VuQuangBao/duocthu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix the F3 out-of-scope gate, close out the V1 feature audit, and clean up project docs

Browse Source 
Also drop .github/ (GitHub-specific CI/CD workflows and ArgoCD
operational scripts) from this mirror -- Gitea auto-picked up
.github/workflows/*.yml as Actions and queued a run against secrets
that don't exist here. Not meaningful outside the GitHub-hosted repo
anyway.
This commit is contained in:
BaoVu2k4
2026-08-25 12:05:00 +07:00
parent 33b16c885b
commit a85b0ccac8
105 changed files with 481 additions and 25857 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs-legacy/README.md
+22 -203
View File
@@ -1,212 +1,31 @@
# Documentation
# docs-legacy — lịch sử dự án
Reverse-engineered from the code in this repository. Every claim here traces to
a file, a command, or an artifact on disk — see
[DOCUMENTATION_PLAN.md](DOCUMENTATION_PLAN.md) for the method and for what was
not verified.
`docs/` là bộ tài liệu chuẩn. Thư mục này **chỉ còn giữ lịch sử**: những gì
không tái tạo được từ code.
## Chọn tài liệu theo việc bạn cần làm
| Mục | Là gì | Vì sao giữ |
|---|---|---|
| `adr/` | 11 Architecture Decision Record | Lịch sử quyết định. `apps/ai-service/routers/rag.py:436` tham chiếu trực tiếp `adr/0006` |
| `pdf-parsing-outlier-catalog.md` | Danh mục ca lỗi khi bóc PDF | **Code đang dùng**: `ingestion/cli.py`, `extract/glyph_order.py`, `extract/models.py` và một test đều trỏ tới file này |
Bộ tài liệu dùng cấu trúc Diataxis: mỗi trang ưu tiên một nhu cầu của người đọc
thay vì cố dạy, hướng dẫn thao tác, liệt kê reference và giải thích kiến trúc
trong cùng một trang.
Nhật ký phát triển chi tiết (`progress-log.md`, ~326 KB, đo lường/ngõ cụt/quyết định
theo từng phiên làm việc) không nằm trong bản mirror này — chỉ có trong repo gốc.
### Học qua thực hành — Tutorial
## Đã xoá 2026-08-24
- [Theo một câu hỏi từ API đến trang PDF nguồn](tutorials/first-grounded-query.md)
Bộ `00-29`, `architecture.md`, các thư mục diataxis (`explanation/`, `how-to/`,
`reference/`, `runbooks/`, `tutorials/`) và các tài liệu kế hoạch/kiểm kê
(`DOCUMENTATION_PLAN.md`, `diataxis-audit.md`, `document-profile.md`,
`bao-cao-kiem-ke-...-2026-08-13.md`, `ke-hoach-showcase-...`,
`pipeline-tu-pdf-den-chatbot-production.md`).
### Hoàn thành một tác vụ — How-to
Lý do: `docs/` đã thay thế chúng và được viết lại từ code, còn bộ này mô tả trạng
thái cũ nên đọc vào dễ hiểu sai. Đã kiểm không file nào trong số đó được code hay
`docs/` tham chiếu.
- [Local development](23-local-development.md)
- [Rebuild và publish corpus](how-to/rebuild-and-publish-corpus.md)
- [Chạy test và evaluation](how-to/run-tests-and-evals.md)
- [Deploy và rollback production](how-to/deploy-and-rollback.md)
- [Lần một request từ người dùng đến evidence](how-to/trace-a-request.md)
- [Production operations](24-production-operations.md)
- [Troubleshooting](25-troubleshooting.md)
Cần đọc lại thì lấy từ lịch sử Git — chúng được track, không mất:
### Tra cứu dữ kiện — Reference
- [Catalog toàn bộ tài liệu](reference/documentation-catalog.md)
- [Repository structure](01-repository-structure.md)
- [API contracts](12-api-architecture.md)
- [Configuration](15-configuration.md)
- [Observability signals](17-observability.md)
- [Known limitations](26-known-limitations.md)
- [Glossary và reason codes](29-glossary.md)
### Hiểu thiết kế — Explanation
- [Pipeline canonical từ PDF đến chatbot](pipeline-tu-pdf-den-chatbot-production.md)
- [Vì sao dùng structured RAG](explanation/why-structured-rag.md)
- [System architecture](02-system-architecture.md)
- [Query understanding](08-query-understanding.md)
- [Retrieval pipeline](09-retrieval-pipeline.md)
- [Generation and grounding](11-generation-and-grounding.md)
- [Audit kiến trúc tài liệu](diataxis-audit.md)
## What this system is
A Vietnamese-language question-answering system over the **Dược thư Quốc gia
Việt Nam 2018** (Vietnamese National Drug Formulary), for doctors and
pharmacists. A user asks a drug question in Vietnamese; the system resolves what
was asked, retrieves the exact monograph section from a vector store, has an LLM
restate it, verifies that restatement against the retrieved text, and returns it
with printed-page citations — or refuses.
Two things distinguish it from a generic RAG app, and both are enforced in code:
- **Retrieval decides what is true; generation only decides how it reads.** A
generated answer is discarded unless every number in it appears verbatim in
the specific evidence block it cites (`rag/grounding.py`) *and* a second LLM
pass confirms the cited block actually says it (`rag/answer.py`).
- **Tables and formulas are quarantined, not linearised.** Content whose numbers
could not be reliably reconstructed from the PDF is never embedded as prose
and never restated; it is surfaced as "check the source page".
Scope boundary: the corpus is **Part 2 monographs only** (printed pages
99–1496). Part 1 general chapters and Part 3 appendices are not ingested.
## Architecture at a glance
```mermaid
flowchart LR
U[Clinician<br/>browser]
CADDY[Caddy 2<br/>TLS + reverse proxy]
WEB["web — Next.js 14<br/>chat UI + BFF routes<br/>+ in-memory rate limit"]
AI["ai-service — FastAPI<br/>RagAgent orchestrator"]
QD[("Qdrant<br/>duocthu_v1<br/>15,100 points")]
PG[("PostgreSQL 16<br/>traces · turns · feedback")]
BR["AWS Bedrock<br/>Cohere embed-v4 · Cohere rerank<br/>Converse generation"]
ING["ingestion — offline batch<br/>PDF → chunks → vectors"]
PDF[/"duoc-thu-quoc-gia-viet-nam-2018.pdf"/]
U --> CADDY --> WEB --> AI
AI --> QD
AI --> PG
AI --> BR
PDF --> ING --> QD
ING --> BR
```bash
git log --oneline -- docs-legacy/00-project-overview.md
git show <sha>^:docs-legacy/00-project-overview.md
```
The `api-gateway`, `auth-service`, `user-service` and `chat-service` directories
in `apps/` contain **only** a `README.md` and a `package.json`. There is no
gateway, no authentication and no chat-service in the request path; `web` calls
`ai-service` directly. See [02-system-architecture.md](02-system-architecture.md).
## Main technology stack
| Layer | Technology | Evidence |
|---|---|---|
| Frontend | Next.js 14 (App Router), React 18, Tailwind, framer-motion | `apps/web/package.json` |
| Backend | Python 3.12, FastAPI, Pydantic Settings, uvicorn | `apps/ai-service/pyproject.toml`, `Dockerfile` |
| Vector store | Qdrant (cosine, 1024-d) | `adapters/qdrant.py`, `ingestion/load/` |
| Relational | PostgreSQL 16 (`psycopg` 3) | `adapters/postgres.py`, `migrations/` |
| Embedding | `cohere.embed-v4:0` on AWS Bedrock | `adapters/embedding.py`, `ingestion/embed/bedrock_cohere.py` |
| Generation | Bedrock Converse API (model id is config) | `adapters/bedrock_converse.py` |
| Rerank | `cohere.rerank-v3-5:0` on Bedrock | `adapters/bedrock_converse.py` |
| PDF parsing | PyMuPDF (`fitz`), pdfplumber for tables only | `ingestion/extract/`, `ingestion/tables/` |
| Observability | Prometheus, OpenTelemetry → OTel Collector → Tempo, Grafana | `rag/telemetry.py`, `infra/docker/` |
| Runtime | Docker Compose on a single EC2 host, Caddy for TLS | `infra/docker/docker-compose.prod.yml` |
| Monorepo | pnpm workspaces + Turborepo (JS side only) | `pnpm-workspace.yaml`, `turbo.json` |
No RAG framework is used. There is no LangChain and no LlamaIndex anywhere in
the dependency set — the orchestration is hand-written in `rag/agent.py`.
## Core runtime services
| Service | Language | Entrypoint | Port |
|---|---|---|---|
| `ai-service` | Python | `apps/ai-service/main.py` → `app` | 8000 |
| `web` | TypeScript | `apps/web/app/` (Next.js) | 3000 |
| `caddy` | — | `infra/docker/Caddyfile` | 80/443 |
| `ingestion` | Python | `python -m ingestion.cli`, `python -m ingestion.load.run` | offline, no port |
## Main data stores
| Store | Holds | Live-path role |
|---|---|---|
| Qdrant `duocthu_v1` | 15,100 chunk points + payload | Every retrieval |
| Qdrant `duocthu_v1__manifest` | One point: corpus sha, model id, dimensions | Startup gate (`bootstrap.py`) |
| PostgreSQL | `rag_retrieval_trace`, `rag_conversation_turn`, `rag_answer_feedback` | Traces + multi-turn history; both fail-open |
| Local disk | `chunks.jsonl`, `monographs.jsonl`, embedding cache | Offline pipeline only |
Redis appears in `infra/docker/docker-compose.yml` (local dev) and in the
pre-existing architecture document. **Nothing in the codebase imports a Redis
client.** It is not deployed in production and not read or written by any code.
## Main pipelines
The single canonical, end-to-end explanation is
[pipeline-tu-pdf-den-chatbot-production.md](pipeline-tu-pdf-den-chatbot-production.md).
The numbered pages below remain the component-level reference.
For a concise demonstration of the changes delivered from 31/07 to 14/08/2026,
use [ke-hoach-showcase-cai-tien-2-tuan.md](ke-hoach-showcase-cai-tien-2-tuan.md).
1. **Ingestion (offline)** — PDF → spans → monographs → chunks → embeddings →
Qdrant. Seven CLI subcommands plus a separate embed/load entrypoint. Has
already been run; re-running the embed step costs real Bedrock spend.
→ [04-ingestion-pipeline.md](04-ingestion-pipeline.md)
2. **Query (live)** — HTTP → understanding LLM call → deterministic route →
Qdrant retrieval → generation LLM call → deterministic grounding →
entailment LLM call → citations → response.
→ [10-rag-orchestration.md](10-rag-orchestration.md)
## Documentation map
**Start here, in order:**
1. [00-project-overview.md](00-project-overview.md) — problem, users, boundaries
2. [02-system-architecture.md](02-system-architecture.md) — components and what is *not* built
3. [03-data-flow.md](03-data-flow.md) — the two end-to-end flows in one page
**For AI/RAG engineers:**
[08-query-understanding.md](08-query-understanding.md) →
[09-retrieval-pipeline.md](09-retrieval-pipeline.md) →
[10-rag-orchestration.md](10-rag-orchestration.md) →
[11-generation-and-grounding.md](11-generation-and-grounding.md) →
[19-rag-evaluation.md](19-rag-evaluation.md).
For the corpus itself: [04](04-ingestion-pipeline.md) →
[05](05-document-parsing.md) → [06](06-document-model-and-chunking.md) →
[07](07-indexing-and-storage.md).
**For backend engineers:**
[12-api-architecture.md](12-api-architecture.md) →
[14-data-stores.md](14-data-stores.md) →
[15-configuration.md](15-configuration.md) →
[18-testing.md](18-testing.md) →
[23-local-development.md](23-local-development.md).
**For frontend engineers:**
[13-frontend-architecture.md](13-frontend-architecture.md) →
[12-api-architecture.md](12-api-architecture.md) (the response contract) →
[16-security.md](16-security.md) (rate limiting lives in the frontend today).
**For DevOps/SRE:**
[20-deployment.md](20-deployment.md) →
[22-ci-cd.md](22-ci-cd.md) →
[17-observability.md](17-observability.md) →
[24-production-operations.md](24-production-operations.md) →
[25-troubleshooting.md](25-troubleshooting.md) →
[21-kubernetes-and-argocd.md](21-kubernetes-and-argocd.md) (unapplied target state).
**For QA:**
[18-testing.md](18-testing.md) →
[19-rag-evaluation.md](19-rag-evaluation.md) →
[26-known-limitations.md](26-known-limitations.md).
**Before planning work:**
[26-known-limitations.md](26-known-limitations.md) →
[27-technical-debt.md](27-technical-debt.md) →
[28-roadmap-from-code.md](28-roadmap-from-code.md).
Terms: [29-glossary.md](29-glossary.md).
## Historical and empirical documents
`architecture.md`, `progress-log.md`, `pdf-parsing-outlier-catalog.md`,
`document-profile.md`, and the ADRs predate the numbered set. They are retained
only for decision history and empirical PDF measurements. Completed plans and
superseded audits were removed; they are not current-state references.