duocthu/docs-legacy/architecture.md

Architecture — Dược Thư RAG Medical Chatbot

Overview

A medical chatbot grounded in the Vietnamese National Drug Formulary (Dược thư quốc gia Việt Nam 2018), built as a microservices monorepo. Users ask drug-related questions through a web chat UI; answers are generated via retrieval-augmented generation (RAG) over the formulary content, always citing the source drug monograph/section, and always carrying a medical disclaimer.

Service responsibilities & communication

Service	Owns	Talks to
api-gateway (NestJS)	Single public entry point; request routing, JWT validation, rate limiting	Routes to auth-service, user-service, chat-service, ai-service over internal REST
auth-service (NestJS)	Signup/login, password hashing, JWT issuance/refresh	Postgres (users); no dependency on other services
user-service (NestJS)	Profile data, preferences, account settings	Postgres (profiles), called by gateway
chat-service (NestJS)	Chat session lifecycle, message history persistence	Postgres (chat_sessions, chat_messages); calls ai-service per user message, persists both turns
ai-service (Python/FastAPI)	RAG orchestration: understand query (LLM) → route to deterministic section/drug retrieval in Qdrant → generate + verify (LLM) → return answer + citations	Qdrant (payload-filtered retrieval), AWS Bedrock (Cohere embed-v4 for query embedding where used, Qwen3 via the Converse API for understanding/generation/entailment, Cohere rerank); conversation history is an in-process dict per RagAgent, not yet durable — see ADR 0008
ingestion (Python, offline batch)	One-time/periodic job: parse PDF → monographs → chunks → embeddings → upsert to Qdrant	Qdrant (write), AWS Bedrock (cohere.embed-v4:0); runs as CLI/CI/k8s Job, never in the live request path
web (Next.js)	Chat UI, auth UI, citation/disclaimer rendering, session list	Calls api-gateway only

Sync vs async: the live chat path (web → gateway → chat-service → ai-service → Qdrant + AWS Bedrock → back) is synchronous request/response. Ingestion is fully decoupled, offline, batch — it populates Qdrant ahead of time and is never triggered by a chat request, since parsing the 37MB PDF and embedding thousands of chunks takes minutes. Internal protocol is REST/JSON for v1; a future gRPC migration is a documented option (see ADRs), not needed now.

Data stores

• Vector DB: Qdrant. Chosen over pgvector because retrieval quality here depends on metadata-filtered ANN search (filter by drug name / section type combined with vector similarity) over a highly structured corpus — Qdrant makes that a first-class, single query. It also scales independently from the transactional Postgres and has a mature Helm chart for the production k8s target. See docs/adr/0001-vector-db-qdrant.md.
• Relational DB: PostgreSQL. One instance, logically separated per service (users/credentials, profiles, chat sessions+messages). As built, only ai-service uses it — for conversation turns (rag_conversation_turn) and retrieval traces (rag_retrieval_trace). The users/profiles/sessions tables belong to services that do not exist yet.
• Redis. Session/refresh-token cache, rate-limit counters, and reserved as the future job-queue backend (BullMQ/Celery) if async admin-triggered re-ingestion or background jobs are added later. Not deployed — nothing in the live path reads or writes Redis, so it was left out of docker-compose.prod.yml rather than run idle.

RAG ingestion pipeline (PDF-specific)

The formulary is a structured per-drug reference, not free prose — the pipeline exploits that structure instead of naive fixed-size chunking. This section reflects an actual empirical investigation of the real PDF (not assumptions) — see docs/adr/0003-pdf-parsing-strategy.md for the full methodology, cross-tool comparison, and validation numbers.

1. Extraction: PyMuPDF (fitz) as primary extractor. This document has no bookmark/outline (doc.get_toc() returns 0 entries — confirmed, do not rely on it) and is a tagged PDF with only a shallow, unusable structure tree (~29 generic H1/P elements covering a fraction of 1668 pages — also confirmed dead-end, not a data source). PyMuPDF's reading order was cross-validated against pdfplumber and opendataloader-pdf on real sample pages: pdfplumber's default text order is unreliable for this layout (scrambles paragraph order, leaks marked-content artifacts) — use it only for its dedicated table-extraction API, never for body text. Raw per-page extraction is persisted to ingestion/data/interim/ so re-segmentation doesn't require re-running the expensive extraction step.
2. Segmentation: drug-entry boundaries are detected via bold-font spans (PyMuPDF span font containing "Bold"), not font-size alone — font size for title/heading spans varies between monographs (confirmed: 10.0pt and 9.5pt both occur for genuine drug-title headings), so bold is the reliable signal, all-caps + short length narrows it to monograph titles specifically. Section headings inside a monograph are also bold spans, cross-checked against a canonical taxonomy (chi_dinh, chong_chi_dinh, lieu_dung, tac_dung_phu, tuong_tac_thuoc, plus real observed extras like ten_thuong_mai "Tên thương mại" not in the book's own documented 19-field list — treat the taxonomy as open/ extensible, not a fixed enum). Multi-line wrapped titles/headings (long Vietnamese names/vaccine names) must be merged across consecutive bold+all-caps lines before matching — this was the single largest source of missed detections in validation. Output: {drug_id, drug_name, source_page_range, sections: {...}} per drug, persisted to ingestion/data/processed/monographs.jsonl and validated both automatically (see ADR 0003) and via manual spot-check in ingestion/notebooks/.
3. Chunking (monograph range only, pp. 99-1496 — see docs/adr/0004-chunking-strategy.md for the full measured rationale): each (drug_id, section_key) pair is the chunk unit; a section stays one chunk if it's under an 800-token ceiling (chars/4 estimate — a validated line, not a guess: whole-corpus measurement across 682 monographs shows ~16 of 18 section types clear it comfortably at their p90). Two sections routinely exceed it — dược lý và cơ chế tác dụng (35.7% of monographs that have it) and liều lượng và cách dùng (29.6%) — sub-chunking is the routine path for those two, not a rare edge case. Oversized sections are split with a sentence-boundary-aware sliding window (~600-700 tokens/sub-chunk, ~1 sentence/50-80 token overlap), never a blind character/line window — PDF line-wrap points are not safe cut points, and a mid-sentence split risks separating an adult/child dosing instruction (a measured, common pattern — outlier catalog item 17) into two chunks. Every chunk carries chunk_id, drug_id, drug_name, section_key, section_display_name, atc_codes, source_page_range, part_index/part_count as Qdrant payload — this is what makes citations possible. Known open gaps (see ADR 0004): sub-compound tagging inside class-level/multi-ATC monographs (25.5% of the corpus) is not yet solved; source_page_range is monograph-level, not sub-chunk-exact; chunking for general chapters/ appendices is a separate, not-yet-designed task; a confirmed header/footer-boilerplate leak into section text (98.4% of monographs affected) must be fixed upstream before this design runs against real data.
4. Embedding + load: AWS Bedrock cohere.embed-v4:0 in batches (cached by (model_id, input_kind, text_sha256) so a reload needs no repeat cloud calls), upserted into Qdrant collection duocthu_v1 (15,100 points, live) keyed by uuid5(chunk_id) for idempotent re-runs; a <collection>__manifest sidecar records the corpus sha/model/dimensions and ai-service refuses to start against a mismatched one (F-05).
5. Batch job, not synchronous: runs as a CLI command locally, and as a Kubernetes Job/CronJob in production — never inside the ai-service request path.

Safety / guardrails

• System prompt instructs the model to answer only from retrieved context, never state a dosage/contraindication/interaction not present in it, always append a disclaimer, and say "not found in the formulary" rather than guess when retrieval is irrelevant.
• Deterministic routing, not a similarity-confidence gate. The live path resolves drug + section by exact payload filter (section_key routing moved contraindication hit@1 from 0.05 to 1.00 — similarity ranking alone was not reliable enough to gate on). A quarantined table/ formula in the retrieved evidence, or missing page provenance, forces VERIFY_PDF/abstain deterministically — never an LLM-reported confidence score. Dense vector similarity search exists (QdrantRetriever.search()) but is reachable only in the legacy no-generator-configured mode, not the live agent path. See ADR 0008.
• Citations from metadata, not LLM prose: the citations list is built directly from retrieved-chunk metadata, independent of what the LLM says, so the frontend can always show verifiable sources.
• Disclaimer enforced at multiple layers: system prompt + a non-LLM-generated static string always appended to the API response + a persistent, non-dismissible UI banner.
• Scoped refusal: out-of-scope questions (e.g. general symptom diagnosis) get a scoped refusal directing to a professional, not an ungrounded general-knowledge answer.

Build roadmap

1. Ingestion pipeline + populated, queryable vector DB. Done when a CLI run populates Qdrant and a test script retrieves the correct drug/section chunk for a sample query — no API, no LLM call yet.

2. ai-service (FastAPI) wrapping RAG + AWS Bedrock. Done when a curl to /v1/rag/query returns a grounded answer with a traceable citation and an always-present disclaimer. Done — live since 2026-08-05, see ADR 0008.

3. auth/user/chat services + api-gateway. Done when register → login → chat message flows end-to-end through the gateway only, persisted in Postgres. Not started — all four directories still hold only a README.md and a package.json. Phases 4-6 were done around this gap, so the live system has no gateway and no auth (see below).

4. Next.js frontend chat UI. Done when a browser user can log in, ask a question, and see a grounded answer with citation + disclaimer banner. Done except the login half — chat, citations, evidence panel and the disclaimer banner are live; there is no login because Phase 3 does not exist. The browser calls apps/web's own route handlers, which proxy directly to ai-service.

5. Containerize + docker-compose local. Done when docker compose up from a clean checkout brings up the full stack and the Phase 4 flow works. Done — 2026-08-10. infra/docker/docker-compose.prod.yml is what production actually runs.

6. Kubernetes/Helm + Terraform + CI + ArgoCD (GitOps) deployment. Done when CI builds/tests/pushes an image and bumps the target environment's Helm values file, the team's ArgoCD instance (see infra/argocd/, docs/adr/0002-argocd-gitops.md) picks up the change and syncs the cluster, and the Phase 4 flow works against the k8s-hosted stack. CI never runs kubectl/helm directly against a cluster. Cloud provider choice (AWS/GCP/Azure) only affects the Terraform module implementations, not this repo's structure. Still the destination — not started, not dropped. Production was shipped ahead of it on an interim single-box setup (see "Deployment as actually built" below), which is a stopgap, not a replacement: ADR 0002 remains Accepted. Nothing here exists yet — infra/k8s/, infra/helm/medical-chatbot/templates/ and infra/terraform/ are empty scaffolds (.gitkeep only), the chart is version 0.0.0, and every ArgoCD Application manifest still carries unresolved TODOs for project, repo URL and destination cluster.

   This phase also includes a repository move to the team's self-hosted Gitea on the company domain, which is where the GitOps repo is intended to live; the project stays on private GitHub until that move is made deliberately. Hard boundary meanwhile: the team's existing git.vinmec.tech/ai-team/gitops repository is reference-only — never push this project into it.

Deployment as actually built (2026-08-10)

Production is not the Phase 6 design. It is a single AWS EC2 t3.large running infra/docker/docker-compose.prod.yml — postgres, qdrant, ai-service, web, and Caddy terminating TLS for realvuxbaro.me via automatic Let's Encrypt. Bedrock is reached through an IAM instance role, so no long-lived AWS key exists on the box or in any env file.

CI/CD is .github/workflows/deploy.yml: a push to master SSHes in, resets the checkout, rebuilds only ai-service/web, runs migrations and health-checks both. It does not touch postgres/qdrant/caddy, so the 15,100 Qdrant points survive deploys (they live in a named volume).

This is an interim setup, not a decision against Phase 6. It exists because a working public demo was needed sooner than the Kubernetes path could deliver one. The expensive prerequisite for that path — containerising both apps — is exactly what this work produced, so the Dockerfiles and compose services port over when the Gitea + team-ArgoCD migration is actually done. Phase 6 and ADR 0002 both stand as written.

See docs/adr/ for architecture decision records. docs/runbooks/ is still empty — the operational knowledge that would live there (restoring a Qdrant snapshot onto a fresh box, what a failed deploy looks like, why uvicorn --reload must not be used on Windows here) currently only exists in docs/progress-log.md.