Add read-only production runtime audit

This commit is contained in:
2026-08-17 11:17:40 +07:00
parent 057d4ed9dc
commit a1de4715a4
106 changed files with 6869 additions and 1782 deletions
+45
View File
@@ -0,0 +1,45 @@
# ADR 0001: Use Qdrant as the vector database
## Status
Accepted
## Context
The RAG pipeline needs a vector store for drug-monograph chunks. The main
alternative considered was **pgvector** (a Postgres extension), which would
let us reuse the Postgres instance already needed for users/chat history —
one fewer moving part to operate.
The corpus is not free-flowing prose: it's a structured per-drug reference
with rich per-chunk metadata (drug name, section type, page range). The
common retrieval pattern this domain calls for is "vector similarity search,
filtered by metadata" — e.g. "search only within chỉ định sections" or
"filter to a specific drug the user named" combined with the semantic query.
## Decision
Use **Qdrant** as a dedicated vector database, separate from Postgres.
## Rationale
- Qdrant gives first-class combined payload-filtering + ANN search in a
single query, which is exactly the retrieval pattern this structured
corpus needs — pgvector supports filtering too, but it's a less natural
fit layered on top of a general-purpose relational engine.
- Vector search becomes its own independent scaling axis, separate from the
transactional Postgres workload (users/chat) — re-indexing or re-ingesting
the formulary doesn't contend with transactional traffic.
- Mature standalone Docker image for local dev, a well-supported Python
client, and a Helm chart for the production Kubernetes deployment target.
- Corpus size (tens of thousands of chunks) is trivial for Qdrant's HNSW
indexing.
## Consequences
- One additional service to operate/deploy/monitor compared to pgvector
(which would ride on the existing Postgres).
- Revisit if operational overhead becomes a real burden at our actual scale,
or if we want tighter transactional consistency between chat data and
retrieval — pgvector remains a viable fallback documented here for that
case.
+68
View File
@@ -0,0 +1,68 @@
# ADR 0002: Use the team's existing ArgoCD instance for deployment (GitOps)
## Status
**Accepted — still the target, not yet implemented.** Not superseded by the
current production setup.
Since 2026-08-10 the project has a *different*, interim deployment: a single
EC2 box running `infra/docker/docker-compose.prod.yml` behind Caddy, deployed
by `.github/workflows/deploy.yml` over SSH. That was built to get a working
demo online, not to replace this decision. Migrating to the team's Kubernetes
+ ArgoCD remains planned work, and the expensive prerequisite — containerising
both apps — is already done, so the Dockerfiles and compose services port over.
Two things must still happen and neither has been started:
1. **Repository move to the team's self-hosted Gitea** (company domain), which
is where the GitOps repo is meant to live. The project stays on private
GitHub until that move is deliberately made. Note the hard boundary already
in force: the team's existing `git.vinmec.tech/ai-team/gitops` repository is
**reference-only** — never push this project into it.
2. **Filling in the scaffolds this ADR assumes exist.** `infra/helm/medical-chatbot/templates/`
and `infra/k8s/**` are empty (`.gitkeep` only), the chart is version `0.0.0`,
and every `infra/argocd/applications/*/app.yaml` still carries unresolved
TODOs for project, repo URL and destination cluster.
## Context
Phase 6 of the build roadmap needs a way to actually deploy the Helm chart to
Kubernetes across dev/staging/prod. The original scaffold (`infra/ci/github-actions/deploy-cd.yml`)
assumed a push-based CI deploy step (CI runs `helm upgrade`/`kubectl apply`
directly against the cluster). The team already runs an ArgoCD instance used
by other projects.
## Decision
Deploy via **GitOps through the team's existing ArgoCD instance** instead of
building a custom push-based CD pipeline. ArgoCD Applications
(`infra/argocd/applications/{dev,staging,prod}/app.yaml`) point at
`infra/helm/medical-chatbot` in this repo; ArgoCD watches the repo and
reconciles the cluster to match.
## Rationale
- Reuses infrastructure the team already operates and trusts, instead of
standing up a parallel deploy mechanism.
- GitOps gives an auditable history of every deploy (it's just git commits
changing values files/image tags) and a built-in rollback path (revert the
commit).
- Removes the need for CI to hold cluster credentials — CI's job shrinks to
"build, test, push image, bump tag," which is a smaller security surface
than "CI can directly mutate the production cluster."
- Prod uses a non-automated `syncPolicy` (manual approval in ArgoCD) while
dev/staging auto-sync, matching normal caution around production changes.
## Consequences
- CI workflows (`infra/ci/github-actions/*.yml`) build/test/push images and
bump the relevant `values-<env>.yaml` image tag + push that commit; they do
**not** call `kubectl`/`helm` against any cluster directly.
- Actual deploy execution and health/sync status live in the team's ArgoCD
UI/CLI, outside this repo — runbooks in `docs/runbooks/` should document how
to check sync status and roll back once the team's ArgoCD instance details
(cluster/server, project, repo URL) are confirmed (see TODOs in
`infra/argocd/README.md`).
- If the team's ArgoCD instance becomes unavailable or this project needs to
fully own its own deploy tooling later, the push-based `deploy-cd.yml`
approach remains a documented fallback.
@@ -0,0 +1,208 @@
# ADR 0003: PDF parsing strategy for the drug formulary — validated empirically
## Status
Accepted (validated against the real 1668-page source PDF, not assumptions)
## Context
The original scaffold's ingestion design (see `docs/architecture.md` history)
assumed generic best practices for structured-PDF parsing: prefer the PDF's
bookmark/outline (`doc.get_toc()`) for section boundaries, fall back to
font-size heuristics. Before writing real ingestion code, this assumption was
tested against the actual `duoc-thu-quoc-gia-viet-nam-2018.pdf` (1668 pages),
because a 1668-page book has enough real-world irregularity that guessing
from a handful of sample pages is not sufficient grounds to trust a parsing
strategy — every claim below was checked against the whole document or a
independently-sourced ground truth, not a small sample.
## What was actually tested
1. **`doc.get_toc()`**: returns **0 entries**. No usable bookmark/outline.
2. **Tagged-PDF structure tree** (`/StructTreeRoot`): exists, but is shallow
— ~29 generic `/H1`/`/P` elements, evidently covering only a small
fraction of the document. Not usable as a structural signal at scale.
Confirmed dead end.
3. **Cross-tool text-extraction comparison** on the same real pages
(a known drug monograph, "Abacavir"):
- **PyMuPDF (`fitz`)**: correct reading order, matches the visual source.
- **pdfplumber** (`extract_text()`): **incorrect** — scrambles paragraph
order on this layout and surfaces a stray marked-content artifact
(`"PB <Header tên thuốc>"`) as if it were visible text. Decision:
pdfplumber is kept **only** for its `extract_tables()` API (a genuinely
different, table-specific algorithm), never for general body text.
- **opendataloader-pdf** (Java-based, benchmarks #1 in public leaderboards
for reading order/tables): correct reading order, and its own computed
font metadata (per-span `font`/`font size` in its JSON output)
**independently agreed** with PyMuPDF's raw span data — two unrelated
tools agreeing on the same font facts is real cross-validation, not
opinion. However, its higher-level paragraph/heading classifier is
**inconsistent**: identical bold section-heading text (e.g. "Dược lý và
cơ chế tác dụng", "Liều lượng và cách dùng") is sometimes promoted to a
markdown `##` heading and sometimes silently merged into the following
body paragraph, for no discernible content-based reason. Conclusion: its
Markdown/heading output is not reliable enough to be the sole
structural signal, but it's a useful independent check and its
header/footer-stripping was notably better than raw PyMuPDF text.
- **docling**: attempted, blocked by a `numpy`/`pyarrow` ABI conflict in
the environment (numpy 2.x vs a pyarrow build expecting numpy 1.x,
pulled in transitively via `torch`/`transformers`). Tested inside an
isolated venv rather than fixed globally, to avoid destabilizing other
tools on the machine. See progress log for current status.
4. **The definitive structural signal — bold font spans**: at the raw
PyMuPDF span level, every section heading and every monograph title is
rendered in a **bold** font (`"...-BoldMT"`), while body text is not.
Italic spans exist too (foreign/Latin species names inline) but are
never confused with headings since they're not bold and appear mid
sentence. This was cross-confirmed by opendataloader's independently
computed font metadata for the same spans (see above) — not a
single-tool guess.
- **Font size is NOT a reliable discriminator on its own**: a monograph
title was observed at both 10.0pt ("ABACAVIR") and 9.5pt ("ACARBOSE")
for equally genuine, equally top-level monograph headings. An earlier
draft of the detector required `size >= 9.8` based on the first
example seen and it silently dropped ~15% of real monographs as a
result — a concrete instance of exactly the "don't generalize from one
example" risk this investigation was meant to guard against. The fix:
drop the size floor; use **bold + all-caps + short line length** for
monograph titles, and **bold** alone (cross-checked against the known
section-name vocabulary) for section headings.
5. **Ground truth for validation**: the book has **two** indexes:
- The front-matter "Danh mục các chuyên luận thuốc" (pages 12-31,
0-indexed): an alphabetical name list with **no page numbers** — useful
only for a name-overlap sanity check, not page-level validation.
- The back-of-book "Mục lục tra cứu" (from page ~1529 printed / ~1528
0-indexed onward): a proper index with **exact page numbers** per
generic-name entry (e.g. `"Abacavir, 101"`), plus brand-name
cross-references (`"Ziagen - Abacavir, 101"`, skipped for ground truth).
This is the real, page-verifiable ground truth and should be used for
any future re-validation, not the front-matter list.
- The front matter's own "NỘI DUNG" (table of contents, page 7 0-indexed)
also gives exact page ranges for the book's 3 parts: general topic
chapters (37-98 printed), individual drug monographs (**99-1496
printed**), appendices (1497-1528), back index (1529+). Any monograph-
boundary detector should be scoped to the 99-1496 printed page range —
scanning the whole book without this scope produces false positives
from front-matter/general-chapter bold-caps lines (org names, decree
headers, chapter titles) that are not drug monographs.
## Decision
- **PyMuPDF is the primary and only general-text extractor.** No TOC
dependency, no reliance on the structure tree.
- **Section/monograph boundary detection uses bold-font spans** (not font
size, not font size + vocabulary alone), scoped to the printed page range
of the actual monograph section (99-1496), with all-caps + short length as
the additional signal narrowing bold spans down to monograph titles
specifically. Multi-line wrapped titles must be merged before matching.
- **pdfplumber is retained only for table extraction** (`extract_tables()`),
never general reading order, per the confirmed scrambling issue.
- **The back-of-book "Mục lục tra cứu" is the ground truth for validation**,
not the front-matter drug list.
- **Validation is a repeatable, whole-document, automated check**, not a
one-time manual read of a handful of pages: a full 1668-page scan runs in
under a minute, so re-running it after every heuristic change is cheap and
should be standard practice before trusting a change.
## Validation results (most recent full-document run)
- Page-verified recall against the back-of-book index: **91.7%** (665/725
primary entries had a detected boundary within ±2 physical pages of the
expected page).
- Remaining misses are overwhelmingly one identified, fixable cause:
**multi-line wrapped ALL-CAPS titles** (long Vietnamese drug/vaccine names
spanning 2+ physical lines) being matched as fragments rather than merged
— not a failure of the bold-span signal itself. A handful of misses are
ground-truth extraction noise (the back-index parser occasionally picks up
a non-drug appendix/table-of-contents line that happens to match the
`"Name, ###"` pattern) rather than real detector failures.
- Expected recall after fixing multi-line merging and cleaning non-drug
entries out of the ground truth: materially higher than 91.7%, to be
re-measured once that fix lands (Phase 1 implementation, not this ADR).
## Follow-up validation: duplicates and cross-page/column data-loss risk
Two further questions were raised and empirically tested against the full
1405-page monograph range (99-1496 printed):
1. **Are any drugs detected twice (real content duplication)?** Scanned for
normalized-name collisions at physically distant pages. Found exactly
**one** candidate: `"GONADOTROPIN"` at physical pages 755 and 1371. On
inspection, this is **not** a real duplicate — page 755 is the genuine
"GONADOTROPIN" monograph (hCG/menotropin/follitropin), while page 1371 is
a different monograph, "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN"
(GnRH-analog drugs), whose title wraps across two lines — the detector
matched only the second line ("GONADOTROPIN"), colliding with the
unrelated monograph's normalized key. This is the **same multi-line
title-wrapping bug** already identified above, now confirmed with a
second concrete example, not a new failure mode. **Conclusion: no real
duplicate monographs found in the corpus**; the multi-line merge fix
(already required for Phase 1) also resolves this collision.
2. **Is the PDF two-column, and can content be lost/corrupted across a page
or column boundary during chunking?** Confirmed via bounding-box
inspection: this document **is** genuinely two-column (left column
x≈44-299, right column x≈308-562, same page). PyMuPDF's block-level
reading order correctly sequences left-column-then-right-column content
(already implicitly validated by the correct Abacavir sample earlier).
However, a **separate, real defect** was found and confirmed: on physical
page 1373, one short run of text has its glyphs in **reversed
(right-to-left) x-order**, producing scrambled output — e.g. `" = tịx 8
yàgn gnàh uềil gnổt( uềihc iổub oàv )magorcim 008 = tịx 4( "`, which
reverses character-by-character back to the correct
`"(4 xịt = 800 microgam) vào buổi chiều (tổng liều hàng ngày 8 xịt = ..."`.
This looks like an isolated PDF-authoring artifact (e.g. an accidental
RTL/BiDi override on one small span during editing), not a systemic
extraction bug. **Initially this scan was scoped to the monograph range
only (1405 of 1668 pages) — an oversight, caught and corrected**:
re-run across all 1668 pages (front matter, general chapters,
monographs, appendices, back index — the entire book, page 0 to the
last page), it still found **exactly 1 affected row, on the same page
1373, and no others** — confirming the defect is genuinely isolated, not
hiding somewhere in the ~260 pages outside the original scan scope.
- The same full-book pass also checked for near-empty pages (<20 chars
extracted): found exactly **6** — physical pages 3, 37, 99, 1495, 1497,
1666 — every one lands exactly at a major section boundary (before
"Các chuyên luận chung" at 37, before "Các chuyên luận thuốc" at 99,
before "Các phụ lục" at 1497, near the book's end at 1666). These are
intentional print-layout blank/separator pages, not lost content —
standard practice to force a new part to start on a fresh page.
**Implications for Phase 1 implementation:**
- Build the pipeline as one **continuous cross-page stream** (text + page
number + bbox per fragment, in reading order), not per-page-isolated
chunks — this is required both for correctly merging multi-line
monograph/section titles (see above) and for never truncating a
paragraph/sentence that spans a page or column break.
- Add an automated **glyph-order sanity check** as a mandatory pass over
100% of pages (not sampled): group text fragments into visual rows by
y-coordinate, verify x-coordinates are non-decreasing, and either
auto-correct (re-sort by x — the fix is deterministic since raw glyph
positions are known) or flag for manual QA. This check is cheap
(~16 seconds over the full monograph range) and should run before every
real ingestion, not just once.
- The book's content must ultimately be captured **from page 0 to the last
page** — but not all of it as drug-monograph chunks: front matter (pages
0-36) is mostly low-value organizational/decree content and can be
largely skipped for RAG purposes; general topic chapters (37-98) and
appendices (1497-1528) are real, valuable content that must be ingested
too, using their own heading-hierarchy-based chunking (not the drug
template) — this was already noted in `docs/architecture.md`'s original
design and is reaffirmed here, not changed. The back-of-book index
(1529+) does not need its own chunks (it's a page-locator, not content)
but remains the validation ground truth.
## Consequences
- The real ingestion pipeline (Phase 1) should implement the bold-span
detector directly (reusing the validated logic, not the exploratory
scratch scripts), scoped to the correct page range, with multi-line
heading merging as a required fix before first real ingestion run.
- Every future change to the segmentation heuristic should be re-validated
with the same whole-document + back-index cross-reference script (or its
Phase 1 equivalent) before being trusted — this is now the project's
standard rigor bar for this pipeline, not an optional nice-to-have.
- `opendataloader-pdf` (Java-based) and `pdfplumber`'s table extraction
remain candidate tools for the table/formula-handling fallback path
described in `docs/architecture.md`; docling's viability is still
unresolved pending the environment fix.
+131
View File
@@ -0,0 +1,131 @@
# ADR 0004: Chunking strategy for drug monographs — validated against real per-section measurements
## Status
Accepted for the monograph range (printed pp. 99-1496) only. General
chapters (pp. 37-98) and appendices (pp. 1497-1528) are explicitly out of
scope — see Consequences.
## Context
`docs/architecture.md`'s original "Chunking" paragraph specified `(drug,
section)` as the chunk unit, a ~500-800 token budget, and a 400-token/
50-overlap sliding window for oversized sections. Those numbers were written
before segmentation existed — a plausible guess, never checked against real
per-section text length.
Phase 1 (extract → segment → validate) is now real, tested code producing
682 real monographs from the full 1668-page source PDF. This session ran
`python -m ingestion.cli run` for real and measured actual per-section
length across the whole corpus with a temporary investigation script
(`ingestion/scratch/chunking_stats_survey.py`, deleted after this ADR
captured its findings, per this project's investigation-script rule) —
something that had never been measured before this ADR.
## What was actually measured (whole corpus, 682 monographs)
- Sections per monograph: min 11, median 17, max 19 (of ~18-19 known
section keys in `segment/vocab.py`'s open taxonomy).
- Whole-monograph length: median 11,480 chars, p90 19,068 chars, max 38,786
chars.
- Per-section length, converted to a **chars/4 token estimate — an
estimate, not a real tokenizer count**:
- Most of the ~18 section types sit comfortably under 800 estimated
tokens even at their p90 (e.g. `chi_dinh` p90≈268 tok, `dang_thuoc_va_
ham_luong` p90≈115 tok, `tac_dung_khong_mong_muon` p90≈481 tok).
- **Two sections routinely exceed 800 tokens**:
`duoc_ly_va_co_che_tac_dung` (242 of 678 monographs that have this
section, 35.7%, max ≈3542 tok) and `lieu_luong_va_cach_dung` (200 of
675, 29.6%, max ≈3631 tok).
- A smaller tail also exceeds it: `than_trong` (25/680, 3.7%),
`tuong_tac_thuoc` (22/642, 3.4%).
- This means: the original 800-token ceiling is directionally correct
(it clears ~16 of 18 section types at their p90 with room to spare),
but "sub-chunk in that case" is not a rare hedge as originally implied
— it is the routine path for roughly a third of all monographs, on two
specific, named, high-clinical-importance sections (mechanism of
action and dosing).
**A separate, blocking bug was found while gathering this data, not fixed
by this ADR** (out of scope — belongs to `extract`/`segment`, owned by a
parallel session at the time of writing): running header/footer
boilerplate ("DTQGVN 2" + page number + repeated drug name, tagged
`column="full_width"` in `extract/spans.py`) is never filtered out of
section body text before it reaches `SectionSpan.text`. Measured:
1,374 of 11,409 sections (12.0%) contain a literal "DTQGVN" string
mid-text; 671 of 682 monographs (98.4%) have at least one affected section
(e.g. MORPHIN SULFAT's `lieu_luong_va_cach_dung`: `"...Nếu\nDTQGVN 2\n1009\n
Morphin sulfat\nuống viên thuốc..."`). This is `docs/pdf-parsing-outlier-
catalog.md` item 13's known risk, measured whole-corpus for the first time
here. **Chunking must not run against real data until this is fixed**
otherwise boilerplate is baked into embeddings and can surface mid-sentence
in a chunk shown to a doctor or pharmacist.
## Decision
1. **Chunk unit stays `(drug_id, section_key)`** — matches
`segment/models.py`'s existing `Monograph.sections: Dict[str,
SectionSpan]`, matches how a doctor/pharmacist would query ("what does
it say about liều dùng"), and lets a citation point at one clinical
section rather than a whole 2,000-19,000-char monograph.
2. **Token budget: keep the 800-token ceiling** (chars/4 estimate) as the
split trigger. Below it, a section is one chunk, verbatim. This is now a
validated choice, not a guess.
3. **Sub-chunking only applies to the long-tail sections above** (~30-36%
of monographs for the two named sections, a few percent for the rest).
Method: **sentence-boundary-aware sliding window**, replacing the
originally-guessed fixed-character window. Target ~600-700 tokens per
sub-chunk (headroom under the 800 ceiling), ~1 sentence / 50-80 token
overlap between adjacent sub-chunks. Split only at a sentence boundary
(`.`/`;`/`:` followed by whitespace + capital letter), explicitly not
treating a Vietnamese decimal comma (e.g. "0,425") as a boundary.
4. **Why sentence-aware, not line- or character-based**: `assembler.py`
joins `body_lines` one line per PyMuPDF *span*, i.e. one PDF visual
line-wrap point — not a semantic paragraph or sentence boundary. A blind
character/line window can split a sentence mid-way. This is a real,
measured risk here, not theoretical: outlier-catalog item 17 found
adult/child dosing splits ("Người lớn"/"Trẻ em") appear on 1,121 of
~1,400 monograph-range pages — a chunk boundary landing inside one of
those sentences would be a patient-safety-relevant defect, not a
cosmetic one.
5. **Chunk metadata / provenance** (extends the existing `drug_name,
section_type, source_page_range, chunk_id` list in `docs/architecture.md`
— per CLAUDE.md's provenance rule): `chunk_id`
(`{drug_id}__{section_key}__{part_index}`), `drug_id`, `drug_name`,
`section_key`, `section_display_name`, `atc_codes` (inherited from the
monograph — enables ATC-class-filtered retrieval), exact per-chunk
`source_page_range` and `printed_page_range`, `part_index`/`part_count`
(`0`/`1` for un-split sections, keeps the schema uniform across all chunks).
6. **Schema v4 separates source from retrieval context.** `source_text` is the
exact contiguous source span and is the basis for lossless reassembly and
page provenance. `text` may prefix repeated route/population labels so a
continuation chunk is independently safe to retrieve. Those retrieval-only
prefixes are recorded in `context_labels` and may not alter `source_text`.
Token counts use `cl100k_base`, not the earlier chars/4 estimate.
## Consequences
- **Scope**: this decision covers the monograph range only. General
chapters and appendices contain real tables and 2D stacked-fraction
formulas (`docs/document-profile.md`, investigation in progress as of
this ADR) that need their own structural survey before any chunking rule
can be designed for them — do not extend this ADR's rules to those ranges
without a fresh investigation.
- **Hard prerequisite**: the boilerplate-leakage bug described above must
be fixed in `extract`/`segment` before this chunking design is run
against real data for ingestion. This ADR does not fix it.
- **Known gap — sub-compound tagging inside class-level monographs**: 25.5%
of the corpus has more than one ATC code per monograph (outlier item
12a), e.g. "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ" documents dosing for 7
different analogues inside one `lieu_luong_va_cach_dung` section. No
reliable structural signal was found in sampled text to split a section
by sub-compound — a chunk from this section is tagged with the class
name only, not the specific analogue a query might target. Deferred to
golden-dataset-driven eval rather than guessed at now.
- **Resolved — sub-chunk page precision**: schema v4 derives exact physical
support from the contiguous `source_text` span and maps it to verified
printed folios. Missing or ambiguous support fails readiness rather than
falling back to monograph-level provenance.
- **Implemented**: the sentence/label-aware splitter is in
`ingestion/ingestion/chunk/` with regression tests for dose continuations,
compound label boundaries, parent route context and lossless reassembly.
@@ -0,0 +1,216 @@
# ADR 0005: `segment/` output contract needed by `chunk/` — structure-preserving, not flattened
## Status
Proposed. **Contract/schema only — no implementation.** `segment/models.py`,
`segment/assembler.py`, and `segment/io.py` are actively owned by a parallel
session on the same checkout at the time of writing; this ADR specifies what
`chunk/` needs from `segment/`'s output precisely enough to implement and
test, but does not touch those files itself. Supersedes part of ADR 0004
(see "Relationship to ADR 0004" below) — ADR 0004's `(drug_id, section_key)`
chunk-unit-as-leaf assumption is corrected here to `(drug_id, section_key)`
as a **parent**, with sentence-window splitting demoted from primary
strategy to fallback.
## Context
ADR 0004 designed chunking against `segment/models.py`'s current output:
`SectionSpan.text` is a single flattened string per section (`"\n".join(
body_line.strip() for ...)`), with all per-line style (`Span.bold`) and
per-line page position discarded once the string is built (confirmed by
reading `assembler.py`: `body_lines.append(span.text.strip())` keeps only
`span.text`, nothing else). Review of ADR 0004 surfaced four real problems
that trace back to this flattening, not to the chunking algorithm itself:
1. **A section is not a single semantic unit.** `liều lượng và cách dùng`
and `tương tác thuốc` routinely contain multiple distinct facts (dosing
per patient population, dosing per organ-function impairment, multiple
separate drug interactions) that a doctor may want to retrieve
independently. Measured: an explicit population marker ("Người lớn"/
"Trẻ em"/"Trẻ sơ sinh"/"Suy thận"/"Suy gan" immediately followed by `:`
or `.`) appears in **303 of 675 monographs (44.9%)** that have a `liều
lượng và cách dùng` section — this is common, not an edge case.
2. **A blind sentence-boundary sliding window (ADR 0004's original
sub-chunking method) can still split two different facts into the same
chunk, or split one fact across two chunks**, because it has no way to
know a population/interaction boundary exists — that information exists
in the source (as a bold or otherwise visually distinct sub-heading, per
direct reading of MORPHIN SULFAT/VITAMIN D section text: lines like
"Thuốc uống", "Cách dùng:" render as isolated bold short lines in the
PDF) but is discarded before `chunk/` ever sees it.
3. **Tables inside the monograph range are not addressed at all.** ADR 0004
implicitly assumed monograph-range sections are prose. `docs/pdf-parsing-
outlier-catalog.md` item 19 already documents a real table (dosing by
renal function, HSV/CMV columns) inside a monograph body (Foscarnet
natri, physical page 698) — flattening a table's rows into
newline-joined body text destroys its row/column structure exactly the
way outlier item 7 already describes for the appendix's 2D nomogram
table. A whole-range survey to size this properly is in progress
alongside this ADR (see "Not yet resolved" below).
4. **Provenance is section-level, not chunk-level**, because per-line
`physical_page`/`y0` (which `Span` already carries — see
`extract/models.py`) is discarded at the same flattening point. For a
section spanning several physical pages, a sub-chunk built from its
final third currently has no way to know its own real page — it can
only inherit the whole monograph's `source_page_range`. For medical
citations this is not precise enough.
**A fifth, independently-found data-quality bug makes precise provenance
even more necessary, not less**: the corpus's last-processed monograph
(ZOLPIDEM) is never closed until true end-of-stream, and `assembler._classify`
calls `match_section()`/`match_section_with_inline_value()` on every span
with **no `in_monograph_range` gate** (unlike `_TextEvent` handling, which
does check it). A spurious bold-text match on physical page 1655 — deep in
the back-of-book "Mục lục tra cứu" brand-name index, confirmed by reading
that page directly — overwrote ZOLPIDEM's real `tương tác thuốc`
`SectionSpan` with an empty one and corrupted its `source_page_range` to
`[1492, 1655]`. This is real content loss (measured: exactly 1 monograph
affected, the last one processed — every other monograph is closed on
schedule by the next monograph title, which *is* range-gated). Flagged for
the session that owns `extract`/`segment`, not fixed here.
## Decision
Extend `segment/models.py`'s `SectionSpan` with a structured, line-level
representation, additive to (not replacing) the existing flat `text` field
`chunk/` becomes a real, structure-aware consumer instead of re-deriving
structure from a flattened string via ad hoc regex.
### New/changed types (`segment/models.py`)
```python
@dataclass(frozen=True)
class BodyLine:
text: str
physical_page: int
y0: float
bold: bool # Span.bold, preserved instead of discarded
@dataclass
class SectionSpan:
key: str
display_name: str
heading: Heading
text: str # UNCHANGED meaning, kept for
# backward compat (see invariant below)
lines: List[BodyLine] = field(default_factory=list) # NEW
```
`lines` carries exactly the per-line signal `chunk/` needs to do its own
job (population/subheading detection, precise page provenance) without
`segment/` having to know anything about chunking — `segment/`'s
responsibility stays "detect boundaries and preserve source structure," not
"decide what a retrieval unit is" (Clean Architecture / SoC, per
CLAUDE.md). Specifically, this is deliberately **not** a `is_subheading:
bool` field computed by `segment/` — classifying "is this line a
subheading a chunker should split on" is a chunking-time decision (what
counts as a good split point can vary by strategy/eval results), not a
segmentation-time one. `segment/` should stop discarding the raw signal
(`bold`, `y0`, `physical_page`) it already has per span; it should not also
start doing chunk-shaping judgment calls.
### Invariants
1. `text == "\n".join(l.text for l in lines).strip()` for every
`SectionSpan`, for the lifetime of this contract — `lines` is a strictly
additive refinement, never a divergent second source of truth. Any
change to how body text is assembled (e.g. the boilerplate-stripping fix
already applied by the other session) must update both fields from the
same filtered span list, not `text` alone.
2. `lines` is in reading order, matching the order `text`'s lines already
implicitly have.
3. Every `BodyLine.physical_page` satisfies `detector.in_monograph_range`
for a `Span` on that page — i.e., **no line in any `SectionSpan.lines`
may come from outside the monograph's real printed-page range**. This is
the ZOLPIDEM bug's exact failure mode stated as an invariant: it was
violated (a spurious section event was accepted from a fully
out-of-range page precisely because no such check existed for section
*events*, only for body *text* events). Enforcing this invariant closes
that bug as a side effect, but the invariant is stated here as a
contract requirement independent of any specific fix implementation.
4. Every currently-open monograph must be finalized exactly once, at either
(a) the next monograph title, or (b) true end-of-stream — with no third
path (e.g., a stray out-of-range section match) able to silently mutate
an already-"complete" monograph's sections after point (a) would
otherwise have applied. (This is a restatement of invariant 3 from the
monograph-lifecycle side, not a new requirement.)
### Migration impact
- **`segment/io.py`** (`_monograph_to_dict`/`_monograph_from_dict`,
`write_monographs_jsonl`/`read_monographs_jsonl`): additive — serialize
`lines` alongside the existing `text`/`heading` fields per section.
Existing consumers reading only `text` (e.g. `segment/atc.py`'s
`extract_atc_codes`, which regexes over `SectionSpan.text`) need no
change, per invariant 1.
- **`ingestion/data/processed/monographs.jsonl`**: schema grows a new
optional-shaped field (`sections[key].lines`). No `schema_version` field
currently exists in the serialized dict (checked `io.py` directly) —
worth adding as part of this change, both for this migration and because
`docs/architecture.md` already assumes "collection aliasing allows
re-ingesting with a changed chunking strategy," which implies the
ingestion output itself should be able to declare which schema shape it
is.
- **Existing 110 tests**: unaffected if invariant 1 holds — no assertion in
the current suite inspects `lines` (it doesn't exist yet), and `text`'s
value/semantics are unchanged.
- **New tests required** (this ADR specifies them; implementation and the
actual test code are not part of this ADR):
1. Regression test reproducing the ZOLPIDEM failure shape: a synthetic
span stream — last monograph's title and real sections, followed by
spans whose `printed_page` is out of `in_monograph_range` but whose
text matches a `vocab.py` section label — asserting the monograph
closes with its real sections intact and the out-of-range spurious
match is ignored, not accepted.
2. `SectionSpan.lines` fixture test: using the real MORPHIN SULFAT
boilerplate-fix fixture already in `tests/test_segment_assembler.py`,
assert `lines` preserves the correct `bold`/`physical_page`/`y0` per
retained line (and that stripped boilerplate lines are absent from
`lines` too, not just from `text`).
3. Round-trip test: `write_monographs_jsonl``read_monographs_jsonl`
preserves `lines` exactly (dataclass equality per line).
4. Whole-corpus invariant-1 check: for a real `cli run` output, assert
`text == "\n".join(l.text for l in lines).strip()` holds for every
section of every monograph, not a sample.
## Relationship to ADR 0004
ADR 0004's chunk-unit decision (`(drug_id, section_key)`) is **not**
discarded — a section is still the natural *parent* grouping (matches how a
clinician thinks, matches `Monograph.sections`). What changes: ADR 0004
described a section as directly *the* chunk when under the 800-token
ceiling, with sentence-window splitting as the fallback for oversized
sections. Per the review above, splitting must instead **first** attempt to
break at real structural boundaries available in `SectionSpan.lines` (a
bold, short, isolated line — the same "subheading" shape already visually
confirmed for route-of-administration/population sub-headers — or an
explicit population/organ-function marker), with the sentence-window method
demoted to a fallback for the remaining prose that has no such marker. The
exact splitting algorithm (how a "subheading-shaped line" is defined
precisely, in code) is a `chunk/`-side implementation detail *enabled* by
this contract, not decided by it.
## Not yet resolved (explicitly out of scope for this ADR)
- **Table/formula content blocks.** A separate whole-monograph-range survey
(pdfplumber `find_tables()` + PyMuPDF math-symbol scan, physical pages
98-1494 excluding blank page 99 — the exact set `detector.
in_monograph_range` accepts, not an assumed offset) is in progress at the
time of writing, per explicit user instruction to measure before deciding
a table/formula chunk-unit strategy. This ADR's `BodyLine`
contract covers **text content only**; a table/formula region should
*not* currently be flattened into `BodyLine`s (doing so would repeat
exactly the "destroys row/column meaning" mistake outlier item 7 already
documents) — but the precise `ContentBlock`/table-row/formula-unit shape
is deferred to a follow-up revision of this ADR once the survey reports
real numbers (how many monographs/sections affected, page-break
continuation frequency, multi-tier headers, merged cells, footnotes).
- **Paragraph-boundary detection** (grouping consecutive `BodyLine`s into a
flowing paragraph vs. a new one) is left to `chunk/`, using the same
kind of y-gap heuristic `segment/merge.py` already validates for
multi-line title wraps (`_MAX_LINE_GAP_PT`) — `BodyLine.y0` is sufficient
raw signal for `chunk/` to compute this itself; `segment/` does not need
to pre-compute paragraph grouping.
- **The actual `chunk/` splitting implementation** (subheading detector,
population-marker regex, sentence-window fallback) is not part of this
ADR — this ADR defines the data contract that implementation will consume.
@@ -0,0 +1,167 @@
# ADR 0006: chunks must carry references to lifted table/formula blocks
## Status
Accepted and implemented in schema v4. Resolves the item ADR
0005 explicitly deferred ("Table/formula content blocks … the precise
`ContentBlock`/table-row/formula-unit shape is deferred to a follow-up
revision of this ADR once the survey reports real numbers"). The survey has
reported.
## Context
`segment/` now lifts table and formula regions out of section prose and
quarantines them (ADR 0003 lineage, outlier-catalog items 7, 8, 24, 25).
That was the right move — linearised, AMPICILIN VÀ SULBACTAM's
Cockcroft-Gault fraction read as `Clcr (ml/phút) = 72 x creatinin huyết
thanh`, i.e. a division presented as a multiplication, in a renal-dosing
section.
But `chunk/models.py` has no field that refers to a lifted block. Measured on
the current whole-corpus output:
| quantity | value |
|---|---|
| lifted blocks represented by descriptor chunks | 151, all quarantined |
| sections affected | 103 |
| **blocks in `lieu_luong_va_cach_dung`** | **125** |
| unverified header rows admitted to embedding text | **0** |
So three quarters of everything removed from prose was removed from the
dosing section, in a drug formulary, for an audience of doctors and
pharmacists.
**The failure this creates is silent, not visible.** A chunk of AMPICILIN VÀ
SULBACTAM's `lieu_luong_va_cach_dung` is grammatical, complete-looking prose
with the renal-dosing table absent and nothing marking the absence. Retrieval
ranks it, the model answers from it, and neither has any way to know a table
was taken out. A visible error would be safer than this.
A second, quieter failure: a table is currently **unreachable**. Nothing in
the index represents it, so "bảng liều theo chức năng thận của ampicilin"
cannot retrieve it even in principle.
## Decision
Chunks reference blocks; blocks' content never becomes embedded text.
### 1. `Chunk` gains typed attachments
```python
@dataclass(frozen=True)
class ChunkAttachment:
block_id: str
kind: str # "table" | "formula"
shape: str # simple_table | multi_level_or_merged_header |
# cross_page_continuation | formula_2d
physical_page: int
printed_page: int
bbox: List[float]
quarantined: bool
header_row: List[str] = () # always empty until separately verified
source_crop: str | None = None
@dataclass(frozen=True)
class Chunk:
...
chunk_kind: str = "prose" # "prose" | "block_descriptor"
attachments: List[ChunkAttachment] = ()
has_quarantined_content: bool = False
```
`has_quarantined_content` is derivable from `attachments`, and is serialized
anyway. A consumer that never looks at `attachments` must still be unable to
miss the fact — the whole defect being fixed here is a consumer not knowing
what it was not told.
### 2. One descriptor chunk per block, built from metadata only
A block also gets its own chunk so it is retrievable at all:
```
chunk_id = "{drug_id}:{section_key}:block:{block_id}"
chunk_kind = "block_descriptor"
text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
trang in 204."
```
The text is assembled only from verified metadata: drug name, section display
name, block kind and printed page. **No cell value or inferred header appears.**
The earlier proposal to use `pdfplumber.find_tables()`'s first row was rejected
after corpus audit: a guessed first row can be a body row or can merge numeric
relationships. Until a separate human-verified header dataset exists,
`header_row` is embargoed for every shape and serialized as empty.
### 3. The answer layer's obligations (binding on `ai-service`)
These obligations are implemented across `ingestion/` and `ai-service` and are
enforced by tests/readiness gates.
1. A retrieved chunk with `has_quarantined_content: true` **must** cause the
answer to state that a table or formula exists at the cited page, and to
surface its rendered crop. The answer may not present itself as complete.
2. A `block_descriptor` chunk may be answered **only** with the crop. It must
never be paraphrased, and its `header_row` must never be presented as the
table's content.
3. No chunk carrying a quarantined attachment may be used to state a numeric
dose. If the dose is in the table, the answer is the crop plus the page.
### 4. `schema_version`
`monographs.jsonl` and the chunk output both gain `schema_version`. ADR 0005
flagged its absence; a schema that now has two chunk kinds and an attachment
list cannot be safely consumed without one.
## Alternatives rejected
- **Flatten the block into the chunk text.** This is the defect, not the fix
— it reproduces `Clcr = 72 x creatinin` exactly.
- **Chunk the block's linearised text as an ordinary chunk.** Worse than
flattening: it makes unsafe text independently retrievable *as prose*, with
its quarantine flag one dereference away from being ignored.
- **Drop the blocks.** Silent loss, and contrary to the standing rule that
unreconstructable content is quarantined with full provenance, never
deleted.
- **Rely on the prose saying "xem bảng".** The prose often does not, and a
retrieval layer cannot act on an unstructured hint.
- **Wait for row/column reconstruction and do this once.** Reconstruction is
days of work and would leave the corpus unchunkable meanwhile; worse, it
would make the schema question look answered when the *silent-incompleteness*
problem is independent of whether the rows are recovered. Reconstruction
later populates `rows` on the same attachment without touching consumers.
## Why a crop is a legitimate answer, not a placeholder
For doctors and pharmacists a rendered crop of the source page is the
highest-fidelity response available: it *is* the book, and it is verifiable at
a glance. Reconstruction earns its keep for a different job — comparing or
combining values across drugs, which is the synthesis use case this product
exists for — not for single-table lookup.
## Invariants and gates
Added to `cli chunk-ready` and to the chunk stage's own tests:
1. `section_with_lifted_block_but_no_chunk_reference = 0`
2. `attachment_block_id_unknown = 0` — every referenced id exists on the
monograph
3. `attachment_without_page_or_bbox = 0`
4. `block_text_leaked_into_chunk_text = 0` — no chunk's embedded text
contains a quarantined block's text
5. `descriptor_chunk_count == block_count`
6. `descriptor_chunk_without_attachment = 0`
7. `attachment_header_row_present = 0`
8. `descriptor_with_unverified_columns = 0`
9. `descriptor_range_not_attachment_page = 0`
10. `attachment_without_printed_page = 0`
## Consequences
- Prose chunks shrink slightly in trustworthiness terms but grow in honesty:
the ones missing a table now say so.
- The current candidate index gains 151 descriptor chunks,
each cheap and none carrying unsafe text.
- `ai-service` cannot answer a dosing question from prose alone for the 103
affected sections without violating a stated contract.
- The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas
answerable as crops today, which they are not now.
@@ -0,0 +1,210 @@
# ADR 0007: Conversational reasoning RAG — state, bounded loop, and how it is measured
**Status:** superseded by ADR 0008 (2026-08-07). See the note below before
reading this as a description of anything currently running.
**Supersedes:** nothing. Extends ADR 0005 (segment output contract) and ADR 0006
(quarantined block references) rather than replacing them.
> **2026-08-07 — why this was superseded, not deleted.** An independent
> 7-agent audit on 2026-08-06 found `bootstrap.py` never constructs any of
> `rag/conversation.py` / `rag/reasoning.py` / `rag/conversational.py` — the
> live agent (`rag/agent.py::RagAgent`, wired in since the F-03 rebuild on
> 2026-08-06) is a fixed one-shot pipeline (understand → route → retrieve
> once → generate → ≤2 same-claim entailment retries), not the PLAN/RETRIEVE/
> ASSESS/REFINE/VERIFY loop or the `Focus`/`ConversationState`/TTL state
> design below. This was a real, deliberate pivot mid-implementation, not an
> abandoned-but-still-intended plan: `rag/agent.py`'s own module docstring
> says outright that `ConversationalLoopService` + `conversation.py` were
> replaced because "the LLM reads a plain turn history and resolves
> ['thuốc đó' / 'còn liều thì sao'] itself" — simpler than maintaining
> `Focus`/TTL/turn-budget state by hand, and proven live across many
> multi-turn conversations since. Section 6 below ("Refused: an LLM
> confidence score as the loop's uncertainty signal") is the clearest
> evidence this is a genuine architecture change, not a gap: the live system
> now uses exactly that — an LLM sufficiency/clarify judgment — as its
> ask-or-answer signal, the opposite of what this ADR chose.
>
> The three modules this ADR specified (1,314 lines) and their five dedicated
> test files (42 tests) were deleted on 2026-08-07 rather than left as dead
> code, once confirmed to have zero live importers anywhere
> (`bootstrap.py`/`main.py`/`agent.py`/`answer.py`/`routers/rag.py`). This
> document is kept, unedited below this notice, as the historical record of
> why that design was chosen and what it traded off — see ADR 0008 for what
> actually runs today, including what this ADR got right that ADR 0008
> still owes (a real request-scoped time/call budget — F-08, still open; a
> durable, cross-worker conversation store — currently an in-process dict).
## Context
The service answers one question at a time. `POST /v1/rag/query` carries no
conversation id, `apps/chat-service` holds zero source files, and every request
re-resolves the drug from scratch. Three consequences, all observed in the UI on
2026-08-05:
- `paracetamol` alone is refused rather than asked about.
- `liều dùng paracetamol cho người lớn` returns the identical answer to
`liều dùng paracetamol` — the qualifier is not used at any stage.
- A follow-up such as *"còn trẻ em thì sao?"* cannot work at all, because
nothing carries the drug forward.
The owner's requirement is a **conversational reasoning RAG**: history, an
internal reasoning stage, and a bounded self-improvement loop.
The binding constraint is that this is a drug formulary for clinicians. Every
capability below is designed so that adding it cannot widen what the system is
allowed to assert.
## Decision
### 1. Conversation state
Two stores with different jobs, deliberately not merged.
**`Focus` — structured, drives routing.** This is what makes *"còn trẻ em thì
sao?"* resolvable without an LLM.
| Field | Purpose |
|---|---|
| `drug_id`, `drug_name` | The drug under discussion |
| `section_key` | The attribute last answered |
| `population` | `nguoi_lon` / `tre_em` / `phu_nu_co_thai` / … |
| `verbosity` | `concise` \| `detailed`, set when the user asks |
| `set_at_turn` | Turn index each field was last set |
**`ConversationState` — the whole record.**
```
conversation_id
recent: tuple[Turn, ...] # last K turns, verbatim
summary: str # rolling prose summary of everything older
focus: Focus
turn_count: int
```
A `Turn` carries `role`, `text`, `at`, and — for assistant turns — the
`drug_id`, `section_key` and `evidence_ids` that produced it. Storing the
evidence ids is what lets the planner answer a follow-up **from evidence
already retrieved** instead of retrieving again.
**Carry-over is never silent.** An inherited `drug_id` that is wrong is a
wrong-drug answer, so any answer built on inherited focus must name what it
inherited: *"Về Metformin, ở trẻ em: …"*. This is a hard rule, not a
presentation preference.
**Focus expires.** A field older than `FOCUS_TTL_TURNS` (6) is dropped rather
than inherited. Conversations drift, and a drug from ten turns ago is not
context, it is a hazard.
### 2. Recent history and summary
- `recent` holds the last **K = 6** turns verbatim (three exchanges).
- When a turn falls out of `recent`, it is folded into `summary`.
- `summary` is regenerated at most every **S = 4** turns, capped at **400
tokens**; `recent` is capped at **2000 tokens**, oldest dropped first.
- **The summary records what was discussed, never clinical content.** It may
say *"đã hỏi liều dùng của Metformin cho người lớn"*; it may not carry a dose.
A dose restated from a summary would have no citation and could not be
grounding-verified — the check compares against retrieved evidence, and a
summary is not evidence.
### 3. Reasoning loop
```mermaid
flowchart TD
A[User turn] --> B[UNDERSTAND<br/>resolve against Focus]
B --> C{Clarify signal?}
C -->|ambiguous drug / no attribute /<br/>multi-attribute| Z[ASK — 1 turn, no loop]
C -->|no| D{Simple?}
D -->|drug + section resolved,<br/>no follow-up ambiguity| E[RETRIEVE]
D -->|complex / decomposable| P[PLAN<br/>sub-questions + retrieval set]
P --> E
E --> F[ASSESS sufficiency]
F -->|insufficient AND rounds left| R[REFINE query] --> E
F -->|sufficient OR rounds exhausted| G[GENERATE]
G --> H[VERIFY<br/>grounding + coverage]
H -->|ungrounded / off-target,<br/>repairs left| G
H -->|grounded| Y[RESPOND]
H -->|repairs exhausted| X[FALL BACK<br/>verbatim source]
F -->|exhausted AND still thin| Z
```
**Continue conditions** — a round is spent only when all hold:
1. `retrieval_rounds < MAX_RETRIEVAL_ROUNDS` (2)
2. the assessor named a *specific* missing thing (a section, a population, a
second drug) — "feels incomplete" is not a reason to spend a round
3. the refined query differs from every query already tried this turn
**Stop conditions** — any one ends the loop:
- sufficiency satisfied
- budget exhausted (rounds, LLM calls, wall-clock, tokens)
- a clarify signal fires (these bypass the loop entirely — asking beats guessing)
- grounding verification fails after `MAX_REPAIRS` (1) → extractive fallback
**Fast path.** When the drug resolves and `SectionResolver` returns a section
and no clarify signal fires, the loop is skipped: retrieve → generate → verify.
This is the majority path and it costs one LLM call.
### 4. Budgets
| Limit | Value | Enforced at |
|---|---|---|
| `MAX_RETRIEVAL_ROUNDS` | 2 | loop guard |
| `MAX_REPAIRS` | 1 | loop guard |
| `MAX_LLM_CALLS` per turn | 4 | budget object, checked before each call |
| `MAX_WALL_CLOCK_MS` | 20000 | checked between stages |
| `MAX_EVIDENCE_TOKENS` | 12000 | evidence assembly, oldest-dropped |
| `FOCUS_TTL_TURNS` | 6 | state update |
The budget is a single object threaded through the loop and **decremented
before** each call, so exhaustion degrades to the best answer so far rather
than to an error.
### 5. Integration
New domain modules, no SDK imports:
- `rag/conversation.py``Focus`, `Turn`, `ConversationState`, window and
focus-update rules. Pure; the follow-up resolution in it needs no LLM.
- `rag/reasoning.py` — the loop, its budget, and its stage protocols.
- `rag/ports.py``ConversationStore` (load/save), `Summariser`, `Planner`,
`SufficiencyAssessor`. Each has a deterministic no-LLM default so the whole
loop runs offline.
New adapter: `adapters/postgres.py` gains `PostgresConversationStore`.
Unchanged and still binding: `GroundedAnswerService` remains the single-turn
engine; `grounding.verify` gates every generated answer; `VERIFY_PDF` evidence
is never generated over.
### 6. Measurement
A capability that cannot be shown to help does not ship. Three modes are run
over the same cases — `single-shot`, `+history`, `+reasoning-loop`:
| Metric | Answers |
|---|---|
| follow-up resolution accuracy | does *"còn trẻ em thì sao?"* reach the right drug+section+population |
| on-target rate | does the answer contain the population/attribute actually asked for |
| grounding rejection rate | does reasoning make fabrication more or less likely |
| clarify rate / clarify precision | does it ask when it should, and only then |
| median + p95 latency, LLM calls, tokens per answered turn | what the capability costs |
The evaluation set is a **new multi-turn golden file** — the existing
`golden_e2e_v1.csv` is single-turn by construction and cannot measure any of
this. Counters land in `rag/metrics.py` and on the existing Grafana dashboard.
## Consequences
**Accepted.** More moving parts and more tokens per turn; a stateful service
where there was a stateless one; a summary that must be kept free of clinical
content by rule rather than by mechanism.
**Refused.** An LLM confidence score as the loop's uncertainty signal. The
signals used are the resolver states that already exist — ambiguous drug,
unresolved section, multi-attribute question — because they are deterministic,
testable, and explainable to a reviewer. "The model felt 0.73 sure" is not a
defensible basis for asking or not asking a clinician a question.
**Unchanged.** Nothing here lets the system assert a figure absent from the
retrieved source. Reasoning chooses *what to look up and how to say it*; it is
not a source of facts.
@@ -0,0 +1,153 @@
# ADR 0008: LLM query understanding + one-shot grounded RAG (what is actually live)
**Status:** accepted, live since 2026-08-06 (F-03), extended 2026-08-07
**Supersedes:** ADR 0007 (conversational reasoning RAG — the `Focus`/
`ConversationState`/TTL state design and the PLAN/RETRIEVE/ASSESS/REFINE/
VERIFY bounded loop). ADR 0007's own `rag/conversation.py`/`rag/reasoning.py`/
`rag/conversational.py` were deleted 2026-08-07 once confirmed unreachable
from `bootstrap.py` — see the notice at the top of ADR 0007 for the full
reasoning.
**Extends:** ADR 0006 (quarantined block references) — unchanged and still
binding: a chunk with `has_quarantined_content` still forces `VERIFY_PDF`
and is never generated over.
## Context
This ADR exists because `docs/architecture.md` and ADR 0007 described a
design that was never fully built, and the modules that partially
implemented it were never wired into `bootstrap.py`. A 2026-08-06
independent 7-agent audit found this the hard way — it cost real time
establishing that `QdrantRetriever.search()` (dense vector search) and the
entire reasoning-loop module set were dead code, contradicting what the
docs claimed was live. The fix is not "finish building ADR 0007" — the
project deliberately moved to a simpler design that already works, proven
across many real multi-turn conversations (see `docs/progress-log.md`,
2026-08-05 through 2026-08-07 entries). This ADR documents that design so
the next reader doesn't have to re-discover it by audit.
## Decision
### 1. One LLM call understands the turn; no separate state object
`rag/understanding.py::LlmQueryUnderstander.understand(turn, history)` reads
the raw current turn plus a **plain list of past turn strings**
(`"Người dùng: …"` / `"Trợ lý: …"`, kept by `RagAgent._history`, a
per-conversation-id in-process dict) and returns a `QueryFrame`: turn type,
resolved `drug_id`s (validated against a candidate set a deterministic
fuzzy/alias pass bounds *before* the model runs — F-04), section attribute,
population, weight, age, indication, route, and a `needs_clarify`/
`clarify_reason`/`quick_replies` triple.
There is no `Focus` struct, no TTL, no separate summariser. The model
re-reads the same history window (last `HISTORY_TURNS * 2` = 12 lines) every
turn and re-derives what's still relevant — cheaper to build and, so far,
more robust than hand-maintained state: it naturally handles "còn trẻ em thì
sao?" and short replies to its own clarify questions (population/route/etc.
— the latter only after a 2026-08-07 fix; see progress-log) without a
resolver state machine to keep in sync.
**Known gap, inherited from ADR 0007 and still open:** this history is an
in-process dict — lost on restart, not shared across workers if the service
ever scales beyond one. ADR 0007's `PostgresConversationStore` was never
built either.
### 2. Routing is a single dispatch, not a loop
`RagAgent._route()` reads `frame.turn_type` and dispatches once:
`interaction` (2+ drugs) → gather each drug's evidence, combine, decide;
`drug_attribute`/`drug_overview`/`dosing_calc`/fallback → one drug, one
retrieval call; `smalltalk`/`out_of_scope` → canned reply, no retrieval;
`symptom_to_drug` with no drug named → an honest "not built yet" clarify.
There is no PLAN/REFINE step and no retrieval-round budget, because there is
only ever one retrieval call per turn.
### 3. Retrieval is deterministic routing, not similarity ranking
`RetrievalService.retrieve_framed(drug_id, section_key, query)`:
- `section_key` given (the dominant case, since `understand()` almost always
resolves it) → `find_by_section`, an **exact Qdrant payload filter**
(`drug_id` + `section_key`), returning the whole section as a scroll.
Score is a hardcoded 1.0 — this is a filter, not a ranked search, and nothing
here is "confidence" in the sense ADR 0007's retrieval-confidence gate meant.
- No section resolved → `find_by_drug` (whole monograph, book order),
trimmed to identity sections for a bare name or reranked (Cohere
cross-encoder over the ~29 sections of that one drug, not a corpus search)
for a free-form question.
- `QdrantRetriever.search()` — real dense vector similarity over the whole
corpus — exists and is unit-tested, but `RagAgent` never calls it. It is
reachable only through the legacy `RetrievalService.retrieve()` entry
point, itself only reachable when `ANSWER_PROVIDER=disabled` (no agent
configured at all — retrieval-only mode). `docs/architecture.md`'s
"Retrieval-confidence gate: below a similarity threshold, skip the LLM
call entirely" describes this legacy-only path, not the live one; that
section has been corrected to say so.
- Measured, and the reason this design was chosen over similarity ranking
for the live path: routing by exact `section_key` moved contraindication
hit@1 from 0.05 to 1.00 (`[[project-retrieval-quality-gap]]`, 2026-08-04).
A quarantined chunk anywhere in the retrieved set still forces the whole
result to `VERIFY_PDF` (`RetrievalService.decide`, a public wrapper added
2026-08-07 so `RagAgent._interaction` applies the same policy to a
combined multi-drug evidence pool instead of hand-rolling it).
### 4. Generation is one call, verified twice, with no confidence score
`GroundedAnswerService.answer_from_result`: sufficiency-check (ask instead of
guessing when the evidence spans multiple populations/routes and the turn
hasn't disambiguated) → generate → `grounding.verify` (every number and
citation traces to the block it cites) → `_verify_entailment` (a second LLM
pass confirming each cited claim's *content*, not just its numbers, is
actually stated by that block; one same-claim retry on a lone reject, since
this call is measurably noisy — 2026-08-06 finding). A generation that fails
any check **abstains** — it does not fall back to a raw extractive quote
when a generator is configured (`[[feedback_no_extractive_fallback_when_llm_configured]]`).
No `MAX_LLM_CALLS`/`MAX_WALL_CLOCK_MS` budget object exists. Each call is
bounded only by its own provider timeout. **This is ADR 0007's F-08 finding,
inherited unchanged and still open** — a real end-to-end request deadline
threaded through `RagAgent`'s sequence of up to 5 sequential Bedrock calls
(understand → sufficiency → generate → ≤2 entailment) is real remaining
work, not solved by this ADR. Measured live 2026-08-07: a single answerable
turn costs ~8-9s wall clock, ~75-80% of it the 4 sequential LLM calls
(understand ~2.6-3.3s dominates — an 80B model doing a classification task
that likely doesn't need one); a clarify chain compounds this linearly since
each round is a fresh request repeating the same call sequence from scratch.
### 5. Context resolved across turns is folded into one self-contained string
Added 2026-08-07, closing a P0 the 2026-08-06 audit named: `frame.population`/
`weight_kg`/`age_text`/`route`/`indication` were extracted by `understand()`
but never reached `retrieve_framed`/`answer_from_result`, which took only
the bare current-turn text — so a reply like "Uống" three turns into a dose
conversation reached the sufficiency/generation LLM calls as literally just
"Uống", with no notion that population=adult was already established two
turns back. `RagAgent._synthesize_query` now folds every resolved field into
one string (`"Uống. Đối tượng: người lớn. Đường dùng: uống."`) before it
reaches retrieval's rerank signal and generation's `query` argument. No-op
for a fresh single-shot question that already states its own context.
## Consequences
**Accepted.** No confidence score, no retrieval-round budget, no PLAN/REFINE
step — the tradeoff ADR 0007 explicitly refused ("an LLM confidence score...
is not a defensible basis for asking or not asking a clinician a question")
is exactly what this design uses instead (an LLM sufficiency/clarify
judgment), because in practice it has been reliable enough and dramatically
simpler to build, extend (route/quick_replies were one schema field + one
prompt rule each, not a new state machine), and debug — every session this
month that touched the ADR 0007 modules found new bugs in the state-machine
edges (TTL boundaries, Focus inheritance correctness) rather than in the
domain logic itself.
**Refused (again, restated from ADR 0007, still true):** an LLM confidence
score as a hard gate for retrieval — `RetrievalService.decide`'s
`VERIFY_PDF`/`ABSTAIN` decisions remain deterministic (quarantine flag,
missing provenance), never a model's self-reported certainty.
**Still open, named rather than hidden:**
- No request-scoped time/call budget (F-08).
- Conversation history is in-process, not durable/shared (inherited from
ADR 0007, never built either way).
- No production-path adversarial regression suite beyond one live-verified
end-to-end case (F-10's remaining scope).
- `dosing_calc` (a real mg/kg calculator) and `symptom_to_drug` (reverse
indication lookup) remain honest "not ready" clarifies, not answers.
+82
View File
@@ -0,0 +1,82 @@
# ADR 0009: No RAG framework — hand-written orchestration behind ports
## Status
Accepted. **Recorded retrospectively** during the 2026-08-12 documentation pass:
the decision is unambiguous in the implementation, but no ADR existed for it.
## Context
The system performs retrieval-augmented generation with query understanding,
multiple retrieval strategies, reranking, prompt construction, structured output
parsing, and post-generation verification — the exact feature set LangChain and
LlamaIndex exist to provide.
## Decision
Neither framework is used. There is no RAG or agent library of any kind.
Verifiable from the repository:
- `apps/ai-service/pyproject.toml` declares six runtime dependencies:
`fastapi`, `httpx`, `psycopg`, `pydantic-settings`, `qdrant-client`,
`uvicorn`. Optional extras add `prometheus-client`, `anthropic` and three
OpenTelemetry packages.
- `apps/ai-service/Dockerfile` installs that set plus `boto3`.
- No file imports `langchain`, `llama_index`, `haystack` or any equivalent.
Instead:
- Orchestration is a plain class with an explicit branch table
(`rag/agent.py::_route`).
- Prompts are module-level constants with JSON schemas (`rag/prompt.py`).
- Providers are injected through `typing.Protocol`s (`rag/ports.py`) and
implemented in `adapters/`, which is the only package importing an SDK — and
always lazily, inside a method.
- `bootstrap.py` is the single composition root.
## Consequences
**Enabled by this choice**
- `rag/` imports no SDK, so the entire domain — including every safety check —
is unit-testable offline with stub objects. All 278 ai-service tests run in
2.6 s with no network.
- Behaviour is inspectable: the retrieval route for a given turn is a readable
`if` chain, not framework dispatch.
- Failure semantics are chosen per call site. The fail-closed/fail-open
asymmetry in [02-system-architecture.md](../02-system-architecture.md#failure-boundaries)
is deliberate and would be hard to express through a framework's uniform
error handling.
- Prompt text is reviewable as domain policy in one file, and swapping providers
cannot silently change what the model was told.
**Costs**
- Retrieval strategies, rank fusion, context packing and evaluation harnesses
are all hand-written. Two of them (`fusion.py`, `expansion.py`) were written
and never wired ([27-technical-debt.md](../27-technical-debt.md#d-12--dead-code-three-tested-modules-with-no-runtime-caller)).
- Optional retriever capabilities are discovered with `getattr` rather than
declared, so the real interface is wider than `ports.py` documents (D-14).
- No community tooling for tracing, caching or evaluation applies; the
observability layer is bespoke.
## Rationale
Partially recoverable. The code does not state "we chose not to use a
framework", but the ports-and-adapters discipline is documented repeatedly in
module docstrings, and one of them makes the intent explicit —
`rag/understanding.py`:
> `rag/` imports no SDK: the LLM is injected as a `JsonLlm` protocol … and a
> deterministic stub runs the whole path offline in tests.
`rag/prompt.py` gives the parallel reason for prompts:
> This is domain policy, not infrastructure … it lives here so it can be read,
> reviewed and tested without an SDK, and so swapping the provider cannot
> silently change what the model was told.
The consistent theme is testability and reviewability of the safety layer.
Whether cost, lock-in or framework maturity also weighed in the decision is not
recoverable from the repository.
@@ -0,0 +1,85 @@
# ADR 0010: Single-host Docker Compose as the interim deployment
## Status
Accepted. **Recorded retrospectively** during the 2026-08-12 documentation pass.
Does **not** supersede [ADR 0002](0002-argocd-gitops.md), whose own status line
says it remains the target:
> **Accepted — still the target, not yet implemented.** Not superseded by the
> current production setup.
## Context
ADR 0002 chose GitOps on the team's ArgoCD instance. A complete Helm chart
(`infra/helm/medical-chatbot/`) and three ArgoCD `Application` manifests exist.
Neither has been applied: each `Application` carries three unresolved `TODO`s
(project/RBAC scope, repo URL, target cluster), `infra/k8s/base|overlays/` hold
only `.gitkeep`, and no image registry is configured anywhere.
Meanwhile the product is live at `https://realvuxbaro.me`.
## Decision
Run production as Docker Compose on a single EC2 host, with Caddy terminating
TLS, and deploy by SSH from GitHub Actions.
Verifiable from the repository:
- `infra/docker/docker-compose.prod.yml` — postgres, qdrant, ai-service, web,
caddy, with named volumes.
- `infra/docker/docker-compose.observability.yml` — the OTel/Prometheus/Tempo/
Grafana overlay, which also sets `OTEL_ENABLED=true`.
- `infra/docker/Caddyfile``realvuxbaro.me``web:3000`, `/grafana/*`
`grafana:3000`.
- `.github/workflows/deploy.yml``appleboy/ssh-action`, `git reset --hard`,
`docker compose up -d --build`, `caddy reload`, `python -m migrate`, then ~18
assertions.
## Consequences
**Accepted trade-offs**
- Images are built on the production host and are untagged, so there is **no
artifact to roll back to**; recovery is a revert commit plus a rebuild.
- Deploys are in-place, with brief per-service downtime.
- No horizontal scaling. That happens to align with the in-process agent state
described in [02-system-architecture.md](../02-system-architecture.md#the-stateful-detail-that-constrains-scaling),
but the alignment is coincidental, not enforced.
- Configuration and secrets live in an uncommitted `.env.prod` on the host, so
production configuration cannot be reviewed in Git.
- `postgres` and `qdrant` are deliberately absent from the workflow's `up -d`
list, so a code deploy never restarts the stateful services — and changes to
their service definitions do not take effect until someone restarts them.
**Preserved despite the simpler runtime**
The deploy script asserts far more than a Compose deploy usually does: service
health, a **real grounded answer** from the real corpus (`decision=answerable`
with a `chi_dinh` citation), both Grafana datasources, the provisioned
dashboard, public reachability of `/grafana/login`, and end-to-end trace
propagation by asserting that a specific `X-Trace-ID` becomes retrievable from
Tempo. That verification block is what makes the simpler runtime defensible.
**Migration path**
The Helm chart already maps every setting in `config.py` to a ConfigMap, mounts
`POSTGRES_DSN` from a Secret, and configures readiness/liveness/startup probes
against the same `/ready` and `/health` endpoints Compose uses. Moving to
Kubernetes therefore needs: an image registry and tagging, a corpus-load or
snapshot-restore step (the chart provisions an **empty** Qdrant, against which
`ai-service`'s manifest check refuses to start), the three ArgoCD `TODO`s
resolved, and the `bump-image-tag` workflow that
`infra/ci/github-actions/README.md` describes but does not contain.
## Rationale
**Decision observed; rationale not fully recoverable from the repository.** The
Compose header comment records one constraint —
> No GPU, no team k3s — Bedrock calls go out over the instance's IAM role … so
> no AWS access keys live in this file or its env files.
— and ADR 0002 remaining un-superseded shows the Kubernetes target was not
abandoned. Beyond that, whether the driver was cost, cluster access, or time to
first deployment is not determinable from the code.
+20
View File
@@ -0,0 +1,20 @@
# Architecture decision records
| ADR | Title | Status | Reflected in code? |
|---|---|---|---|
| [0001](0001-vector-db-qdrant.md) | Use Qdrant as the vector database | Accepted | **Yes**`adapters/qdrant.py`, `ingestion/load/qdrant_repo.py` |
| [0002](0002-argocd-gitops.md) | Use the team's existing ArgoCD instance for deployment (GitOps) | Accepted — target, **not yet implemented** | **No** — production is Docker Compose on EC2 ([20](../20-deployment.md)) |
| [0003](0003-pdf-parsing-strategy.md) | PDF parsing strategy, validated empirically | Accepted | **Yes**`ingestion/extract/`, `ingestion/segment/detector.py` |
| [0004](0004-chunking-strategy.md) | Chunking strategy for drug monographs | Accepted (monograph range only) | **Yes**`ingestion/chunk/chunker.py` |
| [0005](0005-segment-output-contract-for-chunking.md) | `segment/` output contract needed by `chunk/` | Proposed; header says "contract only, no implementation" | **Yes, now implemented**`segment/models.py` + `chunk/` follow it. The status line is stale |
| [0006](0006-quarantined-block-references-in-chunks.md) | Chunks must carry references to lifted table/formula blocks | Accepted, implemented in schema v4 | **Yes**`ChunkAttachment`, `has_quarantined_content`, the ADR-0006 gate set |
| [0007](0007-conversational-reasoning-rag.md) | Conversational reasoning RAG (state + bounded loop) | **Superseded by 0008** | **No**`rag/conversation.py` and `rag/reasoning.py` no longer exist |
| [0008](0008-llm-understanding-one-shot-rag.md) | LLM query understanding + one-shot grounded RAG | Accepted, live since 2026-08-06 | **Yes**`rag/understanding.py`, `rag/agent.py`, `rag/answer.py` |
| [0009](0009-no-rag-framework.md) | No RAG framework — hand-written orchestration behind ports | Accepted (recorded retrospectively) | **Yes** |
| [0010](0010-interim-single-host-compose-deployment.md) | Single-host Docker Compose as the interim deployment | Accepted (recorded retrospectively) | **Yes** |
ADRs 0009 and 0010 were written during the documentation pass described in
[DOCUMENTATION_PLAN.md](../DOCUMENTATION_PLAN.md). They record decisions that are
unambiguously visible in the implementation but had no ADR. Where the rationale
could not be recovered from the repository, they say so rather than inventing
one.