VuQuangBao/duocthu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Wire the guarded conversational RAG answer layer end-to-end

Browse Source
This commit is contained in:
BaoVu2k4
2026-08-05 14:33:13 +07:00
parent 834d9e51b0
commit ef08b4929e
127 changed files with 37921 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/progress-log.md
+793
View File
@@ -1,5 +1,504 @@
# Progress Log
## 2026-08-04 (evening) — Section routing: contraindication retrieval goes from 0.05 to 1.00, at zero cloud cost
The retrieval defect measured earlier today is fixed by routing rather than by
embedding. **No cloud call was made and nothing was re-embedded** — Bedrock
access is still revoked.
**The change.** A question that names its own attribute does not need
similarity to guess which section answers it. `rag/sections.py` maps the
question to a `section_key`; `QdrantRetriever.find_by_section` then filters on
`(drug_id, section_key)` and returns **every** part of that section as a
`scroll`, not a top-k. `RetrievalService` takes that route when it resolves and
falls back to similarity otherwise.
Two rules carry the safety. **Longest phrase wins**: "chống chỉ định" and "chỉ
định" differ by one prefix word and mean opposite things, so every phrase is
sorted by length and the longer is tested first — the same rule keeps "quá
liều" from being read as "liều" and "hướng dẫn xử trí ADR" from being read as
"tác dụng phụ". **No match is not a guess**: an unrecognised question returns
`None` and falls back rather than picking a section it is unsure of.
**Measured against the real `duocthu_v1` collection, no embedding involved:**
| | similarity (measured this afternoon) | section routing |
|---|---|---|
| hit@1, 160 generated cases | 0.544 | **1.000** |
| `chong_chi_dinh` | **0.05** | **1.00** |
| misroutes / empty / leaked sections | — | 0 / 0 / 0 |
**The generated 160 flattered it, and testing on human-written questions said
so.** Those questions use the phrasings the table was built from, so 160/160 is
partly circular. Run against the 16 single-drug questions humans actually wrote
in `Golden Dataset/golden_e2e_v1.csv`, the first version scored **10/16**. The
six failures were two gaps: four questions say just "Liều Metformin cho người
lớn?" — bare "liều", which the table lacked — and one says "Bà bầu", a
colloquial phrasing for pregnancy. Adding those phrases (no code change, which
is what the open/closed table is for) took it to **16/16** while the confusable
pairs still resolve correctly; bare "liều" is safe only because "quá liều" is
longer and tested first, and there is a regression test pinning exactly that.
**A circular import was found and fixed properly rather than worked around.**
`service -> sections -> routing -> service`, because `normalize_name` lived in
`routing.py`. It is a text utility with no knowledge of drugs or sections, so
it moved to `rag/text.py`; `routing.py` re-exports it so existing imports keep
working.
**Also wired, and still unproven:** `BedrockCohereQueryEmbedder` replaces the
SHA-256 hash embedder for the similarity fallback path. It has been
import-checked only — **never run against Bedrock** — so the fallback path
remains unverified end to end. The section route does not depend on it.
Verification actually run: ai-service **37 passed, 3 skipped** (22 before, +15);
ingestion **296 passed** (unchanged, checked for regression); `ruff --select
F,E9,B,ARG` over `rag/`, `adapters/`, `bootstrap.py`, `config.py` and `tests/`
— **all checks passed**; section-route evaluation against the live collection
160/160; human-written golden questions 16/16.
Not established: multi-attribute questions ("liều dùng và chống chỉ định") pick
the longest phrase, which is deterministic but arbitrary; phrase coverage
beyond these 16 human questions is unmeasured; and none of this speaks to
whether the retrieved text is clinically correct.
## 2026-08-04 (afternoon) — First real embeddings exist; retrieval measured at 54% and the cause is not what the small sample said
The corpus is embedded for the first time. Bedrock IAM was opened on the
owner's explicit instruction, all 15,100 chunks were embedded with
`cohere.embed-v4:0`, loaded into Qdrant, and **cloud access was then revoked
and proven revoked** before the owner's 17:00 deadline. Measured spend
**~$0.49** of a personal $138 budget.
**Gate results.** 15,100/15,100 embedded; 15,100 points in `duocthu_v1` over 59
batches; collection point count 15,100 — count gate **PASS**. Manifest records
`cohere.embed-v4:0`, 1024 dimensions, Cosine, corpus SHA
`04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`. Corpus SHA
was re-verified against the morning audit before spending: identical, and
identical to the post-lint copy, so the 12:05 `chunker.py` edit did not change
output.
**Both providers were probed live before choosing.** Titan v2 and Cohere v4
each returned 1024 dimensions with a **measured L2 norm of 1.000000**. That
settles a question left open since 2026-08-03: Cohere's `normalized` field was
`None` because AWS's docs never state it. It is now measured. Cohere was chosen
on two measured grounds — the corpus is Vietnamese and Cohere is explicitly
multilingual, and `bedrock_cohere.py` batches 96 texts per request while
`bedrock_titan.py` sends one, which at a measured 2.3s per call is ~9.6 hours
versus minutes. The $0.41 price difference did not drive it.
**The retrieval number, and a correction to a claim made earlier the same
day.** A 160-case evaluation (20 per section, 8 sections, questions generated
from the corpus so labels are structural) measured **hit@1 0.544, hit@3 0.663,
hit@5 0.738**. Per section:
| section | hit@1 |
|---|---|
| `chong_chi_dinh` | **0.05** (1/20) |
| `chi_dinh` | 0.30 |
| `tac_dung_khong_mong_muon` | 0.40 |
| `lieu_luong_va_cach_dung` | 0.60 |
| `qua_lieu_va_xu_tri` | 0.65 |
| `than_trong` | 0.65 |
| `tuong_tac_thuoc` | 0.80 |
| `thoi_ky_mang_thai` | 0.90 |
An earlier 15-case run gave a similar headline (0.533) but led to the **wrong
diagnosis**: four of its seven failures were contraindication questions
answered with indications, so the cause was reported as embedding weakness at
negation. At 160 cases that pair accounts for only **3** confusions. The
dominant mechanism is different and larger: **`duoc_ly_va_co_che_tac_dung`
absorbs questions from every other section** — 10 from adverse effects, 8 from
contraindications, 7 from dosage, 5 from indications. It is the largest section
(1,896 chunks) and describes the drug in general terms, so it sits close to
almost any question about that drug. This is the small-sample failure mode
CLAUDE.md warns about, reproduced on this project.
**Re-embedding cannot fix this, and the capability to fix it already exists.**
Verified by reading the code, not assumed: `apps/ai-service/adapters/qdrant.py`
`search()` filters on `drug_id` only and lets vector similarity choose the
chunk; `rag/routing.py` resolves drug and intent but **not section**; and
`find_by_payload` — the "return the whole section" method in `ingestion/load/`
— is **never called anywhere in `apps/ai-service`**. Attribute questions
therefore depend on similarity picking the right section, which is what
measures 54%. The fix is to resolve the attribute to a `section_key` and
retrieve that section whole; `ATTRIBUTE_TO_SECTION` already exists in
`embed/benchmark_local.py`.
**A silent-failure hazard found and closed.** `apps/ai-service` embedded
queries with `LocalHashQueryEmbedder` — SHA-256 of tokens, explicitly plumbing
only — while the collection now holds Cohere vectors. Querying across those two
spaces returns hits and raises nothing; the results are simply meaningless.
`BedrockCohereQueryEmbedder` was added and wired behind
`EMBEDDING_PROVIDER=cohere-v4`. **It has only been import-checked — never run
against Bedrock**, because cloud access was revoked first, as instructed.
**Two operational lessons, both paid for.** `bedrock_runtime.py` set no boto3
timeout, so a single throttled response held a socket open for over five
minutes and stalled the whole run; `connect_timeout=10, read_timeout=60` plus
standard retries fixed it. Then the first full run still died at ~14,600/15,100
because the retry backoff (2s, 4s) was far shorter than a per-minute token
quota needs. The disk cache made that survivable: the resumed run recorded
**14,977 cache hits and 123 misses**, so only 123 vectors were paid for twice —
zero, in fact, since the first run's work was already saved.
**Cloud shutdown, verified rather than asserted.** Both policies detached and
deleted; `InvokeModel` and `ListFoundationModels` both now return
`AccessDeniedException`. No EC2 instance, no EBS volume, and — because the
policy never granted `CreateProvisionedModelThroughput` — no way for this
identity to create the one Bedrock resource that bills hourly.
Not established: retrieval quality is not acceptable for clinical use, no
clinician-authored release gate exists, the generated evaluation questions use
template phrasing rather than real clinical language, and no LLM answer layer
has ever run against real evidence.
## 2026-08-04 — Chunk schema v4 passes the embedding-readiness gate
Reviewed the live Claude coordination and its last changes before editing. The
delivery plan was objectively stale: it still described schema v2/15,076 chunks,
empty embed/load/API modules, embedding before content-safety gates, and allowed
unverified inferred table headers as retrieval text. The plan and ADR 0004/0006
now put content safety, exact provenance, fail-closed schema validation and local
pseudo-vector smoke tests before any provider call. Bedrock remains benchmark-
only and requires separate owner approval for any paid/full-corpus run.
Implemented schema v4 and regenerated the canonical chunk artifact. Retrieval
`text` may repeat route/population labels so continuation chunks remain safe in
isolation; contiguous `source_text` remains byte-reassemblable and drives exact
physical/printed page provenance. `context_labels` records retrieval-only
prefixes. All 151 unverified table/formula descriptors embargo `header_row` and
cell-like column text. Attachments now carry physical page, printed page,
`block_id`, `bbox` and optional crop, and those region references survive the
Qdrant adapter and RAG citation response. The loader accepts exactly schema v4,
rejects booleans/non-integers/out-of-range pages, and keeps the normalized
LF/CRLF-stable corpus identity.
Canonical artifact measured after regeneration:
- 15,100 chunks: 14,949 prose + 151 block descriptors;
- 4,105,382 `cl100k_base` tokens; 0 chunks above the 800-token ceiling;
- all `chunk-ready` gates pass: exact provenance, source uniqueness,
reassembly, attachment coverage, descriptor embargo and schema checks all
have 0 failures; 151 descriptors match 151 quarantined blocks;
- raw file SHA-256:
`8dfae08ae6d9222089c5cdb4207a064fe67989f10f7552b555af0aef6331d9a1`;
- normalized corpus SHA-256 used by the Qdrant manifest:
`04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`.
Verification actually run:
- ingestion: **292 passed**; focused post-lint patch: **26 passed**;
- AI service with `RUN_INTEGRATION=1`: **25 passed**, including real local
Qdrant, PostgreSQL and FastAPI round-trips;
- full canonical local smoke with deterministic 4D pseudo-vectors: first and
second loads both upserted 15,100 records and both held exactly 15,100 points;
manifest hash matched; data and sidecar test collections were removed and
Qdrant returned to 0 collections;
- Ruff `F,E9,B,ARG` on the files changed for this gate: clean; `git diff
--check`: clean (Git only reported Windows LF/CRLF conversion warnings).
Conclusion: the canonical corpus is **technically READY TO EMBED**, meaning its
input/schema/provenance/load plumbing meets the measured gates. This does not
authorize a provider call, does not establish retrieval quality for any model,
and does not prove whole-book medical accuracy. Human-reviewed clinical eval,
table reconstruction, and recall for borderless tables/bar-less formulas remain
outside what these gates prove.
## 2026-08-04 — Real local datastore plumbing, guarded RAG API, and printed-page citations
Read the live Claude Code process and coordination before editing. Claude owned
`ingestion/load/` and `embed/cache.py`; it completed the disk cache, Qdrant
port/adapter, idempotent UUID5 upsert, payload indexes and corpus-SHA manifest.
Its real local Qdrant scale check loaded all 15,066 chunk records twice with
1,024-dimensional deterministic pseudo-vectors and held the point count at
15,066. Those vectors are not embeddings and establish no retrieval-quality
claim. No Bedrock call, IAM change, or cloud spend occurred.
Built the first runnable `apps/ai-service` boundary: FastAPI `/health` and
`POST /v1/rag/query`, a Qdrant retriever filtered by resolved `drug_id`, a
PostgreSQL trace repository plus migration, structured human/non-human scope
and fact/recommendation intent gates, parent hydration, quarantine handling,
and an extractive answer layer. The answer layer refuses evidence that has only
a physical page; citations expose only the printed folio, chunk id and optional
source crop. Quarantined tables/formulas return a PDF-verification warning and
never auto-extract numeric content.
Fixed the missing provenance at its source. Chunk schema is now v3 and
`cli chunk` reads the real folio map from the 1,668-page PDF. It refuses a
monograph whose physical range cannot be mapped, and `chunk-ready` has a new
`chunk_without_printed_page_range` gate. Regenerated scope: 684 monographs,
15,066 chunks (14,915 prose + 151 descriptors), zero oversized, and
15,066/15,066 records with a two-value printed-page range. New artifact SHA:
`e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38`.
Verification actually run:
- `python -m pytest -q` and Ruff over `ingestion/`: **258 passed**, lint clean;
- `python -m ingestion.cli chunk-ready`: every gate passed, including printed
page range 0/0 failures;
- ai-service with `RUN_INTEGRATION=1`: **22 passed**, including a real chunk
round-trip through local Qdrant, PostgreSQL migration/insert/read-back, and a
full FastAPI → Qdrant → guarded citation → PostgreSQL trace round-trip;
- local Docker services: PostgreSQL 16 and Qdrant 1.18.3 reachable; integration
collections were UUID-scoped and removed after tests;
- ArgoCD local: namespace, CRD and seven controller pods are running; the
existing unrelated `guestbook` lab app is Synced/Healthy with four history
entries. This repo's three Application YAML files parse and point to
`master`/the Helm chart, but they are not installed and the chart still has
no workload templates, so project sync/rollback was not performed.
Still open: no real embedding exists, no full canonical Qdrant collection can
serve semantic search, `population_tags` are absent, no clinician-authored
release-gate cases exist, and the API currently has no production answer/query
embedding provider. The local hashing provider is explicitly plumbing-only.
## 2026-08-04 — Load stage built and proven against a real Qdrant; bbox rounding found
`ingestion/load/` was a 0-byte `__init__.py`. It now holds the vector-store
boundary: a `VectorStore` port, an `InMemoryVectorStore` that is the reference
implementation of its contract, and `QdrantVectorStore` as the only module that
names `qdrant_client` — imported lazily, the same arrangement that confines
boto3 to `bedrock_runtime`. `embed/cache.py` was added alongside it.
Three design decisions are worth carrying forward.
The cache key is `(model_id, input_kind, text_sha256)`, not `chunk_id` as
§4.A of the delivery plan proposed. Measured reason: `chunks.jsonl` holds
15,066 records but only **14,869 distinct texts**, so 197 records (1.31%) are
repeats that a chunk-keyed cache would pay for twice. The content key also
cannot serve a stale vector after an edit — a changed text is a changed digest,
so it is a miss.
Point ids are `uuid5(chunk_id)`. A random id would make a re-run append a
second copy of a dose and nothing would report an error.
The corpus manifest lives in a `<name>__manifest` sidecar collection rather
than a reserved point inside the data collection, because
`qdrant_point_count != chunk_count` is a v1 gate and a gate needing an
"except the manifest" footnote will eventually be read wrong.
**Whole-corpus check against a real server.** A local Qdrant **1.18.3** was
started from `infra/docker/docker-compose.yml` (local container, no cloud) and
all 15,066 real chunk records were loaded with deterministic pseudo-vectors at
1,024 dimensions — a check of the loading mechanism, **not embeddings, which
still do not exist**. Corpus sha256 `30d5154273e0959a…`. First load: 15,066
points in 59 batches, 14.0s, point-count gate PASS. Second load: still 15,066,
so idempotency holds at real scale, not only against the fake store.
**That sha is already stale, which is the point.** `chunks.jsonl` was
regenerated at 09:53 the same day — `chunker.py` changed two minutes earlier
and every chunk gained `printed_page_range`, 18,229,918 → 18,753,003 bytes,
sha now `e474c83790b450d3…`. Re-measured on the new artifact: still **15,066
chunks, 0 over the 800-token ceiling** (largest exactly 800), all 15,066
carrying `printed_page_range`, 14,915 prose + 151 block descriptors, 197
duplicate texts (1.31%) unchanged because only a field was added. Suite
**258 passed**. Had the old corpus been embedded and loaded, then the new one
loaded into the same collection, two generations would have mixed with no error
at query time — A6 is what refuses that, and it now has a real instance rather
than a hypothetical one.
**A sampled check passed and was wrong.** Comparing 5 payloads gave 5/5
identical. Scrolling the entire collection instead found **86 of 15,066 chunks**
whose payload did not equal its source record. Classifying every differing leaf:
**96 differences, all floats, all inside `attachments[].bbox`, maximum absolute
delta 5.684e-14**, and **zero** non-float differences — every text, id, page
number, page range, token count and boolean round-tripped exactly. A PDF point
is 1/72 inch, so that delta cannot move a rendered crop. It is pinned by a
regression test that fails if the loss reaches another field or grows past 1e-9.
The layer responsible was isolated rather than assumed: the source
`chunks.jsonl` returns the value exactly, our own `json.dumps`/`loads` returns
it exactly, and **Qdrant reached over raw HTTP with no SDK involved** returns it
one ULP low. Nothing needs re-chunking — a regenerated corpus would carry the
identical value and be rounded identically. Qdrant also stores dense vectors as
float32, so precision beyond f32 is discarded at load regardless.
Cache format was decided on measurements, not preference: 300 real chunk texts
at 1,024 dimensions cost **21,098 bytes/record — ~318 MB per model** for the
corpus, with a **7.8s** offset-index rebuild per open. float32 `.npy` (62 MB)
and base64 float32 in JSONL (~87 MB) were measured and set aside; append-only
JSONL survives an interrupted run and stays readable, which outweighs disk at
one or two models. Revisit at three (~950 MB). It lands in
`ingestion/data/processed/`, already excluded by `.gitignore:34`.
**A gap in this work, found and closed the same day.** Payload indexes were
created on `drug_id`, `section_key`, `atc_codes` and `chunk_kind` and reported
as done — but `VectorStore` had no query method, so all that was really proven
is that `create_payload_index` returns without raising. Filtered retrieval is
the whole of mode A. `find_by_payload` now exists on the port and both stores,
as a `scroll` rather than a `search`: it returns **every** match, never a
top-k, because "return the whole section" is the plan's non-negotiable — two of
five contraindications reads as a complete list. Verified on a real server: all
five parts returned with no leak from the PANTOPRAZOL/OMEPRAZOL pair that
measures cosine 1.000 on contraindications; a deliberately 300-part section
(above the 256 scroll page) comes back whole so paging cannot truncate; and a
real multi-part section from `chunks.jsonl` round-trips to exactly its own
chunk ids.
Tests: **255 passed** with Qdrant running (206 before this work, +49);
**247 passed, 8 skipped** with it stopped, so an offline machine and CI see
skips rather than failures. After the mode A work and the other worktree's
`cli.py` fix the suite stands at **268 passed** and
`ruff --select F,E9,B,ARG` reports **no findings at all** across `ingestion/`.
Still missing, and deliberately so: `printed_page_range` and `population_tags`
are not in the payload (open questions to Codex in
`coordination/CLAUDE_TASK_2026-08-04.md`); `cli embed` / `cli load` are not
wired because `cli.py` is Codex's; and **no real embedding vector has ever been
produced** — every vector the load path has carried was synthetic. The Bedrock
request shapes remain documentation-derived and unproven.
Measured cost: **$0**. No Bedrock call, no IAM change, no cloud resource.
## 2026-08-03 — Bedrock embedding boundary built; IAM diagnosed, not yet opened
`ingestion/embed/` was an empty `__init__.py`. It now holds the provider
boundary the model benchmark needs: an `EmbeddingProvider` ABC that owns input
validation, request-size batching and timing, and three adapters behind it —
`amazon.titan-embed-text-v2:0`, `cohere.embed-v4:0`, and `BAAI/bge-m3` as the
zero-cost local control. boto3 is named in exactly one module and imported
lazily, so the package imports and the whole suite runs with no AWS account.
Two design points are worth carrying forward. `input_kind` is a required
argument, not a keyword: Cohere embeds corpus records and queries into
different subspaces, and sending `search_document` for a query raises no error
— recall just drops. And `normalized` is three-valued. Titan is asked to
normalize and says so; the Bedrock docs never state whether Cohere's float
vectors are unit-length, so that field stays `None` instead of guessing, and
`embed.probe` prints a *measured* L2 norm to settle it on the first live call.
The AWS side is diagnosed and stuck. `ai-lab-user` has no inline and no
attached user policy; its one group (`AI-Lab-Group`) grants EC2, IAM, ELB and
VPC full access and nothing else. There is no `bedrock:*` grant anywhere on
the identity — confirmed by running both `list-foundation-models` and
`invoke-model` and reading the two `AccessDeniedException` messages. Two
least-privilege policies are drafted in `infra/aws/iam/` but **deliberately
not applied**: that identity carries `IAMFullAccess` and could attach them
itself, which is exactly why it was left to a human.
Consequence: every request-body shape in the two Bedrock adapters is derived
from the AWS user guide (read today) and **has never been accepted by the
service**. That is unproven, not verified. Tests: 22 new, all with a stub
invoker and zero network; **203 passed** overall, up from 181. Lint clean on
every file added (`--select F,E9,B,ARG`); the one remaining finding is a
pre-existing `cli.py` import owned by the other worktree.
Measured cost so far: **$0**. Nothing was embedded, nothing reached Qdrant.
## 2026-08-03 — Exact hard-10 gate and all-block table chunking experiment
Extended the isolated table/formula sandbox beyond the 100-page sample. An
exact ten-block risk gate covered four cross-page pairs, a merged header, a
fragmented fraction bar, and the bar-less ADENOSIN formula; all ten source crops
were visually checked. The full run then processed all 151 canonical blocks:
141 physical tables, ten formulas, 133 logical table parents, 669 row children,
and seven cross-page logical tables.
Full-scope visual inspection exposed a continuation bug: FAMCICLOVIR p647 and
INSULIN p811 repeat their column headers, while other continuation pages start
directly with data. The linker now distinguishes these cases; repeated headers
are not emitted as data, and INSULIN's changed `Phối hợp` first-column meaning
is preserved. Both branches have regressions.
The expanded, source-derived retrieval suite contains 2,436 cases. With drug
and table/formula lane resolved before ranking, deterministic hybrid character
TF-IDF measured 94.42% Recall@1, 99.79% Recall@5, and 96.90% MRR. Row questions
were 94.82% / 100%; formula questions 100% / 100%. Five ambiguous whole-table
questions fell below top five because the same drug owns several near-identical
tables; production must clarify or route using an additional table anchor.
Neural MiniLM is now opt-in and excluded from the default parsing gate.
Measured chunk design: table-parent tokens min/median/p90/p95/max =
66/188/441/678/1,893; only four of 133 parents exceed 800. Row children are
75-token median, 172 p95, 471 max. Keep every logical parent intact, index both
parent and header-aware rows, never split a row, and hydrate row hits to the
complete parent/source pages. Final checks: **181 tests passed**, readiness
20/20, lint clean.
---
## 2026-08-03 — 100-page table/formula reconstruction and RAG sandbox
Built an isolated experiment under `ingestion/scratch/rag-table-pilot` without
writing sandbox representations into the canonical corpus. The risk-stratified
100-page run reconstructed 120 tables and 10 formula regions, rendered and
manually inspected all 130 crops, and linked four tables continued across page
pairs 132-133, 646-647, 825-826, and 1373-1374.
The retrieval router fixes the drug and data lane before vector ranking. On 461
source-derived queries, hybrid row+whole character TF-IDF reached 92.62%
Recall@1, 98.70% Recall@5, and 95.04% MRR. Cached English-oriented MiniLM was
worse (88.29% / 97.18% / 91.92%). Eighteen row-hit answer previews all hydrated
to the complete parent Markdown table; eight included both pages of a continued
table. A narrow deterministic interval probe passed 172/172 generated cases;
this is a mechanics check, not clinical ground truth.
Visual review exposed one canonical defect: ADENOSIN p147's bar-less printed
formula region ended after its numerator and omitted `Nồng độ adenosin
(3 mg/ml).` The bar-less band now extends 31pt below its synthetic anchor,
capturing the denominator but stopping before `Ví dụ:`; a regression pins that
boundary. Canonical artifacts were regenerated after the fix: 684 monographs,
11,974 sections, 15,066 chunks, 151 descriptors, 0 unassigned spans, all 20
readiness gates passing, **180 tests passed**, and lint clean.
Decision: JSON grid + Markdown answer view, row and whole-table retrieval, and
mandatory parent hydration are viable for the next stage. This remains a
retrieval experiment, not production clinical approval; merged-cell semantics,
unit/multi-axis reasoning, Vietnamese embedding comparison, borderless/bar-less
recall, clinician-authored evals, and final expert review remain open.
---
## 2026-08-03 — Whole-corpus parser repair after manual baseline audit
Implemented and re-ran the parser over all 1,668 pages after manually reading
the high-risk baseline outliers. The fixes are structural, with regressions:
- restored the missing `THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN`
boundary (`Tên chung quốc tế và mã ATC` is its real first anchor), separating
pages 1371–1373 from `THUỐC PHIỆN - OPIAT - OPIOID`;
- require both physical and inferred printed page bounds, so back-index page
1655 can no longer extend ZOLPIDEM's real `[1492, 1494]` range;
- keep plain label-shaped text as body when it is an adjacent wrapped
continuation in the same PDF block (including NADROPARIN's “không phải là
chống chỉ định”);
- classify known table cells before headings, putting WARFARIN and IOBITRIDOL
dosing tables back under `lieu_luong_va_cach_dung`;
- added confirmed heading variants for CLORPHENIRAMIN dosage forms and tetanus
toxoid dosing, and real provenance for combined inline fields;
- made verified formula bands column-aware: NETILMICIN opposite-column prose
is retained while AMPICILIN's gutter-adjacent formula stays quarantined;
- visually inspected all **151/151 unique table/formula regions** against the
rendered PDF; every region is genuinely 2D and remains quarantined;
- emit every physical table/formula region atomically at its first stream
occurrence, fixing split/contradictory ownership on CAPECITABIN, IMATINIB,
CARBOPLATIN, NETILMICIN, and TRASTUZUMAB;
- route explicit `Bảng N. Điều chỉnh liều ...` appendices back to dosage even
when the book prints them after `Tên thương mại` (CAPECITABIN p309);
- added readiness gates for every individual section part's source-span IDs
and duplicate physical-region IDs.
Final regenerated artifacts and evidence:
| check | result |
|---|---|
| tests | **180 passed**; lint clean |
| segmentation | **684 monographs**, 11,974 sections, 8,213,036 prose chars |
| back-index validation | **96.2% recall (678/705), 99.1% precision** |
| quarantined regions | **151 blocks / 151 unique IDs**, all visually checked |
| chunks | **15,066** (14,915 prose + 151 block descriptors), 0 over 800 tokens |
| chunk readiness | **20/20 PASS** (including duplicate-region prevention) |
| coverage | 252,799 spans, **0 unassigned** across all 1,668 pages |
| residual ink | 3,931 classified regions, **0 unclassified** across all pages |
Canonical `ingestion/data/processed/{monographs,chunks,coverage_ledger}` were
regenerated. Remaining limits: no whole-document human-reviewed clinical
ground truth, no row/column reconstruction for quarantined tables, and unknown
recall for borderless tables/bar-less formulas. This is ready for retrieval
experiments, not a claim of production clinical approval.
---
Chronological record of work done on this project, newest entry on top. The
goal is continuity across sessions: if a work session ends unexpectedly
(context/token limit, interruption), whoever picks this up next — human or
@@ -13,6 +512,300 @@ end if that risk is showing.
---
## 2026-08-03 — Independent re-verification, a redundant rule in my own uncommitted fix, a provenance defect, and the measurements a retrieval design has to be built on
No code was changed in this session: the four files from the previous round
are still uncommitted and under external review (Codex). Everything below is
measurement, and the numbers live nowhere else — the investigation scripts
were deleted per the repo rule, so this entry is the record.
### 1. Re-verified the whole tree from scratch
| check | command | result |
|---|---|---|
| tests | `python -m pytest -q` | **164 passed**, 39.31s |
| lint | `ruff check --select F,E9,B,ARG .` | clean |
| gates | `cli chunk-ready` | **18/18 PASS**; 683 monographs, 11,966 sections, 8,212,880 chars, 167 quarantined blocks |
| recall/precision | `cli validate --pdf …` | 683 detected, 705 ground truth, **96.0% (677/705) / 99.1%** |
| span ledger | `cli coverage --pdf …` (re-run) | 252,799 spans after merge, 9,398,772 chars, **unassigned 0** |
| reproducibility | `cli run` → sha256 | **byte-identical** to `monographs.jsonl` (`84f41d96…`) |
| reproducibility | `cli chunk` → sha256 | **byte-identical** to `chunks.jsonl` (`63472db4…`); 15,076 chunks (14,909 prose + 167 descriptors), 0 oversized, 4,072,725 tokens (cl100k_base) |
Recall rose 92.9% → 96.0% because of the uncommitted back-index rejoin, and
the mechanism is the denominator: 725 → 705 ground-truth entries once wrapped
fragments stop counting as entries. The detector did not improve.
**Not re-run: `cli residual-ink`.** `residual_ink.json` is dated 2026-08-01
12:03, before the 17:36 assembler edits. Its stored contents (3,931 regions,
no `unclassified` kind) are last session's numbers, not this session's.
**Doc drift found:** `docs/verification-strategy.md` quotes 252,733 spans /
177,679 `normalized_text` / 12,764 `heading`; measured today 252,799 /
177,754 / 12,752. The `unassigned = 0` conclusion still holds.
### 2. The x0 geometry change in the uncommitted diff is redundant
Assembled the whole book four times with the two new rules toggled:
| variant | monographs | sections | chars |
|---|---|---|---|
| current (x0 + italic) | 683 | 11,966 | 8,212,880 |
| **old x1 rule + italic** | 683 | 11,966 | **8,212,880** — 0 differences of any kind |
| x0, no italic | 683 | 11,966 | 8,212,844 (3 sections differ) |
| x1, no italic (= `bc01782`) | 683 | 11,966 | 8,212,780 (9 sections differ, 11,014 char delta) |
The italic rule alone recovers all 9 sections (CEFAZOLIN dosing 1,062 →
5,620 chars; CALCI LACTAT `than_trong` 582 → 1,504, `tuong_tac_thuoc` 2,346 →
1,439). The x0 rule alone recovers 6 of 9 and adds **nothing** on top of the
italic rule.
Worse, the justification is wrong: the real NEVIRAPIN span on physical page
1045 is `TimesNewRomanPS-ItalicMT` (verified by reading the span's font), so
the italic rule is what fixes that page — not the 0.01pt overlap the code
comment and the new test's docstring credit. The test itself is valid but
pins the geometric rule only, because the `_span()` fixture helper never
produces an italic font. **Either keep the x0 rule as defence-in-depth with
an honest comment, or revert it — but the current comment overstates it.**
### 3. `source_page_range` is wrong for 13 of 683 monographs
Section-level provenance (`parts`) is correct everywhere; the monograph-level
page range is not. 12 monographs overshoot by +1 page; **ZOLPIDEM declares
`[1492, 1655]` while every one of its sections comes from 1492-1494** — a
164-page claim reaching into the back index.
Root cause for ZOLPIDEM, confirmed: physical page 1655 (printed 1656, back
matter) carries a **bold** span reading exactly `Tương tác thuốc`, which
`_classify` emits as a `_SectionEvent`, and the `_SectionEvent` branch at
`segment/assembler.py:496` updates `source_page_range[1]` with **no
`in_monograph_range` guard** — unlike the `_TextEvent` branch at line 511.
Verified that **0 spans past physical 1495 pass `in_monograph_range`**, so no
text was contaminated and `empty_section` is still 0. The defect is confined
to one provenance field.
The +1 cause is **not isolated** — it is not lifted tables (all 12 have
`tables: []`); the likely candidate is a next-page boilerplate span bumping
the range before being excluded, but that was not measured.
### 4. Corpus profile — what a retrieval design actually has to work with
- **13 of 19 fields have p90 < 1,500 chars**, i.e. the whole section fits one
chunk. Only four routinely need splitting: `duoc_ly` (p90 4,939, max
14,099), `lieu_luong` (4,873 / 14,197), `than_trong` (2,419), `tuong_tac`
(2,147). Confirms ADR 0004 on the cleaned corpus.
- **ATC**: 668/683 (97.8%) carry ≥1 code, **171 (25.0%) carry more than one**,
max 20, 1,043 distinct codes.
- **`ten_thuong_mai` present in 492 (72%)** monographs.
- **The back index holds 344 `X - xem Y` lines** — brand → generic aliases —
which `parse_back_index` currently discards wholesale (correct for
validation, but this is the highest-value retrieval asset in the book,
because clinicians type brand names).
- **401 `xem [thêm] mục/chuyên luận` phrases across 261 monographs**; a chunk
containing one is useless retrieved alone.
- **Dosing population markers**: `Trẻ em` 53%, `Người lớn` 51%, `Người cao
tuổi` 15%, `Trẻ sơ sinh` 8%, `Suy thận` 8%, `Suy gan` 6% of 682 dosing
sections — real sub-section boundaries, better split points than token
windows.
- **167 quarantined blocks, 129 (77%) inside `lieu_luong_va_cach_dung`** —
the most dangerous field is the one the tables were lifted out of.
### 5. Cross-drug confusability — the number that decides the architecture
First hypothesis (much repeated boilerplate across drugs) was **refuted**:
only **171 of 11,966 sections** share exact text with another drug (1.4%), and
the six heavy clinical fields are 100% distinct.
Then measured, per field, each drug's TF-IDF cosine against its *nearest other
drug*. **This is a lexical proxy, not an embedding measure** — it bounds the
problem from one side only.
| field | median | p90 | p99 | max | drugs with NN > 0.7 |
|---|---|---|---|---|---|
| `lieu_luong_va_cach_dung` | 0.314 | 0.455 | 0.631 | 0.836 | 4 (0.6%) |
| `tuong_tac_thuoc` | 0.284 | 0.461 | 0.870 | 0.984 | 18 (2.8%) |
| `tac_dung_khong_mong_muon` | 0.300 | 0.437 | 0.856 | 1.000 | 13 (1.9%) |
| `chi_dinh` | 0.408 | 0.637 | 0.885 | 0.924 | 31 (4.5%) |
| `chong_chi_dinh` | 0.346 | 0.633 | 0.898 | **1.000** | 37 (5.4%) |
Named pairs: `PANTOPRAZOL ↔ OMEPRAZOL` (contraindications **1.000**,
indications 0.913) · `BENZATHIN PENICILIN G ↔ PHENOXYMETHYLPENICILIN`
(contraindications **1.000**) · `DIGOXIN ↔ DIGITOXIN` (0.891 / 0.911) ·
`NATRI NITRIT ↔ NATRI THIOSULFAT` (dosing 0.631 — two different steps of the
same cyanide-antidote protocol) · `IOBITRIDOL ↔ ACID IOXAGLIC` (0.984) ·
`ESTRIOL ↔ ESTRON` · `GLICLAZID ↔ GLIMEPIRID` · `NAPHAZOLIN ↔ OXYMETAZOLIN`.
Name layer: **19 drug names are a substring of another drug name**
(`CLOROTHIAZID` in `HYDROCLOROTHIAZID`, `EPHEDRIN` in `PSEUDOEPHEDRIN`,
`LORATADIN` in `DESLORATADIN`, `ATROPIN` in `HOMATROPIN HYDROBROMID` — all
genuinely different drugs), and 106 of 683 names share a 6-character prefix
across 38 clusters.
**Conclusion drawn from this, for the retrieval design: vector similarity must
never be allowed to choose the *drug* — only which passage within an
already-resolved drug.** The dangerous confusions are concentrated in a
small, enumerable set of same-class pairs, which is exactly the population
this project's verification strategy says to census rather than sample.
### Not done yet / next up
Sequenced in **`docs/v1-delivery-plan.md`** (written this session): a
two-week plan to a running v1, scoped down to two deployables (`web` +
`ai-service`) because the four NestJS services measure 0 `.ts` files each.
The items below are the ones that plan depends on.
- The confusable-pair census must become a **committed fixture produced by
production code** (`ingestion/validation/`), not a deleted scratch script.
Until then these numbers are only in this entry.
- ADR 0007 (retrieval architecture) not written. Proposed content: vectors
never pick the drug; the unit returned to the LLM is the **complete
section** (enabled by `section_not_reassemblable_from_chunks = 0`, because a
partial contraindication list reads as "no contraindication"); and eval
split in two — **routing** correctness (ground truth derivable from the
corpus itself, 683 × 19 pairs, no human needed) versus **content**
correctness (requires a clinician; cannot be self-generated without
fabricating evidence).
- Entity/alias layer (683 canonical names + 344 back-index aliases + 492
`ten_thuong_mai` + 1,043 ATC codes) — zero-regret, needed by every
architecture, must use longest-exact-match because of the 19 substring
traps.
- `residual-ink` re-run; `verification-strategy.md` numbers re-synced;
regression test for `parse_back_index` (still has none); the
`source_page_range` guard; the x0-rule comment decision.
- Open question for the user, not a technical one: this is the **2018
edition**; the 3rd edition (2022) exists. For a document with legal force
over prescribing, staying on 2018 should be a deliberate decision, and it
makes edition-independence a real requirement for the pipeline.
- Still untouched: `embed/`, `load/`, Qdrant, `ai-service`, and the general
chapters (printed 37-98) and appendices (printed 1497-1528), which remain
outside the corpus entirely.
## 2026-08-01 (cont'd, 7) — "still errors?" — yes: two more real content-loss bugs, both in dosing sections
Asked whether errors remained after the previous round, the honest answer was
that this session has found real defects every time it looked one level
deeper. It looked again, and found two more.
**1. Chunks ended on a bare population label, with the dose in the next
chunk.** `split_sentences` treats `:` as a sentence boundary and
`_OPENS_SENTENCE` accepts a digit, so `"Người lớn: 500 mg mỗi 8 giờ."` splits
after the colon. When the packer flushed at that point, the chunk ended on the
label. Measured: **38 prose chunks**, e.g. AMOXICILIN's ending on a Lyme
indication followed by a bare `Người lớn:`. Retrieval on that chunk returns a
population with no dose; on the next, a dose with no population. Outlier item
17 counted population markers on 1,121 of ~1,400 monograph pages, so this is
the common shape, not an edge case. The packer now carries trailing label
atoms into the next part instead of flushing on them: **38 → 2**, and chunks
ending on any colon **721 → 19**.
**2. A section name printed mid-line was swallowed as a heading — real text
loss, in dosing sections.** Chasing the last 2 of those 38 showed the defect
was not in chunking at all. CISPLATIN (physical page 402) prints
`Suy thận: Chống chỉ định.` inside `liều lượng và cách dùng`; the second half
is itself a section name, so it was matched as a heading. The result: the
renal-impairment contraindication **disappeared from the dosing text** and the
section ended on a bare `Suy thận:`. ISOPRENALIN had the same shape. Same
family as the FLUOROURACIL bug fixed earlier today, but that rule only covered
a label directly *under* a heading and could not see this one.
Fixed geometrically: a real section heading opens its line, so a non-bold
section name with another span printed to its left is body text. "To the left"
is checked properly — same page/block/line *and* `previous.x1 <= span.x0` —
because the synthetic test fixtures place every span at identical coordinates,
and a looser check passed on real data while breaking the AMITRIPTYLIN
inline-value case.
Verified after the fix: CISPLATIN's dosing section contains
`Suy thận: Chống chỉ định.` again, ISOPRENALIN's `Trẻ em:` is followed by its
doses, and `chong_chi_dinh` is no longer polluted. Monograph and section counts
unchanged at 683 / 11,966 — nothing was traded away for the recovery.
**State:** 18/18 gates pass, **163 tests** (was 161), ruff F/E9/B/ARG clean,
15,077 chunks with 0 over the ceiling, 8,212,780 section characters.
**Standing conclusion, worth writing down:** every round of "is it clean now?"
this session has ended with real defects found — five in the previous round,
two in this one, and four of the previous five were in code written the same
day. The gates and tests prove what those instruments can see. They do not
prove the corpus is correct, and the largest unmeasured area is unchanged:
content accuracy against the source, with no human-reviewed ground truth for
8.2M characters.
## 2026-08-01 (cont'd, 6) — Bug hunt after declaring "clean": the token count was wrong by 2x, 14.7% of chunks were over the ceiling, and two stage boundaries measured different pipelines
I had just reported the tree as clean. It was not. Going looking properly
found five real defects, four of them in code written earlier the same day.
**1. `estimate_tokens` was wrong by a factor of two, and the number it
produced was reported.** ADR 0004 sized chunks with `len(text) // 4`,
described honestly as an estimate. Measured against `cl100k_base` on the real
corpus:
| | |
|---|---|
| estimate (chars/4) | 2,115,427 tokens |
| real tokenizer | **4,093,440 tokens** |
| real/estimate | median **1.95**, p95 2.50, max **6.0** |
| oversized by estimate | **0** |
| oversized in fact | **1,884 of 12,838 = 14.7%**, largest 1,645 tokens |
Vietnamese diacritics cost several byte-pair tokens each. "0 oversized" was
reassuring and false. `chunk/tokens.py` now counts with the real tokenizer,
injected so the chunking logic stays testable without it, with a fallback of
chars/2 that errs small rather than large.
**2. The packer could exceed the ceiling on its own.** Two causes, both
measured on VORICONAZOL's `tương tác thuốc`: an atom of 710 tokens was left
whole because it was under the 800 ceiling, and the overlap builder added
whole atoms until the running total *passed* the budget, so a 251-token atom
produced a 273-token overlap against a 65-token setting. 273 + 710 = 983.
Atoms are now split against the 650 target, leaving room for overlap, and the
overlap stops *before* exceeding its budget.
**3. An over-long comma list was left as one atom.** VORICONAZOL's
interaction list is one "sentence" hundreds of drug names long. Truncated by
an embedding model it reads as "this drug is not listed" — a false negative
in the direction that matters. Split at commas, which is lossless for a list.
After 1-3: **0 chunks over the ceiling**, verified by an independent tiktoken
re-count of the written file, not by the pipeline's own number. 15,049 chunks
(was 12,838 — the rise is real sub-chunking that should have happened all
along).
**4. `cli validate` measured a different pipeline than `cli run`.** It used
the raw span stream (no transcription repair) and passed no table regions, so
recall/precision described a build that is not the one producing the output —
the same class of mismatch already fixed for `coverage`. Now shares
`_extracted_and_repaired_spans` and `_region_index`. Result after the fix is
unchanged at 92.9% / 99.1%.
**5. `chunk/io.py` dropped `SectionPart` when reading monographs back**, so
per-part provenance died at the stage boundary — against CLAUDE.md's explicit
rule. Now carried: 12,290 parts across 11,966 sections.
**Two new gates, and the gate itself was wrong twice before the data was.**
`section_not_reassemblable_from_chunks` rebuilds each section from its own
chunks by removing the deliberate overlap and compares. First version joined
chunk texts with a newline and reported **734** sections missing — the first
one it named was present. Second version probed a 60-character head and
reported **1**, NAPROXEN, where the probe straddled an overlap seam that
legitimately repeats text. The working version compares with whitespace
removed, because each split seam loses exactly one space to `.strip()`
(measured on ABACAVIR: two single spaces in a 4,232-character section,
nothing else). It proves no character of content is lost, reordered or
duplicated beyond the intended overlap. **0.**
**Also fixed:** all 8 real lint findings (`ruff --select F,E9,B,ARG`) — five
unused imports and three `zip()` calls without explicit `strict=`. The zips
were the adjacent-pair idiom and not bugs; `strict=False` now says so. And
the transcription splice could leave a fragment holding only a space, which
showed up as two `whitespace_only` spans; dropped, and proven inert by the
sha256 over every section's text being byte-identical before and after
(`6af13301…`).
**State after the hunt:** 18/18 gates pass (10 corpus + 8 chunk), 161 tests
(was 158), `ruff F/E9/B/ARG` clean, `unassigned = 0`, `cli validate` 92.9% /
99.1%, 15,049 chunks with 0 over the ceiling.
## 2026-08-01 (cont'd, 5) — ADR 0006 implemented: chunks now reference their lifted blocks; `chunk/` runs for the first time; 16/16 gates green
**Why this was needed, in one line**: a chunk of a section whose table had