duocthu/docs/progress-log.md

# Progress Log

## 2026-08-05 (evening 3) — The LLM cloud is LIVE: DeepSeek generation + Cohere rerank on the real corpus

The owner rejected the $0 offline build as the deliverable and set a hard
deadline. The chatbot is now a **real LLM RAG**, grounding kept ON, running the
full HTTP stack (ai-service :8079 ↔ Postgres trace ↔ Qdrant; web :3000). Commit
`9c4273b` (plus `92497ae`/`9e9cef7`/`1b6f399` earlier this session, which
committed the previously-uncommitted evening-1/2 work).

**What was turned on** (live via gitignored `.env`; committed defaults stay
`disabled`/`section-only` so CI/fresh-clone never touches cloud):
- `EMBEDDING_PROVIDER=cohere-v4` — query now embedded in the corpus's
  `cohere.embed-v4:0` space (probe: 1024-dim, L2 1.0, ~1.95s). No re-embed; the
  15,100 vectors already exist.
- `ANSWER_PROVIDER=bedrock-converse` + `deepseek.v3.2` — new
  `adapters/bedrock_converse.py` (Bedrock **Converse** API, boto3,
  model-agnostic; Qwen/GLM = 1 env + 1 ARN). Probe OK. GPT-4o confirmed NOT on
  Bedrock; OpenAI `gpt-oss`, DeepSeek, Qwen, GLM, Mistral, Kimi ARE (checked live).
- `RERANK_ENABLED=true` — `cohere.rerank-v3-5` trims the overview/similarity
  fallback: a free-form drug question no longer dumps all ~29 sections at the
  model (**measured 29 → 6** on the fever/paracetamol case). Section route never
  reranks; fail-open (outage → book order, answer survives).
- `rag/prompt.py` rewritten to current citation-enforced practice: each dose
  carries its population/condition label (no adult/paediatric mixing), cite only
  the supporting block, no `[n]` spam, abstain on insufficient evidence.

**IAM:** managed policy `BedrockEmbeddingInvoke` bumped to v4 (invoke on
titan-embed, cohere.embed-v4, deepseek.v3.2, cohere.rerank-v3-5); repo file
synced. Codex was off, no collision.

**Verified:** ai-service **111 passed, 3 skipped** (+12 this milestone). Live
HTTP `POST /v1/rag/query` returns a grounded LLM answer with a citation and a
Postgres trace id. Golden `golden_e2e` (35 Qs): **19/19 answerable questions
grounded with the correct drug** (incl. typo `paracetamon`, alias
`Acetaminophen`, multi-turn inheritance); 14 adversarial correctly abstained
(fake drugs, weather, symptom→drug reverse-lookup, multi-drug). **Two real
gaps:** a price question answers from the monograph instead of "no price in the
formulary", and "should I double the dose?" is not directly warned. Every
`generated=True` answer passed `grounding.verify`.

**Cost/safety:** Bedrock is pay-per-call — verified **0 EC2** (3 regions) and no
provisioned throughput; idle = ~$0. A few dozen probe/smoke/eval calls this
session, cents-scale on the estimate; exact bill not checked.

**Separate track, NOT done (background subagent started, own worktree):**
reconstruct the 151 quarantined tables with a `needs_expert` flag on uncertain
cells + parse Part 1 (poisoning/pregnancy/hepatic-renal) & Part 3 (BSA/ATC) +
re-embed. This is a multi-hour ingestion pass with the whole-doc validation gate
and will NOT be clinician-validated within the deadline — deliberately kept off
the deadline path.

## 2026-08-05 (evening 2) — Conversational chat core wired LIVE end-to-end (offline, $0); owner wants the LLM cloud next

The chat core is now **live and serving multi-turn**, not just unit-tested. It
runs `$0`/no-cloud because the section-route is a payload filter (no query embed)
and generation is still off (verbatim), but the *conversational* behaviour is
real and smoke-tested against the running service (ai-service :8079, web :3000).

Built (`rag/conversational.py` `ConversationalLoopService`, wrapping the safe
`GroundedAnswerService`; wired through `bootstrap.py`/`main.py`/`routers/rag.py`
with an optional `conversation_id`, plus `route.ts` sending it and a `ChatPanel`
error state):

- **Multi-turn follow-up inheritance.** "Chống chỉ định Metformin" then "còn trẻ
  em thì sao?" carries the drug+section forward and names it ("Về metformin: …").
- **Smalltalk.** "chào bạn" gets a friendly redirect, not a failed-drug-lookup
  refusal.
- **Drug-name-only → the whole monograph.** Typing "PARACETAMOL" now returns all
  18 sections in book order with `【heading】`s and per-section citations
  (`QdrantRetriever.find_by_drug` + `SECTION_ORDER`; `RetrievalService` uses it
  when a drug resolves but no attribute is named) — the earlier "specify an
  attribute" dead-end is gone.
- **Typo → ask, never threshold-guess.** Only an EXACT drug name auto-resolves;
  a fuzzy match is offered as a question ("Ý bạn là: Metformin?") via
  `CatalogDrugResolver.suggest(min_score=0.72)`. A completely-wrong name →
  "Không có thuốc này trong Dược thư Quốc gia." A formulary must not silently
  answer about a *different* drug than the one meant.
- **Autocomplete endpoint** `GET /v1/rag/suggest?q=` (`CatalogDrugResolver.complete`,
  substring/prefix) — the frontend dropdown that consumes it is still to build.
- **BSA calculator** `rag/calculators.py` (Appendix 1, DuBois, tested vs the
  book's own cells).

Verification: **ai-service 99 passed, 3 skipped**; live smoke test of all four
conversation behaviours plus the monograph/typo/not-supported cases. A
refine-loop bug (a refined query dropped the inherited drug and abstained,
discarding a good answer) was found in my own code and removed before shipping —
clarify + inheritance are the loop's value, retrieval-refine is not, and it is
gone from the live path.

**Owner's next-session directive (recorded in memory `project-llm-cloud-plan`):**
stand up the cloud LLM — semantic query embedding (`EMBEDDING_PROVIDER=cohere-v4`,
already IAM-permitted) and answer generation (a cheap model, non-Anthropic OK, via
a Bedrock Converse adapter, needs its ARN added to `BedrockEmbeddingInvoke`). The
offline build was budget/safety-first, not LLM-avoidance; the owner wants the real
AI experience next, with `grounding.verify` and the quarantine contract kept ON.

## 2026-08-05 (late) — Read the source book's own structure; scope + usage-pattern findings (checkpoint before handoff)

Read the Dược thư 2018 front matter directly (printed p8 "Nội dung", p39
"Hướng dẫn sử dụng") to understand what the book is *for* and how clinicians use
it — recorded in memory `reference-duoc-thu-2018-structure`. Key facts that
reshape the chatbot scope:

- The book has **three parts**. The corpus is **Part 2 (drug monographs, printed
  99–1496) ONLY**. **Excluded and clinically important:** Part 1 general chapters
  (printed 37–98: prescribing in the elderly / hepatic-renal impairment /
  children / pregnancy-lactation; disease-class guidance for asthma, epilepsy,
  HIV, antibiotics, TB, hepatitis B, antipsychotics; drug allergy; **poisoning &
  antidotes**; drug-interaction principles) and Part 3 appendices (printed
  1497–1528: **body-surface-area calc**, IV admixture, ATC classification). So
  "how to treat asthma", "antidote for X", "BSA-based dosing" have no data in the
  index today — a coverage limit, not a retrieval bug.
- The 19 monograph fields are fixed and documented on p39; a field is omitted
  when the book has no info (so a missing section is not necessarily a parse bug).
- Field 14 dose is a *general adult+child oral reference dose unless stated*; the
  clinician adjusts. → the tool supplies reference data, not a prescription.

Data checks run this session (against `chunks.jsonl`), correcting earlier
pessimism:
- Indication is searchable: 48 drugs' `chi_dinh` mention "sốt". Reverse lookup
  (symptom → drugs) is feasible from **content**, but retrieval is drug-first, so
  not answerable yet.
- **mg/kg dosing is in PROSE, not tables**: 574 `lieu_luong` chunks contain
  "mg/kg", all prose, across **295 drugs**, 473 of them mentioning trẻ em. So the
  *primary* weight/age dosing (incl. pediatric) is answerable; the 83 quarantined
  dosing tables are mostly the *supplementary* renal-adjustment tables (49 of
  those 83 drugs also have mg/kg prose).
- Pregnancy dosing is mostly **qualitative**: 670 drugs have a
  `thoi_ky_mang_thai` section but only ~23 chunks carry a mg figure — the book
  rarely gives a separate pregnant dose, so answer = pregnancy caution + standard
  dose, never a fabricated pregnant-specific number.
- `drug_id` can be compound (`paracetamol_acetaminophen`); alias resolution must
  map "paracetamol" → that id.

**Design consequence discussed with the owner (not yet built):** the "understand"
stage must classify the *turn type* (smalltalk / medical query / multi-drug
interaction / symptom-indication / out-of-scope / injection-shaped), not just
resolve a drug. Refusing a clinician's symptom→drug question as
"recommendation_out_of_scope" was wrong for this audience — such questions are
indication lookups and should be answered from `chi_dinh`. Multi-drug
interaction/contraindication questions need a real PLAN → gather both drugs →
synthesize step (the ADR-0007 PLAN node, still unimplemented), and an
absence-of-evidence answer must state where it looked, never assert "safe".

**Session state / not yet done (so a fresh session can resume):** the chat
module's domain glue is built and unit-tested (`rag/conversation.py` ports +
summariser, `rag/conversational.py` orchestrator + `is_smalltalk`); it is **not**
wired to the endpoint. Live wiring (turn-type classifier, loop-around-
GroundedAnswerService, Postgres store, `conversation_id` on `/v1/rag/query`,
`route.ts`, ChatPanel error state), the P0 audit fixes (§5 context-mixing
metadata, §8 Qdrant-error degradation), reverse-indication retrieval, and the
table vision-consensus pipeline all remain to do. No code was wired live this
session; the behavior spec is still being clarified with the owner before wiring.

**Owner decision: parse the WHOLE book, re-chunk freely** (not just Part 2
monographs). Current corpus covers physical pages **100–1494** only. To add:
Part 1 general chapters (physical ~36–97) and Part 3 appendices (~1496–1527);
front-matter list (13) and index (1529+) are already used as validation ground
truth. Read `segment/detector.py` to ground the plan — **the machinery already
generalizes**: a chapter title ("NGỘ ĐỘC VÀ THUỐC GIẢI ĐỘC") has the *same shape*
as a monograph title (bold + mostly-upper + short), so `is_monograph_title_candidate`
extends by widening the hardcoded `99–1496` range per `content_type`. Only two
real changes: (1) parametrize the page range + add a `content_type`
(`monograph|chapter|appendix`); (2) chapter/appendix sub-headings are **free-form**
("Hô hấp", "Co giật"), not the 19-key vocab, so `detect_section_headings` needs
an open-taxonomy mode (bold + short = heading, store the text, no `match_section`
requirement). Everything downstream (span extraction, table/formula quarantine,
provenance, chunker) is content-type-agnostic and reused → schema v5 adds
`content_type` + `chapter_id`. **Gate (CLAUDE.md): the span-routing ledger must
account for ALL 1668 pages with `unassigned=0`, not just 99–1496.** Then embed
only the NEW chunks (Cohere, pennies, announce first). This is a focused
ingestion pass (detector + assembler + chunker + whole-doc re-run + validation),
not a one-liner — not attempted this session beyond grounding the plan.

**Done this session (App 1, self-contained, validated):** `rag/calculators.py`
`body_surface_area_m2` replaces Appendix 1's lookup table with the book's DuBois
formula (`S = W^0.425 × H^0.725 × 71.84`), tested against three of the book's own
table cells (165cm/60kg→1.66, 90cm/10kg→0.50, 170cm/70kg→1.81) — `tests/test_calculators.py`,
3 passed. Audit §7 (calculation = tested function, never an LLM).

## 2026-08-05 (evening) — Conversational orchestrator wired to the existing loop; data-quality audit; budget verified live

**Chat module (the glue ADR 0007 specified and nothing had called).** Added the
two missing conversation ports and their offline defaults to `rag/conversation.py`
(`ConversationStore`/`InMemoryConversationStore`, `Summariser`/`DeterministicSummariser`)
and the orchestrator `rag/conversational.py` (`ConversationalRagService`). It owns
no rules of its own: load state → resolve this turn → inherit gaps from `Focus`
→ derive clarify signals from resolver state → `reasoning.run_turn` → update
focus, append turns, summarise overflow, save → name any inherited drug. Runs
with no LLM/service (collaborators are protocols). `DeterministicSummariser`
records only drug/section **labels**, never cell values, so the
no-clinical-content-in-summary rule holds by construction rather than by trust —
closing the summary-bypasses-grounding hole flagged in review. **11 new tests;
full ai-service suite 91 passed, 3 skipped.**

**Still NOT wired live:** a `TurnResolver` bridge over `CatalogDrugResolver` +
`SectionResolver`; bridges from `RetrievalService`→`Retrieve` and the grounded
generation path→`Generate`; `PostgresConversationStore` + migration; a
`conversation_id` on `/v1/rag/query`; `route.ts` sending it and dropping the
hardcoded `intent: fact_lookup`; a `ChatPanel` error state; and the multi-turn
eval run. So no claim yet that history/loop improves answers — designed and unit-
proven, not measured end to end.

**Data-quality audit (self-run this session, not quoted from this log).** 684
drugs; critical-section coverage is strong — dosing missing 0.1% (1), contra-
indication 0.4% (3), indication 0%. **But 83/684 drugs (12%) have their dosing
inside a quarantined table**, so a dose query for them returns `VERIFY_PDF`
(crop, no number) — the largest answer-quality gap for a clinician audience, and
it lands on the single most-asked query. 125 chunks carry a leading `": "`
label-leak artifact (93 in `ten_chung_quoc_te`). Vector-path text loss appears
contained to 22 flagged lines (completeness of detection unverified). Nobody
clinician-side has validated the 8.2M chars against the book — still the largest
unmeasured area.

**Table validation — the instrument that text extraction lacked.** Demonstrated
that vision reads a real quarantined dosing table cell-by-cell: GABAPENTIN's
renal-adjustment table (printed 706) came back exactly by eye where pdfplumber's
text layer could not structure it. Found and corrected a page-index off-by-one
in my own render (data `physical_page` N = `doc[N]`, 0-based) — proof that
correctness must not depend on trusting coordinates. Strategy, given pharmacists
are **end-users, not labelers**: reconstruction powers **retrieval only**; the
displayed answer stays crop + page (clinician verifies at point of use).
Validation is automated — vision↔geometric consensus + round-trip visual +
book invariants — with a per-cell precision-first gate (disagreement → stays
crop-only). Not yet built; 151 blocks is small enough for full census.

**Budget, read live from the billing console** (owner login; `ai-lab-user` has
no billing API perms): **$138.50 remaining, entirely AWS promotional credit,
not the owner's card**; August bill $0. Deploy target chosen: team k3s, but
deferred (mutating a shared cluster). Generation still off (`answer_provider=
disabled`) — extractive/verbatim, which is defensible for clinicians; wiring a
cheap model (Nova/Haiku via Bedrock Converse) needs its ARN added to the
`BedrockEmbeddingInvoke` policy, which today grants invoke on the two embedding
models only.

## 2026-08-05 — An answer layer that cannot state a number the book does not

Today started by walking the **demo path** rather than the test suite, and the
two are not the same thing. The suite was green and the demo was broken.

**What the walk found, by running it rather than reading it.** The backend
answers real Vietnamese questions against the real embedded corpus with real
citations and **zero cloud cost** — the section route is a payload filter, not
a vector search. `Chống chỉ định của Metformin là gì?` returns the true
contraindication text with one citation; `Tương tác thuốc của Warfarin?`
returns two. But `Tôi sốt cao, uống Paracetamol được không?` returned **HTTP
500**: the similarity fallback reached Bedrock, which is revoked, and
`botocore.AccessDeniedException` escaped as an unhandled error. Any question
whose phrasing is outside the section phrase table takes that path.

That crash also **re-verified the cloud shutdown today, live** — the denial
came from the service, not from a claim in a document.

**Four defects, all fixed, all at $0.**

1. **The 500.** `adapters/embedding.py` now translates provider failures into
   the domain error `QueryEmbeddingUnavailable`, and `RetrievalService` catches
   it and abstains with `reason="query_embedding_unavailable"` — deliberately
   distinct from `insufficient_retrieval_score`, so an outage never reads as an
   empty corpus. `rag/` still imports no SDK.
2. **A default config that does not work.** `config.py` pointed at collection
   `duoc_thu_chunks`; the real one is `duocthu_v1`. `embedding_provider`
   defaulted to `disabled`, so `/v1/rag/query` returned 503 on a fresh clone.
3. **Neither existing provider was a safe default.** `local-smoke` searches a
   SHA-256 vector against a Cohere collection — confident, meaningless hits.
   `cohere-v4` spends the boto3 retry budget (~30s) before failing on a revoked
   account. Added `SectionOnlyQueryEmbedder`: refuses locally and instantly, so
   retrieval is confined to the route that measured 16/16.
4. **Safety abstention was incidental, not a gate.** Symptom questions abstain
   with `reason="drug_not_resolved"` — because no drug name was found, not
   because anything recognised a symptom question. Recorded, not yet fixed.

**The answer layer now has an LLM, and a check that makes "it does not
fabricate" measurable rather than promised.** Previously `rag/answer.py` was
extractive: it concatenated retrieved chunks. That is why
`Liều Paracetamol cho người lớn?` opened with `5 - 12 tuổi: Trẻ em 12 - 18
tuổi:` — raw section text, paediatric doses first, for an adult question.

Generation is now three layers, and only the third is load-bearing:

- **Prompt** (`rag/prompt.py`, domain — no SDK): evidence only, figures copied
  character-for-character, `[n]` citations required, insufficient evidence is a
  valid answer. Output shape is pinned by `output_config.format`, so a
  malformed envelope is the provider's error, not our parsing problem.
- **Verification** (`rag/grounding.py`, pure domain): every numeric token in
  the generated answer must appear **exactly** in the evidence, and every `[n]`
  must resolve. Citation markers are stripped before number extraction so `[2]`
  is never read as the quantity 2.
- **Fail-closed** (`rag/answer.py`): ungrounded number, invalid citation,
  malformed output, provider outage, or the model itself reporting insufficient
  evidence — every one falls back to the verbatim source text, which was
  computed first and is therefore always available.

**Numbers are compared as strings, and that is the decision worth keeping.**
No parsing, no normalisation. `1.500` is 1500 under one reading and 1.5 under
another; a normaliser that strips separators maps `7,5` and `75` to the same
key, scoring a **tenfold dose error as a match**. Pinned by
`test_decimal_separators_are_not_interchangeable`. The same rule refuses
`2 g` → `2000 mg`: arithmetically right, but unit conversion is where dosing
errors live, so it is refused rather than interpreted.

Quarantined tables and formulas are **never generated over**. `VERIFY_PDF`
returns before generation — those are precisely the blocks whose numbers were
not reliably reconstructed, so rephrasing them is the one case where fluency
could invent a dose. This keeps ADR 0006's contract intact.

**Provider chosen on the owner's instruction: AWS Bedrock + Claude.**
`adapters/bedrock_claude.py` is the only module naming the `anthropic` SDK,
imported lazily. Two provider facts taken from the Anthropic API reference
today, not from memory: Bedrock model ids carry an `anthropic.` prefix
(`anthropic.claude-opus-5`), and the Messages-API path on Bedrock is
`AnthropicBedrockMantle`, **not** the legacy `bedrock-runtime` InvokeModel route
the embedding adapter uses. A `stop_reason: "refusal"` is a successful HTTP
response with no usable content, so it is routed to the extractive fallback
rather than allowed to raise on `content[0]`.

**This adapter has never been run against Bedrock.** Cloud access is still
revoked and no IAM change was made today. `StubAnswerGenerator` exercises the
entire path — prompt build, schema parse, grounding check, fallback — with no
cloud call, and that is what the end-to-end run below used.

**Observability, because a dashboard is a better answer than a slide.**
`rag/metrics.py` defines the counters in the domain; `adapters/prometheus.py`
is the only module naming `prometheus_client`, imported lazily; `/metrics`
returns 404 rather than an empty 200 when metrics are off, so a scrape cannot
succeed silently with no samples. The headline counter is
`duocthu_generation_rejected_total{reason="ungrounded_number"}` — the measured
form of the no-fabrication claim. A mismatched label drops the sample instead
of raising: metrics must not be able to break a clinical answer.

`infra/docker/` gains Prometheus and Grafana with a provisioned datasource and
dashboard. **Not yet verified running** — the image pull was still in progress
when this was written.

**A section was being served scrambled, and only using the UI found it.**
`liều dùng paracetamol` opened mid-sentence on `5 - 12 tuổi:` and buried
`Liều lượng: Người lớn:` seven hundred words down. `find_by_section` returned
whatever order Qdrant scrolled, and point ids are `uuid5(chunk_id)`, so
PARACETAMOL's five dosing parts came back **3, 4, 1, 2, 0** — verified by
scrolling the real collection, not inferred. `part_index` was in the payload
all along and simply never used. Now sorted by it; a part missing the field
sorts last rather than being dropped, because a silently shortened dose list
is worse than an unordered one. Pinned by `tests/test_section_order.py`,
including the exact 3,4,1,2,0 case. **This is a clinical defect, not a
cosmetic one:** a reader who stops partway through stops in the middle of a
different population's dose. Every section-routed answer given before today —
including the 16/16 golden result — was assembled in this scrambled order;
retrieval picked the right chunks, so the measurement stands, but no
statement about how those answers *read* survives it.

**Conversational reasoning RAG: designed in ADR 0007, domain layer built.**
`rag/conversation.py` carries `Focus` (drug, section, population, verbosity,
each stamped with the turn that set it) and the recent-turn window;
`rag/reasoning.py` is the bounded loop. Both are pure domain and run with no
provider, which is the point: *which drug is this still about* must be
deterministic, not inferred.

Three rules make inheritance safe in a formulary, each pinned by a test: an
explicitly named drug always beats context; focus older than six turns is
dropped rather than carried, because a stale drug is a wrong-drug answer, not
context; and any answer built on an inherited drug must name it.

The loop's uncertainty signal is **not** a model confidence score. It is the
resolver states that already existed and previously dead-ended into `abstain`
— ambiguous drug, unresolved attribute, multi-attribute question — which now
produce a clarifying question. Deterministic, testable, and explainable to a
reviewer in a way that "the model felt 0.73 sure" is not. A clarify signal
short-circuits before any budget is spent, verified by asserting the budget is
untouched and neither retriever nor generator was called.

Budgets are decremented **before** the call they pay for, so exhaustion
degrades to the best answer so far. A retrieval round is bought only by a
*named* gap with a genuinely new query: `test_an_unnamed_gap_does_not_buy_a_round`
and `test_a_refinement_that_changes_nothing_stops_the_loop` are the guards
against a loop that spins on a feeling or re-issues the same query.

`Golden Dataset/golden_multiturn_v1.csv` is new — 8 conversations, 19 turns,
6 of them inheritance-dependent. The existing golden file is single-turn by
construction and can measure none of this. Includes the adversarial turns: a
follow-up after a refused fake drug (must not borrow a drug from elsewhere),
and a follow-up after a symptom question (must not inherit treatment intent).

**Not yet wired:** the loop is not called by `GroundedAnswerService` or the
router, there is no `PostgresConversationStore`, and no evaluation run over the
multi-turn file has been performed — so no claim is made that history or the
loop improves answers. The design states how that will be measured; it has not
been measured.

**LangChain was considered and rejected.** The repo already has the ports and
adapters LangChain would supply, retrieval is already measured, and the
guardrail is already domain code. Adopting it a week before a review would
rewrite the working part for no measured capability gain.

Verification actually run: ai-service **56 passed, 3 skipped** (37 + 3 before,
+19); ingestion **296 passed**, checked for regression, unchanged; `duocthu_v1`
holds **15,100 points** at 1024-dim Cosine, matching the manifest; live service
against the real collection answered three clinical questions with citations
and abstained on six of the seven safety probes; `/metrics` scraped and
returned `duocthu_generation_served_total 2.0` and
`duocthu_abstention_total{reason="drug_not_resolved"} 1.0`.

Not established, and load-bearing for the demo: **`apps/web` is still entirely
mocked** — `packages/api-client/src/sendChatMessage.ts:8` returns
`buildMockResponse(content)` and the whole frontend contains no HTTP call to
the backend, so the working API and the working UI are not connected;
`api-gateway`, `chat-service` and `auth-service` hold **0 source files**; the
Bedrock generator has never been invoked; `intent` is still supplied by the
caller, so the recommendation gate depends on the client declaring it honestly;
and the Prometheus/Grafana stack has not been seen running.

## 2026-08-04 (evening) — Section routing: contraindication retrieval goes from 0.05 to 1.00, at zero cloud cost

The retrieval defect measured earlier today is fixed by routing rather than by
embedding. **No cloud call was made and nothing was re-embedded** — Bedrock
access is still revoked.

**The change.** A question that names its own attribute does not need
similarity to guess which section answers it. `rag/sections.py` maps the
question to a `section_key`; `QdrantRetriever.find_by_section` then filters on
`(drug_id, section_key)` and returns **every** part of that section as a
`scroll`, not a top-k. `RetrievalService` takes that route when it resolves and
falls back to similarity otherwise.

Two rules carry the safety. **Longest phrase wins**: "chống chỉ định" and "chỉ
định" differ by one prefix word and mean opposite things, so every phrase is
sorted by length and the longer is tested first — the same rule keeps "quá
liều" from being read as "liều" and "hướng dẫn xử trí ADR" from being read as
"tác dụng phụ". **No match is not a guess**: an unrecognised question returns
`None` and falls back rather than picking a section it is unsure of.

**Measured against the real `duocthu_v1` collection, no embedding involved:**

| | similarity (measured this afternoon) | section routing |
|---|---|---|
| hit@1, 160 generated cases | 0.544 | **1.000** |
| `chong_chi_dinh` | **0.05** | **1.00** |
| misroutes / empty / leaked sections | — | 0 / 0 / 0 |

**The generated 160 flattered it, and testing on human-written questions said
so.** Those questions use the phrasings the table was built from, so 160/160 is
partly circular. Run against the 16 single-drug questions humans actually wrote
in `Golden Dataset/golden_e2e_v1.csv`, the first version scored **10/16**. The
six failures were two gaps: four questions say just "Liều Metformin cho người
lớn?" — bare "liều", which the table lacked — and one says "Bà bầu", a
colloquial phrasing for pregnancy. Adding those phrases (no code change, which
is what the open/closed table is for) took it to **16/16** while the confusable
pairs still resolve correctly; bare "liều" is safe only because "quá liều" is
longer and tested first, and there is a regression test pinning exactly that.

**A circular import was found and fixed properly rather than worked around.**
`service -> sections -> routing -> service`, because `normalize_name` lived in
`routing.py`. It is a text utility with no knowledge of drugs or sections, so
it moved to `rag/text.py`; `routing.py` re-exports it so existing imports keep
working.

**Also wired, and still unproven:** `BedrockCohereQueryEmbedder` replaces the
SHA-256 hash embedder for the similarity fallback path. It has been
import-checked only — **never run against Bedrock** — so the fallback path
remains unverified end to end. The section route does not depend on it.

Verification actually run: ai-service **37 passed, 3 skipped** (22 before, +15);
ingestion **296 passed** (unchanged, checked for regression); `ruff --select
F,E9,B,ARG` over `rag/`, `adapters/`, `bootstrap.py`, `config.py` and `tests/`
— **all checks passed**; section-route evaluation against the live collection
160/160; human-written golden questions 16/16.

Not established: multi-attribute questions ("liều dùng và chống chỉ định") pick
the longest phrase, which is deterministic but arbitrary; phrase coverage
beyond these 16 human questions is unmeasured; and none of this speaks to
whether the retrieved text is clinically correct.

## 2026-08-04 (afternoon) — First real embeddings exist; retrieval measured at 54% and the cause is not what the small sample said

The corpus is embedded for the first time. Bedrock IAM was opened on the
owner's explicit instruction, all 15,100 chunks were embedded with
`cohere.embed-v4:0`, loaded into Qdrant, and **cloud access was then revoked
and proven revoked** before the owner's 17:00 deadline. Measured spend
**~$0.49** of a personal $138 budget.

**Gate results.** 15,100/15,100 embedded; 15,100 points in `duocthu_v1` over 59
batches; collection point count 15,100 — count gate **PASS**. Manifest records
`cohere.embed-v4:0`, 1024 dimensions, Cosine, corpus SHA
`04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`. Corpus SHA
was re-verified against the morning audit before spending: identical, and
identical to the post-lint copy, so the 12:05 `chunker.py` edit did not change
output.

**Both providers were probed live before choosing.** Titan v2 and Cohere v4
each returned 1024 dimensions with a **measured L2 norm of 1.000000**. That
settles a question left open since 2026-08-03: Cohere's `normalized` field was
`None` because AWS's docs never state it. It is now measured. Cohere was chosen
on two measured grounds — the corpus is Vietnamese and Cohere is explicitly
multilingual, and `bedrock_cohere.py` batches 96 texts per request while
`bedrock_titan.py` sends one, which at a measured 2.3s per call is ~9.6 hours
versus minutes. The $0.41 price difference did not drive it.

**The retrieval number, and a correction to a claim made earlier the same
day.** A 160-case evaluation (20 per section, 8 sections, questions generated
from the corpus so labels are structural) measured **hit@1 0.544, hit@3 0.663,
hit@5 0.738**. Per section:

| section | hit@1 |
|---|---|
| `chong_chi_dinh` | **0.05** (1/20) |
| `chi_dinh` | 0.30 |
| `tac_dung_khong_mong_muon` | 0.40 |
| `lieu_luong_va_cach_dung` | 0.60 |
| `qua_lieu_va_xu_tri` | 0.65 |
| `than_trong` | 0.65 |
| `tuong_tac_thuoc` | 0.80 |
| `thoi_ky_mang_thai` | 0.90 |

An earlier 15-case run gave a similar headline (0.533) but led to the **wrong
diagnosis**: four of its seven failures were contraindication questions
answered with indications, so the cause was reported as embedding weakness at
negation. At 160 cases that pair accounts for only **3** confusions. The
dominant mechanism is different and larger: **`duoc_ly_va_co_che_tac_dung`
absorbs questions from every other section** — 10 from adverse effects, 8 from
contraindications, 7 from dosage, 5 from indications. It is the largest section
(1,896 chunks) and describes the drug in general terms, so it sits close to
almost any question about that drug. This is the small-sample failure mode
CLAUDE.md warns about, reproduced on this project.

**Re-embedding cannot fix this, and the capability to fix it already exists.**
Verified by reading the code, not assumed: `apps/ai-service/adapters/qdrant.py`
`search()` filters on `drug_id` only and lets vector similarity choose the
chunk; `rag/routing.py` resolves drug and intent but **not section**; and
`find_by_payload` — the "return the whole section" method in `ingestion/load/`
— is **never called anywhere in `apps/ai-service`**. Attribute questions
therefore depend on similarity picking the right section, which is what
measures 54%. The fix is to resolve the attribute to a `section_key` and
retrieve that section whole; `ATTRIBUTE_TO_SECTION` already exists in
`embed/benchmark_local.py`.

**A silent-failure hazard found and closed.** `apps/ai-service` embedded
queries with `LocalHashQueryEmbedder` — SHA-256 of tokens, explicitly plumbing
only — while the collection now holds Cohere vectors. Querying across those two
spaces returns hits and raises nothing; the results are simply meaningless.
`BedrockCohereQueryEmbedder` was added and wired behind
`EMBEDDING_PROVIDER=cohere-v4`. **It has only been import-checked — never run
against Bedrock**, because cloud access was revoked first, as instructed.

**Two operational lessons, both paid for.** `bedrock_runtime.py` set no boto3
timeout, so a single throttled response held a socket open for over five
minutes and stalled the whole run; `connect_timeout=10, read_timeout=60` plus
standard retries fixed it. Then the first full run still died at ~14,600/15,100
because the retry backoff (2s, 4s) was far shorter than a per-minute token
quota needs. The disk cache made that survivable: the resumed run recorded
**14,977 cache hits and 123 misses**, so only 123 vectors were paid for twice —
zero, in fact, since the first run's work was already saved.

**Cloud shutdown, verified rather than asserted.** Both policies detached and
deleted; `InvokeModel` and `ListFoundationModels` both now return
`AccessDeniedException`. No EC2 instance, no EBS volume, and — because the
policy never granted `CreateProvisionedModelThroughput` — no way for this
identity to create the one Bedrock resource that bills hourly.

Not established: retrieval quality is not acceptable for clinical use, no
clinician-authored release gate exists, the generated evaluation questions use
template phrasing rather than real clinical language, and no LLM answer layer
has ever run against real evidence.

## 2026-08-04 — Chunk schema v4 passes the embedding-readiness gate

Reviewed the live Claude coordination and its last changes before editing. The
delivery plan was objectively stale: it still described schema v2/15,076 chunks,
empty embed/load/API modules, embedding before content-safety gates, and allowed
unverified inferred table headers as retrieval text. The plan and ADR 0004/0006
now put content safety, exact provenance, fail-closed schema validation and local
pseudo-vector smoke tests before any provider call. Bedrock remains benchmark-
only and requires separate owner approval for any paid/full-corpus run.

Implemented schema v4 and regenerated the canonical chunk artifact. Retrieval
`text` may repeat route/population labels so continuation chunks remain safe in
isolation; contiguous `source_text` remains byte-reassemblable and drives exact
physical/printed page provenance. `context_labels` records retrieval-only
prefixes. All 151 unverified table/formula descriptors embargo `header_row` and
cell-like column text. Attachments now carry physical page, printed page,
`block_id`, `bbox` and optional crop, and those region references survive the
Qdrant adapter and RAG citation response. The loader accepts exactly schema v4,
rejects booleans/non-integers/out-of-range pages, and keeps the normalized
LF/CRLF-stable corpus identity.

Canonical artifact measured after regeneration:

- 15,100 chunks: 14,949 prose + 151 block descriptors;
- 4,105,382 `cl100k_base` tokens; 0 chunks above the 800-token ceiling;
- all `chunk-ready` gates pass: exact provenance, source uniqueness,
  reassembly, attachment coverage, descriptor embargo and schema checks all
  have 0 failures; 151 descriptors match 151 quarantined blocks;
- raw file SHA-256:
  `8dfae08ae6d9222089c5cdb4207a064fe67989f10f7552b555af0aef6331d9a1`;
- normalized corpus SHA-256 used by the Qdrant manifest:
  `04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`.

Verification actually run:

- ingestion: **292 passed**; focused post-lint patch: **26 passed**;
- AI service with `RUN_INTEGRATION=1`: **25 passed**, including real local
  Qdrant, PostgreSQL and FastAPI round-trips;
- full canonical local smoke with deterministic 4D pseudo-vectors: first and
  second loads both upserted 15,100 records and both held exactly 15,100 points;
  manifest hash matched; data and sidecar test collections were removed and
  Qdrant returned to 0 collections;
- Ruff `F,E9,B,ARG` on the files changed for this gate: clean; `git diff
  --check`: clean (Git only reported Windows LF/CRLF conversion warnings).

Conclusion: the canonical corpus is **technically READY TO EMBED**, meaning its
input/schema/provenance/load plumbing meets the measured gates. This does not
authorize a provider call, does not establish retrieval quality for any model,
and does not prove whole-book medical accuracy. Human-reviewed clinical eval,
table reconstruction, and recall for borderless tables/bar-less formulas remain
outside what these gates prove.

## 2026-08-04 — Real local datastore plumbing, guarded RAG API, and printed-page citations

Read the live Claude Code process and coordination before editing. Claude owned
`ingestion/load/` and `embed/cache.py`; it completed the disk cache, Qdrant
port/adapter, idempotent UUID5 upsert, payload indexes and corpus-SHA manifest.
Its real local Qdrant scale check loaded all 15,066 chunk records twice with
1,024-dimensional deterministic pseudo-vectors and held the point count at
15,066. Those vectors are not embeddings and establish no retrieval-quality
claim. No Bedrock call, IAM change, or cloud spend occurred.

Built the first runnable `apps/ai-service` boundary: FastAPI `/health` and
`POST /v1/rag/query`, a Qdrant retriever filtered by resolved `drug_id`, a
PostgreSQL trace repository plus migration, structured human/non-human scope
and fact/recommendation intent gates, parent hydration, quarantine handling,
and an extractive answer layer. The answer layer refuses evidence that has only
a physical page; citations expose only the printed folio, chunk id and optional
source crop. Quarantined tables/formulas return a PDF-verification warning and
never auto-extract numeric content.

Fixed the missing provenance at its source. Chunk schema is now v3 and
`cli chunk` reads the real folio map from the 1,668-page PDF. It refuses a
monograph whose physical range cannot be mapped, and `chunk-ready` has a new
`chunk_without_printed_page_range` gate. Regenerated scope: 684 monographs,
15,066 chunks (14,915 prose + 151 descriptors), zero oversized, and
15,066/15,066 records with a two-value printed-page range. New artifact SHA:
`e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38`.

Verification actually run:

- `python -m pytest -q` and Ruff over `ingestion/`: **258 passed**, lint clean;
- `python -m ingestion.cli chunk-ready`: every gate passed, including printed
  page range 0/0 failures;
- ai-service with `RUN_INTEGRATION=1`: **22 passed**, including a real chunk
  round-trip through local Qdrant, PostgreSQL migration/insert/read-back, and a
  full FastAPI → Qdrant → guarded citation → PostgreSQL trace round-trip;
- local Docker services: PostgreSQL 16 and Qdrant 1.18.3 reachable; integration
  collections were UUID-scoped and removed after tests;
- ArgoCD local: namespace, CRD and seven controller pods are running; the
  existing unrelated `guestbook` lab app is Synced/Healthy with four history
  entries. This repo's three Application YAML files parse and point to
  `master`/the Helm chart, but they are not installed and the chart still has
  no workload templates, so project sync/rollback was not performed.

Still open: no real embedding exists, no full canonical Qdrant collection can
serve semantic search, `population_tags` are absent, no clinician-authored
release-gate cases exist, and the API currently has no production answer/query
embedding provider. The local hashing provider is explicitly plumbing-only.

## 2026-08-04 — Load stage built and proven against a real Qdrant; bbox rounding found

`ingestion/load/` was a 0-byte `__init__.py`. It now holds the vector-store
boundary: a `VectorStore` port, an `InMemoryVectorStore` that is the reference
implementation of its contract, and `QdrantVectorStore` as the only module that
names `qdrant_client` — imported lazily, the same arrangement that confines
boto3 to `bedrock_runtime`. `embed/cache.py` was added alongside it.

Three design decisions are worth carrying forward.

The cache key is `(model_id, input_kind, text_sha256)`, not `chunk_id` as
§4.A of the delivery plan proposed. Measured reason: `chunks.jsonl` holds
15,066 records but only **14,869 distinct texts**, so 197 records (1.31%) are
repeats that a chunk-keyed cache would pay for twice. The content key also
cannot serve a stale vector after an edit — a changed text is a changed digest,
so it is a miss.

Point ids are `uuid5(chunk_id)`. A random id would make a re-run append a
second copy of a dose and nothing would report an error.

The corpus manifest lives in a `<name>__manifest` sidecar collection rather
than a reserved point inside the data collection, because
`qdrant_point_count != chunk_count` is a v1 gate and a gate needing an
"except the manifest" footnote will eventually be read wrong.

**Whole-corpus check against a real server.** A local Qdrant **1.18.3** was
started from `infra/docker/docker-compose.yml` (local container, no cloud) and
all 15,066 real chunk records were loaded with deterministic pseudo-vectors at
1,024 dimensions — a check of the loading mechanism, **not embeddings, which
still do not exist**. Corpus sha256 `30d5154273e0959a…`. First load: 15,066
points in 59 batches, 14.0s, point-count gate PASS. Second load: still 15,066,
so idempotency holds at real scale, not only against the fake store.

**That sha is already stale, which is the point.** `chunks.jsonl` was
regenerated at 09:53 the same day — `chunker.py` changed two minutes earlier
and every chunk gained `printed_page_range`, 18,229,918 → 18,753,003 bytes,
sha now `e474c83790b450d3…`. Re-measured on the new artifact: still **15,066
chunks, 0 over the 800-token ceiling** (largest exactly 800), all 15,066
carrying `printed_page_range`, 14,915 prose + 151 block descriptors, 197
duplicate texts (1.31%) unchanged because only a field was added. Suite
**258 passed**. Had the old corpus been embedded and loaded, then the new one
loaded into the same collection, two generations would have mixed with no error
at query time — A6 is what refuses that, and it now has a real instance rather
than a hypothetical one.

**A sampled check passed and was wrong.** Comparing 5 payloads gave 5/5
identical. Scrolling the entire collection instead found **86 of 15,066 chunks**
whose payload did not equal its source record. Classifying every differing leaf:
**96 differences, all floats, all inside `attachments[].bbox`, maximum absolute
delta 5.684e-14**, and **zero** non-float differences — every text, id, page
number, page range, token count and boolean round-tripped exactly. A PDF point
is 1/72 inch, so that delta cannot move a rendered crop. It is pinned by a
regression test that fails if the loss reaches another field or grows past 1e-9.

The layer responsible was isolated rather than assumed: the source
`chunks.jsonl` returns the value exactly, our own `json.dumps`/`loads` returns
it exactly, and **Qdrant reached over raw HTTP with no SDK involved** returns it
one ULP low. Nothing needs re-chunking — a regenerated corpus would carry the
identical value and be rounded identically. Qdrant also stores dense vectors as
float32, so precision beyond f32 is discarded at load regardless.

Cache format was decided on measurements, not preference: 300 real chunk texts
at 1,024 dimensions cost **21,098 bytes/record — ~318 MB per model** for the
corpus, with a **7.8s** offset-index rebuild per open. float32 `.npy` (62 MB)
and base64 float32 in JSONL (~87 MB) were measured and set aside; append-only
JSONL survives an interrupted run and stays readable, which outweighs disk at
one or two models. Revisit at three (~950 MB). It lands in
`ingestion/data/processed/`, already excluded by `.gitignore:34`.

**A gap in this work, found and closed the same day.** Payload indexes were
created on `drug_id`, `section_key`, `atc_codes` and `chunk_kind` and reported
as done — but `VectorStore` had no query method, so all that was really proven
is that `create_payload_index` returns without raising. Filtered retrieval is
the whole of mode A. `find_by_payload` now exists on the port and both stores,
as a `scroll` rather than a `search`: it returns **every** match, never a
top-k, because "return the whole section" is the plan's non-negotiable — two of
five contraindications reads as a complete list. Verified on a real server: all
five parts returned with no leak from the PANTOPRAZOL/OMEPRAZOL pair that
measures cosine 1.000 on contraindications; a deliberately 300-part section
(above the 256 scroll page) comes back whole so paging cannot truncate; and a
real multi-part section from `chunks.jsonl` round-trips to exactly its own
chunk ids.

Tests: **255 passed** with Qdrant running (206 before this work, +49);
**247 passed, 8 skipped** with it stopped, so an offline machine and CI see
skips rather than failures. After the mode A work and the other worktree's
`cli.py` fix the suite stands at **268 passed** and
`ruff --select F,E9,B,ARG` reports **no findings at all** across `ingestion/`.

Still missing, and deliberately so: `printed_page_range` and `population_tags`
are not in the payload (open questions to Codex in
`coordination/CLAUDE_TASK_2026-08-04.md`); `cli embed` / `cli load` are not
wired because `cli.py` is Codex's; and **no real embedding vector has ever been
produced** — every vector the load path has carried was synthetic. The Bedrock
request shapes remain documentation-derived and unproven.

Measured cost: **$0**. No Bedrock call, no IAM change, no cloud resource.

## 2026-08-03 — Bedrock embedding boundary built; IAM diagnosed, not yet opened

`ingestion/embed/` was an empty `__init__.py`. It now holds the provider
boundary the model benchmark needs: an `EmbeddingProvider` ABC that owns input
validation, request-size batching and timing, and three adapters behind it —
`amazon.titan-embed-text-v2:0`, `cohere.embed-v4:0`, and `BAAI/bge-m3` as the
zero-cost local control. boto3 is named in exactly one module and imported
lazily, so the package imports and the whole suite runs with no AWS account.

Two design points are worth carrying forward. `input_kind` is a required
argument, not a keyword: Cohere embeds corpus records and queries into
different subspaces, and sending `search_document` for a query raises no error
— recall just drops. And `normalized` is three-valued. Titan is asked to
normalize and says so; the Bedrock docs never state whether Cohere's float
vectors are unit-length, so that field stays `None` instead of guessing, and
`embed.probe` prints a *measured* L2 norm to settle it on the first live call.

The AWS side is diagnosed and stuck. `ai-lab-user` has no inline and no
attached user policy; its one group (`AI-Lab-Group`) grants EC2, IAM, ELB and
VPC full access and nothing else. There is no `bedrock:*` grant anywhere on
the identity — confirmed by running both `list-foundation-models` and
`invoke-model` and reading the two `AccessDeniedException` messages. Two
least-privilege policies are drafted in `infra/aws/iam/` but **deliberately
not applied**: that identity carries `IAMFullAccess` and could attach them
itself, which is exactly why it was left to a human.

Consequence: every request-body shape in the two Bedrock adapters is derived
from the AWS user guide (read today) and **has never been accepted by the
service**. That is unproven, not verified. Tests: 22 new, all with a stub
invoker and zero network; **203 passed** overall, up from 181. Lint clean on
every file added (`--select F,E9,B,ARG`); the one remaining finding is a
pre-existing `cli.py` import owned by the other worktree.

Measured cost so far: **$0**. Nothing was embedded, nothing reached Qdrant.

## 2026-08-03 — Exact hard-10 gate and all-block table chunking experiment

Extended the isolated table/formula sandbox beyond the 100-page sample. An
exact ten-block risk gate covered four cross-page pairs, a merged header, a
fragmented fraction bar, and the bar-less ADENOSIN formula; all ten source crops
were visually checked. The full run then processed all 151 canonical blocks:
141 physical tables, ten formulas, 133 logical table parents, 669 row children,
and seven cross-page logical tables.

Full-scope visual inspection exposed a continuation bug: FAMCICLOVIR p647 and
INSULIN p811 repeat their column headers, while other continuation pages start
directly with data. The linker now distinguishes these cases; repeated headers
are not emitted as data, and INSULIN's changed `Phối hợp` first-column meaning
is preserved. Both branches have regressions.

The expanded, source-derived retrieval suite contains 2,436 cases. With drug
and table/formula lane resolved before ranking, deterministic hybrid character
TF-IDF measured 94.42% Recall@1, 99.79% Recall@5, and 96.90% MRR. Row questions
were 94.82% / 100%; formula questions 100% / 100%. Five ambiguous whole-table
questions fell below top five because the same drug owns several near-identical
tables; production must clarify or route using an additional table anchor.
Neural MiniLM is now opt-in and excluded from the default parsing gate.

Measured chunk design: table-parent tokens min/median/p90/p95/max =
66/188/441/678/1,893; only four of 133 parents exceed 800. Row children are
75-token median, 172 p95, 471 max. Keep every logical parent intact, index both
parent and header-aware rows, never split a row, and hydrate row hits to the
complete parent/source pages. Final checks: **181 tests passed**, readiness
20/20, lint clean.

---

## 2026-08-03 — 100-page table/formula reconstruction and RAG sandbox

Built an isolated experiment under `ingestion/scratch/rag-table-pilot` without
writing sandbox representations into the canonical corpus. The risk-stratified
100-page run reconstructed 120 tables and 10 formula regions, rendered and
manually inspected all 130 crops, and linked four tables continued across page
pairs 132-133, 646-647, 825-826, and 1373-1374.

The retrieval router fixes the drug and data lane before vector ranking. On 461
source-derived queries, hybrid row+whole character TF-IDF reached 92.62%
Recall@1, 98.70% Recall@5, and 95.04% MRR. Cached English-oriented MiniLM was
worse (88.29% / 97.18% / 91.92%). Eighteen row-hit answer previews all hydrated
to the complete parent Markdown table; eight included both pages of a continued
table. A narrow deterministic interval probe passed 172/172 generated cases;
this is a mechanics check, not clinical ground truth.

Visual review exposed one canonical defect: ADENOSIN p147's bar-less printed
formula region ended after its numerator and omitted `Nồng độ adenosin
(3 mg/ml).` The bar-less band now extends 31pt below its synthetic anchor,
capturing the denominator but stopping before `Ví dụ:`; a regression pins that
boundary. Canonical artifacts were regenerated after the fix: 684 monographs,
11,974 sections, 15,066 chunks, 151 descriptors, 0 unassigned spans, all 20
readiness gates passing, **180 tests passed**, and lint clean.

Decision: JSON grid + Markdown answer view, row and whole-table retrieval, and
mandatory parent hydration are viable for the next stage. This remains a
retrieval experiment, not production clinical approval; merged-cell semantics,
unit/multi-axis reasoning, Vietnamese embedding comparison, borderless/bar-less
recall, clinician-authored evals, and final expert review remain open.

---

## 2026-08-03 — Whole-corpus parser repair after manual baseline audit

Implemented and re-ran the parser over all 1,668 pages after manually reading
the high-risk baseline outliers. The fixes are structural, with regressions:

- restored the missing `THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN`
  boundary (`Tên chung quốc tế và mã ATC` is its real first anchor), separating
  pages 1371–1373 from `THUỐC PHIỆN - OPIAT - OPIOID`;
- require both physical and inferred printed page bounds, so back-index page
  1655 can no longer extend ZOLPIDEM's real `[1492, 1494]` range;
- keep plain label-shaped text as body when it is an adjacent wrapped
  continuation in the same PDF block (including NADROPARIN's “không phải là
  chống chỉ định”);
- classify known table cells before headings, putting WARFARIN and IOBITRIDOL
  dosing tables back under `lieu_luong_va_cach_dung`;
- added confirmed heading variants for CLORPHENIRAMIN dosage forms and tetanus
  toxoid dosing, and real provenance for combined inline fields;
- made verified formula bands column-aware: NETILMICIN opposite-column prose
  is retained while AMPICILIN's gutter-adjacent formula stays quarantined;
- visually inspected all **151/151 unique table/formula regions** against the
  rendered PDF; every region is genuinely 2D and remains quarantined;
- emit every physical table/formula region atomically at its first stream
  occurrence, fixing split/contradictory ownership on CAPECITABIN, IMATINIB,
  CARBOPLATIN, NETILMICIN, and TRASTUZUMAB;
- route explicit `Bảng N. Điều chỉnh liều ...` appendices back to dosage even
  when the book prints them after `Tên thương mại` (CAPECITABIN p309);
- added readiness gates for every individual section part's source-span IDs
  and duplicate physical-region IDs.

Final regenerated artifacts and evidence:

| check | result |
|---|---|
| tests | **180 passed**; lint clean |
| segmentation | **684 monographs**, 11,974 sections, 8,213,036 prose chars |
| back-index validation | **96.2% recall (678/705), 99.1% precision** |
| quarantined regions | **151 blocks / 151 unique IDs**, all visually checked |
| chunks | **15,066** (14,915 prose + 151 block descriptors), 0 over 800 tokens |
| chunk readiness | **20/20 PASS** (including duplicate-region prevention) |
| coverage | 252,799 spans, **0 unassigned** across all 1,668 pages |
| residual ink | 3,931 classified regions, **0 unclassified** across all pages |

Canonical `ingestion/data/processed/{monographs,chunks,coverage_ledger}` were
regenerated. Remaining limits: no whole-document human-reviewed clinical
ground truth, no row/column reconstruction for quarantined tables, and unknown
recall for borderless tables/bar-less formulas. This is ready for retrieval
experiments, not a claim of production clinical approval.

---

Chronological record of work done on this project, newest entry on top. The
goal is continuity across sessions: if a work session ends unexpectedly
(context/token limit, interruption), whoever picks this up next — human or
Claude — should be able to read the latest entry and know exactly what's
done and what's next, without having to reconstruct it from git history.

**Convention**: add a new entry at the top before ending a session whenever
meaningful progress was made, and proactively the moment it looks like the
session might run out of context/tokens mid-task — don't wait until the very
end if that risk is showing.

---

## 2026-08-03 — Independent re-verification, a redundant rule in my own uncommitted fix, a provenance defect, and the measurements a retrieval design has to be built on

No code was changed in this session: the four files from the previous round
are still uncommitted and under external review (Codex). Everything below is
measurement, and the numbers live nowhere else — the investigation scripts
were deleted per the repo rule, so this entry is the record.

### 1. Re-verified the whole tree from scratch

| check | command | result |
|---|---|---|
| tests | `python -m pytest -q` | **164 passed**, 39.31s |
| lint | `ruff check --select F,E9,B,ARG .` | clean |
| gates | `cli chunk-ready` | **18/18 PASS**; 683 monographs, 11,966 sections, 8,212,880 chars, 167 quarantined blocks |
| recall/precision | `cli validate --pdf …` | 683 detected, 705 ground truth, **96.0% (677/705) / 99.1%** |
| span ledger | `cli coverage --pdf …` (re-run) | 252,799 spans after merge, 9,398,772 chars, **unassigned 0** |
| reproducibility | `cli run` → sha256 | **byte-identical** to `monographs.jsonl` (`84f41d96…`) |
| reproducibility | `cli chunk` → sha256 | **byte-identical** to `chunks.jsonl` (`63472db4…`); 15,076 chunks (14,909 prose + 167 descriptors), 0 oversized, 4,072,725 tokens (cl100k_base) |

Recall rose 92.9% → 96.0% because of the uncommitted back-index rejoin, and
the mechanism is the denominator: 725 → 705 ground-truth entries once wrapped
fragments stop counting as entries. The detector did not improve.

**Not re-run: `cli residual-ink`.** `residual_ink.json` is dated 2026-08-01
12:03, before the 17:36 assembler edits. Its stored contents (3,931 regions,
no `unclassified` kind) are last session's numbers, not this session's.

**Doc drift found:** `docs/verification-strategy.md` quotes 252,733 spans /
177,679 `normalized_text` / 12,764 `heading`; measured today 252,799 /
177,754 / 12,752. The `unassigned = 0` conclusion still holds.

### 2. The x0 geometry change in the uncommitted diff is redundant

Assembled the whole book four times with the two new rules toggled:

| variant | monographs | sections | chars |
|---|---|---|---|
| current (x0 + italic) | 683 | 11,966 | 8,212,880 |
| **old x1 rule + italic** | 683 | 11,966 | **8,212,880** — 0 differences of any kind |
| x0, no italic | 683 | 11,966 | 8,212,844 (3 sections differ) |
| x1, no italic (= `bc01782`) | 683 | 11,966 | 8,212,780 (9 sections differ, 11,014 char delta) |

The italic rule alone recovers all 9 sections (CEFAZOLIN dosing 1,062 →
5,620 chars; CALCI LACTAT `than_trong` 582 → 1,504, `tuong_tac_thuoc` 2,346 →
1,439). The x0 rule alone recovers 6 of 9 and adds **nothing** on top of the
italic rule.

Worse, the justification is wrong: the real NEVIRAPIN span on physical page
1045 is `TimesNewRomanPS-ItalicMT` (verified by reading the span's font), so
the italic rule is what fixes that page — not the 0.01pt overlap the code
comment and the new test's docstring credit. The test itself is valid but
pins the geometric rule only, because the `_span()` fixture helper never
produces an italic font. **Either keep the x0 rule as defence-in-depth with
an honest comment, or revert it — but the current comment overstates it.**

### 3. `source_page_range` is wrong for 13 of 683 monographs

Section-level provenance (`parts`) is correct everywhere; the monograph-level
page range is not. 12 monographs overshoot by +1 page; **ZOLPIDEM declares
`[1492, 1655]` while every one of its sections comes from 1492-1494** — a
164-page claim reaching into the back index.

Root cause for ZOLPIDEM, confirmed: physical page 1655 (printed 1656, back
matter) carries a **bold** span reading exactly `Tương tác thuốc`, which
`_classify` emits as a `_SectionEvent`, and the `_SectionEvent` branch at
`segment/assembler.py:496` updates `source_page_range[1]` with **no
`in_monograph_range` guard** — unlike the `_TextEvent` branch at line 511.
Verified that **0 spans past physical 1495 pass `in_monograph_range`**, so no
text was contaminated and `empty_section` is still 0. The defect is confined
to one provenance field.

The +1 cause is **not isolated** — it is not lifted tables (all 12 have
`tables: []`); the likely candidate is a next-page boilerplate span bumping
the range before being excluded, but that was not measured.

### 4. Corpus profile — what a retrieval design actually has to work with

- **13 of 19 fields have p90 < 1,500 chars**, i.e. the whole section fits one
  chunk. Only four routinely need splitting: `duoc_ly` (p90 4,939, max
  14,099), `lieu_luong` (4,873 / 14,197), `than_trong` (2,419), `tuong_tac`
  (2,147). Confirms ADR 0004 on the cleaned corpus.
- **ATC**: 668/683 (97.8%) carry ≥1 code, **171 (25.0%) carry more than one**,
  max 20, 1,043 distinct codes.
- **`ten_thuong_mai` present in 492 (72%)** monographs.
- **The back index holds 344 `X - xem Y` lines** — brand → generic aliases —
  which `parse_back_index` currently discards wholesale (correct for
  validation, but this is the highest-value retrieval asset in the book,
  because clinicians type brand names).
- **401 `xem [thêm] mục/chuyên luận` phrases across 261 monographs**; a chunk
  containing one is useless retrieved alone.
- **Dosing population markers**: `Trẻ em` 53%, `Người lớn` 51%, `Người cao
  tuổi` 15%, `Trẻ sơ sinh` 8%, `Suy thận` 8%, `Suy gan` 6% of 682 dosing
  sections — real sub-section boundaries, better split points than token
  windows.
- **167 quarantined blocks, 129 (77%) inside `lieu_luong_va_cach_dung`** —
  the most dangerous field is the one the tables were lifted out of.

### 5. Cross-drug confusability — the number that decides the architecture

First hypothesis (much repeated boilerplate across drugs) was **refuted**:
only **171 of 11,966 sections** share exact text with another drug (1.4%), and
the six heavy clinical fields are 100% distinct.

Then measured, per field, each drug's TF-IDF cosine against its *nearest other
drug*. **This is a lexical proxy, not an embedding measure** — it bounds the
problem from one side only.

| field | median | p90 | p99 | max | drugs with NN > 0.7 |
|---|---|---|---|---|---|
| `lieu_luong_va_cach_dung` | 0.314 | 0.455 | 0.631 | 0.836 | 4 (0.6%) |
| `tuong_tac_thuoc` | 0.284 | 0.461 | 0.870 | 0.984 | 18 (2.8%) |
| `tac_dung_khong_mong_muon` | 0.300 | 0.437 | 0.856 | 1.000 | 13 (1.9%) |
| `chi_dinh` | 0.408 | 0.637 | 0.885 | 0.924 | 31 (4.5%) |
| `chong_chi_dinh` | 0.346 | 0.633 | 0.898 | **1.000** | 37 (5.4%) |

Named pairs: `PANTOPRAZOL ↔ OMEPRAZOL` (contraindications **1.000**,
indications 0.913) · `BENZATHIN PENICILIN G ↔ PHENOXYMETHYLPENICILIN`
(contraindications **1.000**) · `DIGOXIN ↔ DIGITOXIN` (0.891 / 0.911) ·
`NATRI NITRIT ↔ NATRI THIOSULFAT` (dosing 0.631 — two different steps of the
same cyanide-antidote protocol) · `IOBITRIDOL ↔ ACID IOXAGLIC` (0.984) ·
`ESTRIOL ↔ ESTRON` · `GLICLAZID ↔ GLIMEPIRID` · `NAPHAZOLIN ↔ OXYMETAZOLIN`.

Name layer: **19 drug names are a substring of another drug name**
(`CLOROTHIAZID` in `HYDROCLOROTHIAZID`, `EPHEDRIN` in `PSEUDOEPHEDRIN`,
`LORATADIN` in `DESLORATADIN`, `ATROPIN` in `HOMATROPIN HYDROBROMID` — all
genuinely different drugs), and 106 of 683 names share a 6-character prefix
across 38 clusters.

**Conclusion drawn from this, for the retrieval design: vector similarity must
never be allowed to choose the *drug* — only which passage within an
already-resolved drug.** The dangerous confusions are concentrated in a
small, enumerable set of same-class pairs, which is exactly the population
this project's verification strategy says to census rather than sample.

### Not done yet / next up

Sequenced in **`docs/v1-delivery-plan.md`** (written this session): a
two-week plan to a running v1, scoped down to two deployables (`web` +
`ai-service`) because the four NestJS services measure 0 `.ts` files each.
The items below are the ones that plan depends on.

- The confusable-pair census must become a **committed fixture produced by
  production code** (`ingestion/validation/`), not a deleted scratch script.
  Until then these numbers are only in this entry.
- ADR 0007 (retrieval architecture) not written. Proposed content: vectors
  never pick the drug; the unit returned to the LLM is the **complete
  section** (enabled by `section_not_reassemblable_from_chunks = 0`, because a
  partial contraindication list reads as "no contraindication"); and eval
  split in two — **routing** correctness (ground truth derivable from the
  corpus itself, 683 × 19 pairs, no human needed) versus **content**
  correctness (requires a clinician; cannot be self-generated without
  fabricating evidence).
- Entity/alias layer (683 canonical names + 344 back-index aliases + 492
  `ten_thuong_mai` + 1,043 ATC codes) — zero-regret, needed by every
  architecture, must use longest-exact-match because of the 19 substring
  traps.
- `residual-ink` re-run; `verification-strategy.md` numbers re-synced;
  regression test for `parse_back_index` (still has none); the
  `source_page_range` guard; the x0-rule comment decision.
- Open question for the user, not a technical one: this is the **2018
  edition**; the 3rd edition (2022) exists. For a document with legal force
  over prescribing, staying on 2018 should be a deliberate decision, and it
  makes edition-independence a real requirement for the pipeline.
- Still untouched: `embed/`, `load/`, Qdrant, `ai-service`, and the general
  chapters (printed 37-98) and appendices (printed 1497-1528), which remain
  outside the corpus entirely.

## 2026-08-01 (cont'd, 7) — "still errors?" — yes: two more real content-loss bugs, both in dosing sections

Asked whether errors remained after the previous round, the honest answer was
that this session has found real defects every time it looked one level
deeper. It looked again, and found two more.

**1. Chunks ended on a bare population label, with the dose in the next
chunk.** `split_sentences` treats `:` as a sentence boundary and
`_OPENS_SENTENCE` accepts a digit, so `"Người lớn: 500 mg mỗi 8 giờ."` splits
after the colon. When the packer flushed at that point, the chunk ended on the
label. Measured: **38 prose chunks**, e.g. AMOXICILIN's ending on a Lyme
indication followed by a bare `Người lớn:`. Retrieval on that chunk returns a
population with no dose; on the next, a dose with no population. Outlier item
17 counted population markers on 1,121 of ~1,400 monograph pages, so this is
the common shape, not an edge case. The packer now carries trailing label
atoms into the next part instead of flushing on them: **38 → 2**, and chunks
ending on any colon **721 → 19**.

**2. A section name printed mid-line was swallowed as a heading — real text
loss, in dosing sections.** Chasing the last 2 of those 38 showed the defect
was not in chunking at all. CISPLATIN (physical page 402) prints
`Suy thận: Chống chỉ định.` inside `liều lượng và cách dùng`; the second half
is itself a section name, so it was matched as a heading. The result: the
renal-impairment contraindication **disappeared from the dosing text** and the
section ended on a bare `Suy thận:`. ISOPRENALIN had the same shape. Same
family as the FLUOROURACIL bug fixed earlier today, but that rule only covered
a label directly *under* a heading and could not see this one.

Fixed geometrically: a real section heading opens its line, so a non-bold
section name with another span printed to its left is body text. "To the left"
is checked properly — same page/block/line *and* `previous.x1 <= span.x0` —
because the synthetic test fixtures place every span at identical coordinates,
and a looser check passed on real data while breaking the AMITRIPTYLIN
inline-value case.

Verified after the fix: CISPLATIN's dosing section contains
`Suy thận: Chống chỉ định.` again, ISOPRENALIN's `Trẻ em:` is followed by its
doses, and `chong_chi_dinh` is no longer polluted. Monograph and section counts
unchanged at 683 / 11,966 — nothing was traded away for the recovery.

**State:** 18/18 gates pass, **163 tests** (was 161), ruff F/E9/B/ARG clean,
15,077 chunks with 0 over the ceiling, 8,212,780 section characters.

**Standing conclusion, worth writing down:** every round of "is it clean now?"
this session has ended with real defects found — five in the previous round,
two in this one, and four of the previous five were in code written the same
day. The gates and tests prove what those instruments can see. They do not
prove the corpus is correct, and the largest unmeasured area is unchanged:
content accuracy against the source, with no human-reviewed ground truth for
8.2M characters.

## 2026-08-01 (cont'd, 6) — Bug hunt after declaring "clean": the token count was wrong by 2x, 14.7% of chunks were over the ceiling, and two stage boundaries measured different pipelines

I had just reported the tree as clean. It was not. Going looking properly
found five real defects, four of them in code written earlier the same day.

**1. `estimate_tokens` was wrong by a factor of two, and the number it
produced was reported.** ADR 0004 sized chunks with `len(text) // 4`,
described honestly as an estimate. Measured against `cl100k_base` on the real
corpus:

| | |
|---|---|
| estimate (chars/4) | 2,115,427 tokens |
| real tokenizer | **4,093,440 tokens** |
| real/estimate | median **1.95**, p95 2.50, max **6.0** |
| oversized by estimate | **0** |
| oversized in fact | **1,884 of 12,838 = 14.7%**, largest 1,645 tokens |

Vietnamese diacritics cost several byte-pair tokens each. "0 oversized" was
reassuring and false. `chunk/tokens.py` now counts with the real tokenizer,
injected so the chunking logic stays testable without it, with a fallback of
chars/2 that errs small rather than large.

**2. The packer could exceed the ceiling on its own.** Two causes, both
measured on VORICONAZOL's `tương tác thuốc`: an atom of 710 tokens was left
whole because it was under the 800 ceiling, and the overlap builder added
whole atoms until the running total *passed* the budget, so a 251-token atom
produced a 273-token overlap against a 65-token setting. 273 + 710 = 983.
Atoms are now split against the 650 target, leaving room for overlap, and the
overlap stops *before* exceeding its budget.

**3. An over-long comma list was left as one atom.** VORICONAZOL's
interaction list is one "sentence" hundreds of drug names long. Truncated by
an embedding model it reads as "this drug is not listed" — a false negative
in the direction that matters. Split at commas, which is lossless for a list.

After 1-3: **0 chunks over the ceiling**, verified by an independent tiktoken
re-count of the written file, not by the pipeline's own number. 15,049 chunks
(was 12,838 — the rise is real sub-chunking that should have happened all
along).

**4. `cli validate` measured a different pipeline than `cli run`.** It used
the raw span stream (no transcription repair) and passed no table regions, so
recall/precision described a build that is not the one producing the output —
the same class of mismatch already fixed for `coverage`. Now shares
`_extracted_and_repaired_spans` and `_region_index`. Result after the fix is
unchanged at 92.9% / 99.1%.

**5. `chunk/io.py` dropped `SectionPart` when reading monographs back**, so
per-part provenance died at the stage boundary — against CLAUDE.md's explicit
rule. Now carried: 12,290 parts across 11,966 sections.

**Two new gates, and the gate itself was wrong twice before the data was.**
`section_not_reassemblable_from_chunks` rebuilds each section from its own
chunks by removing the deliberate overlap and compares. First version joined
chunk texts with a newline and reported **734** sections missing — the first
one it named was present. Second version probed a 60-character head and
reported **1**, NAPROXEN, where the probe straddled an overlap seam that
legitimately repeats text. The working version compares with whitespace
removed, because each split seam loses exactly one space to `.strip()`
(measured on ABACAVIR: two single spaces in a 4,232-character section,
nothing else). It proves no character of content is lost, reordered or
duplicated beyond the intended overlap. **0.**

**Also fixed:** all 8 real lint findings (`ruff --select F,E9,B,ARG`) — five
unused imports and three `zip()` calls without explicit `strict=`. The zips
were the adjacent-pair idiom and not bugs; `strict=False` now says so. And
the transcription splice could leave a fragment holding only a space, which
showed up as two `whitespace_only` spans; dropped, and proven inert by the
sha256 over every section's text being byte-identical before and after
(`6af13301…`).

**State after the hunt:** 18/18 gates pass (10 corpus + 8 chunk), 161 tests
(was 158), `ruff F/E9/B/ARG` clean, `unassigned = 0`, `cli validate` 92.9% /
99.1%, 15,049 chunks with 0 over the ceiling.

## 2026-08-01 (cont'd, 5) — ADR 0006 implemented: chunks now reference their lifted blocks; `chunk/` runs for the first time; 16/16 gates green

**Why this was needed, in one line**: a chunk of a section whose table had
been lifted was grammatical, complete-looking prose with the table absent and
nothing marking the absence — silent incompleteness, in the section where 127
of 167 lifted blocks live (`liều lượng và cách dùng`, 76%).

**Design is in `docs/adr/0006-quarantined-block-references-in-chunks.md`**,
written before any code. It resolves the item ADR 0005 explicitly deferred.

**Implemented:** `ChunkAttachment` (block_id, kind, shape, physical_page,
bbox, quarantined, header_row) on every prose chunk, plus one
`block_descriptor` chunk per block whose text is built **only** from
metadata. `chunk/io.py` now reads `tables` (it silently dropped them before)
and writes `schema_version: 2`.

**`chunk/` executed for the first time**, whole corpus:

| | |
|---|---|
| chunks | **12,838** — 12,671 prose + 167 descriptors |
| prose chunks carrying a lifted block | 185 |
| oversized (>800-token ceiling) | **0** |
| estimated tokens (chars/4, an estimate) | 2,115,427 |

**The condition this work was accepted under — prose chunks must not
change — was measured, not asserted.** Built the corpus both ways and
diffed:

| check | result |
|---|---|
| prose chunk count, both ways | 12,671 / 12,671 |
| chunk id sets identical | yes |
| `prose_text_changed` | **0** |
| `prose_nonattachment_field_changed` | **0** |

Only the two new fields differ. The change is strictly additive.

**A gate caught a real defect in my own design within minutes of existing.**
`block_text_leaked_into_chunk_text` fired on AMIODARON (physical page 183):
pdfplumber reported that table's first row as `"Thời gian liệu pháp tĩnh mạch
Liều 720 mg/ngày (0,5 mg/phút)"` — **a dose, inside what it called a
header**, from an extraction never verified by eye, being embedded as
retrieval text. Measured across the corpus: **42 of 124 simple-table headers
(34%) contain a digit.** Rule added: a header row is embedded only when no
cell contains a digit and every cell is short enough to be a label. 76 of 167
descriptors (46%) keep a header under that rule; the AMIODARON one does not.
A label with no digit cannot be mistaken for a dose.

**Full gate suite, 16/16 pass** — 10 corpus gates plus 6 ADR 0006 gates
(`section_block_without_chunk_reference`, `attachment_block_id_unknown`,
`attachment_without_page_or_bbox`, `block_text_leaked_into_chunk_text`,
`descriptor_chunk_without_attachment`, `descriptor_count_vs_block_count` =
167/167).

Tests: **158 passing** (148 → 158). `chunk/` had no tests at all before this
entry; it now has 10, including the prose-unchanged invariant and the
numeric-header refusal.

**Binding on `ai-service`, stated in ADR 0006 and not implemented here:** a
chunk with `has_quarantined_content` must make the answer say a table or
formula exists at the cited page and surface its crop; a `block_descriptor`
may be answered only with the crop; no chunk carrying a quarantined
attachment may be used to state a numeric dose.

**Still open:** table row/column reconstruction (the opendataloader cell data
is available and matches pdfplumber exactly inside the monograph range);
recall for borderless tables and bar-less formulas; content accuracy against
the source; the general chapters and appendices (9.6% of characters).

## 2026-08-01 (cont'd, 4) — READY TO CHUNK: transcriptions merged back into the text, `cli chunk-ready` gate suite green on all 10 gates, two more real data-loss bugs found and fixed on the way

**The blocker is closed.** The 1,116 transcribed characters are no longer a
file beside the corpus — they are in it. `ingestion/extract/repair.py` splices
each transcribed run back into the span stream geometrically, and every
command that builds monographs now goes through the same repaired stream, so
the ledger and the output describe one pipeline rather than two.

**New gate suite, `cli chunk-ready`** (`ingestion/validation/readiness.py`).
Each invariant gets its own count and its own target — a single verdict would
hide exactly what took this session to find. Run on the whole corpus:

| gate | count | target |
|---|---|---|
| outlined_run_not_merged | 0 | 0 |
| known_corruption_string | 0 | 0 |
| formula_fragment_in_prose | 0 | 0 |
| pua_char | 0 | 0 |
| replacement_char_ufffd | 0 | 0 |
| empty_section | 0 | 0 |
| section_without_provenance | 0 | 0 |
| unflagged_quarantine_block | 0 | 0 |
| duplicate_drug_id | 0 | 0 |
| monograph_without_page_range | 0 | 0 |

Corpus going into chunking: **683 monographs, 11,966 sections, 8,212,712
characters**, plus 167 quarantined table/formula blocks held outside prose.

**Two real bugs surfaced by building the gates, both fixed:**

1. **A 4pt glyph in the column-overlap strip was assigned the wrong column.**
   `classify_column`'s two tolerance bands overlap between x=288 and x=319 and
   left was tested first, so a single `ổ` at x=315 on physical page 714 was
   classified as left-column and could not be matched to its own right-column
   line. `Độ ổn định` stayed `Độ n định` even after the repair ran. Fixed by
   testing exact containment before tolerance. Invisible for a full-width
   block; only a narrow box exposes it.
2. **A plain body line that repeats a section name was read as a heading.**
   FLUOROURACIL (physical page 681), verified by rendering the page, prints
   `Thời kỳ mang thai` / `Chống chỉ định.` and `Thời kỳ cho con bú` /
   `Chống chỉ định.`. Both body lines matched the section vocabulary, so both
   sections came out **empty** and the statement that fluorouracil is
   contraindicated in pregnancy and while breastfeeding was dropped entirely.
   Fixed narrowly: a *non-bold* label directly under a heading is that
   heading's body. Boldness still cannot be required in general (outlier item
   20), hence the position constraint rather than a style rule.

A third placement bug was caught during the merge itself: PyMuPDF emits the
text either side of a dropped glyph as **one span whose box spans the gap**,
so splicing at span boundaries produced `tuở ổi`. `repair.py` now reads
per-character boxes from `rawdict` and splits the containing span at the
character offset the geometry indicates.

**Whole-document re-measurement after all of the above:**

| check | result |
|---|---|
| `cli run` | 683 monographs, 51 runs merged (1,116 chars), 167 blocks lifted / 167 quarantined |
| `cli validate` | 92.9% recall / 99.1% precision — unchanged |
| `cli coverage` | 252,801 spans, **unassigned = 0** |
| `cli chunk-ready` | 10/10 gates pass |
| tests | **148 passing** (145 → 148) |

**What these gates explicitly do NOT prove**, printed by the command itself so
it cannot be quoted out of context: content accuracy against the source (no
whole-document human-reviewed ground truth exists), table row/column
reconstruction, and recall for borderless tables and bar-less formulas.

**Next:** `chunk/` still has no tests and has never been executed. Table
reconstruction from the opendataloader cell data remains available and is not
on the critical path.

## 2026-08-01 (cont'd, 3) — All 23 fraction-bar candidates read by eye (precision 69.6%), 51 outlined runs transcribed, 2D formulas quarantined; prose-leak gate = 0

**All 23 `fraction_bar_candidate` regions were rendered and read.** Verdicts,
one page at a time:

| verdict | count | where |
|---|---|---|
| real 2D formula | **16** | p43, p92 (×5), p202, p325 (×2), p349, p1042, p1043 (×2), p1132, p1402 (×2) |
| not a formula | **7** | p4 (×3 decorative underlines on the Ministry decision page), p63 (ruled box), p845, p878 (table cell borders), p1667 (rule above the colophon) |

**Precision of the candidate rule: 16/23 = 69.6%.** That is why the verified
list is a curated file (`ingestion/data/verified/formula_regions_2d.json`) and
not the detector's raw output — a 70%-precise rule must not quarantine
content on its own. 10 of the 16 are inside the monograph range.

**A formula the detector cannot find, confirmed.** ADENOSIN (physical page
147) prints `Tốc độ truyền dịch (ml/phút) = 0,140 (mg/kg/phút) × trọng lượng
cơ thể (kg) / Nồng độ adenosin (3 mg/ml)` as **three plain lines with no
fraction bar at all** — verified by rendering the region and reading it. No
geometric signal exists to detect it; it surfaced only because a prose-leak
gate matched its text. It is quarantined and flagged, and
`recall_limit` in the verified file records that **the number of bar-less
formulas in the book is UNMEASURED**. The fraction-bar scan must never be
described as complete formula coverage.

**51 outlined runs transcribed** into
`ingestion/data/verified/outlined_text_transcriptions.json` — 22 full lines
plus 29 single glyphs, **1,116 characters** recovered, each with page, bbox,
the run's text and the extracted line it belongs to. Every value there is a
transcription read off a rendered page, labelled as such, never extracted
data.

**The single-glyph runs are the nastier half of that defect.** They are
Vietnamese diacritic characters dropped out of lines that otherwise extract
fine, so the damage is invisible downstream:

| extracted | actual |
|---|---|
| `Độ n định:` | Độ **ổ**n định |
| `≥ 1 tu i` | ≥ 1 tu**ổ**i |
| `Thuốc dùng tại ch :` | tại ch**ỗ** |
| `i nồng độ glucose máu` | (thay đ)**ổ**i nồng độ glucose máu |

**2D formulas are now quarantined in the pipeline.** `SHAPE_FORMULA_2D` was
added to the existing shape taxonomy and to `QUARANTINE_SHAPES` — an entry,
not an edit to matching code. `ingestion/extract/formulas.py` loads the
verified regions and grows each bar into a band covering numerator and
denominator. Whole-book re-run:

| gate | result |
|---|---|
| verified formula regions loaded | 17 on 10 pages |
| blocks lifted out of prose | 169, **169 quarantined** |
| `formula_2d` blocks | 14 |
| `formula_fragment_left_in_prose` | **0** |
| monographs | 683 (unchanged) |
| `cli validate` | 92.9% recall / 99.1% precision (unchanged) |
| tests | **145 passing** (139 → 145) |

The side margin needed two attempts: at 4pt, AMPICILIN VÀ SULBACTAM's
numerator `Thể trọng (kg)` stayed behind in the prose because its span box
carries leading spaces that pull its centre left of the bar. Raised to 95pt
with the reasoning recorded in the module: over-capturing a neighbouring line
into a quarantined block is recoverable, half a formula left in prose is not.

**Still open**: the 1,116 transcribed characters are recorded but **not yet
merged back into the monograph text** — the corpus still contains
`Độ n định`; table row/column reconstruction is untouched (137 simple tables
+ 17 multi-header + 1 continuation remain quarantined); table detection
recall for borderless tables is unmeasured; `chunk/` still has no tests and
has never run.

## 2026-08-01 (cont'd, 2) — Residual-ink check built and run whole-document; found a text-loss class no text-based check could see: 51 runs of type drawn as vector paths on 5 pages

**What was built.** `ingestion/validation/residual_ink.py` (production, plus a
`cli residual-ink` command) renders each page, whites out every pixel covered
by an extracted span, and reports the ink that survives. It needs no ground
truth and no sampling. Measured: **0.06 s/page, all 1668 pages in under two
minutes.** Classification is a pure function over `(region, PageContext)` with
an ordered rule list, so a new kind of residual is a new entry, not an edit.

**Whole-document gate result — all 1668 pages, 3,931 residual regions:**

| kind | regions |
|---|---|
| header_rule | 1,649 |
| text_as_vector_outline | 1,061 |
| table_frame | 959 |
| antialias_speck | 220 |
| fraction_bar_candidate | 31 |
| rule_fragment | 10 |
| header_band_fragment | 1 |
| **unclassified** | **0** |

**The finding: 51 runs of text on 5 pages exist only as vector outlines.**
Physical page 714 (GATIFLOXACIN) prints 17 full lines of ordinary prose that
`page.get_text()` does not return, `page.search_for()` cannot find,
`pdfplumber` does not return and `opendataloader-pdf` does not return.
`page.get_drawings()` shows why: each line is a filled path of 1,126-1,831
items, shaped exactly like one line of type, in the body-text colour. Single
glyphs appear the same way with 39-45 items. Recovery cannot be automatic —
the paths carry no character codes — so `ingestion/extract/outlined_text.py`
detects and reports them for transcription and never guesses.

| physical page | outlined runs |
|---|---|
| 714 | 31 |
| 736 | 16 |
| 1373 | 1 |
| 1444 | 1 |
| 1445 | 2 |

All five are inside the monograph range. Two independent methods agree on the
same five pages: the drawing-shape scan, and counting glyph-shaped leftovers
in the residual mask. Sample of what is missing, read off the rendered page:
`"Nghiên cứu trên động vật, gatifloxacin gây ngộ độc cho thai."` (p714),
`"(Typhoid, inactivated, whole cell), J07AP03 (Typhoid, purified"` (p1445).

**Three instrument bugs were found and fixed before any of the above was
believed** — the measuring device was wrong before the data was, three times:
1. **Horizontal banding merged the two page columns**, so page 209's ADR table
   sat in a box whose centre fell in the gutter and matched no table region.
   Adding a column split then cut single table grids into their individual
   rules. Replaced with 2D connected components (`scipy.ndimage.label`).
2. **A glyph-count ratio was nearly reported as a data-loss measure.** First
   pass gave "extraction ratio 0.6656, 835 pages below 98%". It was wrong:
   `get_texttrace()` counts glyphs painted outside the page rectangle —
   4,717,407 of them, on pages that are visually blank. Clipping to the page
   rect gave 0.8023 and "1642 of 1668 pages below 95%", which was also wrong:
   Vietnamese diacritics are painted as two glyphs and extracted as one
   character, so the deficit is systematic and meaningless. **Neither ratio
   should ever be quoted.** The pixel-based check is the sound one.
3. **Mask padding of 1.0pt ate the fraction bars** it was meant to find.
   Calibrated to 0.5pt against the two known formulas, verified not to add
   noise on a 10-page prose sample.

Incidentally this explains a long-standing note in ADR 0003: `pdfplumber`
"scrambles reading order" on this document because it reads the off-page text
that PyMuPDF correctly clips away.

Tests: **139 passing** (129 → 139), including whole-document regression
fixtures pinning the 51 outlined runs per page and the two fraction-bar
widths (188.6pt on p1042, 118.1pt on p202).

**Not done / next:** the 31 `fraction_bar_candidate` regions on 15 pages have
**not** been looked at yet, so no precision figure for them exists; the 51
outlined runs are detected and flagged but **not transcribed**, so that text
is still absent from the corpus; 2D formulas are still not quarantined in
`segment/`. `unclassified = 0` means every region is *named*, not that every
named verdict has been checked by eye — of the seven kinds, `header_rule`,
`table_frame`, `antialias_speck`, `rule_fragment` and `header_band_fragment`
were confirmed on sampled examples only.

## 2026-08-01 (cont'd) — Two 2D fraction formulas confirmed corrupted in output by reading the source page images; both tools are blind to them, so cross-tool agreement does NOT bound recall

**Finding, visually confirmed on the rendered source, n=2:** stacked-fraction
formulas lose the fraction bar and emit the numerator *before* the `=`, so
the division reads as multiplication.

| drug | physical page | source (read from the page image) | pipeline output |
|---|---|---|---|
| NETILMICIN | 1042 | `Cl_cr (ml/phút) = [(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ)] / [Nồng độ creatinin huyết thanh (micromol/lít) x 0,81]` | `(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ) Clcr (ml/phút) = Nồng độ creatinin huyết thanh (micromol/lít) x 0,81` |
| AMPICILIN VÀ SULBACTAM | 202 | `Cl_cr (ml/phút) = [Thể trọng (kg) x (140 - số tuổi)] / [72 x creatinin huyết thanh (mg/dl)]` | `Thể trọng (kg) x (140 - số tuổi) Clcr (ml/phút) = 72 x  creatinin huyết thanh (mg/dl)` |

Read literally, both now state that clearance is *multiplied* by serum
creatinine. This is a dosing calculation in a renal-impairment section. The
content is **not quarantined and carries no formula flag** — it flows into
`chunk/` as ordinary prose.

**This corrects the weight I put on cross-tool table agreement earlier the
same day.** Measured: on physical page 1042 `pdfplumber.find_tables()`
returns **0** regions and opendataloader returns **0** tables; the same holds
for the formula region on page 202. The two tools agreeing on 112 shared
table pages measures *consistency on what ruling lines make visible*, not
recall — they share the blind spot. Agreement must not be reported as
evidence of coverage.

**Priority consequence:** the 155 table blocks are already `quarantined:
true`, i.e. contained — they cannot poison an answer today. The formulas are
uncontained. Formula handling should therefore come before table
reconstruction, which is the reverse of the plan written earlier today.

**Population sizing, honest limits.** A keyword scan of the output found 185
occurrences of "công thức", of which **93 are "công thức máu/bạch cầu/hồng
cầu"** (blood count, not mathematics) and many of the remaining 92 mean
"formulation" (`thành phần trong công thức`). So keyword counting cannot size
the formula population; only a detector with measured recall can. The two
cases above are the first two regression fixtures.

## 2026-08-01 — Readiness check re-measured from the current artifacts (no code change): text coverage complete, tables quarantined, formulas still unhandled

Question asked: is the data ready to parse 100%, including formulas and
tables? Every number below was recomputed in this session from the files on
disk (`ingestion/data/processed/{monographs.jsonl,coverage_ledger.json}`) and
from a fresh test run — none quoted from earlier entries.

| check | command / scope | result |
|---|---|---|
| unit tests | `python -m pytest -q` (whole `ingestion/`) | **129 passed** |
| monographs / sections | read `monographs.jsonl` | 683 / 11,966 |
| table blocks in output | read `monographs.jsonl` | **155 blocks, 155 quarantined** (simple_table 137, multi_level_or_merged_header 17, cross_page_continuation 1) |
| span coverage ledger | read `coverage_ledger.json`, all pages | 252,733 spans; `unassigned` = **0** |
| ledger states | same | normalized_text 177,754 (8,183,182 ch) / out_of_scope 53,374 (897,692 ch) / heading 12,764 / boilerplate_excluded 4,976 / quarantined 3,862 / structural_excluded 3 |
| page coverage | ledger vs `doc.page_count` | 1666 of 1668 pages carry spans |
| the 2 pages with no spans | rendered physical 99 and 1666 at 110 dpi, read the images | **both genuinely blank** (0 chars, 0 images, only a frame drawing) — not a loss |
| PUA left in output | scan all 11,966 sections | **0** |
| U+FFFD in output | scan all 11,966 sections | **0** — closes the gap flagged in the previous entry as never measured |

Note the block count differs from the previous entry's `148` — this is a
recomputation from the current file, not a correction of a bug; the shape mix
also differs from the 180-region whole-book classification because blocks are
only the regions that fall inside the monograph range.

**Answer: no, not ready for a "100% including formulas and tables" claim.**
What is closed: goal A (full coverage, nothing silently dropped) for the
monograph text path — `unassigned = 0`, both uncovered pages proven blank.
What is open, by name:
- **Formulas: no production stage exists.** `grep -il formula` over
  `ingestion/ingestion/` hits only `chunk/sentences.py` and `cli.py`; all
  formula work lives in `scratch/`. The only detector fired 3,405
  `fraction_bar` hits on 837 of 1668 pages with precision never measured, so
  there is not even a trustworthy formula *count*, let alone reconstruction.
  2D formulas currently linearise into section text unflagged.
- **Tables: detected and quarantined, not reconstructed.** 155/155 blocks are
  `quarantined: true` — provenance kept, unsafe to cite. Borderless tables
  (BSA nomogram, catalog item 7) are invisible to `pdfplumber` by
  construction, so the miss rate is unmeasured and undetected tables still
  contaminate body text.
- **Out-of-scope regions unparsed**: 53,374 spans / 897,692 chars (9.6% of
  ledger chars) — general chapters and appendices — are excluded explicitly
  but have never been structurally parsed.
- **Content accuracy vs. source never measured**; 92.9% / 99.1% is
  boundary detection only, on an uncleaned 1064-entry denominator.
- `chunk/` still has no tests and has never been executed.

## 2026-07-31 (cont'd, 5) — Cleanliness audit before chunking: data is NOT clean; 5 defects measured whole-corpus, incl. ≥/≤ in dosing text lost as PUA glyphs (all 8 PUA codepoints visually confirmed)

**Trigger**: user pushed back on starting the chunk stage ("chưa chunk dữ
liệu phải sạch"), correctly — chunking was about to run against text that
had never been audited for content-level cleanliness. Only boundary
detection had ever been measured, never the text itself.

**Also fixed this session (small)**: `cli.py` crashed with
`UnicodeEncodeError` on Windows cp1258 when printing Vietnamese drug names
in `validate`'s unmatched lists — the metrics printed first so past numbers
were unaffected, but the tail of the report was lost. Added
`sys.stdout/stderr.reconfigure(encoding="utf-8")` in `main()`. Re-ran
`cli validate`: exit 0, Vietnamese renders correctly.

**Timing measured for the first time** (whole 1668-page PDF, PyMuPDF only):
`cli run` = **2m10.6s**, `cli validate` = **44.4s**. Does not cover
pdfplumber/opendataloader/docling cross-checks, which are not part of either
command.

**Boilerplate re-verified independently** against output generated this
session: **0 of 11,409 sections** contain "DTQGVN" (was 1,374), 0 of 682
monographs affected. Also closed the previously-flagged gap of "never
checked with a different signature": scanned for a bare 3-4 digit line
(page number leaking without "DTQGVN" adjacent) — 204 sections matched,
sampled 8, **all legitimate content** (`cytochrom P\n450` split across
lines, dosing values like `250 microgam/kg`), not boilerplate. Scope limit:
8 of 204 inspected, not all.

**Cleanliness audit — whole corpus, 682 monographs / 11,409 sections /
8,241,485 section chars** (`ingestion/scratch/cleanliness_audit.py`,
temporary, to be deleted once this finding is fully captured):

| signal | occurrences | sections hit | % sections |
|---|---|---|---|
| mid-sentence line wrap | 99,501 | 8,197 | 71.8% |
| short fragment lines (<4 chars) | 11,612 | 2,149 | 18.8% |
| bare-number lines | 2,540 | 862 | 7.6% |
| flattened table rows | 25 | 9 | 0.1% |
| PUA chars | 86 | 41 | 0.4% |

**Confirmed: table content IS contaminating section body text.** Real
example — AMPICILIN's `duoc_ly_va_co_che_tac_dung` contains an
antibiotic-resistance table flattened to `'Salmonella typhi\n378\n10,6\n
0,0\n89,4\nShigella flexneri\n120\n41,6...'`, losing all row/column
semantics. The 0.1% figure is only what the all-numeric-row regex catches;
the true table count is pending the inventory scan and will be higher.

**Confirmed, patient-safety relevant: comparison operators in dosing text
are being emitted as raw PUA codepoints.** All 8 distinct PUA codepoints in
the corpus were located in the source PDF, rendered to images, and read
directly (not inferred from context):

| codepoint | count | actual glyph | visual evidence |
|---|---|---|---|
| U+F0B3 | 57 | **≥** | p.141 "trẻ em ≥ 10 tuổi" |
| U+F0A3 | 17 | **≤** | p.169 "liều ≤ 100 mg" |
| U+F061 | 5 | **α** | p.334 "Streptococcus α tan huyết" |
| U+F0AE | 3 | **→** | p.1027 "HCO₃⁻ + H⁺ → H₂CO₃ → CO₂ + H₂O" |
| U+F0D2 | 1 | **®** | p.891 "Plasma Lyte® 56/5%" |
| U+F031 | 1 | **₁** | p.957 "alpha₁-acid glycoprotein" |
| U+F0AF | 1 | **↓** | p.1033 "rhodanese ↓" (catalysis arrow) |
| U+F067 | 1 | **γ** | p.1352 "interferon - γ" |

74 of 86 occurrences are ≥/≤ inside dosing or adverse-effect sentences —
losing the operator changes clinical meaning ("liều ≤ 100 mg" vs "liều 100
mg"). Fonts involved: `SymbolTiger` (7 codepoints) and `Symbol` (1).

**Chunk stage — partially built, then deliberately paused.** Wrote
`ingestion/ingestion/chunk/` (`models.py`, `sentences.py`, `chunker.py`,
`io.py`, `__init__.py`) implementing ADR 0004: `(drug_id, section_key)` unit,
800-token ceiling, sentence-boundary-aware sub-chunking. **Not tested, not
run, and must not run until the cleanliness defects above are fixed** —
chunking dirty text bakes the defects into embeddings. ADR 0004's own
"hard prerequisite" (the boilerplate bug) is satisfied, but this audit found
additional blockers it did not know about.

**Strategy adopted for full-coverage parsing** (written up in
`docs/full-coverage-parsing-plan.md`): separate what is provably clean from
what is not — chunk the clean text, flag-and-exclude untrustworthy tables/
2D formulas with an exact excluded count, and prove nothing was silently
lost via a **character coverage ledger** (every char on all 1668 pages must
land in exactly one bucket: section text / table cell / formula region /
out-of-scope / `unassigned`, with `unassigned` reported as a number plus
page+bbox list). Note the plan explicitly distinguishes goal A (full
coverage, nothing silently dropped — achievable) from goal B (proven 100%
correct — requires manual ground truth for every table/formula, not
achievable in one day).

**Fixes landed after the audit above — new `ingestion/ingestion/normalize/`
stage** (`glyphs.py` = the verified PUA map, `text_flow.py` = geometry-driven
span rejoining). Root cause of defects 1-3 was one line in
`segment/assembler.py`: `body_lines.append(span.text.strip())` made every
*span* its own line, so any visual line the PDF split into multiple spans
(italic run, subscript, symbol font) became multiple lines. Text-level regex
cannot distinguish a mid-word span split from a real line wrap, so the fix
uses geometry instead — PyMuPDF's own `(block, line)` indices identify spans
sharing a visual line, and the horizontal gap (`SPACE_GAP_PT = 1.0`) decides
whether a space belongs. Assembler now collects `Span` objects and joins via
`normalize.join_spans` + `normalize.substitute_pua`.

**Whole-corpus re-measurement after the fix** (same audit script, same scope
— 682 monographs / 11,409 sections):

| signal | before | after |
|---|---|---|
| mid-sentence line wrap | 99,501 | **0** |
| short fragment lines | 11,612 | **7** |
| bare-number lines | 2,540 | **0** |
| flattened table rows (numeric-row regex) | 25 | **0** |
| PUA chars | 86 | **0** |

`cli validate` re-run after the change: **unchanged** at 682 monographs,
92.8% recall, 99.1% precision — normalization does not affect boundary
detection. Tests: **119 passed** (110 before; 9 new in `tests/
test_normalize.py`, covering the real corpus cases — `cytochrom P450`
subscript rejoin, `(feline immunodeficiency virus)` italic rejoin, ≥/≤
restoration in dosing sentences, unmapped-PUA reporting). One existing test
(`test_running_header_boilerplate_stripped_...`) had its expected string
updated: it encoded the old `\n` join for `"...không nhai. Nếu"` + `"uống
viên thuốc..."`, which is exactly the mid-sentence wrap being fixed; its
core assertions (no "DTQGVN", no "1009") are unchanged.

**NOT verified — total section chars dropped 13,224** (8,241,485 →
8,228,261, 0.16%). Reasoning from the code says this is separator characters
only (same-line spans previously contributed a `\n` each, now join directly;
`strip()` only ever removed whitespace and no span is dropped), so
non-whitespace content should be unchanged at 6,552,254 — but **this was
reasoned, not measured**. The character coverage ledger (below) is the
instrument that would actually prove it and has not been run.

**Whole-corpus table/formula inventory completed** (17m17s,
`ingestion/scratch/inventory_tables_formulas.py`, all 1668 pages):
- **200 tables on 152 distinct pages**, 0 page errors. Column distribution:
  3 cols ×78, 2 ×72, 4 ×32, 5 ×10, 1 ×4, 7 ×3, 6 ×1.
- ~~22 header-less-at-top continuation candidates~~ — **this figure was
  wrong and is corrected below**: classifying all 200 regions individually
  showed 17 of those 22 are `not_a_table_full_page` and 3 are
  `not_a_table_degenerate`, leaving **2** real cross-page continuations.
  Cause: the inventory's condition (`starts_near_top AND
  header_textual_cells <= 1`) is satisfied automatically by any full-page
  false-positive region — its bbox starts at y≈0, and its single cell is a
  long text blob rather than a textual header — so every non-table landed
  in the continuation bucket.
- **0 all-numeric wide grids** — but this is a detector limitation, not
  evidence of absence: the known BSA nomogram (item 7) has no ruling lines,
  so `pdfplumber.find_tables()` cannot see it at all.
- **Formula detector over-fires badly and its output must not be quoted**:
  3,405 `fraction_bar` hits across **837 of 1668 pages** (half the book) is
  not credible as a formula count — the thin-horizontal-rect signal is
  evidently matching table rules/underlines/column separators. Precision was
  never measured; this confirms the standing warning that a bbox heuristic
  finds candidates, not formulas. `small_font_numeric` (2,583) is likewise
  unvalidated. Only the PUA count (86) from that scan is trustworthy, and
  only because all 8 codepoints were visually confirmed.

**Section-name spelling variants — a large silent section loss, found and
fixed.** Scanned the whole monograph range for bold heading strings that do
not match the vocabulary, ranked by similarity: **42 distinct near-miss
strings, 542 occurrences**. The dominant one is `"Thông tin qui chế"`
(**469×**) — the book prints "qui" where its own documented template (and
`vocab.py`) says "quy", so `match_section` returned `None` and the section
was never opened. Measured before the fix: only **96 of 682 monographs
(14.1%)** had a `thong_tin_quy_che` section; **586 were missing it entirely**
(the text itself was not lost — it fell into the preceding section's body
unlabelled — but the structure was, so a "thông tin quy chế của X" query
could not retrieve it and citations would name the wrong section).

Two mechanisms were added rather than one long alias list:
- `SectionDef.aliases` for genuinely different wordings ("Mã ACT",
  "Chống chỉ đinh", "Thời kì mang thai", "Hướng dẫn cách sử trí ADR",
  "Quá liều và xử lý", "Dược lí và cơ chế tác dụng", …).
- `_lookup_key()` folds typesetting noise for every entry at once —
  all whitespace removed, case folded, and the U+00D0/U+0110 look-alike
  ("Ðộ" vs "Độ") mapped. This alone absorbs ~14 variants that would
  otherwise each need an alias: "Chỉđịnh", "Chống chỉđịnh", "Độổn định và
  bảo quản", "H ướng dẫn cách xử trí ADR", "Tư ơng kỵ", "Tác dụng
  khôngmong muốn (ADR)", "Thận trọng.", "Liều l ượng và cách dùng", …
- Two near-misses were **deliberately rejected** and recorded in
  `REJECTED_NEAR_MISSES` so a later reader does not add them: "Thể trọng"
  (body weight, 0.84 similar to "Thận trọng"/caution) and "Tác dụng không
  mong muốn của opioid" (a drug-specific sub-heading, not the section).

**Whole-corpus result after the vocabulary fix:**

| | before | after |
|---|---|---|
| monographs | 682 | **683** |
| sections total | 11,409 | **11,966** (+557) |
| `thong_tin_quy_che` present | 96 (14.1%) | **567 (83.0%)** |
| `cli validate` | 92.8% / 99.1% | **92.9% / 99.1%** |
| tests | 119 | **122** |

**Over-joining check (the direction the rejoining work had not tested).**
First attempt used text patterns and had poor precision — sampled examples
were mostly false positives ("Liều lượng có thể tăng…" is ordinary prose,
"Lọ 10, 50, 100 ml" is a volume list, "Wolff - Parkinson - White" is a
hyphenated name), so its counts are not reported here. Redone at the level
where it can actually be judged — the geometry of the two visual lines being
joined — with the same exclusions `assemble()` applies (bold headings and
header-band boilerplate removed, since joins involving those never reach
body text). Monograph range, 180,131 body spans → 154,683 visual lines,
**102,798 joins performed**:

| category | count | % of joins |
|---|---|---|
| clean wrap | 98,611 | **95.9%** |
| indent change | 1,983 | 1.9% |
| vertical gap > 16pt | 514 | 0.5% |
| column change | 634 | 0.6% |
| page change | 567 | 0.6% |
| upward (column/page turn) | 489 | 0.5% |

Vertical gap at join points: median **12.1pt**, p90 12.4pt — a tight
single-leading distribution, i.e. the overwhelming majority are genuine
wraps. **But over-joining is real and it is concentrated in tables**:
physical page 109 shows a dosage-form table being concatenated cell by cell
— `'Viên nén' + '1'`, `'1' + '1 - 4'`, `'1 - 4' + '8 - 12'`, `'8 - 12' +
'Viên nang tác'`, `'18 - 24' + 'Tiêm bắp'`, `'Chưa biết' + 'Tiêm tĩnh'`.
This confirms the risk case predicted before the check was run, and it
settles an ordering question: **table regions must be excluded before
joining, not after.** Not all 1,983 indent-change cases were inspected —
at least one sampled case (`'…(ức chế' + 'alpha-glucosidase).'`) is a
correct wrap with a hanging indent, so that category's precision is
unmeasured.

**Span-level coverage ledger built and run whole-document.** Implemented as
an optional `ledger` argument to `assemble()` plus a `cli coverage` command,
at span level rather than character level (characters cannot balance because
normalization joins and substitutes them). All 1668 pages, 252,733 spans
after merge:

| state | spans | % spans | chars | % chars |
|---|---|---|---|---|
| normalized_text | 181,616 | 71.9% | 8,231,038 | 87.6% |
| out_of_scope | 53,376 | 21.1% | 897,724 | 9.6% |
| heading | 12,764 | 5.1% | 221,266 | 2.4% |
| boilerplate_excluded | 4,976 | 2.0% | 47,609 | 0.5% |
| **unassigned** | **1** | 0.0% | 21 | 0.0% |

The single unassigned span is `"CÁC CHUYÊN LUẬN THUỐC"` on physical page 98
— a part-divider title excluded on purpose via `PART_DIVIDER_TITLES`.

This also **settles the previously-unverified 13,224-character delta**:
`raw_chars_before_merge` = 9,397,658 equals the post-merge total exactly, so
the span-merge step loses no characters; the delta was separator characters
in section assembly, as reasoned earlier but now measured.

**Important limit, learned the hard way in the same session**: the ledger
proves every span was *routed*, not that routed content *survived* into the
output. The section-overwrite bug below was invisible to it — spans were
correctly marked `normalized_text`, then their section was overwritten
downstream.

**Table isolation wired into `assemble()` and gated.** `assemble(spans,
table_index=...)` diverts spans inside a real table region into
`Monograph.tables` (a new `TableBlock` with `table_id`, `shape`,
`physical_page`, `bbox`, `section_key`, `quarantined`). Gate results over
the whole book:

| gate | result |
|---|---|
| `non_table_span_changed` | **0** |
| `table_span_in_normalized_text` | **0** |
| `unintended_duplicate` | **0** |
| `section_emptied` | **0** (was 1 before the overwrite fix) |
| `unassigned` | 1 (the deliberate part divider) |
| lifted blocks | 148, all with unique ids |
| quarantined | **148 / 148** |

Quarantine policy was widened per review: every multi-column shape
(`simple_table`, `multi_level_or_merged_header`, `cross_page_continuation`,
`grid_2d_numeric`) is quarantined until a real row/column reconstruction
exists, because linearised cells are not safe to cite. Only
`single_column_boxed_list` is exempt — one column linearises correctly.

`183 regions loaded but only 148 blocks lifted` is explained, not a loss:
1,676 table spans sit on pages outside the monograph range (e.g. physical
page 42, in the general chapters), where no monograph is open to attach them
to. Those pages are still out of scope entirely.

**Three real bugs found by these gates, all fixed:**
1. **Section overwrite destroyed content in 33 monographs (38 occurrences).**
   A repeated section heading inside one monograph replaced the existing
   `SectionSpan`, discarding everything captured before the repeat.
   CEFAMANDOL's `lieu_luong_va_cach_dung` held only 172 characters of
   flattened renal-dosing table; after the fix it holds **881 characters** of
   real dosing prose ("Cách dùng Thuốc được dùng dưới dạng cefamandol
   nafat…"). Sections are now concatenated, with the first heading kept as
   the provenance anchor. Other affected monographs include CEFAPIRIN NATRI
   and CEFRADIN — also dosing sections.
2. **Duplicate `table_id`.** A region flushed twice emitted two blocks with
   the same id; provenance ids must be unique. Now suffixed (`p339_t0`,
   `p339_t0#1`). Verified: 148 blocks, 148 unique ids.
3. **Table blocks were never written to disk.** `write_monographs_jsonl`
   had no `tables` field, so all 148 lifted blocks were computed, reported
   in the run summary, and then silently dropped at the file boundary. Found
   only because a check script raised `KeyError: 'tables'`. Fixed with a
   round-trip test.

Tests: **129 passing** (122 → 129).

**682 → 683 explained.** A faithful reconstruction of the pre-fix vocabulary
(old `match_section`/`match_section_with_inline_value` patched into the
importing modules, no aliases, no whitespace folding, no Ð/Đ mapping)
reproduces exactly **682**; the current code gives **683**. The difference is
one monograph: **CARBAMAZEPIN**, physical pages 315-319, ATC `N03AF01`, 18
sections, anchor "Carbamazepine.". No monograph disappeared (`GONE` is
empty) and it occurs exactly once, so this is a recovered false negative,
not a duplicate — it is the same `"Carbamazepin, 316"` entry that
`cli validate` had been listing as unmatched ground truth. Two earlier
attempts at this comparison were **invalid** and their numbers (683/683 and
589/683) should be ignored: the first left aliases in `_PREFIX_CANDIDATES`
and kept the new `_lookup_key`, the second built old-style lookup keys but
still queried them through the new whitespace-stripping key function.

**Not done yet / next up:**
  **Design revised** (per review feedback, and it is the better design):
  make it a **span/fragment-level ledger** first and aggregate characters
  from it, because normalization joins, substitutes and drops characters so
  a pure character count cannot balance. States: `normalized_text`, `table`,
  `formula`, `boilerplate_excluded`, `out_of_scope`, `quarantined`,
  `transformed_with_mapping`, `unassigned`.
- PUA reporting should be stated as `known_mapped` / `unknown_pua` /
  `replacement_char_U+FFFD` counts; only `pua_chars = 0` has been measured,
  `U+FFFD` has never been checked.
**All 200 table regions classified individually, then the "not a table"
verdicts checked by rendering every one of them and reading it.** This is
recorded in full because the first two counts reported in this area were
both wrong, and both were wrong the same way — stated from metadata before
anything was looked at:

1. "22 header-less-at-top continuation candidates" — wrong, see the
   correction above; the real figure is 4.
2. "22 of 200 are not tables" — asserted from rules (area ratio ≥ 0.75,
   `n_rows <= 1 or n_cols <= 1`) without opening a single page.

Rendering all 22 and reading them showed **20 correct, 2 wrong**:
- Correct (not tables): p1 copyright page; p3, p5, p1529 blank pages; p7
  table of contents; p9, p10 committee member lists; p12 Vietnamese/English
  drug-name list; p1665 back index; p55 ×3 epilepsy classification lists;
  and p172, p196, p382, p760, p944, p1034, p1230, p1336 — **ordinary
  two-column monograph prose** that `pdfplumber.find_tables()` reports as
  one page-sized table.
- **Wrong**: p62 and p72 are 1×3 regions with visible cell rules — real
  **orphaned continuation rows** of tables broken across a page
  (outlier-catalog item 5). The `n_rows <= 1` rule discarded precisely the
  case where losing content hurts most, since a row without its header
  cannot be interpreted at all.

`classify.py` now treats only `n_cols <= 1` as degenerate and routes a
single row with several columns to `cross_page_continuation`. Corrected
whole-set result:

| shape | count |
|---|---|
| simple_table | 154 |
| multi_level_or_merged_header | 22 |
| not_a_table_full_page | 17 |
| cross_page_continuation | 4 |
| not_a_table_degenerate | 3 |
| **real tables** | **180** |
| **not tables** | **20** |

**Verification scope, explicitly**: all 20 non-table verdicts were confirmed
visually, one page at a time. The 180 real tables' individual shapes
(simple vs multi-level header vs continuation) are **rule-derived only and
have not been checked by eye** — that classification must not be reported as
verified.

**Is "200 tables" trustworthy? Partly — and the limits matter.**
- **No truncation**: 200 records across 152 distinct pages (max 5 on one
  page, spanning physical pages 1-1665). Re-running `find_tables()` over
  just those 152 pages reproduces exactly 200. The round number is a
  coincidence, not a cap. **But this is a reproducibility check with the
  same tool and settings, not independent validation.**
- **Detection recall, measured against the book's own captions**: 33 pages
  carry a `"Bảng N"` caption; 32 of them have a detected table → **97% on
  the captioned subset**. 102 detected-table pages carry no caption, which
  is expected (most tables here are unnumbered). **This measures recall only
  on captioned tables** — borderless tables are invisible to `pdfplumber`
  by construction (the BSA nomogram, outlier item 7, is the known example),
  so the true total is ≥180 and the miss rate for unruled tables is
  **unmeasured**.
- The single captioned miss is physical page 55, captioned `"Bảng 2: Phân
  loại quốc tế các cơn động kinh (1989)"`. Rendering it showed the
  classifier's *structural* verdict was right (one column) but the label
  `not_a_table_degenerate` was semantically wrong — the book numbers it as a
  table, and it is a nested numbered list drawn inside a ruled frame. The
  shape was renamed `single_column_boxed_list` and is counted as a real
  region: single-column content linearises correctly, so it belongs in the
  text, unlike a 2D table. Naming it "not a table" risked a later reader
  discarding it.

**New `ingestion/ingestion/tables/` stage** (`models.py`, `classify.py`,
`detect.py`, `io.py`): table-region detection is production code, not a
scratch script, even though its output is cached (detection takes ≈17
minutes). `pdfplumber` is confined to this module — ADR 0003 established it
must never be used for text on this document. Not yet wired into
`assemble()`; spans inside table regions are still flowing into section body
text.
- Table handling: 200 tables are known but nothing consumes them yet; they
  still flow into section body text as flattened cells (the numeric-row
  regex now reads 0 because rejoining changed the line shape the regex keyed
  on — **that 0 does not mean tables stopped contaminating body text**, and
  claiming otherwise would be wrong).
- Formula detector needs a real precision/recall measurement against a
  golden set before any of its counts are usable.
- Whole-corpus table/formula inventory (`ingestion/scratch/
  inventory_tables_formulas.py`) was still running when this entry was
  written — no counts available yet; `docs/full-coverage-parsing-plan.md`
  has `[chờ đo]` placeholders that must be filled from a real run.
- `chunk/` has no tests yet and has never been executed.
- **Ground truth is not cleaned**: `cli validate`'s 1064-entry denominator
  includes repeated cross-reference index lines (e.g. `"- CoA reductase,
  285"` appears 10+ times in the unmatched list). ADR 0003 used a 725
  denominator, so 91.7% and 92.8% are **not directly comparable**. Neither
  number should be quoted as settled until the ground truth is cleaned.
- Text content accuracy vs. source has still never been measured; the
  recall/precision figures measure monograph-boundary detection only.

---

## 2026-07-31 (cont'd, 4) — Follow-up on the character-diff's remaining unexplained low-similarity pages: sampled 6, all benign/already-known, none newly investigated pipeline bugs

**Scope**: of the ~30-50 pages below 0.95-0.98 similarity left unexplained
by the reversed-column-order investigation (2 entries below), sampled 6 —
1498, 309, 382, 699, 1420, 1369 — chosen to cover the two visible clusters
(1498-1529 near the back-index transition; scattered monograph-range pages)
rather than just the very lowest scores.

**Findings, all benign, none a new production-pipeline bug:**
- **1498, 699**: table/formula content — `opendataloader-pdf` restructures
  it into markdown tables/headings, PyMuPDF's plain text flattens it; same
  underlying content, different presentation. Matches the already-documented
  "no table reconstruction implemented yet" gap (outlier catalog items 7-8),
  not a new finding.
- **309**: the two tools attribute *different* dosing tables to this page
  (PyMuPDF: "Bảng 4" single-agent; opendataloader: "Bảng 3"
  capecitabin+docetaxel combination) — a table-boundary/page-attribution
  disagreement between the two tools, same known gap as above.
- **382, 1420**: the two tools' plain-text page-content genuinely differs
  (different sections of the same drug appear to land on "this page" per
  each tool). **Directly checked against the actual production pathway**
  (`extract_spans()`, dict-mode, already column-sorted) rather than trusting
  the plain-text diff alone: production output for both pages matches
  PyMuPDF's own plain text exactly — the disagreement is opendataloader-pdf
  choosing a different page-boundary cut for overflow text, not a defect in
  this project's pipeline.
- **3, 5, 97**: near/fully blank pages (10-27 chars on one side, 0 on the
  other) — low information content makes the similarity ratio noisy at
  this scale regardless of correctness, not evidence of a real problem.

**Honest scope limit**: only 6 of the ~30-50 unexplained pages were sampled.
All 6 turned out benign or already-documented, which is reassuring but is
not the same claim as "all remaining pages are benign" — that would need
the full set checked, which this session did not do. Investigation scratch
files deleted per CLAUDE.md now that this finding is captured here.

---

## 2026-07-31 (cont'd, 3) — Fixed the boilerplate-leakage bug flagged by the parallel chunking-design session; independently re-verified their numbers before touching any code

**Context**: the parallel session below (ADR 0004 / chunking design) found
and measured a real bug but deliberately left the fix to this session to
avoid a same-file collision. Before writing any fix, independently
reproduced their exact numbers from scratch (not trusted on read) — matched
exactly: 682 monographs, 11,409 sections, 1,374 sections (12.0%) containing
a literal "DTQGVN" string, 671 monographs (98.4%) affected, and the exact
MORPHIN SULFAT `liều lượng và cách dùng` text they quoted. This is the same
discipline applied earlier this session to a mid-session Riboflavin listing
error found in this file — re-verify a reported finding directly against
real data before building on it, even when it looks correct.

**Root cause, confirmed**: `extract/spans.py` already tags the running
header ("DTQGVN 2" + page number + repeated monograph name) as
`column="full_width"`, but nothing in `segment/assembler.py`'s
classification pass excluded it — it matched no section heading and isn't
a real all-caps title, so it fell through into plain body text, landing
mid-sentence whenever a section's text crosses a physical page boundary.
This is exactly outlier-catalog item 13's already-documented risk
("strip the fixed boilerplate before parsing content"), which had a
warning but no enforcing code or test until now — added as item 22 in the
catalog (item 23 also added for the reversed-column bug from the entry
below, which hadn't been given a catalog number yet either).

**Fixed**: new `assembler._is_page_boilerplate(span)` — drops any span with
`column == "full_width"` and `y0 < HEADER_BAND_Y` (same header-band
threshold `page_map.py` already uses for folio detection; exported that
constant as public rather than duplicating the magic number) before any
other classification. Regression test added using the real MORPHIN SULFAT
span shape (`tests/test_segment_assembler.py`).

**Whole-corpus re-measurement after the fix**: 0 of 11,409 sections contain
"DTQGVN" (was 1,374). `cli validate` unchanged: 682 monographs, 92.8%
recall, 99.1% precision — the fix only touches body-text content, not
monograph/section boundaries. 110 tests total (was 109), all passing.

**Not done yet / next up:**
- Chunking (ADR 0004, the parallel session's design) can now safely run
  against real ingestion output for this specific defect — but see the
  entry below's own "not done yet" list (sub-chunk splitter not built,
  general-chapters/appendices scope, sub-compound tagging) for what's still
  actually blocking Phase 2 beyond this fix.
- Only checked for the literal "DTQGVN" substring as this bug's signature
  — did not separately verify whether the page-number token alone (without
  "DTQGVN" adjacent) ever leaks in some other layout shape; the fix itself
  is structural (column+y-position, not text-pattern-based) so it should
  cover that too, but this wasn't independently re-measured after the fix
  with a different detection signature.

---

## 2026-07-31 (cont'd, parallel session) — Phase 2 chunking strategy designed (ADR 0004) from real per-section measurements; found and flagged a new whole-corpus boilerplate-leakage bug for the extract/segment session to pick up

**Context**: this entry comes from a second session running in parallel with
the one still fixing `extract`/`segment` parsing bugs, on the same checkout
(no worktree separation). Per explicit scoping agreed with the user, this
session touched **only** `docs/adr/0004-chunking-strategy.md` (new),
`docs/architecture.md`'s chunking paragraph, this log entry, and a
since-deleted scratch script — it did not touch `extract/*.py`,
`segment/*.py`, or `docs/document-profile.md`, to avoid colliding with the
other session's in-flight edits to those files.

**Done:**
- Ran `python -m ingestion.cli run` for real (full 1668-page PDF) to produce
  `ingestion/data/processed/monographs.jsonl` (682 monographs — gitignored
  output, matches the count already reported elsewhere in this log), then
  measured real per-section text-length distribution across the whole
  corpus for the first time (`ingestion/scratch/chunking_stats_survey.py`,
  now deleted per this project's investigation-script rule, findings
  captured below and in the ADR).
- **Replaced the never-validated chunking guess in `docs/architecture.md`**
  (`(drug, section)` unit, ~500-800 tokens, 400-tok/50-overlap sliding
  window — written before segmentation existed) with a design grounded in
  the real measurement: `(drug_id, section_key)` chunk unit confirmed;
  800-token ceiling (chars/4 estimate) confirmed as directionally right
  (clears ~16/18 section types at p90); **but sub-chunking is the routine
  path, not a rare hedge, for 2 specific sections** — `dược lý và cơ chế
  tác dụng` (242/678 monographs with that section, 35.7%, max ≈3542 est.
  tokens) and `liều lượng và cách dùng` (200/675, 29.6%, max ≈3631 est.
  tokens); a smaller tail also exceeds it (`thận trọng` 3.7%, `tương tác
  thuốc` 3.4%). Chosen sub-chunking method: **sentence-boundary-aware**
  sliding window (~600-700 tok/sub-chunk, ~50-80 tok overlap), not a blind
  character/line window — `assembler.py`'s `body_lines` join one PDF
  visual line-wrap per line, not a semantic boundary, so a blind window
  risks splitting a dosing sentence mid-way (a real, measured risk given
  outlier item 17: adult/child dosing splits appear on 1,121/~1,400
  monograph-range pages). Full rationale, extended chunk metadata schema
  (`chunk_id`, `atc_codes`, `part_index`/`part_count`, etc.), and 4
  explicitly-flagged open gaps (sub-compound tagging inside class-level
  monographs, sub-chunk page-precision, the splitter itself not yet built,
  general-chapters/appendices chunking out of scope) are in
  `docs/adr/0004-chunking-strategy.md`.
- **Found and measured a new whole-corpus bug, not yet fixed, flagged here
  for the `extract`/`segment` session rather than fixed directly** (per
  user's explicit choice this session, to avoid a same-file collision):
  running header/footer boilerplate ("DTQGVN 2" + page number + repeated
  drug name — tagged `column="full_width"` in `extract/spans.py`) is never
  filtered out of section body text; `assembler.py` appends every
  non-title, non-section-heading span to `body_lines` regardless of column
  tag. Measured whole-corpus: **1,374 of 11,409 sections (12.0%) contain a
  literal "DTQGVN" string mid-text; 671 of 682 monographs (98.4%) have at
  least one affected section.** Real example: MORPHIN SULFAT's `liều lượng
  và cách dùng` reads `"...Nếu\nDTQGVN 2\n1009\nMorphin sulfat\nuống viên
  thuốc..."` — the page number and drug name are spliced mid-sentence into
  a real dosing instruction. This is `docs/pdf-parsing-outlier-catalog.md`
  item 13's already-documented risk ("header/footer boilerplate must be
  stripped"), just never actually measured/fixed until this session — it
  should become a new numbered item in that catalog (item 22, or the next
  free number by the time this is read — check the catalog directly) with
  these numbers, but that file is mid-edit in the other session so this
  entry leaves the actual catalog edit to them rather than risking a
  concurrent-write collision. Note: this bug is **separate from** the
  reversed-column-order bug documented in the entry directly below this
  one — that bug was about which *column* content lands in, this one is
  about full-width header-band content never being excluded from body text
  regardless of column. **This is a hard blocker for Phase 2**: chunking
  must not run against real ingestion data until this is fixed, or
  boilerplate gets baked into embeddings and can surface mid-sentence in a
  chunk shown to a doctor/pharmacist.

**Not done yet / next up:**
- The boilerplate-leakage bug above needs a real fix in `extract`/`segment`
  (likely: exclude `column="full_width"` spans from body-text assembly,
  or an explicit boilerplate-pattern filter) plus a regression test and a
  whole-corpus re-measurement to confirm it's actually gone — not done by
  this session, left for whoever owns `extract`/`segment` next.
- `ingestion/ingestion/chunk/` still doesn't exist — ADR 0004 is a design
  only; implementing and unit-testing the sentence-boundary splitter is a
  separate task.
- Chunking design for general chapters (pp. 37-98) and appendices (pp.
  1497-1528) is still blocked on `docs/document-profile.md`'s Group 2
  investigation (tables, 2D stacked-fraction formulas) completing.
- Sub-compound tagging inside class-level/multi-ATC monographs (25.5% of
  corpus) has no design yet — flagged in ADR 0004, deferred to
  golden-dataset-driven eval.

---

## 2026-07-31 (cont'd, 2) — Built a whole-document cross-tool character-diff QA check; it found a real, serious cross-monograph data-corruption bug (reversed column reading order), now fixed and whole-corpus-reverified at zero occurrences

**Why this check was built:** after the ATC-field bug-fixing session below, the
user asked what validation step would catch whether parsing is "correct" at
all — not just "does `cli validate` say recall/precision are high," since
that check only confirms a monograph *exists* at roughly the right name/page,
not that its *content* is complete and correctly attributed. Per
[[feedback-rigorous-validation]], comparing PyMuPDF's own output against
itself can't validate itself — a second, independently-implemented parser
is required as real ground truth. Built a whole-document (all 1668 pages)
per-page character-similarity diff: PyMuPDF's `page.get_text()` vs
`opendataloader-pdf`'s markdown extraction, normalized and compared with
`difflib.SequenceMatcher`.

**Two bugs in the check script itself, found and fixed before trusting any
result (disclosed to the user immediately on discovery, not after):**
1. Wrong page-separator placeholder syntax (`{page}` instead of the tool's
   real `%page-number%`) risked silent page misalignment. Fixed by using the
   real placeholder and parsing the actual page number from each separator
   instead of assuming positional order.
2. Python's `difflib.SequenceMatcher` default `autojunk=True` collapsed the
   similarity ratio to ~0.0065 for a page whose content was actually ~98%
   identical between tools (a long drug-name list trips its "popular
   element" heuristic) — a well-known stdlib gotcha. Fixed with
   `autojunk=False`.

**Whole-document result** (1668/1668 pages compared, mean 0.9892, median
0.9981): a tight cluster of pages — 929, 1099-1106, 1149-1153 — scored only
~0.47-0.53. Investigated instead of dismissed.

**Confirmed real, serious bug in `extract/spans.py`:** the module trusted
PyMuPDF's raw block iteration order to already sequence left-column-before-
right-column, validated only against one example page back in ADR 0003.
Wrong on **12 of 1398 monograph-range pages** (whole-range scan, e.g.
physical page 1100): PyMuPDF's raw block order emits the *right* column
before the *left* column there. Confirmed by rendering the page to an image
and reading it directly, then confirmed in the actual `assemble()` output:
OXYMETAZOLIN's right-column sections (Chống chỉ định, Thận trọng, Thời kỳ
mang thai, Thời kỳ cho con bú, ADR, Hướng dẫn xử trí ADR, Liều lượng và
cách dùng) were being silently attributed to and overwriting the still-open
OXYBUTYNIN monograph's own sections, while OXYMETAZOLIN ended up missing
all 7. Confirmed boundary pairs affected: OXYBUTYNIN/OXYMETAZOLIN,
OXYTETRACYCLIN/OXYTOCIN, OXYTOCIN/PACLITAXEL, PIOGLITAZON/PIPECURONIUM
BROMID; MAGNESI SULFAT, PILOCARPIN, and PACLITAXEL had internal (not
necessarily cross-monograph) ordering corruption. **This is a real,
medical-content-relevant defect** — wrong contraindication/ADR data
silently attached to the wrong drug — not a cosmetic parsing issue.

**Fixed** by explicitly sorting blocks (full_width header band first, then
left column, then right column, each by y-position) instead of trusting
PyMuPDF's raw order. Verified: re-scanned the full 99-1496 range for the
same reversed-order signature — 0 occurrences (was 12). Directly verified
OXYBUTYNIN's and OXYMETAZOLIN's `assemble()`-produced sections are now
distinct and drug-appropriate (spot-checked against the rendered page).
Whole-book `cli validate` after the fix: unchanged at 682 monographs,
92.8% recall, 99.1% precision, 8 zero-ATC (no regression). Also tried a
broader "any within-column y-order violation" scan (670 pages flagged) but
verified a sample and found it's dominated by benign subscript/superscript
baseline noise (e.g. "B" + subscript "6" + ")"), not real bugs — correctly
discarded as evidence rather than reported as 670 new findings.

**Regression test** added (`tests/test_extract_spans.py`) using the exact
real bounding boxes from physical page 1100's raw block order. 109 tests
total (was 103), all passing.

**Not done yet / next up:**
- The whole-document character-diff tooling itself was investigation-only
  (per CLAUDE.md, deleted from `ingestion/scratch/` after this finding was
  captured here + in the regression test + in `spans.py`'s docstring) — if
  this kind of check is wanted as a recurring QA step, it needs to be
  rebuilt as a real `ingestion/validation/` module, not re-derived ad hoc
  each time.
- The character-diff still has ~30-50 pages below a 0.95-0.98 similarity
  threshold that were *not* individually investigated this session (only
  the most extreme cluster was) — front-matter table-like pages (14-31),
  the back-index transition region (1498-1529), and scattered others
  (382, 1420, 57, 68, 309, 194, ...) remain unexplained; could be genuine
  table/formatting differences neither tool handles perfectly, not
  necessarily more instances of this same bug (the specific reversed-column
  signature was already whole-range-scanned to exhaustion above).
- Phase 1.5 (golden dataset) still requires human review by design.
- Phase 2 (chunking) has no code yet and no design decision made.

---

**Correction to the previous entry below, per CLAUDE.md's "never fabricate"
rule:** re-running `assemble()` fresh at the start of this session (same
code, nothing had changed on disk) produced **676** monographs and **48**
zero-ATC-not-stated-absent, not the "680 / 46" the previous entry claimed —
and that entry also self-contradicted (46 in one line, 42 two paragraphs
later). Root cause: the previous session's final numbers were asserted
without a fresh re-run after the very last code edit. No `monographs.jsonl`
artifact existed to diff against, so this can't be proven beyond doubt, but
it's the only explanation consistent with the evidence. Lesson applied
going forward: a number is only "final" if it comes from a command run
*after* the last related edit, in the same message reporting it.

**Method used this session, per two user corrections mid-session**: initial
passes relied only on PyMuPDF span text and coordinate reasoning. The user
first pointed out other installed PDF tools were going unused and that
pages should be rendered to images and read directly rather than trusted
from span dumps alone (per [[feedback-visual-verification]]) — so a first
cross-check used `pdfplumber.extract_text()` plus rendered-page-image
reads. The user then flagged this as still not matching "the strategy from
before." That strategy already existed, in full, in the
[[pdf-parsing-strategy]] memory and `docs/adr/0003-pdf-parsing-strategy.md`:
**4 tools were already evaluated there** (PyMuPDF, pdfplumber,
opendataloader-pdf, docling), and it already concluded
**`pdfplumber.extract_text()` scrambles reading order on this document's
two-column layout and must never be used for general text** — only
PyMuPDF (primary) and `opendataloader-pdf` (independent reading-order +
font-metadata cross-check) are validated for that purpose. The
`MEMORY.md` index line for that memory doesn't carry this detail, only the
full memory file does — this session used the one-line index and never
opened the full file before picking a cross-check tool, which is the actual
process gap (not a memory-setup gap). All findings below were then
re-verified with `opendataloader-pdf` instead, and the earlier pdfplumber
pass was discarded as unreliable evidence, not cited.

**Investigated and closed** (whole-book `cli validate` against the real
back-of-book index, not a sample):
- The 8 detected monographs that didn't match any back-index entry: 2 were
  a real bug in `validation/metrics.py` (substring name-matching let a
  shorter monograph name, e.g. "ISOSORBID", "steal" the ground-truth match
  meant for a longer, textually-overlapping but genuinely distinct
  monograph, e.g. "ISOSORBID DINITRAT" — both are real, correctly segmented
  drugs). Fixed: try an exact normalized-name match before falling back to
  substring. The other 6 are real book-internal inconsistencies, not
  pipeline bugs (compound names containing " - " skipped by the
  already-documented cross-reference filter; title-vs-index spelling
  variants like "HYDROGEN PEROXID" vs the index's "Hydrogen peroxyd").
- The 83 unmatched ground-truth entries: ~40 are back-index line-wrap
  parsing artifacts ("- CoA reductase" / "gonadotropin" fragments from
  wrapped cross-reference lines, not real entries), ~20 are front-matter/
  general-chapter TOC entries (pages 39-98, before the monograph range even
  starts at printed page 99) that `back_index.py` doesn't filter out, a
  handful are the same title-vs-index spelling-variant pattern as above —
  and **7 were genuinely missing monographs**, root-caused to 2 real bugs
  (see below) plus one real book typo (CARBAMAZEPIN's own printed heading
  reads "Ten chung quốc tế", missing the "ê" — confirmed independently by
  both a rendered-page-image read and `opendataloader-pdf`'s text output,
  which shows the same missing "ê"; not fixable without risking false
  positives elsewhere, left as-is).

**4 real bugs found and fixed, each confirmed via a whole-corpus scope
check (not just the sample that surfaced it) and, where the defect could be
page-rendering vs data, a rendered-page-image visual check:**
1. **Same-line diacritic span-fragmentation** (`segment/merge.py`,
   `merge_same_line_bold_fragments`, new): PyMuPDF splits some bold spans
   into multiple fragments around diacritic characters even when the text
   is one unbroken visual line — confirmed by rendering physical page 759
   to an image ("Tên chung quốc tế" looks completely normal to a human
   reader). Cross-checked against `opendataloader-pdf` (the tool
   [[pdf-parsing-strategy]]/ADR 0003 already validated for this — not
   pdfplumber, which that ADR found scrambles reading order on this
   document's two-column layout) on 2 of the 5 affected pages (759
   GUAIFENESIN, 943 MEPHENESIN): both reconstruct the line cleanly, e.g.
   "Tên chung quốc tế: Mephenesin. Mã ATC: M03BX06." with no fragmentation,
   confirming this is a PyMuPDF span-boundary artifact, not a defect in the
   PDF itself. **Correction**: an earlier version of this entry claimed all
   6 candidate pages were cross-checked and listed RIBOFLAVIN among them —
   both wrong. Only 2 of the 5 real pages were actually re-verified with
   opendataloader-pdf just now, and RIBOFLAVIN's failure is the separate
   folio-subscript bug below, not this one — it was never part of the
   diacritic-fragmentation set. Broke the anchor check that gates
   false-positive title filtering, silently dropping whole monographs.
   Confirmed for 5 real monographs (GUAIFENESIN, MEPHENESIN, NATRI
   THIOSULFAT, RAMIPRIL, TENOXICAM) via a full 1668-page scan for the
   fragment signature; the other 3 (NATRI THIOSULFAT, RAMIPRIL, TENOXICAM)
   were not independently cross-tool-verified, only confirmed via PyMuPDF's
   own span coordinates (same-line y-gap).
2. **Folio-detection false conflict** (`extract/page_map.py`, `pick_folio`):
   RIBOFLAVIN's monograph sits high enough on physical page 1243 that its
   own "2" subscript (from "Vitamin B₂", font size 5.83) falls inside the
   header band alongside the real folio "1244" (size 10.0), producing two
   conflicting digit candidates and silently dropping the printed page —
   and the whole monograph with it. Fixed by preferring the largest-font-
   size candidate(s) (a real folio is always set in the header's own
   running size, never a subscript's reduced size); a full-document scan
   confirmed this exact conflict shape occurs on exactly 1 of 1668 pages.
   Confirmed visually by rendering the page.
3. **ATC comma-inside-annotation** (`segment/atc.py`): the field-text
   split on "," ran *before* parenthetical annotations were stripped, so
   an annotation containing its own comma broke the split — e.g. "Mã ATC:
   J07BD01 (Measles, live attenuated)." split into two unrecoverable
   fragments. INSULIN's earlier-fixed Vietnamese annotations ("người",
   "bò") never contain a comma, so this only surfaced with vaccines'
   English annotations — affected 12 vaccine monographs. Fixed by stripping
   *all* parenthetical groups before splitting, not just a trailing one
   per already-split segment. `opendataloader-pdf` cross-check on the real
   VẮC XIN SỞI page confirms the source text genuinely is "Mã ATC: J07BD01
   (Measles, live attenuated)." — the bug was in parsing, not the data.
4. **ATC leading colon from the value span** (`segment/atc.py`): some
   monographs render the bold label as "Mã ATC" (no colon) with the colon
   on the plain *value* span instead (": M03AA01." vs Abacavir's "J05AF06."
   with the colon on the label side) — the section still matched correctly,
   but the leftover leading colon made the stripped candidate 8 characters
   instead of 7, failing the length check. Fixed by stripping a leading
   colon in `normalize_atc_candidate`, symmetric with the existing trailing
   strip. Affected 15 monographs. `opendataloader-pdf` cross-check on the
   real ALCURONIUM CLORID page confirms clean source text ("Mã ATC:
   M03AA01."), same conclusion.

5. **ATC name-prefixed and reversed "CODE: Name" shapes** (`segment/atc.py`,
   same session, found continuing the zero-ATC investigation after the
   above): two more real shapes surfaced once the first 4 fixes cleared the
   noise. (a) 7 monographs with multiple salt/ester forms write each form
   as "Name: CODE" per line, e.g. ARGININ's "Arginin glutamat: A05BA01\n
   Arginin hydroclorid: B05XB01" — the whole segment including the name was
   compared against the 7-char code shape and rejected. (b) The class-level
   "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" monograph writes it the *opposite*
   way, code first — "C10A A01: Simvastatin\nC10A A02: Lovastatin\n...".
   Fixed both with one change: `normalize_atc_candidate` now tries the text
   after the last ":" first, then before, returning whichever side actually
   normalizes to a valid ATC shape — safe because a real drug name never
   happens to match the strict `[A-Z]\d{2}[A-Z]{2}\d{2}` pattern, so there's
   no real ambiguity between the two candidates in practice.

**Net effect, whole-book, before -> after all 5 fixes:**
detected monographs 676 -> **682**; recall 92.2% -> 92.8% (981 -> 987 /
1064); precision 98.8% -> **99.1%**; zero-ATC-not-stated-absent 48 -> **8**.
103 tests total (was 88 at the start of this entry), all passing, each new
fix with a regression test built from the exact real-corpus text that
exposed it.

**The remaining 8 zero-ATC monographs are now all explained, none left
unresolved:**
- 7 (CROTAMITON, INTRALIPID, ISOSORBID, OXYBENZON, PEMIROLAST, SIMETICON,
  the DPT vaccine) have **no "Mã ATC" section anywhere in the book at
  all** — confirmed by reading the actual span sequence after each title
  (goes straight from "Tên chung quốc tế"/"Loại thuốc" to the next section,
  no ATC line ever appears) and by rendering physical page 845 (ISOSORBID)
  to an image and reading it directly. A real, accepted data gap in the
  source — not a parsing bug.
- 1 (SPECTINOMYCIN) is a confirmed real book typo: its own printed heading
  reads **"Mã ACT:"** (letters transposed), not "Mã ATC:" — confirmed by
  rendering physical page 1297 to an image and reading it directly. Same
  category as CARBAMAZEPIN's "Ten chung quốc tế" typo from fix 1 above:
  a real defect in the source document, left unfixed rather than loosening
  vocabulary matching and risking new false positives elsewhere (the
  project's own prior "whack-a-mole" experience with over-loosened
  matching, per outlier-catalog item 21).

**Not done yet / next up:**
- `validation/back_index.py`'s line-wrap and front-matter-entry issues
  (from the investigation above) inflate the "unmatched ground truth"
  count but were left unfixed this session — the user's stated priority
  was the segmentation-pipeline bugs first, not the validation-metric's
  own accuracy.
- The docs/pdf-parsing-outlier-catalog.md items for these 5 new bugs have
  not been added yet (the module docstrings for `merge.py`, `page_map.py`,
  and `atc.py` carry the full evidence in the meantime).
- Only 2 of the ~7 diacritic-fragmentation pages and 2 of the ~15
  leading-colon pages were independently cross-tool-verified with
  opendataloader-pdf (see fix 1's correction note above) — the rest rely on
  PyMuPDF's own span coordinates only, which is weaker evidence.
- No exploration yet of whether the same fragmentation/folio/colon bug
  families affect *other* sections beyond "Tên chung quốc tế" and "Mã
  ATC" (e.g. "Chỉ định", "Liều lượng và cách dùng") — only ATC was swept
  whole-corpus this session.
- Phase 1.5 (golden dataset) still requires human review by design.
- Phase 2 (chunking) has no code yet (`ingestion/chunk/` doesn't exist) and
  no design decision has been made on chunking strategy.

---

## 2026-07-31 — Phase 1.3-1.4 built: assembler, CLI, and validation, with 4 more real bugs found and fixed via whole-book runs

**Done (continuation of the same session, user asked to keep driving
autonomously via `/loop`; visual PDF-page rendering used throughout to
self-verify bugs, per [[feedback-visual-verification]]):**
- Built `assembler.py` (3-pass design: classify spans -> coalesce titles ->
  build Monograph records), `segment/io.py` (JSONL read/write), `cli.py`
  (`run` and `validate` subcommands working end-to-end), and
  `validation/back_index.py` + `metrics.py` (recall/precision against the
  real back-of-book index, parsed from real physical pages 1530+).
- **Found and fixed 4 more real bugs via whole-book `assemble()` runs**,
  each initially surfaced as a wrong number (never trusted the first
  result, per CLAUDE.md):
  1. **ATC trailing-period bug**: "Mã ATC: J05AF06." — the sentence-ending
     period was counted as part of the code, so `normalize_atc_candidate`
     silently returned zero codes for every single-ATC monograph ending in
     "." (a huge fraction of the corpus). Fixed by stripping trailing
     `.,;` before the length check.
  2. **ATC species-annotation bug**: INSULIN's real field lists all 20
     codes each with a parenthetical annotation ("A10AB01 (người); ...") —
     only 2 of 20 survived before the fix (the two that happened to have a
     line-wrap between code and annotation). Fixed by stripping a trailing
     `(...)` group before normalizing. Whole-corpus multi-ATC re-count with
     both fixes: **159/680 (23.4%)** monographs have >1 ATC code (the
     open item from the very first survey session, now closed with a real
     measured number instead of the 25.4%-floor estimate).
  3. **Non-bold combined section heading (outlier item 20)**: AMITRIPTYLIN's
     "Mã ATC:" is a single **non-bold** span combining label and value
     ("Mã ATC: N06AA09."), unlike Abacavir's bold-label-only span — the
     book's ~700 monographs were written by many different authors, so
     styling isn't 100% consistent. Fixed by matching section headings by
     vocabulary **text**, not `span.bold`, plus a new
     `match_section_with_inline_value` for the combined-span case.
  4. **Mixed-case title + false-positive whack-a-mole (outlier item 21)**:
     the class-level monograph "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" embeds
     the mixed-case abbreviation "CoA", which a strict `isupper()` check
     silently dropped from the corpus entirely. Loosening that check (first
     with an absolute lowercase-count tolerance, found wrong, then fixed
     with a **lowercase-letter ratio** instead — "Mã ATC:" has 1/5 = 20%
     lowercase, correctly still rejected, vs. HMG-CoA's 1/27 ≈ 3.7%) then
     exposed a *second* false positive: individual statin sub-headings
     ("SIMVASTATIN", "LOVASTATIN", ...) inside that same class monograph,
     each followed by their own real section but never by "Tên chung quốc
     tế" specifically. The anchor check (added earlier for the HSV/CMV
     table-header false positive, item 19) had been loosened to "any
     section" to pass existing tests — reverted to requiring "Tên chung
     quốc tế" specifically (the one invariant the book's own template
     actually guarantees), and fixed the test fixtures instead of the
     production logic.
- Final whole-book numbers after all fixes: **680 monographs** (matches
  the previously-established count from the original structural survey —
  though this is a count match, not yet a confirmed identical-set match).
  Abacavir ATC now correctly `["J05AF06"]`; Insulin now correctly 20 codes.
  46 monographs remain zero-ATC-and-not-stated-absent (down from an
  initial 48; not yet root-caused further — flagged, not silently accepted
  as final).
- 86 unit tests total, all passing, including a regression test for every
  bug above and for each whack-a-mole cycle (so a future change can't
  silently reintroduce SIMVASTATIN-as-monograph or Mã-ATC-as-title).

**Not done yet / next up:**
- The remaining 42 zero-ATC-not-absent monographs likely hide at least one
  more real pattern (per this session's track record of "one fix reveals
  the next") — worth one more investigation pass before Phase 1.5.
- Phase 1.5 (golden dataset) still requires human review by design — not
  something this session can complete alone, per the approved plan.

---

## 2026-07-31 — Session end: golden dataset NOT started; general chapters + appendices NOT investigated

**Status check requested by user at end of session ("golden dataset bạn để
đâu?" / "đã xem chuyên luận chung và phụ lục chưa?") — answering plainly
here so the next session doesn't have to guess:**

- **Golden dataset (Phase 1.5): not created.** `ingestion/data/qa/` still
  contains only `.gitkeep` — no `golden_pages.jsonl`, no
  `golden_monographs.jsonl`. This is intentional, not an oversight: per the
  approved plan, golden-set ground truth requires human review/sign-off,
  which this session couldn't do alone (dynamic `/loop` autonomy stopped
  here for exactly this reason). [[feedback-visual-verification]] means a
  future session can self-draft much of it (render pages, read them
  directly) but a human still needs to spot-check before it's trustworthy.
- **"Các chuyên luận chung" (general chapters, printed pages 37-98) and
  "Các phụ lục" (appendices, printed pages 1497-1528): NOT investigated
  this session, or any prior session.** All work so far (extract/segment/
  validation, ADR 0003, the outlier catalog) covers only the drug-monograph
  range (printed 99-1496). The only contact with these two ranges was
  incidental: reading physical page 38-39 (inside general chapters) once
  to transcribe the book's own 19-field section template into
  `segment/vocab.py`, and skimming physical ~1526-1528 (inside the
  appendices — specifically "Phân loại thuốc theo mã ATC") only to locate
  where the back-of-book index begins for `validation/back_index.py`.
  Neither range has been structurally surveyed, outlier-cataloged, or
  parsed. This gap has been flagged since the *very first* scaffold session
  (`docs/progress-log.md`'s original Phase 1 roadmap) and remains
  explicitly out of scope of the plan approved this session.
  Known content, not yet verified in depth: general chapters cover topics
  like "Kê đơn thuốc," rational antibiotic use, pediatric dosing
  principles; appendices include the body-surface-area nomogram table
  (already flagged in outlier catalog item 7 as a 2D-table extraction
  problem), IV-admixture compatibility info, and the ATC drug
  classification listing.

**Next session should pick up one of:**
1. Golden dataset drafting (Phase 1.5) — scaffold from current
   extraction/segmentation output, self-verify via page rendering, then
   get human sign-off before trusting it.
2. A first real structural investigation of general chapters + appendices
   (same rigor bar as the monograph range: whole-range scan, not a page or
   two) — needed before any chunking strategy can be designed for them.
3. The 42 remaining zero-ATC-not-absent monographs (Phase 1.4 leftover,
   not blocking).

---

## 2026-07-31 — Phase 1.4 real validation run: 92.2% recall, 98.8% precision (first-ever measurement)

**Done:**
- Ran `python -m ingestion.cli validate` for real against the full
  1668-page book. First result: 91.7% recall / 98.2% precision against
  1064 real back-index ground-truth entries (parsed from physical pages
  1530+, not a sample) — recall matched ADR 0003's original number exactly
  (665/725 there was a different, smaller ground-truth set; this run's
  1064 entries come from parsing the *entire* back index, not a partial
  scan), and **precision was measured for the first time ever** on this
  project, meeting the plan's ≥98% target immediately.
- **Found and fixed one more real bug from this first real run**: 4 of 12
  unmatched detected monographs (ALVERIN CITRAT, OXYMETAZOLIN HYDROCLORID,
  TERBUTALIN SULFAT, TIOTROPIUM BROMID) all shared the same shape — a
  **double space** in the detected title (e.g. "ALVERIN  CITRAT") that
  failed to match ground truth's single-spaced "Alverin citrat" under
  plain strip+upper comparison. Fixed by collapsing whitespace in
  `metrics._normalize_name` before comparing.
- Final numbers after the fix: **recall 92.2% (981/1064), precision 98.8%**
  — both real, measured, whole-book numbers, both improving over the
  already-fixed run (not just over the pre-session 91.7% baseline).
- Remaining unmatched entries are traced to two already-documented, known
  limitations rather than new bugs: (1) `back_index.py`'s own stated
  trade-off of treating any " - " as a brand-cross-reference marker also
  excludes genuine compound-name ground-truth entries ("Carbidopa -
  levodopa", vaccine names like "Vắc xin DPT" that use " - " internally),
  so a handful of correctly-detected monographs (CARBIDOPA - LEVODOPA,
  THUỐC PHIỆN - OPIAT - OPIOID, the DPT/MMR vaccine entries) simply have no
  matchable ground-truth counterpart, not a detection defect; (2) a
  repeating "- CoA reductase, 285" ground-truth artifact (appears ~12
  times) is itself index-parsing noise — likely a long cross-reference
  line wrapping across two physical lines in a way that splits the brand
  name from its "- CoA reductase" continuation, which then doesn't contain
  the " - " marker at its own line start and slips through the
  cross-reference filter as a bogus ground-truth entry.
- 87 unit tests total, all passing.

**Not done yet / next up (Phase 1.5, requires human review by design —
not something a single session can complete alone per the approved plan):**
- Golden dataset authoring: `scaffold-golden` CLI command, golden_pages/
  golden_monographs JSONL schemas, human review of drafted entries.
- The 42 remaining zero-ATC-not-absent monographs and the back_index.py
  compound-name/cross-reference-wrapping noise above are both flagged, not
  blocking — real, moderate-size gaps documented for whoever picks this up
  next.

---

## 2026-07-30 — Phase 1.2 `segment/` pure logic built and validated against real PDF

**Done (real production code, all reused by both the future CLI pipeline
and validation — no logic duplicated):**
- Transcribed the book's own documented 19-field monograph template
  verbatim from its source (physical page 38/39 printed, "HƯỚNG DẪN SỬ DỤNG
  DƯỢC THƯ QUỐC GIA VIỆT NAM") into `vocab.py`'s `SECTION_DEFS`, rather than
  guessing — cross-checked against real bold headings in the Abacavir/
  Acarbose monographs (exact text match, modulo a trailing colon some pages
  have and others don't, now normalized). Added `ten_thuong_mai` ("Tên
  thương mại") as the confirmed 19th, undocumented-but-real field.
- Built `merge.py` (multi-line/multi-fragment title merging), `detector.py`
  (monograph + section boundary detection), `atc.py` (3-state ATC
  extraction: found / recovered-from-noise / stated-absent), `units.py`
  (defensive mg/mcg/mmol validation — see below), `models.py`.
- **Found and fixed a second real title-fragmentation bug by rendering a
  page to an image and reading it directly** (not just reasoning from
  coordinates): "ACICLOVIR" was detected as two separate titles, "ACIC"
  (font size 10.0) and "LOVIR" (font size 9.5) — the same visual word
  rendered at two slightly different sizes in the source PDF. The merge
  logic originally required exact font-size equality (which happened to
  work for the GONADOTROPIN wrap case since both its fragments are size
  9.5) — dropped that requirement per the same "font size is not reliable"
  lesson from ADR 0003, now applied *within* a title's own fragments, not
  just across monographs. Also fixed the join character: a genuine
  same-line split needs no space ("ACIC"+"LOVIR"="ACICLOVIR"); a genuine
  multi-line wrap needs one (GONADOTROPIN case) — distinguished by the y0
  gap. This same fix also resolved two other silent duplicate-name
  artifacts (HSV, CMV) found in the same smoke test.
- Smoke-tested the full detector against the real PDF: 695 monograph titles
  detected (down from 702 pre-fix, closer to the previously-established
  ~680 count), part-divider correctly excluded, ABACAVIR/INSULIN present,
  GONADOTROPIN wrap correctly merged, zero unexplained duplicate names.
- **Investigated the one remaining duplicate name ("SALBUTAMOL", pages 1261
  and 1263) by rendering both pages and reading them directly — confirmed
  it is NOT a bug**: two genuinely different, complete monographs
  ("Dùng trong hô hấp" / respiratory vs. "Dùng trong sản khoa" / obstetric
  use), each with a full 18-section template. Added as outlier-catalog item
  18 with an explicit note for Phase 1.3's assembler: `drug_id` generation
  must fold in the bold, non-all-caps qualifier line beneath the title, or
  it will wrongly treat this legitimate case as a duplicate-title collision.
- Confirmed via a targeted regex scan that the `units.py` whitespace-split
  defense (built by analogy to the confirmed ATC defect) has **zero**
  confirmed real occurrences in this corpus so far — documented honestly as
  a defensive-only check, not a confirmed defect, per CLAUDE.md.
- 44 unit tests total (up from 9), all passing, including regression tests
  for every real bug found this session (kerning jitter, column-merge,
  GONADOTROPIN wrap, ACICLOVIR same-line split).
- Rendering a PDF page to an image and reading it directly (not just
  reasoning from PyMuPDF coordinates) turned out to be a fast, reliable way
  to self-verify segmentation bugs — used for both real bugs found this
  phase (ACICLOVIR, SALBUTAMOL) without needing a human to look at the page.
  This changes the Phase 1.5 golden-dataset plan: much of the
  ground-truth drafting can be self-verified this way before a human spot-
  checks it, rather than requiring a human to author it from scratch.

**Not done yet / next up:**
- Phase 1.3: `assembler.py` (must handle the SALBUTAMOL qualifier-line case
  above), `segment/io.py`, `cli.py run`, wired end-to-end; smoke-test on a
  small page range before a full-book run.
- Phase 1.4: `validation/back_index.py` + `metrics.py` (recall/precision
  against the back-of-book index), `cli validate`.

---

## 2026-07-30 — Phase 1.1 `extract/` module built and validated against real PDF

**Done (real production code, not exploratory scripts — replacing the
empty `ingestion/ingestion/extract/` stub per the approved segmentation +
eval plan):**
- Built `models.py` (`Span` dataclass), `page_map.py` (physical→printed page
  mapping, read per-page rather than assumed as a constant — verified
  correct and constant at +1 across all tested milestone pages: physical 0,
  36, 37, 98, 100, 1496, 1497, plus correctly returns `None` for blank/title
  pages), `spans.py` (continuous cross-page span stream with column
  tagging), `io.py` (JSONL persistence), and `glyph_order.py` (the
  mandatory pre-ingestion sanity gate).
- Added `pytest`/`pymupdf` to `ingestion/pyproject.toml` (previously empty
  `dependencies = []`) plus `[tool.setuptools.packages.find]` to fix a
  package-discovery ambiguity that broke `pip install -e .` — both
  confirmed via a real editable install, not just added and assumed to work.
- **Corrected a real gap in ADR 0003's own validated finding**: re-verifying
  the "reversed glyph order" defect as real tested code (not trusted from
  the prior exploratory script) found **2 genuine occurrences, not 1**
  (physical pages 714 and 1373 — two different defect shapes, see outlier
  catalog item 9's rewrite for full detail). Getting a trustworthy count
  took 3 detector iterations after the first naive whole-book run reported
  1113 false positives (kerning jitter + a column-boundary false-merge bug)
  — full false-positive history and the fix (group by PyMuPDF's own block
  index, not hand-picked x-coordinates) documented in
  `extract/glyph_order.py`'s docstring and the outlier catalog.
- Smoke-tested `extract_spans`/`build_page_map` against the real PDF:
  253,518 spans extracted, 30,728 bold, first monograph title (ABACAVIR)
  correctly located at physical page 100 / printed 101.
- 9 unit tests added (`tests/test_extract_glyph_order.py`), all passing,
  including regression tests for the kerning-jitter and column-merge false
  positives found during validation (so they can't silently regress).

**Not done yet / next up:**
- Phase 1.2: `segment/` pure logic (vocab, merge, detector, atc, units) with
  unit tests reproducing every documented bug case (GONADOTROPIN wrap,
  part-divider false positive, ATC whitespace/O-0, "Chưa có" state) — see
  the approved plan (`ingestion/ingestion/segment/` is still an empty stub).
- The 3 formula-region pages (92, 94, 805) that also trip
  `scan_reading_order` should **not** have their "corrected" text trusted —
  same guidance as outlier catalog item 8 (2D formulas aren't linearly
  recoverable); no auto-correction should be applied to those specifically,
  flag-only.

---

## 2026-07-30 — Eval strategy locked in; Phase 1.0 cheap surveys run

**Done (direct requirement: "phải eval thật kỹ... phải có chiến lược rõ
ràng" — plan mode used to design a full segmentation + eval framework before
writing any real ingestion code):**
- Designed and got user approval on a full implementation plan covering
  `extract/` + `segment/` + a `validation/` package, merging the
  already-validated ADR 0003 methodology (back-index recall, currently
  91.7%) with a 6-point eval framework the user specified (visual diff,
  round-trip test, character-level text coverage, structure validation,
  golden dataset, downstream RAG eval) plus a follow-up list of
  domain-safety checks (adult/child dosing not mixed, mg/mcg/mmol units not
  corrupted, warning/contraindication sections captured, chemical formulas,
  header/footer leakage, page numbers not injected mid-paragraph). Full plan
  is preserved for reference; key decisions below are now the standing
  design, not just a plan-file artifact.
- Confirmed target audience (doctors/pharmacists, not lay users — see
  `project_target_audience` memory) explicitly informs why domain-safety
  checks (dosing-population mixing, unit corruption) are being treated as
  first-class eval dimensions, not nice-to-haves.
- Ran Phase 1.0 whole-book surveys (scratch script, not committed):
  - **Zero embedded images** across all 1668 pages (`get_images(full=True)`,
    measured) — image/caption validation tooling is not needed for this
    corpus.
  - **Adult/child dosing splits are the norm, not rare**: "Người lớn"/"Trẻ
    em"/"Trẻ sơ sinh" terms appear on 1121 of ~1400 monograph-range pages —
    elevates dosing-population-mixing to a standing validation check.
  - **Found and confirmed a real chemical reaction equation** (physical page
    1033, cyanide-antidote mechanism: `Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3`) and a
    **new outlier**: the reaction arrow extracts as a Private-Use-Area glyph
    (`U+F0AF`), not a standard Unicode arrow — added as outlier-catalog item
    16. A regex scan for chemical-formula-shaped tokens found 9 raw hits,
    8 of which were false positives (flu-strain names, receptor names) —
    genuine chemical notation exists but is rare, not systemic.
  - Attempted to pin down the exact shortest monograph name+page, but the
    crude (unmerged, no multi-line-title-merge) scan script produced a
    **different longest-monograph ranking** than the already-documented one
    (previously: "AMOXICILIN VÀ KALI CLAVULANAT" at 45,623 chars; this
    script's top result was INSULIN at 41,799 chars) — flagged as
    unreliable rather than reported as fact, and explicitly deferred to
    Phase 1.2's real detector rather than trusting a quick script's number
    over the previously-validated one. Added to outlier catalog's "not yet
    investigated" list with the reasoning, not silently dropped.
- Added outlier-catalog items 15 (no images), 16 (PUA reaction-arrow
  glyphs), 17 (adult/child dosing prevalence).

**Not done yet / next up:**
- Phase 1.1 onward: build real `ingestion/ingestion/extract/` and
  `segment/` modules (currently still empty stub packages) per the approved
  plan — `page_map.py` first, then `spans.py`/`glyph_order.py`, then the
  segment detector/merge/atc/units logic with unit tests, then wiring
  `cli.py run`, then the `validation/` package (back-index recall+precision,
  golden dataset, char-coverage/structure/domain-safety checks,
  visual-diff). See the approved plan file for the full phase breakdown and
  numeric targets (≥98% monograph recall/precision, ≥99% mean character
  coverage, zero-regression golden-set gate, manual visual-diff sign-off on
  hardest pages) if this session ends before implementation completes.
- `pytest` and `pymupdf` need to be added to `ingestion/pyproject.toml`
  dependencies (currently `dependencies = []`) — confirmed both are already
  available in the global Python 3.12.10 env (PyMuPDF 1.28.0, pytest 7.4.4)
  but not yet pinned in the package's own dependency list.

---

## 2026-07-30 — Whole-corpus structural survey (not just anecdotes)

**Done (direct pushback: "I feel like you're minimizing how complex this
PDF really is — go find another 10-30 outliers, not just Vitamin D"):**
- Built a real per-monograph structural survey across all 680 detected
  monographs (not 2 anecdotes) — computed ATC-code count, known-section
  count, and character length for every one.
- **Multi-ATC monographs are NOT rare**: 173/680 (25.4%) have more than one
  ATC code — INSULIN has 20, BETAMETHASON and DEXAMETHASON 11 each,
  PREDNISOLON 10, HYDROCORTISON 9. The earlier "found 2 examples" framing
  badly understated this. Even 25.4% is a floor (see next point).
- Investigated the 22 apparent "zero ATC" monographs (spot-checked 14):
  found **two distinct real causes of false negatives** — stray internal
  whitespace splitting an ATC code (`"J04A C01"` instead of `"J04AC01"`)
  and digit/letter confusion (`"NO3AX12"` instead of `"N03AX12"`) — 9 of 14
  resolved as real ATC codes hidden by extraction noise (one of them,
  TRIAMCINOLON, turned out to have 5 ATC codes, meaning the true
  multi-ATC percentage is higher than 25.4%). The remaining ~5 genuinely
  say `"Mã ATC: Chưa có."` (not yet assigned) — a valid data state, not an
  error.
- Found and confirmed a **false-positive monograph boundary**: the
  part-divider title "CÁC CHUYÊN LUẬN THUỐC" (Part 2's own section title,
  not a drug) was detected as if it were a monograph.
- Measured real structural variance: monograph length ranges 2,331-45,623
  characters (~20x spread), detected section count ranges 8-20.
- All findings added to `docs/pdf-parsing-outlier-catalog.md` (items 12a
  revised with real numbers, 12c, 12d, 12e — new).
- Verified one of my own debugging steps was itself wrong (read raw page
  text from the top instead of the correctly-bounded monograph segment,
  which briefly looked like a segmentation bug before being traced back to
  a debugging mistake, not a real defect) — corrected before reporting.

**Not done yet / next up:**
- Full-corpus re-count with the relaxed ATC regex (whitespace-tolerant,
  O/0-aware) not yet run — only 14/22 zero-ATC cases spot-checked, and the
  173/680 multi-ATC count still uses the strict (undercounting) regex.
- Phase 1 real implementation still pending overall (see earlier entries).

---

## 2026-07-30 — Confirmed class-level monographs and a real source typo

**Done (direct follow-up: "have you checked drug-class entries like Vitamin
D, or actual spelling/font-size errors?"):**
- Found and confirmed a **second real example of a class-level monograph**
  covering multiple ATC codes/substances: "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"
  (7 ATC codes, one per specific vitamin D analogue) — same pattern as the
  earlier GONADOTROPIN finding, confirming this is recurring, not a one-off.
- Found and confirmed a **genuine spelling/capitalization typo in the
  source PDF itself**: the running header on this monograph's continuation
  pages reads "Vitamin d..." (lowercase d) vs the correct ALL-CAPS heading
  "VITAMIN D...". Verified via font/bbox inspection that this is a real
  source-text inconsistency, not an extraction artifact. The detection
  heuristic still worked correctly here (the typo'd header isn't all-caps
  so it's correctly rejected), but this was incidental, not a designed
  defense against typos.
- Added both findings to `docs/pdf-parsing-outlier-catalog.md` (items 12a,
  12b), with the general lesson: rely on multiple independent structural
  signals, not any single text match, since real source typos do occur.
- Added `CLAUDE.md` with a standing rule: never fabricate or bluff a claim
  (number, test result, capability estimate) — verify before stating,
  explicitly flag estimates as estimates. Grounded in concrete incidents
  from this investigation (the size-threshold bug, the scope-gap bug).

**Not done yet / next up:**
- No systematic scan yet for *other* class-level (multi-ATC) monographs
  beyond the two found incidentally — Phase 1's data model should assume
  ATC code is a list per monograph regardless, rather than trying to
  enumerate every class-level entry in advance.
- Phase 1 real implementation still pending overall (see earlier entries).

---

## 2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns)

**Done (in response to direct follow-up questions about table/formula
handling and full-book coverage):**
- Found and confirmed a **table split across a page break loses its header
  on the continuation page** — real example: "Bảng 4" (ARV rash management
  table) ends with an orphaned, header-less data row on the next page when
  extracted with `pdfplumber`.
- Found the **same header-loss risk also happens across a column boundary
  within a single page** (no page break needed) — real example: "Bảng 6".
- Found and confirmed **2D grid/nomogram tables are not linearly
  recoverable** — the body-surface-area lookup table (appendix) extracts as
  scrambled bare numbers with no row/column association.
- Found **two different formula-rendering outcomes**: a simple inline-
  exponent formula (Du Bois BSA) extracts cleanly as text; a stacked-
  fraction formula (Cockcroft-Gault) extracts as disordered fragments —
  confirmed the determining factor is 1D vs 2D visual layout, not "formulas
  are always broken."
- Found and confirmed a **full-width table that breaks out of the normal
  two-column page grid** (bbox spans nearly the full page width).
- Checked whether front-matter "committee list" pages are genuinely
  multi-column (the user suspected 3 columns) — confirmed via bbox
  inspection they are **not** true structural columns, just single wide
  text blocks with internal whitespace padding between names.
- Consolidated **all** outlier findings from this investigation (this entry
  and the previous one) into a single, reusable, generalized reference:
  `docs/pdf-parsing-outlier-catalog.md` — written so it can guide parsing of
  other similarly-structured PDFs, not just this book.

**Not done yet / next up:**
- No automatic detector exists yet for (a) 2D-formula regions, or (b) 2D
  grid-table reconstruction — both flagged as open items in the catalog,
  not silently skipped.
- Table-continuation re-attachment (page-break and column-break cases) has
  no implementation yet — needed before Phase 1 can trust any multi-row
  table content.
- Phase 1 real implementation still pending overall (see previous entry).

---

## 2026-07-30 — PDF parsing strategy validated empirically (pre-Phase-1)

**Done:**
- Investigated the real PDF structure before writing any ingestion code
  (previous scaffold's assumptions about `doc.get_toc()` turned out wrong).
- Confirmed: 1668 pages, no bookmark/outline (0 TOC entries), tagged-PDF
  structure tree exists but is too shallow to use (~29 elements only).
- Cross-tested 3 extraction tools on real sample pages: PyMuPDF (correct
  reading order — kept as primary), pdfplumber (scrambled reading order on
  this layout — demoted to table-extraction-only use), opendataloader-pdf
  (correct reading order, useful independent font-metadata cross-check, but
  inconsistent heading classification — not trusted as sole signal). Docling
  install hit a numpy/pyarrow ABI conflict in the global Python env; tested
  in an isolated `.venv_docling_test/` (gitignored) instead of risking the
  global environment — see whether that resolved before relying on it.
- Found the real structural ground truth: every section/monograph heading is
  a **bold font span** in the PDF (confirmed at the PyMuPDF span level AND
  independently by opendataloader's own font metadata — two tools agreeing).
  Font **size** is not reliable (10.0pt and 9.5pt both occur for genuine
  monograph titles) — an early size-based threshold silently dropped ~15% of
  real monographs; caught and fixed via whole-document validation, not
  spot-checking.
- Found the real ground truth for validation: the back-of-book "Mục lục tra
  cứu" (page ~1528 onward) has exact page numbers per drug — much stronger
  than the front-matter drug list (which has no page numbers). Also found
  the book's own contents page states individual monographs run printed
  pages 99-1496 exactly.
- Ran automated whole-document (1668-page, ~20-50s per run) validation
  against that page-verified ground truth: **91.7% recall** (665/725), with
  the remaining gap traced to one concrete, fixable cause (multi-line
  wrapped ALL-CAPS titles not yet merged across lines) rather than a flaw in
  the bold-span signal itself.
- Documented the full methodology and results in
  `docs/adr/0003-pdf-parsing-strategy.md` and updated the ingestion section
  of `docs/architecture.md` to match reality (removed the incorrect
  TOC-preference assumption).

**Also validated (in response to direct user questions about correctness):**
- **No real duplicate drug monographs** found across the full 1405-page
  monograph range. The one apparent collision ("GONADOTROPIN" at 2 pages)
  is a detector artifact from the known multi-line-title bug (a different
  monograph's wrapped title fragment collided with it), not real content
  duplication.
- **Confirmed the PDF is genuinely two-column** (bounding-box verified: left
  column x≈44-299, right column x≈308-562). PyMuPDF's reading order across
  columns is correct (already implied by earlier validation).
- **Found and precisely characterized one real data-corruption defect**:
  a single text run on physical page 1373 has reversed (right-to-left)
  glyph order, producing scrambled text — confirmed by reversing the
  string, which recovers the correct Vietnamese sentence. A full scan of
  all 1405 monograph pages (grouping fragments into visual rows, checking
  for descending x-order) found this exact **1 occurrence and no others** —
  rare, isolated, but real, and now has a cheap (~16s) automated detector.
- Full details, methodology, and exact numbers added to
  `docs/adr/0003-pdf-parsing-strategy.md` under "Follow-up validation."
- **Caught a real scope gap**: the glyph-reversal scan above was initially
  run on the monograph range only (1405 of 1668 pages), leaving ~260 pages
  (front matter, appendices, back index) unchecked. Re-ran across the full
  1668 pages: still exactly 1 defect (same page, 1373) — confirmed isolated,
  not hiding elsewhere. Also found 6 near-empty pages (3, 37, 99, 1495, 1497,
  1666), all of which land exactly on major section-transition boundaries —
  intentional print blank pages, not lost content.

**Not done yet / next up:**
- Resolve/confirm docling status in the isolated venv (numpy/pyarrow
  conflict was fixed by using a separate venv; install completed — actual
  parsing comparison against the sample pages still pending).
- Phase 1 real implementation: build `ingestion/` for real using the
  validated bold-span detector (not the exploratory scratch scripts) as one
  continuous cross-page stream (not per-page silos), fix the multi-line
  heading-merge gap, add the glyph-order sanity check as a mandatory
  pre-ingestion pass, re-run the validation script to confirm improved
  recall, then proceed to chunking + embedding + Qdrant upsert.
- Decide and implement chunking strategy for the non-monograph parts of the
  book (general chapters pages 37-98, appendices 1497-1528) — needed so the
  full book (page 0 to last) ends up captured in the RAG corpus in some
  appropriate form, per the user's explicit requirement that no content be
  silently dropped.
- Clean up exploratory `scratch_*` files from the repo root as they
  accumulate during investigation (routinely deleted after findings are
  persisted to docs — not left in git history).

---

## 2026-07-30 — Initial monorepo scaffold

**Done:**
- Designed the microservices architecture (see `docs/architecture.md`):
  Python/FastAPI `ai-service` for RAG, NestJS for `api-gateway`/`auth-service`/
  `user-service`/`chat-service`, Next.js `web`, Qdrant for vectors, Postgres
  for relational data, Redis reserved for caching/queues.
- Scaffolded the full monorepo directory tree (`apps/`, `packages/`,
  `ingestion/`, `infra/`, `docs/`) with baseline config (package.json/
  pyproject.toml stubs, pnpm workspace, docker-compose topology stub).
- Moved `duoc-thu-quoc-gia-viet-nam-2018.pdf` into `ingestion/data/raw/`.
- Decided vector DB: **Qdrant** over pgvector (`docs/adr/0001-vector-db-qdrant.md`).
- Decided deployment: GitOps via the **team's existing ArgoCD instance**,
  not a custom push-based CD pipeline (`docs/adr/0002-argocd-gitops.md`,
  `infra/argocd/`). CI's job is build/test/push image + bump the Helm values
  image tag; ArgoCD does the actual sync.
- `git init` + initial commit (this scaffold).
- Created a private GitHub repo (`BaoVu2k4/vsf-duocthu`, default branch
  `master`) and pushed the initial commit; fixed `targetRevision` in the
  ArgoCD Application manifests to `master` to match.

**Not done yet / next up (Phase 1 of the build roadmap in `docs/architecture.md`):**
- No business logic exists yet anywhere — this was scaffold only.
- Phase 1: build the `ingestion/` pipeline for real (PDF extraction via
  PyMuPDF, monograph/section segmentation, section-aware chunking, OpenAI
  embeddings, Qdrant upsert) and validate retrieval quality via the
  `ingestion/notebooks/` QA step.
- Still pending/TBD: which cloud provider (AWS/GCP/Azure) for Terraform
  (`infra/terraform/README.md`), and the team's ArgoCD instance's actual
  cluster/server + project details (`infra/argocd/README.md` TODOs).