duocthu/docs/progress-log.md

Progress Log

2026-08-05 (evening 3) — The LLM cloud is LIVE: DeepSeek generation + Cohere rerank on the real corpus

The owner rejected the $0 offline build as the deliverable and set a hard deadline. The chatbot is now a real LLM RAG, grounding kept ON, running the full HTTP stack (ai-service :8079 ↔ Postgres trace ↔ Qdrant; web :3000). Commit 9c4273b (plus 92497ae/9e9cef7/1b6f399 earlier this session, which committed the previously-uncommitted evening-1/2 work).

What was turned on (live via gitignored .env; committed defaults stay disabled/section-only so CI/fresh-clone never touches cloud):

• EMBEDDING_PROVIDER=cohere-v4 — query now embedded in the corpus's cohere.embed-v4:0 space (probe: 1024-dim, L2 1.0, ~1.95s). No re-embed; the 15,100 vectors already exist.
• ANSWER_PROVIDER=bedrock-converse + deepseek.v3.2 — new adapters/bedrock_converse.py (Bedrock Converse API, boto3, model-agnostic; Qwen/GLM = 1 env + 1 ARN). Probe OK. GPT-4o confirmed NOT on Bedrock; OpenAI gpt-oss, DeepSeek, Qwen, GLM, Mistral, Kimi ARE (checked live).
• RERANK_ENABLED=true — cohere.rerank-v3-5 trims the overview/similarity fallback: a free-form drug question no longer dumps all ~29 sections at the model (measured 29 → 6 on the fever/paracetamol case). Section route never reranks; fail-open (outage → book order, answer survives).
• rag/prompt.py rewritten to current citation-enforced practice: each dose carries its population/condition label (no adult/paediatric mixing), cite only the supporting block, no [n] spam, abstain on insufficient evidence.

IAM: managed policy BedrockEmbeddingInvoke bumped to v4 (invoke on titan-embed, cohere.embed-v4, deepseek.v3.2, cohere.rerank-v3-5); repo file synced. Codex was off, no collision.

Verified: ai-service 111 passed, 3 skipped (+12 this milestone). Live HTTP POST /v1/rag/query returns a grounded LLM answer with a citation and a Postgres trace id. Golden golden_e2e (35 Qs): 19/19 answerable questions grounded with the correct drug (incl. typo paracetamon, alias Acetaminophen, multi-turn inheritance); 14 adversarial correctly abstained (fake drugs, weather, symptom→drug reverse-lookup, multi-drug). Two real gaps: a price question answers from the monograph instead of "no price in the formulary", and "should I double the dose?" is not directly warned. Every generated=True answer passed grounding.verify.

Cost/safety: Bedrock is pay-per-call — verified 0 EC2 (3 regions) and no provisioned throughput; idle = ~$0. A few dozen probe/smoke/eval calls this session, cents-scale on the estimate; exact bill not checked.

Separate track, NOT done (background subagent started, own worktree): reconstruct the 151 quarantined tables with a needs_expert flag on uncertain cells + parse Part 1 (poisoning/pregnancy/hepatic-renal) & Part 3 (BSA/ATC) + re-embed. This is a multi-hour ingestion pass with the whole-doc validation gate and will NOT be clinician-validated within the deadline — deliberately kept off the deadline path.

2026-08-05 (evening 2) — Conversational chat core wired LIVE end-to-end (offline, $0); owner wants the LLM cloud next

The chat core is now live and serving multi-turn, not just unit-tested. It runs $0/no-cloud because the section-route is a payload filter (no query embed) and generation is still off (verbatim), but the conversational behaviour is real and smoke-tested against the running service (ai-service :8079, web :3000).

Built (rag/conversational.py ConversationalLoopService, wrapping the safe GroundedAnswerService; wired through bootstrap.py/main.py/routers/rag.py with an optional conversation_id, plus route.ts sending it and a ChatPanel error state):

• Multi-turn follow-up inheritance. "Chống chỉ định Metformin" then "còn trẻ em thì sao?" carries the drug+section forward and names it ("Về metformin: …").
• Smalltalk. "chào bạn" gets a friendly redirect, not a failed-drug-lookup refusal.
• Drug-name-only → the whole monograph. Typing "PARACETAMOL" now returns all 18 sections in book order with 【heading】s and per-section citations (QdrantRetriever.find_by_drug + SECTION_ORDER; RetrievalService uses it when a drug resolves but no attribute is named) — the earlier "specify an attribute" dead-end is gone.
• Typo → ask, never threshold-guess. Only an EXACT drug name auto-resolves; a fuzzy match is offered as a question ("Ý bạn là: Metformin?") via CatalogDrugResolver.suggest(min_score=0.72). A completely-wrong name → "Không có thuốc này trong Dược thư Quốc gia." A formulary must not silently answer about a different drug than the one meant.
• Autocomplete endpoint GET /v1/rag/suggest?q= (CatalogDrugResolver.complete, substring/prefix) — the frontend dropdown that consumes it is still to build.
• BSA calculator rag/calculators.py (Appendix 1, DuBois, tested vs the book's own cells).

Verification: ai-service 99 passed, 3 skipped; live smoke test of all four conversation behaviours plus the monograph/typo/not-supported cases. A refine-loop bug (a refined query dropped the inherited drug and abstained, discarding a good answer) was found in my own code and removed before shipping — clarify + inheritance are the loop's value, retrieval-refine is not, and it is gone from the live path.

Owner's next-session directive (recorded in memory project-llm-cloud-plan): stand up the cloud LLM — semantic query embedding (EMBEDDING_PROVIDER=cohere-v4, already IAM-permitted) and answer generation (a cheap model, non-Anthropic OK, via a Bedrock Converse adapter, needs its ARN added to BedrockEmbeddingInvoke). The offline build was budget/safety-first, not LLM-avoidance; the owner wants the real AI experience next, with grounding.verify and the quarantine contract kept ON.

2026-08-05 (late) — Read the source book's own structure; scope + usage-pattern findings (checkpoint before handoff)

Read the Dược thư 2018 front matter directly (printed p8 "Nội dung", p39 "Hướng dẫn sử dụng") to understand what the book is for and how clinicians use it — recorded in memory reference-duoc-thu-2018-structure. Key facts that reshape the chatbot scope:

• The book has three parts. The corpus is Part 2 (drug monographs, printed 99–1496) ONLY. Excluded and clinically important: Part 1 general chapters (printed 37–98: prescribing in the elderly / hepatic-renal impairment / children / pregnancy-lactation; disease-class guidance for asthma, epilepsy, HIV, antibiotics, TB, hepatitis B, antipsychotics; drug allergy; poisoning & antidotes; drug-interaction principles) and Part 3 appendices (printed 1497–1528: body-surface-area calc, IV admixture, ATC classification). So "how to treat asthma", "antidote for X", "BSA-based dosing" have no data in the index today — a coverage limit, not a retrieval bug.
• The 19 monograph fields are fixed and documented on p39; a field is omitted when the book has no info (so a missing section is not necessarily a parse bug).
• Field 14 dose is a general adult+child oral reference dose unless stated; the clinician adjusts. → the tool supplies reference data, not a prescription.

Data checks run this session (against chunks.jsonl), correcting earlier pessimism:

• Indication is searchable: 48 drugs' chi_dinh mention "sốt". Reverse lookup (symptom → drugs) is feasible from content, but retrieval is drug-first, so not answerable yet.
• mg/kg dosing is in PROSE, not tables: 574 lieu_luong chunks contain "mg/kg", all prose, across 295 drugs, 473 of them mentioning trẻ em. So the primary weight/age dosing (incl. pediatric) is answerable; the 83 quarantined dosing tables are mostly the supplementary renal-adjustment tables (49 of those 83 drugs also have mg/kg prose).
• Pregnancy dosing is mostly qualitative: 670 drugs have a thoi_ky_mang_thai section but only ~23 chunks carry a mg figure — the book rarely gives a separate pregnant dose, so answer = pregnancy caution + standard dose, never a fabricated pregnant-specific number.
• drug_id can be compound (paracetamol_acetaminophen); alias resolution must map "paracetamol" → that id.

Design consequence discussed with the owner (not yet built): the "understand" stage must classify the turn type (smalltalk / medical query / multi-drug interaction / symptom-indication / out-of-scope / injection-shaped), not just resolve a drug. Refusing a clinician's symptom→drug question as "recommendation_out_of_scope" was wrong for this audience — such questions are indication lookups and should be answered from chi_dinh. Multi-drug interaction/contraindication questions need a real PLAN → gather both drugs → synthesize step (the ADR-0007 PLAN node, still unimplemented), and an absence-of-evidence answer must state where it looked, never assert "safe".

Session state / not yet done (so a fresh session can resume): the chat module's domain glue is built and unit-tested (rag/conversation.py ports + summariser, rag/conversational.py orchestrator + is_smalltalk); it is not wired to the endpoint. Live wiring (turn-type classifier, loop-around- GroundedAnswerService, Postgres store, conversation_id on /v1/rag/query, route.ts, ChatPanel error state), the P0 audit fixes (§5 context-mixing metadata, §8 Qdrant-error degradation), reverse-indication retrieval, and the table vision-consensus pipeline all remain to do. No code was wired live this session; the behavior spec is still being clarified with the owner before wiring.

Owner decision: parse the WHOLE book, re-chunk freely (not just Part 2 monographs). Current corpus covers physical pages 100–1494 only. To add: Part 1 general chapters (physical ~36–97) and Part 3 appendices (~1496–1527); front-matter list (13) and index (1529+) are already used as validation ground truth. Read segment/detector.py to ground the plan — the machinery already generalizes: a chapter title ("NGỘ ĐỘC VÀ THUỐC GIẢI ĐỘC") has the same shape as a monograph title (bold + mostly-upper + short), so is_monograph_title_candidate extends by widening the hardcoded 99–1496 range per content_type. Only two real changes: (1) parametrize the page range + add a content_type (monograph|chapter|appendix); (2) chapter/appendix sub-headings are free-form ("Hô hấp", "Co giật"), not the 19-key vocab, so detect_section_headings needs an open-taxonomy mode (bold + short = heading, store the text, no match_section requirement). Everything downstream (span extraction, table/formula quarantine, provenance, chunker) is content-type-agnostic and reused → schema v5 adds content_type + chapter_id. Gate (CLAUDE.md): the span-routing ledger must account for ALL 1668 pages with unassigned=0, not just 99–1496. Then embed only the NEW chunks (Cohere, pennies, announce first). This is a focused ingestion pass (detector + assembler + chunker + whole-doc re-run + validation), not a one-liner — not attempted this session beyond grounding the plan.

Done this session (App 1, self-contained, validated): rag/calculators.py body_surface_area_m2 replaces Appendix 1's lookup table with the book's DuBois formula (S = W^0.425 × H^0.725 × 71.84), tested against three of the book's own table cells (165cm/60kg→1.66, 90cm/10kg→0.50, 170cm/70kg→1.81) — tests/test_calculators.py, 3 passed. Audit §7 (calculation = tested function, never an LLM).

2026-08-05 (evening) — Conversational orchestrator wired to the existing loop; data-quality audit; budget verified live

Chat module (the glue ADR 0007 specified and nothing had called). Added the two missing conversation ports and their offline defaults to rag/conversation.py (ConversationStore/InMemoryConversationStore, Summariser/DeterministicSummariser) and the orchestrator rag/conversational.py (ConversationalRagService). It owns no rules of its own: load state → resolve this turn → inherit gaps from Focus → derive clarify signals from resolver state → reasoning.run_turn → update focus, append turns, summarise overflow, save → name any inherited drug. Runs with no LLM/service (collaborators are protocols). DeterministicSummariser records only drug/section labels, never cell values, so the no-clinical-content-in-summary rule holds by construction rather than by trust — closing the summary-bypasses-grounding hole flagged in review. 11 new tests; full ai-service suite 91 passed, 3 skipped.

Still NOT wired live: a TurnResolver bridge over CatalogDrugResolver + SectionResolver; bridges from RetrievalService→Retrieve and the grounded generation path→Generate; PostgresConversationStore + migration; a conversation_id on /v1/rag/query; route.ts sending it and dropping the hardcoded intent: fact_lookup; a ChatPanel error state; and the multi-turn eval run. So no claim yet that history/loop improves answers — designed and unit- proven, not measured end to end.

Data-quality audit (self-run this session, not quoted from this log). 684 drugs; critical-section coverage is strong — dosing missing 0.1% (1), contra- indication 0.4% (3), indication 0%. But 83/684 drugs (12%) have their dosing inside a quarantined table, so a dose query for them returns VERIFY_PDF (crop, no number) — the largest answer-quality gap for a clinician audience, and it lands on the single most-asked query. 125 chunks carry a leading ": " label-leak artifact (93 in ten_chung_quoc_te). Vector-path text loss appears contained to 22 flagged lines (completeness of detection unverified). Nobody clinician-side has validated the 8.2M chars against the book — still the largest unmeasured area.

Table validation — the instrument that text extraction lacked. Demonstrated that vision reads a real quarantined dosing table cell-by-cell: GABAPENTIN's renal-adjustment table (printed 706) came back exactly by eye where pdfplumber's text layer could not structure it. Found and corrected a page-index off-by-one in my own render (data physical_page N = doc[N], 0-based) — proof that correctness must not depend on trusting coordinates. Strategy, given pharmacists are end-users, not labelers: reconstruction powers retrieval only; the displayed answer stays crop + page (clinician verifies at point of use). Validation is automated — vision↔geometric consensus + round-trip visual + book invariants — with a per-cell precision-first gate (disagreement → stays crop-only). Not yet built; 151 blocks is small enough for full census.

Budget, read live from the billing console (owner login; ai-lab-user has no billing API perms): $138.50 remaining, entirely AWS promotional credit, not the owner's card; August bill $0. Deploy target chosen: team k3s, but deferred (mutating a shared cluster). Generation still off (answer_provider= disabled) — extractive/verbatim, which is defensible for clinicians; wiring a cheap model (Nova/Haiku via Bedrock Converse) needs its ARN added to the BedrockEmbeddingInvoke policy, which today grants invoke on the two embedding models only.

2026-08-05 — An answer layer that cannot state a number the book does not

Today started by walking the demo path rather than the test suite, and the two are not the same thing. The suite was green and the demo was broken.

What the walk found, by running it rather than reading it. The backend answers real Vietnamese questions against the real embedded corpus with real citations and zero cloud cost — the section route is a payload filter, not a vector search. Chống chỉ định của Metformin là gì? returns the true contraindication text with one citation; Tương tác thuốc của Warfarin? returns two. But Tôi sốt cao, uống Paracetamol được không? returned HTTP 500: the similarity fallback reached Bedrock, which is revoked, and botocore.AccessDeniedException escaped as an unhandled error. Any question whose phrasing is outside the section phrase table takes that path.

That crash also re-verified the cloud shutdown today, live — the denial came from the service, not from a claim in a document.

Four defects, all fixed, all at $0.

1. The 500. adapters/embedding.py now translates provider failures into the domain error QueryEmbeddingUnavailable, and RetrievalService catches it and abstains with reason="query_embedding_unavailable" — deliberately distinct from insufficient_retrieval_score, so an outage never reads as an empty corpus. rag/ still imports no SDK.
2. A default config that does not work. config.py pointed at collection duoc_thu_chunks; the real one is duocthu_v1. embedding_provider defaulted to disabled, so /v1/rag/query returned 503 on a fresh clone.
3. Neither existing provider was a safe default. local-smoke searches a SHA-256 vector against a Cohere collection — confident, meaningless hits. cohere-v4 spends the boto3 retry budget (~30s) before failing on a revoked account. Added SectionOnlyQueryEmbedder: refuses locally and instantly, so retrieval is confined to the route that measured 16/16.
4. Safety abstention was incidental, not a gate. Symptom questions abstain with reason="drug_not_resolved" — because no drug name was found, not because anything recognised a symptom question. Recorded, not yet fixed.

The answer layer now has an LLM, and a check that makes "it does not fabricate" measurable rather than promised. Previously rag/answer.py was extractive: it concatenated retrieved chunks. That is why Liều Paracetamol cho người lớn? opened with 5 - 12 tuổi: Trẻ em 12 - 18 tuổi: — raw section text, paediatric doses first, for an adult question.

Generation is now three layers, and only the third is load-bearing:

• Prompt (rag/prompt.py, domain — no SDK): evidence only, figures copied character-for-character, [n] citations required, insufficient evidence is a valid answer. Output shape is pinned by output_config.format, so a malformed envelope is the provider's error, not our parsing problem.
• Verification (rag/grounding.py, pure domain): every numeric token in the generated answer must appear exactly in the evidence, and every [n] must resolve. Citation markers are stripped before number extraction so [2] is never read as the quantity 2.
• Fail-closed (rag/answer.py): ungrounded number, invalid citation, malformed output, provider outage, or the model itself reporting insufficient evidence — every one falls back to the verbatim source text, which was computed first and is therefore always available.

Numbers are compared as strings, and that is the decision worth keeping. No parsing, no normalisation. 1.500 is 1500 under one reading and 1.5 under another; a normaliser that strips separators maps 7,5 and 75 to the same key, scoring a tenfold dose error as a match. Pinned by test_decimal_separators_are_not_interchangeable. The same rule refuses 2 g → 2000 mg: arithmetically right, but unit conversion is where dosing errors live, so it is refused rather than interpreted.

Quarantined tables and formulas are never generated over. VERIFY_PDF returns before generation — those are precisely the blocks whose numbers were not reliably reconstructed, so rephrasing them is the one case where fluency could invent a dose. This keeps ADR 0006's contract intact.

Provider chosen on the owner's instruction: AWS Bedrock + Claude. adapters/bedrock_claude.py is the only module naming the anthropic SDK, imported lazily. Two provider facts taken from the Anthropic API reference today, not from memory: Bedrock model ids carry an anthropic. prefix (anthropic.claude-opus-5), and the Messages-API path on Bedrock is AnthropicBedrockMantle, not the legacy bedrock-runtime InvokeModel route the embedding adapter uses. A stop_reason: "refusal" is a successful HTTP response with no usable content, so it is routed to the extractive fallback rather than allowed to raise on content[0].

This adapter has never been run against Bedrock. Cloud access is still revoked and no IAM change was made today. StubAnswerGenerator exercises the entire path — prompt build, schema parse, grounding check, fallback — with no cloud call, and that is what the end-to-end run below used.

Observability, because a dashboard is a better answer than a slide. rag/metrics.py defines the counters in the domain; adapters/prometheus.py is the only module naming prometheus_client, imported lazily; /metrics returns 404 rather than an empty 200 when metrics are off, so a scrape cannot succeed silently with no samples. The headline counter is duocthu_generation_rejected_total{reason="ungrounded_number"} — the measured form of the no-fabrication claim. A mismatched label drops the sample instead of raising: metrics must not be able to break a clinical answer.

infra/docker/ gains Prometheus and Grafana with a provisioned datasource and dashboard. Not yet verified running — the image pull was still in progress when this was written.

A section was being served scrambled, and only using the UI found it. liều dùng paracetamol opened mid-sentence on 5 - 12 tuổi: and buried Liều lượng: Người lớn: seven hundred words down. find_by_section returned whatever order Qdrant scrolled, and point ids are uuid5(chunk_id), so PARACETAMOL's five dosing parts came back 3, 4, 1, 2, 0 — verified by scrolling the real collection, not inferred. part_index was in the payload all along and simply never used. Now sorted by it; a part missing the field sorts last rather than being dropped, because a silently shortened dose list is worse than an unordered one. Pinned by tests/test_section_order.py, including the exact 3,4,1,2,0 case. This is a clinical defect, not a cosmetic one: a reader who stops partway through stops in the middle of a different population's dose. Every section-routed answer given before today — including the 16/16 golden result — was assembled in this scrambled order; retrieval picked the right chunks, so the measurement stands, but no statement about how those answers read survives it.

Conversational reasoning RAG: designed in ADR 0007, domain layer built. rag/conversation.py carries Focus (drug, section, population, verbosity, each stamped with the turn that set it) and the recent-turn window; rag/reasoning.py is the bounded loop. Both are pure domain and run with no provider, which is the point: which drug is this still about must be deterministic, not inferred.

Three rules make inheritance safe in a formulary, each pinned by a test: an explicitly named drug always beats context; focus older than six turns is dropped rather than carried, because a stale drug is a wrong-drug answer, not context; and any answer built on an inherited drug must name it.

The loop's uncertainty signal is not a model confidence score. It is the resolver states that already existed and previously dead-ended into abstain — ambiguous drug, unresolved attribute, multi-attribute question — which now produce a clarifying question. Deterministic, testable, and explainable to a reviewer in a way that "the model felt 0.73 sure" is not. A clarify signal short-circuits before any budget is spent, verified by asserting the budget is untouched and neither retriever nor generator was called.

Budgets are decremented before the call they pay for, so exhaustion degrades to the best answer so far. A retrieval round is bought only by a named gap with a genuinely new query: test_an_unnamed_gap_does_not_buy_a_round and test_a_refinement_that_changes_nothing_stops_the_loop are the guards against a loop that spins on a feeling or re-issues the same query.

Golden Dataset/golden_multiturn_v1.csv is new — 8 conversations, 19 turns, 6 of them inheritance-dependent. The existing golden file is single-turn by construction and can measure none of this. Includes the adversarial turns: a follow-up after a refused fake drug (must not borrow a drug from elsewhere), and a follow-up after a symptom question (must not inherit treatment intent).

Not yet wired: the loop is not called by GroundedAnswerService or the router, there is no PostgresConversationStore, and no evaluation run over the multi-turn file has been performed — so no claim is made that history or the loop improves answers. The design states how that will be measured; it has not been measured.

LangChain was considered and rejected. The repo already has the ports and adapters LangChain would supply, retrieval is already measured, and the guardrail is already domain code. Adopting it a week before a review would rewrite the working part for no measured capability gain.

Verification actually run: ai-service 56 passed, 3 skipped (37 + 3 before, +19); ingestion 296 passed, checked for regression, unchanged; duocthu_v1 holds 15,100 points at 1024-dim Cosine, matching the manifest; live service against the real collection answered three clinical questions with citations and abstained on six of the seven safety probes; /metrics scraped and returned duocthu_generation_served_total 2.0 and duocthu_abstention_total{reason="drug_not_resolved"} 1.0.

Not established, and load-bearing for the demo: apps/web is still entirely mocked — packages/api-client/src/sendChatMessage.ts:8 returns buildMockResponse(content) and the whole frontend contains no HTTP call to the backend, so the working API and the working UI are not connected; api-gateway, chat-service and auth-service hold 0 source files; the Bedrock generator has never been invoked; intent is still supplied by the caller, so the recommendation gate depends on the client declaring it honestly; and the Prometheus/Grafana stack has not been seen running.

2026-08-04 (evening) — Section routing: contraindication retrieval goes from 0.05 to 1.00, at zero cloud cost

The retrieval defect measured earlier today is fixed by routing rather than by embedding. No cloud call was made and nothing was re-embedded — Bedrock access is still revoked.

The change. A question that names its own attribute does not need similarity to guess which section answers it. rag/sections.py maps the question to a section_key; QdrantRetriever.find_by_section then filters on (drug_id, section_key) and returns every part of that section as a scroll, not a top-k. RetrievalService takes that route when it resolves and falls back to similarity otherwise.

Two rules carry the safety. Longest phrase wins: "chống chỉ định" and "chỉ định" differ by one prefix word and mean opposite things, so every phrase is sorted by length and the longer is tested first — the same rule keeps "quá liều" from being read as "liều" and "hướng dẫn xử trí ADR" from being read as "tác dụng phụ". No match is not a guess: an unrecognised question returns None and falls back rather than picking a section it is unsure of.

Measured against the real duocthu_v1 collection, no embedding involved:

	similarity (measured this afternoon)	section routing
hit@1, 160 generated cases	0.544	1.000
chong_chi_dinh	0.05	1.00
misroutes / empty / leaked sections	—	0 / 0 / 0

The generated 160 flattered it, and testing on human-written questions said so. Those questions use the phrasings the table was built from, so 160/160 is partly circular. Run against the 16 single-drug questions humans actually wrote in Golden Dataset/golden_e2e_v1.csv, the first version scored 10/16. The six failures were two gaps: four questions say just "Liều Metformin cho người lớn?" — bare "liều", which the table lacked — and one says "Bà bầu", a colloquial phrasing for pregnancy. Adding those phrases (no code change, which is what the open/closed table is for) took it to 16/16 while the confusable pairs still resolve correctly; bare "liều" is safe only because "quá liều" is longer and tested first, and there is a regression test pinning exactly that.

A circular import was found and fixed properly rather than worked around. service -> sections -> routing -> service, because normalize_name lived in routing.py. It is a text utility with no knowledge of drugs or sections, so it moved to rag/text.py; routing.py re-exports it so existing imports keep working.

Also wired, and still unproven: BedrockCohereQueryEmbedder replaces the SHA-256 hash embedder for the similarity fallback path. It has been import-checked only — never run against Bedrock — so the fallback path remains unverified end to end. The section route does not depend on it.

Verification actually run: ai-service 37 passed, 3 skipped (22 before, +15); ingestion 296 passed (unchanged, checked for regression); ruff --select F,E9,B,ARG over rag/, adapters/, bootstrap.py, config.py and tests/ — all checks passed; section-route evaluation against the live collection 160/160; human-written golden questions 16/16.

Not established: multi-attribute questions ("liều dùng và chống chỉ định") pick the longest phrase, which is deterministic but arbitrary; phrase coverage beyond these 16 human questions is unmeasured; and none of this speaks to whether the retrieved text is clinically correct.

2026-08-04 (afternoon) — First real embeddings exist; retrieval measured at 54% and the cause is not what the small sample said

The corpus is embedded for the first time. Bedrock IAM was opened on the owner's explicit instruction, all 15,100 chunks were embedded with cohere.embed-v4:0, loaded into Qdrant, and cloud access was then revoked and proven revoked before the owner's 17:00 deadline. Measured spend ~$0.49 of a personal $138 budget.

Gate results. 15,100/15,100 embedded; 15,100 points in duocthu_v1 over 59 batches; collection point count 15,100 — count gate PASS. Manifest records cohere.embed-v4:0, 1024 dimensions, Cosine, corpus SHA 04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c. Corpus SHA was re-verified against the morning audit before spending: identical, and identical to the post-lint copy, so the 12:05 chunker.py edit did not change output.

Both providers were probed live before choosing. Titan v2 and Cohere v4 each returned 1024 dimensions with a measured L2 norm of 1.000000. That settles a question left open since 2026-08-03: Cohere's normalized field was None because AWS's docs never state it. It is now measured. Cohere was chosen on two measured grounds — the corpus is Vietnamese and Cohere is explicitly multilingual, and bedrock_cohere.py batches 96 texts per request while bedrock_titan.py sends one, which at a measured 2.3s per call is ~9.6 hours versus minutes. The $0.41 price difference did not drive it.

The retrieval number, and a correction to a claim made earlier the same day. A 160-case evaluation (20 per section, 8 sections, questions generated from the corpus so labels are structural) measured hit@1 0.544, hit@3 0.663, hit@5 0.738. Per section:

section	hit@1
chong_chi_dinh	0.05 (1/20)
chi_dinh	0.30
tac_dung_khong_mong_muon	0.40
lieu_luong_va_cach_dung	0.60
qua_lieu_va_xu_tri	0.65
than_trong	0.65
tuong_tac_thuoc	0.80
thoi_ky_mang_thai	0.90

An earlier 15-case run gave a similar headline (0.533) but led to the wrong diagnosis: four of its seven failures were contraindication questions answered with indications, so the cause was reported as embedding weakness at negation. At 160 cases that pair accounts for only 3 confusions. The dominant mechanism is different and larger: duoc_ly_va_co_che_tac_dung absorbs questions from every other section — 10 from adverse effects, 8 from contraindications, 7 from dosage, 5 from indications. It is the largest section (1,896 chunks) and describes the drug in general terms, so it sits close to almost any question about that drug. This is the small-sample failure mode CLAUDE.md warns about, reproduced on this project.

Re-embedding cannot fix this, and the capability to fix it already exists. Verified by reading the code, not assumed: apps/ai-service/adapters/qdrant.py search() filters on drug_id only and lets vector similarity choose the chunk; rag/routing.py resolves drug and intent but not section; and find_by_payload — the "return the whole section" method in ingestion/load/ — is never called anywhere in apps/ai-service. Attribute questions therefore depend on similarity picking the right section, which is what measures 54%. The fix is to resolve the attribute to a section_key and retrieve that section whole; ATTRIBUTE_TO_SECTION already exists in embed/benchmark_local.py.

A silent-failure hazard found and closed. apps/ai-service embedded queries with LocalHashQueryEmbedder — SHA-256 of tokens, explicitly plumbing only — while the collection now holds Cohere vectors. Querying across those two spaces returns hits and raises nothing; the results are simply meaningless. BedrockCohereQueryEmbedder was added and wired behind EMBEDDING_PROVIDER=cohere-v4. It has only been import-checked — never run against Bedrock, because cloud access was revoked first, as instructed.

Two operational lessons, both paid for. bedrock_runtime.py set no boto3 timeout, so a single throttled response held a socket open for over five minutes and stalled the whole run; connect_timeout=10, read_timeout=60 plus standard retries fixed it. Then the first full run still died at ~14,600/15,100 because the retry backoff (2s, 4s) was far shorter than a per-minute token quota needs. The disk cache made that survivable: the resumed run recorded 14,977 cache hits and 123 misses, so only 123 vectors were paid for twice — zero, in fact, since the first run's work was already saved.

Cloud shutdown, verified rather than asserted. Both policies detached and deleted; InvokeModel and ListFoundationModels both now return AccessDeniedException. No EC2 instance, no EBS volume, and — because the policy never granted CreateProvisionedModelThroughput — no way for this identity to create the one Bedrock resource that bills hourly.

Not established: retrieval quality is not acceptable for clinical use, no clinician-authored release gate exists, the generated evaluation questions use template phrasing rather than real clinical language, and no LLM answer layer has ever run against real evidence.

2026-08-04 — Chunk schema v4 passes the embedding-readiness gate

Reviewed the live Claude coordination and its last changes before editing. The delivery plan was objectively stale: it still described schema v2/15,076 chunks, empty embed/load/API modules, embedding before content-safety gates, and allowed unverified inferred table headers as retrieval text. The plan and ADR 0004/0006 now put content safety, exact provenance, fail-closed schema validation and local pseudo-vector smoke tests before any provider call. Bedrock remains benchmark- only and requires separate owner approval for any paid/full-corpus run.

Implemented schema v4 and regenerated the canonical chunk artifact. Retrieval text may repeat route/population labels so continuation chunks remain safe in isolation; contiguous source_text remains byte-reassemblable and drives exact physical/printed page provenance. context_labels records retrieval-only prefixes. All 151 unverified table/formula descriptors embargo header_row and cell-like column text. Attachments now carry physical page, printed page, block_id, bbox and optional crop, and those region references survive the Qdrant adapter and RAG citation response. The loader accepts exactly schema v4, rejects booleans/non-integers/out-of-range pages, and keeps the normalized LF/CRLF-stable corpus identity.

Canonical artifact measured after regeneration:

• 15,100 chunks: 14,949 prose + 151 block descriptors;
• 4,105,382 cl100k_base tokens; 0 chunks above the 800-token ceiling;
• all chunk-ready gates pass: exact provenance, source uniqueness, reassembly, attachment coverage, descriptor embargo and schema checks all have 0 failures; 151 descriptors match 151 quarantined blocks;
• raw file SHA-256: 8dfae08ae6d9222089c5cdb4207a064fe67989f10f7552b555af0aef6331d9a1;
• normalized corpus SHA-256 used by the Qdrant manifest: 04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c.

Verification actually run:

• ingestion: 292 passed; focused post-lint patch: 26 passed;
• AI service with RUN_INTEGRATION=1: 25 passed, including real local Qdrant, PostgreSQL and FastAPI round-trips;
• full canonical local smoke with deterministic 4D pseudo-vectors: first and second loads both upserted 15,100 records and both held exactly 15,100 points; manifest hash matched; data and sidecar test collections were removed and Qdrant returned to 0 collections;
• Ruff F,E9,B,ARG on the files changed for this gate: clean; git diff --check: clean (Git only reported Windows LF/CRLF conversion warnings).

Conclusion: the canonical corpus is technically READY TO EMBED, meaning its input/schema/provenance/load plumbing meets the measured gates. This does not authorize a provider call, does not establish retrieval quality for any model, and does not prove whole-book medical accuracy. Human-reviewed clinical eval, table reconstruction, and recall for borderless tables/bar-less formulas remain outside what these gates prove.

2026-08-04 — Real local datastore plumbing, guarded RAG API, and printed-page citations

Read the live Claude Code process and coordination before editing. Claude owned ingestion/load/ and embed/cache.py; it completed the disk cache, Qdrant port/adapter, idempotent UUID5 upsert, payload indexes and corpus-SHA manifest. Its real local Qdrant scale check loaded all 15,066 chunk records twice with 1,024-dimensional deterministic pseudo-vectors and held the point count at 15,066. Those vectors are not embeddings and establish no retrieval-quality claim. No Bedrock call, IAM change, or cloud spend occurred.

Built the first runnable apps/ai-service boundary: FastAPI /health and POST /v1/rag/query, a Qdrant retriever filtered by resolved drug_id, a PostgreSQL trace repository plus migration, structured human/non-human scope and fact/recommendation intent gates, parent hydration, quarantine handling, and an extractive answer layer. The answer layer refuses evidence that has only a physical page; citations expose only the printed folio, chunk id and optional source crop. Quarantined tables/formulas return a PDF-verification warning and never auto-extract numeric content.

Fixed the missing provenance at its source. Chunk schema is now v3 and cli chunk reads the real folio map from the 1,668-page PDF. It refuses a monograph whose physical range cannot be mapped, and chunk-ready has a new chunk_without_printed_page_range gate. Regenerated scope: 684 monographs, 15,066 chunks (14,915 prose + 151 descriptors), zero oversized, and 15,066/15,066 records with a two-value printed-page range. New artifact SHA: e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38.

Verification actually run:

• python -m pytest -q and Ruff over ingestion/: 258 passed, lint clean;
• python -m ingestion.cli chunk-ready: every gate passed, including printed page range 0/0 failures;
• ai-service with RUN_INTEGRATION=1: 22 passed, including a real chunk round-trip through local Qdrant, PostgreSQL migration/insert/read-back, and a full FastAPI → Qdrant → guarded citation → PostgreSQL trace round-trip;
• local Docker services: PostgreSQL 16 and Qdrant 1.18.3 reachable; integration collections were UUID-scoped and removed after tests;
• ArgoCD local: namespace, CRD and seven controller pods are running; the existing unrelated guestbook lab app is Synced/Healthy with four history entries. This repo's three Application YAML files parse and point to master/the Helm chart, but they are not installed and the chart still has no workload templates, so project sync/rollback was not performed.

Still open: no real embedding exists, no full canonical Qdrant collection can serve semantic search, population_tags are absent, no clinician-authored release-gate cases exist, and the API currently has no production answer/query embedding provider. The local hashing provider is explicitly plumbing-only.

2026-08-04 — Load stage built and proven against a real Qdrant; bbox rounding found

ingestion/load/ was a 0-byte __init__.py. It now holds the vector-store boundary: a VectorStore port, an InMemoryVectorStore that is the reference implementation of its contract, and QdrantVectorStore as the only module that names qdrant_client — imported lazily, the same arrangement that confines boto3 to bedrock_runtime. embed/cache.py was added alongside it.

Three design decisions are worth carrying forward.

The cache key is (model_id, input_kind, text_sha256), not chunk_id as §4.A of the delivery plan proposed. Measured reason: chunks.jsonl holds 15,066 records but only 14,869 distinct texts, so 197 records (1.31%) are repeats that a chunk-keyed cache would pay for twice. The content key also cannot serve a stale vector after an edit — a changed text is a changed digest, so it is a miss.

Point ids are uuid5(chunk_id). A random id would make a re-run append a second copy of a dose and nothing would report an error.

The corpus manifest lives in a <name>__manifest sidecar collection rather than a reserved point inside the data collection, because qdrant_point_count != chunk_count is a v1 gate and a gate needing an "except the manifest" footnote will eventually be read wrong.

Whole-corpus check against a real server. A local Qdrant 1.18.3 was started from infra/docker/docker-compose.yml (local container, no cloud) and all 15,066 real chunk records were loaded with deterministic pseudo-vectors at 1,024 dimensions — a check of the loading mechanism, not embeddings, which still do not exist. Corpus sha256 30d5154273e0959a…. First load: 15,066 points in 59 batches, 14.0s, point-count gate PASS. Second load: still 15,066, so idempotency holds at real scale, not only against the fake store.

That sha is already stale, which is the point. chunks.jsonl was regenerated at 09:53 the same day — chunker.py changed two minutes earlier and every chunk gained printed_page_range, 18,229,918 → 18,753,003 bytes, sha now e474c83790b450d3…. Re-measured on the new artifact: still 15,066 chunks, 0 over the 800-token ceiling (largest exactly 800), all 15,066 carrying printed_page_range, 14,915 prose + 151 block descriptors, 197 duplicate texts (1.31%) unchanged because only a field was added. Suite 258 passed. Had the old corpus been embedded and loaded, then the new one loaded into the same collection, two generations would have mixed with no error at query time — A6 is what refuses that, and it now has a real instance rather than a hypothetical one.

A sampled check passed and was wrong. Comparing 5 payloads gave 5/5 identical. Scrolling the entire collection instead found 86 of 15,066 chunks whose payload did not equal its source record. Classifying every differing leaf: 96 differences, all floats, all inside attachments[].bbox, maximum absolute delta 5.684e-14, and zero non-float differences — every text, id, page number, page range, token count and boolean round-tripped exactly. A PDF point is 1/72 inch, so that delta cannot move a rendered crop. It is pinned by a regression test that fails if the loss reaches another field or grows past 1e-9.

The layer responsible was isolated rather than assumed: the source chunks.jsonl returns the value exactly, our own json.dumps/loads returns it exactly, and Qdrant reached over raw HTTP with no SDK involved returns it one ULP low. Nothing needs re-chunking — a regenerated corpus would carry the identical value and be rounded identically. Qdrant also stores dense vectors as float32, so precision beyond f32 is discarded at load regardless.

Cache format was decided on measurements, not preference: 300 real chunk texts at 1,024 dimensions cost 21,098 bytes/record — ~318 MB per model for the corpus, with a 7.8s offset-index rebuild per open. float32 .npy (62 MB) and base64 float32 in JSONL (~87 MB) were measured and set aside; append-only JSONL survives an interrupted run and stays readable, which outweighs disk at one or two models. Revisit at three (~950 MB). It lands in ingestion/data/processed/, already excluded by .gitignore:34.

A gap in this work, found and closed the same day. Payload indexes were created on drug_id, section_key, atc_codes and chunk_kind and reported as done — but VectorStore had no query method, so all that was really proven is that create_payload_index returns without raising. Filtered retrieval is the whole of mode A. find_by_payload now exists on the port and both stores, as a scroll rather than a search: it returns every match, never a top-k, because "return the whole section" is the plan's non-negotiable — two of five contraindications reads as a complete list. Verified on a real server: all five parts returned with no leak from the PANTOPRAZOL/OMEPRAZOL pair that measures cosine 1.000 on contraindications; a deliberately 300-part section (above the 256 scroll page) comes back whole so paging cannot truncate; and a real multi-part section from chunks.jsonl round-trips to exactly its own chunk ids.

Tests: 255 passed with Qdrant running (206 before this work, +49); 247 passed, 8 skipped with it stopped, so an offline machine and CI see skips rather than failures. After the mode A work and the other worktree's cli.py fix the suite stands at 268 passed and ruff --select F,E9,B,ARG reports no findings at all across ingestion/.

Still missing, and deliberately so: printed_page_range and population_tags are not in the payload (open questions to Codex in coordination/CLAUDE_TASK_2026-08-04.md); cli embed / cli load are not wired because cli.py is Codex's; and no real embedding vector has ever been produced — every vector the load path has carried was synthetic. The Bedrock request shapes remain documentation-derived and unproven.

Measured cost: $0. No Bedrock call, no IAM change, no cloud resource.

2026-08-03 — Bedrock embedding boundary built; IAM diagnosed, not yet opened

ingestion/embed/ was an empty __init__.py. It now holds the provider boundary the model benchmark needs: an EmbeddingProvider ABC that owns input validation, request-size batching and timing, and three adapters behind it — amazon.titan-embed-text-v2:0, cohere.embed-v4:0, and BAAI/bge-m3 as the zero-cost local control. boto3 is named in exactly one module and imported lazily, so the package imports and the whole suite runs with no AWS account.

Two design points are worth carrying forward. input_kind is a required argument, not a keyword: Cohere embeds corpus records and queries into different subspaces, and sending search_document for a query raises no error — recall just drops. And normalized is three-valued. Titan is asked to normalize and says so; the Bedrock docs never state whether Cohere's float vectors are unit-length, so that field stays None instead of guessing, and embed.probe prints a measured L2 norm to settle it on the first live call.

The AWS side is diagnosed and stuck. ai-lab-user has no inline and no attached user policy; its one group (AI-Lab-Group) grants EC2, IAM, ELB and VPC full access and nothing else. There is no bedrock:* grant anywhere on the identity — confirmed by running both list-foundation-models and invoke-model and reading the two AccessDeniedException messages. Two least-privilege policies are drafted in infra/aws/iam/ but deliberately not applied: that identity carries IAMFullAccess and could attach them itself, which is exactly why it was left to a human.

Consequence: every request-body shape in the two Bedrock adapters is derived from the AWS user guide (read today) and has never been accepted by the service. That is unproven, not verified. Tests: 22 new, all with a stub invoker and zero network; 203 passed overall, up from 181. Lint clean on every file added (--select F,E9,B,ARG); the one remaining finding is a pre-existing cli.py import owned by the other worktree.

Measured cost so far: $0. Nothing was embedded, nothing reached Qdrant.

2026-08-03 — Exact hard-10 gate and all-block table chunking experiment

Extended the isolated table/formula sandbox beyond the 100-page sample. An exact ten-block risk gate covered four cross-page pairs, a merged header, a fragmented fraction bar, and the bar-less ADENOSIN formula; all ten source crops were visually checked. The full run then processed all 151 canonical blocks: 141 physical tables, ten formulas, 133 logical table parents, 669 row children, and seven cross-page logical tables.

Full-scope visual inspection exposed a continuation bug: FAMCICLOVIR p647 and INSULIN p811 repeat their column headers, while other continuation pages start directly with data. The linker now distinguishes these cases; repeated headers are not emitted as data, and INSULIN's changed Phối hợp first-column meaning is preserved. Both branches have regressions.

The expanded, source-derived retrieval suite contains 2,436 cases. With drug and table/formula lane resolved before ranking, deterministic hybrid character TF-IDF measured 94.42% Recall@1, 99.79% Recall@5, and 96.90% MRR. Row questions were 94.82% / 100%; formula questions 100% / 100%. Five ambiguous whole-table questions fell below top five because the same drug owns several near-identical tables; production must clarify or route using an additional table anchor. Neural MiniLM is now opt-in and excluded from the default parsing gate.

Measured chunk design: table-parent tokens min/median/p90/p95/max = 66/188/441/678/1,893; only four of 133 parents exceed 800. Row children are 75-token median, 172 p95, 471 max. Keep every logical parent intact, index both parent and header-aware rows, never split a row, and hydrate row hits to the complete parent/source pages. Final checks: 181 tests passed, readiness 20/20, lint clean.

---

2026-08-03 — 100-page table/formula reconstruction and RAG sandbox

Built an isolated experiment under ingestion/scratch/rag-table-pilot without writing sandbox representations into the canonical corpus. The risk-stratified 100-page run reconstructed 120 tables and 10 formula regions, rendered and manually inspected all 130 crops, and linked four tables continued across page pairs 132-133, 646-647, 825-826, and 1373-1374.

The retrieval router fixes the drug and data lane before vector ranking. On 461 source-derived queries, hybrid row+whole character TF-IDF reached 92.62% Recall@1, 98.70% Recall@5, and 95.04% MRR. Cached English-oriented MiniLM was worse (88.29% / 97.18% / 91.92%). Eighteen row-hit answer previews all hydrated to the complete parent Markdown table; eight included both pages of a continued table. A narrow deterministic interval probe passed 172/172 generated cases; this is a mechanics check, not clinical ground truth.

Visual review exposed one canonical defect: ADENOSIN p147's bar-less printed formula region ended after its numerator and omitted Nồng độ adenosin (3 mg/ml). The bar-less band now extends 31pt below its synthetic anchor, capturing the denominator but stopping before Ví dụ:; a regression pins that boundary. Canonical artifacts were regenerated after the fix: 684 monographs, 11,974 sections, 15,066 chunks, 151 descriptors, 0 unassigned spans, all 20 readiness gates passing, 180 tests passed, and lint clean.

Decision: JSON grid + Markdown answer view, row and whole-table retrieval, and mandatory parent hydration are viable for the next stage. This remains a retrieval experiment, not production clinical approval; merged-cell semantics, unit/multi-axis reasoning, Vietnamese embedding comparison, borderless/bar-less recall, clinician-authored evals, and final expert review remain open.

---

2026-08-03 — Whole-corpus parser repair after manual baseline audit

Implemented and re-ran the parser over all 1,668 pages after manually reading the high-risk baseline outliers. The fixes are structural, with regressions:

• restored the missing THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN boundary (Tên chung quốc tế và mã ATC is its real first anchor), separating pages 1371–1373 from THUỐC PHIỆN - OPIAT - OPIOID;
• require both physical and inferred printed page bounds, so back-index page 1655 can no longer extend ZOLPIDEM's real [1492, 1494] range;
• keep plain label-shaped text as body when it is an adjacent wrapped continuation in the same PDF block (including NADROPARIN's “không phải là chống chỉ định”);
• classify known table cells before headings, putting WARFARIN and IOBITRIDOL dosing tables back under lieu_luong_va_cach_dung;
• added confirmed heading variants for CLORPHENIRAMIN dosage forms and tetanus toxoid dosing, and real provenance for combined inline fields;
• made verified formula bands column-aware: NETILMICIN opposite-column prose is retained while AMPICILIN's gutter-adjacent formula stays quarantined;
• visually inspected all 151/151 unique table/formula regions against the rendered PDF; every region is genuinely 2D and remains quarantined;
• emit every physical table/formula region atomically at its first stream occurrence, fixing split/contradictory ownership on CAPECITABIN, IMATINIB, CARBOPLATIN, NETILMICIN, and TRASTUZUMAB;
• route explicit Bảng N. Điều chỉnh liều ... appendices back to dosage even when the book prints them after Tên thương mại (CAPECITABIN p309);
• added readiness gates for every individual section part's source-span IDs and duplicate physical-region IDs.

Final regenerated artifacts and evidence:

check	result
tests	180 passed; lint clean
segmentation	684 monographs, 11,974 sections, 8,213,036 prose chars
back-index validation	96.2% recall (678/705), 99.1% precision
quarantined regions	151 blocks / 151 unique IDs, all visually checked
chunks	15,066 (14,915 prose + 151 block descriptors), 0 over 800 tokens
chunk readiness	20/20 PASS (including duplicate-region prevention)
coverage	252,799 spans, 0 unassigned across all 1,668 pages
residual ink	3,931 classified regions, 0 unclassified across all pages

Canonical ingestion/data/processed/{monographs,chunks,coverage_ledger} were regenerated. Remaining limits: no whole-document human-reviewed clinical ground truth, no row/column reconstruction for quarantined tables, and unknown recall for borderless tables/bar-less formulas. This is ready for retrieval experiments, not a claim of production clinical approval.

---

Chronological record of work done on this project, newest entry on top. The goal is continuity across sessions: if a work session ends unexpectedly (context/token limit, interruption), whoever picks this up next — human or Claude — should be able to read the latest entry and know exactly what's done and what's next, without having to reconstruct it from git history.

Convention: add a new entry at the top before ending a session whenever meaningful progress was made, and proactively the moment it looks like the session might run out of context/tokens mid-task — don't wait until the very end if that risk is showing.

---

2026-08-03 — Independent re-verification, a redundant rule in my own uncommitted fix, a provenance defect, and the measurements a retrieval design has to be built on

No code was changed in this session: the four files from the previous round are still uncommitted and under external review (Codex). Everything below is measurement, and the numbers live nowhere else — the investigation scripts were deleted per the repo rule, so this entry is the record.

1. Re-verified the whole tree from scratch

check	command	result
tests	python -m pytest -q	164 passed, 39.31s
lint	ruff check --select F,E9,B,ARG .	clean
gates	cli chunk-ready	18/18 PASS; 683 monographs, 11,966 sections, 8,212,880 chars, 167 quarantined blocks
recall/precision	cli validate --pdf …	683 detected, 705 ground truth, 96.0% (677/705) / 99.1%
span ledger	cli coverage --pdf … (re-run)	252,799 spans after merge, 9,398,772 chars, unassigned 0
reproducibility	cli run → sha256	byte-identical to monographs.jsonl (84f41d96…)
reproducibility	cli chunk → sha256	byte-identical to chunks.jsonl (63472db4…); 15,076 chunks (14,909 prose + 167 descriptors), 0 oversized, 4,072,725 tokens (cl100k_base)

Recall rose 92.9% → 96.0% because of the uncommitted back-index rejoin, and the mechanism is the denominator: 725 → 705 ground-truth entries once wrapped fragments stop counting as entries. The detector did not improve.

Not re-run: cli residual-ink. residual_ink.json is dated 2026-08-01 12:03, before the 17:36 assembler edits. Its stored contents (3,931 regions, no unclassified kind) are last session's numbers, not this session's.

Doc drift found: docs/verification-strategy.md quotes 252,733 spans / 177,679 normalized_text / 12,764 heading; measured today 252,799 / 177,754 / 12,752. The unassigned = 0 conclusion still holds.

2. The x0 geometry change in the uncommitted diff is redundant

Assembled the whole book four times with the two new rules toggled:

variant	monographs	sections	chars
current (x0 + italic)	683	11,966	8,212,880
old x1 rule + italic	683	11,966	8,212,880 — 0 differences of any kind
x0, no italic	683	11,966	8,212,844 (3 sections differ)
x1, no italic (= bc01782)	683	11,966	8,212,780 (9 sections differ, 11,014 char delta)

The italic rule alone recovers all 9 sections (CEFAZOLIN dosing 1,062 → 5,620 chars; CALCI LACTAT than_trong 582 → 1,504, tuong_tac_thuoc 2,346 → 1,439). The x0 rule alone recovers 6 of 9 and adds nothing on top of the italic rule.

Worse, the justification is wrong: the real NEVIRAPIN span on physical page 1045 is TimesNewRomanPS-ItalicMT (verified by reading the span's font), so the italic rule is what fixes that page — not the 0.01pt overlap the code comment and the new test's docstring credit. The test itself is valid but pins the geometric rule only, because the _span() fixture helper never produces an italic font. Either keep the x0 rule as defence-in-depth with an honest comment, or revert it — but the current comment overstates it.

3. source_page_range is wrong for 13 of 683 monographs

Section-level provenance (parts) is correct everywhere; the monograph-level page range is not. 12 monographs overshoot by +1 page; ZOLPIDEM declares [1492, 1655] while every one of its sections comes from 1492-1494 — a 164-page claim reaching into the back index.

Root cause for ZOLPIDEM, confirmed: physical page 1655 (printed 1656, back matter) carries a bold span reading exactly Tương tác thuốc, which _classify emits as a _SectionEvent, and the _SectionEvent branch at segment/assembler.py:496 updates source_page_range[1] with no in_monograph_range guard — unlike the _TextEvent branch at line 511. Verified that 0 spans past physical 1495 pass in_monograph_range, so no text was contaminated and empty_section is still 0. The defect is confined to one provenance field.

The +1 cause is not isolated — it is not lifted tables (all 12 have tables: []); the likely candidate is a next-page boilerplate span bumping the range before being excluded, but that was not measured.

4. Corpus profile — what a retrieval design actually has to work with

• 13 of 19 fields have p90 < 1,500 chars, i.e. the whole section fits one chunk. Only four routinely need splitting: duoc_ly (p90 4,939, max 14,099), lieu_luong (4,873 / 14,197), than_trong (2,419), tuong_tac (2,147). Confirms ADR 0004 on the cleaned corpus.
• ATC: 668/683 (97.8%) carry ≥1 code, 171 (25.0%) carry more than one, max 20, 1,043 distinct codes.
• ten_thuong_mai present in 492 (72%) monographs.
• The back index holds 344 X - xem Y lines — brand → generic aliases — which parse_back_index currently discards wholesale (correct for validation, but this is the highest-value retrieval asset in the book, because clinicians type brand names).
• 401 xem [thêm] mục/chuyên luận phrases across 261 monographs; a chunk containing one is useless retrieved alone.
• Dosing population markers: Trẻ em 53%, Người lớn 51%, Người cao tuổi 15%, Trẻ sơ sinh 8%, Suy thận 8%, Suy gan 6% of 682 dosing sections — real sub-section boundaries, better split points than token windows.
• 167 quarantined blocks, 129 (77%) inside lieu_luong_va_cach_dung — the most dangerous field is the one the tables were lifted out of.

5. Cross-drug confusability — the number that decides the architecture

First hypothesis (much repeated boilerplate across drugs) was refuted: only 171 of 11,966 sections share exact text with another drug (1.4%), and the six heavy clinical fields are 100% distinct.

Then measured, per field, each drug's TF-IDF cosine against its nearest other drug. This is a lexical proxy, not an embedding measure — it bounds the problem from one side only.

field	median	p90	p99	max	drugs with NN > 0.7
lieu_luong_va_cach_dung	0.314	0.455	0.631	0.836	4 (0.6%)
tuong_tac_thuoc	0.284	0.461	0.870	0.984	18 (2.8%)
tac_dung_khong_mong_muon	0.300	0.437	0.856	1.000	13 (1.9%)
chi_dinh	0.408	0.637	0.885	0.924	31 (4.5%)
chong_chi_dinh	0.346	0.633	0.898	1.000	37 (5.4%)

Named pairs: PANTOPRAZOL ↔ OMEPRAZOL (contraindications 1.000, indications 0.913) · BENZATHIN PENICILIN G ↔ PHENOXYMETHYLPENICILIN (contraindications 1.000) · DIGOXIN ↔ DIGITOXIN (0.891 / 0.911) · NATRI NITRIT ↔ NATRI THIOSULFAT (dosing 0.631 — two different steps of the same cyanide-antidote protocol) · IOBITRIDOL ↔ ACID IOXAGLIC (0.984) · ESTRIOL ↔ ESTRON · GLICLAZID ↔ GLIMEPIRID · NAPHAZOLIN ↔ OXYMETAZOLIN.

Name layer: 19 drug names are a substring of another drug name (CLOROTHIAZID in HYDROCLOROTHIAZID, EPHEDRIN in PSEUDOEPHEDRIN, LORATADIN in DESLORATADIN, ATROPIN in HOMATROPIN HYDROBROMID — all genuinely different drugs), and 106 of 683 names share a 6-character prefix across 38 clusters.

Conclusion drawn from this, for the retrieval design: vector similarity must never be allowed to choose the drug — only which passage within an already-resolved drug. The dangerous confusions are concentrated in a small, enumerable set of same-class pairs, which is exactly the population this project's verification strategy says to census rather than sample.

Not done yet / next up

Sequenced in docs/v1-delivery-plan.md (written this session): a two-week plan to a running v1, scoped down to two deployables (web + ai-service) because the four NestJS services measure 0 .ts files each. The items below are the ones that plan depends on.

• The confusable-pair census must become a committed fixture produced by production code (ingestion/validation/), not a deleted scratch script. Until then these numbers are only in this entry.
• ADR 0007 (retrieval architecture) not written. Proposed content: vectors never pick the drug; the unit returned to the LLM is the complete section (enabled by section_not_reassemblable_from_chunks = 0, because a partial contraindication list reads as "no contraindication"); and eval split in two — routing correctness (ground truth derivable from the corpus itself, 683 × 19 pairs, no human needed) versus content correctness (requires a clinician; cannot be self-generated without fabricating evidence).
• Entity/alias layer (683 canonical names + 344 back-index aliases + 492 ten_thuong_mai + 1,043 ATC codes) — zero-regret, needed by every architecture, must use longest-exact-match because of the 19 substring traps.
• residual-ink re-run; verification-strategy.md numbers re-synced; regression test for parse_back_index (still has none); the source_page_range guard; the x0-rule comment decision.
• Open question for the user, not a technical one: this is the 2018 edition; the 3rd edition (2022) exists. For a document with legal force over prescribing, staying on 2018 should be a deliberate decision, and it makes edition-independence a real requirement for the pipeline.
• Still untouched: embed/, load/, Qdrant, ai-service, and the general chapters (printed 37-98) and appendices (printed 1497-1528), which remain outside the corpus entirely.

2026-08-01 (cont'd, 7) — "still errors?" — yes: two more real content-loss bugs, both in dosing sections

Asked whether errors remained after the previous round, the honest answer was that this session has found real defects every time it looked one level deeper. It looked again, and found two more.

1. Chunks ended on a bare population label, with the dose in the next chunk. split_sentences treats : as a sentence boundary and _OPENS_SENTENCE accepts a digit, so "Người lớn: 500 mg mỗi 8 giờ." splits after the colon. When the packer flushed at that point, the chunk ended on the label. Measured: 38 prose chunks, e.g. AMOXICILIN's ending on a Lyme indication followed by a bare Người lớn:. Retrieval on that chunk returns a population with no dose; on the next, a dose with no population. Outlier item 17 counted population markers on 1,121 of ~1,400 monograph pages, so this is the common shape, not an edge case. The packer now carries trailing label atoms into the next part instead of flushing on them: 38 → 2, and chunks ending on any colon 721 → 19.

2. A section name printed mid-line was swallowed as a heading — real text loss, in dosing sections. Chasing the last 2 of those 38 showed the defect was not in chunking at all. CISPLATIN (physical page 402) prints Suy thận: Chống chỉ định. inside liều lượng và cách dùng; the second half is itself a section name, so it was matched as a heading. The result: the renal-impairment contraindication disappeared from the dosing text and the section ended on a bare Suy thận:. ISOPRENALIN had the same shape. Same family as the FLUOROURACIL bug fixed earlier today, but that rule only covered a label directly under a heading and could not see this one.

Fixed geometrically: a real section heading opens its line, so a non-bold section name with another span printed to its left is body text. "To the left" is checked properly — same page/block/line and previous.x1 <= span.x0 — because the synthetic test fixtures place every span at identical coordinates, and a looser check passed on real data while breaking the AMITRIPTYLIN inline-value case.

Verified after the fix: CISPLATIN's dosing section contains Suy thận: Chống chỉ định. again, ISOPRENALIN's Trẻ em: is followed by its doses, and chong_chi_dinh is no longer polluted. Monograph and section counts unchanged at 683 / 11,966 — nothing was traded away for the recovery.

State: 18/18 gates pass, 163 tests (was 161), ruff F/E9/B/ARG clean, 15,077 chunks with 0 over the ceiling, 8,212,780 section characters.

Standing conclusion, worth writing down: every round of "is it clean now?" this session has ended with real defects found — five in the previous round, two in this one, and four of the previous five were in code written the same day. The gates and tests prove what those instruments can see. They do not prove the corpus is correct, and the largest unmeasured area is unchanged: content accuracy against the source, with no human-reviewed ground truth for 8.2M characters.

2026-08-01 (cont'd, 6) — Bug hunt after declaring "clean": the token count was wrong by 2x, 14.7% of chunks were over the ceiling, and two stage boundaries measured different pipelines

I had just reported the tree as clean. It was not. Going looking properly found five real defects, four of them in code written earlier the same day.

1. estimate_tokens was wrong by a factor of two, and the number it produced was reported. ADR 0004 sized chunks with len(text) // 4, described honestly as an estimate. Measured against cl100k_base on the real corpus:

estimate (chars/4)	2,115,427 tokens
real tokenizer	4,093,440 tokens
real/estimate	median 1.95, p95 2.50, max 6.0
oversized by estimate	0
oversized in fact	1,884 of 12,838 = 14.7%, largest 1,645 tokens

Vietnamese diacritics cost several byte-pair tokens each. "0 oversized" was reassuring and false. chunk/tokens.py now counts with the real tokenizer, injected so the chunking logic stays testable without it, with a fallback of chars/2 that errs small rather than large.

2. The packer could exceed the ceiling on its own. Two causes, both measured on VORICONAZOL's tương tác thuốc: an atom of 710 tokens was left whole because it was under the 800 ceiling, and the overlap builder added whole atoms until the running total passed the budget, so a 251-token atom produced a 273-token overlap against a 65-token setting. 273 + 710 = 983. Atoms are now split against the 650 target, leaving room for overlap, and the overlap stops before exceeding its budget.

3. An over-long comma list was left as one atom. VORICONAZOL's interaction list is one "sentence" hundreds of drug names long. Truncated by an embedding model it reads as "this drug is not listed" — a false negative in the direction that matters. Split at commas, which is lossless for a list.

After 1-3: 0 chunks over the ceiling, verified by an independent tiktoken re-count of the written file, not by the pipeline's own number. 15,049 chunks (was 12,838 — the rise is real sub-chunking that should have happened all along).

4. cli validate measured a different pipeline than cli run. It used the raw span stream (no transcription repair) and passed no table regions, so recall/precision described a build that is not the one producing the output — the same class of mismatch already fixed for coverage. Now shares _extracted_and_repaired_spans and _region_index. Result after the fix is unchanged at 92.9% / 99.1%.

5. chunk/io.py dropped SectionPart when reading monographs back, so per-part provenance died at the stage boundary — against CLAUDE.md's explicit rule. Now carried: 12,290 parts across 11,966 sections.

Two new gates, and the gate itself was wrong twice before the data was. section_not_reassemblable_from_chunks rebuilds each section from its own chunks by removing the deliberate overlap and compares. First version joined chunk texts with a newline and reported 734 sections missing — the first one it named was present. Second version probed a 60-character head and reported 1, NAPROXEN, where the probe straddled an overlap seam that legitimately repeats text. The working version compares with whitespace removed, because each split seam loses exactly one space to .strip() (measured on ABACAVIR: two single spaces in a 4,232-character section, nothing else). It proves no character of content is lost, reordered or duplicated beyond the intended overlap. 0.

Also fixed: all 8 real lint findings (ruff --select F,E9,B,ARG) — five unused imports and three zip() calls without explicit strict=. The zips were the adjacent-pair idiom and not bugs; strict=False now says so. And the transcription splice could leave a fragment holding only a space, which showed up as two whitespace_only spans; dropped, and proven inert by the sha256 over every section's text being byte-identical before and after (6af13301…).

State after the hunt: 18/18 gates pass (10 corpus + 8 chunk), 161 tests (was 158), ruff F/E9/B/ARG clean, unassigned = 0, cli validate 92.9% / 99.1%, 15,049 chunks with 0 over the ceiling.

2026-08-01 (cont'd, 5) — ADR 0006 implemented: chunks now reference their lifted blocks; chunk/ runs for the first time; 16/16 gates green

Why this was needed, in one line: a chunk of a section whose table had been lifted was grammatical, complete-looking prose with the table absent and nothing marking the absence — silent incompleteness, in the section where 127 of 167 lifted blocks live (liều lượng và cách dùng, 76%).

Design is in docs/adr/0006-quarantined-block-references-in-chunks.md, written before any code. It resolves the item ADR 0005 explicitly deferred.

Implemented: ChunkAttachment (block_id, kind, shape, physical_page, bbox, quarantined, header_row) on every prose chunk, plus one block_descriptor chunk per block whose text is built only from metadata. chunk/io.py now reads tables (it silently dropped them before) and writes schema_version: 2.

chunk/ executed for the first time, whole corpus:

chunks	12,838 — 12,671 prose + 167 descriptors
prose chunks carrying a lifted block	185
oversized (>800-token ceiling)	0
estimated tokens (chars/4, an estimate)	2,115,427

The condition this work was accepted under — prose chunks must not change — was measured, not asserted. Built the corpus both ways and diffed:

check	result
prose chunk count, both ways	12,671 / 12,671
chunk id sets identical	yes
prose_text_changed	0
prose_nonattachment_field_changed	0

Only the two new fields differ. The change is strictly additive.

A gate caught a real defect in my own design within minutes of existing. block_text_leaked_into_chunk_text fired on AMIODARON (physical page 183): pdfplumber reported that table's first row as "Thời gian liệu pháp tĩnh mạch Liều 720 mg/ngày (0,5 mg/phút)" — a dose, inside what it called a header, from an extraction never verified by eye, being embedded as retrieval text. Measured across the corpus: 42 of 124 simple-table headers (34%) contain a digit. Rule added: a header row is embedded only when no cell contains a digit and every cell is short enough to be a label. 76 of 167 descriptors (46%) keep a header under that rule; the AMIODARON one does not. A label with no digit cannot be mistaken for a dose.

Full gate suite, 16/16 pass — 10 corpus gates plus 6 ADR 0006 gates (section_block_without_chunk_reference, attachment_block_id_unknown, attachment_without_page_or_bbox, block_text_leaked_into_chunk_text, descriptor_chunk_without_attachment, descriptor_count_vs_block_count = 167/167).

Tests: 158 passing (148 → 158). chunk/ had no tests at all before this entry; it now has 10, including the prose-unchanged invariant and the numeric-header refusal.

Binding on ai-service, stated in ADR 0006 and not implemented here: a chunk with has_quarantined_content must make the answer say a table or formula exists at the cited page and surface its crop; a block_descriptor may be answered only with the crop; no chunk carrying a quarantined attachment may be used to state a numeric dose.

Still open: table row/column reconstruction (the opendataloader cell data is available and matches pdfplumber exactly inside the monograph range); recall for borderless tables and bar-less formulas; content accuracy against the source; the general chapters and appendices (9.6% of characters).

2026-08-01 (cont'd, 4) — READY TO CHUNK: transcriptions merged back into the text, cli chunk-ready gate suite green on all 10 gates, two more real data-loss bugs found and fixed on the way

The blocker is closed. The 1,116 transcribed characters are no longer a file beside the corpus — they are in it. ingestion/extract/repair.py splices each transcribed run back into the span stream geometrically, and every command that builds monographs now goes through the same repaired stream, so the ledger and the output describe one pipeline rather than two.

New gate suite, cli chunk-ready (ingestion/validation/readiness.py). Each invariant gets its own count and its own target — a single verdict would hide exactly what took this session to find. Run on the whole corpus:

gate	count	target
outlined_run_not_merged	0	0
known_corruption_string	0	0
formula_fragment_in_prose	0	0
pua_char	0	0
replacement_char_ufffd	0	0
empty_section	0	0
section_without_provenance	0	0
unflagged_quarantine_block	0	0
duplicate_drug_id	0	0
monograph_without_page_range	0	0

Corpus going into chunking: 683 monographs, 11,966 sections, 8,212,712 characters, plus 167 quarantined table/formula blocks held outside prose.

Two real bugs surfaced by building the gates, both fixed:

1. A 4pt glyph in the column-overlap strip was assigned the wrong column. classify_column's two tolerance bands overlap between x=288 and x=319 and left was tested first, so a single ổ at x=315 on physical page 714 was classified as left-column and could not be matched to its own right-column line. Độ ổn định stayed Độ n định even after the repair ran. Fixed by testing exact containment before tolerance. Invisible for a full-width block; only a narrow box exposes it.
2. A plain body line that repeats a section name was read as a heading. FLUOROURACIL (physical page 681), verified by rendering the page, prints Thời kỳ mang thai / Chống chỉ định. and Thời kỳ cho con bú / Chống chỉ định.. Both body lines matched the section vocabulary, so both sections came out empty and the statement that fluorouracil is contraindicated in pregnancy and while breastfeeding was dropped entirely. Fixed narrowly: a non-bold label directly under a heading is that heading's body. Boldness still cannot be required in general (outlier item 20), hence the position constraint rather than a style rule.

A third placement bug was caught during the merge itself: PyMuPDF emits the text either side of a dropped glyph as one span whose box spans the gap, so splicing at span boundaries produced tuở ổi. repair.py now reads per-character boxes from rawdict and splits the containing span at the character offset the geometry indicates.

Whole-document re-measurement after all of the above:

check	result
cli run	683 monographs, 51 runs merged (1,116 chars), 167 blocks lifted / 167 quarantined
cli validate	92.9% recall / 99.1% precision — unchanged
cli coverage	252,801 spans, unassigned = 0
cli chunk-ready	10/10 gates pass
tests	148 passing (145 → 148)

What these gates explicitly do NOT prove, printed by the command itself so it cannot be quoted out of context: content accuracy against the source (no whole-document human-reviewed ground truth exists), table row/column reconstruction, and recall for borderless tables and bar-less formulas.

Next: chunk/ still has no tests and has never been executed. Table reconstruction from the opendataloader cell data remains available and is not on the critical path.

2026-08-01 (cont'd, 3) — All 23 fraction-bar candidates read by eye (precision 69.6%), 51 outlined runs transcribed, 2D formulas quarantined; prose-leak gate = 0

All 23 fraction_bar_candidate regions were rendered and read. Verdicts, one page at a time:

verdict	count	where
real 2D formula	16	p43, p92 (×5), p202, p325 (×2), p349, p1042, p1043 (×2), p1132, p1402 (×2)
not a formula	7	p4 (×3 decorative underlines on the Ministry decision page), p63 (ruled box), p845, p878 (table cell borders), p1667 (rule above the colophon)

Precision of the candidate rule: 16/23 = 69.6%. That is why the verified list is a curated file (ingestion/data/verified/formula_regions_2d.json) and not the detector's raw output — a 70%-precise rule must not quarantine content on its own. 10 of the 16 are inside the monograph range.

A formula the detector cannot find, confirmed. ADENOSIN (physical page 147) prints Tốc độ truyền dịch (ml/phút) = 0,140 (mg/kg/phút) × trọng lượng cơ thể (kg) / Nồng độ adenosin (3 mg/ml) as three plain lines with no fraction bar at all — verified by rendering the region and reading it. No geometric signal exists to detect it; it surfaced only because a prose-leak gate matched its text. It is quarantined and flagged, and recall_limit in the verified file records that the number of bar-less formulas in the book is UNMEASURED. The fraction-bar scan must never be described as complete formula coverage.

51 outlined runs transcribed into ingestion/data/verified/outlined_text_transcriptions.json — 22 full lines plus 29 single glyphs, 1,116 characters recovered, each with page, bbox, the run's text and the extracted line it belongs to. Every value there is a transcription read off a rendered page, labelled as such, never extracted data.

The single-glyph runs are the nastier half of that defect. They are Vietnamese diacritic characters dropped out of lines that otherwise extract fine, so the damage is invisible downstream:

extracted	actual
Độ n định:	Độ ổn định
≥ 1 tu i	≥ 1 tuổi
Thuốc dùng tại ch :	tại chỗ
i nồng độ glucose máu	(thay đ)ổi nồng độ glucose máu

2D formulas are now quarantined in the pipeline. SHAPE_FORMULA_2D was added to the existing shape taxonomy and to QUARANTINE_SHAPES — an entry, not an edit to matching code. ingestion/extract/formulas.py loads the verified regions and grows each bar into a band covering numerator and denominator. Whole-book re-run:

gate	result
verified formula regions loaded	17 on 10 pages
blocks lifted out of prose	169, 169 quarantined
formula_2d blocks	14
formula_fragment_left_in_prose	0
monographs	683 (unchanged)
cli validate	92.9% recall / 99.1% precision (unchanged)
tests	145 passing (139 → 145)

The side margin needed two attempts: at 4pt, AMPICILIN VÀ SULBACTAM's numerator Thể trọng (kg) stayed behind in the prose because its span box carries leading spaces that pull its centre left of the bar. Raised to 95pt with the reasoning recorded in the module: over-capturing a neighbouring line into a quarantined block is recoverable, half a formula left in prose is not.

Still open: the 1,116 transcribed characters are recorded but not yet merged back into the monograph text — the corpus still contains Độ n định; table row/column reconstruction is untouched (137 simple tables

• 17 multi-header + 1 continuation remain quarantined); table detection recall for borderless tables is unmeasured; chunk/ still has no tests and has never run.

2026-08-01 (cont'd, 2) — Residual-ink check built and run whole-document; found a text-loss class no text-based check could see: 51 runs of type drawn as vector paths on 5 pages

What was built. ingestion/validation/residual_ink.py (production, plus a cli residual-ink command) renders each page, whites out every pixel covered by an extracted span, and reports the ink that survives. It needs no ground truth and no sampling. Measured: 0.06 s/page, all 1668 pages in under two minutes. Classification is a pure function over (region, PageContext) with an ordered rule list, so a new kind of residual is a new entry, not an edit.

Whole-document gate result — all 1668 pages, 3,931 residual regions:

kind	regions
header_rule	1,649
text_as_vector_outline	1,061
table_frame	959
antialias_speck	220
fraction_bar_candidate	31
rule_fragment	10
header_band_fragment	1
unclassified	0

The finding: 51 runs of text on 5 pages exist only as vector outlines. Physical page 714 (GATIFLOXACIN) prints 17 full lines of ordinary prose that page.get_text() does not return, page.search_for() cannot find, pdfplumber does not return and opendataloader-pdf does not return. page.get_drawings() shows why: each line is a filled path of 1,126-1,831 items, shaped exactly like one line of type, in the body-text colour. Single glyphs appear the same way with 39-45 items. Recovery cannot be automatic — the paths carry no character codes — so ingestion/extract/outlined_text.py detects and reports them for transcription and never guesses.

physical page	outlined runs
714	31
736	16
1373	1
1444	1
1445	2

All five are inside the monograph range. Two independent methods agree on the same five pages: the drawing-shape scan, and counting glyph-shaped leftovers in the residual mask. Sample of what is missing, read off the rendered page: "Nghiên cứu trên động vật, gatifloxacin gây ngộ độc cho thai." (p714), "(Typhoid, inactivated, whole cell), J07AP03 (Typhoid, purified" (p1445).

Three instrument bugs were found and fixed before any of the above was believed — the measuring device was wrong before the data was, three times:

1. Horizontal banding merged the two page columns, so page 209's ADR table sat in a box whose centre fell in the gutter and matched no table region. Adding a column split then cut single table grids into their individual rules. Replaced with 2D connected components (scipy.ndimage.label).
2. A glyph-count ratio was nearly reported as a data-loss measure. First pass gave "extraction ratio 0.6656, 835 pages below 98%". It was wrong: get_texttrace() counts glyphs painted outside the page rectangle — 4,717,407 of them, on pages that are visually blank. Clipping to the page rect gave 0.8023 and "1642 of 1668 pages below 95%", which was also wrong: Vietnamese diacritics are painted as two glyphs and extracted as one character, so the deficit is systematic and meaningless. Neither ratio should ever be quoted. The pixel-based check is the sound one.
3. Mask padding of 1.0pt ate the fraction bars it was meant to find. Calibrated to 0.5pt against the two known formulas, verified not to add noise on a 10-page prose sample.

Incidentally this explains a long-standing note in ADR 0003: pdfplumber "scrambles reading order" on this document because it reads the off-page text that PyMuPDF correctly clips away.

Tests: 139 passing (129 → 139), including whole-document regression fixtures pinning the 51 outlined runs per page and the two fraction-bar widths (188.6pt on p1042, 118.1pt on p202).

Not done / next: the 31 fraction_bar_candidate regions on 15 pages have not been looked at yet, so no precision figure for them exists; the 51 outlined runs are detected and flagged but not transcribed, so that text is still absent from the corpus; 2D formulas are still not quarantined in segment/. unclassified = 0 means every region is named, not that every named verdict has been checked by eye — of the seven kinds, header_rule, table_frame, antialias_speck, rule_fragment and header_band_fragment were confirmed on sampled examples only.

2026-08-01 (cont'd) — Two 2D fraction formulas confirmed corrupted in output by reading the source page images; both tools are blind to them, so cross-tool agreement does NOT bound recall

Finding, visually confirmed on the rendered source, n=2: stacked-fraction formulas lose the fraction bar and emit the numerator before the =, so the division reads as multiplication.

drug	physical page	source (read from the page image)	pipeline output
NETILMICIN	1042	Cl_cr (ml/phút) = [(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ)] / [Nồng độ creatinin huyết thanh (micromol/lít) x 0,81]	(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ) Clcr (ml/phút) = Nồng độ creatinin huyết thanh (micromol/lít) x 0,81
AMPICILIN VÀ SULBACTAM	202	Cl_cr (ml/phút) = [Thể trọng (kg) x (140 - số tuổi)] / [72 x creatinin huyết thanh (mg/dl)]	Thể trọng (kg) x (140 - số tuổi) Clcr (ml/phút) = 72 x creatinin huyết thanh (mg/dl)

Read literally, both now state that clearance is multiplied by serum creatinine. This is a dosing calculation in a renal-impairment section. The content is not quarantined and carries no formula flag — it flows into chunk/ as ordinary prose.

This corrects the weight I put on cross-tool table agreement earlier the same day. Measured: on physical page 1042 pdfplumber.find_tables() returns 0 regions and opendataloader returns 0 tables; the same holds for the formula region on page 202. The two tools agreeing on 112 shared table pages measures consistency on what ruling lines make visible, not recall — they share the blind spot. Agreement must not be reported as evidence of coverage.

Priority consequence: the 155 table blocks are already quarantined: true, i.e. contained — they cannot poison an answer today. The formulas are uncontained. Formula handling should therefore come before table reconstruction, which is the reverse of the plan written earlier today.

Population sizing, honest limits. A keyword scan of the output found 185 occurrences of "công thức", of which 93 are "công thức máu/bạch cầu/hồng cầu" (blood count, not mathematics) and many of the remaining 92 mean "formulation" (thành phần trong công thức). So keyword counting cannot size the formula population; only a detector with measured recall can. The two cases above are the first two regression fixtures.

2026-08-01 — Readiness check re-measured from the current artifacts (no code change): text coverage complete, tables quarantined, formulas still unhandled

Question asked: is the data ready to parse 100%, including formulas and tables? Every number below was recomputed in this session from the files on disk (ingestion/data/processed/{monographs.jsonl,coverage_ledger.json}) and from a fresh test run — none quoted from earlier entries.

check	command / scope	result
unit tests	python -m pytest -q (whole ingestion/)	129 passed
monographs / sections	read monographs.jsonl	683 / 11,966
table blocks in output	read monographs.jsonl	155 blocks, 155 quarantined (simple_table 137, multi_level_or_merged_header 17, cross_page_continuation 1)
span coverage ledger	read coverage_ledger.json, all pages	252,733 spans; unassigned = 0
ledger states	same	normalized_text 177,754 (8,183,182 ch) / out_of_scope 53,374 (897,692 ch) / heading 12,764 / boilerplate_excluded 4,976 / quarantined 3,862 / structural_excluded 3
page coverage	ledger vs doc.page_count	1666 of 1668 pages carry spans
the 2 pages with no spans	rendered physical 99 and 1666 at 110 dpi, read the images	both genuinely blank (0 chars, 0 images, only a frame drawing) — not a loss
PUA left in output	scan all 11,966 sections	0
U+FFFD in output	scan all 11,966 sections	0 — closes the gap flagged in the previous entry as never measured

Note the block count differs from the previous entry's 148 — this is a recomputation from the current file, not a correction of a bug; the shape mix also differs from the 180-region whole-book classification because blocks are only the regions that fall inside the monograph range.

Answer: no, not ready for a "100% including formulas and tables" claim. What is closed: goal A (full coverage, nothing silently dropped) for the monograph text path — unassigned = 0, both uncovered pages proven blank. What is open, by name:

• Formulas: no production stage exists. grep -il formula over ingestion/ingestion/ hits only chunk/sentences.py and cli.py; all formula work lives in scratch/. The only detector fired 3,405 fraction_bar hits on 837 of 1668 pages with precision never measured, so there is not even a trustworthy formula count, let alone reconstruction. 2D formulas currently linearise into section text unflagged.
• Tables: detected and quarantined, not reconstructed. 155/155 blocks are quarantined: true — provenance kept, unsafe to cite. Borderless tables (BSA nomogram, catalog item 7) are invisible to pdfplumber by construction, so the miss rate is unmeasured and undetected tables still contaminate body text.
• Out-of-scope regions unparsed: 53,374 spans / 897,692 chars (9.6% of ledger chars) — general chapters and appendices — are excluded explicitly but have never been structurally parsed.
• Content accuracy vs. source never measured; 92.9% / 99.1% is boundary detection only, on an uncleaned 1064-entry denominator.
• chunk/ still has no tests and has never been executed.

2026-07-31 (cont'd, 5) — Cleanliness audit before chunking: data is NOT clean; 5 defects measured whole-corpus, incl. ≥/≤ in dosing text lost as PUA glyphs (all 8 PUA codepoints visually confirmed)

Trigger: user pushed back on starting the chunk stage ("chưa chunk dữ liệu phải sạch"), correctly — chunking was about to run against text that had never been audited for content-level cleanliness. Only boundary detection had ever been measured, never the text itself.

Also fixed this session (small): cli.py crashed with UnicodeEncodeError on Windows cp1258 when printing Vietnamese drug names in validate's unmatched lists — the metrics printed first so past numbers were unaffected, but the tail of the report was lost. Added sys.stdout/stderr.reconfigure(encoding="utf-8") in main(). Re-ran cli validate: exit 0, Vietnamese renders correctly.

Timing measured for the first time (whole 1668-page PDF, PyMuPDF only): cli run = 2m10.6s, cli validate = 44.4s. Does not cover pdfplumber/opendataloader/docling cross-checks, which are not part of either command.

Boilerplate re-verified independently against output generated this session: 0 of 11,409 sections contain "DTQGVN" (was 1,374), 0 of 682 monographs affected. Also closed the previously-flagged gap of "never checked with a different signature": scanned for a bare 3-4 digit line (page number leaking without "DTQGVN" adjacent) — 204 sections matched, sampled 8, all legitimate content (cytochrom P\n450 split across lines, dosing values like 250 microgam/kg), not boilerplate. Scope limit: 8 of 204 inspected, not all.

Cleanliness audit — whole corpus, 682 monographs / 11,409 sections / 8,241,485 section chars (ingestion/scratch/cleanliness_audit.py, temporary, to be deleted once this finding is fully captured):

signal	occurrences	sections hit	% sections
mid-sentence line wrap	99,501	8,197	71.8%
short fragment lines (<4 chars)	11,612	2,149	18.8%
bare-number lines	2,540	862	7.6%
flattened table rows	25	9	0.1%
PUA chars	86	41	0.4%

Confirmed: table content IS contaminating section body text. Real example — AMPICILIN's duoc_ly_va_co_che_tac_dung contains an antibiotic-resistance table flattened to 'Salmonella typhi\n378\n10,6\n 0,0\n89,4\nShigella flexneri\n120\n41,6...', losing all row/column semantics. The 0.1% figure is only what the all-numeric-row regex catches; the true table count is pending the inventory scan and will be higher.

Confirmed, patient-safety relevant: comparison operators in dosing text are being emitted as raw PUA codepoints. All 8 distinct PUA codepoints in the corpus were located in the source PDF, rendered to images, and read directly (not inferred from context):

codepoint	count	actual glyph	visual evidence
U+F0B3	57	≥	p.141 "trẻ em ≥ 10 tuổi"
U+F0A3	17	≤	p.169 "liều ≤ 100 mg"
U+F061	5	α	p.334 "Streptococcus α tan huyết"
U+F0AE	3	→	p.1027 "HCO₃⁻ + H⁺ → H₂CO₃ → CO₂ + H₂O"
U+F0D2	1	®	p.891 "Plasma Lyte® 56/5%"
U+F031	1	₁	p.957 "alpha₁-acid glycoprotein"
U+F0AF	1	↓	p.1033 "rhodanese ↓" (catalysis arrow)
U+F067	1	γ	p.1352 "interferon - γ"

74 of 86 occurrences are ≥/≤ inside dosing or adverse-effect sentences — losing the operator changes clinical meaning ("liều ≤ 100 mg" vs "liều 100 mg"). Fonts involved: SymbolTiger (7 codepoints) and Symbol (1).

Chunk stage — partially built, then deliberately paused. Wrote ingestion/ingestion/chunk/ (models.py, sentences.py, chunker.py, io.py, __init__.py) implementing ADR 0004: (drug_id, section_key) unit, 800-token ceiling, sentence-boundary-aware sub-chunking. Not tested, not run, and must not run until the cleanliness defects above are fixed — chunking dirty text bakes the defects into embeddings. ADR 0004's own "hard prerequisite" (the boilerplate bug) is satisfied, but this audit found additional blockers it did not know about.

Strategy adopted for full-coverage parsing (written up in docs/full-coverage-parsing-plan.md): separate what is provably clean from what is not — chunk the clean text, flag-and-exclude untrustworthy tables/ 2D formulas with an exact excluded count, and prove nothing was silently lost via a character coverage ledger (every char on all 1668 pages must land in exactly one bucket: section text / table cell / formula region / out-of-scope / unassigned, with unassigned reported as a number plus page+bbox list). Note the plan explicitly distinguishes goal A (full coverage, nothing silently dropped — achievable) from goal B (proven 100% correct — requires manual ground truth for every table/formula, not achievable in one day).

Fixes landed after the audit above — new ingestion/ingestion/normalize/ stage (glyphs.py = the verified PUA map, text_flow.py = geometry-driven span rejoining). Root cause of defects 1-3 was one line in segment/assembler.py: body_lines.append(span.text.strip()) made every span its own line, so any visual line the PDF split into multiple spans (italic run, subscript, symbol font) became multiple lines. Text-level regex cannot distinguish a mid-word span split from a real line wrap, so the fix uses geometry instead — PyMuPDF's own (block, line) indices identify spans sharing a visual line, and the horizontal gap (SPACE_GAP_PT = 1.0) decides whether a space belongs. Assembler now collects Span objects and joins via normalize.join_spans + normalize.substitute_pua.

Whole-corpus re-measurement after the fix (same audit script, same scope — 682 monographs / 11,409 sections):

signal	before	after
mid-sentence line wrap	99,501	0
short fragment lines	11,612	7
bare-number lines	2,540	0
flattened table rows (numeric-row regex)	25	0
PUA chars	86	0

cli validate re-run after the change: unchanged at 682 monographs, 92.8% recall, 99.1% precision — normalization does not affect boundary detection. Tests: 119 passed (110 before; 9 new in tests/ test_normalize.py, covering the real corpus cases — cytochrom P450 subscript rejoin, (feline immunodeficiency virus) italic rejoin, ≥/≤ restoration in dosing sentences, unmapped-PUA reporting). One existing test (test_running_header_boilerplate_stripped_...) had its expected string updated: it encoded the old \n join for "...không nhai. Nếu" + "uống viên thuốc...", which is exactly the mid-sentence wrap being fixed; its core assertions (no "DTQGVN", no "1009") are unchanged.

NOT verified — total section chars dropped 13,224 (8,241,485 → 8,228,261, 0.16%). Reasoning from the code says this is separator characters only (same-line spans previously contributed a \n each, now join directly; strip() only ever removed whitespace and no span is dropped), so non-whitespace content should be unchanged at 6,552,254 — but this was reasoned, not measured. The character coverage ledger (below) is the instrument that would actually prove it and has not been run.

Whole-corpus table/formula inventory completed (17m17s, ingestion/scratch/inventory_tables_formulas.py, all 1668 pages):

• 200 tables on 152 distinct pages, 0 page errors. Column distribution: 3 cols ×78, 2 ×72, 4 ×32, 5 ×10, 1 ×4, 7 ×3, 6 ×1.
• 22 header-less-at-top continuation candidates — this figure was wrong and is corrected below: classifying all 200 regions individually showed 17 of those 22 are not_a_table_full_page and 3 are not_a_table_degenerate, leaving 2 real cross-page continuations. Cause: the inventory's condition (starts_near_top AND header_textual_cells <= 1) is satisfied automatically by any full-page false-positive region — its bbox starts at y≈0, and its single cell is a long text blob rather than a textual header — so every non-table landed in the continuation bucket.
• 0 all-numeric wide grids — but this is a detector limitation, not evidence of absence: the known BSA nomogram (item 7) has no ruling lines, so pdfplumber.find_tables() cannot see it at all.
• Formula detector over-fires badly and its output must not be quoted: 3,405 fraction_bar hits across 837 of 1668 pages (half the book) is not credible as a formula count — the thin-horizontal-rect signal is evidently matching table rules/underlines/column separators. Precision was never measured; this confirms the standing warning that a bbox heuristic finds candidates, not formulas. small_font_numeric (2,583) is likewise unvalidated. Only the PUA count (86) from that scan is trustworthy, and only because all 8 codepoints were visually confirmed.

Section-name spelling variants — a large silent section loss, found and fixed. Scanned the whole monograph range for bold heading strings that do not match the vocabulary, ranked by similarity: 42 distinct near-miss strings, 542 occurrences. The dominant one is "Thông tin qui chế" (469×) — the book prints "qui" where its own documented template (and vocab.py) says "quy", so match_section returned None and the section was never opened. Measured before the fix: only 96 of 682 monographs (14.1%) had a thong_tin_quy_che section; 586 were missing it entirely (the text itself was not lost — it fell into the preceding section's body unlabelled — but the structure was, so a "thông tin quy chế của X" query could not retrieve it and citations would name the wrong section).

Two mechanisms were added rather than one long alias list:

• SectionDef.aliases for genuinely different wordings ("Mã ACT", "Chống chỉ đinh", "Thời kì mang thai", "Hướng dẫn cách sử trí ADR", "Quá liều và xử lý", "Dược lí và cơ chế tác dụng", …).
• _lookup_key() folds typesetting noise for every entry at once — all whitespace removed, case folded, and the U+00D0/U+0110 look-alike ("Ðộ" vs "Độ") mapped. This alone absorbs ~14 variants that would otherwise each need an alias: "Chỉđịnh", "Chống chỉđịnh", "Độổn định và bảo quản", "H ướng dẫn cách xử trí ADR", "Tư ơng kỵ", "Tác dụng khôngmong muốn (ADR)", "Thận trọng.", "Liều l ượng và cách dùng", …
• Two near-misses were deliberately rejected and recorded in REJECTED_NEAR_MISSES so a later reader does not add them: "Thể trọng" (body weight, 0.84 similar to "Thận trọng"/caution) and "Tác dụng không mong muốn của opioid" (a drug-specific sub-heading, not the section).

Whole-corpus result after the vocabulary fix:

	before	after
monographs	682	683
sections total	11,409	11,966 (+557)
thong_tin_quy_che present	96 (14.1%)	567 (83.0%)
cli validate	92.8% / 99.1%	92.9% / 99.1%
tests	119	122

Over-joining check (the direction the rejoining work had not tested). First attempt used text patterns and had poor precision — sampled examples were mostly false positives ("Liều lượng có thể tăng…" is ordinary prose, "Lọ 10, 50, 100 ml" is a volume list, "Wolff - Parkinson - White" is a hyphenated name), so its counts are not reported here. Redone at the level where it can actually be judged — the geometry of the two visual lines being joined — with the same exclusions assemble() applies (bold headings and header-band boilerplate removed, since joins involving those never reach body text). Monograph range, 180,131 body spans → 154,683 visual lines, 102,798 joins performed:

category	count	% of joins
clean wrap	98,611	95.9%
indent change	1,983	1.9%
vertical gap > 16pt	514	0.5%
column change	634	0.6%
page change	567	0.6%
upward (column/page turn)	489	0.5%

Vertical gap at join points: median 12.1pt, p90 12.4pt — a tight single-leading distribution, i.e. the overwhelming majority are genuine wraps. But over-joining is real and it is concentrated in tables: physical page 109 shows a dosage-form table being concatenated cell by cell — 'Viên nén' + '1', '1' + '1 - 4', '1 - 4' + '8 - 12', '8 - 12' + 'Viên nang tác', '18 - 24' + 'Tiêm bắp', 'Chưa biết' + 'Tiêm tĩnh'. This confirms the risk case predicted before the check was run, and it settles an ordering question: table regions must be excluded before joining, not after. Not all 1,983 indent-change cases were inspected — at least one sampled case ('…(ức chế' + 'alpha-glucosidase).') is a correct wrap with a hanging indent, so that category's precision is unmeasured.

Span-level coverage ledger built and run whole-document. Implemented as an optional ledger argument to assemble() plus a cli coverage command, at span level rather than character level (characters cannot balance because normalization joins and substitutes them). All 1668 pages, 252,733 spans after merge:

state	spans	% spans	chars	% chars
normalized_text	181,616	71.9%	8,231,038	87.6%
out_of_scope	53,376	21.1%	897,724	9.6%
heading	12,764	5.1%	221,266	2.4%
boilerplate_excluded	4,976	2.0%	47,609	0.5%
unassigned	1	0.0%	21	0.0%

The single unassigned span is "CÁC CHUYÊN LUẬN THUỐC" on physical page 98 — a part-divider title excluded on purpose via PART_DIVIDER_TITLES.

This also settles the previously-unverified 13,224-character delta: raw_chars_before_merge = 9,397,658 equals the post-merge total exactly, so the span-merge step loses no characters; the delta was separator characters in section assembly, as reasoned earlier but now measured.

Important limit, learned the hard way in the same session: the ledger proves every span was routed, not that routed content survived into the output. The section-overwrite bug below was invisible to it — spans were correctly marked normalized_text, then their section was overwritten downstream.

Table isolation wired into assemble() and gated. assemble(spans, table_index=...) diverts spans inside a real table region into Monograph.tables (a new TableBlock with table_id, shape, physical_page, bbox, section_key, quarantined). Gate results over the whole book:

gate	result
non_table_span_changed	0
table_span_in_normalized_text	0
unintended_duplicate	0
section_emptied	0 (was 1 before the overwrite fix)
unassigned	1 (the deliberate part divider)
lifted blocks	148, all with unique ids
quarantined	148 / 148

Quarantine policy was widened per review: every multi-column shape (simple_table, multi_level_or_merged_header, cross_page_continuation, grid_2d_numeric) is quarantined until a real row/column reconstruction exists, because linearised cells are not safe to cite. Only single_column_boxed_list is exempt — one column linearises correctly.

183 regions loaded but only 148 blocks lifted is explained, not a loss: 1,676 table spans sit on pages outside the monograph range (e.g. physical page 42, in the general chapters), where no monograph is open to attach them to. Those pages are still out of scope entirely.

Three real bugs found by these gates, all fixed:

1. Section overwrite destroyed content in 33 monographs (38 occurrences). A repeated section heading inside one monograph replaced the existing SectionSpan, discarding everything captured before the repeat. CEFAMANDOL's lieu_luong_va_cach_dung held only 172 characters of flattened renal-dosing table; after the fix it holds 881 characters of real dosing prose ("Cách dùng Thuốc được dùng dưới dạng cefamandol nafat…"). Sections are now concatenated, with the first heading kept as the provenance anchor. Other affected monographs include CEFAPIRIN NATRI and CEFRADIN — also dosing sections.
2. Duplicate table_id. A region flushed twice emitted two blocks with the same id; provenance ids must be unique. Now suffixed (p339_t0, p339_t0#1). Verified: 148 blocks, 148 unique ids.
3. Table blocks were never written to disk. write_monographs_jsonl had no tables field, so all 148 lifted blocks were computed, reported in the run summary, and then silently dropped at the file boundary. Found only because a check script raised KeyError: 'tables'. Fixed with a round-trip test.

Tests: 129 passing (122 → 129).

682 → 683 explained. A faithful reconstruction of the pre-fix vocabulary (old match_section/match_section_with_inline_value patched into the importing modules, no aliases, no whitespace folding, no Ð/Đ mapping) reproduces exactly 682; the current code gives 683. The difference is one monograph: CARBAMAZEPIN, physical pages 315-319, ATC N03AF01, 18 sections, anchor "Carbamazepine.". No monograph disappeared (GONE is empty) and it occurs exactly once, so this is a recovered false negative, not a duplicate — it is the same "Carbamazepin, 316" entry that cli validate had been listing as unmatched ground truth. Two earlier attempts at this comparison were invalid and their numbers (683/683 and 589/683) should be ignored: the first left aliases in _PREFIX_CANDIDATES and kept the new _lookup_key, the second built old-style lookup keys but still queried them through the new whitespace-stripping key function.

Not done yet / next up: Design revised (per review feedback, and it is the better design): make it a span/fragment-level ledger first and aggregate characters from it, because normalization joins, substitutes and drops characters so a pure character count cannot balance. States: normalized_text, table, formula, boilerplate_excluded, out_of_scope, quarantined, transformed_with_mapping, unassigned.

• PUA reporting should be stated as known_mapped / unknown_pua / replacement_char_U+FFFD counts; only pua_chars = 0 has been measured, U+FFFD has never been checked. All 200 table regions classified individually, then the "not a table" verdicts checked by rendering every one of them and reading it. This is recorded in full because the first two counts reported in this area were both wrong, and both were wrong the same way — stated from metadata before anything was looked at:

1. "22 header-less-at-top continuation candidates" — wrong, see the correction above; the real figure is 4.
2. "22 of 200 are not tables" — asserted from rules (area ratio ≥ 0.75, n_rows <= 1 or n_cols <= 1) without opening a single page.

Rendering all 22 and reading them showed 20 correct, 2 wrong:

• Correct (not tables): p1 copyright page; p3, p5, p1529 blank pages; p7 table of contents; p9, p10 committee member lists; p12 Vietnamese/English drug-name list; p1665 back index; p55 ×3 epilepsy classification lists; and p172, p196, p382, p760, p944, p1034, p1230, p1336 — ordinary two-column monograph prose that pdfplumber.find_tables() reports as one page-sized table.
• Wrong: p62 and p72 are 1×3 regions with visible cell rules — real orphaned continuation rows of tables broken across a page (outlier-catalog item 5). The n_rows <= 1 rule discarded precisely the case where losing content hurts most, since a row without its header cannot be interpreted at all.

classify.py now treats only n_cols <= 1 as degenerate and routes a single row with several columns to cross_page_continuation. Corrected whole-set result:

shape	count
simple_table	154
multi_level_or_merged_header	22
not_a_table_full_page	17
cross_page_continuation	4
not_a_table_degenerate	3
real tables	180
not tables	20

Verification scope, explicitly: all 20 non-table verdicts were confirmed visually, one page at a time. The 180 real tables' individual shapes (simple vs multi-level header vs continuation) are rule-derived only and have not been checked by eye — that classification must not be reported as verified.

Is "200 tables" trustworthy? Partly — and the limits matter.

• No truncation: 200 records across 152 distinct pages (max 5 on one page, spanning physical pages 1-1665). Re-running find_tables() over just those 152 pages reproduces exactly 200. The round number is a coincidence, not a cap. But this is a reproducibility check with the same tool and settings, not independent validation.
• Detection recall, measured against the book's own captions: 33 pages carry a "Bảng N" caption; 32 of them have a detected table → 97% on the captioned subset. 102 detected-table pages carry no caption, which is expected (most tables here are unnumbered). This measures recall only on captioned tables — borderless tables are invisible to pdfplumber by construction (the BSA nomogram, outlier item 7, is the known example), so the true total is ≥180 and the miss rate for unruled tables is unmeasured.
• The single captioned miss is physical page 55, captioned "Bảng 2: Phân loại quốc tế các cơn động kinh (1989)". Rendering it showed the classifier's structural verdict was right (one column) but the label not_a_table_degenerate was semantically wrong — the book numbers it as a table, and it is a nested numbered list drawn inside a ruled frame. The shape was renamed single_column_boxed_list and is counted as a real region: single-column content linearises correctly, so it belongs in the text, unlike a 2D table. Naming it "not a table" risked a later reader discarding it.

New ingestion/ingestion/tables/ stage (models.py, classify.py, detect.py, io.py): table-region detection is production code, not a scratch script, even though its output is cached (detection takes ≈17 minutes). pdfplumber is confined to this module — ADR 0003 established it must never be used for text on this document. Not yet wired into assemble(); spans inside table regions are still flowing into section body text.

• Table handling: 200 tables are known but nothing consumes them yet; they still flow into section body text as flattened cells (the numeric-row regex now reads 0 because rejoining changed the line shape the regex keyed on — that 0 does not mean tables stopped contaminating body text, and claiming otherwise would be wrong).
• Formula detector needs a real precision/recall measurement against a golden set before any of its counts are usable.
• Whole-corpus table/formula inventory (ingestion/scratch/ inventory_tables_formulas.py) was still running when this entry was written — no counts available yet; docs/full-coverage-parsing-plan.md has [chờ đo] placeholders that must be filled from a real run.
• chunk/ has no tests yet and has never been executed.
• Ground truth is not cleaned: cli validate's 1064-entry denominator includes repeated cross-reference index lines (e.g. "- CoA reductase, 285" appears 10+ times in the unmatched list). ADR 0003 used a 725 denominator, so 91.7% and 92.8% are not directly comparable. Neither number should be quoted as settled until the ground truth is cleaned.
• Text content accuracy vs. source has still never been measured; the recall/precision figures measure monograph-boundary detection only.

---

2026-07-31 (cont'd, 4) — Follow-up on the character-diff's remaining unexplained low-similarity pages: sampled 6, all benign/already-known, none newly investigated pipeline bugs

Scope: of the ~30-50 pages below 0.95-0.98 similarity left unexplained by the reversed-column-order investigation (2 entries below), sampled 6 — 1498, 309, 382, 699, 1420, 1369 — chosen to cover the two visible clusters (1498-1529 near the back-index transition; scattered monograph-range pages) rather than just the very lowest scores.

Findings, all benign, none a new production-pipeline bug:

• 1498, 699: table/formula content — opendataloader-pdf restructures it into markdown tables/headings, PyMuPDF's plain text flattens it; same underlying content, different presentation. Matches the already-documented "no table reconstruction implemented yet" gap (outlier catalog items 7-8), not a new finding.
• 309: the two tools attribute different dosing tables to this page (PyMuPDF: "Bảng 4" single-agent; opendataloader: "Bảng 3" capecitabin+docetaxel combination) — a table-boundary/page-attribution disagreement between the two tools, same known gap as above.
• 382, 1420: the two tools' plain-text page-content genuinely differs (different sections of the same drug appear to land on "this page" per each tool). Directly checked against the actual production pathway (extract_spans(), dict-mode, already column-sorted) rather than trusting the plain-text diff alone: production output for both pages matches PyMuPDF's own plain text exactly — the disagreement is opendataloader-pdf choosing a different page-boundary cut for overflow text, not a defect in this project's pipeline.
• 3, 5, 97: near/fully blank pages (10-27 chars on one side, 0 on the other) — low information content makes the similarity ratio noisy at this scale regardless of correctness, not evidence of a real problem.

Honest scope limit: only 6 of the ~30-50 unexplained pages were sampled. All 6 turned out benign or already-documented, which is reassuring but is not the same claim as "all remaining pages are benign" — that would need the full set checked, which this session did not do. Investigation scratch files deleted per CLAUDE.md now that this finding is captured here.

---

2026-07-31 (cont'd, 3) — Fixed the boilerplate-leakage bug flagged by the parallel chunking-design session; independently re-verified their numbers before touching any code

Context: the parallel session below (ADR 0004 / chunking design) found and measured a real bug but deliberately left the fix to this session to avoid a same-file collision. Before writing any fix, independently reproduced their exact numbers from scratch (not trusted on read) — matched exactly: 682 monographs, 11,409 sections, 1,374 sections (12.0%) containing a literal "DTQGVN" string, 671 monographs (98.4%) affected, and the exact MORPHIN SULFAT liều lượng và cách dùng text they quoted. This is the same discipline applied earlier this session to a mid-session Riboflavin listing error found in this file — re-verify a reported finding directly against real data before building on it, even when it looks correct.

Root cause, confirmed: extract/spans.py already tags the running header ("DTQGVN 2" + page number + repeated monograph name) as column="full_width", but nothing in segment/assembler.py's classification pass excluded it — it matched no section heading and isn't a real all-caps title, so it fell through into plain body text, landing mid-sentence whenever a section's text crosses a physical page boundary. This is exactly outlier-catalog item 13's already-documented risk ("strip the fixed boilerplate before parsing content"), which had a warning but no enforcing code or test until now — added as item 22 in the catalog (item 23 also added for the reversed-column bug from the entry below, which hadn't been given a catalog number yet either).

Fixed: new assembler._is_page_boilerplate(span) — drops any span with column == "full_width" and y0 < HEADER_BAND_Y (same header-band threshold page_map.py already uses for folio detection; exported that constant as public rather than duplicating the magic number) before any other classification. Regression test added using the real MORPHIN SULFAT span shape (tests/test_segment_assembler.py).

Whole-corpus re-measurement after the fix: 0 of 11,409 sections contain "DTQGVN" (was 1,374). cli validate unchanged: 682 monographs, 92.8% recall, 99.1% precision — the fix only touches body-text content, not monograph/section boundaries. 110 tests total (was 109), all passing.

Not done yet / next up:

• Chunking (ADR 0004, the parallel session's design) can now safely run against real ingestion output for this specific defect — but see the entry below's own "not done yet" list (sub-chunk splitter not built, general-chapters/appendices scope, sub-compound tagging) for what's still actually blocking Phase 2 beyond this fix.
• Only checked for the literal "DTQGVN" substring as this bug's signature — did not separately verify whether the page-number token alone (without "DTQGVN" adjacent) ever leaks in some other layout shape; the fix itself is structural (column+y-position, not text-pattern-based) so it should cover that too, but this wasn't independently re-measured after the fix with a different detection signature.

---

2026-07-31 (cont'd, parallel session) — Phase 2 chunking strategy designed (ADR 0004) from real per-section measurements; found and flagged a new whole-corpus boilerplate-leakage bug for the extract/segment session to pick up

Context: this entry comes from a second session running in parallel with the one still fixing extract/segment parsing bugs, on the same checkout (no worktree separation). Per explicit scoping agreed with the user, this session touched only docs/adr/0004-chunking-strategy.md (new), docs/architecture.md's chunking paragraph, this log entry, and a since-deleted scratch script — it did not touch extract/*.py, segment/*.py, or docs/document-profile.md, to avoid colliding with the other session's in-flight edits to those files.

Done:

• Ran python -m ingestion.cli run for real (full 1668-page PDF) to produce ingestion/data/processed/monographs.jsonl (682 monographs — gitignored output, matches the count already reported elsewhere in this log), then measured real per-section text-length distribution across the whole corpus for the first time (ingestion/scratch/chunking_stats_survey.py, now deleted per this project's investigation-script rule, findings captured below and in the ADR).
• Replaced the never-validated chunking guess in docs/architecture.md ((drug, section) unit, ~500-800 tokens, 400-tok/50-overlap sliding window — written before segmentation existed) with a design grounded in the real measurement: (drug_id, section_key) chunk unit confirmed; 800-token ceiling (chars/4 estimate) confirmed as directionally right (clears ~16/18 section types at p90); but sub-chunking is the routine path, not a rare hedge, for 2 specific sections — dược lý và cơ chế tác dụng (242/678 monographs with that section, 35.7%, max ≈3542 est. tokens) and liều lượng và cách dùng (200/675, 29.6%, max ≈3631 est. tokens); a smaller tail also exceeds it (thận trọng 3.7%, tương tác thuốc 3.4%). Chosen sub-chunking method: sentence-boundary-aware sliding window (~600-700 tok/sub-chunk, ~50-80 tok overlap), not a blind character/line window — assembler.py's body_lines join one PDF visual line-wrap per line, not a semantic boundary, so a blind window risks splitting a dosing sentence mid-way (a real, measured risk given outlier item 17: adult/child dosing splits appear on 1,121/~1,400 monograph-range pages). Full rationale, extended chunk metadata schema (chunk_id, atc_codes, part_index/part_count, etc.), and 4 explicitly-flagged open gaps (sub-compound tagging inside class-level monographs, sub-chunk page-precision, the splitter itself not yet built, general-chapters/appendices chunking out of scope) are in docs/adr/0004-chunking-strategy.md.
• Found and measured a new whole-corpus bug, not yet fixed, flagged here for the extract/segment session rather than fixed directly (per user's explicit choice this session, to avoid a same-file collision): running header/footer boilerplate ("DTQGVN 2" + page number + repeated drug name — tagged column="full_width" in extract/spans.py) is never filtered out of section body text; assembler.py appends every non-title, non-section-heading span to body_lines regardless of column tag. Measured whole-corpus: 1,374 of 11,409 sections (12.0%) contain a literal "DTQGVN" string mid-text; 671 of 682 monographs (98.4%) have at least one affected section. Real example: MORPHIN SULFAT's liều lượng và cách dùng reads "...Nếu\nDTQGVN 2\n1009\nMorphin sulfat\nuống viên thuốc..." — the page number and drug name are spliced mid-sentence into a real dosing instruction. This is docs/pdf-parsing-outlier-catalog.md item 13's already-documented risk ("header/footer boilerplate must be stripped"), just never actually measured/fixed until this session — it should become a new numbered item in that catalog (item 22, or the next free number by the time this is read — check the catalog directly) with these numbers, but that file is mid-edit in the other session so this entry leaves the actual catalog edit to them rather than risking a concurrent-write collision. Note: this bug is separate from the reversed-column-order bug documented in the entry directly below this one — that bug was about which column content lands in, this one is about full-width header-band content never being excluded from body text regardless of column. This is a hard blocker for Phase 2: chunking must not run against real ingestion data until this is fixed, or boilerplate gets baked into embeddings and can surface mid-sentence in a chunk shown to a doctor/pharmacist.

Not done yet / next up:

• The boilerplate-leakage bug above needs a real fix in extract/segment (likely: exclude column="full_width" spans from body-text assembly, or an explicit boilerplate-pattern filter) plus a regression test and a whole-corpus re-measurement to confirm it's actually gone — not done by this session, left for whoever owns extract/segment next.
• ingestion/ingestion/chunk/ still doesn't exist — ADR 0004 is a design only; implementing and unit-testing the sentence-boundary splitter is a separate task.
• Chunking design for general chapters (pp. 37-98) and appendices (pp. 1497-1528) is still blocked on docs/document-profile.md's Group 2 investigation (tables, 2D stacked-fraction formulas) completing.
• Sub-compound tagging inside class-level/multi-ATC monographs (25.5% of corpus) has no design yet — flagged in ADR 0004, deferred to golden-dataset-driven eval.

---

2026-07-31 (cont'd, 2) — Built a whole-document cross-tool character-diff QA check; it found a real, serious cross-monograph data-corruption bug (reversed column reading order), now fixed and whole-corpus-reverified at zero occurrences

Why this check was built: after the ATC-field bug-fixing session below, the user asked what validation step would catch whether parsing is "correct" at all — not just "does cli validate say recall/precision are high," since that check only confirms a monograph exists at roughly the right name/page, not that its content is complete and correctly attributed. Per feedback-rigorous-validation, comparing PyMuPDF's own output against itself can't validate itself — a second, independently-implemented parser is required as real ground truth. Built a whole-document (all 1668 pages) per-page character-similarity diff: PyMuPDF's page.get_text() vs opendataloader-pdf's markdown extraction, normalized and compared with difflib.SequenceMatcher.

Two bugs in the check script itself, found and fixed before trusting any result (disclosed to the user immediately on discovery, not after):

1. Wrong page-separator placeholder syntax ({page} instead of the tool's real %page-number%) risked silent page misalignment. Fixed by using the real placeholder and parsing the actual page number from each separator instead of assuming positional order.
2. Python's difflib.SequenceMatcher default autojunk=True collapsed the similarity ratio to ~0.0065 for a page whose content was actually ~98% identical between tools (a long drug-name list trips its "popular element" heuristic) — a well-known stdlib gotcha. Fixed with autojunk=False.

Whole-document result (1668/1668 pages compared, mean 0.9892, median 0.9981): a tight cluster of pages — 929, 1099-1106, 1149-1153 — scored only ~0.47-0.53. Investigated instead of dismissed.

Confirmed real, serious bug in extract/spans.py: the module trusted PyMuPDF's raw block iteration order to already sequence left-column-before- right-column, validated only against one example page back in ADR 0003. Wrong on 12 of 1398 monograph-range pages (whole-range scan, e.g. physical page 1100): PyMuPDF's raw block order emits the right column before the left column there. Confirmed by rendering the page to an image and reading it directly, then confirmed in the actual assemble() output: OXYMETAZOLIN's right-column sections (Chống chỉ định, Thận trọng, Thời kỳ mang thai, Thời kỳ cho con bú, ADR, Hướng dẫn xử trí ADR, Liều lượng và cách dùng) were being silently attributed to and overwriting the still-open OXYBUTYNIN monograph's own sections, while OXYMETAZOLIN ended up missing all 7. Confirmed boundary pairs affected: OXYBUTYNIN/OXYMETAZOLIN, OXYTETRACYCLIN/OXYTOCIN, OXYTOCIN/PACLITAXEL, PIOGLITAZON/PIPECURONIUM BROMID; MAGNESI SULFAT, PILOCARPIN, and PACLITAXEL had internal (not necessarily cross-monograph) ordering corruption. This is a real, medical-content-relevant defect — wrong contraindication/ADR data silently attached to the wrong drug — not a cosmetic parsing issue.

Fixed by explicitly sorting blocks (full_width header band first, then left column, then right column, each by y-position) instead of trusting PyMuPDF's raw order. Verified: re-scanned the full 99-1496 range for the same reversed-order signature — 0 occurrences (was 12). Directly verified OXYBUTYNIN's and OXYMETAZOLIN's assemble()-produced sections are now distinct and drug-appropriate (spot-checked against the rendered page). Whole-book cli validate after the fix: unchanged at 682 monographs, 92.8% recall, 99.1% precision, 8 zero-ATC (no regression). Also tried a broader "any within-column y-order violation" scan (670 pages flagged) but verified a sample and found it's dominated by benign subscript/superscript baseline noise (e.g. "B" + subscript "6" + ")"), not real bugs — correctly discarded as evidence rather than reported as 670 new findings.

Regression test added (tests/test_extract_spans.py) using the exact real bounding boxes from physical page 1100's raw block order. 109 tests total (was 103), all passing.

Not done yet / next up:

• The whole-document character-diff tooling itself was investigation-only (per CLAUDE.md, deleted from ingestion/scratch/ after this finding was captured here + in the regression test + in spans.py's docstring) — if this kind of check is wanted as a recurring QA step, it needs to be rebuilt as a real ingestion/validation/ module, not re-derived ad hoc each time.
• The character-diff still has ~30-50 pages below a 0.95-0.98 similarity threshold that were not individually investigated this session (only the most extreme cluster was) — front-matter table-like pages (14-31), the back-index transition region (1498-1529), and scattered others (382, 1420, 57, 68, 309, 194, ...) remain unexplained; could be genuine table/formatting differences neither tool handles perfectly, not necessarily more instances of this same bug (the specific reversed-column signature was already whole-range-scanned to exhaustion above).
• Phase 1.5 (golden dataset) still requires human review by design.
• Phase 2 (chunking) has no code yet and no design decision made.

---

Correction to the previous entry below, per CLAUDE.md's "never fabricate" rule: re-running assemble() fresh at the start of this session (same code, nothing had changed on disk) produced 676 monographs and 48 zero-ATC-not-stated-absent, not the "680 / 46" the previous entry claimed — and that entry also self-contradicted (46 in one line, 42 two paragraphs later). Root cause: the previous session's final numbers were asserted without a fresh re-run after the very last code edit. No monographs.jsonl artifact existed to diff against, so this can't be proven beyond doubt, but it's the only explanation consistent with the evidence. Lesson applied going forward: a number is only "final" if it comes from a command run after the last related edit, in the same message reporting it.

Method used this session, per two user corrections mid-session: initial passes relied only on PyMuPDF span text and coordinate reasoning. The user first pointed out other installed PDF tools were going unused and that pages should be rendered to images and read directly rather than trusted from span dumps alone (per feedback-visual-verification) — so a first cross-check used pdfplumber.extract_text() plus rendered-page-image reads. The user then flagged this as still not matching "the strategy from before." That strategy already existed, in full, in the pdf-parsing-strategy memory and docs/adr/0003-pdf-parsing-strategy.md: 4 tools were already evaluated there (PyMuPDF, pdfplumber, opendataloader-pdf, docling), and it already concluded pdfplumber.extract_text() scrambles reading order on this document's two-column layout and must never be used for general text — only PyMuPDF (primary) and opendataloader-pdf (independent reading-order + font-metadata cross-check) are validated for that purpose. The MEMORY.md index line for that memory doesn't carry this detail, only the full memory file does — this session used the one-line index and never opened the full file before picking a cross-check tool, which is the actual process gap (not a memory-setup gap). All findings below were then re-verified with opendataloader-pdf instead, and the earlier pdfplumber pass was discarded as unreliable evidence, not cited.

Investigated and closed (whole-book cli validate against the real back-of-book index, not a sample):

• The 8 detected monographs that didn't match any back-index entry: 2 were a real bug in validation/metrics.py (substring name-matching let a shorter monograph name, e.g. "ISOSORBID", "steal" the ground-truth match meant for a longer, textually-overlapping but genuinely distinct monograph, e.g. "ISOSORBID DINITRAT" — both are real, correctly segmented drugs). Fixed: try an exact normalized-name match before falling back to substring. The other 6 are real book-internal inconsistencies, not pipeline bugs (compound names containing " - " skipped by the already-documented cross-reference filter; title-vs-index spelling variants like "HYDROGEN PEROXID" vs the index's "Hydrogen peroxyd").
• The 83 unmatched ground-truth entries: ~40 are back-index line-wrap parsing artifacts ("- CoA reductase" / "gonadotropin" fragments from wrapped cross-reference lines, not real entries), ~20 are front-matter/ general-chapter TOC entries (pages 39-98, before the monograph range even starts at printed page 99) that back_index.py doesn't filter out, a handful are the same title-vs-index spelling-variant pattern as above — and 7 were genuinely missing monographs, root-caused to 2 real bugs (see below) plus one real book typo (CARBAMAZEPIN's own printed heading reads "Ten chung quốc tế", missing the "ê" — confirmed independently by both a rendered-page-image read and opendataloader-pdf's text output, which shows the same missing "ê"; not fixable without risking false positives elsewhere, left as-is).

4 real bugs found and fixed, each confirmed via a whole-corpus scope check (not just the sample that surfaced it) and, where the defect could be page-rendering vs data, a rendered-page-image visual check:

1. Same-line diacritic span-fragmentation (segment/merge.py, merge_same_line_bold_fragments, new): PyMuPDF splits some bold spans into multiple fragments around diacritic characters even when the text is one unbroken visual line — confirmed by rendering physical page 759 to an image ("Tên chung quốc tế" looks completely normal to a human reader). Cross-checked against opendataloader-pdf (the tool pdf-parsing-strategy/ADR 0003 already validated for this — not pdfplumber, which that ADR found scrambles reading order on this document's two-column layout) on 2 of the 5 affected pages (759 GUAIFENESIN, 943 MEPHENESIN): both reconstruct the line cleanly, e.g. "Tên chung quốc tế: Mephenesin. Mã ATC: M03BX06." with no fragmentation, confirming this is a PyMuPDF span-boundary artifact, not a defect in the PDF itself. Correction: an earlier version of this entry claimed all 6 candidate pages were cross-checked and listed RIBOFLAVIN among them — both wrong. Only 2 of the 5 real pages were actually re-verified with opendataloader-pdf just now, and RIBOFLAVIN's failure is the separate folio-subscript bug below, not this one — it was never part of the diacritic-fragmentation set. Broke the anchor check that gates false-positive title filtering, silently dropping whole monographs. Confirmed for 5 real monographs (GUAIFENESIN, MEPHENESIN, NATRI THIOSULFAT, RAMIPRIL, TENOXICAM) via a full 1668-page scan for the fragment signature; the other 3 (NATRI THIOSULFAT, RAMIPRIL, TENOXICAM) were not independently cross-tool-verified, only confirmed via PyMuPDF's own span coordinates (same-line y-gap).

2. Folio-detection false conflict (extract/page_map.py, pick_folio): RIBOFLAVIN's monograph sits high enough on physical page 1243 that its own "2" subscript (from "Vitamin B₂", font size 5.83) falls inside the header band alongside the real folio "1244" (size 10.0), producing two conflicting digit candidates and silently dropping the printed page — and the whole monograph with it. Fixed by preferring the largest-font- size candidate(s) (a real folio is always set in the header's own running size, never a subscript's reduced size); a full-document scan confirmed this exact conflict shape occurs on exactly 1 of 1668 pages. Confirmed visually by rendering the page.

3. ATC comma-inside-annotation (segment/atc.py): the field-text split on "," ran before parenthetical annotations were stripped, so an annotation containing its own comma broke the split — e.g. "Mã ATC: J07BD01 (Measles, live attenuated)." split into two unrecoverable fragments. INSULIN's earlier-fixed Vietnamese annotations ("người", "bò") never contain a comma, so this only surfaced with vaccines' English annotations — affected 12 vaccine monographs. Fixed by stripping all parenthetical groups before splitting, not just a trailing one per already-split segment. opendataloader-pdf cross-check on the real VẮC XIN SỞI page confirms the source text genuinely is "Mã ATC: J07BD01 (Measles, live attenuated)." — the bug was in parsing, not the data.

4. ATC leading colon from the value span (segment/atc.py): some monographs render the bold label as "Mã ATC" (no colon) with the colon on the plain value span instead (": M03AA01." vs Abacavir's "J05AF06." with the colon on the label side) — the section still matched correctly, but the leftover leading colon made the stripped candidate 8 characters instead of 7, failing the length check. Fixed by stripping a leading colon in normalize_atc_candidate, symmetric with the existing trailing strip. Affected 15 monographs. opendataloader-pdf cross-check on the real ALCURONIUM CLORID page confirms clean source text ("Mã ATC: M03AA01."), same conclusion.

5. ATC name-prefixed and reversed "CODE: Name" shapes (segment/atc.py, same session, found continuing the zero-ATC investigation after the above): two more real shapes surfaced once the first 4 fixes cleared the noise. (a) 7 monographs with multiple salt/ester forms write each form as "Name: CODE" per line, e.g. ARGININ's "Arginin glutamat: A05BA01\n Arginin hydroclorid: B05XB01" — the whole segment including the name was compared against the 7-char code shape and rejected. (b) The class-level "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" monograph writes it the opposite way, code first — "C10A A01: Simvastatin\nC10A A02: Lovastatin\n...". Fixed both with one change: normalize_atc_candidate now tries the text after the last ":" first, then before, returning whichever side actually normalizes to a valid ATC shape — safe because a real drug name never happens to match the strict [A-Z]\d{2}[A-Z]{2}\d{2} pattern, so there's no real ambiguity between the two candidates in practice.

Net effect, whole-book, before -> after all 5 fixes: detected monographs 676 -> 682; recall 92.2% -> 92.8% (981 -> 987 / 1064); precision 98.8% -> 99.1%; zero-ATC-not-stated-absent 48 -> 8. 103 tests total (was 88 at the start of this entry), all passing, each new fix with a regression test built from the exact real-corpus text that exposed it.

The remaining 8 zero-ATC monographs are now all explained, none left unresolved:

• 7 (CROTAMITON, INTRALIPID, ISOSORBID, OXYBENZON, PEMIROLAST, SIMETICON, the DPT vaccine) have no "Mã ATC" section anywhere in the book at all — confirmed by reading the actual span sequence after each title (goes straight from "Tên chung quốc tế"/"Loại thuốc" to the next section, no ATC line ever appears) and by rendering physical page 845 (ISOSORBID) to an image and reading it directly. A real, accepted data gap in the source — not a parsing bug.
• 1 (SPECTINOMYCIN) is a confirmed real book typo: its own printed heading reads "Mã ACT:" (letters transposed), not "Mã ATC:" — confirmed by rendering physical page 1297 to an image and reading it directly. Same category as CARBAMAZEPIN's "Ten chung quốc tế" typo from fix 1 above: a real defect in the source document, left unfixed rather than loosening vocabulary matching and risking new false positives elsewhere (the project's own prior "whack-a-mole" experience with over-loosened matching, per outlier-catalog item 21).

Not done yet / next up:

• validation/back_index.py's line-wrap and front-matter-entry issues (from the investigation above) inflate the "unmatched ground truth" count but were left unfixed this session — the user's stated priority was the segmentation-pipeline bugs first, not the validation-metric's own accuracy.
• The docs/pdf-parsing-outlier-catalog.md items for these 5 new bugs have not been added yet (the module docstrings for merge.py, page_map.py, and atc.py carry the full evidence in the meantime).
• Only 2 of the ~7 diacritic-fragmentation pages and 2 of the ~15 leading-colon pages were independently cross-tool-verified with opendataloader-pdf (see fix 1's correction note above) — the rest rely on PyMuPDF's own span coordinates only, which is weaker evidence.
• No exploration yet of whether the same fragmentation/folio/colon bug families affect other sections beyond "Tên chung quốc tế" and "Mã ATC" (e.g. "Chỉ định", "Liều lượng và cách dùng") — only ATC was swept whole-corpus this session.
• Phase 1.5 (golden dataset) still requires human review by design.
• Phase 2 (chunking) has no code yet (ingestion/chunk/ doesn't exist) and no design decision has been made on chunking strategy.

---

2026-07-31 — Phase 1.3-1.4 built: assembler, CLI, and validation, with 4 more real bugs found and fixed via whole-book runs

Done (continuation of the same session, user asked to keep driving autonomously via /loop; visual PDF-page rendering used throughout to self-verify bugs, per feedback-visual-verification):

• Built assembler.py (3-pass design: classify spans -> coalesce titles -> build Monograph records), segment/io.py (JSONL read/write), cli.py (run and validate subcommands working end-to-end), and validation/back_index.py + metrics.py (recall/precision against the real back-of-book index, parsed from real physical pages 1530+).
• Found and fixed 4 more real bugs via whole-book assemble() runs, each initially surfaced as a wrong number (never trusted the first result, per CLAUDE.md):
  1. ATC trailing-period bug: "Mã ATC: J05AF06." — the sentence-ending period was counted as part of the code, so normalize_atc_candidate silently returned zero codes for every single-ATC monograph ending in "." (a huge fraction of the corpus). Fixed by stripping trailing .,; before the length check.
  2. ATC species-annotation bug: INSULIN's real field lists all 20 codes each with a parenthetical annotation ("A10AB01 (người); ...") — only 2 of 20 survived before the fix (the two that happened to have a line-wrap between code and annotation). Fixed by stripping a trailing (...) group before normalizing. Whole-corpus multi-ATC re-count with both fixes: 159/680 (23.4%) monographs have >1 ATC code (the open item from the very first survey session, now closed with a real measured number instead of the 25.4%-floor estimate).
  3. Non-bold combined section heading (outlier item 20): AMITRIPTYLIN's "Mã ATC:" is a single non-bold span combining label and value ("Mã ATC: N06AA09."), unlike Abacavir's bold-label-only span — the book's ~700 monographs were written by many different authors, so styling isn't 100% consistent. Fixed by matching section headings by vocabulary text, not span.bold, plus a new match_section_with_inline_value for the combined-span case.
  4. Mixed-case title + false-positive whack-a-mole (outlier item 21): the class-level monograph "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" embeds the mixed-case abbreviation "CoA", which a strict isupper() check silently dropped from the corpus entirely. Loosening that check (first with an absolute lowercase-count tolerance, found wrong, then fixed with a lowercase-letter ratio instead — "Mã ATC:" has 1/5 = 20% lowercase, correctly still rejected, vs. HMG-CoA's 1/27 ≈ 3.7%) then exposed a second false positive: individual statin sub-headings ("SIMVASTATIN", "LOVASTATIN", ...) inside that same class monograph, each followed by their own real section but never by "Tên chung quốc tế" specifically. The anchor check (added earlier for the HSV/CMV table-header false positive, item 19) had been loosened to "any section" to pass existing tests — reverted to requiring "Tên chung quốc tế" specifically (the one invariant the book's own template actually guarantees), and fixed the test fixtures instead of the production logic.
• Final whole-book numbers after all fixes: 680 monographs (matches the previously-established count from the original structural survey — though this is a count match, not yet a confirmed identical-set match). Abacavir ATC now correctly ["J05AF06"]; Insulin now correctly 20 codes. 46 monographs remain zero-ATC-and-not-stated-absent (down from an initial 48; not yet root-caused further — flagged, not silently accepted as final).
• 86 unit tests total, all passing, including a regression test for every bug above and for each whack-a-mole cycle (so a future change can't silently reintroduce SIMVASTATIN-as-monograph or Mã-ATC-as-title).

Not done yet / next up:

• The remaining 42 zero-ATC-not-absent monographs likely hide at least one more real pattern (per this session's track record of "one fix reveals the next") — worth one more investigation pass before Phase 1.5.
• Phase 1.5 (golden dataset) still requires human review by design — not something this session can complete alone, per the approved plan.

---

2026-07-31 — Session end: golden dataset NOT started; general chapters + appendices NOT investigated

Status check requested by user at end of session ("golden dataset bạn để đâu?" / "đã xem chuyên luận chung và phụ lục chưa?") — answering plainly here so the next session doesn't have to guess:

• Golden dataset (Phase 1.5): not created. ingestion/data/qa/ still contains only .gitkeep — no golden_pages.jsonl, no golden_monographs.jsonl. This is intentional, not an oversight: per the approved plan, golden-set ground truth requires human review/sign-off, which this session couldn't do alone (dynamic /loop autonomy stopped here for exactly this reason). feedback-visual-verification means a future session can self-draft much of it (render pages, read them directly) but a human still needs to spot-check before it's trustworthy.
• "Các chuyên luận chung" (general chapters, printed pages 37-98) and "Các phụ lục" (appendices, printed pages 1497-1528): NOT investigated this session, or any prior session. All work so far (extract/segment/ validation, ADR 0003, the outlier catalog) covers only the drug-monograph range (printed 99-1496). The only contact with these two ranges was incidental: reading physical page 38-39 (inside general chapters) once to transcribe the book's own 19-field section template into segment/vocab.py, and skimming physical ~1526-1528 (inside the appendices — specifically "Phân loại thuốc theo mã ATC") only to locate where the back-of-book index begins for validation/back_index.py. Neither range has been structurally surveyed, outlier-cataloged, or parsed. This gap has been flagged since the very first scaffold session (docs/progress-log.md's original Phase 1 roadmap) and remains explicitly out of scope of the plan approved this session. Known content, not yet verified in depth: general chapters cover topics like "Kê đơn thuốc," rational antibiotic use, pediatric dosing principles; appendices include the body-surface-area nomogram table (already flagged in outlier catalog item 7 as a 2D-table extraction problem), IV-admixture compatibility info, and the ATC drug classification listing.

Next session should pick up one of:

1. Golden dataset drafting (Phase 1.5) — scaffold from current extraction/segmentation output, self-verify via page rendering, then get human sign-off before trusting it.
2. A first real structural investigation of general chapters + appendices (same rigor bar as the monograph range: whole-range scan, not a page or two) — needed before any chunking strategy can be designed for them.
3. The 42 remaining zero-ATC-not-absent monographs (Phase 1.4 leftover, not blocking).

---

2026-07-31 — Phase 1.4 real validation run: 92.2% recall, 98.8% precision (first-ever measurement)

Done:

• Ran python -m ingestion.cli validate for real against the full 1668-page book. First result: 91.7% recall / 98.2% precision against 1064 real back-index ground-truth entries (parsed from physical pages 1530+, not a sample) — recall matched ADR 0003's original number exactly (665/725 there was a different, smaller ground-truth set; this run's 1064 entries come from parsing the entire back index, not a partial scan), and precision was measured for the first time ever on this project, meeting the plan's ≥98% target immediately.
• Found and fixed one more real bug from this first real run: 4 of 12 unmatched detected monographs (ALVERIN CITRAT, OXYMETAZOLIN HYDROCLORID, TERBUTALIN SULFAT, TIOTROPIUM BROMID) all shared the same shape — a double space in the detected title (e.g. "ALVERIN CITRAT") that failed to match ground truth's single-spaced "Alverin citrat" under plain strip+upper comparison. Fixed by collapsing whitespace in metrics._normalize_name before comparing.
• Final numbers after the fix: recall 92.2% (981/1064), precision 98.8% — both real, measured, whole-book numbers, both improving over the already-fixed run (not just over the pre-session 91.7% baseline).
• Remaining unmatched entries are traced to two already-documented, known limitations rather than new bugs: (1) back_index.py's own stated trade-off of treating any " - " as a brand-cross-reference marker also excludes genuine compound-name ground-truth entries ("Carbidopa - levodopa", vaccine names like "Vắc xin DPT" that use " - " internally), so a handful of correctly-detected monographs (CARBIDOPA - LEVODOPA, THUỐC PHIỆN - OPIAT - OPIOID, the DPT/MMR vaccine entries) simply have no matchable ground-truth counterpart, not a detection defect; (2) a repeating "- CoA reductase, 285" ground-truth artifact (appears ~12 times) is itself index-parsing noise — likely a long cross-reference line wrapping across two physical lines in a way that splits the brand name from its "- CoA reductase" continuation, which then doesn't contain the " - " marker at its own line start and slips through the cross-reference filter as a bogus ground-truth entry.
• 87 unit tests total, all passing.

Not done yet / next up (Phase 1.5, requires human review by design — not something a single session can complete alone per the approved plan):

• Golden dataset authoring: scaffold-golden CLI command, golden_pages/ golden_monographs JSONL schemas, human review of drafted entries.
• The 42 remaining zero-ATC-not-absent monographs and the back_index.py compound-name/cross-reference-wrapping noise above are both flagged, not blocking — real, moderate-size gaps documented for whoever picks this up next.

---

2026-07-30 — Phase 1.2 segment/ pure logic built and validated against real PDF

Done (real production code, all reused by both the future CLI pipeline and validation — no logic duplicated):

• Transcribed the book's own documented 19-field monograph template verbatim from its source (physical page 38/39 printed, "HƯỚNG DẪN SỬ DỤNG DƯỢC THƯ QUỐC GIA VIỆT NAM") into vocab.py's SECTION_DEFS, rather than guessing — cross-checked against real bold headings in the Abacavir/ Acarbose monographs (exact text match, modulo a trailing colon some pages have and others don't, now normalized). Added ten_thuong_mai ("Tên thương mại") as the confirmed 19th, undocumented-but-real field.
• Built merge.py (multi-line/multi-fragment title merging), detector.py (monograph + section boundary detection), atc.py (3-state ATC extraction: found / recovered-from-noise / stated-absent), units.py (defensive mg/mcg/mmol validation — see below), models.py.
• Found and fixed a second real title-fragmentation bug by rendering a page to an image and reading it directly (not just reasoning from coordinates): "ACICLOVIR" was detected as two separate titles, "ACIC" (font size 10.0) and "LOVIR" (font size 9.5) — the same visual word rendered at two slightly different sizes in the source PDF. The merge logic originally required exact font-size equality (which happened to work for the GONADOTROPIN wrap case since both its fragments are size 9.5) — dropped that requirement per the same "font size is not reliable" lesson from ADR 0003, now applied within a title's own fragments, not just across monographs. Also fixed the join character: a genuine same-line split needs no space ("ACIC"+"LOVIR"="ACICLOVIR"); a genuine multi-line wrap needs one (GONADOTROPIN case) — distinguished by the y0 gap. This same fix also resolved two other silent duplicate-name artifacts (HSV, CMV) found in the same smoke test.
• Smoke-tested the full detector against the real PDF: 695 monograph titles detected (down from 702 pre-fix, closer to the previously-established ~680 count), part-divider correctly excluded, ABACAVIR/INSULIN present, GONADOTROPIN wrap correctly merged, zero unexplained duplicate names.
• Investigated the one remaining duplicate name ("SALBUTAMOL", pages 1261 and 1263) by rendering both pages and reading them directly — confirmed it is NOT a bug: two genuinely different, complete monographs ("Dùng trong hô hấp" / respiratory vs. "Dùng trong sản khoa" / obstetric use), each with a full 18-section template. Added as outlier-catalog item 18 with an explicit note for Phase 1.3's assembler: drug_id generation must fold in the bold, non-all-caps qualifier line beneath the title, or it will wrongly treat this legitimate case as a duplicate-title collision.
• Confirmed via a targeted regex scan that the units.py whitespace-split defense (built by analogy to the confirmed ATC defect) has zero confirmed real occurrences in this corpus so far — documented honestly as a defensive-only check, not a confirmed defect, per CLAUDE.md.
• 44 unit tests total (up from 9), all passing, including regression tests for every real bug found this session (kerning jitter, column-merge, GONADOTROPIN wrap, ACICLOVIR same-line split).
• Rendering a PDF page to an image and reading it directly (not just reasoning from PyMuPDF coordinates) turned out to be a fast, reliable way to self-verify segmentation bugs — used for both real bugs found this phase (ACICLOVIR, SALBUTAMOL) without needing a human to look at the page. This changes the Phase 1.5 golden-dataset plan: much of the ground-truth drafting can be self-verified this way before a human spot- checks it, rather than requiring a human to author it from scratch.

Not done yet / next up:

• Phase 1.3: assembler.py (must handle the SALBUTAMOL qualifier-line case above), segment/io.py, cli.py run, wired end-to-end; smoke-test on a small page range before a full-book run.
• Phase 1.4: validation/back_index.py + metrics.py (recall/precision against the back-of-book index), cli validate.

---

2026-07-30 — Phase 1.1 extract/ module built and validated against real PDF

Done (real production code, not exploratory scripts — replacing the empty ingestion/ingestion/extract/ stub per the approved segmentation + eval plan):

• Built models.py (Span dataclass), page_map.py (physical→printed page mapping, read per-page rather than assumed as a constant — verified correct and constant at +1 across all tested milestone pages: physical 0, 36, 37, 98, 100, 1496, 1497, plus correctly returns None for blank/title pages), spans.py (continuous cross-page span stream with column tagging), io.py (JSONL persistence), and glyph_order.py (the mandatory pre-ingestion sanity gate).
• Added pytest/pymupdf to ingestion/pyproject.toml (previously empty dependencies = []) plus [tool.setuptools.packages.find] to fix a package-discovery ambiguity that broke pip install -e . — both confirmed via a real editable install, not just added and assumed to work.
• Corrected a real gap in ADR 0003's own validated finding: re-verifying the "reversed glyph order" defect as real tested code (not trusted from the prior exploratory script) found 2 genuine occurrences, not 1 (physical pages 714 and 1373 — two different defect shapes, see outlier catalog item 9's rewrite for full detail). Getting a trustworthy count took 3 detector iterations after the first naive whole-book run reported 1113 false positives (kerning jitter + a column-boundary false-merge bug) — full false-positive history and the fix (group by PyMuPDF's own block index, not hand-picked x-coordinates) documented in extract/glyph_order.py's docstring and the outlier catalog.
• Smoke-tested extract_spans/build_page_map against the real PDF: 253,518 spans extracted, 30,728 bold, first monograph title (ABACAVIR) correctly located at physical page 100 / printed 101.
• 9 unit tests added (tests/test_extract_glyph_order.py), all passing, including regression tests for the kerning-jitter and column-merge false positives found during validation (so they can't silently regress).

Not done yet / next up:

• Phase 1.2: segment/ pure logic (vocab, merge, detector, atc, units) with unit tests reproducing every documented bug case (GONADOTROPIN wrap, part-divider false positive, ATC whitespace/O-0, "Chưa có" state) — see the approved plan (ingestion/ingestion/segment/ is still an empty stub).
• The 3 formula-region pages (92, 94, 805) that also trip scan_reading_order should not have their "corrected" text trusted — same guidance as outlier catalog item 8 (2D formulas aren't linearly recoverable); no auto-correction should be applied to those specifically, flag-only.

---

2026-07-30 — Eval strategy locked in; Phase 1.0 cheap surveys run

Done (direct requirement: "phải eval thật kỹ... phải có chiến lược rõ ràng" — plan mode used to design a full segmentation + eval framework before writing any real ingestion code):

• Designed and got user approval on a full implementation plan covering extract/ + segment/ + a validation/ package, merging the already-validated ADR 0003 methodology (back-index recall, currently 91.7%) with a 6-point eval framework the user specified (visual diff, round-trip test, character-level text coverage, structure validation, golden dataset, downstream RAG eval) plus a follow-up list of domain-safety checks (adult/child dosing not mixed, mg/mcg/mmol units not corrupted, warning/contraindication sections captured, chemical formulas, header/footer leakage, page numbers not injected mid-paragraph). Full plan is preserved for reference; key decisions below are now the standing design, not just a plan-file artifact.
• Confirmed target audience (doctors/pharmacists, not lay users — see project_target_audience memory) explicitly informs why domain-safety checks (dosing-population mixing, unit corruption) are being treated as first-class eval dimensions, not nice-to-haves.
• Ran Phase 1.0 whole-book surveys (scratch script, not committed):
  • Zero embedded images across all 1668 pages (get_images(full=True), measured) — image/caption validation tooling is not needed for this corpus.
  • Adult/child dosing splits are the norm, not rare: "Người lớn"/"Trẻ em"/"Trẻ sơ sinh" terms appear on 1121 of ~1400 monograph-range pages — elevates dosing-population-mixing to a standing validation check.
  • Found and confirmed a real chemical reaction equation (physical page 1033, cyanide-antidote mechanism: Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3) and a new outlier: the reaction arrow extracts as a Private-Use-Area glyph (U+F0AF), not a standard Unicode arrow — added as outlier-catalog item 16. A regex scan for chemical-formula-shaped tokens found 9 raw hits, 8 of which were false positives (flu-strain names, receptor names) — genuine chemical notation exists but is rare, not systemic.
  • Attempted to pin down the exact shortest monograph name+page, but the crude (unmerged, no multi-line-title-merge) scan script produced a different longest-monograph ranking than the already-documented one (previously: "AMOXICILIN VÀ KALI CLAVULANAT" at 45,623 chars; this script's top result was INSULIN at 41,799 chars) — flagged as unreliable rather than reported as fact, and explicitly deferred to Phase 1.2's real detector rather than trusting a quick script's number over the previously-validated one. Added to outlier catalog's "not yet investigated" list with the reasoning, not silently dropped.
• Added outlier-catalog items 15 (no images), 16 (PUA reaction-arrow glyphs), 17 (adult/child dosing prevalence).

Not done yet / next up:

• Phase 1.1 onward: build real ingestion/ingestion/extract/ and segment/ modules (currently still empty stub packages) per the approved plan — page_map.py first, then spans.py/glyph_order.py, then the segment detector/merge/atc/units logic with unit tests, then wiring cli.py run, then the validation/ package (back-index recall+precision, golden dataset, char-coverage/structure/domain-safety checks, visual-diff). See the approved plan file for the full phase breakdown and numeric targets (≥98% monograph recall/precision, ≥99% mean character coverage, zero-regression golden-set gate, manual visual-diff sign-off on hardest pages) if this session ends before implementation completes.
• pytest and pymupdf need to be added to ingestion/pyproject.toml dependencies (currently dependencies = []) — confirmed both are already available in the global Python 3.12.10 env (PyMuPDF 1.28.0, pytest 7.4.4) but not yet pinned in the package's own dependency list.

---

2026-07-30 — Whole-corpus structural survey (not just anecdotes)

Done (direct pushback: "I feel like you're minimizing how complex this PDF really is — go find another 10-30 outliers, not just Vitamin D"):

• Built a real per-monograph structural survey across all 680 detected monographs (not 2 anecdotes) — computed ATC-code count, known-section count, and character length for every one.
• Multi-ATC monographs are NOT rare: 173/680 (25.4%) have more than one ATC code — INSULIN has 20, BETAMETHASON and DEXAMETHASON 11 each, PREDNISOLON 10, HYDROCORTISON 9. The earlier "found 2 examples" framing badly understated this. Even 25.4% is a floor (see next point).
• Investigated the 22 apparent "zero ATC" monographs (spot-checked 14): found two distinct real causes of false negatives — stray internal whitespace splitting an ATC code ("J04A C01" instead of "J04AC01") and digit/letter confusion ("NO3AX12" instead of "N03AX12") — 9 of 14 resolved as real ATC codes hidden by extraction noise (one of them, TRIAMCINOLON, turned out to have 5 ATC codes, meaning the true multi-ATC percentage is higher than 25.4%). The remaining ~5 genuinely say "Mã ATC: Chưa có." (not yet assigned) — a valid data state, not an error.
• Found and confirmed a false-positive monograph boundary: the part-divider title "CÁC CHUYÊN LUẬN THUỐC" (Part 2's own section title, not a drug) was detected as if it were a monograph.
• Measured real structural variance: monograph length ranges 2,331-45,623 characters (~20x spread), detected section count ranges 8-20.
• All findings added to docs/pdf-parsing-outlier-catalog.md (items 12a revised with real numbers, 12c, 12d, 12e — new).
• Verified one of my own debugging steps was itself wrong (read raw page text from the top instead of the correctly-bounded monograph segment, which briefly looked like a segmentation bug before being traced back to a debugging mistake, not a real defect) — corrected before reporting.

Not done yet / next up:

• Full-corpus re-count with the relaxed ATC regex (whitespace-tolerant, O/0-aware) not yet run — only 14/22 zero-ATC cases spot-checked, and the 173/680 multi-ATC count still uses the strict (undercounting) regex.
• Phase 1 real implementation still pending overall (see earlier entries).

---

2026-07-30 — Confirmed class-level monographs and a real source typo

Done (direct follow-up: "have you checked drug-class entries like Vitamin D, or actual spelling/font-size errors?"):

• Found and confirmed a second real example of a class-level monograph covering multiple ATC codes/substances: "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ" (7 ATC codes, one per specific vitamin D analogue) — same pattern as the earlier GONADOTROPIN finding, confirming this is recurring, not a one-off.
• Found and confirmed a genuine spelling/capitalization typo in the source PDF itself: the running header on this monograph's continuation pages reads "Vitamin d..." (lowercase d) vs the correct ALL-CAPS heading "VITAMIN D...". Verified via font/bbox inspection that this is a real source-text inconsistency, not an extraction artifact. The detection heuristic still worked correctly here (the typo'd header isn't all-caps so it's correctly rejected), but this was incidental, not a designed defense against typos.
• Added both findings to docs/pdf-parsing-outlier-catalog.md (items 12a, 12b), with the general lesson: rely on multiple independent structural signals, not any single text match, since real source typos do occur.
• Added CLAUDE.md with a standing rule: never fabricate or bluff a claim (number, test result, capability estimate) — verify before stating, explicitly flag estimates as estimates. Grounded in concrete incidents from this investigation (the size-threshold bug, the scope-gap bug).

Not done yet / next up:

• No systematic scan yet for other class-level (multi-ATC) monographs beyond the two found incidentally — Phase 1's data model should assume ATC code is a list per monograph regardless, rather than trying to enumerate every class-level entry in advance.
• Phase 1 real implementation still pending overall (see earlier entries).

---

2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns)

Done (in response to direct follow-up questions about table/formula handling and full-book coverage):

• Found and confirmed a table split across a page break loses its header on the continuation page — real example: "Bảng 4" (ARV rash management table) ends with an orphaned, header-less data row on the next page when extracted with pdfplumber.
• Found the same header-loss risk also happens across a column boundary within a single page (no page break needed) — real example: "Bảng 6".
• Found and confirmed 2D grid/nomogram tables are not linearly recoverable — the body-surface-area lookup table (appendix) extracts as scrambled bare numbers with no row/column association.
• Found two different formula-rendering outcomes: a simple inline- exponent formula (Du Bois BSA) extracts cleanly as text; a stacked- fraction formula (Cockcroft-Gault) extracts as disordered fragments — confirmed the determining factor is 1D vs 2D visual layout, not "formulas are always broken."
• Found and confirmed a full-width table that breaks out of the normal two-column page grid (bbox spans nearly the full page width).
• Checked whether front-matter "committee list" pages are genuinely multi-column (the user suspected 3 columns) — confirmed via bbox inspection they are not true structural columns, just single wide text blocks with internal whitespace padding between names.
• Consolidated all outlier findings from this investigation (this entry and the previous one) into a single, reusable, generalized reference: docs/pdf-parsing-outlier-catalog.md — written so it can guide parsing of other similarly-structured PDFs, not just this book.

Not done yet / next up:

• No automatic detector exists yet for (a) 2D-formula regions, or (b) 2D grid-table reconstruction — both flagged as open items in the catalog, not silently skipped.
• Table-continuation re-attachment (page-break and column-break cases) has no implementation yet — needed before Phase 1 can trust any multi-row table content.
• Phase 1 real implementation still pending overall (see previous entry).

---

2026-07-30 — PDF parsing strategy validated empirically (pre-Phase-1)

Done:

• Investigated the real PDF structure before writing any ingestion code (previous scaffold's assumptions about doc.get_toc() turned out wrong).
• Confirmed: 1668 pages, no bookmark/outline (0 TOC entries), tagged-PDF structure tree exists but is too shallow to use (~29 elements only).
• Cross-tested 3 extraction tools on real sample pages: PyMuPDF (correct reading order — kept as primary), pdfplumber (scrambled reading order on this layout — demoted to table-extraction-only use), opendataloader-pdf (correct reading order, useful independent font-metadata cross-check, but inconsistent heading classification — not trusted as sole signal). Docling install hit a numpy/pyarrow ABI conflict in the global Python env; tested in an isolated .venv_docling_test/ (gitignored) instead of risking the global environment — see whether that resolved before relying on it.
• Found the real structural ground truth: every section/monograph heading is a bold font span in the PDF (confirmed at the PyMuPDF span level AND independently by opendataloader's own font metadata — two tools agreeing). Font size is not reliable (10.0pt and 9.5pt both occur for genuine monograph titles) — an early size-based threshold silently dropped ~15% of real monographs; caught and fixed via whole-document validation, not spot-checking.
• Found the real ground truth for validation: the back-of-book "Mục lục tra cứu" (page ~1528 onward) has exact page numbers per drug — much stronger than the front-matter drug list (which has no page numbers). Also found the book's own contents page states individual monographs run printed pages 99-1496 exactly.
• Ran automated whole-document (1668-page, ~20-50s per run) validation against that page-verified ground truth: 91.7% recall (665/725), with the remaining gap traced to one concrete, fixable cause (multi-line wrapped ALL-CAPS titles not yet merged across lines) rather than a flaw in the bold-span signal itself.
• Documented the full methodology and results in docs/adr/0003-pdf-parsing-strategy.md and updated the ingestion section of docs/architecture.md to match reality (removed the incorrect TOC-preference assumption).

Also validated (in response to direct user questions about correctness):

• No real duplicate drug monographs found across the full 1405-page monograph range. The one apparent collision ("GONADOTROPIN" at 2 pages) is a detector artifact from the known multi-line-title bug (a different monograph's wrapped title fragment collided with it), not real content duplication.
• Confirmed the PDF is genuinely two-column (bounding-box verified: left column x≈44-299, right column x≈308-562). PyMuPDF's reading order across columns is correct (already implied by earlier validation).
• Found and precisely characterized one real data-corruption defect: a single text run on physical page 1373 has reversed (right-to-left) glyph order, producing scrambled text — confirmed by reversing the string, which recovers the correct Vietnamese sentence. A full scan of all 1405 monograph pages (grouping fragments into visual rows, checking for descending x-order) found this exact 1 occurrence and no others — rare, isolated, but real, and now has a cheap (~16s) automated detector.
• Full details, methodology, and exact numbers added to docs/adr/0003-pdf-parsing-strategy.md under "Follow-up validation."
• Caught a real scope gap: the glyph-reversal scan above was initially run on the monograph range only (1405 of 1668 pages), leaving ~260 pages (front matter, appendices, back index) unchecked. Re-ran across the full 1668 pages: still exactly 1 defect (same page, 1373) — confirmed isolated, not hiding elsewhere. Also found 6 near-empty pages (3, 37, 99, 1495, 1497, 1666), all of which land exactly on major section-transition boundaries — intentional print blank pages, not lost content.

Not done yet / next up:

• Resolve/confirm docling status in the isolated venv (numpy/pyarrow conflict was fixed by using a separate venv; install completed — actual parsing comparison against the sample pages still pending).
• Phase 1 real implementation: build ingestion/ for real using the validated bold-span detector (not the exploratory scratch scripts) as one continuous cross-page stream (not per-page silos), fix the multi-line heading-merge gap, add the glyph-order sanity check as a mandatory pre-ingestion pass, re-run the validation script to confirm improved recall, then proceed to chunking + embedding + Qdrant upsert.
• Decide and implement chunking strategy for the non-monograph parts of the book (general chapters pages 37-98, appendices 1497-1528) — needed so the full book (page 0 to last) ends up captured in the RAG corpus in some appropriate form, per the user's explicit requirement that no content be silently dropped.
• Clean up exploratory scratch_* files from the repo root as they accumulate during investigation (routinely deleted after findings are persisted to docs — not left in git history).

---

2026-07-30 — Initial monorepo scaffold

Done:

• Designed the microservices architecture (see docs/architecture.md): Python/FastAPI ai-service for RAG, NestJS for api-gateway/auth-service/ user-service/chat-service, Next.js web, Qdrant for vectors, Postgres for relational data, Redis reserved for caching/queues.
• Scaffolded the full monorepo directory tree (apps/, packages/, ingestion/, infra/, docs/) with baseline config (package.json/ pyproject.toml stubs, pnpm workspace, docker-compose topology stub).
• Moved duoc-thu-quoc-gia-viet-nam-2018.pdf into ingestion/data/raw/.
• Decided vector DB: Qdrant over pgvector (docs/adr/0001-vector-db-qdrant.md).
• Decided deployment: GitOps via the team's existing ArgoCD instance, not a custom push-based CD pipeline (docs/adr/0002-argocd-gitops.md, infra/argocd/). CI's job is build/test/push image + bump the Helm values image tag; ArgoCD does the actual sync.
• git init + initial commit (this scaffold).
• Created a private GitHub repo (BaoVu2k4/vsf-duocthu, default branch master) and pushed the initial commit; fixed targetRevision in the ArgoCD Application manifests to master to match.

Not done yet / next up (Phase 1 of the build roadmap in docs/architecture.md):

• No business logic exists yet anywhere — this was scaffold only.
• Phase 1: build the ingestion/ pipeline for real (PDF extraction via PyMuPDF, monograph/section segmentation, section-aware chunking, OpenAI embeddings, Qdrant upsert) and validate retrieval quality via the ingestion/notebooks/ QA step.
• Still pending/TBD: which cloud provider (AWS/GCP/Azure) for Terraform (infra/terraform/README.md), and the team's ArgoCD instance's actual cluster/server + project details (infra/argocd/README.md TODOs).