# Progress Log ## 2026-08-10 (cont. 16) — Recovered from the machine-trouble cutoff: Bug 2 live-reverified, quarantine path conclusively exercised, retry rate remeasured Picked up exactly where cont. 15 left off. Docker Desktop was down (machine trouble from last session), so Postgres/Qdrant containers and both app servers were all stopped. Restarted everything: Docker, `docker-postgres-1`/ `docker-qdrant-1` (same volumes, no migration needed — `rag_conversation_turn`/ `rag_retrieval_trace` tables and the Qdrant `duocthu_v1` collection's 15,100 points were confirmed intact, not rebuilt), `ai-service` (`:8079`, no `--reload`, per house rule) and `web` (`:3000`). Full `pytest -q`: 196 passed, 5 skipped, no regression from the crash/restart. **Bug 2 re-verified live in the actual browser** (the one item cont. 15 explicitly flagged as unfinished). Drove the Kanamycin eye-drop-dose question through Chrome by hand, answering the clarify chain (người lớn → indication/renal → weight → indication again) until the model converged. Got exactly the specific, honest reason the fix was supposed to produce: **"Dược thư không nêu liều dùng đường nhỏ mắt của thuốc này"** — not the old generic "chưa xác định đủ cơ sở, vui lòng thử lại" boilerplate. Fix confirmed working after the restart. **Quarantined-table citation path conclusively exercised** — the one gap named in cont. 13's 50-question audit ("did not conclusively exercise... one attempt correctly hit generic out-of-scope instead, not quarantine specifically"). Found a clean known-quarantined chunk via direct Qdrant payload query (`has_quarantined_content: true`): "Thuốc tương tự hormon giải phóng Gonadotropin — Dược lý và cơ chế tác dụng" (p.1372), a `block_descriptor` chunk whose entire content is a table lifted to quarantine (page image only, no extracted text). Asked its mechanism-of- action question live: response correctly tagged **"⚠️ CẦN ĐỐI CHIẾU PDF GỐC"**, body text "Nguồn có bảng hoặc công thức cần đối chiếu trực tiếp với ảnh PDF; không tự động trích số liệu," and the citation panel showed the matching warning card with a working "Mở trang PDF gốc để đối chiếu" deep link to printed page 1372. No fabricated number, source crop shown as designed — matches [[project_quarantined_block_contract]] exactly. **Noisy-entailment retry rate remeasured under light (non-bursty) traffic**, per cont. 13's flag that the old ~4% figure (cont. 11) might have been inflated by that session's own heavy test load. Ran 15 sequential clean single-turn factual questions via the live API (fresh `conversation_id` each, ~2.5s pacing between calls, one manual retry on any non-answerable first attempt) — script at `ingestion`-adjacent scratch path, results not committed (throwaway probe). - **11/15 (73%) answerable on the first attempt**, no retry needed. - **2/15 (13.3%) hit a genuine `abstain` on the first attempt** ("Tương tác thuốc của Warfarin là gì?", "Liều dùng Azithromycin cho người lớn là bao nhiêu?" — both `unsupported_claim`). Warfarin recovered fully to `answerable` on one retry — classic noise-and-recover. Azithromycin's retry downgraded to a `clarify` (asking for indication/route) instead of repeating the unsupported claim — the safety net choosing an honest clarify over a second bad answer, not a full recovery but not a silent wrong answer either. Matches the already-known, already-deferred `dosing_calc`/indication-dependent-dosing gap, not a new bug. - 1/15 (Cefazolin cách dùng) correctly hit `verify_pdf` first try (that section genuinely has quarantined content, independently confirmed via the same Qdrant query above) — **but the identical question retried fresh returned `clarify` instead of `verify_pdf` the second time.** Flagging as a new, small, non-blocking finding: routing/understanding isn't fully deterministic run-to-run on this query, not measured further this session. - 2/15 legitimately needed `clarify` (Insulin storage depends on vial-vs-pen/opened-state; this is a fair question to ask back, not a defect). **Honest reading of the number**: true first-attempt-abstain rate measured at 2/15 ≈ 13.3%, higher than cont. 11's ~4% theoretical estimate — but n=15 is small, and at least one of the two abstains (Azithromycin) looks like a legitimate content-ambiguity case (multiple indication-specific doses) rather than pure entailment noise, so this isn't an apples-to-apples comparison with the old number. Under genuinely light traffic, no case required more than one manual retry to reach either a correct answer or an honest clarify — nothing looped, nothing hung, nothing fabricated. Not proof the noisy-retry math from cont. 11 is wrong, but also not a clean confirmation of the old ~4% figure; worth a larger-n rerun before using either number for an SLA claim. **Also fixed while running the probe**: hit the known `UnicodeEncodeError` on Vietnamese console output (`cp1258` codec) the first run — re-ran with `PYTHONIOENCODING=utf-8` per the standing env gotcha ([[reference_env_operational_gotchas]]); also hit Python's stdout buffering silently swallowing output when redirected to a file under `run_in_background` — fixed with `python -u` (unbuffered) run as a detached shell background process instead. **Still open, not touched this session**: `api-gateway`/`auth-service`/ `user-service`/`chat-service` remain empty scaffolds — no auth, no rate limiting, no `conversationId` ownership check; this is still the largest structural gap standing between this build and production. The other 3 persistent abstains from cont. 13's audit (Aspirin+ulcer caution, Aspirin+Warfarin interaction specifically, Vancomycin rapid-infusion caution) are untouched. The two minor precision bugs from that audit (English-population-ignored, self-referential route question) are untouched. Real Postgres connection pooling (F-09's named remainder) is untouched. ## 2026-08-07 (cont. 15) — 2 more real bugs owner caught live driving the browser, both fixed; session cut short before final re-verify Right after cont. 14's fix, owner drove the actual chat themselves (not me) and hit two more real, live bugs. Both root-caused and fixed same session, committed together in `a723f62`. Machine trouble cut the session short before the second fix could be independently re-verified live — **do that first next session**, see `project_production_readiness_audit_2026_08_07.md` memory for the exact re-check steps. **Bug 1 — citation panel shows the wrong drug's evidence.** Clicking citation `[1]` on an OLDER answer (Omeprazol's own mechanism-of-action citation) displayed a completely unrelated LATER drug (Kanamycin) in the "Bằng Chứng Dược Thư" panel, with the beam-connector line pointing at it too. Root cause: `page.tsx`'s `handleCitationClick(citation, index)` received the correct per-message `citation` object from `ChatBubble` but discarded it, only ever setting `activeCitationIndex` — the panel's `citations` array itself stayed whatever the MOST RECENTLY LOADED answer's list was (set once by `onCitationsLoaded`), never refreshed per click. Any older message's marker index just indexed into that stale, unrelated array. Fixed by threading the clicked message's own citation array through the whole chain (`ChatBubble.tsx`'s `onCitationClick` now passes `allCitations` too → `ChatPanel.tsx` passes it through → `page.tsx` calls `setCitations(allCitations)` before setting the index). Live-verified. **Bug 2 — a good, specific abstain reason gets thrown away for generic boilerplate.** Asked Kanamycin's eye-drop strength; got "Dược thư có nội dung liên quan... nhưng hệ thống chưa xác định đủ cơ sở, vui lòng thử lại" — unhelpful. Manually retrying revealed the model's real, correct judgment was available the whole time: "Bằng chứng không nêu liều dùng đường nhỏ mắt của thuốc này" — specific, honest, actually useful. Root cause: `rag/prompt.py`'s answer contract (rule 5 + `ANSWER_SCHEMA`) allowed `clarifying_question` to stay `null` even when `evidence_sufficient=false` for the "source genuinely lacks this content" case (rule 7's mandate only covered the narrower "user needs to specify more" case) — so whenever the model happened to omit it, `answer.py::_generate` fell through its one internal retry straight to the generic `reject_reason="evidence_insufficient"` → `REFUSALS` boilerplate, discarding the specific reasoning the model actually had. Fixed: prompt rule 5 and the schema's `clarifying_question` description now REQUIRE a short honest explanation whenever `evidence_sufficient=false`, covering both the "ask the user for more" and the "the book doesn't cover this" cases. **Not yet independently live-reverified after the last restart** — session ended mid-check. Full suite 196/196 passing at commit time (no test changes needed for either fix — Bug 1 is TS-only, Bug 2 is a prompt-text-only change, no logic/schema-shape change). ## 2026-08-07 (cont. 14) — Fixed the P0 clarify-loop bug from cont. 13's audit User approved fixing the top blocker from the 50-question audit: the non-terminating multi-turn clarify loop. Two complementary fixes, both in `rag/understanding.py` and `rag/agent.py` (F-11), plus one more real bug the user separately reported live mid-session. **Third live repro found while working**: user typed a correction — "tôi có hỏi liều uống đặt trực tràng đâu" (a negation: "I never asked about the rectal dose") — after the bot answered the wrong route. The bot just repeated the same wrong-route answer, ignoring the correction entirely. Same root cause family as the other two: no dedicated handling for "the user is refuting my last answer," so the model re-derives the same wrong reading. **Fix 1 — structural, `understanding.py`**: `LlmQueryUnderstander.understand()` now takes an optional `prior_frame: QueryFrame`. On a clarify-continuation turn, its known fields (drugs/population/weight/age/route/indication/ attribute) are (a) stated explicitly in the prompt as a "THÔNG TIN ĐÃ XÁC ĐỊNH" block instead of relying on the model to re-derive them from a raw text transcript, and (b) merged back onto the new turn's parsed frame in code (`_merge_with_prior_frame`) whenever this turn doesn't itself resolve a *different* drug — so a dropped field is a non-event, not a re-ask. Guarded: merge/known-block only fire when `prior_frame.needs_clarify` was true (a resolved prior turn has nothing to continue) and never overrides a turn that names its own different drug (that's a real topic change, must not inherit stale slots — this is the direction the OMEPRAZOL bleed ran). Two new `_SYSTEM` prompt rules cover what the merge can't: (1) explicit "this is a brand new unrelated topic, don't carry the old drug over" guidance for the bleed case, (2) explicit "the user is negating/correcting my last answer, don't repeat it — ask what they actually meant" guidance for the rectal-dose correction case. **Fix 2 — circuit breaker, `agent.py`**: `RagAgent` tracks a per-conversation consecutive-clarify streak. After `MAX_CONSECUTIVE_CLARIFY = 4` clarify decisions in a row, it force-abstains with an actionable message ("gõ lại toàn bộ câu hỏi... hoặc bấm Tạo phiên tra cứu mới") instead of asking again. Any non-clarify decision resets the streak. This is the backstop that guarantees no user gets stuck forever regardless of how well Fix 1 works — every other failure mode in this file already degrades to a bounded abstain; this was the one path with no bound at all. **Tests**: 9 new (5 `test_understanding.py` — merge survives a dropped field, merge skipped on a genuine drug change, merge skipped when prior was already resolved, known-facts block present/absent in the actual prompt sent; 4 `test_agent.py` — breaker fires at the threshold, streak resets after the hard stop so the conversation isn't permanently locked, a resolved turn in between resets the streak, `prior_frame` is correctly threaded from the previous turn). All 6 existing fake-understander test doubles in `test_agent.py` updated for the new `prior_frame` kwarg. Full suite 187 -> 196 passed, 5 skipped, no regressions. **Live-verified in the actual browser** (ai-service restarted, no `--reload` per house rule): replayed both reproduced bugs from cont. 13 end to end. "Bảo quản Insulin" -> "Chưa mở lọ" -> "Insulin người" -> "Regular": no longer loops — asks 3 genuinely different narrowing questions (real progress, not a repeat) then the circuit breaker cleanly hard-stops with the actionable message. "Cơ chế tác dụng của Omeprazole" (answered) -> "Tôi bị đau đầu nên uống thuốc gì?": no longer mislabeled OMEPRAZOL or asks for body weight — correctly asks "Anh/chị muốn dùng thuốc gì cho đau đầu? Ví dụ: paracetamol, ibuprofen..." with no stale drug attached; answering "Paracetamol" converges immediately to a correct, grounded, cited answer. Did not re-run the third (rectal-dose correction) case live this session — covered by the same prompt rule mechanism just verified working for the other two, not independently browser-replayed. **Also added**: `clarify_loop_exhausted` entry in `apps/web/app/api/chat/ route.ts`'s `REFUSALS` map (fallback only — the backend supplies its own Vietnamese `answer` text for this reason, same pattern as every other agent- inline abstain since cont. 9's fix). **Still open from cont. 13's audit**, not touched this session: api-gateway/auth-service/user-service/chat-service remain unbuilt scaffolds; the elevated live noisy-retry rate hasn't been re-measured outside heavy test conditions; the 4 persistent (non-loop) abstains from the 50-question run are unchanged; the quarantined-table/`VERIFY_PDF` citation path still hasn't been conclusively exercised. ## 2026-08-07 (cont. 13) — Post-reboot production-readiness audit: 50 hand-typed live browser questions, verdict NOT READY Machine crashed/rebooted mid-session (all background processes killed, Docker Desktop down). Recovered clean: Postgres/Qdrant containers restarted, Qdrant collection intact (15,100 pts), migrations re-applied, ai-service (`:8079`, no `--reload`, per house rule) and web (`:3000`) restarted. Full pytest suite 187 passed/5 skipped immediately after — no regression from the crash. **User then directed a full manual audit**: read the codebase, then type 50 real questions by hand into the actual Chrome UI (not curl) and judge pass/ fail on the rendered answer + citation card, explicitly forbidding any other verification method. Did exactly that — every one of the 50 below was typed into the live textbox, submitted, and judged from the rendered DOM. **Architecture finding (new, not previously logged this precisely)**: `api-gateway`, `auth-service`, `user-service`, `chat-service` are ALL still empty scaffolds (`package.json` + `README.md` only, confirmed via directory listing). `apps/web`'s `route.ts` talks directly to `ai-service:8079` — this is the entire real live path, not a dev shortcut (matches cont. 8's finding). No auth, no gateway rate-limiting, no persisted-by-a-real-backend chat ownership exists yet. **Bug found and fixed before the 50-question run**: user separately reported "mẫu tra cứu nhanh đang bị gửi 2 lần" (quick-prompt sidebar buttons double-sending) — reproduced immediately via a stray click. Root cause: `ChatPanel.tsx`'s `useEffect(() => { if (initialQuery) handleSendMessage(...) }, [initialQuery])` had no guard, and Next.js dev-mode React 18 Strict Mode double-invokes effect setup — each quick-prompt click fired two live identical `/api/chat` POSTs. Fixed with a `useRef` sentinel that records the last-sent `initialQuery` value, persists across the Strict Mode replay, and still sends once for a genuinely new query (`ChatPanel.tsx`). Verified live: one click -> one user bubble -> one POST in the Next.js server log. **50-question live results** (drug resolution + citation page always spot-checked against real pharmacology, not just "did it answer"): - **~40/50 eventually correct** (right drug, right section, citation page matches evidence text) — many only after 1-3 manual "Thử lại" clicks. - **4 persistent abstains** (still wrong after 2-3 retries, not noise): "Thận trọng Aspirin + loét dạ dày" (`evidence_insufficient` 3/3), pediatric weight-based Azithromycin dosing (`dosing_calc` — matches the already-known open F-10 gap), Aspirin+Warfarin interaction (`unsupported_claim` 2/2), Vancomycin rapid-infusion caution (`unsupported_claim` 2/2). - **A systemic multi-turn bug, independently reproduced 3 times in 3 unrelated threads**: mid-clarify-chain, the understanding LLM loses already-established context and either (a) re-asks the exact same clarify question forever (Insulin storage: 5 real answered turns, never converged), (b) forgets an already-stated population and re-asks it (Azithromycin: "bé nặng 20 cân" established, then asked "trẻ em hay người lớn?" again), or (c) drags in a stale unrelated drug from earlier history into a brand-new topic (headache/tension question suddenly labeled OMEPRAZOL, asking for the user's body weight for a recommendation-seeking headache question). This is the single biggest production blocker found this session — a real multi-turn user conversation has a good chance of getting stuck in a non-terminating clarify loop with no escape except starting a new session. - **Two minor precision bugs**: an English-language query ("...for adults") had its explicit population ignored, re-asked for a child's weight; a self-referential question ("Cefotaxime dùng đường nào?" — asking what routes exist) was misread as "which route do you want," asking the user to pick one instead of just listing them. - **Safety nets held up well** in every adversarial case: empty input blocked client-side, gibberish/prompt-injection/English/very-long-repeat input never hallucinated, non-human ("thuốc cho chó") and out-of-corpus (Part 1/3 topics, BSA table) correctly abstained honestly, recommendation- seeking ("tôi đau đầu nên uống gì") correctly did NOT hit the recommendation-refusal gate (per [[feedback_no_recommendation_gate]]). - **Noisy-entailment retry rate looked meaningfully higher than the documented ~4% estimate** from cont. 11 — a large fraction of the 50 needed at least one manual retry to get a real answer. Not conclusively separated from this session's own heavy sequential test traffic; flagged as worth re-measuring under light/normal traffic before trusting the old 4% number for a capacity/SLA decision. - Did not conclusively exercise the quarantined-table/`VERIFY_PDF` path — one attempt (corticoid dose-equivalence table) correctly hit generic out-of-scope instead, not quarantine specifically; needs a targeted follow-up with a known quarantined chunk id. **Verdict**: chatbot is **NOT ready for production**. The RAG/grounding/ citation core is genuinely strong (correct drug+section+page on the large majority of single-turn questions, real safety gates holding under adversarial input) but three things block a ship decision: (1) the non-terminating multi-turn clarify loop — a real, frequent, user-facing dead end, not an edge case; (2) `api-gateway`/`auth-service`/`chat-service` are unbuilt, so there is no auth, no rate limiting, and conversation history lives only in Postgres keyed by a client-supplied `conversationId` with no ownership check; (3) the elevated live retry rate needs re-measurement outside of heavy test conditions before any latency/cost SLA is claimed. ## 2026-08-07 (cont. 12) — REAL root cause of the repeated live failures found: O(history × aliases) candidate resolution, not Bedrock at all User, rightly frustrated that every fix so far was verified via curl/API calls instead of the actual browser ("mở chrome lên gõ tay chat xem thế nào" — go open Chrome, type by hand, see for yourself), directed hands-on browser testing. That reproduced the failure directly: typed a real question by hand, watched it load 30+ seconds, watched it fail with "Dịch vụ đang gặp sự cố tạm thời" — the exact live symptom, not a hypothesis. **Diagnosis, with hard numbers, not guessing**: added timing instrumentation to `RagAgent.handle()` (`t0..t4` around history/understand/route/remember). First real capture: `understand=51.44s` — the understanding call was taking nearly a minute, failing on `RequestBudgetExhausted` before ever reaching Bedrock. Traced into `understanding.py::_candidate_ids`, which calls `CatalogDrugResolver.resolve()` + `.suggest()` once per line of `(turn, *history)` — up to 13 lines per turn. Direct isolated timing: `resolve()` ≈0.65-0.7s, `suggest()` ≈0.94-0.97s per call, over the real 10,164-alias catalog (regex per alias in `resolve`, `SequenceMatcher` per alias in `suggest` — both O(aliases)). **≈1.6-1.7s of pure CPU per history line, called fresh on every single turn — including lines already resolved in every prior turn of the same conversation.** A real multi-turn conversation's accumulated history alone was enough to blow the 20s F-08 budget before the first LLM call ever ran — this had nothing to do with Bedrock, throttling, or the earlier `adaptive`-mode regression; those were real but secondary. This is why the failure was reproducible and worsening turn-over-turn in an actual chat session, not a flaky one-off a single curl call would ever catch. **Fix**: `functools.lru_cache(maxsize=4096)` on `CatalogDrugResolver.resolve` and `.suggest` (`rag/routing.py`) — both are pure functions of their arguments (fixed `self._aliases`/`self._catalog` set once at construction, single read-only caller). Verified in isolation: first pass over 3 lines 5.3s cold, identical second pass **0.0s** (full cache hit) — turns all but the newest line into a dict lookup on every subsequent turn. Full suite 187 passed after. **Live-verified in the actual browser** (not curl): fresh session, "Chống chỉ định của Aspirin là gì?" answered correctly in <10s; immediate follow-up "Liều dùng người lớn thì sao?" (real multi-turn, history now populated) completed in ~15s with `resolved_drug_id` correctly carried over from the first turn — no more `understanding_provider_unavailable`. (That specific follow-up then hit `unsupported_claim` — the separate, already-known noisy-entailment case from cont. 11, not this bug; reported via the new granular reason with an honest message.) **Lesson, stated for the next session**: this was invisible to every curl-based check this session ran, including dozens of them, because a single stateless curl call never accumulates the history that made the cost compound. Only driving the actual multi-turn chat surfaced it. The standing house rule to drive the real chat, not just the API, is not optional politeness — it is what found the actual bug after several API-level "verified" claims that were all individually true but collectively missed the real, user-facing failure. ## 2026-08-07 (cont. 11) — Noisy-check retries, granular error codes, adaptive-mode regression found and reverted same session Follow-up to cont. 10 at the user's direct request: implement the 2 named retry improvements, then a serious live incident hit mid-work. **Retry widening (`rag/answer.py`)**: `_verify_entailment` widened from 2 to 3 attempts (accept on any accept, discard only if all 3 reject) — math check: single-call noise ~33% (2026-08-06 probe) makes 2-attempt discard rate ≈ q²≈11%, matching the observed live abstain rate almost exactly; 3-attempt drops it to ≈q³≈4%. New `_attempt_generation`/`_RawAttempt` split lets `_generate` retry once on a lone `evidence_sufficient=false` with no `clarifying_question` — live-verified: the acid-ascorbic renal-threshold case that abstained now returns the correct answer 3/3 on fresh retry. **Trade-off stated in code, not hidden**: both changes let a genuinely bad claim survive on 1-of-N noisy accepts instead of 1-of-2 — accepted since the probed noise is symmetric, not because residual risk is zero. 5 new tests (`test_grounded_generation.py`) lock in both the recovery and the still-discards/still-abstains cases. Full suite 187 passed after. **Regression found and reverted same session**: while implementing the above, switched Bedrock retry config from `mode: "standard"` to `"adaptive"` (in `adapters/embedding.py`, `adapters/bedrock_converse.py`) — intended to fix throttling, but adaptive mode's client-side rate limiter remembers "throttled" ACROSS requests and paces down even healthy, unrelated calls after a burst. This session's own heavy adversarial test traffic (50-question harness + repeated manual retries) tripped it, and a normal single answerable turn went from ~9s baseline to a measured 1-5 minutes for the user, live, mid-session. Reverted to `mode: "standard"` same session; 3 fresh timing checks after revert: 9.1s / 10s / 10.9s — back to baseline. Lesson: `mode: "adaptive"`'s cross-request memory is exactly wrong for a service that gets bursty *test* traffic sharing the same client/quota as real traffic — `"standard"`'s per-request-independent backoff has no such failure mode. `max_attempts: 3→4` kept either way. **Granular error codes (user's direct request, after this incident)**: every abstain reason had already collapsed into `reason="generation_unavailable"` by the time it reached the API/trace — a real provider outage was indistinguishable from ordinary entailment noise without reading `/metrics` by hand. `_GenOutcome` gained `reject_reason`, threaded through every rejection branch of `_generate`, so `answer_from_result` now returns the SPECIFIC reason (`provider_unavailable`, `malformed_output`, `evidence_insufficient`, `ungrounded_number`, `invalid_citation`, `uncited_claim`, `unsupported_claim`, `request_budget_exhausted`) instead of the generic catch-all. Live-verified: the Risperidon contraindications case (still genuinely abstaining after the retry fix — a real residual case, not eliminated, now at least diagnosable) returns `reason="evidence_insufficient"` instead of the old opaque `"generation_unavailable"`. Also fixed `rag/understanding.py`'s `except AnswerGenerationUnavailable` block, which silently swallowed the real exception with ZERO logging anywhere — now logs `type(exc).__name__: exc` and sets a new `QueryFrame.system_error` field so `RagAgent._route` surfaces `understanding_provider_unavailable`/ `understanding_malformed_output` instead of the same generic `"needs_more_info"` a real clarifying question gets (these were previously indistinguishable from the outside). `apps/web/app/api/chat/route.ts`'s `REFUSALS` map extended with all 8 newly-surfaced `answer.py` codes — this is the SAME class of bug fixed in cont. 9, reintroduced by this round's own propagation change, caught and closed same session rather than left for a future live report. **Also found and fixed**: `ChatPanel.tsx`'s "Thử lại" (retry) button resent the ASSISTANT bubble's own text as the next query instead of the original user question — live-confirmed via a trace row where the query text WAS literally "Dịch vụ đang gặp sự cố tạm thời...". Fixed to walk back to the nearest preceding `role: "user"` message. **Open, named rather than hidden**: the Risperidon case's root cause (why THIS specific short contraindications passage draws a consistent `evidence_insufficient` verdict across many attempts, not just noise) is unsolved — worth a dedicated prompt-tuning look, not chased further this round. User separately proposed a verbatim/no-LLM-generation fast path for simple single-section lookups to cut the ~8-9s baseline under 7s — real idea (the extractive mode already exists for `ANSWER_PROVIDER=disabled`, this would be a per-request criterion instead) — not yet started, awaiting go-ahead given its product-level UX impact. ## 2026-08-07 (cont. 10) — Bedrock retry/backoff + full reason-code contract audit Follow-up to cont. 9: user agreed to skip `dosing_calc` (out of scope) and do the other two named items. **Retry/backoff.** All 3 live boto3 Bedrock client sites (`adapters/embedding.py`'s query embedder, `adapters/bedrock_converse.py`'s answer generator AND reranker) were already retrying transient errors, but only with `mode: "standard"` (reactive backoff-after-failure) and `max_attempts: 3` — measurably not enough during today's throttling burst (the two consecutive "Dịch vụ đang gặp sự cố tạm thời" turns from cont. 9). Switched all 3 to `mode: "adaptive"` (client-side rate limiting that backs off proactively once throttling is detected, botocore-native, no custom retry code) with `max_attempts: 4`. Also gave `adapters/bedrock_claude.py`'s Anthropic-SDK client `max_retries=4` (was the SDK default of 2) for consistency, even though this provider isn't the one actually configured (`ANSWER_PROVIDER=bedrock-converse` per `.env`) — kept in sync in case it's ever selected. Full `pytest -q` suite (184 passed, 5 skipped) re-run clean after the change; ai-service restarted and re-verified live against the same Clonazepam query (still `answerable`/`grounded_evidence_available`, 2 citations). **Contract audit.** Enumerated every `reason=`/`EvidenceDecision.ABSTAIN` site across `rag/routing.py`, `rag/service.py`, `rag/answer.py`, and `rag/agent.py`'s own inline `AgentReply` construction — the full universe of values `routers/rag.py` can put in the API response's `reason` field. Cross-checked each against whether it can reach the frontend with `answer: null` (the only case `REFUSALS` needs to cover — every agent.py inline abstain path already supplies its own real answer text, bypassing the map entirely) and confirmed all are now mapped after cont. 9's fix. Also confirmed `chat-service` (the NestJS hop the documented production topology routes through) has no source files yet — it's unbuilt scaffold — so `apps/web`'s `route.ts` talking directly to `ai-service` on :8079 (per `.env`'s `API_GATEWAY_URL`/`AI_SERVICE_URL`) really is the entire live path today, not a dev-only shortcut with a separate prod code path that could hide the same bug class. No second copy of this mapping exists anywhere else to audit. ## 2026-08-07 (cont. 9) — Live bug report investigated and fixed: misleading abstain message User reported "lỗi nghiêm trọng" (serious error) seen live in the chat UI. Server logs showed zero exceptions/tracebacks and all HTTP 200s, so this was not a crash — required reproducing the actual browser session to find it. **Root cause**: `apps/web/app/api/chat/route.ts`'s `REFUSALS` map only covered 7 of the ~16 abstain `reason` codes the backend can actually emit (enumerated by grepping every `EvidenceDecision.ABSTAIN`/`reason=` site across `rag/routing.py`, `rag/service.py`, `rag/answer.py`). Any unmapped reason silently fell through to a generic string that claims "Hệ thống không tìm thấy căn cứ trong Dược thư" (system found no grounds in the formulary) — **false** for the case that actually happened: asking "dược động học của Clonazepam" (a real monograph section) hit `reason="generation_unavailable"` (`rag/answer.py:208`) — retrieval succeeded, generation/entailment failed its own safety check (this call is documented-noisy, see ADR 0008) and correctly abstained — but the frontend told the clinician the drug had no data at all. Confirmed by replaying the exact same query directly against `/v1/rag/query`: it returned a full, correctly-grounded, cited pharmacokinetics answer on retry. A doctor reading "not found in the formulary" for a drug that IS in the formulary is a real safety-adjacent UX bug, not a cosmetic one. **Fix**: added entries for all 9 previously-unmapped reason codes (`generation_unavailable`, `query_intent_unknown`, `drug_resolution_invalid_state`, `missing_query_or_drug`, `missing_indication`, `no_indication_match`, `parent_hydration_failed`, `missing_provenance`, `missing_printed_page_provenance`), each with an accurate message — `generation_unavailable` explicitly says the formulary DOES have related content and suggests retrying, instead of denying the data exists. Narrowed `GENERIC_REFUSAL` itself from the false "no grounds found" claim to a neutral "cannot process this right now, please retry", since it should now only ever fire for a genuinely unclassified reason. Verified live: hit `/api/chat` directly post-fix with the same Clonazepam query (Next.js dev server hot-reloaded the route with no restart needed) and got the correct grounded answer end-to-end through the real frontend API route, not just the backend. **Also investigated, not a bug**: the same browser tab showed two earlier turns ("Chống chỉ định & Thận trọng khi dùng Amoxicillin") failing with "Dịch vụ đang gặp sự cố tạm thời" — `understanding.py`'s F-10 fail-closed path for a real `understand()` LLM-call failure. Retried the identical query 3x directly against the backend just now: all 3 succeeded with a normal, correct clarify ("người lớn hay trẻ em?"). This looks like a transient Bedrock hiccup/throttle from the session's own rapid testing traffic, not a persistent fault — fail-closed behaved exactly as designed (a graceful clarify message, not a crash), so no code change made here. ## 2026-08-07 (cont. 8) — Phase 5/5 (final): faster-model A/B — negative result, current model kept Ran a live A/B (real Bedrock calls, real catalog/resolver) comparing the current understanding model (`qwen.qwen3-next-80b-a3b`) against the fastest plausible candidate already IAM-permitted (`qwen.qwen3-32b-v1:0`) on a 6-case battery covering today's actual hard cases: simple dose resolution, a fake-drug safety check, 2-drug interaction, symptom_to_drug, weight extraction, and the exact multi-turn route-resolution case fixed earlier today. **Speed**: confirmed, qwen3-32b is genuinely faster — roughly 2.3-2.9s per call vs 2.7-4.5s for the current model on most cases (~30-40% faster). **Two real regressions found, one safety-critical — recommendation: do NOT wire it in.** 1. **Safety.** Asked about the fake drug "aspirinol" (this project's standing regression case for F-04's catalog-bounding), the current model correctly leaves it as `unknown_drugs=('aspirinol',)`. The 32B candidate silently resolved it to the real `acid_acetylsalicylic_aspirin` with no `unknown_drugs` entry at all — and because "aspirinol" fuzzy-matches "aspirin" closely enough to appear in the turn's deterministic candidate set, this substitution is NOT caught by F-04's own candidate-bound check (`_resolve_id` only rejects an id that's outside the shown candidates; this one is inside it). A live user asking about a genuinely nonexistent drug would silently get an answer about aspirin instead, with no indication their drug name didn't match anything. 2. **Instruction-following.** On the exact "Uống" multi-turn case fixed earlier this session (route resolution after a short reply to the model's own prior clarify question), the 32B candidate correctly extracted `route=uong`/`population=nguoi_lon` into the frame fields — the schema/field-level fix from earlier holds regardless of model — but still set `needs_clarify=True` and re-asked a version of the original question, undoing the point of today's earlier fix. The 80B model correctly proceeded (`needs_clarify=False`). This matches a pattern already in memory from 2026-08-05 (`[[project_llm_cloud_live]]`): DeepSeek V3.2 silently ignored the clarify-don't-dump instruction on a different task, which is why Qwen3-80B was chosen in the first place. Smaller models in the same family trading away exactly this kind of careful instruction-following for speed is consistent with that prior finding, not a one-off fluke. **No code changed** — per the plan's own stated criterion ("only wire it in if the smaller model matches quality on the battery; otherwise document the negative result"), this is a complete, valid Phase 5 outcome. `gpt-oss-20b` was not tested — the qwen3-32b result already gives a clear, evidenced negative for the "smaller Bedrock model for understanding" approach in general, and further model exploration should wait for a specific reason to revisit it rather than open-ended search. --- **All 5 phases of the owner-approved plan (`~/.claude/plans/pure-wobbling-llama.md`) are now done or resolved**: symptom_to_drug (built + live-verified), F-10 (built + found/fixed a real unhandled-500 bug), F-08 (built + live-verified), durable conversation history (built + verified across a real process restart), faster understanding model (evaluated, negative result documented, current model kept). `apps/ai-service`: 184 passed. ## 2026-08-07 (cont. 7) — Phase 4/5: durable Postgres conversation history, verified across a real restart **Built**: `adapters/postgres.py::PostgresConversationStore` — same established pattern as `PostgresTraceRepository` (`connect_timeout=5`, one connection per call, no pooling — F-09's accepted tradeoff). Append-only `rag_conversation_turn` table (`migrations/002_rag_conversation_turn.sql`); `id bigserial` insertion order is the "oldest -> newest" ordering the understanding prompt already expects, no separate turn-index column needed. `recent(conversation_id, limit)` windows at READ time (`ORDER BY id DESC LIMIT`), so — unlike the in-process dict it replaces — writes never need to delete old rows; old history just sits unused past the window (same unbounded-growth tradeoff the trace table already has, not a new gap). `RagAgent` gained an optional injected `store: ConversationStore | None` (a tiny local Protocol — not a resurrection of the deleted `conversation.py`'s `ConversationStore`, which was tied to the removed Focus/TTL design). `None` (the default) keeps every existing behavior byte-for-byte unchanged. When configured, `_get_history`/`_remember` read/write through the store instead of the dict, and fail OPEN on any store error — same F-09 fail-open convention as the trace writer, applied by direct analogy rather than a new exception type: read failure → empty history this turn (fresh understanding, not a 500); write failure → this turn's memory is silently lost, the already-computed response still returns. Wired into `bootstrap.py` next to `PostgresTraceRepository`. `migrate.py` now applies both migrations. **Live-verified the actual capability being added, not just the plumbing**: sent turn 1 ("Liều paracetamol hạ sốt là bao nhiêu?", `conversation_id` set) to the real running server → clarify as expected. **Killed and restarted the whole ai-service process** (a fresh Python process, empty in-process dict — under the old design this conversation's memory would be gone). Sent turn 2 ("Người lớn", no drug name at all) with the same `conversation_id` → response's `resolved_drug_id` came back `paracetamol_acetaminophen`, which is only possible if the understanding call received turn 1's history from Postgres, since nothing in-process survived the restart. This is the one live check that actually proves the feature, as opposed to proving the code merely doesn't crash. `apps/ai-service`: **184 passed** (was 181; +3 unit tests with a fake store covering round-trip/read-failure/write-failure). Also added `test_real_postgres_conversation_store_round_trip` to `test_live_datastores.py` (RUN_INTEGRATION=1-gated, matching the existing pattern) — run against the real dev Postgres, passed, covers windowing at the read boundary and an empty read for a never-seen `conversation_id`. **Known limitation, named not hidden**: no retention/cleanup job — the table grows forever, same as `rag_retrieval_trace` already does. Not addressed here; a reasonable follow-up if either table's growth becomes an operational concern. ## 2026-08-07 (cont. 6) — Phase 3/5: F-08 request-scoped budget built and live-verified **Built**: new `rag/budget.py` — `RequestBudget` (deadline + call-count, both must hold) and `RequestBudgetExhausted` (subclasses `AnswerGenerationUnavailable` deliberately, so every existing fail-open/fail-closed handler in the codebase catches it with zero changes — budget exhaustion IS "the provider is unavailable to us right now" from each call site's perspective). Not a resurrection of the deleted `reasoning.py`'s heavier `TurnBudget` — that was tied to the retrieval-refinement loop this system no longer has; this is just a counter + a deadline, checked once per call. `RagAgent.handle()` constructs one `RequestBudget` per turn (defaults: 20s wall clock, 8 calls — sized with headroom above the measured normal case of 4-5 calls / ~8-9s, so ordinary traffic never trips it) and threads it through every LLM call site: `understanding.understand()`, `answer.answer_from_result()` → `_check_sufficiency`/`_generate`/ `_verify_entailment`/`_run_entailment_check`. Each calls `budget.require()` immediately before its actual provider call — exhaustion means the real network call never happens, not that it happens and then gets discarded. Config: `Settings.max_wall_clock_ms`/`max_llm_calls_per_turn`, wired into `bootstrap.py`'s `RagAgent` construction. One deliberate asymmetry, matching each site's existing failure-direction: `_check_sufficiency` fails OPEN on budget exhaustion (skips the clarify heuristic, proceeds to generate — it's a UX heuristic, not a safety gate); every other site fails CLOSED (abstain/reject) — this was already true for provider outages before F-08, budget exhaustion now follows the identical rule at each site rather than introducing a third behavior. **Live-verified two ways**: normal query with the default budget answers unchanged (~7.4s, same as before F-08). A `max_llm_calls_per_turn=1` agent against the real Bedrock/Qdrant stack correctly aborts after the one understand call, cleanly abstains (`generation_unavailable`, no crash, no fabricated answer) instead of proceeding — proving the mechanism holds end to end, not just in unit tests. Note on what this does and doesn't prove: the latency saving in this specific case was modest (~6.2s vs ~7.4s) because `understand()` alone already dominates a normal turn's cost — the budget's real value is bounding the pathological case (one call stuck retrying for minutes against `read_timeout=60s` × up to 3 attempts), which was not separately fault-injected live this session; that would need a deliberately broken/slow fake provider, a reasonable next step if this needs stronger evidence. `apps/ai-service`: **181 passed** (was 173; +8: 6 direct `RequestBudget` unit tests, 2 end-to-end `RagAgent` tests proving a spent budget blocks the generator from ever being called, with a control test proving the same setup succeeds normally under the default budget). ## 2026-08-07 (cont. 5) — Phase 2/5: F-10 adversarial battery — found and fixed a real unhandled-500 bug **Real bug found, not just tests added.** `rag/understanding.py::LlmQueryUnderstander.understand()` was the ONE LLM call site in the whole product with no error handling around it — every other call (`answer.py`'s sufficiency/generate/entailment) catches `AnswerGenerationUnavailable` and fails closed, but `understand()`'s `self._llm.generate(...)` had no try/except, and `routers/rag.py` only wraps the trace-save call, not `agent.handle()` itself. A Bedrock outage during understanding — the FIRST call of every single turn — would have propagated into an unhandled 500 instead of a graceful abstain. Found by asking "what does F-10's provider-outage-mid-conversation category actually cover today" and checking each of the 4 call sites by hand, not by running anything. Fixed: wrapped, fails closed to the same `needs_clarify` shape the JSON- parse-failure path already uses, with an honest "dịch vụ đang gặp sự cố" message instead of "tôi chưa hiểu câu hỏi" (the failure is the service's, not a misunderstanding of the user's phrasing). **Also pinned, not previously tested**: `_check_sufficiency`'s outage behavior is a deliberate fail-OPEN (skip the clarify heuristic, proceed to generate — grounding/entailment remain the real safety net), unlike every other failure mode in the service which fails closed to abstain. This was already the code's behavior; now there's a regression test locking it in as intentional rather than an accident nobody would notice changing. **New coverage**: `conversation_id` presence/absence reaches the same decision on a fresh turn (by construction — both see empty history — now a regression-guarded fact, not just an inference from reading the code); 2 more fake-drug-near-alias shapes beyond the existing `aspirinol` case (brand-like suffix on a real name, a name blending two real drugs) both confirming F-04's candidate-bound rejects even a real catalog id with no turn-specific support. Prompt-injection resistance and the entailment-judge noise case were **not** newly tested — the former only really tests anything with a fake LLM if the "compromise" changes the OUTPUT shape (covered by the near-alias/catalog-bound tests above, which are exactly that); genuine adversarial prompt resistance needs the real model, and today's many live queries already incidentally exercised it without incident. The entailment-judge noise case (warfarin/aspirin, 2026-08-06) was not specifically re-run live this session — time-scoped out, not forgotten. `apps/ai-service`: **173 passed** (was 168). Server restarted, confirmed normal operation unaffected by the fix. ## 2026-08-07 (cont. 4) — Phase 1/5: symptom_to_drug reverse lookup built and live-verified Owner approved a 5-phase plan (`~/.claude/plans/pure-wobbling-llama.md`) for the remaining backlog: symptom_to_drug, F-10 adversarial tests, F-08 request budget, durable conversation history, faster understanding model. Phase 1 done. **Built**: `QdrantRetriever.find_by_indication` (keyword phrase match on `chi_dinh`-section prose chunks, deterministic) + `search_indication` (dense vector fallback restricted to `chi_dinh`, tried only when keyword finds nothing — the one place in the live path dense search is actually used, per ADR 0008). `RetrievalService.retrieve_by_indication` orchestrates the two. `RagAgent._symptom_to_drug` wires this into the `symptom_to_drug` turn type (previously an honest "not ready" clarify), reusing `GroundedAnswerService.answer_from_result` with a new `list_mode` flag so citations/grounding/entailment apply unchanged. **Two real bugs found and fixed by driving it live**, not just unit tests: 1. Without `list_mode`, the generation prompt picked ONE drug out of 8 real symptom matches and silently dropped the rest — `rag/prompt.py::build_request` gained a `list_mode=True` branch instructing the model to enumerate every matching drug (never a ranking — `[[feedback_no_recommendation_gate]]`), and `answer_from_result`/`_generate` thread it through; the sufficiency clarify (right for a single dose question) is skipped in this mode since it doesn't fit a reverse lookup. Verified: "sốt" now correctly cites both paracetamol and artesunat, not just one. 2. The first keyword-matching design (token-SUBSET: every word present *somewhere*, any order) let a long nonsense query built from common filler words ("bệnh chưa từng ghi nhận trong sách…") false-positive against real `chi_dinh` text, reaching a wasted generation call before entailment correctly rejected it. Switched to a word-boundary-anchored CONTIGUOUS phrase match — precise by construction, dense search remains the deliberate fallback for genuine paraphrases. **Known remaining imperfection, not chased further today**: the dense fallback's `minimum_score` gate (reused from `EvidencePolicy`, 0.12) doesn't reject a nonsense query's weak matches before generation — Cohere embed-v4 similarity for unrelated Vietnamese medical text apparently sits above 0.12 often enough that the gate rarely fires. The **safety outcome is still correct** (grounding/entailment cleanly abstains, no fabrication, verified live) — this is a wasted-generation-call efficiency cost, not a correctness gap, and tuning the exact right threshold is a separate exercise from today's scope. `apps/ai-service`: **168 passed** (was 153 before this phase). Live-verified against the real running server (restarted after each code change): "sốt" → real 2-drug answer with citations; a nonsense phrase → clean abstain. ## 2026-08-07 (cont. 3) — Item 3 of yesterday's Top 3 closed: dead reasoning-loop deleted, docs reconciled with what's live Owner said to go ahead and fix the last of yesterday's "Top 3 picked for next session" items: reconcile `architecture.md`/ADR 0007 with what's actually live. **Investigated before touching anything.** Confirmed by grep, not assumption: `rag/reasoning.py`, `rag/conversation.py`, `rag/conversational.py` (1,314 lines) have zero live importers — not in `bootstrap.py`, `main.py`, `agent.py`, `answer.py`, or `routers/rag.py`. Their only consumers were their own 5 dedicated test files (42 tests). `rag/ports.py` never actually gained the `ConversationStore`/`Summariser`/`Planner`/`SufficiencyAssessor` protocols ADR 0007 planned for it, and `adapters/postgres.py` never gained `PostgresConversationStore` either — the whole design was implemented as free-standing modules, then never wired in, confirming the audit's finding that it's genuinely dead, not "integration pending." **Decision: delete + document reality, not wire the old design in.** The live `RagAgent` (LLM-driven one-shot pipeline, plain-history multi-turn) has been proven working across many real multi-turn conversations today and yesterday — including cases ADR 0007's design was explicitly written to handle (follow-up inheritance, under-specified dose clarify). Reviving `Focus`/`ConversationState`/TTL/the PLAN-REFINE loop would mean reintroducing exactly the state-machine complexity `agent.py`'s own docstring says was deliberately removed. ADR 0007 section 6 ("Refused: an LLM confidence score as the loop's uncertainty signal") is itself evidence this was a genuine architecture pivot, not an unfinished build — the live system now uses exactly that judgment as its ask/answer signal. **Done:** - Deleted the 3 dead modules + 5 dedicated test files. `apps/ai-service`: **153 passed** (was 195; 42 tests removed with the dead code, nothing else broke — confirms they were truly isolated). Server restarted, boots clean, a real query still answers correctly. - `docs/adr/0007-conversational-reasoning-rag.md`: status changed to "superseded by ADR 0008," with a note explaining why and pointing to what it got right that's still owed (F-08 request budget, a durable cross-worker conversation store). Kept unedited below the notice — an ADR is a historical decision record, not something to rewrite in place. - New `docs/adr/0008-llm-understanding-one-shot-rag.md`: documents what actually runs today — one LLM call understands the turn against plain history, a single deterministic retrieval dispatch (no PLAN/REFINE round budget because there's only ever one retrieval call), generation verified twice (grounding + entailment) with no confidence score, and the 2026-08-07 context-synthesis fix. States plainly what's still open (F-08, in-process-only history, no adversarial regression suite beyond one case) instead of letting the new doc drift stale the same way the old one did. - `docs/architecture.md`: fixed the audit-flagged false claim ("Retrieval- confidence gate: below a similarity threshold, skip the LLM call entirely" — never true of the live path, only the legacy no-generator fallback) to describe the real deterministic-routing/quarantine-gate design. While in the same sections: also fixed adjacent, equally-stale claims noticed along the way — every "OpenAI" reference (the service actually calls AWS Bedrock: Cohere embed-v4, Qwen3 via Converse, Cohere rerank) and the wrong Qdrant collection name (`drug_monographs_v1` → actual live `duocthu_v1`). Did not do a full audit of the rest of the file (build-roadmap phase claims for auth/chat-service/k8s) — out of scope for this specific reconciliation. `apps/web`: no changes this entry (backend/docs only); typecheck unaffected. ## 2026-08-07 (cont. 2) — Second audit P0 fixed: population/weight/age/route now reach retrieval and generation, not just the frame Owner asked to check why the quick-reply chip loop kept re-asking the same question ("Uống" → same "uống hay đặt trực tràng?" back), and separately asked which of yesterday's "Top 3 picked for next session" items were done. Checked the file directly (`docs/progress-log.md` line 261-267, cont. 12): (1) interaction-quarantine drop — done earlier today; (2) wire population/ weight/age into `retrieve_framed` for real — not done, and turned out to be exactly the root cause of the chip-loop bug; (3) reconcile `architecture.md`/ ADR 0007 with live reality — still untouched, not started this session either. **Root-caused the chip-loop bug with full prompt/response visibility**, not guessing: wrote a throwaway script that monkeypatched the live generator to capture the exact system+user prompt and raw LLM JSON for the failing turn. Two real findings, not one: 1. Conversation history **was** reaching the LLM correctly — the captured prompt showed all 4 prior turns verbatim, and the model correctly read `population=nguoi_lon` from two turns back. Multi-turn history plumbing itself was never the problem. 2. **`QueryFrame` had no field to hold "route of administration."** When the model correctly recognized "Uống" as answering its own prior route question, it had nowhere in its output schema to record that — only `population`/`weight_kg`/`age_text`/`indication` existed. With nothing to write, it could only re-emit the identical `clarify_reason` it asked before. Confirmed directly from the captured raw JSON response. 3. **A second, independent gap, matching exactly the audit's P0-2** named in yesterday's cont. 12 entry: even where the frame *does* correctly resolve population/weight/age, nothing downstream ever reads those fields. `GroundedAnswerService.answer_from_result(query, result)` takes only a bare `query` string with no notion of conversation history — confirmed by `grep`, zero references to `history` anywhere in `rag/answer.py`. So the sufficiency-check and generation LLM calls that decide whether to answer or ask again would have seen only the literal current turn ("Uống"), blind to everything resolved in earlier turns, regardless of whether route existed as a frame field. **Fixed both together** (fixing only one wouldn't have closed the loop): - `rag/understanding.py`: `QueryFrame` gained `route: str | None`; `FRAME_SCHEMA`/`_SYSTEM` updated with an explicit rule — a short reply following the model's own last clarify question must be read as resolving that dimension, keep already-known fields, and flip `needs_clarify=false` once population + route (+ age/weight if a child) are all known, instead of re-emitting the same `clarify_reason` verbatim. - `rag/agent.py`: new `_synthesize_query(turn, frame)` folds every resolved frame field (population/age_text/weight_kg/route/indication) into a self-contained question string — e.g. `"Uống. Đối tượng: người lớn. Đường dùng: uống."` — used in `_single_drug` for both `retrieve_framed`'s rerank signal and (more importantly) as the `query` handed to `answer_from_result`, so sufficiency-check/generation are no longer blind to context resolved in earlier turns. No-op (returns `turn` unchanged) when the frame has no resolved fields, so a fresh single-shot question is unaffected. 6 new tests across `test_understanding.py`/`test_agent.py`, including a direct regression test asserting the exact prior failure case now produces `needs_clarify=false` and a context-carrying query. **Verified live, twice, against the real running server** (restarted after the code change, per house rule): first with a direct script reproducing the exact 3-turn conversation that failed before (`agent.handle()` called 3 times against the live Qdrant/Bedrock stack) — turn 3 ("Uống") now returns a real grounded answer scoped to the oral dose, not a repeated question. Then again through the actual browser UI end to end (chip click → "Người lớn" → typed "Uống") — same result: real answer, `ENTAILED & GROUNDED` + `AI diễn giải, đã kiểm chứng`, 2 real citations, citation beam working. `apps/ai-service`: **195 passed** (was 191, +4 net: 2 route-parsing tests in `test_understanding.py`, 2 query-synthesis tests in `test_agent.py`). `apps/web` typechecks clean (no frontend changes this entry). **Still open, unchanged from this morning:** item (3) from yesterday's Top 3 — reconciling `architecture.md`/ADR 0007 with what's actually live. Also still open: the model-latency investigation's proposed fix (a smaller/faster model for the `understand` step only) — diagnosed, not attempted. ## 2026-08-07 (cont.) — Interaction-quarantine P0 fixed, citation duplicates merged, quick-reply chips added Owner said to go do the outstanding items from the session above, plus asked for clickable quick-reply options on clarifying questions (like this tool's own option-picker). **P0 fixed: `_interaction` no longer silently drops a quarantined drug's evidence.** `rag/agent.py::_interaction` used to keep only `part.decision == ANSWERABLE` parts before combining two drugs' interaction evidence, then hardcoded the combined `RetrievalResult` to `ANSWERABLE` — so if one drug's `tuong_tac_thuoc` section had a quarantined table, its evidence (and the "table exists, verify PDF" notice the quarantine contract requires) was dropped instead of surfaced; a confident interaction answer could omit a real unverified contraindication table for one of the two drugs ([[project-quarantined-block-contract]]). Fixed: added `RetrievalService.decide()` (a public wrapper around the existing `_decide` policy) and `_interaction` now keeps both `ANSWERABLE` and `VERIFY_PDF` parts, then re-derives the combined decision through `decide()` instead of hand-rolling it — the same quarantine policy the single-drug path already applies. New regression test (`test_interaction_with_one_drug_quarantined_never_answers_confidently`) locks this in with a synthetic quarantined case. **Could not be demonstrated live end-to-end**: checked the real corpus and found 0 of 487 quarantined chunks are in `tuong_tac_thuoc` — no real drug pair exists today where this exact path fires, so the fix is proven by unit test + live regression-check of the normal (non-quarantined) interaction case (warfarin+aspirin, unchanged behavior, 2 citations, `generated=true`), not by a live quarantined-interaction probe. **Citation duplication fixed, but it turned out not to be pure duplication.** Investigated the "near-duplicate citation cards" rough edge named at the end of the previous entry. Traced a real case (Acetazolamid, quarantined `duoc_ly_va_co_che_tac_dung` table): the two citations for one evidence block have DIFFERENT physical pages — the prose paragraph sits on physical page 108 (printed 109), the table it mentions sits on physical page 109 (printed 110). So merging them naively would have hidden real information. Fixed properly in `route.ts::toCitations`: citations are grouped by `chunk_id` into one card, using the plain-text ref's page as the card's primary location and keeping the attachment ref's own page as a new `quarantinePhysicalPage` field — `CitationCard`'s "Mở trang PDF gốc" link now opens the TABLE's own page, not the prose's page. `Citation` DTO gained `quarantinePhysicalPage?: number`. **Quick-reply chips added for clarifying questions**, per owner's request ("thêm câu trả lời cho câu hỏi thêm kiểu lựa chọn như của claude ấy"). Two independent clarify sources both needed wiring — found the hard way by testing live: 1. `GroundedAnswerService._check_sufficiency` (the dose-under-specified check) — `rag/prompt.py`'s `SUFFICIENCY_SCHEMA` gained `quick_replies: string[]`, `_check_sufficiency` now returns `(question, quick_replies)`, threaded through `GroundedAnswer.quick_replies` → `AgentReply.quick_replies` → `RagQueryResponse.quick_replies`. 2. **The path real traffic actually hits** (confirmed live — every clarify in this session's testing came from here, not #1): `understanding.py`'s `LlmQueryUnderstander` sets `QueryFrame.needs_clarify`/`clarify_reason` directly from its own single LLM call, and `RagAgent._route()` returns that immediately, short-circuiting before retrieval/`GroundedAnswerService` ever runs. Initially wired only #1 and shipped it — live-tested and found quick_replies came back empty every time; root-caused to this second, dominant path and fixed it too: `QueryFrame` gained `quick_replies`, `FRAME_SCHEMA`/`_SYSTEM` prompt updated, `_parse()` extracts it, `_route` passes it through. 5 new tests across `test_agent.py`/ `test_understanding.py`/`test_citation_and_intro.py`. Frontend: `ChatMessage.quickReplies?: string[]`; `route.ts` only surfaces them when `decision === "clarify"` and the list is non-empty; `ChatBubble` renders them as clickable chips (only under a real `clarify` decision) that call `onQuickReply`, wired in `ChatPanel` straight into `handleSendMessage` — a click sends that exact text as the next turn, no different from typing it. **Verified live** (server restarted after each backend change, per house rule): "Liều paracetamol hạ sốt là bao nhiêu?" → clarify with 4 real chips ("Người lớn", "Trẻ em <1 tuổi", "Trẻ em 1-5 tuổi", "Trẻ em 6-12 tuổi"), clicking "Người lớn" correctly auto-sent it and produced a follow-up clarify ("Uống hay đặt trực tràng?") with its own 2 chips — the chip mechanism itself (render → click → auto-send → new response) works end to end. **New issue found while verifying, not fixed today:** clicking "Uống" (the chip's own suggested answer) got the SAME "uống hay đặt trực tràng?" question back, twice in a row, even though `_remember()` does put "Người dùng: Uống" in the history the very next call reads. The understanding LLM isn't reliably resolving a terse one-word reply against its own immediately-preceding `clarify_reason` — a conversational-memory prompt weakness in `understanding.py`, separate from the chip UI itself (which correctly sent the text every time). Worth a dedicated pass: likely needs the prompt to explicitly say "a short reply with no drug name answers your own last clarify_reason" rather than relying on the model to infer that from bare history lines. `apps/ai-service`: **191 passed** (was 186 before this cont., +5 for the P0 regression test and quick-reply coverage). `apps/web` typechecks clean. ## 2026-08-07 — Citation UI now shows real retrieved data instead of fabricated placeholders Owner asked to fix the UI/UX so it shows precisely what was retrieved and how the LLM answered from it, and to read all memories first. Traced the citation pipeline end to end (`rag/answer.py` → `routers/rag.py` → `apps/web/app/api/ chat/route.ts` → `CitationCard.tsx`) and found it was showing manufactured data at several points, not real data: 1. **`route.ts`'s `RagCitation` interface declared `text_snippet`/ `citation_reason` fields that don't exist on the real backend `CitationResponse`** — always `undefined`, so every citation's snippet was blank and its "reason" silently fell back to a canned boilerplate sentence ("Trích xuất từ mục X làm căn cứ...") presented as if it were real entailment reasoning. 2. **The backend never exposed the retrieved chunk text at all.** `Citation` (`rag/answer.py`) carried only page/block pointers, so even a frontend fix alone could not have shown real evidence. 3. **`CitationCard.tsx`'s `SECTION_LABELS` map used guessed section-key slugs** (`lieu_dung`, `duoc_ly`, `tac_dung_phu`, `qua_lieu`, `bao_quan`) that don't match the corpus's real 19-field schema (`lieu_luong_va_cach_ dung`, `duoc_ly_va_co_che_tac_dung`, `tac_dung_khong_mong_muon`, ...) — every citation fell back to the raw slug instead of a label. 4. **`tra-cuu/page.tsx`'s PDF-jump used the printed page number as the `#page=` fragment.** Verified against the real PDF (rendered physical pages 106-110 with PyMuPDF and read the text) that physical page ≠ printed page — off by 1-3 depending on front-matter offset, confirmed across 1,431 sampled chunks. Right by coincidence in the majority case, wrong the rest of the time. Fixed to use `physical_page + 1` (physical_page is PyMuPDF's 0-indexed page; the `#page=` fragment is 1-indexed — verified directly by opening the resulting PDF tab and reading the rendered page). **Fixed backend** (`rag/answer.py`, `routers/rag.py`): `Citation`/ `CitationResponse` gained `evidence_text` — the literal chunk text handed to the generator/entailment check, not a paraphrase. `RagQueryResponse` gained `generated: bool` so the UI can honestly distinguish an LLM paraphrase (passed grounding + entailment) from a verbatim extractive quote (the `ANSWER_PROVIDER=disabled` mode, or a configured generator's canned `VERIFY_PDF` message). **Fixed frontend:** `Citation` DTO rewritten to match real fields (`chunkId`, `physicalPage`, real `snippet`, `isQuarantined`/`quarantineNotice` in place of the fabricated `reason`); `route.ts` now derives `drugName`/`sectionType` **per citation** from `chunk_id.split("__")` instead of stamping every citation with the turn's single `resolved_drug_id` (wrong on the 2-drug interaction path — verified live with a warfarin+aspirin query, both citations correctly show "WARFARIN", not the old combined string); `CitationCard` renders the real evidence text, a genuine quarantine banner (with a working "open PDF at the right page" link) only when the source pipeline actually flagged that chunk, and the corrected section labels; `ChatBubble` gained a truthful "AI diễn giải, đã kiểm chứng" vs "Trích dẫn nguyên văn" pill — gated to `decision === "answerable"` only, after live testing caught it mislabeling a clarifying question as "verbatim quote." `verify_pdf` and `clarify` decisions now get their own distinct header badges instead of borrowing the grounded/ungrounded binary. **Verified live, not just unit tests** (per house rule): stood up local Qdrant + Postgres (Docker, pre-existing volumes — 15,100 pts intact) and the real ai-service + web servers, drove three real queries through the actual browser: - Simple dose question (paracetamol) → real `evidence_text` shown, correct section label, "AI diễn giải, đã kiểm chứng" pill. - Interaction question (warfarin + aspirin) → both citations correctly show "WARFARIN" (both drawn from warfarin's own `tuong_tac_thuoc` section). - Quarantined-table question (Acetazolamid dược lý, page 110) → "CẦN ĐỐI CHIẾU PDF GỐC" badge, quarantine banner rendered, clicked "Mở trang PDF gốc" and confirmed in the opened PDF tab that it lands exactly on the physical page showing the real quarantined table (the pharmacokinetic timing table) — the page-jump is now provably correct, not just plausible. `apps/ai-service`: **186 passed**, no regressions. `apps/web` typechecks clean (`tsc --noEmit`). **Known rough edges, named rather than hidden, not fixed today:** - `_indexed_citations` emits one `Citation` per `source_ref`, so a quarantined chunk (base prose ref + attachment ref) produces two near-duplicate citation cards with identical snippet text. Pre-existing data shape, not introduced today. A dedup pass needs to preserve the quarantine flag from whichever ref carries it — not attempted, to avoid rushing something that could silently drop the quarantine signal. - This UI fix makes a quarantined citation genuinely visible **when the backend sends it**, but does not fix the already-known P0 where the 2-drug interaction path (`agent.py::_interaction`) silently drops a quarantined drug's evidence instead of surfacing "table exists" for it ([[project-quarantined-block-contract]]). Still next-session work. - `source_crop` is `None` across the entire live corpus (checked: 0/15,100 chunks) — the table-reconstruction pass that would populate it is a separate in-progress track (11 crop PNGs generated so far, not yet loaded). The `` rendering path in `CitationCard` is wired but dormant; it activates automatically once that data lands. Until then, quarantined citations fall back to the "open PDF at the right page" link, which is itself now verified-correct. ## 2026-08-06 (cont. 12) — Independent senior-engineer audit, 7 parallel agents, read-only (no fixes applied yet) Owner asked for a full RAG audit (parsing → chunking → retrieval → query understanding/reasoning → grounding/safety → evaluation → production engineering), API-only architecture, no fine-tuning proposals, no redesign, report only. Ran 7 subagents in parallel, each required to read real implementation + run real tests before concluding. Headline finding: the code running in production (`rag/agent.py`, wired via `bootstrap.py`) is **not** the architecture described in `docs/architecture.md` or ADR 0007 — three separate live/dead-code mismatches independently surfaced by three different agents: - **Dense vector search is dead code live.** `retrieve_framed` (the only method `RagAgent` calls) only ever does exact `find_by_section`/ `find_by_drug` payload-filter scroll, never `QdrantRetriever.search()`. "Hybrid retrieval" doesn't exist in production either (only in `rag/in_memory.py`'s test fallback). - **`architecture.md`'s "retrieval-confidence gate: skip LLM below a similarity threshold" is false for the live path.** The threshold only exists on the legacy `RetrievalService.retrieve()`, which `RagAgent` never calls. - **ADR 0007's entire reasoning-loop design (`reasoning.py`, `conversation.py`, `conversational.py` — Focus/TTL, turn budget, sufficiency-driven retrieval refinement) is dead code.** `bootstrap.py` builds `RagAgent` with none of it; the live agent is a fixed one-shot pipeline (understand → route → retrieve once → generate → ≤2 entailment retries), not an iterative loop that feeds back into retrieval. Two new bugs found (not previously known): 1. **P0 — `_interaction` (agent.py:144-149) silently drops a drug's interaction evidence if it's quarantined (`VERIFY_PDF`)**, without telling the user — violates the [[project-quarantined-block-contract]] obligation ("must make the answer say a table exists") specifically on the 2-drug interaction path; the single-drug path already obeys it. 2. **P0 — `QueryFrame.population/weight_kg/age_text/indication` are extracted by `understanding.py` but never passed into `retrieve_framed`/`answer_from_result`.** This is exactly the bug ADR 0007 was written to fix ("liều paracetamol cho người lớn" vs "liều paracetamol" can retrieve identically) — the structured signal exists, the plumbing into retrieval that would guarantee it does not. Ingestion side (parsing/chunking) came out strong and independently verified against real whole-corpus artifacts, not docs: back-index recall 96.2%/precision 99.1% (live CLI run), all 30 `chunk-ready` gates PASS on the real 684-monograph/15,100-chunk corpus, deterministic rebuild confirmed by sha256 diff + live-Qdrant idempotent-upsert test. Two smaller ingestion bugs found: `extract/spans.py`'s reading-order sort treats every `full_width` block as page-header material — falsified by 9 real mid-page full-width tables in `table_regions.json` (2/9 traced through to final output were fine, 7/9 unverified); and `chunk/chunker.py`'s `_SUBGROUP_LABEL` regex (the guard against splitting mid-subsection) is missing pregnancy/breastfeeding terms (`phụ nữ|mang thai|thai|cho con bú`), 11 real occurrences in-corpus, no confirmed bad split yet but uncovered by the guard. Evaluation-coverage gap: full unit suites pass for real (`apps/ai-service` 186 passed/4 skipped, `ingestion` 296 passed/0 skipped), but there is **no automated regression re-run of the golden sets** — `run_eval.py` is unusable as committed (missing fixtures), `evals/manual_adversarial_ hard10.jsonl` (has exactly the table/formula/vet-abstain/cross-page cases needed) is never read by any script or test, no CI workflow exists, and NDCG/Precision@K are entirely absent repo-wide (MRR exists only in the ingestion embedding benchmark, unwired to retrieval eval). Production engineering: no hardcoded secrets found (checked). k8s/Helm/ Terraform/Dockerfiles are genuinely empty scaffolding (Phase 6, as roadmapped — not a surprise). No circuit breaker, no exception handling around the Qdrant scroll calls actually used live (outage → raw HTTP 500, not a graceful abstain), no end-to-end request timeout budget (F-08 still open — worst case several minutes, no aggregate cutoff), `/health` unconditionally returns ok with no downstream check, 6 of the metric names defined in `rag/metrics.py` aren't registered in `adapters/prometheus.py` (silent no-op if incremented), conversation history is an in-process dict (lost on restart, not shared across workers). **Top 3 picked for next session** (see full report in this session's transcript for file:line detail on every item above): (1) fix the interaction-quarantine silent drop, (2) wire population/weight/age into `retrieve_framed` for real, (3) reconcile `architecture.md`/ADR 0007 with what's actually live — either add the retrieval-confidence floor for real and delete the two dead reasoning-loop modules, or wire them in; stop carrying two contradictory architectures side by side. No code changed this session — read-only audit per owner's explicit instruction. Full agent-by-agent findings (parsing, chunking, retrieval, query-understanding/reasoning-loop, grounding/safety, evaluation, production) not reproduced here in full; re-run the same 7-way audit prompt if the detail is needed again, or ask the owner for the chat transcript. ## 2026-08-06 (cont. 11) — Real bug found by actually running the golden eval set: "thận trọng" silently answered as "chống chỉ định" Owner pointed at a golden dataset (`Golden Dataset/golden_e2e_v1.csv` +4 more, 36-74 hand-authored cases each, dated 2026-08-04/05 — never run this session until asked). Ran the 36-case e2e set live end-to-end. Findings, graded against each case's own pass criteria: - **3/36 (8%) correct answers discarded to an empty abstain** by the F-01 entailment-noise issue already flagged as a known limitation — the golden set turns that into a measured rate, not a hunch. - **2/36 wrong-section content gap, real bug, root-caused and fixed**: "X cần thận trọng gì?" (asking precautions) was classified `attribute= chong_chi_dinh` (contraindications) 9/9 times live-checked — the wrong section entirely, silently dropping the actual precautions content (e.g. metformin's lactic-acidosis warning, gentamicin's oto/nephrotoxicity) in favor of contraindication text. Cause: the prompt gave the model a bare `SECTION_KEYS` slug list with zero definitions — nothing to tell two genuinely adjacent Vietnamese medical concepts apart. Fixed: `rag/understanding.py` gained `SECTION_KEY_HINTS`, a short gloss per key shown inline in the prompt, with `than_trong`'s explicitly stating it is NOT `chong_chi_dinh` and naming the two example warnings that were getting lost. Verified live: 3/3 reclassified correctly to `than_trong` (metformin/gentamicin/ibuprofen), `chong_chi_dinh` questions unaffected, and the two originally-broken answers now contain the exact required content ("nhiễm toan lactic", "độc hại đối với cơ quan thính giác và thận"). 2 new tests in `tests/test_understanding.py` (10 total, was 8). - **Several other gaps found, not code bugs**: `#26` ("nên tự tăng gấp đôi liều?") and the "An toàn (Type 3)" block (`#21-25`) in the golden set model a **lay-patient safety framework** (refuse + "hỏi thầy thuốc") that directly contradicts the owner's explicit correction earlier this same session — this product gates on scope (human/non-human), not on "asks for a recommendation" (`[[feedback_no_recommendation_gate]]`). The golden set predates that correction by two days; treating its Type-3 rows as ground truth would silently re-introduce the exact gate the owner ordered removed. Flagged to the owner rather than "fixed." `#13`/`#20` test the `/v1/rag/suggest` autocomplete flow but were driven through `/v1/rag/query` by mistake — not a valid test of those two rows, not rerun yet. `#14` vs `#15` (bare-name inconsistency), `#30` (price question), `#35` (two-drug wording) are minor, not investigated further today. `apps/ai-service`: **186 passed, 4 skipped**. ## Status at end of today's session (accurate as of cont. 10 below) Codex's `CODEX_RAG_CODE_REVIEW_2026-08-06.md` correction order: **F-01 through F-07, F-09 done; F-08 and F-10 done for their core finding, with named remainder.** Every completed item was live-verified against the real running server, not only unit tests — several real bugs were found *by* that live verification and fixed the same day, not just the ones the review named (grounding fallback removed per owner correction, F-03's `retrieve_framed` sending whole monographs, catalog-naming/id-form/ weight-parsing bugs the owner's own UI test surfaced, F-06's exact overflow repro, F-08/F-09's Postgres connect-timeout hang). **Named remainder, next session's work:** - **F-08**: the Postgres-side unbounded-hang is fixed (`connect_timeout`), but a real end-to-end deadline threaded through `RagAgent`'s own LLM calls (understand → sufficiency → generate → up to 2 entailment retries, up to 5 sequential Bedrock calls per request) does not exist — needs a request-scoped budget object, a real design, not a bolt-on. - **F-10**: the core gap (RagAgent had zero test coverage and was not provably the same dependency graph as the live HTTP service) is closed — `tests/test_live_datastores.py::test_real_rag_agent_end_to_end_through_the_http_api` drives the real `/v1/rag/query` endpoint, real `RagAgent`, real `RetrievalService`/`QdrantRetriever` against a real temporary Qdrant collection, and a real Postgres trace, asserting drug id, citation, and decision — only the nondeterministic cloud model call is faked, since this session's own live probing found real generation/entailment calls too noisy for a regression assertion. **Not built**: the review's full adversarial regression list (prompt injection, fake-drug-near-alias, provider-timeout-and-outage behavior, `conversation_id` presence/absence producing the same safety decision, etc.) — one solid end-to-end case proves the wiring is real and testable; a comprehensive battery is a larger, separate effort. - **`dosing_calc`** (a tested mg/kg calculator) and **`symptom_to_drug`** (reverse indication→drug lookup) remain honest "not ready" clarifies — deliberately not built under today's time pressure; see `[[project_rag_rebuild_2026_08_06]]` on why rushing dosing math is the wrong tradeoff. `apps/ai-service` full suite: **184 passed, 4 skipped** (the new integration test opts in via `RUN_INTEGRATION=1`, verified passing that way), up from 118 passed at the start of today's session. ## 2026-08-06 (cont. 10) — F-10 core done: RagAgent proven live-testable end to end, not just live-tested by hand Every F-01–F-06/F-08/F-09 live verification this session was a one-off Python script run by hand against the real Qdrant/Bedrock/Postgres — real evidence, but not a regression a future change would automatically re-run. F-10 closes that: `tests/test_live_datastores.py` gained `test_real_rag_agent_end_to_end_through_the_http_api`, following the existing `RUN_INTEGRATION=1`-gated pattern in that file (temporary Qdrant collection seeded with one real corpus chunk, real Postgres migration + trace round-trip). What's real in this test: `RagAgent`, `LlmQueryUnderstander`, `RetrievalService`, `QdrantRetriever`/`QdrantParentStore` against a live Qdrant, `GroundedAnswerService`, `PostgresTraceRepository` against a live Postgres, and the actual `/v1/rag/query` FastAPI route via `TestClient` — the identical object graph `bootstrap.build_runtime` wires in production. What's faked: only the LLM boundary (`_FakeJsonLlm`, satisfying both the `JsonLlm` and `AnswerGenerator` protocols with fixed payloads keyed by schema shape) — deliberately, not for convenience: this session's own live probing (F-01's entailment noise, F-03's non-deterministic generations) found real cloud calls too noisy to assert exact drug id / citation / decision against reliably. Asserts (Codex's exact F-10 list): resolved drug id, citation chunk id and printed page, decision, and that the trace persisted and reads back correctly. Verified passing with `RUN_INTEGRATION=1` (4/4 in that file) and correctly skipped by default (184 passed, 4 skipped without it — no cost/flakiness added to the normal suite run). **Scope, stated plainly**: this is the load-bearing first case proving the production path is real and mechanically testable, not the comprehensive adversarial battery the review sketched (prompt injection, fake-drug-near- alias, provider outage/timeout behavior, `conversation_id` presence/absence parity, multi-population-band evidence, etc.). Extending this one case into that full battery is real remaining work, not done today. ## 2026-08-06 (cont. 9) — F-09 done (trace fail-open), F-08 partially: a real unbounded-hang found live and fixed **F-09.** `routers/rag.py` called `traces.save()` synchronously before returning a response; `PostgresTraceRepository.save()` opened a fresh connection per call with no error handling, so a Postgres outage turned an already-computed, safe answer into a 500 for a reason unrelated to whether the answer was safe. Made an explicit fail-open decision (tracing is observability, not the product): the router now wraps the `save()` call, falls back to a locally-generated `trace_id` on any exception, and counts it (`duocthu_trace_write_failed_total`, a new metric — a silent fail-open with nothing to page on is indistinguishable from tracing quietly working). Connection pooling (the other half of the original finding) not done — real pooling needs startup-time lifecycle wiring, out of scope for today. **F-08, live-verified, not fully scoped.** Testing F-09 by pointing `POSTGRES_DSN` at an unreachable host live surfaced a sharper bug: a bare `psycopg.connect()` with no `connect_timeout` hangs on the OS-level TCP timeout (tens of seconds) when the DB is unreachable but not *actively* refusing — which defeats the F-09 try/except just as completely as no try/except at all, since the exception it's waiting for doesn't arrive in time. Added `connect_timeout=5` to every `psycopg.connect()` call in `adapters/postgres.py`. Verified live: same broken-DSN repro that previously hung past a 30s client timeout now returns 200 with the correct grounded answer in ~14.5s (5s bounded connect attempt + normal generation latency). The broader F-08 ask — an end-to-end request deadline threaded through every provider call — is **not done**: `TurnBudget` (`rag/reasoning.py`) exists but belongs to the old `ConversationalLoopService` path, which F-03 stopped constructing live; the new `RagAgent` path (up to 5 sequential Bedrock calls per request: understand, sufficiency, generate, up to 2 entailment retries) has no budget object at all, bounded only by each individual call's own fixed read_timeout (30-60s each). A real fix needs a request-scoped deadline object passed into `RagAgent`/ `GroundedAnswerService` and consulted before each call — a genuine feature to design, not something to bolt on safely in the time remaining today. `apps/ai-service`: **184 passed, 3 skipped**. ## 2026-08-06 (cont. 8) — F-05 done: startup refuses a corpus/model manifest mismatch, live-verified both ways The ingestion loader already writes a sidecar manifest (` __manifest`, one point: corpus SHA, chunk count, embedding model_id, dimensions) recording what a collection was built from (`ingestion/ingestion/load/manifest.py`). Nothing on the ai-service side ever read it — two unrelated embedding models can both produce 1024-dimensional vectors, and Qdrant returns plausible-looking but meaningless nearest neighbours with no error at query time. Added `rag/manifest.py` (`check_manifest` — pure, 6 unit tests) and wired `bootstrap.py::_verify_corpus_manifest` to call it right after the query embedder is constructed, before anything else. `main.py` builds the runtime at import time, so a mismatch crashes startup — the service never comes up against a corpus it wasn't verified against, rather than silently serving degraded search. Hit a real API mismatch immediately (pytest collection caught it, since `test_api.py` imports `main.py`, which calls `build_runtime` against the live Qdrant): this qdrant-client version has no `collection_exists`, and `get_collection` is a known parse-bug risk in this environment (per `reference_env_operational_gotchas`) — switched to `get_collections()` + membership check instead. **Live-verified both directions**, not just unit tests: the real collection's manifest (`model_id=cohere.embed-v4:0, dimensions=1024`) matches the configured embedder and the server starts and answers correctly; a monkeypatched `embedding_dimensions=768` against the same real manifest correctly raises `ManifestMismatch` before any query path is reachable. `apps/ai-service`: **183 passed, 3 skipped**. ## 2026-08-06 (cont. 7) — F-06 done: the overflow-before-truncation bug, exact repro fixed `ConversationState.append()` truncated `recent` to the window immediately; `overflow()` then checked `len(self.recent) > window` on the *already- truncated* tuple, which can never be true. Codex's exact repro (8 turns into a window of 6: `recent=6, turn_count=8, overflow=0`) reproduced first, unchanged from the review. Fixed: `ConversationState` gained a `pending_overflow` field. `append()` computes what it evicts *before* truncating and accumulates it there (accumulates, not overwrites — a live turn calls `append()` twice in a row, user then assistant, and the second call must not lose what the first evicted). `overflow()` now just returns `pending_overflow`. The caller clears it (`replace(state, ..., pending_overflow=())`) after folding into the summary, or the same turns fold again next cycle — `ConversationalLoopService._persist` (the live path) updated to do so; `ConversationalRagService._persist` already reconstructs `ConversationState` directly without passing the field through, so it already clears by construction. Verified the exact repro now returns the 2 actually-dropped turns instead of `()`. 6 new tests in `tests/test_conversation.py`. **Not done, out of scope for the remaining time today:** the second half of the original F-06 finding — `InMemoryConversationStore` loses all state on restart and diverges across multiple workers. That needs a shared (Postgres-backed) store, a real infra addition, not a bug fix; not attempted under today's time pressure rather than risk a rushed, unverified persistence layer. `apps/ai-service`: **177 passed, 3 skipped**. ## 2026-08-06 (cont. 6) — F-04 done: drug candidates bounded deterministically before the LLM picks, live-verified `rag/understanding.py::LlmQueryUnderstander` used to show the model the *entire* ~684-drug catalog every turn and trust any id it returned as long as that id existed somewhere in the catalog (Codex's F-04 finding: catalog membership proves the output is *some* real drug, not that it's the one the user's text actually named — an LLM could satisfy that whitelist while mapping an unrelated/invented name to a different real drug). Reworked: `LlmQueryUnderstander` now takes a `resolver` (the existing `CatalogDrugResolver`, already built in `bootstrap.py` for autocomplete) and computes a deterministic **candidate set** from the turn + raw history text *before* calling the LLM — exact alias matches plus a generous fuzzy `suggest` pass (min_score=0.55, well below the resolver's own 0.84 auto-answer threshold, since the goal here is only to rule out drugs nothing in the conversation plausibly refers to). Only that candidate subset (not the full catalog) is shown to the model, and the model's pick is validated against it — a real id the model names that isn't among the turn's candidates is now treated as unknown, not trusted on catalog membership alone. Also directly closes a separate prompt-cost finding from the same review (sending the full catalog every turn is unbounded token cost) since the shown block is now per-turn-sized, not fixed at ~684 rows. `tests/test_understanding.py` extended (was 0 tests before this session, per Codex's F-10 finding; now 8): covers exact-form and spaced-form resolution, a genuinely invented name staying unknown, **a real catalog id that has no deterministic candidate support still being rejected** (the core F-04 guarantee — catalog membership alone is not enough), and a fuzzy typo still resolving through `suggest`. **Live-verified**, not just unit-tested: `aspirinol` (fake) still correctly abstains out-of-scope; `amoxicillin` (correct INN spelling, a typo-adjacent case) still resolves to `amoxicilin`; `metformin` and the 3-turn paracetamol pediatric-dose conversation from the owner's own UI test both correctly keep the same `resolved_drug_id` across every turn. No latency regression observed (smaller prompt, same ~3-9s range dominated by generation, not catalog size). `apps/ai-service`: **174 passed, 3 skipped**. ## 2026-08-06 (cont. 5) — Three more live bugs found from the owner's own UI test of F-03, all fixed Owner drove the real web UI (not curl) through a multi-turn pediatric dose question and hit a severe regression: "Liều paracetamol cho trẻ em" -> two clarify rounds (age, then weight) -> final turn answered "Không tìm thấy paracetamol trong Dược thư Quốc gia Việt Nam" for a drug that plainly is in it. Root-caused and fixed three distinct bugs in the F-03 wiring, in order: 1. **`_catalog_names` (bootstrap.py) could bury a drug's own name.** It picked the first 3 aliases *alphabetically* per drug to show the LLM understander. Paracetamol has 191 aliases (mostly trade names); the alphabetically-first 3 were "0Frezefev, ABAB, Ace kid 80" — no recognizable name at all. Mid-conversation, once the drug is no longer restated in the raw turn text, the model has only history + this catalog line to re-derive it from; with nothing recognizable shown, it read "paracetamol" as an unknown name. Fixed: always show the drug_id's own name form (`drug_id.replace("_"," ")`, guaranteed present) first, then fill remaining slots preferring short ALL-CAPS aliases (the book's own heading convention, usually the generic name) over dosage-suffixed brand names. `tests/test_bootstrap.py` (new, 4 cases). 2. **That fix immediately exposed a second bug.** With the display name now near-identical to the drug_id ("paracetamol acetaminophen" vs. "paracetamol_acetaminophen"), the model started echoing the *spaced* display form instead of the underscored id, and `LlmQueryUnderstander._parse()`'s strict `d in self._ids` check demoted a correctly-identified drug to `unknown_drugs` — same user-visible failure, different cause. Fixed: `_resolve_id()` accepts either the exact id or its space-substituted form (a deterministic, lossless formatting tolerance — not fuzzy matching, no risk of resolving to an unrelated drug). `tests/test_understanding.py` (new, 7 cases — this module had zero coverage before today, per Codex's F-10 finding). 3. **"30 cân" (colloquial Vietnamese for "30 kg", no unit word) wasn't reliably read as a weight.** Confirmed live: the model missed it entirely in some runs, silently re-asking for weight the user had just given. Added an explicit rule + schema hint that a bare number + "cân"/ "ký" means kilograms. Verified live: 3/3 clean extractions after the fix (was inconsistent before). All three verified against the real running server with the owner's exact repro sequence, not just unit tests — final state: the drug (`resolved_drug_id = paracetamol_acetaminophen`) now stays correctly attached across all three turns, and weight is correctly captured. **Not fixed, deliberately, already flagged (F-07):** `dosing_calc` still doesn't compute an actual mg dose once enough information is gathered — it falls through to ordinary section retrieval (the clinician sees the dosing table, not a calculated number). A weight-based calculator is a real feature to build, not a wiring bug; out of scope for this pass. Also, per owner UX feedback, warmed up the static smalltalk reply (was a terse "Chào anh/chị. Tôi tra cứu... Anh/chị muốn hỏi về thuốc nào?"). `apps/ai-service`: **173 passed, 3 skipped** (was 162 at the end of the F-03 entry below). ## 2026-08-06 (cont. 4) — F-03 done: RagAgent wired into the live server, two real bugs found and fixed by driving it Wired the new LLM-understanding orchestrator (`rag/agent.py` + `rag/ understanding.py`, built last session but never called by anything live — Codex's exact F-03 finding) into `bootstrap.py`/`routers/rag.py`. Both single- and multi-turn requests now go through one path: `RagAgent.handle()`. The old `CatalogDrugResolver`/`QueryRoutingService`/ `ConversationalLoopService` stack stays in the codebase (still unit-tested, still used for autocomplete + the no-generator-configured fallback) but is no longer constructed as the live query path — per Codex, full deletion waits on a production-path parity suite (F-10), not done yet. Added the coverage that didn't exist: `tests/test_agent.py` (14 cases — `RagAgent` had zero tests before this), `tests/test_retrieval_service.py` +2 for `retrieve_framed`, `tests/test_api.py` +4 for the router's agent branch. 162 passed, 3 skipped. **Drove the actual running server** (per house rule: never claim a wiring change works from unit tests with fakes alone) and found two real bugs unit tests couldn't have caught: 1. **`retrieve_framed` had no bare-name/overview case.** `retrieve()` (the old path) always answered a bare drug name from four identity sections only; `retrieve_framed` had no equivalent and always fetched the entire ~29-section monograph, then relied on rerank to trim it — silently sending the whole book as evidence whenever rerank was off or failed open. Live symptom: asking bare "paracetamol" abstained empty every time (answer too long, generation intermittently malformed). Fixed: `retrieve_framed` gained an `is_overview` parameter (driven by the frame's `turn_type == "drug_overview"`), mirroring the old intro-only behavior, and the non-overview rerank branch is now capped at `evidence_limit` even when rerank fails open — an ordering aid failing open must not also remove the size bound. Verified live: 3/3 clean answers after the fix, none of the prior empty-abstain failures. 2. **The entailment judge (added this session, F-01) is noisier than one call suggests.** Same claim/evidence pair, called repeatedly, disagreed with itself — confirmed live on the warfarin/aspirin interaction case, which correctly cites a drug-interaction list evidence block but got rejected 0/2, 1/2, then 3/3 across separate live batches. Added a same-claim retry (`GroundedAnswerService._verify_entailment`): a lone reject retries once, only two agreeing rejects discard the generation. Also sharpened the entailment prompt to explicitly call out dense comma-separated drug-interaction lists, since the specific failing claim named a drug buried mid-list. Owner explicitly capped further spend here (more retries = more tokens for a narrowing edge case) — the retry/prompt change did not fully eliminate this one case in further live testing (still failed 3/3 in the last batch), and it was deliberately **left as a known, safe-direction residual limitation** rather than chased further: the failure mode is abstain (never a fabricated interaction claim), not wrong output. Documented in `_verify_entailment`'s docstring; a cleaner fix (e.g. breaking a multi-drug interaction claim into a per-drug comparison instead of one long prose evidence block) is a good candidate for a future pass, not solved today. Also fixed a mismatched piece of the wiring in `apps/web/app/api/chat/ route.ts`: it discarded `RagAgent`'s specific abstain messages (e.g. "Không tìm thấy X trong Dược thư") in favor of a generic fallback, because it only consulted `answer` when `decision !== "abstain"`. Now prefers `rag.answer` whenever it is non-null, regardless of decision. ## 2026-08-06 (cont. 3) — Investigated "684 vs 700+24 expected" monograph-count question: zero real drug monographs missing, gap is 100% explained Owner asked why the corpus has 684 monographs when the expectation was ~700 drug monographs + 24 general-chapter monographs. Did not rely on any number already sitting in memory/docs — re-ran `ingestion.cli validate` live against the real PDF this session to get a current ground-truth comparison, per [[feedback-rigorous-validation]] / [[feedback-verification-ladder]] ("recompute every number before quoting it"). **Live re-run result** (`python -m ingestion.cli validate --pdf data/raw/duoc-thu-quoc-gia-viet-nam-2018.pdf`): ``` detected monographs: 684 ground-truth entries: 705 (parsed from the book's own back-of-book index, "Mục lục tra cứu", pages 1529+) recall: 96.2% (678/705) precision: 99.1% ``` 27 ground-truth entries didn't match a detected monograph, and 6 detected monographs didn't match a ground-truth entry. Pulled the **full** unmatched list (the CLI only prints the first 20 of 27) via a direct Python call into `ingestion.validation.back_index`/`metrics` — classified all 27 by hand: | Category | Count | Detail | |---|---|---| | Part 1 general-chapter titles (printed pp. 39-95) | 20 distinct (21 lines — "Thuốc chống loạn thần..." p.75 is duplicated in the book's own index) | Hướng dẫn sử dụng(39), Kê đơn thuốc(40), người cao tuổi(41), suy gan/thận(43), trẻ em(45), thai kỳ/cho con bú(47), giảm đau(48), hen phế quản(51), kháng động kinh(55), kháng HIV(61), kháng sinh(70), cephalosporin(72), chống loạn thần×2(75), lao(77), viêm gan B(80), ADR(83), dị ứng thuốc(85), ngộ độc(90), dược động học(93), tương tác thuốc(95) | | Part 3 appendix titles | 2 | BSA calc (1497), pha thuốc tiêm IV (1498) | | Part 2 drug names | 4 | Alphatocoferol(165), Benzoyl peroxyd(246), Hydrogen peroxyd(781), Tretinoin (thuốc uống)(1405) | **Then checked all 4 remaining "drug" entries directly against `ingestion/data/processed/monographs.jsonl`** (not just assumed) — all 4 are already present in the corpus, under a differently-spelled `drug_name`: - Alphatocoferol → `ALPHA TOCOPHEROL (Vitamin E)` (tocoferol/tocopherol) - Benzoyl peroxyd → `BENZOYL PEROXID` (peroxyd/peroxid) - Hydrogen peroxyd → `HYDROGEN PEROXID` (peroxyd/peroxid, same pattern) - Tretinoin (thuốc uống) → `TRETINOIN (UỐNG)` These 4 also account for 2 of the "6 unmatched detected monographs" (HYDROGEN PEROXID, TRETINOIN (UỐNG) both show up on both sides of the diff — same monograph, name-matching miss in `validation/metrics.py`'s `_names_match`, not two different problems). **Conclusion, fully closed, no open unknowns left:** 1. **Zero Part 2 (drug) monographs are actually missing.** Every ground-truth drug entry in the back-index resolves to something already in the 684. The apparent gap was a validator string-matching artifact (Vietnamese `-yd` vs. English `-id`/`-pherol` spelling variants), not missing content. `684` is the correct, complete count for Part 2. 2. **The "24 general chapters" are 0/24 present** — confirmed only 20 distinct chapters exist in the book's own index (not 24; owner should double check where the "24" figure came from), and none of the 20 are extracted, because the pipeline was scoped to Part 2 only from the start (`extract`/`segment`/`assemble` never touch printed pp. 37-98). This matches the already-known, already-documented scope gap in [[project-rag-rebuild-2026-08-06]] / `reference_duoc_thu_2018_structure` memory — not a new discovery, just re-confirmed live. 3. The book's own front-matter "~700 substances" figure is the publisher's approximate active-substance count, not a strict heading-count promise — some monographs bundle multiple substances under one heading (INSULIN = 20 ATC codes / salts under 1 monograph, the HMG-CoA-reductase-inhibitor class monograph, ARGININ's 2 salts), so a smaller heading-count than 700 is expected and consistent with full coverage, not evidence of missing data. **Not done / possible follow-up (not requested this session):** the 6→4 `_names_match` misses above suggest a small, mechanical fix (normalize `-yd`↔`-id`/`-pherol` diacritic-free spelling variants, or add explicit alias pairs) would push CLI-reported recall from 96.2% to ~99.7% without touching extraction at all — cosmetic (metric accuracy), not a data-quality fix, since the underlying monographs already exist either way. The 4 remaining truly-unmatched-detected entries (CARBIDOPA-LEVODOPA, THUỐC PHIỆN-OPIAT-OPIOID, VẮC XIN DPT, VẮC XIN MMR) are compound/hyphenated-name matching gaps in the same function, same category, not investigated further this session. ## 2026-08-06 (cont. 2) — Owner correction: no fallback to raw source text when a generator is configured; F-02 scoped down to subject_scope only Two corrections from the owner mid-F-02, both applied immediately: **1. Dropped intent-based recommendation gating entirely.** Built a keyword detector for `QueryIntent.RECOMMENDATION` ("nên dùng thuốc gì" etc.) as part of F-02's server-side policy derivation — wrong call, reverted same session. **This product is for doctors and pharmacists** (`[[project_target_audience]]`), and a clinician asking "thuốc nào tốt nhất cho bệnh nhân suy thận" is normal, in-scope use of a formulary reference, not a request to abstain on. `rag/ policy.py` now derives `subject_scope` only (veterinary/non-human keyword check — a corpus-coverage fact, not a restriction on clinical questions); `routers/rag.py` passes `intent` through from the caller unchanged, same as before F-02. `tests/test_policy.py` scoped down to match. **2. Removed the extractive-fallback safety net for a CONFIGURED generator that fails.** Previously, any generation failure — provider outage, malformed JSON, `grounding.verify` rejection, entailment rejection — fell back to quoting the retrieved evidence verbatim ("the source is always available because it was computed first"). Owner: that raw citation-stapled paragraph is the retired offline-extractive product shape (`[[project_llm_cloud_plan]]` — "owner wants a REAL LLM chatbot... not the offline extractive build"), and must not reappear as a silent degradation path now that generation is live. `GroundedAnswerService.answer_from_result` (`rag/answer.py`) now branches on whether a generator is configured at all, not just on whether this call produced one: - **No generator configured** (`ANSWER_PROVIDER=disabled`, the default) is unchanged — a deliberate, fully-supported retrieval-only mode, still quotes the source. - **A generator IS configured** and this generation failed any check → the turn **abstains** (`decision=ABSTAIN, reason="generation_unavailable"`, `answer=None`), never a raw source dump. Updated 9 tests across `test_grounded_generation.py` and `test_citation_and_intro.py` whose assertions encoded the old fallback behavior (`grounded.answer.startswith(EVIDENCE_TEXT)` → `grounded.answer is None` + `decision == ABSTAIN`). Live-verified the happy path still works unchanged against the real model (Qwen3/Bedrock Converse, ~3.5s, served correctly) — this change only touches the failure branch. `apps/ai-service`: **140 passed, 3 skipped**. ## 2026-08-06 (cont.) — F-01 fixed: grounding verifier no longer trusts a global number pool or an uncited claim Codex's code-only review (`coordination/CODEX_RAG_CODE_REVIEW_2026-08-06.md`) reproduced three ways `rag/grounding.py::verify` let an unsafe generated answer through. Reproduced all three locally first, byte for byte, before touching code — all three real. Working through the review's proposed correction order (F-01 → F-02 → ... → F-10; tracked as tasks #1-#8). **F-01, done.** Two independent fixes, both proven live (Qwen3 via Bedrock Converse), not just against a fake generator: 1. **Per-citation binding, not global pool.** `verify` used to pool every number from every evidence block into one set and check answer numbers against that pool — so a number true of block 2 passed under a citation to block 1 (`so_sai_nguon`). Rewrote to split the answer at each `[n]` citation group and check only the block(s) that group names. 2. **Citation required for every claim.** A citation-less generated answer used to pass silently as long as it stated no number the pool didn't already contain (`khong_citation`) — trivially true when the answer had no numbers at all. Now any substantive claim with no valid citation is rejected (`uncited_claim`). This also kills the old "attach every retrieved citation when the generated text cites nothing" fallback in `GroundedAnswerService`: that code path is now unreachable, since `grounding.verify` rejects the citation-less generation before it gets there — the extractive fallback (which always cites everything by construction) takes over instead. 3. **Entailment gap (`claim_bia`) — regex can't see meaning.** A fabricated nonnumeric claim with a syntactically valid citation ("Metformin chữa ung thư [1]" citing a block about đái tháo đường) still passed both fixes above: no number, citation in range. Closed with a second LLM call (`GroundedAnswerService._verify_entailment`, `rag/prompt.py`'s `build_entailment_request`) that runs after `grounding.verify` passes: each substantive cited claim, checked only against the evidence block(s) it names, judged by a model told to compare wording, not reason about medicine. Fails closed (provider outage/malformed JSON → reject, not accept). **Live-verified against the real model**, not simulated: ran the actual entailment prompt through `BedrockConverseAnswerGenerator` (Qwen3) on `claim_bia`, a fabricated contraindication, a faithful claim, and a legitimate paraphrase — correctly rejected the two fabrications (`entailed: false`) and passed the two honest ones (`entailed: true`, including the paraphrase, so it isn't just penalizing rewording). Also ran the full `GroundedAnswerService` pipeline live end-to-end (real generator, real multi-call sequence) on a legitimate metformin dose question — served correctly, ~3.4s. `apps/ai-service`: **134 passed, 3 skipped** (was 118p/3s before this session; added `tests/test_grounding.py` — 12 adversarial cases — plus 4 new entailment-path cases in `tests/test_grounded_generation.py`, and updated 3 existing tests whose assertions encoded the old, buggy behavior). **Known residual limit**, stated in `rag/grounding.py`'s docstring: the entailment LLM call is itself a model judgment, not a proof — it is a real improvement over zero semantic check, not a formal guarantee. F-02 through F-10 (scope/intent server-side enforcement, wiring the new `RagAgent`/`LlmQueryUnderstander` orchestrator that's currently dead code, bounding entity candidates, manifest validation, conversation overflow bug, request budgets, trace failure policy, production-path regression suite) are next, in that order — none touched yet this pass. ## 2026-08-06 — RAG rebuild started: live failure diagnosis + LLM query-understanding front-end (replacing the brittle resolver) Owner reported the live chatbot "cực ngu, sai gần hết" and asked to rebuild the RAG from scratch (incl. chunking). Per the never-fabricate rule, drove the REAL running service before designing. **Stack brought up live** (all local, $0 to load): Qdrant `duocthu_v1` already held 15,100 pts @1024-dim (green); Postgres up; ai-service :8079 running with the cloud-live `.env` (cohere-v4 embed + qwen3 generation + Cohere rerank). **Live diagnostic battery (~20 hard VN questions, real `POST /v1/rag/query`).** Finding, evidence-backed: it is NOT "sai hết" and the culprit is NOT chunking — when a single drug resolves cleanly the answer is grounded and correct (paracetamon typo ✓, metfomin typo ✓, multi-turn "nó dùng cho trẻ em" inherited metformin ✓). The failures cluster in the **query-understanding / drug-resolution front-end** (the `CatalogDrugResolver` fuzzy `SequenceMatcher` + keyword `SectionResolver`): - `aspirinol` (fake drug) fuzzy-matched to aspirin and ANSWERED — a safety bug. - `amoxicillin` (correct English INN) tied/ambiguous → abstained; the sentence word "uống" polluted fuzzy scoring (matched `tretinoin_uong`). - `warfarin với aspirin` (interaction) → ambiguous → abstain; no interaction path. - `còn liều dùng thì sao?` follow-up lost the drug (inconsistent inheritance). - `trẻ 5 cân paracetamol` → clarifies forever; no mg/kg weight-based calc node. - symptom→drug and BSA/Part-1/Part-3 → abstain (scope gaps). **Corrected an earlier overstatement (owner was right):** section chunking is NOT uniform — 172/684 monographs (25%) are class monographs cramming many sub-drugs into one section (INSULIN dose = 9,268 chars / 20 ATC, VITAMIN D 14,197 chars), chunked by blind token-window. So re-chunk (sub-drug/population/indication-aware) IS warranted later — but it does not fix the front-end failures above. **Rebuild step 1 — LLM query-understanding front-end (new, PROVEN live).** `apps/ai-service/rag/understanding.py`: `LlmQueryUnderstander` + `QueryFrame`. One LLM call reads the messy turn (+ history + the real 684-drug catalog) → a structured frame (turn_type, drugs [catalog-validated], unknown_drugs, attribute, population, weight_kg, indication). Safety kept: the model may only pick drug_ids from the real catalog; an unrecognised name goes to `unknown_drugs`, never snapped to a near drug. `rag/` stays SDK-free (LLM injected as a `JsonLlm` protocol, satisfied by the existing `BedrockConverseAnswerGenerator`). Proven on the live LLM against all 7 killer cases the old resolver failed — every one now read correctly (amoxicillin→amoxicilin, aspirinol→unknown, warfarin+aspirin→interaction with both drugs, trẻ 5 cân→dosing_calc weight=5.0, sốt cao→symptom_to_drug, follow-up→inherited metformin, chào→smalltalk). **NOT yet done:** the frame is not wired into retrieval/generation — the old `CatalogDrugResolver`/`SectionResolver` still drive `/v1/rag/query`. Next: route on `turn_type` (interaction→gather both drugs; symptom_to_drug→reverse `chi_dinh` lookup; dosing_calc→a tested mg/kg calculator like `rag/calculators.py`), unit + live eval vs the battery, then decide the structure-aware re-chunk (needs owner GO for re-embed ~$0.5). No re-embed or cloud spend beyond cents of diagnostic/proof LLM calls this session. ## 2026-08-05 (night) — Live-chat UX overhaul: reasoning/clarify, multi-turn, Qwen3; plan = finish chatbot tomorrow, deploy next week Owner drove the running web chat with messy real inputs and found the offline-era query layer was a hodgepodge. Fixed the failures found, each **verified by chatting the running service** (not just unit tests). Model switched to **qwen.qwen3-next-80b-a3b** (DeepSeek ignored the clarify instruction; Qwen3 and gpt-oss both follow it — A/B'd). ai-service **118 passed, 3 skipped**. Fixed (commits `6c6a916`, `5feccba`, `553be09`, `7f45d06`): - **Reasoning/clarify (the headline):** a focused sufficiency-check LLM call runs BEFORE generation. An under-specified dose ("paracetamol cho trẻ em") now ASKS age/weight/route/indication instead of dumping every band. Adult dose / CCĐ / interactions answer normally (no false clarify). `answer._check_sufficiency` + `prompt.build_sufficiency_request`; `GroundedAnswer.clarification` → decision "clarify". - **Multi-turn:** "thuốc đó…" was double-resolved (inherited then re-resolved from rewritten text → ambiguous → empty). Now the resolved drug_id is passed straight to retrieval (`routing.retrieve_for_drug`); raw turn drives section routing; overview+rerank finds the part. Verified: Oxymetazolin → "thuốc đó cho trẻ dưới 6 tuổi?" → correct than_trong answer. - **Did-you-mean garbage:** fuzzing a sentence ("EPO…") or "đúng" returned terbinafin/tretinoin in a loop. Now suggestions only for short drug-name misses; confirmations get "which drug?". - **Bare name → drug intro** (class + indication + invite), not a forms dump. - **Citations = only the [n] actually cited** (was ~13 chips for a 1-source line). - Rerank trims overview 29→6; inherited-drug notice uses the display name. **Operational lesson (cost real time):** `uvicorn --reload` does NOT work on this Windows box — the owner chatted STALE servers repeatedly. Must kill :8079 and restart after every edit. Recorded in memory `reference-env-operational-gotchas` and `feedback-chatbot-hard-lessons`. **Cost/safety:** IAM `BedrockEmbeddingInvoke` v6 (embed + rerank + deepseek + qwen3 x2 + gpt-oss x2). Verified 0 EC2, no provisioned throughput — **pay-per-call only, idle ≈ $0**. **Plan — finish the chatbot TOMORROW (2026-08-06), deploy focus next week:** 1. Re-embed the 9 reconstructed tables into Qdrant (owner approved; was wrongly blocked) — ~30–60 min to make them searchable. 2. "EPO"/abbreviation expansion (LLM entity extraction or aliases) — ~2–5h. 3. VERIFY_PDF/crop lookup UX in the web — ~2–4h. 4. UI showing generated-vs-extractive + retrieval path/evidence — ~2–4h. The **coding** fits a day. NOT finishable tomorrow and deliberately off the deadline: reconstructing the other **142 quarantined tables** + a **pharmacist review** of the corpus — that is the clinical-validation long pole (days→weeks, needs a human), separate from "chatbot features done". ## 2026-08-05 (evening 3) — The LLM cloud is LIVE: DeepSeek generation + Cohere rerank on the real corpus The owner rejected the $0 offline build as the deliverable and set a hard deadline. The chatbot is now a **real LLM RAG**, grounding kept ON, running the full HTTP stack (ai-service :8079 ↔ Postgres trace ↔ Qdrant; web :3000). Commit `9c4273b` (plus `92497ae`/`9e9cef7`/`1b6f399` earlier this session, which committed the previously-uncommitted evening-1/2 work). **What was turned on** (live via gitignored `.env`; committed defaults stay `disabled`/`section-only` so CI/fresh-clone never touches cloud): - `EMBEDDING_PROVIDER=cohere-v4` — query now embedded in the corpus's `cohere.embed-v4:0` space (probe: 1024-dim, L2 1.0, ~1.95s). No re-embed; the 15,100 vectors already exist. - `ANSWER_PROVIDER=bedrock-converse` + `deepseek.v3.2` — new `adapters/bedrock_converse.py` (Bedrock **Converse** API, boto3, model-agnostic; Qwen/GLM = 1 env + 1 ARN). Probe OK. GPT-4o confirmed NOT on Bedrock; OpenAI `gpt-oss`, DeepSeek, Qwen, GLM, Mistral, Kimi ARE (checked live). - `RERANK_ENABLED=true` — `cohere.rerank-v3-5` trims the overview/similarity fallback: a free-form drug question no longer dumps all ~29 sections at the model (**measured 29 → 6** on the fever/paracetamol case). Section route never reranks; fail-open (outage → book order, answer survives). - `rag/prompt.py` rewritten to current citation-enforced practice: each dose carries its population/condition label (no adult/paediatric mixing), cite only the supporting block, no `[n]` spam, abstain on insufficient evidence. **IAM:** managed policy `BedrockEmbeddingInvoke` bumped to v4 (invoke on titan-embed, cohere.embed-v4, deepseek.v3.2, cohere.rerank-v3-5); repo file synced. Codex was off, no collision. **Verified:** ai-service **111 passed, 3 skipped** (+12 this milestone). Live HTTP `POST /v1/rag/query` returns a grounded LLM answer with a citation and a Postgres trace id. Golden `golden_e2e` (35 Qs): **19/19 answerable questions grounded with the correct drug** (incl. typo `paracetamon`, alias `Acetaminophen`, multi-turn inheritance); 14 adversarial correctly abstained (fake drugs, weather, symptom→drug reverse-lookup, multi-drug). **Two real gaps:** a price question answers from the monograph instead of "no price in the formulary", and "should I double the dose?" is not directly warned. Every `generated=True` answer passed `grounding.verify`. **Cost/safety:** Bedrock is pay-per-call — verified **0 EC2** (3 regions) and no provisioned throughput; idle = ~$0. A few dozen probe/smoke/eval calls this session, cents-scale on the estimate; exact bill not checked. **Separate track, NOT done (background subagent started, own worktree):** reconstruct the 151 quarantined tables with a `needs_expert` flag on uncertain cells + parse Part 1 (poisoning/pregnancy/hepatic-renal) & Part 3 (BSA/ATC) + re-embed. This is a multi-hour ingestion pass with the whole-doc validation gate and will NOT be clinician-validated within the deadline — deliberately kept off the deadline path. ## 2026-08-05 (evening 2) — Conversational chat core wired LIVE end-to-end (offline, $0); owner wants the LLM cloud next The chat core is now **live and serving multi-turn**, not just unit-tested. It runs `$0`/no-cloud because the section-route is a payload filter (no query embed) and generation is still off (verbatim), but the *conversational* behaviour is real and smoke-tested against the running service (ai-service :8079, web :3000). Built (`rag/conversational.py` `ConversationalLoopService`, wrapping the safe `GroundedAnswerService`; wired through `bootstrap.py`/`main.py`/`routers/rag.py` with an optional `conversation_id`, plus `route.ts` sending it and a `ChatPanel` error state): - **Multi-turn follow-up inheritance.** "Chống chỉ định Metformin" then "còn trẻ em thì sao?" carries the drug+section forward and names it ("Về metformin: …"). - **Smalltalk.** "chào bạn" gets a friendly redirect, not a failed-drug-lookup refusal. - **Drug-name-only → the whole monograph.** Typing "PARACETAMOL" now returns all 18 sections in book order with `【heading】`s and per-section citations (`QdrantRetriever.find_by_drug` + `SECTION_ORDER`; `RetrievalService` uses it when a drug resolves but no attribute is named) — the earlier "specify an attribute" dead-end is gone. - **Typo → ask, never threshold-guess.** Only an EXACT drug name auto-resolves; a fuzzy match is offered as a question ("Ý bạn là: Metformin?") via `CatalogDrugResolver.suggest(min_score=0.72)`. A completely-wrong name → "Không có thuốc này trong Dược thư Quốc gia." A formulary must not silently answer about a *different* drug than the one meant. - **Autocomplete endpoint** `GET /v1/rag/suggest?q=` (`CatalogDrugResolver.complete`, substring/prefix) — the frontend dropdown that consumes it is still to build. - **BSA calculator** `rag/calculators.py` (Appendix 1, DuBois, tested vs the book's own cells). Verification: **ai-service 99 passed, 3 skipped**; live smoke test of all four conversation behaviours plus the monograph/typo/not-supported cases. A refine-loop bug (a refined query dropped the inherited drug and abstained, discarding a good answer) was found in my own code and removed before shipping — clarify + inheritance are the loop's value, retrieval-refine is not, and it is gone from the live path. **Owner's next-session directive (recorded in memory `project-llm-cloud-plan`):** stand up the cloud LLM — semantic query embedding (`EMBEDDING_PROVIDER=cohere-v4`, already IAM-permitted) and answer generation (a cheap model, non-Anthropic OK, via a Bedrock Converse adapter, needs its ARN added to `BedrockEmbeddingInvoke`). The offline build was budget/safety-first, not LLM-avoidance; the owner wants the real AI experience next, with `grounding.verify` and the quarantine contract kept ON. ## 2026-08-05 (late) — Read the source book's own structure; scope + usage-pattern findings (checkpoint before handoff) Read the Dược thư 2018 front matter directly (printed p8 "Nội dung", p39 "Hướng dẫn sử dụng") to understand what the book is *for* and how clinicians use it — recorded in memory `reference-duoc-thu-2018-structure`. Key facts that reshape the chatbot scope: - The book has **three parts**. The corpus is **Part 2 (drug monographs, printed 99–1496) ONLY**. **Excluded and clinically important:** Part 1 general chapters (printed 37–98: prescribing in the elderly / hepatic-renal impairment / children / pregnancy-lactation; disease-class guidance for asthma, epilepsy, HIV, antibiotics, TB, hepatitis B, antipsychotics; drug allergy; **poisoning & antidotes**; drug-interaction principles) and Part 3 appendices (printed 1497–1528: **body-surface-area calc**, IV admixture, ATC classification). So "how to treat asthma", "antidote for X", "BSA-based dosing" have no data in the index today — a coverage limit, not a retrieval bug. - The 19 monograph fields are fixed and documented on p39; a field is omitted when the book has no info (so a missing section is not necessarily a parse bug). - Field 14 dose is a *general adult+child oral reference dose unless stated*; the clinician adjusts. → the tool supplies reference data, not a prescription. Data checks run this session (against `chunks.jsonl`), correcting earlier pessimism: - Indication is searchable: 48 drugs' `chi_dinh` mention "sốt". Reverse lookup (symptom → drugs) is feasible from **content**, but retrieval is drug-first, so not answerable yet. - **mg/kg dosing is in PROSE, not tables**: 574 `lieu_luong` chunks contain "mg/kg", all prose, across **295 drugs**, 473 of them mentioning trẻ em. So the *primary* weight/age dosing (incl. pediatric) is answerable; the 83 quarantined dosing tables are mostly the *supplementary* renal-adjustment tables (49 of those 83 drugs also have mg/kg prose). - Pregnancy dosing is mostly **qualitative**: 670 drugs have a `thoi_ky_mang_thai` section but only ~23 chunks carry a mg figure — the book rarely gives a separate pregnant dose, so answer = pregnancy caution + standard dose, never a fabricated pregnant-specific number. - `drug_id` can be compound (`paracetamol_acetaminophen`); alias resolution must map "paracetamol" → that id. **Design consequence discussed with the owner (not yet built):** the "understand" stage must classify the *turn type* (smalltalk / medical query / multi-drug interaction / symptom-indication / out-of-scope / injection-shaped), not just resolve a drug. Refusing a clinician's symptom→drug question as "recommendation_out_of_scope" was wrong for this audience — such questions are indication lookups and should be answered from `chi_dinh`. Multi-drug interaction/contraindication questions need a real PLAN → gather both drugs → synthesize step (the ADR-0007 PLAN node, still unimplemented), and an absence-of-evidence answer must state where it looked, never assert "safe". **Session state / not yet done (so a fresh session can resume):** the chat module's domain glue is built and unit-tested (`rag/conversation.py` ports + summariser, `rag/conversational.py` orchestrator + `is_smalltalk`); it is **not** wired to the endpoint. Live wiring (turn-type classifier, loop-around- GroundedAnswerService, Postgres store, `conversation_id` on `/v1/rag/query`, `route.ts`, ChatPanel error state), the P0 audit fixes (§5 context-mixing metadata, §8 Qdrant-error degradation), reverse-indication retrieval, and the table vision-consensus pipeline all remain to do. No code was wired live this session; the behavior spec is still being clarified with the owner before wiring. **Owner decision: parse the WHOLE book, re-chunk freely** (not just Part 2 monographs). Current corpus covers physical pages **100–1494** only. To add: Part 1 general chapters (physical ~36–97) and Part 3 appendices (~1496–1527); front-matter list (13) and index (1529+) are already used as validation ground truth. Read `segment/detector.py` to ground the plan — **the machinery already generalizes**: a chapter title ("NGỘ ĐỘC VÀ THUỐC GIẢI ĐỘC") has the *same shape* as a monograph title (bold + mostly-upper + short), so `is_monograph_title_candidate` extends by widening the hardcoded `99–1496` range per `content_type`. Only two real changes: (1) parametrize the page range + add a `content_type` (`monograph|chapter|appendix`); (2) chapter/appendix sub-headings are **free-form** ("Hô hấp", "Co giật"), not the 19-key vocab, so `detect_section_headings` needs an open-taxonomy mode (bold + short = heading, store the text, no `match_section` requirement). Everything downstream (span extraction, table/formula quarantine, provenance, chunker) is content-type-agnostic and reused → schema v5 adds `content_type` + `chapter_id`. **Gate (CLAUDE.md): the span-routing ledger must account for ALL 1668 pages with `unassigned=0`, not just 99–1496.** Then embed only the NEW chunks (Cohere, pennies, announce first). This is a focused ingestion pass (detector + assembler + chunker + whole-doc re-run + validation), not a one-liner — not attempted this session beyond grounding the plan. **Done this session (App 1, self-contained, validated):** `rag/calculators.py` `body_surface_area_m2` replaces Appendix 1's lookup table with the book's DuBois formula (`S = W^0.425 × H^0.725 × 71.84`), tested against three of the book's own table cells (165cm/60kg→1.66, 90cm/10kg→0.50, 170cm/70kg→1.81) — `tests/test_calculators.py`, 3 passed. Audit §7 (calculation = tested function, never an LLM). ## 2026-08-05 (evening) — Conversational orchestrator wired to the existing loop; data-quality audit; budget verified live **Chat module (the glue ADR 0007 specified and nothing had called).** Added the two missing conversation ports and their offline defaults to `rag/conversation.py` (`ConversationStore`/`InMemoryConversationStore`, `Summariser`/`DeterministicSummariser`) and the orchestrator `rag/conversational.py` (`ConversationalRagService`). It owns no rules of its own: load state → resolve this turn → inherit gaps from `Focus` → derive clarify signals from resolver state → `reasoning.run_turn` → update focus, append turns, summarise overflow, save → name any inherited drug. Runs with no LLM/service (collaborators are protocols). `DeterministicSummariser` records only drug/section **labels**, never cell values, so the no-clinical-content-in-summary rule holds by construction rather than by trust — closing the summary-bypasses-grounding hole flagged in review. **11 new tests; full ai-service suite 91 passed, 3 skipped.** **Still NOT wired live:** a `TurnResolver` bridge over `CatalogDrugResolver` + `SectionResolver`; bridges from `RetrievalService`→`Retrieve` and the grounded generation path→`Generate`; `PostgresConversationStore` + migration; a `conversation_id` on `/v1/rag/query`; `route.ts` sending it and dropping the hardcoded `intent: fact_lookup`; a `ChatPanel` error state; and the multi-turn eval run. So no claim yet that history/loop improves answers — designed and unit- proven, not measured end to end. **Data-quality audit (self-run this session, not quoted from this log).** 684 drugs; critical-section coverage is strong — dosing missing 0.1% (1), contra- indication 0.4% (3), indication 0%. **But 83/684 drugs (12%) have their dosing inside a quarantined table**, so a dose query for them returns `VERIFY_PDF` (crop, no number) — the largest answer-quality gap for a clinician audience, and it lands on the single most-asked query. 125 chunks carry a leading `": "` label-leak artifact (93 in `ten_chung_quoc_te`). Vector-path text loss appears contained to 22 flagged lines (completeness of detection unverified). Nobody clinician-side has validated the 8.2M chars against the book — still the largest unmeasured area. **Table validation — the instrument that text extraction lacked.** Demonstrated that vision reads a real quarantined dosing table cell-by-cell: GABAPENTIN's renal-adjustment table (printed 706) came back exactly by eye where pdfplumber's text layer could not structure it. Found and corrected a page-index off-by-one in my own render (data `physical_page` N = `doc[N]`, 0-based) — proof that correctness must not depend on trusting coordinates. Strategy, given pharmacists are **end-users, not labelers**: reconstruction powers **retrieval only**; the displayed answer stays crop + page (clinician verifies at point of use). Validation is automated — vision↔geometric consensus + round-trip visual + book invariants — with a per-cell precision-first gate (disagreement → stays crop-only). Not yet built; 151 blocks is small enough for full census. **Budget, read live from the billing console** (owner login; `ai-lab-user` has no billing API perms): **$138.50 remaining, entirely AWS promotional credit, not the owner's card**; August bill $0. Deploy target chosen: team k3s, but deferred (mutating a shared cluster). Generation still off (`answer_provider= disabled`) — extractive/verbatim, which is defensible for clinicians; wiring a cheap model (Nova/Haiku via Bedrock Converse) needs its ARN added to the `BedrockEmbeddingInvoke` policy, which today grants invoke on the two embedding models only. ## 2026-08-05 — An answer layer that cannot state a number the book does not Today started by walking the **demo path** rather than the test suite, and the two are not the same thing. The suite was green and the demo was broken. **What the walk found, by running it rather than reading it.** The backend answers real Vietnamese questions against the real embedded corpus with real citations and **zero cloud cost** — the section route is a payload filter, not a vector search. `Chống chỉ định của Metformin là gì?` returns the true contraindication text with one citation; `Tương tác thuốc của Warfarin?` returns two. But `Tôi sốt cao, uống Paracetamol được không?` returned **HTTP 500**: the similarity fallback reached Bedrock, which is revoked, and `botocore.AccessDeniedException` escaped as an unhandled error. Any question whose phrasing is outside the section phrase table takes that path. That crash also **re-verified the cloud shutdown today, live** — the denial came from the service, not from a claim in a document. **Four defects, all fixed, all at $0.** 1. **The 500.** `adapters/embedding.py` now translates provider failures into the domain error `QueryEmbeddingUnavailable`, and `RetrievalService` catches it and abstains with `reason="query_embedding_unavailable"` — deliberately distinct from `insufficient_retrieval_score`, so an outage never reads as an empty corpus. `rag/` still imports no SDK. 2. **A default config that does not work.** `config.py` pointed at collection `duoc_thu_chunks`; the real one is `duocthu_v1`. `embedding_provider` defaulted to `disabled`, so `/v1/rag/query` returned 503 on a fresh clone. 3. **Neither existing provider was a safe default.** `local-smoke` searches a SHA-256 vector against a Cohere collection — confident, meaningless hits. `cohere-v4` spends the boto3 retry budget (~30s) before failing on a revoked account. Added `SectionOnlyQueryEmbedder`: refuses locally and instantly, so retrieval is confined to the route that measured 16/16. 4. **Safety abstention was incidental, not a gate.** Symptom questions abstain with `reason="drug_not_resolved"` — because no drug name was found, not because anything recognised a symptom question. Recorded, not yet fixed. **The answer layer now has an LLM, and a check that makes "it does not fabricate" measurable rather than promised.** Previously `rag/answer.py` was extractive: it concatenated retrieved chunks. That is why `Liều Paracetamol cho người lớn?` opened with `5 - 12 tuổi: Trẻ em 12 - 18 tuổi:` — raw section text, paediatric doses first, for an adult question. Generation is now three layers, and only the third is load-bearing: - **Prompt** (`rag/prompt.py`, domain — no SDK): evidence only, figures copied character-for-character, `[n]` citations required, insufficient evidence is a valid answer. Output shape is pinned by `output_config.format`, so a malformed envelope is the provider's error, not our parsing problem. - **Verification** (`rag/grounding.py`, pure domain): every numeric token in the generated answer must appear **exactly** in the evidence, and every `[n]` must resolve. Citation markers are stripped before number extraction so `[2]` is never read as the quantity 2. - **Fail-closed** (`rag/answer.py`): ungrounded number, invalid citation, malformed output, provider outage, or the model itself reporting insufficient evidence — every one falls back to the verbatim source text, which was computed first and is therefore always available. **Numbers are compared as strings, and that is the decision worth keeping.** No parsing, no normalisation. `1.500` is 1500 under one reading and 1.5 under another; a normaliser that strips separators maps `7,5` and `75` to the same key, scoring a **tenfold dose error as a match**. Pinned by `test_decimal_separators_are_not_interchangeable`. The same rule refuses `2 g` → `2000 mg`: arithmetically right, but unit conversion is where dosing errors live, so it is refused rather than interpreted. Quarantined tables and formulas are **never generated over**. `VERIFY_PDF` returns before generation — those are precisely the blocks whose numbers were not reliably reconstructed, so rephrasing them is the one case where fluency could invent a dose. This keeps ADR 0006's contract intact. **Provider chosen on the owner's instruction: AWS Bedrock + Claude.** `adapters/bedrock_claude.py` is the only module naming the `anthropic` SDK, imported lazily. Two provider facts taken from the Anthropic API reference today, not from memory: Bedrock model ids carry an `anthropic.` prefix (`anthropic.claude-opus-5`), and the Messages-API path on Bedrock is `AnthropicBedrockMantle`, **not** the legacy `bedrock-runtime` InvokeModel route the embedding adapter uses. A `stop_reason: "refusal"` is a successful HTTP response with no usable content, so it is routed to the extractive fallback rather than allowed to raise on `content[0]`. **This adapter has never been run against Bedrock.** Cloud access is still revoked and no IAM change was made today. `StubAnswerGenerator` exercises the entire path — prompt build, schema parse, grounding check, fallback — with no cloud call, and that is what the end-to-end run below used. **Observability, because a dashboard is a better answer than a slide.** `rag/metrics.py` defines the counters in the domain; `adapters/prometheus.py` is the only module naming `prometheus_client`, imported lazily; `/metrics` returns 404 rather than an empty 200 when metrics are off, so a scrape cannot succeed silently with no samples. The headline counter is `duocthu_generation_rejected_total{reason="ungrounded_number"}` — the measured form of the no-fabrication claim. A mismatched label drops the sample instead of raising: metrics must not be able to break a clinical answer. `infra/docker/` gains Prometheus and Grafana with a provisioned datasource and dashboard. **Not yet verified running** — the image pull was still in progress when this was written. **A section was being served scrambled, and only using the UI found it.** `liều dùng paracetamol` opened mid-sentence on `5 - 12 tuổi:` and buried `Liều lượng: Người lớn:` seven hundred words down. `find_by_section` returned whatever order Qdrant scrolled, and point ids are `uuid5(chunk_id)`, so PARACETAMOL's five dosing parts came back **3, 4, 1, 2, 0** — verified by scrolling the real collection, not inferred. `part_index` was in the payload all along and simply never used. Now sorted by it; a part missing the field sorts last rather than being dropped, because a silently shortened dose list is worse than an unordered one. Pinned by `tests/test_section_order.py`, including the exact 3,4,1,2,0 case. **This is a clinical defect, not a cosmetic one:** a reader who stops partway through stops in the middle of a different population's dose. Every section-routed answer given before today — including the 16/16 golden result — was assembled in this scrambled order; retrieval picked the right chunks, so the measurement stands, but no statement about how those answers *read* survives it. **Conversational reasoning RAG: designed in ADR 0007, domain layer built.** `rag/conversation.py` carries `Focus` (drug, section, population, verbosity, each stamped with the turn that set it) and the recent-turn window; `rag/reasoning.py` is the bounded loop. Both are pure domain and run with no provider, which is the point: *which drug is this still about* must be deterministic, not inferred. Three rules make inheritance safe in a formulary, each pinned by a test: an explicitly named drug always beats context; focus older than six turns is dropped rather than carried, because a stale drug is a wrong-drug answer, not context; and any answer built on an inherited drug must name it. The loop's uncertainty signal is **not** a model confidence score. It is the resolver states that already existed and previously dead-ended into `abstain` — ambiguous drug, unresolved attribute, multi-attribute question — which now produce a clarifying question. Deterministic, testable, and explainable to a reviewer in a way that "the model felt 0.73 sure" is not. A clarify signal short-circuits before any budget is spent, verified by asserting the budget is untouched and neither retriever nor generator was called. Budgets are decremented **before** the call they pay for, so exhaustion degrades to the best answer so far. A retrieval round is bought only by a *named* gap with a genuinely new query: `test_an_unnamed_gap_does_not_buy_a_round` and `test_a_refinement_that_changes_nothing_stops_the_loop` are the guards against a loop that spins on a feeling or re-issues the same query. `Golden Dataset/golden_multiturn_v1.csv` is new — 8 conversations, 19 turns, 6 of them inheritance-dependent. The existing golden file is single-turn by construction and can measure none of this. Includes the adversarial turns: a follow-up after a refused fake drug (must not borrow a drug from elsewhere), and a follow-up after a symptom question (must not inherit treatment intent). **Not yet wired:** the loop is not called by `GroundedAnswerService` or the router, there is no `PostgresConversationStore`, and no evaluation run over the multi-turn file has been performed — so no claim is made that history or the loop improves answers. The design states how that will be measured; it has not been measured. **LangChain was considered and rejected.** The repo already has the ports and adapters LangChain would supply, retrieval is already measured, and the guardrail is already domain code. Adopting it a week before a review would rewrite the working part for no measured capability gain. Verification actually run: ai-service **56 passed, 3 skipped** (37 + 3 before, +19); ingestion **296 passed**, checked for regression, unchanged; `duocthu_v1` holds **15,100 points** at 1024-dim Cosine, matching the manifest; live service against the real collection answered three clinical questions with citations and abstained on six of the seven safety probes; `/metrics` scraped and returned `duocthu_generation_served_total 2.0` and `duocthu_abstention_total{reason="drug_not_resolved"} 1.0`. Not established, and load-bearing for the demo: **`apps/web` is still entirely mocked** — `packages/api-client/src/sendChatMessage.ts:8` returns `buildMockResponse(content)` and the whole frontend contains no HTTP call to the backend, so the working API and the working UI are not connected; `api-gateway`, `chat-service` and `auth-service` hold **0 source files**; the Bedrock generator has never been invoked; `intent` is still supplied by the caller, so the recommendation gate depends on the client declaring it honestly; and the Prometheus/Grafana stack has not been seen running. ## 2026-08-04 (evening) — Section routing: contraindication retrieval goes from 0.05 to 1.00, at zero cloud cost The retrieval defect measured earlier today is fixed by routing rather than by embedding. **No cloud call was made and nothing was re-embedded** — Bedrock access is still revoked. **The change.** A question that names its own attribute does not need similarity to guess which section answers it. `rag/sections.py` maps the question to a `section_key`; `QdrantRetriever.find_by_section` then filters on `(drug_id, section_key)` and returns **every** part of that section as a `scroll`, not a top-k. `RetrievalService` takes that route when it resolves and falls back to similarity otherwise. Two rules carry the safety. **Longest phrase wins**: "chống chỉ định" and "chỉ định" differ by one prefix word and mean opposite things, so every phrase is sorted by length and the longer is tested first — the same rule keeps "quá liều" from being read as "liều" and "hướng dẫn xử trí ADR" from being read as "tác dụng phụ". **No match is not a guess**: an unrecognised question returns `None` and falls back rather than picking a section it is unsure of. **Measured against the real `duocthu_v1` collection, no embedding involved:** | | similarity (measured this afternoon) | section routing | |---|---|---| | hit@1, 160 generated cases | 0.544 | **1.000** | | `chong_chi_dinh` | **0.05** | **1.00** | | misroutes / empty / leaked sections | — | 0 / 0 / 0 | **The generated 160 flattered it, and testing on human-written questions said so.** Those questions use the phrasings the table was built from, so 160/160 is partly circular. Run against the 16 single-drug questions humans actually wrote in `Golden Dataset/golden_e2e_v1.csv`, the first version scored **10/16**. The six failures were two gaps: four questions say just "Liều Metformin cho người lớn?" — bare "liều", which the table lacked — and one says "Bà bầu", a colloquial phrasing for pregnancy. Adding those phrases (no code change, which is what the open/closed table is for) took it to **16/16** while the confusable pairs still resolve correctly; bare "liều" is safe only because "quá liều" is longer and tested first, and there is a regression test pinning exactly that. **A circular import was found and fixed properly rather than worked around.** `service -> sections -> routing -> service`, because `normalize_name` lived in `routing.py`. It is a text utility with no knowledge of drugs or sections, so it moved to `rag/text.py`; `routing.py` re-exports it so existing imports keep working. **Also wired, and still unproven:** `BedrockCohereQueryEmbedder` replaces the SHA-256 hash embedder for the similarity fallback path. It has been import-checked only — **never run against Bedrock** — so the fallback path remains unverified end to end. The section route does not depend on it. Verification actually run: ai-service **37 passed, 3 skipped** (22 before, +15); ingestion **296 passed** (unchanged, checked for regression); `ruff --select F,E9,B,ARG` over `rag/`, `adapters/`, `bootstrap.py`, `config.py` and `tests/` — **all checks passed**; section-route evaluation against the live collection 160/160; human-written golden questions 16/16. Not established: multi-attribute questions ("liều dùng và chống chỉ định") pick the longest phrase, which is deterministic but arbitrary; phrase coverage beyond these 16 human questions is unmeasured; and none of this speaks to whether the retrieved text is clinically correct. ## 2026-08-04 (afternoon) — First real embeddings exist; retrieval measured at 54% and the cause is not what the small sample said The corpus is embedded for the first time. Bedrock IAM was opened on the owner's explicit instruction, all 15,100 chunks were embedded with `cohere.embed-v4:0`, loaded into Qdrant, and **cloud access was then revoked and proven revoked** before the owner's 17:00 deadline. Measured spend **~$0.49** of a personal $138 budget. **Gate results.** 15,100/15,100 embedded; 15,100 points in `duocthu_v1` over 59 batches; collection point count 15,100 — count gate **PASS**. Manifest records `cohere.embed-v4:0`, 1024 dimensions, Cosine, corpus SHA `04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`. Corpus SHA was re-verified against the morning audit before spending: identical, and identical to the post-lint copy, so the 12:05 `chunker.py` edit did not change output. **Both providers were probed live before choosing.** Titan v2 and Cohere v4 each returned 1024 dimensions with a **measured L2 norm of 1.000000**. That settles a question left open since 2026-08-03: Cohere's `normalized` field was `None` because AWS's docs never state it. It is now measured. Cohere was chosen on two measured grounds — the corpus is Vietnamese and Cohere is explicitly multilingual, and `bedrock_cohere.py` batches 96 texts per request while `bedrock_titan.py` sends one, which at a measured 2.3s per call is ~9.6 hours versus minutes. The $0.41 price difference did not drive it. **The retrieval number, and a correction to a claim made earlier the same day.** A 160-case evaluation (20 per section, 8 sections, questions generated from the corpus so labels are structural) measured **hit@1 0.544, hit@3 0.663, hit@5 0.738**. Per section: | section | hit@1 | |---|---| | `chong_chi_dinh` | **0.05** (1/20) | | `chi_dinh` | 0.30 | | `tac_dung_khong_mong_muon` | 0.40 | | `lieu_luong_va_cach_dung` | 0.60 | | `qua_lieu_va_xu_tri` | 0.65 | | `than_trong` | 0.65 | | `tuong_tac_thuoc` | 0.80 | | `thoi_ky_mang_thai` | 0.90 | An earlier 15-case run gave a similar headline (0.533) but led to the **wrong diagnosis**: four of its seven failures were contraindication questions answered with indications, so the cause was reported as embedding weakness at negation. At 160 cases that pair accounts for only **3** confusions. The dominant mechanism is different and larger: **`duoc_ly_va_co_che_tac_dung` absorbs questions from every other section** — 10 from adverse effects, 8 from contraindications, 7 from dosage, 5 from indications. It is the largest section (1,896 chunks) and describes the drug in general terms, so it sits close to almost any question about that drug. This is the small-sample failure mode CLAUDE.md warns about, reproduced on this project. **Re-embedding cannot fix this, and the capability to fix it already exists.** Verified by reading the code, not assumed: `apps/ai-service/adapters/qdrant.py` `search()` filters on `drug_id` only and lets vector similarity choose the chunk; `rag/routing.py` resolves drug and intent but **not section**; and `find_by_payload` — the "return the whole section" method in `ingestion/load/` — is **never called anywhere in `apps/ai-service`**. Attribute questions therefore depend on similarity picking the right section, which is what measures 54%. The fix is to resolve the attribute to a `section_key` and retrieve that section whole; `ATTRIBUTE_TO_SECTION` already exists in `embed/benchmark_local.py`. **A silent-failure hazard found and closed.** `apps/ai-service` embedded queries with `LocalHashQueryEmbedder` — SHA-256 of tokens, explicitly plumbing only — while the collection now holds Cohere vectors. Querying across those two spaces returns hits and raises nothing; the results are simply meaningless. `BedrockCohereQueryEmbedder` was added and wired behind `EMBEDDING_PROVIDER=cohere-v4`. **It has only been import-checked — never run against Bedrock**, because cloud access was revoked first, as instructed. **Two operational lessons, both paid for.** `bedrock_runtime.py` set no boto3 timeout, so a single throttled response held a socket open for over five minutes and stalled the whole run; `connect_timeout=10, read_timeout=60` plus standard retries fixed it. Then the first full run still died at ~14,600/15,100 because the retry backoff (2s, 4s) was far shorter than a per-minute token quota needs. The disk cache made that survivable: the resumed run recorded **14,977 cache hits and 123 misses**, so only 123 vectors were paid for twice — zero, in fact, since the first run's work was already saved. **Cloud shutdown, verified rather than asserted.** Both policies detached and deleted; `InvokeModel` and `ListFoundationModels` both now return `AccessDeniedException`. No EC2 instance, no EBS volume, and — because the policy never granted `CreateProvisionedModelThroughput` — no way for this identity to create the one Bedrock resource that bills hourly. Not established: retrieval quality is not acceptable for clinical use, no clinician-authored release gate exists, the generated evaluation questions use template phrasing rather than real clinical language, and no LLM answer layer has ever run against real evidence. ## 2026-08-04 — Chunk schema v4 passes the embedding-readiness gate Reviewed the live Claude coordination and its last changes before editing. The delivery plan was objectively stale: it still described schema v2/15,076 chunks, empty embed/load/API modules, embedding before content-safety gates, and allowed unverified inferred table headers as retrieval text. The plan and ADR 0004/0006 now put content safety, exact provenance, fail-closed schema validation and local pseudo-vector smoke tests before any provider call. Bedrock remains benchmark- only and requires separate owner approval for any paid/full-corpus run. Implemented schema v4 and regenerated the canonical chunk artifact. Retrieval `text` may repeat route/population labels so continuation chunks remain safe in isolation; contiguous `source_text` remains byte-reassemblable and drives exact physical/printed page provenance. `context_labels` records retrieval-only prefixes. All 151 unverified table/formula descriptors embargo `header_row` and cell-like column text. Attachments now carry physical page, printed page, `block_id`, `bbox` and optional crop, and those region references survive the Qdrant adapter and RAG citation response. The loader accepts exactly schema v4, rejects booleans/non-integers/out-of-range pages, and keeps the normalized LF/CRLF-stable corpus identity. Canonical artifact measured after regeneration: - 15,100 chunks: 14,949 prose + 151 block descriptors; - 4,105,382 `cl100k_base` tokens; 0 chunks above the 800-token ceiling; - all `chunk-ready` gates pass: exact provenance, source uniqueness, reassembly, attachment coverage, descriptor embargo and schema checks all have 0 failures; 151 descriptors match 151 quarantined blocks; - raw file SHA-256: `8dfae08ae6d9222089c5cdb4207a064fe67989f10f7552b555af0aef6331d9a1`; - normalized corpus SHA-256 used by the Qdrant manifest: `04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`. Verification actually run: - ingestion: **292 passed**; focused post-lint patch: **26 passed**; - AI service with `RUN_INTEGRATION=1`: **25 passed**, including real local Qdrant, PostgreSQL and FastAPI round-trips; - full canonical local smoke with deterministic 4D pseudo-vectors: first and second loads both upserted 15,100 records and both held exactly 15,100 points; manifest hash matched; data and sidecar test collections were removed and Qdrant returned to 0 collections; - Ruff `F,E9,B,ARG` on the files changed for this gate: clean; `git diff --check`: clean (Git only reported Windows LF/CRLF conversion warnings). Conclusion: the canonical corpus is **technically READY TO EMBED**, meaning its input/schema/provenance/load plumbing meets the measured gates. This does not authorize a provider call, does not establish retrieval quality for any model, and does not prove whole-book medical accuracy. Human-reviewed clinical eval, table reconstruction, and recall for borderless tables/bar-less formulas remain outside what these gates prove. ## 2026-08-04 — Real local datastore plumbing, guarded RAG API, and printed-page citations Read the live Claude Code process and coordination before editing. Claude owned `ingestion/load/` and `embed/cache.py`; it completed the disk cache, Qdrant port/adapter, idempotent UUID5 upsert, payload indexes and corpus-SHA manifest. Its real local Qdrant scale check loaded all 15,066 chunk records twice with 1,024-dimensional deterministic pseudo-vectors and held the point count at 15,066. Those vectors are not embeddings and establish no retrieval-quality claim. No Bedrock call, IAM change, or cloud spend occurred. Built the first runnable `apps/ai-service` boundary: FastAPI `/health` and `POST /v1/rag/query`, a Qdrant retriever filtered by resolved `drug_id`, a PostgreSQL trace repository plus migration, structured human/non-human scope and fact/recommendation intent gates, parent hydration, quarantine handling, and an extractive answer layer. The answer layer refuses evidence that has only a physical page; citations expose only the printed folio, chunk id and optional source crop. Quarantined tables/formulas return a PDF-verification warning and never auto-extract numeric content. Fixed the missing provenance at its source. Chunk schema is now v3 and `cli chunk` reads the real folio map from the 1,668-page PDF. It refuses a monograph whose physical range cannot be mapped, and `chunk-ready` has a new `chunk_without_printed_page_range` gate. Regenerated scope: 684 monographs, 15,066 chunks (14,915 prose + 151 descriptors), zero oversized, and 15,066/15,066 records with a two-value printed-page range. New artifact SHA: `e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38`. Verification actually run: - `python -m pytest -q` and Ruff over `ingestion/`: **258 passed**, lint clean; - `python -m ingestion.cli chunk-ready`: every gate passed, including printed page range 0/0 failures; - ai-service with `RUN_INTEGRATION=1`: **22 passed**, including a real chunk round-trip through local Qdrant, PostgreSQL migration/insert/read-back, and a full FastAPI → Qdrant → guarded citation → PostgreSQL trace round-trip; - local Docker services: PostgreSQL 16 and Qdrant 1.18.3 reachable; integration collections were UUID-scoped and removed after tests; - ArgoCD local: namespace, CRD and seven controller pods are running; the existing unrelated `guestbook` lab app is Synced/Healthy with four history entries. This repo's three Application YAML files parse and point to `master`/the Helm chart, but they are not installed and the chart still has no workload templates, so project sync/rollback was not performed. Still open: no real embedding exists, no full canonical Qdrant collection can serve semantic search, `population_tags` are absent, no clinician-authored release-gate cases exist, and the API currently has no production answer/query embedding provider. The local hashing provider is explicitly plumbing-only. ## 2026-08-04 — Load stage built and proven against a real Qdrant; bbox rounding found `ingestion/load/` was a 0-byte `__init__.py`. It now holds the vector-store boundary: a `VectorStore` port, an `InMemoryVectorStore` that is the reference implementation of its contract, and `QdrantVectorStore` as the only module that names `qdrant_client` — imported lazily, the same arrangement that confines boto3 to `bedrock_runtime`. `embed/cache.py` was added alongside it. Three design decisions are worth carrying forward. The cache key is `(model_id, input_kind, text_sha256)`, not `chunk_id` as §4.A of the delivery plan proposed. Measured reason: `chunks.jsonl` holds 15,066 records but only **14,869 distinct texts**, so 197 records (1.31%) are repeats that a chunk-keyed cache would pay for twice. The content key also cannot serve a stale vector after an edit — a changed text is a changed digest, so it is a miss. Point ids are `uuid5(chunk_id)`. A random id would make a re-run append a second copy of a dose and nothing would report an error. The corpus manifest lives in a `__manifest` sidecar collection rather than a reserved point inside the data collection, because `qdrant_point_count != chunk_count` is a v1 gate and a gate needing an "except the manifest" footnote will eventually be read wrong. **Whole-corpus check against a real server.** A local Qdrant **1.18.3** was started from `infra/docker/docker-compose.yml` (local container, no cloud) and all 15,066 real chunk records were loaded with deterministic pseudo-vectors at 1,024 dimensions — a check of the loading mechanism, **not embeddings, which still do not exist**. Corpus sha256 `30d5154273e0959a…`. First load: 15,066 points in 59 batches, 14.0s, point-count gate PASS. Second load: still 15,066, so idempotency holds at real scale, not only against the fake store. **That sha is already stale, which is the point.** `chunks.jsonl` was regenerated at 09:53 the same day — `chunker.py` changed two minutes earlier and every chunk gained `printed_page_range`, 18,229,918 → 18,753,003 bytes, sha now `e474c83790b450d3…`. Re-measured on the new artifact: still **15,066 chunks, 0 over the 800-token ceiling** (largest exactly 800), all 15,066 carrying `printed_page_range`, 14,915 prose + 151 block descriptors, 197 duplicate texts (1.31%) unchanged because only a field was added. Suite **258 passed**. Had the old corpus been embedded and loaded, then the new one loaded into the same collection, two generations would have mixed with no error at query time — A6 is what refuses that, and it now has a real instance rather than a hypothetical one. **A sampled check passed and was wrong.** Comparing 5 payloads gave 5/5 identical. Scrolling the entire collection instead found **86 of 15,066 chunks** whose payload did not equal its source record. Classifying every differing leaf: **96 differences, all floats, all inside `attachments[].bbox`, maximum absolute delta 5.684e-14**, and **zero** non-float differences — every text, id, page number, page range, token count and boolean round-tripped exactly. A PDF point is 1/72 inch, so that delta cannot move a rendered crop. It is pinned by a regression test that fails if the loss reaches another field or grows past 1e-9. The layer responsible was isolated rather than assumed: the source `chunks.jsonl` returns the value exactly, our own `json.dumps`/`loads` returns it exactly, and **Qdrant reached over raw HTTP with no SDK involved** returns it one ULP low. Nothing needs re-chunking — a regenerated corpus would carry the identical value and be rounded identically. Qdrant also stores dense vectors as float32, so precision beyond f32 is discarded at load regardless. Cache format was decided on measurements, not preference: 300 real chunk texts at 1,024 dimensions cost **21,098 bytes/record — ~318 MB per model** for the corpus, with a **7.8s** offset-index rebuild per open. float32 `.npy` (62 MB) and base64 float32 in JSONL (~87 MB) were measured and set aside; append-only JSONL survives an interrupted run and stays readable, which outweighs disk at one or two models. Revisit at three (~950 MB). It lands in `ingestion/data/processed/`, already excluded by `.gitignore:34`. **A gap in this work, found and closed the same day.** Payload indexes were created on `drug_id`, `section_key`, `atc_codes` and `chunk_kind` and reported as done — but `VectorStore` had no query method, so all that was really proven is that `create_payload_index` returns without raising. Filtered retrieval is the whole of mode A. `find_by_payload` now exists on the port and both stores, as a `scroll` rather than a `search`: it returns **every** match, never a top-k, because "return the whole section" is the plan's non-negotiable — two of five contraindications reads as a complete list. Verified on a real server: all five parts returned with no leak from the PANTOPRAZOL/OMEPRAZOL pair that measures cosine 1.000 on contraindications; a deliberately 300-part section (above the 256 scroll page) comes back whole so paging cannot truncate; and a real multi-part section from `chunks.jsonl` round-trips to exactly its own chunk ids. Tests: **255 passed** with Qdrant running (206 before this work, +49); **247 passed, 8 skipped** with it stopped, so an offline machine and CI see skips rather than failures. After the mode A work and the other worktree's `cli.py` fix the suite stands at **268 passed** and `ruff --select F,E9,B,ARG` reports **no findings at all** across `ingestion/`. Still missing, and deliberately so: `printed_page_range` and `population_tags` are not in the payload (open questions to Codex in `coordination/CLAUDE_TASK_2026-08-04.md`); `cli embed` / `cli load` are not wired because `cli.py` is Codex's; and **no real embedding vector has ever been produced** — every vector the load path has carried was synthetic. The Bedrock request shapes remain documentation-derived and unproven. Measured cost: **$0**. No Bedrock call, no IAM change, no cloud resource. ## 2026-08-03 — Bedrock embedding boundary built; IAM diagnosed, not yet opened `ingestion/embed/` was an empty `__init__.py`. It now holds the provider boundary the model benchmark needs: an `EmbeddingProvider` ABC that owns input validation, request-size batching and timing, and three adapters behind it — `amazon.titan-embed-text-v2:0`, `cohere.embed-v4:0`, and `BAAI/bge-m3` as the zero-cost local control. boto3 is named in exactly one module and imported lazily, so the package imports and the whole suite runs with no AWS account. Two design points are worth carrying forward. `input_kind` is a required argument, not a keyword: Cohere embeds corpus records and queries into different subspaces, and sending `search_document` for a query raises no error — recall just drops. And `normalized` is three-valued. Titan is asked to normalize and says so; the Bedrock docs never state whether Cohere's float vectors are unit-length, so that field stays `None` instead of guessing, and `embed.probe` prints a *measured* L2 norm to settle it on the first live call. The AWS side is diagnosed and stuck. `ai-lab-user` has no inline and no attached user policy; its one group (`AI-Lab-Group`) grants EC2, IAM, ELB and VPC full access and nothing else. There is no `bedrock:*` grant anywhere on the identity — confirmed by running both `list-foundation-models` and `invoke-model` and reading the two `AccessDeniedException` messages. Two least-privilege policies are drafted in `infra/aws/iam/` but **deliberately not applied**: that identity carries `IAMFullAccess` and could attach them itself, which is exactly why it was left to a human. Consequence: every request-body shape in the two Bedrock adapters is derived from the AWS user guide (read today) and **has never been accepted by the service**. That is unproven, not verified. Tests: 22 new, all with a stub invoker and zero network; **203 passed** overall, up from 181. Lint clean on every file added (`--select F,E9,B,ARG`); the one remaining finding is a pre-existing `cli.py` import owned by the other worktree. Measured cost so far: **$0**. Nothing was embedded, nothing reached Qdrant. ## 2026-08-03 — Exact hard-10 gate and all-block table chunking experiment Extended the isolated table/formula sandbox beyond the 100-page sample. An exact ten-block risk gate covered four cross-page pairs, a merged header, a fragmented fraction bar, and the bar-less ADENOSIN formula; all ten source crops were visually checked. The full run then processed all 151 canonical blocks: 141 physical tables, ten formulas, 133 logical table parents, 669 row children, and seven cross-page logical tables. Full-scope visual inspection exposed a continuation bug: FAMCICLOVIR p647 and INSULIN p811 repeat their column headers, while other continuation pages start directly with data. The linker now distinguishes these cases; repeated headers are not emitted as data, and INSULIN's changed `Phối hợp` first-column meaning is preserved. Both branches have regressions. The expanded, source-derived retrieval suite contains 2,436 cases. With drug and table/formula lane resolved before ranking, deterministic hybrid character TF-IDF measured 94.42% Recall@1, 99.79% Recall@5, and 96.90% MRR. Row questions were 94.82% / 100%; formula questions 100% / 100%. Five ambiguous whole-table questions fell below top five because the same drug owns several near-identical tables; production must clarify or route using an additional table anchor. Neural MiniLM is now opt-in and excluded from the default parsing gate. Measured chunk design: table-parent tokens min/median/p90/p95/max = 66/188/441/678/1,893; only four of 133 parents exceed 800. Row children are 75-token median, 172 p95, 471 max. Keep every logical parent intact, index both parent and header-aware rows, never split a row, and hydrate row hits to the complete parent/source pages. Final checks: **181 tests passed**, readiness 20/20, lint clean. --- ## 2026-08-03 — 100-page table/formula reconstruction and RAG sandbox Built an isolated experiment under `ingestion/scratch/rag-table-pilot` without writing sandbox representations into the canonical corpus. The risk-stratified 100-page run reconstructed 120 tables and 10 formula regions, rendered and manually inspected all 130 crops, and linked four tables continued across page pairs 132-133, 646-647, 825-826, and 1373-1374. The retrieval router fixes the drug and data lane before vector ranking. On 461 source-derived queries, hybrid row+whole character TF-IDF reached 92.62% Recall@1, 98.70% Recall@5, and 95.04% MRR. Cached English-oriented MiniLM was worse (88.29% / 97.18% / 91.92%). Eighteen row-hit answer previews all hydrated to the complete parent Markdown table; eight included both pages of a continued table. A narrow deterministic interval probe passed 172/172 generated cases; this is a mechanics check, not clinical ground truth. Visual review exposed one canonical defect: ADENOSIN p147's bar-less printed formula region ended after its numerator and omitted `Nồng độ adenosin (3 mg/ml).` The bar-less band now extends 31pt below its synthetic anchor, capturing the denominator but stopping before `Ví dụ:`; a regression pins that boundary. Canonical artifacts were regenerated after the fix: 684 monographs, 11,974 sections, 15,066 chunks, 151 descriptors, 0 unassigned spans, all 20 readiness gates passing, **180 tests passed**, and lint clean. Decision: JSON grid + Markdown answer view, row and whole-table retrieval, and mandatory parent hydration are viable for the next stage. This remains a retrieval experiment, not production clinical approval; merged-cell semantics, unit/multi-axis reasoning, Vietnamese embedding comparison, borderless/bar-less recall, clinician-authored evals, and final expert review remain open. --- ## 2026-08-03 — Whole-corpus parser repair after manual baseline audit Implemented and re-ran the parser over all 1,668 pages after manually reading the high-risk baseline outliers. The fixes are structural, with regressions: - restored the missing `THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN` boundary (`Tên chung quốc tế và mã ATC` is its real first anchor), separating pages 1371–1373 from `THUỐC PHIỆN - OPIAT - OPIOID`; - require both physical and inferred printed page bounds, so back-index page 1655 can no longer extend ZOLPIDEM's real `[1492, 1494]` range; - keep plain label-shaped text as body when it is an adjacent wrapped continuation in the same PDF block (including NADROPARIN's “không phải là chống chỉ định”); - classify known table cells before headings, putting WARFARIN and IOBITRIDOL dosing tables back under `lieu_luong_va_cach_dung`; - added confirmed heading variants for CLORPHENIRAMIN dosage forms and tetanus toxoid dosing, and real provenance for combined inline fields; - made verified formula bands column-aware: NETILMICIN opposite-column prose is retained while AMPICILIN's gutter-adjacent formula stays quarantined; - visually inspected all **151/151 unique table/formula regions** against the rendered PDF; every region is genuinely 2D and remains quarantined; - emit every physical table/formula region atomically at its first stream occurrence, fixing split/contradictory ownership on CAPECITABIN, IMATINIB, CARBOPLATIN, NETILMICIN, and TRASTUZUMAB; - route explicit `Bảng N. Điều chỉnh liều ...` appendices back to dosage even when the book prints them after `Tên thương mại` (CAPECITABIN p309); - added readiness gates for every individual section part's source-span IDs and duplicate physical-region IDs. Final regenerated artifacts and evidence: | check | result | |---|---| | tests | **180 passed**; lint clean | | segmentation | **684 monographs**, 11,974 sections, 8,213,036 prose chars | | back-index validation | **96.2% recall (678/705), 99.1% precision** | | quarantined regions | **151 blocks / 151 unique IDs**, all visually checked | | chunks | **15,066** (14,915 prose + 151 block descriptors), 0 over 800 tokens | | chunk readiness | **20/20 PASS** (including duplicate-region prevention) | | coverage | 252,799 spans, **0 unassigned** across all 1,668 pages | | residual ink | 3,931 classified regions, **0 unclassified** across all pages | Canonical `ingestion/data/processed/{monographs,chunks,coverage_ledger}` were regenerated. Remaining limits: no whole-document human-reviewed clinical ground truth, no row/column reconstruction for quarantined tables, and unknown recall for borderless tables/bar-less formulas. This is ready for retrieval experiments, not a claim of production clinical approval. --- Chronological record of work done on this project, newest entry on top. The goal is continuity across sessions: if a work session ends unexpectedly (context/token limit, interruption), whoever picks this up next — human or Claude — should be able to read the latest entry and know exactly what's done and what's next, without having to reconstruct it from git history. **Convention**: add a new entry at the top before ending a session whenever meaningful progress was made, and proactively the moment it looks like the session might run out of context/tokens mid-task — don't wait until the very end if that risk is showing. --- ## 2026-08-03 — Independent re-verification, a redundant rule in my own uncommitted fix, a provenance defect, and the measurements a retrieval design has to be built on No code was changed in this session: the four files from the previous round are still uncommitted and under external review (Codex). Everything below is measurement, and the numbers live nowhere else — the investigation scripts were deleted per the repo rule, so this entry is the record. ### 1. Re-verified the whole tree from scratch | check | command | result | |---|---|---| | tests | `python -m pytest -q` | **164 passed**, 39.31s | | lint | `ruff check --select F,E9,B,ARG .` | clean | | gates | `cli chunk-ready` | **18/18 PASS**; 683 monographs, 11,966 sections, 8,212,880 chars, 167 quarantined blocks | | recall/precision | `cli validate --pdf …` | 683 detected, 705 ground truth, **96.0% (677/705) / 99.1%** | | span ledger | `cli coverage --pdf …` (re-run) | 252,799 spans after merge, 9,398,772 chars, **unassigned 0** | | reproducibility | `cli run` → sha256 | **byte-identical** to `monographs.jsonl` (`84f41d96…`) | | reproducibility | `cli chunk` → sha256 | **byte-identical** to `chunks.jsonl` (`63472db4…`); 15,076 chunks (14,909 prose + 167 descriptors), 0 oversized, 4,072,725 tokens (cl100k_base) | Recall rose 92.9% → 96.0% because of the uncommitted back-index rejoin, and the mechanism is the denominator: 725 → 705 ground-truth entries once wrapped fragments stop counting as entries. The detector did not improve. **Not re-run: `cli residual-ink`.** `residual_ink.json` is dated 2026-08-01 12:03, before the 17:36 assembler edits. Its stored contents (3,931 regions, no `unclassified` kind) are last session's numbers, not this session's. **Doc drift found:** `docs/verification-strategy.md` quotes 252,733 spans / 177,679 `normalized_text` / 12,764 `heading`; measured today 252,799 / 177,754 / 12,752. The `unassigned = 0` conclusion still holds. ### 2. The x0 geometry change in the uncommitted diff is redundant Assembled the whole book four times with the two new rules toggled: | variant | monographs | sections | chars | |---|---|---|---| | current (x0 + italic) | 683 | 11,966 | 8,212,880 | | **old x1 rule + italic** | 683 | 11,966 | **8,212,880** — 0 differences of any kind | | x0, no italic | 683 | 11,966 | 8,212,844 (3 sections differ) | | x1, no italic (= `bc01782`) | 683 | 11,966 | 8,212,780 (9 sections differ, 11,014 char delta) | The italic rule alone recovers all 9 sections (CEFAZOLIN dosing 1,062 → 5,620 chars; CALCI LACTAT `than_trong` 582 → 1,504, `tuong_tac_thuoc` 2,346 → 1,439). The x0 rule alone recovers 6 of 9 and adds **nothing** on top of the italic rule. Worse, the justification is wrong: the real NEVIRAPIN span on physical page 1045 is `TimesNewRomanPS-ItalicMT` (verified by reading the span's font), so the italic rule is what fixes that page — not the 0.01pt overlap the code comment and the new test's docstring credit. The test itself is valid but pins the geometric rule only, because the `_span()` fixture helper never produces an italic font. **Either keep the x0 rule as defence-in-depth with an honest comment, or revert it — but the current comment overstates it.** ### 3. `source_page_range` is wrong for 13 of 683 monographs Section-level provenance (`parts`) is correct everywhere; the monograph-level page range is not. 12 monographs overshoot by +1 page; **ZOLPIDEM declares `[1492, 1655]` while every one of its sections comes from 1492-1494** — a 164-page claim reaching into the back index. Root cause for ZOLPIDEM, confirmed: physical page 1655 (printed 1656, back matter) carries a **bold** span reading exactly `Tương tác thuốc`, which `_classify` emits as a `_SectionEvent`, and the `_SectionEvent` branch at `segment/assembler.py:496` updates `source_page_range[1]` with **no `in_monograph_range` guard** — unlike the `_TextEvent` branch at line 511. Verified that **0 spans past physical 1495 pass `in_monograph_range`**, so no text was contaminated and `empty_section` is still 0. The defect is confined to one provenance field. The +1 cause is **not isolated** — it is not lifted tables (all 12 have `tables: []`); the likely candidate is a next-page boilerplate span bumping the range before being excluded, but that was not measured. ### 4. Corpus profile — what a retrieval design actually has to work with - **13 of 19 fields have p90 < 1,500 chars**, i.e. the whole section fits one chunk. Only four routinely need splitting: `duoc_ly` (p90 4,939, max 14,099), `lieu_luong` (4,873 / 14,197), `than_trong` (2,419), `tuong_tac` (2,147). Confirms ADR 0004 on the cleaned corpus. - **ATC**: 668/683 (97.8%) carry ≥1 code, **171 (25.0%) carry more than one**, max 20, 1,043 distinct codes. - **`ten_thuong_mai` present in 492 (72%)** monographs. - **The back index holds 344 `X - xem Y` lines** — brand → generic aliases — which `parse_back_index` currently discards wholesale (correct for validation, but this is the highest-value retrieval asset in the book, because clinicians type brand names). - **401 `xem [thêm] mục/chuyên luận` phrases across 261 monographs**; a chunk containing one is useless retrieved alone. - **Dosing population markers**: `Trẻ em` 53%, `Người lớn` 51%, `Người cao tuổi` 15%, `Trẻ sơ sinh` 8%, `Suy thận` 8%, `Suy gan` 6% of 682 dosing sections — real sub-section boundaries, better split points than token windows. - **167 quarantined blocks, 129 (77%) inside `lieu_luong_va_cach_dung`** — the most dangerous field is the one the tables were lifted out of. ### 5. Cross-drug confusability — the number that decides the architecture First hypothesis (much repeated boilerplate across drugs) was **refuted**: only **171 of 11,966 sections** share exact text with another drug (1.4%), and the six heavy clinical fields are 100% distinct. Then measured, per field, each drug's TF-IDF cosine against its *nearest other drug*. **This is a lexical proxy, not an embedding measure** — it bounds the problem from one side only. | field | median | p90 | p99 | max | drugs with NN > 0.7 | |---|---|---|---|---|---| | `lieu_luong_va_cach_dung` | 0.314 | 0.455 | 0.631 | 0.836 | 4 (0.6%) | | `tuong_tac_thuoc` | 0.284 | 0.461 | 0.870 | 0.984 | 18 (2.8%) | | `tac_dung_khong_mong_muon` | 0.300 | 0.437 | 0.856 | 1.000 | 13 (1.9%) | | `chi_dinh` | 0.408 | 0.637 | 0.885 | 0.924 | 31 (4.5%) | | `chong_chi_dinh` | 0.346 | 0.633 | 0.898 | **1.000** | 37 (5.4%) | Named pairs: `PANTOPRAZOL ↔ OMEPRAZOL` (contraindications **1.000**, indications 0.913) · `BENZATHIN PENICILIN G ↔ PHENOXYMETHYLPENICILIN` (contraindications **1.000**) · `DIGOXIN ↔ DIGITOXIN` (0.891 / 0.911) · `NATRI NITRIT ↔ NATRI THIOSULFAT` (dosing 0.631 — two different steps of the same cyanide-antidote protocol) · `IOBITRIDOL ↔ ACID IOXAGLIC` (0.984) · `ESTRIOL ↔ ESTRON` · `GLICLAZID ↔ GLIMEPIRID` · `NAPHAZOLIN ↔ OXYMETAZOLIN`. Name layer: **19 drug names are a substring of another drug name** (`CLOROTHIAZID` in `HYDROCLOROTHIAZID`, `EPHEDRIN` in `PSEUDOEPHEDRIN`, `LORATADIN` in `DESLORATADIN`, `ATROPIN` in `HOMATROPIN HYDROBROMID` — all genuinely different drugs), and 106 of 683 names share a 6-character prefix across 38 clusters. **Conclusion drawn from this, for the retrieval design: vector similarity must never be allowed to choose the *drug* — only which passage within an already-resolved drug.** The dangerous confusions are concentrated in a small, enumerable set of same-class pairs, which is exactly the population this project's verification strategy says to census rather than sample. ### Not done yet / next up Sequenced in **`docs/v1-delivery-plan.md`** (written this session): a two-week plan to a running v1, scoped down to two deployables (`web` + `ai-service`) because the four NestJS services measure 0 `.ts` files each. The items below are the ones that plan depends on. - The confusable-pair census must become a **committed fixture produced by production code** (`ingestion/validation/`), not a deleted scratch script. Until then these numbers are only in this entry. - ADR 0007 (retrieval architecture) not written. Proposed content: vectors never pick the drug; the unit returned to the LLM is the **complete section** (enabled by `section_not_reassemblable_from_chunks = 0`, because a partial contraindication list reads as "no contraindication"); and eval split in two — **routing** correctness (ground truth derivable from the corpus itself, 683 × 19 pairs, no human needed) versus **content** correctness (requires a clinician; cannot be self-generated without fabricating evidence). - Entity/alias layer (683 canonical names + 344 back-index aliases + 492 `ten_thuong_mai` + 1,043 ATC codes) — zero-regret, needed by every architecture, must use longest-exact-match because of the 19 substring traps. - `residual-ink` re-run; `verification-strategy.md` numbers re-synced; regression test for `parse_back_index` (still has none); the `source_page_range` guard; the x0-rule comment decision. - Open question for the user, not a technical one: this is the **2018 edition**; the 3rd edition (2022) exists. For a document with legal force over prescribing, staying on 2018 should be a deliberate decision, and it makes edition-independence a real requirement for the pipeline. - Still untouched: `embed/`, `load/`, Qdrant, `ai-service`, and the general chapters (printed 37-98) and appendices (printed 1497-1528), which remain outside the corpus entirely. ## 2026-08-01 (cont'd, 7) — "still errors?" — yes: two more real content-loss bugs, both in dosing sections Asked whether errors remained after the previous round, the honest answer was that this session has found real defects every time it looked one level deeper. It looked again, and found two more. **1. Chunks ended on a bare population label, with the dose in the next chunk.** `split_sentences` treats `:` as a sentence boundary and `_OPENS_SENTENCE` accepts a digit, so `"Người lớn: 500 mg mỗi 8 giờ."` splits after the colon. When the packer flushed at that point, the chunk ended on the label. Measured: **38 prose chunks**, e.g. AMOXICILIN's ending on a Lyme indication followed by a bare `Người lớn:`. Retrieval on that chunk returns a population with no dose; on the next, a dose with no population. Outlier item 17 counted population markers on 1,121 of ~1,400 monograph pages, so this is the common shape, not an edge case. The packer now carries trailing label atoms into the next part instead of flushing on them: **38 → 2**, and chunks ending on any colon **721 → 19**. **2. A section name printed mid-line was swallowed as a heading — real text loss, in dosing sections.** Chasing the last 2 of those 38 showed the defect was not in chunking at all. CISPLATIN (physical page 402) prints `Suy thận: Chống chỉ định.` inside `liều lượng và cách dùng`; the second half is itself a section name, so it was matched as a heading. The result: the renal-impairment contraindication **disappeared from the dosing text** and the section ended on a bare `Suy thận:`. ISOPRENALIN had the same shape. Same family as the FLUOROURACIL bug fixed earlier today, but that rule only covered a label directly *under* a heading and could not see this one. Fixed geometrically: a real section heading opens its line, so a non-bold section name with another span printed to its left is body text. "To the left" is checked properly — same page/block/line *and* `previous.x1 <= span.x0` — because the synthetic test fixtures place every span at identical coordinates, and a looser check passed on real data while breaking the AMITRIPTYLIN inline-value case. Verified after the fix: CISPLATIN's dosing section contains `Suy thận: Chống chỉ định.` again, ISOPRENALIN's `Trẻ em:` is followed by its doses, and `chong_chi_dinh` is no longer polluted. Monograph and section counts unchanged at 683 / 11,966 — nothing was traded away for the recovery. **State:** 18/18 gates pass, **163 tests** (was 161), ruff F/E9/B/ARG clean, 15,077 chunks with 0 over the ceiling, 8,212,780 section characters. **Standing conclusion, worth writing down:** every round of "is it clean now?" this session has ended with real defects found — five in the previous round, two in this one, and four of the previous five were in code written the same day. The gates and tests prove what those instruments can see. They do not prove the corpus is correct, and the largest unmeasured area is unchanged: content accuracy against the source, with no human-reviewed ground truth for 8.2M characters. ## 2026-08-01 (cont'd, 6) — Bug hunt after declaring "clean": the token count was wrong by 2x, 14.7% of chunks were over the ceiling, and two stage boundaries measured different pipelines I had just reported the tree as clean. It was not. Going looking properly found five real defects, four of them in code written earlier the same day. **1. `estimate_tokens` was wrong by a factor of two, and the number it produced was reported.** ADR 0004 sized chunks with `len(text) // 4`, described honestly as an estimate. Measured against `cl100k_base` on the real corpus: | | | |---|---| | estimate (chars/4) | 2,115,427 tokens | | real tokenizer | **4,093,440 tokens** | | real/estimate | median **1.95**, p95 2.50, max **6.0** | | oversized by estimate | **0** | | oversized in fact | **1,884 of 12,838 = 14.7%**, largest 1,645 tokens | Vietnamese diacritics cost several byte-pair tokens each. "0 oversized" was reassuring and false. `chunk/tokens.py` now counts with the real tokenizer, injected so the chunking logic stays testable without it, with a fallback of chars/2 that errs small rather than large. **2. The packer could exceed the ceiling on its own.** Two causes, both measured on VORICONAZOL's `tương tác thuốc`: an atom of 710 tokens was left whole because it was under the 800 ceiling, and the overlap builder added whole atoms until the running total *passed* the budget, so a 251-token atom produced a 273-token overlap against a 65-token setting. 273 + 710 = 983. Atoms are now split against the 650 target, leaving room for overlap, and the overlap stops *before* exceeding its budget. **3. An over-long comma list was left as one atom.** VORICONAZOL's interaction list is one "sentence" hundreds of drug names long. Truncated by an embedding model it reads as "this drug is not listed" — a false negative in the direction that matters. Split at commas, which is lossless for a list. After 1-3: **0 chunks over the ceiling**, verified by an independent tiktoken re-count of the written file, not by the pipeline's own number. 15,049 chunks (was 12,838 — the rise is real sub-chunking that should have happened all along). **4. `cli validate` measured a different pipeline than `cli run`.** It used the raw span stream (no transcription repair) and passed no table regions, so recall/precision described a build that is not the one producing the output — the same class of mismatch already fixed for `coverage`. Now shares `_extracted_and_repaired_spans` and `_region_index`. Result after the fix is unchanged at 92.9% / 99.1%. **5. `chunk/io.py` dropped `SectionPart` when reading monographs back**, so per-part provenance died at the stage boundary — against CLAUDE.md's explicit rule. Now carried: 12,290 parts across 11,966 sections. **Two new gates, and the gate itself was wrong twice before the data was.** `section_not_reassemblable_from_chunks` rebuilds each section from its own chunks by removing the deliberate overlap and compares. First version joined chunk texts with a newline and reported **734** sections missing — the first one it named was present. Second version probed a 60-character head and reported **1**, NAPROXEN, where the probe straddled an overlap seam that legitimately repeats text. The working version compares with whitespace removed, because each split seam loses exactly one space to `.strip()` (measured on ABACAVIR: two single spaces in a 4,232-character section, nothing else). It proves no character of content is lost, reordered or duplicated beyond the intended overlap. **0.** **Also fixed:** all 8 real lint findings (`ruff --select F,E9,B,ARG`) — five unused imports and three `zip()` calls without explicit `strict=`. The zips were the adjacent-pair idiom and not bugs; `strict=False` now says so. And the transcription splice could leave a fragment holding only a space, which showed up as two `whitespace_only` spans; dropped, and proven inert by the sha256 over every section's text being byte-identical before and after (`6af13301…`). **State after the hunt:** 18/18 gates pass (10 corpus + 8 chunk), 161 tests (was 158), `ruff F/E9/B/ARG` clean, `unassigned = 0`, `cli validate` 92.9% / 99.1%, 15,049 chunks with 0 over the ceiling. ## 2026-08-01 (cont'd, 5) — ADR 0006 implemented: chunks now reference their lifted blocks; `chunk/` runs for the first time; 16/16 gates green **Why this was needed, in one line**: a chunk of a section whose table had been lifted was grammatical, complete-looking prose with the table absent and nothing marking the absence — silent incompleteness, in the section where 127 of 167 lifted blocks live (`liều lượng và cách dùng`, 76%). **Design is in `docs/adr/0006-quarantined-block-references-in-chunks.md`**, written before any code. It resolves the item ADR 0005 explicitly deferred. **Implemented:** `ChunkAttachment` (block_id, kind, shape, physical_page, bbox, quarantined, header_row) on every prose chunk, plus one `block_descriptor` chunk per block whose text is built **only** from metadata. `chunk/io.py` now reads `tables` (it silently dropped them before) and writes `schema_version: 2`. **`chunk/` executed for the first time**, whole corpus: | | | |---|---| | chunks | **12,838** — 12,671 prose + 167 descriptors | | prose chunks carrying a lifted block | 185 | | oversized (>800-token ceiling) | **0** | | estimated tokens (chars/4, an estimate) | 2,115,427 | **The condition this work was accepted under — prose chunks must not change — was measured, not asserted.** Built the corpus both ways and diffed: | check | result | |---|---| | prose chunk count, both ways | 12,671 / 12,671 | | chunk id sets identical | yes | | `prose_text_changed` | **0** | | `prose_nonattachment_field_changed` | **0** | Only the two new fields differ. The change is strictly additive. **A gate caught a real defect in my own design within minutes of existing.** `block_text_leaked_into_chunk_text` fired on AMIODARON (physical page 183): pdfplumber reported that table's first row as `"Thời gian liệu pháp tĩnh mạch Liều 720 mg/ngày (0,5 mg/phút)"` — **a dose, inside what it called a header**, from an extraction never verified by eye, being embedded as retrieval text. Measured across the corpus: **42 of 124 simple-table headers (34%) contain a digit.** Rule added: a header row is embedded only when no cell contains a digit and every cell is short enough to be a label. 76 of 167 descriptors (46%) keep a header under that rule; the AMIODARON one does not. A label with no digit cannot be mistaken for a dose. **Full gate suite, 16/16 pass** — 10 corpus gates plus 6 ADR 0006 gates (`section_block_without_chunk_reference`, `attachment_block_id_unknown`, `attachment_without_page_or_bbox`, `block_text_leaked_into_chunk_text`, `descriptor_chunk_without_attachment`, `descriptor_count_vs_block_count` = 167/167). Tests: **158 passing** (148 → 158). `chunk/` had no tests at all before this entry; it now has 10, including the prose-unchanged invariant and the numeric-header refusal. **Binding on `ai-service`, stated in ADR 0006 and not implemented here:** a chunk with `has_quarantined_content` must make the answer say a table or formula exists at the cited page and surface its crop; a `block_descriptor` may be answered only with the crop; no chunk carrying a quarantined attachment may be used to state a numeric dose. **Still open:** table row/column reconstruction (the opendataloader cell data is available and matches pdfplumber exactly inside the monograph range); recall for borderless tables and bar-less formulas; content accuracy against the source; the general chapters and appendices (9.6% of characters). ## 2026-08-01 (cont'd, 4) — READY TO CHUNK: transcriptions merged back into the text, `cli chunk-ready` gate suite green on all 10 gates, two more real data-loss bugs found and fixed on the way **The blocker is closed.** The 1,116 transcribed characters are no longer a file beside the corpus — they are in it. `ingestion/extract/repair.py` splices each transcribed run back into the span stream geometrically, and every command that builds monographs now goes through the same repaired stream, so the ledger and the output describe one pipeline rather than two. **New gate suite, `cli chunk-ready`** (`ingestion/validation/readiness.py`). Each invariant gets its own count and its own target — a single verdict would hide exactly what took this session to find. Run on the whole corpus: | gate | count | target | |---|---|---| | outlined_run_not_merged | 0 | 0 | | known_corruption_string | 0 | 0 | | formula_fragment_in_prose | 0 | 0 | | pua_char | 0 | 0 | | replacement_char_ufffd | 0 | 0 | | empty_section | 0 | 0 | | section_without_provenance | 0 | 0 | | unflagged_quarantine_block | 0 | 0 | | duplicate_drug_id | 0 | 0 | | monograph_without_page_range | 0 | 0 | Corpus going into chunking: **683 monographs, 11,966 sections, 8,212,712 characters**, plus 167 quarantined table/formula blocks held outside prose. **Two real bugs surfaced by building the gates, both fixed:** 1. **A 4pt glyph in the column-overlap strip was assigned the wrong column.** `classify_column`'s two tolerance bands overlap between x=288 and x=319 and left was tested first, so a single `ổ` at x=315 on physical page 714 was classified as left-column and could not be matched to its own right-column line. `Độ ổn định` stayed `Độ n định` even after the repair ran. Fixed by testing exact containment before tolerance. Invisible for a full-width block; only a narrow box exposes it. 2. **A plain body line that repeats a section name was read as a heading.** FLUOROURACIL (physical page 681), verified by rendering the page, prints `Thời kỳ mang thai` / `Chống chỉ định.` and `Thời kỳ cho con bú` / `Chống chỉ định.`. Both body lines matched the section vocabulary, so both sections came out **empty** and the statement that fluorouracil is contraindicated in pregnancy and while breastfeeding was dropped entirely. Fixed narrowly: a *non-bold* label directly under a heading is that heading's body. Boldness still cannot be required in general (outlier item 20), hence the position constraint rather than a style rule. A third placement bug was caught during the merge itself: PyMuPDF emits the text either side of a dropped glyph as **one span whose box spans the gap**, so splicing at span boundaries produced `tuở ổi`. `repair.py` now reads per-character boxes from `rawdict` and splits the containing span at the character offset the geometry indicates. **Whole-document re-measurement after all of the above:** | check | result | |---|---| | `cli run` | 683 monographs, 51 runs merged (1,116 chars), 167 blocks lifted / 167 quarantined | | `cli validate` | 92.9% recall / 99.1% precision — unchanged | | `cli coverage` | 252,801 spans, **unassigned = 0** | | `cli chunk-ready` | 10/10 gates pass | | tests | **148 passing** (145 → 148) | **What these gates explicitly do NOT prove**, printed by the command itself so it cannot be quoted out of context: content accuracy against the source (no whole-document human-reviewed ground truth exists), table row/column reconstruction, and recall for borderless tables and bar-less formulas. **Next:** `chunk/` still has no tests and has never been executed. Table reconstruction from the opendataloader cell data remains available and is not on the critical path. ## 2026-08-01 (cont'd, 3) — All 23 fraction-bar candidates read by eye (precision 69.6%), 51 outlined runs transcribed, 2D formulas quarantined; prose-leak gate = 0 **All 23 `fraction_bar_candidate` regions were rendered and read.** Verdicts, one page at a time: | verdict | count | where | |---|---|---| | real 2D formula | **16** | p43, p92 (×5), p202, p325 (×2), p349, p1042, p1043 (×2), p1132, p1402 (×2) | | not a formula | **7** | p4 (×3 decorative underlines on the Ministry decision page), p63 (ruled box), p845, p878 (table cell borders), p1667 (rule above the colophon) | **Precision of the candidate rule: 16/23 = 69.6%.** That is why the verified list is a curated file (`ingestion/data/verified/formula_regions_2d.json`) and not the detector's raw output — a 70%-precise rule must not quarantine content on its own. 10 of the 16 are inside the monograph range. **A formula the detector cannot find, confirmed.** ADENOSIN (physical page 147) prints `Tốc độ truyền dịch (ml/phút) = 0,140 (mg/kg/phút) × trọng lượng cơ thể (kg) / Nồng độ adenosin (3 mg/ml)` as **three plain lines with no fraction bar at all** — verified by rendering the region and reading it. No geometric signal exists to detect it; it surfaced only because a prose-leak gate matched its text. It is quarantined and flagged, and `recall_limit` in the verified file records that **the number of bar-less formulas in the book is UNMEASURED**. The fraction-bar scan must never be described as complete formula coverage. **51 outlined runs transcribed** into `ingestion/data/verified/outlined_text_transcriptions.json` — 22 full lines plus 29 single glyphs, **1,116 characters** recovered, each with page, bbox, the run's text and the extracted line it belongs to. Every value there is a transcription read off a rendered page, labelled as such, never extracted data. **The single-glyph runs are the nastier half of that defect.** They are Vietnamese diacritic characters dropped out of lines that otherwise extract fine, so the damage is invisible downstream: | extracted | actual | |---|---| | `Độ n định:` | Độ **ổ**n định | | `≥ 1 tu i` | ≥ 1 tu**ổ**i | | `Thuốc dùng tại ch :` | tại ch**ỗ** | | `i nồng độ glucose máu` | (thay đ)**ổ**i nồng độ glucose máu | **2D formulas are now quarantined in the pipeline.** `SHAPE_FORMULA_2D` was added to the existing shape taxonomy and to `QUARANTINE_SHAPES` — an entry, not an edit to matching code. `ingestion/extract/formulas.py` loads the verified regions and grows each bar into a band covering numerator and denominator. Whole-book re-run: | gate | result | |---|---| | verified formula regions loaded | 17 on 10 pages | | blocks lifted out of prose | 169, **169 quarantined** | | `formula_2d` blocks | 14 | | `formula_fragment_left_in_prose` | **0** | | monographs | 683 (unchanged) | | `cli validate` | 92.9% recall / 99.1% precision (unchanged) | | tests | **145 passing** (139 → 145) | The side margin needed two attempts: at 4pt, AMPICILIN VÀ SULBACTAM's numerator `Thể trọng (kg)` stayed behind in the prose because its span box carries leading spaces that pull its centre left of the bar. Raised to 95pt with the reasoning recorded in the module: over-capturing a neighbouring line into a quarantined block is recoverable, half a formula left in prose is not. **Still open**: the 1,116 transcribed characters are recorded but **not yet merged back into the monograph text** — the corpus still contains `Độ n định`; table row/column reconstruction is untouched (137 simple tables + 17 multi-header + 1 continuation remain quarantined); table detection recall for borderless tables is unmeasured; `chunk/` still has no tests and has never run. ## 2026-08-01 (cont'd, 2) — Residual-ink check built and run whole-document; found a text-loss class no text-based check could see: 51 runs of type drawn as vector paths on 5 pages **What was built.** `ingestion/validation/residual_ink.py` (production, plus a `cli residual-ink` command) renders each page, whites out every pixel covered by an extracted span, and reports the ink that survives. It needs no ground truth and no sampling. Measured: **0.06 s/page, all 1668 pages in under two minutes.** Classification is a pure function over `(region, PageContext)` with an ordered rule list, so a new kind of residual is a new entry, not an edit. **Whole-document gate result — all 1668 pages, 3,931 residual regions:** | kind | regions | |---|---| | header_rule | 1,649 | | text_as_vector_outline | 1,061 | | table_frame | 959 | | antialias_speck | 220 | | fraction_bar_candidate | 31 | | rule_fragment | 10 | | header_band_fragment | 1 | | **unclassified** | **0** | **The finding: 51 runs of text on 5 pages exist only as vector outlines.** Physical page 714 (GATIFLOXACIN) prints 17 full lines of ordinary prose that `page.get_text()` does not return, `page.search_for()` cannot find, `pdfplumber` does not return and `opendataloader-pdf` does not return. `page.get_drawings()` shows why: each line is a filled path of 1,126-1,831 items, shaped exactly like one line of type, in the body-text colour. Single glyphs appear the same way with 39-45 items. Recovery cannot be automatic — the paths carry no character codes — so `ingestion/extract/outlined_text.py` detects and reports them for transcription and never guesses. | physical page | outlined runs | |---|---| | 714 | 31 | | 736 | 16 | | 1373 | 1 | | 1444 | 1 | | 1445 | 2 | All five are inside the monograph range. Two independent methods agree on the same five pages: the drawing-shape scan, and counting glyph-shaped leftovers in the residual mask. Sample of what is missing, read off the rendered page: `"Nghiên cứu trên động vật, gatifloxacin gây ngộ độc cho thai."` (p714), `"(Typhoid, inactivated, whole cell), J07AP03 (Typhoid, purified"` (p1445). **Three instrument bugs were found and fixed before any of the above was believed** — the measuring device was wrong before the data was, three times: 1. **Horizontal banding merged the two page columns**, so page 209's ADR table sat in a box whose centre fell in the gutter and matched no table region. Adding a column split then cut single table grids into their individual rules. Replaced with 2D connected components (`scipy.ndimage.label`). 2. **A glyph-count ratio was nearly reported as a data-loss measure.** First pass gave "extraction ratio 0.6656, 835 pages below 98%". It was wrong: `get_texttrace()` counts glyphs painted outside the page rectangle — 4,717,407 of them, on pages that are visually blank. Clipping to the page rect gave 0.8023 and "1642 of 1668 pages below 95%", which was also wrong: Vietnamese diacritics are painted as two glyphs and extracted as one character, so the deficit is systematic and meaningless. **Neither ratio should ever be quoted.** The pixel-based check is the sound one. 3. **Mask padding of 1.0pt ate the fraction bars** it was meant to find. Calibrated to 0.5pt against the two known formulas, verified not to add noise on a 10-page prose sample. Incidentally this explains a long-standing note in ADR 0003: `pdfplumber` "scrambles reading order" on this document because it reads the off-page text that PyMuPDF correctly clips away. Tests: **139 passing** (129 → 139), including whole-document regression fixtures pinning the 51 outlined runs per page and the two fraction-bar widths (188.6pt on p1042, 118.1pt on p202). **Not done / next:** the 31 `fraction_bar_candidate` regions on 15 pages have **not** been looked at yet, so no precision figure for them exists; the 51 outlined runs are detected and flagged but **not transcribed**, so that text is still absent from the corpus; 2D formulas are still not quarantined in `segment/`. `unclassified = 0` means every region is *named*, not that every named verdict has been checked by eye — of the seven kinds, `header_rule`, `table_frame`, `antialias_speck`, `rule_fragment` and `header_band_fragment` were confirmed on sampled examples only. ## 2026-08-01 (cont'd) — Two 2D fraction formulas confirmed corrupted in output by reading the source page images; both tools are blind to them, so cross-tool agreement does NOT bound recall **Finding, visually confirmed on the rendered source, n=2:** stacked-fraction formulas lose the fraction bar and emit the numerator *before* the `=`, so the division reads as multiplication. | drug | physical page | source (read from the page image) | pipeline output | |---|---|---|---| | NETILMICIN | 1042 | `Cl_cr (ml/phút) = [(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ)] / [Nồng độ creatinin huyết thanh (micromol/lít) x 0,81]` | `(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ) Clcr (ml/phút) = Nồng độ creatinin huyết thanh (micromol/lít) x 0,81` | | AMPICILIN VÀ SULBACTAM | 202 | `Cl_cr (ml/phút) = [Thể trọng (kg) x (140 - số tuổi)] / [72 x creatinin huyết thanh (mg/dl)]` | `Thể trọng (kg) x (140 - số tuổi) Clcr (ml/phút) = 72 x creatinin huyết thanh (mg/dl)` | Read literally, both now state that clearance is *multiplied* by serum creatinine. This is a dosing calculation in a renal-impairment section. The content is **not quarantined and carries no formula flag** — it flows into `chunk/` as ordinary prose. **This corrects the weight I put on cross-tool table agreement earlier the same day.** Measured: on physical page 1042 `pdfplumber.find_tables()` returns **0** regions and opendataloader returns **0** tables; the same holds for the formula region on page 202. The two tools agreeing on 112 shared table pages measures *consistency on what ruling lines make visible*, not recall — they share the blind spot. Agreement must not be reported as evidence of coverage. **Priority consequence:** the 155 table blocks are already `quarantined: true`, i.e. contained — they cannot poison an answer today. The formulas are uncontained. Formula handling should therefore come before table reconstruction, which is the reverse of the plan written earlier today. **Population sizing, honest limits.** A keyword scan of the output found 185 occurrences of "công thức", of which **93 are "công thức máu/bạch cầu/hồng cầu"** (blood count, not mathematics) and many of the remaining 92 mean "formulation" (`thành phần trong công thức`). So keyword counting cannot size the formula population; only a detector with measured recall can. The two cases above are the first two regression fixtures. ## 2026-08-01 — Readiness check re-measured from the current artifacts (no code change): text coverage complete, tables quarantined, formulas still unhandled Question asked: is the data ready to parse 100%, including formulas and tables? Every number below was recomputed in this session from the files on disk (`ingestion/data/processed/{monographs.jsonl,coverage_ledger.json}`) and from a fresh test run — none quoted from earlier entries. | check | command / scope | result | |---|---|---| | unit tests | `python -m pytest -q` (whole `ingestion/`) | **129 passed** | | monographs / sections | read `monographs.jsonl` | 683 / 11,966 | | table blocks in output | read `monographs.jsonl` | **155 blocks, 155 quarantined** (simple_table 137, multi_level_or_merged_header 17, cross_page_continuation 1) | | span coverage ledger | read `coverage_ledger.json`, all pages | 252,733 spans; `unassigned` = **0** | | ledger states | same | normalized_text 177,754 (8,183,182 ch) / out_of_scope 53,374 (897,692 ch) / heading 12,764 / boilerplate_excluded 4,976 / quarantined 3,862 / structural_excluded 3 | | page coverage | ledger vs `doc.page_count` | 1666 of 1668 pages carry spans | | the 2 pages with no spans | rendered physical 99 and 1666 at 110 dpi, read the images | **both genuinely blank** (0 chars, 0 images, only a frame drawing) — not a loss | | PUA left in output | scan all 11,966 sections | **0** | | U+FFFD in output | scan all 11,966 sections | **0** — closes the gap flagged in the previous entry as never measured | Note the block count differs from the previous entry's `148` — this is a recomputation from the current file, not a correction of a bug; the shape mix also differs from the 180-region whole-book classification because blocks are only the regions that fall inside the monograph range. **Answer: no, not ready for a "100% including formulas and tables" claim.** What is closed: goal A (full coverage, nothing silently dropped) for the monograph text path — `unassigned = 0`, both uncovered pages proven blank. What is open, by name: - **Formulas: no production stage exists.** `grep -il formula` over `ingestion/ingestion/` hits only `chunk/sentences.py` and `cli.py`; all formula work lives in `scratch/`. The only detector fired 3,405 `fraction_bar` hits on 837 of 1668 pages with precision never measured, so there is not even a trustworthy formula *count*, let alone reconstruction. 2D formulas currently linearise into section text unflagged. - **Tables: detected and quarantined, not reconstructed.** 155/155 blocks are `quarantined: true` — provenance kept, unsafe to cite. Borderless tables (BSA nomogram, catalog item 7) are invisible to `pdfplumber` by construction, so the miss rate is unmeasured and undetected tables still contaminate body text. - **Out-of-scope regions unparsed**: 53,374 spans / 897,692 chars (9.6% of ledger chars) — general chapters and appendices — are excluded explicitly but have never been structurally parsed. - **Content accuracy vs. source never measured**; 92.9% / 99.1% is boundary detection only, on an uncleaned 1064-entry denominator. - `chunk/` still has no tests and has never been executed. ## 2026-07-31 (cont'd, 5) — Cleanliness audit before chunking: data is NOT clean; 5 defects measured whole-corpus, incl. ≥/≤ in dosing text lost as PUA glyphs (all 8 PUA codepoints visually confirmed) **Trigger**: user pushed back on starting the chunk stage ("chưa chunk dữ liệu phải sạch"), correctly — chunking was about to run against text that had never been audited for content-level cleanliness. Only boundary detection had ever been measured, never the text itself. **Also fixed this session (small)**: `cli.py` crashed with `UnicodeEncodeError` on Windows cp1258 when printing Vietnamese drug names in `validate`'s unmatched lists — the metrics printed first so past numbers were unaffected, but the tail of the report was lost. Added `sys.stdout/stderr.reconfigure(encoding="utf-8")` in `main()`. Re-ran `cli validate`: exit 0, Vietnamese renders correctly. **Timing measured for the first time** (whole 1668-page PDF, PyMuPDF only): `cli run` = **2m10.6s**, `cli validate` = **44.4s**. Does not cover pdfplumber/opendataloader/docling cross-checks, which are not part of either command. **Boilerplate re-verified independently** against output generated this session: **0 of 11,409 sections** contain "DTQGVN" (was 1,374), 0 of 682 monographs affected. Also closed the previously-flagged gap of "never checked with a different signature": scanned for a bare 3-4 digit line (page number leaking without "DTQGVN" adjacent) — 204 sections matched, sampled 8, **all legitimate content** (`cytochrom P\n450` split across lines, dosing values like `250 microgam/kg`), not boilerplate. Scope limit: 8 of 204 inspected, not all. **Cleanliness audit — whole corpus, 682 monographs / 11,409 sections / 8,241,485 section chars** (`ingestion/scratch/cleanliness_audit.py`, temporary, to be deleted once this finding is fully captured): | signal | occurrences | sections hit | % sections | |---|---|---|---| | mid-sentence line wrap | 99,501 | 8,197 | 71.8% | | short fragment lines (<4 chars) | 11,612 | 2,149 | 18.8% | | bare-number lines | 2,540 | 862 | 7.6% | | flattened table rows | 25 | 9 | 0.1% | | PUA chars | 86 | 41 | 0.4% | **Confirmed: table content IS contaminating section body text.** Real example — AMPICILIN's `duoc_ly_va_co_che_tac_dung` contains an antibiotic-resistance table flattened to `'Salmonella typhi\n378\n10,6\n 0,0\n89,4\nShigella flexneri\n120\n41,6...'`, losing all row/column semantics. The 0.1% figure is only what the all-numeric-row regex catches; the true table count is pending the inventory scan and will be higher. **Confirmed, patient-safety relevant: comparison operators in dosing text are being emitted as raw PUA codepoints.** All 8 distinct PUA codepoints in the corpus were located in the source PDF, rendered to images, and read directly (not inferred from context): | codepoint | count | actual glyph | visual evidence | |---|---|---|---| | U+F0B3 | 57 | **≥** | p.141 "trẻ em ≥ 10 tuổi" | | U+F0A3 | 17 | **≤** | p.169 "liều ≤ 100 mg" | | U+F061 | 5 | **α** | p.334 "Streptococcus α tan huyết" | | U+F0AE | 3 | **→** | p.1027 "HCO₃⁻ + H⁺ → H₂CO₃ → CO₂ + H₂O" | | U+F0D2 | 1 | **®** | p.891 "Plasma Lyte® 56/5%" | | U+F031 | 1 | **₁** | p.957 "alpha₁-acid glycoprotein" | | U+F0AF | 1 | **↓** | p.1033 "rhodanese ↓" (catalysis arrow) | | U+F067 | 1 | **γ** | p.1352 "interferon - γ" | 74 of 86 occurrences are ≥/≤ inside dosing or adverse-effect sentences — losing the operator changes clinical meaning ("liều ≤ 100 mg" vs "liều 100 mg"). Fonts involved: `SymbolTiger` (7 codepoints) and `Symbol` (1). **Chunk stage — partially built, then deliberately paused.** Wrote `ingestion/ingestion/chunk/` (`models.py`, `sentences.py`, `chunker.py`, `io.py`, `__init__.py`) implementing ADR 0004: `(drug_id, section_key)` unit, 800-token ceiling, sentence-boundary-aware sub-chunking. **Not tested, not run, and must not run until the cleanliness defects above are fixed** — chunking dirty text bakes the defects into embeddings. ADR 0004's own "hard prerequisite" (the boilerplate bug) is satisfied, but this audit found additional blockers it did not know about. **Strategy adopted for full-coverage parsing** (written up in `docs/full-coverage-parsing-plan.md`): separate what is provably clean from what is not — chunk the clean text, flag-and-exclude untrustworthy tables/ 2D formulas with an exact excluded count, and prove nothing was silently lost via a **character coverage ledger** (every char on all 1668 pages must land in exactly one bucket: section text / table cell / formula region / out-of-scope / `unassigned`, with `unassigned` reported as a number plus page+bbox list). Note the plan explicitly distinguishes goal A (full coverage, nothing silently dropped — achievable) from goal B (proven 100% correct — requires manual ground truth for every table/formula, not achievable in one day). **Fixes landed after the audit above — new `ingestion/ingestion/normalize/` stage** (`glyphs.py` = the verified PUA map, `text_flow.py` = geometry-driven span rejoining). Root cause of defects 1-3 was one line in `segment/assembler.py`: `body_lines.append(span.text.strip())` made every *span* its own line, so any visual line the PDF split into multiple spans (italic run, subscript, symbol font) became multiple lines. Text-level regex cannot distinguish a mid-word span split from a real line wrap, so the fix uses geometry instead — PyMuPDF's own `(block, line)` indices identify spans sharing a visual line, and the horizontal gap (`SPACE_GAP_PT = 1.0`) decides whether a space belongs. Assembler now collects `Span` objects and joins via `normalize.join_spans` + `normalize.substitute_pua`. **Whole-corpus re-measurement after the fix** (same audit script, same scope — 682 monographs / 11,409 sections): | signal | before | after | |---|---|---| | mid-sentence line wrap | 99,501 | **0** | | short fragment lines | 11,612 | **7** | | bare-number lines | 2,540 | **0** | | flattened table rows (numeric-row regex) | 25 | **0** | | PUA chars | 86 | **0** | `cli validate` re-run after the change: **unchanged** at 682 monographs, 92.8% recall, 99.1% precision — normalization does not affect boundary detection. Tests: **119 passed** (110 before; 9 new in `tests/ test_normalize.py`, covering the real corpus cases — `cytochrom P450` subscript rejoin, `(feline immunodeficiency virus)` italic rejoin, ≥/≤ restoration in dosing sentences, unmapped-PUA reporting). One existing test (`test_running_header_boilerplate_stripped_...`) had its expected string updated: it encoded the old `\n` join for `"...không nhai. Nếu"` + `"uống viên thuốc..."`, which is exactly the mid-sentence wrap being fixed; its core assertions (no "DTQGVN", no "1009") are unchanged. **NOT verified — total section chars dropped 13,224** (8,241,485 → 8,228,261, 0.16%). Reasoning from the code says this is separator characters only (same-line spans previously contributed a `\n` each, now join directly; `strip()` only ever removed whitespace and no span is dropped), so non-whitespace content should be unchanged at 6,552,254 — but **this was reasoned, not measured**. The character coverage ledger (below) is the instrument that would actually prove it and has not been run. **Whole-corpus table/formula inventory completed** (17m17s, `ingestion/scratch/inventory_tables_formulas.py`, all 1668 pages): - **200 tables on 152 distinct pages**, 0 page errors. Column distribution: 3 cols ×78, 2 ×72, 4 ×32, 5 ×10, 1 ×4, 7 ×3, 6 ×1. - ~~22 header-less-at-top continuation candidates~~ — **this figure was wrong and is corrected below**: classifying all 200 regions individually showed 17 of those 22 are `not_a_table_full_page` and 3 are `not_a_table_degenerate`, leaving **2** real cross-page continuations. Cause: the inventory's condition (`starts_near_top AND header_textual_cells <= 1`) is satisfied automatically by any full-page false-positive region — its bbox starts at y≈0, and its single cell is a long text blob rather than a textual header — so every non-table landed in the continuation bucket. - **0 all-numeric wide grids** — but this is a detector limitation, not evidence of absence: the known BSA nomogram (item 7) has no ruling lines, so `pdfplumber.find_tables()` cannot see it at all. - **Formula detector over-fires badly and its output must not be quoted**: 3,405 `fraction_bar` hits across **837 of 1668 pages** (half the book) is not credible as a formula count — the thin-horizontal-rect signal is evidently matching table rules/underlines/column separators. Precision was never measured; this confirms the standing warning that a bbox heuristic finds candidates, not formulas. `small_font_numeric` (2,583) is likewise unvalidated. Only the PUA count (86) from that scan is trustworthy, and only because all 8 codepoints were visually confirmed. **Section-name spelling variants — a large silent section loss, found and fixed.** Scanned the whole monograph range for bold heading strings that do not match the vocabulary, ranked by similarity: **42 distinct near-miss strings, 542 occurrences**. The dominant one is `"Thông tin qui chế"` (**469×**) — the book prints "qui" where its own documented template (and `vocab.py`) says "quy", so `match_section` returned `None` and the section was never opened. Measured before the fix: only **96 of 682 monographs (14.1%)** had a `thong_tin_quy_che` section; **586 were missing it entirely** (the text itself was not lost — it fell into the preceding section's body unlabelled — but the structure was, so a "thông tin quy chế của X" query could not retrieve it and citations would name the wrong section). Two mechanisms were added rather than one long alias list: - `SectionDef.aliases` for genuinely different wordings ("Mã ACT", "Chống chỉ đinh", "Thời kì mang thai", "Hướng dẫn cách sử trí ADR", "Quá liều và xử lý", "Dược lí và cơ chế tác dụng", …). - `_lookup_key()` folds typesetting noise for every entry at once — all whitespace removed, case folded, and the U+00D0/U+0110 look-alike ("Ðộ" vs "Độ") mapped. This alone absorbs ~14 variants that would otherwise each need an alias: "Chỉđịnh", "Chống chỉđịnh", "Độổn định và bảo quản", "H ướng dẫn cách xử trí ADR", "Tư ơng kỵ", "Tác dụng khôngmong muốn (ADR)", "Thận trọng.", "Liều l ượng và cách dùng", … - Two near-misses were **deliberately rejected** and recorded in `REJECTED_NEAR_MISSES` so a later reader does not add them: "Thể trọng" (body weight, 0.84 similar to "Thận trọng"/caution) and "Tác dụng không mong muốn của opioid" (a drug-specific sub-heading, not the section). **Whole-corpus result after the vocabulary fix:** | | before | after | |---|---|---| | monographs | 682 | **683** | | sections total | 11,409 | **11,966** (+557) | | `thong_tin_quy_che` present | 96 (14.1%) | **567 (83.0%)** | | `cli validate` | 92.8% / 99.1% | **92.9% / 99.1%** | | tests | 119 | **122** | **Over-joining check (the direction the rejoining work had not tested).** First attempt used text patterns and had poor precision — sampled examples were mostly false positives ("Liều lượng có thể tăng…" is ordinary prose, "Lọ 10, 50, 100 ml" is a volume list, "Wolff - Parkinson - White" is a hyphenated name), so its counts are not reported here. Redone at the level where it can actually be judged — the geometry of the two visual lines being joined — with the same exclusions `assemble()` applies (bold headings and header-band boilerplate removed, since joins involving those never reach body text). Monograph range, 180,131 body spans → 154,683 visual lines, **102,798 joins performed**: | category | count | % of joins | |---|---|---| | clean wrap | 98,611 | **95.9%** | | indent change | 1,983 | 1.9% | | vertical gap > 16pt | 514 | 0.5% | | column change | 634 | 0.6% | | page change | 567 | 0.6% | | upward (column/page turn) | 489 | 0.5% | Vertical gap at join points: median **12.1pt**, p90 12.4pt — a tight single-leading distribution, i.e. the overwhelming majority are genuine wraps. **But over-joining is real and it is concentrated in tables**: physical page 109 shows a dosage-form table being concatenated cell by cell — `'Viên nén' + '1'`, `'1' + '1 - 4'`, `'1 - 4' + '8 - 12'`, `'8 - 12' + 'Viên nang tác'`, `'18 - 24' + 'Tiêm bắp'`, `'Chưa biết' + 'Tiêm tĩnh'`. This confirms the risk case predicted before the check was run, and it settles an ordering question: **table regions must be excluded before joining, not after.** Not all 1,983 indent-change cases were inspected — at least one sampled case (`'…(ức chế' + 'alpha-glucosidase).'`) is a correct wrap with a hanging indent, so that category's precision is unmeasured. **Span-level coverage ledger built and run whole-document.** Implemented as an optional `ledger` argument to `assemble()` plus a `cli coverage` command, at span level rather than character level (characters cannot balance because normalization joins and substitutes them). All 1668 pages, 252,733 spans after merge: | state | spans | % spans | chars | % chars | |---|---|---|---|---| | normalized_text | 181,616 | 71.9% | 8,231,038 | 87.6% | | out_of_scope | 53,376 | 21.1% | 897,724 | 9.6% | | heading | 12,764 | 5.1% | 221,266 | 2.4% | | boilerplate_excluded | 4,976 | 2.0% | 47,609 | 0.5% | | **unassigned** | **1** | 0.0% | 21 | 0.0% | The single unassigned span is `"CÁC CHUYÊN LUẬN THUỐC"` on physical page 98 — a part-divider title excluded on purpose via `PART_DIVIDER_TITLES`. This also **settles the previously-unverified 13,224-character delta**: `raw_chars_before_merge` = 9,397,658 equals the post-merge total exactly, so the span-merge step loses no characters; the delta was separator characters in section assembly, as reasoned earlier but now measured. **Important limit, learned the hard way in the same session**: the ledger proves every span was *routed*, not that routed content *survived* into the output. The section-overwrite bug below was invisible to it — spans were correctly marked `normalized_text`, then their section was overwritten downstream. **Table isolation wired into `assemble()` and gated.** `assemble(spans, table_index=...)` diverts spans inside a real table region into `Monograph.tables` (a new `TableBlock` with `table_id`, `shape`, `physical_page`, `bbox`, `section_key`, `quarantined`). Gate results over the whole book: | gate | result | |---|---| | `non_table_span_changed` | **0** | | `table_span_in_normalized_text` | **0** | | `unintended_duplicate` | **0** | | `section_emptied` | **0** (was 1 before the overwrite fix) | | `unassigned` | 1 (the deliberate part divider) | | lifted blocks | 148, all with unique ids | | quarantined | **148 / 148** | Quarantine policy was widened per review: every multi-column shape (`simple_table`, `multi_level_or_merged_header`, `cross_page_continuation`, `grid_2d_numeric`) is quarantined until a real row/column reconstruction exists, because linearised cells are not safe to cite. Only `single_column_boxed_list` is exempt — one column linearises correctly. `183 regions loaded but only 148 blocks lifted` is explained, not a loss: 1,676 table spans sit on pages outside the monograph range (e.g. physical page 42, in the general chapters), where no monograph is open to attach them to. Those pages are still out of scope entirely. **Three real bugs found by these gates, all fixed:** 1. **Section overwrite destroyed content in 33 monographs (38 occurrences).** A repeated section heading inside one monograph replaced the existing `SectionSpan`, discarding everything captured before the repeat. CEFAMANDOL's `lieu_luong_va_cach_dung` held only 172 characters of flattened renal-dosing table; after the fix it holds **881 characters** of real dosing prose ("Cách dùng Thuốc được dùng dưới dạng cefamandol nafat…"). Sections are now concatenated, with the first heading kept as the provenance anchor. Other affected monographs include CEFAPIRIN NATRI and CEFRADIN — also dosing sections. 2. **Duplicate `table_id`.** A region flushed twice emitted two blocks with the same id; provenance ids must be unique. Now suffixed (`p339_t0`, `p339_t0#1`). Verified: 148 blocks, 148 unique ids. 3. **Table blocks were never written to disk.** `write_monographs_jsonl` had no `tables` field, so all 148 lifted blocks were computed, reported in the run summary, and then silently dropped at the file boundary. Found only because a check script raised `KeyError: 'tables'`. Fixed with a round-trip test. Tests: **129 passing** (122 → 129). **682 → 683 explained.** A faithful reconstruction of the pre-fix vocabulary (old `match_section`/`match_section_with_inline_value` patched into the importing modules, no aliases, no whitespace folding, no Ð/Đ mapping) reproduces exactly **682**; the current code gives **683**. The difference is one monograph: **CARBAMAZEPIN**, physical pages 315-319, ATC `N03AF01`, 18 sections, anchor "Carbamazepine.". No monograph disappeared (`GONE` is empty) and it occurs exactly once, so this is a recovered false negative, not a duplicate — it is the same `"Carbamazepin, 316"` entry that `cli validate` had been listing as unmatched ground truth. Two earlier attempts at this comparison were **invalid** and their numbers (683/683 and 589/683) should be ignored: the first left aliases in `_PREFIX_CANDIDATES` and kept the new `_lookup_key`, the second built old-style lookup keys but still queried them through the new whitespace-stripping key function. **Not done yet / next up:** **Design revised** (per review feedback, and it is the better design): make it a **span/fragment-level ledger** first and aggregate characters from it, because normalization joins, substitutes and drops characters so a pure character count cannot balance. States: `normalized_text`, `table`, `formula`, `boilerplate_excluded`, `out_of_scope`, `quarantined`, `transformed_with_mapping`, `unassigned`. - PUA reporting should be stated as `known_mapped` / `unknown_pua` / `replacement_char_U+FFFD` counts; only `pua_chars = 0` has been measured, `U+FFFD` has never been checked. **All 200 table regions classified individually, then the "not a table" verdicts checked by rendering every one of them and reading it.** This is recorded in full because the first two counts reported in this area were both wrong, and both were wrong the same way — stated from metadata before anything was looked at: 1. "22 header-less-at-top continuation candidates" — wrong, see the correction above; the real figure is 4. 2. "22 of 200 are not tables" — asserted from rules (area ratio ≥ 0.75, `n_rows <= 1 or n_cols <= 1`) without opening a single page. Rendering all 22 and reading them showed **20 correct, 2 wrong**: - Correct (not tables): p1 copyright page; p3, p5, p1529 blank pages; p7 table of contents; p9, p10 committee member lists; p12 Vietnamese/English drug-name list; p1665 back index; p55 ×3 epilepsy classification lists; and p172, p196, p382, p760, p944, p1034, p1230, p1336 — **ordinary two-column monograph prose** that `pdfplumber.find_tables()` reports as one page-sized table. - **Wrong**: p62 and p72 are 1×3 regions with visible cell rules — real **orphaned continuation rows** of tables broken across a page (outlier-catalog item 5). The `n_rows <= 1` rule discarded precisely the case where losing content hurts most, since a row without its header cannot be interpreted at all. `classify.py` now treats only `n_cols <= 1` as degenerate and routes a single row with several columns to `cross_page_continuation`. Corrected whole-set result: | shape | count | |---|---| | simple_table | 154 | | multi_level_or_merged_header | 22 | | not_a_table_full_page | 17 | | cross_page_continuation | 4 | | not_a_table_degenerate | 3 | | **real tables** | **180** | | **not tables** | **20** | **Verification scope, explicitly**: all 20 non-table verdicts were confirmed visually, one page at a time. The 180 real tables' individual shapes (simple vs multi-level header vs continuation) are **rule-derived only and have not been checked by eye** — that classification must not be reported as verified. **Is "200 tables" trustworthy? Partly — and the limits matter.** - **No truncation**: 200 records across 152 distinct pages (max 5 on one page, spanning physical pages 1-1665). Re-running `find_tables()` over just those 152 pages reproduces exactly 200. The round number is a coincidence, not a cap. **But this is a reproducibility check with the same tool and settings, not independent validation.** - **Detection recall, measured against the book's own captions**: 33 pages carry a `"Bảng N"` caption; 32 of them have a detected table → **97% on the captioned subset**. 102 detected-table pages carry no caption, which is expected (most tables here are unnumbered). **This measures recall only on captioned tables** — borderless tables are invisible to `pdfplumber` by construction (the BSA nomogram, outlier item 7, is the known example), so the true total is ≥180 and the miss rate for unruled tables is **unmeasured**. - The single captioned miss is physical page 55, captioned `"Bảng 2: Phân loại quốc tế các cơn động kinh (1989)"`. Rendering it showed the classifier's *structural* verdict was right (one column) but the label `not_a_table_degenerate` was semantically wrong — the book numbers it as a table, and it is a nested numbered list drawn inside a ruled frame. The shape was renamed `single_column_boxed_list` and is counted as a real region: single-column content linearises correctly, so it belongs in the text, unlike a 2D table. Naming it "not a table" risked a later reader discarding it. **New `ingestion/ingestion/tables/` stage** (`models.py`, `classify.py`, `detect.py`, `io.py`): table-region detection is production code, not a scratch script, even though its output is cached (detection takes ≈17 minutes). `pdfplumber` is confined to this module — ADR 0003 established it must never be used for text on this document. Not yet wired into `assemble()`; spans inside table regions are still flowing into section body text. - Table handling: 200 tables are known but nothing consumes them yet; they still flow into section body text as flattened cells (the numeric-row regex now reads 0 because rejoining changed the line shape the regex keyed on — **that 0 does not mean tables stopped contaminating body text**, and claiming otherwise would be wrong). - Formula detector needs a real precision/recall measurement against a golden set before any of its counts are usable. - Whole-corpus table/formula inventory (`ingestion/scratch/ inventory_tables_formulas.py`) was still running when this entry was written — no counts available yet; `docs/full-coverage-parsing-plan.md` has `[chờ đo]` placeholders that must be filled from a real run. - `chunk/` has no tests yet and has never been executed. - **Ground truth is not cleaned**: `cli validate`'s 1064-entry denominator includes repeated cross-reference index lines (e.g. `"- CoA reductase, 285"` appears 10+ times in the unmatched list). ADR 0003 used a 725 denominator, so 91.7% and 92.8% are **not directly comparable**. Neither number should be quoted as settled until the ground truth is cleaned. - Text content accuracy vs. source has still never been measured; the recall/precision figures measure monograph-boundary detection only. --- ## 2026-07-31 (cont'd, 4) — Follow-up on the character-diff's remaining unexplained low-similarity pages: sampled 6, all benign/already-known, none newly investigated pipeline bugs **Scope**: of the ~30-50 pages below 0.95-0.98 similarity left unexplained by the reversed-column-order investigation (2 entries below), sampled 6 — 1498, 309, 382, 699, 1420, 1369 — chosen to cover the two visible clusters (1498-1529 near the back-index transition; scattered monograph-range pages) rather than just the very lowest scores. **Findings, all benign, none a new production-pipeline bug:** - **1498, 699**: table/formula content — `opendataloader-pdf` restructures it into markdown tables/headings, PyMuPDF's plain text flattens it; same underlying content, different presentation. Matches the already-documented "no table reconstruction implemented yet" gap (outlier catalog items 7-8), not a new finding. - **309**: the two tools attribute *different* dosing tables to this page (PyMuPDF: "Bảng 4" single-agent; opendataloader: "Bảng 3" capecitabin+docetaxel combination) — a table-boundary/page-attribution disagreement between the two tools, same known gap as above. - **382, 1420**: the two tools' plain-text page-content genuinely differs (different sections of the same drug appear to land on "this page" per each tool). **Directly checked against the actual production pathway** (`extract_spans()`, dict-mode, already column-sorted) rather than trusting the plain-text diff alone: production output for both pages matches PyMuPDF's own plain text exactly — the disagreement is opendataloader-pdf choosing a different page-boundary cut for overflow text, not a defect in this project's pipeline. - **3, 5, 97**: near/fully blank pages (10-27 chars on one side, 0 on the other) — low information content makes the similarity ratio noisy at this scale regardless of correctness, not evidence of a real problem. **Honest scope limit**: only 6 of the ~30-50 unexplained pages were sampled. All 6 turned out benign or already-documented, which is reassuring but is not the same claim as "all remaining pages are benign" — that would need the full set checked, which this session did not do. Investigation scratch files deleted per CLAUDE.md now that this finding is captured here. --- ## 2026-07-31 (cont'd, 3) — Fixed the boilerplate-leakage bug flagged by the parallel chunking-design session; independently re-verified their numbers before touching any code **Context**: the parallel session below (ADR 0004 / chunking design) found and measured a real bug but deliberately left the fix to this session to avoid a same-file collision. Before writing any fix, independently reproduced their exact numbers from scratch (not trusted on read) — matched exactly: 682 monographs, 11,409 sections, 1,374 sections (12.0%) containing a literal "DTQGVN" string, 671 monographs (98.4%) affected, and the exact MORPHIN SULFAT `liều lượng và cách dùng` text they quoted. This is the same discipline applied earlier this session to a mid-session Riboflavin listing error found in this file — re-verify a reported finding directly against real data before building on it, even when it looks correct. **Root cause, confirmed**: `extract/spans.py` already tags the running header ("DTQGVN 2" + page number + repeated monograph name) as `column="full_width"`, but nothing in `segment/assembler.py`'s classification pass excluded it — it matched no section heading and isn't a real all-caps title, so it fell through into plain body text, landing mid-sentence whenever a section's text crosses a physical page boundary. This is exactly outlier-catalog item 13's already-documented risk ("strip the fixed boilerplate before parsing content"), which had a warning but no enforcing code or test until now — added as item 22 in the catalog (item 23 also added for the reversed-column bug from the entry below, which hadn't been given a catalog number yet either). **Fixed**: new `assembler._is_page_boilerplate(span)` — drops any span with `column == "full_width"` and `y0 < HEADER_BAND_Y` (same header-band threshold `page_map.py` already uses for folio detection; exported that constant as public rather than duplicating the magic number) before any other classification. Regression test added using the real MORPHIN SULFAT span shape (`tests/test_segment_assembler.py`). **Whole-corpus re-measurement after the fix**: 0 of 11,409 sections contain "DTQGVN" (was 1,374). `cli validate` unchanged: 682 monographs, 92.8% recall, 99.1% precision — the fix only touches body-text content, not monograph/section boundaries. 110 tests total (was 109), all passing. **Not done yet / next up:** - Chunking (ADR 0004, the parallel session's design) can now safely run against real ingestion output for this specific defect — but see the entry below's own "not done yet" list (sub-chunk splitter not built, general-chapters/appendices scope, sub-compound tagging) for what's still actually blocking Phase 2 beyond this fix. - Only checked for the literal "DTQGVN" substring as this bug's signature — did not separately verify whether the page-number token alone (without "DTQGVN" adjacent) ever leaks in some other layout shape; the fix itself is structural (column+y-position, not text-pattern-based) so it should cover that too, but this wasn't independently re-measured after the fix with a different detection signature. --- ## 2026-07-31 (cont'd, parallel session) — Phase 2 chunking strategy designed (ADR 0004) from real per-section measurements; found and flagged a new whole-corpus boilerplate-leakage bug for the extract/segment session to pick up **Context**: this entry comes from a second session running in parallel with the one still fixing `extract`/`segment` parsing bugs, on the same checkout (no worktree separation). Per explicit scoping agreed with the user, this session touched **only** `docs/adr/0004-chunking-strategy.md` (new), `docs/architecture.md`'s chunking paragraph, this log entry, and a since-deleted scratch script — it did not touch `extract/*.py`, `segment/*.py`, or `docs/document-profile.md`, to avoid colliding with the other session's in-flight edits to those files. **Done:** - Ran `python -m ingestion.cli run` for real (full 1668-page PDF) to produce `ingestion/data/processed/monographs.jsonl` (682 monographs — gitignored output, matches the count already reported elsewhere in this log), then measured real per-section text-length distribution across the whole corpus for the first time (`ingestion/scratch/chunking_stats_survey.py`, now deleted per this project's investigation-script rule, findings captured below and in the ADR). - **Replaced the never-validated chunking guess in `docs/architecture.md`** (`(drug, section)` unit, ~500-800 tokens, 400-tok/50-overlap sliding window — written before segmentation existed) with a design grounded in the real measurement: `(drug_id, section_key)` chunk unit confirmed; 800-token ceiling (chars/4 estimate) confirmed as directionally right (clears ~16/18 section types at p90); **but sub-chunking is the routine path, not a rare hedge, for 2 specific sections** — `dược lý và cơ chế tác dụng` (242/678 monographs with that section, 35.7%, max ≈3542 est. tokens) and `liều lượng và cách dùng` (200/675, 29.6%, max ≈3631 est. tokens); a smaller tail also exceeds it (`thận trọng` 3.7%, `tương tác thuốc` 3.4%). Chosen sub-chunking method: **sentence-boundary-aware** sliding window (~600-700 tok/sub-chunk, ~50-80 tok overlap), not a blind character/line window — `assembler.py`'s `body_lines` join one PDF visual line-wrap per line, not a semantic boundary, so a blind window risks splitting a dosing sentence mid-way (a real, measured risk given outlier item 17: adult/child dosing splits appear on 1,121/~1,400 monograph-range pages). Full rationale, extended chunk metadata schema (`chunk_id`, `atc_codes`, `part_index`/`part_count`, etc.), and 4 explicitly-flagged open gaps (sub-compound tagging inside class-level monographs, sub-chunk page-precision, the splitter itself not yet built, general-chapters/appendices chunking out of scope) are in `docs/adr/0004-chunking-strategy.md`. - **Found and measured a new whole-corpus bug, not yet fixed, flagged here for the `extract`/`segment` session rather than fixed directly** (per user's explicit choice this session, to avoid a same-file collision): running header/footer boilerplate ("DTQGVN 2" + page number + repeated drug name — tagged `column="full_width"` in `extract/spans.py`) is never filtered out of section body text; `assembler.py` appends every non-title, non-section-heading span to `body_lines` regardless of column tag. Measured whole-corpus: **1,374 of 11,409 sections (12.0%) contain a literal "DTQGVN" string mid-text; 671 of 682 monographs (98.4%) have at least one affected section.** Real example: MORPHIN SULFAT's `liều lượng và cách dùng` reads `"...Nếu\nDTQGVN 2\n1009\nMorphin sulfat\nuống viên thuốc..."` — the page number and drug name are spliced mid-sentence into a real dosing instruction. This is `docs/pdf-parsing-outlier-catalog.md` item 13's already-documented risk ("header/footer boilerplate must be stripped"), just never actually measured/fixed until this session — it should become a new numbered item in that catalog (item 22, or the next free number by the time this is read — check the catalog directly) with these numbers, but that file is mid-edit in the other session so this entry leaves the actual catalog edit to them rather than risking a concurrent-write collision. Note: this bug is **separate from** the reversed-column-order bug documented in the entry directly below this one — that bug was about which *column* content lands in, this one is about full-width header-band content never being excluded from body text regardless of column. **This is a hard blocker for Phase 2**: chunking must not run against real ingestion data until this is fixed, or boilerplate gets baked into embeddings and can surface mid-sentence in a chunk shown to a doctor/pharmacist. **Not done yet / next up:** - The boilerplate-leakage bug above needs a real fix in `extract`/`segment` (likely: exclude `column="full_width"` spans from body-text assembly, or an explicit boilerplate-pattern filter) plus a regression test and a whole-corpus re-measurement to confirm it's actually gone — not done by this session, left for whoever owns `extract`/`segment` next. - `ingestion/ingestion/chunk/` still doesn't exist — ADR 0004 is a design only; implementing and unit-testing the sentence-boundary splitter is a separate task. - Chunking design for general chapters (pp. 37-98) and appendices (pp. 1497-1528) is still blocked on `docs/document-profile.md`'s Group 2 investigation (tables, 2D stacked-fraction formulas) completing. - Sub-compound tagging inside class-level/multi-ATC monographs (25.5% of corpus) has no design yet — flagged in ADR 0004, deferred to golden-dataset-driven eval. --- ## 2026-07-31 (cont'd, 2) — Built a whole-document cross-tool character-diff QA check; it found a real, serious cross-monograph data-corruption bug (reversed column reading order), now fixed and whole-corpus-reverified at zero occurrences **Why this check was built:** after the ATC-field bug-fixing session below, the user asked what validation step would catch whether parsing is "correct" at all — not just "does `cli validate` say recall/precision are high," since that check only confirms a monograph *exists* at roughly the right name/page, not that its *content* is complete and correctly attributed. Per [[feedback-rigorous-validation]], comparing PyMuPDF's own output against itself can't validate itself — a second, independently-implemented parser is required as real ground truth. Built a whole-document (all 1668 pages) per-page character-similarity diff: PyMuPDF's `page.get_text()` vs `opendataloader-pdf`'s markdown extraction, normalized and compared with `difflib.SequenceMatcher`. **Two bugs in the check script itself, found and fixed before trusting any result (disclosed to the user immediately on discovery, not after):** 1. Wrong page-separator placeholder syntax (`{page}` instead of the tool's real `%page-number%`) risked silent page misalignment. Fixed by using the real placeholder and parsing the actual page number from each separator instead of assuming positional order. 2. Python's `difflib.SequenceMatcher` default `autojunk=True` collapsed the similarity ratio to ~0.0065 for a page whose content was actually ~98% identical between tools (a long drug-name list trips its "popular element" heuristic) — a well-known stdlib gotcha. Fixed with `autojunk=False`. **Whole-document result** (1668/1668 pages compared, mean 0.9892, median 0.9981): a tight cluster of pages — 929, 1099-1106, 1149-1153 — scored only ~0.47-0.53. Investigated instead of dismissed. **Confirmed real, serious bug in `extract/spans.py`:** the module trusted PyMuPDF's raw block iteration order to already sequence left-column-before- right-column, validated only against one example page back in ADR 0003. Wrong on **12 of 1398 monograph-range pages** (whole-range scan, e.g. physical page 1100): PyMuPDF's raw block order emits the *right* column before the *left* column there. Confirmed by rendering the page to an image and reading it directly, then confirmed in the actual `assemble()` output: OXYMETAZOLIN's right-column sections (Chống chỉ định, Thận trọng, Thời kỳ mang thai, Thời kỳ cho con bú, ADR, Hướng dẫn xử trí ADR, Liều lượng và cách dùng) were being silently attributed to and overwriting the still-open OXYBUTYNIN monograph's own sections, while OXYMETAZOLIN ended up missing all 7. Confirmed boundary pairs affected: OXYBUTYNIN/OXYMETAZOLIN, OXYTETRACYCLIN/OXYTOCIN, OXYTOCIN/PACLITAXEL, PIOGLITAZON/PIPECURONIUM BROMID; MAGNESI SULFAT, PILOCARPIN, and PACLITAXEL had internal (not necessarily cross-monograph) ordering corruption. **This is a real, medical-content-relevant defect** — wrong contraindication/ADR data silently attached to the wrong drug — not a cosmetic parsing issue. **Fixed** by explicitly sorting blocks (full_width header band first, then left column, then right column, each by y-position) instead of trusting PyMuPDF's raw order. Verified: re-scanned the full 99-1496 range for the same reversed-order signature — 0 occurrences (was 12). Directly verified OXYBUTYNIN's and OXYMETAZOLIN's `assemble()`-produced sections are now distinct and drug-appropriate (spot-checked against the rendered page). Whole-book `cli validate` after the fix: unchanged at 682 monographs, 92.8% recall, 99.1% precision, 8 zero-ATC (no regression). Also tried a broader "any within-column y-order violation" scan (670 pages flagged) but verified a sample and found it's dominated by benign subscript/superscript baseline noise (e.g. "B" + subscript "6" + ")"), not real bugs — correctly discarded as evidence rather than reported as 670 new findings. **Regression test** added (`tests/test_extract_spans.py`) using the exact real bounding boxes from physical page 1100's raw block order. 109 tests total (was 103), all passing. **Not done yet / next up:** - The whole-document character-diff tooling itself was investigation-only (per CLAUDE.md, deleted from `ingestion/scratch/` after this finding was captured here + in the regression test + in `spans.py`'s docstring) — if this kind of check is wanted as a recurring QA step, it needs to be rebuilt as a real `ingestion/validation/` module, not re-derived ad hoc each time. - The character-diff still has ~30-50 pages below a 0.95-0.98 similarity threshold that were *not* individually investigated this session (only the most extreme cluster was) — front-matter table-like pages (14-31), the back-index transition region (1498-1529), and scattered others (382, 1420, 57, 68, 309, 194, ...) remain unexplained; could be genuine table/formatting differences neither tool handles perfectly, not necessarily more instances of this same bug (the specific reversed-column signature was already whole-range-scanned to exhaustion above). - Phase 1.5 (golden dataset) still requires human review by design. - Phase 2 (chunking) has no code yet and no design decision made. --- **Correction to the previous entry below, per CLAUDE.md's "never fabricate" rule:** re-running `assemble()` fresh at the start of this session (same code, nothing had changed on disk) produced **676** monographs and **48** zero-ATC-not-stated-absent, not the "680 / 46" the previous entry claimed — and that entry also self-contradicted (46 in one line, 42 two paragraphs later). Root cause: the previous session's final numbers were asserted without a fresh re-run after the very last code edit. No `monographs.jsonl` artifact existed to diff against, so this can't be proven beyond doubt, but it's the only explanation consistent with the evidence. Lesson applied going forward: a number is only "final" if it comes from a command run *after* the last related edit, in the same message reporting it. **Method used this session, per two user corrections mid-session**: initial passes relied only on PyMuPDF span text and coordinate reasoning. The user first pointed out other installed PDF tools were going unused and that pages should be rendered to images and read directly rather than trusted from span dumps alone (per [[feedback-visual-verification]]) — so a first cross-check used `pdfplumber.extract_text()` plus rendered-page-image reads. The user then flagged this as still not matching "the strategy from before." That strategy already existed, in full, in the [[pdf-parsing-strategy]] memory and `docs/adr/0003-pdf-parsing-strategy.md`: **4 tools were already evaluated there** (PyMuPDF, pdfplumber, opendataloader-pdf, docling), and it already concluded **`pdfplumber.extract_text()` scrambles reading order on this document's two-column layout and must never be used for general text** — only PyMuPDF (primary) and `opendataloader-pdf` (independent reading-order + font-metadata cross-check) are validated for that purpose. The `MEMORY.md` index line for that memory doesn't carry this detail, only the full memory file does — this session used the one-line index and never opened the full file before picking a cross-check tool, which is the actual process gap (not a memory-setup gap). All findings below were then re-verified with `opendataloader-pdf` instead, and the earlier pdfplumber pass was discarded as unreliable evidence, not cited. **Investigated and closed** (whole-book `cli validate` against the real back-of-book index, not a sample): - The 8 detected monographs that didn't match any back-index entry: 2 were a real bug in `validation/metrics.py` (substring name-matching let a shorter monograph name, e.g. "ISOSORBID", "steal" the ground-truth match meant for a longer, textually-overlapping but genuinely distinct monograph, e.g. "ISOSORBID DINITRAT" — both are real, correctly segmented drugs). Fixed: try an exact normalized-name match before falling back to substring. The other 6 are real book-internal inconsistencies, not pipeline bugs (compound names containing " - " skipped by the already-documented cross-reference filter; title-vs-index spelling variants like "HYDROGEN PEROXID" vs the index's "Hydrogen peroxyd"). - The 83 unmatched ground-truth entries: ~40 are back-index line-wrap parsing artifacts ("- CoA reductase" / "gonadotropin" fragments from wrapped cross-reference lines, not real entries), ~20 are front-matter/ general-chapter TOC entries (pages 39-98, before the monograph range even starts at printed page 99) that `back_index.py` doesn't filter out, a handful are the same title-vs-index spelling-variant pattern as above — and **7 were genuinely missing monographs**, root-caused to 2 real bugs (see below) plus one real book typo (CARBAMAZEPIN's own printed heading reads "Ten chung quốc tế", missing the "ê" — confirmed independently by both a rendered-page-image read and `opendataloader-pdf`'s text output, which shows the same missing "ê"; not fixable without risking false positives elsewhere, left as-is). **4 real bugs found and fixed, each confirmed via a whole-corpus scope check (not just the sample that surfaced it) and, where the defect could be page-rendering vs data, a rendered-page-image visual check:** 1. **Same-line diacritic span-fragmentation** (`segment/merge.py`, `merge_same_line_bold_fragments`, new): PyMuPDF splits some bold spans into multiple fragments around diacritic characters even when the text is one unbroken visual line — confirmed by rendering physical page 759 to an image ("Tên chung quốc tế" looks completely normal to a human reader). Cross-checked against `opendataloader-pdf` (the tool [[pdf-parsing-strategy]]/ADR 0003 already validated for this — not pdfplumber, which that ADR found scrambles reading order on this document's two-column layout) on 2 of the 5 affected pages (759 GUAIFENESIN, 943 MEPHENESIN): both reconstruct the line cleanly, e.g. "Tên chung quốc tế: Mephenesin. Mã ATC: M03BX06." with no fragmentation, confirming this is a PyMuPDF span-boundary artifact, not a defect in the PDF itself. **Correction**: an earlier version of this entry claimed all 6 candidate pages were cross-checked and listed RIBOFLAVIN among them — both wrong. Only 2 of the 5 real pages were actually re-verified with opendataloader-pdf just now, and RIBOFLAVIN's failure is the separate folio-subscript bug below, not this one — it was never part of the diacritic-fragmentation set. Broke the anchor check that gates false-positive title filtering, silently dropping whole monographs. Confirmed for 5 real monographs (GUAIFENESIN, MEPHENESIN, NATRI THIOSULFAT, RAMIPRIL, TENOXICAM) via a full 1668-page scan for the fragment signature; the other 3 (NATRI THIOSULFAT, RAMIPRIL, TENOXICAM) were not independently cross-tool-verified, only confirmed via PyMuPDF's own span coordinates (same-line y-gap). 2. **Folio-detection false conflict** (`extract/page_map.py`, `pick_folio`): RIBOFLAVIN's monograph sits high enough on physical page 1243 that its own "2" subscript (from "Vitamin B₂", font size 5.83) falls inside the header band alongside the real folio "1244" (size 10.0), producing two conflicting digit candidates and silently dropping the printed page — and the whole monograph with it. Fixed by preferring the largest-font- size candidate(s) (a real folio is always set in the header's own running size, never a subscript's reduced size); a full-document scan confirmed this exact conflict shape occurs on exactly 1 of 1668 pages. Confirmed visually by rendering the page. 3. **ATC comma-inside-annotation** (`segment/atc.py`): the field-text split on "," ran *before* parenthetical annotations were stripped, so an annotation containing its own comma broke the split — e.g. "Mã ATC: J07BD01 (Measles, live attenuated)." split into two unrecoverable fragments. INSULIN's earlier-fixed Vietnamese annotations ("người", "bò") never contain a comma, so this only surfaced with vaccines' English annotations — affected 12 vaccine monographs. Fixed by stripping *all* parenthetical groups before splitting, not just a trailing one per already-split segment. `opendataloader-pdf` cross-check on the real VẮC XIN SỞI page confirms the source text genuinely is "Mã ATC: J07BD01 (Measles, live attenuated)." — the bug was in parsing, not the data. 4. **ATC leading colon from the value span** (`segment/atc.py`): some monographs render the bold label as "Mã ATC" (no colon) with the colon on the plain *value* span instead (": M03AA01." vs Abacavir's "J05AF06." with the colon on the label side) — the section still matched correctly, but the leftover leading colon made the stripped candidate 8 characters instead of 7, failing the length check. Fixed by stripping a leading colon in `normalize_atc_candidate`, symmetric with the existing trailing strip. Affected 15 monographs. `opendataloader-pdf` cross-check on the real ALCURONIUM CLORID page confirms clean source text ("Mã ATC: M03AA01."), same conclusion. 5. **ATC name-prefixed and reversed "CODE: Name" shapes** (`segment/atc.py`, same session, found continuing the zero-ATC investigation after the above): two more real shapes surfaced once the first 4 fixes cleared the noise. (a) 7 monographs with multiple salt/ester forms write each form as "Name: CODE" per line, e.g. ARGININ's "Arginin glutamat: A05BA01\n Arginin hydroclorid: B05XB01" — the whole segment including the name was compared against the 7-char code shape and rejected. (b) The class-level "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" monograph writes it the *opposite* way, code first — "C10A A01: Simvastatin\nC10A A02: Lovastatin\n...". Fixed both with one change: `normalize_atc_candidate` now tries the text after the last ":" first, then before, returning whichever side actually normalizes to a valid ATC shape — safe because a real drug name never happens to match the strict `[A-Z]\d{2}[A-Z]{2}\d{2}` pattern, so there's no real ambiguity between the two candidates in practice. **Net effect, whole-book, before -> after all 5 fixes:** detected monographs 676 -> **682**; recall 92.2% -> 92.8% (981 -> 987 / 1064); precision 98.8% -> **99.1%**; zero-ATC-not-stated-absent 48 -> **8**. 103 tests total (was 88 at the start of this entry), all passing, each new fix with a regression test built from the exact real-corpus text that exposed it. **The remaining 8 zero-ATC monographs are now all explained, none left unresolved:** - 7 (CROTAMITON, INTRALIPID, ISOSORBID, OXYBENZON, PEMIROLAST, SIMETICON, the DPT vaccine) have **no "Mã ATC" section anywhere in the book at all** — confirmed by reading the actual span sequence after each title (goes straight from "Tên chung quốc tế"/"Loại thuốc" to the next section, no ATC line ever appears) and by rendering physical page 845 (ISOSORBID) to an image and reading it directly. A real, accepted data gap in the source — not a parsing bug. - 1 (SPECTINOMYCIN) is a confirmed real book typo: its own printed heading reads **"Mã ACT:"** (letters transposed), not "Mã ATC:" — confirmed by rendering physical page 1297 to an image and reading it directly. Same category as CARBAMAZEPIN's "Ten chung quốc tế" typo from fix 1 above: a real defect in the source document, left unfixed rather than loosening vocabulary matching and risking new false positives elsewhere (the project's own prior "whack-a-mole" experience with over-loosened matching, per outlier-catalog item 21). **Not done yet / next up:** - `validation/back_index.py`'s line-wrap and front-matter-entry issues (from the investigation above) inflate the "unmatched ground truth" count but were left unfixed this session — the user's stated priority was the segmentation-pipeline bugs first, not the validation-metric's own accuracy. - The docs/pdf-parsing-outlier-catalog.md items for these 5 new bugs have not been added yet (the module docstrings for `merge.py`, `page_map.py`, and `atc.py` carry the full evidence in the meantime). - Only 2 of the ~7 diacritic-fragmentation pages and 2 of the ~15 leading-colon pages were independently cross-tool-verified with opendataloader-pdf (see fix 1's correction note above) — the rest rely on PyMuPDF's own span coordinates only, which is weaker evidence. - No exploration yet of whether the same fragmentation/folio/colon bug families affect *other* sections beyond "Tên chung quốc tế" and "Mã ATC" (e.g. "Chỉ định", "Liều lượng và cách dùng") — only ATC was swept whole-corpus this session. - Phase 1.5 (golden dataset) still requires human review by design. - Phase 2 (chunking) has no code yet (`ingestion/chunk/` doesn't exist) and no design decision has been made on chunking strategy. --- ## 2026-07-31 — Phase 1.3-1.4 built: assembler, CLI, and validation, with 4 more real bugs found and fixed via whole-book runs **Done (continuation of the same session, user asked to keep driving autonomously via `/loop`; visual PDF-page rendering used throughout to self-verify bugs, per [[feedback-visual-verification]]):** - Built `assembler.py` (3-pass design: classify spans -> coalesce titles -> build Monograph records), `segment/io.py` (JSONL read/write), `cli.py` (`run` and `validate` subcommands working end-to-end), and `validation/back_index.py` + `metrics.py` (recall/precision against the real back-of-book index, parsed from real physical pages 1530+). - **Found and fixed 4 more real bugs via whole-book `assemble()` runs**, each initially surfaced as a wrong number (never trusted the first result, per CLAUDE.md): 1. **ATC trailing-period bug**: "Mã ATC: J05AF06." — the sentence-ending period was counted as part of the code, so `normalize_atc_candidate` silently returned zero codes for every single-ATC monograph ending in "." (a huge fraction of the corpus). Fixed by stripping trailing `.,;` before the length check. 2. **ATC species-annotation bug**: INSULIN's real field lists all 20 codes each with a parenthetical annotation ("A10AB01 (người); ...") — only 2 of 20 survived before the fix (the two that happened to have a line-wrap between code and annotation). Fixed by stripping a trailing `(...)` group before normalizing. Whole-corpus multi-ATC re-count with both fixes: **159/680 (23.4%)** monographs have >1 ATC code (the open item from the very first survey session, now closed with a real measured number instead of the 25.4%-floor estimate). 3. **Non-bold combined section heading (outlier item 20)**: AMITRIPTYLIN's "Mã ATC:" is a single **non-bold** span combining label and value ("Mã ATC: N06AA09."), unlike Abacavir's bold-label-only span — the book's ~700 monographs were written by many different authors, so styling isn't 100% consistent. Fixed by matching section headings by vocabulary **text**, not `span.bold`, plus a new `match_section_with_inline_value` for the combined-span case. 4. **Mixed-case title + false-positive whack-a-mole (outlier item 21)**: the class-level monograph "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" embeds the mixed-case abbreviation "CoA", which a strict `isupper()` check silently dropped from the corpus entirely. Loosening that check (first with an absolute lowercase-count tolerance, found wrong, then fixed with a **lowercase-letter ratio** instead — "Mã ATC:" has 1/5 = 20% lowercase, correctly still rejected, vs. HMG-CoA's 1/27 ≈ 3.7%) then exposed a *second* false positive: individual statin sub-headings ("SIMVASTATIN", "LOVASTATIN", ...) inside that same class monograph, each followed by their own real section but never by "Tên chung quốc tế" specifically. The anchor check (added earlier for the HSV/CMV table-header false positive, item 19) had been loosened to "any section" to pass existing tests — reverted to requiring "Tên chung quốc tế" specifically (the one invariant the book's own template actually guarantees), and fixed the test fixtures instead of the production logic. - Final whole-book numbers after all fixes: **680 monographs** (matches the previously-established count from the original structural survey — though this is a count match, not yet a confirmed identical-set match). Abacavir ATC now correctly `["J05AF06"]`; Insulin now correctly 20 codes. 46 monographs remain zero-ATC-and-not-stated-absent (down from an initial 48; not yet root-caused further — flagged, not silently accepted as final). - 86 unit tests total, all passing, including a regression test for every bug above and for each whack-a-mole cycle (so a future change can't silently reintroduce SIMVASTATIN-as-monograph or Mã-ATC-as-title). **Not done yet / next up:** - The remaining 42 zero-ATC-not-absent monographs likely hide at least one more real pattern (per this session's track record of "one fix reveals the next") — worth one more investigation pass before Phase 1.5. - Phase 1.5 (golden dataset) still requires human review by design — not something this session can complete alone, per the approved plan. --- ## 2026-07-31 — Session end: golden dataset NOT started; general chapters + appendices NOT investigated **Status check requested by user at end of session ("golden dataset bạn để đâu?" / "đã xem chuyên luận chung và phụ lục chưa?") — answering plainly here so the next session doesn't have to guess:** - **Golden dataset (Phase 1.5): not created.** `ingestion/data/qa/` still contains only `.gitkeep` — no `golden_pages.jsonl`, no `golden_monographs.jsonl`. This is intentional, not an oversight: per the approved plan, golden-set ground truth requires human review/sign-off, which this session couldn't do alone (dynamic `/loop` autonomy stopped here for exactly this reason). [[feedback-visual-verification]] means a future session can self-draft much of it (render pages, read them directly) but a human still needs to spot-check before it's trustworthy. - **"Các chuyên luận chung" (general chapters, printed pages 37-98) and "Các phụ lục" (appendices, printed pages 1497-1528): NOT investigated this session, or any prior session.** All work so far (extract/segment/ validation, ADR 0003, the outlier catalog) covers only the drug-monograph range (printed 99-1496). The only contact with these two ranges was incidental: reading physical page 38-39 (inside general chapters) once to transcribe the book's own 19-field section template into `segment/vocab.py`, and skimming physical ~1526-1528 (inside the appendices — specifically "Phân loại thuốc theo mã ATC") only to locate where the back-of-book index begins for `validation/back_index.py`. Neither range has been structurally surveyed, outlier-cataloged, or parsed. This gap has been flagged since the *very first* scaffold session (`docs/progress-log.md`'s original Phase 1 roadmap) and remains explicitly out of scope of the plan approved this session. Known content, not yet verified in depth: general chapters cover topics like "Kê đơn thuốc," rational antibiotic use, pediatric dosing principles; appendices include the body-surface-area nomogram table (already flagged in outlier catalog item 7 as a 2D-table extraction problem), IV-admixture compatibility info, and the ATC drug classification listing. **Next session should pick up one of:** 1. Golden dataset drafting (Phase 1.5) — scaffold from current extraction/segmentation output, self-verify via page rendering, then get human sign-off before trusting it. 2. A first real structural investigation of general chapters + appendices (same rigor bar as the monograph range: whole-range scan, not a page or two) — needed before any chunking strategy can be designed for them. 3. The 42 remaining zero-ATC-not-absent monographs (Phase 1.4 leftover, not blocking). --- ## 2026-07-31 — Phase 1.4 real validation run: 92.2% recall, 98.8% precision (first-ever measurement) **Done:** - Ran `python -m ingestion.cli validate` for real against the full 1668-page book. First result: 91.7% recall / 98.2% precision against 1064 real back-index ground-truth entries (parsed from physical pages 1530+, not a sample) — recall matched ADR 0003's original number exactly (665/725 there was a different, smaller ground-truth set; this run's 1064 entries come from parsing the *entire* back index, not a partial scan), and **precision was measured for the first time ever** on this project, meeting the plan's ≥98% target immediately. - **Found and fixed one more real bug from this first real run**: 4 of 12 unmatched detected monographs (ALVERIN CITRAT, OXYMETAZOLIN HYDROCLORID, TERBUTALIN SULFAT, TIOTROPIUM BROMID) all shared the same shape — a **double space** in the detected title (e.g. "ALVERIN CITRAT") that failed to match ground truth's single-spaced "Alverin citrat" under plain strip+upper comparison. Fixed by collapsing whitespace in `metrics._normalize_name` before comparing. - Final numbers after the fix: **recall 92.2% (981/1064), precision 98.8%** — both real, measured, whole-book numbers, both improving over the already-fixed run (not just over the pre-session 91.7% baseline). - Remaining unmatched entries are traced to two already-documented, known limitations rather than new bugs: (1) `back_index.py`'s own stated trade-off of treating any " - " as a brand-cross-reference marker also excludes genuine compound-name ground-truth entries ("Carbidopa - levodopa", vaccine names like "Vắc xin DPT" that use " - " internally), so a handful of correctly-detected monographs (CARBIDOPA - LEVODOPA, THUỐC PHIỆN - OPIAT - OPIOID, the DPT/MMR vaccine entries) simply have no matchable ground-truth counterpart, not a detection defect; (2) a repeating "- CoA reductase, 285" ground-truth artifact (appears ~12 times) is itself index-parsing noise — likely a long cross-reference line wrapping across two physical lines in a way that splits the brand name from its "- CoA reductase" continuation, which then doesn't contain the " - " marker at its own line start and slips through the cross-reference filter as a bogus ground-truth entry. - 87 unit tests total, all passing. **Not done yet / next up (Phase 1.5, requires human review by design — not something a single session can complete alone per the approved plan):** - Golden dataset authoring: `scaffold-golden` CLI command, golden_pages/ golden_monographs JSONL schemas, human review of drafted entries. - The 42 remaining zero-ATC-not-absent monographs and the back_index.py compound-name/cross-reference-wrapping noise above are both flagged, not blocking — real, moderate-size gaps documented for whoever picks this up next. --- ## 2026-07-30 — Phase 1.2 `segment/` pure logic built and validated against real PDF **Done (real production code, all reused by both the future CLI pipeline and validation — no logic duplicated):** - Transcribed the book's own documented 19-field monograph template verbatim from its source (physical page 38/39 printed, "HƯỚNG DẪN SỬ DỤNG DƯỢC THƯ QUỐC GIA VIỆT NAM") into `vocab.py`'s `SECTION_DEFS`, rather than guessing — cross-checked against real bold headings in the Abacavir/ Acarbose monographs (exact text match, modulo a trailing colon some pages have and others don't, now normalized). Added `ten_thuong_mai` ("Tên thương mại") as the confirmed 19th, undocumented-but-real field. - Built `merge.py` (multi-line/multi-fragment title merging), `detector.py` (monograph + section boundary detection), `atc.py` (3-state ATC extraction: found / recovered-from-noise / stated-absent), `units.py` (defensive mg/mcg/mmol validation — see below), `models.py`. - **Found and fixed a second real title-fragmentation bug by rendering a page to an image and reading it directly** (not just reasoning from coordinates): "ACICLOVIR" was detected as two separate titles, "ACIC" (font size 10.0) and "LOVIR" (font size 9.5) — the same visual word rendered at two slightly different sizes in the source PDF. The merge logic originally required exact font-size equality (which happened to work for the GONADOTROPIN wrap case since both its fragments are size 9.5) — dropped that requirement per the same "font size is not reliable" lesson from ADR 0003, now applied *within* a title's own fragments, not just across monographs. Also fixed the join character: a genuine same-line split needs no space ("ACIC"+"LOVIR"="ACICLOVIR"); a genuine multi-line wrap needs one (GONADOTROPIN case) — distinguished by the y0 gap. This same fix also resolved two other silent duplicate-name artifacts (HSV, CMV) found in the same smoke test. - Smoke-tested the full detector against the real PDF: 695 monograph titles detected (down from 702 pre-fix, closer to the previously-established ~680 count), part-divider correctly excluded, ABACAVIR/INSULIN present, GONADOTROPIN wrap correctly merged, zero unexplained duplicate names. - **Investigated the one remaining duplicate name ("SALBUTAMOL", pages 1261 and 1263) by rendering both pages and reading them directly — confirmed it is NOT a bug**: two genuinely different, complete monographs ("Dùng trong hô hấp" / respiratory vs. "Dùng trong sản khoa" / obstetric use), each with a full 18-section template. Added as outlier-catalog item 18 with an explicit note for Phase 1.3's assembler: `drug_id` generation must fold in the bold, non-all-caps qualifier line beneath the title, or it will wrongly treat this legitimate case as a duplicate-title collision. - Confirmed via a targeted regex scan that the `units.py` whitespace-split defense (built by analogy to the confirmed ATC defect) has **zero** confirmed real occurrences in this corpus so far — documented honestly as a defensive-only check, not a confirmed defect, per CLAUDE.md. - 44 unit tests total (up from 9), all passing, including regression tests for every real bug found this session (kerning jitter, column-merge, GONADOTROPIN wrap, ACICLOVIR same-line split). - Rendering a PDF page to an image and reading it directly (not just reasoning from PyMuPDF coordinates) turned out to be a fast, reliable way to self-verify segmentation bugs — used for both real bugs found this phase (ACICLOVIR, SALBUTAMOL) without needing a human to look at the page. This changes the Phase 1.5 golden-dataset plan: much of the ground-truth drafting can be self-verified this way before a human spot- checks it, rather than requiring a human to author it from scratch. **Not done yet / next up:** - Phase 1.3: `assembler.py` (must handle the SALBUTAMOL qualifier-line case above), `segment/io.py`, `cli.py run`, wired end-to-end; smoke-test on a small page range before a full-book run. - Phase 1.4: `validation/back_index.py` + `metrics.py` (recall/precision against the back-of-book index), `cli validate`. --- ## 2026-07-30 — Phase 1.1 `extract/` module built and validated against real PDF **Done (real production code, not exploratory scripts — replacing the empty `ingestion/ingestion/extract/` stub per the approved segmentation + eval plan):** - Built `models.py` (`Span` dataclass), `page_map.py` (physical→printed page mapping, read per-page rather than assumed as a constant — verified correct and constant at +1 across all tested milestone pages: physical 0, 36, 37, 98, 100, 1496, 1497, plus correctly returns `None` for blank/title pages), `spans.py` (continuous cross-page span stream with column tagging), `io.py` (JSONL persistence), and `glyph_order.py` (the mandatory pre-ingestion sanity gate). - Added `pytest`/`pymupdf` to `ingestion/pyproject.toml` (previously empty `dependencies = []`) plus `[tool.setuptools.packages.find]` to fix a package-discovery ambiguity that broke `pip install -e .` — both confirmed via a real editable install, not just added and assumed to work. - **Corrected a real gap in ADR 0003's own validated finding**: re-verifying the "reversed glyph order" defect as real tested code (not trusted from the prior exploratory script) found **2 genuine occurrences, not 1** (physical pages 714 and 1373 — two different defect shapes, see outlier catalog item 9's rewrite for full detail). Getting a trustworthy count took 3 detector iterations after the first naive whole-book run reported 1113 false positives (kerning jitter + a column-boundary false-merge bug) — full false-positive history and the fix (group by PyMuPDF's own block index, not hand-picked x-coordinates) documented in `extract/glyph_order.py`'s docstring and the outlier catalog. - Smoke-tested `extract_spans`/`build_page_map` against the real PDF: 253,518 spans extracted, 30,728 bold, first monograph title (ABACAVIR) correctly located at physical page 100 / printed 101. - 9 unit tests added (`tests/test_extract_glyph_order.py`), all passing, including regression tests for the kerning-jitter and column-merge false positives found during validation (so they can't silently regress). **Not done yet / next up:** - Phase 1.2: `segment/` pure logic (vocab, merge, detector, atc, units) with unit tests reproducing every documented bug case (GONADOTROPIN wrap, part-divider false positive, ATC whitespace/O-0, "Chưa có" state) — see the approved plan (`ingestion/ingestion/segment/` is still an empty stub). - The 3 formula-region pages (92, 94, 805) that also trip `scan_reading_order` should **not** have their "corrected" text trusted — same guidance as outlier catalog item 8 (2D formulas aren't linearly recoverable); no auto-correction should be applied to those specifically, flag-only. --- ## 2026-07-30 — Eval strategy locked in; Phase 1.0 cheap surveys run **Done (direct requirement: "phải eval thật kỹ... phải có chiến lược rõ ràng" — plan mode used to design a full segmentation + eval framework before writing any real ingestion code):** - Designed and got user approval on a full implementation plan covering `extract/` + `segment/` + a `validation/` package, merging the already-validated ADR 0003 methodology (back-index recall, currently 91.7%) with a 6-point eval framework the user specified (visual diff, round-trip test, character-level text coverage, structure validation, golden dataset, downstream RAG eval) plus a follow-up list of domain-safety checks (adult/child dosing not mixed, mg/mcg/mmol units not corrupted, warning/contraindication sections captured, chemical formulas, header/footer leakage, page numbers not injected mid-paragraph). Full plan is preserved for reference; key decisions below are now the standing design, not just a plan-file artifact. - Confirmed target audience (doctors/pharmacists, not lay users — see `project_target_audience` memory) explicitly informs why domain-safety checks (dosing-population mixing, unit corruption) are being treated as first-class eval dimensions, not nice-to-haves. - Ran Phase 1.0 whole-book surveys (scratch script, not committed): - **Zero embedded images** across all 1668 pages (`get_images(full=True)`, measured) — image/caption validation tooling is not needed for this corpus. - **Adult/child dosing splits are the norm, not rare**: "Người lớn"/"Trẻ em"/"Trẻ sơ sinh" terms appear on 1121 of ~1400 monograph-range pages — elevates dosing-population-mixing to a standing validation check. - **Found and confirmed a real chemical reaction equation** (physical page 1033, cyanide-antidote mechanism: `Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3`) and a **new outlier**: the reaction arrow extracts as a Private-Use-Area glyph (`U+F0AF`), not a standard Unicode arrow — added as outlier-catalog item 16. A regex scan for chemical-formula-shaped tokens found 9 raw hits, 8 of which were false positives (flu-strain names, receptor names) — genuine chemical notation exists but is rare, not systemic. - Attempted to pin down the exact shortest monograph name+page, but the crude (unmerged, no multi-line-title-merge) scan script produced a **different longest-monograph ranking** than the already-documented one (previously: "AMOXICILIN VÀ KALI CLAVULANAT" at 45,623 chars; this script's top result was INSULIN at 41,799 chars) — flagged as unreliable rather than reported as fact, and explicitly deferred to Phase 1.2's real detector rather than trusting a quick script's number over the previously-validated one. Added to outlier catalog's "not yet investigated" list with the reasoning, not silently dropped. - Added outlier-catalog items 15 (no images), 16 (PUA reaction-arrow glyphs), 17 (adult/child dosing prevalence). **Not done yet / next up:** - Phase 1.1 onward: build real `ingestion/ingestion/extract/` and `segment/` modules (currently still empty stub packages) per the approved plan — `page_map.py` first, then `spans.py`/`glyph_order.py`, then the segment detector/merge/atc/units logic with unit tests, then wiring `cli.py run`, then the `validation/` package (back-index recall+precision, golden dataset, char-coverage/structure/domain-safety checks, visual-diff). See the approved plan file for the full phase breakdown and numeric targets (≥98% monograph recall/precision, ≥99% mean character coverage, zero-regression golden-set gate, manual visual-diff sign-off on hardest pages) if this session ends before implementation completes. - `pytest` and `pymupdf` need to be added to `ingestion/pyproject.toml` dependencies (currently `dependencies = []`) — confirmed both are already available in the global Python 3.12.10 env (PyMuPDF 1.28.0, pytest 7.4.4) but not yet pinned in the package's own dependency list. --- ## 2026-07-30 — Whole-corpus structural survey (not just anecdotes) **Done (direct pushback: "I feel like you're minimizing how complex this PDF really is — go find another 10-30 outliers, not just Vitamin D"):** - Built a real per-monograph structural survey across all 680 detected monographs (not 2 anecdotes) — computed ATC-code count, known-section count, and character length for every one. - **Multi-ATC monographs are NOT rare**: 173/680 (25.4%) have more than one ATC code — INSULIN has 20, BETAMETHASON and DEXAMETHASON 11 each, PREDNISOLON 10, HYDROCORTISON 9. The earlier "found 2 examples" framing badly understated this. Even 25.4% is a floor (see next point). - Investigated the 22 apparent "zero ATC" monographs (spot-checked 14): found **two distinct real causes of false negatives** — stray internal whitespace splitting an ATC code (`"J04A C01"` instead of `"J04AC01"`) and digit/letter confusion (`"NO3AX12"` instead of `"N03AX12"`) — 9 of 14 resolved as real ATC codes hidden by extraction noise (one of them, TRIAMCINOLON, turned out to have 5 ATC codes, meaning the true multi-ATC percentage is higher than 25.4%). The remaining ~5 genuinely say `"Mã ATC: Chưa có."` (not yet assigned) — a valid data state, not an error. - Found and confirmed a **false-positive monograph boundary**: the part-divider title "CÁC CHUYÊN LUẬN THUỐC" (Part 2's own section title, not a drug) was detected as if it were a monograph. - Measured real structural variance: monograph length ranges 2,331-45,623 characters (~20x spread), detected section count ranges 8-20. - All findings added to `docs/pdf-parsing-outlier-catalog.md` (items 12a revised with real numbers, 12c, 12d, 12e — new). - Verified one of my own debugging steps was itself wrong (read raw page text from the top instead of the correctly-bounded monograph segment, which briefly looked like a segmentation bug before being traced back to a debugging mistake, not a real defect) — corrected before reporting. **Not done yet / next up:** - Full-corpus re-count with the relaxed ATC regex (whitespace-tolerant, O/0-aware) not yet run — only 14/22 zero-ATC cases spot-checked, and the 173/680 multi-ATC count still uses the strict (undercounting) regex. - Phase 1 real implementation still pending overall (see earlier entries). --- ## 2026-07-30 — Confirmed class-level monographs and a real source typo **Done (direct follow-up: "have you checked drug-class entries like Vitamin D, or actual spelling/font-size errors?"):** - Found and confirmed a **second real example of a class-level monograph** covering multiple ATC codes/substances: "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ" (7 ATC codes, one per specific vitamin D analogue) — same pattern as the earlier GONADOTROPIN finding, confirming this is recurring, not a one-off. - Found and confirmed a **genuine spelling/capitalization typo in the source PDF itself**: the running header on this monograph's continuation pages reads "Vitamin d..." (lowercase d) vs the correct ALL-CAPS heading "VITAMIN D...". Verified via font/bbox inspection that this is a real source-text inconsistency, not an extraction artifact. The detection heuristic still worked correctly here (the typo'd header isn't all-caps so it's correctly rejected), but this was incidental, not a designed defense against typos. - Added both findings to `docs/pdf-parsing-outlier-catalog.md` (items 12a, 12b), with the general lesson: rely on multiple independent structural signals, not any single text match, since real source typos do occur. - Added `CLAUDE.md` with a standing rule: never fabricate or bluff a claim (number, test result, capability estimate) — verify before stating, explicitly flag estimates as estimates. Grounded in concrete incidents from this investigation (the size-threshold bug, the scope-gap bug). **Not done yet / next up:** - No systematic scan yet for *other* class-level (multi-ATC) monographs beyond the two found incidentally — Phase 1's data model should assume ATC code is a list per monograph regardless, rather than trying to enumerate every class-level entry in advance. - Phase 1 real implementation still pending overall (see earlier entries). --- ## 2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns) **Done (in response to direct follow-up questions about table/formula handling and full-book coverage):** - Found and confirmed a **table split across a page break loses its header on the continuation page** — real example: "Bảng 4" (ARV rash management table) ends with an orphaned, header-less data row on the next page when extracted with `pdfplumber`. - Found the **same header-loss risk also happens across a column boundary within a single page** (no page break needed) — real example: "Bảng 6". - Found and confirmed **2D grid/nomogram tables are not linearly recoverable** — the body-surface-area lookup table (appendix) extracts as scrambled bare numbers with no row/column association. - Found **two different formula-rendering outcomes**: a simple inline- exponent formula (Du Bois BSA) extracts cleanly as text; a stacked- fraction formula (Cockcroft-Gault) extracts as disordered fragments — confirmed the determining factor is 1D vs 2D visual layout, not "formulas are always broken." - Found and confirmed a **full-width table that breaks out of the normal two-column page grid** (bbox spans nearly the full page width). - Checked whether front-matter "committee list" pages are genuinely multi-column (the user suspected 3 columns) — confirmed via bbox inspection they are **not** true structural columns, just single wide text blocks with internal whitespace padding between names. - Consolidated **all** outlier findings from this investigation (this entry and the previous one) into a single, reusable, generalized reference: `docs/pdf-parsing-outlier-catalog.md` — written so it can guide parsing of other similarly-structured PDFs, not just this book. **Not done yet / next up:** - No automatic detector exists yet for (a) 2D-formula regions, or (b) 2D grid-table reconstruction — both flagged as open items in the catalog, not silently skipped. - Table-continuation re-attachment (page-break and column-break cases) has no implementation yet — needed before Phase 1 can trust any multi-row table content. - Phase 1 real implementation still pending overall (see previous entry). --- ## 2026-07-30 — PDF parsing strategy validated empirically (pre-Phase-1) **Done:** - Investigated the real PDF structure before writing any ingestion code (previous scaffold's assumptions about `doc.get_toc()` turned out wrong). - Confirmed: 1668 pages, no bookmark/outline (0 TOC entries), tagged-PDF structure tree exists but is too shallow to use (~29 elements only). - Cross-tested 3 extraction tools on real sample pages: PyMuPDF (correct reading order — kept as primary), pdfplumber (scrambled reading order on this layout — demoted to table-extraction-only use), opendataloader-pdf (correct reading order, useful independent font-metadata cross-check, but inconsistent heading classification — not trusted as sole signal). Docling install hit a numpy/pyarrow ABI conflict in the global Python env; tested in an isolated `.venv_docling_test/` (gitignored) instead of risking the global environment — see whether that resolved before relying on it. - Found the real structural ground truth: every section/monograph heading is a **bold font span** in the PDF (confirmed at the PyMuPDF span level AND independently by opendataloader's own font metadata — two tools agreeing). Font **size** is not reliable (10.0pt and 9.5pt both occur for genuine monograph titles) — an early size-based threshold silently dropped ~15% of real monographs; caught and fixed via whole-document validation, not spot-checking. - Found the real ground truth for validation: the back-of-book "Mục lục tra cứu" (page ~1528 onward) has exact page numbers per drug — much stronger than the front-matter drug list (which has no page numbers). Also found the book's own contents page states individual monographs run printed pages 99-1496 exactly. - Ran automated whole-document (1668-page, ~20-50s per run) validation against that page-verified ground truth: **91.7% recall** (665/725), with the remaining gap traced to one concrete, fixable cause (multi-line wrapped ALL-CAPS titles not yet merged across lines) rather than a flaw in the bold-span signal itself. - Documented the full methodology and results in `docs/adr/0003-pdf-parsing-strategy.md` and updated the ingestion section of `docs/architecture.md` to match reality (removed the incorrect TOC-preference assumption). **Also validated (in response to direct user questions about correctness):** - **No real duplicate drug monographs** found across the full 1405-page monograph range. The one apparent collision ("GONADOTROPIN" at 2 pages) is a detector artifact from the known multi-line-title bug (a different monograph's wrapped title fragment collided with it), not real content duplication. - **Confirmed the PDF is genuinely two-column** (bounding-box verified: left column x≈44-299, right column x≈308-562). PyMuPDF's reading order across columns is correct (already implied by earlier validation). - **Found and precisely characterized one real data-corruption defect**: a single text run on physical page 1373 has reversed (right-to-left) glyph order, producing scrambled text — confirmed by reversing the string, which recovers the correct Vietnamese sentence. A full scan of all 1405 monograph pages (grouping fragments into visual rows, checking for descending x-order) found this exact **1 occurrence and no others** — rare, isolated, but real, and now has a cheap (~16s) automated detector. - Full details, methodology, and exact numbers added to `docs/adr/0003-pdf-parsing-strategy.md` under "Follow-up validation." - **Caught a real scope gap**: the glyph-reversal scan above was initially run on the monograph range only (1405 of 1668 pages), leaving ~260 pages (front matter, appendices, back index) unchecked. Re-ran across the full 1668 pages: still exactly 1 defect (same page, 1373) — confirmed isolated, not hiding elsewhere. Also found 6 near-empty pages (3, 37, 99, 1495, 1497, 1666), all of which land exactly on major section-transition boundaries — intentional print blank pages, not lost content. **Not done yet / next up:** - Resolve/confirm docling status in the isolated venv (numpy/pyarrow conflict was fixed by using a separate venv; install completed — actual parsing comparison against the sample pages still pending). - Phase 1 real implementation: build `ingestion/` for real using the validated bold-span detector (not the exploratory scratch scripts) as one continuous cross-page stream (not per-page silos), fix the multi-line heading-merge gap, add the glyph-order sanity check as a mandatory pre-ingestion pass, re-run the validation script to confirm improved recall, then proceed to chunking + embedding + Qdrant upsert. - Decide and implement chunking strategy for the non-monograph parts of the book (general chapters pages 37-98, appendices 1497-1528) — needed so the full book (page 0 to last) ends up captured in the RAG corpus in some appropriate form, per the user's explicit requirement that no content be silently dropped. - Clean up exploratory `scratch_*` files from the repo root as they accumulate during investigation (routinely deleted after findings are persisted to docs — not left in git history). --- ## 2026-07-30 — Initial monorepo scaffold **Done:** - Designed the microservices architecture (see `docs/architecture.md`): Python/FastAPI `ai-service` for RAG, NestJS for `api-gateway`/`auth-service`/ `user-service`/`chat-service`, Next.js `web`, Qdrant for vectors, Postgres for relational data, Redis reserved for caching/queues. - Scaffolded the full monorepo directory tree (`apps/`, `packages/`, `ingestion/`, `infra/`, `docs/`) with baseline config (package.json/ pyproject.toml stubs, pnpm workspace, docker-compose topology stub). - Moved `duoc-thu-quoc-gia-viet-nam-2018.pdf` into `ingestion/data/raw/`. - Decided vector DB: **Qdrant** over pgvector (`docs/adr/0001-vector-db-qdrant.md`). - Decided deployment: GitOps via the **team's existing ArgoCD instance**, not a custom push-based CD pipeline (`docs/adr/0002-argocd-gitops.md`, `infra/argocd/`). CI's job is build/test/push image + bump the Helm values image tag; ArgoCD does the actual sync. - `git init` + initial commit (this scaffold). - Created a private GitHub repo (`BaoVu2k4/vsf-duocthu`, default branch `master`) and pushed the initial commit; fixed `targetRevision` in the ArgoCD Application manifests to `master` to match. **Not done yet / next up (Phase 1 of the build roadmap in `docs/architecture.md`):** - No business logic exists yet anywhere — this was scaffold only. - Phase 1: build the `ingestion/` pipeline for real (PDF extraction via PyMuPDF, monograph/section segmentation, section-aware chunking, OpenAI embeddings, Qdrant upsert) and validate retrieval quality via the `ingestion/notebooks/` QA step. - Still pending/TBD: which cloud provider (AWS/GCP/Azure) for Terraform (`infra/terraform/README.md`), and the team's ArgoCD instance's actual cluster/server + project details (`infra/argocd/README.md` TODOs).