Checkpoint frontend UI/UX overhaul and ingestion embed benchmark work

docs/progress-log.md

@@ -1,5 +1,660 @@
 # Progress Log
 
+## 2026-08-06 (cont. 11) — Real bug found by actually running the golden eval set: "thận trọng" silently answered as "chống chỉ định"
+
+Owner pointed at a golden dataset (`Golden Dataset/golden_e2e_v1.csv` +4
+more, 36-74 hand-authored cases each, dated 2026-08-04/05 — never run this
+session until asked). Ran the 36-case e2e set live end-to-end. Findings,
+graded against each case's own pass criteria:
+
+- **3/36 (8%) correct answers discarded to an empty abstain** by the F-01
+  entailment-noise issue already flagged as a known limitation — the golden
+  set turns that into a measured rate, not a hunch.
+- **2/36 wrong-section content gap, real bug, root-caused and fixed**: "X
+  cần thận trọng gì?" (asking precautions) was classified `attribute=
+  chong_chi_dinh` (contraindications) 9/9 times live-checked — the wrong
+  section entirely, silently dropping the actual precautions content (e.g.
+  metformin's lactic-acidosis warning, gentamicin's oto/nephrotoxicity) in
+  favor of contraindication text. Cause: the prompt gave the model a bare
+  `SECTION_KEYS` slug list with zero definitions — nothing to tell two
+  genuinely adjacent Vietnamese medical concepts apart. Fixed:
+  `rag/understanding.py` gained `SECTION_KEY_HINTS`, a short gloss per key
+  shown inline in the prompt, with `than_trong`'s explicitly stating it is
+  NOT `chong_chi_dinh` and naming the two example warnings that were
+  getting lost. Verified live: 3/3 reclassified correctly to `than_trong`
+  (metformin/gentamicin/ibuprofen), `chong_chi_dinh` questions unaffected,
+  and the two originally-broken answers now contain the exact required
+  content ("nhiễm toan lactic", "độc hại đối với cơ quan thính giác và
+  thận"). 2 new tests in `tests/test_understanding.py` (10 total, was 8).
+- **Several other gaps found, not code bugs**: `#26` ("nên tự tăng gấp đôi
+  liều?") and the "An toàn (Type 3)" block (`#21-25`) in the golden set
+  model a **lay-patient safety framework** (refuse + "hỏi thầy thuốc")
+  that directly contradicts the owner's explicit correction earlier this
+  same session — this product gates on scope (human/non-human), not on
+  "asks for a recommendation" (`[[feedback_no_recommendation_gate]]`). The
+  golden set predates that correction by two days; treating its Type-3
+  rows as ground truth would silently re-introduce the exact gate the
+  owner ordered removed. Flagged to the owner rather than "fixed."
+  `#13`/`#20` test the `/v1/rag/suggest` autocomplete flow but were driven
+  through `/v1/rag/query` by mistake — not a valid test of those two rows,
+  not rerun yet. `#14` vs `#15` (bare-name inconsistency), `#30` (price
+  question), `#35` (two-drug wording) are minor, not investigated further
+  today.
+
+`apps/ai-service`: **186 passed, 4 skipped**.
+
+## Status at end of today's session (accurate as of cont. 10 below)
+
+Codex's `CODEX_RAG_CODE_REVIEW_2026-08-06.md` correction order: **F-01
+through F-07, F-09 done; F-08 and F-10 done for their core finding, with
+named remainder.** Every completed item was live-verified against the real
+running server, not only unit tests — several real bugs were found *by*
+that live verification and fixed the same day, not just the ones the
+review named (grounding fallback removed per owner correction, F-03's
+`retrieve_framed` sending whole monographs, catalog-naming/id-form/
+weight-parsing bugs the owner's own UI test surfaced, F-06's exact overflow
+repro, F-08/F-09's Postgres connect-timeout hang).
+
+**Named remainder, next session's work:**
+- **F-08**: the Postgres-side unbounded-hang is fixed (`connect_timeout`),
+  but a real end-to-end deadline threaded through `RagAgent`'s own LLM
+  calls (understand → sufficiency → generate → up to 2 entailment retries,
+  up to 5 sequential Bedrock calls per request) does not exist — needs a
+  request-scoped budget object, a real design, not a bolt-on.
+- **F-10**: the core gap (RagAgent had zero test coverage and was not
+  provably the same dependency graph as the live HTTP service) is closed —
+  `tests/test_live_datastores.py::test_real_rag_agent_end_to_end_through_the_http_api`
+  drives the real `/v1/rag/query` endpoint, real `RagAgent`, real
+  `RetrievalService`/`QdrantRetriever` against a real temporary Qdrant
+  collection, and a real Postgres trace, asserting drug id, citation, and
+  decision — only the nondeterministic cloud model call is faked, since this
+  session's own live probing found real generation/entailment calls too
+  noisy for a regression assertion. **Not built**: the review's full
+  adversarial regression list (prompt injection, fake-drug-near-alias,
+  provider-timeout-and-outage behavior, `conversation_id` presence/absence
+  producing the same safety decision, etc.) — one solid end-to-end case
+  proves the wiring is real and testable; a comprehensive battery is a
+  larger, separate effort.
+- **`dosing_calc`** (a tested mg/kg calculator) and **`symptom_to_drug`**
+  (reverse indication→drug lookup) remain honest "not ready" clarifies —
+  deliberately not built under today's time pressure; see
+  `[[project_rag_rebuild_2026_08_06]]` on why rushing dosing math is the
+  wrong tradeoff.
+
+`apps/ai-service` full suite: **184 passed, 4 skipped** (the new
+integration test opts in via `RUN_INTEGRATION=1`, verified passing that
+way), up from 118 passed at the start of today's session.
+
+## 2026-08-06 (cont. 10) — F-10 core done: RagAgent proven live-testable end to end, not just live-tested by hand
+
+Every F-01–F-06/F-08/F-09 live verification this session was a one-off
+Python script run by hand against the real Qdrant/Bedrock/Postgres — real
+evidence, but not a regression a future change would automatically re-run.
+F-10 closes that: `tests/test_live_datastores.py` gained
+`test_real_rag_agent_end_to_end_through_the_http_api`, following the
+existing `RUN_INTEGRATION=1`-gated pattern in that file (temporary Qdrant
+collection seeded with one real corpus chunk, real Postgres migration +
+trace round-trip).
+
+What's real in this test: `RagAgent`, `LlmQueryUnderstander`,
+`RetrievalService`, `QdrantRetriever`/`QdrantParentStore` against a live
+Qdrant, `GroundedAnswerService`, `PostgresTraceRepository` against a live
+Postgres, and the actual `/v1/rag/query` FastAPI route via `TestClient` —
+the identical object graph `bootstrap.build_runtime` wires in production.
+What's faked: only the LLM boundary (`_FakeJsonLlm`, satisfying both the
+`JsonLlm` and `AnswerGenerator` protocols with fixed payloads keyed by
+schema shape) — deliberately, not for convenience: this session's own live
+probing (F-01's entailment noise, F-03's non-deterministic generations)
+found real cloud calls too noisy to assert exact drug id / citation /
+decision against reliably. Asserts (Codex's exact F-10 list): resolved drug
+id, citation chunk id and printed page, decision, and that the trace
+persisted and reads back correctly.
+
+Verified passing with `RUN_INTEGRATION=1` (4/4 in that file) and correctly
+skipped by default (184 passed, 4 skipped without it — no cost/flakiness
+added to the normal suite run).
+
+**Scope, stated plainly**: this is the load-bearing first case proving the
+production path is real and mechanically testable, not the comprehensive
+adversarial battery the review sketched (prompt injection, fake-drug-near-
+alias, provider outage/timeout behavior, `conversation_id` presence/absence
+parity, multi-population-band evidence, etc.). Extending this one case into
+that full battery is real remaining work, not done today.
+
+## 2026-08-06 (cont. 9) — F-09 done (trace fail-open), F-08 partially: a real unbounded-hang found live and fixed
+
+**F-09.** `routers/rag.py` called `traces.save()` synchronously before
+returning a response; `PostgresTraceRepository.save()` opened a fresh
+connection per call with no error handling, so a Postgres outage turned an
+already-computed, safe answer into a 500 for a reason unrelated to whether
+the answer was safe. Made an explicit fail-open decision (tracing is
+observability, not the product): the router now wraps the `save()` call,
+falls back to a locally-generated `trace_id` on any exception, and counts
+it (`duocthu_trace_write_failed_total`, a new metric — a silent fail-open
+with nothing to page on is indistinguishable from tracing quietly working).
+Connection pooling (the other half of the original finding) not done —
+real pooling needs startup-time lifecycle wiring, out of scope for today.
+
+**F-08, live-verified, not fully scoped.** Testing F-09 by pointing
+`POSTGRES_DSN` at an unreachable host live surfaced a sharper bug: a bare
+`psycopg.connect()` with no `connect_timeout` hangs on the OS-level TCP
+timeout (tens of seconds) when the DB is unreachable but not *actively*
+refusing — which defeats the F-09 try/except just as completely as no
+try/except at all, since the exception it's waiting for doesn't arrive in
+time. Added `connect_timeout=5` to every `psycopg.connect()` call in
+`adapters/postgres.py`. Verified live: same broken-DSN repro that
+previously hung past a 30s client timeout now returns 200 with the correct
+grounded answer in ~14.5s (5s bounded connect attempt + normal generation
+latency). The broader F-08 ask — an end-to-end request deadline threaded
+through every provider call — is **not done**: `TurnBudget`
+(`rag/reasoning.py`) exists but belongs to the old `ConversationalLoopService`
+path, which F-03 stopped constructing live; the new `RagAgent` path (up to
+5 sequential Bedrock calls per request: understand, sufficiency, generate,
+up to 2 entailment retries) has no budget object at all, bounded only by
+each individual call's own fixed read_timeout (30-60s each). A real fix
+needs a request-scoped deadline object passed into `RagAgent`/
+`GroundedAnswerService` and consulted before each call — a genuine feature
+to design, not something to bolt on safely in the time remaining today.
+
+`apps/ai-service`: **184 passed, 3 skipped**.
+
+## 2026-08-06 (cont. 8) — F-05 done: startup refuses a corpus/model manifest mismatch, live-verified both ways
+
+The ingestion loader already writes a sidecar manifest (`<collection>
+__manifest`, one point: corpus SHA, chunk count, embedding model_id,
+dimensions) recording what a collection was built from
+(`ingestion/ingestion/load/manifest.py`). Nothing on the ai-service side
+ever read it — two unrelated embedding models can both produce
+1024-dimensional vectors, and Qdrant returns plausible-looking but
+meaningless nearest neighbours with no error at query time.
+
+Added `rag/manifest.py` (`check_manifest` — pure, 6 unit tests) and wired
+`bootstrap.py::_verify_corpus_manifest` to call it right after the query
+embedder is constructed, before anything else. `main.py` builds the runtime
+at import time, so a mismatch crashes startup — the service never comes up
+against a corpus it wasn't verified against, rather than silently serving
+degraded search.
+
+Hit a real API mismatch immediately (pytest collection caught it, since
+`test_api.py` imports `main.py`, which calls `build_runtime` against the
+live Qdrant): this qdrant-client version has no `collection_exists`, and
+`get_collection` is a known parse-bug risk in this environment (per
+`reference_env_operational_gotchas`) — switched to `get_collections()` +
+membership check instead. **Live-verified both directions**, not just unit
+tests: the real collection's manifest (`model_id=cohere.embed-v4:0,
+dimensions=1024`) matches the configured embedder and the server starts and
+answers correctly; a monkeypatched `embedding_dimensions=768` against the
+same real manifest correctly raises `ManifestMismatch` before any query
+path is reachable.
+
+`apps/ai-service`: **183 passed, 3 skipped**.
+
+## 2026-08-06 (cont. 7) — F-06 done: the overflow-before-truncation bug, exact repro fixed
+
+`ConversationState.append()` truncated `recent` to the window immediately;
+`overflow()` then checked `len(self.recent) > window` on the *already-
+truncated* tuple, which can never be true. Codex's exact repro (8 turns into
+a window of 6: `recent=6, turn_count=8, overflow=0`) reproduced first,
+unchanged from the review.
+
+Fixed: `ConversationState` gained a `pending_overflow` field. `append()`
+computes what it evicts *before* truncating and accumulates it there
+(accumulates, not overwrites — a live turn calls `append()` twice in a row,
+user then assistant, and the second call must not lose what the first
+evicted). `overflow()` now just returns `pending_overflow`. The caller
+clears it (`replace(state, ..., pending_overflow=())`) after folding into
+the summary, or the same turns fold again next cycle —
+`ConversationalLoopService._persist` (the live path) updated to do so;
+`ConversationalRagService._persist` already reconstructs `ConversationState`
+directly without passing the field through, so it already clears by
+construction.
+
+Verified the exact repro now returns the 2 actually-dropped turns instead
+of `()`. 6 new tests in `tests/test_conversation.py`. **Not done, out of
+scope for the remaining time today:** the second half of the original F-06
+finding — `InMemoryConversationStore` loses all state on restart and
+diverges across multiple workers. That needs a shared (Postgres-backed)
+store, a real infra addition, not a bug fix; not attempted under today's
+time pressure rather than risk a rushed, unverified persistence layer.
+
+`apps/ai-service`: **177 passed, 3 skipped**.
+
+## 2026-08-06 (cont. 6) — F-04 done: drug candidates bounded deterministically before the LLM picks, live-verified
+
+`rag/understanding.py::LlmQueryUnderstander` used to show the model the
+*entire* ~684-drug catalog every turn and trust any id it returned as long
+as that id existed somewhere in the catalog (Codex's F-04 finding: catalog
+membership proves the output is *some* real drug, not that it's the one the
+user's text actually named — an LLM could satisfy that whitelist while
+mapping an unrelated/invented name to a different real drug).
+
+Reworked: `LlmQueryUnderstander` now takes a `resolver` (the existing
+`CatalogDrugResolver`, already built in `bootstrap.py` for autocomplete) and
+computes a deterministic **candidate set** from the turn + raw history text
+*before* calling the LLM — exact alias matches plus a generous fuzzy
+`suggest` pass (min_score=0.55, well below the resolver's own 0.84
+auto-answer threshold, since the goal here is only to rule out drugs
+nothing in the conversation plausibly refers to). Only that candidate
+subset (not the full catalog) is shown to the model, and the model's pick
+is validated against it — a real id the model names that isn't among the
+turn's candidates is now treated as unknown, not trusted on catalog
+membership alone. Also directly closes a separate prompt-cost finding from
+the same review (sending the full catalog every turn is unbounded token
+cost) since the shown block is now per-turn-sized, not fixed at ~684 rows.
+
+`tests/test_understanding.py` extended (was 0 tests before this session,
+per Codex's F-10 finding; now 8): covers exact-form and spaced-form
+resolution, a genuinely invented name staying unknown, **a real catalog id
+that has no deterministic candidate support still being rejected** (the
+core F-04 guarantee — catalog membership alone is not enough), and a fuzzy
+typo still resolving through `suggest`.
+
+**Live-verified**, not just unit-tested: `aspirinol` (fake) still correctly
+abstains out-of-scope; `amoxicillin` (correct INN spelling, a typo-adjacent
+case) still resolves to `amoxicilin`; `metformin` and the 3-turn paracetamol
+pediatric-dose conversation from the owner's own UI test both correctly
+keep the same `resolved_drug_id` across every turn. No latency regression
+observed (smaller prompt, same ~3-9s range dominated by generation, not
+catalog size).
+
+`apps/ai-service`: **174 passed, 3 skipped**.
+
+## 2026-08-06 (cont. 5) — Three more live bugs found from the owner's own UI test of F-03, all fixed
+
+Owner drove the real web UI (not curl) through a multi-turn pediatric dose
+question and hit a severe regression: "Liều paracetamol cho trẻ em" -> two
+clarify rounds (age, then weight) -> final turn answered "Không tìm thấy
+paracetamol trong Dược thư Quốc gia Việt Nam" for a drug that plainly is in
+it. Root-caused and fixed three distinct bugs in the F-03 wiring, in order:
+
+1. **`_catalog_names` (bootstrap.py) could bury a drug's own name.** It
+   picked the first 3 aliases *alphabetically* per drug to show the LLM
+   understander. Paracetamol has 191 aliases (mostly trade names); the
+   alphabetically-first 3 were "0Frezefev, ABAB, Ace kid 80" — no
+   recognizable name at all. Mid-conversation, once the drug is no longer
+   restated in the raw turn text, the model has only history + this catalog
+   line to re-derive it from; with nothing recognizable shown, it read
+   "paracetamol" as an unknown name. Fixed: always show the drug_id's own
+   name form (`drug_id.replace("_"," ")`, guaranteed present) first, then
+   fill remaining slots preferring short ALL-CAPS aliases (the book's own
+   heading convention, usually the generic name) over dosage-suffixed brand
+   names. `tests/test_bootstrap.py` (new, 4 cases).
+
+2. **That fix immediately exposed a second bug.** With the display name now
+   near-identical to the drug_id ("paracetamol acetaminophen" vs.
+   "paracetamol_acetaminophen"), the model started echoing the *spaced*
+   display form instead of the underscored id, and
+   `LlmQueryUnderstander._parse()`'s strict `d in self._ids` check demoted
+   a correctly-identified drug to `unknown_drugs` — same user-visible
+   failure, different cause. Fixed: `_resolve_id()` accepts either the exact
+   id or its space-substituted form (a deterministic, lossless formatting
+   tolerance — not fuzzy matching, no risk of resolving to an unrelated
+   drug). `tests/test_understanding.py` (new, 7 cases — this module had
+   zero coverage before today, per Codex's F-10 finding).
+
+3. **"30 cân" (colloquial Vietnamese for "30 kg", no unit word) wasn't
+   reliably read as a weight.** Confirmed live: the model missed it
+   entirely in some runs, silently re-asking for weight the user had just
+   given. Added an explicit rule + schema hint that a bare number + "cân"/
+   "ký" means kilograms. Verified live: 3/3 clean extractions after the fix
+   (was inconsistent before).
+
+All three verified against the real running server with the owner's exact
+repro sequence, not just unit tests — final state: the drug (`resolved_drug_id
+= paracetamol_acetaminophen`) now stays correctly attached across all three
+turns, and weight is correctly captured. **Not fixed, deliberately, already
+flagged (F-07):** `dosing_calc` still doesn't compute an actual mg dose once
+enough information is gathered — it falls through to ordinary section
+retrieval (the clinician sees the dosing table, not a calculated number). A
+weight-based calculator is a real feature to build, not a wiring bug; out of
+scope for this pass.
+
+Also, per owner UX feedback, warmed up the static smalltalk reply (was a
+terse "Chào anh/chị. Tôi tra cứu... Anh/chị muốn hỏi về thuốc nào?").
+
+`apps/ai-service`: **173 passed, 3 skipped** (was 162 at the end of the F-03
+entry below).
+
+## 2026-08-06 (cont. 4) — F-03 done: RagAgent wired into the live server, two real bugs found and fixed by driving it
+
+Wired the new LLM-understanding orchestrator (`rag/agent.py` + `rag/
+understanding.py`, built last session but never called by anything live —
+Codex's exact F-03 finding) into `bootstrap.py`/`routers/rag.py`. Both
+single- and multi-turn requests now go through one path:
+`RagAgent.handle()`. The old `CatalogDrugResolver`/`QueryRoutingService`/
+`ConversationalLoopService` stack stays in the codebase (still unit-tested,
+still used for autocomplete + the no-generator-configured fallback) but is
+no longer constructed as the live query path — per Codex, full deletion
+waits on a production-path parity suite (F-10), not done yet.
+
+Added the coverage that didn't exist: `tests/test_agent.py` (14 cases —
+`RagAgent` had zero tests before this), `tests/test_retrieval_service.py`
++2 for `retrieve_framed`, `tests/test_api.py` +4 for the router's agent
+branch. 162 passed, 3 skipped.
+
+**Drove the actual running server** (per house rule: never claim a wiring
+change works from unit tests with fakes alone) and found two real bugs unit
+tests couldn't have caught:
+
+1. **`retrieve_framed` had no bare-name/overview case.** `retrieve()` (the
+   old path) always answered a bare drug name from four identity sections
+   only; `retrieve_framed` had no equivalent and always fetched the entire
+   ~29-section monograph, then relied on rerank to trim it — silently
+   sending the whole book as evidence whenever rerank was off or failed
+   open. Live symptom: asking bare "paracetamol" abstained empty every
+   time (answer too long, generation intermittently malformed). Fixed:
+   `retrieve_framed` gained an `is_overview` parameter (driven by the
+   frame's `turn_type == "drug_overview"`), mirroring the old intro-only
+   behavior, and the non-overview rerank branch is now capped at
+   `evidence_limit` even when rerank fails open — an ordering aid failing
+   open must not also remove the size bound. Verified live: 3/3 clean
+   answers after the fix, none of the prior empty-abstain failures.
+
+2. **The entailment judge (added this session, F-01) is noisier than one
+   call suggests.** Same claim/evidence pair, called repeatedly, disagreed
+   with itself — confirmed live on the warfarin/aspirin interaction case,
+   which correctly cites a drug-interaction list evidence block but got
+   rejected 0/2, 1/2, then 3/3 across separate live batches. Added a
+   same-claim retry (`GroundedAnswerService._verify_entailment`): a lone
+   reject retries once, only two agreeing rejects discard the generation.
+   Also sharpened the entailment prompt to explicitly call out dense
+   comma-separated drug-interaction lists, since the specific failing claim
+   named a drug buried mid-list. Owner explicitly capped further spend
+   here (more retries = more tokens for a narrowing edge case) — the
+   retry/prompt change did not fully eliminate this one case in further
+   live testing (still failed 3/3 in the last batch), and it was
+   deliberately **left as a known, safe-direction residual limitation**
+   rather than chased further: the failure mode is abstain (never a
+   fabricated interaction claim), not wrong output. Documented in
+   `_verify_entailment`'s docstring; a cleaner fix (e.g. breaking a
+   multi-drug interaction claim into a per-drug comparison instead of one
+   long prose evidence block) is a good candidate for a future pass, not
+   solved today.
+
+Also fixed a mismatched piece of the wiring in `apps/web/app/api/chat/
+route.ts`: it discarded `RagAgent`'s specific abstain messages (e.g. "Không
+tìm thấy X trong Dược thư") in favor of a generic fallback, because it only
+consulted `answer` when `decision !== "abstain"`. Now prefers `rag.answer`
+whenever it is non-null, regardless of decision.
+
+## 2026-08-06 (cont. 3) — Investigated "684 vs 700+24 expected" monograph-count question: zero real drug monographs missing, gap is 100% explained
+
+Owner asked why the corpus has 684 monographs when the expectation was
+~700 drug monographs + 24 general-chapter monographs. Did not rely on any
+number already sitting in memory/docs — re-ran `ingestion.cli validate`
+live against the real PDF this session to get a current ground-truth
+comparison, per [[feedback-rigorous-validation]] / [[feedback-verification-ladder]]
+("recompute every number before quoting it").
+
+**Live re-run result** (`python -m ingestion.cli validate --pdf
+data/raw/duoc-thu-quoc-gia-viet-nam-2018.pdf`):
+```
+detected monographs: 684
+ground-truth entries: 705   (parsed from the book's own back-of-book index,
+                              "Mục lục tra cứu", pages 1529+)
+recall:    96.2% (678/705)
+precision: 99.1%
+```
+27 ground-truth entries didn't match a detected monograph, and 6 detected
+monographs didn't match a ground-truth entry. Pulled the **full** unmatched
+list (the CLI only prints the first 20 of 27) via a direct Python call
+into `ingestion.validation.back_index`/`metrics` — classified all 27 by
+hand:
+
+| Category | Count | Detail |
+|---|---|---|
+| Part 1 general-chapter titles (printed pp. 39-95) | 20 distinct (21 lines — "Thuốc chống loạn thần..." p.75 is duplicated in the book's own index) | Hướng dẫn sử dụng(39), Kê đơn thuốc(40), người cao tuổi(41), suy gan/thận(43), trẻ em(45), thai kỳ/cho con bú(47), giảm đau(48), hen phế quản(51), kháng động kinh(55), kháng HIV(61), kháng sinh(70), cephalosporin(72), chống loạn thần×2(75), lao(77), viêm gan B(80), ADR(83), dị ứng thuốc(85), ngộ độc(90), dược động học(93), tương tác thuốc(95) |
+| Part 3 appendix titles | 2 | BSA calc (1497), pha thuốc tiêm IV (1498) |
+| Part 2 drug names | 4 | Alphatocoferol(165), Benzoyl peroxyd(246), Hydrogen peroxyd(781), Tretinoin (thuốc uống)(1405) |
+
+**Then checked all 4 remaining "drug" entries directly against
+`ingestion/data/processed/monographs.jsonl`** (not just assumed) — all 4
+are already present in the corpus, under a differently-spelled
+`drug_name`:
+- Alphatocoferol → `ALPHA TOCOPHEROL (Vitamin E)` (tocoferol/tocopherol)
+- Benzoyl peroxyd → `BENZOYL PEROXID` (peroxyd/peroxid)
+- Hydrogen peroxyd → `HYDROGEN PEROXID` (peroxyd/peroxid, same pattern)
+- Tretinoin (thuốc uống) → `TRETINOIN (UỐNG)`
+
+These 4 also account for 2 of the "6 unmatched detected monographs"
+(HYDROGEN PEROXID, TRETINOIN (UỐNG) both show up on both sides of the
+diff — same monograph, name-matching miss in `validation/metrics.py`'s
+`_names_match`, not two different problems).
+
+**Conclusion, fully closed, no open unknowns left:**
+1. **Zero Part 2 (drug) monographs are actually missing.** Every
+   ground-truth drug entry in the back-index resolves to something already
+   in the 684. The apparent gap was a validator string-matching artifact
+   (Vietnamese `-yd` vs. English `-id`/`-pherol` spelling variants), not
+   missing content. `684` is the correct, complete count for Part 2.
+2. **The "24 general chapters" are 0/24 present** — confirmed only 20
+   distinct chapters exist in the book's own index (not 24; owner should
+   double check where the "24" figure came from), and none of the 20 are
+   extracted, because the pipeline was scoped to Part 2 only from the
+   start (`extract`/`segment`/`assemble` never touch printed pp. 37-98).
+   This matches the already-known, already-documented scope gap in
+   [[project-rag-rebuild-2026-08-06]] / `reference_duoc_thu_2018_structure`
+   memory — not a new discovery, just re-confirmed live.
+3. The book's own front-matter "~700 substances" figure is the
+   publisher's approximate active-substance count, not a strict
+   heading-count promise — some monographs bundle multiple substances
+   under one heading (INSULIN = 20 ATC codes / salts under 1 monograph,
+   the HMG-CoA-reductase-inhibitor class monograph, ARGININ's 2 salts),
+   so a smaller heading-count than 700 is expected and consistent with
+   full coverage, not evidence of missing data.
+
+**Not done / possible follow-up (not requested this session):** the 6→4
+`_names_match` misses above suggest a small, mechanical fix (normalize
+`-yd`↔`-id`/`-pherol` diacritic-free spelling variants, or add explicit
+alias pairs) would push CLI-reported recall from 96.2% to ~99.7% without
+touching extraction at all — cosmetic (metric accuracy), not a data-quality
+fix, since the underlying monographs already exist either way. The 4
+remaining truly-unmatched-detected entries (CARBIDOPA-LEVODOPA, THUỐC
+PHIỆN-OPIAT-OPIOID, VẮC XIN DPT, VẮC XIN MMR) are compound/hyphenated-name
+matching gaps in the same function, same category, not investigated
+further this session.
+
+## 2026-08-06 (cont. 2) — Owner correction: no fallback to raw source text when a generator is configured; F-02 scoped down to subject_scope only
+
+Two corrections from the owner mid-F-02, both applied immediately:
+
+**1. Dropped intent-based recommendation gating entirely.** Built a keyword
+detector for `QueryIntent.RECOMMENDATION` ("nên dùng thuốc gì" etc.) as part
+of F-02's server-side policy derivation — wrong call, reverted same session.
+**This product is for doctors and pharmacists** (`[[project_target_audience]]`),
+and a clinician asking "thuốc nào tốt nhất cho bệnh nhân suy thận" is normal,
+in-scope use of a formulary reference, not a request to abstain on. `rag/
+policy.py` now derives `subject_scope` only (veterinary/non-human keyword
+check — a corpus-coverage fact, not a restriction on clinical questions);
+`routers/rag.py` passes `intent` through from the caller unchanged, same as
+before F-02. `tests/test_policy.py` scoped down to match.
+
+**2. Removed the extractive-fallback safety net for a CONFIGURED generator
+that fails.** Previously, any generation failure — provider outage, malformed
+JSON, `grounding.verify` rejection, entailment rejection — fell back to
+quoting the retrieved evidence verbatim ("the source is always available
+because it was computed first"). Owner: that raw citation-stapled paragraph
+is the retired offline-extractive product shape (`[[project_llm_cloud_plan]]`
+— "owner wants a REAL LLM chatbot... not the offline extractive build"), and
+must not reappear as a silent degradation path now that generation is live.
+
+`GroundedAnswerService.answer_from_result` (`rag/answer.py`) now branches on
+whether a generator is configured at all, not just on whether this call
+produced one:
+- **No generator configured** (`ANSWER_PROVIDER=disabled`, the default) is
+  unchanged — a deliberate, fully-supported retrieval-only mode, still quotes
+  the source.
+- **A generator IS configured** and this generation failed any check → the
+  turn **abstains** (`decision=ABSTAIN, reason="generation_unavailable"`,
+  `answer=None`), never a raw source dump.
+
+Updated 9 tests across `test_grounded_generation.py` and
+`test_citation_and_intro.py` whose assertions encoded the old fallback
+behavior (`grounded.answer.startswith(EVIDENCE_TEXT)` → `grounded.answer is
+None` + `decision == ABSTAIN`). Live-verified the happy path still works
+unchanged against the real model (Qwen3/Bedrock Converse, ~3.5s, served
+correctly) — this change only touches the failure branch.
+
+`apps/ai-service`: **140 passed, 3 skipped**.
+
+## 2026-08-06 (cont.) — F-01 fixed: grounding verifier no longer trusts a global number pool or an uncited claim
+
+Codex's code-only review (`coordination/CODEX_RAG_CODE_REVIEW_2026-08-06.md`)
+reproduced three ways `rag/grounding.py::verify` let an unsafe generated
+answer through. Reproduced all three locally first, byte for byte, before
+touching code — all three real. Working through the review's proposed
+correction order (F-01 → F-02 → ... → F-10; tracked as tasks #1-#8).
+
+**F-01, done.** Two independent fixes, both proven live (Qwen3 via Bedrock
+Converse), not just against a fake generator:
+
+1. **Per-citation binding, not global pool.** `verify` used to pool every
+   number from every evidence block into one set and check answer numbers
+   against that pool — so a number true of block 2 passed under a citation
+   to block 1 (`so_sai_nguon`). Rewrote to split the answer at each `[n]`
+   citation group and check only the block(s) that group names.
+2. **Citation required for every claim.** A citation-less generated answer
+   used to pass silently as long as it stated no number the pool didn't
+   already contain (`khong_citation`) — trivially true when the answer had
+   no numbers at all. Now any substantive claim with no valid citation is
+   rejected (`uncited_claim`). This also kills the old "attach every
+   retrieved citation when the generated text cites nothing" fallback in
+   `GroundedAnswerService`: that code path is now unreachable, since
+   `grounding.verify` rejects the citation-less generation before it gets
+   there — the extractive fallback (which always cites everything by
+   construction) takes over instead.
+3. **Entailment gap (`claim_bia`) — regex can't see meaning.** A fabricated
+   nonnumeric claim with a syntactically valid citation ("Metformin chữa
+   ung thư [1]" citing a block about đái tháo đường) still passed both
+   fixes above: no number, citation in range. Closed with a second LLM
+   call (`GroundedAnswerService._verify_entailment`, `rag/prompt.py`'s
+   `build_entailment_request`) that runs after `grounding.verify` passes:
+   each substantive cited claim, checked only against the evidence block(s)
+   it names, judged by a model told to compare wording, not reason about
+   medicine. Fails closed (provider outage/malformed JSON → reject, not
+   accept). **Live-verified against the real model**, not simulated: ran
+   the actual entailment prompt through `BedrockConverseAnswerGenerator`
+   (Qwen3) on `claim_bia`, a fabricated contraindication, a faithful claim,
+   and a legitimate paraphrase — correctly rejected the two fabrications
+   (`entailed: false`) and passed the two honest ones (`entailed: true`,
+   including the paraphrase, so it isn't just penalizing rewording). Also
+   ran the full `GroundedAnswerService` pipeline live end-to-end (real
+   generator, real multi-call sequence) on a legitimate metformin dose
+   question — served correctly, ~3.4s.
+
+`apps/ai-service`: **134 passed, 3 skipped** (was 118p/3s before this
+session; added `tests/test_grounding.py` — 12 adversarial cases — plus 4 new
+entailment-path cases in `tests/test_grounded_generation.py`, and updated 3
+existing tests whose assertions encoded the old, buggy behavior).
+
+**Known residual limit**, stated in `rag/grounding.py`'s docstring: the
+entailment LLM call is itself a model judgment, not a proof — it is a real
+improvement over zero semantic check, not a formal guarantee. F-02 through
+F-10 (scope/intent server-side enforcement, wiring the new
+`RagAgent`/`LlmQueryUnderstander` orchestrator that's currently dead code,
+bounding entity candidates, manifest validation, conversation overflow bug,
+request budgets, trace failure policy, production-path regression suite)
+are next, in that order — none touched yet this pass.
+
+## 2026-08-06 — RAG rebuild started: live failure diagnosis + LLM query-understanding front-end (replacing the brittle resolver)
+
+Owner reported the live chatbot "cực ngu, sai gần hết" and asked to rebuild the
+RAG from scratch (incl. chunking). Per the never-fabricate rule, drove the REAL
+running service before designing.
+
+**Stack brought up live** (all local, $0 to load): Qdrant `duocthu_v1` already
+held 15,100 pts @1024-dim (green); Postgres up; ai-service :8079 running with the
+cloud-live `.env` (cohere-v4 embed + qwen3 generation + Cohere rerank).
+
+**Live diagnostic battery (~20 hard VN questions, real `POST /v1/rag/query`).**
+Finding, evidence-backed: it is NOT "sai hết" and the culprit is NOT chunking —
+when a single drug resolves cleanly the answer is grounded and correct
+(paracetamon typo ✓, metfomin typo ✓, multi-turn "nó dùng cho trẻ em" inherited
+metformin ✓). The failures cluster in the **query-understanding / drug-resolution
+front-end** (the `CatalogDrugResolver` fuzzy `SequenceMatcher` + keyword
+`SectionResolver`):
+- `aspirinol` (fake drug) fuzzy-matched to aspirin and ANSWERED — a safety bug.
+- `amoxicillin` (correct English INN) tied/ambiguous → abstained; the sentence
+  word "uống" polluted fuzzy scoring (matched `tretinoin_uong`).
+- `warfarin với aspirin` (interaction) → ambiguous → abstain; no interaction path.
+- `còn liều dùng thì sao?` follow-up lost the drug (inconsistent inheritance).
+- `trẻ 5 cân paracetamol` → clarifies forever; no mg/kg weight-based calc node.
+- symptom→drug and BSA/Part-1/Part-3 → abstain (scope gaps).
+
+**Corrected an earlier overstatement (owner was right):** section chunking is NOT
+uniform — 172/684 monographs (25%) are class monographs cramming many sub-drugs
+into one section (INSULIN dose = 9,268 chars / 20 ATC, VITAMIN D 14,197 chars),
+chunked by blind token-window. So re-chunk (sub-drug/population/indication-aware)
+IS warranted later — but it does not fix the front-end failures above.
+
+**Rebuild step 1 — LLM query-understanding front-end (new, PROVEN live).**
+`apps/ai-service/rag/understanding.py`: `LlmQueryUnderstander` + `QueryFrame`.
+One LLM call reads the messy turn (+ history + the real 684-drug catalog) → a
+structured frame (turn_type, drugs [catalog-validated], unknown_drugs, attribute,
+population, weight_kg, indication). Safety kept: the model may only pick drug_ids
+from the real catalog; an unrecognised name goes to `unknown_drugs`, never snapped
+to a near drug. `rag/` stays SDK-free (LLM injected as a `JsonLlm` protocol,
+satisfied by the existing `BedrockConverseAnswerGenerator`). Proven on the live
+LLM against all 7 killer cases the old resolver failed — every one now read
+correctly (amoxicillin→amoxicilin, aspirinol→unknown, warfarin+aspirin→interaction
+with both drugs, trẻ 5 cân→dosing_calc weight=5.0, sốt cao→symptom_to_drug,
+follow-up→inherited metformin, chào→smalltalk).
+
+**NOT yet done:** the frame is not wired into retrieval/generation — the old
+`CatalogDrugResolver`/`SectionResolver` still drive `/v1/rag/query`. Next: route on
+`turn_type` (interaction→gather both drugs; symptom_to_drug→reverse `chi_dinh`
+lookup; dosing_calc→a tested mg/kg calculator like `rag/calculators.py`), unit +
+live eval vs the battery, then decide the structure-aware re-chunk (needs owner GO
+for re-embed ~$0.5). No re-embed or cloud spend beyond cents of diagnostic/proof
+LLM calls this session.
+
+## 2026-08-05 (night) — Live-chat UX overhaul: reasoning/clarify, multi-turn, Qwen3; plan = finish chatbot tomorrow, deploy next week
+
+Owner drove the running web chat with messy real inputs and found the offline-era
+query layer was a hodgepodge. Fixed the failures found, each **verified by
+chatting the running service** (not just unit tests). Model switched to
+**qwen.qwen3-next-80b-a3b** (DeepSeek ignored the clarify instruction; Qwen3 and
+gpt-oss both follow it — A/B'd). ai-service **118 passed, 3 skipped**.
+
+Fixed (commits `6c6a916`, `5feccba`, `553be09`, `7f45d06`):
+- **Reasoning/clarify (the headline):** a focused sufficiency-check LLM call runs
+  BEFORE generation. An under-specified dose ("paracetamol cho trẻ em") now ASKS
+  age/weight/route/indication instead of dumping every band. Adult dose / CCĐ /
+  interactions answer normally (no false clarify). `answer._check_sufficiency` +
+  `prompt.build_sufficiency_request`; `GroundedAnswer.clarification` → decision
+  "clarify".
+- **Multi-turn:** "thuốc đó…" was double-resolved (inherited then re-resolved
+  from rewritten text → ambiguous → empty). Now the resolved drug_id is passed
+  straight to retrieval (`routing.retrieve_for_drug`); raw turn drives section
+  routing; overview+rerank finds the part. Verified: Oxymetazolin → "thuốc đó cho
+  trẻ dưới 6 tuổi?" → correct than_trong answer.
+- **Did-you-mean garbage:** fuzzing a sentence ("EPO…") or "đúng" returned
+  terbinafin/tretinoin in a loop. Now suggestions only for short drug-name misses;
+  confirmations get "which drug?".
+- **Bare name → drug intro** (class + indication + invite), not a forms dump.
+- **Citations = only the [n] actually cited** (was ~13 chips for a 1-source line).
+- Rerank trims overview 29→6; inherited-drug notice uses the display name.
+
+**Operational lesson (cost real time):** `uvicorn --reload` does NOT work on this
+Windows box — the owner chatted STALE servers repeatedly. Must kill :8079 and
+restart after every edit. Recorded in memory `reference-env-operational-gotchas`
+and `feedback-chatbot-hard-lessons`.
+
+**Cost/safety:** IAM `BedrockEmbeddingInvoke` v6 (embed + rerank + deepseek +
+qwen3 x2 + gpt-oss x2). Verified 0 EC2, no provisioned throughput — **pay-per-call
+only, idle ≈ $0**.
+
+**Plan — finish the chatbot TOMORROW (2026-08-06), deploy focus next week:**
+1. Re-embed the 9 reconstructed tables into Qdrant (owner approved; was wrongly
+   blocked) — ~30–60 min to make them searchable.
+2. "EPO"/abbreviation expansion (LLM entity extraction or aliases) — ~2–5h.
+3. VERIFY_PDF/crop lookup UX in the web — ~2–4h.
+4. UI showing generated-vs-extractive + retrieval path/evidence — ~2–4h.
+The **coding** fits a day. NOT finishable tomorrow and deliberately off the
+deadline: reconstructing the other **142 quarantined tables** + a **pharmacist
+review** of the corpus — that is the clinical-validation long pole (days→weeks,
+needs a human), separate from "chatbot features done".
+
 ## 2026-08-05 (evening 3) — The LLM cloud is LIVE: DeepSeek generation + Cohere rerank on the real corpus
 
 The owner rejected the $0 offline build as the deliverable and set a hard