VuQuangBao/duocthu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Checkpoint frontend UI/UX overhaul and ingestion embed benchmark work

Browse Source
This commit is contained in:
BaoVu2k4
2026-08-06 17:21:21 +07:00
parent 1e8cbdb586
commit a4b8e1c4db
78 changed files with 6761 additions and 654 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/progress-log.md
+655
View File
@@ -1,5 +1,660 @@
# Progress Log
## 2026-08-06 (cont. 11) — Real bug found by actually running the golden eval set: "thận trọng" silently answered as "chống chỉ định"
Owner pointed at a golden dataset (`Golden Dataset/golden_e2e_v1.csv` +4
more, 36-74 hand-authored cases each, dated 2026-08-04/05 — never run this
session until asked). Ran the 36-case e2e set live end-to-end. Findings,
graded against each case's own pass criteria:
- **3/36 (8%) correct answers discarded to an empty abstain** by the F-01
entailment-noise issue already flagged as a known limitation — the golden
set turns that into a measured rate, not a hunch.
- **2/36 wrong-section content gap, real bug, root-caused and fixed**: "X
cần thận trọng gì?" (asking precautions) was classified `attribute=
chong_chi_dinh` (contraindications) 9/9 times live-checked — the wrong
section entirely, silently dropping the actual precautions content (e.g.
metformin's lactic-acidosis warning, gentamicin's oto/nephrotoxicity) in
favor of contraindication text. Cause: the prompt gave the model a bare
`SECTION_KEYS` slug list with zero definitions — nothing to tell two
genuinely adjacent Vietnamese medical concepts apart. Fixed:
`rag/understanding.py` gained `SECTION_KEY_HINTS`, a short gloss per key
shown inline in the prompt, with `than_trong`'s explicitly stating it is
NOT `chong_chi_dinh` and naming the two example warnings that were
getting lost. Verified live: 3/3 reclassified correctly to `than_trong`
(metformin/gentamicin/ibuprofen), `chong_chi_dinh` questions unaffected,
and the two originally-broken answers now contain the exact required
content ("nhiễm toan lactic", "độc hại đối với cơ quan thính giác và
thận"). 2 new tests in `tests/test_understanding.py` (10 total, was 8).
- **Several other gaps found, not code bugs**: `#26` ("nên tự tăng gấp đôi
liều?") and the "An toàn (Type 3)" block (`#21-25`) in the golden set
model a **lay-patient safety framework** (refuse + "hỏi thầy thuốc")
that directly contradicts the owner's explicit correction earlier this
same session — this product gates on scope (human/non-human), not on
"asks for a recommendation" (`[[feedback_no_recommendation_gate]]`). The
golden set predates that correction by two days; treating its Type-3
rows as ground truth would silently re-introduce the exact gate the
owner ordered removed. Flagged to the owner rather than "fixed."
`#13`/`#20` test the `/v1/rag/suggest` autocomplete flow but were driven
through `/v1/rag/query` by mistake — not a valid test of those two rows,
not rerun yet. `#14` vs `#15` (bare-name inconsistency), `#30` (price
question), `#35` (two-drug wording) are minor, not investigated further
today.
`apps/ai-service`: **186 passed, 4 skipped**.
## Status at end of today's session (accurate as of cont. 10 below)
Codex's `CODEX_RAG_CODE_REVIEW_2026-08-06.md` correction order: **F-01
through F-07, F-09 done; F-08 and F-10 done for their core finding, with
named remainder.** Every completed item was live-verified against the real
running server, not only unit tests — several real bugs were found *by*
that live verification and fixed the same day, not just the ones the
review named (grounding fallback removed per owner correction, F-03's
`retrieve_framed` sending whole monographs, catalog-naming/id-form/
weight-parsing bugs the owner's own UI test surfaced, F-06's exact overflow
repro, F-08/F-09's Postgres connect-timeout hang).
**Named remainder, next session's work:**
- **F-08**: the Postgres-side unbounded-hang is fixed (`connect_timeout`),
but a real end-to-end deadline threaded through `RagAgent`'s own LLM
calls (understand → sufficiency → generate → up to 2 entailment retries,
up to 5 sequential Bedrock calls per request) does not exist — needs a
request-scoped budget object, a real design, not a bolt-on.
- **F-10**: the core gap (RagAgent had zero test coverage and was not
provably the same dependency graph as the live HTTP service) is closed —
`tests/test_live_datastores.py::test_real_rag_agent_end_to_end_through_the_http_api`
drives the real `/v1/rag/query` endpoint, real `RagAgent`, real
`RetrievalService`/`QdrantRetriever` against a real temporary Qdrant
collection, and a real Postgres trace, asserting drug id, citation, and
decision — only the nondeterministic cloud model call is faked, since this
session's own live probing found real generation/entailment calls too
noisy for a regression assertion. **Not built**: the review's full
adversarial regression list (prompt injection, fake-drug-near-alias,
provider-timeout-and-outage behavior, `conversation_id` presence/absence
producing the same safety decision, etc.) — one solid end-to-end case
proves the wiring is real and testable; a comprehensive battery is a
larger, separate effort.
- **`dosing_calc`** (a tested mg/kg calculator) and **`symptom_to_drug`**
(reverse indication→drug lookup) remain honest "not ready" clarifies —
deliberately not built under today's time pressure; see
`[[project_rag_rebuild_2026_08_06]]` on why rushing dosing math is the
wrong tradeoff.
`apps/ai-service` full suite: **184 passed, 4 skipped** (the new
integration test opts in via `RUN_INTEGRATION=1`, verified passing that
way), up from 118 passed at the start of today's session.
## 2026-08-06 (cont. 10) — F-10 core done: RagAgent proven live-testable end to end, not just live-tested by hand
Every F-01–F-06/F-08/F-09 live verification this session was a one-off
Python script run by hand against the real Qdrant/Bedrock/Postgres — real
evidence, but not a regression a future change would automatically re-run.
F-10 closes that: `tests/test_live_datastores.py` gained
`test_real_rag_agent_end_to_end_through_the_http_api`, following the
existing `RUN_INTEGRATION=1`-gated pattern in that file (temporary Qdrant
collection seeded with one real corpus chunk, real Postgres migration +
trace round-trip).
What's real in this test: `RagAgent`, `LlmQueryUnderstander`,
`RetrievalService`, `QdrantRetriever`/`QdrantParentStore` against a live
Qdrant, `GroundedAnswerService`, `PostgresTraceRepository` against a live
Postgres, and the actual `/v1/rag/query` FastAPI route via `TestClient` —
the identical object graph `bootstrap.build_runtime` wires in production.
What's faked: only the LLM boundary (`_FakeJsonLlm`, satisfying both the
`JsonLlm` and `AnswerGenerator` protocols with fixed payloads keyed by
schema shape) — deliberately, not for convenience: this session's own live
probing (F-01's entailment noise, F-03's non-deterministic generations)
found real cloud calls too noisy to assert exact drug id / citation /
decision against reliably. Asserts (Codex's exact F-10 list): resolved drug
id, citation chunk id and printed page, decision, and that the trace
persisted and reads back correctly.
Verified passing with `RUN_INTEGRATION=1` (4/4 in that file) and correctly
skipped by default (184 passed, 4 skipped without it — no cost/flakiness
added to the normal suite run).
**Scope, stated plainly**: this is the load-bearing first case proving the
production path is real and mechanically testable, not the comprehensive
adversarial battery the review sketched (prompt injection, fake-drug-near-
alias, provider outage/timeout behavior, `conversation_id` presence/absence
parity, multi-population-band evidence, etc.). Extending this one case into
that full battery is real remaining work, not done today.
## 2026-08-06 (cont. 9) — F-09 done (trace fail-open), F-08 partially: a real unbounded-hang found live and fixed
**F-09.** `routers/rag.py` called `traces.save()` synchronously before
returning a response; `PostgresTraceRepository.save()` opened a fresh
connection per call with no error handling, so a Postgres outage turned an
already-computed, safe answer into a 500 for a reason unrelated to whether
the answer was safe. Made an explicit fail-open decision (tracing is
observability, not the product): the router now wraps the `save()` call,
falls back to a locally-generated `trace_id` on any exception, and counts
it (`duocthu_trace_write_failed_total`, a new metric — a silent fail-open
with nothing to page on is indistinguishable from tracing quietly working).
Connection pooling (the other half of the original finding) not done —
real pooling needs startup-time lifecycle wiring, out of scope for today.
**F-08, live-verified, not fully scoped.** Testing F-09 by pointing
`POSTGRES_DSN` at an unreachable host live surfaced a sharper bug: a bare
`psycopg.connect()` with no `connect_timeout` hangs on the OS-level TCP
timeout (tens of seconds) when the DB is unreachable but not *actively*
refusing — which defeats the F-09 try/except just as completely as no
try/except at all, since the exception it's waiting for doesn't arrive in
time. Added `connect_timeout=5` to every `psycopg.connect()` call in
`adapters/postgres.py`. Verified live: same broken-DSN repro that
previously hung past a 30s client timeout now returns 200 with the correct
grounded answer in ~14.5s (5s bounded connect attempt + normal generation
latency). The broader F-08 ask — an end-to-end request deadline threaded
through every provider call — is **not done**: `TurnBudget`
(`rag/reasoning.py`) exists but belongs to the old `ConversationalLoopService`
path, which F-03 stopped constructing live; the new `RagAgent` path (up to
5 sequential Bedrock calls per request: understand, sufficiency, generate,
up to 2 entailment retries) has no budget object at all, bounded only by
each individual call's own fixed read_timeout (30-60s each). A real fix
needs a request-scoped deadline object passed into `RagAgent`/
`GroundedAnswerService` and consulted before each call — a genuine feature
to design, not something to bolt on safely in the time remaining today.
`apps/ai-service`: **184 passed, 3 skipped**.
## 2026-08-06 (cont. 8) — F-05 done: startup refuses a corpus/model manifest mismatch, live-verified both ways
The ingestion loader already writes a sidecar manifest (`<collection>
__manifest`, one point: corpus SHA, chunk count, embedding model_id,
dimensions) recording what a collection was built from
(`ingestion/ingestion/load/manifest.py`). Nothing on the ai-service side
ever read it — two unrelated embedding models can both produce
1024-dimensional vectors, and Qdrant returns plausible-looking but
meaningless nearest neighbours with no error at query time.
Added `rag/manifest.py` (`check_manifest` — pure, 6 unit tests) and wired
`bootstrap.py::_verify_corpus_manifest` to call it right after the query
embedder is constructed, before anything else. `main.py` builds the runtime
at import time, so a mismatch crashes startup — the service never comes up
against a corpus it wasn't verified against, rather than silently serving
degraded search.
Hit a real API mismatch immediately (pytest collection caught it, since
`test_api.py` imports `main.py`, which calls `build_runtime` against the
live Qdrant): this qdrant-client version has no `collection_exists`, and
`get_collection` is a known parse-bug risk in this environment (per
`reference_env_operational_gotchas`) — switched to `get_collections()` +
membership check instead. **Live-verified both directions**, not just unit
tests: the real collection's manifest (`model_id=cohere.embed-v4:0,
dimensions=1024`) matches the configured embedder and the server starts and
answers correctly; a monkeypatched `embedding_dimensions=768` against the
same real manifest correctly raises `ManifestMismatch` before any query
path is reachable.
`apps/ai-service`: **183 passed, 3 skipped**.
## 2026-08-06 (cont. 7) — F-06 done: the overflow-before-truncation bug, exact repro fixed
`ConversationState.append()` truncated `recent` to the window immediately;
`overflow()` then checked `len(self.recent) > window` on the *already-
truncated* tuple, which can never be true. Codex's exact repro (8 turns into
a window of 6: `recent=6, turn_count=8, overflow=0`) reproduced first,
unchanged from the review.
Fixed: `ConversationState` gained a `pending_overflow` field. `append()`
computes what it evicts *before* truncating and accumulates it there
(accumulates, not overwrites — a live turn calls `append()` twice in a row,
user then assistant, and the second call must not lose what the first
evicted). `overflow()` now just returns `pending_overflow`. The caller
clears it (`replace(state, ..., pending_overflow=())`) after folding into
the summary, or the same turns fold again next cycle —
`ConversationalLoopService._persist` (the live path) updated to do so;
`ConversationalRagService._persist` already reconstructs `ConversationState`
directly without passing the field through, so it already clears by
construction.
Verified the exact repro now returns the 2 actually-dropped turns instead
of `()`. 6 new tests in `tests/test_conversation.py`. **Not done, out of
scope for the remaining time today:** the second half of the original F-06
finding — `InMemoryConversationStore` loses all state on restart and
diverges across multiple workers. That needs a shared (Postgres-backed)
store, a real infra addition, not a bug fix; not attempted under today's
time pressure rather than risk a rushed, unverified persistence layer.
`apps/ai-service`: **177 passed, 3 skipped**.
## 2026-08-06 (cont. 6) — F-04 done: drug candidates bounded deterministically before the LLM picks, live-verified
`rag/understanding.py::LlmQueryUnderstander` used to show the model the
*entire* ~684-drug catalog every turn and trust any id it returned as long
as that id existed somewhere in the catalog (Codex's F-04 finding: catalog
membership proves the output is *some* real drug, not that it's the one the
user's text actually named — an LLM could satisfy that whitelist while
mapping an unrelated/invented name to a different real drug).
Reworked: `LlmQueryUnderstander` now takes a `resolver` (the existing
`CatalogDrugResolver`, already built in `bootstrap.py` for autocomplete) and
computes a deterministic **candidate set** from the turn + raw history text
*before* calling the LLM — exact alias matches plus a generous fuzzy
`suggest` pass (min_score=0.55, well below the resolver's own 0.84
auto-answer threshold, since the goal here is only to rule out drugs
nothing in the conversation plausibly refers to). Only that candidate
subset (not the full catalog) is shown to the model, and the model's pick
is validated against it — a real id the model names that isn't among the
turn's candidates is now treated as unknown, not trusted on catalog
membership alone. Also directly closes a separate prompt-cost finding from
the same review (sending the full catalog every turn is unbounded token
cost) since the shown block is now per-turn-sized, not fixed at ~684 rows.
`tests/test_understanding.py` extended (was 0 tests before this session,
per Codex's F-10 finding; now 8): covers exact-form and spaced-form
resolution, a genuinely invented name staying unknown, **a real catalog id
that has no deterministic candidate support still being rejected** (the
core F-04 guarantee — catalog membership alone is not enough), and a fuzzy
typo still resolving through `suggest`.
**Live-verified**, not just unit-tested: `aspirinol` (fake) still correctly
abstains out-of-scope; `amoxicillin` (correct INN spelling, a typo-adjacent
case) still resolves to `amoxicilin`; `metformin` and the 3-turn paracetamol
pediatric-dose conversation from the owner's own UI test both correctly
keep the same `resolved_drug_id` across every turn. No latency regression
observed (smaller prompt, same ~3-9s range dominated by generation, not
catalog size).
`apps/ai-service`: **174 passed, 3 skipped**.
## 2026-08-06 (cont. 5) — Three more live bugs found from the owner's own UI test of F-03, all fixed
Owner drove the real web UI (not curl) through a multi-turn pediatric dose
question and hit a severe regression: "Liều paracetamol cho trẻ em" -> two
clarify rounds (age, then weight) -> final turn answered "Không tìm thấy
paracetamol trong Dược thư Quốc gia Việt Nam" for a drug that plainly is in
it. Root-caused and fixed three distinct bugs in the F-03 wiring, in order:
1. **`_catalog_names` (bootstrap.py) could bury a drug's own name.** It
picked the first 3 aliases *alphabetically* per drug to show the LLM
understander. Paracetamol has 191 aliases (mostly trade names); the
alphabetically-first 3 were "0Frezefev, ABAB, Ace kid 80" — no
recognizable name at all. Mid-conversation, once the drug is no longer
restated in the raw turn text, the model has only history + this catalog
line to re-derive it from; with nothing recognizable shown, it read
"paracetamol" as an unknown name. Fixed: always show the drug_id's own
name form (`drug_id.replace("_"," ")`, guaranteed present) first, then
fill remaining slots preferring short ALL-CAPS aliases (the book's own
heading convention, usually the generic name) over dosage-suffixed brand
names. `tests/test_bootstrap.py` (new, 4 cases).
2. **That fix immediately exposed a second bug.** With the display name now
near-identical to the drug_id ("paracetamol acetaminophen" vs.
"paracetamol_acetaminophen"), the model started echoing the *spaced*
display form instead of the underscored id, and
`LlmQueryUnderstander._parse()`'s strict `d in self._ids` check demoted
a correctly-identified drug to `unknown_drugs` — same user-visible
failure, different cause. Fixed: `_resolve_id()` accepts either the exact
id or its space-substituted form (a deterministic, lossless formatting
tolerance — not fuzzy matching, no risk of resolving to an unrelated
drug). `tests/test_understanding.py` (new, 7 cases — this module had
zero coverage before today, per Codex's F-10 finding).
3. **"30 cân" (colloquial Vietnamese for "30 kg", no unit word) wasn't
reliably read as a weight.** Confirmed live: the model missed it
entirely in some runs, silently re-asking for weight the user had just
given. Added an explicit rule + schema hint that a bare number + "cân"/
"ký" means kilograms. Verified live: 3/3 clean extractions after the fix
(was inconsistent before).
All three verified against the real running server with the owner's exact
repro sequence, not just unit tests — final state: the drug (`resolved_drug_id
= paracetamol_acetaminophen`) now stays correctly attached across all three
turns, and weight is correctly captured. **Not fixed, deliberately, already
flagged (F-07):** `dosing_calc` still doesn't compute an actual mg dose once
enough information is gathered — it falls through to ordinary section
retrieval (the clinician sees the dosing table, not a calculated number). A
weight-based calculator is a real feature to build, not a wiring bug; out of
scope for this pass.
Also, per owner UX feedback, warmed up the static smalltalk reply (was a
terse "Chào anh/chị. Tôi tra cứu... Anh/chị muốn hỏi về thuốc nào?").
`apps/ai-service`: **173 passed, 3 skipped** (was 162 at the end of the F-03
entry below).
## 2026-08-06 (cont. 4) — F-03 done: RagAgent wired into the live server, two real bugs found and fixed by driving it
Wired the new LLM-understanding orchestrator (`rag/agent.py` + `rag/
understanding.py`, built last session but never called by anything live —
Codex's exact F-03 finding) into `bootstrap.py`/`routers/rag.py`. Both
single- and multi-turn requests now go through one path:
`RagAgent.handle()`. The old `CatalogDrugResolver`/`QueryRoutingService`/
`ConversationalLoopService` stack stays in the codebase (still unit-tested,
still used for autocomplete + the no-generator-configured fallback) but is
no longer constructed as the live query path — per Codex, full deletion
waits on a production-path parity suite (F-10), not done yet.
Added the coverage that didn't exist: `tests/test_agent.py` (14 cases —
`RagAgent` had zero tests before this), `tests/test_retrieval_service.py`
+2 for `retrieve_framed`, `tests/test_api.py` +4 for the router's agent
branch. 162 passed, 3 skipped.
**Drove the actual running server** (per house rule: never claim a wiring
change works from unit tests with fakes alone) and found two real bugs unit
tests couldn't have caught:
1. **`retrieve_framed` had no bare-name/overview case.** `retrieve()` (the
old path) always answered a bare drug name from four identity sections
only; `retrieve_framed` had no equivalent and always fetched the entire
~29-section monograph, then relied on rerank to trim it — silently
sending the whole book as evidence whenever rerank was off or failed
open. Live symptom: asking bare "paracetamol" abstained empty every
time (answer too long, generation intermittently malformed). Fixed:
`retrieve_framed` gained an `is_overview` parameter (driven by the
frame's `turn_type == "drug_overview"`), mirroring the old intro-only
behavior, and the non-overview rerank branch is now capped at
`evidence_limit` even when rerank fails open — an ordering aid failing
open must not also remove the size bound. Verified live: 3/3 clean
answers after the fix, none of the prior empty-abstain failures.
2. **The entailment judge (added this session, F-01) is noisier than one
call suggests.** Same claim/evidence pair, called repeatedly, disagreed
with itself — confirmed live on the warfarin/aspirin interaction case,
which correctly cites a drug-interaction list evidence block but got
rejected 0/2, 1/2, then 3/3 across separate live batches. Added a
same-claim retry (`GroundedAnswerService._verify_entailment`): a lone
reject retries once, only two agreeing rejects discard the generation.
Also sharpened the entailment prompt to explicitly call out dense
comma-separated drug-interaction lists, since the specific failing claim
named a drug buried mid-list. Owner explicitly capped further spend
here (more retries = more tokens for a narrowing edge case) — the
retry/prompt change did not fully eliminate this one case in further
live testing (still failed 3/3 in the last batch), and it was
deliberately **left as a known, safe-direction residual limitation**
rather than chased further: the failure mode is abstain (never a
fabricated interaction claim), not wrong output. Documented in
`_verify_entailment`'s docstring; a cleaner fix (e.g. breaking a
multi-drug interaction claim into a per-drug comparison instead of one
long prose evidence block) is a good candidate for a future pass, not
solved today.
Also fixed a mismatched piece of the wiring in `apps/web/app/api/chat/
route.ts`: it discarded `RagAgent`'s specific abstain messages (e.g. "Không
tìm thấy X trong Dược thư") in favor of a generic fallback, because it only
consulted `answer` when `decision !== "abstain"`. Now prefers `rag.answer`
whenever it is non-null, regardless of decision.
## 2026-08-06 (cont. 3) — Investigated "684 vs 700+24 expected" monograph-count question: zero real drug monographs missing, gap is 100% explained
Owner asked why the corpus has 684 monographs when the expectation was
~700 drug monographs + 24 general-chapter monographs. Did not rely on any
number already sitting in memory/docs — re-ran `ingestion.cli validate`
live against the real PDF this session to get a current ground-truth
comparison, per [[feedback-rigorous-validation]] / [[feedback-verification-ladder]]
("recompute every number before quoting it").
**Live re-run result** (`python -m ingestion.cli validate --pdf
data/raw/duoc-thu-quoc-gia-viet-nam-2018.pdf`):
```
detected monographs: 684
ground-truth entries: 705 (parsed from the book's own back-of-book index,
"Mục lục tra cứu", pages 1529+)
recall: 96.2% (678/705)
precision: 99.1%
```
27 ground-truth entries didn't match a detected monograph, and 6 detected
monographs didn't match a ground-truth entry. Pulled the **full** unmatched
list (the CLI only prints the first 20 of 27) via a direct Python call
into `ingestion.validation.back_index`/`metrics` — classified all 27 by
hand:
| Category | Count | Detail |
|---|---|---|
| Part 1 general-chapter titles (printed pp. 39-95) | 20 distinct (21 lines — "Thuốc chống loạn thần..." p.75 is duplicated in the book's own index) | Hướng dẫn sử dụng(39), Kê đơn thuốc(40), người cao tuổi(41), suy gan/thận(43), trẻ em(45), thai kỳ/cho con bú(47), giảm đau(48), hen phế quản(51), kháng động kinh(55), kháng HIV(61), kháng sinh(70), cephalosporin(72), chống loạn thần×2(75), lao(77), viêm gan B(80), ADR(83), dị ứng thuốc(85), ngộ độc(90), dược động học(93), tương tác thuốc(95) |
| Part 3 appendix titles | 2 | BSA calc (1497), pha thuốc tiêm IV (1498) |
| Part 2 drug names | 4 | Alphatocoferol(165), Benzoyl peroxyd(246), Hydrogen peroxyd(781), Tretinoin (thuốc uống)(1405) |
**Then checked all 4 remaining "drug" entries directly against
`ingestion/data/processed/monographs.jsonl`** (not just assumed) — all 4
are already present in the corpus, under a differently-spelled
`drug_name`:
- Alphatocoferol → `ALPHA TOCOPHEROL (Vitamin E)` (tocoferol/tocopherol)
- Benzoyl peroxyd → `BENZOYL PEROXID` (peroxyd/peroxid)
- Hydrogen peroxyd → `HYDROGEN PEROXID` (peroxyd/peroxid, same pattern)
- Tretinoin (thuốc uống) → `TRETINOIN (UỐNG)`
These 4 also account for 2 of the "6 unmatched detected monographs"
(HYDROGEN PEROXID, TRETINOIN (UỐNG) both show up on both sides of the
diff — same monograph, name-matching miss in `validation/metrics.py`'s
`_names_match`, not two different problems).
**Conclusion, fully closed, no open unknowns left:**
1. **Zero Part 2 (drug) monographs are actually missing.** Every
ground-truth drug entry in the back-index resolves to something already
in the 684. The apparent gap was a validator string-matching artifact
(Vietnamese `-yd` vs. English `-id`/`-pherol` spelling variants), not
missing content. `684` is the correct, complete count for Part 2.
2. **The "24 general chapters" are 0/24 present** — confirmed only 20
distinct chapters exist in the book's own index (not 24; owner should
double check where the "24" figure came from), and none of the 20 are
extracted, because the pipeline was scoped to Part 2 only from the
start (`extract`/`segment`/`assemble` never touch printed pp. 37-98).
This matches the already-known, already-documented scope gap in
[[project-rag-rebuild-2026-08-06]] / `reference_duoc_thu_2018_structure`
memory — not a new discovery, just re-confirmed live.
3. The book's own front-matter "~700 substances" figure is the
publisher's approximate active-substance count, not a strict
heading-count promise — some monographs bundle multiple substances
under one heading (INSULIN = 20 ATC codes / salts under 1 monograph,
the HMG-CoA-reductase-inhibitor class monograph, ARGININ's 2 salts),
so a smaller heading-count than 700 is expected and consistent with
full coverage, not evidence of missing data.
**Not done / possible follow-up (not requested this session):** the 6→4
`_names_match` misses above suggest a small, mechanical fix (normalize
`-yd`↔`-id`/`-pherol` diacritic-free spelling variants, or add explicit
alias pairs) would push CLI-reported recall from 96.2% to ~99.7% without
touching extraction at all — cosmetic (metric accuracy), not a data-quality
fix, since the underlying monographs already exist either way. The 4
remaining truly-unmatched-detected entries (CARBIDOPA-LEVODOPA, THUỐC
PHIỆN-OPIAT-OPIOID, VẮC XIN DPT, VẮC XIN MMR) are compound/hyphenated-name
matching gaps in the same function, same category, not investigated
further this session.
## 2026-08-06 (cont. 2) — Owner correction: no fallback to raw source text when a generator is configured; F-02 scoped down to subject_scope only
Two corrections from the owner mid-F-02, both applied immediately:
**1. Dropped intent-based recommendation gating entirely.** Built a keyword
detector for `QueryIntent.RECOMMENDATION` ("nên dùng thuốc gì" etc.) as part
of F-02's server-side policy derivation — wrong call, reverted same session.
**This product is for doctors and pharmacists** (`[[project_target_audience]]`),
and a clinician asking "thuốc nào tốt nhất cho bệnh nhân suy thận" is normal,
in-scope use of a formulary reference, not a request to abstain on. `rag/
policy.py` now derives `subject_scope` only (veterinary/non-human keyword
check — a corpus-coverage fact, not a restriction on clinical questions);
`routers/rag.py` passes `intent` through from the caller unchanged, same as
before F-02. `tests/test_policy.py` scoped down to match.
**2. Removed the extractive-fallback safety net for a CONFIGURED generator
that fails.** Previously, any generation failure — provider outage, malformed
JSON, `grounding.verify` rejection, entailment rejection — fell back to
quoting the retrieved evidence verbatim ("the source is always available
because it was computed first"). Owner: that raw citation-stapled paragraph
is the retired offline-extractive product shape (`[[project_llm_cloud_plan]]`
— "owner wants a REAL LLM chatbot... not the offline extractive build"), and
must not reappear as a silent degradation path now that generation is live.
`GroundedAnswerService.answer_from_result` (`rag/answer.py`) now branches on
whether a generator is configured at all, not just on whether this call
produced one:
- **No generator configured** (`ANSWER_PROVIDER=disabled`, the default) is
unchanged — a deliberate, fully-supported retrieval-only mode, still quotes
the source.
- **A generator IS configured** and this generation failed any check → the
turn **abstains** (`decision=ABSTAIN, reason="generation_unavailable"`,
`answer=None`), never a raw source dump.
Updated 9 tests across `test_grounded_generation.py` and
`test_citation_and_intro.py` whose assertions encoded the old fallback
behavior (`grounded.answer.startswith(EVIDENCE_TEXT)` → `grounded.answer is
None` + `decision == ABSTAIN`). Live-verified the happy path still works
unchanged against the real model (Qwen3/Bedrock Converse, ~3.5s, served
correctly) — this change only touches the failure branch.
`apps/ai-service`: **140 passed, 3 skipped**.
## 2026-08-06 (cont.) — F-01 fixed: grounding verifier no longer trusts a global number pool or an uncited claim
Codex's code-only review (`coordination/CODEX_RAG_CODE_REVIEW_2026-08-06.md`)
reproduced three ways `rag/grounding.py::verify` let an unsafe generated
answer through. Reproduced all three locally first, byte for byte, before
touching code — all three real. Working through the review's proposed
correction order (F-01 → F-02 → ... → F-10; tracked as tasks #1-#8).
**F-01, done.** Two independent fixes, both proven live (Qwen3 via Bedrock
Converse), not just against a fake generator:
1. **Per-citation binding, not global pool.** `verify` used to pool every
number from every evidence block into one set and check answer numbers
against that pool — so a number true of block 2 passed under a citation
to block 1 (`so_sai_nguon`). Rewrote to split the answer at each `[n]`
citation group and check only the block(s) that group names.
2. **Citation required for every claim.** A citation-less generated answer
used to pass silently as long as it stated no number the pool didn't
already contain (`khong_citation`) — trivially true when the answer had
no numbers at all. Now any substantive claim with no valid citation is
rejected (`uncited_claim`). This also kills the old "attach every
retrieved citation when the generated text cites nothing" fallback in
`GroundedAnswerService`: that code path is now unreachable, since
`grounding.verify` rejects the citation-less generation before it gets
there — the extractive fallback (which always cites everything by
construction) takes over instead.
3. **Entailment gap (`claim_bia`) — regex can't see meaning.** A fabricated
nonnumeric claim with a syntactically valid citation ("Metformin chữa
ung thư [1]" citing a block about đái tháo đường) still passed both
fixes above: no number, citation in range. Closed with a second LLM
call (`GroundedAnswerService._verify_entailment`, `rag/prompt.py`'s
`build_entailment_request`) that runs after `grounding.verify` passes:
each substantive cited claim, checked only against the evidence block(s)
it names, judged by a model told to compare wording, not reason about
medicine. Fails closed (provider outage/malformed JSON → reject, not
accept). **Live-verified against the real model**, not simulated: ran
the actual entailment prompt through `BedrockConverseAnswerGenerator`
(Qwen3) on `claim_bia`, a fabricated contraindication, a faithful claim,
and a legitimate paraphrase — correctly rejected the two fabrications
(`entailed: false`) and passed the two honest ones (`entailed: true`,
including the paraphrase, so it isn't just penalizing rewording). Also
ran the full `GroundedAnswerService` pipeline live end-to-end (real
generator, real multi-call sequence) on a legitimate metformin dose
question — served correctly, ~3.4s.
`apps/ai-service`: **134 passed, 3 skipped** (was 118p/3s before this
session; added `tests/test_grounding.py` — 12 adversarial cases — plus 4 new
entailment-path cases in `tests/test_grounded_generation.py`, and updated 3
existing tests whose assertions encoded the old, buggy behavior).
**Known residual limit**, stated in `rag/grounding.py`'s docstring: the
entailment LLM call is itself a model judgment, not a proof — it is a real
improvement over zero semantic check, not a formal guarantee. F-02 through
F-10 (scope/intent server-side enforcement, wiring the new
`RagAgent`/`LlmQueryUnderstander` orchestrator that's currently dead code,
bounding entity candidates, manifest validation, conversation overflow bug,
request budgets, trace failure policy, production-path regression suite)
are next, in that order — none touched yet this pass.
## 2026-08-06 — RAG rebuild started: live failure diagnosis + LLM query-understanding front-end (replacing the brittle resolver)
Owner reported the live chatbot "cực ngu, sai gần hết" and asked to rebuild the
RAG from scratch (incl. chunking). Per the never-fabricate rule, drove the REAL
running service before designing.
**Stack brought up live** (all local, $0 to load): Qdrant `duocthu_v1` already
held 15,100 pts @1024-dim (green); Postgres up; ai-service :8079 running with the
cloud-live `.env` (cohere-v4 embed + qwen3 generation + Cohere rerank).
**Live diagnostic battery (~20 hard VN questions, real `POST /v1/rag/query`).**
Finding, evidence-backed: it is NOT "sai hết" and the culprit is NOT chunking —
when a single drug resolves cleanly the answer is grounded and correct
(paracetamon typo ✓, metfomin typo ✓, multi-turn "nó dùng cho trẻ em" inherited
metformin ✓). The failures cluster in the **query-understanding / drug-resolution
front-end** (the `CatalogDrugResolver` fuzzy `SequenceMatcher` + keyword
`SectionResolver`):
- `aspirinol` (fake drug) fuzzy-matched to aspirin and ANSWERED — a safety bug.
- `amoxicillin` (correct English INN) tied/ambiguous → abstained; the sentence
word "uống" polluted fuzzy scoring (matched `tretinoin_uong`).
- `warfarin với aspirin` (interaction) → ambiguous → abstain; no interaction path.
- `còn liều dùng thì sao?` follow-up lost the drug (inconsistent inheritance).
- `trẻ 5 cân paracetamol` → clarifies forever; no mg/kg weight-based calc node.
- symptom→drug and BSA/Part-1/Part-3 → abstain (scope gaps).
**Corrected an earlier overstatement (owner was right):** section chunking is NOT
uniform — 172/684 monographs (25%) are class monographs cramming many sub-drugs
into one section (INSULIN dose = 9,268 chars / 20 ATC, VITAMIN D 14,197 chars),
chunked by blind token-window. So re-chunk (sub-drug/population/indication-aware)
IS warranted later — but it does not fix the front-end failures above.
**Rebuild step 1 — LLM query-understanding front-end (new, PROVEN live).**
`apps/ai-service/rag/understanding.py`: `LlmQueryUnderstander` + `QueryFrame`.
One LLM call reads the messy turn (+ history + the real 684-drug catalog) → a
structured frame (turn_type, drugs [catalog-validated], unknown_drugs, attribute,
population, weight_kg, indication). Safety kept: the model may only pick drug_ids
from the real catalog; an unrecognised name goes to `unknown_drugs`, never snapped
to a near drug. `rag/` stays SDK-free (LLM injected as a `JsonLlm` protocol,
satisfied by the existing `BedrockConverseAnswerGenerator`). Proven on the live
LLM against all 7 killer cases the old resolver failed — every one now read
correctly (amoxicillin→amoxicilin, aspirinol→unknown, warfarin+aspirin→interaction
with both drugs, trẻ 5 cân→dosing_calc weight=5.0, sốt cao→symptom_to_drug,
follow-up→inherited metformin, chào→smalltalk).
**NOT yet done:** the frame is not wired into retrieval/generation — the old
`CatalogDrugResolver`/`SectionResolver` still drive `/v1/rag/query`. Next: route on
`turn_type` (interaction→gather both drugs; symptom_to_drug→reverse `chi_dinh`
lookup; dosing_calc→a tested mg/kg calculator like `rag/calculators.py`), unit +
live eval vs the battery, then decide the structure-aware re-chunk (needs owner GO
for re-embed ~$0.5). No re-embed or cloud spend beyond cents of diagnostic/proof
LLM calls this session.
## 2026-08-05 (night) — Live-chat UX overhaul: reasoning/clarify, multi-turn, Qwen3; plan = finish chatbot tomorrow, deploy next week
Owner drove the running web chat with messy real inputs and found the offline-era
query layer was a hodgepodge. Fixed the failures found, each **verified by
chatting the running service** (not just unit tests). Model switched to
**qwen.qwen3-next-80b-a3b** (DeepSeek ignored the clarify instruction; Qwen3 and
gpt-oss both follow it — A/B'd). ai-service **118 passed, 3 skipped**.
Fixed (commits `6c6a916`, `5feccba`, `553be09`, `7f45d06`):
- **Reasoning/clarify (the headline):** a focused sufficiency-check LLM call runs
BEFORE generation. An under-specified dose ("paracetamol cho trẻ em") now ASKS
age/weight/route/indication instead of dumping every band. Adult dose / CCĐ /
interactions answer normally (no false clarify). `answer._check_sufficiency` +
`prompt.build_sufficiency_request`; `GroundedAnswer.clarification` → decision
"clarify".
- **Multi-turn:** "thuốc đó…" was double-resolved (inherited then re-resolved
from rewritten text → ambiguous → empty). Now the resolved drug_id is passed
straight to retrieval (`routing.retrieve_for_drug`); raw turn drives section
routing; overview+rerank finds the part. Verified: Oxymetazolin → "thuốc đó cho
trẻ dưới 6 tuổi?" → correct than_trong answer.
- **Did-you-mean garbage:** fuzzing a sentence ("EPO…") or "đúng" returned
terbinafin/tretinoin in a loop. Now suggestions only for short drug-name misses;
confirmations get "which drug?".
- **Bare name → drug intro** (class + indication + invite), not a forms dump.
- **Citations = only the [n] actually cited** (was ~13 chips for a 1-source line).
- Rerank trims overview 29→6; inherited-drug notice uses the display name.
**Operational lesson (cost real time):** `uvicorn --reload` does NOT work on this
Windows box — the owner chatted STALE servers repeatedly. Must kill :8079 and
restart after every edit. Recorded in memory `reference-env-operational-gotchas`
and `feedback-chatbot-hard-lessons`.
**Cost/safety:** IAM `BedrockEmbeddingInvoke` v6 (embed + rerank + deepseek +
qwen3 x2 + gpt-oss x2). Verified 0 EC2, no provisioned throughput — **pay-per-call
only, idle ≈ $0**.
**Plan — finish the chatbot TOMORROW (2026-08-06), deploy focus next week:**
1. Re-embed the 9 reconstructed tables into Qdrant (owner approved; was wrongly
blocked) — ~30–60 min to make them searchable.
2. "EPO"/abbreviation expansion (LLM entity extraction or aliases) — ~2–5h.
3. VERIFY_PDF/crop lookup UX in the web — ~2–4h.
4. UI showing generated-vs-extractive + retrieval path/evidence — ~2–4h.
The **coding** fits a day. NOT finishable tomorrow and deliberately off the
deadline: reconstructing the other **142 quarantined tables** + a **pharmacist
review** of the corpus — that is the clinical-validation long pole (days→weeks,
needs a human), separate from "chatbot features done".
## 2026-08-05 (evening 3) — The LLM cloud is LIVE: DeepSeek generation + Cohere rerank on the real corpus
The owner rejected the $0 offline build as the deliverable and set a hard
docs/rag-rebuild-plan.md
+90
View File
@@ -0,0 +1,90 @@
# RAG rebuild plan — chatbot tra cứu hoàn thiện cho Dược thư Quốc gia VN 2018
Ngày 2026-08-06. Grounded bằng: (1) chẩn đoán live service thật, (2) đo toàn
corpus đã xử lý, (3) **đọc trực tiếp PDF gốc** (render ảnh, text-layer hỏng).
## 1. Cuốn sách thật ra sao (đọc từ PDF gốc, không suy đoán)
- **1668 trang, 2 cột, 3 phần.** Text-layer PDF **hỏng/đảo trang** (trang
Paracetamol chèn "papaverin hydroclorid") → mọi trích xuất phải dựa vào
**span in đậm + hình học + kiểm tra bằng mắt**, không dùng pdfplumber text.
- **Part 2 — chuyên luận thuốc (vật lý ~99–1496):** CÓ 19 field in đậm cố định
(Tên chung quốc tế, Mã ATC, Loại thuốc, Dạng thuốc, Dược lý, Chỉ định, Chống
chỉ định, Thận trọng, Thời kỳ mang thai/cho con bú, Tác dụng KMM, Hướng dẫn xử
trí ADR, Liều lượng, Tương tác, Quá liều, Độ ổn định, Tương kỵ, Thông tin qui
chế, Tên thương mại). 684 chuyên luận.
- **NHƯNG cấu trúc bên trong field KHÔNG đồng nhất:** 172/684 (25%) là chuyên
luận NHÓM. INSULIN gộp ~20 mã ATC, section liều 9.268 ký tự; VITAMIN D 14.197
ký tự/8 ATC. Thuốc con + đối tượng + chỉ định nằm **lẫn trong prose** (nhãn
kết thúc bằng ":", ví dụ "Đái tháo đường typ 1:", "Người lớn:"), **không có
heading riêng**. Đây là gốc của "không phải section nào cũng như thế".
- **Part 1 — chương tổng quát (vật lý ~37–98):** free-form — dàn ý đánh số phân
cấp, bảng phân loại đóng khung, heading tự do. Nội dung lâm sàng quan trọng:
ngộ độc & thuốc giải độc, kê đơn cho người suy gan/thận/trẻ em/thai kỳ, hướng
dẫn theo bệnh. **Corpus hiện KHÔNG có.**
- **Part 3 — phụ lục (vật lý ~1497–1528):** phân loại ATC, tính BSA, pha tiêm
tĩnh mạch. Dạng tra cứu/bảng. **Corpus hiện KHÔNG có.**
## 2. Gốc bệnh đã đo được (bằng chứng live, không phải cảm tính)
Khi resolve trúng 1 thuốc → trả lời ĐÚNG, grounded, có nguồn. Lỗi tập trung ở
**tầng hiểu-câu/định-tuyến + thiếu node**, và ở **độ hạt chunk cho section phi
đồng nhất** — KHÔNG phải ở embedding/generation:
1. Resolver fuzzy vừa nhận nhầm thuốc bịa (`aspirinol`→aspirin) vừa chết ở tên
Anh đúng (`amoxicillin`). **[ĐÃ THAY — mục 3.1]**
2. Không có node: tương tác 2 thuốc, triệu chứng→thuốc, tính liều mg/kg×cân, BSA.
3. Kế thừa đa lượt chập chờn.
4. Section chunk trộn nhiều đối tượng (đo: 72% chunk liều) + chuyên luận nhóm bị
cắt mù token-window (INSULIN 8 mảnh, không theo ranh giới thuốc con/chỉ định).
5. Thiếu phạm vi Part 1 + Part 3.
## 3. Kế hoạch đập & xây (phân đợt, mỗi đợt có gate + eval)
**GIỮ (xương sống an toàn, đã kiểm chứng — không đập):** extract span-đậm hình
học + provenance; quarantine bảng/công thức (ADR 0006); `grounding.verify` (chặn
bịa số); skeleton vòng lặp `reasoning.py`.
### 3.1. Não mới — hiểu câu bằng LLM ✅ ĐÃ LÀM + PROVEN (session này)
`rag/understanding.py` `LlmQueryUnderstander` → `QueryFrame` (turn_type, drugs
[chỉ từ 684 thuốc thật], unknown_drugs, attribute, population, weight_kg,
indication). Chạy đúng cả 7 ca killer trên LLM thật. `rag/` không import SDK.
Đã dọn 2 embedder 0d giả (SectionOnly/LocalHash), chỉ còn cohere-v4; 118 test pass.
### 3.2. Nối frame vào đường trả lời + thêm node ⬅️ TIẾP THEO ($0, không re-embed)
Thay `CatalogDrugResolver` fuzzy + `SectionResolver` keyword bằng router theo
`turn_type`:
- `drug_attribute`/`drug_overview` → retrieve theo drug_id + attribute.
- `interaction` → gom cả 2 thuốc (mục tương tác của mỗi bên) → tổng hợp; "không
thấy bằng chứng" phải nói *đã tra ở đâu*, không khẳng định "an toàn".
- `symptom_to_drug` → tra ngược `chi_dinh` (48 thuốc chứa "sốt"…).
- `dosing_calc` → node tính mg/kg×cân nặng (hàm CÓ TEST, kiểu `calculators.py`,
không để LLM nhân số) + BSA.
Eval: unit + chạy lại bộ battery live + golden; grounding vẫn bật. Xoá resolver cũ.
### 3.3. Re-chunk structure-aware (đợt lớn — cần owner GO cho re-embed ~$0.5)
- **Part 2:** cắt section lớn/nhóm theo **cấu trúc thật trong prose** — nhãn chỉ
định ("Đái tháo đường typ 1:"), nhãn đối tượng ("Người lớn:", "Trẻ em:"), tên
thuốc con — thành child chunk; parent = cả section để hydrate. Thêm metadata
`population_tags`, `indication_tags`, `subdrug_tags` để lọc. Gỡ 72% trộn +
INSULIN blob.
- **Part 1:** chunk phân cấp open-taxonomy (đường dẫn heading từ dàn ý đánh số);
bảng đóng khung → quarantine/tái dựng.
- **Part 3:** phụ lục riêng (bảng ATC; BSA → calculator).
- Schema v5 (+`content_type` monograph|chapter|appendix, +`chapter_id`). Embed
chunk MỚI (cohere, announce trước). **Gate CLAUDE-cũ:** span-ledger phủ đủ
**1668 trang, unassigned=0**, provenance còn nguyên, không mất chuyên luận.
### 3.4. Tái dựng 151 bảng quarantine cho retrieval (vision↔geometric consensus)
Cell nghi ngờ gắn `needs_expert`; hiển thị vẫn crop+trang (bác sĩ tự đối chiếu).
### 3.5. Eval cuối
Golden + battery live, so trước/sau, grounding on; whole-doc gate cho re-chunk.
## 4. Ràng buộc
- Codex làm song song → claim ownership trong `coordination/` trước khi sửa.
- Không cloud spend nếu chưa có owner GO cụ thể (Part 2 re-embed + Part 1/3 embed).
- AWS $138.50 credit khuyến mãi, pay-per-call, idle ≈ $0.
## 5. Thứ tự đề xuất
3.2 trước (bot hết ngu ngay, $0) → 3.3 (mở phạm vi + sửa chunk, cần GO) → 3.4 → 3.5.