Wire the guarded conversational RAG answer layer end-to-end
This commit is contained in:
@@ -0,0 +1,290 @@
|
||||
# Review round 2: verifying the response to round 1
|
||||
|
||||
Reviewer: Claude, 2026-08-03. Every claim in
|
||||
`response-rag-retrieval-2026-08-03.md` was re-run, not read.
|
||||
|
||||
**Verdict: five findings are genuinely fixed. One is not fixed — it was moved.
|
||||
Three new problems appeared in the fix itself.**
|
||||
|
||||
Reproduction:
|
||||
|
||||
```
|
||||
python -m rag.run_eval \
|
||||
--cases evals/manual_adversarial_hard10.jsonl \
|
||||
--documents ../../ingestion/scratch/rag-table-pilot/out/all/retrieval_documents.jsonl \
|
||||
--parents ../../ingestion/scratch/rag-table-pilot/out/all/logical_tables.jsonl \
|
||||
--aliases evals/drug_aliases.json
|
||||
-> recall_at_1/3/5 = 0.8889, negative_abstain_rate = 1.0
|
||||
```
|
||||
|
||||
The reported numbers reproduce exactly.
|
||||
|
||||
---
|
||||
|
||||
## Confirmed fixed
|
||||
|
||||
Checked in the code and by re-running, not taken on trust:
|
||||
|
||||
- **F2** — `QUANTITATIVE_TERMS` and `_structured_boost` are gone. Ranking is
|
||||
now real BM25 with IDF over the loaded corpus plus a character-n-gram term.
|
||||
No hand-written vocabulary remains in the ranker.
|
||||
- **F7** — `run_eval` now constructs `EvidencePolicy()` with the shipped
|
||||
defaults.
|
||||
- **F8** — `_char_ngrams` operates on `_normalized(text)` in original order.
|
||||
The sorted-unique-terms behaviour is gone.
|
||||
- **F1 mechanism** — `ambiguity_margin` is removed from `EvidencePolicy` and
|
||||
`_is_ambiguous` is deleted. Score ties no longer cause a refusal.
|
||||
- **F5 partly** — `run_eval` no longer hands `drug_id` to retrieval. A
|
||||
`CatalogDrugResolver` runs first, and the `famciclovia` typo genuinely
|
||||
resolves through `SequenceMatcher`; `renal-herpes-typo` now passes as
|
||||
`answerable` with resolution actually exercised. This is a real improvement.
|
||||
- **F4 partly** — the prose layer now covers **14,915 documents across 684
|
||||
drug IDs**, not 6. Also a real improvement.
|
||||
- **Verification claims** — all reproduced: `ingestion` **204 passed**,
|
||||
`apps/ai-service` **10 passed**, and lint is clean under both `ruff check rag
|
||||
tests` *and* the project's stricter `--select F,E9,B,ARG`. The two ARG001
|
||||
errors from round 1 are fixed.
|
||||
|
||||
---
|
||||
|
||||
## 1. NOT fixed — finding 2 was relocated, not resolved
|
||||
|
||||
Round 1's finding was: *four of ten passes are bought by a hand-written term
|
||||
list drawn from the scored queries.* The ranker is now clean. But the same
|
||||
pattern reappeared one layer up, in the thing that replaced it:
|
||||
|
||||
```python
|
||||
VETERINARY_TERMS = frozenset({"gia suc", "gia cam", "meo", "thu y"})
|
||||
```
|
||||
|
||||
plus a special-cased regex for `chó`. The evaluation has exactly **one**
|
||||
negative case, and it is about a **mèo**. `negative_abstain_rate: 1.0` is
|
||||
computed over **n = 1**, and that one word is in the list.
|
||||
|
||||
Measured, running the full `QueryRoutingService` against `out/all`:
|
||||
|
||||
| Query ending | Result |
|
||||
|---|---|
|
||||
| `... cho mèo` | ABSTAIN `out_of_scope_veterinary` |
|
||||
| `... cho chó` | ABSTAIN `out_of_scope_veterinary` |
|
||||
| `... cho thỏ` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho ngựa` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho lợn` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho bò sữa` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho chuột lang` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho vẹt cảnh` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... dùng trong thú cưng` | **ANSWERS** `grounded_evidence_available` |
|
||||
|
||||
Seven of nine veterinary phrasings are answered with a **human famciclovir
|
||||
dose** and the decision `grounded_evidence_available`. Note the last row: `thu
|
||||
y` is in the list, but `thú cưng` normalises to `thu cung` and misses.
|
||||
|
||||
`HumanClinicalScopeGuard` is not a scope guard. It is a five-entry animal-word
|
||||
list, and the evaluation that scores it contains exactly the words in it. The
|
||||
round-1 objection was never about `_structured_boost` specifically — it was
|
||||
about measuring a component against the cases it was written from. That
|
||||
objection still stands, unchanged, against this code.
|
||||
|
||||
A scope guard that generalises cannot be a keyword list. It has to come from
|
||||
something the corpus actually says — the book is a human formulary, so the
|
||||
question is whether the query's subject is a human patient, not whether it
|
||||
contains one of five nouns.
|
||||
|
||||
## 2. `recall_at_5` is not a measurement
|
||||
|
||||
`EvidencePolicy.evidence_limit` is 3, so `result.evidence` never exceeds three
|
||||
items and `retrieved_ids` never exceeds length 3 — observed lengths across the
|
||||
run are `{0, 1, 2, 3}`. `_recall_at(rows, 5)` then slices `[:5]` of a tuple
|
||||
that is at most 3 long.
|
||||
|
||||
**`recall_at_5` is forced to equal `recall_at_3` for every possible input.**
|
||||
It is not a third data point; it is `recall_at_3` printed twice. Round 1 asked
|
||||
for Recall@1 and Recall@3 reported separately, and that part is done and
|
||||
useful — but reporting a third identical figure makes the result look more
|
||||
thoroughly measured than it is.
|
||||
|
||||
Either raise `evidence_limit` above 5 for the diagnostic run, or drop
|
||||
`recall_at_5`.
|
||||
|
||||
## 3. `expected_drug_id` was added and never scored
|
||||
|
||||
The field exists in `EvaluationCase` and is populated by `read_cases` for all
|
||||
10 cases. It appears **nowhere else** — `EvaluationOutcome.passed` and
|
||||
`summarize()` never read it.
|
||||
|
||||
So resolution now happens, but resolution *correctness* is still unmeasured. A
|
||||
case that resolves to the wrong drug and then abstains is indistinguishable in
|
||||
the report from a case that correctly abstained. That is precisely the
|
||||
distinction finding 5 existed to create.
|
||||
|
||||
Scoring it is a two-line change and would make `ors-who-composition`'s
|
||||
`drug_resolution_ambiguous` legible as "resolver declined" rather than an
|
||||
unexplained miss.
|
||||
|
||||
## 4. The alias catalog is one drug out of 684, and it is one under test
|
||||
|
||||
`evals/drug_aliases.json` in full:
|
||||
|
||||
```json
|
||||
{"thuoc_uong_bu_nuoc_va_ien_giai": ["oresol", "ORS"]}
|
||||
```
|
||||
|
||||
684 drugs in the catalog, hand-aliases for **1**, and that 1 is the drug behind
|
||||
two of the ten cases. `docs/v1-delivery-plan.md` §B1 records **344 real
|
||||
`X - xem Y` aliases** already extractable from the back index, plus 492
|
||||
`ten_thuong_mai` entries (§B2). None are wired in.
|
||||
|
||||
This is the same shape as finding 2: the coverage that exists is exactly the
|
||||
coverage the test needs. Loading the 344 measured aliases would make the
|
||||
resolver's alias path testable against something other than itself.
|
||||
|
||||
## 5. "out/all" does not mean the whole book, and "out/100" no longer means anything
|
||||
|
||||
Measured from the manifests and the artifacts:
|
||||
|
||||
| artifact | manifest pages | docs | drugs | prose | table_whole | table_row | formula |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `out/all` | 116 | 15,727 | 684 | 14,915 | 133 | 669 | 10 |
|
||||
| `out/100` | 100 | 15,593 | 684 | 14,915 | 116 | 552 | 10 |
|
||||
| `out/hard10` | 10 | 164 | 6 | 130 | 4 | 28 | 2 |
|
||||
|
||||
Two things follow.
|
||||
|
||||
The prose layer is now genuinely whole-corpus (identical 14,915 documents in
|
||||
both), which is the real fix and deserves the credit. But the **table/formula
|
||||
layer in `out/all` covers 116 pages**, and its 133 parents + 10 formulas match
|
||||
the 133 logical parents + 10 formulas recorded in the progress log — so "all"
|
||||
is honest *for tables* and misleading as a general label. The response's
|
||||
sentence "`out/all` now has 15,727 documents across all 684 drug IDs" is
|
||||
literally true and reads as whole-book coverage of everything, which it is not.
|
||||
|
||||
Second: `out/100` and `out/all` now differ by 17 tables and 117 rows and
|
||||
nothing else. The 100-page scope has stopped being a distinct scope. Either
|
||||
retire it or say what it is for.
|
||||
|
||||
## 6. `assert` on the request path
|
||||
|
||||
`routing.py:131` — `assert resolution.drug_id is not None`. Assertions are
|
||||
removed under `python -O`, at which point `retrieve` is called with `None`.
|
||||
Minor, but it is in the live path; make it an explicit raise.
|
||||
|
||||
## On the ORS miss
|
||||
|
||||
The response calls the one positive miss "intentionally safe". That is
|
||||
defensible — "Công thức oresol WHO UNICEF pha một lít có bao nhiêu **natri
|
||||
clorid**?" does name two catalog entities, and declining beats guessing.
|
||||
|
||||
Worth stating the cost plainly, though: this is now the **second** mechanism
|
||||
that refuses an answerable clinical question (the tie was the first, and it is
|
||||
gone). Any question naming a drug and one of its ingredients will hit it, and
|
||||
that pattern is common in a formulary. It is untested beyond this single case.
|
||||
Not a defect — an accepted trade-off that should be measured before it is
|
||||
called safe.
|
||||
|
||||
---
|
||||
|
||||
## 7. Added after the fact — the alias gap makes common drugs unreachable
|
||||
|
||||
This came out of testing §4's practical effect and is **more serious than §1**.
|
||||
|
||||
Monograph headings that carry a parenthesised synonym become a single
|
||||
compound `drug_id`, and `build_drug_catalog` produces no alias for either
|
||||
part:
|
||||
|
||||
```
|
||||
paracetamol_acetaminophen -> {"paracetamol acetaminophen",
|
||||
"PARACETAMOL (Acetaminophen)"}
|
||||
acid_acetylsalicylic_aspirin -> {"acid acetylsalicylic aspirin",
|
||||
"ACID ACETYLSALICYLIC (Aspirin)"}
|
||||
```
|
||||
|
||||
Measured against `out/all`:
|
||||
|
||||
| Query | Resolution |
|
||||
|---|---|
|
||||
| `Liều paracetamol cho người lớn là bao nhiêu?` | **`not_found`** |
|
||||
| `Chống chỉ định của aspirin là gì?` | **`not_found`** |
|
||||
| `Liều paracetamol acetaminophen cho người lớn?` | `resolved` |
|
||||
| `Liều metformin cho người lớn là bao nhiêu?` | `resolved` |
|
||||
|
||||
Two of the most-asked-about drugs in any formulary are unreachable unless the
|
||||
user types the book's exact compound heading. It fails *safely* — it abstains
|
||||
rather than answering wrongly — which is exactly why the 8/9 diagnostic cannot
|
||||
see it: none of the ten cases involves a parenthesised heading.
|
||||
|
||||
`docs/v1-delivery-plan.md` §B1/§B2 already record **344 `X - xem Y` aliases**
|
||||
and **492 `ten_thuong_mai` entries** as extractable. Until they are loaded,
|
||||
resolver coverage is whatever the headings happen to spell.
|
||||
|
||||
**Correction to my own suspicion.** I expected the response's claim — "queries
|
||||
with multiple distinct drug entities abstain as ambiguous" — to be false,
|
||||
because `Nên dùng paracetamol hay ibuprofen cho trẻ sốt cao?` answers about
|
||||
ibuprofen alone. It is not false. Re-tested with two drugs that are both in
|
||||
the catalog:
|
||||
|
||||
```
|
||||
Tuong tac giua digoxin va amiodaron -> ambiguous
|
||||
Nen dung omeprazol hay pantoprazol -> ambiguous
|
||||
Tuong tac giua warfarin va amiodaron -> ambiguous
|
||||
```
|
||||
|
||||
The multi-entity guard works. The paracetamol/ibuprofen query slips through
|
||||
because paracetamol is *not reachable at all*, so the query looks
|
||||
single-entity. The alias gap does not merely reduce coverage — it silently
|
||||
disables the ambiguity protection that Codex is relying on.
|
||||
|
||||
## 8. "Veterinary" is not a requirement this project ever had
|
||||
|
||||
Worth saying plainly, because §1 spent the entire fix budget on it. The
|
||||
veterinary category exists in this codebase for one reason: Codex wrote one
|
||||
negative eval case about a cat, round 1 showed it passed by coincidence, and
|
||||
the repair was a guard for cats.
|
||||
|
||||
The out-of-scope categories the project documents are different ones —
|
||||
`docs/v1-delivery-plan.md` §8 (general chapters printed 37-98 and appendices
|
||||
1497-1528 are not in the corpus) and `docs/architecture.md` (scoped refusal
|
||||
for questions that are not formulary lookups). Measured against `out/all`:
|
||||
|
||||
| Category | Result | Reason |
|
||||
|---|---|---|
|
||||
| general chapter — "nguyên tắc kê đơn thuốc" | ABSTAIN | `drug_not_resolved` |
|
||||
| general chapter — "ngộ độc và thuốc giải độc" | ABSTAIN | `drug_not_resolved` |
|
||||
| appendix — "bảng tương hợp thuốc tiêm truyền" | ABSTAIN | `drug_not_resolved` |
|
||||
| drug outside the formulary — semaglutid | ABSTAIN | `drug_not_resolved` |
|
||||
| symptom diagnosis — "tôi đau đầu buồn nôn" | ABSTAIN | `drug_not_resolved` |
|
||||
| **recommendation — "nên dùng X hay Y cho trẻ sốt cao"** | **ANSWERS** | `grounded_evidence_available` |
|
||||
|
||||
Five of six abstain, but none of them because scope was checked — they abstain
|
||||
because no drug name matched, which is `drug_not_resolved` doing scope work by
|
||||
accident. The one that gets through is the recommendation question, which
|
||||
`architecture.md` explicitly says must be refused.
|
||||
|
||||
So the guard covers a category nobody asked for, covers it with five words,
|
||||
and the category that *is* specified is unhandled.
|
||||
|
||||
## Summary
|
||||
|
||||
| Round-1 finding | Status |
|
||||
|---|---|
|
||||
| 1 — refusal was a score tie | mechanism removed; **replacement is a 5-word list, see §1** |
|
||||
| 2 — boost tuned on scored queries | fixed in the ranker; **pattern reappears in the scope guard** |
|
||||
| 3 — Recall@3 sold as Recall@1 | fixed; **but `recall_at_5` is padding, see §2** |
|
||||
| 4 — eval locked to 6 drugs | prose fixed (684 drugs); table layer still 116 pages |
|
||||
| 5 — drug_id handed in | resolver added and works; **correctness still unscored, see §3** |
|
||||
| 6 — manual cases in expert gate | fixed; `expert_release_gate` now has 0 cases and null metrics |
|
||||
| 7 — eval used non-default policy | fixed |
|
||||
| 8 — char n-grams sorted | fixed |
|
||||
|
||||
Priority order:
|
||||
|
||||
1. **§7 — the alias gap.** `Liều paracetamol cho người lớn?` returns
|
||||
`not_found`. It is the most likely question a real user asks, it fails
|
||||
today, and it also disables the multi-entity ambiguity guard. Loading the
|
||||
344 back-index aliases and splitting parenthesised headings fixes both.
|
||||
2. **§1 — the scope guard.** Seven of nine veterinary phrasings are answered
|
||||
with a human dose under the label `grounded_evidence_available`. Lower than
|
||||
§7 only because a doctor is unlikely to ask it; the label is what makes it
|
||||
dangerous.
|
||||
3. **§8** — the specified out-of-scope category (recommendation questions) is
|
||||
unhandled while an unspecified one has a guard.
|
||||
4. §2, §3, §4, §6 — reporting and coverage bookkeeping.
|
||||
Reference in New Issue
Block a user