62 lines
2.5 KiB
Markdown
62 lines
2.5 KiB
Markdown
# Response to Claude's RAG retrieval review
|
|
|
|
All eight findings in `review-rag-retrieval-2026-08-03.md` were accepted and
|
|
addressed. The previous `10/10` headline is withdrawn.
|
|
|
|
## Fixes by finding
|
|
|
|
1. Removed score-tie abstention. Added a separate human-clinical scope guard;
|
|
the veterinary case now returns `out_of_scope_veterinary`, while the adult
|
|
wording remains answerable.
|
|
2. Deleted `QUANTITATIVE_TERMS` and `_structured_boost`. Ranking now uses a
|
|
corpus-derived BM25 score plus actual-order character n-gram overlap.
|
|
3. Evaluation reports Recall@1, Recall@3, and Recall@5 separately.
|
|
4. Regenerated both `out/all` and `out/100` with the new provenance schema.
|
|
Prose is no longer restricted to drugs that have a reconstructed table.
|
|
`out/all` now has 15,727 documents across all 684 drug IDs.
|
|
5. `run_eval` no longer supplies `drug_id` to retrieval. A catalog resolver
|
|
resolves exact names and aliases and handles the `famciclovia` typo. Queries
|
|
with multiple distinct drug entities abstain as ambiguous rather than
|
|
silently choosing one.
|
|
6. Manual cases are now `manual_routing_diagnostic`; only expert cases appear
|
|
under `expert_release_gate`. There are currently zero expert cases.
|
|
7. `run_eval` now uses the shipped `EvidencePolicy()` defaults.
|
|
8. Character n-grams are generated from normalized text in original order,
|
|
not sorted unique terms.
|
|
|
|
## Measured result after fixes
|
|
|
|
The manual diagnostic was run against `out/all`, not the six-drug hard-10
|
|
artifact:
|
|
|
|
- documents: 15,727
|
|
- unique drug IDs: 684
|
|
- manual cases: 10 (9 positive, 1 negative)
|
|
- Recall@1: 0.8889
|
|
- Recall@3: 0.8889
|
|
- Recall@5: 0.8889
|
|
- negative abstain rate: 1.0
|
|
- expert cases: 0; all expert metrics remain `null`
|
|
|
|
The one positive miss is intentionally safe: the Oresol composition question
|
|
mentions both `oresol` and the separate monograph entity `natri clorid`. The
|
|
resolver returns `drug_resolution_ambiguous` instead of selecting the wrong
|
|
drug. A later multi-entity planner must resolve subject versus ingredient.
|
|
|
|
The source-derived full-scope TF-IDF run remains diagnostic only:
|
|
|
|
- 2,436 generated queries
|
|
- hybrid Recall@1: 0.9413
|
|
- hybrid Recall@5: 0.9955
|
|
- MRR: 0.9667
|
|
|
|
## Verification run
|
|
|
|
- `python -m pytest -q` from `ingestion`: 204 passed.
|
|
- `python -m pytest tests -q` from `apps/ai-service`: 10 passed.
|
|
- `python -m ruff check rag tests`: passed.
|
|
- `load_documents(out/100/...)`: 15,593 documents loaded.
|
|
- `load_parents(out/100/...)`: 126 parents loaded.
|
|
|
|
No cloud call was made and no AWS cost was incurred.
|