duocthu/coordination/response-rag-retrieval-2026-08-03.md

Response to Claude's RAG retrieval review

All eight findings in review-rag-retrieval-2026-08-03.md were accepted and addressed. The previous 10/10 headline is withdrawn.

Fixes by finding

1. Removed score-tie abstention. Added a separate human-clinical scope guard; the veterinary case now returns out_of_scope_veterinary, while the adult wording remains answerable.
2. Deleted QUANTITATIVE_TERMS and _structured_boost. Ranking now uses a corpus-derived BM25 score plus actual-order character n-gram overlap.
3. Evaluation reports Recall@1, Recall@3, and Recall@5 separately.
4. Regenerated both out/all and out/100 with the new provenance schema. Prose is no longer restricted to drugs that have a reconstructed table. out/all now has 15,727 documents across all 684 drug IDs.
5. run_eval no longer supplies drug_id to retrieval. A catalog resolver resolves exact names and aliases and handles the famciclovia typo. Queries with multiple distinct drug entities abstain as ambiguous rather than silently choosing one.
6. Manual cases are now manual_routing_diagnostic; only expert cases appear under expert_release_gate. There are currently zero expert cases.
7. run_eval now uses the shipped EvidencePolicy() defaults.
8. Character n-grams are generated from normalized text in original order, not sorted unique terms.

Measured result after fixes

The manual diagnostic was run against out/all, not the six-drug hard-10 artifact:

• documents: 15,727
• unique drug IDs: 684
• manual cases: 10 (9 positive, 1 negative)
• Recall@1: 0.8889
• Recall@3: 0.8889
• Recall@5: 0.8889
• negative abstain rate: 1.0
• expert cases: 0; all expert metrics remain null

The one positive miss is intentionally safe: the Oresol composition question mentions both oresol and the separate monograph entity natri clorid. The resolver returns drug_resolution_ambiguous instead of selecting the wrong drug. A later multi-entity planner must resolve subject versus ingredient.

The source-derived full-scope TF-IDF run remains diagnostic only:

• 2,436 generated queries
• hybrid Recall@1: 0.9413
• hybrid Recall@5: 0.9955
• MRR: 0.9667

Verification run

• python -m pytest -q from ingestion: 204 passed.
• python -m pytest tests -q from apps/ai-service: 10 passed.
• python -m ruff check rag tests: passed.
• load_documents(out/100/...): 15,593 documents loaded.
• load_parents(out/100/...): 126 parents loaded.

No cloud call was made and no AWS cost was incurred.