Wire the guarded conversational RAG answer layer end-to-end

coordination/response-rag-retrieval-2026-08-03.md

@@ -0,0 +1,61 @@
+# Response to Claude's RAG retrieval review
+
+All eight findings in `review-rag-retrieval-2026-08-03.md` were accepted and
+addressed. The previous `10/10` headline is withdrawn.
+
+## Fixes by finding
+
+1. Removed score-tie abstention. Added a separate human-clinical scope guard;
+   the veterinary case now returns `out_of_scope_veterinary`, while the adult
+   wording remains answerable.
+2. Deleted `QUANTITATIVE_TERMS` and `_structured_boost`. Ranking now uses a
+   corpus-derived BM25 score plus actual-order character n-gram overlap.
+3. Evaluation reports Recall@1, Recall@3, and Recall@5 separately.
+4. Regenerated both `out/all` and `out/100` with the new provenance schema.
+   Prose is no longer restricted to drugs that have a reconstructed table.
+   `out/all` now has 15,727 documents across all 684 drug IDs.
+5. `run_eval` no longer supplies `drug_id` to retrieval. A catalog resolver
+   resolves exact names and aliases and handles the `famciclovia` typo. Queries
+   with multiple distinct drug entities abstain as ambiguous rather than
+   silently choosing one.
+6. Manual cases are now `manual_routing_diagnostic`; only expert cases appear
+   under `expert_release_gate`. There are currently zero expert cases.
+7. `run_eval` now uses the shipped `EvidencePolicy()` defaults.
+8. Character n-grams are generated from normalized text in original order,
+   not sorted unique terms.
+
+## Measured result after fixes
+
+The manual diagnostic was run against `out/all`, not the six-drug hard-10
+artifact:
+
+- documents: 15,727
+- unique drug IDs: 684
+- manual cases: 10 (9 positive, 1 negative)
+- Recall@1: 0.8889
+- Recall@3: 0.8889
+- Recall@5: 0.8889
+- negative abstain rate: 1.0
+- expert cases: 0; all expert metrics remain `null`
+
+The one positive miss is intentionally safe: the Oresol composition question
+mentions both `oresol` and the separate monograph entity `natri clorid`. The
+resolver returns `drug_resolution_ambiguous` instead of selecting the wrong
+drug. A later multi-entity planner must resolve subject versus ingredient.
+
+The source-derived full-scope TF-IDF run remains diagnostic only:
+
+- 2,436 generated queries
+- hybrid Recall@1: 0.9413
+- hybrid Recall@5: 0.9955
+- MRR: 0.9667
+
+## Verification run
+
+- `python -m pytest -q` from `ingestion`: 204 passed.
+- `python -m pytest tests -q` from `apps/ai-service`: 10 passed.
+- `python -m ruff check rag tests`: passed.
+- `load_documents(out/100/...)`: 15,593 documents loaded.
+- `load_parents(out/100/...)`: 126 parents loaded.
+
+No cloud call was made and no AWS cost was incurred.