141 lines
7.1 KiB
Markdown
141 lines
7.1 KiB
Markdown
# 19 — RAG evaluation
|
||
|
||
## What exists
|
||
|
||
| Asset | Location | Size | Runner |
|
||
|---|---|---|---|
|
||
| Retrieval eval harness | `rag/run_eval.py` | — | Manual CLI; uses in-memory stores, **not** Qdrant |
|
||
| Retrieval eval types | `rag/evaluation.py` | — | — |
|
||
| Condition→drug metrics | `rag/condition_evaluation.py` | — | **No runner** — only `tests/test_condition_evaluation.py` |
|
||
| Adversarial hard set | `evals/manual_adversarial_hard10.jsonl` | 10 cases | No automated runner |
|
||
| Condition→drug set | `evals/condition_to_drug_v1.jsonl` | 20 cases | No automated runner |
|
||
| Production battery | `evals/production_manual_60.jsonl` | 60 cases | `scripts/run_manual_battery.py` (live HTTP) |
|
||
| Golden datasets | `Golden Dataset/*.csv` | 5 files, 209 rows | **No runner anywhere** |
|
||
| Deploy smoke assertion | `.github/workflows/deploy.yml` | 1 case | Runs on every deploy |
|
||
|
||
## The datasets
|
||
|
||
### `Golden Dataset/` — hand-labelled, Vietnamese, unwired
|
||
|
||
| File | Rows | Columns |
|
||
|---|---|---|
|
||
| `golden_intent_v1.csv` | 73 | question, correct intent, labelling rationale, group, difficulty |
|
||
| `golden_entity_v1.csv` | 50 | question, correct drug, correct attribute, disease, symptom |
|
||
| `golden_e2e_v1.csv` | 36 | scenario, question, expected intent/drug/attribute, **required content**, expected citation, pass criteria, actual-result column |
|
||
| `golden_summary_v1.csv` | 32 | drug, attribute, source page, **verbatim source text**, meanings that must be preserved, numbers that must be copied exactly, max length, faithfulness / coverage / readability scores 0–2 |
|
||
| `golden_multiturn_v1.csv` | 19 | conversation id, turn, question, expected behaviour, expected drug/section/population, what should be inherited |
|
||
|
||
These are genuinely useful — `golden_summary_v1.csv` carries the exact source
|
||
paragraph and the exact numbers that must survive, which is precisely the
|
||
property `grounding.verify` enforces. But **no code in the repository reads
|
||
them**. The scoring columns are blank, i.e. filled in by hand.
|
||
|
||
### `evals/production_manual_60.jsonl`
|
||
|
||
The most structured set. Each case declares observable invariants:
|
||
|
||
```json
|
||
{"id":"G01","category":"general_condition","query":"Tăng huyết áp dùng thuốc gì?",
|
||
"decision":"answerable","condition_mode":"general",
|
||
"expected_any_drug_ids":["methyldopa","quinapril","labetalol_hydroclorid"],
|
||
"must_have_citations":true,"max_drugs":8}
|
||
```
|
||
|
||
`scripts/run_manual_battery.py` is deliberately **a transparent HTTP recorder,
|
||
not an LLM judge** — it posts each case to a running service and writes every
|
||
full response to JSONL for human review against the rendered PDF pages. The
|
||
docstring states the rationale: each case has observable invariants (decision,
|
||
relation/section, candidate bound, citations, drug provenance), so an exact
|
||
comparison is auditable in a way a judge model is not.
|
||
|
||
### `evals/condition_to_drug_v1.jsonl`
|
||
|
||
20 cases with `expected_intent`, `expected_condition`, `expected_relation`,
|
||
`expected_clarification` — designed for `condition_evaluation.py`'s metrics.
|
||
|
||
### `evals/manual_adversarial_hard10.jsonl`
|
||
|
||
10 hard cases, each targeting a known parsing hazard — cross-page contrast
|
||
dosing, a formula with no printed fraction bar — with `expected_drug_id` and an
|
||
`expected_id` pointing at a specific block.
|
||
|
||
## Metrics the code can compute
|
||
|
||
`rag/condition_evaluation.py::summarize_condition_outcomes` is fully implemented
|
||
and deterministic — no judge model:
|
||
|
||
| Metric | Definition |
|
||
|---|---|
|
||
| `intent_accuracy` | exact match on turn type |
|
||
| `condition_normalization_accuracy` | exact match on the normalised condition |
|
||
| `ambiguity_clarification_accuracy` | did it clarify exactly when it should |
|
||
| `indication_recall_at_8` | any expected drug in the top-8 retrieved |
|
||
| `drug_precision_at_8` | expected ∩ retrieved / retrieved |
|
||
| `section_correctness` | every retrieved section is `chi_dinh` |
|
||
| `relation_correctness` | indication vs adverse-effect vs contraindication |
|
||
| `unsupported_drug_rate` | generated drugs not present in retrieval |
|
||
| `citation_correctness` | mean over per-citation validity flags |
|
||
| `groundedness` | mean over per-claim grounded flags |
|
||
| `patient_context_extraction_accuracy` | field-by-field match on `PatientContext` |
|
||
| `safety_evidence_retrieval_accuracy` | expected safety facets actually retrieved |
|
||
|
||
`rag/evaluation.py::summarize` covers retrieval-only outcomes (drug resolution
|
||
status and retrieved-id match).
|
||
|
||
**Neither summariser has a production runner.** `run_eval.py` uses
|
||
`InMemoryLexicalRetriever` over JSONL artifacts, so it measures the resolver and
|
||
the section router — not the deployed Qdrant retrieval.
|
||
|
||
## What is *not* measured anywhere
|
||
|
||
| Standard RAG metric | State |
|
||
|---|---|
|
||
| Retrieval recall@k / precision@k against the live corpus | **Not found** — the code exists for condition→drug only, with no runner |
|
||
| MRR / NDCG | **Not found** |
|
||
| Hit-rate on the section route | Measured once by hand (0.544 overall, 0.05 on `chong_chi_dinh` for the *similarity* route, 2026-08-04) — that number is recorded in code comments and ADRs, not reproducible by any committed script |
|
||
| Faithfulness / answer correctness scoring | Manual only (`golden_summary_v1.csv` columns) |
|
||
| LLM-as-judge | **Deliberately absent** — `condition_evaluation.py` says so explicitly |
|
||
| Latency distribution | Measured by hand once (n=8), recorded in `ChatPanel.tsx` |
|
||
| Regression gate in CI | **Not found** — nothing blocks a merge on eval results |
|
||
|
||
## The one automated quality gate
|
||
|
||
`.github/workflows/deploy.yml` runs, on every deploy, a single condition→drug
|
||
case:
|
||
|
||
```
|
||
query: "Đợt gout cấp có thuốc nào được Dược thư ghi chỉ định?"
|
||
assert: response contains "decision":"answerable"
|
||
assert: response contains "section_key":"chi_dinh"
|
||
```
|
||
|
||
Plus a second query used to assert trace propagation. If either fails, the
|
||
deploy fails and the last 200 lines of `ai-service` logs are dumped. This is a
|
||
smoke test on one behaviour, not an evaluation — but it is the only quality
|
||
assertion that runs without a human.
|
||
|
||
## Honest assessment
|
||
|
||
The repository has **good evaluation *material*** and **no evaluation *system***.
|
||
209 hand-labelled golden rows, 90 JSONL cases and two implemented deterministic
|
||
metric summarisers exist; the wiring between them — a runner that executes a set
|
||
against the live service, computes the metrics and compares against a baseline —
|
||
does not.
|
||
|
||
Consequently, no claim of the form "retrieval quality is X" or "the system is
|
||
production-ready because it passes evaluation" can be supported from this
|
||
repository today. What *can* be supported is that the safety mechanisms are
|
||
unit-tested (555 tests) and that one end-to-end behaviour is asserted on every
|
||
deploy.
|
||
|
||
## Suggested minimum wiring (from what already exists)
|
||
|
||
1. A runner that feeds `evals/condition_to_drug_v1.jsonl` through the live
|
||
service into `summarize_condition_outcomes` and prints the metric table.
|
||
2. A CSV reader for `Golden Dataset/golden_e2e_v1.csv` that fills its
|
||
`ket_qua_thuc_te` column automatically.
|
||
3. A stored baseline plus a threshold comparison so a regression fails a check
|
||
rather than being noticed in production.
|
||
|
||
All three are new code; none requires new design.
|