duocthu/docs-legacy/adr/0008-llm-understanding-one-shot-rag.md

# ADR 0008: LLM query understanding + one-shot grounded RAG (what is actually live)

**Status:** accepted, live since 2026-08-06 (F-03), extended 2026-08-07
**Supersedes:** ADR 0007 (conversational reasoning RAG — the `Focus`/
`ConversationState`/TTL state design and the PLAN/RETRIEVE/ASSESS/REFINE/
VERIFY bounded loop). ADR 0007's own `rag/conversation.py`/`rag/reasoning.py`/
`rag/conversational.py` were deleted 2026-08-07 once confirmed unreachable
from `bootstrap.py` — see the notice at the top of ADR 0007 for the full
reasoning.
**Extends:** ADR 0006 (quarantined block references) — unchanged and still
binding: a chunk with `has_quarantined_content` still forces `VERIFY_PDF`
and is never generated over.

## Context

This ADR exists because `docs/architecture.md` and ADR 0007 described a
design that was never fully built, and the modules that partially
implemented it were never wired into `bootstrap.py`. A 2026-08-06
independent 7-agent audit found this the hard way — it cost real time
establishing that `QdrantRetriever.search()` (dense vector search) and the
entire reasoning-loop module set were dead code, contradicting what the
docs claimed was live. The fix is not "finish building ADR 0007" — the
project deliberately moved to a simpler design that already works, proven
across many real multi-turn conversations (see `docs/progress-log.md`,
2026-08-05 through 2026-08-07 entries). This ADR documents that design so
the next reader doesn't have to re-discover it by audit.

## Decision

### 1. One LLM call understands the turn; no separate state object

`rag/understanding.py::LlmQueryUnderstander.understand(turn, history)` reads
the raw current turn plus a **plain list of past turn strings**
(`"Người dùng: …"` / `"Trợ lý: …"`, kept by `RagAgent._history`, a
per-conversation-id in-process dict) and returns a `QueryFrame`: turn type,
resolved `drug_id`s (validated against a candidate set a deterministic
fuzzy/alias pass bounds *before* the model runs — F-04), section attribute,
population, weight, age, indication, route, and a `needs_clarify`/
`clarify_reason`/`quick_replies` triple.

There is no `Focus` struct, no TTL, no separate summariser. The model
re-reads the same history window (last `HISTORY_TURNS * 2` = 12 lines) every
turn and re-derives what's still relevant — cheaper to build and, so far,
more robust than hand-maintained state: it naturally handles "còn trẻ em thì
sao?" and short replies to its own clarify questions (population/route/etc.
— the latter only after a 2026-08-07 fix; see progress-log) without a
resolver state machine to keep in sync.

**Known gap, inherited from ADR 0007 and still open:** this history is an
in-process dict — lost on restart, not shared across workers if the service
ever scales beyond one. ADR 0007's `PostgresConversationStore` was never
built either.

### 2. Routing is a single dispatch, not a loop

`RagAgent._route()` reads `frame.turn_type` and dispatches once:
`interaction` (2+ drugs) → gather each drug's evidence, combine, decide;
`drug_attribute`/`drug_overview`/`dosing_calc`/fallback → one drug, one
retrieval call; `smalltalk`/`out_of_scope` → canned reply, no retrieval;
`symptom_to_drug` with no drug named → an honest "not built yet" clarify.
There is no PLAN/REFINE step and no retrieval-round budget, because there is
only ever one retrieval call per turn.

### 3. Retrieval is deterministic routing, not similarity ranking

`RetrievalService.retrieve_framed(drug_id, section_key, query)`:
- `section_key` given (the dominant case, since `understand()` almost always
  resolves it) → `find_by_section`, an **exact Qdrant payload filter**
  (`drug_id` + `section_key`), returning the whole section as a scroll.
  Score is a hardcoded 1.0 — this is a filter, not a ranked search, and nothing
  here is "confidence" in the sense ADR 0007's retrieval-confidence gate meant.
- No section resolved → `find_by_drug` (whole monograph, book order),
  trimmed to identity sections for a bare name or reranked (Cohere
  cross-encoder over the ~29 sections of that one drug, not a corpus search)
  for a free-form question.
- `QdrantRetriever.search()` — real dense vector similarity over the whole
  corpus — exists and is unit-tested, but `RagAgent` never calls it. It is
  reachable only through the legacy `RetrievalService.retrieve()` entry
  point, itself only reachable when `ANSWER_PROVIDER=disabled` (no agent
  configured at all — retrieval-only mode). `docs/architecture.md`'s
  "Retrieval-confidence gate: below a similarity threshold, skip the LLM
  call entirely" describes this legacy-only path, not the live one; that
  section has been corrected to say so.
- Measured, and the reason this design was chosen over similarity ranking
  for the live path: routing by exact `section_key` moved contraindication
  hit@1 from 0.05 to 1.00 (`[[project-retrieval-quality-gap]]`, 2026-08-04).
  A quarantined chunk anywhere in the retrieved set still forces the whole
  result to `VERIFY_PDF` (`RetrievalService.decide`, a public wrapper added
  2026-08-07 so `RagAgent._interaction` applies the same policy to a
  combined multi-drug evidence pool instead of hand-rolling it).

### 4. Generation is one call, verified twice, with no confidence score

`GroundedAnswerService.answer_from_result`: sufficiency-check (ask instead of
guessing when the evidence spans multiple populations/routes and the turn
hasn't disambiguated) → generate → `grounding.verify` (every number and
citation traces to the block it cites) → `_verify_entailment` (a second LLM
pass confirming each cited claim's *content*, not just its numbers, is
actually stated by that block; one same-claim retry on a lone reject, since
this call is measurably noisy — 2026-08-06 finding). A generation that fails
any check **abstains** — it does not fall back to a raw extractive quote
when a generator is configured (`[[feedback_no_extractive_fallback_when_llm_configured]]`).

No `MAX_LLM_CALLS`/`MAX_WALL_CLOCK_MS` budget object exists. Each call is
bounded only by its own provider timeout. **This is ADR 0007's F-08 finding,
inherited unchanged and still open** — a real end-to-end request deadline
threaded through `RagAgent`'s sequence of up to 5 sequential Bedrock calls
(understand → sufficiency → generate → ≤2 entailment) is real remaining
work, not solved by this ADR. Measured live 2026-08-07: a single answerable
turn costs ~8-9s wall clock, ~75-80% of it the 4 sequential LLM calls
(understand ~2.6-3.3s dominates — an 80B model doing a classification task
that likely doesn't need one); a clarify chain compounds this linearly since
each round is a fresh request repeating the same call sequence from scratch.

### 5. Context resolved across turns is folded into one self-contained string

Added 2026-08-07, closing a P0 the 2026-08-06 audit named: `frame.population`/
`weight_kg`/`age_text`/`route`/`indication` were extracted by `understand()`
but never reached `retrieve_framed`/`answer_from_result`, which took only
the bare current-turn text — so a reply like "Uống" three turns into a dose
conversation reached the sufficiency/generation LLM calls as literally just
"Uống", with no notion that population=adult was already established two
turns back. `RagAgent._synthesize_query` now folds every resolved field into
one string (`"Uống. Đối tượng: người lớn. Đường dùng: uống."`) before it
reaches retrieval's rerank signal and generation's `query` argument. No-op
for a fresh single-shot question that already states its own context.

## Consequences

**Accepted.** No confidence score, no retrieval-round budget, no PLAN/REFINE
step — the tradeoff ADR 0007 explicitly refused ("an LLM confidence score...
is not a defensible basis for asking or not asking a clinician a question")
is exactly what this design uses instead (an LLM sufficiency/clarify
judgment), because in practice it has been reliable enough and dramatically
simpler to build, extend (route/quick_replies were one schema field + one
prompt rule each, not a new state machine), and debug — every session this
month that touched the ADR 0007 modules found new bugs in the state-machine
edges (TTL boundaries, Focus inheritance correctness) rather than in the
domain logic itself.

**Refused (again, restated from ADR 0007, still true):** an LLM confidence
score as a hard gate for retrieval — `RetrievalService.decide`'s
`VERIFY_PDF`/`ABSTAIN` decisions remain deterministic (quarantine flag,
missing provenance), never a model's self-reported certainty.

**Still open, named rather than hidden:**
- No request-scoped time/call budget (F-08).
- Conversation history is in-process, not durable/shared (inherited from
  ADR 0007, never built either way).
- No production-path adversarial regression suite beyond one live-verified
  end-to-end case (F-10's remaining scope).
- `dosing_calc` (a real mg/kg calculator) and `symptom_to_drug` (reverse
  indication lookup) remain honest "not ready" clarifies, not answers.