duocthu/docs-legacy/adr/0008-llm-understanding-one-shot-rag.md

ADR 0008: LLM query understanding + one-shot grounded RAG (what is actually live)

Status: accepted, live since 2026-08-06 (F-03), extended 2026-08-07 Supersedes: ADR 0007 (conversational reasoning RAG — the Focus/ ConversationState/TTL state design and the PLAN/RETRIEVE/ASSESS/REFINE/ VERIFY bounded loop). ADR 0007's own rag/conversation.py/rag/reasoning.py/ rag/conversational.py were deleted 2026-08-07 once confirmed unreachable from bootstrap.py — see the notice at the top of ADR 0007 for the full reasoning. Extends: ADR 0006 (quarantined block references) — unchanged and still binding: a chunk with has_quarantined_content still forces VERIFY_PDF and is never generated over.

Context

This ADR exists because docs/architecture.md and ADR 0007 described a design that was never fully built, and the modules that partially implemented it were never wired into bootstrap.py. A 2026-08-06 independent 7-agent audit found this the hard way — it cost real time establishing that QdrantRetriever.search() (dense vector search) and the entire reasoning-loop module set were dead code, contradicting what the docs claimed was live. The fix is not "finish building ADR 0007" — the project deliberately moved to a simpler design that already works, proven across many real multi-turn conversations (see docs/progress-log.md, 2026-08-05 through 2026-08-07 entries). This ADR documents that design so the next reader doesn't have to re-discover it by audit.

Decision

1. One LLM call understands the turn; no separate state object

rag/understanding.py::LlmQueryUnderstander.understand(turn, history) reads the raw current turn plus a plain list of past turn strings ("Người dùng: …" / "Trợ lý: …", kept by RagAgent._history, a per-conversation-id in-process dict) and returns a QueryFrame: turn type, resolved drug_ids (validated against a candidate set a deterministic fuzzy/alias pass bounds before the model runs — F-04), section attribute, population, weight, age, indication, route, and a needs_clarify/ clarify_reason/quick_replies triple.

There is no Focus struct, no TTL, no separate summariser. The model re-reads the same history window (last HISTORY_TURNS * 2 = 12 lines) every turn and re-derives what's still relevant — cheaper to build and, so far, more robust than hand-maintained state: it naturally handles "còn trẻ em thì sao?" and short replies to its own clarify questions (population/route/etc. — the latter only after a 2026-08-07 fix; see progress-log) without a resolver state machine to keep in sync.

Known gap, inherited from ADR 0007 and still open: this history is an in-process dict — lost on restart, not shared across workers if the service ever scales beyond one. ADR 0007's PostgresConversationStore was never built either.

2. Routing is a single dispatch, not a loop

RagAgent._route() reads frame.turn_type and dispatches once: interaction (2+ drugs) → gather each drug's evidence, combine, decide; drug_attribute/drug_overview/dosing_calc/fallback → one drug, one retrieval call; smalltalk/out_of_scope → canned reply, no retrieval; symptom_to_drug with no drug named → an honest "not built yet" clarify. There is no PLAN/REFINE step and no retrieval-round budget, because there is only ever one retrieval call per turn.

3. Retrieval is deterministic routing, not similarity ranking

RetrievalService.retrieve_framed(drug_id, section_key, query):

• section_key given (the dominant case, since understand() almost always resolves it) → find_by_section, an exact Qdrant payload filter (drug_id + section_key), returning the whole section as a scroll. Score is a hardcoded 1.0 — this is a filter, not a ranked search, and nothing here is "confidence" in the sense ADR 0007's retrieval-confidence gate meant.
• No section resolved → find_by_drug (whole monograph, book order), trimmed to identity sections for a bare name or reranked (Cohere cross-encoder over the ~29 sections of that one drug, not a corpus search) for a free-form question.
• QdrantRetriever.search() — real dense vector similarity over the whole corpus — exists and is unit-tested, but RagAgent never calls it. It is reachable only through the legacy RetrievalService.retrieve() entry point, itself only reachable when ANSWER_PROVIDER=disabled (no agent configured at all — retrieval-only mode). docs/architecture.md's "Retrieval-confidence gate: below a similarity threshold, skip the LLM call entirely" describes this legacy-only path, not the live one; that section has been corrected to say so.
• Measured, and the reason this design was chosen over similarity ranking for the live path: routing by exact section_key moved contraindication hit@1 from 0.05 to 1.00 ([[project-retrieval-quality-gap]], 2026-08-04). A quarantined chunk anywhere in the retrieved set still forces the whole result to VERIFY_PDF (RetrievalService.decide, a public wrapper added 2026-08-07 so RagAgent._interaction applies the same policy to a combined multi-drug evidence pool instead of hand-rolling it).

4. Generation is one call, verified twice, with no confidence score

GroundedAnswerService.answer_from_result: sufficiency-check (ask instead of guessing when the evidence spans multiple populations/routes and the turn hasn't disambiguated) → generate → grounding.verify (every number and citation traces to the block it cites) → _verify_entailment (a second LLM pass confirming each cited claim's content, not just its numbers, is actually stated by that block; one same-claim retry on a lone reject, since this call is measurably noisy — 2026-08-06 finding). A generation that fails any check abstains — it does not fall back to a raw extractive quote when a generator is configured ([[feedback_no_extractive_fallback_when_llm_configured]]).

No MAX_LLM_CALLS/MAX_WALL_CLOCK_MS budget object exists. Each call is bounded only by its own provider timeout. This is ADR 0007's F-08 finding, inherited unchanged and still open — a real end-to-end request deadline threaded through RagAgent's sequence of up to 5 sequential Bedrock calls (understand → sufficiency → generate → ≤2 entailment) is real remaining work, not solved by this ADR. Measured live 2026-08-07: a single answerable turn costs ~8-9s wall clock, ~75-80% of it the 4 sequential LLM calls (understand ~2.6-3.3s dominates — an 80B model doing a classification task that likely doesn't need one); a clarify chain compounds this linearly since each round is a fresh request repeating the same call sequence from scratch.

5. Context resolved across turns is folded into one self-contained string

Added 2026-08-07, closing a P0 the 2026-08-06 audit named: frame.population/ weight_kg/age_text/route/indication were extracted by understand() but never reached retrieve_framed/answer_from_result, which took only the bare current-turn text — so a reply like "Uống" three turns into a dose conversation reached the sufficiency/generation LLM calls as literally just "Uống", with no notion that population=adult was already established two turns back. RagAgent._synthesize_query now folds every resolved field into one string ("Uống. Đối tượng: người lớn. Đường dùng: uống.") before it reaches retrieval's rerank signal and generation's query argument. No-op for a fresh single-shot question that already states its own context.

Consequences

Accepted. No confidence score, no retrieval-round budget, no PLAN/REFINE step — the tradeoff ADR 0007 explicitly refused ("an LLM confidence score... is not a defensible basis for asking or not asking a clinician a question") is exactly what this design uses instead (an LLM sufficiency/clarify judgment), because in practice it has been reliable enough and dramatically simpler to build, extend (route/quick_replies were one schema field + one prompt rule each, not a new state machine), and debug — every session this month that touched the ADR 0007 modules found new bugs in the state-machine edges (TTL boundaries, Focus inheritance correctness) rather than in the domain logic itself.

Refused (again, restated from ADR 0007, still true): an LLM confidence score as a hard gate for retrieval — RetrievalService.decide's VERIFY_PDF/ABSTAIN decisions remain deterministic (quarantine flag, missing provenance), never a model's self-reported certainty.

Still open, named rather than hidden:

• No request-scoped time/call budget (F-08).
• Conversation history is in-process, not durable/shared (inherited from ADR 0007, never built either way).
• No production-path adversarial regression suite beyond one live-verified end-to-end case (F-10's remaining scope).
• dosing_calc (a real mg/kg calculator) and symptom_to_drug (reverse indication lookup) remain honest "not ready" clarifies, not answers.