Add read-only production runtime audit
This commit is contained in:
@@ -0,0 +1,210 @@
|
||||
# ADR 0007: Conversational reasoning RAG — state, bounded loop, and how it is measured
|
||||
|
||||
**Status:** superseded by ADR 0008 (2026-08-07). See the note below before
|
||||
reading this as a description of anything currently running.
|
||||
**Supersedes:** nothing. Extends ADR 0005 (segment output contract) and ADR 0006
|
||||
(quarantined block references) rather than replacing them.
|
||||
|
||||
> **2026-08-07 — why this was superseded, not deleted.** An independent
|
||||
> 7-agent audit on 2026-08-06 found `bootstrap.py` never constructs any of
|
||||
> `rag/conversation.py` / `rag/reasoning.py` / `rag/conversational.py` — the
|
||||
> live agent (`rag/agent.py::RagAgent`, wired in since the F-03 rebuild on
|
||||
> 2026-08-06) is a fixed one-shot pipeline (understand → route → retrieve
|
||||
> once → generate → ≤2 same-claim entailment retries), not the PLAN/RETRIEVE/
|
||||
> ASSESS/REFINE/VERIFY loop or the `Focus`/`ConversationState`/TTL state
|
||||
> design below. This was a real, deliberate pivot mid-implementation, not an
|
||||
> abandoned-but-still-intended plan: `rag/agent.py`'s own module docstring
|
||||
> says outright that `ConversationalLoopService` + `conversation.py` were
|
||||
> replaced because "the LLM reads a plain turn history and resolves
|
||||
> ['thuốc đó' / 'còn liều thì sao'] itself" — simpler than maintaining
|
||||
> `Focus`/TTL/turn-budget state by hand, and proven live across many
|
||||
> multi-turn conversations since. Section 6 below ("Refused: an LLM
|
||||
> confidence score as the loop's uncertainty signal") is the clearest
|
||||
> evidence this is a genuine architecture change, not a gap: the live system
|
||||
> now uses exactly that — an LLM sufficiency/clarify judgment — as its
|
||||
> ask-or-answer signal, the opposite of what this ADR chose.
|
||||
>
|
||||
> The three modules this ADR specified (1,314 lines) and their five dedicated
|
||||
> test files (42 tests) were deleted on 2026-08-07 rather than left as dead
|
||||
> code, once confirmed to have zero live importers anywhere
|
||||
> (`bootstrap.py`/`main.py`/`agent.py`/`answer.py`/`routers/rag.py`). This
|
||||
> document is kept, unedited below this notice, as the historical record of
|
||||
> why that design was chosen and what it traded off — see ADR 0008 for what
|
||||
> actually runs today, including what this ADR got right that ADR 0008
|
||||
> still owes (a real request-scoped time/call budget — F-08, still open; a
|
||||
> durable, cross-worker conversation store — currently an in-process dict).
|
||||
|
||||
## Context
|
||||
|
||||
The service answers one question at a time. `POST /v1/rag/query` carries no
|
||||
conversation id, `apps/chat-service` holds zero source files, and every request
|
||||
re-resolves the drug from scratch. Three consequences, all observed in the UI on
|
||||
2026-08-05:
|
||||
|
||||
- `paracetamol` alone is refused rather than asked about.
|
||||
- `liều dùng paracetamol cho người lớn` returns the identical answer to
|
||||
`liều dùng paracetamol` — the qualifier is not used at any stage.
|
||||
- A follow-up such as *"còn trẻ em thì sao?"* cannot work at all, because
|
||||
nothing carries the drug forward.
|
||||
|
||||
The owner's requirement is a **conversational reasoning RAG**: history, an
|
||||
internal reasoning stage, and a bounded self-improvement loop.
|
||||
|
||||
The binding constraint is that this is a drug formulary for clinicians. Every
|
||||
capability below is designed so that adding it cannot widen what the system is
|
||||
allowed to assert.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. Conversation state
|
||||
|
||||
Two stores with different jobs, deliberately not merged.
|
||||
|
||||
**`Focus` — structured, drives routing.** This is what makes *"còn trẻ em thì
|
||||
sao?"* resolvable without an LLM.
|
||||
|
||||
| Field | Purpose |
|
||||
|---|---|
|
||||
| `drug_id`, `drug_name` | The drug under discussion |
|
||||
| `section_key` | The attribute last answered |
|
||||
| `population` | `nguoi_lon` / `tre_em` / `phu_nu_co_thai` / … |
|
||||
| `verbosity` | `concise` \| `detailed`, set when the user asks |
|
||||
| `set_at_turn` | Turn index each field was last set |
|
||||
|
||||
**`ConversationState` — the whole record.**
|
||||
|
||||
```
|
||||
conversation_id
|
||||
recent: tuple[Turn, ...] # last K turns, verbatim
|
||||
summary: str # rolling prose summary of everything older
|
||||
focus: Focus
|
||||
turn_count: int
|
||||
```
|
||||
|
||||
A `Turn` carries `role`, `text`, `at`, and — for assistant turns — the
|
||||
`drug_id`, `section_key` and `evidence_ids` that produced it. Storing the
|
||||
evidence ids is what lets the planner answer a follow-up **from evidence
|
||||
already retrieved** instead of retrieving again.
|
||||
|
||||
**Carry-over is never silent.** An inherited `drug_id` that is wrong is a
|
||||
wrong-drug answer, so any answer built on inherited focus must name what it
|
||||
inherited: *"Về Metformin, ở trẻ em: …"*. This is a hard rule, not a
|
||||
presentation preference.
|
||||
|
||||
**Focus expires.** A field older than `FOCUS_TTL_TURNS` (6) is dropped rather
|
||||
than inherited. Conversations drift, and a drug from ten turns ago is not
|
||||
context, it is a hazard.
|
||||
|
||||
### 2. Recent history and summary
|
||||
|
||||
- `recent` holds the last **K = 6** turns verbatim (three exchanges).
|
||||
- When a turn falls out of `recent`, it is folded into `summary`.
|
||||
- `summary` is regenerated at most every **S = 4** turns, capped at **400
|
||||
tokens**; `recent` is capped at **2000 tokens**, oldest dropped first.
|
||||
- **The summary records what was discussed, never clinical content.** It may
|
||||
say *"đã hỏi liều dùng của Metformin cho người lớn"*; it may not carry a dose.
|
||||
A dose restated from a summary would have no citation and could not be
|
||||
grounding-verified — the check compares against retrieved evidence, and a
|
||||
summary is not evidence.
|
||||
|
||||
### 3. Reasoning loop
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[User turn] --> B[UNDERSTAND<br/>resolve against Focus]
|
||||
B --> C{Clarify signal?}
|
||||
C -->|ambiguous drug / no attribute /<br/>multi-attribute| Z[ASK — 1 turn, no loop]
|
||||
C -->|no| D{Simple?}
|
||||
D -->|drug + section resolved,<br/>no follow-up ambiguity| E[RETRIEVE]
|
||||
D -->|complex / decomposable| P[PLAN<br/>sub-questions + retrieval set]
|
||||
P --> E
|
||||
E --> F[ASSESS sufficiency]
|
||||
F -->|insufficient AND rounds left| R[REFINE query] --> E
|
||||
F -->|sufficient OR rounds exhausted| G[GENERATE]
|
||||
G --> H[VERIFY<br/>grounding + coverage]
|
||||
H -->|ungrounded / off-target,<br/>repairs left| G
|
||||
H -->|grounded| Y[RESPOND]
|
||||
H -->|repairs exhausted| X[FALL BACK<br/>verbatim source]
|
||||
F -->|exhausted AND still thin| Z
|
||||
```
|
||||
|
||||
**Continue conditions** — a round is spent only when all hold:
|
||||
1. `retrieval_rounds < MAX_RETRIEVAL_ROUNDS` (2)
|
||||
2. the assessor named a *specific* missing thing (a section, a population, a
|
||||
second drug) — "feels incomplete" is not a reason to spend a round
|
||||
3. the refined query differs from every query already tried this turn
|
||||
|
||||
**Stop conditions** — any one ends the loop:
|
||||
- sufficiency satisfied
|
||||
- budget exhausted (rounds, LLM calls, wall-clock, tokens)
|
||||
- a clarify signal fires (these bypass the loop entirely — asking beats guessing)
|
||||
- grounding verification fails after `MAX_REPAIRS` (1) → extractive fallback
|
||||
|
||||
**Fast path.** When the drug resolves and `SectionResolver` returns a section
|
||||
and no clarify signal fires, the loop is skipped: retrieve → generate → verify.
|
||||
This is the majority path and it costs one LLM call.
|
||||
|
||||
### 4. Budgets
|
||||
|
||||
| Limit | Value | Enforced at |
|
||||
|---|---|---|
|
||||
| `MAX_RETRIEVAL_ROUNDS` | 2 | loop guard |
|
||||
| `MAX_REPAIRS` | 1 | loop guard |
|
||||
| `MAX_LLM_CALLS` per turn | 4 | budget object, checked before each call |
|
||||
| `MAX_WALL_CLOCK_MS` | 20000 | checked between stages |
|
||||
| `MAX_EVIDENCE_TOKENS` | 12000 | evidence assembly, oldest-dropped |
|
||||
| `FOCUS_TTL_TURNS` | 6 | state update |
|
||||
|
||||
The budget is a single object threaded through the loop and **decremented
|
||||
before** each call, so exhaustion degrades to the best answer so far rather
|
||||
than to an error.
|
||||
|
||||
### 5. Integration
|
||||
|
||||
New domain modules, no SDK imports:
|
||||
|
||||
- `rag/conversation.py` — `Focus`, `Turn`, `ConversationState`, window and
|
||||
focus-update rules. Pure; the follow-up resolution in it needs no LLM.
|
||||
- `rag/reasoning.py` — the loop, its budget, and its stage protocols.
|
||||
- `rag/ports.py` — `ConversationStore` (load/save), `Summariser`, `Planner`,
|
||||
`SufficiencyAssessor`. Each has a deterministic no-LLM default so the whole
|
||||
loop runs offline.
|
||||
|
||||
New adapter: `adapters/postgres.py` gains `PostgresConversationStore`.
|
||||
|
||||
Unchanged and still binding: `GroundedAnswerService` remains the single-turn
|
||||
engine; `grounding.verify` gates every generated answer; `VERIFY_PDF` evidence
|
||||
is never generated over.
|
||||
|
||||
### 6. Measurement
|
||||
|
||||
A capability that cannot be shown to help does not ship. Three modes are run
|
||||
over the same cases — `single-shot`, `+history`, `+reasoning-loop`:
|
||||
|
||||
| Metric | Answers |
|
||||
|---|---|
|
||||
| follow-up resolution accuracy | does *"còn trẻ em thì sao?"* reach the right drug+section+population |
|
||||
| on-target rate | does the answer contain the population/attribute actually asked for |
|
||||
| grounding rejection rate | does reasoning make fabrication more or less likely |
|
||||
| clarify rate / clarify precision | does it ask when it should, and only then |
|
||||
| median + p95 latency, LLM calls, tokens per answered turn | what the capability costs |
|
||||
|
||||
The evaluation set is a **new multi-turn golden file** — the existing
|
||||
`golden_e2e_v1.csv` is single-turn by construction and cannot measure any of
|
||||
this. Counters land in `rag/metrics.py` and on the existing Grafana dashboard.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Accepted.** More moving parts and more tokens per turn; a stateful service
|
||||
where there was a stateless one; a summary that must be kept free of clinical
|
||||
content by rule rather than by mechanism.
|
||||
|
||||
**Refused.** An LLM confidence score as the loop's uncertainty signal. The
|
||||
signals used are the resolver states that already exist — ambiguous drug,
|
||||
unresolved section, multi-attribute question — because they are deterministic,
|
||||
testable, and explainable to a reviewer. "The model felt 0.73 sure" is not a
|
||||
defensible basis for asking or not asking a clinician a question.
|
||||
|
||||
**Unchanged.** Nothing here lets the system assert a figure absent from the
|
||||
retrieved source. Reasoning chooses *what to look up and how to say it*; it is
|
||||
not a source of facts.
|
||||
Reference in New Issue
Block a user