duocthu/docs/adr/0007-conversational-reasoning-rag.md

# ADR 0007: Conversational reasoning RAG — state, bounded loop, and how it is measured

**Status:** superseded by ADR 0008 (2026-08-07). See the note below before
reading this as a description of anything currently running.
**Supersedes:** nothing. Extends ADR 0005 (segment output contract) and ADR 0006
(quarantined block references) rather than replacing them.

> **2026-08-07 — why this was superseded, not deleted.** An independent
> 7-agent audit on 2026-08-06 found `bootstrap.py` never constructs any of
> `rag/conversation.py` / `rag/reasoning.py` / `rag/conversational.py` — the
> live agent (`rag/agent.py::RagAgent`, wired in since the F-03 rebuild on
> 2026-08-06) is a fixed one-shot pipeline (understand → route → retrieve
> once → generate → ≤2 same-claim entailment retries), not the PLAN/RETRIEVE/
> ASSESS/REFINE/VERIFY loop or the `Focus`/`ConversationState`/TTL state
> design below. This was a real, deliberate pivot mid-implementation, not an
> abandoned-but-still-intended plan: `rag/agent.py`'s own module docstring
> says outright that `ConversationalLoopService` + `conversation.py` were
> replaced because "the LLM reads a plain turn history and resolves
> ['thuốc đó' / 'còn liều thì sao'] itself" — simpler than maintaining
> `Focus`/TTL/turn-budget state by hand, and proven live across many
> multi-turn conversations since. Section 6 below ("Refused: an LLM
> confidence score as the loop's uncertainty signal") is the clearest
> evidence this is a genuine architecture change, not a gap: the live system
> now uses exactly that — an LLM sufficiency/clarify judgment — as its
> ask-or-answer signal, the opposite of what this ADR chose.
>
> The three modules this ADR specified (1,314 lines) and their five dedicated
> test files (42 tests) were deleted on 2026-08-07 rather than left as dead
> code, once confirmed to have zero live importers anywhere
> (`bootstrap.py`/`main.py`/`agent.py`/`answer.py`/`routers/rag.py`). This
> document is kept, unedited below this notice, as the historical record of
> why that design was chosen and what it traded off — see ADR 0008 for what
> actually runs today, including what this ADR got right that ADR 0008
> still owes (a real request-scoped time/call budget — F-08, still open; a
> durable, cross-worker conversation store — currently an in-process dict).

## Context

The service answers one question at a time. `POST /v1/rag/query` carries no
conversation id, `apps/chat-service` holds zero source files, and every request
re-resolves the drug from scratch. Three consequences, all observed in the UI on
2026-08-05:

- `paracetamol` alone is refused rather than asked about.
- `liều dùng paracetamol cho người lớn` returns the identical answer to
  `liều dùng paracetamol` — the qualifier is not used at any stage.
- A follow-up such as *"còn trẻ em thì sao?"* cannot work at all, because
  nothing carries the drug forward.

The owner's requirement is a **conversational reasoning RAG**: history, an
internal reasoning stage, and a bounded self-improvement loop.

The binding constraint is that this is a drug formulary for clinicians. Every
capability below is designed so that adding it cannot widen what the system is
allowed to assert.

## Decision

### 1. Conversation state

Two stores with different jobs, deliberately not merged.

**`Focus` — structured, drives routing.** This is what makes *"còn trẻ em thì
sao?"* resolvable without an LLM.

| Field | Purpose |
|---|---|
| `drug_id`, `drug_name` | The drug under discussion |
| `section_key` | The attribute last answered |
| `population` | `nguoi_lon` / `tre_em` / `phu_nu_co_thai` / … |
| `verbosity` | `concise` \| `detailed`, set when the user asks |
| `set_at_turn` | Turn index each field was last set |

**`ConversationState` — the whole record.**

```
conversation_id
recent: tuple[Turn, ...]     # last K turns, verbatim
summary: str                 # rolling prose summary of everything older
focus: Focus
turn_count: int
```

A `Turn` carries `role`, `text`, `at`, and — for assistant turns — the
`drug_id`, `section_key` and `evidence_ids` that produced it. Storing the
evidence ids is what lets the planner answer a follow-up **from evidence
already retrieved** instead of retrieving again.

**Carry-over is never silent.** An inherited `drug_id` that is wrong is a
wrong-drug answer, so any answer built on inherited focus must name what it
inherited: *"Về Metformin, ở trẻ em: …"*. This is a hard rule, not a
presentation preference.

**Focus expires.** A field older than `FOCUS_TTL_TURNS` (6) is dropped rather
than inherited. Conversations drift, and a drug from ten turns ago is not
context, it is a hazard.

### 2. Recent history and summary

- `recent` holds the last **K = 6** turns verbatim (three exchanges).
- When a turn falls out of `recent`, it is folded into `summary`.
- `summary` is regenerated at most every **S = 4** turns, capped at **400
  tokens**; `recent` is capped at **2000 tokens**, oldest dropped first.
- **The summary records what was discussed, never clinical content.** It may
  say *"đã hỏi liều dùng của Metformin cho người lớn"*; it may not carry a dose.
  A dose restated from a summary would have no citation and could not be
  grounding-verified — the check compares against retrieved evidence, and a
  summary is not evidence.

### 3. Reasoning loop

```mermaid
flowchart TD
    A[User turn] --> B[UNDERSTAND<br/>resolve against Focus]
    B --> C{Clarify signal?}
    C -->|ambiguous drug / no attribute /<br/>multi-attribute| Z[ASK — 1 turn, no loop]
    C -->|no| D{Simple?}
    D -->|drug + section resolved,<br/>no follow-up ambiguity| E[RETRIEVE]
    D -->|complex / decomposable| P[PLAN<br/>sub-questions + retrieval set]
    P --> E
    E --> F[ASSESS sufficiency]
    F -->|insufficient AND rounds left| R[REFINE query] --> E
    F -->|sufficient OR rounds exhausted| G[GENERATE]
    G --> H[VERIFY<br/>grounding + coverage]
    H -->|ungrounded / off-target,<br/>repairs left| G
    H -->|grounded| Y[RESPOND]
    H -->|repairs exhausted| X[FALL BACK<br/>verbatim source]
    F -->|exhausted AND still thin| Z
```

**Continue conditions** — a round is spent only when all hold:
1. `retrieval_rounds < MAX_RETRIEVAL_ROUNDS` (2)
2. the assessor named a *specific* missing thing (a section, a population, a
   second drug) — "feels incomplete" is not a reason to spend a round
3. the refined query differs from every query already tried this turn

**Stop conditions** — any one ends the loop:
- sufficiency satisfied
- budget exhausted (rounds, LLM calls, wall-clock, tokens)
- a clarify signal fires (these bypass the loop entirely — asking beats guessing)
- grounding verification fails after `MAX_REPAIRS` (1) → extractive fallback

**Fast path.** When the drug resolves and `SectionResolver` returns a section
and no clarify signal fires, the loop is skipped: retrieve → generate → verify.
This is the majority path and it costs one LLM call.

### 4. Budgets

| Limit | Value | Enforced at |
|---|---|---|
| `MAX_RETRIEVAL_ROUNDS` | 2 | loop guard |
| `MAX_REPAIRS` | 1 | loop guard |
| `MAX_LLM_CALLS` per turn | 4 | budget object, checked before each call |
| `MAX_WALL_CLOCK_MS` | 20000 | checked between stages |
| `MAX_EVIDENCE_TOKENS` | 12000 | evidence assembly, oldest-dropped |
| `FOCUS_TTL_TURNS` | 6 | state update |

The budget is a single object threaded through the loop and **decremented
before** each call, so exhaustion degrades to the best answer so far rather
than to an error.

### 5. Integration

New domain modules, no SDK imports:

- `rag/conversation.py` — `Focus`, `Turn`, `ConversationState`, window and
  focus-update rules. Pure; the follow-up resolution in it needs no LLM.
- `rag/reasoning.py` — the loop, its budget, and its stage protocols.
- `rag/ports.py` — `ConversationStore` (load/save), `Summariser`, `Planner`,
  `SufficiencyAssessor`. Each has a deterministic no-LLM default so the whole
  loop runs offline.

New adapter: `adapters/postgres.py` gains `PostgresConversationStore`.

Unchanged and still binding: `GroundedAnswerService` remains the single-turn
engine; `grounding.verify` gates every generated answer; `VERIFY_PDF` evidence
is never generated over.

### 6. Measurement

A capability that cannot be shown to help does not ship. Three modes are run
over the same cases — `single-shot`, `+history`, `+reasoning-loop`:

| Metric | Answers |
|---|---|
| follow-up resolution accuracy | does *"còn trẻ em thì sao?"* reach the right drug+section+population |
| on-target rate | does the answer contain the population/attribute actually asked for |
| grounding rejection rate | does reasoning make fabrication more or less likely |
| clarify rate / clarify precision | does it ask when it should, and only then |
| median + p95 latency, LLM calls, tokens per answered turn | what the capability costs |

The evaluation set is a **new multi-turn golden file** — the existing
`golden_e2e_v1.csv` is single-turn by construction and cannot measure any of
this. Counters land in `rag/metrics.py` and on the existing Grafana dashboard.

## Consequences

**Accepted.** More moving parts and more tokens per turn; a stateful service
where there was a stateless one; a summary that must be kept free of clinical
content by rule rather than by mechanism.

**Refused.** An LLM confidence score as the loop's uncertainty signal. The
signals used are the resolver states that already exist — ambiguous drug,
unresolved section, multi-attribute question — because they are deterministic,
testable, and explainable to a reviewer. "The model felt 0.73 sure" is not a
defensible basis for asking or not asking a clinician a question.

**Unchanged.** Nothing here lets the system assert a figure absent from the
retrieved source. Reasoning chooses *what to look up and how to say it*; it is
not a source of facts.