# 09 — Retrieval pipeline

Implementation: `apps/ai-service/rag/service.py` (`RetrievalService`, 741 lines),
`apps/ai-service/adapters/qdrant.py` (501 lines), `rag/sections.py`,
`rag/context.py`.
Tests: `tests/test_retrieval_service.py`, `tests/test_section_routing.py`,
`tests/test_qdrant_adapter.py`, `tests/test_rerank_overview.py`,
`tests/test_section_order.py`.

## What retrieval is here

**Similarity is the fallback, not the default.** Measured 2026-08-04: letting
vector similarity choose the section gives hit@1 **0.544** overall and **0.05**
on `chong_chi_dinh`, because `duoc_ly_va_co_che_tac_dung` is the largest section
and sits close to almost any question about the drug. When the question names
the section it wants, a payload filter answers it exactly.

That single measurement is the reason the architecture looks the way it does.

## Retrieval routes

```mermaid
flowchart TD
    IN["retrieve_framed(drug_id, section_key, query, is_overview)"]
    S{section_key given?}
    SEC["find_by_section(drug_id, section_key)<br/>Qdrant scroll, payload filter, NO vector<br/>score = 1.0 by construction"]
    POOL["_pooled_neighbour_hits<br/>only when section == than_trong"]
    OV["find_by_drug(drug_id)<br/>every prose section, book order"]
    ISOV{is_overview?}
    INTRO["keep INTRO_SECTIONS only:<br/>ten_chung_quoc_te, loai_thuoc,<br/>chi_dinh, duoc_ly_va_co_che_tac_dung"]
    RR["_rerank(query, hits)<br/>Cohere rerank-v3.5, top_k=6, fail-open"]
    PACK["pack_evidence(max_tokens=6000)"]
    DEC["_decide(evidence)"]

    IN --> S
    S -->|yes| SEC --> POOL --> DEC
    S -->|no| OV
    OV -->|None| AB["ABSTAIN insufficient_retrieval_score"]
    OV --> ISOV
    ISOV -->|yes| INTRO --> DEC
    ISOV -->|no| RR --> PACK --> DEC
```

### 1. Section route (the primary path)

`find_by_section` is a **`scroll`, not a `search`** — it must not be a top-k.
Paging continues until the offset is exhausted, because Qdrant's default page is
256 and a long section silently truncated would read as a complete answer.
Results are re-sorted by `part_index` (see
[07-indexing-and-storage.md](07-indexing-and-storage.md) for why). No evidence
limit is applied — the whole section is the answer, and a truncated list of
contraindications reads as a complete one.

Score is `1.0` because the match is exact by construction. It is **not** a
similarity and must not be compared to one.

### 2. Bounded cross-section pooling

`_pooled_neighbour_hits` uses `search_lexical` to find another section of the
*same drug* whose text matches the query strongly. It exists for one measured
case: a `thận trọng` question about a specific condition (loét dạ dày) whose
real answer was filed only under `chống chỉ định`.

It is deliberately narrow:

```python
_LEXICAL_POOL_ENABLED_SECTIONS  = {"than_trong"}                     # only this route
_LEXICAL_POOL_EXCLUDED_SECTIONS = {"duoc_ly_va_co_che_tac_dung"}     # the known attractor
_LEXICAL_POOL_MIN_SCORE = 5.0
_MAX_LEXICAL_POOLED_SECTIONS = 2
```

The excluded section is excluded outright rather than by score margin: on the
exact query that motivated the mechanism, the true positive scored 7 matched
terms and that attractor scored 6 — too close for a threshold to separate.
Applying pooling to every section leaked a lexically-overlapping interaction
section into a dosage answer, so it stayed opt-in.

### 3. Drug overview (a bare drug name)

`find_by_drug` scrolls every **prose** chunk of the drug (block descriptors stay
out of a text answer), orders by `SECTION_ORDER` then `part_index`, and prefixes
each section's first chunk with `【display name】`.

For `turn_type == "drug_overview"` only the four `INTRO_SECTIONS` are kept.
Without that split a bare drug name sent the entire ~29-section monograph as
evidence for every generation call — wrong retrieval, and an answer long enough
to intermittently fail generation outright.

### 4. Free-form question about a resolved drug

The full monograph is reranked to `rerank_top_k=6`, then packed to a **token**
budget rather than a flat count:

```python
max_context_tokens = 6000   # pack_evidence, rag/context.py
```

`pack_evidence` packs whole blocks in retrieval order and never truncates
clinical text; anything that does not fit is recorded in
`omitted_evidence_ids`. The cap is applied even when rerank is disabled or
fails open — an ordering aid must not also remove the size bound.

### 5. Reverse lookup: condition/indication → drugs

`retrieve_by_indication` is a two-stage lookup, keyword first:

1. **`find_by_indication`** — scroll every `chi_dinh` prose chunk and require the
   normalized indication to appear as a **contiguous, word-boundary-anchored
   phrase**. Not a substring (false positives after diacritic stripping), and
   explicitly not a token-subset match: a nonsense phrase built from common
   filler words previously false-positived against real `chi_dinh` text and
   reached generation before being caught.
   Score rewards an early, concise mention:
   `1 + 1/(1+position) + 1/(1 + words/40)`.
2. **`search_indication`** — dense fallback, tried only when the keyword pass
   finds nothing, filtered to `section_key=chi_dinh` and `chunk_kind=prose`.
   **This is the only place in the live path where dense vector search is
   actually used** (ADR 0008). A weak top score (`< evidence_minimum_score`)
   discards the hits, because dense search always returns its nearest
   neighbours — a made-up phrase still got 8 unrelated "matches" live.

The adapter returns a ranked **chunk** pool; `_rank_indication_drugs` groups by
`drug_id`, takes the **max** score per drug (never a sum or count, so a drug with
more chunks does not win), optionally reranks the groups, and the service caps
at 8 drugs × 2 evidence chunks.

### 6. Patient-specific safety evidence (stage 2)

`assess_patient_candidates` / `retrieve_patient_drug_context` never create
candidates. For each already-indicated drug they run separate, relation-specific
lexical searches:

| Facet | Query source | Sections searched |
|---|---|---|
| interaction | `patient.interaction_query()` | `tuong_tac_thuoc` |
| warnings | `patient.warning_query()` | `chong_chi_dinh`, `than_trong` (requires a clinical-anchor match) |
| dosage context | `patient.dosage_context_query()` | `lieu_luong_va_cach_dung` (requires a clinical-anchor match) |
| pregnancy / breastfeeding | direct section route | `thoi_ky_mang_thai`, `thoi_ky_cho_con_bu` |

Keeping the queries separate is the point: a current medicine may select an
interaction chunk only when *that medicine* matches inside the interaction
section — CKD or age terms from another facet cannot make an unrelated
interaction look supported. `_patient_context_matches` requires a real clinical
anchor rather than overlap on generic words like `chức năng`.

Absence of a hit is recorded as `CandidateStatus.INSUFFICIENT_EVIDENCE` — never
as "safe".

## The evidence decision — `_decide`

```python
if not evidence:                                   ABSTAIN  "parent_hydration_failed"
if any(not item.source_refs for item in evidence): ABSTAIN  "missing_provenance"
if any(item.requires_visual_check ...):            VERIFY_PDF "visual_verification_required"
else:                                              ANSWERABLE "grounded_evidence_available"
```

`decide()` is exposed publicly so a caller assembling its own pool across several
retrieve calls — `RagAgent._interaction` — gets the same quarantine and
provenance policy. Bypassing it is precisely how the interaction path once
silently dropped a quarantined drug's evidence instead of surfacing `VERIFY_PDF`.

`requires_visual_check` is read from the payload as
`requires_visual_check OR has_quarantined_content`.

## Policy constants — `EvidencePolicy`

| Setting | Default | Applies to |
|---|---|---|
| `minimum_score` | 0.12 (`EVIDENCE_MINIMUM_SCORE`) | dense routes only |
| `candidate_limit` | 5 | `retrieve()`'s dense search |
| `evidence_limit` | 3 | `_hydrate` default; **not** used by the section route |
| `rerank_top_k` | 6 | overview/free-form rerank |
| `max_context_tokens` | 6000 | overview/free-form packing |
| `indication_candidate_limit` | 8 | drugs shown for a reverse lookup |
| `indication_retrieval_limit` | 40 | chunk pool before grouping |
| `indication_evidence_per_drug` | 2 | |
| `patient_candidate_limit` | 2 | stage-2 safety |
| `safety_hits_per_section` | 1 | |
| `safety_sections_per_candidate` | 4 | |

## What this pipeline is *not*

Stated plainly because the terms get reused loosely:

- **Not BM25.** `search_lexical` scores a hit as *the count of distinct matched
  query tokens* — no term frequency, no IDF, no length normalisation. The
  docstring calls it "a transparent stand-in for a real BM25 score".
- **Not hybrid search.** `rag/fusion.py` implements reciprocal-rank fusion and is
  tested, but **no runtime code calls it**. Dense and lexical results are never
  fused.
- **No multi-query / query expansion.** `rag/expansion.py` (sibling expansion)
  exists and is tested but has **no runtime caller**. No rewritten-query
  retrieval exists anywhere.
- **No parent-child hydration in practice.** The code path exists
  (`_hydrate` → `ParentStore.get`) but no chunk in the loaded corpus carries a
  `parent_id`.
- **No filters on `atc_codes`.** The field is indexed and stored; nothing
  queries it.

## Section keyword resolver — `rag/sections.py`

Used by the legacy `retrieve()` path (no generator configured). Two rules make
it safe:

- **Longest phrase wins.** All phrases across all sections are sorted by length,
  so `chống chỉ định` is tested before `chỉ định` — they differ by one prefix
  word and mean opposite things. The same rule keeps `quá liều` from being read
  as `liều`.
- **No match is not a guess.** An unrecognised question returns `None` and the
  caller falls back to similarity. This layer never picks a section it is unsure
  of.

Adding a phrasing means adding an entry to `SECTION_PHRASES`, never editing the
matching code.