# 03 — Data flow

Two flows exist. They meet only at the Qdrant collection.

## Flow A — document ingestion (offline)

```mermaid
flowchart TD
    PDF[/"data/raw/duoc-thu-quoc-gia-viet-nam-2018.pdf<br/>1,668 pages"/]
    SPANS["extract_spans (PyMuPDF)<br/>+ merge_outlined_runs"]
    GLYPH["scan_glyph_order / scan_reading_order<br/>sanity gate, reports only"]
    REG["_region_index:<br/>table_regions.json + formula_regions_2d.json"]
    ASM["segment.assemble<br/>monograph + section detection,<br/>table lift-out, quarantine"]
    MONO[/"data/processed/monographs.jsonl<br/>684 monographs"/]
    PMAP["build_page_map<br/>physical → printed folio"]
    CHUNK["chunk_all<br/>section → chunk, 800-token ceiling"]
    CHUNKS[/"data/processed/chunks.jsonl<br/>15,100 chunks, schema v4"/]
    GATES["cli chunk-ready<br/>named gates, all must be 0"]
    EMBED["load.run: CachingEmbeddingProvider<br/>cohere.embed-v4:0, input_type=search_document"]
    CACHE[/"data/processed/embeddings/*.jsonl<br/>keyed by (model, kind, sha256(text))"/]
    LOADER["ChunkLoader<br/>uuid5 point ids, batch 256"]
    QD[("Qdrant duocthu_v1")]
    MAN[("Qdrant duocthu_v1__manifest<br/>corpus sha · model · dims")]

    PDF --> SPANS --> ASM
    PDF --> GLYPH
    REG --> ASM
    ASM --> MONO --> CHUNK --> CHUNKS
    PDF --> PMAP --> CHUNK
    MONO --> GATES
    CHUNKS --> GATES
    CHUNKS --> EMBED --> CACHE --> LOADER --> QD
    LOADER --> MAN
```

Intermediate artifacts are real files that exist on disk today
([04-ingestion-pipeline.md](04-ingestion-pipeline.md) lists their sizes). The
embed step is separable (`--embed-only`) and cached, so an interrupted run
resumes without re-paying Bedrock.

## Flow B — a user question (live)

```mermaid
sequenceDiagram
    autonumber
    actor U as Clinician
    participant W as web (Next.js)
    participant MW as middleware.ts
    participant API as ai-service /v1/rag/query
    participant AG as RagAgent
    participant LLM as Bedrock Converse
    participant RS as RetrievalService
    participant QD as Qdrant
    participant GA as GroundedAnswerService
    participant PG as PostgreSQL

    U->>W: POST /api/chat {content, conversationId}
    W->>MW: rate-limit by client IP
    MW-->>W: allow (or 429)
    W->>API: POST /v1/rag/query<br/>{query, subject_scope:"human", intent:"fact_lookup", conversation_id}
    Note over API: resolve_subject_scope() re-derives scope<br/>from the text — the caller's claim cannot widen it
    API->>AG: handle(turn, conversation_id)
    AG->>PG: recent(conversation_id, 12) — fail-open
    AG->>AG: CatalogDrugResolver bounds candidate drug_ids
    AG->>LLM: [1] understanding → QueryFrame (JSON)
    AG->>AG: _route(): turn_type + deterministic guards
    alt clarify / abstain / smalltalk
        AG-->>API: AgentReply (no retrieval)
    else answerable
        AG->>RS: retrieve_framed(drug_id, section_key, query)
        RS->>QD: scroll by payload filter (whole section)
        QD-->>RS: chunks, re-sorted by part_index
        RS->>RS: _decide(): provenance + quarantine gate
        AG->>GA: answer_from_result(...)
        GA->>LLM: [2] generation → {claims[], evidence_sufficient, ...}
        GA->>GA: grounding.verify() — numbers/citations, deterministic
        GA->>LLM: [3] entailment → {entailed, unsupported, complete, missing_evidence}
        GA-->>AG: GroundedAnswer + citations
    end
    AG->>PG: append(conversation_id, lines) — fail-open
    API->>PG: save(trace) — fail-open
    API-->>W: RagQueryResponse (decision, answer, blocks, citations, disclaimer)
    W->>W: map reason → Vietnamese; group citations by chunk_id
    W-->>U: SendMessageResponse
```

## What is carried at each hop

| Hop | Payload |
|---|---|
| Browser → web | `{content, conversationId}` |
| web → ai-service | `{query, subject_scope, intent, conversation_id}` + `X-Correlation-ID`, optional `traceparent`/`tracestate` |
| understanding LLM | Candidate drug shortlist (drug_id + name), 19 section keys with glosses, prior known-facts block, history, current turn |
| Qdrant | Payload filter only for the section route (`drug_id` + `section_key`); a 1024-d vector for the dense fallback |
| generation LLM | Numbered evidence blocks, each prefixed `(drug_id=…; thuốc=…; mục=…)`, plus a presentation plan and the fenced user question |
| entailment LLM | Each claim paired with only the evidence block(s) it cited, plus the whole selected evidence set |
| ai-service → web | `decision`, `reason`, `answer`, `blocks[]`, `citations[]`, `quick_replies[]`, `answer_plan`, `candidate_assessments[]`, `disclaimer`, `trace_id`, `correlation_id`, `otel_trace_id` |

## Identifier flow

One identifier threads the whole system:

```
chunk_id = "{drug_id}__{section_key}__{part_index}"
```

- **Written** by `ingestion/chunk/chunker.py`
- **Point id** = `uuid5(POINT_NAMESPACE, chunk_id)` — derived, so a re-load
  overwrites rather than duplicates (`ingestion/load/models.py`)
- **Filtered on** in Qdrant (`chunk_id` has a keyword index)
- **Returned** as `Citation.chunk_id` and as `AnswerClaim.source_ids`
- **Split** by `answer.py::_section_key` to pick a block title, and by
  `web/app/api/chat/route.ts` to recover the drug slug per citation
- **Persisted** in `rag_retrieval_trace.citations` (jsonb)

The block-descriptor variant is
`{drug_id}__{section_key}__block__{table_id}`.

Correlation identifiers: `X-Correlation-ID` (validated against
`^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$`, regenerated if malformed) and the
OpenTelemetry trace id are both echoed in response headers and stored on the
trace row (`migrations/003`).

## Error / fallback flow

```mermaid
flowchart TD
    Q[Turn received] --> SCOPE{looks_non_human?}
    SCOPE -->|yes| AB1["abstain: out_of_scope"]
    SCOPE -->|no| UND[understanding LLM]
    UND -->|provider error| AB2["abstain: understanding_provider_unavailable"]
    UND -->|unparseable JSON| AB3["abstain: understanding_malformed_output"]
    UND --> ROUTE{route}
    ROUTE -->|missing required field| CLR[clarify]
    CLR --> BRK{4th consecutive clarify?}
    BRK -->|yes| AB4["abstain: clarify_loop_exhausted"]
    BRK -->|no| OUT1[return question]
    ROUTE --> RET[retrieval]
    RET -->|no evidence| AB5["abstain: parent_hydration_failed"]
    RET -->|missing source_refs| AB6["abstain: missing_provenance"]
    RET -->|quarantined content| VP["verify_pdf: notice + source page"]
    RET -->|ok| GEN[generation LLM]
    GEN -->|budget out| AB7["abstain: request_budget_exhausted"]
    GEN -->|provider error| AB8["abstain: provider_unavailable"]
    GEN -->|bad JSON| AB9["abstain: malformed_output"]
    GEN -->|insufficient ×2| AB10["abstain: evidence_insufficient"]
    GEN --> GR[grounding.verify]
    GR -->|number not in cited block| AB11["abstain: ungrounded_number"]
    GR -->|marker out of range| AB12["abstain: invalid_citation"]
    GR -->|claim with no citation| AB13["abstain: uncited_claim"]
    GR --> ENT[entailment LLM]
    ENT -->|not entailed| AB14["abstain: unsupported_claim"]
    ENT -->|incomplete| REP[repair regeneration]
    REP -->|still incomplete| AB15["abstain: incomplete_answer"]
    ENT -->|ok| OK[answerable + citations]
```

Every terminal box above is a distinct `reason` string, and every one of them
has an explicit Vietnamese message in
`apps/web/app/api/chat/route.ts::REFUSALS`. That mapping is load-bearing: an
unmapped reason falls through to `GENERIC_REFUSAL`, which reads as "no data in
the formulary" and would misdescribe an outage.