03 — Data flow

Two flows exist. They meet only at the Qdrant collection.

Flow A — document ingestion (offline)

flowchart TD
    PDF[/"data/raw/duoc-thu-quoc-gia-viet-nam-2018.pdf<br/>1,668 pages"/]
    SPANS["extract_spans (PyMuPDF)<br/>+ merge_outlined_runs"]
    GLYPH["scan_glyph_order / scan_reading_order<br/>sanity gate, reports only"]
    REG["_region_index:<br/>table_regions.json + formula_regions_2d.json"]
    ASM["segment.assemble<br/>monograph + section detection,<br/>table lift-out, quarantine"]
    MONO[/"data/processed/monographs.jsonl<br/>684 monographs"/]
    PMAP["build_page_map<br/>physical → printed folio"]
    CHUNK["chunk_all<br/>section → chunk, 800-token ceiling"]
    CHUNKS[/"data/processed/chunks.jsonl<br/>15,100 chunks, schema v4"/]
    GATES["cli chunk-ready<br/>named gates, all must be 0"]
    EMBED["load.run: CachingEmbeddingProvider<br/>cohere.embed-v4:0, input_type=search_document"]
    CACHE[/"data/processed/embeddings/*.jsonl<br/>keyed by (model, kind, sha256(text))"/]
    LOADER["ChunkLoader<br/>uuid5 point ids, batch 256"]
    QD[("Qdrant duocthu_v1")]
    MAN[("Qdrant duocthu_v1__manifest<br/>corpus sha · model · dims")]

    PDF --> SPANS --> ASM
    PDF --> GLYPH
    REG --> ASM
    ASM --> MONO --> CHUNK --> CHUNKS
    PDF --> PMAP --> CHUNK
    MONO --> GATES
    CHUNKS --> GATES
    CHUNKS --> EMBED --> CACHE --> LOADER --> QD
    LOADER --> MAN

Intermediate artifacts are real files that exist on disk today (04-ingestion-pipeline.md lists their sizes). The embed step is separable (--embed-only) and cached, so an interrupted run resumes without re-paying Bedrock.

Flow B — a user question (live)

sequenceDiagram
    autonumber
    actor U as Clinician
    participant W as web (Next.js)
    participant MW as middleware.ts
    participant API as ai-service /v1/rag/query
    participant AG as RagAgent
    participant LLM as Bedrock Converse
    participant RS as RetrievalService
    participant QD as Qdrant
    participant GA as GroundedAnswerService
    participant PG as PostgreSQL

    U->>W: POST /api/chat {content, conversationId}
    W->>MW: rate-limit by client IP
    MW-->>W: allow (or 429)
    W->>API: POST /v1/rag/query<br/>{query, subject_scope:"human", intent:"fact_lookup", conversation_id}
    Note over API: resolve_subject_scope() re-derives scope<br/>from the text — the caller's claim cannot widen it
    API->>AG: handle(turn, conversation_id)
    AG->>PG: recent(conversation_id, 12) — fail-open
    AG->>AG: CatalogDrugResolver bounds candidate drug_ids
    AG->>LLM: [1] understanding → QueryFrame (JSON)
    AG->>AG: _route(): turn_type + deterministic guards
    alt clarify / abstain / smalltalk
        AG-->>API: AgentReply (no retrieval)
    else answerable
        AG->>RS: retrieve_framed(drug_id, section_key, query)
        RS->>QD: scroll by payload filter (whole section)
        QD-->>RS: chunks, re-sorted by part_index
        RS->>RS: _decide(): provenance + quarantine gate
        AG->>GA: answer_from_result(...)
        GA->>LLM: [2] generation → {claims[], evidence_sufficient, ...}
        GA->>GA: grounding.verify() — numbers/citations, deterministic
        GA->>LLM: [3] entailment → {entailed, unsupported, complete, missing_evidence}
        GA-->>AG: GroundedAnswer + citations
    end
    AG->>PG: append(conversation_id, lines) — fail-open
    API->>PG: save(trace) — fail-open
    API-->>W: RagQueryResponse (decision, answer, blocks, citations, disclaimer)
    W->>W: map reason → Vietnamese; group citations by chunk_id
    W-->>U: SendMessageResponse

What is carried at each hop

Hop	Payload
Browser → web	`{content, conversationId}`
web → ai-service	`{query, subject_scope, intent, conversation_id}` + `X-Correlation-ID`, optional `traceparent`/`tracestate`
understanding LLM	Candidate drug shortlist (drug_id + name), 19 section keys with glosses, prior known-facts block, history, current turn
Qdrant	Payload filter only for the section route (`drug_id` + `section_key`); a 1024-d vector for the dense fallback
generation LLM	Numbered evidence blocks, each prefixed `(drug_id=…; thuốc=…; mục=…)`, plus a presentation plan and the fenced user question
entailment LLM	Each claim paired with only the evidence block(s) it cited, plus the whole selected evidence set
ai-service → web	`decision`, `reason`, `answer`, `blocks[]`, `citations[]`, `quick_replies[]`, `answer_plan`, `candidate_assessments[]`, `disclaimer`, `trace_id`, `correlation_id`, `otel_trace_id`

Identifier flow

One identifier threads the whole system:

chunk_id = "{drug_id}__{section_key}__{part_index}"

Written by ingestion/chunk/chunker.py
Point id = uuid5(POINT_NAMESPACE, chunk_id) — derived, so a re-load overwrites rather than duplicates (ingestion/load/models.py)
Filtered on in Qdrant (chunk_id has a keyword index)
Returned as Citation.chunk_id and as AnswerClaim.source_ids
Split by answer.py::_section_key to pick a block title, and by web/app/api/chat/route.ts to recover the drug slug per citation
Persisted in rag_retrieval_trace.citations (jsonb)

The block-descriptor variant is {drug_id}__{section_key}__block__{table_id}.

Correlation identifiers: X-Correlation-ID (validated against ^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$, regenerated if malformed) and the OpenTelemetry trace id are both echoed in response headers and stored on the trace row (migrations/003).

Error / fallback flow

flowchart TD
    Q[Turn received] --> SCOPE{looks_non_human?}
    SCOPE -->|yes| AB1["abstain: out_of_scope"]
    SCOPE -->|no| UND[understanding LLM]
    UND -->|provider error| AB2["abstain: understanding_provider_unavailable"]
    UND -->|unparseable JSON| AB3["abstain: understanding_malformed_output"]
    UND --> ROUTE{route}
    ROUTE -->|missing required field| CLR[clarify]
    CLR --> BRK{4th consecutive clarify?}
    BRK -->|yes| AB4["abstain: clarify_loop_exhausted"]
    BRK -->|no| OUT1[return question]
    ROUTE --> RET[retrieval]
    RET -->|no evidence| AB5["abstain: parent_hydration_failed"]
    RET -->|missing source_refs| AB6["abstain: missing_provenance"]
    RET -->|quarantined content| VP["verify_pdf: notice + source page"]
    RET -->|ok| GEN[generation LLM]
    GEN -->|budget out| AB7["abstain: request_budget_exhausted"]
    GEN -->|provider error| AB8["abstain: provider_unavailable"]
    GEN -->|bad JSON| AB9["abstain: malformed_output"]
    GEN -->|insufficient ×2| AB10["abstain: evidence_insufficient"]
    GEN --> GR[grounding.verify]
    GR -->|number not in cited block| AB11["abstain: ungrounded_number"]
    GR -->|marker out of range| AB12["abstain: invalid_citation"]
    GR -->|claim with no citation| AB13["abstain: uncited_claim"]
    GR --> ENT[entailment LLM]
    ENT -->|not entailed| AB14["abstain: unsupported_claim"]
    ENT -->|incomplete| REP[repair regeneration]
    REP -->|still incomplete| AB15["abstain: incomplete_answer"]
    ENT -->|ok| OK[answerable + citations]

Every terminal box above is a distinct reason string, and every one of them has an explicit Vietnamese message in apps/web/app/api/chat/route.ts::REFUSALS. That mapping is load-bearing: an unmapped reason falls through to GENERIC_REFUSAL, which reads as "no data in the formulary" and would misdescribe an outage.

7.4 KiB Raw Blame History Unescape Escape