163 lines
7.4 KiB
Markdown
163 lines
7.4 KiB
Markdown
# 03 — Data flow
|
||
|
||
Two flows exist. They meet only at the Qdrant collection.
|
||
|
||
## Flow A — document ingestion (offline)
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
PDF[/"data/raw/duoc-thu-quoc-gia-viet-nam-2018.pdf<br/>1,668 pages"/]
|
||
SPANS["extract_spans (PyMuPDF)<br/>+ merge_outlined_runs"]
|
||
GLYPH["scan_glyph_order / scan_reading_order<br/>sanity gate, reports only"]
|
||
REG["_region_index:<br/>table_regions.json + formula_regions_2d.json"]
|
||
ASM["segment.assemble<br/>monograph + section detection,<br/>table lift-out, quarantine"]
|
||
MONO[/"data/processed/monographs.jsonl<br/>684 monographs"/]
|
||
PMAP["build_page_map<br/>physical → printed folio"]
|
||
CHUNK["chunk_all<br/>section → chunk, 800-token ceiling"]
|
||
CHUNKS[/"data/processed/chunks.jsonl<br/>15,100 chunks, schema v4"/]
|
||
GATES["cli chunk-ready<br/>named gates, all must be 0"]
|
||
EMBED["load.run: CachingEmbeddingProvider<br/>cohere.embed-v4:0, input_type=search_document"]
|
||
CACHE[/"data/processed/embeddings/*.jsonl<br/>keyed by (model, kind, sha256(text))"/]
|
||
LOADER["ChunkLoader<br/>uuid5 point ids, batch 256"]
|
||
QD[("Qdrant duocthu_v1")]
|
||
MAN[("Qdrant duocthu_v1__manifest<br/>corpus sha · model · dims")]
|
||
|
||
PDF --> SPANS --> ASM
|
||
PDF --> GLYPH
|
||
REG --> ASM
|
||
ASM --> MONO --> CHUNK --> CHUNKS
|
||
PDF --> PMAP --> CHUNK
|
||
MONO --> GATES
|
||
CHUNKS --> GATES
|
||
CHUNKS --> EMBED --> CACHE --> LOADER --> QD
|
||
LOADER --> MAN
|
||
```
|
||
|
||
Intermediate artifacts are real files that exist on disk today
|
||
([04-ingestion-pipeline.md](04-ingestion-pipeline.md) lists their sizes). The
|
||
embed step is separable (`--embed-only`) and cached, so an interrupted run
|
||
resumes without re-paying Bedrock.
|
||
|
||
## Flow B — a user question (live)
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
autonumber
|
||
actor U as Clinician
|
||
participant W as web (Next.js)
|
||
participant MW as middleware.ts
|
||
participant API as ai-service /v1/rag/query
|
||
participant AG as RagAgent
|
||
participant LLM as Bedrock Converse
|
||
participant RS as RetrievalService
|
||
participant QD as Qdrant
|
||
participant GA as GroundedAnswerService
|
||
participant PG as PostgreSQL
|
||
|
||
U->>W: POST /api/chat {content, conversationId}
|
||
W->>MW: rate-limit by client IP
|
||
MW-->>W: allow (or 429)
|
||
W->>API: POST /v1/rag/query<br/>{query, subject_scope:"human", intent:"fact_lookup", conversation_id}
|
||
Note over API: resolve_subject_scope() re-derives scope<br/>from the text — the caller's claim cannot widen it
|
||
API->>AG: handle(turn, conversation_id)
|
||
AG->>PG: recent(conversation_id, 12) — fail-open
|
||
AG->>AG: CatalogDrugResolver bounds candidate drug_ids
|
||
AG->>LLM: [1] understanding → QueryFrame (JSON)
|
||
AG->>AG: _route(): turn_type + deterministic guards
|
||
alt clarify / abstain / smalltalk
|
||
AG-->>API: AgentReply (no retrieval)
|
||
else answerable
|
||
AG->>RS: retrieve_framed(drug_id, section_key, query)
|
||
RS->>QD: scroll by payload filter (whole section)
|
||
QD-->>RS: chunks, re-sorted by part_index
|
||
RS->>RS: _decide(): provenance + quarantine gate
|
||
AG->>GA: answer_from_result(...)
|
||
GA->>LLM: [2] generation → {claims[], evidence_sufficient, ...}
|
||
GA->>GA: grounding.verify() — numbers/citations, deterministic
|
||
GA->>LLM: [3] entailment → {entailed, unsupported, complete, missing_evidence}
|
||
GA-->>AG: GroundedAnswer + citations
|
||
end
|
||
AG->>PG: append(conversation_id, lines) — fail-open
|
||
API->>PG: save(trace) — fail-open
|
||
API-->>W: RagQueryResponse (decision, answer, blocks, citations, disclaimer)
|
||
W->>W: map reason → Vietnamese; group citations by chunk_id
|
||
W-->>U: SendMessageResponse
|
||
```
|
||
|
||
## What is carried at each hop
|
||
|
||
| Hop | Payload |
|
||
|---|---|
|
||
| Browser → web | `{content, conversationId}` |
|
||
| web → ai-service | `{query, subject_scope, intent, conversation_id}` + `X-Correlation-ID`, optional `traceparent`/`tracestate` |
|
||
| understanding LLM | Candidate drug shortlist (drug_id + name), 19 section keys with glosses, prior known-facts block, history, current turn |
|
||
| Qdrant | Payload filter only for the section route (`drug_id` + `section_key`); a 1024-d vector for the dense fallback |
|
||
| generation LLM | Numbered evidence blocks, each prefixed `(drug_id=…; thuốc=…; mục=…)`, plus a presentation plan and the fenced user question |
|
||
| entailment LLM | Each claim paired with only the evidence block(s) it cited, plus the whole selected evidence set |
|
||
| ai-service → web | `decision`, `reason`, `answer`, `blocks[]`, `citations[]`, `quick_replies[]`, `answer_plan`, `candidate_assessments[]`, `disclaimer`, `trace_id`, `correlation_id`, `otel_trace_id` |
|
||
|
||
## Identifier flow
|
||
|
||
One identifier threads the whole system:
|
||
|
||
```
|
||
chunk_id = "{drug_id}__{section_key}__{part_index}"
|
||
```
|
||
|
||
- **Written** by `ingestion/chunk/chunker.py`
|
||
- **Point id** = `uuid5(POINT_NAMESPACE, chunk_id)` — derived, so a re-load
|
||
overwrites rather than duplicates (`ingestion/load/models.py`)
|
||
- **Filtered on** in Qdrant (`chunk_id` has a keyword index)
|
||
- **Returned** as `Citation.chunk_id` and as `AnswerClaim.source_ids`
|
||
- **Split** by `answer.py::_section_key` to pick a block title, and by
|
||
`web/app/api/chat/route.ts` to recover the drug slug per citation
|
||
- **Persisted** in `rag_retrieval_trace.citations` (jsonb)
|
||
|
||
The block-descriptor variant is
|
||
`{drug_id}__{section_key}__block__{table_id}`.
|
||
|
||
Correlation identifiers: `X-Correlation-ID` (validated against
|
||
`^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$`, regenerated if malformed) and the
|
||
OpenTelemetry trace id are both echoed in response headers and stored on the
|
||
trace row (`migrations/003`).
|
||
|
||
## Error / fallback flow
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
Q[Turn received] --> SCOPE{looks_non_human?}
|
||
SCOPE -->|yes| AB1["abstain: out_of_scope"]
|
||
SCOPE -->|no| UND[understanding LLM]
|
||
UND -->|provider error| AB2["abstain: understanding_provider_unavailable"]
|
||
UND -->|unparseable JSON| AB3["abstain: understanding_malformed_output"]
|
||
UND --> ROUTE{route}
|
||
ROUTE -->|missing required field| CLR[clarify]
|
||
CLR --> BRK{4th consecutive clarify?}
|
||
BRK -->|yes| AB4["abstain: clarify_loop_exhausted"]
|
||
BRK -->|no| OUT1[return question]
|
||
ROUTE --> RET[retrieval]
|
||
RET -->|no evidence| AB5["abstain: parent_hydration_failed"]
|
||
RET -->|missing source_refs| AB6["abstain: missing_provenance"]
|
||
RET -->|quarantined content| VP["verify_pdf: notice + source page"]
|
||
RET -->|ok| GEN[generation LLM]
|
||
GEN -->|budget out| AB7["abstain: request_budget_exhausted"]
|
||
GEN -->|provider error| AB8["abstain: provider_unavailable"]
|
||
GEN -->|bad JSON| AB9["abstain: malformed_output"]
|
||
GEN -->|insufficient ×2| AB10["abstain: evidence_insufficient"]
|
||
GEN --> GR[grounding.verify]
|
||
GR -->|number not in cited block| AB11["abstain: ungrounded_number"]
|
||
GR -->|marker out of range| AB12["abstain: invalid_citation"]
|
||
GR -->|claim with no citation| AB13["abstain: uncited_claim"]
|
||
GR --> ENT[entailment LLM]
|
||
ENT -->|not entailed| AB14["abstain: unsupported_claim"]
|
||
ENT -->|incomplete| REP[repair regeneration]
|
||
REP -->|still incomplete| AB15["abstain: incomplete_answer"]
|
||
ENT -->|ok| OK[answerable + citations]
|
||
```
|
||
|
||
Every terminal box above is a distinct `reason` string, and every one of them
|
||
has an explicit Vietnamese message in
|
||
`apps/web/app/api/chat/route.ts::REFUSALS`. That mapping is load-bearing: an
|
||
unmapped reason falls through to `GENERIC_REFUSAL`, which reads as "no data in
|
||
the formulary" and would misdescribe an outage.
|