Wire the guarded conversational RAG answer layer end-to-end

This commit is contained in:
2026-08-05 14:33:13 +07:00
parent 834d9e51b0
commit ef08b4929e
127 changed files with 37921 additions and 169 deletions
+16 -15
View File
@@ -93,9 +93,15 @@ in a chunk shown to a doctor or pharmacist.
— per CLAUDE.md's provenance rule): `chunk_id`
(`{drug_id}__{section_key}__{part_index}`), `drug_id`, `drug_name`,
`section_key`, `section_display_name`, `atc_codes` (inherited from the
monograph — enables ATC-class-filtered retrieval), `source_page_range`
(monograph-level, see Consequences), `part_index`/`part_count` (`0`/`1`
for un-split sections, keeps the schema uniform across all chunks).
monograph — enables ATC-class-filtered retrieval), exact per-chunk
`source_page_range` and `printed_page_range`, `part_index`/`part_count`
(`0`/`1` for un-split sections, keeps the schema uniform across all chunks).
6. **Schema v4 separates source from retrieval context.** `source_text` is the
exact contiguous source span and is the basis for lossless reassembly and
page provenance. `text` may prefix repeated route/population labels so a
continuation chunk is independently safe to retrieve. Those retrieval-only
prefixes are recorded in `context_labels` and may not alter `source_text`.
Token counts use `cl100k_base`, not the earlier chars/4 estimate.
## Consequences
@@ -116,15 +122,10 @@ in a chunk shown to a doctor or pharmacist.
by sub-compound — a chunk from this section is tagged with the class
name only, not the specific analogue a query might target. Deferred to
golden-dataset-driven eval rather than guessed at now.
- **Known gap — sub-chunk page precision**: `source_page_range` is
monograph-level, not sub-chunk-exact. A sub-chunk from late in a
multi-page section inherits the whole monograph's page range rather than
its own precise page, because per-line page tracking doesn't currently
exist in `SectionSpan`/`Heading`. The monograph + section-heading page is
still real, checkable provenance, but this is a known precision gap, not
full sub-chunk traceability. Flagged as a future improvement.
- **Not yet built**: the Vietnamese sentence-boundary splitter itself
(abbreviation handling, decimal-comma handling, ATC-code-period handling)
is specified here as a rule, not implemented or unit-tested. Building and
testing it is a separate, later task (`ingestion/ingestion/chunk/`, which
does not exist yet).
- **Resolved — sub-chunk page precision**: schema v4 derives exact physical
support from the contiguous `source_text` span and maps it to verified
printed folios. Missing or ambiguous support fails readiness rather than
falling back to monograph-level provenance.
- **Implemented**: the sentence/label-aware splitter is in
`ingestion/ingestion/chunk/` with regression tests for dose continuations,
compound label boundaries, parent route context and lossless reassembly.