Wire the guarded conversational RAG answer layer end-to-end
This commit is contained in:
@@ -93,9 +93,15 @@ in a chunk shown to a doctor or pharmacist.
|
||||
— per CLAUDE.md's provenance rule): `chunk_id`
|
||||
(`{drug_id}__{section_key}__{part_index}`), `drug_id`, `drug_name`,
|
||||
`section_key`, `section_display_name`, `atc_codes` (inherited from the
|
||||
monograph — enables ATC-class-filtered retrieval), `source_page_range`
|
||||
(monograph-level, see Consequences), `part_index`/`part_count` (`0`/`1`
|
||||
for un-split sections, keeps the schema uniform across all chunks).
|
||||
monograph — enables ATC-class-filtered retrieval), exact per-chunk
|
||||
`source_page_range` and `printed_page_range`, `part_index`/`part_count`
|
||||
(`0`/`1` for un-split sections, keeps the schema uniform across all chunks).
|
||||
6. **Schema v4 separates source from retrieval context.** `source_text` is the
|
||||
exact contiguous source span and is the basis for lossless reassembly and
|
||||
page provenance. `text` may prefix repeated route/population labels so a
|
||||
continuation chunk is independently safe to retrieve. Those retrieval-only
|
||||
prefixes are recorded in `context_labels` and may not alter `source_text`.
|
||||
Token counts use `cl100k_base`, not the earlier chars/4 estimate.
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -116,15 +122,10 @@ in a chunk shown to a doctor or pharmacist.
|
||||
by sub-compound — a chunk from this section is tagged with the class
|
||||
name only, not the specific analogue a query might target. Deferred to
|
||||
golden-dataset-driven eval rather than guessed at now.
|
||||
- **Known gap — sub-chunk page precision**: `source_page_range` is
|
||||
monograph-level, not sub-chunk-exact. A sub-chunk from late in a
|
||||
multi-page section inherits the whole monograph's page range rather than
|
||||
its own precise page, because per-line page tracking doesn't currently
|
||||
exist in `SectionSpan`/`Heading`. The monograph + section-heading page is
|
||||
still real, checkable provenance, but this is a known precision gap, not
|
||||
full sub-chunk traceability. Flagged as a future improvement.
|
||||
- **Not yet built**: the Vietnamese sentence-boundary splitter itself
|
||||
(abbreviation handling, decimal-comma handling, ATC-code-period handling)
|
||||
is specified here as a rule, not implemented or unit-tested. Building and
|
||||
testing it is a separate, later task (`ingestion/ingestion/chunk/`, which
|
||||
does not exist yet).
|
||||
- **Resolved — sub-chunk page precision**: schema v4 derives exact physical
|
||||
support from the contiguous `source_text` span and maps it to verified
|
||||
printed folios. Missing or ambiguous support fails readiness rather than
|
||||
falling back to monograph-level provenance.
|
||||
- **Implemented**: the sentence/label-aware splitter is in
|
||||
`ingestion/ingestion/chunk/` with regression tests for dose continuations,
|
||||
compound label boundaries, parent route context and lossless reassembly.
|
||||
|
||||
Reference in New Issue
Block a user