VuQuangBao/duocthu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Wire the guarded conversational RAG answer layer end-to-end

Browse Source
This commit is contained in:
BaoVu2k4
2026-08-05 14:33:13 +07:00
parent 834d9e51b0
commit ef08b4929e
127 changed files with 37921 additions and 169 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/0004-chunking-strategy.md
+16 -15
View File
@@ -93,9 +93,15 @@ in a chunk shown to a doctor or pharmacist.
— per CLAUDE.md's provenance rule): `chunk_id`
(`{drug_id}__{section_key}__{part_index}`), `drug_id`, `drug_name`,
`section_key`, `section_display_name`, `atc_codes` (inherited from the
monograph — enables ATC-class-filtered retrieval), `source_page_range`
(monograph-level, see Consequences), `part_index`/`part_count` (`0`/`1`
for un-split sections, keeps the schema uniform across all chunks).
monograph — enables ATC-class-filtered retrieval), exact per-chunk
`source_page_range` and `printed_page_range`, `part_index`/`part_count`
(`0`/`1` for un-split sections, keeps the schema uniform across all chunks).
6. **Schema v4 separates source from retrieval context.** `source_text` is the
exact contiguous source span and is the basis for lossless reassembly and
page provenance. `text` may prefix repeated route/population labels so a
continuation chunk is independently safe to retrieve. Those retrieval-only
prefixes are recorded in `context_labels` and may not alter `source_text`.
Token counts use `cl100k_base`, not the earlier chars/4 estimate.
## Consequences
@@ -116,15 +122,10 @@ in a chunk shown to a doctor or pharmacist.
by sub-compound — a chunk from this section is tagged with the class
name only, not the specific analogue a query might target. Deferred to
golden-dataset-driven eval rather than guessed at now.
- **Known gap — sub-chunk page precision**: `source_page_range` is
monograph-level, not sub-chunk-exact. A sub-chunk from late in a
multi-page section inherits the whole monograph's page range rather than
its own precise page, because per-line page tracking doesn't currently
exist in `SectionSpan`/`Heading`. The monograph + section-heading page is
still real, checkable provenance, but this is a known precision gap, not
full sub-chunk traceability. Flagged as a future improvement.
- **Not yet built**: the Vietnamese sentence-boundary splitter itself
(abbreviation handling, decimal-comma handling, ATC-code-period handling)
is specified here as a rule, not implemented or unit-tested. Building and
testing it is a separate, later task (`ingestion/ingestion/chunk/`, which
does not exist yet).
- **Resolved — sub-chunk page precision**: schema v4 derives exact physical
support from the contiguous `source_text` span and maps it to verified
printed folios. Missing or ambiguous support fails readiness rather than
falling back to monograph-level provenance.
- **Implemented**: the sentence/label-aware splitter is in
`ingestion/ingestion/chunk/` with regression tests for dose continuations,
compound label boundaries, parent route context and lossless reassembly.
docs/adr/0006-quarantined-block-references-in-chunks.md
+24 -28
View File
@@ -2,7 +2,7 @@
## Status
Proposed, with implementation to follow immediately. Resolves the item ADR
Accepted and implemented in schema v4. Resolves the item ADR
0005 explicitly deferred ("Table/formula content blocks … the precise
`ContentBlock`/table-row/formula-unit shape is deferred to a follow-up
revision of this ADR once the survey reports real numbers"). The survey has
@@ -22,12 +22,10 @@ the current whole-corpus output:
| quantity | value |
|---|---|
| lifted blocks | 167, all quarantined |
| monographs affected | 96 of 683 (**14.1%**) |
| sections affected | 108 |
| **blocks in `lieu_luong_va_cach_dung`** | **127 (76%)** |
| next largest section | `duoc_ly_va_co_che_tac_dung`, 16 |
| shapes | simple_table 136, multi_level_or_merged_header 16, formula_2d 14, cross_page_continuation 1 |
| lifted blocks represented by descriptor chunks | 151, all quarantined |
| sections affected | 103 |
| **blocks in `lieu_luong_va_cach_dung`** | **125** |
| unverified header rows admitted to embedding text | **0** |
So three quarters of everything removed from prose was removed from the
dosing section, in a drug formulary, for an audience of doctors and
@@ -57,9 +55,11 @@ class ChunkAttachment:
shape: str # simple_table | multi_level_or_merged_header |
# cross_page_continuation | formula_2d
physical_page: int
printed_page: int
bbox: List[float]
quarantined: bool
header_row: List[str] = () # simple_table only; see caveat below
header_row: List[str] = () # always empty until separately verified
source_crop: str | None = None
@dataclass(frozen=True)
class Chunk:
@@ -82,28 +82,20 @@ A block also gets its own chunk so it is retrievable at all:
chunk_id = "{drug_id}:{section_key}:block:{block_id}"
chunk_kind = "block_descriptor"
text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
trang in 204. Cột: Độ thanh thải creatinin | Nửa đời |
Liều ampicilin/sulbactam."
trang in 204."
```
The text is assembled from the drug name, the section display name, the kind,
the printed page and — for `simple_table` only — the header row. **No cell
value ever appears.** A header row is a row of labels; linearising it cannot
invent a numeric relationship, which is precisely what linearising a body row
does. For every other shape the header is omitted, because
`multi_level_or_merged_header` is the shape whose header extraction is least
trustworthy.
Caveat recorded in the schema itself: `header_row` comes from
`pdfplumber.find_tables()`'s first row and has **not** been verified by eye
(the 180 real tables' individual shapes are rule-derived; only the 20
"not a table" verdicts were visually confirmed). It is retrieval bait, never
an answer.
The text is assembled only from verified metadata: drug name, section display
name, block kind and printed page. **No cell value or inferred header appears.**
The earlier proposal to use `pdfplumber.find_tables()`'s first row was rejected
after corpus audit: a guessed first row can be a body row or can merge numeric
relationships. Until a separate human-verified header dataset exists,
`header_row` is embargoed for every shape and serialized as empty.
### 3. The answer layer's obligations (binding on `ai-service`)
These are stated here because they are the reason the schema exists; they are
not implemented by `ingestion/`.
These obligations are implemented across `ingestion/` and `ai-service` and are
enforced by tests/readiness gates.
1. A retrieved chunk with `has_quarantined_content: true` **must** cause the
answer to state that a table or formula exists at the cited page, and to
@@ -158,14 +150,18 @@ Added to `cli chunk-ready` and to the chunk stage's own tests:
contains a quarantined block's text
5. `descriptor_chunk_count == block_count`
6. `descriptor_chunk_without_attachment = 0`
7. `attachment_header_row_present = 0`
8. `descriptor_with_unverified_columns = 0`
9. `descriptor_range_not_attachment_page = 0`
10. `attachment_without_printed_page = 0`
## Consequences
- Prose chunks shrink slightly in trustworthiness terms but grow in honesty:
the ones missing a table now say so.
- The index gains 167 descriptor chunks (≈1.4% of the expected chunk count),
- The current candidate index gains 151 descriptor chunks,
each cheap and none carrying unsafe text.
- `ai-service` cannot be built to answer a dosing question from prose alone
for the 108 affected sections without violating a stated contract.
- `ai-service` cannot answer a dosing question from prose alone for the 103
affected sections without violating a stated contract.
- The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas
answerable as crops today, which they are not now.