168 lines
7.1 KiB
Markdown
168 lines
7.1 KiB
Markdown
# ADR 0006: chunks must carry references to lifted table/formula blocks
|
|
|
|
## Status
|
|
|
|
Accepted and implemented in schema v4. Resolves the item ADR
|
|
0005 explicitly deferred ("Table/formula content blocks … the precise
|
|
`ContentBlock`/table-row/formula-unit shape is deferred to a follow-up
|
|
revision of this ADR once the survey reports real numbers"). The survey has
|
|
reported.
|
|
|
|
## Context
|
|
|
|
`segment/` now lifts table and formula regions out of section prose and
|
|
quarantines them (ADR 0003 lineage, outlier-catalog items 7, 8, 24, 25).
|
|
That was the right move — linearised, AMPICILIN VÀ SULBACTAM's
|
|
Cockcroft-Gault fraction read as `Clcr (ml/phút) = 72 x creatinin huyết
|
|
thanh`, i.e. a division presented as a multiplication, in a renal-dosing
|
|
section.
|
|
|
|
But `chunk/models.py` has no field that refers to a lifted block. Measured on
|
|
the current whole-corpus output:
|
|
|
|
| quantity | value |
|
|
|---|---|
|
|
| lifted blocks represented by descriptor chunks | 151, all quarantined |
|
|
| sections affected | 103 |
|
|
| **blocks in `lieu_luong_va_cach_dung`** | **125** |
|
|
| unverified header rows admitted to embedding text | **0** |
|
|
|
|
So three quarters of everything removed from prose was removed from the
|
|
dosing section, in a drug formulary, for an audience of doctors and
|
|
pharmacists.
|
|
|
|
**The failure this creates is silent, not visible.** A chunk of AMPICILIN VÀ
|
|
SULBACTAM's `lieu_luong_va_cach_dung` is grammatical, complete-looking prose
|
|
with the renal-dosing table absent and nothing marking the absence. Retrieval
|
|
ranks it, the model answers from it, and neither has any way to know a table
|
|
was taken out. A visible error would be safer than this.
|
|
|
|
A second, quieter failure: a table is currently **unreachable**. Nothing in
|
|
the index represents it, so "bảng liều theo chức năng thận của ampicilin"
|
|
cannot retrieve it even in principle.
|
|
|
|
## Decision
|
|
|
|
Chunks reference blocks; blocks' content never becomes embedded text.
|
|
|
|
### 1. `Chunk` gains typed attachments
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class ChunkAttachment:
|
|
block_id: str
|
|
kind: str # "table" | "formula"
|
|
shape: str # simple_table | multi_level_or_merged_header |
|
|
# cross_page_continuation | formula_2d
|
|
physical_page: int
|
|
printed_page: int
|
|
bbox: List[float]
|
|
quarantined: bool
|
|
header_row: List[str] = () # always empty until separately verified
|
|
source_crop: str | None = None
|
|
|
|
@dataclass(frozen=True)
|
|
class Chunk:
|
|
...
|
|
chunk_kind: str = "prose" # "prose" | "block_descriptor"
|
|
attachments: List[ChunkAttachment] = ()
|
|
has_quarantined_content: bool = False
|
|
```
|
|
|
|
`has_quarantined_content` is derivable from `attachments`, and is serialized
|
|
anyway. A consumer that never looks at `attachments` must still be unable to
|
|
miss the fact — the whole defect being fixed here is a consumer not knowing
|
|
what it was not told.
|
|
|
|
### 2. One descriptor chunk per block, built from metadata only
|
|
|
|
A block also gets its own chunk so it is retrievable at all:
|
|
|
|
```
|
|
chunk_id = "{drug_id}:{section_key}:block:{block_id}"
|
|
chunk_kind = "block_descriptor"
|
|
text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
|
|
trang in 204."
|
|
```
|
|
|
|
The text is assembled only from verified metadata: drug name, section display
|
|
name, block kind and printed page. **No cell value or inferred header appears.**
|
|
The earlier proposal to use `pdfplumber.find_tables()`'s first row was rejected
|
|
after corpus audit: a guessed first row can be a body row or can merge numeric
|
|
relationships. Until a separate human-verified header dataset exists,
|
|
`header_row` is embargoed for every shape and serialized as empty.
|
|
|
|
### 3. The answer layer's obligations (binding on `ai-service`)
|
|
|
|
These obligations are implemented across `ingestion/` and `ai-service` and are
|
|
enforced by tests/readiness gates.
|
|
|
|
1. A retrieved chunk with `has_quarantined_content: true` **must** cause the
|
|
answer to state that a table or formula exists at the cited page, and to
|
|
surface its rendered crop. The answer may not present itself as complete.
|
|
2. A `block_descriptor` chunk may be answered **only** with the crop. It must
|
|
never be paraphrased, and its `header_row` must never be presented as the
|
|
table's content.
|
|
3. No chunk carrying a quarantined attachment may be used to state a numeric
|
|
dose. If the dose is in the table, the answer is the crop plus the page.
|
|
|
|
### 4. `schema_version`
|
|
|
|
`monographs.jsonl` and the chunk output both gain `schema_version`. ADR 0005
|
|
flagged its absence; a schema that now has two chunk kinds and an attachment
|
|
list cannot be safely consumed without one.
|
|
|
|
## Alternatives rejected
|
|
|
|
- **Flatten the block into the chunk text.** This is the defect, not the fix
|
|
— it reproduces `Clcr = 72 x creatinin` exactly.
|
|
- **Chunk the block's linearised text as an ordinary chunk.** Worse than
|
|
flattening: it makes unsafe text independently retrievable *as prose*, with
|
|
its quarantine flag one dereference away from being ignored.
|
|
- **Drop the blocks.** Silent loss, and contrary to the standing rule that
|
|
unreconstructable content is quarantined with full provenance, never
|
|
deleted.
|
|
- **Rely on the prose saying "xem bảng".** The prose often does not, and a
|
|
retrieval layer cannot act on an unstructured hint.
|
|
- **Wait for row/column reconstruction and do this once.** Reconstruction is
|
|
days of work and would leave the corpus unchunkable meanwhile; worse, it
|
|
would make the schema question look answered when the *silent-incompleteness*
|
|
problem is independent of whether the rows are recovered. Reconstruction
|
|
later populates `rows` on the same attachment without touching consumers.
|
|
|
|
## Why a crop is a legitimate answer, not a placeholder
|
|
|
|
For doctors and pharmacists a rendered crop of the source page is the
|
|
highest-fidelity response available: it *is* the book, and it is verifiable at
|
|
a glance. Reconstruction earns its keep for a different job — comparing or
|
|
combining values across drugs, which is the synthesis use case this product
|
|
exists for — not for single-table lookup.
|
|
|
|
## Invariants and gates
|
|
|
|
Added to `cli chunk-ready` and to the chunk stage's own tests:
|
|
|
|
1. `section_with_lifted_block_but_no_chunk_reference = 0`
|
|
2. `attachment_block_id_unknown = 0` — every referenced id exists on the
|
|
monograph
|
|
3. `attachment_without_page_or_bbox = 0`
|
|
4. `block_text_leaked_into_chunk_text = 0` — no chunk's embedded text
|
|
contains a quarantined block's text
|
|
5. `descriptor_chunk_count == block_count`
|
|
6. `descriptor_chunk_without_attachment = 0`
|
|
7. `attachment_header_row_present = 0`
|
|
8. `descriptor_with_unverified_columns = 0`
|
|
9. `descriptor_range_not_attachment_page = 0`
|
|
10. `attachment_without_printed_page = 0`
|
|
|
|
## Consequences
|
|
|
|
- Prose chunks shrink slightly in trustworthiness terms but grow in honesty:
|
|
the ones missing a table now say so.
|
|
- The current candidate index gains 151 descriptor chunks,
|
|
each cheap and none carrying unsafe text.
|
|
- `ai-service` cannot answer a dosing question from prose alone for the 103
|
|
affected sections without violating a stated contract.
|
|
- The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas
|
|
answerable as crops today, which they are not now.
|