duocthu/docs/adr/0006-quarantined-block-references-in-chunks.md

ADR 0006: chunks must carry references to lifted table/formula blocks

Status

Accepted and implemented in schema v4. Resolves the item ADR 0005 explicitly deferred ("Table/formula content blocks … the precise ContentBlock/table-row/formula-unit shape is deferred to a follow-up revision of this ADR once the survey reports real numbers"). The survey has reported.

Context

segment/ now lifts table and formula regions out of section prose and quarantines them (ADR 0003 lineage, outlier-catalog items 7, 8, 24, 25). That was the right move — linearised, AMPICILIN VÀ SULBACTAM's Cockcroft-Gault fraction read as Clcr (ml/phút) = 72 x creatinin huyết thanh, i.e. a division presented as a multiplication, in a renal-dosing section.

But chunk/models.py has no field that refers to a lifted block. Measured on the current whole-corpus output:

quantity	value
lifted blocks represented by descriptor chunks	151, all quarantined
sections affected	103
blocks in lieu_luong_va_cach_dung	125
unverified header rows admitted to embedding text	0

So three quarters of everything removed from prose was removed from the dosing section, in a drug formulary, for an audience of doctors and pharmacists.

The failure this creates is silent, not visible. A chunk of AMPICILIN VÀ SULBACTAM's lieu_luong_va_cach_dung is grammatical, complete-looking prose with the renal-dosing table absent and nothing marking the absence. Retrieval ranks it, the model answers from it, and neither has any way to know a table was taken out. A visible error would be safer than this.

A second, quieter failure: a table is currently unreachable. Nothing in the index represents it, so "bảng liều theo chức năng thận của ampicilin" cannot retrieve it even in principle.

Decision

Chunks reference blocks; blocks' content never becomes embedded text.

1. Chunk gains typed attachments

@dataclass(frozen=True)
class ChunkAttachment:
    block_id: str
    kind: str            # "table" | "formula"
    shape: str           # simple_table | multi_level_or_merged_header |
                         # cross_page_continuation | formula_2d
    physical_page: int
    printed_page: int
    bbox: List[float]
    quarantined: bool
    header_row: List[str] = ()   # always empty until separately verified
    source_crop: str | None = None

@dataclass(frozen=True)
class Chunk:
    ...
    chunk_kind: str = "prose"            # "prose" | "block_descriptor"
    attachments: List[ChunkAttachment] = ()
    has_quarantined_content: bool = False

has_quarantined_content is derivable from attachments, and is serialized anyway. A consumer that never looks at attachments must still be unable to miss the fact — the whole defect being fixed here is a consumer not knowing what it was not told.

2. One descriptor chunk per block, built from metadata only

A block also gets its own chunk so it is retrievable at all:

chunk_id  = "{drug_id}:{section_key}:block:{block_id}"
chunk_kind = "block_descriptor"
text       = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
              trang in 204."

The text is assembled only from verified metadata: drug name, section display name, block kind and printed page. No cell value or inferred header appears. The earlier proposal to use pdfplumber.find_tables()'s first row was rejected after corpus audit: a guessed first row can be a body row or can merge numeric relationships. Until a separate human-verified header dataset exists, header_row is embargoed for every shape and serialized as empty.

3. The answer layer's obligations (binding on ai-service)

These obligations are implemented across ingestion/ and ai-service and are enforced by tests/readiness gates.

1. A retrieved chunk with has_quarantined_content: true must cause the answer to state that a table or formula exists at the cited page, and to surface its rendered crop. The answer may not present itself as complete.
2. A block_descriptor chunk may be answered only with the crop. It must never be paraphrased, and its header_row must never be presented as the table's content.
3. No chunk carrying a quarantined attachment may be used to state a numeric dose. If the dose is in the table, the answer is the crop plus the page.

4. schema_version

monographs.jsonl and the chunk output both gain schema_version. ADR 0005 flagged its absence; a schema that now has two chunk kinds and an attachment list cannot be safely consumed without one.

Alternatives rejected

• Flatten the block into the chunk text. This is the defect, not the fix — it reproduces Clcr = 72 x creatinin exactly.
• Chunk the block's linearised text as an ordinary chunk. Worse than flattening: it makes unsafe text independently retrievable as prose, with its quarantine flag one dereference away from being ignored.
• Drop the blocks. Silent loss, and contrary to the standing rule that unreconstructable content is quarantined with full provenance, never deleted.
• Rely on the prose saying "xem bảng". The prose often does not, and a retrieval layer cannot act on an unstructured hint.
• Wait for row/column reconstruction and do this once. Reconstruction is days of work and would leave the corpus unchunkable meanwhile; worse, it would make the schema question look answered when the silent-incompleteness problem is independent of whether the rows are recovered. Reconstruction later populates rows on the same attachment without touching consumers.

Why a crop is a legitimate answer, not a placeholder

For doctors and pharmacists a rendered crop of the source page is the highest-fidelity response available: it is the book, and it is verifiable at a glance. Reconstruction earns its keep for a different job — comparing or combining values across drugs, which is the synthesis use case this product exists for — not for single-table lookup.

Invariants and gates

Added to cli chunk-ready and to the chunk stage's own tests:

1. section_with_lifted_block_but_no_chunk_reference = 0
2. attachment_block_id_unknown = 0 — every referenced id exists on the monograph
3. attachment_without_page_or_bbox = 0
4. block_text_leaked_into_chunk_text = 0 — no chunk's embedded text contains a quarantined block's text
5. descriptor_chunk_count == block_count
6. descriptor_chunk_without_attachment = 0
7. attachment_header_row_present = 0
8. descriptor_with_unverified_columns = 0
9. descriptor_range_not_attachment_page = 0
10. attachment_without_printed_page = 0

Consequences

• Prose chunks shrink slightly in trustworthiness terms but grow in honesty: the ones missing a table now say so.
• The current candidate index gains 151 descriptor chunks, each cheap and none carrying unsafe text.
• ai-service cannot answer a dosing question from prose alone for the 103 affected sections without violating a stated contract.
• The 14 formula_2d attachments make the two Cockcroft-Gault formulas answerable as crops today, which they are not now.