# ADR 0006: chunks must carry references to lifted table/formula blocks ## Status Proposed, with implementation to follow immediately. Resolves the item ADR 0005 explicitly deferred ("Table/formula content blocks … the precise `ContentBlock`/table-row/formula-unit shape is deferred to a follow-up revision of this ADR once the survey reports real numbers"). The survey has reported. ## Context `segment/` now lifts table and formula regions out of section prose and quarantines them (ADR 0003 lineage, outlier-catalog items 7, 8, 24, 25). That was the right move — linearised, AMPICILIN VÀ SULBACTAM's Cockcroft-Gault fraction read as `Clcr (ml/phút) = 72 x creatinin huyết thanh`, i.e. a division presented as a multiplication, in a renal-dosing section. But `chunk/models.py` has no field that refers to a lifted block. Measured on the current whole-corpus output: | quantity | value | |---|---| | lifted blocks | 167, all quarantined | | monographs affected | 96 of 683 (**14.1%**) | | sections affected | 108 | | **blocks in `lieu_luong_va_cach_dung`** | **127 (76%)** | | next largest section | `duoc_ly_va_co_che_tac_dung`, 16 | | shapes | simple_table 136, multi_level_or_merged_header 16, formula_2d 14, cross_page_continuation 1 | So three quarters of everything removed from prose was removed from the dosing section, in a drug formulary, for an audience of doctors and pharmacists. **The failure this creates is silent, not visible.** A chunk of AMPICILIN VÀ SULBACTAM's `lieu_luong_va_cach_dung` is grammatical, complete-looking prose with the renal-dosing table absent and nothing marking the absence. Retrieval ranks it, the model answers from it, and neither has any way to know a table was taken out. A visible error would be safer than this. A second, quieter failure: a table is currently **unreachable**. Nothing in the index represents it, so "bảng liều theo chức năng thận của ampicilin" cannot retrieve it even in principle. ## Decision Chunks reference blocks; blocks' content never becomes embedded text. ### 1. `Chunk` gains typed attachments ```python @dataclass(frozen=True) class ChunkAttachment: block_id: str kind: str # "table" | "formula" shape: str # simple_table | multi_level_or_merged_header | # cross_page_continuation | formula_2d physical_page: int bbox: List[float] quarantined: bool header_row: List[str] = () # simple_table only; see caveat below @dataclass(frozen=True) class Chunk: ... chunk_kind: str = "prose" # "prose" | "block_descriptor" attachments: List[ChunkAttachment] = () has_quarantined_content: bool = False ``` `has_quarantined_content` is derivable from `attachments`, and is serialized anyway. A consumer that never looks at `attachments` must still be unable to miss the fact — the whole defect being fixed here is a consumer not knowing what it was not told. ### 2. One descriptor chunk per block, built from metadata only A block also gets its own chunk so it is retrievable at all: ``` chunk_id = "{drug_id}:{section_key}:block:{block_id}" chunk_kind = "block_descriptor" text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng, trang in 204. Cột: Độ thanh thải creatinin | Nửa đời | Liều ampicilin/sulbactam." ``` The text is assembled from the drug name, the section display name, the kind, the printed page and — for `simple_table` only — the header row. **No cell value ever appears.** A header row is a row of labels; linearising it cannot invent a numeric relationship, which is precisely what linearising a body row does. For every other shape the header is omitted, because `multi_level_or_merged_header` is the shape whose header extraction is least trustworthy. Caveat recorded in the schema itself: `header_row` comes from `pdfplumber.find_tables()`'s first row and has **not** been verified by eye (the 180 real tables' individual shapes are rule-derived; only the 20 "not a table" verdicts were visually confirmed). It is retrieval bait, never an answer. ### 3. The answer layer's obligations (binding on `ai-service`) These are stated here because they are the reason the schema exists; they are not implemented by `ingestion/`. 1. A retrieved chunk with `has_quarantined_content: true` **must** cause the answer to state that a table or formula exists at the cited page, and to surface its rendered crop. The answer may not present itself as complete. 2. A `block_descriptor` chunk may be answered **only** with the crop. It must never be paraphrased, and its `header_row` must never be presented as the table's content. 3. No chunk carrying a quarantined attachment may be used to state a numeric dose. If the dose is in the table, the answer is the crop plus the page. ### 4. `schema_version` `monographs.jsonl` and the chunk output both gain `schema_version`. ADR 0005 flagged its absence; a schema that now has two chunk kinds and an attachment list cannot be safely consumed without one. ## Alternatives rejected - **Flatten the block into the chunk text.** This is the defect, not the fix — it reproduces `Clcr = 72 x creatinin` exactly. - **Chunk the block's linearised text as an ordinary chunk.** Worse than flattening: it makes unsafe text independently retrievable *as prose*, with its quarantine flag one dereference away from being ignored. - **Drop the blocks.** Silent loss, and contrary to the standing rule that unreconstructable content is quarantined with full provenance, never deleted. - **Rely on the prose saying "xem bảng".** The prose often does not, and a retrieval layer cannot act on an unstructured hint. - **Wait for row/column reconstruction and do this once.** Reconstruction is days of work and would leave the corpus unchunkable meanwhile; worse, it would make the schema question look answered when the *silent-incompleteness* problem is independent of whether the rows are recovered. Reconstruction later populates `rows` on the same attachment without touching consumers. ## Why a crop is a legitimate answer, not a placeholder For doctors and pharmacists a rendered crop of the source page is the highest-fidelity response available: it *is* the book, and it is verifiable at a glance. Reconstruction earns its keep for a different job — comparing or combining values across drugs, which is the synthesis use case this product exists for — not for single-table lookup. ## Invariants and gates Added to `cli chunk-ready` and to the chunk stage's own tests: 1. `section_with_lifted_block_but_no_chunk_reference = 0` 2. `attachment_block_id_unknown = 0` — every referenced id exists on the monograph 3. `attachment_without_page_or_bbox = 0` 4. `block_text_leaked_into_chunk_text = 0` — no chunk's embedded text contains a quarantined block's text 5. `descriptor_chunk_count == block_count` 6. `descriptor_chunk_without_attachment = 0` ## Consequences - Prose chunks shrink slightly in trustworthiness terms but grow in honesty: the ones missing a table now say so. - The index gains 167 descriptor chunks (≈1.4% of the expected chunk count), each cheap and none carrying unsafe text. - `ai-service` cannot be built to answer a dosing question from prose alone for the 108 affected sections without violating a stated contract. - The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas answerable as crops today, which they are not now.