Fix every real lint finding and drop degenerate splice fragments
This commit is contained in:
@@ -0,0 +1,171 @@
|
||||
# ADR 0006: chunks must carry references to lifted table/formula blocks
|
||||
|
||||
## Status
|
||||
|
||||
Proposed, with implementation to follow immediately. Resolves the item ADR
|
||||
0005 explicitly deferred ("Table/formula content blocks … the precise
|
||||
`ContentBlock`/table-row/formula-unit shape is deferred to a follow-up
|
||||
revision of this ADR once the survey reports real numbers"). The survey has
|
||||
reported.
|
||||
|
||||
## Context
|
||||
|
||||
`segment/` now lifts table and formula regions out of section prose and
|
||||
quarantines them (ADR 0003 lineage, outlier-catalog items 7, 8, 24, 25).
|
||||
That was the right move — linearised, AMPICILIN VÀ SULBACTAM's
|
||||
Cockcroft-Gault fraction read as `Clcr (ml/phút) = 72 x creatinin huyết
|
||||
thanh`, i.e. a division presented as a multiplication, in a renal-dosing
|
||||
section.
|
||||
|
||||
But `chunk/models.py` has no field that refers to a lifted block. Measured on
|
||||
the current whole-corpus output:
|
||||
|
||||
| quantity | value |
|
||||
|---|---|
|
||||
| lifted blocks | 167, all quarantined |
|
||||
| monographs affected | 96 of 683 (**14.1%**) |
|
||||
| sections affected | 108 |
|
||||
| **blocks in `lieu_luong_va_cach_dung`** | **127 (76%)** |
|
||||
| next largest section | `duoc_ly_va_co_che_tac_dung`, 16 |
|
||||
| shapes | simple_table 136, multi_level_or_merged_header 16, formula_2d 14, cross_page_continuation 1 |
|
||||
|
||||
So three quarters of everything removed from prose was removed from the
|
||||
dosing section, in a drug formulary, for an audience of doctors and
|
||||
pharmacists.
|
||||
|
||||
**The failure this creates is silent, not visible.** A chunk of AMPICILIN VÀ
|
||||
SULBACTAM's `lieu_luong_va_cach_dung` is grammatical, complete-looking prose
|
||||
with the renal-dosing table absent and nothing marking the absence. Retrieval
|
||||
ranks it, the model answers from it, and neither has any way to know a table
|
||||
was taken out. A visible error would be safer than this.
|
||||
|
||||
A second, quieter failure: a table is currently **unreachable**. Nothing in
|
||||
the index represents it, so "bảng liều theo chức năng thận của ampicilin"
|
||||
cannot retrieve it even in principle.
|
||||
|
||||
## Decision
|
||||
|
||||
Chunks reference blocks; blocks' content never becomes embedded text.
|
||||
|
||||
### 1. `Chunk` gains typed attachments
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ChunkAttachment:
|
||||
block_id: str
|
||||
kind: str # "table" | "formula"
|
||||
shape: str # simple_table | multi_level_or_merged_header |
|
||||
# cross_page_continuation | formula_2d
|
||||
physical_page: int
|
||||
bbox: List[float]
|
||||
quarantined: bool
|
||||
header_row: List[str] = () # simple_table only; see caveat below
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Chunk:
|
||||
...
|
||||
chunk_kind: str = "prose" # "prose" | "block_descriptor"
|
||||
attachments: List[ChunkAttachment] = ()
|
||||
has_quarantined_content: bool = False
|
||||
```
|
||||
|
||||
`has_quarantined_content` is derivable from `attachments`, and is serialized
|
||||
anyway. A consumer that never looks at `attachments` must still be unable to
|
||||
miss the fact — the whole defect being fixed here is a consumer not knowing
|
||||
what it was not told.
|
||||
|
||||
### 2. One descriptor chunk per block, built from metadata only
|
||||
|
||||
A block also gets its own chunk so it is retrievable at all:
|
||||
|
||||
```
|
||||
chunk_id = "{drug_id}:{section_key}:block:{block_id}"
|
||||
chunk_kind = "block_descriptor"
|
||||
text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
|
||||
trang in 204. Cột: Độ thanh thải creatinin | Nửa đời |
|
||||
Liều ampicilin/sulbactam."
|
||||
```
|
||||
|
||||
The text is assembled from the drug name, the section display name, the kind,
|
||||
the printed page and — for `simple_table` only — the header row. **No cell
|
||||
value ever appears.** A header row is a row of labels; linearising it cannot
|
||||
invent a numeric relationship, which is precisely what linearising a body row
|
||||
does. For every other shape the header is omitted, because
|
||||
`multi_level_or_merged_header` is the shape whose header extraction is least
|
||||
trustworthy.
|
||||
|
||||
Caveat recorded in the schema itself: `header_row` comes from
|
||||
`pdfplumber.find_tables()`'s first row and has **not** been verified by eye
|
||||
(the 180 real tables' individual shapes are rule-derived; only the 20
|
||||
"not a table" verdicts were visually confirmed). It is retrieval bait, never
|
||||
an answer.
|
||||
|
||||
### 3. The answer layer's obligations (binding on `ai-service`)
|
||||
|
||||
These are stated here because they are the reason the schema exists; they are
|
||||
not implemented by `ingestion/`.
|
||||
|
||||
1. A retrieved chunk with `has_quarantined_content: true` **must** cause the
|
||||
answer to state that a table or formula exists at the cited page, and to
|
||||
surface its rendered crop. The answer may not present itself as complete.
|
||||
2. A `block_descriptor` chunk may be answered **only** with the crop. It must
|
||||
never be paraphrased, and its `header_row` must never be presented as the
|
||||
table's content.
|
||||
3. No chunk carrying a quarantined attachment may be used to state a numeric
|
||||
dose. If the dose is in the table, the answer is the crop plus the page.
|
||||
|
||||
### 4. `schema_version`
|
||||
|
||||
`monographs.jsonl` and the chunk output both gain `schema_version`. ADR 0005
|
||||
flagged its absence; a schema that now has two chunk kinds and an attachment
|
||||
list cannot be safely consumed without one.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
- **Flatten the block into the chunk text.** This is the defect, not the fix
|
||||
— it reproduces `Clcr = 72 x creatinin` exactly.
|
||||
- **Chunk the block's linearised text as an ordinary chunk.** Worse than
|
||||
flattening: it makes unsafe text independently retrievable *as prose*, with
|
||||
its quarantine flag one dereference away from being ignored.
|
||||
- **Drop the blocks.** Silent loss, and contrary to the standing rule that
|
||||
unreconstructable content is quarantined with full provenance, never
|
||||
deleted.
|
||||
- **Rely on the prose saying "xem bảng".** The prose often does not, and a
|
||||
retrieval layer cannot act on an unstructured hint.
|
||||
- **Wait for row/column reconstruction and do this once.** Reconstruction is
|
||||
days of work and would leave the corpus unchunkable meanwhile; worse, it
|
||||
would make the schema question look answered when the *silent-incompleteness*
|
||||
problem is independent of whether the rows are recovered. Reconstruction
|
||||
later populates `rows` on the same attachment without touching consumers.
|
||||
|
||||
## Why a crop is a legitimate answer, not a placeholder
|
||||
|
||||
For doctors and pharmacists a rendered crop of the source page is the
|
||||
highest-fidelity response available: it *is* the book, and it is verifiable at
|
||||
a glance. Reconstruction earns its keep for a different job — comparing or
|
||||
combining values across drugs, which is the synthesis use case this product
|
||||
exists for — not for single-table lookup.
|
||||
|
||||
## Invariants and gates
|
||||
|
||||
Added to `cli chunk-ready` and to the chunk stage's own tests:
|
||||
|
||||
1. `section_with_lifted_block_but_no_chunk_reference = 0`
|
||||
2. `attachment_block_id_unknown = 0` — every referenced id exists on the
|
||||
monograph
|
||||
3. `attachment_without_page_or_bbox = 0`
|
||||
4. `block_text_leaked_into_chunk_text = 0` — no chunk's embedded text
|
||||
contains a quarantined block's text
|
||||
5. `descriptor_chunk_count == block_count`
|
||||
6. `descriptor_chunk_without_attachment = 0`
|
||||
|
||||
## Consequences
|
||||
|
||||
- Prose chunks shrink slightly in trustworthiness terms but grow in honesty:
|
||||
the ones missing a table now say so.
|
||||
- The index gains 167 descriptor chunks (≈1.4% of the expected chunk count),
|
||||
each cheap and none carrying unsafe text.
|
||||
- `ai-service` cannot be built to answer a dosing question from prose alone
|
||||
for the 108 affected sections without violating a stated contract.
|
||||
- The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas
|
||||
answerable as crops today, which they are not now.
|
||||
Reference in New Issue
Block a user