Add read-only production runtime audit
This commit is contained in:
@@ -0,0 +1,167 @@
|
||||
# ADR 0006: chunks must carry references to lifted table/formula blocks
|
||||
|
||||
## Status
|
||||
|
||||
Accepted and implemented in schema v4. Resolves the item ADR
|
||||
0005 explicitly deferred ("Table/formula content blocks … the precise
|
||||
`ContentBlock`/table-row/formula-unit shape is deferred to a follow-up
|
||||
revision of this ADR once the survey reports real numbers"). The survey has
|
||||
reported.
|
||||
|
||||
## Context
|
||||
|
||||
`segment/` now lifts table and formula regions out of section prose and
|
||||
quarantines them (ADR 0003 lineage, outlier-catalog items 7, 8, 24, 25).
|
||||
That was the right move — linearised, AMPICILIN VÀ SULBACTAM's
|
||||
Cockcroft-Gault fraction read as `Clcr (ml/phút) = 72 x creatinin huyết
|
||||
thanh`, i.e. a division presented as a multiplication, in a renal-dosing
|
||||
section.
|
||||
|
||||
But `chunk/models.py` has no field that refers to a lifted block. Measured on
|
||||
the current whole-corpus output:
|
||||
|
||||
| quantity | value |
|
||||
|---|---|
|
||||
| lifted blocks represented by descriptor chunks | 151, all quarantined |
|
||||
| sections affected | 103 |
|
||||
| **blocks in `lieu_luong_va_cach_dung`** | **125** |
|
||||
| unverified header rows admitted to embedding text | **0** |
|
||||
|
||||
So three quarters of everything removed from prose was removed from the
|
||||
dosing section, in a drug formulary, for an audience of doctors and
|
||||
pharmacists.
|
||||
|
||||
**The failure this creates is silent, not visible.** A chunk of AMPICILIN VÀ
|
||||
SULBACTAM's `lieu_luong_va_cach_dung` is grammatical, complete-looking prose
|
||||
with the renal-dosing table absent and nothing marking the absence. Retrieval
|
||||
ranks it, the model answers from it, and neither has any way to know a table
|
||||
was taken out. A visible error would be safer than this.
|
||||
|
||||
A second, quieter failure: a table is currently **unreachable**. Nothing in
|
||||
the index represents it, so "bảng liều theo chức năng thận của ampicilin"
|
||||
cannot retrieve it even in principle.
|
||||
|
||||
## Decision
|
||||
|
||||
Chunks reference blocks; blocks' content never becomes embedded text.
|
||||
|
||||
### 1. `Chunk` gains typed attachments
|
||||
|
||||
```python
|
||||
@dataclass(frozen=True)
|
||||
class ChunkAttachment:
|
||||
block_id: str
|
||||
kind: str # "table" | "formula"
|
||||
shape: str # simple_table | multi_level_or_merged_header |
|
||||
# cross_page_continuation | formula_2d
|
||||
physical_page: int
|
||||
printed_page: int
|
||||
bbox: List[float]
|
||||
quarantined: bool
|
||||
header_row: List[str] = () # always empty until separately verified
|
||||
source_crop: str | None = None
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Chunk:
|
||||
...
|
||||
chunk_kind: str = "prose" # "prose" | "block_descriptor"
|
||||
attachments: List[ChunkAttachment] = ()
|
||||
has_quarantined_content: bool = False
|
||||
```
|
||||
|
||||
`has_quarantined_content` is derivable from `attachments`, and is serialized
|
||||
anyway. A consumer that never looks at `attachments` must still be unable to
|
||||
miss the fact — the whole defect being fixed here is a consumer not knowing
|
||||
what it was not told.
|
||||
|
||||
### 2. One descriptor chunk per block, built from metadata only
|
||||
|
||||
A block also gets its own chunk so it is retrievable at all:
|
||||
|
||||
```
|
||||
chunk_id = "{drug_id}:{section_key}:block:{block_id}"
|
||||
chunk_kind = "block_descriptor"
|
||||
text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
|
||||
trang in 204."
|
||||
```
|
||||
|
||||
The text is assembled only from verified metadata: drug name, section display
|
||||
name, block kind and printed page. **No cell value or inferred header appears.**
|
||||
The earlier proposal to use `pdfplumber.find_tables()`'s first row was rejected
|
||||
after corpus audit: a guessed first row can be a body row or can merge numeric
|
||||
relationships. Until a separate human-verified header dataset exists,
|
||||
`header_row` is embargoed for every shape and serialized as empty.
|
||||
|
||||
### 3. The answer layer's obligations (binding on `ai-service`)
|
||||
|
||||
These obligations are implemented across `ingestion/` and `ai-service` and are
|
||||
enforced by tests/readiness gates.
|
||||
|
||||
1. A retrieved chunk with `has_quarantined_content: true` **must** cause the
|
||||
answer to state that a table or formula exists at the cited page, and to
|
||||
surface its rendered crop. The answer may not present itself as complete.
|
||||
2. A `block_descriptor` chunk may be answered **only** with the crop. It must
|
||||
never be paraphrased, and its `header_row` must never be presented as the
|
||||
table's content.
|
||||
3. No chunk carrying a quarantined attachment may be used to state a numeric
|
||||
dose. If the dose is in the table, the answer is the crop plus the page.
|
||||
|
||||
### 4. `schema_version`
|
||||
|
||||
`monographs.jsonl` and the chunk output both gain `schema_version`. ADR 0005
|
||||
flagged its absence; a schema that now has two chunk kinds and an attachment
|
||||
list cannot be safely consumed without one.
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
- **Flatten the block into the chunk text.** This is the defect, not the fix
|
||||
— it reproduces `Clcr = 72 x creatinin` exactly.
|
||||
- **Chunk the block's linearised text as an ordinary chunk.** Worse than
|
||||
flattening: it makes unsafe text independently retrievable *as prose*, with
|
||||
its quarantine flag one dereference away from being ignored.
|
||||
- **Drop the blocks.** Silent loss, and contrary to the standing rule that
|
||||
unreconstructable content is quarantined with full provenance, never
|
||||
deleted.
|
||||
- **Rely on the prose saying "xem bảng".** The prose often does not, and a
|
||||
retrieval layer cannot act on an unstructured hint.
|
||||
- **Wait for row/column reconstruction and do this once.** Reconstruction is
|
||||
days of work and would leave the corpus unchunkable meanwhile; worse, it
|
||||
would make the schema question look answered when the *silent-incompleteness*
|
||||
problem is independent of whether the rows are recovered. Reconstruction
|
||||
later populates `rows` on the same attachment without touching consumers.
|
||||
|
||||
## Why a crop is a legitimate answer, not a placeholder
|
||||
|
||||
For doctors and pharmacists a rendered crop of the source page is the
|
||||
highest-fidelity response available: it *is* the book, and it is verifiable at
|
||||
a glance. Reconstruction earns its keep for a different job — comparing or
|
||||
combining values across drugs, which is the synthesis use case this product
|
||||
exists for — not for single-table lookup.
|
||||
|
||||
## Invariants and gates
|
||||
|
||||
Added to `cli chunk-ready` and to the chunk stage's own tests:
|
||||
|
||||
1. `section_with_lifted_block_but_no_chunk_reference = 0`
|
||||
2. `attachment_block_id_unknown = 0` — every referenced id exists on the
|
||||
monograph
|
||||
3. `attachment_without_page_or_bbox = 0`
|
||||
4. `block_text_leaked_into_chunk_text = 0` — no chunk's embedded text
|
||||
contains a quarantined block's text
|
||||
5. `descriptor_chunk_count == block_count`
|
||||
6. `descriptor_chunk_without_attachment = 0`
|
||||
7. `attachment_header_row_present = 0`
|
||||
8. `descriptor_with_unverified_columns = 0`
|
||||
9. `descriptor_range_not_attachment_page = 0`
|
||||
10. `attachment_without_printed_page = 0`
|
||||
|
||||
## Consequences
|
||||
|
||||
- Prose chunks shrink slightly in trustworthiness terms but grow in honesty:
|
||||
the ones missing a table now say so.
|
||||
- The current candidate index gains 151 descriptor chunks,
|
||||
each cheap and none carrying unsafe text.
|
||||
- `ai-service` cannot answer a dosing question from prose alone for the 103
|
||||
affected sections without violating a stated contract.
|
||||
- The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas
|
||||
answerable as crops today, which they are not now.
|
||||
Reference in New Issue
Block a user