209 lines
9.2 KiB
Markdown
209 lines
9.2 KiB
Markdown
# Verification strategy — how extraction is actually measured
|
|
|
|
**Short answer to "do you compare characters?": no.** Character comparison
|
|
was tried and rejected twice, for reasons recorded below. What is used
|
|
instead is a ladder of instruments, each answering a *different* question,
|
|
each with a stated blind spot. No single number means "the parse is correct",
|
|
and this document exists so nobody later mistakes one rung for another.
|
|
|
|
Status: written 2026-08-01, after the residual-ink work. Every figure quoted
|
|
here was measured on the whole 1668-page document unless said otherwise.
|
|
|
|
---
|
|
|
|
## The rule that governs everything below
|
|
|
|
**An instrument must be checked before its output is believed.** In this
|
|
project the measuring device has been wrong before the data was, repeatedly.
|
|
Only after an instrument survives its own check does its number get quoted.
|
|
|
|
Three confirmed cases, all from 2026-08-01:
|
|
|
|
| what was nearly reported | why it was wrong |
|
|
|---|---|
|
|
| "extraction ratio 0.6656, 835 pages below 98%" | `get_texttrace()` counts glyphs painted *outside* the page rectangle — 4,717,407 of them, on pages that are visually blank |
|
|
| "ratio 0.8023, 1642 of 1668 pages below 95%" (after clipping to the page) | Vietnamese diacritics are painted as two glyphs and extracted as one character, so the deficit is systematic and meaningless |
|
|
| "page 209's ADR table is unaccounted-for ink" | the residual scan's horizontal banding merged the left and right columns, so the box's centre landed in the gutter and matched no table |
|
|
|
|
Earlier sessions add three more: a gate comparing post-merge spans against
|
|
raw spans, one ordering parts by page-y in a two-column book, and one
|
|
treating a legitimately resuming section as an ordering violation.
|
|
|
|
Corollary: **a non-zero gate is not automatically a data bug.** Check the
|
|
gate, then the data.
|
|
|
|
---
|
|
|
|
## Why not character comparison
|
|
|
|
1. **Characters cannot be balanced across normalization.** The pipeline joins
|
|
spans that share a visual line, substitutes PUA codepoints for real
|
|
glyphs, and strips separators. A character in, character out ledger cannot
|
|
close, so a mismatch tells you nothing.
|
|
2. **Glyph counts cannot stand in for characters.** See the table above —
|
|
both attempts produced confident, wrong numbers.
|
|
3. **Comparing extracted text against another extractor's text measures
|
|
agreement, not truth**, and on this document the tools share a blind spot
|
|
(§3).
|
|
|
|
What replaced it: balance at the **span** level (a unit that survives the
|
|
pipeline), and verify at the **pixel** level (a unit that owes nothing to any
|
|
extractor).
|
|
|
|
---
|
|
|
|
## Layer 1 — Span routing ledger: did every span land somewhere?
|
|
|
|
`cli coverage`. Each of the 252,733 merged spans is assigned exactly one
|
|
state and characters are aggregated from the states.
|
|
|
|
| state | spans | chars |
|
|
|---|---|---|
|
|
| normalized_text | 177,679 | 8,182,049 |
|
|
| out_of_scope | 53,374 | 897,692 |
|
|
| heading | 12,764 | 221,266 |
|
|
| boilerplate_excluded | 4,976 | 47,609 |
|
|
| quarantined | 3,937 | 48,989 |
|
|
| structural_excluded | 3 | 53 |
|
|
| **unassigned** | **0** | **0** |
|
|
|
|
**Proves:** nothing the extractor produced was dropped without a name.
|
|
**Does not prove:** that routed content survived downstream. A section-
|
|
overwrite bug was invisible to this ledger — spans were correctly marked
|
|
`normalized_text`, then their section was overwritten later.
|
|
**Does not prove:** that the extractor produced everything on the page. That
|
|
is Layer 2's job, and it is the gap that mattered most.
|
|
|
|
---
|
|
|
|
## Layer 2 — Residual ink: what is on the page that no span accounts for?
|
|
|
|
`cli residual-ink`. Render the page, white out every pixel covered by an
|
|
extracted span's bbox, measure the ink that survives, and give every
|
|
surviving region a name. Needs no ground truth, no sampling, and no second
|
|
tool. Measured cost: **0.06 s/page, all 1668 pages in under two minutes.**
|
|
|
|
| kind | regions |
|
|
|---|---|
|
|
| header_rule | 1,649 |
|
|
| text_as_vector_outline | 1,061 |
|
|
| table_frame | 959 |
|
|
| antialias_speck | 220 |
|
|
| fraction_bar_candidate | 23 |
|
|
| rule_fragment | 10 |
|
|
| header_band_fragment | 9 |
|
|
| **unclassified** | **0** |
|
|
|
|
This is the only instrument here that does not ask a text layer a question,
|
|
which is why it found what everything else missed: **51 runs of type that
|
|
exist only as vector paths** (outlier-catalog item 24), invisible to
|
|
PyMuPDF, pdfplumber and opendataloader-pdf alike.
|
|
|
|
**Proves:** every mark on all 1668 pages is accounted for by name.
|
|
**Does not prove:** that the names are right. `unclassified = 0` means every
|
|
region was *named*, not that every verdict was checked by eye. Of the seven
|
|
kinds, only `text_as_vector_outline` and `fraction_bar_candidate` were
|
|
confirmed exhaustively; the rest were confirmed on sampled examples.
|
|
**Calibration matters:** at 1.0pt of mask padding the check ate the very
|
|
fraction bars it exists to find (page 1042's bar shrank from 188.6pt to
|
|
9.1pt). 0.5pt was chosen by measurement, and a regression test pins it.
|
|
|
|
---
|
|
|
|
## Layer 3 — Cross-tool agreement: useful, and routinely over-claimed
|
|
|
|
Inside the monograph range, `pdfplumber` and `opendataloader-pdf` agree
|
|
*exactly* on where tables are: same 112 pages, same per-page count, zero
|
|
pages found by only one. That looks like strong evidence and is not.
|
|
|
|
**On physical page 1042, both report zero tables.** There is a
|
|
Cockcroft-Gault fraction on that page. Both tools need ruling lines; the bar
|
|
is a drawn line but not a table, so neither sees it. The same holds on 202.
|
|
|
|
**Rule adopted:** agreement between two tools that share a failure mode
|
|
measures *consistency*, never *recall*. Cross-tool agreement may be reported
|
|
as a reproducibility check and never as coverage evidence.
|
|
|
|
Where it is genuinely useful: opendataloader's whole-book JSON carries 141
|
|
tables / 826 rows / 2,468 cells with per-cell page, bbox, row, column and
|
|
span — a second independent source of table structure, already on disk.
|
|
|
|
---
|
|
|
|
## Layer 4 — Visual census: the only instrument that yields content verdicts
|
|
|
|
Render the region, read the image, record the verdict. This is what turns a
|
|
candidate into a fact, and it is the only layer that can say what the text
|
|
*says*.
|
|
|
|
**Census when the population is small enough to enumerate.** This is stronger
|
|
than any confidence interval, so prefer it whenever possible:
|
|
|
|
| population | size | status |
|
|
|---|---|---|
|
|
| fraction-bar candidates | 23 | **all 23 read.** 16 real, 7 not → precision **69.6%** |
|
|
| vector-outlined runs | 51 | **all 51 read and transcribed** (1,116 characters) |
|
|
| "not a table" verdicts | 20 | all 20 read (found 2 wrong) |
|
|
| table blocks | 155 | not started |
|
|
|
|
**Sampling only when a census is impossible**, and then with the arithmetic
|
|
stated. Rule of three: inspect *n* items, find **0** defects, and the 95%
|
|
upper bound on the defect rate is ≈ 3/n. So "≤ 1% error" costs **n ≥ 300 with
|
|
zero defects**; "≤ 5%" costs n ≥ 60. Any "99%" claim that cannot name its *n*
|
|
is not a measurement.
|
|
|
|
**Risk-based, not random**, when sampling: 100% of table pages, formula
|
|
pages, monograph boundaries, parser-warning pages and unusual-layout pages,
|
|
plus a sample of normal pages.
|
|
|
|
---
|
|
|
|
## Layer 5 — Invariants the book itself supplies
|
|
|
|
The source is redundant, and each redundancy is a free check that needs no
|
|
human ground truth. A violation is a proof of a defect.
|
|
|
|
- back-of-book index → monograph boundaries (in use: 92.9% recall / 99.1%
|
|
precision, on a denominator that is **not yet cleaned**)
|
|
- `"Bảng N"` captions → every caption must have a detected table (in use:
|
|
32/33)
|
|
- cross-references (`"xem Liều lượng và cách dùng"`) → must resolve to a
|
|
section that exists in the same monograph (**not built**)
|
|
- ATC codes → must match the WHO shape `[A-Z]\d\d[A-Z][A-Z]\d\d` (**not
|
|
built**)
|
|
- dose ranges (`"4 - 7,5 mg/kg"`) → must parse as two ordered numbers
|
|
(**not built**)
|
|
|
|
---
|
|
|
|
## Layer 6 — Fail safe at the point of use
|
|
|
|
Detection is never complete, so the system must stay safe when it misses.
|
|
|
|
- every chunk carries `page` + `bbox`; every answer carries a citation
|
|
- the UI shows the **rendered source crop** beside the answer, so a
|
|
pharmacist verifies against the book in seconds
|
|
- `quarantined` content and `formula_kind: 2d` never enter the model's
|
|
context as prose — crop or refuse, never linearised text
|
|
|
|
This is what makes the two Cockcroft-Gault formulas safe *today*, before any
|
|
reconstruction exists: left in prose they read as multiplication, which is a
|
|
dosing error.
|
|
|
|
---
|
|
|
|
## What may and may not be said in a report
|
|
|
|
- Name the **denominator** every time. "99%" of characters, pages, tables,
|
|
formulas and monographs are five different claims.
|
|
- Distinguish **detected / named / verified**. `unclassified = 0` is "named".
|
|
- A heuristic finds **candidates**; it never proves absence. The fraction-bar
|
|
rule is 69.6% precise and its recall is unknown — and known to be below
|
|
100%, because ADENOSIN (page 147) prints a fraction with no bar at all.
|
|
- Never write "100%", "complete", "all", "no data lost" or "production-ready"
|
|
unless the checks performed support the literal claim.
|
|
- The honest current shape: *"the parser processed 1668/1668 pages;
|
|
structural checks and 145 tests pass; nothing is lost without being
|
|
counted. Content accuracy is NOT confirmed at 100% because there is no
|
|
human-reviewed ground truth for the whole document to diff against."*
|