9.2 KiB
Verification strategy — how extraction is actually measured
Short answer to "do you compare characters?": no. Character comparison was tried and rejected twice, for reasons recorded below. What is used instead is a ladder of instruments, each answering a different question, each with a stated blind spot. No single number means "the parse is correct", and this document exists so nobody later mistakes one rung for another.
Status: written 2026-08-01, after the residual-ink work. Every figure quoted here was measured on the whole 1668-page document unless said otherwise.
The rule that governs everything below
An instrument must be checked before its output is believed. In this project the measuring device has been wrong before the data was, repeatedly. Only after an instrument survives its own check does its number get quoted.
Three confirmed cases, all from 2026-08-01:
| what was nearly reported | why it was wrong |
|---|---|
| "extraction ratio 0.6656, 835 pages below 98%" | get_texttrace() counts glyphs painted outside the page rectangle — 4,717,407 of them, on pages that are visually blank |
| "ratio 0.8023, 1642 of 1668 pages below 95%" (after clipping to the page) | Vietnamese diacritics are painted as two glyphs and extracted as one character, so the deficit is systematic and meaningless |
| "page 209's ADR table is unaccounted-for ink" | the residual scan's horizontal banding merged the left and right columns, so the box's centre landed in the gutter and matched no table |
Earlier sessions add three more: a gate comparing post-merge spans against raw spans, one ordering parts by page-y in a two-column book, and one treating a legitimately resuming section as an ordering violation.
Corollary: a non-zero gate is not automatically a data bug. Check the gate, then the data.
Why not character comparison
- Characters cannot be balanced across normalization. The pipeline joins spans that share a visual line, substitutes PUA codepoints for real glyphs, and strips separators. A character in, character out ledger cannot close, so a mismatch tells you nothing.
- Glyph counts cannot stand in for characters. See the table above — both attempts produced confident, wrong numbers.
- Comparing extracted text against another extractor's text measures agreement, not truth, and on this document the tools share a blind spot (§3).
What replaced it: balance at the span level (a unit that survives the pipeline), and verify at the pixel level (a unit that owes nothing to any extractor).
Layer 1 — Span routing ledger: did every span land somewhere?
cli coverage. Each of the 252,733 merged spans is assigned exactly one
state and characters are aggregated from the states.
| state | spans | chars |
|---|---|---|
| normalized_text | 177,679 | 8,182,049 |
| out_of_scope | 53,374 | 897,692 |
| heading | 12,764 | 221,266 |
| boilerplate_excluded | 4,976 | 47,609 |
| quarantined | 3,937 | 48,989 |
| structural_excluded | 3 | 53 |
| unassigned | 0 | 0 |
Proves: nothing the extractor produced was dropped without a name.
Does not prove: that routed content survived downstream. A section-
overwrite bug was invisible to this ledger — spans were correctly marked
normalized_text, then their section was overwritten later.
Does not prove: that the extractor produced everything on the page. That
is Layer 2's job, and it is the gap that mattered most.
Layer 2 — Residual ink: what is on the page that no span accounts for?
cli residual-ink. Render the page, white out every pixel covered by an
extracted span's bbox, measure the ink that survives, and give every
surviving region a name. Needs no ground truth, no sampling, and no second
tool. Measured cost: 0.06 s/page, all 1668 pages in under two minutes.
| kind | regions |
|---|---|
| header_rule | 1,649 |
| text_as_vector_outline | 1,061 |
| table_frame | 959 |
| antialias_speck | 220 |
| fraction_bar_candidate | 23 |
| rule_fragment | 10 |
| header_band_fragment | 9 |
| unclassified | 0 |
This is the only instrument here that does not ask a text layer a question, which is why it found what everything else missed: 51 runs of type that exist only as vector paths (outlier-catalog item 24), invisible to PyMuPDF, pdfplumber and opendataloader-pdf alike.
Proves: every mark on all 1668 pages is accounted for by name.
Does not prove: that the names are right. unclassified = 0 means every
region was named, not that every verdict was checked by eye. Of the seven
kinds, only text_as_vector_outline and fraction_bar_candidate were
confirmed exhaustively; the rest were confirmed on sampled examples.
Calibration matters: at 1.0pt of mask padding the check ate the very
fraction bars it exists to find (page 1042's bar shrank from 188.6pt to
9.1pt). 0.5pt was chosen by measurement, and a regression test pins it.
Layer 3 — Cross-tool agreement: useful, and routinely over-claimed
Inside the monograph range, pdfplumber and opendataloader-pdf agree
exactly on where tables are: same 112 pages, same per-page count, zero
pages found by only one. That looks like strong evidence and is not.
On physical page 1042, both report zero tables. There is a Cockcroft-Gault fraction on that page. Both tools need ruling lines; the bar is a drawn line but not a table, so neither sees it. The same holds on 202.
Rule adopted: agreement between two tools that share a failure mode measures consistency, never recall. Cross-tool agreement may be reported as a reproducibility check and never as coverage evidence.
Where it is genuinely useful: opendataloader's whole-book JSON carries 141 tables / 826 rows / 2,468 cells with per-cell page, bbox, row, column and span — a second independent source of table structure, already on disk.
Layer 4 — Visual census: the only instrument that yields content verdicts
Render the region, read the image, record the verdict. This is what turns a candidate into a fact, and it is the only layer that can say what the text says.
Census when the population is small enough to enumerate. This is stronger than any confidence interval, so prefer it whenever possible:
| population | size | status |
|---|---|---|
| fraction-bar candidates | 23 | all 23 read. 16 real, 7 not → precision 69.6% |
| vector-outlined runs | 51 | all 51 read and transcribed (1,116 characters) |
| "not a table" verdicts | 20 | all 20 read (found 2 wrong) |
| table blocks | 155 | not started |
Sampling only when a census is impossible, and then with the arithmetic stated. Rule of three: inspect n items, find 0 defects, and the 95% upper bound on the defect rate is ≈ 3/n. So "≤ 1% error" costs n ≥ 300 with zero defects; "≤ 5%" costs n ≥ 60. Any "99%" claim that cannot name its n is not a measurement.
Risk-based, not random, when sampling: 100% of table pages, formula pages, monograph boundaries, parser-warning pages and unusual-layout pages, plus a sample of normal pages.
Layer 5 — Invariants the book itself supplies
The source is redundant, and each redundancy is a free check that needs no human ground truth. A violation is a proof of a defect.
- back-of-book index → monograph boundaries (in use: 92.9% recall / 99.1% precision, on a denominator that is not yet cleaned)
"Bảng N"captions → every caption must have a detected table (in use: 32/33)- cross-references (
"xem Liều lượng và cách dùng") → must resolve to a section that exists in the same monograph (not built) - ATC codes → must match the WHO shape
[A-Z]\d\d[A-Z][A-Z]\d\d(not built) - dose ranges (
"4 - 7,5 mg/kg") → must parse as two ordered numbers (not built)
Layer 6 — Fail safe at the point of use
Detection is never complete, so the system must stay safe when it misses.
- every chunk carries
page+bbox; every answer carries a citation - the UI shows the rendered source crop beside the answer, so a pharmacist verifies against the book in seconds
quarantinedcontent andformula_kind: 2dnever enter the model's context as prose — crop or refuse, never linearised text
This is what makes the two Cockcroft-Gault formulas safe today, before any reconstruction exists: left in prose they read as multiplication, which is a dosing error.
What may and may not be said in a report
- Name the denominator every time. "99%" of characters, pages, tables, formulas and monographs are five different claims.
- Distinguish detected / named / verified.
unclassified = 0is "named". - A heuristic finds candidates; it never proves absence. The fraction-bar rule is 69.6% precise and its recall is unknown — and known to be below 100%, because ADENOSIN (page 147) prints a fraction with no bar at all.
- Never write "100%", "complete", "all", "no data lost" or "production-ready" unless the checks performed support the literal claim.
- The honest current shape: "the parser processed 1668/1668 pages; structural checks and 145 tests pass; nothing is lost without being counted. Content accuracy is NOT confirmed at 100% because there is no human-reviewed ground truth for the whole document to diff against."