Fix every real lint finding and drop degenerate splice fragments
This commit is contained in:
@@ -0,0 +1,208 @@
|
||||
# Verification strategy — how extraction is actually measured
|
||||
|
||||
**Short answer to "do you compare characters?": no.** Character comparison
|
||||
was tried and rejected twice, for reasons recorded below. What is used
|
||||
instead is a ladder of instruments, each answering a *different* question,
|
||||
each with a stated blind spot. No single number means "the parse is correct",
|
||||
and this document exists so nobody later mistakes one rung for another.
|
||||
|
||||
Status: written 2026-08-01, after the residual-ink work. Every figure quoted
|
||||
here was measured on the whole 1668-page document unless said otherwise.
|
||||
|
||||
---
|
||||
|
||||
## The rule that governs everything below
|
||||
|
||||
**An instrument must be checked before its output is believed.** In this
|
||||
project the measuring device has been wrong before the data was, repeatedly.
|
||||
Only after an instrument survives its own check does its number get quoted.
|
||||
|
||||
Three confirmed cases, all from 2026-08-01:
|
||||
|
||||
| what was nearly reported | why it was wrong |
|
||||
|---|---|
|
||||
| "extraction ratio 0.6656, 835 pages below 98%" | `get_texttrace()` counts glyphs painted *outside* the page rectangle — 4,717,407 of them, on pages that are visually blank |
|
||||
| "ratio 0.8023, 1642 of 1668 pages below 95%" (after clipping to the page) | Vietnamese diacritics are painted as two glyphs and extracted as one character, so the deficit is systematic and meaningless |
|
||||
| "page 209's ADR table is unaccounted-for ink" | the residual scan's horizontal banding merged the left and right columns, so the box's centre landed in the gutter and matched no table |
|
||||
|
||||
Earlier sessions add three more: a gate comparing post-merge spans against
|
||||
raw spans, one ordering parts by page-y in a two-column book, and one
|
||||
treating a legitimately resuming section as an ordering violation.
|
||||
|
||||
Corollary: **a non-zero gate is not automatically a data bug.** Check the
|
||||
gate, then the data.
|
||||
|
||||
---
|
||||
|
||||
## Why not character comparison
|
||||
|
||||
1. **Characters cannot be balanced across normalization.** The pipeline joins
|
||||
spans that share a visual line, substitutes PUA codepoints for real
|
||||
glyphs, and strips separators. A character in, character out ledger cannot
|
||||
close, so a mismatch tells you nothing.
|
||||
2. **Glyph counts cannot stand in for characters.** See the table above —
|
||||
both attempts produced confident, wrong numbers.
|
||||
3. **Comparing extracted text against another extractor's text measures
|
||||
agreement, not truth**, and on this document the tools share a blind spot
|
||||
(§3).
|
||||
|
||||
What replaced it: balance at the **span** level (a unit that survives the
|
||||
pipeline), and verify at the **pixel** level (a unit that owes nothing to any
|
||||
extractor).
|
||||
|
||||
---
|
||||
|
||||
## Layer 1 — Span routing ledger: did every span land somewhere?
|
||||
|
||||
`cli coverage`. Each of the 252,733 merged spans is assigned exactly one
|
||||
state and characters are aggregated from the states.
|
||||
|
||||
| state | spans | chars |
|
||||
|---|---|---|
|
||||
| normalized_text | 177,679 | 8,182,049 |
|
||||
| out_of_scope | 53,374 | 897,692 |
|
||||
| heading | 12,764 | 221,266 |
|
||||
| boilerplate_excluded | 4,976 | 47,609 |
|
||||
| quarantined | 3,937 | 48,989 |
|
||||
| structural_excluded | 3 | 53 |
|
||||
| **unassigned** | **0** | **0** |
|
||||
|
||||
**Proves:** nothing the extractor produced was dropped without a name.
|
||||
**Does not prove:** that routed content survived downstream. A section-
|
||||
overwrite bug was invisible to this ledger — spans were correctly marked
|
||||
`normalized_text`, then their section was overwritten later.
|
||||
**Does not prove:** that the extractor produced everything on the page. That
|
||||
is Layer 2's job, and it is the gap that mattered most.
|
||||
|
||||
---
|
||||
|
||||
## Layer 2 — Residual ink: what is on the page that no span accounts for?
|
||||
|
||||
`cli residual-ink`. Render the page, white out every pixel covered by an
|
||||
extracted span's bbox, measure the ink that survives, and give every
|
||||
surviving region a name. Needs no ground truth, no sampling, and no second
|
||||
tool. Measured cost: **0.06 s/page, all 1668 pages in under two minutes.**
|
||||
|
||||
| kind | regions |
|
||||
|---|---|
|
||||
| header_rule | 1,649 |
|
||||
| text_as_vector_outline | 1,061 |
|
||||
| table_frame | 959 |
|
||||
| antialias_speck | 220 |
|
||||
| fraction_bar_candidate | 23 |
|
||||
| rule_fragment | 10 |
|
||||
| header_band_fragment | 9 |
|
||||
| **unclassified** | **0** |
|
||||
|
||||
This is the only instrument here that does not ask a text layer a question,
|
||||
which is why it found what everything else missed: **51 runs of type that
|
||||
exist only as vector paths** (outlier-catalog item 24), invisible to
|
||||
PyMuPDF, pdfplumber and opendataloader-pdf alike.
|
||||
|
||||
**Proves:** every mark on all 1668 pages is accounted for by name.
|
||||
**Does not prove:** that the names are right. `unclassified = 0` means every
|
||||
region was *named*, not that every verdict was checked by eye. Of the seven
|
||||
kinds, only `text_as_vector_outline` and `fraction_bar_candidate` were
|
||||
confirmed exhaustively; the rest were confirmed on sampled examples.
|
||||
**Calibration matters:** at 1.0pt of mask padding the check ate the very
|
||||
fraction bars it exists to find (page 1042's bar shrank from 188.6pt to
|
||||
9.1pt). 0.5pt was chosen by measurement, and a regression test pins it.
|
||||
|
||||
---
|
||||
|
||||
## Layer 3 — Cross-tool agreement: useful, and routinely over-claimed
|
||||
|
||||
Inside the monograph range, `pdfplumber` and `opendataloader-pdf` agree
|
||||
*exactly* on where tables are: same 112 pages, same per-page count, zero
|
||||
pages found by only one. That looks like strong evidence and is not.
|
||||
|
||||
**On physical page 1042, both report zero tables.** There is a
|
||||
Cockcroft-Gault fraction on that page. Both tools need ruling lines; the bar
|
||||
is a drawn line but not a table, so neither sees it. The same holds on 202.
|
||||
|
||||
**Rule adopted:** agreement between two tools that share a failure mode
|
||||
measures *consistency*, never *recall*. Cross-tool agreement may be reported
|
||||
as a reproducibility check and never as coverage evidence.
|
||||
|
||||
Where it is genuinely useful: opendataloader's whole-book JSON carries 141
|
||||
tables / 826 rows / 2,468 cells with per-cell page, bbox, row, column and
|
||||
span — a second independent source of table structure, already on disk.
|
||||
|
||||
---
|
||||
|
||||
## Layer 4 — Visual census: the only instrument that yields content verdicts
|
||||
|
||||
Render the region, read the image, record the verdict. This is what turns a
|
||||
candidate into a fact, and it is the only layer that can say what the text
|
||||
*says*.
|
||||
|
||||
**Census when the population is small enough to enumerate.** This is stronger
|
||||
than any confidence interval, so prefer it whenever possible:
|
||||
|
||||
| population | size | status |
|
||||
|---|---|---|
|
||||
| fraction-bar candidates | 23 | **all 23 read.** 16 real, 7 not → precision **69.6%** |
|
||||
| vector-outlined runs | 51 | **all 51 read and transcribed** (1,116 characters) |
|
||||
| "not a table" verdicts | 20 | all 20 read (found 2 wrong) |
|
||||
| table blocks | 155 | not started |
|
||||
|
||||
**Sampling only when a census is impossible**, and then with the arithmetic
|
||||
stated. Rule of three: inspect *n* items, find **0** defects, and the 95%
|
||||
upper bound on the defect rate is ≈ 3/n. So "≤ 1% error" costs **n ≥ 300 with
|
||||
zero defects**; "≤ 5%" costs n ≥ 60. Any "99%" claim that cannot name its *n*
|
||||
is not a measurement.
|
||||
|
||||
**Risk-based, not random**, when sampling: 100% of table pages, formula
|
||||
pages, monograph boundaries, parser-warning pages and unusual-layout pages,
|
||||
plus a sample of normal pages.
|
||||
|
||||
---
|
||||
|
||||
## Layer 5 — Invariants the book itself supplies
|
||||
|
||||
The source is redundant, and each redundancy is a free check that needs no
|
||||
human ground truth. A violation is a proof of a defect.
|
||||
|
||||
- back-of-book index → monograph boundaries (in use: 92.9% recall / 99.1%
|
||||
precision, on a denominator that is **not yet cleaned**)
|
||||
- `"Bảng N"` captions → every caption must have a detected table (in use:
|
||||
32/33)
|
||||
- cross-references (`"xem Liều lượng và cách dùng"`) → must resolve to a
|
||||
section that exists in the same monograph (**not built**)
|
||||
- ATC codes → must match the WHO shape `[A-Z]\d\d[A-Z][A-Z]\d\d` (**not
|
||||
built**)
|
||||
- dose ranges (`"4 - 7,5 mg/kg"`) → must parse as two ordered numbers
|
||||
(**not built**)
|
||||
|
||||
---
|
||||
|
||||
## Layer 6 — Fail safe at the point of use
|
||||
|
||||
Detection is never complete, so the system must stay safe when it misses.
|
||||
|
||||
- every chunk carries `page` + `bbox`; every answer carries a citation
|
||||
- the UI shows the **rendered source crop** beside the answer, so a
|
||||
pharmacist verifies against the book in seconds
|
||||
- `quarantined` content and `formula_kind: 2d` never enter the model's
|
||||
context as prose — crop or refuse, never linearised text
|
||||
|
||||
This is what makes the two Cockcroft-Gault formulas safe *today*, before any
|
||||
reconstruction exists: left in prose they read as multiplication, which is a
|
||||
dosing error.
|
||||
|
||||
---
|
||||
|
||||
## What may and may not be said in a report
|
||||
|
||||
- Name the **denominator** every time. "99%" of characters, pages, tables,
|
||||
formulas and monographs are five different claims.
|
||||
- Distinguish **detected / named / verified**. `unclassified = 0` is "named".
|
||||
- A heuristic finds **candidates**; it never proves absence. The fraction-bar
|
||||
rule is 69.6% precise and its recall is unknown — and known to be below
|
||||
100%, because ADENOSIN (page 147) prints a fraction with no bar at all.
|
||||
- Never write "100%", "complete", "all", "no data lost" or "production-ready"
|
||||
unless the checks performed support the literal claim.
|
||||
- The honest current shape: *"the parser processed 1668/1668 pages;
|
||||
structural checks and 145 tests pass; nothing is lost without being
|
||||
counted. Content accuracy is NOT confirmed at 100% because there is no
|
||||
human-reviewed ground truth for the whole document to diff against."*
|
||||
Reference in New Issue
Block a user