Add comprehensive, reusable PDF-parsing outlier catalog
This commit is contained in:
@@ -0,0 +1,208 @@
|
||||
# ADR 0003: PDF parsing strategy for the drug formulary — validated empirically
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (validated against the real 1668-page source PDF, not assumptions)
|
||||
|
||||
## Context
|
||||
|
||||
The original scaffold's ingestion design (see `docs/architecture.md` history)
|
||||
assumed generic best practices for structured-PDF parsing: prefer the PDF's
|
||||
bookmark/outline (`doc.get_toc()`) for section boundaries, fall back to
|
||||
font-size heuristics. Before writing real ingestion code, this assumption was
|
||||
tested against the actual `duoc-thu-quoc-gia-viet-nam-2018.pdf` (1668 pages),
|
||||
because a 1668-page book has enough real-world irregularity that guessing
|
||||
from a handful of sample pages is not sufficient grounds to trust a parsing
|
||||
strategy — every claim below was checked against the whole document or a
|
||||
independently-sourced ground truth, not a small sample.
|
||||
|
||||
## What was actually tested
|
||||
|
||||
1. **`doc.get_toc()`**: returns **0 entries**. No usable bookmark/outline.
|
||||
2. **Tagged-PDF structure tree** (`/StructTreeRoot`): exists, but is shallow
|
||||
— ~29 generic `/H1`/`/P` elements, evidently covering only a small
|
||||
fraction of the document. Not usable as a structural signal at scale.
|
||||
Confirmed dead end.
|
||||
3. **Cross-tool text-extraction comparison** on the same real pages
|
||||
(a known drug monograph, "Abacavir"):
|
||||
- **PyMuPDF (`fitz`)**: correct reading order, matches the visual source.
|
||||
- **pdfplumber** (`extract_text()`): **incorrect** — scrambles paragraph
|
||||
order on this layout and surfaces a stray marked-content artifact
|
||||
(`"PB <Header tên thuốc>"`) as if it were visible text. Decision:
|
||||
pdfplumber is kept **only** for its `extract_tables()` API (a genuinely
|
||||
different, table-specific algorithm), never for general body text.
|
||||
- **opendataloader-pdf** (Java-based, benchmarks #1 in public leaderboards
|
||||
for reading order/tables): correct reading order, and its own computed
|
||||
font metadata (per-span `font`/`font size` in its JSON output)
|
||||
**independently agreed** with PyMuPDF's raw span data — two unrelated
|
||||
tools agreeing on the same font facts is real cross-validation, not
|
||||
opinion. However, its higher-level paragraph/heading classifier is
|
||||
**inconsistent**: identical bold section-heading text (e.g. "Dược lý và
|
||||
cơ chế tác dụng", "Liều lượng và cách dùng") is sometimes promoted to a
|
||||
markdown `##` heading and sometimes silently merged into the following
|
||||
body paragraph, for no discernible content-based reason. Conclusion: its
|
||||
Markdown/heading output is not reliable enough to be the sole
|
||||
structural signal, but it's a useful independent check and its
|
||||
header/footer-stripping was notably better than raw PyMuPDF text.
|
||||
- **docling**: attempted, blocked by a `numpy`/`pyarrow` ABI conflict in
|
||||
the environment (numpy 2.x vs a pyarrow build expecting numpy 1.x,
|
||||
pulled in transitively via `torch`/`transformers`). Tested inside an
|
||||
isolated venv rather than fixed globally, to avoid destabilizing other
|
||||
tools on the machine. See progress log for current status.
|
||||
4. **The definitive structural signal — bold font spans**: at the raw
|
||||
PyMuPDF span level, every section heading and every monograph title is
|
||||
rendered in a **bold** font (`"...-BoldMT"`), while body text is not.
|
||||
Italic spans exist too (foreign/Latin species names inline) but are
|
||||
never confused with headings since they're not bold and appear mid
|
||||
sentence. This was cross-confirmed by opendataloader's independently
|
||||
computed font metadata for the same spans (see above) — not a
|
||||
single-tool guess.
|
||||
- **Font size is NOT a reliable discriminator on its own**: a monograph
|
||||
title was observed at both 10.0pt ("ABACAVIR") and 9.5pt ("ACARBOSE")
|
||||
for equally genuine, equally top-level monograph headings. An earlier
|
||||
draft of the detector required `size >= 9.8` based on the first
|
||||
example seen and it silently dropped ~15% of real monographs as a
|
||||
result — a concrete instance of exactly the "don't generalize from one
|
||||
example" risk this investigation was meant to guard against. The fix:
|
||||
drop the size floor; use **bold + all-caps + short line length** for
|
||||
monograph titles, and **bold** alone (cross-checked against the known
|
||||
section-name vocabulary) for section headings.
|
||||
5. **Ground truth for validation**: the book has **two** indexes:
|
||||
- The front-matter "Danh mục các chuyên luận thuốc" (pages 12-31,
|
||||
0-indexed): an alphabetical name list with **no page numbers** — useful
|
||||
only for a name-overlap sanity check, not page-level validation.
|
||||
- The back-of-book "Mục lục tra cứu" (from page ~1529 printed / ~1528
|
||||
0-indexed onward): a proper index with **exact page numbers** per
|
||||
generic-name entry (e.g. `"Abacavir, 101"`), plus brand-name
|
||||
cross-references (`"Ziagen - Abacavir, 101"`, skipped for ground truth).
|
||||
This is the real, page-verifiable ground truth and should be used for
|
||||
any future re-validation, not the front-matter list.
|
||||
- The front matter's own "NỘI DUNG" (table of contents, page 7 0-indexed)
|
||||
also gives exact page ranges for the book's 3 parts: general topic
|
||||
chapters (37-98 printed), individual drug monographs (**99-1496
|
||||
printed**), appendices (1497-1528), back index (1529+). Any monograph-
|
||||
boundary detector should be scoped to the 99-1496 printed page range —
|
||||
scanning the whole book without this scope produces false positives
|
||||
from front-matter/general-chapter bold-caps lines (org names, decree
|
||||
headers, chapter titles) that are not drug monographs.
|
||||
|
||||
## Decision
|
||||
|
||||
- **PyMuPDF is the primary and only general-text extractor.** No TOC
|
||||
dependency, no reliance on the structure tree.
|
||||
- **Section/monograph boundary detection uses bold-font spans** (not font
|
||||
size, not font size + vocabulary alone), scoped to the printed page range
|
||||
of the actual monograph section (99-1496), with all-caps + short length as
|
||||
the additional signal narrowing bold spans down to monograph titles
|
||||
specifically. Multi-line wrapped titles must be merged before matching.
|
||||
- **pdfplumber is retained only for table extraction** (`extract_tables()`),
|
||||
never general reading order, per the confirmed scrambling issue.
|
||||
- **The back-of-book "Mục lục tra cứu" is the ground truth for validation**,
|
||||
not the front-matter drug list.
|
||||
- **Validation is a repeatable, whole-document, automated check**, not a
|
||||
one-time manual read of a handful of pages: a full 1668-page scan runs in
|
||||
under a minute, so re-running it after every heuristic change is cheap and
|
||||
should be standard practice before trusting a change.
|
||||
|
||||
## Validation results (most recent full-document run)
|
||||
|
||||
- Page-verified recall against the back-of-book index: **91.7%** (665/725
|
||||
primary entries had a detected boundary within ±2 physical pages of the
|
||||
expected page).
|
||||
- Remaining misses are overwhelmingly one identified, fixable cause:
|
||||
**multi-line wrapped ALL-CAPS titles** (long Vietnamese drug/vaccine names
|
||||
spanning 2+ physical lines) being matched as fragments rather than merged
|
||||
— not a failure of the bold-span signal itself. A handful of misses are
|
||||
ground-truth extraction noise (the back-index parser occasionally picks up
|
||||
a non-drug appendix/table-of-contents line that happens to match the
|
||||
`"Name, ###"` pattern) rather than real detector failures.
|
||||
- Expected recall after fixing multi-line merging and cleaning non-drug
|
||||
entries out of the ground truth: materially higher than 91.7%, to be
|
||||
re-measured once that fix lands (Phase 1 implementation, not this ADR).
|
||||
|
||||
## Follow-up validation: duplicates and cross-page/column data-loss risk
|
||||
|
||||
Two further questions were raised and empirically tested against the full
|
||||
1405-page monograph range (99-1496 printed):
|
||||
|
||||
1. **Are any drugs detected twice (real content duplication)?** Scanned for
|
||||
normalized-name collisions at physically distant pages. Found exactly
|
||||
**one** candidate: `"GONADOTROPIN"` at physical pages 755 and 1371. On
|
||||
inspection, this is **not** a real duplicate — page 755 is the genuine
|
||||
"GONADOTROPIN" monograph (hCG/menotropin/follitropin), while page 1371 is
|
||||
a different monograph, "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN"
|
||||
(GnRH-analog drugs), whose title wraps across two lines — the detector
|
||||
matched only the second line ("GONADOTROPIN"), colliding with the
|
||||
unrelated monograph's normalized key. This is the **same multi-line
|
||||
title-wrapping bug** already identified above, now confirmed with a
|
||||
second concrete example, not a new failure mode. **Conclusion: no real
|
||||
duplicate monographs found in the corpus**; the multi-line merge fix
|
||||
(already required for Phase 1) also resolves this collision.
|
||||
|
||||
2. **Is the PDF two-column, and can content be lost/corrupted across a page
|
||||
or column boundary during chunking?** Confirmed via bounding-box
|
||||
inspection: this document **is** genuinely two-column (left column
|
||||
x≈44-299, right column x≈308-562, same page). PyMuPDF's block-level
|
||||
reading order correctly sequences left-column-then-right-column content
|
||||
(already implicitly validated by the correct Abacavir sample earlier).
|
||||
However, a **separate, real defect** was found and confirmed: on physical
|
||||
page 1373, one short run of text has its glyphs in **reversed
|
||||
(right-to-left) x-order**, producing scrambled output — e.g. `" = tịx 8
|
||||
yàgn gnàh uềil gnổt( uềihc iổub oàv )magorcim 008 = tịx 4( "`, which
|
||||
reverses character-by-character back to the correct
|
||||
`"(4 xịt = 800 microgam) vào buổi chiều (tổng liều hàng ngày 8 xịt = ..."`.
|
||||
This looks like an isolated PDF-authoring artifact (e.g. an accidental
|
||||
RTL/BiDi override on one small span during editing), not a systemic
|
||||
extraction bug. **Initially this scan was scoped to the monograph range
|
||||
only (1405 of 1668 pages) — an oversight, caught and corrected**:
|
||||
re-run across all 1668 pages (front matter, general chapters,
|
||||
monographs, appendices, back index — the entire book, page 0 to the
|
||||
last page), it still found **exactly 1 affected row, on the same page
|
||||
1373, and no others** — confirming the defect is genuinely isolated, not
|
||||
hiding somewhere in the ~260 pages outside the original scan scope.
|
||||
- The same full-book pass also checked for near-empty pages (<20 chars
|
||||
extracted): found exactly **6** — physical pages 3, 37, 99, 1495, 1497,
|
||||
1666 — every one lands exactly at a major section boundary (before
|
||||
"Các chuyên luận chung" at 37, before "Các chuyên luận thuốc" at 99,
|
||||
before "Các phụ lục" at 1497, near the book's end at 1666). These are
|
||||
intentional print-layout blank/separator pages, not lost content —
|
||||
standard practice to force a new part to start on a fresh page.
|
||||
|
||||
**Implications for Phase 1 implementation:**
|
||||
- Build the pipeline as one **continuous cross-page stream** (text + page
|
||||
number + bbox per fragment, in reading order), not per-page-isolated
|
||||
chunks — this is required both for correctly merging multi-line
|
||||
monograph/section titles (see above) and for never truncating a
|
||||
paragraph/sentence that spans a page or column break.
|
||||
- Add an automated **glyph-order sanity check** as a mandatory pass over
|
||||
100% of pages (not sampled): group text fragments into visual rows by
|
||||
y-coordinate, verify x-coordinates are non-decreasing, and either
|
||||
auto-correct (re-sort by x — the fix is deterministic since raw glyph
|
||||
positions are known) or flag for manual QA. This check is cheap
|
||||
(~16 seconds over the full monograph range) and should run before every
|
||||
real ingestion, not just once.
|
||||
- The book's content must ultimately be captured **from page 0 to the last
|
||||
page** — but not all of it as drug-monograph chunks: front matter (pages
|
||||
0-36) is mostly low-value organizational/decree content and can be
|
||||
largely skipped for RAG purposes; general topic chapters (37-98) and
|
||||
appendices (1497-1528) are real, valuable content that must be ingested
|
||||
too, using their own heading-hierarchy-based chunking (not the drug
|
||||
template) — this was already noted in `docs/architecture.md`'s original
|
||||
design and is reaffirmed here, not changed. The back-of-book index
|
||||
(1529+) does not need its own chunks (it's a page-locator, not content)
|
||||
but remains the validation ground truth.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The real ingestion pipeline (Phase 1) should implement the bold-span
|
||||
detector directly (reusing the validated logic, not the exploratory
|
||||
scratch scripts), scoped to the correct page range, with multi-line
|
||||
heading merging as a required fix before first real ingestion run.
|
||||
- Every future change to the segmentation heuristic should be re-validated
|
||||
with the same whole-document + back-index cross-reference script (or its
|
||||
Phase 1 equivalent) before being trusted — this is now the project's
|
||||
standard rigor bar for this pipeline, not an optional nice-to-have.
|
||||
- `opendataloader-pdf` (Java-based) and `pdfplumber`'s table extraction
|
||||
remain candidate tools for the table/formula-handling fallback path
|
||||
described in `docs/architecture.md`; docling's viability is still
|
||||
unresolved pending the environment fix.
|
||||
Reference in New Issue
Block a user