ADR 0003: PDF parsing strategy for the drug formulary — validated empirically

Status

Accepted (validated against the real 1668-page source PDF, not assumptions)

Context

The original scaffold's ingestion design (see docs/architecture.md history) assumed generic best practices for structured-PDF parsing: prefer the PDF's bookmark/outline (doc.get_toc()) for section boundaries, fall back to font-size heuristics. Before writing real ingestion code, this assumption was tested against the actual duoc-thu-quoc-gia-viet-nam-2018.pdf (1668 pages), because a 1668-page book has enough real-world irregularity that guessing from a handful of sample pages is not sufficient grounds to trust a parsing strategy — every claim below was checked against the whole document or a independently-sourced ground truth, not a small sample.

What was actually tested

doc.get_toc(): returns 0 entries. No usable bookmark/outline.
Tagged-PDF structure tree (/StructTreeRoot): exists, but is shallow — ~29 generic /H1//P elements, evidently covering only a small fraction of the document. Not usable as a structural signal at scale. Confirmed dead end.
Cross-tool text-extraction comparison on the same real pages (a known drug monograph, "Abacavir"):
- PyMuPDF (fitz): correct reading order, matches the visual source.
- pdfplumber (extract_text()): incorrect — scrambles paragraph order on this layout and surfaces a stray marked-content artifact ("PB <Header tên thuốc>") as if it were visible text. Decision: pdfplumber is kept only for its extract_tables() API (a genuinely different, table-specific algorithm), never for general body text.
- opendataloader-pdf (Java-based, benchmarks #1 in public leaderboards for reading order/tables): correct reading order, and its own computed font metadata (per-span font/font size in its JSON output) independently agreed with PyMuPDF's raw span data — two unrelated tools agreeing on the same font facts is real cross-validation, not opinion. However, its higher-level paragraph/heading classifier is inconsistent: identical bold section-heading text (e.g. "Dược lý và cơ chế tác dụng", "Liều lượng và cách dùng") is sometimes promoted to a markdown ## heading and sometimes silently merged into the following body paragraph, for no discernible content-based reason. Conclusion: its Markdown/heading output is not reliable enough to be the sole structural signal, but it's a useful independent check and its header/footer-stripping was notably better than raw PyMuPDF text.
- docling: attempted, blocked by a numpy/pyarrow ABI conflict in the environment (numpy 2.x vs a pyarrow build expecting numpy 1.x, pulled in transitively via torch/transformers). Tested inside an isolated venv rather than fixed globally, to avoid destabilizing other tools on the machine. See progress log for current status.
The definitive structural signal — bold font spans: at the raw PyMuPDF span level, every section heading and every monograph title is rendered in a bold font ("...-BoldMT"), while body text is not. Italic spans exist too (foreign/Latin species names inline) but are never confused with headings since they're not bold and appear mid sentence. This was cross-confirmed by opendataloader's independently computed font metadata for the same spans (see above) — not a single-tool guess.
- Font size is NOT a reliable discriminator on its own: a monograph title was observed at both 10.0pt ("ABACAVIR") and 9.5pt ("ACARBOSE") for equally genuine, equally top-level monograph headings. An earlier draft of the detector required size >= 9.8 based on the first example seen and it silently dropped ~15% of real monographs as a result — a concrete instance of exactly the "don't generalize from one example" risk this investigation was meant to guard against. The fix: drop the size floor; use bold + all-caps + short line length for monograph titles, and bold alone (cross-checked against the known section-name vocabulary) for section headings.
Ground truth for validation: the book has two indexes:
- The front-matter "Danh mục các chuyên luận thuốc" (pages 12-31, 0-indexed): an alphabetical name list with no page numbers — useful only for a name-overlap sanity check, not page-level validation.
- The back-of-book "Mục lục tra cứu" (from page ~1529 printed / ~1528 0-indexed onward): a proper index with exact page numbers per generic-name entry (e.g. "Abacavir, 101"), plus brand-name cross-references ("Ziagen - Abacavir, 101", skipped for ground truth). This is the real, page-verifiable ground truth and should be used for any future re-validation, not the front-matter list.
- The front matter's own "NỘI DUNG" (table of contents, page 7 0-indexed) also gives exact page ranges for the book's 3 parts: general topic chapters (37-98 printed), individual drug monographs (99-1496 printed), appendices (1497-1528), back index (1529+). Any monograph- boundary detector should be scoped to the 99-1496 printed page range — scanning the whole book without this scope produces false positives from front-matter/general-chapter bold-caps lines (org names, decree headers, chapter titles) that are not drug monographs.

Decision

PyMuPDF is the primary and only general-text extractor. No TOC dependency, no reliance on the structure tree.
Section/monograph boundary detection uses bold-font spans (not font size, not font size + vocabulary alone), scoped to the printed page range of the actual monograph section (99-1496), with all-caps + short length as the additional signal narrowing bold spans down to monograph titles specifically. Multi-line wrapped titles must be merged before matching.
pdfplumber is retained only for table extraction (extract_tables()), never general reading order, per the confirmed scrambling issue.
The back-of-book "Mục lục tra cứu" is the ground truth for validation, not the front-matter drug list.
Validation is a repeatable, whole-document, automated check, not a one-time manual read of a handful of pages: a full 1668-page scan runs in under a minute, so re-running it after every heuristic change is cheap and should be standard practice before trusting a change.

Validation results (most recent full-document run)

Page-verified recall against the back-of-book index: 91.7% (665/725 primary entries had a detected boundary within ±2 physical pages of the expected page).
Remaining misses are overwhelmingly one identified, fixable cause: multi-line wrapped ALL-CAPS titles (long Vietnamese drug/vaccine names spanning 2+ physical lines) being matched as fragments rather than merged — not a failure of the bold-span signal itself. A handful of misses are ground-truth extraction noise (the back-index parser occasionally picks up a non-drug appendix/table-of-contents line that happens to match the "Name, ###" pattern) rather than real detector failures.
Expected recall after fixing multi-line merging and cleaning non-drug entries out of the ground truth: materially higher than 91.7%, to be re-measured once that fix lands (Phase 1 implementation, not this ADR).

Follow-up validation: duplicates and cross-page/column data-loss risk

Two further questions were raised and empirically tested against the full 1405-page monograph range (99-1496 printed):

Are any drugs detected twice (real content duplication)? Scanned for normalized-name collisions at physically distant pages. Found exactly one candidate: "GONADOTROPIN" at physical pages 755 and 1371. On inspection, this is not a real duplicate — page 755 is the genuine "GONADOTROPIN" monograph (hCG/menotropin/follitropin), while page 1371 is a different monograph, "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN" (GnRH-analog drugs), whose title wraps across two lines — the detector matched only the second line ("GONADOTROPIN"), colliding with the unrelated monograph's normalized key. This is the same multi-line title-wrapping bug already identified above, now confirmed with a second concrete example, not a new failure mode. Conclusion: no real duplicate monographs found in the corpus; the multi-line merge fix (already required for Phase 1) also resolves this collision.
Is the PDF two-column, and can content be lost/corrupted across a page or column boundary during chunking? Confirmed via bounding-box inspection: this document is genuinely two-column (left column x≈44-299, right column x≈308-562, same page). PyMuPDF's block-level reading order correctly sequences left-column-then-right-column content (already implicitly validated by the correct Abacavir sample earlier). However, a separate, real defect was found and confirmed: on physical page 1373, one short run of text has its glyphs in reversed (right-to-left) x-order, producing scrambled output — e.g. " = tịx 8 yàgn gnàh uềil gnổt( uềihc iổub oàv )magorcim 008 = tịx 4( ", which reverses character-by-character back to the correct "(4 xịt = 800 microgam) vào buổi chiều (tổng liều hàng ngày 8 xịt = ...". This looks like an isolated PDF-authoring artifact (e.g. an accidental RTL/BiDi override on one small span during editing), not a systemic extraction bug. Initially this scan was scoped to the monograph range only (1405 of 1668 pages) — an oversight, caught and corrected: re-run across all 1668 pages (front matter, general chapters, monographs, appendices, back index — the entire book, page 0 to the last page), it still found exactly 1 affected row, on the same page 1373, and no others — confirming the defect is genuinely isolated, not hiding somewhere in the ~260 pages outside the original scan scope.
- The same full-book pass also checked for near-empty pages (<20 chars extracted): found exactly 6 — physical pages 3, 37, 99, 1495, 1497, 1666 — every one lands exactly at a major section boundary (before "Các chuyên luận chung" at 37, before "Các chuyên luận thuốc" at 99, before "Các phụ lục" at 1497, near the book's end at 1666). These are intentional print-layout blank/separator pages, not lost content — standard practice to force a new part to start on a fresh page.

Implications for Phase 1 implementation:

Build the pipeline as one continuous cross-page stream (text + page number + bbox per fragment, in reading order), not per-page-isolated chunks — this is required both for correctly merging multi-line monograph/section titles (see above) and for never truncating a paragraph/sentence that spans a page or column break.
Add an automated glyph-order sanity check as a mandatory pass over 100% of pages (not sampled): group text fragments into visual rows by y-coordinate, verify x-coordinates are non-decreasing, and either auto-correct (re-sort by x — the fix is deterministic since raw glyph positions are known) or flag for manual QA. This check is cheap (~16 seconds over the full monograph range) and should run before every real ingestion, not just once.
The book's content must ultimately be captured from page 0 to the last page — but not all of it as drug-monograph chunks: front matter (pages 0-36) is mostly low-value organizational/decree content and can be largely skipped for RAG purposes; general topic chapters (37-98) and appendices (1497-1528) are real, valuable content that must be ingested too, using their own heading-hierarchy-based chunking (not the drug template) — this was already noted in docs/architecture.md's original design and is reaffirmed here, not changed. The back-of-book index (1529+) does not need its own chunks (it's a page-locator, not content) but remains the validation ground truth.

Consequences

The real ingestion pipeline (Phase 1) should implement the bold-span detector directly (reusing the validated logic, not the exploratory scratch scripts), scoped to the correct page range, with multi-line heading merging as a required fix before first real ingestion run.
Every future change to the segmentation heuristic should be re-validated with the same whole-document + back-index cross-reference script (or its Phase 1 equivalent) before being trusted — this is now the project's standard rigor bar for this pipeline, not an optional nice-to-have.
opendataloader-pdf (Java-based) and pdfplumber's table extraction remain candidate tools for the table/formula-handling fallback path described in docs/architecture.md; docling's viability is still unresolved pending the environment fix.

13 KiB Raw Blame History