duocthu/docs/adr/0003-pdf-parsing-strategy.md

# ADR 0003: PDF parsing strategy for the drug formulary — validated empirically

## Status

Accepted (validated against the real 1668-page source PDF, not assumptions)

## Context

The original scaffold's ingestion design (see `docs/architecture.md` history)
assumed generic best practices for structured-PDF parsing: prefer the PDF's
bookmark/outline (`doc.get_toc()`) for section boundaries, fall back to
font-size heuristics. Before writing real ingestion code, this assumption was
tested against the actual `duoc-thu-quoc-gia-viet-nam-2018.pdf` (1668 pages),
because a 1668-page book has enough real-world irregularity that guessing
from a handful of sample pages is not sufficient grounds to trust a parsing
strategy — every claim below was checked against the whole document or a
independently-sourced ground truth, not a small sample.

## What was actually tested

1. **`doc.get_toc()`**: returns **0 entries**. No usable bookmark/outline.
2. **Tagged-PDF structure tree** (`/StructTreeRoot`): exists, but is shallow
   — ~29 generic `/H1`/`/P` elements, evidently covering only a small
   fraction of the document. Not usable as a structural signal at scale.
   Confirmed dead end.
3. **Cross-tool text-extraction comparison** on the same real pages
   (a known drug monograph, "Abacavir"):
   - **PyMuPDF (`fitz`)**: correct reading order, matches the visual source.
   - **pdfplumber** (`extract_text()`): **incorrect** — scrambles paragraph
     order on this layout and surfaces a stray marked-content artifact
     (`"PB <Header tên thuốc>"`) as if it were visible text. Decision:
     pdfplumber is kept **only** for its `extract_tables()` API (a genuinely
     different, table-specific algorithm), never for general body text.
   - **opendataloader-pdf** (Java-based, benchmarks #1 in public leaderboards
     for reading order/tables): correct reading order, and its own computed
     font metadata (per-span `font`/`font size` in its JSON output)
     **independently agreed** with PyMuPDF's raw span data — two unrelated
     tools agreeing on the same font facts is real cross-validation, not
     opinion. However, its higher-level paragraph/heading classifier is
     **inconsistent**: identical bold section-heading text (e.g. "Dược lý và
     cơ chế tác dụng", "Liều lượng và cách dùng") is sometimes promoted to a
     markdown `##` heading and sometimes silently merged into the following
     body paragraph, for no discernible content-based reason. Conclusion: its
     Markdown/heading output is not reliable enough to be the sole
     structural signal, but it's a useful independent check and its
     header/footer-stripping was notably better than raw PyMuPDF text.
   - **docling**: attempted, blocked by a `numpy`/`pyarrow` ABI conflict in
     the environment (numpy 2.x vs a pyarrow build expecting numpy 1.x,
     pulled in transitively via `torch`/`transformers`). Tested inside an
     isolated venv rather than fixed globally, to avoid destabilizing other
     tools on the machine. See progress log for current status.
4. **The definitive structural signal — bold font spans**: at the raw
   PyMuPDF span level, every section heading and every monograph title is
   rendered in a **bold** font (`"...-BoldMT"`), while body text is not.
   Italic spans exist too (foreign/Latin species names inline) but are
   never confused with headings since they're not bold and appear mid
   sentence. This was cross-confirmed by opendataloader's independently
   computed font metadata for the same spans (see above) — not a
   single-tool guess.
   - **Font size is NOT a reliable discriminator on its own**: a monograph
     title was observed at both 10.0pt ("ABACAVIR") and 9.5pt ("ACARBOSE")
     for equally genuine, equally top-level monograph headings. An earlier
     draft of the detector required `size >= 9.8` based on the first
     example seen and it silently dropped ~15% of real monographs as a
     result — a concrete instance of exactly the "don't generalize from one
     example" risk this investigation was meant to guard against. The fix:
     drop the size floor; use **bold + all-caps + short line length** for
     monograph titles, and **bold** alone (cross-checked against the known
     section-name vocabulary) for section headings.
5. **Ground truth for validation**: the book has **two** indexes:
   - The front-matter "Danh mục các chuyên luận thuốc" (pages 12-31,
     0-indexed): an alphabetical name list with **no page numbers** — useful
     only for a name-overlap sanity check, not page-level validation.
   - The back-of-book "Mục lục tra cứu" (from page ~1529 printed / ~1528
     0-indexed onward): a proper index with **exact page numbers** per
     generic-name entry (e.g. `"Abacavir, 101"`), plus brand-name
     cross-references (`"Ziagen - Abacavir, 101"`, skipped for ground truth).
     This is the real, page-verifiable ground truth and should be used for
     any future re-validation, not the front-matter list.
   - The front matter's own "NỘI DUNG" (table of contents, page 7 0-indexed)
     also gives exact page ranges for the book's 3 parts: general topic
     chapters (37-98 printed), individual drug monographs (**99-1496
     printed**), appendices (1497-1528), back index (1529+). Any monograph-
     boundary detector should be scoped to the 99-1496 printed page range —
     scanning the whole book without this scope produces false positives
     from front-matter/general-chapter bold-caps lines (org names, decree
     headers, chapter titles) that are not drug monographs.

## Decision

- **PyMuPDF is the primary and only general-text extractor.** No TOC
  dependency, no reliance on the structure tree.
- **Section/monograph boundary detection uses bold-font spans** (not font
  size, not font size + vocabulary alone), scoped to the printed page range
  of the actual monograph section (99-1496), with all-caps + short length as
  the additional signal narrowing bold spans down to monograph titles
  specifically. Multi-line wrapped titles must be merged before matching.
- **pdfplumber is retained only for table extraction** (`extract_tables()`),
  never general reading order, per the confirmed scrambling issue.
- **The back-of-book "Mục lục tra cứu" is the ground truth for validation**,
  not the front-matter drug list.
- **Validation is a repeatable, whole-document, automated check**, not a
  one-time manual read of a handful of pages: a full 1668-page scan runs in
  under a minute, so re-running it after every heuristic change is cheap and
  should be standard practice before trusting a change.

## Validation results (most recent full-document run)

- Page-verified recall against the back-of-book index: **91.7%** (665/725
  primary entries had a detected boundary within ±2 physical pages of the
  expected page).
- Remaining misses are overwhelmingly one identified, fixable cause:
  **multi-line wrapped ALL-CAPS titles** (long Vietnamese drug/vaccine names
  spanning 2+ physical lines) being matched as fragments rather than merged
  — not a failure of the bold-span signal itself. A handful of misses are
  ground-truth extraction noise (the back-index parser occasionally picks up
  a non-drug appendix/table-of-contents line that happens to match the
  `"Name, ###"` pattern) rather than real detector failures.
- Expected recall after fixing multi-line merging and cleaning non-drug
  entries out of the ground truth: materially higher than 91.7%, to be
  re-measured once that fix lands (Phase 1 implementation, not this ADR).

## Follow-up validation: duplicates and cross-page/column data-loss risk

Two further questions were raised and empirically tested against the full
1405-page monograph range (99-1496 printed):

1. **Are any drugs detected twice (real content duplication)?** Scanned for
   normalized-name collisions at physically distant pages. Found exactly
   **one** candidate: `"GONADOTROPIN"` at physical pages 755 and 1371. On
   inspection, this is **not** a real duplicate — page 755 is the genuine
   "GONADOTROPIN" monograph (hCG/menotropin/follitropin), while page 1371 is
   a different monograph, "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN"
   (GnRH-analog drugs), whose title wraps across two lines — the detector
   matched only the second line ("GONADOTROPIN"), colliding with the
   unrelated monograph's normalized key. This is the **same multi-line
   title-wrapping bug** already identified above, now confirmed with a
   second concrete example, not a new failure mode. **Conclusion: no real
   duplicate monographs found in the corpus**; the multi-line merge fix
   (already required for Phase 1) also resolves this collision.

2. **Is the PDF two-column, and can content be lost/corrupted across a page
   or column boundary during chunking?** Confirmed via bounding-box
   inspection: this document **is** genuinely two-column (left column
   x≈44-299, right column x≈308-562, same page). PyMuPDF's block-level
   reading order correctly sequences left-column-then-right-column content
   (already implicitly validated by the correct Abacavir sample earlier).
   However, a **separate, real defect** was found and confirmed: on physical
   page 1373, one short run of text has its glyphs in **reversed
   (right-to-left) x-order**, producing scrambled output — e.g. `" = tịx 8
   yàgn gnàh uềil gnổt( uềihc iổub oàv )magorcim 008 = tịx 4( "`, which
   reverses character-by-character back to the correct
   `"(4 xịt = 800 microgam) vào buổi chiều (tổng liều hàng ngày 8 xịt = ..."`.
   This looks like an isolated PDF-authoring artifact (e.g. an accidental
   RTL/BiDi override on one small span during editing), not a systemic
   extraction bug. **Initially this scan was scoped to the monograph range
   only (1405 of 1668 pages) — an oversight, caught and corrected**:
   re-run across all 1668 pages (front matter, general chapters,
   monographs, appendices, back index — the entire book, page 0 to the
   last page), it still found **exactly 1 affected row, on the same page
   1373, and no others** — confirming the defect is genuinely isolated, not
   hiding somewhere in the ~260 pages outside the original scan scope.
   - The same full-book pass also checked for near-empty pages (<20 chars
     extracted): found exactly **6** — physical pages 3, 37, 99, 1495, 1497,
     1666 — every one lands exactly at a major section boundary (before
     "Các chuyên luận chung" at 37, before "Các chuyên luận thuốc" at 99,
     before "Các phụ lục" at 1497, near the book's end at 1666). These are
     intentional print-layout blank/separator pages, not lost content —
     standard practice to force a new part to start on a fresh page.

**Implications for Phase 1 implementation:**
- Build the pipeline as one **continuous cross-page stream** (text + page
  number + bbox per fragment, in reading order), not per-page-isolated
  chunks — this is required both for correctly merging multi-line
  monograph/section titles (see above) and for never truncating a
  paragraph/sentence that spans a page or column break.
- Add an automated **glyph-order sanity check** as a mandatory pass over
  100% of pages (not sampled): group text fragments into visual rows by
  y-coordinate, verify x-coordinates are non-decreasing, and either
  auto-correct (re-sort by x — the fix is deterministic since raw glyph
  positions are known) or flag for manual QA. This check is cheap
  (~16 seconds over the full monograph range) and should run before every
  real ingestion, not just once.
- The book's content must ultimately be captured **from page 0 to the last
  page** — but not all of it as drug-monograph chunks: front matter (pages
  0-36) is mostly low-value organizational/decree content and can be
  largely skipped for RAG purposes; general topic chapters (37-98) and
  appendices (1497-1528) are real, valuable content that must be ingested
  too, using their own heading-hierarchy-based chunking (not the drug
  template) — this was already noted in `docs/architecture.md`'s original
  design and is reaffirmed here, not changed. The back-of-book index
  (1529+) does not need its own chunks (it's a page-locator, not content)
  but remains the validation ground truth.

## Consequences

- The real ingestion pipeline (Phase 1) should implement the bold-span
  detector directly (reusing the validated logic, not the exploratory
  scratch scripts), scoped to the correct page range, with multi-line
  heading merging as a required fix before first real ingestion run.
- Every future change to the segmentation heuristic should be re-validated
  with the same whole-document + back-index cross-reference script (or its
  Phase 1 equivalent) before being trusted — this is now the project's
  standard rigor bar for this pipeline, not an optional nice-to-have.
- `opendataloader-pdf` (Java-based) and `pdfplumber`'s table extraction
  remain candidate tools for the table/formula-handling fallback path
  described in `docs/architecture.md`; docling's viability is still
  unresolved pending the environment fix.