Add comprehensive, reusable PDF-parsing outlier catalog

docs/pdf-parsing-outlier-catalog.md

@@ -0,0 +1,294 @@
+# PDF Parsing Outlier Catalog
+
+A generalized checklist of structural risks found while parsing
+`duoc-thu-quoc-gia-viet-nam-2018.pdf` (1668 pages). Every item here was
+**confirmed with real evidence** (bounding-box inspection, cross-tool
+comparison, or a whole-document scan) — not assumed. The goal of this
+document is reuse: if this project (or a future one) needs to parse another
+structured reference PDF — another national formulary, a different
+government-published multi-part document, any dense print-layout book —
+this is the checklist of "things that go wrong that a small page sample
+won't reveal," and how to actually check for each one cheaply (most checks
+here run over the whole 1668-page book in under a minute).
+
+For the narrative investigation and drug-formulary-specific numbers, see
+`docs/adr/0003-pdf-parsing-strategy.md`. This document is the distilled,
+reusable checklist form of the same findings, plus items found afterward.
+
+---
+
+## Structural discovery risks (before you even parse content)
+
+### 1. No bookmarks/TOC
+**What it looks like:** `doc.get_toc()` (PyMuPDF) returns an empty list.
+**Why it matters:** the obvious, easiest structural signal for section
+boundaries simply doesn't exist — don't design a pipeline that assumes it
+will.
+**Check:** one line, `len(doc.get_toc())`. Do this first, always, before
+assuming a bookmark-based approach.
+**Generalizes:** yes, directly — always check this before designing around
+bookmarks, for any PDF.
+
+### 2. Shallow/unusable tagged-PDF structure tree
+**What it looks like:** the PDF has a `/StructTreeRoot` (looks promising —
+"tagged PDF"), but it only covers a handful of generic `/H1`/`/P` elements
+for a fraction of the document (here: ~29 elements for 1668 pages).
+**Why it matters:** easy to assume "tagged PDF = rich semantic structure
+available"; in practice many tagging tools produce a minimal
+compliance-only tree that covers almost nothing.
+**Check:** walk the struct tree (`doc.xref_object` on `/StructTreeRoot`,
+recurse into `/K`) and count real leaf elements vs. total page count. If the
+ratio is tiny, it's not a usable data source.
+**Generalizes:** yes — always verify depth/coverage before trusting a
+struct tree, don't just check for its existence.
+
+---
+
+## Page layout risks
+
+### 3. Multi-column body layout
+**What it looks like:** body pages are genuinely two-column (confirmed via
+bounding boxes: left column x≈44-299, right column x≈308-562, page width
+≈595). Front-matter pages that *look* like a multi-column name grid to the
+eye turned out, on inspection, to be single wide text blocks with internal
+whitespace padding between names — not a real structural column split.
+**Why it matters:** a naive "read text top-to-bottom regardless of x" pass
+would interleave left- and right-column content into nonsense. Conversely,
+assuming every visually grid-like page is column-split leads to wasted
+effort — verify per page/section, don't generalize from appearance alone.
+**Check:** for any suspicious page, dump block bounding boxes
+(`page.get_text("dict")["blocks"]`) and look at the actual x0/x1 ranges. A
+real column split shows two clusters of x-ranges; a padded single-column
+list shows one wide range per line.
+**Handling:** PyMuPDF's default block-level reading order handled the real
+two-column case correctly here (validated against a known monograph) — the
+tool most likely to get column order wrong was `pdfplumber`'s general
+`extract_text()` (see item 8), not PyMuPDF.
+**Generalizes:** yes — this exact check (dump bboxes, look at x-clusters)
+works on any PDF to determine real column count before writing extraction
+logic.
+
+### 4. Full-width content breaking out of the column grid
+**What it looks like:** some pages have a table (or could have a figure)
+that spans nearly the entire page width (confirmed: a body-surface-area
+lookup table's blocks span x≈35 to x≈553, i.e. across both normal columns),
+overriding the page's usual two-column layout.
+**Why it matters:** logic written to always split a page into "left column"
+and "right column" text will misbehave on these pages — the content isn't
+in either column, it's a single full-width unit.
+**Check:** for any block, compare its x-width against the known
+single-column width; if a block's x-range spans (or nearly spans) both
+known column ranges, treat it as a full-width unit, not part of a column.
+**Generalizes:** yes — any multi-column layout can have occasional
+full-width breakout elements (tables, figures, pull-quotes); always check
+for this rather than assuming rigid column adherence everywhere.
+
+---
+
+## Table-specific risks
+
+### 5. Tables split across a page break lose their header on the continuation page
+**What it looks like:** confirmed directly — "Bảng 4: Xử trí về điều trị ARV
+theo mức độ phát ban" (a 3-column table) starts on one page with its header
+row (`['Mức độ', 'Biểu hiện', 'Xử trí']`) and 3 data rows; its 4th data row
+("Mức độ 4...") appears on the **next page**, extracted by `pdfplumber`
+as a **separate table object with no header row at all**.
+**Why it matters:** if a pipeline treats each `find_tables()`/
+`extract_tables()` result as an independent, self-contained table, the
+orphaned continuation row is meaningless on its own — you lose the column
+semantics for that row entirely.
+**Check:** for any table-like structure, check whether the page/column
+immediately preceding it ends with a same-shaped table lacking a natural
+final row (e.g. an incomplete-looking sequence) — a strong heuristic is
+"table starts at the very top of a page/column, no header, same column
+count as the table ending at the bottom of the previous page/column."
+**Handling:** never treat page-extracted tables as independent; track
+continuation explicitly and re-attach the original header to orphaned
+continuation rows before using them.
+**Generalizes:** yes — this is a generic multi-page-table risk in any
+paginated PDF with tall tables; the detection heuristic (position at
+page/column top + no header + matching column count to the previous
+table) applies broadly.
+
+### 6. Tables can also split across a column boundary on the *same* page
+**What it looks like:** confirmed — "Bảng 6" (ARV drug toxicity table)
+starts in the left column near the bottom of a page (header + first data
+row) and its remaining data rows appear at the **top of the right column of
+the very same page**, again with no header repeated.
+**Why it matters:** this is easy to miss because there's no literal page
+break — it's tempting to assume "if it's the same page, it's not split,"
+but a table can still be taller than one column's usable height.
+**Check:** same heuristic as item 5, but also check column position, not
+just page number — a header-less table fragment starting at the top of a
+column (regardless of page) is a suspect continuation.
+**Generalizes:** yes, wherever content flows in columns at all — this risk
+exists any time column height is shorter than table height.
+
+### 7. Two-dimensional grid/nomogram tables are not linearly recoverable
+**What it looks like:** confirmed — a body-surface-area lookup table
+(height across the top, weight down the side, a BSA value at each
+intersection) extracts as a scrambled sequence of numbers with no
+recoverable row/column association from plain text alone (e.g. `"0,50
+0,52 0,54 0,56"` followed by `"0,55 0,57 0,59 0,61"` — these are almost
+certainly column-wise fragments, not the visual rows).
+**Why it matters:** unlike a normal bordered table (rows of related
+values), a 2D lookup grid's *meaning* depends entirely on 2D position — a
+number is meaningless without knowing both its row header (weight) and
+column header (height). Flattened text extraction destroys exactly the
+information needed to interpret it.
+**Check:** any table where extracted "cells" are bare numbers with no
+inline label, laid out in a dense grid, is a candidate — cross-check
+against the source's own stated formula/description (this table is
+explicitly a lookup version of a stated formula, see item 8).
+**Handling:** for RAG purposes, prefer **not** to chunk this table as
+literal text at all; either (a) reconstruct it properly using per-number
+bounding-box position matched against header row/column bboxes (real 2D
+table reconstruction, non-trivial), or (b) rely on the accompanying formula
+being available for the LLM to compute from directly, and explicitly flag
+this table's raw text as unreliable/do-not-cite in metadata.
+**Generalizes:** yes — any nomogram, nutrition-fact grid, or nCk-style
+lookup table in any PDF has this exact problem; detect by the "bare number
+grid" pattern, don't assume normal table extraction works.
+
+---
+
+## Formula / equation risks
+
+### 8. Formula rendering is inconsistent — some survive as linear text, some don't
+**What it looks like:** two real formulas found, two different outcomes.
+The Du Bois body-surface-area formula (simple inline exponents,
+`"S = W0,425 × H0,725 × 71,84"`) extracted **cleanly as readable text**. The
+Cockcroft-Gault creatinine-clearance formula (a stacked fraction —
+numerator over denominator, visually 2D) extracted as **scattered,
+disordered fragments** with no linear reading order.
+**Why it matters:** it's tempting to write one rule ("formulas are
+unreliable, always flag them") or its opposite ("formulas extract fine, no
+special handling needed") — neither is true here. The determining factor is
+whether the formula's visual layout is fundamentally 1D (left-to-right,
+like an inline exponent) or 2D (a fraction, a matrix, stacked terms).
+**Check:** no cheap automatic detector was built for this distinction yet —
+treat any equation/formula-like content as a manual-review candidate,
+especially anything with a fraction bar, until a real detector exists
+(e.g. checking for large vertical bbox gaps between adjacent glyphs that
+should be visually stacked).
+**Generalizes:** yes — any technical/medical/scientific PDF with inline
+math will have this exact split; don't assume all formulas behave the same
+way in extraction.
+
+---
+
+## Character/glyph-level risks
+
+### 9. Rare reversed (right-to-left) glyph-order defect
+**What it looks like:** confirmed exactly once across the entire
+1668-page book (physical page 1373): one short text run's glyphs are
+positioned in **descending x-order** rather than ascending, producing
+scrambled output (e.g. `" = tịx 8 yàgn gnàh uềil gnổt(..."`) that reverses
+character-by-character back to the correct Vietnamese sentence
+(`"(4 xịt = 800 microgam) vào buổi chiều..."`).
+**Why it matters:** this is a genuine, confirmed data-corruption risk, not
+theoretical — but it's also extremely rare (1 occurrence in 1668 pages), so
+it must be *detected*, not assumed to be either absent or common.
+**Check:** group text fragments into visual rows by rounded y-coordinate,
+then check whether x-coordinates are non-decreasing across the row; flag
+(and optionally auto-correct by re-sorting on x) any row that isn't. This
+full-book check runs in about 20 seconds.
+**Generalizes:** yes, directly — this is a cheap, universal sanity check
+worth running on any PDF text-extraction pipeline as a standing QA gate,
+regardless of source document, since it catches a class of PDF-authoring
+defects (RTL/BiDi overrides, corrupted content streams) that have nothing
+to do with this book specifically.
+
+---
+
+## Section/heading detection risks
+
+### 10. Font size is not a reliable heading signal — bold is
+**What it looks like:** confirmed two genuine, equally top-level monograph
+titles at different font sizes (10.0pt and 9.5pt). An early detector
+gated on `size >= 9.8` and silently dropped ~15% of real monographs as a
+result.
+**Why it matters:** a threshold calibrated from one or two examples will
+look correct until validated at scale — this is the single clearest
+"don't generalize from a small sample" lesson from this whole
+investigation.
+**Check:** whole-document validation against an independent ground truth
+(here, the back-of-book page-numbered index) is what caught this — a
+sample of 2-3 pages would not have.
+**Generalizes:** yes — for any PDF, prefer a binary style signal (bold/not
+bold, a specific font name) over a numeric threshold (size, weight value)
+wherever possible, and always validate any numeric threshold against the
+whole document, not a handful of examples.
+
+### 11. Multi-line wrapped titles/headings must be merged before matching
+**What it looks like:** confirmed as the dominant cause of missed
+detections in whole-document validation — long titles (e.g. "CÁC CHẤT ỨC
+CHẾ HMG-CoA REDUCTASE", "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN")
+wrap across 2+ physical lines; a per-line detector catches only fragments,
+which then fail to match a name-based ground truth AND can produce false
+name collisions with an unrelated single-line heading elsewhere in the
+document (this happened: a wrapped title's second line, "GONADOTROPIN",
+collided with a genuine, different, single-line "GONADOTROPIN" monograph
+elsewhere).
+**Check:** whole-document recall measurement against ground truth; misses
+clustered around long/compound names are the signature of this bug.
+**Handling:** merge consecutive bold+all-caps lines (with compatible
+positioning) into one candidate title before matching/keying, rather than
+treating each line independently.
+**Generalizes:** yes — any document with long titles/headings that can wrap
+will have this exact failure mode; always merge candidate multi-line
+headings before using them as unique keys.
+
+### 12. The documented taxonomy is not exhaustive — keep it open
+**What it looks like:** the book explicitly documents a 19-field template
+for every drug monograph (page 38), but real monographs contain at least
+one undocumented extra field ("Tên thương mại" — brand/trade names) not in
+that list.
+**Why it matters:** treating a documented schema as a closed enum will
+silently misclassify or drop real content that doesn't fit it.
+**Generalizes:** yes — any document that describes its own structure in a
+preface/README should still be validated against real instances; documented
+schemas are frequently incomplete in practice.
+
+---
+
+## Noise / boilerplate risks
+
+### 13. Header/footer boilerplate must be stripped, but can double as a signal
+**What it looks like:** every page carries a page number and a repeating
+string (`"DTQGVN 2"`), and body pages additionally carry a running header
+naming the current monograph/section.
+**Handling:** strip the fixed boilerplate before parsing content, but the
+running monograph-name header is a **useful secondary cross-check** for
+"which monograph is this page's body text currently part of" — don't
+discard it as pure noise.
+**Generalizes:** yes — running headers/footers are common in print-derived
+PDFs and are usually worth extracting as metadata, not just filtering out.
+
+### 14. Blank/near-empty separator pages at section transitions are expected
+**What it looks like:** exactly 6 near-empty pages (<20 characters) found
+across the whole 1668-page book, and every single one lands exactly on a
+major section-transition boundary (before general chapters, before
+individual monographs, before appendices, near the book's end).
+**Why it matters:** a naive pipeline might treat a near-empty page as an
+extraction failure and error out or flag it, when it's actually an
+intentional print-layout convention (forcing a new part to start on a
+fresh page).
+**Check:** whole-document scan for pages under some small character
+threshold; cross-reference their positions against known section
+boundaries before treating them as errors.
+**Generalizes:** yes — this print convention is extremely common in
+formally typeset books; always expect and gracefully skip near-empty pages
+rather than treating them as failures.
+
+---
+
+## Not yet investigated (flagged for future work, not silently ignored)
+
+- **Footnote-style superscript reference markers** (seen as `a, b, c, d` in
+  one table) — not yet checked for whether the footnote text stays
+  correctly associated with its marker/row during extraction.
+- **Formula detection heuristic** (item 8) — no automatic detector exists
+  yet to flag 2D-formula regions before they're trusted as chunk content.
+- **2D grid table reconstruction** (item 7) — no implementation yet for
+  recovering row/column-correct values from a nomogram-style table.