duocthu/docs/pdf-parsing-outlier-catalog.md

# PDF Parsing Outlier Catalog

A generalized checklist of structural risks found while parsing
`duoc-thu-quoc-gia-viet-nam-2018.pdf` (1668 pages). Every item here was
**confirmed with real evidence** (bounding-box inspection, cross-tool
comparison, or a whole-document scan) — not assumed. The goal of this
document is reuse: if this project (or a future one) needs to parse another
structured reference PDF — another national formulary, a different
government-published multi-part document, any dense print-layout book —
this is the checklist of "things that go wrong that a small page sample
won't reveal," and how to actually check for each one cheaply (most checks
here run over the whole 1668-page book in under a minute).

For the narrative investigation and drug-formulary-specific numbers, see
`docs/adr/0003-pdf-parsing-strategy.md`. This document is the distilled,
reusable checklist form of the same findings, plus items found afterward.

---

## Structural discovery risks (before you even parse content)

### 1. No bookmarks/TOC
**What it looks like:** `doc.get_toc()` (PyMuPDF) returns an empty list.
**Why it matters:** the obvious, easiest structural signal for section
boundaries simply doesn't exist — don't design a pipeline that assumes it
will.
**Check:** one line, `len(doc.get_toc())`. Do this first, always, before
assuming a bookmark-based approach.
**Generalizes:** yes, directly — always check this before designing around
bookmarks, for any PDF.

### 2. Shallow/unusable tagged-PDF structure tree
**What it looks like:** the PDF has a `/StructTreeRoot` (looks promising —
"tagged PDF"), but it only covers a handful of generic `/H1`/`/P` elements
for a fraction of the document (here: ~29 elements for 1668 pages).
**Why it matters:** easy to assume "tagged PDF = rich semantic structure
available"; in practice many tagging tools produce a minimal
compliance-only tree that covers almost nothing.
**Check:** walk the struct tree (`doc.xref_object` on `/StructTreeRoot`,
recurse into `/K`) and count real leaf elements vs. total page count. If the
ratio is tiny, it's not a usable data source.
**Generalizes:** yes — always verify depth/coverage before trusting a
struct tree, don't just check for its existence.

---

## Page layout risks

### 3. Multi-column body layout
**What it looks like:** body pages are genuinely two-column (confirmed via
bounding boxes: left column x≈44-299, right column x≈308-562, page width
≈595). Front-matter pages that *look* like a multi-column name grid to the
eye turned out, on inspection, to be single wide text blocks with internal
whitespace padding between names — not a real structural column split.
**Why it matters:** a naive "read text top-to-bottom regardless of x" pass
would interleave left- and right-column content into nonsense. Conversely,
assuming every visually grid-like page is column-split leads to wasted
effort — verify per page/section, don't generalize from appearance alone.
**Check:** for any suspicious page, dump block bounding boxes
(`page.get_text("dict")["blocks"]`) and look at the actual x0/x1 ranges. A
real column split shows two clusters of x-ranges; a padded single-column
list shows one wide range per line.
**Handling:** PyMuPDF's default block-level reading order handled the real
two-column case correctly here (validated against a known monograph) — the
tool most likely to get column order wrong was `pdfplumber`'s general
`extract_text()` (see item 8), not PyMuPDF.
**Generalizes:** yes — this exact check (dump bboxes, look at x-clusters)
works on any PDF to determine real column count before writing extraction
logic.

### 4. Full-width content breaking out of the column grid
**What it looks like:** some pages have a table (or could have a figure)
that spans nearly the entire page width (confirmed: a body-surface-area
lookup table's blocks span x≈35 to x≈553, i.e. across both normal columns),
overriding the page's usual two-column layout.
**Why it matters:** logic written to always split a page into "left column"
and "right column" text will misbehave on these pages — the content isn't
in either column, it's a single full-width unit.
**Check:** for any block, compare its x-width against the known
single-column width; if a block's x-range spans (or nearly spans) both
known column ranges, treat it as a full-width unit, not part of a column.
**Generalizes:** yes — any multi-column layout can have occasional
full-width breakout elements (tables, figures, pull-quotes); always check
for this rather than assuming rigid column adherence everywhere.

---

## Table-specific risks

### 5. Tables split across a page break lose their header on the continuation page
**What it looks like:** confirmed directly — "Bảng 4: Xử trí về điều trị ARV
theo mức độ phát ban" (a 3-column table) starts on one page with its header
row (`['Mức độ', 'Biểu hiện', 'Xử trí']`) and 3 data rows; its 4th data row
("Mức độ 4...") appears on the **next page**, extracted by `pdfplumber`
as a **separate table object with no header row at all**.
**Why it matters:** if a pipeline treats each `find_tables()`/
`extract_tables()` result as an independent, self-contained table, the
orphaned continuation row is meaningless on its own — you lose the column
semantics for that row entirely.
**Check:** for any table-like structure, check whether the page/column
immediately preceding it ends with a same-shaped table lacking a natural
final row (e.g. an incomplete-looking sequence) — a strong heuristic is
"table starts at the very top of a page/column, no header, same column
count as the table ending at the bottom of the previous page/column."
**Handling:** never treat page-extracted tables as independent; track
continuation explicitly and re-attach the original header to orphaned
continuation rows before using them.
**Generalizes:** yes — this is a generic multi-page-table risk in any
paginated PDF with tall tables; the detection heuristic (position at
page/column top + no header + matching column count to the previous
table) applies broadly.

### 6. Tables can also split across a column boundary on the *same* page
**What it looks like:** confirmed — "Bảng 6" (ARV drug toxicity table)
starts in the left column near the bottom of a page (header + first data
row) and its remaining data rows appear at the **top of the right column of
the very same page**, again with no header repeated.
**Why it matters:** this is easy to miss because there's no literal page
break — it's tempting to assume "if it's the same page, it's not split,"
but a table can still be taller than one column's usable height.
**Check:** same heuristic as item 5, but also check column position, not
just page number — a header-less table fragment starting at the top of a
column (regardless of page) is a suspect continuation.
**Generalizes:** yes, wherever content flows in columns at all — this risk
exists any time column height is shorter than table height.

### 7. Two-dimensional grid/nomogram tables are not linearly recoverable
**What it looks like:** confirmed — a body-surface-area lookup table
(height across the top, weight down the side, a BSA value at each
intersection) extracts as a scrambled sequence of numbers with no
recoverable row/column association from plain text alone (e.g. `"0,50
0,52 0,54 0,56"` followed by `"0,55 0,57 0,59 0,61"` — these are almost
certainly column-wise fragments, not the visual rows).
**Why it matters:** unlike a normal bordered table (rows of related
values), a 2D lookup grid's *meaning* depends entirely on 2D position — a
number is meaningless without knowing both its row header (weight) and
column header (height). Flattened text extraction destroys exactly the
information needed to interpret it.
**Check:** any table where extracted "cells" are bare numbers with no
inline label, laid out in a dense grid, is a candidate — cross-check
against the source's own stated formula/description (this table is
explicitly a lookup version of a stated formula, see item 8).
**Handling:** for RAG purposes, prefer **not** to chunk this table as
literal text at all; either (a) reconstruct it properly using per-number
bounding-box position matched against header row/column bboxes (real 2D
table reconstruction, non-trivial), or (b) rely on the accompanying formula
being available for the LLM to compute from directly, and explicitly flag
this table's raw text as unreliable/do-not-cite in metadata.
**Generalizes:** yes — any nomogram, nutrition-fact grid, or nCk-style
lookup table in any PDF has this exact problem; detect by the "bare number
grid" pattern, don't assume normal table extraction works.

---

## Formula / equation risks

### 8. Formula rendering is inconsistent — some survive as linear text, some don't
**What it looks like:** two real formulas found, two different outcomes.
The Du Bois body-surface-area formula (simple inline exponents,
`"S = W0,425 × H0,725 × 71,84"`) extracted **cleanly as readable text**. The
Cockcroft-Gault creatinine-clearance formula (a stacked fraction —
numerator over denominator, visually 2D) extracted as **scattered,
disordered fragments** with no linear reading order.
**Why it matters:** it's tempting to write one rule ("formulas are
unreliable, always flag them") or its opposite ("formulas extract fine, no
special handling needed") — neither is true here. The determining factor is
whether the formula's visual layout is fundamentally 1D (left-to-right,
like an inline exponent) or 2D (a fraction, a matrix, stacked terms).
**Check:** no cheap automatic detector was built for this distinction yet —
treat any equation/formula-like content as a manual-review candidate,
especially anything with a fraction bar, until a real detector exists
(e.g. checking for large vertical bbox gaps between adjacent glyphs that
should be visually stacked).
**Generalizes:** yes — any technical/medical/scientific PDF with inline
math will have this exact split; don't assume all formulas behave the same
way in extraction.

---

## Character/glyph-level risks

### 9. Rare reversed (right-to-left) glyph-order defect
**What it looks like:** confirmed exactly once across the entire
1668-page book (physical page 1373): one short text run's glyphs are
positioned in **descending x-order** rather than ascending, producing
scrambled output (e.g. `" = tịx 8 yàgn gnàh uềil gnổt(..."`) that reverses
character-by-character back to the correct Vietnamese sentence
(`"(4 xịt = 800 microgam) vào buổi chiều..."`).
**Why it matters:** this is a genuine, confirmed data-corruption risk, not
theoretical — but it's also extremely rare (1 occurrence in 1668 pages), so
it must be *detected*, not assumed to be either absent or common.
**Check:** group text fragments into visual rows by rounded y-coordinate,
then check whether x-coordinates are non-decreasing across the row; flag
(and optionally auto-correct by re-sorting on x) any row that isn't. This
full-book check runs in about 20 seconds.
**Generalizes:** yes, directly — this is a cheap, universal sanity check
worth running on any PDF text-extraction pipeline as a standing QA gate,
regardless of source document, since it catches a class of PDF-authoring
defects (RTL/BiDi overrides, corrupted content streams) that have nothing
to do with this book specifically.

---

## Section/heading detection risks

### 10. Font size is not a reliable heading signal — bold is
**What it looks like:** confirmed two genuine, equally top-level monograph
titles at different font sizes (10.0pt and 9.5pt). An early detector
gated on `size >= 9.8` and silently dropped ~15% of real monographs as a
result.
**Why it matters:** a threshold calibrated from one or two examples will
look correct until validated at scale — this is the single clearest
"don't generalize from a small sample" lesson from this whole
investigation.
**Check:** whole-document validation against an independent ground truth
(here, the back-of-book page-numbered index) is what caught this — a
sample of 2-3 pages would not have.
**Generalizes:** yes — for any PDF, prefer a binary style signal (bold/not
bold, a specific font name) over a numeric threshold (size, weight value)
wherever possible, and always validate any numeric threshold against the
whole document, not a handful of examples.

### 11. Multi-line wrapped titles/headings must be merged before matching
**What it looks like:** confirmed as the dominant cause of missed
detections in whole-document validation — long titles (e.g. "CÁC CHẤT ỨC
CHẾ HMG-CoA REDUCTASE", "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN")
wrap across 2+ physical lines; a per-line detector catches only fragments,
which then fail to match a name-based ground truth AND can produce false
name collisions with an unrelated single-line heading elsewhere in the
document (this happened: a wrapped title's second line, "GONADOTROPIN",
collided with a genuine, different, single-line "GONADOTROPIN" monograph
elsewhere).
**Check:** whole-document recall measurement against ground truth; misses
clustered around long/compound names are the signature of this bug.
**Handling:** merge consecutive bold+all-caps lines (with compatible
positioning) into one candidate title before matching/keying, rather than
treating each line independently.
**Generalizes:** yes — any document with long titles/headings that can wrap
will have this exact failure mode; always merge candidate multi-line
headings before using them as unique keys.

### 12. The documented taxonomy is not exhaustive — keep it open
**What it looks like:** the book explicitly documents a 19-field template
for every drug monograph (page 38), but real monographs contain at least
one undocumented extra field ("Tên thương mại" — brand/trade names) not in
that list.
**Why it matters:** treating a documented schema as a closed enum will
silently misclassify or drop real content that doesn't fit it.
**Generalizes:** yes — any document that describes its own structure in a
preface/README should still be validated against real instances; documented
schemas are frequently incomplete in practice.

---

## Noise / boilerplate risks

### 13. Header/footer boilerplate must be stripped, but can double as a signal
**What it looks like:** every page carries a page number and a repeating
string (`"DTQGVN 2"`), and body pages additionally carry a running header
naming the current monograph/section.
**Handling:** strip the fixed boilerplate before parsing content, but the
running monograph-name header is a **useful secondary cross-check** for
"which monograph is this page's body text currently part of" — don't
discard it as pure noise.
**Generalizes:** yes — running headers/footers are common in print-derived
PDFs and are usually worth extracting as metadata, not just filtering out.

### 14. Blank/near-empty separator pages at section transitions are expected
**What it looks like:** exactly 6 near-empty pages (<20 characters) found
across the whole 1668-page book, and every single one lands exactly on a
major section-transition boundary (before general chapters, before
individual monographs, before appendices, near the book's end).
**Why it matters:** a naive pipeline might treat a near-empty page as an
extraction failure and error out or flag it, when it's actually an
intentional print-layout convention (forcing a new part to start on a
fresh page).
**Check:** whole-document scan for pages under some small character
threshold; cross-reference their positions against known section
boundaries before treating them as errors.
**Generalizes:** yes — this print convention is extremely common in
formally typeset books; always expect and gracefully skip near-empty pages
rather than treating them as failures.

---

## Not yet investigated (flagged for future work, not silently ignored)

- **Footnote-style superscript reference markers** (seen as `a, b, c, d` in
  one table) — not yet checked for whether the footnote text stays
  correctly associated with its marker/row during extraction.
- **Formula detection heuristic** (item 8) — no automatic detector exists
  yet to flag 2D-formula regions before they're trusted as chunk content.
- **2D grid table reconstruction** (item 7) — no implementation yet for
  recovering row/column-correct values from a nomogram-style table.