# PDF Parsing Outlier Catalog A generalized checklist of structural risks found while parsing `duoc-thu-quoc-gia-viet-nam-2018.pdf` (1668 pages). Every item here was **confirmed with real evidence** (bounding-box inspection, cross-tool comparison, or a whole-document scan) — not assumed. The goal of this document is reuse: if this project (or a future one) needs to parse another structured reference PDF — another national formulary, a different government-published multi-part document, any dense print-layout book — this is the checklist of "things that go wrong that a small page sample won't reveal," and how to actually check for each one cheaply (most checks here run over the whole 1668-page book in under a minute). For the narrative investigation and drug-formulary-specific numbers, see `docs/adr/0003-pdf-parsing-strategy.md`. This document is the distilled, reusable checklist form of the same findings, plus items found afterward. --- ## Structural discovery risks (before you even parse content) ### 1. No bookmarks/TOC **What it looks like:** `doc.get_toc()` (PyMuPDF) returns an empty list. **Why it matters:** the obvious, easiest structural signal for section boundaries simply doesn't exist — don't design a pipeline that assumes it will. **Check:** one line, `len(doc.get_toc())`. Do this first, always, before assuming a bookmark-based approach. **Generalizes:** yes, directly — always check this before designing around bookmarks, for any PDF. ### 2. Shallow/unusable tagged-PDF structure tree **What it looks like:** the PDF has a `/StructTreeRoot` (looks promising — "tagged PDF"), but it only covers a handful of generic `/H1`/`/P` elements for a fraction of the document (here: ~29 elements for 1668 pages). **Why it matters:** easy to assume "tagged PDF = rich semantic structure available"; in practice many tagging tools produce a minimal compliance-only tree that covers almost nothing. **Check:** walk the struct tree (`doc.xref_object` on `/StructTreeRoot`, recurse into `/K`) and count real leaf elements vs. total page count. If the ratio is tiny, it's not a usable data source. **Generalizes:** yes — always verify depth/coverage before trusting a struct tree, don't just check for its existence. --- ## Page layout risks ### 3. Multi-column body layout **What it looks like:** body pages are genuinely two-column (confirmed via bounding boxes: left column x≈44-299, right column x≈308-562, page width ≈595). Front-matter pages that *look* like a multi-column name grid to the eye turned out, on inspection, to be single wide text blocks with internal whitespace padding between names — not a real structural column split. **Why it matters:** a naive "read text top-to-bottom regardless of x" pass would interleave left- and right-column content into nonsense. Conversely, assuming every visually grid-like page is column-split leads to wasted effort — verify per page/section, don't generalize from appearance alone. **Check:** for any suspicious page, dump block bounding boxes (`page.get_text("dict")["blocks"]`) and look at the actual x0/x1 ranges. A real column split shows two clusters of x-ranges; a padded single-column list shows one wide range per line. **Handling:** PyMuPDF's default block-level reading order handled the real two-column case correctly here (validated against a known monograph) — the tool most likely to get column order wrong was `pdfplumber`'s general `extract_text()` (see item 8), not PyMuPDF. **Generalizes:** yes — this exact check (dump bboxes, look at x-clusters) works on any PDF to determine real column count before writing extraction logic. ### 4. Full-width content breaking out of the column grid **What it looks like:** some pages have a table (or could have a figure) that spans nearly the entire page width (confirmed: a body-surface-area lookup table's blocks span x≈35 to x≈553, i.e. across both normal columns), overriding the page's usual two-column layout. **Why it matters:** logic written to always split a page into "left column" and "right column" text will misbehave on these pages — the content isn't in either column, it's a single full-width unit. **Check:** for any block, compare its x-width against the known single-column width; if a block's x-range spans (or nearly spans) both known column ranges, treat it as a full-width unit, not part of a column. **Generalizes:** yes — any multi-column layout can have occasional full-width breakout elements (tables, figures, pull-quotes); always check for this rather than assuming rigid column adherence everywhere. --- ## Table-specific risks ### 5. Tables split across a page break lose their header on the continuation page **What it looks like:** confirmed directly — "Bảng 4: Xử trí về điều trị ARV theo mức độ phát ban" (a 3-column table) starts on one page with its header row (`['Mức độ', 'Biểu hiện', 'Xử trí']`) and 3 data rows; its 4th data row ("Mức độ 4...") appears on the **next page**, extracted by `pdfplumber` as a **separate table object with no header row at all**. **Why it matters:** if a pipeline treats each `find_tables()`/ `extract_tables()` result as an independent, self-contained table, the orphaned continuation row is meaningless on its own — you lose the column semantics for that row entirely. **Check:** for any table-like structure, check whether the page/column immediately preceding it ends with a same-shaped table lacking a natural final row (e.g. an incomplete-looking sequence) — a strong heuristic is "table starts at the very top of a page/column, no header, same column count as the table ending at the bottom of the previous page/column." **Handling:** never treat page-extracted tables as independent; track continuation explicitly and re-attach the original header to orphaned continuation rows before using them. **Generalizes:** yes — this is a generic multi-page-table risk in any paginated PDF with tall tables; the detection heuristic (position at page/column top + no header + matching column count to the previous table) applies broadly. ### 6. Tables can also split across a column boundary on the *same* page **What it looks like:** confirmed — "Bảng 6" (ARV drug toxicity table) starts in the left column near the bottom of a page (header + first data row) and its remaining data rows appear at the **top of the right column of the very same page**, again with no header repeated. **Why it matters:** this is easy to miss because there's no literal page break — it's tempting to assume "if it's the same page, it's not split," but a table can still be taller than one column's usable height. **Check:** same heuristic as item 5, but also check column position, not just page number — a header-less table fragment starting at the top of a column (regardless of page) is a suspect continuation. **Generalizes:** yes, wherever content flows in columns at all — this risk exists any time column height is shorter than table height. ### 7. Two-dimensional grid/nomogram tables are not linearly recoverable **What it looks like:** confirmed — a body-surface-area lookup table (height across the top, weight down the side, a BSA value at each intersection) extracts as a scrambled sequence of numbers with no recoverable row/column association from plain text alone (e.g. `"0,50 0,52 0,54 0,56"` followed by `"0,55 0,57 0,59 0,61"` — these are almost certainly column-wise fragments, not the visual rows). **Why it matters:** unlike a normal bordered table (rows of related values), a 2D lookup grid's *meaning* depends entirely on 2D position — a number is meaningless without knowing both its row header (weight) and column header (height). Flattened text extraction destroys exactly the information needed to interpret it. **Check:** any table where extracted "cells" are bare numbers with no inline label, laid out in a dense grid, is a candidate — cross-check against the source's own stated formula/description (this table is explicitly a lookup version of a stated formula, see item 8). **Handling:** for RAG purposes, prefer **not** to chunk this table as literal text at all; either (a) reconstruct it properly using per-number bounding-box position matched against header row/column bboxes (real 2D table reconstruction, non-trivial), or (b) rely on the accompanying formula being available for the LLM to compute from directly, and explicitly flag this table's raw text as unreliable/do-not-cite in metadata. **Generalizes:** yes — any nomogram, nutrition-fact grid, or nCk-style lookup table in any PDF has this exact problem; detect by the "bare number grid" pattern, don't assume normal table extraction works. --- ## Formula / equation risks ### 8. Formula rendering is inconsistent — some survive as linear text, some don't **What it looks like:** two real formulas found, two different outcomes. The Du Bois body-surface-area formula (simple inline exponents, `"S = W0,425 × H0,725 × 71,84"`) extracted **cleanly as readable text**. The Cockcroft-Gault creatinine-clearance formula (a stacked fraction — numerator over denominator, visually 2D) extracted as **scattered, disordered fragments** with no linear reading order. **Why it matters:** it's tempting to write one rule ("formulas are unreliable, always flag them") or its opposite ("formulas extract fine, no special handling needed") — neither is true here. The determining factor is whether the formula's visual layout is fundamentally 1D (left-to-right, like an inline exponent) or 2D (a fraction, a matrix, stacked terms). **Check:** no cheap automatic detector was built for this distinction yet — treat any equation/formula-like content as a manual-review candidate, especially anything with a fraction bar, until a real detector exists (e.g. checking for large vertical bbox gaps between adjacent glyphs that should be visually stacked). **Generalizes:** yes — any technical/medical/scientific PDF with inline math will have this exact split; don't assume all formulas behave the same way in extraction. --- ## Character/glyph-level risks ### 9. Rare reversed (right-to-left) glyph-order defect **What it looks like:** confirmed exactly once across the entire 1668-page book (physical page 1373): one short text run's glyphs are positioned in **descending x-order** rather than ascending, producing scrambled output (e.g. `" = tịx 8 yàgn gnàh uềil gnổt(..."`) that reverses character-by-character back to the correct Vietnamese sentence (`"(4 xịt = 800 microgam) vào buổi chiều..."`). **Why it matters:** this is a genuine, confirmed data-corruption risk, not theoretical — but it's also extremely rare (1 occurrence in 1668 pages), so it must be *detected*, not assumed to be either absent or common. **Check:** group text fragments into visual rows by rounded y-coordinate, then check whether x-coordinates are non-decreasing across the row; flag (and optionally auto-correct by re-sorting on x) any row that isn't. This full-book check runs in about 20 seconds. **Generalizes:** yes, directly — this is a cheap, universal sanity check worth running on any PDF text-extraction pipeline as a standing QA gate, regardless of source document, since it catches a class of PDF-authoring defects (RTL/BiDi overrides, corrupted content streams) that have nothing to do with this book specifically. --- ## Section/heading detection risks ### 10. Font size is not a reliable heading signal — bold is **What it looks like:** confirmed two genuine, equally top-level monograph titles at different font sizes (10.0pt and 9.5pt). An early detector gated on `size >= 9.8` and silently dropped ~15% of real monographs as a result. **Why it matters:** a threshold calibrated from one or two examples will look correct until validated at scale — this is the single clearest "don't generalize from a small sample" lesson from this whole investigation. **Check:** whole-document validation against an independent ground truth (here, the back-of-book page-numbered index) is what caught this — a sample of 2-3 pages would not have. **Generalizes:** yes — for any PDF, prefer a binary style signal (bold/not bold, a specific font name) over a numeric threshold (size, weight value) wherever possible, and always validate any numeric threshold against the whole document, not a handful of examples. ### 11. Multi-line wrapped titles/headings must be merged before matching **What it looks like:** confirmed as the dominant cause of missed detections in whole-document validation — long titles (e.g. "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE", "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN") wrap across 2+ physical lines; a per-line detector catches only fragments, which then fail to match a name-based ground truth AND can produce false name collisions with an unrelated single-line heading elsewhere in the document (this happened: a wrapped title's second line, "GONADOTROPIN", collided with a genuine, different, single-line "GONADOTROPIN" monograph elsewhere). **Check:** whole-document recall measurement against ground truth; misses clustered around long/compound names are the signature of this bug. **Handling:** merge consecutive bold+all-caps lines (with compatible positioning) into one candidate title before matching/keying, rather than treating each line independently. **Generalizes:** yes — any document with long titles/headings that can wrap will have this exact failure mode; always merge candidate multi-line headings before using them as unique keys. ### 12. The documented taxonomy is not exhaustive — keep it open **What it looks like:** the book explicitly documents a 19-field template for every drug monograph (page 38), but real monographs contain at least one undocumented extra field ("Tên thương mại" — brand/trade names) not in that list. **Why it matters:** treating a documented schema as a closed enum will silently misclassify or drop real content that doesn't fit it. **Generalizes:** yes — any document that describes its own structure in a preface/README should still be validated against real instances; documented schemas are frequently incomplete in practice. --- ## Noise / boilerplate risks ### 13. Header/footer boilerplate must be stripped, but can double as a signal **What it looks like:** every page carries a page number and a repeating string (`"DTQGVN 2"`), and body pages additionally carry a running header naming the current monograph/section. **Handling:** strip the fixed boilerplate before parsing content, but the running monograph-name header is a **useful secondary cross-check** for "which monograph is this page's body text currently part of" — don't discard it as pure noise. **Generalizes:** yes — running headers/footers are common in print-derived PDFs and are usually worth extracting as metadata, not just filtering out. ### 14. Blank/near-empty separator pages at section transitions are expected **What it looks like:** exactly 6 near-empty pages (<20 characters) found across the whole 1668-page book, and every single one lands exactly on a major section-transition boundary (before general chapters, before individual monographs, before appendices, near the book's end). **Why it matters:** a naive pipeline might treat a near-empty page as an extraction failure and error out or flag it, when it's actually an intentional print-layout convention (forcing a new part to start on a fresh page). **Check:** whole-document scan for pages under some small character threshold; cross-reference their positions against known section boundaries before treating them as errors. **Generalizes:** yes — this print convention is extremely common in formally typeset books; always expect and gracefully skip near-empty pages rather than treating them as failures. --- ## Not yet investigated (flagged for future work, not silently ignored) - **Footnote-style superscript reference markers** (seen as `a, b, c, d` in one table) — not yet checked for whether the footnote text stays correctly associated with its marker/row during extraction. - **Formula detection heuristic** (item 8) — no automatic detector exists yet to flag 2D-formula regions before they're trusted as chunk content. - **2D grid table reconstruction** (item 7) — no implementation yet for recovering row/column-correct values from a nomogram-style table.