duocthu/docs-legacy/pdf-parsing-outlier-catalog.md

# PDF Parsing Outlier Catalog

A generalized checklist of structural risks found while parsing
`duoc-thu-quoc-gia-viet-nam-2018.pdf` (1668 pages). Every item here was
**confirmed with real evidence** (bounding-box inspection, cross-tool
comparison, or a whole-document scan) — not assumed. The goal of this
document is reuse: if this project (or a future one) needs to parse another
structured reference PDF — another national formulary, a different
government-published multi-part document, any dense print-layout book —
this is the checklist of "things that go wrong that a small page sample
won't reveal," and how to actually check for each one cheaply (most checks
here run over the whole 1668-page book in under a minute).

For the narrative investigation and drug-formulary-specific numbers, see
`docs/adr/0003-pdf-parsing-strategy.md`. This document is the distilled,
reusable checklist form of the same findings, plus items found afterward.

---

## Structural discovery risks (before you even parse content)

### 1. No bookmarks/TOC
**What it looks like:** `doc.get_toc()` (PyMuPDF) returns an empty list.
**Why it matters:** the obvious, easiest structural signal for section
boundaries simply doesn't exist — don't design a pipeline that assumes it
will.
**Check:** one line, `len(doc.get_toc())`. Do this first, always, before
assuming a bookmark-based approach.
**Generalizes:** yes, directly — always check this before designing around
bookmarks, for any PDF.

### 2. Shallow/unusable tagged-PDF structure tree
**What it looks like:** the PDF has a `/StructTreeRoot` (looks promising —
"tagged PDF"), but it only covers a handful of generic `/H1`/`/P` elements
for a fraction of the document (here: ~29 elements for 1668 pages).
**Why it matters:** easy to assume "tagged PDF = rich semantic structure
available"; in practice many tagging tools produce a minimal
compliance-only tree that covers almost nothing.
**Check:** walk the struct tree (`doc.xref_object` on `/StructTreeRoot`,
recurse into `/K`) and count real leaf elements vs. total page count. If the
ratio is tiny, it's not a usable data source.
**Generalizes:** yes — always verify depth/coverage before trusting a
struct tree, don't just check for its existence.

---

## Page layout risks

### 3. Multi-column body layout
**What it looks like:** body pages are genuinely two-column (confirmed via
bounding boxes: left column x≈44-299, right column x≈308-562, page width
≈595). Front-matter pages that *look* like a multi-column name grid to the
eye turned out, on inspection, to be single wide text blocks with internal
whitespace padding between names — not a real structural column split.
**Why it matters:** a naive "read text top-to-bottom regardless of x" pass
would interleave left- and right-column content into nonsense. Conversely,
assuming every visually grid-like page is column-split leads to wasted
effort — verify per page/section, don't generalize from appearance alone.
**Check:** for any suspicious page, dump block bounding boxes
(`page.get_text("dict")["blocks"]`) and look at the actual x0/x1 ranges. A
real column split shows two clusters of x-ranges; a padded single-column
list shows one wide range per line.
**Handling:** PyMuPDF's default block-level reading order handled the real
two-column case correctly here (validated against a known monograph) — the
tool most likely to get column order wrong was `pdfplumber`'s general
`extract_text()` (see item 8), not PyMuPDF.
**Generalizes:** yes — this exact check (dump bboxes, look at x-clusters)
works on any PDF to determine real column count before writing extraction
logic.

### 4. Full-width content breaking out of the column grid
**What it looks like:** some pages have a table (or could have a figure)
that spans nearly the entire page width (confirmed: a body-surface-area
lookup table's blocks span x≈35 to x≈553, i.e. across both normal columns),
overriding the page's usual two-column layout.
**Why it matters:** logic written to always split a page into "left column"
and "right column" text will misbehave on these pages — the content isn't
in either column, it's a single full-width unit.
**Check:** for any block, compare its x-width against the known
single-column width; if a block's x-range spans (or nearly spans) both
known column ranges, treat it as a full-width unit, not part of a column.
**Generalizes:** yes — any multi-column layout can have occasional
full-width breakout elements (tables, figures, pull-quotes); always check
for this rather than assuming rigid column adherence everywhere.

---

## Table-specific risks

### 5. Tables split across a page break lose their header on the continuation page
**What it looks like:** confirmed directly — "Bảng 4: Xử trí về điều trị ARV
theo mức độ phát ban" (a 3-column table) starts on one page with its header
row (`['Mức độ', 'Biểu hiện', 'Xử trí']`) and 3 data rows; its 4th data row
("Mức độ 4...") appears on the **next page**, extracted by `pdfplumber`
as a **separate table object with no header row at all**.
**Why it matters:** if a pipeline treats each `find_tables()`/
`extract_tables()` result as an independent, self-contained table, the
orphaned continuation row is meaningless on its own — you lose the column
semantics for that row entirely.
**Check:** for any table-like structure, check whether the page/column
immediately preceding it ends with a same-shaped table lacking a natural
final row (e.g. an incomplete-looking sequence) — a strong heuristic is
"table starts at the very top of a page/column, no header, same column
count as the table ending at the bottom of the previous page/column."
**Handling:** never treat page-extracted tables as independent; track
continuation explicitly and re-attach the original header to orphaned
continuation rows before using them.
**Generalizes:** yes — this is a generic multi-page-table risk in any
paginated PDF with tall tables; the detection heuristic (position at
page/column top + no header + matching column count to the previous
table) applies broadly.

### 6. Tables can also split across a column boundary on the *same* page
**What it looks like:** confirmed — "Bảng 6" (ARV drug toxicity table)
starts in the left column near the bottom of a page (header + first data
row) and its remaining data rows appear at the **top of the right column of
the very same page**, again with no header repeated.
**Why it matters:** this is easy to miss because there's no literal page
break — it's tempting to assume "if it's the same page, it's not split,"
but a table can still be taller than one column's usable height.
**Check:** same heuristic as item 5, but also check column position, not
just page number — a header-less table fragment starting at the top of a
column (regardless of page) is a suspect continuation.
**Generalizes:** yes, wherever content flows in columns at all — this risk
exists any time column height is shorter than table height.

### 7. Two-dimensional grid/nomogram tables are not linearly recoverable
**What it looks like:** confirmed — a body-surface-area lookup table
(height across the top, weight down the side, a BSA value at each
intersection) extracts as a scrambled sequence of numbers with no
recoverable row/column association from plain text alone (e.g. `"0,50
0,52 0,54 0,56"` followed by `"0,55 0,57 0,59 0,61"` — these are almost
certainly column-wise fragments, not the visual rows).
**Why it matters:** unlike a normal bordered table (rows of related
values), a 2D lookup grid's *meaning* depends entirely on 2D position — a
number is meaningless without knowing both its row header (weight) and
column header (height). Flattened text extraction destroys exactly the
information needed to interpret it.
**Check:** any table where extracted "cells" are bare numbers with no
inline label, laid out in a dense grid, is a candidate — cross-check
against the source's own stated formula/description (this table is
explicitly a lookup version of a stated formula, see item 8).
**Handling:** for RAG purposes, prefer **not** to chunk this table as
literal text at all; either (a) reconstruct it properly using per-number
bounding-box position matched against header row/column bboxes (real 2D
table reconstruction, non-trivial), or (b) rely on the accompanying formula
being available for the LLM to compute from directly, and explicitly flag
this table's raw text as unreliable/do-not-cite in metadata.
**Generalizes:** yes — any nomogram, nutrition-fact grid, or nCk-style
lookup table in any PDF has this exact problem; detect by the "bare number
grid" pattern, don't assume normal table extraction works.

---

## Formula / equation risks

### 8. Formula rendering is inconsistent — some survive as linear text, some don't
**What it looks like:** two real formulas found, two different outcomes.
The Du Bois body-surface-area formula (simple inline exponents,
`"S = W0,425 × H0,725 × 71,84"`) extracted **cleanly as readable text**. The
Cockcroft-Gault creatinine-clearance formula (a stacked fraction —
numerator over denominator, visually 2D) extracted as **scattered,
disordered fragments** with no linear reading order.
**Why it matters:** it's tempting to write one rule ("formulas are
unreliable, always flag them") or its opposite ("formulas extract fine, no
special handling needed") — neither is true here. The determining factor is
whether the formula's visual layout is fundamentally 1D (left-to-right,
like an inline exponent) or 2D (a fraction, a matrix, stacked terms).
**Check:** a detector now exists — `residual_ink.py`'s
`fraction_bar_candidate`, which finds the bar as ink no extracted span
accounts for. Measured on this book: **precision 16/23 = 69.6%** (the misses
are decorative underlines and table borders), recall unknown, and it is blind
by construction to a fraction printed without a bar (item 25). Its output is
therefore a review queue, not a verdict: all 23 candidates were rendered and
read one at a time before any was acted on, and only the confirmed ones went
into `ingestion/data/verified/formula_regions_2d.json`.
**Generalizes:** yes — any technical/medical/scientific PDF with inline
math will have this exact split; don't assume all formulas behave the same
way in extraction.

---

## Character/glyph-level risks

### 9. Rare reversed/misordered glyph defects — corrected count: 2, not 1
**What it looks like:** re-implemented as real, tested production code
(`ingestion/ingestion/extract/glyph_order.py`) rather than trusted from the
earlier exploratory script's claim. Found **two distinct shapes**, not the
one originally reported:
1. **Within-span character reversal** (physical page 1373, the originally
   reported case): one span's glyphs are positioned in descending x-order,
   producing `" = tịx 8 yàgn gnàh uềil gnổt(..."`, which reverses
   character-by-character back to `"(4 xịt = 800 microgam) vào buổi
   chiều..."`.
2. **Cross-fragment row misordering, newly found** (physical page 714): a
   single visual row is split by PyMuPDF into multiple `line` objects
   *within one block* that are then emitted out of left-to-right order —
   each fragment's own characters are fine, but concatenating fragments in
   extraction order produces `"...bảo quản  ộđ tệihn "` instead of the
   correct `"...bảo quản  nhiệt độ "`. This is a different underlying shape
   from item 1 (multiple mis-ordered fragments, not one reversed span) and
   was missed by the original narrower (within-span-only) check — the
   ADR 0003 claim of "exactly 1 occurrence in the whole book" undercounted
   the real defect population; corrected here.
**Getting a trustworthy count took three detector iterations** (documented
in the module's own docstring) — the first naive whole-book implementation
of the row-level check reported **1113** "issues," almost all false
positives from two mechanisms: (a) ordinary font-kerning jitter (e.g. in
"mefloquin," two adjacent glyphs differ by 0.095pt — normal kerning, not a
defect) treated as a reversal with no decrease-tolerance, which then
actively *corrupted* correct text into "mefolquin"; and (b) reconstructing
"visual rows" from raw x/y coordinates using a hand-picked column-boundary
threshold, which misclassifies a paragraph that happens to start near the
natural column gap (confirmed real case: a right-column paragraph starting
at x=299.4 got merged with an unrelated left-column paragraph at the same
y). The fix that survived whole-book testing: group by PyMuPDF's own
`block` index (already validated in ADR 0003 to respect this document's
column structure) instead of re-deriving columns from coordinates, plus a
minimum-decrease threshold (1.0pt — safely between the ~0.3pt kerning noise
floor and the >2pt real-defect magnitude). Final whole-book result: **11
row-level issues on 5 pages** — 3 of those pages (92, 94, 805) are formula
regions already flagged as unreliable in item 8 below (2D-layout formulas
scramble on extraction; this check's "corrected" text for those rows should
**not** be trusted or auto-applied, same as item 8's existing guidance),
leaving exactly the 2 genuine prose defects above (pages 714, 1373).
**Why it matters:** both genuine defects are confirmed real data-corruption
risks, not theoretical — but both are also extremely rare (2 occurrences in
1668 pages of prose), so they must be *detected*, not assumed either absent
or common. Equally important: a naive implementation of "the obvious check"
can itself introduce false positives and even actively corrupt correct
text — this detector's own false-positive history is as important a lesson
as the defects it catches.
**Check:** `ingestion.extract.scan_glyph_order` (within-span) and
`ingestion.extract.scan_reading_order` (cross-fragment, grouped by real
PyMuPDF block index + row y, with a 1.0pt minimum-decrease threshold and
header-band exclusion). Both run in seconds over the full book.
**Generalizes:** yes, directly — this is a cheap, universal sanity check
worth running on any PDF text-extraction pipeline as a standing QA gate,
regardless of source document. The false-positive history also generalizes:
any "reconstruct visual rows from raw coordinates" approach needs a
decrease-tolerance (font kerning is universal) and should prefer the
source tool's own layout-analysis groupings (blocks/lines) over hand-picked
coordinate thresholds wherever available.

---

## Section/heading detection risks

### 10. Font size is not a reliable heading signal — bold is
**What it looks like:** confirmed two genuine, equally top-level monograph
titles at different font sizes (10.0pt and 9.5pt). An early detector
gated on `size >= 9.8` and silently dropped ~15% of real monographs as a
result.
**Why it matters:** a threshold calibrated from one or two examples will
look correct until validated at scale — this is the single clearest
"don't generalize from a small sample" lesson from this whole
investigation.
**Check:** whole-document validation against an independent ground truth
(here, the back-of-book page-numbered index) is what caught this — a
sample of 2-3 pages would not have.
**Generalizes:** yes — for any PDF, prefer a binary style signal (bold/not
bold, a specific font name) over a numeric threshold (size, weight value)
wherever possible, and always validate any numeric threshold against the
whole document, not a handful of examples.

### 11. Multi-line wrapped titles/headings must be merged before matching
**What it looks like:** confirmed as the dominant cause of missed
detections in whole-document validation — long titles (e.g. "CÁC CHẤT ỨC
CHẾ HMG-CoA REDUCTASE", "THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN")
wrap across 2+ physical lines; a per-line detector catches only fragments,
which then fail to match a name-based ground truth AND can produce false
name collisions with an unrelated single-line heading elsewhere in the
document (this happened: a wrapped title's second line, "GONADOTROPIN",
collided with a genuine, different, single-line "GONADOTROPIN" monograph
elsewhere).
**Check:** whole-document recall measurement against ground truth; misses
clustered around long/compound names are the signature of this bug.
**Handling:** merge consecutive bold+all-caps lines (with compatible
positioning) into one candidate title before matching/keying, rather than
treating each line independently.
**Generalizes:** yes — any document with long titles/headings that can wrap
will have this exact failure mode; always merge candidate multi-line
headings before using them as unique keys.

### 12a. Class-level monographs cover multiple active ingredients (multiple ATC codes) — this is NOT rare
**What it looks like:** first noticed via two incidental examples
("GONADOTROPIN", "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"), then actually measured
across the whole 680-monograph corpus (not assumed from the 2 examples —
this distinction matters, see below). **Real, whole-corpus number: 173 of
680 detected monographs (25.4%) have more than one distinct ATC code**,
ranging up to extreme cases — INSULIN alone lists **20** different ATC
codes, BETAMETHASON and DEXAMETHASON 11 each, PREDNISOLON 10,
HYDROCORTISON 9. This is a quarter of the entire corpus, not a couple of
edge cases — the 2 incidental examples badly understated how common this
is, and stating "found 2 examples, pattern confirmed" without the
whole-corpus count would have been exactly the kind of unverified claim
this project's own validation standard now forbids.
**Even the 25.4% is a floor, not the true number** — see item 12c below:
ATC-code text-extraction noise (stray whitespace, O/0 confusion) caused
some genuinely multi-ATC monographs (e.g. "TRIAMCINOLON", 5 codes) to be
undercounted by a naive regex. The true proportion is measurably higher
than 25.4%; re-measure after fixing the regex, don't keep citing 25.4% as
final.
**Why it matters:** a data model that assumes "one monograph = one drug =
one ATC code" is wrong for roughly a quarter or more of the corpus.
**Handling:** store ATC code (and dosage-form sub-entries) as a **list**
per monograph, not a scalar; when chunking, consider whether a
class-level monograph's sections should be tagged with the whole class
name, the specific sub-compound, or both, depending on what the retrieval
use case needs.
**Generalizes:** yes — any reference work organized primarily by drug
class or by generic substance will have entries that don't map 1:1 to a
single identifier. More importantly, the *methodology* generalizes: when
you notice a pattern from 1-2 examples, measure its real prevalence across
the whole corpus before deciding how much engineering effort it deserves —
"found 2 examples" and "25.4% of everything" call for very different
levels of investment, and you can't tell which one you're dealing with
without the whole-corpus count.

### 12c. ATC codes (and likely other structured codes) have real text-extraction noise
**What it looks like:** while investigating why 22/680 (3.2%) monographs
appeared to have zero ATC codes, spot-checked 14 of them directly and found
**two distinct, confirmed causes**, both text-extraction noise rather than
missing content:
- **Stray internal whitespace** splitting one code into two tokens, e.g.
  `"L01X X02"` (should be `L01XX02`), `"J04A C01"` (should be `J04AC01`),
  `"N05B A06"` (should be `N05BA06`).
- **Digit/letter confusion**: a literal "0" rendered/typeset as the letter
  "O", e.g. `"NO3AX12"` (should be `N03AX12`), `"JO1DC07"` (should be
  `J01DC07`).
A relaxed regex tolerating both patterns resolved **9 of the 14** spot-checked
cases as real ATC codes hiding behind extraction noise. The **remaining
~5 of 14** were genuinely different: the source text explicitly states
`"Mã ATC: Chưa có."` or `"Mã ATC: Không có."` ("not yet available" / "none")
— a real, valid data state, not an error, and not something to paper over
as if a code exists.
**Why it matters:** a strict ATC-code regex silently undercounts real ATC
data; distinguishing "extraction noise hiding a real code" from "the book
says there is no code" requires checking the actual field text, not just
whether a regex matched.
**Handling:** normalize ATC-code-shaped text before matching (strip internal
whitespace between the letter/digit groups, treat a digit-position "O" as
"0") and explicitly check for the "Chưa có"/"Không có" literal strings as a
valid "no ATC" state rather than a parse failure.
**Generalizes:** yes — any structured code/identifier extracted from a PDF
(product codes, classification codes, reference numbers) can suffer this
same whitespace-injection and O/0 confusion; validate structured-looking
fields against their expected format and investigate exceptions rather than
assuming a strict pattern match is reliable.

### 12d. A section-title (part-divider) page can be falsely detected as a monograph
**What it looks like:** confirmed — the very first item in a whole-corpus
boundary scan was "CÁC CHUYÊN LUẬN THUỐC" (the literal title of Part 2 of
the book, "The Drug Monographs" — a part-divider heading, not a drug) at
physical page 98, picked up as a false-positive monograph boundary because
it happened to be bold, all-caps, short, and was followed (a few real
monograph-boundaries later) by some "Tên chung quốc tế" text from the
actual first real monograph.
**Why it matters:** without a whole-corpus scan this would have gone
unnoticed indefinitely — it doesn't look wrong from a single-page read of
Abacavir, and the discovery methodology this catalog is built on is
exhaustive scans, so this is a good example of a defect that only surfaces
at full scale.
**Handling:** exclude a small, known set of non-drug part/section-divider
strings ("CÁC CHUYÊN LUẬN THUỐC", "CÁC CHUYÊN LUẬN CHUNG", "CÁC PHỤ LỤC",
etc. — enumerable from the book's own table of contents) from the
monograph-boundary detector, or require the anchor phrase ("Tên chung quốc
tế") within a tighter line-distance so an unrelated real monograph several
lines away doesn't false-confirm a divider title.
**Generalizes:** yes — any document with part/section-divider title pages
styled similarly to its content headings (bold, prominent, short) risks
this exact false positive; explicitly exclude known structural/navigational
titles from content-boundary detectors.

### 12b. Genuine spelling/capitalization typos exist in the source text
**What it looks like:** confirmed real example — the running header on the
Vitamin D monograph's continuation pages reads `"Vitamin d và các thuốc
tương tự"` (lowercase "d"), while the real ALL-CAPS heading correctly reads
`"VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"`. This is a genuine typesetting mistake
in the 2018 print, confirmed via font/bbox inspection (same bold 10pt font
as the correct heading — not an extraction artifact, the source text itself
has the typo). The page's bottom running *footer* uses yet another variant,
the short form `"Vitamin D"` (correctly capitalized) — meaning the same
monograph has **three different boilerplate text variants** across one
page (top header with a typo, the real heading, bottom footer).
**Why it matters:** don't treat running headers/footers as a perfectly
clean, typo-free secondary signal (item 13 in this catalog already
recommends using them as a cross-check) — they can themselves contain
source-level errors. In this specific case, the detection heuristic
(strict ALL-CAPS requirement, item 10) happened to still work correctly,
because "Vitamin d và các thuốc tương tự" and "Vitamin D" are not fully
uppercase and so are correctly rejected as monograph-boundary candidates —
but this was not a designed defense against typos specifically, just a
side effect of the all-caps requirement. A future/different typo (e.g. an
accidentally all-caps running header) would not be caught the same way.
**Check:** no systematic typo-detection was built (out of scope — this is
about parsing robustness, not proofreading the source); the practical
takeaway is to keep relying on the strict structural signals (bold + all
caps + short + anchor phrase) as primary, and treat any single text-based
signal (including running headers) as fallible.
**Generalizes:** yes — any real-world print-to-PDF source will have some
rate of genuine typos/inconsistencies; parsing logic should be robust to
them by relying on multiple independent structural signals (font,
position, anchor phrases) rather than trusting any single text match to be
error-free.

### 12e. Monograph length and section coverage vary enormously — measured, not assumed
**What it looks like:** across all 680 detected monographs, length ranges
from **2,331 to 45,623 characters** (~20x spread) and the number of known
section labels found per monograph ranges from as few as **8** up to
**20** (out of a ~19-20 item known vocabulary) — most cluster around
16-19, but the tails are real: "ASPARAGINASE"-adjacent short entries around
2,300-4,300 chars vs. "AMOXICILIN VÀ KALI CLAVULANAT" at 45,623 chars.
**Why it matters:** don't design chunking limits (e.g. a fixed max tokens
per monograph, or an assumption that "a monograph roughly fits in N
chunks") around a single example — the real distribution has a long tail
on both ends.
**Check:** this came from the same whole-corpus survey used for items 12a
and 12c — computing length and detected-section-count per monograph is
cheap and worth keeping as a standing sanity metric (e.g. flag any
monograph outside some percentile range for manual review).
**Generalizes:** yes — any corpus of "similar" documents (monographs,
product entries, articles) will have a real length/completeness
distribution; measure it before assuming uniformity.

### 12. The documented taxonomy is not exhaustive — keep it open
**What it looks like:** the book explicitly documents a 19-field template
for every drug monograph (page 38), but real monographs contain at least
one undocumented extra field ("Tên thương mại" — brand/trade names) not in
that list.
**Why it matters:** treating a documented schema as a closed enum will
silently misclassify or drop real content that doesn't fit it.
**Generalizes:** yes — any document that describes its own structure in a
preface/README should still be validated against real instances; documented
schemas are frequently incomplete in practice.

---

## Noise / boilerplate risks

### 13. Header/footer boilerplate must be stripped, but can double as a signal
**What it looks like:** every page carries a page number and a repeating
string (`"DTQGVN 2"`), and body pages additionally carry a running header
naming the current monograph/section.
**Handling:** strip the fixed boilerplate before parsing content, but the
running monograph-name header is a **useful secondary cross-check** for
"which monograph is this page's body text currently part of" — don't
discard it as pure noise.
**Generalizes:** yes — running headers/footers are common in print-derived
PDFs and are usually worth extracting as metadata, not just filtering out.

### 14. Blank/near-empty separator pages at section transitions are expected
**What it looks like:** exactly 6 near-empty pages (<20 characters) found
across the whole 1668-page book, and every single one lands exactly on a
major section-transition boundary (before general chapters, before
individual monographs, before appendices, near the book's end).
**Why it matters:** a naive pipeline might treat a near-empty page as an
extraction failure and error out or flag it, when it's actually an
intentional print-layout convention (forcing a new part to start on a
fresh page).
**Check:** whole-document scan for pages under some small character
threshold; cross-reference their positions against known section
boundaries before treating them as errors.
**Generalizes:** yes — this print convention is extremely common in
formally typeset books; always expect and gracefully skip near-empty pages
rather than treating them as failures.

---

### 15. No embedded images anywhere in the book — measured, not assumed
**What it looks like:** a whole-book scan of `page.get_images(full=True)` across
all 1668 pages returns **zero** embedded raster/vector images, confirmed via
PyMuPDF's own image extraction API (not just "the text doesn't mention an
image").
**Why it matters:** avoids over-investing in image/caption validation tooling
for a corpus that has no images to validate — but this must be a measured
fact, not an assumption from the book's general description as "text-heavy."
**Generalizes:** the check (`get_images(full=True)` summed over every page)
is a cheap one-line whole-document verification worth running on any PDF
before deciding whether image-handling code is needed at all.

### 16. Chemical reaction arrows render as Private-Use-Area glyphs, not Unicode arrows
**What it looks like:** confirmed real example — physical page 1033 contains a
genuine chemical reaction equation (`Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3`, part of
the cyanide-antidote/rhodanese mechanism description). The reaction arrow
extracts as a Private-Use-Area codepoint (``), not a standard Unicode
arrow (`→`) — the source PDF's font maps a custom symbol glyph (likely from a
symbol/wingdings-style embedded font) into a PUA slot, and raw text extraction
faithfully returns that codepoint rather than a human-readable arrow.
**Why it matters:** any pipeline that treats extracted text as directly
human-readable/citable will surface a mangled or invisible character where a
reaction arrow should be; a naive keyword/embedding step over raw text would
either silently drop it (if PUA codepoints get filtered as junk) or leave a
confusing tofu/box character in a chunk shown to a doctor or pharmacist.
**Check:** scan extracted text for codepoints in the Unicode Private Use Area
ranges (`U+E000–U+F8FF`) — cheap and generalizes to any custom-glyph symbol
substitution, not just arrows.
**Handling:** for now, flag any monograph/section containing a PUA codepoint
for manual review or map known PUA codepoints (e.g. this book's `` →
`→`) via an explicit substitution table; do not pass raw PUA codepoints
through to chunking/embedding untranslated.
**Generalizes:** yes — any PDF built from print-authoring software that uses a
symbol font for arrows/special glyphs (common in scientific/medical/chemistry
documents) can exhibit this; always check for PUA codepoints in extracted
text as a standing sanity check, not just assume standard Unicode symbols.
**Confirmed real chemical formula in the corpus, but rare:** a regex scan for
molecular-formula-shaped tokens (`[A-Z][a-z]?\d{1,3}` repeated) across the
monograph page range found 9 raw hits; manual inspection found most are
**false positives** (`H5N1` = flu strain name, `P2Y12` = a receptor name, not
molecular formulas) and only one confirmed genuine chemical formula/equation
(the Na2S2O3 case above) — real chemical notation exists in this corpus but
is genuinely rare, not a systemic pattern requiring a general chemistry
parser.

### 17. Adult/child dosing-population splits are the norm, not an edge case
**What it looks like:** measured via a whole-monograph-range text scan for
"Người lớn"/"Trẻ em"/"Trẻ sơ sinh" (adult/child/newborn) — these terms appear
on **1121 of ~1400** monograph-range pages, i.e. the large majority of drug
monographs split dosing by patient population.
**Why it matters:** this is exactly the kind of structural content where a
segmentation/chunking bug that interleaves or merges adjacent subsections
(e.g. a table/list continuation bug, see items 5-6) would be a genuine
patient-safety risk, not just a data-quality nicety — mixing an adult dose
into a child-dose chunk (or vice versa) is a plausible, concrete failure
mode given how common this structure is.
**Handling:** treat "does this monograph's dosing section correctly keep
adult/child/newborn subsections un-interleaved" as a standing validation
check (not a rare-case afterthought), given the measured prevalence.
**Generalizes:** yes — any clinical/pharmacological reference document
organized with population-specific subsections has this same risk profile;
measure real prevalence before deciding how much validation effort a
structural risk deserves (same methodology lesson as item 12a).

### 18. A monograph title can legitimately repeat — disambiguated by a bold, non-caps qualifier line
**What it looks like:** confirmed real example, found while smoke-testing
the real `segment/detector.py` against the full book: "SALBUTAMOL" is
detected as a monograph title **twice** (physical pages 1261 and 1263).
Rendering both pages to images and reading them directly (not inferred from
coordinates) confirmed these are two genuinely different, complete
monographs — "SALBUTAMOL (Dùng trong hô hấp)" (respiratory use) and
"SALBUTAMOL (Dùng trong sản khoa)" (obstetric/tocolytic use) — each with
its own full 18-section template. The qualifier ("(Dùng trong hô hấp)" /
"(Dùng trong sản khoa)") is a bold line immediately below the all-caps
title, but is **not itself all-caps** (mixed case inside the parens), so it
is correctly excluded from `detect_monograph_titles`'s all-caps candidate
filter — it must instead be captured as a *separate* signal and folded into
the monograph's disambiguating identity downstream.
**Why it matters:** an assembler that derives `drug_id` from the title text
alone (e.g. a simple slug of "SALBUTAMOL") will produce a real collision
between two legitimately different monographs — this is **not** the same
failure mode as the already-fixed GONADOTROPIN false-collision (that one
was a detector artifact from unmerged multi-line wrapping; this one is a
genuine same-name-different-monograph case that must be preserved, not
merged away).
**Handling (for Phase 1.3's assembler):** after detecting a monograph title,
check for an immediately-following bold, parenthesized, non-all-caps line
directly below it (same page, small y-gap) and include it in `drug_id`
generation when present, so "salbutamol_ho_hap" and "salbutamol_san_khoa"
remain distinct rather than colliding as "salbutamol" twice. The
`assembler.py` duplicate-drug_id check (outlier-catalog reasoning already
established: raise on a genuine duplicate rather than silently overwriting)
must be designed with this real case in mind, or it will incorrectly reject
a legitimate second "SALBUTAMOL" entry.
**Generalizes:** yes — any drug/entity reference work that documents the
same base substance under multiple distinct use-contexts (formulation,
indication, route) can have this exact pattern; never assume a title string
alone is a unique key without checking for a disambiguating qualifier line.

### 19. Table column headers can be bold + all-caps + short — identical shape to a real title
**What it looks like:** confirmed real example, found via a whole-book
`assemble()` run raising a duplicate-drug_id error: "HSV" and "CMV" each
appear twice as bold, all-caps, short (3-char) spans on physical page 698 —
not drug names at all, but **column headers in a dosing-by-renal-function
table** inside the "Foscarnet natri" monograph ("Liều đối với HSV / HSV /
CMV / CMV"). Bold+all-caps+short is exactly the monograph-title signal
(item 10/12d), so this is a genuine detector ambiguity, not a coding bug.
**Why it matters:** unlike item 12d's part-divider titles (a small,
enumerable, fixed set of known strings), a table's column headers are
unbounded and content-dependent (any future table could use "HSV", "CMV",
or something else entirely as a header) — an exclusion list approach
doesn't generalize here the way it did for part-dividers.
**Handling:** require a **structural anchor** rather than a text exclusion
list: a real monograph title is always followed shortly by at least one
recognized section heading from the vocabulary (in practice, always "Tên
chung quốc tế" first) before the next title-shaped candidate. A
table-header false positive is not — the table's own cells are numbers/
plain text, matching no vocabulary entry. Implemented as
`assembler._filter_false_positive_titles` (lookahead of 6 events, checked
against the same coalesced event stream already built for assembly — no
separate detection pass, no duplicated logic).
**Generalizes:** yes, more broadly than item 12d — any document where
section/entity boundaries are marked by a *shape* (bold+caps+short) that a
table, list, or figure caption could coincidentally also match should
verify a **structural follow-on anchor**, not just a shape match or a
denylist of known bad strings, since the space of possible false-shaped
content (table headers, figure labels, pull-quotes) is unbounded while the
space of "what a real boundary is followed by" is small and known.

### 20. Section headings are not consistently bold across monographs — some combine label+value in one plain span
**What it looks like:** confirmed real example, found by investigating why
a whole-book `assemble()` run showed 48 monographs with zero ATC codes and
not stated-absent (far more than the ~13-14 the original spot-check
extrapolated). AMITRIPTYLIN's real "Mã ATC:" field is a **single, plain
(non-bold)** span containing the label AND value together: `"Mã ATC:
N06AA09."` — unlike Abacavir's equivalent, which is a bold `"Mã ATC: "`
label span followed by a separate plain `"J05AF06."` value span. Both
render visually similar but have completely different span/style
structure. Given the book's own foreword states it was "biên soạn bởi
nhiều tác giả" (written by many authors), this kind of per-author styling
inconsistency across ~700 individually-authored monographs is plausible
and, once checked, confirmed real — not a one-off.
**Why it matters:** a detector that requires `span.bold` to recognize a
section heading (reasonable-looking given every *title* is confirmed bold)
silently drops entire sections for a meaningful fraction of the corpus —
this directly caused undercounted ATC codes (and, structurally, would
equally affect any other section) for monographs using this looser style.
**Handling:** match section headings by **vocabulary text**, not by
boldness — the same "don't gate on a styling attribute, only content is
reliable" lesson as item 10 (font size), now applied to boldness. Also
handle the "label + value combined in one span" shape explicitly (a prefix
match: does the span start with a known label followed by ":", with the
remainder treated as the section's inline value) rather than assuming
label and value are always separate spans.
**Generalizes:** yes — any print-authored reference work assembled from
many individual authors/editors over a long production process should
expect inconsistent low-level styling of nominally-identical structural
elements; verify a structural signal (styling) against the *content* it's
supposed to correlate with, across a large real sample, before trusting it
as a universal discriminator — the same methodology lesson as item 10,
found again independently here.

### 21. "All-caps" is not 100% reliable either — and a class-level monograph's own internal sub-headings can masquerade as new monographs
**What it looks like:** two distinct confirmed real findings from the same
investigation:
1. The class-level monograph "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" embeds the
   mixed-case abbreviation "CoA" (Coenzyme A) inside an otherwise all-caps
   title. A strict `text.isupper()` check requires *zero* lowercase
   letters, so this single embedded abbreviation caused the entire
   monograph to be silently dropped from the corpus — found only by
   directly checking whether this specific, previously-known (outlier item
   12a) class-level monograph was present in a real whole-book `assemble()`
   run, and discovering it was not.
2. Within that same class-level monograph, individual statin names
   ("SIMVASTATIN", "LOVASTATIN", "PRAVASTATIN", "FLUVASTATIN") appear as
   their own bold+all-caps+short sub-headings, each introducing its own
   "Liều lượng và cách dùng" sub-section — shape-identical to a real
   monograph title, and (after fix 1 above made the loosened "any known
   section" anchor check pass) briefly became a second false-positive
   category alongside item 19's table headers, since these sub-headings
   *are* followed by a recognized section, just never by "Tên chung quốc
   tế" specifically (that section belongs only to the parent).
**Why it matters:** together these show that neither "all-caps" nor "loosen
the anchor to any section" is safe in isolation — the fix for one false
positive (item 19, HSV/CMV) reopened a different one (SIMVASTATIN) until
the anchor check was tightened back to the *specific* section the book's
own template guarantees is always first for a genuine top-level monograph.
**Handling:** `detector._is_mostly_upper` uses a **lowercase-letter ratio**
(≤10%), not an absolute count — an earlier absolute-count version (≤2
lowercase letters) let a real regression through: "Mã ATC:" has only 1
lowercase letter (a normal Vietnamese diacritic, 'ã') but that's 20% of its
5 letters, correctly rejected by the ratio while HMG-CoA's 1/27 ≈ 3.7%
correctly passes. `assembler._has_anchor_ahead`
requires specifically the "ten_chung_quoc_te" section key, not just any
recognized section, since that is the one invariant the book's documented
template actually guarantees is unique to real top-level monographs.
**Generalizes:** yes — (1) don't assume a styling/casing convention holds
with zero exceptions across an entire corpus, even one confirmed exception
matters at whole-corpus scale; (2) when a document has nested substructure
that mimics top-level structure (a class monograph containing per-item
sub-entries), the anchor used to confirm a real boundary must be the most
*specific* invariant available, not just "some known follow-on content" —
a looser check that fixes one false positive can silently reopen another.

### 22. Running-header boilerplate was never actually stripped, despite item 13's warning — measured whole-corpus at 98.4% of monographs affected
**What it looks like:** the running header at the top of every physical page
("DTQGVN 2" + printed page number + the current monograph's name, e.g.
physical page 1008's "DTQGVN 2" / "1009" / "Morphin sulfat", tagged
`column="full_width"` by `extract/spans.py`) matches no section heading and
isn't a real all-caps title, so it fell through every classification branch
in `assembler._classify` into plain body text — splicing itself into the
*middle* of whatever section is open when a physical page turns. Real
example, MORPHIN SULFAT's `liều lượng và cách dùng`: `"...Nếu\nDTQGVN 2\n
1009\nMorphin sulfat\nuống viên thuốc..."` — the header text lands inside a
real dosing sentence.
**Why it matters:** item 13 (above) already *warned* "strip the fixed
boilerplate before parsing content" back when the extraction layer was
first built, but that step was never actually implemented in `assembler.py`
— the warning existed in the catalog without a corresponding code path or
test enforcing it, and nothing caught the gap until a whole-corpus
measurement was actually run. Measured: **1,374 of 11,409 sections (12.0%)
contained a literal "DTQGVN" string mid-text; 671 of 682 monographs (98.4%)
had at least one affected section** — this is not a rare edge case, it's
the default outcome for any section whose text happens to cross a physical
page boundary (i.e. most sections longer than about half a page). Left
unfixed, boilerplate gets baked into chunks and embeddings and can surface
mid-sentence in a citation shown to a doctor/pharmacist.
**Handling:** `assembler._is_page_boilerplate` drops any span with
`column == "full_width"` and `y0 < HEADER_BAND_Y` (the same header-band
threshold `page_map.py` already uses to read the folio) before it reaches
any other classification branch. Whole-corpus re-measurement after the fix:
0 of 11,409 sections contain "DTQGVN". Regression test uses the exact real
MORPHIN SULFAT span shape.
**Generalizes:** a documented risk in this catalog is not the same as a
verified-fixed risk — "we know this could happen" needs a whole-corpus
measurement (not just a warning paragraph) before it can be crossed off,
and ideally a regression test that would fail if the fix were ever reverted.

### 23. PyMuPDF's raw block order doesn't reliably sequence left-column-before-right-column — confirmed wrong on 12 of 1398 pages
**What it looks like:** `extract/spans.py` originally trusted PyMuPDF's own
block iteration order to already emit left-column content before
right-column content, validated only against one example page during ADR
0003. On physical page 1100 (the OXYBUTYNIN/OXYMETAZOLIN monograph
boundary) and 11 other pages, PyMuPDF's raw block order emits the *right*
column first. Since `assembler.assemble` appends section content to
whichever monograph is currently open, this silently attributed
OXYMETAZOLIN's right-column sections (Chống chỉ định, Thận trọng, Thời kỳ
mang thai, Thời kỳ cho con bú, ADR, Hướng dẫn xử trí ADR, Liều lượng và
cách dùng) to the still-open OXYBUTYNIN monograph — overwriting
OXYBUTYNIN's real sections and leaving OXYMETAZOLIN missing all 7.
**Why it matters:** medically relevant — wrong contraindication/ADR content
silently attached to the wrong drug. Found via a whole-document
(1668-page) character-similarity diff against an independent parser
(`opendataloader-pdf`), not from a sample; confirmed by rendering the page
to an image and reading it directly, then confirmed again in the actual
`assemble()` output.
**Handling:** `extract.spans._sort_blocks_reading_order` explicitly sorts
each page's blocks by (full_width header band first, then left column,
then right column) and then by y-position, instead of trusting raw PyMuPDF
order. Whole-range (99-1496) re-scan after the fix: 0 pages with the
reversed-order signature (was 12). Directly verified OXYBUTYNIN's and
OXYMETAZOLIN's `assemble()`-produced sections are now distinct and
drug-appropriate.
**Generalizes:** don't trust an upstream library's element ordering just
because it happened to be correct on the one page checked during initial
validation — for a whole-corpus pipeline, explicitly sort by the actual
signal you care about (here: visual column position) rather than an
implicit "the library probably does this right" assumption.

### 24. Some text exists only as vector outlines — no text extractor can read it, and single dropped glyphs corrupt otherwise-clean sentences
**What it looks like:** physical page 714 prints 17 full lines of ordinary
GATIFLOXACIN prose that `page.get_text()` does not return, `page.search_for()`
cannot find, and neither `pdfplumber` nor `opendataloader-pdf` returns either.
`page.get_drawings()` shows why: each line is a filled path of 1,126-1,831
items, shaped exactly like one line of type and filled with the body-text
colour. The same defect occurs at glyph granularity (39-45 path items), and
that form is far more dangerous — a single Vietnamese diacritic character
drops out of a line that otherwise extracts perfectly: `Độ ổn định` extracts
as `Độ n định`, `≥ 1 tuổi` as `≥ 1 tu i`, `tại chỗ` as `tại ch `. The result
reads as ordinary text, so no structural check, no count and no cross-tool
comparison notices it.
**Why it matters:** this is silent loss of clinical prose in a drug
formulary, and it is invisible to every check that asks a text layer a
question. It survived a whole-document span-coverage ledger reporting
`unassigned = 0`, because the spans that existed were all routed correctly —
the missing content was never a span at all.
**Check:** render the page, white out every extracted span's bbox, and look
at the ink that survives (`ingestion/validation/residual_ink.py`, ~0.06
s/page). Confirm with `page.get_drawings()`: a filled path with ≥30 items
whose box is 3-20pt tall is type, not decoration (real decoration on this
book carries 1-2 items).
**Handling:** `ingestion/extract/outlined_text.py` detects the runs;
recovery cannot be automatic because the paths carry no character codes, so
each run was rendered and transcribed by reading it, into
`ingestion/data/verified/outlined_text_transcriptions.json` with page, bbox,
and the extracted line it belongs to. Whole-document scope: **51 runs on 5
pages** (714 ×31, 736 ×16, 1373, 1444, 1445 ×2), 1,116 characters.
**Generalizes:** yes — any PDF produced by a layout tool that converts
selected text to outlines (common when a font cannot be embedded) has this.
Never treat "the text layer returned something for this page" as evidence
the page was fully extracted; compare against the rendered pixels.

### 25. A fraction can be printed with no fraction bar at all, so no geometric detector can find it
**What it looks like:** ADENOSIN (physical page 147) prints its infusion-rate
formula as three plain lines — `Tốc độ truyền dịch (ml/phút) = 0,140
(mg/kg/phút) × trọng lượng cơ thể (kg)` / `Nồng độ adenosin (3 mg/ml).` —
with **no rule drawn between numerator and denominator**, confirmed by
rendering the region and reading it. Extracted linearly it reads as a
multiplication chain, i.e. the division silently disappears.
**Why it matters:** it defeats the detector that catches every other 2D
formula in this book. The fraction-bar signal (item 8, and
`residual_ink.py`'s `fraction_bar_candidate`) finds ink; there is no ink to
find here. It was caught only because a prose-leak gate matched its text.
**Check:** there is no cheap automatic check. Treat any line ending in a
unit-bearing quantity immediately followed by a line that is itself a
unit-bearing quantity as a division candidate for human review.
**Handling:** quarantined via the verified region list with
`source_prints_no_bar: true`. The count of bar-less formulas in this book is
**unmeasured** — recorded as `recall_limit` in
`ingestion/data/verified/formula_regions_2d.json` so the bar scan is never
mistaken for complete formula coverage.
**Generalizes:** yes — measured precision of the fraction-bar rule on this
book is **16/23 = 69.6%**, and its recall is unknown. A geometric heuristic
finds candidates; it never proves absence.

### 26. Exact section vocabulary can occur as wrapped prose or inside tables; context must precede label matching
**What it looks like:** several unrelated defects shared one cause. A wrapped
body sentence can put `chống chỉ định.` alone on the next visual line
(NADROPARIN, physical page 1016); a dosing-table cell can literally be named
`Chỉ định` (WARFARIN p1485 and IOBITRIDOL p826); and a verified fraction band
widened to capture its numerator can geometrically overlap prose in the other
column (NETILMICIN p1042). Exact vocabulary matching alone classified these as
structure or quarantined content.

**Why it matters:** the output remains grammatical while moving or deleting a
clinically decisive phrase, assigning a dosing table to indications, or hiding
a cross-reference. Aggregate “all spans assigned” and section-level provenance
gates all passed before these defects were found.

**Handling:** classify out-of-scope spans and known table regions before title/
section matching; treat a non-bold exact label as prose when it is the adjacent
line of an unterminated span in the same PDF block; require a formula region's
column to agree with the source span's column; and validate source-span IDs on
every individual part. Confirmed aliases (`Tên chung quốc tế và mã ATC`, `Dạng
bào chế và hàm lượng`, and the tetanus-toxoid dosing heading) are recorded in
the open vocabulary.

**Whole-corpus result:** 684 monographs (was 683), maximum monograph range 7
pages (was the false 164-page ZOLPIDEM range), 11,974 sections, 151 quarantined
blocks, 15,066 chunks, 0 unassigned spans, and every readiness gate passing.

**Generalizes:** vocabulary is evidence, not sufficient context. Apply known
geometric scope (page, table, column, visual-line continuity) before interpreting
a label-shaped string as document structure.

### 27. One physical table can be non-contiguous in PDF block order
**What it looks like:** a table is contiguous on the rendered page, but the PDF
content stream interleaves a visually later section heading between its cells.
This split CAPECITABIN p308 and IMATINIB p795 into multiple blocks with the same
region ID and conflicting section owners. CAPECITABIN p309 adds a second case:
two explicitly captioned dose-adjustment tables are printed after the ordinary
`Tên thương mại` field without repeating the dosage heading.

**Why it matters:** sorting or classifying one extracted span at a time makes a
single physical object acquire several meanings. The flattened text remains
plausible, so ordinary text and coverage gates do not expose the defect.

**Handling:** collect all spans belonging to a verified region before semantic
classification and emit the region atomically at its first occurrence. A narrow
caption rule maps only `Bảng N. Điều chỉnh liều ...` appendices to
`lieu_luong_va_cach_dung`; generic occurrences of the word “liều” are not used.
A readiness gate now requires unique physical-region IDs.

**Verification:** all **151/151 unique regions** were rendered and read against
the PDF. The regenerated corpus has 151 blocks, 151 unique IDs, and zero
duplicate-ID gate failures; CAPECITABIN p309 tables are both owned by dosage.

**Generalizes:** physical-region identity must outrank text-stream adjacency for
tables, formulas, figures, and other layout objects.

### 28. A bar-less formula needs an asymmetric band, but geometry cannot prove its operator
**What it looks like:** ADENOSIN p147 prints a wrapped numerator followed by
`Nồng độ adenosin (3 mg/ml).` with no horizontal fraction rule. The generic
symmetric formula band captured the numerator only, making a plausible but
incomplete source crop.

**Why it matters:** the missing denominator changes the calculation. Visual
review of all reconstructed sandbox crops found the defect even though ordinary
readiness and block-count gates passed.

**Handling:** verified bar-less regions use a 31pt lower margin from the
synthetic anchor. On this page the denominator ends about 29pt below the anchor;
the following `Ví dụ:` begins immediately after the new boundary. A regression
requires the denominator boundary and excludes that prose. The reconstructed
record still sets `requires_human_operator_confirmation`: layout supplies no
bar from which multiplication versus division can be proven.

**Generalizes:** expand a verified crop to preserve all visible operands, but
never invent a mathematical operator that the source geometry does not encode.

## Not yet investigated (flagged for future work, not silently ignored)

- **Footnote-style superscript reference markers** (seen as `a, b, c, d` in
  one table) — not yet checked for whether the footnote text stays
  correctly associated with its marker/row during extraction.
- **How many bar-less formulas exist** (item 25) — one confirmed, total
  unmeasured; no geometric signal can bound it.
- **Production 2D grid reconstruction** (item 7) — the 100-page sandbox now
  reconstructs grids and logical cross-page tables, but merged-cell semantics
  and whole-book recall are not yet production gates.
- **Exact shortest monograph name+page** — a quick unmerged crude scan (no
  multi-line title merge) gave a different longest-monograph ranking than
  the already-documented authoritative one (item 12e: "AMOXICILIN VÀ KALI
  CLAVULANAT", 45,623 chars), meaning the crude scan's numbers are not
  reliable enough to name an exact shortest monograph — deferred to the real
  Phase 1.2 detector (with proper multi-line merge and back-index-validated
  boundaries), which will produce a trustworthy number as a side effect of
  its own validation run, rather than trusting today's quick, differently-
  scoped script.