Add comprehensive, reusable PDF-parsing outlier catalog

docs/architecture.md

@@ -46,21 +46,40 @@ needed now.
 ## RAG ingestion pipeline (PDF-specific)
 
 The formulary is a structured per-drug reference, not free prose — the
-pipeline exploits that structure instead of naive fixed-size chunking:
+pipeline exploits that structure instead of naive fixed-size chunking. This
+section reflects an actual empirical investigation of the real PDF (not
+assumptions) — see `docs/adr/0003-pdf-parsing-strategy.md` for the full
+methodology, cross-tool comparison, and validation numbers.
 
-1. **Extraction**: PyMuPDF (`fitz`) as primary extractor (font size/style/
-   position metadata enables heading detection); pdfplumber as a fallback
-   specifically for tabular content (dosing/interaction tables). Raw
-   per-page extraction is persisted to `ingestion/data/interim/` so
+1. **Extraction**: PyMuPDF (`fitz`) as primary extractor. This document has
+   **no bookmark/outline** (`doc.get_toc()` returns 0 entries — confirmed,
+   do not rely on it) and is a **tagged PDF with only a shallow, unusable
+   structure tree** (~29 generic H1/P elements covering a fraction of 1668
+   pages — also confirmed dead-end, not a data source). PyMuPDF's reading
+   order was cross-validated against `pdfplumber` and `opendataloader-pdf` on
+   real sample pages: pdfplumber's default text order is **unreliable** for
+   this layout (scrambles paragraph order, leaks marked-content artifacts) —
+   use it only for its dedicated table-extraction API, never for body text.
+   Raw per-page extraction is persisted to `ingestion/data/interim/` so
    re-segmentation doesn't require re-running the expensive extraction step.
-2. **Segmentation**: detect drug-entry boundaries (prefer the PDF's
-   bookmark/outline via `doc.get_toc()` when present, else font-size/style
-   heuristics), then classify each heading against a canonical section
-   taxonomy (`chi_dinh`, `chong_chi_dinh`, `lieu_dung`, `tac_dung_phu`,
-   `tuong_tac_thuoc`, etc., Vietnamese diacritic-insensitive matching).
-   Output: `{drug_id, drug_name, source_page_range, sections: {...}}` per
-   drug, persisted to `ingestion/data/processed/monographs.jsonl` and
-   manually spot-checked via `ingestion/notebooks/`.
+2. **Segmentation**: drug-entry boundaries are detected via **bold-font
+   spans** (PyMuPDF span `font` containing `"Bold"`), not font-size alone —
+   font size for title/heading spans varies between monographs (confirmed:
+   10.0pt and 9.5pt both occur for genuine drug-title headings), so bold is
+   the reliable signal, all-caps + short length narrows it to monograph
+   titles specifically. Section headings inside a monograph are also bold
+   spans, cross-checked against a canonical taxonomy (`chi_dinh`,
+   `chong_chi_dinh`, `lieu_dung`, `tac_dung_phu`, `tuong_tac_thuoc`, plus
+   real observed extras like `ten_thuong_mai` "Tên thương mại" not in the
+   book's own documented 19-field list — treat the taxonomy as open/
+   extensible, not a fixed enum). Multi-line wrapped titles/headings (long
+   Vietnamese names/vaccine names) must be merged across consecutive
+   bold+all-caps lines before matching — this was the single largest source
+   of missed detections in validation. Output: `{drug_id, drug_name,
+   source_page_range, sections: {...}}` per drug, persisted to
+   `ingestion/data/processed/monographs.jsonl` and validated both
+   automatically (see ADR 0003) and via manual spot-check in
+   `ingestion/notebooks/`.
 3. **Chunking**: each `(drug, section)` pair is the natural chunk unit;
    never split a section unless it exceeds a token budget (~500-800 tokens),
    in which case sub-chunk with a sliding window (400 tokens, 50 overlap),