# 05 — Document parsing

How 1,668 PDF pages become 684 structured monographs. The empirical background
is in the pre-existing `docs/adr/0003-pdf-parsing-strategy.md`,
`docs/document-profile.md` and `docs/pdf-parsing-outlier-catalog.md`; this page
describes the code that resulted.

## Parsing pipeline

```mermaid
flowchart TD
    PDF[/PDF page/]
    SP["extract_spans<br/>text + bold flag + bbox + page"]
    PM["build_page_map<br/>printed folio per physical page"]
    OT["merge_outlined_runs<br/>put vector-path-only text back"]
    NG["normalize/glyphs.py<br/>PUA + known-corruption substitution"]
    NF["normalize/text_flow.py<br/>visual-line joining"]
    CL["assembler._classify<br/>span → Span | _SectionEvent | _TextEvent"]
    MT["detect_monograph_titles<br/>bold + mostly-upper + 3..60 chars + page range"]
    SH["detect_section_headings<br/>bold + match_section(vocab)"]
    CO["_coalesce_titles<br/>merge multi-line headings"]
    FP["_filter_false_positive_titles<br/>needs an anchor section ahead"]
    AS["assemble<br/>emit Monograph"]

    PDF --> SP --> OT --> NG --> NF --> CL
    PDF --> PM --> SP
    CL --> MT --> CO --> FP --> AS
    CL --> SH --> AS
```

## Monograph title detection — `segment/detector.py`

Rule (validated, ADR 0003): **bold + mostly-upper + short line + inside the
monograph page range**. Font *size* is explicitly not part of the rule — a
`size >= 9.8` threshold was measured dropping ~15% of real monographs.

```python
MONOGRAPH_PRINTED_PAGE_START = 99      # both printed AND physical bounds
MONOGRAPH_PRINTED_PAGE_END   = 1496    # are checked; either alone has
MONOGRAPH_PHYSICAL_PAGE_START = 99     # known failure modes
MONOGRAPH_PHYSICAL_PAGE_END   = 1496
_MIN_TITLE_LEN = 3
_MAX_TITLE_LEN = 60
_MAX_LOWERCASE_RATIO = 0.10
```

`_is_mostly_upper` tolerates up to 10% lowercase letters rather than requiring
`str.isupper()`. The reason is a real regression: the class-level monograph
`CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE` embeds the mixed-case `CoA`, and a strict
check silently dropped the whole monograph. The threshold is a *ratio* because
an earlier absolute-count version let the short label `Mã ATC:` through as a
false title.

Known false positive, excluded by name rather than tuned around: part-divider
titles like `CÁC CHUYÊN LUẬN THUỐC` sit exactly at the printed-page-99 boundary
and are bold + all-caps + short — `vocab.is_part_divider` rejects them.

A second guard, `_filter_false_positive_titles` + `_has_anchor_ahead`, requires a
plausible section heading to follow a candidate title before it is accepted.

## Section heading detection — `segment/detector.py` + `vocab.py`

Bold spans within the page range are matched against an open vocabulary
(`segment/vocab.py::match_section`). There is **no** all-caps requirement here,
because most section headings (`Chỉ định`, `Liều lượng và cách dùng`) are not
all-caps. The vocabulary is data, so adding a phrasing is an entry, not a code
change.

The 19 canonical section keys are listed in
[06-document-model-and-chunking.md](06-document-model-and-chunking.md) and
duplicated (deliberately, as a closed vocabulary for the LLM) in
`apps/ai-service/rag/understanding.py::SECTION_KEYS`.

## Line-level heuristics — `segment/assembler.py`

The classifier is where most of the accumulated PDF-specific knowledge lives:

| Helper | Purpose |
|---|---|
| `_is_page_boilerplate` | Drop running headers/footers |
| `_starts_its_visual_line` / `_continues_previous_visual_line` | Rebuild visual lines from spans |
| `_is_body_line_that_reads_like_a_label` | Stop body prose being read as a heading |
| `_is_mid_line_label` | A label appearing mid-line, not at line start |
| `_is_italic_cross_reference` | Italic "see also" runs |
| `_is_qualifier_line` | Parenthetical qualifiers under a title |
| `_slugify` | Drug name → `drug_id` |

Text between a monograph title and its first section heading is captured as
`Monograph.preamble` rather than dropped — the code names the case: `ARTEMETHER`
(physical page 210) opens with the regulatory notice that single-agent
artemisinin products were withdrawn.

## Table and formula handling

### Detection — `tables/detect.py`, `tables/classify.py`

Regions are located and classified into shapes:

| Shape | Meaning |
|---|---|
| `simple_table` | Regular rows/columns |
| `multi_level_or_merged_header` | Merged/multi-level header |
| `cross_page_continuation` | Continues onto the next page |
| `grid_2d_numeric` | 2-D numeric lookup grid |
| `formula_2d` | A 2-D formula (from `data/verified/formula_regions_2d.json`) |
| `not_a_table_full_page` | False positive, full-page region |
| `single_column_boxed_list` | Boxed list, not a table |

`QUARANTINE_SHAPES` is the subset whose flattened text would be actively
misleading. `assembler.py` marks spans inside those regions
`SPAN_STATE_QUARANTINED`; everything else inside a region is `SPAN_STATE_TABLE`.

### The quarantine contract

A quarantined block:

- is **lifted out of** the section's prose (`SectionSpan.prose_text` filters
  `quarantined` parts);
- becomes a `TableBlock` on the monograph with its own `table_id`, `bbox`,
  `physical_page` and `shape`;
- produces a `block_descriptor` chunk whose text is built **only from
  metadata** — drug name, section display name, "bảng"/"công thức", printed
  page, and the sentence *"Nội dung chỉ tra cứu được trên ảnh trang gốc, không
  trích dẫn được dưới dạng văn bản."* No cell value ever appears;
- sets `has_quarantined_content=True` on every prose chunk of that section, which
  the retrieval layer reads as `requires_visual_check` and turns into a
  `VERIFY_PDF` decision.

Header rows are deliberately **not** embedded either
(`chunker.py::_attachment` forces `header_row=[]`). The measured reason: 42 of
124 simple-table headers contain a digit, and `AMIODARON`'s (physical page 183)
"header" was a dose — `Thời gian liệu pháp tĩnh mạch Liều 720 mg/ngày (0,5
mg/phút)`.

## Normalization

| Concern | Module |
|---|---|
| Private-use-area and known-corruption glyph substitution | `normalize/glyphs.py` |
| Joining spans into flowing text, hyphenation, line breaks | `normalize/text_flow.py` |
| Diacritic-stripped casefolding for matching (never for storage) | `apps/ai-service/rag/text.py::normalize_name` |

Gates `pua_char` and `replacement_char_ufffd` both target zero, so a surviving
U+FFFD or PUA codepoint fails the readiness check rather than being embedded.

## Verification instruments (no ground truth required)

Three independent instruments, each answering a different question:

| Command | Question | Output |
|---|---|---|
| `validate` | Did we find the monographs the book's own back index lists? | recall / precision, plus unmatched entries both ways (`validation/back_index.py`) |
| `coverage` | Where did every extracted span end up? | span + character counts per state, with the `unassigned` bucket broken out by page |
| `residual-ink` | What ink is on the page that no span accounts for? | region census by kind; **gate: `unclassified` must be 0** (`validation/residual_ink.py`) |

`residual-ink` is the one that needs no extraction at all to be trusted — it
rasterises the page and asks what the text layer failed to emit.

## Known parsing limits, stated by the code itself

- `scan_glyph_order` / `scan_reading_order` **report** glyph and reading-order
  defects; they do not correct them. Formula-region issues are expected and left
  alone.
- Table row/column reconstruction is not verified — the `chunk-ready` output
  says so.
- Recall for borderless tables and bar-less formulas is unquantified.
- `pdfplumber` is used only for its table API; its body-text order is unreliable
  for this layout.