Add read-only production runtime audit

docs-legacy/document-profile.md

@@ -0,0 +1,230 @@
+# Document Profile — Dược thư quốc gia Việt Nam 2018
+
+Reverse-engineering survey of the source PDF (`duoc-thu-quoc-gia-viet-nam-2018.pdf`,
+1668 pages) to catalog every distinct page/content type BEFORE deciding what
+parser modules to build. **Classification only — nothing here changes the
+parsing pipeline.** Purpose: give real numbers to decide which content types
+are common enough to deserve a dedicated pipeline stage, per the "leverage
+the existing pipeline + add supplementary handling" direction agreed with
+the user (not a full architecture rewrite).
+
+Method, per this project's standing rules ([[feedback-rigorous-validation]],
+ADR 0003): every count below is a **whole-document** scan (all 1668 pages,
+not a sample), classification rules are stated explicitly so any number can
+be independently re-checked, and every non-trivial claim is cross-checked
+with a second tool (`opendataloader-pdf`, the tool ADR 0003 validated for
+this purpose — **not** `pdfplumber`, which ADR 0003 already found scrambles
+reading order on this document) and/or a rendered-page-image visual read.
+
+Reproducible script: `ingestion/scratch/document_profile_group1.py`
+(investigation code per CLAUDE.md's rules — temporary, not imported by
+production code; delete once this doc + any resulting regression fixtures
+fully capture its findings).
+
+**Note on page numbering**: all page numbers below are physical/0-indexed
+(PyMuPDF convention). A PDF viewer's page counter is 1-indexed:
+`viewer page N == physical page N-1`.
+
+## Group 1 — objectively measurable (done, verified)
+
+| Category | Rule | Count | Verification |
+|---|---|---|---|
+| 2-column | page has both `column="left"` and `column="right"` spans (ADR 0003 bbox ranges) | 1628 | rule-based, matches known monograph-body layout |
+| Mixed/other layout | page has a set of column tags not matching the other 3 buckets | 32 | **100% manually viewed** (rendered every page) — see breakdown below, zero anomalies |
+| Full-width only | only `column="full_width"` spans | 5 | pages 3, 5, 37, 97, 1497 — all print-layout blank/divider-adjacent pages |
+| No text extracted | zero spans on the page | 2 | pages 99, 1666 |
+| Single-column-side | only `left` or only `right`, no `full_width` | 1 | page 1495 — near-empty (1 span), boundary page right at the monograph range end (1496) |
+| Near-empty (<20 chars) | `doc[p].get_text().strip()` length | 7 | pages 3, 5, 37, 99, 1495, 1497, 1666 — all print-layout blank/separator pages, consistent with ADR 0003's earlier finding of 6 (this scan found 1 more, page 5, confirmed same nature by direct read) |
+| Embedded images | `doc[p].get_images(full=True)` non-empty | 0 | 2 independent scans, 2 sessions, same result — **zero scanned pages in this document, no OCR needed** |
+| Chemical reaction equations (confirmed) | manual read of every regex candidate's context | **2** | see "Formula/notation" below — corrected from an initial loose-regex count of 25 |
+| Ion/electrolyte notation (Na+, Ca2+, Cl-, etc.) | same regex, reclassified after context read | ~23 pages (of the 25 original candidates) | common prose notation, not a "formula" needing special parsing — but subscript/superscript preservation matters, see below |
+| Comparison-operator notation (ADR frequency thresholds, "ADR > 1/100") | regex: digit adjacent to `<`/`>` | **933** | this is a **standard template pattern**, not an outlier — appears in the "Tác dụng không mong muốn (ADR)" section of most monographs, flagged by the user directly from a real page (Zolpidem, physical page 1494) |
+
+### Mixed/other layout — full breakdown (32/32 pages viewed)
+
+None are parsing anomalies. All are legitimate non-monograph content:
+
+- **Front-matter title/cover/copyright pages**: 0, 1, 2
+- **Foreword**: 6
+- **Committee/personnel roster** (name lists, 2-column but different geometry than monograph body): 7, 9, 10, 11
+- **Table of contents**: 8
+- **"Danh mục các chuyên luận thuốc"** — Vietnamese\|English drug-name reference table, 2-column but different bbox geometry than the monograph body column rule (hence not tagged `two_column`): 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31 (18 pages; pages 20 and 23 of this same table happened to match the monograph-body bbox rule and are already counted under `two_column`)
+- **"Ký hiệu chữ viết tắt"** — abbreviation table, 3 columns (abbreviation \| English \| Vietnamese): 33
+- **Part-divider title pages**: 36 ("CÁC CHUYÊN LUẬN CHUNG"), 98 ("CÁC CHUYÊN LUẬN THUỐC"), 1496 ("CÁC PHỤ LỤC"), 1528 ("MỤC LỤC TRA CỨU")
+- **Blank separator**: 1529
+- **Colophon (print/publisher info)**: 1667
+
+Potentially useful finding for future scope: the Vietnamese\|English name table
+(18-20 pages) could seed a synonym/alias table for search, if that's ever
+wanted — currently out of scope, noted only.
+
+### Formula/notation — corrected finding
+
+An initial loose regex found 25 candidate pages. **Reading the actual context
+of every match (cross-checked with `opendataloader-pdf`, not just PyMuPDF)
+showed this was the wrong classification** — most matches are ion/electrolyte
+charge notation (Na⁺, K⁺, Ca²⁺, Cl⁻, Mg²⁺, Fe²⁺/Fe³⁺, HCO₃⁻, PO₄³⁻, NH₄⁺),
+which is common, ordinary prose notation throughout the pharmacology text,
+not a distinct "formula" content type. Two unrelated `+`-adjacent patterns
+were also caught by the same regex and are semantically different again:
+"CD4+" (immunology cell-marker notation, not a chemical charge) and
+"O2 + N2O" (anesthetic gas mixture percentages).
+
+**Only 2 pages have a genuine chemical reaction equation:**
+1. Physical page 1033 (already known, outlier-catalog item 16): cyanide
+   antidote mechanism, `Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3` — the reaction arrow
+   extracts as a Private-Use-Area glyph (U+F0AF), not standard Unicode.
+2. Physical page 1027 (**new finding this session**, printed page 1028,
+   "Natri bicarbonat"): buffer equation `HCO₃⁻ + H⁺ → H₂CO₃ → CO₂ + H₂O`,
+   confirmed by rendering the page to an image — the source PDF renders
+   this with real visual subscript/superscript.
+
+**Real cross-cutting issue found, not yet sized or fixed**: both PyMuPDF's
+and `opendataloader-pdf`'s plain-text extraction **flatten subscript/
+superscript formatting** — the bicarbonate equation extracts as flat text
+("HCO-3+ H+ ... H2CO3 ... CO2 + H2O", digits inline, no vertical
+positioning info kept in the text string alone, though bbox/font-size data
+for the small subscript run is still recoverable from raw spans if a future
+stage needs to reconstruct it). This affects ion notation too, and likely
+also formula-adjacent abbreviations like "CD4", "Ca²⁺", "vitamin B₂/B₆/B₁₂"
+site-wide, not just these 2 pages — **the true scope of subscript/superscript
+loss has not been measured yet**, only observed on this one confirmed page.
+
+### Mathematical formulas — separate from chemistry, found after the user
+asked "what about math" (this profile initially only scanned for chemistry-
+shaped tokens and missed this category entirely — a real gap, not a
+deliberate scope decision)
+
+Whole-document regex scan for math symbols (full 1668 pages), initially run
+with PyMuPDF only — **caught by the user re-checking my methodology**
+("đừng dùng 1 con pymu" — don't rely on just one tool) — then re-verified
+against `opendataloader-pdf`'s independent whole-document text extraction
+(125s for all 1668 pages):
+
+| Symbol | Meaning | Pages found (PyMuPDF) | Total occurrences: PyMuPDF | Total occurrences: opendataloader-pdf |
+|---|---|---|---|---|
+| `±` | mean ± SD | 44 | 95 | 95 ✅ |
+| `≤` | less-than-or-equal (dosing/lab thresholds) | 91 | 178 | 178 ✅ |
+| `≥` | greater-than-or-equal (dosing/lab thresholds) | 144 | 244 | 245 (off by 1, unexplained, not chased further — negligible vs. the total) |
+| `×` | multiplication | 19 | 50 | 50 ✅ |
+| `√`, `÷` | square root, division | 0 | 0 | 0 ✅ |
+
+Two independent tools agree almost exactly (only the `≥` total differs, by
+1 out of 245) — real cross-tool evidence the symbol counts aren't a
+single-tool artifact, not just an assertion.
+
+`≤`/`≥` join the already-found `<`/`>` (933 pages) as further evidence that
+**threshold/comparison notation is a pervasive, standard part of this book's
+dosing and lab-value template**, not a rare outlier — same conclusion as
+before, now with more symbols confirmed.
+
+**`×` (19 pages) was individually context-checked (not just counted)** —
+splits into two real, different things:
+- **9 pages** use `×` only as dosing-frequency shorthand ("200 mg × 1
+  lần/ngày" = "200mg, once a day") or scientific notation ("18 × 10⁶")
+  — not a standalone formula: pages 61, 91, 153, 155, 516, 716, 794, 974, 1412.
+- **10 pages have genuine standalone calculation formulas** (variable =
+  expression), found in the general-chapters section (printed 37-98,
+  physical ~36-97) and one appendix: pages 43, 92, 94, 147, 206, 699, 853,
+  1274, 1359, 1498. Examples: Cockcroft-Gault creatinine clearance
+  (`Clcr(nam) = (140-tuổi)×thể trọng / (Ccr×72)`), MDRD GFR (`GFR(nam) =
+  186 × (Ccr)^-1,154 × (tuổi)^-0,203`), the DuBois body-surface-area formula
+  (`S = W^0.425 × H^0.725 × 71.84`, physical page 1498, Appendix 1),
+  elimination half-life (`t½ = 0,693×Vd/Cl`), clearance (`Cl = Q×E`).
+
+**Severe finding, confirmed visually, worse than the subscript-flattening
+issue above**: physical pages 43 and 94 (printed 44, 95 — "Sử dụng thuốc ở
+người suy giảm chức năng gan, thận" and the pharmacokinetics general
+chapter) were rendered to images and read directly. The PDF itself shows
+clean, properly typeset **stacked fractions** (numerator over denominator,
+e.g. `Cl_TP = D/AUC`, `t½ = 0,693×Vd/Cl`). But the plain-text extraction of
+these same formulas comes out **scrambled, not just subscript-flattened** —
+e.g. page 94's `Cl = Q × E = (Ca-Cv)/Ca` extracts as the fragment sequence
+`"Cl = Q × E = | a | v | a | C | C | C | Q | − | × |"`, unreadable and not
+recoverable by a simple flatten-subscript fix. This is a genuine reading-
+order defect specific to stacked-fraction layout, distinct from (and more
+severe than) the subscript-loss issue, confirmed on 2 pages so far — **not
+yet measured across all 10 real-formula pages**, only these 2 were rendered
+and read.
+
+**Scope honesty**: the `×`/`±`/`≤`/`≥` regex families are still just
+*candidate* signals for "this page has notable math content" — a formula
+using only `/` for a fraction, or only superscript exponents with no `×` at
+all, would not be caught by this scan. The 10-page "genuine formula" count
+should be read as a lower bound, not a confirmed total.
+
+**This also confirms a bigger open gap**: both real formulas and real data
+tables (Bảng 3, Bảng 4 — bordered tables with rows/columns, seen on page 43
+during the visual check) live in the **general chapters section (printed
+37-98)**, which per [[project-medical-chatbot-status]] memory has "never
+been structurally investigated." Group 2 below must cover this range, not
+just the monograph body.
+
+## Group 2 — heading / table / list types
+
+### Tables — in progress, NOT yet a trustworthy number
+
+`opendataloader-pdf`'s JSON output (whole-document, converted in 99s) has
+built-in structural typing (`heading`/`table`/`list`/`paragraph`/`caption`),
+so this was tried first instead of hand-writing a table detector.
+
+**Indexing pitfall caught before it became a wrong report**: opendataloader's
+`page number` field is **1-indexed** (confirmed via the RIBOFLAVIN reference
+point — its title lands at `page number: 1244`, and this document's
+physical(0-indexed)+1 == printed page always coincide, per ADR 0003's
+confirmed constant +1 offset — so `page number - 1 == PyMuPDF physical
+page`). An initial table-count query used the raw `page number` value
+unconverted and produced a count that only *coincidentally* matched a
+"2 tables" ground-truth check by luck — re-verified correctly afterward:
+physical page 43 (`page number 44`) shows 2 tables with captions "Bảng 3.
+Phân loại mức độ suy thận theo creatinin..." and "Bảng 4: ...tốc độ lọc cầu
+thận (GFR)" — an exact match to the page rendered and read directly
+earlier in this investigation.
+
+**Current whole-document numbers from opendataloader-pdf alone (converted
+to physical 0-indexed pages)**:
+- 170 table elements, on 129 distinct pages.
+- 107 of those pages are inside the monograph range (98-1495 physical); 22
+  are in the general-chapters range (physical 42-92, i.e. printed 43-93);
+  none found yet in the appendices range beyond page 1498 and 1509.
+
+**This count is NOT yet trustworthy as a final number** — it comes from a
+single tool, spot-checked correct on only 1 of 129 pages so far. Per
+ADR 0003, opendataloader's higher-level structural classifier (confirmed
+inconsistent for headings specifically) has an unknown reliability for
+tables specifically. Cross-checking now with `pdfplumber`'s
+`find_tables()`/`extract_tables()` — the tool ADR 0003 explicitly kept
+around *only* for table extraction (unlike its general text extraction,
+which is confirmed broken on this document) — whole-document run in
+progress, slower than opendataloader's, not complete as of this entry.
+**Do not cite the 170/129 numbers above as confirmed until this second
+tool's results are compared.**
+
+### Headings, lists — not started
+
+Requires proposing a taxonomy from real samples (per the "propose first,
+user reviews" approach agreed for this doc), since unlike Group 1's layout
+checks there's no purely objective rule to classify these — pending. The
+opendataloader JSON also has `heading` (3165) and `list` (1624) element
+counts whole-document, but per the table-count lesson above these should
+not be quoted as real numbers until cross-checked the same way.
+
+## Known gaps in this profile itself
+
+- Comparison-operator (933 pages) and ion-notation (~23 pages) candidates
+  were pattern-matched but not each individually opened — the sample checks
+  done (Zolpidem page for comparison-operators, all formula-regex contexts
+  for ion notation) are consistent enough to trust the *category*, but a
+  page-by-page audit of all 933/23 was not performed.
+- No table detection exists yet in this profile (Group 2 will need to define
+  a table-detection rule before it can be counted). Confirmed real bordered
+  tables exist at least on physical page 43 ("Bảng 3", "Bảng 4" — suy thận
+  classification), found incidentally while visually checking a math
+  formula, not from a deliberate table search.
+- General chapters (37-98 printed) and appendices (1497-1528 printed) have
+  only been surveyed for Group 1's layout/blank/image/formula/math
+  dimensions here — their own internal structure (headings, lists, full
+  table inventory within those sections) is still unsurveyed. This range
+  is now confirmed to contain real formulas and real tables (see Math
+  section above), so it must be explicitly in scope for Group 2, not
+  treated as monograph-adjacent filler.