Document Profile — Dược thư quốc gia Việt Nam 2018

Reverse-engineering survey of the source PDF (duoc-thu-quoc-gia-viet-nam-2018.pdf, 1668 pages) to catalog every distinct page/content type BEFORE deciding what parser modules to build. Classification only — nothing here changes the parsing pipeline. Purpose: give real numbers to decide which content types are common enough to deserve a dedicated pipeline stage, per the "leverage the existing pipeline + add supplementary handling" direction agreed with the user (not a full architecture rewrite).

Method, per this project's standing rules (feedback-rigorous-validation, ADR 0003): every count below is a whole-document scan (all 1668 pages, not a sample), classification rules are stated explicitly so any number can be independently re-checked, and every non-trivial claim is cross-checked with a second tool (opendataloader-pdf, the tool ADR 0003 validated for this purpose — not pdfplumber, which ADR 0003 already found scrambles reading order on this document) and/or a rendered-page-image visual read.

Reproducible script: ingestion/scratch/document_profile_group1.py (investigation code per CLAUDE.md's rules — temporary, not imported by production code; delete once this doc + any resulting regression fixtures fully capture its findings).

Note on page numbering: all page numbers below are physical/0-indexed (PyMuPDF convention). A PDF viewer's page counter is 1-indexed: viewer page N == physical page N-1.

Group 1 — objectively measurable (done, verified)

Category	Rule	Count	Verification
2-column	page has both `column="left"` and `column="right"` spans (ADR 0003 bbox ranges)	1628	rule-based, matches known monograph-body layout
Mixed/other layout	page has a set of column tags not matching the other 3 buckets	32	100% manually viewed (rendered every page) — see breakdown below, zero anomalies
Full-width only	only `column="full_width"` spans	5	pages 3, 5, 37, 97, 1497 — all print-layout blank/divider-adjacent pages
No text extracted	zero spans on the page	2	pages 99, 1666
Single-column-side	only `left` or only `right`, no `full_width`	1	page 1495 — near-empty (1 span), boundary page right at the monograph range end (1496)
Near-empty (<20 chars)	`doc[p].get_text().strip()` length	7	pages 3, 5, 37, 99, 1495, 1497, 1666 — all print-layout blank/separator pages, consistent with ADR 0003's earlier finding of 6 (this scan found 1 more, page 5, confirmed same nature by direct read)
Embedded images	`doc[p].get_images(full=True)` non-empty	0	2 independent scans, 2 sessions, same result — zero scanned pages in this document, no OCR needed
Chemical reaction equations (confirmed)	manual read of every regex candidate's context	2	see "Formula/notation" below — corrected from an initial loose-regex count of 25
Ion/electrolyte notation (Na+, Ca2+, Cl-, etc.)	same regex, reclassified after context read	~23 pages (of the 25 original candidates)	common prose notation, not a "formula" needing special parsing — but subscript/superscript preservation matters, see below
Comparison-operator notation (ADR frequency thresholds, "ADR > 1/100")	regex: digit adjacent to `<`/`>`	933	this is a standard template pattern, not an outlier — appears in the "Tác dụng không mong muốn (ADR)" section of most monographs, flagged by the user directly from a real page (Zolpidem, physical page 1494)

Mixed/other layout — full breakdown (32/32 pages viewed)

None are parsing anomalies. All are legitimate non-monograph content:

Front-matter title/cover/copyright pages: 0, 1, 2
Foreword: 6
Committee/personnel roster (name lists, 2-column but different geometry than monograph body): 7, 9, 10, 11
Table of contents: 8
"Danh mục các chuyên luận thuốc" — Vietnamese|English drug-name reference table, 2-column but different bbox geometry than the monograph body column rule (hence not tagged two_column): 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31 (18 pages; pages 20 and 23 of this same table happened to match the monograph-body bbox rule and are already counted under two_column)
"Ký hiệu chữ viết tắt" — abbreviation table, 3 columns (abbreviation | English | Vietnamese): 33
Part-divider title pages: 36 ("CÁC CHUYÊN LUẬN CHUNG"), 98 ("CÁC CHUYÊN LUẬN THUỐC"), 1496 ("CÁC PHỤ LỤC"), 1528 ("MỤC LỤC TRA CỨU")
Blank separator: 1529
Colophon (print/publisher info): 1667

Potentially useful finding for future scope: the Vietnamese|English name table (18-20 pages) could seed a synonym/alias table for search, if that's ever wanted — currently out of scope, noted only.

Formula/notation — corrected finding

An initial loose regex found 25 candidate pages. Reading the actual context of every match (cross-checked with opendataloader-pdf, not just PyMuPDF) showed this was the wrong classification — most matches are ion/electrolyte charge notation (Na⁺, K⁺, Ca²⁺, Cl⁻, Mg²⁺, Fe²⁺/Fe³⁺, HCO₃⁻, PO₄³⁻, NH₄⁺), which is common, ordinary prose notation throughout the pharmacology text, not a distinct "formula" content type. Two unrelated +-adjacent patterns were also caught by the same regex and are semantically different again: "CD4+" (immunology cell-marker notation, not a chemical charge) and "O2 + N2O" (anesthetic gas mixture percentages).

Only 2 pages have a genuine chemical reaction equation:

Physical page 1033 (already known, outlier-catalog item 16): cyanide antidote mechanism, Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3 — the reaction arrow extracts as a Private-Use-Area glyph (U+F0AF), not standard Unicode.
Physical page 1027 (new finding this session, printed page 1028, "Natri bicarbonat"): buffer equation HCO₃⁻ + H⁺ → H₂CO₃ → CO₂ + H₂O, confirmed by rendering the page to an image — the source PDF renders this with real visual subscript/superscript.

Real cross-cutting issue found, not yet sized or fixed: both PyMuPDF's and opendataloader-pdf's plain-text extraction flatten subscript/ superscript formatting — the bicarbonate equation extracts as flat text ("HCO-3+ H+ ... H2CO3 ... CO2 + H2O", digits inline, no vertical positioning info kept in the text string alone, though bbox/font-size data for the small subscript run is still recoverable from raw spans if a future stage needs to reconstruct it). This affects ion notation too, and likely also formula-adjacent abbreviations like "CD4", "Ca²⁺", "vitamin B₂/B₆/B₁₂" site-wide, not just these 2 pages — the true scope of subscript/superscript loss has not been measured yet, only observed on this one confirmed page.

Mathematical formulas — separate from chemistry, found after the user

asked "what about math" (this profile initially only scanned for chemistry- shaped tokens and missed this category entirely — a real gap, not a deliberate scope decision)

Whole-document regex scan for math symbols (full 1668 pages), initially run with PyMuPDF only — caught by the user re-checking my methodology ("đừng dùng 1 con pymu" — don't rely on just one tool) — then re-verified against opendataloader-pdf's independent whole-document text extraction (125s for all 1668 pages):

Symbol	Meaning	Pages found (PyMuPDF)	Total occurrences: PyMuPDF	Total occurrences: opendataloader-pdf
`±`	mean ± SD	44	95	95 ✅
`≤`	less-than-or-equal (dosing/lab thresholds)	91	178	178 ✅
`≥`	greater-than-or-equal (dosing/lab thresholds)	144	244	245 (off by 1, unexplained, not chased further — negligible vs. the total)
`×`	multiplication	19	50	50 ✅
`√`, `÷`	square root, division	0	0	0 ✅

Two independent tools agree almost exactly (only the ≥ total differs, by 1 out of 245) — real cross-tool evidence the symbol counts aren't a single-tool artifact, not just an assertion.

≤/≥ join the already-found </> (933 pages) as further evidence that threshold/comparison notation is a pervasive, standard part of this book's dosing and lab-value template, not a rare outlier — same conclusion as before, now with more symbols confirmed.

× (19 pages) was individually context-checked (not just counted) — splits into two real, different things:

9 pages use × only as dosing-frequency shorthand ("200 mg × 1 lần/ngày" = "200mg, once a day") or scientific notation ("18 × 10⁶") — not a standalone formula: pages 61, 91, 153, 155, 516, 716, 794, 974, 1412.
10 pages have genuine standalone calculation formulas (variable = expression), found in the general-chapters section (printed 37-98, physical ~36-97) and one appendix: pages 43, 92, 94, 147, 206, 699, 853, 1274, 1359, 1498. Examples: Cockcroft-Gault creatinine clearance (Clcr(nam) = (140-tuổi)×thể trọng / (Ccr×72)), MDRD GFR (GFR(nam) = 186 × (Ccr)^-1,154 × (tuổi)^-0,203), the DuBois body-surface-area formula (S = W^0.425 × H^0.725 × 71.84, physical page 1498, Appendix 1), elimination half-life (t½ = 0,693×Vd/Cl), clearance (Cl = Q×E).

Severe finding, confirmed visually, worse than the subscript-flattening issue above: physical pages 43 and 94 (printed 44, 95 — "Sử dụng thuốc ở người suy giảm chức năng gan, thận" and the pharmacokinetics general chapter) were rendered to images and read directly. The PDF itself shows clean, properly typeset stacked fractions (numerator over denominator, e.g. Cl_TP = D/AUC, t½ = 0,693×Vd/Cl). But the plain-text extraction of these same formulas comes out scrambled, not just subscript-flattened — e.g. page 94's Cl = Q × E = (Ca-Cv)/Ca extracts as the fragment sequence "Cl = Q × E = | a | v | a | C | C | C | Q | − | × |", unreadable and not recoverable by a simple flatten-subscript fix. This is a genuine reading- order defect specific to stacked-fraction layout, distinct from (and more severe than) the subscript-loss issue, confirmed on 2 pages so far — not yet measured across all 10 real-formula pages, only these 2 were rendered and read.

Scope honesty: the ×/±/≤/≥ regex families are still just candidate signals for "this page has notable math content" — a formula using only / for a fraction, or only superscript exponents with no × at all, would not be caught by this scan. The 10-page "genuine formula" count should be read as a lower bound, not a confirmed total.

This also confirms a bigger open gap: both real formulas and real data tables (Bảng 3, Bảng 4 — bordered tables with rows/columns, seen on page 43 during the visual check) live in the general chapters section (printed 37-98), which per project-medical-chatbot-status memory has "never been structurally investigated." Group 2 below must cover this range, not just the monograph body.

Group 2 — heading / table / list types

Tables — in progress, NOT yet a trustworthy number

opendataloader-pdf's JSON output (whole-document, converted in 99s) has built-in structural typing (heading/table/list/paragraph/caption), so this was tried first instead of hand-writing a table detector.

Indexing pitfall caught before it became a wrong report: opendataloader's page number field is 1-indexed (confirmed via the RIBOFLAVIN reference point — its title lands at page number: 1244, and this document's physical(0-indexed)+1 == printed page always coincide, per ADR 0003's confirmed constant +1 offset — so page number - 1 == PyMuPDF physical page). An initial table-count query used the raw page number value unconverted and produced a count that only coincidentally matched a "2 tables" ground-truth check by luck — re-verified correctly afterward: physical page 43 (page number 44) shows 2 tables with captions "Bảng 3. Phân loại mức độ suy thận theo creatinin..." and "Bảng 4: ...tốc độ lọc cầu thận (GFR)" — an exact match to the page rendered and read directly earlier in this investigation.

Current whole-document numbers from opendataloader-pdf alone (converted to physical 0-indexed pages):

170 table elements, on 129 distinct pages.
107 of those pages are inside the monograph range (98-1495 physical); 22 are in the general-chapters range (physical 42-92, i.e. printed 43-93); none found yet in the appendices range beyond page 1498 and 1509.

This count is NOT yet trustworthy as a final number — it comes from a single tool, spot-checked correct on only 1 of 129 pages so far. Per ADR 0003, opendataloader's higher-level structural classifier (confirmed inconsistent for headings specifically) has an unknown reliability for tables specifically. Cross-checking now with pdfplumber's find_tables()/extract_tables() — the tool ADR 0003 explicitly kept around only for table extraction (unlike its general text extraction, which is confirmed broken on this document) — whole-document run in progress, slower than opendataloader's, not complete as of this entry. Do not cite the 170/129 numbers above as confirmed until this second tool's results are compared.

Headings, lists — not started

Requires proposing a taxonomy from real samples (per the "propose first, user reviews" approach agreed for this doc), since unlike Group 1's layout checks there's no purely objective rule to classify these — pending. The opendataloader JSON also has heading (3165) and list (1624) element counts whole-document, but per the table-count lesson above these should not be quoted as real numbers until cross-checked the same way.

Known gaps in this profile itself

Comparison-operator (933 pages) and ion-notation (~23 pages) candidates were pattern-matched but not each individually opened — the sample checks done (Zolpidem page for comparison-operators, all formula-regex contexts for ion notation) are consistent enough to trust the category, but a page-by-page audit of all 933/23 was not performed.
No table detection exists yet in this profile (Group 2 will need to define a table-detection rule before it can be counted). Confirmed real bordered tables exist at least on physical page 43 ("Bảng 3", "Bảng 4" — suy thận classification), found incidentally while visually checking a math formula, not from a deliberate table search.
General chapters (37-98 printed) and appendices (1497-1528 printed) have only been surveyed for Group 1's layout/blank/image/formula/math dimensions here — their own internal structure (headings, lists, full table inventory within those sections) is still unsurveyed. This range is now confirmed to contain real formulas and real tables (see Math section above), so it must be explicitly in scope for Group 2, not treated as monograph-adjacent filler.

15 KiB Raw Blame History Unescape Escape