VuQuangBao/duocthu
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Fix every real lint finding and drop degenerate splice fragments

Browse Source
This commit is contained in:
BaoVu2k4
2026-08-01 13:51:38 +07:00
parent 967b917001
commit 834d9e51b0
69 changed files with 10119 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/adr/0004-chunking-strategy.md
+130
View File
@@ -0,0 +1,130 @@
# ADR 0004: Chunking strategy for drug monographs — validated against real per-section measurements
## Status
Accepted for the monograph range (printed pp. 99-1496) only. General
chapters (pp. 37-98) and appendices (pp. 1497-1528) are explicitly out of
scope — see Consequences.
## Context
`docs/architecture.md`'s original "Chunking" paragraph specified `(drug,
section)` as the chunk unit, a ~500-800 token budget, and a 400-token/
50-overlap sliding window for oversized sections. Those numbers were written
before segmentation existed — a plausible guess, never checked against real
per-section text length.
Phase 1 (extract → segment → validate) is now real, tested code producing
682 real monographs from the full 1668-page source PDF. This session ran
`python -m ingestion.cli run` for real and measured actual per-section
length across the whole corpus with a temporary investigation script
(`ingestion/scratch/chunking_stats_survey.py`, deleted after this ADR
captured its findings, per this project's investigation-script rule) —
something that had never been measured before this ADR.
## What was actually measured (whole corpus, 682 monographs)
- Sections per monograph: min 11, median 17, max 19 (of ~18-19 known
section keys in `segment/vocab.py`'s open taxonomy).
- Whole-monograph length: median 11,480 chars, p90 19,068 chars, max 38,786
chars.
- Per-section length, converted to a **chars/4 token estimate — an
estimate, not a real tokenizer count**:
- Most of the ~18 section types sit comfortably under 800 estimated
tokens even at their p90 (e.g. `chi_dinh` p90≈268 tok, `dang_thuoc_va_
ham_luong` p90≈115 tok, `tac_dung_khong_mong_muon` p90≈481 tok).
- **Two sections routinely exceed 800 tokens**:
`duoc_ly_va_co_che_tac_dung` (242 of 678 monographs that have this
section, 35.7%, max ≈3542 tok) and `lieu_luong_va_cach_dung` (200 of
675, 29.6%, max ≈3631 tok).
- A smaller tail also exceeds it: `than_trong` (25/680, 3.7%),
`tuong_tac_thuoc` (22/642, 3.4%).
- This means: the original 800-token ceiling is directionally correct
(it clears ~16 of 18 section types at their p90 with room to spare),
but "sub-chunk in that case" is not a rare hedge as originally implied
— it is the routine path for roughly a third of all monographs, on two
specific, named, high-clinical-importance sections (mechanism of
action and dosing).
**A separate, blocking bug was found while gathering this data, not fixed
by this ADR** (out of scope — belongs to `extract`/`segment`, owned by a
parallel session at the time of writing): running header/footer
boilerplate ("DTQGVN 2" + page number + repeated drug name, tagged
`column="full_width"` in `extract/spans.py`) is never filtered out of
section body text before it reaches `SectionSpan.text`. Measured:
1,374 of 11,409 sections (12.0%) contain a literal "DTQGVN" string
mid-text; 671 of 682 monographs (98.4%) have at least one affected section
(e.g. MORPHIN SULFAT's `lieu_luong_va_cach_dung`: `"...Nếu\nDTQGVN 2\n1009\n
Morphin sulfat\nuống viên thuốc..."`). This is `docs/pdf-parsing-outlier-
catalog.md` item 13's known risk, measured whole-corpus for the first time
here. **Chunking must not run against real data until this is fixed** —
otherwise boilerplate is baked into embeddings and can surface mid-sentence
in a chunk shown to a doctor or pharmacist.
## Decision
1. **Chunk unit stays `(drug_id, section_key)`** — matches
`segment/models.py`'s existing `Monograph.sections: Dict[str,
SectionSpan]`, matches how a doctor/pharmacist would query ("what does
it say about liều dùng"), and lets a citation point at one clinical
section rather than a whole 2,000-19,000-char monograph.
2. **Token budget: keep the 800-token ceiling** (chars/4 estimate) as the
split trigger. Below it, a section is one chunk, verbatim. This is now a
validated choice, not a guess.
3. **Sub-chunking only applies to the long-tail sections above** (~30-36%
of monographs for the two named sections, a few percent for the rest).
Method: **sentence-boundary-aware sliding window**, replacing the
originally-guessed fixed-character window. Target ~600-700 tokens per
sub-chunk (headroom under the 800 ceiling), ~1 sentence / 50-80 token
overlap between adjacent sub-chunks. Split only at a sentence boundary
(`.`/`;`/`:` followed by whitespace + capital letter), explicitly not
treating a Vietnamese decimal comma (e.g. "0,425") as a boundary.
4. **Why sentence-aware, not line- or character-based**: `assembler.py`
joins `body_lines` one line per PyMuPDF *span*, i.e. one PDF visual
line-wrap point — not a semantic paragraph or sentence boundary. A blind
character/line window can split a sentence mid-way. This is a real,
measured risk here, not theoretical: outlier-catalog item 17 found
adult/child dosing splits ("Người lớn"/"Trẻ em") appear on 1,121 of
~1,400 monograph-range pages — a chunk boundary landing inside one of
those sentences would be a patient-safety-relevant defect, not a
cosmetic one.
5. **Chunk metadata / provenance** (extends the existing `drug_name,
section_type, source_page_range, chunk_id` list in `docs/architecture.md`
— per CLAUDE.md's provenance rule): `chunk_id`
(`{drug_id}__{section_key}__{part_index}`), `drug_id`, `drug_name`,
`section_key`, `section_display_name`, `atc_codes` (inherited from the
monograph — enables ATC-class-filtered retrieval), `source_page_range`
(monograph-level, see Consequences), `part_index`/`part_count` (`0`/`1`
for un-split sections, keeps the schema uniform across all chunks).
## Consequences
- **Scope**: this decision covers the monograph range only. General
chapters and appendices contain real tables and 2D stacked-fraction
formulas (`docs/document-profile.md`, investigation in progress as of
this ADR) that need their own structural survey before any chunking rule
can be designed for them — do not extend this ADR's rules to those ranges
without a fresh investigation.
- **Hard prerequisite**: the boilerplate-leakage bug described above must
be fixed in `extract`/`segment` before this chunking design is run
against real data for ingestion. This ADR does not fix it.
- **Known gap — sub-compound tagging inside class-level monographs**: 25.5%
of the corpus has more than one ATC code per monograph (outlier item
12a), e.g. "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ" documents dosing for 7
different analogues inside one `lieu_luong_va_cach_dung` section. No
reliable structural signal was found in sampled text to split a section
by sub-compound — a chunk from this section is tagged with the class
name only, not the specific analogue a query might target. Deferred to
golden-dataset-driven eval rather than guessed at now.
- **Known gap — sub-chunk page precision**: `source_page_range` is
monograph-level, not sub-chunk-exact. A sub-chunk from late in a
multi-page section inherits the whole monograph's page range rather than
its own precise page, because per-line page tracking doesn't currently
exist in `SectionSpan`/`Heading`. The monograph + section-heading page is
still real, checkable provenance, but this is a known precision gap, not
full sub-chunk traceability. Flagged as a future improvement.
- **Not yet built**: the Vietnamese sentence-boundary splitter itself
(abbreviation handling, decimal-comma handling, ATC-code-period handling)
is specified here as a rule, not implemented or unit-tested. Building and
testing it is a separate, later task (`ingestion/ingestion/chunk/`, which
does not exist yet).
docs/adr/0005-segment-output-contract-for-chunking.md
+216
View File
@@ -0,0 +1,216 @@
# ADR 0005: `segment/` output contract needed by `chunk/` — structure-preserving, not flattened
## Status
Proposed. **Contract/schema only — no implementation.** `segment/models.py`,
`segment/assembler.py`, and `segment/io.py` are actively owned by a parallel
session on the same checkout at the time of writing; this ADR specifies what
`chunk/` needs from `segment/`'s output precisely enough to implement and
test, but does not touch those files itself. Supersedes part of ADR 0004
(see "Relationship to ADR 0004" below) — ADR 0004's `(drug_id, section_key)`
chunk-unit-as-leaf assumption is corrected here to `(drug_id, section_key)`
as a **parent**, with sentence-window splitting demoted from primary
strategy to fallback.
## Context
ADR 0004 designed chunking against `segment/models.py`'s current output:
`SectionSpan.text` is a single flattened string per section (`"\n".join(
body_line.strip() for ...)`), with all per-line style (`Span.bold`) and
per-line page position discarded once the string is built (confirmed by
reading `assembler.py`: `body_lines.append(span.text.strip())` keeps only
`span.text`, nothing else). Review of ADR 0004 surfaced four real problems
that trace back to this flattening, not to the chunking algorithm itself:
1. **A section is not a single semantic unit.** `liều lượng và cách dùng`
and `tương tác thuốc` routinely contain multiple distinct facts (dosing
per patient population, dosing per organ-function impairment, multiple
separate drug interactions) that a doctor may want to retrieve
independently. Measured: an explicit population marker ("Người lớn"/
"Trẻ em"/"Trẻ sơ sinh"/"Suy thận"/"Suy gan" immediately followed by `:`
or `.`) appears in **303 of 675 monographs (44.9%)** that have a `liều
lượng và cách dùng` section — this is common, not an edge case.
2. **A blind sentence-boundary sliding window (ADR 0004's original
sub-chunking method) can still split two different facts into the same
chunk, or split one fact across two chunks**, because it has no way to
know a population/interaction boundary exists — that information exists
in the source (as a bold or otherwise visually distinct sub-heading, per
direct reading of MORPHIN SULFAT/VITAMIN D section text: lines like
"Thuốc uống", "Cách dùng:" render as isolated bold short lines in the
PDF) but is discarded before `chunk/` ever sees it.
3. **Tables inside the monograph range are not addressed at all.** ADR 0004
implicitly assumed monograph-range sections are prose. `docs/pdf-parsing-
outlier-catalog.md` item 19 already documents a real table (dosing by
renal function, HSV/CMV columns) inside a monograph body (Foscarnet
natri, physical page 698) — flattening a table's rows into
newline-joined body text destroys its row/column structure exactly the
way outlier item 7 already describes for the appendix's 2D nomogram
table. A whole-range survey to size this properly is in progress
alongside this ADR (see "Not yet resolved" below).
4. **Provenance is section-level, not chunk-level**, because per-line
`physical_page`/`y0` (which `Span` already carries — see
`extract/models.py`) is discarded at the same flattening point. For a
section spanning several physical pages, a sub-chunk built from its
final third currently has no way to know its own real page — it can
only inherit the whole monograph's `source_page_range`. For medical
citations this is not precise enough.
**A fifth, independently-found data-quality bug makes precise provenance
even more necessary, not less**: the corpus's last-processed monograph
(ZOLPIDEM) is never closed until true end-of-stream, and `assembler._classify`
calls `match_section()`/`match_section_with_inline_value()` on every span
with **no `in_monograph_range` gate** (unlike `_TextEvent` handling, which
does check it). A spurious bold-text match on physical page 1655 — deep in
the back-of-book "Mục lục tra cứu" brand-name index, confirmed by reading
that page directly — overwrote ZOLPIDEM's real `tương tác thuốc`
`SectionSpan` with an empty one and corrupted its `source_page_range` to
`[1492, 1655]`. This is real content loss (measured: exactly 1 monograph
affected, the last one processed — every other monograph is closed on
schedule by the next monograph title, which *is* range-gated). Flagged for
the session that owns `extract`/`segment`, not fixed here.
## Decision
Extend `segment/models.py`'s `SectionSpan` with a structured, line-level
representation, additive to (not replacing) the existing flat `text` field
— `chunk/` becomes a real, structure-aware consumer instead of re-deriving
structure from a flattened string via ad hoc regex.
### New/changed types (`segment/models.py`)
```python
@dataclass(frozen=True)
class BodyLine:
text: str
physical_page: int
y0: float
bold: bool # Span.bold, preserved instead of discarded
@dataclass
class SectionSpan:
key: str
display_name: str
heading: Heading
text: str # UNCHANGED meaning, kept for
# backward compat (see invariant below)
lines: List[BodyLine] = field(default_factory=list) # NEW
```
`lines` carries exactly the per-line signal `chunk/` needs to do its own
job (population/subheading detection, precise page provenance) without
`segment/` having to know anything about chunking — `segment/`'s
responsibility stays "detect boundaries and preserve source structure," not
"decide what a retrieval unit is" (Clean Architecture / SoC, per
CLAUDE.md). Specifically, this is deliberately **not** a `is_subheading:
bool` field computed by `segment/` — classifying "is this line a
subheading a chunker should split on" is a chunking-time decision (what
counts as a good split point can vary by strategy/eval results), not a
segmentation-time one. `segment/` should stop discarding the raw signal
(`bold`, `y0`, `physical_page`) it already has per span; it should not also
start doing chunk-shaping judgment calls.
### Invariants
1. `text == "\n".join(l.text for l in lines).strip()` for every
`SectionSpan`, for the lifetime of this contract — `lines` is a strictly
additive refinement, never a divergent second source of truth. Any
change to how body text is assembled (e.g. the boilerplate-stripping fix
already applied by the other session) must update both fields from the
same filtered span list, not `text` alone.
2. `lines` is in reading order, matching the order `text`'s lines already
implicitly have.
3. Every `BodyLine.physical_page` satisfies `detector.in_monograph_range`
for a `Span` on that page — i.e., **no line in any `SectionSpan.lines`
may come from outside the monograph's real printed-page range**. This is
the ZOLPIDEM bug's exact failure mode stated as an invariant: it was
violated (a spurious section event was accepted from a fully
out-of-range page precisely because no such check existed for section
*events*, only for body *text* events). Enforcing this invariant closes
that bug as a side effect, but the invariant is stated here as a
contract requirement independent of any specific fix implementation.
4. Every currently-open monograph must be finalized exactly once, at either
(a) the next monograph title, or (b) true end-of-stream — with no third
path (e.g., a stray out-of-range section match) able to silently mutate
an already-"complete" monograph's sections after point (a) would
otherwise have applied. (This is a restatement of invariant 3 from the
monograph-lifecycle side, not a new requirement.)
### Migration impact
- **`segment/io.py`** (`_monograph_to_dict`/`_monograph_from_dict`,
`write_monographs_jsonl`/`read_monographs_jsonl`): additive — serialize
`lines` alongside the existing `text`/`heading` fields per section.
Existing consumers reading only `text` (e.g. `segment/atc.py`'s
`extract_atc_codes`, which regexes over `SectionSpan.text`) need no
change, per invariant 1.
- **`ingestion/data/processed/monographs.jsonl`**: schema grows a new
optional-shaped field (`sections[key].lines`). No `schema_version` field
currently exists in the serialized dict (checked `io.py` directly) —
worth adding as part of this change, both for this migration and because
`docs/architecture.md` already assumes "collection aliasing allows
re-ingesting with a changed chunking strategy," which implies the
ingestion output itself should be able to declare which schema shape it
is.
- **Existing 110 tests**: unaffected if invariant 1 holds — no assertion in
the current suite inspects `lines` (it doesn't exist yet), and `text`'s
value/semantics are unchanged.
- **New tests required** (this ADR specifies them; implementation and the
actual test code are not part of this ADR):
1. Regression test reproducing the ZOLPIDEM failure shape: a synthetic
span stream — last monograph's title and real sections, followed by
spans whose `printed_page` is out of `in_monograph_range` but whose
text matches a `vocab.py` section label — asserting the monograph
closes with its real sections intact and the out-of-range spurious
match is ignored, not accepted.
2. `SectionSpan.lines` fixture test: using the real MORPHIN SULFAT
boilerplate-fix fixture already in `tests/test_segment_assembler.py`,
assert `lines` preserves the correct `bold`/`physical_page`/`y0` per
retained line (and that stripped boilerplate lines are absent from
`lines` too, not just from `text`).
3. Round-trip test: `write_monographs_jsonl` → `read_monographs_jsonl`
preserves `lines` exactly (dataclass equality per line).
4. Whole-corpus invariant-1 check: for a real `cli run` output, assert
`text == "\n".join(l.text for l in lines).strip()` holds for every
section of every monograph, not a sample.
## Relationship to ADR 0004
ADR 0004's chunk-unit decision (`(drug_id, section_key)`) is **not**
discarded — a section is still the natural *parent* grouping (matches how a
clinician thinks, matches `Monograph.sections`). What changes: ADR 0004
described a section as directly *the* chunk when under the 800-token
ceiling, with sentence-window splitting as the fallback for oversized
sections. Per the review above, splitting must instead **first** attempt to
break at real structural boundaries available in `SectionSpan.lines` (a
bold, short, isolated line — the same "subheading" shape already visually
confirmed for route-of-administration/population sub-headers — or an
explicit population/organ-function marker), with the sentence-window method
demoted to a fallback for the remaining prose that has no such marker. The
exact splitting algorithm (how a "subheading-shaped line" is defined
precisely, in code) is a `chunk/`-side implementation detail *enabled* by
this contract, not decided by it.
## Not yet resolved (explicitly out of scope for this ADR)
- **Table/formula content blocks.** A separate whole-monograph-range survey
(pdfplumber `find_tables()` + PyMuPDF math-symbol scan, physical pages
98-1494 excluding blank page 99 — the exact set `detector.
in_monograph_range` accepts, not an assumed offset) is in progress at the
time of writing, per explicit user instruction to measure before deciding
a table/formula chunk-unit strategy. This ADR's `BodyLine`
contract covers **text content only**; a table/formula region should
*not* currently be flattened into `BodyLine`s (doing so would repeat
exactly the "destroys row/column meaning" mistake outlier item 7 already
documents) — but the precise `ContentBlock`/table-row/formula-unit shape
is deferred to a follow-up revision of this ADR once the survey reports
real numbers (how many monographs/sections affected, page-break
continuation frequency, multi-tier headers, merged cells, footnotes).
- **Paragraph-boundary detection** (grouping consecutive `BodyLine`s into a
flowing paragraph vs. a new one) is left to `chunk/`, using the same
kind of y-gap heuristic `segment/merge.py` already validates for
multi-line title wraps (`_MAX_LINE_GAP_PT`) — `BodyLine.y0` is sufficient
raw signal for `chunk/` to compute this itself; `segment/` does not need
to pre-compute paragraph grouping.
- **The actual `chunk/` splitting implementation** (subheading detector,
population-marker regex, sentence-window fallback) is not part of this
ADR — this ADR defines the data contract that implementation will consume.
docs/adr/0006-quarantined-block-references-in-chunks.md
+171
View File
@@ -0,0 +1,171 @@
# ADR 0006: chunks must carry references to lifted table/formula blocks
## Status
Proposed, with implementation to follow immediately. Resolves the item ADR
0005 explicitly deferred ("Table/formula content blocks … the precise
`ContentBlock`/table-row/formula-unit shape is deferred to a follow-up
revision of this ADR once the survey reports real numbers"). The survey has
reported.
## Context
`segment/` now lifts table and formula regions out of section prose and
quarantines them (ADR 0003 lineage, outlier-catalog items 7, 8, 24, 25).
That was the right move — linearised, AMPICILIN VÀ SULBACTAM's
Cockcroft-Gault fraction read as `Clcr (ml/phút) = 72 x creatinin huyết
thanh`, i.e. a division presented as a multiplication, in a renal-dosing
section.
But `chunk/models.py` has no field that refers to a lifted block. Measured on
the current whole-corpus output:
| quantity | value |
|---|---|
| lifted blocks | 167, all quarantined |
| monographs affected | 96 of 683 (**14.1%**) |
| sections affected | 108 |
| **blocks in `lieu_luong_va_cach_dung`** | **127 (76%)** |
| next largest section | `duoc_ly_va_co_che_tac_dung`, 16 |
| shapes | simple_table 136, multi_level_or_merged_header 16, formula_2d 14, cross_page_continuation 1 |
So three quarters of everything removed from prose was removed from the
dosing section, in a drug formulary, for an audience of doctors and
pharmacists.
**The failure this creates is silent, not visible.** A chunk of AMPICILIN VÀ
SULBACTAM's `lieu_luong_va_cach_dung` is grammatical, complete-looking prose
with the renal-dosing table absent and nothing marking the absence. Retrieval
ranks it, the model answers from it, and neither has any way to know a table
was taken out. A visible error would be safer than this.
A second, quieter failure: a table is currently **unreachable**. Nothing in
the index represents it, so "bảng liều theo chức năng thận của ampicilin"
cannot retrieve it even in principle.
## Decision
Chunks reference blocks; blocks' content never becomes embedded text.
### 1. `Chunk` gains typed attachments
```python
@dataclass(frozen=True)
class ChunkAttachment:
block_id: str
kind: str # "table" | "formula"
shape: str # simple_table | multi_level_or_merged_header |
# cross_page_continuation | formula_2d
physical_page: int
bbox: List[float]
quarantined: bool
header_row: List[str] = () # simple_table only; see caveat below
@dataclass(frozen=True)
class Chunk:
...
chunk_kind: str = "prose" # "prose" | "block_descriptor"
attachments: List[ChunkAttachment] = ()
has_quarantined_content: bool = False
```
`has_quarantined_content` is derivable from `attachments`, and is serialized
anyway. A consumer that never looks at `attachments` must still be unable to
miss the fact — the whole defect being fixed here is a consumer not knowing
what it was not told.
### 2. One descriptor chunk per block, built from metadata only
A block also gets its own chunk so it is retrievable at all:
```
chunk_id = "{drug_id}:{section_key}:block:{block_id}"
chunk_kind = "block_descriptor"
text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
trang in 204. Cột: Độ thanh thải creatinin | Nửa đời |
Liều ampicilin/sulbactam."
```
The text is assembled from the drug name, the section display name, the kind,
the printed page and — for `simple_table` only — the header row. **No cell
value ever appears.** A header row is a row of labels; linearising it cannot
invent a numeric relationship, which is precisely what linearising a body row
does. For every other shape the header is omitted, because
`multi_level_or_merged_header` is the shape whose header extraction is least
trustworthy.
Caveat recorded in the schema itself: `header_row` comes from
`pdfplumber.find_tables()`'s first row and has **not** been verified by eye
(the 180 real tables' individual shapes are rule-derived; only the 20
"not a table" verdicts were visually confirmed). It is retrieval bait, never
an answer.
### 3. The answer layer's obligations (binding on `ai-service`)
These are stated here because they are the reason the schema exists; they are
not implemented by `ingestion/`.
1. A retrieved chunk with `has_quarantined_content: true` **must** cause the
answer to state that a table or formula exists at the cited page, and to
surface its rendered crop. The answer may not present itself as complete.
2. A `block_descriptor` chunk may be answered **only** with the crop. It must
never be paraphrased, and its `header_row` must never be presented as the
table's content.
3. No chunk carrying a quarantined attachment may be used to state a numeric
dose. If the dose is in the table, the answer is the crop plus the page.
### 4. `schema_version`
`monographs.jsonl` and the chunk output both gain `schema_version`. ADR 0005
flagged its absence; a schema that now has two chunk kinds and an attachment
list cannot be safely consumed without one.
## Alternatives rejected
- **Flatten the block into the chunk text.** This is the defect, not the fix
— it reproduces `Clcr = 72 x creatinin` exactly.
- **Chunk the block's linearised text as an ordinary chunk.** Worse than
flattening: it makes unsafe text independently retrievable *as prose*, with
its quarantine flag one dereference away from being ignored.
- **Drop the blocks.** Silent loss, and contrary to the standing rule that
unreconstructable content is quarantined with full provenance, never
deleted.
- **Rely on the prose saying "xem bảng".** The prose often does not, and a
retrieval layer cannot act on an unstructured hint.
- **Wait for row/column reconstruction and do this once.** Reconstruction is
days of work and would leave the corpus unchunkable meanwhile; worse, it
would make the schema question look answered when the *silent-incompleteness*
problem is independent of whether the rows are recovered. Reconstruction
later populates `rows` on the same attachment without touching consumers.
## Why a crop is a legitimate answer, not a placeholder
For doctors and pharmacists a rendered crop of the source page is the
highest-fidelity response available: it *is* the book, and it is verifiable at
a glance. Reconstruction earns its keep for a different job — comparing or
combining values across drugs, which is the synthesis use case this product
exists for — not for single-table lookup.
## Invariants and gates
Added to `cli chunk-ready` and to the chunk stage's own tests:
1. `section_with_lifted_block_but_no_chunk_reference = 0`
2. `attachment_block_id_unknown = 0` — every referenced id exists on the
monograph
3. `attachment_without_page_or_bbox = 0`
4. `block_text_leaked_into_chunk_text = 0` — no chunk's embedded text
contains a quarantined block's text
5. `descriptor_chunk_count == block_count`
6. `descriptor_chunk_without_attachment = 0`
## Consequences
- Prose chunks shrink slightly in trustworthiness terms but grow in honesty:
the ones missing a table now say so.
- The index gains 167 descriptor chunks (≈1.4% of the expected chunk count),
each cheap and none carrying unsafe text.
- `ai-service` cannot be built to answer a dosing question from prose alone
for the 108 affected sections without violating a stated contract.
- The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas
answerable as crops today, which they are not now.
docs/architecture.md
+25 -6
View File
@@ -80,12 +80,31 @@ methodology, cross-tool comparison, and validation numbers.
`ingestion/data/processed/monographs.jsonl` and validated both
automatically (see ADR 0003) and via manual spot-check in
`ingestion/notebooks/`.
3. **Chunking**: each `(drug, section)` pair is the natural chunk unit;
never split a section unless it exceeds a token budget (~500-800 tokens),
in which case sub-chunk with a sliding window (400 tokens, 50 overlap),
tagging the same drug+section metadata plus `part_index`. Every chunk
carries `drug_name`, `section_type`, `source_page_range`, `chunk_id` as
Qdrant payload — this is what makes citations possible.
3. **Chunking** (monograph range only, pp. 99-1496 — see
`docs/adr/0004-chunking-strategy.md` for the full measured rationale):
each `(drug_id, section_key)` pair is the chunk unit; a section stays one
chunk if it's under an **800-token ceiling** (chars/4 estimate — a
validated line, not a guess: whole-corpus measurement across 682
monographs shows ~16 of 18 section types clear it comfortably at their
p90). Two sections routinely exceed it — `dược lý và cơ chế tác dụng`
(35.7% of monographs that have it) and `liều lượng và cách dùng`
(29.6%) — sub-chunking is the **routine** path for those two, not a rare
edge case. Oversized sections are split with a **sentence-boundary-aware
sliding window** (~600-700 tokens/sub-chunk, ~1 sentence/50-80 token
overlap), never a blind character/line window — PDF line-wrap points
are not safe cut points, and a mid-sentence split risks separating an
adult/child dosing instruction (a measured, common pattern — outlier
catalog item 17) into two chunks. Every chunk carries `chunk_id`,
`drug_id`, `drug_name`, `section_key`, `section_display_name`,
`atc_codes`, `source_page_range`, `part_index`/`part_count` as Qdrant
payload — this is what makes citations possible. **Known open gaps**
(see ADR 0004): sub-compound tagging inside class-level/multi-ATC
monographs (25.5% of the corpus) is not yet solved; `source_page_range`
is monograph-level, not sub-chunk-exact; chunking for general chapters/
appendices is a separate, not-yet-designed task; a confirmed
header/footer-boilerplate leak into section text (98.4% of monographs
affected) must be fixed upstream before this design runs against real
data.
4. **Embedding + load**: OpenAI `text-embedding-3-small` in batches, upserted
into a versioned Qdrant collection (`drug_monographs_v1`) keyed by
`chunk_id` for idempotent re-runs; collection aliasing allows re-ingesting
docs/document-profile.md
+230
View File
@@ -0,0 +1,230 @@
# Document Profile — Dược thư quốc gia Việt Nam 2018
Reverse-engineering survey of the source PDF (`duoc-thu-quoc-gia-viet-nam-2018.pdf`,
1668 pages) to catalog every distinct page/content type BEFORE deciding what
parser modules to build. **Classification only — nothing here changes the
parsing pipeline.** Purpose: give real numbers to decide which content types
are common enough to deserve a dedicated pipeline stage, per the "leverage
the existing pipeline + add supplementary handling" direction agreed with
the user (not a full architecture rewrite).
Method, per this project's standing rules ([[feedback-rigorous-validation]],
ADR 0003): every count below is a **whole-document** scan (all 1668 pages,
not a sample), classification rules are stated explicitly so any number can
be independently re-checked, and every non-trivial claim is cross-checked
with a second tool (`opendataloader-pdf`, the tool ADR 0003 validated for
this purpose — **not** `pdfplumber`, which ADR 0003 already found scrambles
reading order on this document) and/or a rendered-page-image visual read.
Reproducible script: `ingestion/scratch/document_profile_group1.py`
(investigation code per CLAUDE.md's rules — temporary, not imported by
production code; delete once this doc + any resulting regression fixtures
fully capture its findings).
**Note on page numbering**: all page numbers below are physical/0-indexed
(PyMuPDF convention). A PDF viewer's page counter is 1-indexed:
`viewer page N == physical page N-1`.
## Group 1 — objectively measurable (done, verified)
| Category | Rule | Count | Verification |
|---|---|---|---|
| 2-column | page has both `column="left"` and `column="right"` spans (ADR 0003 bbox ranges) | 1628 | rule-based, matches known monograph-body layout |
| Mixed/other layout | page has a set of column tags not matching the other 3 buckets | 32 | **100% manually viewed** (rendered every page) — see breakdown below, zero anomalies |
| Full-width only | only `column="full_width"` spans | 5 | pages 3, 5, 37, 97, 1497 — all print-layout blank/divider-adjacent pages |
| No text extracted | zero spans on the page | 2 | pages 99, 1666 |
| Single-column-side | only `left` or only `right`, no `full_width` | 1 | page 1495 — near-empty (1 span), boundary page right at the monograph range end (1496) |
| Near-empty (<20 chars) | `doc[p].get_text().strip()` length | 7 | pages 3, 5, 37, 99, 1495, 1497, 1666 — all print-layout blank/separator pages, consistent with ADR 0003's earlier finding of 6 (this scan found 1 more, page 5, confirmed same nature by direct read) |
| Embedded images | `doc[p].get_images(full=True)` non-empty | 0 | 2 independent scans, 2 sessions, same result — **zero scanned pages in this document, no OCR needed** |
| Chemical reaction equations (confirmed) | manual read of every regex candidate's context | **2** | see "Formula/notation" below — corrected from an initial loose-regex count of 25 |
| Ion/electrolyte notation (Na+, Ca2+, Cl-, etc.) | same regex, reclassified after context read | ~23 pages (of the 25 original candidates) | common prose notation, not a "formula" needing special parsing — but subscript/superscript preservation matters, see below |
| Comparison-operator notation (ADR frequency thresholds, "ADR > 1/100") | regex: digit adjacent to `<`/`>` | **933** | this is a **standard template pattern**, not an outlier — appears in the "Tác dụng không mong muốn (ADR)" section of most monographs, flagged by the user directly from a real page (Zolpidem, physical page 1494) |
### Mixed/other layout — full breakdown (32/32 pages viewed)
None are parsing anomalies. All are legitimate non-monograph content:
- **Front-matter title/cover/copyright pages**: 0, 1, 2
- **Foreword**: 6
- **Committee/personnel roster** (name lists, 2-column but different geometry than monograph body): 7, 9, 10, 11
- **Table of contents**: 8
- **"Danh mục các chuyên luận thuốc"** — Vietnamese\|English drug-name reference table, 2-column but different bbox geometry than the monograph body column rule (hence not tagged `two_column`): 12, 13, 14, 15, 16, 17, 18, 19, 21, 22, 24, 25, 26, 27, 28, 29, 30, 31 (18 pages; pages 20 and 23 of this same table happened to match the monograph-body bbox rule and are already counted under `two_column`)
- **"Ký hiệu chữ viết tắt"** — abbreviation table, 3 columns (abbreviation \| English \| Vietnamese): 33
- **Part-divider title pages**: 36 ("CÁC CHUYÊN LUẬN CHUNG"), 98 ("CÁC CHUYÊN LUẬN THUỐC"), 1496 ("CÁC PHỤ LỤC"), 1528 ("MỤC LỤC TRA CỨU")
- **Blank separator**: 1529
- **Colophon (print/publisher info)**: 1667
Potentially useful finding for future scope: the Vietnamese\|English name table
(18-20 pages) could seed a synonym/alias table for search, if that's ever
wanted — currently out of scope, noted only.
### Formula/notation — corrected finding
An initial loose regex found 25 candidate pages. **Reading the actual context
of every match (cross-checked with `opendataloader-pdf`, not just PyMuPDF)
showed this was the wrong classification** — most matches are ion/electrolyte
charge notation (Na⁺, K⁺, Ca²⁺, Cl⁻, Mg²⁺, Fe²⁺/Fe³⁺, HCO₃⁻, PO₄³⁻, NH₄⁺),
which is common, ordinary prose notation throughout the pharmacology text,
not a distinct "formula" content type. Two unrelated `+`-adjacent patterns
were also caught by the same regex and are semantically different again:
"CD4+" (immunology cell-marker notation, not a chemical charge) and
"O2 + N2O" (anesthetic gas mixture percentages).
**Only 2 pages have a genuine chemical reaction equation:**
1. Physical page 1033 (already known, outlier-catalog item 16): cyanide
antidote mechanism, `Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3` — the reaction arrow
extracts as a Private-Use-Area glyph (U+F0AF), not standard Unicode.
2. Physical page 1027 (**new finding this session**, printed page 1028,
"Natri bicarbonat"): buffer equation `HCO₃⁻ + H⁺ → H₂CO₃ → CO₂ + H₂O`,
confirmed by rendering the page to an image — the source PDF renders
this with real visual subscript/superscript.
**Real cross-cutting issue found, not yet sized or fixed**: both PyMuPDF's
and `opendataloader-pdf`'s plain-text extraction **flatten subscript/
superscript formatting** — the bicarbonate equation extracts as flat text
("HCO-3+ H+ ... H2CO3 ... CO2 + H2O", digits inline, no vertical
positioning info kept in the text string alone, though bbox/font-size data
for the small subscript run is still recoverable from raw spans if a future
stage needs to reconstruct it). This affects ion notation too, and likely
also formula-adjacent abbreviations like "CD4", "Ca²⁺", "vitamin B₂/B₆/B₁₂"
site-wide, not just these 2 pages — **the true scope of subscript/superscript
loss has not been measured yet**, only observed on this one confirmed page.
### Mathematical formulas — separate from chemistry, found after the user
asked "what about math" (this profile initially only scanned for chemistry-
shaped tokens and missed this category entirely — a real gap, not a
deliberate scope decision)
Whole-document regex scan for math symbols (full 1668 pages), initially run
with PyMuPDF only — **caught by the user re-checking my methodology**
("đừng dùng 1 con pymu" — don't rely on just one tool) — then re-verified
against `opendataloader-pdf`'s independent whole-document text extraction
(125s for all 1668 pages):
| Symbol | Meaning | Pages found (PyMuPDF) | Total occurrences: PyMuPDF | Total occurrences: opendataloader-pdf |
|---|---|---|---|---|
| `±` | mean ± SD | 44 | 95 | 95 ✅ |
| `≤` | less-than-or-equal (dosing/lab thresholds) | 91 | 178 | 178 ✅ |
| `≥` | greater-than-or-equal (dosing/lab thresholds) | 144 | 244 | 245 (off by 1, unexplained, not chased further — negligible vs. the total) |
| `×` | multiplication | 19 | 50 | 50 ✅ |
| `√`, `÷` | square root, division | 0 | 0 | 0 ✅ |
Two independent tools agree almost exactly (only the `≥` total differs, by
1 out of 245) — real cross-tool evidence the symbol counts aren't a
single-tool artifact, not just an assertion.
`≤`/`≥` join the already-found `<`/`>` (933 pages) as further evidence that
**threshold/comparison notation is a pervasive, standard part of this book's
dosing and lab-value template**, not a rare outlier — same conclusion as
before, now with more symbols confirmed.
**`×` (19 pages) was individually context-checked (not just counted)** —
splits into two real, different things:
- **9 pages** use `×` only as dosing-frequency shorthand ("200 mg × 1
lần/ngày" = "200mg, once a day") or scientific notation ("18 × 10⁶")
— not a standalone formula: pages 61, 91, 153, 155, 516, 716, 794, 974, 1412.
- **10 pages have genuine standalone calculation formulas** (variable =
expression), found in the general-chapters section (printed 37-98,
physical ~36-97) and one appendix: pages 43, 92, 94, 147, 206, 699, 853,
1274, 1359, 1498. Examples: Cockcroft-Gault creatinine clearance
(`Clcr(nam) = (140-tuổi)×thể trọng / (Ccr×72)`), MDRD GFR (`GFR(nam) =
186 × (Ccr)^-1,154 × (tuổi)^-0,203`), the DuBois body-surface-area formula
(`S = W^0.425 × H^0.725 × 71.84`, physical page 1498, Appendix 1),
elimination half-life (`t½ = 0,693×Vd/Cl`), clearance (`Cl = Q×E`).
**Severe finding, confirmed visually, worse than the subscript-flattening
issue above**: physical pages 43 and 94 (printed 44, 95 — "Sử dụng thuốc ở
người suy giảm chức năng gan, thận" and the pharmacokinetics general
chapter) were rendered to images and read directly. The PDF itself shows
clean, properly typeset **stacked fractions** (numerator over denominator,
e.g. `Cl_TP = D/AUC`, `t½ = 0,693×Vd/Cl`). But the plain-text extraction of
these same formulas comes out **scrambled, not just subscript-flattened** —
e.g. page 94's `Cl = Q × E = (Ca-Cv)/Ca` extracts as the fragment sequence
`"Cl = Q × E = | a | v | a | C | C | C | Q | − | × |"`, unreadable and not
recoverable by a simple flatten-subscript fix. This is a genuine reading-
order defect specific to stacked-fraction layout, distinct from (and more
severe than) the subscript-loss issue, confirmed on 2 pages so far — **not
yet measured across all 10 real-formula pages**, only these 2 were rendered
and read.
**Scope honesty**: the `×`/`±`/`≤`/`≥` regex families are still just
*candidate* signals for "this page has notable math content" — a formula
using only `/` for a fraction, or only superscript exponents with no `×` at
all, would not be caught by this scan. The 10-page "genuine formula" count
should be read as a lower bound, not a confirmed total.
**This also confirms a bigger open gap**: both real formulas and real data
tables (Bảng 3, Bảng 4 — bordered tables with rows/columns, seen on page 43
during the visual check) live in the **general chapters section (printed
37-98)**, which per [[project-medical-chatbot-status]] memory has "never
been structurally investigated." Group 2 below must cover this range, not
just the monograph body.
## Group 2 — heading / table / list types
### Tables — in progress, NOT yet a trustworthy number
`opendataloader-pdf`'s JSON output (whole-document, converted in 99s) has
built-in structural typing (`heading`/`table`/`list`/`paragraph`/`caption`),
so this was tried first instead of hand-writing a table detector.
**Indexing pitfall caught before it became a wrong report**: opendataloader's
`page number` field is **1-indexed** (confirmed via the RIBOFLAVIN reference
point — its title lands at `page number: 1244`, and this document's
physical(0-indexed)+1 == printed page always coincide, per ADR 0003's
confirmed constant +1 offset — so `page number - 1 == PyMuPDF physical
page`). An initial table-count query used the raw `page number` value
unconverted and produced a count that only *coincidentally* matched a
"2 tables" ground-truth check by luck — re-verified correctly afterward:
physical page 43 (`page number 44`) shows 2 tables with captions "Bảng 3.
Phân loại mức độ suy thận theo creatinin..." and "Bảng 4: ...tốc độ lọc cầu
thận (GFR)" — an exact match to the page rendered and read directly
earlier in this investigation.
**Current whole-document numbers from opendataloader-pdf alone (converted
to physical 0-indexed pages)**:
- 170 table elements, on 129 distinct pages.
- 107 of those pages are inside the monograph range (98-1495 physical); 22
are in the general-chapters range (physical 42-92, i.e. printed 43-93);
none found yet in the appendices range beyond page 1498 and 1509.
**This count is NOT yet trustworthy as a final number** — it comes from a
single tool, spot-checked correct on only 1 of 129 pages so far. Per
ADR 0003, opendataloader's higher-level structural classifier (confirmed
inconsistent for headings specifically) has an unknown reliability for
tables specifically. Cross-checking now with `pdfplumber`'s
`find_tables()`/`extract_tables()` — the tool ADR 0003 explicitly kept
around *only* for table extraction (unlike its general text extraction,
which is confirmed broken on this document) — whole-document run in
progress, slower than opendataloader's, not complete as of this entry.
**Do not cite the 170/129 numbers above as confirmed until this second
tool's results are compared.**
### Headings, lists — not started
Requires proposing a taxonomy from real samples (per the "propose first,
user reviews" approach agreed for this doc), since unlike Group 1's layout
checks there's no purely objective rule to classify these — pending. The
opendataloader JSON also has `heading` (3165) and `list` (1624) element
counts whole-document, but per the table-count lesson above these should
not be quoted as real numbers until cross-checked the same way.
## Known gaps in this profile itself
- Comparison-operator (933 pages) and ion-notation (~23 pages) candidates
were pattern-matched but not each individually opened — the sample checks
done (Zolpidem page for comparison-operators, all formula-regex contexts
for ion notation) are consistent enough to trust the *category*, but a
page-by-page audit of all 933/23 was not performed.
- No table detection exists yet in this profile (Group 2 will need to define
a table-detection rule before it can be counted). Confirmed real bordered
tables exist at least on physical page 43 ("Bảng 3", "Bảng 4" — suy thận
classification), found incidentally while visually checking a math
formula, not from a deliberate table search.
- General chapters (37-98 printed) and appendices (1497-1528 printed) have
only been surveyed for Group 1's layout/blank/image/formula/math
dimensions here — their own internal structure (headings, lists, full
table inventory within those sections) is still unsurveyed. This range
is now confirmed to contain real formulas and real tables (see Math
section above), so it must be explicitly in scope for Group 2, not
treated as monograph-adjacent filler.
docs/full-coverage-parsing-plan.md
+116
View File
@@ -0,0 +1,116 @@
# Kế hoạch phủ toàn bộ nội dung PDF (text + bảng + công thức + outlier)
**Trạng thái**: kế hoạch đang thực thi, lập 2026-07-31. Các ô ghi `[chờ đo]`
là số liệu chưa có tại thời điểm viết — không được trích dẫn cho đến khi
điền bằng kết quả chạy thật.
## Mục tiêu, phát biểu chính xác
Có hai mục tiêu thường bị gộp làm một. Kế hoạch này chỉ nhận mục tiêu A cho
cuối ngày, và phát biểu rõ B là việc dài hơn.
| | Mục tiêu | Nhận cho cuối ngày? |
|---|---|---|
| **A** | **Phủ toàn bộ, không mất âm thầm**: mọi ký tự trong 1668 trang đều rơi vào đúng một rổ đầu ra hoặc vào rổ `unassigned` đếm được; mọi đối tượng không đáng tin đều bị gắn cờ tường minh; provenance giữ nguyên | **Có** |
| **B** | **Đúng 100% đã chứng minh**: mọi bảng và công thức đã đối chiếu ground truth | **Không** — cần đối chiếu thủ công toàn bộ, là công người, không phải công máy |
Tuyên bố "parse được toàn bộ" chỉ hợp lệ theo nghĩa A. Bất kỳ báo cáo nào
cũng phải nói rõ đang nói về A hay B.
## Vì sao không xây một bộ reconstruct tổng quát
Chưa biết trong sách có bao nhiêu bảng, bao nhiêu dạng cấu trúc, bao nhiêu
trang continuation. Xây một bộ tổng quát trước khi biết phân bố dạng là đầu
tư mù. Thứ tự bắt buộc: **kiểm kê → phân loại dạng → chọn đường xử lý theo
từng dạng → mới code**.
## Giai đoạn
### A. Kiểm kê toàn corpus (đang chạy)
Script tạm `ingestion/scratch/inventory_tables_formulas.py`, scope toàn bộ
1668 trang, xuất provenance từng đối tượng để soi lại được.
| Đại lượng | Kết quả |
|---|---|
| Số bảng pdfplumber tìm được / số trang có bảng | `[chờ đo]` |
| Phân bố số cột | `[chờ đo]` |
| Ứng viên continuation (bảng ở đầu trang/cột, không header) | `[chờ đo]` |
| Lưới toàn số ≥4 cột (ứng viên 2D lookup, catalog item 7) | `[chờ đo]` |
| Ứng viên công thức: fraction_bar / PUA / small_font_numeric | `[chờ đo]` |
Kiểm kê này **cố tình thiên về recall**: bắt thừa còn hơn bỏ sót; độ chính
xác đo sau bằng kiểm tra trực quan.
### B. Sổ cái phủ ký tự — đây là eval chứng minh "trích xuất được"
Với mỗi trang trong 1668 trang, đối chiếu:
```
chars_trên_trang_gốc == chars_vào_section_text
+ chars_vào_ô_bảng
+ chars_vào_vùng_công_thức
+ chars_vào_front_matter / phụ lục
+ chars_unassigned
```
`unassigned` phải ra **một con số cụ thể kèm danh sách trang/bbox**, không
phải một lời khẳng định. Đây là điểm khác biệt so với mọi eval trước đó
trong dự án: recall/precision hiện tại chỉ đo **phát hiện ranh giới chuyên
luận**, không đo nội dung; sổ cái này đo nội dung ở mức ký tự, whole-document,
không phải mẫu.
Giới hạn phải nói rõ: sổ cái chứng minh **không mất**, không chứng minh
**đúng thứ tự** hay **đúng ngữ nghĩa**. Thứ tự đã có kiểm tra riêng
(`scan_reading_order`, `scan_glyph_order`); ngữ nghĩa thuộc mục tiêu B.
### C. Định tuyến theo dạng, mỗi dạng một đường
| Dạng | Xử lý | Metadata bắt buộc |
|---|---|---|
| Bảng có kẻ khung, header dạng chữ | Trích ô thật | `table_id`, `row`, `col`, `page`, `bbox` |
| Bảng ngắt trang/cột (catalog item 5-6) | Gắn lại header gốc vào phần tiếp | thêm `continues_from` |
| Lưới toàn số 2D (item 7) | **Không** chunk thành text | `do_not_cite: true` + giữ công thức đi kèm |
| Công thức 1D (mũ inline) | Giữ nguyên text | `formula_kind: "1d"` |
| Công thức 2D (có fraction bar) | Gắn cờ, giữ bbox + ảnh crop | `needs_review: true` |
| Ký tự PUA (item: mũi tên lỗi) | Bảng thay thế tường minh | `pua_substituted` |
Mở/đóng theo SOLID: thêm một dạng mới = thêm một entry định tuyến, không
sửa code đang chạy.
### D. Vùng ngoài chuyên luận
General chapters (tr. 37-98) và phụ lục (tr. 1497-1528) hiện **nằm ngoài
phạm vi hoàn toàn** — pipeline chỉ sinh 682 chuyên luận. Hai vùng này phải
hoặc vào sổ cái phủ, hoặc bị loại trừ tường minh kèm con số ký tự bị loại.
Không được im lặng bỏ qua.
### E. Artifact bằng chứng
Mỗi đối tượng bị gắn cờ sinh một ảnh crop theo bbox đặt cạnh text trích ra,
để mọi tuyên bố eval soi tận mắt được. Tự đọc ảnh để kiểm chứng, không đẩy
việc kiểm tra sang người dùng.
## Số đo cần báo riêng, không gộp
Theo yêu cầu tránh gộp chỉ số che lấp điểm yếu:
- **detection recall** của detector trên golden set — bắt được bao nhiêu %
đối tượng thật
- **false positive** — bắt nhầm bao nhiêu
- **số đối tượng chưa phân loại** — bao nhiêu cái detector không biết xếp vào
đâu
- **structural accuracy** — bảng tái tạo đúng hàng/cột bao nhiêu %
- **semantic fidelity** — nội dung ô đúng bao nhiêu %
Detector dựa trên bbox là **heuristic**: nó tìm ứng viên, không chứng minh
đã bắt hết mọi phân số, chỉ số, căn, ma trận hay lưới 2D. Mọi báo cáo phải
đi kèm ba số đầu, không được nói suông "detector hoạt động tốt".
## Nợ kỹ thuật đã biết, chưa xử lý
- Ground truth từ Mục lục tra cứu **chưa được làm sạch**: chứa entry tham
chiếu chéo lặp (ví dụ `"- CoA reductase, 285"` xuất hiện hơn 10 lần trong
danh sách unmatched). Mẫu số 1064 hiện tại vì thế không đáng tin để chốt;
ADR 0003 dùng mẫu số 725 nên hai lần đo **không so sánh trực tiếp được**.
- Nội dung text chuyên luận chưa từng được đo độ chính xác so với nguồn.
docs/pdf-parsing-outlier-catalog.md
+404 -24
View File
@@ -166,11 +166,14 @@ unreliable, always flag them") or its opposite ("formulas extract fine, no
special handling needed") — neither is true here. The determining factor is
whether the formula's visual layout is fundamentally 1D (left-to-right,
like an inline exponent) or 2D (a fraction, a matrix, stacked terms).
**Check:** no cheap automatic detector was built for this distinction yet —
treat any equation/formula-like content as a manual-review candidate,
especially anything with a fraction bar, until a real detector exists
(e.g. checking for large vertical bbox gaps between adjacent glyphs that
should be visually stacked).
**Check:** a detector now exists — `residual_ink.py`'s
`fraction_bar_candidate`, which finds the bar as ink no extracted span
accounts for. Measured on this book: **precision 16/23 = 69.6%** (the misses
are decorative underlines and table borders), recall unknown, and it is blind
by construction to a fraction printed without a bar (item 25). Its output is
therefore a review queue, not a verdict: all 23 candidates were rendered and
read one at a time before any was acted on, and only the confirmed ones went
into `ingestion/data/verified/formula_regions_2d.json`.
**Generalizes:** yes — any technical/medical/scientific PDF with inline
math will have this exact split; don't assume all formulas behave the same
way in extraction.
@@ -179,25 +182,65 @@ way in extraction.
## Character/glyph-level risks
### 9. Rare reversed (right-to-left) glyph-order defect
**What it looks like:** confirmed exactly once across the entire
1668-page book (physical page 1373): one short text run's glyphs are
positioned in **descending x-order** rather than ascending, producing
scrambled output (e.g. `" = tịx 8 yàgn gnàh uềil gnổt(..."`) that reverses
character-by-character back to the correct Vietnamese sentence
(`"(4 xịt = 800 microgam) vào buổi chiều..."`).
**Why it matters:** this is a genuine, confirmed data-corruption risk, not
theoretical — but it's also extremely rare (1 occurrence in 1668 pages), so
it must be *detected*, not assumed to be either absent or common.
**Check:** group text fragments into visual rows by rounded y-coordinate,
then check whether x-coordinates are non-decreasing across the row; flag
(and optionally auto-correct by re-sorting on x) any row that isn't. This
full-book check runs in about 20 seconds.
### 9. Rare reversed/misordered glyph defects — corrected count: 2, not 1
**What it looks like:** re-implemented as real, tested production code
(`ingestion/ingestion/extract/glyph_order.py`) rather than trusted from the
earlier exploratory script's claim. Found **two distinct shapes**, not the
one originally reported:
1. **Within-span character reversal** (physical page 1373, the originally
reported case): one span's glyphs are positioned in descending x-order,
producing `" = tịx 8 yàgn gnàh uềil gnổt(..."`, which reverses
character-by-character back to `"(4 xịt = 800 microgam) vào buổi
chiều..."`.
2. **Cross-fragment row misordering, newly found** (physical page 714): a
single visual row is split by PyMuPDF into multiple `line` objects
*within one block* that are then emitted out of left-to-right order —
each fragment's own characters are fine, but concatenating fragments in
extraction order produces `"...bảo quản ộđ tệihn "` instead of the
correct `"...bảo quản nhiệt độ "`. This is a different underlying shape
from item 1 (multiple mis-ordered fragments, not one reversed span) and
was missed by the original narrower (within-span-only) check — the
ADR 0003 claim of "exactly 1 occurrence in the whole book" undercounted
the real defect population; corrected here.
**Getting a trustworthy count took three detector iterations** (documented
in the module's own docstring) — the first naive whole-book implementation
of the row-level check reported **1113** "issues," almost all false
positives from two mechanisms: (a) ordinary font-kerning jitter (e.g. in
"mefloquin," two adjacent glyphs differ by 0.095pt — normal kerning, not a
defect) treated as a reversal with no decrease-tolerance, which then
actively *corrupted* correct text into "mefolquin"; and (b) reconstructing
"visual rows" from raw x/y coordinates using a hand-picked column-boundary
threshold, which misclassifies a paragraph that happens to start near the
natural column gap (confirmed real case: a right-column paragraph starting
at x=299.4 got merged with an unrelated left-column paragraph at the same
y). The fix that survived whole-book testing: group by PyMuPDF's own
`block` index (already validated in ADR 0003 to respect this document's
column structure) instead of re-deriving columns from coordinates, plus a
minimum-decrease threshold (1.0pt — safely between the ~0.3pt kerning noise
floor and the >2pt real-defect magnitude). Final whole-book result: **11
row-level issues on 5 pages** — 3 of those pages (92, 94, 805) are formula
regions already flagged as unreliable in item 8 below (2D-layout formulas
scramble on extraction; this check's "corrected" text for those rows should
**not** be trusted or auto-applied, same as item 8's existing guidance),
leaving exactly the 2 genuine prose defects above (pages 714, 1373).
**Why it matters:** both genuine defects are confirmed real data-corruption
risks, not theoretical — but both are also extremely rare (2 occurrences in
1668 pages of prose), so they must be *detected*, not assumed either absent
or common. Equally important: a naive implementation of "the obvious check"
can itself introduce false positives and even actively corrupt correct
text — this detector's own false-positive history is as important a lesson
as the defects it catches.
**Check:** `ingestion.extract.scan_glyph_order` (within-span) and
`ingestion.extract.scan_reading_order` (cross-fragment, grouped by real
PyMuPDF block index + row y, with a 1.0pt minimum-decrease threshold and
header-band exclusion). Both run in seconds over the full book.
**Generalizes:** yes, directly — this is a cheap, universal sanity check
worth running on any PDF text-extraction pipeline as a standing QA gate,
regardless of source document, since it catches a class of PDF-authoring
defects (RTL/BiDi overrides, corrupted content streams) that have nothing
to do with this book specifically.
regardless of source document. The false-positive history also generalizes:
any "reconstruct visual rows from raw coordinates" approach needs a
decrease-tolerance (font kerning is universal) and should prefer the
source tool's own layout-analysis groupings (blocks/lines) over hand-picked
coordinate thresholds wherever available.
---
@@ -424,12 +467,349 @@ rather than treating them as failures.
---
### 15. No embedded images anywhere in the book — measured, not assumed
**What it looks like:** a whole-book scan of `page.get_images(full=True)` across
all 1668 pages returns **zero** embedded raster/vector images, confirmed via
PyMuPDF's own image extraction API (not just "the text doesn't mention an
image").
**Why it matters:** avoids over-investing in image/caption validation tooling
for a corpus that has no images to validate — but this must be a measured
fact, not an assumption from the book's general description as "text-heavy."
**Generalizes:** the check (`get_images(full=True)` summed over every page)
is a cheap one-line whole-document verification worth running on any PDF
before deciding whether image-handling code is needed at all.
### 16. Chemical reaction arrows render as Private-Use-Area glyphs, not Unicode arrows
**What it looks like:** confirmed real example — physical page 1033 contains a
genuine chemical reaction equation (`Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3`, part of
the cyanide-antidote/rhodanese mechanism description). The reaction arrow
extracts as a Private-Use-Area codepoint (``), not a standard Unicode
arrow (`→`) — the source PDF's font maps a custom symbol glyph (likely from a
symbol/wingdings-style embedded font) into a PUA slot, and raw text extraction
faithfully returns that codepoint rather than a human-readable arrow.
**Why it matters:** any pipeline that treats extracted text as directly
human-readable/citable will surface a mangled or invisible character where a
reaction arrow should be; a naive keyword/embedding step over raw text would
either silently drop it (if PUA codepoints get filtered as junk) or leave a
confusing tofu/box character in a chunk shown to a doctor or pharmacist.
**Check:** scan extracted text for codepoints in the Unicode Private Use Area
ranges (`U+E000–U+F8FF`) — cheap and generalizes to any custom-glyph symbol
substitution, not just arrows.
**Handling:** for now, flag any monograph/section containing a PUA codepoint
for manual review or map known PUA codepoints (e.g. this book's `` →
`→`) via an explicit substitution table; do not pass raw PUA codepoints
through to chunking/embedding untranslated.
**Generalizes:** yes — any PDF built from print-authoring software that uses a
symbol font for arrows/special glyphs (common in scientific/medical/chemistry
documents) can exhibit this; always check for PUA codepoints in extracted
text as a standing sanity check, not just assume standard Unicode symbols.
**Confirmed real chemical formula in the corpus, but rare:** a regex scan for
molecular-formula-shaped tokens (`[A-Z][a-z]?\d{1,3}` repeated) across the
monograph page range found 9 raw hits; manual inspection found most are
**false positives** (`H5N1` = flu strain name, `P2Y12` = a receptor name, not
molecular formulas) and only one confirmed genuine chemical formula/equation
(the Na2S2O3 case above) — real chemical notation exists in this corpus but
is genuinely rare, not a systemic pattern requiring a general chemistry
parser.
### 17. Adult/child dosing-population splits are the norm, not an edge case
**What it looks like:** measured via a whole-monograph-range text scan for
"Người lớn"/"Trẻ em"/"Trẻ sơ sinh" (adult/child/newborn) — these terms appear
on **1121 of ~1400** monograph-range pages, i.e. the large majority of drug
monographs split dosing by patient population.
**Why it matters:** this is exactly the kind of structural content where a
segmentation/chunking bug that interleaves or merges adjacent subsections
(e.g. a table/list continuation bug, see items 5-6) would be a genuine
patient-safety risk, not just a data-quality nicety — mixing an adult dose
into a child-dose chunk (or vice versa) is a plausible, concrete failure
mode given how common this structure is.
**Handling:** treat "does this monograph's dosing section correctly keep
adult/child/newborn subsections un-interleaved" as a standing validation
check (not a rare-case afterthought), given the measured prevalence.
**Generalizes:** yes — any clinical/pharmacological reference document
organized with population-specific subsections has this same risk profile;
measure real prevalence before deciding how much validation effort a
structural risk deserves (same methodology lesson as item 12a).
### 18. A monograph title can legitimately repeat — disambiguated by a bold, non-caps qualifier line
**What it looks like:** confirmed real example, found while smoke-testing
the real `segment/detector.py` against the full book: "SALBUTAMOL" is
detected as a monograph title **twice** (physical pages 1261 and 1263).
Rendering both pages to images and reading them directly (not inferred from
coordinates) confirmed these are two genuinely different, complete
monographs — "SALBUTAMOL (Dùng trong hô hấp)" (respiratory use) and
"SALBUTAMOL (Dùng trong sản khoa)" (obstetric/tocolytic use) — each with
its own full 18-section template. The qualifier ("(Dùng trong hô hấp)" /
"(Dùng trong sản khoa)") is a bold line immediately below the all-caps
title, but is **not itself all-caps** (mixed case inside the parens), so it
is correctly excluded from `detect_monograph_titles`'s all-caps candidate
filter — it must instead be captured as a *separate* signal and folded into
the monograph's disambiguating identity downstream.
**Why it matters:** an assembler that derives `drug_id` from the title text
alone (e.g. a simple slug of "SALBUTAMOL") will produce a real collision
between two legitimately different monographs — this is **not** the same
failure mode as the already-fixed GONADOTROPIN false-collision (that one
was a detector artifact from unmerged multi-line wrapping; this one is a
genuine same-name-different-monograph case that must be preserved, not
merged away).
**Handling (for Phase 1.3's assembler):** after detecting a monograph title,
check for an immediately-following bold, parenthesized, non-all-caps line
directly below it (same page, small y-gap) and include it in `drug_id`
generation when present, so "salbutamol_ho_hap" and "salbutamol_san_khoa"
remain distinct rather than colliding as "salbutamol" twice. The
`assembler.py` duplicate-drug_id check (outlier-catalog reasoning already
established: raise on a genuine duplicate rather than silently overwriting)
must be designed with this real case in mind, or it will incorrectly reject
a legitimate second "SALBUTAMOL" entry.
**Generalizes:** yes — any drug/entity reference work that documents the
same base substance under multiple distinct use-contexts (formulation,
indication, route) can have this exact pattern; never assume a title string
alone is a unique key without checking for a disambiguating qualifier line.
### 19. Table column headers can be bold + all-caps + short — identical shape to a real title
**What it looks like:** confirmed real example, found via a whole-book
`assemble()` run raising a duplicate-drug_id error: "HSV" and "CMV" each
appear twice as bold, all-caps, short (3-char) spans on physical page 698 —
not drug names at all, but **column headers in a dosing-by-renal-function
table** inside the "Foscarnet natri" monograph ("Liều đối với HSV / HSV /
CMV / CMV"). Bold+all-caps+short is exactly the monograph-title signal
(item 10/12d), so this is a genuine detector ambiguity, not a coding bug.
**Why it matters:** unlike item 12d's part-divider titles (a small,
enumerable, fixed set of known strings), a table's column headers are
unbounded and content-dependent (any future table could use "HSV", "CMV",
or something else entirely as a header) — an exclusion list approach
doesn't generalize here the way it did for part-dividers.
**Handling:** require a **structural anchor** rather than a text exclusion
list: a real monograph title is always followed shortly by at least one
recognized section heading from the vocabulary (in practice, always "Tên
chung quốc tế" first) before the next title-shaped candidate. A
table-header false positive is not — the table's own cells are numbers/
plain text, matching no vocabulary entry. Implemented as
`assembler._filter_false_positive_titles` (lookahead of 6 events, checked
against the same coalesced event stream already built for assembly — no
separate detection pass, no duplicated logic).
**Generalizes:** yes, more broadly than item 12d — any document where
section/entity boundaries are marked by a *shape* (bold+caps+short) that a
table, list, or figure caption could coincidentally also match should
verify a **structural follow-on anchor**, not just a shape match or a
denylist of known bad strings, since the space of possible false-shaped
content (table headers, figure labels, pull-quotes) is unbounded while the
space of "what a real boundary is followed by" is small and known.
### 20. Section headings are not consistently bold across monographs — some combine label+value in one plain span
**What it looks like:** confirmed real example, found by investigating why
a whole-book `assemble()` run showed 48 monographs with zero ATC codes and
not stated-absent (far more than the ~13-14 the original spot-check
extrapolated). AMITRIPTYLIN's real "Mã ATC:" field is a **single, plain
(non-bold)** span containing the label AND value together: `"Mã ATC:
N06AA09."` — unlike Abacavir's equivalent, which is a bold `"Mã ATC: "`
label span followed by a separate plain `"J05AF06."` value span. Both
render visually similar but have completely different span/style
structure. Given the book's own foreword states it was "biên soạn bởi
nhiều tác giả" (written by many authors), this kind of per-author styling
inconsistency across ~700 individually-authored monographs is plausible
and, once checked, confirmed real — not a one-off.
**Why it matters:** a detector that requires `span.bold` to recognize a
section heading (reasonable-looking given every *title* is confirmed bold)
silently drops entire sections for a meaningful fraction of the corpus —
this directly caused undercounted ATC codes (and, structurally, would
equally affect any other section) for monographs using this looser style.
**Handling:** match section headings by **vocabulary text**, not by
boldness — the same "don't gate on a styling attribute, only content is
reliable" lesson as item 10 (font size), now applied to boldness. Also
handle the "label + value combined in one span" shape explicitly (a prefix
match: does the span start with a known label followed by ":", with the
remainder treated as the section's inline value) rather than assuming
label and value are always separate spans.
**Generalizes:** yes — any print-authored reference work assembled from
many individual authors/editors over a long production process should
expect inconsistent low-level styling of nominally-identical structural
elements; verify a structural signal (styling) against the *content* it's
supposed to correlate with, across a large real sample, before trusting it
as a universal discriminator — the same methodology lesson as item 10,
found again independently here.
### 21. "All-caps" is not 100% reliable either — and a class-level monograph's own internal sub-headings can masquerade as new monographs
**What it looks like:** two distinct confirmed real findings from the same
investigation:
1. The class-level monograph "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" embeds the
mixed-case abbreviation "CoA" (Coenzyme A) inside an otherwise all-caps
title. A strict `text.isupper()` check requires *zero* lowercase
letters, so this single embedded abbreviation caused the entire
monograph to be silently dropped from the corpus — found only by
directly checking whether this specific, previously-known (outlier item
12a) class-level monograph was present in a real whole-book `assemble()`
run, and discovering it was not.
2. Within that same class-level monograph, individual statin names
("SIMVASTATIN", "LOVASTATIN", "PRAVASTATIN", "FLUVASTATIN") appear as
their own bold+all-caps+short sub-headings, each introducing its own
"Liều lượng và cách dùng" sub-section — shape-identical to a real
monograph title, and (after fix 1 above made the loosened "any known
section" anchor check pass) briefly became a second false-positive
category alongside item 19's table headers, since these sub-headings
*are* followed by a recognized section, just never by "Tên chung quốc
tế" specifically (that section belongs only to the parent).
**Why it matters:** together these show that neither "all-caps" nor "loosen
the anchor to any section" is safe in isolation — the fix for one false
positive (item 19, HSV/CMV) reopened a different one (SIMVASTATIN) until
the anchor check was tightened back to the *specific* section the book's
own template guarantees is always first for a genuine top-level monograph.
**Handling:** `detector._is_mostly_upper` uses a **lowercase-letter ratio**
(≤10%), not an absolute count — an earlier absolute-count version (≤2
lowercase letters) let a real regression through: "Mã ATC:" has only 1
lowercase letter (a normal Vietnamese diacritic, 'ã') but that's 20% of its
5 letters, correctly rejected by the ratio while HMG-CoA's 1/27 ≈ 3.7%
correctly passes. `assembler._has_anchor_ahead`
requires specifically the "ten_chung_quoc_te" section key, not just any
recognized section, since that is the one invariant the book's documented
template actually guarantees is unique to real top-level monographs.
**Generalizes:** yes — (1) don't assume a styling/casing convention holds
with zero exceptions across an entire corpus, even one confirmed exception
matters at whole-corpus scale; (2) when a document has nested substructure
that mimics top-level structure (a class monograph containing per-item
sub-entries), the anchor used to confirm a real boundary must be the most
*specific* invariant available, not just "some known follow-on content" —
a looser check that fixes one false positive can silently reopen another.
### 22. Running-header boilerplate was never actually stripped, despite item 13's warning — measured whole-corpus at 98.4% of monographs affected
**What it looks like:** the running header at the top of every physical page
("DTQGVN 2" + printed page number + the current monograph's name, e.g.
physical page 1008's "DTQGVN 2" / "1009" / "Morphin sulfat", tagged
`column="full_width"` by `extract/spans.py`) matches no section heading and
isn't a real all-caps title, so it fell through every classification branch
in `assembler._classify` into plain body text — splicing itself into the
*middle* of whatever section is open when a physical page turns. Real
example, MORPHIN SULFAT's `liều lượng và cách dùng`: `"...Nếu\nDTQGVN 2\n
1009\nMorphin sulfat\nuống viên thuốc..."` — the header text lands inside a
real dosing sentence.
**Why it matters:** item 13 (above) already *warned* "strip the fixed
boilerplate before parsing content" back when the extraction layer was
first built, but that step was never actually implemented in `assembler.py`
— the warning existed in the catalog without a corresponding code path or
test enforcing it, and nothing caught the gap until a whole-corpus
measurement was actually run. Measured: **1,374 of 11,409 sections (12.0%)
contained a literal "DTQGVN" string mid-text; 671 of 682 monographs (98.4%)
had at least one affected section** — this is not a rare edge case, it's
the default outcome for any section whose text happens to cross a physical
page boundary (i.e. most sections longer than about half a page). Left
unfixed, boilerplate gets baked into chunks and embeddings and can surface
mid-sentence in a citation shown to a doctor/pharmacist.
**Handling:** `assembler._is_page_boilerplate` drops any span with
`column == "full_width"` and `y0 < HEADER_BAND_Y` (the same header-band
threshold `page_map.py` already uses to read the folio) before it reaches
any other classification branch. Whole-corpus re-measurement after the fix:
0 of 11,409 sections contain "DTQGVN". Regression test uses the exact real
MORPHIN SULFAT span shape.
**Generalizes:** a documented risk in this catalog is not the same as a
verified-fixed risk — "we know this could happen" needs a whole-corpus
measurement (not just a warning paragraph) before it can be crossed off,
and ideally a regression test that would fail if the fix were ever reverted.
### 23. PyMuPDF's raw block order doesn't reliably sequence left-column-before-right-column — confirmed wrong on 12 of 1398 pages
**What it looks like:** `extract/spans.py` originally trusted PyMuPDF's own
block iteration order to already emit left-column content before
right-column content, validated only against one example page during ADR
0003. On physical page 1100 (the OXYBUTYNIN/OXYMETAZOLIN monograph
boundary) and 11 other pages, PyMuPDF's raw block order emits the *right*
column first. Since `assembler.assemble` appends section content to
whichever monograph is currently open, this silently attributed
OXYMETAZOLIN's right-column sections (Chống chỉ định, Thận trọng, Thời kỳ
mang thai, Thời kỳ cho con bú, ADR, Hướng dẫn xử trí ADR, Liều lượng và
cách dùng) to the still-open OXYBUTYNIN monograph — overwriting
OXYBUTYNIN's real sections and leaving OXYMETAZOLIN missing all 7.
**Why it matters:** medically relevant — wrong contraindication/ADR content
silently attached to the wrong drug. Found via a whole-document
(1668-page) character-similarity diff against an independent parser
(`opendataloader-pdf`), not from a sample; confirmed by rendering the page
to an image and reading it directly, then confirmed again in the actual
`assemble()` output.
**Handling:** `extract.spans._sort_blocks_reading_order` explicitly sorts
each page's blocks by (full_width header band first, then left column,
then right column) and then by y-position, instead of trusting raw PyMuPDF
order. Whole-range (99-1496) re-scan after the fix: 0 pages with the
reversed-order signature (was 12). Directly verified OXYBUTYNIN's and
OXYMETAZOLIN's `assemble()`-produced sections are now distinct and
drug-appropriate.
**Generalizes:** don't trust an upstream library's element ordering just
because it happened to be correct on the one page checked during initial
validation — for a whole-corpus pipeline, explicitly sort by the actual
signal you care about (here: visual column position) rather than an
implicit "the library probably does this right" assumption.
### 24. Some text exists only as vector outlines — no text extractor can read it, and single dropped glyphs corrupt otherwise-clean sentences
**What it looks like:** physical page 714 prints 17 full lines of ordinary
GATIFLOXACIN prose that `page.get_text()` does not return, `page.search_for()`
cannot find, and neither `pdfplumber` nor `opendataloader-pdf` returns either.
`page.get_drawings()` shows why: each line is a filled path of 1,126-1,831
items, shaped exactly like one line of type and filled with the body-text
colour. The same defect occurs at glyph granularity (39-45 path items), and
that form is far more dangerous — a single Vietnamese diacritic character
drops out of a line that otherwise extracts perfectly: `Độ ổn định` extracts
as `Độ n định`, `≥ 1 tuổi` as `≥ 1 tu i`, `tại chỗ` as `tại ch `. The result
reads as ordinary text, so no structural check, no count and no cross-tool
comparison notices it.
**Why it matters:** this is silent loss of clinical prose in a drug
formulary, and it is invisible to every check that asks a text layer a
question. It survived a whole-document span-coverage ledger reporting
`unassigned = 0`, because the spans that existed were all routed correctly —
the missing content was never a span at all.
**Check:** render the page, white out every extracted span's bbox, and look
at the ink that survives (`ingestion/validation/residual_ink.py`, ~0.06
s/page). Confirm with `page.get_drawings()`: a filled path with ≥30 items
whose box is 3-20pt tall is type, not decoration (real decoration on this
book carries 1-2 items).
**Handling:** `ingestion/extract/outlined_text.py` detects the runs;
recovery cannot be automatic because the paths carry no character codes, so
each run was rendered and transcribed by reading it, into
`ingestion/data/verified/outlined_text_transcriptions.json` with page, bbox,
and the extracted line it belongs to. Whole-document scope: **51 runs on 5
pages** (714 ×31, 736 ×16, 1373, 1444, 1445 ×2), 1,116 characters.
**Generalizes:** yes — any PDF produced by a layout tool that converts
selected text to outlines (common when a font cannot be embedded) has this.
Never treat "the text layer returned something for this page" as evidence
the page was fully extracted; compare against the rendered pixels.
### 25. A fraction can be printed with no fraction bar at all, so no geometric detector can find it
**What it looks like:** ADENOSIN (physical page 147) prints its infusion-rate
formula as three plain lines — `Tốc độ truyền dịch (ml/phút) = 0,140
(mg/kg/phút) × trọng lượng cơ thể (kg)` / `Nồng độ adenosin (3 mg/ml).` —
with **no rule drawn between numerator and denominator**, confirmed by
rendering the region and reading it. Extracted linearly it reads as a
multiplication chain, i.e. the division silently disappears.
**Why it matters:** it defeats the detector that catches every other 2D
formula in this book. The fraction-bar signal (item 8, and
`residual_ink.py`'s `fraction_bar_candidate`) finds ink; there is no ink to
find here. It was caught only because a prose-leak gate matched its text.
**Check:** there is no cheap automatic check. Treat any line ending in a
unit-bearing quantity immediately followed by a line that is itself a
unit-bearing quantity as a division candidate for human review.
**Handling:** quarantined via the verified region list with
`source_prints_no_bar: true`. The count of bar-less formulas in this book is
**unmeasured** — recorded as `recall_limit` in
`ingestion/data/verified/formula_regions_2d.json` so the bar scan is never
mistaken for complete formula coverage.
**Generalizes:** yes — measured precision of the fraction-bar rule on this
book is **16/23 = 69.6%**, and its recall is unknown. A geometric heuristic
finds candidates; it never proves absence.
## Not yet investigated (flagged for future work, not silently ignored)
- **Footnote-style superscript reference markers** (seen as `a, b, c, d` in
one table) — not yet checked for whether the footnote text stays
correctly associated with its marker/row during extraction.
- **Formula detection heuristic** (item 8) — no automatic detector exists
yet to flag 2D-formula regions before they're trusted as chunk content.
- **How many bar-less formulas exist** (item 25) — one confirmed, total
unmeasured; no geometric signal can bound it.
- **Merging the 51 transcribed outlined runs back into monograph text**
(item 24) — transcribed and stored, but the corpus still contains
`Độ n định`.
- **2D grid table reconstruction** (item 7) — no implementation yet for
recovering row/column-correct values from a nomogram-style table.
- **Exact shortest monograph name+page** — a quick unmerged crude scan (no
multi-line title merge) gave a different longest-monograph ranking than
the already-documented authoritative one (item 12e: "AMOXICILIN VÀ KALI
CLAVULANAT", 45,623 chars), meaning the crude scan's numbers are not
reliable enough to name an exact shortest monograph — deferred to the real
Phase 1.2 detector (with proper multi-line merge and back-index-validated
boundaries), which will produce a trustworthy number as a side effect of
its own validation run, rather than trusting today's quick, differently-
scoped script.
docs/progress-log.md
+1581
View File
File diff suppressed because it is too large Load Diff
docs/verification-strategy.md
+208
View File
@@ -0,0 +1,208 @@
# Verification strategy — how extraction is actually measured
**Short answer to "do you compare characters?": no.** Character comparison
was tried and rejected twice, for reasons recorded below. What is used
instead is a ladder of instruments, each answering a *different* question,
each with a stated blind spot. No single number means "the parse is correct",
and this document exists so nobody later mistakes one rung for another.
Status: written 2026-08-01, after the residual-ink work. Every figure quoted
here was measured on the whole 1668-page document unless said otherwise.
---
## The rule that governs everything below
**An instrument must be checked before its output is believed.** In this
project the measuring device has been wrong before the data was, repeatedly.
Only after an instrument survives its own check does its number get quoted.
Three confirmed cases, all from 2026-08-01:
| what was nearly reported | why it was wrong |
|---|---|
| "extraction ratio 0.6656, 835 pages below 98%" | `get_texttrace()` counts glyphs painted *outside* the page rectangle — 4,717,407 of them, on pages that are visually blank |
| "ratio 0.8023, 1642 of 1668 pages below 95%" (after clipping to the page) | Vietnamese diacritics are painted as two glyphs and extracted as one character, so the deficit is systematic and meaningless |
| "page 209's ADR table is unaccounted-for ink" | the residual scan's horizontal banding merged the left and right columns, so the box's centre landed in the gutter and matched no table |
Earlier sessions add three more: a gate comparing post-merge spans against
raw spans, one ordering parts by page-y in a two-column book, and one
treating a legitimately resuming section as an ordering violation.
Corollary: **a non-zero gate is not automatically a data bug.** Check the
gate, then the data.
---
## Why not character comparison
1. **Characters cannot be balanced across normalization.** The pipeline joins
spans that share a visual line, substitutes PUA codepoints for real
glyphs, and strips separators. A character in, character out ledger cannot
close, so a mismatch tells you nothing.
2. **Glyph counts cannot stand in for characters.** See the table above —
both attempts produced confident, wrong numbers.
3. **Comparing extracted text against another extractor's text measures
agreement, not truth**, and on this document the tools share a blind spot
(§3).
What replaced it: balance at the **span** level (a unit that survives the
pipeline), and verify at the **pixel** level (a unit that owes nothing to any
extractor).
---
## Layer 1 — Span routing ledger: did every span land somewhere?
`cli coverage`. Each of the 252,733 merged spans is assigned exactly one
state and characters are aggregated from the states.
| state | spans | chars |
|---|---|---|
| normalized_text | 177,679 | 8,182,049 |
| out_of_scope | 53,374 | 897,692 |
| heading | 12,764 | 221,266 |
| boilerplate_excluded | 4,976 | 47,609 |
| quarantined | 3,937 | 48,989 |
| structural_excluded | 3 | 53 |
| **unassigned** | **0** | **0** |
**Proves:** nothing the extractor produced was dropped without a name.
**Does not prove:** that routed content survived downstream. A section-
overwrite bug was invisible to this ledger — spans were correctly marked
`normalized_text`, then their section was overwritten later.
**Does not prove:** that the extractor produced everything on the page. That
is Layer 2's job, and it is the gap that mattered most.
---
## Layer 2 — Residual ink: what is on the page that no span accounts for?
`cli residual-ink`. Render the page, white out every pixel covered by an
extracted span's bbox, measure the ink that survives, and give every
surviving region a name. Needs no ground truth, no sampling, and no second
tool. Measured cost: **0.06 s/page, all 1668 pages in under two minutes.**
| kind | regions |
|---|---|
| header_rule | 1,649 |
| text_as_vector_outline | 1,061 |
| table_frame | 959 |
| antialias_speck | 220 |
| fraction_bar_candidate | 23 |
| rule_fragment | 10 |
| header_band_fragment | 9 |
| **unclassified** | **0** |
This is the only instrument here that does not ask a text layer a question,
which is why it found what everything else missed: **51 runs of type that
exist only as vector paths** (outlier-catalog item 24), invisible to
PyMuPDF, pdfplumber and opendataloader-pdf alike.
**Proves:** every mark on all 1668 pages is accounted for by name.
**Does not prove:** that the names are right. `unclassified = 0` means every
region was *named*, not that every verdict was checked by eye. Of the seven
kinds, only `text_as_vector_outline` and `fraction_bar_candidate` were
confirmed exhaustively; the rest were confirmed on sampled examples.
**Calibration matters:** at 1.0pt of mask padding the check ate the very
fraction bars it exists to find (page 1042's bar shrank from 188.6pt to
9.1pt). 0.5pt was chosen by measurement, and a regression test pins it.
---
## Layer 3 — Cross-tool agreement: useful, and routinely over-claimed
Inside the monograph range, `pdfplumber` and `opendataloader-pdf` agree
*exactly* on where tables are: same 112 pages, same per-page count, zero
pages found by only one. That looks like strong evidence and is not.
**On physical page 1042, both report zero tables.** There is a
Cockcroft-Gault fraction on that page. Both tools need ruling lines; the bar
is a drawn line but not a table, so neither sees it. The same holds on 202.
**Rule adopted:** agreement between two tools that share a failure mode
measures *consistency*, never *recall*. Cross-tool agreement may be reported
as a reproducibility check and never as coverage evidence.
Where it is genuinely useful: opendataloader's whole-book JSON carries 141
tables / 826 rows / 2,468 cells with per-cell page, bbox, row, column and
span — a second independent source of table structure, already on disk.
---
## Layer 4 — Visual census: the only instrument that yields content verdicts
Render the region, read the image, record the verdict. This is what turns a
candidate into a fact, and it is the only layer that can say what the text
*says*.
**Census when the population is small enough to enumerate.** This is stronger
than any confidence interval, so prefer it whenever possible:
| population | size | status |
|---|---|---|
| fraction-bar candidates | 23 | **all 23 read.** 16 real, 7 not → precision **69.6%** |
| vector-outlined runs | 51 | **all 51 read and transcribed** (1,116 characters) |
| "not a table" verdicts | 20 | all 20 read (found 2 wrong) |
| table blocks | 155 | not started |
**Sampling only when a census is impossible**, and then with the arithmetic
stated. Rule of three: inspect *n* items, find **0** defects, and the 95%
upper bound on the defect rate is ≈ 3/n. So "≤ 1% error" costs **n ≥ 300 with
zero defects**; "≤ 5%" costs n ≥ 60. Any "99%" claim that cannot name its *n*
is not a measurement.
**Risk-based, not random**, when sampling: 100% of table pages, formula
pages, monograph boundaries, parser-warning pages and unusual-layout pages,
plus a sample of normal pages.
---
## Layer 5 — Invariants the book itself supplies
The source is redundant, and each redundancy is a free check that needs no
human ground truth. A violation is a proof of a defect.
- back-of-book index → monograph boundaries (in use: 92.9% recall / 99.1%
precision, on a denominator that is **not yet cleaned**)
- `"Bảng N"` captions → every caption must have a detected table (in use:
32/33)
- cross-references (`"xem Liều lượng và cách dùng"`) → must resolve to a
section that exists in the same monograph (**not built**)
- ATC codes → must match the WHO shape `[A-Z]\d\d[A-Z][A-Z]\d\d` (**not
built**)
- dose ranges (`"4 - 7,5 mg/kg"`) → must parse as two ordered numbers
(**not built**)
---
## Layer 6 — Fail safe at the point of use
Detection is never complete, so the system must stay safe when it misses.
- every chunk carries `page` + `bbox`; every answer carries a citation
- the UI shows the **rendered source crop** beside the answer, so a
pharmacist verifies against the book in seconds
- `quarantined` content and `formula_kind: 2d` never enter the model's
context as prose — crop or refuse, never linearised text
This is what makes the two Cockcroft-Gault formulas safe *today*, before any
reconstruction exists: left in prose they read as multiplication, which is a
dosing error.
---
## What may and may not be said in a report
- Name the **denominator** every time. "99%" of characters, pages, tables,
formulas and monographs are five different claims.
- Distinguish **detected / named / verified**. `unclassified = 0` is "named".
- A heuristic finds **candidates**; it never proves absence. The fraction-bar
rule is 69.6% precise and its recall is unknown — and known to be below
100%, because ADENOSIN (page 147) prints a fraction with no bar at all.
- Never write "100%", "complete", "all", "no data lost" or "production-ready"
unless the checks performed support the literal claim.
- The honest current shape: *"the parser processed 1668/1668 pages;
structural checks and 145 tests pass; nothing is lost without being
counted. Content accuracy is NOT confirmed at 100% because there is no
human-reviewed ground truth for the whole document to diff against."*