duocthu/docs/progress-log.md

# Progress Log

Chronological record of work done on this project, newest entry on top. The
goal is continuity across sessions: if a work session ends unexpectedly
(context/token limit, interruption), whoever picks this up next — human or
Claude — should be able to read the latest entry and know exactly what's
done and what's next, without having to reconstruct it from git history.

**Convention**: add a new entry at the top before ending a session whenever
meaningful progress was made, and proactively the moment it looks like the
session might run out of context/tokens mid-task — don't wait until the very
end if that risk is showing.

---

## 2026-08-01 (cont'd, 5) — ADR 0006 implemented: chunks now reference their lifted blocks; `chunk/` runs for the first time; 16/16 gates green

**Why this was needed, in one line**: a chunk of a section whose table had
been lifted was grammatical, complete-looking prose with the table absent and
nothing marking the absence — silent incompleteness, in the section where 127
of 167 lifted blocks live (`liều lượng và cách dùng`, 76%).

**Design is in `docs/adr/0006-quarantined-block-references-in-chunks.md`**,
written before any code. It resolves the item ADR 0005 explicitly deferred.

**Implemented:** `ChunkAttachment` (block_id, kind, shape, physical_page,
bbox, quarantined, header_row) on every prose chunk, plus one
`block_descriptor` chunk per block whose text is built **only** from
metadata. `chunk/io.py` now reads `tables` (it silently dropped them before)
and writes `schema_version: 2`.

**`chunk/` executed for the first time**, whole corpus:

| | |
|---|---|
| chunks | **12,838** — 12,671 prose + 167 descriptors |
| prose chunks carrying a lifted block | 185 |
| oversized (>800-token ceiling) | **0** |
| estimated tokens (chars/4, an estimate) | 2,115,427 |

**The condition this work was accepted under — prose chunks must not
change — was measured, not asserted.** Built the corpus both ways and
diffed:

| check | result |
|---|---|
| prose chunk count, both ways | 12,671 / 12,671 |
| chunk id sets identical | yes |
| `prose_text_changed` | **0** |
| `prose_nonattachment_field_changed` | **0** |

Only the two new fields differ. The change is strictly additive.

**A gate caught a real defect in my own design within minutes of existing.**
`block_text_leaked_into_chunk_text` fired on AMIODARON (physical page 183):
pdfplumber reported that table's first row as `"Thời gian liệu pháp tĩnh mạch
Liều 720 mg/ngày (0,5 mg/phút)"` — **a dose, inside what it called a
header**, from an extraction never verified by eye, being embedded as
retrieval text. Measured across the corpus: **42 of 124 simple-table headers
(34%) contain a digit.** Rule added: a header row is embedded only when no
cell contains a digit and every cell is short enough to be a label. 76 of 167
descriptors (46%) keep a header under that rule; the AMIODARON one does not.
A label with no digit cannot be mistaken for a dose.

**Full gate suite, 16/16 pass** — 10 corpus gates plus 6 ADR 0006 gates
(`section_block_without_chunk_reference`, `attachment_block_id_unknown`,
`attachment_without_page_or_bbox`, `block_text_leaked_into_chunk_text`,
`descriptor_chunk_without_attachment`, `descriptor_count_vs_block_count` =
167/167).

Tests: **158 passing** (148 → 158). `chunk/` had no tests at all before this
entry; it now has 10, including the prose-unchanged invariant and the
numeric-header refusal.

**Binding on `ai-service`, stated in ADR 0006 and not implemented here:** a
chunk with `has_quarantined_content` must make the answer say a table or
formula exists at the cited page and surface its crop; a `block_descriptor`
may be answered only with the crop; no chunk carrying a quarantined
attachment may be used to state a numeric dose.

**Still open:** table row/column reconstruction (the opendataloader cell data
is available and matches pdfplumber exactly inside the monograph range);
recall for borderless tables and bar-less formulas; content accuracy against
the source; the general chapters and appendices (9.6% of characters).

## 2026-08-01 (cont'd, 4) — READY TO CHUNK: transcriptions merged back into the text, `cli chunk-ready` gate suite green on all 10 gates, two more real data-loss bugs found and fixed on the way

**The blocker is closed.** The 1,116 transcribed characters are no longer a
file beside the corpus — they are in it. `ingestion/extract/repair.py` splices
each transcribed run back into the span stream geometrically, and every
command that builds monographs now goes through the same repaired stream, so
the ledger and the output describe one pipeline rather than two.

**New gate suite, `cli chunk-ready`** (`ingestion/validation/readiness.py`).
Each invariant gets its own count and its own target — a single verdict would
hide exactly what took this session to find. Run on the whole corpus:

| gate | count | target |
|---|---|---|
| outlined_run_not_merged | 0 | 0 |
| known_corruption_string | 0 | 0 |
| formula_fragment_in_prose | 0 | 0 |
| pua_char | 0 | 0 |
| replacement_char_ufffd | 0 | 0 |
| empty_section | 0 | 0 |
| section_without_provenance | 0 | 0 |
| unflagged_quarantine_block | 0 | 0 |
| duplicate_drug_id | 0 | 0 |
| monograph_without_page_range | 0 | 0 |

Corpus going into chunking: **683 monographs, 11,966 sections, 8,212,712
characters**, plus 167 quarantined table/formula blocks held outside prose.

**Two real bugs surfaced by building the gates, both fixed:**

1. **A 4pt glyph in the column-overlap strip was assigned the wrong column.**
   `classify_column`'s two tolerance bands overlap between x=288 and x=319 and
   left was tested first, so a single `ổ` at x=315 on physical page 714 was
   classified as left-column and could not be matched to its own right-column
   line. `Độ ổn định` stayed `Độ n định` even after the repair ran. Fixed by
   testing exact containment before tolerance. Invisible for a full-width
   block; only a narrow box exposes it.
2. **A plain body line that repeats a section name was read as a heading.**
   FLUOROURACIL (physical page 681), verified by rendering the page, prints
   `Thời kỳ mang thai` / `Chống chỉ định.` and `Thời kỳ cho con bú` /
   `Chống chỉ định.`. Both body lines matched the section vocabulary, so both
   sections came out **empty** and the statement that fluorouracil is
   contraindicated in pregnancy and while breastfeeding was dropped entirely.
   Fixed narrowly: a *non-bold* label directly under a heading is that
   heading's body. Boldness still cannot be required in general (outlier item
   20), hence the position constraint rather than a style rule.

A third placement bug was caught during the merge itself: PyMuPDF emits the
text either side of a dropped glyph as **one span whose box spans the gap**,
so splicing at span boundaries produced `tuở ổi`. `repair.py` now reads
per-character boxes from `rawdict` and splits the containing span at the
character offset the geometry indicates.

**Whole-document re-measurement after all of the above:**

| check | result |
|---|---|
| `cli run` | 683 monographs, 51 runs merged (1,116 chars), 167 blocks lifted / 167 quarantined |
| `cli validate` | 92.9% recall / 99.1% precision — unchanged |
| `cli coverage` | 252,801 spans, **unassigned = 0** |
| `cli chunk-ready` | 10/10 gates pass |
| tests | **148 passing** (145 → 148) |

**What these gates explicitly do NOT prove**, printed by the command itself so
it cannot be quoted out of context: content accuracy against the source (no
whole-document human-reviewed ground truth exists), table row/column
reconstruction, and recall for borderless tables and bar-less formulas.

**Next:** `chunk/` still has no tests and has never been executed. Table
reconstruction from the opendataloader cell data remains available and is not
on the critical path.

## 2026-08-01 (cont'd, 3) — All 23 fraction-bar candidates read by eye (precision 69.6%), 51 outlined runs transcribed, 2D formulas quarantined; prose-leak gate = 0

**All 23 `fraction_bar_candidate` regions were rendered and read.** Verdicts,
one page at a time:

| verdict | count | where |
|---|---|---|
| real 2D formula | **16** | p43, p92 (×5), p202, p325 (×2), p349, p1042, p1043 (×2), p1132, p1402 (×2) |
| not a formula | **7** | p4 (×3 decorative underlines on the Ministry decision page), p63 (ruled box), p845, p878 (table cell borders), p1667 (rule above the colophon) |

**Precision of the candidate rule: 16/23 = 69.6%.** That is why the verified
list is a curated file (`ingestion/data/verified/formula_regions_2d.json`) and
not the detector's raw output — a 70%-precise rule must not quarantine
content on its own. 10 of the 16 are inside the monograph range.

**A formula the detector cannot find, confirmed.** ADENOSIN (physical page
147) prints `Tốc độ truyền dịch (ml/phút) = 0,140 (mg/kg/phút) × trọng lượng
cơ thể (kg) / Nồng độ adenosin (3 mg/ml)` as **three plain lines with no
fraction bar at all** — verified by rendering the region and reading it. No
geometric signal exists to detect it; it surfaced only because a prose-leak
gate matched its text. It is quarantined and flagged, and
`recall_limit` in the verified file records that **the number of bar-less
formulas in the book is UNMEASURED**. The fraction-bar scan must never be
described as complete formula coverage.

**51 outlined runs transcribed** into
`ingestion/data/verified/outlined_text_transcriptions.json` — 22 full lines
plus 29 single glyphs, **1,116 characters** recovered, each with page, bbox,
the run's text and the extracted line it belongs to. Every value there is a
transcription read off a rendered page, labelled as such, never extracted
data.

**The single-glyph runs are the nastier half of that defect.** They are
Vietnamese diacritic characters dropped out of lines that otherwise extract
fine, so the damage is invisible downstream:

| extracted | actual |
|---|---|
| `Độ n định:` | Độ **ổ**n định |
| `≥ 1 tu i` | ≥ 1 tu**ổ**i |
| `Thuốc dùng tại ch :` | tại ch**ỗ** |
| `i nồng độ glucose máu` | (thay đ)**ổ**i nồng độ glucose máu |

**2D formulas are now quarantined in the pipeline.** `SHAPE_FORMULA_2D` was
added to the existing shape taxonomy and to `QUARANTINE_SHAPES` — an entry,
not an edit to matching code. `ingestion/extract/formulas.py` loads the
verified regions and grows each bar into a band covering numerator and
denominator. Whole-book re-run:

| gate | result |
|---|---|
| verified formula regions loaded | 17 on 10 pages |
| blocks lifted out of prose | 169, **169 quarantined** |
| `formula_2d` blocks | 14 |
| `formula_fragment_left_in_prose` | **0** |
| monographs | 683 (unchanged) |
| `cli validate` | 92.9% recall / 99.1% precision (unchanged) |
| tests | **145 passing** (139 → 145) |

The side margin needed two attempts: at 4pt, AMPICILIN VÀ SULBACTAM's
numerator `Thể trọng (kg)` stayed behind in the prose because its span box
carries leading spaces that pull its centre left of the bar. Raised to 95pt
with the reasoning recorded in the module: over-capturing a neighbouring line
into a quarantined block is recoverable, half a formula left in prose is not.

**Still open**: the 1,116 transcribed characters are recorded but **not yet
merged back into the monograph text** — the corpus still contains
`Độ n định`; table row/column reconstruction is untouched (137 simple tables
+ 17 multi-header + 1 continuation remain quarantined); table detection
recall for borderless tables is unmeasured; `chunk/` still has no tests and
has never run.

## 2026-08-01 (cont'd, 2) — Residual-ink check built and run whole-document; found a text-loss class no text-based check could see: 51 runs of type drawn as vector paths on 5 pages

**What was built.** `ingestion/validation/residual_ink.py` (production, plus a
`cli residual-ink` command) renders each page, whites out every pixel covered
by an extracted span, and reports the ink that survives. It needs no ground
truth and no sampling. Measured: **0.06 s/page, all 1668 pages in under two
minutes.** Classification is a pure function over `(region, PageContext)` with
an ordered rule list, so a new kind of residual is a new entry, not an edit.

**Whole-document gate result — all 1668 pages, 3,931 residual regions:**

| kind | regions |
|---|---|
| header_rule | 1,649 |
| text_as_vector_outline | 1,061 |
| table_frame | 959 |
| antialias_speck | 220 |
| fraction_bar_candidate | 31 |
| rule_fragment | 10 |
| header_band_fragment | 1 |
| **unclassified** | **0** |

**The finding: 51 runs of text on 5 pages exist only as vector outlines.**
Physical page 714 (GATIFLOXACIN) prints 17 full lines of ordinary prose that
`page.get_text()` does not return, `page.search_for()` cannot find,
`pdfplumber` does not return and `opendataloader-pdf` does not return.
`page.get_drawings()` shows why: each line is a filled path of 1,126-1,831
items, shaped exactly like one line of type, in the body-text colour. Single
glyphs appear the same way with 39-45 items. Recovery cannot be automatic —
the paths carry no character codes — so `ingestion/extract/outlined_text.py`
detects and reports them for transcription and never guesses.

| physical page | outlined runs |
|---|---|
| 714 | 31 |
| 736 | 16 |
| 1373 | 1 |
| 1444 | 1 |
| 1445 | 2 |

All five are inside the monograph range. Two independent methods agree on the
same five pages: the drawing-shape scan, and counting glyph-shaped leftovers
in the residual mask. Sample of what is missing, read off the rendered page:
`"Nghiên cứu trên động vật, gatifloxacin gây ngộ độc cho thai."` (p714),
`"(Typhoid, inactivated, whole cell), J07AP03 (Typhoid, purified"` (p1445).

**Three instrument bugs were found and fixed before any of the above was
believed** — the measuring device was wrong before the data was, three times:
1. **Horizontal banding merged the two page columns**, so page 209's ADR table
   sat in a box whose centre fell in the gutter and matched no table region.
   Adding a column split then cut single table grids into their individual
   rules. Replaced with 2D connected components (`scipy.ndimage.label`).
2. **A glyph-count ratio was nearly reported as a data-loss measure.** First
   pass gave "extraction ratio 0.6656, 835 pages below 98%". It was wrong:
   `get_texttrace()` counts glyphs painted outside the page rectangle —
   4,717,407 of them, on pages that are visually blank. Clipping to the page
   rect gave 0.8023 and "1642 of 1668 pages below 95%", which was also wrong:
   Vietnamese diacritics are painted as two glyphs and extracted as one
   character, so the deficit is systematic and meaningless. **Neither ratio
   should ever be quoted.** The pixel-based check is the sound one.
3. **Mask padding of 1.0pt ate the fraction bars** it was meant to find.
   Calibrated to 0.5pt against the two known formulas, verified not to add
   noise on a 10-page prose sample.

Incidentally this explains a long-standing note in ADR 0003: `pdfplumber`
"scrambles reading order" on this document because it reads the off-page text
that PyMuPDF correctly clips away.

Tests: **139 passing** (129 → 139), including whole-document regression
fixtures pinning the 51 outlined runs per page and the two fraction-bar
widths (188.6pt on p1042, 118.1pt on p202).

**Not done / next:** the 31 `fraction_bar_candidate` regions on 15 pages have
**not** been looked at yet, so no precision figure for them exists; the 51
outlined runs are detected and flagged but **not transcribed**, so that text
is still absent from the corpus; 2D formulas are still not quarantined in
`segment/`. `unclassified = 0` means every region is *named*, not that every
named verdict has been checked by eye — of the seven kinds, `header_rule`,
`table_frame`, `antialias_speck`, `rule_fragment` and `header_band_fragment`
were confirmed on sampled examples only.

## 2026-08-01 (cont'd) — Two 2D fraction formulas confirmed corrupted in output by reading the source page images; both tools are blind to them, so cross-tool agreement does NOT bound recall

**Finding, visually confirmed on the rendered source, n=2:** stacked-fraction
formulas lose the fraction bar and emit the numerator *before* the `=`, so
the division reads as multiplication.

| drug | physical page | source (read from the page image) | pipeline output |
|---|---|---|---|
| NETILMICIN | 1042 | `Cl_cr (ml/phút) = [(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ)] / [Nồng độ creatinin huyết thanh (micromol/lít) x 0,81]` | `(140 - tuổi) x cân nặng (kg) (x 0,85 đối với nữ) Clcr (ml/phút) = Nồng độ creatinin huyết thanh (micromol/lít) x 0,81` |
| AMPICILIN VÀ SULBACTAM | 202 | `Cl_cr (ml/phút) = [Thể trọng (kg) x (140 - số tuổi)] / [72 x creatinin huyết thanh (mg/dl)]` | `Thể trọng (kg) x (140 - số tuổi) Clcr (ml/phút) = 72 x  creatinin huyết thanh (mg/dl)` |

Read literally, both now state that clearance is *multiplied* by serum
creatinine. This is a dosing calculation in a renal-impairment section. The
content is **not quarantined and carries no formula flag** — it flows into
`chunk/` as ordinary prose.

**This corrects the weight I put on cross-tool table agreement earlier the
same day.** Measured: on physical page 1042 `pdfplumber.find_tables()`
returns **0** regions and opendataloader returns **0** tables; the same holds
for the formula region on page 202. The two tools agreeing on 112 shared
table pages measures *consistency on what ruling lines make visible*, not
recall — they share the blind spot. Agreement must not be reported as
evidence of coverage.

**Priority consequence:** the 155 table blocks are already `quarantined:
true`, i.e. contained — they cannot poison an answer today. The formulas are
uncontained. Formula handling should therefore come before table
reconstruction, which is the reverse of the plan written earlier today.

**Population sizing, honest limits.** A keyword scan of the output found 185
occurrences of "công thức", of which **93 are "công thức máu/bạch cầu/hồng
cầu"** (blood count, not mathematics) and many of the remaining 92 mean
"formulation" (`thành phần trong công thức`). So keyword counting cannot size
the formula population; only a detector with measured recall can. The two
cases above are the first two regression fixtures.

## 2026-08-01 — Readiness check re-measured from the current artifacts (no code change): text coverage complete, tables quarantined, formulas still unhandled

Question asked: is the data ready to parse 100%, including formulas and
tables? Every number below was recomputed in this session from the files on
disk (`ingestion/data/processed/{monographs.jsonl,coverage_ledger.json}`) and
from a fresh test run — none quoted from earlier entries.

| check | command / scope | result |
|---|---|---|
| unit tests | `python -m pytest -q` (whole `ingestion/`) | **129 passed** |
| monographs / sections | read `monographs.jsonl` | 683 / 11,966 |
| table blocks in output | read `monographs.jsonl` | **155 blocks, 155 quarantined** (simple_table 137, multi_level_or_merged_header 17, cross_page_continuation 1) |
| span coverage ledger | read `coverage_ledger.json`, all pages | 252,733 spans; `unassigned` = **0** |
| ledger states | same | normalized_text 177,754 (8,183,182 ch) / out_of_scope 53,374 (897,692 ch) / heading 12,764 / boilerplate_excluded 4,976 / quarantined 3,862 / structural_excluded 3 |
| page coverage | ledger vs `doc.page_count` | 1666 of 1668 pages carry spans |
| the 2 pages with no spans | rendered physical 99 and 1666 at 110 dpi, read the images | **both genuinely blank** (0 chars, 0 images, only a frame drawing) — not a loss |
| PUA left in output | scan all 11,966 sections | **0** |
| U+FFFD in output | scan all 11,966 sections | **0** — closes the gap flagged in the previous entry as never measured |

Note the block count differs from the previous entry's `148` — this is a
recomputation from the current file, not a correction of a bug; the shape mix
also differs from the 180-region whole-book classification because blocks are
only the regions that fall inside the monograph range.

**Answer: no, not ready for a "100% including formulas and tables" claim.**
What is closed: goal A (full coverage, nothing silently dropped) for the
monograph text path — `unassigned = 0`, both uncovered pages proven blank.
What is open, by name:
- **Formulas: no production stage exists.** `grep -il formula` over
  `ingestion/ingestion/` hits only `chunk/sentences.py` and `cli.py`; all
  formula work lives in `scratch/`. The only detector fired 3,405
  `fraction_bar` hits on 837 of 1668 pages with precision never measured, so
  there is not even a trustworthy formula *count*, let alone reconstruction.
  2D formulas currently linearise into section text unflagged.
- **Tables: detected and quarantined, not reconstructed.** 155/155 blocks are
  `quarantined: true` — provenance kept, unsafe to cite. Borderless tables
  (BSA nomogram, catalog item 7) are invisible to `pdfplumber` by
  construction, so the miss rate is unmeasured and undetected tables still
  contaminate body text.
- **Out-of-scope regions unparsed**: 53,374 spans / 897,692 chars (9.6% of
  ledger chars) — general chapters and appendices — are excluded explicitly
  but have never been structurally parsed.
- **Content accuracy vs. source never measured**; 92.9% / 99.1% is
  boundary detection only, on an uncleaned 1064-entry denominator.
- `chunk/` still has no tests and has never been executed.

## 2026-07-31 (cont'd, 5) — Cleanliness audit before chunking: data is NOT clean; 5 defects measured whole-corpus, incl. ≥/≤ in dosing text lost as PUA glyphs (all 8 PUA codepoints visually confirmed)

**Trigger**: user pushed back on starting the chunk stage ("chưa chunk dữ
liệu phải sạch"), correctly — chunking was about to run against text that
had never been audited for content-level cleanliness. Only boundary
detection had ever been measured, never the text itself.

**Also fixed this session (small)**: `cli.py` crashed with
`UnicodeEncodeError` on Windows cp1258 when printing Vietnamese drug names
in `validate`'s unmatched lists — the metrics printed first so past numbers
were unaffected, but the tail of the report was lost. Added
`sys.stdout/stderr.reconfigure(encoding="utf-8")` in `main()`. Re-ran
`cli validate`: exit 0, Vietnamese renders correctly.

**Timing measured for the first time** (whole 1668-page PDF, PyMuPDF only):
`cli run` = **2m10.6s**, `cli validate` = **44.4s**. Does not cover
pdfplumber/opendataloader/docling cross-checks, which are not part of either
command.

**Boilerplate re-verified independently** against output generated this
session: **0 of 11,409 sections** contain "DTQGVN" (was 1,374), 0 of 682
monographs affected. Also closed the previously-flagged gap of "never
checked with a different signature": scanned for a bare 3-4 digit line
(page number leaking without "DTQGVN" adjacent) — 204 sections matched,
sampled 8, **all legitimate content** (`cytochrom P\n450` split across
lines, dosing values like `250 microgam/kg`), not boilerplate. Scope limit:
8 of 204 inspected, not all.

**Cleanliness audit — whole corpus, 682 monographs / 11,409 sections /
8,241,485 section chars** (`ingestion/scratch/cleanliness_audit.py`,
temporary, to be deleted once this finding is fully captured):

| signal | occurrences | sections hit | % sections |
|---|---|---|---|
| mid-sentence line wrap | 99,501 | 8,197 | 71.8% |
| short fragment lines (<4 chars) | 11,612 | 2,149 | 18.8% |
| bare-number lines | 2,540 | 862 | 7.6% |
| flattened table rows | 25 | 9 | 0.1% |
| PUA chars | 86 | 41 | 0.4% |

**Confirmed: table content IS contaminating section body text.** Real
example — AMPICILIN's `duoc_ly_va_co_che_tac_dung` contains an
antibiotic-resistance table flattened to `'Salmonella typhi\n378\n10,6\n
0,0\n89,4\nShigella flexneri\n120\n41,6...'`, losing all row/column
semantics. The 0.1% figure is only what the all-numeric-row regex catches;
the true table count is pending the inventory scan and will be higher.

**Confirmed, patient-safety relevant: comparison operators in dosing text
are being emitted as raw PUA codepoints.** All 8 distinct PUA codepoints in
the corpus were located in the source PDF, rendered to images, and read
directly (not inferred from context):

| codepoint | count | actual glyph | visual evidence |
|---|---|---|---|
| U+F0B3 | 57 | **≥** | p.141 "trẻ em ≥ 10 tuổi" |
| U+F0A3 | 17 | **≤** | p.169 "liều ≤ 100 mg" |
| U+F061 | 5 | **α** | p.334 "Streptococcus α tan huyết" |
| U+F0AE | 3 | **→** | p.1027 "HCO₃⁻ + H⁺ → H₂CO₃ → CO₂ + H₂O" |
| U+F0D2 | 1 | **®** | p.891 "Plasma Lyte® 56/5%" |
| U+F031 | 1 | **₁** | p.957 "alpha₁-acid glycoprotein" |
| U+F0AF | 1 | **↓** | p.1033 "rhodanese ↓" (catalysis arrow) |
| U+F067 | 1 | **γ** | p.1352 "interferon - γ" |

74 of 86 occurrences are ≥/≤ inside dosing or adverse-effect sentences —
losing the operator changes clinical meaning ("liều ≤ 100 mg" vs "liều 100
mg"). Fonts involved: `SymbolTiger` (7 codepoints) and `Symbol` (1).

**Chunk stage — partially built, then deliberately paused.** Wrote
`ingestion/ingestion/chunk/` (`models.py`, `sentences.py`, `chunker.py`,
`io.py`, `__init__.py`) implementing ADR 0004: `(drug_id, section_key)` unit,
800-token ceiling, sentence-boundary-aware sub-chunking. **Not tested, not
run, and must not run until the cleanliness defects above are fixed** —
chunking dirty text bakes the defects into embeddings. ADR 0004's own
"hard prerequisite" (the boilerplate bug) is satisfied, but this audit found
additional blockers it did not know about.

**Strategy adopted for full-coverage parsing** (written up in
`docs/full-coverage-parsing-plan.md`): separate what is provably clean from
what is not — chunk the clean text, flag-and-exclude untrustworthy tables/
2D formulas with an exact excluded count, and prove nothing was silently
lost via a **character coverage ledger** (every char on all 1668 pages must
land in exactly one bucket: section text / table cell / formula region /
out-of-scope / `unassigned`, with `unassigned` reported as a number plus
page+bbox list). Note the plan explicitly distinguishes goal A (full
coverage, nothing silently dropped — achievable) from goal B (proven 100%
correct — requires manual ground truth for every table/formula, not
achievable in one day).

**Fixes landed after the audit above — new `ingestion/ingestion/normalize/`
stage** (`glyphs.py` = the verified PUA map, `text_flow.py` = geometry-driven
span rejoining). Root cause of defects 1-3 was one line in
`segment/assembler.py`: `body_lines.append(span.text.strip())` made every
*span* its own line, so any visual line the PDF split into multiple spans
(italic run, subscript, symbol font) became multiple lines. Text-level regex
cannot distinguish a mid-word span split from a real line wrap, so the fix
uses geometry instead — PyMuPDF's own `(block, line)` indices identify spans
sharing a visual line, and the horizontal gap (`SPACE_GAP_PT = 1.0`) decides
whether a space belongs. Assembler now collects `Span` objects and joins via
`normalize.join_spans` + `normalize.substitute_pua`.

**Whole-corpus re-measurement after the fix** (same audit script, same scope
— 682 monographs / 11,409 sections):

| signal | before | after |
|---|---|---|
| mid-sentence line wrap | 99,501 | **0** |
| short fragment lines | 11,612 | **7** |
| bare-number lines | 2,540 | **0** |
| flattened table rows (numeric-row regex) | 25 | **0** |
| PUA chars | 86 | **0** |

`cli validate` re-run after the change: **unchanged** at 682 monographs,
92.8% recall, 99.1% precision — normalization does not affect boundary
detection. Tests: **119 passed** (110 before; 9 new in `tests/
test_normalize.py`, covering the real corpus cases — `cytochrom P450`
subscript rejoin, `(feline immunodeficiency virus)` italic rejoin, ≥/≤
restoration in dosing sentences, unmapped-PUA reporting). One existing test
(`test_running_header_boilerplate_stripped_...`) had its expected string
updated: it encoded the old `\n` join for `"...không nhai. Nếu"` + `"uống
viên thuốc..."`, which is exactly the mid-sentence wrap being fixed; its
core assertions (no "DTQGVN", no "1009") are unchanged.

**NOT verified — total section chars dropped 13,224** (8,241,485 →
8,228,261, 0.16%). Reasoning from the code says this is separator characters
only (same-line spans previously contributed a `\n` each, now join directly;
`strip()` only ever removed whitespace and no span is dropped), so
non-whitespace content should be unchanged at 6,552,254 — but **this was
reasoned, not measured**. The character coverage ledger (below) is the
instrument that would actually prove it and has not been run.

**Whole-corpus table/formula inventory completed** (17m17s,
`ingestion/scratch/inventory_tables_formulas.py`, all 1668 pages):
- **200 tables on 152 distinct pages**, 0 page errors. Column distribution:
  3 cols ×78, 2 ×72, 4 ×32, 5 ×10, 1 ×4, 7 ×3, 6 ×1.
- ~~22 header-less-at-top continuation candidates~~ — **this figure was
  wrong and is corrected below**: classifying all 200 regions individually
  showed 17 of those 22 are `not_a_table_full_page` and 3 are
  `not_a_table_degenerate`, leaving **2** real cross-page continuations.
  Cause: the inventory's condition (`starts_near_top AND
  header_textual_cells <= 1`) is satisfied automatically by any full-page
  false-positive region — its bbox starts at y≈0, and its single cell is a
  long text blob rather than a textual header — so every non-table landed
  in the continuation bucket.
- **0 all-numeric wide grids** — but this is a detector limitation, not
  evidence of absence: the known BSA nomogram (item 7) has no ruling lines,
  so `pdfplumber.find_tables()` cannot see it at all.
- **Formula detector over-fires badly and its output must not be quoted**:
  3,405 `fraction_bar` hits across **837 of 1668 pages** (half the book) is
  not credible as a formula count — the thin-horizontal-rect signal is
  evidently matching table rules/underlines/column separators. Precision was
  never measured; this confirms the standing warning that a bbox heuristic
  finds candidates, not formulas. `small_font_numeric` (2,583) is likewise
  unvalidated. Only the PUA count (86) from that scan is trustworthy, and
  only because all 8 codepoints were visually confirmed.

**Section-name spelling variants — a large silent section loss, found and
fixed.** Scanned the whole monograph range for bold heading strings that do
not match the vocabulary, ranked by similarity: **42 distinct near-miss
strings, 542 occurrences**. The dominant one is `"Thông tin qui chế"`
(**469×**) — the book prints "qui" where its own documented template (and
`vocab.py`) says "quy", so `match_section` returned `None` and the section
was never opened. Measured before the fix: only **96 of 682 monographs
(14.1%)** had a `thong_tin_quy_che` section; **586 were missing it entirely**
(the text itself was not lost — it fell into the preceding section's body
unlabelled — but the structure was, so a "thông tin quy chế của X" query
could not retrieve it and citations would name the wrong section).

Two mechanisms were added rather than one long alias list:
- `SectionDef.aliases` for genuinely different wordings ("Mã ACT",
  "Chống chỉ đinh", "Thời kì mang thai", "Hướng dẫn cách sử trí ADR",
  "Quá liều và xử lý", "Dược lí và cơ chế tác dụng", …).
- `_lookup_key()` folds typesetting noise for every entry at once —
  all whitespace removed, case folded, and the U+00D0/U+0110 look-alike
  ("Ðộ" vs "Độ") mapped. This alone absorbs ~14 variants that would
  otherwise each need an alias: "Chỉđịnh", "Chống chỉđịnh", "Độổn định và
  bảo quản", "H ướng dẫn cách xử trí ADR", "Tư ơng kỵ", "Tác dụng
  khôngmong muốn (ADR)", "Thận trọng.", "Liều l ượng và cách dùng", …
- Two near-misses were **deliberately rejected** and recorded in
  `REJECTED_NEAR_MISSES` so a later reader does not add them: "Thể trọng"
  (body weight, 0.84 similar to "Thận trọng"/caution) and "Tác dụng không
  mong muốn của opioid" (a drug-specific sub-heading, not the section).

**Whole-corpus result after the vocabulary fix:**

| | before | after |
|---|---|---|
| monographs | 682 | **683** |
| sections total | 11,409 | **11,966** (+557) |
| `thong_tin_quy_che` present | 96 (14.1%) | **567 (83.0%)** |
| `cli validate` | 92.8% / 99.1% | **92.9% / 99.1%** |
| tests | 119 | **122** |

**Over-joining check (the direction the rejoining work had not tested).**
First attempt used text patterns and had poor precision — sampled examples
were mostly false positives ("Liều lượng có thể tăng…" is ordinary prose,
"Lọ 10, 50, 100 ml" is a volume list, "Wolff - Parkinson - White" is a
hyphenated name), so its counts are not reported here. Redone at the level
where it can actually be judged — the geometry of the two visual lines being
joined — with the same exclusions `assemble()` applies (bold headings and
header-band boilerplate removed, since joins involving those never reach
body text). Monograph range, 180,131 body spans → 154,683 visual lines,
**102,798 joins performed**:

| category | count | % of joins |
|---|---|---|
| clean wrap | 98,611 | **95.9%** |
| indent change | 1,983 | 1.9% |
| vertical gap > 16pt | 514 | 0.5% |
| column change | 634 | 0.6% |
| page change | 567 | 0.6% |
| upward (column/page turn) | 489 | 0.5% |

Vertical gap at join points: median **12.1pt**, p90 12.4pt — a tight
single-leading distribution, i.e. the overwhelming majority are genuine
wraps. **But over-joining is real and it is concentrated in tables**:
physical page 109 shows a dosage-form table being concatenated cell by cell
— `'Viên nén' + '1'`, `'1' + '1 - 4'`, `'1 - 4' + '8 - 12'`, `'8 - 12' +
'Viên nang tác'`, `'18 - 24' + 'Tiêm bắp'`, `'Chưa biết' + 'Tiêm tĩnh'`.
This confirms the risk case predicted before the check was run, and it
settles an ordering question: **table regions must be excluded before
joining, not after.** Not all 1,983 indent-change cases were inspected —
at least one sampled case (`'…(ức chế' + 'alpha-glucosidase).'`) is a
correct wrap with a hanging indent, so that category's precision is
unmeasured.

**Span-level coverage ledger built and run whole-document.** Implemented as
an optional `ledger` argument to `assemble()` plus a `cli coverage` command,
at span level rather than character level (characters cannot balance because
normalization joins and substitutes them). All 1668 pages, 252,733 spans
after merge:

| state | spans | % spans | chars | % chars |
|---|---|---|---|---|
| normalized_text | 181,616 | 71.9% | 8,231,038 | 87.6% |
| out_of_scope | 53,376 | 21.1% | 897,724 | 9.6% |
| heading | 12,764 | 5.1% | 221,266 | 2.4% |
| boilerplate_excluded | 4,976 | 2.0% | 47,609 | 0.5% |
| **unassigned** | **1** | 0.0% | 21 | 0.0% |

The single unassigned span is `"CÁC CHUYÊN LUẬN THUỐC"` on physical page 98
— a part-divider title excluded on purpose via `PART_DIVIDER_TITLES`.

This also **settles the previously-unverified 13,224-character delta**:
`raw_chars_before_merge` = 9,397,658 equals the post-merge total exactly, so
the span-merge step loses no characters; the delta was separator characters
in section assembly, as reasoned earlier but now measured.

**Important limit, learned the hard way in the same session**: the ledger
proves every span was *routed*, not that routed content *survived* into the
output. The section-overwrite bug below was invisible to it — spans were
correctly marked `normalized_text`, then their section was overwritten
downstream.

**Table isolation wired into `assemble()` and gated.** `assemble(spans,
table_index=...)` diverts spans inside a real table region into
`Monograph.tables` (a new `TableBlock` with `table_id`, `shape`,
`physical_page`, `bbox`, `section_key`, `quarantined`). Gate results over
the whole book:

| gate | result |
|---|---|
| `non_table_span_changed` | **0** |
| `table_span_in_normalized_text` | **0** |
| `unintended_duplicate` | **0** |
| `section_emptied` | **0** (was 1 before the overwrite fix) |
| `unassigned` | 1 (the deliberate part divider) |
| lifted blocks | 148, all with unique ids |
| quarantined | **148 / 148** |

Quarantine policy was widened per review: every multi-column shape
(`simple_table`, `multi_level_or_merged_header`, `cross_page_continuation`,
`grid_2d_numeric`) is quarantined until a real row/column reconstruction
exists, because linearised cells are not safe to cite. Only
`single_column_boxed_list` is exempt — one column linearises correctly.

`183 regions loaded but only 148 blocks lifted` is explained, not a loss:
1,676 table spans sit on pages outside the monograph range (e.g. physical
page 42, in the general chapters), where no monograph is open to attach them
to. Those pages are still out of scope entirely.

**Three real bugs found by these gates, all fixed:**
1. **Section overwrite destroyed content in 33 monographs (38 occurrences).**
   A repeated section heading inside one monograph replaced the existing
   `SectionSpan`, discarding everything captured before the repeat.
   CEFAMANDOL's `lieu_luong_va_cach_dung` held only 172 characters of
   flattened renal-dosing table; after the fix it holds **881 characters** of
   real dosing prose ("Cách dùng Thuốc được dùng dưới dạng cefamandol
   nafat…"). Sections are now concatenated, with the first heading kept as
   the provenance anchor. Other affected monographs include CEFAPIRIN NATRI
   and CEFRADIN — also dosing sections.
2. **Duplicate `table_id`.** A region flushed twice emitted two blocks with
   the same id; provenance ids must be unique. Now suffixed (`p339_t0`,
   `p339_t0#1`). Verified: 148 blocks, 148 unique ids.
3. **Table blocks were never written to disk.** `write_monographs_jsonl`
   had no `tables` field, so all 148 lifted blocks were computed, reported
   in the run summary, and then silently dropped at the file boundary. Found
   only because a check script raised `KeyError: 'tables'`. Fixed with a
   round-trip test.

Tests: **129 passing** (122 → 129).

**682 → 683 explained.** A faithful reconstruction of the pre-fix vocabulary
(old `match_section`/`match_section_with_inline_value` patched into the
importing modules, no aliases, no whitespace folding, no Ð/Đ mapping)
reproduces exactly **682**; the current code gives **683**. The difference is
one monograph: **CARBAMAZEPIN**, physical pages 315-319, ATC `N03AF01`, 18
sections, anchor "Carbamazepine.". No monograph disappeared (`GONE` is
empty) and it occurs exactly once, so this is a recovered false negative,
not a duplicate — it is the same `"Carbamazepin, 316"` entry that
`cli validate` had been listing as unmatched ground truth. Two earlier
attempts at this comparison were **invalid** and their numbers (683/683 and
589/683) should be ignored: the first left aliases in `_PREFIX_CANDIDATES`
and kept the new `_lookup_key`, the second built old-style lookup keys but
still queried them through the new whitespace-stripping key function.

**Not done yet / next up:**
  **Design revised** (per review feedback, and it is the better design):
  make it a **span/fragment-level ledger** first and aggregate characters
  from it, because normalization joins, substitutes and drops characters so
  a pure character count cannot balance. States: `normalized_text`, `table`,
  `formula`, `boilerplate_excluded`, `out_of_scope`, `quarantined`,
  `transformed_with_mapping`, `unassigned`.
- PUA reporting should be stated as `known_mapped` / `unknown_pua` /
  `replacement_char_U+FFFD` counts; only `pua_chars = 0` has been measured,
  `U+FFFD` has never been checked.
**All 200 table regions classified individually, then the "not a table"
verdicts checked by rendering every one of them and reading it.** This is
recorded in full because the first two counts reported in this area were
both wrong, and both were wrong the same way — stated from metadata before
anything was looked at:

1. "22 header-less-at-top continuation candidates" — wrong, see the
   correction above; the real figure is 4.
2. "22 of 200 are not tables" — asserted from rules (area ratio ≥ 0.75,
   `n_rows <= 1 or n_cols <= 1`) without opening a single page.

Rendering all 22 and reading them showed **20 correct, 2 wrong**:
- Correct (not tables): p1 copyright page; p3, p5, p1529 blank pages; p7
  table of contents; p9, p10 committee member lists; p12 Vietnamese/English
  drug-name list; p1665 back index; p55 ×3 epilepsy classification lists;
  and p172, p196, p382, p760, p944, p1034, p1230, p1336 — **ordinary
  two-column monograph prose** that `pdfplumber.find_tables()` reports as
  one page-sized table.
- **Wrong**: p62 and p72 are 1×3 regions with visible cell rules — real
  **orphaned continuation rows** of tables broken across a page
  (outlier-catalog item 5). The `n_rows <= 1` rule discarded precisely the
  case where losing content hurts most, since a row without its header
  cannot be interpreted at all.

`classify.py` now treats only `n_cols <= 1` as degenerate and routes a
single row with several columns to `cross_page_continuation`. Corrected
whole-set result:

| shape | count |
|---|---|
| simple_table | 154 |
| multi_level_or_merged_header | 22 |
| not_a_table_full_page | 17 |
| cross_page_continuation | 4 |
| not_a_table_degenerate | 3 |
| **real tables** | **180** |
| **not tables** | **20** |

**Verification scope, explicitly**: all 20 non-table verdicts were confirmed
visually, one page at a time. The 180 real tables' individual shapes
(simple vs multi-level header vs continuation) are **rule-derived only and
have not been checked by eye** — that classification must not be reported as
verified.

**Is "200 tables" trustworthy? Partly — and the limits matter.**
- **No truncation**: 200 records across 152 distinct pages (max 5 on one
  page, spanning physical pages 1-1665). Re-running `find_tables()` over
  just those 152 pages reproduces exactly 200. The round number is a
  coincidence, not a cap. **But this is a reproducibility check with the
  same tool and settings, not independent validation.**
- **Detection recall, measured against the book's own captions**: 33 pages
  carry a `"Bảng N"` caption; 32 of them have a detected table → **97% on
  the captioned subset**. 102 detected-table pages carry no caption, which
  is expected (most tables here are unnumbered). **This measures recall only
  on captioned tables** — borderless tables are invisible to `pdfplumber`
  by construction (the BSA nomogram, outlier item 7, is the known example),
  so the true total is ≥180 and the miss rate for unruled tables is
  **unmeasured**.
- The single captioned miss is physical page 55, captioned `"Bảng 2: Phân
  loại quốc tế các cơn động kinh (1989)"`. Rendering it showed the
  classifier's *structural* verdict was right (one column) but the label
  `not_a_table_degenerate` was semantically wrong — the book numbers it as a
  table, and it is a nested numbered list drawn inside a ruled frame. The
  shape was renamed `single_column_boxed_list` and is counted as a real
  region: single-column content linearises correctly, so it belongs in the
  text, unlike a 2D table. Naming it "not a table" risked a later reader
  discarding it.

**New `ingestion/ingestion/tables/` stage** (`models.py`, `classify.py`,
`detect.py`, `io.py`): table-region detection is production code, not a
scratch script, even though its output is cached (detection takes ≈17
minutes). `pdfplumber` is confined to this module — ADR 0003 established it
must never be used for text on this document. Not yet wired into
`assemble()`; spans inside table regions are still flowing into section body
text.
- Table handling: 200 tables are known but nothing consumes them yet; they
  still flow into section body text as flattened cells (the numeric-row
  regex now reads 0 because rejoining changed the line shape the regex keyed
  on — **that 0 does not mean tables stopped contaminating body text**, and
  claiming otherwise would be wrong).
- Formula detector needs a real precision/recall measurement against a
  golden set before any of its counts are usable.
- Whole-corpus table/formula inventory (`ingestion/scratch/
  inventory_tables_formulas.py`) was still running when this entry was
  written — no counts available yet; `docs/full-coverage-parsing-plan.md`
  has `[chờ đo]` placeholders that must be filled from a real run.
- `chunk/` has no tests yet and has never been executed.
- **Ground truth is not cleaned**: `cli validate`'s 1064-entry denominator
  includes repeated cross-reference index lines (e.g. `"- CoA reductase,
  285"` appears 10+ times in the unmatched list). ADR 0003 used a 725
  denominator, so 91.7% and 92.8% are **not directly comparable**. Neither
  number should be quoted as settled until the ground truth is cleaned.
- Text content accuracy vs. source has still never been measured; the
  recall/precision figures measure monograph-boundary detection only.

---

## 2026-07-31 (cont'd, 4) — Follow-up on the character-diff's remaining unexplained low-similarity pages: sampled 6, all benign/already-known, none newly investigated pipeline bugs

**Scope**: of the ~30-50 pages below 0.95-0.98 similarity left unexplained
by the reversed-column-order investigation (2 entries below), sampled 6 —
1498, 309, 382, 699, 1420, 1369 — chosen to cover the two visible clusters
(1498-1529 near the back-index transition; scattered monograph-range pages)
rather than just the very lowest scores.

**Findings, all benign, none a new production-pipeline bug:**
- **1498, 699**: table/formula content — `opendataloader-pdf` restructures
  it into markdown tables/headings, PyMuPDF's plain text flattens it; same
  underlying content, different presentation. Matches the already-documented
  "no table reconstruction implemented yet" gap (outlier catalog items 7-8),
  not a new finding.
- **309**: the two tools attribute *different* dosing tables to this page
  (PyMuPDF: "Bảng 4" single-agent; opendataloader: "Bảng 3"
  capecitabin+docetaxel combination) — a table-boundary/page-attribution
  disagreement between the two tools, same known gap as above.
- **382, 1420**: the two tools' plain-text page-content genuinely differs
  (different sections of the same drug appear to land on "this page" per
  each tool). **Directly checked against the actual production pathway**
  (`extract_spans()`, dict-mode, already column-sorted) rather than trusting
  the plain-text diff alone: production output for both pages matches
  PyMuPDF's own plain text exactly — the disagreement is opendataloader-pdf
  choosing a different page-boundary cut for overflow text, not a defect in
  this project's pipeline.
- **3, 5, 97**: near/fully blank pages (10-27 chars on one side, 0 on the
  other) — low information content makes the similarity ratio noisy at
  this scale regardless of correctness, not evidence of a real problem.

**Honest scope limit**: only 6 of the ~30-50 unexplained pages were sampled.
All 6 turned out benign or already-documented, which is reassuring but is
not the same claim as "all remaining pages are benign" — that would need
the full set checked, which this session did not do. Investigation scratch
files deleted per CLAUDE.md now that this finding is captured here.

---

## 2026-07-31 (cont'd, 3) — Fixed the boilerplate-leakage bug flagged by the parallel chunking-design session; independently re-verified their numbers before touching any code

**Context**: the parallel session below (ADR 0004 / chunking design) found
and measured a real bug but deliberately left the fix to this session to
avoid a same-file collision. Before writing any fix, independently
reproduced their exact numbers from scratch (not trusted on read) — matched
exactly: 682 monographs, 11,409 sections, 1,374 sections (12.0%) containing
a literal "DTQGVN" string, 671 monographs (98.4%) affected, and the exact
MORPHIN SULFAT `liều lượng và cách dùng` text they quoted. This is the same
discipline applied earlier this session to a mid-session Riboflavin listing
error found in this file — re-verify a reported finding directly against
real data before building on it, even when it looks correct.

**Root cause, confirmed**: `extract/spans.py` already tags the running
header ("DTQGVN 2" + page number + repeated monograph name) as
`column="full_width"`, but nothing in `segment/assembler.py`'s
classification pass excluded it — it matched no section heading and isn't
a real all-caps title, so it fell through into plain body text, landing
mid-sentence whenever a section's text crosses a physical page boundary.
This is exactly outlier-catalog item 13's already-documented risk
("strip the fixed boilerplate before parsing content"), which had a
warning but no enforcing code or test until now — added as item 22 in the
catalog (item 23 also added for the reversed-column bug from the entry
below, which hadn't been given a catalog number yet either).

**Fixed**: new `assembler._is_page_boilerplate(span)` — drops any span with
`column == "full_width"` and `y0 < HEADER_BAND_Y` (same header-band
threshold `page_map.py` already uses for folio detection; exported that
constant as public rather than duplicating the magic number) before any
other classification. Regression test added using the real MORPHIN SULFAT
span shape (`tests/test_segment_assembler.py`).

**Whole-corpus re-measurement after the fix**: 0 of 11,409 sections contain
"DTQGVN" (was 1,374). `cli validate` unchanged: 682 monographs, 92.8%
recall, 99.1% precision — the fix only touches body-text content, not
monograph/section boundaries. 110 tests total (was 109), all passing.

**Not done yet / next up:**
- Chunking (ADR 0004, the parallel session's design) can now safely run
  against real ingestion output for this specific defect — but see the
  entry below's own "not done yet" list (sub-chunk splitter not built,
  general-chapters/appendices scope, sub-compound tagging) for what's still
  actually blocking Phase 2 beyond this fix.
- Only checked for the literal "DTQGVN" substring as this bug's signature
  — did not separately verify whether the page-number token alone (without
  "DTQGVN" adjacent) ever leaks in some other layout shape; the fix itself
  is structural (column+y-position, not text-pattern-based) so it should
  cover that too, but this wasn't independently re-measured after the fix
  with a different detection signature.

---

## 2026-07-31 (cont'd, parallel session) — Phase 2 chunking strategy designed (ADR 0004) from real per-section measurements; found and flagged a new whole-corpus boilerplate-leakage bug for the extract/segment session to pick up

**Context**: this entry comes from a second session running in parallel with
the one still fixing `extract`/`segment` parsing bugs, on the same checkout
(no worktree separation). Per explicit scoping agreed with the user, this
session touched **only** `docs/adr/0004-chunking-strategy.md` (new),
`docs/architecture.md`'s chunking paragraph, this log entry, and a
since-deleted scratch script — it did not touch `extract/*.py`,
`segment/*.py`, or `docs/document-profile.md`, to avoid colliding with the
other session's in-flight edits to those files.

**Done:**
- Ran `python -m ingestion.cli run` for real (full 1668-page PDF) to produce
  `ingestion/data/processed/monographs.jsonl` (682 monographs — gitignored
  output, matches the count already reported elsewhere in this log), then
  measured real per-section text-length distribution across the whole
  corpus for the first time (`ingestion/scratch/chunking_stats_survey.py`,
  now deleted per this project's investigation-script rule, findings
  captured below and in the ADR).
- **Replaced the never-validated chunking guess in `docs/architecture.md`**
  (`(drug, section)` unit, ~500-800 tokens, 400-tok/50-overlap sliding
  window — written before segmentation existed) with a design grounded in
  the real measurement: `(drug_id, section_key)` chunk unit confirmed;
  800-token ceiling (chars/4 estimate) confirmed as directionally right
  (clears ~16/18 section types at p90); **but sub-chunking is the routine
  path, not a rare hedge, for 2 specific sections** — `dược lý và cơ chế
  tác dụng` (242/678 monographs with that section, 35.7%, max ≈3542 est.
  tokens) and `liều lượng và cách dùng` (200/675, 29.6%, max ≈3631 est.
  tokens); a smaller tail also exceeds it (`thận trọng` 3.7%, `tương tác
  thuốc` 3.4%). Chosen sub-chunking method: **sentence-boundary-aware**
  sliding window (~600-700 tok/sub-chunk, ~50-80 tok overlap), not a blind
  character/line window — `assembler.py`'s `body_lines` join one PDF
  visual line-wrap per line, not a semantic boundary, so a blind window
  risks splitting a dosing sentence mid-way (a real, measured risk given
  outlier item 17: adult/child dosing splits appear on 1,121/~1,400
  monograph-range pages). Full rationale, extended chunk metadata schema
  (`chunk_id`, `atc_codes`, `part_index`/`part_count`, etc.), and 4
  explicitly-flagged open gaps (sub-compound tagging inside class-level
  monographs, sub-chunk page-precision, the splitter itself not yet built,
  general-chapters/appendices chunking out of scope) are in
  `docs/adr/0004-chunking-strategy.md`.
- **Found and measured a new whole-corpus bug, not yet fixed, flagged here
  for the `extract`/`segment` session rather than fixed directly** (per
  user's explicit choice this session, to avoid a same-file collision):
  running header/footer boilerplate ("DTQGVN 2" + page number + repeated
  drug name — tagged `column="full_width"` in `extract/spans.py`) is never
  filtered out of section body text; `assembler.py` appends every
  non-title, non-section-heading span to `body_lines` regardless of column
  tag. Measured whole-corpus: **1,374 of 11,409 sections (12.0%) contain a
  literal "DTQGVN" string mid-text; 671 of 682 monographs (98.4%) have at
  least one affected section.** Real example: MORPHIN SULFAT's `liều lượng
  và cách dùng` reads `"...Nếu\nDTQGVN 2\n1009\nMorphin sulfat\nuống viên
  thuốc..."` — the page number and drug name are spliced mid-sentence into
  a real dosing instruction. This is `docs/pdf-parsing-outlier-catalog.md`
  item 13's already-documented risk ("header/footer boilerplate must be
  stripped"), just never actually measured/fixed until this session — it
  should become a new numbered item in that catalog (item 22, or the next
  free number by the time this is read — check the catalog directly) with
  these numbers, but that file is mid-edit in the other session so this
  entry leaves the actual catalog edit to them rather than risking a
  concurrent-write collision. Note: this bug is **separate from** the
  reversed-column-order bug documented in the entry directly below this
  one — that bug was about which *column* content lands in, this one is
  about full-width header-band content never being excluded from body text
  regardless of column. **This is a hard blocker for Phase 2**: chunking
  must not run against real ingestion data until this is fixed, or
  boilerplate gets baked into embeddings and can surface mid-sentence in a
  chunk shown to a doctor/pharmacist.

**Not done yet / next up:**
- The boilerplate-leakage bug above needs a real fix in `extract`/`segment`
  (likely: exclude `column="full_width"` spans from body-text assembly,
  or an explicit boilerplate-pattern filter) plus a regression test and a
  whole-corpus re-measurement to confirm it's actually gone — not done by
  this session, left for whoever owns `extract`/`segment` next.
- `ingestion/ingestion/chunk/` still doesn't exist — ADR 0004 is a design
  only; implementing and unit-testing the sentence-boundary splitter is a
  separate task.
- Chunking design for general chapters (pp. 37-98) and appendices (pp.
  1497-1528) is still blocked on `docs/document-profile.md`'s Group 2
  investigation (tables, 2D stacked-fraction formulas) completing.
- Sub-compound tagging inside class-level/multi-ATC monographs (25.5% of
  corpus) has no design yet — flagged in ADR 0004, deferred to
  golden-dataset-driven eval.

---

## 2026-07-31 (cont'd, 2) — Built a whole-document cross-tool character-diff QA check; it found a real, serious cross-monograph data-corruption bug (reversed column reading order), now fixed and whole-corpus-reverified at zero occurrences

**Why this check was built:** after the ATC-field bug-fixing session below, the
user asked what validation step would catch whether parsing is "correct" at
all — not just "does `cli validate` say recall/precision are high," since
that check only confirms a monograph *exists* at roughly the right name/page,
not that its *content* is complete and correctly attributed. Per
[[feedback-rigorous-validation]], comparing PyMuPDF's own output against
itself can't validate itself — a second, independently-implemented parser
is required as real ground truth. Built a whole-document (all 1668 pages)
per-page character-similarity diff: PyMuPDF's `page.get_text()` vs
`opendataloader-pdf`'s markdown extraction, normalized and compared with
`difflib.SequenceMatcher`.

**Two bugs in the check script itself, found and fixed before trusting any
result (disclosed to the user immediately on discovery, not after):**
1. Wrong page-separator placeholder syntax (`{page}` instead of the tool's
   real `%page-number%`) risked silent page misalignment. Fixed by using the
   real placeholder and parsing the actual page number from each separator
   instead of assuming positional order.
2. Python's `difflib.SequenceMatcher` default `autojunk=True` collapsed the
   similarity ratio to ~0.0065 for a page whose content was actually ~98%
   identical between tools (a long drug-name list trips its "popular
   element" heuristic) — a well-known stdlib gotcha. Fixed with
   `autojunk=False`.

**Whole-document result** (1668/1668 pages compared, mean 0.9892, median
0.9981): a tight cluster of pages — 929, 1099-1106, 1149-1153 — scored only
~0.47-0.53. Investigated instead of dismissed.

**Confirmed real, serious bug in `extract/spans.py`:** the module trusted
PyMuPDF's raw block iteration order to already sequence left-column-before-
right-column, validated only against one example page back in ADR 0003.
Wrong on **12 of 1398 monograph-range pages** (whole-range scan, e.g.
physical page 1100): PyMuPDF's raw block order emits the *right* column
before the *left* column there. Confirmed by rendering the page to an image
and reading it directly, then confirmed in the actual `assemble()` output:
OXYMETAZOLIN's right-column sections (Chống chỉ định, Thận trọng, Thời kỳ
mang thai, Thời kỳ cho con bú, ADR, Hướng dẫn xử trí ADR, Liều lượng và
cách dùng) were being silently attributed to and overwriting the still-open
OXYBUTYNIN monograph's own sections, while OXYMETAZOLIN ended up missing
all 7. Confirmed boundary pairs affected: OXYBUTYNIN/OXYMETAZOLIN,
OXYTETRACYCLIN/OXYTOCIN, OXYTOCIN/PACLITAXEL, PIOGLITAZON/PIPECURONIUM
BROMID; MAGNESI SULFAT, PILOCARPIN, and PACLITAXEL had internal (not
necessarily cross-monograph) ordering corruption. **This is a real,
medical-content-relevant defect** — wrong contraindication/ADR data
silently attached to the wrong drug — not a cosmetic parsing issue.

**Fixed** by explicitly sorting blocks (full_width header band first, then
left column, then right column, each by y-position) instead of trusting
PyMuPDF's raw order. Verified: re-scanned the full 99-1496 range for the
same reversed-order signature — 0 occurrences (was 12). Directly verified
OXYBUTYNIN's and OXYMETAZOLIN's `assemble()`-produced sections are now
distinct and drug-appropriate (spot-checked against the rendered page).
Whole-book `cli validate` after the fix: unchanged at 682 monographs,
92.8% recall, 99.1% precision, 8 zero-ATC (no regression). Also tried a
broader "any within-column y-order violation" scan (670 pages flagged) but
verified a sample and found it's dominated by benign subscript/superscript
baseline noise (e.g. "B" + subscript "6" + ")"), not real bugs — correctly
discarded as evidence rather than reported as 670 new findings.

**Regression test** added (`tests/test_extract_spans.py`) using the exact
real bounding boxes from physical page 1100's raw block order. 109 tests
total (was 103), all passing.

**Not done yet / next up:**
- The whole-document character-diff tooling itself was investigation-only
  (per CLAUDE.md, deleted from `ingestion/scratch/` after this finding was
  captured here + in the regression test + in `spans.py`'s docstring) — if
  this kind of check is wanted as a recurring QA step, it needs to be
  rebuilt as a real `ingestion/validation/` module, not re-derived ad hoc
  each time.
- The character-diff still has ~30-50 pages below a 0.95-0.98 similarity
  threshold that were *not* individually investigated this session (only
  the most extreme cluster was) — front-matter table-like pages (14-31),
  the back-index transition region (1498-1529), and scattered others
  (382, 1420, 57, 68, 309, 194, ...) remain unexplained; could be genuine
  table/formatting differences neither tool handles perfectly, not
  necessarily more instances of this same bug (the specific reversed-column
  signature was already whole-range-scanned to exhaustion above).
- Phase 1.5 (golden dataset) still requires human review by design.
- Phase 2 (chunking) has no code yet and no design decision made.

---

**Correction to the previous entry below, per CLAUDE.md's "never fabricate"
rule:** re-running `assemble()` fresh at the start of this session (same
code, nothing had changed on disk) produced **676** monographs and **48**
zero-ATC-not-stated-absent, not the "680 / 46" the previous entry claimed —
and that entry also self-contradicted (46 in one line, 42 two paragraphs
later). Root cause: the previous session's final numbers were asserted
without a fresh re-run after the very last code edit. No `monographs.jsonl`
artifact existed to diff against, so this can't be proven beyond doubt, but
it's the only explanation consistent with the evidence. Lesson applied
going forward: a number is only "final" if it comes from a command run
*after* the last related edit, in the same message reporting it.

**Method used this session, per two user corrections mid-session**: initial
passes relied only on PyMuPDF span text and coordinate reasoning. The user
first pointed out other installed PDF tools were going unused and that
pages should be rendered to images and read directly rather than trusted
from span dumps alone (per [[feedback-visual-verification]]) — so a first
cross-check used `pdfplumber.extract_text()` plus rendered-page-image
reads. The user then flagged this as still not matching "the strategy from
before." That strategy already existed, in full, in the
[[pdf-parsing-strategy]] memory and `docs/adr/0003-pdf-parsing-strategy.md`:
**4 tools were already evaluated there** (PyMuPDF, pdfplumber,
opendataloader-pdf, docling), and it already concluded
**`pdfplumber.extract_text()` scrambles reading order on this document's
two-column layout and must never be used for general text** — only
PyMuPDF (primary) and `opendataloader-pdf` (independent reading-order +
font-metadata cross-check) are validated for that purpose. The
`MEMORY.md` index line for that memory doesn't carry this detail, only the
full memory file does — this session used the one-line index and never
opened the full file before picking a cross-check tool, which is the actual
process gap (not a memory-setup gap). All findings below were then
re-verified with `opendataloader-pdf` instead, and the earlier pdfplumber
pass was discarded as unreliable evidence, not cited.

**Investigated and closed** (whole-book `cli validate` against the real
back-of-book index, not a sample):
- The 8 detected monographs that didn't match any back-index entry: 2 were
  a real bug in `validation/metrics.py` (substring name-matching let a
  shorter monograph name, e.g. "ISOSORBID", "steal" the ground-truth match
  meant for a longer, textually-overlapping but genuinely distinct
  monograph, e.g. "ISOSORBID DINITRAT" — both are real, correctly segmented
  drugs). Fixed: try an exact normalized-name match before falling back to
  substring. The other 6 are real book-internal inconsistencies, not
  pipeline bugs (compound names containing " - " skipped by the
  already-documented cross-reference filter; title-vs-index spelling
  variants like "HYDROGEN PEROXID" vs the index's "Hydrogen peroxyd").
- The 83 unmatched ground-truth entries: ~40 are back-index line-wrap
  parsing artifacts ("- CoA reductase" / "gonadotropin" fragments from
  wrapped cross-reference lines, not real entries), ~20 are front-matter/
  general-chapter TOC entries (pages 39-98, before the monograph range even
  starts at printed page 99) that `back_index.py` doesn't filter out, a
  handful are the same title-vs-index spelling-variant pattern as above —
  and **7 were genuinely missing monographs**, root-caused to 2 real bugs
  (see below) plus one real book typo (CARBAMAZEPIN's own printed heading
  reads "Ten chung quốc tế", missing the "ê" — confirmed independently by
  both a rendered-page-image read and `opendataloader-pdf`'s text output,
  which shows the same missing "ê"; not fixable without risking false
  positives elsewhere, left as-is).

**4 real bugs found and fixed, each confirmed via a whole-corpus scope
check (not just the sample that surfaced it) and, where the defect could be
page-rendering vs data, a rendered-page-image visual check:**
1. **Same-line diacritic span-fragmentation** (`segment/merge.py`,
   `merge_same_line_bold_fragments`, new): PyMuPDF splits some bold spans
   into multiple fragments around diacritic characters even when the text
   is one unbroken visual line — confirmed by rendering physical page 759
   to an image ("Tên chung quốc tế" looks completely normal to a human
   reader). Cross-checked against `opendataloader-pdf` (the tool
   [[pdf-parsing-strategy]]/ADR 0003 already validated for this — not
   pdfplumber, which that ADR found scrambles reading order on this
   document's two-column layout) on 2 of the 5 affected pages (759
   GUAIFENESIN, 943 MEPHENESIN): both reconstruct the line cleanly, e.g.
   "Tên chung quốc tế: Mephenesin. Mã ATC: M03BX06." with no fragmentation,
   confirming this is a PyMuPDF span-boundary artifact, not a defect in the
   PDF itself. **Correction**: an earlier version of this entry claimed all
   6 candidate pages were cross-checked and listed RIBOFLAVIN among them —
   both wrong. Only 2 of the 5 real pages were actually re-verified with
   opendataloader-pdf just now, and RIBOFLAVIN's failure is the separate
   folio-subscript bug below, not this one — it was never part of the
   diacritic-fragmentation set. Broke the anchor check that gates
   false-positive title filtering, silently dropping whole monographs.
   Confirmed for 5 real monographs (GUAIFENESIN, MEPHENESIN, NATRI
   THIOSULFAT, RAMIPRIL, TENOXICAM) via a full 1668-page scan for the
   fragment signature; the other 3 (NATRI THIOSULFAT, RAMIPRIL, TENOXICAM)
   were not independently cross-tool-verified, only confirmed via PyMuPDF's
   own span coordinates (same-line y-gap).
2. **Folio-detection false conflict** (`extract/page_map.py`, `pick_folio`):
   RIBOFLAVIN's monograph sits high enough on physical page 1243 that its
   own "2" subscript (from "Vitamin B₂", font size 5.83) falls inside the
   header band alongside the real folio "1244" (size 10.0), producing two
   conflicting digit candidates and silently dropping the printed page —
   and the whole monograph with it. Fixed by preferring the largest-font-
   size candidate(s) (a real folio is always set in the header's own
   running size, never a subscript's reduced size); a full-document scan
   confirmed this exact conflict shape occurs on exactly 1 of 1668 pages.
   Confirmed visually by rendering the page.
3. **ATC comma-inside-annotation** (`segment/atc.py`): the field-text
   split on "," ran *before* parenthetical annotations were stripped, so
   an annotation containing its own comma broke the split — e.g. "Mã ATC:
   J07BD01 (Measles, live attenuated)." split into two unrecoverable
   fragments. INSULIN's earlier-fixed Vietnamese annotations ("người",
   "bò") never contain a comma, so this only surfaced with vaccines'
   English annotations — affected 12 vaccine monographs. Fixed by stripping
   *all* parenthetical groups before splitting, not just a trailing one
   per already-split segment. `opendataloader-pdf` cross-check on the real
   VẮC XIN SỞI page confirms the source text genuinely is "Mã ATC: J07BD01
   (Measles, live attenuated)." — the bug was in parsing, not the data.
4. **ATC leading colon from the value span** (`segment/atc.py`): some
   monographs render the bold label as "Mã ATC" (no colon) with the colon
   on the plain *value* span instead (": M03AA01." vs Abacavir's "J05AF06."
   with the colon on the label side) — the section still matched correctly,
   but the leftover leading colon made the stripped candidate 8 characters
   instead of 7, failing the length check. Fixed by stripping a leading
   colon in `normalize_atc_candidate`, symmetric with the existing trailing
   strip. Affected 15 monographs. `opendataloader-pdf` cross-check on the
   real ALCURONIUM CLORID page confirms clean source text ("Mã ATC:
   M03AA01."), same conclusion.

5. **ATC name-prefixed and reversed "CODE: Name" shapes** (`segment/atc.py`,
   same session, found continuing the zero-ATC investigation after the
   above): two more real shapes surfaced once the first 4 fixes cleared the
   noise. (a) 7 monographs with multiple salt/ester forms write each form
   as "Name: CODE" per line, e.g. ARGININ's "Arginin glutamat: A05BA01\n
   Arginin hydroclorid: B05XB01" — the whole segment including the name was
   compared against the 7-char code shape and rejected. (b) The class-level
   "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" monograph writes it the *opposite*
   way, code first — "C10A A01: Simvastatin\nC10A A02: Lovastatin\n...".
   Fixed both with one change: `normalize_atc_candidate` now tries the text
   after the last ":" first, then before, returning whichever side actually
   normalizes to a valid ATC shape — safe because a real drug name never
   happens to match the strict `[A-Z]\d{2}[A-Z]{2}\d{2}` pattern, so there's
   no real ambiguity between the two candidates in practice.

**Net effect, whole-book, before -> after all 5 fixes:**
detected monographs 676 -> **682**; recall 92.2% -> 92.8% (981 -> 987 /
1064); precision 98.8% -> **99.1%**; zero-ATC-not-stated-absent 48 -> **8**.
103 tests total (was 88 at the start of this entry), all passing, each new
fix with a regression test built from the exact real-corpus text that
exposed it.

**The remaining 8 zero-ATC monographs are now all explained, none left
unresolved:**
- 7 (CROTAMITON, INTRALIPID, ISOSORBID, OXYBENZON, PEMIROLAST, SIMETICON,
  the DPT vaccine) have **no "Mã ATC" section anywhere in the book at
  all** — confirmed by reading the actual span sequence after each title
  (goes straight from "Tên chung quốc tế"/"Loại thuốc" to the next section,
  no ATC line ever appears) and by rendering physical page 845 (ISOSORBID)
  to an image and reading it directly. A real, accepted data gap in the
  source — not a parsing bug.
- 1 (SPECTINOMYCIN) is a confirmed real book typo: its own printed heading
  reads **"Mã ACT:"** (letters transposed), not "Mã ATC:" — confirmed by
  rendering physical page 1297 to an image and reading it directly. Same
  category as CARBAMAZEPIN's "Ten chung quốc tế" typo from fix 1 above:
  a real defect in the source document, left unfixed rather than loosening
  vocabulary matching and risking new false positives elsewhere (the
  project's own prior "whack-a-mole" experience with over-loosened
  matching, per outlier-catalog item 21).

**Not done yet / next up:**
- `validation/back_index.py`'s line-wrap and front-matter-entry issues
  (from the investigation above) inflate the "unmatched ground truth"
  count but were left unfixed this session — the user's stated priority
  was the segmentation-pipeline bugs first, not the validation-metric's
  own accuracy.
- The docs/pdf-parsing-outlier-catalog.md items for these 5 new bugs have
  not been added yet (the module docstrings for `merge.py`, `page_map.py`,
  and `atc.py` carry the full evidence in the meantime).
- Only 2 of the ~7 diacritic-fragmentation pages and 2 of the ~15
  leading-colon pages were independently cross-tool-verified with
  opendataloader-pdf (see fix 1's correction note above) — the rest rely on
  PyMuPDF's own span coordinates only, which is weaker evidence.
- No exploration yet of whether the same fragmentation/folio/colon bug
  families affect *other* sections beyond "Tên chung quốc tế" and "Mã
  ATC" (e.g. "Chỉ định", "Liều lượng và cách dùng") — only ATC was swept
  whole-corpus this session.
- Phase 1.5 (golden dataset) still requires human review by design.
- Phase 2 (chunking) has no code yet (`ingestion/chunk/` doesn't exist) and
  no design decision has been made on chunking strategy.

---

## 2026-07-31 — Phase 1.3-1.4 built: assembler, CLI, and validation, with 4 more real bugs found and fixed via whole-book runs

**Done (continuation of the same session, user asked to keep driving
autonomously via `/loop`; visual PDF-page rendering used throughout to
self-verify bugs, per [[feedback-visual-verification]]):**
- Built `assembler.py` (3-pass design: classify spans -> coalesce titles ->
  build Monograph records), `segment/io.py` (JSONL read/write), `cli.py`
  (`run` and `validate` subcommands working end-to-end), and
  `validation/back_index.py` + `metrics.py` (recall/precision against the
  real back-of-book index, parsed from real physical pages 1530+).
- **Found and fixed 4 more real bugs via whole-book `assemble()` runs**,
  each initially surfaced as a wrong number (never trusted the first
  result, per CLAUDE.md):
  1. **ATC trailing-period bug**: "Mã ATC: J05AF06." — the sentence-ending
     period was counted as part of the code, so `normalize_atc_candidate`
     silently returned zero codes for every single-ATC monograph ending in
     "." (a huge fraction of the corpus). Fixed by stripping trailing
     `.,;` before the length check.
  2. **ATC species-annotation bug**: INSULIN's real field lists all 20
     codes each with a parenthetical annotation ("A10AB01 (người); ...") —
     only 2 of 20 survived before the fix (the two that happened to have a
     line-wrap between code and annotation). Fixed by stripping a trailing
     `(...)` group before normalizing. Whole-corpus multi-ATC re-count with
     both fixes: **159/680 (23.4%)** monographs have >1 ATC code (the
     open item from the very first survey session, now closed with a real
     measured number instead of the 25.4%-floor estimate).
  3. **Non-bold combined section heading (outlier item 20)**: AMITRIPTYLIN's
     "Mã ATC:" is a single **non-bold** span combining label and value
     ("Mã ATC: N06AA09."), unlike Abacavir's bold-label-only span — the
     book's ~700 monographs were written by many different authors, so
     styling isn't 100% consistent. Fixed by matching section headings by
     vocabulary **text**, not `span.bold`, plus a new
     `match_section_with_inline_value` for the combined-span case.
  4. **Mixed-case title + false-positive whack-a-mole (outlier item 21)**:
     the class-level monograph "CÁC CHẤT ỨC CHẾ HMG-CoA REDUCTASE" embeds
     the mixed-case abbreviation "CoA", which a strict `isupper()` check
     silently dropped from the corpus entirely. Loosening that check (first
     with an absolute lowercase-count tolerance, found wrong, then fixed
     with a **lowercase-letter ratio** instead — "Mã ATC:" has 1/5 = 20%
     lowercase, correctly still rejected, vs. HMG-CoA's 1/27 ≈ 3.7%) then
     exposed a *second* false positive: individual statin sub-headings
     ("SIMVASTATIN", "LOVASTATIN", ...) inside that same class monograph,
     each followed by their own real section but never by "Tên chung quốc
     tế" specifically. The anchor check (added earlier for the HSV/CMV
     table-header false positive, item 19) had been loosened to "any
     section" to pass existing tests — reverted to requiring "Tên chung
     quốc tế" specifically (the one invariant the book's own template
     actually guarantees), and fixed the test fixtures instead of the
     production logic.
- Final whole-book numbers after all fixes: **680 monographs** (matches
  the previously-established count from the original structural survey —
  though this is a count match, not yet a confirmed identical-set match).
  Abacavir ATC now correctly `["J05AF06"]`; Insulin now correctly 20 codes.
  46 monographs remain zero-ATC-and-not-stated-absent (down from an
  initial 48; not yet root-caused further — flagged, not silently accepted
  as final).
- 86 unit tests total, all passing, including a regression test for every
  bug above and for each whack-a-mole cycle (so a future change can't
  silently reintroduce SIMVASTATIN-as-monograph or Mã-ATC-as-title).

**Not done yet / next up:**
- The remaining 42 zero-ATC-not-absent monographs likely hide at least one
  more real pattern (per this session's track record of "one fix reveals
  the next") — worth one more investigation pass before Phase 1.5.
- Phase 1.5 (golden dataset) still requires human review by design — not
  something this session can complete alone, per the approved plan.

---

## 2026-07-31 — Session end: golden dataset NOT started; general chapters + appendices NOT investigated

**Status check requested by user at end of session ("golden dataset bạn để
đâu?" / "đã xem chuyên luận chung và phụ lục chưa?") — answering plainly
here so the next session doesn't have to guess:**

- **Golden dataset (Phase 1.5): not created.** `ingestion/data/qa/` still
  contains only `.gitkeep` — no `golden_pages.jsonl`, no
  `golden_monographs.jsonl`. This is intentional, not an oversight: per the
  approved plan, golden-set ground truth requires human review/sign-off,
  which this session couldn't do alone (dynamic `/loop` autonomy stopped
  here for exactly this reason). [[feedback-visual-verification]] means a
  future session can self-draft much of it (render pages, read them
  directly) but a human still needs to spot-check before it's trustworthy.
- **"Các chuyên luận chung" (general chapters, printed pages 37-98) and
  "Các phụ lục" (appendices, printed pages 1497-1528): NOT investigated
  this session, or any prior session.** All work so far (extract/segment/
  validation, ADR 0003, the outlier catalog) covers only the drug-monograph
  range (printed 99-1496). The only contact with these two ranges was
  incidental: reading physical page 38-39 (inside general chapters) once
  to transcribe the book's own 19-field section template into
  `segment/vocab.py`, and skimming physical ~1526-1528 (inside the
  appendices — specifically "Phân loại thuốc theo mã ATC") only to locate
  where the back-of-book index begins for `validation/back_index.py`.
  Neither range has been structurally surveyed, outlier-cataloged, or
  parsed. This gap has been flagged since the *very first* scaffold session
  (`docs/progress-log.md`'s original Phase 1 roadmap) and remains
  explicitly out of scope of the plan approved this session.
  Known content, not yet verified in depth: general chapters cover topics
  like "Kê đơn thuốc," rational antibiotic use, pediatric dosing
  principles; appendices include the body-surface-area nomogram table
  (already flagged in outlier catalog item 7 as a 2D-table extraction
  problem), IV-admixture compatibility info, and the ATC drug
  classification listing.

**Next session should pick up one of:**
1. Golden dataset drafting (Phase 1.5) — scaffold from current
   extraction/segmentation output, self-verify via page rendering, then
   get human sign-off before trusting it.
2. A first real structural investigation of general chapters + appendices
   (same rigor bar as the monograph range: whole-range scan, not a page or
   two) — needed before any chunking strategy can be designed for them.
3. The 42 remaining zero-ATC-not-absent monographs (Phase 1.4 leftover,
   not blocking).

---

## 2026-07-31 — Phase 1.4 real validation run: 92.2% recall, 98.8% precision (first-ever measurement)

**Done:**
- Ran `python -m ingestion.cli validate` for real against the full
  1668-page book. First result: 91.7% recall / 98.2% precision against
  1064 real back-index ground-truth entries (parsed from physical pages
  1530+, not a sample) — recall matched ADR 0003's original number exactly
  (665/725 there was a different, smaller ground-truth set; this run's
  1064 entries come from parsing the *entire* back index, not a partial
  scan), and **precision was measured for the first time ever** on this
  project, meeting the plan's ≥98% target immediately.
- **Found and fixed one more real bug from this first real run**: 4 of 12
  unmatched detected monographs (ALVERIN CITRAT, OXYMETAZOLIN HYDROCLORID,
  TERBUTALIN SULFAT, TIOTROPIUM BROMID) all shared the same shape — a
  **double space** in the detected title (e.g. "ALVERIN  CITRAT") that
  failed to match ground truth's single-spaced "Alverin citrat" under
  plain strip+upper comparison. Fixed by collapsing whitespace in
  `metrics._normalize_name` before comparing.
- Final numbers after the fix: **recall 92.2% (981/1064), precision 98.8%**
  — both real, measured, whole-book numbers, both improving over the
  already-fixed run (not just over the pre-session 91.7% baseline).
- Remaining unmatched entries are traced to two already-documented, known
  limitations rather than new bugs: (1) `back_index.py`'s own stated
  trade-off of treating any " - " as a brand-cross-reference marker also
  excludes genuine compound-name ground-truth entries ("Carbidopa -
  levodopa", vaccine names like "Vắc xin DPT" that use " - " internally),
  so a handful of correctly-detected monographs (CARBIDOPA - LEVODOPA,
  THUỐC PHIỆN - OPIAT - OPIOID, the DPT/MMR vaccine entries) simply have no
  matchable ground-truth counterpart, not a detection defect; (2) a
  repeating "- CoA reductase, 285" ground-truth artifact (appears ~12
  times) is itself index-parsing noise — likely a long cross-reference
  line wrapping across two physical lines in a way that splits the brand
  name from its "- CoA reductase" continuation, which then doesn't contain
  the " - " marker at its own line start and slips through the
  cross-reference filter as a bogus ground-truth entry.
- 87 unit tests total, all passing.

**Not done yet / next up (Phase 1.5, requires human review by design —
not something a single session can complete alone per the approved plan):**
- Golden dataset authoring: `scaffold-golden` CLI command, golden_pages/
  golden_monographs JSONL schemas, human review of drafted entries.
- The 42 remaining zero-ATC-not-absent monographs and the back_index.py
  compound-name/cross-reference-wrapping noise above are both flagged, not
  blocking — real, moderate-size gaps documented for whoever picks this up
  next.

---

## 2026-07-30 — Phase 1.2 `segment/` pure logic built and validated against real PDF

**Done (real production code, all reused by both the future CLI pipeline
and validation — no logic duplicated):**
- Transcribed the book's own documented 19-field monograph template
  verbatim from its source (physical page 38/39 printed, "HƯỚNG DẪN SỬ DỤNG
  DƯỢC THƯ QUỐC GIA VIỆT NAM") into `vocab.py`'s `SECTION_DEFS`, rather than
  guessing — cross-checked against real bold headings in the Abacavir/
  Acarbose monographs (exact text match, modulo a trailing colon some pages
  have and others don't, now normalized). Added `ten_thuong_mai` ("Tên
  thương mại") as the confirmed 19th, undocumented-but-real field.
- Built `merge.py` (multi-line/multi-fragment title merging), `detector.py`
  (monograph + section boundary detection), `atc.py` (3-state ATC
  extraction: found / recovered-from-noise / stated-absent), `units.py`
  (defensive mg/mcg/mmol validation — see below), `models.py`.
- **Found and fixed a second real title-fragmentation bug by rendering a
  page to an image and reading it directly** (not just reasoning from
  coordinates): "ACICLOVIR" was detected as two separate titles, "ACIC"
  (font size 10.0) and "LOVIR" (font size 9.5) — the same visual word
  rendered at two slightly different sizes in the source PDF. The merge
  logic originally required exact font-size equality (which happened to
  work for the GONADOTROPIN wrap case since both its fragments are size
  9.5) — dropped that requirement per the same "font size is not reliable"
  lesson from ADR 0003, now applied *within* a title's own fragments, not
  just across monographs. Also fixed the join character: a genuine
  same-line split needs no space ("ACIC"+"LOVIR"="ACICLOVIR"); a genuine
  multi-line wrap needs one (GONADOTROPIN case) — distinguished by the y0
  gap. This same fix also resolved two other silent duplicate-name
  artifacts (HSV, CMV) found in the same smoke test.
- Smoke-tested the full detector against the real PDF: 695 monograph titles
  detected (down from 702 pre-fix, closer to the previously-established
  ~680 count), part-divider correctly excluded, ABACAVIR/INSULIN present,
  GONADOTROPIN wrap correctly merged, zero unexplained duplicate names.
- **Investigated the one remaining duplicate name ("SALBUTAMOL", pages 1261
  and 1263) by rendering both pages and reading them directly — confirmed
  it is NOT a bug**: two genuinely different, complete monographs
  ("Dùng trong hô hấp" / respiratory vs. "Dùng trong sản khoa" / obstetric
  use), each with a full 18-section template. Added as outlier-catalog item
  18 with an explicit note for Phase 1.3's assembler: `drug_id` generation
  must fold in the bold, non-all-caps qualifier line beneath the title, or
  it will wrongly treat this legitimate case as a duplicate-title collision.
- Confirmed via a targeted regex scan that the `units.py` whitespace-split
  defense (built by analogy to the confirmed ATC defect) has **zero**
  confirmed real occurrences in this corpus so far — documented honestly as
  a defensive-only check, not a confirmed defect, per CLAUDE.md.
- 44 unit tests total (up from 9), all passing, including regression tests
  for every real bug found this session (kerning jitter, column-merge,
  GONADOTROPIN wrap, ACICLOVIR same-line split).
- Rendering a PDF page to an image and reading it directly (not just
  reasoning from PyMuPDF coordinates) turned out to be a fast, reliable way
  to self-verify segmentation bugs — used for both real bugs found this
  phase (ACICLOVIR, SALBUTAMOL) without needing a human to look at the page.
  This changes the Phase 1.5 golden-dataset plan: much of the
  ground-truth drafting can be self-verified this way before a human spot-
  checks it, rather than requiring a human to author it from scratch.

**Not done yet / next up:**
- Phase 1.3: `assembler.py` (must handle the SALBUTAMOL qualifier-line case
  above), `segment/io.py`, `cli.py run`, wired end-to-end; smoke-test on a
  small page range before a full-book run.
- Phase 1.4: `validation/back_index.py` + `metrics.py` (recall/precision
  against the back-of-book index), `cli validate`.

---

## 2026-07-30 — Phase 1.1 `extract/` module built and validated against real PDF

**Done (real production code, not exploratory scripts — replacing the
empty `ingestion/ingestion/extract/` stub per the approved segmentation +
eval plan):**
- Built `models.py` (`Span` dataclass), `page_map.py` (physical→printed page
  mapping, read per-page rather than assumed as a constant — verified
  correct and constant at +1 across all tested milestone pages: physical 0,
  36, 37, 98, 100, 1496, 1497, plus correctly returns `None` for blank/title
  pages), `spans.py` (continuous cross-page span stream with column
  tagging), `io.py` (JSONL persistence), and `glyph_order.py` (the
  mandatory pre-ingestion sanity gate).
- Added `pytest`/`pymupdf` to `ingestion/pyproject.toml` (previously empty
  `dependencies = []`) plus `[tool.setuptools.packages.find]` to fix a
  package-discovery ambiguity that broke `pip install -e .` — both
  confirmed via a real editable install, not just added and assumed to work.
- **Corrected a real gap in ADR 0003's own validated finding**: re-verifying
  the "reversed glyph order" defect as real tested code (not trusted from
  the prior exploratory script) found **2 genuine occurrences, not 1**
  (physical pages 714 and 1373 — two different defect shapes, see outlier
  catalog item 9's rewrite for full detail). Getting a trustworthy count
  took 3 detector iterations after the first naive whole-book run reported
  1113 false positives (kerning jitter + a column-boundary false-merge bug)
  — full false-positive history and the fix (group by PyMuPDF's own block
  index, not hand-picked x-coordinates) documented in
  `extract/glyph_order.py`'s docstring and the outlier catalog.
- Smoke-tested `extract_spans`/`build_page_map` against the real PDF:
  253,518 spans extracted, 30,728 bold, first monograph title (ABACAVIR)
  correctly located at physical page 100 / printed 101.
- 9 unit tests added (`tests/test_extract_glyph_order.py`), all passing,
  including regression tests for the kerning-jitter and column-merge false
  positives found during validation (so they can't silently regress).

**Not done yet / next up:**
- Phase 1.2: `segment/` pure logic (vocab, merge, detector, atc, units) with
  unit tests reproducing every documented bug case (GONADOTROPIN wrap,
  part-divider false positive, ATC whitespace/O-0, "Chưa có" state) — see
  the approved plan (`ingestion/ingestion/segment/` is still an empty stub).
- The 3 formula-region pages (92, 94, 805) that also trip
  `scan_reading_order` should **not** have their "corrected" text trusted —
  same guidance as outlier catalog item 8 (2D formulas aren't linearly
  recoverable); no auto-correction should be applied to those specifically,
  flag-only.

---

## 2026-07-30 — Eval strategy locked in; Phase 1.0 cheap surveys run

**Done (direct requirement: "phải eval thật kỹ... phải có chiến lược rõ
ràng" — plan mode used to design a full segmentation + eval framework before
writing any real ingestion code):**
- Designed and got user approval on a full implementation plan covering
  `extract/` + `segment/` + a `validation/` package, merging the
  already-validated ADR 0003 methodology (back-index recall, currently
  91.7%) with a 6-point eval framework the user specified (visual diff,
  round-trip test, character-level text coverage, structure validation,
  golden dataset, downstream RAG eval) plus a follow-up list of
  domain-safety checks (adult/child dosing not mixed, mg/mcg/mmol units not
  corrupted, warning/contraindication sections captured, chemical formulas,
  header/footer leakage, page numbers not injected mid-paragraph). Full plan
  is preserved for reference; key decisions below are now the standing
  design, not just a plan-file artifact.
- Confirmed target audience (doctors/pharmacists, not lay users — see
  `project_target_audience` memory) explicitly informs why domain-safety
  checks (dosing-population mixing, unit corruption) are being treated as
  first-class eval dimensions, not nice-to-haves.
- Ran Phase 1.0 whole-book surveys (scratch script, not committed):
  - **Zero embedded images** across all 1668 pages (`get_images(full=True)`,
    measured) — image/caption validation tooling is not needed for this
    corpus.
  - **Adult/child dosing splits are the norm, not rare**: "Người lớn"/"Trẻ
    em"/"Trẻ sơ sinh" terms appear on 1121 of ~1400 monograph-range pages —
    elevates dosing-population-mixing to a standing validation check.
  - **Found and confirmed a real chemical reaction equation** (physical page
    1033, cyanide-antidote mechanism: `Na2S2O3 + CN⁻ → SCN⁻ + Na2SO3`) and a
    **new outlier**: the reaction arrow extracts as a Private-Use-Area glyph
    (`U+F0AF`), not a standard Unicode arrow — added as outlier-catalog item
    16. A regex scan for chemical-formula-shaped tokens found 9 raw hits,
    8 of which were false positives (flu-strain names, receptor names) —
    genuine chemical notation exists but is rare, not systemic.
  - Attempted to pin down the exact shortest monograph name+page, but the
    crude (unmerged, no multi-line-title-merge) scan script produced a
    **different longest-monograph ranking** than the already-documented one
    (previously: "AMOXICILIN VÀ KALI CLAVULANAT" at 45,623 chars; this
    script's top result was INSULIN at 41,799 chars) — flagged as
    unreliable rather than reported as fact, and explicitly deferred to
    Phase 1.2's real detector rather than trusting a quick script's number
    over the previously-validated one. Added to outlier catalog's "not yet
    investigated" list with the reasoning, not silently dropped.
- Added outlier-catalog items 15 (no images), 16 (PUA reaction-arrow
  glyphs), 17 (adult/child dosing prevalence).

**Not done yet / next up:**
- Phase 1.1 onward: build real `ingestion/ingestion/extract/` and
  `segment/` modules (currently still empty stub packages) per the approved
  plan — `page_map.py` first, then `spans.py`/`glyph_order.py`, then the
  segment detector/merge/atc/units logic with unit tests, then wiring
  `cli.py run`, then the `validation/` package (back-index recall+precision,
  golden dataset, char-coverage/structure/domain-safety checks,
  visual-diff). See the approved plan file for the full phase breakdown and
  numeric targets (≥98% monograph recall/precision, ≥99% mean character
  coverage, zero-regression golden-set gate, manual visual-diff sign-off on
  hardest pages) if this session ends before implementation completes.
- `pytest` and `pymupdf` need to be added to `ingestion/pyproject.toml`
  dependencies (currently `dependencies = []`) — confirmed both are already
  available in the global Python 3.12.10 env (PyMuPDF 1.28.0, pytest 7.4.4)
  but not yet pinned in the package's own dependency list.

---

## 2026-07-30 — Whole-corpus structural survey (not just anecdotes)

**Done (direct pushback: "I feel like you're minimizing how complex this
PDF really is — go find another 10-30 outliers, not just Vitamin D"):**
- Built a real per-monograph structural survey across all 680 detected
  monographs (not 2 anecdotes) — computed ATC-code count, known-section
  count, and character length for every one.
- **Multi-ATC monographs are NOT rare**: 173/680 (25.4%) have more than one
  ATC code — INSULIN has 20, BETAMETHASON and DEXAMETHASON 11 each,
  PREDNISOLON 10, HYDROCORTISON 9. The earlier "found 2 examples" framing
  badly understated this. Even 25.4% is a floor (see next point).
- Investigated the 22 apparent "zero ATC" monographs (spot-checked 14):
  found **two distinct real causes of false negatives** — stray internal
  whitespace splitting an ATC code (`"J04A C01"` instead of `"J04AC01"`)
  and digit/letter confusion (`"NO3AX12"` instead of `"N03AX12"`) — 9 of 14
  resolved as real ATC codes hidden by extraction noise (one of them,
  TRIAMCINOLON, turned out to have 5 ATC codes, meaning the true
  multi-ATC percentage is higher than 25.4%). The remaining ~5 genuinely
  say `"Mã ATC: Chưa có."` (not yet assigned) — a valid data state, not an
  error.
- Found and confirmed a **false-positive monograph boundary**: the
  part-divider title "CÁC CHUYÊN LUẬN THUỐC" (Part 2's own section title,
  not a drug) was detected as if it were a monograph.
- Measured real structural variance: monograph length ranges 2,331-45,623
  characters (~20x spread), detected section count ranges 8-20.
- All findings added to `docs/pdf-parsing-outlier-catalog.md` (items 12a
  revised with real numbers, 12c, 12d, 12e — new).
- Verified one of my own debugging steps was itself wrong (read raw page
  text from the top instead of the correctly-bounded monograph segment,
  which briefly looked like a segmentation bug before being traced back to
  a debugging mistake, not a real defect) — corrected before reporting.

**Not done yet / next up:**
- Full-corpus re-count with the relaxed ATC regex (whitespace-tolerant,
  O/0-aware) not yet run — only 14/22 zero-ATC cases spot-checked, and the
  173/680 multi-ATC count still uses the strict (undercounting) regex.
- Phase 1 real implementation still pending overall (see earlier entries).

---

## 2026-07-30 — Confirmed class-level monographs and a real source typo

**Done (direct follow-up: "have you checked drug-class entries like Vitamin
D, or actual spelling/font-size errors?"):**
- Found and confirmed a **second real example of a class-level monograph**
  covering multiple ATC codes/substances: "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"
  (7 ATC codes, one per specific vitamin D analogue) — same pattern as the
  earlier GONADOTROPIN finding, confirming this is recurring, not a one-off.
- Found and confirmed a **genuine spelling/capitalization typo in the
  source PDF itself**: the running header on this monograph's continuation
  pages reads "Vitamin d..." (lowercase d) vs the correct ALL-CAPS heading
  "VITAMIN D...". Verified via font/bbox inspection that this is a real
  source-text inconsistency, not an extraction artifact. The detection
  heuristic still worked correctly here (the typo'd header isn't all-caps
  so it's correctly rejected), but this was incidental, not a designed
  defense against typos.
- Added both findings to `docs/pdf-parsing-outlier-catalog.md` (items 12a,
  12b), with the general lesson: rely on multiple independent structural
  signals, not any single text match, since real source typos do occur.
- Added `CLAUDE.md` with a standing rule: never fabricate or bluff a claim
  (number, test result, capability estimate) — verify before stating,
  explicitly flag estimates as estimates. Grounded in concrete incidents
  from this investigation (the size-threshold bug, the scope-gap bug).

**Not done yet / next up:**
- No systematic scan yet for *other* class-level (multi-ATC) monographs
  beyond the two found incidentally — Phase 1's data model should assume
  ATC code is a list per monograph regardless, rather than trying to
  enumerate every class-level entry in advance.
- Phase 1 real implementation still pending overall (see earlier entries).

---

## 2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns)

**Done (in response to direct follow-up questions about table/formula
handling and full-book coverage):**
- Found and confirmed a **table split across a page break loses its header
  on the continuation page** — real example: "Bảng 4" (ARV rash management
  table) ends with an orphaned, header-less data row on the next page when
  extracted with `pdfplumber`.
- Found the **same header-loss risk also happens across a column boundary
  within a single page** (no page break needed) — real example: "Bảng 6".
- Found and confirmed **2D grid/nomogram tables are not linearly
  recoverable** — the body-surface-area lookup table (appendix) extracts as
  scrambled bare numbers with no row/column association.
- Found **two different formula-rendering outcomes**: a simple inline-
  exponent formula (Du Bois BSA) extracts cleanly as text; a stacked-
  fraction formula (Cockcroft-Gault) extracts as disordered fragments —
  confirmed the determining factor is 1D vs 2D visual layout, not "formulas
  are always broken."
- Found and confirmed a **full-width table that breaks out of the normal
  two-column page grid** (bbox spans nearly the full page width).
- Checked whether front-matter "committee list" pages are genuinely
  multi-column (the user suspected 3 columns) — confirmed via bbox
  inspection they are **not** true structural columns, just single wide
  text blocks with internal whitespace padding between names.
- Consolidated **all** outlier findings from this investigation (this entry
  and the previous one) into a single, reusable, generalized reference:
  `docs/pdf-parsing-outlier-catalog.md` — written so it can guide parsing of
  other similarly-structured PDFs, not just this book.

**Not done yet / next up:**
- No automatic detector exists yet for (a) 2D-formula regions, or (b) 2D
  grid-table reconstruction — both flagged as open items in the catalog,
  not silently skipped.
- Table-continuation re-attachment (page-break and column-break cases) has
  no implementation yet — needed before Phase 1 can trust any multi-row
  table content.
- Phase 1 real implementation still pending overall (see previous entry).

---

## 2026-07-30 — PDF parsing strategy validated empirically (pre-Phase-1)

**Done:**
- Investigated the real PDF structure before writing any ingestion code
  (previous scaffold's assumptions about `doc.get_toc()` turned out wrong).
- Confirmed: 1668 pages, no bookmark/outline (0 TOC entries), tagged-PDF
  structure tree exists but is too shallow to use (~29 elements only).
- Cross-tested 3 extraction tools on real sample pages: PyMuPDF (correct
  reading order — kept as primary), pdfplumber (scrambled reading order on
  this layout — demoted to table-extraction-only use), opendataloader-pdf
  (correct reading order, useful independent font-metadata cross-check, but
  inconsistent heading classification — not trusted as sole signal). Docling
  install hit a numpy/pyarrow ABI conflict in the global Python env; tested
  in an isolated `.venv_docling_test/` (gitignored) instead of risking the
  global environment — see whether that resolved before relying on it.
- Found the real structural ground truth: every section/monograph heading is
  a **bold font span** in the PDF (confirmed at the PyMuPDF span level AND
  independently by opendataloader's own font metadata — two tools agreeing).
  Font **size** is not reliable (10.0pt and 9.5pt both occur for genuine
  monograph titles) — an early size-based threshold silently dropped ~15% of
  real monographs; caught and fixed via whole-document validation, not
  spot-checking.
- Found the real ground truth for validation: the back-of-book "Mục lục tra
  cứu" (page ~1528 onward) has exact page numbers per drug — much stronger
  than the front-matter drug list (which has no page numbers). Also found
  the book's own contents page states individual monographs run printed
  pages 99-1496 exactly.
- Ran automated whole-document (1668-page, ~20-50s per run) validation
  against that page-verified ground truth: **91.7% recall** (665/725), with
  the remaining gap traced to one concrete, fixable cause (multi-line
  wrapped ALL-CAPS titles not yet merged across lines) rather than a flaw in
  the bold-span signal itself.
- Documented the full methodology and results in
  `docs/adr/0003-pdf-parsing-strategy.md` and updated the ingestion section
  of `docs/architecture.md` to match reality (removed the incorrect
  TOC-preference assumption).

**Also validated (in response to direct user questions about correctness):**
- **No real duplicate drug monographs** found across the full 1405-page
  monograph range. The one apparent collision ("GONADOTROPIN" at 2 pages)
  is a detector artifact from the known multi-line-title bug (a different
  monograph's wrapped title fragment collided with it), not real content
  duplication.
- **Confirmed the PDF is genuinely two-column** (bounding-box verified: left
  column x≈44-299, right column x≈308-562). PyMuPDF's reading order across
  columns is correct (already implied by earlier validation).
- **Found and precisely characterized one real data-corruption defect**:
  a single text run on physical page 1373 has reversed (right-to-left)
  glyph order, producing scrambled text — confirmed by reversing the
  string, which recovers the correct Vietnamese sentence. A full scan of
  all 1405 monograph pages (grouping fragments into visual rows, checking
  for descending x-order) found this exact **1 occurrence and no others** —
  rare, isolated, but real, and now has a cheap (~16s) automated detector.
- Full details, methodology, and exact numbers added to
  `docs/adr/0003-pdf-parsing-strategy.md` under "Follow-up validation."
- **Caught a real scope gap**: the glyph-reversal scan above was initially
  run on the monograph range only (1405 of 1668 pages), leaving ~260 pages
  (front matter, appendices, back index) unchecked. Re-ran across the full
  1668 pages: still exactly 1 defect (same page, 1373) — confirmed isolated,
  not hiding elsewhere. Also found 6 near-empty pages (3, 37, 99, 1495, 1497,
  1666), all of which land exactly on major section-transition boundaries —
  intentional print blank pages, not lost content.

**Not done yet / next up:**
- Resolve/confirm docling status in the isolated venv (numpy/pyarrow
  conflict was fixed by using a separate venv; install completed — actual
  parsing comparison against the sample pages still pending).
- Phase 1 real implementation: build `ingestion/` for real using the
  validated bold-span detector (not the exploratory scratch scripts) as one
  continuous cross-page stream (not per-page silos), fix the multi-line
  heading-merge gap, add the glyph-order sanity check as a mandatory
  pre-ingestion pass, re-run the validation script to confirm improved
  recall, then proceed to chunking + embedding + Qdrant upsert.
- Decide and implement chunking strategy for the non-monograph parts of the
  book (general chapters pages 37-98, appendices 1497-1528) — needed so the
  full book (page 0 to last) ends up captured in the RAG corpus in some
  appropriate form, per the user's explicit requirement that no content be
  silently dropped.
- Clean up exploratory `scratch_*` files from the repo root as they
  accumulate during investigation (routinely deleted after findings are
  persisted to docs — not left in git history).

---

## 2026-07-30 — Initial monorepo scaffold

**Done:**
- Designed the microservices architecture (see `docs/architecture.md`):
  Python/FastAPI `ai-service` for RAG, NestJS for `api-gateway`/`auth-service`/
  `user-service`/`chat-service`, Next.js `web`, Qdrant for vectors, Postgres
  for relational data, Redis reserved for caching/queues.
- Scaffolded the full monorepo directory tree (`apps/`, `packages/`,
  `ingestion/`, `infra/`, `docs/`) with baseline config (package.json/
  pyproject.toml stubs, pnpm workspace, docker-compose topology stub).
- Moved `duoc-thu-quoc-gia-viet-nam-2018.pdf` into `ingestion/data/raw/`.
- Decided vector DB: **Qdrant** over pgvector (`docs/adr/0001-vector-db-qdrant.md`).
- Decided deployment: GitOps via the **team's existing ArgoCD instance**,
  not a custom push-based CD pipeline (`docs/adr/0002-argocd-gitops.md`,
  `infra/argocd/`). CI's job is build/test/push image + bump the Helm values
  image tag; ArgoCD does the actual sync.
- `git init` + initial commit (this scaffold).
- Created a private GitHub repo (`BaoVu2k4/vsf-duocthu`, default branch
  `master`) and pushed the initial commit; fixed `targetRevision` in the
  ArgoCD Application manifests to `master` to match.

**Not done yet / next up (Phase 1 of the build roadmap in `docs/architecture.md`):**
- No business logic exists yet anywhere — this was scaffold only.
- Phase 1: build the `ingestion/` pipeline for real (PDF extraction via
  PyMuPDF, monograph/section segmentation, section-aware chunking, OpenAI
  embeddings, Qdrant upsert) and validate retrieval quality via the
  `ingestion/notebooks/` QA step.
- Still pending/TBD: which cloud provider (AWS/GCP/Azure) for Terraform
  (`infra/terraform/README.md`), and the team's ArgoCD instance's actual
  cluster/server + project details (`infra/argocd/README.md` TODOs).