Add comprehensive, reusable PDF-parsing outlier catalog
This commit is contained in:
+127
-1
@@ -13,6 +13,130 @@ end if that risk is showing.
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns)
|
||||
|
||||
**Done (in response to direct follow-up questions about table/formula
|
||||
handling and full-book coverage):**
|
||||
- Found and confirmed a **table split across a page break loses its header
|
||||
on the continuation page** — real example: "Bảng 4" (ARV rash management
|
||||
table) ends with an orphaned, header-less data row on the next page when
|
||||
extracted with `pdfplumber`.
|
||||
- Found the **same header-loss risk also happens across a column boundary
|
||||
within a single page** (no page break needed) — real example: "Bảng 6".
|
||||
- Found and confirmed **2D grid/nomogram tables are not linearly
|
||||
recoverable** — the body-surface-area lookup table (appendix) extracts as
|
||||
scrambled bare numbers with no row/column association.
|
||||
- Found **two different formula-rendering outcomes**: a simple inline-
|
||||
exponent formula (Du Bois BSA) extracts cleanly as text; a stacked-
|
||||
fraction formula (Cockcroft-Gault) extracts as disordered fragments —
|
||||
confirmed the determining factor is 1D vs 2D visual layout, not "formulas
|
||||
are always broken."
|
||||
- Found and confirmed a **full-width table that breaks out of the normal
|
||||
two-column page grid** (bbox spans nearly the full page width).
|
||||
- Checked whether front-matter "committee list" pages are genuinely
|
||||
multi-column (the user suspected 3 columns) — confirmed via bbox
|
||||
inspection they are **not** true structural columns, just single wide
|
||||
text blocks with internal whitespace padding between names.
|
||||
- Consolidated **all** outlier findings from this investigation (this entry
|
||||
and the previous one) into a single, reusable, generalized reference:
|
||||
`docs/pdf-parsing-outlier-catalog.md` — written so it can guide parsing of
|
||||
other similarly-structured PDFs, not just this book.
|
||||
|
||||
**Not done yet / next up:**
|
||||
- No automatic detector exists yet for (a) 2D-formula regions, or (b) 2D
|
||||
grid-table reconstruction — both flagged as open items in the catalog,
|
||||
not silently skipped.
|
||||
- Table-continuation re-attachment (page-break and column-break cases) has
|
||||
no implementation yet — needed before Phase 1 can trust any multi-row
|
||||
table content.
|
||||
- Phase 1 real implementation still pending overall (see previous entry).
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-30 — PDF parsing strategy validated empirically (pre-Phase-1)
|
||||
|
||||
**Done:**
|
||||
- Investigated the real PDF structure before writing any ingestion code
|
||||
(previous scaffold's assumptions about `doc.get_toc()` turned out wrong).
|
||||
- Confirmed: 1668 pages, no bookmark/outline (0 TOC entries), tagged-PDF
|
||||
structure tree exists but is too shallow to use (~29 elements only).
|
||||
- Cross-tested 3 extraction tools on real sample pages: PyMuPDF (correct
|
||||
reading order — kept as primary), pdfplumber (scrambled reading order on
|
||||
this layout — demoted to table-extraction-only use), opendataloader-pdf
|
||||
(correct reading order, useful independent font-metadata cross-check, but
|
||||
inconsistent heading classification — not trusted as sole signal). Docling
|
||||
install hit a numpy/pyarrow ABI conflict in the global Python env; tested
|
||||
in an isolated `.venv_docling_test/` (gitignored) instead of risking the
|
||||
global environment — see whether that resolved before relying on it.
|
||||
- Found the real structural ground truth: every section/monograph heading is
|
||||
a **bold font span** in the PDF (confirmed at the PyMuPDF span level AND
|
||||
independently by opendataloader's own font metadata — two tools agreeing).
|
||||
Font **size** is not reliable (10.0pt and 9.5pt both occur for genuine
|
||||
monograph titles) — an early size-based threshold silently dropped ~15% of
|
||||
real monographs; caught and fixed via whole-document validation, not
|
||||
spot-checking.
|
||||
- Found the real ground truth for validation: the back-of-book "Mục lục tra
|
||||
cứu" (page ~1528 onward) has exact page numbers per drug — much stronger
|
||||
than the front-matter drug list (which has no page numbers). Also found
|
||||
the book's own contents page states individual monographs run printed
|
||||
pages 99-1496 exactly.
|
||||
- Ran automated whole-document (1668-page, ~20-50s per run) validation
|
||||
against that page-verified ground truth: **91.7% recall** (665/725), with
|
||||
the remaining gap traced to one concrete, fixable cause (multi-line
|
||||
wrapped ALL-CAPS titles not yet merged across lines) rather than a flaw in
|
||||
the bold-span signal itself.
|
||||
- Documented the full methodology and results in
|
||||
`docs/adr/0003-pdf-parsing-strategy.md` and updated the ingestion section
|
||||
of `docs/architecture.md` to match reality (removed the incorrect
|
||||
TOC-preference assumption).
|
||||
|
||||
**Also validated (in response to direct user questions about correctness):**
|
||||
- **No real duplicate drug monographs** found across the full 1405-page
|
||||
monograph range. The one apparent collision ("GONADOTROPIN" at 2 pages)
|
||||
is a detector artifact from the known multi-line-title bug (a different
|
||||
monograph's wrapped title fragment collided with it), not real content
|
||||
duplication.
|
||||
- **Confirmed the PDF is genuinely two-column** (bounding-box verified: left
|
||||
column x≈44-299, right column x≈308-562). PyMuPDF's reading order across
|
||||
columns is correct (already implied by earlier validation).
|
||||
- **Found and precisely characterized one real data-corruption defect**:
|
||||
a single text run on physical page 1373 has reversed (right-to-left)
|
||||
glyph order, producing scrambled text — confirmed by reversing the
|
||||
string, which recovers the correct Vietnamese sentence. A full scan of
|
||||
all 1405 monograph pages (grouping fragments into visual rows, checking
|
||||
for descending x-order) found this exact **1 occurrence and no others** —
|
||||
rare, isolated, but real, and now has a cheap (~16s) automated detector.
|
||||
- Full details, methodology, and exact numbers added to
|
||||
`docs/adr/0003-pdf-parsing-strategy.md` under "Follow-up validation."
|
||||
- **Caught a real scope gap**: the glyph-reversal scan above was initially
|
||||
run on the monograph range only (1405 of 1668 pages), leaving ~260 pages
|
||||
(front matter, appendices, back index) unchecked. Re-ran across the full
|
||||
1668 pages: still exactly 1 defect (same page, 1373) — confirmed isolated,
|
||||
not hiding elsewhere. Also found 6 near-empty pages (3, 37, 99, 1495, 1497,
|
||||
1666), all of which land exactly on major section-transition boundaries —
|
||||
intentional print blank pages, not lost content.
|
||||
|
||||
**Not done yet / next up:**
|
||||
- Resolve/confirm docling status in the isolated venv (numpy/pyarrow
|
||||
conflict was fixed by using a separate venv; install completed — actual
|
||||
parsing comparison against the sample pages still pending).
|
||||
- Phase 1 real implementation: build `ingestion/` for real using the
|
||||
validated bold-span detector (not the exploratory scratch scripts) as one
|
||||
continuous cross-page stream (not per-page silos), fix the multi-line
|
||||
heading-merge gap, add the glyph-order sanity check as a mandatory
|
||||
pre-ingestion pass, re-run the validation script to confirm improved
|
||||
recall, then proceed to chunking + embedding + Qdrant upsert.
|
||||
- Decide and implement chunking strategy for the non-monograph parts of the
|
||||
book (general chapters pages 37-98, appendices 1497-1528) — needed so the
|
||||
full book (page 0 to last) ends up captured in the RAG corpus in some
|
||||
appropriate form, per the user's explicit requirement that no content be
|
||||
silently dropped.
|
||||
- Clean up exploratory `scratch_*` files from the repo root as they
|
||||
accumulate during investigation (routinely deleted after findings are
|
||||
persisted to docs — not left in git history).
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-30 — Initial monorepo scaffold
|
||||
|
||||
**Done:**
|
||||
@@ -30,7 +154,9 @@ end if that risk is showing.
|
||||
`infra/argocd/`). CI's job is build/test/push image + bump the Helm values
|
||||
image tag; ArgoCD does the actual sync.
|
||||
- `git init` + initial commit (this scaffold).
|
||||
- Created a private GitHub repo and pushed the initial commit.
|
||||
- Created a private GitHub repo (`BaoVu2k4/vsf-duocthu`, default branch
|
||||
`master`) and pushed the initial commit; fixed `targetRevision` in the
|
||||
ArgoCD Application manifests to `master` to match.
|
||||
|
||||
**Not done yet / next up (Phase 1 of the build roadmap in `docs/architecture.md`):**
|
||||
- No business logic exists yet anywhere — this was scaffold only.
|
||||
|
||||
Reference in New Issue
Block a user