duocthu/docs/progress-log.md

Progress Log

Chronological record of work done on this project, newest entry on top. The goal is continuity across sessions: if a work session ends unexpectedly (context/token limit, interruption), whoever picks this up next — human or Claude — should be able to read the latest entry and know exactly what's done and what's next, without having to reconstruct it from git history.

Convention: add a new entry at the top before ending a session whenever meaningful progress was made, and proactively the moment it looks like the session might run out of context/tokens mid-task — don't wait until the very end if that risk is showing.

---

2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns)

Done (in response to direct follow-up questions about table/formula handling and full-book coverage):

• Found and confirmed a table split across a page break loses its header on the continuation page — real example: "Bảng 4" (ARV rash management table) ends with an orphaned, header-less data row on the next page when extracted with pdfplumber.
• Found the same header-loss risk also happens across a column boundary within a single page (no page break needed) — real example: "Bảng 6".
• Found and confirmed 2D grid/nomogram tables are not linearly recoverable — the body-surface-area lookup table (appendix) extracts as scrambled bare numbers with no row/column association.
• Found two different formula-rendering outcomes: a simple inline- exponent formula (Du Bois BSA) extracts cleanly as text; a stacked- fraction formula (Cockcroft-Gault) extracts as disordered fragments — confirmed the determining factor is 1D vs 2D visual layout, not "formulas are always broken."
• Found and confirmed a full-width table that breaks out of the normal two-column page grid (bbox spans nearly the full page width).
• Checked whether front-matter "committee list" pages are genuinely multi-column (the user suspected 3 columns) — confirmed via bbox inspection they are not true structural columns, just single wide text blocks with internal whitespace padding between names.
• Consolidated all outlier findings from this investigation (this entry and the previous one) into a single, reusable, generalized reference: docs/pdf-parsing-outlier-catalog.md — written so it can guide parsing of other similarly-structured PDFs, not just this book.

Not done yet / next up:

• No automatic detector exists yet for (a) 2D-formula regions, or (b) 2D grid-table reconstruction — both flagged as open items in the catalog, not silently skipped.
• Table-continuation re-attachment (page-break and column-break cases) has no implementation yet — needed before Phase 1 can trust any multi-row table content.
• Phase 1 real implementation still pending overall (see previous entry).

---

2026-07-30 — PDF parsing strategy validated empirically (pre-Phase-1)

Done:

• Investigated the real PDF structure before writing any ingestion code (previous scaffold's assumptions about doc.get_toc() turned out wrong).
• Confirmed: 1668 pages, no bookmark/outline (0 TOC entries), tagged-PDF structure tree exists but is too shallow to use (~29 elements only).
• Cross-tested 3 extraction tools on real sample pages: PyMuPDF (correct reading order — kept as primary), pdfplumber (scrambled reading order on this layout — demoted to table-extraction-only use), opendataloader-pdf (correct reading order, useful independent font-metadata cross-check, but inconsistent heading classification — not trusted as sole signal). Docling install hit a numpy/pyarrow ABI conflict in the global Python env; tested in an isolated .venv_docling_test/ (gitignored) instead of risking the global environment — see whether that resolved before relying on it.
• Found the real structural ground truth: every section/monograph heading is a bold font span in the PDF (confirmed at the PyMuPDF span level AND independently by opendataloader's own font metadata — two tools agreeing). Font size is not reliable (10.0pt and 9.5pt both occur for genuine monograph titles) — an early size-based threshold silently dropped ~15% of real monographs; caught and fixed via whole-document validation, not spot-checking.
• Found the real ground truth for validation: the back-of-book "Mục lục tra cứu" (page ~1528 onward) has exact page numbers per drug — much stronger than the front-matter drug list (which has no page numbers). Also found the book's own contents page states individual monographs run printed pages 99-1496 exactly.
• Ran automated whole-document (1668-page, ~20-50s per run) validation against that page-verified ground truth: 91.7% recall (665/725), with the remaining gap traced to one concrete, fixable cause (multi-line wrapped ALL-CAPS titles not yet merged across lines) rather than a flaw in the bold-span signal itself.
• Documented the full methodology and results in docs/adr/0003-pdf-parsing-strategy.md and updated the ingestion section of docs/architecture.md to match reality (removed the incorrect TOC-preference assumption).

Also validated (in response to direct user questions about correctness):

• No real duplicate drug monographs found across the full 1405-page monograph range. The one apparent collision ("GONADOTROPIN" at 2 pages) is a detector artifact from the known multi-line-title bug (a different monograph's wrapped title fragment collided with it), not real content duplication.
• Confirmed the PDF is genuinely two-column (bounding-box verified: left column x≈44-299, right column x≈308-562). PyMuPDF's reading order across columns is correct (already implied by earlier validation).
• Found and precisely characterized one real data-corruption defect: a single text run on physical page 1373 has reversed (right-to-left) glyph order, producing scrambled text — confirmed by reversing the string, which recovers the correct Vietnamese sentence. A full scan of all 1405 monograph pages (grouping fragments into visual rows, checking for descending x-order) found this exact 1 occurrence and no others — rare, isolated, but real, and now has a cheap (~16s) automated detector.
• Full details, methodology, and exact numbers added to docs/adr/0003-pdf-parsing-strategy.md under "Follow-up validation."
• Caught a real scope gap: the glyph-reversal scan above was initially run on the monograph range only (1405 of 1668 pages), leaving ~260 pages (front matter, appendices, back index) unchecked. Re-ran across the full 1668 pages: still exactly 1 defect (same page, 1373) — confirmed isolated, not hiding elsewhere. Also found 6 near-empty pages (3, 37, 99, 1495, 1497, 1666), all of which land exactly on major section-transition boundaries — intentional print blank pages, not lost content.

Not done yet / next up:

• Resolve/confirm docling status in the isolated venv (numpy/pyarrow conflict was fixed by using a separate venv; install completed — actual parsing comparison against the sample pages still pending).
• Phase 1 real implementation: build ingestion/ for real using the validated bold-span detector (not the exploratory scratch scripts) as one continuous cross-page stream (not per-page silos), fix the multi-line heading-merge gap, add the glyph-order sanity check as a mandatory pre-ingestion pass, re-run the validation script to confirm improved recall, then proceed to chunking + embedding + Qdrant upsert.
• Decide and implement chunking strategy for the non-monograph parts of the book (general chapters pages 37-98, appendices 1497-1528) — needed so the full book (page 0 to last) ends up captured in the RAG corpus in some appropriate form, per the user's explicit requirement that no content be silently dropped.
• Clean up exploratory scratch_* files from the repo root as they accumulate during investigation (routinely deleted after findings are persisted to docs — not left in git history).

---

2026-07-30 — Initial monorepo scaffold

Done:

• Designed the microservices architecture (see docs/architecture.md): Python/FastAPI ai-service for RAG, NestJS for api-gateway/auth-service/ user-service/chat-service, Next.js web, Qdrant for vectors, Postgres for relational data, Redis reserved for caching/queues.
• Scaffolded the full monorepo directory tree (apps/, packages/, ingestion/, infra/, docs/) with baseline config (package.json/ pyproject.toml stubs, pnpm workspace, docker-compose topology stub).
• Moved duoc-thu-quoc-gia-viet-nam-2018.pdf into ingestion/data/raw/.
• Decided vector DB: Qdrant over pgvector (docs/adr/0001-vector-db-qdrant.md).
• Decided deployment: GitOps via the team's existing ArgoCD instance, not a custom push-based CD pipeline (docs/adr/0002-argocd-gitops.md, infra/argocd/). CI's job is build/test/push image + bump the Helm values image tag; ArgoCD does the actual sync.
• git init + initial commit (this scaffold).
• Created a private GitHub repo (BaoVu2k4/vsf-duocthu, default branch master) and pushed the initial commit; fixed targetRevision in the ArgoCD Application manifests to master to match.

Not done yet / next up (Phase 1 of the build roadmap in docs/architecture.md):

• No business logic exists yet anywhere — this was scaffold only.
• Phase 1: build the ingestion/ pipeline for real (PDF extraction via PyMuPDF, monograph/section segmentation, section-aware chunking, OpenAI embeddings, Qdrant upsert) and validate retrieval quality via the ingestion/notebooks/ QA step.
• Still pending/TBD: which cloud provider (AWS/GCP/Azure) for Terraform (infra/terraform/README.md), and the team's ArgoCD instance's actual cluster/server + project details (infra/argocd/README.md TODOs).