duocthu/docs/progress-log.md

# Progress Log

Chronological record of work done on this project, newest entry on top. The
goal is continuity across sessions: if a work session ends unexpectedly
(context/token limit, interruption), whoever picks this up next — human or
Claude — should be able to read the latest entry and know exactly what's
done and what's next, without having to reconstruct it from git history.

**Convention**: add a new entry at the top before ending a session whenever
meaningful progress was made, and proactively the moment it looks like the
session might run out of context/tokens mid-task — don't wait until the very
end if that risk is showing.

---

## 2026-07-30 — Whole-corpus structural survey (not just anecdotes)

**Done (direct pushback: "I feel like you're minimizing how complex this
PDF really is — go find another 10-30 outliers, not just Vitamin D"):**
- Built a real per-monograph structural survey across all 680 detected
  monographs (not 2 anecdotes) — computed ATC-code count, known-section
  count, and character length for every one.
- **Multi-ATC monographs are NOT rare**: 173/680 (25.4%) have more than one
  ATC code — INSULIN has 20, BETAMETHASON and DEXAMETHASON 11 each,
  PREDNISOLON 10, HYDROCORTISON 9. The earlier "found 2 examples" framing
  badly understated this. Even 25.4% is a floor (see next point).
- Investigated the 22 apparent "zero ATC" monographs (spot-checked 14):
  found **two distinct real causes of false negatives** — stray internal
  whitespace splitting an ATC code (`"J04A C01"` instead of `"J04AC01"`)
  and digit/letter confusion (`"NO3AX12"` instead of `"N03AX12"`) — 9 of 14
  resolved as real ATC codes hidden by extraction noise (one of them,
  TRIAMCINOLON, turned out to have 5 ATC codes, meaning the true
  multi-ATC percentage is higher than 25.4%). The remaining ~5 genuinely
  say `"Mã ATC: Chưa có."` (not yet assigned) — a valid data state, not an
  error.
- Found and confirmed a **false-positive monograph boundary**: the
  part-divider title "CÁC CHUYÊN LUẬN THUỐC" (Part 2's own section title,
  not a drug) was detected as if it were a monograph.
- Measured real structural variance: monograph length ranges 2,331-45,623
  characters (~20x spread), detected section count ranges 8-20.
- All findings added to `docs/pdf-parsing-outlier-catalog.md` (items 12a
  revised with real numbers, 12c, 12d, 12e — new).
- Verified one of my own debugging steps was itself wrong (read raw page
  text from the top instead of the correctly-bounded monograph segment,
  which briefly looked like a segmentation bug before being traced back to
  a debugging mistake, not a real defect) — corrected before reporting.

**Not done yet / next up:**
- Full-corpus re-count with the relaxed ATC regex (whitespace-tolerant,
  O/0-aware) not yet run — only 14/22 zero-ATC cases spot-checked, and the
  173/680 multi-ATC count still uses the strict (undercounting) regex.
- Phase 1 real implementation still pending overall (see earlier entries).

---

## 2026-07-30 — Confirmed class-level monographs and a real source typo

**Done (direct follow-up: "have you checked drug-class entries like Vitamin
D, or actual spelling/font-size errors?"):**
- Found and confirmed a **second real example of a class-level monograph**
  covering multiple ATC codes/substances: "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"
  (7 ATC codes, one per specific vitamin D analogue) — same pattern as the
  earlier GONADOTROPIN finding, confirming this is recurring, not a one-off.
- Found and confirmed a **genuine spelling/capitalization typo in the
  source PDF itself**: the running header on this monograph's continuation
  pages reads "Vitamin d..." (lowercase d) vs the correct ALL-CAPS heading
  "VITAMIN D...". Verified via font/bbox inspection that this is a real
  source-text inconsistency, not an extraction artifact. The detection
  heuristic still worked correctly here (the typo'd header isn't all-caps
  so it's correctly rejected), but this was incidental, not a designed
  defense against typos.
- Added both findings to `docs/pdf-parsing-outlier-catalog.md` (items 12a,
  12b), with the general lesson: rely on multiple independent structural
  signals, not any single text match, since real source typos do occur.
- Added `CLAUDE.md` with a standing rule: never fabricate or bluff a claim
  (number, test result, capability estimate) — verify before stating,
  explicitly flag estimates as estimates. Grounded in concrete incidents
  from this investigation (the size-threshold bug, the scope-gap bug).

**Not done yet / next up:**
- No systematic scan yet for *other* class-level (multi-ATC) monographs
  beyond the two found incidentally — Phase 1's data model should assume
  ATC code is a list per monograph regardless, rather than trying to
  enumerate every class-level entry in advance.
- Phase 1 real implementation still pending overall (see earlier entries).

---

## 2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns)

**Done (in response to direct follow-up questions about table/formula
handling and full-book coverage):**
- Found and confirmed a **table split across a page break loses its header
  on the continuation page** — real example: "Bảng 4" (ARV rash management
  table) ends with an orphaned, header-less data row on the next page when
  extracted with `pdfplumber`.
- Found the **same header-loss risk also happens across a column boundary
  within a single page** (no page break needed) — real example: "Bảng 6".
- Found and confirmed **2D grid/nomogram tables are not linearly
  recoverable** — the body-surface-area lookup table (appendix) extracts as
  scrambled bare numbers with no row/column association.
- Found **two different formula-rendering outcomes**: a simple inline-
  exponent formula (Du Bois BSA) extracts cleanly as text; a stacked-
  fraction formula (Cockcroft-Gault) extracts as disordered fragments —
  confirmed the determining factor is 1D vs 2D visual layout, not "formulas
  are always broken."
- Found and confirmed a **full-width table that breaks out of the normal
  two-column page grid** (bbox spans nearly the full page width).
- Checked whether front-matter "committee list" pages are genuinely
  multi-column (the user suspected 3 columns) — confirmed via bbox
  inspection they are **not** true structural columns, just single wide
  text blocks with internal whitespace padding between names.
- Consolidated **all** outlier findings from this investigation (this entry
  and the previous one) into a single, reusable, generalized reference:
  `docs/pdf-parsing-outlier-catalog.md` — written so it can guide parsing of
  other similarly-structured PDFs, not just this book.

**Not done yet / next up:**
- No automatic detector exists yet for (a) 2D-formula regions, or (b) 2D
  grid-table reconstruction — both flagged as open items in the catalog,
  not silently skipped.
- Table-continuation re-attachment (page-break and column-break cases) has
  no implementation yet — needed before Phase 1 can trust any multi-row
  table content.
- Phase 1 real implementation still pending overall (see previous entry).

---

## 2026-07-30 — PDF parsing strategy validated empirically (pre-Phase-1)

**Done:**
- Investigated the real PDF structure before writing any ingestion code
  (previous scaffold's assumptions about `doc.get_toc()` turned out wrong).
- Confirmed: 1668 pages, no bookmark/outline (0 TOC entries), tagged-PDF
  structure tree exists but is too shallow to use (~29 elements only).
- Cross-tested 3 extraction tools on real sample pages: PyMuPDF (correct
  reading order — kept as primary), pdfplumber (scrambled reading order on
  this layout — demoted to table-extraction-only use), opendataloader-pdf
  (correct reading order, useful independent font-metadata cross-check, but
  inconsistent heading classification — not trusted as sole signal). Docling
  install hit a numpy/pyarrow ABI conflict in the global Python env; tested
  in an isolated `.venv_docling_test/` (gitignored) instead of risking the
  global environment — see whether that resolved before relying on it.
- Found the real structural ground truth: every section/monograph heading is
  a **bold font span** in the PDF (confirmed at the PyMuPDF span level AND
  independently by opendataloader's own font metadata — two tools agreeing).
  Font **size** is not reliable (10.0pt and 9.5pt both occur for genuine
  monograph titles) — an early size-based threshold silently dropped ~15% of
  real monographs; caught and fixed via whole-document validation, not
  spot-checking.
- Found the real ground truth for validation: the back-of-book "Mục lục tra
  cứu" (page ~1528 onward) has exact page numbers per drug — much stronger
  than the front-matter drug list (which has no page numbers). Also found
  the book's own contents page states individual monographs run printed
  pages 99-1496 exactly.
- Ran automated whole-document (1668-page, ~20-50s per run) validation
  against that page-verified ground truth: **91.7% recall** (665/725), with
  the remaining gap traced to one concrete, fixable cause (multi-line
  wrapped ALL-CAPS titles not yet merged across lines) rather than a flaw in
  the bold-span signal itself.
- Documented the full methodology and results in
  `docs/adr/0003-pdf-parsing-strategy.md` and updated the ingestion section
  of `docs/architecture.md` to match reality (removed the incorrect
  TOC-preference assumption).

**Also validated (in response to direct user questions about correctness):**
- **No real duplicate drug monographs** found across the full 1405-page
  monograph range. The one apparent collision ("GONADOTROPIN" at 2 pages)
  is a detector artifact from the known multi-line-title bug (a different
  monograph's wrapped title fragment collided with it), not real content
  duplication.
- **Confirmed the PDF is genuinely two-column** (bounding-box verified: left
  column x≈44-299, right column x≈308-562). PyMuPDF's reading order across
  columns is correct (already implied by earlier validation).
- **Found and precisely characterized one real data-corruption defect**:
  a single text run on physical page 1373 has reversed (right-to-left)
  glyph order, producing scrambled text — confirmed by reversing the
  string, which recovers the correct Vietnamese sentence. A full scan of
  all 1405 monograph pages (grouping fragments into visual rows, checking
  for descending x-order) found this exact **1 occurrence and no others** —
  rare, isolated, but real, and now has a cheap (~16s) automated detector.
- Full details, methodology, and exact numbers added to
  `docs/adr/0003-pdf-parsing-strategy.md` under "Follow-up validation."
- **Caught a real scope gap**: the glyph-reversal scan above was initially
  run on the monograph range only (1405 of 1668 pages), leaving ~260 pages
  (front matter, appendices, back index) unchecked. Re-ran across the full
  1668 pages: still exactly 1 defect (same page, 1373) — confirmed isolated,
  not hiding elsewhere. Also found 6 near-empty pages (3, 37, 99, 1495, 1497,
  1666), all of which land exactly on major section-transition boundaries —
  intentional print blank pages, not lost content.

**Not done yet / next up:**
- Resolve/confirm docling status in the isolated venv (numpy/pyarrow
  conflict was fixed by using a separate venv; install completed — actual
  parsing comparison against the sample pages still pending).
- Phase 1 real implementation: build `ingestion/` for real using the
  validated bold-span detector (not the exploratory scratch scripts) as one
  continuous cross-page stream (not per-page silos), fix the multi-line
  heading-merge gap, add the glyph-order sanity check as a mandatory
  pre-ingestion pass, re-run the validation script to confirm improved
  recall, then proceed to chunking + embedding + Qdrant upsert.
- Decide and implement chunking strategy for the non-monograph parts of the
  book (general chapters pages 37-98, appendices 1497-1528) — needed so the
  full book (page 0 to last) ends up captured in the RAG corpus in some
  appropriate form, per the user's explicit requirement that no content be
  silently dropped.
- Clean up exploratory `scratch_*` files from the repo root as they
  accumulate during investigation (routinely deleted after findings are
  persisted to docs — not left in git history).

---

## 2026-07-30 — Initial monorepo scaffold

**Done:**
- Designed the microservices architecture (see `docs/architecture.md`):
  Python/FastAPI `ai-service` for RAG, NestJS for `api-gateway`/`auth-service`/
  `user-service`/`chat-service`, Next.js `web`, Qdrant for vectors, Postgres
  for relational data, Redis reserved for caching/queues.
- Scaffolded the full monorepo directory tree (`apps/`, `packages/`,
  `ingestion/`, `infra/`, `docs/`) with baseline config (package.json/
  pyproject.toml stubs, pnpm workspace, docker-compose topology stub).
- Moved `duoc-thu-quoc-gia-viet-nam-2018.pdf` into `ingestion/data/raw/`.
- Decided vector DB: **Qdrant** over pgvector (`docs/adr/0001-vector-db-qdrant.md`).
- Decided deployment: GitOps via the **team's existing ArgoCD instance**,
  not a custom push-based CD pipeline (`docs/adr/0002-argocd-gitops.md`,
  `infra/argocd/`). CI's job is build/test/push image + bump the Helm values
  image tag; ArgoCD does the actual sync.
- `git init` + initial commit (this scaffold).
- Created a private GitHub repo (`BaoVu2k4/vsf-duocthu`, default branch
  `master`) and pushed the initial commit; fixed `targetRevision` in the
  ArgoCD Application manifests to `master` to match.

**Not done yet / next up (Phase 1 of the build roadmap in `docs/architecture.md`):**
- No business logic exists yet anywhere — this was scaffold only.
- Phase 1: build the `ingestion/` pipeline for real (PDF extraction via
  PyMuPDF, monograph/section segmentation, section-aware chunking, OpenAI
  embeddings, Qdrant upsert) and validate retrieval quality via the
  `ingestion/notebooks/` QA step.
- Still pending/TBD: which cloud provider (AWS/GCP/Azure) for Terraform
  (`infra/terraform/README.md`), and the team's ArgoCD instance's actual
  cluster/server + project details (`infra/argocd/README.md` TODOs).