Scaffold web frontend: mock-backed chat + PDF split-view, Tailwind/shadcn

docs/pdf-parsing-outlier-catalog.md

@@ -239,6 +239,147 @@ treating each line independently.
 will have this exact failure mode; always merge candidate multi-line
 headings before using them as unique keys.
 
+### 12a. Class-level monographs cover multiple active ingredients (multiple ATC codes) — this is NOT rare
+**What it looks like:** first noticed via two incidental examples
+("GONADOTROPIN", "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"), then actually measured
+across the whole 680-monograph corpus (not assumed from the 2 examples —
+this distinction matters, see below). **Real, whole-corpus number: 173 of
+680 detected monographs (25.4%) have more than one distinct ATC code**,
+ranging up to extreme cases — INSULIN alone lists **20** different ATC
+codes, BETAMETHASON and DEXAMETHASON 11 each, PREDNISOLON 10,
+HYDROCORTISON 9. This is a quarter of the entire corpus, not a couple of
+edge cases — the 2 incidental examples badly understated how common this
+is, and stating "found 2 examples, pattern confirmed" without the
+whole-corpus count would have been exactly the kind of unverified claim
+this project's CLAUDE.md now forbids.
+**Even the 25.4% is a floor, not the true number** — see item 12c below:
+ATC-code text-extraction noise (stray whitespace, O/0 confusion) caused
+some genuinely multi-ATC monographs (e.g. "TRIAMCINOLON", 5 codes) to be
+undercounted by a naive regex. The true proportion is measurably higher
+than 25.4%; re-measure after fixing the regex, don't keep citing 25.4% as
+final.
+**Why it matters:** a data model that assumes "one monograph = one drug =
+one ATC code" is wrong for roughly a quarter or more of the corpus.
+**Handling:** store ATC code (and dosage-form sub-entries) as a **list**
+per monograph, not a scalar; when chunking, consider whether a
+class-level monograph's sections should be tagged with the whole class
+name, the specific sub-compound, or both, depending on what the retrieval
+use case needs.
+**Generalizes:** yes — any reference work organized primarily by drug
+class or by generic substance will have entries that don't map 1:1 to a
+single identifier. More importantly, the *methodology* generalizes: when
+you notice a pattern from 1-2 examples, measure its real prevalence across
+the whole corpus before deciding how much engineering effort it deserves —
+"found 2 examples" and "25.4% of everything" call for very different
+levels of investment, and you can't tell which one you're dealing with
+without the whole-corpus count.
+
+### 12c. ATC codes (and likely other structured codes) have real text-extraction noise
+**What it looks like:** while investigating why 22/680 (3.2%) monographs
+appeared to have zero ATC codes, spot-checked 14 of them directly and found
+**two distinct, confirmed causes**, both text-extraction noise rather than
+missing content:
+- **Stray internal whitespace** splitting one code into two tokens, e.g.
+  `"L01X X02"` (should be `L01XX02`), `"J04A C01"` (should be `J04AC01`),
+  `"N05B A06"` (should be `N05BA06`).
+- **Digit/letter confusion**: a literal "0" rendered/typeset as the letter
+  "O", e.g. `"NO3AX12"` (should be `N03AX12`), `"JO1DC07"` (should be
+  `J01DC07`).
+A relaxed regex tolerating both patterns resolved **9 of the 14** spot-checked
+cases as real ATC codes hiding behind extraction noise. The **remaining
+~5 of 14** were genuinely different: the source text explicitly states
+`"Mã ATC: Chưa có."` or `"Mã ATC: Không có."` ("not yet available" / "none")
+— a real, valid data state, not an error, and not something to paper over
+as if a code exists.
+**Why it matters:** a strict ATC-code regex silently undercounts real ATC
+data; distinguishing "extraction noise hiding a real code" from "the book
+says there is no code" requires checking the actual field text, not just
+whether a regex matched.
+**Handling:** normalize ATC-code-shaped text before matching (strip internal
+whitespace between the letter/digit groups, treat a digit-position "O" as
+"0") and explicitly check for the "Chưa có"/"Không có" literal strings as a
+valid "no ATC" state rather than a parse failure.
+**Generalizes:** yes — any structured code/identifier extracted from a PDF
+(product codes, classification codes, reference numbers) can suffer this
+same whitespace-injection and O/0 confusion; validate structured-looking
+fields against their expected format and investigate exceptions rather than
+assuming a strict pattern match is reliable.
+
+### 12d. A section-title (part-divider) page can be falsely detected as a monograph
+**What it looks like:** confirmed — the very first item in a whole-corpus
+boundary scan was "CÁC CHUYÊN LUẬN THUỐC" (the literal title of Part 2 of
+the book, "The Drug Monographs" — a part-divider heading, not a drug) at
+physical page 98, picked up as a false-positive monograph boundary because
+it happened to be bold, all-caps, short, and was followed (a few real
+monograph-boundaries later) by some "Tên chung quốc tế" text from the
+actual first real monograph.
+**Why it matters:** without a whole-corpus scan this would have gone
+unnoticed indefinitely — it doesn't look wrong from a single-page read of
+Abacavir, and the discovery methodology this catalog is built on is
+exhaustive scans, so this is a good example of a defect that only surfaces
+at full scale.
+**Handling:** exclude a small, known set of non-drug part/section-divider
+strings ("CÁC CHUYÊN LUẬN THUỐC", "CÁC CHUYÊN LUẬN CHUNG", "CÁC PHỤ LỤC",
+etc. — enumerable from the book's own table of contents) from the
+monograph-boundary detector, or require the anchor phrase ("Tên chung quốc
+tế") within a tighter line-distance so an unrelated real monograph several
+lines away doesn't false-confirm a divider title.
+**Generalizes:** yes — any document with part/section-divider title pages
+styled similarly to its content headings (bold, prominent, short) risks
+this exact false positive; explicitly exclude known structural/navigational
+titles from content-boundary detectors.
+
+### 12b. Genuine spelling/capitalization typos exist in the source text
+**What it looks like:** confirmed real example — the running header on the
+Vitamin D monograph's continuation pages reads `"Vitamin d và các thuốc
+tương tự"` (lowercase "d"), while the real ALL-CAPS heading correctly reads
+`"VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"`. This is a genuine typesetting mistake
+in the 2018 print, confirmed via font/bbox inspection (same bold 10pt font
+as the correct heading — not an extraction artifact, the source text itself
+has the typo). The page's bottom running *footer* uses yet another variant,
+the short form `"Vitamin D"` (correctly capitalized) — meaning the same
+monograph has **three different boilerplate text variants** across one
+page (top header with a typo, the real heading, bottom footer).
+**Why it matters:** don't treat running headers/footers as a perfectly
+clean, typo-free secondary signal (item 13 in this catalog already
+recommends using them as a cross-check) — they can themselves contain
+source-level errors. In this specific case, the detection heuristic
+(strict ALL-CAPS requirement, item 10) happened to still work correctly,
+because "Vitamin d và các thuốc tương tự" and "Vitamin D" are not fully
+uppercase and so are correctly rejected as monograph-boundary candidates —
+but this was not a designed defense against typos specifically, just a
+side effect of the all-caps requirement. A future/different typo (e.g. an
+accidentally all-caps running header) would not be caught the same way.
+**Check:** no systematic typo-detection was built (out of scope — this is
+about parsing robustness, not proofreading the source); the practical
+takeaway is to keep relying on the strict structural signals (bold + all
+caps + short + anchor phrase) as primary, and treat any single text-based
+signal (including running headers) as fallible.
+**Generalizes:** yes — any real-world print-to-PDF source will have some
+rate of genuine typos/inconsistencies; parsing logic should be robust to
+them by relying on multiple independent structural signals (font,
+position, anchor phrases) rather than trusting any single text match to be
+error-free.
+
+### 12e. Monograph length and section coverage vary enormously — measured, not assumed
+**What it looks like:** across all 680 detected monographs, length ranges
+from **2,331 to 45,623 characters** (~20x spread) and the number of known
+section labels found per monograph ranges from as few as **8** up to
+**20** (out of a ~19-20 item known vocabulary) — most cluster around
+16-19, but the tails are real: "ASPARAGINASE"-adjacent short entries around
+2,300-4,300 chars vs. "AMOXICILIN VÀ KALI CLAVULANAT" at 45,623 chars.
+**Why it matters:** don't design chunking limits (e.g. a fixed max tokens
+per monograph, or an assumption that "a monograph roughly fits in N
+chunks") around a single example — the real distribution has a long tail
+on both ends.
+**Check:** this came from the same whole-corpus survey used for items 12a
+and 12c — computing length and detected-section-count per monograph is
+cheap and worth keeping as a standing sanity metric (e.g. flag any
+monograph outside some percentile range for manual review).
+**Generalizes:** yes — any corpus of "similar" documents (monographs,
+product entries, articles) will have a real length/completeness
+distribution; measure it before assuming uniformity.
+
 ### 12. The documented taxonomy is not exhaustive — keep it open
 **What it looks like:** the book explicitly documents a 19-field template
 for every drug monograph (page 38), but real monographs contain at least

docs/progress-log.md

@@ -13,6 +13,79 @@ end if that risk is showing.
 
 ---
 
+## 2026-07-30 — Whole-corpus structural survey (not just anecdotes)
+
+**Done (direct pushback: "I feel like you're minimizing how complex this
+PDF really is — go find another 10-30 outliers, not just Vitamin D"):**
+- Built a real per-monograph structural survey across all 680 detected
+  monographs (not 2 anecdotes) — computed ATC-code count, known-section
+  count, and character length for every one.
+- **Multi-ATC monographs are NOT rare**: 173/680 (25.4%) have more than one
+  ATC code — INSULIN has 20, BETAMETHASON and DEXAMETHASON 11 each,
+  PREDNISOLON 10, HYDROCORTISON 9. The earlier "found 2 examples" framing
+  badly understated this. Even 25.4% is a floor (see next point).
+- Investigated the 22 apparent "zero ATC" monographs (spot-checked 14):
+  found **two distinct real causes of false negatives** — stray internal
+  whitespace splitting an ATC code (`"J04A C01"` instead of `"J04AC01"`)
+  and digit/letter confusion (`"NO3AX12"` instead of `"N03AX12"`) — 9 of 14
+  resolved as real ATC codes hidden by extraction noise (one of them,
+  TRIAMCINOLON, turned out to have 5 ATC codes, meaning the true
+  multi-ATC percentage is higher than 25.4%). The remaining ~5 genuinely
+  say `"Mã ATC: Chưa có."` (not yet assigned) — a valid data state, not an
+  error.
+- Found and confirmed a **false-positive monograph boundary**: the
+  part-divider title "CÁC CHUYÊN LUẬN THUỐC" (Part 2's own section title,
+  not a drug) was detected as if it were a monograph.
+- Measured real structural variance: monograph length ranges 2,331-45,623
+  characters (~20x spread), detected section count ranges 8-20.
+- All findings added to `docs/pdf-parsing-outlier-catalog.md` (items 12a
+  revised with real numbers, 12c, 12d, 12e — new).
+- Verified one of my own debugging steps was itself wrong (read raw page
+  text from the top instead of the correctly-bounded monograph segment,
+  which briefly looked like a segmentation bug before being traced back to
+  a debugging mistake, not a real defect) — corrected before reporting.
+
+**Not done yet / next up:**
+- Full-corpus re-count with the relaxed ATC regex (whitespace-tolerant,
+  O/0-aware) not yet run — only 14/22 zero-ATC cases spot-checked, and the
+  173/680 multi-ATC count still uses the strict (undercounting) regex.
+- Phase 1 real implementation still pending overall (see earlier entries).
+
+---
+
+## 2026-07-30 — Confirmed class-level monographs and a real source typo
+
+**Done (direct follow-up: "have you checked drug-class entries like Vitamin
+D, or actual spelling/font-size errors?"):**
+- Found and confirmed a **second real example of a class-level monograph**
+  covering multiple ATC codes/substances: "VITAMIN D VÀ CÁC THUỐC TƯƠNG TỰ"
+  (7 ATC codes, one per specific vitamin D analogue) — same pattern as the
+  earlier GONADOTROPIN finding, confirming this is recurring, not a one-off.
+- Found and confirmed a **genuine spelling/capitalization typo in the
+  source PDF itself**: the running header on this monograph's continuation
+  pages reads "Vitamin d..." (lowercase d) vs the correct ALL-CAPS heading
+  "VITAMIN D...". Verified via font/bbox inspection that this is a real
+  source-text inconsistency, not an extraction artifact. The detection
+  heuristic still worked correctly here (the typo'd header isn't all-caps
+  so it's correctly rejected), but this was incidental, not a designed
+  defense against typos.
+- Added both findings to `docs/pdf-parsing-outlier-catalog.md` (items 12a,
+  12b), with the general lesson: rely on multiple independent structural
+  signals, not any single text match, since real source typos do occur.
+- Added `CLAUDE.md` with a standing rule: never fabricate or bluff a claim
+  (number, test result, capability estimate) — verify before stating,
+  explicitly flag estimates as estimates. Grounded in concrete incidents
+  from this investigation (the size-threshold bug, the scope-gap bug).
+
+**Not done yet / next up:**
+- No systematic scan yet for *other* class-level (multi-ATC) monographs
+  beyond the two found incidentally — Phase 1's data model should assume
+  ATC code is a list per monograph regardless, rather than trying to
+  enumerate every class-level entry in advance.
+- Phase 1 real implementation still pending overall (see earlier entries).
+
+---
+
 ## 2026-07-30 — Comprehensive PDF outlier catalog (tables, formulas, columns)
 
 **Done (in response to direct follow-up questions about table/formula