Fix every real lint finding and drop degenerate splice fragments

This commit is contained in:
2026-08-01 13:51:38 +07:00
parent 967b917001
commit 834d9e51b0
69 changed files with 10119 additions and 39 deletions
+25 -6
View File
@@ -80,12 +80,31 @@ methodology, cross-tool comparison, and validation numbers.
`ingestion/data/processed/monographs.jsonl` and validated both
automatically (see ADR 0003) and via manual spot-check in
`ingestion/notebooks/`.
3. **Chunking**: each `(drug, section)` pair is the natural chunk unit;
never split a section unless it exceeds a token budget (~500-800 tokens),
in which case sub-chunk with a sliding window (400 tokens, 50 overlap),
tagging the same drug+section metadata plus `part_index`. Every chunk
carries `drug_name`, `section_type`, `source_page_range`, `chunk_id` as
Qdrant payload — this is what makes citations possible.
3. **Chunking** (monograph range only, pp. 99-1496 — see
`docs/adr/0004-chunking-strategy.md` for the full measured rationale):
each `(drug_id, section_key)` pair is the chunk unit; a section stays one
chunk if it's under an **800-token ceiling** (chars/4 estimate — a
validated line, not a guess: whole-corpus measurement across 682
monographs shows ~16 of 18 section types clear it comfortably at their
p90). Two sections routinely exceed it — `dược lý và cơ chế tác dụng`
(35.7% of monographs that have it) and `liều lượng và cách dùng`
(29.6%) — sub-chunking is the **routine** path for those two, not a rare
edge case. Oversized sections are split with a **sentence-boundary-aware
sliding window** (~600-700 tokens/sub-chunk, ~1 sentence/50-80 token
overlap), never a blind character/line window — PDF line-wrap points
are not safe cut points, and a mid-sentence split risks separating an
adult/child dosing instruction (a measured, common pattern — outlier
catalog item 17) into two chunks. Every chunk carries `chunk_id`,
`drug_id`, `drug_name`, `section_key`, `section_display_name`,
`atc_codes`, `source_page_range`, `part_index`/`part_count` as Qdrant
payload — this is what makes citations possible. **Known open gaps**
(see ADR 0004): sub-compound tagging inside class-level/multi-ATC
monographs (25.5% of the corpus) is not yet solved; `source_page_range`
is monograph-level, not sub-chunk-exact; chunking for general chapters/
appendices is a separate, not-yet-designed task; a confirmed
header/footer-boilerplate leak into section text (98.4% of monographs
affected) must be fixed upstream before this design runs against real
data.
4. **Embedding + load**: OpenAI `text-embedding-3-small` in batches, upserted
into a versioned Qdrant collection (`drug_monographs_v1`) keyed by
`chunk_id` for idempotent re-runs; collection aliasing allows re-ingesting