Fix every real lint finding and drop degenerate splice fragments
This commit is contained in:
+25
-6
@@ -80,12 +80,31 @@ methodology, cross-tool comparison, and validation numbers.
|
||||
`ingestion/data/processed/monographs.jsonl` and validated both
|
||||
automatically (see ADR 0003) and via manual spot-check in
|
||||
`ingestion/notebooks/`.
|
||||
3. **Chunking**: each `(drug, section)` pair is the natural chunk unit;
|
||||
never split a section unless it exceeds a token budget (~500-800 tokens),
|
||||
in which case sub-chunk with a sliding window (400 tokens, 50 overlap),
|
||||
tagging the same drug+section metadata plus `part_index`. Every chunk
|
||||
carries `drug_name`, `section_type`, `source_page_range`, `chunk_id` as
|
||||
Qdrant payload — this is what makes citations possible.
|
||||
3. **Chunking** (monograph range only, pp. 99-1496 — see
|
||||
`docs/adr/0004-chunking-strategy.md` for the full measured rationale):
|
||||
each `(drug_id, section_key)` pair is the chunk unit; a section stays one
|
||||
chunk if it's under an **800-token ceiling** (chars/4 estimate — a
|
||||
validated line, not a guess: whole-corpus measurement across 682
|
||||
monographs shows ~16 of 18 section types clear it comfortably at their
|
||||
p90). Two sections routinely exceed it — `dược lý và cơ chế tác dụng`
|
||||
(35.7% of monographs that have it) and `liều lượng và cách dùng`
|
||||
(29.6%) — sub-chunking is the **routine** path for those two, not a rare
|
||||
edge case. Oversized sections are split with a **sentence-boundary-aware
|
||||
sliding window** (~600-700 tokens/sub-chunk, ~1 sentence/50-80 token
|
||||
overlap), never a blind character/line window — PDF line-wrap points
|
||||
are not safe cut points, and a mid-sentence split risks separating an
|
||||
adult/child dosing instruction (a measured, common pattern — outlier
|
||||
catalog item 17) into two chunks. Every chunk carries `chunk_id`,
|
||||
`drug_id`, `drug_name`, `section_key`, `section_display_name`,
|
||||
`atc_codes`, `source_page_range`, `part_index`/`part_count` as Qdrant
|
||||
payload — this is what makes citations possible. **Known open gaps**
|
||||
(see ADR 0004): sub-compound tagging inside class-level/multi-ATC
|
||||
monographs (25.5% of the corpus) is not yet solved; `source_page_range`
|
||||
is monograph-level, not sub-chunk-exact; chunking for general chapters/
|
||||
appendices is a separate, not-yet-designed task; a confirmed
|
||||
header/footer-boilerplate leak into section text (98.4% of monographs
|
||||
affected) must be fixed upstream before this design runs against real
|
||||
data.
|
||||
4. **Embedding + load**: OpenAI `text-embedding-3-small` in batches, upserted
|
||||
into a versioned Qdrant collection (`drug_monographs_v1`) keyed by
|
||||
`chunk_id` for idempotent re-runs; collection aliasing allows re-ingesting
|
||||
|
||||
Reference in New Issue
Block a user