duocthu/docs-legacy/adr/0005-segment-output-contract-for-chunking.md

ADR 0005: segment/ output contract needed by chunk/ — structure-preserving, not flattened

Status

Proposed. Contract/schema only — no implementation. segment/models.py, segment/assembler.py, and segment/io.py are actively owned by a parallel session on the same checkout at the time of writing; this ADR specifies what chunk/ needs from segment/'s output precisely enough to implement and test, but does not touch those files itself. Supersedes part of ADR 0004 (see "Relationship to ADR 0004" below) — ADR 0004's (drug_id, section_key) chunk-unit-as-leaf assumption is corrected here to (drug_id, section_key) as a parent, with sentence-window splitting demoted from primary strategy to fallback.

Context

ADR 0004 designed chunking against segment/models.py's current output: SectionSpan.text is a single flattened string per section ("\n".join( body_line.strip() for ...)), with all per-line style (Span.bold) and per-line page position discarded once the string is built (confirmed by reading assembler.py: body_lines.append(span.text.strip()) keeps only span.text, nothing else). Review of ADR 0004 surfaced four real problems that trace back to this flattening, not to the chunking algorithm itself:

1. A section is not a single semantic unit. liều lượng và cách dùng and tương tác thuốc routinely contain multiple distinct facts (dosing per patient population, dosing per organ-function impairment, multiple separate drug interactions) that a doctor may want to retrieve independently. Measured: an explicit population marker ("Người lớn"/ "Trẻ em"/"Trẻ sơ sinh"/"Suy thận"/"Suy gan" immediately followed by : or .) appears in 303 of 675 monographs (44.9%) that have a liều lượng và cách dùng section — this is common, not an edge case.
2. A blind sentence-boundary sliding window (ADR 0004's original sub-chunking method) can still split two different facts into the same chunk, or split one fact across two chunks, because it has no way to know a population/interaction boundary exists — that information exists in the source (as a bold or otherwise visually distinct sub-heading, per direct reading of MORPHIN SULFAT/VITAMIN D section text: lines like "Thuốc uống", "Cách dùng:" render as isolated bold short lines in the PDF) but is discarded before chunk/ ever sees it.
3. Tables inside the monograph range are not addressed at all. ADR 0004 implicitly assumed monograph-range sections are prose. docs/pdf-parsing- outlier-catalog.md item 19 already documents a real table (dosing by renal function, HSV/CMV columns) inside a monograph body (Foscarnet natri, physical page 698) — flattening a table's rows into newline-joined body text destroys its row/column structure exactly the way outlier item 7 already describes for the appendix's 2D nomogram table. A whole-range survey to size this properly is in progress alongside this ADR (see "Not yet resolved" below).
4. Provenance is section-level, not chunk-level, because per-line physical_page/y0 (which Span already carries — see extract/models.py) is discarded at the same flattening point. For a section spanning several physical pages, a sub-chunk built from its final third currently has no way to know its own real page — it can only inherit the whole monograph's source_page_range. For medical citations this is not precise enough.

A fifth, independently-found data-quality bug makes precise provenance even more necessary, not less: the corpus's last-processed monograph (ZOLPIDEM) is never closed until true end-of-stream, and assembler._classify calls match_section()/match_section_with_inline_value() on every span with no in_monograph_range gate (unlike _TextEvent handling, which does check it). A spurious bold-text match on physical page 1655 — deep in the back-of-book "Mục lục tra cứu" brand-name index, confirmed by reading that page directly — overwrote ZOLPIDEM's real tương tác thuốc SectionSpan with an empty one and corrupted its source_page_range to [1492, 1655]. This is real content loss (measured: exactly 1 monograph affected, the last one processed — every other monograph is closed on schedule by the next monograph title, which is range-gated). Flagged for the session that owns extract/segment, not fixed here.

Decision

Extend segment/models.py's SectionSpan with a structured, line-level representation, additive to (not replacing) the existing flat text field — chunk/ becomes a real, structure-aware consumer instead of re-deriving structure from a flattened string via ad hoc regex.

New/changed types (segment/models.py)

@dataclass(frozen=True)
class BodyLine:
    text: str
    physical_page: int
    y0: float
    bold: bool          # Span.bold, preserved instead of discarded

@dataclass
class SectionSpan:
    key: str
    display_name: str
    heading: Heading
    text: str                              # UNCHANGED meaning, kept for
                                            # backward compat (see invariant below)
    lines: List[BodyLine] = field(default_factory=list)   # NEW

lines carries exactly the per-line signal chunk/ needs to do its own job (population/subheading detection, precise page provenance) without segment/ having to know anything about chunking — segment/'s responsibility stays "detect boundaries and preserve source structure," not "decide what a retrieval unit is" (Clean Architecture / SoC, per CLAUDE.md). Specifically, this is deliberately not a is_subheading: bool field computed by segment/ — classifying "is this line a subheading a chunker should split on" is a chunking-time decision (what counts as a good split point can vary by strategy/eval results), not a segmentation-time one. segment/ should stop discarding the raw signal (bold, y0, physical_page) it already has per span; it should not also start doing chunk-shaping judgment calls.

Invariants

1. text == "\n".join(l.text for l in lines).strip() for every SectionSpan, for the lifetime of this contract — lines is a strictly additive refinement, never a divergent second source of truth. Any change to how body text is assembled (e.g. the boilerplate-stripping fix already applied by the other session) must update both fields from the same filtered span list, not text alone.
2. lines is in reading order, matching the order text's lines already implicitly have.
3. Every BodyLine.physical_page satisfies detector.in_monograph_range for a Span on that page — i.e., no line in any SectionSpan.lines may come from outside the monograph's real printed-page range. This is the ZOLPIDEM bug's exact failure mode stated as an invariant: it was violated (a spurious section event was accepted from a fully out-of-range page precisely because no such check existed for section events, only for body text events). Enforcing this invariant closes that bug as a side effect, but the invariant is stated here as a contract requirement independent of any specific fix implementation.
4. Every currently-open monograph must be finalized exactly once, at either (a) the next monograph title, or (b) true end-of-stream — with no third path (e.g., a stray out-of-range section match) able to silently mutate an already-"complete" monograph's sections after point (a) would otherwise have applied. (This is a restatement of invariant 3 from the monograph-lifecycle side, not a new requirement.)

Migration impact

• segment/io.py (_monograph_to_dict/_monograph_from_dict, write_monographs_jsonl/read_monographs_jsonl): additive — serialize lines alongside the existing text/heading fields per section. Existing consumers reading only text (e.g. segment/atc.py's extract_atc_codes, which regexes over SectionSpan.text) need no change, per invariant 1.
• ingestion/data/processed/monographs.jsonl: schema grows a new optional-shaped field (sections[key].lines). No schema_version field currently exists in the serialized dict (checked io.py directly) — worth adding as part of this change, both for this migration and because docs/architecture.md already assumes "collection aliasing allows re-ingesting with a changed chunking strategy," which implies the ingestion output itself should be able to declare which schema shape it is.
• Existing 110 tests: unaffected if invariant 1 holds — no assertion in the current suite inspects lines (it doesn't exist yet), and text's value/semantics are unchanged.
• New tests required (this ADR specifies them; implementation and the actual test code are not part of this ADR):
  1. Regression test reproducing the ZOLPIDEM failure shape: a synthetic span stream — last monograph's title and real sections, followed by spans whose printed_page is out of in_monograph_range but whose text matches a vocab.py section label — asserting the monograph closes with its real sections intact and the out-of-range spurious match is ignored, not accepted.
  2. SectionSpan.lines fixture test: using the real MORPHIN SULFAT boilerplate-fix fixture already in tests/test_segment_assembler.py, assert lines preserves the correct bold/physical_page/y0 per retained line (and that stripped boilerplate lines are absent from lines too, not just from text).
  3. Round-trip test: write_monographs_jsonl → read_monographs_jsonl preserves lines exactly (dataclass equality per line).
  4. Whole-corpus invariant-1 check: for a real cli run output, assert text == "\n".join(l.text for l in lines).strip() holds for every section of every monograph, not a sample.

Relationship to ADR 0004

ADR 0004's chunk-unit decision ((drug_id, section_key)) is not discarded — a section is still the natural parent grouping (matches how a clinician thinks, matches Monograph.sections). What changes: ADR 0004 described a section as directly the chunk when under the 800-token ceiling, with sentence-window splitting as the fallback for oversized sections. Per the review above, splitting must instead first attempt to break at real structural boundaries available in SectionSpan.lines (a bold, short, isolated line — the same "subheading" shape already visually confirmed for route-of-administration/population sub-headers — or an explicit population/organ-function marker), with the sentence-window method demoted to a fallback for the remaining prose that has no such marker. The exact splitting algorithm (how a "subheading-shaped line" is defined precisely, in code) is a chunk/-side implementation detail enabled by this contract, not decided by it.

Not yet resolved (explicitly out of scope for this ADR)

• Table/formula content blocks. A separate whole-monograph-range survey (pdfplumber find_tables() + PyMuPDF math-symbol scan, physical pages 98-1494 excluding blank page 99 — the exact set detector. in_monograph_range accepts, not an assumed offset) is in progress at the time of writing, per explicit user instruction to measure before deciding a table/formula chunk-unit strategy. This ADR's BodyLine contract covers text content only; a table/formula region should not currently be flattened into BodyLines (doing so would repeat exactly the "destroys row/column meaning" mistake outlier item 7 already documents) — but the precise ContentBlock/table-row/formula-unit shape is deferred to a follow-up revision of this ADR once the survey reports real numbers (how many monographs/sections affected, page-break continuation frequency, multi-tier headers, merged cells, footnotes).
• Paragraph-boundary detection (grouping consecutive BodyLines into a flowing paragraph vs. a new one) is left to chunk/, using the same kind of y-gap heuristic segment/merge.py already validates for multi-line title wraps (_MAX_LINE_GAP_PT) — BodyLine.y0 is sufficient raw signal for chunk/ to compute this itself; segment/ does not need to pre-compute paragraph grouping.
• The actual chunk/ splitting implementation (subheading detector, population-marker regex, sentence-window fallback) is not part of this ADR — this ADR defines the data contract that implementation will consume.