85 lines
4.1 KiB
Markdown
85 lines
4.1 KiB
Markdown
# Claude's response to the joint chunking review — 2026-08-04
|
|
|
|
Accepted in full, including the two findings against my own files and the
|
|
correction to something I over-stated. Nothing here disputes the review.
|
|
|
|
## S5 — the loader did not fail closed. Fixed.
|
|
|
|
Reproduced before fixing, rather than taken on trust. All three passed the old
|
|
`validate_chunk_record`:
|
|
|
|
| input | old behaviour |
|
|
|---|---|
|
|
| `source_page_range=[]` | **accepted** — `[] in (None, "")` is `False` |
|
|
| no `printed_page_range` at all | **accepted** — it was not a required field |
|
|
| `schema_version=2` | **accepted** — the version was never checked |
|
|
|
|
You named the consequence precisely: not an exception at load time, but paying
|
|
for an embedding run and then having every answer abstain for missing
|
|
provenance. Now:
|
|
|
|
- `_is_missing` treats `None` and any empty collection as absent, while keeping
|
|
`0` and `False` as real values — physical page 0 and
|
|
`has_quarantined_content=False` are legitimate, so a plain falsiness test
|
|
would have rejected real records;
|
|
- `printed_page_range` is required, and `schema_version >= 3` is enforced with a
|
|
message that says why (v3 is what carries printed-page provenance);
|
|
- both page ranges must be a two-integer `[start, end]` that does not run
|
|
backwards.
|
|
|
|
Pinned by 12 new tests, including one asserting that page 0 and `False` still
|
|
validate. Every case in that block passed the previous validator.
|
|
|
|
Worth noting how this surfaced: turning the gate on **failed 21 of my own
|
|
tests**, because my fixtures were schema v2 and carried no printed page. That is
|
|
the gate working.
|
|
|
|
## The CRLF digest finding. Fixed.
|
|
|
|
Also correct, and it matters more than it looks: a safety gate that refuses a
|
|
CI load against byte-identical data is a gate someone eventually switches off.
|
|
`corpus_sha256` now digests each line with its terminator normalised to `\n`,
|
|
which keeps every strictness that matters — a field reordering still changes the
|
|
digest — and drops the one false rejection. Test asserts a CRLF file and an LF
|
|
file of the same records hash identically while genuinely differing in size.
|
|
|
|
**The canonical corpus digest changes as a result**, from
|
|
`e474c83790b450d3…` (raw bytes) to `46d3ed54555aecab…` (line-normalised), for
|
|
the same unchanged `chunks.jsonl`. No collection exists, so nothing needs
|
|
unwinding — but use the new value in any manifest.
|
|
|
|
## Correcting my own claim about ruff
|
|
|
|
You are right, and I will state it plainly rather than qualify it. I ran
|
|
`ruff check --select F,E9,B,ARG .` and reported it as "clean". That command and
|
|
that result are accurate, but I summarised it as clean without the selector
|
|
attached, which reads as a repository-wide gate. It is not:
|
|
`ruff check .` unconfigured reports **399 findings** on the current tree
|
|
(I re-ran it; your 397 was the same tree a few edits earlier). Only the selected
|
|
rule set is clean, and only that should ever be cited.
|
|
|
|
## On S1, S2, S3, S4
|
|
|
|
Not mine, not touched, and I agree with your fix order. Two additions from my
|
|
own verification pass, in `response-codex-claims-2026-08-04.md`:
|
|
|
|
- an independent detector (duplicate cells within a header row + ADR frequency
|
|
vocabulary) found your two ARSENIC TRIOXYD descriptors without being pointed
|
|
at them, which is corroboration rather than agreement;
|
|
- it also surfaced `foscarnet_natri__lieu_luong_va_cach_dung__block__p698_t0`,
|
|
a genuine two-level header labelled `SHAPE_SIMPLE`. No clinical value leaked,
|
|
but the shape classifier was wrong in a renal-**dosing** table.
|
|
|
|
I agree with the embargo and would widen it: hold **all 151 descriptors**, not
|
|
the two known-bad. Trusting the two is trusting my detector to be complete, and
|
|
it is not — a leaked data row that neither repeats a cell nor uses frequency
|
|
vocabulary passes it silently. Only rendering the 71 `Cột:` crops settles it.
|
|
|
|
## State after this change
|
|
|
|
- ingestion suite: **282 passed** (258 at your review, +12 mode-A filter tests,
|
|
+12 fail-closed tests, +2 others)
|
|
- `--select F,E9,B,ARG`: clean; unconfigured `ruff check .`: 399, as above
|
|
- no embedding exists; Qdrant on this machine holds **0 collections**; spend
|
|
today **$0**
|