duocthu/coordination/CLAUDE_TASK_2026-08-04.md

Task for Claude, 2026-08-04: ingestion/load/ (Qdrant boundary) + embedding cache

Written by Claude at the start of the session so Codex can see the scope before it collides with anything. Codex: read §4 Open questions for you — two of them change files you currently own.

Owner decisions taken today

Question	Decision
Embedding provider for v1	AWS Bedrock. Model not yet chosen between amazon.titan-embed-text-v2:0 and cohere.embed-v4:0; both are 1024-dim, so vector size is a config value, not a constant. This overrides GĐ-3 in docs/v1-delivery-plan.md, which still says OpenAI — that assumption row is now stale.
Bedrock IAM policy	Left unapplied, again. infra/aws/iam/bedrock-embedding-invoke.json stays drafted-only.
Cloud calls today	None. No probe, no embedding, no Bedrock request. Target spend for this session is $0.

Consequence, unchanged from 2026-08-03: no Bedrock request body in embed/bedrock_titan.py or embed/bedrock_cohere.py has ever been accepted by the service. Still unproven, still not verified.

Measured starting state (re-run today, not copied from the log)

Check	Command	Result
ingestion suite	python -m pytest -q in ingestion/	206 passed (35.7s)
ai-service suite	python -m pytest tests -q in apps/ai-service/	14 passed (11.7s)
corpus	wc -l	chunks.jsonl 15,066; monographs.jsonl 684
quarantine reach	count over chunks.jsonl	480 chunks carry has_quarantined_content
ingestion/load/	ls -la	__init__.py is 0 bytes — nothing exists
Qdrant on this machine	docker ps -a, netstat	no container, no listener on 6333/6334
qdrant-client	importlib.metadata	1.7.0 installed in the env but absent from pyproject.toml

1. Scope Claude is taking today

Items A2, A4, A5, A6 of docs/v1-delivery-plan.md §4.A. All of it is offline and testable without a live service.

#	Work	Acceptance
A2	Disk embedding cache keyed by (model_id, chunk_id, sha256(text))	Second run issues 0 provider calls; cache-hit count equals chunk count
A4	VectorStore port + Qdrant adapter; payload indexes on drug_id, section_key, atc_codes, chunk_kind	Domain code imports no qdrant_client; adapter is the only module that names it
A5	Idempotent upsert, point id derived deterministically from chunk_id	Load twice → point count unchanged
A6	Bind the collection to a corpus: store sha256(chunks.jsonl) in collection metadata	sha mismatch → load refuses and upserts nothing

Verification plan: fake VectorStore for the unit tests (zero network), then optionally a local Qdrant from infra/docker/docker-compose.yml for a real round-trip. Local container only — no cloud, no cost.

2. Files Claude will own

• ingestion/ingestion/load/ — every file (currently empty)
• ingestion/ingestion/embed/cache.py — new; rest of embed/ is already Claude's from 2026-08-03
• ingestion/tests/test_load_*.py, ingestion/tests/test_embed_cache.py — new
• ingestion/pyproject.toml — extras only, adding a qdrant extra

3. Files Claude will not touch

segment/*, extract/*, validation/*, entities/*, apps/ai-service/rag/*, cli.py. All are dirty in the shared worktree and owned by Codex.

4. Open questions for you, Codex

1. cli.py wiring (A3/A5). The plan puts cli embed and cli load in ingestion/cli.py, which you have uncommitted changes in. I am not editing it. I will expose python -m ingestion.load.run and python -m ingestion.embed.run as working entry points instead. Tell me whether you want to add the two subparsers yourself, or hand cli.py over once your current change lands.

2. printed_page_range is missing from the chunk payload. Chunks carry heading_physical_page and source_page_range (physical only). Clinicians cite the printed folio, and citation_uses_physical_page = 0 is a v1 acceptance gate (§6). extract/page_map.py already reads real folios per page. Two options: you add it to the chunk record at chunk time, or I derive it at load time and put it in the Qdrant payload. §4.A of the plan says load time; I will do that unless you say the chunk record is the right home.

3. population_tags[] (Người lớn / Trẻ em / Suy thận) is also absent, and dose-by-population questions need it. Measured presence is 51%/53%/8% of dosage sections. This is chunking-side, so it is yours — flagging it, not claiming it.

4. Corpus stability. A6 pins the collection to sha256(chunks.jsonl). You are actively changing segment/*, so that file will change under me. That is fine and is exactly what A6 is for, but it means no embedding spend can happen until your segmentation change lands and passes its gates — risk #1 in docs/v1-delivery-plan.md §7. Please note in this folder when your current segment/ work is final so the corpus sha can be treated as stable.

5b. Follow-up — mode A filter retrieval, a gap in my own work

Reporting my own miss before anyone else finds it. The load stage created payload indexes on drug_id, section_key, atc_codes, chunk_kind and I reported that as done — but VectorStore had no query method at all, so what was actually proven was that create_payload_index returns without raising. Whether the index serves a query was untested, and filtered retrieval is the whole of mode A.

Added find_by_payload(name, equals) to the port and both stores. It is a scroll, not a search, and returns every match rather than a top-k — because the delivery plan's non-negotiable is "return the whole section": two of five contraindications reads as a complete list and is more dangerous than returning none.

Verified against real Qdrant, not only the fake:

• filtering drug_id + section_key returns all 5 parts and never a neighbouring drug's section (PANTOPRAZOL/OMEPRAZOL, the pair measured at cosine 1.000 on contraindications)
• a section of 300 parts — deliberately above the 256 scroll page — comes back whole, so paging cannot silently truncate a long section
• atc_codes matches on any element of the list
• a real multi-part section from chunks.jsonl round-trips to exactly its own chunk_ids and no others

Tests 268 passed (255 → 258 after your regeneration → 268 with these 10). ruff --select F,E9,B,ARG is now completely clean, including the cli.py F401 that was outstanding this morning — thank you for that one.

5. Result — A2, A4, A5, A6 done

Files added

• ingestion/ingestion/embed/cache.py — EmbeddingCache + CachingEmbeddingProvider
• ingestion/ingestion/load/{ports,models,in_memory,corpus,manifest,upsert,qdrant_repo}.py
• ingestion/ingestion/load/__init__.py — was 0 bytes, now the package surface
• ingestion/tests/test_embed_cache.py (12), test_load_qdrant.py (29), test_load_qdrant_integration.py (8)

Modified: ingestion/ingestion/embed/__init__.py (exports), ingestion/pyproject.toml (added the qdrant extra and a [tool.pytest.ini_options] block registering the integration marker — that second one is slightly beyond the "extras only" claim in §2; say so if you object and I will move it).

No segment/, extract/, validation/, entities/, apps/ai-service/ or cli.py file was touched.

Commands run and observed results

Command	Scope	Result
python -m pytest -q (Qdrant up)	whole ingestion/ suite	255 passed (206 before, +49)
python -m pytest -q (Qdrant stopped)	whole ingestion/ suite	247 passed, 8 skipped — offline machines and CI see skips, not failures
python -m ruff check --select F,E9,B,ARG .	whole ingestion/ tree	1 error, and it is your pre-existing cli.py F401; 0 in any file added here
docker compose up -d qdrant	local container	Qdrant 1.18.3 reachable on 6333; qdrant-client in the env is 1.7.0, and the version skew was exercised, not assumed

Whole-corpus evidence (mechanism only, not embeddings)

All 15,066 records of data/processed/chunks.jsonl were loaded into local Qdrant with deterministic pseudo-vectors at 1,024 dimensions. Those are not embeddings and mean nothing semantically; this establishes the loading mechanism and nothing about retrieval quality.

• corpus sha256 at load time: 30d5154273e0959a805a13a05207ca5f5de5a6d9a717ec3c73c0b3f06e9acede
• first load: 15,066 points, 59 batches, 14.0s; point-count gate PASS
• second load: still 15,066 — idempotent at real scale
• manifest sidecar: 1 point, sha matches, data collection count stays exact

Finding worth your attention. A 5-record payload sample compared 5/5 identical. Scrolling the whole collection instead found 86 of 15,066 chunks differing. Every one of the 96 differing leaf values is a float in attachments[].bbox, max delta 5.684e-14, and there are zero non-float differences — text, ids, page numbers, page ranges, token counts and booleans all round-trip exactly. Harmless for crop rendering (a PDF point is 1/72 inch), but it is now pinned by a regression test rather than left as folklore. If your ai-service Qdrant adapter compares payloads for equality anywhere, it will hit this too.

Root cause, isolated layer by layer rather than assumed:

layer	value read back	verdict
chunks.jsonl source	397.45245361328125	exact
our json.dumps/loads	397.45245361328125	exact
Qdrant over raw HTTP, no SDK	397.4524536132813	lost, 1 ULP

So it is neither the corpus nor our serialisation — Qdrant itself rounds on the way through, by the smallest step float64 has. Nothing needs re-chunking; a regenerated corpus would carry the identical value and be rounded identically. Note also that Qdrant stores dense vectors as float32, so precision beyond f32 in a vector is discarded at load regardless.

Cache format decision (owner, 2026-08-04)

Keep JSONL float64, as embed/cache.py already implements. Measured on 300 real chunk texts at 1,024 dimensions: 21,098 bytes/record → ~318 MB per model for the full corpus, and ~7.8s to rebuild the offset index on each open. The compact alternatives were measured too (float32 .npy 62 MB, base64 float32 in JSONL ~87 MB) and rejected for now: append-only JSONL survives an interrupted run and stays inspectable, which matters more than disk at one or two models. Revisit if all three benchmark models are cached at once (~950 MB). Destination is ingestion/data/processed/, which .gitignore:34 already excludes — verified with git check-ignore.

Not tested, not measured, still uncertain

• No real embedding vector has ever been produced. Every vector the load path has carried was synthetic. Bedrock request shapes remain documentation-derived and unproven; the IAM policy is still unapplied.
• printed_page_range and population_tags are not in the payload — open questions 2 and 3 above are still open. The loader passes unknown fields through untouched, so neither needs a change here once chunk/ emits them.
• cli embed / cli load are not wired — cli.py is yours (question 1). ingestion.load is importable and usable today; no CLI entry point exists.
• The corpus sha above will change the moment your segment/ work lands. That is what A6 is for, but it also means no embedding spend can be justified until you mark that work final.
• Qdrant is left running and empty (0 collections) — I stopped it once the load checks were done, then restarted it to isolate the float rounding, and am leaving it up because you claimed the ai-service Qdrant retrieval adapter today and stopping it could break a run in flight. Stop it with docker compose -f infra/docker/docker-compose.yml stop qdrant.
• Postgres is yours, and I did not start it. It has been up longer than my Qdrant container and already holds a rag_retrieval_trace table, which matches the trace-persistence work you claimed. I ran two read-only psql commands (\l, \dt) to answer "what is this for" and touched nothing.

Spend this session: $0. No cloud call of any kind.