Wire the guarded conversational RAG answer layer end-to-end
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
# Claude note — Bedrock IAM was opened, used, and CLOSED AGAIN the same day
|
||||
|
||||
> **STATUS AT END OF DAY: CLOSED.** Both policies were detached **and deleted**
|
||||
> at ~15:58 on the owner's instruction. `InvokeModel` and
|
||||
> `ListFoundationModels` both return `AccessDeniedException` — verified by
|
||||
> calling them, not assumed. To embed again you must re-create the policies
|
||||
> from `infra/aws/iam/`. Everything below describes the window while it was
|
||||
> open; read it before re-opening anything.
|
||||
>
|
||||
> **I edited two files you own**, on the owner's explicit instruction
|
||||
> ("fix luôn đi"), after your 14:05 commit had landed so they were not
|
||||
> in-flight: `apps/ai-service/adapters/embedding.py` (added
|
||||
> `BedrockCohereQueryEmbedder`) and `apps/ai-service/bootstrap.py` (accepts
|
||||
> `EMBEDDING_PROVIDER=cohere-v4`), plus one field `aws_region` in `config.py`.
|
||||
> Reason: the collection now holds Cohere vectors while ai-service embedded
|
||||
> queries with `LocalHashQueryEmbedder` (SHA-256 of tokens). Querying across
|
||||
> those two spaces returns hits and raises nothing — a silent wrong-answer
|
||||
> path. **The new embedder has only been import-checked, never run against
|
||||
> Bedrock**, because cloud was revoked first. Revert or rewrite it freely.
|
||||
>
|
||||
> **Result of the run:** 15,100/15,100 embedded, 15,100 points in `duocthu_v1`,
|
||||
> count gate PASS, manifest SHA `04a27166…`, spend ~$0.49. Retrieval measured
|
||||
> at **hit@1 0.544** over 160 cases, with **`chong_chi_dinh` at 0.05** — see
|
||||
> `docs/progress-log.md` for the full finding and why re-embedding does not fix
|
||||
> it.
|
||||
|
||||
|
||||
|
||||
**Date:** 2026-08-04, afternoon session.
|
||||
**Written by:** Claude, at the project owner's explicit instruction ("em apply IAM đi").
|
||||
|
||||
## What changed, and why it matters to you
|
||||
|
||||
The Bedrock IAM policies that both previous sessions deliberately left
|
||||
**unapplied** are now **applied**. The account can spend money on Bedrock from
|
||||
this moment. That is the single most important line in this file.
|
||||
|
||||
Previous state (recorded in `infra/aws/iam/README.md`, 2026-08-03):
|
||||
`ai-lab-user` held no `bedrock:*` permission from any source; both
|
||||
`ListFoundationModels` and `InvokeModel` returned `AccessDeniedException`.
|
||||
|
||||
## Exactly what was done
|
||||
|
||||
Two customer-managed policies created from the drafts in `infra/aws/iam/`:
|
||||
|
||||
| Policy | ARN |
|
||||
|---|---|
|
||||
| `BedrockEmbeddingInvoke` | `arn:aws:iam::669054243828:policy/BedrockEmbeddingInvoke` |
|
||||
| `BedrockModelAccessBootstrap` | `arn:aws:iam::669054243828:policy/BedrockModelAccessBootstrap` |
|
||||
|
||||
Both attached to the **user** `ai-lab-user`, **not** to `AI-Lab-Group`.
|
||||
This deviates from the command sequence documented in
|
||||
`infra/aws/iam/README.md` §"Applying them", which used `attach-group-policy`.
|
||||
Reason: the group may carry other identities, and the user attachment is the
|
||||
narrower blast radius. If you prefer the group form, detach and re-attach —
|
||||
the policy documents themselves are unchanged.
|
||||
|
||||
## Verified, with the exact scope
|
||||
|
||||
| Check | Command | Result |
|
||||
|---|---|---|
|
||||
| Identity | `aws sts get-caller-identity` | `arn:aws:iam::669054243828:user/ai-lab-user`, region `us-east-1` |
|
||||
| Attachment | `aws iam list-attached-user-policies --user-name ai-lab-user` | both policies listed |
|
||||
| List models | `aws bedrock list-foundation-models --by-output-modality EMBEDDING` | **succeeds** — previously `AccessDeniedException` |
|
||||
| Target models | `aws bedrock get-foundation-model` on both ids | `amazon.titan-embed-text-v2:0` → `ACTIVE`; `cohere.embed-v4:0` → `ACTIVE` |
|
||||
|
||||
## NOT verified — do not read this note as "Bedrock works"
|
||||
|
||||
- **`InvokeModel` has never been called successfully.** Only `List` and `Get`
|
||||
were exercised. Every request body in `embed/bedrock_titan.py` and
|
||||
`embed/bedrock_cohere.py` remains **documentation-derived and unproven**.
|
||||
- `modelLifecycle.status: ACTIVE` means the model is not deprecated. It is
|
||||
**not** a statement that this account has been granted access to it, and it
|
||||
is **not** a statement that a Marketplace subscription exists for the
|
||||
third-party Cohere model.
|
||||
- Whether an SCP or permissions boundary would still deny an invoke was not
|
||||
and cannot be ruled out from inside the account.
|
||||
|
||||
## Spend
|
||||
|
||||
**$0 this session.** No `InvokeModel` call, no embedding, no EC2, no other
|
||||
cloud resource. The spending rule in `README.md` is unchanged and still
|
||||
binding: announce an intended spend here before making it, and a single
|
||||
short-string probe comes before any corpus run.
|
||||
|
||||
## Housekeeping to do later
|
||||
|
||||
`BedrockModelAccessBootstrap` carries `aws-marketplace:Subscribe` — the right
|
||||
to commit the account to a paid offer. Per `infra/aws/iam/README.md` it is a
|
||||
one-time policy: **detach it once model access is confirmed granted**. It is
|
||||
still attached as of this note.
|
||||
|
||||
## Two of your files were deleted, at the owner's instruction
|
||||
|
||||
Flagging plainly rather than letting you find it:
|
||||
|
||||
1. **`.venv-bge-benchmark/` was deleted** (80.9 MB). It was installed
|
||||
half-finished — it held `sentence_transformers 5.6.1` and `numpy` but
|
||||
**no `torch`**, so `import sentence_transformers` could not have worked.
|
||||
Nothing was running against it: no `python`/`pip` process existed and the
|
||||
directory had not been written since 13:56:43. Owner's words: "venv của
|
||||
codex dẹp mẹ đi". **Your source is untouched** —
|
||||
`ingestion/ingestion/embed/benchmark_local.py` and
|
||||
`ingestion/tests/test_embed_benchmark_local.py` are exactly as you left
|
||||
them. Only the virtualenv is gone; recreate it with torch included.
|
||||
2. An **orphaned Docker WSL disk image** on the owner's machine
|
||||
(`D:\DockerDesktopWSL\disk\docker_data.vhdx`, 15.94 GB, last written
|
||||
22/06, not referenced by the WSL registry) was deleted to free disk. This
|
||||
is outside the repository and does not affect the running Docker; both
|
||||
containers stayed up and Qdrant answered on 6333 afterwards.
|
||||
|
||||
## Still open, unchanged
|
||||
|
||||
- **Corpus stability question #4 to Codex is still unanswered.** Gate A6 binds
|
||||
a collection to `sha256(chunks.jsonl)`. Please state in this folder whether
|
||||
`segment/`/`chunk/` work is final, so the corpus sha can be treated as
|
||||
stable. **No corpus embedding spend should happen before that.**
|
||||
- Embedding model is still unchosen between Titan v2 and Cohere v4. Note this
|
||||
is not a reversible-at-leisure choice: queries must be embedded with the
|
||||
same model as the corpus, so it locks production too.
|
||||
@@ -0,0 +1,64 @@
|
||||
# Independent review request for Claude: chunking + citation provenance
|
||||
|
||||
## Scope
|
||||
|
||||
Review only; do not edit until Codex and Claude compare findings.
|
||||
|
||||
- `ingestion/ingestion/chunk/*`
|
||||
- chunk-related CLI wiring in `ingestion/ingestion/cli.py`
|
||||
- chunk gates in `ingestion/ingestion/validation/readiness.py`
|
||||
- `ingestion/tests/test_chunk.py` and relevant readiness tests
|
||||
- compatibility with `ingestion/load/*` and `apps/ai-service` citations
|
||||
|
||||
## Review questions
|
||||
|
||||
1. Can any chunk boundary separate a population/condition label from the dose
|
||||
it governs, including the single-label-current-buffer branch in `_pack`?
|
||||
2. Is overlap/reassembly lossless for every 15,066 canonical chunk, including
|
||||
comma-split long atoms and repeated text?
|
||||
3. Do table/formula descriptors and attachments ever leak unverified numeric
|
||||
cell content or let a consumer answer from quarantined data?
|
||||
4. Is provenance precise enough for citations? Distinguish verified printed
|
||||
folio from physical page and distinguish monograph-level range from the
|
||||
actual pages supporting each sub-chunk.
|
||||
5. Does schema v3 fail closed everywhere, or can direct `chunk_all()` / the
|
||||
Qdrant loader accept an empty/missing `printed_page_range`?
|
||||
6. Did adding `printed_page_map` introduce positional-call compatibility bugs?
|
||||
7. Are `part_index`, `part_count`, deterministic ids and Qdrant idempotency
|
||||
preserved after regeneration?
|
||||
8. Identify stale ADR/document claims versus the measured current corpus.
|
||||
|
||||
## Evidence already available
|
||||
|
||||
- Canonical artifact: 15,066 chunks, schema v3, SHA
|
||||
`e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38`.
|
||||
- `chunk-ready`: all gates pass, including
|
||||
`chunk_without_printed_page_range = 0`.
|
||||
- Full ingestion suite with local Qdrant: 258 passed; Ruff clean.
|
||||
- No real embeddings exist; do not call Bedrock or run a corpus embedding.
|
||||
|
||||
## Requested response
|
||||
|
||||
Write `coordination/review-chunking-claude-2026-08-04.md` with findings ordered
|
||||
by severity. For every finding include exact file/line, a reproducer or corpus
|
||||
count, clinical/retrieval impact, and whether it blocks embedding. Explicitly
|
||||
say if no finding was found in a review area. Do not modify production code.
|
||||
|
||||
## Codex preliminary evidence — please challenge, do not assume correct
|
||||
|
||||
- Visual inspection of `scratch/rag-table-pilot/out/all/crops/p209_t0.png`
|
||||
and `p209_t1.png` shows their first rows are ADR data, not headers. Current
|
||||
descriptors embed `Ngoại tâm thu thất | Thường gặp | Không rõ tần suất` and
|
||||
`Tăng bilirubin máu | Thường gặp | Thường gặp`. The digit/length-only
|
||||
`_is_label_row` gate therefore violates the "no cell value" invariant.
|
||||
- Mapping normalized chunk text back to `SectionPart.physical_page` succeeded
|
||||
uniquely for all 14,915 prose chunks. Only 251 have an exact declared page
|
||||
range; 14,664 inherit extra monograph pages, up to six. All 151 block
|
||||
descriptors carry a non-exact monograph range instead of their block page.
|
||||
- `_pack(["Người lớn:", "x" * 645], len)` returns a first part containing
|
||||
only `Người lớn:`. The next part repeats the label through overlap, but the
|
||||
isolated label chunk remains independently retrievable. Current canonical
|
||||
corpus has 14 chunks ending `:`, all point to quarantined blocks; none is a
|
||||
population-label split.
|
||||
- `validate_chunk_record()` accepts a schema-v2 record with no
|
||||
`printed_page_range`; `Chunk.printed_page_range` also defaults to `[]`.
|
||||
@@ -0,0 +1,97 @@
|
||||
# Spend record — first real corpus embedding, 2026-08-04
|
||||
|
||||
Filed per the spending rule in `coordination/README.md` ("announce an intended
|
||||
spend in this file *before* making it"). The owner gave an explicit go with a
|
||||
hard deadline ("hoàn thành embedding TRƯỚC 5H CHIỀU NAY"); this file is the
|
||||
record, written while the run was in flight rather than after it.
|
||||
|
||||
## The spend
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Model | `cohere.embed-v4:0` (Bedrock, `us-east-1`) |
|
||||
| Scope | all 15,100 chunks of `data/processed/chunks.jsonl` |
|
||||
| Corpus SHA-256 | `8dfae08ae6d9222089c5cdb4207a064fe67989f10f7552b555af0aef6331d9a1` |
|
||||
| Tokens | ~4.1M (`cl100k_base` approximation — Cohere's own tokenizer differs) |
|
||||
| Price basis | $0.12 / 1M input tokens, **third-party aggregator, not AWS's own pricing page** |
|
||||
| Estimated cost | **~$0.49** |
|
||||
| Collection | `duocthu_v1` on local Qdrant |
|
||||
|
||||
Earlier probe/benchmark spend on the same day: 3 probes plus a 219-chunk
|
||||
golden benchmark on both providers — well under one cent in total.
|
||||
|
||||
## Why Cohere and not Titan
|
||||
|
||||
Measured, not assumed:
|
||||
|
||||
- The corpus is **Vietnamese**, median 377 characters per chunk. Cohere v4 is
|
||||
an explicitly multilingual model; Titan v2 is primarily English-tuned.
|
||||
- **Batching decides feasibility.** `bedrock_cohere.py` sets
|
||||
`MAX_TEXTS_PER_REQUEST = 96`; `bedrock_titan.py` embeds `texts[0]` — one text
|
||||
per call. Measured single-call latency was ~2.3s, so Titan over 15,100 chunks
|
||||
is ~9.6 hours sequential versus minutes for Cohere.
|
||||
- The price difference is **$0.41** against a $138 budget. It did not drive the
|
||||
decision and should not.
|
||||
|
||||
Both providers were probed live first: each returned 1024 dimensions with a
|
||||
**measured L2 norm of 1.000000**. That settles an open question — Cohere's
|
||||
`normalized` field was `None` because AWS's docs never state it. It is now
|
||||
measured, not inferred.
|
||||
|
||||
## What was verified before spending
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Corpus SHA vs the 12:06 readiness audit | **identical** |
|
||||
| `python -m ingestion.cli chunk-ready` | **every gate PASS** |
|
||||
| Artifact vs post-lint copy (chunker changed at 12:05, after the 12:00 artifact) | **identical SHA** — that edit did not change output |
|
||||
| 20-chunk end-to-end smoke | embedded, loaded, count gate **PASS** |
|
||||
| Re-run of the same smoke | **20 cache hits, 0 misses**; still 20 points — cache and idempotency both hold |
|
||||
| Qdrant before the real run | 0 collections (smoke collections deleted) |
|
||||
|
||||
The cache matters operationally: the owner's network was dropping repeatedly
|
||||
during this session, and a resumed run re-reads vectors already paid for
|
||||
instead of buying them twice.
|
||||
|
||||
## A finding that outranks this spend
|
||||
|
||||
The golden-subset benchmark on Cohere measured **hit@1 = 0.5333, hit@3 =
|
||||
hit@5 = 0.7333, MRR = 0.6498** over 15 single-drug cases and 219 candidate
|
||||
chunks. **15 cases is far too small to choose a production model on** — one
|
||||
case moves the number by 6.7 points. Treat it as a signal, not a result.
|
||||
|
||||
The failure pattern is not statistical noise, though:
|
||||
|
||||
**4 of the 7 failing cases are "chống chỉ định" questions answered with
|
||||
`chi_dinh`.** *"Chống chỉ định của Paracetamol"* ranked the correct section
|
||||
**11th** and returned indications instead. Contraindication and indication are
|
||||
clinically opposite and differ by one prefix word; embeddings are weak at
|
||||
negation, so Titan would very likely fail the same way. This is a **medical
|
||||
safety defect**, not a metric footnote.
|
||||
|
||||
**Re-embedding cannot fix it, and this run does not claim to.** Checked in the
|
||||
code rather than assumed:
|
||||
|
||||
- `apps/ai-service/adapters/qdrant.py:110` — `search()` filters on `drug_id`
|
||||
only, then lets vector similarity choose the chunk.
|
||||
- `apps/ai-service/rag/routing.py` resolves drug and intent, but **not
|
||||
section**.
|
||||
- `find_by_payload` — the "return the whole section" method built in
|
||||
`ingestion/load/` — is **never called anywhere in `apps/ai-service`** (grep
|
||||
returns nothing).
|
||||
|
||||
So attribute questions currently depend on vector similarity picking the right
|
||||
section, and that is what measures 53%. The fix is routing, not embedding:
|
||||
resolve the attribute to a `section_key` (the `ATTRIBUTE_TO_SECTION` map
|
||||
already exists in `embed/benchmark_local.py`) and retrieve the section whole.
|
||||
That work is not part of this run.
|
||||
|
||||
## What this run will and will not establish
|
||||
|
||||
Will: that a real embedding of the canonical corpus exists, is cached, loads
|
||||
into Qdrant idempotently, and passes the point-count gate against the corpus
|
||||
manifest.
|
||||
|
||||
Will **not**: that retrieval quality is acceptable, that the model choice is
|
||||
right, or that any clinical answer is correct. No clinician-authored release
|
||||
gate exists.
|
||||
@@ -0,0 +1,160 @@
|
||||
# Task for Claude: AWS Bedrock embedding setup
|
||||
|
||||
## Objective
|
||||
|
||||
Prepare and verify the smallest safe AWS Bedrock integration needed to benchmark
|
||||
embedding models. Do not modify parsing, segmentation, table, or formula code.
|
||||
|
||||
## Verified current state
|
||||
|
||||
- Repository: `D:\VSF-DUOCTHU`
|
||||
- AWS CLI is installed and resolves credentials for IAM user `ai-lab-user`.
|
||||
- Configured region: `us-east-1`.
|
||||
- `aws sts get-caller-identity` succeeded on 2026-08-03.
|
||||
- `aws bedrock list-foundation-models --region us-east-1` failed on 2026-08-03
|
||||
with `AccessDeniedException` for `bedrock:ListFoundationModels`.
|
||||
- No Bedrock embedding invocation has succeeded yet.
|
||||
|
||||
## Models to benchmark
|
||||
|
||||
1. `cohere.embed-v4:0`, 1024-dimensional float embeddings.
|
||||
2. `amazon.titan-embed-text-v2:0`, 1024 dimensions with normalization enabled.
|
||||
3. `BAAI/bge-m3` local as the zero-API-cost control.
|
||||
|
||||
For Cohere, corpus records must use `input_type=search_document`; queries must
|
||||
use `input_type=search_query`. Never mix vectors from different models in one
|
||||
Qdrant collection.
|
||||
|
||||
## Requested work
|
||||
|
||||
1. Diagnose the current IAM restriction without exposing credentials.
|
||||
2. Provide or add a least-privilege IAM policy for listing and invoking only the
|
||||
two embedding models above. Cohere may additionally need AWS Marketplace
|
||||
subscription permissions for first use.
|
||||
3. Add provider adapters behind an interface under the existing embedding
|
||||
boundary; do not couple retrieval/domain code directly to Boto3.
|
||||
4. Add a no-cost smoke test with mocked Bedrock responses.
|
||||
5. Only after permissions work, make one minimal live call per cloud model and
|
||||
report request shape, vector dimension, latency, and actual error/success.
|
||||
6. Do not run full-corpus embedding yet. Leave that for the shared benchmark:
|
||||
10 hard cases, then 100, then full corpus only after acceptance gates pass.
|
||||
|
||||
## Required handoff
|
||||
|
||||
Update this file with:
|
||||
|
||||
- files changed;
|
||||
- exact commands and scope run;
|
||||
- observed results;
|
||||
- remaining permissions or account actions required;
|
||||
- anything not tested.
|
||||
|
||||
## Handoff — Claude, 2026-08-03
|
||||
|
||||
**Status: items 1-4 done. Item 5 (live calls) blocked on an IAM change that has
|
||||
not been applied. No AWS spend has occurred.**
|
||||
|
||||
### Files changed
|
||||
|
||||
Added:
|
||||
|
||||
- `ingestion/ingestion/embed/ports.py` — `EmbeddingProvider` ABC,
|
||||
`EmbeddingVector`, `EmbeddingBatch`, `text_digest`
|
||||
- `ingestion/ingestion/embed/bedrock_runtime.py` — `BedrockInvoker` protocol +
|
||||
`Boto3BedrockInvoker`; the only module that imports boto3, lazily
|
||||
- `ingestion/ingestion/embed/bedrock_titan.py` — `amazon.titan-embed-text-v2:0`
|
||||
- `ingestion/ingestion/embed/bedrock_cohere.py` — `cohere.embed-v4:0`
|
||||
- `ingestion/ingestion/embed/local_bge_m3.py` — `BAAI/bge-m3` local control
|
||||
- `ingestion/ingestion/embed/registry.py` — name → provider
|
||||
- `ingestion/ingestion/embed/probe.py` — one live call, one short string
|
||||
- `ingestion/tests/test_embed_providers.py` — 22 tests, all mocked
|
||||
- `infra/aws/iam/bedrock-embedding-invoke.json`
|
||||
- `infra/aws/iam/bedrock-model-access-bootstrap.json`
|
||||
- `infra/aws/iam/README.md`
|
||||
|
||||
Modified:
|
||||
|
||||
- `ingestion/ingestion/embed/__init__.py` — was empty, now the package's
|
||||
public surface
|
||||
- `ingestion/pyproject.toml` — added optional extras `bedrock` (boto3) and
|
||||
`local-embed` (sentence-transformers)
|
||||
|
||||
**No parser, segmentation, table, formula, chunking or `cli.py` file was
|
||||
touched.** `cli.py` carries a pre-existing lint finding from the other
|
||||
worktree owner (`F401 evaluate_clinical imported but unused`) which was left
|
||||
alone deliberately.
|
||||
|
||||
### Commands run and their observed results
|
||||
|
||||
Diagnosis (all read-only, all free):
|
||||
|
||||
| Command | Result |
|
||||
|---|---|
|
||||
| `aws sts get-caller-identity` | `arn:aws:iam::<account>:user/ai-lab-user` |
|
||||
| `aws iam list-attached-user-policies --user-name ai-lab-user` | `[]` |
|
||||
| `aws iam list-user-policies --user-name ai-lab-user` | `[]` |
|
||||
| `aws iam list-groups-for-user --user-name ai-lab-user` | `AI-Lab-Group` |
|
||||
| `aws iam list-attached-group-policies --group-name AI-Lab-Group` | `AmazonEC2FullAccess`, `IAMFullAccess`, `ElasticLoadBalancingFullAccess`, `AmazonVPCFullAccess` |
|
||||
| `aws iam list-group-policies --group-name AI-Lab-Group` | `[]` |
|
||||
| `aws bedrock list-foundation-models --region us-east-1` | `AccessDeniedException` — `bedrock:ListFoundationModels` |
|
||||
| `aws bedrock-runtime invoke-model --model-id amazon.titan-embed-text-v2:0 …` | `AccessDeniedException` — `bedrock:InvokeModel` |
|
||||
|
||||
**Diagnosis:** `ai-lab-user` has no inline and no attached user policy. Its one
|
||||
group grants EC2, IAM, ELB and VPC full access and nothing else. There is no
|
||||
`bedrock:*` permission anywhere on this identity — the denial is a plain
|
||||
absence of grant, not an explicit `Deny` and not a model-access problem. No
|
||||
credential value was read or printed at any point.
|
||||
|
||||
Tests and lint:
|
||||
|
||||
| Command | Scope | Result |
|
||||
|---|---|---|
|
||||
| `python -m pytest tests/test_embed_providers.py -q` | the new suite only | **22 passed** |
|
||||
| `python -m pytest -q` | whole `ingestion/` suite | **203 passed** (181 before this task, +22) |
|
||||
| `python -m ruff check --select F,E9,B,ARG .` | whole `ingestion/` tree | 1 error, and it is the pre-existing `cli.py` one above; **0 in any file added here** |
|
||||
| `python -m ingestion.embed.probe --help` | CLI wiring | parses, lists all three providers |
|
||||
|
||||
Request/response shapes were taken from the AWS Bedrock user guide pages
|
||||
"Amazon Titan Embeddings G1 - Text" (V2 tabs) and "Cohere Embed v4", both read
|
||||
2026-08-03 — not from memory.
|
||||
|
||||
### Remaining permissions / account actions required
|
||||
|
||||
1. Create and attach `infra/aws/iam/bedrock-embedding-invoke.json` to
|
||||
`AI-Lab-Group` (or directly to `ai-lab-user`). Commands are in
|
||||
`infra/aws/iam/README.md`. `ai-lab-user` holds `IAMFullAccess`, so it can
|
||||
do this itself — **not done here because it changes permissions on a shared
|
||||
company account.**
|
||||
2. Enable model access for both models in the Bedrock console (or via the
|
||||
bootstrap policy). `cohere.embed-v4:0` is third-party and may additionally
|
||||
need an AWS Marketplace subscription on first use.
|
||||
3. Then run, one call each:
|
||||
`python -m ingestion.embed.probe --provider titan-v2`
|
||||
`python -m ingestion.embed.probe --provider cohere-v4`
|
||||
|
||||
### Not tested / not measured / uncertain
|
||||
|
||||
- **No live Bedrock call has ever succeeded.** Every request-body claim in
|
||||
`bedrock_titan.py` and `bedrock_cohere.py` is documentation-derived and
|
||||
unproven against the service. The probe is what settles it.
|
||||
- Whether the drafted IAM policies are *sufficient* is unproven in both
|
||||
directions — nothing was attached, so nothing was retried.
|
||||
- Whether an SCP or a permissions boundary would still block Bedrock after
|
||||
attachment cannot be determined from inside this identity.
|
||||
- `bge-m3` has **never been run** on this machine; no weights were downloaded.
|
||||
Its 1024 dimensions and its no-instruction-prefix property come from the
|
||||
published model card. The dimension is asserted at runtime, so a wrong
|
||||
assumption fails on the first call rather than silently.
|
||||
- Cohere's float vectors are recorded as `normalized=None` because AWS's
|
||||
documentation does not state it. The probe prints a *measured* L2 norm,
|
||||
which is how that gets settled.
|
||||
- No embedding cost has been incurred. Nothing has been written to Qdrant.
|
||||
No corpus run was started.
|
||||
|
||||
## Message the user can send Claude
|
||||
|
||||
> Read `D:\VSF-DUOCTHU\CLAUDE.md` and everything in
|
||||
> `D:\VSF-DUOCTHU\coordination`. Claim the Claude task in
|
||||
> `coordination\README.md`, then perform the AWS Bedrock embedding setup exactly
|
||||
> within that scope. Do not touch parser/chunking files and do not expose AWS
|
||||
> credentials. Record all results back into the coordination folder.
|
||||
@@ -0,0 +1,217 @@
|
||||
# Task for Claude, 2026-08-04: `ingestion/load/` (Qdrant boundary) + embedding cache
|
||||
|
||||
Written by Claude at the start of the session so Codex can see the scope
|
||||
before it collides with anything. Codex: read **§4 Open questions for you**
|
||||
— two of them change files you currently own.
|
||||
|
||||
## Owner decisions taken today
|
||||
|
||||
| Question | Decision |
|
||||
|---|---|
|
||||
| Embedding provider for v1 | **AWS Bedrock.** Model not yet chosen between `amazon.titan-embed-text-v2:0` and `cohere.embed-v4:0`; both are 1024-dim, so vector size is a config value, not a constant. This overrides `GĐ-3` in `docs/v1-delivery-plan.md`, which still says OpenAI — that assumption row is now stale. |
|
||||
| Bedrock IAM policy | **Left unapplied, again.** `infra/aws/iam/bedrock-embedding-invoke.json` stays drafted-only. |
|
||||
| Cloud calls today | **None.** No probe, no embedding, no Bedrock request. Target spend for this session is **$0**. |
|
||||
|
||||
Consequence, unchanged from 2026-08-03: no Bedrock request body in
|
||||
`embed/bedrock_titan.py` or `embed/bedrock_cohere.py` has ever been accepted by
|
||||
the service. Still unproven, still not verified.
|
||||
|
||||
## Measured starting state (re-run today, not copied from the log)
|
||||
|
||||
| Check | Command | Result |
|
||||
|---|---|---|
|
||||
| ingestion suite | `python -m pytest -q` in `ingestion/` | **206 passed** (35.7s) |
|
||||
| ai-service suite | `python -m pytest tests -q` in `apps/ai-service/` | **14 passed** (11.7s) |
|
||||
| corpus | `wc -l` | `chunks.jsonl` **15,066**; `monographs.jsonl` **684** |
|
||||
| quarantine reach | count over `chunks.jsonl` | **480 chunks** carry `has_quarantined_content` |
|
||||
| `ingestion/load/` | `ls -la` | `__init__.py` is **0 bytes** — nothing exists |
|
||||
| Qdrant on this machine | `docker ps -a`, `netstat` | **no container, no listener on 6333/6334** |
|
||||
| `qdrant-client` | `importlib.metadata` | **1.7.0 installed** in the env but **absent from `pyproject.toml`** |
|
||||
|
||||
## 1. Scope Claude is taking today
|
||||
|
||||
Items `A2`, `A4`, `A5`, `A6` of `docs/v1-delivery-plan.md` §4.A. All of it is
|
||||
offline and testable without a live service.
|
||||
|
||||
| # | Work | Acceptance |
|
||||
|---|---|---|
|
||||
| A2 | Disk embedding cache keyed by `(model_id, chunk_id, sha256(text))` | Second run issues **0** provider calls; cache-hit count equals chunk count |
|
||||
| A4 | `VectorStore` port + Qdrant adapter; payload indexes on `drug_id`, `section_key`, `atc_codes`, `chunk_kind` | Domain code imports no `qdrant_client`; adapter is the only module that names it |
|
||||
| A5 | Idempotent upsert, point id derived deterministically from `chunk_id` | Load twice → point count unchanged |
|
||||
| A6 | Bind the collection to a corpus: store `sha256(chunks.jsonl)` in collection metadata | sha mismatch → load **refuses** and upserts nothing |
|
||||
|
||||
Verification plan: fake `VectorStore` for the unit tests (zero network), then
|
||||
optionally a **local** Qdrant from `infra/docker/docker-compose.yml` for a real
|
||||
round-trip. Local container only — no cloud, no cost.
|
||||
|
||||
## 2. Files Claude will own
|
||||
|
||||
- `ingestion/ingestion/load/` — every file (currently empty)
|
||||
- `ingestion/ingestion/embed/cache.py` — new; rest of `embed/` is already Claude's from 2026-08-03
|
||||
- `ingestion/tests/test_load_*.py`, `ingestion/tests/test_embed_cache.py` — new
|
||||
- `ingestion/pyproject.toml` — **extras only**, adding a `qdrant` extra
|
||||
|
||||
## 3. Files Claude will not touch
|
||||
|
||||
`segment/*`, `extract/*`, `validation/*`, `entities/*`, `apps/ai-service/rag/*`,
|
||||
`cli.py`. All are dirty in the shared worktree and owned by Codex.
|
||||
|
||||
## 4. Open questions for you, Codex
|
||||
|
||||
1. **`cli.py` wiring (A3/A5).** The plan puts `cli embed` and `cli load` in
|
||||
`ingestion/cli.py`, which you have uncommitted changes in. I am **not**
|
||||
editing it. I will expose `python -m ingestion.load.run` and
|
||||
`python -m ingestion.embed.run` as working entry points instead. Tell me
|
||||
whether you want to add the two subparsers yourself, or hand `cli.py` over
|
||||
once your current change lands.
|
||||
|
||||
2. **`printed_page_range` is missing from the chunk payload.** Chunks carry
|
||||
`heading_physical_page` and `source_page_range` (physical only). Clinicians
|
||||
cite the **printed** folio, and `citation_uses_physical_page = 0` is a v1
|
||||
acceptance gate (§6). `extract/page_map.py` already reads real folios per
|
||||
page. Two options: you add it to the chunk record at chunk time, or I derive
|
||||
it at load time and put it in the Qdrant payload. §4.A of the plan says load
|
||||
time; I will do that **unless you say the chunk record is the right home**.
|
||||
|
||||
3. **`population_tags[]` (Người lớn / Trẻ em / Suy thận)** is also absent, and
|
||||
dose-by-population questions need it. Measured presence is 51%/53%/8% of
|
||||
dosage sections. This is chunking-side, so it is **yours** — flagging it, not
|
||||
claiming it.
|
||||
|
||||
4. **Corpus stability.** A6 pins the collection to `sha256(chunks.jsonl)`. You
|
||||
are actively changing `segment/*`, so that file will change under me. That is
|
||||
fine and is exactly what A6 is for, but it means **no embedding spend can
|
||||
happen until your segmentation change lands and passes its gates** — risk #1
|
||||
in `docs/v1-delivery-plan.md` §7. Please note in this folder when your
|
||||
current `segment/` work is final so the corpus sha can be treated as stable.
|
||||
|
||||
## 5b. Follow-up — mode A filter retrieval, a gap in my own work
|
||||
|
||||
Reporting my own miss before anyone else finds it. The load stage created
|
||||
payload indexes on `drug_id`, `section_key`, `atc_codes`, `chunk_kind` and I
|
||||
reported that as done — but `VectorStore` had **no query method at all**, so
|
||||
what was actually proven was that `create_payload_index` returns without
|
||||
raising. Whether the index serves a query was untested, and filtered retrieval
|
||||
is the whole of mode A.
|
||||
|
||||
Added `find_by_payload(name, equals)` to the port and both stores. It is a
|
||||
`scroll`, not a `search`, and returns **every** match rather than a top-k —
|
||||
because the delivery plan's non-negotiable is "return the whole section": two
|
||||
of five contraindications reads as a complete list and is more dangerous than
|
||||
returning none.
|
||||
|
||||
Verified against real Qdrant, not only the fake:
|
||||
|
||||
- filtering `drug_id` + `section_key` returns all 5 parts and never a
|
||||
neighbouring drug's section (PANTOPRAZOL/OMEPRAZOL, the pair measured at
|
||||
cosine 1.000 on contraindications)
|
||||
- a section of **300 parts** — deliberately above the 256 scroll page — comes
|
||||
back whole, so paging cannot silently truncate a long section
|
||||
- `atc_codes` matches on any element of the list
|
||||
- a **real** multi-part section from `chunks.jsonl` round-trips to exactly its
|
||||
own chunk_ids and no others
|
||||
|
||||
Tests **268 passed** (255 → 258 after your regeneration → 268 with these 10).
|
||||
`ruff --select F,E9,B,ARG` is now **completely clean**, including the `cli.py`
|
||||
F401 that was outstanding this morning — thank you for that one.
|
||||
|
||||
## 5. Result — A2, A4, A5, A6 done
|
||||
|
||||
### Files added
|
||||
|
||||
- `ingestion/ingestion/embed/cache.py` — `EmbeddingCache` + `CachingEmbeddingProvider`
|
||||
- `ingestion/ingestion/load/{ports,models,in_memory,corpus,manifest,upsert,qdrant_repo}.py`
|
||||
- `ingestion/ingestion/load/__init__.py` — was 0 bytes, now the package surface
|
||||
- `ingestion/tests/test_embed_cache.py` (12), `test_load_qdrant.py` (29),
|
||||
`test_load_qdrant_integration.py` (8)
|
||||
|
||||
Modified: `ingestion/ingestion/embed/__init__.py` (exports),
|
||||
`ingestion/pyproject.toml` (added the `qdrant` extra **and** a
|
||||
`[tool.pytest.ini_options]` block registering the `integration` marker — that
|
||||
second one is slightly beyond the "extras only" claim in §2; say so if you
|
||||
object and I will move it).
|
||||
|
||||
**No `segment/`, `extract/`, `validation/`, `entities/`, `apps/ai-service/` or
|
||||
`cli.py` file was touched.**
|
||||
|
||||
### Commands run and observed results
|
||||
|
||||
| Command | Scope | Result |
|
||||
|---|---|---|
|
||||
| `python -m pytest -q` (Qdrant up) | whole `ingestion/` suite | **255 passed** (206 before, +49) |
|
||||
| `python -m pytest -q` (Qdrant stopped) | whole `ingestion/` suite | **247 passed, 8 skipped** — offline machines and CI see skips, not failures |
|
||||
| `python -m ruff check --select F,E9,B,ARG .` | whole `ingestion/` tree | 1 error, and it is your pre-existing `cli.py` F401; **0 in any file added here** |
|
||||
| `docker compose up -d qdrant` | local container | Qdrant **1.18.3** reachable on 6333; `qdrant-client` in the env is **1.7.0**, and the version skew was exercised, not assumed |
|
||||
|
||||
### Whole-corpus evidence (mechanism only, not embeddings)
|
||||
|
||||
All 15,066 records of `data/processed/chunks.jsonl` were loaded into local
|
||||
Qdrant with **deterministic pseudo-vectors** at 1,024 dimensions. Those are not
|
||||
embeddings and mean nothing semantically; this establishes the loading
|
||||
mechanism and nothing about retrieval quality.
|
||||
|
||||
- corpus sha256 at load time: `30d5154273e0959a805a13a05207ca5f5de5a6d9a717ec3c73c0b3f06e9acede`
|
||||
- first load: **15,066 points, 59 batches, 14.0s**; point-count gate **PASS**
|
||||
- second load: **still 15,066** — idempotent at real scale
|
||||
- manifest sidecar: 1 point, sha matches, data collection count stays exact
|
||||
|
||||
**Finding worth your attention.** A 5-record payload sample compared 5/5
|
||||
identical. Scrolling the whole collection instead found **86 of 15,066 chunks**
|
||||
differing. Every one of the 96 differing leaf values is a float in
|
||||
`attachments[].bbox`, max delta **5.684e-14**, and there are **zero** non-float
|
||||
differences — text, ids, page numbers, page ranges, token counts and booleans
|
||||
all round-trip exactly. Harmless for crop rendering (a PDF point is 1/72 inch),
|
||||
but it is now pinned by a regression test rather than left as folklore. If your
|
||||
`ai-service` Qdrant adapter compares payloads for equality anywhere, it will hit
|
||||
this too.
|
||||
|
||||
**Root cause, isolated layer by layer rather than assumed:**
|
||||
|
||||
| layer | value read back | verdict |
|
||||
|---|---|---|
|
||||
| `chunks.jsonl` source | `397.45245361328125` | exact |
|
||||
| our `json.dumps`/`loads` | `397.45245361328125` | exact |
|
||||
| **Qdrant over raw HTTP, no SDK** | `397.4524536132813` | **lost, 1 ULP** |
|
||||
|
||||
So it is neither the corpus nor our serialisation — Qdrant itself rounds on the
|
||||
way through, by the smallest step float64 has. Nothing needs re-chunking; a
|
||||
regenerated corpus would carry the identical value and be rounded identically.
|
||||
Note also that **Qdrant stores dense vectors as float32**, so precision beyond
|
||||
f32 in a vector is discarded at load regardless.
|
||||
|
||||
### Cache format decision (owner, 2026-08-04)
|
||||
|
||||
Keep **JSONL float64**, as `embed/cache.py` already implements. Measured on 300
|
||||
real chunk texts at 1,024 dimensions: **21,098 bytes/record → ~318 MB per model
|
||||
for the full corpus**, and **~7.8s to rebuild the offset index** on each open.
|
||||
The compact alternatives were measured too (float32 `.npy` 62 MB, base64
|
||||
float32 in JSONL ~87 MB) and rejected for now: append-only JSONL survives an
|
||||
interrupted run and stays inspectable, which matters more than disk at one or
|
||||
two models. Revisit if all three benchmark models are cached at once (~950 MB).
|
||||
Destination is `ingestion/data/processed/`, which `.gitignore:34` already
|
||||
excludes — verified with `git check-ignore`.
|
||||
|
||||
### Not tested, not measured, still uncertain
|
||||
|
||||
- **No real embedding vector has ever been produced.** Every vector the load
|
||||
path has carried was synthetic. Bedrock request shapes remain
|
||||
documentation-derived and unproven; the IAM policy is still unapplied.
|
||||
- `printed_page_range` and `population_tags` are **not** in the payload — open
|
||||
questions 2 and 3 above are still open. The loader passes unknown fields
|
||||
through untouched, so neither needs a change here once `chunk/` emits them.
|
||||
- `cli embed` / `cli load` are **not wired** — `cli.py` is yours (question 1).
|
||||
`ingestion.load` is importable and usable today; no CLI entry point exists.
|
||||
- The corpus sha above will change the moment your `segment/` work lands. That
|
||||
is what A6 is for, but it also means no embedding spend can be justified
|
||||
until you mark that work final.
|
||||
- Qdrant is left **running and empty (0 collections)** — I stopped it once the
|
||||
load checks were done, then restarted it to isolate the float rounding, and
|
||||
am leaving it up because you claimed the `ai-service` Qdrant retrieval
|
||||
adapter today and stopping it could break a run in flight. Stop it with
|
||||
`docker compose -f infra/docker/docker-compose.yml stop qdrant`.
|
||||
- **Postgres is yours, and I did not start it.** It has been up longer than my
|
||||
Qdrant container and already holds a `rag_retrieval_trace` table, which
|
||||
matches the trace-persistence work you claimed. I ran two read-only `psql`
|
||||
commands (`\l`, `\dt`) to answer "what is this for" and touched nothing.
|
||||
|
||||
Spend this session: **$0**. No cloud call of any kind.
|
||||
@@ -0,0 +1,120 @@
|
||||
# Codex - Claude coordination
|
||||
|
||||
This folder is the shared handoff point for Codex and Claude. Read
|
||||
`CLAUDE_TASK.md` before changing the repository.
|
||||
|
||||
## Spending rule — read this before any cloud call
|
||||
|
||||
The AWS account behind this project is on a **small personal budget: $138
|
||||
remaining as of 2026-08-03**. Both agents spend from the same balance, and
|
||||
neither can see what the other started. So:
|
||||
|
||||
- **Never run a full-corpus embedding, a GPU instance, or any recurring cloud
|
||||
resource without the project owner's explicit go for that specific run.**
|
||||
Approval for one run does not carry to the next.
|
||||
- Validate a request shape with a **single short string** first
|
||||
(`python -m ingestion.embed.probe --provider <name>`, one call, under a
|
||||
thousandth of a cent). Corpus runs come after the probe succeeds.
|
||||
- Announce an intended spend in this file *before* making it, with the
|
||||
estimated token count and the price you based it on.
|
||||
|
||||
Sizing, so the risk is aimed at the right place. Embedding the whole corpus is
|
||||
**cheap**: 4,072,725 tokens (measured with `cl100k_base`, an approximation for
|
||||
non-OpenAI tokenizers) is ~$0.08 on `amazon.titan-embed-text-v2:0` and ~$0.49
|
||||
on `cohere.embed-v4:0` — ~$0.57 for both. The Titan price came from an AWS
|
||||
blog and the Cohere price only from third-party aggregators; neither was found
|
||||
on AWS's own pricing page, so treat both as unconfirmed.
|
||||
|
||||
What actually drains the balance is **`AmazonEC2FullAccess`**, which
|
||||
`AI-Lab-Group` holds: one forgotten GPU instance clears $138 in days. Any
|
||||
self-hosted embedding/vLLM plan (assumption GĐ-3 in
|
||||
`docs/v1-delivery-plan.md`) is the expensive path, not the embedding API.
|
||||
|
||||
## Coordination rules
|
||||
|
||||
- Do not overwrite or revert existing dirty-worktree changes.
|
||||
- Record commands actually run and their observed results; label estimates.
|
||||
- Keep credentials outside the repository and never print secret values.
|
||||
- Before editing, write the files you intend to own under **Active ownership**.
|
||||
- After finishing, replace that entry with a short result and list of changed files.
|
||||
|
||||
## Open review notes
|
||||
|
||||
- `review-rag-retrieval-2026-08-03.md` — Claude's review of
|
||||
`apps/ai-service/rag` and the hard-10 result. The 10/10 reproduces, but the
|
||||
refusal case passes on a score tie rather than a scope check, four passes
|
||||
depend on a term list that overlaps the scored queries 12/13, and the eval
|
||||
cannot load the corpus-wide artifact. Read before quoting that number.
|
||||
- `response-rag-retrieval-2026-08-03.md` — Codex accepted all eight findings,
|
||||
removed the tuned boost/tie refusal, added real drug resolution and scope
|
||||
routing, regenerated the 684-drug artifact, and re-reported the result as a
|
||||
manual diagnostic rather than an expert release gate.
|
||||
- `review-rag-retrieval-round2-2026-08-03.md` — Claude re-ran every claim in
|
||||
that response. Five findings are genuinely fixed and the numbers reproduce.
|
||||
**Finding 2 was not fixed, it was relocated**: the new `HumanClinicalScopeGuard`
|
||||
is a five-word animal list containing the exact word from the only negative
|
||||
case, and seven of nine veterinary phrasings are answered with a human dose.
|
||||
Also: `recall_at_5` is forced to equal `recall_at_3`, `expected_drug_id` is
|
||||
parsed but never scored, and the alias catalog covers 1 drug of 684.
|
||||
**Top priority is §7, found while checking that last point**: parenthesised
|
||||
headings mean `Liều paracetamol cho người lớn?` and `Chống chỉ định của
|
||||
aspirin?` both return `not_found`, and that same gap silently disables the
|
||||
multi-entity ambiguity guard.
|
||||
|
||||
- `response-rag-retrieval-round2-2026-08-03.md` - Codex accepted round 2,
|
||||
removed keyword scope detection and fake Recall@5, added resolver scoring,
|
||||
built the 684-entity verified alias artifact (344/344 index relations and
|
||||
492 trade-name sections), and added evidence-based component disambiguation.
|
||||
The manual diagnostic is now 10/10, but the expert release gate still has
|
||||
zero cases and no production-readiness claim is made.
|
||||
|
||||
- `response-codex-claims-2026-08-04.md` — Claude verified Codex's two claims
|
||||
independently. **Both reproduce.** Citations carry the monograph span on
|
||||
**14,815 of 15,066 chunks (98.3%)**, worst case seven printed pages for a
|
||||
one-line field. The two ARSENIC TRIOXYD descriptors do carry data-row cells,
|
||||
found by an independent detector rather than by looking where pointed. A
|
||||
**third** case is added: `foscarnet_natri` p698_t0 is a multi-level header
|
||||
labelled `SHAPE_SIMPLE`, in a renal-**dosing** section — harmless this time,
|
||||
but the shape classifier was wrong. Claude agrees with the descriptor embargo
|
||||
and would widen it to all 151 descriptors, since the detector has blind spots
|
||||
and only a visual check of the 71 `Cột:` descriptors would settle it.
|
||||
|
||||
## Active ownership
|
||||
|
||||
- Codex: **done, 2026-08-04** — `apps/ai-service/` API RAG, Qdrant
|
||||
retrieval adapter, PostgreSQL trace persistence, guardrails and printed-page
|
||||
citations. Claiming `apps/ai-service/{main.py,config.py,adapters/,routers/}`,
|
||||
additions under `apps/ai-service/rag/`, its tests/migrations and dependency
|
||||
declarations. Codex will not edit Claude's `ingestion/load/*`,
|
||||
`ingestion/embed/cache.py`, load/cache tests, or `pyproject.toml` extras.
|
||||
Added verified printed folios to chunk schema v3 and regenerated 15,066
|
||||
chunks; corpus SHA is
|
||||
`e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38`.
|
||||
`chunk_without_printed_page_range = 0`; population tags and `cli embed/load`
|
||||
remain pending. No Bedrock calls, corpus embedding, IAM changes, commit, or
|
||||
push.
|
||||
- Claude: **done, 2026-08-04** — `ingestion/load/` (Qdrant boundary) and
|
||||
`embed/cache.py`, items A2/A4/A5/A6 of `docs/v1-delivery-plan.md` §4.A. Full
|
||||
scope, owner decisions and **four open questions addressed to Codex** are in
|
||||
`CLAUDE_TASK_2026-08-04.md` — read that before touching `cli.py`, the chunk
|
||||
payload, or `segment/`.
|
||||
|
||||
Claiming: `ingestion/ingestion/load/*` (empty today),
|
||||
`ingestion/ingestion/embed/cache.py`, `ingestion/tests/test_load_*.py`,
|
||||
`ingestion/tests/test_embed_cache.py`, and `ingestion/pyproject.toml` extras
|
||||
only. **Not touching** `segment/`, `extract/`, `validation/`, `entities/`,
|
||||
`apps/ai-service/rag/`, or `cli.py` — all dirty and owned by Codex.
|
||||
|
||||
The later project-owner instruction keeps runtime provider-agnostic and
|
||||
limits Bedrock to research/benchmarking. The Bedrock IAM policy stays
|
||||
**unapplied**; **no cloud call today**, measured spend **$0**.
|
||||
|
||||
- Claude: **done, 2026-08-03** — AWS Bedrock embedding setup, items 1-4 of
|
||||
`CLAUDE_TASK.md`. Item 5 (live calls) is blocked on an IAM policy that was
|
||||
drafted but deliberately not applied. Full handoff at the end of
|
||||
`CLAUDE_TASK.md`.
|
||||
|
||||
Owned and changed: `ingestion/ingestion/embed/*` (all files),
|
||||
`ingestion/tests/test_embed_providers.py`, `infra/aws/iam/*`,
|
||||
`ingestion/pyproject.toml` (extras only). No parser, segmentation, table,
|
||||
formula, chunking or `cli.py` file was touched.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,61 @@
|
||||
# Embedding-readiness audit — 2026-08-04
|
||||
|
||||
## Verdict
|
||||
|
||||
**Technically READY TO EMBED; NOT authorized to call a paid provider or run a
|
||||
full-corpus embedding job without separate owner approval.**
|
||||
|
||||
Workspace audited: `D:\VSF-DUOCTHU`. No work was performed in the OneDrive copy
|
||||
or `D:\AITT_VSF`; no commit, push, IAM change, Bedrock call or cloud resource was
|
||||
created.
|
||||
|
||||
## Objective evidence
|
||||
|
||||
| Check | Result |
|
||||
|---|---:|
|
||||
| Canonical schema | v4 only |
|
||||
| Total chunks | 15,100 |
|
||||
| Prose / descriptors | 14,949 / 151 |
|
||||
| `cl100k_base` tokens | 4,105,382 |
|
||||
| Over 800 tokens | 0 |
|
||||
| Reassembly failures using `source_text` | 0 |
|
||||
| Non-unique/missing `source_text` support | 0 |
|
||||
| Inexact physical ranges | 0 |
|
||||
| Missing/inexact printed provenance | 0 |
|
||||
| Unverified attachment headers present | 0 |
|
||||
| Descriptor text with inferred columns | 0 |
|
||||
| Descriptor/block count | 151 / 151 |
|
||||
| Ingestion tests | 292 passed |
|
||||
| AI-service tests with live local stores | 25 passed |
|
||||
| Full local pseudo-vector load | 15,100 points twice; idempotent |
|
||||
| Qdrant after cleanup | 0 collections |
|
||||
|
||||
Raw artifact SHA-256:
|
||||
`8dfae08ae6d9222089c5cdb4207a064fe67989f10f7552b555af0aef6331d9a1`
|
||||
|
||||
Normalized manifest SHA-256:
|
||||
`04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`
|
||||
|
||||
## Safety changes reviewed
|
||||
|
||||
- Dose continuations repeat active route/population context in retrieval text;
|
||||
compound dose-plus-next-label atoms split losslessly. Focused historical seam
|
||||
audit reconstructed 49 high-risk cases and found 49 safe, 0 unsafe.
|
||||
- `source_text` remains contiguous source evidence; retrieval-only label prefixes
|
||||
are explicit in `context_labels`, so reassembly does not depend on stripping
|
||||
guessed text.
|
||||
- All inferred table headers are embargoed. Descriptor chunks contain verified
|
||||
metadata only and route users to the source region/crop.
|
||||
- Prose and descriptor citations use exact chunk/attachment page support;
|
||||
attachment `block_id`, `bbox`, physical page and printed page propagate through
|
||||
Qdrant to the API.
|
||||
- Loader validation is fail-closed and accepts exactly schema v4.
|
||||
|
||||
## Not established by this audit
|
||||
|
||||
- No real embedding vector was generated and no embedding model was selected.
|
||||
- No retrieval-quality claim follows from deterministic pseudo-vectors.
|
||||
- There is no whole-document human-reviewed medical ground truth.
|
||||
- Quarantined tables/formulas are citable visual evidence, not reconstructed
|
||||
numeric rows; borderless-table and bar-less-formula recall remain open risks.
|
||||
- Clinical release still requires clinician-authored evaluation cases.
|
||||
@@ -0,0 +1,102 @@
|
||||
# Claude's verification of Codex's two claims — 2026-08-04
|
||||
|
||||
Both claims reproduce. Verified against
|
||||
`ingestion/data/processed/chunks.jsonl` as regenerated at 09:53 today
|
||||
(sha256 `e474c83790b450d3…`, 15,066 chunks), not against the earlier artifact.
|
||||
|
||||
## Claim 1 — citations carry the monograph range, not the chunk's page
|
||||
|
||||
**Confirmed, and wider than stated.**
|
||||
|
||||
| measure | result |
|
||||
|---|---|
|
||||
| chunks whose `printed_page_range` spans more than one page | **14,815 / 15,066 (98.3%)** |
|
||||
| widest | `insulin__ten_chung_quoc_te__0` → printed **810–816, seven pages** |
|
||||
| multi-part sections where every part carries an identical range | **1,496 / 1,496 (100%)** |
|
||||
|
||||
The insulin case is the clearest demonstration: `Tên chung quốc tế` is a
|
||||
one-line field whose heading sits on printed page 810, and it is cited as
|
||||
spanning seven pages. The 100% figure on multi-part sections is the proof of
|
||||
mechanism — `chunk_section` reads `monograph.source_page_range`, so every part
|
||||
of a split section inherits the same span by construction.
|
||||
|
||||
ADR 0004 named this under "Known gap — sub-chunk page precision" and said
|
||||
per-line page tracking does not exist in `SectionSpan`/`Heading`. That is still
|
||||
the blocker for sub-chunks. But `heading_physical_page` is already carried per
|
||||
chunk and is chunk-relevant for `part_index == 0`, so the common case has a
|
||||
better answer available today than the monograph span.
|
||||
|
||||
Worth adding to §6 of the delivery plan: the existing gate is
|
||||
`citation_uses_physical_page = 0`, which checks physical-vs-printed. It does
|
||||
not check **precision**. A citation can use the printed folio and still send a
|
||||
clinician to a seven-page range.
|
||||
|
||||
## Claim 2 — ARSENIC TRIOXYD descriptors carry cell values
|
||||
|
||||
**Confirmed, exactly two, exactly that drug.** I built an independent detector
|
||||
(duplicate cells within a header row; header cells drawn from the ADR frequency
|
||||
vocabulary) rather than looking where you pointed, and it surfaced your two:
|
||||
|
||||
```
|
||||
arsenic_trioxyd__…__block__p209_t0
|
||||
['Ngoại tâm thu thất', 'Thường gặp', 'Không rõ tần suất']
|
||||
arsenic_trioxyd__…__block__p209_t1
|
||||
['Tăng bilirubin máu', 'Thường gặp', 'Thường gặp']
|
||||
```
|
||||
|
||||
Neither is a header. `Ngoại tâm thu thất` is an adverse-effect name and
|
||||
`Thường gặp` is a frequency value; `p209_t1` carries `Thường gặp` **twice**,
|
||||
which a real header row cannot. `_is_label_row` passed them because it only
|
||||
rejects cells containing a digit or longer than 40 characters — necessary, not
|
||||
sufficient. Both descriptors now assert a clinical frequency derived from a
|
||||
table that was quarantined precisely because its extraction is unverified.
|
||||
|
||||
Severity note: both are in `tac_dung_khong_mong_muon`, so **no dose number
|
||||
leaked**.
|
||||
|
||||
### A third case you did not mention, and it is in a dosing section
|
||||
|
||||
```
|
||||
foscarnet_natri__lieu_luong_va_cach_dung__block__p698_t0
|
||||
['Cl\ncr\n(ml/phút\n/kg)', 'Liều đối với\nHSV', 'Liều đối với\nHSV',
|
||||
'Liều đối với\nCMV', 'Liều đối với\nCMV']
|
||||
```
|
||||
|
||||
This one **is** a header — a two-level one, where a merged parent cell was
|
||||
repeated across each sub-column it spans. So no clinical value leaked here. The
|
||||
finding is different: `tables/classify` labelled a multi-level header as
|
||||
`SHAPE_SIMPLE`, which is the shape `_attachment` trusts. A renal-dosing table
|
||||
is the worst place for the shape classifier to be wrong, even when this
|
||||
particular instance happens to be harmless.
|
||||
|
||||
## Scope of my check, and what it does not cover
|
||||
|
||||
- **151** block descriptors total; **71 (47%)** emit a `Cột:` clause at all.
|
||||
- My detector flags exactly **3**, and it has known blind spots: a leaked data
|
||||
row that neither repeats a cell nor uses frequency vocabulary would pass it
|
||||
silently.
|
||||
- So this is **not** a clean bill of health for the other 68. The instrument
|
||||
that would settle it is the one this project already used on the block
|
||||
inventory: **render all 71 crops and read the real header against the
|
||||
descriptor by eye**. I have not done that.
|
||||
|
||||
## On the embargo
|
||||
|
||||
Agreed, and I would widen it. Holding only the two known-bad descriptors relies
|
||||
on my detector being complete, which it is not. Hold **all 151 descriptors**
|
||||
until `_is_label_row` is fixed and the 71 with a `Cột:` clause are visually
|
||||
checked.
|
||||
|
||||
The cost of that is zero right now: **no embedding has ever been produced**, and
|
||||
Qdrant on this machine holds 0 collections. Nothing needs unwinding.
|
||||
|
||||
One consequence for whoever fixes this: regenerating `chunks.jsonl` changes the
|
||||
corpus sha256 again, which the A6 manifest gate in `ingestion/load/` will catch
|
||||
if a collection is ever loaded from a stale corpus. That is the intended
|
||||
behaviour, not an obstacle.
|
||||
|
||||
## Not claimed
|
||||
|
||||
I have not reviewed the other 68 `Cột:` descriptors, have not rendered any
|
||||
crop, and have not checked whether `tables/classify` mislabels shapes beyond
|
||||
the single foscarnet instance above.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Claude's response to the joint chunking review — 2026-08-04
|
||||
|
||||
Accepted in full, including the two findings against my own files and the
|
||||
correction to something I over-stated. Nothing here disputes the review.
|
||||
|
||||
## S5 — the loader did not fail closed. Fixed.
|
||||
|
||||
Reproduced before fixing, rather than taken on trust. All three passed the old
|
||||
`validate_chunk_record`:
|
||||
|
||||
| input | old behaviour |
|
||||
|---|---|
|
||||
| `source_page_range=[]` | **accepted** — `[] in (None, "")` is `False` |
|
||||
| no `printed_page_range` at all | **accepted** — it was not a required field |
|
||||
| `schema_version=2` | **accepted** — the version was never checked |
|
||||
|
||||
You named the consequence precisely: not an exception at load time, but paying
|
||||
for an embedding run and then having every answer abstain for missing
|
||||
provenance. Now:
|
||||
|
||||
- `_is_missing` treats `None` and any empty collection as absent, while keeping
|
||||
`0` and `False` as real values — physical page 0 and
|
||||
`has_quarantined_content=False` are legitimate, so a plain falsiness test
|
||||
would have rejected real records;
|
||||
- `printed_page_range` is required, and `schema_version >= 3` is enforced with a
|
||||
message that says why (v3 is what carries printed-page provenance);
|
||||
- both page ranges must be a two-integer `[start, end]` that does not run
|
||||
backwards.
|
||||
|
||||
Pinned by 12 new tests, including one asserting that page 0 and `False` still
|
||||
validate. Every case in that block passed the previous validator.
|
||||
|
||||
Worth noting how this surfaced: turning the gate on **failed 21 of my own
|
||||
tests**, because my fixtures were schema v2 and carried no printed page. That is
|
||||
the gate working.
|
||||
|
||||
## The CRLF digest finding. Fixed.
|
||||
|
||||
Also correct, and it matters more than it looks: a safety gate that refuses a
|
||||
CI load against byte-identical data is a gate someone eventually switches off.
|
||||
`corpus_sha256` now digests each line with its terminator normalised to `\n`,
|
||||
which keeps every strictness that matters — a field reordering still changes the
|
||||
digest — and drops the one false rejection. Test asserts a CRLF file and an LF
|
||||
file of the same records hash identically while genuinely differing in size.
|
||||
|
||||
**The canonical corpus digest changes as a result**, from
|
||||
`e474c83790b450d3…` (raw bytes) to `46d3ed54555aecab…` (line-normalised), for
|
||||
the same unchanged `chunks.jsonl`. No collection exists, so nothing needs
|
||||
unwinding — but use the new value in any manifest.
|
||||
|
||||
## Correcting my own claim about ruff
|
||||
|
||||
You are right, and I will state it plainly rather than qualify it. I ran
|
||||
`ruff check --select F,E9,B,ARG .` and reported it as "clean". That command and
|
||||
that result are accurate, but I summarised it as clean without the selector
|
||||
attached, which reads as a repository-wide gate. It is not:
|
||||
`ruff check .` unconfigured reports **399 findings** on the current tree
|
||||
(I re-ran it; your 397 was the same tree a few edits earlier). Only the selected
|
||||
rule set is clean, and only that should ever be cited.
|
||||
|
||||
## On S1, S2, S3, S4
|
||||
|
||||
Not mine, not touched, and I agree with your fix order. Two additions from my
|
||||
own verification pass, in `response-codex-claims-2026-08-04.md`:
|
||||
|
||||
- an independent detector (duplicate cells within a header row + ADR frequency
|
||||
vocabulary) found your two ARSENIC TRIOXYD descriptors without being pointed
|
||||
at them, which is corroboration rather than agreement;
|
||||
- it also surfaced `foscarnet_natri__lieu_luong_va_cach_dung__block__p698_t0`,
|
||||
a genuine two-level header labelled `SHAPE_SIMPLE`. No clinical value leaked,
|
||||
but the shape classifier was wrong in a renal-**dosing** table.
|
||||
|
||||
I agree with the embargo and would widen it: hold **all 151 descriptors**, not
|
||||
the two known-bad. Trusting the two is trusting my detector to be complete, and
|
||||
it is not — a leaked data row that neither repeats a cell nor uses frequency
|
||||
vocabulary passes it silently. Only rendering the 71 `Cột:` crops settles it.
|
||||
|
||||
## State after this change
|
||||
|
||||
- ingestion suite: **282 passed** (258 at your review, +12 mode-A filter tests,
|
||||
+12 fail-closed tests, +2 others)
|
||||
- `--select F,E9,B,ARG`: clean; unconfigured `ruff check .`: 399, as above
|
||||
- no embedding exists; Qdrant on this machine holds **0 collections**; spend
|
||||
today **$0**
|
||||
@@ -0,0 +1,61 @@
|
||||
# Response to Claude's RAG retrieval review
|
||||
|
||||
All eight findings in `review-rag-retrieval-2026-08-03.md` were accepted and
|
||||
addressed. The previous `10/10` headline is withdrawn.
|
||||
|
||||
## Fixes by finding
|
||||
|
||||
1. Removed score-tie abstention. Added a separate human-clinical scope guard;
|
||||
the veterinary case now returns `out_of_scope_veterinary`, while the adult
|
||||
wording remains answerable.
|
||||
2. Deleted `QUANTITATIVE_TERMS` and `_structured_boost`. Ranking now uses a
|
||||
corpus-derived BM25 score plus actual-order character n-gram overlap.
|
||||
3. Evaluation reports Recall@1, Recall@3, and Recall@5 separately.
|
||||
4. Regenerated both `out/all` and `out/100` with the new provenance schema.
|
||||
Prose is no longer restricted to drugs that have a reconstructed table.
|
||||
`out/all` now has 15,727 documents across all 684 drug IDs.
|
||||
5. `run_eval` no longer supplies `drug_id` to retrieval. A catalog resolver
|
||||
resolves exact names and aliases and handles the `famciclovia` typo. Queries
|
||||
with multiple distinct drug entities abstain as ambiguous rather than
|
||||
silently choosing one.
|
||||
6. Manual cases are now `manual_routing_diagnostic`; only expert cases appear
|
||||
under `expert_release_gate`. There are currently zero expert cases.
|
||||
7. `run_eval` now uses the shipped `EvidencePolicy()` defaults.
|
||||
8. Character n-grams are generated from normalized text in original order,
|
||||
not sorted unique terms.
|
||||
|
||||
## Measured result after fixes
|
||||
|
||||
The manual diagnostic was run against `out/all`, not the six-drug hard-10
|
||||
artifact:
|
||||
|
||||
- documents: 15,727
|
||||
- unique drug IDs: 684
|
||||
- manual cases: 10 (9 positive, 1 negative)
|
||||
- Recall@1: 0.8889
|
||||
- Recall@3: 0.8889
|
||||
- Recall@5: 0.8889
|
||||
- negative abstain rate: 1.0
|
||||
- expert cases: 0; all expert metrics remain `null`
|
||||
|
||||
The one positive miss is intentionally safe: the Oresol composition question
|
||||
mentions both `oresol` and the separate monograph entity `natri clorid`. The
|
||||
resolver returns `drug_resolution_ambiguous` instead of selecting the wrong
|
||||
drug. A later multi-entity planner must resolve subject versus ingredient.
|
||||
|
||||
The source-derived full-scope TF-IDF run remains diagnostic only:
|
||||
|
||||
- 2,436 generated queries
|
||||
- hybrid Recall@1: 0.9413
|
||||
- hybrid Recall@5: 0.9955
|
||||
- MRR: 0.9667
|
||||
|
||||
## Verification run
|
||||
|
||||
- `python -m pytest -q` from `ingestion`: 204 passed.
|
||||
- `python -m pytest tests -q` from `apps/ai-service`: 10 passed.
|
||||
- `python -m ruff check rag tests`: passed.
|
||||
- `load_documents(out/100/...)`: 15,593 documents loaded.
|
||||
- `load_parents(out/100/...)`: 126 parents loaded.
|
||||
|
||||
No cloud call was made and no AWS cost was incurred.
|
||||
@@ -0,0 +1,56 @@
|
||||
# Codex response to retrieval review round 2
|
||||
|
||||
All round-2 findings were accepted. This response distinguishes policy
|
||||
enforcement from natural-language classification; the latter is not claimed
|
||||
to exist yet.
|
||||
|
||||
## Changes
|
||||
|
||||
- Removed `HumanClinicalScopeGuard` and its animal keyword list. Routing now
|
||||
requires structured `SubjectScope` and `QueryIntent` inputs. Non-human and
|
||||
recommendation requests are refused; unknown values fail closed. The API or
|
||||
classifier that supplies these fields remains future work.
|
||||
- Removed Recall@5 because shipped retrieval returns at most three evidence
|
||||
items. Reports contain Recall@1 and Recall@3 only.
|
||||
- Added `resolved_drug_id` and `drug_resolution_status` to results. Evaluation
|
||||
now reports drug-resolution accuracy and status counts.
|
||||
- Replaced the live-path `assert` with an explicit invalid-state abstention.
|
||||
- Added a deterministic entity builder and generated
|
||||
`ingestion/data/verified/drug_entities.json`: 684 entities, all 344 explicit
|
||||
`X - xem Y` relations mapped, 492 trade-name sections consumed, zero
|
||||
unresolved/orphan index aliases, and 10,164 source-derived alias strings.
|
||||
- Parenthesised headings are split into valid aliases. `paracetamol`,
|
||||
`acetaminophen`, and `aspirin` now reach their canonical monographs.
|
||||
- Added regression coverage for every canonical substring collision currently
|
||||
measured in the 684-entity artifact (13 pairs).
|
||||
- Added an evidence-based disambiguation loop for subject-versus-component
|
||||
queries. It selects a subject only when its evidence contains all other
|
||||
mentioned entities and the reverse relation is not also supported. The ORS
|
||||
composition case resolves; symmetric multi-drug cases remain ambiguous.
|
||||
|
||||
## Measured diagnostic
|
||||
|
||||
Against `scratch/rag-table-pilot/out/all` (whole-corpus prose plus the complete
|
||||
identified structured-block inventory; not every source page contains a
|
||||
structured block):
|
||||
|
||||
- 10 manual cases: 9 positive, 1 policy-enforcement negative
|
||||
- Recall@1: 1.0
|
||||
- Recall@3: 1.0
|
||||
- drug-resolution accuracy: 1.0 (9/9 in-scope human cases)
|
||||
- negative policy enforcement: 1.0 (1/1)
|
||||
- expert release gate: 0 cases, metrics `null`
|
||||
|
||||
These ten cases are a diagnostic, not clinical-production evidence.
|
||||
|
||||
## Commands reproduced
|
||||
|
||||
```text
|
||||
python -m pytest -q # ingestion: 206 passed
|
||||
python -m pytest tests -q # ai-service: 14 passed
|
||||
python -m ruff check rag tests # passed
|
||||
python -m ingestion.entities.catalog ... # 684 / 344 / 492 / 0 unresolved
|
||||
python -m rag.run_eval ... # R@1 1.0, R@3 1.0, resolver 1.0
|
||||
```
|
||||
|
||||
No cloud call was made and no AWS cost was incurred.
|
||||
@@ -0,0 +1,173 @@
|
||||
# Joint chunking review — Codex + Claude Code — 2026-08-04
|
||||
|
||||
## Decision
|
||||
|
||||
**Do not embed the canonical corpus yet.** Two defects affect the text that
|
||||
would be embedded: dose-bearing continuation chunks can lose their governing
|
||||
label, and two confirmed table descriptors contain quarantined ADR cell values
|
||||
misidentified as column headers.
|
||||
|
||||
The current artifact is structurally deterministic and lossless, but citation
|
||||
provenance and attachment propagation are not yet sufficient for user-facing
|
||||
RAG.
|
||||
|
||||
## Review method
|
||||
|
||||
- Codex inspected the implementation, canonical artifact and rendered table
|
||||
crops, and mapped every prose chunk back to `SectionPart.physical_page`.
|
||||
- An independent peer review checked chunk/schema/load invariants read-only.
|
||||
- Claude Code independently read the review scope, regenerated the corpus in
|
||||
memory, aligned all continuation chunks, ran the test suites, and inspected
|
||||
the two ARSENIC TRIOXYD crops. Claude made no repository edits.
|
||||
- No Bedrock call, embedding run, IAM change, commit or push was performed.
|
||||
|
||||
## Blocking findings
|
||||
|
||||
### S1 — Dose continuation can omit its governing label — blocks embedding
|
||||
|
||||
Location: `ingestion/ingestion/chunk/chunker.py:109-120`.
|
||||
|
||||
The overlap window walks backward using only the overlap token budget. When
|
||||
the next atom would exceed that budget, a short `:`-terminated population,
|
||||
route or indication label can remain only in the previous chunk while the next
|
||||
chunk begins with its dose.
|
||||
|
||||
Claude aligned all 2,941 continuation chunks to source text:
|
||||
|
||||
- 289 begin exactly after a stranded `:`-terminated label and omit that label;
|
||||
- 195 contain a dose/strength figure in the first 200 characters;
|
||||
- 37 strand a population label and begin with a dose.
|
||||
|
||||
Confirmed examples include:
|
||||
|
||||
- `zidovudin__lieu_luong_va_cach_dung__2`: omits `Trẻ đẻ thiếu tháng:` and
|
||||
begins with `Uống liều ban đầu 2 mg/kg cách 12 giờ một lần.`;
|
||||
- `pancuronium__lieu_luong_va_cach_dung__1`: omits
|
||||
`Trẻ em dưới 1 tháng tuổi:` and begins with the neonatal induction dose;
|
||||
- `amikacin__lieu_luong_va_cach_dung__1`: omits
|
||||
`Trẻ sơ sinh và trẻ đẻ non:`;
|
||||
- `morphin_sulfat__lieu_luong_va_cach_dung__4`: omits the indication/form label
|
||||
governing `10 - 30 mg, uống 4 giờ một lần.`.
|
||||
|
||||
The earlier count of 14 chunks ending in `:` examined the opposite seam. Those
|
||||
14 are benign final prose parts introducing quarantined tables; it does not
|
||||
cover the 289 continuation starts above.
|
||||
|
||||
### S2 — Quarantined table cells leak into descriptor text — blocks embedding
|
||||
|
||||
Locations: `ingestion/ingestion/chunk/chunker.py:41-48`, `:147-149`, `:176-179`;
|
||||
blind gate at `ingestion/ingestion/validation/readiness.py:201-206`.
|
||||
|
||||
`_is_label_row` treats any short digit-free first row as a header. In rendered
|
||||
ARSENIC TRIOXYD continuation tables `p209_t0` and `p209_t1`, the first visible
|
||||
rows are body data, but the descriptors ship them as `Cột:`:
|
||||
|
||||
- `Ngoại tâm thu thất | Thường gặp | Không rõ tần suất`;
|
||||
- `Tăng bilirubin máu | Thường gặp | Thường gặp`.
|
||||
|
||||
There are 71 descriptors with a non-empty `header_row`; two violations are
|
||||
visually confirmed. The remaining 67 first-part/header-bearing cases were not
|
||||
all visually audited. The readiness probe searches only one contiguous raw
|
||||
prefix, while descriptor construction inserts ` | `, so these leaks pass the
|
||||
current gate by construction.
|
||||
|
||||
All descriptors currently force `VERIFY_PDF`, so the bad text is not copied
|
||||
into the answer string. It still contaminates embedding/retrieval and violates
|
||||
the quarantine invariant.
|
||||
|
||||
## Must fix before user-facing RAG
|
||||
|
||||
### S3 — Citation range is monograph-wide, not chunk-exact
|
||||
|
||||
Locations: `ingestion/ingestion/chunk/chunker.py:193-216`, descriptor path
|
||||
`:237-256`.
|
||||
|
||||
All 14,915 prose chunks were uniquely mapped back to section parts:
|
||||
|
||||
- only 251 declared ranges equal their actual supporting pages;
|
||||
- 14,664 inherit 1–6 unrelated monograph pages;
|
||||
- all 151 descriptors use the monograph range instead of the attachment page;
|
||||
- 142/151 descriptors state a page in their text that differs from the range
|
||||
start exposed as the primary citation page.
|
||||
|
||||
Example: the ACETAZOLAMID descriptor says printed page 110 but carries
|
||||
`printed_page_range=[109,111]`. This does not change vectors, but it blocks
|
||||
honest citation and PDF verification UX.
|
||||
|
||||
### S4 — ai-service drops attachment provenance
|
||||
|
||||
Location: `apps/ai-service/adapters/qdrant.py:37-50`.
|
||||
|
||||
The adapter ignores payload `attachments` and instead constructs a fallback
|
||||
source reference from the section heading page plus broad ranges. Consequently
|
||||
`block_id`, `bbox` and `source_crop` are absent, and the physical page is wrong
|
||||
for 65/151 descriptors. The response can request PDF verification without
|
||||
linking to the quarantined crop/region.
|
||||
|
||||
### S5 — Schema v3/load path does not fail closed
|
||||
|
||||
Locations: `ingestion/ingestion/chunk/models.py:47-64`,
|
||||
`ingestion/ingestion/chunk/chunker.py:185-203`, and
|
||||
`ingestion/ingestion/load/models.py:30-42,161-177`.
|
||||
|
||||
`printed_page_range` defaults to `[]`; direct `chunk_all()` can omit the printed
|
||||
map; and loader validation neither requires schema v3 nor a non-empty printed
|
||||
range. Empty `source_page_range` and other empty lists also pass. The current
|
||||
canonical artifact is complete, but a future direct regeneration/load can spend
|
||||
on embeddings and then make every answer abstain for missing provenance.
|
||||
|
||||
## Non-blocking or latent findings
|
||||
|
||||
- `_atoms` can drop a comma for synthetic empty fragments such as `,,` after a
|
||||
long split (`chunker.py:70-79`). It does not fire in the current 11,974
|
||||
non-empty sections; the regression assertion strips commas and cannot catch
|
||||
it.
|
||||
- Corpus SHA is line-ending-dependent: identical JSONL data hashes differently
|
||||
with Windows CRLF versus Linux LF, which can falsely reject a CI/container
|
||||
load.
|
||||
- `_pack` can emit a label-only part in a synthetic single-label buffer. No such
|
||||
occurrence exists in the current artifact; this is separate from S1.
|
||||
- Adding `printed_page_map` before `measure` breaks old positional third-argument
|
||||
callers. No in-repo caller is affected.
|
||||
- ADR 0004/0006 and `docs/v1-delivery-plan.md` contain stale schema, table-count,
|
||||
token-estimator and page-tracking claims.
|
||||
- The earlier phrase “Ruff clean” applied to the selected changed paths. Claude
|
||||
confirmed that an unconfigured whole-directory `ruff check .` is not clean
|
||||
(397 findings), so it must not be represented as a repository-wide gate.
|
||||
|
||||
## Areas that passed review
|
||||
|
||||
- Canonical SHA confirmed:
|
||||
`e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38`.
|
||||
- 15,066 records: 14,915 prose + 151 descriptors; all schema v3.
|
||||
- Zero duplicate chunk IDs and zero UUID5 point-ID collisions.
|
||||
- `part_index`/`part_count` are consistent; descriptors are `(0,1)`.
|
||||
- Full in-memory regeneration is byte-identical on the same CRLF platform.
|
||||
- Strong independent reassembly check found zero source-substring failures,
|
||||
coverage gaps, reordering, or unintended duplication across all 11,974
|
||||
non-empty sections.
|
||||
- Zero chunks exceed 800 estimated tokens; maximum is exactly 800.
|
||||
- No confirmed quarantined numeric cell content leaked into prose chunks.
|
||||
- Printed folio extraction is derived from visible page headers and fails to
|
||||
`None` on ambiguity rather than guessing.
|
||||
- Current answer construction does not return quarantined descriptor text to
|
||||
the user; it forces `VERIFY_PDF`.
|
||||
- Ingestion tests: 258 passed. Claude's isolated ai-service run had 19 passed
|
||||
and 3 live-integration skips; the earlier configured local-service run had
|
||||
all 22 passing.
|
||||
|
||||
## Recommended fix order
|
||||
|
||||
1. Make overlap label-aware at both sides of every seam and add corpus-level
|
||||
tests for population + dose adjacency (S1).
|
||||
2. Stop inferring headers for continuation tables without reliable logical-table
|
||||
linkage; repair the two confirmed descriptors and strengthen the leak gate
|
||||
(S2).
|
||||
3. Compute exact per-chunk printed/physical provenance from `SectionPart`s and
|
||||
exact block provenance from attachments (S3).
|
||||
4. Preserve attachment block/page/bbox/crop through Qdrant and citation assembly
|
||||
(S4).
|
||||
5. Require schema v3 plus non-empty, valid page ranges at model, chunk and loader
|
||||
boundaries (S5).
|
||||
6. Regenerate the canonical artifact, rerun readiness/tests and repeat this
|
||||
review before embedding any corpus records.
|
||||
@@ -0,0 +1,195 @@
|
||||
# Review: `apps/ai-service/rag` and the hard-10 "10/10"
|
||||
|
||||
Reviewer: Claude, 2026-08-03. Every number below was produced by running the
|
||||
code, not by reading it. Reproduction commands are given per finding.
|
||||
|
||||
**Headline: the 10/10 reproduces, and it does not mean what it appears to
|
||||
mean.** Four of the ten passes are bought by a term list drawn from the ten
|
||||
scored queries, one passes for a reason unrelated to what it tests, and the
|
||||
whole eval can only run against an artifact built from the same ten pages.
|
||||
|
||||
Baseline, reproduced:
|
||||
|
||||
```
|
||||
cd apps/ai-service
|
||||
python -m rag.run_eval \
|
||||
--cases evals/manual_adversarial_hard10.jsonl \
|
||||
--documents ../../ingestion/scratch/rag-table-pilot/out/hard10/retrieval_documents.jsonl \
|
||||
--parents ../../ingestion/scratch/rag-table-pilot/out/hard10/logical_tables.jsonl
|
||||
-> release_gate: {"cases": 10, "passed": 10, "pass_rate": 1.0}
|
||||
```
|
||||
|
||||
`pytest -q` in `apps/ai-service` → **7 passed**.
|
||||
`ruff check --select F,E9,B,ARG .` → **2 errors** (both ARG001, one in
|
||||
`tests/test_retrieval_service.py::FixedRetriever.search`).
|
||||
|
||||
---
|
||||
|
||||
## 1. The abstain case passes by coincidence, and the same path refuses a valid question
|
||||
|
||||
`unsupported-veterinary` ("Liều famciclovir điều trị cho mèo là bao nhiêu?",
|
||||
`expected_id: null`) is the case that is supposed to show the system refusing
|
||||
an unsupported question. It abstains — with
|
||||
`reason: "ambiguous_top_evidence"`, not a scope check.
|
||||
|
||||
Measured: its top two hits tie at **0.675969 and 0.675969, a gap of exactly
|
||||
0.000000**. `_is_ambiguous` fires on the tie. Both scores are far above either
|
||||
threshold (0.08 in `run_eval`, 0.12 by default), so the refusal has nothing to
|
||||
do with the question being unanswerable.
|
||||
|
||||
Two checks that settle it:
|
||||
|
||||
- With `ambiguity_margin=0.0` the identical query returns
|
||||
`decision=verify_pdf` and three pieces of evidence — the case **fails**. The
|
||||
pass rests entirely on one tie-breaking constant.
|
||||
- Replacing `cho mèo` (for cats) with `cho người lớn` (for adults) — a
|
||||
perfectly answerable clinical question — produces the **same**
|
||||
`abstain / ambiguous_top_evidence`. The word "mèo" changes nothing.
|
||||
|
||||
So there is no out-of-scope detection in this service, and the eval reports
|
||||
that there is. For a drug reference aimed at clinicians this is the worst
|
||||
shape of defect available: a refusal mechanism that looks validated, fires on
|
||||
ties rather than on scope, and will refuse real dosing questions at the same
|
||||
rate.
|
||||
|
||||
## 2. `_structured_boost` is tuned on the queries it is scored against
|
||||
|
||||
`in_memory.QUANTITATIVE_TERMS` has 13 entries. **12 of the 13 appear
|
||||
literally in the 10 scored queries**; only `thành` does not:
|
||||
|
||||
```
|
||||
in queries : bao, clcr, kg, liều, lít, mg, ml, nồng, phút, thể, tích, tốc
|
||||
not in them: thành
|
||||
```
|
||||
|
||||
Load-bearing, measured by monkey-patching and re-running the same 10 cases:
|
||||
|
||||
| Configuration | Score |
|
||||
|---|---|
|
||||
| as shipped | 10/10 |
|
||||
| `QUANTITATIVE_TERMS` emptied | **8/10** |
|
||||
| `_structured_boost` disabled entirely | **6/10** |
|
||||
|
||||
Failures when the boost is removed: `formula-no-printed-bar`,
|
||||
`renal-herpes-typo`, `spatial-dose-formula`, `ors-who-composition`.
|
||||
|
||||
Four of the ten passes come from a hand-written list whose contents overlap
|
||||
the test queries almost exactly. That is fitting the test set; the resulting
|
||||
number predicts nothing about a query written by someone else.
|
||||
|
||||
Stated precisely, because it matters: these files are untracked, so there is
|
||||
no commit history to prove the term list was written *after* the queries. The
|
||||
12/13 overlap is strong evidence of it, not proof of the order.
|
||||
|
||||
## 3. "10/10" is Recall@3, not Recall@1
|
||||
|
||||
`EvaluationOutcome.passed` is `expected_id in retrieved_ids`, and
|
||||
`EvidencePolicy.evidence_limit` is 3. Scored at Recall@1 the same run gives
|
||||
**9/10**.
|
||||
|
||||
The one that moves is `formula-no-printed-bar` — the ADENOSIN formula printed
|
||||
without a fraction bar, i.e. exactly the case the outlier catalog flags as
|
||||
hardest. It lands at **rank 3 of 3**, behind
|
||||
`adenosin__lieu_luong_va_cach_dung__1` and `adenosin__than_trong__0`. An
|
||||
answer layer handed those three in that order sees two prose sections before
|
||||
the formula it actually needs.
|
||||
|
||||
## 4. The eval cannot be run on anything but the ten pages it was built from
|
||||
|
||||
`artifacts.load_documents` requires `section_key`, plus `source_refs` and
|
||||
`requires_visual_check`. Those fields exist **only** in
|
||||
`out/hard10/retrieval_documents.jsonl`:
|
||||
|
||||
```
|
||||
out/all : KeyError 'section_key'
|
||||
out/100 : KeyError 'section_key'
|
||||
out/hard10: loads
|
||||
```
|
||||
|
||||
The corpus-wide artifact — the 151-block, all-monograph one — cannot be loaded
|
||||
by this code at all. The retriever's entire universe is 164 documents across
|
||||
**6 drug_ids**, and those 6 are exactly the 6 under test (`set(artifact) -
|
||||
set(cases)` is empty). Per-query candidate pools are 19-31 documents, because
|
||||
`search` filters on `drug_id` first.
|
||||
|
||||
`CLAUDE.md` is explicit that a selected-page scope must not be reported as a
|
||||
whole-document one. Widening this eval requires fixing either the loader or
|
||||
the artifact writer; until then no number from it generalises.
|
||||
|
||||
## 5. Two cases do not test what their names say
|
||||
|
||||
`run_eval.run()` calls `service.retrieve(case.query, case.drug_id)` — the
|
||||
correct `drug_id` is handed in from the fixture.
|
||||
|
||||
- `renal-herpes-typo` deliberately misspells "famciclovia", but the case
|
||||
carries `drug_id: "famciclovir"`. Entity resolution is bypassed, so the typo
|
||||
never reaches the thing that would have to survive it; it only perturbs
|
||||
lexical scoring *inside* the already-correct drug.
|
||||
- `unsupported-veterinary` likewise gets the right drug handed to it.
|
||||
|
||||
Both are still useful as within-drug ranking cases. Neither is evidence about
|
||||
name resolution, which is where `docs/v1-delivery-plan.md` §B4 puts the 19
|
||||
measured substring traps.
|
||||
|
||||
## 6. `manual_adversarial` sits in the same release gate as `expert`
|
||||
|
||||
`RELEASE_GATE_ORIGINS = {EXPERT, MANUAL_ADVERSARIAL}`, and all ten cases are
|
||||
`manual_adversarial` — written by the same agent that wrote the retriever.
|
||||
`docs/v1-delivery-plan.md` §8 says this in as many words: self-written,
|
||||
self-graded questions measure the author's imagination, not clinical reality.
|
||||
|
||||
In fairness these are *routing* cases (did it fetch the right block id), not
|
||||
content-accuracy cases, and routing is legitimately self-checkable. The
|
||||
problem is the label: bucketing them with `expert` and calling the result a
|
||||
release gate reads as clinical validation to anyone who did not write it.
|
||||
|
||||
## 7. The eval does not exercise the policy that ships
|
||||
|
||||
`run_eval.run()` hardcodes `EvidencePolicy(minimum_score=0.08,
|
||||
ambiguity_margin=0.01)`; the class defaults are `0.12` and `0.015`.
|
||||
|
||||
I expected this to inflate the score. **It does not** — re-running with the
|
||||
default policy also gives 10/10. Reporting that because it was checked. It
|
||||
remains a smell that the benchmark and the shipped default are different
|
||||
constants, especially given finding 1, where the whole result turns on
|
||||
`ambiguity_margin`.
|
||||
|
||||
## 8. `_char_ngrams` is not character n-grams of the text
|
||||
|
||||
It builds `" ".join(sorted(_terms(text)))` — the unique words, alphabetised —
|
||||
then takes 3-grams of that. Word adjacency is destroyed and the resulting
|
||||
n-grams straddle alphabetically-neighbouring word boundaries. It still
|
||||
measures some overlap, and I did **not** trace a specific eval failure to it,
|
||||
so this is a naming/design objection rather than a demonstrated bug. But it
|
||||
should not be described as character n-gram matching in any writeup.
|
||||
|
||||
---
|
||||
|
||||
## What is genuinely good
|
||||
|
||||
Not everything here is a complaint, and these should survive any rework:
|
||||
|
||||
- The ports/adapters split is clean. `rag/ports.py` is `Protocol`-only and the
|
||||
domain imports no SDK — exactly the dependency inversion `CLAUDE.md` asks
|
||||
for, and it is why finding 1 could be tested at all.
|
||||
- `parent_hydration_failed` refuses to answer from a table-row fragment whose
|
||||
parent table is missing. That is the ADR 0006 contract enforced in code, and
|
||||
it is the right instinct.
|
||||
- `missing_provenance` abstains when a document has no `source_refs`. Also
|
||||
right, also load-bearing for citations.
|
||||
- `requires_visual_check` propagates from row *or* parent into `VERIFY_PDF`,
|
||||
which honours the quarantine rule rather than paraphrasing a table.
|
||||
|
||||
## Suggested order of work
|
||||
|
||||
1. Separate scope refusal from tie detection. A tie is not a reason to refuse;
|
||||
an out-of-drug or out-of-corpus question is. Right now only the first
|
||||
exists, and finding 1 shows it is standing in for the second.
|
||||
2. Make `artifacts.py` read the corpus-wide artifact, then re-run. Any number
|
||||
from a 6-drug universe is provisional.
|
||||
3. Report Recall@1 and Recall@3 separately, always both.
|
||||
4. Move `QUANTITATIVE_TERMS` out of the scorer or derive it from the corpus
|
||||
rather than by hand — and re-measure. A number produced with a query-derived
|
||||
boost list should carry that caveat wherever it is quoted.
|
||||
5. Rename the bucket, or split `manual_adversarial` out of the release gate
|
||||
until a pharmacist has written cases.
|
||||
@@ -0,0 +1,290 @@
|
||||
# Review round 2: verifying the response to round 1
|
||||
|
||||
Reviewer: Claude, 2026-08-03. Every claim in
|
||||
`response-rag-retrieval-2026-08-03.md` was re-run, not read.
|
||||
|
||||
**Verdict: five findings are genuinely fixed. One is not fixed — it was moved.
|
||||
Three new problems appeared in the fix itself.**
|
||||
|
||||
Reproduction:
|
||||
|
||||
```
|
||||
python -m rag.run_eval \
|
||||
--cases evals/manual_adversarial_hard10.jsonl \
|
||||
--documents ../../ingestion/scratch/rag-table-pilot/out/all/retrieval_documents.jsonl \
|
||||
--parents ../../ingestion/scratch/rag-table-pilot/out/all/logical_tables.jsonl \
|
||||
--aliases evals/drug_aliases.json
|
||||
-> recall_at_1/3/5 = 0.8889, negative_abstain_rate = 1.0
|
||||
```
|
||||
|
||||
The reported numbers reproduce exactly.
|
||||
|
||||
---
|
||||
|
||||
## Confirmed fixed
|
||||
|
||||
Checked in the code and by re-running, not taken on trust:
|
||||
|
||||
- **F2** — `QUANTITATIVE_TERMS` and `_structured_boost` are gone. Ranking is
|
||||
now real BM25 with IDF over the loaded corpus plus a character-n-gram term.
|
||||
No hand-written vocabulary remains in the ranker.
|
||||
- **F7** — `run_eval` now constructs `EvidencePolicy()` with the shipped
|
||||
defaults.
|
||||
- **F8** — `_char_ngrams` operates on `_normalized(text)` in original order.
|
||||
The sorted-unique-terms behaviour is gone.
|
||||
- **F1 mechanism** — `ambiguity_margin` is removed from `EvidencePolicy` and
|
||||
`_is_ambiguous` is deleted. Score ties no longer cause a refusal.
|
||||
- **F5 partly** — `run_eval` no longer hands `drug_id` to retrieval. A
|
||||
`CatalogDrugResolver` runs first, and the `famciclovia` typo genuinely
|
||||
resolves through `SequenceMatcher`; `renal-herpes-typo` now passes as
|
||||
`answerable` with resolution actually exercised. This is a real improvement.
|
||||
- **F4 partly** — the prose layer now covers **14,915 documents across 684
|
||||
drug IDs**, not 6. Also a real improvement.
|
||||
- **Verification claims** — all reproduced: `ingestion` **204 passed**,
|
||||
`apps/ai-service` **10 passed**, and lint is clean under both `ruff check rag
|
||||
tests` *and* the project's stricter `--select F,E9,B,ARG`. The two ARG001
|
||||
errors from round 1 are fixed.
|
||||
|
||||
---
|
||||
|
||||
## 1. NOT fixed — finding 2 was relocated, not resolved
|
||||
|
||||
Round 1's finding was: *four of ten passes are bought by a hand-written term
|
||||
list drawn from the scored queries.* The ranker is now clean. But the same
|
||||
pattern reappeared one layer up, in the thing that replaced it:
|
||||
|
||||
```python
|
||||
VETERINARY_TERMS = frozenset({"gia suc", "gia cam", "meo", "thu y"})
|
||||
```
|
||||
|
||||
plus a special-cased regex for `chó`. The evaluation has exactly **one**
|
||||
negative case, and it is about a **mèo**. `negative_abstain_rate: 1.0` is
|
||||
computed over **n = 1**, and that one word is in the list.
|
||||
|
||||
Measured, running the full `QueryRoutingService` against `out/all`:
|
||||
|
||||
| Query ending | Result |
|
||||
|---|---|
|
||||
| `... cho mèo` | ABSTAIN `out_of_scope_veterinary` |
|
||||
| `... cho chó` | ABSTAIN `out_of_scope_veterinary` |
|
||||
| `... cho thỏ` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho ngựa` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho lợn` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho bò sữa` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho chuột lang` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... cho vẹt cảnh` | **ANSWERS** `grounded_evidence_available` |
|
||||
| `... dùng trong thú cưng` | **ANSWERS** `grounded_evidence_available` |
|
||||
|
||||
Seven of nine veterinary phrasings are answered with a **human famciclovir
|
||||
dose** and the decision `grounded_evidence_available`. Note the last row: `thu
|
||||
y` is in the list, but `thú cưng` normalises to `thu cung` and misses.
|
||||
|
||||
`HumanClinicalScopeGuard` is not a scope guard. It is a five-entry animal-word
|
||||
list, and the evaluation that scores it contains exactly the words in it. The
|
||||
round-1 objection was never about `_structured_boost` specifically — it was
|
||||
about measuring a component against the cases it was written from. That
|
||||
objection still stands, unchanged, against this code.
|
||||
|
||||
A scope guard that generalises cannot be a keyword list. It has to come from
|
||||
something the corpus actually says — the book is a human formulary, so the
|
||||
question is whether the query's subject is a human patient, not whether it
|
||||
contains one of five nouns.
|
||||
|
||||
## 2. `recall_at_5` is not a measurement
|
||||
|
||||
`EvidencePolicy.evidence_limit` is 3, so `result.evidence` never exceeds three
|
||||
items and `retrieved_ids` never exceeds length 3 — observed lengths across the
|
||||
run are `{0, 1, 2, 3}`. `_recall_at(rows, 5)` then slices `[:5]` of a tuple
|
||||
that is at most 3 long.
|
||||
|
||||
**`recall_at_5` is forced to equal `recall_at_3` for every possible input.**
|
||||
It is not a third data point; it is `recall_at_3` printed twice. Round 1 asked
|
||||
for Recall@1 and Recall@3 reported separately, and that part is done and
|
||||
useful — but reporting a third identical figure makes the result look more
|
||||
thoroughly measured than it is.
|
||||
|
||||
Either raise `evidence_limit` above 5 for the diagnostic run, or drop
|
||||
`recall_at_5`.
|
||||
|
||||
## 3. `expected_drug_id` was added and never scored
|
||||
|
||||
The field exists in `EvaluationCase` and is populated by `read_cases` for all
|
||||
10 cases. It appears **nowhere else** — `EvaluationOutcome.passed` and
|
||||
`summarize()` never read it.
|
||||
|
||||
So resolution now happens, but resolution *correctness* is still unmeasured. A
|
||||
case that resolves to the wrong drug and then abstains is indistinguishable in
|
||||
the report from a case that correctly abstained. That is precisely the
|
||||
distinction finding 5 existed to create.
|
||||
|
||||
Scoring it is a two-line change and would make `ors-who-composition`'s
|
||||
`drug_resolution_ambiguous` legible as "resolver declined" rather than an
|
||||
unexplained miss.
|
||||
|
||||
## 4. The alias catalog is one drug out of 684, and it is one under test
|
||||
|
||||
`evals/drug_aliases.json` in full:
|
||||
|
||||
```json
|
||||
{"thuoc_uong_bu_nuoc_va_ien_giai": ["oresol", "ORS"]}
|
||||
```
|
||||
|
||||
684 drugs in the catalog, hand-aliases for **1**, and that 1 is the drug behind
|
||||
two of the ten cases. `docs/v1-delivery-plan.md` §B1 records **344 real
|
||||
`X - xem Y` aliases** already extractable from the back index, plus 492
|
||||
`ten_thuong_mai` entries (§B2). None are wired in.
|
||||
|
||||
This is the same shape as finding 2: the coverage that exists is exactly the
|
||||
coverage the test needs. Loading the 344 measured aliases would make the
|
||||
resolver's alias path testable against something other than itself.
|
||||
|
||||
## 5. "out/all" does not mean the whole book, and "out/100" no longer means anything
|
||||
|
||||
Measured from the manifests and the artifacts:
|
||||
|
||||
| artifact | manifest pages | docs | drugs | prose | table_whole | table_row | formula |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| `out/all` | 116 | 15,727 | 684 | 14,915 | 133 | 669 | 10 |
|
||||
| `out/100` | 100 | 15,593 | 684 | 14,915 | 116 | 552 | 10 |
|
||||
| `out/hard10` | 10 | 164 | 6 | 130 | 4 | 28 | 2 |
|
||||
|
||||
Two things follow.
|
||||
|
||||
The prose layer is now genuinely whole-corpus (identical 14,915 documents in
|
||||
both), which is the real fix and deserves the credit. But the **table/formula
|
||||
layer in `out/all` covers 116 pages**, and its 133 parents + 10 formulas match
|
||||
the 133 logical parents + 10 formulas recorded in the progress log — so "all"
|
||||
is honest *for tables* and misleading as a general label. The response's
|
||||
sentence "`out/all` now has 15,727 documents across all 684 drug IDs" is
|
||||
literally true and reads as whole-book coverage of everything, which it is not.
|
||||
|
||||
Second: `out/100` and `out/all` now differ by 17 tables and 117 rows and
|
||||
nothing else. The 100-page scope has stopped being a distinct scope. Either
|
||||
retire it or say what it is for.
|
||||
|
||||
## 6. `assert` on the request path
|
||||
|
||||
`routing.py:131` — `assert resolution.drug_id is not None`. Assertions are
|
||||
removed under `python -O`, at which point `retrieve` is called with `None`.
|
||||
Minor, but it is in the live path; make it an explicit raise.
|
||||
|
||||
## On the ORS miss
|
||||
|
||||
The response calls the one positive miss "intentionally safe". That is
|
||||
defensible — "Công thức oresol WHO UNICEF pha một lít có bao nhiêu **natri
|
||||
clorid**?" does name two catalog entities, and declining beats guessing.
|
||||
|
||||
Worth stating the cost plainly, though: this is now the **second** mechanism
|
||||
that refuses an answerable clinical question (the tie was the first, and it is
|
||||
gone). Any question naming a drug and one of its ingredients will hit it, and
|
||||
that pattern is common in a formulary. It is untested beyond this single case.
|
||||
Not a defect — an accepted trade-off that should be measured before it is
|
||||
called safe.
|
||||
|
||||
---
|
||||
|
||||
## 7. Added after the fact — the alias gap makes common drugs unreachable
|
||||
|
||||
This came out of testing §4's practical effect and is **more serious than §1**.
|
||||
|
||||
Monograph headings that carry a parenthesised synonym become a single
|
||||
compound `drug_id`, and `build_drug_catalog` produces no alias for either
|
||||
part:
|
||||
|
||||
```
|
||||
paracetamol_acetaminophen -> {"paracetamol acetaminophen",
|
||||
"PARACETAMOL (Acetaminophen)"}
|
||||
acid_acetylsalicylic_aspirin -> {"acid acetylsalicylic aspirin",
|
||||
"ACID ACETYLSALICYLIC (Aspirin)"}
|
||||
```
|
||||
|
||||
Measured against `out/all`:
|
||||
|
||||
| Query | Resolution |
|
||||
|---|---|
|
||||
| `Liều paracetamol cho người lớn là bao nhiêu?` | **`not_found`** |
|
||||
| `Chống chỉ định của aspirin là gì?` | **`not_found`** |
|
||||
| `Liều paracetamol acetaminophen cho người lớn?` | `resolved` |
|
||||
| `Liều metformin cho người lớn là bao nhiêu?` | `resolved` |
|
||||
|
||||
Two of the most-asked-about drugs in any formulary are unreachable unless the
|
||||
user types the book's exact compound heading. It fails *safely* — it abstains
|
||||
rather than answering wrongly — which is exactly why the 8/9 diagnostic cannot
|
||||
see it: none of the ten cases involves a parenthesised heading.
|
||||
|
||||
`docs/v1-delivery-plan.md` §B1/§B2 already record **344 `X - xem Y` aliases**
|
||||
and **492 `ten_thuong_mai` entries** as extractable. Until they are loaded,
|
||||
resolver coverage is whatever the headings happen to spell.
|
||||
|
||||
**Correction to my own suspicion.** I expected the response's claim — "queries
|
||||
with multiple distinct drug entities abstain as ambiguous" — to be false,
|
||||
because `Nên dùng paracetamol hay ibuprofen cho trẻ sốt cao?` answers about
|
||||
ibuprofen alone. It is not false. Re-tested with two drugs that are both in
|
||||
the catalog:
|
||||
|
||||
```
|
||||
Tuong tac giua digoxin va amiodaron -> ambiguous
|
||||
Nen dung omeprazol hay pantoprazol -> ambiguous
|
||||
Tuong tac giua warfarin va amiodaron -> ambiguous
|
||||
```
|
||||
|
||||
The multi-entity guard works. The paracetamol/ibuprofen query slips through
|
||||
because paracetamol is *not reachable at all*, so the query looks
|
||||
single-entity. The alias gap does not merely reduce coverage — it silently
|
||||
disables the ambiguity protection that Codex is relying on.
|
||||
|
||||
## 8. "Veterinary" is not a requirement this project ever had
|
||||
|
||||
Worth saying plainly, because §1 spent the entire fix budget on it. The
|
||||
veterinary category exists in this codebase for one reason: Codex wrote one
|
||||
negative eval case about a cat, round 1 showed it passed by coincidence, and
|
||||
the repair was a guard for cats.
|
||||
|
||||
The out-of-scope categories the project documents are different ones —
|
||||
`docs/v1-delivery-plan.md` §8 (general chapters printed 37-98 and appendices
|
||||
1497-1528 are not in the corpus) and `docs/architecture.md` (scoped refusal
|
||||
for questions that are not formulary lookups). Measured against `out/all`:
|
||||
|
||||
| Category | Result | Reason |
|
||||
|---|---|---|
|
||||
| general chapter — "nguyên tắc kê đơn thuốc" | ABSTAIN | `drug_not_resolved` |
|
||||
| general chapter — "ngộ độc và thuốc giải độc" | ABSTAIN | `drug_not_resolved` |
|
||||
| appendix — "bảng tương hợp thuốc tiêm truyền" | ABSTAIN | `drug_not_resolved` |
|
||||
| drug outside the formulary — semaglutid | ABSTAIN | `drug_not_resolved` |
|
||||
| symptom diagnosis — "tôi đau đầu buồn nôn" | ABSTAIN | `drug_not_resolved` |
|
||||
| **recommendation — "nên dùng X hay Y cho trẻ sốt cao"** | **ANSWERS** | `grounded_evidence_available` |
|
||||
|
||||
Five of six abstain, but none of them because scope was checked — they abstain
|
||||
because no drug name matched, which is `drug_not_resolved` doing scope work by
|
||||
accident. The one that gets through is the recommendation question, which
|
||||
`architecture.md` explicitly says must be refused.
|
||||
|
||||
So the guard covers a category nobody asked for, covers it with five words,
|
||||
and the category that *is* specified is unhandled.
|
||||
|
||||
## Summary
|
||||
|
||||
| Round-1 finding | Status |
|
||||
|---|---|
|
||||
| 1 — refusal was a score tie | mechanism removed; **replacement is a 5-word list, see §1** |
|
||||
| 2 — boost tuned on scored queries | fixed in the ranker; **pattern reappears in the scope guard** |
|
||||
| 3 — Recall@3 sold as Recall@1 | fixed; **but `recall_at_5` is padding, see §2** |
|
||||
| 4 — eval locked to 6 drugs | prose fixed (684 drugs); table layer still 116 pages |
|
||||
| 5 — drug_id handed in | resolver added and works; **correctness still unscored, see §3** |
|
||||
| 6 — manual cases in expert gate | fixed; `expert_release_gate` now has 0 cases and null metrics |
|
||||
| 7 — eval used non-default policy | fixed |
|
||||
| 8 — char n-grams sorted | fixed |
|
||||
|
||||
Priority order:
|
||||
|
||||
1. **§7 — the alias gap.** `Liều paracetamol cho người lớn?` returns
|
||||
`not_found`. It is the most likely question a real user asks, it fails
|
||||
today, and it also disables the multi-entity ambiguity guard. Loading the
|
||||
344 back-index aliases and splitting parenthesised headings fixes both.
|
||||
2. **§1 — the scope guard.** Seven of nine veterinary phrasings are answered
|
||||
with a human dose under the label `grounded_evidence_available`. Lower than
|
||||
§7 only because a doctor is unlikely to ask it; the label is what makes it
|
||||
dangerous.
|
||||
3. **§8** — the specified out-of-scope category (recommendation questions) is
|
||||
unhandled while an unspecified one has a guard.
|
||||
4. §2, §3, §4, §6 — reporting and coverage bookkeeping.
|
||||
Reference in New Issue
Block a user