Wire the guarded conversational RAG answer layer end-to-end
This commit is contained in:
@@ -93,9 +93,15 @@ in a chunk shown to a doctor or pharmacist.
|
||||
— per CLAUDE.md's provenance rule): `chunk_id`
|
||||
(`{drug_id}__{section_key}__{part_index}`), `drug_id`, `drug_name`,
|
||||
`section_key`, `section_display_name`, `atc_codes` (inherited from the
|
||||
monograph — enables ATC-class-filtered retrieval), `source_page_range`
|
||||
(monograph-level, see Consequences), `part_index`/`part_count` (`0`/`1`
|
||||
for un-split sections, keeps the schema uniform across all chunks).
|
||||
monograph — enables ATC-class-filtered retrieval), exact per-chunk
|
||||
`source_page_range` and `printed_page_range`, `part_index`/`part_count`
|
||||
(`0`/`1` for un-split sections, keeps the schema uniform across all chunks).
|
||||
6. **Schema v4 separates source from retrieval context.** `source_text` is the
|
||||
exact contiguous source span and is the basis for lossless reassembly and
|
||||
page provenance. `text` may prefix repeated route/population labels so a
|
||||
continuation chunk is independently safe to retrieve. Those retrieval-only
|
||||
prefixes are recorded in `context_labels` and may not alter `source_text`.
|
||||
Token counts use `cl100k_base`, not the earlier chars/4 estimate.
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -116,15 +122,10 @@ in a chunk shown to a doctor or pharmacist.
|
||||
by sub-compound — a chunk from this section is tagged with the class
|
||||
name only, not the specific analogue a query might target. Deferred to
|
||||
golden-dataset-driven eval rather than guessed at now.
|
||||
- **Known gap — sub-chunk page precision**: `source_page_range` is
|
||||
monograph-level, not sub-chunk-exact. A sub-chunk from late in a
|
||||
multi-page section inherits the whole monograph's page range rather than
|
||||
its own precise page, because per-line page tracking doesn't currently
|
||||
exist in `SectionSpan`/`Heading`. The monograph + section-heading page is
|
||||
still real, checkable provenance, but this is a known precision gap, not
|
||||
full sub-chunk traceability. Flagged as a future improvement.
|
||||
- **Not yet built**: the Vietnamese sentence-boundary splitter itself
|
||||
(abbreviation handling, decimal-comma handling, ATC-code-period handling)
|
||||
is specified here as a rule, not implemented or unit-tested. Building and
|
||||
testing it is a separate, later task (`ingestion/ingestion/chunk/`, which
|
||||
does not exist yet).
|
||||
- **Resolved — sub-chunk page precision**: schema v4 derives exact physical
|
||||
support from the contiguous `source_text` span and maps it to verified
|
||||
printed folios. Missing or ambiguous support fails readiness rather than
|
||||
falling back to monograph-level provenance.
|
||||
- **Implemented**: the sentence/label-aware splitter is in
|
||||
`ingestion/ingestion/chunk/` with regression tests for dose continuations,
|
||||
compound label boundaries, parent route context and lossless reassembly.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Status
|
||||
|
||||
Proposed, with implementation to follow immediately. Resolves the item ADR
|
||||
Accepted and implemented in schema v4. Resolves the item ADR
|
||||
0005 explicitly deferred ("Table/formula content blocks … the precise
|
||||
`ContentBlock`/table-row/formula-unit shape is deferred to a follow-up
|
||||
revision of this ADR once the survey reports real numbers"). The survey has
|
||||
@@ -22,12 +22,10 @@ the current whole-corpus output:
|
||||
|
||||
| quantity | value |
|
||||
|---|---|
|
||||
| lifted blocks | 167, all quarantined |
|
||||
| monographs affected | 96 of 683 (**14.1%**) |
|
||||
| sections affected | 108 |
|
||||
| **blocks in `lieu_luong_va_cach_dung`** | **127 (76%)** |
|
||||
| next largest section | `duoc_ly_va_co_che_tac_dung`, 16 |
|
||||
| shapes | simple_table 136, multi_level_or_merged_header 16, formula_2d 14, cross_page_continuation 1 |
|
||||
| lifted blocks represented by descriptor chunks | 151, all quarantined |
|
||||
| sections affected | 103 |
|
||||
| **blocks in `lieu_luong_va_cach_dung`** | **125** |
|
||||
| unverified header rows admitted to embedding text | **0** |
|
||||
|
||||
So three quarters of everything removed from prose was removed from the
|
||||
dosing section, in a drug formulary, for an audience of doctors and
|
||||
@@ -57,9 +55,11 @@ class ChunkAttachment:
|
||||
shape: str # simple_table | multi_level_or_merged_header |
|
||||
# cross_page_continuation | formula_2d
|
||||
physical_page: int
|
||||
printed_page: int
|
||||
bbox: List[float]
|
||||
quarantined: bool
|
||||
header_row: List[str] = () # simple_table only; see caveat below
|
||||
header_row: List[str] = () # always empty until separately verified
|
||||
source_crop: str | None = None
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Chunk:
|
||||
@@ -82,28 +82,20 @@ A block also gets its own chunk so it is retrievable at all:
|
||||
chunk_id = "{drug_id}:{section_key}:block:{block_id}"
|
||||
chunk_kind = "block_descriptor"
|
||||
text = "AMPICILIN VÀ SULBACTAM — Liều lượng và cách dùng — bảng,
|
||||
trang in 204. Cột: Độ thanh thải creatinin | Nửa đời |
|
||||
Liều ampicilin/sulbactam."
|
||||
trang in 204."
|
||||
```
|
||||
|
||||
The text is assembled from the drug name, the section display name, the kind,
|
||||
the printed page and — for `simple_table` only — the header row. **No cell
|
||||
value ever appears.** A header row is a row of labels; linearising it cannot
|
||||
invent a numeric relationship, which is precisely what linearising a body row
|
||||
does. For every other shape the header is omitted, because
|
||||
`multi_level_or_merged_header` is the shape whose header extraction is least
|
||||
trustworthy.
|
||||
|
||||
Caveat recorded in the schema itself: `header_row` comes from
|
||||
`pdfplumber.find_tables()`'s first row and has **not** been verified by eye
|
||||
(the 180 real tables' individual shapes are rule-derived; only the 20
|
||||
"not a table" verdicts were visually confirmed). It is retrieval bait, never
|
||||
an answer.
|
||||
The text is assembled only from verified metadata: drug name, section display
|
||||
name, block kind and printed page. **No cell value or inferred header appears.**
|
||||
The earlier proposal to use `pdfplumber.find_tables()`'s first row was rejected
|
||||
after corpus audit: a guessed first row can be a body row or can merge numeric
|
||||
relationships. Until a separate human-verified header dataset exists,
|
||||
`header_row` is embargoed for every shape and serialized as empty.
|
||||
|
||||
### 3. The answer layer's obligations (binding on `ai-service`)
|
||||
|
||||
These are stated here because they are the reason the schema exists; they are
|
||||
not implemented by `ingestion/`.
|
||||
These obligations are implemented across `ingestion/` and `ai-service` and are
|
||||
enforced by tests/readiness gates.
|
||||
|
||||
1. A retrieved chunk with `has_quarantined_content: true` **must** cause the
|
||||
answer to state that a table or formula exists at the cited page, and to
|
||||
@@ -158,14 +150,18 @@ Added to `cli chunk-ready` and to the chunk stage's own tests:
|
||||
contains a quarantined block's text
|
||||
5. `descriptor_chunk_count == block_count`
|
||||
6. `descriptor_chunk_without_attachment = 0`
|
||||
7. `attachment_header_row_present = 0`
|
||||
8. `descriptor_with_unverified_columns = 0`
|
||||
9. `descriptor_range_not_attachment_page = 0`
|
||||
10. `attachment_without_printed_page = 0`
|
||||
|
||||
## Consequences
|
||||
|
||||
- Prose chunks shrink slightly in trustworthiness terms but grow in honesty:
|
||||
the ones missing a table now say so.
|
||||
- The index gains 167 descriptor chunks (≈1.4% of the expected chunk count),
|
||||
- The current candidate index gains 151 descriptor chunks,
|
||||
each cheap and none carrying unsafe text.
|
||||
- `ai-service` cannot be built to answer a dosing question from prose alone
|
||||
for the 108 affected sections without violating a stated contract.
|
||||
- `ai-service` cannot answer a dosing question from prose alone for the 103
|
||||
affected sections without violating a stated contract.
|
||||
- The 14 `formula_2d` attachments make the two Cockcroft-Gault formulas
|
||||
answerable as crops today, which they are not now.
|
||||
|
||||
@@ -792,6 +792,81 @@ mistaken for complete formula coverage.
|
||||
book is **16/23 = 69.6%**, and its recall is unknown. A geometric heuristic
|
||||
finds candidates; it never proves absence.
|
||||
|
||||
### 26. Exact section vocabulary can occur as wrapped prose or inside tables; context must precede label matching
|
||||
**What it looks like:** several unrelated defects shared one cause. A wrapped
|
||||
body sentence can put `chống chỉ định.` alone on the next visual line
|
||||
(NADROPARIN, physical page 1016); a dosing-table cell can literally be named
|
||||
`Chỉ định` (WARFARIN p1485 and IOBITRIDOL p826); and a verified fraction band
|
||||
widened to capture its numerator can geometrically overlap prose in the other
|
||||
column (NETILMICIN p1042). Exact vocabulary matching alone classified these as
|
||||
structure or quarantined content.
|
||||
|
||||
**Why it matters:** the output remains grammatical while moving or deleting a
|
||||
clinically decisive phrase, assigning a dosing table to indications, or hiding
|
||||
a cross-reference. Aggregate “all spans assigned” and section-level provenance
|
||||
gates all passed before these defects were found.
|
||||
|
||||
**Handling:** classify out-of-scope spans and known table regions before title/
|
||||
section matching; treat a non-bold exact label as prose when it is the adjacent
|
||||
line of an unterminated span in the same PDF block; require a formula region's
|
||||
column to agree with the source span's column; and validate source-span IDs on
|
||||
every individual part. Confirmed aliases (`Tên chung quốc tế và mã ATC`, `Dạng
|
||||
bào chế và hàm lượng`, and the tetanus-toxoid dosing heading) are recorded in
|
||||
the open vocabulary.
|
||||
|
||||
**Whole-corpus result:** 684 monographs (was 683), maximum monograph range 7
|
||||
pages (was the false 164-page ZOLPIDEM range), 11,974 sections, 151 quarantined
|
||||
blocks, 15,066 chunks, 0 unassigned spans, and every readiness gate passing.
|
||||
|
||||
**Generalizes:** vocabulary is evidence, not sufficient context. Apply known
|
||||
geometric scope (page, table, column, visual-line continuity) before interpreting
|
||||
a label-shaped string as document structure.
|
||||
|
||||
### 27. One physical table can be non-contiguous in PDF block order
|
||||
**What it looks like:** a table is contiguous on the rendered page, but the PDF
|
||||
content stream interleaves a visually later section heading between its cells.
|
||||
This split CAPECITABIN p308 and IMATINIB p795 into multiple blocks with the same
|
||||
region ID and conflicting section owners. CAPECITABIN p309 adds a second case:
|
||||
two explicitly captioned dose-adjustment tables are printed after the ordinary
|
||||
`Tên thương mại` field without repeating the dosage heading.
|
||||
|
||||
**Why it matters:** sorting or classifying one extracted span at a time makes a
|
||||
single physical object acquire several meanings. The flattened text remains
|
||||
plausible, so ordinary text and coverage gates do not expose the defect.
|
||||
|
||||
**Handling:** collect all spans belonging to a verified region before semantic
|
||||
classification and emit the region atomically at its first occurrence. A narrow
|
||||
caption rule maps only `Bảng N. Điều chỉnh liều ...` appendices to
|
||||
`lieu_luong_va_cach_dung`; generic occurrences of the word “liều” are not used.
|
||||
A readiness gate now requires unique physical-region IDs.
|
||||
|
||||
**Verification:** all **151/151 unique regions** were rendered and read against
|
||||
the PDF. The regenerated corpus has 151 blocks, 151 unique IDs, and zero
|
||||
duplicate-ID gate failures; CAPECITABIN p309 tables are both owned by dosage.
|
||||
|
||||
**Generalizes:** physical-region identity must outrank text-stream adjacency for
|
||||
tables, formulas, figures, and other layout objects.
|
||||
|
||||
### 28. A bar-less formula needs an asymmetric band, but geometry cannot prove its operator
|
||||
**What it looks like:** ADENOSIN p147 prints a wrapped numerator followed by
|
||||
`Nồng độ adenosin (3 mg/ml).` with no horizontal fraction rule. The generic
|
||||
symmetric formula band captured the numerator only, making a plausible but
|
||||
incomplete source crop.
|
||||
|
||||
**Why it matters:** the missing denominator changes the calculation. Visual
|
||||
review of all reconstructed sandbox crops found the defect even though ordinary
|
||||
readiness and block-count gates passed.
|
||||
|
||||
**Handling:** verified bar-less regions use a 31pt lower margin from the
|
||||
synthetic anchor. On this page the denominator ends about 29pt below the anchor;
|
||||
the following `Ví dụ:` begins immediately after the new boundary. A regression
|
||||
requires the denominator boundary and excludes that prose. The reconstructed
|
||||
record still sets `requires_human_operator_confirmation`: layout supplies no
|
||||
bar from which multiplication versus division can be proven.
|
||||
|
||||
**Generalizes:** expand a verified crop to preserve all visible operands, but
|
||||
never invent a mathematical operator that the source geometry does not encode.
|
||||
|
||||
## Not yet investigated (flagged for future work, not silently ignored)
|
||||
|
||||
- **Footnote-style superscript reference markers** (seen as `a, b, c, d` in
|
||||
@@ -799,11 +874,9 @@ finds candidates; it never proves absence.
|
||||
correctly associated with its marker/row during extraction.
|
||||
- **How many bar-less formulas exist** (item 25) — one confirmed, total
|
||||
unmeasured; no geometric signal can bound it.
|
||||
- **Merging the 51 transcribed outlined runs back into monograph text**
|
||||
(item 24) — transcribed and stored, but the corpus still contains
|
||||
`Độ n định`.
|
||||
- **2D grid table reconstruction** (item 7) — no implementation yet for
|
||||
recovering row/column-correct values from a nomogram-style table.
|
||||
- **Production 2D grid reconstruction** (item 7) — the 100-page sandbox now
|
||||
reconstructs grids and logical cross-page tables, but merged-cell semantics
|
||||
and whole-book recall are not yet production gates.
|
||||
- **Exact shortest monograph name+page** — a quick unmerged crude scan (no
|
||||
multi-line title merge) gave a different longest-monograph ranking than
|
||||
the already-documented authoritative one (item 12e: "AMOXICILIN VÀ KALI
|
||||
|
||||
@@ -1,5 +1,504 @@
|
||||
# Progress Log
|
||||
|
||||
## 2026-08-04 (evening) — Section routing: contraindication retrieval goes from 0.05 to 1.00, at zero cloud cost
|
||||
|
||||
The retrieval defect measured earlier today is fixed by routing rather than by
|
||||
embedding. **No cloud call was made and nothing was re-embedded** — Bedrock
|
||||
access is still revoked.
|
||||
|
||||
**The change.** A question that names its own attribute does not need
|
||||
similarity to guess which section answers it. `rag/sections.py` maps the
|
||||
question to a `section_key`; `QdrantRetriever.find_by_section` then filters on
|
||||
`(drug_id, section_key)` and returns **every** part of that section as a
|
||||
`scroll`, not a top-k. `RetrievalService` takes that route when it resolves and
|
||||
falls back to similarity otherwise.
|
||||
|
||||
Two rules carry the safety. **Longest phrase wins**: "chống chỉ định" and "chỉ
|
||||
định" differ by one prefix word and mean opposite things, so every phrase is
|
||||
sorted by length and the longer is tested first — the same rule keeps "quá
|
||||
liều" from being read as "liều" and "hướng dẫn xử trí ADR" from being read as
|
||||
"tác dụng phụ". **No match is not a guess**: an unrecognised question returns
|
||||
`None` and falls back rather than picking a section it is unsure of.
|
||||
|
||||
**Measured against the real `duocthu_v1` collection, no embedding involved:**
|
||||
|
||||
| | similarity (measured this afternoon) | section routing |
|
||||
|---|---|---|
|
||||
| hit@1, 160 generated cases | 0.544 | **1.000** |
|
||||
| `chong_chi_dinh` | **0.05** | **1.00** |
|
||||
| misroutes / empty / leaked sections | — | 0 / 0 / 0 |
|
||||
|
||||
**The generated 160 flattered it, and testing on human-written questions said
|
||||
so.** Those questions use the phrasings the table was built from, so 160/160 is
|
||||
partly circular. Run against the 16 single-drug questions humans actually wrote
|
||||
in `Golden Dataset/golden_e2e_v1.csv`, the first version scored **10/16**. The
|
||||
six failures were two gaps: four questions say just "Liều Metformin cho người
|
||||
lớn?" — bare "liều", which the table lacked — and one says "Bà bầu", a
|
||||
colloquial phrasing for pregnancy. Adding those phrases (no code change, which
|
||||
is what the open/closed table is for) took it to **16/16** while the confusable
|
||||
pairs still resolve correctly; bare "liều" is safe only because "quá liều" is
|
||||
longer and tested first, and there is a regression test pinning exactly that.
|
||||
|
||||
**A circular import was found and fixed properly rather than worked around.**
|
||||
`service -> sections -> routing -> service`, because `normalize_name` lived in
|
||||
`routing.py`. It is a text utility with no knowledge of drugs or sections, so
|
||||
it moved to `rag/text.py`; `routing.py` re-exports it so existing imports keep
|
||||
working.
|
||||
|
||||
**Also wired, and still unproven:** `BedrockCohereQueryEmbedder` replaces the
|
||||
SHA-256 hash embedder for the similarity fallback path. It has been
|
||||
import-checked only — **never run against Bedrock** — so the fallback path
|
||||
remains unverified end to end. The section route does not depend on it.
|
||||
|
||||
Verification actually run: ai-service **37 passed, 3 skipped** (22 before, +15);
|
||||
ingestion **296 passed** (unchanged, checked for regression); `ruff --select
|
||||
F,E9,B,ARG` over `rag/`, `adapters/`, `bootstrap.py`, `config.py` and `tests/`
|
||||
— **all checks passed**; section-route evaluation against the live collection
|
||||
160/160; human-written golden questions 16/16.
|
||||
|
||||
Not established: multi-attribute questions ("liều dùng và chống chỉ định") pick
|
||||
the longest phrase, which is deterministic but arbitrary; phrase coverage
|
||||
beyond these 16 human questions is unmeasured; and none of this speaks to
|
||||
whether the retrieved text is clinically correct.
|
||||
|
||||
## 2026-08-04 (afternoon) — First real embeddings exist; retrieval measured at 54% and the cause is not what the small sample said
|
||||
|
||||
The corpus is embedded for the first time. Bedrock IAM was opened on the
|
||||
owner's explicit instruction, all 15,100 chunks were embedded with
|
||||
`cohere.embed-v4:0`, loaded into Qdrant, and **cloud access was then revoked
|
||||
and proven revoked** before the owner's 17:00 deadline. Measured spend
|
||||
**~$0.49** of a personal $138 budget.
|
||||
|
||||
**Gate results.** 15,100/15,100 embedded; 15,100 points in `duocthu_v1` over 59
|
||||
batches; collection point count 15,100 — count gate **PASS**. Manifest records
|
||||
`cohere.embed-v4:0`, 1024 dimensions, Cosine, corpus SHA
|
||||
`04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`. Corpus SHA
|
||||
was re-verified against the morning audit before spending: identical, and
|
||||
identical to the post-lint copy, so the 12:05 `chunker.py` edit did not change
|
||||
output.
|
||||
|
||||
**Both providers were probed live before choosing.** Titan v2 and Cohere v4
|
||||
each returned 1024 dimensions with a **measured L2 norm of 1.000000**. That
|
||||
settles a question left open since 2026-08-03: Cohere's `normalized` field was
|
||||
`None` because AWS's docs never state it. It is now measured. Cohere was chosen
|
||||
on two measured grounds — the corpus is Vietnamese and Cohere is explicitly
|
||||
multilingual, and `bedrock_cohere.py` batches 96 texts per request while
|
||||
`bedrock_titan.py` sends one, which at a measured 2.3s per call is ~9.6 hours
|
||||
versus minutes. The $0.41 price difference did not drive it.
|
||||
|
||||
**The retrieval number, and a correction to a claim made earlier the same
|
||||
day.** A 160-case evaluation (20 per section, 8 sections, questions generated
|
||||
from the corpus so labels are structural) measured **hit@1 0.544, hit@3 0.663,
|
||||
hit@5 0.738**. Per section:
|
||||
|
||||
| section | hit@1 |
|
||||
|---|---|
|
||||
| `chong_chi_dinh` | **0.05** (1/20) |
|
||||
| `chi_dinh` | 0.30 |
|
||||
| `tac_dung_khong_mong_muon` | 0.40 |
|
||||
| `lieu_luong_va_cach_dung` | 0.60 |
|
||||
| `qua_lieu_va_xu_tri` | 0.65 |
|
||||
| `than_trong` | 0.65 |
|
||||
| `tuong_tac_thuoc` | 0.80 |
|
||||
| `thoi_ky_mang_thai` | 0.90 |
|
||||
|
||||
An earlier 15-case run gave a similar headline (0.533) but led to the **wrong
|
||||
diagnosis**: four of its seven failures were contraindication questions
|
||||
answered with indications, so the cause was reported as embedding weakness at
|
||||
negation. At 160 cases that pair accounts for only **3** confusions. The
|
||||
dominant mechanism is different and larger: **`duoc_ly_va_co_che_tac_dung`
|
||||
absorbs questions from every other section** — 10 from adverse effects, 8 from
|
||||
contraindications, 7 from dosage, 5 from indications. It is the largest section
|
||||
(1,896 chunks) and describes the drug in general terms, so it sits close to
|
||||
almost any question about that drug. This is the small-sample failure mode
|
||||
CLAUDE.md warns about, reproduced on this project.
|
||||
|
||||
**Re-embedding cannot fix this, and the capability to fix it already exists.**
|
||||
Verified by reading the code, not assumed: `apps/ai-service/adapters/qdrant.py`
|
||||
`search()` filters on `drug_id` only and lets vector similarity choose the
|
||||
chunk; `rag/routing.py` resolves drug and intent but **not section**; and
|
||||
`find_by_payload` — the "return the whole section" method in `ingestion/load/`
|
||||
— is **never called anywhere in `apps/ai-service`**. Attribute questions
|
||||
therefore depend on similarity picking the right section, which is what
|
||||
measures 54%. The fix is to resolve the attribute to a `section_key` and
|
||||
retrieve that section whole; `ATTRIBUTE_TO_SECTION` already exists in
|
||||
`embed/benchmark_local.py`.
|
||||
|
||||
**A silent-failure hazard found and closed.** `apps/ai-service` embedded
|
||||
queries with `LocalHashQueryEmbedder` — SHA-256 of tokens, explicitly plumbing
|
||||
only — while the collection now holds Cohere vectors. Querying across those two
|
||||
spaces returns hits and raises nothing; the results are simply meaningless.
|
||||
`BedrockCohereQueryEmbedder` was added and wired behind
|
||||
`EMBEDDING_PROVIDER=cohere-v4`. **It has only been import-checked — never run
|
||||
against Bedrock**, because cloud access was revoked first, as instructed.
|
||||
|
||||
**Two operational lessons, both paid for.** `bedrock_runtime.py` set no boto3
|
||||
timeout, so a single throttled response held a socket open for over five
|
||||
minutes and stalled the whole run; `connect_timeout=10, read_timeout=60` plus
|
||||
standard retries fixed it. Then the first full run still died at ~14,600/15,100
|
||||
because the retry backoff (2s, 4s) was far shorter than a per-minute token
|
||||
quota needs. The disk cache made that survivable: the resumed run recorded
|
||||
**14,977 cache hits and 123 misses**, so only 123 vectors were paid for twice —
|
||||
zero, in fact, since the first run's work was already saved.
|
||||
|
||||
**Cloud shutdown, verified rather than asserted.** Both policies detached and
|
||||
deleted; `InvokeModel` and `ListFoundationModels` both now return
|
||||
`AccessDeniedException`. No EC2 instance, no EBS volume, and — because the
|
||||
policy never granted `CreateProvisionedModelThroughput` — no way for this
|
||||
identity to create the one Bedrock resource that bills hourly.
|
||||
|
||||
Not established: retrieval quality is not acceptable for clinical use, no
|
||||
clinician-authored release gate exists, the generated evaluation questions use
|
||||
template phrasing rather than real clinical language, and no LLM answer layer
|
||||
has ever run against real evidence.
|
||||
|
||||
## 2026-08-04 — Chunk schema v4 passes the embedding-readiness gate
|
||||
|
||||
Reviewed the live Claude coordination and its last changes before editing. The
|
||||
delivery plan was objectively stale: it still described schema v2/15,076 chunks,
|
||||
empty embed/load/API modules, embedding before content-safety gates, and allowed
|
||||
unverified inferred table headers as retrieval text. The plan and ADR 0004/0006
|
||||
now put content safety, exact provenance, fail-closed schema validation and local
|
||||
pseudo-vector smoke tests before any provider call. Bedrock remains benchmark-
|
||||
only and requires separate owner approval for any paid/full-corpus run.
|
||||
|
||||
Implemented schema v4 and regenerated the canonical chunk artifact. Retrieval
|
||||
`text` may repeat route/population labels so continuation chunks remain safe in
|
||||
isolation; contiguous `source_text` remains byte-reassemblable and drives exact
|
||||
physical/printed page provenance. `context_labels` records retrieval-only
|
||||
prefixes. All 151 unverified table/formula descriptors embargo `header_row` and
|
||||
cell-like column text. Attachments now carry physical page, printed page,
|
||||
`block_id`, `bbox` and optional crop, and those region references survive the
|
||||
Qdrant adapter and RAG citation response. The loader accepts exactly schema v4,
|
||||
rejects booleans/non-integers/out-of-range pages, and keeps the normalized
|
||||
LF/CRLF-stable corpus identity.
|
||||
|
||||
Canonical artifact measured after regeneration:
|
||||
|
||||
- 15,100 chunks: 14,949 prose + 151 block descriptors;
|
||||
- 4,105,382 `cl100k_base` tokens; 0 chunks above the 800-token ceiling;
|
||||
- all `chunk-ready` gates pass: exact provenance, source uniqueness,
|
||||
reassembly, attachment coverage, descriptor embargo and schema checks all
|
||||
have 0 failures; 151 descriptors match 151 quarantined blocks;
|
||||
- raw file SHA-256:
|
||||
`8dfae08ae6d9222089c5cdb4207a064fe67989f10f7552b555af0aef6331d9a1`;
|
||||
- normalized corpus SHA-256 used by the Qdrant manifest:
|
||||
`04a27166eaa255b516829f8364227e65ad700e51446b569609d18b5efd11189c`.
|
||||
|
||||
Verification actually run:
|
||||
|
||||
- ingestion: **292 passed**; focused post-lint patch: **26 passed**;
|
||||
- AI service with `RUN_INTEGRATION=1`: **25 passed**, including real local
|
||||
Qdrant, PostgreSQL and FastAPI round-trips;
|
||||
- full canonical local smoke with deterministic 4D pseudo-vectors: first and
|
||||
second loads both upserted 15,100 records and both held exactly 15,100 points;
|
||||
manifest hash matched; data and sidecar test collections were removed and
|
||||
Qdrant returned to 0 collections;
|
||||
- Ruff `F,E9,B,ARG` on the files changed for this gate: clean; `git diff
|
||||
--check`: clean (Git only reported Windows LF/CRLF conversion warnings).
|
||||
|
||||
Conclusion: the canonical corpus is **technically READY TO EMBED**, meaning its
|
||||
input/schema/provenance/load plumbing meets the measured gates. This does not
|
||||
authorize a provider call, does not establish retrieval quality for any model,
|
||||
and does not prove whole-book medical accuracy. Human-reviewed clinical eval,
|
||||
table reconstruction, and recall for borderless tables/bar-less formulas remain
|
||||
outside what these gates prove.
|
||||
|
||||
## 2026-08-04 — Real local datastore plumbing, guarded RAG API, and printed-page citations
|
||||
|
||||
Read the live Claude Code process and coordination before editing. Claude owned
|
||||
`ingestion/load/` and `embed/cache.py`; it completed the disk cache, Qdrant
|
||||
port/adapter, idempotent UUID5 upsert, payload indexes and corpus-SHA manifest.
|
||||
Its real local Qdrant scale check loaded all 15,066 chunk records twice with
|
||||
1,024-dimensional deterministic pseudo-vectors and held the point count at
|
||||
15,066. Those vectors are not embeddings and establish no retrieval-quality
|
||||
claim. No Bedrock call, IAM change, or cloud spend occurred.
|
||||
|
||||
Built the first runnable `apps/ai-service` boundary: FastAPI `/health` and
|
||||
`POST /v1/rag/query`, a Qdrant retriever filtered by resolved `drug_id`, a
|
||||
PostgreSQL trace repository plus migration, structured human/non-human scope
|
||||
and fact/recommendation intent gates, parent hydration, quarantine handling,
|
||||
and an extractive answer layer. The answer layer refuses evidence that has only
|
||||
a physical page; citations expose only the printed folio, chunk id and optional
|
||||
source crop. Quarantined tables/formulas return a PDF-verification warning and
|
||||
never auto-extract numeric content.
|
||||
|
||||
Fixed the missing provenance at its source. Chunk schema is now v3 and
|
||||
`cli chunk` reads the real folio map from the 1,668-page PDF. It refuses a
|
||||
monograph whose physical range cannot be mapped, and `chunk-ready` has a new
|
||||
`chunk_without_printed_page_range` gate. Regenerated scope: 684 monographs,
|
||||
15,066 chunks (14,915 prose + 151 descriptors), zero oversized, and
|
||||
15,066/15,066 records with a two-value printed-page range. New artifact SHA:
|
||||
`e474c83790b450d3262f532e81abf6526a485e3a98e376413247da23f4619c38`.
|
||||
|
||||
Verification actually run:
|
||||
|
||||
- `python -m pytest -q` and Ruff over `ingestion/`: **258 passed**, lint clean;
|
||||
- `python -m ingestion.cli chunk-ready`: every gate passed, including printed
|
||||
page range 0/0 failures;
|
||||
- ai-service with `RUN_INTEGRATION=1`: **22 passed**, including a real chunk
|
||||
round-trip through local Qdrant, PostgreSQL migration/insert/read-back, and a
|
||||
full FastAPI → Qdrant → guarded citation → PostgreSQL trace round-trip;
|
||||
- local Docker services: PostgreSQL 16 and Qdrant 1.18.3 reachable; integration
|
||||
collections were UUID-scoped and removed after tests;
|
||||
- ArgoCD local: namespace, CRD and seven controller pods are running; the
|
||||
existing unrelated `guestbook` lab app is Synced/Healthy with four history
|
||||
entries. This repo's three Application YAML files parse and point to
|
||||
`master`/the Helm chart, but they are not installed and the chart still has
|
||||
no workload templates, so project sync/rollback was not performed.
|
||||
|
||||
Still open: no real embedding exists, no full canonical Qdrant collection can
|
||||
serve semantic search, `population_tags` are absent, no clinician-authored
|
||||
release-gate cases exist, and the API currently has no production answer/query
|
||||
embedding provider. The local hashing provider is explicitly plumbing-only.
|
||||
|
||||
## 2026-08-04 — Load stage built and proven against a real Qdrant; bbox rounding found
|
||||
|
||||
`ingestion/load/` was a 0-byte `__init__.py`. It now holds the vector-store
|
||||
boundary: a `VectorStore` port, an `InMemoryVectorStore` that is the reference
|
||||
implementation of its contract, and `QdrantVectorStore` as the only module that
|
||||
names `qdrant_client` — imported lazily, the same arrangement that confines
|
||||
boto3 to `bedrock_runtime`. `embed/cache.py` was added alongside it.
|
||||
|
||||
Three design decisions are worth carrying forward.
|
||||
|
||||
The cache key is `(model_id, input_kind, text_sha256)`, not `chunk_id` as
|
||||
§4.A of the delivery plan proposed. Measured reason: `chunks.jsonl` holds
|
||||
15,066 records but only **14,869 distinct texts**, so 197 records (1.31%) are
|
||||
repeats that a chunk-keyed cache would pay for twice. The content key also
|
||||
cannot serve a stale vector after an edit — a changed text is a changed digest,
|
||||
so it is a miss.
|
||||
|
||||
Point ids are `uuid5(chunk_id)`. A random id would make a re-run append a
|
||||
second copy of a dose and nothing would report an error.
|
||||
|
||||
The corpus manifest lives in a `<name>__manifest` sidecar collection rather
|
||||
than a reserved point inside the data collection, because
|
||||
`qdrant_point_count != chunk_count` is a v1 gate and a gate needing an
|
||||
"except the manifest" footnote will eventually be read wrong.
|
||||
|
||||
**Whole-corpus check against a real server.** A local Qdrant **1.18.3** was
|
||||
started from `infra/docker/docker-compose.yml` (local container, no cloud) and
|
||||
all 15,066 real chunk records were loaded with deterministic pseudo-vectors at
|
||||
1,024 dimensions — a check of the loading mechanism, **not embeddings, which
|
||||
still do not exist**. Corpus sha256 `30d5154273e0959a…`. First load: 15,066
|
||||
points in 59 batches, 14.0s, point-count gate PASS. Second load: still 15,066,
|
||||
so idempotency holds at real scale, not only against the fake store.
|
||||
|
||||
**That sha is already stale, which is the point.** `chunks.jsonl` was
|
||||
regenerated at 09:53 the same day — `chunker.py` changed two minutes earlier
|
||||
and every chunk gained `printed_page_range`, 18,229,918 → 18,753,003 bytes,
|
||||
sha now `e474c83790b450d3…`. Re-measured on the new artifact: still **15,066
|
||||
chunks, 0 over the 800-token ceiling** (largest exactly 800), all 15,066
|
||||
carrying `printed_page_range`, 14,915 prose + 151 block descriptors, 197
|
||||
duplicate texts (1.31%) unchanged because only a field was added. Suite
|
||||
**258 passed**. Had the old corpus been embedded and loaded, then the new one
|
||||
loaded into the same collection, two generations would have mixed with no error
|
||||
at query time — A6 is what refuses that, and it now has a real instance rather
|
||||
than a hypothetical one.
|
||||
|
||||
**A sampled check passed and was wrong.** Comparing 5 payloads gave 5/5
|
||||
identical. Scrolling the entire collection instead found **86 of 15,066 chunks**
|
||||
whose payload did not equal its source record. Classifying every differing leaf:
|
||||
**96 differences, all floats, all inside `attachments[].bbox`, maximum absolute
|
||||
delta 5.684e-14**, and **zero** non-float differences — every text, id, page
|
||||
number, page range, token count and boolean round-tripped exactly. A PDF point
|
||||
is 1/72 inch, so that delta cannot move a rendered crop. It is pinned by a
|
||||
regression test that fails if the loss reaches another field or grows past 1e-9.
|
||||
|
||||
The layer responsible was isolated rather than assumed: the source
|
||||
`chunks.jsonl` returns the value exactly, our own `json.dumps`/`loads` returns
|
||||
it exactly, and **Qdrant reached over raw HTTP with no SDK involved** returns it
|
||||
one ULP low. Nothing needs re-chunking — a regenerated corpus would carry the
|
||||
identical value and be rounded identically. Qdrant also stores dense vectors as
|
||||
float32, so precision beyond f32 is discarded at load regardless.
|
||||
|
||||
Cache format was decided on measurements, not preference: 300 real chunk texts
|
||||
at 1,024 dimensions cost **21,098 bytes/record — ~318 MB per model** for the
|
||||
corpus, with a **7.8s** offset-index rebuild per open. float32 `.npy` (62 MB)
|
||||
and base64 float32 in JSONL (~87 MB) were measured and set aside; append-only
|
||||
JSONL survives an interrupted run and stays readable, which outweighs disk at
|
||||
one or two models. Revisit at three (~950 MB). It lands in
|
||||
`ingestion/data/processed/`, already excluded by `.gitignore:34`.
|
||||
|
||||
**A gap in this work, found and closed the same day.** Payload indexes were
|
||||
created on `drug_id`, `section_key`, `atc_codes` and `chunk_kind` and reported
|
||||
as done — but `VectorStore` had no query method, so all that was really proven
|
||||
is that `create_payload_index` returns without raising. Filtered retrieval is
|
||||
the whole of mode A. `find_by_payload` now exists on the port and both stores,
|
||||
as a `scroll` rather than a `search`: it returns **every** match, never a
|
||||
top-k, because "return the whole section" is the plan's non-negotiable — two of
|
||||
five contraindications reads as a complete list. Verified on a real server: all
|
||||
five parts returned with no leak from the PANTOPRAZOL/OMEPRAZOL pair that
|
||||
measures cosine 1.000 on contraindications; a deliberately 300-part section
|
||||
(above the 256 scroll page) comes back whole so paging cannot truncate; and a
|
||||
real multi-part section from `chunks.jsonl` round-trips to exactly its own
|
||||
chunk ids.
|
||||
|
||||
Tests: **255 passed** with Qdrant running (206 before this work, +49);
|
||||
**247 passed, 8 skipped** with it stopped, so an offline machine and CI see
|
||||
skips rather than failures. After the mode A work and the other worktree's
|
||||
`cli.py` fix the suite stands at **268 passed** and
|
||||
`ruff --select F,E9,B,ARG` reports **no findings at all** across `ingestion/`.
|
||||
|
||||
Still missing, and deliberately so: `printed_page_range` and `population_tags`
|
||||
are not in the payload (open questions to Codex in
|
||||
`coordination/CLAUDE_TASK_2026-08-04.md`); `cli embed` / `cli load` are not
|
||||
wired because `cli.py` is Codex's; and **no real embedding vector has ever been
|
||||
produced** — every vector the load path has carried was synthetic. The Bedrock
|
||||
request shapes remain documentation-derived and unproven.
|
||||
|
||||
Measured cost: **$0**. No Bedrock call, no IAM change, no cloud resource.
|
||||
|
||||
## 2026-08-03 — Bedrock embedding boundary built; IAM diagnosed, not yet opened
|
||||
|
||||
`ingestion/embed/` was an empty `__init__.py`. It now holds the provider
|
||||
boundary the model benchmark needs: an `EmbeddingProvider` ABC that owns input
|
||||
validation, request-size batching and timing, and three adapters behind it —
|
||||
`amazon.titan-embed-text-v2:0`, `cohere.embed-v4:0`, and `BAAI/bge-m3` as the
|
||||
zero-cost local control. boto3 is named in exactly one module and imported
|
||||
lazily, so the package imports and the whole suite runs with no AWS account.
|
||||
|
||||
Two design points are worth carrying forward. `input_kind` is a required
|
||||
argument, not a keyword: Cohere embeds corpus records and queries into
|
||||
different subspaces, and sending `search_document` for a query raises no error
|
||||
— recall just drops. And `normalized` is three-valued. Titan is asked to
|
||||
normalize and says so; the Bedrock docs never state whether Cohere's float
|
||||
vectors are unit-length, so that field stays `None` instead of guessing, and
|
||||
`embed.probe` prints a *measured* L2 norm to settle it on the first live call.
|
||||
|
||||
The AWS side is diagnosed and stuck. `ai-lab-user` has no inline and no
|
||||
attached user policy; its one group (`AI-Lab-Group`) grants EC2, IAM, ELB and
|
||||
VPC full access and nothing else. There is no `bedrock:*` grant anywhere on
|
||||
the identity — confirmed by running both `list-foundation-models` and
|
||||
`invoke-model` and reading the two `AccessDeniedException` messages. Two
|
||||
least-privilege policies are drafted in `infra/aws/iam/` but **deliberately
|
||||
not applied**: that identity carries `IAMFullAccess` and could attach them
|
||||
itself, which is exactly why it was left to a human.
|
||||
|
||||
Consequence: every request-body shape in the two Bedrock adapters is derived
|
||||
from the AWS user guide (read today) and **has never been accepted by the
|
||||
service**. That is unproven, not verified. Tests: 22 new, all with a stub
|
||||
invoker and zero network; **203 passed** overall, up from 181. Lint clean on
|
||||
every file added (`--select F,E9,B,ARG`); the one remaining finding is a
|
||||
pre-existing `cli.py` import owned by the other worktree.
|
||||
|
||||
Measured cost so far: **$0**. Nothing was embedded, nothing reached Qdrant.
|
||||
|
||||
## 2026-08-03 — Exact hard-10 gate and all-block table chunking experiment
|
||||
|
||||
Extended the isolated table/formula sandbox beyond the 100-page sample. An
|
||||
exact ten-block risk gate covered four cross-page pairs, a merged header, a
|
||||
fragmented fraction bar, and the bar-less ADENOSIN formula; all ten source crops
|
||||
were visually checked. The full run then processed all 151 canonical blocks:
|
||||
141 physical tables, ten formulas, 133 logical table parents, 669 row children,
|
||||
and seven cross-page logical tables.
|
||||
|
||||
Full-scope visual inspection exposed a continuation bug: FAMCICLOVIR p647 and
|
||||
INSULIN p811 repeat their column headers, while other continuation pages start
|
||||
directly with data. The linker now distinguishes these cases; repeated headers
|
||||
are not emitted as data, and INSULIN's changed `Phối hợp` first-column meaning
|
||||
is preserved. Both branches have regressions.
|
||||
|
||||
The expanded, source-derived retrieval suite contains 2,436 cases. With drug
|
||||
and table/formula lane resolved before ranking, deterministic hybrid character
|
||||
TF-IDF measured 94.42% Recall@1, 99.79% Recall@5, and 96.90% MRR. Row questions
|
||||
were 94.82% / 100%; formula questions 100% / 100%. Five ambiguous whole-table
|
||||
questions fell below top five because the same drug owns several near-identical
|
||||
tables; production must clarify or route using an additional table anchor.
|
||||
Neural MiniLM is now opt-in and excluded from the default parsing gate.
|
||||
|
||||
Measured chunk design: table-parent tokens min/median/p90/p95/max =
|
||||
66/188/441/678/1,893; only four of 133 parents exceed 800. Row children are
|
||||
75-token median, 172 p95, 471 max. Keep every logical parent intact, index both
|
||||
parent and header-aware rows, never split a row, and hydrate row hits to the
|
||||
complete parent/source pages. Final checks: **181 tests passed**, readiness
|
||||
20/20, lint clean.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-03 — 100-page table/formula reconstruction and RAG sandbox
|
||||
|
||||
Built an isolated experiment under `ingestion/scratch/rag-table-pilot` without
|
||||
writing sandbox representations into the canonical corpus. The risk-stratified
|
||||
100-page run reconstructed 120 tables and 10 formula regions, rendered and
|
||||
manually inspected all 130 crops, and linked four tables continued across page
|
||||
pairs 132-133, 646-647, 825-826, and 1373-1374.
|
||||
|
||||
The retrieval router fixes the drug and data lane before vector ranking. On 461
|
||||
source-derived queries, hybrid row+whole character TF-IDF reached 92.62%
|
||||
Recall@1, 98.70% Recall@5, and 95.04% MRR. Cached English-oriented MiniLM was
|
||||
worse (88.29% / 97.18% / 91.92%). Eighteen row-hit answer previews all hydrated
|
||||
to the complete parent Markdown table; eight included both pages of a continued
|
||||
table. A narrow deterministic interval probe passed 172/172 generated cases;
|
||||
this is a mechanics check, not clinical ground truth.
|
||||
|
||||
Visual review exposed one canonical defect: ADENOSIN p147's bar-less printed
|
||||
formula region ended after its numerator and omitted `Nồng độ adenosin
|
||||
(3 mg/ml).` The bar-less band now extends 31pt below its synthetic anchor,
|
||||
capturing the denominator but stopping before `Ví dụ:`; a regression pins that
|
||||
boundary. Canonical artifacts were regenerated after the fix: 684 monographs,
|
||||
11,974 sections, 15,066 chunks, 151 descriptors, 0 unassigned spans, all 20
|
||||
readiness gates passing, **180 tests passed**, and lint clean.
|
||||
|
||||
Decision: JSON grid + Markdown answer view, row and whole-table retrieval, and
|
||||
mandatory parent hydration are viable for the next stage. This remains a
|
||||
retrieval experiment, not production clinical approval; merged-cell semantics,
|
||||
unit/multi-axis reasoning, Vietnamese embedding comparison, borderless/bar-less
|
||||
recall, clinician-authored evals, and final expert review remain open.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-03 — Whole-corpus parser repair after manual baseline audit
|
||||
|
||||
Implemented and re-ran the parser over all 1,668 pages after manually reading
|
||||
the high-risk baseline outliers. The fixes are structural, with regressions:
|
||||
|
||||
- restored the missing `THUỐC TƯƠNG TỰ HORMON GIẢI PHÓNG GONADOTROPIN`
|
||||
boundary (`Tên chung quốc tế và mã ATC` is its real first anchor), separating
|
||||
pages 1371–1373 from `THUỐC PHIỆN - OPIAT - OPIOID`;
|
||||
- require both physical and inferred printed page bounds, so back-index page
|
||||
1655 can no longer extend ZOLPIDEM's real `[1492, 1494]` range;
|
||||
- keep plain label-shaped text as body when it is an adjacent wrapped
|
||||
continuation in the same PDF block (including NADROPARIN's “không phải là
|
||||
chống chỉ định”);
|
||||
- classify known table cells before headings, putting WARFARIN and IOBITRIDOL
|
||||
dosing tables back under `lieu_luong_va_cach_dung`;
|
||||
- added confirmed heading variants for CLORPHENIRAMIN dosage forms and tetanus
|
||||
toxoid dosing, and real provenance for combined inline fields;
|
||||
- made verified formula bands column-aware: NETILMICIN opposite-column prose
|
||||
is retained while AMPICILIN's gutter-adjacent formula stays quarantined;
|
||||
- visually inspected all **151/151 unique table/formula regions** against the
|
||||
rendered PDF; every region is genuinely 2D and remains quarantined;
|
||||
- emit every physical table/formula region atomically at its first stream
|
||||
occurrence, fixing split/contradictory ownership on CAPECITABIN, IMATINIB,
|
||||
CARBOPLATIN, NETILMICIN, and TRASTUZUMAB;
|
||||
- route explicit `Bảng N. Điều chỉnh liều ...` appendices back to dosage even
|
||||
when the book prints them after `Tên thương mại` (CAPECITABIN p309);
|
||||
- added readiness gates for every individual section part's source-span IDs
|
||||
and duplicate physical-region IDs.
|
||||
|
||||
Final regenerated artifacts and evidence:
|
||||
|
||||
| check | result |
|
||||
|---|---|
|
||||
| tests | **180 passed**; lint clean |
|
||||
| segmentation | **684 monographs**, 11,974 sections, 8,213,036 prose chars |
|
||||
| back-index validation | **96.2% recall (678/705), 99.1% precision** |
|
||||
| quarantined regions | **151 blocks / 151 unique IDs**, all visually checked |
|
||||
| chunks | **15,066** (14,915 prose + 151 block descriptors), 0 over 800 tokens |
|
||||
| chunk readiness | **20/20 PASS** (including duplicate-region prevention) |
|
||||
| coverage | 252,799 spans, **0 unassigned** across all 1,668 pages |
|
||||
| residual ink | 3,931 classified regions, **0 unclassified** across all pages |
|
||||
|
||||
Canonical `ingestion/data/processed/{monographs,chunks,coverage_ledger}` were
|
||||
regenerated. Remaining limits: no whole-document human-reviewed clinical
|
||||
ground truth, no row/column reconstruction for quarantined tables, and unknown
|
||||
recall for borderless tables/bar-less formulas. This is ready for retrieval
|
||||
experiments, not a claim of production clinical approval.
|
||||
|
||||
---
|
||||
|
||||
Chronological record of work done on this project, newest entry on top. The
|
||||
goal is continuity across sessions: if a work session ends unexpectedly
|
||||
(context/token limit, interruption), whoever picks this up next — human or
|
||||
@@ -13,6 +512,300 @@ end if that risk is showing.
|
||||
|
||||
---
|
||||
|
||||
## 2026-08-03 — Independent re-verification, a redundant rule in my own uncommitted fix, a provenance defect, and the measurements a retrieval design has to be built on
|
||||
|
||||
No code was changed in this session: the four files from the previous round
|
||||
are still uncommitted and under external review (Codex). Everything below is
|
||||
measurement, and the numbers live nowhere else — the investigation scripts
|
||||
were deleted per the repo rule, so this entry is the record.
|
||||
|
||||
### 1. Re-verified the whole tree from scratch
|
||||
|
||||
| check | command | result |
|
||||
|---|---|---|
|
||||
| tests | `python -m pytest -q` | **164 passed**, 39.31s |
|
||||
| lint | `ruff check --select F,E9,B,ARG .` | clean |
|
||||
| gates | `cli chunk-ready` | **18/18 PASS**; 683 monographs, 11,966 sections, 8,212,880 chars, 167 quarantined blocks |
|
||||
| recall/precision | `cli validate --pdf …` | 683 detected, 705 ground truth, **96.0% (677/705) / 99.1%** |
|
||||
| span ledger | `cli coverage --pdf …` (re-run) | 252,799 spans after merge, 9,398,772 chars, **unassigned 0** |
|
||||
| reproducibility | `cli run` → sha256 | **byte-identical** to `monographs.jsonl` (`84f41d96…`) |
|
||||
| reproducibility | `cli chunk` → sha256 | **byte-identical** to `chunks.jsonl` (`63472db4…`); 15,076 chunks (14,909 prose + 167 descriptors), 0 oversized, 4,072,725 tokens (cl100k_base) |
|
||||
|
||||
Recall rose 92.9% → 96.0% because of the uncommitted back-index rejoin, and
|
||||
the mechanism is the denominator: 725 → 705 ground-truth entries once wrapped
|
||||
fragments stop counting as entries. The detector did not improve.
|
||||
|
||||
**Not re-run: `cli residual-ink`.** `residual_ink.json` is dated 2026-08-01
|
||||
12:03, before the 17:36 assembler edits. Its stored contents (3,931 regions,
|
||||
no `unclassified` kind) are last session's numbers, not this session's.
|
||||
|
||||
**Doc drift found:** `docs/verification-strategy.md` quotes 252,733 spans /
|
||||
177,679 `normalized_text` / 12,764 `heading`; measured today 252,799 /
|
||||
177,754 / 12,752. The `unassigned = 0` conclusion still holds.
|
||||
|
||||
### 2. The x0 geometry change in the uncommitted diff is redundant
|
||||
|
||||
Assembled the whole book four times with the two new rules toggled:
|
||||
|
||||
| variant | monographs | sections | chars |
|
||||
|---|---|---|---|
|
||||
| current (x0 + italic) | 683 | 11,966 | 8,212,880 |
|
||||
| **old x1 rule + italic** | 683 | 11,966 | **8,212,880** — 0 differences of any kind |
|
||||
| x0, no italic | 683 | 11,966 | 8,212,844 (3 sections differ) |
|
||||
| x1, no italic (= `bc01782`) | 683 | 11,966 | 8,212,780 (9 sections differ, 11,014 char delta) |
|
||||
|
||||
The italic rule alone recovers all 9 sections (CEFAZOLIN dosing 1,062 →
|
||||
5,620 chars; CALCI LACTAT `than_trong` 582 → 1,504, `tuong_tac_thuoc` 2,346 →
|
||||
1,439). The x0 rule alone recovers 6 of 9 and adds **nothing** on top of the
|
||||
italic rule.
|
||||
|
||||
Worse, the justification is wrong: the real NEVIRAPIN span on physical page
|
||||
1045 is `TimesNewRomanPS-ItalicMT` (verified by reading the span's font), so
|
||||
the italic rule is what fixes that page — not the 0.01pt overlap the code
|
||||
comment and the new test's docstring credit. The test itself is valid but
|
||||
pins the geometric rule only, because the `_span()` fixture helper never
|
||||
produces an italic font. **Either keep the x0 rule as defence-in-depth with
|
||||
an honest comment, or revert it — but the current comment overstates it.**
|
||||
|
||||
### 3. `source_page_range` is wrong for 13 of 683 monographs
|
||||
|
||||
Section-level provenance (`parts`) is correct everywhere; the monograph-level
|
||||
page range is not. 12 monographs overshoot by +1 page; **ZOLPIDEM declares
|
||||
`[1492, 1655]` while every one of its sections comes from 1492-1494** — a
|
||||
164-page claim reaching into the back index.
|
||||
|
||||
Root cause for ZOLPIDEM, confirmed: physical page 1655 (printed 1656, back
|
||||
matter) carries a **bold** span reading exactly `Tương tác thuốc`, which
|
||||
`_classify` emits as a `_SectionEvent`, and the `_SectionEvent` branch at
|
||||
`segment/assembler.py:496` updates `source_page_range[1]` with **no
|
||||
`in_monograph_range` guard** — unlike the `_TextEvent` branch at line 511.
|
||||
Verified that **0 spans past physical 1495 pass `in_monograph_range`**, so no
|
||||
text was contaminated and `empty_section` is still 0. The defect is confined
|
||||
to one provenance field.
|
||||
|
||||
The +1 cause is **not isolated** — it is not lifted tables (all 12 have
|
||||
`tables: []`); the likely candidate is a next-page boilerplate span bumping
|
||||
the range before being excluded, but that was not measured.
|
||||
|
||||
### 4. Corpus profile — what a retrieval design actually has to work with
|
||||
|
||||
- **13 of 19 fields have p90 < 1,500 chars**, i.e. the whole section fits one
|
||||
chunk. Only four routinely need splitting: `duoc_ly` (p90 4,939, max
|
||||
14,099), `lieu_luong` (4,873 / 14,197), `than_trong` (2,419), `tuong_tac`
|
||||
(2,147). Confirms ADR 0004 on the cleaned corpus.
|
||||
- **ATC**: 668/683 (97.8%) carry ≥1 code, **171 (25.0%) carry more than one**,
|
||||
max 20, 1,043 distinct codes.
|
||||
- **`ten_thuong_mai` present in 492 (72%)** monographs.
|
||||
- **The back index holds 344 `X - xem Y` lines** — brand → generic aliases —
|
||||
which `parse_back_index` currently discards wholesale (correct for
|
||||
validation, but this is the highest-value retrieval asset in the book,
|
||||
because clinicians type brand names).
|
||||
- **401 `xem [thêm] mục/chuyên luận` phrases across 261 monographs**; a chunk
|
||||
containing one is useless retrieved alone.
|
||||
- **Dosing population markers**: `Trẻ em` 53%, `Người lớn` 51%, `Người cao
|
||||
tuổi` 15%, `Trẻ sơ sinh` 8%, `Suy thận` 8%, `Suy gan` 6% of 682 dosing
|
||||
sections — real sub-section boundaries, better split points than token
|
||||
windows.
|
||||
- **167 quarantined blocks, 129 (77%) inside `lieu_luong_va_cach_dung`** —
|
||||
the most dangerous field is the one the tables were lifted out of.
|
||||
|
||||
### 5. Cross-drug confusability — the number that decides the architecture
|
||||
|
||||
First hypothesis (much repeated boilerplate across drugs) was **refuted**:
|
||||
only **171 of 11,966 sections** share exact text with another drug (1.4%), and
|
||||
the six heavy clinical fields are 100% distinct.
|
||||
|
||||
Then measured, per field, each drug's TF-IDF cosine against its *nearest other
|
||||
drug*. **This is a lexical proxy, not an embedding measure** — it bounds the
|
||||
problem from one side only.
|
||||
|
||||
| field | median | p90 | p99 | max | drugs with NN > 0.7 |
|
||||
|---|---|---|---|---|---|
|
||||
| `lieu_luong_va_cach_dung` | 0.314 | 0.455 | 0.631 | 0.836 | 4 (0.6%) |
|
||||
| `tuong_tac_thuoc` | 0.284 | 0.461 | 0.870 | 0.984 | 18 (2.8%) |
|
||||
| `tac_dung_khong_mong_muon` | 0.300 | 0.437 | 0.856 | 1.000 | 13 (1.9%) |
|
||||
| `chi_dinh` | 0.408 | 0.637 | 0.885 | 0.924 | 31 (4.5%) |
|
||||
| `chong_chi_dinh` | 0.346 | 0.633 | 0.898 | **1.000** | 37 (5.4%) |
|
||||
|
||||
Named pairs: `PANTOPRAZOL ↔ OMEPRAZOL` (contraindications **1.000**,
|
||||
indications 0.913) · `BENZATHIN PENICILIN G ↔ PHENOXYMETHYLPENICILIN`
|
||||
(contraindications **1.000**) · `DIGOXIN ↔ DIGITOXIN` (0.891 / 0.911) ·
|
||||
`NATRI NITRIT ↔ NATRI THIOSULFAT` (dosing 0.631 — two different steps of the
|
||||
same cyanide-antidote protocol) · `IOBITRIDOL ↔ ACID IOXAGLIC` (0.984) ·
|
||||
`ESTRIOL ↔ ESTRON` · `GLICLAZID ↔ GLIMEPIRID` · `NAPHAZOLIN ↔ OXYMETAZOLIN`.
|
||||
|
||||
Name layer: **19 drug names are a substring of another drug name**
|
||||
(`CLOROTHIAZID` in `HYDROCLOROTHIAZID`, `EPHEDRIN` in `PSEUDOEPHEDRIN`,
|
||||
`LORATADIN` in `DESLORATADIN`, `ATROPIN` in `HOMATROPIN HYDROBROMID` — all
|
||||
genuinely different drugs), and 106 of 683 names share a 6-character prefix
|
||||
across 38 clusters.
|
||||
|
||||
**Conclusion drawn from this, for the retrieval design: vector similarity must
|
||||
never be allowed to choose the *drug* — only which passage within an
|
||||
already-resolved drug.** The dangerous confusions are concentrated in a
|
||||
small, enumerable set of same-class pairs, which is exactly the population
|
||||
this project's verification strategy says to census rather than sample.
|
||||
|
||||
### Not done yet / next up
|
||||
|
||||
Sequenced in **`docs/v1-delivery-plan.md`** (written this session): a
|
||||
two-week plan to a running v1, scoped down to two deployables (`web` +
|
||||
`ai-service`) because the four NestJS services measure 0 `.ts` files each.
|
||||
The items below are the ones that plan depends on.
|
||||
|
||||
- The confusable-pair census must become a **committed fixture produced by
|
||||
production code** (`ingestion/validation/`), not a deleted scratch script.
|
||||
Until then these numbers are only in this entry.
|
||||
- ADR 0007 (retrieval architecture) not written. Proposed content: vectors
|
||||
never pick the drug; the unit returned to the LLM is the **complete
|
||||
section** (enabled by `section_not_reassemblable_from_chunks = 0`, because a
|
||||
partial contraindication list reads as "no contraindication"); and eval
|
||||
split in two — **routing** correctness (ground truth derivable from the
|
||||
corpus itself, 683 × 19 pairs, no human needed) versus **content**
|
||||
correctness (requires a clinician; cannot be self-generated without
|
||||
fabricating evidence).
|
||||
- Entity/alias layer (683 canonical names + 344 back-index aliases + 492
|
||||
`ten_thuong_mai` + 1,043 ATC codes) — zero-regret, needed by every
|
||||
architecture, must use longest-exact-match because of the 19 substring
|
||||
traps.
|
||||
- `residual-ink` re-run; `verification-strategy.md` numbers re-synced;
|
||||
regression test for `parse_back_index` (still has none); the
|
||||
`source_page_range` guard; the x0-rule comment decision.
|
||||
- Open question for the user, not a technical one: this is the **2018
|
||||
edition**; the 3rd edition (2022) exists. For a document with legal force
|
||||
over prescribing, staying on 2018 should be a deliberate decision, and it
|
||||
makes edition-independence a real requirement for the pipeline.
|
||||
- Still untouched: `embed/`, `load/`, Qdrant, `ai-service`, and the general
|
||||
chapters (printed 37-98) and appendices (printed 1497-1528), which remain
|
||||
outside the corpus entirely.
|
||||
|
||||
## 2026-08-01 (cont'd, 7) — "still errors?" — yes: two more real content-loss bugs, both in dosing sections
|
||||
|
||||
Asked whether errors remained after the previous round, the honest answer was
|
||||
that this session has found real defects every time it looked one level
|
||||
deeper. It looked again, and found two more.
|
||||
|
||||
**1. Chunks ended on a bare population label, with the dose in the next
|
||||
chunk.** `split_sentences` treats `:` as a sentence boundary and
|
||||
`_OPENS_SENTENCE` accepts a digit, so `"Người lớn: 500 mg mỗi 8 giờ."` splits
|
||||
after the colon. When the packer flushed at that point, the chunk ended on the
|
||||
label. Measured: **38 prose chunks**, e.g. AMOXICILIN's ending on a Lyme
|
||||
indication followed by a bare `Người lớn:`. Retrieval on that chunk returns a
|
||||
population with no dose; on the next, a dose with no population. Outlier item
|
||||
17 counted population markers on 1,121 of ~1,400 monograph pages, so this is
|
||||
the common shape, not an edge case. The packer now carries trailing label
|
||||
atoms into the next part instead of flushing on them: **38 → 2**, and chunks
|
||||
ending on any colon **721 → 19**.
|
||||
|
||||
**2. A section name printed mid-line was swallowed as a heading — real text
|
||||
loss, in dosing sections.** Chasing the last 2 of those 38 showed the defect
|
||||
was not in chunking at all. CISPLATIN (physical page 402) prints
|
||||
`Suy thận: Chống chỉ định.` inside `liều lượng và cách dùng`; the second half
|
||||
is itself a section name, so it was matched as a heading. The result: the
|
||||
renal-impairment contraindication **disappeared from the dosing text** and the
|
||||
section ended on a bare `Suy thận:`. ISOPRENALIN had the same shape. Same
|
||||
family as the FLUOROURACIL bug fixed earlier today, but that rule only covered
|
||||
a label directly *under* a heading and could not see this one.
|
||||
|
||||
Fixed geometrically: a real section heading opens its line, so a non-bold
|
||||
section name with another span printed to its left is body text. "To the left"
|
||||
is checked properly — same page/block/line *and* `previous.x1 <= span.x0` —
|
||||
because the synthetic test fixtures place every span at identical coordinates,
|
||||
and a looser check passed on real data while breaking the AMITRIPTYLIN
|
||||
inline-value case.
|
||||
|
||||
Verified after the fix: CISPLATIN's dosing section contains
|
||||
`Suy thận: Chống chỉ định.` again, ISOPRENALIN's `Trẻ em:` is followed by its
|
||||
doses, and `chong_chi_dinh` is no longer polluted. Monograph and section counts
|
||||
unchanged at 683 / 11,966 — nothing was traded away for the recovery.
|
||||
|
||||
**State:** 18/18 gates pass, **163 tests** (was 161), ruff F/E9/B/ARG clean,
|
||||
15,077 chunks with 0 over the ceiling, 8,212,780 section characters.
|
||||
|
||||
**Standing conclusion, worth writing down:** every round of "is it clean now?"
|
||||
this session has ended with real defects found — five in the previous round,
|
||||
two in this one, and four of the previous five were in code written the same
|
||||
day. The gates and tests prove what those instruments can see. They do not
|
||||
prove the corpus is correct, and the largest unmeasured area is unchanged:
|
||||
content accuracy against the source, with no human-reviewed ground truth for
|
||||
8.2M characters.
|
||||
|
||||
## 2026-08-01 (cont'd, 6) — Bug hunt after declaring "clean": the token count was wrong by 2x, 14.7% of chunks were over the ceiling, and two stage boundaries measured different pipelines
|
||||
|
||||
I had just reported the tree as clean. It was not. Going looking properly
|
||||
found five real defects, four of them in code written earlier the same day.
|
||||
|
||||
**1. `estimate_tokens` was wrong by a factor of two, and the number it
|
||||
produced was reported.** ADR 0004 sized chunks with `len(text) // 4`,
|
||||
described honestly as an estimate. Measured against `cl100k_base` on the real
|
||||
corpus:
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| estimate (chars/4) | 2,115,427 tokens |
|
||||
| real tokenizer | **4,093,440 tokens** |
|
||||
| real/estimate | median **1.95**, p95 2.50, max **6.0** |
|
||||
| oversized by estimate | **0** |
|
||||
| oversized in fact | **1,884 of 12,838 = 14.7%**, largest 1,645 tokens |
|
||||
|
||||
Vietnamese diacritics cost several byte-pair tokens each. "0 oversized" was
|
||||
reassuring and false. `chunk/tokens.py` now counts with the real tokenizer,
|
||||
injected so the chunking logic stays testable without it, with a fallback of
|
||||
chars/2 that errs small rather than large.
|
||||
|
||||
**2. The packer could exceed the ceiling on its own.** Two causes, both
|
||||
measured on VORICONAZOL's `tương tác thuốc`: an atom of 710 tokens was left
|
||||
whole because it was under the 800 ceiling, and the overlap builder added
|
||||
whole atoms until the running total *passed* the budget, so a 251-token atom
|
||||
produced a 273-token overlap against a 65-token setting. 273 + 710 = 983.
|
||||
Atoms are now split against the 650 target, leaving room for overlap, and the
|
||||
overlap stops *before* exceeding its budget.
|
||||
|
||||
**3. An over-long comma list was left as one atom.** VORICONAZOL's
|
||||
interaction list is one "sentence" hundreds of drug names long. Truncated by
|
||||
an embedding model it reads as "this drug is not listed" — a false negative
|
||||
in the direction that matters. Split at commas, which is lossless for a list.
|
||||
|
||||
After 1-3: **0 chunks over the ceiling**, verified by an independent tiktoken
|
||||
re-count of the written file, not by the pipeline's own number. 15,049 chunks
|
||||
(was 12,838 — the rise is real sub-chunking that should have happened all
|
||||
along).
|
||||
|
||||
**4. `cli validate` measured a different pipeline than `cli run`.** It used
|
||||
the raw span stream (no transcription repair) and passed no table regions, so
|
||||
recall/precision described a build that is not the one producing the output —
|
||||
the same class of mismatch already fixed for `coverage`. Now shares
|
||||
`_extracted_and_repaired_spans` and `_region_index`. Result after the fix is
|
||||
unchanged at 92.9% / 99.1%.
|
||||
|
||||
**5. `chunk/io.py` dropped `SectionPart` when reading monographs back**, so
|
||||
per-part provenance died at the stage boundary — against CLAUDE.md's explicit
|
||||
rule. Now carried: 12,290 parts across 11,966 sections.
|
||||
|
||||
**Two new gates, and the gate itself was wrong twice before the data was.**
|
||||
`section_not_reassemblable_from_chunks` rebuilds each section from its own
|
||||
chunks by removing the deliberate overlap and compares. First version joined
|
||||
chunk texts with a newline and reported **734** sections missing — the first
|
||||
one it named was present. Second version probed a 60-character head and
|
||||
reported **1**, NAPROXEN, where the probe straddled an overlap seam that
|
||||
legitimately repeats text. The working version compares with whitespace
|
||||
removed, because each split seam loses exactly one space to `.strip()`
|
||||
(measured on ABACAVIR: two single spaces in a 4,232-character section,
|
||||
nothing else). It proves no character of content is lost, reordered or
|
||||
duplicated beyond the intended overlap. **0.**
|
||||
|
||||
**Also fixed:** all 8 real lint findings (`ruff --select F,E9,B,ARG`) — five
|
||||
unused imports and three `zip()` calls without explicit `strict=`. The zips
|
||||
were the adjacent-pair idiom and not bugs; `strict=False` now says so. And
|
||||
the transcription splice could leave a fragment holding only a space, which
|
||||
showed up as two `whitespace_only` spans; dropped, and proven inert by the
|
||||
sha256 over every section's text being byte-identical before and after
|
||||
(`6af13301…`).
|
||||
|
||||
**State after the hunt:** 18/18 gates pass (10 corpus + 8 chunk), 161 tests
|
||||
(was 158), `ruff F/E9/B/ARG` clean, `unassigned = 0`, `cli validate` 92.9% /
|
||||
99.1%, 15,049 chunks with 0 over the ceiling.
|
||||
|
||||
## 2026-08-01 (cont'd, 5) — ADR 0006 implemented: chunks now reference their lifted blocks; `chunk/` runs for the first time; 16/16 gates green
|
||||
|
||||
**Why this was needed, in one line**: a chunk of a section whose table had
|
||||
|
||||
@@ -0,0 +1,366 @@
|
||||
# Kế hoạch giao bản v1 chạy được — 2 tuần
|
||||
|
||||
**Lập ngày 2026-08-03. Hạn: ~2026-08-17.**
|
||||
|
||||
## Cập nhật bắt buộc 2026-08-04 — gate trước embedding
|
||||
|
||||
Phần hiện trạng ngày 2026-08-03 bên dưới được giữ làm lịch sử, nhưng không còn
|
||||
được dùng để quyết định chạy embedding. Candidate schema v4 đã được tạo và đo
|
||||
trên toàn corpus: **15.100 chunks** (14.949 prose + 151 block descriptor),
|
||||
4.105.382 token `cl100k_base`, 0 chunk quá 800 token. Candidate chưa phải artifact
|
||||
canonical cho đến khi vượt toàn bộ gate và thay thế `data/processed/chunks.jsonl`.
|
||||
|
||||
Thứ tự bắt buộc từ đây:
|
||||
|
||||
1. khóa an toàn nội dung: label/liều không tách rời; `source_text` ghép lại đúng
|
||||
section; toàn bộ header bảng chưa kiểm chứng bị embargo khỏi text embedding;
|
||||
2. khóa provenance: range vật lý và range trang in phải chính xác theo từng chunk;
|
||||
attachment phải mang `block_id`, `bbox`, trang vật lý và trang in;
|
||||
3. khóa consumer: loader chỉ nhận đúng schema v4, từ chối schema cũ/mới và metadata
|
||||
sai kiểu hoặc sai miền;
|
||||
4. chạy test + `chunk-ready` trên candidate; chỉ khi mọi gate bằng 0 mới tái sinh
|
||||
artifact canonical và ghi SHA-256;
|
||||
5. smoke-test local bằng vector giả để kiểm plumbing/idempotency; xóa collection test;
|
||||
6. **chỉ sau phê duyệt riêng của chủ dự án** mới gọi provider có chi phí hoặc chạy
|
||||
embedding toàn corpus. Bedrock chỉ dùng để tìm hiểu/benchmark, không phải runtime
|
||||
dependency.
|
||||
|
||||
Định nghĩa **READY TO EMBED**: canonical là schema v4; toàn bộ readiness gate bằng
|
||||
0; test ingestion và AI service liên quan đều pass; SHA corpus đã ghi; Qdrant không
|
||||
còn collection test; không có header/cell chưa kiểm chứng trong embedding text.
|
||||
Trạng thái này chỉ cho phép bước chuẩn bị kỹ thuật, không tự động cấp phép phát sinh
|
||||
chi phí.
|
||||
|
||||
Quy ước của tài liệu này, theo đúng luật trong `CLAUDE.md`:
|
||||
|
||||
- **(đo)** = đã chạy thật trong phiên 2026-08-03, lệnh và kết quả ghi trong
|
||||
`docs/progress-log.md`.
|
||||
- **(ước lượng)** = phỏng đoán, chưa đo, có thể sai. Mọi con số thời gian
|
||||
trong tài liệu này đều là ước lượng — không có ngoại lệ.
|
||||
- `[chờ xác nhận]` = phụ thuộc quyết định của người chủ dự án, không được tự
|
||||
chọn thay.
|
||||
|
||||
Ước lượng thời gian giả định **1 người, ~6 giờ làm việc hiệu quả/ngày, 10
|
||||
ngày công**. Nếu thực tế là bán thời gian thì mục §8 (ngoài phạm vi) phải
|
||||
dài thêm, chứ không phải ép các mục còn lại chạy nhanh hơn.
|
||||
|
||||
---
|
||||
|
||||
## 0. Hiện trạng — đo, không phải nhớ
|
||||
|
||||
| Thành phần | Trạng thái |
|
||||
|---|---|
|
||||
| `ingestion/` extract → segment → chunk | Baseline 2026-08-03 đã hoàn thành; candidate schema v4 ngày 2026-08-04 có 15.100 chunks và đang chờ gate cuối trước khi trở thành canonical |
|
||||
| `ingestion/embed/` | Đã có provider ports, cache, local BGE-M3 và adapter Bedrock; chưa được phép chạy provider trả phí/full corpus |
|
||||
| `ingestion/load/` | Đã có validation fail-closed schema v4, manifest/hash, upsert idempotent và adapter Qdrant; còn nghiệm thu artifact canonical mới |
|
||||
| `apps/ai-service/` | Đã có FastAPI/RAG, adapter Qdrant/Postgres, guardrails và citation theo region; còn nghiệm thu tích hợp trên corpus canonical mới |
|
||||
| `apps/api-gateway`, `auth-service`, `chat-service`, `user-service` | **0 file `.ts`** mỗi service |
|
||||
| `apps/web/` | 18 file, chat UI + PDF split-view chạy được, backend là mock (`sendChatMessage` = `setTimeout(400ms)` + fixture) |
|
||||
| `packages/shared-types` | DTO `ChatMessage` / `Citation` đã có |
|
||||
| Dockerfile | **0 cái trong toàn repo** |
|
||||
| `infra/helm/medical-chatbot/templates/` | **rỗng**, chỉ có `.gitkeep`; `values.yaml` chỉ có 2 dòng comment |
|
||||
| `infra/argocd/applications/{dev,staging,prod}/app.yaml` | Có sẵn, trỏ `path: infra/helm/medical-chatbot`, `targetRevision: master`; còn 3 `TODO` (project, repoURL, destination cluster) |
|
||||
| `infra/docker/docker-compose.yml` | Chỉ có `postgres`, `qdrant`, `redis` — không có service ứng dụng |
|
||||
| CI | Chỉ có `infra/ci/github-actions/README.md` |
|
||||
| Tracing | Không có gì |
|
||||
|
||||
**Tài sản không nằm trong repo nhưng có thật**: quyền truy cập k3s của team,
|
||||
ArgoCD (admin), kubeconfig đã hoạt động. Đây là lý do phần deploy không bắt
|
||||
đầu từ số 0.
|
||||
|
||||
---
|
||||
|
||||
## 1. Phạm vi v1 — cắt gì, và vì sao đó không phải "ăn bớt"
|
||||
|
||||
### Cắt khỏi v1: `api-gateway`, `auth-service`, `user-service`, `chat-service`
|
||||
|
||||
Bốn service này cộng lại đang là **0 dòng code (đo)**. Viết cả bốn bằng
|
||||
NestJS trong 2 tuần, song song với mọi việc khác, là thứ giết deadline — và
|
||||
không service nào trong bốn cái đó **thêm năng lực** cho bản chạy được:
|
||||
gateway là định tuyến, auth là đăng nhập, user là hồ sơ, chat là lịch sử.
|
||||
|
||||
Thay thế trong v1:
|
||||
|
||||
| Nhu cầu | Cách làm trong v1 | Nợ kỹ thuật để lại |
|
||||
|---|---|---|
|
||||
| Chặn người ngoài | Basic-auth ở ingress (hoặc header token dùng chung) | Không có tài khoản cá nhân, không phân quyền |
|
||||
| Lịch sử hội thoại | `ai-service` ghi thẳng Postgres, bảng `conversation` / `message` | Không có service riêng, không có sync đa thiết bị |
|
||||
| Hồ sơ người dùng | Không có | Toàn bộ |
|
||||
| Định tuyến | `web` gọi thẳng `ai-service` | Không có rate-limit/gateway policy tập trung |
|
||||
|
||||
Điều này **không mâu thuẫn** với Clean Architecture đã ghi trong `CLAUDE.md`:
|
||||
domain là retrieval + grounding, còn auth/history/profile là hạ tầng. Tách
|
||||
chúng ra service riêng sau này không phải sửa domain — nếu domain được viết
|
||||
đúng ngay từ đầu (xem §4.C).
|
||||
|
||||
### Ba thứ tuyệt đối không cắt, dù trễ
|
||||
|
||||
1. **Vector không bao giờ được chọn *thuốc*.** Danh tính thuốc resolve tất
|
||||
định. Lý do đo được: cặp `PANTOPRAZOL ↔ OMEPRAZOL` có cosine chống chỉ
|
||||
định **1,000**, `DIGOXIN ↔ DIGITOXIN` 0,891/0,911, `NATRI NITRIT ↔ NATRI
|
||||
THIOSULFAT` 0,631 ở phần liều. Để cosine chọn thuốc là chấp nhận rủi ro
|
||||
trả nhầm liều của thuốc khác.
|
||||
2. **Trả về cả section, không phải top-k mảnh.** Trả 2/5 chống chỉ định
|
||||
nguy hiểm hơn trả 0, vì thiếu sẽ bị đọc thành "không có chống chỉ định".
|
||||
Đã có bảo chứng: gate `section_not_reassemblable_from_chunks` = 0.
|
||||
3. **Không đọc số liều từ 167 block quarantine** (129 block = 77% nằm trong
|
||||
`lieu_luong_va_cach_dung`) — phải hiện ảnh crop trang gốc.
|
||||
|
||||
---
|
||||
|
||||
## 2. Giả định phải xác nhận trước khi bắt đầu
|
||||
|
||||
| # | Giả định mặc định của kế hoạch này | Nếu khác thì đổi gì |
|
||||
|---|---|---|
|
||||
| GĐ-1 | Đích deploy là **k3s của team qua ArgoCD** | Nếu chỉ cần `docker-compose` demo: bỏ §4.E5-E8, tiết kiệm ~2 ngày (ước lượng) |
|
||||
| GĐ-2 | "Tracing" = **trace LLM/RAG** (câu hỏi → thực thể resolve → chunk lấy ra → prompt → câu trả lời → latency/token) | Nếu là distributed tracing OTel giữa các service: v1 chỉ có 2 service nên giá trị thấp; xem §4.F |
|
||||
| GĐ-3 | Runtime giữ **provider-agnostic**; Bedrock chỉ để benchmark embedding, không là dependency bắt buộc | Không gọi Bedrock/full corpus hoặc tạo chi phí nếu chưa có phê duyệt riêng; local smoke vector chỉ kiểm tra plumbing, không dùng làm số đo retrieval |
|
||||
| GĐ-4 | Dùng **bản 2018** đang có | Chuyển sang bản 2022 = chạy lại toàn bộ ingestion + validate lại từ đầu; **không khả thi trong 2 tuần** |
|
||||
| GĐ-5 | Câu hỏi runtime **có thể chứa thông tin bệnh nhân** | Nội dung sách là tài liệu công khai nên embedding offline không rò rỉ gì; nhưng **câu hỏi của bác sĩ thì có thể** — cần quyết định chính sách trước khi mở cho người thật dùng `[chờ xác nhận]` |
|
||||
|
||||
---
|
||||
|
||||
## 3. Kiến trúc v1
|
||||
|
||||
```
|
||||
[ web (Next.js) ] ──HTTP──> [ ai-service (FastAPI) ] ──> Qdrant (chunk + vector)
|
||||
│ └─> Postgres (hội thoại + trace)
|
||||
└──> provider cấu hình (không bắt buộc Bedrock)
|
||||
|
||||
[ ingestion CLI ] (offline, chạy tay) ──> Qdrant
|
||||
```
|
||||
|
||||
Luồng trả lời, chế độ A (biết tên thuốc — chiếm phần lớn câu hỏi):
|
||||
|
||||
```
|
||||
câu hỏi
|
||||
→ resolve thực thể (khớp chính xác dài nhất trên bảng tên+alias) → drug_id
|
||||
→ phân loại ý định → section_key (+ population nếu là câu hỏi liều)
|
||||
→ LẤY TẤT CẢ chunk của (drug_id, section_key) từ Qdrant bằng FILTER, không phải bằng vector
|
||||
→ ghép lại thành section đầy đủ
|
||||
→ nếu section có attachment quarantine → kèm ảnh crop, và cấm mô hình đọc số từ đó
|
||||
→ LLM soạn câu trả lời, bắt buộc trích: tên thuốc + tên mục + số trang IN
|
||||
```
|
||||
|
||||
Luồng chế độ B (biết khái niệm, không biết thuốc — "thuốc nào trị tăng huyết áp"):
|
||||
|
||||
```
|
||||
câu hỏi → embedding → vector search CHỈ trên section_key ∈ {chi_dinh, duoc_ly}
|
||||
→ gom theo drug_id → trả DANH SÁCH thuốc ứng viên, không trả một thuốc
|
||||
→ người dùng chọn → quay về chế độ A
|
||||
```
|
||||
|
||||
Lý do chế độ B trả danh sách chứ không trả một thuốc: `chi_dinh` là field có
|
||||
độ giống chéo cao nhất (median 0,408, 27,5% số thuốc có hàng xóm > 0,5). Với
|
||||
nhóm PPI thì omeprazol và pantoprazol trùng chỉ định là **đúng y học** — trả
|
||||
cả nhóm mới đúng.
|
||||
|
||||
---
|
||||
|
||||
## 4. Công việc chi tiết
|
||||
|
||||
Ký hiệu kích thước (ước lượng): **S** ≈ nửa buổi · **M** ≈ 1 buổi · **L** ≈
|
||||
1 ngày · **XL** ≈ 2 ngày.
|
||||
|
||||
### A. `ingestion/embed/` + `ingestion/load/`
|
||||
|
||||
Chunk record candidate là **schema v4**. `text` là văn bản retrieval có thể lặp
|
||||
nhãn ngữ cảnh an toàn; `source_text` là đoạn nguồn liên tục dùng cho kiểm chứng và
|
||||
reassembly. Payload còn có `context_labels`, `source_page_range`,
|
||||
`printed_page_range`; mỗi attachment mang trang vật lý, trang in, `block_id`, `bbox`
|
||||
và crop nếu có. Loader không tự suy luận provenance và từ chối mọi schema khác v4.
|
||||
|
||||
| # | Việc | File | Nghiệm thu | Size |
|
||||
|---|---|---|---|---|
|
||||
| A1 | Cổng embedding (interface) + adapter OpenAI, batch + retry + backoff | `ingestion/embed/ports.py`, `embed/openai_provider.py` | Test với provider giả, không gọi mạng | M |
|
||||
| A2 | Cache embedding ra đĩa theo `chunk_id` + sha256(text) | `embed/cache.py`, `data/processed/embeddings.jsonl` | Chạy lần 2 không gọi lại API; đếm cache-hit = 100% | M |
|
||||
| A3 | `cli embed` | `ingestion/cli.py` | In: số chunk, số token thật, số call, chi phí; ghi file | S |
|
||||
| A4 | Schema collection Qdrant + adapter | `load/qdrant_repo.py` | Tạo collection, index payload cho `drug_id`, `section_key`, `atc_codes`, `chunk_kind` | M |
|
||||
| A5 | `cli load` — upsert idempotent, point id sinh tất định từ `chunk_id` | `ingestion/cli.py`, `load/upsert.py` | Chạy 2 lần → số point không đổi | M |
|
||||
| A6 | **Gắn corpus vào collection**: lưu sha256 của `chunks.jsonl` vào metadata collection | `load/qdrant_repo.py` | Gate: sha256 lệch → `cli load` từ chối chạy, không upsert lẫn lộn hai đời corpus | S |
|
||||
|
||||
**Khối lượng candidate**: 4.105.382 token (đo bằng `cl100k_base`). Đơn giá phải
|
||||
tra bảng giá hiện hành trước khi chạy — không trích từ trí nhớ. Đây là hạng
|
||||
mục phải có phê duyệt riêng dù ước tính nhỏ.
|
||||
|
||||
Hai thiếu hụt từng chặn embedding — trang in và ngữ cảnh đối tượng/đường dùng —
|
||||
đã được xử lý trong schema v4. Chỉ được coi là xong khi audit toàn corpus trên
|
||||
artifact canonical xác nhận range chính xác và mọi chunk continuation giữ đủ
|
||||
nhãn ngữ cảnh.
|
||||
|
||||
### B. Tầng thực thể / alias — **làm sớm nhất, zero-regret**
|
||||
|
||||
| # | Việc | File | Nghiệm thu | Size |
|
||||
|---|---|---|---|---|
|
||||
| B1 | Trích 344 dòng `X - xem Y` từ back index thành bảng alias | `ingestion/validation/back_index.py` (thêm hàm mới, **không** đổi `parse_back_index` đang dùng cho validate) | Đếm ra đúng 344 (đo); test hồi quy | M |
|
||||
| B2 | Gom tên biệt dược từ 492 mục `ten_thuong_mai` | `ingestion/segment/` hoặc module mới `entities/` | Đếm được số alias thu thêm | M |
|
||||
| B3 | Xuất `data/verified/drug_entities.json`: 683 tên chuẩn + alias + 1.043 mã ATC → `drug_id` | mới | Mọi `drug_id` phải tồn tại trong `monographs.jsonl`; 0 alias mồ côi | M |
|
||||
| B4 | Bộ resolve **khớp chính xác dài nhất**, có test cho **19 cái bẫy substring** | `entities/resolver.py` | `HYDROCLOROTHIAZID` không ra `CLOROTHIAZID`; `PSEUDOEPHEDRIN` không ra `EPHEDRIN`; `DESLORATADIN` không ra `LORATADIN`; `HOMATROPIN HYDROBROMID` không ra `ATROPIN` | L |
|
||||
|
||||
### C. `apps/ai-service`
|
||||
|
||||
Cấu trúc theo Clean Architecture (`CLAUDE.md`): domain không import SDK.
|
||||
|
||||
| # | Việc | File | Nghiệm thu | Size |
|
||||
|---|---|---|---|---|
|
||||
| C1 | Khung FastAPI + `/health` + config qua env | `main.py`, `config.py` | `curl /health` | S |
|
||||
| C2 | Cổng (interface): `VectorStore`, `Embedder`, `Chat`, `PageRenderer` | `domain/ports.py` | Domain test chạy không cần dịch vụ sống | M |
|
||||
| C3 | Adapter Qdrant / OpenAI embed / OpenAI chat / PyMuPDF render | `adapters/` | Test tích hợp riêng, đánh dấu `@pytest.mark.integration` | L |
|
||||
| C4 | Hiểu truy vấn: tách thực thể thuốc (B4) + phân loại `section_key` + nhận diện đối tượng | `rag/understand.py` | Bộ test câu hỏi mẫu; ca không resolve được phải trả "không chắc", không đoán | L |
|
||||
| C5 | Chế độ A: lấy theo **filter**, ghép section đầy đủ | `rag/retrieve.py` | Ghép lại đúng text section (so với `monographs.jsonl`) | M |
|
||||
| C6 | Chế độ B: vector search giới hạn `section_key`, gom theo thuốc, trả danh sách | `rag/discover.py` | Trả ≥1 ứng viên cho câu hỏi chỉ định mẫu | M |
|
||||
| C7 | Soạn câu trả lời + trích dẫn bắt buộc + từ chối khi không có căn cứ | `rag/answer.py` | Không có chunk → trả "không tìm thấy trong Dược thư", **không** để LLM tự bịa | L |
|
||||
| C8 | Xử lý block quarantine: trả `attachment` + endpoint `/crop?page=&bbox=` render ảnh | `routers/crop.py` | Crop đúng vùng của `p109_t0` (ACETAZOLAMID, trang vật lý 109) | M |
|
||||
| C9 | Lưu hội thoại + trace vào Postgres | `adapters/pg.py`, migration | Hỏi 1 câu → 1 hàng trace đọc lại được | M |
|
||||
|
||||
### D. `apps/web`
|
||||
|
||||
| # | Việc | Nghiệm thu | Size |
|
||||
|---|---|---|---|
|
||||
| D1 | Bỏ mock, gọi thật `ai-service` (giữ nguyên DTO trong `shared-types`) | Chat trả lời thật | M |
|
||||
| D2 | Mở rộng `Citation`: thêm `printedPage`, `chunkId`, `attachment?` | Type check pass | S |
|
||||
| D3 | Click trích dẫn → nhảy đúng trang PDF (trang **in**, không phải trang vật lý) | Kiểm bằng mắt 5 ca | M |
|
||||
| D4 | Hiện ảnh crop cho block quarantine + nhãn cảnh báo "không trích số từ bảng này" | Kiểm bằng mắt trên 1 ca có bảng liều | M |
|
||||
|
||||
### E. Deploy
|
||||
|
||||
| # | Việc | Nghiệm thu | Size |
|
||||
|---|---|---|---|
|
||||
| E1 | `Dockerfile` cho `ai-service` | Build + chạy local | M |
|
||||
| E2 | `Dockerfile` cho `web` (Next.js standalone) | Build + chạy local | M |
|
||||
| E3 | Bổ sung 2 service vào `docker-compose.yml` | `docker compose up` ra bản chạy đầy đủ local | M |
|
||||
| E4 | Nạp dữ liệu Qdrant: chạy `cli embed` + `cli load` qua port-forward, viết runbook | `docs/runbooks/load-qdrant.md` (thư mục đang rỗng) | M |
|
||||
| E5 | Helm templates: deployment/service/ingress cho 2 app + Qdrant (statefulset + PVC) | `helm template` render sạch | XL |
|
||||
| E6 | `values-dev.yaml` thật + Secret cho OpenAI key (**không commit key**) | Secret tạo bằng tay hoặc sealed-secret | M |
|
||||
| E7 | Gỡ 3 `TODO` trong ArgoCD Application (project, repoURL, destination) | ArgoCD sync xanh | M |
|
||||
| E8 | CI: build + test + push image + bump tag trong values | 1 lần chạy thật xanh | L |
|
||||
|
||||
**Ràng buộc đã ghi trong bộ nhớ dự án**: repo gitops nội bộ
|
||||
(`git.vinmec.tech/ai-team/gitops`) là chỉ-đọc, **không đẩy gì lên đó**.
|
||||
ArgoCD Application trong repo này trỏ về chính repo này.
|
||||
|
||||
### F. Tracing
|
||||
|
||||
Theo GĐ-2 (trace LLM/RAG). Đề xuất **làm theo 2 mức, mức 1 trước**:
|
||||
|
||||
| Mức | Nội dung | Size |
|
||||
|---|---|---|
|
||||
| **1 — bắt buộc** | Mỗi request sinh `trace_id`; ghi Postgres: câu hỏi, thực thể resolve được, `section_key`, danh sách `chunk_id` lấy ra, prompt gửi đi, câu trả lời, token in/out, latency từng bước, có/không dùng block quarantine. Kèm endpoint nội bộ `/traces/{id}` đọc lại | L |
|
||||
| **2 — nếu còn thời gian** | Self-host Langfuse hoặc export OTel sang stack sẵn có của team | XL |
|
||||
|
||||
Nói thẳng: **mức 2 không phải một buổi chiều.** Langfuse bản mới cần thêm
|
||||
Clickhouse + Redis + object storage — đó là một hạng mục triển khai riêng.
|
||||
Mức 1 phục vụ đúng mục đích thật (debug một câu trả lời y khoa sai thì truy
|
||||
ngược được tới chunk và trang nào), và nó là thứ hợp với văn hoá provenance
|
||||
của dự án này.
|
||||
|
||||
### G. Eval + gate
|
||||
|
||||
Tách đôi, không gộp:
|
||||
|
||||
| # | Việc | Nghiệm thu | Size |
|
||||
|---|---|---|---|
|
||||
| G1 | **Eval định tuyến** — sinh tự động từ chính corpus: với mỗi (thuốc, field) tạo truy vấn mẫu, kiểm hệ có trả đúng `drug_id` + `section_key`. Ground truth suy ra từ dữ liệu, **không bịa một câu nào** | Báo cáo % đúng; không đặt mục tiêu giả | L |
|
||||
| G2 | **Tập đối kháng** — các cặp confusable đã đo (PPI, penicilin, digoxin/digitoxin, estriol/estron, contrast media, nitrit/thiosulfat) | Gate `wrong_drug_returned` = **0** | M |
|
||||
| G3 | Truy vấn bằng **tên biệt dược** trên 344 alias | Gate `brand_name_query_unresolved` = 0 | M |
|
||||
| G4 | **Eval nội dung** — cần dược sĩ/bác sĩ chấm | **Không tự làm được.** Xem §8 | — |
|
||||
|
||||
---
|
||||
|
||||
## 5. Lịch 2 tuần (ước lượng, không phải cam kết)
|
||||
|
||||
Nguyên tắc xếp lịch: **sau mỗi ngày phải luôn có thứ demo được**, để nếu
|
||||
trễ thì trễ ở phần đuôi chứ không phải mất trắng.
|
||||
|
||||
| Ngày | Nội dung | Cuối ngày có gì |
|
||||
|---|---|---|
|
||||
| 1 | Gate chunk v4: seam label/liều, embargo descriptor, provenance, schema fail-closed | Mọi readiness gate bằng 0; artifact canonical + SHA được chốt |
|
||||
| 2 | B1-B4 (thực thể/alias) + smoke A4-A6 bằng vector giả | Gõ "Panadol" ra `paracetamol`; local Qdrant load đủ 15.100 point, idempotent, rồi dọn collection test |
|
||||
| 3 | C1-C3 (khung + cổng + adapter) | `/health`, gọi được Qdrant + OpenAI |
|
||||
| 4 | C4-C5 (hiểu truy vấn + chế độ A) | Hỏi "chống chỉ định metformin" ra đúng section qua HTTP |
|
||||
| 5 | C7 + C9 (soạn câu trả lời + trace mức 1) | Câu trả lời có trích dẫn, có trace đọc lại được |
|
||||
| 6 | D1-D3 (web nối thật) | **Demo đầu tiên end-to-end trên máy local** |
|
||||
| 7 | C6 + C8 + D4 (chế độ B + crop bảng) | Hỏi theo chỉ định ra danh sách; bảng liều hiện ảnh |
|
||||
| 8 | G1-G3 (eval + 3 gate) | Có số thật về độ đúng định tuyến |
|
||||
| 9 | E1-E4 | `docker compose up` ra bản đầy đủ; runbook nạp dữ liệu |
|
||||
| 10 | E5-E7 | Chạy trên k3s qua ArgoCD |
|
||||
| Dự phòng | E8 (CI), F mức 2, vá lỗi | |
|
||||
|
||||
Embedding thật không được gắn cứng vào “ngày 2”: chỉ chạy sau khi gate ngày 1
|
||||
đã pass và chủ dự án phê duyệt provider, model, phạm vi và chi phí.
|
||||
|
||||
**Không có ngày trống trong 10 ngày.** Đây là rủi ro số 1 của kế hoạch: mọi
|
||||
sự cố đều ăn thẳng vào phần đuôi (CI, tracing mức 2).
|
||||
|
||||
---
|
||||
|
||||
## 6. Gate nghiệm thu v1
|
||||
|
||||
Theo phong cách sẵn có của dự án — có tên, có mục tiêu bằng 0.
|
||||
|
||||
| Gate | Mục tiêu | Đo bằng |
|
||||
|---|---|---|
|
||||
| `wrong_drug_returned` (tập đối kháng) | **0** | G2 |
|
||||
| `answer_without_citation` | 0 | G1 |
|
||||
| `dose_stated_from_quarantined_block` | 0 | rà tay trên các ca có attachment |
|
||||
| `citation_uses_physical_page` (phải là trang **in**) | 0 | G1 |
|
||||
| `brand_name_query_unresolved` (344 alias) | 0 | G3 |
|
||||
| `qdrant_point_count ≠ chunk_count` | 0 | A5 |
|
||||
| `collection_corpus_sha_mismatch` | 0 | A6 |
|
||||
| Độ đúng định tuyến (thuốc, field) | **báo số thật**, không đặt ngưỡng giả | G1 |
|
||||
| p95 latency | **đo rồi báo**, không hứa trước | tracing mức 1 |
|
||||
|
||||
---
|
||||
|
||||
## 7. Rủi ro, xếp theo mức độ
|
||||
|
||||
1. **Segmentation đang bị viết lại (Codex, ngay lúc này).** Nếu `assembler/
|
||||
detector/vocab` đổi thì `chunks.jsonl` đổi, và **mọi embedding đã trả
|
||||
tiền phải tính lại**. → Không chạy `cli embed` cho tới khi bản mới qua
|
||||
đủ: 164 test, 18/18 gate, `cli validate` ≥ 96,0%/99,1%, và so sha256
|
||||
output với mốc đã lưu. Mốc: `monographs 84f41d96…`, `chunks 63472db4…`.
|
||||
2. **Helm viết từ trống (E5).** Không có gì để copy trong repo. Đây là hạng
|
||||
mục dễ vỡ tiến độ nhất sau #1.
|
||||
3. **Tracing mức 2 phình ra.** → Chốt cứng: mức 1 là bắt buộc, mức 2 chỉ
|
||||
làm nếu ngày dự phòng còn trống.
|
||||
4. **Một người, 10 ngày, không có slack.** → Thứ tự trong §5 đã xếp sao cho
|
||||
ngày 6 đã có demo; nếu trễ thì trễ ở CI/tracing chứ không mất demo.
|
||||
5. **Chưa có ai chấm nội dung y khoa.** Gate ở §6 chứng minh hệ *lấy đúng
|
||||
mục của đúng thuốc* — **không** chứng minh câu trả lời đúng về y học.
|
||||
|
||||
---
|
||||
|
||||
## 8. Ngoài phạm vi v1 — nói thẳng, không giấu
|
||||
|
||||
- `api-gateway`, `auth-service`, `user-service`, `chat-service` (§1).
|
||||
- **Các chương tổng quát (in tr. 37-98) và phụ lục (in tr. 1497-1528)** vẫn
|
||||
chưa vào corpus. Hỏi "Kê đơn thuốc", "Ngộ độc và thuốc giải độc" sẽ **không
|
||||
ra gì**. Cần nói trước với người dùng thử.
|
||||
- Benchmark chọn embedding model (bge-m3 vs multilingual-e5 vs provider khác).
|
||||
Runtime vẫn provider-agnostic; chưa chọn provider/model cho full corpus và
|
||||
không được gọi dịch vụ có chi phí khi chưa có phê duyệt riêng.
|
||||
- Tái dựng bảng 2D và nomogram — vẫn quarantine, chỉ hiện ảnh.
|
||||
- Đánh giá nội dung y khoa (G4): **bắt buộc có dược sĩ/bác sĩ chấm.** Tôi tự
|
||||
viết câu hỏi rồi tự chấm thì chỉ đo được trí tưởng tượng của mình, không
|
||||
đo được thực tế lâm sàng — đúng loại bằng chứng giả mà `CLAUDE.md` cấm.
|
||||
- Bản Dược thư 2022 (xuất bản lần 3). Bản đang dùng là 2018.
|
||||
- Mobile app.
|
||||
|
||||
---
|
||||
|
||||
## 9. Số nào đo, số nào đoán
|
||||
|
||||
**Baseline lịch sử đã đo (2026-08-03, không dùng để load/embedding):** toàn bộ
|
||||
bảng §0; 15.076 chunk; 4.072.725 token
|
||||
`cl100k_base`; 683/11.966/8.212.880; recall 96,0% (677/705), precision
|
||||
99,1%; 18/18 gate; 164 test; 167 block quarantine (129 trong phần liều);
|
||||
344 alias `- xem`; 401 cụm cross-reference; 492 mục `ten_thuong_mai`; 1.043
|
||||
mã ATC; 19 tên thuốc là substring của tên khác; bảng cosine chéo giữa các
|
||||
thuốc; schema chunk v2. Các số này đã bị candidate schema v4 ở đầu tài liệu
|
||||
thay thế và chỉ còn giá trị đối chiếu lịch sử.
|
||||
|
||||
**Chưa đo, là phỏng đoán:** mọi ước lượng thời gian ở §4 và §5; chi phí
|
||||
embedding; p95 latency; độ khó thật của E5 (Helm) và F mức 2 (Langfuse);
|
||||
tỷ lệ câu hỏi rơi vào chế độ A so với chế độ B.
|
||||
|
||||
**Chưa biết, chờ người quyết:** GĐ-1, GĐ-2, GĐ-5 ở §2.
|
||||
Reference in New Issue
Block a user