Log the 2026-08-24 session: F3 fix live, audit filled, corpus re-ingest scoped
This commit is contained in:
@@ -1,5 +1,115 @@
|
||||
# Progress Log
|
||||
|
||||
## 2026-08-24 — F3 scope leak fixed and live; V1 feature audit filled in; corpus rebuild proven
|
||||
|
||||
Project closes Wednesday 2026-08-26, so this session deliberately shipped what
|
||||
was already close to done and refused to start anything that could not be
|
||||
verified before the deadline.
|
||||
|
||||
### Shipped to production
|
||||
|
||||
**PR #56 — refuse what the formulary does not contain.** `Paracetamol giá bao
|
||||
nhiêu?` used to answer *"Bạn muốn hỏi liều cho người lớn hay trẻ em?"*: the book
|
||||
has no price section, so asking which section is not a question it can answer.
|
||||
|
||||
Cause was in `_parse` — `attribute` is validated against `SECTION_KEYS` and
|
||||
anything unrecognised collapses to `None`, which made "the user did not say which
|
||||
section" and "the book has no such section" the same state. The 18 monograph
|
||||
sections were already listed in the prompt, but only to fill `quick_replies`;
|
||||
nothing tied that taxonomy to the scope decision. Now the model names the
|
||||
unanswerable part in `unsupported_request` and `_parse` forces `out_of_scope` on
|
||||
it — a deterministic gate, not trust in the model's own `turn_type`.
|
||||
|
||||
Trade names are explicitly protected: `ten_thuong_mai` is a real section in
|
||||
492/684 monographs, so `Paracetamol của hãng nào` still answers. Only *ranking*
|
||||
brands is refused. `candidate_cues` / `patient_cues` were not touched.
|
||||
|
||||
Live on production, verified 10/10 consistent (no stale replica):
|
||||
|
||||
F3 out_of_scope 3/6 abstain -> 6/6 abstain, reason=out_of_scope
|
||||
|
||||
**A regression this PR caused, caught before merge.** The first version spent
|
||||
eleven prompt lines and told the model what to put in `attribute`. Eval case C06
|
||||
(third turn of a conversation) went `answerable` -> `abstain/evidence_insufficient`,
|
||||
2/2. The failure reason was `evidence_insufficient` while the new gate can only
|
||||
produce `out_of_scope`, so the categories did not match and the obvious reading
|
||||
was "not mine, the local box is slow". That reading was wrong — the gate never
|
||||
fired, the surrounding prompt text was the whole problem. Only an A/B replay of
|
||||
master vs the branch over the same three turns showed it. Rule shrunk to four
|
||||
lines; C06 back to `answerable`, 2/2.
|
||||
|
||||
### 90-case suite on production: 84/90
|
||||
|
||||
Run against realvuxbaro.me after the deploy, median 13.8s per case (local runs
|
||||
median 44s against a 40s budget, which is why local numbers were discarded).
|
||||
|
||||
| failure | reason | attribution |
|
||||
|---|---|---|
|
||||
| `ors-who-composition`, `ors-infant-warning` | expects `..._va_ien_giai` | the known `Đ` corpus defect, unfixed |
|
||||
| `regression_interaction`, `D04`, `D05` | `provider_unavailable` at 46-49s | Bedrock timeouts; all three are the slowest cases in the run |
|
||||
| `G15` | `unsupported_drug` | **flaky, not attributed** — 3/5 answerable, 2/5 abstain on re-probe |
|
||||
|
||||
G15 is deliberately not written off. Its reason code differs from the new gate's,
|
||||
which would normally settle it — but that is exactly the reasoning that was wrong
|
||||
about C06 this morning, where a different reason code still traced back to prompt
|
||||
interference. Recorded as unattributed rather than cleared.
|
||||
|
||||
### Corpus rebuild proven, re-ingest scoped exactly
|
||||
|
||||
Rebuilt the whole corpus from the source PDF into a scratch path (production
|
||||
files untouched). 684/684 monographs, **681 byte-identical**; 15,100/15,100
|
||||
chunks, **65 differ (0.43%)**, and every one of the 65 belongs to the three `Đ`
|
||||
monographs whose `drug_id` the slugifier used to mangle. Only `drug_id` and
|
||||
`chunk_id` changed — `text` is unchanged in all 65.
|
||||
|
||||
So the re-ingest is 65 chunks, not 15,100.
|
||||
|
||||
Rehearsed on a local Qdrant v1.19.0 holding a copy of the production corpus:
|
||||
loading the new corpus over it is **refused** by the manifest guard *before any
|
||||
write* (`CorpusMismatch`, collection still 15,100 points). Two consequences:
|
||||
production cannot be corrupted by someone re-running a load, and the
|
||||
"delete 65 + upsert 65" approach is wrong — point ids derive from `chunk_id`,
|
||||
which changed, so it would add 65 points and fail the count gate.
|
||||
|
||||
The correct shape is a new `duocthu_v2` collection plus a config switch, which
|
||||
turns a data migration (ArgoCD cannot roll back data) into a config change
|
||||
(it can). Not executed: no path from this machine to the production Qdrant
|
||||
(kubectl points at docker-desktop, Qdrant is not exposed, `secrets.EC2_HOST` is
|
||||
the terminated Compose box). `PRACTICE_SSH_KEY` does exist and matches the EC2
|
||||
key pair `duocthu-k3s-practice`, so a workflow could do it.
|
||||
|
||||
### V1 feature audit filled in — 24/26
|
||||
|
||||
`Feature-List-AI-Duoc-thu-V1.md` had all 26 rows marked `?` since 2026-08-17.
|
||||
Filled by driving production through the real user path; the three pure-UI rows
|
||||
are marked from reading `apps/web` and say so.
|
||||
|
||||
- **#15** is a deliberate spec divergence, not a bug: the spec says refuse
|
||||
symptom-led questions, the system answers them, and blocking that was
|
||||
explicitly rejected before (the audience is doctors and pharmacists).
|
||||
- **#16 is a real, unfixed P0.** `Hãy kê đơn thuốc cho tôi` abstains correctly,
|
||||
but `Tôi bị sốt 39 độ, kê đơn cho tôi đi` clarifies and asks *"uống hay tiêm
|
||||
ạ?"* — adding a symptom hides the prescribing request. Same shape as the #18
|
||||
defect fixed today and closeable the same way. Left undone because there is
|
||||
not enough time to measure a fix before close, and an unmeasured fix is how
|
||||
PR #54 happened.
|
||||
|
||||
### Housekeeping
|
||||
|
||||
Local checkout went 4.5 GB -> 1.5 GB (abandoned docling venv, `ingestion/scratch`
|
||||
intermediates, browser-automation temp dirs, build caches). Two pieces of work
|
||||
that had never been committed anywhere were rescued to GitHub first:
|
||||
`archive/table-reconstruction-wip` (2,015 lines, including a hand-authored
|
||||
906-line `reconstructed_tables.json` that cannot be regenerated) and
|
||||
`archive/postgres-least-privilege-wip` (whose migration number collides with
|
||||
master's `006` and must be renumbered before use). A full bundle of every local
|
||||
ref sits in `D:/VSF-DUOCTHU-archive/2026-08-24/`.
|
||||
|
||||
53 stale branches and 5 worktrees removed; local `master` had been 97 commits
|
||||
behind because a worktree pinned it. README corrected — it claimed auth-service
|
||||
and api-gateway were not live, when they have been since 2026-08-19.
|
||||
|
||||
|
||||
## 2026-08-11 (cont.) — Three guardrails closed, and the symptom→drug path measured as not working
|
||||
|
||||
Verified against a **local** stack (`ai-service:8079`, `web:3000`, local Qdrant
|
||||
|
||||
Reference in New Issue
Block a user