Soften the tone of the docs and comments written today
This commit is contained in:
@@ -1,5 +1,126 @@
|
||||
# Progress Log
|
||||
|
||||
## 2026-08-11 — Five production bugs found by driving the live site, fixed, deployed and re-verified across 37 live cases
|
||||
|
||||
**Starting point: two items in the docs had moved on**, found by reading the
|
||||
code and driving production rather than by re-reading the docs:
|
||||
|
||||
1. The structured-claims refactor that
|
||||
`coordination/CLAUDE_HANDOFF_2026-08-10.md` describes as in progress
|
||||
shipped the same day (`dfdbf52`, then `9c3acd0`).
|
||||
2. Entailment majority-vote (2-of-3) is no longer in the code. `df55af4`
|
||||
introduced it; `9c3acd0` replaced it with a single pass
|
||||
(`_ENTAILMENT_MAX_ATTEMPTS = 1`).
|
||||
|
||||
The previous entry here (cont. 17) also predates five commits —
|
||||
`9c3acd0`/`33154d4`/`01e44ad`/`480bd1a`/`4438c5f`, 17:09-17:27 on
|
||||
2026-08-10. `9c3acd0` is substantial: `QueryFrame` gained
|
||||
`section_overview`/`standalone_query`/`depends_on_previous_turn`,
|
||||
`population`/`route` became enum-validated, `CatalogDrugResolver.resolve()`
|
||||
went from ~10k regexes per query to a token-span index, and `page.tsx`
|
||||
stopped losing messages on session switch. Worth remembering generally:
|
||||
entries in this log are written at a point in time, so `git log` is the
|
||||
reliable check for current state.
|
||||
|
||||
### Findings (all from driving `https://realvuxbaro.me`)
|
||||
|
||||
- **A 25s client abort against a 40s backend budget.** `ChatPanel.tsx`
|
||||
aborted every request at 25s; `config.py`'s `max_wall_clock_ms` is 40s and
|
||||
can overrun by one in-flight call (`read_timeout=20`), putting the
|
||||
backend's ceiling near 60s. Measured n=8 sequential: 6.2/6.4/8.4/10.9/12.4/
|
||||
21.7/**25.1**/**40.3**s. The 25.1s case was a correct `answerable`,
|
||||
grounded, 2-citation answer that never reached the user — the UI showed
|
||||
"Yêu cầu vượt quá 25 giây... thử lại với câu hỏi cụ thể hơn", which points
|
||||
at the question when the cause was timing. Caddy and the BFF set no timeout
|
||||
of their own, so this constant was the only binding limit.
|
||||
- **Availability failures reaching the user as content failures.**
|
||||
`_run_entailment_check` returned a bare `None` for budget exhaustion,
|
||||
provider outage and an unparseable judge reply alike, and the caller mapped
|
||||
all three to `unsupported_claim`, i.e. "the answer doesn't match the
|
||||
source", in cases where the judge was never consulted. The
|
||||
completeness-repair path had the same shape: it fell through to
|
||||
`incomplete_answer`, whose text tells the clinician the answer was
|
||||
cancelled for omitting source information. Live example: Isosorbid dinitrat
|
||||
dosage, 40.3s against a 40s budget, reported as `incomplete_answer`. The
|
||||
failure taxonomy in `docs/current-rag-pipeline-audit.md` §4 keeps
|
||||
availability and content failures separate for this reason.
|
||||
- **The pediatric dosing gate asked again for what the user had given.**
|
||||
`agent.py`'s fallback was the static "Bé bao nhiêu tuổi và cân nặng bao
|
||||
nhiêu kg?". Reproduced 5/5: "18 ký", "18 cân", explicit "18 kg", and
|
||||
"Trẻ 5 tuổi" all received it. Worth noting for anyone revisiting it: this
|
||||
is not a Vietnamese colloquial-weight parsing issue — explicit "kg"
|
||||
behaved identically, so the parser is not the place to change. The effect
|
||||
was also more visible the better `understanding.py` did, since a frame that
|
||||
parsed the weight and set `needs_clarify=false` reached the static string.
|
||||
- **A retry constant with no effect.** `_verify_entailment`'s
|
||||
`for _ in range(_ENTAILMENT_MAX_ATTEMPTS)` returned on its first iteration
|
||||
on every path, so raising it adds no retries, and the unreachable
|
||||
`return False` after it returns a `bool` where callers read `.supported`.
|
||||
- **Citation chips that render identically.** Deduped by `chunkId` but
|
||||
labelled only drug+section+page, so three distinct chunks appeared as three
|
||||
identical "METFORMIN · Liều lượng & Cách dùng · tr. 957" chips.
|
||||
|
||||
### What was changed, and what deliberately was NOT
|
||||
|
||||
The pediatric gate **still requires both age and weight** — the formulary
|
||||
bands paracetamol by age ("Trẻ em 4-6 tuổi: 240 mg") *and* by mg/kg ("10-50
|
||||
kg: 15 mg/kg"), so one field alone cannot pick a regimen. Only the question
|
||||
changed, and it now echoes the known value back so a mis-parse is visible.
|
||||
Chips are **not** collapsed by label — each opens a different evidence block
|
||||
and provenance is a hard guardrail — they carry the number the evidence
|
||||
panel already shows. The completeness judge was **not** relaxed: making that
|
||||
symptom disappear by loosening it would ship incomplete medical answers.
|
||||
Reason codes reused are ones the BFF already maps
|
||||
(`request_budget_exhausted`, `provider_unavailable`, `malformed_output`);
|
||||
an unmapped code silently reads as "no data in the formulary".
|
||||
|
||||
### Verification (37 live cases, not one)
|
||||
|
||||
`pytest`: **230 passed** (was 219; 9 added). Ruff, `tsc --noEmit` and the
|
||||
Next production build all pass. One existing test changed on purpose —
|
||||
`test_entailment_provider_outage_fails_closed_to_abstain` asserted the old
|
||||
`unsupported_claim` label; its fail-closed assertions are untouched.
|
||||
|
||||
Post-deploy, against production: a **31-case battery** (cases that must
|
||||
change, cases that must NOT, plus neighbouring behaviour) and a **6-run
|
||||
repeat** of one flaky query.
|
||||
- Pediatric clarify verified across 7 variants: "Bé 18 ký…" → "Bé nặng 18
|
||||
kg, vậy bé bao nhiêu tuổi?"; "Trẻ 5 tuổi…" → "Bé 5 tuổi nặng bao nhiêu
|
||||
kg?"; "Bé 8 tháng tuổi…" → "Bé 8 tháng tuổi nặng bao nhiêu kg?".
|
||||
- Multi-turn resolves in **both** directions (clarify→age and clarify→weight
|
||||
both reach `answerable` with 2 citations).
|
||||
- Timeout fix proven in the browser: a Metformin adult-dosing question ran
|
||||
past **33s** — dead 8s earlier under the old limit — and returned a full
|
||||
grounded answer with 3 citations. Deployed bundle contains `65e3`/`15e3`
|
||||
and **no** `25e3`.
|
||||
- Chips render `[1] [2] [3]` matching evidence-panel cards 1/2/3, all three
|
||||
preserved.
|
||||
- Regression guards all held: adult dosing untouched by the pediatric gate,
|
||||
quarantine `verify_pdf` intact (4 and 6 citations), fake drug →
|
||||
`drug_not_in_formulary`, veterinary → `out_of_scope`, ordinary facet
|
||||
lookups still answerable.
|
||||
|
||||
### Known-remaining, deliberately not claimed as fixed
|
||||
|
||||
- **The reason-code split is unit-tested but was NOT observed live**: nothing
|
||||
in the verification run exhausted the budget (max 24.2s), so no live
|
||||
`request_budget_exhausted` from the entailment/repair path was seen.
|
||||
- **Zolpidem ADR is flaky**: 6 repeats gave 5 `answerable`, 1
|
||||
`ungrounded_number` (~17%). Pre-existing generation variance —
|
||||
`grounding.verify` runs *before* any code changed here — not a regression.
|
||||
- **The same "already told you" defect survives in the LLM-generated clarify
|
||||
question**: "Bé 12 cân uống paracetamol…" is answered with "Đường dùng là
|
||||
uống hay tiêm ạ?" although the user said "uống". That text comes from
|
||||
`understanding.py`'s own `clarify_reason`, not the code-level fallback
|
||||
fixed here.
|
||||
- Latency is unchanged — the timeout fix stops discarding good answers, it
|
||||
does not make anything faster. Streaming is still the real fix.
|
||||
- **Sildenafil ADR is not a deterministic `ungrounded_number` failure**, as
|
||||
`docs/current-rag-pipeline-audit.md` states: two runs gave 46.8s
|
||||
`abstain/unsupported_claim` and 25.1s `answerable/grounded`.
|
||||
- Task #4 (real BM25 via Qdrant native sparse vectors) remains **not
|
||||
started**.
|
||||
|
||||
## 2026-08-10 (cont. 17) — First production deployment: EC2 + Docker + CI/CD, live at realvuxbaro.me
|
||||
|
||||
Owner and Codex agreed a work split mid-session
|
||||
|
||||
Reference in New Issue
Block a user