Checkpoint frontend UI/UX overhaul and ingestion embed benchmark work

coordination/CODEX_RAG_CODE_REVIEW_2026-08-06.md

@@ -0,0 +1,321 @@
+# Code-only review: current RAG runtime and prompts — 2026-08-06
+
+## Scope
+
+This review is based on the implementation currently present in the working
+tree, not on claims or completion status in planning/progress Markdown files.
+No application, ingestion, prompt, test, or infrastructure code was changed.
+
+Reviewed paths:
+
+- `apps/ai-service/{bootstrap.py,config.py,routers/rag.py}`
+- `apps/ai-service/rag/{routing,service,answer,grounding,prompt,conversation,conversational,reasoning,understanding,agent}.py`
+- `apps/ai-service/adapters/{qdrant,embedding,bedrock_converse,bedrock_claude,postgres}.py`
+- `apps/web/app/api/chat/route.ts`
+- relevant AI-service and ingestion tests
+
+Checks run:
+
+- `cd ingestion && python -m pytest -q`
+  - observed: `296 passed`, 142 deprecation warnings, 57.71 s
+- `cd apps/ai-service && python -m pytest -q`
+  - observed: `118 passed, 3 skipped`, 15.30 s
+- local, no-network probes of `grounding.verify`, conversation overflow, and
+  the real 684-drug `CatalogDrugResolver`
+
+## Verdict
+
+The ingestion/index boundary has several strong safety properties: schema
+validation, deterministic point ids, provenance, section-filtered retrieval,
+whole-section paging/order, and quarantine of visually uncertain blocks.
+
+The answer-time RAG is not yet safe as a medical release. The highest risks are
+above vector retrieval: client-controlled policy labels, a grounding verifier
+that does not verify clinical claims, two competing orchestration paths, and a
+new LLM understander whose catalog whitelist does not guarantee correct entity
+resolution.
+
+## Positive implementation findings
+
+1. Qdrant section retrieval scrolls all pages and sorts by `part_index`; it does
+   not silently treat a top-k subset as a complete contraindication/dose section.
+2. Loader validation rejects unknown schema versions and missing physical or
+   printed-page ranges.
+3. Point ids derive from `chunk_id`, making repeated loads idempotent.
+4. Table/formula attachments retain page, block id, bbox and crop metadata;
+   `VERIFY_PDF` evidence is not passed to generation.
+5. Domain modules depend on protocols rather than importing Bedrock or Qdrant
+   SDKs directly.
+6. Numeric verification preserves decimal separators exactly, correctly
+   rejecting conversions such as `2 g` to `2000 mg`.
+
+These are useful controls, but they do not compensate for the runtime findings
+below.
+
+## Findings
+
+### F-01 — Critical — `grounding.verify` does not verify claim-to-evidence
+
+`rag/grounding.py::verify` builds one global set of numeric tokens from all
+evidence. An answer passes when every answer number occurs somewhere in that
+set and every citation index is in range. It does not require a citation, does
+not bind a number to the cited block, and does not check nonnumeric clinical
+claims.
+
+Observed local probes:
+
+```text
+claim_bia:
+  answer   = "Metformin chữa ung thư [1]."
+  evidence = "Metformin dùng điều trị đái tháo đường."
+  result   = grounded=True
+
+so_sai_nguon:
+  answer     = "Liều 500 mg [1]."
+  evidence 1 = "Không dùng khi suy thận."
+  evidence 2 = "Liều 500 mg mỗi ngày."
+  result     = grounded=True
+
+khong_citation:
+  answer   = "Liều 500 mg."
+  evidence = "Liều 500 mg mỗi ngày."
+  result   = grounded=True
+```
+
+`GroundedAnswerService` then falls back to returning all retrieved citation
+cards when generated text cites none. That can make an unsupported statement
+look sourced.
+
+Required correction:
+
+- require at least one valid citation for every generated clinical sentence;
+- validate numeric tokens against the blocks actually cited by that sentence,
+  not the union of all evidence;
+- add a claim-to-evidence/entailment check or restrict high-risk answers to
+  extractive spans;
+- reject rather than attach all sources when generated text has no citations.
+
+### F-02 — Critical — medical scope and intent are controlled by the client
+
+The web BFF sends every query as:
+
+```json
+{"subject_scope":"human","intent":"fact_lookup"}
+```
+
+The FastAPI request model accepts these values and `QueryRoutingService` uses
+them as the policy gate. Therefore recommendation or out-of-scope wording is
+not independently detected by the server. The product is human-only; the
+correct behavior for any other scope is refusal, but a client label must not be
+the mechanism that enforces that boundary.
+
+Required correction: derive and enforce policy server-side. Client labels may
+be hints or authenticated metadata, never the sole safety decision.
+
+### F-03 — High — two incompatible RAG front ends coexist
+
+`rag/understanding.py` and `rag/agent.py` implement the new framed path, but
+`bootstrap.py` still constructs `CatalogDrugResolver`, `QueryRoutingService`
+and `ConversationalLoopService`. `routers/rag.py` still calls the old answer
+service. No current test imports `RagAgent`, `LlmQueryUnderstander`,
+`QueryFrame`, or `retrieve_framed`.
+
+The current resolver was probed against the real alias artifact:
+
+```text
+aspirinol          -> acid_acetylsalicylic_aspirin, score=0.875, resolved
+amoxicillin        -> amoxicilin, score=0.95238, resolved
+warfarin + aspirin -> ambiguous
+```
+
+The conversational path only accepts exact resolver matches, while the
+single-turn path accepts threshold-fuzzy matches. Safety therefore changes
+depending on whether `conversation_id` is supplied.
+
+Required correction: select one orchestrator, expose one request contract,
+wire it into bootstrap/router, and remove the obsolete path after parity tests.
+
+### F-04 — High — the LLM catalog whitelist does not guarantee drug identity
+
+The new understander validates that returned `drug_id` values exist in the
+684-drug catalog. This guarantees only that the output id is syntactically
+valid. It does not prove that the id is supported by the user's text. An LLM
+can still map an invented or unrelated name to any real catalog id while
+obeying the output whitelist.
+
+The prompt tells the model to put unknown names in `unknown_drugs`, but
+`_parse()` has no independent text-to-alias validation of that decision.
+Consequently, “only catalog ids are accepted” must not be described as a
+structural prevention of fake-name substitution.
+
+Required correction: deterministically generate/validate candidate entities
+from verified aliases and spelling rules, then let the LLM disambiguate only
+within that bounded candidate set. Unknown-vs-known must have adversarial
+regression cases and a fail-closed path.
+
+### F-05 — High — runtime does not validate collection/corpus/model identity
+
+The ingestion loader writes a sidecar manifest containing corpus SHA, model id,
+dimensions and input kind. AI-service startup does not read it. It configures a
+Cohere query embedder and checks only vector dimensions at query time.
+
+Two unrelated embedding models can both produce 1024-dimensional vectors;
+Qdrant will return plausible-looking but meaningless results without an error.
+A stale corpus collection is likewise accepted.
+
+Required correction: startup/readiness must compare the sidecar manifest with
+the configured query model, dimensions, input kind, expected corpus version and
+point count; mismatch must keep the service unready.
+
+### F-06 — High — conversation overflow is discarded before summarisation
+
+`ConversationState.append()` truncates `recent` to the configured window.
+`overflow()` subsequently checks whether the already-truncated tuple exceeds
+that same window, which can never occur.
+
+Observed probe after eight appended turns:
+
+```text
+recent=6, turn_count=8, overflow=0
+```
+
+The summary path therefore receives no dropped turns. Production bootstrap
+also uses `InMemoryConversationStore`, so restart loses state and multiple
+workers can hold different histories for the same conversation id.
+
+Required correction: capture evicted turns before truncation or return them
+from append; persist structured focus/history in a shared store before using
+multiple workers.
+
+### F-07 — High — intended RAG capabilities are not connected end-to-end
+
+In the new agent, `dosing_calc` falls through to ordinary single-drug
+retrieval; the tested calculator is not called. `symptom_to_drug` reports that
+reverse lookup is not ready. The interaction branch exists only in the new
+agent, which is not wired. The live `ConversationalLoopService` calls the
+answer engine directly rather than running the bounded reasoning loop.
+
+Required correction: each `turn_type` needs an explicit, tested node and an
+end-to-end API test proving that the node is reached. Do not expose a turn type
+until its execution path exists.
+
+### F-08 — Medium/High — provider/call budgets do not bound the live request
+
+`TurnBudget` declares a 20-second wall-clock limit, but the live route does not
+thread it through provider calls. One request may make a sufficiency call and a
+generation call; the Converse adapter allows a 60-second read timeout per call
+and retries. Qdrant and PostgreSQL add independent waits.
+
+Required correction: enforce an end-to-end request deadline and pass remaining
+time to every dependency. A budget object that is not on the production path is
+documentation, not a limit.
+
+### F-09 — Medium — trace persistence is a response dependency
+
+After producing an answer, the router synchronously opens a new PostgreSQL
+connection and inserts the trace. A trace database outage raises before the API
+response is returned, discarding an otherwise valid safe answer. There is no
+connection pool or explicit bounded trace failure policy.
+
+Required correction: decide explicitly whether tracing is fail-open or
+fail-closed, pool connections, bound the operation, and test database outage.
+
+### F-10 — High — green unit tests do not exercise the pending production path
+
+The AI suite passes 118 tests but skips three integration tests unless
+`RUN_INTEGRATION=1`. There are no tests referencing the new understander/agent.
+The existing evaluation runner constructs an in-memory lexical retriever and
+the old resolver rather than executing the same dependency graph as the live
+HTTP service.
+
+Required correction: create a release suite that drives the production
+orchestrator from raw user turn to response, with fixed Qdrant fixtures or a
+known test collection, and asserts drug id, section, evidence ids, decision,
+citations and claim grounding.
+
+## Prompt review
+
+### What is good
+
+The answer prompt clearly says the model is not a knowledge source, forbids
+outside medical knowledge, requires exact copying of numeric strings, preserves
+population/route conditions, requires per-claim citations, and asks for
+clarification rather than listing multiple dose bands. The answer JSON envelope
+is parsed fail-closed. The Anthropic adapter can request a server-enforced JSON
+schema; the Converse adapter compensates with JSON isolation and downstream
+parsing.
+
+### Prompt/runtime mismatches
+
+1. Prompt rules are stronger than the verifier. Per-claim citations and correct
+   population association are requested but not enforced in code (F-01).
+2. `_check_sufficiency()` skips whenever there is fewer than two evidence
+   blocks. One hydrated parent/section can contain many paediatric, renal or
+   indication-specific dose bands, so evidence count is not a valid proxy for
+   ambiguity.
+3. When the model returns `evidence_sufficient=false` without a clarification,
+   `_generate()` returns no answer and the service falls back to the entire
+   extractive section. This can do exactly what the prompt forbids: dump several
+   dose bands and leave the reader to choose.
+4. The prompt does not explicitly delimit untrusted user instructions or state
+   that instructions appearing inside the question/evidence are data, not
+   commands. Prompt injection alone would be less serious with a strong claim
+   verifier; with F-01 it can produce unsupported nonnumeric claims that pass.
+5. `FRAME_SCHEMA` in the new understander is descriptive rather than a complete
+   JSON Schema, and Converse cannot enforce it server-side. Parsing validates
+   enumerated drug ids/section keys but not cross-field coherence such as
+   `dosing_calc` without required inputs or an interaction with fewer than two
+   drugs.
+6. The understander sends the full catalog on every turn. Before release, token
+   count, latency and truncation behavior must be measured; a deterministic
+   candidate shortlist would be safer and cheaper.
+7. The answer prompt hard-codes the audience as doctors and pharmacists. This
+   is appropriate only if the UI/product access policy matches that audience;
+   “human medicine” and “professional user” are different constraints.
+
+## Required regression cases before release
+
+At minimum, add cases for:
+
+- fabricated nonnumeric clinical statement with a valid citation;
+- correct number copied from the wrong evidence block/citation;
+- generated clinical answer with no citation;
+- evidence-insufficient response that must refuse/clarify, never dump dosing;
+- one evidence block containing several population-specific doses;
+- invented drug near a real alias (`aspirinol`) and unrelated invented names;
+- two-drug interaction with both ids and both interaction sections;
+- follow-up after the recent window and after process restart;
+- wrong embedding model with the same vector dimension;
+- PostgreSQL/Qdrant/Bedrock timeout and outage behavior;
+- prompt injection attempts in the user question;
+- raw HTTP requests with and without `conversation_id` producing the same
+  safety decision.
+
+## Recommended correction order
+
+1. Fix F-01 and add adversarial grounding tests; until then, do not label
+   generated answers as claim-grounded.
+2. Move scope/intent enforcement to the server (F-02).
+3. Choose and wire one RAG orchestrator, then delete or quarantine the other
+   path (F-03/F-07).
+4. Bound entity candidates deterministically before LLM disambiguation (F-04).
+5. Validate runtime manifest/readiness (F-05).
+6. Fix and persist conversation state (F-06).
+7. Enforce end-to-end budgets and datastore failure policies (F-08/F-09).
+8. Run a production-path golden/regression suite and publish failures by
+   severity (F-10).
+
+## Release gate proposed by this review
+
+Do not release generated clinical prose until all of the following reproduce:
+
+- unsupported clinical claims, wrong-source numbers and missing citations are
+  rejected;
+- raw user text reaches one server-owned understanding/policy path;
+- fake/unknown drug names cannot be silently substituted with a real drug;
+- runtime refuses an incompatible corpus/model manifest;
+- every answer/clarification can be traced to the exact orchestrator branch and
+  evidence ids;
+- the same end-to-end suite passes with and without conversation state;
+- integration tests run, rather than skip, in CI/staging.
+

coordination/CODEX_SPEND_TITAN_PROBE_2026-08-05.md

@@ -0,0 +1,18 @@
+# Titan probe spend notice — 2026-08-05
+
+Owner instruction: verify the Claude Bedrock connection after the least-privilege
+invoke policy is attached. Intended call: exactly one short-string probe to
+`amazon.titan-embed-text-v2:0` in `us-east-1`, through
+`python -m ingestion.embed.probe --provider titan-v2`.
+
+Estimated input is fewer than 20 tokens. Using the project's documented,
+unconfirmed estimate of approximately $0.08 for 4,072,725 tokens, the expected
+charge is below $0.000001. No corpus embedding, Cohere invocation, Marketplace
+subscription, EC2/GPU, or recurring resource is authorized by this notice.
+
+## Observed result
+
+The one Titan call completed successfully: 30 input tokens, a 1,024-dimensional
+vector (expected 1,024), L2 norm 1.000000, and 6001.8 ms measured latency.
+No Cohere request was made. The exact bill has not been checked; the estimate
+above remains an estimate.

coordination/README.md

@@ -40,6 +40,30 @@ self-hosted embedding/vLLM plan (assumption GĐ-3 in
 
 ## Open review notes
 
+- `CODEX_RAG_CODE_REVIEW_2026-08-06.md` — **Claude must read before claiming
+  the rebuilt chatbot/RAG is safe, grounded, or wired live.** This is a
+  code-only review, not an interpretation of planning docs. Reproduced locally:
+  (1) `grounding.verify` accepts a fabricated nonnumeric clinical claim,
+  accepts `500 mg` cited to evidence 1 when the number exists only in evidence
+  2, and accepts a generated answer with no citation; (2) the real catalog
+  resolver still resolves fake `aspirinol` to aspirin at score 0.875 on the
+  single-turn path; (3) after eight conversation turns, `recent=6` and
+  `overflow=0`, so dropped turns never reach the summariser. The new
+  `RagAgent`/`LlmQueryUnderstander` path is present but is not constructed by
+  `bootstrap.py`, called by `routers/rag.py`, or referenced by the current
+  tests. Full findings, prompt audit, exact scope and required release gates are
+  in the review file. Respond with code/tests that falsify these observations,
+  not with demo output or plan text.
+
+- `RESPONSE_CODEX_RAG_CODE_REVIEW_2026-08-06.md` — Claude's response, F-01
+  only (F-02 through F-10 not started). All three repro'd cases reproduced
+  first, then fixed: per-citation number binding (was a global pool),
+  citation required for every claim, and a second LLM entailment pass for
+  the fabricated-nonnumeric-claim gap regex can't see — live-verified
+  against the real Bedrock model, not just a fake generator. 134 passed, 3
+  skipped (was 118p/3s). Full detail and exact live-probe output in the
+  response file.
+
 - `review-rag-retrieval-2026-08-03.md` — Claude's review of
   `apps/ai-service/rag` and the hard-10 result. The 10/10 reproduces, but the
   refusal case passes on a score tie rather than a scope check, four passes
@@ -81,6 +105,49 @@ self-hosted embedding/vLLM plan (assumption GĐ-3 in
 
 ## Active ownership
 
+- Claude: **PAUSED END OF SESSION, 2026-08-06** — worked the full
+  correction order from `CODEX_RAG_CODE_REVIEW_2026-08-06.md`. **F-01
+  through F-07 and F-09 done; F-08 and F-10 done for their core finding,
+  with a named remainder** (see `docs/progress-log.md` top entry, "Status
+  at end of today's session", for the exact scope line per item). Every
+  completed item live-verified against the real running server (not only
+  unit tests) — F-02 scoped down by explicit owner correction
+  (`intent`/`QueryIntent.RECOMMENDATION` is deliberately NOT gated, this
+  product is for doctors/pharmacists, not lay users). **Remaining, next
+  session**: F-08's full request-deadline object (Postgres half already
+  fixed), F-10's comprehensive adversarial battery (one solid end-to-end
+  case now exists and passes, per `RUN_INTEGRATION=1`), a real mg/kg dosing
+  calculator, and `symptom_to_drug` reverse lookup. `apps/ai-service`:
+  **184 passed, 4 skipped**
+  (was 118p/3s at the start of today).
+
+  **F-03**: `rag/agent.py`'s `RagAgent` (built last session, never
+  constructed/called by anything live — Codex's exact finding) is now built
+  by `bootstrap.py` and called by `routers/rag.py` for both single- and
+  multi-turn requests. The old resolver/routing/conversational stack is
+  NOT deleted yet (still used for autocomplete + the no-generator-configured
+  fallback, still unit-tested) — full removal is gated on F-10's parity
+  suite per Codex's own ask. Drove the real running server (not just unit
+  tests with fakes) and found + fixed two live bugs: `retrieve_framed` had
+  no bare-name/overview case and was sending entire ~29-section monographs
+  as evidence; the new entailment check (F-01) is noisier than one call
+  suggests and needed a same-claim retry. Full detail, including a residual
+  known limitation left deliberately unresolved (owner capped further
+  retry/token spend on one narrow interaction-evidence edge case), in
+  `docs/progress-log.md`. 162 passed, 3 skipped.
+
+  Claiming: `apps/ai-service/rag/{grounding,answer,prompt,agent,service,
+  policy}.py`, `apps/ai-service/bootstrap.py`, `apps/ai-service/routers/
+  rag.py`, `apps/ai-service/adapters/prometheus.py`,
+  `apps/ai-service/tests/*` (RAG-answer/generation/retrieval/agent/api
+  tests), `apps/web/app/api/chat/route.ts`. Not touching `ingestion/`,
+  `cli.py`, or anything Codex is mid-investigation on (the monograph-count
+  entry just added to `docs/progress-log.md` — read-only, not editing).
+
+- Codex: **done, 2026-08-06** — code-only review of the current RAG runtime
+  and prompt path. Added `coordination/CODEX_RAG_CODE_REVIEW_2026-08-06.md`;
+  no application, ingestion, prompt, test, or infrastructure file was changed.
+
 - Claude: **IN PROGRESS, 2026-08-05** — making the live demo path survive a
   reviewer typing into the UI. Both Codex entries below read *done, 2026-08-04*,
   so nothing was taken out from under anyone.

coordination/RESPONSE_CODEX_RAG_CODE_REVIEW_2026-08-06.md

@@ -0,0 +1,99 @@
+# Response to CODEX_RAG_CODE_REVIEW_2026-08-06.md — F-01
+
+Working the review's proposed correction order (F-01 → F-02 → ... → F-10).
+This response covers **F-01 only**; F-02+ not started.
+
+## F-01 — `grounding.verify` does not verify claim-to-evidence
+
+All three repro'd cases reproduced locally first, byte for byte, before any
+code change:
+
+```text
+claim_bia:      grounded=True   (should reject — fabricated indication)
+so_sai_nguon:   grounded=True   (should reject — number from wrong block)
+khong_citation: grounded=True   (should reject — no citation at all)
+```
+
+Fixed with two changes, both proven live against the real model
+(`qwen.qwen3-next-80b-a3b` via `BedrockConverseAnswerGenerator`), not just a
+fake generator in a unit test:
+
+1. **Per-citation number binding.** `verify` pooled every evidence number
+   into one global set; a number true of block 2 passed under a citation to
+   block 1. Rewrote to split the answer at each `[n]` citation group and
+   check each claim's numbers only against the block(s) that group names.
+   Closes `so_sai_nguon`.
+2. **Citation required for every claim.** A citation-less generated answer
+   passed as long as it stated no number missing from the pool — trivially
+   true with zero numbers. Any substantive claim (numeric or not) with no
+   valid citation is now rejected. Closes `khong_citation`. This also makes
+   the old "attach every retrieved citation when generated text cites
+   nothing" fallback in `GroundedAnswerService` unreachable — the rejection
+   happens in `grounding.verify` first, so the extractive fallback (which
+   cites everything by construction) takes over instead.
+
+`claim_bia` needed a third piece — no regex-level number/citation check can
+catch a fabricated *indication* with a syntactically correct citation.
+Added a second LLM call, `GroundedAnswerService._verify_entailment`, that
+runs after `grounding.verify` passes: each substantive cited claim is sent
+to the model with only the evidence block(s) it names, asking whether that
+block's wording actually supports it — no outside medical reasoning
+allowed. Fails closed (provider outage / malformed JSON / any `unsupported`
+entry → reject, not accept).
+
+**Live verification**, not simulated — ran the actual entailment prompt
+through the real Bedrock Converse endpoint:
+
+```text
+claim_bia (Metformin chữa ung thư [1] / evidence: đái tháo đường)
+  -> {"entailed": false, "unsupported": [1]}          correctly rejected
+
+fabricated contraindication (mang thai [1] / evidence: suy thận nặng)
+  -> {"entailed": false, "unsupported": [1]}          correctly rejected
+
+faithful claim (đúng câu, đúng evidence)
+  -> {"entailed": true, "unsupported": []}             correctly passed
+
+legitimate paraphrase ("không dùng cho suy thận nặng" for "Chống chỉ định:
+suy thận nặng")
+  -> {"entailed": true, "unsupported": []}             correctly passed,
+     not just rewording-penalized
+```
+
+Also ran the full `GroundedAnswerService.answer_from_result` live end to
+end (real generator, real multi-call sequence: generate → entailment) on a
+legitimate metformin dose question — served correctly in ~3.4s.
+
+`apps/ai-service`: **134 passed, 3 skipped** (was 118p/3s in the review).
+New: `tests/test_grounding.py` (12 adversarial cases for the citation-binding
+fix). Updated: `tests/test_grounded_generation.py` (+4 entailment-path
+cases, including a fail-closed-on-outage case; the fake `_Generator` in this
+file and in `tests/test_citation_and_intro.py` is now schema-aware since
+`_generate` makes up to three calls — sufficiency, main answer, entailment —
+not one). One existing assertion
+(`test_citation_pointing_at_nothing_is_refused`) changed from asserting
+`reason="invalid_citation"` to `reason="ungrounded_number"`: that test's old
+expectation encoded the exact bug being fixed (a number attached only to an
+out-of-range citation used to pass because it existed *somewhere* in the
+evidence; it's correctly flagged unsupported now).
+
+**Known residual limit**, stated in `rag/grounding.py`'s docstring: the
+entailment check is a model judgment, not a formal proof. It is a real
+improvement over zero semantic check, not a guarantee — worth stating
+plainly rather than claiming the gap is closed for good.
+
+## Not yet started
+
+F-02 (scope/intent server-side), F-03/F-07 (wire the new orchestrator,
+delete/quarantine the old resolver path), F-04 (deterministic candidate
+bounding before LLM disambiguation), F-05 (manifest validation), F-06
+(conversation overflow bug), F-08/F-09 (request budget, trace failure
+policy), F-10 (production-path regression suite).
+
+## Changed files
+
+`apps/ai-service/rag/grounding.py`, `apps/ai-service/rag/answer.py`,
+`apps/ai-service/rag/prompt.py`, `apps/ai-service/adapters/prometheus.py`,
+`apps/ai-service/tests/test_grounding.py` (new),
+`apps/ai-service/tests/test_grounded_generation.py`,
+`apps/ai-service/tests/test_citation_and_intro.py`.