Soften the tone of the docs and comments written today
This commit is contained in:
+63
-13
@@ -1,23 +1,73 @@
|
||||
# ai-service
|
||||
|
||||
FastAPI service for drug resolution, guarded retrieval, printed-page citations,
|
||||
and PostgreSQL retrieval traces.
|
||||
FastAPI service holding the entire live RAG path: query understanding, drug
|
||||
resolution, deterministic section routing, guarded retrieval, grounded
|
||||
generation, claim/citation verification, printed-page citations, and
|
||||
PostgreSQL conversation + retrieval traces. In production this is what
|
||||
`apps/web` calls — there is no gateway in front of it.
|
||||
|
||||
Local infrastructure:
|
||||
## Local infrastructure
|
||||
|
||||
```powershell
|
||||
docker compose -f ..\..\infra\docker\docker-compose.yml up -d postgres qdrant
|
||||
python -m migrate
|
||||
uvicorn main:app --reload
|
||||
python -m uvicorn main:app --port 8079
|
||||
```
|
||||
|
||||
`GET /health` is always available. `POST /v1/rag/query` requires structured
|
||||
`subject_scope` and `intent`; unknown/non-human/recommendation requests fail
|
||||
closed. The default `EMBEDDING_PROVIDER=disabled` intentionally leaves the RAG
|
||||
backend unavailable until the collection and matching query embedder are
|
||||
configured.
|
||||
> **Use a plain restart rather than `--reload` on Windows.** The reloader has
|
||||
> been observed serving the previous code after an edit on this project, so a
|
||||
> change can appear to have no effect (or appear to work when it has not been
|
||||
> loaded). Restarting the process avoids the ambiguity.
|
||||
|
||||
`EMBEDDING_PROVIDER=local-smoke` is only for local plumbing checks. Its hashing
|
||||
vectors are deterministic but not semantic and must not be used for retrieval
|
||||
quality claims. Bedrock is not called by this service and no IAM change is
|
||||
required.
|
||||
`GET /health` is always available. `POST /v1/rag/query` is the live agent
|
||||
endpoint. Structured `subject_scope`/`intent` requests fail closed on
|
||||
unknown/non-human input.
|
||||
|
||||
## Bedrock is required for real answers
|
||||
|
||||
Since 2026-08-05 the live path calls AWS Bedrock and needs credentials with
|
||||
invoke permission:
|
||||
|
||||
| Role | Default | Setting |
|
||||
|---|---|---|
|
||||
| Query embedding | `cohere.embed-v4:0` | `embedding_provider` (default `cohere-v4`) |
|
||||
| Answer generation | Bedrock Converse | `generation_provider` (`bedrock-converse` / `bedrock-claude`) |
|
||||
| Rerank | Cohere rerank | `rerank_enabled` — **off** by default; only affects the similarity fallback, never the deterministic section route |
|
||||
|
||||
Credentials come from the environment (an IAM instance role in production, no
|
||||
long-lived keys). With no working generator configured, a rejected generation
|
||||
**abstains** — it must never degrade into dumping raw source text at a
|
||||
clinician.
|
||||
|
||||
`EMBEDDING_PROVIDER=local-smoke` is only for local plumbing checks. Its
|
||||
hashing vectors are deterministic but not semantic and must not be used for
|
||||
any retrieval-quality claim.
|
||||
|
||||
## Corpus coupling
|
||||
|
||||
Startup verifies the `duocthu_v1__manifest` sidecar and **refuses to run**
|
||||
against a collection whose corpus sha / model / dimensions do not match
|
||||
(F-05). A machine with an empty Qdrant cannot serve answers until a snapshot
|
||||
is restored or `ingestion/` is re-run — the latter costs real Bedrock spend.
|
||||
|
||||
## Request budget and timeouts
|
||||
|
||||
A turn makes several sequential Bedrock calls, bounded by
|
||||
`max_wall_clock_ms` (40s) and `max_llm_calls_per_turn` (8) in `config.py`.
|
||||
The budget is checked **between** calls and cannot cancel one already in
|
||||
flight, which is separately bounded by `read_timeout` in
|
||||
`adapters/bedrock_converse.py`. Real ceiling ≈ 40s + one in-flight call, so
|
||||
any client calling this service must allow more than that — `apps/web`'s
|
||||
`ChatPanel.tsx` uses 65s for exactly this reason.
|
||||
|
||||
## Tests
|
||||
|
||||
```powershell
|
||||
python -m pytest -q # needs Postgres + Qdrant up
|
||||
python -m pytest -q --ignore=tests/test_api.py --ignore=tests/test_live_datastores.py
|
||||
```
|
||||
|
||||
230 pass without live datastores. Coverage is uneven and worth checking
|
||||
before relying on it: `rag/agent.py`'s dosing state machine has been covered
|
||||
only since 2026-08-11, and the repo has no frontend test setup, so a green
|
||||
suite does not by itself confirm a user-visible change works.
|
||||
|
||||
@@ -259,12 +259,27 @@ class RagAgent:
|
||||
if frame.population in {"tre_em", "tre_so_sinh"} and (
|
||||
frame.age_text is None or frame.weight_kg is None
|
||||
):
|
||||
# Both fields stay required. The formulary branches pediatric
|
||||
# dosing on BOTH — paracetamol prints an age band ("Trẻ em
|
||||
# 4-6 tuổi: 240 mg") *and* a weight rule ("10-50 kg: 15
|
||||
# mg/kg") — so answering with only one of them would mean
|
||||
# picking a regimen the source does not let us pick.
|
||||
#
|
||||
# What changed on 2026-08-11 is the *question*, not the gate.
|
||||
# The fallback used to ask for age and weight every time,
|
||||
# including for whichever field the user had already given
|
||||
# (reproduced 5/5 live: "Bé 18 ký ...", "Bé nặng 18 kg ...",
|
||||
# "Trẻ 5 tuổi ..." all received the same sentence). The
|
||||
# effect was most visible when understanding.py did well: a
|
||||
# frame that parsed the weight and set needs_clarify=false
|
||||
# reaches this fallback, so a clearer question from the user
|
||||
# produced a more redundant question back.
|
||||
return AgentReply(
|
||||
"clarify", "missing_pediatric_age_or_weight",
|
||||
clarification=(
|
||||
frame.clarify_reason
|
||||
if frame.needs_clarify and frame.clarify_reason
|
||||
else "Bé bao nhiêu tuổi và cân nặng bao nhiêu kg?"
|
||||
else _pediatric_clarify_question(frame)
|
||||
),
|
||||
drugs=frame.drugs, turn_type=tt,
|
||||
quick_replies=(
|
||||
@@ -498,6 +513,32 @@ def _display_name(drug_id: str) -> str:
|
||||
return drug_id.replace("_", " ").title()
|
||||
|
||||
|
||||
def _pediatric_clarify_question(frame: QueryFrame) -> str:
|
||||
"""Ask for the pediatric field that is actually missing.
|
||||
|
||||
The caller still requires both age and weight; this only stops the
|
||||
question from asking for something the user already supplied in the very
|
||||
same sentence, which reads as not having been listened to and invites
|
||||
them to repeat themselves into the clarify-loop breaker.
|
||||
|
||||
When a field IS known it is echoed back, so the user can see the value
|
||||
was received and correct it if the parse was wrong (e.g. a colloquial
|
||||
"18 ký" read as 18 kg).
|
||||
"""
|
||||
has_age = frame.age_text is not None
|
||||
has_weight = frame.weight_kg is not None
|
||||
if has_age and not has_weight:
|
||||
return f"Bé {frame.age_text} nặng bao nhiêu kg?"
|
||||
if has_weight and not has_age:
|
||||
return f"Bé nặng {_format_kg(frame.weight_kg)} kg, vậy bé bao nhiêu tuổi?"
|
||||
return "Bé bao nhiêu tuổi và cân nặng bao nhiêu kg?"
|
||||
|
||||
|
||||
def _format_kg(weight_kg: float) -> str:
|
||||
"""Render a weight without a trailing '.0' on whole kilograms."""
|
||||
return f"{weight_kg:g}"
|
||||
|
||||
|
||||
def _is_section_overview(turn: str, frame: QueryFrame) -> bool:
|
||||
"""Separate a handbook survey from a patient-specific decision.
|
||||
|
||||
|
||||
@@ -22,7 +22,13 @@ from .routing import QueryRoutingService
|
||||
# Repeating the same temperature-0 prompt against the same model is a
|
||||
# correlated retry, not an independent vote. One fail-closed semantic pass is
|
||||
# kept after structured-claim parsing and deterministic grounding.
|
||||
_ENTAILMENT_MAX_ATTEMPTS = 1
|
||||
#
|
||||
# This is expressed as a single call rather than a `range(1)` loop on
|
||||
# purpose: the loop this replaced returned on its first iteration on every
|
||||
# path, so raising the constant looked like it added retries while silently
|
||||
# doing nothing. If a future change genuinely wants more passes, they must
|
||||
# be different prompts (or a different judge) to be independent evidence —
|
||||
# see this function's own reasoning above.
|
||||
|
||||
_QUICK_REPLY_MAX_ITEMS = 4
|
||||
_QUICK_REPLY_MAX_CHARS = 40
|
||||
@@ -112,6 +118,29 @@ class GroundedAnswer:
|
||||
plan: AnswerPlan | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _CheckNotRun:
|
||||
"""The entailment judge could not be consulted at all.
|
||||
|
||||
Distinct from a negative verdict on purpose. Fail-closed behaviour is
|
||||
identical either way — the answer is still discarded — but the reason
|
||||
code should not report an unsupported claim when the check never ran.
|
||||
`docs/current-rag-pipeline-audit.md` §4's failure taxonomy keeps
|
||||
availability and content failures separate for this reason. Observed
|
||||
live 2026-08-11: a request that ran out of wall-clock budget mid-
|
||||
verification reached the user as "bước đối chiếu chưa xác nhận được câu
|
||||
trả lời khớp với nguồn", which describes the answer rather than the
|
||||
timeout that actually occurred.
|
||||
|
||||
`reason` is deliberately one of the codes that already exist and are
|
||||
already mapped in `apps/web/app/api/chat/route.ts`; a code with no entry
|
||||
there falls back to wording that reads as "no data in the formulary",
|
||||
which would misdescribe these cases.
|
||||
"""
|
||||
|
||||
reason: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class _GenOutcome:
|
||||
answer: str | None = None
|
||||
@@ -714,7 +743,15 @@ class GroundedAnswerService:
|
||||
verification = self._verify_entailment(
|
||||
query, attempt.claims, shown_evidence, budget=budget
|
||||
)
|
||||
if verification is None or not verification.supported:
|
||||
if isinstance(verification, _CheckNotRun):
|
||||
# The judge never ran. Still fail closed, but report why: a
|
||||
# timeout or outage recorded as "unsupported claim" would sit in
|
||||
# the content-failure bucket and be hard to spot in metrics.
|
||||
self._metrics.increment(
|
||||
metric_names.GENERATION_REJECTED, reason=verification.reason
|
||||
)
|
||||
return _GenOutcome(reject_reason=verification.reason)
|
||||
if not verification.supported:
|
||||
self._metrics.increment(
|
||||
metric_names.GENERATION_REJECTED, reason="unsupported_claim"
|
||||
)
|
||||
@@ -737,6 +774,25 @@ class GroundedAnswerService:
|
||||
schema=request.schema,
|
||||
)
|
||||
repaired = self._attempt_generation(repair_request, budget)
|
||||
# The repair roughly doubles a turn's model calls, so it is the
|
||||
# most likely place to run out of wall-clock budget. Observed
|
||||
# live 2026-08-11 (Isosorbid dinitrat dosage, 40.3s against a 40s
|
||||
# budget): running out here fell through to `incomplete_answer`,
|
||||
# which describes the answer as missing source information rather
|
||||
# than reporting that the repair did not finish. Report the
|
||||
# availability failure as itself, as the first attempt above
|
||||
# already does.
|
||||
if repaired.budget_exhausted:
|
||||
self._metrics.increment(
|
||||
metric_names.GENERATION_REJECTED,
|
||||
reason="request_budget_exhausted",
|
||||
)
|
||||
return _GenOutcome(reject_reason="request_budget_exhausted")
|
||||
if repaired.outage:
|
||||
self._metrics.increment(
|
||||
metric_names.GENERATION_REJECTED, reason="provider_unavailable"
|
||||
)
|
||||
return _GenOutcome(reject_reason="provider_unavailable")
|
||||
if repaired.answer is not None:
|
||||
repaired_report = grounding.verify(repaired.answer, evidence_texts)
|
||||
repaired_verification = self._verify_entailment(
|
||||
@@ -747,9 +803,14 @@ class GroundedAnswerService:
|
||||
repaired.claims,
|
||||
repaired_verification,
|
||||
)
|
||||
if isinstance(repaired_verification, _CheckNotRun):
|
||||
self._metrics.increment(
|
||||
metric_names.GENERATION_REJECTED,
|
||||
reason=repaired_verification.reason,
|
||||
)
|
||||
return _GenOutcome(reject_reason=repaired_verification.reason)
|
||||
if (
|
||||
repaired_report.grounded
|
||||
and repaired_verification is not None
|
||||
and repaired_verification.supported
|
||||
and repaired_verification.complete
|
||||
):
|
||||
@@ -766,7 +827,7 @@ class GroundedAnswerService:
|
||||
structured_claims: tuple[tuple[str, tuple[int, ...]], ...],
|
||||
evidence_texts: tuple[str, ...],
|
||||
budget: RequestBudget | None = None,
|
||||
) -> _VerificationOutcome | None:
|
||||
) -> _VerificationOutcome | _CheckNotRun:
|
||||
"""A second, adversarial LLM pass over an answer that already passed
|
||||
`grounding.verify`.
|
||||
|
||||
@@ -808,49 +869,56 @@ class GroundedAnswerService:
|
||||
return _VerificationOutcome(supported=True, complete=True)
|
||||
|
||||
request = build_entailment_request(query, claims, evidence_texts)
|
||||
for _ in range(_ENTAILMENT_MAX_ATTEMPTS):
|
||||
verdict = self._run_entailment_check(
|
||||
request, evidence_texts=evidence_texts, budget=budget
|
||||
)
|
||||
if verdict is None:
|
||||
return None
|
||||
if (
|
||||
verdict.supported
|
||||
and not verdict.complete
|
||||
and _missing_is_already_explicit(verdict.missing, structured_claims)
|
||||
):
|
||||
return _VerificationOutcome(supported=True, complete=True)
|
||||
verdict = self._run_entailment_check(
|
||||
request, evidence_texts=evidence_texts, budget=budget
|
||||
)
|
||||
if isinstance(verdict, _CheckNotRun):
|
||||
return verdict
|
||||
return False
|
||||
if (
|
||||
verdict.supported
|
||||
and not verdict.complete
|
||||
and _missing_is_already_explicit(verdict.missing, structured_claims)
|
||||
):
|
||||
return _VerificationOutcome(supported=True, complete=True)
|
||||
return verdict
|
||||
|
||||
def _run_entailment_check(
|
||||
self,
|
||||
request,
|
||||
evidence_texts: tuple[str, ...],
|
||||
budget: RequestBudget | None = None,
|
||||
) -> _VerificationOutcome | None:
|
||||
"""One entailment call. `None` = outage/malformed/budget-exhausted
|
||||
(fails closed by the caller without a retry); `True`/`False` = the
|
||||
judge's verdict."""
|
||||
) -> _VerificationOutcome | _CheckNotRun:
|
||||
"""One entailment call.
|
||||
|
||||
Returns `_CheckNotRun` when the judge could not be consulted at all
|
||||
(budget exhausted, provider outage, or a reply this code cannot
|
||||
parse) and a `_VerificationOutcome` when it ran and reached a
|
||||
verdict. Both make the caller fail closed; they differ only in the
|
||||
reason reported, which used to collapse into `unsupported_claim`
|
||||
for all of them.
|
||||
"""
|
||||
try:
|
||||
if budget is not None:
|
||||
budget.require()
|
||||
raw = self._generator.generate(request.system, request.user, request.schema)
|
||||
except RequestBudgetExhausted:
|
||||
# Must be caught before its parent below — see `budget.py`.
|
||||
return _CheckNotRun("request_budget_exhausted")
|
||||
except AnswerGenerationUnavailable:
|
||||
return None
|
||||
return _CheckNotRun("provider_unavailable")
|
||||
try:
|
||||
payload = json.loads(raw)
|
||||
entailed = payload["entailed"]
|
||||
unsupported = payload["unsupported"]
|
||||
missing_evidence = payload.get("missing_evidence", [])
|
||||
except (ValueError, TypeError, KeyError):
|
||||
return None
|
||||
return _CheckNotRun("malformed_output")
|
||||
if (
|
||||
not isinstance(entailed, bool)
|
||||
or not isinstance(unsupported, list)
|
||||
or not isinstance(missing_evidence, list)
|
||||
):
|
||||
return None
|
||||
return _CheckNotRun("malformed_output")
|
||||
if not entailed or unsupported:
|
||||
return _VerificationOutcome(supported=False, complete=False)
|
||||
# A completeness objection is itself a factual claim about the raw
|
||||
@@ -866,11 +934,11 @@ class GroundedAnswerService:
|
||||
grounded_missing: list[str] = []
|
||||
for item in missing_evidence:
|
||||
if not isinstance(item, dict):
|
||||
return None
|
||||
return _CheckNotRun("malformed_output")
|
||||
description = item.get("description")
|
||||
evidence_quote = item.get("evidence_quote")
|
||||
if not isinstance(description, str) or not isinstance(evidence_quote, str):
|
||||
return None
|
||||
return _CheckNotRun("malformed_output")
|
||||
description = description.strip()
|
||||
quote = _normalise_for_coverage(evidence_quote)
|
||||
if (
|
||||
|
||||
@@ -142,6 +142,84 @@ def test_drug_attribute_without_an_attribute_does_not_fall_into_overview_retriev
|
||||
assert retrieval.calls == []
|
||||
|
||||
|
||||
# --- the pediatric dosing gate. This code path gained its first test
|
||||
# coverage on 2026-08-11, after driving production reproduced the same
|
||||
# behaviour 5/5: the clarify question asked for both age and weight every
|
||||
# time, including the field the user had just supplied ("Bé 18 ký ...",
|
||||
# "Bé nặng 18 kg ...", "Trẻ 5 tuổi ..." all received "Bé bao nhiêu tuổi và
|
||||
# cân nặng bao nhiêu kg?"). Requiring BOTH fields is deliberate and stays —
|
||||
# the formulary bands paracetamol by age *and* by mg/kg — so these tests pin
|
||||
# the question text without loosening the requirement. --
|
||||
|
||||
|
||||
def _pediatric_agent(**frame_kwargs) -> RagAgent:
|
||||
return _agent(QueryFrame(
|
||||
turn_type="dosing_calc",
|
||||
drugs=("paracetamol_acetaminophen",),
|
||||
population="tre_em",
|
||||
**frame_kwargs,
|
||||
))
|
||||
|
||||
|
||||
def test_pediatric_gate_with_a_known_weight_asks_only_for_the_age():
|
||||
reply = _pediatric_agent(weight_kg=18.0).handle("bé 18 ký sốt cao uống paracetamol liều bao nhiêu")
|
||||
|
||||
assert reply.decision == "clarify"
|
||||
assert reply.reason == "missing_pediatric_age_or_weight"
|
||||
assert "tuổi" in reply.clarification
|
||||
# The weight is echoed back so the user can see it was received (and
|
||||
# catch a mis-parse), but is never asked for again.
|
||||
assert "18 kg" in reply.clarification
|
||||
assert "bao nhiêu kg" not in reply.clarification
|
||||
|
||||
|
||||
def test_pediatric_gate_with_a_known_age_asks_only_for_the_weight():
|
||||
reply = _pediatric_agent(age_text="5 tuổi").handle("trẻ 5 tuổi sốt cao uống paracetamol liều bao nhiêu")
|
||||
|
||||
assert reply.decision == "clarify"
|
||||
assert "kg" in reply.clarification
|
||||
assert "bao nhiêu tuổi" not in reply.clarification
|
||||
|
||||
|
||||
def test_pediatric_gate_with_neither_field_still_asks_for_both():
|
||||
reply = _pediatric_agent().handle("liều paracetamol cho trẻ em")
|
||||
|
||||
assert reply.decision == "clarify"
|
||||
assert "tuổi" in reply.clarification
|
||||
assert "kg" in reply.clarification
|
||||
|
||||
|
||||
def test_pediatric_gate_still_requires_both_fields_before_answering():
|
||||
"""The safety property, asserted directly: knowing only one of the two
|
||||
must NOT be treated as enough to pick a regimen. Both single-field cases
|
||||
above stop at `clarify` — this states the invariant so a future change
|
||||
that "helpfully" answers with weight alone fails here loudly."""
|
||||
for kwargs in ({"weight_kg": 18.0}, {"age_text": "5 tuổi"}, {}):
|
||||
reply = _pediatric_agent(**kwargs).handle("liều paracetamol cho bé")
|
||||
assert reply.decision == "clarify", kwargs
|
||||
assert reply.reason == "missing_pediatric_age_or_weight", kwargs
|
||||
|
||||
|
||||
def test_a_whole_number_weight_is_not_echoed_with_a_trailing_zero():
|
||||
reply = _pediatric_agent(weight_kg=18.0).handle("liều paracetamol cho bé 18 ký")
|
||||
|
||||
assert "18 kg" in reply.clarification
|
||||
assert "18.0" not in reply.clarification
|
||||
|
||||
|
||||
def test_the_models_own_clarify_question_still_wins_over_the_generated_one():
|
||||
"""Unchanged precedence: when understanding.py produced a question of its
|
||||
own it is still preferred, because it can see phrasing/context this
|
||||
code-level fallback cannot."""
|
||||
reply = _pediatric_agent(
|
||||
weight_kg=18.0,
|
||||
needs_clarify=True,
|
||||
clarify_reason="Bé mấy tháng tuổi rồi ạ?",
|
||||
).handle("liều paracetamol cho bé 18 ký")
|
||||
|
||||
assert reply.clarification == "Bé mấy tháng tuổi rồi ạ?"
|
||||
|
||||
|
||||
def test_needs_clarify_frame_is_surfaced_directly():
|
||||
agent = _agent(QueryFrame(
|
||||
turn_type="dosing_calc", drugs=("paracetamol",),
|
||||
|
||||
@@ -12,6 +12,7 @@ import pytest
|
||||
|
||||
from rag import grounding
|
||||
from rag.answer import GroundedAnswerService
|
||||
from rag.budget import RequestBudgetExhausted
|
||||
from rag.metrics import GENERATION_REJECTED, GENERATION_SERVED, InMemoryMetrics
|
||||
from rag.models import (
|
||||
Evidence,
|
||||
@@ -319,6 +320,16 @@ def test_entailment_accepts_after_one_semantic_pass():
|
||||
|
||||
|
||||
def test_entailment_provider_outage_fails_closed_to_abstain():
|
||||
"""Fail-closed is unchanged; only the label it fails closed *under* is.
|
||||
|
||||
This previously asserted `unsupported_claim`, which reports a claim the
|
||||
evidence did not support, in a case where the judge was never reachable.
|
||||
`apps/web/app/api/chat/route.ts` renders that as "bước đối chiếu chưa
|
||||
xác nhận được câu trả lời khớp với nguồn", describing the answer rather
|
||||
than the outage, and places it in the content-failure bucket that the
|
||||
failure taxonomy in `docs/current-rag-pipeline-audit.md` §4 keeps
|
||||
separate from availability.
|
||||
"""
|
||||
grounded, metrics = _answer(
|
||||
{"claims": [{"text": "Người lớn: 500 mg, 2 lần/ngày", "citations": [1]}],
|
||||
"evidence_sufficient": True},
|
||||
@@ -328,7 +339,125 @@ def test_entailment_provider_outage_fails_closed_to_abstain():
|
||||
assert grounded.generated is False
|
||||
assert grounded.answer is None
|
||||
assert grounded.result.decision == EvidenceDecision.ABSTAIN
|
||||
assert grounded.result.reason == "provider_unavailable"
|
||||
assert metrics.total(GENERATION_REJECTED, reason="provider_unavailable") == 1
|
||||
# And specifically NOT counted as a content failure.
|
||||
assert metrics.total(GENERATION_REJECTED, reason="unsupported_claim") == 0
|
||||
|
||||
|
||||
def test_entailment_budget_exhaustion_is_reported_as_a_timeout_not_a_bad_claim():
|
||||
"""Observed live 2026-08-11 against production.
|
||||
|
||||
`RequestBudgetExhausted` subclasses `AnswerGenerationUnavailable`, so it
|
||||
has to be caught first to be distinguishable from an ordinary outage;
|
||||
previously both arrived as `unsupported_claim`. The user-facing string
|
||||
for `request_budget_exhausted` already exists in the BFF mapping, so no
|
||||
new reason code is introduced here.
|
||||
"""
|
||||
grounded, metrics = _answer(
|
||||
{"claims": [{"text": "Người lớn: 500 mg, 2 lần/ngày", "citations": [1]}],
|
||||
"evidence_sufficient": True},
|
||||
entailment_payload=RequestBudgetExhausted(),
|
||||
)
|
||||
|
||||
assert grounded.generated is False
|
||||
assert grounded.answer is None
|
||||
assert grounded.result.decision == EvidenceDecision.ABSTAIN
|
||||
assert grounded.result.reason == "request_budget_exhausted"
|
||||
assert metrics.total(GENERATION_REJECTED, reason="request_budget_exhausted") == 1
|
||||
assert metrics.total(GENERATION_REJECTED, reason="unsupported_claim") == 0
|
||||
assert metrics.total(GENERATION_REJECTED, reason="provider_unavailable") == 0
|
||||
|
||||
|
||||
def test_unparseable_judge_reply_is_reported_as_malformed_not_as_a_bad_claim():
|
||||
"""A judge reply this code cannot read is not a verdict against the answer."""
|
||||
grounded, metrics = _answer(
|
||||
{"claims": [{"text": "Người lớn: 500 mg, 2 lần/ngày", "citations": [1]}],
|
||||
"evidence_sufficient": True},
|
||||
entailment_payload="{not json at all",
|
||||
)
|
||||
|
||||
assert grounded.generated is False
|
||||
assert grounded.answer is None
|
||||
assert grounded.result.reason == "malformed_output"
|
||||
assert metrics.total(GENERATION_REJECTED, reason="unsupported_claim") == 0
|
||||
|
||||
|
||||
def test_budget_running_out_during_completeness_repair_is_not_called_incomplete():
|
||||
"""Pins the production case observed live 2026-08-11.
|
||||
|
||||
"Liều dùng của Isosorbid dinitrat theo Dược thư là gì?" took 40.3s
|
||||
against a 40s budget and returned `incomplete_answer`, whose user-facing
|
||||
text says the answer was cancelled because the source had information it
|
||||
left out — while what actually happened is that the repair generation did
|
||||
not run to completion. The completeness repair roughly doubles a turn's
|
||||
model calls, so it is the likeliest place to exhaust the budget, and it
|
||||
reports that the same way the first attempt does.
|
||||
"""
|
||||
grounded, metrics = _answer(
|
||||
[
|
||||
{"claims": [{"text": "Người lớn uống 500 mg", "citations": [1]}],
|
||||
"evidence_sufficient": True},
|
||||
RequestBudgetExhausted(),
|
||||
],
|
||||
entailment_payload={
|
||||
"entailed": True,
|
||||
"unsupported": [],
|
||||
"complete": False,
|
||||
"missing_evidence": [{
|
||||
"description": "2 lần mỗi ngày và liều tối đa 2 g mỗi ngày",
|
||||
"evidence_quote": EVIDENCE_TEXT,
|
||||
}],
|
||||
},
|
||||
)
|
||||
|
||||
assert grounded.answer is None
|
||||
assert grounded.result.reason == "request_budget_exhausted"
|
||||
assert metrics.total(GENERATION_REJECTED, reason="request_budget_exhausted") == 1
|
||||
# The user must not be told their answer was missing source information
|
||||
# when the repair simply ran out of time.
|
||||
assert metrics.total(GENERATION_REJECTED, reason="incomplete_answer") == 0
|
||||
|
||||
|
||||
def test_a_genuinely_incomplete_repair_is_still_called_incomplete():
|
||||
"""Guards the other side of the split above: when the repair really does
|
||||
run and still comes back incomplete, `incomplete_answer` must survive."""
|
||||
incomplete_verdict = {
|
||||
"entailed": True,
|
||||
"unsupported": [],
|
||||
"complete": False,
|
||||
"missing_evidence": [{
|
||||
"description": "2 lần mỗi ngày và liều tối đa 2 g mỗi ngày",
|
||||
"evidence_quote": EVIDENCE_TEXT,
|
||||
}],
|
||||
}
|
||||
grounded, metrics = _answer(
|
||||
{"claims": [{"text": "Người lớn uống 500 mg", "citations": [1]}],
|
||||
"evidence_sufficient": True},
|
||||
entailment_payload=[incomplete_verdict, incomplete_verdict],
|
||||
)
|
||||
|
||||
assert grounded.answer is None
|
||||
assert grounded.result.reason == "incomplete_answer"
|
||||
assert metrics.total(GENERATION_REJECTED, reason="incomplete_answer") == 1
|
||||
assert metrics.total(GENERATION_REJECTED, reason="request_budget_exhausted") == 0
|
||||
|
||||
|
||||
def test_a_real_negative_verdict_is_still_an_unsupported_claim():
|
||||
"""The counterpart to the three tests above: when the judge DID run and
|
||||
said no, the reason must stay a content failure. Splitting the
|
||||
availability cases out must not quietly reclassify genuine rejections."""
|
||||
grounded, metrics = _answer(
|
||||
{"claims": [{"text": "Metformin chữa ung thư", "citations": [1]}],
|
||||
"evidence_sufficient": True},
|
||||
entailment_payload={"entailed": False, "unsupported": [1]},
|
||||
)
|
||||
|
||||
assert grounded.generated is False
|
||||
assert grounded.result.reason == "unsupported_claim"
|
||||
assert metrics.total(GENERATION_REJECTED, reason="unsupported_claim") == 1
|
||||
assert metrics.total(GENERATION_REJECTED, reason="provider_unavailable") == 0
|
||||
assert metrics.total(GENERATION_REJECTED, reason="request_budget_exhausted") == 0
|
||||
|
||||
|
||||
def test_entailment_check_is_skipped_when_there_are_no_claims():
|
||||
|
||||
Reference in New Issue
Block a user