Wire token-budget packing into the overview/rerank fallback path

.github/workflows/deploy.yml

@@ -0,0 +1,27 @@
+name: Deploy to production
+
+on:
+  push:
+    branches: [master]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Deploy over SSH
+        uses: appleboy/ssh-action@v1.0.3
+        with:
+          host: ${{ secrets.EC2_HOST }}
+          username: ubuntu
+          key: ${{ secrets.EC2_SSH_KEY }}
+          script: |
+            set -e
+            cd ~/app
+            git fetch origin master
+            git reset --hard origin/master
+            cd infra/docker
+            sudo docker compose -f docker-compose.prod.yml up -d --build ai-service web
+            sudo docker exec docker-ai-service-1 python -m migrate
+            sleep 5
+            sudo docker run --rm --network docker_default curlimages/curl -sf http://ai-service:8000/health
+            sudo docker run --rm --network docker_default curlimages/curl -sf -o /dev/null http://web:3000

apps/ai-service/adapters/qdrant.py

@@ -5,6 +5,16 @@ from typing import Any, Protocol, Sequence
 from rag.models import ParentDocument, RetrievalDocument, SearchHit, SourceRef
 
 
+# Vietnamese function words dropped before lexical matching — high-frequency,
+# low-signal; matching on these alone would make `search_lexical` return
+# near-arbitrary same-drug chunks instead of ones sharing real query terms.
+_LEXICAL_STOPWORDS = frozenset({
+    "cua", "va", "la", "cho", "khi", "co", "gi", "duoc", "voi", "the", "nao",
+    "nhu", "o", "trong", "de", "hay", "mot", "nay", "day", "thi", "bi",
+    "khong", "da", "se", "neu", "nen", "phai", "sao",
+})
+
+
 class QueryEmbedder(Protocol):
     @property
     def dimensions(self) -> int: ...
@@ -93,6 +103,9 @@ def _document(payload: dict[str, Any]) -> RetrievalDocument:
             bool(payload.get("requires_visual_check"))
             or bool(payload.get("has_quarantined_content"))
         ),
+        part_index=payload.get("part_index"),
+        part_count=payload.get("part_count"),
+        context_labels=tuple(payload.get("context_labels") or ()),
     )
 
 
@@ -180,6 +193,59 @@ class QdrantRetriever:
         return [hit for _, hit in hits]
 
 
+    def search_lexical(self, query: str, drug_id: str, limit: int) -> list[SearchHit]:
+        """Keyword/BM25-style candidates across ALL of one drug's sections,
+        ranked by term overlap with `query`.
+
+        Two live callers, both in `rag.service.RetrievalService`: (1) inside
+        the deterministic `_section_hits` route, to find a NEIGHBOUR section
+        whose text lexically matches strongly enough to pool in alongside
+        the one the keyword route resolved (a "thận trọng" question can have
+        its real answer only in "chống chỉ định" — see `_pooled_neighbour_
+        hits`'s docstring); (2) available for hybrid fusion with `search`'s
+        dense results (`rag.fusion.reciprocal_rank_fusion`) in the
+        similarity-fallback path, for a free-form question naming no
+        section a paraphrase makes the exact-phrase `find_by_indication`-
+        style match miss.
+
+        Qdrant's `text` index tokenizes and matches individual query tokens
+        (OR semantics across a `should` filter — no `min_should_match` needed
+        since the caller fuses ranks, not raw hits). Common short function
+        words are dropped before matching so they don't dilute every result
+        with the same handful of low-signal hits; score is the count of
+        distinct matched tokens, a transparent stand-in for a real BM25 score
+        given no term-frequency/IDF statistics are computed here.
+        """
+        from qdrant_client.models import FieldCondition, Filter, MatchText, MatchValue
+
+        from rag.text import normalize_name
+
+        tokens = sorted(set(normalize_name(query).split()) - _LEXICAL_STOPWORDS)
+        tokens = [token for token in tokens if len(token) >= 2]
+        if not tokens:
+            return []
+
+        points, _ = self._client.scroll(
+            collection_name=self._collection_name,
+            scroll_filter=Filter(
+                must=[FieldCondition(key="drug_id", match=MatchValue(value=drug_id))],
+                should=[FieldCondition(key="text", match=MatchText(text=t)) for t in tokens],
+            ),
+            limit=max(limit * 4, 20),
+            offset=None,
+            with_payload=True,
+        )
+        scored: list[tuple[int, SearchHit]] = []
+        for point in points:
+            payload = dict(point.payload or {})
+            text_normalized = normalize_name(payload.get("text", ""))
+            matched = sum(1 for t in tokens if t in text_normalized.split())
+            if matched == 0:
+                continue
+            scored.append((matched, SearchHit(_document(payload), float(matched))))
+        scored.sort(key=lambda item: -item[0])
+        return [hit for _, hit in scored[:limit]]
+
     def find_by_drug(self, drug_id: str) -> list[SearchHit]:
         """Every prose section of one drug, in book order — the monograph view.
 

apps/ai-service/config.py

@@ -7,6 +7,20 @@ from pydantic import Field
 from pydantic_settings import BaseSettings, SettingsConfigDict
 
 
+def _default_entities_path() -> Path:
+    """2 parents up from apps/ai-service/config.py in a full repo checkout.
+
+    A container image that flattens apps/ai-service/ into its own root
+    (found live 2026-08-10: the deploy image does exactly this) doesn't have
+    that depth — ENTITIES_PATH env override is for it; this fallback just
+    keeps the class from crashing at import time when it's shallower.
+    """
+    here = Path(__file__).resolve()
+    return (
+        here.parents[2] if len(here.parents) > 2 else here.parent
+    ) / "ingestion/data/verified/drug_entities.json"
+
+
 class Settings(BaseSettings):
     model_config = SettingsConfigDict(env_file=".env", extra="ignore")
 
@@ -36,10 +50,7 @@ class Settings(BaseSettings):
     # never uses it.
     rerank_enabled: bool = False
     metrics_enabled: bool = True
-    entities_path: Path = (
-        Path(__file__).resolve().parents[2]
-        / "ingestion/data/verified/drug_entities.json"
-    )
+    entities_path: Path = _default_entities_path()
     # F-08: a per-turn budget across RagAgent's sequential Bedrock calls
     # (understand, sufficiency, generate, up to 2 entailment retries).
     # Defaults sized with headroom above what a normal turn measures live

apps/ai-service/rag/artifacts.py

@@ -42,6 +42,9 @@ def load_documents(path: Path) -> list[RetrievalDocument]:
             parent_id=raw.get("parent_id"),
             requires_visual_check=raw.get("requires_visual_check", False),
             drug_name=raw.get("drug_name"),
+            part_index=raw.get("part_index"),
+            part_count=raw.get("part_count"),
+            context_labels=tuple(raw.get("context_labels") or ()),
         ))
     return documents
 

apps/ai-service/rag/context.py

@@ -0,0 +1,61 @@
+"""Evidence-safe context packing for generation."""
+from __future__ import annotations
+
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass
+
+from .models import Evidence
+
+
+TokenCounter = Callable[[str], int]
+
+
+def conservative_token_count(text: str) -> int:
+    """Dependency-free estimate, conservative for Vietnamese text."""
+    return (len(text) + 2) // 3 if text else 0
+
+
+@dataclass(frozen=True)
+class PackedContext:
+    evidence: tuple[Evidence, ...]
+    omitted_evidence_ids: tuple[str, ...]
+    estimated_tokens: int
+
+
+def pack_evidence(
+    evidence: Sequence[Evidence],
+    *,
+    max_tokens: int,
+    max_items: int | None = None,
+    token_counter: TokenCounter = conservative_token_count,
+    block_overhead_tokens: int = 4,
+) -> PackedContext:
+    """Pack whole blocks in retrieval order; never truncate clinical text."""
+    if max_tokens < 1:
+        raise ValueError("max_tokens must be positive")
+    if max_items is not None and max_items < 1:
+        raise ValueError("max_items must be positive or None")
+    if block_overhead_tokens < 0:
+        raise ValueError("block_overhead_tokens must be non-negative")
+
+    selected: list[Evidence] = []
+    omitted: list[str] = []
+    seen: set[str] = set()
+    used = 0
+    for item in evidence:
+        if item.evidence_id in seen:
+            continue
+        seen.add(item.evidence_id)
+        count = token_counter(item.text)
+        if count < 0:
+            raise ValueError("token_counter must return a non-negative value")
+        cost = count + block_overhead_tokens
+        if (
+            (max_items is not None and len(selected) >= max_items)
+            or used + cost > max_tokens
+        ):
+            omitted.append(item.evidence_id)
+            continue
+        selected.append(item)
+        used += cost
+    return PackedContext(tuple(selected), tuple(omitted), used)

apps/ai-service/rag/expansion.py

@@ -0,0 +1,62 @@
+"""Bounded sibling expansion for split clinical sections."""
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from .models import RetrievalDocument, SearchHit
+
+
+def _compatible(left: RetrievalDocument, right: RetrievalDocument) -> bool:
+    """Prevent expansion across two explicitly different clinical scopes."""
+    if left.drug_id != right.drug_id or left.section_key != right.section_key:
+        return False
+    if left.context_labels and right.context_labels:
+        return bool(set(left.context_labels) & set(right.context_labels))
+    return not left.context_labels and not right.context_labels
+
+
+def expand_siblings(
+    seeds: Sequence[SearchHit],
+    documents: Sequence[RetrievalDocument],
+    *,
+    window: int = 1,
+    limit: int | None = None,
+) -> list[SearchHit]:
+    """Add adjacent parts without crossing drug, section, or scope labels.
+
+    Each seed remains first in its group. Neighbours retain the seed score so
+    later context packing treats them as context for that match, not as a new
+    independently scored retrieval result.
+    """
+    if window < 0:
+        raise ValueError("window must be non-negative")
+    if limit is not None and limit < 0:
+        raise ValueError("limit must be non-negative or None")
+
+    by_position = {
+        (doc.drug_id, doc.section_key, doc.part_index): doc
+        for doc in documents
+        if doc.part_index is not None
+    }
+    output: list[SearchHit] = []
+    seen: set[str] = set()
+
+    for seed in seeds:
+        doc = seed.document
+        group = [doc]
+        if doc.part_index is not None and window:
+            group = []
+            for index in range(doc.part_index - window, doc.part_index + window + 1):
+                sibling = by_position.get((doc.drug_id, doc.section_key, index))
+                if sibling is not None and _compatible(doc, sibling):
+                    group.append(sibling)
+            group.sort(key=lambda item: item.part_index if item.part_index is not None else 0)
+
+        for item in group:
+            if item.doc_id in seen:
+                continue
+            seen.add(item.doc_id)
+            output.append(SearchHit(item, seed.score))
+            if limit is not None and len(output) >= limit:
+                return output
+    return output

apps/ai-service/rag/fusion.py

@@ -0,0 +1,55 @@
+"""Rank fusion primitives for hybrid and multi-query retrieval.
+
+This module is deliberately store-agnostic: dense, lexical, and rewritten-query
+retrievers only need to return ranked ``SearchHit`` lists. Keeping fusion pure
+makes its ordering deterministic and lets the live service add parallel I/O
+without coupling domain code to Qdrant or PostgreSQL.
+"""
+from __future__ import annotations
+
+from collections.abc import Sequence
+
+from .models import SearchHit
+
+
+def reciprocal_rank_fusion(
+    rankings: Sequence[Sequence[SearchHit]],
+    *,
+    rank_constant: int = 60,
+    limit: int | None = None,
+) -> list[SearchHit]:
+    """Fuse ranked candidate lists using reciprocal-rank fusion (RRF).
+
+    A document contributes at most once per input ranking. The returned score
+    is the accumulated RRF score, not a provider-specific similarity score, so
+    dense and lexical results remain comparable without score normalization.
+    Ties are stable by first appearance, which keeps results reproducible.
+    """
+    if rank_constant < 1:
+        raise ValueError("rank_constant must be positive")
+    if limit is not None and limit < 0:
+        raise ValueError("limit must be non-negative or None")
+
+    scores: dict[str, float] = {}
+    documents = {}
+    first_seen: dict[str, int] = {}
+    seen_order = 0
+
+    for ranking in rankings:
+        seen_in_ranking: set[str] = set()
+        for rank, hit in enumerate(ranking, start=1):
+            doc_id = hit.document.doc_id
+            if doc_id in seen_in_ranking:
+                continue
+            seen_in_ranking.add(doc_id)
+            if doc_id not in documents:
+                documents[doc_id] = hit.document
+                first_seen[doc_id] = seen_order
+                seen_order += 1
+            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (rank_constant + rank)
+
+    ordered_ids = sorted(scores, key=lambda doc_id: (-scores[doc_id], first_seen[doc_id]))
+    if limit is not None:
+        ordered_ids = ordered_ids[:limit]
+    return [SearchHit(document=documents[doc_id], score=scores[doc_id]) for doc_id in ordered_ids]
+

apps/ai-service/rag/models.py

@@ -45,6 +45,9 @@ class RetrievalDocument:
     parent_id: str | None = None
     requires_visual_check: bool = False
     drug_name: str | None = None
+    part_index: int | None = None
+    part_count: int | None = None
+    context_labels: tuple[str, ...] = field(default_factory=tuple)
 
 
 @dataclass(frozen=True)

apps/ai-service/rag/prompt.py

@@ -103,6 +103,13 @@ Quy tắc:
   tuổi và cân nặng). "người lớn" thường ĐỦ cho liều người lớn tiêu chuẩn.
 - Câu hỏi KHÔNG về liều (chống chỉ định, tương tác, tác dụng phụ, giới thiệu
   thuốc…) thì thường ĐỦ.
+- Câu hỏi "thận trọng"/"an toàn khi dùng cho bệnh nhân [tình trạng cụ thể]"
+  mà BẰNG CHỨNG có đoạn CHỐNG CHỈ ĐỊNH nói đúng tình trạng đó → ĐỦ. Chống chỉ
+  định là mức cảnh báo MẠNH HƠN thận trọng (không được dùng, thay vì dùng có
+  theo dõi) — nó TRẢ LỜI được câu hỏi "có an toàn không", chỉ là câu trả lời
+  nghiêm trọng hơn người hỏi hình dung, không phải "sách không đề cập". Đừng
+  yêu cầu đúng từ "thận trọng" xuất hiện — khớp theo Ý (tình trạng bệnh nhân),
+  không khớp theo NHÃN mục sách xếp nó vào.
 
 Trả về DUY NHẤT JSON: {"sufficient": bool, "clarifying_question": string|null,
 "quick_replies": string[]}.

apps/ai-service/rag/service.py

@@ -2,6 +2,7 @@ from __future__ import annotations
 
 from dataclasses import dataclass
 
+from .context import pack_evidence
 from .models import Evidence, EvidenceDecision, RetrievalResult, SearchHit
 from .ports import (
     ParentStore,
@@ -32,6 +33,13 @@ class EvidencePolicy:
     # A free-form question about a resolved drug otherwise hands the LLM the
     # entire monograph; rerank trims it to the sections that actually answer.
     rerank_top_k: int = 6
+    # Token budget for the overview/rerank fallback's evidence pool (2026-08-10
+    # pipeline audit, priority #1). Replaces a flat evidence_limit COUNT: 3
+    # short chunks wastes budget a real model has, 3 long ones can silently
+    # exceed it. Never applied to the deterministic section route — a whole
+    # section is the answer there, and a truncated contraindication list
+    # reads as a complete one (see `_section_hits`'s own comment).
+    max_context_tokens: int = 6000
     # symptom_to_drug: a common symptom can match far more drugs than is
     # useful to show in one answer.
     indication_candidate_limit: int = 8
@@ -94,6 +102,9 @@ class RetrievalService:
             if find_by_section is not None:
                 hits = find_by_section(drug_id, section_key)
                 if hits:
+                    hits = hits + self._pooled_neighbour_hits(
+                        query, drug_id, section_key, find_by_section
+                    )
                     return self._decide(self._hydrate(hits, limit=None))
 
         overview_hits = self._drug_overview(drug_id)
@@ -110,9 +121,12 @@ class RetrievalService:
         # Capped even when rerank is disabled/unavailable and fails open to
         # the unfiltered list — an ordering aid must never remove the size
         # bound too, or the same 29-section explosion returns through here.
-        return self._decide(
-            self._hydrate(overview_hits, limit=self._policy.evidence_limit)
-        )
+        # Token budget (not a flat count): rerank already put the best match
+        # first, so packing in that order keeps as much of it as a real
+        # model's context can hold instead of an arbitrary fixed count.
+        hydrated = self._hydrate(overview_hits, limit=None)
+        packed = pack_evidence(hydrated, max_tokens=self._policy.max_context_tokens)
+        return self._decide(packed.evidence)
 
     def retrieve(self, query: str, drug_id: str) -> RetrievalResult:
         if not query.strip() or not drug_id.strip():
@@ -235,6 +249,28 @@ class RetrievalService:
         hits = find_by_drug(drug_id)
         return hits or None
 
+    # Bounded: a query naming a specific condition can legitimately need one
+    # neighbouring section (than_trong -> chong_chi_dinh was the case found
+    # live); more than this starts pooling tangential sections rather than
+    # the one true answer, which is precision loss dressed as recall.
+    _MAX_LEXICAL_POOLED_SECTIONS = 2
+    # Measured live 2026-08-10 on the real corpus: a genuine neighbour match
+    # (chong_chi_dinh, the true positive) scored 7 matched terms; the same
+    # query's false-positive attractor scored 6 — close enough that no
+    # threshold alone separates them (see the exclusion below instead). 5
+    # keeps clearly-incidental overlap (3-4, seen on unrelated sections in
+    # the same measurement) out while still admitting real matches.
+    _LEXICAL_POOL_MIN_SCORE = 5.0
+    # `duoc_ly_va_co_che_tac_dung` is the corpus's documented false-positive
+    # attractor (see `sections.py`'s own module docstring: it's the largest,
+    # most generic section and "sits close to any question about the drug")
+    # — true for embedding similarity there, and measured true for lexical
+    # overlap here too: it scored a close second (6) right behind the real
+    # answer (7) on the exact query that motivated this pooling mechanism.
+    # Excluded from pooling outright rather than trusting a score margin
+    # that isn't reliably wide enough on its own.
+    _LEXICAL_POOL_EXCLUDED_SECTIONS = frozenset({"duoc_ly_va_co_che_tac_dung"})
+
     def _section_hits(self, query: str, drug_id: str) -> list[SearchHit] | None:
         """Hits for an explicitly named section, or None to fall back.
 
@@ -250,19 +286,43 @@ class RetrievalService:
         if match is None:
             return None
         hits = find_by_section(drug_id, match.section_key)
-        if match.section_key == "than_trong":
-            # A "thận trọng" question about a specific condition sometimes has
-            # its real answer filed under "chống chỉ định" instead — found live
-            # 2026-08-10: Aspirin's own "thận trọng" text never says "loét dạ
-            # dày", the fact only exists in its "chống chỉ định" text ("loét
-            # dạ dày hoặc tá tràng đang hoạt động"). The two are the closest
-            # pair of "is this safe for my patient" categories the book has,
-            # and chống chỉ định text is short — pooling it costs nothing on
-            # a drug where than_trong already answers, and prevents a false
-            # "not in this source" clarify/abstain on one where it doesn't.
-            hits = hits + find_by_section(drug_id, "chong_chi_dinh")
+        hits = hits + self._pooled_neighbour_hits(query, drug_id, match.section_key, find_by_section)
         return hits or None
 
+    def _pooled_neighbour_hits(
+        self, query: str, drug_id: str, resolved_section: str, find_by_section
+    ) -> list[SearchHit]:
+        """Other sections of the SAME drug whose text lexically matches the
+        query strongly enough to suggest the resolved section alone may not
+        answer it — found live 2026-08-10: a "thận trọng" question about a
+        specific condition (loét dạ dày) had its real answer filed only
+        under "chống chỉ định" instead, a category the deterministic
+        keyword route never considers once "thận trọng" itself matched.
+        `search_lexical` generalizes the one hardcoded pairing this started
+        as into a query-driven check across every section, still bounded and
+        still whole-section (never a partial, out-of-context fragment).
+        """
+        search_lexical = getattr(self._retriever, "search_lexical", None)
+        if search_lexical is None:
+            return []
+        lexical_hits = search_lexical(query, drug_id, limit=20)
+        pooled: list[SearchHit] = []
+        ineligible = {resolved_section} | self._LEXICAL_POOL_EXCLUDED_SECTIONS
+        pooled_sections: set[str] = set()
+        for hit in lexical_hits:
+            section = hit.document.section_key
+            if (
+                section in ineligible
+                or section in pooled_sections
+                or hit.score < self._LEXICAL_POOL_MIN_SCORE
+            ):
+                continue
+            pooled_sections.add(section)
+            pooled.extend(find_by_section(drug_id, section))
+            if len(pooled_sections) >= self._MAX_LEXICAL_POOLED_SECTIONS:
+                break
+        return pooled
+
     def decide(self, evidence: tuple[Evidence, ...]) -> RetrievalResult:
         """Public entry point for a caller that assembles its own evidence
         pool across several `retrieve_framed` calls — e.g. `RagAgent`'s

apps/ai-service/tests/test_fusion.py

@@ -0,0 +1,57 @@
+import pytest
+
+from rag.fusion import reciprocal_rank_fusion
+from rag.models import RetrievalDocument, SearchHit, SourceRef
+
+
+SOURCE = SourceRef(physical_page=1, precision="page")
+
+
+def _hit(doc_id: str, score: float = 1.0) -> SearchHit:
+    return SearchHit(
+        RetrievalDocument(
+            doc_id=doc_id,
+            drug_id="drug",
+            kind="prose",
+            text=doc_id,
+            section_key="section",
+            source_refs=(SOURCE,),
+        ),
+        score,
+    )
+
+
+def test_rrf_promotes_candidate_supported_by_dense_and_lexical_rankings() -> None:
+    fused = reciprocal_rank_fusion(
+        [[_hit("dense-only"), _hit("shared")], [_hit("shared"), _hit("lexical-only")]],
+        rank_constant=60,
+    )
+    assert [hit.document.doc_id for hit in fused] == [
+        "shared", "dense-only", "lexical-only",
+    ]
+
+
+def test_rrf_deduplicates_a_document_within_one_ranking() -> None:
+    fused = reciprocal_rank_fusion(
+        [[_hit("duplicate"), _hit("duplicate")], [_hit("other")]],
+        rank_constant=10,
+    )
+    duplicate = next(hit for hit in fused if hit.document.doc_id == "duplicate")
+    assert duplicate.score == pytest.approx(1 / 11)
+
+
+def test_rrf_limit_and_ties_are_deterministic() -> None:
+    fused = reciprocal_rank_fusion(
+        [[_hit("first")], [_hit("second")]], rank_constant=60, limit=1,
+    )
+    assert [hit.document.doc_id for hit in fused] == ["first"]
+
+
+@pytest.mark.parametrize(
+    ("kwargs", "message"),
+    [({"rank_constant": 0}, "rank_constant"), ({"limit": -1}, "limit")],
+)
+def test_rrf_rejects_invalid_configuration(kwargs, message: str) -> None:
+    with pytest.raises(ValueError, match=message):
+        reciprocal_rank_fusion([], **kwargs)
+

apps/ai-service/tests/test_qdrant_adapter.py

@@ -159,3 +159,51 @@ def test_find_by_indication_respects_the_limit():
     hits = retriever.find_by_indication("đau", limit=2)
 
     assert len(hits) == 2
+
+
+def _section_payload(drug_id: str, section_key: str, text: str) -> dict:
+    return {
+        "chunk_id": f"{drug_id}__{section_key}__0", "drug_id": drug_id,
+        "drug_name": drug_id.upper(), "section_key": section_key,
+        "chunk_kind": "prose", "text": text,
+        "heading_physical_page": 100, "printed_page_range": [101, 101],
+    }
+
+
+def test_search_lexical_ranks_by_distinct_matched_term_count():
+    client = _FakeScrollClient([
+        _section_payload("aspirin", "than_trong", "Thận trọng với suy thận."),
+        _section_payload(
+            "aspirin", "chong_chi_dinh",
+            "Không dùng cho người có loét dạ dày tá tràng đang hoạt động.",
+        ),
+    ])
+    retriever = QdrantRetriever(client, "duocthu_v1", embedder=None)
+
+    hits = retriever.search_lexical(
+        "Thận trọng khi dùng aspirin cho bệnh nhân loét dạ dày", "aspirin", limit=5
+    )
+
+    assert [h.document.section_key for h in hits] == ["chong_chi_dinh", "than_trong"]
+    assert hits[0].score > hits[1].score
+
+
+def test_search_lexical_drops_stopword_only_queries():
+    client = _FakeScrollClient([
+        _section_payload("aspirin", "than_trong", "Thận trọng với suy thận."),
+    ])
+    retriever = QdrantRetriever(client, "duocthu_v1", embedder=None)
+
+    assert retriever.search_lexical("là gì và của", "aspirin", limit=5) == []
+
+
+def test_search_lexical_excludes_non_matching_sections():
+    client = _FakeScrollClient([
+        _section_payload("aspirin", "than_trong", "Thận trọng với suy thận."),
+        _section_payload("aspirin", "chi_dinh", "Giảm đau hạ sốt."),
+    ])
+    retriever = QdrantRetriever(client, "duocthu_v1", embedder=None)
+
+    hits = retriever.search_lexical("loét dạ dày", "aspirin", limit=5)
+
+    assert hits == []

apps/ai-service/tests/test_retrieval_service.py

@@ -59,10 +59,19 @@ def table_service(*, visual: bool = False) -> RetrievalService:
 class _OverviewRetriever:
     """A fake with `find_by_drug`/`find_by_section` (the Qdrant adapter's
     shape) — `InMemoryLexicalRetriever` doesn't implement either, so
-    `retrieve_framed`'s overview path is otherwise untestable."""
+    `retrieve_framed`'s overview path is otherwise untestable.
 
-    def __init__(self, documents: list[RetrievalDocument]) -> None:
+    `search_lexical` is scripted per test (`lexical_hits`), not real text
+    matching — this file is about `retrieve_framed`'s own wiring, not the
+    scorer (see `test_qdrant_adapter.py`/`test_section_routing.py` for
+    that)."""
+
+    def __init__(
+        self, documents: list[RetrievalDocument], lexical_hits: list[SearchHit] = ()
+    ) -> None:
         self._documents = documents
+        self._lexical_hits = lexical_hits
+        self.lexical_calls: list[tuple[str, str]] = []
 
     def find_by_drug(self, drug_id: str) -> list[SearchHit]:
         return [
@@ -80,6 +89,10 @@ class _OverviewRetriever:
     def search(self, query: str, drug_id: str, limit: int) -> list[SearchHit]:
         return []
 
+    def search_lexical(self, query: str, drug_id: str, limit: int) -> list[SearchHit]:  # noqa: ARG002
+        self.lexical_calls.append((query, drug_id))
+        return list(self._lexical_hits)
+
 
 _MONOGRAPH_SECTIONS = (
     "ten_chung_quoc_te", "ma_atc", "loai_thuoc", "dang_thuoc_va_ham_luong",
@@ -89,7 +102,7 @@ _MONOGRAPH_SECTIONS = (
 )
 
 
-def _monograph_service() -> RetrievalService:
+def _monograph_service(max_context_tokens: int = 6000) -> RetrievalService:
     documents = [
         RetrievalDocument(
             doc_id=f"paracetamol::{section}::0", drug_id="paracetamol",
@@ -100,7 +113,7 @@ def _monograph_service() -> RetrievalService:
     ]
     return RetrievalService(
         _OverviewRetriever(documents), InMemoryParentStore([]),
-        EvidencePolicy(evidence_limit=3),
+        EvidencePolicy(evidence_limit=3, max_context_tokens=max_context_tokens),
     )
 
 
@@ -123,13 +136,65 @@ def test_retrieve_framed_overview_answers_from_intro_sections_only():
 
 def test_retrieve_framed_question_without_section_is_capped_even_without_rerank():
     # No reranker configured: `_rerank` fails open and returns everything
-    # unfiltered. Hydration must still bound it — an ordering aid failing
-    # open must not also remove the size cap.
-    result = _monograph_service().retrieve_framed(
+    # unfiltered. Hydration must still bound it by TOKEN budget (2026-08-10:
+    # was a flat evidence_limit count, now pack_evidence) — an ordering aid
+    # failing open must not also remove the size cap.
+    result = _monograph_service(max_context_tokens=50).retrieve_framed(
         "paracetamol", None, "thuốc này có tác dụng phụ gì", is_overview=False
     )
     assert result.decision == EvidenceDecision.ANSWERABLE
-    assert len(result.evidence) <= 3
+    assert len(result.evidence) < len(_MONOGRAPH_SECTIONS)
+
+
+def test_retrieve_framed_packs_overview_by_token_budget_not_flat_count():
+    """2026-08-10 pipeline audit priority #1: a tight token budget can admit
+    FEWER than the old flat evidence_limit=3 when blocks are long, and a
+    generous one can admit MORE when blocks are short — proving this is
+    genuinely token-driven, not a renamed count cap."""
+    tiny_budget_result = _monograph_service(max_context_tokens=20).retrieve_framed(
+        "paracetamol", None, "thuốc này có tác dụng phụ gì", is_overview=False
+    )
+    generous_budget_result = _monograph_service(max_context_tokens=6000).retrieve_framed(
+        "paracetamol", None, "thuốc này có tác dụng phụ gì", is_overview=False
+    )
+    assert len(tiny_budget_result.evidence) < 3
+    assert len(generous_budget_result.evidence) > 3
+
+
+def test_retrieve_framed_pools_lexically_strong_neighbour_section():
+    """The LIVE agent path (`RagAgent` -> understanding -> `retrieve_framed`)
+    must get the same neighbour-pooling `retrieve()` does — found live
+    2026-08-10 that the first version of this fix only wired into
+    `retrieve()`, which the real HTTP request path does not call at all;
+    `retrieve_framed` has its own separate `if section_key:` branch."""
+    documents = [
+        RetrievalDocument(
+            doc_id=f"aspirin::{section}::0", drug_id="aspirin",
+            kind="prose", section_key=section,
+            text=f"Nội dung mục {section}.", source_refs=(SOURCE,),
+        )
+        for section in ("than_trong", "chong_chi_dinh")
+    ]
+    chong_chi_dinh_hit = SearchHit(
+        document=next(d for d in documents if d.section_key == "chong_chi_dinh"),
+        score=1.0,
+    )
+    retriever = _OverviewRetriever(documents, lexical_hits=[
+        SearchHit(document=chong_chi_dinh_hit.document, score=7.0),
+    ])
+    service = RetrievalService(retriever, InMemoryParentStore([]), EvidencePolicy())
+
+    result = service.retrieve_framed(
+        "aspirin", "than_trong",
+        "Thận trọng khi dùng aspirin cho bệnh nhân loét dạ dày là gì?",
+    )
+
+    assert result.decision == EvidenceDecision.ANSWERABLE
+    returned_sections = {e.matched_doc_id.split("::")[1] for e in result.evidence}
+    assert returned_sections == {"than_trong", "chong_chi_dinh"}
+    assert retriever.lexical_calls == [
+        ("Thận trọng khi dùng aspirin cho bệnh nhân loét dạ dày là gì?", "aspirin"),
+    ]
 
 
 def test_row_hit_hydrates_complete_parent_and_keeps_citation():

apps/ai-service/tests/test_section_routing.py

@@ -56,6 +56,22 @@ class SectionAwareRetriever:
         ]
 
 
+class LexicalAwareRetriever(SectionAwareRetriever):
+    """Adds `search_lexical`, scripted per test rather than doing real
+    text matching — this suite is about `_pooled_neighbour_hits`' bounding
+    and threshold logic, not the lexical scorer itself (see
+    `adapters/qdrant.py`'s own coverage for that)."""
+
+    def __init__(self, docs: list[RetrievalDocument], lexical_hits: list[SearchHit]) -> None:
+        super().__init__(docs)
+        self._lexical_hits = lexical_hits
+        self.lexical_calls: list[tuple[str, str]] = []
+
+    def search_lexical(self, query: str, drug_id: str, limit: int) -> list[SearchHit]:  # noqa: ARG002
+        self.lexical_calls.append((query, drug_id))
+        return self._lexical_hits
+
+
 class SimilarityOnlyRetriever:
     def __init__(self, docs: list[RetrievalDocument]) -> None:
         self._docs = docs
@@ -78,7 +94,8 @@ CONTRA = [
 INDICATION = [_doc("i1", "chi_dinh", "Giảm đau, hạ sốt, chống viêm.")]
 PHARMACOLOGY = [_doc("p1", "duoc_ly_va_co_che_tac_dung", "Ức chế cyclooxygenase.")]
 PRECAUTION = [_doc("t1", "than_trong", "Thận trọng với người suy thận.")]
-ALL_DOCS = CONTRA + INDICATION + PHARMACOLOGY + PRECAUTION
+INTERACTION = [_doc("x1", "tuong_tac_thuoc", "Tương tác với thuốc chống đông máu.")]
+ALL_DOCS = CONTRA + INDICATION + PHARMACOLOGY + PRECAUTION + INTERACTION
 
 
 def _service(retriever, resolver: SectionResolver | None) -> RetrievalService:
@@ -208,22 +225,89 @@ class TestSectionRouting:
         assert retriever.section_calls == []
         assert retriever.search_calls
 
-    def test_than_trong_also_pools_chong_chi_dinh(self) -> None:
+    def test_lexically_strong_neighbour_section_is_pooled_in(self) -> None:
         """A precaution some drug's own "thận trọng" text never mentions can
         still be filed under "chống chỉ định" (found live: Aspirin + loét dạ
-        dày). Pooling both keeps that answerable instead of a false "not in
-        this source" clarify/abstain."""
-        retriever = SectionAwareRetriever(ALL_DOCS)
+        dày). A neighbour section with real term overlap (score above
+        threshold) gets pooled in whole, not just the matching fragment."""
+        retriever = LexicalAwareRetriever(
+            ALL_DOCS, lexical_hits=[SearchHit(CONTRA[1], 6.0)],  # c2: "chong_chi_dinh"
+        )
         result = _service(retriever, SectionResolver()).retrieve(
-            "Thận trọng khi dùng aspirin là gì?", "aspirin"
+            "Thận trọng khi dùng aspirin cho bệnh nhân loét dạ dày là gì?", "aspirin"
         )
         assert retriever.section_calls == [
             ("aspirin", "than_trong"), ("aspirin", "chong_chi_dinh"),
         ]
+        assert retriever.lexical_calls == [
+            ("Thận trọng khi dùng aspirin cho bệnh nhân loét dạ dày là gì?", "aspirin"),
+        ]
+        # Whole section pooled (c1-c5), not just the one matching chunk (c2).
         assert {item.evidence_id for item in result.evidence} == {
             "t1", "c1", "c2", "c3", "c4", "c5",
         }
 
+    def test_weak_lexical_match_is_not_pooled(self) -> None:
+        """One incidental token overlap must not drag in a whole tangential
+        section — only real term overlap (>= threshold) counts."""
+        retriever = LexicalAwareRetriever(
+            ALL_DOCS, lexical_hits=[SearchHit(CONTRA[1], 4.0)],
+        )
+        result = _service(retriever, SectionResolver()).retrieve(
+            "Thận trọng khi dùng aspirin là gì?", "aspirin"
+        )
+        assert retriever.section_calls == [("aspirin", "than_trong")]
+        assert {item.evidence_id for item in result.evidence} == {"t1"}
+
+    def test_lexical_pooling_is_bounded(self) -> None:
+        """At most 2 neighbour sections pool in, even if more score above
+        threshold — unbounded pooling is precision loss dressed as recall."""
+        retriever = LexicalAwareRetriever(
+            ALL_DOCS,
+            lexical_hits=[
+                SearchHit(CONTRA[0], 7.0),      # chong_chi_dinh
+                SearchHit(INDICATION[0], 6.0),  # chi_dinh
+                SearchHit(INTERACTION[0], 5.0), # tuong_tac_thuoc — 3rd, over bound
+            ],
+        )
+        result = _service(retriever, SectionResolver()).retrieve(
+            "Thận trọng khi dùng aspirin là gì?", "aspirin"
+        )
+        assert retriever.section_calls == [
+            ("aspirin", "than_trong"),
+            ("aspirin", "chong_chi_dinh"),
+            ("aspirin", "chi_dinh"),
+        ]
+        # tuong_tac_thuoc (3rd-ranked lexical hit) never pooled — the bound
+        # (2 neighbours) already used by chong_chi_dinh + chi_dinh.
+        assert "x1" not in {item.evidence_id for item in result.evidence}
+
+    def test_generic_pharmacology_section_is_never_pooled(self) -> None:
+        """`duoc_ly_va_co_che_tac_dung` is the corpus's documented
+        false-positive attractor (see `sections.py`) — excluded outright,
+        even with a lexical score that would otherwise clear the threshold
+        (measured live: it scored a close second right behind the real
+        answer on the exact query that motivated this whole mechanism)."""
+        retriever = LexicalAwareRetriever(
+            ALL_DOCS, lexical_hits=[SearchHit(PHARMACOLOGY[0], 9.0)],
+        )
+        result = _service(retriever, SectionResolver()).retrieve(
+            "Thận trọng khi dùng aspirin là gì?", "aspirin"
+        )
+        assert retriever.section_calls == [("aspirin", "than_trong")]
+        assert "p1" not in {item.evidence_id for item in result.evidence}
+
+    def test_retriever_without_lexical_search_still_works(self) -> None:
+        """No `search_lexical` on the retriever (interface segregation, same
+        pattern as `find_by_section`/`find_by_drug`) — pooling is skipped,
+        not an error."""
+        retriever = SectionAwareRetriever(ALL_DOCS)
+        result = _service(retriever, SectionResolver()).retrieve(
+            "Thận trọng khi dùng aspirin là gì?", "aspirin"
+        )
+        assert retriever.section_calls == [("aspirin", "than_trong")]
+        assert {item.evidence_id for item in result.evidence} == {"t1"}
+
     def test_named_but_empty_section_falls_back(self) -> None:
         """A drug with no such section must not abstain — similarity still tries."""
         retriever = SectionAwareRetriever(INDICATION + PHARMACOLOGY)

apps/web/Dockerfile

@@ -21,4 +21,4 @@ ENV NODE_ENV=production
 COPY --from=build /repo /repo
 WORKDIR /repo/apps/web
 EXPOSE 3000
-CMD ["pnpm", "start", "--", "-p", "3000", "-H", "0.0.0.0"]
+CMD ["node_modules/.bin/next", "start", "-p", "3000", "-H", "0.0.0.0"]

apps/web/app/_components/Composer.tsx

@@ -40,20 +40,36 @@ export function Composer({
     }
   }, [initialValue]);
 
-  // Fetch suggestions from API route when query length > 1
+  // Fetch suggestions from API route when query length > 1.
+  //
+  // `term` is the LAST WORD being typed, not the whole textarea — a natural
+  // sentence like "cho tôi thuốc parace" has no drug alias containing that
+  // full phrase, so matching against it always returns empty (found live
+  // 2026-08-10: this silently killed every suggestion once a user typed a
+  // sentence instead of a bare drug name, which is the more common case).
   useEffect(() => {
-    const term = value.trim();
+    const words = value.trim().split(/\s+/);
+    const term = words[words.length - 1] ?? "";
     if (term.length < 2) {
       setSuggestions([]);
       setShowSuggestions(false);
       return;
     }
 
+    // Race guard: a slower request for an earlier keystroke must not
+    // overwrite a faster one for a later keystroke — found live 2026-08-10
+    // typing fast could show a stale, unrelated suggestion list because the
+    // fetch callback set state unconditionally regardless of whether `term`
+    // (and therefore `value`) had already changed by the time it resolved.
+    let cancelled = false;
+
     const timer = setTimeout(async () => {
       try {
         const res = await fetch(`/api/suggest?q=${encodeURIComponent(term)}`);
+        if (cancelled) return;
         if (res.ok) {
           const data = await res.json();
+          if (cancelled) return;
           if (data?.suggestions && Array.isArray(data.suggestions) && data.suggestions.length > 0) {
             setSuggestions(data.suggestions);
             setShowSuggestions(true);
@@ -64,15 +80,21 @@ export function Composer({
         const filtered = SAMPLE_SUGGESTIONS.filter((s) =>
           s.toLowerCase().includes(term.toLowerCase())
         );
+        if (cancelled) return;
         setSuggestions(filtered);
         setShowSuggestions(filtered.length > 0);
       } catch {
+        if (!cancelled) {
           setSuggestions([]);
           setShowSuggestions(false);
         }
+      }
     }, 200);
 
-    return () => clearTimeout(timer);
+    return () => {
+      cancelled = true;
+      clearTimeout(timer);
+    };
   }, [value]);
 
   // Close suggestions on outside click

coordination/README.md

@@ -105,6 +105,14 @@ self-hosted embedding/vLLM plan (assumption GĐ-3 in
 
 ## Active ownership
 
+- Codex parallel session: **STOPPED, 2026-08-10** — owner ended the session.
+  Left `rag/expansion.py` and `rag/context.py` finished and tested but not
+  wired into any live retrieval path; `rag/fusion.py`/`tests/test_fusion.py`
+  (a third, separate ChatGPT session's work, per Codex's own note above)
+  likewise finished-but-unwired. Claude took over this scope at the owner's
+  explicit direction same session — see `project_production_deployment_live`
+  memory and `docs/progress-log.md` for what got wired in and why.
+
 - Claude: **PAUSED END OF SESSION, 2026-08-06** — worked the full
   correction order from `CODEX_RAG_CODE_REVIEW_2026-08-06.md`. **F-01
   through F-07 and F-09 done; F-08 and F-10 done for their core finding,

docs/progress-log.md

@@ -1,5 +1,96 @@
 # Progress Log
 
+## 2026-08-10 (cont. 17) — First production deployment: EC2 + Docker + CI/CD, live at realvuxbaro.me
+
+Owner and Codex agreed a work split mid-session
+(`coordination/WORK_SPLIT_2026-08-10.md`): Codex owns `rag/**` (multi-query,
+hybrid retrieval, fusion, guardrails); Claude owns deployment (Dockerfiles,
+runtime env, Compose, hosting, CI/CD). Stopped the in-flight RAG bug-fixing
+work (Aspirin/Warfarin/Vancomycin abstains, tasks 8/9) at the owner's
+direction and pivoted entirely to standing up a real production deployment,
+separate from the team's k3s/ArgoCD — a personal AWS account + a bought
+domain (`realvuxbaro.me`, Namecheap).
+
+**Infra provisioned** (`ai-lab-user`'s AWS account, confirmed personal by
+the owner, not the team-shared one memory previously described): EC2
+`t3.large` in us-east-1 (`i-039fc8f6102467a54`, Elastic IP `52.0.158.61`),
+a dedicated security group (22/80/443), and an IAM instance role
+(`duocthu-prod-ec2-role`) with the same Bedrock policies `ai-lab-user`
+has — no long-lived AWS access keys anywhere on the server or in any env
+file; boto3 picks up credentials from instance metadata.
+
+**Containerized for the first time** — neither app had a Dockerfile before
+today:
+- `apps/ai-service/Dockerfile`: the flat module layout (`rag/`, `adapters/`,
+  `routers/` all top-level) isn't pip-installable as a package — setuptools
+  rejects "multiple top-level packages" — so deps are pip-installed
+  directly instead of via `pip install .`, plus `boto3` (used for Bedrock,
+  never declared in `pyproject.toml`).
+- `apps/web/Dockerfile`: pnpm-workspace multi-stage build. Needed a new
+  root `.dockerignore` — a host `node_modules` from an earlier accidental
+  `npm run dev` (should have been `pnpm`) was copying over the container's
+  correctly pnpm-installed `node_modules` and breaking the Next.js build.
+- `infra/docker/docker-compose.prod.yml` + `Caddyfile`: single-box
+  topology — postgres, qdrant, ai-service, web, Caddy for automatic Let's
+  Encrypt SSL. Redis/Prometheus/Grafana left out (redis is unused anywhere
+  in the live path; observability can come back later).
+
+**Two real crash-loop bugs found and fixed, both packaging-assumption
+bugs, neither RAG logic**:
+1. `config.py`'s `entities_path` default did
+   `Path(__file__).resolve().parents[2]` to find the repo root and load
+   `ingestion/data/verified/drug_entities.json` — assumed a full monorepo
+   checkout depth. The deploy image flattens `apps/ai-service/` into `/app`,
+   so this raised `IndexError` at class-definition time, before any env
+   override could apply — crashed the container on every single start.
+   Fixed with a depth-guarded fallback in a small `_default_entities_path()`
+   helper, plus the file itself baked into the image and pointed at via
+   `ENTITIES_PATH` in `.env.prod`. 202/202 tests still pass.
+2. `web`'s `CMD ["pnpm", "start", "--", "-p", "3000", "-H", "0.0.0.0"]`
+   didn't forward the flags through pnpm to `next start` in this pnpm
+   version — `next` received `-p` as a literal project-directory argument
+   and crashed every time. Fixed by invoking `next`'s own binary directly,
+   sidestepping pnpm's arg-forwarding.
+
+**Qdrant data migrated via snapshot, not re-embedded** — free and exact,
+no new Bedrock spend: snapshotted both `duocthu_v1` (15,100 points) and
+`duocthu_v1__manifest` locally, scp'd the ~118MB total to the server, and
+restored via Qdrant's multipart `/snapshots/upload` endpoint (first attempt
+used `PUT` with a raw body per a wrong guess at the API shape — 404;
+`POST` with `-F` multipart is correct). Verified live on the server after
+restore: `points_count: 15100`, `status: green` — and confirmed it
+survived Compose recreating the container afterward (named volume, not the
+container, holds the data).
+
+**DNS**: `realvuxbaro.me`'s existing Namecheap ALIAS/CNAME records (pointed
+at Namecheap's own parking page) replaced with `A` records for `@` and
+`www` → `52.0.158.61`. Caddy's automatic ACME issuance failed twice before
+DNS propagated (expected — logged, not a bug), then succeeded within
+seconds of a manual restart once `nslookup` confirmed propagation.
+
+**End-to-end live-verified in the actual browser over real HTTPS**
+(not curl): `https://realvuxbaro.me` — asked about Amoxicillin
+contraindications, got correctly routed to `AMOXICILIN` and a genuine
+absolute-vs-relative clarifying question with working quick-reply chips.
+Full round trip through Caddy → web → ai-service → Qdrant/Postgres →
+Bedrock, all over the public domain.
+
+**CI/CD**: `.github/workflows/deploy.yml` — push to `master` SSHes into
+the box (key + host in GitHub Actions secrets, `EC2_SSH_KEY`/`EC2_HOST`),
+`git reset --hard origin/master`, rebuilds+restarts only `ai-service`/`web`
+(postgres/qdrant/caddy untouched), runs migrations, health-checks both
+services. First real run (triggered by its own commit) succeeded in 21s;
+site confirmed still up and correct after.
+
+**Explicitly deferred, not done this session**: no real k3s/ArgoCD (owner
+asked about it mid-session, decided current Docker-image approach is easy
+to migrate to later since the hard part — containerizing — is already
+done); the 3 persistent RAG abstains and 2 minor precision bugs from
+cont. 13's audit are untouched, back with Codex per the work split;
+`api-gateway`/`auth-service`/`chat-service` still unbuilt scaffolds — this
+deployment is `web` talking directly to `ai-service`, same as local dev,
+now just reachable over the internet with no additional auth layer.
+
 ## 2026-08-10 (cont. 16) — Recovered from the machine-trouble cutoff: Bug 2 live-reverified, quarantine path conclusively exercised, retry rate remeasured
 
 Picked up exactly where cont. 15 left off. Docker Desktop was down (machine