duocthu/docs/26-known-limitations.md

# 26 — Known limitations

Objective statement of what is incomplete, fragile or unverified. Debt with a
suggested remediation is in [27-technical-debt.md](27-technical-debt.md); this
page is the honest inventory.

## Product scope

- **Only Part 2 of the book is ingested** (printed pages 99–1496, 684
  monographs). Part 1 general chapters — special-population guidance, poisoning
  management, interaction principles — and Part 3 appendices — BSA table, IV
  preparation, ATC index — are excluded by construction
  (`segment/detector.py`). Questions about them abstain, which is correct but is
  a real coverage gap for a clinician.
- **No reverse relations.** "Which drugs cause X" and "which drugs are
  contraindicated in X" are routed to an explicit abstain.
- **No dose calculation.** `rag/calculators.py` implements the book's own DuBois
  BSA formula and is tested, but **no runtime code calls it**, so a BSA-based
  dose still depends on a quarantined table the system will not read.
- **No recommendation or ranking**, by design (prompt rule 10) — but this is
  enforced only by the prompt, not machine-checked.

## Unfinished services

`apps/api-gateway`, `apps/auth-service`, `apps/user-service`,
`apps/chat-service` and `apps/mobile` contain a `README.md` and (for four of
them) a four-line `package.json`. There is no source. Consequences:

- no authentication or authorization anywhere;
- no user accounts, no per-user history, no session ownership;
- rate limiting lives in the frontend because the gateway that should own it
  does not exist;
- `infra/k8s/base/{api-gateway,auth-service,chat-service,user-service}/` are
  empty placeholder directories.

## Safety and correctness caveats

- **The entailment judge is one LLM pass.** Deliberate (repeating a
  temperature-0 prompt is a correlated retry, not an independent vote), but it
  means a single false acceptance is not caught by redundancy — and its accuracy
  is not measured by any committed eval run.
- **Quarantined content is surfaced, not reconstructed.** 151 block descriptors
  exist; their numbers are unavailable to the system. A dosing table the
  clinician needs may simply not be answerable.
- **Table row/column reconstruction is unverified**, and recall for borderless
  tables and bar-less formulas is unquantified — `cli chunk-ready` says so in
  its own output.
- **No whole-document human-reviewed ground truth exists**, so content accuracy
  against the source is not proven by any gate.
- **`prose_text` vs `text`.** The retrieval payload embeds `text`, which may
  carry repeated context labels. That is deliberate for retrieval, but it means
  the embedded string is not byte-identical to the book.
- **Grounding cannot check non-numeric semantics** — that is the entailment
  pass's job, and it is the weaker of the two checks.

## Retrieval limitations

- **Dense search is used in exactly one place**: the indication fallback. A
  question phrased unlike the book, about a drug's section, relies on the
  keyword section resolver or on rerank over the whole monograph.
- **No hybrid search.** `rag/fusion.py` (RRF) is implemented and tested but has
  no runtime caller.
- **No query expansion / multi-query.** `rag/expansion.py` likewise.
- **`search_lexical` is not BM25** — its score is the count of distinct matched
  tokens, with no term frequency, IDF or length normalisation.
- **`text` is not in `INDEXED_PAYLOAD_FIELDS`**, yet `search_lexical` issues
  `MatchText` conditions against it. Qdrant needs an explicit full-text index
  for that; the effective behaviour of those filters on the deployed collection
  was not verified in this pass.
- **Cross-section pooling is enabled for `than_trong` only**, on the strength of
  one measured case. The same class of miss in other sections is not covered.
- **Parent/child hydration is inert** — no chunk in the corpus sets `parent_id`.
- **`atc_codes` is indexed but never queried.**

## Conversation and state

- `RagAgent._last_frame` and `_clarify_streak` are **in-process dicts**. They are
  lost on restart and not shared across replicas, so multi-turn quality degrades
  silently if `ai-service` is scaled horizontally — nothing detects this.
- `conversation_id` is an unauthenticated, client-chosen string with no
  ownership check; anyone who guesses one reads its history into their prompt.
- `rag_conversation_turn` grows without bound. No retention, no deletion.

## Performance

- **No streaming.** The UI shows a spinner for the whole turn. Measured (n=8,
  one user, sequential, 2026-08-11): 6.2–40.3 s.
- **No caching of any kind at request time** — identical questions re-pay for
  every model call.
- Sequential model calls: 3 on the happy path, up to 8 under the budget.
- `CatalogDrugResolver` is O(catalog) on a fuzzy miss; the `lru_cache` fixes
  repeat lookups but a genuinely new typo still costs ~1 s of CPU.
- `/api/pdf` reads a 37 MB file into memory per request, with no range support,
  no caching headers, and **no rate limit** (the middleware has no rule for that
  prefix).

## Testing and evaluation

- **Zero frontend tests.** Every hard-won fix in `ChatPanel.tsx`,
  `middleware.ts` and `route.ts` — the 65 s timeout derivation, the abort
  handling, the Strict-Mode duplicate guard, the `REFUSALS` map, citation
  grouping — can regress silently.
- **No test runs in CI.** A commit that breaks all 555 tests still deploys.
- `apps/ai-service` tests cannot be collected without `EMBEDDING_PROVIDER=disabled`
  or a reachable Qdrant, and that is documented nowhere in the repository.
- **No evaluation runner.** 209 golden rows and 90 JSONL cases exist; nothing
  executes them and no metric is tracked over time. No regression gate.
- No load, performance or security testing.
- Migrations are never exercised by a test.
- The Helm chart is never rendered or linted.

## Deployment and operations

- Images are built on the production host and untagged, so **rollback requires
  a rebuild** and there is no known-good artifact.
- Migrations are forward-only; a rollback across one is uncovered.
- No staging environment is actually deployed.
- No backup automation for PostgreSQL or Qdrant.
- `qdrant/qdrant:latest` is unpinned.
- There is **no Python lockfile**; the Dockerfile installs unpinned ranges
  (`"boto3"` has no bound at all), so two builds of the same commit can differ.
- The Kubernetes/ArgoCD path is written but unapplied, with three `TODO`
  placeholders per environment and no image registry.

## Security

Full detail in [16-security.md](16-security.md). Headline gaps: no
authentication, no authorization, no conversation ownership, a committed default
PostgreSQL credential, containers running as root, no security context or
NetworkPolicy in the chart, no dependency scanning, no security headers, and no
retention or redaction for user-supplied patient context.

## Observability

- **No alerting at all** — no Alertmanager, no rule files, no Grafana alerts.
- **No log aggregation** and no structured logging; `agent.py` logs routine
  timings at WARNING because uvicorn does not wire the root logger.
- **`web` is entirely uninstrumented.**
- Four metric names are registered but never incremented.
- No SLOs or error budgets.

## Documentation/code discrepancies

Found by comparing the pre-existing documents against the code. The code wins in
every case.

| Claim | Where | Reality |
|---|---|---|
| "conversation history is an in-process dict per `RagAgent`, not yet durable" | `docs/architecture.md` service table | `PostgresConversationStore` **is** wired in `bootstrap.py` and backs `recent()`/`append()`. Only `_last_frame` and `_clarify_streak` remain in-process |
| "`web` … Calls api-gateway only" | `docs/architecture.md` service table | `web` calls `ai-service` directly via `AI_SERVICE_URL`; no gateway exists |
| "Qwen3 via the Converse API for understanding/generation/entailment" | `docs/architecture.md` | The model is configuration. Code default `deepseek.v3.2`; local `.env` `qwen.qwen3-next-80b-a3b`; production value is in an uncommitted `.env.prod` and **cannot be verified from the repository** |
| api-gateway / auth-service / user-service / chat-service described with owned responsibilities and data | `docs/architecture.md` service table | Not built. The document does flag this elsewhere, but the table reads as current state |
| Redis "session/refresh-token cache, rate-limit counters" | `docs/architecture.md` | No Redis client is imported anywhere. Present only in the local-dev Compose file |
| "`fusion.py`/`context.py`/`expand_siblings` are dead code" | `docs/current-rag-pipeline-audit.md` | `context.py::pack_evidence` **is** now wired into `RetrievalService.retrieve_framed`. `fusion.py` and `expansion.py` remain unwired |
| "trace has no per-stage timing" | `docs/current-rag-pipeline-audit.md` | `telemetry.stage()` now emits `duocthu_stage_duration_seconds` and per-stage spans |
| ADR 0007's `Focus`/`ConversationState` and the bounded PLAN/RETRIEVE/ASSESS/REFINE/VERIFY loop | `docs/adr/0007` | Superseded by ADR 0008; `rag/conversation.py` and `rag/reasoning.py` no longer exist. The `LOOP_*` metric names survive as dead constants |
| `infra/ci/github-actions/README.md` lists five CI workflows | that README | None exists; the only workflow is `deploy.yml` |
| ADR 0005 "Contract/schema only — no implementation" | `docs/adr/0005` | The contract is implemented — `segment/models.py` and `chunk/` both follow it |

Dated planning documents (`v1-delivery-plan.md`, `rag-rebuild-plan.md`,
`answer-experience-implementation-plan.md`,
`condition-to-drug-audit-and-design.md`, `full-coverage-parsing-plan.md`) record
intent on their date. They were not audited line-by-line here; treat them as
history, not status.