Add read-only production runtime audit

docs-legacy/26-known-limitations.md

@@ -0,0 +1,162 @@
+# 26 — Known limitations
+
+Objective statement of what is incomplete, fragile or unverified. Debt with a
+suggested remediation is in [27-technical-debt.md](27-technical-debt.md); this
+page is the honest inventory.
+
+## Product scope
+
+- **Only Part 2 of the book is ingested** (printed pages 99–1496, 684
+  monographs). Part 1 general chapters — special-population guidance, poisoning
+  management, interaction principles — and Part 3 appendices — BSA table, IV
+  preparation, ATC index — are excluded by construction
+  (`segment/detector.py`). Questions about them abstain, which is correct but is
+  a real coverage gap for a clinician.
+- **No reverse relations.** "Which drugs cause X" and "which drugs are
+  contraindicated in X" are routed to an explicit abstain.
+- **No dose calculation.** `rag/calculators.py` implements the book's own DuBois
+  BSA formula and is tested, but **no runtime code calls it**, so a BSA-based
+  dose still depends on a quarantined table the system will not read.
+- **No recommendation or ranking**, by design (prompt rule 10) — but this is
+  enforced only by the prompt, not machine-checked.
+
+## Unfinished services
+
+`apps/api-gateway`, `apps/auth-service`, `apps/user-service`,
+`apps/chat-service` and `apps/mobile` contain a `README.md` and (for four of
+them) a four-line `package.json`. There is no source. Consequences:
+
+- no authentication or authorization anywhere;
+- no user accounts, no per-user history, no session ownership;
+- rate limiting lives in the frontend because the gateway that should own it
+  does not exist;
+- `infra/k8s/base/{api-gateway,auth-service,chat-service,user-service}/` are
+  empty placeholder directories.
+
+## Safety and correctness caveats
+
+- **The entailment judge is one LLM pass.** Deliberate (repeating a
+  temperature-0 prompt is a correlated retry, not an independent vote), but it
+  means a single false acceptance is not caught by redundancy — and its accuracy
+  is not measured by any committed eval run.
+- **Quarantined content is surfaced, not reconstructed.** 151 block descriptors
+  exist; their numbers are unavailable to the system. A dosing table the
+  clinician needs may simply not be answerable.
+- **Table row/column reconstruction is unverified**, and recall for borderless
+  tables and bar-less formulas is unquantified — `cli chunk-ready` says so in
+  its own output.
+- **No whole-document human-reviewed ground truth exists**, so content accuracy
+  against the source is not proven by any gate.
+- **`prose_text` vs `text`.** The retrieval payload embeds `text`, which may
+  carry repeated context labels. That is deliberate for retrieval, but it means
+  the embedded string is not byte-identical to the book.
+- **Grounding cannot check non-numeric semantics** — that is the entailment
+  pass's job, and it is the weaker of the two checks.
+
+## Retrieval limitations
+
+- **Dense search is used in exactly one place**: the indication fallback. A
+  question phrased unlike the book, about a drug's section, relies on the
+  keyword section resolver or on rerank over the whole monograph.
+- **No hybrid search.** `rag/fusion.py` (RRF) is implemented and tested but has
+  no runtime caller.
+- **No query expansion / multi-query.** `rag/expansion.py` likewise.
+- **`search_lexical` is not BM25** — its score is the count of distinct matched
+  tokens, with no term frequency, IDF or length normalisation.
+- **`text` is not in `INDEXED_PAYLOAD_FIELDS`**, yet `search_lexical` issues
+  `MatchText` conditions against it. Qdrant needs an explicit full-text index
+  for that; the effective behaviour of those filters on the deployed collection
+  was not verified in this pass.
+- **Cross-section pooling is enabled for `than_trong` only**, on the strength of
+  one measured case. The same class of miss in other sections is not covered.
+- **Parent/child hydration is inert** — no chunk in the corpus sets `parent_id`.
+- **`atc_codes` is indexed but never queried.**
+
+## Conversation and state
+
+- `RagAgent._last_frame` and `_clarify_streak` are **in-process dicts**. They are
+  lost on restart and not shared across replicas, so multi-turn quality degrades
+  silently if `ai-service` is scaled horizontally — nothing detects this.
+- `conversation_id` is an unauthenticated, client-chosen string with no
+  ownership check; anyone who guesses one reads its history into their prompt.
+- `rag_conversation_turn` grows without bound. No retention, no deletion.
+
+## Performance
+
+- **No streaming.** The UI shows a spinner for the whole turn. Measured (n=8,
+  one user, sequential, 2026-08-11): 6.2–40.3 s.
+- **No caching of any kind at request time** — identical questions re-pay for
+  every model call.
+- Sequential model calls: 3 on the happy path, up to 8 under the budget.
+- `CatalogDrugResolver` is O(catalog) on a fuzzy miss; the `lru_cache` fixes
+  repeat lookups but a genuinely new typo still costs ~1 s of CPU.
+- `/api/pdf` reads a 37 MB file into memory per request, with no range support,
+  no caching headers, and **no rate limit** (the middleware has no rule for that
+  prefix).
+
+## Testing and evaluation
+
+- **Zero frontend tests.** Every hard-won fix in `ChatPanel.tsx`,
+  `middleware.ts` and `route.ts` — the 65 s timeout derivation, the abort
+  handling, the Strict-Mode duplicate guard, the `REFUSALS` map, citation
+  grouping — can regress silently.
+- **CI does not mechanically gate deploy.** `ci.yml` runs Python tests and web
+  lint/build, but `deploy.yml` triggers independently on matching `master`
+  changes; a red CI run does not itself cancel or block deploy.
+- `apps/ai-service` tests cannot be collected without `EMBEDDING_PROVIDER=disabled`
+  or a reachable Qdrant, and that is documented nowhere in the repository.
+- **No evaluation runner.** 209 golden rows and 90 JSONL cases exist; nothing
+  executes them and no metric is tracked over time. No regression gate.
+- No load, performance or security testing.
+- Migrations are never exercised by a test.
+- The Helm chart is never rendered or linted.
+
+## Deployment and operations
+
+- Images are built on the production host and untagged, so **rollback requires
+  a rebuild** and there is no known-good artifact.
+- Migrations are forward-only; a rollback across one is uncovered.
+- No staging environment is actually deployed.
+- No backup automation for PostgreSQL or Qdrant.
+- `qdrant/qdrant:latest` is unpinned.
+- There is **no Python lockfile**; the Dockerfile installs unpinned ranges
+  (`"boto3"` has no bound at all), so two builds of the same commit can differ.
+- The Kubernetes/ArgoCD path is written but unapplied, with three `TODO`
+  placeholders per environment and no image registry.
+
+## Security
+
+Full detail in [16-security.md](16-security.md). Headline gaps: no
+authentication, no authorization, no conversation ownership, a committed default
+PostgreSQL credential, containers running as root, no security context or
+NetworkPolicy in the chart, no dependency scanning, no security headers, and no
+retention or redaction for user-supplied patient context.
+
+## Observability
+
+- **No alerting at all** — no Alertmanager, no rule files, no Grafana alerts.
+- **No log aggregation** and no structured logging; `agent.py` logs routine
+  timings at WARNING because uvicorn does not wire the root logger.
+- **`web` is entirely uninstrumented.**
+- Four metric names are registered but never incremented.
+- No SLOs or error budgets.
+
+## Documentation/code discrepancies
+
+Found by comparing the pre-existing documents against the code. The code wins in
+every case.
+
+| Claim | Where | Reality |
+|---|---|---|
+| "conversation history is an in-process dict per `RagAgent`, not yet durable" | `docs/architecture.md` service table | `PostgresConversationStore` **is** wired in `bootstrap.py` and backs `recent()`/`append()`. Only `_last_frame` and `_clarify_streak` remain in-process |
+| "`web` … Calls api-gateway only" | `docs/architecture.md` service table | `web` calls `ai-service` directly via `AI_SERVICE_URL`; no gateway exists |
+| "Qwen3 via the Converse API for understanding/generation/entailment" | `docs/architecture.md` | The model is configuration. Code default `deepseek.v3.2`; local `.env` `qwen.qwen3-next-80b-a3b`; production value is in an uncommitted `.env.prod` and **cannot be verified from the repository** |
+| api-gateway / auth-service / user-service / chat-service described with owned responsibilities and data | `docs/architecture.md` service table | Not built. The document does flag this elsewhere, but the table reads as current state |
+| Redis "session/refresh-token cache, rate-limit counters" | `docs/architecture.md` | No Redis client is imported anywhere. Present only in the local-dev Compose file |
+| ADR 0007's `Focus`/`ConversationState` and the bounded PLAN/RETRIEVE/ASSESS/REFINE/VERIFY loop | `docs/adr/0007` | Superseded by ADR 0008; `rag/conversation.py` and `rag/reasoning.py` no longer exist. The `LOOP_*` metric names survive as dead constants |
+| `infra/ci/github-actions/README.md` lists five CI workflows | that README | None exists; the only workflow is `deploy.yml` |
+| ADR 0005 "Contract/schema only — no implementation" | `docs/adr/0005` | The contract is implemented — `segment/models.py` and `chunk/` both follow it |
+
+Completed planning documents and superseded pipeline audits have been removed.
+Use `pipeline-tu-pdf-den-chatbot-production.md` and the numbered documentation
+for current behaviour; use ADRs and `git log` for historical intent.