duocthu/docs/26-known-limitations.md

26 — Known limitations

Objective statement of what is incomplete, fragile or unverified. Debt with a suggested remediation is in 27-technical-debt.md; this page is the honest inventory.

Product scope

• Only Part 2 of the book is ingested (printed pages 99–1496, 684 monographs). Part 1 general chapters — special-population guidance, poisoning management, interaction principles — and Part 3 appendices — BSA table, IV preparation, ATC index — are excluded by construction (segment/detector.py). Questions about them abstain, which is correct but is a real coverage gap for a clinician.
• No reverse relations. "Which drugs cause X" and "which drugs are contraindicated in X" are routed to an explicit abstain.
• No dose calculation. rag/calculators.py implements the book's own DuBois BSA formula and is tested, but no runtime code calls it, so a BSA-based dose still depends on a quarantined table the system will not read.
• No recommendation or ranking, by design (prompt rule 10) — but this is enforced only by the prompt, not machine-checked.

Unfinished services

apps/api-gateway, apps/auth-service, apps/user-service, apps/chat-service and apps/mobile contain a README.md and (for four of them) a four-line package.json. There is no source. Consequences:

• no authentication or authorization anywhere;
• no user accounts, no per-user history, no session ownership;
• rate limiting lives in the frontend because the gateway that should own it does not exist;
• infra/k8s/base/{api-gateway,auth-service,chat-service,user-service}/ are empty placeholder directories.

Safety and correctness caveats

• The entailment judge is one LLM pass. Deliberate (repeating a temperature-0 prompt is a correlated retry, not an independent vote), but it means a single false acceptance is not caught by redundancy — and its accuracy is not measured by any committed eval run.
• Quarantined content is surfaced, not reconstructed. 151 block descriptors exist; their numbers are unavailable to the system. A dosing table the clinician needs may simply not be answerable.
• Table row/column reconstruction is unverified, and recall for borderless tables and bar-less formulas is unquantified — cli chunk-ready says so in its own output.
• No whole-document human-reviewed ground truth exists, so content accuracy against the source is not proven by any gate.
• prose_text vs text. The retrieval payload embeds text, which may carry repeated context labels. That is deliberate for retrieval, but it means the embedded string is not byte-identical to the book.
• Grounding cannot check non-numeric semantics — that is the entailment pass's job, and it is the weaker of the two checks.

Retrieval limitations

• Dense search is used in exactly one place: the indication fallback. A question phrased unlike the book, about a drug's section, relies on the keyword section resolver or on rerank over the whole monograph.
• No hybrid search. rag/fusion.py (RRF) is implemented and tested but has no runtime caller.
• No query expansion / multi-query. rag/expansion.py likewise.
• search_lexical is not BM25 — its score is the count of distinct matched tokens, with no term frequency, IDF or length normalisation.
• text is not in INDEXED_PAYLOAD_FIELDS, yet search_lexical issues MatchText conditions against it. Qdrant needs an explicit full-text index for that; the effective behaviour of those filters on the deployed collection was not verified in this pass.
• Cross-section pooling is enabled for than_trong only, on the strength of one measured case. The same class of miss in other sections is not covered.
• Parent/child hydration is inert — no chunk in the corpus sets parent_id.
• atc_codes is indexed but never queried.

Conversation and state

• RagAgent._last_frame and _clarify_streak are in-process dicts. They are lost on restart and not shared across replicas, so multi-turn quality degrades silently if ai-service is scaled horizontally — nothing detects this.
• conversation_id is an unauthenticated, client-chosen string with no ownership check; anyone who guesses one reads its history into their prompt.
• rag_conversation_turn grows without bound. No retention, no deletion.

Performance

• No streaming. The UI shows a spinner for the whole turn. Measured (n=8, one user, sequential, 2026-08-11): 6.2–40.3 s.
• No caching of any kind at request time — identical questions re-pay for every model call.
• Sequential model calls: 3 on the happy path, up to 8 under the budget.
• CatalogDrugResolver is O(catalog) on a fuzzy miss; the lru_cache fixes repeat lookups but a genuinely new typo still costs ~1 s of CPU.
• /api/pdf reads a 37 MB file into memory per request, with no range support, no caching headers, and no rate limit (the middleware has no rule for that prefix).

Testing and evaluation

• Zero frontend tests. Every hard-won fix in ChatPanel.tsx, middleware.ts and route.ts — the 65 s timeout derivation, the abort handling, the Strict-Mode duplicate guard, the REFUSALS map, citation grouping — can regress silently.
• No test runs in CI. A commit that breaks all 555 tests still deploys.
• apps/ai-service tests cannot be collected without EMBEDDING_PROVIDER=disabled or a reachable Qdrant, and that is documented nowhere in the repository.
• No evaluation runner. 209 golden rows and 90 JSONL cases exist; nothing executes them and no metric is tracked over time. No regression gate.
• No load, performance or security testing.
• Migrations are never exercised by a test.
• The Helm chart is never rendered or linted.

Deployment and operations

• Images are built on the production host and untagged, so rollback requires a rebuild and there is no known-good artifact.
• Migrations are forward-only; a rollback across one is uncovered.
• No staging environment is actually deployed.
• No backup automation for PostgreSQL or Qdrant.
• qdrant/qdrant:latest is unpinned.
• There is no Python lockfile; the Dockerfile installs unpinned ranges ("boto3" has no bound at all), so two builds of the same commit can differ.
• The Kubernetes/ArgoCD path is written but unapplied, with three TODO placeholders per environment and no image registry.

Security

Full detail in 16-security.md. Headline gaps: no authentication, no authorization, no conversation ownership, a committed default PostgreSQL credential, containers running as root, no security context or NetworkPolicy in the chart, no dependency scanning, no security headers, and no retention or redaction for user-supplied patient context.

Observability

• No alerting at all — no Alertmanager, no rule files, no Grafana alerts.
• No log aggregation and no structured logging; agent.py logs routine timings at WARNING because uvicorn does not wire the root logger.
• web is entirely uninstrumented.
• Four metric names are registered but never incremented.
• No SLOs or error budgets.

Documentation/code discrepancies

Found by comparing the pre-existing documents against the code. The code wins in every case.

Claim	Where	Reality
"conversation history is an in-process dict per RagAgent, not yet durable"	docs/architecture.md service table	PostgresConversationStore is wired in bootstrap.py and backs recent()/append(). Only _last_frame and _clarify_streak remain in-process
"web … Calls api-gateway only"	docs/architecture.md service table	web calls ai-service directly via AI_SERVICE_URL; no gateway exists
"Qwen3 via the Converse API for understanding/generation/entailment"	docs/architecture.md	The model is configuration. Code default deepseek.v3.2; local .env qwen.qwen3-next-80b-a3b; production value is in an uncommitted .env.prod and cannot be verified from the repository
api-gateway / auth-service / user-service / chat-service described with owned responsibilities and data	docs/architecture.md service table	Not built. The document does flag this elsewhere, but the table reads as current state
Redis "session/refresh-token cache, rate-limit counters"	docs/architecture.md	No Redis client is imported anywhere. Present only in the local-dev Compose file
"fusion.py/context.py/expand_siblings are dead code"	docs/current-rag-pipeline-audit.md	context.py::pack_evidence is now wired into RetrievalService.retrieve_framed. fusion.py and expansion.py remain unwired
"trace has no per-stage timing"	docs/current-rag-pipeline-audit.md	telemetry.stage() now emits duocthu_stage_duration_seconds and per-stage spans
ADR 0007's Focus/ConversationState and the bounded PLAN/RETRIEVE/ASSESS/REFINE/VERIFY loop	docs/adr/0007	Superseded by ADR 0008; rag/conversation.py and rag/reasoning.py no longer exist. The LOOP_* metric names survive as dead constants
infra/ci/github-actions/README.md lists five CI workflows	that README	None exists; the only workflow is deploy.yml
ADR 0005 "Contract/schema only — no implementation"	docs/adr/0005	The contract is implemented — segment/models.py and chunk/ both follow it

Dated planning documents (v1-delivery-plan.md, rag-rebuild-plan.md, answer-experience-implementation-plan.md, condition-to-drug-audit-and-design.md, full-coverage-parsing-plan.md) record intent on their date. They were not audited line-by-line here; treat them as history, not status.