Fix migration workflow: upload as artifact instead of scp to practice EC2
This commit is contained in:
@@ -0,0 +1,164 @@
|
||||
# 26 — Known limitations
|
||||
|
||||
Objective statement of what is incomplete, fragile or unverified. Debt with a
|
||||
suggested remediation is in [27-technical-debt.md](27-technical-debt.md); this
|
||||
page is the honest inventory.
|
||||
|
||||
## Product scope
|
||||
|
||||
- **Only Part 2 of the book is ingested** (printed pages 99–1496, 684
|
||||
monographs). Part 1 general chapters — special-population guidance, poisoning
|
||||
management, interaction principles — and Part 3 appendices — BSA table, IV
|
||||
preparation, ATC index — are excluded by construction
|
||||
(`segment/detector.py`). Questions about them abstain, which is correct but is
|
||||
a real coverage gap for a clinician.
|
||||
- **No reverse relations.** "Which drugs cause X" and "which drugs are
|
||||
contraindicated in X" are routed to an explicit abstain.
|
||||
- **No dose calculation.** `rag/calculators.py` implements the book's own DuBois
|
||||
BSA formula and is tested, but **no runtime code calls it**, so a BSA-based
|
||||
dose still depends on a quarantined table the system will not read.
|
||||
- **No recommendation or ranking**, by design (prompt rule 10) — but this is
|
||||
enforced only by the prompt, not machine-checked.
|
||||
|
||||
## Unfinished services
|
||||
|
||||
`apps/api-gateway`, `apps/auth-service`, `apps/user-service`,
|
||||
`apps/chat-service` and `apps/mobile` contain a `README.md` and (for four of
|
||||
them) a four-line `package.json`. There is no source. Consequences:
|
||||
|
||||
- no authentication or authorization anywhere;
|
||||
- no user accounts, no per-user history, no session ownership;
|
||||
- rate limiting lives in the frontend because the gateway that should own it
|
||||
does not exist;
|
||||
- `infra/k8s/base/{api-gateway,auth-service,chat-service,user-service}/` are
|
||||
empty placeholder directories.
|
||||
|
||||
## Safety and correctness caveats
|
||||
|
||||
- **The entailment judge is one LLM pass.** Deliberate (repeating a
|
||||
temperature-0 prompt is a correlated retry, not an independent vote), but it
|
||||
means a single false acceptance is not caught by redundancy — and its accuracy
|
||||
is not measured by any committed eval run.
|
||||
- **Quarantined content is surfaced, not reconstructed.** 151 block descriptors
|
||||
exist; their numbers are unavailable to the system. A dosing table the
|
||||
clinician needs may simply not be answerable.
|
||||
- **Table row/column reconstruction is unverified**, and recall for borderless
|
||||
tables and bar-less formulas is unquantified — `cli chunk-ready` says so in
|
||||
its own output.
|
||||
- **No whole-document human-reviewed ground truth exists**, so content accuracy
|
||||
against the source is not proven by any gate.
|
||||
- **`prose_text` vs `text`.** The retrieval payload embeds `text`, which may
|
||||
carry repeated context labels. That is deliberate for retrieval, but it means
|
||||
the embedded string is not byte-identical to the book.
|
||||
- **Grounding cannot check non-numeric semantics** — that is the entailment
|
||||
pass's job, and it is the weaker of the two checks.
|
||||
|
||||
## Retrieval limitations
|
||||
|
||||
- **Dense search is used in exactly one place**: the indication fallback. A
|
||||
question phrased unlike the book, about a drug's section, relies on the
|
||||
keyword section resolver or on rerank over the whole monograph.
|
||||
- **No hybrid search.** `rag/fusion.py` (RRF) is implemented and tested but has
|
||||
no runtime caller.
|
||||
- **No query expansion / multi-query.** `rag/expansion.py` likewise.
|
||||
- **`search_lexical` is not BM25** — its score is the count of distinct matched
|
||||
tokens, with no term frequency, IDF or length normalisation.
|
||||
- **`text` is not in `INDEXED_PAYLOAD_FIELDS`**, yet `search_lexical` issues
|
||||
`MatchText` conditions against it. Qdrant needs an explicit full-text index
|
||||
for that; the effective behaviour of those filters on the deployed collection
|
||||
was not verified in this pass.
|
||||
- **Cross-section pooling is enabled for `than_trong` only**, on the strength of
|
||||
one measured case. The same class of miss in other sections is not covered.
|
||||
- **Parent/child hydration is inert** — no chunk in the corpus sets `parent_id`.
|
||||
- **`atc_codes` is indexed but never queried.**
|
||||
|
||||
## Conversation and state
|
||||
|
||||
- `RagAgent._last_frame` and `_clarify_streak` are **in-process dicts**. They are
|
||||
lost on restart and not shared across replicas, so multi-turn quality degrades
|
||||
silently if `ai-service` is scaled horizontally — nothing detects this.
|
||||
- `conversation_id` is an unauthenticated, client-chosen string with no
|
||||
ownership check; anyone who guesses one reads its history into their prompt.
|
||||
- `rag_conversation_turn` grows without bound. No retention, no deletion.
|
||||
|
||||
## Performance
|
||||
|
||||
- **No streaming.** The UI shows a spinner for the whole turn. Measured (n=8,
|
||||
one user, sequential, 2026-08-11): 6.2–40.3 s.
|
||||
- **No caching of any kind at request time** — identical questions re-pay for
|
||||
every model call.
|
||||
- Sequential model calls: 3 on the happy path, up to 8 under the budget.
|
||||
- `CatalogDrugResolver` is O(catalog) on a fuzzy miss; the `lru_cache` fixes
|
||||
repeat lookups but a genuinely new typo still costs ~1 s of CPU.
|
||||
- `/api/pdf` reads a 37 MB file into memory per request, with no range support,
|
||||
no caching headers, and **no rate limit** (the middleware has no rule for that
|
||||
prefix).
|
||||
|
||||
## Testing and evaluation
|
||||
|
||||
- **Zero frontend tests.** Every hard-won fix in `ChatPanel.tsx`,
|
||||
`middleware.ts` and `route.ts` — the 65 s timeout derivation, the abort
|
||||
handling, the Strict-Mode duplicate guard, the `REFUSALS` map, citation
|
||||
grouping — can regress silently.
|
||||
- **No test runs in CI.** A commit that breaks all 555 tests still deploys.
|
||||
- `apps/ai-service` tests cannot be collected without `EMBEDDING_PROVIDER=disabled`
|
||||
or a reachable Qdrant, and that is documented nowhere in the repository.
|
||||
- **No evaluation runner.** 209 golden rows and 90 JSONL cases exist; nothing
|
||||
executes them and no metric is tracked over time. No regression gate.
|
||||
- No load, performance or security testing.
|
||||
- Migrations are never exercised by a test.
|
||||
- The Helm chart is never rendered or linted.
|
||||
|
||||
## Deployment and operations
|
||||
|
||||
- Images are built on the production host and untagged, so **rollback requires
|
||||
a rebuild** and there is no known-good artifact.
|
||||
- Migrations are forward-only; a rollback across one is uncovered.
|
||||
- No staging environment is actually deployed.
|
||||
- No backup automation for PostgreSQL or Qdrant.
|
||||
- `qdrant/qdrant:latest` is unpinned.
|
||||
- There is **no Python lockfile**; the Dockerfile installs unpinned ranges
|
||||
(`"boto3"` has no bound at all), so two builds of the same commit can differ.
|
||||
- The Kubernetes/ArgoCD path is written but unapplied, with three `TODO`
|
||||
placeholders per environment and no image registry.
|
||||
|
||||
## Security
|
||||
|
||||
Full detail in [16-security.md](16-security.md). Headline gaps: no
|
||||
authentication, no authorization, no conversation ownership, a committed default
|
||||
PostgreSQL credential, containers running as root, no security context or
|
||||
NetworkPolicy in the chart, no dependency scanning, no security headers, and no
|
||||
retention or redaction for user-supplied patient context.
|
||||
|
||||
## Observability
|
||||
|
||||
- **No alerting at all** — no Alertmanager, no rule files, no Grafana alerts.
|
||||
- **No log aggregation** and no structured logging; `agent.py` logs routine
|
||||
timings at WARNING because uvicorn does not wire the root logger.
|
||||
- **`web` is entirely uninstrumented.**
|
||||
- Four metric names are registered but never incremented.
|
||||
- No SLOs or error budgets.
|
||||
|
||||
## Documentation/code discrepancies
|
||||
|
||||
Found by comparing the pre-existing documents against the code. The code wins in
|
||||
every case.
|
||||
|
||||
| Claim | Where | Reality |
|
||||
|---|---|---|
|
||||
| "conversation history is an in-process dict per `RagAgent`, not yet durable" | `docs/architecture.md` service table | `PostgresConversationStore` **is** wired in `bootstrap.py` and backs `recent()`/`append()`. Only `_last_frame` and `_clarify_streak` remain in-process |
|
||||
| "`web` … Calls api-gateway only" | `docs/architecture.md` service table | `web` calls `ai-service` directly via `AI_SERVICE_URL`; no gateway exists |
|
||||
| "Qwen3 via the Converse API for understanding/generation/entailment" | `docs/architecture.md` | The model is configuration. Code default `deepseek.v3.2`; local `.env` `qwen.qwen3-next-80b-a3b`; production value is in an uncommitted `.env.prod` and **cannot be verified from the repository** |
|
||||
| api-gateway / auth-service / user-service / chat-service described with owned responsibilities and data | `docs/architecture.md` service table | Not built. The document does flag this elsewhere, but the table reads as current state |
|
||||
| Redis "session/refresh-token cache, rate-limit counters" | `docs/architecture.md` | No Redis client is imported anywhere. Present only in the local-dev Compose file |
|
||||
| "`fusion.py`/`context.py`/`expand_siblings` are dead code" | `docs/current-rag-pipeline-audit.md` | `context.py::pack_evidence` **is** now wired into `RetrievalService.retrieve_framed`. `fusion.py` and `expansion.py` remain unwired |
|
||||
| "trace has no per-stage timing" | `docs/current-rag-pipeline-audit.md` | `telemetry.stage()` now emits `duocthu_stage_duration_seconds` and per-stage spans |
|
||||
| ADR 0007's `Focus`/`ConversationState` and the bounded PLAN/RETRIEVE/ASSESS/REFINE/VERIFY loop | `docs/adr/0007` | Superseded by ADR 0008; `rag/conversation.py` and `rag/reasoning.py` no longer exist. The `LOOP_*` metric names survive as dead constants |
|
||||
| `infra/ci/github-actions/README.md` lists five CI workflows | that README | None exists; the only workflow is `deploy.yml` |
|
||||
| ADR 0005 "Contract/schema only — no implementation" | `docs/adr/0005` | The contract is implemented — `segment/models.py` and `chunk/` both follow it |
|
||||
|
||||
Dated planning documents (`v1-delivery-plan.md`, `rag-rebuild-plan.md`,
|
||||
`answer-experience-implementation-plan.md`,
|
||||
`condition-to-drug-audit-and-design.md`, `full-coverage-parsing-plan.md`) record
|
||||
intent on their date. They were not audited line-by-line here; treat them as
|
||||
history, not status.
|
||||
Reference in New Issue
Block a user