Add read-only production runtime audit
This commit is contained in:
@@ -0,0 +1,185 @@
|
||||
# 02 — System architecture
|
||||
|
||||
## Architectural style
|
||||
|
||||
**As built:** a two-service application (`web` + `ai-service`) plus an offline
|
||||
batch pipeline, deployed as Docker Compose services on one host. Communication
|
||||
is synchronous HTTP/JSON. There is no message broker, no queue, no async
|
||||
worker, and no service mesh.
|
||||
|
||||
**As designed on paper:** a seven-service microservices platform
|
||||
(`api-gateway`, `auth-service`, `user-service`, `chat-service`, `ai-service`,
|
||||
`web`, `ingestion`), described in the pre-existing `docs/architecture.md`. Four
|
||||
of those seven do not exist — their directories hold a `README.md` and a
|
||||
four-line `package.json` with no `dependencies` and no source files. The
|
||||
monorepo scaffolding (pnpm workspace entries, `infra/k8s/base/<service>/`
|
||||
directories) still reserves their names.
|
||||
|
||||
Both facts matter: the second explains why `apps/`, `pnpm-workspace.yaml` and
|
||||
the Helm chart look bigger than the running system.
|
||||
|
||||
## Component diagram — what actually runs
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph browser["Browser"]
|
||||
UI["Chat UI<br/>ChatPanel.tsx · 65s abort"]
|
||||
end
|
||||
|
||||
subgraph ec2["EC2 host — docker compose"]
|
||||
CADDY["caddy:2-alpine<br/>:80 :443 · ACME TLS"]
|
||||
subgraph webc["web (Next.js 14, :3000)"]
|
||||
MW["middleware.ts<br/>in-memory IP rate limit"]
|
||||
BFF["/api/chat · /api/suggest<br/>/api/feedback · /api/pdf"]
|
||||
end
|
||||
subgraph aic["ai-service (FastAPI, :8000)"]
|
||||
HTTP["routers/rag.py"]
|
||||
AGENT["RagAgent"]
|
||||
RET["RetrievalService"]
|
||||
ANS["GroundedAnswerService"]
|
||||
end
|
||||
PG[("postgres:16-alpine")]
|
||||
QD[("qdrant/qdrant")]
|
||||
subgraph obs["observability overlay"]
|
||||
OTELC["otel-collector"]
|
||||
TEMPO["tempo"]
|
||||
PROM["prometheus"]
|
||||
GRAF["grafana :3002 (127.0.0.1)"]
|
||||
end
|
||||
end
|
||||
|
||||
BEDROCK["AWS Bedrock<br/>embed-v4 · rerank-v3.5 · Converse"]
|
||||
|
||||
UI -->|HTTPS| CADDY
|
||||
CADDY -->|"/*"| MW --> BFF
|
||||
CADDY -->|"/grafana/*"| GRAF
|
||||
BFF -->|"POST /v1/rag/query"| HTTP
|
||||
HTTP --> AGENT
|
||||
AGENT --> RET
|
||||
AGENT --> ANS
|
||||
RET --> QD
|
||||
ANS -->|generation + entailment| BEDROCK
|
||||
AGENT -->|understanding| BEDROCK
|
||||
RET -->|embed + rerank| BEDROCK
|
||||
HTTP --> PG
|
||||
AGENT --> PG
|
||||
aic -->|OTLP/HTTP| OTELC --> TEMPO
|
||||
PROM -->|scrape /metrics| aic
|
||||
GRAF --> PROM
|
||||
GRAF --> TEMPO
|
||||
```
|
||||
|
||||
`ai-service` publishes **no host port** in `docker-compose.prod.yml`; it is
|
||||
reachable only on the Compose network. Caddy proxies `web` and `grafana` only.
|
||||
|
||||
## Service boundaries
|
||||
|
||||
| Service | Owns | Depends on | Stateless? |
|
||||
|---|---|---|---|
|
||||
| `web` | Rendering, reason-code → Vietnamese message mapping, citation grouping, rate limiting | `ai-service` over HTTP; the source PDF on a bind-mounted path | **No** — rate-limit counters are per-process in memory |
|
||||
| `ai-service` | Understanding, retrieval, generation, grounding, citations, traces | Qdrant, PostgreSQL, Bedrock | **Mostly** — `RagAgent` keeps two in-process dicts (`_last_frame`, `_clarify_streak`); conversation *text* is in PostgreSQL |
|
||||
| `ingestion` | Turning the PDF into `chunks.jsonl` and Qdrant points | Qdrant, Bedrock, local disk | N/A — batch |
|
||||
|
||||
### The stateful detail that constrains scaling
|
||||
|
||||
`RagAgent` (`rag/agent.py`) holds three dicts:
|
||||
|
||||
```python
|
||||
self._history: dict[str, list[str]] # unused when a ConversationStore is configured
|
||||
self._last_frame: dict[str, QueryFrame] # ALWAYS in-process
|
||||
self._clarify_streak: dict[str, int] # ALWAYS in-process
|
||||
```
|
||||
|
||||
`PostgresConversationStore` replaces `_history` only. `_last_frame` carries the
|
||||
structured merge that stops the model re-asking an answered clarify question,
|
||||
and `_clarify_streak` drives the clarify circuit breaker. Both are lost on
|
||||
restart and **not shared between replicas**. Running more than one `ai-service`
|
||||
replica therefore degrades multi-turn quality in a way nothing detects. The Helm
|
||||
chart's `aiService.replicaCount` defaults to `1`; nothing enforces it.
|
||||
|
||||
## Module boundaries inside `ai-service`
|
||||
|
||||
Ports-and-adapters, enforced by import discipline rather than by tooling:
|
||||
|
||||
- `rag/ports.py` declares `Retriever`, `SectionRetriever`, `ParentStore`,
|
||||
`AnswerGenerator`, `Reranker`, plus three exception types
|
||||
(`QueryEmbeddingUnavailable`, `AnswerGenerationUnavailable`,
|
||||
`RerankUnavailable`) that adapters raise and the domain catches.
|
||||
- `adapters/` is the only place `qdrant_client`, `boto3`, `psycopg` and
|
||||
`prometheus_client` are imported — and always **lazily**, inside a method, so
|
||||
the domain imports cleanly on a machine with none of them installed.
|
||||
- `bootstrap.py` is the composition root. Nothing else constructs an adapter.
|
||||
|
||||
One boundary is looser than the protocol suggests: `RetrievalService` reaches
|
||||
optional retriever capabilities with `getattr(self._retriever, "find_by_section",
|
||||
None)` rather than through a declared protocol. `find_by_indication`,
|
||||
`search_indication`, `search_lexical` and `find_by_drug` are all discovered this
|
||||
way and none of them appear in `ports.py`. A retriever missing one silently
|
||||
disables a whole route instead of failing a type check.
|
||||
|
||||
## Synchronous communication
|
||||
|
||||
Every hop is a blocking HTTP or SDK call. One answerable turn issues, in
|
||||
sequence:
|
||||
|
||||
1. `POST /api/chat` (browser → web)
|
||||
2. `POST /v1/rag/query` (web → ai-service)
|
||||
3. Bedrock Converse — understanding
|
||||
4. Qdrant `scroll`/`query_points` — retrieval (1–N calls)
|
||||
5. Bedrock Converse — generation (plus one retry on `evidence_sufficient=false`)
|
||||
6. Bedrock Converse — entailment
|
||||
7. optional Bedrock Converse ×2 — completeness repair + its re-verification
|
||||
8. PostgreSQL insert — trace
|
||||
|
||||
Measured production latencies recorded in `ChatPanel.tsx` (n=8, 2026-08-11):
|
||||
6.2 / 6.4 / 8.4 / 10.9 / 12.4 / 21.7 / 25.1 / 40.3 seconds.
|
||||
|
||||
## Asynchronous communication
|
||||
|
||||
**Not found.** No broker, no queue, no background worker, no SSE, no
|
||||
WebSocket, no streaming response. `web`'s `/api/chat` awaits the full upstream
|
||||
response before replying.
|
||||
|
||||
## Failure boundaries
|
||||
|
||||
| Boundary | Policy | Implemented in |
|
||||
|---|---|---|
|
||||
| Corpus/model manifest mismatch at startup | **Fail closed, crash the process** | `bootstrap.py::_verify_corpus_manifest` → `rag/manifest.py` |
|
||||
| Query embedder unreachable | **Fail closed** — abstain, never a 500 | `rag/service.py`, `rag/ports.py` |
|
||||
| Generator unreachable / malformed / budget exhausted | **Fail closed** — abstain with a specific reason code | `rag/answer.py::_generate` |
|
||||
| Understanding call fails | **Fail closed** — abstain, tagged `system_error` | `rag/understanding.py::understand` |
|
||||
| Grounding or entailment rejects | **Fail closed** — abstain | `rag/answer.py` |
|
||||
| Reranker unreachable | **Fail open** — keep original order | `rag/service.py::_rerank` |
|
||||
| Sufficiency check unreachable | **Fail open** — proceed to generate | `rag/answer.py::_check_sufficiency` |
|
||||
| PostgreSQL trace write fails | **Fail open** — answer returned, `TRACE_WRITE_FAILED` counter | `routers/rag.py` |
|
||||
| Conversation store read/write fails | **Fail open** — this turn has no memory | `rag/agent.py::_get_history` / `_remember` |
|
||||
| Metrics package missing | **Degrade** — `NullMetrics` | `bootstrap.py::_build_metrics` |
|
||||
| OpenTelemetry packages missing / `OTEL_ENABLED=false` | **Degrade** — no-op tracer | `rag/telemetry.py` |
|
||||
|
||||
The asymmetry is deliberate and documented in-code: anything that could change
|
||||
*what is stated* fails closed; anything that only affects quality or
|
||||
observability fails open.
|
||||
|
||||
## Deployment units
|
||||
|
||||
| Unit | Image | Built by |
|
||||
|---|---|---|
|
||||
| `ai-service` | `apps/ai-service/Dockerfile` — `python:3.12-slim`, deps pinned inline (not from `pyproject.toml`) | `docker compose up --build` on the EC2 host |
|
||||
| `web` | `apps/web/Dockerfile` — 3-stage node:20-slim, `pnpm --filter @duoc-thu/web build` | same |
|
||||
| `postgres`, `qdrant`, `caddy`, `prometheus`, `tempo`, `grafana`, `otel-collector` | Upstream images | pulled |
|
||||
|
||||
There is **no container registry**. Images are built on the production host at
|
||||
deploy time. The Helm chart assumes registry images
|
||||
(`duocthu-ai-service:<tag>`) that nothing currently produces.
|
||||
|
||||
## Scaling implications
|
||||
|
||||
- `ai-service` is CPU-light and latency-bound on Bedrock. Horizontal scaling is
|
||||
blocked by the in-process `_last_frame`/`_clarify_streak` state above.
|
||||
- `web`'s rate limiter is per-process; a second replica doubles the effective
|
||||
allowance. `middleware.ts` says so explicitly.
|
||||
- Qdrant and PostgreSQL are single containers with named Docker volumes on one
|
||||
EBS-backed host. No replication, no backup job in the repository.
|
||||
- The `evidence_limit`/`max_context_tokens` policy (`rag/service.py`,
|
||||
`EvidencePolicy`) bounds prompt size; nothing bounds concurrent Bedrock calls
|
||||
beyond the per-request budget.
|
||||
Reference in New Issue
Block a user