Add read-only production runtime audit

2026-08-17 11:17:40 +07:00
parent 057d4ed9dc
commit a1de4715a4
106 changed files with 6869 additions and 1782 deletions
@@ -0,0 +1,140 @@
+# 20 — Deployment
+
+## Current state
+
+A single EC2 host running Docker Compose, with Caddy terminating TLS for
+`realvuxbaro.me`. Images are built **on the host** at deploy time; there is no
+container registry and no orchestrator.
+
+```mermaid
+flowchart TB
+    subgraph internet["Internet"]
+        USER[Clinician]
+        OPS[Operator]
+        LE[Let's Encrypt]
+    end
+
+    subgraph host["EC2 instance — docker compose project 'docker'"]
+        CADDY["caddy:2-alpine<br/>:80 :443<br/>volumes: Caddyfile, caddy-data, caddy-config"]
+        WEB["web<br/>build apps/web/Dockerfile<br/>AI_SERVICE_URL=http://ai-service:8000"]
+        AI["ai-service<br/>build apps/ai-service/Dockerfile<br/>env_file .env.prod (not in repo)"]
+        PG[("postgres:16-alpine<br/>vol postgres-data")]
+        QD[("qdrant/qdrant:latest<br/>vol qdrant-data")]
+        PROM["prometheus<br/>127.0.0.1:9090"]
+        TEMPO["tempo"]
+        OTEL["otel-collector"]
+        GRAF["grafana<br/>127.0.0.1:3002"]
+    end
+
+    BR["AWS Bedrock<br/>via instance IAM role"]
+
+    USER -->|https| CADDY
+    OPS -->|https .../grafana/| CADDY
+    LE <-->|ACME| CADDY
+    CADDY --> WEB --> AI
+    AI --> PG
+    AI --> QD
+    AI --> BR
+    AI -.OTLP.-> OTEL --> TEMPO
+    PROM -.scrape.-> AI
+    GRAF --> PROM
+    GRAF --> TEMPO
+    CADDY --> GRAF
+```
+
+## Files that define it
+
+| File | Role |
+|---|---|
+| `infra/docker/docker-compose.prod.yml` | Base topology: postgres, qdrant, ai-service, web, caddy |
+| `infra/docker/docker-compose.observability.yml` | Overlay: turns on OTel in `ai-service`, adds prometheus/tempo/otel-collector/grafana |
+| `infra/docker/Caddyfile` | `realvuxbaro.me` → `web:3000`, `/grafana/*` → `grafana:3000`, `/grafana` → 308 redirect |
+| `apps/ai-service/Dockerfile` | `python:3.12-slim`, deps pinned inline, `uvicorn main:app --host 0.0.0.0 --port 8000` |
+| `apps/web/Dockerfile` | 3-stage node:20-slim, `next start -p 3000 -H 0.0.0.0` |
+| `.github/workflows/deploy.yml` | The deploy itself, over SSH |
+
+## Port and exposure map
+
+| Service | Host port | Reachable from |
+|---|---|---|
+| caddy | 80, 443 | Internet |
+| web | none | Compose network + Caddy |
+| ai-service | **none** | Compose network only |
+| postgres | none | Compose network only |
+| qdrant | none | Compose network only |
+| prometheus | `127.0.0.1:9090` | The host only (SSH tunnel) |
+| grafana | `127.0.0.1:3002` | The host, plus the internet via Caddy `/grafana/` |
+| tempo, otel-collector | none | Compose network only |
+
+## Deploy sequence
+
+Triggered by a push to `master` or a manual `workflow_dispatch`.
+`appleboy/ssh-action` runs a `set -e` script on the host as `ubuntu`:
+
+```mermaid
+sequenceDiagram
+    participant GH as GitHub Actions
+    participant EC2 as EC2 host
+    participant DC as docker compose
+    participant SVC as running stack
+
+    GH->>EC2: ssh (EC2_HOST, EC2_SSH_KEY), env GRAFANA_ADMIN_PASSWORD
+    EC2->>EC2: test -n "$GRAFANA_ADMIN_PASSWORD"  (fail fast)
+    EC2->>EC2: cd ~/app && git fetch origin master && git reset --hard origin/master
+    EC2->>DC: compose -f prod -f observability up -d --build<br/>ai-service web prometheus tempo otel-collector grafana caddy
+    DC-->>SVC: rebuilt + restarted
+    EC2->>SVC: caddy validate && caddy reload
+    EC2->>SVC: docker exec ai-service python -m migrate
+    EC2->>EC2: sleep 10
+    EC2->>SVC: GET /health, GET /ready, GET web:3000
+    EC2->>SVC: POST /v1/rag/query (gout) — assert answerable + chi_dinh
+    EC2->>SVC: prometheus /-/ready, tempo /ready (retry 12x5s), grafana /api/health
+    EC2->>SVC: assert both Grafana datasources + the dashboard exist
+    EC2->>SVC: GET https://realvuxbaro.me/grafana/login
+    EC2->>SVC: POST /v1/rag/query with X-Correlation-ID; assert X-Trace-ID matches ^[0-9a-f]{32}$
+    EC2->>EC2: sleep 20
+    EC2->>SVC: assert duocthu_requests_total in Prometheus
+    EC2->>SVC: assert the exact trace id retrievable from Tempo (retry 12x5s)
+```
+
+Note what the `up -d` line does **not** include: `postgres` and `qdrant`. They
+are left running from a previous deploy (both carry `restart: unless-stopped`),
+so the stateful services are never restarted by a code deploy. That is
+deliberate-looking and safe for uptime, but it also means a change to the
+postgres/qdrant service definitions in the compose file will not take effect
+until someone restarts them by hand.
+
+## Migrations
+
+`docker exec docker-ai-service-1 python -m migrate` runs after the containers
+are up. `migrate.py` applies every `migrations/*.sql` in sorted order, each
+idempotent. There is no version table, no ordering guard beyond the filename,
+and no rollback.
+
+## Rollback
+
+There is no rollback command. The recovery path is `git revert` (or reset) on
+`master` followed by another deploy, because the deploy script does
+`git reset --hard origin/master` and rebuilds. Since images are built on the
+host and not tagged, there is **no previously-built image to roll back to**.
+
+## What the repository does not contain
+
+- Any container registry configuration (ECR, GHCR, Docker Hub).
+- Any image tagging or versioning scheme — `web` and `ai-service` are rebuilt
+  from `latest` source each time.
+- Terraform for the EC2 host: `infra/terraform/` holds only empty module and
+  environment directories plus a README.
+- Blue/green, canary, or any staged rollout — the deploy is in-place.
+- A database backup or restore procedure.
+- A staging environment. `infra/helm/values-staging.yaml` and
+  `infra/argocd/applications/staging/` exist but were never applied.
+
+## Target deployment (not applied)
+
+The Helm chart and ArgoCD manifests describe a Kubernetes deployment. See
+[21-kubernetes-and-argocd.md](21-kubernetes-and-argocd.md). They are current
+intent, not current state — ADR 0002's status line says exactly that:
+
+> **Accepted — still the target, not yet implemented.** Not superseded by the
+> current production setup.