20 — Deployment

Current state

A single EC2 host running Docker Compose, with Caddy terminating TLS for realvuxbaro.me. Images are built on the host at deploy time; there is no container registry and no orchestrator.

flowchart TB
    subgraph internet["Internet"]
        USER[Clinician]
        OPS[Operator]
        LE[Let's Encrypt]
    end

    subgraph host["EC2 instance — docker compose project 'docker'"]
        CADDY["caddy:2-alpine<br/>:80 :443<br/>volumes: Caddyfile, caddy-data, caddy-config"]
        WEB["web<br/>build apps/web/Dockerfile<br/>AI_SERVICE_URL=http://ai-service:8000"]
        AI["ai-service<br/>build apps/ai-service/Dockerfile<br/>env_file .env.prod (not in repo)"]
        PG[("postgres:16-alpine<br/>vol postgres-data")]
        QD[("qdrant/qdrant:latest<br/>vol qdrant-data")]
        PROM["prometheus<br/>127.0.0.1:9090"]
        TEMPO["tempo"]
        OTEL["otel-collector"]
        GRAF["grafana<br/>127.0.0.1:3002"]
    end

    BR["AWS Bedrock<br/>via instance IAM role"]

    USER -->|https| CADDY
    OPS -->|https .../grafana/| CADDY
    LE <-->|ACME| CADDY
    CADDY --> WEB --> AI
    AI --> PG
    AI --> QD
    AI --> BR
    AI -.OTLP.-> OTEL --> TEMPO
    PROM -.scrape.-> AI
    GRAF --> PROM
    GRAF --> TEMPO
    CADDY --> GRAF

Files that define it

File	Role
`infra/docker/docker-compose.prod.yml`	Base topology: postgres, qdrant, ai-service, web, caddy
`infra/docker/docker-compose.observability.yml`	Overlay: turns on OTel in `ai-service`, adds prometheus/tempo/otel-collector/grafana
`infra/docker/Caddyfile`	`realvuxbaro.me` → `web:3000`, `/grafana/*` → `grafana:3000`, `/grafana` → 308 redirect
`apps/ai-service/Dockerfile`	`python:3.12-slim`, deps pinned inline, `uvicorn main:app --host 0.0.0.0 --port 8000`
`apps/web/Dockerfile`	3-stage node:20-slim, `next start -p 3000 -H 0.0.0.0`
`.github/workflows/deploy.yml`	The deploy itself, over SSH

Port and exposure map

Service	Host port	Reachable from
caddy	80, 443	Internet
web	none	Compose network + Caddy
ai-service	none	Compose network only
postgres	none	Compose network only
qdrant	none	Compose network only
prometheus	`127.0.0.1:9090`	The host only (SSH tunnel)
grafana	`127.0.0.1:3002`	The host, plus the internet via Caddy `/grafana/`
tempo, otel-collector	none	Compose network only

Deploy sequence

Triggered by a push to master or a manual workflow_dispatch. appleboy/ssh-action runs a set -e script on the host as ubuntu:

sequenceDiagram
    participant GH as GitHub Actions
    participant EC2 as EC2 host
    participant DC as docker compose
    participant SVC as running stack

    GH->>EC2: ssh (EC2_HOST, EC2_SSH_KEY), env GRAFANA_ADMIN_PASSWORD
    EC2->>EC2: test -n "$GRAFANA_ADMIN_PASSWORD"  (fail fast)
    EC2->>EC2: cd ~/app && git fetch origin master && git reset --hard origin/master
    EC2->>DC: compose -f prod -f observability up -d --build<br/>ai-service web prometheus tempo otel-collector grafana caddy
    DC-->>SVC: rebuilt + restarted
    EC2->>SVC: caddy validate && caddy reload
    EC2->>SVC: docker exec ai-service python -m migrate
    EC2->>EC2: sleep 10
    EC2->>SVC: GET /health, GET /ready, GET web:3000
    EC2->>SVC: POST /v1/rag/query (gout) — assert answerable + chi_dinh
    EC2->>SVC: prometheus /-/ready, tempo /ready (retry 12x5s), grafana /api/health
    EC2->>SVC: assert both Grafana datasources + the dashboard exist
    EC2->>SVC: GET https://realvuxbaro.me/grafana/login
    EC2->>SVC: POST /v1/rag/query with X-Correlation-ID; assert X-Trace-ID matches ^[0-9a-f]{32}$
    EC2->>EC2: sleep 20
    EC2->>SVC: assert duocthu_requests_total in Prometheus
    EC2->>SVC: assert the exact trace id retrievable from Tempo (retry 12x5s)

Note what the up -d line does not include: postgres and qdrant. They are left running from a previous deploy (both carry restart: unless-stopped), so the stateful services are never restarted by a code deploy. That is deliberate-looking and safe for uptime, but it also means a change to the postgres/qdrant service definitions in the compose file will not take effect until someone restarts them by hand.

Migrations

docker exec docker-ai-service-1 python -m migrate runs after the containers are up. migrate.py applies every migrations/*.sql in sorted order, each idempotent. There is no version table, no ordering guard beyond the filename, and no rollback.

Rollback

There is no rollback command. The recovery path is git revert (or reset) on master followed by another deploy, because the deploy script does git reset --hard origin/master and rebuilds. Since images are built on the host and not tagged, there is no previously-built image to roll back to.

What the repository does not contain

Any container registry configuration (ECR, GHCR, Docker Hub).
Any image tagging or versioning scheme — web and ai-service are rebuilt from latest source each time.
Terraform for the EC2 host: infra/terraform/ holds only empty module and environment directories plus a README.
Blue/green, canary, or any staged rollout — the deploy is in-place.
A database backup or restore procedure.
A staging environment. infra/helm/values-staging.yaml and infra/argocd/applications/staging/ exist but were never applied.

Target deployment (not applied)

The Helm chart and ArgoCD manifests describe a Kubernetes deployment. See 21-kubernetes-and-argocd.md. They are current intent, not current state — ADR 0002's status line says exactly that:

Accepted — still the target, not yet implemented. Not superseded by the current production setup.

5.7 KiB Raw Blame History