duocthu/docs/20-deployment.md

# 20 — Deployment

## Current state

A single EC2 host running Docker Compose, with Caddy terminating TLS for
`realvuxbaro.me`. Images are built **on the host** at deploy time; there is no
container registry and no orchestrator.

```mermaid
flowchart TB
    subgraph internet["Internet"]
        USER[Clinician]
        OPS[Operator]
        LE[Let's Encrypt]
    end

    subgraph host["EC2 instance — docker compose project 'docker'"]
        CADDY["caddy:2-alpine<br/>:80 :443<br/>volumes: Caddyfile, caddy-data, caddy-config"]
        WEB["web<br/>build apps/web/Dockerfile<br/>AI_SERVICE_URL=http://ai-service:8000"]
        AI["ai-service<br/>build apps/ai-service/Dockerfile<br/>env_file .env.prod (not in repo)"]
        PG[("postgres:16-alpine<br/>vol postgres-data")]
        QD[("qdrant/qdrant:latest<br/>vol qdrant-data")]
        PROM["prometheus<br/>127.0.0.1:9090"]
        TEMPO["tempo"]
        OTEL["otel-collector"]
        GRAF["grafana<br/>127.0.0.1:3002"]
    end

    BR["AWS Bedrock<br/>via instance IAM role"]

    USER -->|https| CADDY
    OPS -->|https .../grafana/| CADDY
    LE <-->|ACME| CADDY
    CADDY --> WEB --> AI
    AI --> PG
    AI --> QD
    AI --> BR
    AI -.OTLP.-> OTEL --> TEMPO
    PROM -.scrape.-> AI
    GRAF --> PROM
    GRAF --> TEMPO
    CADDY --> GRAF
```

## Files that define it

| File | Role |
|---|---|
| `infra/docker/docker-compose.prod.yml` | Base topology: postgres, qdrant, ai-service, web, caddy |
| `infra/docker/docker-compose.observability.yml` | Overlay: turns on OTel in `ai-service`, adds prometheus/tempo/otel-collector/grafana |
| `infra/docker/Caddyfile` | `realvuxbaro.me` → `web:3000`, `/grafana/*` → `grafana:3000`, `/grafana` → 308 redirect |
| `apps/ai-service/Dockerfile` | `python:3.12-slim`, deps pinned inline, `uvicorn main:app --host 0.0.0.0 --port 8000` |
| `apps/web/Dockerfile` | 3-stage node:20-slim, `next start -p 3000 -H 0.0.0.0` |
| `.github/workflows/deploy.yml` | The deploy itself, over SSH |

## Port and exposure map

| Service | Host port | Reachable from |
|---|---|---|
| caddy | 80, 443 | Internet |
| web | none | Compose network + Caddy |
| ai-service | **none** | Compose network only |
| postgres | none | Compose network only |
| qdrant | none | Compose network only |
| prometheus | `127.0.0.1:9090` | The host only (SSH tunnel) |
| grafana | `127.0.0.1:3002` | The host, plus the internet via Caddy `/grafana/` |
| tempo, otel-collector | none | Compose network only |

## Deploy sequence

Triggered by a push to `master` or a manual `workflow_dispatch`.
`appleboy/ssh-action` runs a `set -e` script on the host as `ubuntu`:

```mermaid
sequenceDiagram
    participant GH as GitHub Actions
    participant EC2 as EC2 host
    participant DC as docker compose
    participant SVC as running stack

    GH->>EC2: ssh (EC2_HOST, EC2_SSH_KEY), env GRAFANA_ADMIN_PASSWORD
    EC2->>EC2: test -n "$GRAFANA_ADMIN_PASSWORD"  (fail fast)
    EC2->>EC2: cd ~/app && git fetch origin master && git reset --hard origin/master
    EC2->>DC: compose -f prod -f observability up -d --build<br/>ai-service web prometheus tempo otel-collector grafana caddy
    DC-->>SVC: rebuilt + restarted
    EC2->>SVC: caddy validate && caddy reload
    EC2->>SVC: docker exec ai-service python -m migrate
    EC2->>EC2: sleep 10
    EC2->>SVC: GET /health, GET /ready, GET web:3000
    EC2->>SVC: POST /v1/rag/query (gout) — assert answerable + chi_dinh
    EC2->>SVC: prometheus /-/ready, tempo /ready (retry 12x5s), grafana /api/health
    EC2->>SVC: assert both Grafana datasources + the dashboard exist
    EC2->>SVC: GET https://realvuxbaro.me/grafana/login
    EC2->>SVC: POST /v1/rag/query with X-Correlation-ID; assert X-Trace-ID matches ^[0-9a-f]{32}$
    EC2->>EC2: sleep 20
    EC2->>SVC: assert duocthu_requests_total in Prometheus
    EC2->>SVC: assert the exact trace id retrievable from Tempo (retry 12x5s)
```

Note what the `up -d` line does **not** include: `postgres` and `qdrant`. They
are left running from a previous deploy (both carry `restart: unless-stopped`),
so the stateful services are never restarted by a code deploy. That is
deliberate-looking and safe for uptime, but it also means a change to the
postgres/qdrant service definitions in the compose file will not take effect
until someone restarts them by hand.

## Migrations

`docker exec docker-ai-service-1 python -m migrate` runs after the containers
are up. `migrate.py` applies every `migrations/*.sql` in sorted order, each
idempotent. There is no version table, no ordering guard beyond the filename,
and no rollback.

## Rollback

There is no rollback command. The recovery path is `git revert` (or reset) on
`master` followed by another deploy, because the deploy script does
`git reset --hard origin/master` and rebuilds. Since images are built on the
host and not tagged, there is **no previously-built image to roll back to**.

## What the repository does not contain

- Any container registry configuration (ECR, GHCR, Docker Hub).
- Any image tagging or versioning scheme — `web` and `ai-service` are rebuilt
  from `latest` source each time.
- Terraform for the EC2 host: `infra/terraform/` holds only empty module and
  environment directories plus a README.
- Blue/green, canary, or any staged rollout — the deploy is in-place.
- A database backup or restore procedure.
- A staging environment. `infra/helm/values-staging.yaml` and
  `infra/argocd/applications/staging/` exist but were never applied.

## Target deployment (not applied)

The Helm chart and ArgoCD manifests describe a Kubernetes deployment. See
[21-kubernetes-and-argocd.md](21-kubernetes-and-argocd.md). They are current
intent, not current state — ADR 0002's status line says exactly that:

> **Accepted — still the target, not yet implemented.** Not superseded by the
> current production setup.