# 24 — Production operations

Production is one EC2 host running Docker Compose. Every command below assumes
an SSH session on that host in `~/app/infra/docker`, matching what
`.github/workflows/deploy.yml` does.

The Compose project name is `docker` (the directory name), so containers are
named `docker-<service>-1`.

## Compose invocation

Both overlay files are always used together:

```bash
cd ~/app/infra/docker
COMPOSE="sudo docker compose -f docker-compose.prod.yml -f docker-compose.observability.yml"
```

The observability overlay is what sets `OTEL_ENABLED=true` on `ai-service`, so
omitting it silently disables tracing.

## Start / stop / restart

```bash
$COMPOSE ps
$COMPOSE up -d ai-service web caddy                  # start/refresh app tier
$COMPOSE restart ai-service                          # restart one service
$COMPOSE stop ai-service
$COMPOSE logs -f --tail 200 ai-service
```

`ai-service` builds its whole runtime at import time, so a restart re-runs the
corpus-manifest check. If that check fails the container exits immediately and
keeps restarting — check the logs for `ManifestMismatch` before assuming a crash
loop is resource-related.

## Deploy

Normal path: push to `master`. The workflow SSHes in, resets the checkout,
rebuilds, reloads Caddy, migrates and runs ~18 assertions
([22-ci-cd.md](22-ci-cd.md)).

Manual equivalent:

```bash
cd ~/app && git fetch origin master && git reset --hard origin/master
cd infra/docker
export GRAFANA_ADMIN_PASSWORD='<value>'
sudo -E docker compose -f docker-compose.prod.yml -f docker-compose.observability.yml \
  up -d --build ai-service web prometheus tempo otel-collector grafana caddy
sudo docker exec docker-caddy-1 caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
sudo docker exec docker-caddy-1 caddy reload   --config /etc/caddy/Caddyfile --adapter caddyfile
sudo docker exec docker-ai-service-1 python -m migrate
```

Note `postgres` and `qdrant` are deliberately absent from that list — a code
deploy never restarts the stateful services.

## Rollback

There is no image to roll back to (images are built on the host, untagged). The
procedure is:

```bash
cd ~/app
git reset --hard <last-good-sha>      # or push a revert to master and let CI deploy
cd infra/docker && sudo -E docker compose -f docker-compose.prod.yml \
  -f docker-compose.observability.yml up -d --build ai-service web
```

A rollback that crosses a migration is **not covered** — migrations are
forward-only with no down scripts.

## Health checks

```bash
NET=docker_default
sudo docker run --rm --network $NET curlimages/curl -sf http://ai-service:8000/health
sudo docker run --rm --network $NET curlimages/curl -sf http://ai-service:8000/ready
sudo docker run --rm --network $NET curlimages/curl -sf -o /dev/null http://web:3000
sudo docker run --rm --network $NET curlimages/curl -sf http://prometheus:9090/-/ready
sudo docker run --rm --network $NET curlimages/curl -sf http://tempo:3200/ready
sudo docker run --rm --network $NET curlimages/curl -sf http://grafana:3000/api/health
```

`ai-service` publishes no host port, so every check goes through a throwaway
container on the Compose network — the same technique the deploy workflow uses.

## Smoke test a real answer

```bash
sudo docker run --rm --network docker_default curlimages/curl -sf \
  -X POST http://ai-service:8000/v1/rag/query \
  -H 'Content-Type: application/json' \
  --data '{"query":"Đợt gout cấp có thuốc nào được Dược thư ghi chỉ định?",
           "subject_scope":"human","intent":"fact_lookup",
           "conversation_id":"ops-smoke"}'
```

Expect `"decision":"answerable"` and at least one citation with
`"section_key":"chi_dinh"` — the same two assertions the deploy makes.

## Database operations

```bash
# psql
sudo docker exec -it docker-postgres-1 psql -U duoc_thu -d duoc_thu

# apply migrations
sudo docker exec docker-ai-service-1 python -m migrate
```

Useful queries:

```sql
-- recent decisions
SELECT created_at, decision, reason, resolved_drug_id
FROM rag_retrieval_trace ORDER BY created_at DESC LIMIT 50;

-- abstain reasons over the last day
SELECT reason, count(*) FROM rag_retrieval_trace
WHERE decision = 'abstain' AND created_at > now() - interval '1 day'
GROUP BY reason ORDER BY 2 DESC;

-- find a support request by either correlation id
SELECT * FROM rag_retrieval_trace WHERE correlation_id = '<id>';
SELECT * FROM rag_retrieval_trace WHERE otel_trace_id  = '<32-hex>';

-- negative feedback with the question that caused it
SELECT f.created_at, f.rating, f.comment, t.query_text, t.decision, t.reason
FROM rag_answer_feedback f JOIN rag_retrieval_trace t USING (trace_id)
WHERE f.rating = 'not_helpful' ORDER BY f.created_at DESC LIMIT 50;
```

## Backup and restore

**No backup automation exists in this repository.** What the code supports:

```bash
# PostgreSQL logical dump
sudo docker exec docker-postgres-1 pg_dump -U duoc_thu duoc_thu > duoc_thu_$(date +%F).sql

# Qdrant snapshot (HTTP API, from inside the network)
sudo docker run --rm --network docker_default curlimages/curl -s -X POST \
  http://qdrant:6333/collections/duocthu_v1/snapshots
```

Both are manual. Whether EBS snapshots are configured on the instance cannot be
determined from the repository.

## Re-indexing / re-ingestion

Two situations, with very different costs:

**Corpus content unchanged, moving or restoring it** — snapshot and restore the
Qdrant collection. Free and exact; `ingestion/README.md` recommends it
explicitly.

**Corpus content changed** — the full pipeline must re-run and the embed step
**costs real AWS Bedrock spend on a personal account**. `ingestion/README.md`
requires explicit approval for any specific run. Order:

1. `python -m ingestion.cli run` → new `monographs.jsonl`
2. `python -m ingestion.cli chunk` → new `chunks.jsonl`
3. `python -m ingestion.cli chunk-ready` — **must exit 0**
4. `python -m ingestion.load.run --provider cohere-v4 --collection duocthu_v2 …`

Use a **new collection name**. The loader refuses to write a different
`corpus_sha256` into an existing collection (`CorpusMismatch`), which is the
intended behaviour, not an obstacle to work around. Then point
`QDRANT_COLLECTION` at the new collection and restart `ai-service`; the startup
manifest check verifies the binding. Keep the old collection until the new one
is confirmed — that is the rollback.

The embedding cache in `ingestion/data/processed/embeddings/` is keyed by
content hash, so unchanged chunks are not re-paid for.

## Grafana

Reachable at `https://realvuxbaro.me/grafana/` with the admin credentials from
`GRAFANA_ADMIN_PASSWORD`. Locally on the host: `http://127.0.0.1:3002`.
Provisioned datasources `prometheus` and `tempo`; dashboard uid
`duocthu-observability`.

## Following one request end to end

1. Take `X-Correlation-ID` or `X-Trace-ID` from the user's response headers (the
   UI surfaces `traceId` on each message).
2. `SELECT * FROM rag_retrieval_trace WHERE correlation_id = …` → the resolved
   scope, decision, reason and citations.
3. Open the trace id in Grafana → Tempo → per-stage spans
   (`rag.stage.understanding`, `retrieval`, `generation`, `entailment`) with
   `duocthu.*` attributes.
4. Cross-check `duocthu_generation_rejected_total{reason=…}` and
   `duocthu_abstention_total{reason=…}` in Prometheus for the same window.

## Cost control

Every chat turn makes 3–8 Bedrock calls on a personal AWS account. The only
guard is the in-memory rate limiter in `apps/web/middleware.ts`
(12/min, 120/hour per IP for `/api/chat`). There is no budget alarm, no
per-day cap and no authentication in the repository. Scaling `web` past one
replica multiplies the effective allowance.

## Incident quick reference

| Symptom | First check |
|---|---|
| Every answer is an abstain | `duocthu_abstention_total{reason}` — a single dominant reason points at a provider or corpus problem |
| `ai-service` restart loop | `docker logs docker-ai-service-1` for `ManifestMismatch` |
| 503 from `/v1/rag/query` | `EMBEDDING_PROVIDER` in `.env.prod`, and whether the manifest check passed |
| Answers take ~60 s then fail | `duocthu_generation_rejected_total{reason="request_budget_exhausted"}` |
| 429s | Rate limiter; `X-RateLimit-*` headers on the response |
| No traces in Grafana | Was the observability overlay included in the last `up`? |

Full table in [25-troubleshooting.md](25-troubleshooting.md).