217 lines
8.3 KiB
Markdown
217 lines
8.3 KiB
Markdown
# 24 — Production operations
|
||
|
||
Production is one EC2 host running Docker Compose. Every command below assumes
|
||
an SSH session on that host in `~/app/infra/docker`, matching what
|
||
`.github/workflows/deploy.yml` does.
|
||
|
||
The Compose project name is `docker` (the directory name), so containers are
|
||
named `docker-<service>-1`.
|
||
|
||
## Compose invocation
|
||
|
||
Both overlay files are always used together:
|
||
|
||
```bash
|
||
cd ~/app/infra/docker
|
||
COMPOSE="sudo docker compose -f docker-compose.prod.yml -f docker-compose.observability.yml"
|
||
```
|
||
|
||
The observability overlay is what sets `OTEL_ENABLED=true` on `ai-service`, so
|
||
omitting it silently disables tracing.
|
||
|
||
## Start / stop / restart
|
||
|
||
```bash
|
||
$COMPOSE ps
|
||
$COMPOSE up -d ai-service web caddy # start/refresh app tier
|
||
$COMPOSE restart ai-service # restart one service
|
||
$COMPOSE stop ai-service
|
||
$COMPOSE logs -f --tail 200 ai-service
|
||
```
|
||
|
||
`ai-service` builds its whole runtime at import time, so a restart re-runs the
|
||
corpus-manifest check. If that check fails the container exits immediately and
|
||
keeps restarting — check the logs for `ManifestMismatch` before assuming a crash
|
||
loop is resource-related.
|
||
|
||
## Deploy
|
||
|
||
Normal path: push to `master`. The workflow SSHes in, resets the checkout,
|
||
rebuilds, reloads Caddy, migrates and runs ~18 assertions
|
||
([22-ci-cd.md](22-ci-cd.md)).
|
||
|
||
Manual equivalent:
|
||
|
||
```bash
|
||
cd ~/app && git fetch origin master && git reset --hard origin/master
|
||
cd infra/docker
|
||
export GRAFANA_ADMIN_PASSWORD='<value>'
|
||
sudo -E docker compose -f docker-compose.prod.yml -f docker-compose.observability.yml \
|
||
up -d --build ai-service web prometheus tempo otel-collector grafana caddy
|
||
sudo docker exec docker-caddy-1 caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
|
||
sudo docker exec docker-caddy-1 caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
|
||
sudo docker exec docker-ai-service-1 python -m migrate
|
||
```
|
||
|
||
Note `postgres` and `qdrant` are deliberately absent from that list — a code
|
||
deploy never restarts the stateful services.
|
||
|
||
## Rollback
|
||
|
||
There is no image to roll back to (images are built on the host, untagged). The
|
||
procedure is:
|
||
|
||
```bash
|
||
cd ~/app
|
||
git reset --hard <last-good-sha> # or push a revert to master and let CI deploy
|
||
cd infra/docker && sudo -E docker compose -f docker-compose.prod.yml \
|
||
-f docker-compose.observability.yml up -d --build ai-service web
|
||
```
|
||
|
||
A rollback that crosses a migration is **not covered** — migrations are
|
||
forward-only with no down scripts.
|
||
|
||
## Health checks
|
||
|
||
```bash
|
||
NET=docker_default
|
||
sudo docker run --rm --network $NET curlimages/curl -sf http://ai-service:8000/health
|
||
sudo docker run --rm --network $NET curlimages/curl -sf http://ai-service:8000/ready
|
||
sudo docker run --rm --network $NET curlimages/curl -sf -o /dev/null http://web:3000
|
||
sudo docker run --rm --network $NET curlimages/curl -sf http://prometheus:9090/-/ready
|
||
sudo docker run --rm --network $NET curlimages/curl -sf http://tempo:3200/ready
|
||
sudo docker run --rm --network $NET curlimages/curl -sf http://grafana:3000/api/health
|
||
```
|
||
|
||
`ai-service` publishes no host port, so every check goes through a throwaway
|
||
container on the Compose network — the same technique the deploy workflow uses.
|
||
|
||
## Smoke test a real answer
|
||
|
||
```bash
|
||
sudo docker run --rm --network docker_default curlimages/curl -sf \
|
||
-X POST http://ai-service:8000/v1/rag/query \
|
||
-H 'Content-Type: application/json' \
|
||
--data '{"query":"Đợt gout cấp có thuốc nào được Dược thư ghi chỉ định?",
|
||
"subject_scope":"human","intent":"fact_lookup",
|
||
"conversation_id":"ops-smoke"}'
|
||
```
|
||
|
||
Expect `"decision":"answerable"` and at least one citation with
|
||
`"section_key":"chi_dinh"` — the same two assertions the deploy makes.
|
||
|
||
## Database operations
|
||
|
||
```bash
|
||
# psql
|
||
sudo docker exec -it docker-postgres-1 psql -U duoc_thu -d duoc_thu
|
||
|
||
# apply migrations
|
||
sudo docker exec docker-ai-service-1 python -m migrate
|
||
```
|
||
|
||
Useful queries:
|
||
|
||
```sql
|
||
-- recent decisions
|
||
SELECT created_at, decision, reason, resolved_drug_id
|
||
FROM rag_retrieval_trace ORDER BY created_at DESC LIMIT 50;
|
||
|
||
-- abstain reasons over the last day
|
||
SELECT reason, count(*) FROM rag_retrieval_trace
|
||
WHERE decision = 'abstain' AND created_at > now() - interval '1 day'
|
||
GROUP BY reason ORDER BY 2 DESC;
|
||
|
||
-- find a support request by either correlation id
|
||
SELECT * FROM rag_retrieval_trace WHERE correlation_id = '<id>';
|
||
SELECT * FROM rag_retrieval_trace WHERE otel_trace_id = '<32-hex>';
|
||
|
||
-- negative feedback with the question that caused it
|
||
SELECT f.created_at, f.rating, f.comment, t.query_text, t.decision, t.reason
|
||
FROM rag_answer_feedback f JOIN rag_retrieval_trace t USING (trace_id)
|
||
WHERE f.rating = 'not_helpful' ORDER BY f.created_at DESC LIMIT 50;
|
||
```
|
||
|
||
## Backup and restore
|
||
|
||
**No backup automation exists in this repository.** What the code supports:
|
||
|
||
```bash
|
||
# PostgreSQL logical dump
|
||
sudo docker exec docker-postgres-1 pg_dump -U duoc_thu duoc_thu > duoc_thu_$(date +%F).sql
|
||
|
||
# Qdrant snapshot (HTTP API, from inside the network)
|
||
sudo docker run --rm --network docker_default curlimages/curl -s -X POST \
|
||
http://qdrant:6333/collections/duocthu_v1/snapshots
|
||
```
|
||
|
||
Both are manual. Whether EBS snapshots are configured on the instance cannot be
|
||
determined from the repository.
|
||
|
||
## Re-indexing / re-ingestion
|
||
|
||
Two situations, with very different costs:
|
||
|
||
**Corpus content unchanged, moving or restoring it** — snapshot and restore the
|
||
Qdrant collection. Free and exact; `ingestion/README.md` recommends it
|
||
explicitly.
|
||
|
||
**Corpus content changed** — the full pipeline must re-run and the embed step
|
||
**costs real AWS Bedrock spend on a personal account**. `ingestion/README.md`
|
||
requires explicit approval for any specific run. Order:
|
||
|
||
1. `python -m ingestion.cli run` → new `monographs.jsonl`
|
||
2. `python -m ingestion.cli chunk` → new `chunks.jsonl`
|
||
3. `python -m ingestion.cli chunk-ready` — **must exit 0**
|
||
4. `python -m ingestion.load.run --provider cohere-v4 --collection duocthu_v2 …`
|
||
|
||
Use a **new collection name**. The loader refuses to write a different
|
||
`corpus_sha256` into an existing collection (`CorpusMismatch`), which is the
|
||
intended behaviour, not an obstacle to work around. Then point
|
||
`QDRANT_COLLECTION` at the new collection and restart `ai-service`; the startup
|
||
manifest check verifies the binding. Keep the old collection until the new one
|
||
is confirmed — that is the rollback.
|
||
|
||
The embedding cache in `ingestion/data/processed/embeddings/` is keyed by
|
||
content hash, so unchanged chunks are not re-paid for.
|
||
|
||
## Grafana
|
||
|
||
Reachable at `https://realvuxbaro.me/grafana/` with the admin credentials from
|
||
`GRAFANA_ADMIN_PASSWORD`. Locally on the host: `http://127.0.0.1:3002`.
|
||
Provisioned datasources `prometheus` and `tempo`; dashboard uid
|
||
`duocthu-observability`.
|
||
|
||
## Following one request end to end
|
||
|
||
1. Take `X-Correlation-ID` or `X-Trace-ID` from the user's response headers (the
|
||
UI surfaces `traceId` on each message).
|
||
2. `SELECT * FROM rag_retrieval_trace WHERE correlation_id = …` → the resolved
|
||
scope, decision, reason and citations.
|
||
3. Open the trace id in Grafana → Tempo → per-stage spans
|
||
(`rag.stage.understanding`, `retrieval`, `generation`, `entailment`) with
|
||
`duocthu.*` attributes.
|
||
4. Cross-check `duocthu_generation_rejected_total{reason=…}` and
|
||
`duocthu_abstention_total{reason=…}` in Prometheus for the same window.
|
||
|
||
## Cost control
|
||
|
||
Every chat turn makes 3–8 Bedrock calls on a personal AWS account. The only
|
||
guard is the in-memory rate limiter in `apps/web/middleware.ts`
|
||
(12/min, 120/hour per IP for `/api/chat`). There is no budget alarm, no
|
||
per-day cap and no authentication in the repository. Scaling `web` past one
|
||
replica multiplies the effective allowance.
|
||
|
||
## Incident quick reference
|
||
|
||
| Symptom | First check |
|
||
|---|---|
|
||
| Every answer is an abstain | `duocthu_abstention_total{reason}` — a single dominant reason points at a provider or corpus problem |
|
||
| `ai-service` restart loop | `docker logs docker-ai-service-1` for `ManifestMismatch` |
|
||
| 503 from `/v1/rag/query` | `EMBEDDING_PROVIDER` in `.env.prod`, and whether the manifest check passed |
|
||
| Answers take ~60 s then fail | `duocthu_generation_rejected_total{reason="request_budget_exhausted"}` |
|
||
| 429s | Rate limiter; `X-RateLimit-*` headers on the response |
|
||
| No traces in Grafana | Was the observability overlay included in the last `up`? |
|
||
|
||
Full table in [25-troubleshooting.md](25-troubleshooting.md).
|