Fix migration workflow: upload as artifact instead of scp to practice EC2

2026-08-13 11:14:25 +07:00
parent 7ebbe1f309
commit a4819b8653
51 changed files with 6830 additions and 8 deletions
@@ -0,0 +1,216 @@
+# 24 — Production operations
+
+Production is one EC2 host running Docker Compose. Every command below assumes
+an SSH session on that host in `~/app/infra/docker`, matching what
+`.github/workflows/deploy.yml` does.
+
+The Compose project name is `docker` (the directory name), so containers are
+named `docker-<service>-1`.
+
+## Compose invocation
+
+Both overlay files are always used together:
+
+```bash
+cd ~/app/infra/docker
+COMPOSE="sudo docker compose -f docker-compose.prod.yml -f docker-compose.observability.yml"
+```
+
+The observability overlay is what sets `OTEL_ENABLED=true` on `ai-service`, so
+omitting it silently disables tracing.
+
+## Start / stop / restart
+
+```bash
+$COMPOSE ps
+$COMPOSE up -d ai-service web caddy                  # start/refresh app tier
+$COMPOSE restart ai-service                          # restart one service
+$COMPOSE stop ai-service
+$COMPOSE logs -f --tail 200 ai-service
+```
+
+`ai-service` builds its whole runtime at import time, so a restart re-runs the
+corpus-manifest check. If that check fails the container exits immediately and
+keeps restarting — check the logs for `ManifestMismatch` before assuming a crash
+loop is resource-related.
+
+## Deploy
+
+Normal path: push to `master`. The workflow SSHes in, resets the checkout,
+rebuilds, reloads Caddy, migrates and runs ~18 assertions
+([22-ci-cd.md](22-ci-cd.md)).
+
+Manual equivalent:
+
+```bash
+cd ~/app && git fetch origin master && git reset --hard origin/master
+cd infra/docker
+export GRAFANA_ADMIN_PASSWORD='<value>'
+sudo -E docker compose -f docker-compose.prod.yml -f docker-compose.observability.yml \
+  up -d --build ai-service web prometheus tempo otel-collector grafana caddy
+sudo docker exec docker-caddy-1 caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
+sudo docker exec docker-caddy-1 caddy reload   --config /etc/caddy/Caddyfile --adapter caddyfile
+sudo docker exec docker-ai-service-1 python -m migrate
+```
+
+Note `postgres` and `qdrant` are deliberately absent from that list — a code
+deploy never restarts the stateful services.
+
+## Rollback
+
+There is no image to roll back to (images are built on the host, untagged). The
+procedure is:
+
+```bash
+cd ~/app
+git reset --hard <last-good-sha>      # or push a revert to master and let CI deploy
+cd infra/docker && sudo -E docker compose -f docker-compose.prod.yml \
+  -f docker-compose.observability.yml up -d --build ai-service web
+```
+
+A rollback that crosses a migration is **not covered** — migrations are
+forward-only with no down scripts.
+
+## Health checks
+
+```bash
+NET=docker_default
+sudo docker run --rm --network $NET curlimages/curl -sf http://ai-service:8000/health
+sudo docker run --rm --network $NET curlimages/curl -sf http://ai-service:8000/ready
+sudo docker run --rm --network $NET curlimages/curl -sf -o /dev/null http://web:3000
+sudo docker run --rm --network $NET curlimages/curl -sf http://prometheus:9090/-/ready
+sudo docker run --rm --network $NET curlimages/curl -sf http://tempo:3200/ready
+sudo docker run --rm --network $NET curlimages/curl -sf http://grafana:3000/api/health
+```
+
+`ai-service` publishes no host port, so every check goes through a throwaway
+container on the Compose network — the same technique the deploy workflow uses.
+
+## Smoke test a real answer
+
+```bash
+sudo docker run --rm --network docker_default curlimages/curl -sf \
+  -X POST http://ai-service:8000/v1/rag/query \
+  -H 'Content-Type: application/json' \
+  --data '{"query":"Đợt gout cấp có thuốc nào được Dược thư ghi chỉ định?",
+           "subject_scope":"human","intent":"fact_lookup",
+           "conversation_id":"ops-smoke"}'
+```
+
+Expect `"decision":"answerable"` and at least one citation with
+`"section_key":"chi_dinh"` — the same two assertions the deploy makes.
+
+## Database operations
+
+```bash
+# psql
+sudo docker exec -it docker-postgres-1 psql -U duoc_thu -d duoc_thu
+
+# apply migrations
+sudo docker exec docker-ai-service-1 python -m migrate
+```
+
+Useful queries:
+
+```sql
+-- recent decisions
+SELECT created_at, decision, reason, resolved_drug_id
+FROM rag_retrieval_trace ORDER BY created_at DESC LIMIT 50;
+
+-- abstain reasons over the last day
+SELECT reason, count(*) FROM rag_retrieval_trace
+WHERE decision = 'abstain' AND created_at > now() - interval '1 day'
+GROUP BY reason ORDER BY 2 DESC;
+
+-- find a support request by either correlation id
+SELECT * FROM rag_retrieval_trace WHERE correlation_id = '<id>';
+SELECT * FROM rag_retrieval_trace WHERE otel_trace_id  = '<32-hex>';
+
+-- negative feedback with the question that caused it
+SELECT f.created_at, f.rating, f.comment, t.query_text, t.decision, t.reason
+FROM rag_answer_feedback f JOIN rag_retrieval_trace t USING (trace_id)
+WHERE f.rating = 'not_helpful' ORDER BY f.created_at DESC LIMIT 50;
+```
+
+## Backup and restore
+
+**No backup automation exists in this repository.** What the code supports:
+
+```bash
+# PostgreSQL logical dump
+sudo docker exec docker-postgres-1 pg_dump -U duoc_thu duoc_thu > duoc_thu_$(date +%F).sql
+
+# Qdrant snapshot (HTTP API, from inside the network)
+sudo docker run --rm --network docker_default curlimages/curl -s -X POST \
+  http://qdrant:6333/collections/duocthu_v1/snapshots
+```
+
+Both are manual. Whether EBS snapshots are configured on the instance cannot be
+determined from the repository.
+
+## Re-indexing / re-ingestion
+
+Two situations, with very different costs:
+
+**Corpus content unchanged, moving or restoring it** — snapshot and restore the
+Qdrant collection. Free and exact; `ingestion/README.md` recommends it
+explicitly.
+
+**Corpus content changed** — the full pipeline must re-run and the embed step
+**costs real AWS Bedrock spend on a personal account**. `ingestion/README.md`
+requires explicit approval for any specific run. Order:
+
+1. `python -m ingestion.cli run` → new `monographs.jsonl`
+2. `python -m ingestion.cli chunk` → new `chunks.jsonl`
+3. `python -m ingestion.cli chunk-ready` — **must exit 0**
+4. `python -m ingestion.load.run --provider cohere-v4 --collection duocthu_v2 …`
+
+Use a **new collection name**. The loader refuses to write a different
+`corpus_sha256` into an existing collection (`CorpusMismatch`), which is the
+intended behaviour, not an obstacle to work around. Then point
+`QDRANT_COLLECTION` at the new collection and restart `ai-service`; the startup
+manifest check verifies the binding. Keep the old collection until the new one
+is confirmed — that is the rollback.
+
+The embedding cache in `ingestion/data/processed/embeddings/` is keyed by
+content hash, so unchanged chunks are not re-paid for.
+
+## Grafana
+
+Reachable at `https://realvuxbaro.me/grafana/` with the admin credentials from
+`GRAFANA_ADMIN_PASSWORD`. Locally on the host: `http://127.0.0.1:3002`.
+Provisioned datasources `prometheus` and `tempo`; dashboard uid
+`duocthu-observability`.
+
+## Following one request end to end
+
+1. Take `X-Correlation-ID` or `X-Trace-ID` from the user's response headers (the
+   UI surfaces `traceId` on each message).
+2. `SELECT * FROM rag_retrieval_trace WHERE correlation_id = …` → the resolved
+   scope, decision, reason and citations.
+3. Open the trace id in Grafana → Tempo → per-stage spans
+   (`rag.stage.understanding`, `retrieval`, `generation`, `entailment`) with
+   `duocthu.*` attributes.
+4. Cross-check `duocthu_generation_rejected_total{reason=…}` and
+   `duocthu_abstention_total{reason=…}` in Prometheus for the same window.
+
+## Cost control
+
+Every chat turn makes 3–8 Bedrock calls on a personal AWS account. The only
+guard is the in-memory rate limiter in `apps/web/middleware.ts`
+(12/min, 120/hour per IP for `/api/chat`). There is no budget alarm, no
+per-day cap and no authentication in the repository. Scaling `web` past one
+replica multiplies the effective allowance.
+
+## Incident quick reference
+
+| Symptom | First check |
+|---|---|
+| Every answer is an abstain | `duocthu_abstention_total{reason}` — a single dominant reason points at a provider or corpus problem |
+| `ai-service` restart loop | `docker logs docker-ai-service-1` for `ManifestMismatch` |
+| 503 from `/v1/rag/query` | `EMBEDDING_PROVIDER` in `.env.prod`, and whether the manifest check passed |
+| Answers take ~60 s then fail | `duocthu_generation_rejected_total{reason="request_budget_exhausted"}` |
+| 429s | Rate limiter; `X-RateLimit-*` headers on the response |
+| No traces in Grafana | Was the observability overlay included in the last `up`? |
+
+Full table in [25-troubleshooting.md](25-troubleshooting.md).