Fix migration workflow: upload as artifact instead of scp to practice EC2
This commit is contained in:
@@ -0,0 +1,120 @@
|
||||
# 22 — CI/CD
|
||||
|
||||
## What exists
|
||||
|
||||
Exactly one workflow: `.github/workflows/deploy.yml`.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[push to master] --> D["job: deploy<br/>ubuntu-latest"]
|
||||
D --> S["appleboy/ssh-action@v1.0.3<br/>ssh to EC2_HOST as ubuntu"]
|
||||
S --> G["git fetch + reset --hard origin/master"]
|
||||
G --> B["docker compose up -d --build<br/>(prod + observability overlays)"]
|
||||
B --> R["caddy validate + reload"]
|
||||
R --> M["python -m migrate"]
|
||||
M --> V["verification block — 15+ assertions"]
|
||||
V -->|any fails| F["job fails; ai-service logs dumped"]
|
||||
V -->|all pass| OK[done]
|
||||
```
|
||||
|
||||
There is **no CI** in the usual sense — the pipeline stops at the *first* box of
|
||||
the conventional diagram and jumps straight to deploy:
|
||||
|
||||
```
|
||||
commit → [ lint ✗ ] → [ tests ✗ ] → [ build ✓ on prod host ] →
|
||||
[ registry ✗ ] → [ manifest update ✗ ] → [ ArgoCD ✗ ] → rollout ✓
|
||||
```
|
||||
|
||||
Concretely, none of the following runs anywhere in CI:
|
||||
|
||||
- `ruff` (configured in `apps/ai-service/pyproject.toml`, never invoked)
|
||||
- `pytest` for either suite (555 tests)
|
||||
- `tsc` / `next lint` / `turbo run lint` / `turbo run build`
|
||||
- `helm lint` or `helm template`
|
||||
- Any dependency or image vulnerability scan
|
||||
|
||||
A commit that breaks every test deploys to production.
|
||||
|
||||
## Triggers
|
||||
|
||||
```yaml
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
workflow_dispatch:
|
||||
```
|
||||
|
||||
No `pull_request` trigger, so a PR receives no automated feedback at all. No
|
||||
environment protection rule, no required approval.
|
||||
|
||||
## Secrets used
|
||||
|
||||
| Secret | Use |
|
||||
|---|---|
|
||||
| `EC2_HOST` | SSH target |
|
||||
| `EC2_SSH_KEY` | SSH private key |
|
||||
| `GRAFANA_ADMIN_PASSWORD` | Passed through `envs:`; the script `test -n`s it and exports it for Compose |
|
||||
|
||||
No AWS credentials are needed — Bedrock is reached through the instance role.
|
||||
|
||||
## The verification block is the real quality gate
|
||||
|
||||
Everything after `docker compose up` is assertion, and `set -e` makes each one
|
||||
fatal. In order:
|
||||
|
||||
| # | Assertion |
|
||||
|---|---|
|
||||
| 1 | `caddy validate --config /etc/caddy/Caddyfile` then `caddy reload` |
|
||||
| 2 | `python -m migrate` inside the ai-service container |
|
||||
| 3 | `GET ai-service:8000/health` |
|
||||
| 4 | `GET ai-service:8000/ready` |
|
||||
| 5 | `GET web:3000` |
|
||||
| 6 | `POST /v1/rag/query` with a real condition→drug question; on failure, dump the last 200 ai-service log lines |
|
||||
| 7 | Response contains `"decision":"answerable"` |
|
||||
| 8 | Response contains `"section_key":"chi_dinh"` |
|
||||
| 9 | `GET prometheus:9090/-/ready` |
|
||||
| 10 | `GET tempo:3200/ready`, retried 12 × 5 s, dumping tempo logs on final failure |
|
||||
| 11 | `GET grafana:3000/api/health` |
|
||||
| 12 | Grafana datasource `prometheus` exists (admin-authenticated) |
|
||||
| 13 | Grafana datasource `tempo` exists |
|
||||
| 14 | Grafana dashboard `duocthu-observability` exists |
|
||||
| 15 | `GET https://realvuxbaro.me/grafana/login` — through the public edge |
|
||||
| 16 | A second `POST /v1/rag/query` with a generated correlation id; the `X-Trace-ID` response header must match `^[0-9a-f]{32}$` |
|
||||
| 17 | After 20 s, `duocthu_requests_total` is queryable in Prometheus |
|
||||
| 18 | That exact trace id is retrievable from `tempo:3200/api/traces/<id>`, retried 12 × 5 s |
|
||||
|
||||
Assertions 6–8 and 16–18 are unusually strong for a deploy script: one verifies
|
||||
a real grounded answer from the real corpus, the other verifies that a specific
|
||||
request's trace actually landed in Tempo.
|
||||
|
||||
## Consequences of the current design
|
||||
|
||||
| Property | Effect |
|
||||
|---|---|
|
||||
| Build happens on the production host | A build failure occurs *after* `git reset --hard`, so the checkout has already moved even if the new image never starts |
|
||||
| No image tags | No artifact to roll back to; recovery is a revert commit plus a full rebuild |
|
||||
| No test gate | Regressions are caught by the deploy smoke test (one behaviour) or by users |
|
||||
| No PR feedback | Review is unassisted |
|
||||
| Deploy is in-place | Brief downtime per service while it rebuilds and restarts |
|
||||
| `postgres`/`qdrant` are not in the `up` list | Stateful services are never restarted by a deploy — good for uptime, but changes to their compose definitions silently do not apply |
|
||||
|
||||
## What `infra/ci/github-actions/README.md` promises
|
||||
|
||||
Five workflows, described as "not yet functional — filled in during Phase 6":
|
||||
`ai-service-ci.yml`, `node-services-ci.yml`, `web-ci.yml`, `ingestion-ci.yml`,
|
||||
`bump-image-tag.yml`. **None of them exists.** `bump-image-tag.yml` is the
|
||||
linchpin of the GitOps flow described in
|
||||
[21-kubernetes-and-argocd.md](21-kubernetes-and-argocd.md), so that flow cannot
|
||||
run.
|
||||
|
||||
## Lowest-effort improvements, in order
|
||||
|
||||
1. Add a `pull_request` + `push` workflow that runs both pytest suites — the
|
||||
commands are two lines and already work
|
||||
([18-testing.md](18-testing.md)), and `EMBEDDING_PROVIDER=disabled` is the
|
||||
only setup needed.
|
||||
2. Add `ruff check` for `apps/ai-service` (config already present) and
|
||||
`turbo run lint build` for the JS workspace.
|
||||
3. Make `deploy` depend on those jobs.
|
||||
4. Build and tag images in CI, push to a registry, and have the host pull a tag
|
||||
— which also makes rollback possible.
|
||||
Reference in New Issue
Block a user