Add production condition retrieval smoke test

This commit is contained in:
2026-08-11 14:58:28 +07:00
parent 59e6ad2d0d
commit 7ebbe1f309
38 changed files with 3752 additions and 121 deletions
+27 -13
View File
@@ -160,20 +160,33 @@ docker compose `
up -d
```
Only Grafana is mapped to the EC2 host (`3002:3000`) by the production overlay;
Prometheus and Tempo stay on the internal Compose network. The EC2 security
group does not expose port 3002 publicly. View Grafana through an SSH tunnel:
Grafana is available directly through Caddy and the existing production TLS
certificate at `https://realvuxbaro.me/grafana/`. Anonymous access is disabled;
sign in with the Grafana admin account. A dedicated
`grafana.realvuxbaro.me` hostname can replace this path after its Namecheap A
record exists.
Grafana and Prometheus are also bound to EC2 loopback only. This keeps both SSH
fallbacks available without exposing their native ports to the Internet:
```powershell
ssh -L 3002:127.0.0.1:3002 <ssh-user>@52.0.158.61
ssh `
-L 3002:127.0.0.1:3002 `
-L 9090:127.0.0.1:9090 `
<ssh-user>@52.0.158.61
```
Keep that session open and visit `http://localhost:3002`. Set
`GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD` in the production environment
before deployment; do not use the fallback password in production. Prometheus
metrics are available in **Grafana -> Explore -> Prometheus**. To investigate a
slow request, open the request-latency panel, follow its exemplar/trace link, or
paste the returned `X-Trace-ID` into **Explore -> Tempo**.
Keep that session open and use `http://localhost:3002/grafana/` for Grafana or
`http://localhost:9090` for the raw Prometheus UI. The same Grafana account is
used through both the public HTTPS path and the SSH tunnel. The production
password lives in the GitHub Actions secret `GRAFANA_ADMIN_PASSWORD`; do not use
the Compose fallback password in production.
Prometheus intentionally has no public URL. Normally use
**Grafana -> Explore -> Prometheus**; use its SSH tunnel only for low-level
target or PromQL diagnostics. To investigate a slow request, open the
request-latency panel, follow its exemplar/trace link, or paste the returned
`X-Trace-ID` into **Explore -> Tempo**.
## Production
@@ -183,9 +196,10 @@ web, Caddy for automatic Let's Encrypt TLS). Bedrock is reached through an IAM
instance role — there are no long-lived AWS keys on the box or in any env file.
Pushing to `master` deploys: `.github/workflows/deploy.yml` SSHes in, resets to
the pushed commit, rebuilds only `ai-service`/`web`, runs migrations and
health-checks both. Postgres/Qdrant/Caddy are left untouched, so the vector
data survives deploys (it lives in a named volume, not the container).
the pushed commit, rebuilds `ai-service`/`web`, reconciles the observability
containers, reloads Caddy, runs migrations and verifies health, metrics and an
exact request trace. Postgres and Qdrant data survive deploys because they live
in named volumes rather than the containers.
This is **interim infrastructure**, not the end state. The intended target is
still the team's self-hosted **Gitea** (company domain) plus their **ArgoCD**