Add production condition retrieval smoke test
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# Codex observability handoff — 2026-08-11
|
||||
|
||||
This is the durable memory for the observability work completed on the owner's
|
||||
personal AWS infrastructure. It records measured state, not the older target
|
||||
architecture. Read it together with `CLAUDE_CLAIM_2026-08-11.md` before making
|
||||
further changes.
|
||||
|
||||
## Ownership boundary
|
||||
|
||||
- Production remains the owner's EC2 + Docker Compose deployment. No team
|
||||
k3s, Gitea, ArgoCD repository or cluster was changed.
|
||||
- Do not overwrite Claude's active work or the user's existing untracked logs,
|
||||
screenshots and `.codex/` files.
|
||||
- At this handoff, the working tree has unrelated, uncommitted changes in
|
||||
`apps/ai-service/config.py`, `main.py`, `rag/prompt.py`, `routers/rag.py`,
|
||||
`tests/test_api.py`, `apps/web/app/api/chat/route.ts`, plus an untracked
|
||||
`tests/test_prompt_untrusted_input.py`. They were not staged or committed by
|
||||
this work.
|
||||
|
||||
## Live topology
|
||||
|
||||
- EC2: `duocthu-prod`, `t3.large`, `us-east-1`, public IP `52.0.158.61`.
|
||||
- Application: Caddy -> Next.js BFF -> FastAPI -> Qdrant/Bedrock/PostgreSQL.
|
||||
- Observability: FastAPI/OpenTelemetry -> OTel Collector -> Tempo;
|
||||
FastAPI `/metrics` -> Prometheus; Grafana queries Prometheus and Tempo.
|
||||
- Production base Compose remains `infra/docker/docker-compose.prod.yml`.
|
||||
Observability is the additive
|
||||
`infra/docker/docker-compose.observability.yml` overlay.
|
||||
|
||||
## What is deployed
|
||||
|
||||
- Correlation ID and W3C trace propagation from the Next.js BFF into FastAPI.
|
||||
- OpenTelemetry spans for receive, understanding, routing, retrieval,
|
||||
rerank/evidence, generation, grounding/entailment, persistence and response.
|
||||
- Prometheus request/domain counters and latency histograms, bounded labels and
|
||||
trace exemplars.
|
||||
- Prometheus 3.3.0, Grafana 11.5.2, Tempo 2.7.2 and OTel Collector 0.123.0.
|
||||
- Grafana datasource UIDs `prometheus` and `tempo`.
|
||||
- Dashboard UID `duocthu-observability`, title
|
||||
`Dược Thư — Request path observability`, eight panels.
|
||||
- PostgreSQL migration `003_rag_trace_correlation.sql` is applied in production.
|
||||
|
||||
## Access and security
|
||||
|
||||
- Public Grafana login: `https://realvuxbaro.me/grafana/`.
|
||||
- Anonymous Grafana access is disabled. Admin user is `admin`; its generated
|
||||
password is stored only as GitHub Actions secret
|
||||
`GRAFANA_ADMIN_PASSWORD` and was not printed into logs or committed.
|
||||
- Grafana fallback tunnel:
|
||||
`ssh -L 3002:127.0.0.1:3002 ubuntu@52.0.158.61`, then open
|
||||
`http://localhost:3002/grafana/`.
|
||||
- Prometheus intentionally has no public route. Use Grafana Explore normally,
|
||||
or tunnel it with
|
||||
`ssh -L 9090:127.0.0.1:9090 ubuntu@52.0.158.61` and open
|
||||
`http://localhost:9090`.
|
||||
- EC2 native ports 3002 and 9090 were externally probed and both were closed.
|
||||
- The EC2 security group exposes only 22, 80 and 443.
|
||||
|
||||
## Answer lineage
|
||||
|
||||
Use three views together:
|
||||
|
||||
1. The web evidence panel shows selected source chunks, pages and evidence.
|
||||
2. Grafana Explore -> Tempo shows the executed pipeline stages, latency,
|
||||
decision/reason, failures and trace/correlation IDs.
|
||||
3. PostgreSQL `rag_retrieval_trace` stores the durable query, resolved drug,
|
||||
decision/reason, citation/evidence payload, correlation ID and OTel trace ID.
|
||||
|
||||
This is execution/provenance tracing, not chain-of-thought capture. Full prompts,
|
||||
model hidden reasoning, all rejected candidates and every ranking score are not
|
||||
stored.
|
||||
|
||||
## Production verification
|
||||
|
||||
- GitHub Actions run `31459113823` completed successfully for commit `90b67fa`.
|
||||
- FastAPI `/health` and `/ready`, Next.js, Prometheus readiness, Tempo readiness
|
||||
and Grafana health all passed from the EC2 Compose network.
|
||||
- Grafana APIs confirmed both datasources and dashboard UID
|
||||
`duocthu-observability` were provisioned.
|
||||
- A real RAG smoke request produced `duocthu_requests_total` in Prometheus.
|
||||
- The deploy check extracted that request's 32-character `X-Trace-ID` and Tempo
|
||||
returned the exact `/api/traces/<trace_id>` record.
|
||||
- An external request to `https://realvuxbaro.me/api/chat` returned HTTP 200,
|
||||
echoed the supplied correlation ID and returned an OTel trace ID.
|
||||
- Public `https://realvuxbaro.me/grafana/login` returned HTTP 200 through Caddy.
|
||||
|
||||
## Relevant commits
|
||||
|
||||
- `ceb12d7` — application tracing, metrics, Compose stack and Helm manifests.
|
||||
- `9826407` — observability access and answer-lineage documentation.
|
||||
- `2159dfc`, `963760b` — production observability deployment and readiness retry.
|
||||
- `640270a`, `8238eeb` — exact trace and Grafana provisioning verification.
|
||||
- `90b67fa` — public `/grafana/` route plus loopback-only Grafana/Prometheus
|
||||
tunnel ports.
|
||||
|
||||
## Pending DNS follow-up
|
||||
|
||||
`realvuxbaro.me` is managed by Namecheap. At the final check,
|
||||
`grafana.realvuxbaro.me` did not resolve and no Namecheap API credential was
|
||||
available in the environment or GitHub secrets. The owner was entering this
|
||||
record manually:
|
||||
|
||||
- Type: `A Record`
|
||||
- Host: `grafana`
|
||||
- Value: `52.0.158.61`
|
||||
- TTL: `Automatic`
|
||||
|
||||
After it resolves, add the subdomain to Caddy, obtain/verify its automatic TLS
|
||||
certificate and decide which URL is canonical. Preserve both the existing
|
||||
`/grafana/` entry path (a redirect is acceptable) and the SSH fallback. Keep
|
||||
Prometheus private.
|
||||
Reference in New Issue
Block a user