duocthu/coordination/CODEX_OBSERVABILITY_HANDOFF_2026-08-11.md

Codex observability handoff — 2026-08-11

This is the durable memory for the observability work completed on the owner's personal AWS infrastructure. It records measured state, not the older target architecture. Read it together with CLAUDE_CLAIM_2026-08-11.md before making further changes.

Ownership boundary

• Production remains the owner's EC2 + Docker Compose deployment. No team k3s, Gitea, ArgoCD repository or cluster was changed.
• Do not overwrite Claude's active work or the user's existing untracked logs, screenshots and .codex/ files.
• At this handoff, the working tree has unrelated, uncommitted changes in apps/ai-service/config.py, main.py, rag/prompt.py, routers/rag.py, tests/test_api.py, apps/web/app/api/chat/route.ts, plus an untracked tests/test_prompt_untrusted_input.py. They were not staged or committed by this work.

Live topology

• EC2: duocthu-prod, t3.large, us-east-1, public IP 52.0.158.61.
• Application: Caddy -> Next.js BFF -> FastAPI -> Qdrant/Bedrock/PostgreSQL.
• Observability: FastAPI/OpenTelemetry -> OTel Collector -> Tempo; FastAPI /metrics -> Prometheus; Grafana queries Prometheus and Tempo.
• Production base Compose remains infra/docker/docker-compose.prod.yml. Observability is the additive infra/docker/docker-compose.observability.yml overlay.

What is deployed

• Correlation ID and W3C trace propagation from the Next.js BFF into FastAPI.
• OpenTelemetry spans for receive, understanding, routing, retrieval, rerank/evidence, generation, grounding/entailment, persistence and response.
• Prometheus request/domain counters and latency histograms, bounded labels and trace exemplars.
• Prometheus 3.3.0, Grafana 11.5.2, Tempo 2.7.2 and OTel Collector 0.123.0.
• Grafana datasource UIDs prometheus and tempo.
• Dashboard UID duocthu-observability, title Dược Thư — Request path observability, eight panels.
• PostgreSQL migration 003_rag_trace_correlation.sql is applied in production.

Access and security

• Public Grafana login: https://realvuxbaro.me/grafana/.
• Anonymous Grafana access is disabled. Admin user is admin; its generated password is stored only as GitHub Actions secret GRAFANA_ADMIN_PASSWORD and was not printed into logs or committed.
• Grafana fallback tunnel: ssh -L 3002:127.0.0.1:3002 ubuntu@52.0.158.61, then open http://localhost:3002/grafana/.
• Prometheus intentionally has no public route. Use Grafana Explore normally, or tunnel it with ssh -L 9090:127.0.0.1:9090 ubuntu@52.0.158.61 and open http://localhost:9090.
• EC2 native ports 3002 and 9090 were externally probed and both were closed.
• The EC2 security group exposes only 22, 80 and 443.

Answer lineage

Use three views together:

1. The web evidence panel shows selected source chunks, pages and evidence.
2. Grafana Explore -> Tempo shows the executed pipeline stages, latency, decision/reason, failures and trace/correlation IDs.
3. PostgreSQL rag_retrieval_trace stores the durable query, resolved drug, decision/reason, citation/evidence payload, correlation ID and OTel trace ID.

This is execution/provenance tracing, not chain-of-thought capture. Full prompts, model hidden reasoning, all rejected candidates and every ranking score are not stored.

Production verification

• GitHub Actions run 31459113823 completed successfully for commit 90b67fa.
• FastAPI /health and /ready, Next.js, Prometheus readiness, Tempo readiness and Grafana health all passed from the EC2 Compose network.
• Grafana APIs confirmed both datasources and dashboard UID duocthu-observability were provisioned.
• A real RAG smoke request produced duocthu_requests_total in Prometheus.
• The deploy check extracted that request's 32-character X-Trace-ID and Tempo returned the exact /api/traces/<trace_id> record.
• An external request to https://realvuxbaro.me/api/chat returned HTTP 200, echoed the supplied correlation ID and returned an OTel trace ID.
• Public https://realvuxbaro.me/grafana/login returned HTTP 200 through Caddy.

Relevant commits

• ceb12d7 — application tracing, metrics, Compose stack and Helm manifests.
• 9826407 — observability access and answer-lineage documentation.
• 2159dfc, 963760b — production observability deployment and readiness retry.
• 640270a, 8238eeb — exact trace and Grafana provisioning verification.
• 90b67fa — public /grafana/ route plus loopback-only Grafana/Prometheus tunnel ports.

Pending DNS follow-up

realvuxbaro.me is managed by Namecheap. At the final check, grafana.realvuxbaro.me did not resolve and no Namecheap API credential was available in the environment or GitHub secrets. The owner was entering this record manually:

• Type: A Record
• Host: grafana
• Value: 52.0.158.61
• TTL: Automatic

After it resolves, add the subdomain to Caddy, obtain/verify its automatic TLS certificate and decide which URL is canonical. Preserve both the existing /grafana/ entry path (a redirect is acceptable) and the SSH fallback. Keep Prometheus private.