# Codex observability handoff — 2026-08-11

This is the durable memory for the observability work completed on the owner's
personal AWS infrastructure. It records measured state, not the older target
architecture. Read it together with `CLAUDE_CLAIM_2026-08-11.md` before making
further changes.

## Ownership boundary

- Production remains the owner's EC2 + Docker Compose deployment. No team
  k3s, Gitea, ArgoCD repository or cluster was changed.
- Do not overwrite Claude's active work or the user's existing untracked logs,
  screenshots and `.codex/` files.
- At this handoff, the working tree has unrelated, uncommitted changes in
  `apps/ai-service/config.py`, `main.py`, `rag/prompt.py`, `routers/rag.py`,
  `tests/test_api.py`, `apps/web/app/api/chat/route.ts`, plus an untracked
  `tests/test_prompt_untrusted_input.py`. They were not staged or committed by
  this work.

## Live topology

- EC2: `duocthu-prod`, `t3.large`, `us-east-1`, public IP `52.0.158.61`.
- Application: Caddy -> Next.js BFF -> FastAPI -> Qdrant/Bedrock/PostgreSQL.
- Observability: FastAPI/OpenTelemetry -> OTel Collector -> Tempo;
  FastAPI `/metrics` -> Prometheus; Grafana queries Prometheus and Tempo.
- Production base Compose remains `infra/docker/docker-compose.prod.yml`.
  Observability is the additive
  `infra/docker/docker-compose.observability.yml` overlay.

## What is deployed

- Correlation ID and W3C trace propagation from the Next.js BFF into FastAPI.
- OpenTelemetry spans for receive, understanding, routing, retrieval,
  rerank/evidence, generation, grounding/entailment, persistence and response.
- Prometheus request/domain counters and latency histograms, bounded labels and
  trace exemplars.
- Prometheus 3.3.0, Grafana 11.5.2, Tempo 2.7.2 and OTel Collector 0.123.0.
- Grafana datasource UIDs `prometheus` and `tempo`.
- Dashboard UID `duocthu-observability`, title
  `Dược Thư — Request path observability`, eight panels.
- PostgreSQL migration `003_rag_trace_correlation.sql` is applied in production.

## Access and security

- Public Grafana login: `https://realvuxbaro.me/grafana/`.
- Anonymous Grafana access is disabled. Admin user is `admin`; its generated
  password is stored only as GitHub Actions secret
  `GRAFANA_ADMIN_PASSWORD` and was not printed into logs or committed.
- Grafana fallback tunnel:
  `ssh -L 3002:127.0.0.1:3002 ubuntu@52.0.158.61`, then open
  `http://localhost:3002/grafana/`.
- Prometheus intentionally has no public route. Use Grafana Explore normally,
  or tunnel it with
  `ssh -L 9090:127.0.0.1:9090 ubuntu@52.0.158.61` and open
  `http://localhost:9090`.
- EC2 native ports 3002 and 9090 were externally probed and both were closed.
- The EC2 security group exposes only 22, 80 and 443.

## Answer lineage

Use three views together:

1. The web evidence panel shows selected source chunks, pages and evidence.
2. Grafana Explore -> Tempo shows the executed pipeline stages, latency,
   decision/reason, failures and trace/correlation IDs.
3. PostgreSQL `rag_retrieval_trace` stores the durable query, resolved drug,
   decision/reason, citation/evidence payload, correlation ID and OTel trace ID.

This is execution/provenance tracing, not chain-of-thought capture. Full prompts,
model hidden reasoning, all rejected candidates and every ranking score are not
stored.

## Production verification

- GitHub Actions run `31459113823` completed successfully for commit `90b67fa`.
- FastAPI `/health` and `/ready`, Next.js, Prometheus readiness, Tempo readiness
  and Grafana health all passed from the EC2 Compose network.
- Grafana APIs confirmed both datasources and dashboard UID
  `duocthu-observability` were provisioned.
- A real RAG smoke request produced `duocthu_requests_total` in Prometheus.
- The deploy check extracted that request's 32-character `X-Trace-ID` and Tempo
  returned the exact `/api/traces/<trace_id>` record.
- An external request to `https://realvuxbaro.me/api/chat` returned HTTP 200,
  echoed the supplied correlation ID and returned an OTel trace ID.
- Public `https://realvuxbaro.me/grafana/login` returned HTTP 200 through Caddy.

## Relevant commits

- `ceb12d7` — application tracing, metrics, Compose stack and Helm manifests.
- `9826407` — observability access and answer-lineage documentation.
- `2159dfc`, `963760b` — production observability deployment and readiness retry.
- `640270a`, `8238eeb` — exact trace and Grafana provisioning verification.
- `90b67fa` — public `/grafana/` route plus loopback-only Grafana/Prometheus
  tunnel ports.

## Pending DNS follow-up

`realvuxbaro.me` is managed by Namecheap. At the final check,
`grafana.realvuxbaro.me` did not resolve and no Namecheap API credential was
available in the environment or GitHub secrets. The owner was entering this
record manually:

- Type: `A Record`
- Host: `grafana`
- Value: `52.0.158.61`
- TTL: `Automatic`

After it resolves, add the subdomain to Caddy, obtain/verify its automatic TLS
certificate and decide which URL is canonical. Preserve both the existing
`/grafana/` entry path (a redirect is acceptable) and the SSH fallback. Keep
Prometheus private.