Verify exact production traces and record rollout
This commit is contained in:
@@ -30,8 +30,9 @@ today.
|
||||
> | `apps/web/` | Done — chat UI with citation/evidence panel |
|
||||
> | `apps/api-gateway`, `auth-service`, `user-service`, `chat-service` | **Not built** — `README.md` + `package.json` only |
|
||||
> | `apps/mobile/` | **Not built** — reserved |
|
||||
> | `infra/docker/` | Done — this is what production actually runs |
|
||||
> | `infra/k8s`, `helm`, `terraform`, `argocd` | **Not built yet** — empty scaffold. Still the target (ADR 0002), not abandoned: the plan is the team's self-hosted Gitea + ArgoCD; the current EC2/Compose setup is an interim stopgap |
|
||||
> | `infra/docker/` | Done — production runs Compose, including the Prometheus/Grafana/Tempo observability overlay |
|
||||
> | `infra/helm/medical-chatbot/` | Built and validated as an offline migration kit; **not deployed** to Docker Desktop, k3s or ArgoCD |
|
||||
> | `infra/k8s`, `terraform`, `argocd` | **Not built yet** — still the target (ADR 0002), not abandoned: the plan is the team's self-hosted Gitea + ArgoCD; the current EC2/Compose setup is an interim stopgap |
|
||||
>
|
||||
> Because the gateway and auth services do not exist, `apps/web` talks
|
||||
> **directly** to `apps/ai-service`; there is no authentication layer. See
|
||||
@@ -96,6 +97,84 @@ Tests: `cd apps/ai-service && python -m pytest -q` — 230 pass. `test_api.py` a
|
||||
with `--ignore` when the stack is down. `apps/web` has **no test setup at all**,
|
||||
so a green suite says nothing about the frontend — drive it in a browser.
|
||||
|
||||
## Observability: Prometheus, Grafana and Tempo
|
||||
|
||||
The observability stack is provisioned in the repository and has been deployed
|
||||
to the production EC2 instance since 2026-08-11.
|
||||
|
||||
- **Prometheus** scrapes `/metrics` from `ai-service`. It records request rate
|
||||
and latency, latency for each RAG stage, routing decisions and reasons,
|
||||
provider failures, trace-write failures and the existing domain counters.
|
||||
- **Grafana** is the user interface for dashboards and metric queries. Its
|
||||
datasource and the Dược Thư dashboard are provisioned automatically.
|
||||
- **Tempo** stores OpenTelemetry traces. A trace contains the receive,
|
||||
understanding, routing, retrieval, rerank/evidence, generation,
|
||||
grounding/entailment, persistence and response stages. Correlation and trace
|
||||
IDs follow the request from the Next.js BFF into FastAPI.
|
||||
- **OpenTelemetry Collector** receives spans from `ai-service` and exports them
|
||||
to Tempo. Grafana exemplars link aggregate latency metrics to an individual
|
||||
Tempo trace.
|
||||
|
||||
For answer lineage, use the three views together:
|
||||
|
||||
1. The web citation/evidence panel shows which source chunks, pages and exact
|
||||
evidence text were selected for the answer.
|
||||
2. **Grafana -> Explore -> Tempo** shows which pipeline stages ran, their
|
||||
nesting and timing, the final decision/reason, provider failures and the
|
||||
persisted trace ID.
|
||||
3. PostgreSQL table `rag_retrieval_trace` is the durable audit record. It stores
|
||||
the query, resolved drug, decision/reason, selected citations/evidence,
|
||||
correlation ID and OpenTelemetry trace ID, so a returned `trace_id` can be
|
||||
joined to its Tempo trace.
|
||||
|
||||
This is provenance and execution tracing, not model chain-of-thought logging.
|
||||
Full prompts/responses, hidden reasoning, every rejected retrieval candidate
|
||||
and every ranking score are deliberately not stored today. If deeper debugging
|
||||
is needed, add bounded audit fields rather than putting sensitive prompt or
|
||||
patient content into metric labels or span names.
|
||||
|
||||
Start the local stack from the repository root:
|
||||
|
||||
```powershell
|
||||
docker compose -f infra\docker\docker-compose.yml up -d prometheus tempo otel-collector grafana
|
||||
```
|
||||
|
||||
Local endpoints:
|
||||
|
||||
| Service | Address | Use |
|
||||
|---|---|---|
|
||||
| Grafana | `http://localhost:3002` | Dashboards and Explore |
|
||||
| Prometheus | `http://localhost:9090` | Raw targets, PromQL and metrics |
|
||||
| Tempo | `http://localhost:3200` | Trace backend; normally queried through Grafana |
|
||||
| ai-service metrics | `http://localhost:8079/metrics` | Raw OpenMetrics output when ai-service runs on port 8079 |
|
||||
|
||||
For the existing EC2 Compose deployment, the optional overlay is
|
||||
`infra/docker/docker-compose.observability.yml`. It leaves
|
||||
`docker-compose.prod.yml` unchanged. A deployment, when explicitly approved,
|
||||
uses both files:
|
||||
|
||||
```powershell
|
||||
docker compose `
|
||||
-f infra/docker/docker-compose.prod.yml `
|
||||
-f infra/docker/docker-compose.observability.yml `
|
||||
up -d
|
||||
```
|
||||
|
||||
Only Grafana is mapped to the EC2 host (`3002:3000`) by the production overlay;
|
||||
Prometheus and Tempo stay on the internal Compose network. The EC2 security
|
||||
group does not expose port 3002 publicly. View Grafana through an SSH tunnel:
|
||||
|
||||
```powershell
|
||||
ssh -L 3002:127.0.0.1:3002 <ssh-user>@52.0.158.61
|
||||
```
|
||||
|
||||
Keep that session open and visit `http://localhost:3002`. Set
|
||||
`GRAFANA_ADMIN_USER` and `GRAFANA_ADMIN_PASSWORD` in the production environment
|
||||
before deployment; do not use the fallback password in production. Prometheus
|
||||
metrics are available in **Grafana -> Explore -> Prometheus**. To investigate a
|
||||
slow request, open the request-latency panel, follow its exemplar/trace link, or
|
||||
paste the returned `X-Trace-ID` into **Explore -> Tempo**.
|
||||
|
||||
## Production
|
||||
|
||||
Live at [realvuxbaro.me](https://realvuxbaro.me): a single EC2 `t3.large`
|
||||
|
||||
Reference in New Issue
Block a user