duocthu/README.md

# Dược Thư RAG — Medical Chatbot Platform

Medical chatbot grounded in the Vietnamese National Drug Formulary
(Dược thư quốc gia Việt Nam 2018), built as a microservices monorepo.

Start with the [canonical documentation set](docs/README.md). It is a compact,
code-verified set covering architecture, PDF ingestion, RAG/chat, local development,
operations, API, configuration, evaluation and documentation governance.

The former numbered `00–29` material and historical plans are retained in
[`docs-legacy/`](docs-legacy/) as raw input only. Architecture decisions also remain
there until reviewed. See the canonical
[documentation policy](docs/documentation-policy.md) for source precedence.

> **Status** (2026-08-24): **live in production at
> [realvuxbaro.me](https://realvuxbaro.me)**, running on **k3s + ArgoCD**
> since the 2026-08-17 cutover — a real RAG chatbot over the whole
> formulary, not a scaffold. What exists and what does not:
>
> | Part | State |
> |---|---|
> | `ingestion/` | Done — 15,100 chunks embedded and loaded into Qdrant `duocthu_v1` |
> | `apps/ai-service/` | Done — live grounded RAG (retrieval, generation, grounding, abstention, citations, traces) |
> | `apps/web/` | Done — chat UI with citation/evidence panel, optional login |
> | `apps/auth-service`, `apps/api-gateway` | **Live since 2026-08-19** (real NestJS: register/login/JWT, `/auth/*` proxy) with a `/login` page. Verified 2026-08-24: `POST /api/auth/login` with the seed `demo` account returns a session. Login stays **optional** — anonymous chat is unaffected. Seed accounts (`admin`/`demo`) still use placeholder passwords and are **not** safe to expose publicly as-is |
> | `apps/user-service`, `apps/chat-service` | **Not built** — `README.md` + `package.json` only |
> | `apps/mobile/` | **Not built** — reserved |
> | `infra/docker/` | Compose is **stopped** (was production through 2026-08-17); still used for local dev (Postgres/Qdrant/observability), not for deploying anywhere |
> | `infra/helm/medical-chatbot/` | **This is production.** Two ArgoCD Applications, `medical-chatbot-app` and `medical-chatbot-data`, both tracked in `infra/argocd/applications/` |
> | Team's Gitea + ArgoCD (ADR 0002) | **Still not started**, not abandoned — the personal k3s/ArgoCD instance above is a separate, owner-operated cluster, not the team's shared infrastructure |
>
> `apps/web`'s chat/history/feedback/sections routes still call
> `apps/ai-service` **directly** — auth-service/api-gateway exist but chat
> traffic doesn't route through the gateway yet. Login is optional and only
> gates `/admin`; anonymous chat is unaffected. See
> [canonical architecture document](docs/architecture.md) for the implemented
> topology (not yet updated for this — verify against code, not that doc,
> until it is).

## Directory map

```
apps/
  web/            Next.js frontend (also hosts the BFF route the browser calls)
  ai-service/     Python FastAPI — RAG orchestration + AWS Bedrock calls
  api-gateway/    NestJS — public entry point, routes to internal services
  auth-service/   NestJS — signup/login/JWT
  user-service/   NestJS — profile/preferences
  chat-service/   NestJS — chat session + message history
  mobile/         reserved for a future mobile app
packages/
  shared-types/   TS DTOs shared across Node services + web
  api-client/     typed HTTP client for web
  ui/             shared React components
  config/         shared eslint/tsconfig presets
ingestion/        offline batch pipeline: PDF -> monographs -> chunks -> embeddings -> Qdrant
infra/            docker-compose, k8s/Helm, Terraform, CI
docs/             canonical project documentation
docs-legacy/      raw notes, historical plans and ADRs pending review
```

## Prerequisites

- Node.js + pnpm (JS workspace: `apps/web`, `apps/auth-service`,
  `apps/api-gateway`, `packages/*`; `apps/user-service`/`apps/chat-service`
  are still unbuilt placeholders)
- Python 3.11+ (`apps/ai-service`, `ingestion`)
- Docker (local Postgres + Qdrant via `infra/docker/docker-compose.yml`)
- AWS credentials with Bedrock invoke permission, for anything that generates
  an answer. Without them `ai-service` still starts, but every answer abstains
  rather than falling back to raw source text.

## Running it locally

```powershell
docker compose -f infra\docker\docker-compose.yml up -d postgres qdrant
cd apps\ai-service
python -m migrate
python -m uvicorn main:app --port 8079     # NOT --reload, see below
```

```powershell
pnpm install
pnpm --filter web dev                      # http://localhost:3000
```

`ai-service` needs a populated Qdrant collection to serve answers: it verifies
a `duocthu_v1__manifest` sidecar at startup and refuses to run against a corpus
whose sha/model/dimensions do not match. A fresh machine either restores a
Qdrant snapshot or re-runs `ingestion/` (the latter costs real Bedrock spend).

> **Prefer a plain restart over `uvicorn --reload` on Windows here.** The
> reloader has been observed serving the previous code after an edit on this
> project, which makes it hard to tell whether a change took effect.

Tests: `cd apps/ai-service && python -m pytest -q` — 230 pass. `test_api.py` and
`test_live_datastores.py` need Postgres and Qdrant actually running; skip them
with `--ignore` when the stack is down. `apps/web` has **no test setup at all**,
so a green suite says nothing about the frontend — drive it in a browser.

## Observability: Prometheus, Grafana and Tempo

The observability stack is provisioned in the repository and has been deployed
to the production EC2 instance since 2026-08-11.

- **Prometheus** scrapes `/metrics` from `ai-service`. It records request rate
  and latency, latency for each RAG stage, routing decisions and reasons,
  provider failures, trace-write failures and the existing domain counters.
- **Grafana** is the user interface for dashboards and metric queries. Its
  datasource and the Dược Thư dashboard are provisioned automatically.
- **Tempo** stores OpenTelemetry traces. A trace contains the receive,
  understanding, routing, retrieval, rerank/evidence, generation,
  grounding/entailment, persistence and response stages. Correlation and trace
  IDs follow the request from the Next.js BFF into FastAPI.
- **OpenTelemetry Collector** receives spans from `ai-service` and exports them
  to Tempo. Grafana exemplars link aggregate latency metrics to an individual
  Tempo trace.

For answer lineage, use the three views together:

1. The web citation/evidence panel shows which source chunks, pages and exact
   evidence text were selected for the answer.
2. **Grafana -> Explore -> Tempo** shows which pipeline stages ran, their
   nesting and timing, the final decision/reason, provider failures and the
   persisted trace ID.
3. PostgreSQL table `rag_retrieval_trace` is the durable audit record. It stores
   the query, resolved drug, decision/reason, selected citations/evidence,
   correlation ID and OpenTelemetry trace ID, so a returned `trace_id` can be
   joined to its Tempo trace.

This is provenance and execution tracing, not model chain-of-thought logging.
Full prompts/responses, hidden reasoning, every rejected retrieval candidate
and every ranking score are deliberately not stored today. If deeper debugging
is needed, add bounded audit fields rather than putting sensitive prompt or
patient content into metric labels or span names.

Start the local stack from the repository root:

```powershell
docker compose -f infra\docker\docker-compose.yml up -d prometheus tempo otel-collector grafana
```

Local endpoints:

| Service | Address | Use |
|---|---|---|
| Grafana | `http://localhost:3002` | Dashboards and Explore |
| Prometheus | `http://localhost:9090` | Raw targets, PromQL and metrics |
| Tempo | `http://localhost:3200` | Trace backend; normally queried through Grafana |
| ai-service metrics | `http://localhost:8079/metrics` | Raw OpenMetrics output when ai-service runs on port 8079 |

Grafana is served through the same k3s ingress as the app, at
`https://realvuxbaro.me/grafana/`. Anonymous access is on but demoted to
**Viewer** (dashboards load with no login; write actions need the admin
account). Prometheus has no public URL — use **Grafana -> Explore ->
Prometheus** for PromQL, or **Explore -> Tempo** with a returned
`X-Trace-ID` to investigate a slow request.

## Production

Live at [realvuxbaro.me](https://realvuxbaro.me), running on **k3s +
ArgoCD** (a personal, owner-operated cluster — not the team's shared
infrastructure) since the 2026-08-17 cutover. Two ArgoCD Applications:
`medical-chatbot-app` (ai-service, web, observability) and
`medical-chatbot-data` (PostgreSQL, Qdrant — a separate release so an app
redeploy or prune can never touch persistent data). Both are tracked in
`infra/argocd/applications/`; `infra/helm/medical-chatbot/` is the chart both
render from. Bedrock is reached through an IAM instance role — no long-lived
AWS keys on the box or in any manifest.

Two things deploy independently:

- **App code** (`apps/ai-service/**`, `apps/web/**`, `packages/**`) — a push
  to `master` triggers `.github/workflows/build-practice-images.yml`, which
  builds and pushes GHCR images tagged by commit SHA, then repoints
  `medical-chatbot-app` at the new tag. `ci.yml` runs in parallel and does
  **not** gate this — a red test suite does not block a deploy.
- **Chart/config** (`infra/helm/**`) — `helm-chart.yml` lints and asserts
  render invariants on the PR; once merged, ArgoCD's own `selfHeal` picks up
  the change automatically. No CI step applies it directly.

Rollback is `.github/workflows/rollback-k3s.yml` (`workflow_dispatch`,
`target_sha`) — see `docs/operations.md` for the full runbook, including its
current gaps (no automated rollback for a config-only change, and the
Grafana admin password still lives inline on the ArgoCD Application rather
than in a real Kubernetes Secret).

The former EC2 Docker Compose deployment (`i-039fc8f6102467a54`) is
**stopped**, not deleted — see `docs/operations.md` if it's ever needed as a
manual DNS fallback again, though its corpus/schema will drift further out
of date the longer it stays off.

The team's self-hosted **Gitea** + **ArgoCD** (per
`docs-legacy/adr/0002-argocd-gitops.md`) remains the longer-term target for
this project and is **still not started** — not abandoned, just a separate
decision from the personal-cluster cutover above. Until it is deliberately
started, the project stays on private GitHub, and the team's existing
`git.vinmec.tech/ai-team/gitops` repository is reference-only: never push
this project into it.