Add read-only production runtime audit

docs-legacy/01-repository-structure.md

@@ -0,0 +1,185 @@
+# 01 — Repository structure
+
+A pnpm/Turborepo monorepo for the JavaScript side, with two independent Python
+projects (`apps/ai-service`, `ingestion`) that are **not** part of the pnpm
+workspace and are not built by Turbo.
+
+## Top level
+
+| Path | Purpose | Runtime relevance |
+|---|---|---|
+| `apps/` | Deployable applications | `ai-service` and `web` only |
+| `packages/` | Shared TypeScript packages | Build-time for `web` |
+| `ingestion/` | Offline PDF → vector pipeline + its data | Never in the request path |
+| `infra/` | Docker, Helm, ArgoCD, Terraform scaffold, AWS IAM policies | Deployment |
+| `docs/` | This documentation set + pre-existing design records | None |
+| `coordination/` | Hand-off notes between two AI agents working the repo | None |
+| `Golden Dataset/` | Five hand-labelled CSV evaluation sets | Manual QA only — no runner reads them |
+| `.github/workflows/` | One workflow: `deploy.yml` | CI/CD |
+| `output/presentations/` | Untracked scratch output | None |
+
+Untracked noise at the repo root (`.codex-*.log`, `.codex-*.png`, `tmp/`,
+`.venv_docling_test/`, `.next/`) is working residue, not part of the system.
+
+## `apps/ai-service/` — the RAG service
+
+Flat module layout, **not** an installable package (see the `Dockerfile`
+comment: setuptools rejects the multiple top-level packages).
+
+| Path | Purpose | Key files |
+|---|---|---|
+| `main.py` | FastAPI app factory + module-level `app`. Builds the whole runtime at **import time**. | `create_app`, `/health`, `/ready`, `/metrics` |
+| `bootstrap.py` | Composition root. Decides which adapters exist and wires the object graph. | `build_runtime` |
+| `config.py` | Pydantic `Settings`; the single definition of every env var | `Settings`, `get_settings` |
+| `migrate.py` | Applies `migrations/*.sql` in sorted order | — |
+| `routers/rag.py` | The only router: `/v1/rag/query`, `/suggest`, `/feedback` | request/response models |
+| `rag/` | Pure domain — imports no SDK | see below |
+| `adapters/` | The only modules that import `qdrant_client`, `psycopg`, `boto3`, `prometheus_client` | `qdrant.py`, `postgres.py`, `embedding.py`, `bedrock_converse.py`, `bedrock_claude.py`, `prometheus.py` |
+| `migrations/` | Four idempotent `CREATE TABLE IF NOT EXISTS` / `ALTER` scripts | — |
+| `evals/` | Three JSONL eval sets + `drug_aliases.json` | [19](19-rag-evaluation.md) |
+| `scripts/run_manual_battery.py` | HTTP recorder for the 60-case production battery | [19](19-rag-evaluation.md) |
+| `tests/` | 26 test modules, 278 tests | [18](18-testing.md) |
+
+### `apps/ai-service/rag/` — domain modules
+
+| Module | Lines | Role | Reached at runtime? |
+|---|---|---|---|
+| `agent.py` | 776 | The orchestrator: `RagAgent.handle()` routes a turn | Yes — the live path |
+| `answer.py` | 1171 | Generation, grounding, entailment, citation assembly | Yes |
+| `understanding.py` | 1030 | LLM query understanding → `QueryFrame` | Yes |
+| `service.py` | 741 | `RetrievalService` — every retrieval strategy | Yes |
+| `prompt.py` | 485 | All three system prompts + JSON schemas | Yes |
+| `clinical.py` | 415 | `PatientContext`, `ConditionQuery`, candidate assessment types | Yes |
+| `routing.py` | 333 | `CatalogDrugResolver` (fuzzy) + `QueryRoutingService` (legacy path) | Partly — resolver yes, `QueryRoutingService.retrieve` only when no generator |
+| `instrumentation.py` | 268 | Subclass wrappers adding spans/metrics | Yes |
+| `telemetry.py` | 217 | Correlation ids, OTel spans, stage timing | Yes |
+| `sections.py` | 210 | Keyword → `section_key` resolver + book section order | Yes |
+| `grounding.py` | 180 | Per-citation number/citation verification | Yes |
+| `condition_evaluation.py` | 111 | Deterministic condition→drug metrics | Test-only |
+| `models.py` | 97 | `Evidence`, `RetrievalResult`, `SourceRef`, enums | Yes |
+| `metrics.py` | 97 | Metric-name constants + `Metrics` protocol | Yes |
+| `in_memory.py` | 97 | In-memory retriever/parent store | Test + `run_eval` only |
+| `evaluation.py` | 93 | Retrieval eval case/outcome types | Test + `run_eval` only |
+| `run_eval.py` | 97 | Offline retrieval eval CLI | Manual only |
+| `ports.py` | 78 | Protocols + the three provider-unavailable exceptions | Yes |
+| `policy.py` | 71 | Server-derived subject scope (non-human guard) | Yes |
+| `budget.py` | 64 | Per-request wall-clock + call budget | Yes |
+| `manifest.py` | 62 | Startup corpus/model manifest check | Yes |
+| `expansion.py` | 62 | Sibling-chunk expansion | **Test-only — no runtime caller** |
+| `context.py` | 61 | Token-budgeted evidence packing | Yes (`service.py::retrieve_framed`) |
+| `fusion.py` | 55 | Reciprocal-rank fusion | **Test-only — no runtime caller** |
+| `calculators.py` | 24 | DuBois body-surface-area | **Test-only — no runtime caller** |
+| `artifacts.py` | 89 | Loads `drug_entities.json` and offline JSONL artifacts | `load_aliases` yes; the rest `run_eval` only |
+| `text.py` | 23 | `normalize_name` (casefold + strip diacritics) | Yes |
+
+## `apps/web/` — Next.js 14 chat UI
+
+| Path | Purpose |
+|---|---|
+| `app/page.tsx` | Chat page shell |
+| `app/tra-cuu/page.tsx` | "Tra cứu" (lookup) page |
+| `app/_components/ChatPanel.tsx` | Chat state, fetch, 65s client timeout, starter questions |
+| `app/_components/Composer.tsx` | Input + autocomplete |
+| `app/_components/EvidencePanel.tsx` | Citation cards |
+| `app/_components/AnswerFeedback.tsx` | Thumbs up/down → `/api/feedback` |
+| `app/_components/Sidebar.tsx`, `NavTabs.tsx` | Navigation |
+| `app/api/chat/route.ts` | **BFF**: calls `ai-service` `/v1/rag/query`, maps reason codes to Vietnamese |
+| `app/api/suggest/route.ts` | Proxies `/v1/rag/suggest` |
+| `app/api/feedback/route.ts` | Proxies `/v1/rag/feedback` |
+| `app/api/pdf/route.ts` | Streams the 37MB source PDF from disk |
+| `middleware.ts` | In-memory IP rate limiting on `/api/*` |
+
+## `packages/`
+
+| Package | Contents | Consumed by |
+|---|---|---|
+| `shared-types` | `dto/chat.ts` (`Citation`, `ChatMessage`, `AnswerBlock`, `AnswerPlan`, …), `dto/session.ts` | `web`, `api-client`, `ui` |
+| `ui` | `ChatBubble`, `CitationCard`, `CitationBeamOverlay`, `DisclaimerBanner`, `ThemeContext`, shadcn-style primitives | `web` |
+| `api-client` | `sendChatMessage`, `getDrugSuggestions`, `mockFixtures` | **Declared as a `web` dependency but the live chat path calls `fetch("/api/chat")` directly** |
+| `config` | `tsconfig-base.json`, empty `eslint-preset/` | build config |
+
+## `ingestion/`
+
+| Path | Purpose |
+|---|---|
+| `ingestion/cli.py` | `run`, `validate`, `detect-tables`, `coverage`, `residual-ink`, `chunk-ready`, `chunk` (+ two `NotImplementedError` stubs) |
+| `ingestion/extract/` | PyMuPDF span extraction, glyph/reading-order scan, printed-page map, vector-outlined text repair, formula regions |
+| `ingestion/normalize/` | Glyph substitution, text-flow joining |
+| `ingestion/segment/` | Monograph/section detection, assembly, ATC parsing, section vocabulary |
+| `ingestion/tables/` | Table region detection + shape classification |
+| `ingestion/chunk/` | Section → chunk packing, sentence splitting, token counting |
+| `ingestion/embed/` | Provider adapters (Cohere/Titan/local BGE-M3), disk cache, registry, probe, benchmark |
+| `ingestion/load/` | Qdrant vector store, chunk loader, corpus manifest, `run.py` entrypoint |
+| `ingestion/entities/` | Drug entity catalog build |
+| `ingestion/validation/` | Named acceptance gates, back-index recall/precision, residual-ink census |
+| `ingestion/data/raw/` | The 37MB source PDF (committed) |
+| `ingestion/data/processed/` | `monographs.jsonl` (31MB), `chunks.jsonl` (30MB), `coverage_ledger.json` (52MB), `table_regions.json`, `residual_ink.json`, `embeddings/` cache |
+| `ingestion/data/verified/` | `drug_entities.json` (684 entities / 10,164 aliases), `formula_regions_2d.json`, `outlined_text_transcriptions.json` |
+| `ingestion/data/reconstruction/crops/` | PNG crops of quarantined tables/formulas |
+| `tests/` | 24 test modules, 277 tests |
+
+## `infra/`
+
+| Path | State |
+|---|---|
+| `docker/docker-compose.prod.yml` | **Live** — the production topology |
+| `docker/docker-compose.observability.yml` | **Live** — overlay applied by the deploy workflow |
+| `docker/docker-compose.yml` | Local dev infra (postgres, qdrant, redis, prometheus, grafana, tempo, otel-collector); app services are commented out |
+| `docker/Caddyfile` | **Live** — TLS + `/grafana/*` subpath |
+| `docker/{prometheus,grafana,tempo,otel}/` | Scrape config, provisioned datasources + one dashboard, Tempo config, collector pipeline |
+| `helm/medical-chatbot/` | Complete chart (ai-service, web, postgres, qdrant, observability, ingress, secret, ServiceMonitor). **Never applied** |
+| `argocd/applications/{dev,staging,prod}/app.yaml` | Three `Application` CRs with three `TODO` placeholders each. **Never applied** |
+| `k8s/base/*`, `k8s/overlays/*` | Empty directories (`.gitkeep` only) |
+| `terraform/` | Empty module/env directories (`.gitkeep` only) + a README |
+| `ci/github-actions/README.md` | Placeholder describing five workflows that **do not exist** |
+| `aws/iam/*.json` | Two IAM policy documents for Bedrock model access |
+
+## Module dependency direction
+
+```mermaid
+flowchart TD
+    subgraph aisvc["apps/ai-service"]
+        MAIN[main.py]
+        BOOT[bootstrap.py]
+        ROUTER[routers/rag.py]
+        CFG[config.py]
+        subgraph domain["rag/ — no SDK imports"]
+            AGENT[agent.py]
+            ANSWER[answer.py]
+            UND[understanding.py]
+            SVC[service.py]
+            GRND[grounding.py]
+            PROMPT[prompt.py]
+            PORTS[ports.py]
+        end
+        subgraph ad["adapters/ — SDK edge"]
+            QA[qdrant.py]
+            PGA[postgres.py]
+            EMB[embedding.py]
+            GEN[bedrock_converse.py]
+            PROM[prometheus.py]
+        end
+    end
+
+    MAIN --> BOOT
+    MAIN --> ROUTER
+    BOOT --> CFG
+    BOOT --> ad
+    BOOT --> domain
+    ROUTER --> ANSWER
+    AGENT --> UND
+    AGENT --> SVC
+    AGENT --> ANSWER
+    ANSWER --> GRND
+    ANSWER --> PROMPT
+    SVC --> PORTS
+    ad -. implements .-> PORTS
+```
+
+The direction is enforced by convention and visible in the imports: no file
+under `rag/` imports `qdrant_client`, `boto3`, `psycopg` or `prometheus_client`.
+`adapters/qdrant.py` imports *from* `rag.models`/`rag.text`/`rag.sections`, not
+the other way round.
+
+`ingestion/` and `apps/ai-service/` share **no** code. The Cohere request body
+is duplicated in both on purpose (`adapters/embedding.py` docstring).