Fix migration workflow: upload as artifact instead of scp to practice EC2

docs/README.md

@@ -0,0 +1,166 @@
+# Documentation
+
+Reverse-engineered from the code in this repository. Every claim here traces to
+a file, a command, or an artifact on disk — see
+[DOCUMENTATION_PLAN.md](DOCUMENTATION_PLAN.md) for the method and for what was
+not verified.
+
+## What this system is
+
+A Vietnamese-language question-answering system over the **Dược thư Quốc gia
+Việt Nam 2018** (Vietnamese National Drug Formulary), for doctors and
+pharmacists. A user asks a drug question in Vietnamese; the system resolves what
+was asked, retrieves the exact monograph section from a vector store, has an LLM
+restate it, verifies that restatement against the retrieved text, and returns it
+with printed-page citations — or refuses.
+
+Two things distinguish it from a generic RAG app, and both are enforced in code:
+
+- **Retrieval decides what is true; generation only decides how it reads.** A
+  generated answer is discarded unless every number in it appears verbatim in
+  the specific evidence block it cites (`rag/grounding.py`) *and* a second LLM
+  pass confirms the cited block actually says it (`rag/answer.py`).
+- **Tables and formulas are quarantined, not linearised.** Content whose numbers
+  could not be reliably reconstructed from the PDF is never embedded as prose
+  and never restated; it is surfaced as "check the source page".
+
+Scope boundary: the corpus is **Part 2 monographs only** (printed pages
+99–1496). Part 1 general chapters and Part 3 appendices are not ingested.
+
+## Architecture at a glance
+
+```mermaid
+flowchart LR
+    U[Clinician<br/>browser]
+    CADDY[Caddy 2<br/>TLS + reverse proxy]
+    WEB["web — Next.js 14<br/>chat UI + BFF routes<br/>+ in-memory rate limit"]
+    AI["ai-service — FastAPI<br/>RagAgent orchestrator"]
+    QD[("Qdrant<br/>duocthu_v1<br/>15,100 points")]
+    PG[("PostgreSQL 16<br/>traces · turns · feedback")]
+    BR["AWS Bedrock<br/>Cohere embed-v4 · Cohere rerank<br/>Converse generation"]
+    ING["ingestion — offline batch<br/>PDF → chunks → vectors"]
+    PDF[/"duoc-thu-quoc-gia-viet-nam-2018.pdf"/]
+
+    U --> CADDY --> WEB --> AI
+    AI --> QD
+    AI --> PG
+    AI --> BR
+    PDF --> ING --> QD
+    ING --> BR
+```
+
+The `api-gateway`, `auth-service`, `user-service` and `chat-service` directories
+in `apps/` contain **only** a `README.md` and a `package.json`. There is no
+gateway, no authentication and no chat-service in the request path; `web` calls
+`ai-service` directly. See [02-system-architecture.md](02-system-architecture.md).
+
+## Main technology stack
+
+| Layer | Technology | Evidence |
+|---|---|---|
+| Frontend | Next.js 14 (App Router), React 18, Tailwind, framer-motion | `apps/web/package.json` |
+| Backend | Python 3.12, FastAPI, Pydantic Settings, uvicorn | `apps/ai-service/pyproject.toml`, `Dockerfile` |
+| Vector store | Qdrant (cosine, 1024-d) | `adapters/qdrant.py`, `ingestion/load/` |
+| Relational | PostgreSQL 16 (`psycopg` 3) | `adapters/postgres.py`, `migrations/` |
+| Embedding | `cohere.embed-v4:0` on AWS Bedrock | `adapters/embedding.py`, `ingestion/embed/bedrock_cohere.py` |
+| Generation | Bedrock Converse API (model id is config) | `adapters/bedrock_converse.py` |
+| Rerank | `cohere.rerank-v3-5:0` on Bedrock | `adapters/bedrock_converse.py` |
+| PDF parsing | PyMuPDF (`fitz`), pdfplumber for tables only | `ingestion/extract/`, `ingestion/tables/` |
+| Observability | Prometheus, OpenTelemetry → OTel Collector → Tempo, Grafana | `rag/telemetry.py`, `infra/docker/` |
+| Runtime | Docker Compose on a single EC2 host, Caddy for TLS | `infra/docker/docker-compose.prod.yml` |
+| Monorepo | pnpm workspaces + Turborepo (JS side only) | `pnpm-workspace.yaml`, `turbo.json` |
+
+No RAG framework is used. There is no LangChain and no LlamaIndex anywhere in
+the dependency set — the orchestration is hand-written in `rag/agent.py`.
+
+## Core runtime services
+
+| Service | Language | Entrypoint | Port |
+|---|---|---|---|
+| `ai-service` | Python | `apps/ai-service/main.py` → `app` | 8000 |
+| `web` | TypeScript | `apps/web/app/` (Next.js) | 3000 |
+| `caddy` | — | `infra/docker/Caddyfile` | 80/443 |
+| `ingestion` | Python | `python -m ingestion.cli`, `python -m ingestion.load.run` | offline, no port |
+
+## Main data stores
+
+| Store | Holds | Live-path role |
+|---|---|---|
+| Qdrant `duocthu_v1` | 15,100 chunk points + payload | Every retrieval |
+| Qdrant `duocthu_v1__manifest` | One point: corpus sha, model id, dimensions | Startup gate (`bootstrap.py`) |
+| PostgreSQL | `rag_retrieval_trace`, `rag_conversation_turn`, `rag_answer_feedback` | Traces + multi-turn history; both fail-open |
+| Local disk | `chunks.jsonl`, `monographs.jsonl`, embedding cache | Offline pipeline only |
+
+Redis appears in `infra/docker/docker-compose.yml` (local dev) and in the
+pre-existing architecture document. **Nothing in the codebase imports a Redis
+client.** It is not deployed in production and not read or written by any code.
+
+## Main pipelines
+
+1. **Ingestion (offline)** — PDF → spans → monographs → chunks → embeddings →
+   Qdrant. Seven CLI subcommands plus a separate embed/load entrypoint. Has
+   already been run; re-running the embed step costs real Bedrock spend.
+   → [04-ingestion-pipeline.md](04-ingestion-pipeline.md)
+2. **Query (live)** — HTTP → understanding LLM call → deterministic route →
+   Qdrant retrieval → generation LLM call → deterministic grounding →
+   entailment LLM call → citations → response.
+   → [10-rag-orchestration.md](10-rag-orchestration.md)
+
+## Documentation map
+
+**Start here, in order:**
+
+1. [00-project-overview.md](00-project-overview.md) — problem, users, boundaries
+2. [02-system-architecture.md](02-system-architecture.md) — components and what is *not* built
+3. [03-data-flow.md](03-data-flow.md) — the two end-to-end flows in one page
+
+**For AI/RAG engineers:**
+[08-query-understanding.md](08-query-understanding.md) →
+[09-retrieval-pipeline.md](09-retrieval-pipeline.md) →
+[10-rag-orchestration.md](10-rag-orchestration.md) →
+[11-generation-and-grounding.md](11-generation-and-grounding.md) →
+[19-rag-evaluation.md](19-rag-evaluation.md).
+For the corpus itself: [04](04-ingestion-pipeline.md) →
+[05](05-document-parsing.md) → [06](06-document-model-and-chunking.md) →
+[07](07-indexing-and-storage.md).
+
+**For backend engineers:**
+[12-api-architecture.md](12-api-architecture.md) →
+[14-data-stores.md](14-data-stores.md) →
+[15-configuration.md](15-configuration.md) →
+[18-testing.md](18-testing.md) →
+[23-local-development.md](23-local-development.md).
+
+**For frontend engineers:**
+[13-frontend-architecture.md](13-frontend-architecture.md) →
+[12-api-architecture.md](12-api-architecture.md) (the response contract) →
+[16-security.md](16-security.md) (rate limiting lives in the frontend today).
+
+**For DevOps/SRE:**
+[20-deployment.md](20-deployment.md) →
+[22-ci-cd.md](22-ci-cd.md) →
+[17-observability.md](17-observability.md) →
+[24-production-operations.md](24-production-operations.md) →
+[25-troubleshooting.md](25-troubleshooting.md) →
+[21-kubernetes-and-argocd.md](21-kubernetes-and-argocd.md) (unapplied target state).
+
+**For QA:**
+[18-testing.md](18-testing.md) →
+[19-rag-evaluation.md](19-rag-evaluation.md) →
+[26-known-limitations.md](26-known-limitations.md).
+
+**Before planning work:**
+[26-known-limitations.md](26-known-limitations.md) →
+[27-technical-debt.md](27-technical-debt.md) →
+[28-roadmap-from-code.md](28-roadmap-from-code.md).
+
+Terms: [29-glossary.md](29-glossary.md).
+
+## Pre-existing documents in this directory
+
+`architecture.md`, `progress-log.md`, `pdf-parsing-outlier-catalog.md`,
+`document-profile.md`, `verification-strategy.md`, the dated plan/audit files,
+and `adr/0001`–`adr/0008` predate this set. They are kept for their reasoning
+and their empirical measurements. Where they describe current behaviour, they
+have drifted in places — the drift is listed in
+[26-known-limitations.md](26-known-limitations.md#documentationcode-discrepancies).