Fix migration workflow: upload as artifact instead of scp to practice EC2

2026-08-13 11:14:25 +07:00
parent 7ebbe1f309
commit a4819b8653
51 changed files with 6830 additions and 8 deletions
@@ -0,0 +1,85 @@
+# Documentation plan
+
+How the `docs/` set in this directory was produced, what was inspected, what
+was executed, and what is deliberately left unverified. Kept so a later reader
+can judge how much weight each page carries.
+
+## Source-of-truth order
+
+1. Production code (`apps/ai-service/`, `apps/web/`, `ingestion/`, `packages/`)
+2. Runtime configuration (`apps/ai-service/config.py`, `.env`, Helm values,
+   Compose files)
+3. Tests (`apps/ai-service/tests/`, `ingestion/tests/`)
+4. Deployment manifests (`infra/`, `.github/workflows/`)
+5. Database migrations (`apps/ai-service/migrations/`)
+6. CI/CD (`.github/workflows/deploy.yml`)
+7. Scripts (`ingestion/ingestion/cli.py`, `ingestion/ingestion/load/run.py`,
+   `apps/ai-service/scripts/`)
+8. Pre-existing documentation — read for context, **never** used as evidence
+   that the system behaves a certain way
+
+Where a pre-existing document and the code disagree, the code wins and the
+disagreement is recorded in [26-known-limitations.md](26-known-limitations.md).
+
+## State vocabulary used throughout
+
+| Label | Meaning |
+|---|---|
+| **Implemented** | Code exists and is reachable from a runtime entrypoint |
+| **Partially implemented** | Reachable, but with a named gap |
+| **Configured, not verified** | Config/manifest exists; no evidence it runs |
+| **Test-only** | Code exists and is tested but no runtime caller reaches it |
+| **Planned / TODO** | Explicit TODO, placeholder, or `NotImplementedError` |
+| **Not found** | Searched for, does not exist |
+| **Unable to verify** | Would require access this session did not have |
+
+## Phases
+
+| Phase | Scope | Output |
+|---|---|---|
+| 1 | Repository inventory: `git ls-files`, per-file line counts, entrypoint identification | [01-repository-structure.md](01-repository-structure.md) |
+| 2 | Runtime architecture: `main.py`, `bootstrap.py`, `config.py`, `routers/rag.py`, import-graph checks for dead code | [00](00-project-overview.md), [02](02-system-architecture.md), [03](03-data-flow.md) |
+| 3 | Ingestion: `ingestion/ingestion/**`, CLI subcommands, gates, artifacts on disk | [04](04-ingestion-pipeline.md), [05](05-document-parsing.md), [06](06-document-model-and-chunking.md), [07](07-indexing-and-storage.md) |
+| 4 | RAG: understanding, retrieval, orchestration, generation, grounding, prompts | [08](08-query-understanding.md), [09](09-retrieval-pipeline.md), [10](10-rag-orchestration.md), [11](11-generation-and-grounding.md) |
+| 5 | API + frontend: FastAPI routes, Next.js BFF routes, middleware, shared DTOs | [12](12-api-architecture.md), [13](13-frontend-architecture.md) |
+| 6 | Infrastructure: Compose, Caddy, Helm, ArgoCD, CI, config/secret surface, security | [14](14-data-stores.md), [15](15-configuration.md), [16-security.md](16-security.md), [17](17-observability.md), [20](20-deployment.md), [21](21-kubernetes-and-argocd.md), [22](22-ci-cd.md) |
+| 7 | Testing + evaluation: both suites executed, eval datasets and metric code read | [18](18-testing.md), [19](19-rag-evaluation.md) |
+| 8 | Operations: local dev, production runbook, troubleshooting | [23](23-local-development.md), [24](24-production-operations.md), [25](25-troubleshooting.md) |
+| 9 | Consistency review: gaps, debt, code-derived roadmap, glossary | [26](26-known-limitations.md), [27](27-technical-debt.md), [28](28-roadmap-from-code.md), [29](29-glossary.md) |
+
+## Verification actually executed
+
+| Command | Result |
+|---|---|
+| `cd ingestion && python -m pytest tests -q` | 277 passed, 12 skipped (32.9s) |
+| `cd apps/ai-service && python -m pytest tests -q` | **Collection error** — `tests/test_api.py` imports `main`, which builds the runtime at import time and tries to reach Qdrant |
+| `cd apps/ai-service && EMBEDDING_PROVIDER=disabled python -m pytest tests -q` | 278 passed, 6 skipped (2.6s) |
+| Corpus census over `ingestion/data/processed/chunks.jsonl` | 15,100 chunks; 14,949 `prose` + 151 `block_descriptor`; 684 distinct `drug_id`; 19 distinct `section_key`; all `schema_version=4` |
+| Census over `ingestion/data/verified/drug_entities.json` | 684 entities, 10,164 aliases |
+| Line count over `ingestion/data/processed/monographs.jsonl` | 684 monographs |
+| Import-graph grep for every `rag/` module | Identified three test-only modules (see [27-technical-debt.md](27-technical-debt.md)) |
+
+## Not verified in this pass
+
+- Live behaviour of <https://realvuxbaro.me> (no request was sent to production).
+- Contents of `apps/ai-service/.env.prod` — gitignored, lives on the EC2 host.
+  Every production-only configuration claim is marked accordingly.
+- Qdrant/PostgreSQL round-trips: `tests/test_live_datastores.py` is gated behind
+  `RUN_INTEGRATION=1` and was not run (no local datastores).
+- Any AWS Bedrock call (costs money on a personal account).
+- Helm chart rendering and the ArgoCD `Application` manifests: never applied to
+  a cluster from this repository.
+- Frontend behaviour: there is no frontend test suite to run.
+
+## Pre-existing documents kept, not rewritten
+
+These predate this set, record what was known on their date, and are kept for
+their reasoning. They are **not** current-state references:
+
+`architecture.md`, `progress-log.md`, `v1-delivery-plan.md`,
+`rag-rebuild-plan.md`, `current-rag-pipeline-audit.md`,
+`answer-experience-implementation-plan.md`,
+`condition-to-drug-audit-and-design.md`, `full-coverage-parsing-plan.md`,
+`document-profile.md`, `pdf-parsing-outlier-catalog.md`,
+`verification-strategy.md`, `pipeline-tu-pdf-den-chatbot-production.md`,
+and `adr/0001`–`adr/0008`.