Stop mislabeling the Compose box as production in read-only audits

coordination/ARGOCD_PRODUCTION_MIGRATION_STATE_2026-08-17.md

@@ -461,16 +461,95 @@ certificate for that name — the SNI handshake fails with an internal error, an
 the certificate it does hold is apex-only. Decide whether to point `www` at k3s
 and add it to `tls`, or drop the record.
 
+## Grafana exposure closed (2026-08-17, post-cutover)
+
+The cutover moved production onto the cluster whose Grafana ran
+`GF_AUTH_ANONYMOUS_ORG_ROLE=Admin` with `GF_AUTH_DISABLE_LOGIN_FORM=true` —
+defensible on a throwaway rehearsal box, not on a public production hostname.
+Confirmed live before the fix: `/grafana/api/org` and `/grafana/api/datasources`
+both answered HTTP 200 with no credentials, listing internal service URLs.
+
+Fixed at `39b1159` plus one out-of-band step:
+
+- Anonymous access kept, but demoted to `Viewer`, so dashboards stay open for a
+  demo while the datasource and dashboard write APIs are refused (verified:
+  `POST /api/datasources` → 403).
+- `GF_AUTH_DISABLE_LOGIN_FORM` is now hardcoded `false`. Combined with
+  anonymous Admin it had previously left no way to sign in as a real admin.
+- `ingress.host` is now `realvuxbaro.me`, with the rehearsal name moved to
+  `extraHosts`. Grafana builds `GF_SERVER_ROOT_URL` from `ingress.host`, so it
+  had continued advertising the rehearsal hostname after the cutover.
+- CI now asserts the rendered role is `Viewer`, the login form is not disabled,
+  and the root URL is the production hostname.
+
+**Setting the admin password through Helm was not enough.** Grafana persists
+its user table in SQLite on a PVC, so `GF_SECURITY_ADMIN_PASSWORD` did not
+overwrite the existing credential: after the rollout, `admin:change-me` still
+authenticated and the new password did not. It required
+`grafana cli admin reset-admin-password --password-from-stdin` inside the pod.
+Anyone rotating this password later must do the same — changing the Secret
+alone is silently ineffective.
+
+Final state verified: anonymous → `/api/admin/settings` 403; admin with the new
+password 200; `admin:change-me` 403; anonymous dashboard search 200; app 200.
+The password lives in the gitignored `.env.k3s-practice` as
+`GRAFANA_ADMIN_PASSWORD` and in the ArgoCD Application's inline values. It is
+not in Git.
+
+## Repository now describes what is actually live
+
+`values-practice.yaml` / `values-practice-data.yaml` became
+`values-production.yaml` / `values-production-data.yaml`, and
+`global.environment` is now `production` (confirmed in the running ConfigMap).
+The separate, never-deployed `values-prod.yaml` is deleted — two "prod" files
+beside a "practice" file that was the real one was the worst of both.
+
+Done in three commits so no sync ever referenced a missing path: add the new
+files (`527d3a8`), repoint both Applications and confirm Synced/Healthy, then
+delete the old ones (`6468b16`). The ai-service Pod rolled cleanly on the
+environment change and the site stayed up throughout.
+
+The immutable-tag guard moved onto the file that is actually live and now
+checks both directions: an empty tag must fail the render, and a supplied SHA
+must resolve the GHCR package rather than the chart's local development name.
+
+## Live chat driven directly against production
+
+Not a golden-set replay — unaccented text, a typo, missing dimensions and
+pronoun follow-ups within one conversation, through the real `/api/chat`:
+
+| Input | Behaviour |
+| --- | --- |
+| `lieu paracetamol` | `clarify` — asks adult or child, offers both as quick replies; answers with a citation once told `nguoi lon` |
+| `paracetamol cho tre em` → `be 3 tuoi` | `clarify` twice: age, then still demands weight. The paediatric gate requiring **both** is intact |
+| `metformim co tac dung phu gi` | Resolves the typo and answers metformin's GI adverse effects with a citation |
+| `amlodipin dung the nao` → `the con chong chi dinh thi sao` | Carries the referent across turns and answers the contraindication |
+| `gia thuoc paracetamol ... bao nhieu tien` | Abstains cleanly: the formulary does not carry prices |
+
+Cosmetic only: the paediatric follow-up echoes the user's unaccented text back
+inside an otherwise accented sentence ("Bé 3 tuoi nặng bao nhiêu kg?").
+
+## Final verified state
+
+`https://realvuxbaro.me/` 200 from `44.206.194.195` with a valid certificate;
+live query `answerable` in 14.3 s with a citation; Grafana anonymous read 200
+but admin 403, admin login 200; `readytochat.realvuxbaro.me` 200;
+`argocd.realvuxbaro.me` 200; and the Compose host still answers the apex with
+a valid certificate, so the rollback is a 60-second A-record revert.
+
+Temporary SSH allowances on `sg-018fc3cde8282f26d` were revoked; only the
+pre-existing `103.238.70.200/32` remains on port 22. Note that this
+workstation's SSH egress was `101.99.23.84`, so re-opening will be needed for
+future cluster access — and the three IP-reporting services disagree, so read
+the real one from the host's own `auth.log` rather than trusting any of them.
+
 ## Migration risks currently open
 
 1. `www.realvuxbaro.me` is broken (see above) and always was.
 2. Secrets, resource limits and failure recovery still need explicit rehearsal
    gates.
-3. Answer-content parity between environments is still unverified — decisions
-   and latency match, answer length does not always.
-4. The cluster still serves production from values files named `values-practice*`.
-   Rename once the acceptance window closes and the Compose host is retired;
-   `values-prod.yaml` is now unused by anything live.
+3. The Compose EC2 is still running as the rollback. Stop — never terminate —
+   once the acceptance window closes.
 3. Cluster-level inspection is currently blocked from this machine: the
    practice security group scopes port 6443 to the operator's own outbound IP,
    which has changed again, so `kubectl` hangs. ArgoCD's API and the public

coordination/CLAUDE_PLAN_CICD_SAFETY_2026-08-18.md

@@ -0,0 +1,204 @@
+> **Decision update, same day, later session:** owner confirmed intent to run
+> **one EC2 only** (k3s). This supersedes PR A step 2 below (reduce to
+> `workflow_dispatch`) — `deploy.yml`, `rollback.yml`, and the never-applied
+> `infra/argocd/applications/**` scaffold (wrong Application names, dead
+> `values-prod.yaml` reference, placeholder team-repo TODOs) were deleted
+> outright on branch `agent/retire-compose-cicd`, not disabled. D1/D2/D3/D4 are
+> closed by deletion rather than by fixing the probe. `docs/operations.md`
+> Deploy/Rollback sections rewritten to describe the actual k3s/ArgoCD path.
+> Compose EC2 itself (`52.0.158.61`) is untouched pending an explicit stop/
+> terminate decision (PR D) — do not stop or terminate it without that go-ahead.
+> PR B (Helm hygiene) and PR C (rollback runbook, now mostly covered in
+> `docs/operations.md`) remain open.
+
+# Plan — make the CI/CD path safe after the k3s cutover (2026-08-18)
+
+Written by Claude (Opus) for execution by another agent. Every claim below was
+verified on 2026-08-18 by the command shown next to it. **Re-verify before
+acting** — runtime state can change after this snapshot.
+
+## 1. The topology changed and two workflows never noticed
+
+This is the root cause of everything in this plan. The 2026-08-17 cutover moved
+`realvuxbaro.me` from the Compose EC2 to the k3s/ArgoCD cluster, but the
+Compose-era workflows still describe and probe the old world.
+
+| Fact | Verified by |
+| --- | --- |
+| `realvuxbaro.me` → `44.206.194.195` (k3s) | `nslookup realvuxbaro.me 8.8.8.8` |
+| `readytochat.realvuxbaro.me` → `44.206.194.195` — **same cluster, same release** | same |
+| Compose EC2 `52.0.158.61` is off DNS; rollback target only | `values-production.yaml` `ingress.host` + DNS above |
+| Compose is still at `df57e6b` | `git log df57e6b..52e8828 -- <deploy.yml paths>` returns empty |
+| App code at `df57e6b` == app code at `master` | the intervening commits touch only docs/CI/coordination |
+
+So: **the Compose rollback is currently valid**, and the window to fix this is
+now, before the first post-cutover app change lands.
+
+`values-production.yaml` sets `ingress.host: realvuxbaro.me` and is rendered by
+ArgoCD Application `medical-chatbot-app` — the same Application
+`.github/scripts/sync_practice_argocd.py` repoints (`APP_NAME =
+"medical-chatbot-app"`). **"Practice" and production are the same release.**
+
+## 2. Confirmed defects, most dangerous first
+
+### D1 — both Compose workflows verify the wrong machine (P0)
+
+`deploy.yml:88` and `rollback.yml:57` both end with:
+
+```
+docker run --rm --network docker_default curlimages/curl -sf \
+  -o /dev/null https://realvuxbaro.me/grafana/login
+```
+
+That hostname now resolves to **k3s**, not the box the workflow just rebuilt.
+The check passes by hitting a completely different server.
+
+This is worst in `rollback.yml`, whose whole purpose is to be trustworthy in an
+emergency: it prints `Rollback to <sha> verified healthy` on the strength of a
+probe that never touched the rolled-back box. A broken rollback would report
+success.
+
+### D2 — one bad commit poisons production *and* the rollback (P0)
+
+`deploy.yml` and `build-practice-images.yml` trigger on the **same four paths**:
+`apps/ai-service/**`, `apps/web/**`, `packages/**`,
+`ingestion/data/verified/drug_entities.json`.
+
+A single push to `master` therefore rolls production forward on k3s **and**
+rebuilds the Compose box that is supposed to be the known-good fallback. The
+"proven rollback" only holds while Compose stays on a good commit.
+
+### D3 — `build-practice-images.yml` header comment is now false (P1)
+
+> *"Does not touch deploy.yml or the production EC2/Compose stack — production
+> never pulls a GHCR image and isn't ArgoCD-managed at all, so this workflow
+> has no path to affect it."*
+
+Production **is** ArgoCD-managed and **does** pull GHCR images. This workflow is
+the production deploy pipeline. The comment invites exactly the push that breaks
+production, and its name reinforces the error.
+
+### D4 — a dead ArgoCD manifest sits in Git (P1)
+
+`infra/argocd/applications/prod/app.yaml` lists
+`valueFiles: [values.yaml, values-prod.yaml]`, but `values-prod.yaml` was
+deleted in `6468b16`. It also declares `syncPolicy: {}` with a comment claiming
+prod sync needs manual approval — the live Application is automated. Applying
+this file would fail or deploy something wrong. `dev/` and `staging/` under the
+same directory have not been checked and may share the defect.
+
+### D5 — the WIP Helm change shadows an existing variable (P2)
+
+Uncommitted in the main worktree. The chart **already** emits `AWS_REGION` in
+the ConfigMap from `aiService.config.awsRegion`
+(`templates/ai-service.yaml:15`, asserted in `helm-chart.yml`). The WIP diff
+adds a second `AWS_REGION` as a container `env:` entry — and in Kubernetes an
+explicit `env:` **overrides** `envFrom`, so enabling it would silently shadow
+the ConfigMap. Inert today only because `aws.region` defaults to `""`.
+
+The static-credentials half is a genuine gap and worth keeping.
+
+### D6 — `helm-chart.yml` asserts invariants but never diffs (P2)
+
+It renders both live releases and checks a strong list (Qwen, rerank, TLS
+secrets, Grafana role, `refute volumeClaimTemplates`). But nothing compares the
+render against the previous commit, so a chart change that alters anything
+*outside* that list reaches production silently.
+
+### D7 — production has no rollback workflow at all (P2)
+
+`rollback.yml` targets `secrets.EC2_HOST` — the Compose box. Nothing rolls back
+k3s. Real production rollback today is a manual Namecheap A-record revert
+(~60s TTL) or an ArgoCD revision/tag revert, neither written down.
+
+## 3. Execution plan
+
+### Ground rules
+
+- **Never push to `master`.** Every change goes through a PR. ArgoCD auto-syncs
+  `master` with `selfHeal` + `prune`; a merge touching `infra/helm/**` applies
+  to production with no human gate.
+- `ci.yml` runs on every PR (ruff, pytest, ingestion tests, web lint+build).
+  `helm-chart.yml` runs on PRs touching `infra/helm/**`. Both must be green.
+- Do not touch `git.vinmec.tech`, team ArgoCD/k3s, or team repos.
+- Do not start, stop, or terminate any EC2 instance without an explicit go.
+- Do not edit files in the other worktrees (`D:\VSF-DUOCTHU-codex-*`,
+  `D:\VSF-DUOCTHU-claude-gitops`); check `coordination/` for active claims and
+  file your own claim before starting.
+
+### PR A — workflow safety (D1, D2, D3, D4)
+
+Touches only `.github/**` and `infra/argocd/**`. ArgoCD renders
+`infra/helm/medical-chatbot`, so **this PR cannot alter production manifests**.
+Confirm that rather than assume it.
+
+1. **Fix the misdirected probes (D1).** In `deploy.yml` and `rollback.yml`,
+   make the Caddy/Grafana check target the box being deployed instead of a
+   public DNS name that now points elsewhere — e.g. resolve the hostname to the
+   local Caddy container so TLS and routing are still exercised.
+   **Verification gate:** prove the fixed check *fails* when Caddy is broken.
+   A probe that cannot fail is the defect being fixed, not a fix. (See the
+   `set -e` / `! grep` silent-pass traps already documented in
+   `helm-chart.yml`.)
+2. **Stop `deploy.yml` firing on push (D2).** Reduce it to `workflow_dispatch`
+   only. Compose then stays pinned at `df57e6b` — a stable rollback rather than
+   one that tracks `master`. State the trade-off in the commit message: the
+   fallback stops drifting, but also stops receiving fixes, so it goes stale as
+   production moves. That is acceptable for a time-boxed acceptance window and
+   is the subject of PR D.
+3. **Tell the truth in `build-practice-images.yml` (D3).** Replace the false
+   header comment. Renaming the workflow to name it as the production deploy
+   path is preferable — **trap:** it self-references in its own `paths:` filter
+   and the sync script path, so both must be updated together or the workflow
+   silently stops triggering.
+4. **Remove or correct `infra/argocd/applications/prod/app.yaml` (D4).** Check
+   `dev/` and `staging/` in the same directory for the same rot. Deleting is
+   fine if nothing applies them; verify that first.
+
+### PR B — Helm hygiene (D5, D6)
+
+Touches `infra/helm/**`, so merging **does** reach production. Highest care.
+
+5. **Drop the redundant `aws.region` block from the WIP diff (D5)**; keep the
+   static-credentials support, still defaulted off. Guard against a nil `aws`
+   key so a values file that omits it cannot break the render — a template
+   error here means ArgoCD cannot sync production at all.
+6. **Add a baseline render diff to `helm-chart.yml` (D6).** Render
+   `values-production.yaml` and `values-production-data.yaml` at the PR base and
+   at HEAD, then surface the diff in the job summary. The goal is that no chart
+   change ever reaches production without a human having seen exactly what it
+   does to the manifests.
+   **Verification gate:** the diff for PR B itself must be **empty** — the
+   static-credentials change is defaults-off and must render byte-identically.
+   If it is not empty, stop and explain why before merging.
+
+Note: `helm` is **not installed** on this workstation (`helm: command not found`
+in both bash and PowerShell). Either install it or rely on the CI render — but
+do not claim the render is unchanged without one of the two actually running it.
+
+### PR C — document the real rollback (D7)
+
+7. Write the production rollback runbook: revert the ArgoCD Application to the
+   previous image tag, and/or revert the `realvuxbaro.me` A record to
+   `52.0.158.61`. Note that the Namecheap edit is a **manual owner step** — a
+   harness permission classifier has blocked agent form input on that page
+   before, so the runbook must not assume an agent can do it.
+
+### PR D — decide the Compose lifecycle (owner call, not an agent call)
+
+8. Compose is a second `t3.large` running purely as a fallback. Once the
+   acceptance window closes, ArgoCD's own revision history covers rollback and
+   the instance is redundant. Surface the choice and the monthly cost; **do not
+   act on it without an explicit go.**
+
+### Out of scope
+
+`Feature-List-AI-Duoc-thu-V1.md`, `presentation/`, and `.claude/skills/` are
+untracked and match no workflow trigger path. Committing them is inert and can
+be a separate trivial commit — keep it out of PRs A–D.
+
+## 4. Order and why
+
+D1 first: an untrustworthy rollback is worse than no rollback, because it fails
+silently at the moment of maximum pressure. D2 next: it is the defect that would
+consume the rollback. Everything after is hardening.