> **Decision update, same day, later session:** owner confirmed intent to run
> **one EC2 only** (k3s). This supersedes PR A step 2 below (reduce to
> `workflow_dispatch`) — `deploy.yml`, `rollback.yml`, and the never-applied
> `infra/argocd/applications/**` scaffold (wrong Application names, dead
> `values-prod.yaml` reference, placeholder team-repo TODOs) were deleted
> outright on branch `agent/retire-compose-cicd`, not disabled. D1/D2/D3/D4 are
> closed by deletion rather than by fixing the probe. `docs/operations.md`
> Deploy/Rollback sections rewritten to describe the actual k3s/ArgoCD path.
> PR #23 (both commits above) merged to `master` as `e804817`.
>
> **Update, same day, further into the session:** owner gave explicit go-ahead
> and Compose EC2 `i-039fc8f6102467a54` was **stopped** (not terminated) via
> `aws ec2 stop-instances`. Confirmed transition `running` → `stopping`. Root
> EBS still carries `DeleteOnTermination=true`, so it is intact and
> restartable, but it is no longer a live rollback target — `docs/operations.md`
> Rollback section now documents this as the current state, including the
> manual `start-instances` + compatibility-check steps needed before ever
> trusting it as a DNS fallback again. Nothing else about the box (AMI, EBS,
> tags, security group) was touched.
>
> PR B (Helm hygiene: drop the redundant `AWS_REGION` env block, add a
> baseline render-diff to `helm-chart.yml`) and the remaining half of PR C
> (an actual one-command k3s rollback script/workflow, not just the manual
> runbook now in `docs/operations.md`) remain open.

# Plan — make the CI/CD path safe after the k3s cutover (2026-08-18)

Written by Claude (Opus) for execution by another agent. Every claim below was
verified on 2026-08-18 by the command shown next to it. **Re-verify before
acting** — runtime state can change after this snapshot.

## 1. The topology changed and two workflows never noticed

This is the root cause of everything in this plan. The 2026-08-17 cutover moved
`realvuxbaro.me` from the Compose EC2 to the k3s/ArgoCD cluster, but the
Compose-era workflows still describe and probe the old world.

| Fact | Verified by |
| --- | --- |
| `realvuxbaro.me` → `44.206.194.195` (k3s) | `nslookup realvuxbaro.me 8.8.8.8` |
| `readytochat.realvuxbaro.me` → `44.206.194.195` — **same cluster, same release** | same |
| Compose EC2 `52.0.158.61` is off DNS; rollback target only | `values-production.yaml` `ingress.host` + DNS above |
| Compose is still at `df57e6b` | `git log df57e6b..52e8828 -- <deploy.yml paths>` returns empty |
| App code at `df57e6b` == app code at `master` | the intervening commits touch only docs/CI/coordination |

So: **the Compose rollback is currently valid**, and the window to fix this is
now, before the first post-cutover app change lands.

`values-production.yaml` sets `ingress.host: realvuxbaro.me` and is rendered by
ArgoCD Application `medical-chatbot-app` — the same Application
`.github/scripts/sync_practice_argocd.py` repoints (`APP_NAME =
"medical-chatbot-app"`). **"Practice" and production are the same release.**

## 2. Confirmed defects, most dangerous first

### D1 — both Compose workflows verify the wrong machine (P0)

`deploy.yml:88` and `rollback.yml:57` both end with:

```
docker run --rm --network docker_default curlimages/curl -sf \
  -o /dev/null https://realvuxbaro.me/grafana/login
```

That hostname now resolves to **k3s**, not the box the workflow just rebuilt.
The check passes by hitting a completely different server.

This is worst in `rollback.yml`, whose whole purpose is to be trustworthy in an
emergency: it prints `Rollback to <sha> verified healthy` on the strength of a
probe that never touched the rolled-back box. A broken rollback would report
success.

### D2 — one bad commit poisons production *and* the rollback (P0)

`deploy.yml` and `build-practice-images.yml` trigger on the **same four paths**:
`apps/ai-service/**`, `apps/web/**`, `packages/**`,
`ingestion/data/verified/drug_entities.json`.

A single push to `master` therefore rolls production forward on k3s **and**
rebuilds the Compose box that is supposed to be the known-good fallback. The
"proven rollback" only holds while Compose stays on a good commit.

### D3 — `build-practice-images.yml` header comment is now false (P1)

> *"Does not touch deploy.yml or the production EC2/Compose stack — production
> never pulls a GHCR image and isn't ArgoCD-managed at all, so this workflow
> has no path to affect it."*

Production **is** ArgoCD-managed and **does** pull GHCR images. This workflow is
the production deploy pipeline. The comment invites exactly the push that breaks
production, and its name reinforces the error.

### D4 — a dead ArgoCD manifest sits in Git (P1)

`infra/argocd/applications/prod/app.yaml` lists
`valueFiles: [values.yaml, values-prod.yaml]`, but `values-prod.yaml` was
deleted in `6468b16`. It also declares `syncPolicy: {}` with a comment claiming
prod sync needs manual approval — the live Application is automated. Applying
this file would fail or deploy something wrong. `dev/` and `staging/` under the
same directory have not been checked and may share the defect.

### D5 — the WIP Helm change shadows an existing variable (P2)

Uncommitted in the main worktree. The chart **already** emits `AWS_REGION` in
the ConfigMap from `aiService.config.awsRegion`
(`templates/ai-service.yaml:15`, asserted in `helm-chart.yml`). The WIP diff
adds a second `AWS_REGION` as a container `env:` entry — and in Kubernetes an
explicit `env:` **overrides** `envFrom`, so enabling it would silently shadow
the ConfigMap. Inert today only because `aws.region` defaults to `""`.

The static-credentials half is a genuine gap and worth keeping.

### D6 — `helm-chart.yml` asserts invariants but never diffs (P2)

It renders both live releases and checks a strong list (Qwen, rerank, TLS
secrets, Grafana role, `refute volumeClaimTemplates`). But nothing compares the
render against the previous commit, so a chart change that alters anything
*outside* that list reaches production silently.

### D7 — production has no rollback workflow at all (P2)

`rollback.yml` targets `secrets.EC2_HOST` — the Compose box. Nothing rolls back
k3s. Real production rollback today is a manual Namecheap A-record revert
(~60s TTL) or an ArgoCD revision/tag revert, neither written down.

## 3. Execution plan

### Ground rules

- **Never push to `master`.** Every change goes through a PR. ArgoCD auto-syncs
  `master` with `selfHeal` + `prune`; a merge touching `infra/helm/**` applies
  to production with no human gate.
- `ci.yml` runs on every PR (ruff, pytest, ingestion tests, web lint+build).
  `helm-chart.yml` runs on PRs touching `infra/helm/**`. Both must be green.
- Do not touch `git.vinmec.tech`, team ArgoCD/k3s, or team repos.
- Do not start, stop, or terminate any EC2 instance without an explicit go.
- Do not edit files in the other worktrees (`D:\VSF-DUOCTHU-codex-*`,
  `D:\VSF-DUOCTHU-claude-gitops`); check `coordination/` for active claims and
  file your own claim before starting.

### PR A — workflow safety (D1, D2, D3, D4)

Touches only `.github/**` and `infra/argocd/**`. ArgoCD renders
`infra/helm/medical-chatbot`, so **this PR cannot alter production manifests**.
Confirm that rather than assume it.

1. **Fix the misdirected probes (D1).** In `deploy.yml` and `rollback.yml`,
   make the Caddy/Grafana check target the box being deployed instead of a
   public DNS name that now points elsewhere — e.g. resolve the hostname to the
   local Caddy container so TLS and routing are still exercised.
   **Verification gate:** prove the fixed check *fails* when Caddy is broken.
   A probe that cannot fail is the defect being fixed, not a fix. (See the
   `set -e` / `! grep` silent-pass traps already documented in
   `helm-chart.yml`.)
2. **Stop `deploy.yml` firing on push (D2).** Reduce it to `workflow_dispatch`
   only. Compose then stays pinned at `df57e6b` — a stable rollback rather than
   one that tracks `master`. State the trade-off in the commit message: the
   fallback stops drifting, but also stops receiving fixes, so it goes stale as
   production moves. That is acceptable for a time-boxed acceptance window and
   is the subject of PR D.
3. **Tell the truth in `build-practice-images.yml` (D3).** Replace the false
   header comment. Renaming the workflow to name it as the production deploy
   path is preferable — **trap:** it self-references in its own `paths:` filter
   and the sync script path, so both must be updated together or the workflow
   silently stops triggering.
4. **Remove or correct `infra/argocd/applications/prod/app.yaml` (D4).** Check
   `dev/` and `staging/` in the same directory for the same rot. Deleting is
   fine if nothing applies them; verify that first.

### PR B — Helm hygiene (D5, D6)

Touches `infra/helm/**`, so merging **does** reach production. Highest care.

5. **Drop the redundant `aws.region` block from the WIP diff (D5)**; keep the
   static-credentials support, still defaulted off. Guard against a nil `aws`
   key so a values file that omits it cannot break the render — a template
   error here means ArgoCD cannot sync production at all.
6. **Add a baseline render diff to `helm-chart.yml` (D6).** Render
   `values-production.yaml` and `values-production-data.yaml` at the PR base and
   at HEAD, then surface the diff in the job summary. The goal is that no chart
   change ever reaches production without a human having seen exactly what it
   does to the manifests.
   **Verification gate:** the diff for PR B itself must be **empty** — the
   static-credentials change is defaults-off and must render byte-identically.
   If it is not empty, stop and explain why before merging.

Note: `helm` is **not installed** on this workstation (`helm: command not found`
in both bash and PowerShell). Either install it or rely on the CI render — but
do not claim the render is unchanged without one of the two actually running it.

### PR C — document the real rollback (D7)

7. Write the production rollback runbook: revert the ArgoCD Application to the
   previous image tag, and/or revert the `realvuxbaro.me` A record to
   `52.0.158.61`. Note that the Namecheap edit is a **manual owner step** — a
   harness permission classifier has blocked agent form input on that page
   before, so the runbook must not assume an agent can do it.

### PR D — decide the Compose lifecycle (owner call, not an agent call)

8. Compose is a second `t3.large` running purely as a fallback. Once the
   acceptance window closes, ArgoCD's own revision history covers rollback and
   the instance is redundant. Surface the choice and the monthly cost; **do not
   act on it without an explicit go.**

### Out of scope

`Feature-List-AI-Duoc-thu-V1.md`, `presentation/`, and `.claude/skills/` are
untracked and match no workflow trigger path. Committing them is inert and can
be a separate trivial commit — keep it out of PRs A–D.

## 4. Order and why

D1 first: an untrustworthy rollback is worse than no rollback, because it fails
silently at the moment of maximum pressure. D2 next: it is the defect that would
consume the rollback. Everything after is hardening.