Fingerprint the Qdrant corpus by content, not by count

coordination/ARGOCD_PRODUCTION_MIGRATION_STATE_2026-08-17.md

@@ -213,14 +213,97 @@ the absolute approximately 12-second response time would be a separate RAG
 architecture change because a normal turn currently performs understanding,
 generation and entailment as sequential Qwen calls.
 
+## GitOps source-of-truth remediation applied (Claude, 2026-08-17 afternoon)
+
+Both practice Applications now render from tracked values files instead of
+untracked inline values. This closes what was open risk 1.
+
+- `infra/helm/medical-chatbot/values-practice.yaml` and
+  `values-practice-data.yaml` hold every stable non-secret setting of the two
+  releases. Pushed to `master` at `a5d1b86`.
+- Only the `aiService.image` / `web.image` blocks remain inline on
+  `medical-chatbot-app`, because `.github/scripts/sync_practice_argocd.py`
+  regex-rewrites those tags on every push; a tag committed to Git would be
+  stale by design. The rewrite contract (exactly one `repository:`/`tag:` pair
+  per image) was asserted against the new inline block before the PUT.
+  `medical-chatbot-data` now carries no inline values at all.
+- Nothing secret was inline, so nothing secret moved. Bedrock still
+  authenticates through the node's instance role.
+- `helm-chart.yml` now renders both practice releases and asserts the
+  production behavioural contract on them, plus that the app release owns
+  neither data StatefulSet — so an app-side sync or prune cannot delete the
+  corpus or the query history.
+
+Evidence that the refactor changed nothing:
+
+- The tracked files were checked to parse to structures identical to the live
+  inline values minus the image blocks, before any Application was edited.
+- ArgoCD effective manifests were captured before and after. Both releases are
+  **byte-for-byte identical** (23 and 6 manifests). Both converged
+  Synced/Healthy at `a5d1b86`. The unchanged `checksum/runtime-config` means
+  no Pod was rolled.
+
+Independent interleaved re-measurement, five *different* clinical questions
+(Codex's controlled sample repeated one Zolpidem query), fresh conversation
+each, alternating which environment ran first:
+
+| Scope | Practice | Production |
+| --- | ---: | ---: |
+| Average | 12.37 s | 13.06 s |
+| Median | 11.90 s | 12.77 s |
+| Maximum | 14.27 s | 16.25 s |
+| Errors/timeouts | 0 | 0 |
+
+All ten answers were `answerable` with at least one citation, and all five
+question pairs agreed on decision. This corroborates the latency-parity verdict
+across question variety rather than a single repeated query. Answer depth is
+still not deterministic between environments: Q1 returned 271 characters with
+one citation on practice against 769 characters with two on production, while
+Q2 and Q5 were character-identical. Decision parity is established; content
+parity is not, and needs a golden-set comparison before cutover.
+
+## Immutable production images applied (Claude, 2026-08-17 afternoon)
+
+`values-prod.yaml` was not deployable as written, beyond the known `latest`
+problem: it pinned no `image.repository`, so both images fell back to the
+chart's local development names (`duocthu-ai-service`, `duocthu-web`) and would
+never have resolved the GHCR packages, and it carried no `imagePullSecrets` for
+those private packages.
+
+Fixed on `master` at `b66c7e5`:
+
+- Both GHCR repositories pinned, `ghcr-pull-secret` declared.
+- Both tags left empty on purpose. Production must run an immutable commit SHA
+  supplied per deploy — the same shape practice already uses — because a
+  `latest` tag makes "which commit is production running?" unanswerable and
+  silently breaks rollback when the tag later points at different content.
+- A new `medical-chatbot.image` helper demands the tag explicitly, so an unset
+  tag fails the render instead of degrading into the chart's `local` default.
+- CI asserts that failure directly (renders `values-prod.yaml` with no tag and
+  requires a non-zero exit plus the guard's message), so the guard cannot rot
+  into a silent default unnoticed.
+
+Still open before this file can be used: the production ArgoCD Application does
+not exist yet, and the `medical-chatbot-prod` Secret it expects
+(`secret.create: false`) must be created in the target namespace first.
+
+Verification that neither afternoon change disturbed the running cluster: after
+both merges, ArgoCD converged Synced/Healthy at `b66c7e5` and the effective
+manifests of both releases remained **byte-for-byte identical to the original
+pre-refactor baseline**. A live query returned `answerable` with one citation
+and character-identical content on both environments (11.36 s practice,
+11.66 s production).
+
 ## Migration risks currently open
 
-1. The live practice Applications use untracked inline Helm values. This is not
-   yet a complete Git source of truth.
-2. `values-prod.yaml` still uses mutable `latest` image tags; it is not safe for
-   cutover as written even though its behavior settings now match production.
-3. PostgreSQL/Qdrant snapshot, restore, rollback, ingress/TLS/DNS, secrets,
+1. PostgreSQL/Qdrant snapshot, restore, rollback, ingress/TLS/DNS, secrets,
    resource limits, and failure recovery still need explicit rehearsal gates.
+3. Cluster-level inspection is currently blocked from this machine: the
+   practice security group scopes port 6443 to the operator's own outbound IP,
+   which has changed again, so `kubectl` hangs. ArgoCD's API and the public
+   ingress were used instead. Re-open the SG rule before any step that needs
+   direct Pod/exec access.
+4. Answer-content parity between environments is unverified (see above).
 
 ## Workspace ownership and current edits
 
@@ -228,23 +311,25 @@ generation and entailment as sequential Qwen calls.
   pre-existing uncommitted Claude/user changes in three Helm files concerning
   optional AWS static credentials plus untracked slide/material files. Do not
   stage, overwrite, or discard them accidentally.
-- Codex created clean worktree `D:\VSF-DUOCTHU-codex-argocd`, branch
-  `agent/argocd-prod-migration`, from `origin/master` to isolate migration work.
-- No Helm migration change has been made in that clean worktree yet at this
-  checkpoint.
+- Codex worktree `D:\VSF-DUOCTHU-codex-argocd`, branch
+  `agent/argocd-prod-migration`. Its migration work landed on `master` through
+  `5a1a600`..`1684e10`; the worktree is clean with nothing in flight.
+- Claude worktree `D:\VSF-DUOCTHU-claude-gitops`, branch
+  `agent/gitops-tracked-values`, holds the GitOps source-of-truth work above.
+  Merged fast-forward into `master` at `a5d1b86`. See
+  `CLAUDE_CLAIM_2026-08-17.md`.
 - A stale Claude worktree under `.claude/worktrees/agent-a1f5e73fc2ce814e8`
   contains unrelated, uncommitted table-reconstruction work from 2026-08-05.
 
 ## Next safe execution order
 
-1. Move the practice Application's stable non-secret config out of inline Helm
-   values and into a tracked values file; keep only dynamic image tags and
-   secret references outside Git.
-2. Verify the actual running Pod environment, image digest, Qdrant identity,
-   health, readiness and traces through a cluster-level inspection path.
-3. Run a larger interleaved production/practice benchmark and compare
-   p50/p95/p99,
-   timeout rate, answer decisions and citations.
+1. Create the production ArgoCD Application against `values-prod.yaml`, and the
+   `medical-chatbot-prod` Secret it expects, without pointing DNS at it yet.
+2. Restore the security-group rule for the operator's current IP, then verify
+   the running Pod environment, image digest, Qdrant identity, health,
+   readiness and traces through a cluster-level inspection path.
+3. Run the golden set against both environments and compare answer content, not
+   just decision and latency.
 4. Rehearse state restore and DNS rollback. Do not repoint `realvuxbaro.me`
    until the gates and rollback path pass.