Track the practice cluster's Helm values in Git

This commit is contained in:
2026-08-17 13:59:39 +07:00
parent a1de4715a4
commit c778f7e9cc
8 changed files with 484 additions and 0 deletions
+80
View File
@@ -0,0 +1,80 @@
name: Validate Helm chart
on:
push:
paths:
- infra/helm/**
- .github/workflows/helm-chart.yml
pull_request:
paths:
- infra/helm/**
- .github/workflows/helm-chart.yml
permissions:
contents: read
jobs:
validate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: azure/setup-helm@v4
with:
version: v3.17.3
- name: Lint chart
run: helm lint infra/helm/medical-chatbot
- name: Render default and production manifests
run: |
helm template default infra/helm/medical-chatbot > /tmp/default.yaml
helm template production infra/helm/medical-chatbot \
--values infra/helm/medical-chatbot/values-prod.yaml \
> /tmp/production.yaml
grep -q 'ANSWER_MODEL_ID: "qwen.qwen3-next-80b-a3b"' /tmp/production.yaml
grep -q 'RERANK_ENABLED: "true"' /tmp/production.yaml
grep -q 'checksum/runtime-config:' /tmp/production.yaml
# The practice cluster is only evidence for the production migration
# while it renders the same behavioural contract as production, so both
# of its releases are asserted here rather than trusted by review.
- name: Render practice manifests
run: |
helm template medical-chatbot-app infra/helm/medical-chatbot \
--values infra/helm/medical-chatbot/values-practice.yaml \
> /tmp/practice-app.yaml
helm template medical-chatbot-data infra/helm/medical-chatbot \
--values infra/helm/medical-chatbot/values-practice-data.yaml \
> /tmp/practice-data.yaml
# Behavioural parity with the audited production runtime contract.
grep -q 'ANSWER_MODEL_ID: "qwen.qwen3-next-80b-a3b"' /tmp/practice-app.yaml
grep -q 'ANSWER_PROVIDER: "bedrock-converse"' /tmp/practice-app.yaml
grep -q 'EMBEDDING_PROVIDER: "cohere-v4"' /tmp/practice-app.yaml
grep -q 'EMBEDDING_DIMENSIONS: "1024"' /tmp/practice-app.yaml
grep -q 'EVIDENCE_MINIMUM_SCORE: "0.12"' /tmp/practice-app.yaml
grep -q 'RERANK_ENABLED: "true"' /tmp/practice-app.yaml
grep -q 'AWS_REGION: "us-east-1"' /tmp/practice-app.yaml
grep -q 'checksum/runtime-config:' /tmp/practice-app.yaml
grep -q 'host: readytochat.realvuxbaro.me' /tmp/practice-app.yaml
# The app release must own neither data StatefulSet: PostgreSQL and
# Qdrant belong to the data release, so an app-side sync failure or
# prune can never delete the corpus or the query history. Only those
# two use volumeClaimTemplates — the observability PVCs are the app
# release's own and are expected here.
#
# `set -e` ignores a command whose status is inverted with `!`, so
# every must-NOT-contain assertion is written as an explicit exit.
refute() {
if grep -q "$2" "$1"; then
echo "::error::$1 must not contain: $2"
exit 1
fi
}
refute /tmp/practice-app.yaml 'volumeClaimTemplates'
grep -q 'medical-chatbot-data-medical-chatbot-qdrant' /tmp/practice-app.yaml
# ...and the data release must own nothing else.
refute /tmp/practice-data.yaml 'medical-chatbot-data-medical-chatbot-ai-service'
refute /tmp/practice-data.yaml 'kind: Ingress'
grep -q 'volumeClaimTemplates' /tmp/practice-data.yaml
@@ -0,0 +1,250 @@
# ArgoCD production migration state — 2026-08-17
## Purpose
Shared handoff for Codex and Claude. This is the current operational reference
for the personal production-to-ArgoCD migration. Re-check live state before a
mutation because runtime and Git revisions can change after this snapshot.
## Current verdict
**Latency parity achieved for the measured post-fix scope.** After aligning the
personal k3s practice environment with production on Qwen and reranking, a
controlled five-pair test showed practice at 12.194 seconds average and
production at 12.842 seconds average, with zero errors or timeouts. There is no
measured systematic k3s latency penalty remaining.
This verdict is specifically about runtime latency parity. It does not mean the
production migration is ready to cut over: tracked ArgoCD values, immutable
production images, state restore, DNS rollback and broader soak/regression gates
remain open.
## Owner intent
- Production must eventually run through ArgoCD.
- The separate personal k3s/ArgoCD EC2 is the rehearsal environment used to
discover migration risk before cutover.
- The intended cutover is to move `realvuxbaro.me` to the proven k3s workload.
- Keep the existing Compose production EC2 intact as the DNS-level rollback
until the ArgoCD deployment has passed its acceptance window.
## Hard boundary
- Do not access, modify, or push to `git.vinmec.tech`.
- Do not access or mutate the team's ArgoCD/k3s infrastructure.
- The active scope is the owner's personal GitHub repository, personal AWS
account, Compose production EC2, and personal practice k3s/ArgoCD EC2.
- No team Gitea or team infrastructure action occurred in the 2026-08-17 Codex
session. Reading an old Claude memory that mentioned Gitea did not authorize
or cause a connection to it.
## Topology at this snapshot
| Role | Runtime | Address / identity | State |
| --- | --- | --- | --- |
| Current production | Docker Compose on EC2 `i-039fc8f6102467a54` (`t3.large`) | `realvuxbaro.me`, `52.0.158.61` | Live; rollback source |
| Migration rehearsal | k3s + ArgoCD on EC2 `i-035cd1f80f4462455` (`t3.large`) | `readytochat.realvuxbaro.me`, `argocd.realvuxbaro.me` | Live, Synced, Healthy |
| Source repository | Private GitHub | `BaoVu2k4/vsf-duocthu` | Active source for both personal environments |
The practice instance is currently `t3.large`. Older Claude memory that says it
is still `t3.medium` is stale.
## Production evidence collected by Codex
Codex added and manually dispatched the read-only workflow
`.github/workflows/audit-production-runtime.yml`.
- Commit: `2ff65d9` (`Add read-only production runtime audit`).
- GitHub Actions run: `31993964589`, passed.
- The commit touched only the new manual audit workflow. It did not match the
existing production deploy workflow's path filters and did not restart the
Compose stack.
- Production checkout revision: `df57e6b0806daccb13970dff2aecb7b7dd33eddd`,
branch `master`.
- `ai-service`, `web`, PostgreSQL, and Qdrant containers: running.
- `/health`: HTTP 200; `/ready`: HTTP 200.
- Qdrant collection: `duocthu_v1`, 15,100 points, green, cosine, 1,024
dimensions.
- Persistent data mounts: Docker volumes `docker_postgres-data` and
`docker_qdrant-data`.
Safe production AI runtime contract:
| Setting | Effective value |
| --- | --- |
| `ANSWER_PROVIDER` | `bedrock-converse` |
| `ANSWER_MODEL_ID` | `qwen.qwen3-next-80b-a3b` |
| `EMBEDDING_PROVIDER` | `cohere-v4` |
| `EMBEDDING_DIMENSIONS` | `1024` |
| `EVIDENCE_MINIMUM_SCORE` | `0.12` |
| `RERANK_ENABLED` | `true` |
| `MAX_WALL_CLOCK_MS` | `40000` |
| `MAX_LLM_CALLS_PER_TURN` | `8` |
| `AWS_REGION` | `us-east-1` |
| `QDRANT_COLLECTION` | `duocthu_v1` |
| `OTEL_ENABLED` | `true` |
| `OTEL_SAMPLE_RATIO` | `1.0` |
No credential, secret value, patient data, or raw `.env.prod` content was
printed or stored.
## Practice evidence collected by Codex
The ArgoCD API was read using the gitignored local practice credential file.
No practice Application or cluster resource was mutated during this audit.
- Application `medical-chatbot-app`: target `master`, compared revision
`f9c20a67943e1df3c9e0cb08d01ebaaee48c0470`, Synced, Healthy, automated
prune and self-heal enabled.
- AI image:
`ghcr.io/baovu2k4/vsf-duocthu-ai-service:f9c20a67943e1df3c9e0cb08d01ebaaee48c0470`.
- Application `medical-chatbot-data` separately owns PostgreSQL and Qdrant.
- Practice AI config explicitly selects `deepseek.v3.2` and does not expose a
Helm value for `RERANK_ENABLED`, so the application default is `false`.
- Practice and production therefore differ in at least two behavior-changing
settings: answer model and rerank enablement.
## Corrected conclusions
- Config drift is real: production runs Qwen with reranking enabled; practice
runs DeepSeek with reranking disabled.
- The drift is a strong explanation for practice tail latency and
`request_budget_exhausted`, but causality must be confirmed by running the
same model/config and an interleaved benchmark.
- Claude's claim that both environments were on the exact same commit
`f9c20a6` was false. Production was observed at `df57e6b`; practice was at
`f9c20a6`. The commits between them did not change AI application source, so
this correction does not remove the proven configuration drift.
- Production behavior/data/config is the migration baseline. Docker Compose is
not the target architecture; it remains the rollback implementation.
## Runtime parity remediation applied
Codex fixed and deployed the main behavior-changing drift to the personal
practice environment on 2026-08-17.
- Commit `5a1a600` added Helm mappings for embedding dimensions, evidence
threshold, AWS region and rerank enablement. It also added a runtime-config
checksum to the AI Deployment Pod template so a ConfigMap change causes an
actual rollout instead of leaving the old process alive.
- `values-prod.yaml` now records the observed production baseline: Qwen
`qwen.qwen3-next-80b-a3b`, Cohere v4, 1,024 dimensions, evidence threshold
`0.12`, reranking enabled and AWS region `us-east-1`.
- Helm validation run `31995055293`: one chart linted, zero failed; default and
production manifests rendered and asserted.
- Full CI run `31995055272`: ingestion pytest, AI ruff+pytest, and web
lint+build all passed.
- The personal ArgoCD practice Application accepted the Qwen+rerank baseline
and converged at revision `5a1a600`: Synced and Healthy.
- Effective practice manifest after convergence: Qwen, rerank `true`, Cohere
v4, 1,024 dimensions, threshold `0.12`, `us-east-1`, and a non-empty
`checksum/runtime-config` Pod annotation.
- CloudWatch over the post-rollout six-minute window recorded 24 Qwen
invocations and no DeepSeek invocation. This confirms the exercised path no
longer used DeepSeek; CloudWatch remains account-aggregate rather than
environment-labelled.
Small live parity sample through each environment's real `/api/chat` path:
| Scope | Practice | Production |
| --- | ---: | ---: |
| Initial smoke | 8.75 s | 8.94 s |
| Three explicit attribute questions — average | 15.33 s | 11.96 s |
| Three explicit attribute questions — maximum | 22.84 s | 15.54 s |
| Errors/timeouts in the three-question sample | 0 | 0 |
All three explicit questions were answerable with grounded evidence and one
citation in both environments. The initial unaccented overview question was
answerable on practice but clarified for a missing attribute on production, so
behavioral determinism still needs a larger test set.
Conclusion: the major model/rerank drift is fixed and the observed 23x tail /
40-second timeout pattern did not reproduce after parity. Practice remained
about 28% slower on the three-question sample and had a 22.84-second maximum,
so this is evidence of a material fix, not proof that all residual latency is
gone. Run a larger interleaved soak before production cutover.
### Residual-latency investigation
The apparent 28% residual above did not survive a controlled follow-up. Five
additional pairs used the same explicit Zolpidem attribute query, alternated
which environment ran first, used fresh conversations, and captured each Tempo
trace.
| Environment | N | Average | Median | Minimum | Maximum | Errors |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| Practice | 5 | 12.194 s | 12.036 s | 11.876 s | 12.691 s | 0 |
| Production | 5 | 12.842 s | 12.333 s | 10.803 s | 15.217 s | 0 |
Practice was 0.648 seconds faster on average in this controlled sample. There
is no remaining systematic practice slowdown demonstrated by the data.
Tempo provider-span averages:
| Operation | Practice | Production |
| --- | ---: | ---: |
| Qwen understanding | 5.456 s | 4.795 s |
| Qwen generation | 5.271 s | 5.793 s |
| Qwen entailment | 0.711 s | 0.640 s |
The distributions overlap and the faster side changes per call/stage. Output
length explains the clearest tail: production round 5 returned 1,426
characters and spent 8.048 seconds in generation; the common 910-character
outputs spent roughly 4.55.4 seconds. Restricting comparison to the common
910-character outputs gave generation averages of about 5.14 seconds on
practice and 5.23 seconds on production.
Infrastructure checks also ruled out the earlier throttle hypothesis:
- both instances are `t3.large` in the same `us-east-1d` availability zone,
subnet and VPC;
- practice CPU averaged 15.7% and peaked at 34.87% over the observed 30-minute
window;
- practice retained roughly 599 CPU credits and had zero surplus-credit usage;
- retrieval remained millisecond-scale, while the varying time was inside
Bedrock provider spans.
Final latency diagnosis: the former large gap was caused by the now-fixed
DeepSeek/rerank configuration drift. The smaller post-fix gap from the first
three queries was sampling noise from managed Bedrock inference and variable
generated-output length, not a persistent k3s penalty. No additional
environment-specific latency fix is justified by the measured state. Reducing
the absolute approximately 12-second response time would be a separate RAG
architecture change because a normal turn currently performs understanding,
generation and entailment as sequential Qwen calls.
## Migration risks currently open
1. The live practice Applications use untracked inline Helm values. This is not
yet a complete Git source of truth.
2. `values-prod.yaml` still uses mutable `latest` image tags; it is not safe for
cutover as written even though its behavior settings now match production.
3. PostgreSQL/Qdrant snapshot, restore, rollback, ingress/TLS/DNS, secrets,
resource limits, and failure recovery still need explicit rehearsal gates.
## Workspace ownership and current edits
- Main worktree `D:\VSF-DUOCTHU`, branch `agent/query-history`, contains
pre-existing uncommitted Claude/user changes in three Helm files concerning
optional AWS static credentials plus untracked slide/material files. Do not
stage, overwrite, or discard them accidentally.
- Codex created clean worktree `D:\VSF-DUOCTHU-codex-argocd`, branch
`agent/argocd-prod-migration`, from `origin/master` to isolate migration work.
- No Helm migration change has been made in that clean worktree yet at this
checkpoint.
- A stale Claude worktree under `.claude/worktrees/agent-a1f5e73fc2ce814e8`
contains unrelated, uncommitted table-reconstruction work from 2026-08-05.
## Next safe execution order
1. Move the practice Application's stable non-secret config out of inline Helm
values and into a tracked values file; keep only dynamic image tags and
secret references outside Git.
2. Verify the actual running Pod environment, image digest, Qdrant identity,
health, readiness and traces through a cluster-level inspection path.
3. Run a larger interleaved production/practice benchmark and compare
p50/p95/p99,
timeout rate, answer decisions and citations.
4. Rehearse state restore and DNS rollback. Do not repoint `realvuxbaro.me`
until the gates and rollback path pass.
+40
View File
@@ -0,0 +1,40 @@
# Claude claim — 2026-08-17 (afternoon)
Continuing the Codex ArgoCD migration thread at the owner's instruction. Codex's
own worktree `D:\VSF-DUOCTHU-codex-argocd` was clean at `1684e10` with nothing
in flight when this claim was taken.
## Owned by Claude in this session
- `infra/helm/medical-chatbot/values-practice.yaml` (new)
- `infra/helm/medical-chatbot/values-practice-data.yaml` (new)
- `.github/workflows/helm-chart.yml` (practice render + assertions)
- The two personal practice ArgoCD Application specs
(`medical-chatbot-app`, `medical-chatbot-data`) — moving inline values into
the tracked files above.
Worktree: `D:\VSF-DUOCTHU-claude-gitops`, branch `agent/gitops-tracked-values`.
## Not touched
- `git.vinmec.tech`, team ArgoCD/k3s, team repos — hard boundary, unchanged.
- The Compose production EC2 and `realvuxbaro.me`. `infra/helm/**` is not in
`deploy.yml`'s path filters, so pushing this work cannot restart production.
- The uncommitted AWS-static-credentials Helm diff in the main worktree
`D:\VSF-DUOCTHU` (branch `agent/query-history`). Left exactly as found.
## Which migration risk this closes
Open risk 1 in `ARGOCD_PRODUCTION_MIGRATION_STATE_2026-08-17.md`: the live
practice Applications carried their entire configuration in untracked inline
`spec.source.helm.values`, so the cluster could drift from the repository with
no commit recording it — the same class of failure that left practice on
DeepSeek with reranking off while production ran Qwen with reranking on.
Image tags stay inline deliberately: `.github/scripts/sync_practice_argocd.py`
regex-rewrites them on every push, so a tag committed to Git would be stale by
design. Nothing secret was inline, so nothing secret moves.
Equivalence was checked before the Application specs were edited: both tracked
files parse to structures identical to the live inline values with only the
`aiService.image` / `web.image` blocks removed.
@@ -10,8 +10,12 @@ data:
QDRANT_URL: {{ default (printf "http://%s-qdrant:6333" (include "medical-chatbot.fullname" .)) .Values.qdrant.url | quote }} QDRANT_URL: {{ default (printf "http://%s-qdrant:6333" (include "medical-chatbot.fullname" .)) .Values.qdrant.url | quote }}
QDRANT_COLLECTION: {{ .Values.aiService.config.qdrantCollection | quote }} QDRANT_COLLECTION: {{ .Values.aiService.config.qdrantCollection | quote }}
EMBEDDING_PROVIDER: {{ .Values.aiService.config.embeddingProvider | quote }} EMBEDDING_PROVIDER: {{ .Values.aiService.config.embeddingProvider | quote }}
EMBEDDING_DIMENSIONS: {{ .Values.aiService.config.embeddingDimensions | quote }}
EVIDENCE_MINIMUM_SCORE: {{ .Values.aiService.config.evidenceMinimumScore | quote }}
AWS_REGION: {{ .Values.aiService.config.awsRegion | quote }}
ANSWER_PROVIDER: {{ .Values.aiService.config.answerProvider | quote }} ANSWER_PROVIDER: {{ .Values.aiService.config.answerProvider | quote }}
ANSWER_MODEL_ID: {{ .Values.aiService.config.answerModelId | quote }} ANSWER_MODEL_ID: {{ .Values.aiService.config.answerModelId | quote }}
RERANK_ENABLED: {{ .Values.aiService.config.rerankEnabled | quote }}
ENTITIES_PATH: {{ .Values.aiService.config.entitiesPath | quote }} ENTITIES_PATH: {{ .Values.aiService.config.entitiesPath | quote }}
METRICS_ENABLED: {{ .Values.aiService.config.metricsEnabled | quote }} METRICS_ENABLED: {{ .Values.aiService.config.metricsEnabled | quote }}
OTEL_ENABLED: {{ and .Values.observability.enabled .Values.aiService.config.otelEnabled | quote }} OTEL_ENABLED: {{ and .Values.observability.enabled .Values.aiService.config.otelEnabled | quote }}
@@ -40,6 +44,7 @@ spec:
app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/instance: {{ .Release.Name }}
app.kubernetes.io/component: ai-service app.kubernetes.io/component: ai-service
annotations: annotations:
checksum/runtime-config: {{ dict "config" .Values.aiService.config "environment" .Values.global.environment "qdrantUrl" .Values.qdrant.url "observabilityEnabled" .Values.observability.enabled | toJson | sha256sum | quote }}
prometheus.io/scrape: "true" prometheus.io/scrape: "true"
prometheus.io/path: /metrics prometheus.io/path: /metrics
prometheus.io/port: "8000" prometheus.io/port: "8000"
@@ -0,0 +1,28 @@
# k3s practice cluster — the `medical-chatbot-data` release.
#
# Companion to values-practice.yaml. The practice cluster deliberately splits
# the chart into two ArgoCD Applications so that PostgreSQL and Qdrant survive
# every app rollout, prune and self-heal: only this release owns the
# PersistentVolumeClaims, so an app-side sync failure can never delete the
# 15,100-point corpus or the query history.
#
# It therefore runs *only* the stateful half — PostgreSQL and Qdrant are left
# at their chart defaults (`enabled: true`) and everything else is switched off.
#
# Kept inline on the Application rather than here: nothing. This release has no
# image tag that CI rewrites and no secret, so it is fully tracked in Git.
global:
environment: k3s-practice
aiService:
enabled: false
web:
enabled: false
observability:
enabled: false
ingress:
enabled: false
@@ -0,0 +1,72 @@
# k3s practice cluster — the `medical-chatbot-app` release.
#
# This file is the Git source of truth for every stable, non-secret setting of
# the practice rehearsal environment. It exists because the same values used to
# live only inside the ArgoCD Application's inline `spec.source.helm.values`,
# where they were invisible to review, diff and rollback: the cluster could
# drift from the repository without a single commit recording it. That is how
# practice ended up on DeepSeek with reranking off while production ran Qwen
# with reranking on.
#
# Two things deliberately stay OUT of this file and remain inline on the
# Application:
#
# - `aiService.image` / `web.image` — rewritten on every push by
# `.github/scripts/sync_practice_argocd.py`, which regex-replaces the tag
# in the inline values. A tag committed here would be stale by design.
# - anything secret. Nothing in the practice release currently needs one;
# Bedrock authenticates through the node's instance role.
#
# ArgoCD applies `valueFiles` in order and then the inline values last, so the
# image block inline still wins over anything the chart defaults set here.
#
# The behavioural settings below mirror the production runtime contract audited
# on 2026-08-17 (see coordination/ARGOCD_PRODUCTION_MIGRATION_STATE_2026-08-17.md).
# Practice is only useful as a migration rehearsal while they stay in sync.
global:
environment: k3s-practice
# The GHCR packages are private; without this the Pods fail ImagePullBackOff.
imagePullSecrets:
- name: ghcr-pull-secret
# PostgreSQL and Qdrant are owned by the separate `medical-chatbot-data`
# release so that redeploying the app never touches persistent state. This
# release therefore runs neither, and points at the data release's Services.
postgres:
enabled: false
qdrant:
enabled: false
url: http://medical-chatbot-data-medical-chatbot-qdrant.medical-chatbot-data.svc.cluster.local:6333
secret:
postgresHost: medical-chatbot-data-medical-chatbot-postgres.medical-chatbot-data.svc.cluster.local
aiService:
config:
embeddingProvider: cohere-v4
embeddingDimensions: 1024
evidenceMinimumScore: 0.12
awsRegion: us-east-1
answerProvider: bedrock-converse
answerModelId: qwen.qwen3-next-80b-a3b
rerankEnabled: true
observability:
grafana:
# Practice-only convenience: the rehearsal cluster is a throwaway box and
# Grafana sits behind the same ingress host. Never set this on production.
anonymousAdmin: true
ingress:
enabled: true
className: traefik
host: readytochat.realvuxbaro.me
annotations:
cert-manager.io/cluster-issuer: letsencrypt-prod
traefik.ingress.kubernetes.io/router.entrypoints: websecure
tls:
- hosts:
- readytochat.realvuxbaro.me
secretName: readytochat-tls
@@ -7,7 +7,12 @@ aiService:
tag: latest tag: latest
config: config:
embeddingProvider: cohere-v4 embeddingProvider: cohere-v4
embeddingDimensions: 1024
evidenceMinimumScore: 0.12
awsRegion: us-east-1
answerProvider: bedrock-converse answerProvider: bedrock-converse
answerModelId: qwen.qwen3-next-80b-a3b
rerankEnabled: true
otelSampleRatio: 0.25 otelSampleRatio: 0.25
web: web:
+4
View File
@@ -31,8 +31,12 @@ aiService:
port: 8000 port: 8000
config: config:
embeddingProvider: disabled embeddingProvider: disabled
embeddingDimensions: 1024
evidenceMinimumScore: 0.12
awsRegion: us-east-1
answerProvider: disabled answerProvider: disabled
answerModelId: deepseek.v3.2 answerModelId: deepseek.v3.2
rerankEnabled: false
# The Dockerfile bakes the catalog file in at this path; config.py's own # The Dockerfile bakes the catalog file in at this path; config.py's own
# repo-relative default only resolves inside a full monorepo checkout, # repo-relative default only resolves inside a full monorepo checkout,
# not this flattened image layout. Matches apps/ai-service/.env.example. # not this flattened image layout. Matches apps/ai-service/.env.example.