Track the practice cluster's Helm values in Git
This commit is contained in:
@@ -0,0 +1,250 @@
|
||||
# ArgoCD production migration state — 2026-08-17
|
||||
|
||||
## Purpose
|
||||
|
||||
Shared handoff for Codex and Claude. This is the current operational reference
|
||||
for the personal production-to-ArgoCD migration. Re-check live state before a
|
||||
mutation because runtime and Git revisions can change after this snapshot.
|
||||
|
||||
## Current verdict
|
||||
|
||||
**Latency parity achieved for the measured post-fix scope.** After aligning the
|
||||
personal k3s practice environment with production on Qwen and reranking, a
|
||||
controlled five-pair test showed practice at 12.194 seconds average and
|
||||
production at 12.842 seconds average, with zero errors or timeouts. There is no
|
||||
measured systematic k3s latency penalty remaining.
|
||||
|
||||
This verdict is specifically about runtime latency parity. It does not mean the
|
||||
production migration is ready to cut over: tracked ArgoCD values, immutable
|
||||
production images, state restore, DNS rollback and broader soak/regression gates
|
||||
remain open.
|
||||
|
||||
## Owner intent
|
||||
|
||||
- Production must eventually run through ArgoCD.
|
||||
- The separate personal k3s/ArgoCD EC2 is the rehearsal environment used to
|
||||
discover migration risk before cutover.
|
||||
- The intended cutover is to move `realvuxbaro.me` to the proven k3s workload.
|
||||
- Keep the existing Compose production EC2 intact as the DNS-level rollback
|
||||
until the ArgoCD deployment has passed its acceptance window.
|
||||
|
||||
## Hard boundary
|
||||
|
||||
- Do not access, modify, or push to `git.vinmec.tech`.
|
||||
- Do not access or mutate the team's ArgoCD/k3s infrastructure.
|
||||
- The active scope is the owner's personal GitHub repository, personal AWS
|
||||
account, Compose production EC2, and personal practice k3s/ArgoCD EC2.
|
||||
- No team Gitea or team infrastructure action occurred in the 2026-08-17 Codex
|
||||
session. Reading an old Claude memory that mentioned Gitea did not authorize
|
||||
or cause a connection to it.
|
||||
|
||||
## Topology at this snapshot
|
||||
|
||||
| Role | Runtime | Address / identity | State |
|
||||
| --- | --- | --- | --- |
|
||||
| Current production | Docker Compose on EC2 `i-039fc8f6102467a54` (`t3.large`) | `realvuxbaro.me`, `52.0.158.61` | Live; rollback source |
|
||||
| Migration rehearsal | k3s + ArgoCD on EC2 `i-035cd1f80f4462455` (`t3.large`) | `readytochat.realvuxbaro.me`, `argocd.realvuxbaro.me` | Live, Synced, Healthy |
|
||||
| Source repository | Private GitHub | `BaoVu2k4/vsf-duocthu` | Active source for both personal environments |
|
||||
|
||||
The practice instance is currently `t3.large`. Older Claude memory that says it
|
||||
is still `t3.medium` is stale.
|
||||
|
||||
## Production evidence collected by Codex
|
||||
|
||||
Codex added and manually dispatched the read-only workflow
|
||||
`.github/workflows/audit-production-runtime.yml`.
|
||||
|
||||
- Commit: `2ff65d9` (`Add read-only production runtime audit`).
|
||||
- GitHub Actions run: `31993964589`, passed.
|
||||
- The commit touched only the new manual audit workflow. It did not match the
|
||||
existing production deploy workflow's path filters and did not restart the
|
||||
Compose stack.
|
||||
- Production checkout revision: `df57e6b0806daccb13970dff2aecb7b7dd33eddd`,
|
||||
branch `master`.
|
||||
- `ai-service`, `web`, PostgreSQL, and Qdrant containers: running.
|
||||
- `/health`: HTTP 200; `/ready`: HTTP 200.
|
||||
- Qdrant collection: `duocthu_v1`, 15,100 points, green, cosine, 1,024
|
||||
dimensions.
|
||||
- Persistent data mounts: Docker volumes `docker_postgres-data` and
|
||||
`docker_qdrant-data`.
|
||||
|
||||
Safe production AI runtime contract:
|
||||
|
||||
| Setting | Effective value |
|
||||
| --- | --- |
|
||||
| `ANSWER_PROVIDER` | `bedrock-converse` |
|
||||
| `ANSWER_MODEL_ID` | `qwen.qwen3-next-80b-a3b` |
|
||||
| `EMBEDDING_PROVIDER` | `cohere-v4` |
|
||||
| `EMBEDDING_DIMENSIONS` | `1024` |
|
||||
| `EVIDENCE_MINIMUM_SCORE` | `0.12` |
|
||||
| `RERANK_ENABLED` | `true` |
|
||||
| `MAX_WALL_CLOCK_MS` | `40000` |
|
||||
| `MAX_LLM_CALLS_PER_TURN` | `8` |
|
||||
| `AWS_REGION` | `us-east-1` |
|
||||
| `QDRANT_COLLECTION` | `duocthu_v1` |
|
||||
| `OTEL_ENABLED` | `true` |
|
||||
| `OTEL_SAMPLE_RATIO` | `1.0` |
|
||||
|
||||
No credential, secret value, patient data, or raw `.env.prod` content was
|
||||
printed or stored.
|
||||
|
||||
## Practice evidence collected by Codex
|
||||
|
||||
The ArgoCD API was read using the gitignored local practice credential file.
|
||||
No practice Application or cluster resource was mutated during this audit.
|
||||
|
||||
- Application `medical-chatbot-app`: target `master`, compared revision
|
||||
`f9c20a67943e1df3c9e0cb08d01ebaaee48c0470`, Synced, Healthy, automated
|
||||
prune and self-heal enabled.
|
||||
- AI image:
|
||||
`ghcr.io/baovu2k4/vsf-duocthu-ai-service:f9c20a67943e1df3c9e0cb08d01ebaaee48c0470`.
|
||||
- Application `medical-chatbot-data` separately owns PostgreSQL and Qdrant.
|
||||
- Practice AI config explicitly selects `deepseek.v3.2` and does not expose a
|
||||
Helm value for `RERANK_ENABLED`, so the application default is `false`.
|
||||
- Practice and production therefore differ in at least two behavior-changing
|
||||
settings: answer model and rerank enablement.
|
||||
|
||||
## Corrected conclusions
|
||||
|
||||
- Config drift is real: production runs Qwen with reranking enabled; practice
|
||||
runs DeepSeek with reranking disabled.
|
||||
- The drift is a strong explanation for practice tail latency and
|
||||
`request_budget_exhausted`, but causality must be confirmed by running the
|
||||
same model/config and an interleaved benchmark.
|
||||
- Claude's claim that both environments were on the exact same commit
|
||||
`f9c20a6` was false. Production was observed at `df57e6b`; practice was at
|
||||
`f9c20a6`. The commits between them did not change AI application source, so
|
||||
this correction does not remove the proven configuration drift.
|
||||
- Production behavior/data/config is the migration baseline. Docker Compose is
|
||||
not the target architecture; it remains the rollback implementation.
|
||||
|
||||
## Runtime parity remediation applied
|
||||
|
||||
Codex fixed and deployed the main behavior-changing drift to the personal
|
||||
practice environment on 2026-08-17.
|
||||
|
||||
- Commit `5a1a600` added Helm mappings for embedding dimensions, evidence
|
||||
threshold, AWS region and rerank enablement. It also added a runtime-config
|
||||
checksum to the AI Deployment Pod template so a ConfigMap change causes an
|
||||
actual rollout instead of leaving the old process alive.
|
||||
- `values-prod.yaml` now records the observed production baseline: Qwen
|
||||
`qwen.qwen3-next-80b-a3b`, Cohere v4, 1,024 dimensions, evidence threshold
|
||||
`0.12`, reranking enabled and AWS region `us-east-1`.
|
||||
- Helm validation run `31995055293`: one chart linted, zero failed; default and
|
||||
production manifests rendered and asserted.
|
||||
- Full CI run `31995055272`: ingestion pytest, AI ruff+pytest, and web
|
||||
lint+build all passed.
|
||||
- The personal ArgoCD practice Application accepted the Qwen+rerank baseline
|
||||
and converged at revision `5a1a600`: Synced and Healthy.
|
||||
- Effective practice manifest after convergence: Qwen, rerank `true`, Cohere
|
||||
v4, 1,024 dimensions, threshold `0.12`, `us-east-1`, and a non-empty
|
||||
`checksum/runtime-config` Pod annotation.
|
||||
- CloudWatch over the post-rollout six-minute window recorded 24 Qwen
|
||||
invocations and no DeepSeek invocation. This confirms the exercised path no
|
||||
longer used DeepSeek; CloudWatch remains account-aggregate rather than
|
||||
environment-labelled.
|
||||
|
||||
Small live parity sample through each environment's real `/api/chat` path:
|
||||
|
||||
| Scope | Practice | Production |
|
||||
| --- | ---: | ---: |
|
||||
| Initial smoke | 8.75 s | 8.94 s |
|
||||
| Three explicit attribute questions — average | 15.33 s | 11.96 s |
|
||||
| Three explicit attribute questions — maximum | 22.84 s | 15.54 s |
|
||||
| Errors/timeouts in the three-question sample | 0 | 0 |
|
||||
|
||||
All three explicit questions were answerable with grounded evidence and one
|
||||
citation in both environments. The initial unaccented overview question was
|
||||
answerable on practice but clarified for a missing attribute on production, so
|
||||
behavioral determinism still needs a larger test set.
|
||||
|
||||
Conclusion: the major model/rerank drift is fixed and the observed 2–3x tail /
|
||||
40-second timeout pattern did not reproduce after parity. Practice remained
|
||||
about 28% slower on the three-question sample and had a 22.84-second maximum,
|
||||
so this is evidence of a material fix, not proof that all residual latency is
|
||||
gone. Run a larger interleaved soak before production cutover.
|
||||
|
||||
### Residual-latency investigation
|
||||
|
||||
The apparent 28% residual above did not survive a controlled follow-up. Five
|
||||
additional pairs used the same explicit Zolpidem attribute query, alternated
|
||||
which environment ran first, used fresh conversations, and captured each Tempo
|
||||
trace.
|
||||
|
||||
| Environment | N | Average | Median | Minimum | Maximum | Errors |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| Practice | 5 | 12.194 s | 12.036 s | 11.876 s | 12.691 s | 0 |
|
||||
| Production | 5 | 12.842 s | 12.333 s | 10.803 s | 15.217 s | 0 |
|
||||
|
||||
Practice was 0.648 seconds faster on average in this controlled sample. There
|
||||
is no remaining systematic practice slowdown demonstrated by the data.
|
||||
|
||||
Tempo provider-span averages:
|
||||
|
||||
| Operation | Practice | Production |
|
||||
| --- | ---: | ---: |
|
||||
| Qwen understanding | 5.456 s | 4.795 s |
|
||||
| Qwen generation | 5.271 s | 5.793 s |
|
||||
| Qwen entailment | 0.711 s | 0.640 s |
|
||||
|
||||
The distributions overlap and the faster side changes per call/stage. Output
|
||||
length explains the clearest tail: production round 5 returned 1,426
|
||||
characters and spent 8.048 seconds in generation; the common 910-character
|
||||
outputs spent roughly 4.5–5.4 seconds. Restricting comparison to the common
|
||||
910-character outputs gave generation averages of about 5.14 seconds on
|
||||
practice and 5.23 seconds on production.
|
||||
|
||||
Infrastructure checks also ruled out the earlier throttle hypothesis:
|
||||
|
||||
- both instances are `t3.large` in the same `us-east-1d` availability zone,
|
||||
subnet and VPC;
|
||||
- practice CPU averaged 15.7% and peaked at 34.87% over the observed 30-minute
|
||||
window;
|
||||
- practice retained roughly 599 CPU credits and had zero surplus-credit usage;
|
||||
- retrieval remained millisecond-scale, while the varying time was inside
|
||||
Bedrock provider spans.
|
||||
|
||||
Final latency diagnosis: the former large gap was caused by the now-fixed
|
||||
DeepSeek/rerank configuration drift. The smaller post-fix gap from the first
|
||||
three queries was sampling noise from managed Bedrock inference and variable
|
||||
generated-output length, not a persistent k3s penalty. No additional
|
||||
environment-specific latency fix is justified by the measured state. Reducing
|
||||
the absolute approximately 12-second response time would be a separate RAG
|
||||
architecture change because a normal turn currently performs understanding,
|
||||
generation and entailment as sequential Qwen calls.
|
||||
|
||||
## Migration risks currently open
|
||||
|
||||
1. The live practice Applications use untracked inline Helm values. This is not
|
||||
yet a complete Git source of truth.
|
||||
2. `values-prod.yaml` still uses mutable `latest` image tags; it is not safe for
|
||||
cutover as written even though its behavior settings now match production.
|
||||
3. PostgreSQL/Qdrant snapshot, restore, rollback, ingress/TLS/DNS, secrets,
|
||||
resource limits, and failure recovery still need explicit rehearsal gates.
|
||||
|
||||
## Workspace ownership and current edits
|
||||
|
||||
- Main worktree `D:\VSF-DUOCTHU`, branch `agent/query-history`, contains
|
||||
pre-existing uncommitted Claude/user changes in three Helm files concerning
|
||||
optional AWS static credentials plus untracked slide/material files. Do not
|
||||
stage, overwrite, or discard them accidentally.
|
||||
- Codex created clean worktree `D:\VSF-DUOCTHU-codex-argocd`, branch
|
||||
`agent/argocd-prod-migration`, from `origin/master` to isolate migration work.
|
||||
- No Helm migration change has been made in that clean worktree yet at this
|
||||
checkpoint.
|
||||
- A stale Claude worktree under `.claude/worktrees/agent-a1f5e73fc2ce814e8`
|
||||
contains unrelated, uncommitted table-reconstruction work from 2026-08-05.
|
||||
|
||||
## Next safe execution order
|
||||
|
||||
1. Move the practice Application's stable non-secret config out of inline Helm
|
||||
values and into a tracked values file; keep only dynamic image tags and
|
||||
secret references outside Git.
|
||||
2. Verify the actual running Pod environment, image digest, Qdrant identity,
|
||||
health, readiness and traces through a cluster-level inspection path.
|
||||
3. Run a larger interleaved production/practice benchmark and compare
|
||||
p50/p95/p99,
|
||||
timeout rate, answer decisions and citations.
|
||||
4. Rehearse state restore and DNS rollback. Do not repoint `realvuxbaro.me`
|
||||
until the gates and rollback path pass.
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Claude claim — 2026-08-17 (afternoon)
|
||||
|
||||
Continuing the Codex ArgoCD migration thread at the owner's instruction. Codex's
|
||||
own worktree `D:\VSF-DUOCTHU-codex-argocd` was clean at `1684e10` with nothing
|
||||
in flight when this claim was taken.
|
||||
|
||||
## Owned by Claude in this session
|
||||
|
||||
- `infra/helm/medical-chatbot/values-practice.yaml` (new)
|
||||
- `infra/helm/medical-chatbot/values-practice-data.yaml` (new)
|
||||
- `.github/workflows/helm-chart.yml` (practice render + assertions)
|
||||
- The two personal practice ArgoCD Application specs
|
||||
(`medical-chatbot-app`, `medical-chatbot-data`) — moving inline values into
|
||||
the tracked files above.
|
||||
|
||||
Worktree: `D:\VSF-DUOCTHU-claude-gitops`, branch `agent/gitops-tracked-values`.
|
||||
|
||||
## Not touched
|
||||
|
||||
- `git.vinmec.tech`, team ArgoCD/k3s, team repos — hard boundary, unchanged.
|
||||
- The Compose production EC2 and `realvuxbaro.me`. `infra/helm/**` is not in
|
||||
`deploy.yml`'s path filters, so pushing this work cannot restart production.
|
||||
- The uncommitted AWS-static-credentials Helm diff in the main worktree
|
||||
`D:\VSF-DUOCTHU` (branch `agent/query-history`). Left exactly as found.
|
||||
|
||||
## Which migration risk this closes
|
||||
|
||||
Open risk 1 in `ARGOCD_PRODUCTION_MIGRATION_STATE_2026-08-17.md`: the live
|
||||
practice Applications carried their entire configuration in untracked inline
|
||||
`spec.source.helm.values`, so the cluster could drift from the repository with
|
||||
no commit recording it — the same class of failure that left practice on
|
||||
DeepSeek with reranking off while production ran Qwen with reranking on.
|
||||
|
||||
Image tags stay inline deliberately: `.github/scripts/sync_practice_argocd.py`
|
||||
regex-rewrites them on every push, so a tag committed to Git would be stale by
|
||||
design. Nothing secret was inline, so nothing secret moves.
|
||||
|
||||
Equivalence was checked before the Application specs were edited: both tracked
|
||||
files parse to structures identical to the live inline values with only the
|
||||
`aiService.image` / `web.image` blocks removed.
|
||||
Reference in New Issue
Block a user