Skip to main content

Dead Env Cleanup Rollout CrashLoop

This retrospective covers the May 26, 2026 rollout where GitOps removed backend env vars before the image that no longer needed them had actually rolled out. Kubernetes prevented an outage, but the incident clarified an important delivery rule: config cleanup that depends on a code change must wait for image deployment, not just for a merged commit.

Context​

The cleanup followed the backend removal of the in-monolith Spring AI path. In production, the surviving runtime contract is now easy to see:

  • lumie-infra/applications/lumie/backend/common-values.yaml still sets CHATBOT_SVC_URL.
  • That same production values file no longer sets GEMINI_API_KEY or LUMIE_AI_LANGGRAPH_ENABLED.
  • lumie-backend/app/src/main/resources/application.yaml still reads the chatbot worker URL through lumie.services.chatbot.url.

There is one intentional environment split to call out explicitly: lumie-backend/Tiltfile still carries LUMIE_AI_LANGGRAPH_ENABLED for the dev cutover flow. That was not the production contract that failed during this rollout.

What Broke Or Changed​

Argo CD applied the env-var cleanup first, which changed the Deployment pod template and created a new ReplicaSet immediately. The image tag had not advanced yet, so the new surge pod still started from the old backend image.

That old image still expected the removed Spring AI key at startup, so the new pod entered CrashLoopBackOff while the old ready pods continued serving traffic.

The rollout was noisy but not user-visible because the production strategy held:

  • lumie-infra/applications/lumie/backend/common-values.yaml still uses maxUnavailable: 0
  • the same file still uses maxSurge: 1

With that strategy, the broken surge pod never replaced the healthy old replicas.

Root Cause​

The confirmed root cause was timing, not Helm syntax or Argo CD state drift.

Two changes had different readiness conditions:

  • the code change became safe only when the new backend image was running everywhere;
  • the env cleanup became active as soon as Argo CD applied the manifest diff.

Those two events were separated by one CI/build/deploy cycle. During that window, Kubernetes tried to start a pod from the old image with the new env contract.

Resolution​

The durable repository state matches the post-incident contract:

  • production values keep CHATBOT_SVC_URL and no longer carry the removed Spring AI env vars;
  • application runtime still reads only the worker URL in application.yaml;
  • the deployment strategy keeps zero unavailable pods during rollout;
  • local and CI verification are easier to line up because lumie-backend/build.gradle.kts now wires lumieLint into check.

No manual rollback was required during the incident. The rollout recovered once CI published the newer backend image and GitOps advanced the deployment to the image that matched the cleaned-up env contract.

Prevention​

  • Treat config cleanup as image-dependent work whenever startup bindings or settings classes are being removed from code.
  • Check the running pod image before removing a now-dead env var from GitOps manifests.
  • Keep readiness and rolling-update safeguards strict enough that a broken surge pod cannot evict healthy traffic-serving replicas.
  • Call out intentional dev-versus-prod config drift explicitly when grep results would otherwise look contradictory.