KEDA Rollout Probe Warmup

Symptom

Use this guide for either of these repeatable failures:

• keda-metrics-apiserver or the KEDA operator enters CrashLoopBackOff or never becomes Ready after a rollout.
• lumie-backend scales up on first light traffic even though sustained CPU demand is low.

What the current repo expects

The checked-in KEDA control-plane values already carry longer probe warmup and larger memory ceilings than the early April defaults:

resources:
  operator:
    requests:
      cpu: 50m
      memory: 128Mi
    limits:
      memory: 192Mi
...
operator:
  livenessProbe:
    initialDelaySeconds: 45
...
metricsServer:
  readinessProbe:
    initialDelaySeconds: 20

Source: lumie-infra/platform/keda/helm-values.yaml

The backend ScaledObject also carries explicit scale-up damping:

minReplicaCount: 1
maxReplicaCount: 2
advanced:
  horizontalPodAutoscalerConfig:
    behavior:
      scaleUp:
        stabilizationWindowSeconds: 120
        policies:
          - type: Pods
            value: 1
            periodSeconds: 60

Source: lumie-infra/applications/lumie/backend/manifests/scaled-object.yaml

Why this failure mode happened

Commit	Change	Operational lesson
55ec012b	Fixed the Helm values key structure	Wrong nesting meant the chart ignored the intended resource settings
6650bd47	Raised metrics-server memory and probe delays	The metrics API server needed about 19 seconds to start
865c7ece	Raised operator probe thresholds and removed CPU limits	Slow API calls could make default probes flap even when the process was healthy
f4581593	Added backend scale-up stabilization	JVM warmup bursts should not immediately drive the deployment to max replicas

Diagnostic path

If the KEDA rollout is failing

kubectl get pods -n keda-system
kubectl describe pod -n keda-system -l app.kubernetes.io/name=keda-metrics-apiserver
kubectl describe pod -n keda-system -l app.kubernetes.io/name=keda-operator

Look for probe failures before assuming the scaler logic itself is broken. The April fixes were about startup and API-response timing, not about RabbitMQ or CPU trigger syntax.

If lumie-backend is over-scaling on warmup

kubectl get scaledobject,hpa -n lumie-backend
kubectl describe scaledobject -n lumie-backend lumie-backend
kubectl get deploy -n lumie-backend lumie-backend -o yaml | rg -n "cpu:|memory:"

The scaling trigger is still CPU utilization 70, and KEDA computes that against the pod CPU request. That denominator matters as much as the ScaledObject.

Current drift to verify before changing anything

The June 1, 2026 warmup fix explicitly raised the backend CPU request to 250m, but the current checked-in backend values now show:

resources:
  requests:
    cpu: 25m
    memory: 512Mi

Source: lumie-infra/applications/lumie/backend/common-values.yaml

So the current repo preserved the ScaledObject stabilization guard, but it does not preserve the original higher CPU-request denominator described in commit f4581593. If warmup-driven over-scaling returns, treat that as live contract drift rather than assuming the June fix is still fully present.

Fix

1. For control-plane rollout failures, keep the probe and memory settings in platform/keda/helm-values.yaml aligned with the current checked-in values.
2. For backend warmup spikes, preserve the scaleUp stabilization block in the backend ScaledObject.
3. Re-check the backend CPU request whenever you change CPU-trigger autoscaling. The request is part of the scaling contract.
4. Verify worker queue scalers only after KEDA itself is healthy. For example, the current report-svc contract is queue length 40 with minReplicaCount: 3 and maxReplicaCount: 5 in applications/lumie/worker/report-svc/manifests/scaled-object.yaml.

Prevention

• Keep KEDA chart value keys in the exact structure the chart expects. Early April failures happened because intended resource overrides were ignored.
• Treat rollout probes and autoscaling thresholds as one operational surface. A healthy ScaledObject is not useful if the metrics server never becomes Ready.
• When changing backend autoscaling, review both manifests/scaled-object.yaml and common-values.yaml together.

Verification

kubectl get applications.argoproj.io -n argocd keda
kubectl get pods -n keda-system
kubectl get scaledobject,hpa -n lumie-backend
kubectl get scaledobject -n lumie-worker report-svc -o yaml

Success means the KEDA control-plane pods are Ready, the backend ScaledObject has the stabilization block, and worker scalers render from Git without probe-related control-plane churn.

Related pages

• KEDA platform reference

Operational Detail

This guide covers two related but different failure classes:

Failure class	Main signal	Primary owner
KEDA control-plane rollout instability	KEDA pods fail probes or crash-loop.	platform/keda/helm-values.yaml.
Workload warmup over-scaling	Application replicas jump on transient startup CPU.	Workload ScaledObject plus CPU requests.

Do not tune workload scaling while the KEDA metrics API server is unhealthy. The workload symptoms may just be a side effect of an unstable scaling control plane.

The backend warmup case is especially sensitive because CPU utilization is a ratio. Lowering CPU requests makes the same JVM warmup burst look larger to the HPA/KEDA math.

Recovery Playbook

1. Check KEDA pods first. If they are not Ready, fix control-plane resources and probes before changing any ScaledObject.
2. Confirm the chart values are rendered under the keys the Helm chart actually consumes.
3. For backend over-scaling, compare the live HPA target, backend CPU request, and scaleUp behavior.
4. Reproduce with first-request or cold-start traffic rather than steady warmed traffic.
5. If the CPU request changed after the June fix, evaluate whether the stabilization window alone still protects the workload.

Verification Evidence

Capture:

• KEDA operator and metrics API server readiness,
• live resource requests and limits for KEDA pods,
• live backend ScaledObject and generated HPA behavior,
• backend CPU request used as the autoscaling denominator,
• replica count during warmup and after the stabilization window.

Anti-Patterns

• Increasing maxReplicaCount to hide warmup jitter.
• Changing CPU trigger thresholds without reviewing CPU requests.
• Tuning worker queue scalers while KEDA control-plane probes are failing.
• Removing probe delays after a chart upgrade without checking actual startup time.