Argo CD Stale Cache Hard Resync

This page is an incident retrospective for the April 16-18, 2026 Argo CD stale-cache incident. The steady-state controller contract lives in the infrastructure GitOps Argo CD reference page.

What Failed

New commits were reaching the Git source, but Argo CD kept reporting no meaningful application change and did not auto-sync. The first change in 19be45e0 updated configs.cm.timeout.hard.reconciliation, but that alone did not fix the issue because the controller was still started with overriding CLI flags in applications/argocd/helm-values.yaml.

The current file still shows the split that mattered during the incident:

configs:
  cm:
    timeout.reconciliation: "60000"
    timeout.hard.reconciliation: "86400000"
controller:
  extraArgs:
    - --app-resync=60
    - --app-hard-resync=120

configs.cm documents ConfigMap intent, but controller.extraArgs is the active resync contract for the application controller.

Timeline

Commit	What changed	Why it mattered
19be45e0	Added timeout.hard.reconciliation to configs.cm	Partial fix only; the controller flags were still overriding resync behavior.
d131eedc	Changed controller.extraArgs from --app-resync=3600 and --app-hard-resync=0 to 60 and 120	This was the real stale-cache fix. It also doubled status and operation processors.
58ffd7c8	Raised repo-server memory from 512Mi to 1Gi	Faster resync immediately increased render pressure across 63 apps.
e11da2d1	Raised repo-server limit to 1500Mi and controller limit to 1280Mi	Restart-time recaching still caused OOM pressure after the cadence change.

The current repo has since increased repo-server headroom further to 3Gi, but the incident sequence explains why the extra capacity was needed.

Root Cause

Two separate contracts had to be true at once:

• Argo CD needed a non-zero hard re-fetch interval so manifest cache entries would be invalidated.
• The application controller needed matching CLI args, because those args overrode the ConfigMap values that looked authoritative at first glance.

The incident lasted longer than expected because the first fix changed the visible ConfigMap but not the controller process arguments that actually drove reconciliation.

Lasting Invariants

• Treat controller.extraArgs as authoritative whenever Argo CD resync timing is in doubt.
• Pair more aggressive resync settings with repo-server and controller memory reviews; stale-cache fixes can become OOM problems immediately.
• Verify live behavior from rendered args and pod stability, not only from Synced status.

Verification

cd lumie-infra
rg -n "timeout\\.hard\\.reconciliation|app-hard-resync|app-resync|status-processors|operation-processors" \
  applications/argocd/helm-values.yaml

kubectl get pods -n argocd
kubectl get deploy -n argocd argocd-repo-server -o yaml
kubectl get statefulset -n argocd argocd-application-controller -o yaml

Success means the repo still shows --app-resync=60 and --app-hard-resync=120, and the live repo-server plus controller workloads are stable without restart churn after reconciliation bursts.