Tempo Single-Binary OOM

Symptom

Use this guide when tempo-0 is repeatedly OOMKilled, especially on a roughly 30-minute loop, or when traces disappear after each restart because the pod stores data on emptyDir.

What the current repo expects

The checked-in Tempo values keep the deployment in single-binary mode with local ephemeral storage, a 512Mi memory limit, GOMEMLIMIT, and the metrics generator disabled:

resources:
  requests:
    cpu: 15m
    memory: 128Mi
  limits:
    memory: 512Mi
extraEnv:
  - name: GOMEMLIMIT
    value: "460MiB"
metricsGenerator:
  enabled: false

Source: lumie-infra/observability/tempo/helm-values.yaml

Why this failure mode happened

The inspected April 25 to April 27, 2026 commit sequence shows four separate causes that stacked on top of each other:

Commit	Change	What it proved
b9eb5593	105Mi limit to 256Mi	The earlier limit was below VPA guidance and too small even before deeper analysis
0afb7859	Disabled metricsGenerator	Failed remote writes let Tempo's WAL grow until the pod OOMed
5f6c0f19	Added GOMEMLIMIT	Go GC alone did not respect the container ceiling soon enough
4688ce0d	Raised limit to 512Mi and GOMEMLIMIT to 460MiB	Single-binary Tempo also needs headroom for file-backed and page-cache memory outside the Go heap

Diagnostic path

Check for ordinary OOMKilled restarts

kubectl describe pod -n tempo tempo-0
kubectl get pod -n tempo tempo-0 -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}{"\n"}'

If the last termination reason is OOMKilled, continue with the checks below.

Confirm the repo still guards the two known causes

• metricsGenerator.enabled: false should still be present in lumie-infra/observability/tempo/helm-values.yaml.
• GOMEMLIMIT should still be set below the memory limit in the same file.
• persistence.enabled: false and the emptyDir mount under /var/tempo explain why every restart also clears trace history.

Re-check the Prometheus side before re-enabling trace-derived metrics

The April 26 fix was tied to failed remote writes. The current Prometheus values still set:

enableRemoteWriteReceiver: false

Source: lumie-infra/observability/prometheus/helm-values.yaml

Do not re-enable Tempo metrics generation unless that contract changes first.

Fix

1. Keep the 512Mi Tempo memory limit unless the deployment model changes away from single-binary local storage.
2. Keep GOMEMLIMIT aligned to the container ceiling. The current repo uses 460MiB for a 512Mi limit.
3. Leave metricsGenerator.enabled: false until Prometheus is explicitly prepared for Tempo remote writes.
4. If you lower the memory limit or re-enable the metrics generator, treat that as a regression test point for this exact failure mode.

Prevention

• Single-binary Tempo sizing has to account for more than Go heap growth. The April 27 fix explicitly called out file-backed and page-cache memory.
• Because storage is emptyDir, every OOM is also a short-retention data-loss event.
• Any future chart change that touches resources, extraEnv, or metricsGenerator should be reviewed together, not independently.

Verification

kubectl get applications.argoproj.io -n argocd tempo
kubectl get pod -n tempo tempo-0
kubectl logs -n tempo statefulset/tempo --tail=200
kubectl get application -n argocd prometheus -o yaml | rg enableRemoteWriteReceiver

Success means the Tempo pod stays Running without fresh OOMKills, logs do not show failed metrics-generator remote writes, and the values still match the guarded single-binary contract described above.

Related pages

• Tempo operations reference
• Prometheus operations reference

Operational Detail

Tempo single-binary mode compresses several concerns into one pod: ingestion, query, local storage, compaction behavior, and optional metrics generation. A memory fix that only adjusts one knob can be temporary if the other knobs still push memory growth.

Knob	Why it matters
resources.limits.memory	Kubernetes kills the pod when total container memory crosses this boundary.
GOMEMLIMIT	Nudges Go GC before the container reaches the kill boundary.
metricsGenerator.enabled	Can create WAL and remote-write pressure when Prometheus is not accepting writes.
persistence.enabled: false	Makes every restart a trace-retention loss event.

OOM triage should record both the memory failure and the trace-retention impact. In this deployment shape, they are linked.

Recovery Playbook

1. Confirm the last termination reason is OOMKilled.
2. Check whether restarts line up with traffic bursts, metrics-generator errors, or steady periodic growth.
3. Verify the checked-in memory limit and GOMEMLIMIT are applied in the live StatefulSet.
4. Confirm metricsGenerator.enabled is still false before changing resource limits.
5. If traces are needed for the incident itself, export or inspect what is available before restarting the pod again.
6. Treat persistence enablement as a separate architecture change, not an emergency toggle.

Verification Evidence

Record:

• previous restart count and last OOM timestamp,
• live memory limit and GOMEMLIMIT,
• metrics-generator setting and Prometheus remote-write receiver setting,
• pod uptime after the fix,
• whether trace data loss occurred because of emptyDir.

Anti-Patterns

• Raising memory without checking failed remote-write or metrics-generator behavior.
• Setting GOMEMLIMIT equal to the Kubernetes limit with no headroom.
• Re-enabling metrics generation while Prometheus remote-write receiver is disabled.
• Assuming emptyDir is acceptable for all future tracing needs because it is acceptable for the current single-binary footprint.