Skip to main content

Server OOM Authentik Upgrade

This page is an incident retrospective for the January 13, 2026 master-node OOM event that occurred during an Authentik upgrade. The trigger was an Authentik memory spike, but the lasting lesson was about node role isolation and overall control-plane headroom.

What failed​

During the upgrade, Authentik gunicorn workers were repeatedly killed by cgroup OOM limits. That was only the first failure. The master node was already carrying too many non-control-plane workloads, so the extra pressure spread outward:

  • SSH became unavailable
  • the host entered emergency-mode recovery
  • K3s API calls timed out
  • Longhorn probes failed and containers restarted in sequence

The incident note described a single-master cluster with forty-six pods colocated on that node, including supporting tools such as Falco, VPA, Goldilocks, and Trivy.

Confirmed causes​

  • The new Authentik version needed more memory than the prior 512Mi container limit allowed.
  • The master node was not isolated for control-plane work, so one application's memory spike could starve the whole cluster.

Recovery and lasting fix​

The effective remediation had two layers:

  1. immediate host recovery and K3s restart
  2. structural reduction of control-plane contention

The structural changes were the durable ones:

  • raise Authentik memory limits so the upgrade path had enough room
  • stop running optional support workloads on the control-plane node
  • taint the master node so ordinary workloads are evicted instead of competing with the API server

What the current repo still proves​

The inspected repo no longer contains Authentik manifests, so it cannot prove the exact memory-limit change from the incident.

It does still show the later footprint-reduction direction that came out of the event:

  • lumie-infra/applications/kustomization.yaml comments out optional tools such as Headlamp and Coder for the small-node footprint plan
  • lumie-infra/observability/kustomization.yaml comments out Goldilocks and VPA for the same footprint reason
  • lumie-infra/security/kustomization.yaml comments out Trivy and Keycloak for the reduced-memory operating mode

Those files do not recreate the incident, but they do show the same operational principle: optional services must yield to platform headroom on small nodes.

Drift to note​

Because Authentik is no longer present in the inspected repo, this page is archival for the incident itself. The current repo only preserves the downstream scheduling and footprint lessons, not the original application manifest that triggered the OOM.

Lasting lessons​

  • Separate control-plane capacity from optional platform workloads as early as possible.
  • Treat large application upgrades as resource-profile changes, not only version bumps.
  • When one pod starts OOMing on a small control-plane node, assume secondary failures will appear in storage and cluster API paths next.
  • Keep reduced-footprint toggles obvious in Git so operational tradeoffs stay visible.

Verification​

cd Lumie
sed -n '1,60p' lumie-infra/applications/kustomization.yaml
sed -n '1,40p' lumie-infra/observability/kustomization.yaml
sed -n '1,40p' lumie-infra/security/kustomization.yaml

Success means the repo still exposes the reduced-footprint decisions that came out of the incident, even though the original Authentik deployment itself is no longer checked in.