Skip to main content

OCI NLB Reserved IP Incident: May 5, 2026

This page is a troubleshooting retrospective for the first reserved-IP outage on Lumie's OCI app ingress NLB. The original incident note restored service by switching away from the reserved-IP path. The current Terraform contract in the repo shows that the durable fix was later captured as a provider-side private_ip_id association guard, not as a permanent OCI-only workaround.

Source paths​

  • todo/incidents/trouble/oci-nlb-reserved-ip-incident-2026-05-05.md
  • lumie-infra/provision/terraform/nlb_0214.tf
  • lumie-infra/provision/terraform/outputs.tf

What failed on May 5​

The outage started immediately after the app NLB backend pool was changed during the worker-1 removal work in tenancy 0214.

CheckResultWhat it ruled out
External TCP/443 to the NLB reserved public IPtimed outPublic edge path was broken
OCI NLB lifecycle and backend healthACTIVE, backends OKControl plane and private-side health checks were still green
Requests to the NLB internal IP from inside the VCNsucceededThe NLB object and backend pool still worked privately
Requests directly to worker public IPssucceededTraefik and workloads were still serving

That combination matters: backends were healthy and the private-side path still worked, but the reserved public IP stopped delivering ingress.

What the repo now treats as the lasting fix​

The immediate recovery on May 5 was to move traffic off the broken reserved-IP path, first by pointing Cloudflare at a worker public IP and then by rebuilding the NLB with an ephemeral public IP. The later checked-in Terraform contract explains why that A/B test worked:

resource "oci_core_public_ip" "nlb_public_ip_0214" {
  lifetime = "RESERVED"

  lifecycle {
    ignore_changes = [private_ip_id]
  }
}

Source: lumie-infra/provision/terraform/nlb_0214.tf

The inline comments in that file say the NLB's reserved public IP must ignore private_ip_id because OCI assigns that field to the NLB's floating private IP, and later Terraform runs can otherwise clear the association and leave TCP/443 black-holed while backend health still looks normal.

Contract drift from the original incident note​

The original May 5 note framed the root cause as an OCI reserved-IP data-plane glitch that would need OCI Support. The current repo no longer treats that as the source-of-truth explanation:

  • nlb_0214.tf records a verified May 24, 2026 fix for Terraform-managed reserved IPs.
  • The same lifecycle guard is also present on the dedicated Teleport NLB in nlb_teleport_0213.tf.
  • The fuller root-cause writeup is captured in OCI NLB Reserved IP Incident: May 24, 2026.

This page therefore preserves the May 5 failure signature and recovery path, but not the original "OCI-only glitch" conclusion as the lasting contract.

Lessons that survived​

  • Treat "backends healthy, public ingress timing out" as an edge-association problem until proven otherwise.
  • A reserved-IP versus ephemeral-IP A/B test is a fast way to separate backend problems from public-IP attachment problems.
  • Keep a documented bypass path for Cloudflare-to-origin cutover so service can be restored before the root cause is fully confirmed.

Verification​

cd lumie-infra/provision/terraform
rg -n "ignore_changes = \\[private_ip_id\\]" nlb_0214.tf
terraform output nlb_public_ip
terraform output nlb_id

Success means the repo still carries the reserved-IP lifecycle guard on the app NLB, and the applied Terraform state still exposes the public IP and NLB OCID operators use during live verification.