Skip to main content

OCI NLB Reserved IP Incident: May 24, 2026

This page is a troubleshooting retrospective for the second reserved-IP ingress failure on Lumie's OCI app NLB. Unlike the earlier May 5 outage, this investigation ended with a checked-in root cause and a repo-managed fix that now applies to both public NLBs.

Source paths​

  • todo/incidents/trouble/oci-nlb-reserved-ip-incident-2026-05-24.md
  • lumie-infra/provision/terraform/nlb_0214.tf
  • lumie-infra/provision/terraform/nlb_teleport_0213.tf
  • lumie-infra/provision/terraform/outputs.tf

Verified failure signature​

The decisive test kept the same NLB shape and healthy backends, but swapped only the public IP type:

Public IP on the NLBBackend healthExternal TCP/443Outcome
Ephemeral public IPhealthysucceededReal backend response reached Traefik
Reserved public IPhealthytimed outNo ingress reached the NLB edge

That ruled out the shared pieces of the path, including the backend pool, private subnet routing, and the cross-tenancy worker reachability used by nlb_0214.tf.

Root cause now encoded in Terraform​

The current repo records the fix directly on the reserved public IP resource:

resource "oci_core_public_ip" "nlb_public_ip_0214" {
  lifetime = "RESERVED"

  lifecycle {
    ignore_changes = [private_ip_id]
  }
}

Source: lumie-infra/provision/terraform/nlb_0214.tf

The file comment is explicit about the contract: OCI assigns the reserved public IP to the NLB's floating private IP, and later Terraform runs must not reset private_ip_id to null. If that association is stripped, the NLB can keep reporting healthy backends while the public edge times out on TCP/443.

The same safeguard is duplicated for the Teleport NLB in lumie-infra/provision/terraform/nlb_teleport_0213.tf, which is the clearest sign that this was not a one-off patch on the app ingress path.

Why the backends still looked healthy​

The backend set in nlb_0214.tf uses TCP health checks against port 443 on worker private IPs. Those checks validate the private NLB-to-worker path:

  • they do not prove that the reserved public IP is still attached to the NLB's floating private IP
  • they do not prove that Cloudflare-to-origin ingress can reach the edge

That is why this incident produced the confusing but repeatable state of "green backends, dead public ingress."

Recovery pattern that survived into the docs​

  • Restore service on a known-good path first, such as an ephemeral NLB IP or a temporary Cloudflare bypass, before attempting more NLB recreates.
  • If an NLB recreate partially succeeds and Terraform reports an object already exists, prefer importing the existing OCI object into state over deleting it from the control plane.

Representative shape:

terraform import oci_network_load_balancer_backend_set.nlb_backend_set_0214 <nlb_ocid>/<backend_set_name>

The original incident note used that recovery to reconcile a partially created backend set without throwing away working server-side resources.

Verification​

cd lumie-infra/provision/terraform
rg -n "ignore_changes = \\[private_ip_id\\]" nlb_0214.tf nlb_teleport_0213.tf
terraform output nlb_public_ip
terraform output nlb_teleport_public_ip
terraform output nlb_backend_targets
terraform plan

Success means both reserved-IP NLB resources still protect private_ip_id, the applied state exposes the expected public addresses, and terraform plan does not try to undo the OCI-managed IP association.