Skip to main content

CNPG Replica Join Cascading Failure

This page is an incident retrospective for the February 8, 2026 infra-db consolidation and scale-out failure. The incident mattered because three separate CNPG blockers appeared one after another, and recovery only worked when they were cleared in the right order.

What failed​

The intended change was to consolidate several small infrastructure databases into infra-db and scale the cluster from one instance to two. The first replica never joined successfully, and the failure cascaded:

  1. pg_basebackup could not allocate a replication slot.
  2. The failed join Job remained behind and blocked the operator from progressing to the next step.
  3. A stale PVC from the failed join attempt blocked the next replica bootstrap even after the primary was healthy again.

Confirmed causes​

  • max_replication_slots was still tuned for the old single-instance shape.
  • The failed join Job remained in place, and CNPG did not progress to the next reconciliation step until that artifact was removed.
  • The failed join also left a PVC behind in an initializing state, and CNPG kept trying to reuse it.

The recovery sequence was therefore strict:

  1. raise the replication-slot ceiling
  2. remove the failed join Job
  3. remove the stale PVC

Recovery​

The incident note's effective recovery sequence was:

kubectl delete job infra-db-2-join -n infra-db
kubectl delete pvc infra-db-2 -n infra-db

Those manual deletions only worked after the cluster definition had already been corrected to allow enough replication slots for the join flow.

What the current repo still proves​

The current checked-in manifest no longer reflects the exact incident topology, but it does preserve the lasting parameter correction:

spec:
  instances: 1
  postgresql:
    parameters:
      max_replication_slots: "8"

Source: lumie-infra/storage/infra-db/manifests/cluster.yaml

That is important drift to understand:

  • the incident happened while moving from instances: 1 to instances: 2
  • the current repo is back to instances: 1
  • the higher replication-slot value remained, which matches the lesson from the failure

Why this incident lasted longer than expected​

The first error message made the problem look like a single PostgreSQL parameter mistake. It was not. Once the join failed, operator cleanup and storage cleanup became separate blockers. Fixing only the parameter left the cluster stuck in a new way.

That is the durable operational lesson: in CNPG, a replica join failure can leave behind control-plane artifacts and storage artifacts that must both be cleared before reconciliation can continue.

Lasting lessons​

  • Review instances and replication parameters together. They are one change, not two.
  • Check for failed join Jobs before forcing broader failover or restart actions.
  • Treat stale PVCs as first-class blockers when using local-path-style storage.
  • Write the recovery order down before acting. This incident was order-sensitive.

Verification​

cd Lumie
sed -n '1,80p' lumie-infra/storage/infra-db/manifests/cluster.yaml
kubectl get clusters.postgresql.cnpg.io -n infra-db infra-db
kubectl get jobs -n infra-db
kubectl get pvc -n infra-db

Success means the repo still shows the raised replication-slot setting and the live cluster is not carrying leftover join Jobs or initializing PVCs from a failed scale-out attempt.