RabbitMQ On Kubernetes Operations

Symptom

Use this guide when RabbitMQ works inconsistently on Kubernetes:

• the Cluster Operator or Topology Operator fails to start
• Queue or Exchange CRs report healthy, but the broker is missing the objects
• Spring or worker consumers crash with 404 NOT_FOUND - no queue ...
• a broker restart exposes PRECONDITION_FAILED exchange-type mismatches
• app startup probes fail even though the service log looks healthy

What The Current Repo Expects

Today the RabbitMQ GitOps split is:

• broker runtime in lumie-infra/platform/rabbitmq/base/rabbitmq-cluster.yaml
• topology CRs in lumie-infra/platform/rabbitmq/base/topology/*.yaml
• dev and prod ownership patches in lumie-infra/platform/rabbitmq/overlays/dev/kustomization.yaml and lumie-infra/platform/rabbitmq/overlays/prod/kustomization.yaml

The base still carries a load_definitions bootstrap path and a rabbitmq-definitions ConfigMap, but both overlays now remove that path and repoint topology CRs to rabbitmqClusterReference.name. That means the repo already reflects the later operator-owned direction, even though the base files still show the older bootstrap mechanism.

lumie-infra/platform/rabbitmq/base/configmap.yaml also documents the intended split clearly: only vhosts, users, and permissions belong in definitions.json; exchanges, queues, bindings, and policies belong under base/topology/.

Why These Failures Happen

The incident that produced this guide was not one bug. It was a stack of operational traps:

• chart defaults assumed a different image or entrypoint than the one actually deployed
• the operator needed RBAC the chart version did not grant
• a ConfigMap existed, but RabbitMQ was never told to load it or mount it
• the same exchange was declared from more than one authority with different types
• a health probe targeted an endpoint that the running Spring Boot version did not expose

The common lesson is that RabbitMQ on Kubernetes is not one control plane. The chart, operator, broker config, topology CRs, and consuming apps all have to agree on who owns each layer.

Diagnostic Path

Start with the operator layer

If the operator pod is crashing, inspect the rendered Deployment before debugging the broker itself:

• image and command
• ClusterRole rules
• the chart version versus the image version actually deployed

An operator that never reaches a stable reconcile loop cannot prove anything about broker state.

Separate "object desired" from "object exists on the broker"

Ready=True on a Queue, Exchange, Binding, or Policy CR only proves the controller reconciled against some broker target. It does not prove the app-facing broker has the object you expect. If the app sees 404 NOT_FOUND - no queue ..., trust the broker and then inspect the CR's rabbitmqClusterReference.

Verify definitions loading as a full path

A ConfigMap alone does nothing. The historical bootstrap path required both:

• load_definitions = /etc/rabbitmq/definitions/definitions.json
• a volume mount that put the ConfigMap at that path

If either side is missing, the broker starts without the intended definitions even though the ConfigMap exists in the cluster.

Check for declaration drift across authorities

This is the classic PRECONDITION_FAILED case. If a queue or exchange is declared from app code and also from infra manifests, the broker treats any type mismatch as a hard contract violation. The current topology CRs define lumie.dlx as a topic exchange in lumie-infra/platform/rabbitmq/base/topology/exchanges.yaml, so any app-side declaration must match that contract.

Treat health-probe paths as runtime contracts

If a Spring app looks healthy in logs but Kubernetes never marks it ready, confirm the exact probe path is still exposed by the runtime. Historical /actuator/health/startup breakage came from assuming a Boot endpoint group existed by default.

Fix

1. Keep one authority per layer: broker runtime in RabbitmqCluster, topology in Topology Operator CRs, and app listeners as consumers only.
2. If a definitions bootstrap is still required, configure both load_definitions and the matching mount path. Do not stop at the ConfigMap.
3. If a Topology CR points at the wrong broker and the reference field is immutable, delete and recreate the CR only after confirming rabbitmq.com/keepAfterDeletion: "true" is present.
4. Resolve exchange-type mismatches by choosing one declared type and deleting or recreating the conflicting broker object. RabbitMQ will not mutate an existing exchange type in place.
5. If startup probes 404, either switch to a path the app already exposes or explicitly enable the endpoint group the probe expects.

Prevention

• Audit chart defaults any time the deployed image differs from the chart's assumed image.
• Keep topology reconciliation outside application startup. The current repo already moves queues, exchanges, bindings, and policies into Topology Operator CRs.
• Avoid duplicate declaration paths. If infra owns lumie.dlx, app code must not quietly redefine it with a different type.
• Treat Ready=True on external-target controllers as "the API call succeeded," not "the app-facing broker is correct."

Verification

cd Lumie
rg -n "load_definitions|rabbitmq-definitions|auth_backends\\.1 = internal" \
  lumie-infra/platform/rabbitmq/base/rabbitmq-cluster.yaml \
  lumie-infra/platform/rabbitmq/overlays/dev/kustomization.yaml \
  lumie-infra/platform/rabbitmq/overlays/prod/kustomization.yaml

rg -n "kind: (Queue|Exchange|Binding|Policy)|keepAfterDeletion|rabbitmqClusterReference" \
  lumie-infra/platform/rabbitmq/base/topology \
  lumie-infra/platform/rabbitmq/overlays/dev \
  lumie-infra/platform/rabbitmq/overlays/prod

Success means the repo still shows a clean broker-versus-topology ownership split, both overlays still remove the old definitions bootstrap, and topology objects remain declarative CRs rather than app-startup side effects.

Source Incident Detail

The January 30, 2026 source writeup captured the operational model for RabbitMQ on Kubernetes after several smaller failures. The important distinction is broker ownership versus topology ownership.

Detail	Value
Broker owner	RabbitMQ Cluster Operator
Topology owner	RabbitMQ Topology Operator CRs
Deprecated bootstrap	load_definitions for app topology and credentials
Main operational risk	treating controller readiness as proof that app-facing broker state is correct
Durable model	broker lifecycle and app topology converge through separate Kubernetes resources

The source record keeps these troubleshooting checks:

Check	Why
cluster CR status	proves the broker API object reconciled
pod readiness and logs	proves broker pods are actually running
topology CR status	proves queue/exchange/binding/policy API calls succeeded
live broker inspection	proves the app-facing object exists where clients expect it
application env	proves apps use generated credentials and the correct vhost/host

Do not collapse these layers. A Ready=True topology object means the operator's API call succeeded; it does not automatically prove publishers and consumers are using the expected credentials, vhost, or queue names.