RabbitMQ Dev Dual Credential Authority

Symptom

Use this guide when the dev messaging path breaks even though the Topology Operator says the queue objects are healthy:

• lumie-backend or a worker crashes with 404 NOT_FOUND - no queue ...
• Queue CRs show Ready=True, but rabbitmq-dev is missing the queue
• topology reconciles only after hitting a different broker than the one the apps use
• repointing the CRs to rabbitmq-dev switches the failure from missing queue to 401 Unauthorized

What The Current Repo Expects

The current dev overlay already captures the resolved design:

• lumie-infra/platform/rabbitmq/overlays/dev/kustomization.yaml removes load_definitions, deletes the dev copy of rabbitmq-topology-auth-vss, and rewrites all topology CRs to rabbitmqClusterReference.name: rabbitmq-dev
• lumie-infra/platform/rabbitmq/overlays/dev/app-users.yaml creates operator-managed User and Permission CRs for lumie, grading-svc, and report-svc
• lumie-infra/applications/lumie/backend/common-values.yaml
• lumie-infra/applications/lumie/worker/grading-svc/common-values.yaml
• lumie-infra/applications/lumie/worker/report-svc/common-values.yaml

Those app values now source RABBITMQ_USERNAME and RABBITMQ_PASSWORD from Topology Operator-generated secrets instead of from a shared Vault-rendered broker password.

Why This Failure Mode Happened

The incident had two linked problems:

1. Topology CRs were declaring against the wrong broker because a shared connectionSecret resolved to the prod/shared management URL instead of rabbitmq-dev.
2. The dev broker also had a custom load_definitions user authority, which suppressed the Cluster Operator's default user. After the CRs were repointed, the Topology Operator authenticated with a user the broker had never created and received 401 Unauthorized.

The unifying bug was dual credential authority. A broker managed by operators cannot safely keep a second, static authority for topology credentials in parallel.

Diagnostic Path

Separate CR health from broker reality

If a Queue CR is green but rabbitmq-dev is missing the queue, the question is not "why did reconcile fail?" but "which broker did reconcile succeed against?"

Inspect the topology reference target

The old failure mode used rabbitmqClusterReference.connectionSecret. The current fixed overlay replaces that reference with name: rabbitmq-dev. If the dev overlay drifts back to a shared connectionSecret, inspect the rendered secret URI before trusting the CR status.

Check whether the broker has more than one credential authority

This failure returns as 401 after repointing because load_definitions seeded a static user while the Cluster Operator expected its default user to exist. If load_definitions is active, RabbitMQ treats the definitions file as the user authority and does not create the operator default user on first boot.

Remember the PVC rule

Removing load_definitions is not enough on an already-used volume. RabbitMQ persists users on disk, and the operator default user is only created on the first boot of an empty database.

Fix

1. Make the RabbitMQ operators the sole credential authority for dev.
2. Repoint all topology CRs to rabbitmqClusterReference.name: rabbitmq-dev, which is the pattern already expressed in lumie-infra/platform/rabbitmq/overlays/dev/kustomization.yaml.
3. Use Topology Operator User and Permission CRs for app credentials, which is the pattern already expressed in lumie-infra/platform/rabbitmq/overlays/dev/app-users.yaml.
4. Recreate immutable topology CRs when the reference target changes. Do that only after confirming rabbitmq.com/keepAfterDeletion: "true" is present.
5. If the broker is moving from definitions-managed users to operator-managed users, plan for a fresh PVC or an equivalent empty-database boot. A plain restart will not create the missing default user.

Prevention

• Do not mix load_definitions user seeding with operator-managed broker credentials in dev.
• Do not use a shared Vault-rendered topology secret across dev and prod brokers.
• Keep app credentials sourced from per-service operator-generated secrets, not a single shared admin password.
• Treat a green external-target CR as "declared somewhere," not "declared on the broker my app uses."

Verification

cd Lumie
rg -n "load_definitions|rabbitmq-topology-auth-vss|rabbitmqClusterReference|name: rabbitmq-dev" \
  lumie-infra/platform/rabbitmq/overlays/dev/kustomization.yaml

rg -n "kind: User|kind: Permission|name: lumie|name: grading-svc|name: report-svc" \
  lumie-infra/platform/rabbitmq/overlays/dev/app-users.yaml

rg -n "RABBITMQ_USERNAME|RABBITMQ_PASSWORD|user-credentials" \
  lumie-infra/applications/lumie/backend/common-values.yaml \
  lumie-infra/applications/lumie/worker/grading-svc/common-values.yaml \
  lumie-infra/applications/lumie/worker/report-svc/common-values.yaml

Success means dev topology targets rabbitmq-dev by cluster name, the overlay no longer depends on the shared topology secret, and each consuming app reads RabbitMQ credentials from operator-generated secrets.

Source Incident Detail

The May 18, 2026 source incident was a dev-environment credential authority failure. Two systems were trying to own RabbitMQ credentials at once: static imported definitions and RabbitMQ Topology Operator resources.

Detail	Value
Environment	dev RabbitMQ cluster
Primary symptom	application credentials did not match the live broker state
Conflicting authorities	load_definitions bootstrap secrets and operator-generated User credentials
Impact	dev messaging instability; no production user impact recorded
Durable fix	make Topology Operator CRs and generated secrets the credential authority

The source diagnosis separated three planes:

Plane	What to check
broker bootstrap	whether load_definitions still imports users or permissions
topology CRs	whether User, Permission, queue, exchange, binding, and policy objects target the intended cluster
application env	whether apps read operator-generated *-user-credentials secrets

The lasting lesson is that RabbitMQ credentials should have one writer. If imported definitions and the Topology Operator both define users, Kubernetes desired state and broker runtime state will drift in ways that look like password bugs but are really authority bugs.