RabbitMQ Stuck Message Defense In Depth

Symptom

Use this guide when a RabbitMQ work queue stops making progress even though consumers are still connected:

• users see grading or report work stall indefinitely
• messages_unacknowledged stays above 0
• the consumer pod is still running, so the broker does not requeue the message
• logs show Task exception was never retrieved, callback failures, or an exception before the consumer callback finishes

What The Current Repo Already Defends

The current worker and infra layout already bakes in the recovery lessons from this incident:

• lumie-worker/libs/common/mq.py centralizes the message lifecycle in universal_process(...)
• lumie-infra/platform/rabbitmq/base/topology/policies.yaml applies delivery-limit: 5 to the request and callback queues
• lumie-infra/platform/rabbitmq/base/podmonitor.yaml scrapes both /metrics and /metrics/per-object, which is required for per-queue stuck-message alerts
• lumie-infra/observability/prometheus/helm-values.yaml defines OMRQueueStuck and ReportQueueStuck
• lumie-worker/libs/common/observability.py explicitly avoids AioPikaInstrumentor because the old aio_pika wrapper crash happened before the consumer callback ran

Representative shape from lumie-worker/libs/common/mq.py:

async def universal_process(message, *, handler, callback, ...):
    try:
        ...
        await callback(payload)
        await message.ack()
    except Exception:
        if not message.processed:
            await message.nack(requeue=True)

Why This Failure Mode Happened

Two separate classes of failure can produce the same stuck queue symptom:

1. The consumer callback starts, but an exception escapes before ack(), nack(), or reject() runs.
2. A wrapper crashes before the worker callback starts at all, so the worker's own try/except never gets a chance to run.

The first incident was a swallowed async task exception after a callback-side network failure. The second incident was an OpenTelemetry aio_pika compatibility bug that crashed before process_message(...) ran. The shared lesson is that application code alone is not enough. The broker must bound redelivery, and monitoring must detect a queue that is stuck even when the worker callback never starts.

Diagnostic Path

Check the broker first

If the queue has connected consumers and non-zero messages_unacknowledged, the broker is usually waiting on a consumer that picked up work and never completed the lifecycle.

rabbitmqctl list_queues name messages consumers messages_unacknowledged

Compare queue state with processing metrics

The current Prometheus rules treat "unacked messages with zero processing rate" as the primary stuck signal. Inspect the alert expressions in lumie-infra/observability/prometheus/helm-values.yaml before assuming the issue is only a queue-depth spike.

Inspect worker logs for the class of failure

Look for:

• Task exception was never retrieved
• Background job callback failed
• Background job unexpected failure
• connection or DNS errors during callback delivery
• wrapper or instrumentation exceptions before the callback code path

If the error happens before universal_process(...) runs, focus on the code around queue.consume(process_message) and on any instrumentation or wrapper layer that runs before process_message(...).

Verify the service still uses the shared lifecycle primitive

The grading and report consumers both delegate to universal_process(...) from lumie-worker/libs/common/mq.py. If a new consumer reintroduced custom lifecycle code, the old unacked-message failure mode can return even if the shared libraries are fixed.

Fix

1. If a live consumer is holding the message and cannot recover, restart the affected pod so the AMQP channel closes and RabbitMQ requeues the message.
2. Keep each consumer on universal_process(...) so every return path ends with exactly one of ack, nack, or reject.
3. Keep retries bounded inside the worker and let the broker-owned delivery-limit move poison or unreachable-callback messages to DLQ.
4. Do not re-enable AioPikaInstrumentor unless the exact aio_pika version in use is verified against the wrapper. The current repo intentionally documents this as unsafe in lumie-worker/libs/common/observability.py.

Prevention

• Treat "exactly one lifecycle decision before return" as a hard invariant for every RabbitMQ consumer.
• Keep per-queue metrics enabled through /metrics/per-object; /metrics alone is not enough for stuck-queue detection.
• Keep queue policies in the Topology Operator CRs, not in app code, because delivery-limit belongs to the broker.
• Treat alerting as a separate defense layer from application exception handling. A queue can be stuck even when application code never runs.

Verification

cd Lumie
rg -n "universal_process|message\\.processed|AioPikaInstrumentor" \
  lumie-worker/libs/common lumie-worker/services

rg -n "delivery-limit|metrics/per-object|OMRQueueStuck|ReportQueueStuck" \
  lumie-infra/platform/rabbitmq/base \
  lumie-infra/observability/prometheus/helm-values.yaml

Success means the worker lifecycle still routes through universal_process(...), broker policies still bound redelivery, and per-queue alerts still exist outside the application process.

Source Incident Detail

The April 18, 2026 source writeup records a defense-in-depth response to stuck RabbitMQ messages. The key lesson was that no single layer owns the whole failure mode: consumer code, broker policy, metrics, and alerts all need a guard.

Detail	Value
Affected pattern	worker messages that are delivered but not completed correctly
Application defense	exactly one lifecycle decision before returning from the consumer
Broker defense	delivery-limit policy so redelivery cannot continue forever
Observability defense	per-object RabbitMQ metrics, not only aggregate /metrics
Alert defense	queue-specific stuck-message alerts outside the app process

The source record's invariant is:

every consumed message must end in exactly one ack, nack/requeue, reject/dead-letter, or explicit failure path

That invariant belongs in shared worker code, not in every service handler. The broker policy then caps what happens if the application layer still fails, and Prometheus catches the case where neither code nor broker-level behavior is producing forward progress.

When debugging a stuck queue, preserve all four views: consumer logs, RabbitMQ message state, policy/redelivery settings, and per-queue metrics.