keystone/RabbitMQ

RabbitMQ: Acks, Dead-Letter Queues, and Redelivery

A broker-managed queue makes a different promise than a log: it deletes what it has successfully handed off. That single decision is why RabbitMQ needs acks, redelivery, prefetch, and dead-letter queues in a way a log-based broker does not.

TLDR

Quick brief

Mental Model

The clearest way to place RabbitMQ is against Kafka, because they answer the same question, "how does a message get from a producer to a consumer reliably," with opposite retention policies. Kafka is a log: messages are appended, retained for a configured window, and consumers track their own position (an offset) into that log. Nothing is removed on read; a consumer can rewind, replay, or fall behind and catch up later, and ten different consumer groups can each read the same message at their own pace.

RabbitMQ is a queue: a message is handed to exactly one consumer (per queue, among however many are competing for it) and is deleted from the broker once that consumer acknowledges it. There is no offset, no replay, and no "read it again later." If a consumer nacks the message instead, or crashes before acking it, RabbitMQ's only options are to redeliver it, or, if it will never succeed, hand it somewhere else. That somewhere else is a dead-letter queue. A log doesn't need one, because a log never has to decide where an unprocessable message goes; it just sits at its offset forever, available to be skipped, replayed, or inspected. A queue has to decide, on every single nack, what happens next.

Ground-Up Explanation

Exchanges, bindings, and routing keys

A publisher never sends directly to a queue. It sends to an exchange with a routing key, and a binding between the exchange and a queue decides which messages land there. A direct exchange matches the routing key exactly; a topic exchange matches wildcard patterns against it; a fanout exchange ignores the routing key and copies the message to every bound queue; a headers exchange matches on message headers instead of the routing key. This indirection is what lets a single publish reach zero, one, or many queues without the publisher knowing which queues exist.

Queues and competing consumers

A queue is an ordered (with caveats, see below) buffer that one or more consumers pull from. When multiple consumers subscribe to the same queue, each message goes to exactly one of them, round-robin by default; this is the competing consumers pattern, and it is how RabbitMQ scales a single logical stream of work across many workers without any of them coordinating with each other.

A publisher sends to an exchange with a routing key. A binding routes matching messages into a queue. Two competing consumers pull from the same queue with manual ack. Acking removes the message. Nacking without requeue follows the dead-letter branch to a dead-letter exchange and its queue instead of being discarded.

Concept Deep Dive

ack, nack, reject, and requeue

A consumer that accepts manual acknowledgement (auto_ack=false) has three things it can do with a delivered message: ack it (done, delete it from the queue), nack or reject it with requeue=true (broker makes it available for redelivery, to this consumer or another one competing for the queue), or nack/reject it with requeue=false (drop it, or, if the queue is configured with a dead-letter target, route it there instead). This is the entire vocabulary redelivery is built from; everything else, retry counting, poison handling, backoff, is application logic layered on top of these three outcomes.

Prefetch (QoS) and head-of-line blocking

basic.qos(prefetch_count=N) caps how many unacked messages the broker will hand a consumer at once. A low prefetch limits how much work one crashed or stuck consumer can be holding onto, which is the point: without it, a fast-producing publisher can hand a slow consumer thousands of unacked messages, all of which get redelivered to someone else the moment that consumer dies. But the same bound has a cost. If one of those in-flight messages can never be acked, a low prefetch means that message occupies a disproportionate share, sometimes all, of the consumer's delivery slots, and nothing behind it in the queue gets delivered until it is resolved. This is head-of-line blocking: a queueing-theory term for exactly this shape of problem, one stuck item blocking everything queued behind it, that shows up anywhere work is dispatched from a bounded window rather than dispatched freely.

Dead-letter exchanges, TTL, and poison messages

A dead-letter exchange is a queue argument (x-dead-letter-exchange, optionally x-dead-letter-routing-key) that tells RabbitMQ where to route a message that is rejected without requeue, or that expires via a per-message or per-queue TTL, or that is dropped because the queue hit a max-length limit. Point it at another exchange bound to a dead-letter queue, and instead of vanishing, the message is preserved somewhere an operator or a replay job can find it. A poison message is one a consumer can never successfully process, a malformed payload, a business rule the consumer has no way to satisfy, a bug the consumer will hit on every retry. Without a dead-letter target, the only options for a poison message are "redeliver it forever" or "drop it silently"; neither is acceptable, which is why the dead-letter exchange exists specifically for this case.

RabbitMQ does not give a message a running redelivery count the way a database row might carry a version number; the redelivered flag is boolean, not a counter, until a message has actually been dead-lettered at least once, at which point an x-death header array starts recording cycles. To cap retries before the first dead-letter, an application-level counter is the standard approach: a custom header the consumer increments and forwards on each retry, most commonly by republishing to the tail of the queue and acking the original rather than relying on native requeue. That last detail matters: native nack(requeue=true) tends to make the message available again near the front of the queue, which is exactly what causes head-of-line blocking; republishing to the tail is a deliberate choice to trade that away.

Left panel: without a dead-letter target, a poison message is nacked with requeue, redelivered, fails again, and repeats forever, occupying a prefetch slot a good message could use. Right panel: with a retry-count guard and a dead-letter exchange, the same poison message is retried a bounded number of times, then nacked without requeue and routed to a dead-letter queue, leaving the main queue free to drain.

Ordering guarantees, and how requeue breaks them

A RabbitMQ queue with a single consumer and no requeues delivers in the order messages were published: first in, first out. Two things break that promise in practice. First, competing consumers: with more than one consumer on a queue, per-consumer ordering still holds, but the interleaving across consumers is not globally ordered, the same limitation Kafka solves by pinning ordering to a partition (one consumer per partition per group) rather than promising it across the whole topic. Second, and specific to queues rather than logs, requeue: a message that gets nacked and put back is, by definition, delivered again later than its original position, out of the order it was produced in. A message can be redelivered dozens of times while messages produced well after it complete first. This is not a bug, it is the direct consequence of "retry" meaning "try again later" in a system with no offset to rewind to; a log-based broker does not have this failure mode because replaying a message from an offset never removes or reorders anything else in the log.

Quorum queues versus classic queues

Classic queues are the original non-replicated queue type. Classic mirroring was deprecated and removed in RabbitMQ 4.x, so data-safety-sensitive queues should use quorum queues or streams instead. Quorum queues use Raft replication, support broker-side poison-message handling, and expose a delivery-count header that classic queues do not. That can replace an application retry counter for quorum queues. The trade-off is the usual one for consensus-backed replication: more nodes involved per write, more network round trips, and different throughput limits than a non-replicated classic queue.

RabbitMQ, Kafka, and SQS compared

RabbitMQKafkaAmazon SQS
Delivery modelBroker pushes to a consumer; message deleted on ackConsumer pulls by offset; message retained regardless of consumptionConsumer polls and receives a receipt handle; message hidden, not deleted, until deleted explicitly
OrderingFIFO per queue with a single consumer; broken by requeue or multiple competing consumersStrict per-partition order; no ordering across partitionsBest-effort on standard queues; strict per-message-group on FIFO queues
RetentionUntil consumed (acked); not designed for replayTime- or size-based retention independent of consumption; built for replayConfigurable retention (up to 14 days); a consumed-and-deleted message is gone
Redelivery / visibilityRedelivered immediately on nack/requeue or consumer disconnectConsumer re-reads from its last committed offset; broker has no per-message redelivery conceptMessage hidden for a visibility timeout after receipt; reappears automatically if not deleted in time
DLQ mechanismExplicit dead-letter exchange bound to a queue; triggered by nack(requeue=false), TTL, or length limitNo native DLQ; conventionally a separate topic an application routes to itselfA redrive policy names a target DLQ and a maxReceiveCount; SQS moves the message automatically once that receive count is exceeded
Scaling unitCompeting consumers on a queue; more consumers, more parallelism, no ordering unitPartitions; one consumer per partition per group is the parallelism ceilingNearly unlimited consumer count on standard queues; FIFO queues cap throughput per message group

SQS's visibility timeout is worth naming explicitly because it solves a similar failure shape with a different mechanism: a message becomes invisible to other consumers the moment it is received, and reappears automatically if the receiving consumer never confirms it by deleting it in time. RabbitMQ has no per-message visibility timeout for normal acknowledgements; an unacked message is redelivered when the consumer or channel goes away, or when the consumer explicitly rejects or nacks it with requeue. SQS's maxReceiveCount redrive policy does natively what this topic builds by hand for classic queues with a custom header: cap the number of redeliveries before the message is moved somewhere else.

Idempotency still applies

Redelivery, whatever the broker, is an at-least-once contract: the same message can be delivered and processed more than once, whether that's a RabbitMQ redelivery after a crash before ack, a Kafka consumer replaying from its last committed offset, or an SQS message reappearing after its visibility timeout expires with no delete. None of these mechanisms make a consumer's side effects idempotent on their own; that is still the consumer's job. See Delivery Semantics for the idempotent-consumer patterns that apply here exactly as they do to a log-based consumer.

Implementation Details

Production examples

RabbitMQ fits work that should be claimed by one worker: sending emails, resizing images, generating reports, dispatching webhooks, or running fraud-review jobs. A successful worker acks the message. A worker that cannot process the message decides whether to retry, requeue, or dead-letter it.

Dead-letter queues matter most when one malformed or impossible message can block useful work behind it. Examples: an email job with an invalid template id, an image job for a corrupt file, a webhook delivery with an unsupported destination, or a report job whose input references deleted data. Retrying forever hides the problem and wastes worker capacity; dead-lettering parks the failed message where an operator or replay tool can inspect it.

Prefetch is the production tuning knob. A low prefetch limits how much work a crashed worker can hold, but it makes head-of-line blocking sharper. A higher prefetch improves throughput for fast homogeneous jobs, but increases the amount of in-flight work that must be redelivered after a worker dies.

Lab Evidence

The runnable lab is labs/messaging/rabbitmq: one topology, lab.direct exchange bound to work.queue, lab.dlx bound to work.dlq, and a producer/consumer pair that switches between "no dead-letter target" and "dead-letter target + retry-count guard" purely through queue arguments set at declare time.

make break

No dead-letter target, prefetch=1. The poison message is nacked with requeue every time, capped at 20 redeliveries so the demo terminates. With only one unacked slot, nothing produced after the poison is ever delivered.

make test

Dead-letter target + a 3-attempt retry-count guard, prefetch=5. The poison is dead-lettered on the third attempt; the main queue drains to 0, the DLQ holds exactly 1 message, and all good messages complete.

reported metrics

Good messages processed, poison redelivery/attempt count, produced order vs completed order, good-message throughput, and queue depths pulled from the management API.

Measured: the same poison message, two outcomes

Real run on 2026-07-08, 30 messages, poison at position 5:

break (no DLQ, prefetch=1):
  good_processed=4/29  poison_attempts=20 (capped)  poison_resolved=false
  good_throughput=169.1 msg/s
  produced_order:  [1 2 3 4 5 6 7 ... 30]
  completed_order: [1 2 3 4]                      (nothing after the poison is ever delivered)
  work.queue messages=26  work.dlq messages=0

test (DLQ + retry-count guard, MAX_ATTEMPTS=3, prefetch=5):
  good_processed=29/29  poison_attempts=3  poison_resolved=true
  good_throughput=168.9 msg/s
  produced_order:  [1 2 3 4 5 6 7 ... 30]
  completed_order: [1 2 3 4 6 7 8 ... 30 5]       (5 completes dead last)
  ordering: poison produced at position 5, completed at position 30 (delta=25)
  work.queue messages=0   work.dlq messages=1

The prefetch difference is the point, not a side detail. At prefetch=1 in break, the broker has exactly one unacked slot and keeps refilling it with the just-requeued poison instead of the 25 good messages queued behind it; only the 4 messages produced before the poison (seq 1-4) are ever delivered, and work.queue ends the run holding 26 messages, the poison plus everything stuck behind it. That is head-of-line blocking measured, not described. At prefetch=5 in test, there is room for other messages to flow around the poison while it retries: all 29 good messages complete in strict produced order, and the poison itself resolves on schedule, three attempts, then dead-lettered, completing dead last (position 30 of 30) instead of never completing at all. The operational signal changes from "a redelivery counter climbing somewhere, unbounded" to "the DLQ depth is 1," a queue depth alert instead of a rate an operator has to notice.

Production Notes

Code Pointers

CodeWhy it matters
src/main.goIdempotent topology declaration, the manual ack/nack branches for break vs test mode, and the produced-vs-completed order tracking.
README.mdScenario explanation, measured break/test output, and RabbitMQ concept notes.
MakefileThe break/test scenario env overrides and the management-API queue-depth checks.
compose.yamlPinned rabbitmq:3.13-management, the healthcheck/--wait lifecycle, and the non-guest user needed for cross-container AMQP logins.

Further Reading & Watching