Messaging Β· 05

RabbitMQ Smart Broker

RabbitMQ is the broker with opinions: it routes, tracks, retries, delays and dead-letters for you. This page explains the AMQP 0-9-1 model and every exchange type, how classic, quorum and stream queues differ and what RabbitMQ 4.0 changed, how to publish and consume without losing messages, how prefetch shapes throughput, how dead lettering and delayed retry work, how to run and monitor a cluster, how to use it from Spring AMQP, and when Kafka is the better tool, finishing with a lab you can run in minutes.

13
Chapters
4.x
RabbitMQ version
3
Queue types
1
Local lab
01
Chapter One - Why It Exists

The Smart Broker and the AMQP 0-9-1 Model

RabbitMQ is a message broker: a server that accepts messages from producers, decides where they should go, holds them safely, and hands them to consumers. Page 01 introduced the generic vocabulary (message, queue, ack, redelivery, dead letter). RabbitMQ is the best-known example of the smart broker, simple consumer design: the broker owns routing, per-message acknowledgment, redelivery, expiry and priority, so each consumer can stay small. Kafka (pages 02 to 04) takes the opposite stance and keeps the broker simple.

The AMQP 0-9-1 model: producers publish to exchanges, never to queues
Producer Exchange routes, stores nothing Queue A stores messages Queue B stores messages Consumer 1 Consumer 2 publish binding A binding B Everything lives inside one virtual host The exchange applies each binding (queue + pattern) to the routing key and copies the message into every queue that matches.

The core protocol is AMQP 0-9-1, and its central idea is a layer of indirection: a producer never writes to a queue. It publishes to an exchange together with a routing key, and the exchange uses its bindings to decide which queues receive a copy. That split is what lets you add a new consumer (new queue plus binding) without touching the producer.

TermWhat it isRemember
Virtual host (vhost)A namespace: its own exchanges, queues, bindings, users permissions and limitsDefault is /. One vhost per environment or tenant
ConnectionA long-lived TCP (optionally TLS) connection from a clientExpensive to open; keep few
ChannelA lightweight virtual connection multiplexed over one connectionWhere publishing, consuming, acks and basic.qos happen
ExchangeReceives published messages and routes themTypes: direct, fanout, topic, headers (chapter 03)
QueueOrdered buffer that stores messages until consumers ack themTypes: classic, quorum, stream (chapter 04)
BindingA rule linking an exchange to a queue (or another exchange), often with a patternRouting logic lives here
Routing keyA string the publisher attaches; matched against binding keysMeaning depends on the exchange type
MessageBody (bytes) plus properties (headers, delivery mode, expiration, priority, correlation id...)RabbitMQ never looks at the body
Protocols and versions

RabbitMQ speaks several protocols. AMQP 0-9-1 is the classic one and the one every Spring AMQP and Java client example on this page uses. Since RabbitMQ 4.0, AMQP 1.0 is supported natively by the core broker (no plugin): the same port 5672 accepts both, and the client chooses with a protocol header. Streams have their own binary protocol (a plugin), and MQTT and STOMP are available through plugins. A message published with one protocol can be consumed with another, with a conversion step when needed.

This page targets RabbitMQ 4.x. Where a feature needs a specific minor version it is stated (for example 4.3 introduced strict priorities and delayed retry for quorum queues). The hands-on chapter runs the rabbitmq:4-management image, which was 4.4.0 when this page was written.

⚠ Publishing to a queue "directly"
  • There is no such thing in AMQP 0-9-1. Publishing to the default exchange (empty name) with the queue name as the routing key looks direct, but an exchange is still in the path (chapter 03). Thinking in exchanges from day one keeps your design extensible.
Key insight: An exchange is a post office sorting room, not a mailbox. The sender hands over a letter with an address (routing key). The sorting room looks at its rules (bindings) and drops a copy into each matching mailbox (queue). Mailboxes hold letters until the owner collects and signs for them (ack).
02
Chapter Two - Connections

Connections, Channels and Virtual Hosts

A connection is a TCP connection with an AMQP handshake on top (and TLS if enabled). Opening one costs several network round trips plus authentication, so applications open it once and keep it for their lifetime. Inside it you open channels: independent, lightweight sessions multiplexed over the one connection. Publishing, consuming, acking and basic.qos all happen on a channel.

One TCP connection carries many channels; vhosts isolate resources on the broker
Application instance (one connection) channel 1: publishing (confirms on) channel 2: consumer for orders.created channel 3: consumer for billing.events AMQP vhost /shop exchanges: orders.x, orders.dlx queues: orders.created, orders.created.dlq user shop-app: configure/write/read on /shop only vhost /billing exchanges: billing.x queues: billing.events invisible from /shop: same names allowed in both A connection is bound to exactly one vhost, chosen at connect time. Channels are cheap and are where the work happens; connections are heavy.
TopicGuidanceWhy
ConnectionsLong-lived. Typically one connection per application process (often one for publishers and a separate one for consumers)A burst of connection opens is expensive and shows up as connection churn in monitoring
ChannelsOpen one per thread or task that needs it; never share a channel between threads for publishingChannels are not thread-safe in the Java client; sharing causes interleaved frames and errors
Channel countBound it. Servers negotiate channel_max (2047 in the 4.4.0 image used for the lab)Every channel uses memory on the broker and the client; leaked channels are a classic slow leak
Publishers vs consumersUse separate connections for publishing and consumingWhen the broker applies flow control or blocks a publishing connection (chapter 08), TCP back pressure would otherwise also stall your consumers' acks
HeartbeatsKeep them on. The suggested default is 60 secondsThey detect dead peers and keep idle connections alive through load balancers
ErrorsA channel-level error (for example acking an unknown delivery tag) closes that channel; a connection-level error closes everythingHandle channel closure and re-open; do not assume a channel lives forever
Virtual hostsOne vhost per environment, tenant or bounded context; give each application a user with permissions on its vhost onlyIsolation of names, permissions, limits and policies in a single cluster
Recovery is your job (or your framework's)

The broker never "remembers" a client. If the connection drops, every channel, consumer, unacknowledged delivery and non-durable declaration of that client is gone. Unacked messages go back to their queue and will be redelivered (with the redelivered flag set), possibly to a different consumer. Good clients re-open the connection, recreate the topology (declarations are idempotent) and resume consuming. Spring AMQP and the Java client do this automatically by default; chapter 09 shows where the knobs are.

Declare, do not assume
  • Queue, exchange and binding declarations are idempotent as long as the arguments match. Many applications declare their topology at startup (Spring does it through Queue, Exchange and Binding beans). If a queue already exists with different arguments, the declaration fails with PRECONDITION_FAILED and the channel closes. Queue arguments cannot be changed in place, which is why policies (chapter 07) are preferred for settings that may evolve.
03
Chapter Three - Routing

Exchanges: Direct, Fanout, Topic, Headers and the Default Exchange

The exchange type decides how the routing key (or headers) is matched against bindings. Four types ship with the broker; the default exchange is a special direct exchange. Each diagram below shows one message arriving and which queues get a copy. Gray dashed paths are bindings that do not match.

Direct exchange

Routes to the queues whose binding key equals the routing key. Use it for classification by a fixed label: log severity, job type, a tenant name.

Direct exchange: exact key match
Publisher key = error direct exchange queue: logs.error queue: logs.warn queue: logs.info key = error warn info Exact match: the message goes to every queue whose binding key equals the routing key (here only logs.error). Several queues may use the same key, then each gets a copy. Same-named keys on two bindings to one queue deliver once.
Fanout exchange

Copies every message to all bound queues and ignores the routing key. Use it for broadcast: notify audit, email and search-indexing independently, each with its own queue and consumers.

Fanout exchange: copy to every bound queue
Publisher key ignored fanout exchange queue: audit queue: email queue: search-index Broadcast: every bound queue gets its own copy. The routing key is not looked at. This is how RabbitMQ does publish/subscribe: one queue per subscriber.
Topic exchange

Routing keys are dot-separated words (order.eu.created). Binding keys are patterns: * substitutes exactly one word, # substitutes zero or more words. This is the workhorse for event routing because consumers choose their slice with a pattern.

Topic exchange: patterns with * and #
Publisher order.eu.created topic exchange Q1 binding: order.*.created Q2 binding: order.# Q3 binding: *.us.* Q4 binding: #.shipped * matches exactly one word; # matches zero or more words. Words are separated by dots. order.eu.created reaches Q1 and Q2. It does not reach Q3 (middle word is eu, not us) or Q4 (does not end in shipped).
Routing keyorder.*.createdorder.#*.us.*#.shipped#
order.eu.createdmatchmatchnonomatch
order.us.shippednomatchmatchmatchmatch
order.createdno (needs a middle word)matchnonomatch
ordernomatch (# is zero words)nonomatch
shippednononomatch (# is zero words)match
Design topic keys on purpose
  • Pick a stable word order such as domain.entity.event or region.service.severity, put the most selective word first, and document it. A binding of # makes a topic exchange behave like fanout for that queue.
Headers exchange

Ignores the routing key and matches on the message headers instead. Each binding carries a map of header values plus x-match: all (default) requires every listed header to match, any requires at least one. It is useful when routing depends on several attributes at once and encoding them into a dotted key would be awkward. In practice topic exchanges cover most needs and are easier to read and reason about.

Headers exchange: match on message headers
Publisher format=pdf region=eu headers exchange Q1 all: format=pdf, type=report Q2 any: format=pdf, type=report Q3 all: region=eu Matching uses message headers, not the routing key. x-match=all needs every listed header to match, x-match=any needs at least one. Q1 is skipped (type is missing); Q2 matches on format; Q3 matches on region.
Default exchange

Every vhost has a pre-declared direct exchange with the empty name. When a queue is declared it is bound to it automatically with the queue name as key, so publishing with exchange "" and routing key tasks lands in queue tasks.

Default exchange: key equals queue name
Publisher exchange "" , key = tasks default exchange name is the empty string queue: tasks queue: emails queue: reports key = queue name A pre-declared direct exchange. Every queue is bound to it automatically, using its own name as the binding key. You cannot bind or unbind queues on it. Handy for tutorials and tasks; real systems usually publish to a named exchange.
TypeMatches onTypical useCost to evaluate
directRouting key equals binding keyJob types, severities, per-tenant queuesVery cheap
fanoutNothing (all bindings)Broadcast, pub/sub to several servicesCheap, but copies grow with queue count
topicPattern on the routing key (*, #)Event routing by domain, region, severityCheap to moderate
headersHeader values with x-match all or anyMulti-attribute routingModerate
default ("")Routing key equals queue nameSending to one known queueVery cheap
consistent-hash (plugin)Hash of routing key or a headerSharding for per-key ordering (chapter 07)Cheap
⚠ Unroutable messages vanish silently
  • If no binding matches, the exchange drops the message and the publisher is told nothing by default. Symptom: the publisher reports success, the queue stays empty. Fix: publish with the mandatory flag (chapter 05), configure an alternate exchange on the exchange, or at least monitor the unroutable-dropped metric.
04
Chapter Four - Queue Types

Classic, Quorum and Stream Queues, and What Changed in 4.0

A queue's type is chosen when it is declared (x-queue-type, or QueueBuilder.quorum() in Spring) and fixes how messages are stored and replicated. RabbitMQ 4.x has three: classic, quorum and stream.

ClassicQuorumStream
Storage modelQueue: message removed once ackedQueue (replicated): removed once ackedAppend-only log: messages stay until retention removes them
ReplicationNone since 4.0 (mirroring removed)Yes: Raft consensus across membersYes: replicated log
Survives node lossNo (the queue lives on one node)Yes while a majority of members is upYes while enough replicas are up
Consume semanticsDestructive (ack deletes)Destructive (ack deletes)Non-destructive: read again by offset or timestamp
DurabilityDurable or transient; persistence per messageAlways durable, always persistedAlways persistent
TTL and length limitsYesYes (but not reject-publish-dlx)Retention by age or size instead
PrioritiesYes (x-max-priority, use few levels)Yes, strict, 0 to 31, from 4.3No
Poison message limitNoYes: delivery-limit, default 20 since 4.0Not applicable
Typical useTransient, non-critical, single-node workDefault for important work queuesFan-out to many readers, replay, large backlogs
Which queue type? A short decision path
New queue needed Many readers, replay of old messages, or a huge backlog? STREAM append-only log yes Must survive the loss of a node and never lose acked data? QUORUM replicated, Raft no yes Short-lived, single node, low value? CLASSIC not replicated no yes Default for production work queues: quorum. Classic is the exception, not the default.
What changed in RabbitMQ 4.0 and later
  • Classic queue mirroring was removed in 4.0 (deprecated since 2021). Classic queues are now a non-replicated type. If you used mirrored classic queues for high availability, migrate to quorum queues (or streams). The docs describe a blue-green strategy for moving from 3.13.
  • **Quorum queues got a default delivery-limit of 20 in 4.0.** A message returned to the queue more than 20 times is dropped, or dead-lettered if a DLX is configured. In 3.13 there was no limit. Set x-delivery-limit=-1 (or the delivery-limit policy key) to restore unlimited redelivery, which the docs advise only for rare compatibility cases. Since 4.3 the limit counts failed deliveries (x-delivery-count) and plain returns via nack do not count.
  • Native AMQP 1.0 support in the core broker since 4.0, alongside AMQP 0-9-1 on the same port.
  • Classic queue version 1 storage was removed in 4.0; version 2 is the only implementation and existing queues are converted at node start.
  • Khepri, a Raft-based metadata store, is fully supported from 4.0 and is the recommended replacement for Mnesia. In the 4.4.0 image used on this page the khepri_db feature flag was enabled.
  • 4.3 added delayed retry with linear back-off for quorum queues and strict 0 to 31 message priorities. It also made non-durable, non-exclusive classic queues impossible to declare unless you set deprecated_features.permit.transient_nonexcl_queues = true.
  • Global QoS (a prefetch limit shared by all consumers of a channel) is a deprecated feature. In the 4.4.0 image rabbitmqctl list_deprecated_features showed global_qos as denied_by_default. Use per-consumer prefetch.
⚠ Upgrading a mirrored-queue cluster and expecting HA
  • Classic queue mirroring no longer exists in 4.x, so a mirroring policy no longer gives you replication (rabbitmq-diagnostics check_if_cluster_has_classic_queue_mirroring_policy reports such policies). Queues that were mirrored are plain single-node classic queues: lose the node and you lose the queue. Migrate to quorum queues before relying on the upgraded cluster.
Quorum queues: the Raft basics

A quorum queue has a leader and followers, together called members (3 by default; x-quorum-initial-group-size changes it, and an odd number is recommended). Publishing goes to the leader, which appends the message to its replicated log and sends it to followers. A message is committed when a majority of members (N/2 + 1) have written it, and only then is the publisher confirmed. If the leader dies, the followers elect a new leader from the members that have the committed data, with no data loss for confirmed messages.

Quorum queue with 3 members: commit means a majority has the message
Quorum queue orders.created, 3 members Node 1 LEADER log: m1 m2 m3 m4 Node 2 follower log: m1 m2 m3 m4 Node 3 follower (slow) log: m1 m2 Publisher 1 publish to the leader 2 leader appends to its log and replicates to followers 3 leader + node 2 = 2 of 3 = majority message is committed 4 only now: basic.ack to publisher (node 3 catches up in the background)
Cluster size / queue membersMajority neededMember failures tolerated
110
321
532
4 (even)31 (no better than 3)

Consequences you should know: writes need a majority, so a minority partition cannot accept publishes to that queue; quorum queues always persist to disk and never keep the whole queue in memory; they do not support non-durable or exclusive queues (use single active consumer instead of exclusivity); and consumer prefetch is capped at 2,000 even if you ask for unlimited.

Declare a quorum queue with a delivery limit and a dead letter exchange (policy, preferred)
$ rabbitmqctl set_policy qq-defaults "^orders\." \ '{"delivery-limit": 5, "dead-letter-exchange": "orders.dlx"}' \ --apply-to quorum_queues --priority 10

A policy applies settings to every queue whose name matches the pattern and can be changed at any time without redeclaring. Arguments set in code (x-delivery-limit, x-dead-letter-exchange) are baked in at declaration and cannot be changed without deleting the queue, which is why the docs recommend policies for anything that may change.

Streams in one paragraph

A stream is an append-only, replicated log inside RabbitMQ. Consumers read non-destructively from an offset, timestamp or the beginning, so many consumers can read the same data and replay it, much like a Kafka partition (page 02). Retention is by x-max-age or x-max-length-bytes, evaluated per segment file (default segment 500,000,000 bytes). Streams can be used from AMQP 0-9-1 clients (acks required, per-consumer prefetch required, no global QoS) or, for best throughput, through the dedicated stream protocol and its clients. Super streams partition a stream for parallelism. Streams complement queues; they do not replace them.

Choosing quickly
  • Work queue where each message is a task to complete: quorum. Events many independent services must read, or that you may need to replay: stream (or Kafka, chapter 11). Scratch queue you can lose, single node, low volume: classic. When unsure, pick quorum.
05
Chapter Five - Publishing

Reliable Publishing: Durability, Confirms and the Mandatory Flag

"The broker accepted my publish" is not one fact, it is three: the message was routed to at least one queue, the queue(s) stored it safely, and the publisher learned about both. RabbitMQ gives you a switch for each, and skipping one is how messages get lost without an error anywhere.

Durability and persistence are two switches
SettingApplies toWhat it protectsWhere set
Durable exchange or queueThe definitionThe queue or exchange still exists after a broker restartdurable=true on declare; QueueBuilder.durable(...)
Persistent message (delivery_mode=2)The messageThe message is written to disk, not only held in memoryMessage property; Spring AMQP sets persistent by default
Quorum queue or streamThe queue typeAlways durable and always persisted, so both switches are impliedx-queue-type

For a classic queue you need both: a durable queue and persistent messages. A persistent message in a non-durable queue vanishes at restart because the queue itself is deleted; a transient message in a durable queue is discarded when the node boots. Quorum queues and streams remove the foot-gun by always doing both.

Publisher confirms

A publisher puts its channel in confirm mode once. The broker then answers every publish, asynchronously, with basic.ack or basic.nack, identified by a sequence number the client keeps. Confirms are the only reliable way to know the broker took responsibility for a message. (AMQP transactions also exist but confirms are the recommended mechanism.)

Confirm and return flow for a routable and an unroutable message
Publisher Exchange Queue (quorum) A. Routable message basic.publish (mandatory) route to queue replicated to a majority and written to disk basic.ack (after the queue accepted it) B. Unroutable message published with mandatory = true basic.publish (mandatory) no binding matches basic.return (reply 312 NO_ROUTE) comes first then basic.ack: the exchange is done with it
SituationWhat the broker sendsMeaning
Routed to one or more queuesbasic.ack once all target queues accepted itPersistent message in durable queue: written to disk. Quorum queue: a quorum of replicas accepted it
Not routed, mandatory=false (default)basic.ackThe exchange verified there is no queue and dropped the message. It looks like success
Not routed, mandatory=truebasic.return, then basic.ackPublisher is told the message was unroutable
Internal error in the queue processbasic.nackThe broker refuses responsibility; publisher may republish
No answer yet-No guarantee how soon a confirm arrives; do not block forever

Two details matter. A confirm says the message is safe in the queue, not that anyone consumed it; consumer acknowledgments are a separate mechanism (chapter 06). And for persistent messages the broker batches fsync calls, so a confirm for a single message on an idle queue can take a moment: that is the price of durability, not a fault.

Publish strategyHowTrade-off
One at a timePublish, wait for the confirm, repeatSimplest; throughput limited by round-trip latency
BatchPublish N, wait for all confirmsMuch faster; on failure you may have to republish the whole batch
AsynchronousPublish continuously, handle ack/nack in a callback keyed by sequence number or correlation idBest throughput; most code (Spring's CorrelationData does the bookkeeping)
The mandatory flag and alternate exchanges

With mandatory=true, an unroutable message is returned to the publisher through basic.return (Spring: a ReturnsCallback) before the confirm. Alternatively an alternate exchange (alternate-exchange policy key or argument) receives everything the primary exchange could not route, which lets the broker catch it into a "parking" queue with no publisher involvement.

⚠ Confirms on, but no retry plan
  • A nack or a missing confirm means "unknown", not "lost": the message may in fact be stored. Republishing then creates a duplicate. The consumer side must be idempotent (page 01): set a stable message-id on each message and deduplicate on it, or make the operation naturally idempotent.
Publisher sends, node crashes before the confirm arrives
β–Ά
Client times out. The message is either stored (and will be delivered) or not. Republish is safe only with idempotent consumers
Exchange has no matching binding, mandatory=false
β–Ά
Message dropped, publisher sees an ack. Symptom: "published OK" but queue empty. Fix: mandatory plus a returns handler, or an alternate exchange
Queue is classic and non-durable, node restarts
β–Ά
Queue and messages gone. Fix: durable queue plus persistent messages, or use quorum
Memory alarm raised
β–Ά
Publishing connections are blocked; publishes stall instead of failing (chapter 08). Fix: add consumers, fix the backlog, raise capacity
06
Chapter Six - Consuming

Acknowledgments, Requeueing and Prefetch

RabbitMQ pushes messages to consumers registered with basic.consume (the alternative, basic.get, polls one message per call and is meant for tests and rare cases). The delivery guarantee on the consuming side comes from acknowledgments.

ModeHow the broker behavesDelivery guaranteeUse when
Automatic ack (autoAck=true, Spring AcknowledgeMode.NONE)Treats a message as handled the moment it is written to the socketAt most once. A crash after delivery loses the messageLosing messages is acceptable (metrics, cache refresh). Rarely right
Manual ack (Spring AUTO or MANUAL)Keeps the message until the consumer sends basic.ack; requeues it if the channel closesAt least once: duplicates possible, so handlers must be idempotentAlmost everything
Consumer responseProtocol methodEffect
Successbasic.ack (optionally multiple=true to ack all tags up to this one)Message removed from the queue
Failure, try againbasic.nack or basic.reject with requeue=trueMessage goes back to the queue (position may differ) and is redelivered with redelivered=true
Failure, give upbasic.nack or basic.reject with requeue=falseDead-lettered if a DLX is configured, otherwise discarded
Consumer or connection dies(none)Unacked messages are requeued automatically

basic.nack can reject several deliveries at once (the multiple flag), basic.reject is single-message only. Acking the same delivery tag twice, or on another channel, closes the channel with PRECONDITION_FAILED - unknown delivery tag.

⚠ Requeue forever
  • A handler that fails on a bad message and always nacks with requeue=true creates a hot loop: the broker redelivers immediately, CPU burns, logs fill, and the message is never removed. Symptom: the same stack trace repeating several times a second and one message with a growing redelivery count. Fix: dead-letter after a bounded number of attempts (quorum delivery-limit, Spring retry plus a DLX) and treat clearly invalid messages as permanent failures straight away.
Prefetch (basic.qos)

Without a limit the broker pushes the entire queue at a consumer as fast as the network allows. Prefetch (basic.qos) caps the number of unacknowledged deliveries a consumer may hold. When it is reached the broker stops sending until an ack frees a slot. It is the single most influential consumer setting for both throughput and fairness.

Prefetch window and fairness
Prefetch = 3: at most 3 unacknowledged deliveries per consumer m1 m2 m3 m4 m5 m6 m7 m8 m1 to m3 are in flight on the consumer (unacked). m4 to m8 stay ready in the queue. ack(m1) lets the broker send m4. Consumer (processing m1, m2, m3 buffered) Fairness with two consumers and 6 slow messages at the head Unlimited prefetch slow consumer A: holds m1 to m6 fast consumer B: idle, queue is empty Prefetch = 1 slow consumer A: holds m1 fast consumer B: works through m2 to m6 A big window is faster per consumer but lets one slow consumer hoard work. A tiny window is fair but pays a round trip per message.
PrefetchThroughputFairness and memoryNotes
1Lowest: one round trip per messageBest: work is spread by actual speedUse for slow, heavy jobs or when strict fairness matters
10 to 50Good for moderately fast handlersGoodA sensible starting range for mixed work
100 to 300The docs say values in this range usually give optimal throughputOne slow consumer can hold up to this many messagesHigher values hit diminishing returns
0 (unlimited)Highest delivery rateWorst: consumer memory grows, one consumer can hoard the backlogQuorum queues still cap it at 2,000
  • RabbitMQ applies prefetch_count per consumer on a channel, which differs from the AMQP 0-9-1 text (shared across the channel). A global (channel-wide) limit exists but is a deprecated feature and slower; avoid it.
  • Spring AMQP's prefetchCount defaults to 250 (since version 2.0) and is ignored when the ack mode is NONE. Set it explicitly per listener container based on handler duration.
  • basic.qos has no effect on basic.get.
  • Example sizing (illustrative arithmetic): if a handler takes 5 ms and the network round trip is 1 ms, a prefetch of 1 leaves the consumer idle for roughly 1 ms out of every 6 ms. A prefetch of a few messages hides the round trip entirely. For handlers that take seconds, the round trip is irrelevant and a small prefetch (1 to 5) keeps work balanced.
A rule that works
  • Prefetch roughly equals the number of messages you can process during one network round trip, plus a small buffer, and never more than you are willing to redeliver if the consumer dies. Measure with the unacked count (chapter 08).
The delivery acknowledgment timeout

If a consumer holds a delivery without acking it, the broker eventually closes the channel. The default timeout is 30 minutes (consumer_timeout, configurable globally or per queue with a policy). As of RabbitMQ 4.3 the timeout is enforced for quorum queues only, because stuck deliveries bloat the Raft log. Symptom of hitting it: the consumer logs a channel error mentioning a delivery acknowledgement timeout, and the message is redelivered. Fix: speed up the handler, raise the timeout deliberately, or hand long work to a separate worker and ack when it is accepted.

⚠ Auto-ack "because it is faster"
  • It is faster precisely because it removes the safety net: the broker forgets the message as soon as it is on the wire, so a consumer crash, a deserialization error or a restart loses it with no trace. Use manual acknowledgment and tune prefetch for speed instead.
07
Chapter Seven - Failure and Flow Control

Dead Letters, TTL, Retry, Priorities, Single Active Consumer and Ordering

Dead lettering

A dead letter exchange (DLX) is an ordinary exchange to which a queue republishes messages that die. A message is dead-lettered in four cases:

Reason (x-death reason)Trigger
rejectedConsumer sent basic.reject or basic.nack with requeue=false (or an AMQP 1.0 rejected outcome)
expiredThe per-message or per-queue TTL elapsed
maxlenThe queue exceeded its length limit and dropped the message (drop-head)
delivery_limitA quorum queue message was returned more than delivery-limit times

If a whole queue expires (x-expires), its messages are not dead-lettered. Configure the DLX with a policy (dead-letter-exchange, optional dead-letter-routing-key); if no routing key is set the message keeps its original key(s). If the DLX does not exist when needed, messages are silently dropped, so create it first. Each dead-lettering **adds an x-death header** and the first and last events are summarised in x-first-death-* and x-last-death-* headers. The real output below came from the lab in chapter 10: a message rejected from queue orders.new and read from the dead letter queue.

Headers on a dead-lettered message (output from the lab, formatted for reading)
x-death: [ { queue: "orders.new", reason: "rejected", count: 1, exchange: "orders.topic", routing-keys: ["order.eu.created"], time: 1793468854 } ] x-first-death-queue: orders.new x-first-death-reason: rejected x-last-death-queue: orders.new x-last-death-reason: rejected x-delivery-count: 1 x-acquired-count: 1
  • The x-death array is ordered most recent first and is compressed by the {queue, reason} pair: the count field says how many times it happened. That is exactly the counter a retry loop needs.
  • By default, dead-lettered messages are republished without publisher confirms, so in a cluster a dead-letter hop can lose messages if the target is unavailable. Quorum queues offer dead-letter-strategy=at-least-once, which retains the message until the DLX target confirmed it. It needs overflow=reject-publish and a configured DLX, and costs more memory and CPU. The default is at-most-once.
  • RabbitMQ detects a dead-letter cycle and drops the message if nothing in the cycle was a rejection. A TTL-driven retry loop (below) is fine because the first hop is a rejection.
Every work queue gets a DLX
  • A queue without a dead letter configuration silently discards rejected and expired messages. For quorum queues the docs recommend a dead letter configuration on all of them so a poison message is parked, not lost.
TTL and expiration
KindSet withBehaviour
Per-queue message TTLpolicy key message-ttl or argument x-message-ttl (milliseconds)Messages older than this expire. The server guarantees expired messages are not delivered
Per-message TTLmessage property expiration (a string of milliseconds)If the queue also has a TTL, the lower value wins
Queue TTLpolicy key expires or argument x-expiresThe queue is deleted after being unused (no consumers) for that long; messages are not dead-lettered
TTL of 0either of the first twoMessage expires on arrival unless a consumer can take it immediately
⚠ Per-message TTL does not expire from the middle
  • Expiry is only acted on when a message reaches the head of the queue. A short-TTL message queued behind a long-TTL one stays in the queue (and in the queue statistics) until the one in front is consumed or expires. If you need predictable expiry, use a queue-level TTL so all messages share one value.
Delayed retry with TTL and a DLX

Retrying immediately usually hits the same transient failure again. The classic RabbitMQ pattern delays the retry with two queues: the work queue dead-letters rejected messages into a retry queue that has a message TTL and a DLX pointing back at the work queue. After the TTL the message expires and returns for another attempt. The x-death count tells the consumer how many times that has happened.

Delayed retry loop built from a TTL queue and dead-lettering
orders.created work queue (quorum) retry.x (exchange) orders.retry message-ttl 30000 orders.parking-lot after max attempts 1 reject, requeue = false dead-letter 2 waits 30 s, then expires count over limit 3 expiry dead-letters via the default exchange (key = orders.created): back to the work queue, now with an x-death header whose count grows with every round trip. 4 the consumer reads x-death; after N attempts it publishes to the parking lot instead of retrying. Quorum queues on 4.3+ can do the delay natively (delayed retry), with no helper queue.
Policy-free sketch: a retry queue that returns messages to the work queue after 3 seconds (verified in the lab)
$ rabbitmqadmin queues declare --name work --type quorum --durable true $ rabbitmqadmin queues declare --name work.retry --type quorum --durable true \ --arguments '{"x-message-ttl":3000,"x-dead-letter-exchange":"","x-dead-letter-routing-key":"work"}' $ rabbitmqadmin publish message -k work.retry -m job-1 # after about 3 seconds the message is in "work", with x-death reason "expired"

This pattern gives a fixed delay per retry queue; for growing delays you create several retry queues (5 s, 30 s, 5 min) and route by attempt. From RabbitMQ 4.3, quorum queues have native delayed retry with linear back-off: set delayed-retry-type (disabled, all, failed or returned), delayed-retry-min and optionally delayed-retry-max in milliseconds, and the delay for a returned message becomes min(min * delivery_count, max). The docs recommend it over a dead-letter cycle because it does not rewrite the message and cannot lose it. It is the better choice on 4.3 or later; the TTL+DLX loop remains the portable pattern for older versions and classic queues.

Priority queues

A priority queue delivers higher-priority messages first. Classic queues need x-max-priority (1 to 255, but the docs strongly recommend a small single-digit number such as 2 to 4, because each level costs resources). Quorum queues from 4.3 always support strict priorities 0 to 31 (messages without a priority count as 4) and ignore x-max-priority; before 4.3 they mapped priorities onto just two buckets. Priorities interact with prefetch: messages already delivered to a consumer cannot be overtaken, so a large prefetch weakens the effect. Use a priority queue only when one queue must mix urgencies; separate queues per urgency are usually simpler and avoid starvation questions.

Single active consumer and ordering

Competing consumers on one queue give parallelism but no ordering: two consumers can finish messages out of order, and redelivery reorders them further. When order matters, enable single active consumer (SAC): declare the queue with x-single-active-consumer=true (Spring: QueueBuilder...singleActiveConsumer()). Many consumers may register, but the queue delivers only to one at a time; if it dies, another takes over. This gives ordered processing plus automatic failover. (SAC on streams needs a native stream client.)

One active consumer means no parallelism, which is the cost of ordering. To keep per-key order while scaling, shard the stream of messages across several queues, each with its own SAC. The consistent hash exchange plugin (rabbitmq_consistent_hash_exchange, exchange type x-consistent-hash) does the sharding: the binding key is a weight (a number as a string), and the exchange hashes the routing key (or a header named by the hash-header exchange argument) to pick one bound queue, so all messages with the same key go to the same queue.

Consistent hash exchange plus single active consumer: parallel across keys, ordered within a key
Publisher routing key = customerId x-consistent-hash exchange q.shard-0 keys: c1, c4, c9 SAC consumer q.shard-1 keys: c2, c5 SAC consumer q.shard-2 keys: c3, c7, c8 SAC consumer The hash of the routing key picks the queue, so all messages for one customer land in one queue. One consumer (single active consumer) per queue keeps their order; the three queues run in parallel.
  • The hash distributes routing keys, not messages: one very hot key still lands in one queue.
  • Use one binding per queue. Changing the number of queues changes the mapping, so a key may move to another queue during a resize and ordering is only guaranteed once the old queue has drained.
  • This is the RabbitMQ analogue of Kafka's key-to-partition rule (page 02), implemented with queues instead of partitions.
08
Chapter Eight - Operations

Clustering, Flow Control, Alarms and Monitoring

Clusters and high availability

A cluster is several RabbitMQ nodes sharing metadata (vhosts, users, exchanges, queue definitions, bindings, policies). Message data lives in queues, and only replicated queue types give high availability: quorum queues and streams. Plan on an odd number of nodes (3 or 5). Clustering is designed for a low-latency LAN. The docs say a cluster can be stretched across availability zones or data centers because quorum queues and Khepri use Raft, but that needs deliberate planning around latency and node placement; for moving messages between distant sites, Shovel and Federation are the usual tools.

Three-node cluster with quorum queue leaders spread across nodes
Node 1 orders.created LEADER payments follower metadata store (Raft) Node 2 orders.created follower payments LEADER metadata store (Raft) Node 3 orders.created follower payments follower metadata store (Raft) Each quorum queue has its own Raft group; leaders are spread over the nodes. The cluster metadata (vhosts, users, queue definitions, bindings) is replicated by the Raft-based metadata store (Khepri). Clients may connect to any node of the cluster.
  • Metadata store: Mnesia was the original; Khepri (Raft-based) is fully supported from 4.0 and recommended. Both are internal; what you see is how the cluster behaves during a network partition.
  • Leader placement: keep leaders balanced, otherwise one node does most of the work. Quorum queue leader placement and member reconciliation are configurable.
  • Node loss: with 3 members a quorum queue keeps working after one failure. Replace the node and rebalance members; the CLI offers rabbitmq-queues add_member, grow, shrink, rebalance and quorum_status.
  • Rolling upgrades: upgrade one node at a time and check rabbitmq-diagnostics check_if_node_is_quorum_critical before stopping a node, so you do not take down a majority.
  • Health checks: rabbitmq-diagnostics check_running for liveness, check_alarms for resource alarms, check_for_quorum_queues_without_an_elected_leader for queue health. (check_local_alarms is deprecated and a no-op in 4.4.)
Flow control, memory and disk alarms

Publishers can outrun queues, disks and replication. RabbitMQ defends itself in two layers:

MechanismTriggerEffect on clientsYou see
Credit-based flow controlA queue, channel or connection falls behind a fast publisherThe publishing connection is slowed to the speed the system can sustainConnection state flow in the UI and rabbitmqctl. No configuration, no errors, publisher just looks slower
Memory alarmNode memory crosses vm_memory_high_watermark (relative 0.6 of available RAM by default)All publishing connections are blocked cluster-wide until the alarm clears. Consumers continue, which is how the backlog drainsConnection state blocked in the UI and rabbitmqctl
Disk alarmFree disk space falls below disk_free_limit (default 50 MB; raise it, for example to an absolute value in the GB range or relative to RAM)Publishers blocked, as aboveSame, plus a disk alarm in the UI

The memory watermark is a throttle point, not a hard cap: a node can use more than the limit. Leave headroom (RAM for the OS and file cache) and do not run a node at the edge. When an alarm is on, look for the cause, not the switch: a queue with no consumers, consumers that are slow or crashed, unacked messages piling up, or a queue type that holds too much in memory.

⚠ Blocked publishers also stop your own consumers' acks
  • If an application publishes and consumes on the same connection, a blocked connection can stall consumer acknowledgments, which keeps the backlog alive, which keeps the alarm on. Use separate connections for publishing and consuming (chapter 02). In Spring AMQP, RabbitTemplate.setUsePublisherConnection(true) makes the template use a connection separate from the listener containers.
Monitoring

Enable the Prometheus plugin (rabbitmq_prometheus, port 15692, /metrics) and scrape it; the management UI (rabbitmq_management, port 15672) is for interactive inspection, and its built-in metric collection is a deprecated feature. In the rabbitmq:4-management image used for the lab both plugins were already enabled.

Metric (Prometheus name)What it tells youAlert idea
rabbitmq_queue_messages_readyBacklog waiting for consumers (queue depth)Sustained growth, or older than your SLO
rabbitmq_queue_messages_unackedDelivered but not ackedLarge and flat: stuck consumers or prefetch too high
rabbitmq_queue_messages_published_total, ..._delivered_ack_total, ..._redelivered_totalPublish, consume and redelivery rates (use rate())Publish rate above consume rate; redelivery spike
rabbitmq_connections, rabbitmq_channels and the ..._opened_total countersConnection and channel counts and churnRapid opens: connections not reused
rabbitmq_resident_memory_limit_bytes vs rabbitmq_process_resident_memory_bytesDistance to the memory alarmAbove about 80 percent of the limit
rabbitmq_disk_space_available_bytes vs rabbitmq_disk_space_available_limit_bytesDistance to the disk alarmWithin a safety margin of the limit
rabbitmq_detailed_queue_messages_ready (from /metrics/detailed?family=queue_coarse_metrics)Per-queue depth, with vhost and queue labelsPer-queue alerts; mind the label cardinality with many queues

The default /metrics endpoint aggregates across queues to stay cheap. Per-queue series come from the detailed endpoint, where you choose metric families and optionally a vhost; avoid scraping every family from a broker with thousands of queues. Alongside metrics, log the redelivered flag and the x-death count in your consumers, and watch the dead letter queue depth: a non-empty DLQ is an alert, not a graveyard.

Production checklist highlights
  • Three or five nodes; quorum queues for important data; a DLX on every work queue; per-vhost users, no guest; TLS; bounded prefetch; separate publish and consume connections; alarms and queue depth monitored; definitions exported for disaster recovery. The official production checklist page covers the rest.
09
Chapter Nine - Spring

RabbitMQ with Spring AMQP

Spring AMQP (spring-amqp and spring-rabbit) and its Spring Boot starter spring-boot-starter-amqp wrap the Java client with a template for sending, listener containers for receiving, and declarative topology. The code below was compiled against Spring AMQP 4.1.2 and Spring Boot 4.1.2 (the current releases on Maven Central when this page was written) and run against the rabbitmq:4-management container from chapter 10. Property names are Boot 4.1 names; Boot 3.x uses max-attempts where Boot 4 uses max-retries.

The Spring application wired to the topology in the code below
OrderPublisher RabbitTemplate orders.x topic exchange orders.created quorum, delivery-limit 5 order.eu.created @RabbitListener container + retry deliver orders.dlx fanout orders.created.dlq quorum rejected Retries run inside the listener container. When they are exhausted, the recoverer rejects the message without requeue and the queue's own dead letter exchange moves it to the DLQ.
Topology as beans

Queue, Exchange and Binding beans are declared on the broker by RabbitAdmin when the first connection opens. Use the builders: QueueBuilder.durable(name).quorum() sets the queue type, deliveryLimit, deadLetterExchange and singleActiveConsumer set the matching arguments, ExchangeBuilder creates exchanges, and BindingBuilder.bind(queue).to(exchange).with(key) binds. A MessageConverter bean (here JacksonJsonMessageConverter, the Jackson 3 based converter in Spring AMQP 4) is picked up by the auto-configured RabbitTemplate and by listeners.

Topology.java
package demo; import org.springframework.amqp.core.Binding; import org.springframework.amqp.core.BindingBuilder; import org.springframework.amqp.core.ExchangeBuilder; import org.springframework.amqp.core.FanoutExchange; import org.springframework.amqp.core.Queue; import org.springframework.amqp.core.QueueBuilder; import org.springframework.amqp.core.TopicExchange; import org.springframework.amqp.support.converter.JacksonJsonMessageConverter; import org.springframework.amqp.support.converter.MessageConverter; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class Topology { public static final String EXCHANGE = "orders.x"; public static final String QUEUE = "orders.created"; public static final String DLX = "orders.dlx"; public static final String DLQ = "orders.created.dlq"; @Bean TopicExchange ordersExchange() { return ExchangeBuilder.topicExchange(EXCHANGE).durable(true).build(); } @Bean Queue ordersCreated() { return QueueBuilder.durable(QUEUE) .quorum() // x-queue-type=quorum .deliveryLimit(5) // x-delivery-limit (4.x default is 20) .deadLetterExchange(DLX) // x-dead-letter-exchange .build(); } @Bean Binding ordersCreatedBinding(Queue ordersCreated, TopicExchange ordersExchange) { return BindingBuilder.bind(ordersCreated).to(ordersExchange).with("order.*.created"); } @Bean FanoutExchange deadLetterExchange() { return ExchangeBuilder.fanoutExchange(DLX).durable(true).build(); } @Bean Queue ordersCreatedDlq() { return QueueBuilder.durable(DLQ).quorum().build(); } @Bean Binding dlqBinding(Queue ordersCreatedDlq, FanoutExchange deadLetterExchange) { return BindingBuilder.bind(ordersCreatedDlq).to(deadLetterExchange); } // JSON on the wire; Spring Boot applies a MessageConverter bean to RabbitTemplate and listeners. @Bean MessageConverter jsonConverter() { return new JacksonJsonMessageConverter(); } }
Arguments in code are frozen
  • The x-delivery-limit and x-dead-letter-exchange above are queue arguments: they are fixed when the queue is first declared and a later change fails with PRECONDITION_FAILED because the arguments no longer match. The RabbitMQ docs recommend policies for settings that may change, and the Queue bean for what must not.
Publishing with confirms and returns

With publisher-confirm-type: correlated each publish may carry a CorrelationData, whose getFuture() completes with a Confirm (isAck(), getReason()). With publisher-returns: true and mandatory set on the template, an unroutable message comes back as a ReturnedMessage, which is stored in the same CorrelationData. Waiting on the future per message is simple but slow; for volume, publish a batch and then wait for all futures, or register callbacks.

OrderPublisher.java
package demo; import java.util.UUID; import java.util.concurrent.TimeUnit; import org.springframework.amqp.rabbit.connection.CorrelationData; import org.springframework.amqp.rabbit.core.RabbitTemplate; import org.springframework.stereotype.Service; @Service public class OrderPublisher { public record OrderCreated(String orderId, String region, long totalCents) { } private final RabbitTemplate template; public OrderPublisher(RabbitTemplate template) { this.template = template; } /** Publishes and waits for the broker confirm. Needs publisher-confirm-type=correlated. */ public void publish(OrderCreated event) throws Exception { CorrelationData cd = new CorrelationData(UUID.randomUUID().toString()); template.convertAndSend(Topology.EXCHANGE, "order." + event.region() + ".created", event, cd); CorrelationData.Confirm confirm = cd.getFuture().get(5, TimeUnit.SECONDS); if (!confirm.isAck()) { throw new IllegalStateException("Broker nacked " + cd.getId() + ": " + confirm.getReason()); } if (cd.getReturned() != null) { // needs publisher-returns=true and mandatory=true throw new IllegalStateException("Unroutable: " + cd.getReturned().getReplyText()); } } }
Listening, acknowledging and failing

@RabbitListener turns a method into a consumer inside a SimpleMessageListenerContainer (the default container type; a direct container is available via spring.rabbitmq.listener.type=direct). The acknowledge mode is container-level:

Spring modeBehaviourNotes
AUTO (default)Container acks after the listener returns; if it throws, the message is rejectedWhether a rejected message is requeued is defaultRequeueRejected, default true. Throw AmqpRejectAndDontRequeueException to dead-letter explicitly
MANUALYour code calls channel.basicAck/basicNack (inject Channel and the AmqpHeaders.DELIVERY_TAG header)You must ack every message or the channel stalls at the prefetch limit
NONEBroker auto-ack: no acks are sentAt most once. Prefetch is ignored
OrderListeners.java
package demo; import java.io.IOException; import com.rabbitmq.client.Channel; import org.springframework.amqp.AmqpRejectAndDontRequeueException; import org.springframework.amqp.rabbit.annotation.RabbitListener; import org.springframework.amqp.support.AmqpHeaders; import org.springframework.messaging.handler.annotation.Header; import org.springframework.stereotype.Component; import demo.OrderPublisher.OrderCreated; @Component public class OrderListeners { /** * Container ack mode AUTO (the default): ack on normal return, reject on exception. * With retry enabled, the exception is retried in-process first. */ @RabbitListener(queues = Topology.QUEUE, concurrency = "2-4") public void handle(OrderCreated event) { if (event.totalCents() < 0) { // A bad message will never succeed: reject it without requeue so it goes to the DLX. throw new AmqpRejectAndDontRequeueException("negative total for " + event.orderId()); } process(event); // must be idempotent: delivery is at-least-once } /** * Manual mode: set spring.rabbitmq.listener.simple.acknowledge-mode=manual, * and register this method in place of handle(...). */ public void handleManually(OrderCreated event, Channel channel, @Header(AmqpHeaders.DELIVERY_TAG) long tag) throws IOException { try { process(event); channel.basicAck(tag, false); // multiple=false: just this delivery } catch (TransientException e) { channel.basicNack(tag, false, true); // requeue=true: try again later } catch (Exception e) { channel.basicNack(tag, false, false); // requeue=false: dead-letter it } } private void process(OrderCreated event) { /* business logic */ } static class TransientException extends RuntimeException { } }

Some errors are never worth retrying. Spring's default ConditionalRejectingErrorHandler rejects without requeue the failures that cannot succeed on redelivery, such as MessageConversionException, MethodArgumentNotValidException and MethodArgumentTypeMismatchException; an x-death header on such a message makes it discard instead of loop. Other exceptions follow defaultRequeueRejected.

Retry and dead letter queue configuration

Spring Boot can wrap the listener in stateless, in-process retry with back-off. When retries are exhausted the default recoverer, RejectAndDontRequeueRecoverer, rejects the message without requeue, so the queue's DLX (set on the queue in the topology) routes it to the DLQ. The configuration below gives 1 initial attempt plus 3 retries.

application.yml
spring: rabbitmq: host: localhost port: 5672 virtual-host: / # username and password: supply via SPRING_RABBITMQ_USERNAME / SPRING_RABBITMQ_PASSWORD publisher-confirm-type: correlated # async confirms, with CorrelationData publisher-returns: true # deliver basic.return to a ReturnsCallback template: mandatory: true # unroutable messages are returned, not dropped listener: simple: acknowledge-mode: auto # container acks on return, rejects on exception prefetch: 20 # per consumer; Spring's own default is 250 concurrency: 2 max-concurrency: 4 default-requeue-rejected: false # failures are dead-lettered, not requeued forever retry: enabled: true # in-process, stateless retry max-retries: 3 # retries after the first attempt (4 calls in total) initial-interval: 500ms multiplier: 2.0 max-interval: 10s

I ran this configuration against the lab broker with a listener that always throws. The handler was invoked 4 times (the first attempt plus max-retries: 3 retries) with growing pauses, then Spring logged Retries exhausted from RejectAndDontRequeueRecoverer, and the message appeared in orders.created.dlq carrying an x-death header with reason: rejected.

To keep the failure reason with the message, register a RepublishMessageRecoverer bean. Spring Boot picks a MessageRecoverer bean up automatically. It republishes the failed message to an exchange you choose with the extra headers x-exception-message, x-exception-stacktrace, x-original-exchange and x-original-routingKey, then acks the original. In the lab the message arrived in orders.dlq with x-exception-message: boom and x-original-routingKey: order.eu.created.

RecovererConfig.java
package demo; import org.springframework.amqp.rabbit.core.RabbitTemplate; import org.springframework.amqp.rabbit.retry.MessageRecoverer; import org.springframework.amqp.rabbit.retry.RepublishMessageRecoverer; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @Configuration public class RecovererConfig { /** * Optional. Without this bean, Spring Boot's listener retry ends with * RejectAndDontRequeueRecoverer, so the queue's own DLX takes over. * This one republishes the failed message with x-exception-message, * x-exception-stacktrace, x-original-exchange and x-original-routingKey * headers, then acks the original. */ @Bean MessageRecoverer republishRecoverer(RabbitTemplate template) { return new RepublishMessageRecoverer(template, Topology.DLX, "failed"); } }
⚠ In-process retry blocks the consumer
  • While retrying with sleeps, the listener thread is busy and its prefetched messages wait. Long back-offs at scale mean long stalls. Keep in-process retry for quick, transient glitches (a dropped connection, a lock timeout) and use a delayed-retry queue (chapter 07) or quorum delayed retry (4.3+) for failures that need minutes.
  • Concurrency: concurrency and max-concurrency set the number of consumer threads per listener. With single active consumer only one of them receives messages.
  • Observability: name your connections (spring.rabbitmq has a connection name strategy) so they are identifiable in the management UI.
  • Testing: use a real broker with Testcontainers (RabbitMQContainer) rather than mocking the template; most bugs live in the topology and the acknowledgment flow.
10
Chapter Ten - Hands-On

A Local Lab with Docker: Route, Reject, Retry, Observe

This lab runs RabbitMQ in Docker, builds a small topology with the CLI tools that ship inside the image, and shows routing, dead lettering, TTL retry, policies and metrics. The outputs below were captured from a real run on rabbitmq:4-management, which was RabbitMQ 4.4.0 that day; tags move, so your version string may differ. Some outputs are shortened and say so.

Step 1: start the broker
Start the container and check the version
$ docker run -d --name rabbit -p 5672:5672 -p 15672:15672 -p 15692:15692 rabbitmq:4-management $ docker exec rabbit rabbitmq-diagnostics -q ping Ping succeeded $ docker exec rabbit rabbitmqctl version 4.4.0 $ docker exec rabbit rabbitmq-plugins list -e -m rabbitmq_management rabbitmq_management_agent rabbitmq_prometheus rabbitmq_web_dispatch

Port 5672 is AMQP, 15672 the management UI and HTTP API (open http://localhost:15672, login guest / guest, which is acceptable only for a throwaway lab), and 15692 the Prometheus endpoint. The tool we use is **rabbitmqadmin**, the v2 command-line tool (the project is called rabbitmqadmin-ng; the binary is rabbitmqadmin, version 2.35.0 in this image). It talks to the management HTTP API. rabbitmqctl and rabbitmq-diagnostics talk to the node directly.

Step 2: declare a topology and route messages
Exchanges, quorum queues with a DLX and a delivery limit, bindings
$ A="docker exec rabbit rabbitmqadmin --non-interactive" $ $A exchanges declare --name orders.topic --type topic --durable true $ $A exchanges declare --name orders.dlx --type fanout --durable true $ $A queues declare --name orders.dlq --type quorum --durable true $ $A queues declare --name orders.new --type quorum --durable true \ --arguments '{"x-dead-letter-exchange":"orders.dlx","x-delivery-limit":5}' $ $A queues declare --name orders.all --type quorum --durable true $ $A bindings declare --source orders.topic --destination-type queue --destination orders.new --routing-key 'order.*.created' $ $A bindings declare --source orders.topic --destination-type queue --destination orders.all --routing-key 'order.#' $ $A bindings declare --source orders.dlx --destination-type queue --destination orders.dlq --routing-key ''

--non-interactive is the flag for scripts. The declare commands print nothing on success. Now publish three messages: one that matches both patterns, one that matches only order.#, and one that matches nothing.

Publish (the tool states whether each message was routed)
$ $A publish message -e orders.topic -k order.eu.created -m '{"id":1}' Message published and routed successfully $ $A publish message -e orders.topic -k order.eu.shipped -m '{"id":2}' Message published and routed successfully $ $A publish message -e orders.topic -k nothing.matches -m lost Message published but NOT routed $ docker exec rabbit rabbitmqctl list_queues name type messages_ready messages_unacknowledged orders.new quorum 1 0 orders.all quorum 2 0 orders.dlq quorum 0 0

orders.all (binding order.#) has both routed messages, orders.new (binding order.*.created) has only the first, and the unroutable one was dropped, exactly as chapter 03 predicts. Queue statistics refresh on an interval of a few seconds, so a number read immediately after an action can lag; the output above is abridged to the table rows.

Step 3: reject a message and read its death certificate
Reject with requeue=false, then read the dead letter queue
$ $A get messages -q orders.new -a reject_requeue_false $ $A get messages -q orders.dlq -a ack_requeue_false (abridged) exchange orders.dlx, routing key order.eu.created, payload {"id":1} headers: {"x-death":[{"count":1,"exchange":"orders.topic","queue":"orders.new", "reason":"rejected","routing-keys":["order.eu.created"],"time":1793468854}], "x-delivery-count":1,"x-first-death-queue":"orders.new","x-first-death-reason":"rejected", ...}

The message left orders.new, passed through orders.dlx and landed in orders.dlq, with the full history in x-death. (rabbitmqadmin get messages is a polling call meant for development; use a real client in applications.)

Step 4: a delayed retry queue
A queue whose messages expire into the work queue
$ $A queues declare --name work --type quorum --durable true $ $A queues declare --name work.retry --type quorum --durable true \ --arguments '{"x-message-ttl":3000,"x-dead-letter-exchange":"","x-dead-letter-routing-key":"work"}' $ $A publish message -k work.retry -m job-1 # wait about 5 seconds $ $A get messages -q work -a ack_requeue_false (abridged) headers: {"x-death":[{"count":1,"queue":"work.retry","reason":"expired", ...}], "x-first-death-reason":"expired", ...} payload: job-1

The message waited in work.retry for its 3 second TTL, expired, was dead-lettered through the default exchange (empty name, routing key work) and arrived in work with reason: expired.

Step 5: policies, headers exchange and health checks
Policy, headers exchange matching, and diagnostics
$ docker exec rabbit rabbitmqctl set_policy ttl-demo "^tmp\." '{"message-ttl":60000}' --apply-to queues --priority 1 $ docker exec rabbit rabbitmqctl list_policies vhost name pattern apply-to definition priority / ttl-demo ^tmp\. queues {"message-ttl":60000} 1 # headers exchange: Q h.all needs both headers, Q h.any needs one $ $A exchanges declare --name h.x --type headers --durable true $ $A bindings declare --source h.x --destination-type queue --destination h.all --routing-key '' \ --arguments '{"x-match":"all","format":"pdf","type":"report"}' $ $A bindings declare --source h.x --destination-type queue --destination h.any --routing-key '' \ --arguments '{"x-match":"any","format":"pdf","type":"report"}' $ $A publish message -e h.x -m a -p '{"headers":{"format":"pdf"}}' $ $A publish message -e h.x -m b -p '{"headers":{"format":"pdf","type":"report"}}' # result after a few seconds: h.any holds 2 messages, h.all holds 1 $ docker exec rabbit rabbitmq-diagnostics check_running RabbitMQ on node rabbit@... is fully booted and running $ docker exec rabbit rabbitmq-queues quorum_status orders.new (abridged) one row per member: Raft State leader, Membership voter

With one node every quorum queue has a single member, so quorum_status shows one leader row. A real cluster shows a leader and followers (chapter 08).

Step 6: run the Spring application against it

Put the four Java files and application.yml from chapter 09 in a Spring Boot 4.1 project with spring-boot-starter-amqp, set spring.rabbitmq.username and password to the lab credentials, and publish an OrderCreated through OrderPublisher. A message whose handler always throws shows 4 invocations and then arrives in orders.created.dlq; a message with a negative total is rejected on the first failure. Check the queues again:

Inspect the result
$ docker exec rabbit rabbitmqctl list_queues name messages_ready messages_unacknowledged $ docker exec rabbit rabbitmqadmin --non-interactive get messages -q orders.created.dlq -a ack_requeue_true
Step 7: the HTTP API and Prometheus
Management API and metrics
$ curl -s -u guest:guest http://localhost:15672/api/overview (abridged) JSON with "rabbitmq_version": "4.4.0" and message and queue totals $ curl -s http://localhost:15692/metrics | grep -E "^rabbitmq_(queue_messages_ready|queues|connections) " rabbitmq_queue_messages_ready 2 rabbitmq_queues 3 rabbitmq_connections 0 $ curl -s "http://localhost:15692/metrics/detailed?family=queue_coarse_metrics" | grep "^rabbitmq_detailed_queue_messages_ready" (abridged) one series per queue, labelled vhost and queue

The numbers were what the lab showed at that moment, with three queues defined; yours will match your own state. This is the same endpoint a Prometheus server would scrape.

Clean up
  • docker rm -f rabbit removes the broker and all its data.
11
Chapter Eleven - Decisions and Pitfalls

RabbitMQ or Kafka, and the Mistakes to Avoid

This is a decision between two good tools built on different primitives. Page 01 gave the vocabulary: Kafka is a log (consumers keep an offset, messages stay until retention) and RabbitMQ is, in its classic form, a queue (the broker tracks each message until it is acknowledged). RabbitMQ streams add a log-style type, and Kafka offers queue-like features through newer mechanisms, so the lines blur; the comparison below is about the defaults and the centre of gravity of each.

QuestionRabbitMQKafka
PrimitiveExchanges route into queues; streams as a second modelPartitioned, replicated, append-only log
RoutingRich and broker-side: direct, topic, headers, fanout, pluginsTopic plus key; consumers filter themselves. Routing logic lives in applications or in stream processing
Per-message controlAck, nack, requeue, TTL, priority, delayed retry, dead lettering per messageNone per message; commit an offset. Retry and DLQ patterns are built on top (page 04)
ConsumptionBroker pushes with prefetch; competing consumersConsumers pull; one consumer per partition within a group
ReplayNot for queues. Yes with streamsYes: reset offsets within retention
Many independent readers of the same dataA queue per reader (cheap fan-out), or one stream for many readersNative: each consumer group has its own offsets
OrderingPer queue, with caveats (competing consumers, redelivery); SAC or consistent hash for per-keyPer partition, strong and simple, by key
Scaling outMore queues, consumers, nodes; a single queue is limited to one leader nodeMore partitions; throughput scales with partition count
Typical retentionUntil consumed (queues); by age or size (streams)Time or size, independent of consumption
OperationsFewer moving parts for small clusters; know alarms, prefetch, queue typesPartition sizing, rebalancing, retention and broker disk planning
Strong fitTask queues, RPC-style flows, complex routing, per-message retry and delay, mixed protocols (AMQP, MQTT, STOMP)Event streaming, event sourcing, analytics pipelines, high-volume ingestion, stream processing, long retention
"Run this job, retry it if it fails, send poison messages aside"
β–Ά
Queue semantics with per-message acks and a DLX: RabbitMQ (or SQS, page 06)
"Many services react to the same business events, and I may need to replay history"
β–Ά
A log: Kafka, or a RabbitMQ stream if you already run RabbitMQ and the volume is moderate
"Route by several attributes, mix MQTT devices and AMQP services"
β–Ά
RabbitMQ: exchanges and multi-protocol support are its home turf
"Millions of events per second, stream processing, months of retention"
β–Ά
Kafka
"Small team, both kinds of needs"
β–Ά
Prefer one system and use its second model (RabbitMQ streams, or Kafka with application-level retry topics) until the mismatch actually hurts

Neither answer is wrong in every case, and using both is common: Kafka as the event backbone, RabbitMQ for the task and workflow queues that hang off it. Page 07 shows patterns (outbox, saga) that sit above either, and page 06 covers the AWS equivalents.

Field guide: common mistakes
MistakeWhat you seeFix
Unbounded prefetch (0, or a huge value)Consumer memory climbs; one consumer hoards the queue while others idle; a crash redelivers thousands of messagesSet prefetch per consumer from handler duration (chapter 06); Spring default is 250
Auto-ack for important workMessages vanish when a consumer crashes or a handler throws; no trace anywhereManual or container ack with a DLX
Unacked pile-upmessages_unacked large and flat; channel closed by the 30-minute ack timeout; redelivery burstsAlways ack or nack on every path; find the stuck handler; bound prefetch
No DLX / no delivery limitPoison message loops forever, or rejected messages are silently droppedDLX on every work queue; quorum delivery-limit; park and alert
Too many queues, channels or connectionsMemory and CPU rise with the count; connection churn; slow management UIReuse connections; one channel per thread; avoid a queue per user or per request; use routing and consumer-side filtering instead
Classic queue where HA is neededQueue lost with its nodeQuorum queues (or streams)
Ignoring unroutable messagesPublisher says OK, nothing arrivesmandatory plus returns, or an alternate exchange
Non-idempotent consumersDuplicate side effects after redelivery or republishDeduplicate on a message id; make operations idempotent (page 01)
Publish and consume on one connectionWhen the broker blocks publishers, acks stall too and the backlog never drainsSeparate connections
Settings hard-coded as queue argumentsPRECONDITION_FAILED on redeploy when a value changesPolicies for tunable settings
Queue as a databaseMillions of messages parked for days; memory alarms; slow recoveryStore state in a database; a queue is a buffer, not a store
12
Chapter Twelve - Interview

Questions You Should Be Able to Answer

Crisp answers for the questions that come up in interviews on this topic.

  • Q: Explain the AMQP 0-9-1 model.
    Producers publish messages with a routing key to an exchange. The exchange applies its bindings to route copies into queues. Consumers read from queues over channels that are multiplexed on a connection. Everything lives inside a virtual host. Producers never write to a queue directly.
  • Q: What exchange types exist and when do you use each?
    Direct (exact key match, classification), fanout (broadcast to all bound queues), topic (pattern match on dotted keys with * for one word and # for zero or more), headers (match on message headers with x-match all or any). The default exchange is a direct exchange where each queue is bound under its own name.
  • Q: What do * and # mean in a topic binding?
    * matches exactly one word and # matches zero or more words, where words are dot-separated. order.*.created matches order.eu.created but not order.created; order.# matches both and also order.
  • Q: Classic, quorum and stream queues: how do you choose?
    Quorum for important work queues because it is replicated with Raft and never loses confirmed messages while a majority is up. Stream for a replicated log with replay and many non-destructive readers. Classic for transient, non-critical, single-node cases only.
  • Q: What changed in RabbitMQ 4.0?
    Classic queue mirroring was removed, so replication means quorum queues or streams. Quorum queues got a default delivery-limit of 20. AMQP 1.0 became natively supported. Classic queue storage version 1 was removed. The Raft-based Khepri metadata store became fully supported.
  • Q: How does a quorum queue provide durability?
    It runs Raft: the leader appends a message to its log and replicates it to followers; once a majority (N/2+1) has it, the message is committed and the publisher is confirmed. If the leader fails, a follower with the committed log becomes leader. Tolerates the loss of a minority of members.
  • Q: Difference between durable and persistent?
    Durable describes the queue or exchange definition surviving a restart. Persistent describes a message being written to disk. A classic queue needs both to keep messages across a restart. Quorum queues and streams always do both.
  • Q: How do you avoid losing messages when publishing?
    Durable (or quorum) queue, persistent messages, publisher confirms to learn the broker accepted them, and the mandatory flag or an alternate exchange so unroutable messages are noticed. Handle nacks and timeouts by republishing, and make consumers idempotent because that creates duplicates.
  • Q: What are acks, nacks and requeue?
    basic.ack removes a message after successful handling. basic.nack or basic.reject with requeue=true puts it back; with requeue=false it is dead-lettered or discarded. Unacked messages are requeued when the channel closes. Auto-ack gives at-most-once and is unsafe for important work.
  • Q: What is prefetch and how do you pick a value?
    The limit on unacknowledged deliveries per consumer (basic.qos). Higher values raise throughput and memory use and reduce fairness; 1 is fair but slow. Start small (10 to 50 for short handlers, 1 to 5 for long ones), and measure; the docs say 100 to 300 usually gives optimal throughput. Unlimited prefetch is dangerous.
  • Q: How does dead lettering work and what is x-death?
    A queue republishes a message to its DLX when it is rejected without requeue, expires (TTL), is dropped by a length limit, or exceeds a quorum queue delivery limit. x-death records queue, reason, count and times so a consumer can count attempts. Use at-least-once dead-lettering on quorum queues if dead letters must not be lost.
  • Q: How do you implement delayed retry?
    Classic pattern: dead-letter failures to a retry queue with a message TTL whose DLX points back to the work queue, and count x-death to give up after N attempts. On RabbitMQ 4.3+ quorum queues support native delayed retry with linear back-off (delayed-retry-type, delayed-retry-min, delayed-retry-max).
  • Q: How do you preserve ordering with several consumers?
    Competing consumers lose ordering. Use single active consumer for strict order on one queue with failover, and shard by key across several queues with the consistent hash exchange, each queue with its own single active consumer, to keep per-key order while scaling.
  • Q: What do flow control and the memory alarm do?
    Flow control slows a connection that publishes faster than queues, disks or replication can keep up. The memory alarm (default 60 percent of RAM) and the disk alarm block all publishing connections until cleared; consumers keep running so the backlog can drain. Use separate connections for publishing and consuming.
  • Q: RabbitMQ or Kafka?
    RabbitMQ for task queues, per-message retry and delay, rich routing and multi-protocol messaging. Kafka for event streams, replay by many consumer groups, very high volume and stream processing. RabbitMQ streams and Kafka's newer features blur the line, so pick on workload and team experience rather than slogans.
  • Q: How do you monitor RabbitMQ?
    Prometheus plugin on port 15692 plus Grafana, the management UI for inspection. Watch queue depth (messages_ready), unacked count, publish/deliver/redelivery rates, connection and channel counts, memory and disk headroom, alarms, and the dead letter queue depth. Health checks via rabbitmq-diagnostics.
13
Chapter Thirteen - Summary

Cheat Sheet and Recap

Cheat sheet
ConceptRememberValue (RabbitMQ 4.x)
RoutingProducer to exchange to bindings to queuesTypes: direct, fanout, topic, headers; default exchange ""
Topic patterns* = one word, # = zero or moreWords are dot-separated
Default queue type for important workQuorum (Raft, majority)Group size 3 by default; odd sizes
Classic queuesNot replicated; mirroring removed4.0
Quorum delivery-limitPoison message protectionDefault 20 since 4.0; -1 disables
Reliable publishDurable or quorum + persistent + confirms + mandatoryConfirm = safe in queue, not consumed
Reliable consumeManual or container ack, idempotent handlersAuto-ack = at most once
PrefetchPer-consumer cap on unacked deliveriesDocs: 100 to 300 usually best; Spring default 250; quorum cap 2,000
Dead letter reasonsrejected, expired, maxlen, delivery_limitx-death header, most recent first
TTLPer queue message-ttl, per message expiration (string), queue expiresLower value wins; expiry acts at the queue head
Delayed retryTTL + DLX loop; native on quorum queuesNative from 4.3
PriorityClassic x-max-priority (2 to 4 recommended); quorum strict 0 to 31Quorum strict from 4.3
OrderingSingle active consumer; consistent hash for per-key shardingx-single-active-consumer, x-consistent-hash
AlarmsMemory and disk alarms block publishersMemory 0.6 of RAM, disk 50 MB by default
PortsAMQP, management, Prometheus5672, 15672, 15692
Delivery ack timeoutChannel closed if a delivery is not acked in time30 minutes default; quorum queues only from 4.3
βˆ‘ RabbitMQ - what to remember
  • A smart broker: routing, acks, redelivery, TTL, priority and dead lettering are broker features, which keeps consumers simple.
  • Exchanges decide, queues store. Choose the exchange type for the routing you need; topic exchanges cover most event routing.
  • Quorum queues are the default for data you care about, classic queues are no longer replicated, and streams give a replayable log.
  • Reliability needs three switches: durable and persistent (or quorum), publisher confirms with mandatory, and manual acknowledgments with idempotent consumers.
  • Bound prefetch. It trades throughput for fairness and memory; unlimited prefetch and auto-ack are the two classic ways to lose control.
  • Every work queue needs a DLX and a retry story: delivery limit, delayed retry, and an alert on the dead letter queue.
  • Operate it: odd cluster sizes, alarms understood, separate publish and consume connections, and Prometheus metrics for depth, unacked and rates.
  • RabbitMQ and Kafka solve different problems: queues with per-message control versus a replayable log. Many systems use both.
Version note

Checked on 2026-10-31 against the RabbitMQ documentation sources for the current 4.x series (quorum queues, classic queues, streams, queues, confirms, consumers, consumer prefetch, dead letter exchanges, TTL, priority, flow control, memory and disk alarms, clustering, metadata store, Prometheus, management CLI, deprecated features list, consistent hash exchange README) and the Spring AMQP reference and Spring Boot 4.1.2 configuration metadata. The Docker lab ran rabbitmq:4-management reporting 4.4.0, rabbitmqadmin 2.35.0; the Spring code was compiled and run with Spring AMQP 4.1.2 and Spring Boot 4.1.2. Features tied to a minor version (delayed retry, strict quorum priorities, ack timeout for quorum queues only, transient non-exclusive queues denied by default: 4.3) should be re-checked against your broker version. Not verified: the stream protocol port and client details, and anything about Kafka share groups. See the Kafka pages for current Kafka behaviour.

Next in this section