RabbitMQ Smart Broker
RabbitMQ is the broker with opinions: it routes, tracks, retries, delays and dead-letters for you. This page explains the AMQP 0-9-1 model and every exchange type, how classic, quorum and stream queues differ and what RabbitMQ 4.0 changed, how to publish and consume without losing messages, how prefetch shapes throughput, how dead lettering and delayed retry work, how to run and monitor a cluster, how to use it from Spring AMQP, and when Kafka is the better tool, finishing with a lab you can run in minutes.
The Smart Broker and the AMQP 0-9-1 Model
RabbitMQ is a message broker: a server that accepts messages from producers, decides where they should go, holds them safely, and hands them to consumers. Page 01 introduced the generic vocabulary (message, queue, ack, redelivery, dead letter). RabbitMQ is the best-known example of the smart broker, simple consumer design: the broker owns routing, per-message acknowledgment, redelivery, expiry and priority, so each consumer can stay small. Kafka (pages 02 to 04) takes the opposite stance and keeps the broker simple.
The core protocol is AMQP 0-9-1, and its central idea is a layer of indirection: a producer never writes to a queue. It publishes to an exchange together with a routing key, and the exchange uses its bindings to decide which queues receive a copy. That split is what lets you add a new consumer (new queue plus binding) without touching the producer.
| Term | What it is | Remember |
|---|---|---|
| Virtual host (vhost) | A namespace: its own exchanges, queues, bindings, users permissions and limits | Default is /. One vhost per environment or tenant |
| Connection | A long-lived TCP (optionally TLS) connection from a client | Expensive to open; keep few |
| Channel | A lightweight virtual connection multiplexed over one connection | Where publishing, consuming, acks and basic.qos happen |
| Exchange | Receives published messages and routes them | Types: direct, fanout, topic, headers (chapter 03) |
| Queue | Ordered buffer that stores messages until consumers ack them | Types: classic, quorum, stream (chapter 04) |
| Binding | A rule linking an exchange to a queue (or another exchange), often with a pattern | Routing logic lives here |
| Routing key | A string the publisher attaches; matched against binding keys | Meaning depends on the exchange type |
| Message | Body (bytes) plus properties (headers, delivery mode, expiration, priority, correlation id...) | RabbitMQ never looks at the body |
RabbitMQ speaks several protocols. AMQP 0-9-1 is the classic one and the one every Spring AMQP and Java client example on this page uses. Since RabbitMQ 4.0, AMQP 1.0 is supported natively by the core broker (no plugin): the same port 5672 accepts both, and the client chooses with a protocol header. Streams have their own binary protocol (a plugin), and MQTT and STOMP are available through plugins. A message published with one protocol can be consumed with another, with a conversion step when needed.
This page targets RabbitMQ 4.x. Where a feature needs a specific minor version it is stated (for example 4.3 introduced strict priorities and delayed retry for quorum queues). The hands-on chapter runs the rabbitmq:4-management image, which was 4.4.0 when this page was written.
- There is no such thing in AMQP 0-9-1. Publishing to the default exchange (empty name) with the queue name as the routing key looks direct, but an exchange is still in the path (chapter 03). Thinking in exchanges from day one keeps your design extensible.
Connections, Channels and Virtual Hosts
A connection is a TCP connection with an AMQP handshake on top (and TLS if enabled). Opening one costs several network round trips plus authentication, so applications open it once and keep it for their lifetime. Inside it you open channels: independent, lightweight sessions multiplexed over the one connection. Publishing, consuming, acking and basic.qos all happen on a channel.
| Topic | Guidance | Why |
|---|---|---|
| Connections | Long-lived. Typically one connection per application process (often one for publishers and a separate one for consumers) | A burst of connection opens is expensive and shows up as connection churn in monitoring |
| Channels | Open one per thread or task that needs it; never share a channel between threads for publishing | Channels are not thread-safe in the Java client; sharing causes interleaved frames and errors |
| Channel count | Bound it. Servers negotiate channel_max (2047 in the 4.4.0 image used for the lab) | Every channel uses memory on the broker and the client; leaked channels are a classic slow leak |
| Publishers vs consumers | Use separate connections for publishing and consuming | When the broker applies flow control or blocks a publishing connection (chapter 08), TCP back pressure would otherwise also stall your consumers' acks |
| Heartbeats | Keep them on. The suggested default is 60 seconds | They detect dead peers and keep idle connections alive through load balancers |
| Errors | A channel-level error (for example acking an unknown delivery tag) closes that channel; a connection-level error closes everything | Handle channel closure and re-open; do not assume a channel lives forever |
| Virtual hosts | One vhost per environment, tenant or bounded context; give each application a user with permissions on its vhost only | Isolation of names, permissions, limits and policies in a single cluster |
The broker never "remembers" a client. If the connection drops, every channel, consumer, unacknowledged delivery and non-durable declaration of that client is gone. Unacked messages go back to their queue and will be redelivered (with the redelivered flag set), possibly to a different consumer. Good clients re-open the connection, recreate the topology (declarations are idempotent) and resume consuming. Spring AMQP and the Java client do this automatically by default; chapter 09 shows where the knobs are.
- Queue, exchange and binding declarations are idempotent as long as the arguments match. Many applications declare their topology at startup (Spring does it through
Queue,ExchangeandBindingbeans). If a queue already exists with different arguments, the declaration fails withPRECONDITION_FAILEDand the channel closes. Queue arguments cannot be changed in place, which is why policies (chapter 07) are preferred for settings that may evolve.
Exchanges: Direct, Fanout, Topic, Headers and the Default Exchange
The exchange type decides how the routing key (or headers) is matched against bindings. Four types ship with the broker; the default exchange is a special direct exchange. Each diagram below shows one message arriving and which queues get a copy. Gray dashed paths are bindings that do not match.
Routes to the queues whose binding key equals the routing key. Use it for classification by a fixed label: log severity, job type, a tenant name.
Copies every message to all bound queues and ignores the routing key. Use it for broadcast: notify audit, email and search-indexing independently, each with its own queue and consumers.
Routing keys are dot-separated words (order.eu.created). Binding keys are patterns: * substitutes exactly one word, # substitutes zero or more words. This is the workhorse for event routing because consumers choose their slice with a pattern.
| Routing key | order.*.created | order.# | *.us.* | #.shipped | # |
|---|---|---|---|---|---|
order.eu.created | match | match | no | no | match |
order.us.shipped | no | match | match | match | match |
order.created | no (needs a middle word) | match | no | no | match |
order | no | match (# is zero words) | no | no | match |
shipped | no | no | no | match (# is zero words) | match |
- Pick a stable word order such as
domain.entity.eventorregion.service.severity, put the most selective word first, and document it. A binding of#makes a topic exchange behave like fanout for that queue.
Ignores the routing key and matches on the message headers instead. Each binding carries a map of header values plus x-match: all (default) requires every listed header to match, any requires at least one. It is useful when routing depends on several attributes at once and encoding them into a dotted key would be awkward. In practice topic exchanges cover most needs and are easier to read and reason about.
Every vhost has a pre-declared direct exchange with the empty name. When a queue is declared it is bound to it automatically with the queue name as key, so publishing with exchange "" and routing key tasks lands in queue tasks.
| Type | Matches on | Typical use | Cost to evaluate |
|---|---|---|---|
| direct | Routing key equals binding key | Job types, severities, per-tenant queues | Very cheap |
| fanout | Nothing (all bindings) | Broadcast, pub/sub to several services | Cheap, but copies grow with queue count |
| topic | Pattern on the routing key (*, #) | Event routing by domain, region, severity | Cheap to moderate |
| headers | Header values with x-match all or any | Multi-attribute routing | Moderate |
default ("") | Routing key equals queue name | Sending to one known queue | Very cheap |
| consistent-hash (plugin) | Hash of routing key or a header | Sharding for per-key ordering (chapter 07) | Cheap |
- If no binding matches, the exchange drops the message and the publisher is told nothing by default. Symptom: the publisher reports success, the queue stays empty. Fix: publish with the
mandatoryflag (chapter 05), configure an alternate exchange on the exchange, or at least monitor the unroutable-dropped metric.
Classic, Quorum and Stream Queues, and What Changed in 4.0
A queue's type is chosen when it is declared (x-queue-type, or QueueBuilder.quorum() in Spring) and fixes how messages are stored and replicated. RabbitMQ 4.x has three: classic, quorum and stream.
| Classic | Quorum | Stream | |
|---|---|---|---|
| Storage model | Queue: message removed once acked | Queue (replicated): removed once acked | Append-only log: messages stay until retention removes them |
| Replication | None since 4.0 (mirroring removed) | Yes: Raft consensus across members | Yes: replicated log |
| Survives node loss | No (the queue lives on one node) | Yes while a majority of members is up | Yes while enough replicas are up |
| Consume semantics | Destructive (ack deletes) | Destructive (ack deletes) | Non-destructive: read again by offset or timestamp |
| Durability | Durable or transient; persistence per message | Always durable, always persisted | Always persistent |
| TTL and length limits | Yes | Yes (but not reject-publish-dlx) | Retention by age or size instead |
| Priorities | Yes (x-max-priority, use few levels) | Yes, strict, 0 to 31, from 4.3 | No |
| Poison message limit | No | Yes: delivery-limit, default 20 since 4.0 | Not applicable |
| Typical use | Transient, non-critical, single-node work | Default for important work queues | Fan-out to many readers, replay, large backlogs |
- Classic queue mirroring was removed in 4.0 (deprecated since 2021). Classic queues are now a non-replicated type. If you used mirrored classic queues for high availability, migrate to quorum queues (or streams). The docs describe a blue-green strategy for moving from 3.13.
- **Quorum queues got a default
delivery-limitof 20 in 4.0.** A message returned to the queue more than 20 times is dropped, or dead-lettered if a DLX is configured. In 3.13 there was no limit. Setx-delivery-limit=-1(or thedelivery-limitpolicy key) to restore unlimited redelivery, which the docs advise only for rare compatibility cases. Since 4.3 the limit counts failed deliveries (x-delivery-count) and plain returns vianackdo not count. - Native AMQP 1.0 support in the core broker since 4.0, alongside AMQP 0-9-1 on the same port.
- Classic queue version 1 storage was removed in 4.0; version 2 is the only implementation and existing queues are converted at node start.
- Khepri, a Raft-based metadata store, is fully supported from 4.0 and is the recommended replacement for Mnesia. In the 4.4.0 image used on this page the
khepri_dbfeature flag was enabled. - 4.3 added delayed retry with linear back-off for quorum queues and strict 0 to 31 message priorities. It also made non-durable, non-exclusive classic queues impossible to declare unless you set
deprecated_features.permit.transient_nonexcl_queues = true. - Global QoS (a prefetch limit shared by all consumers of a channel) is a deprecated feature. In the 4.4.0 image
rabbitmqctl list_deprecated_featuresshowedglobal_qosasdenied_by_default. Use per-consumer prefetch.
- Classic queue mirroring no longer exists in 4.x, so a mirroring policy no longer gives you replication (
rabbitmq-diagnostics check_if_cluster_has_classic_queue_mirroring_policyreports such policies). Queues that were mirrored are plain single-node classic queues: lose the node and you lose the queue. Migrate to quorum queues before relying on the upgraded cluster.
A quorum queue has a leader and followers, together called members (3 by default; x-quorum-initial-group-size changes it, and an odd number is recommended). Publishing goes to the leader, which appends the message to its replicated log and sends it to followers. A message is committed when a majority of members (N/2 + 1) have written it, and only then is the publisher confirmed. If the leader dies, the followers elect a new leader from the members that have the committed data, with no data loss for confirmed messages.
| Cluster size / queue members | Majority needed | Member failures tolerated |
|---|---|---|
| 1 | 1 | 0 |
| 3 | 2 | 1 |
| 5 | 3 | 2 |
| 4 (even) | 3 | 1 (no better than 3) |
Consequences you should know: writes need a majority, so a minority partition cannot accept publishes to that queue; quorum queues always persist to disk and never keep the whole queue in memory; they do not support non-durable or exclusive queues (use single active consumer instead of exclusivity); and consumer prefetch is capped at 2,000 even if you ask for unlimited.
A policy applies settings to every queue whose name matches the pattern and can be changed at any time without redeclaring. Arguments set in code (x-delivery-limit, x-dead-letter-exchange) are baked in at declaration and cannot be changed without deleting the queue, which is why the docs recommend policies for anything that may change.
A stream is an append-only, replicated log inside RabbitMQ. Consumers read non-destructively from an offset, timestamp or the beginning, so many consumers can read the same data and replay it, much like a Kafka partition (page 02). Retention is by x-max-age or x-max-length-bytes, evaluated per segment file (default segment 500,000,000 bytes). Streams can be used from AMQP 0-9-1 clients (acks required, per-consumer prefetch required, no global QoS) or, for best throughput, through the dedicated stream protocol and its clients. Super streams partition a stream for parallelism. Streams complement queues; they do not replace them.
- Work queue where each message is a task to complete: quorum. Events many independent services must read, or that you may need to replay: stream (or Kafka, chapter 11). Scratch queue you can lose, single node, low volume: classic. When unsure, pick quorum.
Reliable Publishing: Durability, Confirms and the Mandatory Flag
"The broker accepted my publish" is not one fact, it is three: the message was routed to at least one queue, the queue(s) stored it safely, and the publisher learned about both. RabbitMQ gives you a switch for each, and skipping one is how messages get lost without an error anywhere.
| Setting | Applies to | What it protects | Where set |
|---|---|---|---|
| Durable exchange or queue | The definition | The queue or exchange still exists after a broker restart | durable=true on declare; QueueBuilder.durable(...) |
Persistent message (delivery_mode=2) | The message | The message is written to disk, not only held in memory | Message property; Spring AMQP sets persistent by default |
| Quorum queue or stream | The queue type | Always durable and always persisted, so both switches are implied | x-queue-type |
For a classic queue you need both: a durable queue and persistent messages. A persistent message in a non-durable queue vanishes at restart because the queue itself is deleted; a transient message in a durable queue is discarded when the node boots. Quorum queues and streams remove the foot-gun by always doing both.
A publisher puts its channel in confirm mode once. The broker then answers every publish, asynchronously, with basic.ack or basic.nack, identified by a sequence number the client keeps. Confirms are the only reliable way to know the broker took responsibility for a message. (AMQP transactions also exist but confirms are the recommended mechanism.)
| Situation | What the broker sends | Meaning |
|---|---|---|
| Routed to one or more queues | basic.ack once all target queues accepted it | Persistent message in durable queue: written to disk. Quorum queue: a quorum of replicas accepted it |
Not routed, mandatory=false (default) | basic.ack | The exchange verified there is no queue and dropped the message. It looks like success |
Not routed, mandatory=true | basic.return, then basic.ack | Publisher is told the message was unroutable |
| Internal error in the queue process | basic.nack | The broker refuses responsibility; publisher may republish |
| No answer yet | - | No guarantee how soon a confirm arrives; do not block forever |
Two details matter. A confirm says the message is safe in the queue, not that anyone consumed it; consumer acknowledgments are a separate mechanism (chapter 06). And for persistent messages the broker batches fsync calls, so a confirm for a single message on an idle queue can take a moment: that is the price of durability, not a fault.
| Publish strategy | How | Trade-off |
|---|---|---|
| One at a time | Publish, wait for the confirm, repeat | Simplest; throughput limited by round-trip latency |
| Batch | Publish N, wait for all confirms | Much faster; on failure you may have to republish the whole batch |
| Asynchronous | Publish continuously, handle ack/nack in a callback keyed by sequence number or correlation id | Best throughput; most code (Spring's CorrelationData does the bookkeeping) |
With mandatory=true, an unroutable message is returned to the publisher through basic.return (Spring: a ReturnsCallback) before the confirm. Alternatively an alternate exchange (alternate-exchange policy key or argument) receives everything the primary exchange could not route, which lets the broker catch it into a "parking" queue with no publisher involvement.
- A
nackor a missing confirm means "unknown", not "lost": the message may in fact be stored. Republishing then creates a duplicate. The consumer side must be idempotent (page 01): set a stablemessage-idon each message and deduplicate on it, or make the operation naturally idempotent.
mandatory=falseAcknowledgments, Requeueing and Prefetch
RabbitMQ pushes messages to consumers registered with basic.consume (the alternative, basic.get, polls one message per call and is meant for tests and rare cases). The delivery guarantee on the consuming side comes from acknowledgments.
| Mode | How the broker behaves | Delivery guarantee | Use when |
|---|---|---|---|
Automatic ack (autoAck=true, Spring AcknowledgeMode.NONE) | Treats a message as handled the moment it is written to the socket | At most once. A crash after delivery loses the message | Losing messages is acceptable (metrics, cache refresh). Rarely right |
Manual ack (Spring AUTO or MANUAL) | Keeps the message until the consumer sends basic.ack; requeues it if the channel closes | At least once: duplicates possible, so handlers must be idempotent | Almost everything |
| Consumer response | Protocol method | Effect |
|---|---|---|
| Success | basic.ack (optionally multiple=true to ack all tags up to this one) | Message removed from the queue |
| Failure, try again | basic.nack or basic.reject with requeue=true | Message goes back to the queue (position may differ) and is redelivered with redelivered=true |
| Failure, give up | basic.nack or basic.reject with requeue=false | Dead-lettered if a DLX is configured, otherwise discarded |
| Consumer or connection dies | (none) | Unacked messages are requeued automatically |
basic.nack can reject several deliveries at once (the multiple flag), basic.reject is single-message only. Acking the same delivery tag twice, or on another channel, closes the channel with PRECONDITION_FAILED - unknown delivery tag.
- A handler that fails on a bad message and always nacks with
requeue=truecreates a hot loop: the broker redelivers immediately, CPU burns, logs fill, and the message is never removed. Symptom: the same stack trace repeating several times a second and one message with a growing redelivery count. Fix: dead-letter after a bounded number of attempts (quorumdelivery-limit, Spring retry plus a DLX) and treat clearly invalid messages as permanent failures straight away.
Without a limit the broker pushes the entire queue at a consumer as fast as the network allows. Prefetch (basic.qos) caps the number of unacknowledged deliveries a consumer may hold. When it is reached the broker stops sending until an ack frees a slot. It is the single most influential consumer setting for both throughput and fairness.
| Prefetch | Throughput | Fairness and memory | Notes |
|---|---|---|---|
| 1 | Lowest: one round trip per message | Best: work is spread by actual speed | Use for slow, heavy jobs or when strict fairness matters |
| 10 to 50 | Good for moderately fast handlers | Good | A sensible starting range for mixed work |
| 100 to 300 | The docs say values in this range usually give optimal throughput | One slow consumer can hold up to this many messages | Higher values hit diminishing returns |
| 0 (unlimited) | Highest delivery rate | Worst: consumer memory grows, one consumer can hoard the backlog | Quorum queues still cap it at 2,000 |
- RabbitMQ applies
prefetch_countper consumer on a channel, which differs from the AMQP 0-9-1 text (shared across the channel). A global (channel-wide) limit exists but is a deprecated feature and slower; avoid it. - Spring AMQP's
prefetchCountdefaults to 250 (since version 2.0) and is ignored when the ack mode isNONE. Set it explicitly per listener container based on handler duration. basic.qoshas no effect onbasic.get.- Example sizing (illustrative arithmetic): if a handler takes 5 ms and the network round trip is 1 ms, a prefetch of 1 leaves the consumer idle for roughly 1 ms out of every 6 ms. A prefetch of a few messages hides the round trip entirely. For handlers that take seconds, the round trip is irrelevant and a small prefetch (1 to 5) keeps work balanced.
- Prefetch roughly equals the number of messages you can process during one network round trip, plus a small buffer, and never more than you are willing to redeliver if the consumer dies. Measure with the unacked count (chapter 08).
If a consumer holds a delivery without acking it, the broker eventually closes the channel. The default timeout is 30 minutes (consumer_timeout, configurable globally or per queue with a policy). As of RabbitMQ 4.3 the timeout is enforced for quorum queues only, because stuck deliveries bloat the Raft log. Symptom of hitting it: the consumer logs a channel error mentioning a delivery acknowledgement timeout, and the message is redelivered. Fix: speed up the handler, raise the timeout deliberately, or hand long work to a separate worker and ack when it is accepted.
- It is faster precisely because it removes the safety net: the broker forgets the message as soon as it is on the wire, so a consumer crash, a deserialization error or a restart loses it with no trace. Use manual acknowledgment and tune prefetch for speed instead.
Dead Letters, TTL, Retry, Priorities, Single Active Consumer and Ordering
A dead letter exchange (DLX) is an ordinary exchange to which a queue republishes messages that die. A message is dead-lettered in four cases:
Reason (x-death reason) | Trigger |
|---|---|
rejected | Consumer sent basic.reject or basic.nack with requeue=false (or an AMQP 1.0 rejected outcome) |
expired | The per-message or per-queue TTL elapsed |
maxlen | The queue exceeded its length limit and dropped the message (drop-head) |
delivery_limit | A quorum queue message was returned more than delivery-limit times |
If a whole queue expires (x-expires), its messages are not dead-lettered. Configure the DLX with a policy (dead-letter-exchange, optional dead-letter-routing-key); if no routing key is set the message keeps its original key(s). If the DLX does not exist when needed, messages are silently dropped, so create it first. Each dead-lettering **adds an x-death header** and the first and last events are summarised in x-first-death-* and x-last-death-* headers. The real output below came from the lab in chapter 10: a message rejected from queue orders.new and read from the dead letter queue.
- The
x-deatharray is ordered most recent first and is compressed by the {queue, reason} pair: thecountfield says how many times it happened. That is exactly the counter a retry loop needs. - By default, dead-lettered messages are republished without publisher confirms, so in a cluster a dead-letter hop can lose messages if the target is unavailable. Quorum queues offer
dead-letter-strategy=at-least-once, which retains the message until the DLX target confirmed it. It needsoverflow=reject-publishand a configured DLX, and costs more memory and CPU. The default isat-most-once. - RabbitMQ detects a dead-letter cycle and drops the message if nothing in the cycle was a rejection. A TTL-driven retry loop (below) is fine because the first hop is a rejection.
- A queue without a dead letter configuration silently discards rejected and expired messages. For quorum queues the docs recommend a dead letter configuration on all of them so a poison message is parked, not lost.
| Kind | Set with | Behaviour |
|---|---|---|
| Per-queue message TTL | policy key message-ttl or argument x-message-ttl (milliseconds) | Messages older than this expire. The server guarantees expired messages are not delivered |
| Per-message TTL | message property expiration (a string of milliseconds) | If the queue also has a TTL, the lower value wins |
| Queue TTL | policy key expires or argument x-expires | The queue is deleted after being unused (no consumers) for that long; messages are not dead-lettered |
| TTL of 0 | either of the first two | Message expires on arrival unless a consumer can take it immediately |
- Expiry is only acted on when a message reaches the head of the queue. A short-TTL message queued behind a long-TTL one stays in the queue (and in the queue statistics) until the one in front is consumed or expires. If you need predictable expiry, use a queue-level TTL so all messages share one value.
Retrying immediately usually hits the same transient failure again. The classic RabbitMQ pattern delays the retry with two queues: the work queue dead-letters rejected messages into a retry queue that has a message TTL and a DLX pointing back at the work queue. After the TTL the message expires and returns for another attempt. The x-death count tells the consumer how many times that has happened.
This pattern gives a fixed delay per retry queue; for growing delays you create several retry queues (5 s, 30 s, 5 min) and route by attempt. From RabbitMQ 4.3, quorum queues have native delayed retry with linear back-off: set delayed-retry-type (disabled, all, failed or returned), delayed-retry-min and optionally delayed-retry-max in milliseconds, and the delay for a returned message becomes min(min * delivery_count, max). The docs recommend it over a dead-letter cycle because it does not rewrite the message and cannot lose it. It is the better choice on 4.3 or later; the TTL+DLX loop remains the portable pattern for older versions and classic queues.
A priority queue delivers higher-priority messages first. Classic queues need x-max-priority (1 to 255, but the docs strongly recommend a small single-digit number such as 2 to 4, because each level costs resources). Quorum queues from 4.3 always support strict priorities 0 to 31 (messages without a priority count as 4) and ignore x-max-priority; before 4.3 they mapped priorities onto just two buckets. Priorities interact with prefetch: messages already delivered to a consumer cannot be overtaken, so a large prefetch weakens the effect. Use a priority queue only when one queue must mix urgencies; separate queues per urgency are usually simpler and avoid starvation questions.
Competing consumers on one queue give parallelism but no ordering: two consumers can finish messages out of order, and redelivery reorders them further. When order matters, enable single active consumer (SAC): declare the queue with x-single-active-consumer=true (Spring: QueueBuilder...singleActiveConsumer()). Many consumers may register, but the queue delivers only to one at a time; if it dies, another takes over. This gives ordered processing plus automatic failover. (SAC on streams needs a native stream client.)
One active consumer means no parallelism, which is the cost of ordering. To keep per-key order while scaling, shard the stream of messages across several queues, each with its own SAC. The consistent hash exchange plugin (rabbitmq_consistent_hash_exchange, exchange type x-consistent-hash) does the sharding: the binding key is a weight (a number as a string), and the exchange hashes the routing key (or a header named by the hash-header exchange argument) to pick one bound queue, so all messages with the same key go to the same queue.
- The hash distributes routing keys, not messages: one very hot key still lands in one queue.
- Use one binding per queue. Changing the number of queues changes the mapping, so a key may move to another queue during a resize and ordering is only guaranteed once the old queue has drained.
- This is the RabbitMQ analogue of Kafka's key-to-partition rule (page 02), implemented with queues instead of partitions.
Clustering, Flow Control, Alarms and Monitoring
A cluster is several RabbitMQ nodes sharing metadata (vhosts, users, exchanges, queue definitions, bindings, policies). Message data lives in queues, and only replicated queue types give high availability: quorum queues and streams. Plan on an odd number of nodes (3 or 5). Clustering is designed for a low-latency LAN. The docs say a cluster can be stretched across availability zones or data centers because quorum queues and Khepri use Raft, but that needs deliberate planning around latency and node placement; for moving messages between distant sites, Shovel and Federation are the usual tools.
- Metadata store: Mnesia was the original; Khepri (Raft-based) is fully supported from 4.0 and recommended. Both are internal; what you see is how the cluster behaves during a network partition.
- Leader placement: keep leaders balanced, otherwise one node does most of the work. Quorum queue leader placement and member reconciliation are configurable.
- Node loss: with 3 members a quorum queue keeps working after one failure. Replace the node and rebalance members; the CLI offers
rabbitmq-queues add_member,grow,shrink,rebalanceandquorum_status. - Rolling upgrades: upgrade one node at a time and check
rabbitmq-diagnostics check_if_node_is_quorum_criticalbefore stopping a node, so you do not take down a majority. - Health checks:
rabbitmq-diagnostics check_runningfor liveness,check_alarmsfor resource alarms,check_for_quorum_queues_without_an_elected_leaderfor queue health. (check_local_alarmsis deprecated and a no-op in 4.4.)
Publishers can outrun queues, disks and replication. RabbitMQ defends itself in two layers:
| Mechanism | Trigger | Effect on clients | You see |
|---|---|---|---|
| Credit-based flow control | A queue, channel or connection falls behind a fast publisher | The publishing connection is slowed to the speed the system can sustain | Connection state flow in the UI and rabbitmqctl. No configuration, no errors, publisher just looks slower |
| Memory alarm | Node memory crosses vm_memory_high_watermark (relative 0.6 of available RAM by default) | All publishing connections are blocked cluster-wide until the alarm clears. Consumers continue, which is how the backlog drains | Connection state blocked in the UI and rabbitmqctl |
| Disk alarm | Free disk space falls below disk_free_limit (default 50 MB; raise it, for example to an absolute value in the GB range or relative to RAM) | Publishers blocked, as above | Same, plus a disk alarm in the UI |
The memory watermark is a throttle point, not a hard cap: a node can use more than the limit. Leave headroom (RAM for the OS and file cache) and do not run a node at the edge. When an alarm is on, look for the cause, not the switch: a queue with no consumers, consumers that are slow or crashed, unacked messages piling up, or a queue type that holds too much in memory.
- If an application publishes and consumes on the same connection, a blocked connection can stall consumer acknowledgments, which keeps the backlog alive, which keeps the alarm on. Use separate connections for publishing and consuming (chapter 02). In Spring AMQP,
RabbitTemplate.setUsePublisherConnection(true)makes the template use a connection separate from the listener containers.
Enable the Prometheus plugin (rabbitmq_prometheus, port 15692, /metrics) and scrape it; the management UI (rabbitmq_management, port 15672) is for interactive inspection, and its built-in metric collection is a deprecated feature. In the rabbitmq:4-management image used for the lab both plugins were already enabled.
| Metric (Prometheus name) | What it tells you | Alert idea |
|---|---|---|
rabbitmq_queue_messages_ready | Backlog waiting for consumers (queue depth) | Sustained growth, or older than your SLO |
rabbitmq_queue_messages_unacked | Delivered but not acked | Large and flat: stuck consumers or prefetch too high |
rabbitmq_queue_messages_published_total, ..._delivered_ack_total, ..._redelivered_total | Publish, consume and redelivery rates (use rate()) | Publish rate above consume rate; redelivery spike |
rabbitmq_connections, rabbitmq_channels and the ..._opened_total counters | Connection and channel counts and churn | Rapid opens: connections not reused |
rabbitmq_resident_memory_limit_bytes vs rabbitmq_process_resident_memory_bytes | Distance to the memory alarm | Above about 80 percent of the limit |
rabbitmq_disk_space_available_bytes vs rabbitmq_disk_space_available_limit_bytes | Distance to the disk alarm | Within a safety margin of the limit |
rabbitmq_detailed_queue_messages_ready (from /metrics/detailed?family=queue_coarse_metrics) | Per-queue depth, with vhost and queue labels | Per-queue alerts; mind the label cardinality with many queues |
The default /metrics endpoint aggregates across queues to stay cheap. Per-queue series come from the detailed endpoint, where you choose metric families and optionally a vhost; avoid scraping every family from a broker with thousands of queues. Alongside metrics, log the redelivered flag and the x-death count in your consumers, and watch the dead letter queue depth: a non-empty DLQ is an alert, not a graveyard.
- Three or five nodes; quorum queues for important data; a DLX on every work queue; per-vhost users, no
guest; TLS; bounded prefetch; separate publish and consume connections; alarms and queue depth monitored; definitions exported for disaster recovery. The official production checklist page covers the rest.
RabbitMQ with Spring AMQP
Spring AMQP (spring-amqp and spring-rabbit) and its Spring Boot starter spring-boot-starter-amqp wrap the Java client with a template for sending, listener containers for receiving, and declarative topology. The code below was compiled against Spring AMQP 4.1.2 and Spring Boot 4.1.2 (the current releases on Maven Central when this page was written) and run against the rabbitmq:4-management container from chapter 10. Property names are Boot 4.1 names; Boot 3.x uses max-attempts where Boot 4 uses max-retries.
Queue, Exchange and Binding beans are declared on the broker by RabbitAdmin when the first connection opens. Use the builders: QueueBuilder.durable(name).quorum() sets the queue type, deliveryLimit, deadLetterExchange and singleActiveConsumer set the matching arguments, ExchangeBuilder creates exchanges, and BindingBuilder.bind(queue).to(exchange).with(key) binds. A MessageConverter bean (here JacksonJsonMessageConverter, the Jackson 3 based converter in Spring AMQP 4) is picked up by the auto-configured RabbitTemplate and by listeners.
- The
x-delivery-limitandx-dead-letter-exchangeabove are queue arguments: they are fixed when the queue is first declared and a later change fails withPRECONDITION_FAILEDbecause the arguments no longer match. The RabbitMQ docs recommend policies for settings that may change, and theQueuebean for what must not.
With publisher-confirm-type: correlated each publish may carry a CorrelationData, whose getFuture() completes with a Confirm (isAck(), getReason()). With publisher-returns: true and mandatory set on the template, an unroutable message comes back as a ReturnedMessage, which is stored in the same CorrelationData. Waiting on the future per message is simple but slow; for volume, publish a batch and then wait for all futures, or register callbacks.
@RabbitListener turns a method into a consumer inside a SimpleMessageListenerContainer (the default container type; a direct container is available via spring.rabbitmq.listener.type=direct). The acknowledge mode is container-level:
| Spring mode | Behaviour | Notes |
|---|---|---|
AUTO (default) | Container acks after the listener returns; if it throws, the message is rejected | Whether a rejected message is requeued is defaultRequeueRejected, default true. Throw AmqpRejectAndDontRequeueException to dead-letter explicitly |
MANUAL | Your code calls channel.basicAck/basicNack (inject Channel and the AmqpHeaders.DELIVERY_TAG header) | You must ack every message or the channel stalls at the prefetch limit |
NONE | Broker auto-ack: no acks are sent | At most once. Prefetch is ignored |
Some errors are never worth retrying. Spring's default ConditionalRejectingErrorHandler rejects without requeue the failures that cannot succeed on redelivery, such as MessageConversionException, MethodArgumentNotValidException and MethodArgumentTypeMismatchException; an x-death header on such a message makes it discard instead of loop. Other exceptions follow defaultRequeueRejected.
Spring Boot can wrap the listener in stateless, in-process retry with back-off. When retries are exhausted the default recoverer, RejectAndDontRequeueRecoverer, rejects the message without requeue, so the queue's DLX (set on the queue in the topology) routes it to the DLQ. The configuration below gives 1 initial attempt plus 3 retries.
I ran this configuration against the lab broker with a listener that always throws. The handler was invoked 4 times (the first attempt plus max-retries: 3 retries) with growing pauses, then Spring logged Retries exhausted from RejectAndDontRequeueRecoverer, and the message appeared in orders.created.dlq carrying an x-death header with reason: rejected.
To keep the failure reason with the message, register a RepublishMessageRecoverer bean. Spring Boot picks a MessageRecoverer bean up automatically. It republishes the failed message to an exchange you choose with the extra headers x-exception-message, x-exception-stacktrace, x-original-exchange and x-original-routingKey, then acks the original. In the lab the message arrived in orders.dlq with x-exception-message: boom and x-original-routingKey: order.eu.created.
- While retrying with sleeps, the listener thread is busy and its prefetched messages wait. Long back-offs at scale mean long stalls. Keep in-process retry for quick, transient glitches (a dropped connection, a lock timeout) and use a delayed-retry queue (chapter 07) or quorum delayed retry (4.3+) for failures that need minutes.
- Concurrency:
concurrencyandmax-concurrencyset the number of consumer threads per listener. With single active consumer only one of them receives messages. - Observability: name your connections (
spring.rabbitmqhas a connection name strategy) so they are identifiable in the management UI. - Testing: use a real broker with Testcontainers (
RabbitMQContainer) rather than mocking the template; most bugs live in the topology and the acknowledgment flow.
A Local Lab with Docker: Route, Reject, Retry, Observe
This lab runs RabbitMQ in Docker, builds a small topology with the CLI tools that ship inside the image, and shows routing, dead lettering, TTL retry, policies and metrics. The outputs below were captured from a real run on rabbitmq:4-management, which was RabbitMQ 4.4.0 that day; tags move, so your version string may differ. Some outputs are shortened and say so.
Port 5672 is AMQP, 15672 the management UI and HTTP API (open http://localhost:15672, login guest / guest, which is acceptable only for a throwaway lab), and 15692 the Prometheus endpoint. The tool we use is **rabbitmqadmin**, the v2 command-line tool (the project is called rabbitmqadmin-ng; the binary is rabbitmqadmin, version 2.35.0 in this image). It talks to the management HTTP API. rabbitmqctl and rabbitmq-diagnostics talk to the node directly.
--non-interactive is the flag for scripts. The declare commands print nothing on success. Now publish three messages: one that matches both patterns, one that matches only order.#, and one that matches nothing.
orders.all (binding order.#) has both routed messages, orders.new (binding order.*.created) has only the first, and the unroutable one was dropped, exactly as chapter 03 predicts. Queue statistics refresh on an interval of a few seconds, so a number read immediately after an action can lag; the output above is abridged to the table rows.
The message left orders.new, passed through orders.dlx and landed in orders.dlq, with the full history in x-death. (rabbitmqadmin get messages is a polling call meant for development; use a real client in applications.)
The message waited in work.retry for its 3 second TTL, expired, was dead-lettered through the default exchange (empty name, routing key work) and arrived in work with reason: expired.
With one node every quorum queue has a single member, so quorum_status shows one leader row. A real cluster shows a leader and followers (chapter 08).
Put the four Java files and application.yml from chapter 09 in a Spring Boot 4.1 project with spring-boot-starter-amqp, set spring.rabbitmq.username and password to the lab credentials, and publish an OrderCreated through OrderPublisher. A message whose handler always throws shows 4 invocations and then arrives in orders.created.dlq; a message with a negative total is rejected on the first failure. Check the queues again:
The numbers were what the lab showed at that moment, with three queues defined; yours will match your own state. This is the same endpoint a Prometheus server would scrape.
docker rm -f rabbitremoves the broker and all its data.
RabbitMQ or Kafka, and the Mistakes to Avoid
This is a decision between two good tools built on different primitives. Page 01 gave the vocabulary: Kafka is a log (consumers keep an offset, messages stay until retention) and RabbitMQ is, in its classic form, a queue (the broker tracks each message until it is acknowledged). RabbitMQ streams add a log-style type, and Kafka offers queue-like features through newer mechanisms, so the lines blur; the comparison below is about the defaults and the centre of gravity of each.
| Question | RabbitMQ | Kafka |
|---|---|---|
| Primitive | Exchanges route into queues; streams as a second model | Partitioned, replicated, append-only log |
| Routing | Rich and broker-side: direct, topic, headers, fanout, plugins | Topic plus key; consumers filter themselves. Routing logic lives in applications or in stream processing |
| Per-message control | Ack, nack, requeue, TTL, priority, delayed retry, dead lettering per message | None per message; commit an offset. Retry and DLQ patterns are built on top (page 04) |
| Consumption | Broker pushes with prefetch; competing consumers | Consumers pull; one consumer per partition within a group |
| Replay | Not for queues. Yes with streams | Yes: reset offsets within retention |
| Many independent readers of the same data | A queue per reader (cheap fan-out), or one stream for many readers | Native: each consumer group has its own offsets |
| Ordering | Per queue, with caveats (competing consumers, redelivery); SAC or consistent hash for per-key | Per partition, strong and simple, by key |
| Scaling out | More queues, consumers, nodes; a single queue is limited to one leader node | More partitions; throughput scales with partition count |
| Typical retention | Until consumed (queues); by age or size (streams) | Time or size, independent of consumption |
| Operations | Fewer moving parts for small clusters; know alarms, prefetch, queue types | Partition sizing, rebalancing, retention and broker disk planning |
| Strong fit | Task queues, RPC-style flows, complex routing, per-message retry and delay, mixed protocols (AMQP, MQTT, STOMP) | Event streaming, event sourcing, analytics pipelines, high-volume ingestion, stream processing, long retention |
Neither answer is wrong in every case, and using both is common: Kafka as the event backbone, RabbitMQ for the task and workflow queues that hang off it. Page 07 shows patterns (outbox, saga) that sit above either, and page 06 covers the AWS equivalents.
| Mistake | What you see | Fix |
|---|---|---|
| Unbounded prefetch (0, or a huge value) | Consumer memory climbs; one consumer hoards the queue while others idle; a crash redelivers thousands of messages | Set prefetch per consumer from handler duration (chapter 06); Spring default is 250 |
| Auto-ack for important work | Messages vanish when a consumer crashes or a handler throws; no trace anywhere | Manual or container ack with a DLX |
| Unacked pile-up | messages_unacked large and flat; channel closed by the 30-minute ack timeout; redelivery bursts | Always ack or nack on every path; find the stuck handler; bound prefetch |
| No DLX / no delivery limit | Poison message loops forever, or rejected messages are silently dropped | DLX on every work queue; quorum delivery-limit; park and alert |
| Too many queues, channels or connections | Memory and CPU rise with the count; connection churn; slow management UI | Reuse connections; one channel per thread; avoid a queue per user or per request; use routing and consumer-side filtering instead |
| Classic queue where HA is needed | Queue lost with its node | Quorum queues (or streams) |
| Ignoring unroutable messages | Publisher says OK, nothing arrives | mandatory plus returns, or an alternate exchange |
| Non-idempotent consumers | Duplicate side effects after redelivery or republish | Deduplicate on a message id; make operations idempotent (page 01) |
| Publish and consume on one connection | When the broker blocks publishers, acks stall too and the backlog never drains | Separate connections |
| Settings hard-coded as queue arguments | PRECONDITION_FAILED on redeploy when a value changes | Policies for tunable settings |
| Queue as a database | Millions of messages parked for days; memory alarms; slow recovery | Store state in a database; a queue is a buffer, not a store |
Questions You Should Be Able to Answer
Crisp answers for the questions that come up in interviews on this topic.
- Q: Explain the AMQP 0-9-1 model.
Producers publish messages with a routing key to an exchange. The exchange applies its bindings to route copies into queues. Consumers read from queues over channels that are multiplexed on a connection. Everything lives inside a virtual host. Producers never write to a queue directly. - Q: What exchange types exist and when do you use each?
Direct (exact key match, classification), fanout (broadcast to all bound queues), topic (pattern match on dotted keys with*for one word and#for zero or more), headers (match on message headers withx-matchall or any). The default exchange is a direct exchange where each queue is bound under its own name. - Q: What do
*and#mean in a topic binding?*matches exactly one word and#matches zero or more words, where words are dot-separated.order.*.createdmatchesorder.eu.createdbut notorder.created;order.#matches both and alsoorder. - Q: Classic, quorum and stream queues: how do you choose?
Quorum for important work queues because it is replicated with Raft and never loses confirmed messages while a majority is up. Stream for a replicated log with replay and many non-destructive readers. Classic for transient, non-critical, single-node cases only. - Q: What changed in RabbitMQ 4.0?
Classic queue mirroring was removed, so replication means quorum queues or streams. Quorum queues got a defaultdelivery-limitof 20. AMQP 1.0 became natively supported. Classic queue storage version 1 was removed. The Raft-based Khepri metadata store became fully supported. - Q: How does a quorum queue provide durability?
It runs Raft: the leader appends a message to its log and replicates it to followers; once a majority (N/2+1) has it, the message is committed and the publisher is confirmed. If the leader fails, a follower with the committed log becomes leader. Tolerates the loss of a minority of members. - Q: Difference between durable and persistent?
Durable describes the queue or exchange definition surviving a restart. Persistent describes a message being written to disk. A classic queue needs both to keep messages across a restart. Quorum queues and streams always do both. - Q: How do you avoid losing messages when publishing?
Durable (or quorum) queue, persistent messages, publisher confirms to learn the broker accepted them, and the mandatory flag or an alternate exchange so unroutable messages are noticed. Handle nacks and timeouts by republishing, and make consumers idempotent because that creates duplicates. - Q: What are acks, nacks and requeue?
basic.ackremoves a message after successful handling.basic.nackorbasic.rejectwithrequeue=trueputs it back; withrequeue=falseit is dead-lettered or discarded. Unacked messages are requeued when the channel closes. Auto-ack gives at-most-once and is unsafe for important work. - Q: What is prefetch and how do you pick a value?
The limit on unacknowledged deliveries per consumer (basic.qos). Higher values raise throughput and memory use and reduce fairness; 1 is fair but slow. Start small (10 to 50 for short handlers, 1 to 5 for long ones), and measure; the docs say 100 to 300 usually gives optimal throughput. Unlimited prefetch is dangerous. - Q: How does dead lettering work and what is
x-death?
A queue republishes a message to its DLX when it is rejected without requeue, expires (TTL), is dropped by a length limit, or exceeds a quorum queue delivery limit.x-deathrecords queue, reason, count and times so a consumer can count attempts. Use at-least-once dead-lettering on quorum queues if dead letters must not be lost. - Q: How do you implement delayed retry?
Classic pattern: dead-letter failures to a retry queue with a message TTL whose DLX points back to the work queue, and countx-deathto give up after N attempts. On RabbitMQ 4.3+ quorum queues support native delayed retry with linear back-off (delayed-retry-type,delayed-retry-min,delayed-retry-max). - Q: How do you preserve ordering with several consumers?
Competing consumers lose ordering. Use single active consumer for strict order on one queue with failover, and shard by key across several queues with the consistent hash exchange, each queue with its own single active consumer, to keep per-key order while scaling. - Q: What do flow control and the memory alarm do?
Flow control slows a connection that publishes faster than queues, disks or replication can keep up. The memory alarm (default 60 percent of RAM) and the disk alarm block all publishing connections until cleared; consumers keep running so the backlog can drain. Use separate connections for publishing and consuming. - Q: RabbitMQ or Kafka?
RabbitMQ for task queues, per-message retry and delay, rich routing and multi-protocol messaging. Kafka for event streams, replay by many consumer groups, very high volume and stream processing. RabbitMQ streams and Kafka's newer features blur the line, so pick on workload and team experience rather than slogans. - Q: How do you monitor RabbitMQ?
Prometheus plugin on port 15692 plus Grafana, the management UI for inspection. Watch queue depth (messages_ready), unacked count, publish/deliver/redelivery rates, connection and channel counts, memory and disk headroom, alarms, and the dead letter queue depth. Health checks viarabbitmq-diagnostics.
Cheat Sheet and Recap
| Concept | Remember | Value (RabbitMQ 4.x) |
|---|---|---|
| Routing | Producer to exchange to bindings to queues | Types: direct, fanout, topic, headers; default exchange "" |
| Topic patterns | * = one word, # = zero or more | Words are dot-separated |
| Default queue type for important work | Quorum (Raft, majority) | Group size 3 by default; odd sizes |
| Classic queues | Not replicated; mirroring removed | 4.0 |
Quorum delivery-limit | Poison message protection | Default 20 since 4.0; -1 disables |
| Reliable publish | Durable or quorum + persistent + confirms + mandatory | Confirm = safe in queue, not consumed |
| Reliable consume | Manual or container ack, idempotent handlers | Auto-ack = at most once |
| Prefetch | Per-consumer cap on unacked deliveries | Docs: 100 to 300 usually best; Spring default 250; quorum cap 2,000 |
| Dead letter reasons | rejected, expired, maxlen, delivery_limit | x-death header, most recent first |
| TTL | Per queue message-ttl, per message expiration (string), queue expires | Lower value wins; expiry acts at the queue head |
| Delayed retry | TTL + DLX loop; native on quorum queues | Native from 4.3 |
| Priority | Classic x-max-priority (2 to 4 recommended); quorum strict 0 to 31 | Quorum strict from 4.3 |
| Ordering | Single active consumer; consistent hash for per-key sharding | x-single-active-consumer, x-consistent-hash |
| Alarms | Memory and disk alarms block publishers | Memory 0.6 of RAM, disk 50 MB by default |
| Ports | AMQP, management, Prometheus | 5672, 15672, 15692 |
| Delivery ack timeout | Channel closed if a delivery is not acked in time | 30 minutes default; quorum queues only from 4.3 |
- A smart broker: routing, acks, redelivery, TTL, priority and dead lettering are broker features, which keeps consumers simple.
- Exchanges decide, queues store. Choose the exchange type for the routing you need; topic exchanges cover most event routing.
- Quorum queues are the default for data you care about, classic queues are no longer replicated, and streams give a replayable log.
- Reliability needs three switches: durable and persistent (or quorum), publisher confirms with mandatory, and manual acknowledgments with idempotent consumers.
- Bound prefetch. It trades throughput for fairness and memory; unlimited prefetch and auto-ack are the two classic ways to lose control.
- Every work queue needs a DLX and a retry story: delivery limit, delayed retry, and an alert on the dead letter queue.
- Operate it: odd cluster sizes, alarms understood, separate publish and consume connections, and Prometheus metrics for depth, unacked and rates.
- RabbitMQ and Kafka solve different problems: queues with per-message control versus a replayable log. Many systems use both.
Checked on 2026-10-31 against the RabbitMQ documentation sources for the current 4.x series (quorum queues, classic queues, streams, queues, confirms, consumers, consumer prefetch, dead letter exchanges, TTL, priority, flow control, memory and disk alarms, clustering, metadata store, Prometheus, management CLI, deprecated features list, consistent hash exchange README) and the Spring AMQP reference and Spring Boot 4.1.2 configuration metadata. The Docker lab ran rabbitmq:4-management reporting 4.4.0, rabbitmqadmin 2.35.0; the Spring code was compiled and run with Spring AMQP 4.1.2 and Spring Boot 4.1.2. Features tied to a minor version (delayed retry, strict quorum priorities, ack timeout for quorum queues only, transient non-exclusive queues denied by default: 4.3) should be re-checked against your broker version. Not verified: the stream protocol port and client details, and anything about Kafka share groups. See the Kafka pages for current Kafka behaviour.
- 06 AWS Messaging - SQS, SNS and EventBridge: the managed counterparts of queues, fan-out and routing.
- 07 Event-Driven Patterns - outbox, saga and idempotency patterns that sit on top of RabbitMQ or Kafka.
- 08 Reliability & Operations - retries, DLQs and incident handling across brokers.
- Messaging Fundamentals - the delivery guarantees and vocabulary used throughout this page.
- Kafka & RabbitMQ CLI cheat sheet - the commands from this page on one sheet.