Lag in offsets is what Kafka tools print: log end offset minus the group's committed offset. It is easy to read and almost useless for urgency on its own, because 50,000 messages of lag is a few seconds on a topic taking 20,000 messages per second and a day on one taking 2 per second.

Lag in time is how old the record at the consumer's position is: now - timestamp of the record being processed. That is what a user experiences as delay. Kafka does not publish it as a single broker metric. You derive it: either the consumer records now - record.timestamp() as it processes (cheap, accurate, and a good SLO input), or a lag exporter estimates it from offsets and their timestamps. Dashboard both: offsets show how much, time shows how bad.

Read it by column. All partitions lagging points at slow processing or too few consumers. One partition lagging points at a hot key or a stuck consumer holding that partition. Lag that grows only right after a restart points at a rebalance (chapter 6). --state shows whether the group is stable or rebalancing.

The --under-replicated-partitions, --under-min-isr-partitions and --unavailable-partitions options of kafka-topics.sh --describe filter the output to just the partitions in trouble, which makes them good first commands in an incident and good health checks before the next step of a rolling restart (chapter 7).

• Producer: record-error-rate and record-retry-rate (are sends failing or being retried), request-latency-avg, and buffer-available-bytes (when the buffer is exhausted, send() blocks up to max.block.ms, default 60 s, which looks like a hung request thread).
• Consumer: records-lag-max, last-poll-seconds-ago compared to max.poll.interval.ms (default 5 minutes: a consumer that does not poll in time is removed from the group), and in the coordinator metrics rebalance-rate-per-hour and failed-rebalance-total.
• These live in your application process, so export them through your metrics library (Micrometer exposes the Kafka client metrics for Spring apps, page 04).

RabbitMQ reports queue and node state through the management HTTP API and the Prometheus plugin. The queue counters to know, using the names from the RabbitMQ monitoring documentation (4.x):

The two alarms deserve emphasis because of what they do to producers. While either alarm is active, RabbitMQ stops reading from connections that publish, so producers see their calls hang rather than fail, and consumer-only connections keep working. A queue that is growing because of slow consumers can therefore turn into a publisher outage when memory or disk runs out. Alert on the alarm itself, and on queue growth long before it.

check_if_node_is_quorum_critical exits non-zero if stopping this node would cost any quorum queue, stream or internal component its online quorum. It is the gate before restarting a node (chapter 7).

SQS publishes CloudWatch metrics in the AWS/SQS namespace with a single dimension, QueueName. These are the ones that matter:

Because SQS metrics are only reported while the queue is active, an empty queue can show missing data instead of zero. Alarms on these metrics should set how missing data is treated (notBreaching for age and depth alarms is the usual choice) or they will sit in INSUFFICIENT_DATA exactly when everything is fine.

This CloudWatch alarm fires when the oldest message in the queue orders has been older than five minutes for five consecutive one-minute periods, and stays quiet when the queue is empty and the metric has no data. The flags are those of aws cloudwatch put-metric-alarm; the SNS topic ARN is a placeholder.

• Top row, per important queue or consumer group: age or time-lag, depth or offset lag, consume rate versus produce rate, DLQ depth. This row answers "is anything late?" in five seconds.
• Second row, per cluster: under-min-ISR and offline partitions, URP, ISR shrink rate, request handler idle, produce and fetch latency, disk used. This row answers "is the platform why?".
• Third row, per consumer app: processing time, error rate, retries, rebalance count, JVM pauses. Often the real cause lives here, not in the broker.
• Put produce rate and consume rate on the same chart. The gap between the lines is the backlog growing or shrinking, and it is the quickest way to see whether a backlog will drain.

Capacity is not a one-time sizing exercise (page 02, chapter 09 gave the starting arithmetic). It is a loop you run on a schedule and after every large change in traffic.

A worked example, with made-up numbers for illustration: a consumer takes 40 ms per message, one thread, and peak arrival is 600 messages per second. One thread handles 25 messages per second, so you need at least 24 concurrent threads at peak, and with a safety margin of 50% about 36. On Kafka that also means at least 36 partitions for that topic, because one partition feeds only one consumer in a group. On SQS the same arithmetic gives the number of concurrent pollers; on RabbitMQ, consumers times prefetch.

An SQS dead-letter queue must be in the same account and Region as its source queue. For a standard queue the original enqueue timestamp is kept when a message moves to the DLQ, and expiry is computed from it, so give the DLQ a longer retention period than the source queue or messages can expire in the DLQ before anyone looks at them.

Received messages become invisible for the visibility timeout and then reappear, so this is a safe peek as long as you do not delete them. Read the body and the attributes (ApproximateReceiveCount, SentTimestamp) to understand the failure.

When the fix is deployed, use the message move task (the API behind the console's "Start DLQ redrive"). It runs asynchronously in the background, moves messages from the DLQ to the original source queue by default or to another queue of the same type, and takes a maximum rate.

• The documented maximum custom rate is 500 messages per second, and the guidance is to start with a small value and increase it while watching the source queue.
• Only one active move task per queue is supported, a task can run for at most 36 hours, and you cannot filter or modify messages during a redrive. To fix a message before replay you must read it, correct it and send it yourself.
• Redriven messages are new messages: they get a new message ID and enqueue time, and the retention period starts again. For FIFO queues, redriven messages interleave with new traffic from producers, so strict order across the replay is not guaranteed.
• Cancelling a running task leaves already-moved messages in the destination.

Kafka has no built-in DLQ. Your consumer, or Spring Kafka's dead-letter publishing recoverer (page 04), writes the failed record plus failure headers to a dead-letter topic. Redrive is a small consumer on that topic that republishes to the original topic. Because the original is a log, you can also fix the bug and reset the group's offsets instead (kafka-consumer-groups.sh --reset-offsets, which only prints the plan until you add --execute; the group must be stopped), which replays everything and is only safe for idempotent consumers.

RabbitMQ uses dead-letter exchanges (page 05). Quorum queues also count failed deliveries and, since RabbitMQ 4.0, dead-letter or drop a message after a delivery limit that defaults to 20, exposed in the x-delivery-count header (the limit is set by the delivery-limit policy argument; the 3.13 behaviour of no limit is restored with -1). Redrive from a DLX queue is done with a shovel or a small script publishing back to the source exchange.

A poison message is one that always fails. Its damage depends on the broker: on Kafka it blocks its partition (everything behind it waits), on a queue it burns retries and CPU and can starve the others. The symptoms are a stuck offset on one partition with the consumer alive, or a message with a very high delivery count.

• Detect: one partition with lag growing while the others are fine and the consumer logs the same exception with the same offset; SQS messages with a high ApproximateReceiveCount; RabbitMQ messages with a growing x-delivery-count.
• Contain: make sure retries are bounded and non-retryable errors (deserialization, validation) go to the DLQ immediately, not after the full retry schedule. This is a code and configuration property you set before the incident.
• Unblock (Kafka): the cleanest fix is the dead-letter publish above. In an emergency, with the consumer stopped, advance that partition's offset by one with --reset-offsets --shift-by 1 for the specific topic and partition, after saving the record somewhere. That is data loss for that record: write it down in the incident log.
• Prevent: validate on the producer side, version your schemas, and test consumers against malformed input (chapter 9).

Every time group membership changes, partitions are reassigned and, with the classic protocol, processing stops for the affected consumers while it happens. One rebalance is normal. A storm is rebalances back to back: lag grows, throughput collapses, and the system looks "up" while doing no work.

Kafka 4.0 also made the new consumer group protocol (KIP-848, group.protocol=consumer) generally available. It moves assignment to the broker and removes the global "stop the world" synchronisation of the classic protocol, so rebalances stop pausing unaffected consumers. The default is still classic in 4.0, so you opt in per consumer, and some classic settings such as session.timeout.ms are not supported with it. Test before adopting it.

Page 01 defined backpressure as the consumer slowing the producer down. In a real system you pick, in advance, which of three things gives way when load exceeds capacity. If you do not choose, the system chooses for you, usually with an outage.

• Set a maximum useful age. A "send the verification code" message that is 20 minutes old is worse than none. Put a TTL on it (SQS retention or per-message TTL in RabbitMQ), or have the consumer check the timestamp and drop it.
• Prioritise by queue, not by hoping. Separate queues or topics for "payments" and "newsletters" so a flood of one cannot starve the other. Kafka has no per-message priority; separate topics is the idiom.
• Make the limit explicit. A consumer with max.poll.records, a bounded thread pool and a bounded in-memory queue applies backpressure by design. An unbounded in-memory queue just moves the failure into your heap.

• Before starting: the cluster is healthy: no under-replicated, under-min-ISR or offline partitions, and no reassignment in progress. Read the upgrade notes for the target version.
• Stop gracefully. With controlled.shutdown.enable=true the broker syncs its logs to disk (fast restart, no log recovery) and migrates partition leadership to other replicas before stopping, which keeps each partition unavailable only for a few milliseconds. Controlled shutdown succeeds only if every partition on the broker has another live replica, which is one more reason for replication factor 3.
• Do not hard-kill (kill -9) a broker unless it is hung: it skips both optimisations, and it leaves followers to time out of the ISR.
• Start it and wait. The broker rejoins as a follower and catches up. Wait until under-replicated partitions are back to 0 before touching the next broker.
• Restore leadership balance (optional). auto.leader.rebalance.enable (default true) moves leaders back to their preferred replicas over time; kafka-leader-election.sh --election-type preferred --all-topic-partitions does it immediately.
• Finish an upgrade. After all brokers run the new version and you have verified behaviour, finalize with kafka-features.sh upgrade --release-version 4.0 (the Kafka 4.0 upgrade notes' example). Upgrade clients separately and one at a time; clients must be on 2.1 or higher before a 4.0 broker upgrade.

Give systemctl a long enough stop timeout: a broker with many partitions needs time to finish controlled shutdown, and a service manager that kills it after 90 seconds defeats the point. In KRaft clusters with separate controller nodes, restart the controller quorum the same way, one node at a time, and never take down enough controllers to lose the quorum majority.

RabbitMQ's rolling upgrade is also node by node, and the dangerous part is the same: quorum. A quorum queue stays available only while a majority of its replicas are online. In a 3-node cluster, restarting node B while node A is still down leaves some queues with one of three replicas and no quorum.

• Gate: run rabbitmq-diagnostics check_if_node_is_quorum_critical against the node you are about to stop. A non-zero exit means stopping it would cost some queue, stream or internal component its quorum. When automating, rabbitmq-upgrade await_online_quorum_plus_one blocks the shutdown until enough nodes are up.
• Maintenance mode: rabbitmq-upgrade drain transfers queue primaries off the node and stops it accepting client connections; rabbitmq-upgrade revive brings it back to regular duty.
• Pre-checks from the RabbitMQ upgrade guide: no alarms in effect, no ongoing queue or stream replica synchronisation, all stable feature flags already enabled, and a supported rolling-upgrade path between the two versions. After the last node, enable the new stable feature flags.
• Leader balance: after a rolling upgrade, queue leaders are unevenly spread. rabbitmq-queues rebalance quorum redistributes them.

Use SASL_SSL so credentials and data are both encrypted: SCRAM, for instance, must only be used over TLS, as the Kafka documentation stresses. Enable the authorizer and deny by default: the broker setting allow.everyone.if.no.acl.found=true makes a resource with no ACLs open to everyone, which is the wrong default for production.

Keystore and key passwords are omitted on purpose: load them from a secrets store, never from a committed file. Grant each service the least it needs with the convenience options --producer and --consumer, which expand to the operations those roles require (a consumer also needs a group):

• **Delete the default guest user** or restrict it; by default it can only connect from localhost, but it is a well-known credential.
• One user per application, one virtual host per environment or tenant, with the narrowest permission patterns: rabbitmqctl set_permissions -p orders-prod billing "^billing\." "^billing\." "^orders\." (configure, write, read regular expressions on resource names).
• Use TLS on port 5671 and, where possible, verify client certificates (ssl_options.verify = verify_peer, ssl_options.fail_if_no_peer_cert = true).
• Do not publish the management UI to the internet; put it behind your SSO or a private network.

• IAM, not credentials in config. Give each service a role that allows only sqs:ReceiveMessage, sqs:DeleteMessage, sqs:ChangeMessageVisibility (consumer) or sqs:SendMessage (producer) on that one queue ARN. Redrive needs additional permissions: see the "queue permissions for dead-letter queue redrive" section of the SQS guide.
• Encrypt with SSE. SSE-SQS uses SQS-managed keys; SSE-KMS uses a KMS key you control, which adds auditability and key policy as a second access control, at the price of KMS calls and a dependency on key availability. Producers and consumers then need KMS permissions too, and a queue that uses the default AWS-managed key cannot be used by other accounts for some integrations (see the SSE page of the SQS guide).
• Resource policies for fan-out: an SNS topic can only send to an SQS queue if the queue policy allows sns.amazonaws.com for that topic ARN. A missing statement is the commonest reason "the subscription is confirmed but nothing arrives".

Two numbers define the problem. RPO (recovery point objective) is how much data you can afford to lose; RTO (recovery time objective) is how long you can be down. Copying messages to another region is almost always asynchronous, so RPO is greater than zero: the last few seconds before the disaster may not have arrived. Decide whether that is acceptable before you buy complexity.

• Practise the failover. A DR plan that has not been run is a hypothesis. Run a game day at least once a year: promote the secondary, repoint a consumer, measure the time and the lost tail, and write down what surprised you.
• Consumers must be idempotent to be safely repointed: after a failover a consumer replays from translated offsets that may be slightly behind where it really was.
• Keep configuration as code so the secondary has the same topics, ACLs, DLQs and alarms as the primary, not a hand-built approximation.

Reliability that has never been tested is a belief. Messaging systems fail in specific ways, so test those specific ways rather than only "does it work".

For Kafka, kafka-producer-perf-test.sh generates load with a controlled rate and reports throughput and latency. In the 4.0 source its main options are --topic, --num-records, --record-size (or a payload file), --throughput (messages per second, required; -1 disables throttling), and --producer-props such as bootstrap.servers=... and acks=all. kafka-consumer-perf-test.sh is the consumer-side counterpart (--bootstrap-server, --topic, --messages).

• Test with **the same replication factor, acks, compression and message size as production**. A benchmark with acks=1 and tiny messages tells you nothing about your real limits.
• Run the load test while killing a broker. Steady-state numbers are the easy part; the interesting number is throughput and latency during recovery.
• Use a throwaway topic and delete it afterwards. Never load-test against a shared production topic.
• For application-level chaos (kill, delay, drop) use whatever fault-injection tool your platform already has; the scenarios above matter more than the tool.