Every service call has a hidden contract: the callee must be up, reachable and fast at the moment of the call. When a request fans out to several services, those requirements multiply. The user waits for the slowest dependency, and the request fails if any one of them fails. This is temporal coupling, and it is the problem a message broker removes: the producer hands a message to a durable intermediary and moves on; consumers pick it up whenever they are able to.

A broker buys four distinct things, and it is worth naming them because each one is a reason you may or may not need messaging in a given design:

Every system in this section, Kafka, RabbitMQ, SQS, SNS, EventBridge, Kinesis, fits the same skeleton: producers write messages, a broker stores and routes them, consumers read them. The products differ in how routing works, how long messages are kept, and who tracks which messages a consumer has seen. The words also differ, which makes first contact with a new broker confusing, so here is a translation table.

"Message" covers three different intents, and mixing them up is the most common design error. The intent decides the name, the number of consumers and the coupling between teams.

A delivery guarantee answers one question: if something goes wrong, can a message be lost, duplicated, or neither? The network cannot tell the difference between "the receiver is slow", "the receiver is dead" and "the receiver finished but its reply got lost", so a sender that wants certainty has to choose which mistake it prefers.

"Exactly-once" deserves care, because vendors use the phrase for narrower things. Kafka offers idempotent producers and transactions that make read-process-write cycles inside Kafka atomic (page 03). SQS FIFO deduplicates identical sends within a 5-minute window. Both are real and useful, but neither extends into your database or an external HTTP call. The moment a consumer writes to a database, sends an email or charges a card, the guarantee is yours to build.

A handler is idempotent when running it twice has the same effect as running it once. You can get there in three ways, and the best designs use the first whenever possible.

Strict global ordering means one lane and one consumer: the next message cannot start until the previous one finished. That is correct and slow. Every real system therefore guarantees order only within a key, and parallelises across keys.

A broker decouples speeds, it does not remove the speed limit. If producers send faster than consumers process, the queue grows until it hits a limit: disk, retention, memory, or the patience of the business. The unprocessed backlog is called lag (Kafka) or queue depth (RabbitMQ, SQS).

Not every failure is the same, and the right reaction depends on the kind. Transient failures (timeout, deadlock, a downstream restart) succeed if tried again. Permanent failures (malformed JSON, a business rule violation, a bug) never will. A message that always fails is a poison message.

Each broker implements these ideas differently: Kafka has no built-in DLQ (you publish to a dead-letter topic yourself or let Spring Kafka do it), RabbitMQ has dead-letter exchanges and quorum-queue delivery limits, and SQS has a redrive policy with maxReceiveCount. Pages 04, 05, 06 and 08 cover the concrete mechanisms.

Building the mechanism once makes every broker's documentation easier to read. This 70-line queue has the three behaviours that define at-least-once delivery: receiving hides a message instead of deleting it, an ack deletes it, and a message that is not acked becomes visible again after a visibility timeout. A receive counter moves poison messages to a dead-letter list. It is the model behind SQS and, with different bookkeeping, RabbitMQ.

Crisp answers for the questions that come up in interviews on this topic.

• Q: Why use a message broker instead of a direct REST call?
  To remove temporal coupling (the consumer need not be up), level load spikes, fan out one event to many consumers and isolate failures. The cost is eventual consistency, harder debugging and the need to handle duplicates and ordering. Use a direct call when the caller needs the answer now.
• Q: Explain at-most-once, at-least-once and exactly-once.
  At-most-once acks before processing: may lose, never duplicates. At-least-once acks after processing: never loses, may duplicate. Exactly-once is not achievable over an unreliable network end to end; in practice it means at-least-once delivery plus an idempotent consumer (or a transaction that covers both the effect and the offset).
• Q: Why can a message be delivered twice even with a perfect broker?
  The ack can be lost, the consumer can crash after processing but before acking, processing can exceed the visibility or session timeout, a rebalance can replay uncommitted work, and a producer can retry after a lost response.
• Q: How do you make a consumer idempotent?
  Prefer naturally idempotent writes (set absolute state, upsert by business key). Otherwise record the message id in a dedup table in the same transaction as the effect, or use a conditional update with a version check.
• Q: Why must the dedup record share a transaction with the business write?
  If they commit separately, a crash between them either skips work that was never done (record first) or repeats work that was done (work first).
• Q: How do you preserve ordering at scale?
  Order within a key, not globally: partition by entity id (Kafka partition key, SQS FIFO message group). One consumer per lane, no parallel processing inside a lane, and make handlers tolerant of stale events with versions.
• Q: What is a poison message and how do you handle it?
  A message that can never be processed successfully. Bound retries, then move it to a dead-letter queue with failure metadata, alert on the DLQ, and provide a way to fix and redrive. Never retry indefinitely in an ordered lane, because it blocks everything behind it.
• Q: Command vs event?
  A command is an instruction to one owner, named in the imperative; the sender expects it done. An event is a past-tense fact published to whoever cares; the sender does not know the consumers and cannot demand action.
• Q: A queue is growing. What do you check?
  Compare arrival and processing rates; check consumer count against partition count (Kafka); check consumer errors and retries; check the downstream dependency (database, API); check for a poison message blocking a lane; only then scale.
• Q: What is the dual-write problem?
  Updating a database and publishing a message are two systems with no shared transaction, so a crash between them leaves them inconsistent. The usual solution is the transactional outbox pattern (page 07).