Messaging · Map
The Messaging Map
Messaging is how services talk without waiting for each other. Every product in this space is one of three shapes, a queue, a publish-subscribe topic or a retained log, and picking the right shape matters more than picking the right brand. This page maps the landscape, compares the guarantees side by side, gives a decision guide, and lays out the pages that follow.
3
Shapes
8
Technologies
9
Pages planned
4
Questions to choose
Map
·
Fundamentals
·
Kafka Architecture
·
Kafka Clients
·
Kafka + Spring
·
RabbitMQ
·
AWS
·
Patterns
·
Operations
·
Interview
01
Chapter One - The Map
Three Families of Brokers
There are dozens of messaging products, but they fall into three shapes. Knowing the shape tells you most of what a product will and will not do, before you read any documentation.
Three shapes of messaging
| Shape | Reading a message | Consumers | Strengths | Examples |
|---|---|---|---|---|
| Queue | Removes it for everyone | Compete: each message processed by one consumer | Job distribution, retries, dead-letter queues, per-message control | Amazon SQS, RabbitMQ queues, ActiveMQ |
| Publish-subscribe | Copies it to every subscriber | Each subscriber receives every message | Fan-out, loose coupling, routing by topic or filter | Amazon SNS, EventBridge, RabbitMQ topic/fanout exchanges |
| Log (stream) | Does not remove it; moves a reader's offset | Independent groups, each reads everything at its own pace | Replay, huge throughput, event history, stream processing | Apache Kafka, Amazon MSK, Kinesis, RabbitMQ Streams, Redis Streams |
Real systems combine shapes. A very common AWS pattern is SNS fan-out to several SQS queues: pub-sub on the front for fan-out, a queue per consumer for buffering, retries and DLQs. Kafka covers the first and third shapes in one product, and RabbitMQ implements all three.
Key insight: Ask one question first: does the consumer need to see history? If a new service must be able to read what happened last week, you need a log. If you are handing out jobs that are finished and forgotten, you need a queue. If you are announcing something to whoever is listening, you need pub-sub.
02
Chapter Two - The Technologies
What Each One Is
| Technology | Shape | Operated how | In one sentence |
|---|---|---|---|
| Apache Kafka | Log | Self-managed, Confluent Cloud, or Amazon MSK | A distributed, replicated, partitioned commit log for high-volume event streams with replay |
| RabbitMQ | Queue + pub-sub (+ streams) | Self-managed, Amazon MQ, CloudAMQP and others | A broker with flexible routing (exchanges, bindings), per-message acks, TTL, priorities and request-reply |
| Amazon SQS | Queue | Fully managed, serverless | A simple, practically unlimited queue: Standard (at-least-once, best-effort order) or FIFO (ordered, deduplicated) |
| Amazon SNS | Pub-sub | Fully managed | Topics that push to SQS, Lambda, HTTP, email and SMS subscribers, with filter policies and FIFO topics |
| Amazon EventBridge | Event bus (pub-sub) | Fully managed | A serverless bus with rules that match event content, route to targets, run schedules and ingest AWS and SaaS events |
| Amazon Kinesis Data Streams | Log | Fully managed | A managed stream of shards, ordered per partition key, with retention (24 hours by default, extendable) and replay |
| Amazon MSK | Log | Managed Kafka | Kafka brokers run for you on AWS; you keep the Kafka API and the design decisions |
| Amazon MQ | Queue + pub-sub | Managed ActiveMQ or RabbitMQ | For lifting existing JMS/AMQP applications to AWS without rewriting them |
⚠ Choosing by popularity
- Kafka is the most talked-about, and it is also the most expensive to get wrong: partitions, consumer groups, rebalancing and retention all need to be understood to run it well. A team that only needs "process these jobs reliably" is better served by SQS in an afternoon. Pick the simplest shape that covers the requirement.
03
Chapter Three - Compare
Guarantees Side by Side
Each cell below is a statement about the default or standard configuration. Limits change, so treat the table as orientation and check the vendor documentation for any number you will rely on.
| Kafka | RabbitMQ | SQS Standard | SQS FIFO | SNS (Standard) | Kinesis | |
|---|---|---|---|---|---|---|
| Order | Per partition | Per queue (one consumer) | Best-effort only | Per message group | None | Per shard (partition key) |
| Delivery | At-least-once; exactly-once inside Kafka with transactions | At-least-once with publisher confirms and consumer acks | At-least-once | Deduplicated sends (5-minute window); process-once semantics | At-least-once | At-least-once |
| After read | Retained | Deleted on ack (streams retain) | Deleted by consumer | Deleted by consumer | No storage; push with retries | Retained |
| Replay | Yes (offsets) | Only with streams | No | No | No (archive via EventBridge) | Yes |
| Retention | 7 days default, configurable | Until consumed | 4 days default; 60 s to 14 days | Same | n/a | 24 h default, extendable |
| Throughput | Very high, scale by partitions | High, scale by queues and nodes | Nearly unlimited | 300 API calls/s per partition, ten times that batched; high-throughput mode available | Very high | Per-shard limits |
| Message size | 1 MiB default (message.max.bytes) | Configurable | 1 MiB | 1 MiB | See quotas | See quotas |
| Ops burden | High (or pay for MSK/Confluent) | Medium | None | None | None | Low |
Those SQS values come straight from the AWS quota pages; the Kafka values from the Kafka 4.0 configuration reference. The remaining pages in this section go deep on each technology and repeat the numbers where they matter, with their sources.
04
Chapter Four - Choose
A Decision Guide
Start from the requirement
| If you need... | Reach for | Because |
|---|---|---|
| Hand out background jobs with retries and a DLQ, on AWS | SQS (Standard unless order matters) | Zero operations, native redrive policy, scales automatically |
| Strict order per customer or per order, with deduplication, at moderate rates | SQS FIFO with a message group id | Order and dedup are built in; groups give parallelism |
| Announce an event to several independent services on AWS | SNS to SQS fan-out, or EventBridge if you need content-based rules | Each consumer gets its own queue for buffering and failure isolation |
| A durable event stream, many consumers, replay, hundreds of thousands of events per second | Kafka / MSK, or Kinesis if you want fully managed and AWS-only | The retained log is the architecture |
| Stream processing: joins, windows, aggregations over events | Kafka with Kafka Streams or Flink | The ecosystem is built around the log |
| Complex routing, per-message TTL, priorities, request-reply, protocol variety (AMQP, MQTT, STOMP) | RabbitMQ | Exchanges and bindings give routing no log-based system offers |
| React to AWS service events, SaaS webhooks, or scheduled triggers | EventBridge | Native integrations, rules, schedules and archive/replay |
| Move a JMS application to AWS unchanged | Amazon MQ | Same protocol, managed servers |
Four questions that settle most choices
- Replay: will a future consumer need yesterday's events? (Yes means a log.)
- Order: what exactly must be ordered, and per which key?
- Volume: events per second at peak, and how large? (Thousands: almost anything works. Hundreds of thousands: think log.)
- Operations: who runs it at 3 a.m.? A managed service is a feature.
05
Chapter Five - Putting It Together
Combining Shapes
Mature systems rarely pick one broker for everything. A common split: facts go on a log, work goes on a queue. The order service writes to its database and, through an outbox (page 07), publishes OrderPlaced to a Kafka topic. Services that care subscribe as consumer groups. A consumer whose handling is a discrete, retryable job, such as "generate invoice PDF", puts that job on an SQS queue, where visibility timeouts, redrive and DLQs are first-class.
A typical combination: a log for facts, queues for work
- Do not fan one event to many services by chaining queues; use a topic or log so the producer stays ignorant of consumers.
- Do not store business truth only in a topic with short retention. The system of record is a database; the log is how changes travel.
- Do give every consumer its own consumer group or queue, so one slow consumer never affects another.
06
Chapter Six - The Learning Path
Pages in This Section
The pages build on each other. Read Fundamentals first; the rest can be taken in the order your project needs.
- 01 · Messaging Fundamentals - delivery guarantees, idempotency, ordering, lag, retries, DLQs, plus a plain-Java at-least-once queue
- 02 · Kafka Architecture - the log, partitions, replication, KRaft, retention, sizing, a hands-on lab
- 03 · Kafka Producers & Consumers - batching, idempotence, transactions, consumer groups, offsets, rebalancing
- 04 · Kafka with Spring -
KafkaTemplate, listeners, error handling, retry topics, dead-letter topics, testing - 05 · RabbitMQ - exchanges, bindings, quorum queues, acks, dead lettering, streams
- 06 · AWS Messaging - SQS, SNS, EventBridge, Kinesis and MSK in practice
- 07 · Event-Driven Patterns - outbox, saga, idempotency, CQRS, event sourcing
- 08 · Reliability & Operations - lag, DLQs, poison messages, monitoring, capacity, incident playbooks
- 09 · Interview Q&A - questions across all of the above
Related sections
- Spring Guide - Spring Kafka, transactions and resilience build on the Spring container.
- Performance Engineering - capacity, concurrency and database limits that decide consumer throughput.
- System Design - where messaging sits in larger architectures.
07
Chapter Seven - Summary
Cheat Sheet and Recap
Cheat sheet
| Question | Answer |
|---|---|
| Which shape? | History or replay: log. Jobs: queue. Announcements: pub-sub |
| Default delivery guarantee | At-least-once: make consumers idempotent |
| Where does order exist? | Per partition (Kafka), per message group (SQS FIFO), per shard (Kinesis), per queue with one consumer (RabbitMQ) |
| Simplest on AWS | SQS for work, SNS to SQS for fan-out |
| Most powerful stream platform | Kafka (or MSK), with the operational cost that implies |
| Most flexible routing | RabbitMQ |
| Never forget | Dead-letter queue with an alert; bounded retries with backoff and jitter |
∑ The messaging map - what to remember
- Three shapes: queue (work), pub-sub (announce), log (history). Most products are one shape plus add-ons.
- Pick the simplest shape that meets the requirement, and prefer managed services unless you need what only a log gives you.
- Every delivery is at-least-once in practice. Idempotent consumers are not optional.
- Order is per key. Choose the key deliberately.
- Facts on a log, work on a queue. Keep the database as the source of truth.