Kafka was built at LinkedIn to move activity data between systems, and its central decision was to make the append-only log the thing you talk to, instead of a queue that deletes messages as they are read. A log is simple: records are added at the end, each gets a sequential number (the offset), and nothing is modified. Reading is a separate concern. A reader remembers its own position, and many readers can sit at different positions in the same log.

That one difference explains most of Kafka's behaviour: replay (move the offset back), independent consumers (each group keeps its own offset), retention by time or size rather than by consumption, and very high throughput (sequential appends and sequential reads are the cheapest thing a disk does).

A topic is a named stream. To scale it, a topic is split into partitions, and each partition is its own append-only log. Every record in a partition gets the next offset, a 64-bit sequence number that never changes and is never reused. A record is therefore addressed by topic + partition + offset.

A Kafka cluster is a set of brokers (servers). Each partition is stored on several brokers, controlled by the topic's replication factor. For each partition exactly one replica is the leader and the rest are followers.

One broker, the active controller, manages the cluster: it detects broker failures, elects partition leaders, and publishes the result as cluster metadata. How the controller is elected and where the metadata lives changed fundamentally in recent Kafka versions (chapter 05).

Replication is where Kafka's guarantees come from, and the settings interact, so it pays to understand the moving parts before touching them.

For its first decade Kafka stored cluster metadata (topics, partitions, leaders, configs, ACLs) in Apache ZooKeeper. That meant running and operating a second distributed system, and the controller had to reload all metadata from ZooKeeper after a failover, which limited cluster size and recovery time. KRaft (Kafka Raft) removes the dependency: a set of controller nodes run the Raft consensus protocol and keep the metadata in an internal, replicated Kafka log named __cluster_metadata. Brokers are consumers of that log.

Kafka routinely handles hundreds of thousands of records per second per broker on ordinary hardware, with durable storage on disk. There is no trick; it follows the grain of the machine in five ways.

The partition a record lands in decides its ordering group, its consumer, and the load on the broker. The producer's partitioner chooses it:

Kafka keeps data according to the topic's cleanup.policy, which can be delete, compact, or both.

• Retention deletes a segment only when all records in it are old enough, and never touches the active segment. A quiet topic can therefore keep data longer than retention.ms. Lower segment.ms if that matters.
• Compaction never changes offsets and does not reorder: it only removes superseded records. A consumer reading a compacted topic from the start sees the latest value for every key, plus some older ones that have not been cleaned yet.
• A record with a key and a null value is a tombstone: it marks the key as deleted. It is retained for delete.retention.ms so slow consumers can observe the delete, then removed.
• Compaction requires keys: producing a record without a key to a compacted topic is rejected.

Partition count is the hardest setting to change later, so size it from numbers. Use measured throughput from your own producers and consumers; the figures below are examples of the method, not benchmarks.

Nothing makes the previous chapters stick like watching them happen. This needs only Docker. The official image runs a single combined broker+controller in KRaft mode.

Crisp answers for the questions that come up in interviews on this topic.

• Q: How is Kafka different from a message queue?
  Kafka is a distributed, partitioned, replicated log. Records are retained for a period regardless of consumption, consumers track their own offsets, and many consumer groups can read the same data independently and replay it. A queue deletes a message once acknowledged and tracks delivery per message.
• Q: What is a partition and why does Kafka have them?
  A partition is one ordered append-only log of a topic. Partitions give parallelism (one consumer per partition within a group), scalability (spread across brokers) and the unit of ordering (order is guaranteed only within a partition).
• Q: What is an offset?
  A sequential, immutable position of a record within one partition. A consumer commits the next offset it wants to read, so after a restart it resumes where it left off.
• Q: How does Kafka guarantee ordering?
  Only within a partition. Records with the same key hash to the same partition, so they stay in order for a single consumer of that partition. There is no ordering across partitions.
• Q: Explain leader, follower, ISR and the high watermark.
  Each partition has one leader that serves all reads and writes; followers replicate from it. The ISR is the set of replicas caught up within replica.lag.time.max.ms. The high watermark is the highest offset replicated to all ISR members; consumers read only below it.
• Q: How do you configure Kafka so that you never lose acknowledged data?
  Replication factor 3, min.insync.replicas=2, producer acks=all with idempotence enabled, unclean.leader.election.enable=false, and replicas across racks or zones. This tolerates one broker failure without loss and refuses writes rather than risk loss when two are down.
• Q: What does acks=all mean if min.insync.replicas=1?
  The leader alone can satisfy it when the ISR has shrunk to one member, so it is effectively acks=1. acks=all is only as strong as min.insync.replicas.
• Q: What changed in Kafka 4.0?
  ZooKeeper support was removed; KRaft is the only metadata mode. Controllers run Raft and store metadata in the __cluster_metadata log. The new consumer group protocol (KIP-848) became generally available, and linger.ms default became 5 ms. Brokers need Java 17.
• Q: Why is Kafka fast?
  Sequential disk I/O, reliance on the OS page cache, zero-copy sendfile for plaintext, batching and compression of whole record batches, and horizontal scaling through partitions.
• Q: Delete vs compact cleanup?
  delete removes whole old segments by age or size. compact keeps the latest record per key and removes older ones in the background; tombstones (null values) mark deletions. Compacted topics still contain old duplicates for some time.
• Q: Why is changing the partition count of a keyed topic risky?
  The key-to-partition mapping depends on the partition count, so existing keys can move to different partitions, breaking per-key ordering and any consumer state keyed by partition assignment. Partitions can be added but never removed.
• Q: How many partitions should a topic have?
  Divide target throughput by the measured per-partition throughput of the slower side (usually the consumer), add headroom, and remember it caps consumer parallelism. Too many partitions cost memory, file handles, recovery time and replication overhead.