The pairing is decided by the Spring Boot dependency management, not by you. The table was read from the spring-boot-dependencies POMs on Maven Central on 2026-10-31.

So "Boot 3.x" means spring-kafka 3.3.x and Kafka clients 3.9.x. The brokers you talk to can be newer (a 4.0 broker accepts 3.9 clients). On Boot 4.x the headline differences you will hit first are in testing: spring-kafka 4.0 removed the ZooKeeper-based embedded broker because Kafka 4.0 is KRaft only (chapter 10).

When a KafkaAdmin bean exists (Boot creates one), every NewTopic bean in the context is created on startup if missing. TopicBuilder is the convenient way to declare it:

Block with get() when the caller must not continue unless the broker has the record (a request handler that returns 202 only after the event is stored, for example). Do not call get() per record inside a loop to "make it safe": you serialize the whole producer to one in-flight request at a time and lose batching, which is the main source of Kafka throughput.

• The key decides the partition, hence the ordering scope. Use the entity id (orderId) for everything that must stay ordered per entity. A null key spreads records over partitions with no per-entity order (page 02).
• Headers carry metadata that is not part of the business payload: an event id for deduplication, a correlation id, a schema version. Add them on a ProducerRecord or build a Message with MessageBuilder.

• Each thread polls its own partitions, so ordering inside a partition is preserved: one thread, one partition, records in order.
• With concurrency = 3 on a 6-partition topic each thread owns two partitions. With concurrency = 8 two threads get nothing (diagram above).
• Running two application instances with concurrency 3 each also splits 6 partitions into 6 consumers. Total consumers across all instances should not exceed the partition count.
• For several topics the default RangeAssignor can leave threads idle even when there are enough partitions overall; the reference suggests a RoundRobinAssignor through partition.assignment.strategy for that case.

Spring Kafka sets enable.auto.commit to false unless you set it yourself and commits through the container. The ack mode says when. The default is BATCH. Commits use commitSync unless you turn off syncCommits (default true).

All of these are at-least-once: the commit happens after the work, so a crash between work and commit replays records. That is why chapter 8 adds an idempotent listener. Spring never commits before the listener returns in any of these modes.

Acknowledgment also has nack(Duration) (and index variants for batches) that makes the container discard the rest of the polled records, pause for the given time, and redeliver from the failed record. Manual modes require a listener that accepts the Acknowledgment parameter; with MANUAL and MANUAL_IMMEDIATE and async-acks you can even acknowledge out of order, but then see the out-of-order commit section of the reference before relying on it.

A batch listener receives everything one poll() returned as a List. Set spring.kafka.listener.type: batch, factory.setBatchListener(true), or batch = "true" on one annotation. It suits bulk writes (insert 500 rows in one statement).

Throwing BatchListenerFailedException with the failing index lets the DefaultErrorHandler commit the earlier records, retry or recover only the bad one, and redeliver the rest. Any other exception makes it retry the whole batch. Remember the limits: non-blocking retries (@RetryableTopic) do not work with batch listeners, and Micrometer observations do not support them either (timers are used instead).

The JsonDeserializer refuses to build classes from packages it was not told to trust, because a type name read from a message header is attacker-controlled input. With the default trusted list (java.util, java.lang) your first JSON message of type com.shop.OrderPlaced fails with an exception that says the class is not in the trusted packages. The quick fix people find is *. Prefer the narrow fix:

A deserializer runs **inside poll()**, before your listener or any Spring error handler sees the record. If it throws, the consumer cannot return that record, does not advance, and the same record fails again on the next poll: an endless loop at 100 percent CPU that no listener try/catch can touch. ErrorHandlingDeserializer wraps the real deserializer, catches the exception and returns null plus a DeserializationException header that holds the cause and the raw bytes. For a record listener the container then sends the record to the error handler instead of the listener.

Do the same for keys with ErrorHandlingDeserializer and spring.deserializer.key.delegate.class if keys can be malformed. In the lab, a record whose value is the text this is not json landed in the dead-letter topic with its original bytes intact (chapter 6).

After a deserialization failure the record value is the raw byte[]. If the KafkaTemplate that publishes to the dead-letter topic uses JsonSerializer, those bytes get JSON-encoded (base64 inside quotes) and the original payload is damaged. The reference recommends a template of type <Object, Object> with a DelegatingByTypeSerializer that handles byte[] separately. This is the template the lab uses for both the app and the recoverer:

One more thing the lab taught: when you pass a serializer instance that you configured with setters (setAddTypeInfo(false)), it must not also receive spring.json.* properties. Spring Kafka throws JsonSerializer must be configured with property setters, or via configuration properties; not both. The cfg.remove(...) line above is the fix.

With Boot you only declare a CommonErrorHandler bean: Boot attaches it to the auto-configured container factory (the same applies to RecordFilterStrategy, AfterRollbackProcessor and a rebalance listener). This is the configuration used in the lab:

ExponentialBackOffWithMaxRetries(3) retries after 200, 400 and 800 ms (multiplier 2, capped at 2 s) before giving up, so four deliveries in total. A plain ExponentialBackOff retries indefinitely unless you compute a maxElapsedTime, which is why Spring Kafka provides the ...WithMaxRetries subclass.

DeadLetterPublishingRecoverer needs a KafkaTemplate<Object, Object>. By default it publishes to **<original topic>-dlt on the same partition number** as the original record, so the DLT needs at least as many partitions as the source topic. Pass a destination function (record, exception) -> TopicPartition to route differently; a negative partition lets Kafka choose.

In the lab three bad records were sent to orders: one with quantity 0 (throws the non-retryable InvalidOrderException), one that is not JSON, and a duplicate of a good one. This is what the dead-letter records looked like when read back with a plain consumer (abridged: stack traces shortened, binary headers decoded by the test):

• The value is the original payload (for the bad JSON, the original bytes). That is what lets you fix and replay it.
• The record landed in orders-dlt-0 and orders-dlt-1: the same partition numbers as in orders (keys o-2 and o-3 hash differently).
• The headers answer the four questions of an incident: what failed (exception class, message, stack trace), where it came from (topic, partition, offset, timestamp) and who failed to process it (consumer group). Combine topic, partition and offset to find the record in the source topic while retention still holds it.

• The listener fails: the record is published to the first retry topic with a back-off timestamp header, and the main offset is committed.
• A consumer of the retry topic checks the timestamp. If it is not yet due it pauses that partition and resumes when due. It is never processed before its due time; it can be late if the consumer is busy (best effort).
• On failure again the record moves to the next retry topic. When attempts are exhausted it goes to the dead-letter topic (DLT), where an optional @DltHandler method consumes it.
• The framework creates the topics (through KafkaAdmin) and the listener containers for you.

With a main topic payments, attempts = 4 and exponential back-off from 500 ms the framework created exactly these topics in the lab:

The rules from the reference: one retry topic per retry (attempts - 1 of them), suffixed with the delay for varying delays and with the index (-retry-0, -retry-1) for fixed delays. With a fixed back-off you can reuse a single retry topic (sameIntervalTopicReuseStrategy = SINGLE_TOPIC), and since 3.2 an exponential back-off reuses the last topic for every attempt at maxDelay, which makes long retry schedules (hundreds of attempts) practical.

Kafka transactions make a consume, process, produce cycle atomic *within Kafka*: the produced records and the consumed offsets commit together, or neither does (page 03 shows the client API). In Spring you switch it on with one property, and Boot does the wiring.

• Setting transaction-id-prefix makes the producer factory transactional and Boot creates a **KafkaTransactionManager**, wired into the default container factory. The listener then runs inside a Kafka transaction and the consumed offsets are sent to that transaction.
• Each producer in the factory gets transactional.id = prefix + n. With several application instances the prefix must be unique per instance (the reference says so; Spring Kafka 3.x supports only EOSMode.V2, which no longer needs the same id across restarts).
• Consumers see aborted records unless they set isolation.level=read_committed (Kafka default is read_uncommitted).
• Outside a listener, @Transactional methods and template.executeInTransaction(...) start a producer-initiated transaction.

This is the arrangement the reference recommends now that ChainedKafkaTransactionManager is deprecated: the container starts the Kafka transaction through the KafkaTransactionManager, and the @Transactional method starts the database transaction. Name the database manager explicitly: with two PlatformTransactionManager beans an unqualified @Transactional is ambiguous.

The lab is a normal Maven project (Boot parent 3.5.16, dependencies: spring-kafka, JDBC with H2, Actuator; test dependencies spring-kafka-test and Awaitility). The test below starts an in-process broker, so there is no Docker step. It is the whole test for this chapter:

• $ mvn test -Dtest=RetryTopicFlowTest runs it. The first start is slow while the embedded KRaft broker boots.
• The listener trail lives in a thread-safe list, so the test can await the final state instead of sleeping. That is the Awaitility pattern from chapter 10.
• To watch the real topics in a longer-running setup, list them with $ bin/kafka-topics.sh --bootstrap-server localhost:9092 --list and consume payments-dlt with kafka-console-consumer.sh --property print.headers=true to see the headers.

spring-kafka-test provides @EmbeddedKafka, an in-JVM broker. Point Boot at it with bootstrapServersProperty:

• On spring-kafka 3.3 the embedded broker defaults to the ZooKeeper-based implementation; kraft = true (since 3.2) selects KRaft, which is what you want against Kafka 3.9. On spring-kafka 4.0 only KRaft remains and the ZooKeeper attributes are gone.
• Create every topic your flow touches (orders-dlt, retry topics are created by the framework). The DLT needs at least the source topic's partition count when the default same-partition rule applies.
• Use KafkaTestUtils (consumerProps, getRecords, getSingleRecord) to read what your code published. The lab reads the DLT with ByteArrayDeserializer to inspect raw bytes and headers.

• Turning observations on replaces the timers (the timers become part of each observation). Observations are not supported for batch listeners, which keep timers.
• Tracing needs a bridge on the classpath (micrometer-tracing-bridge-otel or -brave) and an exporter, per the Spring Boot tracing page.
• Consumer lag is the number to alert on. In Kafka it appears as records-lag-max in the consumer-fetch-manager-metrics group (Kafka 3.9 monitoring docs); the Micrometer binder exposes it under a dotted kafka.consumer.fetch.manager... name, so check your registry for the exact spelling. From outside the app, describe the group:

The columns are GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID. Output depends on your cluster, so none is pasted here. Rising LAG on all partitions means the group is too slow; on one partition it means a stuck or slow record (or a hot key).