The symptom, measured with the code in chapter 3: 400 concurrent clients, each request needs a 50 ms downstream PaymentGateway call. With Tomcat limited to 10 request threads:

No errors, no CPU. Just a flat 2 second latency for every request and 198 requests per second. A 50 ms handler became a 2000 ms response. Chapter 3 shows why that is exactly arithmetic, not mystery.

The request-thread pool is the bottleneck: 10 threads can only have 10 requests in flight, the other 390 queue. (Little's Law, see Measure First: in-flight requests L = throughput × latency.)

Tomcat's own view of the pool (tomcat.threads.busy / tomcat.threads.config.max in Micrometer, or a jcmd <pid> Thread.print showing every http-nio-8080-exec-N thread parked in the downstream call) and the load tool's latency histogram. Below we use both a thread dump and the closed-loop client.

All 10 workers busy 100 % of the time and zero idle capacity; the queue is in front of them (in the OS accept backlog / connector, invisible to application timers). Latency measured at the client includes that wait; http.server.requests measured inside the app does not. That gap - client sees 2000 ms, server metric says 50 ms - is the classic fingerprint of an exhausted request pool.

Change exactly one variable: server.tomcat.threads.max. Then, separately, replace the pool with virtual threads. Chapter 3 and 4 do both and compare.

Same client, same handler, same machine. Result teaser: 10 threads → 198 req/s; 50 → 983; 200 → 3,874; virtual threads → 5,950. Full table in chapter 3.

Throughput stalls at 198 req/s and every response takes 2 seconds, with 400 clients and almost no CPU use (chapter 1).

Throughput = threads / service time = 10 / 0.050 = 200 req/s. Latency for 400 closed-loop clients = 400 / 200 = 2.0 s. If the hypothesis is right, 50 threads should give 1000 req/s and 0.4 s, and 200 threads 4000 req/s and 0.1 s.

A closed-loop client: 400 virtual threads, each looping HttpClient.send() over keep-alive connections, 2 s warm-up then 8 s measured. The server and client are in one JVM per run; full source in chapter 10. Command:

The raw result of all four runs (laptop-class 12-core machine, noisy shared load):

Two things stand out. First, the pool sizes match Little's Law to within 3 %: this is a queueing problem, not a code problem. Second, latency has a floor of 50 ms (the handler) and the thread-per-request model only reaches it once threads ≥ concurrency (200 threads, 400 clients: still queueing, latency 103 ms = 2 × 50).

Closed-loop, 400 clients; embedded Tomcat 10.1.30; Java 21.0.2; shared 12-core machine.

Dots are measured averages; the dashed line is the formula.

Setting threads.max to 2000 is the instinctive fix and it often moves the bottleneck downstream: 2000 request threads now hit a database pool of 10, or a PaymentGateway that rate-limits at 100 concurrent calls. The queue merely relocates (from Tomcat's executor to Hikari's getConnection() wait) and the timeouts you set there start firing. Size the pool from the narrowest downstream resource, not from the incoming concurrency:

Even after raising threads.max to 200, 400 concurrent clients still queue (latency 103 ms for a 50 ms handler). To serve 400 in flight we would need 400+ platform threads; at 5,000 concurrent calls, 5,000 threads: stack memory, scheduler pressure, and GC root scanning grow with every one.

If requests spend their time blocked, giving every request its own virtual thread removes the pool as a limit; latency should fall to about the handler time (50 ms) until some other resource saturates.

Same TomcatLab, same client; the only difference is the executor Tomcat uses (the mechanism spring.threads.virtual.enabled=true configures for Boot's Tomcat):

Throughput rose 54 % over the 200-thread pool and average latency fell from 103 ms to 67 ms with no tuning knob at all. p99 (160 ms) is not better than the 200-thread run (148 ms): at this load the machine is also running the client and 12 cores are busy scheduling, so the tail is CPU contention, not queueing. Virtual threads moved the bottleneck from "pool size" to "CPU and downstream".

What it does in Boot 3.2+: the embedded Tomcat protocol handler gets a virtual-thread-per-task executor, and Boot's auto-configured applicationTaskExecutor uses virtual threads. server.tomcat.threads.max then no longer bounds concurrency.

Mechanism diagram (JEP 444). The 12 carriers are this machine's core count.

WebFlux (Reactor, Netty by default) uses a few event-loop threads and never blocks; any blocking call on the event loop stalls every request on it. For OrderService - blocking JDBC/JPA, blocking RestClient calls - WebFlux would force a rewrite of the data layer (R2DBC or wrapping blocking calls on a scheduler) for no throughput gain that virtual threads do not already give.

Judgement framework, not measured. Reactive code wins on streaming and backpressure, not automatically on request/response latency.

GET /customers/{id}/orders returns a list of full Order entities - the graph the ORM loaded: items, both addresses, internal notes and an audit trail. The mobile team complains about 130 KB responses; the profiler shows Jackson in the top frames of the flame graph.

Serialisation CPU and bytes scale with the object graph, not with what the client uses. A slim DTO will cut both by an order of magnitude; gzip will cut bytes further but spends CPU.

Payload.java (chapter 10) serialises 100 orders both ways with Jackson 2.17.2 (the version Boot 3.3.4 manages), gzips the bytes, then times 5,000 iterations after 3,000 warm-up iterations. Run twice because the machine was loaded:

Run 1 on the same loaded machine gave 1,252 us / 88 us / 1,483 us: the absolute CPU numbers move by up to 2x with neighbour load, the ratios (about 10-15x between graph and DTO) do not. Note the surprising line: gzip CPU (about 1.5 ms) is larger than the whole Jackson serialisation (0.7-1.2 ms) for this body, and on a fast network it costs more than the transfer time it saves.

Measured with Jackson 2.17.2 and java.util.zip; synthetic but representative order data.

Two separate changes, in order of payoff. First, return a projection:

Second, decide on compression where it pays - typically at the CDN / load balancer edge for clients on slow links, not in every origin pod:

Computed from the measured byte counts and measured gzip CPU time; transfer time = bytes x 8 / link speed, ignoring latency and TCP effects.

OrderService calls the PaymentGateway once per order. The service team sees GET /orders/{id} at 12 ms p50, but a trace shows 8 ms of it is spent before the request leaves, in "connect". The calling code creates a new HTTP client (or sets Connection: close) per call.

Connection setup (TCP + TLS) dominates small requests; reusing the connection removes it. Expect HTTPS to gain far more than plain HTTP because of the TLS handshake.

ConnReuse.java starts a com.sun.net.httpserver server (HTTP or HTTPS with a self-signed RSA-2048 key from keytool) and sends 2,000 sequential GETs (500 warm-up) with HttpURLConnection, either with Connection: close or with keep-alive. Three repetitions of each; medians reported.

On loopback a new HTTPS connection costs about 5 ms of handshake that a reused one does not. On a real network add 2 round trips of RTT (cross-AZ: very roughly 1 ms each; across regions tens of ms - illustrative) on top.

Measured on loopback, shared 12-core machine, median of 3 runs of 2,000 sequential requests.

Reuse connections: one long-lived, pooled client per downstream (chapter 7), never a client per call. The JDK HttpClient and Apache HttpClient pool by default; HttpURLConnection keeps connections alive unless http.keepAlive=false or a close header is sent.

Throughput of the single-threaded client rose from 784 to 1,757 req/s (HTTP) and from 167 to 1,055 req/s (HTTPS). The next section is the surprise that appeared while measuring.

Our very first keep-alive run, with the JDK server in its default configuration, reported 43.7 ms average per request - 40x worse than opening a new connection:

A constant ~40 ms is the signature of Nagle's algorithm interacting with delayed ACKs: the server writes the response in two small segments (headers, then body); Nagle refuses to send the second until the first is acknowledged, and the client delays its ACK. Setting TCP_NODELAY (the flag above does it for this server) removes the wait. Tomcat/Netty set TCP_NODELAY by default, but any hand-rolled socket client, proxy or legacy server can bring the stall back, and the symptom - "exactly 40 ms, only on reused connections" - looks like nothing in the application profiler.

Mechanism diagram; the 43 ms figure is measured here, the cause (Nagle + delayed ACK) is the standard explanation and was confirmed by the flag fixing it.

p99 of POST /orders is 1.4 s, but the PaymentGateway team swears their service answers in 40 ms. A distributed trace shows the gap inside OrderService, before the request leaves.

The HTTP client's connection pool is smaller than the concurrency that wants it, so requests queue for a pooled connection (client-side Little's Law again: pool_size / call_time = max calls per second to that host).

Client-side metrics: Micrometer exposes Apache HttpClient pool stats via httpcomponents.httpclient.pool.total.max, .connections (state=leased/available) and .pending when you bind the pool manager (PoolingHttpClientConnectionManagerMetricsBinder). Without those, a thread dump during the slow period shows many threads in PoolingHttpClientConnectionManager.lease / Future.get:

The tell-tale: workers parked (WAITING), not RUNNABLE, with a client-pool frame on top (see the thread-dump reading guide on the Profiling page).

Mechanism illustration with example numbers, not a measurement.

Little's Law applied to the pool: with 20 connections and a 40 ms call, the ceiling is 20 / 0.040 = 500 calls/s. Offered load of 1,000 calls/s (200 threads x 5 calls/s) means half the calls wait, and the wait grows until a timeout fires.

Size the pool from the same formula, set every timeout, reuse one client bean, and keep it bounded:

Spring Boot 3.4 added spring.http.client.* properties for connect/read timeouts on auto-configured clients; if you are on 3.3 (as this lab is) configure the factory in code as above.

Re-run the load test and watch httpcomponents.httpclient.pool.connections{state=leased} stay below pool.total.max, .pending near zero, and the OrderService trace gap disappear. Do not claim a number here without running it: pool tuning gains depend entirely on the downstream.

The JVM caches successful DNS lookups. Without a security manager the default is 30 seconds (the networkaddress.cache.ttl security property in java.security); a value of -1 caches forever. Failed lookups are cached for 10 s by default (networkaddress.cache.negative.ttl). With AWS endpoints that rotate IPs (ELB, RDS/Aurora writer failover, ElastiCache), a long TTL means connections keep going to an old address after a failover.

Pooled connections add a second layer: a kept-alive socket is not re-resolved at all until it is closed, so also bound connection lifetime (Hikari maxLifetime, Apache timeToLive) so the pool turns over and re-resolves. This is how a failover to a new RDS writer IP actually takes effect.

After deploying OrderService behind an ALB, users see sporadic 502 Bad Gateway - maybe 1 in 2,000 requests - with no error in the application log and no 5xx in the target group. The ALB access log shows elb_status_code=502 and target_status_code=-.

The ALB reuses a kept-alive connection that Tomcat has already closed. ALB's idle timeout defaults to 60 seconds; Tomcat's keep-alive timeout (falling back to connectionTimeout) is shorter, so there is a window where the ALB sends a request into a half-closed connection. The ALB (the "client" of your pod) sees the connection reset and answers the user with 502.

Reproduce the mechanism with real Tomcat 10.1.30 and a raw socket acting as the load balancer. The server's keep-alive timeout is 2 s; the "ALB" idles for 3 s then reuses the connection:

read() returned -1 is the client-side view of the server having closed the idle connection: exactly what the ALB experiences when it reuses a connection Tomcat closed. In production the ALB observes it as a reset/EOF and answers the user with 502.

Mechanism reproduced in the lab; the 60 s ALB idle timeout is the documented default, configurable on the load balancer. The failure rate depends on traffic patterns.

Diagnostic signature: CloudWatch HTTPCode_ELB_502_Count non-zero while HTTPCode_Target_5XX_Count is zero; ALB access log target_status_code is -; occurrences cluster on low-traffic connections that sat idle between 20 s and 60 s.

Make the server the last one to close an idle connection: its keep-alive timeout must exceed the ALB idle timeout (60 s by default; raise this together with the target's).

For the same reason outbound: your own pooled clients should evict idle connections before the downstream's timeout (Apache evictIdleConnections below the far side's idle limit). NLB behaves differently (it passes TCP through; documented idle timeout for TCP flows is 350 seconds - check the current doc) and you may need TCP keep-alive probes from the application to keep flows alive.

After the change, 502 count returns to zero while the target-group metrics are unchanged. We could not run this against AWS here; in a real rollout do it during a quiet hour and watch HTTPCode_ELB_502_Count and TargetConnectionErrorCount for a full idle cycle (> 60 s).