Performance · 10

Distributed Systems & Messaging

OrderService stopped being one process the day it started calling PaymentGateway, InventoryService and three more. Every network hop adds latency, a failure mode and a place for the tail to hide. This lab measures how fan-out turns a 1 % slow call into a 5 % slow request, how hedging buys the tail back, how a naive retry policy turns a 5-second brown-out into a permanent outage, and how timeouts, breakers, bulkheads, queues and idempotency keep a small failure small. Every number comes from a simulation or experiment run on a 12-core laptop; absolute values will differ for you, the shape will not.

14
Chapters
16
Diagrams
4
Real experiments
4.9 %
Slow-call share at fan-out 5
Hub · 01 Measure First · 02 Benchmarking & Load Tests · 03 JVM Memory & GC · 04 JIT, Startup & Native · 05 Profiling Toolbox · 06 Concurrency & Virtual Threads · 07 Spring, HTTP & Network · 08 Database Performance · 09 Caching · 10 Distributed Systems & Messaging · 11 Capacity & Failure · 12 AWS Compute & Containers · 13 Scaling & Observability · 14 Cost & Capstone · 15 Scenario Q&A
01
Chapter One - Why This Matters

Every Hop Adds a Tail

In Measure First we blamed one request for 800 ms and attributed it to a database. This chapter is the other half of that story: GET /customers/{id}/orders in OrderService is not slow because of any single dependency. It calls the PaymentGateway for payment status, InventoryService for stock, a CustomerService, a PricingService and a ShippingService, in parallel, and answers only when the last one has replied. Each dependency, looked at on its own dashboard, is healthy: p50 16 ms, p95 31 ms, and 99 % of calls under 90 ms. Yet the endpoint that depends on all five has a p99 of nearly 580 ms.

The mathematics is cruel and simple. If one call is slow with probability p, a request that makes N independent calls and waits for all of them is slow with probability 1 - (1 - p)^N. With p = 1 % and N = 5 that is 4.9 %, which is already five times the share of requests that your p99 is supposed to ignore. The tail of the end-to-end latency is no longer a rare event; it has become part of the common case.

OrderService fans out to five downstreams and waits for all of them
OrderServicewaits for all five Paymentp99 85 ms Inventorythis call: 580 ms Customerp99 85 ms Pricingp99 85 ms Shippingp99 85 ms Responseat the speed of the slowest Each dependency alone: 1 % of calls take about 500-650 ms (the "slow tail"), the rest ~16 ms. P(request touches at least one slow call) = 1 - 0.99^5 = 4.9 %, so the request p95 is already slow (measured below). Tail amplification: the more you fan out, the more your p99 is decided by the worst dependency, not the average one.
Before: what the dashboards say vs what users see
Metric (measured)One dependencyRequest that fans out to 5
p5016.6 ms25.7 ms
p9531.3 ms426 ms
p9959.7 ms579 ms
p99.9592 ms619 ms
Share of requests that include a slow call1 %4.9 % (theory 4.90 %, measured 4.84 %)

Real virtual-thread run, open model at 1000 requests/s, 12,000 requests per row, latency measured from each request's scheduled start time. Source in Chapter 12.

Key insight: A relay team is only as fast as its slowest runner, and a wedding with 100 guests almost certainly has one who is late. Fan-out turns "rare" per-component events into "routine" per-request events. This is why high-percentile latency, not the average, is the design target in distributed systems - and why the fix is rarely "make the slow call 1 % faster".
02
Chapter Two - Mental Model

Max, Sum, Product: Three Ways Latency Composes

Before the labs, fix the mental model. Four effects govern almost everything in this chapter, and each has a one-line formula you can apply in a design review.

Fan-out (parallel): latency = max of N calls
▶
P(slow) = 1 - (1 - p)N. At p = 1 %: N=5 → 4.9 %, N=20 → 18.2 %, N=100 → 63.4 %. The median itself moves once N is large (Lab 1).
Chain (sequential): latency = sum of N calls
▶
Averages add, variances add; the tail of a sum of five is still dominated by the single slow term. Measured chain of five: p50 88.8 ms, p99 653 ms.
Availability multiplies across a synchronous chain
▶
Five services at 99.9 % each: 0.9995 = 99.5 % - about 3.6 hours of "something is down" per month instead of 43 minutes.
Retries multiply across layers
▶
k attempts per layer over L layers = kL work at the bottom. Three layers with 4 attempts each → 64 calls reach the struggling dependency per user request (Lab 3).
Queues absorb, they do not fix
▶
A queue decouples producer rate from consumer rate. It only helps if the average consumer rate exceeds the average producer rate (Chapters 9-10).
Share of requests that hit at least one slow call (measured points sit on the theory curve)
0 % 20 % 40 % 60 % 80 % 100 % 1 5 10 20 30 40 50 60 70 80 90 100 p = 1 %: one dependency N=5: 4.9 % N=20: 18 % 63 % at N=100 N = number of parallel downstream calls per request share of requests with a slow call theory 1-0.99^N measured (200k requests each)

The model has one hidden assumption: the calls are independent. Real slow calls are correlated (a shared network link, a GC pause in the caller, a noisy host), which makes real tails slightly less terrible than the formula for fan-out - but correlated failures are worse for availability. Always treat the formula as a first estimate, then measure.

03
Chapter Three - Lab 1 - Tail Amplification

Measuring Fan-Out: 1 % Slow Becomes 5 % Slow

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
OrderService p99 for GET /customers/{id}/orders is 579 ms although the dashboard for every downstream shows p99 < 90 ms. p50 is fine (26 ms).
2 Hypothesis
▶
The endpoint fans out to five calls and waits for all of them. With a 1 % slow tail per call, 1 - 0.995 = 4.9 % of requests should contain a slow call, which would put p95 and p99 inside the tail.
3 Measurement
▶
A plain-Java model (code in Chapter 12): each call = 12 ms + exponential(mean 6 ms) body; 1 % of calls add 400-600 ms. Run (a) 1,000,000 draws for a single call; (b) Monte Carlo for N = 1...100; (c) a real virtual-thread run at 1000 rps; (d) the share of requests per latency bucket.
4 Diagnosis
▶
See the tables below. The measured share of slow requests matches the formula within 0.1 percentage points at every N. By N = 10 even the p95 sits in the tail; by N = 100 the median does.
5 Optimization
▶
Change one variable: the number of calls per request (N). Do not touch the downstream services at all. This is a design change - fewer, coarser calls, or removing calls from the critical path - not a tuning flag.
6 Verification
▶
Observed p99 for N = 1 vs 5 vs 20 vs 100: 63 → 577 → 608 → 619 ms; observed p50: 16 → 25 → 34 → 480 ms. The tail is capped by the slow-call size (about 600 ms); what grows is the probability of landing in it.
Monte Carlo output, 200,000 requests per row (latency = max of N parallel calls)
single call (1M draws): p50=16.2 p90=26.4 p95=31.3 p99=85.1 p99.9=597.8 ms Monte Carlo, fan-out N (request latency = max of N parallel calls) N theory P(>=1 slow) observed(>=300ms) p50 p95 p99 p99.9 1 0.0100 0.0098 16.2 31.2 62.6 599.9 2 0.0199 0.0204 19.5 37.2 521.2 606.5 5 0.0490 0.0484 24.7 60.7 577.2 613.5 10 0.0956 0.0944 29.1 515.4 597.9 618.9 20 0.1821 0.1812 34.3 566.5 607.7 622.2 50 0.3950 0.3938 45.3 597.1 614.3 627.5 100 0.6340 0.6344 480.2 607.9 618.6 632.2
Where requests land: share per latency bucket, N = 1, 5 and 20 (measured Monte Carlo, 200k requests each)
0 % 20 % 40 % 60 % 80 % 73 21 <20 ms 21 53 29 20-30 ms 4.8 21 49 30-50 ms 2.8 50-100 ms 100-300 ms 2.0 7.0 300-500 ms 3.0 11 500-700 ms N=1 (one call) N=5 N=20 The "slow lump" at 300-700 ms is the same size at every N; fan-out only moves probability mass into it (2.0 % → 6.0 % → 18.2 %).
Real virtual-thread run, open model 1000 rps (latency from scheduled start)
Real virtual-thread run, open model 1000 rps, 12000 requests per row config p50 p95 p99 p99.9 extra calls fan-out 1, no hedge 16.6 31.3 59.7 592.1 0% fan-out 5, no hedge 25.7 426.3 579.1 618.5 0% fan-out 5, hedge after 31 ms 25.7 45.9 54.7 74.8 4.9% (2932 of 60000)
End-to-end p99 (ms) by fan-out width: the tail is reached by N = 2, the median by N = 100
0 200 400 600 63 N=1 p50 16 521 N=2 p50 20 577 N=5 p50 25 598 N=10 p50 29 608 N=20 p50 34 614 N=50 p50 45 619 N=100 p50 480 p99 latency (ms)

Why N = 1 shows p99 = 63 ms and the 1,000,000-draw run shows 85 ms. The slow share is exactly 1 %, so p99 sits on the knife-edge between the body and the tail; tiny sampling differences move it a lot. This is a real-world warning: a percentile that coincides with a fault rate is unstable - choose SLOs and alerts at a percentile clearly above or below the fault rate (p99.5 here would be unambiguously "slow").

Sequential is different: the sum

If instead the five calls were made one after the other (a naive blocking implementation), the median is the sum of five medians and the tail is still one slow term on top: p50 88.8 ms, p95 153.8 ms, p99 652.8 ms, p99.9 921 ms. Parallelising turned p50 from 89 ms into 25 ms but could do nothing for the slow lump - which is why the next two chapters (hedging and deadlines) matter.

Sequential chain of five (Monte Carlo, 200k requests)
Sequential chain of 5 (sum), Monte Carlo 200k chain of 5: p50=88.8 p95=153.8 p99=652.8 p99.9=921.1 ms
⚠ Production gotchas
  • Reading downstream dashboards in isolation. Every service reports "healthy" while the caller's p99 is bad. Fix: measure the caller's end-to-end histogram per endpoint and a per-dependency client-side timer (http.client.requests with the uri/client.name tag).
  • Averaging percentiles across instances or dependencies. p99 does not add or average; merge histograms instead (see Scaling & Observability).
  • "Parallel" that is not. A CompletableFuture chain run on a saturated common pool, or parallelStream() doing blocking IO, serialises silently. Diagnose with a trace waterfall: parallel spans must start within a millisecond of each other.
  • Unbounded fan-out. A loop that issues one call per order line (N+1 over the network) is the distributed twin of the database N+1; batch the call or add a bulk endpoint.
  • A single slow shard hurts everyone. Scatter-gather over S shards behaves exactly like fan-out N = S; the usual cure is hedging (next chapter) or answering from a subset with a "partial result" flag.
04
Chapter Four - Lab 2 - Hedged Requests

Cutting the Tail with a Second Opinion

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
After Lab 1 the fan-out-5 endpoint has p95 = 426 ms and p99 = 579 ms. The dependencies are owned by other teams; we cannot make their tails shorter this quarter.
2 Hypothesis
▶
The slow calls are not slow because of the request - they landed on a bad moment or a bad replica. If a duplicate call to another replica is sent when the first is "unusually late" (after the p95 of a single call, 31 ms), the second will almost always be fast, so the min of the two will cut the tail.
3 Measurement
▶
Same experiment, same open-model 1000 rps, 12,000 requests, but each downstream call becomes a hedged call: if the first attempt has not returned after 31 ms (measured single-call p95), issue a second independent attempt and take whichever returns first. Count how many extra calls were sent.
4 Diagnosis
▶
4.9 % of calls (2,932 of 60,000) triggered a hedge, which is close to the 5 % that exceed the p95 by definition. The slow lump is cut because the probability that both attempts are slow is 1 % × 1 % = 0.01 %.
5 Optimization
▶
One variable: add the hedge, with the hedge delay set at the p95 of the single-call latency. No change to service code, only to the caller.
6 Verification
▶
Fan-out-5 p99: 579 → 54.7 ms; p99.9: 618 → 74.8 ms; p95: 426 → 45.9 ms; cost: 4.9 % more downstream requests.
Fan-out 5 (measured, 1000 rps)p50p95p99p99.9Extra downstream calls
No hedge25.7 ms426.3 ms579.1 ms618.5 ms0 %
Hedge after 31 ms (p95 of one call)25.7 ms45.9 ms54.7 ms74.8 ms4.9 % (2,932 / 60,000)
Single call, no fan-out (reference)16.6 ms31.3 ms59.7 ms592.1 ms0 %
One hedged call in the slow-primary case (illustrative timeline using the measured parameters)
0 ms 100 ms 200 ms 300 ms 400 ms 500 ms 600 ms call 1 to replica A (slow, 1 % case) 580 ms hedge timer: fires at p95 = 31 ms 31 ms call 2 to replica B (normal) 17 ms winner at 48 ms; call 1 cancelled 6.0 ms
Java - a hedged call, sketch (Java 21; StructuredTaskScope is a preview API in 21 so plain executors are used; the measured run is in Chapter 12)
static <T> T hedged(Callable<T> call, Duration hedgeAfter, ExecutorService ex) throws Exception { Future<T> first = ex.submit(call); try { return first.get(hedgeAfter.toMillis(), TimeUnit.MILLISECONDS); // fast path: no extra load } catch (TimeoutException late) { Future<T> second = ex.submit(call); // hedge: independent attempt // wait for whichever finishes first while (true) { if (first.isDone()) { second.cancel(true); return first.get(); } if (second.isDone()) { first.cancel(true); return second.get(); } Thread.sleep(1); // illustrative polling; use CompletableFuture.anyOf in real code } } }
When hedging is safe, and when it is not
ConditionHedge?Why
Idempotent read (GET inventory, GET price)YesDuplicate execution has no side effect
Non-idempotent write (POST /payments)No, unless an idempotency key is sent (Chapter 11)Two charges for one order
Downstream already near saturationNoThe extra 5 % is the load that tips it over; combine with a hedging budget and stop hedging when error rate or queue depth rises
Hedge delay below the p50NoYou would double the traffic for no gain; the delay should be at or above p95
Replicas are correlated (same host/AZ slow)WeakSend the second attempt to a different replica or AZ (watch cross-AZ latency)
⚠ Production gotchas
  • Hedge delay chosen from the average, not the percentile. A hedge at 20 ms against a 16 ms median fires on ~30 % of calls - a 30 % traffic increase. Derive the delay from a live p95 per dependency.
  • Hedging into a brown-out. When the dependency slows for everyone, every call exceeds the hedge delay and traffic doubles exactly when capacity is lowest. Cap hedges at 5-10 % of requests with a token bucket (a "retry budget" idea, Chapter 6).
  • No cancellation. If the loser keeps running you pay twice. Propagate cancellation (Future.cancel(true), HTTP/2 RST_STREAM, gRPC cancel).
  • Hedging is the cousin of speculative execution in MapReduce and "backup requests" in large-scale search; the technique and its cost profile are well studied - the lab above is the same effect at laptop scale.
05
Chapter Five - Timeouts & Deadlines

Budgets That Shrink Toward the Leaf

A hedge needs a delay; every other defence in this chapter needs a timeout. Without timeouts a slow dependency steals threads, connections and memory from callers; with badly chosen timeouts it causes the retry storm of the next chapter. The rules fit on one page.

Every network call has a timeout
▶
Java's HttpClient has none by default for the request; JDBC and Redis clients often default to "forever". Set connect (cheap, ~100-500 ms in-region) and response/read timeouts (derived from the dependency's p99.9 plus slack).
The caller's timeout is smaller than the caller's caller
▶
If the gateway gives up after 2 s while OrderService waits 5 s for Payment, the work continues after nobody is listening: wasted capacity, and a stale answer nobody reads (the cause of the metastable state in Lab 3).
Propagate the remaining budget
▶
Pass a deadline (header such as X-Request-Deadline, or gRPC deadlines) so the callee can refuse work it cannot finish and stop wasting effort on already-expired requests.
Timeouts are not retries
▶
A timeout means "I do not know whether it happened". For a non-idempotent call, a retry needs an idempotency key (Chapter 11).
Deadline propagation: every hop gets "remaining budget", never a fresh timeout (design illustration)
0 ms 500 ms 1000 ms 1500 ms 2000 ms Client / browser deadline 2000 ms API gateway (timeout 2000) 1980 ms OrderService handler 1940 ms call PaymentGateway: min(remaining, 800) 800 ms call InventoryService: min(remaining, 400) 400 ms JSON serialization + response 60 ms Budget left when Payment replies at 1100 ms 840 ms
Java - per-call timeouts and a propagated deadline with Spring RestClient / JDK HttpClient (Spring Framework 6.1+, Boot 3.2+)
HttpClient jdk = HttpClient.newBuilder() .connectTimeout(Duration.ofMillis(300)) // connect budget, in-region .build(); JdkClientHttpRequestFactory factory = new JdkClientHttpRequestFactory(jdk); factory.setReadTimeout(Duration.ofMillis(800)); // response budget: p99.9 + slack RestClient payment = RestClient.builder().baseUrl("http://payment").requestFactory(factory).build(); PaymentStatus status(String orderId, Instant deadline) { long remaining = Duration.between(Instant.now(), deadline).toMillis(); if (remaining <= 50) throw new DeadlineExceededException(); // do not start work we cannot finish return payment.get().uri("/payments/{id}", orderId) .header("X-Deadline-Remaining-Ms", String.valueOf(remaining)) .retrieve().body(PaymentStatus.class); }
Deriving a timeout from data: timeout ≈ the dependency's measured p99.9 under production load + a small slack, never less than p99. In Lab 1 the slow lump ends near 650 ms, so 800 ms catches almost all calls; a 100 ms timeout would turn the whole 1 % tail into errors (and retries). Pick it from the histogram, not from a round number.
⚠ Production gotchas
  • Same timeout at every layer. Gateway 30 s, service 30 s, client 30 s: when the DB stalls, all three give up at once and every layer retries. Make them shrink toward the leaf.
  • ALB / proxy timeouts shorter than the app's. An ALB idle timeout of 60 s cuts long requests with a 504 while the server keeps working; see Spring, HTTP & Network.
  • Only a connect timeout. Symptom: threads parked in SocketInputStream.socketRead0 in a thread dump (Profiling); fix: add a read/response timeout.
  • The timeout fires but the server keeps going. Work done for an abandoned request is pure waste; the next lab quantifies how much that matters.
06
Chapter Six - Lab 3 - Retry Storms

When Retries Turn a Blip into an Outage

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
PaymentGateway has a 5-second brown-out (capacity 1000 → 200 req/s). OrderService offers 700 req/s with a 1 s client timeout. The brown-out ends at t = 15 s, but the dashboards show goodput of zero for as long as we watch - 45 seconds later the service is still down.
2 Hypothesis
▶
The clients retry timed-out requests. During the brown-out the queue fills; every request now waits longer than the client timeout, so the server spends its capacity on answers nobody is waiting for, the clients time out again, retry again, and the offered load stays above capacity forever. That is a metastable failure: the trigger is gone, the system stays broken.
3 Measurement
▶
A discrete-time simulation in plain Java (10 ms ticks, Chapter 12): one server with a FIFO queue and fixed service rate, clients with a 1 s timeout and 3 retries, six client policies. Record attempts per logical request, peak queue, success %, and when the queue drains.
4 Diagnosis
▶
Immediate retries multiply offered load to 3.4 attempts per request (5,938 attempts/s at the peak vs 700 offered). Backoff and jitter spread the arrivals but with a 3000-deep queue the server still serves stale work and never recovers. A retry budget or a circuit breaker removes the amplification; a bounded queue or a deadline-aware server removes the stale work.
5 Optimization
▶
Change one variable per run: client policy (none → immediate → backoff → +jitter → +budget → +breaker), then server policy (3000-deep queue → 500-deep queue → drop expired requests).
6 Verification
▶
Naive server, immediate retries: 17.7 % success, queue never drains. Same server with retry budget: 80.2 %, drains at t = 26 s. With circuit breaker: 92.0 %, drains at t = 15 s. With a 500-deep queue even immediate retries recover at t = 16 s.
Output of RetryStorm.java (seed 7, deterministic; full source in Chapter 12)
capacity 1000 rps (brown-out 200 rps for t=10..15 s), offered 700 rps, client timeout 1000 ms A) naive server: FIFO queue of 3000, serves stale requests, no cancellation policy attempts/req peak sent/s peak queue success% fast-fail% queue<=50 from NO_RETRY 1.00 759 2509 84.0 0.0 t=23 s IMMEDIATE 3.42 5938 2998 17.7 0.0 never (within 60 s) BACKOFF 3.40 4843 2998 17.7 0.0 never (within 60 s) BACKOFF_JITTER 3.40 4621 2998 17.7 0.0 never (within 60 s) JITTER_BUDGET 1.02 796 2776 80.2 0.0 t=26 s JITTER_BREAKER 0.94 759 885 92.0 8.0 t=15 s B) bounded queue (500 = 0.5 s of work), rejects when full policy attempts/req peak sent/s peak queue success% fast-fail% queue<=50 from NO_RETRY 1.00 759 498 92.7 0.0 t=16 s IMMEDIATE 1.22 3323 498 92.8 0.0 t=16 s BACKOFF 1.22 3345 498 93.4 0.0 t=17 s BACKOFF_JITTER 1.22 3215 498 93.5 0.0 t=17 s JITTER_BUDGET 1.01 840 498 92.7 0.0 t=16 s JITTER_BREAKER 0.95 759 498 93.4 6.6 t=15 s C) deadline-aware server: drops requests whose client already timed out policy attempts/req peak sent/s peak queue success% fast-fail% queue<=50 from NO_RETRY 1.00 759 782 95.8 0.0 t=17 s IMMEDIATE 1.11 2216 2278 97.5 0.0 t=19 s BACKOFF 1.13 2151 2198 98.4 0.0 t=20 s BACKOFF_JITTER 1.12 2111 2142 98.5 0.0 t=21 s JITTER_BUDGET 1.01 815 830 96.0 0.0 t=17 s JITTER_BREAKER 0.95 759 860 93.2 6.8 t=15 s
Naive server (queue 3000): load offered to the server per second
0 1000 2000 3000 4000 5000 6000 6 s 10 s 15 s 20 s 25 s 30 s 35 s capacity 1000/s brown-out starts ends time (s); brown-out from t = 10 s to t = 15 s attempts reaching the server per second immediate retries backoff + jitter jitter + circuit breaker
Naive server: goodput stays at zero with retries; the breaker restores it the moment the server is healthy
0 250 500 750 1000 6 s 10 s 15 s 20 s 25 s 30 s 35 s offered 700/s brown-out starts ends time (s) useful responses per second (goodput) immediate retries backoff + jitter jitter + circuit breaker
Same retry policies, bounded queue (500 = 0.5 s of work): the system recovers by itself within about 2 s of the end of the brown-out
0 250 500 750 1000 6 s 10 s 15 s 20 s 25 s 30 s 35 s offered 700/s brown-out ends time (s) useful responses per second immediate retries, queue 500 backoff + jitter, queue 500 immediate, queue 3000 (from above)
Retry amplification across layers: k attempts per layer, L layers = k^L work at the bottom (arithmetic)
Browser / mobile appattempts: 1 x4 API gatewayattempts: 4 x4 OrderServiceattempts: 16 x4 PaymentGatewayattempts: 64 Each layer: 1 try + 3 retries = 4 attempts. When the leaf is down and every layer retries blindly: 4 x 4 x 4 = 64 calls hit the struggling service for ONE user request (and 4 x 4 = 16 hit the middle tier). Fix: retry at ONE layer (closest to the failure, and only where it is safe), or give every layer a retry budget (10 % of requests).
Per-second view: seconds 6-35, "sent attempts / good responses / queue length" for immediate retries on the naive server (excerpt)
sec sent/s good/s queue 9 719 719 0 10 695 201 494 11 1097 84 1391 12 1784 0 2975 13 5938 0 2998 14 3026 0 2998 15 961 0 2885 16 2744 0 2990 20 2743 0 2990 30 2827 0 2990 35 2771 0 2990
Why backoff and jitter alone did not save the naive server

This is the result that surprises people. The textbook answer - "exponential backoff with jitter" - cut peak attempts from 5,938/s to 4,621/s but left success at 17.7 %. The reason: with a 3000-deep FIFO queue the wait for a new request is 3000 / 1000 per second = 3 s, three times the client timeout, so every request served from the queue is already abandoned. Backoff spreads the retries in time but cannot shorten that queue. Jitter's real job is to prevent synchronized waves (all clients retrying at 100 ms, 200 ms, 400 ms in lock-step); it does not reduce total work.

DefenceMechanismNaive queue (3000)Bounded queue (500)Deadline-aware server
No retriesRemove amplification84.0 % ok, drains t=23 s92.7 %, t=16 s95.8 %, t=17 s
Immediate retries x3Amplifies 3.4x17.7 %, never92.8 %, t=16 s97.5 %, t=19 s
Exponential backoffDelays retries17.7 %, never93.4 %, t=17 s98.4 %, t=20 s
Backoff + full jitterDe-synchronises17.7 %, never93.5 %, t=17 s98.5 %, t=21 s
Jitter + retry budget (10 %)Caps retries at 10 % of traffic80.2 %, t=26 s92.7 %, t=16 s96.0 %, t=17 s
Jitter + circuit breakerStops sending while it is down92.0 %, t=15 s93.4 %, t=15 s93.2 %, t=15 s

Read the table row by row: success % over the whole 60-second run (including the 5 s outage) and the second from which the queue stays under 50 entries. "Deadline-aware" gets the highest raw success because stale requests are dropped for free - but note it still recovers later than the breaker (t = 19-21 s vs t = 15 s) when retries are on: the retries are still real load.

⚠ Production gotchas and how to diagnose them
  • Symptom: the incident is "over" (downstream healthy) but errors and queue depth stay high for minutes. Diagnosis: offered load = new + retries > capacity; look at attempts/s vs logical requests/s at the callee (http.server.requests count vs the caller's request count - their ratio is your amplification factor).
  • Retries at every layer. Count attempts per layer in code review; the product is the multiplier (64x above).
  • Retrying non-retryable errors. Retry on timeouts, connection resets, 429/503 with Retry-After; never retry 400/401/403/404/409 or failed business validation.
  • A server that cannot shed. An unbounded queue (or an unbounded thread pool queue) converts overload into latency, then into a stale-work spiral. Bound it (Tomcat accept-count, executor queue) so the overload shows up as fast 503s.
  • Model caveat: this is a simulation with a deliberately simple server (fixed rate, FIFO, no CPU contention). It demonstrates the mechanism, not the exact seconds or percentages of your system.
07
Chapter Seven - Breakers, Bulkheads, Shedding

Containing a Failure Before It Spreads

The retry-storm lab used four defences without explaining them. Each answers a different failure question, and they are used together: timeouts bound how long you wait, retries with budgets bound how often you ask again, a circuit breaker bounds how long you keep asking a service that is down, a bulkhead bounds how much of your own capacity one dependency can consume, and load shedding / backpressure bound what you accept in the first place.

Circuit breaker state machine with the exact thresholds used in the simulation
CLOSEDcalls flow; failures counted OPENfail fast, no calls HALF-OPEN10 probe calls allowed failure rate >= 50 % of >= 20 calls in last 1 s after 3 s wait elapsed all probes succeed → close a probe fails → open again Parameters used in the Chapter 6 simulation. In OPEN the call is rejected immediately (fast-fail 6.6-8 % of logical requests in the run) and, crucially, nothing reaches the server: attempts/request drops from 3.4 to 0.94, the queue drains by t = 15 s.
PatternQuestion it answersWhat it limitsTypical setting (OrderService → Payment)Failure it introduces if misused
TimeoutHow long do I wait?Time per callconnect 300 ms, read 800 ms (p99.9 + slack)Too short → artificial errors and retries
Retry + backoff + jitter + budgetShould I ask again?Attempts per request, retry traffic2 retries, 100-400 ms base, full jitter, budget 10 %Amplification (64x) and synchronized waves
Circuit breakerIs it worth calling at all?Calls to a failing dependencyopen at 50 % failures over 20+ calls; 3-10 s open; 10 probesFlapping; opens on a single bad endpoint and hides healthy ones
Bulkhead (semaphore or thread pool)How much of MY capacity may this use?Concurrent calls per dependency25 concurrent calls to Payment, fail fast beyond (Chapter 11 lab)Sized too low → needless rejections; sized too high → useless
Load sheddingShould I accept this request?Work admittedreject when queue wait > 100 ms (Chapter 11 lab)Rejecting important traffic first (shed by priority!)
BackpressureCan the producer slow down?Producer ratebounded queues, Retry-After, reactive demand, consumer-driven pull (SQS/Kafka)Unbounded buffers hide the problem until OOM
YAML - Resilience4j settings with real property names (Spring Boot 3; see Spring "Caching & Resilience" for the mechanics)
resilience4j: circuitbreaker: instances: payment: slidingWindowType: TIME_BASED slidingWindowSize: 1 # seconds minimumNumberOfCalls: 20 failureRateThreshold: 50 # percent slowCallDurationThreshold: 800ms slowCallRateThreshold: 80 waitDurationInOpenState: 3s permittedNumberOfCallsInHalfOpenState: 10 retry: instances: payment: maxAttempts: 3 # 1 try + 2 retries waitDuration: 100ms enableExponentialBackoff: true exponentialBackoffMultiplier: 2 enableRandomizedWait: true randomizedWaitFactor: 0.5 bulkhead: instances: payment: maxConcurrentCalls: 25 maxWaitDuration: 0 # fail fast instead of queueing timelimiter: instances: payment: timeoutDuration: 800ms

The annotation order matters: Resilience4j's aspect applies Retry ( CircuitBreaker ( RateLimiter ( TimeLimiter ( Bulkhead ( function ))))) by default, so a retry wraps the breaker - an open breaker's CallNotPermittedException should not be retried. Configure ignoreExceptions/retryExceptions accordingly. Background and the Spring wiring are in Caching & Resilience.

⚠ Production gotchas
  • One breaker for a whole service. A slow /refund endpoint opens the breaker for /status. Use one instance per dependency and per criticality class.
  • Breaker without a fallback or alarm. It "works" and quietly returns errors; publish resilience4j.circuitbreaker.state and alert on OPEN.
  • Bulkhead larger than the shared pool. A semaphore of 200 in front of a 20-connection pool limits nothing; size it below the scarce resource so other endpoints keep theirs (the Chapter 11 lab shows /ping staying at 1 ms).
  • Not testing the fallback path. Degraded behaviour (cached status, "payment pending") is code too - rehearse it in a game day (Capacity & Failure).
08
Chapter Eight - Sync vs Async

What Must Be on the Critical Path?

Every synchronous call ties the caller's latency and availability to the callee's. Moving a step off the critical path into a queue breaks that coupling - at the price of eventual consistency, duplicate delivery and a new thing to monitor. For OrderService the question is concrete: which of the work in POST /orders must finish before we tell the user "order placed"?

Sync chain vs queue-decoupled tail (availability = product of the components in the user-facing path; arithmetic)
A Everything synchronous Gateway OrderSvc Payment Inventory Email Availability = 0.999^5 = 99.50 % (≈ 3.6 h/month) Latency = sum of all five; an email outage fails the order B Only the money-critical path synchronous; the rest via a queue Gateway OrderSvc Payment SQS order-events Inventory Email Analytics Availability on the user path = 0.999^3 = 99.70 %; slow or failed consumers add lag, not errors Price: eventual consistency, at-least-once delivery (duplicates!), a backlog to monitor (Chapters 9-11)
DimensionSynchronous callAsynchronous (queue / log)
User-visible latencySum (or max) of dependenciesOnly the enqueue (a few ms)
Failure couplingCallee down → caller errorsCallee down → backlog grows, user unaffected
ConsistencyImmediateEventual; need status endpoints / "pending" UX
DeliveryExactly one response (or timeout)At-least-once (SQS standard, Kafka by default): duplicates are normal
OrderingNaturalPer message-group / per partition only
BackpressureTimeouts, 429/503Queue depth + age; consumers pull at their own rate
DebuggingOne traceCorrelation id across hops; trace context in message attributes/headers
Capacity planningPeak rate x latencyAverage rate; the queue absorbs peaks - if the drain rate exceeds the arrival rate
⚠ Production gotchas
  • Async as a way to hide a slow dependency. If the consumer is permanently slower than the producer the queue grows without bound (Chapter 10 lag arithmetic): fix the rate, not the symptom.
  • Fire-and-forget without a reply path. The user needs to learn "payment failed" - provide a status endpoint, a webhook or a push.
  • Dual write. Saving the order and publishing the event are two systems; use a transactional outbox (Transactions) so a crash between them cannot lose or invent events.
09
Chapter Nine - Queues on AWS

SQS: Visibility, Batching, Long Polling, DLQ

Amazon SQS is the default way to decouple OrderService from slow or flaky consumers on AWS. Its performance behaviour comes from a handful of mechanisms; understanding them avoids the classic surprises (duplicate processing, "stuck" messages, empty receives that cost money, a DLQ filling up).

SQS message lifecycle: visibility timeout is what makes delivery at-least-once
Visible in queuereceive count = n In flight (invisible)visibility timeout running Deleted (success) Dead-letter queueafter maxReceiveCount ReceiveMessage DeleteMessage redrive timeout expires before delete (slow or crashed consumer): visible again, duplicate delivery Standard queue: at-least-once, best-effort ordering. FIFO queue: ordering per message-group-id, dedup window. Defaults to know: visibility timeout 30 s (max 12 h); long-poll WaitTimeSeconds up to 20; batches of up to 10.
LeverEffect on performance / costGuidance
Visibility timeoutToo short → duplicates while a slow consumer is still working; too long → slow retry after a crash≈ 6x the p99 processing time; extend with ChangeMessageVisibility for long jobs
Long polling (WaitTimeSeconds = 20)Fewer empty receives (cost) and lower delivery delay than short pollingAlways on; ReceiveMessageWaitTimeSeconds on the queue
Batching (send / receive / delete up to 10)Up to 10x fewer API calls per message, higher throughput per consumer threadBatch delete; handle partial batch failures (per-entry results)
Consumer concurrencyThroughput = consumers x batch / processing time (Little's Law)Scale on ApproximateNumberOfMessagesVisible / age, not CPU (Scaling & Observability)
Dead-letter queue + maxReceiveCountStops a poison message from retrying forever and starving good onesSet 3-5; alarm on DLQ depth > 0; keep a redrive procedure
FIFO message groupsParallelism = number of distinct group ids in flightUse a fine-grained group id (per customer), not one global id
Java - batch consumer with explicit failure handling (AWS SDK v2 SqsClient; sketch, not run here)
ReceiveMessageRequest req = ReceiveMessageRequest.builder() .queueUrl(queueUrl) .maxNumberOfMessages(10) // batch .waitTimeSeconds(20) // long polling .visibilityTimeout(60) // > p99 processing time .build(); for (Message m : sqs.receiveMessage(req).messages()) { try { handler.handle(m); // MUST be idempotent (Chapter 11) sqs.deleteMessage(b -> b.queueUrl(queueUrl).receiptHandle(m.receiptHandle())); } catch (RuntimeException e) { // do not delete: message becomes visible again after the timeout and counts toward maxReceiveCount } }
Measured vs illustrative: The SQS limits above are documented service behaviour (check the current AWS quotas page); I did not have AWS access in the sandbox, so no SQS throughput numbers appear on this page. The duplicate-delivery effect is reproduced for real in Chapter 11.
⚠ Production gotchas
  • Duplicate side effects. A consumer that takes longer than the visibility timeout is processed twice. Symptom: double emails/charges; confirm with ApproximateReceiveCount > 1 on messages.
  • Poison message failing instantly and being retried until maxReceiveCount: without a DLQ it burns consumer capacity for the whole retention period.
  • Scaling consumers on CPU. Consumers are mostly waiting on IO; CPU stays low while the backlog grows. Scale on queue age / depth per consumer.
  • Message age ignored. Depth alone can be large but harmless; ApproximateAgeOfOldestMessage is the user-facing latency of the async path - put your SLO on it.
10
Chapter Ten - Streams

Kafka: Partitions, Batching, Consumer Lag

Kafka is a partitioned, replicated log. Its throughput comes from batching, sequential disk IO, compression and parallelism = number of partitions; its latency problems come from the same knobs set in the wrong direction. The model to carry: producer batches → partition leader → consumer group pulls by partition, and the one metric that summarises health is consumer lag (log end offset minus committed offset).

Partitions are the unit of parallelism: assignment inside one consumer group
Topic: orders (6 partitions) - consumer group "inventory" P0 P1 P2 P3 P4 P5 Consumer 12 partitions each Consumer 22 partitions each Consumer 32 partitions each 8 consumers on 6 partitions: P0 P1 P2 P3 P4 P5 C1 C2 C3 C4 C5 C6 C7 idle C8 idle The partition count is the ceiling on consumer parallelism within a group; extra consumers sit idle. Plan the partition count for the peak number of consumers you will ever want.
SettingDefault (check your client version)Direction for throughputDirection for latencyTrade-off
batch.size (producer)16384 byteslarger batchessmallermemory per partition; more records lost on crash if acks weak
linger.ms (producer)0 ms in older clients (newer Kafka releases changed the default - check)raise (5-20 ms) to fill batcheskeep lowadds up to linger.ms to every send
compression.typenonelz4 / zstd cuts bytes on the wire and on disk-CPU on producer and consumer
acksall since Kafka 3.0 (idempotent producer on)1 is faster1 is lowerall + min.insync.replicas=2 survives a broker loss
max.in.flight.requests.per.connection5higher pipelines more-ordering preserved only with idempotence on
fetch.min.bytes / fetch.max.wait.ms (consumer)1 byte / 500 msraise min bytes for bigger fetcheskeep smalladds up to max.wait to a quiet topic
max.poll.records500larger batches per pollsmallerthe whole batch must finish within max.poll.interval.ms (300000) or the consumer is evicted and the group rebalances
partitions-more partitions = more parallelism-more files/leaders, longer rebalances; per-key ordering only inside a partition
Java - producer and consumer settings for a throughput-oriented OrderService event stream (property names are the real Kafka client configs)
Properties prod = new Properties(); prod.put("bootstrap.servers", brokers); prod.put("acks", "all"); // durability prod.put("enable.idempotence", "true"); // no duplicates from producer retries prod.put("compression.type", "lz4"); prod.put("batch.size", 65536); // 4x default prod.put("linger.ms", 10); // trade 10 ms for fuller batches Properties cons = new Properties(); cons.put("group.id", "inventory"); cons.put("enable.auto.commit", "false"); // commit AFTER processing -> at-least-once cons.put("max.poll.records", 200); // must finish within max.poll.interval.ms cons.put("fetch.min.bytes", 32768);
Consumer lag arithmetic (calculation, not measurement): producer 3000/s, consumers 2400/s for 10 min, then 4000/s
0k 100k 200k 300k 400k 0 min 2 min 4 min 6 min 8 min 10 min 12 min 14 min 16 min fix deployed: 4000/s +36,000 / min (3000 in, 2400 out) -60,000 / min time since consumers fell behind (min) consumer lag (messages) backlog (messages)
Output of Capacity.java step 7 (arithmetic, illustrative rates)
producer 3000/s, consumers 2400/s for 10 min -> lag grows 36000 msgs/min, backlog 360000 after fix consumers do 4000/s: drain time = backlog / (consume - produce) = 360000 / 1000 = 6.0 min

The drain-time formula is the one to memorise: time = backlog / (consume_rate - produce_rate). If the consumers are only 5 % faster than the producers, a 360,000 message backlog takes 360000 / 150 = 40 minutes to drain; at 1 % it takes hours. This is why headroom in consumer capacity (not just peak absorbing) is part of capacity planning (next page).

⚠ Production gotchas
  • Rebalance storms. A consumer exceeding max.poll.interval.ms is removed, the group rebalances, partitions pause, lag jumps. Lower max.poll.records or move slow work to a worker pool and commit after it completes.
  • Hot partitions. A skewed key (one giant customer) sends most traffic to one partition; per-partition lag is high while the group average looks fine. Monitor lag per partition.
  • Auto-commit + slow processing = message loss on crash; manual commit after processing = at-least-once (duplicates!) - hence Chapter 11.
  • More partitions is not free. Each adds file handles, replication traffic and failover time; increasing the count later changes key-to-partition mapping.
11
Chapter Eleven - Lab 4 - Idempotency

At-Least-Once Needs Idempotent Consumers

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
Finance reports that 1,020 orders were charged twice last night after a consumer restart; the total over-charge is 55,221.96.
2 Hypothesis
▶
The message consumer is at-least-once (SQS visibility timeout / Kafka manual commit). Redelivered messages repeat the charge because the handler is not idempotent.
3 Measurement
▶
Reproduce with a plain-Java consumer (4 threads) fed 10,000 unique messages plus 10 % random redeliveries; compare totals charged with and without an idempotency key checked atomically before the side effect.
4 Diagnosis
▶
1,020 duplicate deliveries → 1,020 duplicate charges, total 601,919.35 vs the expected 546,697.39.
5 Optimization
▶
One change: record the message id before performing the side effect, in an atomic add (in production: a unique index on message_id, or a conditional write; the check and the side effect must share a transaction or be ordered so a crash replays safely).
6 Verification
▶
Charges 11,020 → 10,000; over-charge 55,221.96 → 0.00.
Idem.java output (real run)
unique messages 10000, duplicate deliveries 1020, expected total = 546697.39 no idempotency key : charges=11020 total=601919.35 overcharged by 55221.96 with idempotency key: charges=10000 total=546697.39 overcharged by 0.00
Total amount charged (measured, 10,000 orders with 1,020 redeliveries)
0 200,000 400,000 600,000 546,697 expected (10,000 unique) 601,919 no key (11,020 charges) 546,697 with key (10,000 charges) total charged (currency units)
Designing the key
ChoiceGood forWatch out
Message id (SQS MessageId / Kafka topic-partition-offset)Deduplicating redelivery of the same messageProducer retries can publish the same business event twice with different ids
Business key (orderId + operation)Exactly-once effect per business actionMust be stable across retries and unique per intended operation
Client-supplied Idempotency-Key header (as payment APIs do)Safe retries of POST from mobile/web, hedging writesStore key + response for a retention window; reject same key with different payload
Natural idempotence (SET status = PAID, upsert)Operations that can be defined as "set to this value"Increments and appends are not idempotent
Java / Spring - idempotent handler using a unique constraint (shape only, not run here; ON CONFLICT is PostgreSQL syntax)
@Transactional public void handle(OrderPaid event) { int inserted = jdbc.update( "insert into processed_events(message_id) values (?) on conflict do nothing", event.messageId()); if (inserted == 0) { // duplicate delivery: marker already present log.info("duplicate delivery {}", event.messageId()); return; // no side effect } ledger.charge(event.orderId(), event.amountCents()); // same transaction as the marker row }

Why not catch DataIntegrityViolationException from repository.save()? The repository call runs inside the same transaction, and a runtime exception leaving a transactional proxy marks it rollback-only, so the later commit fails with UnexpectedRollbackException. A conditional insert avoids the exception entirely (see Transactions).

⚠ Production gotchas
  • Check-then-act race. "SELECT exists, then INSERT" lets two consumers both pass the check; use a unique index or atomic conditional write (the lab uses an atomic Set.add for the same reason).
  • Side effect outside the transaction. Sending an email or HTTP call cannot be rolled back with the database; use an outbox so the effect is itself triggered by an idempotent row.
  • Idempotency table grows forever. Give it a TTL longer than the maximum redelivery window (SQS retention up to 14 days, Kafka retention) and index for the lookup.
  • FIFO queues deduplicate only within a window and do not stop consumer redeliveries after a timeout: the consumer still needs idempotence.
12
Chapter Twelve - Try It Yourself

The Plain-Java Source I Ran

Everything above ran from plain Java 21 files, no dependencies. Create the files, then javac FanOut.java && java FanOut and javac RetryStorm.java && java RetryStorm. Output varies slightly for FanOut (random, real sleeps); RetryStorm is seeded and deterministic.

FanOut.java - fan-out amplification, hedging, sequential chain (real virtual-thread run; ~40 s)
import java.util.*; import java.util.concurrent.*; import java.util.concurrent.atomic.*; /** Fan-out tail amplification + hedged requests, real threads (virtual) with sleeps. */ public class FanOut { static final double SLOW_P = 0.01; // 1 % of calls are slow static final ThreadLocal<SplittableRandom> RND = ThreadLocal.withInitial(() -> new SplittableRandom(System.nanoTime())); /** one downstream call latency in ms: ~12 ms + exp(mean 6) body, 1 % chance of +400..600 ms */ static double sample(SplittableRandom r) { double ms = 12 + (-Math.log(1 - r.nextDouble())) * 6; if (r.nextDouble() < SLOW_P) ms += 400 + r.nextDouble() * 200; return ms; } static void callSleep(double ms) throws InterruptedException { Thread.sleep((long) ms, (int) ((ms - Math.floor(ms)) * 1_000_000)); } static double pct(double[] sorted, double p) { int idx = (int) Math.ceil(p / 100.0 * sorted.length) - 1; return sorted[Math.max(0, Math.min(sorted.length - 1, idx))]; } static double[] runFanOut(int fan, int requests, int rps, boolean hedge, double hedgeAfterMs, AtomicLong extra) throws Exception { double[] lat = new double[requests]; CountDownLatch done = new CountDownLatch(requests); try (ExecutorService ex = Executors.newVirtualThreadPerTaskExecutor()) { long start = System.nanoTime(); long gapNs = 1_000_000_000L / rps; for (int i = 0; i < requests; i++) { final int idx = i; long scheduled = start + i * gapNs; // open model: fixed schedule long wait = scheduled - System.nanoTime(); if (wait > 0) LockSupportSleep.nanos(wait); ex.submit(() -> { try { List<Future<?>> fs = new ArrayList<>(); for (int c = 0; c < fan; c++) { fs.add(ex.submit(() -> { SplittableRandom r = RND.get(); if (!hedge) { callSleep(sample(r)); return null; } double a = sample(r); if (a <= hedgeAfterMs) { callSleep(a); return null; } extra.incrementAndGet(); // hedge fired double b = hedgeAfterMs + sample(r); callSleep(Math.min(a, b)); return null; })); } for (Future<?> f : fs) f.get(); lat[idx] = (System.nanoTime() - scheduled) / 1e6; // measured from SCHEDULED time } catch (Exception e) { lat[idx] = Double.NaN; } done.countDown(); }); } done.await(); } Arrays.sort(lat); return lat; } static class LockSupportSleep { static void nanos(long n) { java.util.concurrent.locks.LockSupport.parkNanos(n); } } public static void main(String[] a) throws Exception { // 1) single-call distribution (1,000,000 Monte Carlo draws) SplittableRandom r = new SplittableRandom(42); double[] s = new double[1_000_000]; for (int i = 0; i < s.length; i++) s[i] = sample(r); Arrays.sort(s); System.out.printf("single call (1M draws): p50=%.1f p90=%.1f p95=%.1f p99=%.1f p99.9=%.1f ms%n", pct(s, 50), pct(s, 90), pct(s, 95), pct(s, 99), pct(s, 99.9)); // 2) Monte Carlo fan-out, max of N, 200k requests each System.out.println("\nMonte Carlo, fan-out N (request latency = max of N parallel calls)"); System.out.println("N theory P(>=1 slow) observed(>=300ms) p50 p95 p99 p99.9"); for (int n : new int[]{1, 2, 5, 10, 20, 50, 100}) { int reqs = 200_000; double[] m = new double[reqs]; int slow = 0; for (int i = 0; i < reqs; i++) { double mx = 0; for (int c = 0; c < n; c++) mx = Math.max(mx, sample(r)); m[i] = mx; if (mx >= 300) slow++; } Arrays.sort(m); System.out.printf("%-3d %.4f %.4f %-6.1f %-7.1f %-7.1f %-7.1f%n", n, 1 - Math.pow(1 - SLOW_P, n), slow / (double) reqs, pct(m, 50), pct(m, 95), pct(m, 99), pct(m, 99.9)); } // 2b) histogram of request latency, share of requests per bucket double[] edges = {20, 30, 50, 100, 300, 500, 700}; System.out.println("\nShare of requests per latency bucket (ms): <20 | 20-30 | 30-50 | 50-100 | 100-300 | 300-500 | 500-700 | >=700"); for (int n : new int[]{1, 5, 20}) { int[] h = new int[edges.length + 1]; for (int i = 0; i < 200_000; i++) { double mx = 0; for (int c = 0; c < n; c++) mx = Math.max(mx, sample(r)); int b = 0; while (b < edges.length && mx >= edges[b]) b++; h[b]++; } StringBuilder sb = new StringBuilder(String.format("N=%-3d", n)); for (int v : h) sb.append(String.format(" %6.2f%%", v / 2000.0)); System.out.println(sb); } // 3) REAL threads: 5-way fan-out, 1000 rps open model, latency from scheduled time System.out.println("\nReal virtual-thread run, open model 1000 rps, 12000 requests per row"); System.out.println("config p50 p95 p99 p99.9 extra calls"); for (int n : new int[]{1, 5}) { double[] l = runFanOut(n, 12000, 1000, false, 0, new AtomicLong()); System.out.printf("fan-out %d, no hedge %-7.1f %-7.1f %-7.1f %-7.1f 0%%%n", n, pct(l, 50), pct(l, 95), pct(l, 99), pct(l, 99.9)); } AtomicLong extra = new AtomicLong(); double hedgeAt = pct(s, 95); double[] l = runFanOut(5, 12000, 1000, true, hedgeAt, extra); System.out.printf("fan-out 5, hedge after %.0f ms %-7.1f %-7.1f %-7.1f %-7.1f %.1f%% (%d of %d)%n", hedgeAt, pct(l, 50), pct(l, 95), pct(l, 99), pct(l, 99.9), extra.get() * 100.0 / (12000 * 5), extra.get(), 12000 * 5); // 4) sequential chain of 5 System.out.println("\nSequential chain of 5 (sum), Monte Carlo 200k"); double[] sq = new double[200_000]; for (int i = 0; i < sq.length; i++) { double t = 0; for (int c = 0; c < 5; c++) t += sample(r); sq[i] = t; } Arrays.sort(sq); System.out.printf("chain of 5: p50=%.1f p95=%.1f p99=%.1f p99.9=%.1f ms%n", pct(sq, 50), pct(sq, 95), pct(sq, 99), pct(sq, 99.9)); } }
RetryStorm.java - discrete-time retry-storm simulation with six client policies and three server policies
import java.util.*; /** * Retry-storm simulation. One server (FIFO queue, fixed service rate), clients with a 1 s timeout. * Time step = 10 ms. A 5 s brown-out cuts service capacity from 1000 to 200 req/s, then it recovers. * The server does NOT know a client gave up, so it still spends capacity on stale requests. */ public class RetryStorm { static final int TICK_MS = 10, TPS = 100; static final int DURATION_S = 60, OUT_START_S = 10, OUT_END_S = 15; static final double NORMAL_CAP = 1000, DEGRADED_CAP = 200, LOAD = 700; static final int TIMEOUT_T = 100; enum Policy { NO_RETRY, IMMEDIATE, BACKOFF, BACKOFF_JITTER, JITTER_BUDGET, JITTER_BREAKER } static boolean dropExpired = false; static int maxQueue = 3000; static double budget = 0; static class Logical { int attempts; boolean done; } static class Attempt { Logical l; int sentT; boolean ok, expired; } static class Result { Policy p; int logical, success, failed, fastFailed; long attempts; int peakQueue; int drainedAtS = -1; int[] sentPerS = new int[DURATION_S], goodPerS = new int[DURATION_S], queueAtS = new int[DURATION_S]; int peakSentPerS; } static Result run(Policy pol, long seed) { Random rnd = new Random(seed); budget = 0; Result res = new Result(); res.p = pol; ArrayDeque<Attempt> queue = new ArrayDeque<>(); ArrayDeque<Attempt> timeouts = new ArrayDeque<>(); List<List<Logical>> retryAt = new ArrayList<>(); int totalT = DURATION_S * TPS; for (int i = 0; i <= totalT + 2000; i++) retryAt.add(null); double tokens = 0; // breaker state int bState = 0; // 0 closed, 1 open, 2 half-open int openUntil = 0, probesLeft = 0, probesOk = 0, probesFail = 0, probesOut = 0; int[] winOk = new int[TPS], winFail = new int[TPS]; for (int t = 0; t < totalT; t++) { int sec = t / TPS; double cap = (sec >= OUT_START_S && sec < OUT_END_S) ? DEGRADED_CAP : NORMAL_CAP; // --- sends this tick: new + due retries List<Logical> toSend = new ArrayList<>(); int n = poisson(rnd, LOAD / TPS); for (int i = 0; i < n; i++) { toSend.add(new Logical()); res.logical++; budget = Math.min(10, budget + 0.1); } List<Logical> due = retryAt.get(t); if (due != null) toSend.addAll(due); for (Logical l : toSend) { if (l.done) continue; // breaker gate if (pol == Policy.JITTER_BREAKER) { if (bState == 1 && t >= openUntil) { bState = 2; probesLeft = 10; probesOk = probesFail = probesOut = 0; } if (bState == 1 || (bState == 2 && probesLeft <= 0)) { l.done = true; res.failed++; res.fastFailed++; continue; } if (bState == 2) { probesLeft--; probesOut++; } } Attempt a = new Attempt(); a.l = l; a.sentT = t; l.attempts++; res.attempts++; res.sentPerS[sec]++; if (queue.size() >= maxQueue) { // rejected immediately a.expired = true; onFail(pol, a, t, rnd, retryAt, res, bState); winFail[t % TPS]++; if (bState == 2) { probesFail++; probesOut--; } continue; } queue.add(a); timeouts.add(a); } // --- server serves tokens += cap / TPS; while (tokens >= 1 && !queue.isEmpty()) { tokens -= 1; Attempt a = queue.poll(); if (dropExpired && a.expired) { tokens += 1; continue; } // deadline-aware server: skip stale work for free if (!a.expired && !a.l.done) { a.ok = true; a.l.done = true; res.success++; res.goodPerS[sec]++; winOk[t % TPS]++; if (bState == 2) { probesOk++; probesOut--; } } else if (!a.expired && a.l.done) { a.ok = true; } // expired attempt: wasted work } if (queue.isEmpty()) tokens = Math.min(tokens, 1); // --- timeouts while (!timeouts.isEmpty() && timeouts.peek().sentT + TIMEOUT_T <= t) { Attempt a = timeouts.poll(); if (a.ok) continue; a.expired = true; winFail[t % TPS]++; if (bState == 2) { probesFail++; probesOut--; } onFail(pol, a, t, rnd, retryAt, res, bState); } // --- breaker transitions if (pol == Policy.JITTER_BREAKER) { if (bState == 0) { int ok = 0, fail = 0; for (int i = 0; i < TPS; i++) { ok += winOk[i]; fail += winFail[i]; } if (ok + fail >= 20 && fail >= 0.5 * (ok + fail)) { bState = 1; openUntil = t + 300; } } else if (bState == 2 && probesLeft <= 0 && probesOut <= 0) { if (probesFail == 0) { bState = 0; Arrays.fill(winOk, 0); Arrays.fill(winFail, 0); } else { bState = 1; openUntil = t + 300; } } winOk[(t + 1) % TPS] = 0; winFail[(t + 1) % TPS] = 0; } res.peakQueue = Math.max(res.peakQueue, queue.size()); if ((t + 1) % TPS == 0) res.queueAtS[sec] = queue.size(); } for (int s = 0; s < DURATION_S; s++) res.peakSentPerS = Math.max(res.peakSentPerS, res.sentPerS[s]); for (int s = OUT_END_S; s < DURATION_S; s++) { boolean stay = true; for (int k = s; k < Math.min(DURATION_S, s + 5); k++) if (res.queueAtS[k] > 50) stay = false; if (stay && res.queueAtS[s] <= 50) { res.drainedAtS = s; break; } } return res; } static void onFail(Policy pol, Attempt a, int t, Random rnd, List<List<Logical>> retryAt, Result res, int bState) { Logical l = a.l; if (l.done) return; int maxAttempts = (pol == Policy.NO_RETRY) ? 1 : 4; // 1 try + 3 retries if (l.attempts >= maxAttempts) { l.done = true; res.failed++; return; } int retryNo = l.attempts; // 1,2,3 int delayT; switch (pol) { case IMMEDIATE: delayT = 0; break; case BACKOFF: delayT = (int) (10 * Math.pow(2, retryNo - 1)); break; // 100,200,400 ms default: delayT = (int) (rnd.nextDouble() * 10 * Math.pow(2, retryNo - 1) * 2); break; // full jitter in [0, 2*base) } if (pol == Policy.JITTER_BUDGET) { // retry budget: 0.1 token earned per request, 1 token per retry if (budget < 1) { l.done = true; res.failed++; return; } budget -= 1; } int at = t + 1 + delayT; if (retryAt.get(at) == null) retryAt.set(at, new ArrayList<>()); retryAt.get(at).add(l); } static int poisson(Random r, double lambda) { double L = Math.exp(-lambda), p = 1; int k = 0; do { k++; p *= r.nextDouble(); } while (p > L); return k - 1; } static void table(String title) { System.out.println(title); System.out.println("policy attempts/req peak sent/s peak queue success% fast-fail% queue<=50 from"); for (Policy p : Policy.values()) { Result r = run(p, 7); System.out.printf("%-23s %-13.2f %-12d %-11d %-9.1f %-11.1f %s%n", p, r.attempts / (double) r.logical, r.peakSentPerS, r.peakQueue, 100.0 * r.success / r.logical, 100.0 * r.fastFailed / r.logical, r.drainedAtS < 0 ? "never (within 60 s)" : "t=" + r.drainedAtS + " s"); } System.out.println(); } public static void main(String[] args) { System.out.printf("capacity %.0f rps (brown-out %.0f rps for t=%d..%d s), offered %.0f rps, client timeout %d ms%n%n", NORMAL_CAP, DEGRADED_CAP, OUT_START_S, OUT_END_S, LOAD, TIMEOUT_T * TICK_MS); table("A) naive server: FIFO queue of 3000, serves stale requests, no cancellation"); maxQueue = 500; table("B) bounded queue (500 = 0.5 s of work), rejects when full"); maxQueue = 3000; dropExpired = true; table("C) deadline-aware server: drops requests whose client already timed out"); dropExpired = false; maxQueue = 3000; System.out.println("SERIES (naive server): second:sent/good/queue"); for (Policy p : new Policy[]{Policy.IMMEDIATE, Policy.BACKOFF_JITTER, Policy.JITTER_BREAKER}) { Result r = run(p, 7); StringBuilder sb = new StringBuilder(); System.out.println("SERIES " + p); for (int s = 6; s < 36; s++) sb.append(String.format("%d:%d/%d/%d ", s, r.sentPerS[s], r.goodPerS[s], r.queueAtS[s])); System.out.println(sb); } maxQueue = 500; System.out.println("SERIES (bounded queue 500)"); for (Policy p : new Policy[]{Policy.IMMEDIATE, Policy.BACKOFF_JITTER}) { Result r = run(p, 7); StringBuilder sb = new StringBuilder(); System.out.println("SERIES " + p); for (int s = 6; s < 36; s++) sb.append(String.format("%d:%d/%d/%d ", s, r.sentPerS[s], r.goodPerS[s], r.queueAtS[s])); System.out.println(sb); } } }
Idem.java - at-least-once delivery with and without an idempotency key
import java.util.*; import java.util.concurrent.*; import java.util.concurrent.atomic.*; /** At-least-once delivery + a consumer: with and without an idempotency key. 4 consumer threads, 10 % of messages redelivered. */ public class Idem { record Msg(String messageId, String orderId, int amountCents) {} static final AtomicLong chargedCents = new AtomicLong(); // the side effect: money moved static final AtomicInteger charges = new AtomicInteger(); static void charge(Msg m) { charges.incrementAndGet(); chargedCents.addAndGet(m.amountCents()); } static void consume(List<Msg> delivered, boolean idempotent) throws Exception { chargedCents.set(0); charges.set(0); Set<String> processed = ConcurrentHashMap.newKeySet(); // stand-in for a unique index on message_id try (ExecutorService ex = Executors.newFixedThreadPool(4)) { for (Msg m : delivered) ex.submit(() -> { if (idempotent && !processed.add(m.messageId())) return; // atomic check-and-insert, duplicate -> skip charge(m); }); } } public static void main(String[] a) throws Exception { Random r = new Random(1); List<Msg> unique = new ArrayList<>(); long expected = 0; for (int i = 0; i < 10_000; i++) { Msg m = new Msg("m-" + i, "o-" + i, 1000 + r.nextInt(9000)); unique.add(m); expected += m.amountCents(); } List<Msg> delivered = new ArrayList<>(unique); int dup = 0; for (Msg m : unique) if (r.nextDouble() < 0.10) { delivered.add(m); dup++; } // visibility-timeout expiry / consumer crash -> redelivery Collections.shuffle(delivered, r); System.out.printf("unique messages %d, duplicate deliveries %d, expected total = %.2f%n", unique.size(), dup, expected / 100.0); consume(delivered, false); System.out.printf("no idempotency key : charges=%d total=%.2f overcharged by %.2f%n", charges.get(), chargedCents.get() / 100.0, (chargedCents.get() - expected) / 100.0); consume(delivered, true); System.out.printf("with idempotency key: charges=%d total=%.2f overcharged by %.2f%n", charges.get(), chargedCents.get() / 100.0, (chargedCents.get() - expected) / 100.0); } }
What to try next: Change SLOW_P to 0.001 and see how much further the fan-out can grow before p99 moves; set the hedge delay to the p50 and measure the extra load; in RetryStorm set maxQueue = 1500 or TIMEOUT_T = 300 and find the boundary where the naive server flips from recovering to metastable. Treat them as models: the point is to find the mechanism, then look for it in your traces.
13
Chapter Thirteen - Interview Angles

Questions You Should Be Able to Answer

Questions on distributed performance that come up in system-design and senior backend interviews, with the answer an interviewer wants.

  • Q: Each of our 5 downstream services has p99 = 100 ms. What is the p99 of an endpoint that calls all five in parallel?
    At least 100 ms and usually much more. The chance that at least one call is in its slowest 1 % is 1 - 0.99^5 ≈ 4.9 %, so the endpoint's p95 already reaches the downstream tail and its p99 is dominated by the slowest dependencies. Measure end-to-end; do not read dependencies in isolation.
  • Q: What is a hedged request and what is the cost?
    Send a duplicate request to another replica after the p95 delay and take the first answer. It cut p99 from 579 ms to 55 ms in our lab for 4.9 % extra traffic. Only for idempotent calls, with cancellation and a budget so it cannot double load during a brown-out.
  • Q: Why can retries make an outage worse?
    Offered load = new requests + retries. With k attempts per layer over L layers the leaf sees k^L; a recovering server with a deep queue spends capacity on already-abandoned requests (metastable failure). Fix: retry budgets, backoff with jitter, circuit breaker, bounded queues, deadline propagation, retry at a single layer.
  • Q: Backoff with jitter did not help in your simulation. Why?
    Jitter de-synchronises retry waves but does not reduce total attempts; with a 3000-deep queue every served request was stale. The fix there was server-side (bounded queue or dropping expired work) or reducing retries (budget, breaker).
  • Q: How do you choose a timeout?
    From the dependency's measured p99.9 under production load plus slack, and always smaller than the caller's timeout. Propagate the remaining deadline so callees can refuse hopeless work.
  • Q: Circuit breaker vs bulkhead?
    Breaker stops calling a dependency that is failing (time-based protection). Bulkhead limits how much of your own capacity a dependency can use even when it is merely slow (resource isolation). They are complementary; the Chapter 11 lab on the next page shows a bulkhead keeping a healthy endpoint at 1 ms while the DB is 5x slower.
  • Q: SQS delivered the same message twice. Bug or normal?
    Normal: standard queues are at-least-once, and a consumer that exceeds the visibility timeout (or crashes) causes redelivery. Make the consumer idempotent; set the visibility timeout above p99 processing time; use a DLQ with maxReceiveCount.
  • Q: Consumer lag is growing. First three checks?
    Is the arrival rate above the consume rate (lag slope)? Is one partition hot (per-partition lag)? Are consumers being evicted/rebalancing (max.poll.interval.ms, GC pauses)? Drain time = backlog / (consume - produce).
  • Q: When should a call be asynchronous?
    When the user does not need its result to proceed and the work tolerates eventual consistency (email, analytics, inventory reservation updates). Keep money-critical, user-visible validation synchronous; use an outbox to avoid dual writes.
  • Q: What does "exactly once" really mean?
    Delivery is at-least-once; "exactly once" is achieved by making the effect idempotent (dedupe key / transactional consume-and-produce in Kafka), not by the network.
14
Chapter Fourteen - Summary

Cheat Sheet and Recap

Cheat sheet: symptom to distributed-systems cause
SymptomLikely causeEvidence to collectFix
p99 bad, every dependency healthyFan-out tail amplificationEnd-to-end histogram vs per-dependency client timers; trace waterfallFewer/coarser calls, hedging, partial results, timeouts at p99.9
Errors persist after dependency recoversRetry storm / metastable failureattempts/s at callee vs logical requests/s; queue depth; stale workRetry budget, breaker, bounded queue, deadlines
One slow dependency slows unrelated endpointsShared pool exhaustionThread dump, pool metrics (Chapter 11 lab)Bulkhead / separate pools, shed early
Duplicate emails or chargesAt-least-once + non-idempotent handlerApproximateReceiveCount, duplicate idsIdempotency key, unique index, outbox
Backlog grows after deployconsume rate < produce rateLag slope, per-partition lag, rebalance eventsScale consumers (up to partition count), faster handler, batching
Latency doubles when you add a hopNetwork + serialization + extra tailTrace span durations; cross-AZ placementCollapse hops, cache, co-locate
∑ Distributed performance - what to remember
  • Fan-out multiplies tail probability: P(slow) = 1 - (1 - p)^N. Measured 4.9 % at N = 5, 18 % at N = 20, 63 % at N = 100.
  • Hedging (after ~p95, idempotent calls only, budgeted) cut fan-out-5 p99 from 579 to 55 ms for 4.9 % extra calls.
  • Timeouts shrink toward the leaf; propagate the remaining deadline; derive from p99.9, not round numbers.
  • Retries multiply (k^L). Backoff + jitter de-synchronises but does not cap load; retry budgets and breakers do. Bound server queues so overload is visible and recoverable.
  • Bulkheads and shedding keep healthy endpoints healthy; a circuit breaker removes load from a failing dependency.
  • Queues decouple but deliver at-least-once: visibility timeout, DLQ, consumer lag, and idempotent handlers are the four things to design.
  • Everything on this page is a laptop-class simulation or experiment - the shapes transfer, the exact numbers do not.
Related pages: