Performance · 11

Capacity Planning & Failure Scenarios

Capacity planning answers "how much do we need?" and failure engineering answers "what happens when something is not as expected?". Both are applications of the same two ideas: Little's Law and queueing. This page models OrderService with real arithmetic (RPS per instance, instances at peak, the DB connection budget, the point where adding instances stops helping), runs a queueing simulation to show the latency hockey stick, shows what happens when the database becomes 5x slower and how a bulkhead contains it, and walks six failure scenarios as cards with symptoms, blast radius, protection and rehearsal. Simulations and arithmetic ran on a 12-core laptop; the model inputs (service times, pool sizes, DB ceilings) are illustrative assumptions you replace with your own measurements.

14
Chapters
6
Failure cards
4
Real programs
120 %
Load that breaks it
01
Chapter One - Why This Matters

The Question Is Never "How Fast", It Is "What Runs Out First"

Marketing tells the platform team that next month's campaign will double OrderService traffic. The default engineering answer, "we autoscale, it will be fine", is a hypothesis. Autoscaling adds instances; it does not add database capacity, connection slots, downstream rate limits or Kafka partitions, and it takes minutes to react (Scaling & Observability). Capacity planning is the discipline of finding which resource runs out first - before the traffic does.

OrderService capacity: the first ceiling you hit decides everything (calculation with illustrative inputs)
Users1500 rps peak ALBper-AZ 10 OrderService instances (2 vCPU) Tomcat threadslimit 4444 rps CPU 2 vCPUlimit 250 rps Hikari pool 10limit 286 rps plan: 60 % CPU150 rps / instance fleet: 10 x 150 = 1500 rps Database (shared)15,000 queries/s ceiling 6 queries / request = DB wall at 2500 rps Each instance has three independent ceilings; the fleet shares one DB. At 2x traffic the app tier scales (10 to 20 instances), the DB does not.

The same thinking applies to failure. "What happens when the DB is 5x slower?" is a capacity question in disguise: a slower resource has one fifth of the capacity, so the same offered load is now an overload. The response (timeouts, bulkheads, shedding, graceful degradation) is the same toolbox whether the cause is traffic, a slow dependency or a dead cache.

Key insight: A motorway is not limited by its average speed but by its narrowest bridge. Adding lanes elsewhere does nothing; and when a bridge closes, the question is not "how fast is the road" but "where does the queue form, and does it block the junction behind it?"
02
Chapter Two - Mental Model

Little's Law, Bottlenecks and the Knee of the Curve

Little's Law: L = λ · W
▶
Average number in the system = arrival rate x time in system. Threads busy = RPS x seconds per request. 1500 rps x 45 ms = 67 threads in use on a fleet; 200 threads per instance is plenty, a pool of 10 connections at 35 ms DB time per request tops out at 286 rps.
Throughput ceiling = min over resources
▶
Every resource has capacity = units / time-per-request-on-that-unit. The smallest wins. The bottleneck moves when you fix it (CPU → pool → DB).
Utilisation drives latency non-linearly
▶
Mean response time in an M/M/1 queue = service time / (1 - ρ). At 50 % busy it doubles, at 90 % it is 10x, at 98 % it is 50x (Lab 1, measured).
Headroom is capacity you pay for on purpose
▶
Target 50-70 % utilisation at peak so that one AZ loss, a deploy, a GC pause or a 20 % traffic bump does not push you to the knee of the curve.
Shared resources break linear scaling
▶
N instances multiply the app tier, but the DB, the lock, the downstream rate limit are shared: the Universal Scalability Law adds contention (α) and coherency (β) terms (Chapter 6).
Bottleneck = minimum over resources (calculation; values from Capacity.java in Chapter 12)
A request needs ONE unit of EACH resource for a while; throughput is set by the scarcest CPU cores 2 vCPU / 8 ms 250 rps per instance DB connections 10 / 35 ms 286 rps per instance Tomcat threads 200 / 45 ms 4444 rps per instance DB queries/s (shared) 15000 / 6 per req 2500 rps fleet-wide Bar length is logarithmic. Per instance: min(250, 286, 4444) = 250 rps = the CPU. Whole fleet: min(N x 150, 2500). When you double the fleet the CPU line doubles; the DB line does not move.
03
Chapter Three - Lab 1 - The Hockey Stick

Why 90 % Utilisation Feels Like an Outage

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
During the campaign OrderService runs at "only" 85-90 % CPU on average according to the dashboard, but p99 has gone from 90 ms to 1 second. Nothing is failing; everything is just slow.
2 Hypothesis
▶
Queueing theory predicts that response time is flat at low utilisation and explodes near 100 %. The system is on the steep part of the curve, so a few percent more traffic costs a lot of latency; a larger worker pool should push the knee further right.
3 Measurement
▶
Plain-Java discrete-event simulation (Hockey.java, Chapter 12): Poisson arrivals, exponential service time with mean 20 ms, FIFO. One configuration with a single server (M/M/1) and one with a pool of 8 workers (M/M/8). 1,000,000 requests per utilisation level; compare with the exact formula for the mean.
4 Diagnosis
▶
Simulation matches theory to 1-3 % for the mean (the 1M-sample tail is noisier). Single server: mean 2x at 50 %, 10x at 90 %, 44x at 98 %. Pool of 8: the same utilisation is far cheaper - 1.8x at 90 %, 6x at 98 %.
5 Optimization
▶
One variable: utilisation (offered load / capacity) on the x-axis; and in the second chart pooling (1 vs 8 servers) at equal utilisation. The practical lever is to keep the busiest resource below the knee: add capacity before 70-80 % (single resource) or 85-90 % (large pool).
6 Verification
▶
M/M/1 at 90 %: p99 877 ms; at 50 % 185 ms (4.7x). M/M/8 at 90 %: p99 138 ms; at 50 %: 93 ms (1.5x). Same mean service time, same utilisation: the 8-worker pool is 6x more tolerant at the p99.
Hockey.java output (real run, 1,000,000 requests per row; times in ms)
Service time: exponential, mean 20 ms. 1,000,000 requests per row. Times in ms. --- 1 server --- util mean p50 p95 p99 theory-mean mean/20ms 10% 22.2 15.4 66.4 102.3 22.2 1.1x 30% 28.5 19.8 85.3 131.5 28.6 1.4x 50% 40.0 27.7 119.7 185.1 40.0 2.0x 60% 50.0 34.7 149.6 231.5 50.0 2.5x 70% 66.6 46.2 199.1 309.1 66.7 3.3x 80% 99.7 69.2 296.2 463.1 100.0 5.0x 85% 131.9 91.6 391.9 611.8 133.3 6.6x 90% 194.4 136.9 573.6 876.9 200.0 9.7x 95% 377.4 273.4 1089.7 1594.0 400.0 18.9x 98% 890.2 666.1 2499.0 3658.8 1000.0 44.5x --- 8 servers (one pool of 8 workers) --- util mean p50 p95 p99 theory-mean mean/20ms 10% 20.0 13.9 59.9 92.1 20.0 1.0x 30% 20.0 13.9 59.9 92.1 20.0 1.0x 50% 20.3 14.2 60.3 92.5 20.3 1.0x 60% 20.9 14.9 61.2 93.3 20.9 1.0x 70% 22.3 16.3 63.4 95.5 22.3 1.1x 80% 25.7 19.7 70.2 103.4 25.7 1.3x 85% 29.4 23.0 78.4 114.2 29.5 1.5x 90% 36.9 29.4 96.0 137.6 37.5 1.8x 95% 59.6 47.5 154.0 220.8 62.2 3.0x 98% 123.6 97.3 326.5 469.7 137.0 6.2x
One server (M/M/1, mean service 20 ms): the latency hockey stick (measured)
0 1000 2000 3000 4000 0 % 20 % 40 % 60 % 80 % 90 % 100 % knee (~80 %) utilisation (arrival rate / capacity) response time (ms) p99 p95 mean p50
Pool of 8 workers (M/M/8, same service time): the knee moves right (measured)
0 100 200 300 400 500 0 % 20 % 40 % 60 % 80 % 90 % 100 % knee (~90 %) utilisation (arrival rate / total capacity) response time (ms) p99 p95 mean p50
Mean latency as a multiple of service time: pooling is the reason big fleets can run hotter than single boxes
0x 10x 20x 30x 40x 50x 0 % 20 % 40 % 60 % 80 % 90 % 100 % utilisation mean latency / service time 1 worker 8 workers
UtilisationM/M/1 mean (sim / theory)M/M/1 p99M/M/8 mean (sim / theory)M/M/8 p99
50 %40.0 / 40.0 ms185 ms20.3 / 20.3 ms93 ms
70 %66.6 / 66.7 ms309 ms22.3 / 22.3 ms96 ms
80 %99.7 / 100.0 ms463 ms25.7 / 25.7 ms103 ms
90 %194 / 200 ms877 ms36.9 / 37.5 ms138 ms
95 %377 / 400 ms1594 ms59.6 / 62.2 ms221 ms
98 %890 / 1000 ms3659 ms124 / 137 ms470 ms

Caveats. Real services are not M/M/c: service times have heavier tails (GC, cache misses), arrivals are bursty, work is not perfectly shared. That makes the real knee earlier. The simulation is a floor on how bad it is, not a prediction for your service. Take the shape; measure your own knee with a stepped load test (Benchmarking & Load Testing).

⚠ Production gotchas
  • Planning to 100 % of CPU. "250 rps per instance" is the cliff, not the target. Plan at 50-70 % (Chapter 4).
  • Looking at the average CPU over 5 minutes. A 90 % average with bursts spends many seconds beyond 100 % demand; use p95 of per-second CPU and the run-queue / throttling metrics.
  • Mixing resources with different knees. A 10-connection pool (single-server-like) hits its knee at ~70-80 % pool utilisation (hikaricp.connections.pending becomes non-zero), long before the 200-thread Tomcat pool does.
  • Confusing latency with a CPU bug. If p99 explodes while CPU is merely high, profile queue wait before profiling code: add a "time in queue" span/metric (Profiling Toolbox).
04
Chapter Four - Lab 2 - Capacity Model

RPS per Instance and Instances at Peak

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
We need a number for the campaign: peak is forecast at 1500 req/s for GET /orders/{id}. How many OrderService instances (2 vCPU, Tomcat 200 threads, Hikari pool 10) must run?
2 Hypothesis
▶
Per-instance throughput is bounded by three resources. Use Little's Law on each: capacity / time-held-per-request. The smallest decides.
3 Measurement
▶
Measured per request from a profile/trace (Profiling): 8 ms of CPU, 35 ms holding a DB connection, 2 ms other waiting. These are the inputs (illustrative here; in your service read them from JFR/CPU time and Hikari hikaricp.connections.usage). The program Capacity.java only does the arithmetic.
4 Diagnosis
▶
CPU: 2 / 0.008 = 250 rps; pool: 10 / 0.035 = 286 rps; threads: 200 / 0.045 = 4444 rps. CPU is the bottleneck. Target 60 % CPU at peak = 150 rps per instance.
5 Optimization
▶
No tuning here: choose the instance count. Peak 1500 / 150 = 10 instances. To survive losing one of three AZs (two AZs must carry the peak): ceil(10 / 2) = 5 per AZ = 15 instances.
6 Verification
▶
After sizing: 15 instances x 150 rps = 2250 rps normal capacity, 10 in two AZs = 1500 rps in the failure case at 60 % CPU - the knee is not reached even during an AZ outage.
Capacity.java, steps 1-2 (arithmetic on illustrative inputs)
== Step 1: per-instance ceiling = min over resources (Little's Law: throughput = capacity / time-per-request) == wall time per request = 45 ms (cpu 8 + db 35 + other 2) thread limit 200 / 0.045 s = 4444 rps CPU limit 2 / 0.008 s = 250 rps (100% busy) DB pool limit 10 / 0.035 s = 286 rps ceiling = 250 rps ; planned (CPU 60%, pool/threads 70%) = 150 rps -> bottleneck: CPU == Step 2: instances for peak, N+1 per AZ failure headroom == peak 1500 rps / 150 = 10 instances; to survive losing 1 of 3 AZs: 5 per AZ = 15 total
Per-instance ceilings (rps): the minimum is the CPU; the bar for threads is clipped
0 100 200 300 400 4444 (clipped) threads 200 / 45 ms 286 DB pool 10 / 35 ms 250 CPU 2 vCPU / 8 ms 150 planned (60 % CPU) requests / second / instance
StepFormulaWorked result
Wall time per requestcpu + db + other8 + 35 + 2 = 45 ms
Thread limitthreads / wall time200 / 0.045 = 4,444 rps
CPU limitvCPU / cpu time2 / 0.008 = 250 rps
DB pool limitpool / db time10 / 0.035 = 286 rps
Planned rpsmin x target utilisation250 x 0.60 = 150 rps
Instancesceil(peak / planned)ceil(1500 / 150) = 10
With AZ headroomceil(n / (AZs - 1)) per AZceil(10 / 2) = 5 per AZ = 15 total
⚠ Production gotchas
  • Using wall time for the CPU term. CPU time per request comes from profiling (process.cpu.usage / rps, or JFR jdk.CPULoad divided by throughput); wall time includes waiting.
  • Forgetting that downstream calls hold threads but not CPU - a 2 s Payment timeout turns the thread limit from 4444 rps into 100 rps (Chapter 10, card 4).
  • Capacity numbers from an idle test environment. Warm JIT, cold caches, data size, noisy neighbours and burstable (T-family) CPU credits all change them; verify with a load test against production-like data.
  • Ignoring the deploy. During a rolling deploy 25-50 % of instances may be starting (cold JIT, ~50 % capacity - see Scaling & Observability); size for that or deploy off-peak.
05
Chapter Five - Lab 3 - The Connection Budget

instances x pool <= max_connections

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
The campaign needs 30 instances at peak with autoscaling headroom (60 if you allow a 2x burst). Last week a different team scaled out and the database started refusing connections: FATAL: remaining connection slots are reserved for non-replication superuser connections (PostgreSQL wording) or Too many connections (MySQL).
2 Hypothesis
▶
Total connections = instances x pool size, and each database has a hard max_connections that depends on instance class and parameter group. Scaling the app tier silently consumes the shared connection budget.
3 Measurement
▶
Step 3 of Capacity.java: budget = max_connections - reserve (admin, replication, migrations, monitoring). Compare to instances x maximumPoolSize for realistic fleet sizes.
4 Diagnosis
▶
With max_connections = 500 (example) and a reserve of 50, the budget is 450. 10, 15 and 30 instances x 10 = 100, 150, 300: fine. At 60 instances: 600 > 450, which exceeds the budget.
5 Optimization
▶
One variable: pool size per instance or the connection architecture. Options: shrink the pool (a 2-vCPU instance rarely needs more than 10); use RDS Proxy / PgBouncer to multiplex many app connections on fewer DB connections; use fewer, larger instances; separate pools for read replicas.
6 Verification
▶
Pool 10 → 6 at 60 instances = 360 connections (fits, at the cost of 286 → 171 rps per instance on the pool limit - which is still above the 150 rps planned). Always recompute the pool-limit line from Chapter 4 after shrinking.
Capacity.java, step 3
== Step 3: DB connection budget (instances x pool <= max_connections minus reserve) == 10 instances x pool 10 = 100 connections vs budget 450 -> ok 15 instances x pool 10 = 150 connections vs budget 450 -> ok 30 instances x pool 10 = 300 connections vs budget 450 -> ok 60 instances x pool 10 = 600 connections vs budget 450 -> EXCEEDS (reduce pool, add a proxy, or fewer bigger instances)
Connections used = instances x pool size, against the database budget (calculation, example max_connections)
0 150 300 450 600 100 10 inst x 10 150 15 inst x 10 300 30 inst x 10 600 60 inst x 10 360 60 inst x 6 DB connections budget 450 (max_connections 500 - 50 reserve)
YAML - sizing the Hikari pool and exposing the numbers (Spring Boot 3, property names are real)
spring: datasource: hikari: maximum-pool-size: 6 # per instance; instances x this must fit the DB budget minimum-idle: 6 # fixed-size pool: avoids connection churn connection-timeout: 1000 # ms: fail fast instead of the 30000 default max-lifetime: 1500000 # ms: shorter than any DB/proxy idle cut-off management: metrics: tags: application: order-service

How to watch it in production: hikaricp.connections.active, hikaricp.connections.pending (non-zero means threads are waiting for a connection), hikaricp.connections.timeout and hikaricp.connections.acquire per instance, plus the database's own connection count (RDS/Aurora DatabaseConnections in CloudWatch). Deep dive in Database Performance.

⚠ Production gotchas
  • Autoscaling max not tied to the DB budget. ASG/ECS "max tasks = 100" with pool 10 = 1000 connections. Derive the maximum from the budget.
  • Rolling deploys double connections briefly (old and new tasks overlap). Include surge capacity in the budget.
  • Connection storms on failover. After an Aurora/RDS failover every instance reconnects at once; a short connection-timeout plus jittered retries keep it from becoming a self-inflicted outage.
  • Memory per DB connection is real (PostgreSQL forks a process per connection), so "just raise max_connections" has a cost.
06
Chapter Six - Lab 4 - Diminishing Returns

When Adding Instances Stops Helping

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
We added instances during the campaign (10 → 24) and throughput rose less each time. After instance 17 it did not rise at all, while the DB CPU sat at 100 %.
2 Hypothesis
▶
Two separate effects: (a) a shared resource (the DB) caps total throughput regardless of the app tier; (b) even before that cap, coordination costs (shared locks, hot rows, chatty calls) make each additional instance contribute less.
3 Measurement
▶
Step 4 and 5 of Capacity.java: (a) fleet throughput = min(N x per-instance, DB ceiling / queries per request); (b) Universal Scalability Law X(N) = N x X1 / (1 + α(N-1) + β N(N-1)) with illustrative α = 0.04 (contention) and β = 0.0006 (coherency).
4 Diagnosis
▶
DB ceiling 15,000 queries/s at 6 queries per request = 2500 rps, reached at 17 instances; beyond that the instances only share the same cap. With the USL the curve peaks near N = 40 and then declines - retrograde scaling.
5 Optimization
▶
The variable that matters is not instance count but the shared bottleneck: reduce queries per request (fix N+1, cache), scale the DB (read replicas, bigger class, sharding), or remove coordination. Verify with a load test at 2-3 fleet sizes and fit α and β.
6 Verification
▶
Halving queries per request from 6 to 3 doubles the DB-bound ceiling from 2500 to 5000 rps without adding a single instance - more effective than going from 17 to 24 instances.
Capacity.java, steps 4-5
== Step 4: when does adding instances stop helping? shared DB ceiling == DB can sustain 15000 queries/s, each request issues 6 -> whole fleet capped at 2500 rps instances app capacity fleet throughput (min(app, DB)) 4 600 600 8 1200 1200 12 1800 1800 16 2400 2400 17 2550 2500 20 3000 2500 24 3600 2500 32 4800 2500 == Step 5: Universal Scalability Law, X(N) = N*X1 / (1 + a*(N-1) + b*N*(N-1)) == a (contention) = 0.04, b (coherency) = 0.0006, X1 = 150 rps [illustrative coefficients] N linear USL efficiency 1 150 150 100% 2 300 288 96% 4 600 532 89% 8 1200 914 76% 12 1800 1185 66% 16 2400 1376 57% 24 3600 1599 44% 32 4800 1693 35% 48 7200 1701 24% 64 9600 1616 17% peak of the curve: N = 40 (formula sqrt((1-a)/b) = 40.0), X = 1716 rps
Why more instances stop helping: shared-resource cap (red) and coordination costs (blue) - calculation with illustrative coefficients
0 1000 2000 3000 4000 5000 0 10 17 20 30 40 50 60 DB ceiling 2500 rps DB cap reached number of OrderService instances (150 rps each) fleet throughput (rps) linear (ideal) capped by shared DB (2500 rps) USL (alpha 0.04, beta 0.0006)
InstancesLinearDB-cappedUSL (illustrative)USL efficiency
460060053289 %
81200120091476 %
1624002400137657 %
2436002500159944 %
40 (peak of USL curve)60002500171629 %
6496002500161617 %
⚠ Production gotchas
  • Scaling the app tier to "fix" a DB-bound system. The signature: throughput flat, app CPU low, DB CPU/IOPS or connection wait high, hikaricp.connections.pending > 0 on every instance.
  • Retrograde scaling from coordination. Distributed locks, a shared counter row, chatty cache invalidation: more instances = more contention. A load test showing lower throughput at higher N is the signature; profile lock waits (Profiling).
  • Fitting USL with two points. You need at least 5-6 measurements at different concurrencies/instances to fit α and β; treat the page's coefficients as placeholders.
  • Hot partitions/rows. One celebrity customer row or Kafka partition caps throughput independent of fleet size.
07
Chapter Seven - Worked Example

What Happens When Traffic Doubles?

Now the worked example the campaign needs. Take the model above and double the peak from 1500 rps to 3000 rps. Compute every tier's requirement instead of assuming it scales.

Capacity.java, step 6
== Step 6: 'what if traffic doubles?' == peak 3000 rps -> app tier: 20 instances (was 10); DB connections 100 -> 200; DB query rate 9000 -> 18000 qps vs ceiling 15000 qps (DB IS THE FIRST WALL) DB utilisation today: 60% of ceiling; at 2x it would need 120% of ceiling max traffic the DB tier allows: 2500 rps = 1.7x today (before any DB change)
Tier / resourceToday (1500 rps)At 2x (3000 rps)Verdict
App instances at 60 % CPU1020Scales horizontally; check autoscaling max and scale-out speed
DB connections (pool 10)100 of 450 (22 %)200 of 450 (44 %)Fine
DB queries/s (ceiling 15,000)9,000 (60 %)18,000 (120 %)First wall. Needs a change before the campaign
Downstream Payment rate limit (example 2,000 rps)1,500 (75 %)3,000 (150 %)Second wall. Ask the owner team now; add hedging only if idempotent
Kafka partitions / consumers12 / 812 / 16 useful max 12Plan more partitions or batch
"What happens when traffic doubles?": utilisation of each tier at 2x peak (calculation, illustrative limits)
0 % 40 % 80 % 100 % 120 % 160 % 60 % App CPU (20 inst) 44 % DB conn budget 120 % DB queries (15k/s) 150 % Payment rate limit % of the tier limit at 2x traffic 100 % = limit

The answer in one sentence for the stakeholder: the application tier will scale, the database and the payment provider will not, and we have a 1.7x runway on the DB before we must fix queries per request, add a read replica or cache. That is a capacity plan; "autoscaling will handle it" is not.

The planning checklist: peak (not average) rate with a growth factor; per-request resource cost (CPU ms, DB ms, queries, bytes); per-resource ceiling from a load test; N+1 / AZ headroom; shared-resource budget (connections, DB QPS, downstream limits, partitions); autoscaling min/max derived from the above; a documented plan for what to shed when the plan is wrong.
08
Chapter Eight - Lab 5 - Overload & Load Shedding

Sustained 120 % Load: Reject Early or Wait Forever

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
A bad deploy, a bot or a viral link pushes offered load to 120 % of capacity. After 20 minutes the p50 of the service is 124 seconds. Nobody is waiting that long: every user has already left.
2 Hypothesis
▶
When arrival rate permanently exceeds capacity the queue grows without bound and every request waits behind all earlier ones. The only stable policy is to reject some work. Shed early and cheaply, based on predicted queue wait, so the admitted requests stay fast.
3 Measurement
▶
Same simulator: 8 workers (capacity 400 rps), offered 480 rps for 600,000 requests (21 minutes of traffic). Three policies: no limit; shed if the predicted wait exceeds 100 ms; shed if it exceeds 40 ms.
4 Diagnosis
▶
Without a limit, mean latency 124 s and max 251 s (it grows linearly with time). With shedding the service serves its capacity (83 % of offered load) with bounded latency; the other 17 % are rejected immediately.
5 Optimization
▶
One variable: the shedding threshold (predicted wait). A queue limit expressed in time (max_queue_wait) rather than in items adapts to changes in service time.
6 Verification
▶
p99: 248,225 ms (no limit) → 182 ms (shed at 100 ms wait) → 123 ms (shed at 40 ms wait). Accepted-request mean: 124,359 ms → 108 ms → 49 ms. 17 % of requests receive a fast "try again" instead of a 2-minute hang.
Hockey.java, overload section (real run)
--- Overload: 8 workers (capacity 400 rps), offered 480 rps (120%), 600,000 requests = 21 min of traffic --- policy served% mean p50 p99 max no limit (queue grows) 100.0 124359 123984 248225 250851 shed if predicted wait > 100 ms 83.3 108 104 182 341 shed if predicted wait > 40 ms 82.9 49 45 123 278
Mean latency of accepted requests under 120 % offered load for 21 minutes (measured): 124 s vs 108 ms vs 49 ms
0 ms 40,000 ms 80,000 ms 120,000 ms 124,359 ms no limit mean 108 ms shed >100 ms mean 49 ms shed >40 ms mean mean latency (ms)
MechanismWhereBehaviour at overloadTypical setting
Server concurrency limit + bounded accept queueTomcat server.tomcat.threads.max, accept-count, max-connectionsExcess connections are refused or wait in the OS backlogaccept-count = a fraction of one second of work
Admission control by queue waitFilter / interceptor timing the time from accept to startReturn 503 + Retry-After when predicted wait > limit40-100 ms
Per-client rate limit (token bucket)API Gateway / WAF / filter429 for the offending client onlyburst and sustained rate per API key
Priority sheddingGateway or appDrop analytics/prefetch first, keep checkout2-3 priority classes
AutoscalingECS/EKSAdds capacity in minutes (too slow for a spike)not a substitute for shedding
⚠ Production gotchas
  • Unbounded executor queue. Executors.newFixedThreadPool has an unbounded LinkedBlockingQueue: overload becomes memory growth and latency; use a bounded queue with an explicit rejection policy (Concurrency).
  • Shedding after the expensive work. Reject before parsing bodies, authenticating against a remote service or touching the DB.
  • Clients that retry 503 instantly turn shedding into a retry storm; send Retry-After and make clients honour it, with jitter (Distributed Systems).
  • Health checks that shed. If the load balancer's health check goes through the overloaded path, instances are marked unhealthy and capacity shrinks exactly when it is needed (the next lab shows /ping failing).
09
Chapter Nine - Lab 6 - Failure: The DB Gets 5x Slower

A Slow Dependency Is Worse Than a Dead One

1 Problem
2 Hypothesis
3 Measurement
4 Diagnosis
5 Optimization
6 Verification
1 Problem
▶
At 10:42 the database becomes 5x slower (a bad plan after a statistics refresh, a noisy neighbour, a failing disk). The load balancer health check /ping, which never touches the DB, starts failing and instances are rotated out of service - and the incident widens.
2 Hypothesis
▶
The slow DB makes every /orders request hold a Tomcat thread much longer (waiting for a connection and for the DB). With 50 threads shared by all endpoints and a Hikari pool of 20, the thread pool saturates and even /ping waits for a thread. A bulkhead that limits how many threads may be inside the DB section should keep /ping alive.
3 Measurement
▶
Discrete-event model of one instance (DbSlow.java, Chapter 12): Tomcat-like pool of 50 threads with accept queue 100, Hikari-like pool of 20 connections with 1 s acquire timeout, offered 400 rps (70 % /orders, 30 % /ping). Phases: 0-30 s normal (DB 20 ms), 30-60 s DB 100 ms, 60-90 s normal. Run twice: without and with a bulkhead of 25 concurrent /orders in the DB section (excess rejected immediately).
4 Diagnosis
▶
In the slow phase DB demand = 280 orders/s x 0.1 s = 28 connections-worth of work against a pool of 20, so connection waiters pile up while still holding their Tomcat threads. All 50 threads end up blocked; the accept queue overflows (25.8 % of /ping rejected) and the surviving /ping calls wait 338 ms at the median.
5 Optimization
▶
One variable: a bulkhead of 25 concurrent requests in the DB section (a Semaphore / Resilience4j Bulkhead). Orders beyond it fail fast with 503; threads return to the pool within milliseconds.
6 Verification
▶
/ping during the incident: p99 413 ms and 25.8 % errors → p99 1 ms and 0 % errors. /orders: errors stay ~27-29 % in both (the DB really cannot serve them), but p50 drops from 577 ms to 110 ms: failing fast instead of failing slowly. After recovery both variants are back to p99 65-69 ms at once.
DbSlow.java (real run, deterministic seed)
== NO bulkhead (shared 50 threads, Hikari 20) == phase endpoint requests errors% p50 ms p99 ms 0-30 s normal /ping 3528 0.0 1 1 0-30 s normal /orders 8345 0.0 25 65 30-60 s DB 5x slower /ping 3526 25.8 338 413 30-60 s DB 5x slower /orders 8328 26.8 577 769 60-90 s recovered /ping 3504 0.1 1 1 60-90 s recovered /orders 8381 0.0 25 69 == WITH bulkhead (25 concurrent /orders in DB section, fail fast beyond) == phase endpoint requests errors% p50 ms p99 ms 0-30 s normal /ping 3528 0.0 1 1 0-30 s normal /orders 8345 0.0 25 65 30-60 s DB 5x slower /ping 3611 0.0 1 1 30-60 s DB 5x slower /orders 8415 29.4 110 315 60-90 s recovered /ping 3507 0.0 1 1 60-90 s recovered /orders 8389 0.0 25 65
Mechanism: a shared thread pool lets one slow dependency starve everything; a bulkhead partitions it (model of the run above)
Without a bulkhead: one slow resource drains the shared pool 400 rps arrive280 + 120 rps Tomcat pool: 50 threads50 busy, accept queue 100 full Hikari pool: 20 connsall busy, waiters hold threads DB 5x slower100 ms per query /ping needs a thread for 1 ms, but all 50 threads are parked waiting for connections: queue → 338 ms median, 25.8 % rejected. With a bulkhead (25 concurrent in the DB section; rejects fail fast) 400 rps arrive280 + 120 rps Tomcat pool: 50 threads25 in DB section + 25 free Bulkhead: 25 permitsexcess: 503 in < 1 ms DB 5x slowerstill slow, isolated /ping always finds one of the 25 free threads: 1 ms median and p99, 0 % errors. /orders p50 is 110 ms (fast failure) instead of 577 ms.
DB 5x slower: latency during the incident window, with and without the bulkhead (measured in the simulation)
0 ms 200 ms 400 ms 600 ms 800 ms 413 ms /ping p99 no bulkhead 1 ms /ping p99 bulkhead 577 ms /orders p50 no bulkhead 110 ms /orders p50 bulkhead 769 ms /orders p99 no bulkhead 315 ms /orders p99 bulkhead latency (ms)
Java - a bulkhead is a Semaphore with tryAcquire (plain Java; Resilience4j Bulkhead does the same with metrics)
private final Semaphore dbSection = new Semaphore(25); Order load(long id) { if (!dbSection.tryAcquire()) { // fail fast, thread goes straight back to the pool throw new ResponseStatusException(HttpStatus.SERVICE_UNAVAILABLE, "order store saturated"); } try { return repository.findById(id).orElseThrow(); // the slow, scarce resource } finally { dbSection.release(); } }
⚠ Production gotchas
  • Health check shares the failing path. Keep liveness trivial and readiness cheap (Actuator & Observability); do not make readiness depend on the DB if you would rather degrade than lose all instances.
  • Bulkhead = pool size. A semaphore of 200 in front of a pool of 20 isolates nothing; the number must be smaller than the shared pool so other work keeps threads (here 25 of 50).
  • Model caveat. This is one instance in a discrete-event model, with exponential-ish DB time and no GC, client retries or CPU contention; real incidents add retry load (see Distributed Systems, Lab 3). It shows the mechanism and the direction of the fix.
  • Diagnosis in production: thread dump (jcmd <pid> Thread.print) full of threads in HikariPool.getConnection; hikaricp.connections.pending > 0; tomcat.threads.busy = tomcat.threads.config.max; DB latency/wait events up.
10
Chapter Ten - Failure Scenario Cards

Six Ways Production Gets Slow

Each card follows the same template: what you will see, how far it spreads, what protects you and how to rehearse it. Numbers are arithmetic from the formulas above; none of these six were reproduced in the sandbox beyond what the earlier labs show (marked).

🗄

1. Database 5x slower (simulated in Lab 6)

  • Symptoms: p50/p99 of every DB-backed endpoint up, hikaricp.connections.pending > 0, Tomcat threads all busy, SQLTransientConnectionException: ... Connection is not available, request timed out after 30000ms if the acquire timeout is left at the default.
  • Blast radius: Every endpoint sharing the thread pool; health checks; upstream callers.
  • Protects you: Short acquire timeout, bulkhead on the DB section, query timeouts (spring.jpa.properties.jakarta.persistence.query.timeout), cache for read-mostly data, read-replica for reports.
  • Rehearse: Add 200 ms latency on the DB port with Toxiproxy or tc netem in staging; watch /ping.
⚡

2. Redis / ElastiCache is down

  • Symptoms: Cache calls time out or throw; DB read QPS multiplies by 1 / (1 - hit ratio): 10x at 90 % hits, 20x at 95 %, 100x at 99 % (arithmetic). Latency up, then DB overload.
  • Blast radius: The database and everything behind the cache; stampede on restart (cold cache).
  • Protects you: Cache timeouts of 10-30 ms, circuit breaker that bypasses the cache, request coalescing so one miss loads the DB once, a small local Caffeine layer, DB protected by a bulkhead; warm the cache before taking traffic. See Caching.
  • Rehearse: Kill the Redis container / block its port; confirm DB QPS stays inside its ceiling.
📨

3. Kafka consumer lag growing

  • Symptoms: Lag slope positive on records-lag-max / consumer-lag metric; downstream data (inventory, emails) increasingly stale; with SQS, ApproximateAgeOfOldestMessage rising.
  • Blast radius: Everything that reads from the stream; users see "my order is not updated". Not the synchronous path.
  • Protects you: Headroom in consumer rate (drain time = backlog / (consume - produce); 360,000 backlog at +1000/s spare = 6 min), autoscale consumers on lag up to the partition count, DLQ for poison messages, backpressure on producers if bounded. Calculation in Distributed Systems.
  • Rehearse: Pause a consumer group for 10 minutes in staging and time the drain.
⏱

4. A downstream (PaymentGateway) times out

  • Symptoms: Thread hold time = timeout. With 50 threads and a 2 s timeout, throughput of payment-dependent requests is capped at 50 / 2 = 25 rps (Little's Law); threads pile up, callers see 504/timeout.
  • Blast radius: All requests that call Payment; if not isolated, everything on the pool.
  • Protects you: Timeout at p99.9 (not 30 s), circuit breaker, bulkhead sized below the pool, retry budget, graceful degradation ("payment pending", async confirmation), idempotency keys. Simulations in Distributed Systems.
  • Rehearse: Toxiproxy timeout toxic on the payment proxy; check the fallback path and the dashboards.
🌊

5. A 10x traffic spike

  • Symptoms: Queue wait dominates, p99 in seconds, 5xx from the load balancer, autoscaling reacts minutes later (see Scaling & Observability). In Lab 5 a mere 120 % turned p50 into 124 s without shedding.
  • Blast radius: The whole service; shared dependencies get hammered.
  • Protects you: Rate limits per client at the edge, admission control by queue wait, CDN/cache for reads, queue-based leveling for writes, pre-scaling for known events, priority shedding.
  • Rehearse: Load-test at 3x and 10x with an open-model generator; verify rejection is fast and recovery is automatic.
🧪

6. A deployment ships a memory leak

  • Symptoms: Heap baseline after each GC rises; GC time share grows; p99 climbs before the OOM; then java.lang.OutOfMemoryError: Java heap space or a container OOMKilled. Example arithmetic: 1536 MB heap, 500 MB live set, +4 MB/min leak = OOM after about 259 minutes - after the deploy has been declared a success.
  • Blast radius: A rolling deploy hides it: instances die one by one hours later, then restart and leak again (a sawtooth in availability).
  • Protects you: Canary with automatic rollback on SLO burn and on heap-after-GC trend, -XX:+HeapDumpOnOutOfMemoryError, alerts on old-gen occupancy slope, restarts as a mitigation not a fix. See JVM Memory & GC.
  • Rehearse: Deploy a build with a deliberate leak flag to the canary and check that the pipeline rolls back by itself.
Which parts of the system each failure scenario touches (illustrative, derived from the mechanisms above)
Blast radius by scenario (qualitative - which layers are touched) Edge / ALB App threads DB Cache Consumers Users 1 DB 5x slower - hit hard hit hard - degraded hit hard 2 Cache down - degraded hit hard hit hard - degraded 3 Consumer lag - - - - hit hard degraded 4 Downstream timeout - hit hard - - - hit hard 5 10x spike hit hard hit hard hit hard degraded degraded hit hard 6 Memory leak deploy - hit hard - - - hit hard Containment tools shrink the red cells: bulkheads keep threads free, breakers isolate the dependency, shedding protects the edge, request coalescing protects the DB.
11
Chapter Eleven - Rehearsing Failure

Game Days, Fault Injection and Abort Criteria

A protection you have never exercised is a hypothesis. Rehearsal turns it into evidence, in three increasing levels: tests in CI, fault injection in staging, and game days in production with a controlled blast radius.

Game day: steady state → hypothesis → inject → observe → fix
1 Steady state define "healthy": p99 < 300 ms 2 Hypothesis "Redis dies: DB QPS stays < 12k/s" 3 Inject one fault, small blast radius, abort rules 4 Observe dashboards and alerts: did they fire? 5 Fix & repeat fix the gap, re-run, automate in CI The game-day loop (process, not measurement) Abort conditions are written BEFORE injecting: error rate > 2 % for 2 min, p99 > 2 s, or any customer-visible payment failure. Run in staging first, then production at low traffic, one instance/AZ, with the team watching.
Fault to injectTool (how-to, not run in this sandbox)What it proves
Latency / timeout / dropped connections to a dependencytoxiproxy-cli toxic add -t latency -a latency=1000 -a jitter=200 payment (Toxiproxy proxies a TCP service and adds "toxics")Timeouts, breaker, bulkhead and fallback work
Dependency fully downtoxiproxy-cli toxic add -t timeout -a timeout=0 redis or stop the containerCache-down path; DB stays inside its ceiling
Network delay/loss on a hosttc qdisc add dev eth0 root netem delay 200ms 50ms loss 1% (Linux, needs root)Hedging, retries, timeouts under packet loss
Kill instances / tasks / AZAWS Fault Injection Service experiment templates (stop tasks/instances, network disruption - check the current action list), or Chaos Mesh / Litmus on EKS (PodChaos, NetworkChaos)Autoscaling, health checks, AZ headroom
CPU or memory pressurestress-ng, FIS resource actions, or a leak flag in a canary buildThrottling behaviour, OOM handling, canary rollback
Traffic surgek6/Gatling open-model stages at 3x / 10x of peakShedding, autoscaling lag, rate limits
Shell - a minimal Toxiproxy session (how-to; commands as documented by the Toxiproxy project)
toxiproxy-server & # start the proxy daemon toxiproxy-cli create -l localhost:26379 -u localhost:6379 redis # clients connect to 26379 toxiproxy-cli toxic add -t latency -a latency=300 -a jitter=100 redis # ... run the load test, watch p99, DB QPS, breaker state ... toxiproxy-cli toxic remove -n latency_downstream redis
⚠ Production gotchas
  • Chaos without observability. If you cannot see p99, queue depth and breaker state in real time, you are only breaking things.
  • No abort criteria or no owner. One person has the stop button; abort limits are numeric.
  • Testing only the happy fault. Total outage is easier to survive than "5x slower" or "1 % packet loss" - inject gradual and partial faults too.
  • One-off exercises. Capacity and failure behaviour regress with every release; automate the cheap scenarios in the pipeline and repeat game days each quarter.
12
Chapter Twelve - Try It Yourself

The Plain-Java Source I Ran

Plain Java 21, no dependencies: javac Hockey.java && java Hockey, javac DbSlow.java && java DbSlow, javac Capacity.java && java Capacity. All three are deterministic (fixed seeds) except for tiny differences you get from changing the seed.

Hockey.java - M/M/c queue simulation and overload shedding
import java.util.*; /** Queueing simulation: M/M/c FIFO. Shows the latency hockey stick vs utilisation, and overload with/without shedding. */ public class Hockey { static final double SVC_MS = 20; static double[] simulate(int servers, double util, int n, long seed, double shedIfWaitOverMs, int[] rejected) { Random r = new Random(seed); double lambdaPerMs = util * servers / SVC_MS; PriorityQueue<Double> free = new PriorityQueue<>(); for (int i = 0; i < servers; i++) free.add(0.0); double t = 0; double[] resp = new double[n]; int k = 0; for (int i = 0; i < n; i++) { t += -Math.log(1 - r.nextDouble()) / lambdaPerMs; double svc = -Math.log(1 - r.nextDouble()) * SVC_MS; double f = free.peek(); double start = Math.max(t, f); if (shedIfWaitOverMs > 0 && start - t > shedIfWaitOverMs) { if (rejected != null) rejected[0]++; continue; } free.poll(); double done = start + svc; free.add(done); resp[k++] = done - t; } double[] out = Arrays.copyOf(resp, k); Arrays.sort(out); return out; } static double pct(double[] s, double p) { return s[Math.max(0, Math.min(s.length - 1, (int) Math.ceil(p / 100 * s.length) - 1))]; } static double mean(double[] s) { double a = 0; for (double v : s) a += v; return a / s.length; } /** Erlang C mean response time for M/M/c */ static double erlangMean(int c, double rho) { double a = rho * c, sum = 0, term = 1; for (int k = 0; k < c; k++) { if (k > 0) term *= a / k; sum += term; } double last = term * a / c; // a^c / c! double top = last / (1 - rho); double pw = top / (sum + top); return SVC_MS + pw * SVC_MS / (c * (1 - rho)); } public static void main(String[] args) { System.out.println("Service time: exponential, mean 20 ms. 1,000,000 requests per row. Times in ms."); for (int c : new int[]{1, 8}) { System.out.printf("%n--- %d server%s ---%n", c, c == 1 ? "" : "s (one pool of 8 workers)"); System.out.println("util mean p50 p95 p99 theory-mean mean/20ms"); for (double u : new double[]{0.10, 0.30, 0.50, 0.60, 0.70, 0.80, 0.85, 0.90, 0.95, 0.98}) { double[] s = simulate(c, u, 1_000_000, 11, 0, null); System.out.printf("%3.0f%% %-8.1f %-8.1f %-8.1f %-8.1f %-13.1f %.1fx%n", u * 100, mean(s), pct(s, 50), pct(s, 95), pct(s, 99), erlangMean(c, u), mean(s) / SVC_MS); } } System.out.println("\n--- Overload: 8 workers (capacity 400 rps), offered 480 rps (120%), 600,000 requests = 21 min of traffic ---"); System.out.println("policy served% mean p50 p99 max"); for (double shed : new double[]{0, 100, 40}) { int[] rej = new int[1]; double[] s = simulate(8, 1.2, 600_000, 5, shed, rej); System.out.printf("%-31s %-9.1f %-9.0f %-9.0f %-9.0f %.0f%n", shed == 0 ? "no limit (queue grows)" : "shed if predicted wait > " + (int) shed + " ms", 100.0 * s.length / 600_000, mean(s), pct(s, 50), pct(s, 99), s[s.length - 1]); } } }
DbSlow.java - discrete-event model of thread pool, connection pool and a slow database, with and without a bulkhead
import java.util.*; /** * "The database becomes 5x slower" - discrete-event model of one OrderService instance. * Tomcat-like pool: 50 threads, accept queue 100. Hikari-like pool: 20 connections, wait timeout 1000 ms. * /orders: 5 ms app work, then DB call (20 ms normal, x5 in phase 2), then 3 ms serialization. /ping: 1 ms, no DB. * Offered: 400 rps (70 % /orders, 30 % /ping). Phases: 0-30 s normal, 30-60 s DB 5x slower, 60-90 s normal. * Variant "bulkhead": at most 25 /orders requests may be inside the DB section at once, others are rejected at once. */ public class DbSlow { enum Ev { ARRIVE, APP_DONE, DB_DONE, FIN, CONN_TIMEOUT } static class Req { boolean order; double arr; boolean waitingConn; boolean dead; int phase; boolean failed; } record E(double t, Ev ev, Req r, long seq) {} static final int THREADS = 50, ACCEPT = 100, CONNS = 20, BULKHEAD = 25; static final double CONN_TIMEOUT = 1000; static void run(boolean bulkhead) { Random rnd = new Random(3); PriorityQueue<E> pq = new PriorityQueue<>(Comparator.comparingDouble(E::t).thenComparingLong(E::seq)); long[] seq = {0}; ArrayDeque<Req> threadQ = new ArrayDeque<>(), connQ = new ArrayDeque<>(); int[] busyThreads = {0}, busyConns = {0}, inDb = {0}; List<double[]>[] lat = new List[6]; // [phase*2 + (order?1:0)] for (int i = 0; i < 6; i++) lat[i] = new ArrayList<>(); int[] fail = new int[6], total = new int[6]; double t = 0, end = 90_000; double lambdaMs = 0.4; // generate arrivals lazily t = -Math.log(1 - rnd.nextDouble()) / lambdaMs; Req first = new Req(); first.arr = t; first.order = rnd.nextDouble() < 0.7; pq.add(new E(t, Ev.ARRIVE, first, seq[0]++)); while (!pq.isEmpty()) { E e = pq.poll(); double now = e.t(); if (now > end) break; Req r = e.r(); switch (e.ev()) { case ARRIVE -> { // schedule next arrival double nt = now + (-Math.log(1 - rnd.nextDouble()) / lambdaMs); Req nx = new Req(); nx.arr = nt; nx.order = rnd.nextDouble() < 0.7; pq.add(new E(nt, Ev.ARRIVE, nx, seq[0]++)); r.phase = now < 30_000 ? 0 : now < 60_000 ? 1 : 2; total[r.phase * 2 + (r.order ? 1 : 0)]++; if (busyThreads[0] < THREADS) { start(pq, seq, r, now, busyThreads); } else if (threadQ.size() < ACCEPT) threadQ.add(r); else { r.failed = true; fail[r.phase * 2 + (r.order ? 1 : 0)]++; } } case APP_DONE -> { if (!r.order) { finish(pq, seq, r, now, 0); break; } if (bulkhead && inDb[0] >= BULKHEAD) { fin(r, now, lat, fail, true, pq, seq, busyThreads, threadQ); break; } inDb[0]++; if (busyConns[0] < CONNS) { busyConns[0]++; dbStart(pq, seq, r, now, rnd); } else { r.waitingConn = true; connQ.add(r); pq.add(new E(now + CONN_TIMEOUT, Ev.CONN_TIMEOUT, r, seq[0]++)); } } case CONN_TIMEOUT -> { if (r.waitingConn) { r.waitingConn = false; r.dead = true; connQ.remove(r); inDb[0]--; fin(r, now, lat, fail, true, pq, seq, busyThreads, threadQ); } } case DB_DONE -> { inDb[0]--; Req nxt = connQ.poll(); while (nxt != null && !nxt.waitingConn) nxt = connQ.poll(); if (nxt != null) { nxt.waitingConn = false; dbStart(pq, seq, nxt, now, rnd); } else busyConns[0]--; finish(pq, seq, r, now, 3); } case FIN -> fin(r, now, lat, fail, false, pq, seq, busyThreads, threadQ); } } System.out.println(bulkhead ? "\n== WITH bulkhead (25 concurrent /orders in DB section, fail fast beyond) ==" : "== NO bulkhead (shared 50 threads, Hikari 20) =="); System.out.println("phase endpoint requests errors% p50 ms p99 ms"); String[] ph = {"0-30 s normal ", "30-60 s DB 5x slower", "60-90 s recovered "}; for (int p = 0; p < 3; p++) for (int o = 0; o < 2; o++) { int i = p * 2 + o; List<double[]> L = lat[i]; double[] a = new double[L.size()]; for (int k = 0; k < a.length; k++) a[k] = L.get(k)[0]; Arrays.sort(a); System.out.printf("%s %-9s %-9d %-9.1f %-8.0f %.0f%n", ph[p], o == 1 ? "/orders" : "/ping", total[i], 100.0 * fail[i] / Math.max(1, total[i]), a.length == 0 ? 0 : a[a.length / 2], a.length == 0 ? 0 : a[(int) Math.min(a.length - 1, Math.ceil(0.99 * a.length) - 1)]); } } static void start(PriorityQueue<E> pq, long[] seq, Req r, double now, int[] busyThreads) { busyThreads[0]++; pq.add(new E(now + (r.order ? 5 : 1), Ev.APP_DONE, r, seq[0]++)); } static void dbStart(PriorityQueue<E> pq, long[] seq, Req r, double now, Random rnd) { // DB slowness follows the wall clock, not the request's arrival phase double mean = (now >= 30_000 && now < 60_000) ? 100 : 20; double d = 0.5 * mean + (-Math.log(1 - rnd.nextDouble())) * 0.5 * mean; pq.add(new E(now + d, Ev.DB_DONE, r, seq[0]++)); } static void finish(PriorityQueue<E> pq, long[] seq, Req r, double now, double extra) { pq.add(new E(now + extra, Ev.FIN, r, seq[0]++)); } static void fin(Req r, double now, List<double[]>[] lat, int[] fail, boolean failed, PriorityQueue<E> pq, long[] seq, int[] busyThreads, ArrayDeque<Req> threadQ) { int i = r.phase * 2 + (r.order ? 1 : 0); if (failed) fail[i]++; else lat[i].add(new double[]{now - r.arr}); busyThreads[0]--; Req n = threadQ.poll(); if (n != null) start(pq, seq, n, now, busyThreads); } public static void main(String[] a) { run(false); run(true); } }
Capacity.java - the capacity arithmetic (illustrative inputs at the top; replace with your own measurements)
/** Capacity arithmetic for OrderService. Inputs are ILLUSTRATIVE assumptions; the code only does the math. */ public class Capacity { public static void main(String[] a) { // ---- inputs (per request, GET /orders/{id}) ---- double cpuMs = 8; // CPU time burnt in the JVM per request double dbMs = 35; // time holding a DB connection per request (waiting on DB) double otherMs = 2; // other waiting (downstream, serialization IO) double wallMs = cpuMs + dbMs + otherMs; int vcpu = 2, tomcatThreads = 200, hikari = 10; double targetCpuUtil = 0.60; double peakRps = 1500; int azs = 3; System.out.println("== Step 1: per-instance ceiling = min over resources (Little's Law: throughput = capacity / time-per-request) =="); double byThreads = tomcatThreads / (wallMs / 1000); double byCpu = vcpu / (cpuMs / 1000); double byPool = hikari / (dbMs / 1000); System.out.printf("wall time per request = %.0f ms (cpu %.0f + db %.0f + other %.0f)%n", wallMs, cpuMs, dbMs, otherMs); System.out.printf("thread limit 200 / %.3f s = %6.0f rps%n", wallMs / 1000, byThreads); System.out.printf("CPU limit %d / %.3f s = %6.0f rps (100%% busy)%n", vcpu, cpuMs / 1000, byCpu); System.out.printf("DB pool limit %d / %.3f s = %6.0f rps%n", hikari, dbMs / 1000, byPool); double ceiling = Math.min(byThreads, Math.min(byCpu, byPool)); double usable = Math.min(byThreads * 0.7, Math.min(byCpu * targetCpuUtil, byPool * 0.7)); System.out.printf("ceiling = %.0f rps ; planned (CPU %.0f%%, pool/threads 70%%) = %.0f rps -> bottleneck: %s%n", ceiling, targetCpuUtil * 100, usable, byCpu <= byPool && byCpu <= byThreads ? "CPU" : byPool <= byThreads ? "DB pool" : "threads"); System.out.println("\n== Step 2: instances for peak, N+1 per AZ failure headroom =="); int base = (int) Math.ceil(peakRps / usable); int perAz = (int) Math.ceil(base / (double) (azs - 1)); // survive losing one AZ System.out.printf("peak %.0f rps / %.0f = %d instances; to survive losing 1 of %d AZs: %d per AZ = %d total%n", peakRps, usable, base, azs, perAz, perAz * azs); System.out.println("\n== Step 3: DB connection budget (instances x pool <= max_connections minus reserve) =="); int maxConn = 500, reserve = 50; for (int inst : new int[]{base, perAz * azs, 30, 60}) { int need = inst * hikari; System.out.printf("%2d instances x pool %d = %3d connections vs budget %d -> %s%n", inst, hikari, need, maxConn - reserve, need <= maxConn - reserve ? "ok" : "EXCEEDS (reduce pool, add a proxy, or fewer bigger instances)"); } System.out.println("\n== Step 4: when does adding instances stop helping? shared DB ceiling =="); double dbQps = 15000, queriesPerReq = 6; double dbCeilingRps = dbQps / queriesPerReq; System.out.printf("DB can sustain %.0f queries/s, each request issues %.0f -> whole fleet capped at %.0f rps%n", dbQps, queriesPerReq, dbCeilingRps); System.out.println("instances app capacity fleet throughput (min(app, DB))"); for (int n : new int[]{4, 8, 12, 16, 17, 20, 24, 32}) { double app = n * usable; System.out.printf("%-10d %-14.0f %.0f%n", n, app, Math.min(app, dbCeilingRps)); } System.out.println("\n== Step 5: Universal Scalability Law, X(N) = N*X1 / (1 + a*(N-1) + b*N*(N-1)) =="); double x1 = usable, alpha = 0.04, beta = 0.0006; System.out.printf("a (contention) = %.2f, b (coherency) = %.4f, X1 = %.0f rps [illustrative coefficients]%n", alpha, beta, x1); System.out.println("N linear USL efficiency"); double best = 0; int bestN = 0; for (int n : new int[]{1, 2, 4, 8, 12, 16, 24, 32, 48, 64}) { double x = n * x1 / (1 + alpha * (n - 1) + beta * n * (n - 1)); System.out.printf("%-4d %-10.0f %-10.0f %.0f%%%n", n, n * x1, x, 100 * x / (n * x1)); } for (int n = 1; n <= 200; n++) { double x = n * x1 / (1 + alpha * (n - 1) + beta * n * (n - 1)); if (x > best) { best = x; bestN = n; } } System.out.printf("peak of the curve: N = %d (formula sqrt((1-a)/b) = %.1f), X = %.0f rps%n", bestN, Math.sqrt((1 - alpha) / beta), best); System.out.println("\n== Step 6: 'what if traffic doubles?' =="); double newPeak = peakRps * 2; int newInst = (int) Math.ceil(newPeak / usable); System.out.printf("peak %.0f rps -> app tier: %d instances (was %d); DB connections %d -> %d; DB query rate %.0f -> %.0f qps vs ceiling %.0f qps (%s)%n", newPeak, newInst, base, base * hikari, newInst * hikari, peakRps * queriesPerReq, newPeak * queriesPerReq, dbQps, newPeak * queriesPerReq > dbQps ? "DB IS THE FIRST WALL" : "ok"); System.out.printf("DB utilisation today: %.0f%% of ceiling; at 2x it would need %.0f%% of ceiling%n", 100 * peakRps * queriesPerReq / dbQps, 100 * newPeak * queriesPerReq / dbQps); System.out.printf("max traffic the DB tier allows: %.0f rps = %.1fx today (before any DB change)%n", dbCeilingRps, dbCeilingRps / peakRps); System.out.println("\n== Step 7: consumer lag arithmetic (queue backlog) =="); double produce = 3000, consume = 2400; // msgs/s double outageMin = 10; double backlog = (produce - consume) * outageMin * 60; System.out.printf("producer %.0f/s, consumers %.0f/s for %.0f min -> lag grows %.0f msgs/min, backlog %.0f%n", produce, consume, outageMin, (produce - consume) * 60, backlog); double fullConsume = 4000; System.out.printf("after fix consumers do %.0f/s: drain time = backlog / (consume - produce) = %.0f / %.0f = %.1f min%n", fullConsume, backlog, fullConsume - produce, backlog / (fullConsume - produce) / 60); System.out.printf("memory leak: heap 1536 MB, live set 500 MB, leak 4 MB/min -> OOM after about %.0f min%n", (1536 - 500) / 4.0); } }
What to try next: Change SVC_MS to a heavy-tailed distribution (lognormal) in Hockey.java and watch the knee move left; give DbSlow 100 threads or a 200 ms acquire timeout and see which variable changes the incident most; replace Capacity's inputs with numbers from your own JFR/Hikari metrics.
13
Chapter Thirteen - Interview Angles

Questions You Should Be Able to Answer

  • Q: How do you decide how many instances a service needs?
    Per instance: min over resources of capacity / time-per-request (CPU, threads, connections), times a target utilisation (50-70 %). Fleet: ceil(peak / planned) plus N+1 (or AZ) headroom. Then check every shared resource (DB QPS, connections, downstream limits, partitions) for the same peak.
  • Q: Why does latency explode near 100 % utilisation?
    Queueing: mean response time = service time / (1 - ρ) for M/M/1 - 2x at 50 %, 10x at 90 %, 50x at 98 %. The tail is worse. Pooling (more servers) delays the knee: an 8-worker pool is 1.8x at 90 %.
  • Q: We doubled the instances and throughput did not move. Why?
    The bottleneck is shared: DB CPU/IO or QPS, connection slots, a lock, a downstream rate limit, a hot partition. Evidence: app CPU low, hikaricp.connections.pending high, DB saturated. Fix queries per request, scale the shared tier, or remove coordination.
  • Q: How does instances x pool size interact with the database?
    Total connections = instances x maximumPoolSize must stay under the DB budget (max_connections minus reserve); autoscaling max must be derived from it. RDS Proxy / PgBouncer multiplex when it does not fit.
  • Q: The DB is slow and unrelated endpoints are failing. Why?
    Shared Tomcat threads are blocked waiting on the DB. Isolate with a bulkhead or separate pools, fail fast on connection acquire, keep health checks off the DB path. Our model: /ping p99 413 ms and 25.8 % errors → 1 ms and 0 %.
  • Q: What is load shedding and where do you put it?
    Rejecting some work early so admitted work stays fast. At the edge (rate limits), then admission control by queue wait in the service, before expensive work. Send 503 + Retry-After; clients back off with jitter.
  • Q: A new release leaks 4 MB/min. Why is that hard to catch?
    The OOM comes after hours (259 min in the example) when the deploy looks healthy. Canary + automatic rollback on heap-after-GC slope and SLO burn, heap dump on OOM, and restarts as mitigation only.
  • Q: How do you rehearse failure safely?
    Steady-state hypothesis, small blast radius, numeric abort criteria, tool-driven faults (Toxiproxy, tc netem, FIS), run in staging first, then production at low traffic; automate the cheap ones in CI.
  • Q: What does the Universal Scalability Law add to Amdahl's Law?
    A coherency term β: contention (α) limits speed-up to a plateau (Amdahl), coherency cost makes throughput decline beyond a peak N = sqrt((1-α)/β). Fit α and β from at least 5 measurements.
  • Q: Why is autoscaling not a capacity plan?
    It adds app-tier capacity after minutes of delay and does nothing for DB, quotas, partitions, or warm-up; it can amplify DB overload by adding connections. Plan the shared limits and pre-scale known events.
14
Chapter Fourteen - Summary

Cheat Sheet and Recap

Cheat sheet
QuestionFormula / ruleOrderService example
Threads or connections neededL = λ · W1500 rps x 45 ms = 67 threads fleet-wide; pool 10 / 35 ms = 286 rps
Per-instance ceilingmin(capacity / time-per-request) over resourcesmin(4444, 286, 250) = 250 rps (CPU)
Instances at peakceil(peak / (ceiling x target util))ceil(1500 / 150) = 10; 15 with AZ headroom
DB connection budgetinstances x pool <= max_connections - reserve60 x 10 = 600 > 450: shrink pool or add a proxy
Shared-tier wallDB QPS ceiling / queries per request15000 / 6 = 2500 rps (1.7x today)
Latency vs utilisationW = S / (1 - ρ) (M/M/1)2x at 50 %, 10x at 90 %, 50x at 98 %
Queue drain timebacklog / (consume - produce)360000 / 1000 = 6 min
Thread hold with slow downstreamthreads / timeout50 / 2 s = 25 rps
∑ Capacity and failure - what to remember
  • Throughput = min over resources; find the minimum per instance, then the shared minimum for the fleet.
  • Utilisation is a latency dial: the knee is ~80 % for a single resource, ~90 % for a pool of 8; plan peaks at 50-70 %.
  • Connections are a shared budget: instances x pool must fit; autoscaling max follows from it.
  • Adding instances stops helping at the shared bottleneck (here at 17 instances) and may reverse past the USL peak.
  • Overload needs rejection: shed early by predicted queue wait; otherwise 120 % load means a 2-minute p50.
  • A slow dependency is worse than a dead one: bulkhead the thread pool, short timeouts, fast failure; keep health checks off the failing path.
  • Rehearse every protection (Toxiproxy, netem, FIS, canary leak) with abort criteria. Numbers here are model results and arithmetic on illustrative inputs - re-derive them with your own measurements.
Related pages: