Capacity Planning & Failure Scenarios
Capacity planning answers "how much do we need?" and failure engineering answers "what happens when something is not as expected?". Both are applications of the same two ideas: Little's Law and queueing. This page models OrderService with real arithmetic (RPS per instance, instances at peak, the DB connection budget, the point where adding instances stops helping), runs a queueing simulation to show the latency hockey stick, shows what happens when the database becomes 5x slower and how a bulkhead contains it, and walks six failure scenarios as cards with symptoms, blast radius, protection and rehearsal. Simulations and arithmetic ran on a 12-core laptop; the model inputs (service times, pool sizes, DB ceilings) are illustrative assumptions you replace with your own measurements.
The Question Is Never "How Fast", It Is "What Runs Out First"
Marketing tells the platform team that next month's campaign will double OrderService traffic. The default engineering answer, "we autoscale, it will be fine", is a hypothesis. Autoscaling adds instances; it does not add database capacity, connection slots, downstream rate limits or Kafka partitions, and it takes minutes to react (Scaling & Observability). Capacity planning is the discipline of finding which resource runs out first - before the traffic does.
The same thinking applies to failure. "What happens when the DB is 5x slower?" is a capacity question in disguise: a slower resource has one fifth of the capacity, so the same offered load is now an overload. The response (timeouts, bulkheads, shedding, graceful degradation) is the same toolbox whether the cause is traffic, a slow dependency or a dead cache.
Little's Law, Bottlenecks and the Knee of the Curve
Why 90 % Utilisation Feels Like an Outage
| Utilisation | M/M/1 mean (sim / theory) | M/M/1 p99 | M/M/8 mean (sim / theory) | M/M/8 p99 |
|---|---|---|---|---|
| 50 % | 40.0 / 40.0 ms | 185 ms | 20.3 / 20.3 ms | 93 ms |
| 70 % | 66.6 / 66.7 ms | 309 ms | 22.3 / 22.3 ms | 96 ms |
| 80 % | 99.7 / 100.0 ms | 463 ms | 25.7 / 25.7 ms | 103 ms |
| 90 % | 194 / 200 ms | 877 ms | 36.9 / 37.5 ms | 138 ms |
| 95 % | 377 / 400 ms | 1594 ms | 59.6 / 62.2 ms | 221 ms |
| 98 % | 890 / 1000 ms | 3659 ms | 124 / 137 ms | 470 ms |
Caveats. Real services are not M/M/c: service times have heavier tails (GC, cache misses), arrivals are bursty, work is not perfectly shared. That makes the real knee earlier. The simulation is a floor on how bad it is, not a prediction for your service. Take the shape; measure your own knee with a stepped load test (Benchmarking & Load Testing).
- Planning to 100 % of CPU. "250 rps per instance" is the cliff, not the target. Plan at 50-70 % (Chapter 4).
- Looking at the average CPU over 5 minutes. A 90 % average with bursts spends many seconds beyond 100 % demand; use p95 of per-second CPU and the run-queue / throttling metrics.
- Mixing resources with different knees. A 10-connection pool (single-server-like) hits its knee at ~70-80 % pool utilisation (
hikaricp.connections.pendingbecomes non-zero), long before the 200-thread Tomcat pool does. - Confusing latency with a CPU bug. If p99 explodes while CPU is merely high, profile queue wait before profiling code: add a "time in queue" span/metric (Profiling Toolbox).
RPS per Instance and Instances at Peak
GET /orders/{id}. How many OrderService instances (2 vCPU, Tomcat 200 threads, Hikari pool 10) must run?hikaricp.connections.usage). The program Capacity.java only does the arithmetic.| Step | Formula | Worked result |
|---|---|---|
| Wall time per request | cpu + db + other | 8 + 35 + 2 = 45 ms |
| Thread limit | threads / wall time | 200 / 0.045 = 4,444 rps |
| CPU limit | vCPU / cpu time | 2 / 0.008 = 250 rps |
| DB pool limit | pool / db time | 10 / 0.035 = 286 rps |
| Planned rps | min x target utilisation | 250 x 0.60 = 150 rps |
| Instances | ceil(peak / planned) | ceil(1500 / 150) = 10 |
| With AZ headroom | ceil(n / (AZs - 1)) per AZ | ceil(10 / 2) = 5 per AZ = 15 total |
- Using wall time for the CPU term. CPU time per request comes from profiling (
process.cpu.usage/ rps, or JFRjdk.CPULoaddivided by throughput); wall time includes waiting. - Forgetting that downstream calls hold threads but not CPU - a 2 s Payment timeout turns the thread limit from 4444 rps into 100 rps (Chapter 10, card 4).
- Capacity numbers from an idle test environment. Warm JIT, cold caches, data size, noisy neighbours and burstable (T-family) CPU credits all change them; verify with a load test against production-like data.
- Ignoring the deploy. During a rolling deploy 25-50 % of instances may be starting (cold JIT, ~50 % capacity - see Scaling & Observability); size for that or deploy off-peak.
instances x pool <= max_connections
FATAL: remaining connection slots are reserved for non-replication superuser connections (PostgreSQL wording) or Too many connections (MySQL).How to watch it in production: hikaricp.connections.active, hikaricp.connections.pending (non-zero means threads are waiting for a connection), hikaricp.connections.timeout and hikaricp.connections.acquire per instance, plus the database's own connection count (RDS/Aurora DatabaseConnections in CloudWatch). Deep dive in Database Performance.
- Autoscaling max not tied to the DB budget. ASG/ECS "max tasks = 100" with pool 10 = 1000 connections. Derive the maximum from the budget.
- Rolling deploys double connections briefly (old and new tasks overlap). Include surge capacity in the budget.
- Connection storms on failover. After an Aurora/RDS failover every instance reconnects at once; a short
connection-timeoutplus jittered retries keep it from becoming a self-inflicted outage. - Memory per DB connection is real (PostgreSQL forks a process per connection), so "just raise max_connections" has a cost.
When Adding Instances Stops Helping
| Instances | Linear | DB-capped | USL (illustrative) | USL efficiency |
|---|---|---|---|---|
| 4 | 600 | 600 | 532 | 89 % |
| 8 | 1200 | 1200 | 914 | 76 % |
| 16 | 2400 | 2400 | 1376 | 57 % |
| 24 | 3600 | 2500 | 1599 | 44 % |
| 40 (peak of USL curve) | 6000 | 2500 | 1716 | 29 % |
| 64 | 9600 | 2500 | 1616 | 17 % |
- Scaling the app tier to "fix" a DB-bound system. The signature: throughput flat, app CPU low, DB CPU/IOPS or connection wait high,
hikaricp.connections.pending> 0 on every instance. - Retrograde scaling from coordination. Distributed locks, a shared counter row, chatty cache invalidation: more instances = more contention. A load test showing lower throughput at higher N is the signature; profile lock waits (Profiling).
- Fitting USL with two points. You need at least 5-6 measurements at different concurrencies/instances to fit α and β; treat the page's coefficients as placeholders.
- Hot partitions/rows. One celebrity customer row or Kafka partition caps throughput independent of fleet size.
What Happens When Traffic Doubles?
Now the worked example the campaign needs. Take the model above and double the peak from 1500 rps to 3000 rps. Compute every tier's requirement instead of assuming it scales.
| Tier / resource | Today (1500 rps) | At 2x (3000 rps) | Verdict |
|---|---|---|---|
| App instances at 60 % CPU | 10 | 20 | Scales horizontally; check autoscaling max and scale-out speed |
| DB connections (pool 10) | 100 of 450 (22 %) | 200 of 450 (44 %) | Fine |
| DB queries/s (ceiling 15,000) | 9,000 (60 %) | 18,000 (120 %) | First wall. Needs a change before the campaign |
| Downstream Payment rate limit (example 2,000 rps) | 1,500 (75 %) | 3,000 (150 %) | Second wall. Ask the owner team now; add hedging only if idempotent |
| Kafka partitions / consumers | 12 / 8 | 12 / 16 useful max 12 | Plan more partitions or batch |
The answer in one sentence for the stakeholder: the application tier will scale, the database and the payment provider will not, and we have a 1.7x runway on the DB before we must fix queries per request, add a read replica or cache. That is a capacity plan; "autoscaling will handle it" is not.
Sustained 120 % Load: Reject Early or Wait Forever
max_queue_wait) rather than in items adapts to changes in service time.| Mechanism | Where | Behaviour at overload | Typical setting |
|---|---|---|---|
| Server concurrency limit + bounded accept queue | Tomcat server.tomcat.threads.max, accept-count, max-connections | Excess connections are refused or wait in the OS backlog | accept-count = a fraction of one second of work |
| Admission control by queue wait | Filter / interceptor timing the time from accept to start | Return 503 + Retry-After when predicted wait > limit | 40-100 ms |
| Per-client rate limit (token bucket) | API Gateway / WAF / filter | 429 for the offending client only | burst and sustained rate per API key |
| Priority shedding | Gateway or app | Drop analytics/prefetch first, keep checkout | 2-3 priority classes |
| Autoscaling | ECS/EKS | Adds capacity in minutes (too slow for a spike) | not a substitute for shedding |
- Unbounded executor queue.
Executors.newFixedThreadPoolhas an unboundedLinkedBlockingQueue: overload becomes memory growth and latency; use a bounded queue with an explicit rejection policy (Concurrency). - Shedding after the expensive work. Reject before parsing bodies, authenticating against a remote service or touching the DB.
- Clients that retry 503 instantly turn shedding into a retry storm; send
Retry-Afterand make clients honour it, with jitter (Distributed Systems). - Health checks that shed. If the load balancer's health check goes through the overloaded path, instances are marked unhealthy and capacity shrinks exactly when it is needed (the next lab shows
/pingfailing).
A Slow Dependency Is Worse Than a Dead One
/ping, which never touches the DB, starts failing and instances are rotated out of service - and the incident widens.Semaphore / Resilience4j Bulkhead). Orders beyond it fail fast with 503; threads return to the pool within milliseconds.- Health check shares the failing path. Keep liveness trivial and readiness cheap (Actuator & Observability); do not make readiness depend on the DB if you would rather degrade than lose all instances.
- Bulkhead = pool size. A semaphore of 200 in front of a pool of 20 isolates nothing; the number must be smaller than the shared pool so other work keeps threads (here 25 of 50).
- Model caveat. This is one instance in a discrete-event model, with exponential-ish DB time and no GC, client retries or CPU contention; real incidents add retry load (see Distributed Systems, Lab 3). It shows the mechanism and the direction of the fix.
- Diagnosis in production: thread dump (
jcmd <pid> Thread.print) full of threads inHikariPool.getConnection;hikaricp.connections.pending> 0;tomcat.threads.busy=tomcat.threads.config.max; DB latency/wait events up.
Six Ways Production Gets Slow
Each card follows the same template: what you will see, how far it spreads, what protects you and how to rehearse it. Numbers are arithmetic from the formulas above; none of these six were reproduced in the sandbox beyond what the earlier labs show (marked).
1. Database 5x slower (simulated in Lab 6)
- Symptoms: p50/p99 of every DB-backed endpoint up,
hikaricp.connections.pending> 0, Tomcat threads all busy,SQLTransientConnectionException: ... Connection is not available, request timed out after 30000msif the acquire timeout is left at the default. - Blast radius: Every endpoint sharing the thread pool; health checks; upstream callers.
- Protects you: Short acquire timeout, bulkhead on the DB section, query timeouts (
spring.jpa.properties.jakarta.persistence.query.timeout), cache for read-mostly data, read-replica for reports. - Rehearse: Add 200 ms latency on the DB port with Toxiproxy or
tc netemin staging; watch /ping.
2. Redis / ElastiCache is down
- Symptoms: Cache calls time out or throw; DB read QPS multiplies by 1 / (1 - hit ratio): 10x at 90 % hits, 20x at 95 %, 100x at 99 % (arithmetic). Latency up, then DB overload.
- Blast radius: The database and everything behind the cache; stampede on restart (cold cache).
- Protects you: Cache timeouts of 10-30 ms, circuit breaker that bypasses the cache, request coalescing so one miss loads the DB once, a small local Caffeine layer, DB protected by a bulkhead; warm the cache before taking traffic. See Caching.
- Rehearse: Kill the Redis container / block its port; confirm DB QPS stays inside its ceiling.
3. Kafka consumer lag growing
- Symptoms: Lag slope positive on
records-lag-max/ consumer-lag metric; downstream data (inventory, emails) increasingly stale; with SQS,ApproximateAgeOfOldestMessagerising. - Blast radius: Everything that reads from the stream; users see "my order is not updated". Not the synchronous path.
- Protects you: Headroom in consumer rate (drain time = backlog / (consume - produce); 360,000 backlog at +1000/s spare = 6 min), autoscale consumers on lag up to the partition count, DLQ for poison messages, backpressure on producers if bounded. Calculation in Distributed Systems.
- Rehearse: Pause a consumer group for 10 minutes in staging and time the drain.
4. A downstream (PaymentGateway) times out
- Symptoms: Thread hold time = timeout. With 50 threads and a 2 s timeout, throughput of payment-dependent requests is capped at 50 / 2 = 25 rps (Little's Law); threads pile up, callers see 504/timeout.
- Blast radius: All requests that call Payment; if not isolated, everything on the pool.
- Protects you: Timeout at p99.9 (not 30 s), circuit breaker, bulkhead sized below the pool, retry budget, graceful degradation ("payment pending", async confirmation), idempotency keys. Simulations in Distributed Systems.
- Rehearse: Toxiproxy
timeouttoxic on the payment proxy; check the fallback path and the dashboards.
5. A 10x traffic spike
- Symptoms: Queue wait dominates, p99 in seconds, 5xx from the load balancer, autoscaling reacts minutes later (see Scaling & Observability). In Lab 5 a mere 120 % turned p50 into 124 s without shedding.
- Blast radius: The whole service; shared dependencies get hammered.
- Protects you: Rate limits per client at the edge, admission control by queue wait, CDN/cache for reads, queue-based leveling for writes, pre-scaling for known events, priority shedding.
- Rehearse: Load-test at 3x and 10x with an open-model generator; verify rejection is fast and recovery is automatic.
6. A deployment ships a memory leak
- Symptoms: Heap baseline after each GC rises; GC time share grows; p99 climbs before the OOM; then
java.lang.OutOfMemoryError: Java heap spaceor a containerOOMKilled. Example arithmetic: 1536 MB heap, 500 MB live set, +4 MB/min leak = OOM after about 259 minutes - after the deploy has been declared a success. - Blast radius: A rolling deploy hides it: instances die one by one hours later, then restart and leak again (a sawtooth in availability).
- Protects you: Canary with automatic rollback on SLO burn and on heap-after-GC trend,
-XX:+HeapDumpOnOutOfMemoryError, alerts on old-gen occupancy slope, restarts as a mitigation not a fix. See JVM Memory & GC. - Rehearse: Deploy a build with a deliberate leak flag to the canary and check that the pipeline rolls back by itself.
Game Days, Fault Injection and Abort Criteria
A protection you have never exercised is a hypothesis. Rehearsal turns it into evidence, in three increasing levels: tests in CI, fault injection in staging, and game days in production with a controlled blast radius.
| Fault to inject | Tool (how-to, not run in this sandbox) | What it proves |
|---|---|---|
| Latency / timeout / dropped connections to a dependency | toxiproxy-cli toxic add -t latency -a latency=1000 -a jitter=200 payment (Toxiproxy proxies a TCP service and adds "toxics") | Timeouts, breaker, bulkhead and fallback work |
| Dependency fully down | toxiproxy-cli toxic add -t timeout -a timeout=0 redis or stop the container | Cache-down path; DB stays inside its ceiling |
| Network delay/loss on a host | tc qdisc add dev eth0 root netem delay 200ms 50ms loss 1% (Linux, needs root) | Hedging, retries, timeouts under packet loss |
| Kill instances / tasks / AZ | AWS Fault Injection Service experiment templates (stop tasks/instances, network disruption - check the current action list), or Chaos Mesh / Litmus on EKS (PodChaos, NetworkChaos) | Autoscaling, health checks, AZ headroom |
| CPU or memory pressure | stress-ng, FIS resource actions, or a leak flag in a canary build | Throttling behaviour, OOM handling, canary rollback |
| Traffic surge | k6/Gatling open-model stages at 3x / 10x of peak | Shedding, autoscaling lag, rate limits |
- Chaos without observability. If you cannot see p99, queue depth and breaker state in real time, you are only breaking things.
- No abort criteria or no owner. One person has the stop button; abort limits are numeric.
- Testing only the happy fault. Total outage is easier to survive than "5x slower" or "1 % packet loss" - inject gradual and partial faults too.
- One-off exercises. Capacity and failure behaviour regress with every release; automate the cheap scenarios in the pipeline and repeat game days each quarter.
The Plain-Java Source I Ran
Plain Java 21, no dependencies: javac Hockey.java && java Hockey, javac DbSlow.java && java DbSlow, javac Capacity.java && java Capacity. All three are deterministic (fixed seeds) except for tiny differences you get from changing the seed.
SVC_MS to a heavy-tailed distribution (lognormal) in Hockey.java and watch the knee move left; give DbSlow 100 threads or a 200 ms acquire timeout and see which variable changes the incident most; replace Capacity's inputs with numbers from your own JFR/Hikari metrics.
Questions You Should Be Able to Answer
- Q: How do you decide how many instances a service needs?
Per instance: min over resources of capacity / time-per-request (CPU, threads, connections), times a target utilisation (50-70 %). Fleet: ceil(peak / planned) plus N+1 (or AZ) headroom. Then check every shared resource (DB QPS, connections, downstream limits, partitions) for the same peak. - Q: Why does latency explode near 100 % utilisation?
Queueing: mean response time = service time / (1 - ρ) for M/M/1 - 2x at 50 %, 10x at 90 %, 50x at 98 %. The tail is worse. Pooling (more servers) delays the knee: an 8-worker pool is 1.8x at 90 %. - Q: We doubled the instances and throughput did not move. Why?
The bottleneck is shared: DB CPU/IO or QPS, connection slots, a lock, a downstream rate limit, a hot partition. Evidence: app CPU low,hikaricp.connections.pendinghigh, DB saturated. Fix queries per request, scale the shared tier, or remove coordination. - Q: How does instances x pool size interact with the database?
Total connections = instances x maximumPoolSize must stay under the DB budget (max_connections minus reserve); autoscaling max must be derived from it. RDS Proxy / PgBouncer multiplex when it does not fit. - Q: The DB is slow and unrelated endpoints are failing. Why?
Shared Tomcat threads are blocked waiting on the DB. Isolate with a bulkhead or separate pools, fail fast on connection acquire, keep health checks off the DB path. Our model: /ping p99 413 ms and 25.8 % errors → 1 ms and 0 %. - Q: What is load shedding and where do you put it?
Rejecting some work early so admitted work stays fast. At the edge (rate limits), then admission control by queue wait in the service, before expensive work. Send 503 + Retry-After; clients back off with jitter. - Q: A new release leaks 4 MB/min. Why is that hard to catch?
The OOM comes after hours (259 min in the example) when the deploy looks healthy. Canary + automatic rollback on heap-after-GC slope and SLO burn, heap dump on OOM, and restarts as mitigation only. - Q: How do you rehearse failure safely?
Steady-state hypothesis, small blast radius, numeric abort criteria, tool-driven faults (Toxiproxy, tc netem, FIS), run in staging first, then production at low traffic; automate the cheap ones in CI. - Q: What does the Universal Scalability Law add to Amdahl's Law?
A coherency term β: contention (α) limits speed-up to a plateau (Amdahl), coherency cost makes throughput decline beyond a peak N = sqrt((1-α)/β). Fit α and β from at least 5 measurements. - Q: Why is autoscaling not a capacity plan?
It adds app-tier capacity after minutes of delay and does nothing for DB, quotas, partitions, or warm-up; it can amplify DB overload by adding connections. Plan the shared limits and pre-scale known events.
Cheat Sheet and Recap
| Question | Formula / rule | OrderService example |
|---|---|---|
| Threads or connections needed | L = λ · W | 1500 rps x 45 ms = 67 threads fleet-wide; pool 10 / 35 ms = 286 rps |
| Per-instance ceiling | min(capacity / time-per-request) over resources | min(4444, 286, 250) = 250 rps (CPU) |
| Instances at peak | ceil(peak / (ceiling x target util)) | ceil(1500 / 150) = 10; 15 with AZ headroom |
| DB connection budget | instances x pool <= max_connections - reserve | 60 x 10 = 600 > 450: shrink pool or add a proxy |
| Shared-tier wall | DB QPS ceiling / queries per request | 15000 / 6 = 2500 rps (1.7x today) |
| Latency vs utilisation | W = S / (1 - ρ) (M/M/1) | 2x at 50 %, 10x at 90 %, 50x at 98 % |
| Queue drain time | backlog / (consume - produce) | 360000 / 1000 = 6 min |
| Thread hold with slow downstream | threads / timeout | 50 / 2 s = 25 rps |
- Throughput = min over resources; find the minimum per instance, then the shared minimum for the fleet.
- Utilisation is a latency dial: the knee is ~80 % for a single resource, ~90 % for a pool of 8; plan peaks at 50-70 %.
- Connections are a shared budget: instances x pool must fit; autoscaling max follows from it.
- Adding instances stops helping at the shared bottleneck (here at 17 instances) and may reverse past the USL peak.
- Overload needs rejection: shed early by predicted queue wait; otherwise 120 % load means a 2-minute p50.
- A slow dependency is worse than a dead one: bulkhead the thread pool, short timeouts, fast failure; keep health checks off the failing path.
- Rehearse every protection (Toxiproxy, netem, FIS, canary leak) with abort criteria. Numbers here are model results and arithmetic on illustrative inputs - re-derive them with your own measurements.
- Measure First - Little's Law, percentiles, USL basics.
- Distributed Systems & Messaging - retries, breakers, hedging, queues.
- Database Performance - the Hikari pool and the DB wall in depth.
- AWS Compute & Containers - the next page: CPU limits, throttling and container memory.
- Scaling & Observability - reacting to load in time and watching it.
- Spring: Caching & Resilience - Resilience4j bulkhead/breaker wiring.