The symptom. OrderService, the lab system used on every page of this guide, has a p99 of 800 ms on GET /orders/{id} and the SLO says 200 ms. A team spends a week on what sounds right: a bigger heap, a different garbage collector, a faster JSON library, more pods. Latency barely moves. Nobody measured where the 800 ms was spent.

Performance work has an unusual property: intuition is usually wrong, and wrong in a consistent direction - we tune the code we understand and can see, while the time is spent in a network hop, a lock, a database plan or a queue that we cannot see without instruments. The cure is not cleverness; it is a loop that forces evidence before every change.

Every performance investigation, from a microbenchmark to a production incident, is the same four-step loop. What separates good engineers is not knowing more JVM flags but running the loop honestly: writing the hypothesis down, changing one variable, rerunning an identical test.

Four quantities appear in every conversation and are routinely confused. Latency is how long one request takes. Throughput is how many complete per second. Utilisation is the fraction of time a resource is busy. Saturation is work that is waiting because the resource is already busy (a queue). They are linked: as utilisation approaches 100 %, queues form and latency explodes while throughput flattens (Chapter 06 and the hockey-stick lab on page 2 show the real curve).

The OrderService dashboard shows average latency 26 ms and everybody relaxes. Support, meanwhile, is forwarding screenshots of checkout pages that hang for two seconds. Both are true: the average is a single number computed from a distribution that is nothing like a bell curve.

Hypothesis: most requests take the fast path (cache hit, ~15 ms) and a small share fall onto a slow path (a cold query, ~150 ms) or a pathological one (lock wait, retry, ~1-2 s). If so the mean sits in a valley where no real request lives and says nothing about the tail.

We need the distribution, not one number. The program below synthesises 100,000 request latencies from exactly that mixture (97 % ~15 ms, 2.5 % 120-200 ms, 0.5 % 0.8-2 s), sorts them and reads off nearest-rank percentiles. The generator is invented; the arithmetic and the traps it demonstrates are real. Full source in Chapter 09.

• Mean 26.0 ms vs p50 15.1 ms: the mean is pulled up by a 0.5 % tail - the 518 requests above 500 ms (0.5 %) contribute ~7 ms of the 26 ms average all by themselves.
• p99 = 184.5 ms and p99.9 = 1,803.6 ms: one in a hundred users sees ~12x the median; one in a thousand sees ~120x.
• The histogram is multi-modal: three clusters (15 ms, 150 ms, 1.4 s) that correspond to three different code paths. A percentile curve exposes that; a mean erases it.

The optimisation here is to the measurement: replace the average with p50 / p95 / p99 / p99.9 from a histogram, record them per route, and aggregate by merging histogram buckets, never by averaging percentiles. In Spring Boot this is management.metrics.distribution.percentiles-histogram.http.server.requests=true plus SLO buckets (see Actuator & Observability and Scaling & Observability). Which percentile? p99 for a service-level objective; p99.9 where a page fans out to many calls; p50 only to describe the typical case.

Verification of a measurement change is simple: the new view must reveal a problem the old view hid. It did.

Operations froze the OrderService instance for five seconds (think stop-the-world pause, failover or a saturated pool). The load-test report written by a naive closed-loop client says p99 = 5 ms. Customers disagree. Coordinated omission (Gil Tene's term) is the name for this: the measuring client slows down together with the system under test, so it politely stops sending exactly the requests that would have measured the problem.

A client that waits for each response before sending the next only records one slow sample for the whole freeze. A client that sends on a fixed schedule and measures from the intended start time records one slow sample for every request that should have been sent during the freeze - hundreds of them. The p99 should differ by orders of magnitude.

The simulator models a 60 s test at a target of 100 req/s (one intended request every 10 ms). The server answers in 5 ms except between t = 20 s and t = 25 s, when it is frozen. Three clients: (1) closed loop measuring from actual send time, (2) the same client but measuring from the intended send time, (3) an open-model client that sends on schedule regardless of outstanding responses (ground truth). Virtual time makes the result deterministic; page 2 repeats the experiment against a real HTTP server.

• The naive closed loop recorded 5,501 samples instead of 6,000 - 499 requests that should have happened in the freeze were silently never issued. Its p99 is 5.0 ms; max 5,005 ms is a single sample.
• Measuring from intended time (client 2) restores the missing samples: p99 = 4,705 ms. The open model gives 4,405 ms; the small difference is that client 2 is still one connection and queues behind itself.
• A tell-tale of CO in a report: achieved throughput below the target and a max latency far above a tiny p99.9.

Switch the generator from a closed model (N workers, each waits for a reply) to an open model (arrivals on a schedule, e.g. k6 constant-arrival-rate / ramping-arrival-rate executors, Gatling constantUsersPerSec, wrk2 -R), and record latency from the intended time. Real users do not wait for each other, so an open model is also the more honest model of internet traffic.

On page 2 the same effect is measured against a real HTTP endpoint with a real 1.5 s freeze: closed loop p99 = 29.0 ms, open model p99 = 1,410 ms.

OrderService has a Hikari pool of 10 connections. At 200 req/s, each request holds a connection for ~50 ms. It runs fine in a test at 100 req/s and starts throwing SQLTransientConnectionException: ... Connection is not available, request timed out after 30000ms at peak. Nobody changed the code.

Little's Law says the average number of items inside any stable system equals the arrival rate times the average time each spends inside: L = λ x W. For the pool: connections in use = requests/s x seconds each holds a connection. If that product exceeds the pool size, requests must queue - and queues grow without bound.

The program injects open arrivals at a fixed rate, holds each "request" for a fixed service time (a sleep, on a virtual thread), and measures three things independently: arrival rate λ, mean time in system W, and the mean number in the system L by sampling an in-flight counter every 5 ms. If the law holds, the measured L must match λ x W.

The law held to within about 1 % in every row. That is why it is useful: it needs no assumption about the distribution of arrivals or service times, only a stable system. Note the last row - latency changes capacity. A slow dependency does not just slow requests, it multiplies the number of connections, threads and memory you need at the same traffic.

• Required concurrency L = λ x W; add 20-30 % headroom, because utilisation near 100 % queues badly (hockey-stick, see page 11).
• For pools: pool size ≥ peak rps x p95 hold time. 250 rps x 0.060 s = 15; pick ~20, not 10 and not 200.
• The same math sizes thread pools (blocking I/O: threads ≈ rps x time-in-request), and the instance count: instances = peak L / per-instance concurrency.
• Better: shrink W. Cutting query time from 50 ms to 25 ms halves the pool and thread requirement at no hardware cost.

Verify in production with the metric pair hikaricp.connections.active and hikaricp.connections.pending: pending should sit at ~0; active should track λ x W. (Rows above are the law applied to measured L; the pool outcomes are consequences of the law, not separate runs.)

Going from 4 to 8 OrderService instances raised throughput by only ~1.5x; going to 16 added almost nothing. Two classic models explain why, and give you a number to plan with.

• Amdahl's Law. If a fraction s of the work is serial (one shared lock, one primary database writer, one single-threaded step), speed-up with N workers is 1 / (s + (1 - s) / N) and can never exceed 1/s.
• Universal Scalability Law (Neil Gunther). Adds a coherency term κ: workers also pay to keep shared state consistent (cache-line bouncing, lock hand-off, distributed coordination). C(N) = N / (1 + σ(N-1) + κN(N-1)). Because the quadratic term eventually wins, throughput peaks and then falls at N* = sqrt((1-σ)/κ).

This program only evaluates the formulas - it is a calculator, not a measurement of any real service. σ = 5 % and κ = 0.002 are illustrative parameters; in practice you fit them to measured throughput at several concurrency levels (e.g. the step test on page 2).

• With s = 5 %, the best case is 20x; 16 workers give 9.1x (57 % efficiency), 64 give 15.4x.
• With sigma = 5 % and kappa = 0.002 the USL peaks at N = 22 (C = 7.40); at 64 the system delivers 5.24x - less than at 12 (6.62x). Adding capacity made it worse.
• What are the serial parts in OrderService? A single database primary taking every write, one Hikari pool shared by all threads in an instance, a synchronized audit log, a cache with a global lock. Page 5 finds such a lock with a real profile.

Prove a scaling fix by re-running a step test across several N and refitting: the curve should shift up and the knee move right. If throughput(N) is flat or falling, look for the shared resource (USE method) instead of buying nodes. Page 11 turns this into a capacity-planning calculation.

A single GET /orders/{id} takes about 800 ms against a 200 ms target. The architect proposes switching to a faster JSON library; the DBA proposes a bigger instance; the platform team proposes more pods. Which is right? The honest answer is "show me the breakdown".

In the OrderService lab, time is distributed roughly as 30 ms application code, 650 ms database, 20 ms serialization, 100 ms other (downstream HTTP, filters, GC). These are the lab's scenario numbers, chosen so the exercise has a clear answer; your own system will have its own split.

The cheapest attribution tool is a timer around each stage: a span. Real systems use Micrometer Observation / OpenTelemetry (page 13) and read the trace waterfall; the plain-Java version below shows the mechanism with no dependencies. The four stages are simulated with sleeps to the lab's numbers, so what you see reproduced is the bookkeeping and the arithmetic, not a measurement of a real database.

Apply Amdahl to the request: if a stage that takes fraction f of the time becomes k times faster, the overall speed-up is 1 / ((1 - f) + f / k). The what-if table in the output answers every argument at once:

• 10x faster JSON: 1.02x overall (806 ms). The proposed rewrite is a rounding error.
• 10x faster app code: 1.03x. Even a perfect JIT cannot save this request.
• 10x faster downstream call: 1.12x - worth doing, but second.
• 10x faster database stage: 3.45x (824 to 239 ms). Only this change alone moves the request toward the 200 ms target.

Go after the 650 ms and nothing else. What that looks like is page 8 (Database): count the queries (N+1), look at the plan, add the missing index on orders.customer_id. Resist the temptation to also "tidy up" the JSON mapper in the same change - you would pollute the experiment.

Notice the bottleneck moved: after the database fix the downstream call is the largest slice - exactly what the loop predicts. The projected 216 ms is arithmetic from the what-if, labelled as such; the capstone (page 14) performs the real run.

Everything above runs with a bare JDK 21 (javac and java, no dependencies). Save each file under its class name and run it. Absolute numbers will differ on your machine - the shape (mean vs tail, omission, L = λW) will not. Results shown on this page were produced on a 12-core, laptop-class machine that was shared with other builds, which is itself a reminder that measurement environments are noisy.

• Q: Why is average latency a poor SLI?
  Latency distributions are skewed and often multi-modal; the mean is dominated by neither the typical request nor the worst. Use percentiles (p50 for typical, p99/p99.9 for the experience of unlucky users) computed from histograms.
• Q: Can you average p99s from several instances?
  No. Percentiles are not additive. Merge the underlying histograms (sum the buckets) or raw samples, then compute the percentile. Averaging p99 of a healthy and a sick instance gave 510 ms; the true p99 was 940 ms in the lab run.
• Q: What is coordinated omission?
  A measurement bug where the load generator waits for slow responses and therefore stops issuing the very requests that would have recorded the slowness. Fix: open-model arrivals plus latency measured from the intended send time (or HdrHistogram's corrected recording).
• Q: State Little's Law and use it.
  L = lambda x W. At 250 req/s with 60 ms per request, 15 requests are in flight on average, so a 10-connection pool must queue. Also: latency growth multiplies concurrency demand.
• Q: USE vs RED?
  USE inspects resources (utilisation, saturation, errors) to find what is constrained; RED inspects services (rate, errors, duration) to see whether users are affected. Use RED to alert, USE to diagnose.
• Q: Why did doubling instances not double throughput?
  A shared serial resource (DB primary, pool, lock) bounds speed-up (Amdahl) and coordination overhead can even reduce it (USL). Find the shared resource with USE, then fix or shard it before adding nodes.
• Q: A 10x faster component but no visible improvement - why?
  It was a small fraction of the request. Overall speed-up is 1/((1-f)+f/k): 10x on a 3 % stage yields 1.03x.
• Q: How many samples do I need for a p99.9?
  p99.9 means 1 request in 1,000, so a window of 10,000 requests contains only about 10 tail samples. Aggregate over longer windows, or alert on the count of requests above a threshold.
• Q: What is wrong with load-testing against a warm laptop?
  Cold JIT, GC and caches, noisy neighbours, different data volume and no network - see Benchmarking for the discipline of baselines.