The symptom. A pull request on OrderService claims "string building is 30 % faster, measured". Two weeks later the load-test report says the new build sustains 2x the traffic - and production p99 is unchanged. Both numbers were honest measurements. Both were wrong in a way the tool made easy to miss.

Benchmarking has two failure modes. A microbenchmark measures a few lines of code in isolation and is easily fooled by the JIT compiler (dead-code elimination, constant folding, warm-up, profile pollution). A load test measures a whole system and is easily fooled by its own methodology (closed loops, coordinated omission, unrealistic data, a noisy machine). This page runs real experiments on both: a genuine JMH 1.37 run, a hand-rolled timer that lies, and a plain-Java open-model load generator against a local HTTP endpoint.

Choose the level of benchmark by the question. Never use a microbenchmark to predict system throughput, and never use a system load test to choose between two string-building idioms.

The PR compares building an order-line string with += in a loop against a StringBuilder. The author timed one call of each with System.nanoTime() and also a 20,000-iteration loop. Reviewers ask: can we believe it?

A hand-rolled timer measures whatever the JVM is doing at that moment: class loading, interpreter, C1, C2, and whichever method happened to run first. If true, merely swapping the order of the two measurements will change the answer.

NaiveBench times each variant once (cold), then times 20,000 calls of each in both orders. Run three times:

• The first plus() call cost 52-181 ms - not string work but the one-off bootstrap of the invokedynamic string-concat machinery. A single cold timing measures class loading.
• The loop result for plus ranged from 2.4 us to 17.8 us per call depending only on whether it ran first or second, and across identical runs (14.5 us vs 17.8 us). Whoever runs first pays for compilation of shared code and for a cold CPU.
• The ranking stayed plus > builder here, but the "30 %" is unsupported: the ratio swung from 1.6x to 3.8x (e.g. 2,376 vs 1,470 and 14,494 vs 3,859).

Replace the hand-rolled timer by JMH (next chapter). Everything else - the two methods, the 50 order lines, the machine - stays the same.

Chapter 04 gives the JMH answer, with confidence intervals. The verification here is negative: the old method could not have produced a repeatable number, as the three runs show.

We now want trustworthy answers to four questions about OrderService internals: (a) how expensive are += and StringBuilder for 50 lines, (b) loop vs IntStream for summing line totals, and - to see the traps - (c) what does a benchmark report when the JIT deletes its work, and (d) when it folds the work into a constant.

• StringBuilder beats += by 2-3x at 50 appends (+= allocates a new String each iteration).
• A plain loop beats a stream for a 50-element sum.
• A benchmark whose result is unused will report ~1 ns; one that computes Math.sqrt of a constant will report ~1 ns too. Both numbers will be meaningless.

JMH is a Maven Central library with an annotation processor. A pom is enough to download it; the commands below (plain javac, no shading) are what produced the numbers on this page. JMH needs the benchmark in a named package.

• deadCode = 1.68 ns vs returned = 247 ns for the same 50-element sqrt loop: with the result unused the JIT removed the loop. A 150x "speed-up" that is really "nothing was measured".
• constantFolded = 1.09 ns vs notFolded = 6.39 ns: Math.sqrt(144.0) was folded into the constant 12.0; reading x from a state field forces the real operation.
• concatPlus 3,113 +/- 1,460 vs concatBuilder 1,148 +/- 321 ns: intervals do not overlap, so builder is faster (about 2.7x). The wide intervals say the machine was noisy.
• sumLoop 32.7 +/- 9.6 vs sumStream 79.8 +/- 35.3 ns: non-overlapping again; the stream pays set-up cost that dominates for 50 elements.
• returned (247 +/- 163) vs blackholed (408 +/- 160) are statistically the same code; the gap is noise. This is exactly why you read the error column before telling a story.
• In fork 1 of blackholed the iterations read 93, 121, then ~500 ns: with no code change, a competing process stole the cores. Evidence is in the raw log, so keep it.

Within OrderService, the single decision these results support is: build per-order strings with a StringBuilder (or String.join/a streaming JSON writer) rather than += in a loop. Note what it supports not: it does not say the request will be faster, only that this operation is cheaper. Whether 2 us of savings matters against a 650 ms database call is a Chapter 08 question (Measure First).

Verification means a rerun that agrees within error bars. Re-running the whole suite twice more (see Chapter 08) gave concatPlus 2,769 / 3,163 / 3,113 ns and concatBuilder consistently several times lower, so the ordering is trustworthy; the absolute values are not. A claim of "30 % faster" would have needed non-overlapping intervals from a quiet machine.

The OrderService test endpoint is hammered with a closed-loop tool (two workers) and the report shows p99 = 29 ms. A release that includes a 1.5 s stop-the-world-style freeze looks perfectly healthy. In Chapter 05 of Measure First this was a simulation; here it is a real HTTP server and a real client.

A closed-loop client stops sending during the freeze (both workers are stuck waiting), so it records ~2 slow samples. An open-model client keeps issuing 100 requests per second on schedule, so ~150 requests experience the freeze. We expect a p99 near 1.4 s vs near 29 ms.

The server (com.sun.net.httpserver, 64 worker threads) sleeps 20 ms per request as a stand-in for a database call. For the "stall" variant, the handler additionally sleeps until a deadline 5 s into the run, creating a 1.5 s freeze that every in-flight request experiences. The open-model client computes intended = t0 + i x interval, spins until that moment, submits the request on a virtual thread and records completion - intended. Source in Chapter 09.

• No stall: all three measures agree (p99 27.5 / 28.1 / 28.8 ms). The tool is fine when nothing hurts.
• With the freeze: closed loop p99 = 29.0 ms, but p99.9 = 1,510 ms - the freeze is visible only as 2 samples out of 1,226. The loop also reports 81.7 rps instead of the intended 100: the tell-tale of omission.
• Open model: p95 = 821 ms, p99 = 1,410 ms, mean 103.5 ms; 1,500 samples, exactly 100 rps.
• Measuring from send vs from intended time differs by under 4 ms here (1,410.2 vs 1,413.7): virtual threads let the generator launch every request on time, so the client itself did not lag. If the generator had been saturated (e.g. too few threads), only the "intended" column would have exposed it.

Switch from closed (N workers, think-less) to open (fixed arrival rate), keep everything else. Where your tool supports it: k6 constant-arrival-rate / ramping-arrival-rate executors, Gatling open injection profiles (constantUsersPerSec, rampUsersPerSec), wrk2 (-R rate, records latency from the scheduled time). These are named from tool documentation; they were not run in this sandbox.

Capacity planning starts with one number per instance: the arrival rate at which the SLO breaks. Guessing ("it handled 300 in the demo") is how outages are born.

By Little's Law the test server (8 worker threads, 20 ms per request) can hold 8 requests in service, so max throughput is about 8 / 0.0207 s = 386 requests/s. Below that, latency should be flat at ~21 ms; above it, a queue forms and latency should grow with time under load.

LoadGen step offers 100, 200, 300, 340, 370, 400 and 450 req/s for 6 s each in the open model against the 8-worker server. Latency is measured from intended send time. (The leading "rps" in the report is the offered rate n / seconds.)

• From 200 to 340 req/s p99 stays 21-23 ms: the server is not queueing. Even at 370 req/s (96 % of capacity) p99 is only 29.9 ms - but that is the last stable step.
• At 400 req/s (4 % above the 386/s limit) p50 jumps from 21 ms to 179 ms and p99 to 270 ms; at 450 it is p50 578 ms / p99 1,005 ms. Latency grows with test duration because the backlog grows - the system is unstable, not slow.
• The knee matches the prediction within a few percent. The SLO (say p99 < 50 ms) is met up to ~370 req/s; the safe operating point is lower, ~300 (78 % utilisation), to leave headroom for jitter.

The fix for a saturated tier is one of: raise capacity (more workers or instances), lower service time (the 20 ms is the lever - Chapter 06 of Measure First), or shed load. We change none of them here; the experiment's product is the number "~370 rps per instance at p99 < 50 ms", which feeds Capacity Planning.

Because the unstable steps lasted only 6 s, their latencies are a function of how long you ran them. A soak at 400 rps would show seconds, then timeouts. Report such steps as "saturated", not as a latency.

The experiments above used a trivial endpoint. A load test of real OrderService adds three realism requirements: the right traffic shape, realistic data and mix, and a complete record of what was run.

While producing this page, the benchmark suite was run three times unchanged. During all of them the sandbox was shared with other compile and test jobs: uptime reported load averages of 15-38 on a 12-core machine.

A noisy neighbour steals CPU time and cache; short, CPU-bound benchmarks suffer most, and the effect is unevenly distributed between iterations, forks and runs. Expect scores to wander and error bars to widen.

(Run 1 used the first draft of the dead-code benchmark so only the four unchanged benchmarks are comparable. Run 2 load average fell from 38 to 21; run 3 stayed ~21-22. The load values are the uptime readings taken at start and end of each run.)

• sumStream varied from 47 to 157 ns - 3.3x - for identical code. concatBuilder in run 2 (2,960 ns) was not faster than concatPlus (3,163 +/- 3,169): error bars overlapped completely, so run 2 alone would have refuted the correct conclusion.
• The error column honestly warned about it: the run with the worst ranking also had +/- 3,169 ns on a 3,163 ns score (100 %).
• Fork-level logs showed intra-run swings (93 ns to 500 ns inside one fork) - a sign of CPU theft, not of JIT behaviour.

A claim is verified when a reviewer can rerun the recorded procedure on a quiet machine and obtain results whose confidence intervals overlap yours. The summary you publish should state: hardware, load at start/end, forks, iterations, and for each comparison whether the intervals are disjoint.

All numbers on this page were produced by the code below on JDK 21.0.2 with JMH 1.37, on a 12-core laptop-class machine that was busy with other work (load average 15-38). Expect different absolute numbers; the shapes - dead code at ~1 ns, a hockey stick at the capacity predicted by workers / service time, and an open-model p99 that sees the freeze - should reproduce.

• Q: Why do we need warm-up in a Java benchmark?
  Code starts interpreted, then C1, then C2 compiles hot paths using the profile gathered so far. Early iterations measure compilation and class loading; in the lab the first 50 ms iteration of a builder loop was 3x steady state.
• Q: What is dead-code elimination in JMH terms?
  If a benchmark computes a value nobody uses the JIT may delete the computation. In the lab, an unused 50-element sqrt loop reported 1.7 ns while the same loop with its result returned took 247 ns. Fix: return the value or pass it to a Blackhole.
• Q: Why must benchmark inputs come from @State fields?
  Compile-time constants let the JIT fold the work away (Math.sqrt of a literal is evaluated at compile time).
• Q: What do forks add?
  Fresh JVMs, so one run's JIT decisions do not contaminate or favour another; disagreement between forks is a warning.
• Q: Open vs closed workload - when each?
  Open when arrivals are independent of the system (web traffic); closed when a fixed population loops (workers behind a pool, batch jobs). Closed tools under-report outages (coordinated omission).
• Q: How do you find single-instance capacity?
  Step test in the open model, watch p99 and the queue; capacity is the last step meeting the SLO, minus headroom. Cross-check with workers / service time (Little).
• Q: Your benchmark shows 20 % improvement. What do you check before merging?
  Error bars/CI disjoint, repeated on a quiet machine, same JDK and flags, other benchmark in the same suite not regressed, and whether the changed code is even on the critical path of a real request.
• Q: What makes a load test realistic?
  Production-shaped data and request mix, open arrivals, realistic downstream latency including tails, warm and cold cache states, a generator that is not the bottleneck.