The campaign from Capacity & Failure is on. Traffic to OrderService ramps from 600 to 1800 requests/s over two minutes. The ECS service autoscaling policy is textbook: target tracking on average CPU at 60 %. The dashboard shows what looks like a bug: CPU is at 100 % for minutes, p99 is 4+ seconds, yet the service "is scaling". It is - but each step of the loop takes time, and the delays add up to more than the ramp lasts.

Observability is the other half of the same problem. The first symptom in this incident was not an alarm but a user complaint, because the dashboard plotted average latency and average CPU across instances. This page covers both halves: how to make capacity arrive before the load (Chapters 2-5), and how to see - fast and truthfully - what the service is doing (Chapters 6-11).

If the loop has dead time, shorten it where you can and compensate where you cannot. Three levers: choose a signal that moves before CPU does, lower the number of confirmations you need, and use feed-forward (schedules, predictions, queue depth) for known patterns.

A new task is not new capacity until it is ready and warm. Three mechanisms decide how soon: health probes (when does traffic arrive?), load balancer slow start (how fast does it ramp?), and graceful shutdown (what happens to in-flight requests on scale-in and deploys).

You need a small, fixed set of metrics on every service dashboard. The classic frames are the RED method (Rate, Errors, Duration) for request-driven services and USE (Utilisation, Saturation, Errors) for resources, both introduced in Measure First. With Micrometer in Spring Boot the names below are what you get.

Metrics tell you that p99 is bad; a trace tells you where one slow request spent its time. A trace is a tree of spans (name, start, duration, parent) joined by a trace id that travels in HTTP headers (W3C traceparent) or message headers. In Spring Boot 3 the pieces are Micrometer Observation (instrumentation API), Micrometer Tracing (bridge) and OpenTelemetry (exporter).

Below is a real trace from the hand-rolled tracer in Obs.java (spans are explicit; the durations are real Thread.sleep times of a slow request modelled on the 800 ms OrderService request from Measure First): one slow GET /orders/42.

How to read a waterfall. (1) Find the widest span under the root: that is where time went. (2) Look for repetition - many short identical children means N+1; (3) look at gaps between a parent and its children - time spent in the parent itself or waiting for a thread; (4) look at stairs - sequential children that could be parallel; (5) compare with a fast trace of the same endpoint (is it always slow or only sometimes?). The 650 ms of the OrderService request is the database span group, exactly the attribution exercise from the methodology page.

An SLO turns "fast enough" into a number you can alert on. Define a service level indicator (SLI) as a ratio of good events to total events (for example: requests answered successfully in under 300 ms), an SLO as the target (99.9 % over 30 days) and an error budget as the allowed failures (0.1 % - about 43 minutes of full outage). Then alert on how fast you are spending the budget, not on a raw error count.

The pairs are the multiwindow, multi-burn-rate pattern from the Google SRE workbook: the long window proves it is not a blip, the short window proves it is still happening (so the alert resolves quickly). 14.4x for 1 hour consumes 14.4 x 1/720 = 2 % of a 30-day budget, which is why that number appears in the table.

On AWS the same telemetry reaches three families of destinations. None of this could be run in the sandbox (no AWS access); the architecture and formats are what the services document, so check current docs for limits and prices.

When the SLO alert fires, the order of questions matters more than the tools. This is the shortest path from "burn-rate page" to a cause, using exactly the signals built in this page and the earlier ones.

Plain Java 21, no dependencies: javac Obs.java && java Obs (about 12 seconds; random load, so numbers vary a little between runs) and javac Scale.java && java Scale (instant; deterministic, writes scale.csv).

• Q: Why did autoscaling not save us during the traffic ramp?
  The loop has dead time: 1-minute metric averages, several breaching datapoints, task start (60-120 s), then JIT warm-up. In the simulation CPU target tracking decided at t = 239 s for a ramp that started at t = 60 and left users 243 seconds degraded. Use faster signals, fewer confirmations, headroom or scheduled scaling.
• Q: Which metric should drive autoscaling for an HTTP service?
  Requests per target (ALBRequestCountPerTarget) or in-flight/busy threads, because they lead CPU; keep CPU as a guardrail. For workers use queue depth/age (KEDA, CloudWatch). Latency is a guardrail, not a driver.
• Q: Why can you not average p99 across instances?
  Percentiles are not additive. Merge histogram buckets across instances and then compute the quantile; our run: average of p99s 242 ms vs merged 304 ms vs truth 349 ms.
• Q: Histogram vs summary in Micrometer?
  Histogram buckets (percentiles-histogram / slo) are aggregatable across instances in Prometheus; client-side percentiles are per-instance and cannot be merged. Put bucket boundaries at your SLO thresholds.
• Q: How do you find the 650 ms in a slow request?
  A trace waterfall: the widest span group (here 3 repeated "order_lines x8" spans + 1 query = ~630 ms, an N+1), sequential HTTP calls that could be parallel, and gaps in the parent. Then confirm with DB-side evidence.
• Q: Explain burn rate.
  Burn rate = observed error ratio / allowed error ratio (1 - SLO). Burn rate 1 spends exactly the budget over the window; 14.4 for an hour spends 2 % of a 30-day budget. Alert on multiple windows (long to prove, short to confirm).
• Q: Why readiness vs liveness probes?
  Readiness controls traffic (remove from rotation, e.g. while warming or dependency down); liveness restarts a stuck process. Mixing them causes restarts that make an overload worse.
• Q: We see 502s only during deploys. Causes?
  No graceful shutdown/preStop delay, deregistration delay too short, in-flight requests cut off, or keep-alive shorter than the ALB idle timeout. Fix with graceful shutdown, preStop sleep, and aligned timeouts.
• Q: What goes into an EMF record and what must not?
  Metric values plus a small set of low-cardinality dimensions; high-cardinality identifiers (traceId, userId) stay as plain log fields so they do not multiply metric series.
• Q: How would you tune head vs tail sampling?
  Head sampling is cheap but random; tail sampling in the collector keeps all errors and slow traces. Use head sampling at a low rate plus tail rules for error/latency, and exemplars to link a histogram bucket to a trace.

• Measure First - RED/USE, percentiles, the 800 ms attribution.
• Capacity & Failure - what runs out first; shedding and bulkheads.
• JIT, Startup & Native Image - the warm-up and start-time inputs to the scaling model.
• AWS Compute, Containers & the JVM - CPU limits and throttling, which distort CPU-based scaling.
• Cost & Capstone - the cost of idle headroom vs the cost of late scaling.
• Spring: Actuator & Observability - Micrometer, health groups, tracing configuration.