Monitoring answers questions you decided to ask in advance: is the CPU above 90 percent, is the error rate above 1 percent, is the health check failing. You build dashboards and alerts for known failure modes. Observability is the property of a system that lets you ask new questions about its behaviour from the outside, without shipping new code: why are only requests from tenant 42 slow, why did latency double right after the 14:05 deploy. In practice the difference is whether your telemetry is rich enough (high-detail events, consistent identifiers, traces) to debug failures you did not predict.

Telemetry comes in three signal types, usually called the three pillars. They are complementary, not competing. Each is good at one question and poor at the others.

Two further signals appear in the OpenTelemetry world: profiles (continuous profiling, still maturing) and events. This page covers the three that every team needs and how to wire them together, using Prometheus, Grafana, Loki, Tempo and OpenTelemetry, with Spring Boot as the example application. Page 05 CI/CD Pipelines shows where these checks run after a deploy; page 08 Deployment Strategies uses the metrics from this page to gate canary releases.

Prometheus is a time-series database plus a scraper plus a query language. Its data model is small. A time series is identified by a metric name and a set of labels (key/value pairs); each series stores (timestamp, float64) samples. http_server_requests_seconds_count{method="POST",status="200",uri="/orders"} is one series; changing any label value gives a different series.

Prometheus pulls: every scrape interval it sends GET /metrics (or whatever metrics_path says) to every target and stores what comes back. The text exposition format is plain lines of name{labels} value, plus # HELP and # TYPE comments. Pull has practical benefits: a dead target is detected immediately (the scrape fails, up becomes 0), you can curl any target by hand to see exactly what Prometheus sees, and targets need no knowledge of where Prometheus lives. Push is available for short-lived batch jobs through the Pushgateway, but it is the exception.

Histogram buckets are cumulative: the le="0.1" bucket counts every observation less than or equal to 0.1 seconds, so it includes the 0.05 bucket. _count equals the +Inf bucket. Average latency is _sum / _count; percentiles are estimated from the buckets by interpolation, so their accuracy depends on how well the bucket boundaries fit your real latencies.

A minimal prometheus.yml has global timing, optional rule files and a list of scrape jobs. scrape_interval controls resolution and cost; the default is 1 minute, and 15 to 30 seconds is common. The job label is set from the job_name, and instance from the target address.

In Kubernetes you rarely list targets by hand. kubernetes_sd_configs discovers pods, and relabel_configs decides which to scrape and which labels to attach (usually through an annotation convention or, with the Prometheus Operator, ServiceMonitor and PodMonitor objects). The pipeline is always: discover, relabel (filter and add labels), scrape, optionally metric_relabel_configs to drop series before storing.

A selector like http_server_requests_seconds_count{uri="/orders"} returns an instant vector (one sample per matching series). Adding [5m] makes a range vector (the last five minutes of samples per series), which functions such as rate() consume. Aggregations (sum, avg, max, topk) collapse series, and by (...) or without (...) chooses which labels survive.

I ran queries 2, 3, 4 and 5 against the sample application of chapter 11 with Prometheus 3.5.0. With about 10 percent of orders failing by design, the error ratio settled near 0.11, the 95th percentile of the payment call came out at about 0.2 seconds (the simulated call sleeps between 20 and 200 ms), and the status breakdown showed roughly one 502 for every eight 200 responses. Your numbers will differ; the point is that every query is a rate over a window, aggregated, and only then converted to a ratio or quantile.

In the lab this alert went from inactive to pending to firing as the error ratio stayed above 5 percent: for: 1m means the condition must hold continuously for a minute before the alert fires, which filters out single noisy evaluations.

Cardinality is the number of distinct time series. Prometheus keeps recent series in memory and indexes every label value, so its cost scales with the number of series, not with request volume. A busy service with a handful of label values is cheap; a quiet service with an unbounded label is not.

Cardinality is why exemplars exist: a histogram bucket can carry a single sample trace id, so you can jump from a latency spike to one concrete trace without ever adding a trace-id label. Spring Boot supports Prometheus exemplars when a Micrometer Tracing span context is available (Spring Boot reference, Actuator metrics section).

Micrometer is the metrics facade of the Spring ecosystem, the SLF4J of metrics: your code talks to a MeterRegistry, and a registry implementation translates to a backend (Prometheus, OTLP, CloudWatch, Datadog and others). Spring Boot Actuator auto-configures Micrometer, instruments the JVM, the HTTP server, the HTTP clients, the datasource pool and more, and exposes management endpoints over HTTP.

To expose metrics to Prometheus you need two things: the Actuator starter, and the Prometheus registry. The endpoint is /actuator/prometheus; by default it is not exposed, so you must list it in management.endpoints.web.exposure.include (the default there is only health). The Spring Boot reference documents exactly this endpoint and the matching scrape_config.

Start the application and curl localhost:8080/actuator/prometheus. You immediately get JVM, process, Tomcat-independent HTTP and logging metrics without writing code. I built and ran the sample of this chapter (Spring Boot 3.5.16, Java 21) and the endpoint returned the series shown in chapter 2, including http_server_requests_seconds_count labelled by method, status, uri and outcome.

Inject the MeterRegistry and register meters once (in the constructor), then update them on the hot path. Register a meter once and keep the reference; building the same meter again on every request works (the registry returns the existing one) but wastes time on a hot path.

By default the http.server.requests timer exports a summary-style count, sum and max: enough for average latency and rate but not for percentiles across pods. Turn on a histogram for that meter (or for a prefix) in configuration, then use histogram_quantile as in chapter 3.

The slo boundaries add buckets exactly at the thresholds you care about, which makes "fraction of requests under 300 ms" an exact count rather than an interpolation. Each added bucket multiplies series by the number of tag combinations, so enable histograms where you need them, not everywhere.

A log line like Order 8841 failed for user 17: payment declined is readable by humans and nearly useless to machines: to count failures per user you need a regex. Structured logging writes each event as a JSON object with named fields, so a log backend can filter and aggregate on fields without parsing prose.

Spring Boot 3.4 and later has built-in structured logging. One property switches the console output to JSON in a chosen format; the formats it ships are Elastic Common Schema (ecs), Logstash (logstash) and Graylog Extended Log Format (gelf).

A request in a busy service interleaves with hundreds of others in the same log stream. The fix is an identifier on every line of that request. SLF4J provides the MDC (Mapped Diagnostic Context), a thread-local map whose entries the logging framework adds to each event. Put the id in at the edge (a servlet filter), and it appears on every log line produced by that thread without passing it around.

Note how the structured format picked up requestId from the MDC automatically: that is the reason to use a structured encoder instead of a hand-written pattern. Also note MDC.remove in finally: server threads are pooled, and an id left behind would be attributed to the next, unrelated request.

A request id is useful inside one service. A trace id is the same idea across services, generated by the tracing library and propagated in headers (chapter 7). When Micrometer Tracing is on the classpath, Spring Boot puts traceId and spanId in the MDC for you and includes them in the default log pattern; the chapter 8 lab shows them in the JSON output. Prefer the trace id as your correlation id once tracing exists, because it also links the logs to the trace view in Grafana.

Both are valid choices. The qualitative difference is the trade between index everything, search anything (Elasticsearch) and index little, store cheaply, filter by label first (Loki). Teams that already live in Grafana for metrics and traces often pick Loki; teams that need heavy ad hoc log analytics pick Elasticsearch or OpenSearch. In Kubernetes the usual pattern is: the application writes JSON to stdout, the container runtime stores it on the node, and a DaemonSet agent tails and ships it.

• Log cost is roughly events x average size x retention, plus indexing. A debug line inside a hot loop can double a bill.
• Never log secrets, tokens, full card numbers or personal data. Redact at the source; downstream scrubbing is unreliable.
• Log an error once, at the place that handles it, with the stack trace. Logging and rethrowing at every layer produces duplicates that bury the real signal.
• Sample or rate-limit repeated identical lines (a retry loop that logs every 10 ms for an hour).
• Retention is a policy decision: keep detailed logs for days, aggregates and audit data longer.

A trace is the record of one request as it crosses services. It is a tree of spans. A span is one timed unit of work: an incoming HTTP request, an outgoing call, a database query, a function you chose to wrap. Each span has a name, start time, duration, key/value attributes (http.request.method, db.system), a status, a span id, and the id of its parent span. All spans of a trace share one trace id.

A waterfall like this is why tracing exists. Metrics told you p99 latency of /checkout doubled; the trace shows that the time sits in one span, the payment service calling an external card provider. No amount of staring at service-level averages shows that.

For spans from different processes to join one trace, the caller must pass the trace context to the callee, and the callee must pick it up. This is context propagation. The standard is W3C Trace Context, which defines the traceparent and tracestate HTTP headers. It replaced vendor formats such as Zipkin B3 as the default in OpenTelemetry.

tracestate is a second header carrying vendor-specific key/value pairs alongside traceparent. Applications rarely touch either by hand: the tracing library injects them into outgoing HTTP, gRPC and messaging calls and extracts them from incoming ones. In the lab of chapter 8 I sent a request with the specification's example header, and the application's log lines for that request carried "traceId":"0af7651916cd43dd8448eb211c80319c": it continued the caller's trace instead of starting a new one.

Recording and storing every span of every request is expensive at scale, so tracing systems sample. The decision can be made at two moments.

OpenTelemetry (OTel) is a CNCF project that standardises how telemetry is produced and shipped, independent of the backend. It defines an API (what instrumentation calls), an SDK (the implementation that samples, batches and exports), semantic conventions (standard attribute names), the OTLP wire protocol, and the Collector, a standalone service that receives, processes and exports telemetry.

For Spring Boot 3.x the supported way is Micrometer Tracing with the OpenTelemetry bridge. The Boot 3.5 reference lists two dependencies for tracing with OTLP: io.micrometer:micrometer-tracing-bridge-otel (bridges the Observation API to OpenTelemetry) and io.opentelemetry:opentelemetry-exporter-otlp (reports to an OTLP endpoint), configured with management.otlp.tracing.*. In Spring Boot 4 the reference instead names a single starter, org.springframework.boot:spring-boot-starter-opentelemetry, with management.opentelemetry.tracing.export.otlp.* properties. The concepts are identical; only the packaging and property prefix changed, so check the reference for your Boot version.

The lab run: I started an OpenTelemetry Collector (image otel/opentelemetry-collector, version 0.164.0) with an OTLP receiver and the debug exporter, started the sample application with these two dependencies added, and sent two requests. The collector logged "resource spans": 2, "spans": 2, and each application log line carried traceId and spanId fields. The second request sent the traceparent header of chapter 7, and its log line carried that same trace id:

A realistic production pipeline replaces the debug exporter with exporters for your backends (for example otlp to Tempo, prometheusremotewrite (contrib distribution) or a Prometheus scrape of the collector for metrics, otlphttp to Loki), and adds memory_limiter first and batch last in the processor list. The collector configuration has receivers, processors, exporters, connectors, extensions and a service section whose pipelines decide what is wired together; a component that is defined but not listed in a pipeline does nothing.

Alerting on raw thresholds (CPU above 80 percent, error rate above 1 percent for 5 minutes) asks the wrong question. The right question is: are users getting the reliability we promised, and are we on course to break that promise? Site Reliability Engineering turns this into three terms.

The error budget turns reliability into a shared, measurable resource. While budget remains, the team ships features and takes risks (canary releases, chaos experiments). When it is spent, the team shifts effort to reliability work. It also ends the argument "is 100 percent the goal": it is not, because each extra nine costs far more and users cannot tell the difference beyond their own network's reliability.

• Example service: 10,000,000 requests in 30 days, SLO 99.9 percent. Error budget = 0.001 x 10,000,000 = 10,000 failed requests.
• A bad deploy fails 2 percent of requests (burn rate = 0.02 / 0.001 = 20). At 200 requests per second that is 4 failures per second, so the 10,000-request budget lasts 10,000 / 4 = 2,500 seconds, about 42 minutes, even though the service is "98 percent up".
• The same budget at a constant 0.1 percent error ratio (burn rate 1) lasts exactly the 30 days.
• These are worked examples with made-up traffic; use your own request counts.

A simple alert "error ratio above X for 5 minutes" fires too often on harmless blips and too late on slow burns. The Google SRE Workbook (chapter "Alerting on SLOs") compares several approaches and recommends multiple burn-rate alerts with a long and a short window: the long window proves the burn is significant (it already consumed a meaningful share of the budget), the short window proves it is still happening, so the alert resets quickly after the fix. For a 99.9 percent SLO it recommends:

Check the arithmetic: budget consumed = burn rate x window / SLO period. For the first row 14.4 x 1 h / 720 h = 0.02, so 2 percent; the second 6 x 6 / 720 = 0.05; the third 1 x 72 / 720 = 0.10. The threshold on the error ratio is burn rate x (1 - SLO), for example 14.4 x 0.001 = 1.44 percent. The short window is one twelfth of the long window.

Prometheus evaluates alert rules; it does not send notifications. Firing alerts go to Alertmanager, which deduplicates, groups, silences, inhibits and routes them to receivers (a pager, chat, email, a webhook).

Grafana queries many data sources (Prometheus, Loki, Tempo, CloudWatch, SQL) and draws them. Good dashboards are designed top-down: one overview with the golden signals and SLO status for the service, then drill-down rows (per route, per dependency, JVM, database pool), with template variables ($service, $env) so one dashboard serves every deployment.

• Put the question in the panel title ("Are we serving errors?") and the unit on the axis.
• Show the SLO line and the current burn rate; users should see "healthy or not" in two seconds.
• Use rate() windows tied to $__rate_interval rather than hard-coded [5m], so panels stay correct when you zoom.
• Provision data sources and dashboards as code (the lab mounts a provisioning YAML for the data source) so a rebuilt Grafana is not empty; keep dashboard JSON in Git.
• Link panels to logs and traces: with a trace id in the log line and Loki and Tempo configured, Grafana can jump from a log line to its trace and back, and exemplars jump from a latency graph to one trace.

This lab was run on Linux with Docker Compose, Prometheus 3.5.0 and Grafana 12.1.0 (the image tags used), and a Spring Boot 3.5.16 application on Java 21. The application runs on the host and the containers reach it through host.docker.internal (mapped with extra_hosts and host-gateway, which works on Linux). Docker Desktop provides that name automatically.

Create the project from chapter 5 and chapter 6 (pom.xml with web, actuator and micrometer-registry-prometheus; ShopApplication, OrderController, CorrelationFilter; the application.yml shown there plus logging.structured.format.console: ecs). POST /orders sleeps 20 to 200 ms to simulate a payment call and fails with 502 about one time in ten. Build and run it.

The first command prints nothing on success. The health output is the real response from the lab; startup took about ten seconds on this machine.

Open http://localhost:9090/targets to see the target, http://localhost:9090/alerts for the rule, and http://localhost:3000 for Grafana (login admin with the password set in the compose file; the data source is pre-provisioned). Prometheus took a few scrape intervals before rates were non-zero: rate() needs at least two samples after the traffic starts.

Both the p95 and the mean are consistent with the code: the simulated call is uniform between 20 and 200 ms, so the mean should be near 110 ms and the 95th percentile near 190 ms. The alert OrdersHighErrorRatio moved to pending while the ratio stayed above 5 percent (the check ran moments before the minute elapsed, so I did not see firing; the state machine is exactly what the for: clause describes).

Add the two dependencies and the management.otlp.tracing and management.tracing.sampling properties of chapter 8, start the Collector with otel-collector.yml (for example docker run --network host -v $PWD/otel-collector.yml:/c.yml:ro otel/opentelemetry-collector:latest --config=/c.yml), and send requests. With verbosity: basic the debug exporter prints a count of spans per batch; raise it to detailed to see every span.

Crisp answers for the observability questions that come up in DevOps and backend interviews.

• Q: What is the difference between monitoring and observability?
  Monitoring checks known conditions with predefined dashboards and alerts. Observability means the system emits enough rich, correlated data (metrics, logs, traces) that you can investigate failures you did not anticipate by asking new questions, without deploying new code.
• Q: Name the three pillars and when you use each.
  Metrics for detection, trends and SLOs (cheap, aggregated). Traces for locating the slow or failing hop in a distributed request. Logs for the detailed reason in one case. They are linked by shared identifiers: labels, trace ids.
• Q: Counter vs gauge vs histogram vs summary?
  A counter only goes up (use rate). A gauge goes up and down (read it directly). A histogram counts observations into cumulative buckets and can be aggregated across instances, with quantiles estimated by histogram_quantile. A summary computes quantiles in the client, cannot be aggregated across instances.
• Q: Why does Prometheus pull instead of push?
  Pull makes target health visible (up), lets you inspect any target with curl, centralises scrape config and avoids targets needing to know the server. Short-lived batch jobs that cannot be scraped can push to a Pushgateway.
• Q: Write a PromQL query for the 99th percentile latency across all pods.
  histogram_quantile(0.99, sum by (le) (rate(http_server_requests_seconds_bucket[5m]))), with histogram buckets enabled for that timer. Sum the buckets across pods first, then take the quantile; never average per-pod percentiles.
• Q: What is cardinality and why is it dangerous?
  The number of distinct time series (unique metric name plus label set). Cost in memory and index grows with it. A label like user_id or request_id creates a series per value and can take Prometheus down; keep such values in logs and traces.
• Q: RED vs USE vs golden signals?
  RED (rate, errors, duration) describes request-driven services. USE (utilisation, saturation, errors) describes resources. The four golden signals are latency, traffic, errors, saturation: RED plus saturation.
• Q: How do you expose Spring Boot metrics to Prometheus?
  Add spring-boot-starter-actuator and micrometer-registry-prometheus, include prometheus in management.endpoints.web.exposure.include (only health is exposed by default), and scrape /actuator/prometheus.
• Q: How do you correlate logs across services?
  Log JSON, put a correlation id in the MDC on every request, propagate it (preferably as the W3C trace id via Micrometer Tracing or OpenTelemetry) and include it in every log line, so a backend can show all lines of one request.
• Q: What is in a W3C traceparent header?
  Four dash-separated fields: version (00), 32-hex trace id, 16-hex parent span id, and trace flags (bit 0 is the sampled flag). Example: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01.
• Q: Head vs tail sampling?
  Head sampling decides at the start of a trace (cheap, but blind to outcome). Tail sampling decides after the trace completes in a collector, so it can keep all errors and slow traces, at the cost of buffering and routing all spans of a trace to one instance.
• Q: OTel Java agent or Spring Boot Micrometer Tracing?
  The agent needs no code changes and covers many libraries and any JVM app. The Boot route is native, unifies metrics, traces and log correlation through the Observation API. Do not use both on one service.
• Q: What is an error budget?
  One minus the SLO: the unreliability you may spend. A 99.9 percent SLO over 30 days allows 0.1 percent bad events (43.2 minutes of full outage). Budget left permits risky changes; budget spent shifts work to reliability.
• Q: Explain multi-window burn-rate alerting.
  Burn rate is observed error ratio divided by the allowed one. Page when a long window (for example 1 h at burn 14.4, 2 percent of a 30-day budget) and a short window (5 min) both exceed the threshold: the long one proves significance, the short one makes the alert reset quickly after the fix.
• Q: Symptom vs cause alerts?
  Symptom alerts describe user-visible impact and should page. Cause alerts (CPU, disk, restarts) are noisy; use them for dashboards and tickets, and page only when they predict an imminent symptom.
• Q: How do you reduce alert fatigue?
  Page only on SLO-tied symptoms, add for: durations, group and inhibit in Alertmanager, require a runbook for every alert, and review and delete alerts nobody acts on.
• Q: Loki vs Elasticsearch?
  Loki indexes only labels and stores log text cheaply (query scans text after label filtering); Elasticsearch indexes all content for fast full-text search and analytics at higher cost and operational effort.