Spring · 12

Actuator & Observability

How to know your Spring Boot service is working, not merely running. Actuator endpoints and how to expose and secure them, health groups with Kubernetes liveness and readiness probes, Micrometer counters, gauges and timers (and the cardinality trap), distributed tracing, structured logging with MDC, and the Prometheus and Grafana pipeline — plus a plain-Java metrics registry you can run.

Chapters

Diagrams

Pillars: health, metrics, traces, logs, control

Runnable demo

Spring Hub · 01 IoC & DI · 02 Bean Scopes & Lifecycle · 03 AOP & Proxies · 04 Boot Internals · 05 REST & MVC · 06 Validation & Errors · 07 Spring Data JPA · 08 Transactions · 09 Security · 10 JWT & OAuth2 · 11 Configuration & Profiles · 12 Actuator & Observability · 13 Caching & Resilience · 14 Testing · 15 Interview Q&A

Chapter One · Why It Exists

Running Is Not the Same as Working

A service that is running is not a service that is working. The process can be alive while the connection pool is exhausted, a downstream is timing out, or one customer's requests take eleven seconds. Without telemetry your only diagnostic tools are SSH, grep and guesswork, and the first signal of failure is a customer ticket.

Observability is the property that you can ask new questions about a running system from the outside, using the data it emits: metrics (numbers aggregated over time), traces (the path of one request across services) and logs (discrete events with context). Spring Boot's answer is Actuator (operational endpoints), Micrometer (a vendor-neutral metrics and observation API) and Micrometer Tracing (distributed trace context).

Before — the "monitoring" most teams start with

@RestController
class PingController {
  @GetMapping("/ping")
  String ping() { return "pong"; }          // always 200, even with the database down
}

@Service
class OrderService {
  Order create(OrderRequest req) {
      long start = System.currentTimeMillis();
      Order o = repo.save(map(req));
      System.out.println("created order in " + (System.currentTimeMillis() - start) + " ms");   // unqueryable, unaggregated
      return o;
  }
}

After — Actuator + Micrometer, almost no code

# pom.xml: spring-boot-starter-actuator + micrometer-registry-prometheus

@Service
class OrderService {
  private final Timer createTimer;
  OrderService(MeterRegistry registry) {
      this.createTimer = Timer.builder("orders.create").publishPercentileHistogram().register(registry);
  }
  Order create(OrderRequest req) {
      return createTimer.record(() -> repo.save(map(req)));   // count, sum, max and histogram buckets
  }
}
# GET /actuator/health            -> real dependency checks (DB, disk, ...)
# GET /actuator/prometheus        -> every metric, scrape-ready

Three signals, three different questions — use them in this order during an incident

Analogy: A car dashboard. The fuel and temperature gauges (metrics) tell you something is off long before the engine stops; the diagnostic port (traces) shows which subsystem; the black-box recorder (logs) explains the details afterwards. You would not drive without gauges, relying on the sound of the engine.

Monitoring vs observability Monitoring answers questions you anticipated ("is CPU > 90%?"). Observability means the telemetry is rich enough — tags, high-cardinality IDs in traces and logs, correlation — to answer questions you did not anticipate ("is it only EU customers on the new build calling the payment retry path?").

Chapter Two · Mental Model

Endpoints, Registries and Observations

An @Endpoint bean declares operations with @ReadOperation / @WriteOperation / @DeleteOperation

▶

The endpoint is technology-neutral. Boot adapts the same bean to HTTP (Spring MVC, WebFlux or Jersey) or JMX through WebMvcEndpointHandlerMapping and friends

Every endpoint passes two gates: access (is it enabled?) and exposure (is it published on this technology?)

▶

By default only health is exposed over HTTP. management.endpoints.web.exposure.include opens more; an endpoint that is enabled but not exposed simply returns 404

Actuator lives under a base path (/actuator) and optionally on a separate management port

▶

management.server.port starts a second, isolated child context, so the public port never routes to operational endpoints

Micrometer is the facade: code talks to MeterRegistry; a registry implementation per backend does the export

▶

Boot auto-configures one CompositeMeterRegistry plus a concrete registry for each backend on the classpath (micrometer-registry-prometheus, -datadog, ...); app code never changes when the backend does

MeterBinder beans register the built-in meters at startup

▶

JVM, system, Tomcat, HikariCP, Logback, executor and HTTP metrics appear without you writing any instrumentation

The Observation API (Boot 3) instruments once, observes many ways

▶

One Observation around a unit of work is turned by handlers into a timer metric and a trace span and log correlation — http.server.requests is produced this way

GET /actuator/health — from socket to JSON

Two consequences are worth internalising. First, because endpoints are normal request handlers, they are subject to your filters, your security configuration and your thread pool — a saturated Tomcat pool also stalls /actuator/health, which is why production setups put management on its own port. Second, because Micrometer is a facade, metric names are chosen once, in dotted form, and each backend maps them to its own convention (Prometheus: orders.create becomes orders_create_seconds_count).

Chapter Three · Actuator

Endpoints, Exposure and Securing Them

Add spring-boot-starter-actuator and decide, deliberately, which endpoints to publish. The safe baseline in production is: health and info for platforms, prometheus for the scraper, everything else only to authenticated operators on a private port.

Endpoint	Shows / does	Sensitivity	Notes
`health`	Aggregated status, optional component details	Low (details: medium)	Only one exposed over HTTP by default; probes at `/health/liveness`, `/health/readiness`
`info`	Build, git and custom info	Low	Needs `build-info` goal / git-commit-id plugin; env contributor needs `management.info.env.enabled=true`
`metrics`	Browse meters: `/metrics/jvm.memory.used?tag=area:heap`	Medium	Handy for ad-hoc checks without a TSDB
`prometheus`	All meters in Prometheus text format	Medium	Requires `micrometer-registry-prometheus`
`env` / `configprops`	Property sources / bound config beans	High	Values masked by default since Boot 3.0 (`show-values`)
`beans` / `mappings` / `conditions`	Bean graph, URL mappings, auto-config report	Medium	Superb for debugging wiring
`loggers`	Read and change log levels at runtime	High (write)	`POST` `{"configuredLevel":"DEBUG"}`
`threaddump` / `heapdump`	JVM thread dump / heap dump file	Very high	Heap dump contains secrets and PII; never expose publicly
`httpexchanges`	Last N HTTP exchanges	High	Replaces Boot 2 `httptrace`; needs an `HttpExchangeRepository` bean
`scheduledtasks` / `caches` / `flyway` / `liquibase`	Schedulers, cache managers, migration state	Low-medium	Cache endpoint can clear caches (write)
`shutdown`	Gracefully stops the app	Critical	Disabled by default; leave it so

A production baseline

management:
server:
  port: 8081                          # separate port: not routed by the public load balancer
endpoints:
  web:
    exposure:
      include: health,info,prometheus,loggers   # explicit allow-list; avoid "*"
    base-path: /actuator
endpoint:
  health:
    probes:
      enabled: true
    show-details: when-authorized
  env:
    show-values: NEVER
info:
  env:
    enabled: true
info:
app:
  name: orders
  team: payments-platform

Securing it with Spring Security 6 — ops-only, except health and info

@Configuration
class ActuatorSecurity {

  @Bean
  @Order(1)                                                   // before the application's own chain
  SecurityFilterChain actuatorChain(HttpSecurity http) throws Exception {
      http.securityMatcher(EndpointRequest.toAnyEndpoint())
          .authorizeHttpRequests(auth -> auth
              .requestMatchers(EndpointRequest.to("health", "info")).permitAll()
              .anyRequest().hasRole("OPS"))
          .httpBasic(Customizer.withDefaults())
          .csrf(csrf -> csrf.ignoringRequestMatchers(EndpointRequest.toAnyEndpoint()));   // POST /loggers etc.
      return http.build();
  }
}

Writing your own endpoint

@Component
@Endpoint(id = "featureflags")                        // exposure must include "featureflags"
class FeatureFlagsEndpoint {
  private final Map<String, Boolean> flags = new ConcurrentHashMap<>(Map.of("new-checkout", false));

  @ReadOperation                                    // GET  /actuator/featureflags
  public Map<String, Boolean> all() { return flags; }

  @ReadOperation                                    // GET  /actuator/featureflags/new-checkout
  public Boolean one(@Selector String name) { return flags.get(name); }

  @WriteOperation                                   // POST /actuator/featureflags/new-checkout  {"enabled":true}
  public void set(@Selector String name, boolean enabled) { flags.put(name, enabled); }
}

Endpoint access control is evolving: up to Boot 3.3 each endpoint has management.endpoint.<id>.enabled; from Boot 3.4 the finer management.endpoint.<id>.access (none, read-only, unrestricted) is available. Either way, exposure and access are independent switches.

⚠ Production gotchas & how to diagnose them:

include: "*" in production exposes env, heapdump, threaddump, loggers, beans. Real-world breaches started from an exposed /actuator/heapdump or /actuator/env. Use an allow-list and a separate port.
404 on /actuator/prometheus — one of three causes: the registry dependency is missing, the endpoint is not in exposure.include, or you are calling the main port while management.server.port moved it. Check /actuator (the discovery document) first.
401/403 from the kubelet or Prometheus after adding Spring Security: the default chain secures /actuator/**. Permit probe paths and give the scraper a token or network-level access.
Management port reachable from the internet because the Service or Ingress forwards every port. Expose 8081 only to the monitoring namespace (NetworkPolicy), never via the Ingress.
Custom endpoint not visible — the bean is fine, the id is simply not included in exposure.include. Endpoint ids must be lowercase alphanumeric.
Slow health endpoint blocks the probe thread: it runs every indicator synchronously on a request thread (see Chapter 04).

Chapter Four · Health

Health Indicators, Groups, Liveness and Readiness

/actuator/health is the most important and most misused endpoint. It answers two different questions for two different consumers, and mixing them causes outages: liveness ("is this process wedged so badly that only a restart helps?") and readiness ("should the load balancer send me traffic right now?"). Everything that is not one of those is a diagnostic health check for humans and dashboards.

The health document (details visible) — a composite of contributors

GET /actuator/health
{
"status": "DOWN",
"components": {
  "db":        { "status": "UP",   "details": { "database": "PostgreSQL", "validationQuery": "isValid()" } },
  "diskSpace": { "status": "UP",   "details": { "total": 107374182400, "free": 61203283968, "threshold": 10485760 } },
  "ping":      { "status": "UP" },
  "payment":   { "status": "DOWN", "details": { "error": "java.net.SocketTimeoutException: Read timed out" } }
}
}
// overall status = worst component (payment is DOWN) => HTTP 503

Status	Meaning	Default HTTP code	Aggregation order
`DOWN`	Component failed	503	1 (worst)
`OUT_OF_SERVICE`	Deliberately taken out (maintenance)	503	2
`UP`	Healthy	200	3
`UNKNOWN`	No information	200	4 (ignored by aggregate when anything else is present)

Custom HealthIndicator that does not hammer its dependency

@Component                                           // bean name paymentGatewayHealthIndicator -> component "paymentGateway"
class PaymentGatewayHealthIndicator implements HealthIndicator {

  private final RestClient client;
  private final AtomicReference<Health> last = new AtomicReference<>(Health.unknown().build());

  PaymentGatewayHealthIndicator(RestClient.Builder builder, PaymentProperties props) {
      this.client = builder.baseUrl(props.url()).build();
  }

  @Scheduled(fixedDelay = 10_000)                   // needs @EnableScheduling; probes in the background, not per kubelet request
  void refresh() {
      try {
          long t0 = System.nanoTime();
          client.head().uri("/status").retrieve().toBodilessEntity();
          long ms = (System.nanoTime() - t0) / 1_000_000;
          last.set(Health.up().withDetail("latencyMs", ms).build());
      } catch (RuntimeException e) {
          last.set(Health.down(e).build());
      }
  }

  @Override public Health health() { return last.get(); }       // O(1), never blocks
}

Health groups and Kubernetes probes

application.yml

management:
endpoint:
  health:
    probes:
      enabled: true                  # auto-enabled when running on Kubernetes
    show-details: when-authorized
    group:
      liveness:
        include: livenessState       # ONLY the internal state: no external dependencies
      readiness:
        include: readinessState,db   # can this instance serve traffic?
      diagnostics:                   # custom group for humans / dashboards
        include: db,diskSpace,paymentGateway
        show-details: always
server:
shutdown: graceful
spring:
lifecycle:
  timeout-per-shutdown-phase: 30s

Pod spec

livenessProbe:
httpGet: { path: /actuator/health/liveness, port: 8081 }
periodSeconds: 10
failureThreshold: 3                  # ~30s of failures before a restart
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: 8081 }
periodSeconds: 5
failureThreshold: 3
startupProbe:                           # protects slow starts from the liveness probe
httpGet: { path: /actuator/health/liveness, port: 8081 }
periodSeconds: 5
failureThreshold: 30                  # up to 150s to boot
terminationGracePeriodSeconds: 45       # > timeout-per-shutdown-phase

What the kubelet does with each probe (and how Spring flips them)

	Liveness	Readiness
Question	Is the process irrecoverably broken?	Can it serve a request right now?
Failure action	Kill and restart the container	Remove from load balancing; keep running
Should include	Internal state only (deadlock detector, fatal error flag)	Things the instance cannot work without: DB pool, required caches
Must not include	External dependencies — a DB blip would restart every pod at once	Optional/soft dependencies (recommendations service)
Typical bad config	Probing `/health` with the DB check	Failing readiness for a downstream that has a fallback

Marking the app broken from code

@Component
class FatalErrorHandler {
  private final ApplicationEventPublisher publisher;
  FatalErrorHandler(ApplicationEventPublisher publisher) { this.publisher = publisher; }

  void onUnrecoverable(Throwable t) {                 // e.g. out-of-sync internal state, poisoned cache
      log.error("unrecoverable state, requesting restart", t);
      AvailabilityChangeEvent.publish(publisher, this, LivenessState.BROKEN);
  }
}

⚠ Production gotchas & how to diagnose them:

Database in the liveness group → the DB restarts, every pod's liveness probe fails together, Kubernetes kills them all, they restart into a still-recovering database — a restart storm that turns a 30-second blip into a 10-minute outage. Symptom in kubectl describe pod: Liveness probe failed: HTTP probe failed with statuscode: 503 followed by Container ... failed liveness probe, will be restarted.
No startupProbe on a slow JVM: Spring Boot with Flyway and a cold cache takes 90 s; a liveness probe with failureThreshold: 3 kills it at 30 s → CrashLoopBackOff with no stack trace. Use startupProbe or initialDelaySeconds.
Health check that calls a remote service per probe. With 3 probes/10s × 50 pods you have generated 15 req/s of synthetic load against the dependency, and a slow dependency now makes /health itself slow (Readiness probe failed: Get ...: context deadline exceeded). Cache the result (as above) or use a short timeout.
show-details: always on a public port leaks hostnames, versions and error messages. Use when-authorized, or reveal details only on a custom diagnostics group.
"Always UP" /ping as readiness: traffic arrives while the instance still cannot reach its database — 500s at every deploy. Use the readiness group with real dependencies.
Probes on the main port with server.tomcat.threads.max exhausted: the probe waits in the queue and fails, causing needless restarts under load. A separate management port has its own connector and pool.
Forgetting the grace period: terminationGracePeriodSeconds (default 30) shorter than your drain time kills in-flight requests with SIGKILL. Keep it larger than spring.lifecycle.timeout-per-shutdown-phase.

Chapter Five · Metrics

Micrometer: Counters, Gauges, Timers and Tags

Micrometer is to metrics what SLF4J is to logging: you instrument against io.micrometer.core.instrument.* and a registry maps the meters onto a backend. A meter is identified by name + set of tags; each distinct combination of tag values is a separate time series. That sentence contains both the power and the main danger of metrics.

The meter types and what their samples look like over time

Instrumenting a service by hand

@Service
public class OrderService {

  private final OrderRepository repo;
  private final Counter accepted;
  private final Counter rejected;
  private final Timer createTimer;
  private final DistributionSummary basketSize;

  public OrderService(OrderRepository repo, MeterRegistry registry) {
      this.repo = repo;
      this.accepted = Counter.builder("orders.created").description("Orders accepted")
              .tag("channel", "web").register(registry);                 // tag VALUES must be a small, closed set
      this.rejected = Counter.builder("orders.rejected").register(registry);
      this.createTimer = Timer.builder("orders.create")
              .description("Time to validate and persist an order")
              .publishPercentileHistogram()                              // exports buckets: aggregatable p95 in Prometheus
              .register(registry);
      this.basketSize = DistributionSummary.builder("orders.basket.items").register(registry);
      Gauge.builder("orders.pending", repo, OrderRepository::countPending)   // sampled when scraped
              .description("Orders awaiting fulfilment").register(registry);
  }

  public Order create(OrderRequest req) {
      return createTimer.record(() -> {
          if (req.items().isEmpty()) { rejected.increment(); throw new IllegalArgumentException("empty basket"); }
          Order saved = repo.save(Order.from(req));
          accepted.increment();
          basketSize.record(req.items().size());
          return saved;
      });
  }
}

Auto-configured meters (no code)	Names	What to watch
JVM memory / GC / threads	`jvm.memory.used`, `jvm.gc.pause`, `jvm.threads.live`	Heap after GC trending up = leak; long GC pauses
CPU / uptime	`process.cpu.usage`, `system.cpu.usage`, `process.uptime`	Saturation
HTTP server	`http.server.requests` (tags `method uri status outcome exception`)	RED: rate, errors, duration per route
HTTP client	`http.client.requests` (RestClient / RestTemplate / WebClient built from Boot builders)	Downstream latency and errors
Connection pool	`hikaricp.connections.active / pending / timeout / max`	Pending > 0 = threads waiting for a DB connection
Tomcat	`tomcat.threads.busy`, `tomcat.sessions.*` (needs `server.tomcat.mbeanregistry.enabled=true`)	Busy threads at max = saturated
Logback	`logback.events` by level	ERROR rate as a cheap alert
Executors / caches	`executor.*`, `cache.gets` (hit/miss)	Queue growth; hit ratio

Distribution and common tags

management:
metrics:
  tags:
    application: ${spring.application.name}       # added to every meter; join/filter in Grafana
    region: eu-west-1
  distribution:
    percentiles-histogram:
      http.server.requests: true                    # buckets for histogram_quantile() (server-side, aggregatable)
    slo:
      http.server.requests: 50ms,100ms,250ms,500ms,1s   # cheap fixed buckets matching your SLO
    percentiles:
      orders.create: 0.5,0.95,0.99                  # client-side quantiles: NOT mergeable across instances

The Observation API (Boot 3)

One instrumentation, metrics + traces + logs

@Service
class ShippingService {
  private final ObservationRegistry observations;
  ShippingService(ObservationRegistry observations) { this.observations = observations; }

  Label label(Order o) {
      return Observation.createNotStarted("shipping.label", observations)
              .lowCardinalityKeyValue("carrier", o.carrier())        // becomes a metric tag AND a span tag
              .highCardinalityKeyValue("orderId", o.id().toString()) // span/log only; never a metric tag
              .observe(() -> carrierApi.createLabel(o));             // timer + span, error recorded on exception
  }
}
// Declarative variant: @Observed(name = "shipping.label") on a bean method,
// requires an ObservedAspect bean and spring-boot-starter-aop (or aspectj) on the classpath.

⚠ Production gotchas & how to diagnose them:

High-cardinality tags. .tag("userId", id) or a raw URL as a tag creates one time series per value; Prometheus memory and your bill explode and dashboards time out. Symptoms: scrape size in MBs, prometheus_tsdb_head_series climbing, scrape timeouts. Tag with bounded sets (channel, outcome, status class); put IDs in traces and logs. Guard with a MeterFilter.maximumAllowableTags(...).
Gauge disappears (NaN). A gauge keeps only a weak reference to its state object; registry.gauge("n", new AtomicInteger()) or a boxed Integer is garbage-collected and the series goes NaN. Hold a strong reference to the object yourself.
Using a counter for something that goes down (queue length, active sessions) — use a gauge. Using a gauge for something that only accumulates (requests) — you lose rate() semantics and resets break.
Averages hide tails. _sum / _count over a window of 6 requests at 4, 8, 35, 42, 120, 480 ms gives 115 ms while one customer waited 480 ms. Alert and plot percentiles from histogram buckets.
Client-side percentiles averaged across pods are statistically meaningless. Publish histogram buckets, sum them across instances, then compute histogram_quantile.
@Timed / @Counted doing nothing on a service method: outside controllers they need a TimedAspect / CountedAspect bean and AOP on the classpath, and they are subject to the proxy self-invocation limitation (AOP & Proxies).
new RestTemplate() / WebClient.create() bypass Boot's instrumentation: no http.client.requests, no trace propagation. Inject RestClient.Builder, RestTemplateBuilder or WebClient.Builder.

Chapter Six · Traces

Distributed Tracing with Micrometer Tracing

A metric tells you that p95 latency doubled. A trace tells you where: it follows one request through every service as a tree of spans, each with a start, a duration, tags and a parent. All spans of one request share a traceId; each span has its own spanId and points to its parentId.

A request arrives with (or without) a W3C traceparent header: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

▶

Without one, a new trace starts; with one, the server span joins that trace as a child of 00f067aa0ba902b7

The tracing bridge (micrometer-tracing-bridge-otel or -brave) puts traceId/spanId into the logging MDC for the current thread

▶

Every log line written during the request automatically carries the IDs — this is how you jump from a trace to its logs

Instrumented HTTP clients (built from Boot's builders) add traceparent to outgoing requests

▶

The next service continues the same trace. A client built with new RestTemplate() breaks the chain: two unrelated traces appear

Only a sampled fraction of traces is exported (management.tracing.sampling.probability, default 0.1)

▶

Tracing everything is expensive. 10% is typical; use 1.0 in dev, and tail-based sampling in the collector if you need all errors

An exporter (OTLP over HTTP or gRPC) ships finished spans to a collector, then to Tempo, Jaeger, Zipkin or a vendor

▶

Export is asynchronous and batched; a dead collector must not take your service down (it drops spans)

Dependencies and configuration

<!-- pom.xml -->
<dependency><groupId>io.micrometer</groupId><artifactId>micrometer-tracing-bridge-otel</artifactId></dependency>
<dependency><groupId>io.opentelemetry</groupId><artifactId>opentelemetry-exporter-otlp</artifactId></dependency>

application.yml

management:
tracing:
  sampling:
    probability: 1.0                 # dev. Default 0.1 -> 1 in 10 requests is traced
otlp:
  tracing:
    endpoint: http://otel-collector:4318/v1/traces
spring:
application:
  name: orders                        # becomes the service name on every span

A log line during a traced request (Boot 3.2+ correlation, abridged)

2026-10-29T10:15:04.120Z  INFO 1 --- [orders] [nio-8080-exec-3] [4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7] c.e.orders.OrderService : order 981 accepted

One trace, five spans: the waterfall that answers "where did the 480 ms go?"

Adding your own span (rarely needed: prefer Observation)

@Service
class FraudChecker {
  private final Tracer tracer;                       // io.micrometer.tracing.Tracer
  FraudChecker(Tracer tracer) { this.tracer = tracer; }

  boolean isSuspicious(Order o) {
      Span span = tracer.nextSpan().name("fraud-check").tag("order.value.band", band(o)).start();
      try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
          return rules.evaluate(o);
      } catch (RuntimeException e) {
          span.error(e);
          throw e;
      } finally {
          span.end();                                // forgetting end() leaks the span: it is never exported
      }
  }
}

⚠ Production gotchas & how to diagnose them:

Broken trace across a thread hop. Context is thread-local; @Async, executors and reactive schedulers need context propagation (a TaskDecorator or Micrometer's context-propagation integration) or the child work starts a new trace. Symptom: spans with no parent, logs with traceId empty after an async call.
Sampling surprise. You look for a failed request's trace and it is missing because only 10% were sampled. For debugging keep 1.0 in lower environments; in prod rely on exemplars or tail sampling in the OpenTelemetry Collector.
Tracing dependency present, nothing exported. A tracer bridge without an exporter/endpoint creates spans (IDs in logs) but ships nothing — check management.otlp.tracing.endpoint and collector connectivity.
Sensitive data in span tags (emails, tokens, full SQL with literals). Tags are stored and visible to everyone with tracing access. Use identifiers, not payloads.
Health/probe endpoints flooding traces. Kubelet polls every few seconds; register an ObservationPredicate bean that returns false for requests whose path starts with /actuator, otherwise they dominate sampled traces.

Chapter Seven · Logs

Structured Logging, MDC and Correlation

Logs are the highest-fidelity and most expensive signal. Make them queryable rather than merely readable: one event per line, stable key names, a correlation ID on every line, and parameterised messages instead of concatenation. Boot uses Logback through SLF4J by default; logging.level.* controls verbosity per package.

Levels and groups

logging:
level:
  root: INFO
  com.example.orders: INFO
  org.hibernate.SQL: DEBUG                  # print SQL (no bind values)
  org.hibernate.orm.jdbc.bind: TRACE        # bind parameters (Hibernate 6)
group:
  sql: org.hibernate.SQL,org.hibernate.orm.jdbc.bind

Change a level at runtime — no restart, no redeploy

$ curl -s -X POST localhost:8081/actuator/loggers/com.example.orders.PaymentClient \
     -H 'Content-Type: application/json' -d '{"configuredLevel":"DEBUG"}'
$ curl -s localhost:8081/actuator/loggers/com.example.orders.PaymentClient
{"configuredLevel":"DEBUG","effectiveLevel":"DEBUG"}
# ...reproduce the issue, then restore:
$ curl -s -X POST localhost:8081/actuator/loggers/com.example.orders.PaymentClient -H 'Content-Type: application/json' -d '{"configuredLevel":null}'

Structured logs (Boot 3.4+): one JSON object per line, ready for Loki / Elasticsearch

logging:
structured:
  format:
    console: ecs                 # also: logstash, gelf (or a custom StructuredLogFormatter)

Resulting line (ECS, abridged)

{"@timestamp":"2026-10-29T10:15:04.120Z","log.level":"INFO","process.thread.name":"nio-8080-exec-3","service.name":"orders","log.logger":"com.example.orders.OrderService","message":"order accepted","orderId":"981","traceId":"4bf92f3577b34da6a3ce929d0e0e4736","ecs.version":"8.11"}

MDC — context that follows the thread

The Mapped Diagnostic Context is a thread-local map that Logback merges into every event logged on that thread. Tracing writes traceId/spanId into it automatically; add your own business context (requestId, tenant, userId) in a servlet filter. Because it is thread-local, it must be cleaned up at the end of the request and copied explicitly across thread hops.

Request-scoped MDC filter

@Component
@Order(Ordered.HIGHEST_PRECEDENCE + 10)
class RequestContextFilter extends OncePerRequestFilter {

  @Override
  protected void doFilterInternal(HttpServletRequest req, HttpServletResponse res, FilterChain chain)
          throws ServletException, IOException {
      String requestId = Optional.ofNullable(req.getHeader("X-Request-Id"))
              .filter(s -> !s.isBlank()).orElseGet(() -> UUID.randomUUID().toString());
      MDC.put("requestId", requestId);
      res.setHeader("X-Request-Id", requestId);          // lets callers quote it in a support ticket
      try {
          chain.doFilter(req, res);
      } finally {
          MDC.remove("requestId");                        // pooled threads are reused: always clean up
      }
  }
}

Propagating MDC to @Async / executor threads

@Bean
TaskDecorator mdcTaskDecorator() {                          // Boot applies a TaskDecorator bean to its applicationTaskExecutor
  return runnable -> {
      Map<String, String> captured = MDC.getCopyOfContextMap();   // captured on the submitting thread
      return () -> {
          Map<String, String> previous = MDC.getCopyOfContextMap();
          if (captured != null) MDC.setContextMap(captured); else MDC.clear();
          try { runnable.run(); }
          finally { if (previous != null) MDC.setContextMap(previous); else MDC.clear(); }
      };
  };
}

Level	Use for	Not for
`ERROR`	Operation failed and a human or retry must act; alert on rate	Validation failures caused by bad client input
`WARN`	Recovered anomaly: retry succeeded, fallback used, deprecated call	Normal business outcomes
`INFO`	Business milestones (order accepted), startup/shutdown, config summary	Per-item loops, payloads
`DEBUG` / `TRACE`	Developer detail; enable per package at runtime via `/loggers`	Always-on production default

⚠ Production gotchas & how to diagnose them:

MDC lost in a thread pool (see the real output in Chapter 09): the async task logs traceId=-. Add the TaskDecorator above, or use Micrometer's context propagation.
MDC leak. Without finally-cleanup, a pooled thread keeps the previous request's userId; the next request's logs name the wrong user. Symptom: one customer's ID in another customer's log lines.
String concatenation in log calls (log.debug("x=" + expensive())) builds the string even when DEBUG is off. Use log.debug("x={}", x), and a guard for expensive arguments.
Logging the same exception at every layer → one failure produces ten stack traces and inflates ERROR counts. Log once, at the boundary that handles it (usually @ControllerAdvice); lower layers wrap and rethrow.
PII and secrets in logs (tokens, card numbers, full request bodies). Log IDs; mask at the source; remember log retention is usually longer than DB retention.
DEBUG left on after an incident. A forgotten POST /loggers keeps verbosity until restart; the disk and the log bill fill up. Reset with configuredLevel: null when done.

Chapter Eight · The Pipeline

From /actuator/prometheus to a Grafana Alert

Prometheus is a pull-based time-series database: every 15 seconds it HTTP-GETs /actuator/prometheus from each instance, stores the samples with their labels, and answers queries (PromQL). Grafana draws dashboards from those queries; Alertmanager routes alerts. Pull has a useful side effect: if a scrape fails, the synthetic series up becomes 0 — you detect a dead instance without it having to report anything.

The metrics pipeline end to end

What Prometheus scrapes (excerpt of GET /actuator/prometheus)

# HELP http_server_requests_seconds Duration of HTTP server request handling
# TYPE http_server_requests_seconds histogram
http_server_requests_seconds_bucket{application="orders",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}",le="0.1"} 812
http_server_requests_seconds_bucket{application="orders",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}",le="0.5"} 987
http_server_requests_seconds_bucket{application="orders",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}",le="+Inf"} 1000
http_server_requests_seconds_count{application="orders",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}"} 1000
http_server_requests_seconds_sum{application="orders",method="GET",outcome="SUCCESS",status="200",uri="/orders/{id}"} 54.2
# TYPE hikaricp_connections_active gauge
hikaricp_connections_active{application="orders",pool="HikariPool-1"} 7.0

prometheus.yml and a ServiceMonitor (Prometheus Operator)

scrape_configs:
- job_name: orders
  metrics_path: /actuator/prometheus
  scrape_interval: 15s
  static_configs:
    - targets: ["orders-1:8081", "orders-2:8081"]
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata: { name: orders }
spec:
selector: { matchLabels: { app: orders } }
endpoints:
  - port: management                 # named port 8081 on the Service
    path: /actuator/prometheus
    interval: 15s

Question (RED / USE)	PromQL	Reads as
Request rate per route	`sum by (uri) (rate(http_server_requests_seconds_count{application="orders"}[5m]))`	requests per second
Error ratio	`sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m])) / sum(rate(http_server_requests_seconds_count[5m]))`	0.02 = 2% of requests fail
p95 latency per route	`histogram_quantile(0.95, sum by (le, uri) (rate(http_server_requests_seconds_bucket[5m])))`	seconds; sum before quantile
DB pool saturation	`hikaricp_connections_active / hikaricp_connections_max`	1.0 = pool exhausted
Heap pressure	`jvm_memory_used_bytes{area="heap"} / jvm_memory_max_bytes{area="heap"}`	sustained > 0.9 = trouble
Dead instance	`up{job="orders"} == 0`	scrape failed

An alert rule that avoids flapping

groups:
- name: orders
  rules:
    - alert: OrdersHighErrorRate
      expr: |
        sum(rate(http_server_requests_seconds_count{application="orders",status=~"5.."}[5m]))
          / sum(rate(http_server_requests_seconds_count{application="orders"}[5m])) > 0.05
      for: 10m                                  # must hold for 10 minutes before firing
      labels: { severity: page }
      annotations:
        summary: "orders 5xx ratio above 5% for 10m"
        runbook: "https://wiki.example.com/runbooks/orders-5xx"

⚠ Production gotchas & how to diagnose them:

Averaging quantiles / summing percentiles. avg(http_server_requests_seconds{quantile="0.95"}) is not the fleet p95. Export buckets and use histogram_quantile over sum by (le).
rate() window shorter than 4× the scrape interval (e.g. [30s] at a 15 s scrape) yields gaps and noisy graphs. Use at least [1m], typically [5m].
Counters reset on restart; always wrap them in rate()/increase(), never graph the raw value or subtract it yourself.
Alerting on a cause, not a symptom (CPU > 80%) wakes people for non-incidents. Alert on user-visible symptoms (error ratio, latency SLO burn) and keep causes on dashboards.
Scrape timing out because of a huge label set (see cardinality). Symptoms: scrape_duration_seconds near the timeout, target flapping up=0. Fix the tags, not the timeout.
The scraper hits the main port and Spring Security answers 401. Give Prometheus a separate port or a bearer token; do not open the endpoint to the internet instead.

Chapter Nine · Build It Yourself

A Mini Metrics Registry, Health Aggregator and MDC in Plain Java

Reading Micrometer's source is harder than rebuilding the idea. The program below is plain Java 17 with no dependencies. It implements a tagged meter registry (counter, gauge, timer with cumulative histogram buckets), renders it in Prometheus text format, aggregates health statuses with Actuator's worst-wins ordering, demonstrates a cardinality explosion, and shows MDC being lost across a thread pool and fixed by capture-and-restore.

ObsDemo.java

import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.*;
import java.util.function.Supplier;

public class ObsDemo {

  // ---------- tiny Micrometer-style registry ----------
  record Tag(String key, String value) {}

  static String id(String name, Tag... tags) {
      if (tags.length == 0) return name;
      StringJoiner sj = new StringJoiner(",", name + "{", "}");
      Arrays.stream(tags).sorted(Comparator.comparing(Tag::key)).forEach(t -> sj.add(t.key() + "=\"" + t.value() + "\""));
      return sj.toString();
  }

  static class Counter { final DoubleAdder v = new DoubleAdder(); void increment() { v.add(1); } double count() { return v.sum(); } }
  static class Gauge { final Supplier<Double> src; Gauge(Supplier<Double> s) { src = s; } double value() { return src.get(); } }
  static class Timer {
      static final double[] SLO_MS = {10, 50, 100, 500};
      final LongAdder count = new LongAdder(); final LongAdder totalNs = new LongAdder();
      final AtomicLong maxNs = new AtomicLong(); final LongAdder[] buckets = new LongAdder[SLO_MS.length];
      Timer() { for (int i = 0; i < buckets.length; i++) buckets[i] = new LongAdder(); }
      void record(long ns) {
          count.increment(); totalNs.add(ns); maxNs.accumulateAndGet(ns, Math::max);
          for (int i = 0; i < SLO_MS.length; i++) if (ns <= SLO_MS[i] * 1_000_000) buckets[i].increment(); // cumulative
      }
  }

  static class Registry {
      final Map<String, Counter> counters = new TreeMap<>();
      final Map<String, Gauge> gauges = new TreeMap<>();
      final Map<String, Timer> timers = new TreeMap<>();
      Counter counter(String n, Tag... t) { return counters.computeIfAbsent(id(n, t), k -> new Counter()); }
      Timer timer(String n, Tag... t) { return timers.computeIfAbsent(id(n, t), k -> new Timer()); }
      void gauge(String n, Supplier<Double> s, Tag... t) { gauges.put(id(n, t), new Gauge(s)); }
      int meterCount() { return counters.size() + gauges.size() + timers.size(); }

      String prometheus() {
          StringBuilder sb = new StringBuilder();
          counters.forEach((k, c) -> sb.append(withSuffix(k, "_total")).append(' ').append(c.count()).append('\n'));
          gauges.forEach((k, g) -> sb.append(withSuffix(k, "")).append(' ').append(g.value()).append('\n'));
          timers.forEach((k, t) -> {
              sb.append(withSuffix(k, "_seconds_count")).append(' ').append(t.count.sum()).append('\n');
              sb.append(withSuffix(k, "_seconds_sum")).append(' ').append(t.totalNs.sum() / 1e9).append('\n');
              sb.append(withSuffix(k, "_seconds_max")).append(' ').append(t.maxNs.get() / 1e9).append('\n');
              for (int i = 0; i < Timer.SLO_MS.length; i++)
                  sb.append(withSuffix(k, "_seconds_bucket", "le=\"" + Timer.SLO_MS[i] / 1000 + "\"")).append(' ').append(t.buckets[i].sum()).append('\n');
              sb.append(withSuffix(k, "_seconds_bucket", "le=\"+Inf\"")).append(' ').append(t.count.sum()).append('\n');
          });
          return sb.toString();
      }
      static String withSuffix(String id, String suffix) { return withSuffix(id, suffix, null); }
      static String withSuffix(String id, String suffix, String extraLabel) {
          int b = id.indexOf('{');
          String name = (b < 0 ? id : id.substring(0, b)).replace('.', '_') + suffix;
          String labels = b < 0 ? "" : id.substring(b + 1, id.length() - 1);
          if (extraLabel != null) labels = labels.isEmpty() ? extraLabel : labels + "," + extraLabel;
          return labels.isEmpty() ? name : name + "{" + labels + "}";
      }
  }

  // ---------- health aggregation ----------
  enum Status { DOWN, OUT_OF_SERVICE, UP, UNKNOWN }  // declaration order = severity order (worst first)

  record Health(Status status, Map<String, Object> details) {}

  static Status aggregate(Collection<Status> parts) {
      return parts.stream().min(Comparator.comparingInt(Enum::ordinal)).orElse(Status.UNKNOWN);
  }

  // ---------- MDC ----------
  static final ThreadLocal<Map<String, String>> MDC = ThreadLocal.withInitial(HashMap::new);
  static void log(String level, String msg) {
      Map<String, String> m = MDC.get();
      System.out.printf("%s [traceId=%s userId=%s] %s%n", level, m.getOrDefault("traceId", "-"), m.getOrDefault("userId", "-"), msg);
  }

  public static void main(String[] args) throws Exception {
      Registry reg = new Registry();

      // counters with tags (low cardinality)
      for (String[] r : new String[][]{{"GET", "200"}, {"GET", "200"}, {"GET", "404"}, {"POST", "201"}, {"GET", "500"}, {"GET", "200"}})
          reg.counter("http.server.requests", new Tag("method", r[0]), new Tag("status", r[1])).increment();

      // gauge reads live state
      BlockingQueue<String> queue = new LinkedBlockingQueue<>(List.of("a", "b", "c"));
      reg.gauge("orders.queue.size", () -> (double) queue.size());
      queue.poll();

      // timer: record 6 latencies in ms
      Timer t = reg.timer("orders.create", new Tag("outcome", "success"));
      for (long ms : new long[]{4, 8, 35, 42, 120, 480}) t.record(ms * 1_000_000);

      System.out.println("---- GET /actuator/prometheus ----");
      System.out.print(reg.prometheus());

      double avgMs = t.totalNs.sum() / 1e6 / t.count.sum();
      System.out.printf("%navg latency = %.2f ms, max = %.0f ms (avg hides the 480 ms tail)%n", avgMs, t.maxNs.get() / 1e6);

      // cardinality explosion
      int before = reg.meterCount();
      for (int user = 0; user < 10_000; user++) reg.counter("orders.created", new Tag("userId", "u" + user)).increment();
      System.out.println("\nmeters before = " + before + ", after tagging by userId = " + reg.meterCount());

      // health aggregation
      System.out.println("\n---- health ----");
      Map<String, Status> comps = new LinkedHashMap<>();
      comps.put("db", Status.UP); comps.put("diskSpace", Status.UP); comps.put("redis", Status.UP);
      System.out.println("all UP            -> " + aggregate(comps.values()));
      comps.put("redis", Status.DOWN);
      System.out.println("redis DOWN        -> " + aggregate(comps.values()));
      comps.put("redis", Status.OUT_OF_SERVICE);
      System.out.println("redis OOS         -> " + aggregate(comps.values()));
      comps.put("redis", Status.UNKNOWN);
      System.out.println("redis UNKNOWN     -> " + aggregate(comps.values()));

      // MDC across threads
      System.out.println("\n---- MDC ----");
      MDC.get().put("traceId", "4bf92f3577b34da6"); MDC.get().put("userId", "42");
      log("INFO", "request accepted");
      ExecutorService pool = Executors.newSingleThreadExecutor();
      pool.submit(() -> log("INFO", "in pool thread - MDC lost")).get();
      Map<String, String> snapshot = new HashMap<>(MDC.get());
      pool.submit(() -> { MDC.get().putAll(snapshot); try { log("INFO", "in pool thread - MDC propagated"); } finally { MDC.get().clear(); } }).get();
      pool.shutdown();
  }
}

$ javac ObsDemo.java && java ObsDemo (real output)

---- GET /actuator/prometheus ----
http_server_requests_total{method="GET",status="200"} 3.0
http_server_requests_total{method="GET",status="404"} 1.0
http_server_requests_total{method="GET",status="500"} 1.0
http_server_requests_total{method="POST",status="201"} 1.0
orders_queue_size 2.0
orders_create_seconds_count{outcome="success"} 6
orders_create_seconds_sum{outcome="success"} 0.689
orders_create_seconds_max{outcome="success"} 0.48
orders_create_seconds_bucket{outcome="success",le="0.01"} 2
orders_create_seconds_bucket{outcome="success",le="0.05"} 4
orders_create_seconds_bucket{outcome="success",le="0.1"} 4
orders_create_seconds_bucket{outcome="success",le="0.5"} 6
orders_create_seconds_bucket{outcome="success",le="+Inf"} 6

avg latency = 114.83 ms, max = 480 ms (avg hides the 480 ms tail)

meters before = 6, after tagging by userId = 10006

---- health ----
all UP            -> UP
redis DOWN        -> DOWN
redis OOS         -> OUT_OF_SERVICE
redis UNKNOWN     -> UP

---- MDC ----
INFO [traceId=4bf92f3577b34da6 userId=42] request accepted
INFO [traceId=- userId=-] in pool thread - MDC lost
INFO [traceId=4bf92f3577b34da6 userId=42] in pool thread - MDC propagated

Output	What it proves
`http_server_requests_total{method="GET",status="200"} 3.0`	A meter is name + tag set: GET/200, GET/404, GET/500 and POST/201 are four separate series of one logical metric.
`orders_create_seconds_bucket{...le="0.05"} 4`	Histogram buckets are cumulative: 4 of 6 requests took ≤ 50 ms, 6 took ≤ 500 ms, and `+Inf` always equals `_count`. That is what lets Prometheus merge instances and compute quantiles later.
`avg latency = 114.83 ms, max = 480 ms`	The average looks tolerable while one request took 480 ms — why you alert on percentiles, not means.
`meters before = 6, after tagging by userId = 10006`	One innocent-looking tag turned 6 time series into 10,006. Each is memory in the app and in Prometheus.
`redis DOWN -> DOWN`, `OUT_OF_SERVICE`, `UNKNOWN -> UP`	Aggregation is simply the minimum by severity ordinal; an UNKNOWN component does not drag the whole service down.
`in pool thread - MDC lost` then `propagated`	ThreadLocal context does not cross threads; capturing on submit and restoring in the task is exactly what a `TaskDecorator` does.

Where the real Micrometer goes further

Meter IDs are immutable Meter.Id objects with unit and description; registries de-duplicate registrations.
Clocks and step-based rate aggregation per backend (push registries compute rates; Prometheus leaves it to the server).
MeterFilters can rename, add or deny tags and cap cardinality before a meter is created.
Gauges hold weak references so metrics never prevent garbage collection (the origin of the NaN gotcha).

Chapter Ten · In Production

Reading the Signals: Symptom to Cause

Instrumentation is only valuable if you know what to read when something breaks. This is the triage table to keep next to the dashboards: symptom → first signal → confirm with → typical cause.

Symptom	First signal (metric)	Confirm with	Typical cause / fix
p95 latency up, CPU normal	`hikaricp_connections_pending > 0`, `hikaricp_connections_active` at max	Trace waterfall: spans waiting before DB calls; `/actuator/threaddump` shows threads in `HikariPool.getConnection`	Pool too small or a slow query / long transaction holding connections; fix the query, then size the pool
5xx spike after a deploy	`http_server_requests_seconds_count{status=~"5.."}` by `uri`, `exception` tag	Logs filtered by one traceId from an exemplar/trace; `/actuator/loggers` to raise a package to DEBUG	Bad config (see Configuration), missing dependency, schema mismatch
Pods restart repeatedly	`kube_pod_container_status_restarts_total`, `up` flapping	`kubectl describe pod` events: `Liveness probe failed` vs `OOMKilled` (exit 137)	Liveness includes an external dependency; or heap larger than the container limit
Pod Running but gets no traffic	Endpoints count for the Service is 0	`curl :8081/actuator/health/readiness` → `OUT_OF_SERVICE`/503	Required dependency DOWN; readiness group misconfigured; app still draining
Memory climbs, never drops	`jvm_memory_used_bytes{area="heap"}` measured after GC trending up	`/actuator/heapdump` (protected!) opened in a profiler; `jvm_gc_pause_seconds` growing	Unbounded cache or collection; listener/ThreadLocal leak
Requests hang, CPU idle	`tomcat_threads_busy` = max, request queue growing	`/actuator/threaddump`: many threads `BLOCKED` / `WAITING` on the same lock or socket read	Slow downstream with no timeout; synchronized hot spot; need timeouts + bulkheads (Resilience)
One endpoint slow for few users	p99 per `uri` high, p50 normal	Traces filtered by `uri` and duration > 2 s; high-cardinality tags in spans (tenant, orderId)	Data skew (a huge tenant), missing index, N+1 (Spring Data JPA)
Prometheus shows gaps	`up == 0`, `scrape_duration_seconds` high	`curl -w %{time_total} :8081/actuator/prometheus`; `count({job="orders"})` series count	Cardinality explosion, or the scrape hits a saturated main port

A first-five-minutes checklist from a shell

$ curl -s :8081/actuator/health | jq .status                       # is it up at all?
$ curl -s :8081/actuator/health/readiness | jq .status             # does the LB think so?
$ curl -s ':8081/actuator/metrics/hikaricp.connections.pending' | jq '.measurements'
$ curl -s ':8081/actuator/metrics/http.server.requests?tag=outcome:SERVER_ERROR' | jq '.measurements'
$ curl -s :8081/actuator/threaddump | jq '.threads | map(select(.threadState=="BLOCKED")) | length'
$ kubectl describe pod orders-7d9f | grep -A5 Events

Chapter Eleven · Interview Angles

Questions You Will Be Asked

Short answers first; trap is the follow-up an experienced interviewer uses to check you have run this in production.

What is Actuator, and how is it different from Micrometer?

Actuator exposes operational endpoints (health, metrics, loggers...). Micrometer is the metrics/observation library behind the metrics endpoints. Trap: Actuator without a registry dependency has no Prometheus endpoint.

Liveness vs readiness: what goes in each?

Liveness: internal, unrecoverable state only; failure restarts the container. Readiness: ability to serve now, including hard dependencies; failure removes from load balancing. Trap: DB check in liveness causes restart storms.

How does overall health status get computed?

A StatusAggregator picks the worst component status (DOWN > OUT_OF_SERVICE > UP > UNKNOWN); HttpCodeStatusMapper maps DOWN/OUT_OF_SERVICE to 503. Trap: UNKNOWN maps to 200.

Counter vs Gauge vs Timer?

Counter only increases (use rate()); gauge is a sampled current value; timer records count, total time, max and optional histogram. Trap: gauges hold weak references and can go NaN.

What is metric cardinality and why does it matter?

The number of distinct tag combinations per metric = number of time series. Unbounded values (userId, raw URL) explode memory and cost. Trap: put high-cardinality data in traces/logs, not metric tags.

Why histograms instead of client-side percentiles?

Histogram buckets can be summed across instances, then quantiles computed server-side; pre-computed quantiles cannot be averaged. Trap: avg(p95) across pods is wrong.

How does tracing propagate across services?

W3C traceparent header carries traceId/parentSpanId; the instrumented client adds it, the server span continues it. Trap: clients not built from Boot builders, and thread hops, break the chain.

How do you correlate logs with traces?

The tracing bridge puts traceId/spanId in the MDC and the log pattern prints them; filter logs by traceId. Trap: MDC is thread-local; async work needs propagation.

How would you secure Actuator?

Allow-list exposure, separate management port, Spring Security chain with EndpointRequest, mask env/configprops values, never expose heapdump. Trap: include: "*".

Q10

Why is Prometheus pull-based, and what breaks that model?

Pull makes target health visible (up) and keeps instances stateless about the backend. Short-lived jobs vanish before a scrape — push (Pushgateway/OTLP) instead.

Chapter Twelve · Review

Summary & Cheat Sheet

Signal	Tool in Spring Boot	Question it answers	Danger
Health	Actuator `/health`, groups, probes	Should I restart / route to this pod?	External deps in liveness
Metrics	Micrometer + `/prometheus`	Is it wrong? How bad? Trend?	High-cardinality tags
Traces	Micrometer Tracing + OTLP	Where is the time / failure?	Sampling, broken propagation
Logs	Logback, MDC, structured JSON	Why did this request fail?	PII, MDC leaks, duplicate stack traces
Control	`/loggers`, custom `@Endpoint`	Change behaviour without redeploy	Unauthenticated write endpoints

∑ Actuator & Observability — what to remember

Expose deliberately: allow-list endpoints, separate management port, mask values, secure with EndpointRequest.
Liveness = internal state only; readiness = can serve now. Never put shared dependencies in liveness; use a startupProbe for slow boots.
Meter = name + tags. Counter for events, gauge for current state, timer + histogram for latency; bound every tag set.
Percentiles from histograms, summed across instances, not averages and not per-pod quantiles.
Tracing needs propagation: Boot-built clients, context across threads, a sampling rate you chose, and an exporter that is configured.
Logs: structured, correlated (traceId, requestId), cleaned MDC, one stack trace per failure.
Alert on symptoms (error ratio, latency SLO burn), investigate with causes (saturation, GC, pool).

Related pages

Configuration & Profiles — exposure and masking properties, show-values, profile-specific management settings.
Security — the filter chain that guards /actuator.
Caching & Resilience — Resilience4j and cache metrics that surface on /actuator/metrics.
Spring Data JPA — Hikari pool saturation and N+1 queries seen through metrics.
Testing — testing health indicators and endpoints with @SpringBootTest.
AOP & Proxies — how @Timed, @Observed and @Counted work and where self-invocation bites.

← Configuration & Profiles
Spring hub
Caching & Resilience →

LearningTree

Actuator & Observability | Spring Guide

Actuator & Observability

Running Is Not the Same as Working

Endpoints, Registries and Observations

Endpoints, Exposure and Securing Them

Health Indicators, Groups, Liveness and Readiness

Micrometer: Counters, Gauges, Timers and Tags

Distributed Tracing with Micrometer Tracing

Structured Logging, MDC and Correlation

From /actuator/prometheus to a Grafana Alert

A Mini Metrics Registry, Health Aggregator and MDC in Plain Java

Reading the Signals: Symptom to Cause

Questions You Will Be Asked

Summary & Cheat Sheet