DevOps · 04

Kubernetes Operations

A Deployment that runs is not a Deployment that is safe. This page makes the orders service production-grade: probes wired to Spring Boot Actuator, sensible requests and limits, a JVM sized to its container, autoscaling, upgrades and drains without dropped requests, security in layers, a Helm and a Kustomize packaging example, and decision trees with the exact kubectl commands for Pending, CrashLoopBackOff, ImagePullBackOff, OOMKilled, failing readiness and Services with no endpoints.

Chapters

1.37

Kubernetes version

Failure patterns

Diagrams

Map · Linux · Docker · K8s Core · K8s Ops · CI/CD · Terraform · Observability · Deployments · Interview

Chapter One - Production Readiness

What Separates a Demo Deployment from a Production One

Page 03 produced a Deployment that runs. This page makes it safe to run: able to tell Kubernetes whether it is healthy, sized so it neither starves nor gets killed, scaled by load, upgraded and drained without dropping requests, locked down so a compromise stays small, and debuggable at 03:00. The same orders service is used throughout, and the manifests are validated against a real Kubernetes 1.37 cluster.

Concern	Question it answers	Main objects and settings	Chapter
Health	Is it up? Can it take traffic? Is it still starting?	startup / liveness / readiness probes, Actuator health groups	02
Resources	How much does it need and what happens when it exceeds that?	requests, limits, QoS class, JVM memory flags	03, 04
Scaling	How many Pods for the current load?	HPA (`autoscaling/v2`), VPA, KEDA	05
Change	Can I deploy and drain nodes without errors?	rolling update, PodDisruptionBudget, graceful shutdown, preStop	06
Placement	Where do Pods run and how are they spread?	nodeSelector, affinity, taints and tolerations, topology spread	07
Security	Who may do what, and what can a Pod do?	RBAC, ServiceAccount, NetworkPolicy, Pod Security Standards, Secrets	08, 09
Packaging	How do I keep one set of manifests for many environments?	Helm, Kustomize	09
Operations	It is broken. What now?	`describe`, `logs --previous`, events, decision trees	10

Key insight: Kubernetes will do exactly what the manifest says, quickly and tirelessly, including the wrong thing. A missing readiness probe does not produce an error. It produces an outage during the next deploy.

Verified against Kubernetes 1.37 (documentation, and a live kind cluster running v1.37.0) and the Spring Boot 4.1 reference documentation for Actuator and graceful shutdown. Version-specific statements are marked inline.

Chapter Two - Health

Probes and Spring Boot Actuator

The kubelet runs probes against each container. A probe is an HTTP GET, a TCP connect, a gRPC check or a command; the container is judged by the result. The three kinds have different meanings, and mixing them up causes most self-inflicted outages.

Three probes, three different questions, three different reactions

Probe	Question	On failure	Typical endpoint
startupProbe	Has the application finished starting?	Container restarted if it never succeeds within `failureThreshold x periodSeconds`. While it has not succeeded, liveness and readiness probes do not run	Same as liveness
livenessProbe	Is the process wedged in a way only a restart fixes?	Container is restarted	`/actuator/health/liveness`
readinessProbe	Should this Pod receive traffic now?	Pod is removed from Service endpoints; nothing is restarted; it is added back when the probe passes again	`/actuator/health/readiness`

The Kubernetes 1.37 API documentation gives the defaults for probe fields:

Field	Default	Meaning
`periodSeconds`	10	How often to probe (minimum 1)
`timeoutSeconds`	1	Probe times out after this many seconds
`failureThreshold`	3	Consecutive failures before the probe is considered failed
`successThreshold`	1	Consecutive successes to be considered healthy again (must be 1 for liveness and startup)
`initialDelaySeconds`	0	Delay after container start before probing; prefer a startup probe to a long delay

A startup probe is the right way to protect a slow-starting JVM. The Kubernetes documentation suggests a startup probe with the same check as liveness and a failureThreshold x periodSeconds large enough for the worst-case start. Example: periodSeconds: 5 and failureThreshold: 36 allows up to 180 seconds to start, but a Pod that starts in 20 seconds is detected after about 20 seconds, and once the startup probe passes the tight liveness settings take over.

Wiring probes to Spring Boot Actuator

Spring Boot exposes the application's own availability state through two health groups. According to the Spring Boot 4.1 reference, Actuator collects Liveness and Readiness from ApplicationAvailability and publishes them as /actuator/health/liveness and /actuator/health/readiness. The property management.endpoint.health.probes.enabled (default true in the 4.1 properties reference) controls them.

application.yaml - what to set (Spring Boot 4.1 properties verified)

management:
endpoint:
  health:
    probes:
      enabled: true               # default true in 4.1 docs; explicit is harmless
      add-additional-paths: true  # also serve /livez and /readyz on the main port
endpoints:
  web:
    exposure:
      include: health
server:
shutdown: graceful                # enabled by default in the 4.1 reference; explicit is harmless
spring:
lifecycle:
  timeout-per-shutdown-phase: 20s # default 30s

Lifecycle moment	LivenessState	ReadinessState	HTTP server
Starting	BROKEN	REFUSING_TRAFFIC	Not started
Context refreshed, startup tasks running	CORRECT	REFUSING_TRAFFIC	Refuses requests
Ready	CORRECT	ACCEPTING_TRAFFIC	Accepts requests
Graceful shutdown requested	CORRECT	REFUSING_TRAFFIC	New requests rejected, in-flight allowed to finish
Shutdown complete	n/a	n/a	Stopped

This table is from the Spring Boot reference. Note the last two rows: when shutdown starts, readiness flips to REFUSING_TRAFFIC and the HTTP server stops accepting new requests. Both are inside the Pod. Kubernetes still needs a moment to remove the Pod from Service endpoints, which is why chapter 06 adds a preStop sleep.

YAML - probes for the orders container (excerpt of the full manifest below)

startupProbe:
httpGet: { path: /actuator/health/liveness, port: http }
periodSeconds: 5
failureThreshold: 36          # up to 180 s to start
livenessProbe:
httpGet: { path: /actuator/health/liveness, port: http }
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet: { path: /actuator/health/readiness, port: http }
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 3

⚠ Putting dependencies in the liveness check

If liveness fails when the database is down, Kubernetes restarts every instance at once, the restarts add load to a database that is already struggling, and a dependency outage becomes a total outage with a restart storm. The Spring Boot documentation warns about the same trap (checking external state with probes can cause cascading failures). Keep liveness to "is my own process healthy". Use readiness for "can I serve requests", and think carefully before including a shared dependency even there: a database blip that fails readiness on all Pods leaves the Service with no endpoints.

⚠ A readiness probe on a different port than the app

With a separate management port the probe can pass while the main port is dead. The Spring Boot docs recommend making the groups available on the main port in that case (add-additional-paths) so the probe reflects the real server.

Chapter Three - Resources

Requests, Limits, QoS, Throttling and OOMKill

Every container can declare requests and limits for CPU and memory. They look alike and do different jobs.

requests decide where; limits decide what happens at the edge

	Request	Limit
Used by	The scheduler: a node must have this much unreserved capacity. Also the baseline for HPA utilisation and for QoS	The kernel (cgroups), enforced at runtime
CPU	A guaranteed share when the node is busy	Hard ceiling enforced by throttling: the container is paused for part of each period. Never killed for CPU
Memory	Reserved for scheduling	Hard ceiling: exceeding it triggers the kernel OOM killer; the container exits with code 137 and reason `OOMKilled`
If omitted	Treated as 0 for scheduling (unless a LimitRange or admission default applies); if a limit is set, the request defaults to it	No ceiling (up to the node)

CPU and memory differ in kind. CPU is compressible: taking some away only slows work. Memory is not compressible: the only way to reclaim it is to kill. The 1.37 resource-management documentation states it this way: CPU limits are enforced by throttling, memory limits are enforced reactively, and a container may exceed its memory limit but is killed if it does.

Symptom	Cause	How to confirm
Latency spikes, low CPU use on dashboards, `container_cpu_cfs_throttled_periods_total` growing	CPU limit too low: threads are paused mid-request, GC and JIT threads included	Throttling metrics (page 07); remove or raise the CPU limit
Container restarts, `Last State: Terminated, Reason: OOMKilled, Exit Code: 137`	Memory limit exceeded (heap plus off-heap plus native memory)	`kubectl describe pod`; see the real output below
Pod `Evicted` with a message about node pressure	Node ran short of memory or disk and the kubelet evicted Pods, lowest QoS and most over their request first	`kubectl describe pod`, node conditions
Pod `Pending` with `Insufficient cpu`/`memory`	Requests larger than any node's free allocatable capacity	Events on the Pod

QoS classes

Kubernetes assigns each Pod a Quality of Service class from its requests and limits. It is not a field you set; it is derived, and used when the node must evict Pods.

Class	Rule	Meaning under pressure
Guaranteed	Every container has CPU and memory requests and limits, and they are equal	Last to be evicted
Burstable	At least one request or limit is set, but not the Guaranteed pattern	Evicted after BestEffort; those exceeding their requests go first
BestEffort	No requests and no limits anywhere	First to be evicted

Shell - real output (abridged): a Pod that exceeds its memory limit, from the 1.37 test cluster

$ kubectl get pod oom-demo
NAME       READY   STATUS             RESTARTS        AGE
oom-demo   0/1     CrashLoopBackOff   7 (3m40s ago)   14m

$ kubectl describe pod oom-demo
...
  State:          Waiting
    Reason:       CrashLoopBackOff
  Last State:     Terminated
    Reason:       OOMKilled
    Exit Code:    137
  Restart Count:  7
  Limits:
    memory:  50Mi
  Requests:
    memory:     50Mi
...
QoS Class:                   Burstable

The test Pod used a stress tool allocating about 200 MiB under a 50 MiB limit. Note QoS Class: Burstable even though the memory request equals the limit: Guaranteed needs CPU requests and limits as well. Many teams deliberately settle for Burstable (see the next section).

The CPU limit debate

Memory request equal to memory limit is the common advice: it avoids a Pod using more memory than the scheduler accounted for, then being killed or evicted when a neighbour needs it. For CPU the choice is more nuanced, and the right answer depends on the workload:

Choice	Pros	Cons
CPU request, no CPU limit	Uses idle CPU, avoids throttling latency; requests still guarantee a fair share when the node is busy	A noisy Pod can use more than expected; capacity planning rests on requests; you cannot use the Guaranteed class
CPU request = CPU limit	Predictable; Guaranteed QoS (with memory set equally); fits strict multi-tenant policies	Throttling can add tail latency, especially for multi-threaded JVMs with GC and JIT bursts

The manifest in this page uses a CPU request and no CPU limit, with memory request equal to the limit. Namespaces with a ResourceQuota or policies that require limits may force a choice; follow your platform rules.

Cluster guardrails

A LimitRange sets default and maximum requests and limits per container in a namespace, and a ResourceQuota caps the total a namespace may request. Together they stop one team from taking the cluster or deploying a Pod with no resource settings.

Chapter Four - JVM in a Container

Sizing a Spring Boot Pod

The JVM is container-aware: it reads the cgroup memory and CPU limits and sizes itself from them. The key is that heap is only part of what the process uses. A pod limit has to cover heap, metaspace, thread stacks, the code cache, direct buffers, GC structures and native libraries, plus headroom. Set the heap too close to the limit and the kernel kills the container even though the heap never overflowed.

Default and configured heap sizes, measured with JDK 21 (eclipse-temurin:21-jre) in a container started with --memory=512m --cpus=1:

Shell - real output (abridged): heap sizing from the cgroup limit

$ docker run --rm --memory=512m --cpus=1 eclipse-temurin:21-jre \
  java -XX:+PrintFlagsFinal -version | grep -E " MaxHeapSize |MaxRAMPercentage"
 size_t MaxHeapSize          = 134217728     {product} {ergonomic}     # 128 MiB
 double MaxRAMPercentage     = 25.000000     {product} {default}

$ docker run --rm --memory=512m --cpus=1 eclipse-temurin:21-jre \
  java -XX:MaxRAMPercentage=75 -XX:+PrintFlagsFinal -version | grep -E " MaxHeapSize |MaxRAMPercentage"
 size_t MaxHeapSize          = 402653184     {product} {ergonomic}     # 384 MiB
 double MaxRAMPercentage     = 75.000000     {product} {command line}

The default MaxRAMPercentage is 25%, so a container limited to 512 MiB gets a 128 MiB heap by default (the JVM also reported cgroup v2 metrics and an effective CPU count of 1 for --cpus=1). That default wastes memory in a container dedicated to one JVM, which is why -XX:MaxRAMPercentage is usually raised. The JAVA_TOOL_OPTIONS environment variable is a convenient place to set it from a manifest, as the final YAML does.

Component	Rough size drivers	Notes
Heap (`-Xmx` / `MaxRAMPercentage`)	Your live data and allocation rate	The largest part, but not all of it
Metaspace	Number of loaded classes (Spring, Hibernate, proxies)	Grows with libraries; unbounded unless `MaxMetaspaceSize` is set
Thread stacks	Threads x stack size (`-Xss`, default usually 1 MiB on 64-bit Linux)	Hundreds of Tomcat/async threads add up
Code cache, GC data, CDS	JIT output and collector bookkeeping	Tens of MiB
Direct / native memory	NIO buffers, Netty, compression, JNI	Not covered by `-Xmx`

Worked example (an example, not a recommendation)

Suppose measurements of a service under realistic load show a steady live heap of about 300 MiB and a peak process RSS of about 700 MiB with the heap at its maximum. A reasonable first sizing, to be corrected by observation:

Setting	Value	Reasoning (example arithmetic)
Memory limit	1 Gi (1024 MiB)	Peak RSS 700 MiB plus roughly 30% headroom for spikes
Memory request	1 Gi	Equal to the limit: the scheduler reserves what the Pod can really use
`MaxRAMPercentage`	75	75% of 1024 MiB = 768 MiB maximum heap, leaving about 256 MiB for metaspace, stacks, code cache and native memory
CPU request	500m	Average use measured under load, with room before throttling matters. No CPU limit (chapter 03)

Then measure: watch container working-set memory and heap usage during a load test and over days, and adjust. A heap percentage of 75 is a common starting point for a 1 GiB container, but a service with many threads, large direct buffers or a big metaspace may need 60 to 70 to stay clear of the kill line.

⚠ Setting the heap equal to the limit

-Xmx1g in a container limited to 1 Gi guarantees an OOMKill under load, long before a Java OutOfMemoryError: non-heap memory has nowhere to go. The symptom is OOMKilled with exit code 137, no stack trace and no heap dump, and a restart count that climbs. Leave 25 to 40 percent of the limit outside the heap and prefer percentages over fixed -Xmx so the heap follows the limit.

CPU counts

The JVM sets thread pools (GC threads, ForkJoinPool.commonPool, the available-processor count) from the CPU limit if there is one, otherwise from the node. With a request but no limit the JVM may see all the node's cores. Set -XX:ActiveProcessorCount=N when you want an explicit value, and check what the JVM reports with -XshowSettings:system in the container.

Since Kubernetes 1.35, in-place Pod resize is stable: CPU and memory of a running container can be changed without recreating the Pod, subject to the resize policy. A JVM does not re-read its heap setting while running, so a memory resize usually still means a restart in practice; check how your workload reacts before relying on it.

Chapter Five - Scaling

HPA, VPA and KEDA

The HorizontalPodAutoscaler (HPA) changes the number of replicas of a Deployment (or StatefulSet) from observed metrics. Most JVM services scale horizontally: more Pods, same size.

HPA control loop (autoscaling/v2)

The formula, from the Kubernetes 1.37 HPA documentation, is desiredReplicas = ceil[currentReplicas x (currentMetricValue / desiredMetricValue)]. The loop runs periodically; the control plane skips scaling when the ratio is close enough to 1.0 (tolerance, 0.1 by default). With several metrics the HPA computes a result for each and takes the largest. The arithmetic in the diagram is a worked example.

YAML - HPA and PodDisruptionBudget (autoscaling/v2, policy/v1; server dry-run checked on 1.37)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: orders
spec:
scaleTargetRef:
  apiVersion: apps/v1
  kind: Deployment
  name: orders
minReplicas: 3
maxReplicas: 12
metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
      - type: Percent
        value: 50
        periodSeconds: 60
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: orders
spec:
minAvailable: 2
selector:
  matchLabels:
    app.kubernetes.io/name: orders

Requirement or setting	Why
metrics-server (or another `metrics.k8s.io` provider)	Resource metrics come from the Metrics API, usually provided by metrics-server, an add-on. A fresh kind cluster does not have it: `kubectl top nodes` printed `error: Metrics API not available`, and an HPA there would show unknown metrics
CPU requests on the target Pods	Utilisation is a percentage of the request
`behavior.scaleDown.stabilizationWindowSeconds`	Uses the highest desired value over the window to stop flapping; 300 s is the documented default for scale down
`behavior.scaleDown.policies`	Limits how fast replicas are removed (here at most 50% per minute)
`minReplicas` of at least 2 for important services	A single replica has no redundancy during node loss or rollout

Scaling the JVM on CPU is straightforward but imperfect: a JVM can be slow because it is waiting (database, downstream calls) with low CPU, and a freshly started JVM is CPU-hungry while it warms up. Scale on a metric that matches the bottleneck (requests per second or queue depth through custom or external metrics) when CPU does not reflect load. A warm-up period, a readiness probe and a startup probe keep a new replica from being counted too early.

Tool	What it does	Notes (verified against project pages and the 1.37 docs)
HPA	More or fewer replicas	Built into Kubernetes. Since 1.37, scaling to zero (`minReplicas: 0`) is beta and enabled by default for object and external metrics; it requires such a metric
VPA (Vertical Pod Autoscaler)	Adjusts container CPU and memory requests from observed use	A separate add-on from the Kubernetes autoscaler project, not part of core. Do not let it fight an HPA on the same metric. Useful in recommendation mode to find right-sized requests
KEDA	Event-driven autoscaling: scales Deployments, StatefulSets and Jobs from external event sources (queue length, stream lag, schedules), including to and from zero	A separate project (keda.sh) that works with the HPA machinery; useful for consumers of Kafka or SQS (see the Messaging section)
Cluster autoscaling (Cluster Autoscaler, Karpenter)	Adds or removes nodes when Pods are Pending for lack of capacity	Separate from Pod autoscaling; an HPA that wants more Pods needs nodes to put them on

⚠ HPA and a fixed replicas in the Deployment YAML

If your manifest says replicas: 3 and the HPA scaled to 7, the next kubectl apply resets it to 3, and the HPA then has to scale up again. Omit replicas from the Deployment when an HPA owns it (or let your GitOps tool ignore the field).

Chapter Six - Changing Things Safely

Rolling Updates, PodDisruptionBudget and Graceful Shutdown

Rolling update parameters

Chapter 05 of page 03 covered the mechanics. For a production service the usual setting is maxUnavailable: 0 with maxSurge: 1 (or a percentage): a new Pod must become Ready before any old Pod is removed, so serving capacity never drops. It needs room for the surge Pod and a readiness probe that means what it says. minReadySeconds adds a soak time, so a Pod that crashes a few seconds after becoming Ready does not let the rollout proceed. progressDeadlineSeconds (default 600) marks a stuck rollout as failed; it does not roll back automatically: a CI pipeline should watch kubectl rollout status and run kubectl rollout undo on failure (page 05).

PodDisruptionBudget

Rolling updates are controlled by the Deployment. Voluntary disruptions are the other source of Pod removal: kubectl drain during node upgrades, cluster autoscaler scale-down, or anything using the Eviction API. A PodDisruptionBudget (policy/v1) tells the eviction machinery how many Pods of an app must stay: minAvailable or maxUnavailable (one of the two). An eviction that would violate it is refused (HTTP 429) and the drain waits and retries.

A PDB covers voluntary evictions only. Node crashes, OOM kills and your own kubectl delete pod are not blocked by it.
With replicas: 1 and minAvailable: 1 a node can never be drained (the single Pod may never be evicted): upgrades stall. Use at least 2 replicas, or maxUnavailable: 1.
spec.unhealthyPodEvictionPolicy controls whether Pods that are not Ready may be evicted even when the budget is not met; consult the 1.37 disruptions page before setting it.
The PDB in chapter 05 (minAvailable: 2 for 3 replicas) allows one Pod at a time to be drained.

Graceful shutdown with Spring Boot and the preStop sleep

Page 03 chapter 11 described the termination sequence. The practical consequence: removal from endpoints and SIGTERM are not synchronised. For a few seconds after the Pod is marked Terminating, some nodes and load balancers still send it new requests. If SIGTERM makes the app refuse new connections immediately (which Spring Boot does: readiness becomes REFUSING_TRAFFIC and the web server stops accepting), those late requests fail with connection errors during every deploy.

Shutdown timeline of one orders Pod (preStop sleep 10 s, Spring timeout 20 s, grace 60 s)

The fix is a short preStop hook that just waits. The 1.37 lifecycle-hook documentation lists the handlers exec, httpGet and sleep, plus tcpSocket, which is marked deprecated. The built-in **sleep action** runs in the kubelet and needs no sleep binary in the image, so it works with distroless images. The full manifest below uses it.

Setting	Value in the example	Role
`lifecycle.preStop.sleep.seconds`	10	Keep serving while endpoint removal propagates. Typical values are a few seconds up to 15; measure how long your load balancer needs
`server.shutdown`	`graceful`	Stop accepting, let in-flight requests finish
`spring.lifecycle.timeout-per-shutdown-phase`	`20s` (default is 30s)	Longest wait for in-flight requests; set slightly above your slowest request
`terminationGracePeriodSeconds`	60	Must exceed preStop + Spring timeout + margin. The countdown includes the preStop time. Default is 30

⚠ Grace period shorter than the sleep plus the drain

With the default 30-second grace period, a 10-second preStop and a 30-second Spring drain add up to 40 seconds. Kubernetes sends SIGKILL at 30 and the in-flight requests it was protecting are cut. Always do the sum, and put the margin in terminationGracePeriodSeconds.

YAML - the production-ready orders Deployment (apps/v1; server dry-run checked, and it also passes the restricted Pod Security profile)

apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
labels:
  app.kubernetes.io/name: orders
spec:
replicas: 3
selector:
  matchLabels:
    app.kubernetes.io/name: orders
strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0
template:
  metadata:
    labels:
      app.kubernetes.io/name: orders
  spec:
    serviceAccountName: orders
    terminationGracePeriodSeconds: 60
    securityContext:
      runAsNonRoot: true
      runAsUser: 10001
      seccompProfile:
        type: RuntimeDefault
    topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: orders
    containers:
      - name: orders
        image: registry.example.com/shop/orders:1.4.2
        ports:
          - name: http
            containerPort: 8080
        env:
          - name: JAVA_TOOL_OPTIONS
            value: "-XX:MaxRAMPercentage=75"
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop: ["ALL"]
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            memory: 1Gi
        startupProbe:
          httpGet:
            path: /actuator/health/liveness
            port: http
          periodSeconds: 5
          failureThreshold: 36
        livenessProbe:
          httpGet:
            path: /actuator/health/liveness
            port: http
          periodSeconds: 10
          timeoutSeconds: 2
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /actuator/health/readiness
            port: http
          periodSeconds: 5
          timeoutSeconds: 2
          failureThreshold: 3
        lifecycle:
          preStop:
            sleep:
              seconds: 10
        volumeMounts:
          - name: tmp
            mountPath: /tmp
    volumes:
      - name: tmp
        emptyDir: {}

Zero-downtime recipe: two or more replicas, maxUnavailable: 0, readiness probe on the Actuator group, preStop sleep, graceful shutdown, a PDB, and a grace period that covers the sum.
The container runs as non-root, drops all capabilities, forbids privilege escalation and uses a read-only root filesystem with an emptyDir for /tmp (the JVM and Tomcat need a writable temp directory). That is what makes the Pod pass the restricted Pod Security profile in chapter 08.
The topology spread constraint (chapter 07) spreads replicas across zones. serviceAccountName: orders refers to the ServiceAccount defined in chapter 08.

Chapter Seven - Scheduling

Where Pods Run

The scheduler places each Pod on a node in two steps: filter out nodes that cannot run it (not enough requested resources, taints it does not tolerate, selector or affinity mismatch), then score the rest and pick one. You influence both steps with constraints. Use the simplest tool that expresses what you need.

Tool	Expresses	Hard or soft	Typical use
`nodeSelector`	Run only on nodes with these labels	Hard	GPU nodes, a particular architecture or node pool
Node affinity	Richer node-label rules (`In`, `NotIn`, `Exists`...)	`requiredDuringSchedulingIgnoredDuringExecution` is hard; `preferredDuring...` is soft	Prefer spot nodes, require a zone set
Pod affinity / anti-affinity	Place near (or away from) Pods matching a label, within a topology domain	Hard or soft	Keep replicas on different nodes; co-locate a cache with an app
Taints and tolerations	A node repels all Pods except those that tolerate its taint	Effects: `NoSchedule`, `PreferNoSchedule`, `NoExecute`	Dedicated nodes, control-plane nodes, evicting from failing nodes
Topology spread constraints	Keep matching Pods evenly spread across zones, nodes or other domains within a maximum skew	`whenUnsatisfiable: DoNotSchedule` (hard) or `ScheduleAnyway` (soft)	Spread replicas across availability zones

Taints and affinity are opposites in direction: a taint is set on the node and pushes Pods away unless they tolerate it, while affinity and nodeSelector are set on the Pod and pull it toward nodes. Using both gives dedicated node pools: taint the pool, then give the intended Pods a toleration and a node selector (the toleration alone only allows the Pod on those nodes, it does not attract it).

YAML - tolerations, node affinity and pod anti-affinity (excerpt of a Pod spec)

spec:
tolerations:
  - key: dedicated
    operator: Equal
    value: batch
    effect: NoSchedule
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: node.kubernetes.io/instance-type
              operator: In
              values: ["m6i.large", "m6i.xlarge"]
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: orders

For "spread my replicas" the modern, more predictable tool is the topology spread constraint used in the Deployment above: maxSkew: 1 across topology.kubernetes.io/zone keeps replica counts per zone within one of each other, and whenUnsatisfiable: ScheduleAnyway makes it a preference so a zone outage or capacity shortage does not leave Pods Pending. Use DoNotSchedule only when the spread is a hard requirement.

⚠ Hard anti-affinity on a small cluster

requiredDuringScheduling... anti-affinity per hostname with 5 replicas and 3 nodes leaves 2 Pods Pending forever (and breaks rolling updates that need a surge Pod). Prefer the soft form or a topology spread constraint with ScheduleAnyway.

Chapter Eight - Security

RBAC, ServiceAccounts, NetworkPolicy and Pod Security

Assume a container will eventually be compromised, and limit what that buys the attacker. Kubernetes gives you layers, each configured separately, and each worth having.

Defence in layers around one Pod

RBAC and ServiceAccounts

RBAC answers "who may do what to which resources". Four objects: a Role (namespaced permissions) or ClusterRole (cluster-wide, or reusable), and a RoleBinding or ClusterRoleBinding that grants it to subjects (users, groups, ServiceAccounts). Permissions are only additive: there is no "deny".

Every Pod runs as a ServiceAccount (the default one if you do not say). Anything the Pod does against the Kubernetes API is authorised by that account's RBAC. Most application Pods never call the API at all, so give each application its own ServiceAccount, grant it nothing it does not need, and do not mount the token unless the app uses it (automountServiceAccountToken: false). Pods mount a short-lived projected token from 1.22 on; earlier versions used a long-lived static token stored in a Secret.

YAML - least privilege: a ServiceAccount, a Role and RoleBinding, and NetworkPolicies (server dry-run checked)

apiVersion: v1
kind: ServiceAccount
metadata:
name: orders
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: orders-config-reader
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["orders-env", "orders-files"]
  verbs: ["get", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: orders-config-reader
subjects:
- kind: ServiceAccount
  name: orders
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: orders-config-reader
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
spec:
podSelector: {}
policyTypes: ["Ingress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: orders-allow-gateway
spec:
podSelector:
  matchLabels:
    app.kubernetes.io/name: orders
policyTypes: ["Ingress"]
ingress:
  - from:
      - namespaceSelector:
          matchLabels:
            kubernetes.io/metadata.name: ingress
      - podSelector:
          matchLabels:
            app.kubernetes.io/name: cart
    ports:
      - port: 8080
        protocol: TCP

Shell - real output (abridged): checking permissions without running anything

$ kubectl auth can-i get configmap/orders-env --as=system:serviceaccount:dry:orders -n dry
yes
$ kubectl auth can-i delete pods --as=system:serviceaccount:dry:orders -n dry
no

Prefer a Role in one namespace over a ClusterRole. Use resourceNames to pin to specific objects as above.
Avoid * in verbs or resources, and be wary of secrets (read access to Secrets is read access to credentials), pods/exec, and the right to create Pods or bind roles (each can be turned into more privilege).
**Never bind cluster-admin to an application.** Review bindings with kubectl get rolebindings,clusterrolebindings -A.

NetworkPolicy

By default every Pod can talk to every other Pod. A NetworkPolicy (networking.k8s.io/v1) selects Pods and states which ingress and/or egress traffic is allowed to them; once a Pod is selected by a policy for a direction, only traffic matching some policy is allowed. The standard pattern is default-deny plus explicit allows, as in the YAML above.

⚠ NetworkPolicy with a CNI that does not enforce it

Network policies are implemented by the network plugin. The Kubernetes documentation says that creating a NetworkPolicy without a controller that implements it has no effect: no error, no protection. Confirm that your CNI enforces policies and test it (a Pod that should be blocked really is). On EKS the Amazon VPC CNI add-on can enforce them when the feature is turned on, as the EKS guide describes; other CNIs (Calico, Cilium) bring their own enforcement.

Remember egress: a default-deny egress policy also blocks DNS, so allow UDP and TCP port 53 to the DNS Pods, and allow traffic to your database and message broker explicitly.

Pod Security Standards

The Pod Security Standards define three cumulative profiles, enforced by the built-in Pod Security Admission controller using namespace labels:

Profile	What it allows	Use for
privileged	Unrestricted, including known privilege escalations	System and infrastructure workloads (CNI, storage drivers) in trusted namespaces
baseline	Minimally restrictive: blocks known escalations (host namespaces, privileged containers, most host paths...)	A sensible floor for general workloads
restricted	Heavily restricted: non-root, no privilege escalation, capabilities dropped, seccomp profile set, limited volume types	Security-sensitive applications; the target for new services

Each namespace can set the mode per profile: enforce (reject), audit (log) and warn (tell the user). A real rejection from the 1.37 cluster, in a namespace labelled pod-security.kubernetes.io/enforce=restricted:

Shell - real output: enforcing the restricted profile

$ kubectl label ns psa pod-security.kubernetes.io/enforce=restricted
$ kubectl run root-pod --image=nginx:1.28 -n psa
Error from server (Forbidden): pods "root-pod" is forbidden: violates PodSecurity "restricted:latest":
allowPrivilegeEscalation != false (container "root-pod" must set securityContext.allowPrivilegeEscalation=false),
unrestricted capabilities (container "root-pod" must set securityContext.capabilities.drop=["ALL"]),
runAsNonRoot != true (pod or container "root-pod" must set securityContext.runAsNonRoot=true),
seccompProfile (pod or container "root-pod" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

The same cluster accepted the hardened orders Deployment from chapter 06 in that namespace (kubectl apply --dry-run=server succeeded), which is the point of its securityContext block. Roll the labels out in warn and audit first to see what would break, then switch to enforce.

Chapter Nine - Secrets and Packaging

Secrets Management, Helm and Kustomize

Secrets management options

Page 03 explained that a Kubernetes Secret is only base64 and sits unencrypted in etcd by default. The real question for a team is where the secret lives and how it reaches the cluster without a plaintext value in Git. The options, qualitatively:

Approach	How it works	Strengths	Watch out for
Plain Secret + RBAC + etcd encryption	Secret objects, API server encryption at rest (KMS provider recommended)	Simplest; no extra components	Values still pass through your CI or hand; not in Git safely; any reader of the Secret sees plaintext
Sealed Secrets	Encrypt a Secret with a cluster-held key into a `SealedSecret` that is safe to commit; a controller decrypts it in the cluster	GitOps friendly; no external service	Key management and backup of the controller's key; rotation is manual; the decrypted value is still a normal Secret
External Secrets Operator	Custom resources describe which entries to read from an external store (AWS Secrets Manager, Vault, cloud stores); a controller creates and refreshes Kubernetes Secrets	Central store, rotation and audit in the store; nothing secret in Git	Another operator to run; the synced Kubernetes Secret has the usual weaknesses
Secrets Store CSI driver	Mounts secrets from an external store as files in the Pod, optionally syncing to a Secret	Values need not become Secret objects	Provider-specific setup; mounted at Pod start
Cloud KMS / identity-based access	The app reads from the cloud secret store directly with a workload identity (EKS: IAM role for the ServiceAccount)	No long-lived credentials in the cluster; per-workload IAM	App code or sidecar must integrate; cloud-specific

I am describing categories here and not endorsing a product; each project's documentation is the authority for installation, versions and security guarantees. Whatever you pick: workload identity beats static keys, rotate, and keep Secret read access in RBAC narrow.

Helm versus Kustomize

You rarely deploy one hand-written YAML file. Different environments differ in image tags, replica counts, resource sizes and host names. Two tools solve that, with opposite philosophies:

	Helm (checked with v4.3.1)	Kustomize (built into `kubectl`, v5.8.1 in kubectl 1.37.2)
Approach	Templating: Go templates plus a `values.yaml`, packaged as a versioned chart	Patching: plain YAML bases plus overlays that patch them; no template language
Strengths	Reusable, shareable charts (most third-party software ships as a chart); release history, upgrade and rollback of a whole release (`helm upgrade`, `helm rollback`); hooks	No new syntax; the base is valid YAML; overlays are small and readable diffs; built into `kubectl -k`
Weaknesses	Templates mix YAML and logic and get hard to read; indentation bugs; the output is only visible by rendering	Limited logic; sharing and versioning are by Git, not by a package
Typical use	Distributing a product, installing third-party software, or an app with many options	Your own applications across dev / stage / prod
Combine?	Yes: `helm template` output can be post-processed by Kustomize, or a GitOps tool can use either

Kustomize - base kustomization.yaml (base/ also holds deployment.yaml and service.yaml)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml

Kustomize - overlays/prod/kustomization.yaml (rendered with kubectl kustomize overlays/prod)

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-prod
resources:
- ../../base
images:
- name: registry.example.com/shop/orders
  newTag: 1.4.2
replicas:
- name: orders
  count: 3
patches:
- target:
    kind: Deployment
    name: orders
  patch: |-
    - op: add
      path: /spec/template/spec/containers/0/resources
      value:
        requests:
          cpu: 500m
          memory: 1Gi
        limits:
          memory: 1Gi

kubectl kustomize overlays/prod rendered the Deployment with 3 replicas, image registry.example.com/shop/orders:1.4.2, namespace shop-prod and the added resources block, and the Service in the same namespace. Apply it with kubectl apply -k overlays/prod. Preview with kubectl diff -k overlays/prod.

Helm - values.yaml and templates/deployment.yaml of a minimal chart (helm lint passed; helm template rendered it)

replicaCount: 2
image:
repository: registry.example.com/shop/orders
tag: ""
service:
port: 80
resources:
requests:
  cpu: 250m
  memory: 512Mi
limits:
  memory: 512Mi

# --- templates/deployment.yaml ---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-orders
labels:
  app.kubernetes.io/name: orders
  app.kubernetes.io/instance: {{ .Release.Name }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
  matchLabels:
    app.kubernetes.io/name: orders
    app.kubernetes.io/instance: {{ .Release.Name }}
template:
  metadata:
    labels:
      app.kubernetes.io/name: orders
      app.kubernetes.io/instance: {{ .Release.Name }}
  spec:
    containers:
      - name: orders
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
        ports:
          - name: http
            containerPort: 8080
        resources:
          {{- toYaml .Values.resources | nindent 12 }}

Shell - using the chart (helm template and helm lint were run; install and upgrade are the standard commands)

$ helm lint chart/orders
==> Linting chart/orders
1 chart(s) linted, 0 chart(s) failed
$ helm template shop chart/orders --set replicaCount=3 --set image.tag=1.4.2   # render only, to review
$ helm upgrade --install shop chart/orders -n shop-prod --create-namespace \
  --set image.tag=1.4.2 --atomic --timeout 5m
$ helm history shop -n shop-prod
$ helm rollback shop 1 -n shop-prod

Roll pods when a ConfigMap changes

Env vars do not refresh and mounted files refresh only eventually. A common Helm idiom adds an annotation to the Pod template such as checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}, so the Pod template (and therefore the ReplicaSet) changes whenever the config does and a rolling update is triggered. Kustomize generates ConfigMaps with a content hash suffix for the same reason (configMapGenerator).

Chapter Ten - Troubleshooting

Decision Trees and the Exact Commands

Most Kubernetes problems fall into a handful of patterns, and the first two commands are always the same: kubectl describe pod NAME (read Events at the bottom and Last State of each container) and kubectl logs NAME --previous (the log of the container that just died). Every symptom below was reproduced on the 1.37 test cluster and the outputs are real, shortened where marked.

Decision trees: Pending, ImagePullBackOff, CrashLoopBackOff, OOMKilled

Decision trees: readiness failing and Service has no endpoints

Pending

Shell - find out why a Pod is not scheduled (real output, abridged)

$ kubectl get pods
NAME           READY   STATUS    RESTARTS   AGE
pending-demo   0/1     Pending   0          46s
$ kubectl describe pod pending-demo | tail -4
Events:
Type     Reason            Age   From               Message
Warning  FailedScheduling  48s   default-scheduler  0/1 nodes are available: 1 Insufficient cpu. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

The scheduler's message lists every reason per node. Compare the Pod's requests with node allocatable (kubectl describe node NAME shows Allocatable and Allocated resources). Other messages: didn't match Pod's node affinity/selector, had untolerated taint, pod has unbound immediate PersistentVolumeClaims (check kubectl get pvc and the StorageClass), Too many pods. With a cluster autoscaler, a Pod stays Pending briefly while a node is added.

ImagePullBackOff and ErrImagePull

Shell - real output (abridged)

$ kubectl get pod badimg
NAME     READY   STATUS             RESTARTS   AGE
badimg   0/1     ImagePullBackOff   0          25s
$ kubectl describe pod badimg | tail -4
Normal   BackOff  20s  kubelet  Back-off pulling image "nginx:nope"
Warning  Failed   20s  kubelet  Error: ImagePullBackOff
Warning  Failed   3s   kubelet  Failed to pull image "nginx:nope": rpc error: code = NotFound ...
Warning  Failed   3s   kubelet  Error: ErrImagePull

ErrImagePull is the failed attempt, ImagePullBackOff is the kubelet waiting before it retries. Read the message: NotFound means a wrong name or tag; unauthorized or denied means missing credentials: create a docker-registry Secret and reference it in imagePullSecrets (or use node or workload IAM for ECR on EKS). Also check the registry is reachable from the node, and that imagePullPolicy and a mutable tag like latest are not giving you a different image than you think.

CrashLoopBackOff and OOMKilled

Shell - the commands, in order

$ kubectl get pod NAME                       # RESTARTS column, STATUS CrashLoopBackOff
$ kubectl logs NAME --previous               # why the last run died (stack trace, config error)
$ kubectl describe pod NAME                  # Last State: Reason, Exit Code; Events; probes
$ kubectl get pod NAME -o jsonpath='{.status.containerStatuses[0].lastState}'

Last State	Meaning	Usual cause and fix
`Reason: Error`, `Exit Code: 1` (or other small codes)	The process exited with an error	Read `logs --previous`: missing env var or Secret, wrong database URL, failed migration. Fix config
`Reason: OOMKilled`, `Exit Code: 137`	The kernel killed it for exceeding its memory limit	Raise the memory limit or lower the heap percentage (chapter 04); check for a leak
`Exit Code: 137` without OOMKilled, with a liveness failure event	SIGKILL after the kubelet restarted it for a failing liveness probe	Add a startup probe, increase `failureThreshold` and `timeoutSeconds`
`Exit Code: 143`	Terminated by SIGTERM	Normal during shutdown; if unexpected, look for evictions and rollouts
`Exit Code: 126` / `127`	Command cannot execute / not found	Wrong `command` or entrypoint, missing binary

The kubelet restarts with an exponential back-off, so the interval between attempts grows, and the status alternates between Error and CrashLoopBackOff. In the test run a container that exited with code 1 printed its message but kubectl logs --previous sometimes returned "unable to retrieve container logs" for a very short-lived container; if that happens, try again or read the node's container log. Use kubectl debug with an ephemeral container or a copy of the Pod (kubectl debug POD -it --copy-to=debug --container=app -- sh) when the image has no shell.

Readiness failing and Service has no endpoints

Shell - real output (abridged)

$ kubectl get pod notready-demo
NAME            READY   STATUS    RESTARTS   AGE
notready-demo   0/1     Running   0          46s
$ kubectl describe pod notready-demo | tail -2
Warning  Unhealthy  49s  kubelet  Readiness probe failed: Get "http://10.244.0.16:80/ready": dial tcp 10.244.0.16:80: connect: connection refused
Warning  Unhealthy  3s (x10 over 48s)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 404

$ kubectl get endpointslices
NAME         ADDRESSTYPE   PORTS     ENDPOINTS      AGE
nr-br67r     IPv4          80        10.244.0.16    58s      # exists, but ready: false in the slice
typo-b88kf   IPv4          <unset>   <unset>        58s      # selector matches no Pods

Readiness failing: the event text names the cause. First connection refused (nothing listening yet or wrong port), later 404 (the path does not exist). Test from inside: kubectl exec POD -- curl -s localhost:8080/actuator/health/readiness (or use wget on minimal images).
No endpoints: run kubectl describe svc NAME and kubectl get endpointslices -l kubernetes.io/service-name=NAME. Then walk the list: (1) kubectl get pods --show-labels against the Service selector, (2) are the Pods Ready, (3) does targetPort match the container port (the example typo had a selector of app: wbe).
Service has endpoints but calls fail: kubectl port-forward to a Pod to test the app directly, then from another Pod with the Service DNS name; check NetworkPolicies (kubectl get networkpolicy), and CoreDNS (kubectl get pods -n kube-system -l k8s-app=kube-dns).

A quick triage order

Shell - the first minute

$ kubectl get pods -o wide                                   # status, restarts, node
$ kubectl describe pod NAME                                  # Events, Last State, Conditions
$ kubectl logs NAME --previous                               # last crash
$ kubectl get events --sort-by=.lastTimestamp | tail -20     # what the cluster did recently
$ kubectl rollout status deployment/NAME                     # is a rollout stuck?
$ kubectl get endpointslices -l kubernetes.io/service-name=NAME
$ kubectl top pod NAME                                       # needs metrics-server

Chapter Eleven - On AWS, and Pitfalls

EKS Notes and Common Mistakes

EKS notes (qualitative; check the EKS user guide for current details)

Amazon EKS operates the Kubernetes control plane for you, and you manage worker capacity and workloads. These points are stable design facts; limits, prices and supported versions change, so look them up in the EKS documentation instead of trusting a number here.

Topic	What to know
Workers	Managed node groups, self-managed nodes, Karpenter (node provisioning), or Fargate (serverless Pods). The Cluster Autoscaler or Karpenter handles node scaling; the HPA handles Pod scaling
Pod networking	The Amazon VPC CNI gives Pods real VPC IP addresses, so VPC subnet size and per-node IP limits constrain how many Pods you can run. Plan address space early
NetworkPolicy	The VPC CNI can enforce Kubernetes NetworkPolicy when enabled (see the EKS guide "Restrict Pod network traffic with Kubernetes network policies"); otherwise use another CNI that enforces it
Workload identity	EKS Pod Identity and IAM roles for service accounts (IRSA) give Pods short-lived AWS credentials per ServiceAccount instead of node-wide or static keys. Use one of these for S3, SQS or Secrets Manager access
Load balancing	`type: LoadBalancer` Services and Ingress objects need the AWS Load Balancer Controller to create ALBs or NLBs; check the controller's supported versions for your cluster version
Secrets	Envelope encryption of Secrets with a KMS key can be enabled on the cluster; combine with External Secrets or the Secrets Store CSI driver and Pod Identity for AWS Secrets Manager
Upgrades	Control-plane, node group and add-on upgrades are separate steps you plan (see the EKS upgrade documentation for the supported order); PodDisruptionBudgets, multiple replicas and topology spread decide whether node rotation is invisible
Storage	EBS volumes (RWO) through the EBS CSI driver are zonal: a Pod using one is tied to that Availability Zone. EFS gives RWX across zones

Common mistakes

Mistake	Symptom	Fix
No readiness probe	502s during every rollout; new Pod receives traffic while starting	Probe the Actuator readiness group (chapter 02)
Liveness checks dependencies	All Pods restart when the database is slow	Liveness = own process only
Slow JVM start and only a liveness probe	CrashLoopBackOff while the app is still starting; exit 137 on a probe kill	Add a startup probe with a long enough window
Heap close to the memory limit	`OOMKilled`, exit 137, no Java stack trace	`MaxRAMPercentage` 60 to 75, leave room for native memory
No memory request or limit (BestEffort)	Evicted first when a node is under pressure; noisy neighbour	Set requests and limits; use a LimitRange default
CPU limit too low	Latency with low CPU graphs; throttling metric rising	Raise or remove the CPU limit (chapter 03)
Grace period shorter than preStop + drain	In-flight requests cut at every deploy	Compute the sum, set `terminationGracePeriodSeconds` above it
No PodDisruptionBudget (or one that blocks everything)	Node drain takes the whole service down, or drain hangs forever	PDB with `minAvailable` below replicas; at least 2 replicas
`latest` tag	Different nodes run different images; rollbacks do not roll back	Immutable tags or digests
Everything in `default` with the default ServiceAccount and API token mounted	Large blast radius if one Pod is compromised	Namespaces, dedicated ServiceAccount, `automountServiceAccountToken: false`
`replicas` hard-coded next to an HPA	Scale-ups reset on each apply	Omit `replicas` when the HPA owns it
Secrets in Git in plain form	Credentials leaked through history	Sealed Secrets, External Secrets or a cloud store
NetworkPolicy without enforcement	No error, no protection	Use a CNI that enforces it and test it

Chapter Twelve - Interview

Questions You Should Be Able to Answer

Crisp answers for Kubernetes operations questions.

Q: Liveness versus readiness versus startup probes?
Startup: has it finished starting (gates the others; failure restarts). Liveness: is it irrecoverably stuck (failure restarts the container). Readiness: can it serve now (failure removes it from Service endpoints; no restart).
Q: How do you wire Spring Boot to Kubernetes probes?
Actuator exposes health groups: /actuator/health/liveness and /actuator/health/readiness, backed by the application availability state. Point the probes at them, and use a startup probe for slow starts. Keep external dependencies out of liveness.
Q: Requests versus limits?
Requests are used for scheduling and are the baseline for QoS and HPA utilisation. Limits are enforced by the kernel: CPU by throttling, memory by the OOM killer.
Q: What are the QoS classes?
Guaranteed (requests = limits for CPU and memory on every container), Burstable (some requests or limits), BestEffort (none). Under node pressure BestEffort is evicted first, then Burstable, Guaranteed last.
Q: A Java container is OOMKilled but the heap is fine. Why?
The limit covers the whole process: heap plus metaspace, thread stacks, code cache, direct buffers and native memory. Use MaxRAMPercentage around 60 to 75, not -Xmx equal to the limit, and measure RSS.
Q: CPU throttling versus OOMKill?
CPU is compressible: over the limit the container is slowed. Memory is not: over the limit the container is killed (exit 137). Throttling shows as latency, OOMKill as restarts.
Q: How does the HPA decide the number of replicas?
desired = ceil(current x currentMetric / targetMetric), skipped within a 10% tolerance, clamped to min and max, with stabilization windows and policies for scale-down. It needs the Metrics API (metrics-server) and CPU requests for utilisation targets.
Q: HPA versus VPA versus KEDA versus Cluster Autoscaler?
HPA changes replica count; VPA changes requests; KEDA scales from external event sources and can reach zero; Cluster Autoscaler or Karpenter changes the number of nodes.
Q: How do you achieve zero-downtime deployments?
Several replicas, maxUnavailable: 0, readiness probe, preStop sleep so endpoint removal propagates, graceful shutdown in the app, a grace period covering both, and a PodDisruptionBudget for node drains.
Q: What does a PodDisruptionBudget protect against?
Voluntary disruptions through the Eviction API (drains, autoscaler scale-down). It does not stop crashes, OOM kills or direct deletes.
Q: How do taints and tolerations differ from affinity?
A taint is on the node and repels Pods that do not tolerate it. Affinity and nodeSelector are on the Pod and attract it to nodes. Dedicated pools use both.
Q: How does RBAC work and what is least privilege?
Roles list allowed verbs on resources, bindings attach them to users, groups or ServiceAccounts; permissions are additive. Give each app its own ServiceAccount, namespaced Roles with specific resources and resourceNames, and no token if it does not call the API.
Q: What is a NetworkPolicy and what is the catch?
It restricts Pod traffic by labels and ports; default-deny plus explicit allows is the pattern. It only works if the CNI enforces it; otherwise it silently does nothing.
Q: Pod Security Standards?
Three profiles, privileged, baseline and restricted, enforced by Pod Security Admission through namespace labels in enforce, audit and warn modes.
Q: Helm versus Kustomize?
Helm templates YAML with values and packages charts with release history. Kustomize patches plain YAML bases with overlays and is built into kubectl. Helm for distributing software, Kustomize for your own environments; they can be combined.
Q: A Pod is CrashLoopBackOff. Walk me through it.
kubectl describe pod for Last State (exit code, OOMKilled) and events, kubectl logs --previous for the last crash. Exit 1: application or config error. 137 with OOMKilled: memory. 137 with a failing liveness probe: probe timing, add a startup probe.
Q: A Service has no endpoints. Why?
The selector matches no Pods, the Pods are not Ready, or the Pods are in another namespace. Check kubectl get endpointslices, the Service selector against --show-labels, and readiness events.

Chapter Thirteen - Summary

Cheat Sheet and Recap

Cheat sheet

Topic	Rule	Detail (Kubernetes 1.37 unless stated)
Probes	Startup gates, liveness restarts, readiness gates traffic	Defaults: period 10 s, timeout 1 s, failureThreshold 3, successThreshold 1
Actuator	Probe the health groups	`/actuator/health/liveness`, `/actuator/health/readiness` (Spring Boot 4.1 docs)
Memory	Request = limit; heap 60-75% of limit	JDK default `MaxRAMPercentage` is 25% (measured with JDK 21)
CPU	Request yes; limit with care	Throttling versus kill: CPU compressible, memory not
QoS	Guaranteed > Burstable > BestEffort	Guaranteed needs CPU and memory requests = limits
HPA	`ceil(current x metric / target)`	Tolerance 0.1; needs metrics-server and requests; scale to zero beta in 1.37
Rolling update	`maxUnavailable: 0`, `maxSurge: 1` for no capacity loss	No automatic rollback: watch and `rollout undo`
PDB	`policy/v1`; voluntary evictions only	`replicas: 1` + `minAvailable: 1` blocks drains
Shutdown	preStop sleep, `server.shutdown=graceful`, grace > sum	Spring timeout default 30 s; grace default 30 s
Scheduling	Spread with topology constraints; taints repel; affinity attracts	Prefer soft rules on small clusters
RBAC	Own ServiceAccount, Role not ClusterRole, `resourceNames`	`automountServiceAccountToken: false` when unused
NetworkPolicy	Default deny + allow list	Requires an enforcing CNI
Pod Security	privileged, baseline, restricted	Labels `pod-security.kubernetes.io/enforce\|audit\|warn`
Debug	`describe`, `logs --previous`, events, endpointslices	Exit 137 + OOMKilled = memory; ErrImagePull = image or credentials

∑ Kubernetes operations - what to remember

Probes encode intent: liveness for "restart me", readiness for "send me traffic", startup for "give me time". Never put shared dependencies in liveness.
Requests schedule, limits enforce. Memory over its limit is killed (137); CPU over its limit is only throttled.
A JVM needs headroom beyond the heap. Use MaxRAMPercentage below 100 and measure, never -Xmx equal to the limit.
Zero downtime is a recipe, not a setting: readiness, maxUnavailable: 0, preStop sleep, graceful shutdown, a long enough grace period and a PDB.
HPA scales on ratios of an observed metric to a target and needs metrics-server and requests; VPA, KEDA and node autoscalers do different jobs.
Layer the defences: namespace Pod Security, NetworkPolicy (with an enforcing CNI), RBAC with a dedicated ServiceAccount, and Secrets that do not live in Git.
Troubleshoot from the Events and the previous logs, then walk the decision tree for the status you see.

Version note

Checked on 2026-10-31. Kubernetes: the 1.37 documentation (probes, QoS, resource management, HPA including the formula, tolerance and scale to zero beta, disruptions, topology spread, taints, network policies, Pod Security Standards, lifecycle hooks and the sleep action, in-place resize stable since 1.35) and a live kind v0.33.0 cluster with Kubernetes v1.37.0 and kubectl v1.37.2, on which every manifest here passed kubectl apply --dry-run=server and the outputs marked real were captured. Helm v4.3.1 linted and rendered the chart; kubectl kustomize rendered the overlay. Spring Boot: the 4.1 reference and properties pages (health groups, probes default true, graceful shutdown, timeout-per-shutdown-phase 30s). JVM numbers were measured with eclipse-temurin:21-jre under Docker. Not verified live: HPA behaviour (the test cluster had no metrics-server), NetworkPolicy enforcement, VPA and KEDA beyond their project pages, and the EKS notes beyond the existence of the cited EKS guide pages (qualitative, check current EKS docs).

Next in this section

05 CI/CD Pipelines - getting manifests and images to the cluster and rolling back automatically.
07 Observability - metrics for throttling, memory, probes and alerts.
08 Deployment Strategies - blue/green, canary and feature flags on top of the rolling update.
03 Kubernetes Core - the object model this page builds on.

← 03 · Kubernetes Core
DevOps map
05 · CI/CD Pipelines →

LearningTree

Kubernetes Operations | DevOps