Performance · 12

AWS Compute, Containers & the JVM

Why does a Java service that is fine on a laptop get OOMKilled with a half-empty heap, or show 79 ms p99 stalls at 49% average CPU? Because inside EC2, Fargate, EKS and Lambda the JVM does not own the machine: a cgroup limits its memory and CPU, and the JVM sizes its heap, GC and thread pools from what it reads there. This page measures that boundary with real Docker experiments (limits, NMT, an OOM kill, CFS throttling) and explains Graviton, Lambda cold starts and ALB tuning, labelling what could not be run here.

Container experiments

AWS compute models

79 ms

Throttled p99 (measured)

137

OOMKilled exit code

Hub · 01 Measure First · 02 Benchmarking & Load Testing · 03 JVM Memory & GC · 04 JIT, Startup & Native · 05 Profiling Toolbox · 06 Concurrency & Virtual Threads · 07 Spring, HTTP & Network · 08 Database Performance · 09 Caching · 10 Distributed Systems & Messaging · 11 Capacity & Failure · 12 AWS Compute & the JVM · 13 Scaling & Observability · 14 Cost & Capstone · 15 Scenario Q&A

Chapter One - The Problem

Restarts and Spikes Your Profiler Cannot See

OrderService runs on AWS: tasks on ECS Fargate (or pods on EKS) behind an Application Load Balancer, a database behind it. After the code-level fixes from earlier pages two production symptoms remain, and neither is visible in a profiler on your laptop.

Lab step	What happened
1. Problem	(a) Tasks restart with exit code 137 although `jvm.memory.used` for the heap never passes 50%. (b) p99 shows spikes of 80-90 ms while the CPU graph reads 30-50%.
2. Hypothesis	(a) The container limit covers more than the heap, and something off-heap grows. (b) A CPU limit enforces a quota per 100 ms period, and a burst of runnable threads exhausts it.
3. Measurement	Run the JVM in a container with the same limits and observe: `-XX:+PrintFlagsFinal`, Native Memory Tracking, `docker inspect` (OOMKilled), the cgroup file `cpu.stat`.
4. Diagnosis	Chapters 4-7: the JVM derives heap, GC and pools from cgroup values; memory is a sum of regions; throttling freezes every thread at once.
5. Optimization	Size heap as a fraction of the limit, cap off-heap explicitly, decide CPU requests vs limits deliberately, set processor-dependent pools consciously.
6. Verification	Re-run the same container experiment: exit code, `nr_throttled`, probe p99.

Container memory is the sum of regions, not the heap

Key insight: The container limit is the size of a shipping container, the heap is one pallet in it. Metaspace, thread stacks, direct buffers and the code cache are the other pallets. Loading the heap pallet to the brim does not leave room for the rest, and the port authority (the kernel) does not unload anything: it destroys the container.

⚠ Common bugs & edge cases:

Reading "heap used 50%" as "memory is fine". The kernel enforces the cgroup total, not the heap.
Looking at average CPU for latency problems on a CPU-limited container: the throttled stall is invisible in a 1-minute average.

Chapter Two - Mental Model

Cgroups, Limits and What the JVM Reads

A container is a set of kernel cgroup limits plus namespaces. The JVM (with UseContainerSupport on by default) reads those files at startup and derives its configuration. Understanding that one-way flow explains most container-JVM surprises.

Signal	What it means
`memory.max` (the limit)	The JVM sets the default max heap to `MaxRAMPercentage` (default 25%) of it. Cgroup v2 support arrived in JDK 15. Exceeding the limit anywhere triggers the kernel OOM killer: exit code 137 (128 + SIGKILL).
`cpu.max` (quota / period)	A limit of 2 CPUs is `200000 100000`: 200 ms of CPU per 100 ms period, across all threads. The JVM reports ceil(quota/period) as `availableProcessors()`. When the quota is used up the whole cgroup is frozen until the next period (CFS throttling).
CPU request / shares	A request only sets relative weight under contention; it does not cap. In Kubernetes a limit is optional. Without a limit the JVM sees every node CPU and sizes pools for the node.
Ergonomics	From processors and memory the JVM chooses the collector (Serial vs G1), GC thread counts, compiler threads and the ForkJoin common pool.

How the JVM reads the container

Requests vs limits, side by side

	Memory	CPU
Request (soft)	Scheduler reservation; no runtime cap	Relative weight under contention; no cap
Limit (hard)	Enforced: exceed it and the kernel kills the process	Enforced as a quota: exceed it and threads are throttled
JVM reads	The limit	The limit (quota/period). Not the request (JDK 19+ ignores CPU shares)
Failure mode	OOMKilled / exit 137, abrupt, no stack trace	Latency stalls with low average CPU

⚠ Common bugs & edge cases:

On Fargate the task-level CPU and memory are hard limits chosen from fixed combinations; on EKS you set them per container. Either way the JVM only knows the limit it can read.
Removing a CPU limit to avoid throttling also removes the JVM's only processor-count signal: it then sizes pools for the whole node. Set -XX:ActiveProcessorCount explicitly in that case.

Chapter Three - Compute Options

EC2, Fargate, EKS and Lambda for Java

Which compute model a Java service uses decides which of this page's levers matter. General guidance, not a benchmark: none of the rows below was measured here.

Decision flow (general guidance)

	EC2 / ASG	ECS on Fargate	EKS (pods)	Lambda
You manage	OS, AMI, JVM, scaling	Task definition (CPU, memory), image	Pod spec, nodes (or Fargate/Karpenter), cluster	Function code, memory, concurrency
Billing unit	Instance-hour (or second)	vCPU and GiB per second of the task	Node (EC2) or pod (Fargate)	Requests plus GB-seconds
JVM sees	The whole host unless you containerise	Task limits via cgroup	Container limits via cgroup (or node if none)	Configured memory; CPU share scales with it
Startup cost matters	Slow-start in LB, warm-up	Task start + image pull + JVM + Spring	Pod schedule + image pull + JVM + Spring	Every cold start (init phase)
Best fit for JVM	Steady load, big heaps, host control	Steady load, little ops	Platform teams, many services	Spiky or low-volume with SnapStart
Main tuning risk	Noisy neighbours, bin packing	Wrong task size, throttling	Limits vs requests, throttling	Cold starts, memory sizing

⚠ Common bugs & edge cases:

Choosing by price per vCPU. Compare cost per request after measuring throughput on each model (see Cost & Capstone).
Assuming Lambda is cheap for steady high traffic: the duration-based price grows linearly with load while an always-on container does not.

Chapter Four - Lab 1

What the JVM Decides Inside a Container

Lab step	What happened
1. Problem	Same jar, same flags: on a laptop it uses G1 with a ~8 GiB heap; in the production task it is slow and the GC log looks different.
2. Hypothesis	The JVM chose different defaults because the container exposes different memory and CPU limits.
3. Measurement	Run `java -XX:+PrintFlagsFinal -version` and a small `Cpus.java` inside Docker containers with different `--memory` and `--cpus`. JDK 21.0.2 was mounted read-only into a stock Ubuntu-based image; the host has 12 cores and 30 GB.
4. Diagnosis	The matrix below (real output).
5. Optimization	Set `-XX:MaxRAMPercentage` (or `-Xmx`) deliberately and pick the collector explicitly if the default is wrong for the size.
6. Verification	Re-run the matrix with your flags and confirm `MaxHeapSize` and the GC flags.

Real output: the ergonomics matrix (trimmed)

### container limits: none
intx CICompilerCount = 4 {product} {ergonomic}
uint ConcGCThreads = 3 {product} {ergonomic}
size_t MaxHeapSize = 8317304832 {product} {ergonomic}
double MaxRAMPercentage = 25.000000 {product} {default}
uint ParallelGCThreads = 10 {product} {default}
size_t SoftMaxHeapSize = 8317304832 {manageable} {ergonomic}
bool UseG1GC = true {product} {ergonomic}
bool UseSerialGC = false {product} {default}
availableProcessors   = 12
maxMemory (max heap)  = 7932 MiB
commonPool parallelism = 11

### container limits: 1g, 1 cpu
intx CICompilerCount = 2 {product} {ergonomic}
uint ConcGCThreads = 0 {product} {default}
size_t MaxHeapSize = 268435456 {product} {ergonomic}
double MaxRAMPercentage = 25.000000 {product} {default}
uint ParallelGCThreads = 0 {product} {default}
size_t SoftMaxHeapSize = 268435456 {manageable} {ergonomic}
bool UseG1GC = false {product} {default}
bool UseSerialGC = true {product} {ergonomic}
availableProcessors   = 1
maxMemory (max heap)  = 247 MiB
commonPool parallelism = 1

### container limits: 2g, 2 cpus
intx CICompilerCount = 2 {product} {ergonomic}
uint ConcGCThreads = 1 {product} {ergonomic}
size_t MaxHeapSize = 536870912 {product} {ergonomic}
double MaxRAMPercentage = 25.000000 {product} {default}
uint ParallelGCThreads = 2 {product} {default}
size_t SoftMaxHeapSize = 536870912 {manageable} {ergonomic}
bool UseG1GC = true {product} {ergonomic}
bool UseSerialGC = false {product} {default}
availableProcessors   = 2
maxMemory (max heap)  = 512 MiB
commonPool parallelism = 1

### container limits: 1g, 2 cpus, MaxRAMPercentage=75
intx CICompilerCount = 2 {product} {ergonomic}
uint ConcGCThreads = 0 {product} {default}
size_t MaxHeapSize = 805306368 {product} {ergonomic}
double MaxRAMPercentage = 75.000000 {product} {command line}
uint ParallelGCThreads = 0 {product} {default}
size_t SoftMaxHeapSize = 805306368 {manageable} {ergonomic}
bool UseG1GC = false {product} {default}
bool UseSerialGC = true {product} {ergonomic}
availableProcessors   = 2
maxMemory (max heap)  = 742 MiB
commonPool parallelism = 1

### container limits: 2g, 0.5 cpu
intx CICompilerCount = 2 {product} {ergonomic}
uint ConcGCThreads = 0 {product} {default}
size_t MaxHeapSize = 536870912 {product} {ergonomic}
double MaxRAMPercentage = 25.000000 {product} {default}
uint ParallelGCThreads = 0 {product} {default}
size_t SoftMaxHeapSize = 536870912 {manageable} {ergonomic}
bool UseG1GC = false {product} {default}
bool UseSerialGC = true {product} {ergonomic}
availableProcessors   = 1
maxMemory (max heap)  = 494 MiB
commonPool parallelism = 1

Max heap the JVM chose (Runtime.maxMemory in MiB, measured)

Thread pools follow the CPU count

Real output: what the JVM read from the cgroup (-Xlog:os+container=trace, 2 GiB / 2 CPU)

[0.001s][debug][os,container] Detected cgroups v2 unified hierarchy
[0.001s][trace][os,container] Raw value for CPU quota is: 200000
[0.001s][trace][os,container] CPU Quota is: 200000
[0.001s][trace][os,container] CPU Quota count based on quota/period: 2
[0.001s][trace][os,container] OSContainer::active_processor_count: 2
[0.001s][trace][os,container] Memory Limit is: 2147483648
[0.022s][trace][os,container] Memory Limit is: 2147483648

Three facts to remember from this output. 1) The default heap is 25% of the container memory: 256 MiB in a 1 GiB container. 2) A fractional CPU limit rounds up: 0.5 CPU reports 1 processor. 3) The collector depends on the container: with fewer than 2 CPUs or less than about 1792 MiB the JVM selects the Serial collector, so a 1 GiB container with 2 CPUs and MaxRAMPercentage=75 still runs Serial GC. If that is not what you want, say -XX:+UseG1GC.

⚠ Common bugs & edge cases:

Seeing a different GC in production than on the laptop and concluding "the flags are ignored". They are not; the defaults differ. Always log the effective flags at startup (-XX:+PrintFlagsFinal or jcmd <pid> VM.flags).
Setting -Xmx and also MaxRAMPercentage: -Xmx wins; the percentage is then ignored for the maximum.
Using -XX:+UseContainerSupport on ancient JDKs (before 8u191 it did not exist). Check the version of the base image.

Chapter Five - Lab 2

Container Memory Is More Than the Heap

Lab step	What happened
1. Problem	The pod restarts with exit code 137 and the heap graph never exceeded 50%.
2. Hypothesis	Container memory = heap + metaspace + thread stacks + code cache + direct buffers + GC and malloc structures + native libraries, and some region outside the heap grows.
3. Measurement	Two real experiments. (1) NMT on the actual OrderService jar in a 1 GiB container: `-XX:NativeMemoryTracking=summary`, `jcmd 1 VM.native_memory summary`. (2) A program that allocates direct buffers in a 256 MiB container with a 64 MiB heap.
4. Diagnosis	NMT committed 416 MiB; the process RSS was 505 MiB; the heap was 300 MiB of that. In experiment (2) the heap stayed at 7 MiB while direct memory grew until the kernel killed the container.
5. Optimization	Leave 25-40% of the limit for non-heap (so `MaxRAMPercentage` 60-75, not 90), cap direct memory (`-XX:MaxDirectMemorySize`), bound thread counts, and keep NMT off in production unless diagnosing (it adds overhead).
6. Verification	The container survives load; `docker inspect` shows `OOMKilled=false`; NMT total plus margin fits the limit.

Real NMT breakdown of OrderService inside a 1 GiB container (MiB, after a short load run)

Real output: NMT summary for OrderService, 1 GiB container (trimmed)

Total: reserved=1763575KB, committed=426431KB
     malloc: 31059KB #186074
-                 Java Heap (reserved=307200KB, committed=307200KB)
-                     Class (reserved=1049774KB, committed=6510KB)
-                    Thread (reserved=46228KB, committed=4928KB)
-                      Code (reserved=250385KB, committed=28889KB)
-                        GC (reserved=1030KB, committed=1026KB)
-                  Compiler (reserved=552KB, committed=552KB)
-                  Internal (reserved=692KB, committed=692KB)
-                     Other (reserved=194KB, committed=194KB)
-                    Symbol (reserved=12620KB, committed=12620KB)
-    Native Memory Tracking (reserved=2970KB, committed=2970KB)
-        Shared class space (reserved=16384KB, committed=12952KB, readonly=0KB)
-               Arena Chunk (reserved=8651KB, committed=8651KB)
-                    Module (reserved=79KB, committed=79KB)
-                 Safepoint (reserved=8KB, committed=8KB)
-           Synchronization (reserved=939KB, committed=939KB)
-            Serviceability (reserved=17KB, committed=17KB)
-                 Metaspace (reserved=65770KB, committed=38122KB)
-      String Deduplication (reserved=1KB, committed=1KB)
-           Object Monitors (reserved=80KB, committed=80KB)

Real output: process view of the same container

VmRSS:	  516744 kB
Threads:	46
cgroup memory.current: 512241664
cgroup memory.max:     1073741824
docker stats MemUsage: 486.2MiB / 1GiB

Now the failure itself. NativeHog grows direct memory by 16 MiB every 200 ms while the heap stays tiny. With the default settings the JVM protects you: MaxDirectMemorySize defaults to the max heap, so you get a clean, catchable OutOfMemoryError.

Real output: default direct-memory cap (container 256 MiB, -Xmx64m): a clean error

direct= 16 MiB  heapUsed= 6 MiB  heapMax=61 MiB
direct= 32 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct= 48 MiB  heapUsed= 7 MiB  heapMax=61 MiB
Exception in thread "main" java.lang.OutOfMemoryError: Cannot reserve 16777216 bytes of direct buffer memory (allocated: 50339840, limit: 64880640)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178)
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111)
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360)
at NativeHog.main(NativeHog.java:10)
exit code: 1  OOMKilled: false

Libraries such as Netty manage off-heap memory on their own and many deployments raise or remove the cap. Tell the JVM it may use 1 GiB of direct memory inside a 256 MiB container and the same program is killed by the kernel with no stack trace:

Real output: -XX:MaxDirectMemorySize=1g in a 256 MiB container

direct= 16 MiB  heapUsed= 6 MiB  heapMax=61 MiB
direct= 32 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct= 48 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct= 64 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct= 80 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct= 96 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct=112 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct=128 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct=144 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct=160 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct=176 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct=192 MiB  heapUsed= 7 MiB  heapMax=61 MiB
direct=208 MiB  heapUsed= 7 MiB  heapMax=61 MiB
exit code: 137  OOMKilled: true

Off-heap growth ends in a kernel OOM kill

Container memory budget (starting point)

Region	Controlled by	Rule of thumb (verify with NMT)
Java heap	-Xmx / MaxRAMPercentage	60-75% of the limit
Metaspace + class space	MaxMetaspaceSize (unbounded by default)	Tens of MiB for a Spring app (38 + 6 MiB measured)
Code cache	ReservedCodeCacheSize	Tens of MiB (28 MiB committed measured)
Thread stacks	Xss, thread count	Count x stack size; 1 MiB reserved per platform thread by default on Linux x64 (only touched pages are resident)
Direct / NIO buffers	MaxDirectMemorySize, Netty settings	Cap explicitly if you use Netty or large NIO buffers
GC structures, malloc, libraries	Collector choice, glibc arenas (MALLOC_ARENA_MAX)	The 88 MiB gap here; measure with RSS - NMT

⚠ Common bugs & edge cases:

Diagnose with kubectl describe pod (Last State: Terminated, Reason: OOMKilled, Exit Code: 137) or the ECS stopped-task reason ("OutOfMemoryError: Container killed due to memory usage"). No Java stack trace exists for a kernel kill: check dmesg or container events.
"The heap fits" is not an argument: add -XX:+ExitOnOutOfMemoryError and -XX:+HeapDumpOnOutOfMemoryError so a real heap OOM is distinguishable from a kernel kill.
NMT shows only what the JVM tracks. RSS is larger; that gap (88 MiB here) is real memory the cgroup counts.
Page cache counts toward memory.current in cgroup v2, so a container memory graph can look higher than RSS.

Chapter Six - Lab 3

CPU Limits and CFS Throttling

Lab step	What happened
1. Problem	A latency-sensitive path shows p99 spikes of ~80 ms while the container averages under 50% of its CPU limit.
2. Hypothesis	The limit is enforced as 100 ms periods. A burst of many runnable threads (GC workers, JIT, request bursts) consumes the quota in a few ms and the kernel freezes all threads of the cgroup until the next period.
3. Measurement	`Throttle.java`: every second 8 threads each do 100 ms of CPU work at once (a stand-in for GC + JIT + a request burst) while a probe thread times a 1 ms operation every 10 ms. Read the cgroup file `/sys/fs/cgroup/cpu.stat` before and after. Four containers: A limit 1 CPU + 8-thread burst, B limit 1 CPU + 1-thread burst, C limit 4 CPUs, D no limit.
4. Diagnosis	In A, 81 of 201 periods were throttled (`nr_throttled`) and the 1 ms probe saw p99 = 78.8 ms with only 49% average CPU. B, C and D were not throttled for the probe (p99 about 1.3-1.7 ms).
5. Optimization	ONE variable: the limit. Raise it (A vs C), or keep requests and drop the limit, or reduce simultaneous threads.
6. Verification	Probe p99 falls from 78.8 ms (A) to 1.7 ms with 4 CPUs (C) and 1.3 ms with no limit (D); `nr_throttled` goes 81, 36, 0. C was still throttled in some bursts (max probe 45.8 ms) because 8 threads x 100 ms can exceed even a 400 ms quota.

Real output: exp3c (docker run --cpus=... Throttle.java 8 100 20)

### A: --cpus=1, 8 threads x 100 ms burst once per second
availableProcessors=1 unitsPerMs=707560
probe n=1303 p50=0.8ms p99=78.8ms max=89.9ms  probes>20ms=37
cpu.stat: avg CPU used = 0.49 cores | nr_periods=201 nr_throttled=81 throttled_time=56679 ms

### B: --cpus=1, 1 thread x 100 ms burst (fits in the quota)
availableProcessors=1 unitsPerMs=903185
probe n=1808 p50=0.9ms p99=1.3ms max=4.6ms  probes>20ms=0
cpu.stat: avg CPU used = 0.18 cores | nr_periods=200 nr_throttled=0 throttled_time=0 ms

### C: --cpus=4, 8 threads x 100 ms burst
availableProcessors=4 unitsPerMs=904814
probe n=1703 p50=1.0ms p99=1.7ms max=45.8ms  probes>20ms=11
cpu.stat: avg CPU used = 0.86 cores | nr_periods=200 nr_throttled=36 throttled_time=11309 ms

### D: no CPU limit (12 cores), 8 threads x 100 ms burst
availableProcessors=12 unitsPerMs=865849
probe n=1816 p50=0.8ms p99=1.3ms max=1.7ms  probes>20ms=0
cpu.stat: avg CPU used = 0.88 cores | nr_periods=0 nr_throttled=0 throttled_time=0 ms

Probe p99 latency of a 1 ms operation (ms, measured)

Throttling timeline (mechanism)

Kubernetes / ECS signals

Where	Metric / command	Meaning
Kubernetes (cAdvisor)	`container_cpu_cfs_throttled_periods_total` / `container_cpu_cfs_periods_total`	Fraction of periods in which the container was throttled
Inside any container	`cat /sys/fs/cgroup/cpu.stat` (cgroup v2): nr_periods, nr_throttled	Same counters, per container
ECS / Fargate	Container Insights CPU metrics and task CPU sizing	Check which throttling indicators your platform exposes; do not assume

⚠ Common bugs & edge cases:

The raw throttled_time in cpu.stat adds up across CPUs (21 s over a 20 s run in an earlier 8 x 50 ms test). Use the throttled-period ratio and a latency probe.
A limit of 1 CPU does not mean "one thread". The JVM has GC threads, JIT threads and your pools all competing for the same 100 ms quota.
Lowering thread counts beats raising limits when the burst is self-inflicted (a big parallel stream, an oversized pool).
Do not read this as "never set CPU limits": limits give predictability and protect neighbours. Decide with the throttle ratio and latency tail in hand.

Chapter Seven - Lab 4

Active Processor Count, GC Threads and Pools

Lab step	What happened
1. Problem	GC pauses are longer than expected on a small task and `parallelStream()` barely parallelises.
2. Hypothesis	Both are sized from `availableProcessors()`, which is tiny in the container.
3. Measurement	`Cpus.java` and PrintFlagsFinal in several containers (table below, real).
4. Diagnosis	With 2 CPUs: ParallelGCThreads = 2, ConcGCThreads = 1, the common pool has parallelism 1. With 12 CPUs: ParallelGCThreads = 10, ConcGCThreads = 3, common pool 11.
5. Optimization	Where it matters, size explicitly: `-XX:ActiveProcessorCount` (override), `-XX:ParallelGCThreads`, `-Djava.util.concurrent.ForkJoinPool.common.parallelism`, and your own executors.
6. Verification	Re-run Cpus.java with the flag and confirm the numbers.

Real output: override the processor count

availableProcessors     = 3
maxMemory (max heap)    = 7932 MiB
commonPool parallelism  = 2
(java -XX:ActiveProcessorCount=3 Cpus.java on the 12-core host)

Real output: JAVA_TOOL_OPTIONS (applied by the JVM at startup, announced on stderr)

Picked up JAVA_TOOL_OPTIONS: -XX:MaxRAMPercentage=75 -XX:+ExitOnOutOfMemoryError
   bool ExitOnOutOfMemoryError                   = true                                      {product} {environment}
 double MaxRAMPercentage                         = 75.000000                                 {product} {environment}
openjdk version "21.0.2" 2024-01-16 LTS

JAVA_TOOL_OPTIONS is the standard way to inject flags when you cannot edit the command line (ECS task definitions, Kubernetes env, base images). The JVM prints Picked up JAVA_TOOL_OPTIONS: ... so you can verify it in the log; it also applies to every JVM launched in the container (including jcmd and build tools), which is a frequent surprise.

The OrderService capstone showed the same sizing effect indirectly: pinning to 4 cores (Cost & Capstone) changed the processor count the JVM saw, and the application still served the load.

⚠ Common bugs & edge cases:

A common pool of parallelism 1 in a small container means parallelStream() and default CompletableFuture async work do not run in parallel. Never put blocking IO on the common pool in any size container.
JAVA_TOOL_OPTIONS affects jcmd too: a diagnostic JVM started inside the container inherits -Xmx; this can fail with out-of-memory in a small container.
Do not copy GC thread counts from a big host into a small task definition.

Chapter Eight - Graviton

arm64 for Java: Price/Performance, Not Magic

Graviton processors are AWS's arm64 CPUs. For a JVM service the move is mostly an image and build question, not a code change, and the win (if any) is price/performance. Nothing in this chapter was measured here: the sandbox host is x86_64 and has no AWS access, so figures are illustrative.

Cost per request: x86 vs arm64 (ILLUSTRATIVE - not measured in this sandbox)

Lab step	What happened
1. Problem	Compute is the largest line on the bill and the service is already tuned.
2. Hypothesis	Same task size on arm64 costs less per hour and performs about the same or better, lowering cost per request.
3. Measurement	Build a multi-architecture image and rerun the same load test on both architectures; compare RPS, p99 and CPU.
4. Diagnosis	Look for: native dependencies (JNI libraries, agents, APM) without arm64 builds; base images without arm64 variants; JIT and vector differences that move hot paths.
5. Optimization	Change ONE variable (architecture), keep the task size, JVM flags and data identical.
6. Verification	Cost per request = hourly price / (RPS x 3600) for each side. Adopt only if latency SLOs hold.

How-to (not run here): multi-arch image and ECS runtime platform

docker buildx build --platform linux/amd64,linux/arm64 -t myrepo/orderservice:1.0 --push .
# ECS task definition: runtimePlatform { cpuArchitecture: "ARM64", operatingSystemFamily: "LINUX" }
# EKS: schedule on arm64 nodes with a nodeSelector for kubernetes.io/arch=arm64
# Verify at runtime:
java -XX:+PrintFlagsFinal -version | grep -i UseSVE   # (arm64 specific flags differ by JDK)
uname -m                                              # aarch64 vs x86_64

⚠ Common bugs & edge cases:

A multi-arch manifest that is missing the arm64 variant of a base image fails at pull time, not build time. Test deploys on a canary.
JNI-based libraries (compression, crypto, some DB drivers) must ship an arm64 binary. Run the full test-suite on arm64, not just a smoke test.
Do not assume a faster or slower result: load-test. Price/performance is a measured ratio.

Chapter Nine - Lambda

Cold Starts, SnapStart and the Memory-CPU Coupling

Lambda changes the performance model: you rent execution time in memory-sized slices, and an idle function is frozen. For a JVM the dominant issue is the cold start. Not run here: Lambda needs an AWS account; the diagrams are mechanisms and shapes, labelled illustrative.

Cold start anatomy (ILLUSTRATIVE shape, not measured)

Lab step	What happened
1. Problem	The first request after idle or a scale-up takes seconds; later requests take milliseconds.
2. Hypothesis	Cold starts (init phase) dominate the tail, mostly framework startup and class loading, with the JIT cold on top.
3. Measurement	Separate init duration from invocation duration (the `REPORT` log line shows `Init Duration` on cold starts); use X-Ray or Lambda Insights; measure p50/p99 split by cold/warm.
4. Diagnosis	Which of: runtime start, class loading, framework wiring, dependency initialisation (connections, clients), first-call JIT.
5. Optimization	ONE lever at a time: SnapStart (for supported Java runtimes; check current list and constraints), provisioned concurrency, lighter framework, smaller memory footprint, AppCDS-style class sharing, native image (an option, not automatic).
6. Verification	Re-measure cold-start p99 and cost per 1M invocations.

Memory sets CPU, so memory is a performance knob

Lambda allocates CPU in proportion to the configured memory (documented as roughly one vCPU at about 1,769 MB - verify in the current docs). Doubling memory can halve duration for a CPU-bound function, leaving the GB-second cost similar or lower. The cheapest setting is usually in the middle of the range; the open-source AWS Lambda Power Tuning state machine measures it for your function.

Duration and cost vs memory (ILLUSTRATIVE - not measured in this sandbox)

Option	What it does	Trade-off
SnapStart	Restores a snapshot of the initialised execution environment	Supported Java runtimes only; re-establish connections and regenerate unique values after restore; check current constraints
Provisioned concurrency	Keeps N environments initialised	Pay for the provisioned capacity even when idle
More memory	More CPU: faster init and handler	Higher price per ms; often net neutral for CPU-bound code
Lazy / lean init	Defer heavy beans, drop unused dependencies	First use of a lazy path is slower
Native image	Starts in milliseconds, small footprint	Build complexity, peak throughput usually not better (see JIT, Startup & Native)

How-to (not run here): enable SnapStart and provisioned concurrency

aws lambda update-function-configuration --function-name orderservice --snap-start ApplyOn=PublishedVersions
aws lambda publish-version --function-name orderservice
aws lambda put-provisioned-concurrency-config --function-name orderservice --qualifier 1 --provisioned-concurrent-executions 5

⚠ Common bugs & edge cases:

Measuring Lambda performance without separating cold from warm invocations. A good average hides a terrible cold-start tail.
SnapStart restores state: caches, random seeds, open connections and cached DNS from init time can be stale or duplicated after restore.
Treating Lambda as unlimited: account and regional concurrency limits and downstream connection limits (a database does not scale with Lambda) cause throttling and errors.

Chapter Ten - ALB

Target Tuning and Health Checks

The load balancer sits between users and your tasks, with its own timers and health logic. Misaligned timers produce intermittent errors that look like application bugs. Illustrative: no ALB was available here; values are the ones to check, not guarantees.

ALB timers and target group behaviour (ILLUSTRATIVE - not measured in this sandbox)

Lab step	What happened
1. Problem	Sporadic 502 responses and tasks that get killed during traffic spikes.
2. Hypothesis	(a) The app closes idle keep-alive connections earlier than the ALB expects. (b) The health check is too deep or too slow and marks busy but healthy tasks unhealthy.
3. Measurement	ALB access logs (`elb_status_code` 502 with `target_status_code` of "-"), the target response time metric, target group healthy host count, ECS service events.
4. Diagnosis	502 with no target status and no app log line means the connection was closed or reset before a response: compare the app keep-alive timeout with the ALB idle timeout (60 seconds by default).
5. Optimization	ONE change each: app keep-alive timeout above the ALB idle timeout (Spring Boot: `server.tomcat.keep-alive-timeout`); shallow, fast health endpoint; set ECS `healthCheckGracePeriodSeconds` so slow-starting JVMs are not killed during boot.
6. Verification	The 502 count returns to zero over a day of traffic; healthy host count stays stable through a load test.

Health check death spiral (ILLUSTRATIVE - not measured in this sandbox)

Setting	Why it matters for Java	Direction
Idle timeout (ALB)	Must be shorter than the app keep-alive timeout	App keep-alive > ALB idle
Health check path	A deep check (DB, downstream) turns a slow dependency into a full outage	Shallow: process up and accepting requests
Health check timeout / thresholds	A GC pause or burst must not fail enough checks to deregister the task	Tolerate transient slowness
Deregistration delay	In-flight requests drain before the task stops; with a graceful shutdown (`server.shutdown=graceful`) none are cut	Align with request duration and the stop timeout
Slow start	Ramp traffic to a new target so a cold JIT is not hit with full load	Enable for JVM services
Routing algorithm	Least outstanding requests can balance uneven request costs better than round robin	Test with your latency distribution

⚠ Common bugs & edge cases:

502 vs 504: 502 means a bad or closed upstream connection, 504 means the target did not answer within the timeout. They have different causes and different fixes.
A health check that exercises the database makes every dependency blip look like an instance failure. Separate liveness from readiness in Kubernetes the same way.
Ramping traffic to cold JVMs: combine slow start with a warm-up in the readiness probe so the first real request is not the one that triggers JIT.

Chapter Eleven - Try It Yourself

Reproduce the Container Experiments

Everything measured on this page is reproducible with Docker and a JDK. The JDK 21.0.2 install on the host was mounted read-only into a stock Ubuntu-based image so no JDK image had to be pulled. The three Java programs below are plain Java with no dependencies.

Java - Cpus.java (what does the JVM think it has?)

import java.util.concurrent.ForkJoinPool;

public class Cpus {
  public static void main(String[] a) {
      Runtime rt = Runtime.getRuntime();
      System.out.println("availableProcessors     = " + rt.availableProcessors());
      System.out.println("maxMemory (max heap)    = " + rt.maxMemory() / (1024 * 1024) + " MiB");
      System.out.println("commonPool parallelism  = " + ForkJoinPool.commonPool().getParallelism());
  }
}

Java - NativeHog.java (heap tiny, off-heap growing)

import java.nio.ByteBuffer;
import java.util.*;

/** The heap stays tiny; off-heap (direct) memory grows 16 MiB every 200 ms. */
public class NativeHog {
  public static void main(String[] a) throws Exception {
      List<ByteBuffer> keep = new ArrayList<>();
      Runtime rt = Runtime.getRuntime();
      for (int i = 1; i <= 40; i++) {
          ByteBuffer b = ByteBuffer.allocateDirect(16 * 1024 * 1024);
          for (int j = 0; j < b.capacity(); j += 4096) b.put(j, (byte) 1);   // touch pages so they are really resident
          keep.add(b);
          System.out.printf("direct=%3d MiB  heapUsed=%2d MiB  heapMax=%d MiB%n", i * 16,
              (rt.totalMemory() - rt.freeMemory()) / (1024 * 1024), rt.maxMemory() / (1024 * 1024));
          Thread.sleep(200);
      }
  }
}

Java - Throttle.java (CFS throttling probe)

import java.nio.file.*;
import java.util.*;

/** CFS throttling demo. args: burstThreads burstMs seconds
*  Every second, burstThreads threads each do burstMs of CPU work at once (think: GC + JIT + a batch of requests).
*  A probe thread measures the wall time of a ~1 ms unit of work every 10 ms (think: a latency-sensitive request). */
public class Throttle {
  static volatile boolean stop;
  static volatile long sink;
  static long unitsPerMs;

  static void work(long units) { long x = 1; for (long i = 0; i < units; i++) x = x * 31 + i; sink += x; }

  static Map<String, Long> cpuStat() {
      Map<String, Long> m = new HashMap<>();
      try { for (String l : Files.readAllLines(Path.of("/sys/fs/cgroup/cpu.stat"))) { String[] f = l.split(" "); m.put(f[0], Long.parseLong(f[1])); } }
      catch (Exception e) { }
      return m;
  }

  public static void main(String[] a) throws Exception {
      int bt = Integer.parseInt(a[0]), bms = Integer.parseInt(a[1]), secs = Integer.parseInt(a[2]);
      for (int r = 0; r < 5; r++) { long t = System.nanoTime(); work(20_000_000L); unitsPerMs = (long) (20_000_000L / ((System.nanoTime() - t) / 1e6)); }
      System.out.println("availableProcessors=" + Runtime.getRuntime().availableProcessors() + " unitsPerMs=" + unitsPerMs);

      Thread burster = Thread.ofPlatform().start(() -> {
          while (!stop) {
              List<Thread> ts = new ArrayList<>();
              for (int i = 0; i < bt; i++) ts.add(Thread.ofPlatform().start(() -> work(bms * unitsPerMs)));
              try { for (Thread t : ts) t.join(); Thread.sleep(1000 - bms); } catch (InterruptedException e) { return; }
          }
      });

      List<Double> lat = new ArrayList<>();
      Thread.sleep(3000);                                   // warm-up, not recorded
      Map<String, Long> s0 = cpuStat(); long t0 = System.nanoTime();
      while (System.nanoTime() - t0 < secs * 1_000_000_000L) {
          long t = System.nanoTime();
          work(unitsPerMs);                                  // ~1 ms of CPU
          lat.add((System.nanoTime() - t) / 1e6);
          Thread.sleep(10);
      }
      stop = true; burster.interrupt();
      Map<String, Long> s1 = cpuStat(); double el = (System.nanoTime() - t0) / 1e9;
      Collections.sort(lat);
      int n = lat.size(); long over = lat.stream().filter(x -> x > 20).count();
      System.out.printf("probe n=%d p50=%.1fms p99=%.1fms max=%.1fms  probes>20ms=%d%n", n, lat.get(n / 2), lat.get((int) (n * 0.99)), lat.get(n - 1), over);
      if (!s0.isEmpty()) {
          double cpu = (s1.get("usage_usec") - s0.get("usage_usec")) / 1e6 / el;
          System.out.printf("cpu.stat: avg CPU used = %.2f cores | nr_periods=%d nr_throttled=%d throttled_time=%d ms%n", cpu,
              s1.get("nr_periods") - s0.get("nr_periods"), s1.get("nr_throttled") - s0.get("nr_throttled"),
              (s1.get("throttled_usec") - s0.get("throttled_usec")) / 1000);
      }
  }
}

The exact commands (JDK mounted at /jdk, sources at /w)

J=$HOME/.sdkman/candidates/java/21.0.2-tem      # any JDK 21 directory
IMG=tomcat:9.0-jdk17-temurin                   # any glibc-based image works; the JDK comes from the mount
DR="docker run --rm -v $J:/jdk:ro -v $PWD:/w:ro"

# 1. ergonomics
$DR --memory=1g --cpus=1 $IMG /jdk/bin/java -XX:+PrintFlagsFinal -version | grep -E 'MaxHeapSize|UseSerialGC|UseG1GC'
$DR --memory=2g --cpus=2 $IMG /jdk/bin/java /w/Cpus.java

# 2. off-heap OOM kill (read the verdict from the container state)
docker run --name hog --memory=256m --memory-swap=256m -v $J:/jdk:ro -v $PWD:/w:ro $IMG /jdk/bin/java -Xmx64m -XX:MaxDirectMemorySize=1g /w/NativeHog.java
docker inspect hog --format '{{.State.ExitCode}} {{.State.OOMKilled}}'

# 3. CFS throttling probe: threads, burst ms, seconds
$DR --cpus=1 $IMG /jdk/bin/java /w/Throttle.java 8 100 20

# 4. native memory breakdown of a running JVM
docker exec nmt /jdk/bin/jcmd 1 VM.native_memory summary   # started with -XX:NativeMemoryTracking=summary

Absolute numbers (heap sizes follow the host, throttling depends on the kernel scheduler) will differ on your machine; the mechanisms and the method do not. All container numbers on this page were taken on cgroup v2 with Docker 29.

⚠ Common bugs & edge cases:

Run these on a quiet machine; the throttling probe is sensitive to other CPU load.
On cgroup v1 the files are different (/sys/fs/cgroup/cpu/cpu.stat, memory.limit_in_bytes); modern JDKs handle both, but your scripts must too.

Chapter Twelve - Interview Angles

AWS and Container Questions

Why does a JVM pod get OOMKilled with a half-empty heap?

The cgroup limit covers heap plus metaspace, thread stacks, code cache, direct buffers, GC structures and libraries. In the NMT run, a 300 MiB heap sat inside a 505 MiB RSS. Lower MaxRAMPercentage, cap direct memory, and use NMT to find the growing region.

What is the default max heap in a container?

25% of the container memory (MaxRAMPercentage), so 256 MiB in 1 GiB. Set it explicitly.

Which GC does a small container get?

Serial if the container has fewer than 2 CPUs or under about 1792 MiB, even if you gave it 2 CPUs and 1 GiB. Verified here with PrintFlagsFinal.

Why low CPU average but latency spikes?

CFS throttling: the quota is per 100 ms period; a burst of threads freezes the whole cgroup. Probe p99 was 78.8 ms at 49% average CPU. Check nr_throttled.

CPU requests vs limits?

Requests set scheduling weight and do not cap; limits cap via quota. The JVM sizes itself from the limit, not the request (JDK 19+).

Is Graviton free performance?

Often better price/performance, but you must have arm64 images and native libs, and load-test it. Compare cost per request.

How do you cut Java Lambda cold starts?

Measure init vs invoke, then SnapStart for supported runtimes, provisioned concurrency, lighter framework, class-load reduction, more memory (more CPU). Native image is an option, not an automatic win.

Why sporadic 502s behind an ALB?

Keep-alive mismatch: app closes idle connections earlier than the ALB idle timeout (60 s default). Make the app timeout longer and check the access logs.

Chapter Thirteen - Summary

Cheat Sheet, Recap & Where Next

Cheat sheet

Symptom	Likely cause	Measure with	Fix
Exit 137 / OOMKilled, heap half full	Off-heap growth or heap too close to the limit	NMT, `docker inspect`, RSS	MaxRAMPercentage 60-75, cap direct memory
Latency spikes at low average CPU	CFS throttling	`cpu.stat` nr_throttled, probe p99	Raise limit, fewer simultaneous threads
Different GC / small pools in prod	Ergonomics from cgroup values	PrintFlagsFinal, Cpus.java	Set flags explicitly
Slow first request (Lambda)	Cold start	Init Duration, X-Ray	SnapStart, provisioned concurrency
Sporadic 502	Keep-alive mismatch	ALB access logs	App keep-alive > ALB idle
Tasks killed under load	Deep health check	Healthy host count, events	Shallow check, grace period

Recap

The JVM derives heap, GC choice and thread pools from cgroup values: look at the effective flags.
Container memory is heap plus non-heap; the kernel kills at the sum (exit code 137).
CPU limits are quotas per 100 ms: bursts freeze every thread; average CPU hides it.
Graviton and Lambda are measurement questions, not beliefs: build both, load-test, compare cost per request.
ALB timers and health checks must be aligned with JVM keep-alive and startup.
Illustrative here: Graviton, Lambda, ALB. Measured here: ergonomics, NMT, OOM kill, throttling.

Next steps

Previous: Capacity & Failure
Next: Scaling & Observability
Related: JVM Memory & GC · Concurrency & Virtual Threads · Spring, HTTP & Network · Cost & Capstone
Hub: Performance hub (symptom, cause, tool, fix)

LearningTree

AWS Compute, Containers & the JVM | Java Performance Engineering

AWS Compute, Containers & the JVM

Restarts and Spikes Your Profiler Cannot See

Cgroups, Limits and What the JVM Reads

EC2, Fargate, EKS and Lambda for Java

What the JVM Decides Inside a Container

Container Memory Is More Than the Heap

CPU Limits and CFS Throttling

Active Processor Count, GC Threads and Pools

arm64 for Java: Price/Performance, Not Magic

Cold Starts, SnapStart and the Memory-CPU Coupling

Target Tuning and Health Checks

Reproduce the Container Experiments

AWS and Container Questions

Cheat Sheet, Recap & Where Next