p50 of GET /orders/{id} is 20+ ms because every call goes to the database, and during a promotion the same 1,000 popular orders account for most traffic. The database is the shared bottleneck (page 08).

Traffic is highly skewed (a few keys get most reads). A small in-process cache holding the hot keys can absorb most reads, and the database sees only the misses. Expected: effective latency = h × t_cache + (1 − h) × t_db.

Measure the hit ratio, not the cache size: Micrometer exposes cache.gets{result=hit|miss}, cache.puts, cache.evictions for Caffeine/Spring caches (enable recordStats()); Redis exposes keyspace_hits / keyspace_misses (ElastiCache: CacheHitRate). Offline, simulate your own key distribution as we do in chapter 3.

Chapters 3 to 5 show what the data says: the formula holds when the backend is not saturated; hit ratio saturates with cache size (long tail of cold keys); and a naive cache fails in two famous ways (stampede, synchronised expiry).

One variable at a time: cache size, then single-flight, then TTL jitter.

Backend calls and latency percentiles before/after, each table from a real run.

We need to decide how big a cache to buy (memory per pod) and what effect to expect. "Cache more" is not an answer.

(a) The average latency follows h · t_cache + (1 − h) · t_db. (b) With Zipf-distributed keys, hit ratio grows quickly with the first few thousand entries and then flattens. (c) If the backend is saturated, a cache gains more than the formula says because it also removes queueing.

CacheLab ratio (chapter 10): 100,000 keys, Zipf exponent 1.0, 200 virtual-thread callers, a backend that sleeps 20 ms and admits a configurable number of concurrent calls. Caffeine maximumSize is the only variable between rows.

Measured; the ~19 % of misses at 100,000 keys are first-touch (compulsory) misses of 200,000 requests over 100,000 keys.

Dots measured; dashed lines computed from the formula with example t_cache values.

The measured average matches the formula to within a few percent in every row (for instance 8.48 ms measured vs 8.39 ms formula at 58 % hit ratio): the model is right when the backend latency does not change with load.

(Table computed from the formula with t_db = 20 ms; the 0.5 ms remote-cache cost is an illustrative same-AZ figure, not measured.) Note the last rows: past about 95 % hit ratio the average barely moves while the tail is exactly the miss latency - the p99 of a service with 99 % hit ratio is a database call plus whatever queueing misses suffer.

Pick maximumSize from the measured hit-ratio curve: the knee (5,000 keys here) is where each extra entry stops paying. Memory = entries × average entry size (measure with a heap histogram, see JVM Memory & GC).

Measured: same cache sizes, backend limited to 50 concurrent calls vs 1,000.

With the backend saturated, "no cache" costs 84 ms for a 20 ms service time (200 callers / 50 slots = 4 waves of queueing). A 5,000-entry cache reduces that to 30.5 ms. The formula would predict 7.6 ms: it is wrong here because t_db itself was a function of load. The cache removed 62 % of the calls and the queue they formed. This is why caching in front of a database that is hot looks magical, and why you must re-measure the database after the cache goes live rather than assume the number from a quiet benchmark.

Every 60 s the dashboard shows a latency spike and a burst of 500 identical queries for the hottest order, exactly when its TTL expires. The database connection pool saturates for a moment and unrelated requests fail.

500 concurrent readers miss simultaneously and each calls the database. A per-key "single-flight" - the first caller loads, the rest wait for its result - would turn 500 calls into 1.

CacheLab stampede releases 500 virtual threads together on an empty cache for key 7 and counts backend calls, with three strategies. Backend: 20 ms per call, 50 concurrent. Also run with 100 ms (a slow database) to see the effect grow.

Measured.

Naive: 500 backend calls for one key; with 20 ms calls and 50 slots the last reader waits 203 ms, and with a 100 ms backend it waits over a second - and in production those 500 calls would also hold 500 pooled connections. Both single-flight variants make exactly 1 call.

Illustration of the in-flight map used in our CacheLab (a ConcurrentHashMap of CompletableFuture).

Two companions that remove the cause: refresh before expiry so readers never see a miss (refreshAfterWrite in Caffeine serves the old value while one thread reloads asynchronously - the in-process version of HTTP "stale-while-revalidate"), and probabilistic early expiration for distributed caches (each reader may refresh a bit early, with probability rising as expiry nears).

Note on the Caffeine row: its p50 printed as 0.0 ms. We believe this is an artefact of the test, not a feature: Caffeine runs the loader inside ConcurrentHashMap.compute, which holds a monitor, and on JDK 21 a virtual thread blocking inside synchronized pins its carrier; most of the 500 test threads could not even start timing until the load had finished. Read only the call count and p99 for that row. (The pinning discussion is on the Concurrency page; JEP 491 in JDK 24 removes this pinning.)

Database CPU shows a sawtooth with exactly the cache TTL as its period: calm, then a 2,000-query burst, then calm again.

Keys filled at the same instant share an expiry instant. Randomising each TTL by ±20 % ("jitter") spreads expiries over a window instead of a spike.

CacheLab jitter: 2,000 keys, each read every ~50 ms by its own virtual thread, hand-rolled TTL cache, reload = 20 ms. Count backend loads per 250 ms bucket for 6 s. Only the jitter factor differs between runs.

Measured.

Without jitter the database sees 2,000 loads in a single 250 ms bucket, every 2 s, and nothing in between. With ±20 % jitter the first wave (cold start - unavoidable) is the same, but afterwards the worst bucket is 638 loads: 3.1x lower, and the waves widen and eventually blur as the keys desynchronise.

(Caffeine expireAfterWrite is one duration per cache; for per-entry TTLs use Expiry or add the jitter to the stored deadline. Spring's RedisCacheManager takes a TTL per cache, so jitter needs a custom RedisCacheWriter / TTL function or a custom wrapper.)

Cold start needs different medicine: warm gradually, or keep serving stale while refreshing (refreshAfterWrite). Jitter only fixes repeating synchronisation.

Per-pod Caffeine caches give a 70 % hit ratio but each of 20 pods warms separately and an eviction in one pod is invisible to the others. Product wants a shared cache so that a deploy does not zero the hit ratio.

A shared Redis layer (L2) behind the Caffeine L1 keeps hit ratio through deploys and gives one place to invalidate, with an added cost per L2 hit of one network round trip plus (de)serialization.

Measure (a) the serialization cost here with SerLab (chapter 10), and (b) in your VPC, the real Redis round trip: redis-cli --latency -h <endpoint> and the ElastiCache CloudWatch metrics CacheHitRate, Evictions, EngineCPUUtilization, CurrConnections, DatabaseMemoryUsagePercentage, ReplicationLag.

Measured with Jackson 2.17.2 and java.io serialization on a shared machine.

JSON is smaller and faster than Java serialization here, and is readable and language-neutral. Java serialization is also a security liability (deserialisation gadgets) - do not use it for cache values. For hot paths consider a compact binary format (protobuf, MessagePack) - not measured here.

Illustrative topology; behaviour per Redis cluster documentation.

Operational settings: client pool sized like any pool (Lettuce shares one connection per thread-safe client by default; set timeouts), maxmemory and an eviction policy (allkeys-lru/allkeys-lfu for a pure cache; Redis OSS defaults to noeviction and ElastiCache's default parameter group differs - check yours), and SCAN instead of KEYS.

Verify with the metrics above that: L1+L2 combined hit ratio holds across a rolling deploy; EngineCPUUtilization stays well under 100 % on every shard (a single hot key shows as one shard pinned while others idle); p99 for an L2 hit stays near your measured round trip. None of this could be measured in this sandbox.

Support reports orders showing "PAYMENT_PENDING" for ten minutes after payment succeeded. The database is right; the cache is wrong.

Either the invalidation on payment never ran for this path (another service wrote the row), or it ran and then a racing reader repopulated the old value.

Reproduce with a log of cache operations per key (ids, version/updatedAt, source) and compare to the DB row's updated_at. Log the version in the cached value and expose it in a debug endpoint.

Mechanism diagram. Fully reproducible with two threads and a sleep between read and fill.

Reader A read the old row, writer B committed and deleted the key, then A filled the cache with the old value. The delete came before the stale fill. Any cache-aside read-then-write sequence has this window.

Rules that prevent most trouble: evict after commit, not before; evict by delete; keep a TTL even when you evict; make cached values carry a version so you can detect staleness; and treat "another service writes this table" as an invalidation source you must subscribe to.

Write a test that interleaves reader and writer with explicit latches (the race above), assert the final cache value equals the DB value, and run it in CI. Watch the cache hit ratio and a "stale reads" counter (compare cached version to DB version on a sample of reads).

Requests for IDs that do not exist (bots scanning /orders/1..N, deleted orders) always miss and always hit the database. Cache the absence for a short TTL (for example 30-60 s) - a sentinel such as Optional.empty() in Caffeine, or disableCachingNullValues() turned off with a short TTL in Spring Data Redis - and cap the key space (a Bloom filter in front of the cache is the next step). Keep negative TTLs short: a just-created order must not be "not found" for a minute; evict the key on create.