System Design · Building Blocks · CDN & Object Storage

CDN & Object Storage

Serving static assets and files at global scale.

01
Chapter One

What CDNs and Object Storage Are

Two Layers of the Static-Content Stack

A user in Sydney loads your homepage and waits 600 ms while a 2 MB hero image crosses the Pacific from your origin server in Virginia. Multiply that by every static asset, every product photo, every video thumbnail — and your origin spends most of its CPU budget pushing bytes that haven't changed in weeks. This is what CDNs and object storage exist to fix. Object storage gives you a virtually infinite, durable place to put files. A CDN puts copies of those files close to your users so the request never reaches your origin in the first place.

They are different tools that almost always work together. Object storage (S3, GCS, Azure Blob) is the durable source of truth: cheap per GB, eleven nines of durability, accessed by HTTP. CDN (CloudFront, Cloudflare, Fastly, Akamai) is a global cache layer: hundreds of edge locations near users, designed to absorb traffic before it reaches your origin. The CDN talks to object storage; users talk to the CDN; object storage rarely sees direct user traffic.

🗄️

Object Storage — The Durable Bucket

Model: flat namespace of objects (key → blob), accessed over HTTP.

Strength: 11 nines durability, near-infinite capacity, cheap per GB, no provisioning.

Examples: AWS S3, Google Cloud Storage, Azure Blob, Cloudflare R2.

Used for: images, videos, backups, data lakes, static site assets, ML training data.

🌐

CDN — The Edge Cache

Model: hundreds of edge locations (PoPs) caching content close to users.

Strength: low latency for global users, absorbs traffic spikes, offloads origin.

Examples: CloudFront, Cloudflare, Fastly, Akamai, Bunny.

Used for: static assets, video streaming, API acceleration, DDoS shielding.

CDN + Object Storage — The Standard Static Stack
Users SY LO NY CDN Edge (PoPs) ~10ms ~10ms ~10ms Sydney PoP London PoP N.Virginia PoP cache miss ~5% of requests Origin (1 region) Object Storage (S3) durable source of truth N.Virginia bucket ~95% requests served from edge origin sees only cache misses + infrequently-used assets

The geography matters more than people expect. Edge locations (also called PoPs — points of presence) are small caching servers placed in major metros. CloudFront has 600+ PoPs; Cloudflare has 300+; Akamai has 4,200+. The architecture is hierarchical: a user hits the nearest edge, which on cache miss may consult a regional “mid-tier” cache, which on miss queries your origin. The further upstream the request goes, the slower it gets and the more it costs you.

The analogy: object storage is your central warehouse. The CDN is the network of corner stores that hold the products people actually buy. Customers walk to the corner store; the corner store re-orders from the warehouse weekly. The warehouse barely sees an end customer.

Almost every modern web application uses object storage + CDN as the static-content backbone. Skipping either layer is a self-inflicted scaling problem. Storing user uploads on local disk on your application server is a 2008 mistake that still shows up in production today.

📋 Chapter 1 — Summary
  • Object storage is the durable source of truth: HTTP-accessible blobs, cheap per GB, near-infinite capacity.
  • CDN is the edge cache: hundreds of PoPs near users, absorbs traffic before it hits origin.
  • They almost always work together — CDN as cache, object storage as backing store.
  • The user's request rarely reaches your origin once the architecture is set up correctly.
02
Chapter Two

How They Work Internally

Push vs Pull — How Content Reaches the Edge

A CDN supports two ways of getting your files into its caches. Pull (lazy) is the default and the right answer for almost everything: you point the CDN at your origin; on the first request to any URL, the edge fetches from origin, caches it, and serves it. Subsequent requests within the TTL are served from edge. New files are discovered automatically. Push (eager) is the rarer model: you upload directly to the CDN's storage; you control which files exist and when they update. Push made sense for video distribution in the early 2000s; pull dominates modern web because it's self-managing and works equally well for content you didn't know you needed.

The metric that matters is cache hit ratio at the edge. A well-tuned static site CDN runs 95–99% hit ratio — the origin sees only the long-tail traffic. A poorly-tuned one drops to 50–70%, which often means too-short TTLs, varying cache keys (e.g. including useless query parameters), or content that should be cacheable but isn't marked as such. Hit ratio is a direct line to your origin egress bill.

S3 Internals — What Actually Happens Behind a PUT

Object storage looks simple from the outside — a key, a value, an HTTP API — but the engineering behind “eleven nines of durability” is substantial. A PUT to S3 doesn't complete until the object has been written to multiple devices across multiple availability zones. Reads pull from any of those copies. Background processes constantly scan for bit-rot and repair lost replicas. The flat key namespace is an illusion implemented over a partitioned, distributed key-value store; what looks like “folders” in your S3 console is just keys with slashes in them.

📦

Buckets, Keys, and the Flat Namespace

Bucket: globally-unique top-level container. One per region, eventually consistent listing.

Key: the full path-like string identifying an object. users/42/avatar.jpg has no real folders — the slashes are just characters.

Implication: “list folder” is a prefix scan; expensive on large buckets. Don't treat S3 like a filesystem.

🔀

Multipart Upload

Why: single PUT is limited (5 GB on S3); large files (videos, backups) need chunked upload.

How: initiate → upload parts (5 MB–5 GB each, in parallel) → complete with manifest of part ETags.

Resilience: failed part can be retried independently; abandoned uploads must be cleaned up (lifecycle rule).

Storage Classes — Pay for What You Access

Not all data deserves the same storage cost. Object storage providers tier their offerings based on access frequency and retrieval latency. Hot data (regularly accessed): standard storage, ~$0.023/GB-month. Infrequent access: 60% cheaper but with retrieval fees. Archive (Glacier / Archive Tier): ~$0.001/GB-month but minutes-to-hours to retrieve. Lifecycle policies move objects through these tiers automatically — e.g. logs go to standard for 30 days, infrequent for 60 days, glacier for 7 years, deleted after.

CDN Edge Cache Hit / Miss Flow
Cache Hit (~95% of requests) User in Sydney GET Edge: cached TTL valid ~10ms total no origin involved Cache Miss (~5% of requests) User in Sydney Edge: miss expired or new key Origin (S3) N.Virginia populate cache ~200ms total ~190ms is the long-haul edge → origin trip High hit ratio = low origin egress = lower bill a 90% → 99% hit-ratio improvement is a 10× reduction in origin traffic

Cache hit ratio is the single number that determines whether your CDN is doing its job. 95% is a healthy baseline for static assets. 50% means something is misconfigured: TTLs too short, query strings in cache keys, cookies preventing caching, or your assets aren't marked cacheable. Fix the hit ratio before you fix anything else.

📋 Chapter 2 — Summary
  • Pull CDN is the modern default — lazy fetch from origin, automatic discovery of new content.
  • Cache hit ratio at the edge is the metric to watch — 95%+ for static is the bar.
  • S3 is a flat key/value store with multipart upload for large objects and lifecycle policies for tiering.
  • Storage classes let you pay for access frequency — hot, infrequent, archive.
03
Chapter Three

When to Use — and When Not To

CDN: Default On for Public Static Content

For nearly any web application serving public, cacheable assets, the CDN-on default is correct. The cost is small, the operational complexity is low, and the latency win is unambiguous. The cases where a CDN doesn't help are narrower than people imagine: heavily personalised per-request content, internal-only APIs, and content that changes on every request. Even then, edge compute (Cloudflare Workers, Lambda @ Edge) lets you push some logic out so personalisation doesn't mean “always hit origin.”

USE a CDN When…

Static assets: JS bundles, CSS, fonts, images, videos — the obvious case.

Public API responses with cacheable shape: product catalogs, public profiles, search results.

Geographic users: any time your users span more than one continent.

Traffic spikes: launches, campaigns, news events — CDN absorbs the burst.

DDoS shielding: the CDN is the first thing attackers see, not your origin.

DO NOT Use a CDN For…

Highly personalised per-user content: dashboards, account pages with PII — cache key explodes.

Strict freshness requirements: stock prices, live game state — CDN TTL is wrong tool.

Internal-only APIs: traffic stays within your VPC; no benefit from edge proximity.

Tiny low-traffic sites: overhead may not justify the cost — but at zero CDN cost (Cloudflare free), even this is rare today.

Object Storage vs Block vs File — Pick the Right Storage Class

Three storage paradigms exist, and they are not interchangeable. Object storage (S3): HTTP API, immutable objects, infinite scale, eventual-then-strong consistency, no random updates. Block storage (EBS, GCP Persistent Disk): raw block device attached to one VM, supports filesystems, low latency, fixed capacity. File storage (EFS, NFS, FSx): network filesystem accessible by many VMs, supports POSIX semantics, expensive but compatible with legacy software. Object storage wins for “files served by my web application” in nearly every case. Block storage is what your databases and VMs run on. File storage is for legacy workloads that genuinely require a shared POSIX filesystem.

🗄️

USE Object Storage When…

User uploads, ML datasets, video archives, backups, static site assets, data lakes.

Don't use: as a database; for files needing in-place edits; when you need POSIX semantics.

💾

USE Block Storage When…

Database storage, VM root disks, applications expecting low-latency random I/O.

Don't use: for shared multi-VM access (it's single-attach); for huge unstructured archives.

📂

USE File Storage When…

Legacy software needing POSIX/NFS, shared media folders, build pipelines.

Don't use: when object storage suffices — file storage is more expensive and less scalable.

The single most common mistake at the storage layer: storing user uploads on the local disk of the application server. It scales to one server, breaks horizontal scaling, makes deployments destructive, and has no durability guarantees. The fix is always “put it in object storage and store the URL.”

📋 Chapter 3 — Summary
  • CDN is default-on for public static content; default-off for highly personalised or internal-only traffic.
  • Object storage for files served by your app; block storage for databases and VMs; file storage for legacy POSIX needs.
  • Edge compute (Workers, Lambda@Edge) bridges the gap between “static” and “personalised but still fast.”
  • Don't store user uploads on application server disks — ever.
04
Chapter Four

Trade-offs & Comparisons

Cache Invalidation — Three Strategies, Pick Wisely

CDN cache invalidation is the second of the “two hard things in computer science.” You have three real strategies. Versioned URLs (the right answer most of the time): when a file changes, change its URL — app.a3f9b2.js instead of app.js. The new URL is uncached; the old URL keeps serving until natural TTL expiry. Zero invalidation calls, zero coordination. TTL expiry: set a short TTL (e.g. 60 s) and let staleness self-heal. Simple, but every request fetches from origin once per TTL window. Explicit purge: call the CDN API to invalidate a URL or pattern. Slow (propagation across edges takes seconds to minutes), often rate-limited, and historically the most common source of “why is the old version still showing?” bugs.

🔖

Versioned URLs (Best)

Pattern: include hash or version in filename: app.a3f9b2.js.

TTL: 1 year (immutable).

Trade-off: requires build pipeline that fingerprints assets and rewrites references.

Use: JS bundles, CSS, images served by your build.

⏲️

TTL-Based Expiry

Pattern: set a sensible TTL (60 s, 5 min, 1 hour).

Trade-off: always slightly stale; origin sees re-validation traffic.

Use: content where minutes-of-staleness is fine: blog posts, product pages.

🗑️

Explicit Purge (Last Resort)

Pattern: CDN API call to invalidate URL or pattern.

Trade-off: slow propagation (seconds to minutes); rate-limited; brittle.

Use: emergencies (legal takedown, security incidents).

S3 Consistency — Now Strong, but Beware History

S3 used to be eventually consistent for new objects in some regions; it became strongly read-after-write consistent globally in December 2020. PUT then immediate GET now reliably returns the new object. List operations are also strongly consistent. The implication: a lot of legacy code defensively retries or sleeps after writes — that defensive code is no longer necessary on S3 specifically, though it's still required on most other object stores. Don't assume strong consistency on every object store; check the docs for whichever you use.

Storage Class Cost Model — Know What You're Paying For
💵

Hot & Infrequent Tiers

Standard: ~$0.023/GB-month, free retrieval, no minimum duration.

Infrequent Access (Standard-IA): ~$0.0125/GB-month + $0.01/GB retrieval; 30-day minimum duration.

Watch: if your access pattern is bursty, retrieval fees can erase the storage savings. Use S3 Intelligent-Tiering to let AWS decide.

❄️

Archive Tiers

Glacier Instant: ~$0.004/GB-month, milliseconds retrieval, 90-day minimum.

Glacier Flexible: ~$0.0036/GB-month, minutes-to-hours retrieval.

Glacier Deep Archive: ~$0.00099/GB-month, hours retrieval, 180-day minimum. Cheapest cloud storage that exists.

Versioned URLs make CDN invalidation a non-problem. If you find yourself relying on explicit purge calls, your build pipeline is the thing to fix — not the CDN. Five minutes of investment in fingerprinted asset filenames eliminates an entire class of production bugs forever.

📋 Chapter 4 — Summary
  • Versioned URLs are the cleanest invalidation strategy — no coordination, no waiting.
  • S3 is strongly consistent since 2020; legacy retry-loops on writes are no longer required.
  • Storage classes trade access frequency for cost — lifecycle policies automate the tiering.
  • Watch retrieval fees on infrequent / archive tiers — bursty access can erase the savings.
05
Chapter Five

Production Patterns & Common Mistakes

Two Patterns Every Team Should Know

Two patterns turn up in nearly every production system that gets static content right. Presigned URLs: generate a short-lived signed URL that grants temporary access to an object — clients upload directly to S3 without the bytes ever flowing through your application servers. Critical for video uploads, large file uploads, anything that would otherwise saturate your app tier. CDN for API acceleration: even dynamic API responses benefit from edge proximity — the CDN handles TLS termination, request collapsing, and geographic routing even when nothing is cached. The TTFB win is real even at zero cache hit ratio.

🔐

Pattern: Presigned URLs

Goal: let clients upload/download directly to/from object storage without your app proxying bytes.

How: server signs URL with short expiry (5–15 min); client uses URL directly; S3 enforces expiry and signature.

Watch: use POST policy for size/content-type constraints on uploads; rotate signing keys periodically.

Use: profile photo uploads, video uploads, document downloads with access control.

Pattern: CDN-Fronted APIs

Goal: reduce TTFB even on uncacheable APIs.

Mechanism: TLS terminated at edge, request collapsing for cacheable bursts, optimal route to origin.

Watch: the CDN sees all your traffic — pick a vendor whose privacy/jurisdiction model matches yours.

Use: public APIs, mobile app backends, anything user-facing.

The Five Mistakes That Bite In Production
🗄️

Mistake 1 — Files in Database

Storing image/video bytes as BLOBs in PostgreSQL or MongoDB. Database grows uncontrollably, backups become huge, queries slow down. Fix: store the bytes in S3, store the URL in the database.

📻

Mistake 2 — No CDN for Static

Web app serves its own JS, CSS, and images directly from origin. App servers spend CPU on static content. Global users wait. Fix: put CloudFront / Cloudflare in front of static; mark assets cacheable; profit.

🔑

Mistake 3 — Public S3 Bucket

Bucket misconfigured as public; sensitive data leaks. Fix: default-deny bucket policy, “Block all public access” on; serve through CDN with origin access identity; audit with S3 Public Access Reports.

📲

Mistake 4 — Uploads Through App Tier

Mobile app POSTs 50 MB videos to your API server, which proxies to S3. App tier CPU and bandwidth saturate. Fix: presigned URLs — client uploads directly to S3; app tier touches only metadata.

🔑

Mistake 5 — Cache Key Includes Junk

Cache key includes user-specific cookies, tracking query strings, or session IDs — effective hit ratio drops to near zero. Fix: normalise cache key; whitelist only the query params that actually vary content.

♻️

Bonus — Forgotten Multipart Uploads

Aborted multipart uploads accumulate, paying for storage of incomplete data forever. Fix: lifecycle rule to abort multipart uploads older than 7 days; review storage cost reports monthly.

The static-content backbone is one of the highest-leverage architectures in your system. Get it right once and it scales for a decade. Get it wrong and you spend half your engineering budget shipping bytes that didn't need to move.

📋 Chapter 5 — Summary
  • Presigned URLs let clients upload/download directly to object storage — never proxy bytes through your app.
  • CDN-fronted APIs reduce TTFB even at 0% cache hit ratio — TLS termination and routing are wins on their own.
  • The five outage mistakes: files in DB, no CDN for static, public buckets, uploads through app tier, polluted cache keys.
  • Lifecycle rules clean up after you — expire infrequent data, abort orphaned multipart uploads.