System design · Concept 4

Caching

Cache-aside, read-through, write-through and write-back; TTLs, invalidation, eviction, hot keys, stampede protection and CDNs.

7 min read

In short

  • Cache-aside, a TTL and delete-on-write cover most cases
  • TTL from allowed staleness, with jitter
  • A small drop in hit ratio multiplies database load
  • Write-back can lose data: never for money
  • Protect hot keys from stampedes; plan for the cache being down

A cache stores the result of expensive work (a DB query, a rendered page, a remote call) in a faster place so the next request is cheaper. Caching is often the single biggest performance win, and also a common source of stale data and outages. Treat every cache as a deliberate trade of freshness for speed and cost.

Where caches live

Browser cache ─► CDN edge ─► API gateway / reverse proxy ─► in-process cache ─► Redis/Memcached ─► DB buffer pool
   (ms saved)     (geo)           (shared responses)          (ns, per instance)    (sub-ms, shared)    (built in)
LayerLatencyShared?Typical content
Browser / app0 msPer userStatic assets, API responses with Cache-Control
CDN5-30 ms (near user)GlobalImages, JS/CSS, video, cacheable API GETs
In-process (LRU map)~100 nsPer instanceConfig, feature flags, very hot keys
Distributed (Redis)0.3-1 msAll instancesSessions, query results, counters, feeds

Caching patterns

Cache-aside (lazy loading), the default

read(key):
  v = cache.get(key)
  if v != null: return v                 # hit
  v = db.query(key)                      # miss
  cache.set(key, v, ttl = 1h)
  return v

write(key, v):
  db.update(key, v)
  cache.delete(key)                      # invalidate, don't update

The application talks to both the cache and the DB. Only data that's actually read gets cached, and a cache outage degrades to "slow" instead of "down".

Read-through

The cache library or provider loads from the DB on a miss, so the app only talks to the cache. It's the same idea as cache-aside with cleaner code, but you depend on the cache's loader integration.

Write-through

Writes go to the cache and the DB synchronously. The cache is always fresh for written keys, but every write pays both latencies and you cache data that may never be read.

Write-back (write-behind)

Writes go to the cache, and the cache flushes to the DB asynchronously in batches. Writes are very fast and bursts are absorbed, but you lose data if the cache dies before flushing. Use it for counters, metrics and "likes", never for money.

Write-around

Writes go straight to the DB and the cache is filled only on reads. This avoids polluting the cache with write-once data such as logs or uploads.

PatternRead latencyWrite latencyStaleness riskData-loss riskUse when
Cache-asideMiss = slow onceDB onlyUntil TTL / invalidationNoneDefault for read-heavy data
Read-throughSameDB onlySameNoneWant simpler app code
Write-throughAlways fast for written keysDB + cacheLowNoneRead-after-write must be fast
Write-backFastVery fastLowYesCounters, high write rates
Write-aroundFirst read slowDB onlyLowNoneWrite-once, rarely read data

TTL (time to live)

  • Pick the TTL from how stale the data may be, not from memory size: product catalog 10 min, user profile 1 h, short-link mapping 24 h+, stock price 1 s.
  • Add jitter (e.g. TTL ± 10%) so keys written together don't all expire together.
  • Negative caching: cache "not found" for a short TTL (30-60 s) so a flood of lookups for missing keys doesn't hammer the DB.
  • A TTL is your safety net even with explicit invalidation, because invalidations sometimes get lost.

Invalidation

"There are only two hard things in computer science: cache invalidation and naming things."

  • Delete on write, don't update. If two writers update the cache concurrently, the older value can win. Deleting forces the next read to load the fresh value.
  • The classic race: reader A misses and reads the old value from the DB. Writer B updates the DB and deletes the key. A then writes the old value into the cache. Mitigations: a short TTL, versioned values (only set if the version is newer), or a delayed second delete.
  • Versioned keys: user:42:v7. Bump the version on change, and old entries simply age out.
  • Event-driven invalidation: consume DB change events (CDC, outbox) and delete affected keys. This works across services.
  • With read replicas, a miss refilled from a lagging replica can re-cache stale data. Read from the primary for recently written keys, or delay the invalidation.

Eviction policies

When memory is full, something must go:

PolicyEvictsGood for
LRU (least recently used)Oldest accessGeneral purpose default
LFU (least frequently used)Lowest access countStable popularity (top products) and resisting one-off scans
FIFOOldest insertSimple, rarely optimal
TTL-basedSoonest to expireMostly time-bound data
RandomRandom keyCheap, surprisingly decent

Redis offers allkeys-lru, allkeys-lfu, volatile-lru (only keys with a TTL) and noeviction (writes fail when full; right for queues or locks, wrong for caches). Track the hit ratio (aim for >80-90% for hot paths), evictions per second and memory.

Hot keys

One key (a celebrity profile, a flash-sale product) can receive more traffic than one cache shard can serve (~100K ops/s).

  • Local L1 cache: keep hot keys in-process for 1-5 s. This absorbs most reads.
  • Key replication: store copies product:42#0..#9 on different shards and read a random copy.
  • Request coalescing: collapse concurrent identical requests into one backend call.
  • Detect hot keys with sampling (redis-cli --hotkeys, proxy metrics) and alert on them.

Cache stampede (thundering herd)

When a popular key expires, thousands of requests miss at once and all hit the DB.

TechniqueHow
Lock / single-flightThe first miss takes a short lock (SET key:lock NX PX 5000) and recomputes; others wait briefly or serve stale
Stale-while-revalidateServe the expired value while one background refresh runs
Probabilistic early expirationEach read refreshes early with a probability that rises near expiry (XFetch)
Pre-warmingPopulate caches before traffic arrives (deploys, sales)
TTL jitterAvoid synchronized expiry

CDN

A CDN is a geographically distributed cache in front of your origin.

  • Static assets: use fingerprinted file names (app.3f9a1c.js) with Cache-Control: public, max-age=31536000, immutable. You never need to purge.
  • Media: images and video from object storage, with resizing at the edge or via an image service.
  • API responses: cacheable GETs with s-maxage and stale-while-revalidate. Vary on as few headers as possible.
  • Private content: signed URLs or cookies with short expiry.
  • Pull vs push: pull CDNs fetch from the origin on a miss (the default). Push means you upload content up front, which suits large, predictable files.
  • Purge is slow and eventually consistent across POPs. Design so you rarely need it (versioned URLs).
  • A CDN also absorbs DDoS traffic and terminates TLS close to users.

When to use / trade-offs

  • Use a cache when reads greatly outnumber writes, the same data is read repeatedly, and some staleness is acceptable.
  • Avoid or limit caching for data that must be strongly consistent (balances, inventory at checkout). Read those from the source of truth, or use the cache only as a hint.
  • Costs: stale reads, invalidation bugs, extra infrastructure, cold-start load after a cache flush, and memory cost.
  • Failure mode: if the cache disappears, can the DB survive the full load? If not, you need cache replicas, request coalescing and rate limiting toward the DB.
  • Cache-aside + TTL + delete-on-write covers 80% of cases. Reach for write-back or event-driven invalidation only when a requirement demands it.

Checklist

  • What is cached, at which layer, and why (read:write ratio)?
  • Pattern chosen (usually cache-aside) and write path invalidation
  • TTL per data type with jitter; negative caching for misses
  • Eviction policy and memory sizing; hit-ratio monitoring
  • Hot-key and stampede protection
  • CDN for static/media with versioned URLs
  • Behavior when the cache is down

Practise this concept

In the app: a short video and a 11-question concept check. Free with a GitHub account.

Open in the app