System design · Concept 4
Caching
Cache-aside, read-through, write-through and write-back; TTLs, invalidation, eviction, hot keys, stampede protection and CDNs.
In short
- Cache-aside, a TTL and delete-on-write cover most cases
- TTL from allowed staleness, with jitter
- A small drop in hit ratio multiplies database load
- Write-back can lose data: never for money
- Protect hot keys from stampedes; plan for the cache being down
A cache stores the result of expensive work (a DB query, a rendered page, a remote call) in a faster place so the next request is cheaper. Caching is often the single biggest performance win, and also a common source of stale data and outages. Treat every cache as a deliberate trade of freshness for speed and cost.
Where caches live
Browser cache ─► CDN edge ─► API gateway / reverse proxy ─► in-process cache ─► Redis/Memcached ─► DB buffer pool
(ms saved) (geo) (shared responses) (ns, per instance) (sub-ms, shared) (built in)
| Layer | Latency | Shared? | Typical content |
|---|---|---|---|
| Browser / app | 0 ms | Per user | Static assets, API responses with Cache-Control |
| CDN | 5-30 ms (near user) | Global | Images, JS/CSS, video, cacheable API GETs |
| In-process (LRU map) | ~100 ns | Per instance | Config, feature flags, very hot keys |
| Distributed (Redis) | 0.3-1 ms | All instances | Sessions, query results, counters, feeds |
Caching patterns
Cache-aside (lazy loading), the default
read(key):
v = cache.get(key)
if v != null: return v # hit
v = db.query(key) # miss
cache.set(key, v, ttl = 1h)
return v
write(key, v):
db.update(key, v)
cache.delete(key) # invalidate, don't update
The application talks to both the cache and the DB. Only data that's actually read gets cached, and a cache outage degrades to "slow" instead of "down".
Read-through
The cache library or provider loads from the DB on a miss, so the app only talks to the cache. It's the same idea as cache-aside with cleaner code, but you depend on the cache's loader integration.
Write-through
Writes go to the cache and the DB synchronously. The cache is always fresh for written keys, but every write pays both latencies and you cache data that may never be read.
Write-back (write-behind)
Writes go to the cache, and the cache flushes to the DB asynchronously in batches. Writes are very fast and bursts are absorbed, but you lose data if the cache dies before flushing. Use it for counters, metrics and "likes", never for money.
Write-around
Writes go straight to the DB and the cache is filled only on reads. This avoids polluting the cache with write-once data such as logs or uploads.
| Pattern | Read latency | Write latency | Staleness risk | Data-loss risk | Use when |
|---|---|---|---|---|---|
| Cache-aside | Miss = slow once | DB only | Until TTL / invalidation | None | Default for read-heavy data |
| Read-through | Same | DB only | Same | None | Want simpler app code |
| Write-through | Always fast for written keys | DB + cache | Low | None | Read-after-write must be fast |
| Write-back | Fast | Very fast | Low | Yes | Counters, high write rates |
| Write-around | First read slow | DB only | Low | None | Write-once, rarely read data |
TTL (time to live)
- Pick the TTL from how stale the data may be, not from memory size: product catalog 10 min, user profile 1 h, short-link mapping 24 h+, stock price 1 s.
- Add jitter (e.g. TTL ± 10%) so keys written together don't all expire together.
- Negative caching: cache "not found" for a short TTL (30-60 s) so a flood of lookups for missing keys doesn't hammer the DB.
- A TTL is your safety net even with explicit invalidation, because invalidations sometimes get lost.
Invalidation
"There are only two hard things in computer science: cache invalidation and naming things."
- Delete on write, don't update. If two writers update the cache concurrently, the older value can win. Deleting forces the next read to load the fresh value.
- The classic race: reader A misses and reads the old value from the DB. Writer B updates the DB and deletes the key. A then writes the old value into the cache. Mitigations: a short TTL, versioned values (only set if the version is newer), or a delayed second delete.
- Versioned keys:
user:42:v7. Bump the version on change, and old entries simply age out. - Event-driven invalidation: consume DB change events (CDC, outbox) and delete affected keys. This works across services.
- With read replicas, a miss refilled from a lagging replica can re-cache stale data. Read from the primary for recently written keys, or delay the invalidation.
Eviction policies
When memory is full, something must go:
| Policy | Evicts | Good for |
|---|---|---|
| LRU (least recently used) | Oldest access | General purpose default |
| LFU (least frequently used) | Lowest access count | Stable popularity (top products) and resisting one-off scans |
| FIFO | Oldest insert | Simple, rarely optimal |
| TTL-based | Soonest to expire | Mostly time-bound data |
| Random | Random key | Cheap, surprisingly decent |
Redis offers allkeys-lru, allkeys-lfu, volatile-lru (only keys with a TTL) and noeviction (writes fail when full; right for queues or locks, wrong for caches). Track the hit ratio (aim for >80-90% for hot paths), evictions per second and memory.
Hot keys
One key (a celebrity profile, a flash-sale product) can receive more traffic than one cache shard can serve (~100K ops/s).
- Local L1 cache: keep hot keys in-process for 1-5 s. This absorbs most reads.
- Key replication: store copies
product:42#0..#9on different shards and read a random copy. - Request coalescing: collapse concurrent identical requests into one backend call.
- Detect hot keys with sampling (
redis-cli --hotkeys, proxy metrics) and alert on them.
Cache stampede (thundering herd)
When a popular key expires, thousands of requests miss at once and all hit the DB.
| Technique | How |
|---|---|
| Lock / single-flight | The first miss takes a short lock (SET key:lock NX PX 5000) and recomputes; others wait briefly or serve stale |
| Stale-while-revalidate | Serve the expired value while one background refresh runs |
| Probabilistic early expiration | Each read refreshes early with a probability that rises near expiry (XFetch) |
| Pre-warming | Populate caches before traffic arrives (deploys, sales) |
| TTL jitter | Avoid synchronized expiry |
CDN
A CDN is a geographically distributed cache in front of your origin.
- Static assets: use fingerprinted file names (
app.3f9a1c.js) withCache-Control: public, max-age=31536000, immutable. You never need to purge. - Media: images and video from object storage, with resizing at the edge or via an image service.
- API responses: cacheable GETs with
s-maxageandstale-while-revalidate. Vary on as few headers as possible. - Private content: signed URLs or cookies with short expiry.
- Pull vs push: pull CDNs fetch from the origin on a miss (the default). Push means you upload content up front, which suits large, predictable files.
- Purge is slow and eventually consistent across POPs. Design so you rarely need it (versioned URLs).
- A CDN also absorbs DDoS traffic and terminates TLS close to users.
When to use / trade-offs
- Use a cache when reads greatly outnumber writes, the same data is read repeatedly, and some staleness is acceptable.
- Avoid or limit caching for data that must be strongly consistent (balances, inventory at checkout). Read those from the source of truth, or use the cache only as a hint.
- Costs: stale reads, invalidation bugs, extra infrastructure, cold-start load after a cache flush, and memory cost.
- Failure mode: if the cache disappears, can the DB survive the full load? If not, you need cache replicas, request coalescing and rate limiting toward the DB.
- Cache-aside + TTL + delete-on-write covers 80% of cases. Reach for write-back or event-driven invalidation only when a requirement demands it.
Checklist
- What is cached, at which layer, and why (read:write ratio)?
- Pattern chosen (usually cache-aside) and write path invalidation
- TTL per data type with jitter; negative caching for misses
- Eviction policy and memory sizing; hit-ratio monitoring
- Hot-key and stampede protection
- CDN for static/media with versioned URLs
- Behavior when the cache is down
Practise this concept
In the app: a short video and a 11-question concept check. Free with a GitHub account.
Open in the app