System design · Concept 3
Scaling Basics
Vertical vs horizontal scaling, stateless services, load balancing algorithms, autoscaling and where session state should live.
In short
- Scale stateless tiers out, the database up, for as long as you can
- Move sessions, files and jobs out of the process
- Pick the load-balancing algorithm for your request shape
- Autoscale on the right signal, with warm-up and limits
- Data tier: indexes, cache, replicas, then partitioning
Scaling means handling more load (requests, data, users) while keeping latency and availability within targets. Most systems follow a predictable path, and knowing the stages helps you propose the next step instead of jumping straight to the final one.
The scaling journey
Stage 1 [ App + DB on one box ]
Stage 2 [ App ] ──► [ DB ] split tiers, scale each separately
Stage 3 Clients ─► [ LB ] ─► [ App ×N ] ─► [ DB primary ] ─► [ read replicas ]
Stage 4 + CDN for static/media, + cache (Redis), + queue for slow work
Stage 5 + sharded data, + multiple regions, + per-service databases
Each stage adds operational complexity, so move to the next one when a metric tells you to: CPU, p99 latency, replication lag or storage.
Vertical vs horizontal scaling
| Aspect | Vertical (scale up) | Horizontal (scale out) |
|---|---|---|
| How | Bigger machine (more CPU/RAM/IOPS) | More machines behind a load balancer |
| Code changes | None | Service must be stateless or partitioned |
| Ceiling | Hardware limit (largest instance) | Practically unlimited for stateless tiers |
| Availability | Single point of failure | Survives instance and AZ failures |
| Cost curve | Superlinear at the high end | Linear-ish, but more ops overhead |
| Typical use | Databases early on, quick wins | Web/API tiers, workers, caches, sharded DBs |
A pragmatic rule: scale stateless tiers horizontally from day one (it's cheap) and scale the database vertically as long as you can, because sharding is expensive to build and operate.
Stateless services
A service is stateless when any instance can handle any request, so nothing it needs lives only in its own memory or local disk between requests.
Move state out of the process:
| State | Where it should live |
|---|---|
| User sessions | Signed token (JWT/cookie) or a session store (Redis) |
| Uploaded files | Object storage (S3, Blob Storage, Cloud Storage), never the local disk |
| Caches | Distributed cache, or accept per-instance caches that may differ |
| Background jobs | A queue, not in-memory timers |
| Config | Environment variables / config service |
Benefits: instances become disposable, so you can autoscale, roll deploys and replace unhealthy nodes freely. This is the core idea behind the "twelve-factor app".
Load balancing
L4 vs L7
| L4 (transport) | L7 (application) | |
|---|---|---|
| Sees | IP, port, TCP/UDP | HTTP method, path, headers, cookies |
| Can do | Fast connection forwarding | Path routing, TLS termination, header rewrites, retries |
| Use for | Raw TCP, WebSockets at huge scale, non-HTTP | Most web APIs, microservice routing |
| Examples | AWS NLB, Azure Load Balancer, Google Cloud passthrough Network Load Balancer, HAProxy (TCP mode) | AWS ALB, Azure Application Gateway, Google Cloud Application Load Balancer, nginx, Envoy |
Algorithms
| Algorithm | How it works | Good for | Watch out |
|---|---|---|---|
| Round robin | Next server in turn | Uniform requests and servers | Ignores current load |
| Weighted round robin | More turns for bigger servers | Mixed instance sizes, canaries | Static weights |
| Least connections | Fewest open connections | Long-lived or uneven requests | Needs connection tracking |
| Least response time / EWMA | Fastest recent responders | Latency-sensitive services | Can herd onto a recovering node |
| Power of two choices | Pick 2 at random, choose the less loaded | Large fleets, cheap and robust | Slightly less optimal than global view |
| IP hash / consistent hashing | Same key → same server | Cache affinity, sticky routing | Uneven load, rebalancing on change |
Health checks and draining
- Active checks hit
/healthzevery few seconds. Passive checks eject servers that return errors. - Keep a liveness check ("process is up") separate from a readiness check ("can serve traffic": DB reachable, warmed up).
- Connection draining: during deploys, stop sending new requests to an instance and let in-flight ones finish.
- The LB itself must be redundant. Managed LBs are; self-hosted ones run as pairs with a floating IP or behind DNS/anycast.
Autoscaling
Autoscaling adds or removes instances based on a signal:
| Signal | Use for |
|---|---|
| CPU utilization (target ~50-70%) | CPU-bound API servers |
| Requests per instance | I/O-bound services where CPU is a poor proxy |
| Queue depth / consumer lag | Workers: scale on backlog, not CPU |
| Custom latency metric | Services with strict SLOs |
| Schedule | Known peaks (business hours, nightly batch, sales) |
Practical concerns:
- Warm-up time: new instances take 30 s to several minutes (boot, JIT, cache warm-up). Scale out early and scale in slowly (cooldowns).
- Min/max bounds: keep a minimum for availability (at least 2-3 across AZs) and a maximum to protect downstream systems and your budget.
- Downstream limits: 200 app instances × a pool of 20 connections = 4,000 DB connections, which can overwhelm Postgres. Use a connection pooler (PgBouncer, RDS Proxy) and size pools deliberately.
- Thundering herds: if every new instance warms its cache from the DB at once, the DB suffers. Stagger warm-up or share a distributed cache.
Sessions
| Option | How | Pros | Cons |
|---|---|---|---|
| Sticky sessions | LB pins a user to one instance (cookie) | Simple, no shared store | Uneven load; sessions lost when the instance dies; blocks autoscaling |
| Central session store | Session ID cookie → Redis | Any instance serves any user; easy revocation | Extra hop (~0.5 ms), Redis must be HA |
| Client-side token | Signed JWT / encrypted cookie holds the claims | No server state at all, scales trivially | Hard to revoke before expiry; token size; keep short TTL + refresh tokens |
A common combination is short-lived JWT access tokens (5-15 min) plus refresh tokens that are checked against a store, so you get statelessness on the hot path and revocation within minutes.
WebSockets are inherently sticky: a connection lives on one server. Make the server replaceable by keeping a registry (user → gateway node) in Redis and letting clients reconnect anywhere.
Scaling the data tier (preview)
Stateless tiers are easy; data is hard. In order of increasing cost:
- Indexes and query tuning. Often a 10-100x win.
- Caching hot reads (Redis, CDN).
- Read replicas for read-heavy traffic, as long as you can tolerate replication lag.
- Vertical scaling of the primary.
- Functional partitioning: separate databases per service or domain.
- Sharding: split one dataset across many nodes by key.
See the Databases lesson for details.
When to use / trade-offs
- Horizontal scaling for anything stateless. The trade-off is that you need a load balancer, health checks, and stateless code discipline.
- Vertical scaling for databases and for quick relief. It ends at a hard ceiling and is a single point of failure unless you add replicas.
- Sticky sessions only for legacy apps or connection-oriented protocols. Prefer a session store or tokens.
- Autoscaling for variable load. It's not instant, so pair it with headroom and scheduled scaling for known peaks.
- Don't over-scale early: a single well-indexed Postgres plus a few app instances serves a surprising amount of traffic (thousands of QPS).
Checklist
- App tier stateless (sessions, files and jobs moved out of process)
- L7 load balancer with readiness checks and connection draining
- Instances spread across at least 2-3 AZs
- Autoscaling on the right signal with min/max bounds and cooldowns
- DB connection pooling sized against max instance count
- Plan for the data tier: indexes → cache → replicas → partitioning
Practise this concept
In the app: a short video and a 11-question concept check. Free with a GitHub account.
Open in the app