System design · Concept 3

Scaling Basics

Vertical vs horizontal scaling, stateless services, load balancing algorithms, autoscaling and where session state should live.

7 min read

In short

  • Scale stateless tiers out, the database up, for as long as you can
  • Move sessions, files and jobs out of the process
  • Pick the load-balancing algorithm for your request shape
  • Autoscale on the right signal, with warm-up and limits
  • Data tier: indexes, cache, replicas, then partitioning

Scaling means handling more load (requests, data, users) while keeping latency and availability within targets. Most systems follow a predictable path, and knowing the stages helps you propose the next step instead of jumping straight to the final one.

The scaling journey

Stage 1  [ App + DB on one box ]

Stage 2  [ App ] ──► [ DB ]                     split tiers, scale each separately

Stage 3  Clients ─► [ LB ] ─► [ App ×N ] ─► [ DB primary ] ─► [ read replicas ]

Stage 4  + CDN for static/media, + cache (Redis), + queue for slow work

Stage 5  + sharded data, + multiple regions, + per-service databases

Each stage adds operational complexity, so move to the next one when a metric tells you to: CPU, p99 latency, replication lag or storage.

Vertical vs horizontal scaling

AspectVertical (scale up)Horizontal (scale out)
HowBigger machine (more CPU/RAM/IOPS)More machines behind a load balancer
Code changesNoneService must be stateless or partitioned
CeilingHardware limit (largest instance)Practically unlimited for stateless tiers
AvailabilitySingle point of failureSurvives instance and AZ failures
Cost curveSuperlinear at the high endLinear-ish, but more ops overhead
Typical useDatabases early on, quick winsWeb/API tiers, workers, caches, sharded DBs

A pragmatic rule: scale stateless tiers horizontally from day one (it's cheap) and scale the database vertically as long as you can, because sharding is expensive to build and operate.

Stateless services

A service is stateless when any instance can handle any request, so nothing it needs lives only in its own memory or local disk between requests.

Move state out of the process:

StateWhere it should live
User sessionsSigned token (JWT/cookie) or a session store (Redis)
Uploaded filesObject storage (S3, Blob Storage, Cloud Storage), never the local disk
CachesDistributed cache, or accept per-instance caches that may differ
Background jobsA queue, not in-memory timers
ConfigEnvironment variables / config service

Benefits: instances become disposable, so you can autoscale, roll deploys and replace unhealthy nodes freely. This is the core idea behind the "twelve-factor app".

Load balancing

L4 vs L7

L4 (transport)L7 (application)
SeesIP, port, TCP/UDPHTTP method, path, headers, cookies
Can doFast connection forwardingPath routing, TLS termination, header rewrites, retries
Use forRaw TCP, WebSockets at huge scale, non-HTTPMost web APIs, microservice routing
ExamplesAWS NLB, Azure Load Balancer, Google Cloud passthrough Network Load Balancer, HAProxy (TCP mode)AWS ALB, Azure Application Gateway, Google Cloud Application Load Balancer, nginx, Envoy

Algorithms

AlgorithmHow it worksGood forWatch out
Round robinNext server in turnUniform requests and serversIgnores current load
Weighted round robinMore turns for bigger serversMixed instance sizes, canariesStatic weights
Least connectionsFewest open connectionsLong-lived or uneven requestsNeeds connection tracking
Least response time / EWMAFastest recent respondersLatency-sensitive servicesCan herd onto a recovering node
Power of two choicesPick 2 at random, choose the less loadedLarge fleets, cheap and robustSlightly less optimal than global view
IP hash / consistent hashingSame key → same serverCache affinity, sticky routingUneven load, rebalancing on change

Health checks and draining

  • Active checks hit /healthz every few seconds. Passive checks eject servers that return errors.
  • Keep a liveness check ("process is up") separate from a readiness check ("can serve traffic": DB reachable, warmed up).
  • Connection draining: during deploys, stop sending new requests to an instance and let in-flight ones finish.
  • The LB itself must be redundant. Managed LBs are; self-hosted ones run as pairs with a floating IP or behind DNS/anycast.

Autoscaling

Autoscaling adds or removes instances based on a signal:

SignalUse for
CPU utilization (target ~50-70%)CPU-bound API servers
Requests per instanceI/O-bound services where CPU is a poor proxy
Queue depth / consumer lagWorkers: scale on backlog, not CPU
Custom latency metricServices with strict SLOs
ScheduleKnown peaks (business hours, nightly batch, sales)

Practical concerns:

  • Warm-up time: new instances take 30 s to several minutes (boot, JIT, cache warm-up). Scale out early and scale in slowly (cooldowns).
  • Min/max bounds: keep a minimum for availability (at least 2-3 across AZs) and a maximum to protect downstream systems and your budget.
  • Downstream limits: 200 app instances × a pool of 20 connections = 4,000 DB connections, which can overwhelm Postgres. Use a connection pooler (PgBouncer, RDS Proxy) and size pools deliberately.
  • Thundering herds: if every new instance warms its cache from the DB at once, the DB suffers. Stagger warm-up or share a distributed cache.

Sessions

OptionHowProsCons
Sticky sessionsLB pins a user to one instance (cookie)Simple, no shared storeUneven load; sessions lost when the instance dies; blocks autoscaling
Central session storeSession ID cookie → RedisAny instance serves any user; easy revocationExtra hop (~0.5 ms), Redis must be HA
Client-side tokenSigned JWT / encrypted cookie holds the claimsNo server state at all, scales triviallyHard to revoke before expiry; token size; keep short TTL + refresh tokens

A common combination is short-lived JWT access tokens (5-15 min) plus refresh tokens that are checked against a store, so you get statelessness on the hot path and revocation within minutes.

WebSockets are inherently sticky: a connection lives on one server. Make the server replaceable by keeping a registry (user → gateway node) in Redis and letting clients reconnect anywhere.

Scaling the data tier (preview)

Stateless tiers are easy; data is hard. In order of increasing cost:

  1. Indexes and query tuning. Often a 10-100x win.
  2. Caching hot reads (Redis, CDN).
  3. Read replicas for read-heavy traffic, as long as you can tolerate replication lag.
  4. Vertical scaling of the primary.
  5. Functional partitioning: separate databases per service or domain.
  6. Sharding: split one dataset across many nodes by key.

See the Databases lesson for details.

When to use / trade-offs

  • Horizontal scaling for anything stateless. The trade-off is that you need a load balancer, health checks, and stateless code discipline.
  • Vertical scaling for databases and for quick relief. It ends at a hard ceiling and is a single point of failure unless you add replicas.
  • Sticky sessions only for legacy apps or connection-oriented protocols. Prefer a session store or tokens.
  • Autoscaling for variable load. It's not instant, so pair it with headroom and scheduled scaling for known peaks.
  • Don't over-scale early: a single well-indexed Postgres plus a few app instances serves a surprising amount of traffic (thousands of QPS).

Checklist

  • App tier stateless (sessions, files and jobs moved out of process)
  • L7 load balancer with readiness checks and connection draining
  • Instances spread across at least 2-3 AZs
  • Autoscaling on the right signal with min/max bounds and cooldowns
  • DB connection pooling sized against max instance count
  • Plan for the data tier: indexes → cache → replicas → partitioning

Practise this concept

In the app: a short video and a 11-question concept check. Free with a GitHub account.

Open in the app