System design · Concept 8

Reliability Patterns and Observability

Timeouts, retries with jitter, circuit breakers, bulkheads, rate limiting algorithms, SLIs/SLOs/SLAs and the three pillars of observability.

7 min read

In short

  • Timeouts on every call, shrinking with depth
  • Retry transient errors at one layer, with backoff, jitter and a budget
  • Circuit breakers fail fast with a fallback
  • Bulkheads keep one slow dependency contained
  • SLOs and error budgets; alert on symptoms

At scale, something is always failing: a disk, a node, a dependency, a deploy. Reliability engineering is about containing failures so one broken part doesn't take down the whole, and seeing problems before users report them.

Timeouts: the first line of defense

Every remote call needs a timeout. Without one, a slow dependency ties up threads and connections until your service falls over too (cascading failure).

  • Set timeouts from the dependency's p99 latency plus margin (e.g. p99 = 80 ms → timeout 250 ms), not "30 s by default".
  • Keep connect timeout (short, ~100 ms-1 s) separate from read/request timeout.
  • Deadline propagation: if the user request has a 2 s budget and 1.5 s is used, downstream calls get the remaining 0.5 s at most. gRPC does this natively, and HTTP can carry it in a header.
  • Timeouts should shrink as you go deeper. The outermost layer has the longest.

Retries with exponential backoff and jitter

Retries fix transient errors (network blips, 503, leader failover) but amplify load when the system is already struggling.

▶ Try
async function withRetry<T>(fn: () => Promise<T>, attempts = 4, baseMs = 100, capMs = 2000): Promise<T> {
  for (let i = 0; ; i++) {
    try {
      return await fn();
    } catch (e) {
      if (i >= attempts - 1 || !isTransient(e)) throw e;
      const backoff = Math.min(capMs, baseMs * 2 ** i);
      await sleep(Math.random() * backoff); // "full jitter": spreads retries out
    }
  }
}

Rules:

  • Retry only idempotent operations, or make them idempotent with an idempotency key.
  • Retry only transient errors (timeouts, 429/503, connection reset). Never retry 400/401/404.
  • Jitter prevents synchronized retry waves (thundering herd).
  • Retry at one layer. If 3 layers each make up to 3 attempts, a failing backend sees 3 × 3 × 3 = 27x the traffic.
  • Use retry budgets: cap retries at ~10% of requests, so a global outage doesn't triple your load.
  • Honor Retry-After headers.

Circuit breakers

A circuit breaker stops calling a dependency that is clearly failing, fails fast, and gives it room to recover.

          failures > threshold
 CLOSED ─────────────────────────► OPEN  (fail fast, return fallback)
   ▲                                  │ after cool-down (e.g. 30 s)
   │ trial calls succeed              ▼
   └──────────────────────────── HALF-OPEN (let a few requests through)
                 trial fails ──► back to OPEN
  • Trip on error rate or slow-call rate over a rolling window (e.g. >50% failures over at least 20 calls).
  • Pair it with a fallback: cached data, a default value, a degraded feature ("recommendations unavailable"), or a queued write.
  • Libraries: Polly (.NET), resilience4j (Java), opossum (Node), or service-mesh features (Envoy outlier detection).

Bulkheads

Ships have watertight compartments so one leak doesn't sink the vessel. In software:

  • Separate connection/thread pools per dependency, so a slow payment provider can't exhaust the pool used for catalog reads.
  • Separate deployments/queues per workload (transactional vs marketing notifications, premium vs free tenants).
  • Cell-based architecture: split users into independent cells (stacks), so a bad deploy or poison request affects one cell's users only.

Graceful degradation and load shedding

  • Decide in advance what to drop first: recommendations, analytics, avatars. Keep the core path working (checkout, login).
  • Load shedding: when saturated, reject excess requests early with 503/429, prioritizing by request class. Serving 80% of requests well beats serving 100% badly.
  • Feature flags / kill switches to turn off expensive features under load.

Rate limiting algorithms

AlgorithmHowMemory per keyBurstsAccuracy
Token bucketBucket of capacity B refills at rate R; each request takes a token2 numbersAllows bursts up to BGood
Leaky bucketRequests drain from a queue at a constant rateQueue / counterSmooths output, no burstsGood
Fixed windowCounter per minute window1 counterUp to 2x the limit at window boundariesWeak at edges
Sliding window logStore a timestamp per request; count those within the windowO(requests)ExactExact, memory-heavy
Sliding window counterWeighted current + previous window counters2 countersSmooth~Approximate, very good in practice

Token bucket is the usual default (API gateways, AWS, Stripe). Return 429 Too Many Requests with Retry-After and rate-limit headers. See the Rate Limiter design problem for the distributed version.

Redundancy and recovery targets

  • N+1 / N+2 capacity: survive the loss of one instance or one AZ without overload.
  • Multi-AZ is the baseline for production. Multi-region active-passive (failover) or active-active (both serve) protects against regional outages, at a cost in complexity and consistency.
  • RTO (recovery time objective): how long you may be down. RPO (recovery point objective): how much data you may lose. Async cross-region replication means RPO > 0.
  • Test it: game days, chaos experiments, restore drills. A backup you have never restored isn't a backup.

SLI, SLO, SLA

TermMeaningExample
SLI (indicator)A measured ratio of good events% of requests with status < 500 and latency < 300 ms
SLO (objective)Internal target for an SLI over a window99.9% of requests good over 28 days
SLA (agreement)Contract with customers, with penalties99.5% monthly uptime or service credits
Error budget100% − SLO0.1% ≈ 40 min of "bad" per 28 days to spend on risk (deploys, experiments)
AvailabilityDowntime / yearDowntime / 30 days
99%3.65 days7.2 h
99.9%8.8 h43 min
99.95%4.4 h22 min
99.99%53 min4.3 min
99.999%5.3 min26 s

Serial dependencies multiply: a service calling 3 dependencies at 99.9% each can be at most ~99.7% available unless it degrades gracefully.

Observability: logs, metrics, traces

PillarWhatGood forTips
MetricsNumeric time series (counters, gauges, histograms)Dashboards, alerting, trendsLow cost per data point; watch label cardinality
LogsDiscrete structured events (JSON)Debugging specific requests, auditsInclude request/correlation ID; sample noisy logs
TracesSpans across services for one requestFinding where latency/errors come fromOpenTelemetry; propagate trace context through queues too

Useful lenses:

  • RED for services: Rate, Errors, Duration (latency histograms, p50/p95/p99).
  • USE for resources: Utilization, Saturation, Errors (CPU, memory, disk, connection pools, queue depth).
  • Golden signals: latency, traffic, errors, saturation.

Alerting:

  • Alert on symptoms users feel (SLO burn rate, error rate, latency) rather than causes (CPU at 80%).
  • Every page must be actionable and link a runbook.
  • Use multi-window burn-rate alerts on SLOs (fast burn pages, slow burn files a ticket).

When to use / trade-offs

  • Timeouts: always. The trade-off is setting them too low (false failures) vs too high (resource exhaustion).
  • Retries: for transient errors on idempotent operations. Without backoff, jitter and budgets they cause retry storms.
  • Circuit breakers: around remote dependencies that can fail independently. They add tuning and fallback logic.
  • Bulkheads: when workloads have different criticality. You pay with lower resource utilization.
  • Higher SLOs cost exponentially more (redundancy, multi-region, process). Pick the SLO users actually need.
  • Observability costs money (log volume, trace storage, metric cardinality). Sample traces, aggregate metrics, and keep logs structured but lean.

Checklist

  • Timeouts on every remote call, with deadline propagation
  • Retries: transient-only, backoff + jitter, single layer, budgeted
  • Circuit breakers + fallbacks on critical dependencies
  • Bulkheads between critical and non-critical workloads
  • Rate limiting / load shedding at the edge
  • SLIs and SLOs defined, error budget policy
  • Metrics (RED/USE), structured logs with correlation IDs, distributed tracing
  • Multi-AZ, backups tested, RTO/RPO stated

Practise this concept

In the app: a short video and a 11-question concept check. Free with a GitHub account.

Open in the app