System design · Concept 8
Reliability Patterns and Observability
Timeouts, retries with jitter, circuit breakers, bulkheads, rate limiting algorithms, SLIs/SLOs/SLAs and the three pillars of observability.
In short
- Timeouts on every call, shrinking with depth
- Retry transient errors at one layer, with backoff, jitter and a budget
- Circuit breakers fail fast with a fallback
- Bulkheads keep one slow dependency contained
- SLOs and error budgets; alert on symptoms
At scale, something is always failing: a disk, a node, a dependency, a deploy. Reliability engineering is about containing failures so one broken part doesn't take down the whole, and seeing problems before users report them.
Timeouts: the first line of defense
Every remote call needs a timeout. Without one, a slow dependency ties up threads and connections until your service falls over too (cascading failure).
- Set timeouts from the dependency's p99 latency plus margin (e.g. p99 = 80 ms → timeout 250 ms), not "30 s by default".
- Keep connect timeout (short, ~100 ms-1 s) separate from read/request timeout.
- Deadline propagation: if the user request has a 2 s budget and 1.5 s is used, downstream calls get the remaining 0.5 s at most. gRPC does this natively, and HTTP can carry it in a header.
- Timeouts should shrink as you go deeper. The outermost layer has the longest.
Retries with exponential backoff and jitter
Retries fix transient errors (network blips, 503, leader failover) but amplify load when the system is already struggling.
async function withRetry<T>(fn: () => Promise<T>, attempts = 4, baseMs = 100, capMs = 2000): Promise<T> {
for (let i = 0; ; i++) {
try {
return await fn();
} catch (e) {
if (i >= attempts - 1 || !isTransient(e)) throw e;
const backoff = Math.min(capMs, baseMs * 2 ** i);
await sleep(Math.random() * backoff); // "full jitter": spreads retries out
}
}
}
Rules:
- Retry only idempotent operations, or make them idempotent with an idempotency key.
- Retry only transient errors (timeouts, 429/503, connection reset). Never retry 400/401/404.
- Jitter prevents synchronized retry waves (thundering herd).
- Retry at one layer. If 3 layers each make up to 3 attempts, a failing backend sees 3 × 3 × 3 = 27x the traffic.
- Use retry budgets: cap retries at ~10% of requests, so a global outage doesn't triple your load.
- Honor
Retry-Afterheaders.
Circuit breakers
A circuit breaker stops calling a dependency that is clearly failing, fails fast, and gives it room to recover.
failures > threshold
CLOSED ─────────────────────────► OPEN (fail fast, return fallback)
▲ │ after cool-down (e.g. 30 s)
│ trial calls succeed ▼
└──────────────────────────── HALF-OPEN (let a few requests through)
trial fails ──► back to OPEN
- Trip on error rate or slow-call rate over a rolling window (e.g. >50% failures over at least 20 calls).
- Pair it with a fallback: cached data, a default value, a degraded feature ("recommendations unavailable"), or a queued write.
- Libraries: Polly (.NET), resilience4j (Java), opossum (Node), or service-mesh features (Envoy outlier detection).
Bulkheads
Ships have watertight compartments so one leak doesn't sink the vessel. In software:
- Separate connection/thread pools per dependency, so a slow payment provider can't exhaust the pool used for catalog reads.
- Separate deployments/queues per workload (transactional vs marketing notifications, premium vs free tenants).
- Cell-based architecture: split users into independent cells (stacks), so a bad deploy or poison request affects one cell's users only.
Graceful degradation and load shedding
- Decide in advance what to drop first: recommendations, analytics, avatars. Keep the core path working (checkout, login).
- Load shedding: when saturated, reject excess requests early with
503/429, prioritizing by request class. Serving 80% of requests well beats serving 100% badly. - Feature flags / kill switches to turn off expensive features under load.
Rate limiting algorithms
| Algorithm | How | Memory per key | Bursts | Accuracy |
|---|---|---|---|---|
| Token bucket | Bucket of capacity B refills at rate R; each request takes a token | 2 numbers | Allows bursts up to B | Good |
| Leaky bucket | Requests drain from a queue at a constant rate | Queue / counter | Smooths output, no bursts | Good |
| Fixed window | Counter per minute window | 1 counter | Up to 2x the limit at window boundaries | Weak at edges |
| Sliding window log | Store a timestamp per request; count those within the window | O(requests) | Exact | Exact, memory-heavy |
| Sliding window counter | Weighted current + previous window counters | 2 counters | Smooth | ~Approximate, very good in practice |
Token bucket is the usual default (API gateways, AWS, Stripe). Return 429 Too Many Requests with Retry-After and rate-limit headers. See the Rate Limiter design problem for the distributed version.
Redundancy and recovery targets
- N+1 / N+2 capacity: survive the loss of one instance or one AZ without overload.
- Multi-AZ is the baseline for production. Multi-region active-passive (failover) or active-active (both serve) protects against regional outages, at a cost in complexity and consistency.
- RTO (recovery time objective): how long you may be down. RPO (recovery point objective): how much data you may lose. Async cross-region replication means RPO > 0.
- Test it: game days, chaos experiments, restore drills. A backup you have never restored isn't a backup.
SLI, SLO, SLA
| Term | Meaning | Example |
|---|---|---|
| SLI (indicator) | A measured ratio of good events | % of requests with status < 500 and latency < 300 ms |
| SLO (objective) | Internal target for an SLI over a window | 99.9% of requests good over 28 days |
| SLA (agreement) | Contract with customers, with penalties | 99.5% monthly uptime or service credits |
| Error budget | 100% − SLO | 0.1% ≈ 40 min of "bad" per 28 days to spend on risk (deploys, experiments) |
| Availability | Downtime / year | Downtime / 30 days |
|---|---|---|
| 99% | 3.65 days | 7.2 h |
| 99.9% | 8.8 h | 43 min |
| 99.95% | 4.4 h | 22 min |
| 99.99% | 53 min | 4.3 min |
| 99.999% | 5.3 min | 26 s |
Serial dependencies multiply: a service calling 3 dependencies at 99.9% each can be at most ~99.7% available unless it degrades gracefully.
Observability: logs, metrics, traces
| Pillar | What | Good for | Tips |
|---|---|---|---|
| Metrics | Numeric time series (counters, gauges, histograms) | Dashboards, alerting, trends | Low cost per data point; watch label cardinality |
| Logs | Discrete structured events (JSON) | Debugging specific requests, audits | Include request/correlation ID; sample noisy logs |
| Traces | Spans across services for one request | Finding where latency/errors come from | OpenTelemetry; propagate trace context through queues too |
Useful lenses:
- RED for services: Rate, Errors, Duration (latency histograms, p50/p95/p99).
- USE for resources: Utilization, Saturation, Errors (CPU, memory, disk, connection pools, queue depth).
- Golden signals: latency, traffic, errors, saturation.
Alerting:
- Alert on symptoms users feel (SLO burn rate, error rate, latency) rather than causes (CPU at 80%).
- Every page must be actionable and link a runbook.
- Use multi-window burn-rate alerts on SLOs (fast burn pages, slow burn files a ticket).
When to use / trade-offs
- Timeouts: always. The trade-off is setting them too low (false failures) vs too high (resource exhaustion).
- Retries: for transient errors on idempotent operations. Without backoff, jitter and budgets they cause retry storms.
- Circuit breakers: around remote dependencies that can fail independently. They add tuning and fallback logic.
- Bulkheads: when workloads have different criticality. You pay with lower resource utilization.
- Higher SLOs cost exponentially more (redundancy, multi-region, process). Pick the SLO users actually need.
- Observability costs money (log volume, trace storage, metric cardinality). Sample traces, aggregate metrics, and keep logs structured but lean.
Checklist
- Timeouts on every remote call, with deadline propagation
- Retries: transient-only, backoff + jitter, single layer, budgeted
- Circuit breakers + fallbacks on critical dependencies
- Bulkheads between critical and non-critical workloads
- Rate limiting / load shedding at the edge
- SLIs and SLOs defined, error budget policy
- Metrics (RED/USE), structured logs with correlation IDs, distributed tracing
- Multi-AZ, backups tested, RTO/RPO stated
Practise this concept
In the app: a short video and a 11-question concept check. Free with a GitHub account.
Open in the app