System design · Concept 14
Observability and Delivery
Logs, metrics and traces with OpenTelemetry, correlation across services and queues, cardinality, sampling, SLO burn-rate alerts, runbooks and health checks; then CI/CD, infrastructure as code, pipelines that sign in with OIDC, environments, rolling, blue-green and canary releases, feature flags, expand-and-contract migrations and rollback.
In short
- OpenTelemetry everywhere; trace context through queues and outboxes
- Bounded metric labels; counts from metrics; sample traces
- Page on SLO burn rate, with a runbook; liveness checks only the process
- Infrastructure as code; build once; OIDC, one identity per environment
- Canaries, sticky flags and expand and contract, with a practised rollback
Two habits separate teams that ship calmly from teams that fear Fridays: they can see one request across every service it touched, and they can change production in small, reversible steps. This lesson covers both as patterns, then maps them onto AWS, Azure and Google Cloud. The basics (SLI, SLO, SLA, RED and USE, the three pillars) are in Reliability Patterns and Observability.
Part 1: Observability
Logs, metrics and traces
| Signal | What it is | Answers | Cost driver |
|---|---|---|---|
| Metrics | Numbers over time (counters, gauges, histograms), aggregated before storage | "Is it healthy? Is it getting worse?" | Number of distinct time series (cardinality) |
| Logs | Discrete events with fields | "What exactly happened to this request?" | Volume (GB ingested and retained) |
| Traces | A tree of timed spans for one request across services | "Where did the time go, and which hop failed?" | Number of spans kept (sampling) |
They work best linked: a dashboard metric spikes, you jump to example traces from that minute, and from a slow span to the logs that carry the same trace ID.
OpenTelemetry: instrument once
OpenTelemetry (OTel) is the vendor-neutral, CNCF standard for producing telemetry: an API and SDKs for most languages, automatic instrumentation for common frameworks (HTTP servers and clients, database drivers, messaging SDKs), the OTLP wire protocol, and the Collector, a process that receives, filters, samples and forwards telemetry. Your code talks to the OTel API; where the data goes is configuration. All three clouds and the main vendors accept it, so changing backends doesn't mean re-instrumenting.
// A custom span around business logic (illustrative; the SDK is set up at startup)
import { trace, SpanStatusCode } from '@opentelemetry/api';
const tracer = trace.getTracer('orders');
export function reserveStock(orderId: string, lines: OrderLine[]) {
return tracer.startActiveSpan('reserve-stock', async (span) => { // child of the current request
span.setAttribute('order.id', orderId);
span.setAttribute('order.line_count', lines.length);
try {
return await inventory.reserve(orderId, lines);
} catch (err) {
span.recordException(err as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw err;
} finally {
span.end();
}
});
}
Configure each service with the standard environment variables rather than code:
OTEL_SERVICE_NAME=orders-api # one name per service, or the service map is one blob
OTEL_RESOURCE_ATTRIBUTES=service.version=1.42.0,deployment.environment.name=prod
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1 # keep 10% of new traces; follow the parent's decision
service.version is what lets you compare a canary with the baseline later.
Correlation and distributed tracing
Every request gets a trace ID at the edge, and each hop passes it on. The W3C Trace Context standard carries it in the traceparent header (00-<trace-id>-<parent-span-id>-<flags>), and OTel instrumentation reads and writes it for you over HTTP and gRPC.
Browser trace 4bf9...
└─ POST /api/orders traceparent: 00-4bf9...-<span>-01
└─ orders-api: request span ── SQL span
└─ queue send span (traceparent stored in the message's headers or attributes)
└─ payments-worker: process span ── HTTP call to the payment provider
- Queues: put the trace context in message headers or attributes when sending, and extract it when processing. Many messaging SDKs and their OTel instrumentation do this; check yours. For batch consumers, OTel's messaging conventions use span links to each message rather than one parent.
- Anything the instrumentation can't see is your job: an outbox table (store the current
traceparentwith the row, and use it as the parent when the relay sends), a webhook to another system, a scheduled job that picks up earlier work. Otherwise the trace splits into unrelated pieces exactly where incidents happen. - Browsers: a front-end SDK can start the trace and send
traceparentwithfetch. If the API is on another origin, the API's CORS policy must allow thetraceparentandtracestaterequest headers, or the preflight fails. Don't send them to third-party domains. - Return the trace ID to clients (a response header, or on error pages as a "support code"), so a bug report leads straight to the trace.
Structured logging
Log events with fields, not sentences to parse later.
{ "timestamp": "2026-10-23T09:14:02.381Z", "level": "error", "service": "orders-api",
"message": "payment declined", "order_id": "o_8812", "provider_code": "card_declined",
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" }
- Put the trace and span IDs on every log line (OTel's logging bridges do this), so logs and traces join.
- Use consistent field names across services, and log levels on purpose:
infofor business events,debugoff in production unless switched on. - Never log secrets, tokens, full card numbers or more personal data than you need. Logs get copied to more places, and kept longer, than your database.
Cardinality
A metric's cost is its number of time series: one per unique combination of label values. http_requests_total{route, method, status} with 50 routes, 4 methods and 6 status codes is at most 1,200 series. Add user_id with a million users and it's over a billion, which no metrics store will thank you for.
- Labels on metrics should have bounded values: route templates (
/orders/{id}, not/orders/8812), status class, region, version. - High-cardinality detail (user ID, order ID, full URL) belongs on spans and log lines, which are stored per event and searched rather than pre-aggregated.
Sampling
Traces of every request are expensive, so traces are sampled. Metrics are aggregated, so they are never sampled, which is why request and error counts should come from metrics.
| Kind | Decided | Keeps | Trade-off |
|---|---|---|---|
| Head sampling | When the trace starts (for example 10% by trace ID) | A random share of traces, consistently across services via the traceparent flag | Cheap and simple, but may drop the one failing request you need |
| Tail sampling | After the trace is complete, in a Collector | All errors and slow traces, plus a share of the rest | Needs a Collector tier that sees every span of a trace before deciding |
- Use parent-based sampling in every service, so a kept trace keeps all its spans. Mixed rules leave traces with holes.
- If you count from sampled data, weight each kept item by the sampling rate. Some backends store this for you (Application Insights records it as
ItemCount, so queries usesum(ItemCount)instead ofcount()). - Other cost levers: drop health-check requests and chatty dependencies at the Collector, lower log levels, shorter retention or cheaper storage tiers for verbose logs. An ingestion cap protects the bill but stops collection mid-incident, so treat it as a safety net set well above normal, with an alert before it's reached.
Query languages
Each platform has its own, but the moves are the same: filter by time, group, aggregate, take percentiles.
| Platform | Language |
|---|---|
| AWS CloudWatch Logs | Logs Insights query language |
| Azure Monitor (Log Analytics, Application Insights) | KQL |
| Google Cloud Logging | Logging query language; SQL in Log Analytics |
| Prometheus (and managed Prometheus on every cloud) | PromQL |
// KQL: p95 latency and failures per operation in the last hour (sampling-aware)
AppRequests
| where TimeGenerated > ago(1h)
| summarize requests = sum(ItemCount), failures = sumif(ItemCount, Success == false),
p95 = percentile(DurationMs, 95) by Name
| order by failures desc
SLOs, error budgets and burn rate
Define the SLI as a ratio of good events (for example "checkout requests that succeed in under 800 ms"), set an SLO over a window (99.9% over 30 days), and the error budget is the rest (0.1%). Burn rate is how fast you're spending it: observed error rate ÷ budget rate. A burn rate of 1 spends exactly the budget over the window.
# Checkout burn rate over the last hour (alert when > 14.4)
(
sum(rate(http_requests_total{route="/api/orders", code=~"5.."}[1h]))
/
sum(rate(http_requests_total{route="/api/orders"}[1h]))
) / (1 - 0.999)
Common multi-window thresholds for a 30-day SLO, from the Google SRE workbook: page at 14.4× sustained over an hour (2% of the monthly budget gone in one hour) or at 6× over six hours (5% in six hours), and ticket at 1× over three days (10% in three days). Add a short window of 1/12 of the long one to each (5 minutes for the 1-hour alert, 30 minutes for the 6-hour one, 6 hours for the 3-day ticket), so the alert also stops quickly once the problem is fixed.
Alerting, dashboards and runbooks
- Alert on symptoms users feel: SLO burn rate, error rate, latency, the age of the oldest message in a queue, dead-lettered messages. CPU at 85% is a cause; it belongs on a dashboard, not a pager.
- Every page is actionable and links a runbook: what the alert means, the dashboards and queries to check first, safe mitigations (roll back, turn a flag off, scale out, fail over) and who to escalate to.
- Dashboards: one per service with RED (rate, errors, duration) and its key dependencies, with deploy markers so "it started at 14:05" lines up with "we deployed at 14:04".
- Synthetic checks from outside (an uptime probe hitting a real endpoint) catch the failures your own telemetry can't see, like DNS or certificate expiry.
Health checks: liveness and readiness
| Probe | Question | On failure | Should check |
|---|---|---|---|
| Liveness | Is this process stuck beyond repair? | The platform restarts it | Only the process itself (it responds, the event loop isn't blocked) |
| Readiness | Can this instance take traffic right now? | It's removed from the load balancer until it passes | Warm-up finished, critical config loaded, not overloaded |
| Startup | Has it finished starting? | Liveness waits until it passes | Slow initialisation |
Never put a shared dependency (the database, a downstream API) in liveness: a 30-second database blip would restart every instance at once and turn a blip into an outage. Be careful in readiness too: if every instance fails readiness for the same shared dependency, the load balancer has nowhere to send traffic, and serving a clear error or degraded response is often better. Kubernetes names these three probes; load balancers and managed platforms on every cloud have equivalent health checks.
Part 2: Delivery
CI/CD pipelines
- CI runs on every change: build, unit tests, linting, a security scan, and produces one immutable artifact (a container image identified by its digest, or a versioned package).
- CD promotes that same artifact through environments: build once, deploy many. Only configuration changes between dev, staging and production. Rebuilding per environment means production runs something nobody tested.
- Keep pipelines as code in the repository, fast (cache dependencies, run tests in parallel), and the only way to change production.
- GitOps is a variant where the desired state lives in Git and an agent in the cluster (Argo CD, Flux) pulls and applies it.
Infrastructure as code
Every resource is declared in files, reviewed in pull requests, and applied by the pipeline. Manual changes in a console become drift, which the next apply either reverts or trips over.
| Tool | Scope | State | Notes |
|---|---|---|---|
| Terraform / OpenTofu | Any provider: all clouds, GitHub, DNS, monitoring | A state file in a remote backend with locking (a storage bucket, or HCP Terraform) | Declarative HCL; plan previews changes. OpenTofu is an open-source fork of Terraform |
| Pulumi | Any provider | Pulumi Cloud or a self-managed backend | General-purpose languages (TypeScript, Python, Go, C#) |
| AWS CloudFormation / CDK | AWS | AWS keeps the stack state | CDK writes CloudFormation from code; change sets preview |
| Bicep (ARM) | Azure | Azure is the state | what-if previews; compiles to ARM JSON |
| Infrastructure Manager | Google Cloud | Managed for you | Google's managed service for running Terraform; replaces the deprecated Deployment Manager |
- Show the plan in the pull request (
terraform plan, a change set,what-if) so reviewers see infrastructure changes, not only code. - State files contain secrets (generated passwords, connection strings). Store them encrypted, locked and access-controlled, never in Git.
- Know what removal does. Terraform, OpenTofu, Pulumi and CloudFormation delete a resource that leaves the code (unless you mark it to be retained). Azure deployments are incremental by default, so a resource removed from a Bicep file stays in Azure unless you use a deployment stack.
- One owner per resource. Mixing tools is fine across resources, never for the same one.
Secrets in pipelines: OIDC workload identity federation
The old way was a cloud access key or service principal secret in a pipeline variable: long-lived, copied around, rarely rotated, and valid from anywhere. Workload identity federation removes it. The CI system issues a short-lived OIDC token for the running job, signed by its issuer, with claims about the repository, branch and environment. The cloud exchanges it for short-lived cloud credentials because you configured a trust that matches exactly that issuer and subject.
| Cloud | You configure | The job uses |
|---|---|---|
| AWS | An IAM OIDC identity provider and a role whose trust policy checks the sub claim | AssumeRoleWithWebIdentity (for example aws-actions/configure-aws-credentials) |
| Azure | A federated identity credential on a user-assigned managed identity or app registration | azure/login with client, tenant and subscription IDs |
| Google Cloud | A workload identity pool and provider, with an attribute condition | google-github-actions/auth with the provider's name |
# GitHub Actions (illustrative): no cloud secret stored anywhere
permissions:
id-token: write # lets the job request an OIDC token
contents: read
jobs:
deploy-prod:
runs-on: ubuntu-latest
environment: production # required reviewers; the token's subject ends in :environment:production
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::111122223333:role/deploy-orders-prod
aws-region: eu-central-1
- run: ./deploy.sh prod
- The subject reflects the job:
repo:acme/orders:ref:refs/heads/mainfor a branch,repo:acme/orders:environment:productionwhen the job targets an environment (the environment takes precedence),repo:acme/orders:pull_requestfor pull requests. The trust must match it exactly, or sign-in fails. - Use one identity per environment, trusted only for that environment's subject. Then only jobs that passed the production environment's protection rules can touch production, and a pull request workflow can't.
- Azure DevOps, GitLab and other CI systems offer the same federation. The pattern doesn't change: a short-lived token, an exact trust, least privilege.
- Application secrets are a separate concern: the running service reads them from a secret store with its own workload identity at runtime. Don't bake them into images or pipeline logs.
Environments and promotion
- Separate accounts, subscriptions or projects per environment where you can, so a mistake in dev can't reach production, and each environment has its own identities.
- Promotion gates: automated tests and checks first, then human approval only where it adds value (production, or regulated changes).
- Staging is as close to production as you can afford: same IaC, same artifact, smaller scale. Differences between them are where surprises come from.
- Humans get production access just in time, for a reason and a limited time, and it's logged.
Deployment strategies
| Strategy | How it works | Rollback | Watch out for |
|---|---|---|---|
| Rolling | Replace instances a few at a time | Roll forward or redeploy the previous version (slow-ish) | Both versions serve traffic during the roll |
| Blue-green | Bring up the new version beside the old, test it, switch all traffic at once | Switch back | Double capacity during the switch; shared database |
| Canary | Send a small share of real traffic (1-5%) to the new version, compare, then increase | Set its weight to 0 | Needs per-version metrics and enough traffic to compare |
| Feature flag | Same binary everywhere; behaviour turned on per audience | Turn the flag off | Flag debt, and testing both paths |
What makes these safe in practice:
- Compare the canary with the baseline running at the same time, not with last week: filter by
service.version, and gate each step on error rate and p95, automatically. - Traffic splitting is per request unless you add session affinity. A user can bounce between versions, and a single-page app can ask for a JavaScript chunk that only the other version has. Keep old assets around for a while, use affinity, or release with a flag.
- Message contracts must be readable by both versions: a new producer can't send a field an old consumer will choke on.
Feature flags
Flags separate deploying code from releasing behaviour: ship dark, turn on for staff, then 5%, then everyone, and turn off in seconds without a deploy. That's also the cheapest kill switch you'll have during an incident.
- Percentage rollouts must be sticky: bucket by a hash of the user ID and flag name, so each user stays consistently in or out. A fresh random decision on every evaluation makes the same user flip between versions.
- Evaluate flags on the server (or a back end for the front end) and send the result to the browser. Don't ship flag-service credentials to clients.
- Decide the default when the flag service is unreachable (usually "off"), test both paths, and delete flags when a rollout finishes. OpenFeature is the vendor-neutral flag API, the OpenTelemetry of flags.
Database migrations alongside code
During any rolling, blue-green or canary release, two versions run against one database, and a rollback means the old code runs against the new schema. So every schema change must work with both. Use expand and contract across releases:
- Expand: add the new column or table (nullable or with a default). The new release writes both old and new.
- Migrate: backfill existing rows in batches, then switch reads to the new column.
- Contract: once no running version uses the old column, drop it in a later release.
Renaming a column in one step breaks whichever version expects the other name, whatever order you run things in. Run migrations as their own pipeline step, idempotent and short, and keep long backfills out of the deploy's critical path.
Rollback
- Rolling back must be faster than rolling forward. Keep the previous artifact, redeploy it with one command (or shift traffic back), and practise it.
- Rollback undoes code, not data. Schema changes and data written by the new version must stay compatible with the old one, which is what expand and contract buys you.
- Decide in advance what triggers a rollback (the canary's error rate or burn rate over a threshold), so nobody debates it at 3 a.m. Then fix forward calmly.
Cloud mappings
Side note
| Need | AWS | Azure | Google Cloud | Other |
|---|---|---|---|---|
| Metrics, logs, dashboards, alerts | CloudWatch (metrics, Logs, alarms, dashboards) | Azure Monitor, Log Analytics | Cloud Monitoring, Cloud Logging | Prometheus + Grafana, Datadog, New Relic |
| Tracing and APM | X-Ray, CloudWatch Application Signals | Application Insights | Cloud Trace | Jaeger, Grafana Tempo, Honeycomb, Datadog APM |
| OpenTelemetry path | AWS Distro for OpenTelemetry (ADOT) | Azure Monitor OpenTelemetry Distro | OpenTelemetry Collector with Google Cloud exporters | OpenTelemetry Collector to any OTLP backend |
| Managed Prometheus and Grafana | Amazon Managed Service for Prometheus, Amazon Managed Grafana | Azure Monitor managed service for Prometheus, Azure Managed Grafana | Google Cloud Managed Service for Prometheus | Grafana Cloud, self-hosted |
| Synthetic checks | CloudWatch Synthetics | Application Insights availability tests | Cloud Monitoring uptime checks | Checkly, Datadog Synthetics |
| Infrastructure as code | CloudFormation, CDK | Bicep, ARM templates | Infrastructure Manager | Terraform, OpenTofu, Pulumi |
| CI/CD | CodePipeline, CodeBuild | Azure Pipelines (Azure DevOps) | Cloud Build, Cloud Deploy | GitHub Actions, GitLab CI/CD, Jenkins, Argo CD |
| Progressive delivery | CodeDeploy, ALB weighted target groups, Lambda alias weights | App Service deployment slots, Container Apps traffic splitting | Cloud Deploy canaries, Cloud Run traffic splitting | Kubernetes rolling updates, Argo Rollouts, Flagger |
| Pipeline sign-in without secrets | IAM OIDC identity provider and role | Entra federated identity credential | Workload Identity Federation | Vault JWT/OIDC auth |
| Feature flags | AWS AppConfig | Azure App Configuration | Firebase Remote Config | LaunchDarkly, Unleash, Flagsmith (OpenFeature) |
How to say it in a design review
"Every service uses OpenTelemetry with its own service.name and service.version, exporting through a Collector that tail-samples to keep every error. traceparent goes from the browser to the API and rides in message headers and outbox rows, so one trace ID covers browser to payment provider, and every log line carries it. Request and error counts come from metrics, with no user IDs in labels. We page on a 14.4× burn rate of the checkout SLO and on dead-lettered messages, each with a runbook. Liveness checks only the process. Infrastructure is Terraform with the plan on every pull request; pipelines sign in with OIDC federation, one identity per environment, and production sits behind required reviewers. We build one image, release it as a 5%, 25%, 100% canary with automatic checks against the baseline, put new behaviour behind sticky flags, and change schemas with expand and contract so rollback is always safe."
Checklist
- OpenTelemetry in every service with
service.nameandservice.version; logs carry trace and span IDs - Trace context through HTTP, queues, outboxes and webhooks; trace ID returned to clients
- Bounded metric labels; counts from metrics; sampling chosen on purpose (tail sampling to keep errors)
- SLIs and SLOs with burn-rate alerts that page; symptoms on the pager, causes on dashboards; a runbook per alert
- Liveness checks the process only; readiness gates traffic; synthetic checks from outside
- All infrastructure in code, plan shown in pull requests, state protected, one owner per resource
- Build once, deploy many; pipelines sign in with OIDC federation, one identity per environment, no stored secrets
- Canary or blue-green with automatic comparison to the baseline; sticky flags with safe defaults and cleanup
- Expand and contract for schema changes; a practised, one-step rollback
Practise this concept
In the app: a short video and a 12-question concept check. Free with a GitHub account.
Open in the app