System design · Concept 14

Observability and Delivery

Logs, metrics and traces with OpenTelemetry, correlation across services and queues, cardinality, sampling, SLO burn-rate alerts, runbooks and health checks; then CI/CD, infrastructure as code, pipelines that sign in with OIDC, environments, rolling, blue-green and canary releases, feature flags, expand-and-contract migrations and rollback.

18 min read

In short

  • OpenTelemetry everywhere; trace context through queues and outboxes
  • Bounded metric labels; counts from metrics; sample traces
  • Page on SLO burn rate, with a runbook; liveness checks only the process
  • Infrastructure as code; build once; OIDC, one identity per environment
  • Canaries, sticky flags and expand and contract, with a practised rollback

Two habits separate teams that ship calmly from teams that fear Fridays: they can see one request across every service it touched, and they can change production in small, reversible steps. This lesson covers both as patterns, then maps them onto AWS, Azure and Google Cloud. The basics (SLI, SLO, SLA, RED and USE, the three pillars) are in Reliability Patterns and Observability.

Part 1: Observability

Logs, metrics and traces

SignalWhat it isAnswersCost driver
MetricsNumbers over time (counters, gauges, histograms), aggregated before storage"Is it healthy? Is it getting worse?"Number of distinct time series (cardinality)
LogsDiscrete events with fields"What exactly happened to this request?"Volume (GB ingested and retained)
TracesA tree of timed spans for one request across services"Where did the time go, and which hop failed?"Number of spans kept (sampling)

They work best linked: a dashboard metric spikes, you jump to example traces from that minute, and from a slow span to the logs that carry the same trace ID.

OpenTelemetry: instrument once

OpenTelemetry (OTel) is the vendor-neutral, CNCF standard for producing telemetry: an API and SDKs for most languages, automatic instrumentation for common frameworks (HTTP servers and clients, database drivers, messaging SDKs), the OTLP wire protocol, and the Collector, a process that receives, filters, samples and forwards telemetry. Your code talks to the OTel API; where the data goes is configuration. All three clouds and the main vendors accept it, so changing backends doesn't mean re-instrumenting.

// A custom span around business logic (illustrative; the SDK is set up at startup)
import { trace, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('orders');

export function reserveStock(orderId: string, lines: OrderLine[]) {
  return tracer.startActiveSpan('reserve-stock', async (span) => {   // child of the current request
    span.setAttribute('order.id', orderId);
    span.setAttribute('order.line_count', lines.length);
    try {
      return await inventory.reserve(orderId, lines);
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: SpanStatusCode.ERROR });
      throw err;
    } finally {
      span.end();
    }
  });
}

Configure each service with the standard environment variables rather than code:

OTEL_SERVICE_NAME=orders-api                         # one name per service, or the service map is one blob
OTEL_RESOURCE_ATTRIBUTES=service.version=1.42.0,deployment.environment.name=prod
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1                          # keep 10% of new traces; follow the parent's decision

service.version is what lets you compare a canary with the baseline later.

Correlation and distributed tracing

Every request gets a trace ID at the edge, and each hop passes it on. The W3C Trace Context standard carries it in the traceparent header (00-<trace-id>-<parent-span-id>-<flags>), and OTel instrumentation reads and writes it for you over HTTP and gRPC.

Browser                    trace 4bf9...
  └─ POST /api/orders      traceparent: 00-4bf9...-<span>-01
       └─ orders-api: request span ── SQL span
            └─ queue send span        (traceparent stored in the message's headers or attributes)
                 └─ payments-worker: process span ── HTTP call to the payment provider
  • Queues: put the trace context in message headers or attributes when sending, and extract it when processing. Many messaging SDKs and their OTel instrumentation do this; check yours. For batch consumers, OTel's messaging conventions use span links to each message rather than one parent.
  • Anything the instrumentation can't see is your job: an outbox table (store the current traceparent with the row, and use it as the parent when the relay sends), a webhook to another system, a scheduled job that picks up earlier work. Otherwise the trace splits into unrelated pieces exactly where incidents happen.
  • Browsers: a front-end SDK can start the trace and send traceparent with fetch. If the API is on another origin, the API's CORS policy must allow the traceparent and tracestate request headers, or the preflight fails. Don't send them to third-party domains.
  • Return the trace ID to clients (a response header, or on error pages as a "support code"), so a bug report leads straight to the trace.

Structured logging

Log events with fields, not sentences to parse later.

{ "timestamp": "2026-10-23T09:14:02.381Z", "level": "error", "service": "orders-api",
  "message": "payment declined", "order_id": "o_8812", "provider_code": "card_declined",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7" }
  • Put the trace and span IDs on every log line (OTel's logging bridges do this), so logs and traces join.
  • Use consistent field names across services, and log levels on purpose: info for business events, debug off in production unless switched on.
  • Never log secrets, tokens, full card numbers or more personal data than you need. Logs get copied to more places, and kept longer, than your database.

Cardinality

A metric's cost is its number of time series: one per unique combination of label values. http_requests_total{route, method, status} with 50 routes, 4 methods and 6 status codes is at most 1,200 series. Add user_id with a million users and it's over a billion, which no metrics store will thank you for.

  • Labels on metrics should have bounded values: route templates (/orders/{id}, not /orders/8812), status class, region, version.
  • High-cardinality detail (user ID, order ID, full URL) belongs on spans and log lines, which are stored per event and searched rather than pre-aggregated.

Sampling

Traces of every request are expensive, so traces are sampled. Metrics are aggregated, so they are never sampled, which is why request and error counts should come from metrics.

KindDecidedKeepsTrade-off
Head samplingWhen the trace starts (for example 10% by trace ID)A random share of traces, consistently across services via the traceparent flagCheap and simple, but may drop the one failing request you need
Tail samplingAfter the trace is complete, in a CollectorAll errors and slow traces, plus a share of the restNeeds a Collector tier that sees every span of a trace before deciding
  • Use parent-based sampling in every service, so a kept trace keeps all its spans. Mixed rules leave traces with holes.
  • If you count from sampled data, weight each kept item by the sampling rate. Some backends store this for you (Application Insights records it as ItemCount, so queries use sum(ItemCount) instead of count()).
  • Other cost levers: drop health-check requests and chatty dependencies at the Collector, lower log levels, shorter retention or cheaper storage tiers for verbose logs. An ingestion cap protects the bill but stops collection mid-incident, so treat it as a safety net set well above normal, with an alert before it's reached.

Query languages

Each platform has its own, but the moves are the same: filter by time, group, aggregate, take percentiles.

PlatformLanguage
AWS CloudWatch LogsLogs Insights query language
Azure Monitor (Log Analytics, Application Insights)KQL
Google Cloud LoggingLogging query language; SQL in Log Analytics
Prometheus (and managed Prometheus on every cloud)PromQL
// KQL: p95 latency and failures per operation in the last hour (sampling-aware)
AppRequests
| where TimeGenerated > ago(1h)
| summarize requests = sum(ItemCount), failures = sumif(ItemCount, Success == false),
            p95 = percentile(DurationMs, 95) by Name
| order by failures desc

SLOs, error budgets and burn rate

Define the SLI as a ratio of good events (for example "checkout requests that succeed in under 800 ms"), set an SLO over a window (99.9% over 30 days), and the error budget is the rest (0.1%). Burn rate is how fast you're spending it: observed error rate ÷ budget rate. A burn rate of 1 spends exactly the budget over the window.

# Checkout burn rate over the last hour (alert when > 14.4)
(
  sum(rate(http_requests_total{route="/api/orders", code=~"5.."}[1h]))
  /
  sum(rate(http_requests_total{route="/api/orders"}[1h]))
) / (1 - 0.999)

Common multi-window thresholds for a 30-day SLO, from the Google SRE workbook: page at 14.4× sustained over an hour (2% of the monthly budget gone in one hour) or at 6× over six hours (5% in six hours), and ticket at 1× over three days (10% in three days). Add a short window of 1/12 of the long one to each (5 minutes for the 1-hour alert, 30 minutes for the 6-hour one, 6 hours for the 3-day ticket), so the alert also stops quickly once the problem is fixed.

Alerting, dashboards and runbooks

  • Alert on symptoms users feel: SLO burn rate, error rate, latency, the age of the oldest message in a queue, dead-lettered messages. CPU at 85% is a cause; it belongs on a dashboard, not a pager.
  • Every page is actionable and links a runbook: what the alert means, the dashboards and queries to check first, safe mitigations (roll back, turn a flag off, scale out, fail over) and who to escalate to.
  • Dashboards: one per service with RED (rate, errors, duration) and its key dependencies, with deploy markers so "it started at 14:05" lines up with "we deployed at 14:04".
  • Synthetic checks from outside (an uptime probe hitting a real endpoint) catch the failures your own telemetry can't see, like DNS or certificate expiry.

Health checks: liveness and readiness

ProbeQuestionOn failureShould check
LivenessIs this process stuck beyond repair?The platform restarts itOnly the process itself (it responds, the event loop isn't blocked)
ReadinessCan this instance take traffic right now?It's removed from the load balancer until it passesWarm-up finished, critical config loaded, not overloaded
StartupHas it finished starting?Liveness waits until it passesSlow initialisation

Never put a shared dependency (the database, a downstream API) in liveness: a 30-second database blip would restart every instance at once and turn a blip into an outage. Be careful in readiness too: if every instance fails readiness for the same shared dependency, the load balancer has nowhere to send traffic, and serving a clear error or degraded response is often better. Kubernetes names these three probes; load balancers and managed platforms on every cloud have equivalent health checks.

Part 2: Delivery

CI/CD pipelines

  • CI runs on every change: build, unit tests, linting, a security scan, and produces one immutable artifact (a container image identified by its digest, or a versioned package).
  • CD promotes that same artifact through environments: build once, deploy many. Only configuration changes between dev, staging and production. Rebuilding per environment means production runs something nobody tested.
  • Keep pipelines as code in the repository, fast (cache dependencies, run tests in parallel), and the only way to change production.
  • GitOps is a variant where the desired state lives in Git and an agent in the cluster (Argo CD, Flux) pulls and applies it.

Infrastructure as code

Every resource is declared in files, reviewed in pull requests, and applied by the pipeline. Manual changes in a console become drift, which the next apply either reverts or trips over.

ToolScopeStateNotes
Terraform / OpenTofuAny provider: all clouds, GitHub, DNS, monitoringA state file in a remote backend with locking (a storage bucket, or HCP Terraform)Declarative HCL; plan previews changes. OpenTofu is an open-source fork of Terraform
PulumiAny providerPulumi Cloud or a self-managed backendGeneral-purpose languages (TypeScript, Python, Go, C#)
AWS CloudFormation / CDKAWSAWS keeps the stack stateCDK writes CloudFormation from code; change sets preview
Bicep (ARM)AzureAzure is the statewhat-if previews; compiles to ARM JSON
Infrastructure ManagerGoogle CloudManaged for youGoogle's managed service for running Terraform; replaces the deprecated Deployment Manager
  • Show the plan in the pull request (terraform plan, a change set, what-if) so reviewers see infrastructure changes, not only code.
  • State files contain secrets (generated passwords, connection strings). Store them encrypted, locked and access-controlled, never in Git.
  • Know what removal does. Terraform, OpenTofu, Pulumi and CloudFormation delete a resource that leaves the code (unless you mark it to be retained). Azure deployments are incremental by default, so a resource removed from a Bicep file stays in Azure unless you use a deployment stack.
  • One owner per resource. Mixing tools is fine across resources, never for the same one.

Secrets in pipelines: OIDC workload identity federation

The old way was a cloud access key or service principal secret in a pipeline variable: long-lived, copied around, rarely rotated, and valid from anywhere. Workload identity federation removes it. The CI system issues a short-lived OIDC token for the running job, signed by its issuer, with claims about the repository, branch and environment. The cloud exchanges it for short-lived cloud credentials because you configured a trust that matches exactly that issuer and subject.

CloudYou configureThe job uses
AWSAn IAM OIDC identity provider and a role whose trust policy checks the sub claimAssumeRoleWithWebIdentity (for example aws-actions/configure-aws-credentials)
AzureA federated identity credential on a user-assigned managed identity or app registrationazure/login with client, tenant and subscription IDs
Google CloudA workload identity pool and provider, with an attribute conditiongoogle-github-actions/auth with the provider's name
# GitHub Actions (illustrative): no cloud secret stored anywhere
permissions:
  id-token: write          # lets the job request an OIDC token
  contents: read

jobs:
  deploy-prod:
    runs-on: ubuntu-latest
    environment: production   # required reviewers; the token's subject ends in :environment:production
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/deploy-orders-prod
          aws-region: eu-central-1
      - run: ./deploy.sh prod
  • The subject reflects the job: repo:acme/orders:ref:refs/heads/main for a branch, repo:acme/orders:environment:production when the job targets an environment (the environment takes precedence), repo:acme/orders:pull_request for pull requests. The trust must match it exactly, or sign-in fails.
  • Use one identity per environment, trusted only for that environment's subject. Then only jobs that passed the production environment's protection rules can touch production, and a pull request workflow can't.
  • Azure DevOps, GitLab and other CI systems offer the same federation. The pattern doesn't change: a short-lived token, an exact trust, least privilege.
  • Application secrets are a separate concern: the running service reads them from a secret store with its own workload identity at runtime. Don't bake them into images or pipeline logs.

Environments and promotion

  • Separate accounts, subscriptions or projects per environment where you can, so a mistake in dev can't reach production, and each environment has its own identities.
  • Promotion gates: automated tests and checks first, then human approval only where it adds value (production, or regulated changes).
  • Staging is as close to production as you can afford: same IaC, same artifact, smaller scale. Differences between them are where surprises come from.
  • Humans get production access just in time, for a reason and a limited time, and it's logged.

Deployment strategies

StrategyHow it worksRollbackWatch out for
RollingReplace instances a few at a timeRoll forward or redeploy the previous version (slow-ish)Both versions serve traffic during the roll
Blue-greenBring up the new version beside the old, test it, switch all traffic at onceSwitch backDouble capacity during the switch; shared database
CanarySend a small share of real traffic (1-5%) to the new version, compare, then increaseSet its weight to 0Needs per-version metrics and enough traffic to compare
Feature flagSame binary everywhere; behaviour turned on per audienceTurn the flag offFlag debt, and testing both paths

What makes these safe in practice:

  • Compare the canary with the baseline running at the same time, not with last week: filter by service.version, and gate each step on error rate and p95, automatically.
  • Traffic splitting is per request unless you add session affinity. A user can bounce between versions, and a single-page app can ask for a JavaScript chunk that only the other version has. Keep old assets around for a while, use affinity, or release with a flag.
  • Message contracts must be readable by both versions: a new producer can't send a field an old consumer will choke on.

Feature flags

Flags separate deploying code from releasing behaviour: ship dark, turn on for staff, then 5%, then everyone, and turn off in seconds without a deploy. That's also the cheapest kill switch you'll have during an incident.

  • Percentage rollouts must be sticky: bucket by a hash of the user ID and flag name, so each user stays consistently in or out. A fresh random decision on every evaluation makes the same user flip between versions.
  • Evaluate flags on the server (or a back end for the front end) and send the result to the browser. Don't ship flag-service credentials to clients.
  • Decide the default when the flag service is unreachable (usually "off"), test both paths, and delete flags when a rollout finishes. OpenFeature is the vendor-neutral flag API, the OpenTelemetry of flags.

Database migrations alongside code

During any rolling, blue-green or canary release, two versions run against one database, and a rollback means the old code runs against the new schema. So every schema change must work with both. Use expand and contract across releases:

  1. Expand: add the new column or table (nullable or with a default). The new release writes both old and new.
  2. Migrate: backfill existing rows in batches, then switch reads to the new column.
  3. Contract: once no running version uses the old column, drop it in a later release.

Renaming a column in one step breaks whichever version expects the other name, whatever order you run things in. Run migrations as their own pipeline step, idempotent and short, and keep long backfills out of the deploy's critical path.

Rollback

  • Rolling back must be faster than rolling forward. Keep the previous artifact, redeploy it with one command (or shift traffic back), and practise it.
  • Rollback undoes code, not data. Schema changes and data written by the new version must stay compatible with the old one, which is what expand and contract buys you.
  • Decide in advance what triggers a rollback (the canary's error rate or burn rate over a threshold), so nobody debates it at 3 a.m. Then fix forward calmly.

Cloud mappings

Side note
NeedAWSAzureGoogle CloudOther
Metrics, logs, dashboards, alertsCloudWatch (metrics, Logs, alarms, dashboards)Azure Monitor, Log AnalyticsCloud Monitoring, Cloud LoggingPrometheus + Grafana, Datadog, New Relic
Tracing and APMX-Ray, CloudWatch Application SignalsApplication InsightsCloud TraceJaeger, Grafana Tempo, Honeycomb, Datadog APM
OpenTelemetry pathAWS Distro for OpenTelemetry (ADOT)Azure Monitor OpenTelemetry DistroOpenTelemetry Collector with Google Cloud exportersOpenTelemetry Collector to any OTLP backend
Managed Prometheus and GrafanaAmazon Managed Service for Prometheus, Amazon Managed GrafanaAzure Monitor managed service for Prometheus, Azure Managed GrafanaGoogle Cloud Managed Service for PrometheusGrafana Cloud, self-hosted
Synthetic checksCloudWatch SyntheticsApplication Insights availability testsCloud Monitoring uptime checksCheckly, Datadog Synthetics
Infrastructure as codeCloudFormation, CDKBicep, ARM templatesInfrastructure ManagerTerraform, OpenTofu, Pulumi
CI/CDCodePipeline, CodeBuildAzure Pipelines (Azure DevOps)Cloud Build, Cloud DeployGitHub Actions, GitLab CI/CD, Jenkins, Argo CD
Progressive deliveryCodeDeploy, ALB weighted target groups, Lambda alias weightsApp Service deployment slots, Container Apps traffic splittingCloud Deploy canaries, Cloud Run traffic splittingKubernetes rolling updates, Argo Rollouts, Flagger
Pipeline sign-in without secretsIAM OIDC identity provider and roleEntra federated identity credentialWorkload Identity FederationVault JWT/OIDC auth
Feature flagsAWS AppConfigAzure App ConfigurationFirebase Remote ConfigLaunchDarkly, Unleash, Flagsmith (OpenFeature)

How to say it in a design review

"Every service uses OpenTelemetry with its own service.name and service.version, exporting through a Collector that tail-samples to keep every error. traceparent goes from the browser to the API and rides in message headers and outbox rows, so one trace ID covers browser to payment provider, and every log line carries it. Request and error counts come from metrics, with no user IDs in labels. We page on a 14.4× burn rate of the checkout SLO and on dead-lettered messages, each with a runbook. Liveness checks only the process. Infrastructure is Terraform with the plan on every pull request; pipelines sign in with OIDC federation, one identity per environment, and production sits behind required reviewers. We build one image, release it as a 5%, 25%, 100% canary with automatic checks against the baseline, put new behaviour behind sticky flags, and change schemas with expand and contract so rollback is always safe."

Checklist

  • OpenTelemetry in every service with service.name and service.version; logs carry trace and span IDs
  • Trace context through HTTP, queues, outboxes and webhooks; trace ID returned to clients
  • Bounded metric labels; counts from metrics; sampling chosen on purpose (tail sampling to keep errors)
  • SLIs and SLOs with burn-rate alerts that page; symptoms on the pager, causes on dashboards; a runbook per alert
  • Liveness checks the process only; readiness gates traffic; synthetic checks from outside
  • All infrastructure in code, plan shown in pull requests, state protected, one owner per resource
  • Build once, deploy many; pipelines sign in with OIDC federation, one identity per environment, no stored secrets
  • Canary or blue-green with automatic comparison to the baseline; sticky flags with safe defaults and cleanup
  • Expand and contract for schema changes; a practised, one-step rollback

Practise this concept

In the app: a short video and a 12-question concept check. Free with a GitHub account.

Open in the app