System design · Concept 13

Compute and Hosting

Matching workloads to hosting models (virtual machines, PaaS, managed containers, Kubernetes, serverless functions, durable workflow engines, static hosting), plus scaling signals and limits, zero-downtime deploys, private networking, cost, and the equivalent services on AWS, Azure and Google Cloud.

17 min read

In short

  • Choose per workload: API, worker, job, event, workflow, frontend
  • Kubernetes only when you need its API
  • Functions: cold starts, timeouts, concurrency
  • Scale on queue depth or concurrency; size for the database
  • Canary or rolling deploys with backward-compatible schemas

Picking compute is less about "which scales best" (they all scale) and more about the shape of the work (an HTTP API, a queue consumer, a scheduled job, a long workflow, static files) and how much platform your team wants to run. Decide the hosting model first, then name the service your cloud offers for it. The Cloud mappings table below lists the equivalents.

Start from the workload

WorkloadReach for firstWhy
Request/response API with steady trafficManaged containers or PaaS app hostingAlways warm, scales out on concurrency, no cluster to run
Background worker consuming a queueManaged containers or functions with a queue triggerScales on queue depth, down to zero when the queue is empty
Scheduled or batch job (nightly export, reindex)A run-to-completion job triggered by a schedulerRuns, exits, bills only while running; no server idling 23 hours a day
Event handlers and glue (webhooks, file uploaded, row changed)Serverless functionsTriggers built in, per-execution billing, spiky load is fine
Long-running workflow with waits, retries, human approvalA durable workflow engineState, timers and retries survive restarts without your own state machine
Single page app or static siteStatic hosting with a CDNFiles served from the edge, nearly free, nothing to patch
Needs the Kubernetes API itself: operators, Helm charts from vendors, GPU node pools, a platform for many teamsKubernetes (a managed control plane)Full control, at the price of running a cluster
Legacy software, special OS or licensing, full control of the boxVirtual machinesAnything runs; you patch and scale it yourself

One system usually needs several of these. A typical product is a static frontend, one or two APIs on containers, a few workers, a scheduler and maybe a workflow engine for the one process that spans days.

Hosting models

From most control (and most work) to least:

VMs ─► Kubernetes ─► managed containers ─► PaaS app hosting ─► functions ─► static hosting
you run the OS        you ship an image       you ship code       you ship a handler   you ship files

Virtual machines

You get an OS and run whatever you like. Scale with an instance group built from an image (auto scaling groups, scale sets, managed instance groups) behind a load balancer. You own OS patching, runtime upgrades, log shipping and the image pipeline. Choose VMs when nothing higher up fits: software that expects a machine, special kernels or drivers, per-core licensing, or very steady load where reserved instances are cheapest.

PaaS app hosting

You push code (or a container) and the platform runs it on instances you size and count: Elastic Beanstalk, App Service, App Engine. You get TLS, autoscaling rules, health checks and deployment features without touching an OS. The usual model is provisioned instances billed while they run, whether traffic arrives or not; on Azure App Service, several apps can share one plan's instances, which makes small apps cheap to run together. Turn off session affinity (sticky sessions) for stateless APIs, so load spreads evenly.

Containers: managed platforms vs Kubernetes

A container image is the portable unit: the same image runs on a laptop, a managed platform or a cluster.

  • Managed container platforms (ECS on Fargate, Container Apps, Cloud Run) run your image with no nodes to manage. You set CPU, memory, minimum and maximum instances, and a scale rule. Cloud Run and Container Apps can scale to zero and back on traffic, and Container Apps scales on queue length through KEDA. Most teams with a handful of services belong here.
  • Kubernetes (EKS, AKS, GKE) gives you the full API: operators and CRDs, Helm charts, service meshes, special node pools (GPU, spot, Windows) and one shared platform for many teams. You also take on version upgrades several times a year, node images, ingress, network policy, autoscaling at two levels (pods and nodes) and capacity planning. GKE Autopilot, EKS Auto Mode and AKS Automatic manage nodes for you and are the sensible starting point for a new cluster.

The test: "do we need something only Kubernetes gives us?" If the answer is "it's what everyone uses", a managed container platform is less work for the same result.

Serverless functions

You deploy a handler, the platform runs it when a trigger fires (HTTP, queue message, file upload, timer, database change) and bills per execution and memory-time. Three things to design for:

  • Cold starts. A new instance has to start the runtime, load your code and run your startup before it serves anything, which adds latency to the first request on that instance. Keep startup lean (no big caches or network calls at init), create SDK and HTTP clients once and reuse them across invocations, and pay for warm capacity on latency-sensitive paths: provisioned concurrency on Lambda, always-ready instances on Azure Functions Flex Consumption, minimum instances on Cloud Run functions. A path that is busy all day and can't tolerate any cold start probably belongs on containers instead.
  • Timeouts. Every platform caps how long one execution may run (Lambda stops at 15 minutes), and the HTTP path in front can cap it sooner: an HTTP-triggered Azure Function must respond within 230 seconds whatever its timeout setting. Long work returns 202 Accepted with a status URL and continues in the background, through a queue and a worker or a workflow engine.
  • Concurrency. Lambda runs one request per execution environment at a time, so 500 concurrent requests means 500 environments, each with its own database connections. Cloud Run and Azure Functions instances handle several requests at once. Either way, every platform has a concurrency or instance ceiling (Lambda's per-Region account quota is 1,000 concurrent executions by default, shared by every function), so one runaway function can starve the rest unless you reserve or cap concurrency per function.

Durable workflow engines

Some processes span minutes to days: reserve stock, wait for a payment, wait for a human, ship. Writing that as a chain of queues and status columns works, but you end up building your own state machine, timers and retry logic. A workflow engine (Step Functions, Durable Functions or Logic Apps, Google Cloud Workflows, Temporal) persists the workflow's state after each step, so it survives restarts and deploys.

workflow ApproveLargeOrder(order):
  reserveStock(order)                         # activity: retried by the engine
  decision = waitFor("approval", timeout = 24h)   # durable timer, no server waiting
  if decision == approved:
    try:
      chargeCard(order)
    except PaymentFailed:
      releaseStock(order)                     # compensation: undo the earlier step
      return "payment failed"
    return "approved"
  releaseStock(order)
  return "timed out"
  • Retries are declared per step (attempts, backoff), so activities must be idempotent: a step can run twice if the worker dies after doing the work but before the engine records it.
  • Sagas: there's no transaction across services, so each step that changes something gets a compensating step (release stock, refund the charge) that runs when a later step fails.
  • Two styles. Step Functions and Google Cloud Workflows take a definition (JSON or YAML) of steps and transitions. Durable Functions and Temporal let you write the workflow as ordinary code and replay it: after every await, the engine re-runs the function from the start against the recorded history. Replayed code must be deterministic, so take the time, random values and new IDs from the engine's context, and do all I/O inside activities.
  • In-flight instances outlive deploys. Changing a workflow while instances are running can break their replay or their state. Use the engine's versioning, or deploy the new version side by side and let old instances drain.
  • Limits matter. Step Functions has Standard workflows (up to a year, for long processes) and Express workflows (up to five minutes, for high-volume short ones); pick by duration.

Static hosting with a CDN

A single page app or static site is a folder of files. Put it in object storage or a static hosting service with a CDN in front (S3 and CloudFront, Static Web Apps, Cloud Storage and Cloud CDN, Netlify, Cloudflare Pages, Vercel).

  • Fingerprinted asset names (app.3f9a1c.js) cached for a year as immutable; index.html cached briefly or not at all, so a deploy is visible at once.
  • A navigation fallback that serves index.html for unknown paths, so /orders/42 reaches the client router instead of a 404.
  • Keep the previous release's assets for a while. Users with the old app open still request old chunk names, and get a ChunkLoadError if they're gone.
  • Preview environments per pull request come built in with Static Web Apps, Amplify Hosting, Firebase Hosting, Netlify, Vercel and Cloudflare Pages.

Cloud mappings

Side note
RoleAWSAzureGoogle CloudOther
Virtual machines + autoscalingEC2 + Auto Scaling groupsVirtual Machines + Virtual Machine Scale SetsCompute Engine + managed instance groupsAny VPS or bare metal
PaaS app hostingElastic BeanstalkApp ServiceApp EngineHeroku, Render
Managed containersECS on FargateContainer AppsCloud RunFly.io, Render
KubernetesEKS (Auto Mode)AKS (Automatic)GKE (Autopilot)Self-managed Kubernetes
Serverless functionsLambdaFunctions (Flex Consumption for new apps)Cloud Run functions (formerly Cloud Functions)Cloudflare Workers, Vercel Functions
Run-to-completion jobsECS tasks, AWS BatchContainer Apps jobsCloud Run jobsKubernetes Jobs
SchedulerEventBridge SchedulerFunctions timer trigger, Container Apps jobs (cron)Cloud Schedulercron, Kubernetes CronJob
Workflow / orchestratorStep FunctionsDurable Functions, Logic AppsWorkflowsTemporal, Camunda
Static hosting + CDNS3 + CloudFront, Amplify HostingStatic Web Apps, or Blob Storage + Front DoorCloud Storage + Cloud CDN, Firebase HostingNetlify, Cloudflare Pages, Vercel
Traffic shifting for deploysCodeDeploy (blue/green, canary), Lambda alias weightsApp Service deployment slots, Container Apps revisionsCloud Run revisions, App Engine versionsKubernetes rolling updates, Argo Rollouts
Private access to managed servicesVPC interface endpoints (PrivateLink)Private endpoints (Private Link)Private Service Connect
Serverless egress into a private networkLambda attached to VPC subnetsVNet integrationDirect VPC egress
Static outbound IPNAT gatewayNAT GatewayCloud NAT

Scaling: signals, units and limits

Hosting modelUnit that scalesTypical signalScale to zero
VMsInstances in a groupCPU, requests per instance, a scheduleRarely used
PaaS app hostingPlan or app instancesCPU, memory, request rateVaries: App Engine standard can, App Service can't
Managed containersReplicas (tasks)Concurrent requests, CPU, queue lengthCloud Run and Container Apps yes; ECS services don't wake on traffic
KubernetesPods (HPA or KEDA), then nodes (cluster autoscaler)Any metricPods with KEDA; nodes down to the pool minimum
FunctionsInstances per triggerEvents waiting, concurrencyYes
  • Scale on the signal closest to the work. Queue depth (or consumer lag on a stream) for workers, concurrent requests for APIs. CPU lags behind the load and misses I/O-bound work that spends its time waiting on a database.
  • Size a queue target from latency. If one message takes 0.5 s and the backlog should clear within 30 s, one worker handles 60 messages in that time, so target 60 messages per instance: 3,000 waiting messages means 50 workers.
  • Downstream limits don't scale with you. 40 instances × a connection pool of 100 = up to 4,000 connections, and PostgreSQL's default max_connections is 100. Set the maximum instance count from your tightest dependency, size pools for that maximum, use a pooler (PgBouncer, RDS Proxy) where many small instances share one database, and put a queue in front of slow dependencies.
  • Scale-in kills instances. Handle SIGTERM: stop taking new work, finish or release in-flight messages so they return to the queue, then exit. Keep processing idempotent, because a message can be delivered again.
  • Scale to zero means a cold start on the next request. Latency-sensitive APIs keep a minimum of one or two instances (across zones for availability).

Zero-downtime deployments

StrategyHowRollbackWatch out for
RollingReplace instances in batches, each gated by a readiness checkRoll forward or redeploy the old versionBoth versions serve traffic during the rollout
Blue/greenStart the new version beside the old one, test it, switch all trafficSwitch backDouble capacity while both run
CanarySend a small share of traffic (5%) to the new version, compare errors and latency, then increaseSet the weight back to 0Needs good metrics to judge the canary
RecreateStop everything, start the new versionRedeployDowntime; only for dev or things that can't run twice

Every platform has its own primitive. App Service deployment slots warm a staging slot and then swap it with production; settings marked as slot settings stay behind, and a forgotten one is how a staging connection string reaches production. Container Apps and Cloud Run revisions take traffic weights and get tagged URLs for testing before they receive real traffic. Lambda aliases shift weighted traffic between versions, and CodeDeploy automates blue/green and canary for ECS and Lambda. Kubernetes does rolling updates (maxSurge, maxUnavailable) gated by readiness probes, and tools like Argo Rollouts add canaries. On Azure Functions Flex Consumption there are no slots: switch the site update strategy from the default Recreate to RollingUpdate.

Whatever the platform, two versions run side by side for a while, so:

  • Database changes must be backward compatible. Expand (add the new column), deploy code that writes both and reads the new one, backfill, then contract (drop the old column) in a later release. Never rename a column in the same release that starts using the new name.
  • Readiness checks gate traffic. A cheap /healthz that confirms the instance can serve, without calling every dependency (or one slow dependency takes the whole fleet out of rotation).

Networking basics

  • Ingress: the public entry point is an edge service or load balancer with TLS and a WAF (CloudFront, Front Door, Cloud Load Balancing, Cloudflare). The app itself accepts traffic only from that edge, or has no public endpoint at all.
  • Private access to managed services: a private endpoint gives a database, secret store or storage account a private IP in your network; then disable its public access.
  • Egress from serverless and PaaS into a private network needs an explicit link (a Lambda attached to VPC subnets, VNet integration, Direct VPC egress). That link is outbound only: it lets the app reach private resources, but doesn't make the app itself private.
  • Outbound to the internet from private subnets goes through a NAT gateway, which also gives you a static outbound IP for partners' allow-lists. A Lambda attached to a VPC gets no public IP, so without NAT its calls to external APIs time out.
Internet ─► Edge (CDN + WAF) ─► app (no public access)
                                  │ outbound link into the private network
                                  ▼
               private subnet ─► private endpoints: database · secrets · queue
               private DNS: the services' normal hostnames resolve to private IPs
                                  │
                                  └─► NAT gateway ─► partner APIs (static IP)

Private endpoints depend on DNS. The service's normal hostname must resolve to the private IP inside your network (a private DNS zone on Azure, private DNS for interface endpoints on AWS, DNS records for Private Service Connect on Google Cloud). If it doesn't, the hostname resolves to the public IP, the request leaves over the internet, and the service rejects it because public access is off. When a private connection fails, check DNS first (nslookup from the app).

Cost trade-offs

ModelYou pay forCheap whenExpensive when
VMsInstances, 24/7, plus the people who patch themSteady load with reserved pricingIdle capacity, small teams
PaaS app hostingProvisioned instances, 24/7Several apps share instances; steady trafficMany small apps idling at night
Managed containersvCPU and memory while instances run (some bill per request)Scale to zero; spiky or moderate loadMany always-on instances at high utilisation
KubernetesNodes, the control plane fee, and the platform teamLarge, dense, steady workloads; spot node poolsA few services run by a small team
FunctionsExecutions and memory-timeSpiky or low volume, idle most of the dayBusy all day: per-execution pricing overtakes an always-on instance
Static hostingStorage and CDN egress, often a free tierAlmost alwaysRarely the cost driver
  • Steady 24/7 load → committed pricing (AWS Savings Plans and Reserved Instances, Azure reservations and savings plans, Google Cloud committed use discounts). Spiky or idle → pay per use. Interruptible work (batch, CI, stateless workers) can run on spot capacity on any of the three.
  • Hidden costs repeat per environment: NAT gateways and private endpoints bill by the hour and by data processed, log ingestion grows with traffic, and cross-region or internet egress is billed per GB.
  • Count people, not just instances. A Kubernetes cluster that saves 20% on compute but needs a part-time platform engineer is not cheaper for a team of five.

How to say it in a design review

"The React app is static files on object storage behind the CDN, with a preview per pull request. The order API runs on a managed container platform with two minimum instances across zones, scaling on concurrent requests, and a maximum set so its connection pools fit the database. The payment worker scales from zero to 30 on queue depth. The nightly report is a scheduled run-to-completion job, and order approval is a workflow engine with a 24-hour timer and compensation if payment fails. Deploys are canaries at 5% with automatic rollback; schema changes are expand and contract. The app reaches the database through a private endpoint with private DNS, and calls the payment provider through NAT for a fixed IP. On AWS that's ECS on Fargate, Step Functions and EventBridge Scheduler; on Azure, Container Apps, Durable Functions and a Container Apps job; on Google Cloud, Cloud Run, Workflows and Cloud Scheduler."

Checklist

  • Hosting chosen per workload (API, worker, job, event handler, workflow, frontend), not one size for all
  • Kubernetes only when you need its API; otherwise managed containers
  • Functions: cold starts handled on latency-sensitive paths, execution and HTTP timeouts respected, concurrency capped
  • Long processes in a workflow engine: idempotent steps, compensation, versioning; deterministic code if it replays
  • Scale on queue depth or concurrency; minimum instances for latency-sensitive APIs; maximum sized for the database
  • Zero-downtime path (rolling, blue/green or canary) with readiness checks and backward-compatible schema changes
  • Private endpoints with working private DNS; outbound link and NAT where needed; public access disabled
  • Cost: idle capacity, per-environment fixed costs, egress and log ingestion estimated

Practise this concept

In the app: a short video and a 12-question concept check. Free with a GitHub account.

Open in the app