System design · Concept 13
Compute and Hosting
Matching workloads to hosting models (virtual machines, PaaS, managed containers, Kubernetes, serverless functions, durable workflow engines, static hosting), plus scaling signals and limits, zero-downtime deploys, private networking, cost, and the equivalent services on AWS, Azure and Google Cloud.
In short
- Choose per workload: API, worker, job, event, workflow, frontend
- Kubernetes only when you need its API
- Functions: cold starts, timeouts, concurrency
- Scale on queue depth or concurrency; size for the database
- Canary or rolling deploys with backward-compatible schemas
Picking compute is less about "which scales best" (they all scale) and more about the shape of the work (an HTTP API, a queue consumer, a scheduled job, a long workflow, static files) and how much platform your team wants to run. Decide the hosting model first, then name the service your cloud offers for it. The Cloud mappings table below lists the equivalents.
Start from the workload
| Workload | Reach for first | Why |
|---|---|---|
| Request/response API with steady traffic | Managed containers or PaaS app hosting | Always warm, scales out on concurrency, no cluster to run |
| Background worker consuming a queue | Managed containers or functions with a queue trigger | Scales on queue depth, down to zero when the queue is empty |
| Scheduled or batch job (nightly export, reindex) | A run-to-completion job triggered by a scheduler | Runs, exits, bills only while running; no server idling 23 hours a day |
| Event handlers and glue (webhooks, file uploaded, row changed) | Serverless functions | Triggers built in, per-execution billing, spiky load is fine |
| Long-running workflow with waits, retries, human approval | A durable workflow engine | State, timers and retries survive restarts without your own state machine |
| Single page app or static site | Static hosting with a CDN | Files served from the edge, nearly free, nothing to patch |
| Needs the Kubernetes API itself: operators, Helm charts from vendors, GPU node pools, a platform for many teams | Kubernetes (a managed control plane) | Full control, at the price of running a cluster |
| Legacy software, special OS or licensing, full control of the box | Virtual machines | Anything runs; you patch and scale it yourself |
One system usually needs several of these. A typical product is a static frontend, one or two APIs on containers, a few workers, a scheduler and maybe a workflow engine for the one process that spans days.
Hosting models
From most control (and most work) to least:
VMs ─► Kubernetes ─► managed containers ─► PaaS app hosting ─► functions ─► static hosting
you run the OS you ship an image you ship code you ship a handler you ship files
Virtual machines
You get an OS and run whatever you like. Scale with an instance group built from an image (auto scaling groups, scale sets, managed instance groups) behind a load balancer. You own OS patching, runtime upgrades, log shipping and the image pipeline. Choose VMs when nothing higher up fits: software that expects a machine, special kernels or drivers, per-core licensing, or very steady load where reserved instances are cheapest.
PaaS app hosting
You push code (or a container) and the platform runs it on instances you size and count: Elastic Beanstalk, App Service, App Engine. You get TLS, autoscaling rules, health checks and deployment features without touching an OS. The usual model is provisioned instances billed while they run, whether traffic arrives or not; on Azure App Service, several apps can share one plan's instances, which makes small apps cheap to run together. Turn off session affinity (sticky sessions) for stateless APIs, so load spreads evenly.
Containers: managed platforms vs Kubernetes
A container image is the portable unit: the same image runs on a laptop, a managed platform or a cluster.
- Managed container platforms (ECS on Fargate, Container Apps, Cloud Run) run your image with no nodes to manage. You set CPU, memory, minimum and maximum instances, and a scale rule. Cloud Run and Container Apps can scale to zero and back on traffic, and Container Apps scales on queue length through KEDA. Most teams with a handful of services belong here.
- Kubernetes (EKS, AKS, GKE) gives you the full API: operators and CRDs, Helm charts, service meshes, special node pools (GPU, spot, Windows) and one shared platform for many teams. You also take on version upgrades several times a year, node images, ingress, network policy, autoscaling at two levels (pods and nodes) and capacity planning. GKE Autopilot, EKS Auto Mode and AKS Automatic manage nodes for you and are the sensible starting point for a new cluster.
The test: "do we need something only Kubernetes gives us?" If the answer is "it's what everyone uses", a managed container platform is less work for the same result.
Serverless functions
You deploy a handler, the platform runs it when a trigger fires (HTTP, queue message, file upload, timer, database change) and bills per execution and memory-time. Three things to design for:
- Cold starts. A new instance has to start the runtime, load your code and run your startup before it serves anything, which adds latency to the first request on that instance. Keep startup lean (no big caches or network calls at init), create SDK and HTTP clients once and reuse them across invocations, and pay for warm capacity on latency-sensitive paths: provisioned concurrency on Lambda, always-ready instances on Azure Functions Flex Consumption, minimum instances on Cloud Run functions. A path that is busy all day and can't tolerate any cold start probably belongs on containers instead.
- Timeouts. Every platform caps how long one execution may run (Lambda stops at 15 minutes), and the HTTP path in front can cap it sooner: an HTTP-triggered Azure Function must respond within 230 seconds whatever its timeout setting. Long work returns
202 Acceptedwith a status URL and continues in the background, through a queue and a worker or a workflow engine. - Concurrency. Lambda runs one request per execution environment at a time, so 500 concurrent requests means 500 environments, each with its own database connections. Cloud Run and Azure Functions instances handle several requests at once. Either way, every platform has a concurrency or instance ceiling (Lambda's per-Region account quota is 1,000 concurrent executions by default, shared by every function), so one runaway function can starve the rest unless you reserve or cap concurrency per function.
Durable workflow engines
Some processes span minutes to days: reserve stock, wait for a payment, wait for a human, ship. Writing that as a chain of queues and status columns works, but you end up building your own state machine, timers and retry logic. A workflow engine (Step Functions, Durable Functions or Logic Apps, Google Cloud Workflows, Temporal) persists the workflow's state after each step, so it survives restarts and deploys.
workflow ApproveLargeOrder(order):
reserveStock(order) # activity: retried by the engine
decision = waitFor("approval", timeout = 24h) # durable timer, no server waiting
if decision == approved:
try:
chargeCard(order)
except PaymentFailed:
releaseStock(order) # compensation: undo the earlier step
return "payment failed"
return "approved"
releaseStock(order)
return "timed out"
- Retries are declared per step (attempts, backoff), so activities must be idempotent: a step can run twice if the worker dies after doing the work but before the engine records it.
- Sagas: there's no transaction across services, so each step that changes something gets a compensating step (release stock, refund the charge) that runs when a later step fails.
- Two styles. Step Functions and Google Cloud Workflows take a definition (JSON or YAML) of steps and transitions. Durable Functions and Temporal let you write the workflow as ordinary code and replay it: after every await, the engine re-runs the function from the start against the recorded history. Replayed code must be deterministic, so take the time, random values and new IDs from the engine's context, and do all I/O inside activities.
- In-flight instances outlive deploys. Changing a workflow while instances are running can break their replay or their state. Use the engine's versioning, or deploy the new version side by side and let old instances drain.
- Limits matter. Step Functions has Standard workflows (up to a year, for long processes) and Express workflows (up to five minutes, for high-volume short ones); pick by duration.
Static hosting with a CDN
A single page app or static site is a folder of files. Put it in object storage or a static hosting service with a CDN in front (S3 and CloudFront, Static Web Apps, Cloud Storage and Cloud CDN, Netlify, Cloudflare Pages, Vercel).
- Fingerprinted asset names (
app.3f9a1c.js) cached for a year asimmutable;index.htmlcached briefly or not at all, so a deploy is visible at once. - A navigation fallback that serves
index.htmlfor unknown paths, so/orders/42reaches the client router instead of a 404. - Keep the previous release's assets for a while. Users with the old app open still request old chunk names, and get a
ChunkLoadErrorif they're gone. - Preview environments per pull request come built in with Static Web Apps, Amplify Hosting, Firebase Hosting, Netlify, Vercel and Cloudflare Pages.
Cloud mappings
Side note
| Role | AWS | Azure | Google Cloud | Other |
|---|---|---|---|---|
| Virtual machines + autoscaling | EC2 + Auto Scaling groups | Virtual Machines + Virtual Machine Scale Sets | Compute Engine + managed instance groups | Any VPS or bare metal |
| PaaS app hosting | Elastic Beanstalk | App Service | App Engine | Heroku, Render |
| Managed containers | ECS on Fargate | Container Apps | Cloud Run | Fly.io, Render |
| Kubernetes | EKS (Auto Mode) | AKS (Automatic) | GKE (Autopilot) | Self-managed Kubernetes |
| Serverless functions | Lambda | Functions (Flex Consumption for new apps) | Cloud Run functions (formerly Cloud Functions) | Cloudflare Workers, Vercel Functions |
| Run-to-completion jobs | ECS tasks, AWS Batch | Container Apps jobs | Cloud Run jobs | Kubernetes Jobs |
| Scheduler | EventBridge Scheduler | Functions timer trigger, Container Apps jobs (cron) | Cloud Scheduler | cron, Kubernetes CronJob |
| Workflow / orchestrator | Step Functions | Durable Functions, Logic Apps | Workflows | Temporal, Camunda |
| Static hosting + CDN | S3 + CloudFront, Amplify Hosting | Static Web Apps, or Blob Storage + Front Door | Cloud Storage + Cloud CDN, Firebase Hosting | Netlify, Cloudflare Pages, Vercel |
| Traffic shifting for deploys | CodeDeploy (blue/green, canary), Lambda alias weights | App Service deployment slots, Container Apps revisions | Cloud Run revisions, App Engine versions | Kubernetes rolling updates, Argo Rollouts |
| Private access to managed services | VPC interface endpoints (PrivateLink) | Private endpoints (Private Link) | Private Service Connect | |
| Serverless egress into a private network | Lambda attached to VPC subnets | VNet integration | Direct VPC egress | |
| Static outbound IP | NAT gateway | NAT Gateway | Cloud NAT |
Scaling: signals, units and limits
| Hosting model | Unit that scales | Typical signal | Scale to zero |
|---|---|---|---|
| VMs | Instances in a group | CPU, requests per instance, a schedule | Rarely used |
| PaaS app hosting | Plan or app instances | CPU, memory, request rate | Varies: App Engine standard can, App Service can't |
| Managed containers | Replicas (tasks) | Concurrent requests, CPU, queue length | Cloud Run and Container Apps yes; ECS services don't wake on traffic |
| Kubernetes | Pods (HPA or KEDA), then nodes (cluster autoscaler) | Any metric | Pods with KEDA; nodes down to the pool minimum |
| Functions | Instances per trigger | Events waiting, concurrency | Yes |
- Scale on the signal closest to the work. Queue depth (or consumer lag on a stream) for workers, concurrent requests for APIs. CPU lags behind the load and misses I/O-bound work that spends its time waiting on a database.
- Size a queue target from latency. If one message takes 0.5 s and the backlog should clear within 30 s, one worker handles 60 messages in that time, so target 60 messages per instance: 3,000 waiting messages means 50 workers.
- Downstream limits don't scale with you. 40 instances × a connection pool of 100 = up to 4,000 connections, and PostgreSQL's default
max_connectionsis 100. Set the maximum instance count from your tightest dependency, size pools for that maximum, use a pooler (PgBouncer, RDS Proxy) where many small instances share one database, and put a queue in front of slow dependencies. - Scale-in kills instances. Handle
SIGTERM: stop taking new work, finish or release in-flight messages so they return to the queue, then exit. Keep processing idempotent, because a message can be delivered again. - Scale to zero means a cold start on the next request. Latency-sensitive APIs keep a minimum of one or two instances (across zones for availability).
Zero-downtime deployments
| Strategy | How | Rollback | Watch out for |
|---|---|---|---|
| Rolling | Replace instances in batches, each gated by a readiness check | Roll forward or redeploy the old version | Both versions serve traffic during the rollout |
| Blue/green | Start the new version beside the old one, test it, switch all traffic | Switch back | Double capacity while both run |
| Canary | Send a small share of traffic (5%) to the new version, compare errors and latency, then increase | Set the weight back to 0 | Needs good metrics to judge the canary |
| Recreate | Stop everything, start the new version | Redeploy | Downtime; only for dev or things that can't run twice |
Every platform has its own primitive. App Service deployment slots warm a staging slot and then swap it with production; settings marked as slot settings stay behind, and a forgotten one is how a staging connection string reaches production. Container Apps and Cloud Run revisions take traffic weights and get tagged URLs for testing before they receive real traffic. Lambda aliases shift weighted traffic between versions, and CodeDeploy automates blue/green and canary for ECS and Lambda. Kubernetes does rolling updates (maxSurge, maxUnavailable) gated by readiness probes, and tools like Argo Rollouts add canaries. On Azure Functions Flex Consumption there are no slots: switch the site update strategy from the default Recreate to RollingUpdate.
Whatever the platform, two versions run side by side for a while, so:
- Database changes must be backward compatible. Expand (add the new column), deploy code that writes both and reads the new one, backfill, then contract (drop the old column) in a later release. Never rename a column in the same release that starts using the new name.
- Readiness checks gate traffic. A cheap
/healthzthat confirms the instance can serve, without calling every dependency (or one slow dependency takes the whole fleet out of rotation).
Networking basics
- Ingress: the public entry point is an edge service or load balancer with TLS and a WAF (CloudFront, Front Door, Cloud Load Balancing, Cloudflare). The app itself accepts traffic only from that edge, or has no public endpoint at all.
- Private access to managed services: a private endpoint gives a database, secret store or storage account a private IP in your network; then disable its public access.
- Egress from serverless and PaaS into a private network needs an explicit link (a Lambda attached to VPC subnets, VNet integration, Direct VPC egress). That link is outbound only: it lets the app reach private resources, but doesn't make the app itself private.
- Outbound to the internet from private subnets goes through a NAT gateway, which also gives you a static outbound IP for partners' allow-lists. A Lambda attached to a VPC gets no public IP, so without NAT its calls to external APIs time out.
Internet ─► Edge (CDN + WAF) ─► app (no public access)
│ outbound link into the private network
▼
private subnet ─► private endpoints: database · secrets · queue
private DNS: the services' normal hostnames resolve to private IPs
│
└─► NAT gateway ─► partner APIs (static IP)
Private endpoints depend on DNS. The service's normal hostname must resolve to the private IP inside your network (a private DNS zone on Azure, private DNS for interface endpoints on AWS, DNS records for Private Service Connect on Google Cloud). If it doesn't, the hostname resolves to the public IP, the request leaves over the internet, and the service rejects it because public access is off. When a private connection fails, check DNS first (nslookup from the app).
Cost trade-offs
| Model | You pay for | Cheap when | Expensive when |
|---|---|---|---|
| VMs | Instances, 24/7, plus the people who patch them | Steady load with reserved pricing | Idle capacity, small teams |
| PaaS app hosting | Provisioned instances, 24/7 | Several apps share instances; steady traffic | Many small apps idling at night |
| Managed containers | vCPU and memory while instances run (some bill per request) | Scale to zero; spiky or moderate load | Many always-on instances at high utilisation |
| Kubernetes | Nodes, the control plane fee, and the platform team | Large, dense, steady workloads; spot node pools | A few services run by a small team |
| Functions | Executions and memory-time | Spiky or low volume, idle most of the day | Busy all day: per-execution pricing overtakes an always-on instance |
| Static hosting | Storage and CDN egress, often a free tier | Almost always | Rarely the cost driver |
- Steady 24/7 load → committed pricing (AWS Savings Plans and Reserved Instances, Azure reservations and savings plans, Google Cloud committed use discounts). Spiky or idle → pay per use. Interruptible work (batch, CI, stateless workers) can run on spot capacity on any of the three.
- Hidden costs repeat per environment: NAT gateways and private endpoints bill by the hour and by data processed, log ingestion grows with traffic, and cross-region or internet egress is billed per GB.
- Count people, not just instances. A Kubernetes cluster that saves 20% on compute but needs a part-time platform engineer is not cheaper for a team of five.
How to say it in a design review
"The React app is static files on object storage behind the CDN, with a preview per pull request. The order API runs on a managed container platform with two minimum instances across zones, scaling on concurrent requests, and a maximum set so its connection pools fit the database. The payment worker scales from zero to 30 on queue depth. The nightly report is a scheduled run-to-completion job, and order approval is a workflow engine with a 24-hour timer and compensation if payment fails. Deploys are canaries at 5% with automatic rollback; schema changes are expand and contract. The app reaches the database through a private endpoint with private DNS, and calls the payment provider through NAT for a fixed IP. On AWS that's ECS on Fargate, Step Functions and EventBridge Scheduler; on Azure, Container Apps, Durable Functions and a Container Apps job; on Google Cloud, Cloud Run, Workflows and Cloud Scheduler."
Checklist
- Hosting chosen per workload (API, worker, job, event handler, workflow, frontend), not one size for all
- Kubernetes only when you need its API; otherwise managed containers
- Functions: cold starts handled on latency-sensitive paths, execution and HTTP timeouts respected, concurrency capped
- Long processes in a workflow engine: idempotent steps, compensation, versioning; deterministic code if it replays
- Scale on queue depth or concurrency; minimum instances for latency-sensitive APIs; maximum sized for the database
- Zero-downtime path (rolling, blue/green or canary) with readiness checks and backward-compatible schema changes
- Private endpoints with working private DNS; outbound link and NAT where needed; public access disabled
- Cost: idle capacity, per-environment fixed costs, egress and log ingestion estimated
Practise this concept
In the app: a short video and a 12-question concept check. Free with a GitHub account.
Open in the app