System design · Concept 7

Async Messaging: Queues and Streams

When to put a queue or a log between services, delivery guarantees, ordering, dead-letter queues, backpressure and fan-out patterns.

6 min read

In short

  • Go async for buffering, retries and fan-out, not by default
  • Queues for work items, streams for events and replay
  • At-least-once delivery plus idempotent consumers
  • Partition by entity key for ordering; partitions cap parallelism
  • Dead-letter poison messages, and watch backlog age

Synchronous calls couple services in time: if the callee is slow or down, so is the caller. Putting a message broker between them turns "do this now" into "this needs doing" and gives you buffering, retries and independent scaling. The price is eventual completion, duplicate handling and more infrastructure.

Why go async?

BenefitExample
DecouplingOrder service publishes order.placed without knowing email, analytics and fraud exist
Load levelingA 50K/s burst is absorbed by the queue and drained at 5K/s by workers
ResilienceEmail provider is down for 10 minutes; messages wait and retry, nothing is lost
Faster responsesAPI returns 202 Accepted in 20 ms; the 3 s video transcode happens later
Fan-outOne event consumed by many independent services

Keep the call synchronous when the user needs the result now to continue (e.g. "is this username free?", "was the card authorized?").

Queues vs streams

Queue:   producer ─► [ m5 m4 m3 m2 m1 ] ─► worker A  (m1)     each message to ONE consumer,
                                       └─► worker B  (m2)     deleted after ack

Stream:  producer ─► [ e1 e2 e3 e4 e5 … ] (retained log, partitioned)
                         ▲ offset 3: consumer group "email"
                         ▲ offset 5: consumer group "analytics"   each group reads everything
Message queueEvent stream / log
ExamplesSQS, Azure Service Bus, Google Cloud Pub/Sub, RabbitMQKinesis, Azure Event Hubs, Google Cloud Managed Service for Apache Kafka, Kafka, Pulsar
ConsumptionCompeting consumers; each message processed once by one workerEach consumer group reads the whole log at its own offset
RetentionUntil acked (plus DLQ)Time/size based (days, or forever); replayable
OrderingUsually none (FIFO queues: per group)Strict within a partition
Per-message featuresVisibility timeouts, delays, priorities, per-message retry, DLQMostly DIY (retry topics), offsets per partition
Scaling consumersAdd workers freelyParallelism capped by number of partitions
Best forTask/job distribution: send email, resize image, run jobEvent sourcing, fan-out to many services, analytics pipelines, CDC, replay

A common combination: stream for domain events, queues for work items. For example, an SNS topic fanning out to several SQS queues, or a Kafka consumer that enqueues jobs.

Delivery semantics

GuaranteeHowRisk
At-most-onceAck/commit before processingCrash = message lost
At-least-onceAck/commit after processingCrash after work but before ack = duplicate
Exactly-once (effectively)At-least-once + idempotent consumer, or transactional read-process-write (Kafka transactions within Kafka)More complexity; external side effects still need idempotency
receive(m) ─► process(m) ─► write result ─► ack(m)
                                  ▲ crash here → m redelivered → must be safe to redo

Design for at-least-once and make consumers idempotent: dedup on a message ID, use upserts or conditional updates, and pass idempotency keys to third-party APIs. "Exactly-once delivery" across a network isn't achievable in general. Exactly-once processing is achievable through idempotency.

Ordering

  • Global ordering doesn't scale (one partition = one consumer). Ask what needs ordering, which is usually per entity: per user, per order, per conversation.
  • Partition by that entity key (Kafka key, SQS FIFO MessageGroupId). All events for order-42 land in one partition and are consumed in order.
  • Parallelism = number of partitions. Choose enough up front (e.g. 64-256) because increasing partitions remaps keys.
  • Retries can reorder messages. A failed message retried later lands behind newer ones. Options: block the partition while retrying (strict order, head-of-line blocking), or carry a version/sequence number so consumers ignore stale updates.
  • Hot keys make one partition the bottleneck. Spread them with sub-keys if strict per-key order isn't required.

Retries and dead-letter queues

  • Visibility timeout (SQS) / lock duration (Service Bus) / ack deadline (Pub/Sub): a received message is hidden for N seconds. If it isn't acked in time, it reappears. Set N above the p99 processing time, and extend it for long jobs (heartbeat).
  • Retry with exponential backoff (1 s, 2 s, 4 s… plus jitter) via delayed redelivery or retry topics (orders.retry.1m, orders.retry.10m).
  • Dead-letter queue (DLQ): after N attempts (e.g. 5), move the message aside so a poison message doesn't block or burn resources forever.
  • DLQs need alerts, dashboards and a redrive tool. A DLQ nobody watches is a silent data-loss bin.
  • Distinguish transient errors (timeouts, 503: retry) from permanent ones (validation error, 400: DLQ immediately).

Backpressure

When producers outpace consumers, something has to give:

MechanismHow
BufferThe queue absorbs bursts. Watch queue depth / consumer lag
Scale consumersAutoscale workers on backlog (queue depth, lag in seconds)
Bound the queueReject or slow producers when full (429, blocking puts)
Load sheddingDrop or sample low-value work (analytics events) under pressure
Pull-based consumptionConsumers fetch at their own pace (Kafka, SQS), unlike push which can overwhelm
Rate limit downstreamWorkers respect third-party limits (token bucket per provider)

Key metrics: queue depth, age of oldest message, consumer lag (time), processing rate, error rate, DLQ size. Alert on age of oldest message, since that's what users feel.

Fan-out patterns

  • Pub/sub fan-out: one topic, many subscriber queues (SNS → SQS ×3, Kafka consumer groups). Each service gets its own copy and fails independently.
  • Competing consumers: many workers on one queue for throughput.
  • Fan-out on write: one event triggers N per-recipient messages (notifications to group members, feed inserts). Watch the amplification (1 post × 1M followers).
  • Request-reply over messaging: reply-to queue + correlation ID, for long-running work.
  • Claim check: put large payloads (>256 KB-1 MB) in object storage and send only the reference in the message.

Message design

  • Use an envelope: eventId, type, version, occurredAt, source, correlationId, payload.
  • Events state facts in the past tense (order.placed). Commands state intent (send-email).
  • Schema evolution: add optional fields only, never repurpose fields, and use a schema registry (Avro/Protobuf/JSON Schema) for streams.
  • Prefer thin events with IDs when consumers need fresh data anyway, and fat events when consumers shouldn't call back (decoupling, replay).

When to use / trade-offs

  • Use a queue for background tasks, smoothing bursts, calling slow or unreliable third parties, and retries.
  • Use a stream when multiple consumers need the same events, when you need replay or history, or when ordering per key matters at high throughput.
  • Stay synchronous when the caller needs the answer to proceed and the dependency is fast and reliable.
  • Costs: eventual consistency (the user may not see the result immediately), duplicate handling, harder debugging (use correlation IDs and tracing), another system to operate, and poison messages.
  • Don't add a broker "for scale" without a reason. A direct call is simpler until a requirement (spikes, retries, fan-out) justifies the hop.

Checklist

  • Why is this hop async? (latency, buffering, fan-out, retries)
  • Queue or stream, and which partition/ordering key?
  • At-least-once + idempotent consumers (dedup key)
  • Retry policy with backoff, DLQ with alerting and redrive
  • Backpressure: autoscaling on lag, bounded producers, shedding
  • Message schema with versioning; large payloads via claim check
  • Metrics: depth, oldest message age, lag, DLQ size

Practise this concept

In the app: a short video and a 11-question concept check. Free with a GitHub account.

Open in the app