System design · Concept 7
Async Messaging: Queues and Streams
When to put a queue or a log between services, delivery guarantees, ordering, dead-letter queues, backpressure and fan-out patterns.
In short
- Go async for buffering, retries and fan-out, not by default
- Queues for work items, streams for events and replay
- At-least-once delivery plus idempotent consumers
- Partition by entity key for ordering; partitions cap parallelism
- Dead-letter poison messages, and watch backlog age
Synchronous calls couple services in time: if the callee is slow or down, so is the caller. Putting a message broker between them turns "do this now" into "this needs doing" and gives you buffering, retries and independent scaling. The price is eventual completion, duplicate handling and more infrastructure.
Why go async?
| Benefit | Example |
|---|---|
| Decoupling | Order service publishes order.placed without knowing email, analytics and fraud exist |
| Load leveling | A 50K/s burst is absorbed by the queue and drained at 5K/s by workers |
| Resilience | Email provider is down for 10 minutes; messages wait and retry, nothing is lost |
| Faster responses | API returns 202 Accepted in 20 ms; the 3 s video transcode happens later |
| Fan-out | One event consumed by many independent services |
Keep the call synchronous when the user needs the result now to continue (e.g. "is this username free?", "was the card authorized?").
Queues vs streams
Queue: producer ─► [ m5 m4 m3 m2 m1 ] ─► worker A (m1) each message to ONE consumer,
└─► worker B (m2) deleted after ack
Stream: producer ─► [ e1 e2 e3 e4 e5 … ] (retained log, partitioned)
▲ offset 3: consumer group "email"
▲ offset 5: consumer group "analytics" each group reads everything
| Message queue | Event stream / log | |
|---|---|---|
| Examples | SQS, Azure Service Bus, Google Cloud Pub/Sub, RabbitMQ | Kinesis, Azure Event Hubs, Google Cloud Managed Service for Apache Kafka, Kafka, Pulsar |
| Consumption | Competing consumers; each message processed once by one worker | Each consumer group reads the whole log at its own offset |
| Retention | Until acked (plus DLQ) | Time/size based (days, or forever); replayable |
| Ordering | Usually none (FIFO queues: per group) | Strict within a partition |
| Per-message features | Visibility timeouts, delays, priorities, per-message retry, DLQ | Mostly DIY (retry topics), offsets per partition |
| Scaling consumers | Add workers freely | Parallelism capped by number of partitions |
| Best for | Task/job distribution: send email, resize image, run job | Event sourcing, fan-out to many services, analytics pipelines, CDC, replay |
A common combination: stream for domain events, queues for work items. For example, an SNS topic fanning out to several SQS queues, or a Kafka consumer that enqueues jobs.
Delivery semantics
| Guarantee | How | Risk |
|---|---|---|
| At-most-once | Ack/commit before processing | Crash = message lost |
| At-least-once | Ack/commit after processing | Crash after work but before ack = duplicate |
| Exactly-once (effectively) | At-least-once + idempotent consumer, or transactional read-process-write (Kafka transactions within Kafka) | More complexity; external side effects still need idempotency |
receive(m) ─► process(m) ─► write result ─► ack(m)
▲ crash here → m redelivered → must be safe to redo
Design for at-least-once and make consumers idempotent: dedup on a message ID, use upserts or conditional updates, and pass idempotency keys to third-party APIs. "Exactly-once delivery" across a network isn't achievable in general. Exactly-once processing is achievable through idempotency.
Ordering
- Global ordering doesn't scale (one partition = one consumer). Ask what needs ordering, which is usually per entity: per user, per order, per conversation.
- Partition by that entity key (Kafka key, SQS FIFO
MessageGroupId). All events fororder-42land in one partition and are consumed in order. - Parallelism = number of partitions. Choose enough up front (e.g. 64-256) because increasing partitions remaps keys.
- Retries can reorder messages. A failed message retried later lands behind newer ones. Options: block the partition while retrying (strict order, head-of-line blocking), or carry a version/sequence number so consumers ignore stale updates.
- Hot keys make one partition the bottleneck. Spread them with sub-keys if strict per-key order isn't required.
Retries and dead-letter queues
- Visibility timeout (SQS) / lock duration (Service Bus) / ack deadline (Pub/Sub): a received message is hidden for N seconds. If it isn't acked in time, it reappears. Set N above the p99 processing time, and extend it for long jobs (heartbeat).
- Retry with exponential backoff (1 s, 2 s, 4 s… plus jitter) via delayed redelivery or retry topics (
orders.retry.1m,orders.retry.10m). - Dead-letter queue (DLQ): after N attempts (e.g. 5), move the message aside so a poison message doesn't block or burn resources forever.
- DLQs need alerts, dashboards and a redrive tool. A DLQ nobody watches is a silent data-loss bin.
- Distinguish transient errors (timeouts, 503: retry) from permanent ones (validation error, 400: DLQ immediately).
Backpressure
When producers outpace consumers, something has to give:
| Mechanism | How |
|---|---|
| Buffer | The queue absorbs bursts. Watch queue depth / consumer lag |
| Scale consumers | Autoscale workers on backlog (queue depth, lag in seconds) |
| Bound the queue | Reject or slow producers when full (429, blocking puts) |
| Load shedding | Drop or sample low-value work (analytics events) under pressure |
| Pull-based consumption | Consumers fetch at their own pace (Kafka, SQS), unlike push which can overwhelm |
| Rate limit downstream | Workers respect third-party limits (token bucket per provider) |
Key metrics: queue depth, age of oldest message, consumer lag (time), processing rate, error rate, DLQ size. Alert on age of oldest message, since that's what users feel.
Fan-out patterns
- Pub/sub fan-out: one topic, many subscriber queues (SNS → SQS ×3, Kafka consumer groups). Each service gets its own copy and fails independently.
- Competing consumers: many workers on one queue for throughput.
- Fan-out on write: one event triggers N per-recipient messages (notifications to group members, feed inserts). Watch the amplification (1 post × 1M followers).
- Request-reply over messaging: reply-to queue + correlation ID, for long-running work.
- Claim check: put large payloads (>256 KB-1 MB) in object storage and send only the reference in the message.
Message design
- Use an envelope:
eventId,type,version,occurredAt,source,correlationId,payload. - Events state facts in the past tense (
order.placed). Commands state intent (send-email). - Schema evolution: add optional fields only, never repurpose fields, and use a schema registry (Avro/Protobuf/JSON Schema) for streams.
- Prefer thin events with IDs when consumers need fresh data anyway, and fat events when consumers shouldn't call back (decoupling, replay).
When to use / trade-offs
- Use a queue for background tasks, smoothing bursts, calling slow or unreliable third parties, and retries.
- Use a stream when multiple consumers need the same events, when you need replay or history, or when ordering per key matters at high throughput.
- Stay synchronous when the caller needs the answer to proceed and the dependency is fast and reliable.
- Costs: eventual consistency (the user may not see the result immediately), duplicate handling, harder debugging (use correlation IDs and tracing), another system to operate, and poison messages.
- Don't add a broker "for scale" without a reason. A direct call is simpler until a requirement (spikes, retries, fan-out) justifies the hop.
Checklist
- Why is this hop async? (latency, buffering, fan-out, retries)
- Queue or stream, and which partition/ordering key?
- At-least-once + idempotent consumers (dedup key)
- Retry policy with backoff, DLQ with alerting and redrive
- Backpressure: autoscaling on lag, bounded producers, shedding
- Message schema with versioning; large payloads via claim check
- Metrics: depth, oldest message age, lag, DLQ size
Practise this concept
In the app: a short video and a 11-question concept check. Free with a GitHub account.
Open in the app