System design · Concept 6

Consistency, CAP and Distributed Transactions

CAP and PACELC, the consistency spectrum, idempotency, and how sagas and the outbox pattern replace distributed transactions.

6 min read

In short

  • During a partition, choose consistency or availability
  • Every request trades latency against consistency
  • Pick a consistency model per use case
  • Make every retried write idempotent
  • Sagas and the outbox replace distributed transactions

Once data lives on more than one machine, you have to decide what happens when those machines disagree or can't talk. This lesson gives you the vocabulary to make that decision explicitly and the patterns to keep multi-service workflows correct.

CAP in one paragraph

In a distributed system, network partitions will happen (P is not optional). When one happens, a system must choose:

  • Consistency (C): every read sees the latest write or gets an error. Some requests are refused.
  • Availability (A): every request gets a non-error response, possibly with stale data.

So CAP really says: during a partition, choose C or A. A bank balance service picks C (refuse rather than show a wrong balance). A social feed picks A (show slightly stale posts rather than an error page).

PACELC: the more useful version

If Partition → choose Availability or Consistency; Else (normal operation) → choose Latency or Consistency.

Partitions are rare, but the latency vs consistency trade-off is paid on every request. Waiting for replicas in other zones or regions to confirm costs milliseconds to hundreds of milliseconds.

System (default/typical config)During partitionNormal operation
Cassandra, DynamoDB (default reads)AvailabilityLatency (eventual)
Spanner, CockroachDBConsistencyConsistency (pay latency)
Postgres primary + async replicasConsistency on primaryLatency on replicas (stale reads)
MongoDB (majority writes/reads)ConsistencyConsistency

Most of these systems are tunable (per-request consistency levels), so you can choose per use case.

Consistency models (strongest to weakest)

ModelGuaranteeWhere you need it
Linearizable (strong)Behaves like a single copy; reads see the latest completed writeLocks, leader election, balances, inventory decrement, unique usernames
SequentialAll nodes see operations in the same orderReplicated logs
CausalCausally related operations are seen in order (reply after post)Comments, chat threads
Read-your-writesA user always sees their own updatesProfile edits, settings, "I just posted this"
Monotonic readsNever see older data after newerPaging through timelines
EventualReplicas converge if writes stopLike counts, view counts, feeds, analytics

Practical approach: default to eventual where users won't notice, and add targeted guarantees where they will. Read-your-writes for the author, strong consistency for anything involving money or uniqueness.

Quorums

With N replicas, writes acknowledged by W and reads querying R: R + W > N guarantees the read set overlaps the latest write (e.g. N=3, W=2, R=2). Use W=1, R=1 for speed with eventual consistency, and W=N for durability-critical writes.

Idempotency: the foundation of safe retries

Networks drop responses, so clients retry. Message brokers deliver at least once. That means every write handler will eventually see duplicates. An operation is idempotent if applying it twice has the same effect as applying it once.

TechniqueExample
Idempotency keyClient sends Idempotency-Key: 7f3a…; server stores key → result; repeats return the stored result
Natural key / upsertINSERT … ON CONFLICT (order_id) DO NOTHING
Conditional writeUPDATE … SET status='PAID' WHERE id=? AND status='PENDING'
Version / ETagIf-Match: "v7"; reject if the version changed
Dedup tableStore processed message IDs (with TTL) in the same transaction as the side effect
POST /payments  Idempotency-Key: K1
  ├─ BEGIN
  ├─ INSERT INTO idempotency_keys (key, request_hash, status) VALUES (K1, h, 'in_progress')
  │     └─ conflict? → return stored response (or 409 if still in progress, 422 if hash differs)
  ├─ ...do the work, store response...
  └─ COMMIT

SET x = 5 is naturally idempotent. x = x + 5 isn't, so turn increments into "apply event E once".

Distributed transactions: 2PC

Two-phase commit coordinates a commit across several databases:

Coordinator ── prepare ──► DB1, DB2      (each votes yes/no and holds locks)
Coordinator ── commit  ──► DB1, DB2      (only if all voted yes)

Problems:

  • Blocking: if the coordinator dies after prepare, participants hold locks until it recovers.
  • Latency: several round trips, with locks held across the network.
  • Availability: every participant must be up, so availability multiplies down.
  • Limited support across heterogeneous systems (queues, third-party APIs can't join).

2PC is fine inside one database product (distributed SQL uses consensus internally). Avoid it across microservices.

Sagas

A saga splits a business transaction into local transactions, each with a compensating action that semantically undoes it.

Create Order ──► Reserve Inventory ──► Charge Payment ──► Ship
     ▲                  │ fail               │ fail
     └── Cancel Order ◄─┴── Release Stock ◄──┘   (compensations run in reverse)
StyleHowProsCons
OrchestrationA coordinator (workflow engine, state machine) tells each service what to doFlow is explicit and observable, easy to changeCoordinator is extra infrastructure
ChoreographyServices react to each other's eventsLoose coupling, no central componentHard to see and debug the overall flow; cyclic dependencies

Saga design tips:

  • Compensations are business actions (refund, release reservation). They're not rollbacks, and they must themselves be idempotent and retryable.
  • Order steps so that hard-to-undo steps go last (charging a card, sending email). The point of no return is called the pivot transaction.
  • Sagas have no isolation: other requests can see intermediate states. Use status fields (PENDING), semantic locks or reservations.
  • Durable workflow engines (AWS Step Functions, Azure Durable Functions, Google Cloud Workflows, Temporal) implement orchestration with retries and timers.

The outbox pattern: no more dual writes

The dual-write problem: you update your DB and then publish an event to Kafka. If the process crashes between the two, the DB and the rest of the world disagree forever.

Service ──BEGIN──► orders table   (UPDATE status='PAID')
                   outbox table   (INSERT event 'order.paid')
        ──COMMIT──
Relay (poller or CDC e.g. Debezium) ──reads outbox──► Kafka ──► consumers (idempotent)
  • The state change and the event are committed atomically in one local transaction.
  • A relay publishes outbox rows, either by polling or by tailing the DB log (CDC). It may publish twice after a crash, so it's at-least-once.
  • Consumers must be idempotent (dedup by event ID, the inbox pattern).
  • Clean up published outbox rows periodically.

When to use / trade-offs

NeedChooseCost
Money, inventory, uniquenessStrong consistency (single leader / consensus)Higher latency, lower availability during partitions
Feeds, counters, analyticsEventual consistencyStale reads, conflict handling
Atomic change across servicesSaga + outboxCompensation logic, no isolation, eventual completion
Atomic change within one DBLocal ACID transactionNone, so keep data that changes together in one DB
Safe retries anywhereIdempotency keys / conditional writesStorage for keys, more design up front

Rule of thumb: keep transactional boundaries inside one service and one database. Cross those boundaries with events (outbox), make every handler idempotent, and use sagas for multi-step business processes.

Checklist

  • For each data type: what consistency do users actually need?
  • What happens during a partition or region outage: refuse or serve stale?
  • Every retried write is idempotent (keys, upserts, conditional updates)
  • No dual writes: outbox or CDC for DB + event
  • Multi-service workflows modeled as sagas with compensations
  • Read-your-writes where users will notice staleness

Practise this concept

In the app: a short video and a 11-question concept check. Free with a GitHub account.

Open in the app