System design · Concept 6
Consistency, CAP and Distributed Transactions
CAP and PACELC, the consistency spectrum, idempotency, and how sagas and the outbox pattern replace distributed transactions.
In short
- During a partition, choose consistency or availability
- Every request trades latency against consistency
- Pick a consistency model per use case
- Make every retried write idempotent
- Sagas and the outbox replace distributed transactions
Once data lives on more than one machine, you have to decide what happens when those machines disagree or can't talk. This lesson gives you the vocabulary to make that decision explicitly and the patterns to keep multi-service workflows correct.
CAP in one paragraph
In a distributed system, network partitions will happen (P is not optional). When one happens, a system must choose:
- Consistency (C): every read sees the latest write or gets an error. Some requests are refused.
- Availability (A): every request gets a non-error response, possibly with stale data.
So CAP really says: during a partition, choose C or A. A bank balance service picks C (refuse rather than show a wrong balance). A social feed picks A (show slightly stale posts rather than an error page).
PACELC: the more useful version
If Partition → choose Availability or Consistency; Else (normal operation) → choose Latency or Consistency.
Partitions are rare, but the latency vs consistency trade-off is paid on every request. Waiting for replicas in other zones or regions to confirm costs milliseconds to hundreds of milliseconds.
| System (default/typical config) | During partition | Normal operation |
|---|---|---|
| Cassandra, DynamoDB (default reads) | Availability | Latency (eventual) |
| Spanner, CockroachDB | Consistency | Consistency (pay latency) |
| Postgres primary + async replicas | Consistency on primary | Latency on replicas (stale reads) |
| MongoDB (majority writes/reads) | Consistency | Consistency |
Most of these systems are tunable (per-request consistency levels), so you can choose per use case.
Consistency models (strongest to weakest)
| Model | Guarantee | Where you need it |
|---|---|---|
| Linearizable (strong) | Behaves like a single copy; reads see the latest completed write | Locks, leader election, balances, inventory decrement, unique usernames |
| Sequential | All nodes see operations in the same order | Replicated logs |
| Causal | Causally related operations are seen in order (reply after post) | Comments, chat threads |
| Read-your-writes | A user always sees their own updates | Profile edits, settings, "I just posted this" |
| Monotonic reads | Never see older data after newer | Paging through timelines |
| Eventual | Replicas converge if writes stop | Like counts, view counts, feeds, analytics |
Practical approach: default to eventual where users won't notice, and add targeted guarantees where they will. Read-your-writes for the author, strong consistency for anything involving money or uniqueness.
Quorums
With N replicas, writes acknowledged by W and reads querying R: R + W > N guarantees the read set overlaps the latest write (e.g. N=3, W=2, R=2). Use W=1, R=1 for speed with eventual consistency, and W=N for durability-critical writes.
Idempotency: the foundation of safe retries
Networks drop responses, so clients retry. Message brokers deliver at least once. That means every write handler will eventually see duplicates. An operation is idempotent if applying it twice has the same effect as applying it once.
| Technique | Example |
|---|---|
| Idempotency key | Client sends Idempotency-Key: 7f3a…; server stores key → result; repeats return the stored result |
| Natural key / upsert | INSERT … ON CONFLICT (order_id) DO NOTHING |
| Conditional write | UPDATE … SET status='PAID' WHERE id=? AND status='PENDING' |
| Version / ETag | If-Match: "v7"; reject if the version changed |
| Dedup table | Store processed message IDs (with TTL) in the same transaction as the side effect |
POST /payments Idempotency-Key: K1
├─ BEGIN
├─ INSERT INTO idempotency_keys (key, request_hash, status) VALUES (K1, h, 'in_progress')
│ └─ conflict? → return stored response (or 409 if still in progress, 422 if hash differs)
├─ ...do the work, store response...
└─ COMMIT
SET x = 5 is naturally idempotent. x = x + 5 isn't, so turn increments into "apply event E once".
Distributed transactions: 2PC
Two-phase commit coordinates a commit across several databases:
Coordinator ── prepare ──► DB1, DB2 (each votes yes/no and holds locks)
Coordinator ── commit ──► DB1, DB2 (only if all voted yes)
Problems:
- Blocking: if the coordinator dies after prepare, participants hold locks until it recovers.
- Latency: several round trips, with locks held across the network.
- Availability: every participant must be up, so availability multiplies down.
- Limited support across heterogeneous systems (queues, third-party APIs can't join).
2PC is fine inside one database product (distributed SQL uses consensus internally). Avoid it across microservices.
Sagas
A saga splits a business transaction into local transactions, each with a compensating action that semantically undoes it.
Create Order ──► Reserve Inventory ──► Charge Payment ──► Ship
▲ │ fail │ fail
└── Cancel Order ◄─┴── Release Stock ◄──┘ (compensations run in reverse)
| Style | How | Pros | Cons |
|---|---|---|---|
| Orchestration | A coordinator (workflow engine, state machine) tells each service what to do | Flow is explicit and observable, easy to change | Coordinator is extra infrastructure |
| Choreography | Services react to each other's events | Loose coupling, no central component | Hard to see and debug the overall flow; cyclic dependencies |
Saga design tips:
- Compensations are business actions (refund, release reservation). They're not rollbacks, and they must themselves be idempotent and retryable.
- Order steps so that hard-to-undo steps go last (charging a card, sending email). The point of no return is called the pivot transaction.
- Sagas have no isolation: other requests can see intermediate states. Use status fields (
PENDING), semantic locks or reservations. - Durable workflow engines (AWS Step Functions, Azure Durable Functions, Google Cloud Workflows, Temporal) implement orchestration with retries and timers.
The outbox pattern: no more dual writes
The dual-write problem: you update your DB and then publish an event to Kafka. If the process crashes between the two, the DB and the rest of the world disagree forever.
Service ──BEGIN──► orders table (UPDATE status='PAID')
outbox table (INSERT event 'order.paid')
──COMMIT──
Relay (poller or CDC e.g. Debezium) ──reads outbox──► Kafka ──► consumers (idempotent)
- The state change and the event are committed atomically in one local transaction.
- A relay publishes outbox rows, either by polling or by tailing the DB log (CDC). It may publish twice after a crash, so it's at-least-once.
- Consumers must be idempotent (dedup by event ID, the inbox pattern).
- Clean up published outbox rows periodically.
When to use / trade-offs
| Need | Choose | Cost |
|---|---|---|
| Money, inventory, uniqueness | Strong consistency (single leader / consensus) | Higher latency, lower availability during partitions |
| Feeds, counters, analytics | Eventual consistency | Stale reads, conflict handling |
| Atomic change across services | Saga + outbox | Compensation logic, no isolation, eventual completion |
| Atomic change within one DB | Local ACID transaction | None, so keep data that changes together in one DB |
| Safe retries anywhere | Idempotency keys / conditional writes | Storage for keys, more design up front |
Rule of thumb: keep transactional boundaries inside one service and one database. Cross those boundaries with events (outbox), make every handler idempotent, and use sagas for multi-step business processes.
Checklist
- For each data type: what consistency do users actually need?
- What happens during a partition or region outage: refuse or serve stale?
- Every retried write is idempotent (keys, upserts, conditional updates)
- No dual writes: outbox or CDC for DB + event
- Multi-service workflows modeled as sagas with compensations
- Read-your-writes where users will notice staleness
Practise this concept
In the app: a short video and a 11-question concept check. Free with a GitHub account.
Open in the app