Week 9 — Distributed Transactions & Consensus

The Problem

In a monolith, a database transaction is atomic. In a distributed system, you have multiple databases — atomicity across them requires coordination.

Order service (DB A) → debit inventory
Payment service (DB B) → charge customer
Shipping service (DB C) → create shipment

What if step 2 succeeds but step 3 fails?

Two-Phase Commit (2PC)

A coordinator manages atomic commit across participants.

Phase 1 — Prepare:
  Coordinator → "Can you commit?" → Participant A, B, C
  All respond "Yes, prepared" (locks held)

Phase 2 — Commit:
  Coordinator → "Commit" → A, B, C
  All commit and release locks

Fatal flaw: if the coordinator crashes after Phase 1, participants hold locks indefinitely waiting for Phase 2 that never comes. The system is stuck.

Use 2PC only within a single database system (e.g. PostgreSQL’s internal transaction manager). Avoid for cross-service distributed transactions.

Saga Pattern

Instead of enforcing atomicity, embrace eventual consistency. Each step commits locally. If a step fails, compensating transactions undo prior steps.

Step 1: Create order (local commit)
Step 2: Reserve inventory (local commit)
Step 3: Charge customer (local commit)
Step 4: Create shipment (local commit)

Step 3 fails →
  Compensate step 2: release inventory
  Compensate step 1: cancel order

Every step must be idempotent — retries are inevitable. A retry of “charge customer” must not double-charge.

Choreography vs Orchestration

	Choreography	Orchestration
Control	Each service reacts to events	Central coordinator directs each step
Coupling	Loose	Tighter (to coordinator)
Debugging	Hard — flow is implicit	Easier — flow is explicit
SPOF	No	Coordinator can be

Recommendation: use orchestration for business-critical flows where you need visibility.

Raft Consensus

How distributed nodes agree on who is leader and what data is committed.

Leader Election

• Nodes start as followers
• If no heartbeat from leader → become candidate → request votes
• Majority vote → become leader
• Split vote → random timeout → retry

Log Replication

1. Client sends write to leader
2. Leader appends to its log
3. Leader sends to followers
4. Majority ACK → commit
5. Leader tells followers to commit

Fault Tolerance

Cluster size	Tolerated failures
3 nodes	1
5 nodes	2
7 nodes	3

Don’t implement Raft yourself. Use etcd, ZooKeeper, or Consul. Patroni uses etcd for PostgreSQL HA.

ERP Example: Multi-Company Intercompany Transactions

Company A creates an invoice → triggers Company B’s vendor bill automatically.

This is a saga:

1. Post Company A invoice ✓
2. Create Company B bill ✓
3. Link the two ✓

If step 2 fails → compensate step 1 (reset to draft). If step 3 fails → compensate step 2 (delete bill) and step 1.

Every compensating transaction must be idempotent.

Self-check

• Why can’t 2PC guarantee progress if the coordinator crashes?
• What makes a good compensating transaction? What makes a bad one?
• A 5-node Raft cluster: how many nodes can you lose before writes stop?