I’ve written before about why quorums exist; this is the protocol layer above them. Raft (Ongaro & Ousterhout, 2014) was explicitly designed around understandability — the paper’s actual thesis — and it worked: etcd, Consul, CockroachDB, TiKV, and most Kafka deployments (KRaft) now bet their correctness on it. The payoff of learning it properly is that distributed-systems logs stop being mystical: term 847, leader election lost, proposal dropped are all lines you can trace back to specific rules.

The Frame: Terms and One Leader

Time divides into numbered terms; each term has at most one leader, elected by majority vote. All writes flow through the leader — Raft buys simplicity by refusing multi-master entirely. Every message carries the sender’s term; seeing a higher term instantly demotes you to follower. That one rule quietly handles most of the “stale leader” zoo: a partitioned ex-leader rejoins, hears term 848 > its 847, and steps down before doing damage.

Election: followers expect heartbeats; silence past a randomized timeout (150–300 ms class) triggers candidacy — increment term, vote for self, request votes. Majority → leader. The randomization is the whole liveness trick (simultaneous candidacies split votes; random timeouts make repeats improbable), and the vote-granting rule is the first safety pillar: a node refuses its vote to any candidate whose log is less up-to-date than its own. Elected leaders therefore already hold every committed entry — recovery requires no log transfer to the new leader, ever.

Replication: The Log Matching Property

The leader appends client commands to its log and ships them via AppendEntries, each carrying the previous entry’s (index, term) as a consistency probe:

text
  leader:   ... [5:t2] [6:t3] [7:t3]        AppendEntries(prev=6:t3, [7:t3])
  follower: ... [5:t2] [6:t3]           ✓ matches → append 7
  follower: ... [5:t2] [6:t2]           ✗ reject → leader backs up, overwrites

Induction does the rest: if a follower matches at (index, term), its entire prefix matches — divergent suffixes (from crashed leaders’ unreplicated tail entries) get found and overwritten mechanically. An entry is committed once replicated on a majority and — the subtle clause that fixed a real bug class (the paper’s Figure 8) — the leader only counts replication of entries from its own term; older-term entries commit implicitly behind them. Committed entries feed the state machine, and the guarantee stack — elected leaders have all committed entries + log matching + commit rule — yields the property everything else stands on: committed means durable across any minority of failures, in order, exactly once in the log (delivery to your application is another story — see below).

Where Deployments Actually Bleed

The protocol is the easy half; a decade of etcd/Consul incident reports names the hard one:

  • fsync is part of the protocol. Votes and log entries must hit stable storage before being answered — a node that votes, crashes, and forgets its vote can elect two leaders in one term. Slow disks therefore are slow consensus: etcd’s wal_fsync_duration_seconds alerting exists because commit latency is disk latency; running consensus on shared/burstable storage is a well-documented way to spend a weekend.
  • Elections during GC/CPU stalls. A leader that pauses past the timeout gets deposed; the cluster churns through terms (“leadership transferred” storms). Tuning timeouts against your worst-case pause — and using pre-vote (a candidate first checks it could win, without incrementing terms) plus leader stickiness — is what keeps a flaky node from term-bombing a healthy cluster. If your logs show terms climbing by hundreds, it’s this.
  • Membership changes are a protocol, not a config edit. Jumping straight from {A,B,C} to {A,B,C,D,E} lets two disjoint majorities exist mid-change. Raft’s answers (joint consensus, or one-node-at-a-time — what etcd enforces, now with learner/non-voting states to catch new nodes up safely) are why “just add two replicas at once” is refused by good implementations.
  • Reads need care too. Serving reads from any leader replica risks stale data from a deposed leader; linearizable reads use ReadIndex (confirm leadership with a quorum round) or leases — the reason etcd distinguishes --consistency=l from serializable reads, and a knob every Raft-backed store exposes somewhere.

Scope honesty completes the picture: Raft totally orders a log — it says nothing about exactly-once effects in your application (dedup/idempotency still yours), it costs a quorum round-trip per write (don’t put it on hot paths that don’t need it), and a majority loss halts writes by design — availability was traded for the guarantee, which is the correct trade for the configuration/metadata/ coordination tier it dominates.

Takeaways

  • Terms + majority elections + higher-term-wins handle stale leaders; randomized timeouts handle liveness.
  • Log matching lets one (index, term) probe verify entire prefixes; commit = majority replication of a current-term entry.
  • Consensus latency = fsync latency; monitor WAL sync times before blaming the network.
  • Pre-vote and single-node membership changes are what production Raft looks like; their absence is what incidents look like.
  • Raft orders the log — idempotency, read consistency modes, and the cost of quorum rounds remain your problems.