A transaction touches five pages scattered across a B-tree. Crash mid-write and the tree is neither old nor new — it’s shrapnel. You cannot make five random writes atomic; disks don’t offer that. But you can make one sequential append durable, and that asymmetry is the whole foundation: write what you intend to do into a log, make the log durable, then do it. Postgres WAL, InnoDB redo log, ext4’s journal, Kafka, Raft’s log, RocksDB — one idea, a dozen uniforms. Learn it once with the failure modes attached and half of storage engineering unlocks.

The Protocol

text
  1. append to log:  "txn 42: page 7 bytes 96..128 := <new>" (+ more records)
  2. append:         "txn 42 COMMIT"
  3. fsync the log   ◀── the transaction is durable HERE
  4. report success
  5. update the real pages in memory; write them back... eventually

The rule that names the technique: no data page reaches disk before the log records describing its changes (write-ahead). Recovery is then mechanical: replay the log forward from the last checkpoint (redo), and for transactions with no COMMIT record, either ignore their records (redo-only designs) or roll them back (undo). Crash at any step and you get either all of txn 42 or none of it.

Notice what got bought. Performance, not just safety: the durable write path became sequential appends to one file — the access pattern both disks and SSDs love — while the random-write chaos happens lazily in memory and trickles back in the background. And group commit compounds it: ten transactions committing within the same millisecond share a single fsync (one flush, ten durability guarantees) — the mechanism behind every “how does this database do 50k commits/sec on one disk” number, and behind knobs like Postgres’s commit_delay and Kafka producers’ batching. Latency-vs-throughput at the fsync boundary is the tunable of durable systems: relaxations like synchronous_commit = off (ack before flush, bounded loss window on crash) are honest engineering trades when stated, and quiet data-loss bugs when unstated — MongoDB’s early unacknowledged-write defaults taught the industry that lesson publicly.

Where WAL Implementations Get Hurt

The protocol is simple; the storage stack underneath is treacherous, and the classic wounds are all documented incidents:

  • fsync must mean it. Consumer drives ack writes from volatile cache; a power cut eats “durable” bytes. Barriers/FUA and honest disks (or capacitor-backed enterprise SSDs) are prerequisites, not options. Worse: Linux fsync error semantics — a failed fsync could mark pages clean, so retrying it “succeeds” over lost data. That’s fsyncgate (2018): Postgres now PANICs on fsync failure, on the theory that crash-and-recover-from-WAL beats believing a liar.
  • Torn pages. An 8 KB page write isn’t atomic against 4 KB-sector power loss — half old, half new, checksum garbage. Postgres logs full-page images on first touch after each checkpoint (why WAL volume spikes right after checkpoints); InnoDB routes through its doublewrite buffer. Same threat, two armors.
  • Checkpoints exist to bound recovery, and they’re the other latency villain: flushing all dirty pages at once creates I/O storms (hence spread/fuzzy checkpointing, checkpoint_completion_target), while checkpointing rarely means longer crash-recovery replays. The triangle — recovery time, steady-state I/O, WAL retention — has no fourth option.

One more gift for free: the log is a replication feed. Ship WAL to a replica and replay it — physical replication, PITR archives, and (decoded) logical CDC pipelines like Debezium are all “read someone else’s WAL.” The log turns out to be the database’s most reusable interface — which is the observation Kafka generalized into a company, and Raft formalized into consensus: replicate the log, and state machines replaying it agree by construction.

Takeaways

  • Random multi-page atomicity is impossible; one durable sequential append is easy — WAL converts the first problem into the second.
  • Durability point = log fsync; group commit amortizes it and is where commit throughput comes from.
  • The stack lies: verify write caches/barriers, treat fsync failure as fatal (fsyncgate), and armor against torn pages (FPI/doublewrite).
  • Checkpoints trade steady-state I/O against recovery time; tune the spread, accept the triangle.
  • The WAL is also the replication/CDC/consensus interface — the log is the database, everything else is a cache of it.