The most-studied bug in security history is also one of the simplest to write: copy untrusted bytes into a fixed buffer without checking the length. Understanding exactly how that becomes a compromise — and, more importantly, how each mitigation layer breaks a specific link in the chain — is core defensive knowledge. This is written for the defender’s side of the fence: to configure hardening correctly and read a crash for exploitability, you have to know what the attacker needed.

Why the Stack Is the Target

Local variables and the function’s return address share the stack, adjacent, growing toward each other. A buffer that overflows writes upward into the saved return address:

text
   [ char buf[64] ][ saved rbp ][ return address ]
        │ overflow ───────────────────▶ overwritten
   gets(buf); // 200 bytes of attacker input → return addr = attacker's

When the function returns, the CPU jumps to whatever sits in that slot. Control-flow hijack from a copy loop. The historical exploit (Aleph One’s 1996 “Smashing the Stack”, the intellectual starting point for this whole field) placed shellcode in the buffer itself and pointed the return address back at it. strcpy, gets, sprintf, unchecked memcpy — the unbounded-copy family is the classic source, and integer-overflow-into-undersized-malloc the modern one.

The Mitigation Stack, Layer by Layer

No single defense “fixes” this; the durable win came from stacking independent ones, each invalidating an assumption the exploit relied on. Reading them in order is the mental model:

  • Stack canaries (-fstack-protector-strong, on by default). A random value placed between buffers and the return address, checked on function exit; a linear overflow that reaches the return address must first overwrite the canary, and the mismatch aborts (*** stack smashing detected ***). Breaks sequential stack overflow — sidestepped by info-leaks (read the canary first) or writes that skip over it.
  • NX / DEP (W^X — no page both writable and executable). The stack is data, marked non-executable, so shellcode-in-buffer simply faults. Breaks code injection — and attackers answered with ROP (return-oriented programming): don’t inject code, chain together existing snippets (“gadgets”) ending in ret. NX raised the bar from “write shellcode” to “build a ROP chain.”
  • ASLR — randomize the base addresses of stack, heap, libraries, (and PIE) the executable. Now the attacker doesn’t know where the gadgets or their buffer are. Breaks the hardcoded address, including most ROP — its kryptonite is any information leak that discloses one runtime address (from which the rest are offsets), so leaks became the center of gravity of modern exploitation.
  • CFI / shadow stacks (the current frontier: Clang CFI, Intel CET’s hardware shadow stack + endbranch). Even with a leak, constrain indirect jumps/returns to legitimate targets — a shadow stack keeps an untamperable copy of return addresses and faults on mismatch, attacking ROP at its root.

The lesson generalizes past this one bug: defense in depth as assumption-invalidation. Each layer is individually bypassable and the stack is collectively strong, because chaining a bypass for all of them (a leak and a suitable overflow and gadgets and CET evasion) is the difference between a 1999 script and a modern research write-up. It didn’t kill the bug class — memory-safety bugs are still ~70% of the CVEs Microsoft and Chrome report yearly, which is the actual argument for Rust and for -fbounds-safety-style work — but it made each instance dramatically harder to weaponize.

What Defenders Do With This

Concretely: compile with the modern defaults on (-D_FORTIFY_SOURCE=3 turns unbounded libc calls into checked ones when the size is known at compile time; -fstack-protector-strong; -fstack-clash-protection; PIE + full RELRO so the GOT and return path are hardened) and verify they’re present (checksec, hardening-check) rather than assuming the toolchain did it. Find the bugs before shipping with ASan (shadow memory catches the overflow at the write, with a stack trace) and coverage-guided fuzzing on every parser that eats untrusted bytes — that pairing is how the overflow gets caught in CI instead of in a CVE. And when triaging a crash, “SIGSEGV with a controlled register” or a tripped canary is the signal to treat it as security-relevant, not a routine null-deref — the exploitability question is exactly the mitigation-bypass question above.

Takeaways

  • Overflows win by overwriting the saved return address adjacent to stack buffers; unbounded copies and integer-overflow allocations are the sources.
  • The mitigations each kill one link: canaries (sequential overwrite), NX (code injection), ASLR (known addresses), CFI/shadow stack (ROP/return hijack).
  • Every layer is individually bypassable — via info leaks above all — and collectively raises weaponization from scripting to research.
  • Turn the defaults on and check them (checksec, FORTIFY, PIE/RELRO, stack-clash); assume nothing about your build.
  • Catch instances pre-ship with ASan + fuzzing; memory-safe languages remove the class, which is why the industry is migrating.