0.1 + 0.2 == 0.30000000000000004 is the famous one, and it’s the least interesting floating-point surprise. It has a boring cause (0.1 isn’t representable in binary, same as 1/3 isn’t in decimal) and a boring fix (don’t == floats). The genuinely hazardous facts are the ones that make the same program give different answers on different machines, or make a sum depend on the order you added things — because those break reproducibility, tests, and distributed consensus in ways that look like heisenbugs. (I’ve written elsewhere on the bit-level format; this is about the traps that follow from it.)

The Core Sin: Addition Isn’t Associative

In real numbers, (a + b) + c == a + (b + c). In IEEE 754, it often doesn’t, because each operation rounds to the nearest representable value, and where the rounding lands depends on the intermediate magnitudes:

text
  (1e20 + -1e20) + 1.0  =  0.0 + 1.0     =  1.0
   1e20 + (-1e20 + 1.0) =  1e20 + -1e20  =  0.0   ← the 1.0 vanished

The 1.0 was absorbed: added to 1e20, it falls below the precision of that magnitude and rounds away entirely. Reorder, and it survives. This single fact — non-associativity — is the root of most serious float trouble, and it has three teeth:

  • Summation order changes results. Sum a million floats left-to-right vs sorted vs pairwise and you get different answers. For a large sum of mixed magnitudes the naive order can lose most of your significant digits, because each addition rounds against a running total that’s grown large relative to the addends.
  • Parallelism reorders implicitly. Split that sum across 8 threads or SIMD lanes and the reduction order changes with the thread count — so your result depends on core count and scheduling. “Same code, same input, different answer per run” is usually this.
  • The compiler reorders too. -ffast-math (and pieces of it like -fassociative-math) explicitly permits the optimizer to treat float math as associative for speed/vectorization — trading reproducibility and some correctness for throughput. Great for a game’s shading, potentially disastrous for a numerical solver or anything that must match across builds. Know whether it’s on.

Catastrophic Cancellation: Losing Everything at Once

The other classic destroyer: subtracting two nearly-equal numbers. The equal leading digits cancel, leaving only the noisy trailing bits — you can go from 15 significant digits to 2 in one subtraction:

text
  1.2345678901234 - 1.2345678900000 = 0.0000000001234
  15 good digits in each input → ~4 in the result

This is why the textbook quadratic formula is numerically unstable for certain coefficients, why computing variance as E[x²] - E[x]² is a known footgun (Welford’s algorithm exists to dodge exactly this), and why the fix is usually algebraic reformulation to avoid the subtraction, not more precision. Doubling to 64-bit or 128-bit float delays the problem; it doesn’t change that the algorithm is throwing away information.

Where This Bites, and What To Do

The failure modes are concrete: flaky tests that assert float equality (compare with a tolerance appropriate to the magnitude — absolute epsilon fails for large values; use relative or ULP-based comparison); distributed systems that hash or compare computed floats across heterogeneous nodes and disagree (never put raw float results in a consensus/equality path — quantize to integers or fixed-point first); financial code using float for money (use integer cents or a decimal type — float dollars will eventually be off by a penny in an auditable place); and GPU/CPU results that don’t match because fused-multiply-add (one rounding instead of two) changed the last bits.

The defensive toolkit: never == — compare with tolerance; don’t use float for money or exact counts — integers/decimals; mind the algorithm, not just the type — reformulate to avoid cancellation, use Kahan/Neumaier compensated summation or pairwise reduction for large sums (Kahan carries the lost low-order bits in a correction term and recovers most of them); and control the knobs — know your -ffast-math status and pin FMA/rounding modes if cross-platform determinism is a requirement, because by default it isn’t one.

Takeaways

  • The real hazard isn’t representation error, it’s non-associativity: reordering a sum changes the result, so order, parallelism, and fast-math all alter answers.
  • Naive summation of mixed magnitudes loses precision by absorption; Kahan/pairwise summation recovers it.
  • Catastrophic cancellation destroys significant digits in one near-equal subtraction — fix the algorithm (Welford, reformulated quadratic), not the precision.
  • Never == floats; never use them for money or cross-node equality/ consensus — quantize to integers/decimals at those boundaries.
  • Know your -ffast-math and FMA settings; determinism across machines is opt-in, not the default.