“Put a load balancer in front” hides a stack of very different machines: DNS answers, ECMP router hashes, kernel-level packet rewriters, and full userspace HTTP proxies — usually several at once (a request to a large service commonly crosses four balancing tiers before touching application code). Knowing which layer does what — and where the real failure modes live — turns “the LB is acting up” into an actual diagnosis.

The Two Species

L4 balancers operate on connections: they see 5-tuples, pick a backend per new flow, and rewrite/encapsulate packets — NAT-style (rewrite destination, must sit on the return path), tunnel/DSR-style (encapsulate to the backend, which answers directly to the client — the asymmetry that lets one balancer front massive egress, since responses skip it entirely; IPVS speaks all three modes). They’re fast, cheap per connection, and blind: no headers, no requests, no retries — one connection in, one connection out. Google’s Maglev and its descendants (Katran) are this species scaled out: many identical L4 boxes behind router ECMP, using consistent hashing per flow so that when a balancer node dies, most flows still land on the same backend via its siblings — connection state made largely stateless.

L7 balancers (nginx, Envoy, HAProxy in HTTP mode, ALBs) terminate the client’s connection, parse the protocol, and originate a new connection to a backend. That word “terminate” carries everything: per-request (not per-connection) routing — which HTTP/2 made mandatory, since one client connection multiplexes hundreds of requests that would otherwise all pin one backend — plus retries, timeouts, TLS offload, header-based routing, connection pooling. The bill: the balancer is now a full proxy with buffer memory, TLS CPU, its own connection limits, and its own failure domain — plus two observability gotchas that eat debugging days: backends see the proxy’s IP (hence X-Forwarded-For / PROXY protocol), and client idle-timeout mismatches between the tiers produce those intermittent 502s where the proxy reuses a connection the backend just closed.

Algorithms: Less Important Than Advertised, Except When Not

  • Round robin is fine when requests are uniform and backends identical — i.e., in demos. Variance is the enemy: one slow request pins a backend that RR keeps feeding.
  • Least connections adapts to variance — outstanding connections proxy for load — and is the sane default for L7 pools with mixed request costs.
  • Power of two choices (pick two random backends, send to the less loaded) is the one with a beautiful theorem behind it: exponential improvement in max load over random, at O(1) cost, and — crucially for distributed balancers — it degrades gracefully when load information is stale, where global least-connections herd-stampedes the momentarily-idlest node. It’s Envoy’s default for a reason.
  • Hashing (client IP, cookie, header) is for stickiness/affinity — session state, cache locality — and is a constraint, not a balancer: hash-based schemes inherit hot-key imbalance and break sessions on pool resize unless consistent/ring hashing (Maglev, ketama) bounds the churn.

The dirty secret of the algorithm wars: in most real outages the algorithm was irrelevant and health checking was everything. The questions that matter: active checks (probe /healthz) vs passive (watch real failures — outlier ejection); how fast a dead backend is ejected vs how fast a flapping one oscillates; whether the check verifies dependencies (a backend that’s up but its database isn’t — should it fail the check and risk cascading removal of the whole pool? — every serious LB has a “panic threshold”/fail-open mode because someone learned this live); and draining — removing a backend from rotation while letting in-flight requests finish, the difference between deploys being invisible and deploys being a 502 spike. Slow-start/warmup ramps (new backend gets traffic gradually — cold caches, JIT warmup) round out the list of things that page you.

The composed picture, one request deep: DNS/anycast picks a region → ECMP sprays across L4 tier → consistent-hash to an L7 proxy → least-loaded/P2C to a backend — four decisions, three algorithms, and health state at every layer. When “the LB is broken,” start by asking which one.

Takeaways

  • L4 moves connections (NAT/tunnel/DSR — mind the return path); L7 terminates and re-originates, buying per-request routing (mandatory under H2) at proxy-failure-domain prices.
  • Scaled L4 = ECMP + consistent hashing (Maglev/Katran): flow stability without shared state.
  • Least-connections beats RR under variance; power-of-two-choices wins under stale/distributed load info; hashing is affinity, not balancing.
  • Health checks, ejection thresholds, panic/fail-open behavior, and connection draining cause (and prevent) more incidents than any algorithm choice.
  • Requests traverse several balancing tiers — diagnose by naming the tier before naming the symptom.