ss -tan | awk '{print $1}' | sort | uniq -c is one of my first commands on any misbehaving box, and it only means something if the state names do. When I implemented a TCP stack in software, the state machine went from an exam diagram to the actual data structure at the heart of the protocol — and it turns out the production folklore (port exhaustion, connection leaks, half-closed hangs) is all legible directly from the transitions. Eleven states; here’s the tour, weighted by operational pain.

Setup: Three Packets, Two Queues

text
  client                          server
  CLOSED                          LISTEN
    │ ── SYN ──────────────────▶  SYN_RECV
  SYN_SENT                          │
    │ ◀────────────── SYN+ACK ──   │
    │ ── ACK ──────────────────▶  ESTABLISHED
  ESTABLISHED

The handshake exists to synchronize sequence numbers — each side picks an ISN and acknowledges the other’s; both directions of the byte stream get independent numbering. Two operational facts hide in SYN_RECV: the kernel holds embryonic connections in a SYN queue until the final ACK promotes them to the accept queue (where they wait for your accept() call). SYN floods target the first queue — the SYN cookies countermeasure encodes the connection state into the ISN so nothing is stored. A slow application targets the second: when the accept queue overflows (somaxconn, the listen() backlog), Linux silently drops handshake-completing ACKs, and clients see mysteriously hanging connects. ss -lt shows both queue depths per listener; nstat -az TcpExtListenDrops counts the bodies.

Teardown: Four Packets and the Two Problem States

Close is two independent half-closes — each direction’s FIN/ACK pair — because each byte stream ends separately:

text
  active closer                   passive closer
  ESTABLISHED                     ESTABLISHED
    │ ── FIN ──────────────────▶  CLOSE_WAIT   (app must close()!)
  FIN_WAIT_1                        │
    │ ◀──────────────────  ACK ──  │
  FIN_WAIT_2                        │
    │ ◀──────────────────  FIN ──  LAST_ACK
  TIME_WAIT ── ACK ────────────▶  CLOSED
    │ (2×MSL, then CLOSED)

CLOSE_WAIT is an application bug wearing a network costume. The remote side closed; the kernel is waiting for your process to call close(). CLOSE_WAIT sockets accumulating means a code path that stops reading without closing — a leaked fd, an exception past the cleanup. No kernel tunable fixes it; lsof finds the process, the fix is in your code.

TIME_WAIT is by design — and still bites. The active closer holds the socket for 2×MSL (60 s on Linux) to (a) retransmit the final ACK if the peer’s FIN re-arrives and (b) let stray delayed segments die before the 4-tuple is reused, where they could corrupt a successor connection. The famous incident shape: a service making rapid short-lived outbound connections to one backend accumulates tens of thousands of TIME_WAITs, and since the 4-tuple space to one destination is bounded by ~28k ephemeral ports, connect() starts failing with EADDRNOTAVAIL at a few hundred new connections/second. Real fixes, in order of correctness: reuse connections (keep-alive/pooling — the actual answer), net.ipv4.tcp_tw_reuse=1 (safe: timestamps prove segment freshness for outbound reuse), more ephemeral ports / more source IPs. The folklore fix tcp_tw_recycle was so unsafe behind NAT that the kernel removed it entirely in 4.12 — its ghost still haunts old runbooks.

One more: FIN_WAIT_2 with a dead peer. You closed, they ACKed, their FIN never comes (crashed, or a middlebox ate it). For orphaned sockets the kernel times this out (tcp_fin_timeout); for sockets your process still holds, the half-open connection sits until keepalive — which defaults to two hours idle before the first probe. Any long-lived-connection service that doesn’t set TCP_KEEPIDLE/TCP_KEEPINTVL (or application-level heartbeats) is betting its fd table on peers always dying politely.

Watching the Machine Run

The whole post is checkable live: ss -tan state time-wait, ss -o state fin-wait-2 (the -o shows the running timers), watch 'ss -s' during a load test, and nstat deltas for drops and overflows. Implementing the machine yourself is the deep version — my enduring takeaway was that the states aren’t bureaucracy; every one exists because some packet can arrive late, twice, or never, and the diagram is the complete answer to “then what?”

Takeaways

  • Handshake problems are queue problems: SYN queue (floods, cookies) vs accept queue (slow apps, silent ACK drops) — ss -lt shows both.
  • CLOSE_WAIT piling up = your code didn’t close(); no sysctl will save you.
  • TIME_WAIT is protection, not garbage; exhaustion is an outbound-churn problem solved by pooling, tcp_tw_reuse, or more 4-tuple space — never recycle.
  • Half-open connections outlive dead peers for hours unless you configure keepalive or heartbeat at the application layer.
  • ss with state filters and timers makes every claim here verifiable on a live box in seconds.