“Strings are arrays of characters” stopped being true decades ago, but an enormous amount of code still acts like it is — and then someone types an emoji into the name field. Understanding UTF-8 takes about ten minutes, fixes a whole category of bugs, and as a bonus is one of the most elegant pieces of bit-level design you’ll ever see. Ken Thompson and Rob Pike sketched it on a diner placemat in 1992.

Code Points Are Not Bytes

Unicode assigns every character a number — a code point — from U+0000 to U+10FFFF: A is U+0041, é is U+00E9, 中 is U+4E2D, 🚀 is U+1F680. That’s the abstract layer. An encoding decides how those numbers become bytes. UTF-8 is a variable-length encoding: 1 to 4 bytes per code point, and the bit patterns are the whole story:

text
  U+0000..U+007F     0xxxxxxx                              (ASCII, 1 byte)
  U+0080..U+07FF     110xxxxx 10xxxxxx                     (2 bytes)
  U+0800..U+FFFF     1110xxxx 10xxxxxx 10xxxxxx            (3 bytes)
  U+10000..U+10FFFF  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx   (4 bytes)

The code point’s bits are packed into the x positions. é (U+00E9, binary 11101001) becomes 110_00011 10_101001 = 0xC3 0xA9 — which is why broken pipelines show é: someone decoded those two bytes as Latin-1.

The Design Properties That Won

Look at the leading bits and notice what falls out:

  1. ASCII is valid UTF-8, unchanged. Every 7-bit ASCII file ever written was already UTF-8. Adoption required converting nothing.
  2. Self-synchronizing. Continuation bytes always start 10; lead bytes never do. Drop into the middle of a stream and you can find the next character boundary within 3 bytes. A corrupted byte damages one character, not the rest of the file.
  3. No false ASCII. Bytes 0x00–0x7F appear only as themselves — never inside a multi-byte sequence. So strchr(s, '/'), null-terminated strings, and every ASCII-era parser work on UTF-8 without modification. (UTF-16 fails this: it’s full of zero bytes.)
  4. Memcmp sorts correctly. Byte-wise comparison of UTF-8 strings equals code-point order. Databases and filesystems can sort without decoding.
  5. No byte-order problem. It’s a byte stream; there’s no 16-bit unit to swap, no BOM required (Windows tools add one anyway — EF BB BF — and Unix tools choke on it; that’s a compatibility scar, not part of the design).

UTF-16, which Windows, Java, and JavaScript adopted early, has none of properties 2–5 and still turned out variable-length once Unicode outgrew 16 bits — surrogate pairs are the permanent tax on that bet.

Where Code Goes Wrong

Almost every Unicode bug is one confusion: treating byte count, code point count, and what a user sees as the same number. They’re three numbers:

text
  "🇮🇳"  flag of India
    bytes (UTF-8):   8
    code points:     2   (regional indicators I + N)
    what users see:  1

len() in Python counts code points; strlen in C counts bytes; a text cursor should move by grapheme clusters (what users see), which requires Unicode’s segmentation rules — use a library. Truncating a string at byte 100 can cut a character in half and produce invalid UTF-8; truncate at a boundary (back up while (byte & 0xC0) == 0x80). And validate at the edges: overlong encodings (encoding / as two bytes) are illegal precisely because they were used to sneak path traversal past security checks — a validator must reject them, and every competent decoder since has.

There’s one more layer: the same visible text can be different code points — é as U+00E9 or as e + combining accent U+0301. Compare user-visible strings without normalizing (NFC/NFD) and “identical” strings won’t match. Filesystems disagree here too: macOS normalizes filenames, Linux stores bytes.

Takeaways

  • Code points are numbers; UTF-8 is how they become bytes — 1 to 4 of them.
  • The bit layout gives ASCII compatibility, self-synchronization, safe ASCII scanning, and memcmp ordering; that’s why it carries ~98% of the web.
  • Bytes, code points, and grapheme clusters are three different lengths; know which one each API returns.
  • Truncate on boundaries, validate (reject overlong forms) at input edges, normalize before comparing.
  • Declare/assume UTF-8 everywhere; encoding guessing is how é happens.