The model that can explain quantum field theory cannot reliably count the letters in “strawberry.” That contradiction isn’t a reasoning failure — it’s a perception failure, and it happens before the model does any thinking at all, in a preprocessing step most people never look at: the tokenizer. Understanding tokenization explains a surprising share of LLM quirks, and it’s pure computer science — a vocabulary, a greedy merge algorithm, and consequences.

Models Don’t See Characters

A language model operates on tokens, not letters — integer IDs into a fixed vocabulary of maybe 50k–200k entries. The tokenizer maps text to those IDs, and the mapping is subword: common words are one token, rare words split into pieces, and the pieces are learned, not designed:

text
  "tokenizer"  → ["token", "izer"]      2 tokens
  "strawberry" → ["str", "aw", "berry"] 3 opaque chunks
  " the"       → one token (leading space included!)
  "1234567"    → maybe ["123", "456", "7"] — inconsistent digit grouping

The model sees token IDs like [496, 3213], never the characters inside them. So “how many R’s in strawberry” asks the model to introspect the spelling of chunks it perceives as atomic symbols — roughly like asking you how many times the pixel-shape of a letter appears in a word you only know by sound. It can sometimes fake it from training data about spelling, but the information genuinely isn’t in its native representation. Same root cause: reversing strings, character-level edits, and rhyming are all harder than tasks that sound far more complex.

The Algorithm: Byte-Pair Encoding

Most modern tokenizers use BPE (byte-pair encoding — originally a 1994 compression algorithm, repurposed). Training it is simple and greedy: start with individual bytes/characters, then repeatedly find the most frequent adjacent pair in the corpus and merge it into a new token, for a fixed number of merges.

text
  start:  l o w e r   l o w e s t
  merge most frequent pair "l o" → "lo"
  merge "lo w" → "low"     ... continue N times
  result: "low" is one token; rare words stay in small pieces

The elegance: frequent sequences (common words, def , </div>, the) become single tokens automatically, rare ones decompose into reusable parts, and because the base is bytes, nothing is ever unrepresentable — any Unicode, any binary, any emoji falls back to byte tokens (this is why byte-level BPE beat word-level: no out-of-vocabulary problem, ever). It’s a compression codebook that the model then learns to reason over.

The Consequences That Bite

Once you see tokenization as a lossy, frequency-biased compression layer between text and model, a batch of real behaviors explains itself:

  • Arithmetic is hard partly because of tokenization. If “1234” and “1235” tokenize into different chunk groupings, the model can’t rely on positional digit structure — the number’s form is inconsistent. Newer tokenizers deliberately split digits uniformly (one per token, or fixed groups) specifically to help math, a direct fix to a tokenization problem.
  • Token cost varies wildly by language. English is what the vocabulary was optimized on, so it’s ~1 token per short word. Many non-Latin scripts and less-represented languages fragment into many more tokens for the same meaning — sometimes 2–3× — which means they cost 2–3× more per API call and consume context budget faster. A real fairness-and-cost issue baked into the vocabulary.
  • Context limits are in tokens, not characters. “8k context” depends on what you put in it; code with lots of rare identifiers or a non-English language fills it faster than English prose.
  • Code formatting matters more than expected. Whitespace and indentation tokenize in specific ways (some tokenizers have special handling for runs of spaces); this is part of why models are sensitive to code formatting and why tokenizer improvements for code measurably help.
  • The “glitch token” phenomenon. Strings that appeared in the tokenizer’s training corpus but barely in the model’s (famously SolidGoldMagikarp) got a token ID the model never learned to handle, producing bizarre outputs — a seam between two training stages, visible only if you know the two stages exist.

The systems lesson generalizes: there’s a lossy encoding layer between the user’s data and the model’s computation, its design choices leak into behavior, and debugging model quirks often means looking below the model at how the input was represented — the same instinct as checking the wire format before blaming the application.

Takeaways

  • LLMs perceive subword tokens, not characters — the strawberry/ letter-counting/spelling failures are perception limits, not reasoning ones.
  • BPE builds the vocabulary by greedily merging frequent byte pairs; byte-level fallback means nothing is ever out-of-vocabulary.
  • Inconsistent digit tokenization hurts arithmetic — modern tokenizers split digits deliberately to fix it.
  • Token cost and context consumption vary 2–3× by language/script — a cost and fairness consequence encoded in the vocabulary.
  • Tokenization is a lossy encoding layer whose choices leak into behavior; debug quirks by inspecting the tokens, not just the prompt.