Tokenization

Level 2 builds the kit of bricks from nothing, starting with what a token is.

Everyday picture A box of Lego with a fixed set of bricks. Common shapes have a ready-made brick; anything unusual gets assembled from smaller bricks. You can build anything, but some builds take many more bricks than others. A tokenizer is that kit for text: a fixed vocabulary of pieces (typically 32,000 to 200,000 of them), each with a number, its token id.

Tiny worked example With a kit that contains the, un, believ and able, "the" takes 1 brick and "unbelievable" takes 3: un + believ + able. The model never sees the letters; it sees a short list of ids such as [262, 403, 11009, 540].

flowchart LR
  T["'the unbelievable'"] --> TOK["tokenizer<br/>(fixed kit of pieces)"]
  TOK --> P["pieces: 'the' | ' un' | 'believ' | 'able'"]
  P --> IDS["ids: 262, 403, 11009, 540"]
  IDS --> EMB["embedding table<br/>one vector per id"]
  EMB --> M["the model"]

In code: trained_tokenizer builds this lesson's toy kit, and ByteBPE.encode turns any text into its list of ids.

Why it matters Everything downstream is counted in tokens: price, speed, and how much fits in the context window.

Everyday picture A court stenographer inventing shorthand. Whatever pair of letters they write most often gets its own squiggle. Then the most common pair including that squiggle gets one too, and so on, until they have as many squiggles as they can remember.

Tiny worked example Training text "low lower lowest". Start from single letters and repeatedly merge the most frequent neighbouring pair (train_char_bpe reproduces this exactly):

At step 1, l+o and o+w are tied at 3; ties go to the pair seen first.

flowchart LR
  T[Training text] --> S[Split into characters]
  S --> C[Count adjacent pairs]
  C --> M[Merge the most frequent pair<br/>record the merge rule]
  M -->|vocab not full yet| C
  M -->|vocab full| V[Vocabulary + ordered merge list]

The math and the code Each round picks the pair with the largest count:

# f_w: how often each distinct word occurs
f = {"low": 1, "lower": 1, "lowest": 1}
# n_w(a, b): a immediately followed by b inside w
def n(w, a, b):
    return sum(1 for i in range(len(w) - 1) if w[i] == a and w[i + 1] == b)
def count(a, b):
    # Σ_w f_w · n_w(a, b)
    return sum(f_w * n(w, a, b) for w, f_w in f.items())
count("l", "o"), count("e", "r")  # → (3, 1)

In code: train_char_bpe runs the count-and-merge loop and returns one MergeStep per round, holding the winning pair, its count, the new token and the text after the merge (one row of the table above).

Why it matters Frequent strings end up as single tokens and rare ones stay in pieces, which is exactly why token counts differ from word counts, and why a tokenizer trained mostly on English is cheap for English and expensive for everything else (section 6).

Everyday picture Following a recipe card: do step 1 everywhere it applies, then step 2, and so on. Skipping ahead would give a different dish.

Tiny worked example Encode "lowest" with the three merges learned above: l o w e s t → (merge 1) lo w e s t → (merge 2) low e s t → (merge 3) lowe s t. Result: 3 tokens. A word never seen in training still works: "slow" → s low; "newer" matches no merge and stays as letters.

flowchart LR
  W["'lowest' as letters<br/>l o w e s t"] --> R1["merge 1: l+o<br/>lo w e s t"]
  R1 --> R2["merge 2: lo+w<br/>low e s t"]
  R2 --> R3["merge 3: low+e<br/>lowe s t"]
  R3 --> OUT["no learned merge applies<br/>tokens: lowe | s | t"]

The code encode_char_bpe repeatedly finds, among the pairs present, the one learned earliest, merges it, and loops. Training and encoding agree because they apply rules in the same order.

Why it matters Encoding is deterministic: the same text always gives the same ids, which is what makes prompt caching and cost estimates possible.

Everyday picture Every file on a computer, whatever it holds, is a sequence of bytes: numbers from 0 to 255. If the smallest bricks in the kit are those 256 byte values, then no text can ever be unbuildable: emoji, Chinese, source code or garbage all come apart into bytes.

Tiny worked example In UTF-8 (the standard way to store text as bytes) "a" is one byte, 97; "é" is two bytes, 195 169; "🙂" is four bytes, 240 159 153 130. An untrained byte-level tokenizer turns "é🙂" into 6 tokens. After training on English, " password" is one token, id 291, because it was frequent.

flowchart LR
  T["Text: 'Reset your password'"] --> P["Pre-tokenize (regex)<br/>'Reset' | ' your' | ' password'"]
  P --> B["UTF-8 bytes per chunk<br/>' your' = 32 121 111 117 114"]
  B --> R["Replay merges, earliest first<br/>inside each chunk only"]
  R --> I["Token ids<br/>346 377 309 328 291"]
  I --> D["Decode: look up bytes per id,<br/>concatenate, UTF-8 decode"]

The code ByteBPE is train_char_bpe with bytes instead of letters, ids 0-255 for the bytes and 256, 257, ... for each merge. GPT-2, GPT-4's cl100k_base, Llama 3 and most current LLMs work this way.

In code: ByteBPE.train learns the byte merges, ByteBPE.encode and ByteBPE.decode make the round trip, and compression_curve trains tokenizers of growing size and measures each one for the figure above.

Why it matters No "unknown token" failures, ever, for any input. The price is that unfamiliar scripts fall back to near-byte level and cost many more tokens.

Everyday picture BPE glues the most common pair. WordPiece glues the most inseparable pair: two pieces that almost never appear apart, like "Q" and "u" in English. Unigram works the other way round: start with a huge kit and keep throwing out the bricks you'd miss least.

Tiny worked example On "low lower lowest", BPE's first merge is l+o. WordPiece's first merge is s+t: "s" and "t" each appear exactly once, and always together.

flowchart TB
  subgraph BPE["BPE (GPT, Llama)"]
    b1[start small] --> b2[merge most frequent pair] --> b3[grow to target size]
  end
  subgraph WP["WordPiece (BERT)"]
    w1[start small] --> w2["merge pair with best<br/>count(ab) / count(a)·count(b)"] --> w3[grow to target size]
  end
  subgraph UG["Unigram (T5, SentencePiece default)"]
    u1[start huge] --> u2[drop pieces whose loss<br/>hurts likelihood least] --> u3[shrink to target size]
  end

The math WordPiece ranks pairs by

f = {"low": 1, "lower": 1, "lowest": 1}
# how often a symbol, or a pair written together, appears
def count(piece):
    return sum(f_w * w.count(piece) for w, f_w in f.items())
def score(a, b):
    # count(ab) / (count(a) · count(b))
    return count(a + b) / (count(a) * count(b))
score("s", "t"), round(score("l", "o"), 2)  # → (1.0, 0.33)

BERT marks continuation pieces with ## ("un", "##believ", "##able"). SentencePiece is a library that runs BPE or Unigram directly on raw text, writing spaces as the visible symbol ▁.

Why it matters Different model families segment the same text differently, so their token counts and costs differ. Always count with the tokenizer of the model you'll actually call.

Everyday picture A taxi meter that ticks per token, not per mile, with a pricier meter for the return trip: output tokens usually cost several times more than input tokens.

Tiny worked example A prompt of 2,000 tokens with a 500-token answer, at example prices of $3 per million input tokens and $15 per million output tokens: 0.006 + 0.0075 = $0.0135. A million such calls cost $13,500.

flowchart LR
  P["prompt text"] --> TI["count input tokens<br/>meter A: $ per million in"]
  TI --> M[model]
  M --> TO["count output tokens<br/>meter B: $ per million out (pricier)"]
  TO --> BILL["bill = A + B"]
  TI -.-> CTX["also: must fit the<br/>context window"]

The math and the code

T_in, T_out = 2_000, 500
# dollars per million tokens
p_in, p_out = 3, 15
round(T_in / 10**6 * p_in + T_out / 10**6 * p_out, 4)  # → 0.0135

Rule of thumb: English prose averages about 4 characters per token, or 0.75 words per token, so 1,000 tokens ≈ 750 words (estimate_tokens, tokens_to_words). Use it for quick estimates; count with the real tokenizer for anything that matters.

In code: chars_per_token computes each bar: the length of the text divided by the number of ids ByteBPE.encode returns for it.

Why it matters The same request can differ several-fold in cost, latency and context usage depending on language and content: dense JSON, code, tables and non-English text all need more tokens than English prose.

Everyday picture Reading through frosted glass that only shows whole bricks. You can tell which bricks are there, but not the letters printed on them.

Tiny worked example (this module's toy tokenizer):

flowchart LR
  W["' strawberry'"] --> T["tokenizer"] --> I["ids 396, 311"]
  I --> M["model sees two ids<br/>no letters, no count of r"]
  Q["'how many r's?'"] --> M

The code ByteBPE.tokens(text) shows the pieces for any string; the demo prints these cases.

Why it matters These quirks explain surprising behaviour: miscounted letters, shaky arithmetic, odd handling of rare names. Some newer tokenizers split digits into fixed groups of up to 3 to reduce the arithmetic problem.

Answer each one out loud or on paper before you open it. If you can explain it, you know it.