Tokenization algorithms: how a vocabulary is built

Why can a short name use more tokens than a longer familiar word? A tokenizer reuses pieces learned from its training text. The visible character count alone does not tell you which pieces it knows.

Before you begin: You know that tokens are model-specific pieces of text and that token IDs are not meanings.

The tokenizer is part of the model's interface. Replacing it with a different tokenizer while leaving the embedding table unchanged assigns the wrong learned representations to token IDs.

Learn pieces, then reuse them

Byte-pair encoding, or BPE, repeatedly merges frequent adjacent symbols in its training data. With character-based starting symbols, l o w might become lo w, then low. Byte-level variants start from a byte alphabet and use their own preprocessing rules. At inference, the learned merge priorities determine segmentation; the tokenizer does not retrain itself on your prompt.

This miniature example performs one merge. It teaches the local operation, not a production BPE trainer.

# Runnable: Python 3, standard library.
def merge_pair(symbols, pair):
    result = []
    i = 0
    while i < len(symbols):
        if tuple(symbols[i:i + 2]) == pair:
            result.append(''.join(pair))
            i += 2
        else:
            result.append(symbols[i])
            i += 1
    return result

assert merge_pair(list('banana'), ('a', 'n')) == [
    'b', 'an', 'an', 'a'
]
print(merge_pair(list('banana'), ('a', 'n')))

Notice that each input symbol is consumed once. Replacing overlapping pairs carelessly is a common mistake in toy implementations.

Does a merge preserve the original text?

The toy merge turns banana into b, an, an, a. Joining those pieces recovers the original word. Try the overlapping case aaaa with the pair a, a: a left-to-right pass should produce two aa pieces, consuming each original symbol once.

Now imagine a preprocessing step lowercases US before merging. Would joining the output recover the original capitalization?

Add both examples to a tokenizer inspection notebook. Record the original string, normalized form when observable, pieces, IDs, and decoded output. Those columns reveal more than a single token-count total.

Similar names hide different choices

Training and encoding are separate algorithms. For example, describing WordPiece inference as longest-match does not fully specify how its vocabulary was trained. Avoid treating an unofficial reconstruction of a training recipe as the only definition.

SentencePiece is a tokenizer library and text-processing approach. It supports algorithms including BPE and Unigram. It is not a fourth algorithm that is always synonymous with Unigram. Its treatment of whitespace is particularly useful when building tokenizers without assuming spaces separate every language's words.

Inspect the whole pipeline

Normalization can change text before segmentation. Lowercasing makes “US” and “us” harder to distinguish. Unicode normalization can combine or transform characters. Pre-tokenization sets boundaries, the model assigns pieces, and post-processing may insert special tokens.

Chat formatting is another layer: role markers, message separators, and generation prompts need the model's expected template. Text that looks identical in a chat window can produce a different model input when its roles are serialized differently.

A round trip through encode and decode is not always byte-for-byte identical when normalization is enabled. Decide whether that behavior is acceptable for your use case. Preserving source offsets matters for highlighting citations and redacting sensitive text.

Design a tokenizer check

Collect short examples from the actual workload: names, code indentation, punctuation, your users' languages, repeated whitespace, and unusual Unicode. Record token counts and decoded output using the exact tokenizer revision. Compare distributions, not a single English sentence. Longer token sequences can increase cost and reduce the amount of content that fits in context.

Exercise: your new tokenizer reduces English token counts but doubles Telugu token counts. Is it an improvement?

With the input representation settled, the next question is how to divide a training budget between the model and its data.

Sources

The implementation distinctions are documented in Hugging Face Tokenizers components and the SentencePiece project. Use those libraries' versioned tokenizer files for real models rather than this teaching example.

Continue: Model Scaling Laws.