Unit 1.3 · about 3 h
292 / 324Tokenizer design choices: vocabulary size, Unigram and special tokens
A shorter token sequence buys context space, but a larger vocabulary costs parameters and output work. Special-token recognition also decides whether ordinary input can impersonate a document boundary.
- explain
- derive
You can now
- Contrast BPE's ordered merges with Unigram's scored segmentations and vocabulary pruning.
- Derive bytes-per-token, sequence-count and vocabulary-axis parameter consequences of a vocabulary change.
- Define fixed special-token IDs and distinguish trusted boundary insertion from matching untrusted literal text.
- Compare tokenizer efficiency on matched language and code slices without treating compression as model quality.
Covers
- Unigram tokenizer intuition and contrast with BPE
- Vocabulary size, token fertility and sequence-length trade-offs
- Special-token IDs, boundaries and injection policy
- Tokenizer comparisons across language and code slices
Study, in this order
4 steps
- Unigram tokenizationHugging Face LLM Course - tokenizer translation
Training algorithm, Tokenization algorithm and Back to training; the Viterbi explanation and the
'huggun'Now your turn! questionDepth · Stop before Implementing Unigram. Recompute examples from the frequency table: the printed
'pug'products mistakenly use 5 rather than 17 for p and pu. - BPE vocabulary sizes and special tokensModern LLM Notebook - tokenization and data experiments
Vocabulary Sizes in Practice and Appendix: Special Tokens and the Vocabulary, immediately before Summary
Depth · Treat the vocabulary-size ranges as examples, not an optimum for every language or corpus. Skip the later step-by-step BPE appendix.
- Compression and special-token contractsStanford CS336 Assignment 1 - tokenizer contract
Section 2.3 (p. 5), Special tokens in Section 2.4 (pp. 6 to 7), and Problem (
tokenizer_experiments), parts (a) and (b) in Section 2.7 (p. 12)Depth · Use the experiment's matched-sample method on small local slices; do not train the full OpenWebText tokenizer here.
- Reserved-token policyTokenizer and Corpus Pipeline - reduced foundation slice
Architecture constraints: reserved tokens cannot be produced from ordinary text unless explicitly escaped or policy-defined
Focus on
- Unigram maximizes a segmentation's product of token probabilities, or minimizes its summed negative logs. BPE follows merge ranks; neither is simply longest-prefix matching.
- For \(B\) bytes encoded as \(T\) tokens, compression is \(B/T\). Compare the same text and preprocessing on each tokenizer, and report per-slice results rather than only a pooled mean.
- Adding \(\Delta V\) tokens adds \(\Delta V \times d\) embedding parameters, plus the same again for an untied output head. Shorter sequences do not make vocabulary costs disappear.
- CS336 deliberately recognizes configured special strings. A corpus-facing interface needs a separate policy for untrusted literals and trusted boundary insertion.
Reproduce closed-book
'ab' with \(P(a)=0.4\), \(P(b)=0.4\) and \(P(ab)=0.2\), and select [ab]. Then derive that 12,000 bytes at 3 versus 4 bytes/token require 4,000 versus 3,000 tokens, or 8 versus 6 length-512 windows if the tail is padded.Traps
- Assuming the fewest-token Unigram segmentation always wins. Token probabilities, not token count alone, determine the score.
- Calling a tokenizer better because it compresses English prose more. Code, other languages, throughput and downstream behavior can disagree.
- Letting a user-typed EOS spelling silently acquire the trusted EOS ID. Decide whether to reject, escape or encode it as ordinary bytes.
Practice
Hugging Face LLM Course: Unigram `'huggun'` segmentation
Now your turn!: determine the tokenization of 'huggun' and its score
Enumerate valid paths from the given vocabulary and frequency table before using code. The best path is [hug, g, un], with score \((15/210) \times (20/210) \times (16/210)\).
Deep-ML Lab 19: Build a Tokenizer for Language Modeling
Implement train_tokenizer(corpus, vocab_size) in standard Python; the lab trains a bigram model on your tokens and scores held-out bits per character
Compare a byte-level baseline with BPE at the full 256-token budget, and check whether the better compressor also gives the lower bits per character.
Self-test
-
Why can Unigram choose a segmentation that is not the longest available prefix?
It scores the whole path. A shorter first piece can lead to a larger total probability; Viterbi compares complete paths through the token lattice. -
A 32,000-token vocabulary grows to 48,000 at width 512. How many parameters does one embedding matrix gain?
\(16{,}000 \times 512 = 8{,}192{,}000\). An untied output matrix gains another 8,192,000; a tied matrix does not. -
A 1,000-byte sample takes 250 tokens with tokenizer A and 400 with B. What is the compression ratio?
A has 4 bytes/token and B has 2.5. A uses fewer tokens on this sample; that alone says nothing about model quality. -
Does reserving an EOS ID prevent special-token injection?
No. The encoder must also distinguish trusted insertion from untrusted literal text. Reserving an ID only establishes its meaning.
Completion criterion
Stop when
You solve the segmentation and window-count examples without notes, and can defend vocabulary and literal-special-token policies with separate prose, code and non-English examples.
Mark passed, then open 1.4 Only mark this when you can do it without notes.Leads to
- 1.4 A tokenizer you can trust: round trips, identity and streaming Serialize the complete tokenization policy, not just the vocabulary.
- 2.1 Embeddings, positions and the vocabulary axis Account for the vocabulary axis in model shapes.
Comes after
- 1.2 Byte-level BPE: training, merges and tie-breakingWeigh vocabulary size and contrast BPE with Unigram.
Resources for this unit
- Stanford CS336 Assignment 1 - tokenizer contract Read the Assignment 1 tokenizer handout and use the named unicode1, unicode2, train_bpe, tokenizer and tokenizer_experiments specifications plus public tests as an acceptance contract. Implement independently in your own repository and do not copy or modify course solutions.
- Modern LLM Notebook - tokenization and data experiments Run English notebooks 01-tokenizer-basics and 02-bpe-tokenizer. Use only the quality, deduplication and mixture examples from 15-data-engineering. Change the corpus and tie cases, then verify every quantitative claim independently.
- Tokenizer and Corpus Pipeline - reduced foundation slice Use the byte, identity, split, deduplication, contamination, packing, manifest and restart correctness requirements. Keep the corpus small and stop before distributed processing or internet-scale filtering.
- Hugging Face LLM Course - tokenizer translation After the blank-file tokenizer passes, use the tokenizer chapters to map vocabulary, special tokens, fast tokenizers and batching into library APIs. Do not replace the deterministic implementation gate with a pretrained tokenizer call.
Also in this section
- LLMs from Scratch companion - tokenization and data loading Work through the ch02 main notebook, 05_bpe-from-scratch and 04_bonus_dataloader-intuition. Rebuild byte-level BPE, serialization and packed windows in blank files, then add multilingual, control, whitespace and chunk-boundary properties.
- Python unicodedata - normalization forms Read
unicodedata.normalizefor canonical and compatibility equivalence, andunidata_versionfor recording the normalization database version. - Hugging Face Datasets - streaming, sharding and resume Read Shard, Interleave, Save a dataset checkpoint and resume iteration, and Export. Use tiny local data and inspect the warning that shuffle buffers are lost on resume; do not download web-scale corpora or upload to the Hub.