A tokenizer you can trust: round trips, identity and streaming

A round trip can succeed even when token IDs differ from the model's training IDs. Serialization and streaming must preserve the exact encoding, not just readable text.

implement

You can now

Implement encode, decode and reload from a byte vocabulary and ordered merges without changing token IDs.
Test multilingual text, whitespace, controls, overlapping specials and malformed decoded byte sequences against a stated contract.
Preserve one-shot token IDs across streaming splits, retaining unfinished text until a safe boundary is known.
Reject tokenizer artifacts whose vocabulary, merge ranks or preprocessing policy differs from the expected identity.

Covers

Encode, decode, round-trip and serialized tokenizer identity
Streaming inputs and chunk-boundary invariance

Study, in this order

4 steps

Encoding, decoding and iterable input
Stanford CS336 Assignment 1 - tokenizer contract

Section 2.6, including Example (bpe_encoding), Memory considerations, Section 2.6.2 and Problem (tokenizer) (pp. 10 to 12)

Done
Saving, loading and reference tokenizer files
LLMs from Scratch companion - tokenization and data loading

Sections 3.2, Saving and loading the tokenizer, and 3.3, Loading the original GPT-2 BPE tokenizer from OpenAI

Depth · Use the save/reload examples as a workflow, not a byte-correctness oracle. Keep the CS336 bytes-based representation and test against tiktoken through the public suite.

Done
Public tokenizer acceptance cases
Stanford CS336 Assignment 1 - tokenizer contract

test_roundtrip_unicode_string, test_overlapping_special_tokens, the two test_encode_special_token_* cases, and test_encode_iterable_tinystories_matches_tiktoken; test_encode_iterable_memory_usage

Done
Tokenizer identity and adversarial boundaries
Tokenizer and Corpus Pipeline - reduced foundation slice

Correctness requirements: supported round trip and deterministic versioned merges/IDs; Failure-injection requirements: malformed encoding, corrupt model and stale tokenizer

Done

Focus on

Match exact IDs as well as decoded text. Two vocabularies can reconstruct the same bytes while indexing different embedding rows.
Persist the ID-to-bytes map, ordered merges, special IDs, regex and normalization/error policies. Encode bytes losslessly in the artifact and hash a canonical serialization.
A transport chunk is not a pre-token. Retain unfinished pre-tokens, whitespace lookahead and special-token prefixes; use an incremental UTF-8 decoder if the input arrives as bytes.
CS336's iterable tests use file lines. Add every split point of short strings and splits inside special literals; the public suite alone does not prove arbitrary chunk invariance.

Reproduce closed-book

From a blank file, load your byte vocabulary and merge ranks, encode and decode a string, then serialize and reload. For a short multilingual string containing spaces and a configured special token, assert that every two-chunk split yields the same IDs as one-shot encoding under the same special-token policy.

Traps

Encoding every incoming chunk independently. A split inside a pre-token or special literal changes IDs even when concatenated decoded text is unchanged.
Serializing vocabulary bytes as decoded Unicode. Partial UTF-8 token bytes can fail decoding or collide under replacement.
Claiming a fixed memory bound while buffering an arbitrarily long unfinished pre-token. State the maximum-document/pre-token policy and never silently truncate it.

Practice

CS336 Assignment 1: tokenizer

Problem (tokenizer), Section 2.6; connect your implementation to get_tokenizer in tests/adapters.py and run tests/test_tokenizer.py

Implement in your own repository. The encode memory test is expected to fail; the iterable memory test is Linux-only. Add reload identity, arbitrary chunk-split and malformed-byte properties beyond the public cases.

Self-test

Why is decode(encode(text)) == text insufficient to establish tokenizer compatibility?
An input chunk ends with '<|endo'. Can you emit its ordinary-text IDs immediately?
A token ends with byte 0xC3 and the next begins with 0xA9. How should the decoder handle them?
Can a decoder with errors='replace' promise a round trip for arbitrary raw bytes?

Completion criterion

Stop when

The public tokenizer suite has no unexpected failures, and your own reload and every-split-point tests preserve exact IDs while a changed tokenizer identity is rejected.

Mark passed, then open 1.5 Only mark this when you can do it without notes.

Leads to

1.5 From documents to training examples: packing and loss masks Build training examples from a stable token stream.
3.5 Checkpoints that really resume Make tokenizer compatibility part of checkpoint loading.

Comes after

1.2 Byte-level BPE: training, merges and tie-breakingTurn the merge list into an encoder and decoder you can trust.
1.3 Tokenizer design choices: vocabulary size, Unigram and special tokensSerialize the complete tokenization policy, not just the vocabulary.

Resources for this unit

LLMs from Scratch companion - tokenization and data loading
Sebastian Raschkaprimaryimplementation
Work through the ch02 main notebook, 05_bpe-from-scratch and 04_bonus_dataloader-intuition. Rebuild byte-level BPE, serialization and packed windows in blank files, then add multilingual, control, whitespace and chunk-boundary properties.
Stanford CS336 Assignment 1 - tokenizer contract
Stanford Universityrequiredassessment
Read the Assignment 1 tokenizer handout and use the named unicode1, unicode2, train_bpe, tokenizer and tokenizer_experiments specifications plus public tests as an acceptance contract. Implement independently in your own repository and do not copy or modify course solutions.
Tokenizer and Corpus Pipeline - reduced foundation slice
Roadmap-Finalsupportproject
Use the byte, identity, split, deduplication, contamination, packing, manifest and restart correctness requirements. Keep the corpus small and stop before distributed processing or internet-scale filtering.

Also in this section

Modern LLM Notebook - tokenization and data experiments
Walking Labssupportimplementation
Run English notebooks 01-tokenizer-basics and 02-bpe-tokenizer. Use only the quality, deduplication and mixture examples from 15-data-engineering. Change the corpus and tie cases, then verify every quantitative claim independently.
Hugging Face LLM Course - tokenizer translation
Hugging Faceoptionalreference
After the blank-file tokenizer passes, use the tokenizer chapters to map vocabulary, special tokens, fast tokenizers and batching into library APIs. Do not replace the deterministic implementation gate with a pretrained tokenizer call.
Python unicodedata - normalization forms
Python Software Foundationreferencereference
Read unicodedata.normalize for canonical and compatibility equivalence, and unidata_version for recording the normalization database version.
Hugging Face Datasets - streaming, sharding and resume
Hugging Facereferencereference
Read Shard, Interleave, Save a dataset checkpoint and resume iteration, and Export. Use tiny local data and inspect the warning that shuffle buffers are lost on resume; do not download web-scale corpora or upload to the Hub.