Section 1 of 5 · LLM Foundations
Tokenization, Corpus Identity & Packed Data
The model sees token identities and loss masks rather than raw text. This section makes every transformation from bytes to training examples deterministic, reversible where promised and auditable across splits.
7 units 23 topics 0/7
Units
7 in this section
- 1.1 Text as bytes: Unicode, encodings and normalization explain3 h
- 1.2 Byte-level BPE: training, merges and tie-breaking derive · implement6 h
- 1.3 Tokenizer design choices: vocabulary size, Unigram and special tokens explain · derive3 h
- 1.4 A tokenizer you can trust: round trips, identity and streaming implement6 h
- 1.5 From documents to training examples: packing and loss masks derive · implement4 h
- 1.6 Provenance, deduplication and contamination implement4 h
- 1.7 Filtering, mixtures and deterministic shards explain · implement5 h
Before moving on
Checkpoint
Implement and serialize deterministic byte-level BPE, pass adversarial round-trip, merge-tie, special-token and chunk-boundary tests, then publish a small packed split with provenance and an automated contamination check.
Deliberately out of scope
Leave for later
- Internet-scale collection, distributed sharding and data lakes
- Production PII, toxicity and quality classifiers
- Full Unigram training and exhaustive tokenizer comparison
Resources for this section
Shared by every unit above
- LLMs from Scratch companion - tokenization and data loading Work through the ch02 main notebook, 05_bpe-from-scratch and 04_bonus_dataloader-intuition. Rebuild byte-level BPE, serialization and packed windows in blank files, then add multilingual, control, whitespace and chunk-boundary properties.
- Stanford CS336 Assignment 1 - tokenizer contract Read the Assignment 1 tokenizer handout and use the named unicode1, unicode2, train_bpe, tokenizer and tokenizer_experiments specifications plus public tests as an acceptance contract. Implement independently in your own repository and do not copy or modify course solutions.
- Modern LLM Notebook - tokenization and data experiments Run English notebooks 01-tokenizer-basics and 02-bpe-tokenizer. Use only the quality, deduplication and mixture examples from 15-data-engineering. Change the corpus and tie cases, then verify every quantitative claim independently.
- Tokenizer and Corpus Pipeline - reduced foundation slice Use the byte, identity, split, deduplication, contamination, packing, manifest and restart correctness requirements. Keep the corpus small and stop before distributed processing or internet-scale filtering.
- Hugging Face LLM Course - tokenizer translation After the blank-file tokenizer passes, use the tokenizer chapters to map vocabulary, special tokens, fast tokenizers and batching into library APIs. Do not replace the deterministic implementation gate with a pretrained tokenizer call.
- Python unicodedata - normalization forms Read
unicodedata.normalizefor canonical and compatibility equivalence, andunidata_versionfor recording the normalization database version. - Hugging Face Datasets - streaming, sharding and resume Read Shard, Interleave, Save a dataset checkpoint and resume iteration, and Export. Use tiny local data and inspect the warning that shuffle buffers are lost on resume; do not download web-scale corpora or upload to the Hub.
Apply the section
Mini-project · Tokenizer and corpus audit pack (12 to 16 h)
Turn a small public-domain corpus into a packed, auditable training split with your own byte-level BPE. Every packed token traces back to a source document, and every claim about the split, such as no duplicates across splits and no planted evaluation text in training, is backed by a test or a report you can rerun.
Steps
- Start one repository for the whole stage: a uv project with a
srcpackage and atestsfolder. Pick a small public-domain corpus under 20 MB, such as a few Project Gutenberg books or the TinyStories validation file. - Implement BPE training, encoding, decoding and streaming encoding. Connect them to
tests/adapters.pyin your clone of the CS336 assignment and passtests/test_train_bpe.pyandtests/test_tokenizer.py. - Serialize the tokenizer canonically with a content hash, and make loading fail on a changed merge list, special-token set or normalization policy.
- Assign raw documents to train and evaluation splits by a stable hash of their raw ID, before fitting the tokenizer or packing. Record source, raw hash, normalized hash and transform version for each document.
- Remove exact and near duplicates while keeping cluster lineage, and simulate a hash collision to show that your byte comparison catches it. Plant three evaluation passages in training documents, make your contamination probe find them, and review its false matches.
- Apply a reason-coded filter table with quality rules and synthetic PII and toxicity examples, and report document, byte and token yields per source.
- Pack the training split with EOS boundaries, a loss mask and per-token document IDs into ordered checksummed shards with a manifest, showing an interrupted run produces byte-identical shards. Check both masks against small hand-worked examples.
Deliverables
- The tokenizer package with its serialized artifact, identity hash and passing CS336 tokenizer tests.
- The packed training split: shards, manifest, per-token document IDs and loss masks.
- An audit report: split assignment, duplicate clusters, contamination hits with reviewed false matches, filter yields and per-source token shares.
- A resume test that interrupts packing mid-shard and compares checksums with an uninterrupted run.
Checks
- external-tests · CS336 `tests/test_train_bpe.py` and `tests/test_tokenizer.py`
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Tokenizer correctness | Both CS336 tokenizer test files pass with no unexpected failures. | Your own property tests also cover every two-chunk split of multilingual strings and partial special-token prefixes, and a reloaded artifact gives identical IDs. |
| Tokenizer identity | Loading rejects a changed merge list or special-token set. | The hash covers a canonical serialization of vocabulary bytes, merges, special tokens, regex and normalization policy, and the error names what changed. |
| Splits and lineage | Splits are assigned before tokenizer fitting and packing, and every packed token maps to a document ID. | Duplicate clusters that span splits are quarantined or reassigned, and the report counts them. |
| Packing and masks | Your mask tests pass on a hand-worked sequence that packs at least two documents. | You explain which predictions the attention mask alone would not remove, and a deliberately wrong mask makes your tests fail. |
| Deduplication and contamination | The probe finds every planted passage and the report lists its hits. | The report reviews false matches, states what the probe cannot detect, such as paraphrase or translation, and simulates a hash collision. |
| Reproducibility | Rerunning the pipeline gives the same manifest checksums. | An interrupted run resumed from its saved cursor, RNG and partial pack gives byte-identical shards, and a changed config is rejected on resume. |
Feeds into: Capstone artifact 1, the tokenizer and corpus audit pack. Sections 3 and 4 train and evaluate on this split, and every token count in the capstone uses this tokenizer.