Unit 1.7 · about 5 h
296 / 324Filtering, mixtures and deterministic shards
Filter and mixture choices determine which text the model repeatedly sees. A reproducible recipe also needs stable shard contents and restart state, not just a random seed.
- explain
- implement
You can now
- Implement versioned filter decisions and quantify retained documents, bytes and tokens by source and representative slice.
- Distinguish document sampling probabilities from token exposure and defend a small source mixture.
- Publish deterministic, checksummed local shards and resume without omitted or duplicated records under a fixed configuration.
- Use synthetic examples to distinguish PII redaction, toxicity review and quality filtering without claiming production safety.
Covers
- Filtering reasons, yield and representative slices
- Mixture weights, sampling units and token exposure
- Deterministic sharding, manifests and restart behavior
- Small PII, toxicity and quality-filter policy exercise
Study, in this order
3 steps
- Filter decisions and source mixturesModern LLM Notebook - tokenization and data experiments
Section 2.1 with
quality_check; Section 2.3, PII and Safety Filtering, withredact_or_drop; Sections 4.1 and 4.2 with source effective-token amounts and Recipe A/B; Exercises 1 and 3Depth · Skip Section 2.2 and Sections 4.3 onward. The rules and recipe scores are teaching examples, not validated quality thresholds or measured model results. Use only synthetic sensitive-data examples.
- Shard ownership, mixture sampling and iterator stateHugging Face Datasets - streaming, sharding and resume
Shard under Split dataset; Interleave, including probabilities and
stopping_strategy; Save a dataset checkpoint and resume iteration, including its shuffle-buffer warning; ExportDepth · Use tiny local records and the six-record checkpoint example. Skip remote corpus downloads, the training loop and Save to the Hub. Library resume is a comparison, not a substitute for testing your own exact-resume contract.
- Auditable recipes and local shard publicationTokenizer and Corpus Pipeline - reduced foundation slice
Core requirements: mixture weights, deterministic sample/pack construction, checksums, atomic manifests and resume; Architecture constraints: manifests instead of directory discovery; Milestone B: per-stage counts and filter samples; Correctness requirements: complete nonoverlapping shards
Depth · Keep one process and a few local shards. Leave distributed storage, production classifiers and internet-scale processing for later.
Focus on
- Log
keep,redact,rejectorreviewwith a reason and policy version. Report yields by source/language/code slice and by documents, bytes and tokens; a smaller corpus is not automatically better. - For source document probability \(p_s\) and mean retained length \(L_s\), expected token share is \(p_s L_s / \sum_j p_j L_j\), before truncation and exhaustion effects. Declare the sampling unit and replacement policy.
- A manifest fixes ordered shard paths, checksums, counts, split, tokenizer identity and recipe version. Publish only complete shards and reject resume under a changed configuration.
- Exact resume needs input/shard cursor, sampling RNG, any shuffle buffer and the unfinished pack. HF's documented iterable resume loses buffered shuffle examples; a seed alone does not restore them.
Reproduce closed-book
Traps
- Applying English word-length and punctuation rules to code or languages without spaces. Inspect false rejections by slice before choosing thresholds.
- Treating email regex redaction or a toxic-word hit as a safety guarantee. Names, context and obfuscation can escape it; benign quotations can be flagged.
- Rebuilding shard order from a directory listing or restarting from only a seed. Either can change the next record, duplicate work or lose a partially built pack.
Practice
The section mini-project covers this: reason-coded filters on representative slices, a synthetic PII/toxicity/quality policy table, source token-exposure accounting, deterministic shard manifests and interruption/resume equivalence.
Self-test
-
Sources A and B are sampled equally by document, with mean lengths 100 and 900 tokens. What token share is expected?
About 10% A and 90% B before exhaustion or truncation. Equal document probabilities do not imply equal token exposure. -
What should you report if a filter retains 80% of documents but only 30% of tokens?
Both yields and per-slice results. The filter disproportionately removed long documents; neither number establishes a quality improvement. -
Why does HF's iterable state checkpoint not guarantee exact shuffled continuation?
The documented shuffle buffer is lost on resume and refilled. The saved shard/example position alone does not restore the buffered records or their future order. -
A crash leaves a shard file but no manifest entry. Should the reader consume it?
No. Read only completed, validated artifacts declared in the manifest. Recovery must verify or rebuild unpublished work before publishing it. -
What does a synthetic toxicity-policy test demonstrate?
That declared keep/reject/review behavior is implemented on those examples. It does not demonstrate calibrated detection, generalization or production safety.
Completion criterion
Stop when
An interrupted and uninterrupted small run produce identical ordered IDs and artifact checksums, changed-config resume is rejected, and your audit explains filter yields and observed document versus token shares.
Mark passed, then open 2.1 Only mark this when you can do it without notes.Leads to
- 3.2 Feeding the model: batches, initialization and first steps Feed batches from a versioned token corpus.
- 3.5 Checkpoints that really resume Resume the data stream alongside optimizer and model state.
Comes after
- 1.6 Provenance, deduplication and contaminationCarry source and cluster identities into sampling and manifests.
Resources for this unit
- Modern LLM Notebook - tokenization and data experiments Run English notebooks 01-tokenizer-basics and 02-bpe-tokenizer. Use only the quality, deduplication and mixture examples from 15-data-engineering. Change the corpus and tie cases, then verify every quantitative claim independently.
- Tokenizer and Corpus Pipeline - reduced foundation slice Use the byte, identity, split, deduplication, contamination, packing, manifest and restart correctness requirements. Keep the corpus small and stop before distributed processing or internet-scale filtering.
- Hugging Face Datasets - streaming, sharding and resume Read Shard, Interleave, Save a dataset checkpoint and resume iteration, and Export. Use tiny local data and inspect the warning that shuffle buffers are lost on resume; do not download web-scale corpora or upload to the Hub.
Also in this section
- LLMs from Scratch companion - tokenization and data loading Work through the ch02 main notebook, 05_bpe-from-scratch and 04_bonus_dataloader-intuition. Rebuild byte-level BPE, serialization and packed windows in blank files, then add multilingual, control, whitespace and chunk-boundary properties.
- Stanford CS336 Assignment 1 - tokenizer contract Read the Assignment 1 tokenizer handout and use the named unicode1, unicode2, train_bpe, tokenizer and tokenizer_experiments specifications plus public tests as an acceptance contract. Implement independently in your own repository and do not copy or modify course solutions.
- Hugging Face LLM Course - tokenizer translation After the blank-file tokenizer passes, use the tokenizer chapters to map vocabulary, special tokens, fast tokenizers and batching into library APIs. Do not replace the deterministic implementation gate with a pretrained tokenizer call.
- Python unicodedata - normalization forms Read
unicodedata.normalizefor canonical and compatibility equivalence, andunidata_versionfor recording the normalization database version.