Unit 1.6 · about 4 h
295 / 324Provenance, deduplication and contamination
Distinct files can contain the same document, and distinct documents can share an evaluation answer. Deduplication and contamination checks need traceable identities, not just a smaller output corpus.
- implement
You can now
- Implement stable raw document IDs and retain source, transformation versions and duplicate-cluster lineage.
- Remove exact duplicates under a declared byte policy and implement a small shingle-based near-duplicate comparison.
- Probe train/evaluation overlap, review false matches and report what the detector cannot establish.
Covers
- Source and document identities, lineage and provenance
- Exact deduplication and near-duplicate strategy
- Contamination probes and false-match review
Study, in this order
2 steps
- Exact deduplication, shingles and overlap probesModern LLM Notebook - tokenization and data experiments
Sections 3.1 and 3.2 with Exact Dedup and MinHash approximate deduplication code; Section 3.3, Decontamination, with
word_ngramsandmax_overlap; Exercises, Exercise 2: Complete Jaccard similarityDepth · Use exact Jaccard as the small-corpus oracle. Treat the four-position MinHash as a noisy demonstration and the stated corpus-removal percentages as unverified examples, not expected results.
- Provenance and duplicate-cluster contractTokenizer and Corpus Pipeline - reduced foundation slice
Core requirements: versioned schemas, source/document identities, exact/near dedup and contamination probes; Architecture constraints: raw source identity and separate versioned stages; Correctness requirements: cluster lineage; Failure-injection requirements: hash collision simulation and duplicate clusters spanning sources/splits
Depth · Implement a tiny exact pipeline and all-pairs shingle comparison. Leave large-scale approximate search and distributed processing for later.
Focus on
- Keep source ID, stable raw record ID, raw-byte hash, normalized-content hash and transform version separately. A normalization collision is not evidence that two raw records were identical.
- Use the full digest and verify bytes within a candidate hash bucket. Choose a deterministic representative and retain every removed member's source and raw identity.
- Define shingles, Jaccard threshold and empty-set behavior. Similarity is not transitive: connected-component clustering can join two endpoints below the pairwise threshold.
- The notebook's contamination score is shared evaluation shingles divided by evaluation shingles, not Jaccard. Common boilerplate can trigger it, while paraphrases and non-Latin text can escape its ASCII-word regex.
Reproduce closed-book
Traps
- Replacing a raw ID with a normalized-text hash. Different sources or byte spellings can collapse, and the audit loses their origins.
- Hashing and splitting each record independently before handling duplicates. A duplicate cluster can span train and evaluation; fix group ownership or quarantine cross-split matches before packing.
- Calling a no-hit report proof of no contamination. Exact and shingle probes miss transformed content, and short common matches need review.
Practice
The section mini-project covers this: raw and transformed document identities, exact and near-duplicate lineage, planted train/evaluation overlap, collision simulation and false-match review.
Self-test
-
Two records have the same NFC text but different raw bytes. Which identities should survive?
Both raw record IDs and raw hashes survive in lineage. They may share a normalized-content hash and cluster only if the declared dedup policy permits it. -
Shingle sets have three members each and share two. What is their Jaccard score?
\(2/(3+3-2) = 0.5\). The denominator is the union, not the size of the smaller set. -
A four-word evaluation phrase appears in a long training article. Why can containment be 1 while Jaccard is small?
Every evaluation shingle is present, so the containment denominator is fully covered. Jaccard also counts the many unrelated training shingles in its union. -
A-B and B-C exceed the near-duplicate threshold. Must A-C exceed it?
No. Threshold similarity is not transitive. A connected-component policy can group A and C anyway, so document that choice and review its removals.
Completion criterion
Stop when
Your small pipeline retains all removed-record lineage, detects a planted overlap and survives a simulated hash collision, with reviewed false matches and explicit limits on any no-hit claim.
Mark passed, then open 1.7 Only mark this when you can do it without notes.Leads to
- 1.7 Filtering, mixtures and deterministic shards Carry source and cluster identities into sampling and manifests.
- 4.5 Contamination and memorization Separate observed overlap from memorization and capability claims.
Comes after
- 1.1 Text as bytes: Unicode, encodings and normalizationKeep raw identity separate from normalized identity.
- 4.5 Contamination and memorizationTrace the overlap back to pipeline lineage and split discipline.
Resources for this unit
- Modern LLM Notebook - tokenization and data experiments Run English notebooks 01-tokenizer-basics and 02-bpe-tokenizer. Use only the quality, deduplication and mixture examples from 15-data-engineering. Change the corpus and tie cases, then verify every quantitative claim independently.
- Tokenizer and Corpus Pipeline - reduced foundation slice Use the byte, identity, split, deduplication, contamination, packing, manifest and restart correctness requirements. Keep the corpus small and stop before distributed processing or internet-scale filtering.
Also in this section
- LLMs from Scratch companion - tokenization and data loading Work through the ch02 main notebook, 05_bpe-from-scratch and 04_bonus_dataloader-intuition. Rebuild byte-level BPE, serialization and packed windows in blank files, then add multilingual, control, whitespace and chunk-boundary properties.
- Stanford CS336 Assignment 1 - tokenizer contract Read the Assignment 1 tokenizer handout and use the named unicode1, unicode2, train_bpe, tokenizer and tokenizer_experiments specifications plus public tests as an acceptance contract. Implement independently in your own repository and do not copy or modify course solutions.
- Hugging Face LLM Course - tokenizer translation After the blank-file tokenizer passes, use the tokenizer chapters to map vocabulary, special tokens, fast tokenizers and batching into library APIs. Do not replace the deterministic implementation gate with a pretrained tokenizer call.
- Python unicodedata - normalization forms Read
unicodedata.normalizefor canonical and compatibility equivalence, andunidata_versionfor recording the normalization database version. - Hugging Face Datasets - streaming, sharding and resume Read Shard, Interleave, Save a dataset checkpoint and resume iteration, and Export. Use tiny local data and inspect the warning that shuffle buffers are lost on resume; do not download web-scale corpora or upload to the Hub.