Unit 4.5 · about 3 h
315 / 324Contamination and memorization
Training on the Internet means the test set may already be in the weights. Without overlap checks, a benchmark measures recall, not generalization.
- implement
You can now
- Implement exact and approximate contamination checks between train and evaluation texts and produce a clean subset.
- Explain memorization, duplicate inflation and why a contaminated score cannot support a capability claim.
- State the contamination boundary in the release gate with fresh or private evals where needed.
Covers
- Exact and approximate contamination checks
- Memorization, duplicates and unsupported capability claims
Study, in this order
4 steps
- CS336 Lecture 12: validity and train-test overlapStanford CS336 - resource accounting, inference and evaluation
Function
validity()with train-test overlap routes (exchangeability paper 2310.17623, reporting norms 2410.08385 with confidence intervals, fresh evals LiveCodeBench and UncheatableEval with unsafe timestamps, private evals with perplexity as the easiest private check) plus dataset quality (SWE-Bench Verified, Platinum, agentic test gaps with Docent) - Modern LLM Notebook 25-evaluation: contamination pitfallsModern LLM Notebook - evaluation and FLOP accounting
Section 10.1 (contamination as critical with exact, n-gram and embedding matching,
check_integritylimits, stale HumanEval toward HumanEval+ and LiveCodeBench, small-data variance) - lm-eval-harness: decontamination guideLanguage Model Evaluation Harness
13-gram overlap rule from Few-Shot Learners Appendix C with \(N\) 8 to 13,
decontaminate.pyn-gram dicts plus training scan, clean benchmark withdecontaminatesuffix, disabled by default - lm-eval-harness: task decontamination fieldsLanguage Model Evaluation Harness
should_decontaminateanddoc_to_decontamination_queryfields for enabling task decontamination
Focus on
- Run both exact and approximate (n-gram or embedding) checks. Exact catches copies, n-grams catch near copies.
check_integritychecks task and data plumbing, not training overlap. Do not cite it as decontamination.- Duplicates and memorized benchmarks inflate scores without new capability. Prefer fresh or private evals for the headline claim.
- Report full versus clean side by side. A large drop is evidence of leakage, not a worse model.
Reproduce closed-book
decontamination.md: build eval n-grams, scan training n-grams, mark contaminated docs, and report the metric with a decontaminate suffix on the clean subset.Traps
- Equating deduplication with decontamination. Dedup cleans training, while decontamination checks train versus eval overlap.
- Trusting timestamps for freshness. Copied text carries old dates into new pages, so scraped evals still need overlap checks.
- Claiming reasoning from a memorized benchmark. Unique tasks or fresh problems test generalization, while leaked sets test recall.
Practice
The section mini-project covers this: exact and approximate contamination checks between the tiny train corpus and the eval set, with full versus clean reporting.
Self-test
-
Which \(n\) does the harness use for decontamination, and what marks a doc as contaminated?
13. OpenAI used 8 to 13 by dataset, and the harness fixes 13 for simplicity. A doc is contaminated if any 13-gram overlaps a training doc. -
What does
check_integritydo, and what does it not do?It checks task and data integrity such as config and loading, not train-eval overlap. It cannot replace decontamination. -
HumanEval is stale and widely memorized. What should you do?
Move to HumanEval+ or LiveCodeBench, run overlap checks, and cross-validate with more sets, because 164 items carry high variance on their own. -
Full-set accuracy is 85 percent but clean-subset accuracy is 71 percent. What should you claim?
Report both, headline the clean number, and present the drop as leakage evidence. Never claim broad capability from the full score alone.
Completion criterion
Stop when
Your report shows full versus clean scores from an explicit n-gram check, and every capability claim names the evals excluded as contaminated.
Mark passed, then open 4.6 Only mark this when you can do it without notes.Leads to
- 4.4 Uncertainty, robustness and honest claims Report decontaminated deltas with intervals instead of a single full-set mean.
- 1.6 Provenance, deduplication and contamination Trace the overlap back to pipeline lineage and split discipline.
Comes after
- 1.6 Provenance, deduplication and contaminationSeparate observed overlap from memorization and capability claims.
- 4.4 Uncertainty, robustness and honest claimsAdd the contamination boundary to the release gate.
Resources for this unit
- Stanford CS336 - resource accounting, inference and evaluation Use Spring 2026 Lectures 2, 10 and 12 for FLOPs, memory, arithmetic intensity, inference and evaluation. Reproduce the small calculations locally. Assignment 3 IsoFLOPs analysis is optional; do not require its hosted training API or costly experiments.
- Modern LLM Notebook - evaluation and FLOP accounting Use English notebook 25-evaluation and Appendix B FLOP and memory accounting. Build a 20-example evaluator with two slices, raw outputs and bootstrap intervals, and deliberately create one parser, prompt or judge failure.
- Language Model Evaluation Harness After the small evaluator passes, inspect task versioning, request construction and result schemas to understand production evaluation plumbing. Do not use the harness as a substitute for owning the task and metric contract.
Also in this section
- LLMs from Scratch companion - GPT FLOP analysis Use ch04/02_performance-analysis/flops-analysis.ipynb as a concrete calculator, then rebuild the formulas from the model configuration and reconcile parameters and at least one memory or operation estimate with runtime state.
- LLM Foundations Checkpoint - evaluation and arithmetic slice Use the contamination, evaluator-bias, parameter, FLOP, activation, state and KV requirements as the assessment contract. Keep raw outputs and distinguish a verified task result from a broad model capability claim.
- Stanford CS336 Assignment 3 - IsoFLOPs scaling Use Section 2.1 and Problem
chinchilla_isoflopswithdata/isoflops_curves.jsonfor the bounded curve-fitting exercise only. Do not use the hosted training API or costly experiments.