Section 4 of 5 · LLM Foundations
Evaluation, Claims & Model Arithmetic
A score is useful only when the task, parser, sampling policy, uncertainty and contamination boundary are defensible. Cost estimates must likewise state shapes, units and omitted overhead before they guide a design.
7 units 24 topics 0/7
Units
7 in this section
- 4.1 Likelihood and perplexity, and their limits derive3 h
- 4.2 Task contracts and automatic metrics implement4 h
- 4.3 Model judges and human review explain · measure3 h
- 4.4 Uncertainty, robustness and honest claims implement · measure4 h
- 4.5 Contamination and memorization implement3 h
- 4.6 Counting parameters and FLOPs derive4 h
- 4.7 Memory, arithmetic intensity and reconciling estimates derive · measure4 h
Before moving on
Checkpoint
Produce a versioned evaluation with likelihood, one exact or structured metric, two slices, raw outputs, bootstrap uncertainty and contamination checks. Expose one evaluator failure and reconcile parameter, FLOP, activation and optimizer-memory estimates with the tiny model.
Deliberately out of scope
Leave for later
- Leaderboard claims and broad capability conclusions
- Large scaling-law sweeps and hosted-compute experiments
- Full safety, alignment and model-judge research
Resources for this section
Shared by every unit above
- Stanford CS336 - resource accounting, inference and evaluation Use Spring 2026 Lectures 2, 10 and 12 for FLOPs, memory, arithmetic intensity, inference and evaluation. Reproduce the small calculations locally. Assignment 3 IsoFLOPs analysis is optional; do not require its hosted training API or costly experiments.
- LLMs from Scratch companion - GPT FLOP analysis Use ch04/02_performance-analysis/flops-analysis.ipynb as a concrete calculator, then rebuild the formulas from the model configuration and reconcile parameters and at least one memory or operation estimate with runtime state.
- Modern LLM Notebook - evaluation and FLOP accounting Use English notebook 25-evaluation and Appendix B FLOP and memory accounting. Build a 20-example evaluator with two slices, raw outputs and bootstrap intervals, and deliberately create one parser, prompt or judge failure.
- LLM Foundations Checkpoint - evaluation and arithmetic slice Use the contamination, evaluator-bias, parameter, FLOP, activation, state and KV requirements as the assessment contract. Keep raw outputs and distinguish a verified task result from a broad model capability claim.
- Language Model Evaluation Harness After the small evaluator passes, inspect task versioning, request construction and result schemas to understand production evaluation plumbing. Do not use the harness as a substitute for owning the task and metric contract.
- Stanford CS336 Assignment 3 - IsoFLOPs scaling Use Section 2.1 and Problem
chinchilla_isoflopswithdata/isoflops_curves.jsonfor the bounded curve-fitting exercise only. Do not use the hosted training API or costly experiments.
Apply the section
Mini-project · Evaluation and arithmetic dossier (10 to 14 h)
Evaluate the trained model honestly and account for its cost. The evaluation is versioned and rerunnable, every number carries uncertainty, one evaluator failure is exposed on purpose, and every parameter, FLOP and memory estimate is reconciled with a measurement.
Steps
- Report validation likelihood and perplexity with the tokenizer and split named, and explain why the number does not compare across tokenizers.
- Define one small task with at least 20 examples in each of two slices, a versioned prompt, parser and exact or structured metric. Run it on your model and store raw outputs next to the scores.
- Report the metric per slice with bootstrap intervals, compare two checkpoints or decoding settings with a paired bootstrap, and report the range across two prompt versions.
- Expose one evaluator failure on purpose, such as a parser that accepts a wrong format or a judge that prefers longer answers, and show the fix.
- Run exact and n-gram contamination checks between the evaluation set and your training split, report full and clean scores side by side, and state what the checks cannot rule out.
- Write a worksheet of parameters, forward FLOPs, activation memory and AdamW memory for your model, and reconcile each with a measurement:
FlopCounterMode, tensor sizes or peak memory. - Test the arithmetic and statistics code: parameter and matmul-FLOP counts equal exact counts from a reference module, a fixed seed gives identical bootstrap intervals, and on simulated data with a known mean roughly 95 percent of your 95 percent intervals cover it.
Deliverables
- An evaluation manifest: data version, prompt, parser, metric, seeds and decoding config.
- Raw outputs, slice scores with bootstrap intervals and the paired comparison.
- The evaluator-failure case with results before and after the fix.
- The arithmetic worksheet, with each estimate next to its measurement and the reason for any gap.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Versioning | Rerunning the manifest reproduces the scores. | A changed prompt, parser or tokenizer changes the manifest hash, and the report says which changed. |
| Uncertainty | Every headline number has a bootstrap interval. | The comparison uses a paired bootstrap, and the report states which claim the interval supports and which it does not. |
| Evaluator failure | One failure is shown with its fix. | You found the failure by reading raw outputs, and a regression test now covers it. |
| Contamination | The probe runs between evaluation and training data and its result is reported. | The report separates overlap from memorization and lists what the probe cannot detect. |
| Arithmetic | Your counting and bootstrap tests pass, and every estimate is within 10 percent of its measurement. | Every gap above 2 percent is explained, for example by elementwise ops, buffers or allocator overhead. |
Feeds into: Capstone artifact 3, the evaluation and arithmetic half of the dossier. Section 5 extends the memory worksheet with the KV cache.