Unit 4.1 · about 3 h
311 / 324Likelihood and perplexity, and their limits
Perplexity guides training and scaling because it is smooth, but it scores every token including ones the task does not care about. A lower number on different tokenizers or splits is not a better model.
- derive
You can now
- Derive token-weighted perplexity with an explicit valid-token mask and state the conditional form for prompt-conditioned scoring.
- Explain why perplexity comparisons need the same tokenizer, data split and conditioning before any ranking claim.
- Read word, byte and bits-per-byte variants from an evaluation config and say which one removes tokenizer effects.
Covers
- Conditional likelihood and token-weighted perplexity
- Tokenizer dependence and limits of perplexity comparison
Study, in this order
4 steps
- CS336 Lecture 12: `perplexity()`Stanford CS336 - resource accounting, inference and evaluation
Function
perplexity(): \(p(x)\) over token sequences, perplexity \(\left(1/p(D)\right)^{1/|D|}\), PTB, WikiText-103 and One Billion Word splits, GPT-2 WebText 40GB zero-shot on small versus 1BW sets, conditional perplexity \(p(\text{response} \mid \text{prompt})^{1/|\text{response}|}\), the trust warning for perplexity leaderboards, and LAMBADA plus HellaSwag as perplexity in disguise - Modern LLM Notebook 25-evaluation: perplexity and metric choiceModern LLM Notebook - evaluation and FLOP accounting
Intro (perplexity suits language modeling and pretraining monitoring, not chat, reasoning or safety) and Section 9.1 (perplexity row: sum of logprobs, ppl and bpb on WikiText and Lambada)
- lm-eval-harness: wikitext perplexity configLanguage Model Evaluation Harness
wikitext.yaml:loglikelihood_rollingwithword_perplexity,byte_perplexityandbits_per_byte(version 2.0) - lm-eval-harness: perplexity aggregations in `metrics.py`Language Model Evaluation Harness
perplexityas \(\exp(-\text{mean})\),weighted_perplexityas \(\exp(-\text{weighted\_mean})\) andbits_per_byteas \(-\text{weighted\_mean}/\log(2)\)
Focus on
- The mean runs over scored tokens only. Prompt tokens leave both the sum and the count in the conditional form.
- Perplexity is \(\exp\) of the mean negative log likelihood, so it reads as the effective shortlist size at each step.
- A new tokenizer changes the token count for the same text, so the means run over different units. Bits per byte divides by bytes to remove that effect.
- Perplexity is smooth and universal for scaling, but it penalizes tokens the task ignores. Task accuracy still needs its own contract.
Reproduce closed-book
Traps
- Comparing perplexity across tokenizers or splits. Different vocabularies split the same text into different token counts, so the ranking is meaningless.
- Reading a low perplexity as chat quality, reasoning or safety. The lecture and notebook both limit it to language modeling and pretraining monitoring.
- Trusting submitted perplexity numbers without checking that probs sum to one. Accuracy needs only output text, but perplexity needs valid probabilities.
Practice
Modern LLM Notebook 25-evaluation Exercise 1: Perplexity by Hand
Exercises: Exercise 1 with output probs [0.5, 0.3, 0.2], [0.1, 0.7, 0.2], [0.4, 0.1, 0.5]; take the log prob at each step, average, then exponentiate
Take the stated correct-token probs as given and compare your hand result with code. Then rescore the same text after a tokenizer change to see the denominator move.
Self-test
-
Correct-token probs are 0.5, 0.7 and 0.5. What is the perplexity?
About 1.79. The mean negative log likelihood is \(-(\ln 0.5 + \ln 0.7 + \ln 0.5)/3 = 0.58\), and \(\exp(0.58) = 1.79\). -
Two tokenizers score the same text with mean NLL 2.0 over 100 tokens and 1.8 over 150 tokens. Which model is better?
The first, if both scored the same text. Per-token means use different denominators, so compare totals instead: \(2.0 \times 100 = 200\) nats versus \(1.8 \times 150 = 270\) nats. The first model gives the text higher probability even though its per-token perplexity looks worse; bits per byte makes the same comparison. -
Why does the lecture warn that a perplexity leaderboard must trust the model?
Perplexity computes from the model's own probabilities, so they must sum to one. A task accuracy check only reads generated text and does not need that trust. -
When should you report conditional perplexity instead of full perplexity?
When a prompt is given and only the response matters. Score \(p(\text{response} \mid \text{prompt})\) normalized by response length, so prompt tokens do not dilute the number.
Completion criterion
Stop when
You hand-compute 1.79 from the three probs, and you can say for any reported perplexity which tokenizer, split and conditioning it used.
Mark passed, then open 4.2 Only mark this when you can do it without notes.Leads to
- 4.2 Task contracts and automatic metrics Carry perplexity limits into explicit task contracts and metric choices.
- 4.4 Uncertainty, robustness and honest claims Treat perplexity gaps with slices and intervals instead of reading them as wins.
Comes after
- 3.7 Entropy, cross-entropy, KL divergence and mutual informationPerplexity is exponentiated cross-entropy on held-out text.
- 5.7 Stable log-sum-exp, online reductions and mixed precisionUse stable log probabilities and controlled reductions when aggregating likelihoods.
- 3.1 The next-token objective: shifts, masks and perplexityCarry perplexity into evaluation and learn when it misleads.
Resources for this unit
- Stanford CS336 - resource accounting, inference and evaluation Use Spring 2026 Lectures 2, 10 and 12 for FLOPs, memory, arithmetic intensity, inference and evaluation. Reproduce the small calculations locally. Assignment 3 IsoFLOPs analysis is optional; do not require its hosted training API or costly experiments.
- Modern LLM Notebook - evaluation and FLOP accounting Use English notebook 25-evaluation and Appendix B FLOP and memory accounting. Build a 20-example evaluator with two slices, raw outputs and bootstrap intervals, and deliberately create one parser, prompt or judge failure.
- Language Model Evaluation Harness After the small evaluator passes, inspect task versioning, request construction and result schemas to understand production evaluation plumbing. Do not use the harness as a substitute for owning the task and metric contract.
Also in this section
- LLMs from Scratch companion - GPT FLOP analysis Use ch04/02_performance-analysis/flops-analysis.ipynb as a concrete calculator, then rebuild the formulas from the model configuration and reconcile parameters and at least one memory or operation estimate with runtime state.
- LLM Foundations Checkpoint - evaluation and arithmetic slice Use the contamination, evaluator-bias, parameter, FLOP, activation, state and KV requirements as the assessment contract. Keep raw outputs and distinguish a verified task result from a broad model capability claim.
- Stanford CS336 Assignment 3 - IsoFLOPs scaling Use Section 2.1 and Problem
chinchilla_isoflopswithdata/isoflops_curves.jsonfor the bounded curve-fitting exercise only. Do not use the hosted training API or costly experiments.