Unit 4.2 · about 4 h
312 / 324Task contracts and automatic metrics
Most evaluation gains come from parsing, prompts or versions, not from the model. A contract plus raw outputs turns a score into evidence that survives review.
- implement
You can now
- Define a task contract with prompt version, few-shot count, output schema and parsing rules before running any model.
- Implement exact and normalized match plus a structured verifier, and show a counterexample for each automatic metric.
- Retain raw outputs and version the task, prompt, parsing and model so another run reproduces the score.
Covers
- Task contract, prompt version and output schema
- Exact match, normalized match and structured verifiers
- Generation metrics and their counterexamples
- Raw-output retention and evaluator versioning
Study, in this order
5 steps
- Modern LLM Notebook 25-evaluation: metric system and pitfallsModern LLM Notebook - evaluation and FLOP accounting
Section 9.1 (
loglikelihoodaccandacc_normfor MCQ,generate_untilexact_match,pass@kandF1for generation,strict_accfor IFEval) and Sections 10.1 to 10.2 (prompt sensitivity 5 to 15 percent with OLMES prompts, few-shot 0, 5 and 8,seed,temperature0, Chat versus Completions API confusion) - lm-eval-harness: GSM8K task contractLanguage Model Evaluation Harness
doc_to_textanddoc_to_target, metricexact_matchwith strict-match regex#### (number)versus flexible-extract last-number regex,generation_kwargsuntil Questionwithdo_sample falseandtemperature 0.0,num_fewshot 5, metadata version 3.0 - lm-eval-harness: task configuration guideLanguage Model Evaluation Harness
Task naming,
doc_to_text,doc_to_target,doc_to_choice,num_fewshot,output_typegenerate_until,loglikelihoodandmultiple_choice,filter_listpipelines, metadata version - lm-eval-harness: exact match and accuracy metricsLanguage Model Evaluation Harness
exact_match_hf_evaluatewithregexes_to_ignore,ignore_caseandignore_punctuationplus mean aggregation;accandacc_normentries forloglikelihoodandmultiple_choice - CS336 Lecture 12: exam benchmarks and rules of the gameStanford CS336 - resource accounting, inference and evaluation
Function
exam_benchmarks()with MMLU 57 subjects and few-shot prompting, MMLU-Pro 4 to 10 choices with 16 to 33 percent drops, IFEval synthetic constraints that check format not semantics; plushow_to_think_about_evaluation()with methods versus models and the nanogpt speedrun and DataComp-LM rules of the game
Focus on
- The task is the prompt plus few-shot plus parsing plus version, not just the dataset name. Change any part and the score means something new.
- Strict match needs the
####marker, flexible extract takes the last number. The same outputs can pass one and fail the other. accpicks the highest-prob choice,acc_normcorrects for length. IFEvalstrict_accchecks format and keywords, not whether the answer is right.- Keep raw generations and the config hash. Reruns compare filtered outputs, not just the mean.
Reproduce closed-book
#### marker that states the right number, show strict-match failing while flexible-extract passes, then write the version string that would catch the parser change.Traps
- Comparing scores across papers with different prompts, few-shot counts or parsers. The notebook calls such comparisons invalid without aligned configs.
- Treating flexible-extract as strict. Lenient parsing inflates the score without changing the model.
- Reading IFEval strict accuracy as semantic quality. It verifies constraints such as word count and keywords, not the meaning of the response.
Practice
The section mini-project covers this: a versioned task contract with prompt, parser and exact or structured metric, raw outputs kept beside the scores, and one parser failure exposed and fixed.
Self-test
-
An output says The answer is 42 with no
####marker. How do strict-match and flexible-extract score it?Strict fails because there is no#### (number)match, while flexible extracts the last number-like token 42 and passes. Report both pipeline names with the score. -
When should you report
acc_norminstead ofacc?acccompares raw choice loglikelihoods,acc_normnormalizes by length or bytes. Use the normalized form when choices differ in length or tokenization, and state which one the table uses. -
Which temperature and sampling settings suit deterministic task metrics?
Usetemperature0 withdo_sample falsefor exact and structured metrics. Use non-zerotemperatureonly forpass@k, self-consistency or robustness runs, and then fix theseed. -
What must be versioned for a task result to be reproducible?
Task YAML plus commit hash, prompt text, few-shot count and split, filter regexes, generation kwargs, model version and evaluator version, plus raw outputs saved withlog_samplesandoutput_path.
Completion criterion
Stop when
Your harness records prompt, few-shot, parser and versions with raw outputs, and you can show one output where strict and flexible disagree.
Mark passed, then open 4.3 Only mark this when you can do it without notes.Leads to
- 4.3 Model judges and human review Route open-ended outputs from this harness into judged evaluation with bias controls.
- 4.4 Uncertainty, robustness and honest claims Add slices and uncertainty to the harness before claiming any gap is real.
Comes after
- 3.7 Stopping rules and reproducible generationFeed validated, reproducibly generated outputs into task metrics.
- 4.1 Likelihood and perplexity, and their limitsCarry perplexity limits into explicit task contracts and metric choices.
Resources for this unit
- Stanford CS336 - resource accounting, inference and evaluation Use Spring 2026 Lectures 2, 10 and 12 for FLOPs, memory, arithmetic intensity, inference and evaluation. Reproduce the small calculations locally. Assignment 3 IsoFLOPs analysis is optional; do not require its hosted training API or costly experiments.
- Modern LLM Notebook - evaluation and FLOP accounting Use English notebook 25-evaluation and Appendix B FLOP and memory accounting. Build a 20-example evaluator with two slices, raw outputs and bootstrap intervals, and deliberately create one parser, prompt or judge failure.
- Language Model Evaluation Harness After the small evaluator passes, inspect task versioning, request construction and result schemas to understand production evaluation plumbing. Do not use the harness as a substitute for owning the task and metric contract.
Also in this section
- LLMs from Scratch companion - GPT FLOP analysis Use ch04/02_performance-analysis/flops-analysis.ipynb as a concrete calculator, then rebuild the formulas from the model configuration and reconcile parameters and at least one memory or operation estimate with runtime state.
- LLM Foundations Checkpoint - evaluation and arithmetic slice Use the contamination, evaluator-bias, parameter, FLOP, activation, state and KV requirements as the assessment contract. Keep raw outputs and distinguish a verified task result from a broad model capability claim.
- Stanford CS336 Assignment 3 - IsoFLOPs scaling Use Section 2.1 and Problem
chinchilla_isoflopswithdata/isoflops_curves.jsonfor the bounded curve-fitting exercise only. Do not use the hosted training API or costly experiments.