Uncertainty, robustness and honest claims

1. Modern LLM Notebook 25-evaluation: aggregation and composite scores

   Modern LLM Notebook - evaluation and FLOP accounting

   Section 5 comparison table with analysis points (same-scale compare, business-relevant metrics, trends over single numbers, labeled source and config) and Section 5.2 (arithmetic, normalized, weighted, geometric and rank-sum methods with geometric penalizing weak spots)

   Done
2. Modern LLM Notebook 25-evaluation: robustness and pitfalls

   Modern LLM Notebook - evaluation and FLOP accounting

   Section 8.2 (CR@K, prompt, order and sampling robustness) and Sections 10.1 to 10.3 (prompt sensitivity 5 to 15 percent with OLMES prompts, few-shot counts, HumanEval 164 variance, seed, temperature, mean plus minus std, per-subject slices, length bias, best-score reporting)

   Done
3. lm-eval-harness: bootstrap uncertainty in `metrics.py`

   Language Model Evaluation Harness

   bootstrap_stderr with resampling and iters, stderr_for_metric with the bootstrappable list (median, matthews, f1, perplexity, bleu, chrf, ter, nanmean) and closed-form mean and acc_all, plus pooled_sample_stderr for groups

   Done
4. CS336 Lecture 12: safety benchmarks and realism

   Stanford CS336 - resource accounting, inference and evaluation

   Function safety_benchmarks() with HarmBench 510 behaviors, AIR-Bench 314 categories and 5694 prompts, GCG transfer, contextual safety and dual-use CyBench; plus realism() with GDPVal, MedHELM 121 tasks from 29 clinicians, and Clio

   Done

Focus on

• Slice first, then aggregate. Report per-slice means before any composite, and say which slice drives the headline.
• Compare paired deltas on the same prompts, not two independent means. Pairing cancels prompt difficulty, which two independent intervals would both include.
• Bootstrap resamples with replacement and reports the std of the stat. Use the closed-form SE for plain means and bootstrap for medians, F1, perplexity and similar.
• Fix the sampling and prompt budget before comparing. Temperature, seed, few-shot and prompt version are part of the result.

Reproduce closed-book

Traps

• Reading overlapping intervals as proof of no difference without a paired test. Pair the prompts first, then test the \(\delta\).
• Reporting the best prompt or seed. Prompt swings of 5 to 15 percent dwarf small model gaps.
• Claiming safety from a capability average. A model at 90 math and 10 safety averages 50 and is still unsafe, so gate safety separately.

Practice

The section mini-project covers this: 20-example two-slice evaluator with raw outputs, paired deltas and bootstrap intervals, plus one deliberate prompt or sampling failure.

Self-test

1. Exercise 3 scores Model A at 82, 71, 65, 78 and Model B at 78, 75, 70, 72. What do the two means say?

   Show answer Got it Missed

2. When should you use bootstrap_stderr instead of a closed-form SE?

   Show answer Got it Missed

3. Why compare paired deltas rather than two independent means?

   Show answer Got it Missed

4. How should capability and safety results be gated?

   Show answer Got it Missed

Leads to

• 4.5 Contamination and memorization Add the contamination boundary to the release gate.
• 4.7 Memory, arithmetic intensity and reconciling estimates Carry interval discipline into quality-compute comparisons with uncertainty ranges.

Comes after

• 3.5 Bayes rule, conjugate priors and posterior predictivesCarry the posterior-versus-confidence distinction into model uncertainty claims.
• 4.3 Intervals, hypothesis tests, power and effect sizeUse coverage and test assumptions to qualify evaluation claims.
• 4.6 Missingness and selection mechanismsCarry missingness assumptions and sensitivity analysis into evaluation claims.
• 4.7 Randomized experiment design, power and MDEUse predeclared thresholds and a validity audit when reporting evaluation results.
• 4.8 Causal claims, causal graphs, adjustment and identificationKeep causal identification separate from predictive evaluation claims.
• 3.4 Keeping training numerically healthyTreat seed and run variability as uncertainty instead of noise to ignore.
• 4.1 Likelihood and perplexity, and their limitsTreat perplexity gaps with slices and intervals instead of reading them as wins.
• 4.2 Task contracts and automatic metricsAdd slices and uncertainty to the harness before claiming any gap is real.
• 4.3 Model judges and human reviewCarry judge error into slices and intervals instead of quoting a bare win rate.
• 4.5 Contamination and memorizationReport decontaminated deltas with intervals instead of a single full-set mean.

Resources for this unit

• Stanford CS336 - resource accounting, inference and evaluation
  Stanford Universityprimarycourse
  Use Spring 2026 Lectures 2, 10 and 12 for FLOPs, memory, arithmetic intensity, inference and evaluation. Reproduce the small calculations locally. Assignment 3 IsoFLOPs analysis is optional; do not require its hosted training API or costly experiments.
• Modern LLM Notebook - evaluation and FLOP accounting
  Walking Labsrequiredimplementation
  Use English notebook 25-evaluation and Appendix B FLOP and memory accounting. Build a 20-example evaluator with two slices, raw outputs and bootstrap intervals, and deliberately create one parser, prompt or judge failure.
• Language Model Evaluation Harness
  EleutherAIoptionalreference
  After the small evaluator passes, inspect task versioning, request construction and result schemas to understand production evaluation plumbing. Do not use the harness as a substitute for owning the task and metric contract.