Unit 4.5 · about 5 h
76 / 324Regression diagnostics, calibration and uncertainty
A small residual sum of squares is not an assumption check, and a low prediction loss is not a calibration certificate. This unit supplies the statistical properties that later model-evaluation tools try to estimate.
- measure
You can now
- Derive residual orthogonality and conditional estimator covariance, then distinguish a fitting identity from evidence for regression assumptions.
- Audit residual curvature, changing spread, dependence and influential observations without claiming that plots prove exogeneity.
- Define probability calibration and separate calibration, resolution, outcome noise and parameter uncertainty.
Covers
- Regression assumptions and diagnostics
- Calibration and uncertainty decomposition
Study, in this order
4 steps
- 14.310x Lecture 18: assumptions behind regression uncertaintyMIT 14.310x - Data Analysis for Social Scientists
Slides 8 to 22: full column rank, error covariance, least-squares estimator and its properties
Depth · Treat \(X\) as fixed, or condition all error-moment assumptions on random \(X\). Unconditional \(\mathbb{E}[\text{error}] = 0\) alone does not establish exogeneity. Stop before \(F\)-tests.
- 18.05 Class 26 reading: residual plotsMIT 18.05 - Inference practice
Sections 3.1 and 3.2, Residuals and Homoscedasticity, pp. 4 to 5
Depth · Use the plots, but correct p. 4's claim that heteroscedastic data must be transformed before least squares. OLS still exists; its usual homoscedastic standard errors need repair. Constant variance is an assumption about model errors, not identical variances of fitted residuals.
- Probability calibration, before recalibrationProbability calibration: the statistical property
Opening calibration definition, proper-scoring-rule note, and Section 1.16.1, Calibration curves
Depth · Stop before Section 1.16.2. Do not fit Platt scaling here; derive what calibration and the Brier components mean.
- Murphy: noise and parameter uncertaintyProbabilistic Machine Learning: An Introduction
April 18, 2025 online edition: Section 2.1.2, Types of uncertainty, pp. 33 to 34; Section 11.7.4, Computing the posterior predictive, pp. 406 to 407
Depth · Use the two variance terms, not a new Bayesian-regression implementation. More data can reduce parameter uncertainty within a model; it need not repair model misspecification.
Focus on
- Differentiating squared error gives \(X^\top r = 0\) at an OLS fit. With an intercept, fitted residuals sum to zero even under a wrong model. For fixed full-rank \(X\) and \(\mathbb{E}[\epsilon \mid X] = 0\), OLS is unbiased; covariance is \(A\Omega A^\top\), where \(A = (X^\top X)^{-1} X^\top\). Only \(\Omega = \sigma^2 I\) reduces it to \(\sigma^2(X^\top X)^{-1}\).
- Look for curved residual means, a fan-shaped spread, serial dependence and high leverage. Fitted residual covariance is \(\sigma^2(I - H)\) under homoscedastic independent errors, so fitted residuals are not iid. Plots can reveal failures but cannot prove causal exogeneity.
- Binary probability calibration is \(\mathbb{E}[Y \mid P] = P\) under the evaluation population. If \(q(P) = \mathbb{E}[Y \mid P]\) and \(\pi = \mathbb{E}[Y]\), expected Brier loss is \(\mathbb{E}[(P - q)^2] - \operatorname{Var}(q) + \pi(1 - \pi)\). Reliability, resolution and outcome uncertainty are distinct; binning can hide within-bin failures.
- The law of total variance separates \(\mathbb{E}[\operatorname{Var}(Y \mid \theta, D)]\) from \(\operatorname{Var}(\mathbb{E}[Y \mid \theta, D])\). These are conditional outcome noise and posterior parameter uncertainty within the assumed model. Sampling variability of an estimator is a repeated-data property, and misspecification is not automatically represented by either variance term.
Reproduce closed-book
float64 without random draws. Fit \(y = x^2\) at \(x = [-2, -1, 0, 1, 2]\) with X = column_stack([ones(5), x]) and numpy.linalg.lstsq(X, y, rcond=None). The intercept/slope are approximately [2, 0], residuals [2, -1, -2, -1, 2], and the infinity norm of X.T @ residuals was 2.67e-15 or less; accept below 1e-12. On independent evaluation locations [-1.5, -0.5, 0.5, 1.5] with \(y = x^2\), residuals are [0.25, -1.75, -1.75, 0.25], so orthogonality did not fix curvature. Separately freeze probability predictions [0.2]*10 + [0.8]*10, then evaluate labels [1] + [0]*9 + [1]*9 + [0]. The two group frequencies are 0.1 and 0.9. Brier loss is 0.10 = reliability 0.01 - resolution 0.16 + uncertainty 0.25. Forecasts fixed at the group frequencies give 0.09; the calibrated but uninformative forecast 0.5 gives 0.25. Require absolute error below 1e-12 for these three Brier identities. This is a constructed population check, not a recalibrator fitted and scored on the same labels.Traps
- Using zero training residual mean as evidence that errors have zero conditional mean. It is forced by the fitted intercept.
- Assuming homoscedasticity or normality is required to compute OLS. They support particular efficiency or inference claims, not existence of a least-squares fit.
- Calling a lower Brier loss proof of better calibration. A gain in resolution can outweigh worse reliability.
Practice
The section mini-project covers this: generate separate fitting and evaluation data with curvature, heteroscedasticity and dependence; audit residuals and interval coverage; evaluate fixed probabilistic forecasts by reliability, resolution and subgroup calibration without fitting Platt scaling.
Self-test
-
Which assumptions produce the usual OLS covariance formula?
Fixed full-column-rank \(X\) defines \(A = (X^\top X)^{-1} X^\top\). The linear model gives \(\hat{\beta} - \beta = A\epsilon\). Conditional zero-mean errors give unbiasedness; finite error covariance gives \(\operatorname{Cov}(\hat{\beta} \mid X) = A\Omega A^\top\). Homoscedastic uncorrelated errors make \(\Omega = \sigma^2 I\), yielding \(\sigma^2(X^\top X)^{-1}\). Normality is needed for the familiar exact finite-sample normal/\(t\) inference, not this covariance algebra. -
Can a calibrated forecast have poor discrimination?
Yes. A constant forecast equal to the population prevalence is calibrated but separates no cases. For this balanced toy population, \(P = 0.5\) is calibrated and has Brier loss 0.25 versus 0.09 for the calibrated two-group forecast. -
Derive the Brier cross-term cancellation and state its assumption.
Write \(P - Y = (P - q(P)) + (q(P) - Y)\). Since \(q(P) = \mathbb{E}[Y \mid P]\), \(\mathbb{E}[q(P) - Y \mid P] = 0\), so the cross term vanishes after conditioning on \(P\). Binary \(Y\) then gives \(\mathbb{E}[\operatorname{Var}(Y \mid P)] = \mathbb{E}[q(1 - q)] = \pi(1 - \pi) - \operatorname{Var}(q)\). Finite moments hold because \(P\) and \(Y\) lie in \([0, 1]\). -
Why can more data shrink parameter uncertainty while uncertainty claims remain wrong?
Posterior concentration can occur within a misspecified model. The assumed noise law, dependence structure or evaluation population can still be wrong. Small parameter variance does not validate the model or guarantee coverage under shift.
Completion criterion
Stop when
You derive residual orthogonality and the Brier decomposition, pass both deterministic checks, and explain which statistical assumptions neither a good fit nor calibration can establish.
Mark passed, then open 4.6 Only mark this when you can do it without notes.Leads to
- 5.4 Calibration curves and Platt or isotonic recalibration Apply the calibration property to held-out reliability diagrams and recalibration.
- 2.1 Least squares: residuals, normal equations and stable solvers Separate statistical assumptions from numerical conditioning of a fit.
Resources for this unit
- MIT 14.310x - Data Analysis for Social Scientists Follow lectures 11-15, 17-20 and 22 with matching slides and problem sets for inference, regression, causality and experiment design.
- MIT 18.05 - Inference practice Use classes 17-20, 22-24 and 26 with their problem sets for tests, intervals, bootstrap and regression.
- Probabilistic Machine Learning: An Introduction Use the April 18, 2025 online edition: Sections 4.2.3 to 4.2.5, 4.5.1 and 4.6.4.1 for Bernoulli and Gaussian MLE/MAP; the definition and central intervals in 4.6.6; 2.1.2 and 11.7.4 for predictive uncertainty. Exercises 4.1 and 4.2 supply derivation practice.
- Probability calibration: the statistical property Read the opening calibration definition, the proper-scoring-rule note, and Section 1.16.1, Calibration curves. Stop before recalibration methods; the later Classical ML unit implements them.
Also in this section
- Causal Inference: What If Read Chapters 1-3 and 6-8 for causal effects, DAGs, confounding, selection bias and identification reasoning.
- Patterns of trustworthy experimentation Read after the course to connect health checks, guardrails and statistical interpretation to production launch decisions.
- Always Valid Inference Use the introduction and worked motivation to understand optional-stopping failure. Reproducing the theory is outside this gate.
- Permutation nulls and false discovery control Read
permutation_testNotes, Unpaired statistics, for exchangeability and exact partitions;false_discovery_controlNotes and the named thrombolysis example for Bonferroni and Benjamini-Hochberg. Use as an independent implementation oracle, not as a substitute for stating assumptions.