Unit 4.1 · about 4 h
72 / 324Estimands, estimators, bias, variance and consistency
Uncertainty belongs to a sampling procedure, not just to a reported number. This is the foundation for deciding whether an apparently precise model comparison is trustworthy.
- derive
You can now
- Distinguish a population estimand, a random estimator, its sampling distribution and the observed estimate.
- Derive the bias and mean squared error of variance estimates using \(n\) and \(n - 1\), and state where normality is needed.
- Prove consistency of the sample mean under finite variance and compare efficiency only within a stated estimator class.
Covers
- Populations, samples, estimands, estimators and estimates
- Bias, variance, consistency and efficiency
Study, in this order
3 steps
- 14.310x Lecture 11: sample means and estimationMIT 14.310x - Data Analysis for Social Scientists
Slides 2 to 21, the sample mean and CLT; slides 22 to 30, statistics, parameters, estimators and estimates
- 14.310x Lecture 12: assessing estimatorsMIT 14.310x - Data Analysis for Social Scientists
Slides 2 to 25: unbiasedness, sample variance, efficiency, MSE and consistency
Depth · Stop before frameworks for finding estimators. Supply the algebra behind slide 14 rather than treating unbiased sample variance as a fact to memorize.
- 18.05 Class 22 reading: the variance pivotMIT 18.05 - Inference practice
Section 6, Chi-square confidence intervals for the variance
Depth · Use the normal-sample chi-square pivot to obtain moments of sample variance. Leave interval construction for the intervals unit.
Focus on
- For iid \(X_i\) with mean \(\mu\) and finite variance \(\sigma^2\), expand \(\sum (X_i - \bar{X})^2 = \sum (X_i - \mu)^2 - n(\bar{X} - \mu)^2\). Expectations give \((n - 1)\sigma^2\). Independence removes covariance terms in \(\operatorname{Var}(\bar{X})\), not in \(\mathbb{E}[\bar{X}]\).
- Under normal sampling, \((n - 1)S^2/\sigma^2\) is chi-square with \(n - 1\) degrees of freedom. Its variance gives \(\operatorname{Var}(S^2) = 2\sigma^4/(n - 1)\). Multiplying \(S^2\) by \((n - 1)/n\) reduces variance but introduces bias \(-\sigma^2/n\).
- MSE = variance + squared bias. Unbiasedness need not minimize MSE; efficiency comparisons here are between unbiased estimators at the same sample size.
- Consistency is convergence in probability. Chebyshev gives \(P(|\bar{X} - \mu| \ge \epsilon) \le \sigma^2/(n\epsilon^2)\), which tends to zero. The CLT concerns a scaled sampling distribution, not convergence of individual observations.
Reproduce closed-book
Closed-book, derive the variance identity and MSE decomposition. For data
[1, 2, 3], compute \(\bar{X} = 2\), variance with \(n = 2/3\) and with \(n - 1 = 1\) by hand. Then use NumPy float64: rng = numpy.random.default_rng(41); draw x = rng.normal(2, 3, size=(100000, 5)) once, with samples as rows. Compute x.mean(axis=1).var(ddof=0), then v0 = x.var(axis=1, ddof=0) and v1 = x.var(axis=1, ddof=1). The sample-mean variance is 1.789 against theory 1.8. The means of v0 and v1 are 7.216 and 9.020 against 7.2 and 9.0; their variances (ddof=0) are 25.867 and 40.418 against 25.92 and 40.5. Their empirical MSEs relative to 9 are 29.050 and 40.418 against 29.16 and 40.5. Require maximum absolute error below 0.2 over these seven moment checks; this is a Monte Carlo tolerance, not a floating-point identity. Seeded values came from NumPy 2.5.4.Traps
- Calling an estimate unbiased because it is close to truth in one sample. Bias is an expectation over repetitions.
- Using the normal-sample variance formula for an arbitrary distribution. Unbiasedness needs finite second moments; this variance calculation uses normality and hence finite fourth moments.
- Assuming unbiased means consistent. \(X_1\) remains unbiased for \(\mu\) at every \(n\) but its variance never shrinks.
Practice
The section mini-project covers this: compare bias, variance and MSE of \(n\) versus \(n - 1\) variance estimators across sample sizes, with an analytic normal-sampling oracle and a heavy-tailed failure case.
Self-test
-
Where does the lost degree of freedom enter the sample-variance derivation?
Centering by \(\bar{X}\) subtracts \(n(\bar{X} - \mu)^2\). Its expectation is \(\sigma^2\), using unbiasedness of \(\bar{X}\) and \(\operatorname{Var}(\bar{X}) = \sigma^2/n\) from iid finite-variance sampling. The expected residual sum of squares is therefore \(n\sigma^2 - \sigma^2\). -
For \(n = 5\) and \(\sigma^2 = 9\) under normal sampling, which denominator has smaller MSE?
Using \(n\) gives bias -1.8, variance 25.92 and MSE 29.16. Using \(n - 1\) gives zero bias and variance/MSE 40.5. Lower MSE here does not make the biased estimate more efficient within the lecture's unbiased class. -
Prove the sample mean is consistent and name the assumptions at each step.
Common finite mean gives \(\mathbb{E}[\bar{X}] = \mu\) by linearity. Independence makes cross covariances zero, and common finite variance gives \(\operatorname{Var}(\bar{X}) = \sigma^2/n\). Chebyshev bounds the deviation probability by \(\sigma^2/(n\epsilon^2)\). For fixed \(\epsilon > 0\) and fixed \(\sigma^2\), that bound tends to zero.
Completion criterion
Stop when
You derive both variance estimators' bias and normal-sample MSE, pass the seven moment checks, and explain why \(X_1\) is unbiased but not consistent.
Mark passed, then open 4.2 Only mark this when you can do it without notes.Leads to
- 5.1 Holdout, cross-validation and resampling by the right unit Apply sampling-distribution reasoning when rows are not independent units.
Resources for this unit
- MIT 14.310x - Data Analysis for Social Scientists Follow lectures 11-15, 17-20 and 22 with matching slides and problem sets for inference, regression, causality and experiment design.
- MIT 18.05 - Inference practice Use classes 17-20, 22-24 and 26 with their problem sets for tests, intervals, bootstrap and regression.
Also in this section
- Causal Inference: What If Read Chapters 1-3 and 6-8 for causal effects, DAGs, confounding, selection bias and identification reasoning.
- Patterns of trustworthy experimentation Read after the course to connect health checks, guardrails and statistical interpretation to production launch decisions.
- Always Valid Inference Use the introduction and worked motivation to understand optional-stopping failure. Reproducing the theory is outside this gate.
- Probabilistic Machine Learning: An Introduction Use the April 18, 2025 online edition: Sections 4.2.3 to 4.2.5, 4.5.1 and 4.6.4.1 for Bernoulli and Gaussian MLE/MAP; the definition and central intervals in 4.6.6; 2.1.2 and 11.7.4 for predictive uncertainty. Exercises 4.1 and 4.2 supply derivation practice.
- Permutation nulls and false discovery control Read
permutation_testNotes, Unpaired statistics, for exchangeability and exact partitions;false_discovery_controlNotes and the named thrombolysis example for Bonferroni and Benjamini-Hochberg. Use as an independent implementation oracle, not as a substitute for stating assumptions. - Probability calibration: the statistical property Read the opening calibration definition, the proper-scoring-rule note, and Section 1.16.1, Calibration curves. Stop before recalibration methods; the later Classical ML unit implements them.