Unit 4.3 · about 6 h
74 / 324Intervals, hypothesis tests, power and effect size
Intervals and tests are statements about a model and a procedure. Checking coverage reveals whether the procedure earns its advertised uncertainty, while effect size says what the result means in practice.
- derive
- measure
You can now
- Derive a normal-mean confidence interval and test from a pivot, and distinguish confidence, credible and prediction intervals.
- Explain null, alternative, statistic, p-value, Type I/II error, power and effect size without assigning a posterior probability to a p-value.
- Measure interval coverage and the null p-value distribution using repeated independent samples.
Covers
- Confidence, credible and prediction intervals
- Hypothesis tests, errors, power and effect size
Study, in this order
4 steps
- 14.310x Lecture 13: reliability, intervals and testsMIT 14.310x - Data Analysis for Social Scientists
Slides 5 to 26, standard errors, confidence intervals and the chi-square and \(t\) distributions; slides 28 to 38, the known- and unknown-variance normal-mean intervals; slides 40 to 74, hypotheses, error types, critical regions, the \(\alpha\)-\(\beta\) trade-off and power for a simple alternative
Depth · Say 'fail to reject' rather than 'accept'. Use the normal-sample assumption for exact \(t\) inference. Leave experiment sample-size calculations to the design unit.
- 18.05 Class 17 reading: p-valuesMIT 18.05 - Inference practice
Section 4.3, Significance level and power; Section 7, p-values; Section 7.1, Example: z-tests
Depth · Work the normal \(z\)-test, then stop; do not turn a p-value into the probability that a hypothesis is true.
- 18.05 Class 22 reading: pivoting and t intervalsMIT 18.05 - Inference practice
Sections 3 to 5: interval statistics, z confidence intervals and t confidence intervals
- Murphy: posterior intervals and future observationsProbabilistic Machine Learning: An Introduction
April 18, 2025 online edition: Section 4.6.6, p. 146, definition (4.192) and central-interval quantiles; Section 11.7.4, pp. 406 to 407, posterior predictive and observation-noise term
Depth · Stop 4.6.6 before Monte Carlo quantile estimation and HPD regions. In 11.7.4 use the intercept-only model to separate a mean interval from a future-observation interval; do not implement Bayesian regression.
Focus on
- For iid \(\mathcal{N}(\mu, \sigma^2)\) data with known \(\sigma\), \(Z = (\bar{X} - \mu)/(\sigma/\sqrt{n})\) is standard normal. Inverting \(|Z| \le z(1 - \alpha/2)\) gives \(\bar{X} \pm z\sigma/\sqrt{n}\). Replacing \(\sigma\) by \(S\) gives an exact \(t(n - 1)\) pivot under normal sampling; for other distributions it is only approximate.
- Coverage is \(P_\mu(\mu \in C(X))\). A credible interval contains specified posterior mass conditional on data and prior. A prediction interval covers a new \(Y\), so it includes observation noise as well as uncertainty about the mean.
- For a continuous exact null statistic with CDF \(F\), \(F(T)\) is uniform under the null. For a two-sided normal test, \(p = 2[1 - \Phi(|Z|)]\) is also uniform. Discrete valid p-values can be conservative rather than uniform.
- Power is \(P_\delta(\text{reject})\) for a specified alternative \(\delta\); Type II error is its complement. Report \(\delta\) in outcome units and a practical threshold, not only whether \(p\) crosses \(\alpha\).
Reproduce closed-book
float64 with SciPy normal/t quantiles: rng = numpy.random.default_rng(43); first draw x = rng.normal(2, 3, size=(20000, 25)), then future = rng.normal(2, 3, size=20000). Use means by row, c = scipy.stats.norm.ppf(0.975), Z = (means - 2)/0.6 and p = 2*norm.sf(abs(Z)). Mean-interval coverage is 0.9503, prediction-interval coverage 0.95015, \(P(p \le 0.05)\) is 0.0497 and \(P(p \le 0.5)\) is 0.49905. Unknown-\(\sigma\) \(t\) intervals using x.std(axis=1, ddof=1)/5 and scipy.stats.t.ppf(0.975, 24) cover 0.95165. Require maximum absolute deviation below 0.01 from the five theoretical rates [0.95, 0.95, 0.05, 0.5, 0.95]. This is simulation error, not exact equality. Seeded values came from NumPy 2.5.4 with SciPy 1.18.1.Traps
- Reading a 95% confidence interval as a 95% posterior probability for the fixed parameter in the realized interval.
- Using the mean interval to cover individual outcomes. Its width shrinks to zero even when outcome noise remains.
- Taking failure to reject as evidence of equivalence. A wide interval can include both zero and practically important effects.
Practice
18.05 Problem Set 10: Soda and Probabilities for hypotheses
Problem 2(a to c), Soda, p. 1; Problem 5(a to c), Probabilities for hypotheses, pp. 2 to 3
Attempt before opening the official solutions linked on the course Problem Sets page. You may use NumPy/SciPy instead of R; distinguish interval coverage from posterior mass. In 5(b), \(\mathcal{N}(\mu, 5)\) means standard deviation 5, as the next sentence states. The official 98% \(t\) interval in 2(c) prints a half-width of 3.89; its endpoints [352.20, 359.80] use the correct 3.80.
Self-test
-
Show that the two-sided normal p-value is uniform under the null.
For \(0 < u < 1\), \(p \le u\) iff \(|Z| \ge \Phi^{-1}(1 - u/2)\). Under the exact standard-normal null, symmetry gives two tails of probability \(u/2\), hence \(P(p \le u) = u\). The endpoints follow by continuity. Normality and a continuous null distribution are used in the tail calculation. -
What changes if \(\sigma\) is estimated from the same small normal sample?
The standardized mean uses \(S\) and has a \(t(n - 1)\) distribution, not a standard normal distribution. Independence of the normal sample mean and variance justifies this pivot. Using normal critical values ignores extra variance-estimation uncertainty. -
Why does a future-observation interval not shrink to zero with n?
Independent future noise contributes \(\sigma^2\) to \(\operatorname{Var}(Y_{\text{new}} - \bar{X})\), while estimating the mean contributes \(\sigma^2/n\). Only the second term vanishes. -
Can a 95% Bayesian credible interval have less than 95% frequentist coverage?
Yes. It has 95% posterior mass under its prior and model. Repeated-sampling coverage at a fixed parameter is a different property and can be poor, especially with a mismatched strong prior.
Completion criterion
Stop when
You derive both intervals, pass the five rate checks, solve the two named problems, and explain why neither \(p\) nor confidence level is a posterior hypothesis probability.
Mark passed, then open 4.4 Only mark this when you can do it without notes.Leads to
- 4.4 Uncertainty, robustness and honest claims Use coverage and test assumptions to qualify evaluation claims.
Resources for this unit
- MIT 14.310x - Data Analysis for Social Scientists Follow lectures 11-15, 17-20 and 22 with matching slides and problem sets for inference, regression, causality and experiment design.
- MIT 18.05 - Inference practice Use classes 17-20, 22-24 and 26 with their problem sets for tests, intervals, bootstrap and regression.
- Probabilistic Machine Learning: An Introduction Use the April 18, 2025 online edition: Sections 4.2.3 to 4.2.5, 4.5.1 and 4.6.4.1 for Bernoulli and Gaussian MLE/MAP; the definition and central intervals in 4.6.6; 2.1.2 and 11.7.4 for predictive uncertainty. Exercises 4.1 and 4.2 supply derivation practice.
Also in this section
- Causal Inference: What If Read Chapters 1-3 and 6-8 for causal effects, DAGs, confounding, selection bias and identification reasoning.
- Patterns of trustworthy experimentation Read after the course to connect health checks, guardrails and statistical interpretation to production launch decisions.
- Always Valid Inference Use the introduction and worked motivation to understand optional-stopping failure. Reproducing the theory is outside this gate.
- Permutation nulls and false discovery control Read
permutation_testNotes, Unpaired statistics, for exchangeability and exact partitions;false_discovery_controlNotes and the named thrombolysis example for Bonferroni and Benjamini-Hochberg. Use as an independent implementation oracle, not as a substitute for stating assumptions. - Probability calibration: the statistical property Read the opening calibration definition, the proper-scoring-rule note, and Section 1.16.1, Calibration curves. Stop before recalibration methods; the later Classical ML unit implements them.