Section 4 of 5 · Mathematical Foundations
Statistics, Experimentation & Causality
A model result matters only when the estimand, uncertainty and decision rule survive leakage, selection and alternative explanations.
8 units 17 topics 0/8
Units
8 in this section
- 4.1 Estimands, estimators, bias, variance and consistency derive4 h
- 4.2 Maximum likelihood and MAP estimation derive · implement5 h
- 4.3 Intervals, hypothesis tests, power and effect size derive · measure6 h
- 4.4 Bootstrap, permutation tests and multiple comparisons implement · measure6 h
- 4.5 Regression diagnostics, calibration and uncertainty measure5 h
- 4.6 Missingness and selection mechanisms explain4 h
- 4.7 Randomized experiment design, power and MDE derive · measure5 h
- 4.8 Causal claims, causal graphs, adjustment and identification explain · derive6 h
Before moving on
Checkpoint
Audit a flattering experiment, identify its validity failures, and produce a defensible rerun and decision memo.
Deliberately out of scope
Leave for later
- Advanced semiparametric efficiency theory
- Longitudinal causal inference
- Specialized survey methodology
Resources for this section
Shared by every unit above
- MIT 14.310x - Data Analysis for Social Scientists Follow lectures 11-15, 17-20 and 22 with matching slides and problem sets for inference, regression, causality and experiment design.
- MIT 18.05 - Inference practice Use classes 17-20, 22-24 and 26 with their problem sets for tests, intervals, bootstrap and regression.
- Causal Inference: What If Read Chapters 1-3 and 6-8 for causal effects, DAGs, confounding, selection bias and identification reasoning.
- Patterns of trustworthy experimentation Read after the course to connect health checks, guardrails and statistical interpretation to production launch decisions.
- Always Valid Inference Use the introduction and worked motivation to understand optional-stopping failure. Reproducing the theory is outside this gate.
- Probabilistic Machine Learning: An Introduction Use the April 18, 2025 online edition: Sections 4.2.3 to 4.2.5, 4.5.1 and 4.6.4.1 for Bernoulli and Gaussian MLE/MAP; the definition and central intervals in 4.6.6; 2.1.2 and 11.7.4 for predictive uncertainty. Exercises 4.1 and 4.2 supply derivation practice.
- Permutation nulls and false discovery control Read
permutation_testNotes, Unpaired statistics, for exchangeability and exact partitions;false_discovery_controlNotes and the named thrombolysis example for Bonferroni and Benjamini-Hochberg. Use as an independent implementation oracle, not as a substitute for stating assumptions. - Probability calibration: the statistical property Read the opening calibration definition, the proper-scoring-rule note, and Section 1.16.1, Calibration curves. Stop before recalibration methods; the later Classical ML unit implements them.
Apply the section
Mini-project · A statistical audit harness and the memo for a flattering experiment (14 to 18 h)
Build a simulation harness with known truth that tests estimators, intervals, multiplicity, regression, calibration, missingness and causal adjustment, each with an oracle and a failure case. Then audit an experiment that looks too good, rerun it under a plan written first, and write the decision memo.
Steps
- Build the harness on the probability section's estimator: each check names its known truth and reports a Monte Carlo standard error. Compare the two variance estimators across sample sizes against analytic normal-sampling moments, then repeat under a heavy tail and show the normal-theory formula failing.
- Measure percentile-bootstrap coverage for an exponential mean at two sample sizes. Simulate many hypotheses with some true nulls and compare FWER and FDR under Bonferroni and Benjamini-Hochberg, then measure the bias and naive-interval coverage of the largest of several estimates with equal true effects.
- Generate separate fitting and evaluation data with a curved mean, growing variance and dependent errors. Compare the usual and sandwich standard errors against the sampling standard deviation, measure prediction-interval coverage, and score fixed forecasts by Brier decomposition with subgroup calibration.
- Compare complete-case, standardized and inverse-weighted means under MCAR, MAR and MNAR. Add a stratum with zero observation probability, draw a DAG where selection depends on treatment, and report MNAR as a sensitivity range over an assumed shift rather than a single corrected number.
- Define a potential-outcome estimand with a DAG holding a confounder and a selection node. Show observed and interventional means differing, a valid backdoor adjustment recovering the effect, an invalid adjustment failing, and an overlap failure, optionally reporting pre-treatment subgroup effects without claiming individual effects.
- Construct the flattering experiment: event-row analysis of a user-randomized test, daily peeking, best-of-several metrics and a post-treatment segment, plus a sample-ratio mismatch. Write the pre-analysis plan before touching it, then list every validity failure with the check that exposes it, including the assignment-count check.
- Rerun under the plan: size arms for both rejection tails, randomize and analyze users, hold the fixed horizon, and report the effect and interval against the practical threshold and guardrails. Write a one-page memo deciding ship, extend or do not ship, with reasons.
Deliverables
numlab/stats/: the harness and every check, each with its oracle and failure case.- The audit: the pre-analysis plan, validity failures with checks, the rerun and its output.
- The decision memo, the two DAGs, and
notes/statistics.mdwith the derivations behind each check.
Checks
- rubric · Self-grade with the rubric below
Rubric
| Criterion | Pass | Strong |
|---|---|---|
| Estimators and intervals | The variance-estimator comparison matches the normal-theory oracle, the heavy-tail failure is shown, and bootstrap coverage is measured at both sample sizes. | Each Monte Carlo result reports its standard error, and the notes say which assumption each failure breaks. |
| Multiplicity and selection | Realized FWER and FDR match what each procedure controls, and the winner's-curse bias is measured. | The notes distinguish the two error rates and explain why the selected estimate needs its own interval. |
| Regression and calibration | Diagnostics, the sandwich comparison, prediction-interval coverage and the Brier decomposition are all on separate evaluation data. | Subgroup calibration finds a failure that the overall reliability term hides. |
| Missingness and causality | Each missingness mechanism, both overlap failures and the valid and invalid adjustments are shown against the known truth. | The MNAR result is a sensitivity range, and every causal claim names its untestable assumption. |
| Experiment audit and memo | The plan predates the analysis, every validity failure has a check, and the rerun reports the effect against the practical threshold. | The memo makes a decision a reader could act on, with guardrails and residual risks stated. |
Feeds into: Capstone experiment analysis: the harness, the audit with its failures, the rerun and the decision memo.