Unit 4.6 · about 4 h
77 / 324Missingness and selection mechanisms
A carefully analyzed observed sample can still answer the wrong population question. Missingness and selection determine whose outcomes enter the analysis and what assumptions are needed to recover the estimand.
- explain
You can now
- Define MCAR, MAR and MNAR relative to observed and missing variables, without assuming MAR is testable from incomplete data.
- Derive why complete-case means can be biased under MAR and when standardization or inverse-observation weighting identifies the population mean.
- Explain selection as conditioning on a collider and demonstrate a misleading association in a small table and a simulation.
Covers
- Missingness and selection mechanisms
Study, in this order
2 steps
- What If: missingness as an exchangeability questionCausal Inference: What If
August 19, 2026 edition: Section 2.2, p. 18, margin definitions of MCAR, MAR and MNAR; Sections 2.3 and 2.4, pp. 19 to 25, standardization and weighting
Depth · Translate missing counterfactuals to an observation indicator \(R\) for a missing outcome \(Y\) with fully observed \(X\). Reuse the conditioning logic, not the book's treatment notation without adaptation.
- What If: selection and censoringCausal Inference: What If
Sections 8.1 to 8.3, pp. 109 to 115; Section 8.5, pp. 117 to 120, identification conditions for selection adjustment
Depth · Skip Technical Point 8.1 on hazard ratios. Keep the collider examples and exchangeability, positivity and consistency requirements; leave longitudinal censoring models for later.
Focus on
- With \(R = 1\) when \(Y\) is observed and \(X\) fully observed, MCAR means \(R\) is independent of \((X, Y)\). MAR means \(R\) is independent of \(Y\) conditional on \(X\). MNAR allows observation probability to depend on unobserved \(Y\) even given \(X\). These labels depend on what is actually observed.
- Under MAR, \(\mathbb{E}[Y \mid X, R = 1] = \mathbb{E}[Y \mid X]\), but \(\mathbb{E}[Y \mid R = 1]\) need not equal \(\mathbb{E}[Y]\). Standardize conditional observed means using the population distribution of \(X\), not the distribution among complete cases.
- If \(p(X) = P(R = 1 \mid X) > 0\) and MAR holds, \(\mathbb{E}[RY/p(X) \mid X] = \mathbb{E}[Y \mid X]\). Integrability and the correct observation probabilities give an unbiased Horvitz-Thompson mean; normalized weights are a different finite-sample estimator. Small probabilities can create large variance.
- Selection on a common effect can associate otherwise independent variables. Randomization before selection does not guarantee exchangeability after treatment-dependent attrition. MAR and adequate adjustment require substantive assumptions, not just a successful regression fit.
Reproduce closed-book
By hand let \(X \sim \operatorname{Bernoulli}(0.5)\), \(P(Y = 1 \mid X) = 0.2 + 0.6X\) and observation probability \(p(X) = 0.8 - 0.6X\). Derive \(\mathbb{E}[Y] = 0.5\) and \(\mathbb{E}[Y \mid R = 1] = 0.32\) under MAR. If instead observation probability is \(0.8 - 0.6Y\), the MNAR complete-case mean is 0.2. Check in NumPy
float64: rng = numpy.random.default_rng(46); N = 100000; first X = (rng.random(N) < 0.5).astype(numpy.float64), next Y = (rng.random(N) < 0.2 + 0.6*X).astype(numpy.float64), then U = rng.random((3, N)). Define R0 = U[0] < 0.5, R1 = U[1] < 0.8 - 0.6*X, R2 = U[2] < 0.8 - 0.6*Y. Full, MCAR, MAR and MNAR means are 0.4983, 0.4977, 0.3171 and 0.1971. The MAR weighted mean numpy.mean(R1*Y/(0.8 - 0.6*X)) is 0.4922. Require maximum absolute deviation below 0.01 from [0.5, 0.5, 0.32, 0.2, 0.5]. Finally enumerate independent fair bits \(A\) and \(U\) with \(Y = U\) and selection \(S = (A \text{ or } U)\): among selected cases the \(A = 1\) mean is 1/2 and the \(A = 0\) mean is 1, despite zero causal effect. No random draw is needed for this four-row counterexample. Seeded values came from NumPy 2.5.4.Traps
- Equating MAR with an unbiased complete-case marginal mean. Observed covariates can influence both missingness and outcomes.
- Claiming missingness mechanisms are identifiable just from the observed outcomes. Different unobserved-outcome laws can produce the same incomplete data.
- Fixing MNAR by weighting only on X. The assumed observation model omits dependence on missing Y, so the MAR proof no longer applies.
Practice
The section mini-project covers this: compare complete-case, standardized and inverse-observation-weighted means under MCAR, MAR and MNAR; include a zero-overlap failure and a treatment-dependent selection DAG, with a stated sensitivity analysis rather than claiming MNAR is solved.
Self-test
-
Derive the MAR complete-case mean in the toy model.
\(P(R = 1) = 0.5 \times 0.8 + 0.5 \times 0.2 = 0.5\). MAR makes \(Y\) and \(R\) conditionally independent given \(X\), so \(\mathbb{E}[RY] = 0.5 \times 0.8 \times 0.2 + 0.5 \times 0.2 \times 0.8 = 0.16\). Dividing by 0.5 gives 0.32, not the population mean 0.5. -
Where are MAR and positivity used in the weighting proof?
Positivity makes \(1/p(X)\) defined on every population stratum. MAR gives \(\mathbb{E}[R \mid Y, X] = p(X)\). Therefore \(\mathbb{E}[RY/p(X) \mid X] = \mathbb{E}[Y\mathbb{E}[R \mid Y, X]/p(X) \mid X] = \mathbb{E}[Y \mid X]\). Averaging over the population \(X\) distribution identifies \(\mathbb{E}[Y]\), provided the terms are integrable. -
Why does the selected four-row table show a spurious treatment association?
Selection excludes \((A, U) = (0, 0)\). Among selected \(A = 0\) cases, \(U\) must be 1; among selected \(A = 1\) cases, \(U\) remains equally likely 0 or 1. The path \(A \to S \leftarrow U \to Y\) opens when conditioning on \(S\). Changing \(A\) does not change \(Y = U\), so the causal effect is still zero.
Completion criterion
Stop when
You derive the two biased complete-case means, pass the five simulation checks, and identify the collider and the exact failed assumption in a new missingness scenario.
Mark passed, then open 4.7 Only mark this when you can do it without notes.Leads to
- 5.1 Holdout, cross-validation and resampling by the right unit Check population selection before treating observed units as representative.
- 4.4 Uncertainty, robustness and honest claims Carry missingness assumptions and sensitivity analysis into evaluation claims.
Resources for this unit
- Causal Inference: What If Read Chapters 1-3 and 6-8 for causal effects, DAGs, confounding, selection bias and identification reasoning.
Also in this section
- MIT 14.310x - Data Analysis for Social Scientists Follow lectures 11-15, 17-20 and 22 with matching slides and problem sets for inference, regression, causality and experiment design.
- MIT 18.05 - Inference practice Use classes 17-20, 22-24 and 26 with their problem sets for tests, intervals, bootstrap and regression.
- Patterns of trustworthy experimentation Read after the course to connect health checks, guardrails and statistical interpretation to production launch decisions.
- Always Valid Inference Use the introduction and worked motivation to understand optional-stopping failure. Reproducing the theory is outside this gate.
- Probabilistic Machine Learning: An Introduction Use the April 18, 2025 online edition: Sections 4.2.3 to 4.2.5, 4.5.1 and 4.6.4.1 for Bernoulli and Gaussian MLE/MAP; the definition and central intervals in 4.6.6; 2.1.2 and 11.7.4 for predictive uncertainty. Exercises 4.1 and 4.2 supply derivation practice.
- Permutation nulls and false discovery control Read
permutation_testNotes, Unpaired statistics, for exchangeability and exact partitions;false_discovery_controlNotes and the named thrombolysis example for Bonferroni and Benjamini-Hochberg. Use as an independent implementation oracle, not as a substitute for stating assumptions. - Probability calibration: the statistical property Read the opening calibration definition, the proper-scoring-rule note, and Section 1.16.1, Calibration curves. Stop before recalibration methods; the later Classical ML unit implements them.