Unit 4.7 · about 5 h
78 / 324Randomized experiment design, power and MDE
The design determines which uncertainty calculation is valid. Sample size follows from the effect worth detecting and the independent units available, not from how many telemetry rows can be collected.
- derive
- measure
You can now
- Specify an estimand, randomization unit, analysis unit, primary metric, guardrails, practical threshold and stopping/multiplicity policy before an experiment.
- Derive normal-approximation power, sample size and minimum detectable effect for two independent equal-variance arms.
- Audit clustering, interference, attrition, sample-ratio mismatch and post-treatment segment selection before interpreting an effect.
Covers
- Randomization, units, metrics and experiment design
- Power and minimum detectable effect
Study, in this order
4 steps
- 14.310x Lecture 15: randomized comparisonsMIT 14.310x - Data Analysis for Social Scientists
Slides 2 to 8: average treatment effects, difference in means, variance, intervals and testing
Depth · Separate finite-population randomization variance from the independent-superpopulation normal model used in this card's power check. Stop before the Oregon example.
- 14.310x Lecture 13: power calculationsMIT 14.310x - Data Analysis for Social Scientists
Slides 75 to 83: power inputs, normal approximation, stratification and clustering
Depth · Retain both rejection tails when checking power. The sample-size shortcut drops a small tail and is an approximation, not an exact equality.
- 14.310x Lecture 22: units and spilloversMIT 14.310x - Data Analysis for Social Scientists
Slides 2 to 4, interventions, units and stratification; slides 10 to 17, spillovers and two-step randomization
Depth · Use the labor-market example to explain interference. Skip the detailed outcome tables and the later rice-program case.
- Trustworthy experimentation during the experimentPatterns of trustworthy experimentation
Capture Unexpected Effects Early with a Complete Metric Set, including the taxonomy table; Measure Early and Often; Set Up Alerts to Detect Unintended Degradations; Use Segments that are Stable during the A/B Test; Segment by Date
Depth · Use the operational checklist, not its word 'guarantees' for segment validity. Stable segments and no detected sample-ratio mismatch do not prove exchangeability, prevent false alarms or establish correct uncertainty. Separate safety monitoring from fixed-horizon efficacy inference.
Focus on
- Specify assignment and analysis units before choosing a standard error. Difference in means estimates an intention-to-treat effect of assignment when outcomes are observed without selection. Treatment receipt and post-treatment activity are different estimands.
- For independent arms with \(n\) observations each and known common variance \(\sigma^2\), \(D = \text{mean}(T) - \text{mean}(C)\) has variance \(2\sigma^2/n\). With \(\lambda = \delta/\text{SE}\) and \(c = z(1 - \alpha/2)\), two-sided power is \(1 - \Phi(c - \lambda) + \Phi(-c - \lambda)\).
- For positive \(\delta\) and a negligible lower tail, \(n\) per arm is approximately \(2\sigma^2[z(1 - \alpha/2) + z(1 - \beta)]^2/\delta^2\). MDE reverses this equation. Distinguish per-arm \(n\) from total \(2n\) and absolute outcome change from relative percentage change.
- Predeclare one primary metric and how guardrail decisions and multiple metrics will be handled. Test observed assignment counts against the configured allocation, not an assumed 50/50 split. Investigate SRM rather than treating balanced counts as proof that logging is unbiased.
Reproduce closed-book
float64 and scipy.stats.norm to compute c = norm.ppf(0.975), SE = sqrt(2/393), approximate MDE = (c + norm.ppf(0.8))*SE, and exact normal-model two-sided power = norm.sf(c - delta/SE) + norm.cdf(-c - delta/SE). Rounded values are SE 0.07134, MDE 0.1999 and power 0.8006. Check by sampling sufficient statistics, not individual rows: rng = numpy.random.default_rng(47); first control = rng.normal(0, 1/sqrt(393), size=20000), then treatment = rng.normal(0.2, 1/sqrt(393), size=20000). Reject when abs((treatment - control)/SE) > c. Power is 0.7988; its absolute difference from exact power is 0.0018 or less, with a Monte Carlo acceptance tolerance of 0.01. Explain why the same calculation does not apply to 393 observations that come from only a few independent clusters. Seeded values came from NumPy 2.5.4 with SciPy 1.18.1.Traps
- Counting repeated events from the same randomized user as independent participants. More rows need not add independent information.
- Using a significant pilot's selected effect as the unquestioned power input. Winner's curse can make the planned experiment too small.
- Choosing a favorable metric, duration or active-user subgroup after seeing results. Those choices change the estimand and the error policy.
Practice
18.05 Problem Set 8, Problem 3: radar-gun threshold and power
Problem 3(a), 3(b)(i to ii) and 3(c), z-test, p. 2: null, threshold, power at 45 mph and number of cameras
Attempt before reading the official solutions linked on the course Problem Sets page. This is a one-sided test, unlike the reproduce check. The section mini-project should add a pre-analysis plan, assignment-unit audit, SRM and guardrail decision rules.
Self-test
-
Why does halving the target effect require about four times as many observations?
With \(\alpha\), target power and variance fixed, the normal-approximation sample size is proportional to \(1/\delta^2\). Replacing \(\delta\) by \(\delta/2\) multiplies \(n\) by four. This assumes the same independent sampling units and variance model. -
Prove equal allocation minimizes variance when total N and common arm variance are fixed.
For independent arms, \(\operatorname{Var}(D) = \sigma^2(1/n_T + 1/n_C) = \sigma^2 N/(n_T n_C)\), with \(n_T + n_C = N\). The nonnegative square \((n_T - n_C)^2\) gives \(n_T n_C \le N^2/4\), so variance is at least \(4\sigma^2/N\). Equality occurs at equal allocation. Common variance and equal per-unit costs are the conditions for this design comparison. -
Can passing SRM and finding stable pre-treatment segments establish a causal effect?
No. They remove some warning signs but do not rule out outcome-dependent logging, interference, selection or measurement errors. Valid assignment and observation, a well-defined intervention and correct inference still matter. Multiplicity can also create false segment discoveries. -
What is wrong with deciding from \(p\) alone even in a valid experiment?
A tiny effect can be statistically significant in a large sample, and an important effect can be imprecisely estimated in a small sample. Compare the effect and interval with the predeclared practical threshold, guardrails, costs and risks.
Completion criterion
Stop when
You derive power and per-arm sample size, pass the Monte Carlo check, solve the radar-gun problem, and can explain why clustered rows or post-treatment segments invalidate a naive experiment analysis.
Mark passed, then open 4.8 Only mark this when you can do it without notes.Leads to
- 5.1 Holdout, cross-validation and resampling by the right unit Respect dependence and randomization units in uncertainty estimates.
- 4.4 Uncertainty, robustness and honest claims Use predeclared thresholds and a validity audit when reporting evaluation results.
Resources for this unit
- MIT 14.310x - Data Analysis for Social Scientists Follow lectures 11-15, 17-20 and 22 with matching slides and problem sets for inference, regression, causality and experiment design.
- Patterns of trustworthy experimentation Read after the course to connect health checks, guardrails and statistical interpretation to production launch decisions.
Also in this section
- MIT 18.05 - Inference practice Use classes 17-20, 22-24 and 26 with their problem sets for tests, intervals, bootstrap and regression.
- Causal Inference: What If Read Chapters 1-3 and 6-8 for causal effects, DAGs, confounding, selection bias and identification reasoning.
- Always Valid Inference Use the introduction and worked motivation to understand optional-stopping failure. Reproducing the theory is outside this gate.
- Probabilistic Machine Learning: An Introduction Use the April 18, 2025 online edition: Sections 4.2.3 to 4.2.5, 4.5.1 and 4.6.4.1 for Bernoulli and Gaussian MLE/MAP; the definition and central intervals in 4.6.6; 2.1.2 and 11.7.4 for predictive uncertainty. Exercises 4.1 and 4.2 supply derivation practice.
- Permutation nulls and false discovery control Read
permutation_testNotes, Unpaired statistics, for exchangeability and exact partitions;false_discovery_controlNotes and the named thrombolysis example for Bonferroni and Benjamini-Hochberg. Use as an independent implementation oracle, not as a substitute for stating assumptions. - Probability calibration: the statistical property Read the opening calibration definition, the proper-scoring-rule note, and Section 1.16.1, Calibration curves. Stop before recalibration methods; the later Classical ML unit implements them.