Unit 3.3 · about 4 h
67 / 324Expectation, variance, covariance and conditioning laws
Mixture models, hierarchical models and unbiased estimators all rest on iterating expectations correctly. Eve's law is also the variance accounting behind every Monte Carlo error bar later.
- derive
You can now
- Explain expectation, variance, covariance and correlation, and why zero correlation need not imply independence.
- Derive total expectation and total variance for a new two-stage model and state every assumption used.
- Compute conditional expectations by taking out what is known and verify the result by simulation.
Covers
- Expectation, variance, covariance and correlation
- Total probability, expectation and variance
- Conditional expectation
Study, in this order
4 steps
- Linearity, indicators and symmetryHarvard Stat 110 - Probability
Lectures 9 and 10: indicator variables, linearity of expectation and the fundamental bridge
- Covariance, correlation and conditional expectationHarvard Stat 110 - Probability
Lectures 21 and 25 to 27: variance of a sum, covariance, taking out what is known, Adam's law and Eve's law
- Covariance and conditional-expectation drillsStat 110 Strategic Practice
Strategic Practice set 10, Conditional Expectation and Variance Q1-5: Eve's law, residual example and the spam-arrival model
- Moments and the total lawsProbabilistic Machine Learning: An Introduction
Chapter 2, Section 2.2.5 (pp. 40-43): mean, linearity, variance, law of total expectation and law of total variance with the mixture example
Focus on
- Linearity of expectation needs no independence. The variance of a sum needs uncorrelatedness; independence is sufficient, not necessary.
- Eve's law splits variance into explained plus residual: \(\operatorname{Var}(Y)=\operatorname{Var}(\mathbb{E}[Y \mid X])+\mathbb{E}[\operatorname{Var}(Y \mid X)]\). Name which term each modeling choice changes.
- Taking out what is known, \(\mathbb{E}[g(X)Y \mid X]=g(X)\mathbb{E}[Y \mid X]\), needs \(g(X)\) fixed given \(X\). Conditioning on the wrong variable breaks it.
- Zero correlation with clear dependence is one construction away: take \(X \sim \operatorname{Normal}(0, 1)\) and \(Y=X^2\), then \(\operatorname{Cov}(X, Y)=\mathbb{E}[X^3]=0\) by symmetry.
Reproduce closed-book
float64: numpy.random.default_rng(21), n = 400000, branch I from integers(0, 2) and then the die from integers(1, 7), with I = 0 taking the die value and I = 1 taking 10; sample mean 6.7493 and variance 12.023 with standard error of the mean 0.00548.Traps
- Pushing a nonlinear function inside an expectation. \(\mathbb{E}[g(X)]\) differs from \(g(\mathbb{E}[X])\); Jensen fixes the direction for convex \(g\).
- Dropping the covariance term in \(\operatorname{Var}(X+Y)\) without checking uncorrelatedness first.
- Quoting a conditional expectation as a number. \(\mathbb{E}[Y \mid X]\) is a function of \(X\); only \(\mathbb{E}[Y \mid X=x]\) is a number.
Practice
Stat 110 Strategic Practice 10: Conditional Expectation and Variance Q1-5
Strategic Practice Q1-5 with the inline solutions; Penultimate Homework Q1 (same file) for a random-count total with mean, variance and MGF
Attempt each problem before reading its solution. The solutions are printed in the same file.
Stat 110 Strategic Practice 8: Covariance and Correlation Q1-7
Strategic Practice Q1-7: dice sums versus differences, Poisson thinning correlation and Multinomial covariances
Attempt before reading the solutions printed in the same file.
Self-test
-
Prove Eve's law from the definition of variance, naming where each assumption is used.
Write \(\operatorname{Var}(Y)=\mathbb{E}[Y^2]-\mathbb{E}[Y]^2\), then condition: \(\mathbb{E}[Y^2]=\mathbb{E}[\mathbb{E}[Y^2 \mid X]]\) by Adam's law, which needs \(\mathbb{E}|Y^2|\) finite. Subtract \(\mathbb{E}[\mathbb{E}[Y \mid X]]^2\) after writing \(\mathbb{E}[Y]=\mathbb{E}[\mathbb{E}[Y \mid X]]\) by the same law, and regroup into \(\mathbb{E}[\operatorname{Var}(Y \mid X)]+\operatorname{Var}(\mathbb{E}[Y \mid X])\). Both steps use iterated expectation, hence integrability. -
\(X \sim \operatorname{Normal}(0, 1)\) and \(Y=X^2\). Show \(\operatorname{Cov}(X, Y)=0\) while \(X\) and \(Y\) are dependent. Why is symmetry alone not enough?
\(\mathbb{E}[XY]=\mathbb{E}[X^3]=0\) by symmetry and \(\mathbb{E}[X]=0\), so \(\operatorname{Cov}=0\). Dependence: \(P(Y>1 \mid X>1)=1\) while \(P(Y>1)=0.3173\). Symmetry alone fails for \(X=+1\) or \(-1\) with probability 1/2 each: then \(Y=1\) is constant and independent of \(X\). -
In Reproduce, which part of 12.0208 comes from within-branch variation and which from between-branch variation?
\(35/24=1.4583\) is \(\mathbb{E}[\operatorname{Var}(X \mid \text{branch})]\), the within part, all of it from the die since the constant branch has zero variance. \((10-3.5)^2/4=10.5625\) is \(\operatorname{Var}(\mathbb{E}[X \mid \text{branch}])\), the between part. -
Why does the reproduce check quote 0.00548 alongside the mean?
It is the simulation standard error, \(\mathrm{sd}/\sqrt{n}\). Without it the comparison of 6.7493 against 6.75 has no scale; agreement means within a few standard errors.
Completion criterion
Stop when
You derive Eve's law and the 12.0208 without notes, your rerun of the seed-21 simulation matches the mean to 2 decimals with its standard error, and you can construct a zero-correlation dependent pair on demand.
Mark passed, then open 3.4 Only mark this when you can do it without notes.Leads to
- 3.6 Laws of large numbers, CLT, concentration and Monte Carlo Feed these moments into concentration bounds and Monte Carlo standard errors.
- 4.5 Gaussian mixtures and EM EM iterates a conditional expectation of the complete-data log likelihood.
Comes after
- 3.2 Joint, marginal and conditional distributions, and independenceTake expectations of the conditionals built here.
Resources for this unit
- Harvard Stat 110 - Probability Follow lectures 4-14, 17-23 and 25-30 with matching Strategic Practice. Use lectures 1-3 only for prerequisite repair.
- Stat 110 Strategic Practice Solve the matching sets before marking a probability topic Practised; use solutions to audit reasoning, not to begin it.
- Probabilistic Machine Learning: An Introduction Consult targeted parts of Chapters 2, 3, 4 and 6 for ML notation, Monte Carlo, factorization, posterior prediction and information theory.
Also in this section
- MIT 18.05 - Bayesian bridge Use classes 10-16 for MLE, Bayesian updating, posterior prediction, conjugacy, prior choice and credible intervals.
- D2L - Information Theory Use §§22.11.1-22.11.5 for runnable entropy, cross-entropy, KL and mutual-information examples and exercises.