Unit 3.2 · about 3 h
66 / 324Joint, marginal and conditional distributions, and independence
Conditioning is the operation behind every Bayes update, every missing-data marginalization and every independence assumption in a graphical model.
- derive
You can now
- Explain joint, marginal and conditional distributions and apply the chain rule to a new finite model.
- Distinguish independence, conditional independence and uncorrelatedness with a counterexample for each confusion.
- Derive a Bayes-rule conditional flip and verify it by simulation with a stated seed.
Covers
- Joint, marginal and conditional distributions
- Independence and conditional independence
Study, in this order
4 steps
- Conditional probability, Bayes rule and independenceHarvard Stat 110 - Probability
Lectures 4 and 5: conditional probability, Bayes rule, law of total probability and conditional independence
- Joint, conditional and marginal distributionsHarvard Stat 110 - Probability
Lecture 19: joint, conditional and marginal distributions, 2-D LOTUS and the chicken-egg example
- Joint, marginal and conditional drillsStat 110 Strategic Practice
Strategic Practice Q1-5: ball in sphere, uniforms, Rock-Paper-Scissors Multinomial, chicken-egg; Homework Q1-7
- Chain rule and conditional independenceProbabilistic Machine Learning: An Introduction
Chapter 2, Sections 2.1.3.4-2.1.3.6 and 2.2.3-2.2.4 (pp. 35, 38-40): conditional probability, product and chain rules, independence and conditional independence
Focus on
- \(P(A \mid B)=P(A,B)/P(B)\) restricts the sample space to \(B\). Write the restricted space before computing.
- The chain rule factors any joint distribution. An independence claim drops a conditioning bar and must be justified, never assumed silently.
- Conditioning can reverse a comparison, as Simpson examples show. Report the conditioning set with every rate.
- Monty Hall rewards writing the host protocol as a conditional distribution, not as unconditional door odds.
Reproduce closed-book
float64: numpy.random.default_rng(11), a (300000, 2) array from integers(0, 2) with 1 coding a boy; the conditional fraction is 0.33357 over 225364 conditioned pairs, with standard error 0.00099.Traps
- Swapping \(P(A \mid B)\) with \(P(B \mid A)\). Write both conditionals with their different denominators.
- Reading pairwise independence as mutual independence. A three-coin Bernstein-style construction breaks it by hand.
- Confusing conditional with marginal independence in either direction. The chess problem in Strategic Practice 2 (Q5, given the opponent's skill level) separates the two.
Practice
Stat 110 Strategic Practice 7: Joint, Marginal and Conditional Q1-5
Strategic Practice Q1-5 with the inline solutions; Homework Q2-4 for extra conditional drills
Attempt each problem before reading its solution. The solutions are printed in the same file.
Stat 110 Strategic Practice 3: Continuing with Conditioning Q1-2
Strategic Practice Q1-2: Monty Hall with a biased door and conditional-independence statements
Attempt before reading the solutions printed in the same file.
Self-test
-
Derive Bayes rule from the definition of conditional probability and name the step that needs \(P(B)>0\).
\(P(A \mid B)=P(A,B)/P(B)\) and \(P(B \mid A)=P(A,B)/P(A)\); eliminating \(P(A,B)\) gives \(P(A \mid B)=P(B \mid A)P(A)/P(B)\). Division needs \(P(B)>0\), and expanding \(P(B)\) by total probability needs a partition of the space. -
Two fair coins. Are {first is heads} and {at least one head} independent?
No. \(P(\text{first H})=1/2\) and \(P(\text{at least one H})=3/4\), so independence needs \(P(\text{both})=3/8\). The intersection is {first is heads} itself, {HH, HT}, with probability 1/2. -
Give variables that are conditionally independent given \(Z\) but marginally dependent.
Two symptoms given disease status: given \(Z\) they are independent by the modeling assumption, but unconditionally they correlate through the unobserved common cause \(Z\). -
In Reproduce, why is the answer 1/3 and not 1/2?
Conditioning on at least one boy leaves {BB, BG, GB}, three equally likely outcomes. The 1/2 answer conditions on a different event, namely that a specified child is a boy.
Completion criterion
Stop when
You derive the 1/3 and a Bayes flip without notes, your rerun of the seed-11 simulation matches 0.33357 to 3 decimals with its standard error, and you can exhibit a pairwise-but-not-mutual independence example.
Mark passed, then open 3.3 Only mark this when you can do it without notes.Leads to
- 3.3 Expectation, variance, covariance and conditioning laws Take expectations of the conditionals built here.
- 2.7 Naive Bayes: event models, smoothing and log-space scoring Naive Bayes is this unit's conditional-independence assumption applied to features.
Comes after
- 3.1 Probability axioms and random variablesCondition on the events defined here.
Resources for this unit
- Harvard Stat 110 - Probability Follow lectures 4-14, 17-23 and 25-30 with matching Strategic Practice. Use lectures 1-3 only for prerequisite repair.
- Stat 110 Strategic Practice Solve the matching sets before marking a probability topic Practised; use solutions to audit reasoning, not to begin it.
- Probabilistic Machine Learning: An Introduction Consult targeted parts of Chapters 2, 3, 4 and 6 for ML notation, Monte Carlo, factorization, posterior prediction and information theory.
Also in this section
- MIT 18.05 - Bayesian bridge Use classes 10-16 for MLE, Bayesian updating, posterior prediction, conjugacy, prior choice and credible intervals.
- D2L - Information Theory Use §§22.11.1-22.11.5 for runnable entropy, cross-entropy, KL and mutual-information examples and exercises.