Unit 3.5 · about 4 h
69 / 324Bayes rule, conjugate priors and posterior predictives
Posterior prediction is the honest form of a probabilistic forecast: it carries parameter uncertainty into the prediction instead of plugging in a point estimate.
- derive
- implement
You can now
- Explain prior, likelihood, posterior, evidence, posterior predictive and credible interval, and how each differs from its frequentist lookalike.
- Derive a Beta-Binomial conjugate posterior and its posterior predictive distribution, stating every modeling assumption.
- Factor a small joint distribution from a stated graph and run a prior-sensitivity check by repeated simulation.
Covers
- Bayes rule, likelihood, evidence and posterior
- Priors, conjugacy, posterior predictive distributions and sensitivity
- Simple graphical factorization and latent variables
Study, in this order
4 steps
- Bayesian updating with discrete priors and predictionMIT 18.05 - Bayesian bridge
Class 11 reading: Bayesian Updating with Discrete Priors, with its Bayes tables; Class 12a reading: Probabilistic Prediction
- Continuous priors, conjugate priors and prior choiceMIT 18.05 - Bayesian bridge
Class 13a reading: Bayesian Updating with Continuous Priors; Class 15 reading: Conjugate Priors, Beta and Normal; Class 16a and 16b readings: Choosing Priors and Probability Intervals
- Hybrid Bayes rule and Beta-Binomial updatingHarvard Stat 110 - Probability
Lecture 17: hybrid Bayes rule and Laplace succession; Lecture 23: Beta distribution and Bayes billiards
- Beta-Binomial posterior and posterior predictiveProbabilistic Machine Learning: An Introduction
Chapter 4, Sections 4.6.1-4.6.2.10 (pp. 129-136): conjugate priors, Beta-Binomial posterior, MAP, posterior predictive and marginal likelihood
Depth · Stop before Section 4.6.2.11 (mixtures of conjugate priors). The Dirichlet and Gaussian models and the computation methods of Section 4.6.8 belong to later stages.
Focus on
- Conjugacy is algebra: a Beta prior times a Binomial likelihood is Beta up to a constant. The update adds successes to \(a\) and failures to \(b\).
- The posterior predictive averages over the posterior; the plug-in forecast conditions on one estimate. They differ most with little data.
- The evidence normalizes the posterior for fixed data and compares models across hypotheses. It never rescales a single posterior's shape.
- A graphical factorization is a chain-rule product with dropped conditioning bars. Each dropped bar is a conditional-independence claim to defend.
Reproduce closed-book
float64: numpy.random.default_rng(41), 300000 beta(9, 5) draws; posterior mean 0.6430 with standard error 0.00023, one binomial(1, theta) flip per draw from the same generator gives predictive fraction 0.6439, and the 2.5 and 97.5 percent sample quantiles give the 95 percent credible interval [0.385, 0.862] (exact \(\operatorname{Beta}(9, 5)\) quantiles 0.386 and 0.861).Traps
- Reporting the MAP as the posterior mean. For a \(\operatorname{Beta}(a, b)\) posterior they coincide only when \(a=b\); here 0.6667 differs from 0.6429.
- Reading a 95 percent credible interval as a 95 percent confidence interval. One conditions on the observed data with a prior; the other is a pre-data coverage statement.
- Reusing the same data to pick the prior and to update. That double counts the evidence; run a prior-sensitivity check instead.
Practice
MIT 18.05 Problem Set 7: Problems 2 and 6
Problem 2 (Prediction: prior, posterior and posterior predictive) and Problem 6 (Normal-Normal conjugate updating); official solutions are linked beside the set with the same problem numbers
Attempt both before opening the solutions. Problem 2 is the closest match to this unit's implement target.
Deep-ML 336: Compute Posterior Probability using Bayes Theorem
Implement bayes_theorem(priors, likelihoods); the two tests check the normalized prior-times-likelihood posterior to 4 decimals
The function is a discrete Bayes table. Check it against your hand table for Problem Set 7, Problem 2(a), then contrast it with the conjugate update, which needs no table.
Self-test
-
Derive the \(\operatorname{Beta}(9, 5)\) posterior from Bayes rule and name the step that uses conjugacy.
Posterior is proportional to \(\theta^7(1-\theta)^3\) times \(\theta^1(1-\theta)^1=\theta^8(1-\theta)^4\). Recognizing this kernel as \(\operatorname{Beta}(9, 5)\) up to its normalizing constant is the conjugacy step; it uses the Beta integral identity. -
Derive the posterior predictive 9/14 and state the assumption that lets the integral close.
\(P(\text{next head} \mid \text{data})\) = integral of \(\theta\) times \(\operatorname{Beta}(9, 5)\) = \(\mathbb{E}[\theta]=9/14\). The integral closes because the Bernoulli likelihood is linear in \(\theta\) and the posterior is Beta; exchangeability of the flips justifies reusing the same \(\theta\). -
How does the predictive probability move if the prior changes from \(\operatorname{Beta}(2, 2)\) to \(\operatorname{Beta}(20, 20)\) with the same 7-in-10 data?
The posterior becomes \(\operatorname{Beta}(27, 23)\) with mean \(27/50=0.54\), pulled from 0.6429 toward the prior mean 0.5. Sensitivity shrinks as data grow: the prior contributes a fixed pseudo-count (40 here) against \(n\) real trials. -
A two-node graph \(X \to Y\) claims \(p(x, y)=p(x)p(y \mid x)\). What does the missing bar encode?
Nothing is dropped here: it is the chain rule. A claim appears only with more parents or missing edges, for example \(p(y \mid x, z)=p(y \mid x)\), which asserts \(Y\) is independent of \(Z\) given \(X\).
Completion criterion
Stop when
You derive the \(\operatorname{Beta}(9, 5)\) posterior, its mean, MAP and predictive without notes, your rerun of the seed-41 check matches the posterior mean to 3 decimals, and you can show how a stronger prior moves the predictive.
Mark passed, then open 3.6 Only mark this when you can do it without notes.Leads to
- 5.4 Calibration curves and Platt or isotonic recalibration Calibrate the posterior probabilities produced here before trusting them.
- 4.4 Uncertainty, robustness and honest claims Carry the posterior-versus-confidence distinction into model uncertainty claims.
Comes after
- 3.4 Common distribution families and transformationsUse Beta, Binomial and Gaussian families as priors and likelihoods.
Resources for this unit
- Harvard Stat 110 - Probability Follow lectures 4-14, 17-23 and 25-30 with matching Strategic Practice. Use lectures 1-3 only for prerequisite repair.
- MIT 18.05 - Bayesian bridge Use classes 10-16 for MLE, Bayesian updating, posterior prediction, conjugacy, prior choice and credible intervals.
- Probabilistic Machine Learning: An Introduction Consult targeted parts of Chapters 2, 3, 4 and 6 for ML notation, Monte Carlo, factorization, posterior prediction and information theory.
Also in this section
- Stat 110 Strategic Practice Solve the matching sets before marking a probability topic Practised; use solutions to audit reasoning, not to begin it.
- D2L - Information Theory Use §§22.11.1-22.11.5 for runnable entropy, cross-entropy, KL and mutual-information examples and exercises.