Unit 3.4 · about 4 h
68 / 324Common distribution families and transformations
Likelihoods, priors, noise models and initializations are all named distributions. Sampling them and pushing them through transforms is daily work in generative modeling and Bayesian computation.
- derive
- implement
You can now
- Explain the Bernoulli, Binomial, categorical, Gaussian, Exponential and Poisson families, with their means, variances and sampling stories.
- Derive a change-of-variables density with the Jacobian factor and state the support explicitly.
- Implement inverse-CDF sampling for a target density and test it against exact moments and tail probabilities.
Covers
- Bernoulli, binomial, categorical, Gaussian, exponential and Poisson families
- Transformations of random variables
Study, in this order
4 steps
- Named discrete and continuous familiesHarvard Stat 110 - Probability
Lectures 11 to 14 and 16: Poisson, Uniform, Normal, LOTUS and Exponential memorylessness
- MGFs, transformations and convolutionsHarvard Stat 110 - Probability
Lectures 17, 18 and 22: MGFs for Expo and Normal moments, sums of Poissons, transformations, LogNormal and convolutions
- Transformation drillsStat 110 Strategic Practice
Strategic Practice set 8, Transformations Q1-3: Uniform to Beta powers, Box-Muller and the convolution derivation
- Distribution families and change of variablesProbabilistic Machine Learning: An Introduction
Chapter 2, Sections 2.4-2.6 (pp. 49-61) for Bernoulli, categorical and Gaussian; Sections 2.8.3-2.8.5 (pp. 67-72) for bijections, the Jacobian factor and convolutions
Focus on
- Tie each family to a sampling story: counts of trials, waiting times, arrivals per interval. The story selects the family; the data never does by itself.
- A change of variables multiplies by the Jacobian absolute determinant and re-derives the support. A density without a support is unfinished.
- Sums of independent variables convolve densities; MGFs turn convolution into multiplication where the MGF exists near zero.
- Parametrization traps: Exponential and Gamma rate versus scale, Normal variance versus standard deviation. State yours in every signature.
Reproduce closed-book
float64: numpy.random.default_rng(31), 300000 draws via random() then -log; sample mean 0.9983 and variance 1.0024 against exact 1 and 1, and \(P(X>2)=0.1341\) against exact \(e^{-2}=0.1353\) with standard error 0.00062.Traps
- Dropping the Jacobian factor or keeping the old support after a transform. Both silently produce a function that does not integrate to 1.
- Assuming every distribution has an MGF. Cauchy has none; quote the MGF only with its domain.
- Confusing Poisson counts with Exponential waiting times. They describe one process from two sides; the units differ.
Practice
Stat 110 Strategic Practice 8: Transformations Q1-3
Strategic Practice set 8, Transformations Q1-3 with the inline solutions; Homework 8 Q4-6 for extra density derivations
Attempt each problem before reading its solution. The solutions are printed in the same file.
Self-test
-
Derive the density of \(Y=X^2\) for \(X \sim \operatorname{Uniform}(-1, 1)\), naming the non-monotone step.
\(F_Y(y)=P(-\sqrt{y} \le X \le \sqrt{y})=\sqrt{y}\) on \((0,1)\), using the two-sided preimage; differentiating gives \(f_Y(y)=1/(2\sqrt{y})\) on \((0,1)\). The monotone CDF rule applied to one branch alone would halve the answer. -
Why does Box-Muller produce two independent Normals from two Uniforms?
The polar map \((U_1, U_2) \to (R\cos, R\sin)\) with \(R^2=-2\log U_1\) has Jacobian factor that splits the joint density into a product of two standard Normal densities. Independence is read off the factored form. -
\(X \sim \operatorname{Pois}(2)\) counts arrivals per hour and \(T \sim \operatorname{Expo}(2)\) is the wait for the next arrival. Are \(X\) and \(T\) the same object?
No. \(X\) is discrete with mean 2; \(T\) is continuous with mean 1/2 hour. Same rate parameter, different sample spaces and different questions. -
In Reproduce, why is
random()with -log enough, and what would break withintegers(0, 2)?Inverse-CDF needs a continuous \(\operatorname{Uniform}(0,1)\) input so that \(P(U \ge u)=1-u\); the proof step uses continuity. A discrete input would quantize \(X\) and the CDF derivation would fail.random()can also return exactly 0.0 (probability \(2^{-53}\) per draw), so library samplers use-log1p(-U)on the same input.
Completion criterion
Stop when
You derive the \(-\log(U)\) result and one non-monotone transform without notes, and your rerun of the seed-31 check matches the mean, variance and tail to 3 decimals with the stated standard error.
Mark passed, then open 3.5 Only mark this when you can do it without notes.Leads to
- 3.5 Bayes rule, conjugate priors and posterior predictives Use Beta, Binomial and Gaussian families as priors and likelihoods.
- 2.6 LDA, QDA and generative versus discriminative classifiers LDA and QDA fit Gaussian class-conditional densities.
Resources for this unit
- Harvard Stat 110 - Probability Follow lectures 4-14, 17-23 and 25-30 with matching Strategic Practice. Use lectures 1-3 only for prerequisite repair.
- Stat 110 Strategic Practice Solve the matching sets before marking a probability topic Practised; use solutions to audit reasoning, not to begin it.
- Probabilistic Machine Learning: An Introduction Consult targeted parts of Chapters 2, 3, 4 and 6 for ML notation, Monte Carlo, factorization, posterior prediction and information theory.
Also in this section
- MIT 18.05 - Bayesian bridge Use classes 10-16 for MLE, Bayesian updating, posterior prediction, conjugacy, prior choice and credible intervals.
- D2L - Information Theory Use §§22.11.1-22.11.5 for runnable entropy, cross-entropy, KL and mutual-information examples and exercises.