Unit 4.2 · about 5 h
73 / 324Maximum likelihood and MAP estimation
Likelihood connects probability models to fitting objectives. MAP adds a prior, but a posterior mode is neither a posterior mean nor a complete uncertainty statement.
- derive
- implement
You can now
- Derive Bernoulli, categorical and univariate Gaussian MLEs, including boundary cases.
- Derive Bernoulli MAP with a beta prior and Gaussian-mean MAP with known observation variance and a normal prior.
- Implement the objectives and use a separate grid search to check the closed forms and prior sensitivity.
Covers
- Likelihood, maximum likelihood and MAP estimation
Study, in this order
2 steps
- 14.310x Lecture 12: likelihood as a function of parametersMIT 14.310x - Data Analysis for Social Scientists
Slides 33 to 53, maximum likelihood and the \(\operatorname{Uniform}[0, \theta]\) example
Depth · Use strictly increasing transformations such as log to preserve a maximizer. Stop before the second uniform example and the broad MLE-properties summary.
- Murphy: simple likelihood and posterior derivationsProbabilistic Machine Learning: An Introduction
April 18, 2025 online edition: Sections 4.2.3 to 4.2.5 (pp. 110 to 112), 4.5.1 (pp. 121 to 122), and 4.6.4.1 (pp. 141 to 143)
Depth · Use the univariate Gaussian-Gaussian case with known observation variance. Skip multivariate covariance priors and empirical Bayes.
Focus on
- For independent Bernoulli observations with \(k\) successes in \(n\) trials, \(\log L = k\log p + (n - k)\log(1 - p)\). The interior stationary point is \(k/n\), with negative second derivative; all-zero or all-one samples maximize at a boundary.
- Categorical MLE maximizes \(\sum N_k\log p_k\) on the probability simplex. A Lagrange multiplier gives \(p_k = N_k/n\); zero-count categories need boundary treatment.
- For iid Gaussian data, differentiate in \(\mu\) and \(v = \sigma^2 > 0\) to get \(\mu = \bar{X}\) and \(v = \sum (X_i - \bar{X})^2/n\). A zero residual sum of squares has no finite maximum with positive variance.
- A \(\operatorname{Beta}(a, b)\) prior adds \(a - 1\) and \(b - 1\) to the log-likelihood exponents. The interior MAP formula needs both posterior shape parameters greater than one. Gaussian-mean MAP combines prior precision \(1/\tau^2\) and data precision \(n/\sigma^2\); that posterior is Gaussian, so its mode equals its mean, unlike the \(\operatorname{Beta}(5, 3)\) example (2/3 versus 5/8).
Reproduce closed-book
[1, 0, 1, 1] and \(\operatorname{Beta}(2, 2)\), obtain MLE 3/4, posterior \(\operatorname{Beta}(5, 3)\), MAP 2/3 and posterior mean 5/8. With Gaussian data [1, 2, 3], obtain joint MLE \(\mu = 2\), \(v = 2/3\). In a separate known-variance model with \(\sigma^2 = 1\) and prior \(\mu \sim \mathcal{N}(0, 1)\), obtain MAP 1.5. Check in NumPy float64 without random draws: g = numpy.linspace(0.0001, 0.9999, 9999); maximize 3*log(g) + log1p(-g), then 4*log(g) + 2*log1p(-g). Next h = numpy.linspace(-1, 4, 5001); maximize -0.5*sum((y - h)**2) over each grid value, then subtract 0.5*h**2 for MAP. Grid optima are 0.7500, 0.6667, 2.000 and 1.500. The infinity-norm difference from the four analytic optima is 3.34e-5 or less; accept below 1e-4. Widen the Gaussian prior variance and verify that its MAP approaches 2.Traps
- Normalizing likelihood over parameters as though it were already a posterior. A posterior needs a prior and its normalizing integral.
- Using \(n - 1\) in a Gaussian variance MLE. That is an unbiased estimator, not the likelihood maximizer.
- Confusing beta-prior MAP pseudo-counts \(a - 1\), \(b - 1\) with posterior-mean pseudo-counts \(a\), \(b\).
Practice
Murphy Exercises 4.1 and 4.2: Gaussian MLE and MAP
Section 4.8, p. 164 in the April 18, 2025 online edition: Exercise 4.1, MLE for the univariate Gaussian; Exercise 4.2, MAP estimation for 1D Gaussians, parts a to d
Attempt closed-book before comparing with Sections 4.2.5 and 4.6.4.1. The public solutions list only headings for these two exercises, so use the independent grid check as well. In 4.2(a), take the derivative rather than merely stating the answer; check \(n\) growing and prior variance tending to zero or infinity.
Self-test
-
Why does a stationary-point calculation fail for all-one Bernoulli data?
The log likelihood is \(n\log p\) and is increasing on \((0, 1)\). Its maximum on \([0, 1]\) is \(p = 1\). Differentiation alone misses that boundary. -
Derive the Gaussian-mean MAP and identify the assumptions used.
Independence with common known \(\sigma^2\) makes the log likelihood \(-\sum (x_i - \mu)^2/(2\sigma^2)\) plus a constant. A normal prior adds \(-(\mu - m_0)^2/(2\tau^2)\), with \(\tau^2 > 0\). Setting the derivative to zero gives \((\sum x_i/\sigma^2 + m_0/\tau^2)/(n/\sigma^2 + 1/\tau^2)\). Positive precisions make the second derivative negative, so this is the unique mode. -
Does a stronger prior always improve a held-out prediction?
No. It reduces movement away from the prior mean but can introduce bias when the prior is poorly matched. Evaluate predictions on independent data and report sensitivity; optimizing the posterior objective is not a generalization guarantee.
Completion criterion
Stop when
You derive the estimators without copying, pass the independent grid check, solve Exercises 4.1 and 4.2, and explain the Bernoulli and Gaussian boundary failures.
Mark passed, then open 4.3 Only mark this when you can do it without notes.Leads to
- 2.7 Naive Bayes: event models, smoothing and log-space scoring Apply count likelihoods and distinguish smoothing conventions.
- 2.2 Conditioning, basis expansions, ridge, lasso and elastic net Connect Gaussian-prior MAP to ridge regression.
Resources for this unit
- MIT 14.310x - Data Analysis for Social Scientists Follow lectures 11-15, 17-20 and 22 with matching slides and problem sets for inference, regression, causality and experiment design.
- Probabilistic Machine Learning: An Introduction Use the April 18, 2025 online edition: Sections 4.2.3 to 4.2.5, 4.5.1 and 4.6.4.1 for Bernoulli and Gaussian MLE/MAP; the definition and central intervals in 4.6.6; 2.1.2 and 11.7.4 for predictive uncertainty. Exercises 4.1 and 4.2 supply derivation practice.
Also in this section
- MIT 18.05 - Inference practice Use classes 17-20, 22-24 and 26 with their problem sets for tests, intervals, bootstrap and regression.
- Causal Inference: What If Read Chapters 1-3 and 6-8 for causal effects, DAGs, confounding, selection bias and identification reasoning.
- Patterns of trustworthy experimentation Read after the course to connect health checks, guardrails and statistical interpretation to production launch decisions.
- Always Valid Inference Use the introduction and worked motivation to understand optional-stopping failure. Reproducing the theory is outside this gate.
- Permutation nulls and false discovery control Read
permutation_testNotes, Unpaired statistics, for exchangeability and exact partitions;false_discovery_controlNotes and the named thrombolysis example for Bonferroni and Benjamini-Hochberg. Use as an independent implementation oracle, not as a substitute for stating assumptions. - Probability calibration: the statistical property Read the opening calibration definition, the proper-scoring-rule note, and Section 1.16.1, Calibration curves. Stop before recalibration methods; the later Classical ML unit implements them.