Unit 3.7 · about 3 h
71 / 324Entropy, cross-entropy, KL divergence and mutual information
Classification losses, perplexity and variational objectives are cross-entropies and KL terms in disguise. Misreading their direction or units silently flips what the optimizer rewards.
- derive
You can now
- Explain entropy, cross-entropy, KL divergence and mutual information in both probabilistic and coding terms.
- Derive cross-entropy as entropy plus KL divergence and identify which term changes during model fitting.
- Prove \(\mathrm{KL} \ge 0\) and implement stable discrete entropy, cross-entropy, KL and mutual-information estimates.
Covers
- Entropy, cross-entropy, KL divergence and mutual information
Study, in this order
3 steps
- Self-information, entropy and mutual informationD2L - Information Theory
Sections 22.11.1-22.11.3 with the PyTorch tab: card-deck bits, entropy, joint and conditional entropy, mutual information
- KL divergence and cross-entropy as a lossD2L - Information Theory
Sections 22.11.4-22.11.5 with the PyTorch tab: KL definition and properties, cross-entropy identity \(\mathrm{CE}=H+\mathrm{KL}\), binary and multiclass loss
Depth · Run the PyTorch tab, not the MXNet or TensorFlow tab: the TensorFlow conditional-entropy and joint-entropy cells return a vector instead of a scalar, because its
nansumreduces over the last axis only. Treat Section 22.11.4.3 as a symmetry demo only: sorting two sample vectors and scoring them as distributions is not a valid KL estimate. - Entropy, KL non-negativity and mutual informationProbabilistic Machine Learning: An Introduction
Chapter 6, Sections 6.1-6.2 and 6.3.1-6.3.4 (pp. 207-220): entropy and chain rules, Theorem 6.2.1 (\(\mathrm{KL} \ge 0\) by Jensen), KL versus maximum likelihood, mutual information identities
Focus on
- \(\mathrm{CE}(P, Q)=H(P)+\mathrm{KL}(P\|Q)\). Fitting \(Q\) moves only the KL term; \(H(P)\) is constant for fixed data.
- KL direction matters: \(\mathrm{KL}(P\|Q)\) punishes \(Q\) near zero where \(P\) is positive, even infinite for a hard zero. Averaging the wrong direction answers a different question.
- Log base sets the units: base 2 gives bits, base e gives nats, and 1 nat = 1.4427 bits. ProbML Chapter 6 and the D2L entropy helpers use base 2; the D2L cross-entropy loss and PyTorch losses use base e. Convert before comparing.
- Mutual information \(I(X;Y)=H(X)-H(X \mid Y)\) is zero exactly under independence. Correlation only sees linear dependence; MI sees any.
Reproduce closed-book
float64 using log2: \(H=1.5000\), \(\mathrm{KL}=0.0850\), \(\mathrm{CE}=1.5850\), with agreement of \(\mathrm{CE}\) and \(H+\mathrm{KL}\) to 1e-12 in absolute value.Traps
- Calling KL a distance. It is asymmetric and violates the triangle inequality: for the Reproduce pair, \(\mathrm{KL}(P\|Q)=0.0850\) bits but \(\mathrm{KL}(Q\|P)=0.0817\) bits.
- Taking
abs()around a KL implementation, as the D2L helper does. A negative KL signals a bug (unnormalized inputs or swapped logs);abshides it. - Assuming differential entropy is non-negative. Continuous entropy can be negative; only the discrete form \(H \ge 0\) holds.
Practice
D2L Information Theory Exercises 1, 2 and 4
Section 22.11.7: verify the card-deck entropies, prove \(\mathrm{KL} \ge 0\) by Jensen, and derive \(I(X;Y)=H(X)-H(X \mid Y)\)
Run the PyTorch tab for the entropy helpers. Skip the TensorFlow tab per the depth note above.
Stat 110 Penultimate Homework Q5: surprise and entropy
Penultimate Homework Q5 (in the Strategic Practice 10 file): surprise function and the uniform-maximizes-entropy proof by Jensen
Attempt before reading the solution printed in the same file.
Self-test
-
Prove \(\mathrm{KL}(P\|Q) \ge 0\) for discrete distributions and name the step that uses each assumption.
\(\mathrm{KL}=\mathbb{E}_P[-\log(q/p)] \ge -\log(\mathbb{E}_P[q/p])\) by Jensen for the convex \(-\log\); the expectation step needs \(p\) a valid PMF. \(\mathbb{E}_P[q/p]\) = sum over support of \(P\) of \(q \le 1\), with equality iff supports match; \(-\log\) of that is \(\ge 0\), using \(q\) a valid PMF. Equality needs \(q=p\) on the support of \(P\). -
In Reproduce, which term changes if \(Q\) is fitted to \(P\), and what is its minimum?
Only \(\mathrm{KL}(P\|Q)\) changes; \(H(P)=1.5\) is fixed by the data. Its minimum is 0, attained at \(Q=P\), where CE equals \(H(P)\). -
\(Q\) assigns 0 to an outcome with \(P=0.25\). What happens to \(\mathrm{KL}(P\|Q)\) and to a Monte Carlo CE estimate?
KL is infinite: the term \(0.25\log(0.25/0)\) diverges. A finite sample avoiding that outcome still estimates finite CE, which hides the infinite population KL. Support mismatch is a modeling error, not a sampling artifact. -
\(X\) and \(Y\) are independent bits. What are \(H(X \mid Y)\) and \(I(X;Y)\)?
\(H(X \mid Y)=H(X)\) since conditioning on \(Y\) removes nothing; \(I(X;Y)=H(X)-H(X \mid Y)=0\). Independence zeroes mutual information in both directions.
Completion criterion
Stop when
You recompute the 1.5, 0.0850 and 1.5850 without notes, prove \(\mathrm{KL} \ge 0\) naming the Jensen and support steps, and can predict which KL direction explodes for a given support mismatch.
Mark passed, then open 4.1 Only mark this when you can do it without notes.Leads to
- 4.1 Likelihood and perplexity, and their limits Perplexity is exponentiated cross-entropy on held-out text.
- 6.2 The ELBO and reparameterized sampling The ELBO trades a reconstruction term against a KL regularizer.
Resources for this unit
- Probabilistic Machine Learning: An Introduction Consult targeted parts of Chapters 2, 3, 4 and 6 for ML notation, Monte Carlo, factorization, posterior prediction and information theory.
- D2L - Information Theory Use §§22.11.1-22.11.5 for runnable entropy, cross-entropy, KL and mutual-information examples and exercises.
Also in this section
- Harvard Stat 110 - Probability Follow lectures 4-14, 17-23 and 25-30 with matching Strategic Practice. Use lectures 1-3 only for prerequisite repair.
- Stat 110 Strategic Practice Solve the matching sets before marking a probability topic Practised; use solutions to audit reasoning, not to begin it.
- MIT 18.05 - Bayesian bridge Use classes 10-16 for MLE, Bayesian updating, posterior prediction, conjugacy, prior choice and credible intervals.