Unit 6.7 · about 3 h
213 / 324Evaluating and comparing generative model families
Every family optimizes a different proxy, and every metric rewards a different slice. Choosing or trusting a model without this map repeats the section checkpoint failure.
- explain
You can now
- State what likelihood, ELBO, reconstructions and samples each prove and what each hides.
- Compute Inception Score limits, FID behavior and precision-recall splits for a stated failure.
- Contrast flows, VAEs, GANs and diffusion by likelihood, sampling, coverage, stability and evaluation at recognition depth.
Covers
- Reconstruction, likelihood and sample-quality limits
- Autoregressive, latent, adversarial and diffusion comparisons
- Normalizing-flow likelihood contrast
Study, in this order
2 steps
- What makes a good model and how to quantify itUnderstanding Deep Learning - generative models
Sections 14.2 and 14.3 with Figure 14.3, Equations 14.1 to 14.3, and the test likelihood, inception, FID and precision-recall paragraphs (pp. 270 to 275)
- Flow likelihood contrast at recognition depthUnderstanding Deep Learning - generative models
Sections 16.1 to 16.2 with Equations 16.1 to 16.3, and Section 16.3.3 coupling Equations 16.13 and 16.14 (pp. 304 to 312)
Depth · Read flows only far enough to state the change-of-variables likelihood and why coupling keeps the Jacobian triangular. Leave residual, autoregressive and multiscale flow details for later.
Focus on
- Test likelihood needs a density: flows give it exactly (Eqs 16.1 to 16.3), VAE and diffusion give a lower bound, and GANs give nothing because they only sample. Training likelihood is not a generalization claim.
- Inception Score (Eqs 14.2 and 14.3) rewards one realistic example per ImageNet class under one classifier. FID fits Gaussians to Inception features of real and fake sets, so it sees within-class spread but only through retained features. Precision-recall splits the two: realism versus coverage via \(k\)-nearest hyperspheres.
- Figure 14.3 summarizes the trade: no family has efficient sampling, top quality, full coverage, a good latent space and exact likelihood at once. Quote the row, not a slogan.
- Coupling (Eqs 16.13 and 16.14) leaves half the variables fixed and applies an invertible elementwise map to the rest with params from the fixed half. The Jacobian is triangular, so the determinant is the diagonal product and both directions stay cheap.
Reproduce closed-book
Traps
- Comparing likelihood numbers across families or preprocessings. VAE and diffusion report bounds, flows report exact values, and any dequantization or resolution change breaks comparability.
- Reporting one FID from few samples or mismatched Inception weights. FID is a distribution distance under a Gaussian fit; small sample counts and different feature layers move it without any model change.
- Treating low FID as proof of coverage. A mode-dropped GAN can score well; only the recall side or a duplicate test like the Notes 400-sample argument addresses support size.
Practice
UDL Problems 16.1 and 16.2: 1D flow densities
Problems 16.1 and 16.2 (p. 325): uniform through \(x = z^2\) and standard normal through the sigmoid
Derive both densities by CDF differentiation and verify each integrates to 1. These are the smallest exact-likelihood claims in the section.
Self-test
-
Which families give exact likelihood, a lower bound, or nothing?
Flows give exact likelihood by change of variables. VAE and diffusion give a lower bound through the ELBO. GANs give no density; they sample without evaluating probabilities. -
State the 1D flow density for uniform \(z\) through \(x = z^2\).
\(\Pr(x) = \frac{1}{2 \sqrt{x}}\) on \((0, 1]\). The CDF is \(P(X \le x) = \sqrt{x}\); differentiate to get the density, which integrates to 1. -
Why is Inception Score insufficient, and what do FID and precision-recall add?
IS needs ImageNet classes and one classifier and rewards a single good example per class. FID compares Gaussian-fitted Inception features of both sets, capturing within-class spread through retained features. Precision-recall splits realism (fakes inside the real manifold) from coverage (real data inside the fake manifold). -
How does coupling keep flows tractable?
Half the variables pass through unchanged while the rest get an elementwise invertible map with parameters from the fixed half. The Jacobian is triangular, so its determinant is the product of diagonal terms and both directions stay cheap.
Completion criterion
Stop when
You can place any two families on likelihood, sampling cost, coverage and evaluation without notes and name the metric that would catch a mode-dropped winner.
Mark passed, then open 7.1 Only mark this when you can do it without notes.Comes after
- 6.1 Autoencoders and latent-variable modelsKeep the autoencoder versus probabilistic distinction for the family comparison.
- 6.3 Posterior collapse and decoder capacityA collapsed model is the case study for why loss and samples prove different things.
- 6.4 GAN objectives, alternating updates and mode collapseGood samples with dropped modes motivate precision-recall evaluation.
- 6.6 Diffusion sampling schedules, cost and guidanceJudge sampling shortcuts by coverage and cost, not by cherry-picked images.
Resources for this unit
- Understanding Deep Learning - generative models Read Chapter 14 for evaluation limits, 15 for GANs, 17 for VAE and ELBO, and 18 for diffusion. Read Chapter 16 only for flow recognition. Use notebooks 17.1-17.3 and 18.1-18.4 as mathematical and visual practice.
Also in this section
- D2L - GAN and DCGAN Use gan.md and dcgan.md for one bounded adversarial exposure. The required implementation remains a small VAE with separate reconstruction and KL terms on a two-dimensional mixture or binarized MNIST subset.
- Deep-ML candidate - VAE, GAN and diffusion problems After the platform admission check, use at most one of 393 VAE Loss, 1384 Reparameterization, 174 1D GAN or 303 Forward Diffusion as a changed-prompt drill.
- Stanford CS231n - diffusion assignment slice Use the Assignment 3 diffusion component only after the VAE gate if diffusion mechanics remain weak. It is not a second required generative implementation.
- Progressive Distillation for Fast Sampling of Diffusion Models Read Section 4 only for the velocity parameterization \(v = \alpha \epsilon - \sigma x\) and why it stays stable at zero signal-to-noise ratio. Leave Algorithms 1 and 2 and the distillation results for later.