Unit 6.1 · about 3 h
207 / 324Autoencoders and latent-variable models
Every later unit reuses the same split: a prior over \(z\), a decoder likelihood, and an approximate posterior. If this split is vague, the ELBO looks like bookkeeping instead of a bound.
- explain
- derive
You can now
- Derive a mixture-of-Gaussians marginal and state the prior, likelihood and marginal of the nonlinear latent variable model.
- Explain why the VAE is a training architecture while the learned model is the nonlinear latent variable model.
- Contrast a plain autoencoder bottleneck with a probabilistic latent model that defines \(\Pr(x)\).
Covers
- Autoencoder reconstruction objectives
- Latent variables, priors and approximate posteriors
Study, in this order
2 steps
- Latent variable models and the nonlinear generatorUnderstanding Deep Learning - generative models
Sections 17.1 and 17.2, with Equations 17.1 to 17.7 (pp. 327 to 330)
Depth · Stop before Section 17.3. Training and the ELBO belong to the next unit.
- Notebook 17.1: latent variable modelsUnderstanding Deep Learning - generative models
The notebook that accompanies Sections 17.1 and 17.2, margin-noted beside Equation 17.7
Focus on
- The marginal \(\Pr(x) = \int \Pr(x \mid z) \Pr(z) \, dz\) sums over every latent cause. The mixture example (Eq 17.4) is the discrete case; the nonlinear model (Eq 17.7) replaces the table with a network \(f[z, \phi]\).
- The prior is fixed to \(\mathcal{N}(0, I)\) (Eq 17.5) while the likelihood \(\mathcal{N}(f[z, \phi], \sigma^2 I)\) carries the parameters (Eq 17.6). Generation samples \(z\) first and never touches the encoder.
- The chapter introduction warns that calling the VAE the model of \(\Pr(x)\) misleads. The final model is a nonlinear latent variable model with no variational and no autoencoder parts.
- A plain autoencoder minimizes reconstruction through a bottleneck and defines no distribution. The probabilistic model adds prior, likelihood and posterior, which is what makes a likelihood bound possible.
Reproduce closed-book
Traps
- Reading the encoder as part of the generative model. Generation draws \(z\) from the prior and decodes; the encoder only proposes \(q\) during training.
- Treating a low reconstruction error as a likelihood claim. Autoencoders have no \(\Pr(x)\), so their error cannot be compared with ELBO or flow likelihood numbers.
Practice
UDL Problems 17.1 to 17.3: MoG parameters, log concavity and Jensen
Problems 17.1 to 17.3 (p. 348): count mixture parameters with ranges, prove log is concave, prove the second-moment bound
Do these on paper before the ELBO unit. Problem 17.2 is the concave-log fact that Jensen needs in Section 17.3.
Self-test
-
A 1D mixture of Gaussians has 5 components. How many raw parameters does it need, and how many are free?
15 raw numbers: 5 means, 5 variances and 5 weights. Weights lie in [0,1] and sum to 1, which removes one degree of freedom, so 14 are free. Means are real and variances are positive. -
Why does the book say the VAE is an architecture, not the model of \(\Pr(x)\)?
The learned model is the nonlinear latent variable model: prior \(\mathcal{N}(0, I)\) plus decoder likelihood. The encoder and the sampling step only help optimize the bound and are absent from the final \(\Pr(x)\). -
How does a plain autoencoder differ from the VAE bottleneck?
An autoencoder learns a low-dimensional code that reconstructs well and defines no distribution. A VAE learns a prior, a likelihood and an approximate posterior, so it defines \(\Pr(x)\) and a lower bound on its \(\log\) likelihood. -
How do you generate from the Section 17.2 model?
Draw \(z\) from \(\mathcal{N}(0, I)\), compute \(f[z, \phi]\), then draw \(x\) from \(\mathcal{N}(f[z, \phi], \sigma^2 I)\). The encoder plays no role.
Completion criterion
Stop when
You can write the MoG marginal and Eqs 17.5 to 17.7 without notes and explain which parts survive into the final \(\Pr(x)\).
Mark passed, then open 6.2 Only mark this when you can do it without notes.Leads to
- 6.2 The ELBO and reparameterized sampling Turn the prior and likelihood into the ELBO and a trainable sampler.
- 6.7 Evaluating and comparing generative model families Keep the autoencoder versus probabilistic distinction for the family comparison.
Resources for this unit
- Understanding Deep Learning - generative models Read Chapter 14 for evaluation limits, 15 for GANs, 17 for VAE and ELBO, and 18 for diffusion. Read Chapter 16 only for flow recognition. Use notebooks 17.1-17.3 and 18.1-18.4 as mathematical and visual practice.
Also in this section
- D2L - GAN and DCGAN Use gan.md and dcgan.md for one bounded adversarial exposure. The required implementation remains a small VAE with separate reconstruction and KL terms on a two-dimensional mixture or binarized MNIST subset.
- Deep-ML candidate - VAE, GAN and diffusion problems After the platform admission check, use at most one of 393 VAE Loss, 1384 Reparameterization, 174 1D GAN or 303 Forward Diffusion as a changed-prompt drill.
- Stanford CS231n - diffusion assignment slice Use the Assignment 3 diffusion component only after the VAE gate if diffusion mechanics remain weak. It is not a second required generative implementation.
- Progressive Distillation for Fast Sampling of Diffusion Models Read Section 4 only for the velocity parameterization \(v = \alpha \epsilon - \sigma x\) and why it stays stable at zero signal-to-noise ratio. Leave Algorithms 1 and 2 and the distillation results for later.