Unit 6.3 · about 3 h
209 / 324Posterior collapse and decoder capacity
Collapse is the characteristic VAE failure: the loss looks fine while the latent code carries nothing. The checkpoint requires you to produce it on purpose and then bound it.
- measure
You can now
- Detect posterior collapse from per-dimension KL and from resampling tests on a small VAE.
- Explain why a high-capacity decoder lets the ELBO keep reconstruction while the KL goes to zero.
- Apply KL annealing and decoder-capacity bounds and state what each intervention cannot prove.
Covers
- Posterior collapse and decoder capacity
Study, in this order
2 steps
- Posterior collapse and the information preference noteUnderstanding Deep Learning - generative models
Notes: the Posterior collapse paragraph and the Chen et al. paragraph (p. 346)
- Disentanglement and the \(\beta\)-weighted lossUnderstanding Deep Learning - generative models
Section 17.8.4 with Equation 17.30 (pp. 342 to 343)
Depth · Equation 17.30 with \(\beta\) above 1 is still a lower bound, only a looser one. The Notes sentence about gradually increasing the KL term describes annealing: the KL weight starts near 0 and rises to 1.
Focus on
- Collapse means the encoder outputs the prior for every input: per-dimension KL sits near zero and resampling \(z\) leaves reconstructions and samples unchanged.
- Chen et al. observed the mechanism: with a PixelCNN-class likelihood the decoder models \(x\) alone, so the ELBO keeps reconstruction while the KL term pays for nothing and is driven to zero.
- \(\beta\) above 1 in Eq 17.30 pushes \(q\) harder toward the prior and raises collapse risk. Annealing starts the KL weight at 0 and raises it to 1 so the code is used before it is taxed.
- The book names VQ-VAE style discrete latents and total-correlation penalties as further directions. Treat them as pointers, not as claims you verified here.
Reproduce closed-book
Train a tiny VAE on a 2D mixture twice: once with KL weight 1 and once with a large weight such as \(\beta = 10\) (or a decoder that also sees \(x\)). Log per-dimension KL each epoch. In the collapsed run, show that KL sits near zero and that decoding fresh prior samples gives nearly the same output. Then add annealing from 0 to 1 to the weight-1 run and state the schedule you used.
Traps
- Reading a falling total loss as learning. In collapse the reconstruction term improves while the KL term dies; only the split plot shows it.
- Raising the KL weight to fix blurry samples. A larger final weight pressures \(q\) toward the prior and makes collapse more likely, not less.
Practice
The section mini-project covers this: a controlled collapse run with per-dimension KL logs, a decoder-capacity ablation, KL annealing from 0 to 1, and a resampling test.
Self-test
-
What is posterior collapse in one observable sentence?
The encoder predicts the prior for every input: per-dimension KL is near zero and resampling \(z\) does not change reconstructions or samples. -
Why does a powerful autoregressive decoder invite collapse?
It can model \(x\) without \(z\), so the ELBO keeps the reconstruction term while the KL term is pure cost. Optimization then sets \(q\) to the prior. -
What does \(\beta\) above 1 in Equation 17.30 do to collapse risk?
It raises the KL penalty, which pushes \(q\) toward the prior and makes collapse more likely. The objective stays a lower bound, only a looser one. -
Name two measurements that expose collapse.
Per-dimension KL near zero across epochs, and a resampling test where fresh \(z\) with a fixed decoder leaves outputs unchanged while the loss barely moves.
Completion criterion
Stop when
Your split KL plot shows collapse on demand, annealing restores nonzero KL, and the resampling test passes with the schedule written down.
Mark passed, then open 6.4 Only mark this when you can do it without notes.Leads to
- 6.7 Evaluating and comparing generative model families A collapsed model is the case study for why loss and samples prove different things.
- 6.4 GAN objectives, alternating updates and mode collapse Compare latent-ignoring collapse with GAN mode collapse.
Comes after
- 6.2 The ELBO and reparameterized samplingWatch the KL term collapse to zero during training.
- 6.4 GAN objectives, alternating updates and mode collapseCompare GAN mode collapse with VAE posterior collapse.
Resources for this unit
- Understanding Deep Learning - generative models Read Chapter 14 for evaluation limits, 15 for GANs, 17 for VAE and ELBO, and 18 for diffusion. Read Chapter 16 only for flow recognition. Use notebooks 17.1-17.3 and 18.1-18.4 as mathematical and visual practice.
Also in this section
- D2L - GAN and DCGAN Use gan.md and dcgan.md for one bounded adversarial exposure. The required implementation remains a small VAE with separate reconstruction and KL terms on a two-dimensional mixture or binarized MNIST subset.
- Deep-ML candidate - VAE, GAN and diffusion problems After the platform admission check, use at most one of 393 VAE Loss, 1384 Reparameterization, 174 1D GAN or 303 Forward Diffusion as a changed-prompt drill.
- Stanford CS231n - diffusion assignment slice Use the Assignment 3 diffusion component only after the VAE gate if diffusion mechanics remain weak. It is not a second required generative implementation.
- Progressive Distillation for Fast Sampling of Diffusion Models Read Section 4 only for the velocity parameterization \(v = \alpha \epsilon - \sigma x\) and why it stays stable at zero signal-to-noise ratio. Leave Algorithms 1 and 2 and the distillation results for later.