Section 6 of 8 · Deep Learning

Generative Modeling

The core skill is distinguishing objective families, sampling paths and evaluation limits. One compact representative should be implemented well instead of reproducing every family.

7 units 16 topics 0/7

Units

7 in this section

6.1 Autoencoders and latent-variable models
explain · derive
3 h
6.2 The ELBO and reparameterized sampling
derive · implement
4 h
6.3 Posterior collapse and decoder capacity
measure
3 h
6.4 GAN objectives, alternating updates and mode collapse
derive · explain
4 h
6.5 Diffusion: forward corruption, reverse denoising and prediction targets
derive
5 h
6.6 Diffusion sampling schedules, cost and guidance
explain
3 h
6.7 Evaluating and comparing generative model families
explain
3 h

Before moving on

Checkpoint

Derive and implement a compact VAE objective and sampler, expose posterior or mode collapse in a controlled run, and explain what the loss, reconstructions and samples cannot prove.

Deliberately out of scope

Leave for later

Implementing every generative family
Large diffusion reproduction
Domain-specific generative evaluation suites

Resources for this section

Shared by every unit above

Understanding Deep Learning - generative models
Simon J. D. Princetheory

primary

Read Chapter 14 for evaluation limits, 15 for GANs, 17 for VAE and ELBO, and 18 for diffusion. Read Chapter 16 only for flow recognition. Use notebooks 17.1-17.3 and 18.1-18.4 as mathematical and visual practice.
D2L - GAN and DCGAN
D2L authorsimplementation

support

Use gan.md and dcgan.md for one bounded adversarial exposure. The required implementation remains a small VAE with separate reconstruction and KL terms on a two-dimensional mixture or binarized MNIST subset.
Deep-ML candidate - VAE, GAN and diffusion problems
Deep-MLpractice

optional

After the platform admission check, use at most one of 393 VAE Loss, 1384 Reparameterization, 174 1D GAN or 303 Forward Diffusion as a changed-prompt drill.
Stanford CS231n - diffusion assignment slice
Stanford Universitycourse

optional

Use the Assignment 3 diffusion component only after the VAE gate if diffusion mechanics remain weak. It is not a second required generative implementation.
Progressive Distillation for Fast Sampling of Diffusion Models
arXivreference

reference

Read Section 4 only for the velocity parameterization \(v = \alpha \epsilon - \sigma x\) and why it stays stable at zero signal-to-noise ratio. Leave Algorithms 1 and 2 and the distillation results for later.

Apply the section

Mini-project · Compact VAE with a controlled collapse study (8 to 12 h)

Derive and implement a small VAE, then run a controlled study that produces posterior collapse on purpose, measures it per latent dimension and repairs it. Compare with a small GAN on a 2D mixture, and end with what the loss, reconstructions and samples cannot prove.

Steps

Derive the ELBO for a diagonal Gaussian encoder, including the closed-form KL per dimension, and implement it with the reparameterization trick. Pass Deep-ML 393, and test your KL against torch.distributions.kl_divergence and a Monte Carlo estimate.
Train a small VAE on binarized MNIST, or on a 2D mixture if you only have a CPU, and log the reconstruction term, the total KL and the KL of each latent dimension every epoch.
Produce collapse on purpose with a much stronger decoder or a large KL weight such as \(\beta = 10\). Show most dimensions with KL near zero, and a resampling test: decoding fresh prior samples gives nearly the same output.
Ablate decoder capacity or \(\beta\) at three settings, and add KL annealing from 0 to 1 over a stated number of steps. Report active units, the dimensions whose KL exceeds a threshold you state, for each run.
Report the test ELBO in nats per example beside reconstructions and samples, and give one way each of the three could mislead.
Train a small GAN on an eight-Gaussian 2D mixture with alternating updates, and count the modes its samples cover with a fixed sample size and distance threshold, over training and two seeds.

Deliverables

The VAE code with ELBO tests: Deep-ML 393, the KL checks and the resampling test.
The collapse study: per-dimension KL plots, active-unit counts, the ablation and the annealing run, plus the GAN mode-coverage plot.
A limits note on what the loss, reconstructions and samples cannot prove, citing your own runs.

Checks

external-tests · Deep-ML 393: Implement VAE Loss (ELBO)
rubric · Self-grade with the rubric below

Rubric

Criterion	Pass	Strong
Objective	Deep-ML 393 passes, and the closed-form KL matches `torch.distributions` within float tolerance.	The Monte Carlo KL estimate agrees within its standard error, and the note derives the per-dimension KL.
Controlled collapse	One run shows most dimensions with near-zero KL, and the resampling test confirms the decoder ignores \(z\).	The ablation reports active units across two seeds and explains any setting that breaks the trend.
Repair	Annealing or a capacity change restores active units under the stated threshold.	You report the repair's cost in ELBO and reconstruction, not only its benefit.
Mode coverage	GAN coverage of the eight modes is counted with a stated threshold and sample size.	Coverage is tracked over training and across two seeds, and drops are tied to the alternating updates.
Honest limits	The note gives one way each of loss, reconstructions and samples can mislead.	Each claim points to a run in your study that shows it.

Feeds into: Capstone artifact 2 can use this VAE as its trained model, and the collapse study can be a casebook postmortem.