Unit 6.6 · about 3 h
212 / 324Diffusion sampling schedules, cost and guidance
A trained denoiser is not yet a generator. Sampling cost and conditioning decide whether the model is usable, and both are set after training.
- explain
You can now
- Explain why ancestral sampling needs about 1000 serial U-Net evals and what DDIM changes.
- Compare step-count versus quality trade-offs including the 50-step DDIM point and cascades.
- Distinguish classifier guidance from classifier-free guidance and predict the stereotyping effect of overweighting.
Covers
- Schedules, iterative sampling and compute-quality trade-offs
- Conditioning and guidance at recognition depth
Study, in this order
2 steps
- Implementation: U-Net, time embedding, DDIM and cascadesUnderstanding Deep Learning - generative models
Section 18.6 with Algorithms 18.1 and 18.2 and Notebooks 18.3 and 18.4 (pp. 363 to 368)
- Conditioning and guidanceUnderstanding Deep Learning - generative models
Sections 18.6.3 and 18.6.4 with Equation 18.41 on classifier guidance and the classifier-free paragraph (pp. 365 to 368)
Depth · Read for schedule, step-count and guidance mechanics at recognition depth. Do not train a large image model here.
Focus on
- One U-Net maps \(z_t\) to an \(\epsilon\) estimate at one noise level, conditioned on a sinusoidal time embedding. Ancestral sampling (Algorithm 18.2) calls it serially about 1000 times per image, so cost scales with steps, not with parameters alone.
- DDIM is a deterministic integrator of the probability-flow ODE: it drops per-step noise and gives good samples in about 50 steps on the book example. The Notes add latent-space diffusion as the further speedup used by Stable Diffusion.
- Classifier guidance adds \(\sigma_t^2\) times \(\operatorname{grad} \log \Pr(c \mid z_t)\) from a separate noisy classifier (Eq 18.41). Classifier-free guidance trains \(g(z_t, \phi_t, c)\) with randomly dropped labels and mixes conditional and unconditional scores at sample time.
- Overweighting the conditioning improves apparent quality but produces stereotyped outputs. The book draws the same trade as GAN truncation: less variety for cleaner exemplars.
Reproduce closed-book
Traps
- Calling DDIM faster per step. The gain is fewer steps with a different integrator, not a cheaper network; one DDIM step costs about the same as one ancestral step.
- Reading guidance scale as likelihood tempering. It reweights the conditional score at sample time and changes coverage; it does not retrain the density.
- Comparing step counts across different noise schedules. Fifty steps under one schedule is not the same compute-quality point as fifty under another; always fix the schedule first.
Practice
Stanford CS231n Assignment 3 Q3: DDPM notebook (conditional)
Q3 Denoising Diffusion Probabilistic Models, DDPM.ipynb in the starter code (https://cs231n.github.io/assignments/2026/assignment3.zip); attempt the forward-process and loss parts only, after the VAE gate
Use it as a mechanics check if diffusion sampling still feels vague. It is not a second required generative implementation; stop before large-scale image training.
Self-test
-
Why is diffusion sampling slow, in one line of arithmetic?
Each reverse step needs one full U-Net eval and good ancestral images need about 1000 serial steps, so one sample costs about 1000 network passes. -
What does DDIM change?
It integrates the probability-flow ODE deterministically without per-step noise, so the book gets good samples in about 50 steps. It is still slower than most GAN sampling. -
Contrast classifier guidance with classifier-free guidance.
Classifier guidance adds the gradient of a separately trained noisy classifier (Eq 18.41). Classifier-free guidance folds the class into the denoiser, drops it at random during training, and mixes conditional and unconditional predictions at sample time. -
What happens when conditioning is over-weighted?
Higher apparent quality but stereotyped, low-variety samples. UDL notes the same quality-variety trade as GAN truncation.
Completion criterion
Stop when
You can cost a sampler in U-Net evals, explain the 1000 versus 50 step points, and state which guidance form a config uses from its sampling code.
Mark passed, then open 6.7 Only mark this when you can do it without notes.Leads to
- 6.7 Evaluating and comparing generative model families Judge sampling shortcuts by coverage and cost, not by cherry-picked images.
Comes after
- 6.5 Diffusion: forward corruption, reverse denoising and prediction targetsTurn the trained denoiser into a sampler with a schedule and guidance.
Resources for this unit
- Understanding Deep Learning - generative models Read Chapter 14 for evaluation limits, 15 for GANs, 17 for VAE and ELBO, and 18 for diffusion. Read Chapter 16 only for flow recognition. Use notebooks 17.1-17.3 and 18.1-18.4 as mathematical and visual practice.
Also in this section
- D2L - GAN and DCGAN Use gan.md and dcgan.md for one bounded adversarial exposure. The required implementation remains a small VAE with separate reconstruction and KL terms on a two-dimensional mixture or binarized MNIST subset.
- Deep-ML candidate - VAE, GAN and diffusion problems After the platform admission check, use at most one of 393 VAE Loss, 1384 Reparameterization, 174 1D GAN or 303 Forward Diffusion as a changed-prompt drill.
- Stanford CS231n - diffusion assignment slice Use the Assignment 3 diffusion component only after the VAE gate if diffusion mechanics remain weak. It is not a second required generative implementation.
- Progressive Distillation for Fast Sampling of Diffusion Models Read Section 4 only for the velocity parameterization \(v = \alpha \epsilon - \sigma x\) and why it stays stable at zero signal-to-noise ratio. Leave Algorithms 1 and 2 and the distillation results for later.