Diffusion: forward corruption, reverse denoising and prediction targets

Diffusion training is ELBO arithmetic plus one Gaussian closure. Once the closed form and the noise-to-data map are routine, schedules and samplers become cost choices.

derive

You can now

Derive the one-step forward kernel, the closed-form \(x_0\)-to-\(x_t\) sampler and the reverse conditional used for training.
Derive the diffusion ELBO down to the per-step noise matching loss of Algorithm 18.1.
Recognize noise, data and velocity prediction targets and state when each stays stable.

Covers

Diffusion forward corruption and reverse denoising
Noise, data and velocity prediction recognition

Study, in this order

3 steps

Forward encoder and closed-form sampling
Understanding Deep Learning - generative models

Sections 18.1 and 18.2 with Equations 18.1 to 18.8 and Figures 18.2 and 18.3 (pp. 349 to 353)

Done
Reverse decoder, ELBO and the noise reparameterization
Understanding Deep Learning - generative models

Sections 18.3 to 18.5 with Equations 18.16 to 18.40 and Algorithm 18.1 (pp. 356 to 363)

Done
Velocity parameterization
Progressive Distillation for Fast Sampling of Diffusion Models

Section 4: the \(v = \alpha \epsilon - \sigma x\) definition with \(\hat{x} = \alpha z_t - \sigma \hat{v}\), and the low signal-to-noise stability argument

Depth · Read only the parameterization and loss-weighting discussion. Leave Algorithms 1 and 2 and all distillation results for later.

Done

Focus on

The forward kernel is \(q(z_t \mid z_{t-1}) = \mathcal{N}(\sqrt{1 - \beta_t} z_{t-1}, \beta_t I)\) (Eq 18.2). Products of Gaussians close in closed form: \(x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon\) with \(\bar{\alpha}_t\) the product of \((1 - \beta_s)\) (Eqs 18.7 and 18.8).
The reverse conditional \(q(z_{t-1} \mid z_t, x)\) (Eq 18.15) trains the decoder. The ELBO (Eq 18.25) becomes per-step KL terms that reduce to squared error between the true reverse mean and \(f_t\) (Eqs 18.28 and 18.29).
Section 18.5 converts data prediction into noise prediction through Eq 18.30 and trains \(g\) to output \(\epsilon\) with the plain loss of Eq 18.40. The map is linear, so the two targets carry the same information with different conditioning.
Velocity \(v = \alpha \epsilon - \sigma x\) keeps the implied \(x\) prediction stable when signal-to-noise goes to zero, where \(\epsilon\)-to-\(x\) division by \(\alpha\) blows up. UDL never names it, so the paper supplies this one recognition item.

Reproduce closed-book

Closed-book, derive the three-step closed form of Problem 18.3 from Equations 18.5 to 18.7, then verify with NumPy that \(x_0=[1,2]\) at \(t=3\) with \(\epsilon=[0.5,-0.5]\) on linspace(0.1,0.2,5) gives [1.1057,1.3488].

Traps

Scaling the mean by \(1 - \beta\) instead of \(\sqrt{1 - \beta}\). The square root is what keeps the variance at 1 when the input has unit variance (Problem 18.1); without it the signal shrinks faster than the noise is added.
Indexing \(\bar{\alpha}\) at \(t\) instead of \(t - 1\) in the reverse mean. The coefficient \((1 - \alpha_{t-1})/(1 - \alpha_t)\) in Eq 18.28 is sensitive to this shift; test \(t = 1\) separately.
Treating noise, data and velocity targets as different models. They parameterize the same reverse mean; only their numerical behavior at low signal-to-noise differs.

Practice

UDL Problems 18.1 to 18.3: variance preservation and the closed form

Problems 18.1 to 18.3 (pp. 372 to 373): covariance preservation, combining two noises, and the three-step expansion to Eq 18.45

Do all three on paper, then check the [1.2649] and [1.1057,1.3488] numbers with NumPy. These are the forward-process facts every sampler assumes.

Self-test

Write the one-step forward kernel and the direct \(x_0\)-to-\(x_t\) sampler.
With a linear \(\beta\) schedule linspace(0.1,0.2,5), what is \(x_1\) for x0=[1.0] and eps=[1.0]?
How are noise prediction and data prediction related?
State the velocity target and why it helps.

Completion criterion

Stop when

You derive Eq 18.8 without notes, both numeric checks match to 4 decimals, and you can convert a noise output into data and velocity forms.

Mark passed, then open 6.6 Only mark this when you can do it without notes.

Leads to

6.6 Diffusion sampling schedules, cost and guidance Turn the trained denoiser into a sampler with a schedule and guidance.

Comes after

6.2 The ELBO and reparameterized samplingThe diffusion loss is another ELBO with the same reconstruction-versus-prior shape.

Resources for this unit

Understanding Deep Learning - generative models
Simon J. D. Princeprimarytheory
Read Chapter 14 for evaluation limits, 15 for GANs, 17 for VAE and ELBO, and 18 for diffusion. Read Chapter 16 only for flow recognition. Use notebooks 17.1-17.3 and 18.1-18.4 as mathematical and visual practice.
Progressive Distillation for Fast Sampling of Diffusion Models
arXivreferencereference
Read Section 4 only for the velocity parameterization \(v = \alpha \epsilon - \sigma x\) and why it stays stable at zero signal-to-noise ratio. Leave Algorithms 1 and 2 and the distillation results for later.

Also in this section

D2L - GAN and DCGAN
D2L authorssupportimplementation
Use gan.md and dcgan.md for one bounded adversarial exposure. The required implementation remains a small VAE with separate reconstruction and KL terms on a two-dimensional mixture or binarized MNIST subset.
Deep-ML candidate - VAE, GAN and diffusion problems
Deep-MLoptionalpractice
After the platform admission check, use at most one of 393 VAE Loss, 1384 Reparameterization, 174 1D GAN or 303 Forward Diffusion as a changed-prompt drill.
Stanford CS231n - diffusion assignment slice
Stanford Universityoptionalcourse
Use the Assignment 3 diffusion component only after the VAE gate if diffusion mechanics remain weak. It is not a second required generative implementation.