Unit 6.4 · about 4 h
210 / 324GAN objectives, alternating updates and mode collapse
GANs are the adversarial counterpoint to likelihood training: no density, a learned loss, and failures that look like good samples. The minimax argument explains why.
- derive
- explain
You can now
- Derive the minimax losses, the optimal discriminator, and the Jensen-Shannon form of the generator signal.
- Run alternating D-then-G updates with the non-saturating generator loss on a small GAN.
- Distinguish mode dropping from mode collapse and name the imbalance that causes each.
Covers
- GAN minimax objectives and alternating updates
- Generator-discriminator balance and mode collapse
Study, in this order
4 steps
- Minimax, optimal discriminator and the Jensen-Shannon splitUnderstanding Deep Learning - generative models
Sections 15.1 to 15.2.2 with Equations 15.2 to 15.9 (pp. 276 to 283)
- Wasserstein, DCGAN recipe and collapseUnderstanding Deep Learning - generative models
Sections 15.2.3 to 15.2.6 on Wasserstein (pp. 284 to 286); Section 15.1.3 DCGAN recipe and Section 15.1.4 with Figure 15.5 on mode dropping and collapse (pp. 279 to 281)
Depth · Skip Sections 15.3 to 15.6 here: progressive growing, conditional GANs, translation and StyleGAN are later material.
- Basic GAN on a 2D GaussianD2L - GAN and DCGAN
Generate Some Real Data, Generator, Discriminator and Training: 2D Gaussian with
A=[[1,2],[-0.1,0.5]]andb=[1,2], linear generator, 3-layer Tanh discriminator, batch size 8, \(D\) updated before \(G\) each iteration - DCGAN blocks and Pokemon trainingD2L - GAN and DCGAN
The Generator with
G_blockand the 100-dim latent to 64 by 64 map; Discriminator withD_blockand leaky ReLU 0.2; Training with Adambeta10.5
Focus on
- Equations 15.2 to 15.5 turn binary cross-entropy into a minimax game. The practical generator loss is the non-saturating \(-\log D(G(z))\) (label flipped to 1), because the \(\max \log(1 - D)\) form saturates when \(D\) is confident.
- Section 15.2.1 shows the optimal discriminator is \(\Pr(x)/[\Pr(x) + \Pr(\text{fake})]\), and the loss becomes a Jensen-Shannon divergence split into quality and coverage terms. The book notes that the real-data term of Eq 15.6 does not involve the generator, so nothing directly rewards it for covering missed modes; it calls this the putative reason for mode dropping.
- Section 15.1.4 separates the failures: mode dropping generates a realistic subset (no beards), while collapse maps most \(z\) to one or few points and ignores \(z\). Figure 15.5 and the Notes diversity paragraph make this concrete.
- The DCGAN recipe is strided convolutions,
BatchNormin both nets, leaky ReLU in \(D\) and Adam withbeta10.5. D2L updates \(D\) first each iteration with detached fakes so no gradient reaches \(G\) through the \(D\) step.
Reproduce closed-book
lr_D 0.05 and lr_G 0.005 and watch the discriminator loss move toward about 0.69 (\(\log 2\), the value when \(D\) outputs 0.5 everywhere) as the fake scatter covers the real Gaussian.Traps
- Forgetting
detachonfake_Xin the \(D\) step. The \(D\) loss then backpropagates into \(G\): wasted compute, and if \(G\)'s gradients are not zeroed before its own step, \(G\) also moves in the direction that helps \(D\). - Following the DCGAN summary line about
BatchNormliterally. The shippedD_blockappliesBatchNormin every block including the first, while the paper recipe uses no norm on the discriminator input. Implement the paper variant and note the difference. - Reading \(D\) accuracy 0.5 as proof of distributional match. A finite-capacity \(D\) reaches chance on a mode-dropped generator too; the Notes duplicate test shows support can be far smaller than the training set.
Practice
D2L basic GAN: alternating training on the 2D Gaussian
Training with update_D and update_G: batch size 8, latent dim 2, 20 epochs; confirm \(D\) runs before \(G\) and fakes use detach
Plot real versus fake scatter at epochs 1, 5 and 20, and add a discriminator-accuracy printout. State the epoch where accuracy first nears 0.5, and remember that chance accuracy does not prove the distributions match.
UDL Problem 15.4: divergences for shifted uniforms
Problem 15.4 (p. 303): KL, reverse KL, JS and Wasserstein for uniform \([0, 1]\) versus \([a, a+1]\) over \(a\) in \([-3, 3]\)
Do it on paper: KL is infinite for any nonzero \(a\), JS is \(|a| \log 2\) up to 1 and \(\log 2\) after, W1 is \(|a|\). This is the disjoint-support case Wasserstein was introduced for.
Self-test
-
Write the two alternating losses and the update order.
\(D\) minimizes BCE on real data with label 1 and detached fakes with label 0. \(G\) minimizes \(-\log D(G(z))\), the same BCE with fake labels flipped to 1. Each iteration updates \(D\) first, then \(G\). -
Why can the generator ignore coverage, according to Section 15.2.1?
In Equation 15.6 the real-data term does not involve \(G\), so for a fixed discriminator the generator's gradient comes only from its own samples. Generating a realistic subset is not penalized directly. The book offers this as the putative reason for mode dropping. -
Distinguish mode dropping from mode collapse.
Dropping covers a realistic subset of modes. Collapse maps most or all \(z\) to one or few points and ignores the latent input. -
For uniform \([0, 1]\) versus \([a, a+1]\) with \(a = 2\), give KL, JS and W1.
Both KL directions are infinite because the supports are disjoint. JS is \(\log 2\) (1 bit). W1 is \(|a| = 2\).
Completion criterion
Stop when
You derive the JS split without notes, your D2L run shows \(D\)-first updates with detached fakes, and you solve Problem 15.4 for \(a = 0\), 0.5 and 2.
Mark passed, then open 6.5 Only mark this when you can do it without notes.Leads to
- 6.7 Evaluating and comparing generative model families Good samples with dropped modes motivate precision-recall evaluation.
- 6.3 Posterior collapse and decoder capacity Compare GAN mode collapse with VAE posterior collapse.
Comes after
- 6.3 Posterior collapse and decoder capacityCompare latent-ignoring collapse with GAN mode collapse.
Resources for this unit
- Understanding Deep Learning - generative models Read Chapter 14 for evaluation limits, 15 for GANs, 17 for VAE and ELBO, and 18 for diffusion. Read Chapter 16 only for flow recognition. Use notebooks 17.1-17.3 and 18.1-18.4 as mathematical and visual practice.
- D2L - GAN and DCGAN Use gan.md and dcgan.md for one bounded adversarial exposure. The required implementation remains a small VAE with separate reconstruction and KL terms on a two-dimensional mixture or binarized MNIST subset.
Also in this section
- Deep-ML candidate - VAE, GAN and diffusion problems After the platform admission check, use at most one of 393 VAE Loss, 1384 Reparameterization, 174 1D GAN or 303 Forward Diffusion as a changed-prompt drill.
- Stanford CS231n - diffusion assignment slice Use the Assignment 3 diffusion component only after the VAE gate if diffusion mechanics remain weak. It is not a second required generative implementation.
- Progressive Distillation for Fast Sampling of Diffusion Models Read Section 4 only for the velocity parameterization \(v = \alpha \epsilon - \sigma x\) and why it stays stable at zero signal-to-noise ratio. Leave Algorithms 1 and 2 and the distillation results for later.