gan-training-stability-requires-multiple-interventions
IN derived (depth 1)
Created 2026-06-21T10:23:12+00:00 · Reviewed 2026-06-21T15:37:01+00:00
GAN training stability benefits from several complementary design choices — non-saturating loss to address vanishing gradients when the generator is poor, two-timescale update rule for provable convergence to a stationary local Nash equilibrium (though not mode collapse prevention), and deterministic discriminators for optimality of the discriminator — each addressing a distinct failure mode, but none individually sufficient for overall stability.
Justifications
SL — three independent stability mechanisms each address a different failure mode — removing any one destabilizes training
Antecedents (all must be IN):
- IN gan-non-saturating-loss-original-paper — The non-saturating generator loss (L_G = E[ln D(x)]) was recommended in the original GAN paper for faster convergence, addressing vanishing gradients when the generator is poor
- IN gan-ttur-convergence — The Two Time-Scale Update Rule (TTUR) sets the generator's learning rate lower than the discriminator's and provably converges to a stationary local Nash equilibrium, but does not prevent mode collapse
- IN gan-optimal-discriminator-formula — The optimal discriminator for a fixed generator is D*(x) = μ_ref(x) / (μ_ref(x) + μ_G(x)), which is the Radon-Nikodym derivative dμ_ref/d(μ_ref + μ_G)
Dependents
These beliefs depend on this one:
- IN gan-implicit-nature-explains-training-difficulty — GANs' implicit generative nature (no explicit likelihood function) is plausibly connected to their need for multiple complementary training interventions — without a tractable objective to optimize directly, training stability relies on several distinct design choices (non-saturating loss, TTUR, deterministic discriminators) each addressing a different failure mode, which may partly substitute for the more direct optimization signal that explicit-likelihood models enjoy.
- OUT gan-training-stabilizable-given-complete-theory — GAN training would be reliably stabilizable given the complete theoretical characterization (optimal discriminator, JSD minimization, unique equilibrium) and multiple complementary stabilization interventions (non-saturating loss, TTUR, architecture choices) — unless the game-theoretic foundations themselves are limited, with Nash equilibria not guaranteed in general and equilibrium equivalence holding only for the original formulation.
- IN svm-gan-methodology-maturity-contrast — SVMs and GANs illustrate contrasting degrees of methodology codification in ML — SVMs have an unusually prescriptive practical recipe (standardize, default to RBF, grid-search C and gamma), while GAN training stability requires multiple complementary but individually insufficient interventions (non-saturating loss, two-timescale updates, deterministic discriminators) addressing distinct failure modes, suggesting that well-understood convex optimization enables more codified practice than implicit generative modeling with competing failure modes.