Paper Reading
This paper delves into the optimization landscapes of Generative Adversarial Networks (GANs) through a detailed analysis of rotation techniques, local Nash equilibrium, and locally stable stationary points. It investigates various optimization solutions and tools used in GANs, providing insights into the dynamics of training GAN models. The study also examines the discriminator behaviors in different GAN variants, shedding light on the complex interactions within these networks.
Download Presentation
Please find below an Image/Link to download the presentation.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.If you encounter any issues during the download, it is possible that the publisher has removed the file from their server.
You are allowed to download the files provided on this website for personal or commercial use, subject to the condition that they are used lawfully. All files are the property of their respective owners.
The content on the website is provided AS IS for your information and personal use only. It may not be sold, licensed, or shared on other websites without obtaining consent from the author.
E N D
Presentation Transcript
Paper Reading ICLR2020 : A Closer Look at the Optimization Landscapes of Generative Adversarial Networks Presented by Dachao Lin
Outline Rotation in GANs optimization LNE and LSSP optimization solution of GAN Visualization tool Experiments 2
Rotation in GANs optimization Discriminator: Vanilla GAN WGAN Vanilla GAN Direc GAN Real sample at x=0, generator parameter , generate data at x= . 3
Rotation in GANs optimization Direc GAN ?2+ ?2=1 Rotation Graident flow 4
Rotation in GANs optimization WGAN WGAN-GP 5
Local Nash Equilibrium (LNE) Local Nash equilibrium (LNE), i.e. a point only locally true. Being a DNE is not necessary for being a LNE: a local Nash equilibrium may have Hessians that are only semi-definite. NE are commonly used in GANs to describe the goal of the learning procedure. The interaction between the two networks is not taken into account. 6
Locally Stable Stationary Point (LSSP) Intuition: ? ?(? ) ? ? ? 7
Path-angle for NSGAN (top row) and WGAN-GP (bottom row) 12
Eigenvalues of the Jacobian of the game for NSGAN (top row) and WGAN-GP (bottom row) 13
Summary The rotational component is clearly visible. The complex eigenvalues for NSGAN seems to be much more concentrated on the imaginary axis ??? WGAN-GP tends to spread the eigenvalues towards the right of the imaginary axis ???
Summary The generator never reaches a local minimum but instead finds a saddle point. The algorithm converges to a LSSP which is not a LNE. NSGAN converges to a solution with very large positive eigenvalues compared to WGAN-GP. The gradient penalty acts as a regularizer on the discriminator and prevents it from becoming too sharp.
Discussion GANs do not converge to local Nash Equilibria. The optimization landscapes of GANs typically have rotational components. Whether we need a Nash equilibrium to get a generator with good performance in GANs? Handle strong rotational components: extragradient, averaging, gradient penalty based
