solving mode collapse with autoencoder ganselarosca.net/slides/iccv_autoencoder_gans.pdf ·...
TRANSCRIPT
![Page 1: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/1.jpg)
Thanks to: Balaji Lakshminarayanan, David Warde-Farley, Shakir Mohamed
Solving mode collapse with Autoencoder GANs
Mihaela Rosca
![Page 2: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/2.jpg)
![Page 3: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/3.jpg)
![Page 4: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/4.jpg)
![Page 5: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/5.jpg)
Autoencoders
L1/L2 reconstruction loss
Image Credit: mikicon, the Noun Project
code
![Page 6: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/6.jpg)
Adversarial autoencoders
Improve reconstruction quality by adding a GAN loss.
Adversarial AutoencodersA. Makhzani, J. Shlens, N.Jaitly, I. Goodfellow, B. Frey
![Page 7: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/7.jpg)
By construction, autoencoders learn to cover the entire training data.
![Page 8: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/8.jpg)
Autoencoder GANs
Combine the reconstruction power of autoencoders with the sampling power of GANs!
![Page 9: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/9.jpg)
How to sample?
![Page 10: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/10.jpg)
Work on the code space - not data space.
![Page 11: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/11.jpg)
reconstructingImage Credit: mikicon, the Noun Project
1) Learning the code distributionAssumption
learning the code distribution is simpler than learning data distribution
sampling
learn p(codes)
![Page 12: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/12.jpg)
reconstructing sampling
2) Match the code distribution to a desired prior
match prior
![Page 13: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/13.jpg)
Working on the code space
learn
p(cod
es)
match prior
reconstructing sampling
![Page 14: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/14.jpg)
Learning the code distribution
![Page 15: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/15.jpg)
Learning the code distribution: PPGN
Plug and play generative modelsA. Nguyen, J. Clune, Y, Bengio, A. Dosovitskiy, J. Yosinski
![Page 16: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/16.jpg)
PPGN - key ingredients
● Reconstructions○ Autoencoder + GAN + Perceptual loss in feature space
● Samples○ Markov Chain○ Conditioning
![Page 17: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/17.jpg)
PPGN
L1/L2 reconstructionGAN lossPerceptual loss
Not depicted:● Activation Maximization● Encoder is pretrained classifier
![Page 18: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/18.jpg)
PPGN: sampling using a denoising autoencoder
Learn ∇p c) using a DAE, then sample using MCMC
![Page 19: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/19.jpg)
Limitations of PPGN
● Not end to end○ Need to pretrain an encoder on
the same dataset● Depends on labels for:
○ Conditioning samples○ Pretrained encoder
● Markov Chains ○ when to stop?○ missing rejection step
Learning the code distribution
![Page 20: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/20.jpg)
Matching a desired prior
![Page 21: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/21.jpg)
reconstructing sampling
2) Match the code distribution to a desired prior
match prior
![Page 22: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/22.jpg)
Sounds familiar?
![Page 23: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/23.jpg)
Variational inference - the ELBO
likelihood term KL term
good reconstructions match the prior
![Page 24: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/24.jpg)
reconstructing sampling
Variational autoencoders
computed analytically
computed analytically
match prior
![Page 25: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/25.jpg)
AlphaGAN GANsVAEs
AlphaGANcombining GANs and VAEs
● variational ● inference
○ reconstructions○ encoder network
● the posterior latent matches the prior
● implicit ○ encoder ○ decoder
● discriminators used to match distributions
Variational Approaches for Auto-Encoding Generative Adversarial NetworksM. Rosca, B. Lakshminarayanan, D. Warde-Farley, S. Mohamed
![Page 26: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/26.jpg)
reconstructing sampling
AlphaGAN
estimated from samplescodes discriminator
match prior
estimated from samplesdata discriminator
![Page 27: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/27.jpg)
Q: how do we estimate the terms in the ELBO using GANs?
![Page 28: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/28.jpg)
Density ratio trick
Estimate the ratio of two distributions only from samples, by building a binary classifier to distinguish between them.
![Page 29: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/29.jpg)
Using GANs for variational inference - the ELBO
![Page 30: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/30.jpg)
ELBO - likelihood term
![Page 31: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/31.jpg)
ELBO - the KL term
![Page 32: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/32.jpg)
From the ELBO to loss functionsWe want to match:● the reconstruction and data
distributions○ likelihood term
● the code and prior distributions○ the KL
Tools for matching distributions:● (GAN) the density ratio trick● (VAE) observer likelihoods
○ reconstructions losses
![Page 33: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/33.jpg)
reconstructing sampling
New loss functions - via the density ratio trick
match prior
GAN loss(make samples close to reconstructions)
Reconstruction loss(avoid mode collapse)
+GAN loss
(improve recon quality)
![Page 34: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/34.jpg)
![Page 35: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/35.jpg)
![Page 36: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/36.jpg)
Samples
![Page 37: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/37.jpg)
Cifar10 - Inception score
Improved Techniques for Training GANsT. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen
Classifier trained on Imagenet Classifier trained on Cifar10
![Page 38: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/38.jpg)
CelebA - sample diversity
Improved Techniques for Training GANsT. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen
(1 -
MS-
SSIM
)
![Page 39: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/39.jpg)
Matching the prior - AGE
● the encoder is the discriminator○ forces data codes to match the prior○ sample codes to not match the prior
It Takes (Only) Two: Adversarial Generator-Encoder NetworksD. Ulyanov, A. Vedaldi, V. Lempitsky
![Page 40: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/40.jpg)
Analysis of methods
![Page 41: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/41.jpg)
Try Autoencoder GANs if mode collapse is a problem.
Summary
Combining different learning principles results in a family of novel algorithms.
![Page 42: Solving mode collapse with Autoencoder GANselarosca.net/slides/iccv_autoencoder_gans.pdf · Variational Approaches for Auto-Encoding Generative Adversarial Networks M. Rosca, B. Lakshminarayanan,](https://reader031.vdocument.in/reader031/viewer/2022041022/5ed408d38d46b66d226352ac/html5/thumbnails/42.jpg)
ReferencesIt Takes (Only) Two: Adversarial Generator-Encoder NetworksD. Ulyanov, A. Vedaldi, V. Lempitsky
Improved Techniques for Training GANsT. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen
Variational Approaches for Auto-Encoding Generative Adversarial NetworksM. Rosca, B. Lakshminarayanan, D. Warde-Farley, S. Mohamed
Plug and play generative modelsA. Nguyen, J. Clune, Y, Bengio, A. Dosovitskiy, J. Yosinski
Adversarial AutoencodersA. Makhzani, J. Shlens, N.Jaitly, I. Goodfellow, B. Frey
Generative Adversarial NetworksI. Goodfellow, J. Pouget-Abadie, M. Mirza, B.Xu,D. Warde-Farley, S. Ozair, A.Courville, Y. Bengio
Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial NetworksL. Mescheder, S. Nowozin, A. Geiger
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial NetworksA. Radford, L. Metz, S. Chintala
Auto-Encoding Variational BayesD. P. Kingma, M. Welling