Generative Adversarial Networks

Sahin Olut

Department of Computer EngineeringIstanbul Technical University

November 4, 2017

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 1 / 23

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 2 / 23

Motivation Why should we study generative models?

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 3 / 23

Motivation Why should we study generative models?

Motivation

We can restore the missing data by generative models. (Imageinpaiting, super-resolution)

We can generate new examples to enhance our classifier networks.(Data augmentation strategy)

If we want our computers to understand, we have to teach them tocreate. (I do not understand what I cannot create. – RichardFeynman)

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 4 / 23

Motivation Why should we study generative models?

Motivation

We can restore the missing data by generative models. (Imageinpaiting, super-resolution)

We can generate new examples to enhance our classifier networks.(Data augmentation strategy)

If we want our computers to understand, we have to teach them tocreate. (I do not understand what I cannot create. – RichardFeynman)

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 4 / 23

Motivation Why should we study generative models?

Motivation

We can restore the missing data by generative models. (Imageinpaiting, super-resolution)

We can generate new examples to enhance our classifier networks.(Data augmentation strategy)

If we want our computers to understand, we have to teach them tocreate. (I do not understand what I cannot create. – RichardFeynman)

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 4 / 23

Motivation Some results from recent GAN works

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 5 / 23

Motivation Some results from recent GAN works

Recent works

Photo-Realistic Single Image Super-Resolution Using a GenerativeAdversarial Network [Ledig et al., 2016]

Figure: Work by Ledig et al., 2016

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 6 / 23

Motivation Some results from recent GAN works

Recent works

Plug & Play Generative Networks: Conditional Iterative Generation ofImages in Latent Space [Nguyen et al., 2017]

Figure: Synthetic images generated from ImageNet classes.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 7 / 23

How does GAN work? GAN Architecture

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 8 / 23

How does GAN work? GAN Architecture

Discriminator and Generator Networks

What is a generative model?

Discriminator’s role in GAN is to predict whether the input isgenerated or sampled from training data.

The aim of generator is to capture the distribution of the trainingdata.

According to Goodfellow et al., it is a minimax game betweengenerator and discriminator where generator tries to fooldiscriminator.(There are some debates about it.)

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 9 / 23

How does GAN work? GAN Architecture

Discriminator and Generator Networks

What is a generative model?

Discriminator’s role in GAN is to predict whether the input isgenerated or sampled from training data.

The aim of generator is to capture the distribution of the trainingdata.

According to Goodfellow et al., it is a minimax game betweengenerator and discriminator where generator tries to fooldiscriminator.(There are some debates about it.)

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 9 / 23

How does GAN work? Formutalation of GAN

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 10 / 23

How does GAN work? Formutalation of GAN

The description of GANs leads us to formulation for loss:

minG

maxD

V (D,G ) = Ex∼pdata(x)[logD(x)]+Ez∼pz (z)[log(1−D(G (z)))]

(1)

Where both networks rely on gradients flowing through discriminator.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 11 / 23

How does GAN work? Training procedure of GANs

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 12 / 23

How does GAN work? Training procedure of GANs

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 13 / 23

How does GAN work? Training procedure of GANs

GAN Framework

From now and on, we have a basic grasp of GAN, therefore we can codeour own GAN! The starter code can be found in my GitHub Repository:github.com/norveclibalikci/InzvaGanStarter

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 14 / 23

Applications of GANs Computer Vision

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 15 / 23

Applications of GANs Computer Vision

Computer Vision Applications

Image generation(Plug and Play GAN)

Style transfer(CycleGAN)

Image inpainting, super-resolution(SRGAN)

Image to text (Image captioning - Generative Adversarial Text toImage Synthesis)

There are many applications which are not covered above.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 16 / 23

Applications of GANs Computer Vision

Computer Vision Applications

Image generation(Plug and Play GAN)

Style transfer(CycleGAN)

Image inpainting, super-resolution(SRGAN)

Image to text (Image captioning - Generative Adversarial Text toImage Synthesis)

There are many applications which are not covered above.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 16 / 23

Applications of GANs Reinforcement Learning

Outline

1 MotivationWhy should we study generative models?Some results from recent GAN works

2 How does GAN work?GAN ArchitectureFormutalation of GANTraining procedure of GANs

3 Applications of GANsComputer VisionReinforcement Learning

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 17 / 23

Applications of GANs Reinforcement Learning

Applications to real life

Discriminator can be rewarded for labeling correctly and a new losscan be defined by that way. (Still on research)

Generating environment and test for reinforcement learningapplications.

Simulating particle experiments like they do in CERN.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 18 / 23

Limitations of GAN

Many things can be done with GANs however, GANs have limitations aswell.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 19 / 23

Limitations of GAN

There is no training procedure that has been proven to be successful.

Sometimes discriminator learns faster than generator(predictseverything correctly), which leads a gradient problem to generator. Insome cases, it is vice-versa as well.

After from some point, generator keeps generating similar examples.

Losses of models are not meaningful as classifier’s. It just keeposcillating back and forth.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 20 / 23

Limitations of GAN

There is no training procedure that has been proven to be successful.

Sometimes discriminator learns faster than generator(predictseverything correctly), which leads a gradient problem to generator. Insome cases, it is vice-versa as well.

After from some point, generator keeps generating similar examples.

Losses of models are not meaningful as classifier’s. It just keeposcillating back and forth.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 20 / 23

Limitations of GAN

There is no training procedure that has been proven to be successful.

Sometimes discriminator learns faster than generator(predictseverything correctly), which leads a gradient problem to generator. Insome cases, it is vice-versa as well.

After from some point, generator keeps generating similar examples.

Losses of models are not meaningful as classifier’s. It just keeposcillating back and forth.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 20 / 23

Limitations of GAN

There is no training procedure that has been proven to be successful.

Sometimes discriminator learns faster than generator(predictseverything correctly), which leads a gradient problem to generator. Insome cases, it is vice-versa as well.

After from some point, generator keeps generating similar examples.

Losses of models are not meaningful as classifier’s. It just keeposcillating back and forth.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 20 / 23

Limitations of GAN

Loss graphic of a GAN

Figure: Taken from http://www.rricard.me

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 21 / 23

Summary

Summary

GANs are very powerful tools for generating new samples from datayet it has serious issues.

GAN uses semi-supervised approach therefore, there is no need fordata-labeling.

Unsupervised learning is the cake of true AI. (Yann LeCun, Head ofFacebook Research)

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 22 / 23

Appendix For Further Reading

For Further Reading I

Chintala et al.GAN Hacksgithub.com/soumith/ganhacks

Goodfellow et al.Generative Adversarial NetworksarXiv: 1406.2661, 2014.

Sahin Olut (ITU Vision Lab) Generative Adversarial Networks November 4, 2017 23 / 23