an introduction to generative adversarial networks and ... · generative adversarial networks...
Post on 25-Sep-2020
6 Views
Preview:
TRANSCRIPT
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
An Introduction to Generative AdversarialNetworks and Image Deblurring Applications
Dimitrios Gotsis
April 27, 2020
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications1 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Table of Contents
1 Main Message
2 Neural Networks
3 Generative Adversarial Networks
4 Issues with GANs
5 Deblurring with GANs
6 Conclusion
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications2 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Main Message
Brief introduction into neural networks and certain wellknown layers
Give an introduction to Generative Adversarial Networks(GANs) and their potential applications
Introduction to image deblurring problem and potentialsolutions with GANs and its modifications
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications3 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Basic Layers
Will cover some basic layers, many more exist includingvariations on the ones mentioned
ReLU (Rectified Linear Unit): activation function,ReLU(z) = max{0, z}, other activation functions exist likesigmoid, tanh, etc.
Convolutional Layers
Layers that upsample or downsample, e.g. MaxPooling
Linear Layer, y = Wx+ b
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications4 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Structure of a Neural Network
Neural Networks usually pass output from one layer tonext layerMore complex structures can be created that do not just‘stack’, i.e. are not only compositions of functions
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications5 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Training a Neural Network
Typically use gradient descent, i.e. wt+1 = wt − α∇f(wt),or some variant to update weights based on objectivefunction
Need to update weights of all layers, use chain rule toupdate weights due to composition of functions, i.e.[f(g(x))]′ = f ′(g(x))g′(x) = ∂f
∂g∂g∂x
Start with weights from last layer and move backwards totake advantage of chain rule
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications6 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Basic Defintions
A Generative Adversarial Network (GAN) uses two neuralnetworks, a generator (G) and a discriminator (D)
G and D trained in competition to generate images fromnoise to look like they derive from training set
G takes in noise-only images, outputs created images
D takes in created images and real images anddifferentiates between them
When training is complete, discard D and keep G
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications7 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Mathematical Description
Have 0 signify created image and 1 signify real image
Mathematically we have: Dφ : RM×N → [0, 1] andGθ : RL×K → RM×N
M,N dimensions of dimensions of created and real images,K,L dimensions of noise-only image, φ parameters of Dand θ parameters of G
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications8 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Loss Function
The discriminator’s loss function can be shown to be:
LD = −1
2Ex[log(Dφ(x))]− 1
2Ez[log(1−Dφ(Gθ(z)))] (1)
To derive the generator loss function we assume zero sum game,i.e. LD + LG = 0 allowing us to say that:
θ∗ = arg minθ
maxφ
Ex[log(Dφ(x))] + E
z[log(1−Dφ(Gθ(z)))] (2)
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications9 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Training a GAN
Figure: Training a GAN
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications10 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Some Applications
Maps high resolution images to low resolution images well,applications in superresolution and deblurring
Image-to-image translations, such as mapping sketches toimages and aerial photos to maps
Also used in creating artwork
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications11 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Application Examples
Figure: Image-to-Image Translation and Superresolution Examples
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications12 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Mode Collapse
Problem where G maps noise input z to only one potentialoutput point
In practice G maps to some output points but not all whenmode collapse occurs
Caused by the fact that min max is not the same asmax min and during training we simultaneously updateboth networks hoping that solution will be similar
Simultaneous updates do not privilege min max overmax min or vice versa
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications13 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Mode Collapse (cont.)
Figure: No Mode Collapse vs Mode Collapse
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications14 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Vanishing Gradient
During training G or D may overpower one anotherpotentially causing a vanishing gradient value
A small gradient means small updates to weights, if thishappens too soon in training we get bad weights for both Gand D
Many tricks used to avoid problem such as training onenetwork less often during training or changing the modelsize of either network until no vanishing gradient is seen
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications15 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Non-Convergence
Gradient Descent does not converge for all GANs since it isa minimax problem, a competition between two players Gand D
Sometimes you may get oscillatory results meaning thatequilibrium point never reached
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications16 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Deblurring Background
Blurry images very common when taking pictures/video,inconvenient since image information lost
Mathematically can be modeled as IB = h ∗ IS + n, IBblurry image, IS sharp image, h blur kernel, n noise
For simplicity if we assume no noise, and a known kernel
then IS = F−1{IB(ω)H(ω)
}, where F−1 is inverse Fourier
Transform
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications17 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Deblurring Background (cont.)
However, even with very small noise may lead to hugedistortions from fraction if H(ω) has small values
Additionally, h not known in reality and without IS beingknown we can have many potential solutions
Create realistic blur kernels and transform sharp imagesinto blurry images to make dataset
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications18 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Using GANs for Deblurring
Main modification: change input generator input fromnoise-only image to blurry image or blurry image plusnoise-only image pair
Design algorithm for creating realistic blur kernels
In practice other modifications/additions to loss functionalso present in other NN solved problems
See Kupyn et al. (2018)
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications19 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
Conclusion and Summary
NN made up of function compositions, use gradient descent(or variant) plus chain rule to train weights, more complexNN structures also exist
GANs can be used for a variety of different applicationsbut difficult to train due to mode collapse, vanishinggradient and potential non-convergence
Deblurring problem hard to solve due to noise andunknown blur kernel
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications20 / 21
Main MessageNeural Networks
Generative Adversarial NetworksIssues with GANs
Deblurring with GANsConclusion
References
Goodfellow, Ian. “NIPS 2016 Tutorial: GenerativeAdversarial Networks.” ArXiv:1701.00160 [Cs], December31, 2016. http://arxiv.org/abs/1701.00160.
Kupyn, Orest, Volodymyr Budzan, Mykola Mykhailych,Dmytro Mishkin, and Jiri Matas. “DeblurGAN: BlindMotion Deblurring Using Conditional AdversarialNetworks.” In 2018 IEEE/CVF Conference on ComputerVision and Pattern Recognition, 8183–92. Salt Lake City,UT: IEEE, 2018.https://doi.org/10.1109/CVPR.2018.00854.
Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications21 / 21
top related