![Page 1: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/1.jpg)
Introduction of Generative Adversarial Network (GAN)
李宏毅
Hung-yi Lee
![Page 2: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/2.jpg)
Generative Adversarial Network(GAN)• How to pronounce “GAN”?
Google 小姐
![Page 3: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/3.jpg)
Yann LeCun’s comment
https://www.quora.com/What-are-some-recent-and-potentially-upcoming-breakthroughs-in-unsupervised-learning
![Page 4: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/4.jpg)
Yann LeCun’s comment
https://www.quora.com/What-are-some-recent-and-potentially-upcoming-breakthroughs-in-deep-learning
……
![Page 5: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/5.jpg)
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, Shakir Mohamed, “Variational Approaches for Auto-Encoding Generative Adversarial Networks”, arXiv, 2017
All Kinds of GAN … https://github.com/hindupuravinash/the-gan-zoo
GAN
ACGAN
BGAN
DCGAN
EBGAN
fGAN
GoGAN
CGAN
……
![Page 6: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/6.jpg)
ICASSP Keyword search on session index page, so session names are included.
Number of papers whose titles include the keyword
0
5
10
15
20
25
30
35
40
45
2012 2013 2014 2015 2016 2017 2018
generative adversarial reinforcement
GAN becomes a very important technology.
![Page 7: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/7.jpg)
Outline
Basic Idea of GAN
GAN as structured learning
Can Generator learn by itself?
Can Discriminator generate?
A little bit theory
![Page 8: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/8.jpg)
Generation
NNGenerator
Image Generation
Sentence Generation
NNGenerator
We will control what to generate latter. → Conditional Generation
0.1−0.1⋮0.7
−0.30.1⋮0.9
0.1−0.1⋮0.2
−0.30.1⋮0.5
Good morning.
How are you?
0.3−0.1⋮
−0.7
0.3−0.1⋮
−0.7 Good afternoon.
In a specific range
![Page 9: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/9.jpg)
Basic Idea of GAN
Generator
It is a neural network (NN), or a function.
Generator
0.1−3⋮2.40.9
imagevector
Generator
3−3⋮2.40.9
Generator
0.12.1⋮5.40.9
Generator
0.1−3⋮2.43.5
high dimensional vector
Powered by: http://mattya.github.io/chainer-DCGAN/
Each dimension of input vector represents some characteristics.
Longer hair
blue hair Open mouth
![Page 10: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/10.jpg)
Discri-minator
scalarimage
Basic Idea of GAN It is a neural network (NN), or a function.
Larger value means real, smaller value means fake.
Discri-minator
Discri-minator
Discri-minator1.0 1.0
0.1 Discri-minator
0.1
![Page 11: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/11.jpg)
Basic Idea of GAN
Brown veins
Butterflies are not brown
Butterflies do not have veins
……..
Generator
Discriminator
![Page 12: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/12.jpg)
Basic Idea of GAN
NNGenerator
v1
Discri-minator
v1
Real images:
NNGenerator
v2
Discri-minator
v2
NNGenerator
v3
Discri-minator
v3
This is where the term“adversarial” comes from.You can explain the process in different ways…….
![Page 13: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/13.jpg)
Generatorv3
Generatorv2
Basic Idea of GAN(和平的比喻)
Generator(student)
Discriminator(teacher)
Generatorv1 Discriminator
v1
Discriminatorv2
沒有兩個圈
沒有彩色
為什麼不自己學? 為什麼不自己做?
![Page 14: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/14.jpg)
Generator v.s. Discriminator
• 寫作敵人,唸做朋友
![Page 15: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/15.jpg)
• Initialize generator and discriminator
• In each training iteration:
DG
sample
generated objects
G
Algorithm
D
Update
vector
vector
vector
vector
0000
1111
randomly sampled
Database
Step 1: Fix generator G, and update discriminator D
Discriminator learns to assign high scores to real objects and low scores to generated objects.
Fix
![Page 16: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/16.jpg)
• Initialize generator and discriminator
• In each training iteration:
DG
Algorithm
Step 2: Fix discriminator D, and update generator G
Discri-minator
NNGenerator
vector
0.13
hidden layer
update fix
Gradient Ascent
large network
Generator learns to “fool” the discriminator
![Page 17: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/17.jpg)
• In each training iteration:
• Sample m examples 𝑥1, 𝑥2, … , 𝑥𝑚 from database
• Sample m noise samples 𝑧1, 𝑧2, … , 𝑧𝑚 from a distribution
• Obtaining generated data 𝑥1, 𝑥2, … , 𝑥𝑚 , 𝑥𝑖 = 𝐺 𝑧𝑖
• Update discriminator parameters 𝜃𝑑 to maximize
• ෨𝑉 =1
𝑚σ𝑖=1𝑚 𝑙𝑜𝑔𝐷 𝑥𝑖 +
1
𝑚σ𝑖=1𝑚 𝑙𝑜𝑔 1 − 𝐷 𝑥𝑖
• 𝜃𝑑 ← 𝜃𝑑 + 𝜂𝛻 ෨𝑉 𝜃𝑑• Sample m noise samples 𝑧1, 𝑧2, … , 𝑧𝑚 from a
distribution
• Update generator parameters 𝜃𝑔 to maximize
• ෨𝑉 =1
𝑚σ𝑖=1𝑚 𝑙𝑜𝑔 𝐷 𝐺 𝑧𝑖
• 𝜃𝑔 ← 𝜃𝑔 − 𝜂𝛻 ෨𝑉 𝜃𝑔
Algorithm
Learning D
Learning G
Initialize 𝜃𝑑 for D and 𝜃𝑔 for G
![Page 18: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/18.jpg)
Anime Face Generation
100 updates
Source of training data: https://zhuanlan.zhihu.com/p/24767059
![Page 19: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/19.jpg)
Anime Face Generation
1000 updates
![Page 20: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/20.jpg)
Anime Face Generation
2000 updates
![Page 21: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/21.jpg)
Anime Face Generation
5000 updates
![Page 22: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/22.jpg)
Anime Face Generation
10,000 updates
![Page 23: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/23.jpg)
Anime Face Generation
20,000 updates
![Page 24: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/24.jpg)
Anime Face Generation
50,000 updates
![Page 25: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/25.jpg)
The faces generated by machine.
圖片生成:吳宗翰、謝濬丞、陳延昊、錢柏均
![Page 26: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/26.jpg)
感謝陳柏文同學提供實驗結果
0.00.0
G
0.90.9
G
0.10.1
G
0.20.2
G
0.30.3
G
0.40.4
G
0.50.5
G
0.60.6
G
0.70.7
G
0.80.8
G
![Page 27: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/27.jpg)
Outline
Basic Idea of GAN
GAN as structured learning
Can Generator learn by itself?
Can Discriminator generate?
A little bit theory
![Page 28: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/28.jpg)
Structured Learning
YXf :Regression: output a scalar
Classification: output a “class”
Structured Learning/Prediction: output a sequence, a matrix, a graph, a tree ……
(one-hot vector)1 0 0
Machine learning is to find a function f
Output is composed of components with dependency
0 1 0 0 0 1
Class 1 Class 2 Class 3
![Page 29: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/29.jpg)
Output Sequence
(what a user says)
YXf :
“機器學習及其深層與結構化”
“Machine learning and having it deep and structured”
:X :Y
感謝大家來上課”
(response of machine)
:X :Y
Machine Translation
Speech Recognition
Chat-bot
(speech)
:X :Y
(transcription)
(sentence of language 1) (sentence of language 2)
“How are you?” “I’m fine.”
![Page 30: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/30.jpg)
Output Matrix YXf :
ref: https://arxiv.org/pdf/1605.05396.pdf
“this white and yellow flower have thin white petals and a round yellow stamen”
:X :Y
Text to Image
Image to Image
:X :Y
Colorization:
Ref: https://arxiv.org/pdf/1611.07004v1.pdf
![Page 31: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/31.jpg)
Why Structured Learning Challenging?• One-shot/Zero-shot Learning:
• In classification, each class has some examples.
• In structured learning,
• If you consider each possible output as a “class” ……
• Since the output space is huge, most “classes” do not have any training data.
• Machine has to create new stuff during testing.
• Need more intelligence
![Page 32: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/32.jpg)
Why Structured Learning Challenging?• Machine has to learn to do planning
• Machine generates objects component-by-component, but it should have a big picture in its mind.
• Because the output components have dependency, they should be considered globally.
Image Generation
Sentence Generation
這個婆娘不是人
九天玄女下凡塵
![Page 33: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/33.jpg)
Structured Learning Approach
Bottom Up
Top
Down
Generative Adversarial
Network (GAN)
Generator
Discriminator
Learn to generate the object at the component level
Evaluating the whole object, and find the best one
![Page 34: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/34.jpg)
Outline
Basic Idea of GAN
GAN as structured learning
Can Generator learn by itself?
Can Discriminator generate?
A little bit theory
![Page 35: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/35.jpg)
Generator
Image:
code: 0.10.9(where does they
come from?)
NNGenerator
0.1−0.5
0.2−0.1
0.30.2
NNGenerator
cod
e
vectors
cod
eco
de
0.10.9
image
As close as possible
c.f. NNClassifier
𝑦1𝑦2⋮
As close as possible10⋮
![Page 36: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/36.jpg)
Generator
Encoder in auto-encoder provides the code ☺
𝑐NN
Encoder
NNGenerator
cod
e
vectors
cod
eco
de
Image:
code:(where does they come from?)
0.10.9
0.1−0.5
0.2−0.1
0.30.2
![Page 37: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/37.jpg)
Auto-encoder
NNEncoder
NNDecoder
code
Compact representation of the input object
codeCan reconstruct the original object
Learn together28 X 28 = 784
Low dimension
𝑐
Trainable
NNEncoder
NNDecoder
![Page 38: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/38.jpg)
Auto-encoder
As close as possible
NNEncoder
NNDecoder
cod
e
NNDecoder
cod
e
Randomly generate a vector as code Image ?
= Generator
= Generator
![Page 39: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/39.jpg)
Auto-encoder
NNDecoder
code
2D
-1.5 1.5−1.50
NNDecoder
1.50
NNDecoder
(real examples)
image
![Page 40: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/40.jpg)
Auto-encoder
-1.5 1.5
(real examples)
![Page 41: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/41.jpg)
Auto-encoder NNGenerator
cod
e
vectors
cod
eco
de
NNGenerator
aNN
Generatorb
NNGenerator
?a b0.5x + 0.5x
Image:
code:(where does them come from?)
0.10.9
0.1−0.5
0.2−0.1
0.30.2
![Page 42: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/42.jpg)
Auto-encoder
Variational Auto-encoder(VAE)
NNEncoder
input NNDecoder
output
m1m2m3
𝜎1𝜎2𝜎3
𝑒3
𝑒1𝑒2
From a normal distribution
𝑐3
𝑐1𝑐2
X
+
Minimize reconstruction error
𝑖=1
3
𝑒𝑥𝑝 𝜎𝑖 − 1 + 𝜎𝑖 + 𝑚𝑖2
exp
𝑐𝑖 = 𝑒𝑥𝑝 𝜎𝑖 × 𝑒𝑖 +𝑚𝑖
Minimize
NNEncoder
NNDecoder
code
input output
= Generator
![Page 43: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/43.jpg)
What do we miss?
Gas close as
possible
Target Generated Image
It will be fine if the generator can truly copy the target image.
What if the generator makes some mistakes …….
Some mistakes are serious, while some are fine.
![Page 44: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/44.jpg)
What do we miss? Target
1 pixel error 1 pixel error
6 pixel errors 6 pixel errors
我覺得不行 我覺得不行
我覺得其實可以
我覺得其實可以
![Page 45: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/45.jpg)
What do we miss?
我覺得不行 我覺得其實可以
The relation between the components are critical.
Although highly correlated, they cannot influence each other.
Need deep structure to catch the relation between components.
Layer L-1
……
Layer L…
…
……
……
……
Each neural in output layer corresponds to a pixel.
ink
empty
旁邊的,我們產生一樣的顏色
誰管你
![Page 46: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/46.jpg)
(Variational) Auto-encoder
感謝黃淞楓同學提供結果
x1
x2
G𝑧1𝑧2
𝑥1𝑥2
![Page 47: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/47.jpg)
Outline
Basic Idea of GAN
GAN as structured learning
Can Generator learn by itself?
Can Discriminator generate?
A little bit theory
![Page 48: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/48.jpg)
Discriminator
• Discriminator is a function D (network, can deep)
• Input x: an object x (e.g. an image)
• Output D(x): scalar which represents how “good” an object x is
R:D X
D 1.0 D 0.1
Can we use the discriminator to generate objects?Yes.
Evaluation function, Potential Function, Energy Function …
![Page 49: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/49.jpg)
Discriminator
• It is easier to catch the relation between thecomponents by top-down evaluation.
我覺得不行 我覺得其實 OKThis CNN filter is good enough.
![Page 50: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/50.jpg)
Discriminator
• Suppose we already have a good discriminator D(x) …
How to learn the discriminator?
Enumerate all possible x !!!
It is feasible ???
![Page 51: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/51.jpg)
Discriminator - Training
• I have some real images
Discriminator training needs some negative examples.
D scalar 1(real)
D scalar 1(real)
D scalar 1(real)
D scalar 1(real)
Discriminator only learns to output “1” (real).
![Page 52: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/52.jpg)
Discriminator - Training
• Negative examples are critical.
D scalar 1(real)
D scalar 0(fake)
D 0.9 Pretty real
D scalar 1(real)
D scalar 0(fake)
How to generate realistic negative examples?
![Page 53: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/53.jpg)
Discriminator - Training
• General Algorithm
• Given a set of positive examples, randomly generate a set of negative examples.
• In each iteration• Learn a discriminator D that can discriminate
positive and negative examples.
• Generate negative examples by discriminator D
D
xDxXx
maxarg~
v.s.
![Page 54: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/54.jpg)
Discriminator - Training
D(x)
real examples
In practice, you cannot decrease all the x other than real examples.
Discriminator D
objectx
scalarD(x)
![Page 55: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/55.jpg)
Discriminator - Training
𝐷 𝑥
𝐷 𝑥
𝐷 𝑥𝐷 𝑥
In the end ……
real
generated
![Page 56: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/56.jpg)
Graphical Model
Bayesian Network(Directed Graph)
Markov Random Field(Undirected Graph)
Markov Logic Network
Boltzmann Machine
Restricted Boltzmann Machine
Structured Learning ➢ Structured Perceptron➢ Structured SVM➢Gibbs sampling➢Hidden information➢Application: sequence
labelling, summarization
Conditional Random Field
Segmental CRF
(Only list some of the approaches)
Energy-based Model: http://www.cs.nyu.edu/~yann/research/ebm/
![Page 57: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/57.jpg)
Generator v.s. Discriminator
• Generator
• Pros:• Easy to generate even
with deep model
• Cons:• Imitate the appearance
• Hard to learn the correlation between components
• Discriminator
• Pros:• Considering the big
picture
• Cons:• Generation is not
always feasible• Especially when
your model is deep• How to do negative
sampling?
![Page 58: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/58.jpg)
Generator + Discriminator
• General Algorithm
• Given a set of positive examples, randomly generate a set of negative examples.
• In each iteration• Learn a discriminator D that can discriminate
positive and negative examples.
• Generate negative examples by discriminator D
D
xDxXx
maxarg~
v.s.
G x~ =
![Page 59: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/59.jpg)
Benefit of GAN
• From Discriminator’s point of view
• Using generator to generate negative samples
• From Generator’s point of view
• Still generate the object component-by-component
• But it is learned from the discriminator with global view.
xDxXx
maxarg~G x~ =
efficient
![Page 60: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/60.jpg)
GAN
感謝段逸林同學提供結果
https://arxiv.org/abs/1512.09300
VAE
GAN
![Page 61: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/61.jpg)
FID[Martin Heusel, et al., NIPS, 2017]: Smaller is better
[Mario Lucic, et al. arXiv, 2017]
![Page 62: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/62.jpg)
Next Time
• Preview
• https://youtu.be/0CKeqXl5IY0
• https://youtu.be/KSN4QYgAtao
![Page 63: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/63.jpg)
Outline
Basic Idea of GAN
GAN as structured learning
Can Generator learn by itself?
Can Discriminator generate?
A little bit theory
![Page 64: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/64.jpg)
Generator
• A generator G is a network. The network defines a probability distribution 𝑃𝐺
generator G𝑧 𝑥 = 𝐺 𝑧
Normal Distribution
𝑃𝐺(𝑥) 𝑃𝑑𝑎𝑡𝑎 𝑥
as close as possible
How to compute the divergence?
𝐺∗ = 𝑎𝑟𝑔min𝐺
𝐷𝑖𝑣 𝑃𝐺 , 𝑃𝑑𝑎𝑡𝑎Divergence between distributions 𝑃𝐺 and 𝑃𝑑𝑎𝑡𝑎
𝑥: an image (a high-dimensional vector)
![Page 65: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/65.jpg)
Discriminator𝐺∗ = 𝑎𝑟𝑔min
𝐺𝐷𝑖𝑣 𝑃𝐺 , 𝑃𝑑𝑎𝑡𝑎
Although we do not know the distributions of 𝑃𝐺 and 𝑃𝑑𝑎𝑡𝑎, we can sample from them.
sample
G
vector
vector
vector
vector
sample from normal
Database
Sampling from 𝑷𝑮
Sampling from 𝑷𝒅𝒂𝒕𝒂
![Page 66: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/66.jpg)
Discriminator 𝐺∗ = 𝑎𝑟𝑔min𝐺
𝐷𝑖𝑣 𝑃𝐺 , 𝑃𝑑𝑎𝑡𝑎
Discriminator
: data sampled from 𝑃𝑑𝑎𝑡𝑎: data sampled from 𝑃𝐺
train
𝑉 𝐺,𝐷 = 𝐸𝑥∼𝑃𝑑𝑎𝑡𝑎 𝑙𝑜𝑔𝐷 𝑥 + 𝐸𝑥∼𝑃𝐺 𝑙𝑜𝑔 1 − 𝐷 𝑥
Example Objective Function for D
(G is fixed)
𝐷∗ = 𝑎𝑟𝑔max𝐷
𝑉 𝐷, 𝐺Training:
Using the example objective function is exactly the same as training a binary classifier.
The maximum objective value is related to JS divergence.
![Page 67: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/67.jpg)
Discriminator 𝐺∗ = 𝑎𝑟𝑔min𝐺
𝐷𝑖𝑣 𝑃𝐺 , 𝑃𝑑𝑎𝑡𝑎
Discriminator
: data sampled from 𝑃𝑑𝑎𝑡𝑎: data sampled from 𝑃𝐺
train
hard to discriminatesmall divergence
Discriminatortrain
easy to discriminatelarge divergence
𝐷∗ = 𝑎𝑟𝑔max𝐷
𝑉 𝐷, 𝐺
Training:
(cannot make objective large)
![Page 68: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/68.jpg)
𝐺∗ = 𝑎𝑟𝑔min𝐺
𝐷𝑖𝑣 𝑃𝐺 , 𝑃𝑑𝑎𝑡𝑎max𝐷
𝑉 𝐺,𝐷
The maximum objective value is related to JS divergence.
• Initialize generator and discriminator
• In each training iteration:
Step 1: Fix generator G, and update discriminator D
Step 2: Fix discriminator D, and update generator G
𝐷∗ = 𝑎𝑟𝑔max𝐷
𝑉 𝐷, 𝐺
[Goodfellow, et al., NIPS, 2014]
![Page 69: Introduction of Generative Adversarial Network (GAN)speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/GAN (v2).… · •In each training iteration: •Sample m examples 1, 2,…,](https://reader034.vdocument.in/reader034/viewer/2022042218/5ec46b541f6eb751b323856c/html5/thumbnails/69.jpg)
Using the divergence you like ☺
Can we use other divergence?