towards deep generative models in game development · random variables and generative modelling for...
TRANSCRIPT
![Page 1: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/1.jpg)
Towards Deep Generative Models in GameDevelopment
Jorge del Val
Research Engineer / SEED (Electronic Arts)
![Page 2: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/2.jpg)
![Page 3: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/3.jpg)
Agenda
1. Motivation and fundamentals
2. Variational autoencoders (VAE)
3. Generative adversarial networks (GAN)
4. Conditional generative models
5. Some applications to game development
![Page 4: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/4.jpg)
In a sentence…
Models that generate or remix stuff
![Page 5: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/5.jpg)
In a better sentence…
Models that learn the data probability distribution and are able to sample from it
![Page 6: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/6.jpg)
But… why in games?
![Page 7: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/7.jpg)
(Thanks A. Opara ☺)
![Page 8: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/8.jpg)
![Page 9: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/9.jpg)
Photo Wake-Up: 3D Character Animation from a Single Photo. Weng et al. 2018
![Page 10: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/10.jpg)
Which is real?
A Style-Based Generator Architecture for Generative Adversarial Networks. Karras et al. 2018 (NVIDIA)
![Page 11: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/11.jpg)
Which is real?
FAKE FAKE
A Style-Based Generator Architecture for Generative Adversarial Networks. Karras et al. 2018 (NVIDIA)
REAL
![Page 12: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/12.jpg)
https://thispersondoesnotexist.com
(A Style-Based Generator Architecture for Generative Adversarial Networks. Karras et al. 2018)
![Page 13: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/13.jpg)
So what do they actually do?
![Page 14: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/14.jpg)
![Page 15: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/15.jpg)
![Page 16: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/16.jpg)
![Page 17: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/17.jpg)
![Page 18: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/18.jpg)
Image credit Animation Mentor
![Page 19: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/19.jpg)
![Page 20: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/20.jpg)
In the end… It’s all numbers
… in particular, M-dimensional vectors in 𝒳 ⊂ ℝ𝑀.
![Page 21: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/21.jpg)
Data is far from random
![Page 22: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/22.jpg)
Do we need M pixels to represent a face?
M=1000000 pixels!
![Page 23: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/23.jpg)
Data is not really M-dimensional
It rather lays on a lower dimensional manifold!
![Page 24: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/24.jpg)
Manifold?
𝑧2
𝑧1
𝑥 = (𝑥1, 𝑥2, 𝑥3)𝑧 = (𝑧1, 𝑧2)
![Page 25: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/25.jpg)
Latent dimensions of data
Images credit Ward A.D. et al 2007
*Spoiler: Generative models learn both: the intrinsic geometry and the probability distribution!
![Page 26: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/26.jpg)
Auto-Encoding Variational Bayes. Kingma et al. 2013
𝑧2
𝑧1 𝑧1
𝑧2
![Page 27: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/27.jpg)
A walk through the latent space
A Style-Based Generator Architecture for Generative Adversarial Networks. Karras et al. 2018 (NVIDIA)
![Page 28: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/28.jpg)
And how do they work?
![Page 29: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/29.jpg)
Random variables and generative modelling
For us, each datapoint 𝑥𝑖 is just a realization of an underlying random variable.
𝐱 ∼ 𝑝(𝑥)
● Unsupervised learning is the field which attempts to infer properties of x from samples.
● Generative modelling is a subset of unsupervised learning which attempts to approximatex as some parametrized combination of “simple” random variables which you can sample.
𝐱 ≈ 𝑓𝜃 𝐳𝟏, 𝐳𝟐, … , 𝐳𝐊 ≜ ො𝐱
![Page 30: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/30.jpg)
Example: Gaussian MixturesHere every 𝐳𝑖 is normal (gaussian) 𝒩 𝜇𝑖 , Σ𝑖 and the combination 𝑓𝜃(⋅) is a mixture.
![Page 31: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/31.jpg)
Latent variable models
𝐳 ∼ 𝑝(𝑧)
ො𝐱
Transformation
Prior distribution
![Page 32: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/32.jpg)
Architectures: neural networks
𝑓𝜃
𝜃 weights
![Page 33: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/33.jpg)
Architectures: neural networks
Neural networks can approximate any function 𝑓(𝑥) to any precision! *
𝑥 𝑓𝜃(𝑥)
𝜃
*G. Cybenko 1989, K. Hornik 1991, Z. Lu et al 2017, B. Hanin 2017
Image credit Sydney Firmin
![Page 34: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/34.jpg)
But to find the right 𝜃 (training)?
You optimize some loss (error) function!
min𝜃
𝐿 𝜃; data
![Page 35: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/35.jpg)
E.g. Classifier
𝑥𝑓𝜃 𝑥
𝐿 𝜃; 𝑥, 𝑦 = −𝑦𝑐𝑎𝑡 log 𝑓𝜃 𝑥 − 𝑦𝑑𝑜𝑔 log 1 − 𝑓𝜃 𝑥
𝜃𝑦𝑐𝑎𝑡 ∈ {0,1}
𝑦𝑑𝑜𝑔 ∈ {0,1}Cross-entropy
𝑝 𝑐𝑎𝑡I want it to be…
![Page 36: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/36.jpg)
E.g. Classifier
𝑥𝑓𝜃 𝑥
𝐿 𝜃; 𝑥, 𝑦 = −𝑦𝑐𝑎𝑡 log 𝑓𝜃 𝑥 − 𝑦𝑑𝑜𝑔 log 1 − 𝑓𝜃 𝑥
𝜃𝑦𝑐𝑎𝑡 ∈ {0,1}
𝑦𝑑𝑜𝑔 ∈ {0,1}Cross-entropy
𝑝 𝑐𝑎𝑡I want it to be…
![Page 37: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/37.jpg)
But to find the right 𝜃 (training)?
You optimize some loss (error) function!
𝜃𝑡+1 = 𝜃𝑡 − 𝛼∇𝜃𝐿min𝜃
𝐿 𝜃; data
Stochastic Gradient Descent
![Page 38: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/38.jpg)
But to find the right 𝜃 (training)?
You optimize some loss (error) function!
𝜃𝑡+1 = 𝜃𝑡 − 𝛼∇𝜃𝐿min𝜃
𝐿 𝜃; data
Stochastic Gradient Descent
![Page 39: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/39.jpg)
But to find the right 𝜃 (training)?
You optimize some loss (error) function!
𝜃𝑡+1 = 𝜃𝑡 − 𝛼∇𝜃𝐿min𝜃
𝐿 𝜃; data
Easy to gradient descent any function with current frameworks!! ☺
Stochastic Gradient Descent
![Page 40: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/40.jpg)
Latent variable models
𝐳 ∼ 𝑝 𝑧
ො𝐱
Prior distribution
Transformation
![Page 41: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/41.jpg)
Deep latent variable models
𝐳 ∼ 𝑝 𝑧
ො𝐱 = 𝐺𝜃(𝐳)
𝐺𝜃(⋅)Prior distribution
![Page 42: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/42.jpg)
Training
We want to approximate 𝐱 as ො𝐱 = 𝐺𝜃(𝐳). How do we find the optimal θ?
Maximize the likelihood of the data!
max𝜃
log ℒ(𝜃|𝑥𝑡𝑟𝑎𝑖𝑛) = max𝜃
𝑖=1
𝑁
log 𝑝𝜃(𝑥𝑖)
max𝜃
ℒ(𝜃|𝑥𝑡𝑟𝑎𝑖𝑛) = max𝜃
ෑ
𝑖=1
𝑁
𝑝𝜃(𝑥𝑖)
But… we need 𝑝𝜃 𝑥 explicitly!
Probability that the model would generate 𝑥𝑖
Image credit: Colin Raffel
![Page 43: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/43.jpg)
What about deep latent variable models?
𝑝(𝑧) 𝑝𝜃(𝑥|𝑧)
𝐺𝜃(⋅)𝐳 ො𝐱
𝑝𝜃 𝑥 = න𝑝𝜃 𝑥 𝑧 𝑝 𝑧 𝑑𝑧
What’s the total probability of generating 𝑥?
𝑝𝜃(𝑥)??
![Page 44: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/44.jpg)
Different models – different methods
1. We have 𝑝𝜃( ො𝑥) explicitly: maximize the likelihood.
2. 𝑝𝜃 ො𝑥 is intractable: we can approximate it instead
● Markov Chain Monte Carlo (MCMC) methods
● Variational methods (e.g. Variational Autoencoders)
3. We don’t need 𝑝𝜃 ො𝑥 ; it’s implicit.
● Adversarial methods (e.g. GANs)
![Page 45: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/45.jpg)
Goodfellow. 2016
![Page 46: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/46.jpg)
Variational autoencoder (VAE)
𝑝(𝑧) 𝑝𝜃(𝑥|𝑧)
𝐺𝜃(⋅)𝐳 ො𝐱
𝑝𝜃 𝑥 = න𝑝𝜃 𝑥 𝑧 𝑝 𝑧 𝑑𝑧
![Page 47: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/47.jpg)
Variational autoencoder (VAE)
𝑝(𝑧) 𝑝𝜃(𝑥|𝑧)
𝐺𝜃(⋅)𝐳 ො𝐱
𝑞𝜙(𝑧|𝑥)
𝐸𝜙(⋅)𝐱
![Page 48: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/48.jpg)
Variational autoencoder (VAE)
𝐺𝜃(⋅)𝐳 ො𝐱𝐱 𝐸𝜙(⋅)
𝑞𝜙(𝑧|𝑥)
𝑝(𝑧) 𝑝𝜃(𝑥|𝑧)
log 𝑝𝜃 𝑥 ≥ 𝔼𝑧∼𝑞𝜙(𝑧|𝑥) log 𝑝𝜃 𝑥 𝑧 − KL 𝑞𝜙 𝑧 𝑥 ∥ 𝑝 𝑧
Encoder (inference) Decoder (generation)
Maximize this instead!
![Page 49: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/49.jpg)
True likelihood
Lower bound
Iterations
![Page 50: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/50.jpg)
Pros:
● Efficient inference for free!
o Great tool for modelling the hidden structure of data.
● Stable to train.
● Good theoretical ground.
Cons:
● Not very good samples.
Variational autoencoders
![Page 51: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/51.jpg)
Generative adversarial networks (GANs)
𝑝(𝑧)
𝐺𝜃(⋅)𝐳 ො𝐱
Why not just sample a bunch of data and see if they look real?
𝐱
Sample
𝑥1, 𝑥2, … , 𝑥𝐵
ො𝑥1, ො𝑥2, … , ො𝑥𝐵
Objective:Looks similar!
![Page 52: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/52.jpg)
But… how do we measure similarity between groups of samples?
![Page 53: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/53.jpg)
How to measure similarity of samples
One solution: train a classifier 𝐷𝜙(𝑥) to discriminate!
● If the classifier can not tell if a sample is real or fake, both distributions are close.
● Trained with the standard cross-entropy loss:
max𝜙
𝐿𝑑(𝜙) = max𝜙
𝔼𝑥𝑟∼𝑝𝑟𝑒𝑎𝑙 log 𝐷𝜙 𝑥𝑟 + 𝔼𝑥𝑓∼𝑝𝑓𝑎𝑘𝑒 log 1 − 𝐷𝜙 𝑥𝑓
It can be shown that the optimal classifier performance 𝐿𝑑(𝜙∗) is related to the closeness
between both distributions (JS divergence).
![Page 54: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/54.jpg)
The GAN game
We want to minimize “closeness” between the generated and real samples, as measured by
the discriminator loss:
min𝜃
"closeness"
= min𝜃
max𝜙
𝔼𝑥𝑟∼𝑝𝑟𝑒𝑎𝑙 log 𝐷𝜙 𝑥𝑟 + 𝔼𝑥𝑓∼𝑝𝑓𝑎𝑘𝑒 log 1 − 𝐷𝜙 𝑥𝑓
It’s formally a two-player minimax game!!
![Page 55: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/55.jpg)
Generative adversarial networks
𝑝(𝑧)
𝐺𝜃(⋅)𝐳 ො𝐱
Why not just sample a bunch of data and see if they look real?
𝐱
Sample
𝑥1, 𝑥2, … , 𝑥𝐵
ො𝑥1, ො𝑥2, … , ො𝑥𝐵
Objective:Looks similar!𝐷𝜙(⋅)
Objective:Fool 𝐷𝜙!
![Page 56: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/56.jpg)
![Page 57: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/57.jpg)
GANs
● Pros:
▪ Awesome samples
● Cons:
▪ Unstable training
▪ No explicit probability density
▪ No direct inference
Large Scale GAN Training for High Fidelity Natural Image Synthesis. Brock et al. 2018
![Page 58: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/58.jpg)
Bonus: autoregressive methods
𝑝𝜃 𝑥 =ෑ
𝑡=1
𝑇
𝑝𝜃 𝑥𝑡 𝑥1, … , 𝑥𝑡−1
Generate little by little!
Wavenet: A Generative Model for Raw Audio. Van den Oord et al. 2016.
![Page 59: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/59.jpg)
OK!
![Page 60: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/60.jpg)
OK! I can generate stuff.
![Page 61: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/61.jpg)
OK! I can generate stuff.
But how do I influence what I generate?
![Page 62: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/62.jpg)
OK! I can generate stuff.
How do I remix existing stuff??
![Page 63: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/63.jpg)
Conditional generative models
What if I have information 𝐜 to condition the generation/inference, e.g., class labels?
● Just introduce them in the networks!
𝐺𝜃(⋅)𝐳 ො𝐱𝐱 𝐸𝜙(⋅)
𝑞𝜙(𝑧|𝑥)
𝑝(𝑧) 𝑝𝜃(𝑥|𝑧)
Encoder (inference)
Decoder (generation)
Variational Autoencoder
![Page 64: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/64.jpg)
𝑞𝜙(𝑧|𝑥, 𝑐)
𝑝𝜃(𝑥|𝑧, 𝑐)
ConditionalVariational Autoencoder
Conditional generative models
What if I have information 𝐜 to condition the generation/inference, e.g., class labels?
● Just introduce them in the networks!
𝐺𝜃(⋅)𝐳 ො𝐱𝐱 𝐸𝜙(⋅)
𝑝(𝑧)
Encoder (inference)
Decoder (generation)
𝐜 𝐜
![Page 65: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/65.jpg)
Conditional generative models
What if I have information 𝐜 to condition the generation/inference, e.g., class labels?
● Just introduce them in the networks!
GAN
𝑝(𝑧)
𝐺𝜃(⋅)𝐳 ො𝐱
𝐱
Sample
𝑥1, 𝑥2, … , 𝑥𝐵
ො𝑥1, ො𝑥2, … , ො𝑥𝐵
𝐷𝜙(⋅)
![Page 66: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/66.jpg)
ConditionalGAN
Conditional generative models
What if I have information 𝐜 to condition the generation/inference, e.g., class labels?
● Just introduce them in the networks!
𝑝(𝑧)
𝐺𝜃(⋅)𝐳 ො𝐱
𝐱
Sample
𝑥1, 𝑥2, … , 𝑥𝐵
ො𝑥1, ො𝑥2, … , ො𝑥𝐵
𝐷𝜙(⋅)
𝐜
𝐜
![Page 67: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/67.jpg)
Conditional GMs are very important!
Pix2Pix: Image-to-Image Translation with Conditional Adversarial Networks. Isola et al.
![Page 68: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/68.jpg)
Conditional GMs are very important!
Pose Guided Person Image Generation. Ma et al. 2017.
![Page 69: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/69.jpg)
Some applications to game dev so far?
![Page 70: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/70.jpg)
Generation of terrain
Interactive Example-Based Terrain Authoring with Conditional Generative Adversarial Networks. Guérin et al. 2017.
![Page 71: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/71.jpg)
3D Content Generation
DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation.Park et al. 2019.
Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling.Wu et al. 2016
![Page 72: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/72.jpg)
Face generation
GANFIT: Generative Adversarial Network Fitting for High Fidelity 3D Face Reconstruction. Gecer et al. 2019 (FaceSoft.io)
![Page 73: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/73.jpg)
Procedural placement
Deep Convolutional Priors for Indoor Scene Synthesis. Wang et al. 2018.
![Page 74: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/74.jpg)
Generation of behaviour policies
Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow. Peng et al. 2018.
![Page 75: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/75.jpg)
Generation of behaviour policies
Imitation Learning with Concurrent Actions in 3D Games. Harmer et al. 2018 (SEED)
![Page 76: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/76.jpg)
Learn and accelerate physics
Latent-space Physics: Towards Learning the Temporal Evolution of Fluid Flow. Wiewel et al. 2018
![Page 77: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/77.jpg)
![Page 78: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/78.jpg)
Thanks for inspiration and insightful conversations ☺
Anastasia Opara
Camilo Gordillo
Colin Barré-Brisebois
Hector Anadón León
Henrik Johansson
Jack Harmer
Joakim Bergdahl
Johan Andersson
Kristoffer Sjöö
Ken Brown
Linus Gisslen
Martin Singh-Blom
Mattias Teye
Mark Cerny
Mónica Villanueva Aylagas
Magnus Nordin
Olivier Pomarez
Paul Greveson
Roy Harvey
Tomasz Stachowiak
![Page 79: Towards Deep Generative Models in Game Development · Random variables and generative modelling For us, each datapoint 𝑖 is just a realization of an underlying random variable](https://reader030.vdocument.in/reader030/viewer/2022040611/5ed408df8d46b66d226352c3/html5/thumbnails/79.jpg)
S E E D / / S E A R C H F O R E X T R A O R D I N A R Y E X P E R I E N C E S D I V I S I O N
S T O C K H O L M – L O S A N G E L E S – M O N T R É A L – R E M O T E
S E E D . E A . C O M
W E ‘ R E H I R I N G !
Jorge del Val [email protected]