reviewing autoencoding variational bayes...autoencoding variational bayes & deep generative...
TRANSCRIPT
ReviewingAutoencoding Variational Bayes& Deep Generative Models
Elham DolatabadiMarch 2019
Overview
2
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
• Background
• Paper review
• Discussion on deep generative models
Probabilistic graphical models
3
• Our system is a collection of random variables
Latent variables Z
XObserved variables
ϴ Model parameters
might be huge
might be deep &
continuous
Computational Challenge
4
• For complex model or big data it may be infeasible to compute posterior & marginal likelihood
• Intractable posterior -> No EM
• Approaches to compute posterior:
• Analytical Integration (MAP)
• Approximation:
• Stochastic (sampling, MCMC)
• Variational
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
e.g. neural nets as
components
Variational Approximation
5
Z
X
ϴɸ
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
Variational Lower bound
6
how: to minimize KL(qɸ(z|x) || pϴ(z|x)) w.r.t. variational parameters ɸ
KL(qɸ || pϴ) = ∫qɸ(z|x) log qɸ(z|x)/pϴ(z|x) = log p ϴ(x) + ∫qɸ(z|x) log qɸ(z|x)/pϴ(z,x)
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
L, Variational Lower bound,
L = -∫qɸ(z|x) log qɸ(z|x)/pϴ(z,x)
log p(x) = L + KL(q||p)
log p(x) > L
-L
Variational Lower bound
7
Minimize KL(qɸ(z|x) || pϴ(z|x)) w.r.t. ɸ
maximize L
L = -∫qɸ(z|x) log qɸ(z|x)/pϴ(z,x)
p ϴ(x) is fixed w.r.t ɸ
1
2
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
Connection to Auto-encoders
8
L
Stochastic Encoder
qɸ
Stochastic Decoder
pϴ
Encoder maps x into a distribution qɸSampling z ~ qɸ(z|x) is an “encoding” that converts observations to latent code
Z X
Decoder maps z into a distribution pϴSampling x ~ pθ(x|z) is an “decoding” that reconstructs observations from z
X
Sampled latent vector
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
Reconstruction loss or Expected negative loglikelihood
Regularizer
2
Deep Latent Variable Model
9Deep Latent-Variable
ModelsVariational
ApproximationVariational
Autoencoding
Benefits:
1. Representing complex p(x)
2. Representing complicated conditional dependencies
Example model:
p(z) = N(0,I)pϴ(x|z) = N(µ, σ2), µ = fϴ(z) = deep NN
With neural net fϴ(z),
• pϴ(x) would be complicated
• pϴ(x|z) would be intractable
Image source: Kingma talk Nips 2015
Key Reparameterization Trick
10
• Θ*, ɸ* = argmax L(Θ, ɸ; x)Issue: How to take derivatives w.r.t parameters (e.g. backpropagate) because sampling is a stochastic process
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
Image source: Kingma talk Nips 2015
SGVB estimator
11
2
1
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
AEVB
12
Image source: Kingma talk Nips 2015
Deep Latent-Variable Models
Variational Approximation
Variational Autoencoding
AEVB
13Deep Latent-Variable
ModelsVariational
ApproximationVariational
Autoencoding
Stochastic Encoder
qɸ
Stochastic Decoder
pϴ
Z XXµ
σ
Sampled Latent
Z = µ +σ * ϵ
Pixel differences|| x– f(z) || 2
|| x– f(z) || 2f(z)
(½) [exp(σ(x)) + µ(x)2 – 1 - σ(x)]Σ
VAE
14Deep Latent-Variable
ModelsVariational
ApproximationVariational
Autoencoding
The latent variable space q(z|x) of VAE trained on MNIST
VAE as generative model
15Deep Latent-Variable
ModelsVariational
ApproximationVariational
Autoencoding
DecoderTrained on
MNIST
Demo
16Deep Latent-Variable
ModelsVariational
ApproximationVariational
Autoencoding
VAE interactive demo
Summary
17
• VAE is rooted in Bayesian inference and estimates p(x); optimizes its lower bound
• VAE is a scalable generative modeling with continuous latent variables
• Simple and fast
• Potential applications:
• Deep generative models of images, videos, audio
• Broader application of SGVB estimator
Discussion: Deep Generative Models
18
• State of the art deep generative models
image source: https://openai.com/blog/generative-models/#contributions
Discussion: Deep Generative Models
19
• Three main approaches:
• VAE
• GAN
• PixelRNN
Discussion: VAE vs. GAN
20
VAE:• Find q(z|x), map x onto z• Interpretable p(x)• Produces blurry Images
GAN:• No explicit p(x)• Unstable training dynamics/
difficult to optimize• Difficult to find q(z|x)• Produces sharp images
Image Credit: Autoencoding beyond pixels using a learned similarity metric