On the Equivalence Between Deep NADE and Generative

Stochastic Networks Li Yao, Sherjil Ozair, Kyunghyun Cho, Yoshua Bengio

Outline

Deep NADE

How Deep NADE is a GSN

Fast sampling algorithm for Deep NADE

Annealed GSN Sampling

Deep Neural AutoRegressive Distribution Estimators

ith MLP models P(vi|v

Orderless Training

Sample a random permutation of input ordering

Sample a random pivot index j

Backprop through error in indices j to inputs < j

Sampling Sample v1 conditioned on

nothing

Sample v2 conditioned on sampled v1

Sample v3 conditioned on sampled v1, v2

Sample vi conditioned on v1, v2, vi-1

Time complexity of one pass

single-hidden layer: O(#v * #h)

double-hidden layer: O(#v * #h2)

multi-hidden layer: O(#v * k * #h2)

Generative Stochastic Networks

Model P(V | V), where V is a corrupted V

Modeling P(V | V) is easier than modeling P(V)

Gibbs sampling: alternatively sample from P(V | V) and C(V | V)

X

Orderless Training

Sample a random permutation of input ordering

Sample a random pivot index j

Backprop through error in indices j to inputs < j

Also trains a GSN

Corruption process: remove bits j from V

Deep NADE models P(Vj | V

GSN Sampling on NADE

Repeat:

Sample an ordering, and a pivot index j

Sample P(Vj | V

Evaluation: Accuracy

Evaluation: Performance

Evaluation: Qualitative

Thanks!