![Page 1: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/1.jpg)
Poisson Gamma Dynamical Systems
Aaron Schein UMass Amherst
Mingyuan Zhou Univ. Texas at Austin
Hanna Wallach Microsoft Research
Joint work with:
![Page 2: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/2.jpg)
Sequential multivariate count data
Days in 2003
![Page 3: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/3.jpg)
Sequential multivariate count data
Days in 2003
bursty
![Page 4: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/4.jpg)
Sequential multivariate count data
Days in 2003
bursty
![Page 5: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/5.jpg)
Sequentially observed count vectors
It is everywhere.
Sequential multivariate count data
![Page 6: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/6.jpg)
International relations
![Page 7: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/7.jpg)
RUS
PRK
JAP
KOR
International relations
![Page 8: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/8.jpg)
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
June 28, 2003
RUS
PRK
JAP
KOR
Sequentially observed count vectors
![Page 9: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/9.jpg)
6/28
32
2
0
0
0
5
Sequentially observed count vectors
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
June 28, 2003
RUS
PRK
JAP
KOR
![Page 10: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/10.jpg)
Sequentially observed count vectors
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
32
2
0
0
0
5
June 29, 2003
RUS
PRK
JAP
KOR
40
0
0
0
0
0
6/29
![Page 11: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/11.jpg)
Sequentially observed count vectors
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
32
2
0
0
0
5
June 30, 2003
RUS
PRK
JAP
KOR
40
0
0
0
0
0
6/29
7
0
2
5
6
10
6/30
![Page 12: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/12.jpg)
Sequentially observed count vectors
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
32
2
0
0
0
5
July 1, 2003
RUS
PRK
JAP
KOR
40
0
0
0
0
0
6/29
7
0
2
5
6
10
6/30
3
0
0
0
0
13
7/1
![Page 13: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/13.jpg)
Sequentially observed count vectorsJuly 2, 2003
RUS
PRK
JAP
KOR
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
32
2
0
0
0
5
40
0
0
0
0
0
6/29
7
0
2
5
6
10
6/30
3
0
0
0
0
13
7/1
3
0
21
0
0
0
7/2
![Page 14: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/14.jpg)
Sequentially observed count vectors
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
32
2
0
0
0
5
40
0
0
0
0
0
6/29
7
0
2
5
6
10
6/30
3
0
0
0
0
13
7/1
3
0
21
0
0
0
7/2
T
V
![Page 15: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/15.jpg)
Sequentially observed count vectors
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
32
2
0
0
0
5
40
0
0
0
0
0
6/29
7
0
2
5
6
10
6/30
3
0
0
0
0
13
7/1
3
0
21
0
0
0
7/2
sparseV = 6,197
only 4% non-zero!
T
VT = 365
![Page 16: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/16.jpg)
Sequentially observed count vectors
y(1), . . . ,y(T )
y(t) =0
2
0
0
1
7V
![Page 17: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/17.jpg)
Sequentially observed count vectors
y(1), . . . ,y(T )
y(t) =0
2
0
0
1
7V
bursty
RUS-KOR
RUS-PRK
KOR-PRK
JAP-RUS
JAP-PRK
JAP-KOR
6/28
32
2
0
0
05
40
0
0
0
0
0
6/29
7
02
5
6
10
6/30
3
0
0
0
013
7/1
3
021
0
0
0
7/2
sparse
Especially for large V!
![Page 18: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/18.jpg)
Modeling sequential count data
• Prediction (forecasting, smoothing) • Explanation • Exploration (interpretability!)
Many reasons to want a probabilistic model
![Page 19: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/19.jpg)
Modeling sequential count data
• Autoregressive models
• Hidden Markov models
• Dynamic topic models
• Hawkes process models
Many probabilistic models for sequential counts
![Page 20: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/20.jpg)
Modeling sequential count data
• Expressive
• Parsimonious
• Gaussian*
Linear dynamical systems
unnatural for count datamisspecified likelihood… …or non-conjugate
![Page 21: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/21.jpg)
Modeling sequential count data
![Page 22: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/22.jpg)
The case for naturalness
Natural models
• Statistically and computationally efficient • the right inductive bias constrains the hypothesis space • the right inductive bias supplements a lack of data • quantities of interest available in closed form
• Important for measurement
✓ likelihood matches the support of the data ✓ conjugate priors
![Page 23: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/23.jpg)
Our contributions:• Novel generative model
• Matches the form of linear dynamical systems • Natural for count data
• Auxiliary variable-based MCMC inference
• Superior forecasting and smoothing performance
(not size of data matrix, like the Gaussian LDS)
Benefits of the model:• Inference scales with the number of non-zeros
• Highly interpretable latent structure
Poisson gamma dynamical systems
![Page 24: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/24.jpg)
y(t)v ⇠ Pois
!
· · ·
(natural likelihood for counts)
Poisson gamma dynamical systems
![Page 25: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/25.jpg)
y(t)v
how active event type v is in component k
KX
k=1
�kv✓(t)k⇠ Pois
!
Poisson gamma dynamical systems
![Page 26: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/26.jpg)
y(t)v
how active component k is at time step t
KX
k=1
�kv✓(t)k⇠ Pois
!
Poisson gamma dynamical systems
![Page 27: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/27.jpg)
y(t)v
KX
k=1
�kv✓(t)k
h i=
Poisson gamma dynamical systems
![Page 28: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/28.jpg)
⌧0Gam
,
!
⇠✓(t)k
KX
k0=1
⇡kk0✓(t�1)k0⌧0 · · ·
(conjugate prior to Poisson)
Poisson gamma dynamical systems
![Page 29: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/29.jpg)
⌧0Gam
,
!
⇠✓(t)k
how active component k’ is at time step t-1
KX
k0=1
⇡kk0✓(t�1)k0⌧0
Poisson gamma dynamical systems
![Page 30: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/30.jpg)
⌧0Gam
,
!
⇠✓(t)k
KX
k0=1
⇡kk0✓(t�1)k0⌧0
the probability of transitioning from component k’ into k
KX
k=1
⇡kk0 = 1
Poisson gamma dynamical systems
![Page 31: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/31.jpg)
⌧0Gam
,
!
⇠✓(t)k
KX
k0=1
⇡kk0✓(t�1)k0⌧0
concentration hyperparameter
Poisson gamma dynamical systems
![Page 32: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/32.jpg)
⌧0✓(t)k
KX
k0=1
⇡kk0✓(t�1)k0⌧0h i
=
Poisson gamma dynamical systems
![Page 33: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/33.jpg)
✓(t)k
KX
k0=1
⇡kk0✓(t�1)k0
h i=
Poisson gamma dynamical systems
![Page 34: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/34.jpg)
h i= ⇧✓(t�1)✓(t)
h i=y(t) �✓(t)
(matches linear dynamical systems)
Poisson gamma dynamical systems
![Page 35: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/35.jpg)
Inferred latent structure
International relations data from 2003
![Page 36: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/36.jpg)
Inferred latent structure
�kv
The top event types for component k=1
![Page 37: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/37.jpg)
Inferred latent structure
✓(1)k , . . . , ✓(T )k
�kv
![Page 38: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/38.jpg)
Inferred latent structure
![Page 39: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/39.jpg)
Inferred latent structure
![Page 40: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/40.jpg)
Inferred latent structure
![Page 41: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/41.jpg)
Inferred latent structure
![Page 42: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/42.jpg)
Inferred latent structure
![Page 43: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/43.jpg)
Inferred latent structure
![Page 44: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/44.jpg)
Inferred latent structure
![Page 45: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/45.jpg)
Inferred latent structure
![Page 46: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/46.jpg)
Technical challenge LegendPoisson/MultinomialGamma/Dirichlet
⇧
✓(1)
y(1) y(2) y(3)
✓(2) ✓(3)
⌫
�
![Page 47: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/47.jpg)
Technical challenge LegendPoisson/MultinomialGamma/Dirichlet
⇧
✓(1)
y(1) y(2) y(3)
✓(2) ✓(3)
⌫
�
� ⇠(conditional posterior has closed form)
P (⇧ |Y,⇥,⌫)⇧ ⇠
P (⇥ |Y,⇧,⌫)⇥ ⇠
P (� |Y )
![Page 48: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/48.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/Dirichlet
⇧
✓(1)
y(1) y(2) y(3)
✓(2)✓(3)
⌫
�
Original model⇧
y(1) y(2) y(3)
⌫
l(4)
m(3)
l(3)
m(2)
l(2)
m(1)
l(1)
�
Alternative model
⇧ ⇠ P (⇧ | A, Y,⌫)
![Page 49: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/49.jpg)
Three rules
m
�↵
l
m
�↵
l
✓
m
�↵
m
�↵
y
✓ l
y
✓ l
m
LegendPoisson/MultinomialGamma/Dirichlet
![Page 50: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/50.jpg)
Three rules
y
✓ l
y
✓ l
m
LegendPoisson/MultinomialGamma/Dirichlet
Relationship between Poisson and Multinomial
Steel (1953)
![Page 51: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/51.jpg)
Three rules
m
�↵
l
m
�↵
l
✓
m
�↵
m
�↵
y
✓ l
y
✓ l
m Poisson-multinomial relationship
LegendPoisson/MultinomialGamma/Dirichlet
![Page 52: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/52.jpg)
Three rules
✓
m
�↵
m
�↵
LegendPoisson/MultinomialGamma/DirichletNegative binomial
Negative binomial as gamma-PoissonGreenwood & Yule (1920)
![Page 53: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/53.jpg)
Three rules
m
�↵
l
m
�↵
l
✓
m
�↵
m
�↵
y
✓ l
y
✓ l
m Poisson-multinomial relationship
LegendPoisson/MultinomialGamma/Dirichlet
Negative binomial definition
Negative binomial
![Page 54: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/54.jpg)
Three rules
m
�↵
l
m
�↵
l
LegendPoisson/MultinomialGamma/DirichletNegative binomial
P (l,m |↵,�)
Magic bivariate count distributionZhou & Carin (2012)
![Page 55: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/55.jpg)
Three rules
m
�↵
l
m
�↵
l
LegendPoisson/MultinomialGamma/DirichletNegative binomial
P (l,m |↵,�)
Magic bivariate count distributionZhou & Carin (2012)
![Page 56: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/56.jpg)
Three rules
m
�↵
l
m
�↵
l
LegendPoisson/MultinomialGamma/DirichletNegative binomial
P (l,m |↵,�)
Magic bivariate count distributionZhou & Carin (2012)
P (m |↵,�)=
![Page 57: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/57.jpg)
Three rules
m
�↵
l
m
�↵
l
LegendPoisson/MultinomialGamma/DirichletNegative binomial
P (l,m |↵,�)
Magic bivariate count distribution
CRT
Zhou & Carin (2012)
P (m |↵,�)=
P (l |m,↵)
![Page 58: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/58.jpg)
Three rules
m
�↵
l
m
�↵
l
LegendPoisson/MultinomialGamma/DirichletNegative binomial
P (l,m |↵,�)
Magic bivariate count distribution
CRT
Zhou & Carin (2012)
P (m |↵,�)=
P (l |m,↵) P (l |↵,�)=
![Page 59: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/59.jpg)
Three rules
m
�↵
l
m
�↵
l
LegendPoisson/MultinomialGamma/DirichletNegative binomial
P (l,m |↵,�)
Magic bivariate count distribution
CRTSumLog
Zhou & Carin (2012)
P (m |↵,�)=
P (l |m,↵) P (l |↵,�)=
P (m | l,�)
![Page 60: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/60.jpg)
Three rules
m
�↵
l
m
�↵
l
LegendPoisson/MultinomialGamma/DirichletNegative binomial
P (l,m |↵,�)
P (m | l,�)P (l |↵,�)=
P (l |m,↵)P (m |↵,�)=
Magic bivariate count distribution
CRTSumLog
Zhou & Carin (2012)
![Page 61: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/61.jpg)
Three rules
m
�↵
l
m
�↵
l
✓
m
�↵
m
�↵
y
✓ l
y
✓ l
m Poisson-multinomial relationship
LegendPoisson/MultinomialGamma/Dirichlet
Negative binomial definition
Negative binomialCRTSumLog
Magic bivariate count distribution
![Page 62: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/62.jpg)
Augment and Conquer
⇧
✓(1)
y(1) y(2) y(3)
✓(2) ✓(3)
⌫
�
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
![Page 63: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/63.jpg)
Augment and Conquer
⇧
✓(1)
y(1) y(2) y(3)
✓(2) ✓(3)
⌫
l(4)
�
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
![Page 64: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/64.jpg)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
✓(3)
Augment and Conquer
�
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
![Page 65: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/65.jpg)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
Augment and Conquer
�
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
![Page 66: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/66.jpg)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
�
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
![Page 67: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/67.jpg)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
�
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
![Page 68: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/68.jpg)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 69: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/69.jpg)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
m(2)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 70: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/70.jpg)
⇧
✓(1)
y(1) y(2) y(3)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
m(2)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 71: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/71.jpg)
⇧
✓(1)
y(1) y(2) y(3)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
m(2)
l(2)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 72: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/72.jpg)
⇧
✓(1)
y(1) y(2) y(3)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
m(2)
l(2)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 73: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/73.jpg)
⇧
✓(1)
y(1) y(2) y(3)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
m(2)
l(2)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 74: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/74.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog⇧
✓(1)
y(1) y(2) y(3)
⌫
l(4)
m(3)
l(3)
m(2)
l(2)
m(1)
�
![Page 75: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/75.jpg)
⇧
y(1) y(2) y(3)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
m(2)
l(2)
m(1)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 76: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/76.jpg)
⇧
y(1) y(2) y(3)
⌫
l(4)
m(3)
Augment and Conquer
l(3)
m(2)
l(2)
m(1)
l(1)
LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
�
![Page 77: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/77.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog⇧
y(1) y(2) y(3)
⌫
l(4)
m(3)
l(3)
m(2)
l(2)
m(1)
l(1)
�
![Page 78: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/78.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog⇧
y(1) y(2) y(3)
⌫
l(4)
m(3)
l(3)
m(2)
l(2)
m(1)
l(1)
�
⇧ ⇠ P (⇧ | A, Y,⌫)
![Page 79: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/79.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog⇧
y(1) y(2) y(3)
⌫
l(4)
m(3)
l(3)
m(2)
l(2)
m(1)
l(1)
�
![Page 80: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/80.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog⇧
✓(1)
y(1) y(2) y(3)
⌫
l(4)
m(3)
l(3)
m(2)
l(2)
m(1)
�
✓(1) ⇠ P (✓(1) | A, Y,⇧,⌫)
![Page 81: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/81.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
✓(2) ⇠ P (✓(2) |✓(1),A, Y,⇧,⌫)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
l(3)
m(2)
�
![Page 82: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/82.jpg)
Augment and Conquer LegendPoisson/MultinomialGamma/DirichletNegative binomialCRTSumLog
✓(3) ⇠ P (✓(3) |✓(2),✓(1),A, Y,⇧,⌫)
⇧
✓(1)
y(1) y(2) y(3)
✓(2)
⌫
l(4)
m(3)
✓(3)
�
![Page 83: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/83.jpg)
Solution
Forward sampling ✓(t) ⇠ P (✓(t) |✓(t�1),A, Y,⇧,⌫)
Backward filtering A(t) ⇠ P (A(t) | A(t+1), Y,⇥,⇧,⌫)
Backward filtering forward sampling (BFFS)
![Page 84: Poisson Gamma Dynamical Systems - Columbia Universityas5530/ScheinZhouWallach2016_slides.pdf · Our contributions: • Novel generative model • Matches the form of linear dynamical](https://reader033.vdocument.in/reader033/viewer/2022060907/60a250043c8d382a0c28c78c/html5/thumbnails/84.jpg)
Conclusion
Elegant inference in the natural model
and relies on a novel auxiliary variable scheme
that scales with the number of non-zeros
Come to poster #193 tonight!
https://github.com/aschein/pgds