inference in probabilistic graphical models by …2019/04/12 · inference in probabilistic...
TRANSCRIPT
![Page 1: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/1.jpg)
Inference in Probabilistic Graphical Models by GraphNeural Networks
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya,Raquel Urtasun, Richard Zemel, Xaq Pitkow
Presenter: Arshdeep Sekhon
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 1 / 22
![Page 2: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/2.jpg)
Outline
1 Introduction
2 Proposed Method
3 Experiments
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 2 / 22
![Page 3: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/3.jpg)
Probabilistic Graphical Models(PGMs)
Given x ∈ RD , joint probability p(x)
Simplify joint p(x) into a factorization based on conditionalindependence defined by a graph structure.
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 3 / 22
![Page 4: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/4.jpg)
Factor Graphs
Factorization of the joint probability distribution for more efficientcomputationsbipartite graph: two types of nodes, edges connect different nodetypesGiven a factorization:g(X1,X2,X3) = f1(X1)f2(X1,X2)f3(X1,X2)f4(X2,X3)
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 4 / 22
![Page 5: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/5.jpg)
Tasks for PGMs: Inference
Inference Task: Given a graphical model p(x), find marginal probabilitypi (xi )pi (xi ) =
∑x/xi
p(x)
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 5 / 22
![Page 6: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/6.jpg)
Tasks: MAP
Maximum A Posteriori(MAP) Inference: x∗ = argmaxxp(x), Finding themost probable state
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 6 / 22
![Page 7: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/7.jpg)
Belief Propagation
Belief propagation operates on these factor graphs by constructingmessages µi→α and µα→i that are passed between variable(i) andfactor(α) nodes:
µα→i (xi ) =∑xα\xi
ψα(xα)∏
j∈Nα\i
µj→α(xj) (1)
µi→α(xi ) =∏
β∈Ni\α
µβ→i (xi ) (2)
the estimated marginal joint probability of a factor α, namely Bα(xα), isgiven by
Bα(xα) =1
Zψα(xα)
∏i∈Nα
µi→α(xi ) (3)
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 7 / 22
![Page 8: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/8.jpg)
Belief Propagation
Issues:
Exact Inference on tree graphs, but not on graphs with cycles
Update Steps may not have closed form solutions
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 8 / 22
![Page 9: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/9.jpg)
Special Case: Binary Markov Random Field
variables x ∈ {+1,−1}|V |
p(x) = 1Z exp (b · x + x · J · x) (4)
singleton factor: ψi (xi ) = ebixi
pairwise factors: ψi ,j(xi , xj) = eJijxixj
Goal: find p(xi )
J is a symmetric matrix
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 9 / 22
![Page 10: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/10.jpg)
Belief Propagation on Binary Markov Fields
Belief propagation updates messages µij from i to j according to
µij(xj) =∑xi
eJijxixj+bixi∏
k∈Ni\j
µki (xi ) (5)
estimated marginals by pi (xi ) = 1Z e
bixi∏
k∈Niµki (xi )
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 10 / 22
![Page 11: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/11.jpg)
Proposed GNN architecture
mt+1i→j =M(hti ,h
tj , εij) (6)
mt+1i =
∑j∈Ni
mt+1j→i (7)
ht+1i = U(hti ,m
t+1i ) (8)
y = σ(R(h
(T )i )
)(9)
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 11 / 22
![Page 12: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/12.jpg)
Proposed Model: Message-GNN
Convert all messages µi→j into a node in a GNN hi→j
Two GNN nodes v and w are connected if they correspond tomessages µi→j and µj→k
message from vi to vj is computed by mt+1i→j =M(
∑k∈Ni\j h
tk→i , eij).
update its hidden state by ht+1i→j = U(hti→j ,m
t+1i→j ).
Readout: pi (xi ) = R(∑
j∈Nih(T )j→i ).KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 12 / 22
![Page 13: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/13.jpg)
Proposed Model: node-GNN
No representation for factor nodes
information about interactions in εij
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 13 / 22
![Page 14: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/14.jpg)
GNN for inference and MAP
minimize cross entropy loss L(p, p) = −∑
i qi log pi (xi )
For MAP: delta function qi = δxi ,x∗iFor Marginal Inference: qi enumeration of ground truth
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 14 / 22
![Page 15: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/15.jpg)
Experimental Design
generalization under 4 conditions
to unseen graphs of the same structure (I, II),
and to completely different random graphs (III, IV).
These graphs may be the same size (I, III) or larger (II, IV).
structured randomn = 9 I IIIn = 16 II IV
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 15 / 22
![Page 16: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/16.jpg)
Experimental Set up
train on 100 graphical models of 13 classical types
Sample Jij = Jji ∼ N (0, 1)
sample biases bi ∼ N (0, (1/4)2)
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 16 / 22
![Page 17: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/17.jpg)
Within Set Generalization
test graphs had the same size and structure as training graphsbut the values of singleton and edge potentials differedmost notable performance difference between loopy graphs
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 17 / 22
![Page 18: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/18.jpg)
Out of Set Generalization
Train on same graphs
Test on bigger graphs
Metric: the average Kullback-Leibler divergence 〈DKL[pi (xi )‖pi (xi )]〉across the entire set of test graphs with the small and large numberof nodes.
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 18 / 22
![Page 19: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/19.jpg)
Out of Set Generalization: different structure
connected random Erdos Renyi graphs Gn,q,changed connectivity by increasing the edge probability from q = 0.1(sparse) to 0.9 (dense)
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 19 / 22
![Page 20: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/20.jpg)
Convergence of Inference Dynamics
How node states change over time
||htv − ht−1v ||`2
0 100.0
1.0
time step
dist
ance
I
II
III
IV
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 20 / 22
![Page 21: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/21.jpg)
MAP Estimation
x∗ = argmaxxp(x)
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 21 / 22
![Page 22: Inference in Probabilistic Graphical Models by …2019/04/12 · Inference in Probabilistic Graphical Models by Graph Neural Networks Author KiJung Yoon, Renjie Liao, Yuwen Xiong,](https://reader033.vdocument.in/reader033/viewer/2022052717/5f03f4757e708231d40b985f/html5/thumbnails/22.jpg)
Conclusions
limited testing: binary markov random field models only
relatively small graphs
A combination of NNs approximation power to incorporate non linearstructure of inference problems
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard Zemel, Xaq PitkowInference in Probabilistic Graphical Models by Graph Neural NetworksPresenter: Arshdeep Sekhon 22 / 22