junction tree algorithm 10-708:probabilistic graphical models recitation: 10/04/07 ramesh nallapati
TRANSCRIPT
Junction tree Algorithm
10-708:Probabilistic Graphical Models
Recitation: 10/04/07
Ramesh Nallapati
Cluster Graphs
A cluster graph K for a set of factors F is an undirected graph with the following properties: Each node i is associated with a subset Ci ½ X Family preserving property: each factor is such that
scope[] µ Ci
Each edge between Ci and Cj is associated with a sepset Sij = Ci Å Cj
Execution of variable elimination defines a cluster-graph Each factor used in elimination becomes a cluster-node An edge is drawn between two clusters if a message is
passed between them in elimination Example: Next slide
Variable Elimination to Junction Trees:
Original graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
Variable Elimination to Junction Trees:
Moralized graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
Variable Elimination to Junction Trees:
Triangulated graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I
Variable Elimination to Junction Trees:
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
G,S
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,S
G,J
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,SJ,S,L
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,S
J,S,LJ,S,L
L,J
Variable Elimination to Junction Trees:
Elimination ordering: C, D, I, H, G, S, L
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,S
J,S,LJ,S,L
L,J
L,J
Properties of Junction Tree
Cluster-graph G induced by variable elimination is necessarily a tree Reason: each intermediate factor is used atmost
once G satisfies Running Intersection Property (RIP)
(X 2 Ci & X in Cj) ) X 2 CK where Ck is in the path of Ci and Cj
If Ci and Cj are neighboring clusters, and Ci passes message mij to Cj, then scope[mij] = Si,j
Let F be set of factors over X. A cluster tree over F that satisfies RIP is called a junction tree
One can obtain a minimal junction tree by eliminating the sub-cliques No redundancies
C,D
D,I,G
D
G,I,S
G,I
H,G,J
G,J
G,J,S,L
G,S
J,S,LJ,S,L
L,J
L,J
Junction Trees to Variable elimination:
Now we will assume a junction tree and show how to do variable elimination
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
1: C,D
2: G,I,D
3: G,S,I
4: G,J,S,L
5: H,G,J
D
G,I
G,S
G,J
Junction Trees to Variable Elimination:
Initialize potentials first:
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
1: C,D
2: G,I,D
3: G,S,I
4:G,J,S,L
5:H,G,J
D
G,I
G,S
G,J
01(C,D) =
P(C)P(D|C)
02(G,I,D) = P(G|
D,I)
03(G,S,I) =
P(I)P(S|I)
04(G,J,S,L) = P(L|
G)P(J|S,L)
05(H,G,J) = P(H|
G,J)
Junction Trees to Variable Elimination:
Pass messages: (C4 is the root)
1: C,D
2: G,I,D
3: G,S,I
4:G,J,S,L
5:H,G,J
D
G,I
G,S
G,J
01(C,D) =
P(C)P(D|C)
02(G,I,D) = P(G|
D,I)
03(G,S,I) =
P(I)P(S|I)
04(G,J,S,L) = P(L|
G)P(J|S,L)
05(H,G,J) = P(H|
G,J)
1! 2(D) = C 01(C,D)
2! 3(G,I) = D 0
2(G,I,D)1! 2(D)
3! 4(G,S) = I 0
3(G,S,I)2! 3(G,I)
5! 4(G,J) = H 05(H,G,J)
4(G,J,S,L) = 3 ! 4(G,S)5 ! 4(G,J)0
4(G,J,S,L)
Junction Tree calibration
Aim is to compute marginals of each node using least computation Similar to the 2-pass sum-product algorithm
Ci transmits a message to its neighbor Cj after it receives messages from all other neighbors
Called “Shafer-Shenoy” clique tree algorithm
1: C,D 2: G,I,D 3: G,S,I 4:G,J,S,L 5:H,G,J
Message passing with division
Consider calibrated potential at node Ci
whose neighbor is Cj
Consider message from Ci to C
j
Hence, one can write:
Ci
Cj
Message passing with division
Belief-update or Lauritzen-Speigelhalter algorithm Each cluster Ci maintains its fully updated current
beliefs i
Each sepset sij maintains ij, the previous message passed between Ci-Cj regardless of direction
Any new message passed along Ci-Cj is divided by ij
Belief Update message passingExample
1: A,B 2: B,C 3: C,DB C
12 = 1 ! 2(B) 23 = 3 ! 2(C)
2! 1(B)
This is what we expect to send in the regular message passing!
Actual message
Belief Update message passingAnother Example
1: A,B 2: B,C 3: C,DB C
2 ! 3(C) = 023
3 ! 2(C) = 123
This is exactly the message C2 would have received from C3 if C2 didn’t send an uninformed message: Order of messages doesn’t matter!
Belief Update message passingJunction tree invariance
Recall: Junction Tree measure:
A message from Ci to Cj changes only j and ij:
Thus the measure remains unchanged for updated potentials too!
Junction trees from Chordal graphs
Recall: A junction tree can be obtained by the induced graph from variable elimination
Alternative approach: using chordal graphs Recall:
Any chordal graph has a clique tree Can obtain chordal graphs through triangulation
Finding a minimum triangulation, where largest clique has minimum size is NP-hard
Junction trees from Chordal graphsMaximum spanning tree algorithm
Original Graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
Junction trees from Chordal graphsMaximum spanning tree algorithm
Undirected moralized graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
Junction trees from Chordal graphsMaximum spanning tree algorithm
Chordal (Triangulated) graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
Junction trees from Chordal graphsMaximum spanning tree algorithm
Cluster graph
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
1
G,I,S
2
L,S,J
2
G,S,L
2
G,H1 1
1
1
1
Junction trees from Chordal graphsMaximum spanning tree algorithm
Junction tree
Difficulty Intelligence
Coherence
Grade SAT
Happy
Letter
Job
C,D
D,I,G
D
G,I,S
G,I
L,S,J
S,L
G,S,L
G,S
G,HG
Summary
Junction tree data-structure for exact inference on general graphs
Two methods Shafer-Shenoy Belief-update or Lauritzen-Speigelhalter
Constructing Junction tree from chordal graphs Maximum spanning tree approach