3 march, 2003university of glasgow1 statistical-mechanical approach to probabilistic inference ---...
TRANSCRIPT
3 March, 2003 University of Glasgow 1
Statistical-Mechanical Approach to Probabilistic Inference--- Cluster Variation Method
and Generalized Loopy Belief Propagation ---
Statistical-Mechanical Approach to Probabilistic Inference--- Cluster Variation Method
and Generalized Loopy Belief Propagation ---
Kazuyuki TanakaGraduate School of Information Sciences
Tohoku University, [email protected]
http://www.statp.is.tohoku.ac.jp/~kazu/index-e.html
Kazuyuki TanakaGraduate School of Information Sciences
Tohoku University, [email protected]
http://www.statp.is.tohoku.ac.jp/~kazu/index-e.html
3 March, 2003 University of Glasgow 2
Contents
1. Introduction2. Probabilistic Inference3. Cluster Variation Method4. Generalized Loopy Belief Propagation5. Linear Response6. Numerical Experiments7. Concluding Remarks
3 March, 2003 University of Glasgow 3
IntroductionProbabilistic Inference and Belief Propagation
Probabilistic Inference Probabilistic Model
Bayes Formula
BeliefMarginal Probability
BeliefPropagation
Probabilistic models on tree-like networks
with no loops=>Exact Results
Probabilistic models on networks
with some loops=>Good Approximation
Generalization
3 March, 2003 University of Glasgow 4
IntroductionStatistical Mechanics and Belief Propagation
Belief PropagationProbabilistic model with no loop
Probabilistic model with some loops(Lauritzen, Pearl)
Probabilistic model with no loop Transfer Matrix
Recursion Formula for Beliefs and Messages
Probabilistic model with some loops
Bethe/Kikuchi MethodCluster Variation Method
Transfer Matrix =
Belief Propagation
3 March, 2003 University of Glasgow 5
Introduction
Purpose
•Review of generalized loopy belief propagation and cluster variation method.•Calculation of correlations between any pair of nodes by combining the cluster variation method with the linear response theory
3 March, 2003 University of Glasgow 7
Probabilistic InferenceProbabilistic Inference and Probabilistic Model
)()()()(
)()()()(
)|()|(
)|()|(
),|(
),|(1
),,,,,,,(
88776655
44332211
13132424
25256767
436346
658568
87654321
xVxVxVxV
xVxVxVxV
xxVxxV
xxVxxV
xxxV
xxxVZ
xxxxxxxxP
1X
3X
2X
4X
6X 5X
8X7X
13V
67V
24V
25V346V
568V
1V 2V
3V4V
5V
8V7V
6V
3 March, 2003 University of Glasgow 8
Probabilistic InferenceProbabilistic Inference and Probabilistic Model
)()(
),(),(),(
)()()(
),(),,(),,(1
),,,,,,,(
2233
311342245225
44552
66
7667543346865568
87654321
xWxW
xxWxxWxxW
xWxWxW
xxWxxxWxxxW
Z
xxxxxxxxP
)()()(),|(),,(
)()()|(),(
664433436346643346
442224244224
xVxVxVxxxVxxxW
xVxVxxVxxW
1X
3X
2X
4X
6X 5X
13W
67W
24W
25W346W
568W
8X7X
3W4W
5W
6W
3 March, 2003 University of Glasgow 9
Probabilistic Inference
Probabilistic Model and Beliefs
)8,7,6,5,4,3,2,1(
),,,,,,,()(\
87654321
i
xxxxxxxxPxPi
iixx
Statistics: Marginal Probability
Statistical Mechanics: One-body distribution
Probabilistic Inference: Belief
1X
3X
2X
4X
6X 5X
13W
67W
24W
25W346W
568W
8X7X
3W4W
5W
6W
3 March, 2003 University of Glasgow 10
Cluster Variation Method
Probabilistic Inference and Probabilistic Model
)()(
),(),(),(
)()()(
),(),,(),,(
),,,,,,,(
2233
311342245225
44552
66
7667543346865568
87654321
xQxQ
xxQxxQxxQ
xQxQxQ
xxQxxxQxxxQ
xxxxxxxxQ
)( ofy Probabilit Marginal : )()( xxxx\x
QQQ
ZDDDD
DDDDDDPfD
ln2 6543
56867252434613
0ln)(
x
xx
x P
QQPQD
x
x
x
1)(
0
Q
Q 1X
3X
2X
4X
6X 5X
13W
67W
24W
25W346W
568W
8X7X
3W4W
5W
6W
67W
1X
3X
2X
4X
6X 5X
8X7X
13W 24W
25W
346W
568W
3X
3X
6X
5X
5X
4X
4X
6X
6X
2X2X
3W
2W
4W
5W
6W
3 March, 2003 University of Glasgow 11
Cluster Variation Method
ZWQDWQD
WQDWQD
WQDWQD
WQDWQD
WQDWQDPQD
ln|2|
||
||
||
||
6655
4433
5685686767
25252424
3463461313
1 4 6
54
,),,(),()(
,),(),()(
643346311333
5225422422
x x x
xx
xxxQxxQxQ
xxQxxQxQ
1)(),()(]|[minarg}|{\}|{
xxxCxxx
QQQPQDPCQ
1X
3X
2X
4X
6X 5X
8X7X
13W 24W
25W
346W
568W
3X
3X
6X
5X
5X
4X
4X
6X
6X
2X2X
3W
2W
4W
5W
6W
3 March, 2003 University of Glasgow 12
Generalized Loopy Belief Propagation
Extreme Condition of Kullback-Leibler Divergence
3 4 6
3 4
)()(),,(
)()(),,(
)(42443133643346
42443133643346
63466
x x x
x x
xmxmxxxW
xmxmxxxW
xm
Message Passing Algorithm
1X
3X
2X
4X
6X
)( 3133 xm )( 4244 xm
)( 63466 xm
3 March, 2003 University of Glasgow 13
Generalized Loopy Belief Propagation
Expression of Marginal Probability in terms of Messages
3 4 6)()(
)()(),,(
)()(
)()(),,(
),,(
656866676
42443133643346
656866676
42443133643346
643346
x x x xmxm
xmxmxxxW
xmxm
xmxmxxxW
xxxP
1X
3X
2X
4X
6X 5X
8X7X
346W
)( 3133 xm )( 4244 xm
)( 6676 xm
)( 65686 xm
3 March, 2003 University of Glasgow 14
Generalized Loopy Belief Propagation
1X
3X
2X
4X
6X 5X
13W
67W
24W
25W346W
568W
8X7X
3W4W
5W
6W
67W
1X
3X
2X
4X
6X 5X
8X7X
13W 24W
25W
346W
568W
3X
3X
6X
5X
5X
4X
4X
6X
6X
2X2X
3W
2W
4W
5W
6W
)( )(
)( )( )(
),( ),,(
),,( ),(
),( ),(
6655
443322
7667865568
6433465225
42243113
xPxP
xPxPxP
xxPxxxP
xxxPxxP
xxPxxP
5 6
6
3
),,()(
),()(
),()(
86556888
766777
311311
x x
x
x
xxxPxP
xxPxP
xxPxP
3 March, 2003 University of Glasgow 15
Linear Response
ji
ii xiii
xiiiiii xPxxPxPxPxx
Nodeat Field External respect to with Nodeat Average ofDeviation
)()(~
)()(~
xx
xx
i
j
hjiji h
xxxxx
i
P
0
lim
)(in Covariant x
x
x)(Pxx iix
x)(Pxxxx jiji
)exp()(~1
)(~
iixhPZ
P xx
3 March, 2003 University of Glasgow 16
Numerical Experiments
Beliefs
0104.0)1(9896.0)1( 33 PP
0104.0)1(9896.0)1( 33 PP
4393.0)1(5607.0)1( 88 PP
4360.0)1(5640.0)1( 88 PP
Cluster Variation Method
Exact4500.0)1(5500.0)1( 55 PP
4500.0)1(5500.0)1( 55 PP
1X
3X
2X
4X
6X 5X
13W
67W
24W
25W346W
568W
8X7X
3W4W
5W
6W
i
xxxxxxxxPxP iixx \
87654321 ),,,,,,,()(
3 March, 2003 University of Glasgow 17
Numerical Experiments
Correlation Functions8544.061 xx
1334.083 xx
0890.062 xx
0828.072 xx
8544.061 xx
1402.083 xx
0890.062 xx
0828.072 xx
Cluster Variation Method
Exact
1X
3X
2X
4X
6X 5X
13W
67W
24W
25W346W
568W
8X7X
3W4W
5W
6W
i jx x
jiijjijiji xxPxxPxxxx ),()(x
x
3 March, 2003 University of Glasgow 18
Concluding Remarks
Summary
•Review of Cluster Variation Method and Loopy Belief Propagation•Calculation of Correlation Function by means of Linear Response Theory and Cluster Variation Method
Future Problem•Learning of Hyperparameters by means of Maximum Likelihood Estimation