physical fluctuomatics 7th~10th belief propagation appendix
DESCRIPTION
Physical Fluctuomatics 7th~10th Belief propagation Appendix. Kazuyuki Tanaka Graduate School of Information Sciences, Tohoku University [email protected] http://www.smapip.is.tohoku.ac.jp/~kazu/. - PowerPoint PPT PresentationTRANSCRIPT
Physics Fluctuomatics (Tohoku University) 1
Physical Fluctuomatics7th~10th Belief propagation
Appendix
Kazuyuki TanakaGraduate School of Information Sciences, Tohoku University
[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/
Physics Fluctuomatics (Tohoku University) 2
Textbooks
Kazuyuki Tanaka: Introduction of Image Processing by Probabilistic Models, Morikita Publishing Co., Ltd., 2006 (in Japanese) , Chapter 5.
ReferencesH. Nishimori: Statistical Physics of Spin Glasses and Information Processing, ---An Introduction, Oxford University Press, 2001. H. Nishimori, G. Ortiz: Elements of Phase Transitions and Critical Phenomena, Oxford University Press, 2011.M. Mezard, A. Montanari: Information, Physics, and Computation, Oxford University Press, 2010.
Physics Fluctuomatics (Tohoku University) 3
Probabilistic Model for Ferromagnetic Materials
p p
p p
)1,1()1,1()1.1()1.1( PPPP
pPP )1.1()1,1(
11 a
1
12 a
1
11
1 1
p
PP
21
)1.1()1,1(
Physics Fluctuomatics (Tohoku University) 4
Probabilistic Model for Ferromagnetic Materials
Prior probability prefers to the configuration with the least number of red lines.
> >=
Lines Red of #Lines Blue of # )21()( ppaP
p p
11 a 112 a 111 1 1
Physics Fluctuomatics (Tohoku University) 5
More is different in Probabilistic Model for Ferromagnetic Materials
Disordered State
Ordered State
Sampling by Markov Chain Monte Carlo method
p p
Small p Large p
p p
More is different.
p21 p
21
Critical Point(Large fluctuation)
Physics Fluctuomatics (Tohoku University) 6
Fundamental Probabilistic Models for Magnetic Materials
Since h is positive, the probablity of up spin is larger than the one of down spin .
1)exp(
)exp()(
aha
haaP1a
+1 1
he he
)tanh()(1
haaPma
h : External Field
)(tanh1)()( 2
1
2 haPmaaVa
Variance
Average
0h
Physics Fluctuomatics (Tohoku University) 7
Fundamental Probabilistic Models for Magnetic Materials
Since J is positive, (a1,a2)=(+1,+1) and (1,1) have the largest probability .
1 121
2121
1 2
)exp()exp(
),(
a aaJa
aJaaaP
11 a
0),(1 1
21111 2
a a
aaPam
J : Interaction
1),()(1 1
212
1111 2
a a
aaPmaaVVariance
Average
0J
Je Je
+1 +1 1 1
+1 +1 11
12 aJe Je
Physics Fluctuomatics (Tohoku University) 8
Fundamental Probabilistic Models for Magnetic Materials
a
aEZ
))(exp(
Eji
jiVi
i aaJahaE},{
)(
Translational Symmetry
),( EVJJ
h h
)(exp1)( aEZ
aP
),,,( 21 Naaaa
E : Set of All the neighbouring Pairs of Nodes
1ia 1ia
N
i ai aPa
Nm
1)(1
Problem: Compute
)'()()'()( aPaPaEaE
Physics Fluctuomatics (Tohoku University) 9
Fundamental Probabilistic Models for Magnetic Materials
Eji
jiVi
i aaJahaE},{
)(
N
i ai
NhaPa
Nm
10)(1limlim
)(exp1)( aEZ
aP
),,,( 21 Naaaa
1ia
Problem: Compute
Translational Symmetry
),( EV
J
J
h h
Spontaneous Magnetization
Physics Fluctuomatics (Tohoku University) 10
Mean Field Approximation for Ising Model
)},{( 0))(( Ejimama ji We assume that the probability for configurations satisfying
Vi
iaJmhaE )4()(
2mmamaaa ijji
Eji
jiVi
i aaJahaE},{
iJm
Jm
JmJm
h
are large.
Physics Fluctuomatics (Tohoku University) 11
Mean Field Approximation for Ising Model
)4tanh()(1
1JmhaPa
Nm
N
i ai
Vi
ii aPaEZ
aP )())(exp(1)(
Fixed Point Equation of m)(mm
We assume that all random variables ai are independent of each other, approximately.
Vi
iaJmhaE )4()(
Physics Fluctuomatics (Tohoku University) 12
Fixed Point Equation and Iterative Method
• Fixed Point Equation ** MM
Physics Fluctuomatics (Tohoku University) 13
Fixed Point Equation and Iterative Method
• Fixed Point Equation ** MM • Iterative Method
0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 14
Fixed Point Equation and Iterative Method
• Fixed Point Equation ** MM • Iterative Method
0M0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 15
Fixed Point Equation and Iterative Method
• Fixed Point Equation ** MM • Iterative Method
01 MM
0M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 16
Fixed Point Equation and Iterative Method
• Fixed Point Equation ** MM • Iterative Method
12
01
MMMM
0M1M
1M
0
xy
)(xy
y
x*M
Physics Fluctuomatics (Tohoku University) 17
Fixed Point Equation and Iterative Method
• Fixed Point Equation ** MM • Iterative Method
12
01
MMMM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics (Tohoku University) 18
Fixed Point Equation and Iterative Method
• Fixed Point Equation ** MM • Iterative Method
23
12
01
MMMMMM
0M1M
1M
0
xy
)(xy
y
x*M
2M
Physics Fluctuomatics (Tohoku University) 19
Marginal Probability Distribution in Mean Field Approximation
))4exp((1
)()(1 2 1 1
ii
a a a a aii
aJmhZ
aPaPi i N
iJm
Jm
JmJm
h
1
)(ia
iii aPam
))4tanh(( mJhm Jm: Mean Field
Physics Fluctuomatics (Tohoku University) 20
Advanced Mean Field Method
))4exp((1)( ii
ii ahZ
aP
)))(3exp((1),( jijii
jiij aJaaahZ
aaP
h
h
h
1
),()(ja
jiijii aaPaP
))3tanh()(tanh(arctanh hJ
Bethe Approximation
Kikuchi Method (Cluster Variation Meth)
: Effective Field
Fixed Point Equation for
J
Physics Fluctuomatics (Tohoku University) 21
Average of Ising Model on Square Grid Graph
(a) Mean Field Approximation(b) Bethe Approximation(c) Kikuchi Method (Cluster Variation Method)(d) Exact Solution ( L. Onsager , C.N.Yang )
J/1
a
iNh
aPa
)(limlim0
Ejiji
Vii aaJah
ZaP
},{
exp1 ),( EVJJ
h h
Physics Fluctuomatics (Tohoku University) 22
Model Representation in Statistical Physics
),,,(},,,Pr{ 212211 NNN aaaPaAaAaA
a
aEZ
))(exp(
)(}Pr{ aPaA
))(exp(1)( aEZ
aP
),,,( 21 NAAAA
Gibbs Distribution Partition Function
)))(exp(ln(ln a
aEZF
Free Energy
Energy Function
Physics Fluctuomatics (Tohoku University) 23
Gibbs Distribution and Free Energy
Gibbs Distribution
ZPFaQQFaQ
ln][}1)(|][{min
))(exp(1)( aEZ
aP
)(ln)()()(][ aQaQaQaEQFaa
Variational Principle of Free Energy Functional F[Q] under Normalization Condition for Q(a)
Free Energy Functional of Trial Probability Distribution Q(a)
a
aEZ
))(exp(lnlnFree Energy
Physics Fluctuomatics (Tohoku University) 24
Explicit Derivation of Variantional Principle for Minimization of Free Energy Functional
ZPFaQQFaQ
ln][}1)(|][{min
)(
)(exp)(exp)(ˆ aPaEaEaQ
a
1)()())(ln)((1)(
aaaaQaQaQaEaQQFQL
01)(ln)()(
aQaEaQQL
1)(exp)(ˆ aEaQ
Normalization Condition
Physics Fluctuomatics (Tohoku University) 25
Kullback-Leibler Divergence and Free Energy
0)()(ln)(
aP
aQaQPQDa
aaQaQ
1)( ,0)(
ZQF
ZaQaQaEaQPQD
QFaa
ln][
ln)(ln)()()(]|[
][
0)()( PQDaPaQ
))(exp(1)( aEZ
aP
}1)(|]|[{minarg}1)(|][{minarg aQaQ
aQPQDaQQF
Physics Fluctuomatics (Tohoku University) 26
Interpretation of Mean Field Approximation as Information Theory
Vi
ii aQaQ )()(
)(
)(ln)(aPaQaQPQD
a
))(exp(1 aEZ
aP and
Marginal Probability Distributions Qi(ai) are determined so as to minimize D[Q|P]
1 2 1 1 2
)()()(\ a a a a a aaa
iii i i Ni
aQaQaQ
Minimization of Kullback-Leibler Divergence between
Physics Fluctuomatics (Tohoku University) 27
Interpretation of Mean Field Approximation as Information Theory
Eji
jiVi
i aaJahaE},{
)(
Vi a
iiVi a
ii
aPaV
aPaV
m1
)(||
1)(||
1
)(exp1)( aEZ
aP
),,,( ||21 Vaaaa
1ia
Problem: Compute
Translational Symmetry
),( EV
J
J
h h
Magnetization
1 2 1 1 2
)()()(\ a a a a a aaa
iii i i Ni
aPaPaP
Physics Fluctuomatics (Tohoku University) 28
Kullback-Leibler Divergence in Mean Field Approximation for Ising Model
Vi
ii aQaQ )()(
ZViQFPQD i ln|MF
)(
)(ln)(aPaQaQPQD
a
Viii
Ejiji
Viii
QQQQJ
QhViQF
1},{ 11
1MF
ln))()()((
)(}]|[{
1 2 1 1 2
)(
)()(\
a a a a a a
aaii
i i i N
i
aQ
aQaQ
Eji
jiVi
i aaJahaE},{
)( )(exp1)( aE
ZaP
Physics Fluctuomatics (Tohoku University) 29
Minimization of Kullback-Leibler Divergence and Mean Field Equation
)( ))(ˆ(exp1ˆ1
ViQJhZ
Qij
ji
i
} ,1)(|]|[{minarg)}(ˆ{}{
ViQPQDQ iQii
Fixed Point Equations for {Qi|iV}
Variation
1 1
))(ˆ(exp
ij
ji QJhZi
Set of all the neighbouring nodes of the node i
Ejiji },{
Physics Fluctuomatics (Tohoku University) 30
Orthogonal Functional Representation of Marginal Probability Distribution of Ising Model
iiii amaQ21
21)(
1
)(ia
iiia
ii aQaaQam
),,,( 21 Naaaa 1ia
ia
iiia
iia
iii
aii
ai
aii
iiii
maQadddacaaQa
aQccdacaQ
adacaQ
iii
iii
21)(
212)()(
21)(
212)()(
1)( )(
111
111
2
1 2 1 1
)()()(\ a a a a aaa
iii i Ni
aQaQaQ
Physics Fluctuomatics (Tohoku University) 31
Conventional Mean Field Equation in Ising Model
)4tanh( Jmhm
iiiiiaa
ii maamaQaPaPi
21
21
21
21)(ˆ)()(
\
maPaN
N
i ai
1)(1
Fixed Point Equation
mmmm N 21
Eji
jiVi
i aaJahaE},{
)(
))4exp((1))(ˆ(exp1)(ˆ1
ii
iij
ji
ii aJmhZ
aQJhZ
aQ
VJ
J
Translational Symmetry
h h
)( 4|| Vii
Physics Fluctuomatics (Tohoku University) 32
Interpretation of Bethe Approximation (1)
Eji
jiVi
i aaJahaE},{
)(
)(exp1)( aEZ
aP
),,,( ||21 Vaaaa
1ia
Translational Symmetry
),( EV
J
J
h h
1 2 1 1 2
)()()(\ a a a a a aaa
iii i i Ni
aPaPaP
1 2 1 1 2 1 1 2
)()(),(},\{ a a a a a a a a aaaa
jiiji i i j j j Nji
aPaPaaP
Eji
jiij aaZ
aP},{
),(1)(
jijijiij aJaha
jha
iaa
||1
||1exp),(
a Eji
jiij aaZ
},{
),(
Compute
and
Interpretation of Bethe Approximation (2)
ZQFPQD ln
aQaQaaWaaQ
aQaQaaWaQ
aQaQaaWaQQF
aEji a ajiijjiij
aEji a ajiij
aaa
aEjijiij
a
i j
i j ji
ln)(,ln,
ln)(,ln)(
ln)(,ln)(
},{
},{ ,\
},{
0ln)(
aP
aQaQPQDa
Free EnergyKL Divergence
Eji
jiij aaWZ
P},{
,1x
ji aaa
jiij
aQ
aaQ
,
)(
),(
\
33Physics Fluctuomatics (Tohoku
University)
Interpretation of Bethe Approximation (3)
ZQFPQD ln
Ejijjiiijij
Viii
Ejiijij
a
Ejiijij
QQQQQQ
WQ
aQaQ
WQQF
},{
},{
},{
lnln,ln,
ln
,ln,
ln)(
,ln,
Bethe Free Energy
Free EnergyKL Divergence
Eji
jiij aaWZ
aP},{
,1
ji aaajiij aQaaQ
,
)(),(\
iaaii aQaQ
\
)()(
34Physics Fluctuomatics (Tohoku
University)
Interpretation of Bethe Approximation (4)
FPQDQQ
minargminarg
,iji QQ
ZQQFPQD iji ln,Bethe
ijiQQQ
QQFPQDiji
,minargminarg Bethe,
1,
iji QQ
Ejijjiiijij
Viii
Ejiijijiji
QQQQQQ
QQWQQQF
},{
},{Bethe
lnln,ln,
ln,ln,,
35Physics Fluctuomatics (Tohoku
University)
Interpretation of Bethe Approximation (5)
Ejiijij
Viii
Vi ijijijii
ijiiji
QQFQQL
},{
},{,
BetheBethe
1,1
,
,,
1, ,,,minarg Bethe,
ijiijiijiQQ
QQQQQQFiji
Lagrange Multipliers to ensure the constraints
36Physics Fluctuomatics (Tohoku
University)
Interpretation of Bethe Approximation (6)
Ejiijij
Viii
Vi ijijijji
Ejijjiiijij
Viii
Ejiijij
Ejiijij
Viii
Vi ijijijiiijiiji
QQQQ
QQQQQQ
QQQ
QQQQFQQL
},{},{,
},{
},{
},{
},{,BetheBethe
1,1,
lnln,ln,
ln,ln,
1,1
,,,
0,Bethe
iji
ii
QQLxQ
• Extremum Condition
0,, Bethe
ijijiij
QQLxxQ
37Physics Fluctuomatics (Tohoku
University)
Interpretation of Bethe Approximation (7)
FGfP yxyxyx g ,,,
ik
ikiiii ai
aQ )(1||
1exp },{, )()(exp,, 2}2,1{,21}2,1{,121122112 aaaaaaQ
Extremum Condition 0,Bethe
ijiii
QQLxQ 0,
, Bethe
iji
jiij
QQLxxQ
38Physics Fluctuomatics (Tohoku
University)
115114
1131121
111
aMaM
aMaMZ
aQ
2282272262112
11511411312
2112
,
1,
aMaMaMaaW
aMaMaMZ
aaQ
)()(exp\
},{, ijik
ikijii aMa
Interpretation of Bethe Approximation (8)
FGfP yxyxyx g ,,,14 2
5
13M
14M
15M
12M
3
115114
1131121
111
aMaM
aMaMZ
aQ
2282272262112
11511411312
2112
,
1,
aMaMaMaaW
aMaMaMZ
aaQ
Extremum Condition 0,Bethe
ijiii
QQLxQ 0,
, Bethe
iji
jiij
QQLxxQ
39Physics Fluctuomatics (Tohoku
University)
26M
14
5
13M
14M
15M
12W3
2
6
27M
8
7
28M
Interpretation of Bethe Approximation (9)
FGfP yxyxyx g ,,,14 2
5
13M
14M
15M
12M
3
412W
1
5
13M
14M
15M
3
26M2
6
27M
8
7
28M
,121 QQ
115114
1131121
111
aMaM
aMaMZ
aQ
2282272262112
11511411312
2112
,
1,
aMaMaMaa
aMaMaMZ
aaQ
1514
1312
21
,
MM
M
M
Message Update Rule
40Physics Fluctuomatics (Tohoku
University)
Interpretation of Bethe Approximation (10)
15141312
15141312
21 ,
,
MMMW
MMMW
M
1
3
4 2
5
13M
14M
15M
21M
14
5
3
2
6
8
7
2a
14 2
5
3
=
Message Passing Rule of Belief Propagation
It corresponds to Bethe approximation in the statistical mechanics.
41Physics Fluctuomatics (Tohoku
University)
Interpretation of Bethe Approximation (11)
1 1 \
1 \,
,
jikikij
jikikij
ji MW
MW
M
42Physics Fluctuomatics (Tohoku
University)
))tanh()(tanh(arctanh\
jik
ikji hJ
jijiM exp
))3tanh()(tanh(arctanh hJ
ji Translational Symmetry
Physics Fluctuomatics (Tohoku University) 43
Summary
Statistical Physics and Information TheoryProbabilistic Model of FerromagnetismMean Field TheoryGibbs Distribution and Free EnergyFree Energy and Kullback-Leibler DivergenceInterpretation of Mean Field Approximation as Information TheoryInterpretation of Bethe Approximation as Information Theory