14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 1
Statistical performance analysis by loopy belief propagation in
probabilistic image processing Kazuyuki Tanaka
Graduate School of Information Sciences,Tohoku University, Japan
http://www.smapip.is.tohoku.ac.jp/~kazu/
CollaboratorsD. M. Titterington (University of Glasgow, UK)
M. Yasuda (Tohoku University, Japan)S. Kataoka (Tohoku University, Japan)
2
IntroductionBayesian network is originally one of the methods for probabilistic inferences in artificial intelligence. Some probabilistic models for information processing are also regarded as Bayesian networks. Bayesian networks are expressed in terms of products of functions with a couple of random variables and can be associated with graphical representations. Such probabilistic models for Bayesian networks are referred to as Graphical Model.
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
3
Probabilistic Image Processing by Bayesian Network
Probabilistic image processing systems are formulated on square grid graphs. Averages, variances and covariances of the Bayesian network are approximately computed by using the belief propagation on the square grid graph.
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
10 March, 2010 IW-SMI2010 (Kyoto) 4
MRF, Belief Propagation and Statistical Performance
Nishimori and Wong (1999): Physical Review EStatistical Performance Estimation for MRF
(Infinite Range Ising Model and Replica Theory)
Is it possible to estimate the performance of belief propagation statistically?
Tanaka and Morita (1995): Physics Letters ACluster Variation Method for MRF in Image Processing
CVM= Generalized Belief Propagation (GBP)
Geman and Geman (1986): IEEE Transactions on PAMIImage Processing by Markov Random Fields (MRF)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 5
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 6
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
Image Restoration by Bayesian Statistics
Original Image
14 October, 2010 7LRI Seminar 2010 (Univ. Paris-Sud)
Image Restoration by Bayesian Statistics
Original Image
Degraded Image
Transmission
Noise
14 October, 2010 8LRI Seminar 2010 (Univ. Paris-Sud)
Image Restoration by Bayesian Statistics
Original Image
Degraded Image
Transmission
Noise
Estimate
14 October, 2010 9LRI Seminar 2010 (Univ. Paris-Sud)
10
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
Bayes Formula
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
14 October, 2010 10LRI Seminar 2010 (Univ. Paris-Sud)
11
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
Assumption 1: Original images are randomly generated by according to a prior probability.
Bayes Formula
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
14 October, 2010 11LRI Seminar 2010 (Univ. Paris-Sud)
7 July, 2010 12
Image Restoration by Bayesian Statistics
PriorProcessn Degradatio
Posterior
}Image OriginalPr{}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
∝
Bayes Formula
Assumption 2: Degraded images are randomly generated from the original image by according to a conditional probability of degradation process.
Original Image
Degraded Image
Transmission
Noise
Estimate
Posterior
12LRI Seminar 2010 (Univ. Paris-Sud)14 October, 2010
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 13
Bayesian Image Analysis
Prior Probability
−−=
−−∝= ∑∏
∈∈ Ejiji
Ejiji fffffF
},{
2
},{
2 )(21exp)(
21exp}|Pr{ ααα
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
0>α
14
Bayesian Image Analysis
Prior Probability
−−=
−−∝= ∑∏
∈∈ Ejiji
Ejiji fffffF
},{
2
},{
2 )(21exp)(
21exp}|Pr{ ααα
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
0>α
14 October, 2010 14LRI Seminar 2010 (Univ. Paris-Sud)
15
Bayesian Image Analysis
0005.0=α 0030.0=α0001.0=α
Patterns by MCMC.
Prior Probability
−−=
−−∝= ∑∏
∈∈ Ejiji
Ejiji fffffF
},{
2
},{
2 )(21exp)(
21exp}|Pr{ ααα
Assumption: Prior Probability consists of a product of functions defined on the neighbouring pixels.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
0>α
14 October, 2010 15LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 16
Bayesian Image Analysis
Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
∏∈
−−∝
==
Viii fg
fFgG
22 )(
21exp
},|Pr{
σ
σ
0>σV:Set of all the pixels
17
Bayesian Image Analysis
Assumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
14 October, 2010 17LRI Seminar 2010 (Univ. Paris-Sud)
∏∈
−−∝
==
Viii fg
fFgG
22 )(
21exp
},|Pr{
σ
σ
18
Bayesian Image AnalysisAssumption: Degraded image is generated from the original image by Additive White Gaussian Noise.
Prior
Likelihood
Posterior
}Image OriginalPr{
}Image Original|Image DegradedPr{
}Image Degraded|Image OriginalPr{
×
∝
∏∈
−−∝
==
Viii fg
fFgG
22 )(
21exp
},|Pr{
σ
σ
0>σ
14 October, 2010 18LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 19
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
Estimate
20
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
SmoothingData Dominant
Estimate
14 October, 2010 20LRI Seminar 2010 (Univ. Paris-Sud)
11 March, 2010 IW-SMI2010 (Kyoto) 21
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
SmoothingData Dominant
Bayesian Network Estimate
14 October, 2010 21LRI Seminar 2010 (Univ. Paris-Sud)
( ) g
zdgGzFz
ghf
12
},,|Pr{
),,(ˆ
−+=
===
=
∫CI ασ
σα
σα
22
Bayesian Image Analysis
f
g
}|Pr{ αfF
= },|Pr{ σfFgG
== gOriginalImage
Degraded Image
Prior ProbabilityPosterior Probability
Degradation Process
−−−−∝
====
=
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfFfFgG
gGfF
},{
222 )(
21)(
21exp
},|Pr{},Pr{},|Pr{
},,|Pr{
ασ
σαασ
σα
Field Random Markov Gauss),(
⇒
+∞−∞∈iF
SmoothingData Dominant
Bayesian Network Estimate
14 October, 2010 22LRI Seminar 2010 (Univ. Paris-Sud)
∈−∈=∂
=otherwise,0
},{,1|,|
EjiVjii
ji C
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 23
Image Restorations by Gaussian Markov Random Fields
and Conventional Filters
2||ˆ||
||1MSE ∑
∈−=
Viff
V
MSEGauss Markov Random Field 315
Lowpass Filter
(3x3) 388
(5x5) 413
Median Filter
(3x3) 486
(5x5) 445
(3x3) Lowpass (5x5) MedianGauss Markov Random Field
Original Image Degraded Image
RestoredImage V: Set of all
the pixels
Field Random Markov Gauss),(
⇒
+∞−∞∈iF
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 24
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 25
Statistical Performance by Sample Average of Numerical Experiments
f
Original Images
26
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
Original ImagesN
oise
Pr
{G|F
=f,σ
}
ObservedData
14 October, 2010 26LRI Seminar 2010 (Univ. Paris-Sud)
11 March, 2010 IW-SMI2010 (Kyoto) 27
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Post
erio
r Pr
obab
ility
Pr{F
|G=g
, α,σ
}
Original ImagesN
oise
Pr
{G|F
=f,σ
}
Estimated ResultsObserved
Data
14 October, 2010 27LRI Seminar 2010 (Univ. Paris-Sud)
28
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Post
erio
r Pr
obab
ility
Pr{F
|G=g
,α,σ
}
Sample Average of Mean Square Error
Original ImagesN
oise
Pr
{G|F
=f,σ
}
∑=
−≅5
1
2||||51),|,(
nnhffD
σσα
Estimated ResultsObserved
Data
14 October, 2010 28LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 29
Statistical Performance Estimation
gdfFgGfghV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
( ) g
gh
12
),,(−
+= CI ασ
σα
gg
Additive White Gaussian Noise
},|Pr{ σfFgG
==f
},,|Pr{ σαgGfF ==
Posterior Probability
RestoredImage
Original Image Degraded Image
},|Pr{ σfFgG
==
Additive White Gaussian Noise
∈−∈=∂
=otherwise,0
},{,1|,|
EjiVjii
ji C
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 30
Statistical Performance Estimation for Gauss Markov Random Fields
T22
242
22
2
22
||2
2
2
)(||1
)(Tr1
21exp
211
},|Pr{),,(1),|,(
fI
fVV
gdfggfV
gdfFgGghfV
fD
V
CC
CII
CII
ασσα
ασσ
σσπασ
σσασσα
++
+=
−−
+
−=
==−=
∫
∫
∈−∈=∂
=otherwise
EjiVjii
ji,0
},{,1|,|
C
0
200
400
600
0 0.001 0.002 0.003α
σ=40
0
200
400
600
0 0.001 0.002 0.003
σ=40
α
),|,( σσα fD
),|,( σσα fD
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 31
32
Marginal Probability in Belief Propagation
{ } { }∑∑∑ ∑ ===1 3 4
,,,,,PrPr 43212F F F F
NN
FFFFFF gGgG
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
In order to compute the marginal probability Pr{F2|G=g},we take summations over all the pixels except the pixel 2.
33
Marginal Probability in Belief Propagation
∑∑∑ ∑=1 3 4F F F FN
2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
{ } { }∑∑∑ ∑ ===1 3 4
,,,,,PrPr 43212F F F F
NN
FFFFFF gGgG
34
Marginal Probability in Belief Propagation
∑∑∑ ∑=1 3 4F F F FN
2 2≅
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
{ } { }∑∑∑ ∑ ===1 3 4
,,,,,PrPr 43212F F F F
NN
FFFFFF gGgG
In the belief propagation, the marginal probability Pr{F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixel 2.
35
Marginal Probability in Belief Propagation
∑∑ ∑=3 4F F FN
1 2
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
In order to compute the marginal probability Pr{F1,F2|G=g},we take summations over all the pixels except the pixels 1 and2.
{ } { }∑∑ ∑ ===3 4
,,,,,Pr,Pr 432121F F F
NN
FFFFFFF gGgG
36
Marginal Probability in Belief Propagation
∑∑ ∑=3 4F F FN
1 2 1 2≅
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
{ } { }∑∑ ∑ ===3 4
,,,,,Pr,Pr 432121F F F
NN
FFFFFFF gGgG
In the belief propagation, the marginal probability Pr{F1,F2|G=g} is approximately expressed in terms of the messages from the neighbouring region of the pixels 1 and 2.
Belief Propagation in Probabilistic Image Processing
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 37
Belief Propagation in Probabilistic Image Processing
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 38
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 39
Image Restorations by Gaussian Markov Random Fields and Conventional Filters
MSE
Gauss MRF (Exact) 315
Gauss MRF (Belief Propagation) 327
Lowpass Filter
(3x3) 388
(5x5) 413
Median Filter
(3x3) 486
(5x5) 445
Belief PropagationExact
Original Image Degraded Image
RestoredImage
V: Set of all the pixels
Field Random Markov Gauss),(
⇒
+∞−∞∈iF
( )2ˆ||
1MSE ∑∈
−=Vi
ii ffV
40
Gray-Level Image Restoration(Spike Noise)
Original Image
MSE:135MSE: 217
MSE: 371 MSE: 523
MSE: 244
MSE: 3469
MSE: 2075
Degraded Image
Belief Propagation Lowpass Filter Median Filter
MSE: 395
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 41
Binary Image Restoration byLoopy Belief Propagation
α∗=0.465
42
Statistical Performance by Sample Average of Numerical Experiments
1g
f
2g
3g
4g
5g
1h
2h
3h
4h
4h
Post
erio
r Pr
obab
ility
Pr{F
|G=g
,α,σ
}
Sample Average of Mean Square Error
Original ImagesN
oise
Pr
{G|F
=f,σ
}
∑=
−≅5
1
2||||51),|,(
nnhffD
σσα
Estimated ResultsObserved
Data
14 October, 2010 42LRI Seminar 2010 (Univ. Paris-Sud)
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 43
Statistical Performance Estimation for Binary Markov Random Fields
gdfFgGfghV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
||211 1 1 },{
222
22
1 2 ||
)(21)(
21explog
)(2
1exp
Vz z z Eji
jiVi
ii
Viii
dgdgdgzzgz
fg
V
−−−−×
−−
∑ ∑ ∑ ∑∑
∫ ∫ ∫ ∑
±= ±= ±= ∈∈
∞+
∞−
∞+
∞−
∞+
∞−∈
ασ
σ
It can be reduced to the calculation of the average of free energy with respect to locally non-uniform external fields g1, g2,…,g|V|.
Free Energy of Ising Model with Random External Fields
1±=ifLight intensities of the original image can be regarded as spin states of ferromagnetic system.
== >
== >
Eji ∈},{
−−−−∝
==
∑∑∈∈ Eji
jiVi
ii ffgf
gGfF
},{
222 )(
21)(
21exp
},,|Pr{
ασ
σα
Statistical Performance Estimation for Binary Markov Random Fields
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 44
( ) },|Pr{)(),,(||
1
},|Pr{),,(1),|,(
2
2
σλρσαλ
σσασσα
iiiiij
ijijVi
iiij
iji fFgGghfddgV
gdfFgGfghV
fD
==
−
≅
==−=
∏∑∫ ∏∫
∫
∂∈→→
∈
∞+
∞−∂∈
∞+
∞− →
},|Pr{)(
tanh)tanh(tanh
)(
*
}/{
}/{2
1
}{\
σλρ
λσ
αλδ
λλρ
jjjjijk
jkjk
ijljl
iij
Vi ijkjkiijij
fFgG
g
ddg
==
×
+−×
=
∏
∑
∑∫ ∏ ∫
∂∈→→
∂∈→
−→
∈
∞+
∞−∂∈
∞+
∞− →→→
+≅ ∑
∂∈→
ijij
ii
ggh λ
σσα 2sgn),,(
gdfFgGfghV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 45
0
0.1
0.2
0.3
0.4
0 0.2 0.4 0.6 0.8 1
Statistical Performance Estimation for Markov Random Fields
0
200
400
600
0 0.001 0.002 0.003α
σ=40σ=1
α
),|,( σσα fD
Spin Glass Theory in Statistical MechanicsLoopy Belief Propagation
0.8
0.6
0.40.2
),|,( σσα fD
Multi-dimensional Gauss Integral Formulas
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 46
Statistical Performance Estimation for Markov Random Fields
gdfFgGghfV
fD },|Pr{),,(1),|,(
2σσασσα ==−= ∫
Outline
1. Introduction2. Bayesian Image Analysis by Gauss Markov
Random Fields3. Statistical Performance Analysis for Gauss
Markov Random Fields4. Statistical Performance Analysis in Binary
Markov Random Fields by Loopy Belief Propagation
5. Concluding Remarks
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 47
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 48
Summary
Formulation of probabilistic model for image processing by means of conventional statistical schemes has been summarized.Statistical performance analysis of probabilistic image processing by using Gauss Markov Random Fields has been shown.One of extensions of statistical performance estimation to probabilistic image processing with discrete states has been demonstrated.
Image Impainting by Gauss MRF and LBP
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 49
Gauss MRFand LBP
Our framework can be extended to erase a scribbling.
14 October, 2010 LRI Seminar 2010 (Univ. Paris-Sud) 50
References1. K. Tanaka and D. M. Titterington: Statistical Trajectory of Approximate EM
Algorithm for Probabilistic Image Processing, Journal of Physics A: Mathematical and Theoretical, vol.40, no.37, pp.11285-11300, 2007.
2. M. Yasuda and K. Tanaka: The Mathematical Structure of the Approximate Linear Response Relation, Journal of Physics A: Mathematical and Theoretical, vol.40, no.33, pp.9993-10007, 2007.
3. K. Tanaka and K. Tsuda: A Quantum-Statistical-Mechanical Extension of Gaussian Mixture Model, Journal of Physics: Conference Series, vol.95, article no.012023, pp.1-9, January 2008
4. K. Tanaka: Mathematical Structures of Loopy Belief Propagation and Cluster Variation Method, Journal of Physics: Conference Series, vol.143, article no.012023, pp.1-18, 2009
5. M. Yasuda and K. Tanaka: Approximate Learning Algorithm in Boltzmann Machines, Neural Computation, vol.21, no.11, pp.3130-3178, 2009.
6. S. Kataoka, M. Yasuda and K. Tanaka: Statistical Performance Analysis in Probabilistic Image Processing, Journal of the Physical Society of Japan, vol.79, no.2, article no.025001, 2010.