generalized belief propagation for gaussian graphical model in probabilistic image processing

6 September, 20056 September, 2005 SPDSA2005 (Roma)SPDSA2005 (Roma) 11

Generalized Belief Propagation Generalized Belief Propagation for Gaussian Graphical Model for Gaussian Graphical Model

in probabilistic image processingin probabilistic image processing

Generalized Belief Propagation Generalized Belief Propagation for Gaussian Graphical Model for Gaussian Graphical Model

in probabilistic image processingin probabilistic image processing

Kazuyuki TanakaKazuyuki TanakaGraduate School of Information Sciences,Graduate School of Information Sciences,

Tohoku University, JapanTohoku University, Japanhttp://www.smapip.is.tohoku.ac.jp/~kazu/

ReferenceK. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe Approximation for Hyperparameter Estimation in Probabilistic Image Processing, J. Phys. A: Math & Gen., 37, 8675 (2004).


ContentsContents

1.1. IntroductionIntroduction

2.2. Loopy Belief PropagationLoopy Belief Propagation

3.3. Generalized Belief PropagationGeneralized Belief Propagation

4.4. Probabilistic Image ProcessingProbabilistic Image Processing

5.5. Concluding RemarksConcluding Remarks

6 September, 2005 SPDSA2005 (Roma) 3

Bayesian Image Analysis

Likelihood Marginal

yProbabilit PrioriA Processn Degradatio

yProbabilit PosterioriA Image Degraded

Image OriginalImage OriginalImage DegradedImage DegradedImage Original

Pr

PrPr Pr

Original Image Degraded Image

Transmission

Noise

Bayesian Image Analysis + Belief Propagation Bayesian Image Analysis + Belief Propagation →→ Probabilistic Image ProcessingProbabilistic Image Processing

Graphical Model with Loops Graphical Model with Loops =Spin System on Square Lattice=Spin System on Square Lattice


Belief PropagationBelief Propagation

Belief PropagationProbabilistic model with no loop

Probabilistic model with some loops(Lauritzen, Pearl)

Loopy Belief Propagation (LBP) = = Bethe Approximation

Generalized Belief Propagation (GBP) = Cluster Variation Method

= Transfer Matrix = Transfer Matrix

ApproximationApproximation→Loopy Belief Propagation→Loopy Belief Propagation

Generalized Belief Propagation (Yedidia, Freeman, Weiss)

How is the accuracy of LBP and GBP?How is the accuracy of LBP and GBP?


Gaussian Graphical Model

formula. integral Gauss

ldimensiona-multi the usingby

calculated be can

average and

energy Free

fff d

Z

ln

ifif

Nijjiij

iiii ffgf

Z22

2

1

2

1exp

1 f

,if

otherwise 0

/

/1

Nij

ji

iij

Nijjiij

ij

H

gHfff 1 dmatrix :H

6 September, 2005 6SPDSA2005 (Roma)

How can we construct a probabilistic image How can we construct a probabilistic image processing algorithm by using Loopy Belief processing algorithm by using Loopy Belief Propagation and Generalized Belief Propagation and Generalized Belief Propagation? Propagation?

How is the accuracy of Loopy Belief How is the accuracy of Loopy Belief Propagation and Generalized Belief Propagation and Generalized Belief Propagation?Propagation?

In order to clarify both questions, we assume In order to clarify both questions, we assume the Gaussian graphical model as a posterior the Gaussian graphical model as a posterior probabilistic model probabilistic model

Probabilistic Image Processing by Probabilistic Image Processing by Gaussian Graphical Model and Gaussian Graphical Model and Generalized Belief PropagationGeneralized Belief Propagation


ContentsContents







Kullback-Leibler Divergence of Gaussian Graphical Model

ZFdQ

QQD lnln

z

z

zz

zzz dQQ

mmVVV

gmVF

jijijiijNij

iiiii

ln

22

1

2

1

2

2

zz dQzm ii zz dQmzV iii 2

zz dQmzmzV jjiiij

Entropy TermEntropy Term


Loopy Belief Propagation

zz dQzffQ iiii

Nij jii

jiij

iii fQfQ

ffQfQQ

,f

Trial Function

zz dQzfzfffQ jjiijiij ,

Tractable Form



Nij jii

jiij

iii fQfQ

ffQfQQ

,f

mfAmfA

zzzff

1

2

1exp

det2

1 T

dQQ

Trial Function

Marginal Distribution of GGM is also GGM

iiiVA ijij

VA iimm



Nijij

ii

jijijiijNij

iiiii

ijii

Vi

mmVVVgmV

VVmFF

Adet2ln2

112ln

2

11

22

1

2

1

,,

2

22

mfAmf

Af 1

2

1exp

det2

1 TQ

Nij jii

jiij

iii fQfQ

ffQfQQ

,f

Bethe Free Energy in GGMBethe Free Energy in GGM



1

11

Nijjiiij

Nijjijii i

V A

jiij

ijij VVV 2411

2

1

00,,

Nijj

jiijiiii

ijii mmgmm

VVmF

00,, 11

Nijjiiiji

Nijjiji

i

ijii AiV

VVmFA

00,, 1

ijijijij

ijii

V

VVmFA

m is exact

gHm 1

jij

ijiij VV

VVA


Iteration Procedure

Fixed Point Equation *VΨV *

Iteration

)2()3(

)1()2(

)0()1(

VΨV

VΨV

VΨV

)0(M)1(V

)1(V

0

xy

)(xy

y

x*V


Loopy Belief Propagation and TAP Free Energy


0

4112

1

5223

2

ij

ijjiijjiij

jiijij

ij

OVVVV

VVV

Nijijjiij

ii

jijiijNij

iiiii

ijii

OVVV

mmVVgmV

VVmF

42

22

2

12ln

2

11

2

1

2

1

,,

iiiVA ijij

VA

TAP Free TAP Free EnergyEnergy

01

ijijij A

Mean Field Free EnergyMean Field Free Energy


ContentsContents







Generalized Belief PropagationCluster: Set of nodes

C',''

'1

1 2

3 4

1 2

3 4

1

3

2

4

B

BC

414,114,434,334

,323,223,212,112

Example: System consisting of 4 nodes

114342312 14321

BBBC

BBBB

' and for '

' ' and :Set Cluster Basic

' and ' Subcluster

Every subcluster of the Every subcluster of the element of element of BB does not does not belong to belong to BB..


Selection of B in LBP and GBP

B1 2 3

4 5 6

7 8 9

1 2

4 5

1

4

2

5

2 3

5 6

3

6

7 8

4

7

5

8 8 9

6

9

1 2

4 5

2 3

5 6

4 5

7 8

5 6

8 9

B

LBP (Bethe Approx.）

GBP (Square Approx.

in CVM)


Selection of B and C in Loopy Belief Propagation

B

LBP (Bethe Approx.）

CThe set of Basic ClustersThe set of Basic Clusters The Set of Basic Clusters aThe Set of Basic Clusters a

nd Their Subclustersnd Their Subclusters


Selection of B and C in Generalized Belief Propagation

B

GBP (Square Approximation in CVM）

CThe set of Basic ClustersThe set of Basic Clusters The Set of Basic Clusters aThe Set of Basic Clusters a

nd Their Subclustersnd Their Subclusters


Generalized Belief Propagation

C

jijijiijNij

iiiii

ijii

mmVVVgmV

VVmFF

Adet2ln2

11

22

1

2

1

,,

22

C

QQ ff

mfAmf

Azzzff 1

2

1exp

det2

1 TdQQ

Trial Function

Marginal Distribution of GGM is also GGM

iiiVA

ijijVA


00,,

,

1

,,

Ciii

Bijjiji

i

ijii

V

VVmF

A

Generalized Belief Propagation

00,,

,,

Cijjjiijiii

i

ijiimmgm

m

VVmF

00,,

,,

1

Cjiijij

ij

ijii

V

VVmF

A

NijiVV iji ,,V

gHm 1

VΨV

m is exact


ContentsContents









Transmission

Noise

Likelihood Marginal

yProbabilit PrioriA Processn Degradatio

yProbabilit PosterioriA Image Degraded

Image OriginalImage OriginalImage DegradedImage DegradedImage Original

Pr

PrPr Pr


Bayesian Image AnalysisDegradation Process ,, ii gf

iii gfP 2

22

1exp

2

1,

fg

2,0~ Nn

nfg

i

iii


Transmission

Additive White Gaussian Noise



A Priori Probability ,, ji gf

Bijji ff

ZP 2

PR 2

1exp

1

f

Standard Images

Generate

Similar?


Bij

jiij ffWZP

PPP ,

,,

1

,

,,,

POS

gg

ffggf


,, ji gf

22

22

2 2

1

8

1

8

1exp, jijjiijiij ffgfgfffW

A Posteriori Probability

Gaussian Graphical Model

Original Image f Degraded Image g

,,gfP



y

x

ifif igig

fg

fP ,fgP gOriginal Image Degraded Image

,

,,,

g

ffggf

P

PPP

iiiii dffPfdPff

,,,,ˆ gfgf

A Priori Probability

A Posteriori Probability

Degraded Image

Pixels


Hyperparameter Determination by Maximization of Marginal Likelihood

fffgg dPPP ,,

,max argˆ,ˆ,

gP

MarginalizationMarginalization

g ,gP ifif

Original Image

Marginal Likelihood

igig

Degraded Image

y

x

fgf dPff ii ˆ,ˆ,ˆ

f g fP ,fgP g ffggf PPP ,,,


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

fffgg dPPP ,,Marginal Likelihood

igigIncomplete Data

y

x

fgfgfg dPPQ ,,ln',',,',',

0,',',','

gQ

0,

gP

Equivalent

Q-Function


Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

fffgg dPPP ,,Marginal Likelihood

y

x

fgfgfg dPPQ ,,ln',',,',',

Q-Function

.,,maxarg1,1 :Step-M

.,,ln,,,, :Step-E

,ttQtt

dPttPttQ

fgfgf

Iterate the following EM-steps until convergence:

EM Algorithm

A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data

via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).


Image RestorationThe original image is generated from the prior probability. (Hyperparameters: Maximization of Marginal Likelihood)

302.8MSE 37.8,

0.000784

ˆ

ˆ

538.5MSE ,29.1

0.000298

ˆ

ˆ

297.6MSE 39.4,

0.001090

ˆ

ˆ

Mean-Field Method Exact Result

)( 001.0 40)(

Loopy Belief PropagationLoopy Belief Propagation

Degraded ImageOriginal Image


Numerical Experiments of Logarithm of Marginal Likelihood

The original image is generated from the prior probability. (Hyperparameters: Maximization of Marginal Likelihood)

37.8 0.000784, ˆˆ29.1 0.000298, ˆˆ 39.4 0.00109, ˆˆ

Mean-Field Method Exact Result

)( 001.0 40)(

Loopy Belief PropagationLoopy Belief Propagation

Degraded ImageOriginal Image

,ˆln||

1gP

ˆ,ln||

1gP

10 0.00103020 40 50 60 0 0.0020-6.0

-5.5

-5.0 -5.0

-5.5

ExactLPBLPB

MFA

MFALPBLPB

Exact


Numerical Experiments of Logarithm of Marginal Likelihood

)( 001.0

40)( Degraded Image

Original Image

37.8 0.000784, ˆˆ

29.1 0.000298, ˆˆ

39.4 0.00109, ˆˆ

MF

Exact

LBPLBP

0.001

00

50 100

ExactLPBLPB

MFA

t

0.002

EM Algorithm with Belief Propagation


Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model

Original ImageOriginal Image Degraded ImageDegraded Image

MSE: 1529MSE: 1529

MSE: 1512MSE: 1512

EM Algorithm with Belief Propagation


Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model

Original ImageOriginal Image

MSE:315MSE:315MSE:327 MSE:327

MSE:611 MSE:611 MSE: 1512MSE: 1512

Degraded ImageDegraded Image

LBPLBP

Mean Field Method

Exact Solution

2

,,,

ˆ

yx

yxyx ff||

1MSE

GBPGBP

MSE: 315MSE: 315

TAPTAP

MSE:320 MSE:320


Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model

Original ImageOriginal Image

MSE:236MSE:236MSE:260 MSE:260

MSE: 565MSE: 565MSE: 1529MSE: 1529

Degraded ImageDegraded Image

LBPLBP

Mean Field Method

Exact Solution

2

,,,

ˆ

yx

yxyx ff||

1MSE

GBPGBP

MSE:236 MSE:236

TAPTAP

MSE:248 MSE:248


Image Restoration by Gaussian Graphical Model

2

,,,

ˆ

yx

yxyx ff||

1MSE

MSE

MF 611 0.000263 26.918 -5.13083

LBP 327 0.000611 36.302 -5.19201

TAP 320 0.000674 37.170 -5.20265

GBP 315 0.000758 37.909 -5.21172

Exact 315 0.000759 37.919 -5.21444

ˆ,ˆln gP

MSE

MF 565 0.000293 26.353 -5.09121

LBP 260 0.000574 33.998 -5.15241

TAP 248 0.000610 34.475 -5.16297

GBP 236 0.000652 34.971 -5.17256

Exact 236 0.000652 34.975 -5.17528

ˆ,ˆln gP

40

40

,ˆ,ˆ,

gPmax arg


Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters

2

,,,

ˆ

yx

yxyx ff||

1MSE

MSEMSE MSEMSE

MFMF 611611Lowpass Lowpass

FilterFilter

(3x3(3x3))

388388

LBPLBP 327327 (5x5(5x5))

413413

TAPTAP 320320Median Median FilterFilter

(3x3(3x3))

486486

GBPGBP 315315 (5x5(5x5))

445445

ExactExact 315315Wiener Wiener FilterFilter

(3x3(3x3))

864864

(5x5(5x5))

548548

40

GBPGBP

(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener


Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters

2

,,,

ˆ

yx

yxyx ff||

1MSE

MSEMSE MSEMSE

MFMF 565565Lowpass Lowpass

FilterFilter

(3x3(3x3))

241241

LBPLBP 260260 (5x5(5x5))

224224

TAPTAP 248248Median Median FilterFilter

(3x3(3x3))

331331

GBPGBP 236236 (5x5(5x5))

244244

ExactExact 236236Wiener Wiener FilterFilter

(3x3(3x3))

703703

(5x5(5x5))

372372

40

GBPGBP

(5x5) Lowpass(5x5) Lowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener


ContentsContents







SummarySummarySummarySummaryGeneralized Belief Propagation for Gaussian Graphical ModelAccuracy of Generalized Belief PropagationDerivation of TAP Free Energy for Gaussian Graphical Model by Perturbation Expansion of Bethe Approximation


Future ProblemFuture ProblemFuture ProblemFuture ProblemHyperparameter Estimation by TAP Free Energy is better than by Loopy Belief Propagation.

Effectiveness of Higher Order Terms of TAP Free Energy for Hyperparameter Estimation by means of Marginal Likelihood in Bayesian Image Analysis.

Nijijjiij

ii

jijiijNij

iiiii

ijii

OVVV

mmVVgmV

VVmF

42

22

2

12ln

2

11

2

1

2

1

,,

TAP Free TAP Free EnergyEnergy

Mean Field Free EnergyMean Field Free Energy

generalized belief propagation for gaussian graphical model in probabilistic image processing

Documents