probabilistic image processing and bayesian network

RC2005 (19 July, 2005, Sendai)

1

Probabilistic image processing and Bayesian networkKazuyuki Tanaka

Graduate School of Information Sciences,Tohoku University

[email protected]://www.smapip.is.tohoku.ac.jp/~kazu/

ReferencesReferencesK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, vol.35, pp.R81-R150 (2002).K. Tanaka, H. Shouno, M. Okada and D. M. Titterington: Accuracy of the Bethe approximation for hyperparameter estimation in probabilistic image processing, J. Phys. A, vol.37, pp.8675-8695 (2004).


2

Bayesian Network and Belief Propagation

Probabilistic Information Processing

Probabilistic Model

Bayes Formula

Belief Propagation

J. Pearl: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference (Morgan Kaufmann, 1988).C. Berrou and A. Glavieux: Near optimum error correcting coding and decoding: Turbo-codes, IEEE Trans. Comm., 44 (1996).

Bayesian Network

RC2005 (19 July, 2005, Sendai)RC2005 (19 July, 2005, Sendai) 33

Formulation of Belief PropagationFormulation of Belief PropagationLink between Link between belief propagation belief propagation andand statistical statistical mechanics.mechanics.Y. Kabashima and D. Saad, Belief propagation vs. TAP for decoding corrupted messages, Europhys. Lett. 44 (1998). M. Opper and D. Saad (eds), Advanced Mean Field Methods ---Theory and 　 Practice (MIT Press, 2001).

Generalized belief propagationGeneralized belief propagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, 51 (2005).

Information geometrical interpretation Information geometrical interpretation of belief propagationof belief propagation

S. Ikeda, T. Tanaka and S. Amari: Stochastic reasoning, free energy, and information geometry, Neural Computation, 16 (2004).


Application of Belief PropagationApplication of Belief PropagationImage ProcessingImage ProcessingK. Tanaka: Statistical-mechanical approach to image processing (Topical Review), J. Phys. A, 35 (2002).A. S. Willsky: Multiresolution Markov Models for Signal and Image Processing, Proceedings of IEEE, 90 (2002).

Low Density Parity Check CodesLow Density Parity Check CodesY. Kabashima and D. Saad: Statistical mechanics of 　 low-density parity-check codes (Topical Review), J. Phys. A, 37 (2004). S. Ikeda, T. Tanaka and S. Amari: Information geometry of turbo and low-density parity-check codes, IEEE Transactions on Information Theory, 50 (2004).

CDMA Multiuser Detection AlgorithmCDMA Multiuser Detection AlgorithmY. Kabashima: A CDMA multiuser detection algorithm on the basis of belief propagation, J. Phys. A, 36 (2003).T. Tanaka and M. Okada: Approximate Belief propagation, density evolution, and statistical neurodynamics for CDMA multiuser detection, IEEE Transactions on Information Theory, 51 (2005).

SSatisfability atisfability PProblemroblemO. C. Martin, R. Monasson, R. Zecchina: Statistical mechanics methods and phase transitions in optimization problems, Theoretical Computer Science, 265 　 (2001).M. Mezard, G. Parisi, R. Zecchina: Analytic and algorithmic solution of random satisfability problems, Science, 297 (2002).

RC2005 (19 July, 2005, Sendai) 5

ContentsContents1. Introduction2. Belief Propagation3. Bayesian Image Analysis and Gaussian

Graphical Model4. Image Segmentation5. Concluding Remarks


6

How should we treat the calculation of the summation over 2N configurations.

1,0 1,0 1,0

211 2

,,,x x x

NN

xxxW

It is very hard to calculate exactly except some special cases.It is very hard to calculate exactly except some special cases.

Formulation for approximate algorithmAccuracy of the approximate algorithm

Belief Propagation


7

Tractable Model

Factorizable

Not Factorizable

Probabilistic models with no loop are tractable.

Probabilistic models with loop are not tractable.

dcbadcba

dcba

),(),(),(),(

),(),(),(,

xDxCxBxA

xDxCxBxAa b c d

a

b

cd

a

b

cd

a b c d

dcba xW ,,,,


8

Probabilistic Model on a Graph with Loops

Bij

jiijL xxWZ

xxxP ,1,,, 21

1\2111 ,,,

xLxxxPxP

x

Marginal Probability

x Bij

jiij xxWZ ,

21,\212112 ,,,,

xxLxxxPxxP

x


9

Message Passing Rule of Belief Propagation

Fixed Point Equations for Massage MM

15141312

15141312

21 ,

,

MMMW

MMMWM

1

33

44 2

5

13M

14M

15M

21M


10

Approximate Representation of Marginal Probability

1

1

115114113112

115114113112

\11

x

x

xMxMxMxMxMxMxMxM

PxPx

x

144 2

5

13M

14M

15M

12M

33

Fixed Point Equations for Messages

MM


11

Fixed Point Equation and Iterative Method

Fixed Point Equation ** MM

Iterative Method

23

12

01

MM

MM

MM

0M1M

1M

0

xy

)(xy

y

x*M

RC2005 (19 July, 2005, Sendai) 12




13

Bayesian Image Analysis

Original Image Degraded Image

Transmission

Noise

Likelihood Marginal

yProbabilit PrioriA Processn Degradatio

yProbabilit PosterioriA Image Degraded

Image OriginalImage OriginalImage DegradedImage DegradedImage Original

PrPrPr

Pr


14

Bayesian Image AnalysisDegradation Process ,, ii gf

iii gfP 2

221exp

21,

fg

2,0~ Nn

nfg

i

iii

Original Image Degraded Image

Transmission

Additive White Gaussian Noise


15


A Priori Probability ,, ji gf

Bijji ff

ZP 2

PR 21exp1

f

Standard Images

Generate

Similar?


16

Bayesian Image Analysis ,, ji gf

Bijjiij ffW

Z

PPP

P

,,,

1

,,

,,

POS

g

gffg

gf

22

22

2 21

81

81exp, jijjiijiij ffgfgfffW

A Posteriori Probability

Gaussian Graphical Model


17


y

x

ifif igig

fg

fP ,fgP gOriginal Image Degraded Image

,,

,,g

ffggf

PPP

P

iiiii dffPfdPff

,,,,ˆ gfgf

A Priori Probability

A Posteriori Probability

Degraded Image

Pixels


18

Hyperparameter Determination by Maximization of Marginal Likelihood

fffgg dPPP ,,

,max argˆ,ˆ,

gP

MarginalizationMarginalization

g ,gP ifif

Original Image

Marginal Likelihood

igigDegraded Image

y

x

fgf dPff ii ˆ,ˆ,ˆ

f g fP ,fgP g ffggf PPP ,,,


19

Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

fffgg dPPP ,,Marginal Likelihood

igigIncomplete Data

y

x

fgfgfg dPPQ ,,ln',',,',',

0,',',','

gQ

0,

gP

Equivalent

Q-Function


20

Maximization of Marginal Likelihood by EM (Expectation Maximization) Algorithm

fffgg dPPP ,,Marginal Likelihood

y

x

fgfgfg dPPQ ,,ln',',,',',Q-Function

.,,maxarg1,1 :Step-M

.,,ln,,,, :Step-E

,ttQtt

dPttPttQ

fgfgfIterate the following EM-steps until convergence:EM Algorithm

A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” J. Roy. Stat. Soc. B, 39 (1977).

21RC2005 (19 July, 2005, Sendai)

One-Dimensional One-Dimensional SignalSignal

EM Algorithm

i

i

i

0 127 255

0 127 255

0 127 255

100

0

200

100

0

200

100

0

200

if

ig

if

Original Signal

Degraded Signal

Estimated Signal

40

22RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model

Original ImageOriginal Image Degraded ImageDegraded Image

MSE: 1529MSE: 1529

MSE: 1512MSE: 1512

EM Algorithm with Belief Propagation


23

Exact Results of Gaussian Graphical Model

fCffgf

Cffgfgf

dP

T22

T22

21

21exp

21

21exp

,,

g

CICg

CICg 2

T2 2

1expdet2det,

P

gCIf12ˆ

Multi-dimensional Gauss integral formula

y

x

,max argˆ,ˆ,

gP

24RC2005 (19 July, 2005, Sendai)

Comparison of Belief Propagation with Comparison of Belief Propagation with Exact Results in Gaussian Graphical ModelExact Results in Gaussian Graphical Model

2ˆ||

1MSE

i

ii ff

MSEMSE

Belief Belief PropagationPropagation 327327 0.0006110.000611 36.30236.302 -5.19201-5.19201

ExactExact 315315 0.0007590.000759 37.91937.919 -5.21444-5.21444

ˆ,ˆln gP

MSEMSE


ExactExact 236236 0.0006520.000652 34.97534.975 -5.17528-5.17528

ˆ,ˆln gP

40

40

,max argˆ,ˆ,

gP

25RC2005 (19 July, 2005, Sendai)

Image Restoration by Image Restoration by Gaussian Graphical ModelGaussian Graphical Model

Original ImageOriginal Image

MSE:315MSE:315MSE: 325MSE: 325

MSE: 545MSE: 545 MSE: 447MSE: 447MSE: 411MSE: 411

MSE: 1512MSE: 1512

Degraded ImageDegraded Image Belief PropagationBelief Propagation

Lowpass FilterLowpass Filter Median FilterMedian Filter

Exact

Wiener Filter

2ˆ||

1MSE

i

ii ff

26RC2005 (19 July, 2005, Sendai)

Original ImageOriginal Image

MSE236MSE236MSE: 260MSE: 260

MSE: 372MSE: 372 MSE: 244MSE: 244MSE: 224MSE: 224

MSE: 1529MSE: 1529

Degraded ImageDegraded Image Belief PropagationBelief Propagation

Lowpass FilterLowpass Filter Median FilterMedian Filter

Exact

Wiener Filter

2ˆ||

1MSE

i

ii ff

Image Restoration by Gaussian Image Restoration by Gaussian Graphical ModelGraphical Model

27RC2005 (19 July, 2005, Sendai)

Extension of Belief PropagationExtension of Belief Propagation

Generalized Belief PropagationGeneralized Belief PropagationJ. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy J. S. Yedidia, W. T. Freeman and Y. Weiss: Constructing free-energy approximations and generalized belief propagation algorithms, IEEE approximations and generalized belief propagation algorithms, IEEE Transactions on Information Theory, Transactions on Information Theory, 5151 (2005). (2005).

Generalized belief propagation is equivalent Generalized belief propagation is equivalent to the cluster variation method in statistical to the cluster variation method in statistical mechanicsmechanicsR. Kikuchi: A theory of cooperative phenomena, Phys. Rev., R. Kikuchi: A theory of cooperative phenomena, Phys. Rev., 8181 (1951). (1951).T. Morita: Cluster variation method of cooperative phenomena and its T. Morita: Cluster variation method of cooperative phenomena and its generalization I, J. Phys. Soc. Jpn, generalization I, J. Phys. Soc. Jpn, 1212 (1957). (1957).

28RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical ModelImage Restoration by Gaussian Graphical Model

2ˆ||

1MSE

i

ii ff

MSEMSE


Generalized Generalized Belief Belief

PropagationPropagation315315 0.0007580.000758 37.90937.909 -5.21172-5.21172

ExactExact 315315 0.0007590.000759 37.91937.919 -5.21444-5.21444

ˆ,ˆln gP

MSEMSE



PropagationPropagation236236 0.0006520.000652 34.97134.971 -5.17256-5.17256

ExactExact 236236 0.0006520.000652 34.97534.975 -5.17528-5.17528

ˆ,ˆln gP

40

40

,max argˆ,ˆ,

gP

29RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters

2

,,,

ˆ

yx

yxyx ff||

1MSE

MSEMSE MSEMSE

Belief Belief PropagationPropagation 327327 Lowpass Lowpass

FilterFilter(3x3)(3x3) 388388

(5x5)(5x5) 413413


PropagationPropagation315315 Median Median


(5x5)(5x5) 445445

ExactExact 315315 Wiener Wiener FilterFilter

(3x3)(3x3) 864864

(5x5)(5x5) 548548

40

GBPGBP

(3x3) Lowpass(3x3) Lowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener

30RC2005 (19 July, 2005, Sendai)

Image Restoration by Gaussian Graphical Image Restoration by Gaussian Graphical Model and Conventional FiltersModel and Conventional Filters

2

,,,

ˆ

yx

yxyx ff||

1MSE

MSEMSE MSEMSE

Belief Belief PropagationPropagation 260260 Lowpass Lowpass


(5x5)(5x5) 224224


PropagationPropagation236236 Median Median


(5x5)(5x5) 244244

ExactExact 236236 Wiener Wiener FilterFilter

(3x3)(3x3) 703703

(5x5)(5x5) 372372

40

GBPGBP

(5x5) Lowpass(5x5) Lowpass (5x5) Median(5x5) Median (5x5) Wiener(5x5) Wiener

RC2005 (19 July, 2005, Sendai) 31



RC2005 (19 July, 2005, Sendai) 32

Image Segmentation by Image Segmentation by Gauss Mixture ModelGauss Mixture Model

i

iaP γa

iii

ii

afaa

P 222

1exp2

1,,

σμaf

204321

,1924 ,1923,1272 ,641

i

K

k

i

kkf

kk

PP

P

12

2

2exp

2

,,

,,

aγaσμaf

γσμfGauss Mixture Model

γσμf

γaσμaf

γσμfa

,,,,

,,,

PPP

P

RC2005 (19 July, 2005, Sendai) 33

Image Segmentation by Combining Image Segmentation by Combining Gauss Mixture Model with Potts Model Gauss Mixture Model with Potts Model

Bijaa

ii jia

ZP ,

PRexp1

γγa

iii

ii

afaa

P 222

1exp2

1,,

σμaf

204321

,1924 ,1923,1272 ,641

γσμf

γaσμaf

γσμfa

,,,,

,,,

PPP

P

Belief PropagationBelief Propagation

iaia

ifif

iaiˆaPotts Model

34RC2005 (19 July, 2005, Sendai)

Image SegmentationImage Segmentation

Original Image Histogram Gauss Mixture Model

Gauss Mixture Model and Potts Model

Belief Belief PropagationPropagation

0101.05 ,4.145 ,8.2245

3982.04 ,7.114 ,4.16843375.03 ,6.233 ,6.13030711.02 ,0.182 ,2.422

1831.01 ,7.21 ,7.121

35RC2005 (19 July, 2005, Sendai)

Motion DetectionMotion Detection

SegmentationAND

Detection

ba

cb

Gauss Mixture Model and Potts Model with Belief PropagationGauss Mixture Model and Potts Model with Belief Propagation

Segmentation

a

b

c

RC2005 (19 July, 2005, Sendai) 36



37RC2005 (19 July, 2005, Sendai)

SummarySummaryFormulation of belief propagationFormulation of belief propagationAccuracy of belief propagation in Bayesian Accuracy of belief propagation in Bayesian image analysis by means of Gaussian image analysis by means of Gaussian graphical model (Comparison between the graphical model (Comparison between the belief propagation and exact calculation)belief propagation and exact calculation)Some applications of Bayesian image Some applications of Bayesian image analysis and belief propagationanalysis and belief propagation

38RC2005 (19 July, 2005, Sendai)

Related ProblemRelated Problem

fgffggff ddPP ,,,ˆ 2

Statistical Performance Spin Glass Theory

H. Nishimori: Statistical Physics of Spin Glasses and Information Processing: An Introduction, Oxford University Press, Oxford, 2001.

f g fP ,fgP g ffggf PPP ,,,

fgfg dPff ii ,,,,ˆ


確率的情報処理の動向確率的情報処理の動向田中和之・樺島祥介編著 , “ ミニ特集 / ベイズ統計・統計力学と情報処理” , 計測と制御 2003 年 8 月号．田中和之，田中利幸，渡辺治他著，“連載 / 確率的情報処理と統計力学～様々なアプローチとそのチュートリアル～”，数理科学 2004 年 11 月号から開始．田中和之，岡田真人，堀口剛他著，“小特集 / 確率を手なづける秘伝の計算技法～古くて新しい確率・統計モデルのパラダイム～”，電子情報通信学会誌 2005 年 9 月号

probabilistic image processing and bayesian network

Documents

information theory

approximate belief propagation

basis of belief propagation

probabilistic image

information geometry

ieee transactions

statistical mechanics

statistical neurodynamics