bethe free energy, kikuchi approximations, and …sczhu/workshops/sctv01/tr2001-16.pdfbethe free...

24
MITSUBISHI ELECTRIC RESEARCH LABORATORIES http://www.merl.com Bethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia, William T. Freeman, and Yair Weiss TR-2001-16 May 2001 Abstract This is an updated and expanded version of TR2000-26, but it is still in draft form. Belief propagation (BP) was only supposed to work for tree-like networks but works surprisingly well in many applications involving networks with loops, including turbo codes. However, there has been little understanding of the algorithm or the nature of the solutions it finds for general graphs. We show that BP can only converge to a stationary point of an approximate free energy, known as the Bethe free energy in statistical physics. This result characterizes BP fixed-points and makes connections with variational approaches to approximate inference. More importantly, our analysis lets us build on the progress made in statistical physics since Bethe’s approximation was introduced in 1935. Kikuchi and others have shown how to construct more accurate free energy approximations, of which Bethe’s approximation is the simplest. Exploiting the insights from our analysis, we derive generalized belief propagation (GBP) versions of these Kikuchi approximations. These new message passing algorithms can be significantly more accurate than ordinary BP, at an adjustable increase in complexity. We illustrate such a new GBP algorithm on a grid Markov network and show that it gives much more accurate marginal probabilities than those found using ordinary BP. This work may not be copied or reproduced in whole or in part for any commercial purpose. Permission to copy in whole or in part without payment of fee is granted for nonprofit educational and research purposes provided that all such whole or partial copies include the following: a notice that such copying is by permission of Mitsubishi Electric Research Laboratories, Inc.; an acknowledgment of the authors and individual contributions to the work; and all applicable portions of the copyright notice. Copying, reproduction, or republishing for any other purpose shall require a license with payment of fee to Mitsubishi Electric Research Laboratories, Inc. All rights reserved.

Upload: others

Post on 26-Dec-2019

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

MITSUBISHI ELECTRICRESEARCHLABORATORIEShttp://www.merl.com

Bethe free energy, Kikuchi approximations,and belief propagation algorithms

JonathanS.Yedidia,William T. Freeman,andYair Weiss

TR-2001-16 May 2001

Abstract

This is anupdatedandexpandedversionof TR2000-26,but it is still in draft form.Beliefpropagation(BP)wasonlysupposedtowork for tree-likenetworksbutworks

surprisinglywell in many applicationsinvolving networkswith loops,includingturbocodes.However, therehasbeenlittle understandingof thealgorithmor thenatureofthesolutionsit findsfor generalgraphs.

We show that BP canonly converge to a stationarypoint of an approximatefreeenergy, known astheBethefreeenergy in statisticalphysics.This resultcharacterizesBP fixed-pointsand makes connectionswith variationalapproachesto approximateinference.

More importantly, our analysislets us build on the progressmadein statisticalphysicssinceBethe’s approximationwasintroducedin 1935.Kikuchi andothershaveshown how to constructmoreaccuratefree energy approximations,of which Bethe’sapproximationis the simplest. Exploiting the insightsfrom our analysis,we derivegeneralizedbeliefpropagation(GBP)versionsof theseKikuchi approximations.Thesenew messagepassingalgorithmscanbesignificantlymoreaccuratethanordinaryBP,at anadjustableincreasein complexity. We illustratesucha new GBPalgorithmon agrid Markov network andshow thatit givesmuchmoreaccuratemarginalprobabilitiesthanthosefoundusingordinaryBP.

This work may not be copiedor reproducedin whole or in part for any commercialpurpose.Permissionto copy inwholeor in partwithoutpaymentof feeis grantedfor nonprofiteducationalandresearchpurposesprovidedthatall suchwholeor partialcopiesincludethefollowing: anoticethatsuchcopying is by permissionof MitsubishiElectricResearchLaboratories,Inc.;anacknowledgmentof theauthorsandindividualcontributionsto thework; andall applicableportionsof thecopyright notice.Copying, reproduction,or republishingfor any otherpurposeshallrequirealicensewith paymentof feeto MitsubishiElectricResearchLaboratories,Inc. All rightsreserved.

Page 2: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

Copyright c�

MitsubishiElectricResearchLaboratories,Inc., 2001201Broadway, Cambridge,Massachusetts02139

Page 3: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

This is anupdatedandexpandedversionof TR2000-26,but is still in draft form asof its releaseonApril 4, 2001.Smallfurtherupdate(section2.3)onMay 15,2001.

Page 4: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

��������� ����� �� ������������������������! � �"$#%�'& � ���(") �* � �+ ,-��.��'�� �" �� � � ���(")

� .'�/"�0�(����& *

132547698;:<6=4?>A@CB�D�E<FGE<FG6H$IKJML

NPORQTSVUXW(Y(Z9[VY]\^ Y`_ba=UXcdZ9e`f(g Hih O(N=Qkj`lmon W(p=f`qAr=Qts;uvr(N=QwuxsPytz(z{}|;~�� ~9� �P���M|`�G��� �w�R�

��FG���GF�6=����@C�3�tD�D��/6=4H$IKJ�L

NPORQTSVUXW(Y`Z=[VY]\^ Y`_ba=UXc�Z=e`f`g Hih O(N9Q;j`lmKn W`pRf`qAr=Qts;uvr(N9Q}u�s`y`N's� �d|}|`���'�t����|P����� �}�=�

B�6=F�����D�F��;�^ W`_��= 9¡Xf}U�¢=£wc�f}pR£}f¥¤�c�¦'cd§¨c©W(pª ^ S�fkU¨«(f}¬�f}\(g'z'­(y�¢�W9Z=Y�®MY`¬©¬SVf}UX«`fk¬©fk\`g ^ h l`z�sPN`Otu¯Q]s(strmon W`p=f(qAy=QkOPu°r`z'N]u�yPO'NPl{k±o|t� ²x²¯�C�w²k��³P|P��´µ|P� |}{k� |;~=¶

·¹¸Tº`»9¼=½A¾¿»

SVf}¬�c©f}ÀM�=UXW`�RY`e(Yt¡Xc©W(pÂÁ�S mVà [ÄY`§ÅW`p=¬�\/§¨ =�=�¿W(§¨fkZ�¡XW![�W(U¨«ÆÀÇW`U�¡¨UXf}f}u�¬�c�«`fp=fw¡�[VW`UX«9§�a= 9¡<[�W(U¨«9§3§¨ =UX�=U¨cd§¨c©p=e(¬©\�[�fk¬©¬(c©pb_�Y`p�\�YP�R�=¬©cd£}YP¡¨c�W`pR§Èc©p�¦(W`¬�¦'c�p=ep=fw¡�[VW`UX«9§0[Éc©¡ n ¬©W�W(�R§}gKc©pR£}¬© 5Z9c©pReÆ¡X =UXa5W!£wW9Z9f;§}Ê/®MWt[�fk¦`f}U;gC¡ n f}UXf n Y(§a5fkf}p%¬©c©¡x¡X¬©fË =pRZ9fkUX§x¡XY`pRZ9c�p=e�W`À�¡ n f!YP¬�e`W(U¨c©¡ n _ÌW(U�¡ n fÆp5Yt¡¨ RU¨fÆWPÀ0¡ n f§xW(¬© =¡¨c�W`pR§Äc©¡ÉÍRpRZ=§ÉÀÇW`UÉe`fkp=f}U¯YP¬Îe(UXY`� n §kÊÏ f�§ n Wt[-¡ n YP¡�S m £kYPp/W(p=¬©\�£}W`p�¦`fkU¨e(f�¡¨WÆYƧ�¡¯Yt¡Xc©W(pRYPUX\Æ�¿W`c�p'¡ÅW`ÀMYPpYP�=�RU¨W]Ð9c�_�YP¡¨fMÀÇU¨fkf�fkp=f}UXe`\(g'«�p=Wt[Ép�Y`§�¡ n f¥SVf}¡ n fTÀÇUXf}fTf}p=fkU¨e(\�c©p�§�¡¯Yt¡Xc�§xu¡¨cd£}Y`¬3� n \9§xcd£}§kÊ�Ñ n cd§�UXfk§¨ =¬©¡M£ n YPU¯Y`£w¡¨fkU¨c�Ò}f;§VS m Í=Ð9f;Z�u��¿W`c�p'¡X§�Y`pRZ�_�YP«`f;§£wW(p=p=fk£w¡¨c�W`pR§Ä[Éc©¡ n ¦tYPUXcdYt¡¨c�W`p5YP¬�YP�=�RU¨W'Y`£ n f;§o¡XW�Y`�=�=UXW]Ð9c©_�Yt¡Xf�c�p9ÀÇfkU¨fkpR£wf(ÊH W`UXf¥c©_��¿W`U¨¡XY`p(¡X¬©\(g=W` RU�YPpRY`¬©\9§¨c�§Ä¬�fw¡¯§É R§Éa= Rc©¬dZ�W`pÓ¡ n fb�=U¨W(e`UXfk§X§K_�Y(Z9fc©p/§x¡XYP¡¨cd§�¡Xc�£kYP¬A� n \9§¨c�£k§0§xc�pR£}f�S�f}¡ n f(Ô §ÅYP�=�RU¨W]Ð9c�_�YP¡¨c�W`pÆ[VY(§¥c©p'¡¨UXW9Z9 R£}fkZc©p�Q;l`j'y9Ê<Õ¥c�«� R£ n cRY`pRZ�WP¡ n f}U¯§ n Y]¦(f�§ n Wt[Ép n Wt[)¡XW0£wW(pR§�¡XU¨ 5£µ¡K_�W`UXf�Y`£wu£w =U¯Yt¡XfKÀÇUXf}fÄfkp=f}UXe`\0Y`�=�=UXW]Ð�c�_�Yt¡Xc©W(pR§}g]W`ÀR[ n cd£ n SVfw¡ n f`Ô §AYP�R�=U¨W]Ð9c�_�Yt¡¨c�W`pc�§T¡ n f�§¨c©_��=¬�fk§x¡kÊ I Ð9�=¬�W`c©¡¨c�p=eÓ¡ n f�c�pR§¨c©e n ¡¯§TÀÇU¨W(_ÖW` RU¥Y`pRYP¬�\9§xcd§kg3[�f�Z9f}uU¨c�¦`fVe`fkp=f}U¯YP¬�c�Ò}fkZ0a5fk¬©c�fwÀ¿�=U¨W(�RYPe'Yt¡Xc©W(p×ÁGØ¥S m�à ¦(f}U¯§xc�W`pR§ÈWPÀ5¡ n f;§xfÉÕ¥c�«' 5£ n cYP�=�RU¨W]Ð9c�_�YP¡¨c�W`pR§kÊ�Ñ n f;§xfÆp=f}[Ù_�fk§X§XYPe`fi�RY`§X§xc�p=e�Y`¬©e(W`UXc�¡ n _�§×£kYPpÚa¿f§xc�e`pRc�Í5£kYPp'¡¨¬�\�_�W`UXf¥Y`£}£} =U¯Yt¡¨fT¡ n YPp�W`U¯Z9c�pRYPUX\�S m g=Yt¡MYPp�Y`ZtÛ� R§�¡¯YPa=¬�f¥c©p=u£wUXfkY(§xf¥c�pi£wW(_��R¬©f}Ð�c©¡�\`Ê Ï fbc�¬©¬� R§x¡¨U¯Yt¡¨fŧx R£ n Y�pRf}[�Ø¥S m YP¬�e`W(U¨c©¡ n _ÜW`pY0e(U¨cdZ H Y`U¨«(Wt¦bp=f}¡�[�W(U¨«�YPpRZק n Wt[¹¡ n Yt¡Vc�¡Äe`c�¦`f;§K_Å R£ n _�W`UXfMY(£}£w RUXYP¡¨f_�YPUXe`c�pRYP¬Î�=UXW`aRY`a=c�¬©c©¡¨c�fk§V¡ n YPp�¡ n W(§¨f�ÀÇW( =pRZ� R§¨c©pRe�W(UXZ=c©pRY`U¨\�S m Ê

Ý Þ(ß »9¼=àÉáMâM¾¿»=ãxà ßL W9£kYP¬Mä¨a5fk¬©c�fwÀ��=U¨W(�RYPe'Yt¡Xc©W(pRå�Á�S m�à YP¬�e`W`UXc©¡ n _�§M§¨ R£ n Y(§�¡ n W(§¨f0c�p(¡XU¨W9Z9 5£wfkZ$a'\ m f;YPUX¬ÈYPUXfe` 5YPU¯YPp'¡¨fkfkZ�¡XW$£wW(p'¦(f}UXe`fb¡¨W�¡ n f�£wW`UXUXfk£µ¡¥_�YPUXe`c�pRYP¬A�¿W(§x¡¨fkU¨c�W`U¥�=UXW`a5YPa=c�¬©c©¡¨c�fk§¥c�pË¡¨UXf}f}u�¬�c©«(fe`U¯YP� n c�£kYP¬5_�W9Z9fk¬�§kÊAæ=W(U�e(f}p=fkUXY`¬Rp=f}¡�[�W(U¨«9§A[Éc©¡ n ¬©W�W`�5§}g'¡ n f�§¨c�¡X RYt¡Xc©W(p×cd§K_Å R£ n ¬�fk§X§�£}¬©f;YPU;Ê

Page 5: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

ç p¹¡ n fiW`pRf n YPpRZ3gMY!p� =_ba¿f}UÓWPÀ�U¨f;§xf;YPU¯£ n f}U¯§ n Y]¦(f�f}_��=c�U¨cd£}Y`¬©¬�\)Z9fk_�W`pR§x¡¨U¯Yt¡XfkZ¹e`W�W9Z�¿f}U¨ÀÇW`UX_�Y`pR£wf�ÀÇW`UÉS m YP¬�e`W`UXc©¡ n _�§VYP�=�=¬�c�fkZס¨W�e`U¯YP� n §�[Éc©¡ n ¬©W�W`�5§}Ê ç p=f�Z9U¯YP_�Yt¡Xc�£�£}Y(§xfTc�§¡ n fÉp=fkY`U�¢ n Y`p=p=W`p=u�¬�c©_�c©¡o�5fkUxÀÇW(U¨_�Y`pR£wfÄWPÀÄä¨Ñ< =UXa5W�£}W�Z=fk§Xå=gP[ n W'§xf�Z=fk£wW9Z9c�p=e�YP¬�e`W(U¨c©¡ n _�c�§fkè' =c�¦tYP¬�f}p'¡V¡¨W�S m W`p�Y0¬�W�W`��\�e`U¯YP� nÆé j9g�Q;O]ê°Ê ç ¡ n f}Uɧ¨ R£}£}fk§X§�ÀÇ R¬Î£kY`§¨fk§�c©p5£w¬� RZ9f¥£wW`_��= =¡¨f}U¦�c�§¨c�W`p��=UXW`a=¬�f}_�§ÅYPpRZ!_�fkZ=c�£kYP¬�Z=c�Y`e`W`p5§xcd§ é j=gAN�g�Q]N;êV§¨ =e(e`fk§x¡¨c�p=e�¡ n fÓ§¨ R£}£}fk§X§0W`ÀÉS m W(p¬�W'W(��\ie`U¯YP� n §¥c�§0p=WP¡0¬�c©_�c©¡¨f;Z!¡¨W$£}W�Z=c©p=e$Y`�=�=¬�c�£kYt¡Xc©W(pR§}Ê ç pË¡ n f�W`¡ n fkU n Y`pRZ�g<ÀÇW`U0W`¡ n fkUe`U¯YP� n §V[Éc�¡ n ¬�W�W`�R§kg=S m _�Y]\×e`c�¦`f¥�5W�W(UÉU¨f;§x =¬©¡X§ÄW(UÄÀ�YPc�¬Î¡¨W�£}W`p�¦`fkU¨e(f é Q;N;ê°Êæ=W(U�Y�e`fkp=f}U¯YP¬�e`U¯YP� n gM¬©c©¡x¡X¬©f n Y(§×a¿f}fkp =pRZ=f}U¯§�¡XW'W9ZÚY`a5W( 9¡Ó[ n Yt¡�YP�=�=UXW]Ð9c©_�YP¡¨c�W`pÚS mUXf}�=UXfk§¨f}p'¡X§kgoYPp5Z n Wt[Üc�¡�_�c©e n ¡�a¿f�c©_��=UXWt¦`f;Z�Ê�Ñ n cd§��RYP�¿f}U;Ô §�e(W(YP¬Vcd§�¡¨WË�=U¨Wt¦�cdZ9fÓ¡ n YP¡ =pRZ=f}U¯§�¡¯YPpRZ9c�p=e!YPpRZ�c©p'¡XU¨W9Z9 R£}f�Y˧¨fw¡�WPÀMp=fk[ÜYP¬�e`W(U¨c©¡ n _�§ÅUXfk§¨ =¬�¡Xc©pReiÀÇUXW`_ë¡ n Yt¡� =pRZ=f}U¨u§x¡XYPp5Z9c©pReRÊ Ï fק n Wt[�¡ n Yt¡�S m cd§�¡ n f�ÍRU¯§�¡bc�p�Y��=UXW`e`UXfk§X§¨c©W(piWPÀĬ�W�£kYP¬o_�f;§¨§XYPe(fwu°�RY`§X§xc�p=eYP¬�e`W(U¨c©¡ n _�§kg3f;Y`£ n e`c�¦'c�p=e�fkè' =c�¦tYP¬�f}p'¡0U¨f;§x R¬�¡¯§T¡¨W$Y�£wW(U¨UXfk§¨�5W(pRZ9c�p=e�YP�=�=UXW]Ð9c©_�YP¡¨c�W`p�ÀÇUXW`_§x¡XYt¡Xc�§x¡¨cd£}Y`¬5� n \9§¨c�£k§V«'pRWt[Ép×Y`§K¡ n f�äXÕ¥c©«� R£ n cdå0Y`�=�=UXW]Ð�c�_�Yt¡Xc©W(p�¡¨WÅ¡ n f�ØTc©aRaR§�ÀÇU¨fkf�fkp=f}UXe`\(ÊÑ n f;§xf�YP¬�e`W(U¨c©¡ n _�§ n Y]¦`fÓ¡ n f�YP¡x¡XUXY(£µ¡¨c�¦`fÓ�=UXW`�¿f}U¨¡�\ËW`À�a¿f}c�p=eË R§xfkUxuvY`ZtÛ� R§�¡¯YPa=¬�f`qÓa�\��RY]\'uc�p=eק¨W`_�f0Y`Z=Z=c�¡Xc©W(pRYP¬7£wW`_��= =¡XYt¡Xc©W(pRYP¬<£wW(§x¡kg5W`p=fÅ£}YPp�W(a9¡XY`c©p$£}W`pR§¨cdZ9f}U¯YPa=¬�f¥c©_��=UXWt¦`fk_�f}p'¡c�p�¡ n f$Y`£k£w =U¯Y`£}\ËW`À�W(p=f`Ô §�YP�=�=UXW]Ð9c©_�YP¡¨c�W`p3gKY`pRZì£}YPpì§xW(_�fw¡¨c�_�fk§�W(a9¡XY`c©p¹YÆ£}W`p�¦`fkU¨e(f}p'¡_�fk§X§¨Y`e`f}u��RY(§¨§¨c�p=eÅY`¬©e(W`UXc�¡ n _í[ n f}p�W(UXZ9c�pRY`U¨\�S m Z9W�fk§Äp=W`¡M£wW(p'¦(f}UXe`f(Ê

î ï!ðÈñ ã ð3òÅó ¼=à ó ½7ôK½<»=ãxà ßöõ�÷Éð áÄø ó àKã ß »=º�½C¼ ðÂù�ð ¼=à�ôo¼=½Aá�ã ðÈß » ó àKã ß »=ºËà ò»9ú ð?ï!ð »9ú ð�ò ¼ ð3ð%ð<ß�ð ¼=ôoû

æ=W(U�¡ n f��= =UX�5W'§xf�W`ÀÄY`pRYP¬�\�Ò}c�p=e�YPpRZ!Z9f;§¨£}U¨c�a=c�p=e�S m [Vf�Y`§X§¨ =_�fÅ¡ n Yt¡Å[�f�Y`U¨f�e`c�¦`fkpËY`p =pRZ=c©UXfk£w¡¨fkZ�e(UXY`� n cd£}Y`¬9_�W9Z9fk¬=WPÀ3üýp=W9Z9f;§o[Éc©¡ n �RY`c©UX[Éc�§¨fÄ�5W`¡¨fkp(¡Xc�Y`¬�§MÁ�Y��RY`c©UX[Écd§xf H Y`U¨«(Wt¦J Y`pRZ9W`_öæ7c�f}¬dZ Ã Ê h p�\�e`U¯YP� n c�£kYP¬¿_�W�Z=f}¬3£}Y`p×a¿f0£wW(p'¦(f}U¨¡¨f;Z�c©p'¡XWÅ¡ n cd§VÀÇW`UX_ía5f}ÀÇW`UXf¥Z=W`c�p=ec�p9ÀÇf}UXf}pR£}fÄ¡ n UXW` Re n Y�§x =c©¡XY`a=¬�f�£w¬� R§�¡Xf}UXc©pRe�WPÀÎp=W9Z9fk§Ac�p'¡¨W�¬dYPUXe`fkU7pRW�Z=fk§ é Q]skê°ÊAÑ n UXW` =e n ¡ n c�§¡¨U¯YPp5§�ÀÇW(U¨_�Yt¡Xc©W(p3gÈW(p=f�£}Y`p/YP�=�R¬©\ËYP¬�¬o¡ n f�U¨f;§x R¬�¡¯§�c©p/¡ n cd§0�RY`�5fkU�¡¨WÆYPUXa=c�¡XUXY`U¨\ie`U¯YP� n cd£}Y`¬_�W9Z9f}¬d§Kc�pR£}¬© RZ=c©p=eb¡ n W'§xfM[Éc�¡ n×n c�e n fkUxu°W`U¯Z9fkUA�5W`¡¨f}p'¡Xc�Y`¬�§kÊ Ï f�[Éc©¬�¬5e(c©¦(f�fwÐ9�=¬�cd£wc©¡�fwÐ=Y`_��R¬©f;§WPÀ7¡ n c�§Äc�p�¬�YP¡¨f}U�§¨fk£w¡¨c�W`pR§kÊÑ n f�§�¡¯Yt¡XfÅW`ÀKfkY(£ n  =p=W(aR§xfkU¨¦(fkZ�p=W9Z9f�þÉcd§�Z9fkp=WP¡XfkZia'\�ÿ��xg3Y`pRZ$[VfÅY(§¨§¨ =_�fbf;Y`£ n  Rp=W`a9u§¨f}UX¦`fkZ�pRW�Z=fMcd§�£}W`p=p=f;£µ¡XfkZ�¡XWÅYPp×W`aR§¨f}UX¦`f;Z�p=W9Z9f����xÊAÑ n fKÛ�W(c©p'¡V�=U¨W(aRYPaRc©¬�c�¡�\�Z9cd§�¡XU¨c�a= 9¡Xc©W(pÀÇ =pR£w¡¨c�W`p�cd§Ée`c�¦`fkp×a�\� ÁÇÿ���xÿ����� ��� �¨ÿ���� � Ã�� Q��� ����� ��� ÁÇÿ � �xÿ � à � ��� � Á�ÿ � ��� � à Á�Q Ã

[ n fkU¨f � �¨Á�ÿ�������� à c�§T¡ n f�¬©W9£}Y`¬�äxfk¦'cdZ9fkpR£wf;å�ÀÇW`U�pRW�Z=f�þ¯g � ���(ÁÇÿ�� �¨ÿ!� à cd§T¡ n f�£wW(_��RYt¡Xc©a=c�¬�c�¡�\_�Yt¡XU¨c©Ð�a5f}¡�[�fkf}p�p=W9Z9fk§Äþ�YPpRZ�"5g=YPpRZ � c�§ÉY�p=W(U¨_�YP¬�c�ÒkYt¡Xc©W(p×£wW(pR§x¡XYPp'¡;Ê$#vp�[ n YP¡VÀÇW(¬©¬�Wt[�§kg[Vf¥§¨c�_��R¬©c©ÀÇ\×p=W`¡XYP¡¨c�W`p�YPpRZ�[ÉUXc�¡Xf � �XÁÇÿ�� à Y(§É§ n W(Ux¡ n YPp5Z�ÀÇW(U � �XÁÇÿ���� ��� à ÊÑ n f0§x¡XY`pRZ=YPU¯Z×S m  =�ÎZ=Yt¡Xf�U¨ R¬©f;§ÉYPUXf`q% ���(Á�ÿ!� Ã'& (*),+.- � ���'ÁÇÿ����¨ÿ!� à � �XÁÇÿ�� à �/.0 �21 � 3546� % / �XÁÇÿ�� à ÁGN Ã7 �¨ÁÇÿ�� Ã'& ( � �¨Á�ÿ�� à �/.0 �21 ��3 % / �¨Á�ÿ�� à Á�j Ã

[ n fkU¨f ( Z=f}p=W`¡¨fk§ÄYÅp=W`UX_�YP¬�c©Ò;Yt¡Xc©W(pÓ£wW(pR§x¡XYPp'¡ÉYPpRZÓü/ÁÇþ Ã�8 "�_�fkY`pR§ÉYP¬�¬Îp=W9Z9fk§Vp=fkc©e n a¿W`UXc�p=ep=W9Z9f�þ¯g5f}Ð9£}f}�9¡*"5ÊT®Mf}UXf % ���ÅU¨f}ÀÇf}U¯§É¡XW�¡ n fÅ_�fk§X§XYPe`f0¡ n YP¡Tp=W9Z9f�þɧxfkpRZ=§M¡¨WÓp=W9Z9f9"�Y`pRZ7 �7cd§K¡ n f¥a5fk¬©c�fwÀÄÁ�Y`�=�=UXW]Ð�c�_�Yt¡XfM_�YPUXe`c�pRYP¬¿�¿W(§x¡¨f}UXc�W`UV�=U¨W(aRYPaRc©¬�c�¡�\ à Yt¡Äp=W9Z9f¥þ¯g'W(a9¡XY`c©pRfkZ�a�\

Page 6: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

_b R¬�¡Xc©�=¬�\�c©pReÅYP¬�¬Rc�pR£}W`_�c©pReb_�fk§X§¨Y`e`f;§C¡¨W�¡ n Yt¡Vp=W9Z9fMa�\�¡ n fM¬�W�£kYP¬5f}¦�c�Z=f}pR£}f`Êo¢�c�_�c�¬dYPUX¬©\(g`[Vf£}Y`pËZ=fwÍRp=f�¡ n f�a5fk¬©c�fwÀ 7 ���(ÁÇÿ����¨ÿ!� à Yt¡¥¡ n f��RYPc�U¥WPÀ�pRW�Z=fk§�ÁÇÿ����¨ÿ!� à Y(§M¡ n f��=UXW9Z9 R£µ¡�W`ÀK¡ n f¬�W�£kYP¬��¿WP¡Xf}p'¡¨cdYP¬d§ÉYPpRZ�Y`¬©¬�_�fk§X§XYPe`f;§Vc©pR£}W`_�c�p=e�¡¨W�¡ n f��RYPc�UÄWPÀCp=W9Z9fk§kq7 ��� ÁÇÿ � �¨ÿ � Ã:��(<; ��� Á�ÿ � �xÿ � à �/.0 �21 ��3=4>� % / � Á�ÿ � à �?@0 �21 � 3=4�� % ? � ÁÇÿ � à ÁÇz Ã

[ n fkU¨f ; ��� ÁÇÿ � �xÿ � Ã:A � ��� ÁÇÿ � �xÿ � à � � Á�ÿ � à � � ÁÇÿ � à Êæ=W(U$Y/¡XU¨fkf`gT¡ n f/S m c©¡¨f}U¯Yt¡Xc©W(pR§�£wW(p�¦`f}UXe`fi¡¨W�Yì =p=cdè' =fËÍ=Ð9fkZ%�5W(c©p'¡iYPp5ZÚ¡ n f�a5fk¬©c�fwÀ�§YPUXfËfkè' RYP¬T¡XW)¡ n f��¿W(§x¡¨f}UXc�W`U�_�Y`U¨e(c©p5YP¬d§ 7 �xÁÇÿ�� ÃB�C� ÁÇÿ�� à ÁÇf(Ê e5Ê é Q;j]ê à ÊÙ¢�c©_�c�¬�Y`U¨¬�\`gTW`pRf£}Y`p�§ n Wt[�¡ n YP¡�ÀÇW`UbYÓ¡XU¨fkf�¡ n f��5YPc�U¨[Écd§xf�a¿f}¬�c©f}À�§0YPUXf�fkè' RY`¬A¡¨W�¡ n f��5W'§�¡Xf}UXc©W(U�_�YPUXe`c�pRYP¬d§7 ���(ÁÇÿ����¨ÿ!� Ã:��� ÁÇÿ����¨ÿ!� à ÊoÑ n f�ÀÇW`¬�¬�Wt[Éc©p=eÅ£w¬dYPc�_-£ n Y`UXY(£µ¡Xf}UXc©Òkfk§ 7 �<YPpRZ 7 ����ÀÇW(U�Y`p×YPUXa=c©¡¨U¯YPUX\e`U¯YP� n g3§ n Wt[Éc©p=eÓ¡ n Yt¡¥¡ n fk\iYPUXf�fkè' =c�¦]Y`¬©fkp'¡T¡XWס n f�_�YPUXe`c�pRYP¬7�=UXW`aRY`a=c�¬©c©¡¨c�fk§TW`a=¡XYPc�p=f;Zic�p¡ n f0S�f}¡ n f0Y`�=�=UXW]Ð9c©_�Yt¡Xc©W(p×Z9fk¦`f}¬�W`�¿fkZ�c�p�§x¡XYP¡¨cd§�¡Xc�£kYP¬3� n \9§¨c�£k§ é Q}ê�ÊDFE GH@IKJ�L L fw¡NM % ���POÄa5fÄYM§¨fw¡CWPÀ¿S m _�fk§X§¨Y`e`fk§7YPp5Z�¬©f}¡QM 7 ����� 7 �>OVa¿fK¡ n fVa5fk¬©c�fwÀ�§A£kYP¬d£w =¬dYt¡XfkZÀÇUXW`_ ¡ n W'§xfÅ_�f;§¨§XYPe(fk§kÊÉÑ n f}p�¡ n fba¿f}¬�c�fwÀ�§TYPUXf�Í=Ð9fkZ�u°�¿W`c�p(¡¯§�W`ÀA¡ n f�S m YP¬�e`W`UXc©¡ n _ c�ÀKY`pRZW`pR¬©\�c©À7¡ n fk\ÓYPUXf¥Ò}fkU¨W�e`U¯Y`Z9c�f}p'¡É�¿W`c�p(¡¯§ÄWPÀ7¡ n f0SVf}¡ n fTÀÇUXf}f�f}pRf}UXe`\`g!RTS3qRTSÈÁ6M 7 ����� 7 �6O Ã'� U ) ��� )+ -WV +�X<7 ���(ÁÇÿ�� �xÿ<� à ¬�p ; ���'Á�ÿ�� �xÿ<� ÃZY ) � Á5[\� U Q à )@+ - 7 �xÁÇÿ�� à ¬©p � �XÁÇÿ�� ÃY ) ��� )+ -WV +�X 7 ���(ÁÇÿ�� �xÿ<� à ¬�p 7 ���(ÁÇÿ�� �xÿ<� Ã$U ) � Á=[\� U Q à ) + - 7 �¨Á�ÿ�� à ¬�p 7 �xÁ�ÿ�� Ã

Á�y 稠=a9Û�fk£w¡$¡¨WÚ¡ n f/pRW`UX_�Y`¬©c�ÒkYP¡¨c�W`p�YPp5Z%_�Y`U¨e(c©pRY`¬©c�ÒkYP¡¨c�W`p�£wW(pR§�¡XUXY`c©p'¡¯§}q^] +.- 7 �¨Á�ÿ�� Ã_� Q`g] +�- 7 ���(ÁÇÿ����¨ÿ!� Ã�� 7 �(ÁÇÿ<� à Ê�Á5[\�Acd§V¡ n f�p� =_Åa5fkUÉWPÀCp=f}c�e n a¿W`U¯§VWPÀCp=W9Z9f¥þXÊ Ã`Na b.b>c L Ï fbY(Z=Z L Y`e`U¯YPp=e(f¥_b =¬©¡¨c��=¬�c©fkUX§�¡¨W�ÀÇW`UX_�Y L YPe(UXY`p=e`cdYPped�qgfh���'Á�ÿ!� à cd§É¡ n f0_Å =¬�¡Xc�u�=¬�c©fkU0£wW(U¨UXfk§¨�5W(pRZ9c�p=e�¡XW�¡ n f�£wW(pR§x¡¨U¯YPc�p(¡T¡ n Yt¡ 7 ��� ÁÇÿ � �¨ÿ � à _�YPUXe`c�pRYP¬�c�Ò}fk§TZ9Wt[Épi¡¨W 7 � ÁÇÿ � à gYPp5Zji ��� �>i � Y`U¨fA_Å =¬©¡¨c��=¬©c�f}U¯§È£}W`UXU¨f;§x�¿W`p5Z9c©pReÄ¡¨WÉ¡ n fKp=W(U¨_�Y`¬©c�ÒkYP¡¨c�W`p�£wW(pR§�¡XUXY`c©p'¡¯§}ÊÈÑ n fofkè' RYPu¡¨c�W`p k�lk.m - X 1 + -WV +\X 3 � O�e`c�¦`f;§}q7¬�p 7 ��� Á�ÿ � �xÿ � Ã:� ¬�pÈÁ ; ��� Á�ÿ � �xÿ � ÃxÃ<Y f ��� ÁÇÿ � Ã<Y f � � ÁÇÿ � Ã<Y i ��� U Q(ÊÑ n fÉf;è( 5Yt¡¨c�W`p k�lknm - 1 + - 3 � O¥e(c©¦(fk§kqKÁ=[�� U Q à ÁǬ�p 7 �xÁ�ÿ�� Ã�Y Q Ã:� ¬©p � �xÁ�ÿ�� Ã�Y ] � 0 �21 ��3 f!� �XÁ�ÿ�� Ã�Yi � Ê�¢�fw¡¨¡¨c�p=eof ��� ÁÇÿ � Ãp� ¬�p2q /n0 ��1 � 3=4�� % / � ÁÇÿ � à YPpRZ R§¨c©p=eì¡ n f!_�Y`U¨e(c©p5YP¬�c©Ò;Yt¡¨c�W`p%£wW`p=u§x¡¨U¯YPc�p(¡¯§}g7[Vf�ÍRpRZ!¡ n Yt¡Å¡ n f�§�¡¯Yt¡Xc©W(pRYPUX\i£wW(pRZ9c©¡¨c�W`pR§bW`p!¡ n f L YPe`U¯YPpRe`cdYPp!YPUXf�fkè' =c�¦]Y`¬©fkp'¡¡¨W�¡ n f�S m Í=Ð9fkZ9u��¿W`c�p'¡¥£}W`pRZ9c©¡¨c�W`p5§}Ê�ÑÈW�e`W�c�pi¡ n f�W`�=�¿W(§¨c�¡Xf�Z=c©UXfk£w¡¨c�W`p3g3c©À�[Vf�YPUXfÅe(c©¦(f}p7 ��� � 7 � ��f ��� ÁÇÿ � à ¡ n YP¡<£wW(U¨UXfk§¨�¿W`pRZM¡¨WMY�Ò}f}UXWPu°e`U¯Y`Z=c©fkp(¡Î�¿W`c�p(¡;g][�fo§¨fw¡ % ��� ÁÇÿ � Ã:� m X 1 + X 3rWsut 1wv - X 1 +�X 3,3 ÊSVfk£kYP R§¨f 7 ��� � 7 � �xf ��� §¨YP¡¨cd§�ÀÇ\�¡ n f0§�¡¯Yt¡Xc©W(pRYPUXc�¡�\�£}W`pRZ=c�¡Xc©W(pR§}g % ��� Z9f}ÍRp=fkZ�c�pס n cd§É[VY]\�_Å R§x¡a¿f�Y×ÍRÐ�f;ZË�5W(c©p'¡0WPÀÉS m Ê3Ñ n  R§¥¡ n f}UXf�cd§0Y�W`pRfwu�¡¨WPu°W`pRf�£}W`UXU¨fk�R§xW(pRZ9fkpR£wf�a5f}¡�[�fkf}pÆÍ=Ð9fkZ�u�¿W`c�p(¡¯§ÉWPÀCS m Y`pRZ�§�¡¯YPc�W`pRY`U¨\��¿W`c�p'¡X§ÉW`ÀÈ¡ n f0S�f}¡ n f¥ÀÇUXf}f¥fkp=f}UXe`\(ÊNyÁ I _��=c©UXcd£}YP¬�¬�\`g�[�fMÍRpRZס n YP¡2z|{ G�}\E ~ S m Í=Ð9f;Z�u��¿W`c�p'¡X§V£wW(U¨UXfk§¨�5W(pRZ�¡XWŬ�W9£}YP¬ I�H@�hH@I�G W`À3¡ n fSVfw¡ n f¥ÀÇU¨fkf¥f}p=fkU¨e(\`g9U¯Yt¡ n f}UV¡ n Y`p�_�YPÐ9c©_�Y�W`U�§XY`Z=Z=¬©f}u��¿W`c�p'¡X§kÊ Ãª §¨c�p=e�¡ n fb_�YPUXe`c�pRYP¬�c�ÒkYt¡Xc©W(p�£}W`pRZ=c�¡Xc©W(pR§MY`pRZ�¡ n f�Z9fwÍ5p=c�¡Xc©W(p�WPÀ ; ÁÇÿ�� �¨ÿ!� à ¡ n f�S�f}¡ n f0ÀÇU¨fkff}pRf}UXe`\�£}Y`p�YP¬d§xW�a5f�[ÉUXc©¡x¡¨fkp3qRTSÈÁ6M 7 ����� 7 �>O Ã'� U ) ��� )+.- V + X 7 ���(ÁÇÿ����¨ÿ!� à ¬©p � ���(ÁÇÿ����¨ÿ!� ÃFU ) � ),+.-�7 �XÁÇÿ�� à ¬©p � �XÁÇÿ�� ÃY�) ��� )+ -WV +\X 7 ���(ÁÇÿ����¨ÿ!� à ¬©p 7 ���(ÁÇÿ����¨ÿ!� Ã$U�) � Á=[�� U Q Ãh)=+ - 7 �XÁÇÿ�� à ¬�p 7 �¨ÁÇÿ�� Ã

Á�r Ã

Page 7: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

Ï�n fkU¨fTZ9W�fk§A¡ n f�SVf}¡ n fMÀÇU¨fkf�f}p=fkU¨e(\�£}W`_�fÉÀÇUXW`_�� Ï fM[Éc�¬©¬ÎZ9cd§¨£} R§¨§o¡ n fT� n \9§xcd£}§Kc©p'¡X =c�¡Xc©W(pa¿f n c�pRZÆc�¡�c�p˧¨fk£w¡¨c�W`pÆjÓa= 9¡¥¡XW�_�Y`«`f0¡ n f�ÀÇW`UX_b =¬dYÓ¬©f;§¨§T_b\9§x¡¨fkU¨c�W` R§�[VfÅp=Wt[ § n Wt[�¡ n YP¡c©¡�c�§�Y`pÓf}Ð=Y`£µ¡ÄÀÇUXf}f�fkp=f}UXe`\�[ n fkp�¡ n f�e(UXY`� n cd§ÉY�¡¨UXf}f(ÊF#vp�W`¡ n fkUÉ[�W(UXZR§}g9_�c�p=c©_�c�Ò}c�p=e�¡ n fSVfw¡ n f¥ÀÇU¨fkf¥f}p=fkU¨e(\�ÀÇW`UMYÅ¡¨UXf}f¥cd§Äfkè' =c�¦tYP¬�f}p'¡É¡¨W��¿f}U¨ÀÇW`UX_�c�p=e�fwÐ=Y(£µ¡Éc©p=ÀÇf}UXf}pR£}f`ÊÑ n f�f}Ð9Y(£µ¡ÄÀÇUXf}f¥f}p=fkU¨e(\�cd§É§xc�_��=¬©\ס n f0Y]¦(f}U¯YPe(f�f}p=fkU¨e(\�_�c©p� R§V¡ n f�f}p'¡XU¨W(�'\ÎqR�Á>M 7 Á�ÿ à O Ã:� ) + 7 ÁÇÿ Ã�� ÁÇÿ Ã�Y ) + 7 ÁÇÿ à ¬©W(e 7 Á�ÿ à Á�s Ã

[ n fkU¨f 7 Á�ÿ à c�§¥¡ n f/ä¨a5fk¬©c�fwÀ¨å$c©p/Y�§x¡XYP¡¨f�ÿ�Y`pRZ � ÁÇÿ à ¡ n f/äxfkp=f}UXe`\9å�WPÀÄÿ�_�W`¡¨c�¦]YP¡¨f;ZÆa�\SVW`¬©¡¨Òk_�Y`p=p3Ô §Ä¬�Y][ � ÁÇÿF� � Ã�� ��Q����� 1 + 3 q� Á�ÿ Ã���U ¬©W(e � ÁÇÿF� � ÃFU ¬©W(e �o��U )� ���u� ¬©W(e � ����ÁÇÿ�� �xÿ<� ÃFU ) � ¬�W`e � �XÁÇÿ�� à Á�­ Ã� W`¡¨fÅ¡ n YP¡�Röcd§�Á� =�Æ¡¨W�YÓ£}W`pR§x¡XY`p(¡ à ¡ n f�Õ L Z=c©¦(f}UXe`f}p5£wfba¿fw¡�[Vf}fkp 7 Á�ÿ à YPp5Z � Á�ÿ$� � à §¨W¡ n Yt¡�R�cd§É_�c�p=c©_�c�Ò}f;ZÓ[ n f}p 7 ÁÇÿ Ã���� Á�ÿ$� � à ÊSV\ק¨ =aR§x¡¨c©¡¨ 9¡Xc©p=e�f;è' RYt¡Xc©W(p�­�c�p�¡ n f�Y]¦`fkUXY`e`f�f}p=fkU¨e(\�[�f¥W(a9¡XY`c©p3q) +�7 Á�ÿ Ã�� ÁÇÿ Ã��^U�) ��� )+ -6V +\X<7 ��� ÁÇÿ � �¨ÿ � à ¬�p � ��� ÁÇÿ � �xÿ � Ã$UB) � ) + - 7 � ÁÇÿ � à ¬©p � � Á�ÿ � à Á�l Ã

§¨W�¡ n YP¡É¡ n f�Í5UX§x¡É¡�[�W�¡Xf}UX_�§ÉWPÀ7¡ n fÅS�f}¡ n f¥ÀÇUXf}f�fkp=f}UXe`\ÆÁÇfkè¿Ê¿r à YPUXf�fwÐ=Y(£µ¡¨¬�\�¡ n fbY]¦`fkUXY`e`ff}pRf}UXe`\`Ê#°ÀÉ¡ n fÓe(UXY`� n c�§�Y�¡XU¨fkf`g7¡ n fkp�[Vf×£}Y`p/UXfk§x¡¨UXc�£w¡0¡ n fÓ_�c©p=c�_�c©Ò;Yt¡Xc©W(p�¡¨Wi¡ n f�£w¬dY`§X§0WPÀ 7 ÁÇÿ á n Yt¡M§XYt¡¨cd§xÀÇ\�¡ n fb£wW(pRZ9c©¡¨c�W`pRY`¬�c©pRZ=f}�¿f}pRZ9fkpR£wf;§�c�_��=¬©c�fkZ�a�\�¡ n f0e(UXY`� n ÊK¢� R£ n 7 Á�ÿ à £kYPp�a¿f[ÉUXc�¡¨¡¨f}p�Y(§�YÓ�RU¨W9Z9 R£w¡0WPÀVa5fk¬©c�fwÀ�§�Wt¦(f}U�£w¬�cdè( Rfk§kg<Z9c�¦�c�Z9f;ZËa'\ia5fk¬©c�fwÀ�§�Wt¦(f}U�§¨f}�RY`UXYP¡¨W`U¥§¨fw¡X§Á�§¨f}fbf`Ê eRÊ é Qkjtê à ÊN#vpiW` =UM§¨fw¡¨¡¨c�p=eRgR¡ n f�£w¬�c�è' =f;§�YPUXf��RY`c©U¯§ÉWPÀAp=W9Z9f;§MY`pRZ�¡ n f�§xfk�RYPU¯Yt¡XW`Uɧ¨fw¡X§YPUXf�§xc�p=e(¬©f¥p=W9Z9f;§É§xW�¡ n Yt¡kq 7 ÁÇÿ ÃQ� q � ���u� 7 ���(ÁÇÿ����¨ÿ!� Ãq � 7 �¨ÁÇÿ�� Ã�� - � � ÁxQkO Ã#°À�[�fö§¨ =aR§x¡¨c©¡¨ =¡¨fÜf;è' RYt¡Xc©W(pÌQ;O=g¹c�p(¡XW�¡ n fÜfwÐ9�=UXfk§X§xc�W`p ÀÇW`U�¡ n föpRf}e(YP¡¨c�¦`f f}p'¡XU¨W(�'\] + 7 ÁÇÿ à ¬©p 7 ÁÇÿ à [VfTW`a9¡¯YPc�pס n f¥¬�Y(§�¡V¡�[�WÅ¡¨fkU¨_�§VWPÀ<¡ n f�SVfw¡ n f�ÀÇU¨fkfTfkp=f}UXe`\�Á�fkè¿Ê=r à ÊCÑ n  R§ÀÇW`UÓYË¡¨UXf}f�e`U¯YP� n gK¡ n fiSVfw¡ n f�ÀÇUXf}f$fkp=f}UXe`\/cd§�f}Ð9Y(£µ¡×Y`pRZìc�¡×[Éc�¬�¬Ma¿f$_�c©p=c�_�c©ÒkfkZ¹[ n fkp7 ��� ÁÇÿ � �¨ÿ � Ã:��� ÁÇÿ � �¨ÿ � � � à Y`pRZ 7 � Á�ÿ � Ã:�o� Á�ÿ � � � à Ê� @=� >Z�KD��`FG6=�g�`6=�;D��¥8��<�.��2p�`23E7D��Ï fÉÀÇW`¬�¬©Wt[�¡ n f�ÀÇW`UX_b =¬dYt¡Xc©W(p�W`ÀÎÑ< =UXa5W0Z=fk£wW9Z9c�p=e��=UXfk§¨f}p'¡XfkZ�c©p é Q`Qµê°Ê h p� =pR«'pRWt[Ép�aRc©pRY`U¨\¦`f;£µ¡XW`U���cd§�fkpR£wW9Z9f;Z� R§¨c©pReb¡�[VWb£wW(pR§x¡¨c©¡¨ =fkp(¡É£wW9Z9f;§}g�f;Y`£ n WPÀ<[ n c�£ n c�§VfkY(§x\�¡XWÅZ9f;£wW9Z9fMa�\c©¡X§¨f}¬©ÀKYPpRZ�¡ n fbUXfk§¨ =¬©¡X§�YPUXf�¡¨U¯YPpR§¨_�c�¡¨¡¨f;Z�Wt¦`fkU�Y�p=W(c�§¨\�£ n YPp=p=fk¬Èe(c©¦�c�p=e�¡�[VW�W`aR§¨f}UX¦tYt¡Xc©W(pR§�������`Ê�Ñ n fb¡XY`§¨«�WPÀKÑ< =U¨a¿WÓZ=fk£wW9Z9c�p=eÓc�§�¡¨W×c�p9ÀÇfkUM¡ n fŦtY`¬© =f;§�W`Àoa=c�¡¯§MW`À:�iÀÇUXW`_ ¡ n fb¡�[VWW`a5§xfkU¨¦tYt¡Xc©W(pR§����n� ��PÊÚÑ n f�Z=fk£wW9Z9fkU�«�p=Wt[�§�¡ n f��RU¨c�W`U�Z9cd§�¡XU¨c�a= 9¡Xc©W(p¹Wt¦`f}U���Á�[ n c�£ n [VfY`§X§¨ =_�f n f}UXfTcd§Ä =p=c©ÀÇW`UX_ à YPpRZÓ¡ n f¥¡�[�W�£}W`pRZ9c©¡¨c�W`p5YP¬��=UXW`aRY`a=c�¬©c©¡¨c�fk§$ ��XÁ,� Ã:�  7Á,���x� � à ÊÑ n f¥¡X =U¨a¿W�Z9fk£}W9Z9c©pRe�Y`¬©e(W`UXc�¡ n _öc©¡¨fkUXYP¡¨fk§V¡ n f¥ÀÇW`¬�¬�Wt[Éc©p=e�c©¡¨f}U¯Yt¡Xc©W(pR§kq¡ �¨Á=��� Ã'& ( )¢T£ ¢ -=¤Z¥n-   � Á=¦ à ��n§¤ ��¨ �(Á,¦Q� à ÁxQ`Q Ã

¨ �¨Á=��� Ã'& ( )¢T£ ¢ -=¤Z¥n-   Á=¦ à ��n§¤ � ¡ �(Á,¦Q� à ÁxQ;N Ã

Page 8: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

message 1

bit 1 bit 2 bit 3 bit 4 bit 5 bit 6 bit 7

message 2

æ7c©e( =UXf�Q`qbÑ n f�Ñ< =UXa5W$£}W9Z9f�§�¡XU¨ 5£µ¡¨ RU¨f(Ê J  =pRp=c©pRe$S m W(pË¡ n cd§0e(UXY`� n cd§�f;è( Rc©¦tYP¬�f}p'¡�¡XW¡ n f¥¡¨ =UXa5W�Z=fk£wW9Z9c�p=e�YP¬�e`W(U¨c©¡ n _�Ê

YPp5Z�¡ n f�_�YPUXe`c�pRYP¬��=UXW`a5YPa=c�¬©c©¡�\�Wt¦`fkUg���Ccd§ÉYP�=�RU¨W]Ð9c�_�YP¡¨f;Z�a�\Îq7 Á,��� Ã:�©( ¨ �¨Á,��� à ¡ �XÁ,��� à ÁxQkj ÃYtÀ ¡Xf}UÄ¡ n f�c�¡Xf}U¯Yt¡Xc©W(pR§ n Y]¦`f¥£wW(p�¦`f}UXe`f;ZiÁ�W`UÉUXfkY`£ n fkZ×Y�_�YtÐ9c�_�YP¬�p� =_ba¿f}UMWPÀ<c©¡¨fkUXYP¡¨c�W`pR§ à Ê� W`¡¨f�¡ n YP¡�¡ n f¥ÑÈ =UXa¿W�Z9f;£wW9Z9c�p=eÅY`¬©e(W`UXc�¡ n _íY`§KÀÇW`UX_b =¬dYt¡XfkZ n f}UXf�UXfkè' =c�U¨f;§K§¨ =_�_�c©p=e�Wt¦`fkUYPpÅfwÐ9�¿W`p=fkp(¡Xc�Y`¬©¬�\�¬dYPUXe`fKp� =_Åa5fkUCWPÀ¿§x¡XYP¡¨f;§<c�pbf;Y`£ n c�¡Xf}U¯Yt¡¨c�W`pÈÊ<Ñ n fÄUXfkY`§¨W`p¥¡ n cd§Cc�§7�¿W(§X§xc�a=¬�fcd§0¡ n Yt¡Å¡ n fÓ¬©c�«`fk¬©c n W�W9Z=§*  � Á,¦ à �=  Á=¦ à £wW(_�f�ÀÇU¨W(_ Y`p/fkY(§x\Æ¡¨WËZ9fk£}W�Z=f×£wW9Z9f(Ê�#vp/W`¡ n fkU[VW`U¯Z=§}g¿¡ n f}UXf�c�§�Y×À�Y`£w¡¨W`UXc�ÒkYt¡Xc©W(pi§x¡¨UX R£µ¡X =U¨f�W`pÆ¡ n fk§¨f�¬�c�«`f}¬�c n W�W�ZR§M¡ n Yt¡0£}W`UXU¨f;§x�¿W`pRZ�¡¨W�Y§¨c©p=e(¬©\'uv£wW(p=p=fk£w¡¨f;Z�e`U¯YP� n q�¡ n cd§�f}p5YPa=¬�fk§Å¡¨ RU¨a¿WËZ=fk£wW9Z9fkUX§b¡¨W!£wW(_��= 9¡¨f�¡ n f�fwÐ9�5W(p=f}p'¡Xc�Y`¬§¨ =_ÜWt¦(f}U�§x¡XYt¡Xfk§Äc�p�¬©c�p=f;YPUÄ¡¨c�_�f`Ê I §X§xfkp(¡Xc�Y`¬©¬�\`g�¡ n cd§�cd§�Z9W`pRf�a'\ÓYPp�YP¬�e`W(U¨c©¡ n _Üfkè' =c�¦]Y`¬©fkp'¡¡¨W�S m Êh §V§ n Wt[Ép�c�p é Q;­9g9­9gÎQkO]ê°g'¡ n fTÑÈ RU¨a¿WÅZ9f;£wW9Z9c�p=eÅY`¬©e(W`UXc�¡ n _-c�§�fkè' =c�¦tYP¬�f}p'¡K¡XWba¿f}¬�c©f}À<�=UXW`�RYPue(YP¡¨c�W`p3ÊQ#vp�W` =UÉ RpRZ9c�U¨f;£µ¡¨f;Z�ÀÇW`UX_�YP¬�c�§¨_�g=c�¡�c�§Éf;è' =c©¦tY`¬©fkp(¡M¡¨W�c©¡¨f}U¯Yt¡Xc©pRe�¡ n fbS m f;è( 5Yt¡¨c�W`p5§W`pb¡ n fÄe(UXY`� n § n Wt[Épbc�pÅÍRe` =UXfTQ`Ê<Ñ n f�¡XW`��YPpRZÅa5W`¡x¡XW`_�p=W9Z9f;§C£wW`UXUXfk§¨�5W(pRZ¥¡¨W�£}W�Z=f}[VW`U¯Z=§¦ � ��¦ YPpRZÓ¡ n f0p=W9Z9fk§�c©p�¡ n f0_�c�Z=Z=¬©f0¬dY]\`fkUÉ£wW`UXUXfk§¨�5W(pRZ�¡XW� =p=«�p=Wt[Ép�a=c©¡X§kÊ�Ñ n f��¿WP¡Xf}p9u¡¨cdYP¬d§¥a¿fw¡�[Vf}f}p!p=W9Z9f;§�YPUXf�§¨fw¡�¡XW�Z9f}¬©¡XYÓÀÇ =pR£w¡¨c�W`pR§kq � Á=¦ � ����� Ã9� Q�c�ÀV¡ n f�þ�¡ n aRc�¡0W`ÀQ¦ � g¦ �� cd§Kf;è( 5YP¬¿¡¨W����7YPpRZ×Ò}fkU¨W0W`¡ n fkU¨[Écd§¨f`Ê � W9Z9f;§�¦ � Y`pRZ�¦ n Y]¦(f�W`a5§xfkU¨¦tYt¡Xc©W(p��5W`¡¨fkp(¡Xc�Y`¬�§  � Á,¦ à �=  Á=¦ à £wW`UXUXfk§¨�5W(pRZ9c�p=e�¡¨WÆ¡ n fÓ¬�c©«(f}¬�c n W'W9Z�WPÀ�YÆ£wW9Z9f}[VW`U¯Z!ÀÇUXW`_ëW`pRf�£}W`pR§x¡¨c©¡¨ =fkp'¡£wW9Z9f(ÊÏ�n fkp)S m cd§ÅUX =p/W(p�¡ n c�§Åe`U¯YP� n g<¡ n f�_�f;§¨§XYPe(fk§�UXfkY(£ n c�p=eª����Y`U¨f�f}Ð=Y`£µ¡X¬©\ ¨ �>� ¡ �� R§xf;Zc�pÆ¡X =UXa5WiZ=fk£wW9Z9c�p=e�YPp5ZÆf;è' RYt¡Xc©W(pR§�Q`Qu«¿Q]N�ÀÇW(¬©¬�Wt[ Z9c�UXfk£µ¡X¬©\iÀÇU¨W(_Ö¡ n f�§�¡¯YPpRZRYPU¯ZËS m  =�9uZ=YP¡¨fk§ é Qkr]ê°ÊbÑ n f�S m ÀÇW`UX_�Y`¬©cd§¨_�Y`¬�§¨W�YP¬�¬©Wt[�§� R§�¡¨W�£wW(_��R 9¡¨f�a¿f}¬�c©f}À�§¥Wt¦`f}U2¦ � � ¦ e(c©¦(f}pa�\Îq

7 �]Á,¦ ì� (  ­�tÁ=¦ à � � ¨ � Á=¦ � à ÁxQ}z Ã

Page 9: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

7 PÁ,¦ ì� (  �`Á=¦ à � � ¡ � Á=¦ � à ÁxQ;y ÃÁxQkr Ã

h �R�=¬©\�c�p=e�£w¬dYPc�_�QסXWi¡ n f�e`U¯YP� n § n Wt[Ép)c©p)ÍRe( =U¨fÆQÓe(c©¦(fk§�YËS�f}¡ n fÓÀÇUXf}f�f}p=fkU¨e(\Ë¡ n YP¡c�pR£w¬� RZ9f;§�Û�W`c�p'¡Ka¿f}¬�c©f}À�§KW`À¿¡ n fÉÀÇW(U¨_ 7 Á,¦ � � � � à g(a= =¡�a¿fk£}Y` R§¨fÄ¡ n f�f}p=fkU¨e(\bcd§oc�p9ÍRp=c©¡¨fÉÀÇW(U�Y`p'\§¨ R£ n a5fk¬©c�fwÀAÀÇW(U�[ n cd£ n ¦ ���®� � � §¨ R£ n a¿f}¬�c©f}À�§�_b 5§�¡Ma¿f0W`À7¡ n f�ÀÇW(U¨_ 7 Á,¦ � Ã>¯ Á=¦ �� U � � à ÀÇW(U¡ n f�SVfw¡ n fÅÀÇU¨fkfbf}pRf}UXe`\Ó¡¨W�a5fÅÍRp=c©¡¨f(Ê ª §xc�p=eס n cd§�À�Y`£µ¡;gÎ[�f�£}Y`pi§xc�_��=¬�c�ÀÇ\�¡ n f�SVf}¡ n fbÀÇU¨fkff}pRf}UXe`\`ÊCÑ n fM§¨c�_��R¬©c©ÍRfkZ�ÀÇU¨fkfMf}pRf}UXe`\bcd§opRWt[ìW(p=¬�\�Y¥ÀÇ RpR£µ¡Xc©W(p�WPÀ 7 � Á,¦ à � 7 Á,¦ à Y`pRZ 7 �xÁ=��� à ÊD ba b E@E G a|° J�L L fw¡ ¨ ��� ¡ �Ca¿fbY�§xf}¡�WPÀAÑÈ RU¨a¿W�Z9fk£}W9Z9c©pRe�_�f;§¨§XYPe(fk§ÄYPp5ZÓ¬�fw¡±M 7 ��Á=¦ à � 7 �¯Á,��� à Oa¿f0¡ n fba¿f}¬�c©f}À�§T£kYP¬d£w =¬dYt¡XfkZ�ÀÇUXW`_ ¡ n W'§xfÅ_�fk§X§¨Y`e`fk§kÊ�Ñ n fkp�¡ n f�a5fk¬©c�fwÀ�§TYPUXf�Í=Ð9f;Z�u��¿W`c�p'¡X§MWPÀ¡ n fÓÑ< =U¨a¿WiZ9f;£wW9Z9c�p=eÆY`¬©e(W`UXc�¡ n _ c©À�Y`pRZ!W`p=¬�\Ëc�À�¡ n f}\�Y`U¨f�Òkf}UXW$e`U¯Y`Z9c�f}p'¡0�¿W`c�p'¡X§ÅWPÀÉ¡ n fSVfw¡ n f¥ÀÇU¨fkf¥f}p=fkU¨e(\`g<RTS3qR S � Uª) +²7 �PÁ,¦ à ¬�p: ­�tÁ,¦ ÃTY³) ¢ 7 �PÁ,¦ à ¬©p 7 �tÁ,¦ à ÁxQ]s ÃU ).+²7 (Á,¦ à ¬�p: �`Á,¦ ÃTY ) ¢ 7 (Á,¦ à ¬©p 7 `Á,¦ à ÁxQk­ ÃU ) � )@¥ - 7 �¨Á=��� à ¬�p 7 �¨Á=��� à ÁxQkl Ã

§¨ =a9Û�fk£w¡K¡XW0¡ n f�£wW(pR§x¡¨U¯YPc�p(¡¯§C¡ n Yt¡ 7 �tÁ,¦ à Y`pRZ 7 (Á,¦ à _�YPUXe`c�pRYP¬�c©Òkf�Z9Wt[Ép�¡¨W 7 � Á=� � à ÀÇW(U�Y`¬©¬5þXÊ`Na b.b>c L Ñ n cd§oÀÇW(¬©¬�Wt[�§AÀÇUXW`_í£w¬dYPc�_�Q`Ê h ¬©¡¨fkU¨p5Yt¡¨c�¦`fk¬©\(g(W(p=fM£kYPp��=UXWt¦`fÉc©¡ÉZ9c©UXfk£w¡¨¬�\�a'\�Y(Z=Z9c�p=eL Y`e`U¯YPp=e(f�_b =¬©¡¨c��=¬�c©fkUX§�f�´ÈÁ=��� à fkp9ÀÇW`U¯£wf;§0¡ n f�£}W`pR§x¡¨U¯YPc�p'¡0¡ n Yt¡ 7 � Á=¦ à _�Y`U¨e(c©p5YP¬�c©Òkf×Z9Wt[Ép¡¨W 7 �XÁ,��� à Y`pRZef!S<Á=��� à f}p9ÀÇW(UX£}fk§V¡ n f0£wW(pR§�¡XUXY`c©p'¡V¡ n YP¡ 7 Á=¦ à _�YPUXe`c�pRY`¬©c�Ò}fTZ9Wt[Ép�¡¨W 7 �xÁ=��� à ʢ�f}¡x¡¨c�p=e�f!SÈÁ=��� Ã:� ¬©p ¡ Á=��� à �xf�´<Á,��� Ã:� ¬�p ¨ Á=��� à e(c©¦(fk§VYb§¨fw¡ÄWPÀ L YPe(UXY`p=e`fÉ_Å =¬�¡Xc©�R¬©c�f}U¯§�¡ n YP¡§XYt¡¨cd§xÀÇ\/¡ n f$§�¡¯Yt¡¨c�W`p5YPUXc�¡�\�£}W`pRZ=c�¡Xc©W(pR§�c�À0Y`pRZìW`p=¬�\ ¨ � ¡ YPUXf�Í=Ð9fkZ¹�¿W`c�p'¡X§�W`À�¡ n f�¡¨ =UXa¿WZ9f;£wW9Z9c�p=e�YP¬�e`W`UXc©¡ n _�Êgy� @ � >Z�KD��`FG6=�g�`6=�;D�����2�µÖE7D'47�;FÇ8x¶��C69�tFÇ8x¶·�;:7D��.¸·�(23E7D'�æ7c©e( =UXfMN¥§ n Wt[�§A¡ n f��5YPc�U¨[Écd§xf HiJ æÆ£}W`UXU¨f;§x�¿W`p5Z9c©pReM¡XWbY¥¬©Wt[¹Z9fkpR§xc©¡�\��RYPUXc�¡�\Å£ n fk£¯«�£wW9Z9f`ÊÏ f� 5§xf J W`_�YPpƬ©f}¡x¡Xf}U¯§�¬�c�«`f�þÄ¡¨W�Z9f}p=W`¡¨fÅ¡ n f�a=c©¡X§¥YPp5ZÆØTUXf}f}«�¬�fw¡¨¡¨fkUX§T¬�c©«(f ¨ ¡¨W�Z9fkp=WP¡Xf¡ n fM�RY`U¨c©¡�\Å£ n f;£¯«9§}Ê$#vp�¡ n cd§of}Ð=YP_��=¬�f`g(¡ n f�a=c�¡¯§oþC£}Y`p�a5f�c�p�W(p=f�WPÀΡ�[VW0§�¡¯Yt¡Xfk§KZ=f}p=W`¡¨fkZ�a�\ÿ � g¿[ n c©¬�f�¡ n fÅ£ n f;£¯«×a=c©¡X§T£}Y`p�a5fÅc©p$W(p=f�WPÀofkc©e n ¡M§�¡¯Yt¡¨f;§MZ9fkp=WP¡XfkZ�a�\�ÿ ´ ʹ#°ÀA¡ n f0þ�¡ n a=c©¡cd§T£}W`p=pRfk£µ¡XfkZ�¡XWס n f ¨ ¡ n £ n f;£¯«ÓpRW�Z=f`gΡ n f}pi[Vf� R§xfb¡ n f�p=W`¡XYt¡Xc©W(piÿ�´3ÁÇþ à ¡¨W�Z9f}pRWP¡¨fÅ¡ n f§x¡XYt¡XfÄ¡ n YP¡o¡ n f�þ�¡ n p=W9Z9f�§ n W` =¬dZ�a5fMc©p�¡XW0£wW`UXUXfk§¨�5W(pRZb[Éc©¡ n ¡ n f ¨ ¡ n £ n f;£¯«0p=W9Z9fÉa¿f}c�p=e0c�p¡ n f�§x¡XYt¡Xf�ÿ�´ÈÊAÑ n f��RY`c©UX[Écd§xfÉ�¿WP¡Xf}p'¡¨cdYP¬d§ � ÁÇÿ����¨ÿ�´ à YPUXf�§xf}¡K¡XW0W`p=fMc�À3ÿ�´ÈÁ�þ Ã:� ÿ��7YPpRZ�Òkf}UXWWP¡ n f}UX[Écd§xf(Ê�Ñ n f�§xc�p=e(¬©f}¡¨W`pË�¿WP¡¨fkp'¡¨cdYP¬d§�YPUXf�§¨fw¡�¡XW � �¨Á�ÿ�� Ã9�  <Á=��� � ÿ�� à ÀÇW`U�¡ n f�a=c©¡0p=W9Z9fk§[ n c�¬©f � ´ÈÁ�ÿ�´ à cd§TW`p=f�c©ÀKÿ�´Ë£wW(U¨UXfk§¨�¿W`pRZ=§M¡¨W�YPpif}¦`fkpi�RY`U¨c©¡�\�§�¡¯Yt¡Xf�Y`pRZ$Òkf}UXW×WP¡ n f}UX[Écd§xf(ÊÑ n f0S m  =�ÎZ=Yt¡Xf�U¨ =¬�fk§ÄW(pס n cd§ H YPUX«`Wt¦�e(UXY`� n e(c©¦(fM¡ n fbØ¥YP¬�¬dYPe`fkUÉZ9fk£}W�Z=c©p=e�Y`¬©e(W`UXc�¡ n _�q

% ´�� Á�ÿ � Ã'& )+nº £ +nº 1 ��3 ¤Z+.- � ´ ÁÇÿ ´ à �/.0 �21 ´�354 � % / ´ Á�ÿ ´ à Á�NPO Ã

% �w´ ÁÇÿ ´ Ã'& � � ÁÇÿ ´ ÁÇþ èà �S 0 �21 ��3=4x´ % S�� Á�ÿ ´ ÁÇþ Ãxà Á�N9Q Ã

Page 10: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

æ7c©e( =UXfTN9qCÑ n fT�RYPc�UX[Éc�§¨f H Y`U¨«(Wt¦be(UXY`� n ÀÇW`UÄY0U¯Yt¡XfbQn»Pj=gÎÁ�jh�XN à ¬©Wt[ÚZ=f}pR§¨c�¡�\��RY`U¨c©¡�\�£ n fk£¯«£wW9Z9f(Ê�Ñ n fbp=W9Z9fk§Éc�p�¡ n f�a5W`¡x¡¨W(_ ¬dY]\`fkUÉU¨fk�=UXfk§¨f}p'¡É =p=«�p=Wt[Ép�a=c©¡X§MY`pRZÓ¡ n fbp=W9Z9fk§Éc�p�¡ n f¡¨W(�Ó¬dY]\`fkU�UXf}�RU¨f;§xfkp(¡V¡¨UXc��=¬©f}¡X§ÄW`À<a=c©¡X§kÊ I Y(£ n ¡¨UXc©�R¬©f}¡�p=W9Z9f n Y`§ÄYPp�W`a5§xfkU¨¦tYt¡Xc©W(p×£wW(p=p=f;£µ¡¨f;Z¡¨W�c©¡�¡ n Yt¡¥e`c�¦`f;§MÒkf}UXW��5W'§�¡Xf}UXc©W(U��RU¨W(aRYPa=c�¬�c�¡�\�c©Ào¡ n fÅ¡¨UXc��=¬©f}¡ n Y`§TW9Z=Z$�RY`U¨c©¡�\`洄 n f�p=W9Z9fk§c�p�¡ n fMa5W`¡x¡¨W(_ ¬�Y]\(f}U n Y]¦(fÉW`aR§¨f}UX¦tYt¡¨c�W`p�p=W9Z9f;§A¡ n Yt¡Ä£wW`UXUXfk§¨�5W(pRZ0¡XW0p=W(c�§¨\Ŧ(f}U¯§xc�W`p5§CWPÀ�¡ n f =p=«�p=Wt[Épia=c�¡¯§}洄 n fÅ�5W`¡¨f}p'¡Xc�Y`¬�§�a5f}¡�[�fkf}p�¡ n fbaRc�¡¥p=W9Z9fk§TYPp5Z�¡ n fb¡¨UXc©�R¬©f}¡Tp=W9Z9f;§�Y`U¨fŧxf}¡¡¨W�W(p=f�W`UÄÒ}fkU¨W�Z9fk�5fkpRZ9c�p=e�W`p�[ n f}¡ n fkUÉ¡ n f¥¡XU¨c��=¬�fw¡MY`e`UXf}fk§�[Éc©¡ n ¡ n f�aRc�¡MpRW�Z=f`Ê

7 �xÁ�ÿ�� Ã'& � �XÁÇÿ�� à �´ 0 �21 � 3 % ´��¨Á�ÿ�� à Á�N`N Ã7 ´ ÁÇÿ ´ Ã'& � ÁÇÿ ´ à �� 0 �21 ´�3 % �w´ Á�ÿ ´ à Á�NPj Ã

h �R�=¬©\�c�p=e/£w¬dYPc�_ ¡XW!ÍRe` RU¨fiNÆ[Éc�¬©¬TYPe(Y`c©pìe`c�¦`f�Y!SVfw¡ n f�ÀÇUXf}f�f}p=fkU¨e(\�¡ n Yt¡ÓZ9fk�5fkpRZ=§�W(pÛ�W`c�p'¡Va¿f}¬�c�fwÀ�§ÄWPÀ<¡ n f�ÀÇW`UX_ 7 Á�ÿ ´ �xÿ � à a= 9¡Äa5f;£}Y` R§xfTWPÀ<¡ n fTc�p9ÍRp=c©¡¨f¥f}pRf}UXe`\�Y`§X§¨W�£}c�YP¡¨f;Z�[Éc�¡ n£wW(p9ÍRe( =UXYP¡¨c�W`pR§3ÀÇW`U<[ n c�£ n ÿ ´ ÁÇþ à ®� ÿ � g 7 ÁÇÿ ´ �xÿ � à _b 5§�¡7a5fVWPÀ=¡ n foÀÇW(U¨_ 7 ÁÇÿ ´ Ã>¯ Á�ÿ ´ ÁÇþ ÃnU ÿ � à ÊÑ n  R§C[Vf�W(a9¡XY`c©p�Y¥§xc�_��=¬©c©ÍRf;ZÅSVfw¡ n fVÀÇU¨fkfÄf}p=fkU¨e(\¥¡ n YP¡AZ=f}�¿f}pRZ=§CW(p=¬�\0W`p 7 ÁÇÿ ´ à Y`pRZ 7 ÁÇÿ � à ÊD ba b E@E G a|°�¼ L h §¨fw¡VWPÀ�_�fk§X§¨Y`e`f;§KYPp5Z�a5fk¬©c�fwÀ�§VYPUXfÉÍ=Ð9fkZ9u��¿W`c�p'¡X§�WPÀΡ n f¥Ø¥YP¬�¬�Y`e`fkUKZ9fk£}W9Z9c©pReYP¬�e`W(U¨c©¡ n _ ÀÇW`U L ¤ m ^ £wW9Z9f;§Äc�ÀAYPp5ZÓW(p=¬�\�c�À7¡ n f}\ÓY`U¨f¥§x¡XYt¡Xc©W(pRYPUX\��¿W`c�p(¡¯§ÉWPÀ�qR S � U�) � ) +�- 7 � ÁÇÿ � à ¬�p � � Á�ÿ � Ã$Up) ´ ) +nº 7 ´ ÁÇÿ ´ à ¬�p � ´ ÁÇÿ ´ à Á�Ntz ÃY ) ´ ) + º 7 ´ÈÁÇÿ�´ à ¬�p 7 ´3Á�ÿ�´ ÃFU ) � Á5[\� U Q à ),+ - 7 �¨Á�ÿ�� à ¬©p 7 �xÁÇÿ�� à Á�N`y Ã

§¨ =a9Û�fk£w¡É¡¨W�¡ n f0£wW`p5§�¡XUXY`c©p'¡Ä¡ n YP¡ 7 ´ Á�ÿ ´ à _�Y`U¨e(c©p5YP¬�c©ÒkfTZ9Wt[Ép�¡XW 7 � ÁÇÿ � à ÀÇW`U�YP¬�¬�þ�½�ü/Á ¨ à Ê`Na b.b>c L Ñ n c�§AY`e(YPc�p�ÀÇW(¬©¬�Wt[�§3ÀÇUXW`_�£}¬�Y`c©_íQ�W`UA£}Y`pba¿f��RU¨Wt¦(f}pbZ9c�UXfk£µ¡X¬©\�a�\0Y(Z=Z9c�p=e L YPe(UXY`p=e`f_b R¬�¡Xc©�=¬�c�f}U¯§9f �w´ Á�ÿ � à ¡ n Yt¡¥f}p=ÀÇW`U¯£wfÅ¡ n f�£wW(pR§�¡XUXY`c©p'¡�¡ n YP¡ 7 ÁÇÿ ´ à _�Y`U¨e(c©p5YP¬�c©ÒkfbZ9Wt[Ép$¡XWÓÿ � Ê#°ÀA[�f0Z=fwÍRp=f±f ´�� Á�ÿ � Ã�� ¬©p % ´�� Á�ÿ � à [�fTÍRp5Z�¡ n YP¡�¡ n f¥ÍRÐ�f;Z�u°�5W(c©p'¡�fkè' RYP¡¨c�W`pR§ÉÀÇW`U�S m YPUXffkè' =c�¦tYP¬�f}p'¡Ä¡¨W�¡ n f0§�¡¯Yt¡Xc©W(pRYPUXc�¡�\�£}W`pRZ=c�¡Xc©W(pR§VÀÇW`U*R S ʹy� @�¾ >Z�KD��`FG6=�g�`6=�;D��À¿�6h�P8k25��Á5�t6!�C:<�h À�Y`£w¡¨W(U�e`U¯YP� n�é lPêMcd§�YËa=c©�5YPU¨¡¨c©¡¨f�e`U¯YP� n [Éc©¡ npcx ��à { H b � p=W9Z9f;§ÅÄ��Z9fkp=WP¡XfkZìa�\/ÍR¬�¬©f;Z§Xè( 5YPUXfk§ÈY`pRZ�Æ G a H=G�}uE�~ p=W9Z9fk§7ÿ��¿Z=f}p=W`¡¨fkZba�\� =p9ÍR¬�¬©f;Zb£}c©U¯£w¬�fk§kÊ<Ñ n f�ÀÇ =pR£w¡¨c�W`pÅp=W9Z9fk§CZ9fkp=WP¡Xf

Page 11: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

YÉZ9f;£wW`_��¿W(§¨c�¡Xc©W(p¥WPÀ9YÉåPe`¬�W`a5YP¬dåKÀÇ RpR£µ¡Xc©W(p9ÇÎÁ�ÿ à c�p'¡¨W�YÄ�=UXW9Z9 R£µ¡<WPÀ9åP¬�W�£kYP¬då�ÀÇ =pR£µ¡Xc©W(pR§�Ä�´3ÁÇÿ à ÊÏ f�[Éc©¬�¬ÈY`§X§¨ =_�f�¡ n Yt¡¹Ç�UXf}�RU¨f;§xfkp(¡¯§ÄYMÛ�W(c©p'¡MZ9cd§x¡¨UXc©a= =¡¨c�W`p�Wt¦`fkUK¡ n f�¦tYPUXc�Y`a=¬©f¥p=W9Z9f;§}Êh §M§ n Wt[Ép�c�p é Qtskê�g5YPp�\�À�Y`£w¡¨W(U�e`U¯YP� n £}Y`p�a¿f0£wW(p�¦`f}U¨¡¨f;Z×c�p'¡¨W�Y��RY`c©UX[Éc�§¨f H YPUX«`Wt¦�e(UXY`� n¡ n Yt¡�U¨fk�=U¨f;§xfkp'¡X§�¡ n f�§¨Y`_�f�Z9cd§�¡XU¨c�a= 9¡Xc©W(p3Ê�Ñ n f×�=UXW9£wf;Z9 =UXf×cd§bcdZ9f}p'¡Xc�£kYP¬V¡¨Wi¡ n f�W`p=f�Z9fwu§X£wUXc©a¿fkZ�c�pÓ¡ n f��=UXf}¦�c�W` R§É§¨fk£w¡¨c�W`pÓÀÇW`U L ¤ m ^ §}qA[VfVÛ� R§�¡�p=f}f;ZסXW�¡¨UXfkYt¡ÉÀÇ RpR£µ¡Xc©W(p�pRW�Z=fk§Éc�p¡ n f�§XYP_�fb[ÄY]\�[�fb¡¨UXfkYP¡¨fkZÆ£ n fk£¯«�p=W9Z9f;§�c�p$¡ n f L ¤ m ^ Ê�Ñ n cd§M�=UXW9£wf;Z9 =UXfb\�c©fk¬�ZR§TY×�RYPc�U¨u[Écd§xf H YPUX«`Wt¦Óe`U¯YP� n §xWס n YP¡Tc©¡¨fkUXYP¡¨c�p=eס n f�S m fkè' RYP¡¨c�W`pR§�W`pi¡ n cd§Te`U¯YP� n cd§�fkè' =c�¦]Y`¬©fkp'¡¡¨W�UX =p=p=c�p=e�¡ n f0§x R_Åu°�=UXW9Z9 R£µ¡�Y`¬©e(W`UXc�¡ n _íW`p�¡ n f¥À�Y`£µ¡XW`UÉe(UXY`� nÆé Q]skê°ÊÏ f�YPe(Y`c©pËZ9f}pRWP¡¨f�a�\$ÿ�´Æ¡ n f�§¨fw¡0W`À�¦tYPUXcdYPa=¬�fk§�¡ n Yt¡�ÀÇW`UX_Ù¡ n f�Z9W`_�Y`c©pËWPÀVÀÇ =pR£w¡¨c�W`pBÄP´�Êh §ba5f}ÀÇW`UXf`g<¡ n fÓS m Y`¬©e(W`UXc�¡ n _ [Éc�¬�¬�£}Y`¬�£} =¬dYt¡¨f�a¿f}¬�c�fwÀ�§0Wt¦(f}U�c�pRZ9c�¦�c�Z9 5YP¬K¦tY`U¨cdYPa=¬�fk§ 7 �xÁÇÿ�� ÃY`§M[�fk¬©¬CY`§oÛ�W`c�p'¡Ta¿f}¬�c©f}À�§MWt¦(f}UMY`¬©¬7¦tYPUXc�Y`a=¬�fk§É¡ n Yt¡�ÀÇW`UX_ý¡ n f�Z9W`_�YPc�p$W`ÀAYÓ§xc�p=e(¬©fbÀÇ =pR£µ¡Xc©W(p7 ´ÈÁÇÿ�´ à ÊoÑ n f¥SVf}¡ n fTÀÇUXf}f¥f}p=fkU¨e(\�WPÀÈ¡ n f��RY`c©UX[Éc�§¨f H YPUX«`Wt¦�e`U¯YP� n £}Y`pÓa¿f�§xc�_��=¬©c©ÍRf;ZÓY`pRZ[VfTW(a9¡XY`c©p�Y�SVfw¡ n fTÀÇU¨fkf�f}p=fkU¨e(\�¡ n YP¡MZ9f}�¿f}pRZR§ÉW`p=¬�\�W(p 7 1 ÿ�� à Y`pRZ 7 ´3Á�ÿ�´ à ÊD ba b E@E G a|°·È L h §¨fw¡�WPÀ0_�fk§X§XYPe`f;§�YPp5Z�a5fk¬©c�fwÀ�§�YPUXf�Í=Ð9f;Z�u��¿W`c�p'¡X§×WPÀ�¡ n fƧ¨ =_�u��=UXW9Z9 R£w¡YP¬�e`W(U¨c©¡ n _íW`p�YÅÀ�Y`£µ¡XW`UÉe(UXY`� n c©ÀCY`pRZ�W`p=¬�\�c©À<¡ n f}\�YPUXfT§x¡XYP¡¨c�W`pRY`U¨\��¿W`c�p'¡X§ÉW`À�qRTS � U ) ´ ) +nºª7 ´ÈÁÇÿ�´ à ¬�p�ÄP´3Á�ÿ�´ à Á�NPr ÃY ) ´ ) + º 7 ´ ÁÇÿ ´ à ¬�p 7 ´ Á�ÿ ´ ÃFU ) � Á5[ � U Q à ) + - 7 � Á�ÿ � à ¬©p 7 � ÁÇÿ � à Á�N(s Ã

§¨ =a9Û�fk£w¡�¡XW�¡ n f�£}W`pR§x¡¨U¯YPc�p'¡�¡ n Yt¡ 7 ´3Á�ÿ�´ à _�YPUXe`c�pRYP¬�c�Ò}fbZ9Wt[Ép$¡¨W 7 �¨Á�ÿ�� à ÀÇW(UTY`¬©¬7þ*½!ü/Á ¨ à ʮ�fkU¨fÀ[��Ac�§V¡ n f�p� =_ba¿f}UÉW`À<ÀÇ RpR£µ¡Xc©W(pR§Ä¡ n Yt¡�¦tYPUXc�Y`a=¬�f�þ��RYPU¨¡¨cd£wc��RYt¡Xfk§Vc�p3Ê� @ É >Z�KD��`FG6=�g�`6=�;D���E<F��]D��`8kD�E�ÁR�t6!�C:7�ªÊ>Ë�6�¶¿D'�;F�6R4Ú4<D'8xµb25�n¸3�nÌSÄY]\`f;§xcdYPp�p=f}¡�[�W(U¨«9§�YPUXf�Z9c�U¨f;£µ¡XfkZ�e(UXY`� n §�c©pÓ[ n cd£ n ¡ n fT£}W`pRZ9c©¡¨c�W`p5YP¬¿�=U¨W(aRYPaRc©¬�c�¡�\�Wt¦(f}U�Y`¬©¬n cdZ=Z9fkp�pRW�Z=fk§Écd§Ä[ÉU¨c©¡x¡Xf}p�Y`§kq

� ÁÇÿF� � Ã�� Q� � � � ÁÇÿ � � �ÀÍ�Î ÁÇÿ � èÃ>� Á,� � � ÿ � à Á�NP­ Ã

[ n fkU¨f �ÀÍ�Î ÁÇÿ � à _�f;YPpR§V¡ n f0�RY`U¨fkp'¡X§ÉW`À7ÿ � Ê Ï f n Y]¦`f�Y`§X§x R_�f;ZÓ¡ n YP¡Mf}¦(f}UX\ n cdZ=Z9f}p�pRW�Z=fÿ � n Y(§�Y`piW(aR§xfkU¨¦(fkZ�p=W9Z9f�� � £wW(p=p=fk£w¡¨f;Z�¡¨W�c©¡kg3YPp5Z$YPUXfŧ¨ =aR§¨ =_�c©pReס n fÅ�RU¨c�W`UTWt¦`fkU�¡ n fUXW'W`¡Ép=W9Z9f¥c©p'¡¨W�c©¡X§�¬©W9£}Y`¬�f}¦�cdZ9f}pR£}f`Êæ=W(UK£wW(_��=¬©f}¡¨fkp=fk§X§A[Vf�e`c�¦`f m f;YPUX¬GÔ §AYP¬�e`W(U¨c©¡ n _� R§xc�p=e�¡ n fÉÀÇW(U¨_Å =¬�YP¡¨c�W`p�WPÀ m f}W`¡KYPpRZ�¢ n Y`£wu¡¨fkU é Qkztê�Ê I Y(£ n p=W9Z9f�ÿ��¥£wW(_��= 9¡¨f;§�Yia¿f}¬�c�fwÀ 7 ÁÇÿ�� à g�a�\/£wW`_Åa=c�p=c©pReÆ_�fk§X§XYPe`f;§0ÀÇUXW`_ c©¡X§£ n c�¬�Z=U¨fkpef!� �¯ÁÇÿ�� à YPp5ZÓ_�f;§¨§XYPe(fk§�ÀÇU¨W(_öc©¡X§É�RY`U¨fkp'¡X§NÏ / �¯Á�ÿ / à Ê7 Á�ÿ�� Ã�� ¨ f<ÁÇÿ�� à ÏoÁÇÿ�� à Á�NPl Ã[ n fkU¨fT¡ n f±f�YPpRZÐÏ!è' RYPp'¡¨c©¡¨c�fk§ÉY`U¨f¥Z9fkU¨c�¦`f;Z�ÀÇUXW`_ c�¡Xf}U¯Yt¡¨c�p=e5qfÈÁ�ÿ�� Ã:&Ñ� Á=��� � ÿ�� à � � f!� �XÁ�ÿ�� à ÁGj`O Ã

YPp5Z�q ÏoÁ�ÿ � Ã�& )ÒTÓuÔ 1 + 3 � Á,Õ � ÿF� �ÀÍ�Î ÁÇÿ Ãxà � / Ï / � Á�ÿ / à ÁGj=Q Ã

Page 12: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

x1

x2 x3

x5x4

x6

x1

x2 x3

x5

x4

x6

x7=(x2’,x3’)

Y aæ7c©e( =UXf�jRq h p�\iSÄY]\`f;§xcdYPpÆp=fw¡�[VW`UX«$£}Y`pÆa¿f�£wW`p�¦(f}U¨¡¨fkZic©p'¡XW$YPpË =pRZ9c�U¨f;£µ¡XfkZÆe`U¯YP� n [Éc�¡ n�RY`c©UX[Éc�§¨f¥£w¬�c�è' =f;§�a�\�Y(Z=Z9c�p=e�£}¬© 5§�¡Xf}UÄp=W9Z9fk§VÀÇW(UÉYP¬�¬¿�5YPUXf}p'¡X§K¡ n YP¡�§ n Y`U¨f¥Y�£wW`_�_�W`p�£ n c�¬�Z�Ê6�@?h SVY]\(fk§¨c�Y`p/p=f}¡�[�W(U¨«ÎÊ �V@ Ñ n f�£}W`UXU¨f;§x�¿W`p5Z9c©pRei =pRZ=c©UXfk£w¡¨fkZìe(UXY`� n [Éc©¡ n �RYPc�UX[Éc�§¨f£w¬�cdè( Rfk§kÊ h £}¬© R§x¡¨fkUÓpRW�Z=f$ÀÇW`UËÁ=ÖÅ��× Ã×n Y`§×a5fkf}pÂY(Z=Z9fkZ3Ê Ñ n fÆ�5W`¡¨f}p'¡Xc�Y`¬�§Ó£}YPpÂa5f˧xf}¡§¨W�¡ n YP¡�¡ n f�Û�W(c©p'¡0�=UXW`a5YPa=c�¬©c©¡�\$c�p!¡ n f� RpRZ9c�U¨f;£µ¡¨f;Z!p=fw¡�[VW`UX«�cd§0cdZ9f}p'¡¨cd£}Y`¬A¡¨W�¡ n Yt¡0c�p!¡ n fSÄY]\`f;§xcdYPp�p=f}¡�[�W(U¨«ÎÊ�#vp$¡ n cd§¥£}Y`§¨f�¡ n fÅ R�¿Z=YP¡¨f�U¨ R¬©f;§M�=UXfk§¨f}p'¡XfkZ�c�p$¡ n c�§��RYP�¿f}UTUXfkZ9 R£}f0¡XWm fkY`U¨¬�Ô §��RU¨W(�RYPe'Yt¡¨c�W`p×U¨ =¬�fk§Äc�p�¡ n f�W(U¨c�e`c�pRY`¬ÎSÄY]\(fk§¨c�Y`p×p=f}¡�[�W(U¨« é Qkr]ê°Ê

Page 13: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

Ñ n f�_�fk§X§XYPe`fTÿ��o�RY`§X§xf;§�¡¨W�c©¡X§Ä�RY`U¨fkp(¡Éÿ / cd§Ée(c©¦(f}p×a�\Îqfh� / Á�ÿ / Ã�& ¨ )=+�- fÈÁ�ÿ�� à )¥ £ ¥Ø|¤Z+nØ � ÁÇÿ��x� � à ��n§¤ / Ï!� �¯Á,�h� à ÁGj(N Ã

[ n fkU¨f ] ¥ £ ¥ Ø ¤T+ Ø _�fkYPp5§�§x R_ÜWt¦`fkUÉYP¬�¬È£wW(p9ÍRe` RUXYP¡¨c�W`pR§�ÀÇW`U�¡ n f0�5YPUXf}p'¡X§ÄW`À7ÿ � [ n f}UXf¥¡ n f¦tYP¬� =f¥WPÀCÿ / cd§É£w¬dYP_��5f;Z�ÊÑ n f�_�fk§X§XYPe`fTÿ��K§xfkpRZ=§V¡¨W�c©¡X§�£ n c©¬dZ�ÿ!��c�§Äe(c©¦(f}p�a'\ÎqÏ����'ÁÇÿ�� Ã:& ¨ ÏoÁ�ÿ�� Ã>� Á=����� ÿ�� à �/ §¤ � fh� / Á�ÿ�� à ÁGj`j Ã

h §�§ n Wt[ÉpËc©pÆÍRe( =U¨f�j�f}¦(f}UX\$Z9c©UXfk£w¡¨f;Zie(UXY`� n £kYPpËa5f�¡¨U¯YPp5§�ÀÇW(U¨_�fkZic©p'¡XW�Y`pÆ RpRZ9c�U¨f;£µ¡¨f;Ze`U¯YP� n [Éc©¡ n �RYPc�UX[Éc�§¨f¥�5W`¡¨f}p'¡Xc�Y`¬�§kÊAæ= =U¨¡ n fkU¨_�W(U¨f(g�Y(§ÉZ9fw¡¯YPc�¬©f;Z�c©p é Q;r]ê°g=U¨ Rp=p=c�p=e�¡ n f� =p5Z9c�uUXfk£µ¡XfkZÚS m fkè' RYP¡¨c�W`pR§ÓW`p�¡ n fÆ¡¨U¯YPpR§xÀÇW`UX_�fkZ�e(UXY`� n cd§×f;è( Rc©¦tYP¬�f}p'¡�¡¨WìU¨ Rp=p=c�p=e m f;YPUX¬GÔ §YP¬�e`W(U¨c©¡ n _öW`pÓ¡ n fÅZ9c©UXfk£w¡¨f;Z×e(UXY`� n Ê Ï f�f}_�� n Y(§xc�Ò}f¥¡ n Yt¡É¡ n c�§�fkè' =c�¦]Y`¬©fkpR£wf n W`¬dZ=§VÀÇW`U�fk¦'uf}UX\�c�¡Xf}U¯Yt¡¨c�W`pÈqÈf}¦(f}UX\Å_�f;§¨§XYPe(f�§xfkp(¡�a'\� =pRZ=c©UXfk£w¡¨fkZ×S m W`p�¡ n f�¡¨U¯YPp5§�ÀÇW(U¨_�fkZ�e(UXY`� n�n Y`§KY£wW(U¨UXfk§¨�¿W`pRZ9c�p=e¥_�fk§X§¨Y`e`fÄc�p m fkYPUX¬�Ô §CYP¬�e`W(U¨c©¡ n _�W`p�¡ n fMZ9c�U¨f;£µ¡XfkZ�e`U¯YP� n Ê7æRW`Uof}Ð9Y`_��=¬©f(gPÀÇW(Up=W9Z9f;§ n Y]¦�c�p=eÅYŧxc�p=e(¬©f��RYPUXf}p'¡gf�� / ÁÇÿ / à c�§�fwÐ=Y(£µ¡¨¬�\ % � / Á�ÿ / à ÊAæ=W`UÄp=W9Z9f;§ n Y]¦'c�p=eÅ_b =¬©¡¨c��=¬�f�RY`U¨fkp(¡¯§}ghf�� / ÁÇÿ / à cd§Ä¡ n f0_�fk§X§¨Y`e`f¥�RY`§X§¨fkZ×ÀÇU¨W(_ö¡ n f$䨣}¬© R§x¡¨fkUÉp=W9Z9fkå5ÁÇf(Ê e5Êoÿ�Ù�c�p�ÍRe( =U¨f�j á¨W�¡ n Yt¡��RY`U¨fkp'¡kÊÚ  R§�¡�Y`§�[VfÓ£kYPp� 5§xf� =pRZ9c�UXfk£µ¡XfkZìS m ¡¨WËW`a=¡XYPc�p��5YPc�U¨[Écd§xfÓa5fk¬©c�fwÀ�§kgK[Vf�£}YPp) R§¨f m f;YPUX¬GÔ §YP¬�e`W(U¨c©¡ n _Ù¡¨W�£}Y`¬�£} =¬dYt¡¨f c GPI�H@E ° }x~uE�H=~ c zµq0¡ n f¥Û�W`c�p'¡0a5fk¬©c�fwÀÉWt¦`fkU�YÓpRW�Z=f�YPpRZËc©¡X§��RY`U¨fkp'¡X§kÊÏ fbZ9fwÍ5p=f�¡ n c�§Ma5fk¬©c�fwÀ�g 7 ÁÇÿ � � �ÀÍ�Î Á�ÿ � à g¿Y`§Ä¡ n f0�=UXW9Z9 R£µ¡�WPÀÉÁ�Q à YP¬�¬3_�fk§X§¨Y`e`fk§É£}W`_�c�p=e�c©p'¡XWÿ � YPpRZËc©¡X§b�RYPUXf}p'¡X§kg�Á�N à ¡ n f�¬�W9£}YP¬�_�fk§X§¨Y`e`fk§TÀÇW(UbYP¬�¬op=W9Z9f;§0c�pË¡ n f×À�YP_�c©¬�\ÆY`pRZ�Á�j à ¡ n f£wW(pRZ9c©¡¨c�W`pRY`¬ � ÁÇÿ � � �ÀÍ�Î Á�ÿ � Ãxà ÊKÑ n  R§kq7 ÁÇÿ���� �ÀÍ�Î Á�ÿ�� ÃxÃ�� ¨ f<ÁÇÿ�� Ã>� ÁÇÿ���� �ÀÍ�Î ÁÇÿ�� èà �+.Ø 0�ÒTÓ|Ô 1 +.- 3 Ï / �¯ÁÇÿ / à ÁGjPz Ã

ª §¨c�p=e�¡ n c�§È¡¨U¯YPp5§�ÀÇW(U¨_�Yt¡Xc©W(p3g;[�fV£}YPpÅYP�=�=¬�\�£w¬dYPc�_ QKYPpRZ0W`a9¡¯YPc�p�¡ n fÄSVfw¡ n foÀÇUXf}f�f}p=fkU¨e(\MÀÇW(U¡ n f¥¡¨U¯YPpR§xÀÇW`UX_�fkZÓe`U¯YP� n Ê h §Éc©p�¡ n f0£}Y(§xf¥WPÀ7¡¨ RU¨a¿W�£wW9Z9f;§ÉYPpRZ L ¤ m ^ §kg9¡ n c�§ÉÀÇU¨fkfTfkp=f}UXe`\£}Y`p$a¿f�§¨c©_��=¬�c©ÍRfkZia5f;£}YP 5§xf�§xW(_�fÅ£wW`p=ÍRe` =U¯Yt¡Xc©W(pR§�c�p$¡ n f0¡XUXY`pR§xÀÇW`UX_�f;Z�e`U¯YP� n�n Y]¦`f0Òkf}UXW�=UXW`aRY`a=c�¬©c©¡�\`Ê Ï f�£}Y`p�¡ n f}pìU¨fk[ÉU¨c©¡¨f�¡ n f�SVfw¡ n f�ÀÇUXf}f�f}p=fkU¨e(\Æc�p/¡Xf}UX_�§�W`ÀÉ¡ n f�Z9c�U¨f;£µ¡¨f;Ze`U¯YP� n g=YPpRZÓW`a=¡XYPc�pÓ¡ n f¥ÀÇW(¬©¬�Wt[Éc©pRe�£}W`UXW`¬�¬�Y`U¨\(ÊD ba b E@E G a|°ÐÛ L L f}¡�M 7 � ÁÇÿ � à O(g M 7 Á�ÿ � � �ÀÍ�Î ÁÇÿ � èà O�a¿f�Y�§xf}¡0WPÀÄa5fk¬©c�fwÀ�§ÅZ9f}UXc©¦(fkZiÀÇU¨W(_ m f;YPUX¬GÔ §YP¬�e`W(U¨c©¡ n _ýW(pËY�Z9c�U¨f;£µ¡XfkZ$e(UXY`� n [Éc©¡ n YPUXa=c©¡¨U¯YPUX\�¡¨W`�¿W`¬�W`e(\`Ê0Ñ n f�a¿f}¬�c©f}À�§�£wW(U¨UXfk§¨�5W(pRZ�¡XWÍ=Ð9fkZ9u��¿W`c�p'¡X§7WPÀ m f;YPUX¬GÔ §ÈY`¬©e(W`UXc�¡ n _�c�ÀÎYPpRZbW`p=¬�\�c©ÀR¡ n f}\0YPUXf�£}W`pR§x¡¨U¯YPc�p=f;Z0§�¡¯Yt¡Xc©W(pRYPUX\¥�5W(c©p'¡X§WPÀ7¡ n f0S�f}¡ n fTÀÇUXf}f�fkp=f}UXe`\ÎqRTS � U ) � )+.- V ÒTÓ|Ô 1 +.- 3 7 ÁÇÿ���� �ÀÍ�Î Á�ÿ�� Ãxà ¬©W(e � ÁÇÿ��x� �ÀÍ�Î ÁÇÿ�� ÃxÃ:U ) � )=+ - 7 ÁÇÿ�� à ¬©W(e � Á,���x� ÿ�� ÃY ) � )+.- V ÒTÓ|Ô 1 +.- 3 7 ÁÇÿ � � �ÀÍ�Î Á�ÿ � Ãxà ¬©W(e 7 ÁÇÿ � � �ÀÍ�Î Á�ÿ � ÃxÃÜU ) � Á5[ � U Q à ) + - 7 Á�ÿ � à ¬�W`e 7 Á�ÿ � Ã

Á�j'y ÃÑ n fÓ_�c�p=c�_�c©Ò;Yt¡¨c�W`p)c�§�£wW(pR§x¡¨U¯YPc�p=fkZ/§¨W�¡ n Yt¡ 7 ÁÇÿ���� �ÀÍ�Î Á�ÿ�� Ãxà _�YPUXe`c�pRYP¬�c�Ò}fk§bZ9Wt[Ép�¡¨WÆ¡ n f§¨c©p=e(¬©f}¡¨W(pÆa¿f}¬�c©f}À 7 ÁÇÿ!� à ÀÇW(U0YP¬�¬Aÿ!��½ÂÁ�ÿ���� �ÀÍ�Î ÁÇÿ�� èà Ê�®Mf}UXf�[��Äc�§T¡ n f�p� =_ba¿f}U�W`ÀKÀ�YP_�c�¬©c�fk§

Page 14: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

¡ n Yt¡¥p=W9Z9fÅÿ��V�RYPU¨¡¨cd£wc��RYt¡Xfk§Tc�p3g�[ n cd£ n cd§Mf;è( 5YP¬7¡¨W�W`p=fÅ�=¬� R§T¡ n f�p� =_ba¿f}U¥W`À�£ n c�¬�Z9UXf}piWPÀÿ��AÀÇW`U�YP¬�¬�p=W(p=U¨W�W`¡ÄpRW�Z=fk§kÊ� @wÝ Þ}�p�7�GF=�(698kFG2547�Ñ n f¥À�Y(£µ¡É¡ n Yt¡�RTSÈÁ6M 7 ����� 7 �6O à c�§Éa¿W` RpRZ9fkZ�a¿f}¬�Wt[Âc�_��=¬©c�fk§É¡ n YP¡�¡ n f0S m fkè' RYP¡¨c�W`pR§ÉY`¬©[ÄY]\9§�¿W(§X§xf;§¨§TY�Í=Ð9f;Z�u��¿W`c�p'¡�Á�W`a9¡¯YPc�p=fkZÆYt¡M¡ n f�e`¬�W`aRY`¬<_�c�p=c©_Å =_ÙWPÀNRZS à ʥÑ<W×W( =UT«'pRWt[ɬ©f;Z9e`f(g¡ n c�§Écd§V¡ n f¥ÍRUX§x¡É�=UXW'W`À<W`À<fwÐ9cd§�¡Xf}pR£}f¥WPÀ7Í=Ð9fkZ�u°�5W(c©p'¡¯§VÀÇW`U�Y�e`fkp=f}U¯YP¬Îe`U¯YP� n [Éc©¡ n YPUXa=c©¡¨U¯YPUX\�¿WP¡¨fkp'¡¨cdYP¬d§�Á�§¨f}f é Q;y;ê3ÀÇW(U�Y�_�W(U¨f¥�=UXW�WPÀ<ÀÇW`UÄ¡ n f0§x�¿fk£}c�Y`¬3£}Y`§¨f¥WPÀ<¡¨ =UXa5W�£}W9Z9fk§ à ÊÑ n fAÀÇUXf}fAfkp=f}UXe`\�ÀÇW`UX_b =¬dYt¡Xc©W(p¥£}¬�Y`U¨c©ÍRf;§¿¡ n foUXf}¬dYt¡Xc©W(pR§ n c©�T¡XWɦ]Y`U¨cdYt¡Xc©W(pRYP¬PY`�=�=UXW(Y(£ n fk§¿[ n cd£ nYP¬d§¨WÅ_�c�p=c©_�c�Ò}f�YPp�YP�R�=U¨W]Ð9c�_�Yt¡¨fMÀÇU¨fkf¥f}p=fkU¨e(\ é r]ê°Êoæ=W(UÄfwÐ=YP_��=¬�f`g�¡ n f¥_�f;YPpÓÍRf}¬dZ�YP�=�=UXW]Ð�uc�_�YP¡¨c�W`pÓÍRpRZ=§MY�§¨fw¡�W`ÀQM 7 � O�¡ n Yt¡É_�c©pRc©_�c�Ò}f`qRFß�àMÁ6M 7 �6O Ã:��U ) ��� )+ -WV +�X 7 �xÁ�ÿ�� à 7 �(Á�ÿ!� à ¬�p � ����ÁÇÿ�� �xÿ<� à Y ) � ) + - 7 �xÁ�ÿ�� Ã�é ¬©p 7 �xÁÇÿ�� Ã$U ¬�p � �¨Á�ÿ�� à ê

ÁGj`r 稠=a9Û�fk£w¡É¡¨W�¡ n f0£wW`p5§�¡XUXY`c©p'¡ ] � 7 �xÁ�ÿ�� Ã�� Q(ÊÑ n fÉSVfw¡ n fÄÀÇU¨fkfÄf}p=fkU¨e(\�c©pR£}¬© 5Z9fk§7ÍRU¯§x¡xu°W`U¯Z9f}U<¡¨fkU¨_�§ 7 �¨ÁÇÿ�� à Y`§C[Vf}¬�¬=Y`§A§¨fk£}W`pRZ9u�W(UXZ9fkU3¡Xf}UX_�§7 ���(ÁÇÿ����¨ÿ!� à gR[ n c�¬©f0¡ n f0_�f;YPp�ÍRf}¬dZ�ÀÇUXf}f0fkp=f}UXe`\� 5§xf;§�W`pR¬©\ס n f�Í5UX§x¡�W`U¯Z9f}UMW`p=f;§}ÊVÑ n fbS mÀÇUXf}f�f}pRf}UXe`\/cd§�f}Ð9Y(£µ¡�ÀÇW(U�¡¨UXf}f;§�[ n c©¬�f�¡ n f$_�fkY`p)Í5f}¬dZ¹W`p=f�cd§�p=W`¡kÊ%æR =U¨¡ n fkU¨_�W`UXf�¡ n fW`�=¡¨c�_�c�ÒkYP¡¨c�W`pÚ_�f}¡ n W9Z=§�YPUXfiZ9c á¿fkU¨fkp'¡kq�¡�\��=cd£}Y`¬©¬�\_RFß�àícd§Ó_�c�p=c�_�c�Ò}f;ZÚZ9c�U¨f;£µ¡X¬©\¹c�pÚ¡ n f�=UXc©_�Y`¬V¦tY`U¨cdYPa=¬�fk§ÅM 7 �6O(qÓc�¡�cd§�fkY`§¨\Ë¡¨W�Z9f}ÍRp=f�¬©W9£}Y`¬Ä =�¿ZRYt¡¨f;§�ÀÇW`U 7 �XÁÇÿ�� à ¡ n YP¡�_�c�p=c�_�c©ÒkfRFß�à YP¡�fk¦`fkU¨\�c©¡¨fkUXYP¡¨c�W`p3Ê·#vp�£wW(p(¡XUXY(§�¡;go¡ n f�S m c©¡¨fkUXYP¡¨c�W`pR§�[VW`UX«�[Éc�¡ n ¡ n f�_�fk§X§XYPe`f;§[ n cd£ n YPUXfb£}W`_baRc©pRYP¡¨c�W`pR§MWPÀo¡ n fÅZ= RYP¬7¦tYPUXc�Y`a=¬©f;§ÀMfh���'ÁÇÿ<� à O'ÊTÑ n f�S m c©¡¨f}U¯Yt¡Xc©W(pR§MZ=W×p=W`¡p=f;£wfk§X§XYPUXc©¬�\Ó_�c�p=c�_�c�Ò}f�R S Yt¡¥f}¦(f}UX\Óc©¡¨fkUXYP¡¨c�W`p3qÉc�p$À�Y(£µ¡kgÎÀÇW(U�c�p'¡¨fkU¨_�fkZ=c�YP¡¨fbc©¡¨fkUXYP¡¨c�W`pR§M¡ n f7 ��� � 7 � £kYP¬d£w =¬dYt¡XfkZÓ R§¨c©p=e�S m Z9W�p=W`¡É§XYt¡Xc�§xÀÇ\�¡ n f�_�YPUXe`c�pRYP¬�c©Ò;Yt¡Xc©W(pÓ£wW(pR§x¡¨U¯YPc�pR§}ÊÕ�YPa5Y`§ n c©_�YÅYPpRZ�¢9Y`Y(Z é Qkl]ê n Y]¦`fT�=U¨fk¦�c©W( R§x¬�\��¿W`c�p(¡XfkZÓW` 9¡Ä¡ n f�£wW`UXUXfk§¨�5W(pRZ9f}p5£wfMa¿fw¡�[Vf}fkpS m YPpRZË¡ n fÓSVf}¡ n f×Y`�=�=UXW]Ð9c©_�Yt¡Xc©W(pìÁ�fwÐ9�=UXfk§X§xf;ZË R§xc�p=e�¡ n fÓÑ hMm ÀÇW(U¨_�Y`¬©cd§x_ à ÀÇW(Uŧ¨W`_�f§¨�5f;£wc©Í5£¥e`U¯YP� n c�£kYP¬�_�W9Z9f}¬d§É[Éc©¡ n U¯YPpRZ9W(_ Z9cd§¨W`U¯Z9f}U;Ê ç  =U��=UXW�WPÀAYPpR§¨[Vf}U¯§�c�p�¡ n f0Yâ�U¨_�YPu¡¨c�¦`fÓ¡ n f}c�U�è' =f;§�¡Xc©W(p)Y`a5W( 9¡�[ n fw¡ n f}U�¡ n fkU¨f�cd§�Y%ä¨Z=f}f}��e`f}pRf}U¯YP¬V¬©c�p=«�a¿fw¡�[Vf}f}p�¡ n f�¡�[VW_�fw¡ n W9Z=§}Ê å é Q;l]êã ä ã�åÄâ�¾Rú�ãÅ· ó�ó ¼=à ÷ ã�æ-½7»9ãxà ß ºË»9à »=ú ðèç ¼ ð3ðêé�ß�ð ¼=ôoûÑ n f�S�f}¡ n f�YP�=�RU¨W]Ð9c�_�YP¡¨c�W`p3g=ÀÇW`UT[ n c�£ n ¡ n fbfkp=f}UXe`\�YPp5Z�f}p'¡¨UXW`��\�YPUXfbYP�=�RU¨W]Ð9c�_�YP¡¨f;ZÓa�\¡¨fkU¨_�§0¡ n YP¡Åc�p'¦(W`¬�¦`f�YP¡Å_�W(§x¡b�RY`c©U¯§�WPÀ�pRW�Z=fk§kg7cd§b¡ n fק¨c�_��R¬©f;§�¡�¦`f}U¯§¨c©W(pËW`ÀÉ¡ n f�Õ¥c©«� R£ n c䨣}¬© 5§�¡Xf}UɦtYPUXc�YP¡¨c�W`pRY`¬Î_�f}¡ n W9Z�Ê å é s'g=zRg¿y;êT#vp�Y�e`f}pRf}U¯YP¬3Õ¥c�«' 5£ n c3YP�=�RU¨W]Ð9c�_�YP¡¨c�W`p3g'¡ n f�f}p9uf}UXe`\�Á�W`UÈfkp'¡¨UXW`��\ à c�§7YP�R�=U¨W]Ð9c�_�Yt¡¨f;Z�Y`§7YM§¨ =_�WPÀ9¡ n fVf}p=fkU¨e(c©f;§kÁ�W`UÈfkp'¡¨UXW`�=c�fk§ à WPÀ5aRY`§¨c�£K£w¬� R§�u¡¨fkUX§KWPÀ�p=W9Z9f;§}g'_�c©p� R§K¡ n f�f}p=fkU¨e(c©f;§kÁ�W`UAfkp(¡XU¨W(�=c�fk§ à WPÀ�Wt¦`fkUxuv£wW( =p'¡¨f;ZÅ£}¬© R§x¡¨fkUKc�p'¡¨f}U¯§¨fk£µ¡Xc©W(pR§kg_�c©p� R§�¡ n f�f}p=fkU¨e(c©f;§kÁ�W`UTf}p'¡¨UXW`�=c�fk§ à W`ÀK¡ n fÅWt¦(f}U¨u°£}W` =p'¡XfkZ�c�p(¡Xf}U¯§xf;£µ¡Xc©W(pR§TWPÀKc�p'¡¨f}U¯§¨fk£µ¡Xc©W(pR§kgYPp5ZÓ§¨W�W`p3Êæ7c©e( =UXf0z(Y�§ n Wt[�§�Y`p�fwÐ=YP_��=¬�f`Ê�Ñ<W�W(a9¡XY`c©p�¡ n fÅSVf}¡ n fbY`�=�=UXW]Ð9c©_�Yt¡Xc©W(p3g=[Vf0 R§¨fÅY(§�aRY(§xcd££w¬� R§x¡¨fkUX§V¡ n f0§xf}¡�WPÀAYP¬�¬��RYPc�U¯§ÄWPÀCp=W9Z9fk§kÊoÑ n f0SVfw¡ n f�f}p'¡¨UXW`��\�cd§Ä¡ n f0§x =_ WPÀAYP¬�¬�f}p'¡¨UXW`�Rc©f;§c�pÓ¡ n f�aRY(§xcd£¥£w¬� R§�¡Xf}U¯§Ä_�c©p� R§V¡ n f�f}p'¡¨UXW`�Rc©f;§ÄWPÀ<Wt¦(f}U¨u°£}W` =p'¡XfkZ×£}¬© 5§�¡Xf}UÉc�p(¡Xf}U¯§xf;£µ¡Xc©W(pR§}Êë SÈÁ�ÿ Ãì� ë Á�ÿ �� ÃZY�ë ÁÇÿ í ÃTY�ë ÁÇÿ�î�ï ÃTY�ë ÁÇÿ�ï ð à ÁGj's ÃYjë Á�ÿ � î ÃTY�ë ÁÇÿ ï ÃTY�ë ÁÇÿ í ð à ÁGj`­ Ã

Page 15: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

1 2 3

4 5 6

Á�Y Ã

1 2 3

4 5 6

7 8 9

ÁÇa Ãæ7c©e( =U¨f¥z5qCÑÄ[VW�fwÐ=YP_��=¬�fk§VÀÇW(UÄ¡ n f0Õ¥c�«� R£ n c3Y`�=�=UXW]Ð�c�_�Yt¡Xc©W(pR§}ÊA¢�fkf�¡XfwÐ�¡ÉÀÇW`U�Z=fw¡XY`c©¬d§kÊ

U*ë Á�ÿ�� ÃÜU�ë Á�ÿ�í Ã$U�ë ÁÇÿ ð ÃFUñë ÁÇÿ î à ÁGj`l ÃU N ë ÁÇÿ Ã$U N ë ÁÇÿ�ï à Á�z(O Ã[ n fkU¨f0[Vf0Z9fkp=WP¡Xf0a�\ ë Á�ÿ�� à ¡ n f0f}p'¡XU¨W(�'\ÓWPÀAÿ��¨q ë ÁÇÿ�� Ãg�òU ] +.- � Á�ÿ�� à ¬�p � ÁÇÿ�� Ã Ê � WP¡Xf¡ n Yt¡×¡ n cd§×cd§�f}Ð9Y(£µ¡X¬©\)¡ n fËSVfw¡ n fif}p'¡XU¨W(�'\ì R§¨fkZ¹c�p£w¬dYPc�_ Q`Ê^#vp'¡¨ Rc�¡Xc©¦(f}¬�\`g�a¿fk£kYP R§¨f$ÿ �YP�R�5f;YPU¯§Äc©p�¡�[VW�£}¬© 5§�¡Xf}U¯§}g9c©¡X§Éfkp'¡¨UXW`��\�[ÄY`§ÄWt¦`fkUxuv£wW( =p'¡¨f;Z�§xW�c©¡Mcd§É§x =a=¡¨U¯Y`£µ¡XfkZ�W`p5£wf`ÊoÑ n fp=W9Z9f(g=ÿ g=W`p�¡ n f0WP¡ n f}U n YPpRZ�g5YP�=�¿fkY`UX§Äc�p�¡ n UXf}f�£}¬© R§x¡¨fkUX§kgR§¨W�c�¡¯§�f}p'¡XU¨W(�'\×c�§É§¨ =a9¡XUXY(£µ¡¨f;Z¡�[Écd£wf`Êh Z9c á¿fkU¨fkp(¡�Õ¥c�«� R£ n c3YP�R�=U¨W]Ð9c�_�Yt¡¨c�W`p�ÀÇW`UV¡ n f�§XYP_�f�è' RYPp'¡¨c©¡�\�£}Y`p×a¿f�W`a9¡¯YPc�p=fkZ×[ n f}p�[Vf R§¨f�è( 5YPU¨¡¨fw¡¯§VWPÀÈp=W9Z9f;§ÉY`§�¡ n fTa5Y`§¨c�£¥£w¬� R§x¡¨fkUX§kÊAÑ n f�fkp'¡¨UXW`��\�c©pÓÍRe( =U¨fTz(YÅ[VW` =¬dZ�¡ n f}p�a¿fYP�R�=U¨W]Ð9c�_�Yt¡¨f;Z×Y(§}q ë�ó:ô ÁÇÿ ÃQ��ë ÁÇÿ��� î�ï ÃTY�ë ÁÇÿ� í ï ð ÃFUñë ÁÇÿ� ï à Á�zRQ î�fkU¨fTp=W9Z9f;§Äÿ�¥YPpRZ×ÿ ï Y`�=�5f;YPU�¡¨W(e`fw¡ n f}UÄc�pס�[VW�£w¬� R§�¡Xf}U¯§�§¨W�[�f�§¨ =a9¡¨U¯Y`£w¡�¡ n f}c�UÄf}p'¡¨UXW`��\(Êæ7c©e( =UXfTz(a�§ n Wt[�§ÉY�_�W`UXf¥e`fkp=f}UXc�£¥§¨c�¡X RYt¡Xc©W(p3Ê:#°ÀC[�f0Y`e(Y`c©p� R§¨f0è( 5YPU¨¡¨fw¡¯§ÄWPÀCp=W9Z9fk§ÉY(§Ä¡ n faRY(§xcd£¥£w¬� R§�¡Xf}U¯§Ä[�f n Y]¦(f`që ó ôõ� ë Á�ÿ �� îxï Ã�YBë Á�ÿ �í ï�ð ÃTYBë ÁÇÿhîxï�Ù ö ÃZYBë ÁÇÿ�ï�ð�ö ÷ ÃU*ë Á�ÿ ï Ã$Uñë ÁÇÿ�î�ï ÃFUñë ÁÇÿ�ï ð Ã$Upë ÁÇÿ�ï ö ÃYjë Á�ÿ�ï à Á�z'N î�fkU¨f0¡ n f0pRW�Z=fk§�ÿ ÿ�ïbYP�R�5f;YPUÉ¡XW`e`f}¡ n fkU�c�p�¡�[�W�£}¬© R§x¡¨fkUX§�§xW�[Vfb§¨ =a9¡XUXY(£µ¡É¡ n f}c�UMf}p'¡XU¨W(�'\W`p5£wfÅY`pRZ$§¨c©_�c�¬�Y`U¨¬�\�ÀÇW`U�zh�XyÓy!�Xr�YPpRZÆy!�¨­RÊTSV 9¡TW`pR£}fb[VfÅZ9Wס n YP¡T[Vfŧ¨f}fb¡ n YP¡TpRW�Z=fbÿ�ïYP�R�5f;YPU¯§Mc�piÀÇW( =U�£w¬� R§x¡¨f}U¯§¥YPpRZic�¡¯§¥f}p'¡¨UXW`��\�cd§T§¨ =a9¡XUXY(£µ¡XfkZ�ÀÇW( =UT¡¨c�_�fk§�ÁÇc�pÆÀÇW` =U�Z9c á¿fkU¨fkp'¡�RY`c©U¯§ÄWPÀ7p=W9Z9fk§ à §xW�[Vf¥p=f}f;ZסXW�Y`Z=ZÓ¡ n f�f}p'¡¨UXW`��\�WPÀCp=W9Z9f¥ÿ ï YPe(Y`c©pÈÊÑ n f0Õ¥c�«� R£ n cÈY]¦(f}U¯YPe`fTf}pRf}UXe`\�cd§ÉaRY(§xf;ZÓW(p�Y�§xc�_�c©¬dYPU�c�p'¡¨ =c©¡¨c�W`pÈÊ�¤�fwÍRp=f0¡ n f0fkp=f}UXe`\×WPÀAYUXf}e`c�W`pÓa�\¿q � Ô ÁÇÿ Ô Ã:A�U ¬©p � ����� ��� Á�ÿ � �xÿ � ÃFU ¬©p � ��� � ÁÇÿ � Ã:A�U ¬�p � Ô ÁÇÿ Ô Ã Á�z(j Ã

[ n fkU¨fÉ¡ n fT�=U¨W9Z9 5£µ¡X§VYPUXf�Wt¦(f}UKY`¬©¬¿p=W9Z9fk§KW`UV�RYPc�U¯§oWPÀ3pRW�Z=fk§A¡ n Yt¡ÉYPUXfM£}W`p'¡XY`c©p=f;Z�c©p×U¨fke`c�W`pÎ Ê h pRZ¹Z9f}p=W`¡¨f�a�\Ñø� ÁÇÿ Ô�ù 7 à ¡ n f$Y]¦`fkUXY`e`fÓWPÀT¡ n f�fkp=f}UXe`\/[Éc©¡ n UXfk§¨�¿fk£µ¡�¡XW!¡ n f�a¿f}¬�c©f}Àø� Á�ÿ Ô�ù 7 ÃN� ] +.ú 7 Ô Á�ÿ Ô Ã>� Ô ÁÇÿ Ô Ã ÊÉÑ n fkp�¡ n fÅY]¦`f}U¯YPe(f�fkp=f}UXe`\ÓWPÀCÍRe( =U¨f�z(a� R§xc�p=e×è' RY`Ux¡Xfw¡¯§

Page 16: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

WPÀCp=W9Z9f;§Äc�§kq ø� ó ôõ� ø� Á�ÿ �� îxï ÃTY ø� ÁÇÿ �í ï ð ÃTY ø� ÁÇÿ�î�ï�Ù ö ÃTY ø� ÁÇÿ�ï ð�ö�÷ ÃU ø� Á�ÿ ï ÃFU ø� Á�ÿhîxï ÃFU ø� Á�ÿ�ï�ð ÃFU ø� Á�ÿ�ï�ö ÃY ø� Á�ÿ ï à Á�z`z ÃÁÇ[ n f}UXfT[Vf n Y]¦(fT§¨ =�=�=UXfk§X§¨fkZס n f0Z9fk�5fkpRZ9fkpR£wf�W`À�ø� W`p�a Ãæ=W(UMY�e(f}p=fkUXY`¬3e`U¯YP� n g=¬©f}¡2û a5f�Y�§xf}¡MWPÀAUXf}e`c�W`p5§Ä¡ n YP¡Mc�pR£w¬� RZ9fb§¨W`_�fb£ n W(§¨f}p�aRY`§¨cd£�£w¬� R§�u¡¨fkUX§TW`À�p=W9Z9f;§}gΡ n f}c�U�c©p'¡Xf}U¯§xf;£µ¡¨c�W`p5§}gΡ n f�c©p'¡¨fkUX§¨fk£w¡¨c�W`pR§�WPÀ�¡ n f�c�p'¡¨f}U¯§¨fk£µ¡Xc©W(pR§kg3YPpRZƧxW�W`p3ÊÑ n f�£ n W(c�£}f�WPÀÄaRY`§¨cd£�£w¬� R§�¡Xf}U¯§0Z9f}¡¨f}UX_�c©pRfk§¥¡ n f×Õ¥c�«' 5£ n cKYP�R�=U¨W]Ð9c�_�Yt¡¨c�W`p!«(ÀÇW`U�¡ n f×SVfw¡ n fYP�R�=U¨W]Ð9c�_�Yt¡¨c�W`pÈg`¡ n f�aRY`§¨cd£�£}¬© R§x¡¨fkUX§Ä£}W`pR§¨c�§x¡ÉWPÀ<Y`¬©¬�¬�c�p=«`f;Z×�RY`c©U¯§VWPÀ7p=W9Z9fk§kÊ L fw¡Éÿ Ô a¿f¥¡ n f§x¡XYt¡Xf¥WPÀ7¡ n f�p=W9Z9f;§Äc©p�UXf}e(c©W(p Î YPpRZ 7 Ô Á�ÿ Ô Ã a5f¥¡ n f�äxa¿f}¬�c©f}À¨å�c�p�ÿ Ô ÊÑ n f0Õ¥c�«� R£ n cÎÀÇUXf}f¥f}p=fkU¨e(\�c�§R ó � )Ô\0�ü9ý Ô±þ ) +.ú 7 Ô Á�ÿ Ô Ã>� Ô ÁÇÿ Ô ÃTY ) +�ú 7 Ô ÁÇÿ Ô Ã ¬©W(e 7 Ô Á�ÿ Ô Ã6ÿ Á�z'y Ã

[ n fkU¨f ý Ô c�§7¡ n fÄWt¦`f}U¨uv£wW` Rp(¡Xc©pReÉp' R_ba¿f}UAWPÀ5U¨fke`c�W`p Î g(Z=fwÍRp=f;Zba�\Îq ý Ô � Q U ]�� 0 � ¥ ��� Ô 1 Ô 3 ý �[ n fkU¨f��\��  � Î Á ÎPà c�§b¡ n fÓ§¨fw¡�WPÀMYP¬�¬Ä§x =�¿f}U¨u°U¨fke`c�W`pR§�W`À Î ÊËæRW`UÅ¡ n fÓ¬�Y`U¨e(fk§x¡bU¨fke`c�W`pR§bc©p³ûÅgý Ô � Q`ÊÅÑ n f�a¿f}¬�c�fwÀ 7 Ô Á ¨ Ô Ã c�pËU¨fke`c�W`p Î×n Y`§�§¨f}¦(f}U¯YP¬C£wW`p5§�¡XUXY`c©p'¡X§kqMc©¡0_b 5§�¡0§¨ =_Ù¡¨W�W`pRfYPp5Z/a¿f�£}W`pR§¨c�§x¡¨fkp'¡�[Éc©¡ n ¡ n f�a¿f}¬�c©f}À�§�c©p)U¨fke`c�W`pR§Å[ n cd£ n c�p(¡Xf}U¯§xf;£µ¡�[Éc�¡ n³Î ÊB#vp)e`f}pRf}U¯YP¬�gc�pR£wUXfkY(§xc�p=e0¡ n f�§xc�Ò}f¥W`À3¡ n f¥aRY(§xcd£T£w¬� R§�¡Xf}U¯§Vc©_��=UXWt¦`f;§o¡ n f�YP�=�=UXW]Ð9c©_�YP¡¨c�W`p�W(p=f�W(a9¡XY`c©p5§�a�\_�c©pRc©_�c�Ò}c�p=e�¡ n f0Õ¥c�«� R£ n cÎÀÇUXf}f¥f}pRf}UXe`\`Ê� �¹ðÈßMð ¼=½ ñ ã ù�ð áý¸ ðÈñ ã ð3ò�ó ¼Rà ó ½Côo½7»=ã�à ß��Úï ���H c�p=c�_�c�Ò}c�p=e)¡ n f�Õ¥c�«� R£ n cTÀÇU¨fkfifkp=f}UXe`\�§¨ =a9Û�fk£w¡�¡¨W)¡ n f!£wW(pR§�¡XUXY`c©p'¡¯§×W(pÚ¡ n fÆa¿f}¬�c©f}À�§�c�§p=W`¡�§¨c�_��R¬©f(Ê � f;YPUX¬©\�Y`¬©¬�Y`�=�=¬�c�£kYt¡¨c�W`p5§ÅW`À�¡ n f�Õ¥c�«� R£ n cÉYP�R�=U¨W]Ð9c�_�Yt¡¨c�W`p�c©p)¡ n f�� n \9§¨c�£k§¬�c�¡Xf}U¯Yt¡¨ RU¨f�fwÐ9�=¬�W`c©¡b§¨\'_�_�fw¡XU¨c�fk§¥c�pË¡ n f× =pRZ9fkU¨¬�\�c©p=e�� n \9§¨c�£kYP¬o§¨\�§x¡¨fk_ Y`pRZÆ¡ n f�£ n W(c�£}f�WPÀ£w¬� R§x¡¨fkUX§¥¡XW�U¨f;Z9 R£}f�¡ n f�p� =_Åa5fkU0WPÀÄfkè' RYP¡¨c�W`pR§�¡ n YP¡bp=f}f;ZÆ¡¨W�a¿fקxW(¬©¦(fkZÆÀÇUXW`_���ÁGü à ¡XW��Á�Q à ÊoSV 9¡<Û� R§�¡ÉY`§K¡ n fTSVfw¡ n f�ÀÇUXf}f�f}p=fkU¨e(\�£kYPp�a¿fT_�c�p=c�_�c©ÒkfkZ�a�\�¡ n f�S m Y`¬©e(W`UXc�¡ n _�g`[Vfc�p(¡XU¨W9Z9 5£wfÉY¥£w¬dY`§X§<W`ÀÎY`pRYP¬�W`e(W` R§�� ~u��~ a GPE�H��.~���}x~\E�H=~ c���a b�� G � G { H b � ÁGØ¥S m�à Y`¬©e(W`UXc�¡ n _�§<¡ n YP¡_�c©pRc©_�c�Ò}f0YPp�Y`U¨a=c©¡¨U¯YPUX\�Õ¥c�«' 5£ n cÎÀÇU¨fkf�f}p=fkU¨e(\`ÊAÑ n f;§xf0Y`¬©e(W`UXc�¡ n _�§�UXf}�RU¨f;§xfkp(¡�Y`p�Y(Z9¦tYPpR£}fc�p�� n \�§¨cd£}§kg`c�p�¡ n YP¡o¡ n fk\ÅW(�5fkp�¡ n fM[VY]\b¡¨W�¡ n f�fwÐ9�=¬�W`c©¡XYP¡¨c�W`p�WPÀ3Õ¥c�«� R£ n cRY`�=�=UXW]Ð�c�_�Yt¡Xc©W(pR§ÀÇW`UÉc�p n W(_�W`e`fkp=f}W( R§V� n \9§xcd£}Y`¬3§x\9§x¡¨f}_�§kÊÑ n fkU¨f�YPUXf¥c©p�À�Y(£µ¡�_�YPp�\��¿W(§X§xc�a=¬�fbØ¥S m Y`¬©e(W`UXc�¡ n _�§V[ n c�£ n Y`¬©¬È£}W`UXU¨f;§x�¿W`p5Z�¡¨W�¡ n fb§XYP_�fÕ¥c�«' 5£ n cKYP�R�=U¨W]Ð9c�_�Yt¡¨c�W`pÈÊ Ï f��RU¨f;§xfkp(¡0Y¹äX£}YPpRW`p=cd£}Y`¬�å�Ø¥S m YP¬�e`W`UXc©¡ n _ [ n c�£ n!n Y(§¥¡ n fp=cd£wfÉ�=UXW`�¿f}U¨¡�\�WPÀÎUXfkZ9 R£}c©pRe�¡XW¥W`U¯Z9c©p5YPUX\0S m YP¡7¡ n f�SVfw¡ n fɬ©fk¦`fk¬GÊ Ï fÉc�p'¡¨UXW9Z9 R£wfÄ_�fk§X§XYPe`f;§% Ô � Á�ÿ � à a¿fw¡�[Vf}fkp�YP¬�¬MUXf}e`c�W`p5§ Î Y`pRZ)¡ n fkc©U)ä¨Z9c�U¨f;£µ¡�§¨ =a9u°U¨fke`c�W`pR§Xå��(Ê Á�¤�fwÍRpRf�¡ n fƧxf}¡��� 7�� Á Îtà W`ÀbZ9c©UXfk£w¡�§¨ =a9u°U¨fke`c�W`pR§×WPÀ Î ¡XW)a¿f$¡ n W(§¨fiUXf}e(c©W(pR§�¡ n Yt¡�YPUXfi§¨ =a9u°U¨fke`c�W`pR§×WPÀ Îa= 9¡ n Y]¦(fÅpRW�§¨ =�¿f}U¨u�UXf}e(c©W(pR§É¡ n Yt¡0YPUXf�YP¬d§xW�§x =a=u�UXf}e(c©W(pR§MW`À Î g<Y`pRZi§¨c©_�c�¬�Y`U¨¬�\�ÀÇW`UT¡ n f�§xf}¡����  � Î � Á ÎPà W`À�äXZ9c�U¨f;£µ¡�§x R�5fkUxu°U¨fke`c�W`pR§kÊ å à #°¡�cd§ n fk¬©�9ÀÇ R¬�¡¨WË¡ n c©pR«)W`ÀM¡ n c�§�Y(§�Y!_�fk§X§¨Y`e`fÀÇUXW`_ ¡ n W'§xf×p=W9Z9fk§Åc©p Î a= 9¡�p=WP¡�c©p��ËÁ�[ n c�£ n [�fÓZ9f}p=W`¡¨fÓa'\ Î�8 � à ¡¨W�¡ n f�p=W9Z9fk§Åc©p��`Ê#vp'¡¨ =c©¡¨c�¦`fk¬©\(g�[�f�[ÄYPp'¡V_�f;§¨§XYPe(fk§C¡XWb�=UXW`�5YPe(YP¡¨fÉc�p9ÀÇW`UX_�Yt¡Xc©W(p�¡ n YP¡�¬�c©f;§KW( 9¡X§¨c�Z=fMWPÀÈY0U¨fke`c�W`pc�p(¡XW�c©¡kÊ�Ñ n  R§kg=ÀÇW`UTY�e`c�¦`fkp�U¨fke`c�W`p Î g5[Vf0[ÄYPp'¡É¡ n fÅa5fk¬©c�fwÀ 7 Ô ÁÇÿ Ô Ã ¡XW×Z9fk�5fkpRZ�W`p�f}Ð=Y`£µ¡X¬©\¡ n W(§¨f�_�fk§X§¨Y`e`fk§ % Ô�� � � ¡ n YP¡V§x¡XYPU¨¡oW` =¡X§¨c�Z9fMWPÀ�¡ n f�U¨fke`c�W`p Î YPp5Z�e`W�c�p(¡XW�¡ n fMUXf}e`c�W`p Î Ê Ï fZ9f}ÍRp=fÅ¡ n cd§�§xf}¡TW`ÀK_�fk§X§¨Y`e`f;§! Á ÎPà ¡¨W�a¿fb¡ n W(§¨fÅ_�f;§¨§XYPe(fk§ % Ô � � � ÁÇÿ � � à §x 5£ n ¡ n Yt¡¥U¨fke`c�W`pÎ#",8 � "Rn Y`§opRWbp=W9Z9f;§oc�p×£wW(_�_�W(p�[Éc�¡ n UXf}e`c�W`p Î g=YPp5Z�§x R£ n ¡ n YP¡�UXf}e`c�W`p$� " c�§�Yb§¨ =a9u°U¨fke`c�W`p

Page 17: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

WPÀ Î W`U�¡ n f�§XYP_�fÅY(§TU¨fke`c�W`p Î Ê Ï f�YP¬d§xW�Z9fwÍRpRfb¡ n f�§xf}¡% Á Î ��� à W`ÀK_�fk§X§¨Y`e`f;§�¡XWÓa¿f�Y`¬©¬¡ n W(§¨fÓ_�fk§X§XYPe`f;§�¡ n Yt¡�§�¡¯YPU¨¡�c�p)YƧx Ra9u�UXf}e(c©W(p/WPÀ Î YPpRZ)YP¬d§xWÆa5fk¬©W(p=ei¡XW� Á&� à g�Y`pRZ�[VfZ9f}ÍRp=f' Á ÎPÃ�8 Á(� à ¡XW�a5f¥¡ n W(§¨f¥_�f;§¨§XYPe(fk§�¡ n Yt¡MYPUXfTc©p) Á ÎPà aR 9¡�p=WP¡Éc�p) Á(� à ÊÑ n f0£kYPp=W(p=c�£kYP¬Îe`fkp=f}U¯YP¬�c�Ò}fkZ×a5fk¬©c�fwÀA�=UXW`�5YPe(YP¡¨c�W`p� R�¿Z=YP¡¨f�UX =¬�fk§ÉYPUXf`q

% Ô � & ¨ *+ )+ ú-,/. � Ô 4 � Á�ÿ Ô 4 � à �0 ú � � . � � 0 ß 1 Ô 3=4xß 1 � 3 % Ô � � � � �213 » �0 ú � . � 0 ß 1 Ô V � 3 % Ô � � � Á�z(r Ã7 Ô & ¨ � Ô ÁÇÿ Ô Ã �0 ú � . � 0 ß 1 Ô 3 % Ô � � � Á�z�s Ã

[ n fkU¨f�ÀÇW`UVa=UXf}¦�c�¡�\�[Vf n Y]¦`f�§¨ =�=�RU¨f;§¨§¨fkZ�¡ n f�ÀÇ =pR£w¡¨c�W`pRY`¬3Z9f}�¿f}pRZ=f}pR£}fk§VWPÀ3¡ n f¥a5fk¬©c�fwÀ�§ÄY`pRZ_�fk§X§¨Y`e`f;§}ÊMÑ n fb_�fk§X§¨Y`e`f;§�YPUXfb =�ÎZ=YP¡¨fkZi§�¡¯YPU¨¡¨c�p=e×[Éc©¡ n ¡ n f�_�f;§¨§XYPe(fk§Éc�p'¡¨Wס n f�§x_�YP¬�¬�fk§x¡UXf}e`c�W`p5§ÉÍRU¯§�¡;Ê ç p=fÅ£}YPp$¡ n fkpi R§¨f0¡ n fbp=fk[ɬ©\�£}W`_��= 9¡XfkZ$_�f;§¨§XYPe(fk§Éc�p�¡ n fÅ�=UXW9Z9 R£w¡�Wt¦`fkU Á Î ��� à WPÀÈ¡ n f�_�fk§X§¨Y`e`f}u� =�ÎZ=YP¡¨f¥U¨ R¬©f(Ê I _��=c©UXcd£}YP¬�¬�\`g�¡ n c�§ n f}¬��R§É£wW(p'¦(f}UXe`fkpR£wf(ÊDFE GH@I ¼ L L f}¡�M % Ô � Á�ÿ � à O�a5f�Y�§xf}¡0WPÀÄ£}YPpRW`p=cd£}Y`¬AØ¥S m _�fk§X§¨Y`e`f;§TY`pRZƬ©f}¡�M 7 Ô Á�ÿ Ô Ã O�a¿f¡ n f$a5fk¬©c�fwÀ�§Ó£}Y`¬�£} =¬�YP¡¨f;Z�ÀÇUXW`_ ¡ n W(§¨f�_�f;§¨§XYPe(fk§kÊÂÑ n fkp¹¡ n f$a¿f}¬�c�fwÀ�§×Y`U¨f�ÍRÐ�f;Z�u°�5W(c©p'¡X§�WPÀ¡ n f�£kYPp=W(p=cd£}YP¬ÉØ¥S m Y`¬©e(W`UXc�¡ n _ c©ÀTYPp5Z/W`pR¬©\�c©ÀM¡ n f}\�Y`U¨fÓÒ}fkU¨WËe`U¯Y`Z9c�f}p'¡��5W(c©p'¡X§�W`ÀM¡ n f£wW(pR§x¡¨U¯YPc�p=fkZÓÕ¥c©«� R£ n c3ÀÇU¨fkfTfkp=f}UXe`\�R ó ÊÏ f$�=UXWt¦`fÓ¡ n cd§×£}¬�Y`c©_ a'\ìY`Z=Z=c©p=e L YPe`U¯YPpRe`f�_b =¬©¡¨c��=¬�c©fkUX§kq i Ô ¡¨W�f}p=ÀÇW`U¯£wf�¡ n f�p=W`UX_�YP¬©uc�ÒkYt¡Xc©W(p)W`À 7 Ô Y`pRZ_f Ô � ÁÇÿ � à ¡¨W�f}p=ÀÇW`U¯£wfÓ¡ n f$£}W`pR§¨c�§x¡¨fkpR£w\�W`ÀTf;Y`£ n U¨fke`c�W`p Î [Éc�¡ n YP¬�¬ÉWPÀc©¡X§0Z9c�UXfk£µ¡0§¨ =a9u°U¨fke`c�W`pR§%�(Ê�Ñ n cd§�§xf}¡0WPÀV£}W`pR§¨cd§�¡Xf}pR£}\i£wW(pR§�¡XUXY`c©p'¡¯§�cd§�Y`£w¡¨ RY`¬©¬�\$_�W`UXf�¡ n Y`p§¨ !â�£wc�f}p'¡kgÄa= 9¡�¡ n f}UXf�cd§�p=W n Y`U¨_ c�p�Y`Z=Z=c©p=e�f}Ð�¡¨U¯YË£}W`pR§x¡¨U¯YPc�p'¡X§kÊ Ï f�¡ n f}p¹UXWP¡¯Yt¡¨f�¡XWYPpRWP¡ n f}U�§¨fw¡VWPÀ L YPe`U¯YPpRe`fÉ_b R¬�¡Xc©�=¬�c�f}U¯§54 Ô � ÁÇÿ � à W`À3fkè' RY`¬¿Z9c�_�f}pR§¨c�W`pRY`¬©c©¡�\Å[ n c�£ n fkp9ÀÇW`U¯£wf�Y¬�c©p=f;YPUK£}W`_Åa=c©p5Yt¡¨c�W`p�W`À¿¡ n fMW(U¨c�e`c�pRY`¬=£wW`p5§�¡XUXY`c©p'¡X§kq64 Ô � ÁÇÿ � à fkp9ÀÇW`U¯£wf;§oYP¬�¬=¡ n W(§¨f�£wW(pR§�¡XUXY`c©p'¡¯§c�p'¦(W`¬�¦�c©p=eÓ_�YPUXe`c�pRYP¬�c©Ò;Yt¡Xc©W(pR§Ma�\$Y`¬©¬oZ9c�UXfk£µ¡0§¨ =�¿f}U¨u�UXf}e(c©W(pR§ Î#" W`À7��c�p'¡¨W8��fwÐ=£wfk�9¡¥¡ n YP¡�WPÀUXf}e`c�W`p Î c©¡X§¨f}¬©À�Ê�Ñ n f×UXWP¡¯Yt¡Xc©W(p�_�YP¡¨UXc�Ð�cd§bc�p)Yia=¬�W�£¯«ËÀÇW(U¨_ë[ n cd£ n £}Y`p/a¿fÓe( RYPU¯YPp'¡¨fkfkZ¡¨W!a¿f�ÀÇ =¬�¬MU¯YPp=«ÎÊ Ï f$£kYPp)¡ n fkp¹§ n Wt[Ü¡ n YP¡�¡ n f84 Ô � ÁÇÿ � à £wW(pR§�¡XUXY`c©p'¡¯§�£kYPpìa5f�[ÉUXc©¡x¡¨fkpc�p�¡ n fbÀÇW`UX_94 Ô � ÁÇÿ � à ] Ô:�,0�ü 1�; ú/. 3 ý Ô � ] + ú � 7 ÁÇÿ "Ô Ã [ n f}UXf�û�Á<4 Ô � à cd§�¡ n fŧ¨fw¡MW`ÀoYP¬�¬<UXf}e(c©W(pR§[ n cd£ n U¨f;£wf}c�¦`f�¡ n f�_�f;§¨§XYPe(f!4 Ô � c©p�¡ n f�a5fk¬©c�fwÀC =�ÎZ=Yt¡Xf�U¨ =¬�f¥WPÀ7¡ n f0£kYPp=W(p=cd£}YP¬�Y`¬©e(W`UXc�¡ n _�ÊÏ f�¡ n fkp�U¨f}u°Y`U¨U¯YPp=e(f�¡ n f$§x =_ Wt¦`fkU�Y`¬©¬=4AÔ §�c�p'¡¨W�Y!§x R_ Wt¦`fkU�YP¬�¬�U¨fke`c�W`pR§kgK[ n c�£ n¹n Y(§¡ n f0ÀÇW`UX_ ] Ô\0�ü ý Ô ] +.ú 7 Ô Á�ÿ Ô Ã ] ; ú>. 0@?ß 1 Ô 3 4 Ô � ÁÇÿ � à Ê�Á=A Á ÎPà c�§TY�§¨fw¡TWPÀB4 Ô:� � � c�p$W(p=fwu�¡¨W`uW`pRfÅ£wW(U¨UXfk§¨�¿W`pRZ9fkpR£wf�[Éc©¡ n ¡ n f % Ô�� � � c�pC Á ÎPÃ Ê Ã æ<c�pRY`¬©¬�\`gÎ[VfbZ9c á¿fkU¨fkp(¡Xc�YP¡¨fb¡ n f�Õ¥c©«� R£ n cÀÇUXf}f�f}p=fkU¨e(\�[Éc�¡ n UXfk§¨�5f;£µ¡T¡¨W 7 Ô Á ÎPà g<Y`pRZÆc�Z9fkp'¡¨c©ÀÇ\)4 Ô � Á�ÿ � Ãj� ¬©p % Ô � ÁÇÿ � à ¡XWÓW(a9¡XY`c©pÆ¡ n f£}Y`p=W`pRc�£kYP¬VØ¥S m a5fk¬©c�fwÀ� =�¿ZRYt¡¨f�UX =¬©f;§}g I è¿Ê<z9s�Ê ª §¨c©pRe$¡ n fÓa¿f}¬�c©f}À� =�ÎZ=YP¡¨f�U¨ =¬�fk§Åc�p/¡ n f_�YPUXe`c�pRYP¬�c�ÒkYt¡Xc©W(p/£wW(pR§�¡XUXY`c©p'¡¯§}gC[Vf×W`a=¡XYPc�p/¡ n f�£}YPpRW`p=cd£}Y`¬VØ¥S m _�fk§X§XYPe`f× =�¿ZRYt¡¨f�UX =¬�fk§kgI è¿Ê9z'r=ÊI è' RYP¡¨c�W`p�z'r�_�Y]\�Y`�=�¿fkYPUi¡¨W%a5f¹£wW(_��R¬©cd£}YP¡¨fkZ3gba= 9¡!c©¡Ë£}Y`p�a¿f짨c©_��=¬�\�Z9f}UXc�¦`fkZ�a�\_�YPUXe`c�pRYP¬�c�Ò}c�p=e$fkè' RYP¡¨c�W`p/z9s�gA[ n cd£ n cd§ÅU¨f;YP¬�¬©\Ë¡ n fÓ«(f}\!e`fkp=f}U¯YP¬�c©ÒkfkZ�a¿f}¬�c�fwÀM�=UXW`�5YPe(YP¡¨c�W`pfkè' RYP¡¨c�W`p3Ê�#vp'¡¨ =c©¡¨c�¦`fk¬©\(g3fkè' RYP¡¨c�W`pËz�sקxc�_��=¬�\i§¨Y]\9§�¡ n Yt¡¥¡ n f�a¿f}¬�c�fwÀÄc©p!Y�U¨fke`c�W`pÆZ=f}�¿f}pRZ=§W`pi¡ n f��=UXW9Z9 R£µ¡¥WPÀKY`¬©¬7¡ n f�£}W`_��RYP¡¨c�a=c©¬�c©¡�\�_�YP¡¨UXc�£}fk§TYPpRZ$f}¦�c�Z=f}pR£}fb¡ n Yt¡�YPUXfbc�p'¡¨f}UXpRY`¬È¡XW¡ n f0U¨fke`c�W`p�YPpRZ�Y`¬©¬�¡ n f0_�fk§X§XYPe`f;§�¡ n YP¡MW`UXc©e(c©p5Yt¡¨f¥W( 9¡X§¨c�Z=f�WPÀ7¡ n f0UXf}e(c©W(p�Y`pRZ�f}pRZ�c�pR§¨c�Z=fc©¡kÊ#°¡ocd§o£w¬�fkY`UCÀÇU¨W(_�¡ n c�§A�RU¨W�WPÀ¿W` 9¡X¬©c�p=fÄ¡ n Yt¡oW`¡ n fkU�Ø¥S m _�f;§¨§XYPe(fV�5Y`§X§xc�p=e�YP¬�e`W(U¨c©¡ n _�§7[ n cd£ nYPUXf�f;è( Rc©¦tYP¬�f}p'¡�¡¨Wi¡ n f�Õ¥c©«� R£ n cVY`�=�=UXW]Ð�c�_�Yt¡Xc©W(pËf}Ð9c�§x¡kÊ�#°ÀMW`pRf×[ÉUXc�¡Xfk§�YPp�\�§xf}¡ÅWPÀT£wW`p=u§x¡¨U¯YPc�p(¡¯§A[ n cd£ n YPUXf�§x !â�£}c©fkp(¡K¡¨W�c�pR§x RU¨fÄ¡ n fM£wW(pR§¨c�§x¡¨fkpR£w\ÅWPÀ3YP¬�¬RÕ¥c�«' 5£ n c=UXf}e`c�W`p5§}gPW(p=f�£}Y`pY`§X§¨W�£}c�YP¡¨fT¡ n f�fwÐ9�5W(p=f}p'¡Xc�YP¡¨fkZ L YPe`U¯YPpRe`fT_b =¬©¡¨c��=¬�c©fkUX§ÉW`À<¡ n W(§¨f0£wW(pR§x¡¨U¯YPc�p(¡¯§Ä[Éc�¡ n Y�§xf}¡�WPÀ_�fk§X§¨Y`e`f;§}Ê

Page 18: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

1 2 3

4 5 6

7 8 9

Á�Y Ã

36

1245

7878 9

Á�a Ã

2356

1245

5689

2356

478

ÁG£ Ã

2356

1245

5689

2356

4578

ÁGZ Ãæ7c©e( =UXfÅy9q 6�@bh §xc�_��=¬�fbe`U¯YP� n Ê �ED �FD�E Ñ n UXf}fÅ[VY]\9§ÉW`ÀK£w¬� R§x¡¨f}UXc�p=e�¡ n fbe`U¯YP� n Ê J  =pRp=c©pReW`U¯Z9c�pRYPUX\ÅS m W(p�¡ n fMÍRUX§x¡o¡�[VWb£w¬� R§x¡¨fkU¨c�p=e(§MÁÇa3g £ à c�§Kfkè' =c�¦]Y`¬©fkp'¡o¡¨W�Y0Õ¥c©«� R£ n cRYP�=�RU¨W]Ð9c�_�YPu¡¨c�W`pÈg=a= 9¡�p=W`¡ÉÀÇW`UÄ¡ n f¥¡ n c�U¯ZÓ£}¬© R§x¡¨fkU¨c�p=e�Á�Z Ã

G HËñ â�º`» ð ¼ ð á ï!ðÈñ ã ð3òI� ¼=à ó ½Côo½7»=ã�à ß#vp�¡ n f�Ø¥S m YP¬�e`W(U¨c©¡ n _�g`¡ n fT_�fk§X§¨Y`e`fk§o¡XY`«`f�¡ n f�ÀÇW`UX_-W`À3�=UXW`aRY`a=c©¬�c©¡�\�Z=c�§x¡¨UXc©aR 9¡¨c�W`pR§�Wt¦`fkU£w¬� R§x¡¨fkUX§AW`À�p=W9Z9fk§kg([ n f}UXfkY`§Cc�p�W(UXZ9c�pRY`U¨\bS m ¡ n fM_�fk§X§XYPe`f;§7¡XY`«`fÄ¡ n f�ÀÇW`UX_�W`À��=U¨W(aRYPaRc©¬�c�¡�\Z9cd§�¡XU¨c�a= 9¡Xc©W(pR§AWt¦(f}UA§xc�p=e(¬©fÄp=W9Z9f;§}Ê h Z9c á¿fkU¨fkp(¡o[ÄY]\0W`ÀÎZ9fkU¨c�¦�c©pRe¥Y¥S m YP¬�e`W(U¨c©¡ n _�c©p�[ n cd£ n_�fk§X§¨Y`e`f;§MYPUXf�Z9c�§x¡¨UXc�a= 9¡¨c�W`p5§�Wt¦(f}UT£w¬� R§�¡Xf}U¥WPÀ�p=W9Z9fk§Tcd§�c�¬�¬© R§x¡¨U¯Yt¡XfkZic©piÍRe` =UXf�y9Ê Ï fbÍRU¯§x¡e`UXW` R�0£w¬� R§�¡Xf}U<WPÀ9p=W9Z9fk§<c�p(¡XW�䨧¨ =�¿f}U¨u�pRW�Z=fk§XåÉYPpRZ¥¡ n f}pbU¨ =p�W(UXZ=c©pRY`U¨\TS m W`p¥¡ n f�£}¬© R§x¡¨fkU¨f;Ze`U¯YP� n Ê Ï�n fkp�¡ n fb£}¬© 5§�¡Xf}U¯§�YPUXf¥¬�Y`U¨e(fTfkp=W` Re n §xW�¡ n Yt¡�¡ n fb£}¬© R§x¡¨fkU�e`U¯YP� n cd§�Y�¡¨UXf}f(g=¡ n c�§cd§AÛ� R§x¡ m f;YPUX¬GÔ §Ä_�fw¡ n W�Z$WPÀo£w¬� R§x¡¨f}UXc�p=e�ÀÇW`UMfwÐ=Y`£w¡Mc©p=ÀÇf}UXf}pR£}f0c©p$Y`p�\Óe(UXY`� n cd£}YP¬�_�W9Z9fk¬ é Qkj]êÁÇf;è' =c©¦tY`¬©fkp(¡É¡XW�¡ n f�Û� =pR£µ¡Xc©W(p�¡¨UXf}f0Y`¬©e(W`UXc�¡ n _ Ã Ê Ï�n fkp�¡ n fb£}¬© 5§�¡Xf}U�e(UXY`� n §x¡¨c�¬�¬ n Y`§�¬©W�W`�5§c�pÅc©¡kg n Wt[�fk¦`fkUkg;¡ n fÉa5fk¬©c�fwÀ�§AY`U¨fVp=WP¡of}Ð=Y`£µ¡;Ê<®MWt[)YPUXf�¡ n f;§xfÄa¿f}¬�c©f}À�§AU¨fk¬�YP¡¨f;Z�¡¨WT¡ n f�Õ¥c©«� R£ n cYP�R�=U¨W]Ð9c�_�Yt¡¨c�W`p��Ñ n fħxc�_��=¬�fk§x¡7£}Y(§xfK¡¨WTYPpRY`¬©\�Ò}f�c�§<[ n fkp0¡ n fV£}¬© R§x¡¨fkUX§7YPUXfop=W(p9u°Wt¦`f}UX¬dYP�=�=c�p=e�Y(§Èc�p0ÍRe( =UXf�yta3ÊD ba b E@E G a|°$J L L f}¡LKNM�a5fTY�e`U¯YP� n W(a9¡XY`c©pRfkZ�a�\�£w¬� R§�¡Xf}UXc©pRe�p=W9Z9fk§�c©p�Y0e`U¯YP� n K�Ê h §X§¨ =_�f¡ n fÉ�5W`¡¨f}p'¡Xc�Y`¬�§Ac�p$K M Y`U¨fɧ¨fw¡K§¨W¥¡ n YP¡A¡ n fCÛ�W(c©p'¡�Z=c�§x¡¨UXc©aR 9¡¨c�W`p�Z9f}ÍRp=fkZ�a�\@K M c�§ofkè' =c�¦]Y`¬©fkp'¡¡¨Wb¡ n YP¡VZ9f}ÍRp=f;Z�a�\IK�Ê$#°ÀÈ¡ n fT£w¬� R§�¡Xf}U¯§�Y`U¨fMp=W`p9u°Wt¦`fkU¨¬dYP�R�=c©pRe�¡ n f}p�¡ n fTa5fk¬©c�fwÀ�§Ä£kYP¬d£w =¬dYt¡XfkZa�\�U¨ =pRp=c©pReÅS m W(pOKNMÄYPUXfÉ¡ n fT§XYP_�f�Y(§o¡ n W(§¨f�£kYP¬d£w =¬dYt¡XfkZ�a�\�YbÕ¥c�«� R£ n cÎYP�=�RU¨W]Ð9c�_�YP¡¨c�W`pW`p�K [ n f}UXfT¡ n f�UXf}e(c©W(pR§ÄYPUXf¥Y`¬©¬��5YPc�UX§ÄW`À<p=W9Z9f;§Éc©p8KNMkÊÑ n cd§�£wW(U¨W(¬©¬dYPUX\�ÀÇW`¬�¬©Wt[�§ÄZ9c�UXfk£µ¡X¬©\×ÀÇU¨W(_Ü£w¬dYPc�_ÖQ(ÊÏ�n fkpÅ¡ n f�£w¬� R§x¡¨fkUX§AY`U¨fVWt¦`fkU¨¬dYP�=�Rc©p=e5g]¡ n f�§¨c©¡¨ RYP¡¨c�W`p�c�§C_�W`UXfÉ£wW(_��R¬©cd£}YP¡¨fkZ3Ê Ï�n fkp�[�fÄUX =pS m W`p�¡ n f0e`U¯YP� n c�p�ÍRe( =U¨fby`£¥[�f¥Í5pRZ�¡ n YP¡M¡ n f0a¿f}¬�c©f}À�§MYPUXf�c�pRZ9f}f;Z�¡ n fŧ¨Y`_�f0Y(§Ä¡ n W(§¨f

Page 19: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

£}Y`¬�£} =¬dYt¡¨f;Z� 5§xc�p=eÆ¡ n f$Õ¥c�«� R£ n cÉYP�=�RU¨W]Ð9c�_�YP¡¨c�W`p/[Éc©¡ n ¡ n fk§¨f�£w¬� R§�¡Xf}U¯§�Y`§�aRY(§xcd£�U¨fke`c�W`pR§kÊÏ�n fkp/[�fÓU¨ Rp�S m W`p�¡ n f×e(UXY`� n c�p�ÍRe` RU¨fÓy`Z�g n Wt[Vf}¦(f}U;gÈ[Vf×W`a=¡XYPc�p�a¿f}¬�c©f}À�§b¡ n Yt¡�YPUXfè' =c©¡¨f0Z9c á¿fkU¨fkp'¡ÉÀÇU¨W(_í¡ n f0Õ¥c�«� R£ n c�a¿f}¬�c©f}À�§}Êç p=f×Z=c áÎf}UXf}pR£}f�a¿fw¡�[Vf}f}p!¡ n fÓ£}¬© 5§�¡Xf}U¯§0c�p!ÍRe` =UXfÓy`£�YPpRZË¡ n fk§¨f�c�p�ÍRe( =U¨f×y`Z n Y`§0¡¨WiZ=W[Éc©¡ n ¡ n fTWt¦`fkUX£}W` =p'¡¨c�p=e0p� =_Åa5fkUX§ ý Ô Ê$#vpÓÍRe( =U¨f¥yP£`g(¡ n f¥Õ¥c©«� R£ n cÎfkp(¡XU¨W(��\�cd§VWPÀ3¡ n f�ÀÇW`UX_�që � ë ÁÇÿ­�> îxï ÃTY�ë Á�ÿ��í ï�ð ÃZY�ë Á�ÿ ï�ð ö�÷ Ã�YBë Á�ÿ îxÙ ö à Á�z(­ ÃU*ë ÁÇÿ� ï ÃFUñë ÁÇÿ ï ð ÃFUñë ÁÇÿ ö ÃFUñë ÁÇÿ î à Á�z(l á n  R§×fwÐ=£wfk�9¡�ÀÇW(U�¡ n fÆaRY`§¨cd£�£w¬� R§�¡Xf}U¯§}gÄp=W�WP¡ n f}U�UXf}e(c©W(p n Y`§��¿W(§¨c�¡Xc©¦(fiZ9W( =a=¬�fi£wW( =p'¡¨c�p=ep� =_ba¿f}U;Ê�Ñ n cd§��=UXW`�¿f}U¨¡�\�c�§�p=W`¡×§ n YPUXfkZ/a�\�¡ n f$£}¬© R§x¡¨fkUX§�c�pìÍRe` =UXf$yPZ�g n Wt[Vf}¦(f}U;Ê h §fkè' RYP¡¨c�W`p�z�N�§ n Wt[�§}gV¡ n fiUXf}e`c�W`p�£}W`UXU¨f;§x�¿W`pRZ=c©p=e!¡XW�¡ n fƧ¨c©p=e(¬©fip=W9Z9fiÿ�ï n Y(§��¿W(§¨c�¡Xc©¦(fZ9W( =a=¬�f0£wW` Rp(¡Xc©pRe�p� =_Åa5fkUkÊDFE GH@I È L L fw¡:û¹a¿fÄY¥§¨fw¡CWPÀÎÕ¥c©«� R£ n c�UXf}e(c©W(pR§�Á�aRY`§¨cd£�£}¬© R§x¡¨fkUX§kgtc�p(¡Xf}U¯§xf;£µ¡Xc©W(pR§}gtc�p'¡¨fkUX§¨fk£w¡¨c�W`pWPÀ3c�p'¡¨fkUX§¨fk£w¡¨c�W`pR§kg(f}¡X£`Ê Ã [Éc©¡ n Z9W( =a=¬�fT£}W` =p'¡Xc©p=ebp' R_ba¿f}U¯§ ý Ô ÊF#°À ý ÔQP ObW`p=¬�\�ÀÇW`U�¡ n fTaRY(§xcd££w¬� R§x¡¨fkUX§kg¿¡ n fkp$¡ n fba¿f}¬�c©f}À�§T£kYP¬d£w =¬dYt¡XfkZ� 5§xc�p=e�¡ n f�Õ¥c©«� R£ n cCYP�R�=U¨W]Ð9c�_�Yt¡¨c�W`p$Y`U¨fbc�Z9fkp'¡¨cd£}YP¬¡¨W�¡ n W(§¨f�¡ n YP¡¥[VW` R¬�Zia5f�£kYP¬d£w =¬dYt¡XfkZ$ 5§xc�p=e�W`U¯Z9c�pRYPUX\�S m W(pÆY�e`U¯YP� n KNM¥[ n W'§xf�p=W9Z9fk§YPUXfT¡ n f�a5Y`§¨c�£¥£w¬� R§x¡¨fkUX§kÊ`Na b.b>c L Ñ<WM£}W`pR§x¡¨UX R£µ¡È¡ n f�e`U¯YP� n KNM<[VfKY`Z=Z0fkZ9e(fk§3a¿fw¡�[Vf}fkp¥¡�[�WMU¨fke`c�W`pR§ Î �n� Î VZ9f}�¿f}p5Z9c©pReW`p�¡ n f�Z9W` Ra=¬©f�£}W` =p'¡Xc©p=e¥p� =_Åa5fkUoWPÀ5¡ n fkc©Uoc�p'¡¨fkUX§¨fk£w¡¨c�W`p ý � ÀÇW`U5� �oÎ �FR Î 'Ê�#°À ý � �èU QÄ[Vf§¨c©_��=¬�\$£wW(p=p=fk£w¡�¡ n f�£wW`UXUXfk§¨�5W(pRZ9c�p=e�¡�[VW×UXf}e`c�W`p5§}Êj#°À ý �@S U QÓÁÇc�Ê f`Ê¥¡ n f}UXf�Y`U¨fÅ_b =¬©¡¨c��=¬�f�RY`c©U¯§�WPÀ¥U¨fke`c�W`pR§�¡ n Yt¡ n Y]¦`f)�$Y(§�¡ n fkc©U×c©p'¡¨fkUX§¨fk£w¡¨c�W`p à [Vf$£ n W�W(§¨fp� ý � �A§x 5£ n �5YPc�UX§�Y`pRZ£wW(p=p=f;£µ¡Ä¡ n fk_�Ê Ï f�£ n W�W(§¨f�¡ n f¥fkZ=e`fk§É§¨WÅ¡ n YP¡ÉY`¬©¬�UXf}e(c©W(pR§�[ n W(§¨f�c�p'¡¨fkUX§¨fk£w¡¨c�W`p�c�§T�TÀÇW(U¨_Y�£}W`p=pRfk£µ¡XfkZ¹e(UXY`� n Ê�Ñ n fË£wW(_��5Yt¡XY`a=c�¬©c©¡�\)_�YP¡¨UXc�£}fk§8U Ô:V V Ô-W ÁÇÿ Ô�V �xÿ Ô-W à YPUXf ¯ ÀÇ =pR£w¡¨c�W`pR§¡ n Yt¡�U¨f;è( Rc©UXf$ÿ Ô�V YPp5Z�ÿ Ô-W ¡¨WìYPe`UXf}fiW`p�¡ n fƦtY`¬© =fÆWPÀ0ÿ � Ê-SVf;£}YP 5§xf$W`À0¡ n c�§×ÀÇW`UX_ WPÀ¡ n f UÖÀÇ =pR£w¡¨c�W`pR§kgÄ¡ n fÆ_�fk§X§¨Y`e`f�ÀÇUXW`_ Î � ¡XW Î Z9f}�¿f}p5Z=§�W(p=¬©\ìW`p¹¡ n fi¦tY`¬© =fiWPÀ¥¡ n fkc©Uc�p(¡Xf}U¯§xf;£µ¡Xc©W(pËÿ � Ê Ï f�p=Wt[ Y`Z=Z L YPe`U¯YPpRe`f�_b =¬©¡¨c��=¬�c©fkUX§kÊÅ#°À Î � � Î YPUXf�£wW(p=p=fk£w¡¨f;ZÆc�p!¡ n fe`U¯YP� n [�fTY`Z=Z�¡�[VW0¬dYPe(UXY`p=e`fÉ_Å =¬�¡Xc©�R¬©c�f}U¯§Nf Ô:V V Ô-W Á&� à ��f Ô-W V Ô:V ÁÇÿ � à ÊAÑ n f�_b =¬©¡¨c��=¬�c©fkU*f Ô:V V Ô-W Á&� Ãf}p=ÀÇW`U¯£wfk§0¡ n f�£wW(pR§x¡¨U¯YPc�p(¡b¡ n YP¡ 7 Á�ÿ Ô-W à _�YPUXe`c�pRY`¬©c�Ò}f×Z=Wt[Ép!¡¨W 7 ÁÇÿ � à Êñ#°À�[�fÓp=Wt[öZ9f}ÍRp=f`gf Ô-W V Ô�V ÁÇÿ � Ã:� ¬�p % Ô-W V Ô:V ÁÇÿ � à [�fÄÍRp5Zb¡ n Yt¡C¡ n f�§�¡¯Yt¡Xc©W(pRYPUXc�¡�\Å£wW`p5Z9c�¡Xc©W(pR§CW`pÅ¡ n fÄÀÇU¨fkfVfkp=f}UXe`\YPp5Z�¡ n f¥Í=Ð9fkZ��¿W`c�p(¡�f;è' RYt¡Xc©W(pR§VÀÇW`UÄ¡ n f�_�fk§X§¨Y`e`fk§ÄY`U¨fTfkè' =c�¦tYP¬�f}p'¡kʹyh §¨c©_��=¬�f0£wW(pR§¨fkè' =f}p5£wf�WPÀA£w¬dYPc�_ j�cd§Ä¡ n Yt¡M[ n f}p$Y�§¨fw¡�WPÀAÕ¥c�«' 5£ n c3UXf}e`c�W`p5§É§¨YP¡¨cd§�ÍRf;§Ä¡ n f£wW(pRZ9c©¡¨c�W`p�WPÀC£w¬dYPc�_Üj�YPpRZס n f¥e(UXY`� n KNMÉc�§ÉYÅ¡¨UXf}f(g�¡ n fkpס n f0Õ¥c�«' 5£ n c�a5fk¬©c�fwÀ�§ÉY`U¨f¥f}Ð9Y(£µ¡;ÊÝ3@=� X�25�t�/6=�GFZY`D�E\[Å�=�7�}8;D(�tD�E·ËÅD��GF�D�¿T]¥�t2h�C6<Á¿698kFG254#vp$Í5e` =UXfÅyPZ3gÎY`¬©¬C£w¬� R§x¡¨fkUX§T£wW(p(¡¯YPc�p�¡ n f�p=W9Z9f�y9Ê¥¢�c©p5£wfÅ¡ n f�£w¬� R§x¡¨f}U¯§MY`U¨f�£wW(p=p=fk£w¡¨f;Z�c©pÆY¬�W'W(�3gRY`pRZ�¡ n fb�5W`¡¨f}p'¡Xc�Y`¬�§Éa¿fw¡�[Vf}fkp�£w¬� R§�¡Xf}U¯§�YPUXf ¯ ÀÇ =pR£µ¡Xc©W(pR§kgRUX =p=p=c�p=e×S m W`p�ÍRe( =UXf�y`Z[Éc�¬©¬¿¬©f;Y`Z�¡XW$äxc�p9ÍRpRc�¡Xf�Z9W( =a=¬�f�£wW( =p'¡¨c�p=e(åRq7Y`p�\Åfk¦'cdZ9fkpR£wfMYbp=W9Z9f n Y`§KY`a5W( 9¡�ÿ�ï�[Éc�¬©¬�£}W`_�faRY(£¯«�¡¨W�c©¡MY`§V¡ n f�_�f;§¨§XYPe(fk§Ve`W�Y`U¨W( =pRZס n f�¬�W�W`�3ÊK¢�W�c©ÀCY`p�\�WPÀ7¡ n f�pRW�Z=fk§ n Y(§Äf}¦�c�Z=f}pR£}fYPa¿W` =¡Kÿ�ï�¡ n f�W(p=¬�\bÍ=Ð9f;Z�u��¿W`c�p'¡X§KWPÀ3S m YPUXfÄ¡ n f;§xfM[ n f}UXf 7 ÁÇÿ�ï Ãon Y`§KYP¬�¬Rc©¡X§o_�Y(§¨§�£wf}p'¡Xf}UXfkZW`p�W`pRf�¦]Y`¬© RfTW`À7ÿ ï ÊoÑ n cd§�§¨ =e`e(fk§x¡X§�¡ n Yt¡�UX =p=p=c�p=e�S m W(p�§¨ R£ n e`U¯YP� n §Vc�§ÉY�aRY(Z×cdZ9f;Y=Ê^±bPa I�GPE�H��.~�� £w¬� R§x¡¨f}UXfkZ¹S m c�§�Y`p�YP¬�e`W`UXc©¡ n _ë¡ n Yt¡�Í=Ð9f;§�¡ n cd§�¡�\��¿f�WPÀÓä¨c©p=ÍRp=c©¡¨f$Z9W( =a=¬�f£wW( =p'¡¨c�p=e(åiYPpRZ!¡X =UXpR§bW( 9¡�¡¨Wie(c©¦(f�¡ n f�§XYP_�f×Y`pR§x[Vf}U�Y`§b¡ n f�Õ¥c©«� R£ n cVY`�=�=UXW]Ð�c�_�Yt¡Xc©W(p3Êæ7c©e( =UXfMr�c�¬©¬� R§x¡¨U¯Yt¡¨f;§A¡ n f�Y`¬©e(W`UXc�¡ n _�Ê�#°¡Vc�§Kfkè' =c�¦]Y`¬©fkp'¡K¡¨WbW`U¯Z9c©p5YPUX\ÅS m fwÐ=£}f}�9¡�¡ n YP¡V§¨W`_�ffkZ=e`fk§ n Y]¦`f0YËäxp=W(U¨_�Y`¬©c�Ò}fkUXåÅ_�W�Z= =¬©f0YP¡x¡¯Y`£ n f;ZסXW�¡ n fk_�Ê h §Mc©p�W(UXZ9c�pRY`U¨\×£}¬© 5§�¡Xf}UXfkZ�S m g_�fk§X§¨Y`e`f;§ÈYP¬�W`pRe�fkY(£ n fkZ9e(f�£}W`UXU¨f;§x�¿W`pRZT¡XW��RU¨W(aRYPa=c�¬�c�¡�\¥Z9cd§�¡XU¨c�a= 9¡Xc©W(pR§7Wt¦`f}U7YM§¨fw¡CWPÀ=p=W9Z9f;§}ÊÑ n f¥cdZ9fkYÅWPÀ3¡ n f¥p=W`UX_�YP¬�c©Òkf}UV_�W�Z= =¬©f¥cd§�¡¨W�£wW`UXUXfk£µ¡�ÀÇW`UV¡ n f¥Z=W` =a=¬�f�£wW( =p'¡¨c�p=eÅW`À<Y�§x =a5§xf}¡

Page 20: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

2356

1245

5689

2356

4578

N

æ7c©e( =UXfVrRq h p�c©¬�¬� R§�¡XUXYP¡¨c�W`pÅWPÀ=¡ n fÄp=W`UX_�YP¬�c©ÒkfkZ0£}¬© R§x¡¨fkU¨f;ZbS m YP¬�e`W`UXc©¡ n _�ÊT#°¡Acd§7c�Z9fkp'¡¨cd£}YP¬'¡XWW`U¯Z9c�pRYPUX\�S m a= =¡ÉY`¬©W(p=eŧ¨W`_�fTfkZ9e(fk§ÄYÆäxp=W(U¨_�YP¬�c�Ò}f}U¯å�_�W9Z9 =¬�f¥c�§ÄY`ZRZ9fkZÆÁ�Z9fkp=WP¡XfkZÓa'\�Y`pYPUXUXWt[ÆYPp5ZT¡ n f�¬©f}¡x¡¨fkUKÔ � Ô Ã Ê]Ñ n cd§<Y`¬©e(W`UXc�¡ n _Âcd§3f;è( Rc©¦tYP¬�f}p'¡3¡XWÉ¡ n f�Õ¥c�«' 5£ n c(Y`�=�=UXW]Ð�c�_�Yt¡Xc©W(p3ÊWPÀ7¡ n fk§¨f0p=W9Z9fk§kÊ:#vp�ÍRe( =UXf¥r(Y�¡ n fkU¨f�cd§CÛ� R§x¡MW(p=f�p=W`UX_�YP¬�c©Òkf}UÉ_�W9Z9 =¬�f0YPpRZ�c©¡�£wW(U¨UXfk£w¡X§VÀÇW(Uÿ`_ � ÿ ï Ê� W9Z9f;§�£}Y`¬�£} =¬dYt¡¨f¥¡ n f}c�U�W( 9¡¨e(W`c�p=e�_�f;§¨§XYPe(fk§VaRY(§xf;ZÓW(p�c©pR£}W`_�c�p=e�_�fk§X§¨Y`e`fk§ÄY(§Éc©p�p=W(U¨_�Y`¬S m Ê Ï f¥£}Y`¬©¬¿¡ n fk§¨f�_�fk§X§XYPe`f;§ %ba ��� ÁÇÿ � à ÊF#vpס n f¥§xf}¡x¡¨c�p=e�WPÀ<£}¬© 5§�¡Xf}UXfkZ×S m g'¡ n fk§¨f�_�fk§X§XYPe`f;§YPUXfÉÀÇ =p5£µ¡¨c�W`p5§�W`À�¡ n f�c©p'¡Xf}U¯§xf;£µ¡¨c�W`p×a5f}¡�[�fkf}p Î �<Y`pRZ Î �¥§xWÅ[�fTZ9f}p=W`¡¨fM¡ n fk_ %ba ��� Á(� à [ n f}UXf��cd§V¡ n f�c©p'¡¨fkUX§¨fk£w¡¨c�W`p3Êæ=W(UÅf;Z9e`f;§0¡ n Yt¡�Z9WËp=WP¡ n Y]¦`fÓp=W`UX_�YP¬�c©Òkf}UÅ_�W9Z9 R¬©f;§}gA¡ n fk§¨f�W` 9¡Xe`W`c�p=e$_�f;§¨§XYPe(fk§b§¨c�_��R¬©\a¿fk£wW(_�f¥¡ n f0c�p=e(W`c�p=e�_�f;§¨§XYPe(fk§ % ��� � %ba ��� ÊVæ=W`UÉf;Z9e`f;§Ä¡ n Yt¡ n Y]¦(fTpRW`UX_�Y`¬©c�Ò}fkUÉ_�W�Z= =¬©f;§}g¡ n fÆ_�fk§X§XYPe`f;§ % ���!YPpRZ % � ��YPUXfia¿WP¡ n _b R¬�¡Xc©�=¬�c�fkZÚa�\�Yì£wW`UXUXfk£µ¡Xc©W(pìÀ�Y`£µ¡XW`UÓ¡ n YP¡�cd§�YÀÇ =pR£w¡¨c�W`p�WPÀCÿ _

% ��� Á�ÿ � Ã'� c ��� ÁÇÿd_ à % a ��� Á�ÿ � à Á�yPO Ã% � �¨Á�ÿ � Ã'� c ���(ÁÇÿ _ à % a� � Á�ÿ � à Á�y9Q ÃÁ�y`N Ã

[Éc©¡ n q c ��� Á�ÿ`_ Ã:� Qe ] +f. 4 +fg % a ��� Á�ÿ � à % a� � ÁÇÿ � à Á�yPj Ã

DFE GH@I Û L L fw¡:û¹a¿fÄY¥§¨fw¡CWPÀÎÕ¥c©«� R£ n c�UXf}e(c©W(pR§�Á�aRY`§¨cd£�£}¬© R§x¡¨fkUX§kgtc�p(¡Xf}U¯§xf;£µ¡Xc©W(pR§}gtc�p'¡¨fkUX§¨fk£w¡¨c�W`pWPÀ3c�p'¡¨fkUX§¨fk£w¡¨c�W`pR§ à [Éc©¡ n Z=W` =a=¬�f�£}W` =p'¡Xc©p=ebp' R_ba¿f}U¯§ ý Ô ÊF#°À ý ÔQP O0W`p=¬�\�ÀÇW`U�¡ n f�aRY`§¨c�£M£w¬� R§�u¡¨fkUX§�W`U�ÀÇW`UÄc�p'¡¨f}U¯§¨fk£µ¡Xc©W(pR§�WPÀÈc�p'¡¨f}U¯§¨fk£µ¡Xc©W(pR§oW`ÀÈaRY(§xcd£�£}¬© 5§�¡Xf}U¯§}g(¡ n fkpס n fTa5fk¬©c�fwÀ�§É£kYP¬d£w =¬dYt¡XfkZ R§¨c©pRe$¡ n f�Õ¥c�«� R£ n cÄYP�=�=UXW]Ð9c©_�YP¡¨c�W`p�YPUXf×cdZ9f}p'¡Xc�£kYP¬�¡¨W$¡ n W(§¨f�¡ n Yt¡�[VW` =¬dZ�a¿f�£kYP¬d£w =¬dYt¡XfkZ R§¨c©pRe � ba I�GE�H��.~�� S m W`p�Y�e`U¯YP� n KNMÉ[ n W'§xf¥p=W9Z9f;§ÉYPUXfT¡ n f�aRY(§xcd£¥£w¬� R§x¡¨f}U¯§kÊ`Na b.b>c L Ï f�£}W`pR§x¡¨UX R£µ¡A¡ n fÉe`U¯YP� n c�p�¡ n f�§XYP_�fÄ[VY]\0Y(§Ac�p�£}¬�Y`c©_ j=ÊCæ=W`UAfk¦`fkU¨\0c�p'¡¨fkUX§¨fk£w¡¨c�W`pWPÀ5c©p'¡¨fkUX§¨fk£w¡¨c�W`p@h�¡ n YP¡ n Y`§CYTYM�¿W(§¨c�¡Xc©¦(f ý _3[�fÄY`ZRZ ý _3pRW`UX_�Y`¬©c�Ò}fkUÈ_�W9Z9 =¬�fÄW`pbf;Z9e`f;§7[ n W(§¨fc�p(¡Xf}U¯§xf;£µ¡Xc©W(p�£wW`p'¡¯YPc�pR§ ý _¯Ê Ï fM R§¨f�¡ n f�§XYP_�f L YPe(UXY`p=e`cdYPp�Y`§Kc©p×£}¬�Y`c©_íjba= 9¡V£}W`UXU¨f;§x�¿W`p5Z�uc�p=e¥¡¨W0f;Y`£ n p=W`UX_�YP¬�c©Òkf}UA_�W9Z9 =¬�f�[Vf�Y`Z=Z�Y L YPe`U¯YPpRe`fV_b =¬©¡¨c��=¬�c©fkU:i Ô - V Ô X ÁÇÿd_ à ¡ n Yt¡Kfkp9ÀÇW`U¯£wf;§¡ n f0£wW`p5§�¡XUXY`c©p'¡Ä¡ n YP¡ 7 Á�ÿ � à Á�[Éc�¡ n ÿ � ¡ n f�c�p'¡¨fkUX§¨fk£w¡¨c�W`p�WPÀCÿ Ô - �¨ÿ Ô X à _�Y`U¨e(c©p5YP¬�c©Òkf¥Z=Wt[ÉpסXW7 Á�ÿ`_ Ã Ê Ï f¥p=Wt[ÂZ9f}ÍRp=f2i Ô - V Ô X ÁÇÿd_ Ã:� ¬�p c ��� Á�ÿ`_ à [Éc©¡ nbc ÁÇÿ`_ à ÀÇUXW`_ífkè' RYP¡¨c�W`p�yPjbY`pRZ�f ��� Y(§c�pס n f��=UXW'W`ÀÈWPÀA£w¬dYPc�_öj=Ê Ï fTÍRpRZÓ¡ n Yt¡Ä¡ n f0§x¡XYP¡¨c�W`pRY`U¨c©¡�\�£wW(pRZ9c©¡¨c�W`pR§ÄW(pס n fTÀÇU¨fkfTfkp=f}UXe`\YPp5Z�¡ n f¥Í=Ð9fkZ��¿W`c�p(¡�f;è' RYt¡Xc©W(pR§VÀÇW`UÄ¡ n f�_�fk§X§¨Y`e`fk§ÄY`U¨fTfkè' =c�¦tYP¬�f}p'¡kʹy

Page 21: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

i · óMó�ñ ãx¾Î½7»=ã�à ß »9àkj ó�ð ¾�ã õ ¾ml0½7»�»=ãx¾ ð ºÏ f0c©¬�¬� R§�¡XUXYP¡¨fT¡ n fb£}Y`p=W`p=cd£}Y`¬ÈØ¥S m YP¬�e`W`UXc©¡ n _íÀÇW(UÄ¡ n fÅÕ¥c©«� R£ n c<YP�=�=UXW]Ð9c©_�YP¡¨c�W`p×W`À7Wt¦(f}U¨u¬dYP�=�=c�p=e�z`u�p=W9Z9fÉ£}¬© R§x¡¨fkUX§CW(p�YT§Xè' RYPUXfĬ�YP¡x¡Xc�£}fÉWPÀÎp=W9Z9fk§kÊ7æ7c�e` =UXfMsÅÁ�Y à g¿Á�a à g¿Á�£ à c©¬�¬� R§�¡XUXYP¡¨f;§¡ n fÉa5fk¬©c�fwÀ�§�Yt¡KYTp=W9Z9f`g(�RYPc�UAWPÀÎp=W9Z9f;§}g'YPpRZ�Yt¡KY¥£}¬© R§x¡¨fkUoWPÀÎz¥p=W9Z9fk§kg`c�p�¡¨fkU¨_�§AW`À¿_�fk§X§XYPe`f;§�=UXW`�RY`e(YP¡¨fkZ�c�p�¡ n fMp=f}¡�[�W(U¨«ÎÊonAfk£w¡¨W`U¯§KY`U¨fM¡ n fT§xc�p=e(¬©f�c©pRZ=fwÐ�_�fk§X§¨Y`e`fk§KYP¬d§xWÅ R§xf;Z�c�p�W`U¯Z9c©upRY`U¨\�S m ÊdnAf;£µ¡XW`U¯§M[Éc©¡ n ¬©c�p=f�§xfke`_�f}p'¡X§�c�pRZ9cd£}Yt¡Xf0¡ n f�Z=W` =a=¬�fwu°c�pRZ9fwÐ9f;Z$_�fk§X§¨Y`e`f;§�YPUXc�§¨c�p=eÀÇUXW`_ ¡ n fbÕ¥c�«� R£ n c7YP�=�=UXW]Ð9c©_�YP¡¨c�W`p� R§xf;Z n fkU¨f(ÊÄÑ n fk§¨fb£}Y`p�a5f�¡ n W` Re n ¡MWPÀoY(§M£wW(U¨UXfk£w¡¨c�W`p¡¨fkU¨_�§�Y(£}£}W` =p'¡¨c�p=e�ÀÇW(UM£wW(U¨UXf}¬dYt¡Xc©W(pR§Va5f}¡�[�fkf}p�_�fk§X§¨Y`e`f;§V¡ n YP¡�W`U¯Z9c©p5YPUX\×S m ¡¨UXfkYt¡¯§�Y`§�c©p9uZ9fk�5fkpRZ9f}p'¡;ÊTÁGæ=W`UT£wW(_��5YPUXc�§¨W`p3g=æ7c�eRÊ¿s�ÁGZ à gCÁÇf à g7Á À à § n Wt[�§V¡ n fb£}W`UXU¨f;§x�¿W`p5Z9c©pRe�_�Y`U¨e(c©pRY`¬£wW(_��= 9¡XYP¡¨c�W`pR§<ÀÇW(U<¡ n fV¡¨UXc�Y`p=e` R¬�Y`U7¬dYt¡¨¡¨cd£wfV[Éc�¡ n YP¬�¬(¡XU¨cdYPpRe`¬�fk§C£ n W(§¨f}pÅY`§<¡ n fÉa5Y`§¨c�£VÕ¥c©«� R£ n c£w¬� R§x¡¨fkUX§ à ÊÏ f�ÍRpRZÆ¡ n f×_�f;§¨§XYPe(fÅ R�¿Z=YP¡¨f�UX =¬�fk§�a�\$f;è( 5Yt¡¨c�p=e�_�YPUXe`c�pRYP¬�c�ÒkYt¡Xc©W(pR§TWPÀÉæ7c©e5Ê<sËÁÇa à Y`pRZÁ�£ à [Éc�¡ n ¡ n f�a5fk¬©c�fwÀ�§¥c�pËæ7c�eRÊÈsiÁ�Y à Y`pRZ�Á�a à g<U¨f;§x�¿fk£w¡¨c�¦`f}¬�\`ÊÅæ<c�e` RU¨f�­ÆÁGY à YPpRZìÁÇa à § n Wt[ÁÇe(UXY`� n cd£}Y`¬©¬�\ à ¡ n fÉU¨f;§x R¬�¡Xc©p=eTÍ=Ð9f;ZÅ�¿W`c�p'¡Afkè' RYt¡Xc©W(pR§kÊ7Ñ n fÄ =�ÎZ=YP¡¨fÉUX =¬©f0ÁGY à cd§A¬©c�«`fV¡ n Yt¡AÀÇW(UW`U¯Z9c�pRYPUX\�S m g5[Éc©¡ n ¡ n f�Y(Z=Z9c©¡¨c�W`piW`Ào¡�[VW�Z9W( =a=¬�fwu°c©p5Z9fwÐ9fkZi_�fk§X§¨Y`e`fk§kÊ�Ñ n f� =�ÎZ=Yt¡XfÅUX =¬�fÀÇW`UÅ¡ n f�Z=W` =a=¬�fwu°c�pRZ9fwÐ9f;Z�_�fk§X§¨Y`e`fk§Åc�p'¦(W`¬�¦`f;§bZ9c�¦�c�§¨c©W(p/a�\!¡ n f�p=fk[ɬ©\'uv£wW`_��= =¡¨fkZ)§xc�p=e(¬©f}uc�pRZ9fwÐ9f;Z�_�fk§X§¨Y`e`fk§kÊCæ<c©Ð9fkZ��¿W`c�p'¡X§�WPÀ�¡ n fk§¨fM_�fk§X§XYPe`fÉ R�¿Z=YP¡¨fTfkè' RYP¡¨c�W`pR§Ke`c�¦`f�a¿f}¬�c©f}À�§�¡ n YP¡YPUXf�§�¡¯Yt¡Xc©W(pRYPUX\��5W(c©p'¡¯§�ÁÇf}_��=c�UXc�£kYP¬�¬©\$_�c�p=c�_�Y à WPÀ�¡ n f�£}W`UXU¨f;§x�¿W`p5Z9c©pRe�Õ¥c�«� R£ n coYP�R�=U¨W]Ð9c©u_�Yt¡Xc©W(pסXW�¡ n f¥ÀÇUXf}f¥f}pRf}UXe`\`Ê

a

c

bd

e

Á�Y Ãa

c

bd

e f

g

hÁÇa Ãa

c

bd

e f

g

h

i

j k

l

Á�£ Ã

ab

c d

e

il

Á�Z Ã

abp c

d

e

f

g

hil

Á�f Ã

ab

cd

e

f

g

hi

jk

l

Á À Ãæ7c©e( =UXf�s9q H Y`U¨e(c©pRY`¬Ó�RU¨W(aRYPa=c�¬�c�¡Xc©f;§�c�pÜ¡Xf}UX_�§)W`À$¡ n f?pRW�Z=f%¬�c©pR«�§¹Y`pRZýØ¥S m _�f;§�u§XYPe`f;§}Ê æ=W(U Á�Y à p=W9Z9f(g ÁÇa à ¬©c�p=f(g Á�£ à §Xè( 5YPUXfÖ£w¬� R§�¡Xf}U;g� R§¨c©pRe Y Õ¥c©«� R£ n c%YP�=u�=UXW]Ð9c©_�Yt¡Xc©W(p [Éc�¡ n zPu°p=W9Z9f £}¬© R§x¡¨fkUX§ W`p Y §¨è' RY`U¨fÖ¬dYt¡¨¡¨cd£wf(Ê I Ê eRÊ�gÜÁÇa à Z9fk�=cd£µ¡X§Á�YÙ§¨�¿fk£wcdYP¬)£}Y`§¨f WPÀ I è¿Ê�z�s9g)[ÉUXc©¡x¡¨fkp n f}UXfí R§¨c©p=eÙpRW�Z=fí¬�Y`a5fk¬�§ à q 7 Ó m Á�ÿ Ó �¨ÿ m à �¨ � Ó m Á�ÿ Ó �¨ÿ m à � Ó Á�ÿ Ó Ã � m Á�ÿ m à MÓ �Ó �Ó �-qÓ m qm srm utm M tÓ m g?[ n f}UXf�§¨ =�¿f}UöY`pRZ §¨ =a9u§X£wUXc©�9¡¯§Mc�pRZ9cd£}YP¡¨fÅ[ n c�£ n p=W9Z9fk§T_�fk§X§¨Y`e`f@ e(W�fk§ÉÀÇUXW`_ÙY`pRZ�¡¨W5Ê�ÁGZ à goÁ�f à goÁ À à q H Y`U¨e(c©pRY`¬�=UXW`aRY`a=c�¬©c©¡¨c�fk§VÀÇW(UÄ¡¨UXc�Y`p=e` =¬dYPUĬdYt¡¨¡¨cd£wf¥[Éc�¡ n jtu°p=W9Z9f¥Õ¥c©«� R£ n c3£}¬© 5§�¡Xf}U¯§}Ê

a b a b

f

g

hc

e

=ÁGY Ã

a b

fe =a b

f

g

e

d

i

j k

l

ÁÇa Ãæ7c©e( =UXfË­Rq�ØTU¯YP� n c�£kYP¬�Z9fk�=c�£w¡¨c�W`p?WPÀ�_�f;§¨§XYPe(fi =�ÎZ=YP¡¨f�fkè' RYP¡¨c�W`pR§!Á I è¿ÊÉz(r ù _�YPUXe`c�pRYP¬©uc�Ò}fýWt¦`f}U-p=W9Z9f;§-§ n Wt[Ép  Rp9ÍR¬�¬©f;Z à ÀÇW`UöØ¥S m  R§¨c©pRe Wt¦(f}UX¬�Y`�=�=c�p=eÖzPu°p=W9Z9f�Õ¥c©«� R£ n c£w¬� R§x¡¨fkUX§kÊ Á�Y à ª �¿ZRYt¡¨f�f;è( 5Yt¡¨c�W`p ÀÇW`U-¡ n fÙ§¨c©pRe`¬�fwu°c©pRZ=fwÐ _�fk§X§¨Y`e`fk§kqv mÓ ÁÇÿ Ó Ã �¨ ] +xw � m ÁÇÿ m à � Ó m Á�ÿ Ó �¨ÿ m à �yqÓ m qm srm tm uM tÓ m Ê Á�a ÃÚª �¿ZRYt¡¨f�fkè' RYt¡Xc©W(p�ÀÇW`UÂZ=W` =a=¬�fwuc�pRZ9fwÐ9f;Z�_�fk§X§¨Y`e`fk§�ÁÇc�p�¦`W`¬�¦`f;§KYÅZ9c©¦�cd§xc�W`p×a'\�¡ n f�§¨c©pRe`¬�fwu°c©pRZ=fwÐ�_�fk§X§¨Y`e`fk§oW`p�¡ n f�¬�fwÀ ¡ n Y`pRZ§¨c�Z9f à Ê

Page 22: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

z éÓ÷Éó�ð ¼5ã�æ ðÈß »9½ ñ@{¹ð º�â ñ »9ºç U¯Z9c©p5YPUX\ÅS m cd§Kf}Ð��¿fk£w¡¨f;Z�¡¨WÅ�5fkUxÀÇW(U¨_-U¨fk¬�YP¡¨c�¦`fk¬©\��5W�W(U¨¬�\bÀÇW(UKp=f}¡�[�W(U¨«9§A[Éc©¡ n _�YPp�\b¡Xc©e n ¡¬�W'W(�R§kg�£}W`p}|5c�£w¡¨c�p=e�c©p'¡¨fkUXY(£µ¡Xc©W(pR§}g9Y`pRZ�[VfkY`«�f}¦�c�Z=f}pR£}f`Ê Ï f�£}W`pR§x¡¨UX R£µ¡XfkZ�§x 5£ n YÅp=fw¡�[VW`UX«Îg«�p=Wt[ÉpËc©pÆ¡ n f�� n \9§¨c�£k§¥¬©c©¡¨fkUXYP¡¨ =UXf�Y`§T¡ n f�§¨è' RY`U¨f�¬�YP¡x¡Xc�£}f�#�§¨c�p=e�§x�=c�pËe`¬dY`§X§�c�p�Y�UXY`pRZ9W`__�YPe(p=fw¡Xc�£ÓÍRfk¬�Z�Ê%Ñ n f�p=W9Z9fk§�Y`U¨f�W(pìY�§Xè' RYPUXfÓ¬dYt¡¨¡¨cd£wf(gV[Éc©¡ n p=f;YPUXfk§x¡�p=f}c�e n a¿W`U�p=W9Z9fk§£wW(p=p=f;£µ¡¨f;Z�a�\�Yb£}W`_��RYt¡Xc©aRc©¬�c�¡�\�_�Yt¡XU¨c©Ð�WPÀ�¡ n f�ÀÇW`UX_ � ��� ��~ fwÐ9�ÈÁ/����� à f}Ð9�ÈÁ U ����� Ã

fwÐ9�ÈÁ U ����� à f}Ð9�ÈÁ/����� à �YPp5Zi¬�W9£}YP¬Cfk¦'cdZ9fkpR£wf�¦`f;£µ¡¨W(UX§�WPÀ�¡ n f�ÀÇW(U¨_ � � � ÁÇfwÐ9�ÈÁ&�h� à ù fwÐ9�ÈÁ U �h� Ãxà Ê�Ñ<WÓc�pR§x¡XYPp'¡Xc�YP¡¨f�Y�RY`Ux¡Xc�£} =¬�Y`U�p=fw¡�[VW`UX«ÎgΡ n f)������YPp5Z ���Ä�RY`UXY`_�f}¡¨fkUX§¥YPUXf�£ n W(§¨f}pËU¯YPpRZ9W(_�¬©\iY`pRZËc©p5Z9f}�¿f}p9uZ9fkp(¡X¬©\�ÀÇUXW`_ Òkf}UXWPu°_�f;YPp×Ø¥Y` R§¨§¨cdYPp��=UXW`aRY`a=c�¬©c©¡�\�Z9c�§x¡¨UXc�a= 9¡¨c�W`p5§K[Éc©¡ n §�¡¯YPpRZ=Y`UXZ×Z9f}¦�cdYt¡¨c�W`p5§�ËY`pRZ���U¨f;§x�¿fk£w¡¨c�¦`fk¬©\(ÊÑ n fTÀÇW`¬�¬�Wt[Éc©p=e�UXfk§¨ =¬�¡¯§ÄYPUXfMÀÇW(U c a'\ c ¬dYt¡x¡Xc�£}fk§V[Éc©¡ n ¡¨W(U¨W(c�Z=Y`¬5a¿W` RpRZ=YPUX\�£wW(pRZ9c©¡¨c�W`pR§ÄY`pRZ[Éc©¡ n � � Q`g�Y`pRZ�� � Oh�©Q(Ê�Ñ n c�§Ó_�W9Z9f}¬�c�§ÓZ9fk§¨c©e(p=fkZì¡XW�§ n Wt[�WPá¡ n fi[VfkY`«�p=fk§X§xf;§WPÀÄW`U¯Z9c�pRYPUX\$S m g3[ n cd£ n �5fkUxÀÇW(U¨_�§¥[Vf}¬�¬AÀÇW`U�_�YPp�\ÆWP¡ n f}U�p=f}¡�[�W(U¨«9§kÊ ç UXZ9c�pRY`U¨\$S m cd§�Y§¨�5f;£wcdYP¬5£kY`§¨f�WPÀÈ£kYPp=W(p=cd£}YP¬¿Ø¥S m g'§xWb[�f�f}Ð9�=¬©W(c�¡XfkZ�¡ n cd§o¡XW0 R§¨fÉ¡ n fT§¨Y`_�f�e`fkp=f}U¯YP¬©u��R =U¨�¿W(§¨fØ¥S m £}W�Z=fÉÀÇW`UVa5W`¡ n W(UXZ=c©pRY`U¨\�S m YPp5Z�£}Y`p=W`p=cd£}Y`¬5Ø¥S m  R§¨c�p=e0Wt¦`fkU¨¬dYP�R�=c©pRe0§¨è' RY`U¨fÉÀÇW( =U¨up=W9Z9fM£}¬© 5§�¡Xf}U¯§}g`¡ n  R§�_�Y`«�c©p=e0£}W`_��= 9¡¯Yt¡Xc©W(pRYP¬¿£wW(§x¡K£wW(_��RYPUXc�§¨W`p5§CU¨f;Y`§¨W`pRY`a=¬©f(Ê Ï f�§x¡XY`Ux¡XfkZ[Éc©¡ n U¯YPpRZ=W`_�c©ÒkfkZ�_�fk§X§XYPe`f;§KY`pRZ�W`p=¬�\�§�¡Xf}�=�¿fkZ n YP¬©À u�[ÄY]\Å¡¨Wt[ÄYPU¯Z=§C¡ n f�£wW(_��= 9¡¨f;Z�¦tYP¬� =fk§WPÀ¥¡ n f$_�fk§X§¨Y`e`f;§�Yt¡�f;Y`£ n c©¡¨f}U¯Yt¡Xc©W(p¹c©p�W(UXZ9fkU�¡¨W n f}¬��£wW(p'¦(f}UXe`fkpR£wf(Ê Ï fiÀÇW` =p5Zì¡ n YP¡£}Y`p=W`pRc�£kYP¬AØ¥S m ¡¨W�W(«iYPa¿W` 9¡T¡�[Écd£wf�Y`§T¬©W(p=e�Y`§TW`U¯Z9c�pRYPUX\�S m �¿f}U¥c�¡Xf}U¯Yt¡Xc©W(p3g�a= 9¡0[VW` =¬dZ¡�\��=cd£}YP¬�¬�\$U¨f;Y`£ n Y�e`c�¦`f}p!¬�f}¦(f}¬AWPÀÉ£wW(p�¦`f}UXe`fkpR£wfÅc©p!_�YPp�\iÀÇf}[Vf}U�c©¡¨f}U¯Yt¡Xc©W(pR§kÊ�#vpÆÀ�Y(£µ¡;gÈÀÇW(U¡ n f�_�YPÛ�W`UXc�¡�\�W`ÀK¡ n f�Z9W`Òkf}pR§¥W`ÀV§XYP_��=¬�fk§T¡ n Yt¡0[Vf�¬©W�W`«(fkZÆYt¡kg7S m Z=c�ZËp=W`¡�£}W`p�¦`fkU¨e(fÅYP¡YP¬�¬�g3[ n c�¬©f�£}Y`p=W`pRc�£kYP¬AØ¥S m Y`¬©[ÄY]\9§�£}W`p�¦`fkU¨e(fkZ�ÀÇW`U¥¡ n cd§¥_�W9Z9fk¬oYPpRZËYP¬�[ÄY]\�§M¡¨W�Y`£k£w =U¯Yt¡XfYPp5§x[Vf}U¯§}Ê$Á Ï f�ÀÇW` =pRZË¡ n Yt¡bÀÇW`U¥¡ n f�Ò}fkU¨W`uGÍRfk¬�ZËjtuvZ9c�_�f}pR§¨c©W(pRYP¬o§¨�=c�p�e`¬dY`§X§¥[Éc�¡ n ¡¨W`UXW`cdZ=Y`¬a¿W` =pRZRYPUX\i£wW(pRZ9c©¡¨c�W`pR§kg<[ n c�£ n cd§0YPp!f}¦(f}p!_�W`UXf�Z9c â�£} =¬�¡Å_�W9Z9fk¬GgC£}Y`p=W`pRc�£kYP¬KØ¥S m [Éc�¡ nN]ÐRNtÐ=NÅ£w Ra=c�£�£}¬© 5§�¡Xf}U¯§Ä[�W( =¬�Z�YP¬d§¨WbÀ�Y`c©¬�¡XW�£wW`p�¦(f}UXe`f à Êæ=W(U cª� NPO¥W(UA¬�Y`U¨e(f}U;gPc©¡o[VY(§oZ9c â�£} =¬©¡o¡¨W�_�YP«(f�£wW(_��5YPUXc�§¨W`pR§7[Éc©¡ n Y`p'\bWP¡ n f}UKY`¬©e(W`UXc�¡ n _�ga¿fk£}Y` R§¨fbW`U¯Z9c�pRYPUX\ÓS m Z=c�Zip=WP¡¥£wW`p�¦(f}UXe`f�YPpRZ H W`p'¡¨f ^ YPUX¬©W�§¨c�_b =¬dYt¡Xc©W(pR§�§¨ !áÎf}UXfkZ�ÀÇUXW`_fwÐ�¡XU¨fk_�fk¬©\)§x¬�Wt[öfkè' =c�¬�c©a=U¯Yt¡Xc©W(p3Ê)®MWt[�fk¦`fkUkgoe(f}p=fkUXY`¬©c�Ò}f;Z�a¿f}¬�c©f}ÀT�=UXW`�5YPe(YP¡¨c�W`p�£}W`p�¦`fkU¨e(fkZUXfkY`§¨W`p5YPa=¬�\bU¯YP�=cdZ9¬�\b¡XWb�=¬dYP 5§xc�a=¬�fwu°¬©W�W`«�c�p=e�a¿f}¬�c©f}À�§}ÊoæRW`U�§¨_�YP¬�¬ c g'[Vf�£wW( =¬dZ�£wW(_��5YPUXf�[Éc�¡ nfwÐ=Y(£µ¡ÉU¨f;§x R¬�¡¯§}g9a�\� R§¨c�p=e m f;YPUX¬GÔ §Ä£w¬� R§x¡¨f}UXc�p=eÅ_�f}¡ n W9Z�W`p�Y�£ n YPc�pÓW`À c a'\iQ¥§¨ =�¿f}U¨u�pRW�Z=fk§kÊÑ<W�e`c�¦`f�Y�è' RYP¬�c�¡¯Yt¡Xc©¦(f�ÀÇfkf}¬ÈÀÇW`UÉ¡ n f�U¨f;§x R¬�¡¯§}g=[Vf0£wW(_��RYPUXf¥W`U¯Z9c©p5YPUX\×S m g=£}YPpRW`p=cd£}Y`¬ÈØ¥S m gYPp5Z�¡ n f�fwÐ=Y`£w¡KU¨f;§x R¬�¡¯§AÀÇW`UVYPp cª� Q;O�¬dYt¡x¡Xc�£}f�[ n fkU¨f�W(UXZ=c©pRY`U¨\�S m Z9cdZ�£wW(p'¦(f}UXe`f(Ê L cd§�¡Xc©pRe¡ n f�¦tY`¬© =f;§TWPÀo¡ n f�W`p=f}u�p=W9Z9f�_�YPUXe`c�pRYP¬7�=UXW`aRY`a=c�¬©c©¡¨c�fk§Tc�piW`pRfÅW`ÀK¡ n f�UXWt[�§kg¿[VfÅÍRpRZi¡ n YP¡W`U¯Z9c�pRYPUX\�S m e`c�¦`fk§�Á�Ê O`O`z(j`­(O's9g=Ê stz'y`O(N9g5Ê j'NP­(r`r=gRÊ r'N9Qkl(O=g5Ê j�s`s]z�y9g=Ê zRQ]Ntz'j=gRÊ y(sP­Pz�N9g=Ê s]z�y`y`y=gÊ ­(yPjRQ;y=g`Ê l`l`r(j(N à g]£}YPpRW`p=cd£}Y`¬'Ø¥S m e(c©¦(fk§KÁxÊ z'O(N(y`y9g'Ê ytzRQ(Q;y=g`Ê z(l=Q;­Pz5g`Ê ytz'N`j(N=g(Ê z`z'­=Q;N=g`Ê z(­(O=Q}z5gÊ y9Q;y`O=Q(g�Ê y'str`l(j=g�Ê y(s(s�QkORgÎÊ y`l'sPy's à g=YPpRZ�¡ n f0fwÐ=Y(£µ¡�UXfk§¨ =¬�¡¯§�[Vf}UXf�Á�Ê z(O=Q;j=Q(g3Ê yPz(O(j`­=gÎÊ z(­(l(NPjRgÊ ytz'y`O`rRg5Ê z`z�yPj's9gÎÊ z9st­(y`r=g¿Ê y=Qkr(­`r=g¿Ê yP­=Q;O`­Rg¿Ê y(s(stlRQ`g¿Ê y`l`­`­RQ à Ê� �.¸3472�µ���D�E�Á¿D��/D�4Î8k���Ï Y`£¯«�p=Wt[ɬ�fkZ=e`fk§Ä§¨ =�=�¿W`U¨¡Éa'\ H$ªMJ #°u h�J ç uv¤ h�h ®MO`zPuvl`rPuXQ}u°O(jPzRQ(g H$ª�J # � O`O(O=Q}z`u°O(OtuQwuvO`r(j's0YPpRZ � ¢9æ³#�#x¢�u°l(l`­`­(rPz�N9Ê{¹ð3ò¯ð ¼ ðÈß ¾ ð ºé Qwê0®bÊ h ÊRSVfw¡ n f`Ê `Na b Ãf��� b°��­b Ãf��� b ��� b �)� g�Q]yPO=q y`y(N9Ê

Page 23: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

é N]ê Ï Ê Ñ¥ÊPæ=UXf}fk_�Y`p�Y`pRZ I Ê ^ Ê m Y`§¨Òw¡¨W(UkÊ L f;YPUXp=c�p=e�¬�Wt[/¬�f}¦(f}¬9¦�c�§¨c�W`p3Ê!#vp `Na b Ã��`�|� { E��$D b � c �D b I �� { ~ a8� H z H b � g=�RYPe(fk§¥Q(Qk­(Nn«¿Q(Qk­`lRÊ=Qkl(l`lRÊé jtê0SVUXf}pRZ=Y`p Ú Ê5æRU¨fk\`Ê@� a G �F� H=ÃxGEo� b ��~uE z cuba9` G { { ~ a ��DFE G z�z H �NÃ�G { H b �F����G { G�D b I ��a ~ z��z H b �ñG��� D � GP�h��~\E�D b �H@� �PÊ H #�Ñ m U¨f;§¨§kgÎQkl`l(­=Êé zPê Ú ÊÉ®�c Û�_�Y`pR§�YPp5Z Ú ÊĤ�f�SVW'fkUkÊ h p?Y`�=�=UXW]Ð9c©_�Yt¡Xc©W(p�_�fw¡ n W�ZÂÀÇW`U�W(UXZ9fkUxuvZ9cd§xW(UXZ=f}U�=UXW`aR¬©fk_�§kÊ `���° z H=ÃxG g5�RYPe(fk§Äz9s�Q|«�z(­`zRgRQ;l(y(y9Ê

é y]êb¢��¿fk£}c�Y`¬�c�§X§¨ =f�W`p�Õ¥c©«� R£ n c�_�f}¡ n W9Z=§kÊ `Na b � �=� � ~ ba � `5��° z � gÈQ(Q;y9gÎQ;l`lPz5Êé rtê H Ê #µÊ Ú W(UXZ=Y`p3g��7ÊCØ n Y n U¯YP_�YPpRcGgCÑ¥Ê Ú Y`Y`«�«`W`¬dY=g7YPpRZ L ÊC¢=YP =¬�Ê h p)c©p'¡XU¨W9Z9 R£w¡¨c�W`p!¡XW¦tYPUXcdYt¡¨c�W`p5YP¬�_�fw¡ n W9Z=§ÉÀÇW(UMe`U¯YP� n c�£kYP¬�_�W9Z9f}¬d§kÊN#vp H Ê #µÊ Ú W(UXZ=Y`p3g=f;Z9c�¡XW`U;g ��~xG a �hH@� � H@�� a G �F� H=ÃxGPE6� b ��~\E zµÊ H #�Ñ m UXfk§X§}gÎQ;l`l`­RÊ

é s;ê J Ê5Õ¥c©«� R£ n c�Ê `���° z �2~ Æ;g=�RY`e`f�­=Q(q l(­`­=g¿Qkl'y9Q(Êé ­tê0æVÊ J Ê<Õ�§¨£ n cd§X£ n YPp=e�YPpRZËSTÊ Ú Ê3æ=UXf}\`Ê�#°¡Xf}U¯Yt¡Xc©¦(f�Z=fk£wW9Z9c�p=e�WPÀÉ£wW(_��5W( =pRZË£wW9Z9fk§¥a�\�=UXW`a5YPa=c�¬©c©¡�\��=UXW`�5YPe(YP¡¨c�W`pÓc©p�e(UXY`� n cd£}YP¬Î_�W9Z9f}¬d§kÊ �:�5���u� bÂ!a ��GE b � � ~uE�~�à { ~���� a ~xG zH@�BD b I�I  �hH=ÃxG { H b � gÈQ;rRÁGN à q N9Q;ln«9N`j`ORgRQkl(l`­RÊ

é ltê0æVÊ J ÊVÕ�§X£ n cd§¨£ n Y`p=eRgÄSTÊ Ú Ê�æRU¨fk\`g�YPp5Z�®0Ê u h Ê L W'fk¬©c�e`fkUkÊ�æRY(£µ¡XW`U×e(UXY`� n §�Y`pRZì¡ n f§¨ =_�u��RU¨W9Z9 R£w¡�YP¬�e`W(U¨c©¡ n _�Ê ��������� a GP� z G�à { H b � z b �8�|� cubPa I�G { H b ��� � ~ ba|° g3NPO(O=Q(ÊK¡XWY`�=�5f;YPU;Ê

é QkOtê J Ê Ú Ê H £ I ¬�c�fk£wf(go¤bÊ Ú Ê ^ Ê H Y(£¯«�Õ�Y]\(gAYPpRZ Ú Ê æVÊ ^ n f}p=e5Ê/ÑÈ =UXa¿WÆZ9f;£wW9Z9c�p=eËY(§ÅY(§ÅY`pc�pR§x¡XYPp5£wf�WPÀ m fkY`U¨¬�Ô §�  a¿f}¬�c©f}ÀÉ�=U¨W(�RYPe'Yt¡Xc©W(p3Ô�YP¬�e`W`UXc©¡ n _�Ê �������k� bPÂ!a ��GPE b � � ~\E ~xà { ~��� a ~xG z H@�³D b I�I  �hH=ÃxG { H b � g3Q;rRÁGN à q�Q}z'On«¿Q]y`N=gRQkl(l`­RÊ

é Q`Qwê J Ê Ú Ê H £ I ¬�c©f;£wf(g I Ê J W�Z=f}_�c�£ n gPYPpRZ Ú Ê æ�Ê ^ n fkp=eRÊ'Ñ n fVÑ< =UXa5WTZ9fk£}c�§¨c�W`p0YP¬�e`W(U¨c©¡ n _�Ê�#vp`Na�b Ãf� È�È�a �I�2E@E ~ a { b �³D b � c ~ a ~u��Ã�~ b �³D b I�I  �hH=ÃxG { H b � z �jD b � { a b E$G���ªD b I �� { H@� �Pg�RY`e`f;§Vj(r`r«�j'sPl=g H W`p'¡¨cd£wfk¬©¬�WRg!# L g3Qkl(l(y=Ê

é Q;N]ê0Õ�Ê m Ê H  =U¨� n \(g � Ê Ï f}cd§¨§kgKY`pRZ H Ê #µÊ Ú W(UXZRYPp3Ê L W�W`��\!a¿f}¬�c©f}À��=UXW`�RY`e(YP¡¨c�W`pËÀÇW`U�YP�=u�=UXW]Ð9c©_�YP¡¨f¥c©p=ÀÇf}UXf}pR£}f`qKY`p�fk_��=c©UXcd£}YP¬3§x¡¨ 5Z9\`ÊQ#vp `Na b Ã�~x~��H@� �z b�c�¡ ��Ãx~ a { GH@� { ° H@�8�¢� gQ;l`l`lRÊ

é Qkjtê Ú  RZ9f;Y m fkYPUX¬�Ê `Na b }xG�}\H@E�H z|{ H=Ã!��~xG z b ��H@� � H@�$�|� { ~\E@E�H � ~u� { ��° z|{ ~\I z L ^ ~ {(£ bPa:¤ z b�cN` E�G  �z H=}\E ~!�u� c ~ a ~\��Ãx~ Ê H W`UXe(YPp�Õ�YP 9ÀÇ_�Y`p=p3g3Q;l`­`­RÊé Q}zPê H Ê h Ê m fkWP¡ÉYPpRZ J Ê ¤bÊ5¢ n Y`£ n ¡¨fkUkÊ7æ= R§¨c�W`p�YPpRZÓ�=UXW`�RY`e(Yt¡Xc©W(p�[Éc�¡ n _Å =¬©¡¨c��=¬©f�W(aR§¨f}UX¦]YPu

¡Xc©W(pR§Äc©p�a¿f}¬�c©f}À7pRfw¡�[VW`UX«�§kÊ � a { H �NÃuH=GPE��|� { ~\E@E�H � ~u��Ãx~ g=z'­=q NPl(ln«�jRQk­RgRQkl(l=Q(Êé Q;y]ê0Ñ¥Ê Ú Ê J c�£ n Y`UXZR§xW(p3ÊoÑ n f�e`fkW`_�fw¡XU¨\�W`À<¡X =UXa5W`u°Z9f;£wW9Z9c�p=e�Z9\�pRYP_�cd£}§kÊ �:�5���¥� a G�b�|� cub� � ~ bPa|° gRz(r5Á�Q à q ln«9N`j=g9NPO(O`O=Êé Qkrtê � Ê Ï f}cd§X§}Ê ^ W`UXUXfk£µ¡Xp=fk§X§3W`ÀR¬�W�£kYP¬'�=UXW`aRY`a=c©¬�c©¡�\T�RU¨W(�RYPe'Yt¡¨c�W`p�c�pbe(UXY`� n cd£}Y`¬(_�W9Z9f}¬d§7[Éc�¡ n

¬�W�W`�R§kÊ ^ ~ Â!a GEÜD b I �� { G { H b � gÈQ]N9q�Q|«'z5Q`g9NPO(O`O=Êé Q]s;ê � Ê Ï f}cd§¨§7YPp5Z Ï Ê Ñ¥ÊtæRU¨fkf}_�YPp3Ê ç p�¡ n f�W(�9¡¨c�_�YP¬�c�¡�\¥WPÀR§¨W`¬� 9¡Xc©W(pR§3W`À9¡ n fV_�YtÐ�u��RU¨W9Z9 R£w¡

a¿f}¬�c©f}À=�=UXW`�RY`e(Yt¡Xc©W(p�YP¬�e`W`UXc©¡ n _�Ê �:�5���u� a G� z G�à { H b � z b �N�u� cuba I�G { H b � � � ~ ba|° g(N`O`O`ORÊ¡XW�Y`�=�¿fkYPU;Ê

é Qk­tê � Ê Ï c�a¿f}UXeRÊ D b ��~ z G���O��~xà b �PH@� � b � � ~u��~ a GPE � a G �F� zµÊ mKn ¤?¡ n f;§xcd§kg�¤�f}�5YPU¨¡¨_�f}p'¡ÄWPÀI ¬©f;£µ¡¨UXcd£}YP¬ I p=e`c�p=f}fkU¨c�p=e5g ª Ê L c�p=«(W`�=c�p=eRgR¢9[�f;Z9f}p3g�Q;l`l(r=Ê

Page 24: Bethe free energy, Kikuchi approximations, and …sczhu/Workshops/sctv01/TR2001-16.pdfBethe free energy, Kikuchi approximations, and belief propagation algorithms Jonathan S. Yedidia,

é Qkltê � Ê Õ�Y`aRY`§ n c�_�Y�Y`pRZ$¤ÅÊ3¢9Y`Y(Z�ÊMSVf}¬�c©f}Ào�=UXW`�RY`e(Yt¡Xc©W(p�¦�§�¡XY`��ÀÇW`UTZ9fk£}W�Z=c©p=e�£wW(U¨UX =�9¡XfkZ_�fk§X§XYPe`f;§}Ê � Â!a b�� � `��!° z �¦��~ { {Gg9�5YPe`fTz`z5q r(r`­Rg¿Q;l`l(­=Ê