john cunningham and david knowles machine learning rcc 08...
TRANSCRIPT
![Page 1: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/1.jpg)
John Cunninghamand
David Knowles
Machine Learning RCC08 December 2011
Approximate Inference
![Page 2: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/2.jpg)
• Motivation
• Taxonomy
• Summations
• Estimators
• Easier Integrals
• Summary
Outline
![Page 3: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/3.jpg)
• Motivation
• Taxonomy
• Summations
• Estimators
• Easier Integrals
• Summary
Outline
![Page 4: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/4.jpg)
Probabilistic Inference
• Bayes Rule:
![Page 5: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/5.jpg)
Probabilistic Inference
• Bayes Rule:
• (frequentist/statistical inference)
• (Bayesian/non-Bayesian distinction)
• (conjugate models)
• (enumerable simple cases)
![Page 6: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/6.jpg)
• Many (most?) problems of interest in inference can be written as an integral of type:
• Examples:
• Posterior mean and moments:
• Data likelihood and model selection:
• Prediction:
Just an Integral
![Page 7: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/7.jpg)
Central Object of Interest
![Page 8: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/8.jpg)
Central Object of Focus
• Why not...
![Page 9: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/9.jpg)
Central Object of Focus
• Why not...
• message passing on the factor graph?• explains {BP,VB,EP,Gibbs,etc.} nicely• abstracts approximate inference to message calculation• mechanistic, not actually the problem we are trying to solve
![Page 10: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/10.jpg)
Central Object of Focus
• Why not...
• message passing on the factor graph?• explains {BP,VB,EP,Gibbs,etc.} nicely• abstracts approximate inference to message calculation• mechanistic, not actually the problem we are trying to solve
• the posterior?• pretty much the same thing, but again not often the core problem
![Page 11: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/11.jpg)
• A huge field• Bishop PRML: ~100 pages• MacKay Info Theory, Inference, ... : ~180 pages• Murphy ML: A Probabilistic Perspective: ~110 pages• MLSS: ~half a day
Fool’s Errand
![Page 12: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/12.jpg)
• A huge field• Bishop PRML: ~100 pages• MacKay Info Theory, Inference, ... : ~180 pages• Murphy ML: A Probabilistic Perspective: ~110 pages• MLSS: ~half a day
• Scope of this talk: • tutorial view of the field• incorporate by reference where possible• details where (hopefully) valuable
Fool’s Errand
![Page 13: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/13.jpg)
• Motivation
• Taxonomy
• Summations
• Estimators
• Easier Integrals
• Summary
Outline
![Page 14: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/14.jpg)
Approximate Inference Taxonomy
![Page 15: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/15.jpg)
Approximate Inference Taxonomy
![Page 16: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/16.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with summations”
• Sampling methods• Central problem: how to
sample • Monte Carlo, MCMC,
Gibbs, etc.
![Page 17: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/17.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with summations”
• Sampling methods• Central problem: how to
sample • Monte Carlo, MCMC,
Gibbs, etc.
![Page 18: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/18.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with easier integrals”
• Message Passing on Factor Graph
• Central problem: how to find
• VB, EP, etc.• Note (cheat): Also
“replace hard sums with easier sums”: BP, LBP,etc.
• “Replace hard integrals with summations”
• Sampling methods• Central problem: how to
sample • Monte Carlo, MCMC,
Gibbs, etc.
![Page 19: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/19.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with easier integrals”
• Message Passing on Factor Graph
• Central problem: how to find
• VB, EP, etc.• Note (cheat): Also
“replace hard sums with easier sums”: BP, LBP,etc.
• “Replace hard integrals with summations”
• Sampling methods• Central problem: how to
sample • Monte Carlo, MCMC,
Gibbs, etc.
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
![Page 20: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/20.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with easier integrals”
• Message Passing on Factor Graph
• Central problem: how to find
• VB, EP, etc.• Note (cheat): Also
“replace hard sums with easier sums”: BP, LBP,etc.
• “Replace hard integrals with summations”
• Sampling methods• Central problem: how to
sample • Monte Carlo, MCMC,
Gibbs, etc.
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
Deterministic MethodsRandom Methods
![Page 21: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/21.jpg)
• Motivation
• Taxonomy
• Summations
• Estimators
• Easier Integrals
• Summary
Outline
![Page 22: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/22.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with summations”
• Sampling methods• Central problem: how to
sample • Monte Carlo, MCMC,
Gibbs, etc.
![Page 23: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/23.jpg)
Summations
• Two basic types: Sampling and MCMC
• “Instead of choosing [points] randomly, then weighting them..., we choose [points] with a probability... and weight them evenly.” - Metropolis et al (1953).
![Page 24: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/24.jpg)
Summations
• Sampling:
![Page 25: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/25.jpg)
Summations
• Sampling:
![Page 26: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/26.jpg)
Summations
• Sampling:
• “pick an arbitrary point and weight it by what you care about.”
• MC, importance, rejection.
![Page 27: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/27.jpg)
• MH/MCMC:
Summations
• Sampling:
• “pick an arbitrary point and weight it by what you care about.”
• MC, importance, rejection.
![Page 28: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/28.jpg)
• MH/MCMC:
Summations
• Sampling:
• “pick an arbitrary point and weight it by what you care about.”
• MC, importance, rejection.
![Page 29: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/29.jpg)
• MH/MCMC:
Summations
• Sampling:
• “pick an arbitrary point and weight it by what you care about.”
• MC, importance, rejection.
![Page 30: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/30.jpg)
• MH/MCMC:
• “pick a point from what you care about and weight it evenly.”
• MH, MCMC, AIS, Gibbs, HMC, Slice Sampling, ESS, Hamiltonian MCMC, RML, ...
Summations
• Sampling:
• “pick an arbitrary point and weight it by what you care about.”
• MC, importance, rejection.
![Page 31: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/31.jpg)
Big Topic, Incorporated by Reference
• Iain Murray’s MLSS lectures: http://videolectures.net/mlss09uk_murray_mcmc/
![Page 32: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/32.jpg)
• Motivation
• Taxonomy
• Summations
• Estimators
• Easier Integrals
• Summary
Outline
![Page 33: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/33.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
![Page 34: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/34.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
• Laplace
![Page 35: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/35.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
• Laplace
![Page 36: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/36.jpg)
• MAP
Approximate Inference Taxonomy
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
![Page 37: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/37.jpg)
• Nested Laplace• Rue and Martino (2009), “Approximate
Bayesian Inference for latent Gaussian models by using integrated nested Laplace approximations”, JRSSB.
Approximate Inference Taxonomy
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
![Page 38: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/38.jpg)
• Nested Laplace• Rue and Martino (2009), “Approximate
Bayesian Inference for latent Gaussian models by using integrated nested Laplace approximations”, JRSSB.
• ...but see Cseke and Heskes (2011), “Approximate marginals in latent Gaussian models”, JMLR.
Approximate Inference Taxonomy
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
![Page 39: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/39.jpg)
• Motivation
• Taxonomy
• Summations
• Estimators
• Easier Integrals
• Summary
Outline
![Page 40: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/40.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with easier integrals”
• Message Passing on Factor Graph
• Central problem: how to find
• VB, EP, etc.• Note (cheat): Also
“replace hard sums with easier sums”: BP, LBP,etc.
![Page 41: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/41.jpg)
Message Passing on Factor Graph
![Page 42: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/42.jpg)
Belief Propagation / Sum-product
![Page 43: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/43.jpg)
Approximate messages
![Page 44: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/44.jpg)
Expectation Propagation (EP)
![Page 45: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/45.jpg)
Expectation Propagation (EP)
![Page 46: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/46.jpg)
Instead...
~
~
![Page 47: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/47.jpg)
Instead, EP does this...
Moment Match
A - Form “CAVITY” B - Add a true factor and “PROJECT”
~
~
![Page 48: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/48.jpg)
Instead, EP does this...
Moment Match
A - Form “CAVITY” B - Add a true factor and “PROJECT”
![Page 49: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/49.jpg)
At convergence, we have...
~
~
![Page 50: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/50.jpg)
Approximately this...
Moment Match
![Page 51: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/51.jpg)
Beyond Simple EP
![Page 52: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/52.jpg)
Variational Bayes / Variational Message Passing
![Page 53: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/53.jpg)
Variational Bayes / Variational Message Passing
![Page 54: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/54.jpg)
Summary of Message Passing Perspective
![Page 55: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/55.jpg)
• Exclusive (VB mode-seeking) vs. inclusive (EP) KL, consequences for multimodality
• Damping for EP• Power EP• More structured approximations (GBP, tree EP,
structured VB)• Connection to EM • Infer.NET
Things to be aware of
![Page 56: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/56.jpg)
• Motivation
• Taxonomy
• Summations
• Estimators
• Easier Integrals
• Summary
Outline
![Page 57: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/57.jpg)
Approximate Inference Taxonomy
• “Replace hard integrals with easier integrals”
• Message Passing on Factor Graph
• Central problem: how to find
• VB, EP, etc.• Note (cheat): Also
“replace hard sums with easier sums”: BP, LBP,etc.
• “Replace hard integrals with summations”
• Sampling methods• Central problem: how to
sample • Monte Carlo, MCMC,
Gibbs, etc.
• “Replace hard integrals with estimators”
• “Non-Bayesian” methods• Central problem: how to
find • MAP, ML, Laplace, Nested
Laplace, etc.
![Page 58: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/58.jpg)
Summary of Features
![Page 59: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/59.jpg)
Summary of Features
• exact (eventually)
• fast/efficient in big-huge cases (at times the only option)
• poor for model selection
• slow error convergence
![Page 60: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/60.jpg)
• analytically useful
• fits into many MLschemes (bounds)
• fast/efficient in small-medium cases
• no exactness (ignores some features of true integral)
Summary of Features
• exact (eventually)
• fast/efficient in big-huge cases (at times the only option)
• poor for model selection
• slow error convergence
![Page 61: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/61.jpg)
• analytically useful
• fits into many MLschemes (bounds)
• fast/efficient in small-medium cases
• no exactness (ignores some features of true integral)
Summary of Features
• exact (eventually)
• fast/efficient in big-huge cases (at times the only option)
• poor for model selection
• slow error convergence
• quick and dirty
• often works well
• quick and dirty (local... ignores many features of true integral)
![Page 62: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/62.jpg)
• Has many names and duplicate fields, but in the end is just numerical integration
• Disappointingly (necessarily?) fractured field
• Inherently problem-specific
Conclusion
![Page 63: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/63.jpg)
Resources• Books
• Bishop (2006) “Pattern Recognition and Machine Learning”, Chapters 10-11• Murphy (2012) “ML: A Probabilistic Perspective”, Chapters 18 - 22
• Rasmussen and Williams (2006), “Gaussian Processes for Machine Learning”, Chapter 3 (for EP and Laplace).
• MacKay (2003) “Information Theory, Inference, and Learning Algorithms”, Part IV
• Video• MLSS 09: Murray (MCMC): http://videolectures.net/mlss09uk_murray_mcmc/
• MLSS 09: Minka (Min Divergence): http://videolectures.net/mlss09uk_minka_ai/
• Papers• Wainwright and Jordan (2008), “Graphical Models, Exponential Families, and
Variational Inference.” Foundations and Trends in Machine Learning.
• Winn and Bishop (2005), “Variational Message Passing”, JMLR.• Minka and Winn (2009): Gates, NIPS
• Hennig (2011), “Approximate Inference in Graphical Models” (PhD Thesis), Chap 2.• Kuss and Rasmussen (2005), “Assessing Approximate Inference for Binary
Gaussian Process Classification”, JMLR.
![Page 64: John Cunningham and David Knowles Machine Learning RCC 08 ...cbl.eng.cam.ac.uk/pub/Intranet/MLG/ReadingGroup/20111208cambri… · John Cunningham and David Knowles Machine Learning](https://reader034.vdocument.in/reader034/viewer/2022050514/5f9e4ea28885e37c135f4be9/html5/thumbnails/64.jpg)