introduction to bayesian inference part 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides ›...
TRANSCRIPT
![Page 1: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/1.jpg)
INTRODUCTION TOBAYESIAN INFERENCE – PART 2
CHRIS BISHOP
![Page 2: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/2.jpg)
![Page 3: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/3.jpg)
Personal Healthcare Revolution
Electronic health records (CFH)
Personal genomics (DeCode, Navigenics, 23andMe)
X-prize: first $10k human genome technologyNIH: $1k by 2014
Microsoft Research Cambridge:PhD ScholarshipsInternships: 3 monthsPostdoctoral Fellowships
![Page 4: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/4.jpg)
Why Probabilities?
Image vector
Class “cancer” or “normal”
![Page 5: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/5.jpg)
Decisions
One-step solution
train a function to decide the class
Two-step solution
inference : infer posterior probabilities
decision : use probabilities to decide the class
![Page 6: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/6.jpg)
Minimum Misclassification Rate
![Page 7: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/7.jpg)
Why Separate Inference and Decision?
• Minimizing risk (loss matrix may change over time)
• Reject option
• Unbalanced class priors
• Combining models
![Page 8: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/8.jpg)
Loss Matrix
Decision
True class
![Page 9: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/9.jpg)
Minimum Expected Loss
Regions are chosen, at each x, to minimize
![Page 10: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/10.jpg)
Reject Option
![Page 11: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/11.jpg)
Unbalanced class priors
In screening application, cancer is very rare
Use “balanced” data sets to train models, then use Bayes’ theorem to correct the posterior probabilities
![Page 12: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/12.jpg)
Combining models
Image data and blood tests
Assume independent for each class:
![Page 13: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/13.jpg)
Binary Variables (1)
Coin flipping: heads=1, tails=0
Bernoulli Distribution
![Page 14: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/14.jpg)
Expectation and Variance
In general
For Bernoulli
![Page 15: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/15.jpg)
Likelihood function
Data set
Likelihood function
![Page 16: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/16.jpg)
Prior Distribution
Simplification if prior has same functional form as likelihood function
Called conjugate prior
![Page 17: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/17.jpg)
Beta Distribution
![Page 18: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/18.jpg)
Posterior Distribution
![Page 19: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/19.jpg)
Posterior Distribution
![Page 20: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/20.jpg)
Properties of the Posterior
As the size N of the data set increases
![Page 21: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/21.jpg)
Predictive Distribution
What is the probability that the next coin flip will be heads?
![Page 22: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/22.jpg)
The Exponential Family
where ´ is the natural parameter
We can interpret g(´) as the normalization coefficient
![Page 23: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/23.jpg)
Likelihood Function
Give a data set,
Depends on data through sufficient statistics
![Page 24: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/24.jpg)
Expected Sufficient Statistics
![Page 25: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/25.jpg)
Conjugate priors
For the exponential family
Combining with the likelihood function, we get
Prior corresponds to º pseudo-observations with statistic Â
![Page 26: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/26.jpg)
Bernoulli revisited
The Bernoulli distribution
Comparing with the general form we see that
and so
Logistic sigmoid
![Page 27: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/27.jpg)
Bernoulli revisited
The Bernoulli distribution in canonical form
where
![Page 28: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/28.jpg)
The Gaussian Distribution
![Page 29: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/29.jpg)
Likelihood Function
![Page 30: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/30.jpg)
Bayesian Inference – unknown mean
Assume ¾2 is known
Data set
Likelihood function for ¹
![Page 31: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/31.jpg)
Bayesian Inference – unknown mean
Conjugate prior is a Gaussian
which gives a Gaussian posterior
![Page 32: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/32.jpg)
![Page 33: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/33.jpg)
Bayesian Inference – unknown precision
Now assume ¹ is known
Likelihood function for precision ̧ = 1/¾2
![Page 34: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/34.jpg)
Conjugate prior
Gamma distribution
![Page 35: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/35.jpg)
Unknown Mean and Precision
Likelihood function
Gaussian-gamma distribution
![Page 36: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/36.jpg)
Gaussian-gamma Distribution
![Page 37: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/37.jpg)
Linear Regression (1)
Noisy sinusoidal data
![Page 38: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/38.jpg)
Linear Regression (2)
Linear combination of basis functions
Noise model
Likelihood function
![Page 39: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/39.jpg)
Linear Regression (3)
Polynomial basis functions
![Page 40: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/40.jpg)
Linear Regression (4)
Define a conjugate prior over w
Combining with likelihood function gives the posterior
where
![Page 41: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/41.jpg)
Data from straight line with Gaussian noise
First order polynomial model
Simple Example (1)
![Page 42: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/42.jpg)
Simple Example (2)
0 data points observed
Prior Data Space
![Page 43: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/43.jpg)
Simple Example (3)
1 data point observed
Likelihood Posterior Data Space
![Page 44: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/44.jpg)
Simple Example (4)
2 data points observed
Likelihood Posterior Data Space
![Page 45: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/45.jpg)
Simple Example (5)
20 data points observed
Likelihood Posterior Data Space
![Page 46: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/46.jpg)
Predictive Distribution (1)
Predict t for new values of x by integrating over w:
where
![Page 47: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/47.jpg)
Predictive Distribution (3)
Example: Sinusoidal data, 9 Gaussian basis functions, 1 data point
![Page 48: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/48.jpg)
Predictive Distribution (4)
Example: Sinusoidal data, 9 Gaussian basis functions, 2 data points
![Page 49: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/49.jpg)
Predictive Distribution (5)
Example: Sinusoidal data, 9 Gaussian basis functions, 4 data points
![Page 50: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/50.jpg)
Predictive Distribution (6)
Example: Sinusoidal data, 9 Gaussian basis functions, 25 data points
![Page 51: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/51.jpg)
Bayesian Model Comparison (1)
Alternative models Mi, i=1, …,L
Predictive distribution is a mixture
Model selection: keep only most probable model
![Page 52: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/52.jpg)
Bayesian Model Comparison (2)
From Bayes’ theorem
For equal priors, models ranked by marginal likelihood
posterior model evidence
(marginal likelihood)
prior
![Page 53: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/53.jpg)
Bayesian Model Comparison (4)
For a model with parameters w
Note that
![Page 54: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/54.jpg)
Bayesian Model Comparison (5)
Consider model with a single parameter w
![Page 55: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/55.jpg)
Bayesian Model Comparison (6)
Taking logarithms, we obtain
With M parameters, all assumed to have the same ratio , we get
Negative
![Page 56: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/56.jpg)
Linear Regression revisited
Marginal likelihood
![Page 57: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/57.jpg)
Linear Regression revisited
Noisy sinusoidal data
![Page 58: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/58.jpg)
Linear Regression revisited
Polynomial of order M ,
![Page 59: INTRODUCTION TO BAYESIAN INFERENCE PART 2mlg.eng.cam.ac.uk › mlss09 › mlss_slides › Bishop-MLSS-09-2.pdf · BAYESIAN INFERENCE –PART 2 CHRIS BISHOP. Personal Healthcare Revolution](https://reader030.vdocument.in/reader030/viewer/2022040411/5ed851e86664347bbe092342/html5/thumbnails/59.jpg)
Bayesian Model Comparison
Matching data and model complexity