christopher m. bishop pattern recognition and machine learningskirshne/teaching/... · the beta...
TRANSCRIPT
![Page 1: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/1.jpg)
Christopher M. Bishop
PATTERN RECOGNITION AND MACHINE LEARNING
![Page 2: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/2.jpg)
Polynomial Curve Fitting
![Page 3: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/3.jpg)
Sum-of-Squares Error Function
![Page 4: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/4.jpg)
0th Order Polynomial
![Page 5: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/5.jpg)
1st Order Polynomial
![Page 6: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/6.jpg)
3rd Order Polynomial
![Page 7: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/7.jpg)
9th Order Polynomial
![Page 8: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/8.jpg)
Over-fitting
Root-Mean-Square (RMS) Error:
![Page 9: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/9.jpg)
Polynomial Coefficients
![Page 10: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/10.jpg)
Data Set Size:
9th Order Polynomial
![Page 11: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/11.jpg)
Data Set Size:
9th Order Polynomial
![Page 12: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/12.jpg)
Regularization
Penalize large coefficient values
![Page 13: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/13.jpg)
Regularization:
![Page 14: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/14.jpg)
Regularization:
![Page 15: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/15.jpg)
Regularization: vs.
![Page 16: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/16.jpg)
Polynomial Coefficients
![Page 17: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/17.jpg)
The Gaussian Distribution
![Page 18: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/18.jpg)
Gaussian Parameter Estimation
Likelihood function
![Page 19: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/19.jpg)
Maximum (Log) Likelihood
![Page 20: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/20.jpg)
Properties of and
![Page 21: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/21.jpg)
Curve Fitting Re-visited
![Page 22: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/22.jpg)
Maximum Likelihood
Determine by minimizing sum-of-squares error, .
![Page 23: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/23.jpg)
Predictive Distribution
![Page 24: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/24.jpg)
MAP: A Step towards Bayes
Determine by minimizing regularized sum-of-squares error, .
![Page 25: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/25.jpg)
Bayesian Curve Fitting
![Page 26: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/26.jpg)
Bayesian Predictive Distribution
![Page 27: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/27.jpg)
Model Selection
Cross-Validation
![Page 28: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/28.jpg)
Parametric Distributions
Basic building blocks:
Need to determine given
Representation: or ?
Recall Curve Fitting
![Page 29: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/29.jpg)
Binary Variables (1)
Coin flipping: heads=1, tails=0
Bernoulli Distribution
![Page 30: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/30.jpg)
Binary Variables (2)
N coin flips:
Binomial Distribution
![Page 31: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/31.jpg)
Binomial Distribution
![Page 32: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/32.jpg)
Parameter Estimation (1)
ML for BernoulliGiven:
![Page 33: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/33.jpg)
Parameter Estimation (2)
Example:
Prediction: all future tosses will land heads up
Overfitting to D
![Page 34: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/34.jpg)
Beta Distribution
Distribution over .
![Page 35: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/35.jpg)
Bayesian Bernoulli
The Beta distribution provides the conjugate prior for the Bernoulli distribution.
![Page 36: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/36.jpg)
Beta Distribution
![Page 37: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/37.jpg)
Prior ∙ Likelihood = Posterior
![Page 38: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/38.jpg)
Properties of the Posterior
As the size of the data set, N , increase
![Page 39: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/39.jpg)
Prediction under the Posterior
What is the probability that the next coin toss will land heads up?
![Page 40: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/40.jpg)
Multinomial Variables
1-of-K coding scheme:
![Page 41: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/41.jpg)
ML Parameter estimation
Given:
Ensure , use a Lagrange multiplier, ¸.
![Page 42: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/42.jpg)
The Multinomial Distribution
![Page 43: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/43.jpg)
The Dirichlet Distribution
Conjugate prior for the multinomial distribution.
![Page 44: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/44.jpg)
Bayesian Multinomial (1)
![Page 45: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/45.jpg)
Bayesian Multinomial (2)
![Page 46: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/46.jpg)
The Gaussian Distribution
![Page 47: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/47.jpg)
Maximum Likelihood for the Gaussian (1)
Given i.i.d. data , the log likeli-hood function is given by
Sufficient statistics
![Page 48: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/48.jpg)
Maximum Likelihood for the Gaussian (2)
Set the derivative of the log likelihood function to zero,
and solve to obtain
Similarly
![Page 49: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/49.jpg)
Maximum Likelihood for the Gaussian (3)
Under the true distribution
Hence define
![Page 50: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/50.jpg)
Bayesian Inference for the Gaussian (1)
Assume ¾2 is known. Given i.i.d. data, the likelihood function for
¹ is given by
This has a Gaussian shape as a function of ¹ (but it is not a distribution over ¹).
![Page 51: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/51.jpg)
Bayesian Inference for the Gaussian (2)
Combined with a Gaussian prior over ¹,
this gives the posterior
Completing the square over ¹, we see that
![Page 52: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/52.jpg)
Bayesian Inference for the Gaussian (3)
… where
Note:
![Page 53: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/53.jpg)
Bayesian Inference for the Gaussian (4)
Example: for N = 0, 1, 2 and 10.
![Page 54: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/54.jpg)
Bayesian Inference for the Gaussian (5)
Sequential Estimation
The posterior obtained after observing N { 1data points becomes the prior when we observe the N th data point.
![Page 55: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/55.jpg)
Bayesian Inference for the Gaussian (6)
Now assume ¹ is known. The likelihood function for ̧ = 1/¾2 is given by
This has a Gamma shape as a function of ¸.
![Page 56: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/56.jpg)
Bayesian Inference for the Gaussian (7)
The Gamma distribution
![Page 57: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/57.jpg)
Bayesian Inference for the Gaussian (8)
Now we combine a Gamma prior, ,with the likelihood function for ¸ to obtain
which we recognize as with
![Page 58: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/58.jpg)
Bayesian Inference for the Gaussian (9)
If both ¹ and ¸ are unknown, the joint likelihood function is given by
We need a prior with the same functional dependence on ¹ and ¸.
![Page 59: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/59.jpg)
Bayesian Inference for the Gaussian (10)
The Gaussian-gamma distribution
• Quadratic in ¹.• Linear in ¸.
• Gamma distribution over ¸.• Independent of ¹.
![Page 60: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/60.jpg)
Bayesian Inference for the Gaussian (11)
The Gaussian-gamma distribution
![Page 61: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/61.jpg)
Bayesian Inference for the Gaussian (12)
Multivariate conjugate priors• ¹ unknown, ¤ known: p(¹) Gaussian.
• ¤ unknown, ¹ known: p(¤) Wishart,
• ¤ and ¹ unknown: p(¹,¤) Gaussian-Wishart,
![Page 62: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/62.jpg)
where
Infinite mixture of Gaussians.
Student’s t-Distribution
![Page 63: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/63.jpg)
Student’s t-Distribution
![Page 64: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/64.jpg)
Student’s t-Distribution
Robustness to outliers: Gaussian vs t-distribution.
![Page 65: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/65.jpg)
Student’s t-Distribution
The D-variate case:
where .
Properties:
![Page 66: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/66.jpg)
The Exponential Family (1)
where ´ is the natural parameter and
so g(´) can be interpreted as a normalization coefficient.
![Page 67: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/67.jpg)
The Exponential Family (2.1)
The Bernoulli Distribution
Comparing with the general form we see that
and so
Logistic sigmoid
![Page 68: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/68.jpg)
The Exponential Family (2.2)
The Bernoulli distribution can hence be written as
where
![Page 69: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/69.jpg)
The Exponential Family (3.1)
The Multinomial Distribution
where, , and
NOTE: The ´k parameters are not independent since the corresponding ¹k must satisfy
![Page 70: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/70.jpg)
The Exponential Family (3.2)
Let . This leads to
and
Here the ´k parameters are independent. Note that
and
Softmax
![Page 71: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/71.jpg)
The Exponential Family (3.3)
The Multinomial distribution can then be written as
where
![Page 72: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/72.jpg)
The Exponential Family (4)
The Gaussian Distribution
where
![Page 73: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/73.jpg)
ML for the Exponential Family (1)
From the definition of g(´) we get
Thus
![Page 74: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/74.jpg)
ML for the Exponential Family (2)
Give a data set, , the likelihood function is given by
Thus we have
Sufficient statistic
![Page 75: Christopher M. Bishop PATTERN RECOGNITION AND MACHINE LEARNINGskirshne/teaching/... · The Beta distribution provides the . conjugate. prior for the Bernoulli distribution](https://reader030.vdocument.in/reader030/viewer/2022041021/5ed0fe2bd3b82946a13ab7b3/html5/thumbnails/75.jpg)
Conjugate priors
For any member of the exponential family, there exists a prior
Combining with the likelihood function, we get
Prior corresponds to º pseudo-observations with value Â.