1

Bayesian methods for parameter estimation and data assimilation with crop models

Part 2: Likelihood function and prior distribution

David Makowski and Daniel Wallach

INRA, France

October 2006

2

• Notions in probability - Joint probability

- Conditional probability

- Marginal probability

• Bayes’ theorem

Previously

PyP

PyPyP

3

Objectives of part 2

• Introduce the notion of prior distribution.

• Introduce the notion of likelihood function.

• Show how to estimate parameters with a Bayesian method.

4

Part 2: Likelihood function and prior distributions

Estimation of parameters ()

Parameter = numerical value not calculated by the model and not observed.

Information available to estimate parameters

- A set of observations (y).

- Prior knowledge about parameter values.

5

Part 2: Likelihood function and prior distributions

Two distributions in Bayes’ theorem

• Likelihood function = function relating data to parameters.( )P y

• Prior parameter distribution = probability distribution describing our initial knowledge about parameter values.

( )P

6

Part 2: Likelihood function and prior distributions

Measurements

Prior Information about parameter values

Bayesian methodCombined info about parameters

7

ExampleEstimation of crop yield θ by combining a measurement with expert knowledge.

Measurement y = 9 t/ha ± 1

about 5 t/ha ± 2

Expert

Field with unknown yield

Part 2: Likelihood function and prior distributions

Plot

8

ExampleEstimation of crop yield θ by combining a measurement with expert knowledge.

Part 2: Likelihood function and prior distributions

• One parameter to estimate: the crop yield θ.

• Two types of information available:

- A measurement equal to 9 t/ha with a standard error equal to 1 t/ha.

- An estimation provided by an expert equal to 5 t/ha with a

standard error equal to 2 t/ha.

9

Part 2: Likelihood function and prior distributions

Prior distribution

• It describes our belief about the parameter values before we observe the measurements.

• It is based on past studies, expert knowledge, and litterature.

10

Part 2: Likelihood function and prior distributions

Example (continued)

Definition of a prior distribution

θ ~ N( µ, ² )

42

5exp42

1

2exp

2

1 2

2

2

2

P

• Normal probability distribution.

• Expected value equal to 5 t/ha.

• Standard error equal to 2 t/ha

11

Part 2: Likelihood function and prior distributions

Example (continued)

Plot of the prior distribution

0 2 4 6 8 10

Theta (t/ha)

0.0

0.1

0.2

0.3

0.4

0.5

Den

sity

12

Part 2: Likelihood function and prior distributions

Likelihood function

• A likelihood function is a function relating data to parameters.

• It is equal to the probability that the measurements would have been observed given some parameter values.

• Notation: P(y | θ)

13

Part 2: Likelihood function and prior distributions

Example (continued)

Statistical model

y | θ ~ N( θ, σ² )

y = θ + with ~ N( 0, σ² )

14

Part 2: Likelihood function and prior distributions

Example (continued)

Definition of a likelihood function

• Normal probability distribution.

• Measurement y assumed unbiaised and equal to 9 t/ha.

• Standard error σ assumed equal to 1 t/ha

y | θ ~ N( θ, σ² )

2

9exp

2

1

2exp

2

1 2

2

2

2

yyP

15

Part 2: Likelihood function and prior distributions

Example (continued)

Definition of a likelihood function

4 6 8 10 12

Theta (t/ha)

0.0

0.1

0.2

0.3

0.4

De

nsi

ty

Maximum likelihood estimate

16

Part 2: Likelihood function and prior distributions

Maximum likelihood

Likelihood functions are also used by frequentist to implement the maximum likelihood method.

The maximum likelihood estimator is the value of θ maximizing P(y | θ) .

17

0 2 4 6 8 10 12

Theta (t/ha)

0.0

0.1

0.2

0.3

0.4

De

nsi

ty

Part 2: Likelihood function and prior distributions

Likelihood functionPrior probability distribution

18

Part 2: Likelihood function and prior distributions

Example (continued)

Analytical expression of the posterior distribution

θ | y ~ N( µpost, post² )

1 8.2post B B y

8.01 22 Bpost

5

422

2

B

19

Part 2: Likelihood function and prior distributions

0 2 4 6 8 10 12

Theta (t/ha)

0.0

0.1

0.2

0.3

0.4

0.5

De

nsi

ty

Prior probability distribution Likelihood function

Posterior probability distribution

20

1. Result is a probability distribution (posterior distr.)

2. Posterior mean is intermediate between prior mean and observation.

3. Weight of each depends on prior variance and measurement error.

4. Posterior variance is lower than both prior variance and measurement error variance.

5. Used just one data point and still got estimator.

Part 2: Likelihood function and prior distributions

Example (continued)

Discussion of the posterior distribution

21

Frequentist versus Bayesian

Part 2: Likelihood function and prior distributions

Bayesian analysis introduces an element of subjectivity: the prior distribution.

But its representation of the uncertainty is easy to understand

- the uncertainty is assessed conditionally to the observations,

- the calculations are straightforward when the posterior distribution is known.

22

Part 2: Likelihood function and prior distributions

Which is better?

Bayesian methods often lead to

- more realistic estimated parameter values,

- in some cases, more accurate model predictions.

Problems when prior information is wrong and when one has a strong confidence in it.

23

Part 2: Likelihood function and prior distributions

Difficulties for estimating crop model parameters

Which likelihood function ?

- Unbiaised errors ?

- Independent errors ?

Which prior distribution ?

- What do the parameters really represent ?

- Level of uncertainty ?

- Symmetric distribution ?

24

Part 2: Likelihood function and prior distributions

Practical considerations

• The analytical expression of the posterior distribution can be derived for simple applications.

• For complex problems, the posterior distribution must be approximated.

25

Next part

Importance sampling, an algorithm to approximate the posterior probability distribution.