Introduction to Bayesian Econometrics and Decision

Theory

Karsten T. Hansen

January 14, 2002

Lecture notes, 315 Winter 2001.

Bayesian Theory

Introduction

This note will give a short introduction to Bayesian Econometrics and Bayesian Deci-

sion Theory. Econometrics is usually taught from a classical, frequentist perspective.

However, thinking of econometric models from a Bayesian viewpoint can often be illu-

minating.

Here is a non-exhaustive list of arguments for considering a Bayesian analysis of an

econometric model (taken from Berger’s book):

1. Prior information about economically structural parameters is often

available.

In many econometric models we often have information about the underlying

unknown parameters. Often some parameter values just don’t make much sense

in terms of the underlying econometric theory (e.g., own price elasticities be-

ing positive in a demand function). A Bayesian analysis makes it very easy to

incorporate such information directly.

2. Uncertainty = Probabilities. Any conclusion derived from a statistical anal-

ysis should have attached to it an indication of the uncertainty of the conclusion.

For example, a point estimate of an unknown parameter is more or less worthless

without an indication of the uncertainty underlying the estimate.

In classical statistics one can only talk about uncertainty in a repeated sample

framework. Recall the construction of confidence intervals!

A Bayesian analysis will yield statements like “Given the observed data, I believe

with 95 percent probability that this wage elasticity is between .1 and .18”

1

3. Allows for conditioning on data. A Bayesian analysis conditions on the

observed data, where as a classical analysis averages over all possible data struc-

tures.

4. Exact distribution theory. Frequentist distribution theory of estimators of

parameters for all but the most simple non-interesting econometric models rely

on asymptotic approximations. These approximations are sometimes good, some-

times horrible. Bayesian distribution is always exact - never requiring the use of

asymptotic approximations.

5. Coherency and Rationality. It has been shown that any statistical analysis

which is not Bayesian must violate some basic ”common sense” axiom of behavior.

This is related to the fact that a Bayesian analysis is directly based on axiomatic

utility theory.

6. Bayes is optimal from a classical perspective. It has been shown in nu-

merous papers that whenever one finds a class of optimal decision rules from

a classical perspective (optimal with respect to some acceptable principle) they

usually corresponds the class of Bayes decision rules. An example is the many

complete class theorems in the literature (which roughly says that all admissible

decision rules are Bayes decision rules).

7. Operational advantage: “You always know what to do!” Researchers

are often faced with problems like ”How do I estimate the parameters of this

econometric model in a good way?”. In a Bayesian analysis you always do this

the same way - and it is usually with a good answer.

8. Computation. In the past it was often very hard to carry out a Bayesian anal-

ysis in practice due to the need of analytical integration. With the introduction

of cheap high-performance PC’s and the development of Monte Carlo statistical

methods it is now possible to estimate models with several thousand parameters.

2

This is only a partial list. A few more (technical) reasons for considering a Bayesian

approach is

• can easily accomodate inference in non-regular models,

• allows for parameter uncertainty when forming predicitions,

• can test multiple non-nested models,

• allows for automatic James-Stein Shrinkage estimation using hierarchial models.

Probability theory as logic

Probability spaces are usually introduced in the form of the Kolmogorov axioms. A

probability space (Ω,F , P ) consists of a sample space Ω, a set of events F consisting

of subsets of Ω and a probability measure P with the properties

1. F is a σ-field

2. P (A) ≥ 0, for all A ∈ F

3. P (Ω) = 1

4. For a disjoint collection Aj ∈ F,

P (∪Aj) =∑

j

P (Aj)

These are axioms and hence taken as given. The classical interpretation of the

number P (A) is the relative frequency with which A occurs in a repeated random

experiment when the number of trials go to infinity.

But why should we be base probability theory on exactly these axioms? Indeed

many have criticized these axioms as being arbitrary. Can we derive them from deeper

principles that seem less arbitrary? Yes and this also leads to an alternative interpre-

tation of the number P (A).

3

Let us start by noting that in a large part of our lives our human brains are engaged

in plausible reasoning. As an example of plausible reasoning consider the following little

story from Jaynes’ book:

“Suppose some dark night a policeman walks down a street, apparently

deserted; but suddenly he hears a burglar alarm, looks across the street,

and sees a jewelry store with a broken window. Then a gentleman wearing a

mask comes crawling out through the broken window, carrying a bag which

turns out to be full of expensive jewelry. The policeman doesn’t hesitate

at all in deciding that this gentleman is dishonest. But by what reasoning

process does he arrive at this conclusion?”

The policeman’s reasoning is clearly not deductive reasoning which is based on

relationships like

If A is true, then B is true.

Deductive reasoning is then A true =⇒ B true and B false =⇒ A false.

The policeman’s reasoning is better described by the following relationship:

If A is true, then B becomes more plausible

Plausible reasoning is then

B is true =⇒ A becomes more plausible.

How can one formalize this kind of reasoning? In chapter 1 and 2 of Jaynes’ book it

is shown that given some basic desiderata that a theory of plausible reasoning should

satisfy one can derive the laws of probability from scratch. These desiderata are that

(i) degrees of plausibility are represented by real numbers, (ii) if a conclusion can be

reasoned out in more than one way, then every possible way must lead to the same

result (plus some further weak conditions requiring correspondence of the theory to

common sense).

4

So according to this approach probability theory is not a theory about limiting rela-

tive frequencies in random experiments, but a formalization of the process of plausible

reasoning and the interpretation of a probability is

P (A) = “the degree of belief in the proposition A”

This subjective definition of probability can now be used to formalize the idea of learning

in an uncertain environment. Suppose my degree of belief in A is P (A). Then I learn

that the proposition B is true. If I believe there is some connection between A and B

I have then also learned something about A. In particular, the laws of probability (or,

according to the theory above, the laws of plausible reasoning) tells me that

Pr(A|B) =Pr(A ∩B)

Pr(B)=

Pr(B|A)Pr(A)Pr(B)

, (1)

which, of course, is known as Bayes’ rule. If there is no logical connection between A

and B then Pr(B|A) = Pr(B) and in this case Pr(A|B) = Pr(A) and I haven’t learning

anything by observing B. On the other hand, if Pr(B|A) 6= Pr(B) then B contains

information about A and therefore I must update my beliefs about A.

The Bayesian approach to statistics

The Bayesian approach to statistics is based on applying the laws of probability to

statistical inference. To see what this entails simply replace A and B above by

A = the unobserved parameter vector, θ,

B = the observed data vector, y.

Replacing probabilities with pdfs we get

p(θ|y) =p(y|θ)p(θ)

p(y)(2)

Here p(y|θ) is the sample distribution of the data given θ and p(θ) is the prior distri-

bution of θ.

5

So Bayesian statistics is nothing more than a formal model of learning in an uncer-

tain environment applied to statistical inference. The prior expresses my beliefs about

θ before observing the data;The distribution p(θ|y) expresses my updated beliefs about

θ after observing the data.

Definition 1. p(θ) is the prior distribution of θ. p(θ|y) given in (2) is the posterior

distribution of θ, and

p(y) =∫

p(y|θ)p(θ)dθ,

is the marginal distribution of the data.

Carrying out a Bayesian analysis is deceptively simple and always proceed as follows:

• Formulate the sample distribution p(y|θ) and prior p(θ).

• Compute the posterior p(θ|y) according to (2)

That’s it! All information about θ is now contained in the posterior. For example the

probability that θ ∈ A is

Pr(θ ∈ A|y) =∫

Ap(θ|y)dθ.

A couple of things to note:

• In the Bayesian approach “randomness”= uncertainty. The reason something is

random is not because it is generated by a “random experiment” but because it is

unknown. According to the Bayesian approach you are only allowed to condition

on something you know.

• Note that data and parameters are treated symmetrically. Before observing any

data both data and parameters are considered random (since they are unknown).

After observing the data only the parameters are considered random (since now

you know the data but you still don’t know the parameters).

• The final product of a Bayesian analysis is the posterior distribution of θ, p(θ|y).

This distribution summarizes your current state of knowledge about θ. Note that

6

the posterior distribution is not an estimator. An estimator is a function of the

data, θ = θ(y), which given the data y yields a single value of θ. The posterior

distribution is a distribution over the whole parameter space of θ.

While the posterior distribution is the complete representation of your beliefs about

θ, it is sometimes convenient to report a single estimate, e.g. the most likely value of

θ.

How should one can generate Bayes estimators from the posterior distribution?

Since we argued above that the Bayesian approach is just a model of learning we might

as well ask how one should make an optimal decision in an uncertain environment.

Well, as economists we know this: Maximize expected utility or, equivalently, minimize

expected loss.

So let

Y = Data sample space,

Θ = Parameter space,

A = Action space,

where a ∈ A is an action or decision related to the parameter θ, e.g. a = θ(y), an

estimator.

Definition 2. A function L : Θ×A → R with the interpretation that L(θ1, a1) is the

loss incurred if action a = a1 is taken when the parameter is θ = θ1 is called a loss

function

With these definitions we can now define the posterior expected loss of a a given

action.

Definition 3. The posterior expected loss of an action a ∈ A is

ρ(a|y) =∫

L(θ, a)p(θ|y)dθ (3)

Now we can define a Bayes estimator:

7

Definition 4. Given a sample distribution, prior and loss function, a Bayes estimator

θB(y) is any function of y so that

θB(y) = arg mina∈Aρ(a|y)

Some typical loss functions when θ is one dimensional are

L(θ, a) = (θ − a)2, quadratic, (4)

L(θ, a) = |θ − a|, absolut error, (5)

L(θ, a) =

k2(θ − a), if θ > a,

k1(a− θ), otherwise,, generalized absolut error. (6)

Then the corresponding optimal Bayes estimators are

E[θ|y], (Posterior mean), (7)

Q1/2(θ|y), (Posterior median), (8)

Qk2/(k1+k2)(θ|y), (k2/(k2 + k1) fractile of posterior ), (9)

Proof. Consider first the quadratic case. The posterior expected loss is

ρ(a|y) =∫

(θ − a)2p(θ|y)dθ,

which is a continuous and convex function of a so

∂ρ(a|y)∂a

= 0 ⇐⇒∫

(θ − a∗)p(θ|y)dθ = 0 ⇐⇒ a∗ =∫

θp(θ|y)dθ ≡ E[θ|y].

For the generalized absolut error loss case we get

ρ(a|y) =∫

L(θ, a)p(θ|y]dθ

= k1

∫ a

−∞(a− θ)p(θ|y)dθ + k2

∫ ∞

a(θ − a)p(θ|y)dθ

Now using integration by parts,∫ a

−∞(a− θ)p(θ|y)dθ = (a− a)Pr(θ < a|y)− lim

x→−∞(a− x)Pr(θ < x|y) +

∫ a

−∞Pr(θ < x|y)dx

=∫ a

−∞Pr(θ < x|y)dx,

8

and similarly for the second integral. Then

ρ(a|y) = k1

∫ a

−∞Pr(θ < x|y)dx + k2

∫ ∞

aPr(θ > x|y)dx (10)

This is a continuous convex function of a and

∂ρ(a|y)∂a

= k1Pr(θ < a|y)− k2Pr(θ > a|y).

Setting this equal to zero and solving (using that Pr(θ > a|y) = 1 − Pr(θ < a|y)) we

find

Pr(θ < a∗|y) =k2

k1 + k2, (11)

which shows that a∗ is the k2/(k1 +k2) fractile of the posterior. For k1 = k2 we get the

posterior median.

One can also construct loss functions which gives the posterior mode as an optimal

Bayes estimator.

Here is a simple example of a Bayesian analysis.

Example 1. Suppose we have a sample y of size n where by assumption yi is sampled

from a normal distribution with mean µ and variance 1,

yi|µi.i.d.∼ N(µ, 1). (12)

So the sample distribution is

p(y|µ) = (2π)−n/2σ−n exp

− 12σ2

n∑

i=1

(yi − µ)2

. (13)

Suppose we use the prior

p(µ) = (2π)−1/2σ0−1 exp

− 12σ2

0(µ− µ0)2

, (14)

where µ0 and σ20 is the known prior mean and variance. So before observing any data

the best guess of µ is µ0 (at least under squared error loss). Usually one would have

σ0 large to express that little is known about µ before observing the data.

9

The posterior distribution of µ is

p(µ|y) =p(y|µ)p(µ)

p(y)=

p(y|µ)p(µ)∫

p(y|µ)p(µ)dµ(15)

The numerator is

p(y|µ)p(µ) = (2π)−(n+1)/2σ−nσ0−1 exp

− 12σ2

n∑

i=1

(yi − µ)2 − 12σ2

0(µ− µ0)2

Note thatn

∑

i=1

(yi − µ)2 =n

∑

i=1

(yi − y)2 + n(µ− y)2

Then since1

nσ2 (µ− y)2 +

1σ2

0(µ− µ0)2 =

1σ2 (µ− µ)2 +

1σ2

0 + n−1σ2 (y − µ0)2, (16)

where

µ =(n/σ2)y + (1/σ2

0)µ0

(n/σ2) + (1/σ20)

, (17)

σ2 =1

n/σ2 + (1/σ20)

(18)

the term in curly brackets is

− 12σ2

n∑

i=1

(yi − µ)2 − 12σ2

0(µ− µ0)2 = − 1

2σ2 (µ− µ)2 − h(y),

where

h(y) =1

2σ2

n∑

i=1

(yi − y)2 +1

2(σ20 + n−1σ2)

(y − µ0)2.

Then

p(y|µ)p(µ) = p(y)(2π)−1/2σ−1 exp

− 12σ2 (µ− µ)2, (19)

where

p(y) = (2π)−n/2σ−nσσ0−1 exp

− h(y). (20)1A convenient expansion when working with the normal distribution is

a(x− b)2 + c(x− d)2 = (a + c)(

x− ab + cda + c

)2+

aca + c

(b− d)2

10

Then

p(µ|y) =p(y|µ)p(µ)

p(y)= (2π)−1/2σ−1 exp

− 12σ2 (µ− µ)2, (21)

which is the density of a normal distribution with mean µ and variance σ2 so we

conclude that

p(µ|y) = N(µ, σ2). (22)

To derive this we did more calculations than we actually had to. Remember that when

deriving the posterior for µ we only need to include terms where µ enters. Hence,

p(µ|y) =p(y|µ)p(µ)

p(y)

∝ p(y|µ)p(µ)

∝ exp

− 12σ2

n∑

i=1

(yi − µ)2 − 12σ2

0(µ− µ0)2

∝ exp

− 12σ2 (µ− µ)2.

So this quick calculation shows that

p(µ|y) ∝ exp

− 12σ2 (µ− µ)2.

We recognize this as an unnormalized normal density. So we can immediately conclude

(22).

Under squared error loss the Bayes estimator is the posterior mean,

E[µ|y] = µ =(n/σ2)y + (1/σ2

0)µ0

(n/σ2) + (1/σ20)

. (23)

The optimal Bayes estimator is a convex combination of the usual estimator y and the

prior expectation µ0. When n is large and/or σ0 is large most weight is given to y. In

particular,

E[µ|y] → y as n →∞

E[µ|y] → y as σ0 →∞.

11

In this example there was a close correspondence between the optimal Bayes esti-

mator and the classical estimator y. But suppose now we had the knowledge that µ

have to be positive. Suppose also we initially use the prior

p(µ) = I(K > µ > 0)1K

,

where K is a large positive number. Then we can compute the posterior for K < ∞

and then let K approach infinity. Then posterior is then

p(µ|y) ∝ exp

− 12σ2 (µ− µ)2I(K > µ > 0)

1K

,

where now µ = y and σ2 = σ2/n. This is an unormalized doubly truncated normal

distribution so

p(µ|y) =φ(

µ|µ, σ2)

I(K > µ > 0)

Φ(K−µ

σ

)

− Φ(−µ/σ)

→φ(

µ|µ, σ2)

I(∞ > µ > 0)Φ(µ/σ)

, as K →∞.

The posterior is a left truncated normal distribution with mean

E[µ|y] = y + σφ(y/σ)Φ(y/σ)

. (24)

Note that the unrestricted estimate is y which may be negative. Developing the

repeated sample distribution of y under the restriction µ > 0 is a tricky matter. On

the other hand, the posterior analysis is straightforward and E[µ|y] is a reasonable and

intuitive estimator of µ.

Models via exchangeability

In criticisms of Bayesian statistics one often meet statements like “This is too restrictive

since you have to use a prior to do a Bayesian analysis whereas in classical statistics

you don’t”. This is correct but now we will show that under mild conditions there

always exists a prior.

12

Consider the following example. Suppose you wish to estimate the probability of

unemployment for a group of (similar) individuals. The only information you have

is a sample y = (y1, . . . , yn) where yi is one if individual i is employed and zero if

unemployed. Clearly the indices of the observations should not matter in this case.

The joint distribution of the sample p(y1, . . . , yn) should be invariant to permutations

of the indices, i.e.,

p(y1, . . . , yn) = p(yi(1), . . . , yi(n)),

where i(1), . . . , i(n) is a permutation of 1, . . . , n. Such a condition is called ex-

changeability.

Definition 5. A finite set of random quantities z1, . . . , zn are said to be exchangeable if

every permutation of z1, . . . , zn has the same joint distribution as every other permuta-

tion. An infinite collection is exchangeable if every finite subcollection is exchangeable.

The relatively weak assumption of exchangeability turns out to have a profound

consequence as shown by a famous theorem by deFinetti.

Theorem 1. deFinetti’s representation theorem Let z1, . . . be a sequence of 0-1

random quantities. The sequence (z1, . . . , zn) is exchangeable for every n if and only if

p(z1, . . . , zn) =∫ 1

0

(n

∏

i=1

θzi(1− θ)1−zi

)

dF (θ),

where

F (θ) = limn→∞

Pr( 1

n

n∑

i=1

zi ≤ θ)

What does deFinetti’s theorem say? It says that if the sequence z1, z2, . . . is con-

sidered exchangeable then it is as if the zi’s are iid Bernoulli given θ,

zi|θi.i.d.∼ Bernoulli(θ), i = 1, . . . , n,

where θ is a random variable with a distribution which is the limit distribution of the

sample average n−1 ∑ni=1 zi.

13

So one way of defending a model like

zi|θi.i.d.∼ Bernoulli(θ), i = 1, . . . , n, (25)

θ ∼ π(θ), (26)

is to appeal to exchangeability and think about your beliefs about the limit of n−1 ∑ni=1 zi

when you pick the prior π(θ).

DeFinitti’s theorem can be generalized to sequences of continuous random variables,

see the book by Schervish for theorems and proofs.

Frequentist properties of Bayes procedures

It is often of interest to evaluate Bayes estimators from a classical frequentist perspec-

tive.

Consider first the issue of consistency. Suppose we have computed the posterior

distribution of the parameter vector θ,

p(θ|y) =p(y|θ)p(θ)

p(y).

Suppose now we also make the assumption that there is a population distribution f(y).

As a measure of the difference between the sample distribution used to compute the

posterior p(y|θ) and the actual population distribution we can use the Kullback-Leibler

discrepency,

H(θ) =∫

log(p(yi|θ)

f(yi)

)

f(yi)dyi. (27)

Let θ∗ be the value of θ that minimizes this distance. One can show that if f(yi) =

p(yi|θ0), i.e. the sample distribution is correctly specified and the population is indexed

by some true value θ0, then θ∗ = θ0. Then2

Theorem 2. If the parameter space Θ is compact and A is a neighborhood of θ0 with

nonzero prior probability, then

p(θ ∈ A|y) → 1 as n →∞.2For proofs see the textbook by Schervish or Gelman et.al. on the reading list.

14

So in the case of correct specification, f(yi) = p(yi|θ0), the posterior will concentrate

around the true value θ0 asymptotically as long as θ0 is contained in the support of

the prior. Under misspecification the posterior will concentrate around the value of θ0

that minimizes the distance to the true model.

Now we shall consider the frequentist risk properties of Bayes estimators. To this

end we shall first define the frequentist risk of an estimator θ(y). This is

r(θ, θ) =∫

L(θ, θ(y))p(y|θ)dy. (28)

Note the difference between this risk measure and the Bayesian risk measure (3): The

frequentist risk averages over the data for a given parameter θ whereas the Bayesian

risk measure averages over the parameter space given the data. Furthermore, the

frequentist risk is a function of both θ and the proposed estimator θ. The Bayes risk

is only a function of a = θ.

There are two popular ways to choose estimators optimally based on their fre-

quentist risk. These are minimaxity and admissibility It turns out that there is close

relationship between admissibility and Bayes estimators.

Definition 6. An estimator θ is inadmissible if there exists an estimator θ1 which

dominates θ, i.e., such that for every θ,

r(θ, θ) ≥ r(θ, θ1),

and, for at least one value θ0,

r(θ0, θ) > r(θ0, θ1).

If θ is not inadmissible it is admissible.

The idea behing admissibility is to reduce the number of potential estimators to

consider. Indeed is seems hard to defend using an inadmissible estimator.

Under mild conditions Bayes estimators can be shown to be admissible. Under

somewhat stronger conditions one can in fact show the reverse: All admissible estima-

15

tors are Bayes estimators (or limits of Bayes estimators). A theorem proving this is

called a complete class theorem and different versions of complete class theorems exist.3

Comparisons between classical and Bayesian inference

The fundamental difference between classical frequentist inference and Bayesian infer-

ence is in the use of pre-data versus post-data probability statements.

The frequentist approach is limited to pre-data considerations. This approach an-

swers questions of the following form:

(Q1) Before we have seen the data, what data do we expect to get?

(Q2) If we use the as yet unknown data to estimate parameters by some known algo-

rithm, how accurate do we expect the estimates to be?

(Q3) If the hypothesis being tested is in fact true, what is the probability that we shall

get data indicating that it is true?

These questions can also be answered in the Bayesian approach. However, followers

of the Bayesian approach argue that these questions are not relevant for scientific

inference. What is relevant are post-data questions:

(Q1’) After having seen the data, do we have any reason to be surprised by them?

(Q2’) After we have seen the data, what parameter estimates can we now make, and

what accuracy are we entitled to claim?

(Q3) What is the probability conditional on the data, that the hypothesis is true?

Questions (Q1’)-(Q3’) are only meaningful in a Bayesian framework.

In the frequentist approach one cannot talk about the probability of a hypothesis.

The marginal propensity to consume is either .92 or not. A frequentist 95 pct. confi-

dence interval (a, b) does not mean that the probability of a < θ < b is 95 pct. θ either

belongs to the interval (a, b) or not.3For more about this see the book by Berger.

16

Sometimes frequentist and Bayesian procedures give similar results although their

interpretation differ.

Example 2. In example 1 we found the posterior,

p(µ|y) = N(µ, σ2),

where

µ =(n/σ2)y + (1/σ2

0)µ0

(n/σ2) + (1/σ20)

,

σ2 =1

n/σ2 + (1/σ20)

Suppose we look at the “limit prior”, σ0 →∞4 Then

p(µ|y) = N(y, σ2/n),

and the Bayes estimate under squared error loss plus/minus one posterior standard

deviation is

µB = y ± σ√n

. (29)

On the other hand, the repeated sampled distribution of µ = y is

p(y|µ) = N(µ, σ2/n),

and the estimate plus minus one standard deviation of the repeated sample distribution

is

y = µ± σ√n

. (30)

Conceptually (29) and (30) are very different, but the final statements one would make

about µ would be nearly identical.

We stress once again the difference between (30) and (29). (30) answers the question

(Q1) How much would the estimate of µ vary over the class of all data sets that we

might conceivably get?4This is a special case of a “non-informative” prior. We will discuss these later.

17

whereas (29) answers the question

(Q2) How accurately is the value of µ determined by the one data set that we actually

have?

The Bayesian camp has often critized the fact that the frequentist approach takes

data that could have been observed but wasn’t into account when conducting inference

about the parameter vector. Here is a famous example from Berger’s book that shows

why this sometimes can be a relevant critique.

Example 3. Suppose a substance to be analyzed can be sent either to either lab 1 or

lab 2. The two labs seem equally good so a fair coin is flipped to choose between them.

The coin flip results in lab 1. A week later the results come back from lab 1 and a

conclusion is to be made. Should this conclusion take into account the fact that the

coin could have pointed to lab 2 instead? Common sense says no, but according to the

frequentist principle we have to average over all possible samples including the ones

from lab 2.

Here is another often quoted example that shows this also affects testing.

Example 4. Suppose in 12 independent tosses of a coin you observe 9 heads and 3 tails.

Let θ =probability of heads. You wish to test H0 : θ = 1/2 vs. H1 : θ > 1/2. Given

that this is all the information you have, there are two candidates for the likelihood

function:

(1) Binomial. The number n = 12 was fixed beforehand and the random quantity X

was the number of heads observed in n tosses. Then X ∼ Bin(12, θ) and

L1(θ) =(

129

)

θ9(1− θ)3.

(2) Negative binomial. The coint was flipped until the third head appeared. Then

the random component is X =the number of flips required to complete the ex-

periment, so X ∼ NegBin(3, θ) and

L2(θ) =(

119

)

θ9(1− θ)3.

18

Suppose we use the test statistic X =number of heads and decision rule “reject H0 if

X ≥ c”. The p-value is the probability of observing the data X = 9 or something more

extreme under H0. This

α1 = Pr(X ≥ 9|θ = 1/2) =1

∑

j=9

2(

12j

)

(1/2)j(1/2)12−j = .075,

α2 = Pr(X ≥ 9|θ = 1/2) =∞

∑

j=9

(

2 + jj

)

(1/2)j(1/2)3 = .0325.

So using a conventional Type I error level α = .05 the two model assumptions lead to

two different conclusions. But there is nothing in the situation that tells us which of

the two models we should use.

What happens here is that the Neyman-Pearson test procedure allows unobserved

outcomes to effect the results. X values more extreme than 9 was used as evidence

against the null. The prominent Bayesian Harold Jeffreys described this situation as “a

hypothesis that may be true may be rejected because is has not predicted observable

results that have not occurred”.

There is also an important difference between frequentist and Bayesian approaches

to the elimination of nuisance parameters. In the frequentist approach nuisance pa-

rameters are usually eliminated by the plug-in method. Suppose we have an estimator

θ1 of a parameter θ1 which depends on another parameter θ2:

θ1 = θ1(y, θ2).

Typically one would get rid of the dependence of θ2 by plugging in an estimate of θ2:

ˆ1θ = θ1(y, θ2(y)).

In the Bayesian approach one gets rid of nuisance parameters by integration. Suppose

the joint posterior distribution of θ1 and θ2 is p(θ1, θ2|y). Inference about θ1 is then

based on the marginal posterior

p(θ1|y) =∫

p(θ1, θ2|y)dθ2.

19

Note that we can rewrite this integration as

p(θ1|y) =∫

p(θ1, θ2|y)dθ2

=∫

p(θ1|θ2, y)p(θ2|y)dθ2,

so instead of plugging in a single value of θ2 we average over all possible values of θ2

by integrating the conditional posterior of θ1 given θ2 w.r.t. the marginal posterior for

θ2.

20

Bayesian Mechanics

The normal linear regression model

The sampling distribution of the n vector of observable data y is

p(y|X, θ) = N(y|Xβ, τ−1In), (31)

where X is n× k with rank k and θ = (β, τ). Note that the covariance matrix τ−1In is

formulated in terms of the precision τ of the observations. The precision is the inverse

of the variance.

We need a prior for θ. There are two popular choices.

p(β) = N(β|β0, Λ−10 ),

p(τ) = G(τ |α1, α2)

and

p(β, τ) ∝ τ−1 (32)

The first prior specifies that β and τ are prior independent with β having a mul-

tivariate normal prior with mean β0 and covariance Λ−10 and τ having a gamma prior

with shape parameter α1 and inverse scale parameter α2 (We could also have chosen

to work with the variance σ2 = 1/τ . The implied prior on σ2 will be an inverse gamma

distribution).

The second prior is a “non-informative” prior. This is a prior that you may want to

use if you don’t have much prior information about θ available (you may be wondering

why τ−1 represents a “non-informative” prior on τ . This will become clearer below).

Consider the second prior first. The posterior distribution of θ is

p(θ|y) ∝ τn/2−1 exp

− τ2(y −Xβ)′(y −Xβ)

= τn/2−1 exp

− τ2(

(y −Xβ)′(y −Xβ) + (β − β)′X ′X(β − β))

, (33)

where β = (X ′X)−1X ′y.

21

If we are interested primarily in β we can integrate out the nuisance parameter τ

to get the marginal posterior of β. Letting s(y) = (y −Xβ)′(y −Xβ) we get

p(β|y) =∫

p(θ|y)dτ

∝∫

τn/2−1 exp

− τ2(

s(y) + (β − β)′X ′X(β − β))

dτ (34)

∝(

1 +1

s(y)(β − β)X ′X(β − β)

)−n/2, (35)

which is the kernel of a multivariate t distribution,

p(β|y) = tn−k(

β|β, Σ)

, (36)

with n− k degrees of freedom, mean β and scale matrix

Σ =s(y)n− k

(X ′X)−1.

Note that this is exactly equivalent to the repeated sample distribution of β.

We can also derive the marginal posterior of τ . From (33) we get

p(τ |y) =∫

p(θ|y)dβ

∝∫

τn/2−1 exp

− τ2(

s(y) + (β − β)′X ′X(β − β))

dβ

∝ τn/2−1 exp

− τ2s(y)

∫

exp

− τ2(β − β)′X ′X(β − β)

)

dβ (37)

∝ τ (n−k)/2−1 exp

− τ2s(y)

, (38)

which we recognize as the kernel of a gamma distribution with shape parameter (n−

k)/2 and inverse scale s(y)/2,

p(τ |y) = G(

τ |n− k2

,s(y)2

)

. (39)

Note that the mean of this distribution is

E[τ |y] =(n− k)

s(y)=

1σ2 ,

where σ2 = s(y)/(n− k).

22

Now we can see one way the prior p(β, τ) ∝ τ−1 may be considered “non-informative”:

The marginal posterior distributions have properties closely resembling the correspond-

ing repeated sample distributions.

For the first prior we get

p(θ|y) ∝ τn/2 exp

− τ2(y −Xβ)′(y −Xβ)

×

exp

− 12(β − β0)′Λ0(β − β0)

τα1−1 exp−α2τ (40)

This can be rewritten as

p(θ|y) ∝ τn/2+α1−1 exp

− τ2(

s(y) + (β − β)′X ′X(β − β))

×

exp

− 12(β − β0)′Λ0(β − β0)

exp−α2τ (41)

This joint posterior of θ does not lead to convenient expressions for the marginals of β

and τ .

We can, however, derive analytical expressions for the conditional posteriors p(β|τ, y)

and p(τ |β, y). These conditional posteriors turn out to play a fundamental role when

designing simulation algorithms.

Let us first derive the conditional posterior for β given τ . Remember that we then

only need to include terms containing β. Then we get

p(β|τ, y) ∝ exp

− 12(

(β − β)′τX ′X(β − β) + (β − β0)′Λ0(β − β0))

(42)

Now we can use the following convenient expansion5:

Lemma 1. Let z, a, b be k vectors and A,B be symmetric k × k matrices such that

(A + B)−1 exists. Then

(z−a)′A(z−a)+(z−b)′B(z−b) = (z−c)′(A+B)(z−c)+(a−b)′A(A+B)−1B(a−b),

where c = (A + B)−1(Aa + Bb)5For a proof see Box and Tiao (1973), p.418.

23

Applying lemma 1 it follows that

p(β|τ, y) ∝ exp

− 12(β − β)′Σ−1(β − β)

,

where

Σ = (τX ′X + Λ0)−1,

β = Σ−1(τX ′y + Λ0β0).

So

p(β|τ, y) = N(β|β, Σ), (43)

a multivariate normal distribution.

Similarly, from (40) we get

p(τ |β, y) ∝ τn/2+α1−1 exp−τ2(y −Xβ)′(y −Xβ)− τα2,

which is the kernel of a gamma distribution,

p(τ |β, y) = G(

τ |n2

+ α1,(y −Xβ)′(y −Xβ)

2+ α2

)

(44)

We will later on see how it is extremely easy to simulate draws from p(β|y) and

p(τ |y) using these conditional distributions.

The SURE model

Consider now the model

yij = x′ijβj + εij, i = 1, . . . , n; j = 1, . . . , J, (45)

where εi = (εi1, . . . , εiJ) is assumed jointly normal,

εi|Λ ∼ N(0, Λ−1).

We can rewrite this model as

Yi = Xiβ + εi, i = 1, . . . , n. (46)

24

where Yi = (yi1, . . . , yiJ)′ and

Xi =

xi1 0 · · · 0

0 xi2 · · · 0...

... . . . 0

0 0 · · · xiJ

, β =

β1

β2...

βJ

We need to specify a prior on θ = (β, Λ). Again we can consider a non-informative

and an informative prior. The usual non-informative prior for this model is

p(β, Λ) ∝ |Λ|−(J+1)/2. (47)

Alternatively, one can use

p(β) = N(β|β0, Λ−10 ),

p(Λ) = W(Λ|ν, S).

The prior for β is a multivariate normal distribution as before. The Λ prior is a Wishart

distribution. This is the multivariate generalization of the gamma distribution. The

Wishart distribution has mean

E[Λ] = νS.

The posterior under the first prior is

p(β, Λ|y) ∝ |Λ|−(J+1)/2|Λ|n/2 exp(

− 12

n∑

i=1

(Yi −Xiβ)′Λ(Yi −Xiβ))

= |Λ|(n−J−1)/2 exp(

− 12

n∑

i=1

(Yi −Xiβ)′Λ(Yi −Xiβ))

(48)

Using the well known result

n∑

i=1

(Yi −Xiβ)′Λ(Yi −Xiβ) =n

∑

i=1

(Yi −Xiβ(Λ))′Λ(Yi −Xiβ(Λ))+

(β − β(Λ))′n

∑

i=1

X ′iΛXi(β − β(Λ)), (49)

25

where

β(Λ) =(

n∑

i=1

X ′iΛXi

)−1n

∑

i=1

X ′iΛYi,

we find the conditional posterior

p(β|Λ, y) ∝ exp(

− 12(β − β(Λ))′

n∑

i=1

X ′iΛXi(β − β(Λ)

)

(50)

So

p(β|Λ, y) = N(

β|β(Λ),(

n∑

i=1

X ′iΛXi

)−1)

. (51)

The conditional posterior of β is normal with mean equal to the efficient GLS estimator

(when Λ is known).

To get the conditional posterior for Λ note that

n∑

i=1

(Yi −Xiβ)′Λ(Yi −Xiβ) = Tr

ΛM(β)

,

where

M(β) =n

∑

i=1

(Yi −Xiβ)(Yi −Xiβ)′.

From (48) we then get

p(Λ|β, y) ∝ |Λ|(n−J−1)/2 exp(

− 12Tr

ΛM(β))

. (52)

This is the kernel of a Wishart distribution,

p(Λ|β, y) = W(

n,M(β)−1) (53)

Note that the posterior of mean of Λ given β is

E[Λ|β, y] = nM(β)−1.

The inverse of this is n−1M(β) which is the usual estimate of the covariance matrix if

β is known.

26

Next let’s derive the conditional posteriors under the proper prior distributions.

The joint posterior is

p(β, Λ|y) ∝ |Λ|ν−J−1)/2 exp(

− 12Tr

ΛS−1)

exp(

− 12(β − β0)′Λ0(β − β0)

)

×

|Λ|n/2 exp(

− 12

n∑

i=1

(Yi −Xiβ)′Λ(Yi −Xiβ))

(54)

The conditional for β is then

p(β|Λ, y) ∝ exp(

− 12(β − β0)′Λ0(β − β0)

)

exp(

− 12

n∑

i=1

(Yi −Xiβ)′Λ(Yi −Xiβ))

∝ exp(

− 12(β − β0)′Λ0(β − β0)

)

exp(

− 12(β − β(Λ))′(

n∑

i=1

X ′iΛXi)(β − β(Λ)

)

Application of lemma 1 then gives

p(β|Λ, y) = N(

β|β(Λ), Σ(Λ))

, (55)

where

Σ(Λ) =(

Λ0 +n

∑

i=1

X ′iΛXi

)−1, (56)

β(Λ) = Σ(Λ)(

n∑

i=1

X ′iΛYi + Λ0β0

)

. (57)

The conditional for Λ is

p(Λ|β, y) ∝ |Λ|(n+ν−J−1)/2 exp(

− 12Tr

ΛS−1− 12

n∑

i=1

(Yi −Xiβ)′Λ(Yi −Xiβ))

∝ |Λ|(n+ν−J−1)/2 exp(

− 12Tr

ΛS−1− 12Tr

ΛM(β)

∝ |Λ|(n+ν−J−1)/2 exp(

− 12Tr

Λ(

S−1 + M(β))

)

,

which is the kernel of a Wishart distribution,

p(Λ|β, y) = W(

Λ∣

∣n + ν,(

S−1 + M(β))−1

)

(58)

27

Readings

• Bayesian foundations and philosophy

– Jaynes, E.T., (1994), “Probability: The logic of science”, unpublished book.

Chapters may be downloaded from http://bayes.wustl.edu/etj/prob.html

– Jeffreys, H., (1961), “Theory of Probability”, Oxford University Press.

• Bayesian statistics and econometrics

– Gelman, A., Carlin, J.B., Stern, H.S., and Rubin, D.B. (1995), “Bayesian

Data Analysis”, Chapman and Hall.

– Schervish, M.J. (1995) “Theory of Statistics”, Springer.

– Zellner, A. (1971) “An Introduction to Bayesian Inference in Econometrics”,

Wiley.

• Bayesian statistics and Decision Theory

– Berger, J.O., (1985), “Statistical Decision Theory and Bayesian Analysis”,

Springer

– Robert, C.P., (1994), “The Bayesian Choice”, Springer.

28