workshop on bayesian statistics for education research › cemo › dokumenter › kaplan... ·...

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Workshop on Bayesian Statistics for EducationResearch

David Kaplan

Department of Educational Psychology

19 – 21 January, 2015CEMO, Oslo, Norway

1 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

The material for this workshop is drawn from Kaplan, D (2014),Bayesian Statistics for the Social Sciences. New York, GuilfordPress.

c©David Kaplan, 2015

2 / 165

Day 1 AMBayes’Theorem

TheAssumption ofExchangeabil-ity

The PriorDistribution

BayesianHypothesisTesting

PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Day 1 AM: Introduction to Workshop andBayesian Theory

.[I]t is clear that it is not possible to think about learning fromexperience and acting on it without coming to terms withBayes’ theorem. - Jerome Cornfield

3 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Bayesian statistics has long been overlooked in the quantitativemethods training of social scientists.

Typically, the only introduction that a student might have toBayesian ideas is a brief overview of Bayes’ theorem whilestudying probability in an introductory statistics class.

1 Until recently, it was not feasible to conduct statistical modelingfrom a Bayesian perspective owing to its complexity and lack ofavailability.

2 Bayesian statistics represents a powerful alternative to frequentist(classical) statistics, and is therefore, controversial.

Bayesian statistical methods are now more popular owing to thedevelopment of powerful statistical software tools that make theestimation of complex models feasible from a Bayesianperspective.

4 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Day 1AM: Major differences between the Bayesian and frequentistparadigms of statistics; Bayes’ theorem; Bayesian hypothesistesting

PM: MCMC; Bayesian computation with R and rjags

Day 2AM: Bayesian model building; evaluation; Bayesian linearregression

PM: Student analyses – Bayesian regression analysis

Day 3AM: Advanced Topics: HLM, Factor analysis (time permitting)

PM: Wrap-up: Student analyses

5 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Paradigm Differences

For frequentists, the basic idea is that probability is representedby the model of long run frequency.

Frequentist probability underlies the Fisher andNeyman-Pearson schools of statistics – the conventionalmethods of statistics we most often use.

The frequentist formulation rests on the idea of equally probableand stochastically independent events

The physical representation is the coin toss, which relates tothe idea of a very large (actually infinite) number of repeatedexperiments.

6 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

The entire structure of Neyman - Pearson hypothesis testingand Fisherian statistics (together referred to as the frequentistschool) is based on frequentist probability.

Our conclusions regarding null and alternative hypothesespresuppose the idea that we could conduct the sameexperiment an infinite number of times.

Our interpretation of confidence intervals also assumes a fixedparameter and CIs that vary over an infinitely large number ofidentical experiments.

7 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

But there is another view of probability as subjective belief.

The physical model in this case is that of the “bet”.

Consider the situation of betting on who will win the the WorldCup (or the World Series).

Here, probability is not based on an infinite number ofrepeatable and stochastically independent events, but rather onhow much knowledge you have and how much you are willingto bet.

Subjective probability allows one to address questions such as“what is the probability that my team will win the World Cup?”Relative frequency supplies information, but it is not the sameas probability and can be quite different.

This notion of subjective probability underlies Bayesianstatistics.

8 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Bayes’ Theorem

Consider the joint probability of two events, Y and X, forexample observing lung cancer and smoking jointly.

The joint probability can be written as

p(cancer, smoking) = p(cancer|smoking)p(smoking) (1)

Similarly

p(smoking, cancer) = p(smoking|cancer)p(cancer) (2)

9 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Because these are symmetric, we can set them equal to eachother to obtain the following

p(cancer|smoking)p(smoking) = p(smoking|cancer)p(cancer)(3)

.

p(cancer|smoking) =p(smoking|cancer)p(cancer)

p(smoking)(4)

The inverse probability theorem (Bayes’ theorem) states.

p(smoking|cancer) =p(cancer|smoking)p(smoking)

p(cancer)(5)

10 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM



.


p(smoking)(4)

The inverse probability theorem (Bayes’ theorem) states

.


p(cancer)(5)

10 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM



.


p(smoking)(4)

The inverse probability theorem (Bayes’ theorem) states.


p(cancer)(5)

10 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Why do we care?

Because this is how we go from the probability of a patienthaving cancer given that the patient smokes, to the probabilitythat the patient smokes given that he/she has cancer.

We simply need the marginal probability of smoking and themarginal probability of cancer (”base rates” or what we will callprior probabilities).

11 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Statistical Elements of Bayes’ Theorem

What is the role of Bayes’ theorem for statistical inference?

Denote by Y a random variable that takes on a realized value y.

For example, a person’s socio-economic status could beconsidered a random variable taking on a very large set ofpossible values.

Once the person identifies his/her socioeconomic status, therandom variable Y is now realized as y.

Because Y is unobserved and random, we need to specify aprobability model to explain how we obtained the actual datavalues y.

12 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Next, let θ be a parameter that we believe characterizes theprobability model of interest.

The parameter θ can be a scalar, such as the mean or thevariance of a distribution, or it can be vector-valued, such as aset of regression coefficients in regression analysis or factorloadings in factor analysis.

We are concerned with determining the probability of observingy given the unknown parameters θ, which we write as p(y|θ).

In statistical inference, the goal is to obtain estimates of theunknown parameters given the data.

This is expressed as the likelihood of the parameters given thedata, often denoted as L(θ|y).

13 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

A key difference between Bayesian statistical inference andfrequentist statistical inference concerns the nature of theunknown parameters θ.

In the frequentist tradition, the assumption is that θ is unknownand has a fixed value that we wish to estimate. Our uncertaintyabout θ is not taken into account in the frequentist tradition.

In Bayesian statistical inference, θ is also considered unknownbut specify a probability distribution that reflects our uncertaintyabout the true value of θ.

Because both the observed data y and the parameters θ areassumed random, we can model the joint probability of theparameters and the data as a function of the conditionaldistribution of the data given the parameters, and the priordistribution of the parameters.

14 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

More formally,

p(θ, y) = p(y|θ)p(θ). (6)

where p(θ, y) is the joint distribution of the parameters and thedata. Following Bayes’ theorem described earlier, we obtain

Bayes’ Theorem

p(θ|y) =p(θ, y)

p(y)=p(y|θ)p(θ)p(y)

, (7)

where p(θ|y) is referred to as the posterior distribution of theparameters θ given the observed data y.

15 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

More formally,p(θ, y) = p(y|θ)p(θ). (6)

where p(θ, y) is the joint distribution of the parameters and thedata. Following Bayes’ theorem described earlier, we obtain

Bayes’ Theorem

p(θ|y) =p(θ, y)

p(y)=p(y|θ)p(θ)p(y)

, (7)

where p(θ|y) is referred to as the posterior distribution of theparameters θ given the observed data y.

15 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

From equation (7) the posterior distribution of θ give y is equalto the data distribution p(y|θ) times the prior distribution of theparameters p(θ) normalized by p(y) so that the posteriordistribution sums (or integrates) to one.

For discrete variables

p(y) =∑θ

p(y|θ)p(θ), (8)

For continuous variables

p(y) =

∫θ

p(y|θ)p(θ)dθ, (9)

16 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Notice that p(y) does not involve model parameters, so we canomit the term and obtain the unnormalized posterior distribution

.

p(θ|y) ∝ p(y|θ)p(θ). (10)

This can also be expressed in terms of the likelihood

.

p(θ|y) ∝ L(θ|y)p(θ). (11)

17 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Equations (10) and (11) represents the core of Bayesianstatistical inference and is what separates Bayesian statisticsfrom frequentist statistics.

Equation (11) states that our uncertainty regarding theparameters of our model, as expressed by the prior densityp(θ), is weighted by the actual data p(y|θ) (or equivalently,L(θ|y)), yielding an updated estimate of our uncertainty, asexpressed in the posterior density p(θ|y).

18 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Exchangeability

It is common in statistics to invoke the assumption that the datay1, y2, . . . yn are independently and identically distributed – oftenreferred to as the i.i.d assumption.

Bayesians invoke the deeper notion of exchangeability toproduce likelihoods and address the issue of independence.

Exchangeability arises from de Finetti’s RepresentationTheorem and implies that the subscripts of a vector of data, e.g.y1, y2, . . . yn do not carry information that is relevant todescribing the probability distribution of the data.

In other words, the joint distribution of the data, p(y1, y2, . . . yn)is invariant to permutations of the subscripts.

19 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Consider the response that student i (i = 1, 2, . . . , 10) wouldmake to a PIRLS question “I like what I read in school“, where

yi =

1, if student i agrees0, if student i disagrees

(12)

Next, consider three patterns of responses by 10 randomlyselected students

p(1, 0, 1, 1, 0, 1, 0, 1, 0, 0) (13a)p(1, 1, 0, 0, 1, 1, 1, 0, 0, 0) (13b)p(1, 0, 0, 0, 0, 0, 1, 1, 1, 1) (13c)

Note that there are actually 210 possible response patterns.

20 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

If our task were to assign probabilities to all possible outcomes,this could become prohibitively difficult.

However, suppose we now assume that student responses areindependent of one another.

Exchangeability implies that only the proportion of agreementsmatter, not the location of those agreements in the vector.

Given that the sequences are the same length n, we canexchange the response of student i for student j, the withoutchanging our belief about the probability model that generatedthat sequence.

21 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Exchangeability means that we believe that there is aparameter θ that generates the observed data via a stochasticmodel and that we can describe that parameter withoutreference to the particular data at hand.

The fact that we can describe θ without reference to a particularset of data is, in fact, what is implied by the idea of a priordistribution.

22 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

The Prior Distribution

Why do we specify a prior distribution on the parameters?

Our view is that progress in science generally comes about bylearning from previous research findings and incorporatinginformation from these findings into our present studies.

Information from previous research is almost always incorporatedinto our choice of designs, variables to be measured, orconceptual diagrams to be drawn.

Bayesian statistical inference requires that our prior beliefs bemade explicit, but then moderates our prior beliefs by the actualdata in hand.

Moderation of our prior beliefs by the data in hand is the keymeaning behind equations (10) and (11).

23 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

But how do we choose a prior?

The choice of a prior is based on how much information webelieve we have prior to the data collection and how accurate webelieve that information to be.

Leamer (1983), who orders priors on the basis of degree ofconfidence.

Leamer’s hierarchy of confidence is as follow: truths (e.g. axioms)> facts (data) > opinions (e.g. expert judgement) > conventions(e.g. pre-set alpha levels).

The strength of Bayesian inference lies precisely in its ability toincorporate existing knowledge into statistical specifications.

24 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Non-informative priors

In some cases we may not be in possession of enough priorinformation to aid in drawing posterior inferences.

From a Bayesian perspective, this lack of information is stillimportant to consider and incorporate into our statisticalspecifications.

In other words, it is equally important to quantify our ignoranceas it is to quantify our cumulative understanding of a problem athand.

The standard approach to quantifying our ignorance is toincorporate a non-informative prior into our specification.

Non-informative priors are also referred to as reference, defalut,or objective priors.

25 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Perhaps the most common non-informative prior distribution isthe uniform distribution U(α, β) over some sensible range ofvalues from α to β.

The uniform distribution indicates that we believe that the valueof our parameter of interest lies in range β−α and that allvalues have equal probability.

Care must be taken in the choice of the range of values over theuniform distribution. A U [−∞,∞] is an improper priordistribution because it does not integrate to 1.0 as required ofprobability distributions.

26 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Informative (conjugate) priors

It may be the case that some information can be brought tobear on a problem and be systematically incorporated into theprior distribution.

Such “subjective” priors are called informative, personal, orepistemic.

One type of informative prior is based on the notion of aconjugate distribution.

A conjugate prior distribution is one that, when combined withthe likelihood function yields a posterior that is in the samedistributional family as the prior distribution.

27 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Conjugate Priors for Some CommonDistributions

Data Distribution Conjugate Prior

The normal distribution The normal distribution or Uniform Distribution

The Poisson distribution The gamma distribution

The binomial distribution The Beta Distribution

The multinomial distribution The Dirichlet Distribution

28 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

−4 −2 0 2 4

0.0

0.5

1.0

1.5

Prior is Normal (0,0.3)

x

Density

prior

Likelihood

Posterior

−4 −2 0 2 4

0.0

0.5

1.0

1.5


x

Density

prior

Likelihood

Posterior

−4 −2 0 2 4

0.0

0.5

1.0

1.5


Density

prior

Likelihood

Posterior

−4 −2 0 2 4

0.0

0.5

1.0

1.5

Prior is Normal (0,3)

Density

prior

Likelihood

Posterior

Figure 1: Normal distribution, mean unknown/variance known with varyingconjugate priors

29 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

Prior is Gamma(10,0.2)

x

Den

sity

priorLikelihoodPosterior

0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

Prior is Gamma(8,0.5)

x

Den

sity


0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

Prior is Gamma(3,1)

Den

sity


0 2 4 6 8 10

0.0

0.2

0.4

0.6

0.8

1.0

Prior is Gamma(2.1,3)

Den

sity


Figure 2: Poisson distribution with varying gamma-density priors

30 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Schools of Thought

A critically important component of applied statistics ishypothesis testing.

The approach most widely used in the social and behavioralsciences is the Neyman-Pearson approach.

An interesting aspect of the Neyman-Pearson approach tohypothesis testing is that students (as well as many seasonedresearchers) appear to have a very difficult time grasping itsprinciples.

Gigerenzer (2004) argued that much of the difficulty in graspingfrequentist hypothesis testing lies in the conflation of Fisherianhypothesis testing and the Neyman-Pearson approach tohypothesis testing.

31 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Fisher’s early approach to hypothesis testing requiredspecifying only the null hypothesis.

A conventional significance level is chosen (usually the 5%level).

Once the test is conducted, the result is either significant(p < .05) or it is not (p > .05).

If the resulting test is significant, then the null hypothesis isrejected. However, if the resulting test is not significant, then noconclusion can be drawn.

Fisher’s approach was based on looking at how data couldinform evidence for a hypothesis.

32 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

The Neyman and Pearson approach requires that twohypotheses be specified – the null and alternative hypothesis –and is designed to inform specific sets of actions. It’s aboutmaking a choice, not about evidence against a hypothesis.

By specifying two hypotheses, one can compute a desiredtradeoff between two types of errors: Type I errors (α) and TypeII errors (β)

The goal is not to assess the evidence against a hypothesis (ormodel) taken as true. Rather, it is whether the data providesupport for taking one of two competing sets of actions.

In fact, prior belief as well as interest in “the truth” of thehypothesis is irrelevant – only a decision that has a lowprobability of being the wrong decision is relevant.

33 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

The conflation of Fisherian and Neyman-Pearson hypothesistesting lies in the use and interpretation of the p-value.

In Fisher’s paradigm, the p-value is a matter of convention withthe resulting outcome being based on the data.

In the Neyman-Pearson paradigm, α and β are determinedprior to the experiment being conducted and refer to aconsideration of the cost of making one or the other error.

34 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

In the Neyman-Pearson approach, the problem is one of findinga balance between α, power, and sample size.

The Neyman-Pearson approach is best suited for experimentalplanning. Sometimes, it is used this way, followed by theFisherian approach for judging evidence. But, these two ideasmay be incompatible (Royall, 1997).

However, this balance is virtually always ignored and α = 0.05 isused.

The point is that the p-value and α are not the same thing.

35 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

The confusion between these two concepts is made worse bythe fact that statistical software packages often report a numberof p-values that a researcher can choose from after havingconducted the analysis (e.g., .001, .01, .05).

This can lead a researcher to set α ahead of time, as per theNeyman-Pearson school, but then communicate a different levelof “significance” after running the test.

The conventional practice is even worse than described, suchas when describing non-significant results as

.“trending toward significance”

36 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Misunderstanding the Fisher or Neyman-Pearson framework tohypothesis testing and/or poor methodological practice is not acriticism of the approach per se.

However, from the frequentist point of view, a criticism oftenleveled at the Bayesian approach to statistical inference is that itis “subjective”, while the frequentist approach is “objective”.

The objection to “subjectivism” is perplexing insofar asfrequentist hypothesis testing also rests on assumptions that donot involve data.

The simplest and most ubiquitous example is the choice ofvariables in a regression equation or asymptotic distributions oftest statistics.

37 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Point Summaries of the Posterior Distribution

Hypothesis testing begins first by obtaining summaries ofrelevant distributions.

The difference between Bayesian and frequentist statistics isthat with Bayesian statistics we wish to obtain summaries of theposterior distribution.

The expressions for the mean and variance of the posteriordistribution come from expressions for the mean and varianceof conditional distributions generally.

38 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

For the continuous case, the mean of the posterior distributionof θ given the data y is referred to as the expected a posteriorior EAP estimate and can be written as

.

E(θ|y) =

+∞∫−∞

θp(θ|y)dθ. (14)

Similarly, the variance of posterior distribution of θ given y canbe obtained as

.

var(θ|y) = E[(θ − E[(θ|y])2|y),

=

+∞∫−∞

(θ − E[θ|y])2p(θ|y)dθ. (15)

39 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Another common summary measure would be the mode of theposterior distribution – referred to as the maximum a posteriori(MAP) estimate.

The MAP begins with the idea of maximum likelihoodestimation. The MAP can be written as

.

θMAP = arg maxθL(θ|y)p(θ). (16)

where arg maxθ stands for “maximizing the value of theargument”.

40 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Interval Summaries

Posterior Probability Intervals

In addition to point summary measures, it may also be desirableto provide interval summaries of the posterior distribution.

Recall that the frequentist confidence interval requires that weimagine an infinite number of repeated samples from thepopulation characterized by µ.

For any given sample, we can obtain the sample mean x and thenform a 100(1− α)% confidence interval.

The correct frequentist interpretation is that 100(1− α)% of theconfidence intervals formed this way capture the true parameter µunder the null hypothesis. Notice that the probability that theparameter is in the interval is either zero or one.

41 / 165





PointSummaries

IntervalSummaries

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Posterior Probability Intervals (cont’d)

In contrast, the Bayesian framework assumes that a parameterhas a probability distribution.

Sampling from the posterior distribution of the model parameters,we can obtain its quantiles. From the quantiles, we can directlyobtain the probability that a parameter lies within a particularinterval.

So, a 95% posterior probability interval would mean that theprobability that the parameter lies in the interval is 0.95.

Notice that this is entirely different from the frequentistinterpretation, and arguably aligns with common sense.

42 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Day 1 PM: Bayesian Computation with R andrjags

The key reason for the increased popularity of Bayesianmethods in the social and behavioral sciences has been the(re)-discovery of numerical algorithms for estimating theposterior distribution of the model parameters given the data.

Prior to these developments, it was virtually impossible toanalytically derive summary measures of the posteriordistribution, particularly for complex models with manyparameters.

Rather than attempting the impossible task of analyticallysolving for estimates of a complex posterior distribution, we caninstead draw samples from p(θ|y) and summarize thedistribution formed by those samples. This is referred to asMonte Carlo integration.

The two most popular methods of MCMC are the Gibbssampler and the Metropolis-Hastings algorithm.

43 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

A decision must be made regarding the number of Markovchains to be generated, as well as the number of iterations ofthe sampler.

Each chain samples from another location of the posteriordistribution based on starting values.

With multiple chains, fewer iterations are required, particularly ifthere is evidence for the chains converging to the sameposterior mean for each parameter.

Once the chain has stabilized, the burn-in samples arediscarded.

Summaries of the posterior distribution as well as convergencediagnostics are calculated on the post-burn-in iterations.

44 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Brief Introduction to R

R is a programming environment for data analysis and graphics.It is also a language similar to C++.

R is part of the GNU (GNU is Not Unix) Project and associatedwith the Free Software Foundation.

“Free” means freedom not price.

When contributing a package to R, you agree to allow anyoneaccess to the code, to be copied, modified, etc.

45 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Downloading R

1 Go to http://www.r-project.org/

2 Click on CRAN and choose mirror site

3 Click on mirror site and chose OS, e.g. Mac or Windows

4 For Mac click on latest version; for Windows, click on BASE

Finding and Installing Packages

1 Go to http://cran.r-project.org/web/views/

2 Packages are organized topically. For example, click onMCMCpack.

46 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

To install a package for Mac1 At the prompt > type install.packages(”MCMCpack”). Do this

again for the package ”coda” and ”BMA”.

2 Refresh the list. Your package should then show up. Click onbutton and the package is installed.

To install a package for the PC1 Go to Packages − > Set Cran Mirror. Choose a mirror site

2 Go back to Packages − > Install Packages. Choose MCMCpackand then again coda.

3 Packages − > Load Packages. Click on these packages and theywill then be installed.

4 You can now open the package and see the various programsand syntax.

47 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Writing Programs in R

Programs can be written directly into the console or in a script file.

New Script for PC. New Document for Mac. This is to bepreferred.

The file with your script will be save as a .R file.

If you are writing commands in the console, you may want to savethe session when youre finished. Then, all the commands youtyped will reappear again.

48 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Reading in Data

You have to have your data saved as a .txt or .dat file. This can bedone from most other statistical software packages as well asfrom Excel. R will read in other data forms as well.

The general format of reading in data is

objectname <- read.table(datalocation, header=T)

If you’re using .csv files from Excel, then this would read

objectname <- read.csv(datalocation, header=T)

The “objectname” is a new name given to the datafile that R then canread.

“datalocation” is the location of the data using DOS or Mac file locationrules.

49 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Reading in Data (cont’d)

Perhaps the easiest approach is to convert your data set into a.csv file and type the following

objectname <- read.csv(file.choose(),header=T

This will open up a window so you can search for the file.

You can also use the package “foreign“ that will allow you to bring manyother system files, such as SPSS or SAS.

For most text files, the top row contains the variable names, so header=Ttells R that’s the case.

50 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Handling Missing Data in R.

Don’t have any missing data. ,

Assuming this is not feasible, the best thing to do is to convertmissing data into some kind of absurd value such as 999. Thiscan be done easily in SPSS.

Once the data are converted to a .dat file (perhaps via Excel),then the goal is to convert the 999 into the code NA, which Rrecognizes.

Bring in your data set as usual, for example

mydata <- read.csv(file.choose(),header=T)

51 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Handling Missing Data in R (cont’d)

To convert all of the 999s to NAs in one step, type the following

mydata[jmydata==999] = NA

You can check this by typing “mydata”at the command line. That is

> mydata

You should have NAs where the 999s were.

To remove all missing data (coded as NA) (LISTWISE DELETION from ananalysis, you would write, for example

mydata <- na.omit(mydata)

If missing is blank then R will convert to NA.

Most programs will use something like listwise deletion. THIS IS NOTPREFERRED!! R has many good missing data programs.

52 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Manipulating Data in R

A quick way to create dummy codes for categorical variable in Ris as follows: For a dichotomous variable, say coding 1 = female,2 = male, to 0 = male

female <- ifelse(sex==2,0,1)

For three categories

newrace1 = ifelse(race==1,1,0)newrace2 = ifelse(race==2,1,0)

Categorical variables such as sex and race should be treated as factors.In R this is handled as follows. For example

race = c(1,2,3)frace=factors(race)levels(frace) = c(‘‘black", ‘‘white", ‘‘hispanic")

53 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Manipulating Data in R (cont’d)

To select a subset of observations in R, we can write the following:

mydata <- read.csv(file.choose(),header=T)newdata <- subset(mydata, variable > somevalue)

To indicate which columns along with rows to keep, you write

newdata <- subset(mydata, variable > somevalue,select=c(v1,v2))

For numerous transformations, you could write

newdata <- within(mydata,v1 < somevalue,v2 == somevalue,v3 = log(v4),rm(v5,v6))

54 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Manipulating Data in R (cont’d)

To transform data, the simplest approach would be to write

junk <- read.csv(file.choose(),header=T)newdata <- transform(mydata, log.v1 = log(v1))

Combining the columns of two data sets, for example data1 and data2. Tocombine them, write

data3 <- cbind(data1,data2)

55 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Saving Output and plots to a file

“sink” will divert all subsequent output to a file. Place “sink”at thebeginning of the file where you want output to go. For example:

sink("˜/Desktop/test.txt")d <- rnorm(1000,0,1)summary(d)sink()

To send a plot to a file, you can type

pdf("˜/Desktop/test.pdf")d <- rnorm(1000, 0, 1)plot(density(d))dev.off()

This last command prevents the plot from showing up on the screen.

56 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Some Good R Books

1 Crawley, M. J. (2007). The R Book: Chichester, UK: John Wiley &Sons.

2 Dalgaard, P. (2002). Introductory Statistics with R. New York:Springer.

3 Murrell, P. (2006). R Graphics: Boca Ratan: Taylor & Francis.

Some Interesting Web Sites for R

1 http://www.fsf.org/

2 http://www.gnu.org/

3 http://www.r-project.org/

4 http://cran.r-project.org/web/views/

57 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Brief Introduction to rjags

For this workshop we will use the R interface of the program“JAGS” (Just Another Gibbs Sampler ; Plummer, 2015)

JAGS is a BUGS like program (Bayes Under Gibbs Sampling;Spiegelhalter, Thomas and Best 2000).

BUGS was the first program (within WinBUGS) that madeBayesian analysis possible.

58 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Steps when using rjags:

1 Bring in data and use R to do any manipulations - e.g. handlingmissing data, etc.

2 Write model in jags and save as a .bug file.

3 Pass the data and model to JAGS using the jags.modelcommand.

4 Use coda.samples and update to run the Gibbs sample.

5 Use R to inspect results and get plots.

59 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

## Simple normal distribution with conjugate prior #### Install and load packages ##

install.packages("rjags")require(rjags)

# Generate some random data

N <- 10000y <- rnorm(N, 0, 5)

# JAGS Code

modelstring ="model for (i in 1:N)

y[i] ˜ dnorm(mu, tau)# Priorsmu ˜ dnorm(0, 0.0001);tau ˜ dgamma(0.001, 0.001);# Transformationssigma <- 1.0/sqrt(tau);

"# End of JAGS code and back to R

60 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

writeLines(modelstring,con="model.bug")variables <- list(y = y, N=N)parameters <- c("mu","tau","sigma")simplemod <-jags.model("model.bug",data=variables, n.chains=2,

n.adapt=1000)

cat("Burning in the MCMC chain ...\n")update(simplemod, n.iter=5000)cat("Sampling from the final MCMC chain ... \n")codaSamples1 = coda.samples(simplemod,variable.names=parameters,

n.iter=100000, thin=10,seed=5555)

61 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

## Use coda to obtain diagnostic plots ##

par(mar=c(2,2,2,2)) # Help with plotting margins in RStudioplot(codaSamples1)acf(codaSamples1[[1]])geweke.diag(codaSamples1)geweke.plot(codaSamples1)gelman.diag(codaSamples1)gelman.plot(codaSamples1)## Summarize output ##

options(scipen=999) # A nice trick to remove scientific notationsummary(codaSamples1[[1]])

62 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Figure 3: Trace and density plots for two chains63 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

0 50 150 250 350

−1.

0−

0.5

0.0

0.5

1.0

Lag

AC

F

mu

0 50 150 250 350

−1.

0−

0.5

0.0

0.5

1.0

Lag

mu & sigm

0 50 150 250 350

−1.

0−

0.5

0.0

0.5

1.0

Lag

mu & tau

−350 −250 −150 −50 0

−1.

0−

0.5

0.0

0.5

1.0

Lag

AC

Fsigm & mu

0 50 150 250 350−

1.0

−0.

50.

00.

51.

0

Lag

sigma

0 50 150 250 350

−1.

0−

0.5

0.0

0.5

1.0

Lag

sigm & tau

−350 −250 −150 −50 0

−1.

0−

0.5

0.0

0.5

1.0

Lag

AC

F

tau & mu

−350 −250 −150 −50 0

−1.

0−

0.5

0.0

0.5

1.0

Lag

tau & sigm

0 50 150 250 350

−1.

0−

0.5

0.0

0.5

1.0

Lag

tau

Figure 4: ACF plots for two chains64 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM 10000 20000 30000 40000 50000

−2

−1

01

2

First iteration in segment

mu (chain1)

10000 20000 30000 40000 50000−

2−

10

12


Z−

scor

e

sigma (chain1)

10000 20000 30000 40000 50000

−2

−1

01

2

tau (chain1)

10000 20000 30000 40000 50000

−2

−1

01

2

Z−

scor

e

mu (chain2)10000 20000 30000 40000 50000

−2

−1

01

2


sigma (chain2)

10000 20000 30000 40000 50000

−2

−1

01

2


Z−

scor

e

tau (chain2)

Figure 5: Geweke plots

65 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM20000 40000 60000 80000

1.00

01.

005

1.01

01.

015

1.02

01.

025

1.03

0

last iteration in chain

median

97.5%

mu

20000 40000 60000 80000

0.99

51.

000

1.00

51.

010

last iteration in chain

shrin

k fa

ctor

median

97.5%

sigma

20000 40000 60000 80000

0.99

51.

000

1.00

51.

010 median

97.5%

tau

Figure 6: Gelman-Rubin-Brooks plot66 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Fraction in 1st window = 0.1Fraction in 2nd window = 0.5

mu sigma tau0.003232 0.419334 -0.428941

[[2]]Fraction in 1st window = 0.1Fraction in 2nd window = 0.5

mu sigma tau-1.0197 -0.1983 0.1819

Potential scale reduction factors:

Point est. Upper C.I.mu 1 1sigma 1 1tau 1 1

Multivariate psrf

1

1. Empirical mean and standard deviation for each variable,plus standard error of the mean:

Mean SD Naive SE Time-series SEmu 0.009206 0.0503731 0.000503731 0.000503731sigma 5.005702 0.0353989 0.000353989 0.000353989tau 0.039915 0.0005645 0.000005645 0.000005645

2. Quantiles for each variable:

2.5\% 25\% 50\% 75\% 97.5\%mu -0.08950 -0.02500 0.00948 0.04332 0.10741sigma 4.93647 4.98148 5.00554 5.02996 5.07527tau 0.03882 0.03952 0.03991 0.04030 0.04104

67 / 165

Day 1 AM

Day 1 PM

Day 2 AMPosteriorPredictiveChecking

Bayes Factors

BayesianModelAveraging

BayesianLinearRegression

Day 2 PM

Day 3 AM

Day 3 PM

Day 2 AM: Bayesian Model Building

The frequentist and Bayesian goals of model building are thesame.

1 Model specification based on prior knowledge

2 Model estimation and fitting

3 Model evaluation and modification

4 Model choice

68 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Despite these similarities there are important differences.

A major difference between the Bayesian and frequentist goalsof model building lie in the model specification stage.

Because the Bayesian perspective explicitly incorporatesuncertainty regarding model parameters using priors, the firstphase of modeling building requires the specification of a fullprobability model for the data and the parameters of the model.

Model fit implies that the full probability model fits the data.Lack of model fit may be due to incorrect specification oflikelihood, the prior distribution, or both.

69 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Another difference between the Bayesian and frequentist goalsof model building relates to the justification for choosing aparticular model among a set of competing models.

Model building and model choice in the frequentist domain isbased primarily on choosing the model that best fits the data.

This has certainly been the key motivation for model building,respecification, and model choice in the context of structuralequation modeling (Kaplan 2009).

In the Bayesian domain, the choice among a set of competingmodels is based on which model provides the best posteriorpredictions.

That is, the choice among a set of competing models should bebased on which model will best predict what actually happened.

70 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Posterior Predictive Checking

A very natural way of evaluating the quality of a model is toexamine how well the model fits the actual data.

In the context of Bayesian statistics, the approach to examininghow well a model predicts the data is based on the notion ofposterior predictive checks, and the accompanying posteriorpredictive p-value.

The general idea behind posterior predictive checking is thatthere should be little, if any, discrepancy between datagenerated by the model, and the actual data itself.

Posterior predictive checking is a method for assessing thespecification quality of the model. Any deviation between thedata generated from the model and the actual data impliesmodel misspecification.

71 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

In the Bayesian context, the approach to examining model fitand specification utilizes the posterior predictive distribution ofreplicated data.Let yrep be data replicated from our current model.

Posterior Predictive Distribution

p(yrep|y) =

∫p(yrep|θ)p(θ|y)dθ

=

∫p(yrep|θ)p(y|θ)p(θ)dθ (17)

Equation (17) states that the distribution of future observationsgiven the present data, p(yrep|y), is equal to the probabilitydistribution of the future observations given the parameters,p(yrep|θ), weighted by the posterior distribution of the modelparameters.

72 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

To assess model fit, posterior predictive checking implies thatthe replicated data should match the observed data quiteclosely if we are to conclude that the model fits the data.

One approach to model fit in the context of posterior predictivechecking is based on Bayesian p-values.

Denote by T (y) a test statistic (e.g. χ2) based on the data, andlet T (yrep) be the same test statistics for the replicated data(based on MCMC). Then, the Bayesian p-value is defined to be

p-value = pr[T (yrep, θ) ≥ T (y, θ)|y]. (18)

The p-value is the proportion of of replicated test values thatequal or exceed the observed test value. High (or low if signsare reversed) values indicate poor model fit.

73 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayes Factors

A very simple and intuitive approach to model building andmodel selection uses so-called Bayes factors (Kass & Raftery,1995)

In essence, the Bayes factor provides a way to quantify theodds that the data favor one hypothesis over another. A keybenefit of Bayes factors is that models do not have to benested.

Consider two competing models, denoted as M1 and M2, thatcould be nested within a larger space of alternative models. Letθ1 and θ2 be the two parameter vectors associated with thesetwo models.

These could be two regression models with a different numberof variables, or two structural equation models specifying verydifferent directions of mediating effects.

74 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

The goal is to develop a quantity that expresses the extent towhich the data support M1 over M2. One quantity could be theposterior odds of M1 over M2, expressed as

Bayes Factorsp(M1|y)

p(M2|y)=p(y|M1)

p(y|M2)×[p(M1)

p(M2)

]. (19)

Notice that the first term on the right hand side of equation (19)is the ratio of two integrated likelihoods.

This ratio is referred to as the Bayes factor for M1 over M2,denoted here as B12.

Our prior opinion regarding the odds of M1 over M2, given byp(M1)/p(M2) is weighted by our consideration of the data,given by p(y|M1)/p(y|M2).

75 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

This weighting gives rise to our updated view of evidenceprovided by the data for either hypothesis, denoted asp(M1|y)/p(M2|y).

An inspection of equation (19) also suggests that the Bayesfactor is the ratio of the posterior odds to the prior odds.

In practice, there may be no prior preference for one model overthe other. In this case, the prior odds are neutral andp(M1) = p(M2) = 1/2.

When the prior odds ratio equals 1, then the posterior odds isequal to the Bayes factor.

76 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian Information Criterion

A popular measure for model selection used in both frequentistand Bayesian applications is based on an approximation of theBayes factor and is referred to as the Bayesian informationcriterion (BIC), also referred to as the Schwarz criterion.

Consider two models, M1 and M2 with M2 nested in M1. Underconditions where there is little prior information, the BIC can bewritten as

.

BIC = −2 log(θ|y) + p log(n) (20)

where −2 log θ|y describes model fit while p log(n) is a penaltyfor model complexity, where p represents the number ofvariables in the model and n is the sample size.

77 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

The BIC is often used for model comparisons. The differencebetween two BIC measures comparing, say M1 to M2 can bewritten as

.

∆(BIC12) = BIC(M1) −BIC(M2) (21)

= log(θ1|y)− log(θ2|y)− 1

2(p1 − p2) log(n)

78 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Rules of thumb have been developed to assess the quality ofthe evidence favoring one hypothesis over another using Bayesfactors and the comparison of BIC values from two competingmodels. Using M1 as the reference model,

BIC Difference Bayes Factor Evidence against M2

0 to 2 1 to 3 Weak2 to 6 3 to 20 Positive6 to 10 20 to 150 Strong> 10 > 150 Very strong

79 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Deviance Information Criterion

The BIC (ironically) is not fundamentally Bayesian

An explicitly Bayesian approach to model comparison has beendeveloped based on the notion of Bayesian deviance.

Define Bayesian deviance as

Bayes Deviance

D(θ) = −2 log[p(y|θ)] + 2 log[h(y)], (22)

To make this Bayesian this, we obtain a posterior mean over θby defining

.

D(θ) = Eθ[−2log[p(y|θ)|y] + 2log[h(y)]. (23)

80 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Let D(θ) be a posterior estimate of θ. We can define theeffective dimension of the model as qD = D(θ)−D(θ).

We then add the model fit term D(θ) to obtain the devianceinformation criterion (DIC) - namely,

Deviance Information Criterion

DIC = D(θ) + qD = 2D(θ)−D(θ). (24)

The DIC can be obtained by calculating equation (23) overMCMC samples.

Models with the lowest DIC values are preferred.

81 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian Model Averaging

The selection of a particular model from a universe of possiblemodels can also be characterized as a problem of uncertainty.This problem was succinctly stated by Hoeting, Raftery &Madigan (1999) who write

.“Standard statistical practice ignores model uncertainty. Data analyststypically select a model from some class of models and then proceed as ifthe selected model had generated the data. This approach ignores theuncertainty in model selection, leading to over-confident inferences anddecisions that are more risky than one thinks they are.”(pg. 382)

An approach to addressing the problem is the method ofBayesian model averaging (BMA). We will show this in theregression example.

82 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM


Consider a quantity of interest such as a future observationdenoted as Υ.

Next, consider a set of competing models Mk, k = 1, 2, . . . ,Kthat are not necessarily nested.

The posterior distribution of Υ given data y can be written as

.

p(Υ|y) =

K∑k=1

p(Υ|Mk)p(Mk|y). (25)

where p(Mk|y) is the posterior probability of model Mk writtenas

.

p(Mk|y) =p(y|Mk)p(Mk)∑Kl=1 p(y|Ml)p(Ml)

, l 6= k. (26)

83 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM


Consider a quantity of interest such as a future observationdenoted as Υ.

Next, consider a set of competing models Mk, k = 1, 2, . . . ,Kthat are not necessarily nested.

The posterior distribution of Υ given data y can be written as.

p(Υ|y) =

K∑k=1

p(Υ|Mk)p(Mk|y). (25)

where p(Mk|y) is the posterior probability of model Mk writtenas

.

p(Mk|y) =p(y|Mk)p(Mk)∑Kl=1 p(y|Ml)p(Ml)

, l 6= k. (26)

83 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

The interesting feature of equation (26) is that p(Mk|y) will likelybe different for different models.

The term p(y|Mk) can be expressed as an integrated likelihood

.

p(y|Mk) =

∫p(y|θk,Mk)p(θk|Mk)dθk, (27)

where p(θk|Mk) is the prior distribution of θk under model Mk

(Raftery et al., 1997).

BMA provides an approach for combining models specified byresearchers.

Madigan and Raftery (1994) show that BMA provides betterpredictive performance than that of a single model based on alog-score rule.

84 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

BMA is difficult to implement.

1 The number of terms in p(Υ|y) =∑K

k=1 p(Υ|Mk)p(Mk|y) can bequite large and the corresponding integrals are hard to compute.

2 Eliciting p(Mk) may not be straightforward. The uniform prior1/M is often used.

3 Choosing the class of models to average over is also challenging.

The problem of reducing the overall number of models that onecould incorporate in the summation has led to solutions basedon Occam’s window and Markov Chain Monte Carlo ModelComposition – M3 (Madigan and Raftery, 1994).

85 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian Linear Regression

We will start with the basic multiple regression model.

Let y be an n-dimensional vector (y1, y2, . . . , yn)′

(i = 1, 2, . . . , n) of reading scores from n students on the PIRLSreading assessment, and let X be an n× k matrix containing kbackground and attitude measures. Then, the normal linearregression model can be written as

.

y = Xβ + u, (28)

All the usual regression assumptions apply

We assume that student level PIRLS reading scores scores aregenerated from a normal distribution.

86 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Recall that the components of Bayes’ theorem require priors onall model parameters.

We will write the probability distribution for the regression modelas

.

p(X,y|β, σ2) (29)

Conventional statistics stops here and estimates the modelparameters with either maximum likelihood estimation orordinary least square.

But for Bayesian regression we need to specify the priors for allmodel parameters.

87 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

First consider non-informative priors.

In the context of the normal linear regression model, theuniform distribution is typically used as a non-informative prior.

That is, we assign an improper uniform prior to the regressioncoefficient β that allows β to take on values over the support[−∞,∞].

This can be written as p(β) ∝ c, where c is a constant.

88 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Next, we assign a uniform prior to log(σ2) because thistransformation also allows values over the support [0,∞].

From here, the joint posterior distribution of the modelparameters is obtained by multiplying the prior distributions of βand σ2 by the likelihood give in equation (29).

Assuming that β and σ2 are independent, we obtain

.

p(β, σ2|y,X) ∝ p(X,y|β, σ2)p(β)p(σ2). (30)

.In virtually all packages, non-informative or weakly informativepriors are the default.

89 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM



Assuming that β and σ2 are independent, we obtain.

p(β, σ2|y,X) ∝ p(X,y|β, σ2)p(β)p(σ2). (30)

.

In virtually all packages, non-informative or weakly informativepriors are the default.

89 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM



Assuming that β and σ2 are independent, we obtain.

p(β, σ2|y,X) ∝ p(X,y|β, σ2)p(β)p(σ2). (30)

.In virtually all packages, non-informative or weakly informativepriors are the default.

89 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

What about informative priors?

The most sensible conjugate prior distribution for the vector ofregression coefficients β of the linear regression model is themultivariate normal prior.

The conjugate prior for the variance of the disturbance term σ2

is the inverse-Gamma distribution, with shape and scalehyperparameters a and b, respectively.

From here, we can obtain the joint posterior distribution of allmodel parameters using conjugate priors based on expertopinion or prior research.

90 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian regression: Non-informative priors

Data come from a sub-sample of 5000 Canadian students whotook part in the IEA Program for International Reading LiteracyStudy (PIRLS 2011).

The international population for PIRLS 2011 consisted ofstudents in the grade that represents four years of schooling,provided that the mean age at the time of testing was at least9.5 years.

91 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Variables in this model are:

1 male (1=male)

2 ASBG04 = # of books in home

3 ASBGSBS = Bullying/teasing at school (higher values mean lessbullying/teasing)

4 ASBGSMR = Students motivated to read

5 ASBGSCR = Students confidence in their reading

6 ASBR05E = Teacher is easy to understand

7 ASBR05F = Interested in what teacher says

8 ASBR05G = Teacher gives interesting things to read

92 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian Multiple Regression:Non-informative Priors

### Multiple Regression Model ##### Install and load packages ##install.packages("rjags")require(rjags)

## Read in data ##Canadareg <-read.csv(file.choose(),header=TRUE) #browse to select data "Canada.csv"male <- ifelse(Canadareg$ITSEX==2,0,1)Canadareg <- cbind(Canadareg,male)Canadareg <- subset(Canadareg, select=c(ASRREA01,male,ASBG04,ASBGSBS,ASBGSMR,ASBGSCR,

ASBR05E,ASBR05F,ASBR05G))Canadareg[Canadareg==999999]=NACanadareg <- na.omit(Canadareg)Canadareg <- Canadareg[sample(1: nrow(Canadareg), 5000,replace=F),]nData=NROW(Canadareg)

## Begin JAGS Code ##modelstring = "

# Likelihoodmodel for (i in 1:nData)

ASRREA01[i] ˜ dnorm(mu[i], tau)mu[i] <- a + b1*male[i] + b2*ASBG04[i] + b3*ASBGSBS[i] + b4*ASBGSMR[i]

+ b5*ASBGSCR[i] + b6*ASBR05E[i] + b7*ASBR05F[i] + b8*ASBR05G[i]

93 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian Regression: Non-informative Priors

# Priors

a ˜ dnorm(0, .0001)b1 ˜ dnorm(0, .0001)b2 ˜ dnorm(0, .0001)b3 ˜ dnorm(0, .0001)b4 ˜ dnorm(0, .0001)b5 ˜ dnorm(0, .0001)b6 ˜ dnorm(0, .0001)b7 ˜ dnorm(0, .0001)b8 ˜ dnorm(0, .0001)tau ˜ dgamma(.01, 0.01)sigma ˜ dunif(0, 100)

"## End JAGS Code ##

94 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian Regression: Non-informative Priors

attach(Canadareg)writeLines(modelstring,con="model.bug")Canadareg <- list(ASRREA01=ASRREA01,male=male,ASBG04=ASBG04,ASBGSBS=ASBGSBS,

ASBGSMR=ASBGSMR,ASBGSCR=ASBGSCR,ASBR05E=ASBR05E,ASBR05F=ASBR05F,ASBR05G=ASBR05G,nData=nData)

parameters = c("a","b1","b2","b3","b4","b5","b6","b7","b8","tau")lapply(Canadareg,summary)Canadareg1 <-jags.model("model.bug", data=Canadareg,n.chains=2, n.adapt=5000)

cat("Burning in the MCMC chain ...\n")update(Canadareg1, n.iter=5000)cat("Sampling from the final MCMC chain ... \n")codaSamples1 = coda.samples(Canadareg1,variable.names=parameters,

n.iter=10000, thin=10,seed=5555)

## Use coda to obtain diagnostic plots ##par(mar=c(2,2,2,2))plot(codaSamples1)acf(codaSamples1[[1]])geweke.diag(codaSamples1)gelman.plot(codaSamples1)gelman.diag(codaSamples1)

## Summarize output ##options(scipen=999) # A nice trick to remove scientific notationsummary(codaSamples1[[1]])

95 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Figure 7: Trace and density plots for regression example

96 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Figure 8: Trace and density plots for regression example

97 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

0 100 200 300

−0.

50.

5

Lag

AC

F

a

0 100 200 300

−0.

50.

5

Lag

a & b1

0 100 200 300

−0.

50.

5

Lag

a & b2

0 100 200 300

−0.

50.

5

Lag

a & b3

0 100 200 300

−0.

50.

5

Lag

a & b4

−300 −150 0

−0.

50.

5

LagA

CF

b1 & a

0 100 200 300

−0.

50.

5

Lag

b1

0 100 200 300

−0.

50.

5

Lag

b1 & b2

0 100 200 300

−0.

50.

5

Lag

b1 & b3

0 100 200 300

−0.

50.

5

Lag

b1 & b4

−300 −150 0

−0.

50.

5

Lag

AC

Fb2 & a

−300 −150 0

−0.

50.

5

Lag

b2 & b1

0 100 200 300

−0.

50.

5

Lag

b2

0 100 200 300

−0.

50.

5

Lag

b2 & b3

0 100 200 300

−0.

50.

5

Lag

b2 & b4

−300 −150 0

−0.

50.

5

Lag

AC

F

b3 & a

−300 −150 0

−0.

50.

5

Lag

b3 & b1

−300 −150 0

−0.

50.

5

Lag

b3 & b2

0 100 200 300

−0.

50.

5

Lag

b3

0 100 200 300

−0.

50.

5

Lag

b3 & b4

−300 −150 0

−0.

50.

5

Lag

AC

F

b4 & a

−300 −150 0

−0.

50.

5

Lag

b4 & b1

−300 −150 0

−0.

50.

5

Lag

b4 & b2

−300 −150 0

−0.

50.

5

Lag

b4 & b3

0 100 200 300

−0.

50.

5

Lag

b4

[ 1 , 1 ]

Figure 9: Selected ACF plots for regression example

98 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Geweke Diagnostic[[1]]


a b1 b2 b3 b4 b50.2561 -0.4423 -0.6049 0.1415 -0.0185 0.1620

b6 b7 b8 tau-0.7107 0.5939 -0.3631 0.8690

[[2]]


a b1 b2 b3 b4 b5-0.08788 1.19659 -0.14634 -0.19179 0.31039 0.31930

b6 b7 b8 tau-0.11029 0.10144 -0.41664 0.06028

99 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Potential scale reduction factors:

Point est. Upper C.I.a 1 1b1 1 1b2 1 1b3 1 1b4 1 1b5 1 1b6 1 1b7 1 1b8 1 1tau 1 1

Multivariate psrf

1

100 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Iterations = 5010:105000Thinning interval = 10Number of chains = 1Sample size per chain = 10000


Mean SD Naive SE Time-series SEa 330.8184946 7.520770759 0.07520770759 0.22854502729b1 2.3045004 1.756978774 0.01756978774 0.01756978774b2 12.0627318 0.826166021 0.00826166021 0.01264760827b3 5.7351510 0.469627827 0.00469627827 0.01085387141b4 -2.6525502 0.520885540 0.00520885540 0.01060793826b5 12.9309822 0.474212123 0.00474212123 0.01011261341b6 5.7639764 1.487658259 0.01487658259 0.03680520372b7 1.2357292 1.467496377 0.01467496377 0.03208490365b8 -4.8637366 1.423546593 0.01423546593 0.03163075215tau 0.0002656 0.000005305 0.00000005305 0.00000005005


2.5% 25% 50% 75% 97.5%a 315.9381600 325.745992 330.7659447 335.9338771 345.3917959b1 -1.1417712 1.149604 2.2931490 3.4668234 5.7668519b2 10.4474817 11.515152 12.0628648 12.6193506 13.6814649b3 4.7888284 5.430616 5.7385525 6.0461955 6.6517412b4 -3.6861710 -2.997114 -2.6480935 -2.2977976 -1.6436147b5 11.9842282 12.618637 12.9321930 13.2476858 13.8587343b6 2.8608925 4.762251 5.7585103 6.7706293 8.6889859b7 -1.6674899 0.268615 1.2387679 2.2124796 4.1046863b8 -7.6617504 -5.816653 -4.8544442 -3.9017228 -2.0524820tau 0.0002555 0.000262 0.0002655 0.0002691 0.0002761

101 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Posterior predictive checking and the DIC

### Multiple Regression Model with PPC and DIC ##### Install and load packages ##


## Read in data ##

Canadareg <-read.csv(file.choose(),header=TRUE) #browse to select data "Canada.csv"male <- ifelse(Canadareg$ITSEX==2,0,1)Canadareg <- cbind(Canadareg,male)Canadareg <- subset(Canadareg, select=c(ASRREA01,male,ASBG04,ASBGSBS,ASBGSLR,ASBGSMR,ASBGSCR,

ASBR05E,ASBR05F,ASBR05G))Canadareg[Canadareg==999999]=NACanadareg <- na.omit(Canadareg)Canadareg <- Canadareg[sample(1: nrow(Canadareg), 5000,replace=F),]nData=NROW(Canadareg)

## Begin JAGS Code ##

modelstring = "

# Likelihood

model for (i in 1:nData)



## Obtaining information for posterior pred. checks ###res[i] <- ASRREA01[i] - mu[i]ASRREA01.new[i] ˜ dnorm(mu[i], tau)res.new[i] <- ASRREA01.new[i] - mu[i]

102 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM


# Priors

a ˜ dnorm(0, .0001)b1 ˜ dnorm(0, .0001)b2 ˜ dnorm(0, .0001)b3 ˜ dnorm(0, .0001)b4 ˜ dnorm(0, .0001)b5 ˜ dnorm(0, .0001)b6 ˜ dnorm(0, .0001)b7 ˜ dnorm(0, .0001)b8 ˜ dnorm(0, .0001)tau ˜ dgamma(.01, 0.01)sigma ˜ dunif(0, 100)

# Derived parameters for post. pred. checks #fit <- sum(res[ ])fit.new <- sum(res.new[ ])

"

## End JAGS Code ##

103 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM


attach(Canadareg)writeLines(modelstring,con="model.bug")Canadareg <- list(ASRREA01=ASRREA01,male=male,ASBG04=ASBG04,ASBGSBS=ASBGSBS,

ASBGSMR=ASBGSMR,ASBGSCR=ASBGSCR,ASBR05E=ASBR05E,ASBR05F=ASBR05F,ASBR05G=ASBR05G,nData=nData)

parameters = c("a","b1","b2","b3","b4","b5","b6","b7","b8","tau","fit","fit.new")lapply(Canadareg,summary)Canadareg1 <-jags.model("model.bug", data=Canadareg,n.chains=2, n.adapt=5000)

cat("Burning in the MCMC chain ...\n")update(Canadareg1, n.iter=5000)cat("Sampling from the final MCMC chain ... \n")codaSamples1 = coda.samples(Canadareg1,variable.names=parameters,


# Deviance Information Criterion

dic.pD <- dic.samples(Canadareg1, 5000, "pD")


plot(codaSamples1)acf(codaSamples1[[1]])geweke.diag(codaSamples1)geweke.plot(codaSamples1)gelman.diag(codaSamples1)gelman.plot(codaSamples1)

## Summarize output ##


104 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

# Posterior predictive check and DIC #

fit.all <- unlist(codaSamples1[,’fit’]) # unlist removes list params and places in vector #fitnew.all <- unlist(codaSamples1[,’fit.new’])

bayespval <- mean(fit.all > fitnew.all)bayespval

plot(fit.all, fitnew.all, xlab = "Actual Dataset", ylab = "Simulated Dataset",main = paste("Posterior Predictive Check", "\n", "Bayesian P-value =",

round(bayespval, 2)))abline(0,1)

# Deviance Information Criteriondic.pD

105 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Figure 10: Posterior predictive distribution plot and Bayesian p-value –Noninformative priors. Penalized deviance = 55315.

106 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM


## Bayesian Model Averaging ###

#------ Select DV and IVs and run BMA ----------#

install.packages("BMA")require(BMA)

#------- Read in data and delete missing data --------#Canadareg <-read.csv(file.choose(),header=TRUE) #browse to select data "Canada.csv"male <- ifelse(Canadareg$ITSEX==2,0,1)Canadareg <- cbind(Canadareg,male)Canadareg[Canadareg==999999]=NACanadareg <- na.omit(Canadareg)Canadareg <- Canadareg[sample(1: nrow(Canadareg), 5000,replace=F),]CanadaBMA <- subset(Canadareg, select=c(ASRREA01,male,ASBG04,ASBGSBS,ASBGSMR,ASBGSCR,

ASBR05E,ASBR05F,ASBR05G))

y <- CanadaBMA[,1]x <- CanadaBMA[,2:9]

CanadaBMA1 <- bicreg(x,y)summary(CanadaBMA1)par(mar=c(1,1,1,1))plot(CanadaBMA1)

107 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Call:bicreg(x = x, y = y)5 models were selectedBest 5 models (cumulative posterior probability = 1 ):

p!=0 EV SD model 1 model 2 model 3 model 4 model 5Intercept 100.0 356.4863 8.6761 360.295 351.759 359.582 350.999 353.876male 16.0 0.6364 1.6113 . . 3.962 3.991 .ASBG04 100.0 12.0446 0.8391 12.049 12.053 12.019 12.023 12.028ASBGSBS 100.0 4.6593 0.4729 4.745 4.563 4.687 4.504 4.649ASBGSMR 100.0 -3.8255 0.5079 -3.730 -3.982 -3.770 -4.023 -3.594ASBGSCR 100.0 13.6979 0.4647 13.762 13.626 13.739 13.602 13.638ASBR05E 44.8 1.8287 2.2585 . 3.932 . 3.951 5.043ASBR05F 6.1 -0.1949 0.8397 . . . . -3.202ASBR05G 0.0 0.0000 0.0000 . . . . .

nVar 4 5 5 6 6r2 0.229 0.230 0.230 0.231 0.231BIC -1268.212 -1267.486 -1265.017 -1264.364 -1264.169post prob 0.459 0.320 0.093 0.067 0.061

108 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Figure 11: BMA posterior distributions for model parameters. Note that thenarrow spike corresponds to p(βk = 0|y). Thus, in the case of ASBR05F,note that p(β > 0|y) = .06 and therefore p(β = 0|y) = 0.94. correspondingto the spike found in the BMA posterior distribution plot for ASBR05F.

109 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Bayesian Regression: Informative Priors

## Begin JAGS Code ##

modelstring = "

# Likelihood




# Informative Priors

a ˜ dnorm(315, .0001)b1 ˜ dnorm(-1.1, 1)b2 ˜ dnorm(10, 1)b3 ˜ dnorm(5, 1)b4 ˜ dnorm(-3.6, 1)b5 ˜ dnorm(12, 1)b6 ˜ dnorm(3, 1)b7 ˜ dnorm(-1.7, 1)b8 ˜ dnorm(-7.7, 1)tau ˜ dgamma(.01, 0.01)sigma ˜ dunif(0, 100)

"

## End JAGS Code ##

110 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM



Mean SD Naive SE Time-series SEa 348.0506040 7.273790846 0.2300174630 0.7317398168b1 -1.3401687 0.875120906 0.0276737529 0.0276737529b2 11.8532575 0.637352469 0.0201548548 0.0239238682b3 4.8294589 0.411043057 0.0129983228 0.0280732750b4 -3.0136563 0.442032998 0.0139783107 0.0322177582b5 13.4519672 0.392107157 0.0123995170 0.0271735194b6 4.8316944 0.818161822 0.0258725485 0.0405695591b7 0.7978413 0.790084490 0.0249846653 0.0360789736b8 -5.1173077 0.773486777 0.0244597995 0.0319707027tau 0.0002727 0.000005688 0.0000001799 0.0000001799


2.5% 25% 50% 75% 97.5%a 334.6187264 342.7984158 348.0887437 353.0064870 362.2468348b1 -3.0432010 -1.9215012 -1.3255125 -0.7679197 0.3509581b2 10.6225677 11.4304647 11.8582659 12.2618320 13.0869157b3 4.0072161 4.5437243 4.8343362 5.1214499 5.6145668b4 -3.8380865 -3.3225626 -3.0170771 -2.7164481 -2.1221862b5 12.6887714 13.1972972 13.4521819 13.6968835 14.2360563b6 3.2502445 4.2842621 4.8560718 5.4097235 6.3691876b7 -0.8749808 0.2865611 0.8319233 1.3251697 2.3398803b8 -6.5829456 -5.6320692 -5.1112628 -4.5813653 -3.7056908tau 0.0002615 0.0002689 0.0002726 0.0002764 0.0002846

111 / 165

Day 1 AM

Day 1 PM


Bayes Factors



Day 2 PM

Day 3 AM

Day 3 PM

Figure 12: Posterior predictive distribution plot and Bayesian p-value:Informative priors case. Penalized Deviance = 55457.

112 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PM

Day 2 PM: Student Analyses: Regression

1 Choose a country.

2 Run the full analysis with non-informative priors - obtain PPCsand DIC values.

3 Run full analysis again with informative priors on regressioncoefficients only based on Canada results - obtain PPCs andDIC values.

4 Compare DIC values.

5 Make a note of what happens to the PPIs when estimating themodel with informative priors.

113 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AMMultilevelmodeling

BayesianFactor Analysis

Day 3 PM

Day 3 AM: Advanced Topics: Multilevelmodeling and factor analysis

A common feature of data collection in the social sciences isthat units of analysis (e.g. students or employees) are nested inhigher organizational units (e.g. schools or companies,respectively).

The PIRLS study deliberately samples schools (within acountry) and then takes a sample of 4th grade students withinsampled schools.

Such data collection plans are generically referred to asclustered sampling designs.

114 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

In addition to being able to incorporate priors directly into amultilevel model, the Bayesian conception of multilevelmodeling has another advantage – namely it clears up a greatdeal of confusion in the presentation of multilevel models.

The literature on multilevel modeling attempts to make adistinction between so-called “fixed-effects” and“random-effects”.

Gelman and Hill have recognized this issue and present fivedifferent definitions of fixed and random effects.

The advantage of the Bayesian approach is that all parametersare assumed to be random. When conceived as a Bayesianhierarchical model, much of the confusion around terminologydisappears.

115 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Bayesian Random Effects ANOVA

Perhaps the most basic multilevel model is the random effectsanalysis of variance model.

As a simple example consider whether there are differencesamong G schools (g = 1, 2, . . . , G) on the outcome of studentreading performance y obtained from n students(i = 1, 2, . . . , n).

In this example, it is assumed that the G schools are a randomsample from a population of schools.

116 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

The model can be written as a two level hierarchical linearmodel as follows: Let

Level - 1

yig = βg + rig, (31)

The model for the school random effect can be written as

Level - 2

βg = µ+ ug, (32)

Inserting equation (32) into equation (31) yields.

yig = µ+ ug + rig. (33)

117 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM


Level - 1



Level - 2

βg = µ+ ug, (32)

Inserting equation (32) into equation (31) yields

.

yig = µ+ ug + rig. (33)

117 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM


Level - 1



Level - 2

βg = µ+ ug, (32)

Inserting equation (32) into equation (31) yields.

yig = µ+ ug + rig. (33)

117 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

For equation (33), we first specify the distribution of the readingperformance outcome yig given the school effect ug and thewithin school variance σ2. Specifically,

.

yig|ug, σ2 ∼ N(ug, σ2). (34)

Next specify the prior distribution on the remaining modelparameters. For this model, we specify conjugate priors

.

ug|µ, ω2 ∼ N(0, ω2), (35)µ ∼ N(b0, B0), (36)

σ2 ∼ inverse-Gamma(a, b), (37)

ω2 ∼ inverse-Gamma(a, b), (38)

118 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

We can arrange all of the parameters of the random effectsANOVA model into a vector θ and write the prior distribution as

.

p(θ) = p(u1, u2, . . . , uG, µ, σ2, ω2), (39)

where under the assumption of exchangeability of the schooleffects ug we obtain

.

p(θ) =G∏g=1

p(ug|µ, ω2)p(µ)p(σ2)p(ω2). (40)

119 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

We can arrange all of the parameters of the random effectsANOVA model into a vector θ and write the prior distribution as

.

p(θ) = p(u1, u2, . . . , uG, µ, σ2, ω2), (39)

where under the assumption of exchangeability of the schooleffects ug we obtain

.

p(θ) =

G∏g=1

p(ug|µ, ω2)p(µ)p(σ2)p(ω2). (40)

119 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Bayesian Multilevel Regression using rjags

## ONE-WAY RANDOM EFFECTS ANOVA: Non-Informative Priors #### Install and load packages ##install.packages("rjags")require(rjags)# Read in data and make transformations #

Canada <-read.csv(file.choose(),header=TRUE) # browse to select data "Canada.csv"Canada[Canada==999999]=NACanada1 <- na.omit(Canada)Canada1$id <- as.numeric(as.factor(Canada1$IDSCHOOL)) # This creates a new sequential ID.Canadasub <- subset(Canada1, select=c(ASRREA01,id))Canada2 <- Canadasub[1:5000,] # Just to reduce the size of the dataset.nData=NROW(Canada2)nGroups = length(unique(Canada2$id))

## JAGS Code ##modelstring = "model

# Likelihood

for(i in 1:nData) ASRREA01[i] ˜ dnorm(mu.y[i], tau1)mu.y[i] <- beta[id[i]] # Assigns the correct group mean to the specific student

for (g in 1:nGroups) beta[g] ˜ dnorm(mu, tau2)

120 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Bayesian Multilevel Regression using rjags

# Priorsmu ˜ dnorm(0.0,0.001)tau1 ˜ dgamma(.001,.001)tau2 ˜ dgamma(.001,.001)sigma1 <- 1/tau1sigma2 <- 1/tau2ICC <- sigma2/(sigma1+sigma2)

"attach(Canada2)writeLines(modelstring,con="model.bug")Canada2 <- list(ASRREA01=ASRREA01, id=id,nData=nData,nGroups=nGroups)parameters = c("mu","tau1", "tau2","sigma1","sigma2","ICC")lapply(Canada2,summary)CanadaM1 <-jags.model("model.bug", data=Canada2, n.chains=2, n.adapt=1000)

cat("Burning in the MCMC chain ...\n")update(CanadaM1, n.iter=5000)cat("Sampling from the final MCMC chain ... \n")codaSamples1 = coda.samples(CanadaM1,variable.names=parameters,

n.iter=100000, thin=10,seed=5555)


plot(codaSamples1)acf(codaSamples1[[1]])geweke.diag(codaSamples1)geweke.diag(codaSamples1)gelman.diag(codaSamples1)gelman.plot(codaSamples1)



121 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Figure 13: Trace and density plot for one-way random-effects ANOVA

122 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

0 100 250

−1.

00.

5

Lag

AC

F

ICC

0 100 250

−1.

00.

5

Lag

ICC & mu

0 100 250

−1.

00.

5

Lag

ICC & sgm1

0 100 250

−1.

00.

5

Lag

ICC & sgm2

0 100 250

−1.

00.

5

Lag

ICC & tau1

0 100 250

−1.

00.

5

Lag

ICC & tau2

−300 −100

−1.

00.

5

Lag

AC

F

mu & ICC

0 100 250

−1.

00.

5

Lag

mu

0 100 250

−1.

00.

5

Lag

mu & sgm1

0 100 250

−1.

00.

5

Lag

mu & sgm2

0 100 250

−1.

00.

5

Lag

mu & tau1

0 100 250

−1.

00.

5

Lag

mu & tau2

−300 −100

−1.

00.

5

Lag

AC

F

sgm1 & ICC

−300 −100

−1.

00.

5

Lag

sgm1 & mu

0 100 250

−1.

00.

5

Lag

sigma1

0 100 250

−1.

00.

5

Lag

sgm1 & sgm2

0 100 250

−1.

00.

5

Lag

sgm1 & tau1

0 100 250

−1.

00.

5

Lag

sgm1 & tau2

−300 −100

−1.

00.

5

Lag

AC

F

sgm2 & ICC

−300 −100

−1.

00.

5

Lag

sgm2 & mu

−300 −100

−1.

00.

5

Lag

sgm2 & sgm1

0 100 250

−1.

00.

5

Lag

sigma2

0 100 250

−1.

00.

5

Lag

sgm2 & tau1

0 100 250

−1.

00.

5

Lag

sgm2 & tau2

−300 −100

−1.

00.

5

Lag

AC

F

tau1 & ICC

−300 −100

−1.

00.

5

Lag

tau1 & mu

−300 −100

−1.

00.

5

Lag

tau1 & sgm1

−300 −100

−1.

00.

5

Lag

tau1 & sgm2

0 100 250

−1.

00.

5

Lag

tau1

0 100 250

−1.

00.

5

Lag

tau1 & tau2

−300 −100

−1.

00.

5

Lag

AC

F

tau2 & ICC

−300 −100

−1.

00.

5

Lag

tau2 & mu

−300 −100

−1.

00.

5

Lag

tau2 & sgm1

−300 −100

−1.

00.

5

Lag

tau2 & sgm2

−300 −100

−1.

00.

5

Lag

tau2 & tau1

0 100 250

−1.

00.

5

Lag

tau2

Figure 14: ACF plots for one-way random-effects ANOVA

123 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

[[1]]


ICC mu sigma1 sigma2 tau1 tau2-0.3539 1.7102 -2.0538 -0.7617 2.0515 0.9777

[[2]]


ICC mu sigma1 sigma2 tau1 tau2-0.2456 -0.6950 0.2759 -0.1807 -0.2853 0.2354

> gelman.diag(codaSamples1)Potential scale reduction factors:

Point est. Upper C.I.ICC 1 1mu 1 1sigma1 1 1sigma2 1 1tau1 1 1tau2 1 1

Multivariate psrf

1

124 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM



Mean SD Naive SE Time-series SEICC 0.1865017 0.015332535 0.00015332535 0.00015388992mu 552.4813990 1.667775858 0.01667775858 0.01667775858sigma1 3793.7522713 79.370977405 0.79370977405 0.79370977405sigma2 870.8763015 84.689699722 0.84689699722 0.86524045885tau1 0.0002637 0.000005515 0.00000005515 0.00000005515tau2 0.0011592 0.000113268 0.00000113268 0.00000113268


2.5% 25% 50% 75% 97.5%ICC 0.1576265 0.175838 0.1862509 0.1965435 0.2175658mu 549.1980070 551.371135 552.4835792 553.6074288 555.7565580sigma1 3641.4297289 3739.188086 3792.7877359 3845.9567853 3955.0834783sigma2 714.0057804 812.468735 867.9593784 924.9670708 1047.7049816tau1 0.0002528 0.000260 0.0002637 0.0002674 0.0002746tau2 0.0009545 0.001081 0.0011521 0.0012308 0.0014005

125 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Bayesian Multilevel Regression

Suppose interest centers on individual and schoolsocio-economic factors in term of their prediction of readingperformance in PIRLS.

We can write a substantive model as

Level - 1

READINGig = β0ig + β1ig(ASBH17A) + β2ig(ASBH17B) + rig,

where βkig (k = 1, 2, . . . ,K + 1) are the intercept and regressioncoefficients (slopes) that are allowed to vary over the Gschools. The variables ASBG17A = Fathers Education;ASBG17B = Mothers Education

126 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

We can model the intercept and slopes as a function of schoollevel predictors.

Level - 2

β0g = γ00 + γ01(ACBG03A)g + u0g, (41a)β1g = γ10 + γ11(ACBG03A)g + u1g, (41b)β2g = γ20 + γ21(ACBG03A)g + u2g, (41c)

where γ’s are the coefficients relating βkg to the school levelpredictors, and ACBG03A = percent of students in school gfrom economically disadvantaged homes.

127 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

The mixed model can be written as

Mixed Model

READINGig = γ00 + γ01(ACBG03A) (42)+ γ10(ASBG17A) + γ11(ASBG17B)(ACBG03A)

+ γ20(ASBG17B) + γ21(ACBG17B)(ACBG03A)

+ u0g + u1g(ASBG17A) + u2g(ASBG17B) + u3g

+ rig.

In terms of a Bayesian hierarchical model, the priors wouldhave to be chosen for σ2

g and the hyperparameters γg and ω2k.

128 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Bayesian Multilevel RegressionIntercept-Only Model

i## Multilevel Regression: Intercept with covariate model, Non-Informative Priors #### Install and load packages ##


# Read in data and make transformations #

Canada <-read.csv(file.choose(),header=TRUE) # browse to select data "Canada.csv"Canada[Canada==999999]=NACanada1 <- na.omit(Canada)Canada1$schid <- as.numeric(as.factor(Canada1$IDSCHOOL)) # This creates a new sequential ID.Canadasub <- subset(Canada1, select=c(ASRREA01,ASBH17A,ASBH17B,ACBG03A,schid))Canada2 <- Canadasub[1:5000,] # Just to reduce the size of the dataset.

nData=NROW(Canada2)nGroups = length(unique(Canada2$schid))

## JAGS Code ##

modelstring = "model

for (i in 1:nData) mu[i] <- alpha[schid[i]] + beta1[schid[i]]*ASBH17A[i] + beta2[schid[i]]*ASBH17B[i]ASRREA01[i] ˜ dnorm(mu[i], tau.c)

## Obtaining information for posterior pred. checks ###res[i] <- ASRREA01[i] - mu[i]ASRREA01.new[i] ˜ dnorm(mu[i], tau.c)res.new[i] <- ASRREA01.new[i] - mu[i]

for (j in 1:nGroups)

alpha[j] ˜ dnorm(gam00 + gam10*ACBG03A[j], alpha.tau)beta1[j] ˜ dnorm(beta1.mu, beta1.tau)beta2[j] ˜ dnorm(beta2.mu, beta2.tau)

129 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

# Priorsgam00 ˜ dnorm(0,.0001)gam10 ˜ dnorm(0,.0001)beta1.mu ˜ dnorm(0, 1.0E-4)beta2.mu ˜ dnorm(0, 1.0E-4)tau.c ˜ dgamma(1.0E-3, 1.0E-3)alpha.tau ˜ dgamma(1.0E-3, 1.0E-3)beta1.tau ˜ dgamma(1.0E-3, 1.0E-3)beta2.tau ˜ dgamma(1.0E-3, 1.0E-3)

# Transformationsalpha.sigma <- 1.0/sqrt(alpha.tau)beta1.sigma <- 1.0/sqrt(beta1.tau)beta2.sigma <- 1.0/sqrt(beta2.tau)sigma.c <- 1.0/sqrt(tau.c)

# Derived parameters for post. pred. checks #fit <- sum(res[ ])fit.new <- sum(res.new[ ])

"

130 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

attach(Canada2)writeLines(modelstring,con="model.bug")Canada2 <- list(ASRREA01=ASRREA01,ASBH17A=ASBH17A,ASBH17B=ASBH17B,ACBG03A=ACBG03A,schid=schid,nData=nData,nGroups=nGroups)parameters = c("beta1.mu","beta2.mu","tau.c","alpha.tau","beta1.tau","beta2.tau",

"alpha.sigma","beta1.sigma","beta2.sigma","sigma.c","gam00","gam10","fit","fit.new")lapply(Canada2,summary)CanadaM1 <-jags.model("model.bug", data=Canada2, n.chains=2, n.adapt=5000)

cat("Burning in the MCMC chain ...\n")update(CanadaM1, n.iter=5000)cat("Sampling from the final MCMC chain ... \n")codaSamples1 = coda.samples(CanadaM1,variable.names=parameters,


# Deviance Information Criteriondic1.pD <- dic.samples(CanadaM1, 5000, "pD")## Use coda to obtain diagnostic plots ##

plot(codaSamples1)acf(codaSamples1[[1]])geweke.diag(codaSamples1)gelman.diag(codaSamples1)gelman.plot(codaSamples1)



131 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

# Obtain posterior predictive check


fit.all <- unlist(codaSamples1[,’fit’])fitnew.all <- unlist(codaSamples1[,’fit.new’])


pdf("˜/desktop/multilevel1ppc.pdf")plot(fit.all, fitnew.all, xlab = "Actual Dataset", ylab = "Simulated Dataset",

main = paste("Posterior Predictive Check", "\n", "Bayesian P-value =",round(bayespval, 2)))

abline(0,1)dev.off()

# Deviance Information Criterion

dic1.pD

132 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

[[1]]Fraction in 1st window = 0.1Fraction in 2nd window = 0.5

alpha.sigma alpha.tau beta1.mu beta1.sigma beta1.tau beta2.mu beta2.sigma4.8937 -4.9593 -3.0113 -3.7969 1.3645 4.2897 -2.4304

beta2.tau gam00 gam10 sigma.c tau.c1.6123 -6.8907 0.6724 -4.4081 4.3770


alpha.sigma alpha.tau beta1.mu beta1.sigma beta1.tau beta2.mu beta2.sigma-1.4316 1.3128 2.0892 0.5495 0.1426 -4.0355 0.6756

beta2.tau gam00 gam10 sigma.c tau.c-1.3198 -1.7727 0.9650 0.6738 -0.6693

Potential scale reduction factors:Point est. Upper C.I.

alpha.sigma 1.08 1.30alpha.tau 1.09 1.34beta1.mu 1.24 1.82beta1.sigma 1.04 1.16beta1.tau 1.14 1.24beta2.mu 1.77 4.40beta2.sigma 1.78 4.04beta2.tau 1.37 2.91gam00 1.03 1.14gam10 1.00 1.00sigma.c 1.00 1.01tau.c 1.00 1.01

Multivariate psrf

1.89

133 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM


Mean SD Naive SE Time-series SEalpha.sigma 25.8467877 1.617111038 0.02286940362 0.09791926814alpha.tau 0.0015152 0.000198336 0.00000280490 0.00001223128beta1.mu 7.0357425 0.525386594 0.00743008847 0.11540910189beta1.sigma 0.3706912 0.457392750 0.00646851030 0.07845048441beta1.tau 234.7152012 480.698653137 6.79810554680 80.05237700237beta2.mu 4.9892031 0.880284909 0.01244910857 0.19489310019beta2.sigma 0.6932699 0.562461338 0.00795440453 0.09300373934beta2.tau 64.3524557 226.300731351 3.20037563451 27.03025807615gam00 481.5273343 7.335247174 0.10373606037 0.47988158616gam10 1.3938938 1.658904517 0.02346045266 0.05213100139sigma.c 60.2621432 0.635819024 0.00899183888 0.00964639716tau.c 0.0002755 0.000005814 0.00000008222 0.00000008222


2.5% 25% 50% 75% 97.5%alpha.sigma 22.4757574 24.8497642 25.8732026 26.9358635 28.8772000alpha.tau 0.0011992 0.0013783 0.0014938 0.0016194 0.0019796beta1.mu 6.0944046 6.6982800 6.9535584 7.4254125 8.2768577beta1.sigma 0.0239098 0.0662179 0.1638606 0.4925364 1.6452589beta1.tau 0.3694295 4.1221588 37.2435313 228.0597676 1749.2277855beta2.mu 3.3489585 4.2156964 5.0813464 5.7009877 6.4458430beta2.sigma 0.0376242 0.2342708 0.5348191 1.0643539 2.0241580beta2.tau 0.2440682 0.8827300 3.4961194 18.2206481 706.4243779gam00 466.8756075 476.5797154 481.6285088 486.4800117 495.6771803gam10 -1.8043387 0.2673627 1.3644368 2.5045133 4.6860523sigma.c 59.0450342 59.8239919 60.2593910 60.6938671 61.5096669tau.c 0.0002643 0.0002715 0.0002754 0.0002794 0.0002868

Mean deviance: 55205penalty 320.3Penalized deviance: 55525134 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

−15000 −10000 −5000 0 5000 10000 15000

−15

000

−50

000

5000

1000

015

000

Posterior Predictive Check Bayesian P−value = 0.51

Actual Dataset

Sim

ulat

ed D

atas

et

Figure 15: PPC for intercept-only model

135 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Bayesian Multilevel Regression Slopes andIntercept Model

## Multilevel Regression: Intercept and slope with covariate model, Non-Informative Priors ##

## Install and load packages ##


# Read in data and make transformations #

Canada <-read.csv(file.choose(),header=TRUE) # browse to select data "Canada.csv"Canada[Canada==999999]=NACanada1 <- na.omit(Canada)Canada1$schid <- as.numeric(as.factor(Canada1$IDSCHOOL)) # This creates a new sequential ID.Canadasub <- subset(Canada1, select=c(ASRREA01,ASBH17A,ASBH17B,ACBG03A,schid))Canada2 <- Canadasub[1:5000,] # Just to reduce the size of the dataset.

nData=NROW(Canada2)nGroups = length(unique(Canada2$schid))

#### JAGS Code ####

modelstring = "

# Likelihood


mu[i] <- alpha[schid[i]] + beta1[schid[i]]*ASBH17A[i] + beta2[schid[i]]*ASBH17B[i]ASRREA01[i] ˜ dnorm(mu[i], tau.c)

## Obtaining information for posterior pred. checks ###res[i] <- ASRREA01[i] - mu[i]ASRREA01.new[i] ˜ dnorm(mu[i], tau.c)res.new[i] <- ASRREA01.new[i] - mu[i]

136 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

for (j in 1:nGroups) alpha[j] ˜ dnorm(gam00 + gam10*ACBG03A[j], alpha.tau)beta1[j] ˜ dnorm(beta1.mu, beta1.tau)beta2[j] ˜ dnorm(gam01 + gam11*ACBG03A[j], beta2.tau)

# Priors

gam00 ˜ dnorm(0,.0001)gam10 ˜ dnorm(0,.0001)gam01 ˜ dnorm(0,.0001)gam11 ˜ dnorm(0,.0001)beta1.mu ˜ dnorm(0, 1.0E-4)beta2.mu ˜ dnorm(0, 1.0E-4)tau.c ˜ dgamma(1.0E-3, 1.0E-3)alpha.tau ˜ dgamma(1.0E-3, 1.0E-3)beta1.tau ˜ dgamma(1.0E-3, 1.0E-3)beta2.tau ˜ dgamma(1.0E-3, 1.0E-3)

# Transforming precision to standard deviation

alpha.sigma <- 1.0/sqrt(alpha.tau)beta1.sigma <- 1.0/sqrt(beta1.tau)beta2.sigma <- 1.0/sqrt(beta2.tau)sigma.c <- 1.0/sqrt(tau.c)

# Derived parameters for post. pred. checks #fit <- sum(res[ ])

fit.new <- sum(res.new[ ])

"#### End of JAGS Code ####

137 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

attach(Canada2)writeLines(modelstring,con="model.bug")Canada2 <- list(ASRREA01=ASRREA01,ASBH17A=ASBH17A,ASBH17B=ASBH17B,ACBG03A=ACBG03A,schid=schid,nData=nData,nGroups=nGroups)parameters = c("beta1.mu","tau.c","alpha.tau","beta1.tau","beta2.tau",

"alpha.sigma","beta1.sigma","beta2.sigma","sigma.c","gam00","gam10","gam01","gam11")lapply(Canada2,summary)CanadaM2 <-jags.model("model.bug", data=Canada2, n.chains=2, n.adapt=1000)

cat("Burning in the MCMC chain ...\n")update(CanadaM2, n.iter=5000)dic2.pD <- dic.samples(CanadaM2, 5000, "pD") # Deviance Information Criterioncat("Sampling from the final MCMC chain ... \n")codaSamples1 = coda.samples(CanadaM2,variable.names=parameters,



plot(codaSamples1)acf(codaSamples1[[1]])geweke.diag(codaSamples1)gelman.diag(codaSamples1)gelman.plot(codaSamples1)

138 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM




fit.all <- unlist(codaSamples1[,’fit’])fitnew.all <- unlist(codaSamples1[,’fit.new’])


plot(fit.all, fitnew.all, xlab = "Actual Dataset", ylab = "Simulated Dataset",main = paste("Posterior Predictive Check", "\n", "Bayesian P-value =",

round(bayespval, 2)))abline(0,1)

# Deviance Information Criterion and DIC Difference from Model 1 #

dic2.pDdiffdic(dic1.pD, dic2.pD)

139 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM


Mean SD Naive SE Time-series SEalpha.sigma 26.1470609 1.552307516 0.02195294342 0.08742986958alpha.tau 0.0014787 0.000182438 0.00000258006 0.00001016877beta1.mu 7.2461117 0.623289307 0.00881464191 0.27903545459beta1.sigma 0.1853041 0.207443773 0.00293369797 0.02588946802beta1.tau 242.5570675 398.661396022 5.63792353049 46.66551288483beta2.sigma 0.5058206 0.507555968 0.00717792534 0.08639145534beta2.tau 93.2323788 253.880423751 3.59041138490 33.54776990275gam00 474.2382962 14.739020474 0.20844122650 5.14104773211gam01 6.1213248 2.194435837 0.03103400922 0.98394110125gam10 3.0432823 3.849583227 0.05444132809 0.99255029344gam11 -0.3046821 0.628570497 0.00888932921 0.27634470958sigma.c 60.2879355 0.636268435 0.00899819450 0.00968073858tau.c 0.0002752 0.000005811 0.00000008218 0.00000008832

2. Quantiles for each variable:2.5% 25% 50% 75% 97.5%

alpha.sigma 23.002468 25.1842139 26.1627259 27.177529 29.1652713alpha.tau 0.001176 0.0013539 0.0014609 0.001577 0.0018900beta1.mu 6.201361 6.8354337 7.2414476 7.866749 8.1223891beta1.sigma 0.026203 0.0616696 0.1112855 0.230629 0.7320091beta1.tau 1.866244 18.8006242 80.7462729 262.940314 1456.4474738beta2.sigma 0.034131 0.1367330 0.3017579 0.733402 1.8960134beta2.tau 0.278174 1.8591544 10.9820639 53.487712 858.4246073gam00 448.314005 464.2906292 473.9135428 483.445131 507.6447931gam01 1.081862 4.7623234 6.2586801 7.182836 9.9804648gam10 -5.500616 0.6950690 3.1138073 5.695323 9.9758779gam11 -1.302419 -0.6708085 -0.3121674 0.030134 1.2610496sigma.c 59.048764 59.8632633 60.2856261 60.704551 61.5407563tau.c 0.000264 0.0002714 0.0002752 0.000279 0.0002868Mean deviance: 55188penalty 269.6Penalized deviance: 55458Difference: 67.38174140 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Bayesian Factor Analysis

We write the confirmatory factor analysis model as

CFA Model

y = α+ Λη + ε, (43)

Under conventional assumptions we obtain the modelexpressed in terms of the population covariance matrix Σ as

.

Σ = ΛΦΛ′ + Ψ, (44)

141 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

Conjugate Priors for Factor AnalysisParameters

Let θnorm = α,Λ be the set of free model parameters thatare assumed to follow a normal distribution and letθIW = Φ,Ψ be the set of free model parameters that areassumed to follow an inverse-Wishart distribution. Thus,

.

θnorm ∼ N(µ,Ω), (45)

The uniqueness covariance matrix Ψ is assumed to follow aninverse-Wishart distribution. Specifically,

.

θIW ∼ IW (R, δ), (46)

Different choices for R and δ will yield different degrees of“informativeness” for the inverse-Wishart distribution.

142 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

install.packages("rjags")require(rjags)# READ IN DATA AND PREPARE FOR JAGS

CanadaCFA <-read.csv(file.choose(),header=TRUE) #browse to select data "Canada.csv"CanadaCFA <- subset(CanadaCFA, select=c(ASBR08A,ASBR08B,ASBR08C,ASBR08D,ASBR08E,ASBR08F,ASBR08G))CanadaCFA[CanadaCFA==999999]=NACanadaCFA <- na.omit(CanadaCFA)CanadaCFA <- CanadaCFA[sample(1: nrow(CanadaCFA), 5000,replace=F),]

y = as.matrix(CanadaCFA)nData=NROW(y)

# Specify model in JAGSmodelstring = " # Likelhoodmodel for (i in 1 : nData) for (j in 1 : 7) # NCOL(y) = 7y[i,j] ˜ dnorm(mu[i,j],psi[j])

mu[i,1] <- a[1]+1*xi[i,1] # Factor 1, lam[1]=1mu[i,2] <- a[2]+lam[2]*xi[i,1]mu[i,4] <- a[4]+lam[4]*xi[i,1]mu[i,6] <- a[6]+lam[6]*xi[i,1]

mu[i,3] <-a[3]+1*xi[i,2] # Factor 2, lam[2]=1mu[i,5] <- a[5]+lam[5]*xi[i,2]mu[i,7] <- a[7]+lam[7]*xi[i,2]

xi[i,1:2] ˜ dmnorm(u[1:2],phi[1:2,1:2]) # Specifiy distribution of factors

143 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

# Priors on factor loadings (Non-informative)lam[1]<-1lam[2] ˜ dnorm(0, 10ˆ(-6)) # prior mean and prior precisionlam[4] ˜ dnorm(0, 10ˆ(-6))lam[6] ˜ dnorm(0, 10ˆ(-6))

lam[3]<-1lam[5] ˜ dnorm(0, 10ˆ(-6))lam[7] ˜ dnorm(0, 10ˆ(-6))

# Priors on intercepts (Non-informative)a[1] ˜ dnorm(0, 10ˆ(-6))a[2] ˜ dnorm(0, 10ˆ(-6))a[4] ˜ dnorm(0, 10ˆ(-6))a[6] ˜ dnorm(0, 10ˆ(-6))

a[3] ˜ dnorm(0, 10ˆ(-6))a[5] ˜ dnorm(0, 10ˆ(-6))a[7] ˜ dnorm(0, 10ˆ(-6))# Priors on Precisions (Non-informative)

for(j in 1:7) psi[j] ˜ dgamma(1, 0.001) # Precision of uniqunessesuniq[j] <- 1/psi[j] # unique variance

phi[1:2,1:2] ˜ dwish(R[1:2,1:2], 3) # phi is precision matrixSigma[1,1] <- phi[2,2]/(phi[1,1]*phi[2,2]-phi[2,1]ˆ2) # Sigma is covariance matrixSigma[2,1] <- -phi[1,2]/(phi[1,1]*phi[2,2]-phi[2,1]ˆ2)Sigma[2,2] <- phi[1,1]/(phi[1,1]*phi[2,2]-phi[2,1]ˆ2)Sigma[1,2] <- Sigma[2,1]"

144 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

writeLines(modelstring,con="model.bug")# Specify parameters

u=c(0,0) # Factor means set to zeroR=matrix(c(1,0,0,1),nrow=2)CanadaCFA <- list(y=y, nData=nData,u=u,R=R)

# RUN CHAIN# Initialize Modelparameters = c("a","lam","Sigma","uniq","phi") #Specify the Parameters to Be Estimated

cfaModel1 = jags.model("model.bug",data=CanadaCFA,n.chains=2, n.adapt=500)

# Obtain the Posterior Sample of Factor Analysis Parameters: #cat("Burning in the MCMC chain ...\n")update(cfaModel1, n.iter=500)cat("Sampling from the final MCMC chain ... \n")codaSamples1 = coda.samples(cfaModel1, variable.names=parameters,n.iter=1000, thin=1,seed=5555)

145 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

## Diagnostics ##

# Selected Trace plots and Density plots for factor loadings #plot(codaSamples1[[1]][,c(6,8,9,10,11)])

# Selected Trace plots and Density plots for variance terms #plot(codaSamples1[[1]][,c(16:22)])

# Auto-correlation plots for selected factor loadings (first chain) #par(mfrow=c(3,2))acf(codaSamples1[[1]][,6],main="Item 2")acf(codaSamples1[[1]][,8],main="Item 4")acf(codaSamples1[[1]][,9],main="Item 5")acf(codaSamples1[[1]][,10],main="Item 6")acf(codaSamples1[[1]][,11],main="Item 7")

par(mfrow=c(2,2))acf(codaSamples1[[1]][,1],main="Variance of Factor TEABEHA")acf(codaSamples1[[1]][,2],main="Covariance betw TEABEHA and STUDBEHA")acf(codaSamples1[[1]][,4],main="Variance of Factor STUDBEHA")acf(codaSamples1[[1]][,22],main="Error Variance of Item 1")

# Geweke and Gelman Diagnostic and Plotgeweke.diag(codaSamples1[[1]])gelman.diag(codaSamples1, multivariate = FALSE)

146 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

## Summary

# Posterior Mean, posterior SD and posterior probablity interval (PPI)options(scipen=999)summary(codaSamples1[[1]])

# posterior mean of factor correlation #options(scipen=999)mean(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]))

# posterior SD of factor correlation #options(scipen=999)sd(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]))

# 95% PPI of factor corr.#options(scipen=999)quantile(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]),c(.025,.975))

147 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM


Mean SD Naive SE Time-series SESigma[1,1] 12.1957649 0.31295770 0.009896591 0.08129586Sigma[2,1] 5.4785041 0.16348030 0.005169701 0.05874001Sigma[1,2] 5.4785041 0.16348030 0.005169701 0.05874001Sigma[2,2] 3.7166390 0.09514420 0.003008724 0.03366946a[1] 0.1246684 0.03026110 0.000956940 0.02425277a[2] 1.6291422 0.03242008 0.001025213 0.00489975a[3] 0.0791766 0.01817046 0.000574601 0.01234002a[4] 2.4556130 0.05139082 0.001625120 0.00910447a[5] 1.5167426 0.02873283 0.000908612 0.00251472a[6] 1.7455273 0.06138817 0.001941264 0.01318495a[7] 0.7244733 0.02224057 0.000703309 0.00197731lam[1] 1.0000000 0.00000000 0.000000000 0.00000000lam[2] 0.5620899 0.01103100 0.000348831 0.00203238lam[3] 1.0000000 0.00000000 0.000000000 0.00000000lam[4] 0.2698516 0.01394463 0.000440968 0.00238223lam[5] 0.4753352 0.01430177 0.000452262 0.00120354lam[6] 0.4348302 0.01859912 0.000588156 0.00423937lam[7] 0.5134271 0.01046553 0.000330949 0.00076241phi[1,1] 0.2428598 0.00489724 0.000154864 0.00026402phi[2,1] -0.3579597 0.00791961 0.000250440 0.00027702phi[1,2] -0.3579597 0.00791961 0.000250440 0.00027702phi[2,2] 0.7969053 0.01546323 0.000488990 0.00050366uniq[1] 0.0003776 0.00009149 0.000002893 0.00004753uniq[2] 0.3347643 0.00705603 0.000223131 0.00023797uniq[3] 0.0008867 0.00048109 0.000015214 0.00030639uniq[4] 0.7729350 0.01507422 0.000476689 0.00047669uniq[5] 0.9446526 0.01941626 0.000613996 0.00061400uniq[6] 0.7013185 0.01381563 0.000436888 0.00045417uniq[7] 0.5927233 0.01199405 0.000379285 0.00036103

148 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM


2.5% 25% 50% 75% 97.5%Sigma[1,1] 11.5758862 11.9746797 12.2030811 12.4045067 12.7691310Sigma[2,1] 5.1684885 5.3659367 5.4814118 5.5825546 5.7911302Sigma[1,2] 5.1684885 5.3659367 5.4814118 5.5825546 5.7911302Sigma[2,2] 3.5385586 3.6497954 3.7169181 3.7788233 3.9054663a[1] 0.0809455 0.0982338 0.1186369 0.1452640 0.1816772a[2] 1.5706381 1.6070419 1.6269780 1.6489623 1.7000288a[3] 0.0462116 0.0674402 0.0784316 0.0855467 0.1146649a[4] 2.3532690 2.4196956 2.4590488 2.4886161 2.5564497a[5] 1.4631486 1.4970458 1.5160299 1.5375097 1.5743647a[6] 1.6358590 1.7009827 1.7469751 1.7879749 1.8700960a[7] 0.6823280 0.7094545 0.7242721 0.7397865 0.7676564lam[1] 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000lam[2] 0.5401068 0.5542617 0.5625471 0.5698119 0.5832293lam[3] 1.0000000 1.0000000 1.0000000 1.0000000 1.0000000lam[4] 0.2421761 0.2609658 0.2688939 0.2790611 0.2991232lam[5] 0.4469051 0.4657162 0.4755650 0.4851953 0.5027134lam[6] 0.3966582 0.4212391 0.4349353 0.4491616 0.4672959lam[7] 0.4936497 0.5062420 0.5130664 0.5201479 0.5340757phi[1,1] 0.2335481 0.2394177 0.2429008 0.2461530 0.2528012phi[2,1] -0.3729033 -0.3634434 -0.3581288 -0.3526214 -0.3431805phi[1,2] -0.3729033 -0.3634434 -0.3581288 -0.3526214 -0.3431805phi[2,2] 0.7655042 0.7867512 0.7966551 0.8074313 0.8273683uniq[1] 0.0002435 0.0003018 0.0003713 0.0004437 0.0005796uniq[2] 0.3217185 0.3295547 0.3344764 0.3394534 0.3494967uniq[3] 0.0004859 0.0005651 0.0006602 0.0010475 0.0022210uniq[4] 0.7435506 0.7625202 0.7724621 0.7828628 0.8032921uniq[5] 0.9083740 0.9319449 0.9444711 0.9576743 0.9823026uniq[6] 0.6754621 0.6920695 0.7014459 0.7104762 0.7290440uniq[7] 0.5687867 0.5849021 0.5927457 0.6003985 0.6168780

149 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM



Day 3 PM

> # posterior mean of factor correlation #> options(scipen=999)> mean(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]))[1] 0.8136581>> # posterior SD of factor correlation #> options(scipen=999)> sd(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]))[1] 0.005619092>> # 95% PPI of factor corr.#> options(scipen=999)> quantile(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]),c(.025,.975))

2.5% 97.5%0.8022316 0.8245019> # posterior mean of factor correlation #> options(scipen=999)> mean(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]))[1] 0.8136581>> # posterior SD of factor correlation #> options(scipen=999)> sd(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]))[1] 0.005619092>> # 95% PPI of factor corr.#> options(scipen=999)> quantile(codaSamples1[[1]][,2]/sqrt(codaSamples1[[1]][,1]*codaSamples1[[1]][,4]),c(.025,.975))

2.5% 97.5%0.8022316 0.8245019

150 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up

The BayesianAdvantage

Final Thoughts

Wrap-up: Some Philosophical Issues

Bayesian statistics represents a powerful alternative tofrequentist (classical) statistics, and is therefore, controversial.

The controversy lies in differing perspectives regarding thenature of probability, and the implications for statistical practicethat arise from those perspectives.

The frequentist framework views probability as synonymouswith long-run frequency, and that the infinitely repeatingcoin-toss represents the canonical example of the frequentistview. frequency.

In contrast, the Bayesian viewpoint regarding probability was,perhaps, most succinctly expressed by de Finetti

151 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

.Probability does not exist.

- Bruno de Finetti

152 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

That is, probability does not have an objective status, but ratherrepresents the quantification of our experience of uncertainty.

For de Finetti, probability is only to be considered in relation toour subjective experience of uncertainty, and, for de Finetti,uncertainty is all that matters.

.“The only relevant thing is uncertainty – the extent of our known knowledge andignorance. The actual fact that events considered are, in some sense, determined, orknown by other people, and so on, is of no consequence.” (pg. xi).

The only requirement then is that our beliefs be coherent,consistent, and have a reasonable relationship to anyobservable data that might be collected.

153 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

Summarizing the Bayesian Advantage

The major advantages of Bayesian statistical inference overfrequentist statistical inference are

1 Coherence

2 Handling non-nested models

3 Flexibility in handling complex data structures

4 Inferences based on data actually observed

5 Quantifying evidence

6 Incorporating prior knowledge

154 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

Subjective v. Objective Bayes

See papers in Bayesian Analysis, 1 (3), 2006.

Subjective Bayesian practice attempts to bring prior knowledgedirectly into an analysis. This prior knowledge represents theanalysts (or others) degree-of-uncertainity.

An analyst’s degree-of-uncertainty is encoded directly into theprior distribution, and specifically in the degree of precisionaround the parameter of interest.

The advantages according to Press (2003) include

1 Subjective priors are proper

2 Priors can be based on factual prior knowledge

3 Small sample sizes can be handled.

155 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

Subjective v. Objective Bayes

See papers in Bayesian Analysis, 1 (3), 2006.

Subjective Bayesian practice attempts to bring prior knowledgedirectly into an analysis. This prior knowledge represents theanalysts (or others) degree-of-uncertainity.

An analyst’s degree-of-uncertainty is encoded directly into theprior distribution, and specifically in the degree of precisionaround the parameter of interest.

The advantages according to Press (2003) include

1 Subjective priors are proper

2 Priors can be based on factual prior knowledge

3 Small sample sizes can be handled.155 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

The disadvantages to the use of subjective priors according toPress (2003) are

1 It is not always easy to encode prior knowledge into probabilitydistributions.

2 Subjective priors are not always appropriate in public policy orclinical situations.

3 Prior distributions may be analytically intractable unless they areconjugate priors.

156 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

Within objective Bayesian statistics, there is disagreementabout the use of the term “objective”, and the related term“non-informative”.

Specifically, there are a large class of so-called reference priors(Kass and Wasserman,1996).

An important viewpoint regarding the notion of objectivity in theBayesian context comes from Jaynes (1968).

For Jaynes, the “personalistic” school of probability is to bereserved for

.“...the field of psychology and has no place in applied statistics. Or, tostate this more constructively, objectivity requires that a statistical analysisshould make use, not of anybody’s personal opinions, but rather thespecific factual data on which those opinions are based.”(pg. 228)

157 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

In terms of advantages Press (2003) notes that

1 Objective priors can be used as benchmarks against whichchoices of other priors can be compared

2 Objective priors reflect the view that little information is availableabout the process that generated the data

3 An objective prior provides results equivalent to those based on afrequentist analysis

4 Objective priors are sensible public policy priors.

158 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

In terms of disadvantages of objective priors, Press (2003)notes that

1 Objective priors can lead to improper results when the domain ofthe parameters lie on the real number line.

2 Parameters with objective priors are often independent of oneanother, whereas in most multi-parameter statistical models,parameters are correlated.

3 Expressing complete ignorance about a parameter via anobjective prior leads to incorrect inferences about functions of theparameter.

159 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

Kadane (2011) states, among other things:

.“The purpose of an algorithmic prior is to escape from theresponsibility to give an opinion and justify it. At the sametime, it cuts off a useful discussion about what isreasonable to believe about the parameters. Without sucha discussion, appreciation of the posterior distribution ofthe parameters is likely to be less full, and importantscientific information may be neglected.”(pg. 445)

160 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

Kadane (2011) states, among other things:.

“The purpose of an algorithmic prior is to escape from theresponsibility to give an opinion and justify it. At the sametime, it cuts off a useful discussion about what isreasonable to believe about the parameters. Without sucha discussion, appreciation of the posterior distribution ofthe parameters is likely to be less full, and importantscientific information may be neglected.”(pg. 445)

160 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

Final Thoughts: Evidenced-based Bayes

The subjectivist school allows for personal opinion to be elicitedand incorporated into a Bayesian analysis.

The subjectivist school places no restriction on the source,reliability, or validity of the elicited opinion.

The objectivist school views personal opinion as the realm ofpsychology with no place in a statistical analysis.The objectivist school would require formal rules for choosingreference priors.

161 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

“Subjectivism” within the Bayesian framework runs the gamutfrom the elicitation of personal beliefs to making use of the bestavailable historical data available to inform priors.

I agree with Jaynes (1968) – namely that the requirements ofscience demand reference to “specific, factual data on whichthose opinions are based” (pg. 228).

This view is also consistent with Leamer’s hierarchy ofconfidence on which priors should be ordered.

We may refer to this view as an evidence-based form ofsubjective Bayes which acknowledges (1) the subjectivity thatlies in the choice of historical data; (2) the encoding of historicaldata into hyperparameters of the prior distribution; and (3) thechoice among competing models to be used to analyze thedata.

162 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

What if factual historical data are not available?

Berger (2006) states that reference priors should be used “inscenarios in which a subjective analysis is not tenable”,although such scenarios are probably rare.

The goal, nevertheless, is to shift the practice of Bayesianstatistics away from the elicitation of personal opinion (expert orotherwise) which could, in principle, bias results toward aspecific outcome, and instead move Bayesian practice towardthe warranted use prior objective empirical data for thespecification of priors.

The specification of any prior should be explicitly warrantedagainst observable, empirical data and available for critique bythe relevant scholarly community.

163 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

To conclude, the Bayesian school of statistical inference is,arguably, superior to the frequentist school as a means ofcreating and updating new knowledge in the social sciences.

An evidence-based focus that ties the specification of priors toobjective empirical data provides stronger warrants forconclusions drawn from a Bayesian analysis.

In addition predictive criteria should always be used as a meansof testing and choosing among Bayesian models.

As always, the full benefit of the Bayesian approach to researchin the social sciences will be realized when it is more widelyadopted and yields reliable predictions that advance knowledge.

164 / 165

Day 1 AM

Day 1 PM

Day 2 AM

Day 2 PM

Day 3 AM

Day 3 PMWrap-up


Final Thoughts

.

Tusen takk

165 / 165

workshop on bayesian statistics for education research › cemo › dokumenter › kaplan... ·...

Documents