Random and mixed effects

ANOVA

STAT 526

Professor Olga Vitek

January 27, 2011

Reading: KNNL Ch. 25

Faraway Ch. 8

2

One-way random effects

ANOVA

2-1

Random Effects vs Fixed

Effects

• Consider factor with numerous levels

• Want to draw inference on population of levels

• Not concerned with any specific levels

• Example of difference (1=fixed, 2=random)

1. Compare reading ability of 10 2nd grade classesin NY

2. Compare variability among all 2nd grade classesin NY

1. Select a = 10 specific classes of interest.Randomly choose n students from each class-room.

2. Randomly choose a = 10 classes from largenumber of classes. Randomly choose n studentsfrom each classroom.

• Inference broader in random effects case

• Levels chosen randomly → inference on population

2-2

Example: KNNL p.1036

• Interested in studying the variability in therating of job applicants

– Variability among applicants

– Variability among personnel officers

• Y is the job applicant rating

• Factor: officer/interviewer (r = 5)

• Interviewers selected at random from pop-

ulation of personnel officers

• Twenty applicants randomly and equally

assigned (n = 4) to officers

2-3

Means Plot

X <- read.table("CH25TA01.txt", sep="", as.is=TRUE,header=FALSE)

dimnames(X)[[2]] <- c("rating", "officer", "replicate")

pdf("ch25t01.pdf", width=6)plot(X$officer, X$rating, xlab="Oficer is", ylab="Rating",

cex=2, cex.lab=2)lines(1:5, with(X, tapply(rating, officer, mean)))legend("top", lty=1, "mean", cex=2)dev.off()

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

1 2 3 4 5

5060

7080

90

Oficer id

Rat

ing

mean

2-4

Data for One-way Random

Effects Model

• Y is the response variable

• Factor with levels i = 1,2, ..., r

• Yij is the jth observation from cell i

• Consider j = 1,2, ..., n

2-5

Random Effects Model

• Cell means model

Yij = µi + εij,

µiiid∼ N(µ, σ2

µ), εijiid∼ N(0, σ2)

- µi and εij are independent

• Factor effects model

Yij = µ+ τi + εij,

τiiid∼ N(0, σ2

τ ), εijiid∼ N(0, σ2)

- τi and εij are independent

• Called Model II in KNNL

2-6

Equivalent Specification in

Terms of Conditional

Distributions

• Cell means model

µiiid∼ N(µ, σ2

µ)

Yij|µiind∼ N(µi, σ

2)

• Factor effects model

τiiid∼ N(0, σ2

µ)

Yij|τiind∼ N(µ+ τi, σ

2)

• Similarity to Bayesian methods:- parameters µi have a distribution

• Difference from Bayesian methods:-µ, σ2 and σ2

µ do not have a distribution

2-7

Implications of the Random

Effects Model

• There are TWO variance parameters

• Cell means are random variables, not pa-rameters

• Yij ∼ N(µ, σ2µ + σ2)

• The observations are not independent:Cov(Yij, Yi′j′)=0 but Cov(Yij, Yij′)=σ2

µ

• E.g. if r = 2 and n = 2,

Var

Y11

Y12

Y21

Y22

=

σ2µ + σ2 σ2 0 0σ2 σ2

µ + σ2 0 00 0 σ2

µ + σ2 σ2

0 0 σ2 σ2µ + σ2

2-8

ANOVA Table

Source DF SS EMS

Trt r − 1r∑

i=1n(Yi· − Y··)2 σ2 + nσ2

µ

Error nr − rr∑

i=1

n∑j=1

(Yij − Yi·)2 σ2

Total nr − 1

• H0 : σ2µ = 0 vs Ha : σ2

µ > 0

• F0 = MS(Trt)MSE =

H0∼ Fr−1,nr−r

• Conclusion pertains to entire population

• Reject H0:

– The expected ratings of the population ofofficers has a non-zero variance

– The company need to improve consistencybetween the interviewers

2-9

Inference About E{Yij}

• E{Yij} = µ·

• E{Yij} = Y··

• V ar{Y··} =σ2µr + σ2

rn =nσ2

µ+σ2

rn

• V ar{Y··} = MS(Trt)rn

• Testing: t = Y··−µH0·√

MS(Trt)/rn

H0∼ Studentr−1

• CI: Y·· ± t(1− α/2, r − 1)√MS(Trt)/rn

2-10

Intraclass Correlation

Coefficient

• percentage of total variation due to factor

σ2µ

σ2µ + σ2

=σ2µ

σ2Y

• correlation between two observations withthe same i (e.g. between two evaluationsof a same officer)

ρIC =Cov(Yij, Yik)√

Var(Yij)Var(Yik)=

σ2µ

σ2Y

• Small intraclass correlation= little variation among officers

• Large intraclass correlation= little variation among applicants

2-11

Confidence Interval forσ2µ

σ2µ+σ2

• MS(Trt) and MSE are independent r. v.

MS(Trt)nσ2

µ+σ2 / MSEσ2 ∼ F (r − 1, rn− r)

P{Fα/2, r−1, rn−r ≤ MS(Trt)MSE

· σ2

nσ2µ+σ2 ≤ F1−α/2, r−1, rn−r} = 1− α

• Solve forσ2µ

σ2µ+σ2:

L

L + 1≤

σ2µ

σ2 + σ2µ

≤U

U + 1

where

L =1

n

(MS(Trt)

MSE F1−α/2, r−1, rn−r− 1

)U =

1

n

(MS(Trt)

MSE Fα/2, r−1, rn−r− 1

)

• If low limit of the CI is negative, set to zero

2-12

Confidence Interval for σ2

• σ2 = MSE

• Known distribution of a function of MSE

r(n−1)MSEσ2 ∼ χ2(rn− r)

P{χ2α/2, rn−r ≤

r(n−1)MSEσ2 ≤ χ2

1−α/2, rn−r} = 1− α

• Solve for σ2:

r(n− 1)MSE

χ21−α/2, rn−r

≤ σ2 ≤r(n− 1)MSE

χ2α/2, rn−r

• Replace nr with nT in unbalanced designs

2-13

Point Estimate for σ2µ

• E{MS(Trt)} = σ2 + nσ2µ

• E{MSE} = σ2

• σ2µ = E{MS(Trt)}−E{MSE}

n

=

(1

n

)MS(Trt) +

(−

1

n

)MSE

• σ2µ = MS(Trt)−MSE

n

=

(1

n

)MS(Trt) +

(−

1

n

)MSE

• σ2µ is estimated by a linear combination of indepen-

dent MS

• Can adjust denominator in unbalanced experiments

• σ2µ can be negative

2-14

Confidence Interval for σ2µ

• σ2µ = c1E{MS1}+ . . .+ chE{MSh} = L

• σ2µ = c1MS1 + . . .+ chMSh = L

• dfLL

approx∼ χ2(df)

P{χ2α/2, df ≤

dfLL≤ χ2

1−α/2, df}approx

= 1− α

• Solve for L:

df L

χ21−α/2, df

≤ L ≤df L

χ2α/2, df

• Satterwaite approximation of df:

df =(c1MS1 + . . .+ chMSh)2

(c1MS1)2

df1+ . . .+ (chMSh)2

dfh

• Use the nearest integer for the df

2-15

Job Applicants Example:

ANOVA-based estimation

• Use aov in R; could also use lm or glm

# --------------fit fixed-effects ANOVA-------------X$officerF <- factor(X$officer)fit1 <- aov(rating ~ officerF, data=X)

> summary(fit1)Df Sum Sq Mean Sq F value Pr(>F)

officerF 4 1579.70 394.93 5.389 0.006803 **Residuals 15 1099.25 73.28---Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1

> coef(fit1)(Intercept) officerF2 officerF3 officerF4 officerF5

75.00 -4.50 -20.25 4.75 2.25

# ---Calculate estimate of sigma2_mu from the ANOVA table---> table(X$officerF)1 2 3 4 54 4 4 4 4

> sigma2MuHat <- (394.93 - 73.28)/4> sigma2MuHat[1] 80.4125

2-16

Pros and Cons of

ANOVA-based Inference

• Advantages

– Explicit formulae

– Clear insight into the mechanism

• Disadvantages

– σµ can be negative

– For unbalanced designs and in presence of multi-ple factors, ANOVA decomposition is not unique(depends on the order of the factors), and SSare not orthogonal. Therefore the inferencedoes not hold.

– Formulae become more complex with more fac-tors

• Alternative solution:

– Maximum likelihood or restricted maximum like-lihood

2-17

Maximum Likelihood

Estimation

• Define yi = [yi1, yi2, . . . , yini]′ and Vi the

ith block of the variance matrix V

• Assumptions: yi ∼MVN(µ1ni,Vi)

• Likelihood:

L =r∏

i=1

(2π)−ni/2|Vi|−1/2exp

[−

1

2(yi − µ1n1)

′V−1i (yi − µ1n1)

]

• Solutions maximizing log-likelihoodin balanced case:

– µ = Y··

– σ2 = MSE

– σ2µ = 1

n

[(1− 1

r

)MS(Trt)−MSE

]2-18

Problems With MLE

• σ2µ is biased towards smaller values

E{σ2µ} =

1

n

[(1−

1

r

)EMS(Trt)− EMSE

]=

(1−

1

r

)σ2µ − σ2/rn 6= σ2

µ

• σ2µ can still be negative

• Solutions in the restricted parameter space:

σ2(MLE)µ =

{σ2µ, if σ2

µ ≥ 00, otherwise

;

σ2(MLE) =

{σ2, if σ2

µ ≥ 0SSTrn , otherwise

2-19

Interviewers Example: ML

• Use lmer in R; could also use lme (older)

library(lme4)fit2 <- lmer(rating ~ 1 + (1|officerF), data=X, REML=FALSE)

• The data are grouped by ’officer’• The random effect is constant within each group

> summary(fit2)Linear mixed model fit by maximum likelihoodFormula: rating ~ 1 + (1 | officerF)

Data: XAIC BIC logLik deviance REMLdev

156.0 158.9 -74.98 150.0 145.3Random effects:Groups Name Variance Std.Dev.officerF (Intercept) 60.664 7.7887Residual 73.283 8.5606

Number of obs: 20, groups: officerF, 5

Fixed effects:Estimate Std. Error t value

(Intercept) 71.450 3.974 17.98

• Due to the bias, the ML-based estimate of varianceis smaller than the ANOVA-based estimate.

• This can result in too much optimism

2-20

Restricted/Residual ML

(REML)

• Apply ML to linear combinations of y K′y

– K selected such that the distribution of K′y doesnot involve µ (or, more generally, any fixed ef-fects)

– Estimates of variance components are invariantto fixed effects

– Implicitly takes into account the df for fixed ef-fects

• Simple example:

– Suppose Yi ∼ N(µ, σ2), i = 1, . . . , n

– Define Y· =n∑i=1

Yi/n, Syy =n∑i=1

(Yi − Y·)2

– σ2ML = Syy/n, (biased)

– σ2REML = Syy/(n− 1), (unbiased)

• Estimate fixed effects as a second step

2-21

REML in 1-Way Random

Effects ANOVA

• Factor effects model:

– The part of the likelihood that does not involvefixed effects is the part that does not involve µ

L(µ, σ2, σ2µ | Y) = L( µ | Y·· ) · L( σ2, σ2

µ | SS(Trt), SSE )

– Use the second product in the likelihood as thelikelihood for REML

• Solutions in balanced case:

– µ = Y··, σ2 = MSE, σ2µ = 1

n[MS(Trt)−MSE]

• Solutions maximizing log-likelihood in therestricted parameter space:

σ2(REML)µ =

{σ2µ, if σ2

µ ≥ 00, otherwise

; same as ANOVA-based

σ2(REML) =

{σ2, if σ2

µ ≥ 0SSTrn−1

, otherwise

2-22

Interviewers Example:

REML

fit3 <- lmer(rating ~ 1 + (1|officerF), data=X, REML=TRUE)

> summary(fit3)Linear mixed model fit by REMLFormula: rating ~ 1 + (1 | officerF)

Data: XAIC BIC logLik deviance REMLdev

151.2 154.2 -72.62 150.0 145.2

Random effects:Groups Name Variance Std.Dev.officerF (Intercept) 80.410 8.9672Residual 73.283 8.5606

Number of obs: 20, groups: officerF, 5

Fixed effects:Estimate Std. Error t value

(Intercept) 71.450 4.443 16.08

• In balanced designs, σ2µ is similar to the

ANOVA-based estimation

2-23

Predicting Random Effects

• Specification withconditional distributions:

Yij|τiind∼ N(µ+ τi, σ

2); τiiid∼ N(0, σ2

µ)

– Can be of interest to predict τi given the data

– Best predictor is E{τi | Y}

• In 1-way ANOVA with random effects:E{τi | Y} = E{τi | Yi·}

= E{τi}+ cov(τi, Yi·)[var(Yi·)

]−1 (Yi· − E{Yi·}

)= 0 + σ2

µ ·1

σ2µ + σ2/ni

·(Yi· − µ

)E{τi | y} =

σ2µ

σ2µ + σ2/ni

·(Yi· − µ

)• In balanced experiments:E{τi | Y} =

σ2µ

σ2µ + σ2/n

·(Yi· − Y··

)Random effects

τi =(Yi· − Y··

)Fixed effects

- shrinkage of the estimation of random effects ascompared to fixed effects

2-24

Extracting Parameters and

Testing in R

• Extracting values of parameters

fixef(fit3) # predicted fixed effectsvcov(fit3) # variance-covariance of fixed effectsranef(fit3) # predicted random effectsfitted(fit3) # yHat = fixedEffectsHat + randomEffectsHat

• Likelihood Ratio Test H0 : σ2µ = 0 vs Ha : σ2

µ 6= 0

• REML-based likelihoods are not comparable for mod-els that differ in random effects

• Use ML-based estimation

• The test is approximate since based on biased ML

fit4 <- lm(rating ~ 1, data=X, REML=FALSE)

> pchisq( as.numeric(2*( logLik(fit2) - logLik(fit4))) ,1, lower=FALSE)

[1] 0.02919295

• Conclusion: at α = 0.05, there is a significantvariation between the officers

• Weaker evidence than with ANOVA-based F test

2-25

Two-way random effects

ANOVA

(Model II in KNNL)

2-26

Example: KNNL 25.15

• Interested in the fuel efficiency (mpg)

• Response: mpg

– Yijk is the kth observed mpg from driver i andcar j, with k = 1,2, ..., nij

• Two random factors

– Factor A: Driver, levels i = 1,2, ..., a

– Factor B: Car (same model), levels j = 1,2, ..., b

– Each driver drove each car twice over same 40mile course

- balanced design

• Scientific question:

– How much of the overall variability is due todriver and/or car?

2-27

Means PlotX <- read.table("CH25PR15.txt", sep="", as.is=TRUE,

header=FALSE)dimnames(X)[[2]] <- c("mpg", "driver", "car", "replicate")

pdf("ch25pr15.pdf", width=6)plot(X$driver, X$mpg, xlab="Driver id", ylab="Miles/gallon")

m <- with(X, tapply(mpg, list(car, driver), mean))for (i in 1:5) { lines(1:4, m[i,], col=i) }

legend("topright", lty=1, "car", cex=1.5)dev.off()

●●

●

●

●●

●●

●●

●

●

●●

●●

●●

●●

●

●

●●

●●

●●

●●

●●

●●

●

●

●

●●●

1.0 1.5 2.0 2.5 3.0 3.5 4.0

2628

3032

3436

Driver id

Mile

s/ga

llon

car

2-28

Random Factor Effects

Model

• In mathematical notation:

Yijk = µ+ αi + βj + (αβ)ij + εijk

i = 1, . . . , a, j = 1, . . . , b, k = 1, . . . , n

– µ - grand mean (or another reference)

– αi: random deviation of the ith level of factor

A from the reference; αiiid∼ N(0, σ2

α)

– βj: random deviation of the jth level of factor

B from the reference; βjiid∼ N(0, σ2

β)

– (αβ)ij: the joint random effect of ith level of A

and jth level of B; (αβ)ijiid∼ N(0, σ2

αβ)

– εijk: random error; εijkiid∼ N(0, σ2)

– All random terms are independent

2-29

Covariance Structure

• There are for variance parameters

– Covariances:

Cov(Yijk, Yijk) = σ2 + σ2α + σ2

β + σ2αβ

Cov(Yijk, Yijk′) = σ2α + σ2

β + σ2αβ

Cov(Yijk, Yij′k) = σ2α

Cov(Yijk, Yi′jk) = σ2β

Cov(Yijk, Yi′j′k) = 0

• Yijk are not independent

• Questions

– Percentage of total variation due to each factor

– Percentage of cell means variation(i.e., ignoring error variance)

– Pairwise comparisons between levels of factorsare not appropriate

2-30

ANOVA Table

Source DF EMS

A a− 1 σ2 + nσ2αβ + bnσ2

α

B b− 1 σ2 + nσ2αβ + anσ2

β

AB (a− 1)(b− 1) σ2 + nσ2αβ

Error ab(n− 1) σ2

Total ab− 1

• Parameter estimates using mean squares

σ2 = MSE

σ2αβ = (MSAB−MSE)/n

σ2β = (MSB−MSAB)/an

σ2α = (MSA−MSAB)/bn

• Estimates can be negative

• Same procedures for CI, and same adjust-ments for df, as in the one-way randomeffects ANOVA

2-31

ANOVA-Based Hypothesis

Tests

Three tests of variance, no hierarchy

• H0AB: σ2αβ = 0 vs H1AB: σ2

αβ > 0

F = MS(AB)MSE

H0∼ F(a−1)(b−1), ab(n−1)

• H0A: σ2α = 0 vs H1A: σ2

α > 0

F = MS(A)MS(AB)

H0∼ F(a−1), (a−1)(b−1)

• H0B: σ2β = 0 vs H1B: σ2

β > 0

F = MS(B)MS(AB)

H0∼ F(b−1), (a−1)(b−1)

2-32

Car Example:

ANOVA-Based Approach

#--------Fit the fixed-effects 2-way ANOVA---------X$driverF <- factor(X$driver)X$carF <- factor(X$car)fit1 <- aov(mpg ~ driverF*carF, data=X)

#---------------Summary of fit--------------------summary(fit1)

Df Sum Sq Mean Sq F value Pr(>F)driverF 3 280.285 93.428 531.60 < 2.2e-16 ***carF 4 94.714 23.678 134.73 3.664e-14 ***driverF:carF 12 2.447 0.204 1.16 0.3715Residuals 20 3.515 0.176

#---------Estimate variance components-------------> sigma2ABHat <- (0.204 - 0.176)/2[1] 0.014

> sigma2DriverHat <- (93.428 - 0.204)/(2*5)[1] 9.3224

> sigma2CarHat <- (23.678 - 0.204)/(2*4)[1] 2.93425

• Only the F test for the interaction is used directly

2-33

Car Example: ML-Based

Approach

#--------------REML estimation------------------library(lme4)fit2 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF)+(1|driverF:carF),

data=X, REML=TRUE)

#---------Summary for random effects--------------# Similar to ANOVA for variance of main effectsGroups Name Variance Std.Dev.driverF:carF (Intercept) 1.6379e-14 1.2798e-07carF (Intercept) 2.9365e+00 1.7136e+00driverF (Intercept) 9.3242e+00 3.0536e+00Residual 1.8630e-01 4.3162e-01

#----------------ML estimation------------------fit3 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF)+(1|driverF:carF),

data=X, REML=FALSE)

#---------Summary for random effects--------------# Similar to ANOVA for variance of residuals and interactionGroups Name Variance Std.Dev.driverF:carF (Intercept) 0.014080 0.11866carF (Intercept) 2.772125 1.66497driverF (Intercept) 7.413777 2.72283Residual 0.175750 0.41923

2-34

Car Example: REML in SAS

proc mixed data=a1;class car driver;model mpg=;random car driver car*driver / vcorr;

run;

Covariance Parameter EstimatesCov Parm Estimate Alpha Lower Uppercar 2.9343 0.05 1.0464 24.9038driver 9.3224 0.05 2.9864 130.79car*driver 0.01406 0.05 0.001345 3.592E17Residual 0.1757 0.05 0.1029 0.3665

• SAS implementation produces REML estimates thatare identical to ANOVA

• This is due to differences in the implementation ofREML between SAS and R

2-35

Car Example: ML

Additive model

#--------------REML estimation------------------library(lme4)fit4 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF), data=X,

REML=TRUE)

#---------Summary for random effects--------------# Same as REML with interactionRandom effects:Groups Name Variance Std.Dev.carF (Intercept) 2.93651 1.71362driverF (Intercept) 9.32419 3.05355Residual 0.18630 0.43162

#----------------ML estimation------------------fit5 <- lmer(mpg ~ 1 + (1|driverF)+(1|carF), data=X,

REML=FALSE)

#---------Summary for random effects--------------# Smaller estimates of variationGroups Name Variance Std.Dev.carF (Intercept) 2.77436 1.66564driverF (Intercept) 7.41579 2.72319Residual 0.18631 0.43163

2-36

Two-way mixed effects

ANOVA

(Model III in KNNL)

2-37

Model formulation

Yijk = µ+ αi + βj + (αβ)ij + εijk

i = 1, . . . , a, j = 1, . . . , b, k = 1, . . . , n

• µ is the overall mean

• αi is the fixed effect of the i-th level of factor A,∑i αi = 0

• βj is the random effect of the j-th level of factor

B, βjiid∼ N (0, σ2

β)

• (αβ)ij is the joint random effect of the i-th level

of factor A and the j-th level of factor B, (αβ)ijiid∼

N (0, σ2αβ)

• εijk is the random error εijkiid∼ N (0, σ2)

• All random terms are independent

2-38

ANOVA Table

Source Type DF EMS

A F a− 1 σ2 + nσ2αβ + bn

a−1

a∑i=1

α2i

B R b− 1 σ2 + nσ2αβ + anσ2

β

AB R (a− 1)(b− 1) σ2 + nσ2αβ

Error R ab(n− 1) σ2

Total ab− 1

• Parameter estimates using mean squares

σ2 = MSE

σ2αβ = (MSAB−MSE)/n

σ2β = (MSB−MSAB)/an

• Estimates can be negative

• Same procedures for CI, and same adjust-ments for df, as in the one-way randomeffects ANOVA

2-39

ANOVA-Based Hypothesis

Tests

Two tests of variance, one test of fixed effects.In balanced designs:

• H0AB: σ2αβ = 0 vs H1AB: σ2

αβ > 0

F = MS(AB)MSE

H0∼ F(a−1)(b−1), ab(n−1)

• H0B: σ2β = 0 vs H1B: σ2

β > 0

F = MS(B)MS(AB)

H0∼ F(b−1), (a−1)(b−1)

• H0A: αi = 0, all i vs H1A: αi 6= 0, some i

F = MS(A)MS(AB)

H0∼ F(a−1), (a−1)(b−1)

2-40

Balanced Designs: Variance of

Linear Combinations of Means

• Assume balanced design,i.e. nij = n for all i and j

• Suppose we are interested in combinationsof expected values

L = c1(µ+ α1) + . . .+ ca(µ+ αa)

• The inbiased estimator L is

L = c1Y1··+ . . .+ caYa··

• And it’s variance is

V ar{L} =

(nσ2

αβ + σ2

nb

)(a∑

i=1

c2i

)+

(σ2β

b

)(a∑

i=1

ci

)2

2-41

Degrees of Freedom for Linear

Combination of Means

• Suppose the estimate of V ar{L} is

V ar{L} = p1MS1 + . . .+ pkMSk

where

– p1, . . . , pk are nonnegative constraints and

– MS1, . . . ,MSk are the mean squares inthe ANOVA table with degrees of free-dom ν1, . . . , νk

• Then the degrees of freedom associatedwith V ar{L}

ν =(p1MS1 + . . .+ pkMSk)2

p21MS2

1ν1

+ . . .+ p2kMS2

kνk

2-42

Careful: Restricted Mixed

Model Formulation in KNNL

Yijk = µ+ αi + βj + (αβ)ij + εijk

i = 1, . . . , a, j = 1, . . . , b, k = 1, . . . , n

• µ is the overall mean

• αi is the fixed effect of the i-th level of A,∑i αi = 0

• βj is the random effect of the j-th level of B,

βjiid∼ N (0, σ2

β)

• (αβ)ij is the joint random effect of the i-th level ofA and the j-th level of B

(αβ)ijiid∼ N (0, a−1

aσ2αβ),

a∑i=1

(αβ)ij = 0

• εijk is the random error εijkiid∼ N (0, σ2)

• The random terms are not independent:

Cov( (αβ)ij, (αβ)i′j ) = −1aσ2αβ

2-43

Restricted Mixed Model

Formulation: ANOVA Table

Source Type DF EMS

A F a− 1 σ2 + nσ2αβ + bn

a−1

a∑i=1

α2i

B R b− 1 σ2 + anσ2β

AB R (a− 1)(b− 1) σ2 + nσ2αβ

Error R ab(n− 1) σ2

Total ab− 1

• Different EMS for random effect

• Different test statistic to test the randomeffect

– H0B: σ2β = 0 vs H1B: σ2

β > 0

F = MS(B)MSE

H0∼ Fb−1, ab(n−1)

• We will use the non-restricted formulation

2-44

Linear Mixed Models:

General Formulation

• In notation of Faraway:

y = Xβ + Zγ + ε, where

γ ∼ N (0, σ2D) and

ε ∼ N (0, σ2I)

• Or, equivalently:

y|γ ∼ N (Xβ + Zγ, σ2I), where

γ ∼ N (0, σ2D)

• This implies the marginal distribution of y:

y ∼ N (Xβ, σ2(I + ZDZ′))

• Then the likelihood of the data is1

2πn/2|σ2V|1/2e− 1

2σ2(y−Xβ)′V(y−Xβ)

where V = I + ZDZ′

2-45

REML

• Using the general formulation

y ∼ N (Xβ, σ2V), define matrix K where

K′X = 0, therefore

K′y = N (0,K′VK)

• Find estimates of variance parameters on

the transformed scale K′y

• Find estimates of fixed effects assuming

that variance is known

2-46

Likelihood Ratio Test

• Compare a full and a reduced model: test

statistic

2{logL(Full|y)− logL(reduced|y)} H0∼ χ2df

where df is the difference in the number of param-

eters in the model

• Testing fixed effects in a mixed effects model

H0 : αi = 0 for all i, Ha αi 6= 0 for some i

– REML are not comparable (different scale)

– Use ML

– Approximate p-values (too small)

– Alternative: use bootstrap

– Alternative(1) estimate variances from the full model(2) use weighted fixed effects model

as if variances are known

2-47

Likelihood Ratio Test

• Testing random effects in a mixed effects

model

H0 : σ2 = 0, Ha σ2 6= 0 for some variance

– The value under H0 is on the limit of the pa-rameter space

– Approximate test (p-values too large)

– Alternative: use bootstrap

2-48

Predicting Random Effects

• Estimator: expected value of the random

effect conditional on the data

E{β} = DZ′V−1(y −Xβ)

• Basic property:

var(β) = var(E{β|y}) + E{var(β|y)}= var(predictor) + positive value

• Therefore

var(E{β|y} ≤ var(β)

−→ E{β} is a shrinkage estimator

2-49