11. Analysis of Variance (ANOVA)

Analysis of Variance

• Review of T-Test ✔

• The basic ANOVA situation

• How ANOVA works• One factor ANOVA model• ANCOVA and MANOVA

回顾 : T 检验

2

)1()1(

)2(~11

YXT

:H

,YX

,,,Y,,N~Y

X,,,X,,N~X

32

212

210

222

21

21222

21211

0

mn

smsns

mnt

mns

YYY

XX

H

m

n

这里

检验统计量

要检验而且独立与假定

的样本）为（总体

的样本）为（设总体

Analysis of Variance

• Review of T-Test ✔

• The basic ANOVA situation ✔

• How ANOVA works• One factor ANOVA model• ANCOVA and MANOVA

The basic ANOVA situation

Two variables: 1 Categorical, 1 Quantitative

Main Question: Do the (means of) the quantitative variables depend on which group (given by categorical variable) the individual is in?

If categorical variable has only 2 values: • 2-sample t-test

ANOVA allows for 3 or more groups

An example ANOVA situationSubjects: 25 patients with blistersTreatments: Treatment A, Treatment B, PlaceboMeasurement: # of days until blisters heal

Data [and means]:• A: 5,6,6,7,7,8,9,10 [7.25]• B: 7,7,8,9,9,10,10,11 [8.875]• P: 7,9,9,10,10,10,11,12,13 [10.11]

Are these differences significant?

Informal Investigation

Graphical investigation: • side-by-side box plots• multiple histograms

Whether the differences between the groups are significant depends on

• the difference in the means• the standard deviations of each group• the sample sizes

ANOVA determines P-value from the F statistic

Side by Side Boxplots

PBA

13

12

11

10

9

8

7

6

5

treatment

days

What does ANOVA do?

At its simplest ANOVA tests the following hypotheses:

H0: The means of all the groups are equal.

Ha: Not all the means are equal

• doesn’t say how or which ones differ.• Can follow up with “multiple comparisons”

Assumptions of ANOVAeach group is approximately normal

check this by looking at histograms and/or normal quantile plots, or use assumptions

can handle some nonnormality, but not severe outliers

standard deviations of each group are approximately equal rule of thumb: ratio of largest to smallest

sample st. dev. must be less than 2:1

Normality Check

We should check for normality using:• assumptions about population • histograms for each group• normal quantile plot for each group

With such small data sets, there really isn’t a really good way to check normality from data, but we make the common assumption that physical measurements of people tend to be normally distributed.

Standard Deviation Check

Compare largest and smallest standard deviations:• largest: 1.764• smallest: 1.458• 1.458 x 2 = 2.916 > 1.764

Variable treatment N Mean StDev

days A 8 7.250 1.669 B 8 8.875 1.458 P 9 10.111 1.764

Analysis of Variance

• Review of T-Test ✔

• The basic ANOVA situation ✔

• How ANOVA works ✔• One factor ANOVA model• ANCOVA and MANOVA

Notation for ANOVA

• n = number of individuals all together• m = number of groups• = mean for entire data set is

Group j has• nj = # of individuals in group j• xij = value for individual i in group j

• = mean for group j• sj = standard deviation for group j

jx

x

How ANOVA works (outline)ANOVA measures two sources of variation in the data and compares their relative sizes

variation BETWEEN groups for each data value look at the difference

between its group mean and the overall mean

variation WITHIN groups for each data value we look at the difference

between that value and the mean of its group

2

ij jx x

2

jx x

The ANOVA F-statistic

A ratio of the Between Group Variation divided by the Within Group Variation:

MSE

MSG

Within

BetweenF

A large F is evidence against H0, since it indicates that there is more difference between groups than within groups.

Minitab ANOVA Output

Analysis of Variance for days Source DF SS MS F Ptreatment 2 34.74 17.37 6.45 0.006Error 22 59.26 2.69Total 24 94.00

Df Sum Sq Mean Sq F value Pr(>F) treatment 2 34.7 17.4 6.45 0.0063 Residuals 22 59.3 2.7

R ANOVA Output

How are these computations made?

We want to measure the amount of variation due to BETWEEN group variation and WITHIN group variation

For each data value, we calculate its contribution to:

• BETWEEN group variation:

• WITHIN group variation:

2

jx x2( )ij jx x

An even smaller exampleSuppose we have three groups

• Group 1: 5.3, 6.0, 6.7• Group 2: 5.5, 6.2, 6.4, 5.7• Group 3: 7.5, 7.2, 7.9

We get the following statistics:

SUMMARYGroups Count Sum Average Variance

Column 1 3 18 6 0.49Column 2 4 23.8 5.95 0.176667Column 3 3 22.6 7.533333 0.123333

Excel ANOVA OutputANOVASource of Variation SS df MS F P-value F critBetween Groups 5.127333 2 2.563667 10.21575 0.008394 4.737416Within Groups 1.756667 7 0.250952

Total 6.884 9

1 less than number of groups: m-1=2

number of data values - number of groups:n-m=7

1 less than number of individuals:n-1=9

)7,2(05.0F

Computing ANOVA F statistic

overall mean: 6.44 F = 2.5528/0.25025 = 10.21575

The example ANOVA situation again

Subjects: 25 patients with blistersTreatments: Treatment A, Treatment B, PlaceboMeasurement: # of days until blisters heal

Data:• A: 5,6,6,7,7,8,9,10 • B: 7,7,8,9,9,10,10,11• P: 7,9,9,10,10,10,11,12,13

Minitab ANOVA Output

1 less than # of groups: m-1=2

# of data values - # of groups:n-m=22

1 less than # of individuals: n-1=24

Analysis of Variance for days Source DF SS MS F Ptreatment 2 34.74 17.37 6.45 0.006Error 22 59.26 2.69Total 24 94.00

Minitab ANOVA OutputAnalysis of Variance for days Source DF SS MS F Ptreatment 2 34.74 17.37 6.45 0.006Error 22 59.26 2.69Total 24 94.00

2

,

( )ij ji j

x x 2

,

( )ji j

x x2

,

)( xxji

ij

SS stands for sum of squaresMS = SS/DF

Minitab ANOVA Output

MSG = SSG / DFG

Analysis of Variance for days Source DF SS MS F Ptreatment 2 34.74 17.37 6.45 0.006Error 22 59.26 2.69Total 24 94.00

F = MSG / MSE

P-valuecomes fromF(DFG,DFE)

(P-values for the F statistic are in Table E)

MSE = SSE / DFE

So How big is F?

Since F is Mean Square Between / Mean Square Within

= MSG / MSE

A large value of F indicates relatively moredifference between groups than within groups (evidence against H0)

To get the P-value, we compare to F(m-1,n-m)-distribution• m-1 degrees of freedom in numerator (# groups -1)• n-m degrees of freedom in denominator (rest of df)

Connections between SST, MST, and standard deviation

So SST = (n -1) s2, and MST = s2. That is, SST and MST measure the TOTAL variation in the data set.

MST

DFT

SST

n

xxs ij

1

2

2

If ignore the groups for a moment and just compute the standard deviation of the entire data set, we see

Connections between SSE, MSE, and standard deviation

So SS[Within Group j] = (sj2) (dfj )

2

2 [Within Group ]

1ij j

jj j

x x SS js

n df

This means that we can compute SSE from the standard deviations and sizes (df) of each group:

2 2

[ ] [ j]

( 1) ( )j j j j

SSE SS Within SS

s n s df

Within Group

Remember:

Pooled estimate for st. dev

2 2 21 1 2 2 1 12 ( 1) ( 1) ... ( 1)

1p p

w

n s n s n ss

n p

2 2 21 1 2 2 1 12

1 2 1

( ) ( ) ... ( )

...p p

wp

df s df s df ss

df df df

One of the ANOVA assumptions is that all groups have the same standard deviation.

2w

SSEs MSE

DFE so MSE is the pooled

estimate of variance

m=p+1

In Summary2 2

,

2 2

,

2 2

,

( ) ( )

( ) ( )

( ) ( )

; ;

iji j

ij j j ji j i

j j ji j j

SST x x s DFT

SSE x x s d

SSG x x n x x

SS MSGSSE SSG SST MS F

DF MSE

Analysis of Variance

• Review of T-Test ✔

• The basic ANOVA situation ✔

• How ANOVA works ✔• One factor ANOVA model ✔

• ANCOVA and MANOVA

ANOVA 由英国统计学家 R.A.Fisher首创，为纪念 Fisher ，以 F 命名，故方差分析又称 F 检验 （ F test ）。用于推断多个总体均数有无差异

Mechanics: One factor ANOVA model

One factor ANOVA model:

Xij=μ+τj+εij

where

μ is the overall mean of the population,

τj is the treatment effect associated with level j of the experimental factor (i.e., Group j),

εij is assumed to be distributed independently and identically with mean zero and variance σ2.

Mechanics: One factor ANOVA model

We now state the null hypothesis as follows:

H0: τ1= τ2=…= τp+1=0

It can be proved that

12 2 2

1 1

12 2 2

1

1 1( ) ( ) ( )

1 1

1 1( ) ( )

jnp

ij j j jj i j

p

j j j jj j

E MSE E x x E s dfn p n p

E MSG E n x x np p

F Statistic

2

2

2

[ ]

[ ]

1( , 1)

1

, 1

( , 1)

SS Between SSGR

SS Total SST

R n pF F p n p

R p

p

MSGF p n p

MSE

0

线性模型

H

在线性回归分析中，

这里 为回归系数个数。 p+1在方差分析中， 为组的个数，有

SSG/ pF=

SSE/ (n-p-1)

0. 0

0. 2

0. 4

0. 6

0. 8

1. 0

1. 2

1. 4

0 1 2 3 4F

F 分布曲线

10,10 21

5,1 21 5,5 21

221

21

122/

22/

121

21

1

21

)(22

2)(

F

F

Ff

F 分布曲线下面积与概率

Where’s the Difference?

Analysis of Variance for days Source DF SS MS F Ptreatmen 2 34.74 17.37 6.45 0.006Error 22 59.26 2.69Total 24 94.00 Individual 95% CIs For Mean Based on Pooled StDevLevel N Mean StDev ----------+---------+---------+------A 8 7.250 1.669 (-------*-------) B 8 8.875 1.458 (-------*-------) P 9 10.111 1.764 (------*-------) ----------+---------+---------+------Pooled StDev = 1.641 7.5 9.0 10.5

Once ANOVA indicates that the groups do not all appear to have the same means, what do we do?

Clearest difference: P is worse than A (CI’s don’t overlap)

Multiple ComparisonsOnce ANOVA indicates that the groups do not allhave the same means, we can compare them twoby two using the 2-sample t test

• We need to adjust our p-value threshold because we are doing multiple tests with the same data.

•There are several methods for doing this.

• If we really just want to test the difference between one pair of treatments, we should set the study up that way.

Analysis of Variance

• Review of T-Test ✔

• The basic ANOVA situation ✔

• How ANOVA works ✔• One factor ANOVA model ✔

• ANCOVA and MANOVA ✔

Analysis of Covariance (ANCOVA)

ANCOVA allows for the introduction of other independent variables into ANOVA. Here are two reasons:

1. To introduce the precision of an experiment by removing possible sources of variance in the dependent variable that are attributed to factors not being controlled by the researcher.

2. To introduce a covariate to account for systematic differences across treatment groups that for some reason were not controlled for in the experiment design.

One factor ANCOVA model

One factor ANOVA model with covariate Z:

Xij=μ+τj+βZij+εij

where

μ is the overall mean of the population,

τj is the treatment effect associated with level j of the experimental factor (i.e., Group j),

εij is assumed to be distributed independently and identically with mean zero and variance σ2.

Z is a standarized variable.

Mulitiple Analysis of Variance(MANOVA)

In some experiments, there are multiple dependent variables because the same measure is taken from each subject at different points in time following the treatment. For example, after administrating a new drug, it might be important to measure the incidence of side effects or complications short after the treatment and again later. This type of an experiment is known as a repeated-measures design.