the paired t-test, non-parametric tests, and anova july 13, 2004
TRANSCRIPT
The paired t-test, The paired t-test, non-parametric tests, and ANOVAnon-parametric tests, and ANOVA
July 13, 2004July 13, 2004
Review: the Experiment Review: the Experiment (note: exact numbers have been altered)(note: exact numbers have been altered)
Grade 3 at Oak School were given an IQ test at the beginning of the academic year (n=90).
Classroom teachers were given a list of names of students in their classes who had supposedly scored in the top 20 percent; these students were identified as “academic bloomers” (n=18).
BUT: the children on the teachers lists had actually been randomly assigned to the list.
At the end of the year, the same I.Q. test was re-administered.
The resultsThe results
Children who had been randomly assigned to the “top-20 percent” list had mean I.Q. increase of 12.2 points (sd=2.0) vs. children in the control group only had an increase of 8.2 points (sd=2.5)
Confidence interval (more Confidence interval (more information!!)information!!)
95% CI for the difference: 4.0±1.99(.64) = (2.7 – 5.3)
t-curve with 88 df’s has slightly wider cut-off’s for 95% area (t=1.99) than a normal curve (Z=1.96)
The Paired T-testThe Paired T-test
The Paired T-testThe Paired T-test
Paired data means you’ve measured the same person at different time points or measured pairs of people who are related (husbands and wives, siblings, controls pair-matched to cases, etc.
For example, to evaluate whether an observed change in mean (before vs. after) represents a true improvement (or decrease):
Null hypothesis: difference (after-before)=0
The differences are treated The differences are treated like a single random variablelike a single random variable
Xi Yi Xi - Yi
X1 Y1 D1
X2 Y2 D2
X3 Y3 D3
X4 Y4 D4
… … …
Xn Yn Dn
n
D
D
n
i
n
1
nSD
D
n 0T=
21
2
1
)(
D
n
ini
Sn
DD
2
D
Example DataExample Data
baseline Test2 improvement
10 9 -1
10 12 +2
9 13 +4
8 8 0
12 11 -1
11 12 +1
11 13 +2
7 11 +4
6 8 +2
9 9 0
9 8 -1
10 9 -1
9 9 0
Is there a significant increase in scores in this group? Average of differences = +1 Sample Variance = 3.3; sample SD = 1.82 T 12 = 1/(1.82/3.6) = 1.98
data _null_;pval= 1-probt(1.98, 12);put pval;run;0.0355517436
Significant for a one-sided test; borderline for two-sided test
Example 2: Did the control Example 2: Did the control group in the Oak School group in the Oak School
experiment improveexperiment improveat allat all during the year? during the year?
2829.
2.8
725.2
2.8271 t
p-value <.0001
Confidence interval for annual Confidence interval for annual change in IQ test scorechange in IQ test score
95% CI for the increase: 8.2±2.0(.29) = (7.6 – 8.8)
t-curve with 71 df’s has slightly wider cut-off’s for 95% area (t=2.0) than a normal curve (Z=1.96)
Summary: parametric testsSummary: parametric tests
Equal variances
are pooled
Unequal variances (unpooled)
One sample (or paired sample)
Two samples
True standard deviation is known
One-sample Z-test
Two-sample Z-test
Standard deviation is estimated by the sample
One-sample t-test
Two-sample t-test
Non-parametric testsNon-parametric tests
Non-parametric testsNon-parametric tests
t-tests require your outcome variable to be normally distributed (or close enough).
Non-parametric tests are based on RANKS instead of means and standard deviations (=“population parameters”).
Example: non-parametric testsExample: non-parametric tests
10 dieters following Atkin’s diet vs. 10 dieters following Jenny Craig
Hypothetical RESULTS:Atkin’s group loses an average of 34.5 lbs.
J. Craig group loses an average of 18.5 lbs.
Conclusion: Atkin’s is better?
Example: non-parametric testsExample: non-parametric tests
BUT, take a closer look at the individual data…
Atkin’s, change in weight (lbs):+4, +3, 0, -3, -4, -5, -11, -14, -15, -300
J. Craig, change in weight (lbs)-8, -10, -12, -16, -18, -20, -21, -24, -26, -30
Enter data in SAS…Enter data in SAS…data nonparametric;input loss diet $;datalines ;+4 atkins +3 atkins0 atkins-3 atkins-4 atkins-5 atkins -11 atkins-14 atkins-15 atkins-300 atkins-8 jenny-10 jenny-12 jenny-16 jenny-18 jenny -20 jenny-21 jenny-24 jenny-26 jenny-30 jenny
;run;
Jenny CraigJenny Craig
-30 -25 -20 -15 -10 -5 0 5 10 15 20
0
5
10
15
20
25
30
Percent
Weight Change
Atkin’sAtkin’s
-300 -280 -260 -240 -220 -200 -180 -160 -140 -120 -100 -80 -60 -40 -20 0 20
0
5
10
15
20
25
30
Percent
Weight Change
t-test doesn’t work…t-test doesn’t work…
Comparing the mean weight loss of the two groups is not appropriate here.
The distributions do not appear to be normally distributed.
Moreover, there is an extreme outlier (this outlier influences the mean a great deal).
Statistical tests to compare Statistical tests to compare ranks:ranks:
Wilcoxon rank-sum test (equivalent to Mann-Whitney U test) is analogue of two-sample t-test.
Wilcoxon signed-rank test is analogue of one-sample t-test, usually used for paired data
Wilcoxon rank-sum testWilcoxon rank-sum test
RANK the values, 1 being the least weight loss and 20 being the most weight loss.
Atkin’s +4, +3, 0, -3, -4, -5, -11, -14, -15, -300 1, 2, 3, 4, 5, 6, 9, 11, 12, 20 J. Craig -8, -10, -12, -16, -18, -20, -21, -24, -26, -30 7, 8, 10, 13, 14, 15, 16, 17, 18, 19
Wilcoxon “rank-sum” testWilcoxon “rank-sum” test
Sum of Atkin’s ranks: 1+ 2 + 3 + 4 + 5 + 6 + 9 + 11+ 12 + 20=73 Sum of Jenny Craig’s ranks:
7 + 8 +10+ 13+ 14+ 15+16+ 17+ 18+19=137
Jenny Craig clearly ranked higher! P-value *(from computer) = .017
– from ttest, p-value=.60
*Tests in SAS…*Tests in SAS…
/*to get wilcoxon rank-sum test*/proc npar1way wilcoxon data=nonparametric;class diet;var loss;run;
/*To get ttest*/proc ttest data=nonparametric;class diet;var loss;run;
Wilcoxon “signed-rank” testWilcoxon “signed-rank” testH0: median weight loss in Atkin’s group = 0
Ha:median weight loss in Atkin’s not 0Atkin’s +4, +3, 0, -3, -4, -5, -11, -14, -15, -300
Rank absolute values of differences (ignore zeroes):Ordered values: 300, 15, 14, 11, 5, 4, 4, 3, 3, 0Ranks: 1 2 3 4 5 6-7 8-9 - Sum of negative ranks: 1+2+3+4+5+6.5+8.5=30Sum of positive ranks: 6.5+8.5=15P-value*(from computer)=.043; from paired t-test=.27
*Tests in SAS…*Tests in SAS…
/*to get one-sample tests (both student’s t and signed-rank*/
proc univariate data=nonparametric;
var loss;
where diet="atkins";
run;
What if data were paired?What if data were paired?
e.g., one-to-one matching; find pairs of study participants who have same age, gender, socioeconomic status, degree of overweight, etc.
Atkin’s+4, +3, 0, -3, -4, -5, -11, -14, -15, -300J. Craig-8, -10, -12, -16, -18, -20, -21, -24, -26, -30
Enter data differently in SAS…Enter data differently in SAS…10 pairs, rather than 20 10 pairs, rather than 20 individual observationsindividual observations
data piared;input lossa lossj;diff=lossa-lossj;datalines ;+4 -8 +3 -10 0 -12 -3 -16 -4 -18 -5 -20 -11 -21 -14 -24 -15 -26 -300 -30
;run;
*Tests in SAS…*Tests in SAS…/*to get all paired tests*/proc univariate data=paired;var diff;run;/*To get just paired ttest*/proc ttest data=paired;var diff;run;/*To get paired ttest, alternatively*/proc ttest data=paired;paired lossa*lossj;run;
ANOVAANOVAfor comparing means between for comparing means between
more than 2 groupsmore than 2 groups
ANOVA ANOVA ((ANANalysis alysis OOf f VAVAriance)riance)
Idea: For two or more groups, test difference between means, for quantitative normally distributed variables.
Just an extension of the t-test (an ANOVA with only two groups is mathematically equivalent to a t-test).
Like the t-test, ANOVA is “parametric” test—assumes that the outcome variable is roughly normally distributed
The “F-test”The “F-test”
groupswithinyVariabilit
groupsbetweenyVariabilitF
Is the difference in the means of the groups more than background noise (=variability within groups)?
amenorrheic oligomenorrheic eumenorrheic
0.7
0.8
0.9
1.0
1.1
1.2
SPINE
Between group variation
Spine bone density vs. Spine bone density vs. menstrual regularity menstrual regularity
Within group variability
Within group variability
Within group variability
Group means and standard Group means and standard deviationsdeviations
Amenorrheic group (n=11):– Mean spine BMD = .92 g/cm2
– standard deviation = .10 g/cm2
Oligomenorrheic group (n=11)– Mean spine BMD = .94 g/cm2
– standard deviation = .08 g/cm2
Eumenrroheic group (n=11)– Mean spine BMD =1.06 g/cm2
– standard deviation = .11 g/cm2
The F-TestThe F-Test
063.)13
)97.06.1()97.94(.)97.92(.(*11
22222
xbetween nss
0095.)11.08.10(.31 22222 savgswithin
6.60095.
063.2
2
30,2 within
between
s
sF
The size of the groups. The difference of
each group’s mean from the overall mean.
Between-group variation.
The average amount of variation within groups.
Each group’s variance.Large F value indicates that the between group variation exceeds the within group variation (=the background noise).
The F-distributionThe F-distribution
The F-distribution is a continuous probability distribution that depends on two parameters n and m (numerator and denominator degrees of freedom, respectively):
The F-distributionThe F-distribution A ratio of sample variances follows an F-
distribution:
22
220
:
:
withinbetweena
withinbetween
H
H
The F-test tests the hypothesis that two sample variances are equal. F will be close to 1 if sample variances are equal.
mnwithin
between F ,2
2
~
ANOVA TableANOVA Table
Between (k groups)
k-1 SSB(sum of squared deviations of group means from
grand mean)
SSB/k-1 Go to
Fk-1,nk-k
chart
Total variation
nk-1 TSS(sum of squared deviations of observations from grand mean)
Source of variation
d.f.
Sum of squares
Mean Sum of Squares F-statistic p-value
Within(n individuals
per group)
nk-k SSW (sum of squared deviations of observations from their group mean)
s2=SSW/nk-k
knkSSW
kSSB
1
TSS=SSB + SSW
ANOVA=t-testANOVA=t-test
222
22
2
)()()(
npp
ts
YX
s
YXBetween
(2 groups)1 SSB
(squared differenc
e in means)
Squared difference in means
Go to
F1, 2n-2
Chart notice values are just (t 2n-2)
2
Total variation
2n-1 TSS
Source of variation
d.f.
Sum of squares
Mean Sum of Squares F-statistic p-value
Within 2n-2 SSW
equivalent to numerator of pooled variance
Pooled variance
ANOVA summaryANOVA summary
A statistically significant ANOVA (F-test) only tells you that at least two of the groups differ, but not which ones differ.
Determining which groups differ (when it’s unclear) requires more sophisticated analyses to correct for the problem of multiple comparisons…
Question: Question: Why not just do Why not just do 3 pairwise ttests?3 pairwise ttests?
Answer: because, at an error rate of 5% each test, this means you have an overall chance of up to 1-(.95)3= 14% of making a type-I error (if all 3 comparisons were independent)
If you wanted to compare 6 groups, you’d have to do
6C2 = 15 pairwise ttests; which would give you a high chance of finding something significant just by chance (if all tests were independent with a type-I error rate of 5% each); probability of at least one type-I error = 1-(.95)15=54%.
Multiple comparisonsMultiple comparisons
With 18 independent comparisons, we have 60% chance of at least 1 false positive.
Multiple comparisonsMultiple comparisons
With 18 independent comparisons, we expect about 1 false positive.
Correction for multiple Correction for multiple comparisonscomparisons
How to correct for multiple comparisons post-hoc…
Bonferroni’s correction (adjusts p by most conservative amount, assuming all tests independent)
Holm/Hochberg (gives p-cutoff beyond which not significant)
Tukey’s (adjusts p) Scheffe’s (adjusts p)
Non-parametric ANOVANon-parametric ANOVA
Kruskal-Wallis one-way ANOVA
Extension of the Wilcoxon Sign-Rank test for 2 groups; based on ranks
Proc NPAR1WAY in SAS
Reading for this weekReading for this week
Chapters 4-5, 12-13 (last week)Chapters 6-8, 10, 14 (this week)