Download - Effect Size 4-9-2012
-
Understanding and Quantifying EFFECT SIZESEFFECT SIZES
Karabi Nandy, Ph.d.Assistant Adjunct Professor
Translational Sciences Section, School of Nursing , gDepartment of Biostatistics, School of Public Health,
University of California Los Angeles (UCLA)
-
ObjectiveObjective
Effect size comes up in the context of
sample size calculations for proposals
reporting results from pilot studies.
By the end of this talk, you will be able to calculate,
interpret and report effect sizes in your work.
4/9/2012 EffectSize 2
-
OutlineOutline
Why are Effect Sizes (ES) important?
Types of Effect Sizes
Quantifying Magnitudes of Effect Sizes
Calculating Effect SizesCalculating Effect Sizes
Use of Softwares for Calculating Effect Sizes
Specific Guidelines for Reporting Effect Sizes
4/9/2012 EffectSize 3
-
DefinitionDefinition
Effect - A change or changed state occurring as a direct
result of action by somebody or something (Encarta,
2009)2009)
Size - The degree of something in terms of how big or
small it is
'Effect size' is simply a way of quantifying the size of the Effect size is simply a way of quantifying the size of the
difference between two groups.
4/9/2012 EffectSize 4
-
It is particularly valuable for quantifying the
effectiveness of a particular intervention relative toeffectiveness of a particular intervention, relative to
some comparison. It allows us to move beyond the
simplistic, 'Does it work or not?' to the far more
sophisticated 'How well does it workHow well does it work in a range ofsophisticated, How well does it work How well does it work in a range of
contexts?
4/9/2012 EffectSize 5
-
Why is Effect Size ImportantWhy is Effect Size Important
Knowing the magnitude of an effect allows us to
ascertain the practical significance of statistical
significancesignificance
Can always reach statistical significance if there is a large
enough sample size, unless the effect size is 0.
Even a large effect may not be statistically significant if Even a large effect may not be statistically significant if
the sample size is too small.
4/9/2012 EffectSize 6
-
Why is Effect Size ImportantWhy is Effect Size Important
Practical Significance
Even a statistically significant treatment difference may
not be practically important if the effect size is too smallnot be practically important if the effect size is too small.
However, there could still be practical importance even for
small effect sizes, especially in cases where cost and ease
make it easy to be implemented on a large scale. y p g
4/9/2012 EffectSize 7
-
Why is Effect Size ImportantWhy is Effect Size Important Sample Size Calculation for Studies
ES l di t l i l i l l ti fES plays a direct role in sample size calculations for any
study. It is connected to the power of a test, the level of
significance and sample size (n).
or Given any 3 quantities (power ES n) we can find the 4th Given any 3 quantities (power, ES, , n), we can find the 4th.
4/9/2012 EffectSize 8
-
Why is Effect Size ImportantWhy is Effect Size Important
Meta-Analysis
pooling information from many studies to verify results of p g y y
past research and inform future studies.
ES i t d i h t d d th fi di l d ES is computed in each study and the findings are pooled
together to draw overall inferences.
4/9/2012 EffectSize 9
-
Types Types of Effect of Effect SizesSizes Mean Differences between Groups
Eff t Si C h d Effect Size: Cohens d
Correlation/Regression
Effect Size: Pearsons r and R2
Effect Size: Cohens f2
Contingency tablesContingency tables
Effect Size: Odds Ratio or Relative Risk (association
between binary variables)4/9/2012 EffectSize 10
-
Types Types of Effect of Effect SizesSizes
ANOVA or GLMs
Effect Size: Eta-squared
Effect Size: Omega squaredEffect Size: Omega squared
Effect Size: Intraclass correlation (rater equality)
Chi-square tests
Effect Size: Phi (2 binary variables)Effect Size: Phi (2 binary variables)
Effect Size: Cramers Phi or V (categorical variables)
4/9/2012 EffectSize 11
-
Cohens dCohens d
21
222
211
pooledpooled
21 )1()1( wherenn
snsnss
xxd
Standardizes ES of the difference between two means
21pooled
Used for two sample independent t-tests
d ranges from - to +
interpretation: the difference between the mean valuesinterpretation: the difference between the mean values
is d standard deviations, Cohen (1988)
4/9/2012 EffectSize 12
-
ES Example 1ES Example 1
ES = 0 00 means that the average treatmentES 0.00 means that the average treatment
participant outperformed 50% of the control
i iparticipants4/9/2012 EffectSize 13
-
ES Example 2ES Example 2
ES = 0.85 means that the average treatment participant g p p
outperformed 80% of the control participants
4/9/2012 EffectSize 14
-
General GuidelinesGeneral Guidelines
In general, 0.20 is a small effect size, 0.50 is a moderate
effect size and 0.80 is a large effect size (Cohen, 1992)
d standardized Percentage ofd- standardized Percentage of
mean difference variance explained
Small .20 1%
M d t 50 10% Moderate .50 10%
Large .80 25%
4/9/2012 EffectSize 15
-
Cohens dCohens d
Special Cases
For small sample sizes use Hedges G )1()1(h
222
21121 snsnxxd
For unequal group variances, use Glasss 2
)()( where21
2211pooled
pooled
21
nnssdq g p
uses sample sd of the control group only so that effect sizes
would not differ under equal means and unequal varianceswould not differ under equal means and unequal variances
4/9/2012 EffectSize 16
-
Pearsons Pearsons rr
used in the context of correlationmeasuring association
between 2 variablesbetween 2 variables
Interpretation: For every 1-unit standard deviation change
in x, there is a r-unit standard deviation change in y
4/9/2012 EffectSize 17
-
Pearsons Pearsons RR2 2
used in the context of regressionmeasuring how well a regression line fits to a given dataregression line fits to a given data
R - linear association between 2 continuous variables
R2 (Coefficient of Determination) proportion of shared variability between 2 or more variables
Interpretation: R2 *100% is percent variance of the outcome y that can be explained by the linear regression y p y gmodel (i.e. indicates how well the linear regression line fits the data))
4/9/2012 EffectSize 18
-
Odds RatioOdds Ratio
Used in the context of binary/categorical outcomes
Odds of being in one group (eg. success) relative to the odds of
being in a different group (eg. failure)g g p ( g )
OR ranges from 0 to
OR>1 indicates an increase in odds relative to the reference group
OR < 1 indicates a decrease in odds relative to the reference group
4/9/2012 EffectSize 19
-
Relative RiskRelative Risk
RR measures the risk of an event relative to an independent
variablevariable
For small probabilities, the relative risk is approximately equal to
the odds ratio
Interpretation: If RR > 1 then the risk of disease X among p g
smokers is RR times the risk of disease X among non-smokers
(vice versa if RR < 1)(vice versa if RR < 1)
4/9/2012 EffectSize 20
-
EtaEta--Squared (Squared (22) ) and partial Etaand partial Eta Squared (Squared ( 22))and partial Etaand partial Eta--Squared (Squared (pp22))
Used with ANOVA family and GLMs
Measures the degree of association in the sample
St d di Eff t Si f th h d i b t Standardizes Effect Sizes of the shared variance between a
continuous outcome and categorical predictors
Partial eta-squared is the proportion of the total variability
attributable to a given factorattributable to a given factor.
4/9/2012 EffectSize 21
-
EtaEta--Squared (Squared (22) ) and partial Etaand partial Eta--Squared (Squared ( 22))
Interpretation: 2 *100% is percent of the variance in y
and partial Etaand partial Eta--Squared (Squared (pp ))
explained by the variance in x (similar to the R2
interpretation for linear regression (Dattalo, 2008)).p g ( , ))
2 is biased and on average overestimates the variance
explained in the population, but decreases as the sample size
gets larger.
Caution: these effect sizes depend on the number and
magnitude of the other effectsmagnitude of the other effects
4/9/2012 EffectSize 22
-
Cohens Cohens ff22
Used in multiple linear regression, R2 = 2
Standardized effect size is the proportion of explained
variance over unexplained variancevariance over unexplained variance
Estimate is biased and overestimates the effect size for
ANOVA (unbiased estimate is Omega-Squared)
4/9/2012 EffectSize 23
-
OmegaOmega--SquaredSquared
Estimates the proportion of variance in the population
that is explained by the treatment
is always smaller than or since Omega is always smaller than or since Omega
measures the population variance and Eta measures the
sample variance
4/9/2012 EffectSize 24
-
IntraclassIntraclass CorrelationCorrelation ICC is used to measure inter-rater reliability for two or more
raters. It may also be used to assess test-retest reliability. ICC may
be conceptualized as the ratio of between-groups variance to total
variance.
Can be used in ANOVA
Similar interpretation to Omega-Squared
4/9/2012 EffectSize 25
-
PhiPhi
n
2 Used for crosstabs or for chi-square tests (equality of
proportions or tests of independence between 2 binary p p p y
variables) = 0 indicates independence
Phi are related to correlation and Cohens d (for 2 binary
variables)
Interpreted like Pearsons r and R2
4/9/2012 EffectSize 26
-
Cramers Cramers Phi or VPhi or V CramersPhi(CramersV)canbeusedwithcategoricalvariables
withmorethan2categories(m>2)(RxCtables)g ( ) ( )
k i (R C);k=min(R,C)
measurestheintercorrelationofthevariables,butisbiased
sinceitincreaseswiththenumberofcells.IncreaseinRandC
willindicateastrongassociation,whichisjustanartifactofthe
typeofvariableused.
4/9/2012 EffectSize 27
-
Magnitude of Effect Summary TableMagnitude of Effect Summary TableEffect Size Small Medium Large
r 0.10 0.30 0.50
r2 0.01 0.09 0.25
2 0 01 0 06 0 142 0.01 0.06 0.14
R2 0.01 0.06 0.14
Cohens d 0.20 0.50 0.80
/ Cramers V 0.10 0.30 0.50
Cohens f2 0.02 0.15 0.35
OR 1.44 2.47 4.25
4/9/2012 EffectSize
OR 1.44 2.47 4.25
28
-
Effect Size ConversionsEffect Size Conversions
Effect Size Converted to Cohens d
Correlation21
2r
rd
Chi-Square
df =1df =1
df > 1 Nd
24 Odds Ratio (Chinn, 2000)
4/9/2012 EffectSize 29
-
SoftwareSoftware
Calculate effect size
http://www.uccs.edu/~faculty/lbecker/
http://fac lt assar ed /lo r /ne cs html http://faculty.vassar.edu/lowry/newcs.html
Statistical softwares such as SAS, R, STATA, SPSS S s c so w es suc s S S, , S , S SS
can calculate most of the standard Effect Sizes
4/9/2012 EffectSize 30
-
Calculating Cohens d Calculating Cohens d online calculatoronline calculatorhttp://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 31
-
Calculating Cohens d Calculating Cohens d online calculatoronline calculatorhttp://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 32
-
Calculating Cramers Phi Calculating Cramers Phi online calculatoronline calculatorhttp://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 33
-
Calculating Cramers Phi Calculating Cramers Phi online calculatoronline calculatorhttp://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 34
-
Calculating Cramers Phi Calculating Cramers Phi online calculatoronline calculatorhttp://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/~faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/http://www.uccs.edu/ faculty/lbecker/
4/9/2012 EffectSize 35
-
A (notA (not--soso--great) Alternative: great) Alternative: Calculating Cohens d using Calculating Cohens d using GPowerGPowerg gg g
Step 1: Choose t-test from drop down
Step 2: Choose means from drop
Step 3: Choose
drop down menu
from drop down menu
Solution: Calculated effect size
Choose Sensitivity from the drop down menu Step 4: Based on
4/9/2012 EffectSize
menudata from pilot study
36
-
Calculating SS given Cohens d Calculating SS given Cohens d -- GPowerGPowerA better use of Gpower is to calculate sample sizesA better use of Gpower is to calculate sample sizes.
4/9/2012 EffectSize 37
-
Reporting Guidelines and TrendsReporting Guidelines and Trends Reporting effect sizes has three important benefits (APA,
1999):1999):
Meta-analysis
Informing subsequent research
Interpretation and evaluation of results within the context
of related literature
4/9/2012 EffectSize 38
-
Reporting Guidelines and TrendsReporting Guidelines and Trends
What to report (APA, 2010):
Type of effect size
Value of the effect size (in original units such as lbs or Value of the effect size (in original units, such as lbs., or
mean differences on a scale, and/or the effect size statistic)
Interpretation of the effect size
Practical significance of the effect sizePractical significance of the effect size
4/9/2012 EffectSize 39
-
ReferencesReferences
Chinn, S.(2000). A simple method for converting an odds ratio to effect
i f i t l i St ti ti i M di i 19 3127 3131size for use in meta-analysis. Statistics in Medicine, 19, 3127-3131.
Cohen, J.(1992). A power primer. Psychological Bulletin, 112(1), 155-
159.
Cohen J (1988) Statistical power analysis for the behavioral sciencesCohen, J. (1988) Statistical power analysis for the behavioral sciences .
New Jersey: Lawrenced Erlbaum Associates, Inc. Publishers. pp 283-286.
4/9/2012 EffectSize 40
-
ReferencesReferences Dunst Carl J et al (2004) Guidelines for Calculating Effect Sizes for Dunst, Carl J. et al. (2004) Guidelines for Calculating Effect Sizes for
Practice-Based Research Syntheses. Centerscope 3(1)
http://courseweb.unt.edu/gknezek/06spring/5610/centerscopevol3no1.pdf
A presentation on effect size:
http://www.family.umaryland.edu/ryc_research_and_evaluation/publication
product files/selected presentations/presentation files/pdfs/effect%20size_product_files/selected_presentations/presentation_files/pdfs/effect%20size
%20and%20intervention%20research.pdf
Lee B.R., Bright C.L., Svobo da, D., Fakunmoju, S. and Barth R.P.
Outcomes of group care for youth: A review of comparative literature.
Research on Social Work Practice 4/9/2012 EffectSize 41