LECTURE 4EPSY 652 FALL 2009

Computing Effect Sizes- Mean Difference EffectsGlass: e = (MeanExperimental – MeanControl)/SD

o SD = Square Root (average of two variances) for randomized designs

o SD = Control standard deviation when treatment might affect variation (causes statistical problems in estimation)

Hedges: Correct for sampling bias: g = e[ 1 – 3/(4N – 9) ]

where N=total # in experimental and control groups

Sg = [ (Ne + Nc)/NgNc + g2/(2(Ne + Nc) ]½

Computing Effect Sizes- Mean Difference Effects Example from Spencer ADHD Adult study

Glass: e = (MeanExperimental – MeanControl)/SD

= (82 – 101)/21.55= .8817

Hedges: Correct for sampling bias: g = e[ 1 – 3/(4N – 9) ]

= .8817 (1 – 3/(4*110 – 9) = .8762

Note: SD computed from t-statistic of 4.2 given in article:e = t*(1/NE + 1/NC )½

123456789101112

A B C D E F G H I J K L

effect Mean E Mean C SDE SDC d

Hedges g Ctrl N Trmt N N w wd

1 1 0.2 1 1 0.60 0.58 10 13 23 5.43 3.142 0.3 -0.4 1 1 -0.05 -0.05 21 20 41 10.24 -0.533 0.8 0.28 1 1 0.54 0.52 9 9 18 4.35 2.254 0.5 -0.46 1 1 0.02 0.02 18 21 39 9.69 0.205 0.2 -0.8 1 1 -0.30 -0.30 73 94 167 40.65 -12.106 0.4 -0.12 1 1 0.14 0.14 52 71 123 29.94 4.207 1 0.36 1 1 0.68 0.68 117 115 232 54.85 37.178 0.46 -0.5 1 1 -0.02 -0.02 8 8 16 4.00 -0.06

mean 0.2154 38.50 43.88 82.38 159.15 34.28s(mean) 0.0793

Computing Mean Difference Effect Sizes from Summary Statisticst-statistic: e = t*(1/NE + 1/NC )½ F(1,dferror): e = F½ *(1/NE + 1/NC )½ Point-biserial correlation:

e = r*(dfe/(1-r2 ))½ *(1/NE + 1/NC )½ Chi Square (Pearson association):

= 2/(2 + N) e = ½*(N/(1-))½ *(1/NE + 1/NC )½

ANOVA results: Compute R2 = SSTreatment/Sstotal

Treat R as a point biserial correlation

Excel workbook for Mean difference computation

STUDY# OUTCOME#STATISTIC TYPE MEAN E MEAN C SD E SD C Ne Nc N

SUMMARY STATISTIC

COMPUTATION d

intermediate computation

intermediate computation hedges g

1 1means, SDs 101 82 22 21 78 32 110 19 0.8817 0.8817 0.875563

2 1 t-statistic 101 82 78 32 110 4.2 0.881705 0.881705 0.875568

3 1 F-statistic 78 32 110 17.64 0.881705 0.881705 4.2 0.875568

4 1point-biserial r 78 32 110 0.374701 0.881705 0.881705 17.64 4.2 0.875568

5 1 chi square 47 76 123 3.66 0.654634 0.654634 0.169989 0.169989 0.650568

6p(t-statistic) 47 76 123 0.05 0.654634 0.654634 1.979764 0.650568

Story Book ReadingReferences1 Wasik & Bond: Beyond the Pages of a Book: Interactive Book Reading and Language Development in Preschool Classrooms. J. Ed Psych 20012 Justice & Ezell. Use of Storybook Reading to Increase Print Awareness in At-Risk Children. Am J Speech-Language Path 20023 Coyne, Simmons, Kame’enui, & Stoolmiller. Teaching Vocabulary During Shared Storybook Readings: An Examination of Differential Effects. Exceptionality 20044 Fielding-Barnsley & Purdie. Early Intervention in the Home for Children at Risk of Reading Failure. Support for Learning 2003

Coding the Outcome1 open Wasik & Bond pdf2 open excel file “computing mean effects

example”3 in Wasik find Ne and Nc4 decide on effect(s) to be used- three outcomes

are reported: PPVT, receptive, and expressive vocabulary at classroom and student level: what is the unit to be focused on? Multilevel issue of student in classroom, too few classrooms for reasonable MLM estimation, classroom level is too small for good power- use student data

Coding the Outcome5 Determine which reported data is usable: here

the AM and PM data are not usable because we don’t have the breakdowns by teacher-classroom- only summary tests can be used

6 Data for PPVT were analyzed as a pre-post treatment design, approximating a covariance analysis; thus, the interaction is the only usable summary statistic, since it is the differential effect of treatment vs. control adjusting for pretest differences with a regression weight of 1 (ANCOVA with a restricted covariance weight):

Interactionij = Grand Mean – Treat effect –pretest effect = Y… - ai.. – b.j.

Graphically, the Difference of Gain inTreat(post-pre) and Gain in Control (post –pre)

• F for the interaction was F(l,120) = 13.69, p < .001.• Convert this to an effect size using excel file Outcomes

Computation• What do you get? (.6527)

Coding the OutcomeY

Control Treatment

gains

Gain not “predicted” from control

pre

post

Coding the Outcome7 For Expressive and Receptive Vocabulary,

only the F-tests for Treatment-Control posttest results are given:Receptive: F(l, 120) = 76.61, p < .001Expressive: F(l, 120) =128.43, p< .001

What are the effect sizes? Use Outcomes Computation1.5441.999

Getting a Study Effect• Should we average the outcomes to get a single

study effect or• Keep the effects separate as different constructs

to evaluate later (Expressive, Receptive) or• Average the PPVT and receptive outcome as a

total receptive vocabulary effect?Comment- since each effect is based on the same

sample size, the effects here can simply be averaged. If missing data had been involved, then we would need to use the weighted effect size equation, weighting the effects by their respective sample size within the study

Getting a Study EffectFor this example, let’s average the three

effects to put into the Computing mean effects example excel file- note that since we do not have means and SDs, we can put MeanC=0, and MeanE as the effect size we calculated, put in the SDs as 1, and put in the correct sample sizes to get the Hedges g, etc.

(.6567 + 1.553 + 2.01)/3 = 1.4036

2 Justice & EzellReceptive: 0.403Expressive: 0.8606Average = 0.6303

3 Coyne et al• Taught Vocab: 0.9385• Untaught Vocab: 0.3262• Average = 0.6323

4 Fielding• PPVT: -0.0764

Computing mean effect sizeUse e:\\Computing mean effects1.xls

12345678

A B C D E F G H I J K L

Study Mean E Mean C SDE SDC d

Hedges g Ctrl N Trmt N N w wd

1 1.4036 0 1 1 0.65 1.40 61 63 124 24.87 34.912 0.6303 0 1 1 0.63 0.61 15 15 30 7.16 4.393 0.6323 0 1 1 0.63 0.62 30 34 64 15.20 9.494 0.5 -0.46 1 1 0.02 -0.08 23 26 49 12.20 -0.93

mean 0.8054 32.25 34.50 66.75 59.43 47.86s(mean) 0.1297

Mean

Computing Correlation Effect SizesReported Pearson correlation- use thatRegression b-weight: use t-statistic reported,

e = t*(1/NE + 1/NC )½ t-statistics: r = [ t2 / (t2 + dferror) ] ½ Sums of Squares from ANOVA or ANCOVA:

r = (R2partial) ½ R2partial = SSTreatment/Sstotal

Note: Partial ANOVA or ANCOVA results should be noted as such and compared with unadjusted effects

Computing Correlation Effect SizesTo compute correlation-based effects, you can

use the excel program “Outcomes Computation correlations”

The next slide gives an example.Emphasis is on disaggregating effects of

unreliability and sample-based attenuation, and correcting sample-specific bias in correlation estimation

For more information, see Hunter and Schmidt (2004): Methods of Meta-Analysis. Sage.

Correlational meta-analyses have focused more on validity issues for particular tests vs. treatment or status effects using means

Computing Correlation Effects Example

STUDY# OUTCOME# x alpha y alpha Ne Nc N r rcorrected s(r ) Nr N*(r-rmean)disattenuated r Ndisr s(edis)

reliabiltiy reliabiltiy1 1 0.80 0.77 47 76 123 0.352646 0.351381 0.079277 43.21983 -5.1631 0.449313 55.26549 0.1010082 1 0.70 0.80 33 55 88 0.323444 0.32178 0.095995 28.31665 -6.26369 0.432221 38.03544 0.1282793 1 0.75 0.90 22 45 67 0.190571 0.18918 0.118621 12.67504 -13.6715 0.231956 15.54103 0.1443814 1 0.88 0.70 111 111 222 0.67 0.669165 0.037071 148.5545 61.13375 0.853659 189.5123 0.0472335 1 0.77 0.85 34 34 68 0.169989 0.168757 0.118639 11.47548 -15.2751 0.210119 14.28811 0.1466476 1 0.90 0.78 47 45 92 0.177133 0.17619 0.101539 16.20946 -20.0091 0.211412 19.44991 0.12119

N(r-rmean) N(rdis-rdismean)0.229994041 0.356773 r(mean)= 0.3946230.466932722 0.442974 rdis(mean)= 0.503172.827858564 4.92834 Var(rmean )= 0.00113816.73286084 27.27103 s(rmean)= 0.0337393.469040187 5.8397574.389591025 7.831295 s(emean)= 0.080498

s(edismean)=

EFFECT SIZE DISTRIBUTION

Hypothesis: All effects come from the same distribution

What does this look like for studies with different sample sizes?

Funnel plot- originally used to detect bias, can show what the confidence interval around a given mean effect size looks likeNote: it is NOT smooth, since CI depends on

both sample sizes AND the effect size magnitude

EFFECT SIZE DISTRIBUTIONEach mean effect SE can be computed from

SE = 1/ (w)

For our 4 effects: 1: 0.200525 2: 0.373633 3: 0.256502 4: 0.286355

These are used to construct a 95% confidence interval around each effect

EFFECT SIZE DISTRIBUTION- SE of Overall MeanOverall mean effect SE can be computed

fromSE = 1/ (w)

For our effect mean of 0.8054, SE = 0.1297Thus, a 95% CI is approximately (.54, 1.07)The funnel plot can be constructed by

constructing a SE for each sample size pair around the overall mean- this is how the figure below was constructed in SPSS, along with each article effect mean and its CI

EFFECT SIZE DISTRIBUTION- Statistical testHypothesis: All effects come from the same

distribution: Q-testQ is a chi-square statistic based on the

variation of the effects around the mean effect

Q = wi ( g – gmean)2

Q 2 (k-1)

k

Example Computing Q Excel file

effect d w   Qi prob(Qi) sig?

1 0.58 5.43   0.7151598 0.397736175no

2 -0.05 10.24   0.7326248 0.392033721no

3 0.52 4.35   0.3957949 0.52926895no

4 0.02 9.69   0.366319 0.545017585no

5 -0.30 40.65   10.697349 0.001072891yes

6 0.14 29.94   0.1686616 0.681304025no

7 0.68 54.85   11.727452 0.000615849yes

8 -0.02 4.00   0.2125622 0.644766516no

             

  0.2154   Q= 25.015924    

      df 7    

      prob(Q)= 0.0007539    

Computational Excel fileOpen excel file: Computing QEnter the effects for the 4 studies, w for each

study (you can delete the extra lines or add new ones by inserting as needed)

from the Computing mean effect excel fileWhat Q do you get? Q = 39.57 df=3 p<.001

Interpreting QNonsignificant Q means all effects could have

come from the same distribution with a common mean

Significant Q means one or more effects or a linear combination of effects came from two different (or more) distributions

Effect component Q-statistic gives evidence for variation from the mean hypothesized effect

Interpreting Q- nonsignificantSome theorists state you should stop-

incorrect.Homogeneity of overall distribution does not

imply homogeneity with respect to hypotheses regarding mediators or moderators

Example- homogeneous means correlate perfectly with year of publication (ie. r= 1.0, p< .001)

Interpreting Q- significantSignificance means there may be

relationships with hypothesized mediators or moderators

Funnel plot and effect Q-statistics can give evidence for nonconforming effects that may or may not have characteristics you selected and coded for