significance and effect sizes what is the problem with just using p-levels to determine whether one...
TRANSCRIPT
Evaluation Research and Meta-analysis
Significance and effect sizesWhat is the problem with just using p-levels to
determine whether one variable has an effect on another?
Don’t EVER just give p-range!Sample results:
For boys, r (87) = .31, p = .03For girls, r (98) = .24, p = .14
Significance test = effect size x study sizeWhy are effect sizes important? What is the difference between statistical,
practical, and clinical significance?
What should you report? 2 group comparison—treatment vs. control
on anxiety symptoms3 group comparison—positive prime vs.
negative prime vs. no prime on number of problems solved
2 continuous variables—relationship between neuroticism and goal directedness
3 continuous variables—anxiety as a function of self-esteem and authoritarian parenting
2 categorical variables—relationship between answers to 2 multiple choice questions
Narrative vs. quantitative reviewsWhen was the first meta-analysis? When was the term first used? What are the advantages of quant reviews? What are particular critiques of them? What are the three basic principles to guide
meta-analysis?
Steps to meta-analysis
1. define your variables/question1 df contrastsWhat is a contrast?
2. Decide on inclusion criteriaWhat factors do you want to consider here?
3. Collect studies systematicallyWhere do you find studies? File drawer problemRosenthal’s fail-safe N
# studies needed at p < .05= (K/2.706) (K(mean Z squared) = 2.706) Z = Z for that level of p K = number of studies in meta-analysis
Funnel plotRank correlation test for pub biasWhat can you do if publication bias is a problem?
Trim and fillSensitivity analysisWeight studies
Fig. 3. Funnel plots of 11 (subsets of) meta-analyses from 2011 and Greenwald, Poehlman, Uhlman, and Banaij (2009).
Marjan Bakker et al. Perspectives on Psychological Science 2012;7:543-554
Copyright © by Association for Psychological Science
3. Calculate effect sizesIf there is more than 1 effect per study, what
do you do? What does the sign mean on an effect size? What are small, medium, and large effects?How can you convert from one to another? r or d? http://
www.soph.uab.edu/Statgenetics/People/MBeasley/Courses/EffectSizeConversion.pdf
Families of effect sizes2 group comparisons (difference between the means)
Cohen’s dHedge’s gGlass’s d or delta
Continuous or multi-group (proportion of variability)Eta squared η2
Partial eta-squared ηp2
Generalized eta-squared η G2
r, fisher’s z, R2, adjusted R2
ω2 and its partsdifference between η2 and R2 family
Nonparametric effect sizesNonnormal data: convert z to r or dCategorical data:
Rho Cramer’s V Goodman-Kruskal’s Lambda
How can you increase your effect sizes?How can you calculate confidence intervals
around your effect sizes? http://
www.latrobe.edu.au/psy/research/cognitive-and-developmental-psychology/esci
http://www.cem.org/effect-size-calculator
Interpretation of effect sizesRecommended for at least most important
findingsPSUBinomial effect size display (p. 76)
Relative riskOdds ratioRisk difference
4. Look at heterogeneity of effect sizesChi-square testI2 (measure based on Chi-square)Cochran’s QStandard deviations of effect sizesStem and leaf plot (p. 671)Box plotForest plotWhat are common moderators you might
test? How would you do that?
Forest plot
5. Combine effect sizesWhen should you do fixed vs. random effects? Should you weight effect sizes, and if so, on
what?How can you deal with dependent effect
sizes? Hunter and Schmidt method vs. Hedges et al.
methodCredibility intervals vs. confidence intervals
6. Calculate confidence intervals/ 7. Look for moderatorsWhat are common moderators you might
test? How do you compare moderators?
“Meta-analysis”Comparing and combining effect sizes on a
smaller level—when might you want to do this?
How would you do it? Average within-cell r’s with fisher z
transformsTo compare independent r’s: Z = z1-z2/sqrt
((1/n-3) + (1/n-3))To combine independent r’s: z = z1+z2/2
Write-upInclusion criteria, search, what effect sizeWhich m-a tech and whyStem and leaf plots of effect sizes (and maybe
mods)Forest plotsStats on variability of effect sizes, estimate of
pop effect size and confidence intervalsPublication bias analyses
Side noteAnalysis of power (Appendix)
TermsEvolutionary epistemologyEvidence-based practiceSystems thinking
Dynamical systems approachesEvaluation research
Issues with evaluation researchWhat questions are asked?What methods are used?What unique issues emerge?
Types of evaluationFormative
Needs assessmentEvaluability assessmentStructured conceptualizationImplementation evaluationProcess evaluation
SummativeOutcome evaluationImpact evaluationCost-benefit analysisSecondary analysisMeta-analysis
Methods used for different ?sWhat is the scope of the problem?How big is the problem?How should we deliver the program?How well did we deliver it? What type of evaluation can we do? Was the program effective?What parts of the program work?Should we continue the program?
Evidence based medicine (Sackett et al.)Convert problem into questionFind evidenceEvaluate validity, impact, applicabilityIntegrate patient experience and clinical
judgmentReview evaluation
What does the book authorMean by an “evaluation culture”? Is it a good thing?
Post spring breakReadings on analyses (some to be emailed
out)Quant article critique is separate from
thought paper (look for questions at end of syllabus)
One more week then rough drafts due