advanced statistics for researchers
DESCRIPTION
Advanced Statistics for Researchers. Meta-analysis and Systematic Review Avoiding bias in literature review and calculating effect sizes Dr. Chris Rakes October 9, 2013 With Special Thanks to Dr. Jeff Valentine. Research Statistics Framework. Conceptual Framework(s) - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/1.jpg)
Advanced Statistics for Researchers
Meta-analysis and Systematic ReviewAvoiding bias in literature review and calculating effect sizes
Dr. Chris RakesOctober 9, 2013
With Special Thanks to Dr. Jeff Valentine
![Page 2: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/2.jpg)
Research Statistics Framework
Descriptive & Inferential Statistics
w/ Probability
General Linear Model• (M)AN(C)OVA• Simple and Multiple
Regression• Exploratory Factor
Analysis
Meta-AnalysisStructural Equation
ModelingHierarchical Linear
ModelingItem Response TheoryBayesian Estimation
Secondary Data Analysis
Qualitative Data Analysis
Writing for Publication
Conceptual Framework(s)Research Design, Missing Data, Statistical Assumptions, Measurement Reliability &
Validity
![Page 3: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/3.jpg)
In this SessionUsing Systematic Review to
Obtain a Better Literature Review/Conceptual Framework
Minimizing Publication BiasHow to compute various types of
effect sizesFixed vs. Random EffectsComputing a Design Effect
![Page 4: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/4.jpg)
USING SYSTEMATIC REVIEW TO OBTAIN A BETTER LITERATURE REVIEW/CONCEPTUAL FRAMEWORK
![Page 5: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/5.jpg)
Why Systematic Review?Synthesis of results of multiple
studies provides more compelling evidence than results of any single study.◦Less effected than single studies by
sampling error◦More confidence in results: place
single studies in context
![Page 6: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/6.jpg)
Problems with Narrative ReviewsLiterature search virtually never thorough
in scope or in reporting how literature was located.
Under-reported methodology (why were certain studies included or excluded?)◦Often unstated, virtually always arbitrary◦Potential Confirmation Bias
Conflate statistical significance with effect size
Ignore Type II error in primary studiesIgnore publication biasOften employ vote counts
![Page 7: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/7.jpg)
Steps for a Systematic ReviewGoal: Uncover All Relevant
StudiesMore realistic goal: Minimize
differences between retrieved and un-retrieved studies.Populatio
n of StudiesAccessibl
e Populatio
nSample
All Relevant Studies
All Retrievable Relevant Studies
Retrieved Studies
![Page 8: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/8.jpg)
Searching Electronic DatabasesAlways consult with a professional librarian!!!!
Identify potentially relevant databases.Search terms must appear in an indexed
field.◦Often must be exhaustive with terms
Deep substantive knowledge of the research questions is required to capture the relevant terms
Strongly susceptible to disciplinary bias (vet thoroughly)
Full text search capability will help some
![Page 9: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/9.jpg)
Gray LiteratureGeneric search engines such as Google
and Google Scholar can sometimes help identify unpublished material
ProQuest Dissertations and Theses will house dissertation research http://aok.lib.umbc.edu/databases/dblink.php?DBID=370
Research Organizations in Your Field often house technical reports on their websites (e.g., CRESST)
Bibliographies of already-identified relevant studies.
![Page 10: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/10.jpg)
Publication BiasKnown difference in statistical
significance of published vs. unpublished studies.
The best defense is a comprehensive, systematic search for literature.
![Page 11: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/11.jpg)
Key Decisions in Literature ReviewInter-Rater Agreement on Key
Decisions◦Does the study look like it might be
relevant? If yes, retrieve full text of article.
◦Is the study eligible for inclusion? Base final decision on full text.
Double code as much as possible.
![Page 12: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/12.jpg)
Browse UMBC’s Databases
![Page 13: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/13.jpg)
COMPUTING EFFECT SIZES
![Page 14: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/14.jpg)
Statistical SignificanceInterpretation of a p-value
◦Given a true null hypothesis, the probability of observing a relationship at least as large as the one being tested.
◦The confidence we can state the direction of a relationship (positive or negative)
◦Likelihood that a result is due to random chance (i.e., sampling error).
A p-value is a function of sample size and effect size.
![Page 15: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/15.jpg)
Effect SizeEstimates the magnitude (size) of
a relationship (i.e., how much impact?)
Three families of effect size◦Correlation Coefficients (r)◦Odds Ratios (OR; Two Dichotomous
Variables)◦Mean Differences (d)
![Page 16: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/16.jpg)
The role of sample sizeAny non-zero difference in means
will be statistically significant given a large enough sample. Assume:◦MT = 100.1, MC = 100.0, sp = 15
n per group d t-test p-value100 0.01 0.962
1000 0.01 0.88210000 0.01 0.637
100000 0.01 0.136200000 0.01 0.035
![Page 17: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/17.jpg)
Two categories of Effect SizesUnstandardized
◦Effects expressed directly in terms of the measured outcome (e.g., “3 points on an IQ scale”)
◦Most useful when scale is well understood and relevant studies all use the same scale.
Standardized: transforming effects to have similar meaning across scales◦Standard Deviation Units◦Percent Change◦Proportion of Variance Explained
![Page 18: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/18.jpg)
Computing an Odds RatioGraduated Didn’t
GraduateTreatment 2 (a) 6 (b)Control 9 (c) 12 (d)
![Page 19: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/19.jpg)
Standardized Effect Size: Mean Difference (d or Cohen’s d),
◦where is the pooled standard deviation.
Study n1 Ȳ1s1 (s1)2 n2 Ȳ2
s2 (s2)2
1 59 17.25 3.26 10.6276 50 19.32 3.53 12.4609
![Page 20: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/20.jpg)
Computing ES: An ExampleStudy n1 Ȳ1
s1 (s1)2 n2 Ȳ2s2 (s2)2
1 59 17.25 3.26 10.6276 50 19.32 3.53 12.4609
![Page 21: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/21.jpg)
Weighting Effect SizesWeight by the inverse of the
variance of the effect size
Report d n1 n2 d2 n1n2 2n1n2 n1 + n2 n1n2d2 Numerator: 2n1n2(n1 +n2)
Denominator: 2(n1 +n2)2 + n1n2d2 w w*d
=B2^2 =C2*D2 =2*C2*D2 =C2+D2 =C2*D2*(B2^2) =G2*H2 =2*(H2^2)+I2 =IFERROR(J2/K2,0) =IFERROR(L2*B2,0)
davg: Sums: =SUM(L2:L2) =SUM(M2:M2)
=$D$4+(1.96*$N$2)davg HI95:=IF(L4=0,0,M4/L4)
=$D$4-(1.96*$N$2)davg LO95:
![Page 22: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/22.jpg)
Try It! Go to http://csrakes.yolasite.com for the template
n1 Ȳ1s1 n2 Ȳ2
s2
5 3.3 1.2 17 4.2 0.735 1.5 0.5 17 4.2 0.7
5 3.3 1.2 32 3 0.535 1.5 0.5 32 3 0.5
![Page 23: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/23.jpg)
Conversion FormulasCooper, H. (1998). Synthesizing
research. Thousand Oaks, CA: Sage.
If mean, SD, and n available, use regular formula
Convert from r, t, F(1,X), and dichotomous proportions
![Page 24: Advanced Statistics for Researchers](https://reader035.vdocument.in/reader035/viewer/2022062520/56816421550346895dd5e275/html5/thumbnails/24.jpg)
SoftwareComprehensive Meta-Analysis:
http://www.meta-analysis.com/index.php?gclid=CITZk8WSiroCFRCg4AodJhYA7Q
Microsoft Excel: Home-made formulasn1 Ȳ1 s1 (s1)2 n2 Ȳ2 s2 (s2)2 sp d d2 n1n2 n1 + n2 SEd Lo95d Hi95d
=E3^2 =I3^2 =SQRT((F3+J3)/2) =(H3-D3)/K3 =L3^2 =C3*G3 =C3+G3 =SQRT((O3/N3)+(M3/(2*O3))) =L3-(1.96*P3) =L3+(1.96*P3)
Control Group Treatment Group