Systematic Reviews: The Potential of

Meta-analysis

ESRC Research Methods FestivalOxford

5th July, 2012

Professor Steven HigginsDurham Universitys.e.higgins@durham.ac.uk

What is meta-analysis?

A way of combining the results of quantitative research To accumulate evidence from smaller studies To compare results of similar studies - consistency To investigate patterns of association in the findings of

different studies – explaining variation ‘Surveys’ research studies

Key points

Understanding ‘effect-size’ as a common measure

Why do we need meta-analysis? What are its limitations? What is its potential?

What is an “effect size”?

Standardised way of looking at difference Different methods for calculation

Binary (Risk difference, Odds ratio, Risk ratio) Continuous

Correlational (Pearson’s r) Standardised mean difference (d, g, Δ)

Difference between control and intervention group as proportion of the dispersion of scores

Intervention group score – control group score / standard deviation of scores

Examples of Effect Sizes:

ES = 0.2“Equivalent to the difference in heights between 15 and 16 year old girls”

58% of

control group below

mean of experimental

group

Probability you could guess which group a person was in = 0.54Change in the proportion above a given threshold:

from 50% to 58% or from 75% to 81%

“Equivalent to the difference in heights between 13 and 18 year old girls”

79% of

control group below

mean of experimental

group

Probability you could guess which group a person was in = 0.66

ES = 0.8

Change in the proportion above a given threshold:

from 50% to 79% or from 75% to 93%

The rationale for using effect sizes Traditional quantitative reviews focus on statistical

significance testing Highly dependent on sample size Null finding does not carry the same “weight” as a

significant finding Meta-analysis focuses on the direction and

magnitude of the effects across studies From “Is there a difference?” to “How big is the differenc

e?” and “How consistent is the difference?” Direction and magnitude represented by “effect size”

Meta-analysis

Synthesis of quantitative data Cumulative Comparative Correlational

“Surveys” educational research (Lipsey and Wilson, 2001)

Forest plots

Effective way of presenting results Studies, effect sizes, confidence intervals Provides an overview of consistency of effects Summarises an overall effect (with confidence

interval) Useful visual model of a meta-analysis

Anatomy of a forest plot…

Studies

N of studyLine of no effect

C.I

Study effect size

Pooled effect size

Pooled effect size

Study effect size (with C.I.)

Weighting of study in meta-analysis

Issues and challenges in meta-analysis

Conceptual Reductionist - the answer is .42 Comparability - apples and oranges Atheoretical - ‘flat-earth’

Technical Heterogeneity Publication bias Methodological quality

Some recent findings from meta-analysis in education

Klauer & Phye 2008 74 studies, 3,600 children - training in inductive reasoning improves academic

performance (0.69) more than intelligence test performance (0.52).

Gersten et al. 2009 Maths interventions for low attainers - 42 studies ES ranging from 0.21-1.56.

Teaching heuristics and explicit instruction particularly beneficial.

Domino 2010 31 studies / 5288 - pupils those who used manipulatives during mathematics

instruction had higher mathematics achievement than students who were taught by traditional teaching methods - effect size 0.50 (CI 0.34 to 0.65)

Methodological heterogeneity

Study design Sample characteristics Assessment (measures, timing)

Educational heterogeneity

‘Clinical’or ‘pedagogical’ heterogeneity Systematic variation in response to the

intervention Teacher level effects Pupil level effects

Statistical

Due to chance Unexplainable

Statistical methods to identify heterogeneity

Presence Q statistic (Cooper & Hedges, 1994)

Significance level (p-value) 2

2

Extent I2 (Higgins & Thompson, 2002)

If it exceeds 50%, it may be advisable not to combine the studies

All have low power with a small number of studies (Huedo-Medina et al. 2006)

Exploring heterogeneity In a meta-analysis, exploring heterogeneity of effect

can be as or even more important than reporting averages

Exploring to what extent the variation can be explained by factors in the coding of studies (age, gender, duration of intervention etc) through regression

Forming sub-groups with greater homogeneity Identifying the extent of the variation through

further analysis

Coding for exploration

Factors which may relate to variation The intervention

E.g. duration, intensity, design, implementation The sample

E.g. age, gender, ethnicity, particular needs The research

E.g. design (RCT, quasi-experimental), quality, tests/outcomes, comparison group

Pooling the results In a meta-analysis, the effects found across studies are

combined or ‘pooled’ to produce a weighted average effect of all the studies-the summary effect.

Each study is weighted according to some measure of its importance.

In most meta-analyses, this is achieved by giving a weight to each study in inverse proportion to the variance of its effect.

Fixed effect model The difference between the studies is due to

chance Observed study effect = Fixed effect + error

Fixed effect model

Each study is seen as being a sample from a distribution of studies, all estimating the same overall effect, but differing due to random error

Random effects model

Assumes there are two component of variation1. Due to differences within the studies (e.g. different

design, different populations, variations in the intervention, different implementation, etc.)

2. Due to sampling error

Random effects model

Each study is seen as representing the mean of a distribution of studies

There is still a resultant overall effect size

“Random effects” model assumes a different underlying effect for each study.

This model gives relatively more weight to smaller studies and wider confidence intervals than fixed effect models.

The use of this model is recommended if there is heterogeneity between study results.

Also recommended as it provides a more conservative estimate for the pooled effect.

Which model?

Exploring heterogeneity

Conceptual: are the studies sufficiently similar in terms of the intervention or treatment?

Statistical: greater variation than would be predicted

Sensitivity analysis

Provides feedback about whether assumptions and decisions made during the meta-analysis have had a major effect on the results

Repeats the analysis using different assumptions (as a quality check to make sure results are consistent) e.g. Effect of including and excluding low quality studies Excluding and including outliers Undertaking fixed effect and and random-effects

analyses

Meta-regression

Examines the impact of moderator variables on pooled effect size using regression-based techniques.

Estimates the extent to which covariates (e.g. age, intervention length) can explain between study heterogeneity

If covariate is not associated with heterogeneity then it will not be significant in the regression

Interpreting review findings

The standardised mean difference represents the amount of a standard deviation that the two groups differ by

Can therefore be converted back to a more ‘user-friendly’ metric. For example fruit and vegetable consumption was found to have

increased by a standardised mean difference of 0.65 If, on baseline fruit and vegetable consumption was

measured as being 2.4 portions per day with a standard deviation of 0.9, we can say that the intervention increased consumption by 0.585 portions, or from 2.4 to nearly 3 portions per day

Cumulative meta-analysis Meta-analysis can have powerful applications e.g. detecting changes in paradigms

Nykänen H & Koricheva J (2004) Damage-induced changes in woody plants and their effects on insect herbivore performance: a meta-analysis. Oikos, 104, 247-268. They measured the responses of woody plants to natural or simulated damage. “Cumulative meta-analyses revealed dramatic temporal changes in the magnitude and direction of the plant and herbivore responses reported during the last two decades.”Not a change in plant behaviour: a change in human understanding of them (and thus a change in measurement practices).

Comparative meta-analysis

Theory testing Practical value

Ability grouping

Slavin 1990 b (secondary low attainers) -0.06

Lou et al 1996 (on low attainers) -0.12

Kulik & Kulik 1982 (secondary - all) 0.10

Kulik & Kulik 1984 (elementary - all) 0.07

Meta-cognition and self-regulation strategiesAbrami et al. 2008 0.34

Haller et al. 1988 0.71

Klauer & Phye 2008 0.69

Higgins et al. 2004 0.62

Chiu 1998 0.67

Dignath et al. 2008 0.62

Summary Meta-analysis is only as good as the systematic review in

which it is located Systematic bias in search strategy can lead to invalid results Sensitivity analyses are essential in order to explore the robustness

of the findings Heterogeneity must be examined

A statistical method for combining the quantitative results of primary studies Cumulative Comparative

Meta-analysis overcomes a lack of statistical power in small primary studies Offers a more precise estimate of effect Offers a way to explore systematic variation

Can settle controversies from apparently conflicting studies or generate new hypotheses

References, further readings and information

Books and articlesBorenstein, M., Hedges, L.V., Higgins, J.P.T. & Rothstein, H.R. (2009) Introduction to Meta Analysis (Statistics in Practice) Oxford: Wiley

Blackwell.Chambers, E.A. (2004). An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). Journal of

Educational Research, 98, pp 35-44.Cooper, H.M. (1982) Scientific Guidelines for Conducting Integrative Research Reviews Review Of Educational Research 52; 291.*Cooper, H.M. (2009) Research Synthesis and meta-analysis: a step-by-step approach London: SAGE Publications (4th Edition).Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward

reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, Ca.: Jossey-Bass. Glass, G.V. (2000). Meta-analysis at 25. Available at: http://glass.ed.asu.edu/gene/papers/meta25.html (accessed 9/9/08)Lipsey, Mark W., and Wilson, David B. (2001). Practical Meta-Analysis. Applied Social Research Methods Series (Vol. 49). Thousand

Oaks, CA: SAGE Publications.Torgerson, C. (2003) Systematic Reviews and Meta-Analysis (Continuum Research Methods) London: Continuum Press.

WebsitesWhat is an effect size?, by Rob Coe: http://www.cemcentre.org/evidence-based-education/effect-size-resources The meta-analysis of research studies: http://echo.edres.org:8080/meta/The Meta-Analysis Unit, University of Murcia: http://www.um.es/metaanalysis/The PsychWiki: Meta-analysis: http://www.psychwiki.com/wiki/Meta-analysis Meta-Analysis in Educational Research: http://www.dur.ac.uk/education/meta-ed/

Acknowledgements This presentation is an outcome of the work of the ESRC-funded Researcher Development

Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008-2011.

The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas (Institute of Education, London University).

The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta-analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11).

The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.