comparison of bayesian and classical meta-analysis-powerpoint

8/13/2019 Comparison of Bayesian and Classical Meta-Analysis-Powerpoint

1/23

COMPARISON OFBAYESIAN AND CLASSICAL

META-ANALYSISJoe P King

Educational Psychology

Measurement, Statistics, and Research Design


2/23

Meta Analysis

Overview of Meta-Analysis

Traditional Meta Analysis

Bayesian Meta Analysis

Why are they different? Why is it important?

Comparing methods using experimental data

Implications

Conclusions


3/23

Introduction

Meta analysis is used in every field of scientific research

Meta analysis allows us to compile many studies to test a

theory across many studies.

The goal is to take the many studies which have looked atan outcome variable and try to inform on a theory.

This satisfies one of the tenants of science that research

must be reproducible and not one study can confirm or

deny a theory


4/23

Introduction

Methods of Meta Analysis

Problem formulation

Data collection and selection of relevant studies

Evaluation of data collected

Interpretation and analysis

Presentation of results


5/23

Example

Study by Rakes, Valentine, McGatha, and Ronau (2010)

analyzed different methods of algebra instruction they

found 82 relevant studies, 109 independent effect sizes,

and a total sample size of 22424 students.

Searched electronic journals for relevant articles,

calculated effect sizes and in some cases weighted effect

sizes due to different sampling techniques or small

sample size

In 5 strategies of teaching algebra they found statisticalsignificance for each in at least one model, which should

inform on how math instructors teach algebra.


6/23

Treatment

Communities that Care

Seeks to prevent anti-social behaviors among the youth

in a community and strengthen pro-social behaviors.

Provides community leaders with training, materials andtechnical assistance in the advancement of the program

(Coie, Watt, & West, 1993; Mrazek, Haggerty, &

Committee on Prevention of Mental Disorders, 1994).


7/23


8/23

Outcomes

Two Measures were Used for this analysis

Delinquent behavior

Fifth grade, delinquent behaviors were categorized as

stealing, property damage, shoplifting, and attacking

someone with intention of hurting them.

Eighth grade, delinquent behaviors were carrying a gun to

school, beating up someone, stealing a vehicle, selling

drugs, and being arrested.

Binge Drinking - 5 or more drinks at once occasion, andthe measurement was how many instances of binge

drinking per month


9/23

Traditional Meta Analysis

Effect size was collected for each pair of cities

Used multilevel modeling with two levels Level 1 was the pairs of cities and accounting for the

within experiment variance.

Level 2 sought to account for the between experiment

variance.


10/23

Why Bayes?

Specify Priors (this analysis uses non-informative priors)

Can see the effect of adding each experiment on effect

size and precision associated with the estimate.

Less Uncertainty in the experiment Easier to implement


11/23

Bayesian Analysis

Replication of Howard, Maxwell, & Fleming (2000)

They used Bayesian Calculations to calculate a posterior

distribution of 3 studies.

Basics of Bayesian Approach Prior Distribution Initial estimate of effect size

Likelihood Principle Effect size estimate of data

Posterior Distribution Final Effect Size Combining

Prior and Likelihood.


12/23

Calculations

What we know

Effect size for each Comparison

Uncertainty within Effect Size

What we will calculate

Precision

Posterior Effect Size for Each City then All

Cities

Uncertainty Around the Effect Size


13/23

Precision


14/23

Effect Size


15/23

Results Delinquent Behavior

Effect Size SE

Traditional Meta Analysis -0.31 0.1400

Bayesian Meta Analysis -0.17 0.0025


16/23

Results Binge Drinking

Effect Size SE

Bayesian Meta Analysis -0.43 0.0011


17/23

Results for Each ComparisonPosterior Distributions for Deliquent Behavior

Comparison Number

EffectSize

-1.0

-0.8

-0.6

-0.4

-0.2

0.0

0.2

0.4

0.6

0.8

1 2 3 4 5 6 7 8 9 10 11 12


18/23

Cumulative Results After Each IterationPosterior Distributions for Deliquent Behavior

Posterior Iteration

EffectSize

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

1 2 3 4 5 6 7 8 9 10 11 12


19/23

Results for Each ComparisonPosterior Distributions for Binge Drinking

Comparison Number

EffectSize

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

1 2 3 4 5 6 7 8 9 10 11 12


20/23

Cumulative Results After Each IterationPosterior Distributions for Binge Drinking

Posterior Iteration

EffectSize

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

1 2 3 4 5 6 7 8 9 10 11 12


21/23

Summary

Meta Analysis is important to research

Many methods exist yet pose limitations

Bayesian approach is an additional method that shows

promise.


22/23

Thank you

Dr. Charles Peck

Dr. Robert Abbott

Dr. Joe Lott

Kelly Jewell Mom and Dad


23/23

References

Coie, J., Watt, N., & West, S. (1993). The science of prevention: A conceptual

framework and some directions for a national research program.American

Psychologist, 48(10), 1013-1022.

Hedges, L. V., & Hedberg, E. C. (2007). Intraclass Correlation Values for Planning

Group-Randomized Trials in Education. Educational Evaluation and Policy

Analysis, 29(1), 60-87.

Howard, G. S., Maxwell, S. E., & Fleming, K. J. (2000). The proof of the pudding: an

illustration of the relative strengths of null hypothesis, meta-analysis, and

Bayesian analysis. Psychological Methods, 5(3), 315-332.

Mrazek, P. J., Haggerty, R. J., & Committee on Prevention of Mental Disorders, I. on

M. (1994). Reducing risks for mental disorders: Frontiers for preventive

intervention research. Washington, D.C. National Academy Press.Noble, J. H. (2006). Meta-analysis: Methods, strengths, weaknesses, and political

uses. Journal of Laboratory and Clinical Medicine, 147(1), 7-20.

R Development Core Team. (2010). R: A Language and Environment for Statistical

Computing. Vienna Austria: R Foundation for Statistical Computing.

comparison of bayesian and classical meta-analysis-powerpoint

Documents