psychology 290 special topics study course: advanced meta-analysis january 27, 2014
TRANSCRIPT
![Page 1: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/1.jpg)
Psychology 290
Special Topics Study Course: Advanced Meta-analysis
January 27, 2014
![Page 2: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/2.jpg)
Overview
• Discussion of the Hedges / Hanushek exchange.
• Investigation of vote counting as an inferential method.
![Page 3: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/3.jpg)
Education finance exchange
• Larry Hedges was then a professor of educational statistics at the University of Chicago.
• Eric Hanushek was an economist at the University of Rochester.
• Now at Northwestern and Stanford, respectively.
![Page 4: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/4.jpg)
Background of the paper
• Hanushek at the time was extremely active as an expert witness in educational equity lawsuits.
• Paper grew out of a student project in Hedges’ meta-analysis class.
• Discussion.
![Page 5: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/5.jpg)
Vote counting as inference
• To understand vote counting as an inferential method, we need to understand the probability that an individual study will reject the null hypothesis.
• Statisticians have a name for that idea.• Power.
![Page 6: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/6.jpg)
What does power depend on?
• Lots of things: –characteristics of population –choices about how to do inference–characteristics of the sample.
![Page 7: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/7.jpg)
Characteristics of the population
• How strong is the effect? • How much unmodeled variability exists?
![Page 8: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/8.jpg)
Choices about how to do inference
• Alpha level.• One- vs. two-tailed tests.
![Page 9: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/9.jpg)
Characteristics of the sample
• Sample size.
![Page 10: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/10.jpg)
Back to vote counting
• To understand vote counting, we need to understand power.
• We’ve just seen that power is a complex function of lots of factors.
• If we want to understand something that is too complex, what can we do?
• Simplify.
![Page 11: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/11.jpg)
Simplifying
• All of those issues that were characteristics of the population can be simplified by, for the moment, confining our interests to the fixed-effects context.
• In that case, we are assuming that all of the studies are samples from the same population.
![Page 12: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/12.jpg)
Simplifying
• All of the issues that were characteristics of how we do inference are under our control.
• For example, we can simply say that a vote is positive if the null hypothesis is rejected using a two-tailed test with an alpha level of .05.
![Page 13: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/13.jpg)
Simplifying
• The remaining issue that effects power is the sample size of the individual study.
• Obviously, in the real world, different studies will have different N.
• Simplify by assuming that all studies have the same N.
![Page 14: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/14.jpg)
Simplifying
• With these simplifying assumptions, we can treat power (i.e., the probability of a positive vote) as a constant.
![Page 15: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/15.jpg)
Distribution of votes
• We can now think of our studies as a series of independent attempts to vote YES.
• For each attempt, the probability of a YES vote is the same (power).
• We are interested in the total number of YES votes.
• This should sound vaguely familiar.
![Page 16: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/16.jpg)
Distribution of votes
• Suppose that instead of studies and YES votes, I were talking about coin tosses and HEADS outcomes.
• We would be looking at a series of independent coin tosses with a constant probability of success, and would be interested in the probability of a particular number of successes.
![Page 17: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/17.jpg)
Distribution of votes
• With the simplifying assumptions we have made, the number of YES votes follows a binomial distribution.
![Page 18: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/18.jpg)
What should we assume for power?
• Given that Hanushek is arguing that there is no effect, we should be justified in considering it to be “small.”
• Empirical studies of power.
![Page 19: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/19.jpg)
Cohen 1962
• The statistical power of abnormal-social psychological research, Journal of Abnormal and Social Psychology, 65, 145-153.
• Finding: 100% of studies of small effects in that field had power of < .50.
![Page 20: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/20.jpg)
Brewer 1973
• A note on the power of statistical tests in the Journal of Educational Measurement, Journal of Educational Measurement, 10, 71-73.
• Very much the same finding as Cohen.
![Page 21: Psychology 290 Special Topics Study Course: Advanced Meta-analysis January 27, 2014](https://reader036.vdocument.in/reader036/viewer/2022062421/56649ceb5503460f949b7b47/html5/thumbnails/21.jpg)
Understanding vote counting
• What happens with vote counting as the number of studies becomes large?
• (Another digression in R.)• Using the normal approximation to the
binomial distribution.