publication bias in the social sciences: unlocking the file drawer€¦ · motivation publication...
TRANSCRIPT
![Page 1: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/1.jpg)
Publication Bias in the Social Sciences: Unlocking theFile Drawer
Annie Franco, Neil Malhotra, and Gabor Simonovits
BITSS 2014
December 2014
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 1 / 24
![Page 2: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/2.jpg)
Introduction
Paper
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 2 / 24
![Page 3: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/3.jpg)
Introduction
Motivation
Publication bias: greater likelihood of statistically significant resultsbeing published than statistically insignificant results, holding fixedresearch quality
Partial equilibrium (across-study bias): If editors/reviewers rejectinsignificant results (and authors strategically don’t submit them),then the published literature will overestimate effect sizes and bemore likely to report type I errors
General equilibrium (within-study bias): p-hacking/underreporting
I If authors selectively report significant results: inflation of effect sizesI Even if authors randomly report results: type I errors underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 3 / 24
![Page 4: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/4.jpg)
Introduction
Motivation
Publication bias: greater likelihood of statistically significant resultsbeing published than statistically insignificant results, holding fixedresearch quality
Partial equilibrium (across-study bias): If editors/reviewers rejectinsignificant results (and authors strategically don’t submit them),then the published literature will overestimate effect sizes and bemore likely to report type I errors
General equilibrium (within-study bias): p-hacking/underreporting
I If authors selectively report significant results: inflation of effect sizesI Even if authors randomly report results: type I errors underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 3 / 24
![Page 5: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/5.jpg)
Introduction
Motivation
Publication bias: greater likelihood of statistically significant resultsbeing published than statistically insignificant results, holding fixedresearch quality
Partial equilibrium (across-study bias): If editors/reviewers rejectinsignificant results (and authors strategically don’t submit them),then the published literature will overestimate effect sizes and bemore likely to report type I errors
General equilibrium (within-study bias): p-hacking/underreporting
I If authors selectively report significant results: inflation of effect sizesI Even if authors randomly report results: type I errors underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 3 / 24
![Page 6: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/6.jpg)
Introduction
Motivation
Publication bias: greater likelihood of statistically significant resultsbeing published than statistically insignificant results, holding fixedresearch quality
Partial equilibrium (across-study bias): If editors/reviewers rejectinsignificant results (and authors strategically don’t submit them),then the published literature will overestimate effect sizes and bemore likely to report type I errors
General equilibrium (within-study bias): p-hacking/underreportingI If authors selectively report significant results: inflation of effect sizes
I Even if authors randomly report results: type I errors underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 3 / 24
![Page 7: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/7.jpg)
Introduction
Motivation
Publication bias: greater likelihood of statistically significant resultsbeing published than statistically insignificant results, holding fixedresearch quality
Partial equilibrium (across-study bias): If editors/reviewers rejectinsignificant results (and authors strategically don’t submit them),then the published literature will overestimate effect sizes and bemore likely to report type I errors
General equilibrium (within-study bias): p-hacking/underreportingI If authors selectively report significant results: inflation of effect sizesI Even if authors randomly report results: type I errors underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 3 / 24
![Page 8: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/8.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precisionI Assumes outcome variables and effect sizes are comparable across
studiesI Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 9: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/9.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precisionI Assumes outcome variables and effect sizes are comparable across
studiesI Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 10: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/10.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precision
I Assumes outcome variables and effect sizes are comparable acrossstudies
I Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 11: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/11.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precisionI Assumes outcome variables and effect sizes are comparable across
studies
I Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 12: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/12.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precisionI Assumes outcome variables and effect sizes are comparable across
studiesI Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 13: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/13.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precisionI Assumes outcome variables and effect sizes are comparable across
studiesI Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 14: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/14.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precisionI Assumes outcome variables and effect sizes are comparable across
studiesI Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 15: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/15.jpg)
Literature
Existing Approaches
Examine published literature: compare sample sizes against effectsizes
I Indirect evidence; makes assumptions about what published literaturelooks like
I Sensitive to using different measures of precisionI Assumes outcome variables and effect sizes are comparable across
studiesI Mechanism is unclear
Try to find unpublished gray literatures (dissertations, working papers,conference papers, human subjects registries)
I Likely missing a lot of unpublished studies (population of conductedstudies unknown)
I Substantial quality differences (unobserved heterogeneity) betweenpublished and unpublished studies
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 4 / 24
![Page 16: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/16.jpg)
Methodology
Leveraging the Online Archive of TESS studies(2002-2012)
Known population of conducted studies (published and unpublished);not selecting on dependent variable
Time Sharing Experiments for the Social Sciences (TESS) fundssurvey experiments on representative samples
Winning proposals are selected via peer review, meet a minimumthreshold of quality and scholarly interest
Studies are comparable due to similar sampling method and mode ofadministration (Knowledge Networks), but span a large substantivearea
Studies are required to have requisite statistical power
Not obvious why TESS studies are unrepresentative of politicalscience research, but may understate publication bias if anything
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 5 / 24
![Page 17: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/17.jpg)
Methodology
Leveraging the Online Archive of TESS studies(2002-2012)
Known population of conducted studies (published and unpublished);not selecting on dependent variable
Time Sharing Experiments for the Social Sciences (TESS) fundssurvey experiments on representative samples
Winning proposals are selected via peer review, meet a minimumthreshold of quality and scholarly interest
Studies are comparable due to similar sampling method and mode ofadministration (Knowledge Networks), but span a large substantivearea
Studies are required to have requisite statistical power
Not obvious why TESS studies are unrepresentative of politicalscience research, but may understate publication bias if anything
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 5 / 24
![Page 18: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/18.jpg)
Methodology
Leveraging the Online Archive of TESS studies(2002-2012)
Known population of conducted studies (published and unpublished);not selecting on dependent variable
Time Sharing Experiments for the Social Sciences (TESS) fundssurvey experiments on representative samples
Winning proposals are selected via peer review, meet a minimumthreshold of quality and scholarly interest
Studies are comparable due to similar sampling method and mode ofadministration (Knowledge Networks), but span a large substantivearea
Studies are required to have requisite statistical power
Not obvious why TESS studies are unrepresentative of politicalscience research, but may understate publication bias if anything
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 5 / 24
![Page 19: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/19.jpg)
Methodology
Leveraging the Online Archive of TESS studies(2002-2012)
Known population of conducted studies (published and unpublished);not selecting on dependent variable
Time Sharing Experiments for the Social Sciences (TESS) fundssurvey experiments on representative samples
Winning proposals are selected via peer review, meet a minimumthreshold of quality and scholarly interest
Studies are comparable due to similar sampling method and mode ofadministration (Knowledge Networks), but span a large substantivearea
Studies are required to have requisite statistical power
Not obvious why TESS studies are unrepresentative of politicalscience research, but may understate publication bias if anything
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 5 / 24
![Page 20: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/20.jpg)
Methodology
Leveraging the Online Archive of TESS studies(2002-2012)
Known population of conducted studies (published and unpublished);not selecting on dependent variable
Time Sharing Experiments for the Social Sciences (TESS) fundssurvey experiments on representative samples
Winning proposals are selected via peer review, meet a minimumthreshold of quality and scholarly interest
Studies are comparable due to similar sampling method and mode ofadministration (Knowledge Networks), but span a large substantivearea
Studies are required to have requisite statistical power
Not obvious why TESS studies are unrepresentative of politicalscience research, but may understate publication bias if anything
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 5 / 24
![Page 21: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/21.jpg)
Methodology
Leveraging the Online Archive of TESS studies(2002-2012)
Known population of conducted studies (published and unpublished);not selecting on dependent variable
Time Sharing Experiments for the Social Sciences (TESS) fundssurvey experiments on representative samples
Winning proposals are selected via peer review, meet a minimumthreshold of quality and scholarly interest
Studies are comparable due to similar sampling method and mode ofadministration (Knowledge Networks), but span a large substantivearea
Studies are required to have requisite statistical power
Not obvious why TESS studies are unrepresentative of politicalscience research, but may understate publication bias if anything
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 5 / 24
![Page 22: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/22.jpg)
Methodology
Data Collection
Verified status of published articles (author CVs, content search)
Collected any unpublished manuscripts available on the web
Contacted over 100 researchers where we could not find a publishedpaper to find out about the fate of their TESS project
For authors who replied but did not provide details, collected resultsfrom TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 6 / 24
![Page 23: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/23.jpg)
Methodology
Data Collection
Verified status of published articles (author CVs, content search)
Collected any unpublished manuscripts available on the web
Contacted over 100 researchers where we could not find a publishedpaper to find out about the fate of their TESS project
For authors who replied but did not provide details, collected resultsfrom TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 6 / 24
![Page 24: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/24.jpg)
Methodology
Data Collection
Verified status of published articles (author CVs, content search)
Collected any unpublished manuscripts available on the web
Contacted over 100 researchers where we could not find a publishedpaper to find out about the fate of their TESS project
For authors who replied but did not provide details, collected resultsfrom TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 6 / 24
![Page 25: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/25.jpg)
Methodology
Data Collection
Verified status of published articles (author CVs, content search)
Collected any unpublished manuscripts available on the web
Contacted over 100 researchers where we could not find a publishedpaper to find out about the fate of their TESS project
For authors who replied but did not provide details, collected resultsfrom TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 6 / 24
![Page 26: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/26.jpg)
Methodology
Email to Researchers
Dear Professor [X],
Hope all is well! I am very interested in the TESS study that you ran in [year]called ”[title].”
We are in the process of conducting a meta-analysis of a set of survey experiments.I was wondering if the results were published anywhere, or if there is a working orconference paper available. If so, could you please send me the citation and/or acopy of the paper?
If there is no paper available, could you briefly summarize what you found in thesurvey experiment?
Thanks in advance for your time and consideration.
Sincerely,
Neil Malhotra, Stanford University
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 7 / 24
![Page 27: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/27.jpg)
Methodology
Identifying Strength of Results (Independent Variable)
Qualitative approach: Findings are as convincing as framed by thestudy investigators
Papers were independently coded as null, mixed, or strong by theauthors (around 90% agreement)
Coding based on reading abstracts, description of tables/figures, andconclusions
Studies not written up were coded based on email communicationswith PIs and/or description on TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 8 / 24
![Page 28: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/28.jpg)
Methodology
Identifying Strength of Results (Independent Variable)
Qualitative approach: Findings are as convincing as framed by thestudy investigators
Papers were independently coded as null, mixed, or strong by theauthors (around 90% agreement)
Coding based on reading abstracts, description of tables/figures, andconclusions
Studies not written up were coded based on email communicationswith PIs and/or description on TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 8 / 24
![Page 29: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/29.jpg)
Methodology
Identifying Strength of Results (Independent Variable)
Qualitative approach: Findings are as convincing as framed by thestudy investigators
Papers were independently coded as null, mixed, or strong by theauthors (around 90% agreement)
Coding based on reading abstracts, description of tables/figures, andconclusions
Studies not written up were coded based on email communicationswith PIs and/or description on TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 8 / 24
![Page 30: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/30.jpg)
Methodology
Identifying Strength of Results (Independent Variable)
Qualitative approach: Findings are as convincing as framed by thestudy investigators
Papers were independently coded as null, mixed, or strong by theauthors (around 90% agreement)
Coding based on reading abstracts, description of tables/figures, andconclusions
Studies not written up were coded based on email communicationswith PIs and/or description on TESS website
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 8 / 24
![Page 31: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/31.jpg)
Methodology
Example: Strong Result
“Despite the prominence of audience costs in international relations theories,it remains unclear whether and when audience costs exist in practice... Theresults [...] provide unambiguous evidence of audience costs.”
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 9 / 24
![Page 32: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/32.jpg)
Methodology
Example: Mixed Result
“This study investigates the impact of color and phenotypically black facialfeatures on candidate evaluation... Contrary to my expectations, there wasno main effect of candidate race or skin color on vote choice... While seem-ingly irrelevant to vote choice and perceived ideology, race and skin colorhad a large effect on white subjects’ perceptions of the political qualities ofthe opponent.”
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 10 / 24
![Page 33: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/33.jpg)
Methodology
Example: Null Result
“In this paper, I test the hypothesis that testimony can increase the persua-siveness of empirical claims... Regrettably, I show that describing statementsas being made in testimony typically has little effect on respondents’ factualbeliefs.”
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 11 / 24
![Page 34: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/34.jpg)
Methodology
Why We Didn’t Use a Quantitative Approach
One approach would be to analyze the data ourselves and codewhether results are strong, mixed, or null
Impossible to tell whether hypotheses listed on TESS website arepost-hoc (do not have original proposals)
Survey experiments are often complex with multiple possibletreatment group comparisons; hard to know what the estimand(s) ofinterest are
We did not have time to “fish”; we did not know how to optimally“fish” and were less motivated to do so
Takeaway point: What matters for publication is how authorsthemselves perceive/spin/frame their results
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 12 / 24
![Page 35: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/35.jpg)
Methodology
Why We Didn’t Use a Quantitative Approach
One approach would be to analyze the data ourselves and codewhether results are strong, mixed, or null
Impossible to tell whether hypotheses listed on TESS website arepost-hoc (do not have original proposals)
Survey experiments are often complex with multiple possibletreatment group comparisons; hard to know what the estimand(s) ofinterest are
We did not have time to “fish”; we did not know how to optimally“fish” and were less motivated to do so
Takeaway point: What matters for publication is how authorsthemselves perceive/spin/frame their results
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 12 / 24
![Page 36: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/36.jpg)
Methodology
Why We Didn’t Use a Quantitative Approach
One approach would be to analyze the data ourselves and codewhether results are strong, mixed, or null
Impossible to tell whether hypotheses listed on TESS website arepost-hoc (do not have original proposals)
Survey experiments are often complex with multiple possibletreatment group comparisons; hard to know what the estimand(s) ofinterest are
We did not have time to “fish”; we did not know how to optimally“fish” and were less motivated to do so
Takeaway point: What matters for publication is how authorsthemselves perceive/spin/frame their results
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 12 / 24
![Page 37: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/37.jpg)
Methodology
Why We Didn’t Use a Quantitative Approach
One approach would be to analyze the data ourselves and codewhether results are strong, mixed, or null
Impossible to tell whether hypotheses listed on TESS website arepost-hoc (do not have original proposals)
Survey experiments are often complex with multiple possibletreatment group comparisons; hard to know what the estimand(s) ofinterest are
We did not have time to “fish”; we did not know how to optimally“fish” and were less motivated to do so
Takeaway point: What matters for publication is how authorsthemselves perceive/spin/frame their results
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 12 / 24
![Page 38: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/38.jpg)
Methodology
Why We Didn’t Use a Quantitative Approach
One approach would be to analyze the data ourselves and codewhether results are strong, mixed, or null
Impossible to tell whether hypotheses listed on TESS website arepost-hoc (do not have original proposals)
Survey experiments are often complex with multiple possibletreatment group comparisons; hard to know what the estimand(s) ofinterest are
We did not have time to “fish”; we did not know how to optimally“fish” and were less motivated to do so
Takeaway point: What matters for publication is how authorsthemselves perceive/spin/frame their results
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 12 / 24
![Page 39: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/39.jpg)
Methodology
Coding Publication Status (Dependent Variable)
Among published studies, coded them as appearing in top-tier vs.non-top-tier journals
Drop book chapters and books
Distinguished unpublished studies as: (1) written but not published;(2) never written up
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 13 / 24
![Page 40: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/40.jpg)
Methodology
Coding Publication Status (Dependent Variable)
Among published studies, coded them as appearing in top-tier vs.non-top-tier journals
Drop book chapters and books
Distinguished unpublished studies as: (1) written but not published;(2) never written up
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 13 / 24
![Page 41: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/41.jpg)
Methodology
Coding Publication Status (Dependent Variable)
Among published studies, coded them as appearing in top-tier vs.non-top-tier journals
Drop book chapters and books
Distinguished unpublished studies as: (1) written but not published;(2) never written up
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 13 / 24
![Page 42: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/42.jpg)
Results
Sample by field and year
Year Commun. Econ. Pol. Sci. Pub. Health Psychology Sociology Other Total
2002 - - 1 - - - - 12003 - 1 4 - 6 2 1 142004 - 2 9 1 5 - - 172005 2 2 13 - 10 7 1 352006 3 1 12 1 9 6 - 322007 - - 5 - 3 2 - 102008 2 - 11 1 4 2 1 212009 - - 12 1 8 2 3 262010 3 3 22 - 5 6 2 412011 2 0 19 1 9 6 2 392012 1 1 5 1 1 3 1 13
Total 13 10 113 6 60 36 11 249
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 14 / 24
![Page 43: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/43.jpg)
Results
Sample (n=249)
Results Unwritten Unpublished Published Book chapter Missing Total
Null 31 7 10 1 0 49Mixed 10 32 40 3 1 86Strong 4 31 56 1 1 93Missing 6 1 0 2 12 21
Total 51 71 106 7 14 249
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 15 / 24
![Page 44: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/44.jpg)
Results
Final sample (n=221)
Results Unwritten Unpublished Published Book chapter Missing Total
Null 31 7 10Mixed 10 32 40Strong 4 31 56Missing
Total 45 70 106
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 16 / 24
![Page 45: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/45.jpg)
Results
Patterns of Publication Bias
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 17 / 24
![Page 46: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/46.jpg)
Results
Patterns of Publication Bias
Strong Mixed Null
Published (top-tier) 23.1% 11.0% 10.4%Published (non-top-tier) 38.5 37.8 10.4Written but not published 34.1 39.0 14.6Not written 4.4 12.2 64.6
χ2(6) = 80.3, p < .001
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 18 / 24
![Page 47: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/47.jpg)
Results
Robustness Checks
Robust to controls for researcher quality (h-index; number ofpublications), discipline, date the study was ran
No heterogeneity by researcher quality, discipline, date the study wasran
Sensitivity analysis: findings robust to even dramatic and unrealisticrates of misclassification due to self-reporting
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 19 / 24
![Page 48: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/48.jpg)
Results
Robustness Checks
Robust to controls for researcher quality (h-index; number ofpublications), discipline, date the study was ran
No heterogeneity by researcher quality, discipline, date the study wasran
Sensitivity analysis: findings robust to even dramatic and unrealisticrates of misclassification due to self-reporting
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 19 / 24
![Page 49: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/49.jpg)
Results
Robustness Checks
Robust to controls for researcher quality (h-index; number ofpublications), discipline, date the study was ran
No heterogeneity by researcher quality, discipline, date the study wasran
Sensitivity analysis: findings robust to even dramatic and unrealisticrates of misclassification due to self-reporting
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 19 / 24
![Page 50: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/50.jpg)
Conclusions
Summary
Clear relationship between results and publication status: publishedresults are not representative of even ex ante interesting projects
Mechanism: it seems that most null findings (around 2/3) never gotwritten up
Effect sizes are extremely large. Null findings increase probability ofpublication by 40 percentage points and paper writing by 60percentage points.
The patterns we uncovered suggest that type I error rates aresubstantially underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 20 / 24
![Page 51: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/51.jpg)
Conclusions
Summary
Clear relationship between results and publication status: publishedresults are not representative of even ex ante interesting projects
Mechanism: it seems that most null findings (around 2/3) never gotwritten up
Effect sizes are extremely large. Null findings increase probability ofpublication by 40 percentage points and paper writing by 60percentage points.
The patterns we uncovered suggest that type I error rates aresubstantially underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 20 / 24
![Page 52: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/52.jpg)
Conclusions
Summary
Clear relationship between results and publication status: publishedresults are not representative of even ex ante interesting projects
Mechanism: it seems that most null findings (around 2/3) never gotwritten up
Effect sizes are extremely large. Null findings increase probability ofpublication by 40 percentage points and paper writing by 60percentage points.
The patterns we uncovered suggest that type I error rates aresubstantially underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 20 / 24
![Page 53: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/53.jpg)
Conclusions
Summary
Clear relationship between results and publication status: publishedresults are not representative of even ex ante interesting projects
Mechanism: it seems that most null findings (around 2/3) never gotwritten up
Effect sizes are extremely large. Null findings increase probability ofpublication by 40 percentage points and paper writing by 60percentage points.
The patterns we uncovered suggest that type I error rates aresubstantially underestimated
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 20 / 24
![Page 54: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/54.jpg)
Conclusions
Implications
Proposed solutions need to tackle author selection stage: (1)two-stage review; (2) pre-analysis plans and pre-registration
Major value of pre-registration and pre-analysis is for scholarlycommunity to gain access to null results
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 21 / 24
![Page 55: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/55.jpg)
Conclusions
Implications
Proposed solutions need to tackle author selection stage: (1)two-stage review; (2) pre-analysis plans and pre-registration
Major value of pre-registration and pre-analysis is for scholarlycommunity to gain access to null results
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 21 / 24
![Page 56: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/56.jpg)
Conclusions
Patterns of Underreporting: Political Science
0 1 2 3 4 5 6 7
01
23
45
67
Questionnaire
Article
A) Number of experimental conditions
0 5 10 15 20
05
10
15
20
Questionnaire
Article
B) Number of outcome variables
0 5 10 15 20
05
10
15
20
Questionnaire
Article
C) Number of other variables
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 22 / 24
![Page 57: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/57.jpg)
Conclusions
Differences across questionnaires and articles
Design feature Mean SE 95% CI
Experimental conditions Q 3.06 0.18 [2.73,3.43]A 2.61 0.17 [2.29,2.94]
Q-A 0.45 0.12 [0.22,0.71]Outcomes Q 8.67 0.83 [7.08,10.35]
A 5.47 0.62 [4.31,6.73]Q-A 3.2 0.56 [2.18,4.35]
Other items Q 6.1 0.75 [4.69,7.61]A 2.39 0.4 [1.63,3.2]
Q-A 3.71 0.69 [2.45,5.14]
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 23 / 24
![Page 58: Publication Bias in the Social Sciences: Unlocking the File Drawer€¦ · Motivation Publication bias: greater likelihood of statistically signi cant results being published than](https://reader034.vdocument.in/reader034/viewer/2022051912/600294b9f6005e2bc87213ff/html5/thumbnails/58.jpg)
Conclusions
Patterns of Underreporting: Psychology
0 1 2 3 4 5
01
23
45
Questionnaire
Article
A) Number of experimental conditions
0 10 20 30 40
010
20
30
40
QuestionnaireArticle
B) Number of outcome variables
Franco, Malhotra & Simonovits (Stanford) Publication Bias in the Social Sciences December 2014 24 / 24