![Page 1: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/1.jpg)
Challenges for investigators –
generating reproducible research results
University of Minnesota, 3/2018
John P.A. Ioannidis, MD, DSc
C.F. Rehnborg Chair in Disease Prevention
Professor of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics
Stanford University
Co-Director, Meta-Research Innovation Center at Stanford (METRICS)
![Page 2: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/2.jpg)
Goodman, Fanelli, Ioannidis. Science Translational Medicine 2016
![Page 3: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/3.jpg)
Different types of reproducibility
• Reproducibility of methods: the ability to understand or repeat as
exactly as possible the experimental and computational procedures.
• Reproducibility of results: the ability to produce corroborating
results in a new study, having followed the same experimental
methods.
• Reproducibility of inferences: the making of knowledge claims of
similar strength from a study replication.
![Page 4: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/4.jpg)
![Page 5: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/5.jpg)
![Page 6: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/6.jpg)
![Page 7: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/7.jpg)
Scientific discovery has become a boring nuisance: 96% of
the biomedical literature claims significant results (and the
vast majority of them say they are novel)
Chavalarias, Wallach, Li, Ioannidis, JAMA 2016
![Page 8: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/8.jpg)
Typical recipe of research practices
• Solo, siloed investigator
• Small sample size studies
• Cherry-picking of one/best hypothesis
• Post-hoc
• P<0.05 is enough
• No registration
• No data sharing
• No replication
![Page 9: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/9.jpg)
fMRI studies: only very small
studies discover a lot of foci
David et al, PLoS ONE 2013, Ioannidis et al Trends in Cognitive Sciences 2014
![Page 10: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/10.jpg)
![Page 11: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/11.jpg)
Szucs and Ioannidis, PLoS Biology 2017
![Page 12: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/12.jpg)
Empirical studies on fields where
replication practices are common suggest
that most of the initially claimed
statistically significant effects in
observational associations are false
positives or substantially exaggerated
![Page 13: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/13.jpg)
Candidate genes replicated through GWAS:
replication rate = 1.2%
Ioannidis, Tarone, McLaughlin, Epidemiology 2011
![Page 14: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/14.jpg)
Prinz et al., Nature Reviews Drug Discovery 2011
Failed replication in preclinical
research
![Page 15: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/15.jpg)
Replicated: only 6 of 53 landmark studies for
Amgen oncology drug target projects
• “The failure to win “the war on cancer”
has been blamed on many factors, … But
recently a new culprit has emerged: too
many basic scientific discoveries… are
wrong.”
Begley et al. Nature 2012
![Page 16: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/16.jpg)
Begley and Ioannidis, Circulation Research 2015
![Page 17: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/17.jpg)
Begley and Ioannidis, Circulation Research 2015
![Page 18: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/18.jpg)
![Page 19: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/19.jpg)
![Page 20: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/20.jpg)
Potential solutions to improve
reproducibility
• Some solutions have already worked in specific fields and may need to be considered in other fields as well
• Other solutions are more speculative
• Empirical evidence as to their efficacy is needed
• Seemingly effective solutions may also have collateral damages
• Do no harm
![Page 21: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/21.jpg)
Ioannidis, PLoS Medicine 2014
![Page 22: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/22.jpg)
Large-scale collaboration and adoption of
replication culture
![Page 23: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/23.jpg)
Replication – by whom?
• Same investigators
• Different investigators of the same school
• Different investigators of competing theories/hypotheses
• Combinations of the above
• Open to the wide public
![Page 24: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/24.jpg)
Levels of registration
• Level 0: no registration
• Level 1: registration of dataset
• Level 2: registration of protocol
• Level 3: registration of analysis plan
• Level 4: registration of analysis plan and raw data
• Level 5: open live streaming
![Page 25: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/25.jpg)
![Page 26: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/26.jpg)
![Page 27: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/27.jpg)
Sharing
data –
who,
when,
and how?
Doshi, Goodman,
Ioannidis, TiPS
2013
![Page 28: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/28.jpg)
![Page 29: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/29.jpg)
Science, December 2, 2016
![Page 30: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/30.jpg)
Begley and Ioannidis, Circulation Research 2015
![Page 31: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/31.jpg)
Modeling and modeling plus
experimentation
![Page 32: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/32.jpg)
Grimes
et al,
JRSOS
2018
![Page 33: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/33.jpg)
Re-engineering the reward system
Ioannidis and
Khoury, JAMA 2014
![Page 34: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/34.jpg)
![Page 35: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/35.jpg)
![Page 36: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/36.jpg)
![Page 37: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/37.jpg)
Understand and align interests of stakeholders
![Page 38: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/38.jpg)
Concluding comments
• Reproducibility has different meanings across
different scientific disciplines but eventually all of
them aim at enhancing our trust in scientific findings
• There are many possible interventions that may
improve the efficiency of research practices and the
reproducibility of the scientific literature
• Empirical meta-research would be useful not only to
assess the prevalence of problems of reproducibility,
but also to assess the effectiveness and potential
harms of interventions that try to make research
more reproducible.
![Page 39: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/39.jpg)
Daniele Fanelli
Steve Goodman
Shanil Ebrahim
Despina
Contopoulos-
Ioannidis
Georgia Salanti
Chirag Patel
Lars Hemkens
Ann Hsing
Lamberto Manzoli
Maria Elena Flacco
George Siontis
Denes Szucs
Kostas Siontis
Vangelis Evangelou
Kristin Sainani
Muin Khoury
Orestis Panagiotou
Florence Bourgeois
Special thanks
![Page 40: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/40.jpg)
Joseph LauMalcolm MacLeodMarcus MunafoDavid AllisonJosh WallachFotini KarassaAthina TatsioniEvi NtzaniIoanna TzoulakiDemos KatritsisNikos PatsopoulosFainia KavvouraBrian NosekVictoria StoddenΕle ZegginiBelinda BurfordKostas TsilidisJodi Prochaska
Special thanks
![Page 41: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/41.jpg)
Charitini StavropoulouEvropi Theodoratou Nikos PandisHuseyin NaciVanesa BellouΑntony DoufasLazaros BelbasisChris DoucouliagosStelios SerghiouAnna ChaimaniFotini ChatzinasiouStephania PapatheodorouFlorian NaudetTom HardwickePerrine JaniaudIoana-Alina CristeaShannon BrownleeVikas SainiMatthias EggerPatrick BossuytAndre UitterlindenDoug AltmanDeb ZarinKatherine Flegal
Special thanks
![Page 42: Challenges for investigators generating reproducible research results · 2018. 3. 12. · Challenges for investigators – generating reproducible research results University of Minnesota,](https://reader035.vdocument.in/reader035/viewer/2022071418/61167ce77b224e6e2e64f10b/html5/thumbnails/42.jpg)
Shanthi Kapaggoda
Ewoud Schuit
Stefania Boccia
David Chavalarias
Jennifer Ware
Viswam Nair
Stephan Bruns
Dorothy Bishop
Tom Trikalinos
Kristina Sundquist
Johanna Int’hout
Kevin Boyack
Brett Thombs
Raj Manrai
Nazmus Saquib
Elizabeth Iorns
Abraham Verghese
Euan Ashley
Special thanks