irreproducible results - portal · results of reproducibility study for 508 papers 402 9...

14
Coffee & Bit(e)s THE COFFEE LECTURES FOR SCIENTISTS UNIVERSITY LIBRARY BERN Aline Frank, Nuria Plattner, Michael Horn, and Silvan Christen www.unibe.ch/ub/sciencelibrary Dr. Nuria Plattner Irreproducible results in scientific articles Can we trust published literature?

Upload: others

Post on 28-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

Coffee & Bit(e)sTHE COFFEE LECTURES FOR SCIENTISTS

UNIVERSITY LIBRARY BERNAline Frank, Nuria Plattner, Michael Horn, and Silvan Christenwww.unibe.ch/ub/sciencelibrary

Dr. Nuria Plattner

Irreproducible results in

scientific articlesCan we trust published literature?

Page 2: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

2

Scientific Method based on reproducibilityImportance of reproducibility

• A scientific fact can only be established if a reproducibleprocedure to verify it can be defined

• Research topic: challenges for defining reproduciblemethods and documentation differ for different areas

• The reproducibility problem exists in all areas

• Established vs. new topics: the more established aresearch area, the better defined and standardized itsprocedures

https://3c1703fe8d.site.internapcdn.net/newman/gfx/news/hires/2016/howscientist.png

Page 3: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

3

Recent concerns about reproducibilityIs there a reproducibility crisis?

M. Baker, Nature 533, 452, (2017)

• Nature survey, 1576researchers

• 52% agree that thereis a significant crisis inreproducibility

• <31% think that non-reproducible resultsare wrong

• Most scientist still trustpublished literature

Page 4: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

4

Trends in paper retractionCrisis Indicators

A) Retracted articles for specific causes by retraction year

B) Percentage of articles retracted for fraud or suspected fraud by publication year

• Only papers with significantflaws are retracted

• Minor corrections usually addedto paper as erratum (if anything)

• Main causes for retraction: fraudor scientific mistake (conceptual)

F. C. Fang, R. G. Steen, and A. Casadevall, Proc. Natl. Accad. Sci. 109, 17028-17033, (2012)

Page 5: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

5

Scientific result

Publication process and lifecycleHow are irreproducible results detected?

ManuscriptPeer review

Experimental results/observations: only consistency check

Mathematical proofs: full check

Reproductionattempt of result

Nobody interested

Check forown mistakescontact authorCorrection/

Retraction

Prepare publicationabout new finding

?

Scientific publication

Page 6: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

6

Statistical facts about scientific publicationsReasons for reproducibility problems

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3999612/

• Scientists spend time publishing as much aspossible, rather than spending time developingresearch

• No time for confirmation of experiments

• Limited time for peer-review

• Only 45% of articles published in top 4500scientific journals are cited within first 5 years

• Only 42% of the papers receive more than onecitation, 5-25% of these are self-citation

Page 7: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

7

Selective reporting of resultsExamples for lack of reproducibility

• Selective reporting can take place in various ways

• There can be a fine line between selective reporting and scientific fraud

Different forms of selective reporting[1]

[1] Timothy H. Parker et al., Trends Ecol. Evol., 31, 711-719, (2016)

Examples of selective reporting[1]

Page 8: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

8

Missing documentation/codeExamples for lack of reproducibility

• Sharing for repeatability is essential toensure that other researchers canevaluate results based on accurate andcomplete evidence

• Repeatability study 2015 examinsextent to which computer systemsresearchers share their source code andcode builds

• This is referred to as weak repeatability

http://reproducibility.cs.arizona.edu/

Results of reproducibility study for 508 papers

402

Page 9: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

9

Experiments with limited reproducibilityExamples for lack of reproducibility

• Success rates for new development projects in Phase II trials have fallen from 28% to 18%in recent years

• Limitations ofpredictivity

• Validity of targets

F. Prinz, T. Schlange and K. Asadullah, Nat. Rev. Drug Discovery 10, 712, (2011)

Bayer study: reproducibility of preclinical trials

c) relationship of publisheddata to in-house data

a) characterization oftargets

b) approaches used toreprocuded data

Page 10: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

10

Scientific misconductExamples for lack of reproducibility

• Nr. 1) Yoshitaka Fujii, anesthesiology researcher, fabricated clinical study data in at least 169 papers

• 1993: falsified data publishing started• 2000: first publication suspecting fraud• 2012: full investigation

• Nr. 10) Jan Hendrik Schön, physicist at Bell labs; high-impact papers retracted after failure toreproduce superconductivity in organic materials

• 2000: first fraudulent result published• 2001: first suspicion of fraud• 2002: report of scientific misconduct released

https://www.sciencemag.org/news/2018/10/what-massive-database-retracted-papers-reveals-about-science-publishing-s-death-penalty

Medicine

Psychology

Computer Science

Medicine

Medicine

Chemistry

Physics

Biology

Accounting (Economy)

Medicinal Chemistry

Page 11: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

11

Retraction watch databaseHow to keep track of paper retractions?

https://retractionwatch.com

Page 12: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

12

Retraction watch databaseHow to keep track of paper retractions?

https://retractionwatch.com

Page 13: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

13

Possible solutionsHow to improve reproducibility?

• Important: detailed description ofresults and methods

• sharing of data and code, openscience

• More fundamental changes: structuralincentives need to be changed

BORIS Repository

https://boris.unibe.ch/

Page 14: Irreproducible results - Portal · Results of reproducibility study for 508 papers 402 9 Experiments with limited reproducibility Examples for lack of reproducibility • Success

Coffee & Bit(e)sTHE COFFEE LECTURES FOR SCIENTISTS

UNIVERSITY LIBRARY BERNAline Frank, Nuria Plattner, Michael Horn, and Silvan Christenwww.unibe.ch/ub/sciencelibrary

Questions?Thanks for your attention