peer review
DESCRIPTION
This is a copy of a talk i gave at the Vision Sciences Society meeting in 2012. The talk presentas a review of the history and evolution of peer review and scientific publishing, an evaluation of its current function based on survey data, and a proposal for improving the system.TRANSCRIPT
![Page 1: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/1.jpg)
The state of scientific publishing and a new model
Dwight KravitzChris Baker
![Page 2: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/2.jpg)
Motivation: Improving Cost and Quality
Budget per Paper ($1000)N
IH B
ud
get
/ #
Pap
ers
Pu
blis
hed
![Page 3: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/3.jpg)
Peer Review: Outline
• History and evolution
• Modern purpose
• Improving peer review
• Post-publication
![Page 4: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/4.jpg)
History of Peer Review and Publishing
• Problems facing authors– Disseminating results– Claiming ownership
of results
• Publishing– Publishers had
distribution channels– Provides a record of
when results were published
![Page 5: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/5.jpg)
History of Peer Review and Publishing
• Problems facing publishers– Establishing veracity
of results– Deciding which results
to publish given limited space
Henry Oldenburg, 1665Medical Essays and Observations, 1731
• Peer review– Stamp of scientific
approval– Prioritizes papers for
inclusion by publishers
![Page 6: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/6.jpg)
![Page 7: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/7.jpg)
55% Reject42% Revise3% Accept
![Page 8: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/8.jpg)
55% Reject42% Revise3% Accept
![Page 9: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/9.jpg)
![Page 10: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/10.jpg)
Given Revision:98% Accept2% Reject
![Page 11: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/11.jpg)
![Page 12: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/12.jpg)
64% of papers are rejected at least
once
![Page 13: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/13.jpg)
![Page 14: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/14.jpg)
![Page 15: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/15.jpg)
![Page 16: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/16.jpg)
![Page 17: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/17.jpg)
Direct Costs of the Current System
• Total Number of Reviews: 6.3 (2-15)• Total Days under Review: 122 (21-
321)• Total Hours for Revisions: 68 (5-
300)• Total Days to Publish: 221 (21-
533)• Publication Fees: $2000 • Thousands of dollars and hundreds of hours
per paper• Long delays impair every form of assessment• Variance hinders planning research programs
and careers
![Page 18: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/18.jpg)
Peer Review: Outline
• History and evolution– Invention– Current form
• Modern purpose
• Improving peer review
• Post-publication
![Page 19: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/19.jpg)
Original Purpose
Henry Oldenburg, 1665Medical Essays and Observations, 1731
• Reasons for publishing– Disseminating results– Claiming ownership
of results• Reasons for peer review– Establishing veracity
of results– Deciding which results
to publish given limited space
![Page 20: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/20.jpg)
Modern Purpose• Reasons for
publishing– Disseminating results– Claiming ownership
of results• Reasons for peer review– Establishing veracity
of results– Deciding which results
to publish given limited space
Abstract: 9495Full Text: 8564PDF: 2549
![Page 21: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/21.jpg)
Pre-reception
• Peer review allows authors to test the paper before sending it to the whole field– Catches fundamental errors– Highlights important overlooked
issues
![Page 22: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/22.jpg)
Problems with Pre-reception
• Reviews are impure measures of quality
![Page 23: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/23.jpg)
Problems with Pre-reception
• Reviews are impure measures of quality
• Review process is opaque to the wider field
![Page 24: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/24.jpg)
Prioritization of the Literature
• Loop ranks papers based on which journal publishes them– Provides a way of deciding which papers to
read– Compensates for long publication lags
![Page 25: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/25.jpg)
Problems with Prioritization
• Reviews not focused solely on quality• Inefficient (avg. lag is 221 days)• Creates detrimental short-term
incentives
![Page 26: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/26.jpg)
Problems with Prioritization
http://pmretract.heroku.com
Budget per Paper ($1000)
![Page 27: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/27.jpg)
Problems with Prioritization
http://pmretract.heroku.com
Budget per Paper ($1000)Retractions per 100k Papers
![Page 28: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/28.jpg)
Problems with Prioritization
Fang & Casadevall, 2011
![Page 29: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/29.jpg)
Problems with Prioritization
• Reviews not focused on solely on quality
• Inefficient (avg. lag is 221 days)• Creates detrimental short-term
incentives• Doesn’t work
![Page 30: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/30.jpg)
Problems with Prioritization
![Page 31: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/31.jpg)
Problems with Prioritization
![Page 32: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/32.jpg)
Problems with Prioritization
![Page 33: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/33.jpg)
Problems with Prioritization
![Page 34: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/34.jpg)
Peer Review: Outline• History and evolution
– Invention– Current form
• Modern purpose– Pre-reception– Prioritization
• Improving peer review
• Post-publication
![Page 35: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/35.jpg)
Improving the current system• Make publishing more efficient and less
variable• Days to Publish: 221 (21-
533)• Years to First Cite: ~1.5 +
research
• Capture reviews in the prioritization• Total Number of Reviews: 6.3• Total Hours of Reviewing: ~31.5
• Focus reviews on science rather than publication
Decouple peer review from publishing
![Page 36: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/36.jpg)
![Page 37: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/37.jpg)
Proposed System
![Page 38: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/38.jpg)
Decoupling peer review from publishing
• Allows reviewers to focus on scientific merit rather than publication
• Allows reviews to be captured for prioritization and rewards for reviewers
• Makes publishing simpler and more efficient
• Eliminates the short-term incentive to produce least publishable units
![Page 39: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/39.jpg)
Peer Review: Outline• History and evolution
– Invention– Current form
• Modern purpose– Pre-reception– Prioritization
• Improving peer review– Decoupling from publishing
• Post-publication
![Page 40: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/40.jpg)
Format for papers
![Page 41: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/41.jpg)
The need for post-publication review
Field Altering
Useful Datapoint
Why?
Shrug
Pro
port
ion
of
field
![Page 42: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/42.jpg)
Field Altering
Useful Datapoint
Why?
Shrug
The need for post-publication review
We need more reviews to approximate these complex distributions.
Pro
port
ion
of
field Also enables
personalized prioritization of the literature.
![Page 43: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/43.jpg)
Incentivizing post-publication review
![Page 44: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/44.jpg)
Organizing the
literature
• The utility of modern peer review and publishing is prioritizing and organizing the literature.
![Page 45: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/45.jpg)
Organizing the
literature
• The utility of modern peer review and publishing is prioritizing and organizing the literature.
![Page 46: Peer review](https://reader035.vdocument.in/reader035/viewer/2022062319/5576c3b3d8b42ae3108b4a0f/html5/thumbnails/46.jpg)
Peer Review: Outline• History and evolution
– Invention– Current form
• Modern purpose– Pre-reception– Prioritization
• Improving peer review– Decoupling from publishing
• Post-publication review??????