5/14/2018

1

Melissa Blickem

• Senior Peer

Review Analyst,

American Chemical

Society

Margaret Donnelly

• Publisher, IOP Publishing

Lindsay Morton

• Contributor Experience Manager, Public Library of Science

SabinaAlam

• Editorial Director, F1000 Platforms

Moderator: Jonathan Schultz

• American Heart Association

Concurrent #3.4: Peer Review Debate#CSE2018

Peer Review Debate:Single-Blind vs Double-Blind vs

Open vs Post-Publication

Moderator:

Jonathan Schultz

American Heart Association

Blinded Peer Review

• Single-Blind:

– Reviewers know identity of authors

– Reviewer identity unknown to authors & public

• Double-Blind:

– Reviewers do not know identity of authors

– Reviewer identity unknown to authors & public

5/14/2018

2

Unblinded Peer Review

• Open/Transparent:

– Identity of authors & reviewers known throughout the review process

– Reviewer names & reviews typically published with articles

• Post-Publication:

– Articles posted with minimal or no review

– Review occurs in the open with names and reviews visible to authors & public

The Debate

• Goal: Determine which format is the “best” at

generating high-quality research articles in a

practical and sustainable way.

• Format: Brief overview of each format,

followed by Q&A/panel discussion

– Questions from the audience are encouraged

Our Debaters

• Single-Blind: Melissa Blickem, Senior Peer Review

Analyst, American Chemical Society

• Double-Blind: Margaret Donnelly, Publisher, IOP

Publishing

• Open/Transparent: Lindsay Morton, Contributor

Experience Manager, Public Library of Science

• Post-Publication: Sabina Alam, Editorial Director,

F1000 Platforms

5/14/2018

3

American Chemical Society

Single Blind Peer Review

Melissa BlickemLead Peer Review Analyst, Peer Review Services

Editorial Services, Global Journal DevelopmentACS Publications

Peer Review Debate: Double-blind vs Single vs Open vs Post

2018 Council of Science EditorsMay 7, 2018

Peer-Review Survey of Journal Authors and Reviewers

Main purpose - to investigate comments about perceived bias emerging from India trip

Sent to 15,741 total authors/reviewers

Total finished survey: 2,246 (14% response rate)

75% non-US based authors/reviewers, 66 countries represented

For research papers published in your field, to what extent do you agree that the following

types of peer review are/would be effective?

4.2% 4.7%

33.5% 33.5%39.6%

19.6%6.8% 8.8%

26.5% 26.5%24.1%

19.3%

11.7%

20.1%

21.7% 23.5%22.9%

33.0%

31.1%

30.0%

12.5% 11.7%9.8%

20.3%46.2%

36.4%

5.8% 4.9% 3.6%7.8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Single-blind peer

review

Double-blind peer

review

Open peer review Open & published

peer review

Post-publication open

commentary

Blind peer-review

supplemented with

post-publication open

commentary

Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree

5/14/2018

4

2.6% 4.7%

31.6%35.7%

38.5%

22.1%

3.0%5.1%

19.4%

21.6%19.2%

16.0%

6.5%

11.4%

22.7%

23.7%26.7%

29.6%

24.3%

29.5%

16.5%

12.7%10.7%

20.2%

63.6%

49.2%

9.9%6.4% 4.9%

12.1%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Single-blind peer

review

Double-blind peer

review

Open peer review Open & published

peer review

Post-publication open

commentary

Blind peer-review

supplemented with

post-publication open

commentary

Not Willing Somewhat Unwilling Neutral/Don't Know Somewhat Willing Very Willing

As a reviewer, to what extent are you/would you be willing to

participate in the following types of peer-review?

2.7% 3.9%

25.1%31.5%

37.3%

21.0%

4.4% 4.7%

15.4%

19.8%

20.3%

16.6%

8.2%12.1%

25.3%

25.1%

25.9%

30.4%

28.0%

30.2%

20.2%

15.7%

11.9%

21.7%

56.7%

49.0%

14.0%7.9%

4.7%10.3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Single-blind peer

review

Double-blind peer

review

Open peer review Open & published

peer review

Post-publication open

commentary

Blind peer-review

supplemented with

post-publication open

commentary

Not Willing Somewhat Unwilling Neutral/Don't Know Somewhat Willing Very Willing

As an author, to what extent are you/would you be willing to

participate in the following types of peer- review?

In Conclusion

Single-blind peer-review perceived as most effective by reviewers (77% agree)

• Double-blind peer-review also perceived as effective by reviewers (66% agree)

• Open peer-review and post-publication peer-review perceived as not effective.

5/14/2018

5

American Chemical Society

Single Blind Peer Review

Melissa BlickemLead Peer Review Analyst, Peer Review Services

Editorial Services, Global Journal DevelopmentACS Publications

Peer Review Debate: Double-blind vs Single vs Open vs

2018 Council of Science EditorsMay 7, 2018

Double-blind peer review

2018 CSE Annual Meeting

May 7, 2018

Margaret Donnelly

Publisher

Introduction

• Double-blind peer review– The identities of the authors are

blinded to the reviewers

– The identities of the reviewers are not

made public with the publication of

the paper

– IOP Publishing led a year-long pilot on

double-blind peer review on its

Express titles – Materials Research

Express (MRX) and Biomedical Physics

& Engineering Express – “inclusive”

international journals

5/14/2018

6

Double-blind peer review: strengths

• Reduce/eliminate bias in the peer review process– Studies have shown that blinded peer review reduces bias against certain

authors, e.g., female authors, junior authors, controversial authors, authors

from less prestigious institutions, etc. (1)

– Similarly, other studies have shown that referees who know the identities of the

authors are more likely to recommend acceptance for papers from famous

authors and authors from top universities and companies than referees who do

not know the identities or institutions of the authors (2)

Double-blind peer review: strengths

• Potential to improve referee acceptance rates– When the identities of authors are known to potential referees, a 2017 PNAS

study found that referees accept invitations to review 22% fewer papers than

referees who are blinded to the authors’ identities (2)

– In double-blind peer review, referees may be more inclined to agree to review

a paper based on personal interest rather than factors like the authors’

identities or institutions

– There is evidence that referees decline to review at higher rates for journals

with open peer review compared to journals using blinded peer review (3),

leading to longer turnaround times

Double-blinded peer review: strengths

• Referees have the freedom to provide

honest feedback– Referee anonymity ensures that reviewers can

provide critical (relevant, important) feedback

without fear of retaliation

– The critical feedback provided by double-blind

peer review leads to higher rejection rates, and

subsequently higher quality published articles

– Several studies have shown that double-blinded

referees recommend rejection at a higher rate

than referees whose identities are known to the

authors (4,5)Kowalczuk et al. BMJ Open 2015

5/14/2018

7

Summary

• Double-blind peer review is the best format of peer review:

– It reduces bias and is fairer than single-blind

– It encourages more objective, critical reviews, leading to higher

rejection rates and higher quality published articles

– It is the most popular system with referees, leading to faster

publication times

References1. Triggle CR, Triggle DJ. What is the future of peer review? Why is there fraud in science? Is plagiarism out of

control? Why do scientists do bad things? Is it a case of: “all that is necessary for the triumph over evil is that

good men do nothing”? Vasc Health Risk Manag 2007; 3:39-53.

2. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single- versus double-blind peer review. PNAS 2017;114:

12708-12713

3. Kowalczuk M, Samarasinghe M. Comparison of Acceptance of Peer Review Invitations by Peer Review Model:

Open, Single-blind, and Double-blind Peer Review. Presented at the International Congress on Peer Review and

Scientific Publication, September 10-12, 2017. https://peerreviewcongress.org/prc17-0227

4. Kowalczuk MK, Dudbridge F, Nanda S, et al. Retrospective analysis of the quality of reports by author-suggested

and non-author-suggested reviewers in journals operating on open or single-blind peer review models. BMJ

Open 2015;5:e008707

5. Vinther S, Nielsen OH, Rosenberg J, Keiding N, & Schroeder TV. Same review quality in open versus blinded peer

review in “ugeskrift for Laeger. Dan Med J 2012; 59:A4479

6. Snodgrass RT. Single- versus double-blind reviewing. ACM Trans Database Syst (TODS) 2007: 32:1-29

Transparent Peer Review

Lindsay Morton

Contributor Experience Manager, PLOS

lmorton@plos.org

May 2018

5/14/2018

8

What is Open Peer Review?

No universally accepted definition. For purposes of

discussion, we are referring to peer review in which:

⊨ Author, editor, and reviewer identities are known to all

participants throughout the peer review process

◇ Reviewer and editors names appear alongside the

published article

◇ Reviewer comments posted alongside the published article

Opt-in Approaches

Optional open peer review

scenarios

Reviewer name

disclosed to

authors

Reviewer name

published

Reviewer report

published

Author and reviewer both agree to open peer review

✘ ✘ ✘

Author agrees to open peer review but reviewer does not

✘

Reviewer agrees to open peer review but author does

not

✘

No one agrees to open peer review

Why Choose Open Peer Review?

1. Accountability

2. Quality

3. Reviewer credit

4. Enrich scientific record

5. Momentum and innovation

5/14/2018

9

1. Accountability

Each participant in the peer review process is accountable

for their own assessment and remarks

� Ethical benefits:

� Competing interests are more apparent

� Discourages malicious comments, plagiarism, and pursuit

of personal agenda

� Qualitative benefits:

� Encourages thoroughness, courtesy, and fairness

2. Quality

Evidence is not conclusive, but studies suggest that open

peer review feedback is as good as, or slightly better than,

closed models

� More constructive

� More in-depth

� Longer

� Attention to methods

� Substantiating evidence underlying reviewer comments

3. Reviewer credit

Reviewers are hardworking volunteers who want and deserve

acknowledgement. Posting peer reviews publicly enables:

• Enhanced presentation of reviewer activity for professional

records

• Opportunities for reviewers to demonstrate their expertise

• Recognizing contributions of students or colleagues

assisting with review

• Future elevation of review reports to citable scientific

outputs in their own right

5/14/2018

10

When reviews are open to the wider scientific community,

readers:

• Have access to more expert scientific opinions

• Have context for interpreting results

• Have model reviews they can emulate

• Can easily see whether their own questions were

previously addressed in review

4. Enrich scientific record

Open peer review is gaining ground among stakeholders

across the scientific publishing landscape:

• Many journals already mandate or offer an Open Peer

Review option

• Authors like signed peer review

• Reviewers are open to signed and posted reviews

5. Momentum and innovation

Open Invited Peer Review,

After Publication

Sabina Alam

Editorial Director, F1000 Platforms

5/14/2018

11

• Speed of publication

• Transparent (peer review and

data sharing)

• Credit for reviewers (DOI for

reports and ORCID IDs)

THE F1000RESEARCH PUBLISHING AND PEER REVIEW PROCESS

Scope

Language

Reporting guidelines

Data availability

Ethics

Reviewer suitability

(competing interests,

expertise, etc)

Methods & analysisStrength of conclusionsScientific validity

~ 7 days

DOI

DOI

TRANSPARENT REFEREEING AND REVIEW STATUS

https://f1000research.com/articles/6-1698/v3

Indexed once it passes peer review:

or

5/14/2018

12

http://f1000research.com/articles/4-

121

TRANSPARENT REFEREEING AND DISCUSSION

Referees:

� Get credit for contributing to discussion

�Focus on helping authors improve their work

�Their reports provide new form of expert

article-based assessment

Open invited peer review, after publication

The advantage of the open refereeing process becomes completely obvious once you

have experienced it. When referees comment openly on a paper that is already

published, they suddenly become much more constructive and helpful.

Etienne Joly

IPBS, CNRS, Université de Toulouse, France

The open review process pushes authors to submit an

accurately prepared manuscript and reviewers to

provide constructive comments.

Gianmarco Contino

MRC Cancer Unit, University of Cambridge, UK

In my opinion, this publication model represents the future of scientific

publishing. Review is completely transparent to all involved, tyrannical

reviewers are avoided, authors drive the process, it is open access and the

speed is amazing.

Eleftherios Diamandis

Mount Sinai Hospital, Canada

Panel Discussion

• Single-Blind: Melissa Blickem, Senior Peer Review

Analyst, American Chemical Society

• Double-Blind: Margaret Donnelly, Publisher, IOP

Publishing

• Open/Transparent: Lindsay Morton, Contributor

Experience Manager, Public Library of Science

• Post-Publication: Sabina Alam, Editorial Director,

F1000 Platforms