the current state of peer review: criticisms, challenges ... · the current state of peer review:...

The current state of peer review:

criticisms, challenges and innovations

Irene Hames, PhD, FSB Editorial and Publishing Consultant

Council Member, COPE (Committee on Publication Ethics)

Co-Editor-in-Chief, Ethical Editing

ABEC meeting, Brazil, 13 November 2012

Irene Hames, ABEC Brazil, November 2012 2

What is (editorial) peer review?

… the process by which research output is

subjected to scrutiny and critical assessment by

individuals who are experts in those areas

(Hames, 2012, in Academic and Professional Publishing, Chandos

Publishing, Eds Campbell, Pentz and Borthwick, p.16)

and …

…the critical assessment of manuscripts

submitted to journals by experts who are not part

of the editorial staff

(ICMJE, International Committee of Medical Journal Editors,

http://www.icmje.org/)

Irene Hames, ABEC Brazil, November 2012 3

.

*Good practice and quality in peer review is

system and business-model independent*

Irene Hames, ABEC Brazil, November 2012 4

.

Open access has presented opportunities for abuse and

unscrupulous business ventures

Phil Davis (The Scholarly Kitchen blog, 10 June 2009) –

„nonsense‟ manuscript accepted after „peer review‟

Jeffrey Beall‟s list of Predatory OA Publishers

http://metadata.posterous.com/83235355

Irene Hames, ABEC Brazil, November 2012 5

Common misconception that the larger and higher

impact a journal the better the quality of peer review

… many small specialist journals operate rigorous peer

review, have dedicated and knowledgeable editors and

committed editorial teams

Irene Hames, ABEC Brazil, November 2012 6

Bigger is not better

What do people think of peer review?

Editors … value it

Surveys of research community (Ware and

Monkman, 2008; Sense About Science, 2009)

85% & 82% - peer review greatly helps scientific

communication

83% & 84% - without peer review would be no control

in scientific communication

accuracy and quality of work not peer reviewed cannot

be trusted

89% & 91% felt own last accepted paper improved by

peer review

Irene Hames, ABEC Brazil, November 2012 7

“Peer review in scholarly publishing, in one form

or another, is crucial to the reputation and

reliability of scientific research” (Para 277)

Irene Hames, ABEC Brazil, November 2012 8

… but is some dissatisfaction

12% (Ware & Monkman) and 9% (SAS) in the

two surveys

Only about a third in both surveys think current

system of peer review is best that can be

achieved

Researchers want to improve peer review, not

replace it

Irene Hames, ABEC Brazil, November 2012 9

Criticisms of peer review

Irene Hames, ABEC Brazil, November 2012 10

“Peer review is in

crisis”

“Publish all, filter

later”

Unreliable and unfair

No clear standards, idiosyncratic

Open to abuse and bias

Stifles innovation

Slow, causes delays in publication

Poor at detecting errors

Almost useless at detecting fraud and misconduct

Expensive and labour intensive

Reviewers overloaded, working „for free‟

Can ‘fail’ in even the best-run journals [Image, Gideon Burton, Utah, USA (CC BY-SA 2.0)]

Critical role of the Editor

“…[peer review] works as well as can be expected. The critical feature

that makes the system work is the skill and insight of the editor. Astute

editors can use the system well, the less able who follow reviewer

comments uncritically bring the system into disrepute.”

(a respondent, Ware & Monkman, 2008, PRC peer review survey)

“Unfortunately, all too often editors relinquish their responsibilities and

treat the peer review process as a vote … the real problem is editors …

increasingly, one sees editors who don’t use any judgement at all, but

just keep going back to reviewers until there is agreement.”

(Dorothy Bishop, Professor of Developmental Neuropsychology, Oxford University, „In

defence of peer review‟, comment 4 Jan 2011, to R Smith (2010) Breast Cancer

Research, 12(Suppl 4):S13 )

Irene Hames, ABEC Brazil, November 2012 11

Editors have to act as editors

Being an editor is:

not just moving manuscripts automatically

through the peer-review process

not just „counting votes‟

not passing on editor responsibilities to

reviewers

making critical judgements („reviewers advise,

editors decide‟)

Irene Hames, ABEC Brazil, November 2012 12

Some problems due to

Variable quality of peer review

Lack of training for new editors

Inconsistency in decision making

Perceived gaming by journals

Irene Hames, ABEC Brazil, November 2012 13

Basic checks must be done

Three recent cases of „fake reviewers‟

For „suggested reviewers‟, authors provided:

false identities (and emails), which were them or colleagues

names of real people but created email accounts for them which they

or associates had access to

Reviews were done very quickly and were positive

„The peer-review process for the above article was found to have

been compromised and inappropriately influenced by the

corresponding author‟

http://retractionwatch.wordpress.com/, „faked emails‟ category

Involves different disciplines, different countries and different

publishers … and often many published papers …

Irene Hames, ABEC Brazil, November 2012 14

Irene Hames, ABEC Brazil, November 2012 15

A cause for concern

“… and underlying these worries was yet another: that

scientific articles have been hijacked away from their

primary role of communicating scientific discovery to one

of demonstrating academic activity.”

Stephen Lock, „A Difficult Balance. Editorial peer review in

medicine‟, Introduction to third impression, BMJ,1991, p.xi.

Irene Hames, ABEC Brazil, November 2012 16

Increasing pressures & challenges

On researchers

on their time

to publish more

to publish in high-impact journals

On journals and editors

increasing submissions

preliminary work

more rebuttals?

language and quality issues

new tools – plagiarism, inappropriate digital image

manipulation

Irene Hames, ABEC Brazil, November 2012 17

A crisis in reviewing?

Fox and Petchey*: „peer review system is breaking down

and will soon be in crisis‟

„Tragedy of the reviewer commons‟ – individuals exploit

system by submitting manuscripts but little incentive to

review manuscripts from others

*(2010) Bulletin of the Ecological Society of America, 91: 325-33 p.325

Irene Hames, ABEC Brazil, November 2012 18

Current geographical imbalance between who submitting and who reviewing

USA producing ~20% of papers globally, doing ~32% of

reviews*

China producing ~12-15% of papers , doing ~4-5% of

reviews*

Name ambiguity problem – ORCID launched last month

(Open Researcher and Contributor ID,

http://about.orcid.org/ )

*Elsevier (2011a) Evidence given by Mayur Amin to the UK House of Commons Science and

Technology Inquiry into Peer Review, 11 May 2001. Transcript of oral evidence, HC 856, Q127.

http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf

Irene Hames, ABEC Brazil, November 2012 19

„Wastage‟ of reviews?

Rejected manuscripts can go from journal to

journal, fresh reviews at each

„Cascading‟ submissions and reviews

Between publishers: Neuroscience Peer Review

Consortium (http://nprc.incf.org/)

Within publishers and societies

Irene Hames, ABEC Brazil, November 2012 20

.

Innovations in peer review

Irene Hames, ABEC Brazil, November 2012 21

Two functions of peer review separated

PLOS ONE launched 2006, published ~14,000 articles in 2011

(~1.5% of world‟s scientific literature) using >38,400 reviewers

Publication based on „soundness‟ - research methodology, results

and reporting - not novelty, interest or potential impact

Evaluation of interest/impact left for post-publication

Impact Factor 4.411

Open access, „repository‟ type journals - „PLoS ONE clones‟ – being

launched (BMJ Open, Sage Open, Scientific Reports, Biology Open,

AIP Advances, SpringerPLus)

22 Irene Hames, ABEC Brazil, November 2012

More transparent approaches

Publishing reviews, ms versions and editorial

correspondence

BMC series medical journals – „pre-publication history‟

EMBO Journal – „peer review process file‟

BMJ Open – „peer review history‟

Reviewers‟ names may or may not be revealed

„Cross-peer review‟ – EMBO Journal

Open peer review

Irene Hames, ABEC Brazil, November 2012 23

„Open review‟ can mean a number of things

Irene Hames, ABEC Brazil, November 2012 24

Authors’ names

known

Authors’ names

not known

Reviewers

’ names

known

„Open‟ review

Term „open‟ may also include:

(i) Reviewers‟ names being disclosed

for published articles

(ii) Reviewers‟ reports (with or without

names) being included with published

articles

(iii) Editorial correspondence and/or

all versions of the manuscript being

included with published articles

(iv) Community/public being able to

comment during review

(v) Combinations of the above

(unlikely that this system is in

operation anywhere)

Reviewers

’ names

not known

„Single-blind‟ review

The most common form in scientific,

technical and medical (STM) journals

„Double-blind‟ review

The most common form in the

humanities and social sciences

(HSS)

Many new initiatives

In past year:

Peerage of Science (Winner of ALPSP Award

for Publishing Innovation 2012)

http://www.peerageofscience.org/

PeerJ https://peerj.com/

F1000 Research http://f1000research.com/

Rubriq http://www.rubriq.com/

Irene Hames, ABEC Brazil, November 2012 25

Peer review doesn‟t stop at publication

When real peer review starts?

Post-publication review and evaluation

Increasing opportunities for innovation

Challenges and problems

Increasing importance of blogs, twitter and

other social media

Irene Hames, ABEC Brazil, November 2012 26

.

#arseniclife

Carl Zimmer

http://blogs.discovermagazine.com/loom/2012/10/03/weirdly-unweird-a-

better-end-to-the-arseniclife-affair/

Irene Hames, ABEC Brazil, November 2012 27

.

„Online scientific interaction outside the traditional

journal space is becoming more and more

important to academic communication‟

Mark Hahnel, founder, FigShare (http://figshare.com/)

Irene Hames, ABEC Brazil, November 2012 28

A big challenge - data

Massive amounts being generated

Recognition for producing, making usable by

others and curating

Where to put?

Dryad http://datadryad.org/ - international

repository of data underlying peer-reviewed

articles in basic and applied sciences; can be

made securely available for peer review

Irene Hames, ABEC Brazil, November 2012 29

Wiley-Blackwell Copyright 2007 30

A reviewer‟s point of view

„Reviewers peering from under a pile of „omics‟ data‟ J.K.

Nicholson, Nature (2006), 440, 992

“The scientific community needs to reassess the

way it addresses the peer-review problem,

taking into account that referees are only human

and are now being asked to do a superhuman

task on a near-daily basis.”

Peer review ….

Mark Ware: „far from being in crisis, peer review remains

widely supported and diversely innovative‟

Fiona Godlee: (BMJ Editor): „At its best I think we would

all agree that it does improve the quality of scientific

reporting and that it can improve, through the pressure of

the journal, the quality of the science itself and how it is

performed.‟

Ware M (2011) New Review of Information Networking, 16(1): 23-53

Godlee F (2011) Evidence given to the UK House of Commons Science and Technology Inquiry into

Peer Review, 11 May 2001. Transcript of oral evidence, HC856, Q97.

http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf

Irene Hames, ABEC Brazil, November 2012 31

25% discount on all Wiley books

Code ABEC, valid until 31 May 2013

Irene Hames, ABEC Brazil, November 2012 32

Irene Hames, ABEC Brazil, November 2012 33

Thank you!

Dr Irene Hames

email: irene.hames@gmail.com

twitter: @irenehames