the current state of peer review: criticisms, challenges ... · the current state of peer review:...
TRANSCRIPT
The current state of peer review:
criticisms, challenges and innovations
Irene Hames, PhD, FSB Editorial and Publishing Consultant
Council Member, COPE (Committee on Publication Ethics)
Co-Editor-in-Chief, Ethical Editing
ABEC meeting, Brazil, 13 November 2012
Irene Hames, ABEC Brazil, November 2012 2
What is (editorial) peer review?
… the process by which research output is
subjected to scrutiny and critical assessment by
individuals who are experts in those areas
(Hames, 2012, in Academic and Professional Publishing, Chandos
Publishing, Eds Campbell, Pentz and Borthwick, p.16)
and …
…the critical assessment of manuscripts
submitted to journals by experts who are not part
of the editorial staff
(ICMJE, International Committee of Medical Journal Editors,
http://www.icmje.org/)
Irene Hames, ABEC Brazil, November 2012 3
.
*Good practice and quality in peer review is
system and business-model independent*
Irene Hames, ABEC Brazil, November 2012 4
.
Open access has presented opportunities for abuse and
unscrupulous business ventures
Phil Davis (The Scholarly Kitchen blog, 10 June 2009) –
„nonsense‟ manuscript accepted after „peer review‟
Jeffrey Beall‟s list of Predatory OA Publishers
http://metadata.posterous.com/83235355
Irene Hames, ABEC Brazil, November 2012 5
Common misconception that the larger and higher
impact a journal the better the quality of peer review
… many small specialist journals operate rigorous peer
review, have dedicated and knowledgeable editors and
committed editorial teams
Irene Hames, ABEC Brazil, November 2012 6
Bigger is not better
What do people think of peer review?
Editors … value it
Surveys of research community (Ware and
Monkman, 2008; Sense About Science, 2009)
85% & 82% - peer review greatly helps scientific
communication
83% & 84% - without peer review would be no control
in scientific communication
accuracy and quality of work not peer reviewed cannot
be trusted
89% & 91% felt own last accepted paper improved by
peer review
Irene Hames, ABEC Brazil, November 2012 7
“Peer review in scholarly publishing, in one form
or another, is crucial to the reputation and
reliability of scientific research” (Para 277)
Irene Hames, ABEC Brazil, November 2012 8
… but is some dissatisfaction
12% (Ware & Monkman) and 9% (SAS) in the
two surveys
Only about a third in both surveys think current
system of peer review is best that can be
achieved
Researchers want to improve peer review, not
replace it
Irene Hames, ABEC Brazil, November 2012 9
Criticisms of peer review
Irene Hames, ABEC Brazil, November 2012 10
“Peer review is in
crisis”
“Publish all, filter
later”
Unreliable and unfair
No clear standards, idiosyncratic
Open to abuse and bias
Stifles innovation
Slow, causes delays in publication
Poor at detecting errors
Almost useless at detecting fraud and misconduct
Expensive and labour intensive
Reviewers overloaded, working „for free‟
Can ‘fail’ in even the best-run journals [Image, Gideon Burton, Utah, USA (CC BY-SA 2.0)]
Critical role of the Editor
“…[peer review] works as well as can be expected. The critical feature
that makes the system work is the skill and insight of the editor. Astute
editors can use the system well, the less able who follow reviewer
comments uncritically bring the system into disrepute.”
(a respondent, Ware & Monkman, 2008, PRC peer review survey)
“Unfortunately, all too often editors relinquish their responsibilities and
treat the peer review process as a vote … the real problem is editors …
increasingly, one sees editors who don’t use any judgement at all, but
just keep going back to reviewers until there is agreement.”
(Dorothy Bishop, Professor of Developmental Neuropsychology, Oxford University, „In
defence of peer review‟, comment 4 Jan 2011, to R Smith (2010) Breast Cancer
Research, 12(Suppl 4):S13 )
Irene Hames, ABEC Brazil, November 2012 11
Editors have to act as editors
Being an editor is:
not just moving manuscripts automatically
through the peer-review process
not just „counting votes‟
not passing on editor responsibilities to
reviewers
making critical judgements („reviewers advise,
editors decide‟)
Irene Hames, ABEC Brazil, November 2012 12
Some problems due to
Variable quality of peer review
Lack of training for new editors
Inconsistency in decision making
Perceived gaming by journals
Irene Hames, ABEC Brazil, November 2012 13
Basic checks must be done
Three recent cases of „fake reviewers‟
For „suggested reviewers‟, authors provided:
false identities (and emails), which were them or colleagues
names of real people but created email accounts for them which they
or associates had access to
Reviews were done very quickly and were positive
„The peer-review process for the above article was found to have
been compromised and inappropriately influenced by the
corresponding author‟
http://retractionwatch.wordpress.com/, „faked emails‟ category
Involves different disciplines, different countries and different
publishers … and often many published papers …
Irene Hames, ABEC Brazil, November 2012 14
Irene Hames, ABEC Brazil, November 2012 15
A cause for concern
“… and underlying these worries was yet another: that
scientific articles have been hijacked away from their
primary role of communicating scientific discovery to one
of demonstrating academic activity.”
Stephen Lock, „A Difficult Balance. Editorial peer review in
medicine‟, Introduction to third impression, BMJ,1991, p.xi.
Irene Hames, ABEC Brazil, November 2012 16
Increasing pressures & challenges
On researchers
on their time
to publish more
to publish in high-impact journals
On journals and editors
increasing submissions
preliminary work
more rebuttals?
language and quality issues
new tools – plagiarism, inappropriate digital image
manipulation
Irene Hames, ABEC Brazil, November 2012 17
A crisis in reviewing?
Fox and Petchey*: „peer review system is breaking down
and will soon be in crisis‟
„Tragedy of the reviewer commons‟ – individuals exploit
system by submitting manuscripts but little incentive to
review manuscripts from others
*(2010) Bulletin of the Ecological Society of America, 91: 325-33 p.325
Irene Hames, ABEC Brazil, November 2012 18
Current geographical imbalance between who submitting and who reviewing
USA producing ~20% of papers globally, doing ~32% of
reviews*
China producing ~12-15% of papers , doing ~4-5% of
reviews*
Name ambiguity problem – ORCID launched last month
(Open Researcher and Contributor ID,
http://about.orcid.org/ )
*Elsevier (2011a) Evidence given by Mayur Amin to the UK House of Commons Science and
Technology Inquiry into Peer Review, 11 May 2001. Transcript of oral evidence, HC 856, Q127.
http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf
Irene Hames, ABEC Brazil, November 2012 19
„Wastage‟ of reviews?
Rejected manuscripts can go from journal to
journal, fresh reviews at each
„Cascading‟ submissions and reviews
Between publishers: Neuroscience Peer Review
Consortium (http://nprc.incf.org/)
Within publishers and societies
Irene Hames, ABEC Brazil, November 2012 20
.
Innovations in peer review
Irene Hames, ABEC Brazil, November 2012 21
Two functions of peer review separated
PLOS ONE launched 2006, published ~14,000 articles in 2011
(~1.5% of world‟s scientific literature) using >38,400 reviewers
Publication based on „soundness‟ - research methodology, results
and reporting - not novelty, interest or potential impact
Evaluation of interest/impact left for post-publication
Impact Factor 4.411
Open access, „repository‟ type journals - „PLoS ONE clones‟ – being
launched (BMJ Open, Sage Open, Scientific Reports, Biology Open,
AIP Advances, SpringerPLus)
22 Irene Hames, ABEC Brazil, November 2012
More transparent approaches
Publishing reviews, ms versions and editorial
correspondence
BMC series medical journals – „pre-publication history‟
EMBO Journal – „peer review process file‟
BMJ Open – „peer review history‟
Reviewers‟ names may or may not be revealed
„Cross-peer review‟ – EMBO Journal
Open peer review
Irene Hames, ABEC Brazil, November 2012 23
„Open review‟ can mean a number of things
Irene Hames, ABEC Brazil, November 2012 24
Authors’ names
known
Authors’ names
not known
Reviewers
’ names
known
„Open‟ review
Term „open‟ may also include:
(i) Reviewers‟ names being disclosed
for published articles
(ii) Reviewers‟ reports (with or without
names) being included with published
articles
(iii) Editorial correspondence and/or
all versions of the manuscript being
included with published articles
(iv) Community/public being able to
comment during review
(v) Combinations of the above
(unlikely that this system is in
operation anywhere)
Reviewers
’ names
not known
„Single-blind‟ review
The most common form in scientific,
technical and medical (STM) journals
„Double-blind‟ review
The most common form in the
humanities and social sciences
(HSS)
Many new initiatives
In past year:
Peerage of Science (Winner of ALPSP Award
for Publishing Innovation 2012)
http://www.peerageofscience.org/
PeerJ https://peerj.com/
F1000 Research http://f1000research.com/
Rubriq http://www.rubriq.com/
Irene Hames, ABEC Brazil, November 2012 25
Peer review doesn‟t stop at publication
When real peer review starts?
Post-publication review and evaluation
Increasing opportunities for innovation
Challenges and problems
Increasing importance of blogs, twitter and
other social media
Irene Hames, ABEC Brazil, November 2012 26
.
#arseniclife
Carl Zimmer
http://blogs.discovermagazine.com/loom/2012/10/03/weirdly-unweird-a-
better-end-to-the-arseniclife-affair/
Irene Hames, ABEC Brazil, November 2012 27
.
„Online scientific interaction outside the traditional
journal space is becoming more and more
important to academic communication‟
Mark Hahnel, founder, FigShare (http://figshare.com/)
Irene Hames, ABEC Brazil, November 2012 28
A big challenge - data
Massive amounts being generated
Recognition for producing, making usable by
others and curating
Where to put?
Dryad http://datadryad.org/ - international
repository of data underlying peer-reviewed
articles in basic and applied sciences; can be
made securely available for peer review
Irene Hames, ABEC Brazil, November 2012 29
Wiley-Blackwell Copyright 2007 30
A reviewer‟s point of view
„Reviewers peering from under a pile of „omics‟ data‟ J.K.
Nicholson, Nature (2006), 440, 992
“The scientific community needs to reassess the
way it addresses the peer-review problem,
taking into account that referees are only human
and are now being asked to do a superhuman
task on a near-daily basis.”
Peer review ….
Mark Ware: „far from being in crisis, peer review remains
widely supported and diversely innovative‟
Fiona Godlee: (BMJ Editor): „At its best I think we would
all agree that it does improve the quality of scientific
reporting and that it can improve, through the pressure of
the journal, the quality of the science itself and how it is
performed.‟
Ware M (2011) New Review of Information Networking, 16(1): 23-53
Godlee F (2011) Evidence given to the UK House of Commons Science and Technology Inquiry into
Peer Review, 11 May 2001. Transcript of oral evidence, HC856, Q97.
http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/856/856.pdf
Irene Hames, ABEC Brazil, November 2012 31
25% discount on all Wiley books
Code ABEC, valid until 31 May 2013
Irene Hames, ABEC Brazil, November 2012 32
Irene Hames, ABEC Brazil, November 2012 33
Thank you!
Dr Irene Hames
email: [email protected]
twitter: @irenehames