How to be an effective peer reviewer
Dr Diana MarshallHead of Reviewer Programmes
Taylor & Francis Group
Dr Jia YangReviewer Engagement Manager
Taylor & Francis Group
Overview of peer review• Overview of the process
• Why peer review is important
• Types of peer review
Who is involved?
Editor• Assesses the article• Usually selects suitable
reviewers• Makes decisions
Reviewers• Assesses the detail• Give advice and expertise to
the Editor
Journal staff• Check format and journal
requirements• Manage communications• Production processes once
article accepted• Maintain journal systems and
websites
Why is peer review important?
• Editors cannot be experts on the detail of all
subjects
• Peer review is a conversation between the
authors, the editors, and the reviewers -
improve the quality of research and enable
effective communication of knowledge
• 90% of researchers believed their last paper
was improved through peer review - Sense
about Science Peer Review Survey 2019
“After authors, reviewers are
the lifeblood of any journal.”
Mike J. Smith, Editor-in-Chief, Journal of Maps.
Types of peer review
• Reviewers know the identity of the authors• Authors do not know the identity of the reviewers• Most common model of peer review
• Reviewers do not know the identity of the authors• Authors do not know the identity of the reviewers• May fail to hide author identity in 25-50% of cases*
• Reviewers know the identity of the authors• Authors know the identity of the reviewers• Reviewer reports may be published with reviewer
names if article accepted
*O’Connor E, Cousar M, Lentini J, Castillo M, Halm K, Zeffiro T. Efficacy of double-blind peer review in an imaging subspecialty journal. AJNR Am J Neuroradiol. 2017;38(2):230-235.
Single-blind/
Single-anonymous
Double-blind/
Double-anonymous
Open peerreview
Other models of review
• Transparent peer review: reports published without reviewer names• Post-publication peer review: article published without review and process open online
Responsibilities and expectations on reviewers• Expectations on reviewers
• When you receive an invitation letter
Give readers:
• Trusted research integrity of the article
• Ensuring adequately detailed methods to allow readers to judge the merit of the study design
Give authors:
• Detailed feedback
• Highlights any errors or gaps in literature
• Assists with making the manuscript more applicable to the journal readership
Give editors:
• Summarized information on research merit, originality, and validity of results
• Allow editors to assess the suitability of manuscript for publication in the journal
Reviewers
Editors
ReadersAuthors
Expectations on reviewers
When you receive an invitation letter:
• Meeting the deadline is essential for the whole peer review progress• Peer review is not a one-off task, you may be asked to review further versions of that manuscript
Can you complete the review in a timely fashion?
• From the information in the invitation, do you think that you can assess all or part of the manuscript?
Do you have the right expertise?
• Conflicts of interest are connections to an author, idea, or funding organization that could interfere with your ability to be objective
Are there any potential conflicts of interest?
Are you happy with the type of review used by the journal?
When you receive an invitation letter:
• Accept
• Decline - Indicate the reason‒ Declare conflicts of interest if any ‒ Give keywords to make sure journal editor knows your expertise, if the invitation is
not within your subject area‒ Suggest replacement reviewers if you are able (DO NOT contact potential reviewers)
• Unavailable‒ Specify when you will be available ‒ Editors may get back to you with an extended deadline
The Editor would rather have a quick decline than a report which does not help them make a decision
It’s OK to say no
Bear in mind…• DO submit your report and
recommendation in a timely manner.• DO declare any potential conflict of interest
before agreeing to review and any relationship that may potentially bias your review
• DO keep the peer review process confidential – from the moment you get the invitation
• DO judge manuscript on its merits, without regard to race, religion, nationality, sex, seniority, or institutional affiliation of the author(s)
• DON'T use any of the results or novel approaches reported in the manuscript until it is published
• DON'T share or discuss any aspect of the manuscript with anybody else
• DON'T make personal attacks on the author rather than providing assessment of the work itself
• DON'T ask the authors to cite your own or colleagues work unnecessarily
COPE Ethical Guidelines for Peer Reviewers
Critically assessing a manuscript• What reviewers look for
• Getting an overview of the paper
• Detailed assessment
• Should peer review detect fraud and misconduct?
• Assessing language
What do peer reviewers look for?
Presentation Quality
Research significance
OriginalityValidity
• Contribution to current knowledge
• Interest and importance
• Original contribution• Novelty
• Soundness of study• Appropriateness of the
conclusions• Research ethics
• Quality of language• The style of the paper
4 DIMENSIONS OF PEER REVIEW
Getting an overview of the paper
1. Get an idea of the direction that the paper is going• Abstract
• Methods and figures
• Conclusions
2. Read through beginning to end
• Make brief notes if needed
• No need to pick up all the details yet
15
Once you have an overview go back to the beginning and work through section by section for your detailed assessment
Detailed assessment:
Concise, accurate, and informative
A clear, short summary of the full manuscript
Why the study matters and putting the research in context
Appropriate to answer the research question and enough detail to repeat
Clearly describe the outcomes
Interprets the results, discusses limitations
Title
Abstract
Introduction
Method
Result (incl. table and figures)
Discussion &conclusion
Manuscript
Consider what is required from each section of the article
Find our checklist online for the details to check in all the sections
Detailed assessment:
• Need to be suitable to answer the question and enough description to be repeatable
• Consider the methods impartially
Ask questions when assessing:
• Are the study design and methods appropriate for the research question?
• Is there enough detail to repeat the experiments?
• Is it clear how samples were collected or how participants were recruited?
• Is there any potential bias in the sample or in the recruitment of participants?
• Are the correct controls/ validation included?
• Has any randomization been done correctly?
• Is the time-frame of the study sufficient to see outcomes?
• Is there sufficient power and appropriate statistics?
• Do you have any ethical concerns?
Researchers gave articles with the same methodology and different outcomes to 75 reviewers:• More likely to notice errors in methodology if
the results were negative• More positive about results which matched
their theoretical preference
Mahoney 1977, Cognitive Therapy and Research 1(2)Mahoney, M.J. Cogn Ther Res (1977) 1: 161
https://doi.org/10.1007/BF01173636
Pay particular attention to the methods
• Clearly describing outcomes which
would be derived from the methods
presented
• Data should not be selectively reported
• Results should be descriptive and
detailed, not presenting explanation or
discussion of the data
• It is ok for results to be negative
Checklist
• Do the results presented match the methods?
• Have all the relevant data been included?
• Are the results presented clearly and accurately?
• Is there any risk of patients or participants being identified?
• Is the data described in the text consistent with the data showed in figures and tables?
Detailed assessment:
Results
Detailed assessment:
• Is the data presented clearly and accurately in the figures and tables?
• Is the presentation of tables and figures consistent with the description
in text?
• Are the figure legends and table headings detailed enough to stand
alone from the text?
• Do you have any concerns about the manipulation of data?
• Is there any data in the main article which should be in the
supplementary files?
Look carefully at the tables and figures, including the supplementary data
• Interpretation of the findings
• Comparison with current knowledge
• Importance for future research
• Correspond with the aims of study
• Supported by data presented
• Realistic about limitations and what next steps might be needed
Checklist
• Do the authors logically explain the findings?
• Do the authors compare the findings with current findings in the research field?
• Are the implications of the findings for future research and potential applications discussed?
• Are the conclusions supported by the data presented?
• Are any limitations of the study discussed?
• Are any contradictory data discussed?
Detailed assessment:
Discussion and Conclusion
Reporting guidelines
Checklists of key points to cover in different biomedical articles – can help identify what needs covering in a reviewhttp://www.equator-network.org/
Taylor & Francis Author Survey, 2015
Should peer review detect fraud and misconduct?
“Peer review is not intended for, and is not an efficient or effective means for, the
detection of deliberate research fraud, or indeed other forms of misconduct.”
Mark Ware, Peer Review: An Introduction and Guide
• Peer review is not primarily to detect misconduct• But peer reviewers are looking closely at the article
and may see things that others have missed
Research misconduct Publishing misconduct
Author misconduct
If you have concerns then contact the journal
Data fabrication
and falsification
Plagiarism & self-plagiarism
Research ethics
• Deliberately made up data (fabrication) or change data (falsification)
• Re‐use without appropriate citation of the original source
• Any concerns about treatment of patients/ participants/ animals
Multiple publication
Others
• Submitting the same paper to more than one journal at the same time
• Improper author contribution or attribution
• Undeclaration of conflicts of interest
• DO contact the journal editor‒ If you want to see the original
files‒ If you are very concerned then
email immediately‒ If not urgent, could be included
in comments to editors‒ Give the journal specific details‒ Using neutral wording and
reporting
What to do if you are concerned
• DON’T try to contact the author directly
• DON’T accuse the author of deliberate manipulation
• DON’T investigate in detail e.g. run a plagiarism check yourself
Assessing languageShould you correct spelling and grammar errors?
Difficult to understand
Consider rejection orsuggest to editor revisions required
Not clear but understandable
Request a language polishing in review
report
Minor spelling and grammar issues
Ignore, there will be copyediting prior to
publication
79% of recommendations by reviewers were
influenced by grammar and writing style.M Shattell, et al., Journal of Nursing Scholarship, 42(1), 58-65, 2010
Reviewing non-research articles
Review article• Summary of current state of research• Balance and accuracy
Systematic review• Focus on specific methodology• Review the methods in detail
Software articles• Describing development of a new software tool• Suitable testing in relevant conditions• Is the tool available? e.g. code open source on
GitHub as required
Methodology• Detailed description and testing of a novel method• Rationale for having the new method and is it possible
to replicate
Opinion (commentary, editorial)• Variety of articles where the author gives their views
on a topic• Based on current knowledge and contributed to the
research field?• Well-argued?
Check the journal’s instructions for authors
Case report• Describing clinical progression of an unusual
disease, presentation of a disease or side-effect• Diagnosis/treatment/outcomes clearly described
Qualitative research
• Is the aim of the study clear?
• Do authors provide sufficient information about context of the research, data collection and analysis process?
• Is the methodology well described and suitable for the research question?
• Is the questionnaire standard or suitably validated?
• Are the findings and analysis aligned with used methodology?
• What contribution does the study make to knowledge?
Qualitative research seeks in-depth understanding of social phenomena.
It focuses on the “why” and “how” rather than the "what" and “how often” types of questions
Bear in mind…• Neutrally assess the research which is there, not what you wish the
authors had done
• Most research is not perfect, as long as the limitations are clear it
doesn’t need to be
• Take notes constantly by assessing not only the issues to improve but
also the strengths of the manuscript
• It’s not the reviewer’s responsibility to investigate author misconduct
• It’s not the reviewer’s responsibility to polish grammar or spelling
• Communicate with the editor/journal if you have any problems or
concerns
Writing a review report• Structure of the report
• Making your recommendation
• Style of the report
• Re-review
Structure of the review report
• What the manuscript is about• Key findings and conclusions• Contribution of the manuscript• Strengths and weakness
• Essential points the authors must address for publication
• Fundamental points for the current study
Comments and recommendation should match
SUMMARY
MAJOR COMMENTS
MINOR COMMENTS• Still important but will not affect the overall
conclusions• Good suggestions, suggested but not essential
comments to the authors
Major vs. minor issues
Minor issuesStill important but will not affect the overall conclusions• Missing references• Technical clarifications • Unclear labelling of figures and
tables• Spelling, grammar and phrasing
issues
Major issuesEssential points the authors need to address before the manuscript can proceed• Problems with study design • Issues with data or analysis• Experiments not following best
practice
Help the authors improve the manuscript before publicationThese comments can be seen by both editor and authors
Comments to the authors
• Summarize the strengths of manuscript, give positive feedback• Do not mention whether the paper should be published unless the
journal requests this• Be specific so the authors know what they need to do to improve• Number your comments and provide one major comment per
number• Refer to page and line numbers in the manuscript
• Give editors an overall picture about the manuscript and help editors to make their decision
• These comments will not be seen by authors• Do not include detailed concerns about the manuscript which the
authors need to be aware of
Confidential comments to editor
• Comments about the manuscript’s novelty, research significance or suitability for the journal
• Comments not to be shared with the authors‒ Concerns over suspected
author misconduct ‒ Anything that may disclose
your identity‒ Any issues relating to conflicts
of interest
Key questions:• Can your concern be
addressed?• How long will it take to
address your concern?
Recommendation• Methodologically sound
• Fits the journal scope
• Completeness
• Correctness
• Need to redo the study or add significant work
• Major flaws in the methodology • Lack of significance/novelty• Not suitable for this journal
May need no further reviewing
Suitable for publication in its current form
Further reviewingNo guarantee for acceptance
• Grammar/ Format
• Clarifications
• Accept
• Minor revision
• Major revision
• Reject
Style of the report:
• Fair, polite, clearly, and not personal
• Objective & constructive
• Key message is clear and consistent
• Specific not brief
• Check your own spelling and grammar for clarity Cartoon by Nick D Kim, strange-matter.net
Review as you would want to be reviewed
avoid saying… vs. could say…“The writing is terrible.” or “The
writing is so bad, it is practically
unreadable.”
“The authors should revise the language to
improve readability.” or “While the study appears
to be sound, the language is unclear, making it
difficult to follow. I advise the authors improve
the flow and readability of the text.”
“The technical details don’t make
sense.”
“The technical details should be expanded and
clarified to ensure that readers understand exactly
what the researchers studied.”
“The manuscript is fatally flawed.” “The study appears to be missing a key point” or
“Additional work is required to support the
conclusions”.
Example comments
Re-review
• Focus on how well your original comments have been addressed• Avoid raising new concerns unless they have to do with the author’s
revisions• Avoid asking for further experimentation• If the authors decided not to follow your suggestions evaluate their
reasons fairly, don’t take it personally
How to re-review
• The journal may ask you to re-review to confirm whether the points have been addressed
• Consider as part of the commitment of original review• Look carefully at emails
Agree to re-review
Bear in mind…• DO give positive feedback first in your report
• DO keep the focus on the research and not the
author
• BE professional and respectful.
• BE aware of the possibility for bias in your
review
• DO proofread your report and check
everything one more time before submitting it
• DO raise your suspicions in the comments to
editors if you suspect plagiarism, fraud or have
other ethical concerns, providing as much
detail as possible
• DON'T say whether the paper should be
published or not in the comments to authors
• DON'T try to tell the authors exactly how to
revise their manuscript
• DON'T use the review to promote your own
research or hypotheses
• DON'T feel under pressure to detect fraud and
misconduct
• DON'T make comments that could make your
identity obvious to the authors
How to be an effective reviewer?1. It is ok to say no to an invitation
2. Start by getting an overview of the article
3. Consider what is required from each section of the article
4. Pay particular attention to the methods/methodology
5. Look carefully at the tables and figures, including any
supplementary data
6. Start your report with a summary
7. Make it clear which comments are essential
8. Be specific
9. Review as you would want to be reviewed
10. Contact the journal or Editor if you have any questions or problems
Questions?
https://editorresources.taylorandfrancis.com/reviewer-guidelines/
For more information about reviewer training with Taylor & Francis:https://editorresources.taylorandfrancis.com/reviewer-guidelines/peer-review-training/