Yes, You too, can Peer Review

How to contribute to the peer review

process

Presented by: Andrea Boccelli, Publisher, Elsevier and

Gina Sucato, MD, MPH, Director of Adolescent Health, Kaiser Permanente

Seattle Washington

April 21, 2017

Financial Disclosure

Andrea Boccelli is an employee of Elsevier,

Inc, a Science, Technical and Medical

Publisher and the Publisher of JPAG

Gina Sucato has received support from

TEVA to conduct and present findings from

TEVA- funded research projects

2

Objectives

Describe the fundamental principles of peer

review in academic publishing

Identify the critical components of a

constructive review of a scholarly manuscript

Improve skills in providing useful feedback to

journal editors and peer authors

3

Registration The timestamp to officially note who

submitted scientific results first

Dissemination Provide a medium for discoveries and

findings to be shared

Preservation Preserving the minutes and record of

science for posterity

Role of Scientific Publications

Certification Perform peer-review to ensure the validity

and integrity of submissions

JPAG Over Time…

1986

1988 2003

6 times per year!

1996 2012-2015

IF

2013

2017

1.630

1.812 1.683

1.605

JPAG Stats 2016

Articles submitted: 448

Rejection rate: 56%

Submission to Acceptance: 19.8 weeks

Total article downloads: 299,319

6

JPAG Article Growth

0

50

100

150

200

250

300

350

400

450

500

2012 2013 2014 2015 2016

Number of articles submitted

Number of articles accepted

8

Picture: @LDMay

| 9

What is peer review?

Peer review places the reviewer, with the author, at the heart of

scientific publishing

Reviewers make the editorial process work by examining and

commenting on manuscripts

Without peer review there is no control in scientific

communication

Reviewers are the backbone of the whole process

The best editors and reviewers tend to view

themselves as teachers rather than critics.

The goal is to improve the work published – for the

sake of the authors, readers and science overall.

| 10

Purpose of peer review

Improves quality of the published paper

Ensures previous work is acknowledged

Highlights any omissions in the

reference list and any ethics concerns

Determines the importance of findings

Assesses the originality and significance

of the work

| 11

Role and tasks of the reviewer The peer review process is based

on trust

The scientific publishing enterprise depends largely on the quality and integrity of the reviewers

Reviewers should write reports in a collegial and constructive manner

Reviewers should treat all manuscripts in the same manner

| 12

JPAG peer-review process

Source: Modified from Peer review: the nuts and bolts,

Sense About Science, 2012

Author submits

article to journal

Accepted no

revisions required

Rejected after

screening = “Desk

Reject”

Journal Associate

Editor screens

paper

Reviewer Reviewer

Rejected Makes revisions

Associate Editor

assessment of

reviews

Journal Editor-in-

Chief screens

paper

Editor-in-Chief assessment of

recommendation from

Associate Editor

| 13

JPAG peer-review process

Source: Modified from Peer review: the nuts and bolts,

Sense About Science, 2012

Author submits

article to journal

Accepted no

revisions required

Rejected after

screening = “Desk

Reject”

Journal Associate

Editor screens

paper

Reviewer Reviewer

Rejected Makes revisions

Associate Editor

assessment of

reviews

Journal Editor-in-

Chief screens

paper

Editor-in-Chief assessment of

recommendation from

Associate Editor

Rejection without external review

Editor or Associate Editor rejects without

sending out for review

Poor quality

Out of scope/ not of interest to journal readership

Allows submitting authors to submit elsewhere

in a timely way

Respects the time of the reviewers

14

| 15

JPAG peer-review process

Source: Modified from Peer review: the nuts and bolts,

Sense About Science, 2012

Author submits

article to journal

Accepted no

revisions required

Rejected after

screening = “Desk

Reject”

Journal Associate

Editor screens

paper

Reviewer Reviewer

Rejected Makes revisions

Associate Editor

assessment of

reviews

Journal Editor-in-

Chief screens

paper

Editor-in-Chief assessment of

recommendation from

Associate Editor

| 16

Articles are initially reviewed by at least two reviewers

When invited, the reviewer receives the abstract of the manuscript

The Editor generally requests that the article be reviewed within 2 weeks

Articles are revised until the reviewers agree, or until the Editor decides that the reviewer concerns have been adequately addressed

The reviewers’ reports help the Editors to reach a decision on a submitted paper

Review process (I)

| 17

Review process (II)

If report has not been received after 2 weeks, the editorial office

contacts the reviewer

If there is a notable disagreement between the reports of the

reviewers, a third reviewer may be consulted

The anonymity of the reviewers is maintained, unless a reviewer

asks the Editor to have their identity made known

| 18

Review process (III) Reviewers must not communicate directly with authors

All manuscripts and materials must be treated confidentially by

Editors and reviewers

The aim is to have a first decision to the authors by 3-6 weeks

(depending on the field) after submission

Meeting the schedule objectives requires a significant effort by all

involved

Reviewers should treat authors as they themselves would like to

be treated

| 19

Take Home Lesson

Authors sometimes experience peer review as distress they

need to get through to publish their work. But remember, in

your reviewer role you are a teacher not a critic.

| 20

Agree to review then never do it

Give only your opinion (ie, “this is great publish it”)

Fix simple grammar and spelling (the copyeditor

does this)

Use the data from the manuscript

Reviewers- What Not to Do

| 21

Confidential document

Manuscripts are confidential documents where the

data are and remain exclusive property of the

author(s)

Must be destroyed after the final decision from the

Editor

Shared responsibility for the review of the

manuscript with a colleague must be disclosed to the

Editors

| 22

Reviewers -Do

Ensure that you review manuscripts in area of

expertise only

Complete the review on time

Provide an honest and critical assessment

Analyze the strengths and weaknesses

Avoid any conflicts of interest

| 23

A competing interest is anything that interferes

with, or could reasonably be perceived as

interfering with, the full and objective presentation,

peer review, editorial decision-making, or publication

of research or non-research articles. Competing

interests can be financial or non-financial

(sometimes called “private interests”),

professional, or personal.

Conflict of Interest/Competing Interests Or “When should I recuse myself?”

Modified from http://journals.plos.org/plosbiology/s/competing-interests

| 24

Honoraria

Educational Grants

Participation in Speakers Bureaus

Employment

Consultancies

Stock Ownership

Expert testimony

Patent-licensing Arrangements

What are competing interests? Financial- past three years*

*ICMJE Form for Disclosure of Potential Conflicts of Interest

| 25

Personal or professional relationship

Collaborator

Co-author

Grant holder

Same institution/organization

Same research

Can be personal, political, academic, ideological, or religious

Difficult to quantitate

What are competing interests? Personal or “Private”

Being a reviewer

| 27

Why do reviewers review?

General interest in the area

Early awareness of latest research findings

Help with own research or new ideas

Association with journals and editors

Value from mentoring young researchers

Enjoyment in reviewing

Career development – become a better writer

Approaching the review

Step 1: read the article

Read through from start to finish

Make notes in margins

Both substantive and trivial

Detailed enough you’ll know what you meant

later

Set aside for a couple days

28

Conducting the review

Step 2: Perform the review

Carefully reread

Draft comments to author on each section

| 30

A systematic approach to reviewing Article section Description

General Impression Innovative and important

Writing Clear and concise

Title Specifically describes manuscript content

Abstract Accurately reports manuscript content

Introduction Adequately explains rationale for study

Methods Appropriate to answer the question, described in full

Results Not redundant with figures and tables

Figures & Tables Best way to illustrate the data? Clear. Can stand alone.

Discussion Places findings in the context of the current literature

Limitations Are the findings ready for implementation?

Conclusion Implications of the study findings and directions for future work

References Accurate and complete

Conducting the review

Step 2: Perform the review

Carefully reread

Draft comments to author on each section

Draft comments on the title and abstract last

Draft a brief summary of article

List of minor errors (only those necessary)

| 32

Innovative

Moves the field forward

Likely to change practice / be frequently cited

Elicits enthusiasm

Writing

Clear and concise

Organized

General impression

| 33

Title

Specific / not overly broad

Interesting (catchy but not cutesy)

Abstract

Accurately reports the manuscript

No inconsistencies with text

Includes the key results

Does not overstate the findings

Appropriate format for journal

Title and Abstract

| 34

Introduction

Effective, clear, and well organized?

Clear rationale for the study

Suggest changes in organization

3 paragraph introduction

Point authors to appropriate citations if necessary

Be as specific as possible when giving feedback

| 35

Methods

Complete, detailed description Could a colleague reproduce the study?

Proper references to previously published methods

Measures valid for the study population

Appropriate statistical tests chosen?

Sample size large enough? study adequately powered?

Selection bias

Response rate

All enrolled subjects adequately accounted for

IRB approval

Should supplementary material be made available? eg, survey items

Results, Tables and Figures

Are all data relevant to study aims included?

Do tables, figures, and text complement but not

repeat information?

Do the figures and tables stand on their own?

Complete legends, axis labels, units of measurement

Have authors chosen the clearest way to present

their data?

Are additional analyses needed?

Have potential confounders been accounted for?

36

| 37

Discussion

Does the argument proceed logically?

Are the results discussed in the context of all the

existing, relevant literature?

Why are their findings different?

If similar, how do they expand on prior findings?

Are the interpretations justified?

Are recommendations well supported and clearly

articulated?

Limitations acknowledged and addressed

| 38

Do the conclusions result directly from their study?

Not from the general body of knowledge or others’

work or what sounds like a good idea

Findings are not overstated or over-generalized

Comments on importance and validity

Not redundant

Abstract, not the Conclusion, summarizes the study

Recommendations for next steps

Conclusions

| 39

Check accuracy, number, and appropriateness

of citations

Correct interpretation

Most recent

Self citations

Minor edits

Those which copy editor might miss

References, minor edits

40

Writing the review: author comments

Step 3: Write up comments to the author

Start with a brief summary of the article

Comment on each section in order

starting with title and abstract

Provide specific comments to help strengthen the manuscript

Conclude with list of minor errors

only as necessary

Do not discuss decision (accept/revise/reject), as the final

decision depends on more that your own review

Typically ~ 500 words (300-1000)

Writing the review: editor comments

Step 4: Write comments to the editor

Ask yourself “Is the manuscript is worth being

published?”

Importance to the field

Are the findings novel / original / significant

Writing is clear and concise

Need for statistical review?

Any ethical concerns (e.g. duplicate publication)

Remember confidential comments are not disclosed to

the author(s)

| 43

Options:

Strongly disagree Disagree Neutral Agree Strongly agree

The subject addressed in this article is worthy of investigation

The information presented is new

The conclusions are supported by the data

The manuscript is appropriate for the journal

Organization of the manuscript is appropriate

Figures, tables and supplementary data are appropriate

Reviewer Checklist- JPAG

For Editor’s Eyes Only

Writing the review: editor comments

Recommend whether the manuscript is suitable for

publication

Is this the right journal?

Urgency of publication

Appropriate for an editorial?

Would you be willing to write?

If not, whom would you recommend?

Submitting the review

Step 5: Reread review

Are the comments to the author consistent with

the comments to the editor?

Proofread and edit

Is review specific and constructive enough to

improve the article?

Tone: would you feel comfortable reading it

aloud to a junior colleague about his/her work

45

| 46

Comments to the authors

“When reviewing, try to remember that you are an author too and

be professional and constructive in your approach. That can be

hard but don’t let your inner nitpicker get the upper hand. Leave

24 hours between reading the manuscript and writing your

review, to allow time for your reasonable self to rise to the fore.”

- Stephen Curry, Professor of Structural Biology, Imperial

College London

| 47

Provides an objective, thorough, and

comprehensive report

Provides well-founded comments for authors

Gives specific and constructive criticism

Provides a clear recommendation to the Editor

Submits the report on time

Editors’ view: what makes a good reviewer?

Characteristics of Good Reviewers

Study of 690 reviews of 420 manuscripts

Quality of review assessed by 2 editors and

corresponding author

7 item scale with good psychometric properties

Black et al. JAMA 1998

48

Review quality

Did the reviewer address

Importance of the research question

Originality of the question

Strengths and weaknesses of methods

Presentation of paper (organization, writing, illustrations

Interpretation of results

Were comments constructive

Were comments substantiated

49

Characteristics not predictive

Current academic appointment

Current research investigator

Number of papers published in past 5 yrs

Number of journals reviewed for

Member of editorial board, and other similar

positions and characteristics

50

Characteristics predictive

Younger (closer to 40 than 60)

Postgraduate training in epidemiology or statistics

These features had very limited predictive power

(predicted 8% of variance in outcome)

Review quality increased with time spent but not

beyond 3 hours

There are many types of good reviewers

51

Time Taken to Review

(Source: Peer Review Survey 2009)

Improving as a reviewer

Step 6: Read other reviewers comments

Was the article accepted or rejected?

Did the other reviewer(s) comment on same

flaws but communicate more clearly or with more

helpful suggestion for improvement?

Were there major concerns that you had not

noticed?

Destroy article

Additional References

Association for American Medical Colleges: A guide for reviewers.

A series of articles that together make up a guide for reviewers (old but

good)

https://www.aamc.org/members/gea/167684/guide_for_reviewers.html

Cummings and Rivara. Reviewing manuscripts for Archives. Arch Ped

Adolesc Med 2002

Hoppin JG. How I review an original scientific article. Am J Respir Crit Care

Med 2002

Roberts LW, Coverdale J, Edenharder K, Louie Alan. How to review a

manuscript. Academic Pscyhiatry 2004

54

| 55

Further reading at

publishingcampus.com

elsevier.com/authors

elsevier.com/reviewers

elsevier.com/editors

Understanding the Publishing Process

Publishing Ethics

Get Published- tips on writing and reviewing

Get noticed; how to promote your article

| 56

For Editors For Reviewers

Plagiarism detection tool at time of

submission

Tool based on Scopus database to

identify potential reviewers

Free access to

All content published by Elsevier

Free access to

The world's largest abstract and citation

database

Reference-linking and resolution in PDF

of the manuscript

References: tools for reviewers (I)

| 57

References: tools for reviewers (II)

| 58

Elsevier Publishing Campus

www.publishingcampus.com

Sense About Science Publications

www.senseaboutscience.org/pages/peerrevieweduc

ation.htm

More information on journal peer review

www.elsevier.com/reviewers

Thank you

INTERESTED IN REVIEWING FOR JPAG?

Contact JPAG Editor-in-Chief,

Dr. Paula J Adams Hillard

JPAGEditorinChief@gmail.com