Professional Development Review & Critical Appraisal Skills for Peer-

Reviewed JournalsAka “reviewing & responding to reviews”Presented by: Elena M. Andresen, PhD

Chief, Disability & Health Research GroupInstitute on Disability & Development Oregon Health & Science University CDRC room 1272A Office: 503-494-2275email: andresee@ohsu.edu

Sponsored by OHSU Department of Public Health &

Preventive Medicine, School of Nursing, & the Institute on Disability &

Rehabilitation

My Background, Limits, & Disclaimers

• Professional Service– Editorial Boards J Gerontology; Dis Health J– Associate Editor Qual Life Res (Springer). Cumulative

portfolio ~115 manuscripts – Frequent manuscript reviews: medical/clinical, public health

journals (~ 8-15 manuscripts/year)– Frequent U.S. grant reviews: ad hoc & standing panels

• Research & Scholarship – Author/Coauthor of 100+ peer-review publications – PI/Investigator 40+ national, state, international,

& foundation grants & contracts

Background, Limits, & Disclaimers

• Training: – No formal training/advice on reviewing for

journals – No training on responding to journal reviewers – Over time, I observed expert reviews/reviewers,&

found some nicely constructed papers to help me personally, & to share.

• Most importantly: colleagues & collaborators provide exceptional role-modeling, constructive criticism, & encouragement

Your Experiences:

Have you…• Coauthored a journal submission?• Been a lead (corresponding) author for

a journal submission?• Performed manuscript reviewer for a

peer-review journal?• Received harsh, inadequate, or highly

useful feedback from reviewers, readers, instructors, or collaborators?Your experiences are valuable & add to the perspectives in this presentation.

Expectations Today

• Content: Lecture slides, three pdf review examples (de-identifed)

• Suggested readings: Many possible other resources & I have a few suggestions.

• Availability: Email me or make an appointment if I can be of specific help.

Successful “Academic” Careers *

These 4 components vary by setting. Review of your qualifications for jobs, grants, & promotions may include:

• Research (funding & experience)• Productive scholarship (peer-reviewed

papers, presentations, other scholarship)• Service (national & international professional

work in your discipline)• Other career components for some

– Teaching – Clinical duties

* In reality, these are part of most professional positions

And… There is a feedback loop

• Teaching & training ideas are partly derived from your research experience & publications

• Trainees may seek you out based on your peer-reviewed scholarship

• Research funding depends, in part, on your peer-reviewed publication productivity

• Invitations to professional service depend partly on your being “known” via peer-reviewed scholarship

Scholarly peer-reviewed success stimulates success in the other activities.

Professional service provides insight that feeds scholarship & research.

Part 1: Becoming a successful journal peer-reviewer

Peer Reviewing

• Myth: If you review (for a specific journal) your manuscripts are more likely to be accepted– Most journals use management software systems

where this information would not be available to reviewers, & editors may be unlikely to examine authors/reviewers service in their final decisions

– Reviewers may be “blinded” to author identity (journals vary)

• Not a Myth: Constructing reviews of good & bad manuscripts helps you write & submit

Peer Reviewing

• Not a Myth: review quality & reviewer performance is rated in some journal management systems. This may affect your future service for the journal.

• Example: for Quality Life Research & other journals, editors (who search for & invite reviewers) examine…– Reviewer’s average days to submit reviews – Average editorial rating of reviews’ quality– More descriptive issues; e.g., total invitations &

completed reviews; declined, agreed, uninvited & terminated reviews.

Personal Peer Reviewing Benefits

• Another good outcome of journal reviewing: learning to respond to reviews! (Part 2)

• Suggested reading on peer review: Cummings & Rivara. Responding to reviewers comments on submitted

articles. Arch Pediatr Adolesc Med 2002; 156: 105-107.Cummings & Rivara. Reviewing manuscripts for [our journal]. Arch

Pediatr Adolesc Med 2002; 156: 11-13.Schroter et al. What errors do peer reviewers detect, & does training

improve their ability to detect them? J R Soc Med 2008;101: 507-514.Schroter et al. Differences in review quality & recommendations for

publication between peer reviewers suggested by authors or by editors. JAMA 2006; 295: 314-317.

Committee on Publication Ethics (COPE) documents (see website http://www.publicationethics.org.uk/about)

Getting Started for Journal Peer Reviewing

• Get some practice with mentors– Ask to work as a co-reviewer. Discuss your practice

reviews, compare yours to theirs, etc. – Ask if your mentors will share an especially helpful

review they received. • Volunteer if the opportunity arises

– Ask mentors & colleagues to recommend you & don’t let them down.

• Introduce yourself to editors (& associate editors) at meetings– Look at favorite target journals for editor/board names.

Getting Started for Journal Peer Reviewing

• Read articles, expectations, & suggestions from journal editors – Search your favorite journals for recent

editorials about reviewing & expectations, & for good studies about review quality, etc.

• Be prompt & thorough when you are asked & you agree to review– We will spend more time on this.

Selection of Reviewers

• Register at the journal (reviewer) website– Complete name, title, institution(s)– Expertise: Be comprehensive.

• Editors look for reviewers by keywords & categories• Usually this is a list that you check off for your

interests & discipline or experiences• It would be rare that one of these sites has categories

that fit you perfectly, so add text if you have the opportunity. However, the management systems probably uses only the check-box information when an editor is searching. The added text may be used after looking at a possible reviewer record in more detail.

Selection of Reviewers

• Register at the journal (reviewer) website– Don’t add more categories of expertise than is

reasonable – you might refuse more invitations this way, because you will be sent invitations that are far outside your comfort level.• Refusals may be recorded in your reviewer record.

– Keep a record of journal reviewer information; e.g., your login name & password, & update your data as it changes • when you move from graduate student status to a new

institution, email address, etc. – Track your reviews. Someone, usually your

supervisory, will want a report

Selection of Reviewers

• Answer invitations promptly, & be helpful with your suggestions for other reviewers if you decline– A quick refusal is nearly as good as a quick acceptance– Each delay in a reviewer response can affect the overall

time that a manuscript is in review. – When you say no, or automatically when some number of

days expire on your invitation, the system moves to the next listed (pending) reviewer & sends out a request (& notifies the managing editor).

• Manuscripts are delayed by a number of review factors: reviewer delays in responding to invitations are part of total review time (journal decisions to authors)

General Guidance

• Read the abstract to see broad issues of what you are asked to review– Usually part of the initial invitation

• There are multiple reviewers (2-4), so you do not need to know everything about the manuscript to provide a good review – You may provide more on methods than content

/ topic– You may provide more clinical or basic science

input then research methods • Set a reasonable timeline & keep to it

– The sooner you start the better– It takes the same time no matter when you do

the review

General Guidance

• Read Editors’ guidelines for review & format– Do you use a check list of attributes in addition to a text

review? Go to the tab on “submit review” if it is your first time to see what they ask for. May provide orientation.

– Are you required to use a numbered format (I recommend you do so even if not required), & do you need to provide manuscript details like section/row numbers for your comments?

• Start a structured WORD document – name it by journal & use their system manuscript

number, e.g., QualLifeRes9977.doc– Personally: I review on a paper copy & make notes

directly on it before starting a WORD file

General Guidance

• Often you are asked for two review formats– A private summary of your recommendations about

publication & special issues. – Then the full review for the authors, where you typically

do not say if you recommend publication. • It may help to construct 2-3 sections & then write

in your content (draft comments) as you read, e.g.– Mandatory changes (or major comments)– Minor changes (or recommended comments)– Editorial suggestions (please be generous!)

Specifically

• Say something nice! (The Golden Rule)– Consider your role: provide editorial decision support,

constructive criticism, & improve science quality. • Provide an accurate study description,

setting; highlight benefits of (design or importance of the question, etc.)

• Address major comments & questions typically a list: use complete sentences, specifics from the manuscript. Consider: Both strengths & weaknesses OriginalitySample (appropriate to question) Study design Measures & analytic methods Theory

Conclusions/discussion/interpretation

Specifically

• Minor & important – Unclear vs. flawed aspects of the manuscript – Literature & references (Thorough? Current?)– Use of terms & language that match the journal– Tables & figures (lack of clarity, e.g., titles,

complete words, footnoted effectively, etc). – Suggest other editorial issues for clarity, e.g.,

reduce jargon & abbreviations

Specifically

• As noted, some comments don’t go to authors; summarize additional issues only for the editor(s). – Frank advice to the editor, or if your review & advice

conflict (Review: pleasant & constructive. Advice: reject) – Example: a naïve manuscript, that didn’t follow scientific

construction or format. The topic & sample (& apparently available data) would strengthen our field. All the details went to authors. I told the editor why I had not rejected it.

– Example: a reviewer did not find many faults with general analysis & construction of the manuscript, but noted to the authors that there were few new elements in the manuscript. In the comments to the Editor, s/he noted that the manuscript was quite duplicative of other (un cited) studies.

Next

• Set the review aside briefly (a day)• Edit the review for clarity & writing

– Be as clear with your review as the authors should be with their manuscript

• Re-read the manuscript & make final edits• Send the review promptly & confirm that it

was registered – Sometimes electronic journal tracking sites are

imperfect, & sometimes reviewers are imperfect – Typically you receive an email thanks, but you

can check the journal review website to see if you still have a review “pending”

Examples (anonymized) Reviews

Link to document with manuscript reviews for three different journals:

“example manuscript reviews.pdf” These are my reviews with different review formats. Each has been edited to remove aspects that might identify the study, locations, etc., so forgive the multiple places where I insert “letters” replacing details like conditions, measures, etc.

Part 1 Reflections & Discussion

Courtesy of the Colorado Home Initiative & the Florida Office on Disability & Health http://fodh.phhp.ufl.edu/

Angola: young man with polio, WHO. http://www.who.int/features/2005/disability/en/

Part 2: Responding to journal reviews of your manuscripts

Think “Oregon social & political styles.”

Consider the process, & reviewers, as important diverse players in your scholarly environment. As in Oregon, you don’t always agree with what others say & do around you, but you are open to supporting a “better Oregon” & their right to express themselves.

Responding to reviewers: part of methods of successful publication

• Arguments with reviewers may be futile– Feel free to disagree, nicely, describing why

• Use an appreciative, thoughtful style– “We thank the reviewer for catching this misplaced

decimal…”– “Editing recommendations result in clearer language …” – “The Reviewer asked an important question prompting ad

hoc analyses… although the results did not change our conclusions, the process provided an important addition to the discussion section…”

– “The analysis suggested by the Reviewer is an interesting variation of our choice. We considered this with our statistical advisor, & feel that the use of non-parametric methods better suites our non-normal data & sample size & now reference this in…”

Responding to reviewers

• Be thorough – summarize each reviewer comment & describe what you did & where

• ‘There are no page limits to response letters’ – Cummings & Rivara 2002

• I once sent a 9-page response with new tables & analyses (not added to the revised paper) suggested by reviewers– I thanked them for suggestions prompting a more thorough

discussion section & added the ad hoc analyses to methods, & to results & interpretation

– If you do not have room, for page limits, add a comment in the revisions (data available from authors) or similar.

Learning from Journal Peer Reviewing Experience• These somewhat arbitrary suggestions

are based on my experience & colleagues who are journal editors

• Reviewing others’ work helps you to be a productive (successful) scholar

• A few tips & inside experiences follow. – As a reviewer & editor, I often make similar

suggestions to authors

Successfully Communicating to Reviewers: Tables –drop dead easy

• Tables should “stand alone” (from colleague Fred Wolinsky@Iowa & APA criteria)– E.g. use variables in columns that someone can

understand – E.g., use a full informative title with data set,

topic, sample, etc.– E.g., label columns, with numbers where needed.– Be clear about data, %’s, means, variances, etc.– Use leading zeros (0.06)– Use informative footnotes liberally – Avoid abbreviations if not absolutely needed for

brevity (use consistent shorter terms, e.g. not a full question or clinical description: use an informative full word)

Table Example (uninformative)

Variable OR P-valuePrior fall 2.51 *Increasing education 1.10 *CESD 1.50 *BMI < 20 > 30

.65 .99

Arthritis .97Lower body limits 1.05

Table 2: Risk factors for falls

* p<.05

Baseline Independent Variables OR adjusted + P-value

Prior fall reported at baseline 2.51 ***Increasing education (years) 1.10 **Clinically relevant depression symptoms # 1.50 *Body Mass Index < 20 vs. 20-29.9 > 30 vs. 20-29.9

0.650.99

nsns

Self reported arthritis 0.97 nsLower body limits, per point § 1.05 ns

Table 2: Risk factors for incident falls among 888 African Americans aged 50-64 during two years of follow-up

+ Weighted odds ratio from binary logistic regression adjusted by all other model variables * p ≤ 0.05 ** p ≤ 0.01 *** p ≤ 0.001 (ns= not statistically significant p > 0.05)# Measured as ≥ 9 points, Center for Epidemiologic Studies Depression Scale (CESD)§ 6 items summed (difficulty walk ¼ mile; walk up & down 10 steps; stand 2 hours; stooping; lift 10 lbs; push large objects)

Personal Editor Experience

• Somewhat of a myth: Your suggestions for good reviewers will be the basis for selecting & inviting reviewers of your paper– Editors may assume you select colleagues even if

they are not currently at your institution– An editor told me he took a random sample of

submitted papers & tracked web data on reviewer/authors relationships (he said there were lots).

He used author suggestions to build the reviewer database, sending them invitations & asking them to add details about themselves at the journal reviewer website (but rarely used them for the submitted manuscript).

Final Thoughts & Suggestions

• Your science is always your first concern• Consider “packaging” as a task you perform

to maximize the outcome of your science• Consider your overall career goals early, &

don’t apologize for considering these as you prepare, submit, & resubmit your work

• Don’t go against your core values & make decision that feel like you are “selling out”– But push your comfort limits a bit as you develop

your career, & don’t forget how important it is to communicate your work effectively!

Final Thoughts & Suggestions

• Use both successful journal experience & failures to inform future decisions, not to direct your decisions – knowledge is power

• Use coauthors & colleagues for information & feedback for submissions & resubmissions– Consider criticisms in light of “style” vs. “content.”

Style is the prerogative of the first author & the journal • Develop patience & perseverance

– Keep repackaging good science if you do not first succeed!

– It is common to select an inappropriate first target journal – consider target journals carefully

Thank you

View from the Oregon

Health & Science

University