![Page 1: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/1.jpg)
1
REVIEWER ORIENTATION REVIEWER ORIENTATION TO ENHANCED PEER REVIEWTO ENHANCED PEER REVIEW April 2009 April 2009
1
![Page 2: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/2.jpg)
2
Changes to ReviewChanges to ReviewBeginning with May/June 2009 Beginning with May/June 2009 MeetingsMeetings
Enhanced Review Criteria for certain mechanisms
Templates for Structured Critiques
Scoring of Individual Review Criteria– All applications will receive criterion scores
from assigned reviewers New 1 to 9 Scoring Scale
2
![Page 3: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/3.jpg)
3
Goals of the ChangesGoals of the Changes
Clearer understanding of the basis of application ratings
More emphasis on impact and less emphasis on technical details
Succinct, well-focused critiques that evaluate, rather than describe, applications
Routine use of the entire rating scale
3
![Page 4: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/4.jpg)
4
Before the Review MeetingBefore the Review Meeting
When reading applications the assigned reviewers should: Address all applicable criteria and
other review considerations Identify major strengths and
weaknesses Assign scores to each of the 5 “core”
criteria Assign an overall impact/priority score
4
![Page 5: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/5.jpg)
5
Preparation of CritiquesPreparation of Critiques
When writing your critiques:
Use bulleted points to make succinct, focused comments
Short narratives may occasionally be appropriate, but should be rare
Focus on major strengths and weaknesses (ones that impacted your overall rating of the application)
5
![Page 6: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/6.jpg)
6
Features of Critique TemplatesFeatures of Critique Templates
Boxes for evaluating:
– Each core review criterion– Other applicable review criteria and
considerations– Overall impact of the application
A box for “advice to applicants”
Hyperlinks to web pages providing descriptions of review criteria and additional review considerations
6
![Page 7: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/7.jpg)
7
Excerpt from a Critique Template: Excerpt from a Critique Template: CriterionCriterion
List major strengths and weaknesses that influenced the overall impact/priority score
Limit text to ¼ page per criterion, although more text may occasionally be needed
Do not enter scores on critiques
1. Significance Please limit text to ¼ page
Strengths
•
Weaknesses
•
![Page 8: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/8.jpg)
8
Excerpt from a Critique Template: Excerpt from a Critique Template: Protected Form Fields and Drop-downsProtected Form Fields and Drop-downs
Protected elements (Drop-down boxes and form fields) are shaded gray
Part of each template is a PROTECTED form
Reviewers should NOT unprotect the forms!
![Page 9: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/9.jpg)
9
Scoring Individual Review CriteriaScoring Individual Review Criteria
There are 5 “core” criteria for most types of grant applications
For example, the core criteria for R01s are:– Significance– Investigator(s) – Innovation– Approach– Environment
Use the 9-point scale (1 = exceptional, 9 = poor) for the five “core” review criteria.
Do not enter scores in the critique9
![Page 10: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/10.jpg)
10
Overall Impact/Priority ScoresOverall Impact/Priority Scores
Consider criterion strengths and weaknesses of each application in determining an overall impact/priority score
Recognize this is a NEW scoring system and focus on the guidelines for its use
This new scoring system is intended to reflect the “real-world” range of the quality of applications typically seen in actual study sections
It is ESSENTIAL that reviewers take advantage of this unique opportunity to use the entire 1 to 9 range 10
![Page 11: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/11.jpg)
Scoring DescriptionsScoring Descriptions
Impact Score Descriptor Additional Guidance on Strengths/Weaknesses
High
1 Exceptional Exceptionally strong with essentially no weaknesses
2 Outstanding Extremely strong with negligible weaknesses
3 Excellent Very strong with only some minor weaknesses
Medium
4 Very Good Strong but with numerous minor weaknesses
5 Good Strong but with at least one moderate weakness
6 Satisfactory Some strengths but also some moderate weaknesses
Low
7 Fair Some strengths but with at least one major weakness
8 Marginal A few strengths and a few major weaknesses
9 Poor Very few strengths and numerous major weaknesses
Non-numeric score options: NR = Not Recommended for Further Consideration, DF = Deferred, AB = Abstention, CF = Conflict, NP = Not Present, ND = Not Discussed
Minor Weakness: An easily addressable weakness that does not substantially lessen impactModerate Weakness: A weakness that lessens impactMajor Weakness: A weakness that severely limits impact 11
![Page 12: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/12.jpg)
12
Before Attending the Review MeetingBefore Attending the Review Meeting
Post critiques to the Internet Assisted Review (IAR) Web module
Enter criterion scores and overall/priority score in IAR
Do not enter scores as part of the critique!
– Ensures better data integrity– Allows scores to be placed where needed
• i.e. Summary Statements, Commons Status
– Makes scores available for future analysis12
![Page 13: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/13.jpg)
13
IAR: New Drop Down for Five Core Criteria Reviewers will see new drop-down menus
in IAR for entering scores for each criterion
New drop-down
![Page 14: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/14.jpg)
14
IAR: Assigned reviewers must submit a critique to upload scores
Reviewers must close the critique file before submitting
![Page 15: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/15.jpg)
15
IAR: Entering Scores and Critiques
Assigned reviewers may not submit Criterion or Preliminary Scores without a critique
– If a reviewer tries to save the criterion and/or preliminary score without uploading the critique, an error message will occur
The maximum file size for a critique is 1 MB
![Page 16: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/16.jpg)
16
IAR: New Header Information in Critique Preliminary IAR Critique now includes
criterion scores
![Page 17: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/17.jpg)
17
IAR: Updating Criterion Scores
Criterion scores can be updated in IAR during the submit phase, edit phase and the final scoring phase
If criterion scores are edited, the PDF of the critique file is regenerated each time because the critique has header information with the criterion scores
– If the criterion scores change, the PDF critique changes
![Page 18: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/18.jpg)
18
IAR: New Popup Listing Criterion Scores New link on List of Applications screen will
display criterion scores for each application
View All Scores
![Page 19: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/19.jpg)
19
At the Review Meeting:At the Review Meeting:Procedure for Discussed ApplicationsProcedure for Discussed Applications
Assigned reviewers will discuss strengths and weaknesses of each application
– Recommend overall impact/priority score– Criterion scores will not be discussed by the
committee
All eligible members will record an overall impact/priority score (as is presently true)
19
![Page 20: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/20.jpg)
20
IAR: Edit Criterion Scores on Voter Sheet Criterion scores can easily be edited by
using the voter sheet
![Page 21: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/21.jpg)
21
After the Review Meeting: After the Review Meeting: Updating Scores or CritiquesUpdating Scores or Critiques
Assigned reviewers whose opinions changed as a result of discussion at the meeting should use IAR:
– To modify their criterion scores– To post revised critiques
If criterion scores are edited, the PDF of the critique file is regenerated
21
![Page 22: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/22.jpg)
22
Summary StatementsSummary Statements
Overall impact/priority scores of discussed applications will be the average of scores voted by all eligible reviewers, multiplied by 10
Final scores will range from 10-90, in whole numbers
Summary statements for ALL applications will include the criterion scores and critiques posted by assigned reviewers
22
![Page 23: 1 REVIEWER ORIENTATION TO ENHANCED PEER REVIEW April 2009 1](https://reader034.vdocument.in/reader034/viewer/2022051819/551aa65f550346856e8b4838/html5/thumbnails/23.jpg)
23
For additional information:For additional information:
Enhancing Peer Review at NIH Web SiteEnhancing Peer Review at NIH Web Site
http://enhancing-peer-review.nih.gov
Thank you for your review service Thank you for your review service
23