Peer Review at the NIH Center for Peer Review at the NIH Center for Scientific ReviewScientific Review

Dan Gerendasy, Ph.D.Dan Gerendasy, Ph.D.Scientific Review OfficerScientific Review Officer

National Institutes of Health

U.S. Department of Health and Human Services

November 5, 2008

Fundamental Tenets for NIH

1. “The only possible source for adequate support of our medical research is the taxing power of the Federal Government . . .“ (Surgeon General Thomas Parran, 1945).

2. Federal government and the politicians must assure complete freedom for the individual scientists in developing and conducting their research work.

3. Reviews should be conducted by outside experts essentially without compensation.

4. Program management and review functions should be separated.

Dual Review System for Grant Applications

Second Level of Review

NIH Institute/Center Council

First Level of Review

Scientific Review Group (SRG)

Center for Scientific Review

Study Section

Institute

Advisory Councils and Boards

Institute Director

Assigns to Institute & Study Section

Reviews for Scientific Merit

Evaluates for Relevance

Recommends Action

Takes Final Action

Principle Investigator

School orResearch

Center

Allocates Funds

Submits Application

Review Process for an NIH Research Grant

Applications are referred to an Institute or Center as the potential funding component:

• Assignment is based on a match between the research proposed and the overall mission of the Institute or Center. For example:

National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK)

National Heart, Lung, and Blood Institute (NHLBI)

National Institute of Biomedical Imaging and BioEngineering (NIBIB)

National Human Genome Research Institute (NHGRI)

• Applications can be assigned to multiple Institutes or Centers.

Assignment to Institutes

• Assignment is based on specific referral guidelines for each IRG.

• Each of the 25 IRGs has 6-10 standing study sections.

Applications are assigned to Integrated Review Groups (IRGs) for review. IRGs are clusters of scientifically related study sections.

Assignment to CSR Integrated Review Groups

Cystic FibrosisRespiratory Sciences IRGRenal and Urological Sciences IRGHealth of the Population IRG 

DiabetesEndocrinology, Metabolism, Nutrition and Reproductive Sciences IRGHealth of the Population IRGDigestive Sciences IRG 

NanotechnologyBioengineering Sciences and Technologies IRGSurgical Sciences, Biomedical Imaging and Bioengineering IRG 

SensorsSurgical Sciences, Biomedical Imaging and Bioengineering IRGBioengineering Sciences and Technologies IRGBiological Chemistry and Macromolecular Biophysics IRG

Example IRGs

• Standing Study Sections when the subject matter of the application matches the referral guidelines for the study section

• Ad Hoc Special Emphasis Panels (SEPs) when the subject matter does not fit into any study section, or when assignment of an application to the most appropriate study section would create a conflict of interest. Also used for special mechanisms (e.g., fellowships, SBIRs, AREAS)

Within an IRG, applications are assigned for review to:

Assignment to CSR Study Sections

CSR Study Sections

• Each CSR standing study section has about 23-40 members who are primarily from academia.

• CSR standing study sections convene face-to-face, or virtual (electronic) meetings.

• As many as 60-100 applications are reviewed by each study section.

Role of the Scientific Review Officer

• Doctoral level scientist with expertise complimentary to science reviewed in their study section.

• Legal responsibility for study section and management of review including the selection of reviewers

• Provides written (summary statement) reports to Institutes and Centers for secondary review.

Designated Federal official with overall responsibility for the review process

Role of Program Officer

• Observes study section meeting

• Provides feedback to investigator about study section results

• Makes recommendations to Institutes/Centers and National Advisory Councils/Boards about funding decisions

This designated federal official has overall responsibility for recommending meritorious applications for funding. The PO:

Role of Study Section Chair

• Partners with Scientific Review Officers to ensure the best scores for the highest quality science

• Guides study section discussion

• Ensures all study section member opinions are given careful consideration

• Manages meeting logistics during the meeting, e.g., timeliness, thoroughness, scientific integrity

How Reviewers Are Selected

• Demonstrated Scientific Expertise

• Doctoral Degree or Equivalent

• Mature Judgment

• Work Effectively in a Group Context

• Breadth of Perspective

• Impartiality

• Interested in Serving

• Balanced Gender, Minority and Geographical representation (for chartered Study Sections)

Sources for Reviewer Recruitment

Research databases (e.g., CRISP, Pub Med, PsychInfo, etc.)

Professional societies o Volunteer Reviewer Registry: Now includes

recommendations from the Science Foundation Ireland and Department of Employment and Learning (N. Ireland).

Scientific community consultation

NIH Program Staff

Conference attendance

Study section chairs

Study section members

Pre-Meeting Review Process

• SRO assigns appropriate reviewers to each application.

• Conflicts of interest identified.

• Applications sent 6 weeks prior to meeting.

• Reviewers post critiques and provisional scores on NIH Web site 3-4 days prior to meeting.

• Reviewers read colleagues’ critiques (excluding conflicts) prior to meeting.

Conflict of Interest (COI)

• Institutional• Family Member/Close Friend• Collaborator• Longstanding Scientific Disagreement• Personal Bias• Appearance of Conflict

This will certify that in the review of applications and proposals by (study section) on (date) , I did not participate in an evaluation of any application or proposal: (1) from any applicant institution or offer or where I am a full- or part-time salaried employee or where I am negotiating for such employment; (2) from any applicant institution or offer or where I have received or could receive a direct financial benefit in relation to the application or proposal under review or have received or could receive a financial benefit from the applicant institution or offer or principal investigator valued at $10,000 or more per year that is unrelated to the application or proposal under review; (3) submitted by a close personal relative, a member of my household, or professional associate, or if such person receives financial benefits from or provides financial benefits to an applicant or offer or. If there was an appearance or real conflict of interest, or (4) any application submitted by my former (within the past year) employer I recused myself from the review of the application/proposal or was granted an appropriate waiver.

Certification of No Conflict of Interest

SIGNATURESSIGNATURES

Certification of Confidentiality and Non-Disclosure

I fully understand the confidential nature of the review process and agree: (1) to destroy or return all materials related to the evaluation; (2) not to disclose or discuss the materials associated with the review, my evaluation, or the review meeting outside of that meeting or with any other individual except as authorized by the Scientific Review Officer (SRO) or other NIH designated official; (3) not to disclose procurement information prior to the award of a contract; and (4) to refer all inquiries concerning the review to the SR0 or other designated NIH official.

SIGNATURESSIGNATURES

Review Process

• Lower half of applications identified, not discussed (“streamlined”)

• Remaining applications discussed (average of 15 minutes/application)

3 Assigned ReviewersConflicts absent from room

• Assigned reviewers establish a range of priority scores, and all present vote.

Review Criteria

• Significance: Does the study address an important problem? How will scientific knowledge or clinical practice be advanced?

• Approach: Are design and methods well-developed and appropriate? Are problem areas addressed?

• Innovation: Are there novel concepts or approaches? Are the aims original and innovative?

• Investigator: Is the investigator appropriately trained?

• Environment: Does the scientific environment contribute to the probability of success? Are there unique features of the scientific environment or subject populations?

Advice to Applicants:Think about how the reviewers will review it

Other Review Criteria

• Protection of Human Subjects• Enrollment of Women, Children and ethnic

minorities• Vertebrate Animal Care• Biohazards• Budget

What Reviewers Look for In Applications

• Significance/Impact – Will it more the field forward?

• Is it exciting?

• Is it clearly written?

• Does the applicant assume I know more than I do?

• Are the Aims & Timelines realistic or are they overly ambitious?

• Does it contain unnecessarily long descriptions of well known methods and approaches?

• Are the study’s limitations discussed?

• Did they bother to proofread the application?

• For each study section, applications in the upper half generally are scored from 1.0-3.0, with 1.0 the best score. Scores as low as 5.0 are possible.

• Individual scores are averaged and multiplied by 100 to give the final priority score.

plus past two meetings

Priority Scores/Percentile Rank

Post Meeting Review Process

• Reviewers have opportunity to revise written critiques based on discussion & reading of colleagues’ critiques.

• SRO writes Resume & Summary of Discussion for scored applications’ summary statements. Unscored applications receive critiques only.

• Summary statements available within 30 days of meeting (10 days for new investigators).

Summary Statement

• Summary of Review Discussion• Essentially Unedited Critiques• Budget Recommendations• Administrative Notes• Priority Score and Percentile Ranking

Results are documented by SRO in a summary statement and forwarded to the PI and the assigned NIH Institute or Center, where a funding decision is made.

The Summary Statement Contains

What Determines Which Awards Are Made?

• Scientific merit

• Program or public health priorities

• Availability of funds

Center for Scientific Review

Study Section

Institute

Advisory Councils and Boards

Institute Director

Assigns to Institute & Study Section

Reviews for Scientific Merit

Evaluates for Relevance

Recommends Action

Takes Final Action

Principle Investigator

School orResearch

Center

Allocates Funds

Submits Application

Review Process for an NIH Research Grant

http://www.csr.nih.gov/video/video.asp

CSR has developed a video of a mock study section meeting to show how NIH grant applications are reviewed.

Inside the NIH Grant Review Process Video