Objective Review Training 2009 Sponsored by Grants Policy Staff/OMS

Michelle G. Bulls, Director

Presenter:Ms. Tammy G. BagleySenior Grants Policy AnalystGrant Policy Staff, OMS/IHS(301) 443-7172

Objective Reviews What is an Objective Review?

• An Objective Review is a process that involves the thorough and consistent examination of applications based on an unbiased evaluation of scientific or technical merits or other relevant aspects of the proposal.

Purpose of an Objective Review

• To provide a fair and credible forum for making sure that only the applicants selected for funding offer the greatest potential for furthering the program’s goals, objectives and purpose.

Outcomes and Other Purposes for the Review Process

• Provide an evaluation on the merits of an application submitted based on the criteria/guidelines provided in the IHS Grants Funding Opportunity Announcement.

• Provide recommendations for possible funding of the grantee application to the IHS Program Staff.

• Provide technical assistance to potential IHS grantee’s on the strengths and weaknesses of their proposed project (without re- writing the proposal for them).

• ORC’s must be viewed as credible and fair.

• Any circumstance that might introduce a conflict of interest, or the appearance thereof of prejudices, biases or predispositions into the process must be avoided.

• ORC’s are performed by experts in the field of endeavor for which support is requested.

PolicyAAGAM Chapter 2.04.104-C

ORC Members Must consist of the following persons:

Chairperson – Presides over the ORC meeting, a non-voting member of the ORC, prepares the Executive Summary Statements.

Program Official –Serves as an advisor on programmatic related issues.

NOTE: PO’s are not allowed to provide any information that would create a bias in the minds of the reviewer’s. No comments on personal knowledge of the applicants can be shared.

Grants Management Official 1. Provides orientation to the ORC, 2. Secures all score sheets, critiques, and minority reports at the

end of each review, daily, 3. Ensures that the ORC is a fair and creditable process and,4. Serves as an advisor on grants management related issues.

ORC Members con’t

The panel must consist of a minimum of at least 3 people.

Primary Reviewer:• Introduces the application to the ORC, • Reads key aspects of the project, including the highlights of the

project narrative,• Serves as the lead reviewer for the application,• Conducts a thorough critique of all strengths and weakness of

the application and, • Presents them first to the ORC. • Most knowledgeable of all aspects of the proposal.

.

ORC Members con’t

Secondary Reviewer:• Reviews the entire application and,• Participates as a discussant based on their critiques,• The open discussion should be geared around comments

on any additional areas of the application that may not have been presented by the Primary Reviewer. Clearly the discussions must focus on strengths and weaknesses of the application.

Tertiary Reviewer: • Offers a third opinion of the application and,• Describes the key points of the proposal that were not

mentioned by the primary and secondary reviewers.

IMPORTANT NOTE: Tertiary reviewers are not required.

The cognizant program office officials CANNOT serve as reviewers on proposals

funded out of their respective offices.

A Federal employee may be from an office within the Indian Health Service considering the application for

funding other than the cognizant program office.

AAGAM 2.04.104C-6-2

Example:

OPHS- Apples and Oranges Health Review

* An OPHS staff person cannot serve as a reviewer*

Federal Employees

Contents of the Reviewer’s PackageAAGAM Chapter 2.04.104C-7 (D)

• Instructions for Reviewers• Conflict of Interest and Confidential Form• Written Agreement of the Scope of Work• Confidential Financial Disclosure Report (OGE-450) for Federal

Reviewers• IHS Program Announcement • Reviewer’s Score Sheets• Reviewer’s Critique Template • Objective Review Minority Report Form• Objective Review Roster• Review Panel Application Assignments• Review Meeting Order of Review

ApplicationsAAGAM Chapter 2.04.104C-7D-1

• All reviewers should receive a copy of each application. This means even the applications where the reviewer does not serve as the primary, secondary, or tertiary on the application.

Order of ORC Meeting

• Meeting is called to order by the Program Official.

– Brief description of the program that is up for review;

– Expresses appreciation for those that will serve on the review panel;

– Shares the number of applications that are up for review;

– Introduces the Chairperson.

Order of the ORC Meeting

• Chairperson

– Calls for a brief introduction of each Review Member.

– Goes over the ORC protocol for conducting the meeting briefly, discusses breaks, timelines for reviewing applications and presides over the ORC process.

– Keeps track of which reviewer had a Conflict of Interest, for which grant.

– Turns over the meeting to the GMS for the Orientation.

Order of the ORC Meeting

• Grants Management Official conducts the Orientation

– Describes the roles of each participant within the review process,

– Discusses the Conflict of Interest Forms and collections all signed forms,

– Reminds ORC members to adhere to the program announcement criteria,

– Discusses scoring confidentiality -individual scores, collection of score sheets;

– Discusses Critiques and Minority Reports,– Reminds reviewers to dispose of all applications after the ORC

has concluded,– Discusses the confidentiality of the review process after the

ORC ends.

Ready to Start?

• Chairperson

– Announces application to be reviewed,– Ask the panel, if there are any conflicts of interest,– Persons with conflicts must excuse themselves from the room,– Requests primary, secondary, and tertiary (as appropriate)

reviewer to summarize the project narrative and present strengths and weaknesses of the application,

– Second reviewer presents, strengths and weaknesses,– Panel discussion, for clarity if needed,– Call for a motion of approval or disapproval,– Motion is seconded by a panel member,– Votes are taken on the motion,– Reminds reviewers to enter individual scores on their score

sheet.

**Panel members should express their range of scoring based on the scoring chart provided to them by the Program Office.

Individual scores are confidential and should not be shared with the panel at all.

For example the Primary Reviewer might say: After careful review of this application, I feel that it falls in the fundable range

of 80 to 100.

**Reviewers must make sure that their numeric scores support the strengths and weakness outlined in their critiques.

Note: If the reviewer gives the application a low score, then his/her critique must reflect numerous weaknesses to support the low score given. (same for scoring the application on the high end)

Scoring Applications

Scoring Applications (con’t)

• Reviewers cannot re-write the proposal. It must be scored as is.

• Reviewers can only make recommendations for improvement in their critiques, but this will not change the score. This is designed to highlight the weaknesses of the application.

• Reviewers should not add to or disaggregate the project in order to approve it. They should approve it and or make recommendations .

• Approval or Disapproval should be based on the “CONTENTS OF THE APPLICATION ONLY” in line with the criteria outlined in the Funding Opportunity Announcement.

Scoring Applications (con’t)

• Applications - In accordance with HHS policy (AAGAM Chapter 2.04.104C -

7G), all applications that are reviewed must be scored unless the program is using a streamlined method for voluminous applications.

• Disapproved applications– Disapproved applications “receive low scores that are outside of

the fundable range” or unsatisfactory rating.

• If a reviewer is out due to a “Conflict of Interest”, they should make note of that on their score sheets.

Example:

– ‘COI’ if out due to a Conflict or Interest

Scoring Applications (con’t)

• The Chairperson is responsible for making sure all reviewers place a score on their score sheets.

• If using individual score sheets, all sheets should be given to the chairperson before moving to the next application for review.

• If using a master score sheet, the Chairperson will need to collect each master sheet from each reviewer voting at the end of the day and provide them to the Grants Management Official present.

Supposition

• Information contained in the grantee application should be the ONLY basis for reviewers scores.

Reviewers cannot score applications based on speculation, or assumption. Applications must be scored based on the:

•Program Announcement Guidelines•Facts stated in the actual application•Application instructions

Critiques

Each reviewer should pass their critiques to the Chairperson for

collection before moving on to the next application for review.

* Note: if a critique or minority report needs to be written or revised, the Chairperson should note who is responsible for this action and make sure the reports are completed by the end of the day and turned into him/her to be given to the Grants Management Official.

Minority Reports

Minority Reports are required when two or more reviewers disagree with the majority.

Reviewers are required to complete a minority report stating their comments and concerns.

Reports must be signed by all reviewers making comments.

Minority report comments will be placed in the summary statement and shared with the applicant.

Conclusion of the ORC

Housekeeping

*Chairperson- Gives the Grants Mgmt Specialist all– Score sheets – Critiques– Minority Reports

• All applications should be destroyed.

*Reviewers are reminded that all discussions of the ORC are confidential and should not be shared outside of the ORC meeting.

Executive Summary

• Once the ORC is concluded, the GMS will copy all critiques, and minority reports, and give them to the Chairperson.

• The program official must not write the Executive Summary; however, they may designate the Chairperson to write the summary. The goal is to not have the program perform this function to maintain separation of duties and avoid conflicts.

• To avoid the appearance of a conflict, the program office should not be in possession of the reviewers scores or critiques without first obtaining them from the Grants Management Staff.

Reviewer Payments

• Program Official must contact :

Mr. Dale Burson

Division of Acquisitions and Policy

206-615-2456

Regarding payment for reviewers. Either by :

P-Card or,

Cashier’s Check

** Note: the recommended threshold for IHS reviewers is $250.