reviewer manual for scientist reviewers funding program ......reviewer manual for scientist...

Reviewer Manual

for Scientist Reviewers

Funding Program: Treatment of Multiple Sclerosis

Review Cycle: Cycle 3, 2015

Relevant Dates: o February 26th 2016 - March 7th 2016: Reviewers will designate whether they have

a conflict of interest (COI) and their experience/expertise level for each application. o March 11th, 2016: PCORI will notify reviewers of their assigned applications. o March 21st, 2016: One complete written critique (for one application) is due in

PCORI Online or to the MRO. o April 13th, 2016: Written critiques and scores for all assigned applications are due

in PCORI Online. o May 19th-20th, Washington DC metro area: In-person Merit Review Panel

Meeting o July 2016: Funding decisions made

PCORI Cycle 3 2015 Merit Review Cycle

Table of Contents Chapter 1. Setting the Stage ......................................................................................................................... 4

PCORI’s Mission and Strategic Goals ........................................................................................................ 4

Comparative Clinical Effectiveness Research ........................................................................................... 4

Effectiveness versus Efficacy ..................................................................................................................... 4

Decisional Dilemma, or Evidence Gap ...................................................................................................... 5

Dissemination Goals.................................................................................................................................. 5

Usual Care ................................................................................................................................................. 6

Categories of research PCORI does not fund ............................................................................................ 6

Cost-Effectiveness Analysis (CEA) ......................................................................................................... 6

Development of Decision Aids .............................................................................................................. 6

Other categories of research PCORI does not fund .............................................................................. 7

Chapter 2. Treatment of Multiple Sclerosis .................................................................................................. 8

Treatment of Multiple Sclerosis research questions ................................................................................ 8

Types of Studies for Treatment of Multiple Sclerosis PFA ........................................................................ 8

PCORI is not Interested In ......................................................................................................................... 9

Methodology Standards............................................................................................................................ 9

MS applications and programmatic responsiveness .............................................................................. 10

Chapter 3. Merit Review ............................................................................................................................. 11

Merit Review Goals ................................................................................................................................. 11

Preliminary Review Overview ................................................................................................................. 11

Merit Review Criteria .............................................................................................................................. 11

Criterion 1. Potential for the study to fill critical gaps and generate actionable evidence ................ 12

Criterion 2: Potential for the study findings to be adopted into clinical practice and improve delivery of care ................................................................................................................................................. 12

Criterion 3. Scientific merit (research design, analysis, and outcomes) ............................................. 13

Criterion 4. Patient-centeredness ....................................................................................................... 13

Criterion 5: Patient and stakeholder engagement ............................................................................. 14

Understanding Patient-Centeredness vs. Patient Engagement.............................................................. 15

Where to find good examples of Patient and Stakeholder Engagement ............................................... 15

Human Subjects Protections ................................................................................................................... 16

Evaluating Resubmissions ....................................................................................................................... 16

Evaluating Budgets .................................................................................................................................. 17

2


Chapter 4. Critique Writing ......................................................................................................................... 18

Written Critiques Overview .................................................................................................................... 18

Offline critique template......................................................................................................................... 18

Guidelines for Writing Strengths and Weaknesses................................................................................. 18

Major and Minor Strengths and Weaknesses ......................................................................................... 19

Examples: Writing Strengths ................................................................................................................... 19

Examples: Writing Weaknesses .............................................................................................................. 20

Merit Review Scores ............................................................................................................................... 21

Writing an Overall Narrative ................................................................................................................... 22

Assigning an Overall Score ...................................................................................................................... 22

Revising and Resubmitting a Critique ..................................................................................................... 23

Chapter 5. In-person panel meeting ........................................................................................................... 24

In-Person Panel Review Overview .......................................................................................................... 24

Preparing for the In-Person Panel Review .............................................................................................. 24

Panel and Reviewer Roles ....................................................................................................................... 24

Inside the In-Person Panel Review Video ............................................................................................... 25

Preparing In-Person Panel Presentations ............................................................................................... 25

Panel Review Discussion Tips .................................................................................................................. 26

Appendix 1. COI/Expertise .......................................................................................................................... 28

Appendix 2. Methodology Standards ......................................................................................................... 33

Appendix 3. Engagement Rubric ................................................................................................................. 46

Appendix 4. Human Subjects checklist ....................................................................................................... 51

Appendix 5. Offline Critique Template ....................................................................................................... 53

3


Chapter 1. Setting the Stage PCORI’s Mission and Strategic Goals

Comparative Clinical Effectiveness Research Per its authorizing legislation, PCORI funds research that supports comparative clinical effectiveness research (CER)—comparing health outcomes and the clinical effectiveness, risks, and benefits of two or more approaches to healthcare. For example: Which option works better to improve patient outcomes for low-back pain, medication management or surgery?

Effectiveness versus Efficacy: PCORI is interested in Comparative Effectiveness Research—not studies on efficacy.

• Effectiveness is the extent to which an intervention does more good than harm across a broad mix of patients in a range of clinical settings. PCORI seeks to fund studies that compare different treatment options in real-world environments. These real-world environments are more likely to include different medical care

CER research answers patient-centered questions, such as: • Given my personal characteristics, conditions, and preferences, what should I expect

will happen to me? • What are my options, and what are the potential benefits and harms of those options? • What can I do to improve outcomes that are most important to me? • How can clinicians and the care delivery systems they work in help me make the best

decisions about my health and health care?

PCORI helps people make informed healthcare decisions, and improves healthcare delivery and outcomes, by producing and promoting high-integrity, evidence-based information that comes from research guided by patients, caregivers, and the broader healthcare community. PCORI’s Strategic Goals:

• Increase quantity, quality, and timeliness of useful, trustworthy research information available to support health decisions

• Speed the implementation and use of patient-centered outcomes research evidence • Influence research funded by others to be more patient-centered

PCORI seeks to fund studies that will produce information that allows patients to weigh the benefits and risks of clinical alternatives. This will ensure that people receive care according to their needs and have the opportunity to achieve the best possible health outcomes.

4


settings and patient characteristics. Comparative effectiveness research includes studies where two readily available treatments or interventions are compared with one another, rather than with a placebo or no treatment. These types of studies require a much broader patient population to be able to account for differences among patients and settings, and for the smaller differences researchers are likely to find when comparing two or more active treatments. Effectiveness trials should include comorbidities, variable adherence rates, and the presence of other medication.

• In contrast, efficacy is the extent to which an intervention does more good than harm in ideal

patients under ideal circumstances. Efficacy studies show results in an ideal, controlled world, typically compared to placebo or no active treatment. PCORI is not interested in funding efficacy research because the results from efficacy trials have limited generalizability beyond the trial settings and conditions. Challenges associated with the everyday care of patients and the healthcare choices they make tend to affect how well the treatments perform under real-world conditions. PCORI’s legislative mandate specifically prohibits the funding of efficacy research.

Decisional Dilemma, or Evidence Gap The applicant should support the importance of the research topic or question by demonstrating an evidence gap or decisional dilemma.

Decisional dilemma – describes the difficulty the patients, clinicians, and other stakeholders have in understanding treatment options for a specific condition. Decisional dilemmas can be identified through discussions with patients and other stakeholders, or through careful review of the scientific and medical literature on the topic.

Evidence gap – describes an area of missing information that would help patients, clinicians, and other stakeholders make better decisions about health care. Evidence gaps are usually identified through systematic research reviews that demonstrate unclear or incomplete guidelines. Another way a principal investigator may indicate they are looking to fill an evidence gap is by documenting that the evidence gap is high priority as recommended by research, clinical, and/or stakeholder (i.e., decision-maker) groups (such as the Institute of Medicine and AHRQ), including specific recommendations for CER.

Applicants should also document that the existing evidence of efficacy or effectiveness is insufficient to guide current clinical decisions. Dissemination Goals Dissemination and Implementation refer to the processes of enhancing the awareness of new research evidence, and speeding the integration of this evidence into practice.

PCORI is interested in robust research findings that can be rapidly disseminated and implemented in clinical and community practice, thus facilitating improvements in patients’ and other stakeholders’ healthcare decision making.

Applicants are expected to describe the potential of research results for dissemination and implementation. We encourage applicants to think creatively about how to disseminate findings.

5


Traditional dissemination, through research articles or scientific presentations, are unlikely to reach the range of stakeholders who are faced with making clinical decisions.

Usual Care PCORI funds research that compares at least two alternative treatments or interventions, both of which must be available in the real world. Sometimes one of the approaches that is compared is labeled “usual care.” If the researchers are proposing “usual care” they should first justify that choice as the best comparison for the particular condition or intervention. Further, to be an appropriate comparison condition in PCORI research, usual care needs to be carefully described and measured in the study, and be a realistic choice faced by patients and other stakeholders. The clinical characteristics must be specified, and applicants must provide a persuasive rationale for using it a comparator (e.g., it reflects the standard of care, etc.). Categories of research PCORI does not fund: Cost-Effectiveness Analysis (CEA) CEA examines both the costs and health outcomes of alternative intervention strategies, and the results typically are presented in the form of costs per particular health outcomes or life year saved, etc. Per its founding legislation, PCORI may not fund studies that:

• Include formal cost-effectiveness analysis (CEA) or • Directly compare the costs of care between two or more alternative approaches to providing care.

Any applications that propose to conduct CEA are deemed nonresponsive and are not reviewed.

Please alert your Merit Review Officer if you encounter cost-effectiveness analysis in your application review.

Development of Decision Aids Decision aid tools are meant to help patients and their caregivers facing complex decisions. Aids do not replace providers, but function to support a team-approach to decision-making between the patients and their healthcare providers. These tools help prepare the patient for consultations with their providers by increasing health literacy for the patient and help clinicians limit decision conflict.

PCORI does have an interest, however, in studies that address questions about conditions that lead to high costs to the individual or to society. This includes studies that:

• Examine the effect of costs on patients, such as patients’ out-of-pocket costs, hardship or lost opportunity, or costs as a determinant of or barrier to access to care.

• Address cost-related issues, such as the resources needed to replicate or disseminate a successful intervention. • Evaluate interventions to reduce health system waste or increase health system efficiency.

6


Although PCORI is not currently interested in funding the development or testing of decision aids, if the application describes how their research could be used in decision making or later guideline development, that is acceptable. Other categories of research PCORI does not fund: Consistent with PCORI's authorizing law, PCORI does not fund research whose findings will include:

• Creation of clinical practice guidelines or instruments to measure outcomes • Insurance coverage recommendations • Payment or policy recommendations • Establishing efficacy for a new clinical strategy or in a tightly controlled environment • Study of the natural history of disease or comparison of patient characteristics rather than

treatment strategies • Fundamental science or study of biological mechanisms, or how medications work in the body

Please alert your Merit Review Officer if you encounter these topics as aims in the applications you review.

Examples of decision aids could include:

⦁ Brochures ⦁ Audiovisual materials ⦁ Educational sessions ⦁ Websites ⦁ Counseling sessions ⦁ Computer programs

7


Chapter 2. Treatment of Multiple Sclerosis PCORI informs the research community of opportunities to apply for research contracts via PCORI Funding Announcements (PFAs).

Reading your assigned PFA will help provide context for the applications that you review. The Treatment of Multiple Sclerosis (MS) PFA seeks to fund randomized clinical trials (RCTs) or observational studies that compare two or more alternatives for the treatment of multiple sclerosis, with a focus on the effects of therapies on the symptoms experienced by patients with MS and on quality of life and functional status. The PFA specified three areas of research that are of interest: comparisons of the effects of disease-modifying therapies (DMTs) and DMT-based strategies, of non-DMT therapies aimed at specific symptoms, and of telerehabilitation versus conventional direct care, on functional status, fatigue, and quality of life.

Through the Treatment of Multiple Sclerosis PFA, PCORI seeks to fund research that addresses one of the following questions:

Please click here to access the Treatment of Multiple Sclerosis PFA. Types of Studies for Treatment of Multiple Sclerosis PFA You might be asked to review studies that are head-to-head comparisons of two or more different treatment approaches to Multiple Sclerosis. Alternatively, you may be asked to review a project that compares two or more alternative care delivery strategies for management of MS symptoms.

• What are the comparative benefits and harms of different DMTs or therapeutic strategies in patients with relapsing, remitting multiple sclerosis on symptoms, functioning, quality of life, disease activity, and disease progression? Strategies may include comparisons of initial DMT treatment or comparisons of follow-on treatments in patients for whom initial DMT treatment has failed, including strategies for sequencing or combining agents, changing to a different DMT, or escalating DMT dose.

• What are the comparative benefits and harms of different approaches, other than DMTs, for ameliorating important symptoms in people with MS? Symptoms of interest include fatigue, difficulty walking, memory or attention problems (cognition), bladder problems, numbness or tingling, and pain. Studies of patients with progressive forms of MS are of particular interest.

• What is the comparative effectiveness of telerehabilitation versus conventional direct care interventions for improving outcomes in people with MS, such as functional status, fatigue, and quality of life?

o Studies should evaluate the effectiveness of telerehabilitation interventions to enhance community-based primary care or neurology practice for patients who do not have access to specialty centers. Applications that employ intervention(s) already in practice are especially attractive.

o Studies should examine the impact of the telerehabilitation strategies in various subpopulations, including individuals with low socioeconomic status and patients with progressive disease.

8

http://www.pcori.org/sites/default/files/PCORI-PFA-2015-Cycle-3-Multiple-Sclerosis.pdfhttp://www.pcori.org/sites/default/files/PCORI-PFA-2015-Cycle-3-Multiple-Sclerosis.pdf


PCORI is not Interested In: The Treatment of Multiple Sclerosis PFA is not interested in studies that:

• Test efficacy within a tightly-protocol-controlled research setting, as opposed to more real- world, pragmatic CER

• Conduct a formal cost-effectiveness analysis • Directly compare the costs of care between two or more alternative approaches to providing

care • Primarily focus on the natural history of disease or proposes instrument (for data collection)

development, pharmacodynamics, and fundamental science or study of biological mechanisms • Evaluate new or existing decision support tools; this includes the development and evaluation of

a decision support or shared decision tool or system for patients, clinicians, or both patients and clinicians

• Develop clinical prediction or prognostication tools

Applications that include these topics were actively discouraged in the funding announcement and during the letter of intent review process. However, if you identify one of these topics in an application you are reviewing, let your merit review officer know right away. You should continue to review the application while the merit review officer investigates the issue.

Methodology Standards Your PFA includes a section on Methodological Considerations. The PCORI Methodology Committee was tasked with setting standards that describe scientifically sound methods to be used by all PCORI awardees, which resulted in the PCORI Methodology Standards. Please click here to read more about the Methodology Standards.

Why do methods matter to PCORI? • Rigorous methods ensure that studies produce trustworthy information that can be used to

improve healthcare outcomes. • Methods describe how researchers collect data, administer interventions, and analyze results.

Scientist reviewers can help PCORI evaluate adherence of investigators’ proposed research to the Methodology Standards. Chapter 3 will revisit the Methodology Standards as they relate to specific Merit Review criteria.

9

http://www.pcori.org/research-results/research-methodology/pcori-methodology-standardshttp://www.pcori.org/research-results/research-methodology/pcori-methodology-standards


All MS applications undergoing Merit Review are programmatically responsive:

10


Chapter 3. Merit Review Merit Review Goals The goals of PCORI’s Merit Review process are to:

• To identify applications that have the strongest potential to provide useful evidence that helps patients, caregivers, clinicians and other stakeholders make informed decisions to improve patient outcomes;

• To implement a transparent, fair, objective, consistent process to identify these applications;

• To elicit high-quality feedback that reflects a diversity of perspectives to ensure that the research funded by PCORI reflects the interests and views of patients and those who care for them and that it meets the criteria for scientific rigor;

• To identify projects that fill important evidence gaps and have strong implementation potential; and

• To regularly evaluate and continually improve the merit review process and policies in support of PCORI’s mission.

Preliminary Review Overview

Returning reviewers should bear in mind that PFAs are revised often, so they should be sure to thoroughly read the PFA for the applications they are reviewing.

For the Treatment of Multiple Sclerosis funding announcement reviews, patients, scientists, and other stakeholders evaluate applications against all five criteria and human subjects’ protections. Each application is reviewed by two scientists, one patient, and one stakeholder reviewer. Merit Review Criteria Scientist reviewers are required to respond to all five merit review criteria as well as to comment on human subjects protections.

Reviewers should evaluate the strengths and weaknesses of each application based on PCORI’s Merit Review criteria. In your critique, evaluate the application’s adherence to the themes indicated by the bulleted questions under each criterion.

All reviewers should begin the Preliminary Review by reading the full PCORI Funding Announcement (PFA) for the applications they are reviewing, to make sure they understand PCORI’s programmatic and organizational goals. They then carefully evaluate their assigned applications according to PCORI’s Merit Review criteria.

For each of their assigned applications, reviewers will: • Write a critique highlighting the application’s strengths and weaknesses • Assign scores for each of the criteria that align with their written critiques • Write a summary evaluating the application as a whole • Provide an overall score for the application

11


Criterion 1. Potential for the study to fill critical gaps and generate actionable evidence The proposal should address the following questions:

• Does the application convincingly describe clinical burden? • Does the application identify a critical gap in current knowledge as noted in systematic reviews,

guideline development efforts, or previous research prioritizations? • Does the study identify variations in practice patterns that suggest clinical uncertainty? • Does the application describe the decisional dilemmas experienced by patients and other

stakeholders that this study would address? • Does the study/application have the potential to fill these evidence gaps and inform decision

making for key stakeholders (provide example)?

The key considerations are whether or not the applicants make the case that the question they are addressing is one that is of interest to stakeholders in the real world. Is there a true decisional dilemma faced by patients, caregivers, providers, and/or other decision-makers about how to approach a specific disease or condition? Is there a gap in the available research evidence that needs to be filled?

Criterion 2: Potential for the study findings to be adopted into clinical practice and improve delivery of care The application should describe how evidence that is generated from this study could be adopted into clinical practice and delivery of care by others. The application should address the following:

• Does the application identify potential end-users of study findings, such as local and national stakeholders, and does it incorporate strategies to engage these end-users in dissemination of outcomes? Does the application provide information that supports a demand for this kind of a study from end-users?

• How likely is it that positive findings could be reproduced by others, resulting in improvements in practice and patient outcomes? Identify the potential barriers that could hinder adoption of the intervention by others, including generalizability to other health systems or treatment settings, or complexity of the intervention, as applicable.

• Does the application describe a plan for how study findings will be disseminated beyond publication in peer review journals and national conferences?

• Can the study be readily adopted in other settings with minimal adaptations or complexities?

When reviewing your applications for dissemination potential, remember that applicants: • are asked to describe the potential for dissemination and implementation of their research

findings, but • are not expected to disseminate and implement findings during the research period covered by

PCORI funding. For research that produces important findings, subsequent applications to support dissemination and implementation efforts may be submitted for consideration for funding under separate funding announcements.

12


Criterion 3. Scientific merit (research design, analysis, and outcomes) The application should show sufficient technical merit in the research design to ensure that the study goals will be met.

• Does the proposal describe a clear conceptual framework to anchor the background literature and inform the design, key variables, and relationship between interventions and outcomes being tested?

• Does the application provide justification that the outcome measures are validated and appropriate for the population?

• Does the research plan describe rigorous methods that demonstrate adherence to PCORI’s Methodology Standards?

• Are each of the comparators (e.g., active intervention arm and comparator arm) clearly described and well justified? If usual care is one of the arms, is it sufficiently justified and will it be sufficiently measured?

• Are the sample sizes and power estimates based on careful evaluations of the anticipated effect size? Is the effect size adequately justified in relation to the size or dose of the intervention and the research design (e.g., cluster randomized design)?

• Is the study plan feasible? o Is the project timeline realistic, including specific scientific and engagement milestones? o Are planned start-up times realistic, including training of personnel? Have the investigators

considered and addressed the potential barriers to study initiation within the targeted clinical setting?

o Is the strategy for recruiting participants feasible? o Are assumptions about participant attrition realistic and are plans to address patient or site

attrition adequate?

Address each set of bulleted questions, taking special care to consider whether weaknesses are easily fixable, and whether the research design adequately addresses the study aims.

Criterion 4. Patient-centeredness The application should demonstrate that the study focuses on improving patient-centered outcomes and employs a patient-centered research design (i.e., design is informed or endorsed by patients). (Note: a study can be patient-centered even if the end-user is not the patient, as long as patients will benefit from its information.) The proposal should address the following:

• Does the application include a thorough description about which outcomes (both benefits and harms) are important to patients? Are those outcomes included in the study plan?

• Are the interventions being compared in the study available to patients now and are they the best options for comparison (including whether they would be chosen by patients and their healthcare providers for managing the condition being studied)?

Remember that a project can be patient-centered even if the outcomes are not patient reported, or if the interventions are not at the patient level. For instance, an application may be comparing two different clinic structures on how much time providers have to spend with patients. The outcomes involve provider visit time and provider satisfaction. However, the research question may stem from patient complaints that they do not have enough time to talk to their providers during visits. Thus, the application might still be patient centered.

13


Criterion 5: Patient and stakeholder engagement The proposal describes plans for the engagement of and collaboration with relevant stakeholders (e.g., patients, caregivers, clinicians, hospitals and health systems, payers [insurance], purchasers [business], industry, researchers, policy makers, and training institutions) in the conduct of the study. PCORI understands that applicants may not have the resources to establish formal partnerships prior to contract award, but expects applicants to discuss in their application their plans to work with PCORI to create the types of partnerships with national and regional patient and other stakeholder groups that will contribute to refinement of research questions, outcomes, protocols, and study conduct and dissemination.

Successful applicants shall plan to work in collaboration with PCORI staff upon award of the proposed studies to establish a project Study Advisory Committee (SAC) that is comprised of national or regional organizations that represent, at a minimum, patients and families with lived experience, relevant clinicians, payers, and health plans. Other representation may be recommended in collaboration with PCORI, including individual patients with lived experience and other relevant stakeholders, including scientific and methodological experts. The SAC serves to advise and assist the research team with further refinement of the study questions, outcomes, and protocol. It is expected that the SAC will meet regularly in person at least two times per year and may use virtual communications at other times.

These are to be budgeted activities and are to be represented in the project milestones. The proposal should address the following:

• Does the application provide a well-justified and comprehensive description of plans to build an interdisciplinary study team that includes appropriate patient and stakeholder representation?

• Are the plans for a strong partnership among scientists, patients, and others throughout the entire research process (e.g., finalizing questions, identifying outcomes, monitoring study, dissemination, and implementation) appropriate and tailored to the study?

• Are the scope, form, and frequency of patient and stakeholder involvement planned throughout entire research process sufficient to support the study goals?

• Are the roles and the decision-making authority of all study partners clearly described? • Are the organizational structure and resources appropriate to carry out the project?

Successful applicants are expected to eventually partner with national patient and other stakeholder organizations to make sure that the research they are conducting is meaningful and will be useful to real-world decision-makers. The plan for working with such groups should be described in detail, including frequency of meetings and expected outcomes throughout the timeline. PCORI understands that engagement structures and approaches vary widely. Other engagement approaches such as forming stakeholder groups, panels, task forces, working groups, and other bodies, and/or involving individual patient and stakeholder partners in various ways are also permissible to employ, either in addition to or instead of the formation of the SAC. The SAC provision is not meant to require that a separate governance and/or advisory entity must be established beyond the study governance and advisory structure the awardee has planned, if an applicant already has an approach for including the relevant, required stakeholders and patient partners.

14


Understanding Patient-Centeredness vs. Patient Engagement When evaluating Criteria 4 and 5, it is important that reviewers know the difference between patient-centeredness and patient and stakeholder engagement.

• “Patient-centeredness” means:

o The project aims to answer questions or examine outcomes that matter to patients o Research questions and outcomes should reflect what is important to patients and

caregivers • “Patient and stakeholder engagement” means:

o Patients are partners in research, not just “subjects” o Active and meaningful engagement between scientists, patients, and other stakeholders o Community, patient, and caregiver involvement already in existence or a well-thought-out

plan Patient-centeredness focuses on the importance of the research questions and outcomes to patients, while patient and stakeholder engagement focuses on how patients and other stakeholders are involved in the research.

Where to find good examples of Patient and Stakeholder Engagement To help explain what the phrase ‘engagement in research’ means, PCORI worked with its Advisory Panel on Patient Engagement to develop the Engagement Rubric. The rubric was created using promising engagement practices from previous funding cycles, and is meant to be a tool for applicants to formulate their Engagement Plans, not as a scoring rubric for reviewers to evaluate applications. Reviewers should not expect all applications to match the illustrated examples in the rubric. About the Rubric

• Provides some real examples from PCORI-funded projects of options—not strict guidelines—for incorporating engagement into the research process

• Is not intended to be prescriptive and comprehensive

Patient-centered research should strive to answer the following questions patients frequently ask when making health decisions:

1. “Given my personal characteristics, conditions, and preferences, what should I expect will happen to me?”

2. “What are my options, and what are the potential benefits and harms of those options?”

3. “What can I do to improve the outcomes that are most important to me?” 4. “How can clinicians and the care delivery systems they work in help me make

the best decisions about my health and health care?”

15


Elements of the Rubric

• Planning the Study: How might patients be involved in formulating the research question to be studied or in designing various elements of the study?

• Conducting the Study: What are the ways patients and other stakeholders could participate in and monitor the conduct of the project?

• Disseminating the Study Results: How might patients and other stakeholders help plan for and participate in dissemination?

The rubric also defines the six PCOR Engagement Principles reciprocal relationships, co-learning, partnership, trust, transparency, and honesty. See Appendix 3 for the full Engagement Rubric.

Human Subjects Protections PCORI requires that research involving human subjects include adequate safeguards, and Institutional Review Boards selected by awardees have authority for ensuring the protection of human subjects.

After you have evaluated all five Merit Review criteria, please evaluate the adequacy of human subjects protections in your assigned applications.

• PCORI requires that research involving human subjects include adequate safeguards • Institutional Review Boards selected by awardees have authority for ensuring the protection of

human subjects • PCORI seeks your assistance in identifying issues with protection of human subjects that PCORI

staff should review with potential funding awardees • Concerns about protections for human subjects should not be factored into the application’s

score • Flag these concerns for PCORI staff by checking the appropriate box and providing your

comments in PCORI Online See Appendix 4 for detailed guidance for evaluating Human Subjects Protections.

Evaluating Resubmissions What is PCORI’s resubmission policy?

An applicant may resubmit an application that was not funded. An application is considered a resubmission if it has previously completed PCORI’s merit review process, including receipt of the summary statement.

When resubmitting an application, applicants include a resubmission letter with their revised application.

The resubmission letter is an opportunity for applicants to provide an overview of how the application has been strengthened in its scientific merit and responsiveness to the current PFA. Simply responding to previous reviewers’ concerns is not sufficient; the application must be programmatically responsive and demonstrate methodological rigor and patient-centeredness.

16


TIP: Review the resubmission statement letter after reading the application to better understand the changes the investigators made to their original application.

Evaluating Budgets All applicants submit a detailed budget and justification for the duration of the proposed study. The budget reflects the work outlined in the application and must support all objectives.

If a project is awarded, the applicant moves into the post-award phase. Budgets are carefully scrutinized to ensure that the proposed budget is appropriate for the size and scope of the study and does not have any unallowable costs. Since PCORI awards contracts, not grants, negotiations between the awarded organization and PCORI can work out issues in the budget or projected project milestones.

Reviewers do not need to comment or closely analyze the budgets. However, as you look over the application, please flag any issues with the proposed budget in your critiques.

• Provide specific information on any budgetary issues that are not sufficiently described, considered, or justified.

• Look for appropriate budgeting for costs related to engagement, such as financial compensation of patient and other stakeholder partners, costs of meetings, and other facilitators of patient and stakeholder partner participation in research (include these comments under Criterion 5). If you identify a concern in the budget, do not factor it into your score for the application.

17


Chapter 4. Critique Writing

Written Critiques Overview

Offline critique template PCORI provides an offline critique template to all reviewers each cycle to ensure that guidance is consistent and incorporates any changes for that cycle. This document, provided in Appendix 5, is a helpful tool that you should use to write your critiques. Each criterion will have its own section where perceived strengths and weaknesses can be organized, and scores can be provided.

Guidelines for Writing Strengths and Weaknesses When reviewing applications and writing critiques, reviewers should:

Write critiques, not summaries A critique provides a detailed analysis and assessment of the application, not a

summary of the content of the application Provide details Provide an explanation of why a specific point is a weakness or a strength Provide constructive criticism and be specific

Be objective Refrain from discussing your own personal experiences. Generalize your experiences to other patients with different conditions and

experiences. Judge on application’s merit Evaluate each application as submitted. Your score and critique should be based on

the application as-is, not the application’s potential. Do not make assumptions about the principal investigator’s intent. If information

seems to be left out of an application, consider whether or not that is a weakness. Applications should not be compared to one another. You might be assigned two

applications focused on the same disease or condition, and you should be careful to not let the review of one affect the review of the other.

Written critiques will be included in the final summary statements and will be used by several audiences during the Merit Review process:

• By applicants to inform possible resubmissions • By you to prepare your oral presentation of your critique at the In-Person Panel Review • By other In-Person Panel Review Members to help prepare for their participation in the In-

Person Panel discussion • By PCORI staff as they build funding slates and manage projects

The goal of the critique is to ensure that applications, other reviewers, and PCORI staff understands the strengths and weaknesses of each application, based on the merit review criteria. This helps PCORI identify the most meritorious applications and helps applicants understand the strengths and weaknesses of their application, which can inform how they strengthen subsequent submissions.

18


Major and Minor Strengths and Weaknesses Using modifiers—major, moderate, and minor—to describe strengths and weaknesses of your assigned applications can help you determine scores for each Merit Review criterion. The modifiers also help others who read your critiques to understand which points are more important to you and drove your score.

Major Strength: An attribute that is likely to lead to improvements in healthcare and/or outcomes

Moderate Strength: An attribute that would probably lead to improvements in healthcare and/or outcomes

Minor Strength: An attribute that could lead to improvements in healthcare and/or outcomes

Minor Weakness: An easily addressable weakness that does not substantially lessen the impact of the study's results on healthcare and/or outcomes

Moderate Weakness: A weakness that would lessen the impact of the study's results on healthcare and/or outcomes

Major Weakness: A weakness that would seriously limit the impact of the study's results on healthcare and/or outcomes

The upcoming scoring chart provides additional guidance and characteristics to help you determine a score for each criterion based on major and minor strengths and weaknesses.

Feel free to literally copy/paste our wording for these modifiers in your critiques. For example, “The limited role of patient partners would seriously limit the impact of the study’s results on healthcare and/or outcomes because ______________.”

Examples: Writing Strengths Below are sample strengths for Criterion 2: Potential for the study findings to be adopted into clinical practice and improve delivery of care, written from a stakeholder’s perspective.

Review each critique excerpt and decide which one best aligns with PCORI’s guidelines for writing strengths and weaknesses.

Sample Critique 1

Strengths:

• This study proposes to use biological pre-treatment indicators to study the effectiveness of consuming chicken soup for alleviating cold symptoms.

• One of my patients has struggled with a sore throat for many years and she hates taking medicine. I would be happy if I could give this patient chicken soup instead.

19


Examples: Writing Weaknesses Below are sample weaknesses for Criterion 5: Patient and Stakeholder Engagement, written from a patient’s perspective. Take a moment to review each of the critiques and decide which one best aligns with PCORI’s guidelines for writing strengths and weaknesses.

Answer: The correct answer is Sample Critique 2.

Sample Critique 1 is not appropriate because it is vague and personal and omits modifiers. The first point summarizes the aim of the research, but does not follow up by describing WHY it is a major, moderate or minor strength. The second point expresses a personal opinion rather than an objective description.

Sample Critique 2 is appropriate because it directly addresses Criterion 2: Potential for the study to improve healthcare and outcomes findings to be adopted into clinical practice and improve delivery of care. It is objective in tone, and it also mentions the magnitude of the strength.

Sample Critique 2

Strengths:

• Currently there are no biological pre-treatment indicators to predict cold symptom response to chicken soup. This study of biological measures to predict the success of chicken soup will provide caregivers with the information they need to know when to cook chicken soup rather than giving cold medicine to their sick family members. (Major)

• There is a large variability in response to treatment of the common cold. The use of biological measures to predict chicken soup outcomes has the potential to reduce some of that variability for clinicians and may have an impact on the community as a whole. (Moderate)

• The applicants have clearly identified local and national stakeholders who have expressed a need for better understanding of the benefits of chicken soup for the common cold. (Moderate)

• The applicants describe a clear plan for disseminating the results of this study via television cooking programs featuring chicken soup recipes, which will reach a broad general audience. (Major)

Sample Critique 1

Weaknesses: • Patients and caregivers were not formally engaged in the formulation of the research

questions. The application needs to include more details about if and how influenza survivors requested the interventions outlined and, further, if they were interested in the difference between the treatments offered. (Major)

• A more detailed and specific plan is needed describing how patients will be involved in monitoring the conduct and progress of the study, particularly in light of the trust issues between the patient communities and healthcare providers identified in the application. (Moderate)

20


Merit Review Scores Once you have completed your list of strengths and weaknesses for each criterion, you will provide a score for that criterion. The scoring range for the overall application and for the individual criteria consists of a nine-point scale, with lower numbers indicating higher quality. Numerically low scores (such as 1-3) reflect applications that have major strengths in a criterion area, while numerically higher scores (such as scores of 7-9) are associated with applications that are very weak in that criterion area.

The number and magnitude of strengths and weaknesses for each criterion should reflect the criterion score. For example, a score of 1 would correlate with an exceptionally strong application with essentially no weaknesses. An application with an overall score of 4 might have a mix of moderate strengths and moderate weaknesses, with strengths outweighing weaknesses. A score of 7 or 8 might indicate major weaknesses and minor strengths. The scores will depend on the individual reviewer. You should feel free to ask your mentor or Merit Review Officer for guidance on scoring.

This scoring chart can help you determine a score for each criterion.

Align scores to critiques • If you assign a poor score, be clear about the weaknesses of the application (and vice versa). • Text and score alignment will help other reviewers, staff, and the applicant understand your

perspective when they read your reviews.

Answer: The correct answer is Sample Critique 1.

Sample Critique 1 is appropriate because it directly addresses Criterion 5. People representing the population of interest and other relevant stakeholders are engaged in ways that are appropriate and necessary in the given research context. This critique provides details about what could be improved, and makes clear suggestions for improvement.

Sample Critique 2 is not appropriate because it conveys an inflammatory and personal tone.

Sample Critique 2

Weaknesses: • No engagement plan. Is this feasible? • Have you asked Disease X survivors what kind of outcomes they would like to see? • I have worked with this population of patients, and the project is just not going to work

if doctors get to call all the shots in monitoring the project.

21


• The use of modifiers like major, moderate, and minor to describe strengths and weaknesses helps the reader understand what drove your scores The modifiers covered earlier should help you determine a score for each criterion. When determining what score to assign, consider how many strengths versus weaknesses you’ve identified. Also consider whether the gravity of those strengths or weaknesses is more impactful, therefore affecting the balance.

Writing an Overall Narrative The overall narrative should provide a high-level summary of the strengths and weaknesses of the application as a whole. The overall summary should be in paragraph format, instead of in a bulleted list, and should describe the likelihood that the research would exert a sustained, powerful influence on healthcare and patient-centered outcomes.

The overall narrative is a great place to give general feedback to the applicant. Keep in mind that your comments under each criterion should not be copy/pasted or repeated in this section. Repeat information would have no utility for the applicant, and we want reviewers to be providing an overall statement about how the perceived strengths and weaknesses affect each other.

One way to think of writing the overall narrative is to consider it as an “elevator brief.” If you had to cover the most salient points on the merits of the application in the time it would take to ride an elevator 10 floors, what would you say?

Example of a well-written Overall Narrative:

Assigning an Overall Score The overall score:

• Takes into consideration the entire application’s strengths and weaknesses • Accounts for all of the criteria you considered but is not an average of individual criterion scores • Should not fall outside of the range of scores given for each criterion (for example, if you rated

the criteria 3-8, do not assign an overall score of 9)

Overall Summary Score = 5

Please provide your overall comments:

There is a need to improve the delivery of depression care in primary care settings. Most individuals with depression are initially treated in primary care, but identification of depression, referrals, and even medical treatment within the primary care setting are not consistent and usually not evidence based. This project provides an opportunity to identify the optimal model for primary care delivery of depression care. While this is an important and innovative project, the application does have one major and several much more minor weaknesses. The major weakness is the absence of broad input from nurses, physician assistants, and other care providers in primary care offices. Physicians are not the only stakeholders in how primary care offices are organized. More minor weaknesses involve the measures that are used to identify depression, and limited dissemination plan. These weaknesses are easily fixed, but more thought should be given to the engagement of stakeholders.

22


To assign an overall score, use the same 1 to 9 scale you used to score the application on each individual criterion where 1 is the best possible score, and 9 is the worst possible score. This chart provides examples of how your narrative evaluation of an application could translate to an overall score.

Assigning an Overall Score – What to Consider

Potential to Improve Patient Outcomes High Medium Low

Overall Score 1 2 3 4 5 6 7 8 9

Revising and Resubmitting a Critique Almost every reviewer, no matter how experienced, will receive feedback from their Merit Review Officer for how to clarify comments in his or her critiques, especially for the first critique.

When revising and resubmitting a critique, please ensure that you: • Thoroughly review the feedback provided by your MRO or mentor • Pay close attention to all feedback to ensure that you address all questions and concerns • Ask questions to ensure that you have a clear understanding of the specific areas of needed

improvement

Prior to the In-Person Panel Review, Merit Review Officers (MROs) and mentors will review all critiques. This is a critical step in our review process.

e.g. Addresses a problem of critical importance and has a high chance to improve healthcare or outcomes. May have some or no minor weaknesses.

e.g. Addresses a problem of critical importance, but some moderate weaknesses may negatively impact the proposed study’s ability to improve healthcare or outcomes.

Or

e.g. Addresses a problem of moderate importance. May have some or no weaknesses.

e.g. Addresses a problem of minor importance, and some moderate weaknesses may negatively impact the propose study’s ability to improve healthcare or outcomes. Or e.g. Addresses a problem of moderate importance, but the major weaknesses severely impair the proposed study’s ability to improve healthcare or outcomes.

23


Chapter 5. In-person panel meeting In-Person Panel Review Overview

Preparing for the In-Person Panel Review Before the In-Person Panel Review:

� Review the list of applications that will be discussed at the In-Person Panel Review. Your Merit Review Officer will send you this list in discussion order before the panel meeting.

� Review all of the written critiques and preliminary scores for all of your assigned applications � Prepare your brief oral presentation using main strengths and weaknesses from your written

critique. � If you have time, review the abstracts and written critiques for the other applications on the

discussion order. Becoming familiar with all of the applications and their critiques will help you understand and score these applications after hearing the discussion.

Panel and Reviewer Roles The following people will be in the room during the In-Person Panel Review

Panel Reviewers (not PCORI Staff members) • About half of the panel reviewers will be scientists and the other half will be patients and

stakeholders.

Merit Reviewer Officer (MRO) (PCORI staff member) • Serves as main PCORI point of contact for reviewers • Provides guidance to reviewers on PCORI process and policy • Presents brief orientation to panel proceedings • Answers panel members’ questions about PCORI and the review process • Records notes of the panel discussion, which are also incorporated into the final summary

statement

The In-Person Panel Review provides an opportunity for reviewers to discuss the strengths and weaknesses of the applications selected for discussion after the Preliminary Review phase. The full panel has an opportunity to ask questions and fully discuss each application, with facilitation by the Chair. This discussion helps reviewers provide scores for the applications that they were not assigned to review.

• Reviewers receive a brief orientation to the panel process and group rules. • The Panel Chair introduces each application. • Reviewers assigned to each application make individual oral presentations the strengths and

weaknesses that drove their scores of the application. • Members of the full panel ask questions and discuss the application. • The Chair records an oral summary of the panel discussion. • Each panel reviewer provides a final overall score for the application.

24


Panel Manager (PCORI staff member) • Serves as administrative point of contact for reviewers

Provides support for reviewers in the online scoring system and other technical needs

Panel Chair (not a PCORI staff member; usually an experienced senior-level scientist) • Introduces and facilitates discussion of each application

Records an oral summary of the discussion of each application

Observers • Observers often include PCORI program staff members and mentors, who are available on-

site to provide support for patient and other stakeholder reviewers if needed.

Inside the In-Person Panel Review Video Click HERE or below on the embedded video to watch a simulation of an in-person panel review:

Preparing In-Person Panel Presentations Panel presentations give reviewers an opportunity to participate in a thoughtful discussion about the strengths and weaknesses of the selected applications. Reviewers should familiarize themselves with all applications on the discussion list in advance, even if they are not assigned to review them. Your presentation should cover the main strengths and weaknesses of the applications you reviewed.

• Start by thinking about what other reviewers will need to know about the strengths and weaknesses of each application, based on your evaluation, in order to score the application.

25

https://www.youtube.com/watch?v=HlnE5VK4Ijc&feature=youtu.be.https://www.youtube.com/watch?v=HlnE5VK4Ijc


• Review the written critiques that were submitted by the other three reviewers on your assigned applications. This will help you identify topics that are likely to be discussed in the panel.

Remember, you have about two (2) minutes to present. Be clear and to the point. Do not summarize the application, but you can add key details that the Chair may not have covered in his or her introduction. Panel Review Discussion Tips Making sure that the In-Person Panel is a welcoming, safe, and open environment requires more than coming prepared to discuss your critique—it also requires being a thoughtful, respectful, and engaged participant. Here are some tips that can help you manifest these qualities in this stage of the process. Avoid Repetition

Avoid repetition of points made by previous reviewers. This will keep the discussion moving and allow adequate time to review each application on the schedule.

• If you agree with a reviewer’s perspective, you can summarize why you agree: o “I agree with the previous reviewer about…”

• If you disagree with the reviewer’s viewpoint, be sure to voice your views and disagree with the idea, not the person:

o “I view this differently because…” o “I would like to add these important strengths/weaknesses that haven’t been

mentioned…” Maintain a Productive Discussion

A critical part of the In-Person Panel Review is a productive discussion. As a panel reviewer, you can help maintain a productive discussion by:

• Respecting all viewpoints • Engaging in active listening • Asking good questions • Using words such as good, bad, better, or worse to describe scores, instead of high or low; this is

to avoid confusion since low numerical scores indicate high quality • Minimizing the use of acronyms and jargon • Being culturally aware/sensitive

Respect all Viewpoints

Although you may not agree with what other panel reviewers say, it is important that you: • Listen to what others have to say before expressing your viewpoint • Refrain from insulting, name-calling, criticizing, or putting down a panel member • Build on one another’s comments • Work towards shared understanding • Be mindful of cultural differences and the needs of different patient populations • Be respectful of the different constituencies around the table; each person brings a valuable

viewpoint that helps PCORI find the best research to improve patient outcomes

Practice Active Listening

26


A successful panel member uses active listening in order to check assumptions, clarify his or her thoughts, and understand others. Here are a few active listening strategies that you can use during the In-Person Panel Review:

• Mentally commit to listening • Avoid distractions

o Turn off mobile devices o Avoid answering email and visiting web sites not related to the applications o Refrain from side conversations

• Take notes while you listen • Jot down questions that you would like to ask • Face the person who is speaking

Ask Good Questions

Good questions are the key to a productive discussion. Questions can be used to probe for deeper analysis, get clarifications or examples, explore implications, or respectfully challenge opinions or ideas.

Examples of common In-Person Panel Review questions include: • Can you tell me more about the engagement plan? • Would patient advocacy groups be willing to disseminate the findings of the research? • Can a clinician or clinical administrator in the room talk about how well this intervention could

be implemented in clinical practice if this research were to find it effective?

27


Appendix 1. COI/Expertise INFORMATION FOR PCORI MERIT REVIEWERS ON CONFIDENTIALITY, CONFLICT

OF INTEREST, AND RATING EXPERTISE (February 2, 2015)

Click here to access PCORI’s guidance on conflict of interest and rating expertise online.

The Patient-Centered Outcomes Research Institute (PCORI) welcomes a broad array of stakeholder reviewers to participate in the evaluation of research applications (“Reviewers”). Reviewers are essential to helping PCORI fulfill its mission and to fund research that is both scientifically rigorous and truly patient centered. Given the important role of Reviewers in PCORI’s application selection process, PCORI requires Reviewers to abide by a number of policies and commitments that support a fair and objective merit review process. This document provides information about three important obligations of any Reviewers participating in PCORI’s merit review process: A) Confidentiality and Non-Disclosure; B) Conflict of Interest; and C) Rating Expertise.

A. CONFIDENTIALITY AND NONDISCLOSURE

Maintenance of confidentiality is a critical component of merit review. All Reviewers are required to agree to the terms of a PCORI Non-Disclosure Agreement (“NDA”) before they participate in merit review activities. By agreeing to the NDA, Reviewers confirm that they will preserve and not disclose confidential information and that they will not use any confidential information except as required to perform the responsibilities of merit review.

In the context of preserving the confidentiality of the merit review, it is important that materials reviewed before or during the merit review meeting as well as discussion content of the merit review meeting not be disclosed to anyone at any time before, during, or after the merit review meeting except as part of the application evaluations during the actual meeting. Confidential information includes any information that has not been made public, such as information about applications, number of applications discussed, research topics, negative or positive outcomes of the meeting, and any personal information about other reviewers disclosed as part of the merit review process.

In order to maintain the integrity of the review process, Reviewers must not contact any applicants for whom they have access to application material. Merit assessments of applications must be completed using only the information provided by the applicant at the time of submission. Reviewers must not request additional information from applicants once the application has been submitted. If Reviewers need assistance in reviewing applications, he or she may contact their Merit Review Officer (MRO) for help or clarification.

Reviewers must not use social media or other electronic media tools during merit review panel discussions or activities. Reviewers must not discuss the review with other reviewers absenting themselves from the room for conflict of interest (COI) reasons, or with reviewers of any other panel. If a Reviewer is asked to disclose information about the contents of an application or about the nature of review discussions, he or she must inform the person making the request that merit review participants may not disclose such information and must inform the panel MRO that he or she has been contacted directly.

28

http://www.pcori.org/sites/default/files/PCORI-Information-for-Reviewers-Confidentiality-COI-Rating-Expertise.pdf


It is the responsibility of each Reviewer to safeguard the confidentiality of review material while it is in his or her possession, not to share the material with other persons, and to properly dispose of both hard copy and electronic materials at the conclusion of the panel meeting or when directed to do so.

The actions outlined above are among the steps that a Reviewer should take to fulfill his/her obligations under the Non-Disclosure Agreement.

B. CONFLICT OF INTEREST (COI)

PCORI’s Board of Governors has adopted a Conflict of Interest Policy that applies to all PCORI activities, including merit review of applications for research funding. A copy of the PCORI Conflict of Interest Policy is provided to all Reviewers and is available on PCORI’s website. The information here is intended to help Reviewers understand how PCORI implements and interprets the Conflict of Interest Policy in the context of merit review activities, including how conflicts of interest should be disclosed and addressed.

PCORI requires each Reviewer to disclose conflicts of interest as a condition of participating (or being considered for participation) in merit review. PCORI relies on the professionalism and integrity of each Reviewer to identify any financial or personal associations that have the potential to bias or have the appearance of biasing the Reviewer’s activities and decisions in merit review. The appearance or perception of bias can be enough to undermine the public trust. All efforts should be made to identify all associations that may give rise to a conflict of interest. It is important that each Reviewer submit COI disclosures by the requested deadline so that application assignments can be made to the full panel in a timely manner.

A COI in merit review exists when a Reviewer or a close relative or professional associate of the Reviewer has a financial or personal association related to an application, including the applicant and investigators, which may bias the evaluation of the application or create the perception of bias. The term “close relative” includes a parent, spouse, domestic partner, or child. Depending upon factors like financial dependency, cohabitation, and family history, sometimes other relatives could also be considered “close relatives.” Reviewers should use their best judgment in determining when a familial relationship is close enough that the relative’s associations could bias or appear to bias their decision making.

Financial associations often involve relationships or interests that may cause a Reviewer to have a financial stake in whether certain applications are selected for PCORI funding. Regardless of the level of financial involvement or other interest, if a Reviewer feels, or may be perceived as being, unable to provide an objective evaluation, he or she may not participate in the review of the application. Personal associations can be either professional or non-professional relationships with the applicant, the investigator(s), or a person or organization whose interests would be affected by the project under review.

For COI purposes, applicant and investigators include the roles listed below.

• Applicant: Principal Investigator (PI) listed in the application • Investigator: All active participants (PI, co-PI, research partner, collaborator, consultant,

subcontractor, and other senior/key personnel) listed in the application

Based on the nature of the COI, it may be handled at either the PFA or application level. Please review the following examples of COI and how to handle them at each level. While the following are provided as general examples, PCORI reserves the right to address conflicts of interest on a case-by-case basis.

29


1. PCORI Funding Announcement (PFA) Level COI

Certain conflicts of interest can be viewed as involving such powerful influences or deeply-felt relationships that they have the potential to bias (or appear to bias) a Reviewer’s evaluation of all of the applications submitted in response to a PFA. For example, a Reviewer might hold so large a financial interest in one application that it would not reasonably appear the Reviewer could impartially evaluate competing applications that have been submitted in response to the same PFA.

If any of the following types of conditions apply, the Reviewer cannot serve on panel reviewing applications received in response to a particular PFA.

• The Reviewer is an investigator in an application on a PFA reviewed by the panel. • The Reviewer has a close relative who is an investigator on an application reviewed by the

panel.

There may be other circumstances, in addition to those identified above, in which a Reviewer feels unable to serve impartially on a panel evaluating applications submitted in response to a specific PFA, or in which it might appear that the Reviewer cannot do so. The Reviewer should report such conflicts to the panel’s MRO or the Associate Director, Merit Review and self-recuse from participating on the panel for applications related to the specific PFA.

2. Application Level COI

In contrast to PFA-level conflicts of interest, the potential for bias created by other types of conflicts may be confined to the review of a particular application. If any of the following types of conditions apply, the Reviewer can serve on the panel but must recuse himself or herself from the discussion and scoring of the application. The Reviewer will not have access to that application or participate in the discussion or scoring of the application, and the recusal will be documented.

• The Reviewer or his/her close relative currently receives, or within the past 12 months has received, medical care from the applicant entity, principal investigator, or other individuals identified in the application as key personnel.

• The Reviewer or his/her close relative currently has a significant personal or professional relationship with the applicant entity, principal investigator, or other key personnel. (Note that sometimes negative relationships – for example, a professional rivalry – can be a significant personal or professional relationship.)

• The Reviewer or his/her close relative provides, or within the past 12 months has provided, technical assistance to the applicant entity, principal investigator, or other key personnel in any of the following ways:

o Assistance with preparing or submitting the application.

o Providing the applicant entity, principal investigator, or other key personnel with resources for the application that are not freely available to others in the research community -- e.g., specialized data analysis, service, or confidential material.

Note that providing resources that are freely available to anyone in the scientific community (e.g., letter of support, service, equipment, data, or other material) would not be considered a conflict of interest.

• The Reviewer or his/her close relative is employed at the applicant entity.

30


o For multi-campus State institutions, a Reviewer who is primarily employed at one campus of the institution is not considered to have a conflict of interest with respect to an application submitted by another campus of the same institution provided that the reviewer has no institutional responsibilities that would significantly affect the other campus.

o For private institutions and affiliates, a Reviewer who is primarily employed at one affiliate of the institution is not considered to have a conflict of interest with respect to an application submitted by another affiliate of the same institution provided that the reviewer does not have institutional responsibilities that would significantly affect the other affiliate.

• The Reviewer or his/her close relative is actively negotiating, or has an agreement about future employment at the applicant entity.

• The Reviewer or his/her close relative has a professional relationship other than employment with an applicant entity – e.g., consulting or other vendor contract, service on board of directors, service on advisory committee.

• The Reviewer or his/her close relative could receive professional gain or advancement (e.g., publications, scientific prizes, or academic appointments) as the direct result of the application funding decision.

• The Reviewer or his/her close relative could receive a financial benefit exceeding $10,000 per year from individuals or companies that own or manufacture medical treatments, services, or items that the application proposes to study.

C. RATING EXPERTISE

PCORI merit review is designed to incorporate the perspective of scientists, patients, and other healthcare stakeholders, including having merit review panels that incorporate appropriate areas and levels of expertise. To support appropriate composition of merit review panels, Reviewers will be notified when application abstracts and lists of key personnel are available and accessible in PCORI Online. When reading the application title and abstracts, Reviewers should indicate for each application whether their expertise matches with the content of the application and whether that content match is high, medium, low, or none.

For Scientific reviewers, PCORI expects reviewer rating of expertise for specific applications to be made on the basis of the individual’s research expertise and science training. Please note that expertise is an indication of the extent of the reviewer’s subject-matter expertise and is not a reflection of his or her willingness to review an application.

It is not necessary for any patient or stakeholder reviewer to indicate high, medium, or low expertise on applications. Patients and stakeholders provide critical and important perspectives during the review process, independent of technical expertise. If patient or stakeholder reviewers do have specific disease expertise, however, it is appropriate to indicate this in the system. Please also note that the ratings are used to match applications to reviewers and are not a reflection of the relative importance of any reviewer on the panel.

31


Please use the following descriptors to indicate Rating Expertise:

Rating Description For Scientists High

The Reviewer is able to evaluate the application with little or no need to make use of background material or the relevant literature. The Reviewer has likely published in areas closely related to the science presented in the application.

Medium

The Reviewer has most of the knowledge to evaluate the application but will require some review of relevant literature to fill in details or increase familiarity with the system employed. The Reviewer may employ similar methodologies in his or her own work but may need to review the literature for recent data relevant to the application.

Low

The Reviewer understands the broad concepts but is unfamiliar with the specific methodology or other details, and reviewing the application would require considerable preparation.

None The Reviewer has only superficial or no familiarity with the concepts and methodology described in the application.

It is important that each Reviewer submit the expertise information by the requested deadline so that application assignments can be made to the full panel in a timely manner.

D. ENFORCEMENT AND CONCLUSION

If a Reviewer violates his or her obligations as a reviewer, including relating to Confidentiality, Conflicts of Interest, or Rating Expertise, PCORI may implement sanctions or corrective measures, as appropriate. These sanctions may include: removing the Reviewer from the panel; notifying other panel members of the violation; initiating an internal investigation of the Reviewer’s conduct and its consequences; and disqualifying the Reviewer, indefinitely or for a specified period, from participating as a PCORI reviewer.

If any Reviewer has any questions about the information outlined in this document on Confidentiality and Non-Disclosure, Conflicts of Interest, or Rating Expertise, please contact your panel’s MRO or the Associate Director, Merit Review ([email protected]).

PCORI is grateful for the important contributions that Reviewers make to PCORI’s application selection process.

32

mailto:[email protected]


Appendix 2. Methodology Standards PCORI Methodology Standards

Updated: October 13, 2015 Published: February 4, 2014

Click here to access PCORI’s Methodology Standards online.

Cross-Cutting Standards for PCOR

1: Standards for Formulating Research Questions

RQ-1: Identify gaps in evidence Gap analysis and systematic reviews should be used to support the need for a proposed study. If a systematic review is not available, a systematic review should be performed using accepted standards in the field (see standard SR-1), or a strong rationale should be presented for proceeding without a systematic review. In the case where a systematic review is not possible, the methods used to review the literature should be explained and justified.

RQ-2: Develop a formal study protocol Studies should include a formal protocol specifying at least one purpose for which the data were collected (e.g., effectiveness, safety, natural history of disease, quality improvement); data sources and linkage plans, if any; data feasibility and quality, measure(s) of effect; and use of any standardized data dictionaries (nationally or internationally accepted).

RQ-3: Identify specific populations and health decision(s) affected by the research To produce information that is meaningful and useful to people when making specific health decisions, research proposals and protocols should describe: 1) the specific health decision the research is intended to inform; 2) the specific population for whom the health decision is pertinent; and 3) how study results will inform the health decision.

RQ-4: Identify and assess participant subgroups In designing studies, researchers should identify participant subgroups of interest and, where feasible, design the study with adequate precision and power to reach conclusions specific to these subgroups. In addition, subgroup information should be reported for later systematic reviews.

RQ-5: Select appropriate interventions and comparators When evaluating an intervention, the comparator treatment(s) must be chosen to enable accurate evaluation of effectiveness or safety compared to other viable options for similar patients. Researchers should make explicit what the comparators are and how they were selected, focusing on clearly describing how the chosen comparator(s) define the causal question, reduce the potential for biases, and allow direct comparisons. Generally, non-use (or no specific treatment) comparator groups should be avoided unless no specific treatment is a likely option in standard care.

33

http://www.pcori.org/research-results/research-methodology/pcori-methodology-standards


RQ-6: Measure outcomes that people representing the population of interest notice and care about Identify and include outcomes the population of interest notices and cares about (e.g., survival, function, symptoms, health-related quality of life) and that inform an identified health decision. Define outcomes clearly, especially for complex conditions or outcomes that may not have established clinical criteria. Provide information that supports the selection of outcomes as meeting the criteria of “patient-centered” and “relevant to decision makers,” such as patient and decision-maker input from meetings, surveys, or published studies. Select outcomes based on input directly elicited from patient informants and people representative of the population of interest, either in previous studies or in the proposed research.

2: Standards Associated with Patient-Centeredness

PC-1: Engage people representing the population of interest and other relevant stakeholders in ways that are appropriate and necessary in a given research context People representing the population of interest include individuals who have the condition or who are at risk of the condition and, as relevant, their surrogates or caregivers. Other relevant stakeholders may include clinicians, administrators, policy makers, or others involved in healthcare decision making. Stakeholders can be engaged in the processes of:

• Formulating research questions;

• Defining essential characteristics of study participants, comparators, and outcomes;

• Identifying and selecting outcomes that the population of interest notices and cares about (e.g.,

survival, function, symptoms, health-related quality of life) and that inform decision making

relevant to the research topic;

• Monitoring study conduct and progress; and

• Designing/suggesting plans for dissemination and implementation activities.

When applicable, research proposals should describe how these stakeholders will be identified, recruited, and retained. If engagement is not necessary or appropriate in these processes, explain why.

PC-2: Identify, select, recruit, and retain study participants representative of the spectrum of the population of interest and ensure that data are collected thoroughly and systematically from all study participants Research proposals and subsequent study reports should describe: 1) the plan to ensure representativeness of participants; 2) how participants are identified, selected, recruited, enrolled, and retained in the study to reduce or address the potential impact of selection bias; 3) efforts employed to maximize adherence to agreed-on enrollment practices; and 4) methods used to ensure unbiased and systematic data collection from all participants.

If the population of interest includes people who are more difficult to identify, recruit, and/or retain than other study populations (for example, individuals historically underrepresented in healthcare research such as those with multiple disease conditions, low literacy, low socioeconomic status, or poor healthcare access, as well as racial and ethnic minority groups and people living in rural areas), then

34


specify plans to address population-unique issues for

reviewer manual for scientist reviewers funding program ......reviewer manual for scientist...

Documents