1

Objectives

• Describe ways in which the EPC program has engaged stakeholders to improve relevance and applicability of its products

• Present methods used for stakeholder engagement

• Identify challenges in stakeholder engagement and strategies employed by the EPC program

Evidence-based Practice Centers (EPCs) 11 centers throughout the United States and Canada

Systematically and critically appraise existing research and synthesize knowledgeProduce comparative effectiveness and effectiveness reviews

Scientific Resource Center (SRC)Responsible for scientific methodological work of reviews and other research projects for EPCs and DEcIDES

EPCs & SRC

EPCs & SRCBlue Cross and Blue Shield Association, Technology Evaluation Center, Chicago, ILBrown University, Providence Rhode IslandECRI Institute, Plymouth Meeting, PAJohns Hopkins University, Baltimore, MDKaiser Foundation Research Institute, Oakland, CAOregon Health & Science University, Portland, ORRTI International – University of North Carolina at Chapel Hill, Chapel Hill, NCRAND Corporation , Santa Monica, CAUniversity of Alberta, Edmonton, Edmonton, AlbertaUniversity of Minnesota EPC, Minneapolis, MNVanderbilt University Medical Center, Nashville, TN

Scientific Resource Center (SRC) for EPCs and DEcIDES – Portland VA Research Foundation, Portland OR

Public

Nominate Review Topics

Technical Expert Panel

Key Research Questions

Peer and Public Review

Feedback on Summary Guides

Future Research

Disseminate Reports

Key Informants

Opportunities for Stakeholder Input

EHC Report on Engaging Stakeholders for Development and Prioritization of Future Research Needs

• O’Haire C, McPheeters M, Nakamoto E, LaBrant L, Most C , Lee K, Graham E, Cottrell E, Guise J-M. Engaging Stakeholders To Identify and Prioritize Future Research Needs. Rockville (MD): Agency for Healthcare Research and Quality (US); 2011 Jun. (Methods Future Research Needs Reports, No. 4.) Available from: http://www.ncbi.nlm.nih.gov/books/NBK62565/

Literature Synthesis

Phase III:EPC interviews

(Vanderbilt EPC)

Recruitment and Interview Guide Development

Analysis of Interviews

Key Informant Interviews

Phase II: Key Informant Interviews(Oregon EPC)

Phase I:Literature Summary

(Oregon EPC)

Literature Review

Literature Search

Cumulative Synthesis

EPC Protocol Review

Invitations and EPC Discussion Guide

EPC Discussions

Analysis of Discussions

Stakeholder Identification & Recruitment• Identify pertinent stakeholder groups

(intentional based upon stakeholder groupings)• Ensure a balance of stakeholder perspectives

(i.e. consumer, clinician, researcher, research funder, insurer/payer, manufacturer)

• The results of EPC FRN pilot projects suggest that engaging 6-10 stakeholders is appropriate

• OMB restricts the number of participants in surveys to no more than 9 non-federal employees

Research Needs Development

StakeholdersStakeholder Group Description

Consumer (public, patient, and caregiver)

An individual or advocacy group representing individuals who have a health condition, use health care services and/or who are members of the community

Clinician (Health professional)

Health care or public health provider (e.g., academic, rural/frontier, and community) and medical and/or public health organizations

Social Service Organizations An individual or organization that advances human welfare or social work (e.g. community or condition service providers, school-based programs, justice system)

Policymaker An individual or organization who is involved in health care policy (e.g. local, state, provincial and Federal legislators and staff)

Medical organizations Governmental organizations (e.g. VA, AHRQ, etc.)

Researcher An individual who conducts and/or facilitates research activities in:

Basic, translational, clinical sciences; research methodology, public health or health services, and systematic reviews

Research Funder A public or private organization that funds research (e.g. National Institute of Health, Susan G. Komen Foundation, and American Cancer Society)

Insurer/Payer An organization or agency that pays for health-related goods and services (e.g. Blue Cross Blue Shield, Medicaid, and Medicare) or a business group that pays for health insurance (e.g. employers and government)

Manufacturer A business group that produces health-related items (e.g. pharmaceuticals and medical devices)

Methods of Stakeholder Engagement

Engagement Most common: Focus Groups, Symposia/Conferences

Prioritization Most common: Priority Questionnaire, Consensus

Engagement & Prioritization Simultaneously Priority Survey Delphi

1:1 Meetings

1:1 Meetings (phone)

Focus Groups

GroupCalls Symposia

Consensus or Voting Survey Delphi

Short Timeframe • + – + – – + –

↓ Budget – + • + – – + –Geographic

Barriers • + – + – – + +Differing

Perspectives + + – – + – + +Multiple

Stakeholder Groups + + – – + – + +

↓Research Experience + + + – + – – •

Complex Program + + + – + – • –

+GOOD – FAIR • POOR

Topic Generation

Nominate topics via the web

Topic Development

Clarify nomination

intent

Frame clinical

context of nomination

Topic Refinement

Provide “real world” input

Provide input on clinical logic

Public comment via

Web

Review

Provide methodological & clinical

input

Provide input via peer review

Provide public

comment via Web

Research Needs

Development

Participate in discussions to describe research gaps &

prioritize research

needs

Report Translation & Dissemination

Participate in focus groups

Participate in individual interviews

Provide expert

review of summary

guides

EHC Program Research Reviews

Topics Evidence Review Research D&I

Topic Generation ApproachesNominal Group Process In-person Idea generating Intended to promote group participation

Our experience: In-person brainstorming with flip chart used to document topics Each state had 8 stickers to vote for priority topics:

#1 (2 red dots)#2 (2 green dots) #3 (2 blue dots), and #4 (2 yellow dots)

Weighting system applied to priorities: #1 - 4 points for each vote#2 - 3 points#3 – 2 points#4 – 1 point

Topic Generation Approaches

Medicaid Medical Directors

Analytic Process:Weighting system :

#1 - 4 points #2 - 3 points#3 – 2 points#4 – 1 point

Prevention in Women’s HealthModified Process:

Local Key InformantsThree web meetings:

May 11 – Framework & Initial Topics(Extranet & email topic generation)June 8 – Topic Generation(Evidence Scan)

13 – Final PrioritizationPrioritization

Survey – survey monkeyWeb whiteboard during 3rd meeting

Topic Generation Approaches

Stakeholders Engaged in:Development of CER key questions Refinement of topicReview of Draft

Methods used:Group or individual callsWeb-based forums with survey type format to provide

responses

Scoping CER

Preparation of Stakeholders is critical:

• Assume that all stakeholders need orientation to the CER process

• Very important for them to know where they fit in and what to expect from their feedback

• Familiarity with research methods likely to be very variable

• Conflicts of interest are to be expected with stakeholder groups, but must be disclosed.

Scoping CER

Topic Generation

Nominate topics via the web

Topic Development

Clarify nomination

intent

Frame clinical

context of nomination

Topic Refinement

Provide “real world” input

from provider,

patient, and policy

perspectives

Provide input on clinical

logic

Provide public

comment via Web

Research Review

Provide methodological & clinical input

as a TEP member

Provide input via peer review

Provide public comment via

Web

Research Needs

Development

Participate in

discussions to describe research gaps &

prioritize research

needs

Report Translation & Dissemination

Participate in focus groups

Participate in

individual interviews

Provide expert

review of summary

guides

EHC Program Research Reviews

Stakeholder Preparatory Materials

Where We Arein the Comparative

Effectiveness Review Process

Name of Panel EPC Role & What to Expect

 

Describe what has occurred to date

  

 

• Describe Intent of panel

• Numbers and types of individuals

• Numbers of times and ways they may be engaged

• List ways TEP is especially helpful to process

 

• The role of the Evidence-based Practice Center (EPC) is to …….

• List Key Principles• Reviews commissioned under

the Effective Healthcare Program (EHC) are posted publically at different stages of the review process, including at the stage of proposed Key Questions and the draft report stage.

Stakeholder Preparatory Materials

• Comparative Effectiveness Reviews (CERs) critically appraise and synthesize bodies of literature and identify areas where research is lacking or insufficient in either quantity or quality

• There are limitations in funds available to support research

• To fully actualize their potential, CERs should be utilized to inform and guide future research

• Future Research Needs (FRN) documents are designed to facilitate this process by engaging stakeholders to prioritize future research topics

Future Research Needs Development

Purpose

Future Research Needs Development

Step 1

•Research gaps from CER interviews

Step 2

•Identify & Recruit Stakeholders

Step 3

•Orient stakeholders to project

Step 4

•Solicit input on gaps, generate topics

Step 5

•Engage stakeholders to prioritize FRN (at least 2 rounds)

Step 6

•Categorize by PICOs

Step 7

•Search for ongoing Research & Reviews

Step 8

•Write up report

Step 9

•Elicit stakeholder feedback

Step

10

•Final report

Combined Methods:• Semi-structured interviews to identify topics• Modified Delphi process for prioritization

Semi-structured interviews Telephone Conference calls Webinars

Delphi#1 Survey

Topics organized by PICOsLikert Scale – 6-point scale of no, low, highest priority

#2 SurveyLikertRankingIndicate top 3 in order

Research Needs Development

Research Needs DevelopmentPriority Final Rank Weighted

Scorea

Studies to test clinical, institutional, or policy interventions to increase access to “safe” TOL 1 72

Research on barriers to providing safe TOL, including factors that limit the ability of hospitals to meet the “immediately available” requirement

2 51

Studies comparing outcomes for mother and infant in settings where physicians are “immediately available” vs. settings where physicians are “readily available”

3 46

Studies to understand best practice models based on institutions that are currently offering safe TOL

4 41

Development of standardized measures for short-and long-term maternal and infant outcomes

5 41

Surveillance to determine long-term clinical outcomes of TOL vs. ERCD 6 38

Research on how patients understand risk, how they respond to different ways of framing risk, and how best to communicate risks of TOL vs. ERCD

7 37

Clinical and policy relevant studies to address the threat of legal liability on practice patterns regarding TOL vs. ERCD

8 34

Development/utilization of a reliable model or tool to predict the probability of successful VBAC for individual women /tool to predict probability of successful VBAC

9 32

Studies to refine, validate, and implement informed consent templates that are informative, reliable, and able to be well documented

10 26

Stakeholder Involvement: Across Research PhasesRESEARCH CONTINUUM

TopicGeneration

StudyDesign Implementation Analysis/

Interpretation Research

PrioritizationDissemination

CliniciansConsumers

FundersInsurers

ResearchersPolicymakers

CliniciansConsumers

FundersInsurers

ManufacturersResearchers

PayersPolicymakers

CliniciansManufacturers

ResearchersPolicymakers

CliniciansConsumers

FundersInsurers

ManufacturersResearchers

PayersPolicymakers

CliniciansConsumersResearchers

Policymakers

CliniciansResearchers

EPC Stakeholder Engagement Expanded Scope of Future Research Needs Beyond CER

Research Needs Development

SummarySeveral options for methods to use when engaging stakeholdersMethod depends on several factors including

• Intent (identifying topics, prioritization)• Resources• Number and variety of stakeholders• What is already known in topic• Experience

Important to outline approach and intent a prioriRelationships with stakeholders are important

• Contact potential stakeholders through multiple venues• Consistent communication and follow-up in the form of emails or

phone calls is important to ensuring a balanced selection of stakeholders and a high response rate

Preparation of stakeholders is critical to define expectations and allow engagement