introduction to epcts - duke university...to real-world decision-making may come at the expense of...
TRANSCRIPT
Introduction to ePCTs
10 self-paced learning modules on how to design, conduct, and disseminate embedded pragmatic clinical trials (ePCTs)
Objectives
1. Provide investigators with an introduction to the design and conduct of ePCTs
2. Identify important things to know
3. Identify important things to do
4. Point to key resources to advance learning
Learning modules1 What are ePCTs?
2 Engaging All Stakeholders & Aligning with Healthcare System Partners
3 Designing with Implementation in Mind4 Design and Analytic Considerations5 Regulatory and Ethical Challenges of ePCTs6 Measuring Outcomes7 Pilot and Feasibility Testing8 Dissemination of Results9 ePCT Team Composition
10 Developing a Compelling Application
The NIH Collaboratory’s Living Textbook of Pragmatic Clinical Trials: www.rethinkingclinicaltrials.org
Learning resource
1: What are ePCTs?
Contributing authors:
Lesley Curtis, PhD, Duke Center for Pragmatic Health Systems Research
Gloria Coronado, PhD, Kaiser Permanente Center for Health Research
Doug Zatzick, MD, University of Washington School of Medicine
• Identify key characteristics in the rationale and design of ePCTs
• Identify key differences between explanatory and pragmatic trials
• Provide an understanding of the PRECIS-2 tool and its ability to assist teams in the design of an ePCT
Learning goals
• ePCTs are designed to answer important, real-world clinical questions
• Broad stakeholder engagement and support are essential from beginning to end
• Tradeoffs in flexibility, adherence, and generalizability are inevitable
Important things to know
• Conducted within healthcare systems
• Use streamlined procedures and existing infrastructure
• Answer important medical questions
ePCT characteristics
ePCTs have the potential to inform policy and practice with high-quality evidence at reduced cost and increased efficiency compared with traditional clinical trials
Why conduct ePCTs?
Study designed with
input from health system stakeholders
Intervention incorporated into routine
clinical workflow
Data collected through EHR in health care
settings
Diverse, representative
study populations
Outcomes important to
decision makers
ePCTs bridge clinical care and research
Key differences between explanatory and pragmatic trials
Adapted from Zwarenstein M, Treweek S, Gagnier JJ, et al. BMJ. 2008;337:a2390. doi: 10.1136/bmj.a2390. PMID: 19001484
EXPLANATORY PRAGMATICResearch question
Efficacy: Can the intervention work under the best conditions?
Effectiveness: Does the intervention work in routine practice?
Setting Well-resourced “ideal” setting Routine care settings including primary care, community clinics, hospitals
Participants Highly selected More representative with less strict eligibility criteria
Intervention design
Tests against placebo, enforcing strict protocols & adherence
Tests 2 or more real-world treatments using flexible protocols, as would be used in routine practice
Outcomes Often short-term surrogates or process measures; data collected outside of routine care
Clinically important endpoints; at leastsome data collected in routine care
Relevance to practice
Indirect: Not usually designed for making decisions in real-world settings
Direct: Purposely designed for making decisions in real-world settings
Common-sense definition
“Designed for the primary purpose of
informing decision-makers regarding
the comparative balance of benefits,
burdens and risks of a biomedical or
behavioral health intervention at the
individual or population level.”
Califf RM, Sugarman J. Clin Trials. 2015 Oct;12(5):436-41. doi: 10.1177/1740774515598334
• Achieving both relevance and efficiency is a goal of pragmatic trials, yet high relevance to real-world decision-making may come at the expense of trial efficiency
• For example, a trial measuring outcomes that matter most to patients and health systems may not be able to rely exclusively on information from the EHR, and instead need to assess patient-reported outcomes, which is more expensive and less efficient
Balancing relevance and efficiency
Why Are We Talking AboutPragmatic Clinical Trials?
Resource: What are ePCTs?
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Eligibility• Recruitment• Setting• Organization• Flexibility: delivery• Flexibility: adherence• Follow-up• Primary outcomes• Primary analysis
PRECIS-2: Designing trials fit for purpose
Explanatory Pragmatic
Tool assesses trial across 9 domains
Introducing PRECIS-2
PRECIS-2 source: Kirsty Loudon et al. BMJ 2015;350:bmj.h2147. Copyright 2015 by British Medical Journal Publishing Group. Used by permission.
PRECIS-2 can be a useful tool for understanding variability in pragmatic trial characteristics
Pragmatic–Explanatory Continuum Indicator Summary (PRECIS) tool
PRECIS-2: Eligibility
Who is selected to participate in the trial?
Highly selected patients, strict
inclusion criteria
Typical patients, minimal
inclusion criteria
Explanatory Pragmatic
PRECIS-2: Recruitment
How are participants recruited into the trial?
Uses methods and resources outside
of, or in addition to, what is typical
Recruited in usual healthcare settings;
participants may include patients,
providers, or health systems
Explanatory Pragmatic
PRECIS-2: Setting
Where is the trial being done?
Specialist practice or academic
medial center
Settings where the trial’s results will be applied
Explanatory Pragmatic
PRECIS-2: Organization
What expertise and resources are needed to deliver the intervention?
Changes the workflow, adds equipment or staff training, or affects how
care is typically delivered
Changes to clinical delivery and resources are minimal, easy to implement in usual care after the trial
Explanatory Pragmatic
PRECIS-2: Flexibility -delivery
How should the intervention be delivered?
Highly specified, protocol-driven with timing of intervention
tightly defined
Details of intervention
delivery left to the care provider
Explanatory Pragmatic
PRECIS-2: Flexibility - adherence
What measures are in place to ensure participants adhere to the intervention?
Measures to monitor patient adherence
and excludes patients judged not
to be adherent
No special measures to
enforce intervention engagement or
compliance
Explanatory Pragmatic
PRECIS-2: Follow-up
How closely are participants followed up?
Frequent follow-up visits scheduled
outside of clinical encounters, extensive
data collection
Few follow-up visits, outcome data obtained
through EHR, questionnaires, or other data sources
Explanatory Pragmatic
PRECIS-2: Primary outcome
How relevant is it to participants?
Surrogate outcomes or
measures distinct from the research
question
Outcomes of importance to
patients, measured as they would be in
usual care
Explanatory Pragmatic
PRECIS-2: Primary analysis
To what extent are all data included?
Intent-to-treat analysis
Explanatory Pragmatic
Excludes noncompliant participants, dropouts, or
practice variability
Pragmatic Elements:An Introduction to PRECIS-2
Resource: Using PRECIS-2
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• For each domain of PRECIS-2, determine the approach along the pragmatic-explanatory continuum that is most appropriate for answering your research question
• Remember that trials may have some elements that are more pragmatic and some that are more explanatory
Important things to do
2: Engaging Stakeholders & Aligning with Healthcare
System Partners Contributing author:
Leah Tuzzio, MPH, Kaiser Permanente Washington Health Research Institute
Learning goals
• Understand the range of stakeholders to engage and how to partner with them through all phases of the trial
• Identify strategies for understanding the priorities and perspectives of health system leaders and obtaining their support
• Be patient: relationships take time to build and nurture• Get to know your stakeholders, their values, priorities,
and expectations • Consider whether your intervention will add long-term
value to the health system and its patients• Assess the capacity and capabilities of your health
system partners• Engage across all trial phases: design, conduct, and
dissemination
Important things to know
Lessons from NIH Collaboratory
LISTEN TO THE FRONTLINE “The purpose of the healthcare system is not to do research, but to provide good
healthcare. Researchers often have a tail-wagging-the-dog problem. We assume if we think something is a good idea, the
healthcare system will too … We need to remember that we’re the tail and the
healthcare system is the dog.”
– Greg Simon, MD, MPH (SPOT)
• Identifies priorities and perspectives early and throughout the research continuum
• Defines relevant questions and selects high-priority outcomes
• Improves efficiency and diversity of participant enrollment
• Continuously helps improve methods and overcome challenges
• Reduces missing data and loss to follow-up• Increases the uptake and impact of research
What’s the value of engagement?
• Potential stakeholders have varied priorities, values, work cultures, and expectations:• Healthcare delivery organization leaders• Clinicians• Operational personnel• Patients, caregivers, patient advocacy groups• Payers, purchasers• Policymakers, regulators• Research funders• Researchers• Product manufacturers
Who are ePCT stakeholders?
• The wider community of stakeholders is needed to define the question and design the intervention• “We really want to know what you need”
• Local stakeholders are essential to implementing the ePCT at sites• “We really need your help to get this done”
Types of stakeholders
Source: Greg Simon, MD, MPH
Determine which stakeholders are important for your trial
Who can help minimize potential barriers to study
completion?
Who will use the evidence from the
study to make decisions? Who will be affected by those
decisions?
Important things to do
Deciding Who To Engageand
Stakeholder EngagementThroughout the PCT Life Cycle
Resource: Engaging stakeholders
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Carefully choose the research question• Design the intervention for sustainability • Select meaningful outcome measures• Design the protocol to minimize burden on patients
and clinicians • Promote and support the study
Engagement strategies in the design phase
• Develop recruitment strategies• Promote and assess compliance with study
requirements (eg, regulatory)• Engage study champions at each site• Solve problems and remove barriers • Consider privacy and data sharing issues
Engagement strategies in the conduct phase
USE EXISTING WORKFLOWS“The more complicated the
intervention is to the existing workflow, the more difficult it is to
get compliance—you can’t just add on a new thing, you have to change
what happens on the floor.”
– Vincent Mor, PhD (PROVEN)
Lessons from NIH Collaboratory
Challenge SolutionIntervention is in the primary care settingwhere schedules are busy and space is tight
Team with clinicians to understand workflow and schedule study-related patient visits during slower clinic periods
Hold patient visits in less conventional ways (eg, after hours, meet in lobby spaces)
Nurturing relationships: challenges and solutions
Challenge SolutionHigh amounts of leadership turnover at medical director and provider levels due to preexisting pressures and challenges inherent in community clinics
Meet regularly with leadership teams and establish an advisory board and other infrastructure to help engage leaders and gatekeepers
Nurturing relationships: challenges and solutions
Challenge SolutionLeadership approval of the study is delayed because different departments within a single healthcare system are unable to initiate approval before another department does
Hold in-depth discussions of the project with all relevant stakeholders attending in person or by phone or web
A prior history of collaboration among investigators and health system leadership can be instrumental in obtaining approval
Nurturing relationships: challenges and solutions
• Determine key messages for different stakeholder groups
• Identify avenues for dissemination• Assist with the development of manuscripts and
other dissemination materials• Share findings via professional networks and
social media• Support implementation or de-implementation
of intervention• Consider changes to policies and guidelines
Engagement strategies in the dissemination phase
Lessons from NIH Collaboratory
DON’T START FROM SCRATCH, ADAPT
“Each system is going to implement the trial in a slightly different way that works best for them and their
workflows.”
– Miguel Vazquez, MD (ICD Pieces)
Prepare a brief, clear abstract that includes:
Reasons to invest in
intervention
Potential harms or liability issues
Alignment with organizational
priorities
Downstream implications
Alignment with policy makers
Impact on workflows
Sustainability plans
Important things to do
Engaging Stakeholders andBuilding Partnerships to
Ensure a Successful Trial
Resource: Engaging stakeholders
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
3: Designing with Implementation in Mind
Contributing author:Doug Zatzick, MD, University of Washington School of Medicine
Consider how to design ePCTs so findings can be successfully implemented and sustained in real-world healthcare settings
Learning goal
• Pragmatic trials can simultaneously address effectiveness and implementation aims
• Health systems vary in how they change practice based on evidence from a clinical trial
• Methods that integrate pragmatic trials and implementation science frameworks are in development
Important things to know
To design the trial with implementation and sustainability in mind:• Consider how your intervention
fits with the target patient population and setting
• Think about whether your intervention can be delivered in a variety of healthcare settings
Consider implementation early
Key Considerations
Resource: Upfront design considerations
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
If you build it . . . they will come
David Chambers, May 24, 2017, NIH Collaboratory Workshop
Translated to ePCTs:
If you build it pragmatically … they will implement
• Effectiveness aim: Reduce PTSD symptoms
• Implementation aim: Influence U.S. trauma center requirements for sustainable PTSD screening and intervention procedures
Case study from NIH Collaboratory: Trauma Support and Outcomes (TSOS)
• Embed implementation team in the health system
• Spend time in clinical context of trauma care system
• Conduct participant observations• Record field notes and key informant
interviews• Review and document themes related
to trial roll-out and sustainable implementation
Some TSOS implementation tasks
• LIRE tests effectiveness of a simple, inexpensive intervention: Epidemiologic benchmarks inserted into lumbar spine imaging reports
• Total patient N ~250,000• Rated highly pragmatic using PRECIS-2
tool• Stepped-wedge design leaves
intervention “turned on” after study completion
Case study from NIH Collaboratory: Lumbar Imaging with Reporting of Epidemiology (LIRE)
• A few providers/radiologists/clinics: • Adopted the intervention before the
start of the trial • Selectively removed the intervention
from reports• Temporarily discontinued the
intervention during the trial• Can potentially contaminate
comparison groups and interrupt sustainability
• Requires communication between study team and system leadership to find practical solutions
Some LIRE implementation challenges
• What are the needs of those who will use the research findings to make decisions?
• Who is able to deliver the intervention?• Build in tests of training, support, and
adherence/fidelity• During trial roll-out, remove barriers
to high-quality, sustainable intervention delivery
Upfront considerations
Plan for:
How the trial addresses effectiveness
How the trial addresses sustainable implementation
Key policy or practice changes to enhance
sustainable implementationHow the health system
learns
Important things to do
Designing with Implementation and Dissemination in Mind
Resource: Designing with implementation in mind
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
4: Design and Analytic Considerations
Contributing authors:Liz Turner, PhD, Duke University School of Medicine
Liz DeLong, PhD, Duke University School of Medicine
• Determine which randomization scheme makes sense for your ePCT
• Understand special considerations with clustered data• Recognize the analytical challenges of cluster-
randomized and stepped-wedge study designs
Learning goals
• Question drives design, design drives analysis• Randomization
• Individual preferred for statistical reasons• But cluster often needed (cluster-randomized [CRT] design)• Avoiding informed consent is not a reason to favor a CRT design
• Considerations in both design and analysis• Must account for clustering (if CRT design)• Best to account for baseline imbalance
• Good design is difficult but critical • Need input from diverse team, including statistician• Analysis may not be able to overcome design flaws
Important things to know
Introductionand
Statistical Design Considerations
Resources: Introduction to experimental designs
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Deciding on the level of randomization
Individual Cluster
• Examples: patient, caregiver
• Often used in explanatory trials
• Examples: clinic, hospital, region
• Often used in pragmatic trials
Cluster-randomized trial (CRT)or group-randomized trialor community-randomized trial
When the unit of randomization is a cluster, the trial is called . . .
• Target of intervention is a collective instead of an individual• For example, comparing 2 protocols for handling in-hospital infections
or comparing 12-hour nursing shifts to 8-hour shifts
• Target of intervention is an individual, but there is risk of contamination• Contamination occurs when aspects of an intervention are adopted by
members of the group that was randomized to not receive that intervention
• For example, physicians randomized to a new educational program might inadvertently share lessons learned from the program with physicians in their practice that were randomized to control
• Contamination reduces the observed treatment effect
• Logistically easier to implement intervention by cluster
Reasons to randomize clusters instead of individuals
• STOP CRC is a CRT testing a culturally tailored, health system–based program to improve CRC screening rates in community-based collaborative network
• 26 clinical sites• 40,000+ patients
Cluster-level randomization in an NIH Collaboratory study: Strategies and Opportunities to Stop Colorectal Cancer (STOP CRC)
STOP CRC cluster randomization levels
Reduces power to detect treatment effect if same sample size used as under individual randomization
Individual-level outcomes within same clinic expected to be correlated with each other (ie, to cluster)
Level 2: Randomization at clinic (ie, cluster) level
Level 1: Individual-level outcomes nested in clinics
STOP CRC intervention
Screening
Factors related to uptake of CRC screening(eg, age, gender)
• SPOT is a collaborative care model testing treatments intended to reach large groups of adult patients who have serious thoughts of suicide
• 4 clinical sites• 16,000 expected patients
Individual-level randomization in an NIH Collaboratory study: Suicide Prevention Outreach Trial (SPOT)
• Two active arms• Both interventions are individual-level• Intervention contact mostly through electronic health record,
so low risk of contamination is expected
SPOT individual randomization
SPOT study flow
Simon GE, et al. 2016. Trials. 17:452. doi:10.1186/s13063-016-1566-z. PMID: 27634417
Cluster-Randomized Trialsand
Choosing Between Clusterand Individual Randomization
Resources: Randomization schemes
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• When unit of randomization is a cluster or . . .• When unit of randomization is an individual AND the
individual outcomes demonstrate some clustering
When are special statistical analyses needed to accommodate clustering?
What does it mean to say that an outcome is clustered?
• Suppose 10 clinics• Each with 5 age-eligible patients: ie, are not up to date
with colorectal cancer (CRC) screening• Outcome
• Binary outcome: refused screening (Y/N)• No screening within year of enrollment
Clustering of a particular outcome
Understanding clustering: complete clustering
>1 participant/clinic gives no more information than asingle participant/clinic since every participant in a given
clinic has the same outcome
ScreenedNot screened
Understanding clustering: no clustering
ScreenedNot screened
20% uptake of CRC screening in each clinicNo structure by clinic; more like a random sample
of eligible participants
Understanding clustering: some clustering
A more typical situation: proportion screened ranges from 0% - 80%
ScreenedNot screened
ICC, ρ:• Most commonly used measure of clustering• Ranges: 0-1; 0=no clustering; 1=total clustering• Typically <0.2; commonly around 0.01 to 0.05• Ratio of between-cluster variance of outcome to the
total variance
Measure of clustering: intraclass correlation coefficient (ICC)
ICC for continuous outcomes:
Involves both between-cluster and within-cluster variance
• Need measure of clustering for sample size• CV is an alternative to ICC:
• Multiple definitions of ICC for binary outcomes (some authors prefer CV for binary)
Measure of clustering: ICC & coefficient of variation (CV)
µσ Bk =
where μ is overall mean of outcome
Analysis Planand
Intraclass Correlation
Resource: Design and analysis
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Typically use regression-type models of individual-level data• Random effects/mixed effects models• Generalized estimating equations
• Important: Work with a statistician to ensure correct accounting for clustering
Analytic approaches for clustered data
• Power is affected by . . .• Strength of the clustering effect (eg, size of ICC)• Number of clusters• Number of patients per cluster
Adjustment for clustering in the analysis will require larger sample size to have adequate power
Higher ICCs and fewer clusters = lower power (so need to increase sample size to compensate)
ICC=0.03 (ie, like STOP CRC power calculation)
26 clusters - 450/cluster
20 clusters - 585/cluster
32 clusters - 365/cluster
Note: this is the total # clusters across both arms
• Power and sample size• Account for anticipated clustering• Inflate RCT sample size • Work with statistician to do correctly
• Use ICC (or CV) for outcome • ICC often 0.01-0.05• STOP CRC: ICC = 0.03 for primary outcome• Depends on outcome & study characteristics • Different outcome = different ICC, even in same CRT
Accounting for clustering in design
• How to get good initial estimate of ICC for a particular outcome?• It depends on outcome and study characteristics • CONSORT statement on reporting of CRTs recommends
ICC reported• Look at other articles with similar settings
• Be cautious when using pilot data from small study• The ICC might have a wide confidence interval
Estimating ICC to plan study
Design considerations: clustering in STOP CRC
“Assumed equal numbers of subjects per clinic and equal numbers of clinics (n = 13) per group. In practice, the clinic sizes will not be equal, but since almost all clinics have at least 450 active age-eligible patients, we conservatively use this figure for all sites. We based our calculations on the simple paradigm of comparing two binomial proportions with a type I error rate of 5%, and adjusted both for intraclass correlation (ICC) and the reduced degrees-of-freedom (n = 24) for the critical values. Based on analyses by Dr. Green using the data from her Systems Of Support study [12,28], we expect the ICC to be about .03. Using this figure, we will have very good power (>91%) to detect absolute differences as small as 10 percentage points even if the FIT completion rate in the UC arm is as high as 15% (fecal testing rates for 2013 for usual care clinics was 10%). For an ICC of .05 we would still have >91% power for detecting effect sizes of at least 13 percentage points.”
Coronado GD, et al. 2014. Contemp Clin Trials. 38:344-349. doi:10.1016/j.cct.2014.06.006. PMID: 24937017
• Pragmatic CRTs often enroll small # of clusters (<40) • Randomization may not balance baseline covariates• Baseline covariate imbalance threatens internal
validity; ie, comparability of treatment arms• There may be confounding due to non-comparability of
treatment arms
Special consideration for CRTs: Greater potential for imbalance on baseline covariates
Addressing imbalance in baseline covariates
Adjust for imbalance at analysis stage
Prevent imbalance at design stage
Not recommendedRecommended
Restricted randomization
• Use restricted randomization if • Total # clusters <40, and . . .• Know which baseline covariates are predictive of outcome
• Multiple approaches possible• Pair-matching • Stratification• Covariate-constrained randomization • Consult a statistician to choose!
• Analysis must account for whatever type of restricted randomization is used in design
Using restricted randomization
• If you are planning a cluster-randomized design, what cluster-level covariates might be important to balance on?
Consider …
Randomization Methods
Resource: Randomization methods
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• CONSORT extension for cluster RCTs• Recommends at least 4 clusters/arm • This is just a guide
• Statistical reasons may require many more than 8 clusters in total in a 2-arm trial!
• Remember: # clusters drives the power of trial more than # participants
• CRTs require a lot of time and effort• Consider a pilot trial to get procedures in place
Number of clusters: How low can you go?
1. Parallel CRT2. Stepped-wedge CRT
2 major types of CRT
Two types of CRT designs
Stepped-wedgeParallel
IncompleteComplete
In complete designs, measurements are taken from every cluster at every
time point. In incomplete designs, some clusters do not provide measurements
at all time points.
Examples with 8 clusters: 1-year intervention
Types of CRT designs
Complete stepped-wedge design
Incomplete stepped-wedge design
0 1Time since baseline
2 3 4 0 1 2 3 4Time since baseline
Control period Intervention period
Based on: Hemming K, Lilford R, Girling AJ. 2015. Stepped-wedge cluster randomised controlled trials: a generic framework including parallel and multiple-level designs. Stat Med. 34:181-196. doi:10.1002/sim.6325. PMID: 25346484
Parallel design
0 1Time since baseline
Cluster 1
Cluster 8
......
Complete stepped-wedge design
Incomplete stepped-wedge design
0 1Time since baseline
2 3 4 0 1 2 3 4Time since baseline
Parallel design
0 1Time since baseline
Cluster 1
Cluster 8
......
Types of CRT designs
Post-intervention period
Examples with 8 clusters: 1-year intervention
Control period Intervention period
Complete SW design
0 1Time since baseline
2 3 4
Control period Intervention period
Parallel design
0 1Time since baseline
CRT analysis: treatment effects
Estimated (primarily) using between- cluster ie, vertical information
Estimated using both vertical & horizontal (ie, within-cluster) information
• Arguments for stepped-wedge CRT:• Cannot immediately implement intervention in half the clusters • Pragmatic research: plan to eventually implement in all clusters • Have few clusters and might gain power
• Arguments against stepped-wedge CRT:• Risk confounding treatment effect with time effect• Could do staggered-start parallel CRT if cannot start
implementation in half the clusters immediately• Roll out to all clusters at end of evaluation, if effective
Choosing the right type of CRT
• Use a parallel CRT design if you can• If not, plan for time effects in designing and analyzing
stepped-wedge CRT• Work with statistician to account for clustering in
design and analysis of both designs
Recommendations for CRT Design
If you are planning a cluster-randomized design, what are the pros and cons of using a parallel versus stepped-wedge design for your trial?
Consider …
• Intent-to-treat (ITT) versus per-protocol analysis• Concealment and blinding• Monitoring and managing unexpected changes
Other considerations for ePCTs
• Pragmatic nature ITT commonly used• Per protocol often difficult to define
• Screening yes/no is easy• Other interventions might have degrees of adherence to
protocol • Might be interested in other types of treatment effect
• Average treatment effect on the treated
Intent-to-treat vs per protocol analysis
• Concealment of randomization assignment to avoid selection bias• Less a problem in CRTs than RCTs if clusters all
randomized together • Blinding (masking)
• May not be possible or practicable for CRTs• Objective assessment criteria should be consistently applied
Concealment and blinding
• Study designs can be affected by• Changes in study populations• Changes in coverage patterns• Changes in patient perceptions/decisions• Decisions by hospital/health system leadership• Changes in regulations or practice standards• Site turnover
• Careful planning and monitoring are needed
Managing unanticipated changes
Concealment and Blindingand
Unanticipated Changes
Resources: Other design considerations
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Someone who …• Wants to be involved from beginning of development
of research proposal• Has experience with pragmatic trials and is familiar
with the PRECIS-2 tool• Has experience with using EHR data• Has experience with CRT design and analysis (if using
a clustered design)
How do I know I have the right statistician?
Focus on research question
Write & publish a protocol paper
Collaborate early with a statistician
Choose individual randomization, but
only if possible
Weigh statistical choices vs
implementation challenges
Select design features with
analysis in mind
Important things to do
Additional Resources
Resource: ePCT design and analysis
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
5: Regulatory and Ethical Challenges
Contributing author: Kevin Weinfurt, PhD, Duke Clinical Research Institute
Learn about the regulatory and ethical considerations specific to conducting ePCTs
Learning goal
• Ethical analysis for ePCTs is a work in progress• Federal and local policies regarding the oversight of
ePCTs are in flux• There is often confusion and misunderstanding
about ePCTs on the part of patients, providers, IRBs, and DSMBs
Important things to know
ePCTs also raise interesting ethical and regulatory questions
ePCTs are motivated by ethical imperatives
• Informed consent• Data monitoring• Defining minimal risk• Research/quality
improvement distinction• Vulnerable subjects• IRB harmonization
Evolving understanding of unique ethical/regulatory issues for ePCTs
• Identifying direct and indirect subjects
• Gatekeepers• FDA-regulated products• Nature of ePCT
interventions• Privacy
Regulatory & ethical challenges of ePCTs
Ethical, not regulatory, question:Whose rights and welfare need to be protected?
Introductionand
Informed Consent
Resources: Regulatory & ethical challenges of ePCTs
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Current ethics/regulatory in fluxDelayed compliance date of revised final
Common Rule
Your dedicated ethics/regulatory
liaison
Direct Indirect
Types of participants in an ePCT
Immediate or mediated targets of the intervention
Direct participants
Intervention
Intervention
Intervention
Patients
Providers
Clinics
Direct participant
Intervention
Immediate target Mediated target
People affected by routine exposure to the environment (eg, family/caregivers)
Indirect participants
Intervention
• Cluster trial comparing 2 quality improvement strategies to reduce multidrug-resistant organisms and healthcare-related infections in non-ICU population
• 53 hospitals• 331,584 patients
Case study from NIH Collaboratory: Active Bathing to Eliminate (ABATE) Infection
Indirect participants: ABATE example
Routine Care Decolonization
• Who are the direct and indirect participants for your study?
• What are the potential risks and benefits for each?
Consider …
Alternative Approaches to Disclosure and Authorization
Resource: Alternative approaches
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Informed consent
Alterations
Nondisclosure
Broad notification Opt-out
Opt-in
Approaches to notification & authorization
An IRB may waive or alter the requirements of informed consent if all of the below are deemed true:• “The research involves no more than minimal risk to the
subjects• The waiver or alteration will not adversely affect the rights
and welfare of the subjects• The research could not practicably be carried out without
the waiver or alteration and• Whenever appropriate, the subjects will be provided with
additional pertinent information after participation” §46.116
Conditions for waiver of consent
“In evaluating risks and benefits, the IRB should consider only those risks and benefits that may result from the research (as distinguished from risks and benefits of therapies subjects would receive even if not participating in the research).”
Common Rule: CFR 46.111 (a)(2)
“The reasonably foreseeable risks of research include already identified risks of the standards of care being evaluated as a purpose of the research.”
From the OHRP Draft Guidance
Minimal risk
Some debate here!!!
• CRT testing whether a longer hemodialysis session can improve survival and quality of life for patients with kidney failure who require chronic treatment with dialysis
• 256 clinical sites• 7053 patients
Case study from NIH Collaboratory: Time to Reduce Mortality in End-Stage Renal Disease (TiME)
• Facility implementation of ≥4.25-hour dialysis session duration improves outcomes compared with usual care
• Patients starting dialysis at participating facilities are given a brief information document with:• Purpose of the trial• How session duration will be affected by the trial• Toll-free telephone number to obtain additional information
from the research team and to opt-out of participation• Informational posters in participating dialysis facilities
throughout the duration of the trial
Consent process: TiME example
• Tests effectiveness of an intervention that inserts epidemiologic benchmarks into lumbar spine imaging reports
• Goal of intervention is to reduce subsequent diagnostic and therapeutic interventions
• 98 clinical sites• 246,289 patients
Case study from NIH Collaboratory: Lumbar Imaging with Reporting of Epidemiology (LIRE)
• Waiver of consent was granted • Risk of contacting subjects deemed greater than
the risk of study procedures• By informing primary care providers and patients,
they risk invalidating the results
Consent process: LIRE example
• Written consent (with clinical risks included)• Written consent• Oral consent + info sheet• Oral consent• General notification (with opt-out)• Post-notification after study done
Approaches to notification & authorization
What do data suggest about different approaches?
Resource: Comparison of approaches
Weinfurt KP, et al. Med Care. 2017 Nov;55(11):970-978. doi: 10.1097/MLR.0000000000000762
Difficulty understanding aspects of pragmatic trials of accepted medical practices
Nontrivial consent bias, but it’s the same for all approaches to notification and authorization
Less active approaches to notification and authorization viewed as unacceptable for some types of pragmatic research
Including descriptions of background clinical risks increased length of form but did not change any outcome
Active alternatives to written consent—such as oral consent—may not be expected to compromise consent quality
1
2
3
4
5
• Institutional review boards (IRBs)• Data monitoring committees (DMCs)
• Data safety and monitoring boards (DSMBs)
Working with human subjects oversight bodies
• NIH policy on single IRB review, effective January 25, 2018
• Revised Common Rule requires U.S.-based institutions engaged in cooperative research to use a single IRB for regulatory review
• The sites involved in research that uses a single IRB need to• Sign a reliance agreement, which outlines who is responsible
for what (usually for each protocol)• Develop systems for fulfilling institutional responsibilities• Develop mechanisms for reporting relevant institutional
information to reviewing IRB
Major issue: single IRB review
• Stepped-wedge CRT testing innovative intervention for patients with PTSD and comorbidity
• 25 level 1 trauma centers• 960 expected patients
Case study from NIH Collaboratory: Trauma Support and Outcomes (TSOS)
• At the time of study initiation, the University of Washington IRB did not have capacity for “centralization”
• Western IRB serves as the centralized IRB• No single administrative contact• Only 4 sites “cede” to centralized WIRB review• 20 individual site IRB submissions (out of 24 sites)
“Single” IRB experience: TSOS example
Group of experts that reviews the ongoing conduct of a clinical trial to ensure continuing patient safety as
well as the validity and scientific merit of the trial
Data monitoring committee
• Poor adherence to intervention: problem or finding?• Inference about adverse events
• Availability of clinical data to assess relatedness• Should adverse events still be monitored?
• Limited or delayed access to study outcomes during study conduct
• Are interim analyses actionable?
Unique considerations for monitoring ePCTs
Adapted from Greg Simon, PCT Grand Rounds, December 8, 2017
• Describe current practices and beliefs• Test assumptions of an ethical argument• Measure potential impact of different
regulatory policies
Collect data to contribute to the learning
Consent, Disclosure, and Nondisclosure
Resource: Regulatory and ethical challenges of ePCTs
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Designate someone to track local and federal regulatory developments and serve as liaison with regulatory/oversight bodies
• Budget sufficient time for proactive education and negotiations with relevant regulatory/oversight bodies
• Identify all parties who might be affected by the study and its findings; consider protections
Important things to do
Special Issue of Clinical Trials
Resource: Additional readings on regulatory/ethical considerations
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
6: Measuring Outcomes
Contributing authors:Rachel Richesson, PhD, Duke University School of Nursing
Lesley Curtis, PhD, Duke Clinical Research Institute
Describe methods for measuring study outcomes using data sources such as electronic health records (EHRs) and patient-reported outcomes (PROs)
Learning goal
• In pragmatic research, endpoints and outcomes need to be available as part of routine care
• Endpoints and outcomes should be • Meaningful to providers and patients• Relatively easy to collect
• Researchers do not control the type or format of data collected in EHR systems
Important things to know
• An endpoint usually refers to an analyzed parameter (eg, change from baseline at 6 weeks in mean PROMIS Fatigue score)
• An outcome usually refers to a measured variable (eg, peak volume of oxygen or PROMIS Fatigue score)
Endpoints and outcomes
• Is the outcome medically significant such that a patient would seek care?
• Does it require hospitalization?• Will it be medically attended?• Is the treatment generally provided in
inpatient or outpatient settings?
Key questions for choosing endpoints
Choosing and Specifying Endpoints and Outcomes
Resource: Endpoints and outcomes
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Data sources for endpoints in ePCTs
“The first challenge in using big biomedical data effectively is to
identify what the potential sources of health care information are and to determine the value of linking
these together.”
Weber GM, et al. JAMA. 2014;311(24):2479-2480. doi:10.1001/jama.2014.4228 (Figure 1)
• EHR or ancillary health information systems• Patient report • Patient measurement
Data sources for endpoints
• Acute MI• Broken bone• Hospitalization
Choosing and specifying endpoints
• Suicide attempts• Gout flares• Silent MI• Early miscarriage
• EHR (laboratory values, treatments, etc)• Claims data (does the event generate a bill?)
Where is the signal?
Payer claimsInpatient
& outpatient
EHR
Overlap
Simon G, Group Health Research Institute
Payer #1
Payer #2
Outpatient EHR A
Outpatient EHR CInpatient
EHR B
Inpatient EHR B
Overlap
Reality is not straightforward
• To fully capture all care—complete longitudinal data—linking research and insurance claims data is often necessary
• Without explicit consent, getting longitudinal data from an insurance carrier can be an insurmountable hurdle, both technically and legally
Longitudinal data linkage
• Are meaningful to providers and patients• Myocardial infarction vs MACE• Clinical event vs blood test
• Are captured reliably as part of routine clinical care
• Do not require central adjudication
More pragmatic outcomes
• Are surrogate outcomes mainly important to providers (eg, blood test)
• Are composite outcomes less important to patients• Involve tests not used in usual care or are outcomes
that require central adjudication• Are shorter term outcomes for a condition in which
patients are more concerned about longer term outcomes
Less pragmatic outcomes
• What is the phenomenon you are trying to identify or measure?
• What are the sources of error, and how can you reduce the error?
Key questions for using data in research
• Identify variation between populations at different sites or study groups
• Recommend formal assessment of accuracy, completeness and consistency for key data
• Data quality should be described, reported, and informed by workflows
Assessing data quality
• Mixed-methods cluster trial evaluating integration of multidisciplinary services within the primary care environment to improve chronic pain management
• 3 regional health systems• 2000 expected patients
Case study from NIH Collaboratory: Collaborative Care for Chronic Pain in Primary Care (PPACT)
• Patient-reported outcomes (PROs) were needed but were not standardly collected across diverse regions
• Study team worked with national healthcare system to create buy-in for a common instrument
• Local IT team built instrument within each region • A multi-tiered approach supplemented the clinically
collected patient-reported data at 3, 6, 9, and 12 months
• Study team needed a follow-up phone call to maximize data collection at each time point
PPACT data challenges
Via Electronic Health Record
Via Direct Patient Report
Via Mobile Devices
Resource: Methods of measuring outcomes
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Data may be transformed/coded for purposes other than research and clinical care
• Data captured in clinical notes may not be available • EHRs are often highly customized• EHRs may present multiple sources of similar data• EHRs often do not tell a complete story
Caveats when using EHR data for research
• PROs are often the best way to measure quality of life
• Challenge is that PROs are not routinely or consistently used in clinical care and not regularly recorded in the EHR
• Need a mechanism to collect PROs
Direct patient report
• Smartphones, tablets, and portable, implantable, or wearable medical devices (mHealth)• Some mHealth devices transmit data to a data
warehouse every night• Largely considered imperfect measures
• Patient-facing mobile phone apps can be used in ePCTs for passive or active surveillance
Mobile devices
• Clearly define primary and secondary outcome measures
• Report methods used to enhance the quality of measurements
• Explain how selected outcomes and length of follow-up are important to stakeholders
Consider ePCT reporting guidelines when choosing outcomes
General Considerationsand
PCT Reporting Template
Resource: Reporting secondary use of EHR data
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Richesson RL, et al. 2017. J Am Med Inform Assoc. 24:996-1001. doi:10.1093/jamia/ocx016. PMID: 28340241
• Need to capture intervention or control activities
• Including standard of care• Need to enable learning
and research activities into EHR functions
• Competition for IT resources
• Need to optimize clinical data for research
• Only small proportion of research in EHRs
• Ask questions that the data will support, and design trials to minimize new data collection
• Engage EHR and data experts when defining endpoints and outcomes
• Budget for data and systems experts at each site (… and then double it)
• Develop a robust data quality assessment plan to improve value of data and to detect and address data issues
Important things to do
7: Pilot and Feasibility Testing
Contributing author:Wendy Weber, ND, PhD, MPH, National Center for
Complementary and Integrative Health (NCCIH)
Identify approaches to evaluate the capabilities and challenges of the partner healthcare system and test key elements of the intervention
Learning goal
• Pilot testing the ePCT methods increases likelihood of completing the trial and can prevent silly mistakes
• You need a biostatistician in the pilot/feasibility stage• “Process issues” can derail the ePCT• Use the pilot study to maximize acceptability, maintain
affordability, and consider scalability of your intervention
Important things to know
• ePCTs bridge research into clinical care
• Intervention is integrated into a real-world healthcare settings
ePCTs are not efficacy trials
• Establish close partnerships with healthcare system (HCS) personnel
• Test and validate EHR data collection and extraction• Assess how well the intervention can be integrated
into the clinical workflow • Identify local champions at each study site
During the pilot phase
• Is the intervention aligned with the priorities of the partner HCS?
• How ready is the partner?• Are extra resources needed to support the
intervention, identify participants, and extract necessary data?
• How many sites are available to fully participate?• How much provider training will be needed, and
can training use existing HCS infrastructure?• If the intervention proves successful, what
adaptations would be needed to implement it in other healthcare settings?
Build partnerships
Establishing Close Partnerships with Healthcare Systems Leaders and Staff
Resource: Health system partnerships
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Aspects of feasibility that can be piloted
Verify that target population can be identified via the
EHR
Coordinate processes with
local champions
Test phenotypes needed for sample
identification
Test data sample for quality &
accuracy
Test the training materials for
frontline providers & staff
Validate data collection &
extraction methods
Evaluate informed consent materials
• Eligibility• Recruitment• Randomization• Adverse events• Retention• Missing data• Intervention fidelity
Quantify feasibility for pilot study aims
• Demonstrate effective recruitment and retention, which is defined as the ability to recruit an average of 10 patients per month per site and retain 80% of participants for final data collection at 6 months
Quantifying example 1
• Determine whether the intervention can be delivered with reasonable feasibility, defined as 70% of the enrolled participants engage in the intervention
Quantifying example 2
• Demonstrate ability to collect primary outcomes and minimize missing data to less than 5% of primary outcome measures
Quantifying example 3
• If cluster randomization is involved, collect data to confirm estimate of intraclass correlation (ICC) for power calculations
Evaluate power calculations
Pilot Testingand
Feasibility Scenarios
Resource: Pilot and feasibility testing
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Collaborative care model to test treatments intended to reach large groups of adult patients who have serious thoughts of suicide
• 4 clinical sites• 16,000 expected patients
Case study from NIH Collaboratory: Suicide Prevention Outreach Trial (SPOT)
• An NIH Collaboratory Demonstration Project in UH3 phase
• Gregory Simon, MD, MPH, Principal Investigator, Kaiser Permanente Washington Health Research Institute
• Watch the 9-minute webinar (on Vimeo)
Pilot testing in SPOT
• Avoiding silly mistakes• Maximizing acceptability• Maintaining affordability• Remembering scalability
In the end, it’s about
Spotlight on Four Demonstration Projects
Resource: More feasibility examples
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Troubleshooting and iterative testing• Flexibility to accommodate local conditions and
changes over time• Continuous engagement with healthcare system• Readiness tasks
• Recruitment plans are finalized• Ethical/regulatory aspects are addressed• Intervention is fully developed and finalized• Data collection methods are adequately tested• Budget and timeline are realistic and feasible
Ensuring trial readiness
Readiness checklistMilestone CompletedRecruitment plans are finalized
All sites identified (documentation of site commitment)Methods for accurately identifying participants validatedAll agreements for necessary subcontracts in place
Ethical/regulatory aspects are addressedCoordinated IRB oversight in placeFinalized plans for informed consent or waiver of informed consentFinalized data and safety monitoring plan
Intervention is fully developed and finalizedFinalized intervention (including materials and training at sites) ready for site implementation
Finalized protocol is IRB approved (informed consent and data collection forms, if applicable)
Data collection methods are adequately testedValidated methods for the electronic health record informationValidated study surveys, interviews, or other data collection modesDemonstrated quality assurance and harmonization of data elements across healthcare systems/sites
Statistical and data analysis methods have been adequately developed
Budget is realistic, feasible, and accounts for potential changes
Implementation Readiness Checklist
Resource: Trial readiness criteria
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Conduct a pilot or feasibility study of the ePCT intervention
• Work with a great biostatistician and an informatician (if needed)
• Develop a partnership approach to working with your healthcare system
• Identify local champions for all your sites• Anticipate, identify, and make a plan to address
changes in the healthcare system
Important things to do
8: Dissemination of Results
Contributing author:Doug Zatzick, MD, University of Washington School of Medicine
Identify considerations and approaches for dissemination of study results
Learning goal
• Dissemination and implementation (D&I) science can inform the translation of ePCT results into healthcare system practice change
• Case examples from NIH Collaboratory demonstration projects suggest a number of possible approaches to the dissemination of trial results
Important things to know
Dissemination research
NIH Dissemination and Implementation Research in Health PAR-16-238
The scientific study of targeted distribution of information and
intervention materials to a specific public health or clinical
practice audience
The intent is to understand how best to spread and sustain knowledge and the associated
evidence-based interventions
Implementation research
NIH Dissemination and Implementation Research in Health PAR-16-238
The scientific study of the use of strategies to adopt and integrate
evidence-based health interventions into clinical and community settings in order to improve patient outcomes and
benefit population health
D&I Introduction
Resource: Dissemination and implementation
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Randomized evaluation of decolonization versus universal clearance to eliminate methicillin-resistant staphylococcus aureus (REDUCE MRSA)
Dissemination and implementation together: Case study
• Large-scale cluster trial conducted in hospital intensive care units (ICU)
• Tested whether targeted decolonization of MRSA carriers versus universal decolonization of all ICU patients was the most effective intervention
REDUCE MRSA
Universal decolonization led to:• 37% reduction in MRSA clinical cultures• 44% reduction in bloodstream infections
REDUCE MRSA findings
REDUCE MRSA manuscript
Created for clinicians by clinicians, the toolkit is designed to serve as a roadmap for hospital champions and frontline staff
REDUCE MRSA toolkit
• Introduction and welcome• Universal ICU decolonization protocol overview• Scientific rationale• References• Appendices include training and educational materials
Toolkit contents
REDUCE MRSA toolkit available on AHRQ website
Dissemination Approaches For Different Stakeholders
Resource: Dissemination and implementation
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Effectiveness aim: Reduce PTSD symptoms
• Implementation aim: Influence U.S. trauma center requirements for sustainable PTSD screening and intervention procedures
Case study from NIH Collaboratory: Trauma Support and Outcomes (TSOS)
American College of Surgeons guidelines• Main outcome paper and other publications aim to be
cited in College Resources Guide• End-of-study policy summit aims to integrate findings
into College regulatory/verification processes
TSOS dissemination aims to “nudge” practice change through regulatory policy
PTSD and comorbidity:“The incorporation of routine trauma center–based screening and intervention for PTSD & depression is an area that could benefit from the ongoing integration of emerging data and evolving expert opinion.”
American College of Surgeons Resources Guide
Changes to Policies and Guidelines
Resource: Dissemination and implementation
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
• Consider plans for dissemination of your ePCT results• Data sharing can be an essential element of
dissemination• How do your dissemination plans align with NIH data
sharing guidelines?
Important things to do
Data Sharing and Embedded Research
Resource: Dissemination and implementation
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
9: ePCT Team Composition
Contributing author:Lesley Curtis, PhD, Duke Clinical Research Institute
Identify ideal composition and skills needed for your ePCT study team
Learning goal
• ePCTs are a team sport• Necessary expertise depends on the study aims and
how the intervention will be implemented
Important things to know
Who is involved?
Healthcare systempartners delivering
the intervention
Team designing the study
Potential team members
PI/Co-PI Clinical staff HCS leader or executive Lead clinician
Information technology specialist
Biostatistician Professional society leader Site champion
Practice facilitator
Research assistant
Communications specialist
Project coordinator
Patient or patient advocate
• What clinical specialties will be needed to carry out the intervention?
• What roles will support clinic operations?• Who will be the liaison between HCS
departments for interventions that are multidisciplinary?
• What aspects of the trial will require IT staff expertise?
• Will the trial need training videos, online materials, or toolkits?
Consider
• Identify the types of expertise needed for your trial• Recruit team members during the planning phase
and engage them for the duration of the trial
Important things to do
10: Developing a Compelling Application
Contributing author:Marcel Salive, MD, MPH, National Institute on Aging
Provide trainees information on how to develop a compelling ePCT application
Learning goal
• Online resources are available for the development of pragmatic trial grant applications
• NIH has new policies and forms related to clinical trial grant applications
• Some things, such as milestones and safety monitoring, may be negotiable around the time of an award
Important things to know
• NIH is made up of 27 institutes and centers (IC)
• ICs award >80% of the NIH budget each year
• Each IC has a budget and a director, and typically their own review for large trials
National Institutes of Health
IC mission and priorities• Focus on a specific disease area, organ system, or
stage of life• Use Matchmaker tool in NIH RePORTER for
suggestions• Talk to program officials• Consult your mentor and colleagues
Understand NIH: find the right fit
NIH RePORTER
https://projectreporter.nih.gov/reporter_matchmaker.cfm
Matchmaker results
Under assistance relationships:• Grants (R) are used when no substantial programmatic
involvement is anticipated between the Federal agency and the recipient during performance of the assisted activity
• Cooperative agreements (U) are used when substantial programmatic involvement is anticipated between the Federal agency and the recipient during performance of the assisted activity
• Not necessarily important for developing the application
Grant versus cooperative agreement
Scientific contacts from participating NIH Institutes and Centers
NIH Research Collaboratory: RFA-RM-16-019
NCCIH Robin BoineauNCI Erica Breslau NHLBI Barbara WellsNIA Marcel SaliveNIAAA Brett Hagman NIAID Clayton Huntley NIAMS Chuck WashabaughNICHD Sue Marden
NIDA Sarah DuffyNIDCR Dena FischerNIDDK Andy NarvaNIMH Jane Pearson NINDS Robin ConwitNINR Jeri MillerODP Rachael Ballard
• Overly ambitious—beyond the life/length of the application
• Missing or inappropriate control groups• Lack of sufficient expertise or skilled collaborators
needed to complete the studies• Not sufficient publications in the area of proposed
studies• Insufficient statistical power• Cannot recruit the needed population
Common application pitfalls
Avoid receiving these summary statement comments!
No adequate description of how activities in the planning phase would inform activities in the implementation phase
Amount budgeted for a biostatistician is much
too low
The premise of the study is
based on weak evidence
Concerned whether outcomes of this
study would drive a change in clinical
practice
Data provided did not establish the feasibility
of recruitment
• Pose a clear research question• Convince the reviewer your study is
worth doing• Sell your research plan–highlight the
strengths• Identify weaknesses and explain how
you will deal with them• Tailor your application to the funding
agency• Obtain feedback from your collaborators,
consultants, and others
Strategies for success
• Justify the research• Include pilot data• Reduce complexity• Ensure aims are capable of advancing the field• Choose appropriate expert personnel• Link data collection and analysis to aims• Justify use of multiple sites and sample size
Application dos
• Skip any steps (eg, literature review)• Use dense or confusing writing style• Use appendix inappropriately• Include untestable aims• Include non-relevant aims or fishing expeditions• Assume that prior collaboration is irrelevant
Application don’ts
NIH research methods resources
https://researchmethodsresources.nih.gov
• Read relevant Funding Opportunity Announcement multiple times
• Identify program staff at your target NIH Institute/Center and review your Specific Aims and any questions with them
• Obtain adequate feedback on the Research Plan from the entire team
Important things to do
Worksheet to help study teams get started
Download from the Living
Textbook (PDF)
More learning resources …
From the Living Textbook of Pragmatic Clinical Trialswww.rethinkingclinicaltrials.org
Demonstration Projects ePCT Training Resources
The National Institutes of Health (NIH) Health Care Systems Research Collaboratory Coordinating Center is supported by the NIH Common Fund, through a cooperative agreement from the Office of Strategic Coordination within the Office of the NIH Director (Grant # 1U24AT009676-01). The views presented here are solely the responsibility of the authors and do not necessarily represent the official views of the National Institutes of Health.
Funding Statement