1
Incorporating Virtual Patients into Clinical Studies
Adam Himes, Tarek Haddad, Medtronic
Laura Thompson 1 , Telba Irony2 , Rajesh Nair1
1CDRH / FDA, 2CBER / FDA
on behalf of MDIC working group colleagues:
Dawn Bardot, MDIC / Medtronic
Anita Bestelmeyer, BD
Dan Cooke, Boston Scientific
Mark Horner, ANSYS
Russ Klehn, St. Jude Medical
Tina Morrison, OSEL / FDA
Kyle Myers, OSEL / FDA
Val Parvu, BD
MDIC.org
2 http://www.nejm.org/doi/full/10.1056/NEJMra1512592
“If it can be shown that these virtual patients are similar, in a precisely defined way, to
real patients, future trials may be able to rely partially on virtual-patient information,
thus lessening the burden of enrolling additional real patients.”
3
Outline
• Motivation
• Virtual patients: physical + probabilistic modeling
• Combining virtual patients with clinical data
• Test drive: mock submission
• Review
4
Motivation
Rising Demands:
• Duration
• Size
• Data
New Factors:
• Diseases
• Markets
• Cost ModelsJ. Diabetes Sci Tech (2009) 3(1) 44-55
Murbach, et.al. 2017 BMES/FDA Frontiers
Lee, et.al. 2017 BMES/FDA Frontiers
Dharia, et.al. 2016 BMES/FDA Frontiers
Kuntz, Insights on Global Healthcare Trends (2013)
Global demand:C
OS
T
15% year/year
TIME
Modeling advances:
Our ability to simulate clinical
outcomes has never been better
Demand for clinical evidence has
never been higher
Yet it is still challenging to incorporate prior information into new studies
5
Virtual Patients as a New Source of Evidence
Bench
Animal
Human
Computer
Traditional
Virtual Patient
Computer
Bench
Animal
Human
• Integrate the virtual patient in clinical study design
• Use Bayesian statistical methods
• Build on 2010 FDA Guidance
• Maintain clinical and statistical rigor
Future
6
What Makes a Virtual Patient?
Physical Modeling
Probabilistic Modeling
Clinically Relevant
Predictions
Variability:
• Age
• Gender
• Activity level
• Implant factors
• Physical tolerances
Uncertainty:
• Sample size
• Measurement error/bias
• Model bias
Well Characterized Physics:
• Structural
• Electrical
• Heat transfer / fluid flow
Knowledge of Biology /
Physiology:
• Local device ↔ tissue
interactions
• Failure modes
• Insulin response
Clinically Relevant End
Points:
• ICD lead fracture
• Orthopedic implant
survival
• Coronary artery flow
• MRI heating
• Cardiac rhythm
detection
• Blood glucose level
Like running a
virtual clinical
study!
7
Sources of information for virtual
patient models
Virtual patient outcomes have to be exchangeable with
something you’d measure in a clinical study.
• Historical data
−Pilot studies, other geograhies, similar predicate products
• Real-world data
−Electronic medical records, claims data, observational studies
• Engineering and physiological models
Credibility is the biggest hurdle.
8
Virtual Patient Example: ICD Lead Fracture
• Medical devices are particularly well suited to virtual patient modeling
− Local vs. systemic, often iterative, method of action is usually well understood
• Many applicable models, implantable defibrillator lead fracture is a good example
− Relevant, simple, public domain examples
Haddad, et.al., Reliability Engineering and System Safety, 123 (2014): 145-157.
Swerdlow, et.al., JACC, 67 (2016): 1358-1368.
9
Virtual Patient Example: ICD Lead Fracture
• Simulate many combinations of virtual patients
& clinical trial
• Propagate variability and uncertainty to
predict survival with confidence bounds
Field data
Projection with
95% Confidence
Interval
in-vivo
bending
patient
activity
fatigue
strength
INPUT OUTPUT
Haddad, et.al., Reliability Engineering and System Safety,
123 (2014): 145-157
10
Bayesian Statistical Methods
• How much influence is given to
the prior data?
• What if the clinical study data
disagrees?
Challenges: Solution:
• Method developed by MDIC
working group
• Influence of prior data determined
by agreement with study data
• Maintain statistical power with
fewer patients
influen
ce
disagree agreedisagree agree
(ideal state)
prior data
study data
discount
function
Provide a way to incorporate prior data into analysis of a clinical study
Haddad, et.al. (2017). J. Biopharm Stat, 27(6), 1089-1103.
DOI: 10.1080/10543406.2017.1300907
11
Using a Discount Function to Control
the Influence of Virtual Patients
• Influence determined by agreement
between real patients and virtual patients
− Defined before starting the clinical study
− Maximum depends on model maturity
− Shape of function depends on desired characteristics
• If virtual and real patients disagree:
− Number of virtual patients decreases
− Eventually converts to a traditional study
• If virtual and real patients agree:
− Number of virtual patients increases up to pre-
specified maximum
Vir
tual patient
weig
ht
Statistical
agreement
agreedisagree
more
conservative
less
conservative
Haddad, et.al. (2017). J. Biopharm Stat, 27(6), 1089-1103.
DOI: 10.1080/10543406.2017.1300907
12
Incorporate Virtual Patients: Step By Step
1. Compare virtual
patient and current data
2. Compute strength of
prior using discount function
3. Combine virtual
patient and current data
4. Statistical analysis
using combined data
virtual patient data
current data
𝑝
𝑝
𝛼0
𝑛𝑡𝑜𝑡𝑎𝑙 = 𝑛𝑐𝑢𝑟𝑟𝑒𝑛𝑡 + 𝛼0𝑛𝑉𝑃
combined data
This part
is new
Haddad, et.al. (2017). J. Biopharm Stat, 27(6), 1089-1103.
DOI: 10.1080/10543406.2017.1300907
13
Implementation: Mock Submission
• Collaboration between MDIC and FDA CDRH Division of Cardiovascular Devices
• Demonstrate the engineering and statistical framework for virtual patients
• MDIC sponsor team includes industry and FDA
• FDA review team, just like for a real device
http://mdic.org/computer-modeling/virtual-patients/
2014 2015 2016
MDIC
working
group
formed
Mock
submission
team identified
Mock
submission
informational
meeting at FDA
2nd Mock
submission meeting
(engineering model)
at FDA
3rd Mock
submission
meeting (clinical
study) at FDA
FDA
commitment 2017
MDIC project
initiated to
develop methods
for historical data
14
Mock submission details
• Hypothetical new ICD lead
− Changes to insulation thickness and conductor
material, both affect fatigue life
− Expected fracture rate <1% at 18 months
− Single anatomical zone, single failure mode
• Clinical study design
− Objectives
• Fracture rate at 18 months < 3%, type I error < 10%
− Enrollment
• 200 initial patients, interim look every 30, maximum 400
patients. Up to 160 virtual patients (40%)
− Analysis
• No virtual patients, fixed amount, and with a discount
function
http://mdic.org/computer-modeling/virtual-patients/
15
Results
• Traditional study with no virtual
patients is underpowered
− 60% power / 3% type I error
• Fixed number of virtual patients
has unacceptable type I error
− 96% power / 25% type I error
• Studies using virtual patients with
a discount function have
acceptable power AND type I error
− Function #1: 80% power / 5% type I error
− Function #2: 85% power / 10% type I error
Power = % chance of success that you deserve
Type I error = % chance of success you don’t deserve
lower bound of virtual
patient model
performance
performance
requirement
Type I error
suffers without
discount
functionPower suffers
without virtual
patients
http://mdic.org/computer-modeling/virtual-patients/
16
What did we learn?
• Three year process, required a very high level of collaboration
• Stakeholders have to engage in a different way
− Statistician, engineer, clinician, regulator all have a role in the study design
− The bandwidth of the regulatory communication process is a challenge – engage early!
• Agreement on credibility of the prior is the most important topic
− Discount function allows for scaling according to prior credibility
− Introduces a different lens for evaluating historical data and engineering models
• All material available at http://mdic.org/computer-modeling/virtual-patients
2014 2015 2016
MDIC
working
group
formed
Mock
submission
team identified
Mock
submission
informational
meeting at FDA
2nd Mock
submission meeting
(engineering model)
at FDA
3rd Mock
submission
meeting (clinical
study) at FDA
FDA
commitment 2017
MDIC project
initiated to
develop methods
for historical data
17
Model Credibility
• Level of trust in the model is driven by credibility.
• Credibility comes from verification and validation.
• ASME V&V40 risk-informed credibility
assessment methodology
− Model influence: the contribution of the model to
the decision relative to other available evidence
− Decision consequence: the significance of an
incorrect decision (related to the device)
− Model Risk: combination of model influence and
decision consequence for a context of use
18
Summary
• Virtual patients can improve the clinical decision process while exposing
fewer patients to clinical trials
• Bayesian design with a discount function controls the influence of virtual
patients
• The statistical methods are ready – we just need the right applications!
• Model credibility is the most important thing