b 1.ohdsi erlangen 2018 - miracum.org€¦ · • population‐level estimation = causality ... why...
TRANSCRIPT
![Page 1: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/1.jpg)
Results from OHDSI and perspectives for international
data sharing projects
George Hripcsak, MD, MSBiomedical Informatics, Columbia University
Medical Informatics Services, NewYork‐Presbyterian
Biomedical Informaticsdiscovery and impact
![Page 2: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/2.jpg)
Observational Health Data Sciences and Informatics (OHDSI, as “Odyssey”)
Mission: To improve health by empowering a community to collaboratively generate the evidence that promotes better health decisions and better careA multi‐stakeholder, interdisciplinary, international collaborative with a coordinating center at Columbia UniversityAiming for 1,000,000,000 patient data network
http://ohdsi.org
![Page 3: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/3.jpg)
OHDSI’s global research community
• >200 collaborators from 25 different countries• Experts in informatics, statistics, epidemiology, clinical sciences• Active participation from academia, government, industry, providers• Over a billion records on >400 million unique patients in 80 databases
http://ohdsi.org/who‐we‐are/collaborators/
![Page 4: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/4.jpg)
OHDSI’s approach to open science
Open source software
Open science
Enable users to do
something
Generate evidence
• Open science is about sharing the journey to evidence generation • Open‐source software can be part of the journey, but it’s not a final destination• Open processes can enhance the journey through improved reproducibility of
research and expanded adoption of scientific best practices
Data + Analytics + Domain expertise
![Page 5: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/5.jpg)
Evidence OHDSI seeks to generate from observational data
• Clinical characterization = tallying– Natural history: Who has diabetes, and who takes metformin?– Quality improvement: What proportion of patients with
diabetes experience complications?• Population‐level estimation = causality
– Safety surveillance: Does metformin cause lactic acidosis?– Comparative effectiveness: Does metformin cause lactic
acidosis more than glyburide?• Patient‐level prediction = prediction
– Precision medicine: Given everything you know about me, if I take metformin, what is the chance I will get lactic acidosis?
– Disease interception: Given everything you know about me, what is the chance I will develop diabetes?
![Page 6: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/6.jpg)
How OHDSI Works
Source data Source data warehouse, with
identifiable patient‐level data
Standardized, deStandardized, de‐identified patient‐level database (OMOP CDM v5)
ETL
Summary statistics results
repository
OHDSI.org
Consistency
Temporality
Strength Plausibility
Experiment
Coherence
Biological gradient Specificity
Analogy
Comparative effectiveness
Predictive modeling
OHDSI Data Partners
OHDSI Coordinating Center
Standardized large‐scale analytics
Analysis results
Analytics development and testing
Research and education
Data network support
![Page 7: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/7.jpg)
Deep information modelOMOP CDM v5
Concept
Concept_relationship
Concept_ancestor
Vocabulary
Source_to_concept_ma
Relationship
Concept_synonym
Drug_strength
Cohort_definition
Standardized vocabularies
Attribute_definition
Domain
Concept_class
Cohort
Dose_era
Condition_era
Drug_era
Cohort_attribut
Standardized derived elem
ents
Stan
dardize
d clinical data
Drug_exposure
Condition_occurrence
Procedure_occurrence
Visit_occurrence
Measurement
Observation_period
Payer_plan_period
Provider
Care_siteLocation
Death
Cost
Device_exposure
Observation
Note
Standardized health system data
Fact_relationship
SpecimenCDM_source
Standardized meta‐data
Standardized health econom
ics
Person
![Page 8: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/8.jpg)
Extensive vocabularies (80)
![Page 9: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/9.jpg)
Vocabulary
• SNOMED‐CT– OHDSI’s reference terminology for conditions (diagnoses) …
– Mappings from ICD10, ICD10‐CM, ICD9‐CM, Read, MedDRA, …
• Maintained by US Nat’l Library of Medicine and OHDSI– Retain the original term, but most research on the SNOMED term
– Not need language‐specific version• Similar for RxNorm (drug), LOINC (lab), …
![Page 10: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/10.jpg)
Tools to convert your data
Patient‐level data in source system/ schema
Patient‐level data in
OMOP CDM
ETL design
ETL implement ETL test
WhiteRabbit: profile your source data
RabbitInAHat: map your source structure to
CDM tables and fields
ATHENA: standardized vocabularies for all CDM domains
ACHILLES: profile your CDM data; review data quality
assessment; explore
population‐level summaries
OHD
SI to
ols b
uilt to help
CDM: DDL, index,
constraints for Oracle, SQL Server,
PostgresQL; Vocabulary tables
with loading scripts
http://github.com/OHDSI
OHDSI Forums:Public discussions for OMOP CDM Implementers/developers
Usagi: map your
source codes to CDM
vocabulary
![Page 11: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/11.jpg)
Data Validation by ACHILLES Heel
![Page 12: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/12.jpg)
ATLAS to build, visualize, and analyze cohorts
![Page 13: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/13.jpg)
Characterize the cohorts of interest
![Page 14: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/14.jpg)
OHDSI in Action
![Page 15: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/15.jpg)
Treatment Pathways
Public
Industry
Regulator
Academics RCT, ObsLiterature
Lay press
Social media
Guidelines
Formulary
Labels
Advertising Clinician
Patient
Family
Consultant
Indication
Feasibility
Cost
Preference
Local stakeholdersGlobal stakeholders Conduits
Inputs
Evidence
![Page 16: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/16.jpg)
OHDSI in action:Chronic disease treatment pathways
• Conceived at AMIA • Protocol written, code written and tested at 2sites
• Analysis submitted to OHDSI network
• Results submitted for 7 databases
15Nov201430Nov2014
2Dec2014
5Dec2014
![Page 17: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/17.jpg)
OHDSI partners for this query (250M)Abbre‐viation
Name Description Population, millions
AUSOM Ajou University School of Medicine South Korea; inpatient hospital EHR
2
CCAE MarketScan Commercial Claims and EncountersUS private‐payer claims 119
CPRD UK Clinical Practice Research Datalink UK; EHR from general practice 11
CUMC Columbia University Medical Center US; inpatient EHR 4
GE GE Centricity US; outpatient EHR 33
INPC Regenstrief Institute, Indiana Network for Patient Care
US; integrated health exchange 15
JMDC Japan Medical Data Center Japan; private‐payer claims 3
MDCD MarketScan Medicaid Multi‐State US; public‐payer claims 17
MDCR MarketScan Medicare Supplemental and Coordination of Benefits
US; private and public‐payer claims
9
OPTUM Optum ClinFormatics US; private‐payer claims 40STRIDE Stanford Translational Research Integrated
Database EnvironmentUS; inpatient EHR 2
HKU Hong Kong University Hong Kong; EHR 1
![Page 18: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/18.jpg)
Treatment pathway event flow
![Page 19: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/19.jpg)
Proceedings of the National Academy of Sciences, 2016
![Page 20: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/20.jpg)
T2DM : All databases
Treatment pathways for diabetes
First drug
Second drug
Only drug
![Page 21: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/21.jpg)
Type 2 Diabetes Mellitus Hypertension Depression
OPTUM
GE
MDCDCUMC
INPC
MDCR
CPRD
JMDC
CCAE
Population‐level heterogeneity across systems, and patient‐level heterogeneity within systems
![Page 22: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/22.jpg)
HTN: All databases
Patient‐level heterogeneity
25% of HTN patients (10% of others) have a unique path despite 250M pop
![Page 23: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/23.jpg)
Monotherapy – diabetes
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
1989 1994 1999 2004 2009
AUSOM (SKorea*) CCAE (US#) CPRD (UK*) CUMC (US*)GE (US*) INPC (US*#) JMDC (Japan#) MDCD (US#)MDCR (US#) OPTUM (US#) STRIDE (US*)
General upward trend in monotherapy
![Page 24: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/24.jpg)
Monotherapy – HTN
AUSOM (SKorea*) CCAE (US#) CPRD (UK*) CUMC (US*)GE (US*) INPC (US*#) JMDC (Japan#) MDCD (US#)MDCR (US#) OPTUM (US#) STRIDE (US*)
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
1989 1994 1999 2004 2009
Academic medical centers differ from general practices
![Page 25: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/25.jpg)
Monotherapy – diabetes
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
1989 1994 1999 2004 2009
AUSOM (SKorea*) CCAE (US#) CPRD (UK*) CUMC (US*)GE (US*) INPC (US*#) JMDC (Japan#) MDCD (US#)MDCR (US#) OPTUM (US#) STRIDE (US*)
General practices, whether EHR or claims, have similar profiles
![Page 26: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/26.jpg)
Conclusions: Network research
• It is feasible to encode the world population in a single data model– Over 1,000,000,000 records by voluntary effort
• Generating evidence is feasible• Stakeholders willing to share results• Able to accommodate vast differences in privacy and research regulation
![Page 27: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/27.jpg)
What is the quality of the current evidence from observational analyses?
August2010: “Among patients in the UK General Practice Research Database, the use of oral bisphosphonates was not significantly associated with incident esophageal or gastric cancer”
Sept2010: “In this large nested case‐control study within a UK cohort [General Practice Research Database], we found a significantly increased risk of oesophagealcancer in people with previous prescriptions for oral bisphosphonates”
![Page 28: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/28.jpg)
Open science
• Admit that there is a problem• Study it scientifically
– Define that surface and differentiate true variation from confounding …
• Total description of every study• Research into new methods
![Page 29: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/29.jpg)
Take a scientific approach to science
Madigan D, Ryan PB, Schuemie MJ et al, American Journal of Epidemiology, 2013“Evaluating the Impact of Database Heterogeneity on Observational Study Results”
Madigan D, Ryan PB, Schuemie MJ, Therapeutic Advances in Drug Safety, 2013: “Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies”
Ryan PB, Stang PE, Overhage JM et al, Drug Safety, 2013: “A Comparison of the Empirical Performance of Methods for a Risk Identification System”
Schuemie MJ, Ryan PB, DuMouchel W, et al, Statistics in Medicine, 2013:“Interpreting observational studies: why empirical calibration is needed to correct p‐values”
1. Database heterogeneity:Holding analysis constant, different data may yield different estimates
2. Parameter sensitivity:Holding data constant, different analytic design choices may yield different estimates
3. Empirical performance:Most observational methods do not have nominal statistical operating characteristics
4. Empirical calibration can help restore interpretation of study findings
![Page 30: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/30.jpg)
Reproducible research
1. Address confounding that is measured• Propensity stratification• Systematic (not manual) variable selection
• Balance 58,285 variables (“Table 1”)
After stratification on the propensity score, all 58,285 covariates have standardized difference of mean < 0.1
![Page 31: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/31.jpg)
Reproducible research
2. Unmeasured (residual) confounding• Confidence interval calibration
• Adjust for all uncertainty, not just sampling• Many negative controls
• Unique to OHDSI (PNAS in press)
After calibration, 4% have p < 0.05 (was 16%)
![Page 32: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/32.jpg)
Reproducible research
3. Multiple databases, locations, practice types• Exploit international OHDSI network
![Page 33: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/33.jpg)
Reproducible research
4. Open: publish all• Hypotheses• Code• Parameters• Runs
![Page 34: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/34.jpg)
Generating evidence for US FDA
• Protocol completed, code tested, study announced
• 50 viewed protocol, 25 viewed the code, and 7 sites ran the code on 10 databases (5 claims / 5 EHR), 59,367 levetiracetampatients matched with 74,550 phenytoin patients
?
![Page 35: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/35.jpg)
Generating evidence for US FDA
“The study is focused, appears well designed, and provides new insight that should be of interest to clinicians and regulators... This is an important contribution to improved pharmacovigilance.”
Add word to title, move diagram from supplement to body
No evidence of increased angioedema risk with levetiracetamuse compared with phenytoin use
![Page 36: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/36.jpg)
How can we improve the literature
![Page 37: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/37.jpg)
Literature
Effect size (1 = no effect)
Stan
dard error
P = 0.05
![Page 38: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/38.jpg)
Literature
Not significant
Effect size
Stan
dard error
P = 0.05
HarmProtect
![Page 39: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/39.jpg)
Observational research results in literature
85% of exposure‐outcome pairs have p < 0.05
29,982 estimates11,758 papers
![Page 40: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/40.jpg)
Observational research results in literature
29,982 estimates11,758 papers
Publication bias Don’t know the denominator
of negative studies.
![Page 41: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/41.jpg)
Observational research results in literature
29,982 estimates11,758 papers
P‐value hacking
![Page 42: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/42.jpg)
Observational research results in literature
• Individuals may produce good research studies
• In aggregate, the medical research system is a data dredging machine
![Page 43: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/43.jpg)
Look at many outcomes at once
Acute liver injury Hypotension
Acute myocardial infarction Hypothyroidism
Alopecia Insomnia
Constipation Nausea
Decreased libido Open‐angle glaucoma
Delirium Seizure
Diarrhea Stroke
Fracture Suicide and suicidal ideation
Gastrointestinal hemorrhage Tinnitus
HyperprolactinemiaVentricular arrhythmia and sudden cardiac death
Hyponatremia Vertigo
Duloxetine vs. Sertraline for these 22 outcomes:
![Page 44: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/44.jpg)
Many treatments at onceType Class TreatmentDrug Atypical BupropionDrug Atypical MirtazapineProcedure ECT Electroconvulsive therapyProcedure Psychotherapy PsychotherapyDrug SARI TrazodoneDrug SNRI DesvenlafaxineDrug SNRI duloxetineDrug SNRI venlafaxineDrug SSRI CitalopramDrug SSRI EscitalopramDrug SSRI FluoxetineDrug SSRI ParoxetineDrug SSRI SertralineDrug SSRI vilazodoneDrug TCA AmitriptylineDrug TCA DoxepinDrug TCA Nortriptyline
![Page 45: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/45.jpg)
Large‐scale estimation for depression
• 17 treatments• 17 * 16 = 272 comparisons• 22 outcomes• 272 * 22 = 5,984 effect size estimates• 4 databases (Truven CCAE, Truven MDCD, Truven MDCR, Optum)
• 4 * 5,984 = 23,936 estimates
![Page 46: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/46.jpg)
Estimates are in line with expectations
11% of exposure‐outcome pairs have calibrated p < 0.05
![Page 47: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/47.jpg)
Example
Mirtazapine vs. CitalopramConstipationDatabase: Truven MDCD
Calibrated HR = 0.90 (0.70 – 1.12)
![Page 48: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/48.jpg)
Propensity models for all comparisons(Truven CCAE, one outcome)
48
![Page 49: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/49.jpg)
Large‐scale estimation for depression
• How do we use it? Troll for effects?
• Professor what should I study this year?
– Simple, go to Pubmed and find the smallest p-values in the literature; surely those must be the most significant things to study
• Which is safer?
• Seizure in 0.0000000001 to 0.0000000002 (p=0.00001)
• Seizure in 0 to 0.2 (p=.45)
• Large-scale studies become the literature
• Come with hypothesis and ask a question
![Page 50: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/50.jpg)
Large‐scale estimation for depression
• Not “data‐dredging”! – Data‐dredging is not about what you do but about what you throw out
• This can’t be done for literature
• One‐off studies– Wouldn’t it be best to optimize each study?
• Never get 10 or 100 parameters right– Still good to see the distribution
• At the very least, publish every last parameter so it can be reproduced
50
![Page 51: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/51.jpg)
How often
![Page 52: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/52.jpg)
How often do side effects occur?
• New incidence of any condition for any drug on the world market– Show range of answers for disparate databases
• Absolute risk (vs. attributable risk)– Not know if it is causal or not: MI with statin
• More complicated than it looks– Standard framework for reporting incidence
Person timeline
Cohort entry
Time‐at‐risk
Outcomeoccurrence
Cohort exit
Observation period end
Observation period start
![Page 53: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/53.jpg)
howoften.org
![Page 54: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/54.jpg)
International data sharing
![Page 55: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/55.jpg)
International data sharing
• Research benefits from diversity– Climate and other environment– Culture– Genetics– Health care policy– Socioeconomic disparities– Care setting– Data capture process
![Page 56: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/56.jpg)
International data sharing
• Language– Collected and developed mappings from 80 vocabularies to small number of standards
– >$250K/year + volunteer work– Free on Athena on OHDSI.org
![Page 57: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/57.jpg)
International data sharing
• Common data model– International community to assist in ETL
Patient‐level data in source system/schema
Patient‐level data in
Common Data Model
ETL design
ETL implement ETL test
One‐time Repeated
![Page 58: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/58.jpg)
International data sharing
• Privacy– Keep data local– Distribute queries
![Page 59: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/59.jpg)
International data sharing
• Governance– Workgroups and forums
• Many require two per period to cover time zones
– Coordinating center to pay the bills
![Page 60: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/60.jpg)
Tradeoff on data standardization
• More constrained common data model• Deep information model
– Schema, vocabularies, conventions
• More time to convert and verify data• Greatly aids research• For worldwide research, need agreement
![Page 61: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/61.jpg)
Opportunities for standardization• Data structure: tables, fields, data types• Data conventions: set of rules that govern how data are
represented• Data vocabularies: terminologies to codify clinical domains• Cohort definition: algorithms for identifying the set of
patients who meet a collection of criteria for a given interval of time
• Covariate construction: logic to define variables available for use in statistical analysis
• Analysis: collection of decisions and procedures required to produce aggregate summary statistics from patient‐level data
• Results reporting: series of aggregate summary statistics presented in tabular and graphical form
![Page 62: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/62.jpg)
OHDSI Uptake
• Good uptake in the US– OHDSI network sites– All of Us Research Program– eMERGE Consortium– FDA Sentinel BEST contract– PCORnet sources
• Good uptake in Asia– South Korea– Taiwan– Hong Kong– Japan database– China– Australia
• Europe– Netherlands– Sweden– Italy– UK database– EMA investigation– IMI– Germany …
![Page 63: B 1.OHDSI Erlangen 2018 - miracum.org€¦ · • Population‐level estimation = causality ... why empirical calibration is needed to correct p‐values ... Troll for effects?](https://reader031.vdocument.in/reader031/viewer/2022022519/5b16d4c07f8b9a5e6d8e015f/html5/thumbnails/63.jpg)
Summary on OHDSI
• Successful international clinical data collaboration
• Open source tools including mappings (free)• Producing evidence today• Can influence the future of OHDSI
OHDSI.org