march 15, 2011: i. simclinical research 2.0 epi – 206 medical informatics clinical research 2.0...
TRANSCRIPT
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Clinical Research 2.0
Ida Sim, MD, PhD
March 15, 2011
Division of General Internal Medicine, and Graduate Group in Biological and Medical Informatics
UCSF
Copyright Ida Sim, 2011. All federal and state rights reserved for all original material presented in this course through any medium, including lecture or print.
Today
Virtual Patient
Transactions
Raw data
Medical knowledge
Clinical research
transactions
Raw research
data
Dec
isio
n su
ppor
t
Med
ical
logi
c
PATIENT CARE / WELLNES RESEARCH
Workflow modeling and support, usability, cognitive support, computer-supported cooperative work (CSCW), etc.
CRMSsEHRs
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
IRB Funding Agency
Study DB
Data analysis
Results reporting
Contract R
esearch O
rganization (C
RO
)
Protocol
Trial DesignSponsorsAcademic PIs
?Site 1 Site 2 Site 3
Site Management Organization (SMO)
Today
Clinic 2008
FrontDesk
Radiology
MedicalInformationBureau
Walgreens
Pharm BenefitManager
Benefits Check(RxHub)
HealthNet
B&T
UCare
Specialist
ReferralAuthorization
Internet Intranet Phone/Paper/Fax
Lab
UniLab
(HL-7)
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Health Care Quality
• Doing the right thing– based on scientific evidence
• right – without error
• to the right people– e.g., blood pressure meds by ethnicity
• at the right time– beta-blockers at hospital discharge for heart attacks
Clinici
an View
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
General Drivers of Change
• A “grand convergence” of– maturation of the Internet as connective technology
– social computing tipping past critical mass
– ubiquity of microchips in computers, appliances, and sensors
– explosion of data from everywhere and everything (Big Data)
• For all fields, frontiers of research driven by– ability to do large-scale multi-disciplinary data analysis,
visualization, etc.
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Biomedical Drivers of Change• Personalized medicine, geno-pheno correlations
– need genomic and phenotype data in computable form for large-scale small signal correlations
• predictors more likely to be rare vs common variants
• Genomic data will be a commodity– SNPs, whole genome analysis
• Large-scale phenotype is the bottleneck– incl. patient-centered phenotypes (symptoms, side effects)
• Requires tighter connection between research and care – huge volume, complex data that needs to be made sense of
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Health Care Quality--Pt. View
• Stay healthy for as long as possible• Take as few medicines as possible
– they all work, with fewest side effects– at lowest cost– for as short a time as possible
• I know what makes me worse and what makes me better, and can do the right thing
• I have all my questions answered• I know what other people are thinking, and I feel
supported in my overall health status
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Clinical Research/Care 2.0
• Methodological considerations of ICT• Open data/open science on epic scale fully integrated into
personalized person-centered care• “not your grandfather’s clinical research”
• What do we need to get there?– “building the research ‘cyberinfrastructure’ is the single
most important challenge confronting the nation’s science laboratories” (NSF)1
– communities of self-improving practice (evaluation and feedback)
– architecture of sustainability
1http://www.nsf.gov/news/special_reports/cyber/index.jsp
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Outline
• Methodological Considerations of ICT • Towards Clinical Care/Research 2.0• Open Science• Class Summary
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Methodological Considerations
• ICT system design – user interaction design, rapid prototyping
• Study design– beyond RCTs
• Participant recruitment and sample size– selection bias, sampling error
• Outcomes assessment– measurement error and bias
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Intervention Design• User interaction design
– “form factor”: kiosk, desktop, laptop, notebook, tablet, phone… – platform (e.g., Mac), browser (e.g., Safari), operating system
(e.g., Android or iPhone) incompatibilities– use Flash, Java, etc requiring plug-ins or version compatibility – readiblity (font too small), need to scroll, confusing navigation,
bugs• What technology does respondent group use?
– check some browser and platform usage statistics sources • e.g., http://www.statowl.com/web_browser_market_share.php
– need to test and double-test for various configurations
• UCSF Usability Lab pending…consider paying for design services
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Rapid Prototyping & Feedback• Traditional approach
– design a website or system; pilot it, improve it; run RCT; analyze data; publish (over 2-3 years)
• Rapid prototyping, participatory design, agile programming– Google launched 450 search engine improvements in 2007– each one tested rigorously, real-time feedback through
clicks, user logs, etc.• http://googleblog.blogspot.com/2008/08/search-experiments-lar
ge-and-small.html
– combine quantitative with qualitative user studies
• Why wait till end of RCT to find out that “the system” didn’t work or wasn’t used, and have little idea why?
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Embedded RCTs
• Identify part of the system for testing (e.g., videos vs. pictures of inhaler use)
• Randomly assign each person landing on the screen to videos or pictures
• Test for comprehension with short survey• Compare % correct answers, set “stopping rule,” run
study until answer clear or test is over• Can run multiple embedded user interaction RCTs to
optimize design features in parallel
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Methodological Considerations
• ICT system design – user interaction design, rapid prototyping
• Study design– beyond RCTs
• Participant recruitment and sample size– selection bias, sampling error
• Outcomes assessment– measurement error and bias
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Current RCT Research Approach• RCTs are expensive, difficult to conduct, 30-40% of studies never
accrue enough patients– estimated 2 million pts needed/yr for US-based trials
– will be worse with personalized medicine• Studies take years to answer limited questions in limited
populations• Study designs and results are heterogenous, limiting ability to pool
findings or make summary interpretations• Research questions don’t address combination treatments (e.g.,
ACEI and amlodipine)• Research questions don’t track with front-line clinical needs
– no good data on mid- to long-term efficacy or effectiveness of antidepressants
• Overall lack of relevance, generalizability, and sustainabilityMoss, et al. NEJM 2011; 364(9):789-761Crowley, et al. JAMA 2004; 291(9):1120-6etc.
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
New Vogue
• Data (evidence) mining of aggregated EHR data– new “comparative effectiveness research”
– effectiveness, not efficacy
– subject to confounding by indication
– subject to errors and biases of data coding (e.g., IDR Cohort Selection Tool example)
– under-estimating GIGO (garbage in garbage out) problem
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Mining vs. Farming
• Mining– traditional but troubled RCTs1, mining for gold
– in a clinical data warehouse “garbage pit”• may be biased, error-laden population-level effectiveness
estimates
• Sustainable evidence farming– “systematic effort by practitioners to collect and analyze data
on actual practice experience that is used to develop clinical evidence for practice” (R. Kravitz, UC Davis)
– i.e., to cultivate data for use in generating relevant evidence for sharpening therapeutic precision for individual patients
1DeMets, Califf. JAMA 2011; 305(7):713-714.
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Promising Evidence Farms
• Practice-Based Research Networks, Accountable care organizations, etc.– PROS, AAFP, AHRQ, HMO Research Network
– Kaiser Permanente
– UCSF?
• Practitioners (and patients?) own the data and enjoy the harvest – Practice informs research and vice versa
• Practitioners help to select research questions
• Results can be used to craft guidelines, improve practice organization, develop locally relevant clinical evidence
– Joint symptom monitoring and/or N-of-1 trials• using EHR, PHR, mHealth apps
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Hypothetical Case
• MR is a 38 year old man with depression• Condition characterized by depressed mood,
anhedonia, poor sleep, difficulty concentrating; all of moderate severity
• Treatment options are Zoloft and Effexor• Which treatment is best for MR?• Traditional EBM approach applies, in order
– best research evidence
– clinical experience
– patient valuesAcknowledgement for evidence farming slides: R. Kravitz, UC Davis
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
But…
• Latest research indicates an 8-week remission rate that is greater for Effexor than for Zoloft (68% vs. 45%)1
• Among the past 20 patients Dr. Jones has treated, 6 got Effexor and 14 got Zoloft– Among Effexor patients, 3 remitted (50%)– Among Zoloft patients, 12 remitted (86%)
• Should Dr. Jones get with the published evidence?
1Mehtonen OP, et al. Randomized, double-blind comparison of venlafaxine and sertraline in outpatients with major depressive disorder. Venlafaxine 631 Study Group. J Clin Psychiatry. 2000 Feb;61(2):95-100.
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
The Clinical Dilemma
• Guidelines (based on global, exogenous evidence) say one thing
• (Systematic) experience (based on local, endogenous experience) says something else
• Who to listen to?• Both!
– traditional evidence sets the prior based on population-level efficacy data (top down)
– evidence farming methods update the prior using systematic local/personalized observations (bottom up)
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Why Bother with Local Evidence?
• Context matters: RCT evidence not fully generalizable– Patient values and preferences vary across
practices • psychotherapy vs. SSRIs
– Patient risk, responsiveness, and vulnerability vary across practices
• 2-drug therapy for hypertension in blacks
– Practitioners and organizations differ• carotid endarterectomy
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Empirical Bayes Approach
• Empirical Bayes method balances between the two parcels of evidence (borrow from strength)
• Ti,EB = (1 – Bi) * TGLOBAL + Bi * Ti,LOCAL
Bi = sBETWEEN2 / (sBETWEEN
2 + sWITHIN,i2)
• The greater the between-group variance relative to the total variance, the more weight is placed on local evidence
• Local evidence accumulates over time
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Rigorous Symptom Monitoring
• Depression symptom self-monitoring– triggered by GPS activity (staying in bed all day?) or
customized for patient (e.g., after elder care) – timing based on focus group input (e.g., end of day
preferred)– very simple, low overhead data input
• Feedback on Bayesian trends, discontinuities, etc. correlated with current medication (e.g., Zoloft) and dosage
• Or more formal N-of-1 studies1
– randomized and blinded to Effexor and Zoloft in within-subject crossover fashion
1Kravitz, et al. Contemp Clin Trials 2009; 30:436-445
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Barriers to Implementation
• Electronic infrastructure• Culture• Resources & incentives
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Methodological Considerations
• ICT system design – user interaction design, rapid prototyping
• Study design– beyond RCTs
• Participant recruitment and sample size– selection bias, sampling error
• Outcomes assessment– measurement error and bias
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Subject Recruitment• Recruitment is biggest bottleneck of clinical research
– 30-40% of clinical trial costs – >80% of trials have recruitment delays– 1/20 recruited patients actually enroll
• Web-based recruitment can be international, cheap, fast– e.g., www.stopsmoking.ucsf.edu Dec 05 - Feb 07
• 350,000 hits, 60,000 entered data, 20,000 enrolled• 2/3 Spanish-speaking, 1/3 English• 131,517 visits from 121 countries Jan 12, 05 to April 5,
06
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Visits0=>1=>100=>1,000=>10,000
Distribution of Visits to www.stopsmoking.ucsf.edu Jan 12, 2005 to April 5, 2006
(131,517 visits from 121 countries)
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Reducing Sampling Error
• Social sciences and marketing are most advanced in population-level recruitment methodology– e.g., Joint Statistical Meetings of the American Statistical
Association
– http://www.knowledgenetworks.com/
• Two major methods– recruit a representative sample
– use a pre-assembled representative cohort
Disclosure: I have no relationship with KnowledgeNetworks
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Recruit Representative Sample
• Random digit dialing (RDD) with web supplementation equally representative as (land-line) telephone RDD– RDD sampling (landline and cellular)
– if respondent agrees, provide them with free Internet access (via MSN TV, formerly WebTV) or other necessary hardware for duration of participation
– e.g.,http://knowledgenetworks.com/
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Representative Cohorts
• Maintained by e.g., large survey and marketing firms– http://www.knowledgenetworks.com/knpanel/index.html
• KnowledgePanel is representative of US
• can target specific respondents, “survey response rates of 65-75%, abandonment rate <2%”
– www.surveysampling.com• ~6 m people from 72 countries, incl. health-specific panels
– http://www.experimentcentral.org/ • NSF-funded representative panel for social science survey
research
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Other Recruitment Methods
• With higher risk of sampling bias– search engines, with search engine
optimization (SEO) techniques (e.g., Google adwords)
– links from related pages– listservs, Facebook, etc.
• Can combine radio, TV, or print w/ – website (URL, Uniform Resource
Locator)
– text messaging Common Short Codes (http://www.usshortcodes.com/csc_whatDo.html)
– QR code (or matrix bar code) launches URL, vCard, displays text, etc.
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Search Engine Ranking• Search engines have their own (secret) algorithm for ranking
pages– Google uses >100 factors, esp. how many pages link into a
page
• Google AdWords – put in your keywords, see cost-per-click
• https://adwords.google.com/select/KeywordToolExternal?defaultView=3
– pay only if someone clicks, more if the keyword historically brings more traffic
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Note on Sample Size
• Estimating sample size– e.g., Google provides traffic history for various keywords
(adwords.google.com)
• Since incremental cost of web surveys/ICT interventions are often negligible, less pressure to minimize sample size– not unusal to get large samples (>10,000)
• But high sample size = high accuracy!– may be precise (p < 0.05) but inaccurate if sample is non-
representative
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Methodological Considerations
• ICT system design – user interaction design, rapid prototyping
• Study design– beyond RCTs
• Participant recruitment and sample size– selection bias, sampling error
• Outcomes assessment– measurement error and bias
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Measurement Error/Bias
• What you designed may not be what respondent sees
• Client’s browser/phone displays the intervention based on – platform, browser, monitor, screen/window size
– different users see different survey, e.g., • small screen/window size makes “Next” button not visible
• text doesn’t fit on small window, or requires scrolling for some respondents and not others
• colors, graphics (e.g., visual analog scales) may appear differently
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Loss to Follow-up• Influenced by
– respondent familiarity with functions (e.g., using CSCs)– technical design of survey– disability accesibilty (Section 503 compliance)– bandwidth– convenience (can interrupt survey?)
• Can use mixed-mode surveys/interventions to increase completion rates/follow up, e.g., – combined web/voice, web/mail surveys
– combined desktop/laptop and tablet/mobile (I.e., use different form factors for initiating vs. sustaining intervention effect)
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Outline
• Methodological Considerations of ICT • Towards Clinical Care/Research 2.0• Open Science• Class Summary
“In bounds” to “beyond bounds”
• Clinician-driven• Intermittent• Retrospective, delayed• Verbal report to clinician• Research and care
separated• Data is siloed• One patient at a time• Transactional support is
central
• Patient engaged• 24/7/everywhere• Real-time• Multi-modal• Research and care
integrated• Data is shared,
aggregated• “All” patients at a time• Evidence is central
Internet Hourglass Model
• Standardize and make open the “narrow waist”
• Reduces duplication, spurs community innovation, supports commercial and non-profit uses
Health 2.0 Hourglass• Key challenge is what
should be in the narrow waist– interoperation protocols
– privacy architecture
• Phase the introduction– start in mobile health
– tackle heavily siloed enterprise health later
http://openmhealth.org/• Orange waist
– standard connections to PHRs and EHRs
– ultrasecure personal data vault (“questions go to the data”)
• Green “middleware”– shared library of standardized
of outcomes, data variables, vocabularies
– widgets for symptom assessment, graphing, risk communication, surveying, randomization, etc.
– N-of-1 infrastructure
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
What Next at UCSF
• IDR/MyResearch– undergoing campus-wide review for future plans
• mHealth– working group, listserv, Facebook: email Ida to join
– mHealth methods class
– consultation services: design, etc.
– technical and legal services: either common open platform (e.g., openmhealth.org) or preferred contractors
• Planning for major initiative in informatics…TBD– Care Science Institute -- Cyberinfrastructure for Integrated
Care and Research?
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Outline
• Methodological Considerations of ICT • Towards Clinical Care/Research 2.0• Open Science• Class Summary
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Science in Era of “Big Data”
• “Data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time.”– e.g., the whole web, all EHR data, all human studies data
• Science is increasingly open– open source: NIH-funded software expected to be open source
– open data: sequencing data, GWAS, Alzheimers, all of NSF, large NHLBI studies, Science Commons, Community Health Data Initiative, etc.
– open publishing: PLoS, Biomed Central, PubMed Central
– open community curation of knowledge: e.g., CHDwiki, Wikigenes with authorship tracking (the long tail…)
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Data Mining with “Big Data”• Text mining, data mining, model building across ALL
data– within and outside biomedicine
• using shopping data to track diet, or population salt intake
– supervised (e.g, neural net) and unsupervised (e.g., clustering) learning
• not your grandfather’s logistic regression…
• Puts premium on– natural language processing (e.g., Watson)– semantic interoperation– maintaining context
• Alb 1.7 in 20 y/o MVA burn patient, vs. homebound 88 y/o
– developing a culture of sharing, policies for attribution of credit
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Towards Semantic Web
• Web 1.0– web of text
• Web 2.0– web of people
• Web 3.0 aka Semantic Web– web of data: data must be “tagged” with semantics, using
vocabularies, ontologies (e.g., Science Commons, linked open data repositories)
• Web 4.0– web of knowledge: nowhere near this (e.g., computable
statement that diabetes increases MI risk)
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Clinical Care/Research 3.0
Administrative Clinical Care Research
Physical Networking
Standard Communications Protocols (e.g., HL-7)
PracticeManagement
Systems
EHRExecutionAnalysis
Medical BusinessData Model
Clinical CareData Model
Clinical StudyData Models
Open de-identified repositories
OWL Ontologies of Everything
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Open Discussion
• How to balance standardization and comparability (e.g., of EHR notes, of research variables) with flexibility and innovation?
• Medicine and biomedical research is conservative– will all this web 2.0/3.0 stuff pass right by us?
• How will this change what you do/how you think, if at all?
• What would you like to see from academia/UCSF to help you stay as competitive in research as possible?
• ???
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Outline
• Methodological Considerations of ICT • Towards Clinical Care/Research 2.0• Open Science• Class Summary
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
State of Health IT Use• EHR adoption still low
– barriers include finances, lack of organizational change expertise, fragmentation of health care system, misaligned incentives
• Recovery Act will spur EHR adoption, for good or ill• EHR and data warehouses can but don’t always help
research • Limited success of decision support systems• Fundamental tradeoff of coding effort vs. “smartness”
of system limits both EHR and CDSS return on investment
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Computers Must Interoperate• In a networked world, data and actions must be
shared across people and computers– syntatic interoperation: a common grammar for machines
talking to each other in biomedicine (e.g., HL7)– semantic interoperation: predictable and meaningful
exchange of common meaning• requires standard vocabularies and standard data models
• SNOMED most comprehensive but use is unproven
• Other challenging things that need standardization in biomedicine– “common data elements” in research– a standard EHR data model so all EHRs “look” alike– standard protocol models for human studies, etc.
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Summary• Informatics helps make sense of data and knowledge
– is necessary for better care and research
• Today’s technologies promise transactional support – major barriers are economic, policy, and workflow related
• Need brand new technologies for other 3/4 of Big Picture
• Disruptive change to clinical research seems quite possible – as mobile technologies break down time and space barriers– as social computing takes off– as open data becomes more common
March 15, 2011: I. Sim Clinical Research 2.0Epi – 206 Medical Informatics
Take-Home Message• IT focuses on storing, accessing, and exchanging
data • Informatics is use of computers to make sense of
data • The more “computable” the information, the more the
computer can do for us• ...not just individually, but together as a community of
care and science
• Course evaluation– http://psg-mac43.ucsf.edu/ticr/CourseEvaluations/ticreval.asp?id=433