1

Evaluating Health Information Technology: Evaluating Health Information Technology: Putting Theory Into PracticePutting Theory Into Practice

Eric Poon, MD MPHEric Poon, MD MPHClinical Informatics Research and Development,

Partners Information Systems

David F. Lobach, MD, PhD, MSDavid F. Lobach, MD, PhD, MS

Division of Clinical Informatics Division of Clinical Informatics

Department of Community and Family MedicineDepartment of Community and Family Medicine

Duke University Medical Center, Durham, North CarolinaDuke University Medical Center, Durham, North Carolina

AHRQ’s National Resource Center for Health Information TechnologyAHRQ’s National Resource Center for Health Information Technology

Annual Meeting June 2005Annual Meeting June 2005

2

OutlineOutline

Overview of Evaluating HITOverview of Evaluating HIT Why evaluate?Why evaluate? General Approach to EvaluationGeneral Approach to Evaluation Choosing Evaluation MeasuresChoosing Evaluation Measures Study Design TypesStudy Design Types Analytical issues in HIT evaluationsAnalytical issues in HIT evaluations

Evaluation in the ‘real world’Evaluation in the ‘real world’ Duke University Medical Center Duke University Medical Center

3

Why Measure Impact of HIT?Why Measure Impact of HIT?

Impact of HIT often hard to predict Impact of HIT often hard to predict Many “slam dunks” go awryMany “slam dunks” go awry You can’t manage/improve what isn’t measuredYou can’t manage/improve what isn’t measured

Understand how to clear barriers to effective Understand how to clear barriers to effective implementationimplementation Understand what works and what doesn’tUnderstand what works and what doesn’t Invent the wheel only onceInvent the wheel only once

Justify enormous investmentsJustify enormous investments Return on investmentReturn on investment Allow other institutions to make tradeoffs intelligentlyAllow other institutions to make tradeoffs intelligently

Use results to win over late adoptersUse results to win over late adopters

4

General Approach to Evaluating HITGeneral Approach to Evaluating HIT

Understand your interventionUnderstand your intervention

Formulate questions to answerFormulate questions to answer

Select and define measuresSelect and define measures

Pick the study designPick the study design

Data analysisData analysis

5

Getting Started: Get to know your Getting Started: Get to know your interventionintervention

What problem(s) is it trying to solve?What problem(s) is it trying to solve? Think about intermediate processesThink about intermediate processes

Identify potential barriers to successful Identify potential barriers to successful implementation:implementation: Managerial barriersManagerial barriers End-user behavioral barriersEnd-user behavioral barriers

Understand how your peers around the Understand how your peers around the country are addressing (or not) the same country are addressing (or not) the same issues.issues.

6

Formulating QuestionsFormulating Questions

Likely questions:Likely questions: Does the HIT work?Does the HIT work? What would have made it work better?What would have made it work better? What would the next set of designers/implementors What would the next set of designers/implementors

like to know?like to know? Has this question been fully answered before?Has this question been fully answered before?

Don’t reinvent the wheel! (not a big concern)Don’t reinvent the wheel! (not a big concern) What impact would the answer have?What impact would the answer have?

PeersPeers Policy makersPolicy makers

7

Array of MeasuresArray of Measures

Quality and SafetyQuality and Safety Clinical OutcomesClinical Outcomes Clinical ProcessesClinical Processes

KnowledgeKnowledge PatientPatient ProviderProvider

Satisfaction & Satisfaction & AttitudesAttitudes PatientPatient ProviderProvider

Resource utilizationResource utilization Costs and chargesCosts and charges LOSLOS Employee Employee

time/workflowtime/workflow

Lessons learnedLessons learned

8

Choosing Study MeasuresChoosing Study Measures Clinical vs Process MeasuresClinical vs Process Measures

Clinical outcomes (e.g. mortality) desirableClinical outcomes (e.g. mortality) desirable Justifiable to measure process outcomes (e.g. door to abx time) Justifiable to measure process outcomes (e.g. door to abx time)

if relationship between outcome and process already if relationship between outcome and process already demonstrateddemonstrated

Will outcomes be impacted by the intervention?Will outcomes be impacted by the intervention? Will impact on outcomes be detectable during the study Will impact on outcomes be detectable during the study

period?period? ? Rare events, e.g. adverse outcomes? Rare events, e.g. adverse outcomes ? Colon cancer screening? Colon cancer screening

What resources do you have?What resources do you have? Don’t bit off more than what you can chew.Don’t bit off more than what you can chew.

9

Selecting Study TypesSelecting Study Types

Commonly used study types:Commonly used study types: Optimal design: Randomized Controlled TrialsOptimal design: Randomized Controlled Trials

Factorial DesignFactorial Design Before-and-after time series TrialsBefore-and-after time series Trials

Main study design issues:Main study design issues: Secular Trend: Can a simultaneous control group Secular Trend: Can a simultaneous control group

be established?be established? Confounding: Can you randomly assign Confounding: Can you randomly assign

individuals to study groups?individuals to study groups? Study design often influenced by Study design often influenced by

implementation planimplementation plan Need to respect operational needs, but often Need to respect operational needs, but often

there is room for creative designsthere is room for creative designs

10

Randomization Nuts and BoltsRandomization Nuts and Bolts

Justifiable to have a control arm as long as Justifiable to have a control arm as long as benefit not already demonstrated (usual care)benefit not already demonstrated (usual care)

Want to choose a truly random variable Want to choose a truly random variable Not day of the weekNot day of the week

Consideration: Stratified randomizationConsideration: Stratified randomization Ensures that intervention and control group are Ensures that intervention and control group are

similar on important characteristics (e.g. baseline similar on important characteristics (e.g. baseline computer literacy)computer literacy)

Strongest possible interventionStrongest possible intervention

11

Randomization Unit:Randomization Unit:How to Decide?How to Decide?

Small units (patients) vs. Large units (practices Small units (patients) vs. Large units (practices wards)wards) Contamination across randomization unitsContamination across randomization units If risk of contamination is significant, consider If risk of contamination is significant, consider

larger unitslarger units Effect contamination-can underestimate impactEffect contamination-can underestimate impact

However, if you see a difference, impact is presentHowever, if you see a difference, impact is present

Randomization by patient generally undesirableRandomization by patient generally undesirable ContaminationContamination Ethical concernEthical concern

12

Randomization Schemes:Randomization Schemes:Simple RCTSimple RCT

Burn-in periodBurn-in period Give target population time to get used to new Give target population time to get used to new

intervention intervention Data not used in final analysisData not used in final analysis

XX Clinics

Baseline Period

Baseline Data Collection Data Collection for RCT

No Intervention

Intervention Period

3 month burn-in period

Intervention Deployed

Intervention

arm

Control

arm

Control arm gets intervention

Post- Intervention

Period

13

Randomization schemes: Randomization schemes: Factorial DesignFactorial Design

May be used to May be used to concurrently evaluate concurrently evaluate more than one more than one intervention:intervention: Assess interventions Assess interventions

independently and in independently and in combinationcombination

Loss of statistical powerLoss of statistical power

Usually not practical for Usually not practical for more than 2 interventionsmore than 2 interventions

Control (no interventions)

A

B

A+B

14

Randomization Schemes:Randomization Schemes:Staggered DeploymentStaggered Deployment

Advantages of staggeringAdvantages of staggering Easier for user education and trainingEasier for user education and training Can fix IT problems up frontCan fix IT problems up front

Need to account for secular trend and baseline differencesNeed to account for secular trend and baseline differences Time variable in regression analysisTime variable in regression analysis Control for practice characteristicsControl for practice characteristics

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2Practice 1

Practice 2

Practice 3

Practice 4

Practice 5

Practice 6

Practice 7

Practice 8

2006 2007 2008

Intervention Group

Control Group

Intervention Group

Control Group

Intervention Group

Control Group

Intervention Group

Control Group

15

Inherent Limitations of RCTs in Inherent Limitations of RCTs in InformaticsInformatics

Blinding is seldom possibleBlinding is seldom possible

Effect on documentation vs. clinical actionEffect on documentation vs. clinical action

People always question generalizabilityPeople always question generalizability Success is highly implementation independentSuccess is highly implementation independent Efficacy-effectiveness gap: ‘Invented here’ Efficacy-effectiveness gap: ‘Invented here’

effecteffect

16

Mitigating the Limitations of Mitigating the Limitations of Before-and-After Study DesignsBefore-and-After Study Designs

Before-and-after trial common in informaticsBefore-and-after trial common in informatics Concurrent randomization is hardConcurrent randomization is hard

Don’t lose the opportunity to collect baseline Don’t lose the opportunity to collect baseline data!data!

Leave the time gap between before and after Leave the time gap between before and after trends relatively shorttrends relatively short

Look for secular trend in statistical analysis and Look for secular trend in statistical analysis and adjust for it if presentadjust for it if present

17

Common Pitfalls with Data CollectionCommon Pitfalls with Data Collection

Measures you define and collect on your ownMeasures you define and collect on your own Pilot data collection and refine definition Pilot data collection and refine definition earlyearly Ask yourself Ask yourself earlyearly whether data your collect measure whether data your collect measure

what you intended to measure.what you intended to measure. Measures others defined but you collect on your Measures others defined but you collect on your

ownown Do you need to adapt other people’s instruments?Do you need to adapt other people’s instruments?

Measures others define and collect for youMeasures others define and collect for you Understand nuisances and limitations, particular with Understand nuisances and limitations, particular with

administrative data.administrative data.

18

Electronic Data Abstraction: Electronic Data Abstraction: There’s no free lunch!There’s no free lunch!

Convenient and time-saving, but…Convenient and time-saving, but… Some chart review (selected) to get Some chart review (selected) to get

information not available electronicallyinformation not available electronically Get ready for surprisesGet ready for surprises

Documentation effect of EMRsDocumentation effect of EMRs

19

Data Collection Issue: Data Collection Issue: Baseline DifferencesBaseline Differences

Randomization schemes Randomization schemes often lead to imbalance often lead to imbalance between intervention and between intervention and control arms:control arms: Need to collect baseline Need to collect baseline

data and adjust for data and adjust for baseline differences baseline differences

Interaction term ( Time * Interaction term ( Time * Allocation Arm) gives Allocation Arm) gives effect for intervention in effect for intervention in regression analysisregression analysis

20

Data Collection Issue: Data Collection Issue: Completeness of FollowupCompleteness of Followup

The higher the better:The higher the better: Over 90%Over 90% 80-90%80-90% Less than 80%Less than 80%

Intention to treat analysisIntention to treat analysis In an RCT, should analyze outcomes In an RCT, should analyze outcomes

according to the original randomization according to the original randomization assignmentassignment

21

A Common Analytical Issue A Common Analytical Issue The Clustering EffectThe Clustering Effect

Occurs when your observations are not Occurs when your observations are not independent:independent: Example: Each physician treats multiple patients:Example: Each physician treats multiple patients:

May need to increase sample size to account for May need to increase sample size to account for loss of power.loss of power.

Intervention Group Control Group

Physicians

Patient -> Outcome assessed

22

Looking at Usage DataLooking at Usage Data

Great way to tell how well the intervention Great way to tell how well the intervention is goingis going Target your trouble-shooting effortsTarget your trouble-shooting efforts

In terms of evaluating HIT:In terms of evaluating HIT: Correlate usage to implementation/training Correlate usage to implementation/training

strategystrategy Correlate usage to stakeholder characteristicsCorrelate usage to stakeholder characteristics Correlate usage to improved outcomeCorrelate usage to improved outcome

23

Studies on Workflow and UsabilityStudies on Workflow and Usability

How to make observations?How to make observations? Direct observationsDirect observations Stimulated observationsStimulated observations

Random paging methodRandom paging method Subjects must be motivated and cooperativeSubjects must be motivated and cooperative

Usability LabUsability Lab What to look for?What to look for?

Time to accomplish specific tasks:Time to accomplish specific tasks: Need to pre-classify activitiesNeed to pre-classify activities Handheld/Tablet PC tools may be very helpfulHandheld/Tablet PC tools may be very helpful

Workflow analysisWorkflow analysis Asking users to ‘think aloud’Asking users to ‘think aloud’

Unintended consequences of HITUnintended consequences of HIT

24

Cost Benefit AnalysisCost Benefit Analysis

Do the benefits of the technology justify the costs?Do the benefits of the technology justify the costs? Monetary benefits – Monetary costsMonetary benefits – Monetary costs Important in the policy realmImportant in the policy realm

Need to specify perspectiveNeed to specify perspective OrganizationalOrganizational SocietalSocietal

Cost analysis more straight forwardCost analysis more straight forward Prospective data collection preferredProspective data collection preferred Discounting: a dollar spent today worth more than a dollar 10 years from Discounting: a dollar spent today worth more than a dollar 10 years from

nownow Benefits analysis more controversialBenefits analysis more controversial

Cost of illness averted: medical costs, productivity for patient Cost of illness averted: medical costs, productivity for patient What is the cost of suffering due to preventable adverse events?What is the cost of suffering due to preventable adverse events? What is the cost of a life?What is the cost of a life?

25

Using Surveys – Stay Tuned!Using Surveys – Stay Tuned!

Survey of user believes, attitude and Survey of user believes, attitude and behaviorsbehaviors Response rate – responder bias: Aim for Response rate – responder bias: Aim for

response rate > 50-60%response rate > 50-60% Keep the survey conciseKeep the survey concise Pilot survey for readability and clarityPilot survey for readability and clarity Need formal validation if you want plan to Need formal validation if you want plan to

develop a scale/summary scoredevelop a scale/summary score

26

Qualitative Methodologies – Qualitative Methodologies – Don’t touch that dial!Don’t touch that dial!

Major techniquesMajor techniques Direct observationsDirect observations Semi-structured interviewsSemi-structured interviews Focus groupsFocus groups

Adds richness to the evaluationAdds richness to the evaluation Explains successes and failures. Generate Lessons learnedExplains successes and failures. Generate Lessons learned Captures the unexpectedCaptures the unexpected Great for forming hypothesesGreat for forming hypotheses People love to hear storiesPeople love to hear stories

Data analysisData analysis Goal is to make sense of your observationsGoal is to make sense of your observations Iterative & interactiveIterative & interactive

27

Concluding RemarksConcluding Remarks

Don’t bite off more than what you can chewDon’t bite off more than what you can chew Pick a few study outcomes and study them well. Pick a few study outcomes and study them well.

It’s a practical worldIt’s a practical world Balancing operational and research needs is Balancing operational and research needs is

always a challenge.always a challenge. Life (data collection) is like a box of Life (data collection) is like a box of

chocolates…chocolates… You don’t know what you’re going to get until you You don’t know what you’re going to get until you

look, so look early!look, so look early!

28

Thank youThank you

Eric Poon, MD MPHEric Poon, MD MPH Email: Email: epoon@partners.orgepoon@partners.org

AcknowledgementsAcknowledgements Davis Bu, MD MADavis Bu, MD MA

CITL, Partners HealthcareCITL, Partners Healthcare David Bates, MD MScDavid Bates, MD MSc

Chief, Div of General Medicine, Brigham and Chief, Div of General Medicine, Brigham and Women’s HospitalWomen’s Hospital