The role of Usability & UCD

in eHealth ProcurementAnne Miller, PhD, Assoc. Prof

Vanderbilt University Medical Center,University of Sydney,

eHealth NSW

Acknowledgements

The research reported in this presentation was funded by Vanderbilt University Medical Center,

Nashville, Tennessee.

The author acknowledges the contributions of Shilo Anders, Ray Booker, Dan France, Tiercy Forteberry, Thad

Hoffman, Peggy Joseph, Jennifer Slaughter, Marcella Woods, PhD, and Matt Weinger MD, PhD.

The views expressed in this presentation are the presenter’s own and may not reflect those of past or current employers.

e-health implementations• Include:

– Electronic health records– Practice management systems– Dispensing & laboratory

systems– Medical devices,– Personal use apps

HIT implementation outcomes

• 62% of HIT studies report positive outcomes (improvement in an aspect of care without negative effects)

• 38% of HIT studies reported mixed or negative outcomes

• Quantitative efficiency / effectiveness studies more positive outcomes

• Qualitative satisfaction studies find more negative outcomes

Buntin et al (2011) Health Affairs

Nature of HIT is changing…To now… •Isolated functions (e.g., CPOE)•Limited user group (e.g., physicians)•Defined processes (e.g., ordering)•One-to-one mapping intervention & outcomes

Next generation…•Multiple, network functions (e.g., Whatsapp)•Multi-user groups (incl. patients)•Emergent processes (image sharing & messaging)•Many-to-one intervention to outcome mapping

How can Human Factors help…

What is Human Factors?The Science and Practice of Socio-technical Systems design for efficient, effective, safe

and satisfying work.

•Science & Practice application of theory & method•‘Human’ psychology, sociology, anthropology•Business/Organization change management,

strategic directions•Technical systems engineering, physics, chemistry,

biology & domain expertise•Design interaction, interface, graphic, industrial

Usability & User Centered Design

User Centered Design (UCD)A process for achieving software quality that depends on:•Evidence-based understanding of contexts of use•The direct involvement of representative end-users•Iterative design & evaluation•Usability testing

Usability:Allows users to complete tasks in a context of use efficiently,

effectively, safely and with satisfaction

HF & Usability in HIT Procurement: Case Study

Positive Patient Identification project

Problem:VUMC had short listed two barcode blood checking products and wanted to make a purchasing decision. Could HF evaluation help to differentiate?

UCD in HIT Procurement: Case Study

Center for Research & Innovation in Systems Safety (CRISS):

Which of the two products:1.Is most likely to be accepted and used by clinicians; 2.Is easiest to learn/use; has lowest training costs3.Supports clinicians’ cognitive and work processes;4.Reduces blood administration errors, while not introducing new errors.

UCD in HIT Procurement: Case study Method:

Near-realistic multi-agent simulations: Operating Room, Emergency Department, Oncology Clinic & Paediatric Intensive Care Unit.

Participants: 22 clinicians drawn from each context

– 11 participants used product A– 11 participants used product B

Test tasks:1) Routine 2) Emergency compared to manual3) Wrong patient identification; 4) Wrong blood type identification

VUMC simulation center

UCD in HIT Procurement: Case study Results

Efficiency (Time on task): •No statistically significant differences Product A vs Product B•Manual vs Product A/B checking in emergency infusion: 15 vs 21secs

Effectiveness (# interaction errors / product / scenario): Product A clinicians 1.2 ±1.8 vs Product B clinicians 2.8 ± 4 interaction errors (p<0.05)

Safety:All wrong patient / wrong blood type errors identified. One potential error was identified.

Satisfaction ratings:No statistically significant differences in mean ratings

“It doesn’t give clear instructions—it tells you that the scan is wrong but doesn’t tell you which scan [is wrong],”

“You have to rework vitals, you write them and enter later when the screen comes up”

“I hit cancel and had to restart from the beginning.”

“It’s like using a brick - why can’t we have a (scanning) gun like the supermarkets use?”

UCD in HIT Procurement: Case study Conclusions

1.Is most likely to be accepted and used by clinicians; 2.Is easiest to learn/use; has lowest training costs;3.Supports clinicians’ cognitive and work processes;4.Reduces blood administration errors, while not introducing new errors.

“It’s like using a brick - why can’t we have a gun like the supermarkets use”

UCD in HIT Procurement: Case study Outcomes:

1. Each vendor received a confidential & comprehensive product specific report and were present during their product tests2. Results were published in ‘Transfusion’

Anders S, Miller A, et al. (2011) Blood product positive patient identification: Comparative simulation-based usability test of two commercial products. Transfusion. 51: 2311-2318

3. VUMC elected not to purchase either due to technological immaturity

– Levels of frustration experienced by staff mitigates against adoption - system likely to be abandoned

– Unlikely return on investment

Blood transfusion cross checking

Limitations• Simulated environments are not real environments• Effectiveness depends on detailed understanding of

contexts of use for representative test scenarios• Issues identified may not be remediable in current

HIT versions• May not capture issues effectively in more fully

networked HIT configurations

Healthcare without humans!

Conclusions• Human Factors & User Centered Design:

– Methods can provide purchasers and vendors with greater outcome certainty during procurement

– Fills a common gap in procurement processes – involvement by front-end clinicians doing realistic tasks

• As we move towards more networked configurations, Human Factors evaluation approaches are moving towards a more broader ‘pilot’ review approach that better fits real-world complexities

Thank you • Questions?

Nashville, Tennessee