usability methods to improve emrs

1

Usability Methodsto Improve EMRs

Janey Barnes, PhDUser-View, Inc.

Raleigh, NC

Jeff Belden, MD Family & Community MedicineUniversity of Missouri

June 9, 2010

2

Conflict of Interest Disclosure

Janey Barnes, PhD• Contracted Research:

• Allscripts• Patagonia Health• Duke Health Systems

3

Conflict of Interest Disclosure

Jeff Belden, MD • Dr. Belden is on faculty at University of Missouri -

Columbia, and an EMR usability consultant with the Tiger Institute, a technology collaborative between the University of Missouri and Cerner Corporation.

• Dr. Belden has no financial interest.

4

Session Objectives

• Describe stages of software development in which usability methods can be most effective.

• List and describe usability methods applicable to each stage of EMR development.

• Analyze the cost/benefit of implementing usability methods within a user-centered design process.

5

Software Development Process

6

Waterfall modelof software development

7

Waterfall

8

“Agile” modelof software development

9

Agile

10

Usability methods work for both

11

don’t forget the buyer’s sideof usability

12

Selection Implementation

AgileWaterfall

13

Want to be a change-agent?

15

User Centered Design Process

Discover Define Design Develop Beta Launch Post-Launch

16


Incorporating User Centered Designin Software Development Process

•Affinity Diagramming•Personas•Use case scenarios

•Requirements•User stories

•Keep testing

•User acceptance testing•Incorporating feedback•Maintain design intent

•Summative testing

•and testing

•Prototyping•Card sorting•Concept exploration•Usability Testing

17

a few words about usability when selecting &

implementing

Selection Implementation


Software Selection ActivitiesUsability Methods of Interest• Site visit for:

• User surveys of current users like you• Performance testing

• Observe one of their experienced (but not an ace) users• Do clinical scenario test with real data• Define target measures• Set goals

18Selection Implementation


Implementation ActivitiesUsability Methods of Interest• Task analysis & user observation• Card sorting (picking categories, nomenclature)• Design guidelines & Style guides• Paper prototyping• Have clinician create documents for several patients

19Selection Implementation


21

Discovery ActivitiesContextual inquiry1. Affinity diagramming2. Personas3. Use case scenarios

• For each method we present• Goal of the method• Details for planning & executing each method• Best timing the software development schedule• How to deliver impact to the product


22

Contextual Inquiry

Persona


23

Goals• understand the user’s:

• tasks• workflows• environment

• description of a specific person (a target user). Details make them real.


Contextual Inquiry

Persona

Planning • Observe users onsite• Be a fly on the wall• Before, during & after• Half day…or longer!• Take photos of environment.

Execution• Watch. Observe. Learn. • Make notes. • Stand, sit, to side, in middle. • Stay out of way• Collect any artifacts.


Contextual Inquiry

Persona

• Timing• Early• Requirements gathering• Before sprint planning• Before “iteration 1”

• Having Impact• By watching, observing, learning

Contextual Inquiry

Persona


26

Contextual InquiryUse Case Scenarios

27


Goals• Understand the user’s:

• tasks• workflows• environment

• Specify how users carry out their tasks in a specified context.


28



Planning • Observe users onsite• Be a fly on the wall• Before, during & after• Half day…or longer!• Take photos of

environment.

Execution• Watch. Observe. Learn. • Make notes. • Stand, sit, to side, in

middle. • Stay out of way• Collect any artifacts.

29



• Timing• Early• Requirements gathering• Before sprint planning• Before “iteration 1”

• Having Impact• By watching, observing, learning

30

Contextual Inquiry

How long should this take?


5 days first time

Day 1

PlanDay 2

PlanDay 3

On-siteDay 4

CompileDay 5

Compile

Further visits could be…

1 dayOn-site

1.5 daysCompile

31

Contextual Inquiry

How long should we plan?


Plan• Teach team to observe• Not sell, or teach, or fix• Have tools ready

5 days first time

Day 1

PlanDay 2

PlanDay 3

On-siteDay 4

CompileDay 5

Compile

32

Contextual Inquiry

How long should we stay to observe?


It depends• How varied are user groups?• How familiar are you with territory?• Stay until you see patterns emerge

5 days first time

Day 1

PlanDay 2

PlanDay 3

On-siteDay 4

CompileDay 5

Compile

33

Contextual Inquiry

How long to compile findings?


Compile into• Personas• Use case scenarios• Work flows• Task flows

5 days first time

Day 1

PlanDay 2

PlanDay 3

On-siteDay 4

CompileDay 5

Compile

35

Definition Activities

1. Requirements 2. User Stories

• Goal• Planning• Execution• Timing• Impact


36

Requirements gatheringGoal• Identify usability requirements that can be

tested later• For each chosen task and user type estimate:

• acceptable task time & optimum target• how to score effectiveness - agreeing what errors user might

make• the effectiveness target• the satisfaction target.


37

Requirements gathering

• Arrange workshop with users, developers.

• Review tasks• For each task & user type

estimate• Task time & optimum goal• How to score effectiveness• Effectiveness target• Satisfaction Target

• Decide on usability requirements among:• Understandability• Learnability• Operability• Attractiveness

• Recruiting physicians is challenging.

Planning Execution


38

Requirements gathering

Timing• Early

Impact• Shows importance of usability early on• Provides concrete objectives & testable criteria

for usability


39

User Stories


40

User StoriesGoal

• More approachable than a formalized use case

• Slim, give high-level requirements

• Just a sentence or three.• “A surgeon needs latest evolving lab results while moving

through the hospital, without stopping to log-in at a PC”.


41

User StoriesPlanning• Arrange short meeting with

customer

• Bring 3x5 cards for each story

Execution

• User writes user stories

• 3x5 note card• Rewrite until clear• Developer may use

questions to get user going


42

User Stories

Timing• Early

Impact• Short, quick to generate• Low maintenance• Keeps contact with user• Helps estimate the development work effort


44

Design & User Feedback Activities

1. Information Design• Prototyping• Card sorting

2. Interaction Design• Prototyping

3. Visual Design• Concept exploration


45


Prototyping


46

Prototyping

Goal• clarify requirements• create draft of

interaction designs & screen designs to be rapidly simulated & tested


47

Prototyping

Planning• Arrange workshop with

• Users• Developers• Facilitator & scribe

• Four stages• Concept design• Interaction design• Screen design• Screen testing


48

PrototypingExecution• Concept design

• Sketch possible approaches

• See if they meet agree objectives

• Interaction design• Use Post-It notes,

writing each suggested screen or activity

• Group them, name clusters, arrange sequences


49

PrototypingExecution• Screen design

• Brainstorm screen designs with users

• Ask user to carry out a realistic task, pointing among screen sketches

• Screen testing• Mock-up rough designs.

• Walk thru steps, explaining what happens, or get fancier with paper menus, dialog boxes, etc.


50

Prototyping

Timing• Early• Before codeImpact• Find usability problems early before code is

written• Communication between users & designers is promoted• Paper is cheap & quick


51


Card sorting


52

Card SortingGoal• Aids information design• Discover latent structure in an unsorted list

of ideas• Example: “Group these smart-phone functions into

ones that you need, and ones that would be optional. Then rank the needed ones in order of importance.”


53

Card SortingPlanning• Make separate cards for

each statement• Number cards on back• Find at least 6 users

Execution• Shuffle the deck each time.• Have users group cards• Two styles of card sort

• Closed (you supply category)• Open (user names groups)

• Note results. Include names user gave groups, proximity info or comments user made.


54

Card SortingTiming• Design phase• Before codingImpact• Helps discover users’ mental model• Shows how ideas or concepts should be

presented


Speaker handoff

56


Visual DesignConcept Exploration • Goal

• Communicating information organization• Communicating information priority• Engaging brand


57

Visual Design

It’s not just pretty colors…


59

Visual Design

Good visual design communicates: Info organizationInfo priority

and reduces cognitive load


60


Visual DesignConcept Exploration • Planning: Iterative Process• Execution: Research methods to

quantify effectiveness of Visual Design • Timing: In parallel with information &

interaction design• Impact: Critical! Visual Design is not

just “make it pretty”


Use CasesNew customer: Salesman wants to add new customer information, but doesn’t have a truck to spec at this time.

Edit customer: Salesman needs to update the street address of an existing long-time customer.

New Truck/Existing Customer: Salesman needs to create a new spec and relate to an existing customer.

Existing Truck/Existing Customer: Salesman needs to relate existing truck spec to current customer.

Existing Truck/New Customer:Salesman need to input new customer information and related to existing spec

Affinity Diagramming

PrototypingLow-resolution High-resolution

64


Usability Tests


65

Usability Tests


66


Usability Tests - one-on-one sessions where participant user performs key/at risk tasks while researcher gathers data related to performance

Goals


• Formative• Design decisions• Identify usability issues

• Summative• Evaluate usability goals• Validate design

Planning • Test Objectives• Test Design• Test Plan including data analysis plan

• Test Participants• Test Environment• Test Report

Execution• Pilot Test• Logistics• Carry out plan

including test and data analysis

• Share with team & IMPACT design

Usability Tests

Formative and Summative

67

68

Timing

Usability Tests



Formative• Earlier• As soon as you have use cases in a prototype

Summative• Later• Enough• Stability• Early enough to make IMPACT

Fit to sprints• UX a sprint ahead• UX doing formative activities for next sprint and summative activities for current sprint

• Having Impact• Actionable & Prioritized

Recommendations• Here are 3 examples…

Usability Tests



Usability TestsActionable & prioritized recommendations


High Priority

Users (patients) cannot find Clinician’s name in the list of physicians (appointment task). Put names in alphabetical order

High Priority

Users (physicians) cannot find “New Prescription” action button. Make “New Prescription” action button visually apparent.

High Priority

Users (physicians) cannot find patient’s name on open chart quickly. Put the patient’s name in a visually distinct location on the screen and be consistent on all screens.

71

Development Activities

1. User Acceptance Testing (Software)2. Incorporating user feedback3. Maintaining the intent of the design


Keep the door open betweenUX and Development!

72

Development Activities1. User Acceptance Testing (Software)

• Goal: Validate that the application works as intended

• Planning: Begins during Define Phase• Execution: Automatic versus Manual• Timing: Nearing the end• Impact: Too late to IMPACT the application?


http://www.google.com/imgres?imgurl=http://www.clipartpal.com/_thumbs/pd/education/caution.png&imgrefurl=http://www.clipartpal.com/clipart_pd/education/studentssigns_12054.html&usg=__LSR3ShqOtlkyjkv9meCGwrt42K0=&h=908&w=908&sz=59&hl=en&start=1&itbs=1&tbnid=l4Iss7XxWTNW8M:&tbnh=147&tbnw=147&prev=/images?q=caution+sign+free+clip+art&hl=en&sa=G&gbv=2&tbs=isch:1

http://www.google.com/imgres?imgurl=http://www.clipartpal.com/_thumbs/pd/education/caution.png&imgrefurl=http://www.clipartpal.com/clipart_pd/education/studentssigns_12054.html&usg=__LSR3ShqOtlkyjkv9meCGwrt42K0=&h=908&w=908&sz=59&hl=en&start=1&itbs=1&tbnid=l4Iss7XxWTNW8M:&tbnh=147&tbnw=147&prev=/images?q=caution+sign+free+clip+art&hl=en&sa=G&gbv=2&tbs=isch:1

73

Development Activities2. Incorporating user feedback3. Maintaining the intent of the design

• Goal: SUPPORT development team• Planning: Little planning time – rely on process• Execution: SUPPORT• Timing: Nearing the end• Impact: CRITICAL time


74

Beta Testing Activities


GoalFrom Usability Perspective:

• Live usability test to inform current application and next release

• System-wide usability test • Preparation for implementations

Usability Activitiesduring Beta Testing

75

Beta Testing Activities


Planning • Test Objectives• Test Design• Test Plan

including data analysis plan• Test Participants• Test Environment• Test Report


including test and data analysis• Share with team &

IMPACT design, implementation

Usability Activitiesduring Beta Testing

76

Beta Testing ActivitiesUsability Activities

during Beta Testing


Timing: We are nearing the endImpact: Not too late to impact the application

77

Launch Activities


Goal• From Usability Perspective:

• Live usability test to inform current application and next release

• System-wide usability test

Continued Testing

78

Launch ActivitiesContinued Testing


Planning • Test Objectives• Test Design• Test Plan

including data analysis plan• Test Participants• Test Environment• Test Report


including test and data analysis• Share with team &

IMPACT design, implementation

79

Launch Activities


Timing: We are nearing the endImpact: Not too late to impact the application

Continued Testing

80

Post Launch ActivitiesLearnings Discover Activities

DiscoverDefine

Design

Develop

Beta

Laun

ch

Post-

Launch

Post-

Launch

Implementation

Implementation

Selection

81

Cost / Benefit

82

Cost/Benefit

Key to cost-effective product usability is to

Plan & Managethe usability activities within the development process.

Jeff Belden MD

Can link to interactive Usability Methods Table at http://www.usabilitynet.org/tools/methods.htm

83

Cost/Benefit

specific activity designed to gather data to address specific research question

•Affinity Diagramming•Personas•Use case scenarios

•Requirements•User stories

•Keep testing

•User acceptance testing•Incorporating feedback•Maintain design intent

•Summative testing

•and testing•Prototyping•Card sorting•Concept exploration•Usability Testing


Formative Summative Formative

84http://bit.ly/UXmethods

85

CostsIncluding Usabilityin Development Cycle

Planning Design Development Release

Design Alternatives

Cost of Changes

86

Costs Usability in the Development Cycle 1 of 2

• Discount Usability vs Premium Usability• Time (labor)

Employee vs Contractor (expendable)• Design vs Evaluation• Test system setup• Participant recruitment and compensation

87

Costs Usability in the Development Cycle 2 of 2

• Lab equipment & usability lab facility• Travel expenses• Planning Time !!

• Managers must make the time to plan the integration of usability into the development process.

• Sometimes too busy to stop and determine how to integrate usability during development.

88

Measurable Benefitsof Usability Integration

• Improved productivity• Shorter time to

complete tasks• Fewer errors during

tasks• Improved user

satisfaction• Lower training costs

• Reduced tech support & documentation costs

• Increased patient safety• Litigation deterrence• User preferences

• Brand loyalty• Marketing voice

89

Tradeoffs between Usability Costs & Benefits

• Measurable benefits not well documented• “Post” data is often not collected due to

budget and lack of vision• Proprietary data used competitively and not

available to the public• Usability community has made efforts to

quantify benefit of Usability interventions

Janey Barnes

Janey - find more recent case scenario

90

Tradeoffs between Usability Costs & Benefits

• Case scenarios presented as examples that hopefully demonstrate the cost-benefit tradeoffs across several types of HF activities.

• Data for case scenarios taken from:Good Ergonomics Is Good Economics (1996). Hal

Hendrick. Proceedings of the HFES 40th.Usability Professionals’ Association (UPA) http://

www.upassoc.org

http://www.upassoc.org/

http://www.upassoc.org/

Janey Barnes

Janey - find more recent case scenario

91

Best Bang for the BuckKey Usability Activities

• Contextual Inquiry• Usability Testing (1 on 1) with

prototype or early code• Enough functionality to test• Early enough to make changes


92

Session Objectives1. Describe stages of software development in which

usability methods can be most effective.• Requirements & Design

2. List and describe usability methods applicable to each stage of EMR development.• Contextual Inquiry & Usability Testing

3. Analyze the cost/benefit of implementing usability methods within a user-centered design process.• Early usability methods are most cost effective in terms

of planning, carrying out, and incorporating findings in software.

Links• Usability Methods Table | Usability.net• HIMSS EHR Usability Links | Online resources

http://www.usabilitynet.org/tools/methods.htm

http://www.himss.org/ASP/topics_FocusDynamic.asp?faid=358

Questions?For further info, contact:Janey Barnes PhD | [email protected]

Jeff Belden MD | [email protected]

usability methods to improve emrs

Health & Medicine

usability testing

usability methods applicable

usability methodsto

emr usability consultant

software development

usercentered design

user centered design

user stories