himss ehr usability task force report on defining and testing emr usability

8/14/2019 HIMSS EHR Usability Task Force Report on Defining and Testing EMR Usability

1/42

2009HealthcareInformationandManagementSystemsSociety(HIMSS)

DefiningandTestingEMR

Usability:

Pr inc ip les and Proposed Met hodsof EMR Usabi l i t y Evaluat ion and

Rat ing

HIMSSEHRUsabilityTaskForce

June2009


2/42


CONTENTS

EXECUTIVESUMMARY.......................................................................................................................................1

INTRODUCTION..................................................................................................................................................2

WHATISUSABILITY?.......................................................................................................................................................3

THECHALLENGESOFEMRDESIGN....................................................................................................................................

3

SCOPEOFTHISREPORT....................................................................................................................................................4

USABILITYPRINCIPLES........................................................................................................................................4

SIMPLICITY....................................................................................................................................................................5

NATURALNESS...............................................................................................................................................................6

CONSISTENCY................................................................................................................................................................6

MINIMIZINGCOGNITIVELOAD..........................................................................................................................................6

EFFICIENTINTERACTIONS.................................................................................................................................................7

FORGIVENESSANDFEEDBACK...........................................................................................................................................7

EFFECTIVEUSEOFLANGUAGE...........................................................................................................................................7

EFFECTIVEINFORMATIONPRESENTATION............................................................................................................................

8

PRESERVATIONOFCONTEXT...........................................................................................................................................10

USABILITYEVALUATIONANDRATINGMETHODS...............................................................................................10

EVALUATIONMETHODSANDMETRICS.............................................................................................................................12

SELECTINGTASKSFOREVALUATION.................................................................................................................................17

5STARUSABILITYRATINGSYSTEM..................................................................................................................................18

CERTIFICATIONANDEMRUSABILITYRATING....................................................................................................................18

RECOMMENDATIONS........................................................................................................................................19

REFERENCES......................................................................................................................................................20

CONTRIBUTORS................................................................................................................................................24

AUTHORS...................................................................................................................................................................24

USABILITYPRINCIPLESWORKGROUP...............................................................................................................................24

EDITORS.....................................................................................................................................................................25

HIMSSEHRUSABILITYTASKFORCE................................................................................................................................25

APPENDICES......................................................................................................................................................27

A. TESTTASKANDSCENARIOEXAMPLES.......................................................................................................................27

B. BENCHMARKEXAMPLES........................................................................................................................................35

C. PATIENTSAFETYCHECKLISTEXAMPLES.....................................................................................................................36

D.

USABILITYPRINCIPLESWORKGROUPBIOS...............................................................................................................

38


3/42


EXECUTIVE SUMMARY

Electronicmedicalrecord(EMR)adoptionrateshavebeenslowerthanexpectedintheUnitedStates,

especiallyincomparisontootherindustrysectorsandotherdevelopedcountries.Akeyreason,aside

frominitialcostsandlostproductivityduringEMRimplementation,islackofefficiencyandusabilityof

EMRscurrentlyavailable.AchievingthehealthcarereformgoalsofbroadEMRadoptionand

meaningfulusewillrequirethatefficiencyandusabilitybeeffectivelyaddressedatafundamental

level.

Weconductedaliteraturereviewofusabilityprinciples,especiallythoseapplicabletoEMRs.Thekey

principlesidentifiedweresimplicity,naturalness,consistency,minimizingcognitiveload,efficient

interactions,forgivenessandfeedback,effectiveuseoflanguage,effectiveinformationpresentation,

andpreservationofcontext.

Usabilityisoftenmistakenlyequatedwithusersatisfaction,whichisanoversimplification.Wedescribe

methodsof

usability

evaluation,

offering

several

alternative

methods

for

measuring

efficiency

and

effectiveness,includingpatientsafety.Weprovidesamplesofobjective,repeatableandcostefficient

testscenariosapplicabletoevaluatingEMRusabilityasanadjuncttocertification,andwediscussrating

schemaforscoringtheresults.

1


4/42


INTRODUCTION

TherateatwhichEMRshavebeenadoptedinclinicandhospitalsettingswithintheUnitedStateshas

laggedbehindtheadoptionofinformationtechnologythathasbeencommoninotherindustriesfor

morethan20years.19

Multiplecauseshavebeensuggestedincludingcost,resistancetochange,fearoravoidanceof

technology,andingrainedpatternsofbehavior.Increasingly,however,usabilityhasbeenacknowledged

asadeterrenttoadoption40,51,35,23,andonethatmustbeaddressed.

Wesubmitthatusabilityisoneofthemajorfactorspossiblythe

mostimportantfactorhinderingwidespreadadoptionofEMRs.

Usabilityhasastrong,oftendirectrelationshipwithclinical

productivity,errorrate,userfatigueandusersatisfactioncritical

factorsforEMRadoption.Cliniciansloseproductivityduringthe

trainingdays

and

for

months

afterward

as

they

adapt

to

the

new

tools

andworkflow.Someproductivitylossesaresustained,mostlydueto

longertimeneededforencounterdocumentationincomplex

patients31.

Effectivetrainingandimplementationmethodsaffectuseradoption

ratesaswell,buttrainingisbothharderandmorecostly,andimplementationismorecomplexand

difficultwhenusabilityislacking.

IthasproveddifficultforclinicianstoevaluateEMRusabilityaspartofthepurchaseprocessforseveral

reasons.Properevaluationbypurchasersrequiresindepthstudyusingunfamiliarskills.Mostusersof

oneEMR

often

have

not

experienced

other

EMRs,

so

single

product

ratings

are

less

helpful

than

those

whichcomparesystems.9Therehasbeenworkdonebythirdpartyconsultinggroupstosurveycurrent

users,buttheseresultsarerarelyprovideddirectlybyclinicalendusers.Industrysurveyinstrumentsare

generallynotconstructed toprovidereliableusabilitydata;theyalsoonlyprovideusersatisfaction

ratingsasinglecomponentofusability.

Thispaperwill:

1. DescribeanddefineusabilityasitpertainstotheEMR.

2. Identifyasetofwellestablishedprinciplesofusabilityanddesign.

3. Offer

potential

methods

of

assessing

and

rating

EMR

usability.

Wesubmitthattheseprinciplesandmethodscouldbeusedbycertificationorganizationstotestand

rateproductsforusability.Requiringthisadjuncttocertificationmayspurdevelopmentofmoreusable

EMRproducts,andallowdecisionmakersmoreconfidenceinchoosingaproductthatwillbenefit

clinicians.

InBrief:Adoptionratesby

physiciansandhospitalshavebeen

slowerthanexpectedintheUS,in

part,duetopoorefficiencyand

usability.Weexplorewell

establishedusability

principles

and

testingmethodsandpropose

methodstotestandrateEMRsfor

efficiency,effectivenessandsafety.

2


5/42


What is Usabi l i t y?

Usabilityistheeffectiveness,efficiencyandsatisfactionwithwhichspecificuserscanachieveaspecificsetoftasksinaparticularenvironment.39Inessence,asystemwithgoodusabilityiseasytouseandeffective.Itisintuitive,forgivingofmistakesandallowsonetoperformnecessarytasksquickly,

efficientlyandwithaminimumofmentaleffort.Taskswhichcanbeperformedbythesoftware(suchas

dataretrieval,organization,summary,crosschecking,calculating,etc.)aredoneinthebackground,

improvingaccuracyandfreeinguptheuserscognitiveresourcesforothertasks.

Usabilityevaluationisfarbroaderthanthesimpleprocessofmeasuringusersatisfaction.Justas

importantly,usabilitymetricsincludemeasuresofefficiency,effectiveness,cognitiveloadandeaseof

learning.Usabilityemergesfromunderstandingtheneedsoftheusers,usingestablishedmethodsof

iterativedesign,andperformingappropriateusertestingwhenneeded.Thereareawiderangeof

designandevaluationmethodologies,bothsubjectiveandobjective,whicharecontinuallygrowingin

sophistication.BuiltinwebcamsonmodernlaptopPCs,robustwirelessnetworking,remotetesting

software,andcompact,inexpensivevideorecordersmakeitincreasinglyeasiertotestinliveclinical

settings.

The Challeng es of EMR Design

ItisparticularlychallengingtodevelopexcellentusabilityinEMRsystems.Thereisawiderangeof

complexinformationneeds,whichvaryfromsettingtosetting,amongdifferentadministrative,financial

andcliniciangroups,andfromtasktotaskwithinagroup.Thereareover50physicianspecialties(AMA

specialtycodes2)eachwithitsownsoftwareneeds,aswellasthesoftwareneedsofotherclinical

groupssuchasnurses,pharmacists,physicaltherapists,respiratorytherapists,medicaldieticiansand

others.Eachdisciplinemayhaveseveraldifferenttaskscenariosinaworkingday,witheachscenario

demandingadifferentsoftwareinterfacedesign.

Cliniciansareoftenmobile,goingfromroomtoroom,hospitaltoclinic.Theyseldomgivetheirfull

attentiontothesoftware.Theirprimaryfocusshouldbeonthepatient,andcliniciansareoftentalking,

listeningorthinkingwhileusingthesoftware.Theyoftenhaveafrequentlychangingagendaduringa

singlepatientworkflow,andinterruptionsarecommon.

Administrativeandfinancialissuescomplicateevenroutinetasks(providingbillingcodes,discovering

drugformularycoverage,pursuingpriorauthorization)andvarywidelywithdifferentinsurers.Thereisa

burgeoningimpetustomeasurequalityofcare,complicatedbymultiplestandards.

ItmaybechallengingforEMRdeveloperstogetaccesstoclinicianusersforfeedbackortesting.Busy

physiciansallow

only

limited

access

for

user

centered

design

work.

Clinicians

have

other

significant

constraintsthatcomplicateusabilityevaluations,suchasconfidentialityconcernsinalltheirencounters,

theneedtotestintheactualworkenvironment,andfrequentinterruptionsintheirworkflow.

3


6/42


Scope of th is Report

WediscusstheusabilityoftheEMRfromtheperspectiveofclinicianusers(physicians,nurses,

pharmacists,physicaltherapists,respiratorytherapistsandothers)intheambulatory,inpatientand

acutecareenvironments.Weconfinedourselvestoissuesofusercentereddesignandusability

evaluation.Theseconceptsapplytovendorproductdevelopmentprocesses,publicproductusability

ratingmethodsandvendorselectioncriteriaforhealthcareorganizations.Inaddition,thesemethods

shouldbeappliedintheconfigurationofhighlyadaptablesystemsduringimplementation.Wedonot

otherwiseaddressconcernsofimplementation,usertrainingorchangemanagement,thoughthese

issuesdoaffectuseradoptionsuccessrates.

Terminologynote:AnEMRisacomputersystemcomposedofmultiple,integratedapplicationsenabling

clinicianstoorder,documentandstorepatientinformation.Thetermelectronichealthrecord(EHR)is

sometimes,andincorrectly,usedinterchangeably.Incontrast,anEHRispatienthealthinformationfrom

multiplecaredeliveryorganizationsEMRs,comprisingapatientcentric,longitudinalviewofapatients

encounterswithhealthcareproviders.13Forthepurposeofthispaper,thetermEMRwillbeused,aswe

areaddressing

systems

vs.

data.

USABILITY PRINCIPLES

Inrecentyears,usabilityhasbecomeanincreasinglyprevalenttopicinthehealthinformation

technology(HIT)literatureandmedia.ManyHITprofessionals,healthcareinformaticiansand

researchershaveclearlyarticulateddesignproblemsinthecurrentgenerationofclinical

applications.40,20,8,50TheNationalResearchCouncil(NRC)hasassertedthattodaysclinicalsystems

provide

poor

support

for

the

cognitive

tasks

and

workflow

of

clinicians.

28

These

problems

can

dramaticallyimpactuseracceptanceandproductivity.

Patientsafetyisaprominentconcernintheliterature.TheJoint

Commission(formallyknownasJointCommissiononAccreditationof

HealthcareOrganizations)recentlyissuedSentinelEventAlert42

regardingtechnologyrelatedadverseevents(TheJointCommission,

2008).ThissafetyalertincludedEMRs,computerphysicianorderentry

(CPOE)andclinicaldecisionsupport(CDS)systems.Theyreportedthat

approximately25percentofmedicationerrorsincludedinthe2006

PharmacopeiaMEDMARXinvolvedcomputertechnologyasacontributing

cause.Theoverwhelmingmajorityofthese(82percent)stemmedfrom

CPOEandotherdataentryfunctions.Manystudieshavedocumentedthe

issuesofalertfatigue,screenfragmentation,terminologyconfusionandlackofappropriatedefaultsin

CPOEandCDSsystems.8,6,21

InBrief:Wellknownprinciples

suchassimplicity,naturalness,

consistency,protectionagainst

cognitiveoverloadandothers,

definegoodusability.

Incorporatedeepknowledgeof

theuserstasks,contextand

workflows.

4


7/42


Notallauthorsdiscusstheseissuesdirectlyintermsofusability.Farfewerpresenttheseproblemsin

termsoftheunderlyingdesignprinciplesbeingviolated.Itisthislevelthatmustbeaddressedinorder

todesignapplicationsthatwillachievedesiredefficiency,broadusage(aprerequisiteformeaningful

use)andavertsystemfacilitatedsafetyerrors.

Expertsin

usability

and

Human

Factors

have

published

many

compilations

of

principles

and

guidelines

to

aidindesigningthemosteffectiveuserinterfaces.30,32,46,49Mostoftheselistssharecertaincoreideas.

Principlesmustbeevaluatedfortheirimportancetotheparticularcontextofuse.Thosethatareofkey

importancetothedesignofanEMRsystemarediscussedbelow.Theseprincipleswereselectedfor

discussionbasedontheircontributiontotwoessentialfactorsforclinicianacceptanceandsystem

success:

1)Efficiencyofuse.

2)Minimizinglikelihoodofusererror.

Usererrorshaveadirectrelationshiptopotentialpatientsafety.Usererrorsmaybeeithererrorsof

commissionor

errors

of

omission:18

Example,errorsofcommission:

o Selectingthewrongpatient,wrongmedication,wrongdosageorwrongencounter.

Example,errorsofomission:

o Overlookingormisinterpretingkeydataduetopoorinformationdisplay(e.g.,overlooking

criticallyabnormallabresult,orroutinelydismissingacriticallyharmfuldrugdrug

interactionwarning).

o Failingtocompleteatask(perhapsduetointerruption)suchastransmittingordersor

signingdocumentation.

Testingmethodswhichmeasureefficiency,effectiveness,easeoflearningandusersatisfactionhave

beendevelopedtotaketheseusabilityprinciplesintoaccount.Tousethemethodsproperly,the

principlesbehindthemshouldbewellunderstood.

Simpl ic i ty

Simplicityindesignreferstoeverythingfromlackofvisualclutterandconciseinformationdisplayto

inclusionofonlyfunctionalitythatisneededtoeffectivelyaccomplishtasks.Alessismorephilosophy

isappropriate,withemphasisbeinggiventoinformationneededfordecisionmaking.29,26Themore

complexan

application,

the

more

important

this

principle

becomes.

Clinical

systems

are

complex

as

well

asinformationdenseitisessentialforefficiencyaswellaspatientsafetythatdisplaysareeasytoread,

thatimportantinformationstandsout,andthatfunctionoptionsarestraightforward.Simplicityasa

principleshouldnotbeinterpretedassimple.Clear,cleanscreendesignrequiressubstantiallymore

effortthancluttereddisplays;italsomaymeanthatsomecomplexityhasbeenremovedfromthe

5


8/42


surfaceandmovedunderthehood.Simplicityappliestoanydesignregardlessoftheexperiencelevel

ofthetargetuser.

Natura lness

Naturalnessreferstohowautomaticallyfamiliarandeasytousetheapplicationfeelstotheuser.

Factorsthat

contribute

to

this

feeling

include

terminology

used

in

the

interface

and

how

well

the

design

andscreenflowsmaptotheuserstasksandexpectations.30Thisiscriticaltoclinicalapplicationssinceit

isextremelydifficulttoprovideextensivetraining,especiallytoclinicianswhosetimeislimitedand

fragmented.Goodworkflowdesigncancontributesignificantlytoefficiencyandreducecognitiveload.

Naturalworkflowcanvarydramaticallyfromonespecialtytoanotherorinanacutesetting,fromone

departmenttoanother.AnEmergencyDepartmentsworkflowisverydifferentfromthatofaninpatient

medicalsurgicalunit.Likesimplicity,naturalnessalsocontributestoerrorreduction.

Cons is tency

Externalandinternalconsistencyareimportanttothedesignofanyapplication.Externalconsistency

primarilyhas

to

do

with

how

much

an

applications

structure,

interactions

and

behaviors

match

ausers

experiencewithothersoftwareapplications.Themoreausercanapplypriorexperiencetoanew

system,thelowerthelearningcurve,themoreeffectivetheirusage,andthefewertheirerrors.An

internallyconsistentapplicationusesconcepts,behavior,appearanceandlayoutconsistently

throughout.30,46,49Predictabilityisanotherimportantfactorinenablingefficientuseandreducingerrors.

Minim iz ing Cogni t ive Load

Whilethisprinciplemaysoundabitesoteric,itisessentialforacomplex,informationdenseclinical

application.Clinicians,inparticular,arealmostalwaysperformingundersignificanttimepressureandin

environmentsburstingwithmultipledemandsfortheirattention.Combinedwiththestaggering

informationload

faced

by

todays

providers,

this

can

be

arecipe

for

cognitive

overload,

which

could

negativelyimpactpatientsafety.

Presentingalltheinformationneededforthetaskathandreducescognitiveload.Forexample,when

reviewingresultsofalipidprofile,theproviderwillwanttoseethepatientslatestandpriorresults,the

medicationlist,theproblemlistandallergylistallinthesamevisualfieldsothatdecisionsand

subsequentactionsmaybeperformedwithoutchangingscreens.Displayinginformationorganizedby

meaningfulrelationshipsisonemethodofprovidingcognitivesupporttotheuser.50,28

AnEMRmustnotonlyassistwithtaskperformanceanddecisionmaking,butstrivefortransparency.

Indesignterms,transparencymeansthatuseofthesoftwareapplicationdoesnotcreatetoomany

intrusivethoughtsfortheuserlikeHowdoI?,Whatdoesthisdo?orWhereis...?These

mentalinterruptionscancausetheusertolosetheirthoughtprocessaboutthetaskordecisionmaking

processinwhichtheyareengaged.Inotherwords,theusershouldnothavetothinktoomuchabout

theapplicationitself.22

6


9/42


Cognitiveloadisincreasedbyanyaspectsofadesignthatdonotfollowtheprinciplesofsimplicity,

naturalnessandconsistency.Itisalsoincreasedifauserisrequiredtorelyonmemory(recall)rather

thanvisualrecognition,ifausermusttrytorememberinformationfromonescreentoanother,whata

buttonreallydoes,orwhatnamesomethingiscalledasinanorderableslist.Highinformationdensity,

poorfeedbacktotheuserandinadequatecuesfordataentryfieldsalsoaffectcognitiveload.

Ef f ic ient In terac t ions

Oneofthemostdirectwaystofacilitateefficientuserinteractionsistominimizethenumberofstepsit

takestocompletetasksandtoprovideshortcutsforusebyfrequentand/orexperiencedusers.46While

thisissomewhatstatingtheobvious,itisincludedherebecauseofitsimportancetotheuser

acceptanceofaclinicalapplication.Otherexamplesofdesigningforefficientinteractionsincludeauto

tabbing;gooddefaultvalues;largeenoughlistandtextboxestolimitscrolling;andpreventingtheneed

forfrequentswitchingbetweenkeyboardandmouse.49Somewhatlessobviousfactorsincludeattention

tominimizingtheamountofvisualsearchingrequiredtolocateinformationandthedistancethecursor

musttraveltomakeselections.Excessivecursormovementandvisualscanningbothcontributetouser

fatigueand

frustration.

Forgiveness and Feedback

Forgivenessmeansthatadesignallowstheusertodiscoveritthroughexplorationwithoutfearof

disastrousresults.Thisapproachaccelerateslearningwhilebuildinginprotectionsagainstunintended

consequences.30,46Thisisespeciallyhelpfuliftrainingislimited.Goodfeedbacktotheusersupportsthis

goalbyinformingthemabouttheeffectsoftheactionstheyareabouttotake.Campbelletal.8provide

ananalysisofthetypesofunintendedconsequencesrelatedtoCPOE.Forgivenessandfeedbackwork

togethertoreduceusererrorsandprovidegracefulrecoverywhenmistakesaremade.Goodfeedback

alsoreassurestheuserthattheiractionshavehadthedesiredeffect.Likeconsistency,theseprinciples

arestandard

in

the

design

of

any

application,

but

of

special

importance

in

aclinical

information

system

duetotheimpacttheycanhaveonusererrorsaswellascognitiveload.

Effect ive Use of Language

AlllanguageusedinanEMRshouldbeconciseandunambiguous.Terminologyusedalsomustbethat

whichisfamiliarandmeaningfultotheendusersinthecontextoftheirwork;notermsrelatedto

computers,technology,HL7,databases,etc.shouldappearintheuserinterface.Thisappliesto

everything:labels,descriptions,picklistsanderrormessages.

Textshouldneverbedisplayedinalluppercase;thisisconsideredshouting.Itismoredifficultand

takeslonger

to

read,

and

increases

perceived

density.

Even

iflists

of

orderables

or

terms

are

received

by

theEMRinuppercase,theyshouldbetranslatedtotitlecasebeforedisplayintheinterface.Rare

exceptionsincludeoneortwowordmessagesthatareintendedtodrawtheattentionoftheuser.26,49,38

Abbreviationsandacronymsshouldonlybedisplayedwhentheyarecommonlyunderstoodand

unambiguous.49,50Informationthatmustbespelledoutbuttakesmorespacethanavailableshouldhave

ellipsesinsertedtoindicatethereismorewiththefulltextavailableonmouseover.Thisisinparta

7


10/42


patientsafetyissue.Forefficiency,however,alargernumberofcommonabbreviationsandsynonyms

shouldbeavailabletotheuserforthepurposesofdataentryandsearching,expandingifnecessaryfor

display.

AlanguageissuespecifictoEMRdesignistheneedtocapturestructured(discrete)clinicaltermsfrom

providerdocumentation

such

as

visit

notes,

allergies

and

problem

and

medication

lists.

This

data

is

used

toidentifyclinicalrelationshipsinpatientrecords,drivedecisionsupportfunctions,eliminateredundant

dataentryandsupplycodeddataelementstoadministrativeandreportingfunctions.Meaningfuluse

criteriaforhealthreformwilllikelyincluderequirementsfortheroutinecaptureofcodedclinicaldata.

Thechallengewithdiscreteclinicaldataentryisthepresentationofstructuredterminologyintheuser

interface.Vocabularymustbeefficienttonavigate,presentedintermsfamiliartoclinicalpractice(e.g.,

insteadofbilling)andattheappropriatelevelofgranularity.Interfaceterminologyisacomplexissue

andanactiveareaofMedicalInformaticsresearch.36

Ef fec t ive In format ion Presenta t ion

Appropr ia te Densi ty

Whiledensityofinformationonascreenisnotcommonlymeasured(thoughitcanbe),itisavery

importantconcepttobecognizantofwhendesigningEMRscreens.Inclinicalapplications,therecanbe

somuchrelevantinformationtodisplayitcanbetemptingtopackasmuchaspossibleontoascreen.

However,visualsearchtimesandusererrorsincreaseinproportiontodensity.Itischallengingto

balanceprovidingallthenecessaryinformationandlimitthenumberofscreenchangeswhile

maintaininganappropriatescreendensity.Testingactualuserswillrevealwhenthebalancehasbeen

reached.

Charactercount,resolution,font,fontsizeandgroupingtechniquesimpactvisualdensity.Screen

elementssuchaslines,buttons,controls,scrollbarsandiconsalsocontributetodensity,whichisyet

anotherreasonthatsimplicityissoimportant.

Accordingtoergonomicrecommendationsforinformationpresentationoncomputerscreens,15

an

upperlimitof40percentdensityisappropriateforcharacterbaseddisplays(thepercentageofpotential

characterpositionsactuallyfilledbycharacters).Graphicaluserinterfacesmustbeevenlessduetothe

otherelementscontributingtoperceptionofdensity.

Animportantmeansofreducingdensityisviewingdataatasummarylevelbeforedrillingdownto

detail.Roughly,the80/20ruleappliestosummaryscreens80percentofthetimetheinformationat

thesummary

level

is

sufficient

for

decision

making

and

is

the

most

frequently

needed

information;

20

percentofthetimetheuserwillneedtodelvedeeper.

Meaningful Use of Color

Colorisoneofseveralattributesofvisualcommunication.Itissingledoutherefordiscussiondueto

howpoorlyithasbeenutilizedinmanysystemdesignstodate.Skillfuluseofcolorcertainlycontributes

toauserinterfacethatispleasinginappearance.However,aestheticsshouldbethelastconsideration

8


11/42


forusingcolorinanytaskorientedapplication.Firstandforemost,colorshouldbeusedtoconvey

meaningtotheuser.Thisincludesallaspectsofinformationpresentation,navigation,differentiationof

screenareasandstaterepresentationofcontrols.Everythingintheuserstaskareaofthescreen,

includingnavigationbars,needstoobeyameaningfulcolorscheme.Purelyaestheticuseofcolorshould

belimitedtodesignoficons,logosandbannerareas.

Simplicityandconsistencyarebothkeyprinciplesintheuseofcolor.Forcolortoconveymeaning,there

cannotbealargernumberofcolorsusedthantheusercanremember,andtheymustbeused

consistentlythroughouttheapplication.Forinstance,ifbrightyellowisusedasahighlightercolorto

emphasizethenameofthepatientwhoseordersarecurrentlybeingentered,thenbrightyellowshould

onlybeusedasahighlightcolorforkeyinformation. Inconsistentorgratuitoususeofcolorincreases

thelikelihoodofusererrorduetomisinterpretationoroversightofimportantdetails;themeaningwill

belost.

Toaccommodateuserswithcolorblindness,allmeaningconveyedwithcolormustalsobe

differentiatedwithasecondvisualmechanism(redundantencoding)suchasfontcharacteristicsorfill

pattern.Forexample,ifredisusedtodisplaycriticallabvaluesthenthecharactersshouldalsobe

bolded,increasedinsizeorsomeothercharacteristic.Itishighlyrecommendedthatdisplaysbe

designedingrayscalepriortoaddingcolortoensurethatallmeaningisrepresented.Ifnot,theinability

todifferentiatecolorsalsomayleadtousererrorsthathavepatientsafetyconsequences.

Naturalnessisaccomplishedbyadheringtoculturalconventionsofcolormeaning.IntheUnitedStates,

thefollowingcolorinterpretationsarecommonlyunderstood.Comprehensiveguidelinesonuseofcolor

haverecentlybeendevelopedbyHFES15;seealsoHHS49andAccessibilityForum.1

Red: Stop, Hot, Danger, Error, Extreme Warning, Severe Alert, Emergency, Alarm

Yellow: Caution, Potential or Mild Warning, Requires Attention, Slow, Moderate Alert

Green: Go, Safe, Normal, Good, Proceed

Blue: Cold, Advisory

Readabi l i ty

Screenreadabilityalsoisakeyfactorinobjectivesofefficiencyandsafety.Clinicalusersmustbeableto

scaninformationquicklywithhighcomprehension.Thepaceandfrequentinterruptionsinclinical

workflowguaranteethatdecisionswillsometimesbemadebaseduponcursoryscreenreview.

Simplicity,naturalness,languageuse,densityandcolorallcontributetoreadability.Inaddition,

guidelinesrecommendusingafontsizeofnosmallerthan12pointforimportantcontentandnever

smallerthan

9point

as

defaults.

Differences

in

visual

acuity

make

it

necessary

to

allow

users

to

modify

textsizeasneeded.Systemsettingsforcolor,fontsandfontsizeshouldalwaysberespected.1,26,49,15San

seriffontscanbereadmoreeasilyincomputerdisplaysthanseriffonts.Highcontrastbetweentextand

backgroundisalsoimportant;blackonwhiteisthemostreadable.

9


12/42


Preservat ion of Contex t

Preservationofcontextisaveryimportantaspectofdesigningatransparentapplication.Inpractical

terms,thismeanskeepingscreenchangesandvisualinterruptionstoaminimumduringcompletionofa

particulartask.Visualinterruptionsincludeanythingthatforcestheusertoshiftvisualfocusawayfrom

theareaonthescreenwheretheyarecurrentlyreadingand/orworkingtoaddresssomethingelse,and

thenreestablishfocusafterward.Themostfrequentviolatoristhedialogbox,whichalsotendsto

obscureasignificantpartofthescreen.Dialogboxesshouldbekepttoaminimum.Forinstance,whena

dialogormessageboxistriggered,itshouldappearincontext(adjacenttoorjustbelowthecontrolthat

triggeredit).Thislimitsvisualsearchingandmakesitfeellikeitisanaturalpartofthecurrenttask.Allof

theseboxesshouldalsobeassmallaspossiblewithoutcompromisingtheirusability.

Anotherimportantguidelineassociatedwithpreservationofcontextisthatofdirectness.Inpart,thisis

acomponentofthewhatyouseeiswhatyougetphilosophyifyouchangesomethingonthescreen,

youshouldseethechangeimmediatelyandintheformatexpected.Anaspectofdirectnessthat

sometimesfallsthroughthecracksistoavoidmodes.Indataentry,thissometimesoccursintheform

ofviewing

vs.

entry

modes;

these

should

not

be

separate.30

If

auser

is

viewing

information

on

a

formthattheyhavepermissiontoedit,theyshouldbeabletodoso,incontext.Thisdoesnotmeanthat

informationcollectedviaaparticularform(e.g.,allergies)shouldntbedisplayedelsewhereinthe

systemasviewonly.However,anydatapresentedthatispotentiallyusereditable,shouldhavea

mechanismfortakingtheuserdirectlytotheappropriateentryformifupdatingisdesired.

SeeANSI/HFES200:HumanFactorsEngineeringofSoftwareUserInterfaces15,WindowsUserExperience

InteractionGuidelines26andResearchBasedWebDesign&UsabilityGuidelines49forcurrent,

comprehensiveguidelinesondesigningforoptimalusability.Thislastworkisuniqueinthatitsinclusion

criteriaforguidelineswasresearchbasedevidenceratherthanexpertopinion(paralleltothe

evidencebased

medicine

concept).

While

targeted

at

Web

site

design,

most

of

these

guidelines

apply

equallywelltoWebbasedanddesktopapplications.Eachguidelineisaccompaniedbyresearch

referencesandtheequivalentof5starscoresfora)relativeimportance,andb)strengthofthe

evidence.

HavingreviewedtheessentialprinciplesofEMRusability,wenowmovetoadiscussionofevaluation

methods.

USABILITY EVALUATION AND RATING METHODS

ThepeoplewhoselectanduseEMRsoftwarearemakingacommitmentthatcannoteasilybereversed.

Costsofimplementationaretypicallyhigh,andthecostsofabandoninganimplementationorswitching

toanotherproductarevastlyhigher.Reliableusabilityratingschemesofferproductpurchasersatool

forcomparingproductsbeforepurchaseorimplementation.Thesemethodscanfostercompetitionand

innovationbymakingexcellentusabilityvisibletotheentirecommunityofpurchasersandusers.47

10


13/42


Organizationsevaluateandcommunicateusabilityinformationfora

numberofreasons.Commercialorganizationsevaluateusabilityaspart

ofproductimprovement,andasameansofdifferentiatingtheirproducts

fromcompetitors.Thegovernmentevaluatesusabilitytoassuresafety

standardsarenotcompromised.IndependentgroupssuchasConsumer

Reportsprovide

evaluation

data

to

help

consumers

make

informed

purchases.Thissectionwilldescribetwosuchprogramsthatprovide

potentialEMRratingorganizationswithperspectivesonevaluating

usabilityinproductsthatmustmaintainhighsafetystandards.

1. Nat iona l Highw ay Tra f f ic Safe ty Admin is t ra t ion Chi ld Safe ty Seat Usabi l i ty

Rat ing Program

TheNationalHighwayTrafficSafetyAdministration(NHTSA)EaseofUseratingprogramwasdesignedto

createmarket

forces

that

encourage

child

seat

manufacturers

to

include

user

friendly

features,

labeling,

andinstructionmanualsandtoprovideconsumersadditionaldataastheymakechildsafetyseat

purchasedecisions.TheinitialmandateforsucharatingprogramwaspartoftheTransportationRecall

Enhancement,Accountability,andDocumentation(TREAD)Actin2000.NHTSAhadtherequirements

thattheprogrammustberepeatableandmustbeobjective.41NHTSAsEaseofUseratingprogramevolvedoverseveralyears.Thechosenapproachusestrained

evaluatorstoassignratingstofeaturesconsideredtoimpacttheusabilityofinstallingchildsafetyseats.

Weightedaveragesforeachcategoryandoverallusabilityarecommunicatedtothepublicthroughthe

useofa5starratingsystem.41

Severalkey

aspects

of

the

NHTSA

rating

program

may

provide

insight

to

EMR

certification

organizations.

First,ittooktwoyearsfromthemandatetothefirstimplementationofNHTSAsratingprogram.

Second,NHTSAcollecteddatatoevaluatetheeffectivenessoftheirprogramfromtheinitiationofthe

program.Third,NHTSArecognizesthattheEaseofUseratingprogramneedstobeflexibleandchange

asthemarketchanges.In2008NHTSAupdatedtheirEaseofUseratingprogramto:

Makeimprovementswheretheinitialprogramwasweak.

Makeupdateswheretheinitialprogramhadbecomeobsolete.

Accommodatenewsafetyseatfeaturesthatdidnotexistwhentheratingprogramwas

initiated.

Addapre

evaluation

program

to

encourage

continued

innovation.

2. FDA and Human Fact ors Regulat ion and Guidel ines for Device

Manufacturers

TheFoodandDrugAdministration(FDA)requiresdevicemanufacturerstofollowHumanFactors

regulationsandprovidesguidancetoensuresafeuseofmedicaldevices.In1997theFDApresentedthe

InBrief:Objectivemethodsare

availabletoevaluateEMR

usability.Wefocusparticularlyon

efficiency,effectivenessAND

SAFETY,butalsodiscussuser

satisfactionand

cognitive

workload. A5starratingsystem

canhelpinformEMRpurchasers

aboutanEMRsusability.

11


14/42


finalruleforGoodManufacturingPractice.Aspartofthisqualitycontrolmandate,medicaldevice

manufacturesarerequiredtodemonstrateadherencetogooddesignandmanufacturingprocesses.The

objectiveoftheHumanFactorsaspectoftheregulationisaimedatminimizingusererrorsthatcould

causepatientinjuryordeath.Theregulationcallsfordesigninput,designverificationanddesign

validation.17

TheessenceoftheregulationisthatHumanFactorsactivitiesaretobeconductedthroughoutthe

designanddevelopmentofamedicaldevice.Designinputcallsformanufacturerstoestablishand

maintainproceduresthatensuredesignrequirementsareappropriateandaddresstheintendeduseof

thedevice,userneedsandpatientneeds.Designverificationrequiresmanufacturestoestablishand

maintainproceduresforverifyingthedesigninput.HumanFactorsactivitiesmayincludetaskanalysis,

functionalanalyses,userstudies,prototypetestsandmockupreviews.Designvalidationrequiresthat

thedeviceconformstothedefineduserandpatientneeds,andassuressafeuseinbothintendedand

unintendedusesofthedevice.Ariskanalysisaimedatminimizingusererrorthatcanleadtopatient

injuryordeathmustbeincludedaspartofdesignvalidation.16

KeyaspectsoftheFDAsregulationandguidanceusefultoanEMRcertificationprocessincludethe

following:First,theFDAprovidesguidance,placingtheresponsibilityonthemanufacturertobe

educatedinHumanFactorsandtoselectappropriatemethodstomeettheFDAsregulations.Second,

theFDAendorsesandrequiresmanufacturestoadheretostandardsdevelopedbyotherstandards

organizationsinadditiontotheirown.Third,theFDAholdsmanufacturersaccountabletoHuman

Factorsregulationsthroughfieldinspections,premarketreviews,andpostmarketsurveillance.Ineach

situationtheFDAinstructsmanufacturerstoprovideevidenceforappropriateHumanFactorsanalyses

andtestsfortheproductunderreview.17

Evaluat ion Methods and Met r ics

Dependingonthereasonfortesting(e.g.,earlydesignvs.differentiationbetweeninterfaces)

measurementmethodsdiffer.Usabilityexpertsapproachproductevaluationasaprocess.Assuch,

therearespecificgoalsforeachphaseoftheprocessandtherearespecificactivitiesappropriateto

addressphasespecificgoals.42Usabilityevaluationmethodsareoftendescribedasbeingprimarily

formativeorsummativeinnature.Formativeevaluationisusedtoinformandimprovetheproduct

designduringthedevelopmentprocess.Summativeusabilitytestingisavalidationexercisetoevaluate

aproductattheendofthedevelopmentprocess.

Usabilityistheresultofcarefuldesignandevaluationthroughoutproductdevelopment.Duringthe

designanddevelopmentprocess,formativeusabilityactivitiesarecarriedoutinsupportofdefiningthe

application,understanding

the

user

and

user

workflow,

and

making

iterative

improvements

to

the

product.Thedatagatheredduringtheseactivitiestendtobemorequalitativeanddescriptive.The

findingsfromformativeusabilityactivitiesaremeanttodescribeanddefineusersanduserneedsand

productfeatures,aswellashaveanimpactonthedesignoftheproductsuserinterface.

12


15/42


Thisphaseisclearlytheresponsibilityofthesoftwarevendor.Inaddition,thesemethodsshouldbe

appliedintheconfigurationofhighlyadaptablesystemsduringimplementation;configurationcan

involveahighdegreeofscreendesignandworkflowengineering.Formativeusabilityactivitiesinclude

butarenotlimitedto:

ContextualInquiry

FocusGroups

StakeholderMeetings

AffinityDiagramming

TaskAnalysis

RiskAssessment

ExpertReview

Oneononeusabilitytesting

Laterinthedevelopmentprocess,summativeusabilityactivitiesarecarriedouttorefinetheproduct.

Theyalso

may

be

done

after

product

completion

to

validate

the

usability

of

the

product,

or

compare

it

withcompetitorproducts.Recommendedusabilityratingactivitiesclearlyfallafterproductcompletion

andshouldbesummativeinnature.Summativeusabilityactivitiesincludebutarenotlimitedto:

ExpertReview

PerformanceTesting

RiskAssessment

Oneononeusabilitytesting

Summativeusabilityactivitieseachhavespecificgoalswhichtheyappropriatelyaddress.Thedata

gatheredduringtheseactivitiestendtobemorequantitativeandobjective.However,somesummative

researchactivities

are

subjective.

Expert

reviews

as

ameans

to

validate

usability

introduce

subjective

expertinput.Thefindingsfromsummativeactivitiesaremeanttovalidateandconfirmusability.Ifa

vendorhasemployedaniterativeusercentereddesignprocessthroughtheproductdevelopment

process,thereshouldbefewsurprisesthatariseinsummativeusabilitytesting.

Animportanttradeofftoconsiderinanyusabilityevaluationisthetestingenvironment.Assoonasthe

usabilityevaluationismovedoutoftheactualenvironmentandintoatestenvironment,muchofthe

complexitycausedbytheenvironmentisremoved.Assuch,asystemthatisratedhighinusabilityina

testenvironmentmaynotbeeasytouseinthecontextoftheactualenvironmentrifewithinterruptions

andchangingworkpriorities.Usabilitytestingbestpracticessubmitsthatitisalwaysbesttoconduct

formativeusability

testing

in

the

environment

that

is

closest

to

the

users

actual

environment.

Software

vendorsshouldensurethattheirdesignersanddevelopershavetheopportunitytoexperiencetheirend

productinuseinaclinicalsetting.Thisprocesscouldhelpmaketremendousstridestowardminimizing

disconnectbetweenwhattheuserneedsanapplicationtodoandwhatitactuallydoesorhowit

doesit.

13


16/42


Usabilityspecialistshavestarteddevelopingautomatedmethodsforusabilitytesting.Automated

usabilitytesttoolstypicallyevaluateuserinterfacesagainstdesignrequirements.TheWebsite

http://usability.govprovidesanautomatedusabilitytooltofederalagencies(AutomatedUsability).In

addition,commercialproductsareavailableforautomatedusabilitytestingofsoftwareapplicationsand

Webbasedsolutions.

Evaluat ing Eff ic ienc y

Efficiency,asatestmetric,isthespeedwithwhichausercansuccessfullyaccomplishthetaskathand.

Researchactivitiesaimedatevaluatingefficiencyincludesexpertreviewandefficiencystudies.Expert

ReviewisaHumanFactorsexpertreviewoftheproduct.Aspartofthereview,theusabilityspecialist

identifiesareasintheproductwheretheproductconformsorfailstoconformtoHumanFactorsbest

practices.Thereissomeamountofsubjectivejudgmentinvolvedinanexpertreview.25

Thereareanumberofvariantsononeoneoneusabilitytestsaimedatevaluatingefficiency.Atypical

efficiencystudycallsforanexpert,intermediateornoviceusertocompletespecifickeytaskswiththe

application.

Performance

data

is

collected.

Sessions

are

frequently

recorded

with

special

software

that

capturesinteractionswiththegraphicaluserinterfacesandmatchestheinteractionswithtimestamps.

Theresultsareusedtoevaluatetheefficiencyoftheproduct.25

Themostcommonmeasuresofefficiency:

Timetoperformaparticulartask.

Numberofkeypressesorinteractionstoachievetask.

Numberofscreensvisitedtocompleteaspecificworkflowscenario.

NumberofBackbuttonuses.

Timetoexecuteaparticularsetofinstructions.

Evaluat ing Effect iveness

Effectivenessistheaccuracyandcompletenesswithwhichausercanachievetaskgoals.Riskanalysisis

acollectionoftechniquesforidentifyingthemostlikelyhumanerrorpointsinasystem.A

comprehensiveriskanalysiswillidentify,quantifyandmitigateriskswithiterativeassessmentand

implementationthroughoutproductdevelopment.Thesetechniqueshavebeenusedformanyyearsin

numerousindustriessuchasthespaceprogram,shippingandnuclearenergy.Earlyonitwaslearned

thathumanfailuresweremuchmoredifficulttopredictthanmechanicalorelectroniccomponents.24

TheJointCommission,VeteransAdministration,theFDA,andtheDepartmentofDefensehavespenta

greatdeal

of

time

and

effort

developing

variants

of

Failure

Modes

and

Effects

Analysis

(FMEA)

to

identifyandanalyzerisksinhospitalandotherhealthcareprocesses,medicaldevicedevelopmentand

othercomplexsystems.FMEAisoneofthemostwidelyusedformsofbottomupriskanalysisandis

theprevalentformofriskanalysisintheautomotiveandaviationindustries.43AnFMEAforanalyzing

humanerrorisasystematicprocessexaminingtheusersworkflowforpointswhereerrorcouldoccur.

14


17/42


Theprobabilityandseverityoflikelyerrorsareevaluated,andappropriatemitigationforeachpotential

errorisidentified.

InadditiontoFMEA,whichhasawelldevelopedformalism,amoregenericformofriskanalysisisa

topologicalriskanalysis.Thetopologicalriskanalysisisaviablefirstanalysissteptoexaminethe

processin

enough

depth

to

define

its

layout

(or

topology).

A

topological

analysis

can

help

identify

a

numberofriskelementssuchassinglepointfailuresandcommonmodefailures.

Asinglepointfailurewouldbeanyactionbytheclinicianthatresultsinharm,injuryordeathtothe

patientwithoutaredundantsafetycheckinplace.Anexamplewouldbeprescribingadrugthepatient

isallergictobecausedrugallergieswerenotdisplayedontheorderingscreen,anddecisionsupportfor

drugallergycheckingwasnotyetimplemented.Acommonmodefailureiswhenmultipleactionsbythe

clinicianstemfromasinglecausethatresultsinharm,injury,ordeathtothepatient.Forexample,a

prescribingerrorcausedbyforcingtheprovidertoaccessmultiplescreensandholddetailsinmemory

tocompletetheprescribingprocess.Asrisksareidentified,detailedanalysisandmitigationefforts

shouldconcentrateatthesepoints.24

Ausabilityratingprocesscanbedevelopedbyadaptingriskassessmentmethodologiestoobjectively

evaluatethepotentialforusererror.Certaindesignfactorscanleadtousererrorwhichwouldhave

patientsafetyimplications.AssessinganEMRuserinterfaceforthepresence/absenceofthesedesign

factorsprovidesanimportantmeansofevaluatingEMReffectiveness.Examplesofusererrorswith

patientsafetyimplicationsareprovidedinAppendixC.Indepthworkonexamininghowuserinterface

designchoicescancompromisepatientsafetyisbeingconductedbytheePrescribing&

CommonUserInterfaceprogramsoftheNationalHealthService(NHS)intheU.K.Thisgrouphas

designedasafetyfocusedusabilityevaluationmethodbasedonerrortrapssimilartothepatient

safetychecklistconceptproposedinthisdocument.11TheNHSalsoisintheprocessofdeveloping

specificguidelines

for

safe

on

screen

display

of

medication

information.

Effectivenessstudiesareaclassofoneononeusabilityteststhatinvolvecollectingmeasuresof

effectivenesswhenuserscompletespecifickeytaskswiththeapplication.

Commonmeasuresofeffectivenessincludebutarenotlimitedto:

Numberorrateoferrors

Pathtakentocompletetask

Severityoferrors

Requestsforhelp

Evaluat ing Ease of Learning

Improvingusabilityhasbeenshowntoimproveeaseoflearningorlearnability.37Themoreausercan

applypriorexperiencetoanewsystemandthegreatertheinternalconsistency(useofconsistence

concepts,behaviors,layout,etc.)thelowerthelearningcurve.Whenasystemisforgivingofmistakes

15


18/42


andallowsdiscoverythroughexploration,itfostersfasterlearningbyreducingtheusersfearof

unintendedconsequences.Errors,pathtakentocompletetasksandrequestsforhelpeachcorrelate

withhowwellauserknowsthesystem.

Easeoflearningcanbeevaluatedintermsofthetimeittakestheusertoreachaspecifiedlevelof

proficiencyand

in

terms

of

the

time

it

takes

auser

who

has

never

seen

the

system

interface

to

successfullyaccomplishbasictasks.Itisimportanttoconsiderlearningthroughoutthelifetimeofuseof

aproduct.33

Possiblemeasuresofeaseoflearningincludebutarenotlimitedto:

Timetoachieveexpertperformance.

Numberoficonsrememberedaftertaskcompletion.

Timespentusingmanual.

Timetoperformaparticulartaskafteraspecifiedperiodoftimeawayfromtheproduct.

Timetoperformtaskcomparedtoanexpert.

Numberof

times

the

Help

function

is

accessed.

Foramorecompletereviewofmetrics,methodologies,andguidelinesregardingusabilityandeaseof

learningseeGrossman,etal.14

Evaluat ing Cognit ive Load

Manymethodsformeasuringcognitiveloadinvolvecomplextestingthatrequiretheskillsofcognitive

psychologistsorexperiencedHumanFactorsengineers.However,thereareafewwelldevelopedand

validatedinstrumentsthatareadministeredassimplequestionnaires.TwoexamplesaretheNASATask

LoadIndex(NASATLX)andtheSubjectiveWorkloadAssessmentTechnique(SWAT).Itispossibleto

simplifythesemethodssuchthattheymaybeadministeredfairlyeasily.10Cognitiveloadissuchan

importantissueforcliniciansthatthisshouldbeconsideredforinclusioninanEMRusabilityrating

program.

Evaluat ing User Sat is fact ion

Thedefinitionofusabilitytypicallyincludesreferencetousersatisfaction.Usersatisfactionisapersons

subjectiveresponsetotheirinteractionwithasystem.Whenevaluatingusability,satisfactioncanbe

addressedinseveralways.AcommonapproachusesLikertscalequestionnairesaskinguserstorate

theirsatisfactionwithvariousaspectsoftheproduct(e.g.,onascaleofoneto10).Typicallythisisdone

immediatelyafterhandsonusabilitytaskperformanceandattheendofausabilitytestsession.Whatis

weakaboutthisapproachisthatthemethodhasnotbeendevelopedunderscientificscrutiny.Other

researchersusemorescientificratingtoolssuchastheSystemUsabilityScale.7,48,5,44

Thesescalesare

strongerbecausethetoolsareaccompaniedbymeasuresofreliabilityandvalidity.Researchsuggests

thatusersatisfactiondoesnotcorrelatewellwithothermorerigorouslyobtainedmeasuresofusability

suchaseffectivenessandefficiency.4,12

16


19/42


Weagreethatusersatisfactionisonecomponentofusability.However,becauseofthesubjective

natureofevaluatingusersatisfaction,wewillnotproviderecommendationsconcerningmeasuringuser

satisfactionaspartofausabilityratingprogram.WedohighlyrecommendthatEMRpurchasers

performstructuredusersatisfactiontestingaspartoftheirEMRselectionprocess.Aggregationofuser

satisfactiondatafromcurrentEMRenduserswouldalsobeofvalueifcollectedusingappropriate

instrumentsand

methodology.

Select i ng Tasks for Evaluat ion

Testtasksshouldbeselectedbasedonhowthedataaregoingtobeused.Whenevaluatingefficiency,

tasksfocusedonuserandsystemefficiencyshouldbeconsidered:thatis,tasksthatwillbefrequently

performedbyusersandtasksthatareknowntobeinefficient.Whenevaluatingeffectiveness,tasks

focusedonusereffectivenessshouldbeconsidered:thatis,tasksthataredeemeddifficulttoperform

andtasksthatareknowntobeatriskforusererrorbasedonpriorevaluations.Userobservations,

workflowanalysisandtaskanalysisaremethodsusedtoidentifyfrequentanderrorpronetasks.

Surveysandinterviewsaremethodsusedtosolicitinformationfromusers;however,userbehaviors

shouldalways

be

directly

observed

when

possible

because

users

are

not

always

accurate

in

describing

whattheyactuallydo.

OneofthechallengesofevaluatingusabilityinEMRsisthecomplexityofusertasks,workflowsandthe

userenvironment.Considerthetasksimplystatedasrefillamedication.Theactualclinicalworkflow

includesacombinationofelements.Specifically,theproviderneedstoconsiderthefollowing:

1. Pastdatapoints(e.g.,medicationhistory,lastvisitdate,relevantlabvalues,lastclinicnote)

2. Futuredatapoints(e.g.,nextlaborvisitdate).

3. Medicalevidencepersonalizedforthepatient(e.g.,whatisthegoalcholesterolforthispatient,

howoftendolabsneedtobecheckedonthismedication).

4. Contextualrelevanceof#13:Whereisthepatientinthelifecycleofthismedication(e.g.,did

theyjuststartitorhavetheybeenonitforfiveyears,reasonforrefill).

5. Taskofformallycreating/approvingtherefill.

6. Considerationsofcostandformularycoverage,andpossiblealternativeproductswithbetter

formularycoverage.

7. Communicatingwiththeirassistantorthepharmacy.

Forthepurposesofusabilityrating,werecommendselectingtesttasksthatencompassentire

workflows(scenarios).EMRsarecomplexsystemsandusabilityofcomplexsystemsmustincludethe

interactionsbetweentheinformationandusersubtasksthatmakeuptheactualwork.

Oncetasksareselected,successfulcompletioncriteria(suchaserrorstates,deviationsandothers)

mustbedefinedforeachtask.ExamplesoftesttasksandtestscenariosareprovidedinAppendixA.

17


20/42


5-Star Usabi l i t y Rat ing System

Thefivestarratingsystemisreadilyrecognized,sinceitisacommonschemeusedwithconsumer

productsinmanycommercialWebsites.Developmentworkisneededtodefineausabilityrating

system(e.g.,5star=excellent,4star=good,etc.)thatcanbeusedtocommunicatetheresultsofa

usabilityratingprogramtoEMRpurchasers.

Considerationmustbegiventodefinitionofthescaleandassigningmetricintervalcutoffs.Asan

example,shouldtheratingscaleconsistofequalintervals(e.g.,5staris059seconds;4staris6069

seconds,etc.)orshouldtheratingscaleconsistofintervalsbasedonthenormaldistribution(e.g.,3star

isthemiddle50thpercentile).

Anotherconsiderationisthegranularityofthereportingsystem.Oneoptionistocombinethescores

fromthetasksineachcategory(efficiencyandeffectiveness)andreportoneglobalstarrating.Asecond

optionistoreportseparatestarratingsforeachcategoryofmeasure.Initiallythismayonlymean

separateratingsforefficiencyandeffectiveness;ratingsforfactorssuchascognitiveloadandeaseof

learningshould

also

be

considered.

Themostimportantaspectofdevelopingastarratingsystemisdefiningthebenchmarkmetricsforeach

measure.Forefficiencymeasures,goaltasktimesshouldbebasedontheneedsofthecliniciansin

actualclinicalpractice.Expectationsshouldbebothhighandattainable.AppendixBincludesbenchmark

examples.

Cert i f ic at ion and EMR Usabi l i t y Rat i ng

Duringoursurveyoftheliterature,welearnedthattheCertificationCommissionforHealthcare

InformationTechnology(CCHIT)wasactivelyconsideringaddingusabilityasacriteriontoitsEMR

certificationprocess.Subsequently,wecontactedthemtounderstandtheiroverallprogram

requirements,andtobetterunderstandtheneedsofcertificationorganizations.CCHITisaprivate

nonprofitorganizationwiththesolepublicmissionofacceleratingtheadoptionofrobust,interoperable

healthinformationtechnologybycreatingacredible,efficientcertificationprocess.Theydescribedthe

followingcharacteristicstheyfounddesirableinthedevelopmentofausabilityprogram:

Theprogrammustbeobjective.

Theprogrammustberepeatable.

Theprogrammustbecostefficienttoimplement.

Theprogramshouldfocusonevaluatingefficiencyandpatientsafety.

Theprogramshouldevaluateproductsthatarereadyformarket.

Agood

approach

would

be

to

rate

usability

on

ascale

similar

to

star

programs

seen

in

consumer

products.

Usabilityratingshouldbeanadjuncttoproductcertificationwithoutaffectingcertification

outcome.

Theusabilityratingsystemadoptedshouldnotbeapass/failmodel

18


21/42


RECOMMENDATIONS

Asthecertifyingorganizationimplementsitsusabilityratingprogram,

basedonourresearchandpriorknowledgeofusabilityprinciplesand

practiceweproposethefollowingapproach:

Star t smal l .

Beginusabilityratingwithafocusonsimpleefficiencyandeffectiveness

measures,includingsomeinitialpatientsafetycheckpoints.Dontletit

becomeamultiyearprocesstosetupaninitialprogram.Learnfromand

adaptHumanFactorsprocessesestablishedbytheFDA,NHTSAandother

organizations.

Develop measurement s.

Deviseobjectivemeasuresofefficiencythattakesintoaccounttimeontaskandnumberofuser

interactions.Develop

objective

measures

of

effectiveness

that

takes

into

account

system

characteristics

thatimpactpatientsafety.Werecommendinitiallytestingeffectivenessusingapredefinedchecklistof

systeminteractionsthathavepatientsafetyimplicationsifnothandledwellbythesystem.AppendixC

illustratespatientsafetychecklistexamples;alsoseeFoneandLewis.11

Create a 5-s tar ra t ing system.

Definetheusabilityratingsystemusing5starsbasedonanabsolutestandardagainstbenchmarks.An

absolutestandardallowseachproducttostandonitsownanddemonstrateprogressovertimerather

thanincomparisontootherproducts.Thisrequiresthatbenchmarksbeestablishedinadvancedefining

targetscoresforeachmeasurement.Benchmarksshouldreflectuserneedsinactualclinicalpractice.

SeeAppendix

B

for

sample

benchmarks.

Initially,reportstarratingsonafewmeasures.Insubsequentyears,additionalmeasurescanbeadded.

Def ine the process.

Selectasetoftesttasksforevaluatingefficiencythata)arefrequentlyperformedbyproviders;b)areat

riskofbeinginefficient;andc)allowevaluationoftasksandworkflow.Beginwithsimple(butcarefully

planned)scenariobasedusertestingsimilartothediscountusabilityengineeringmethodsdescribed

byNielsen.30Recall,however,thattheprocessesshouldbeengineeredtobesummativeinnature.

Forefficiencyevaluationwerecommendamultistepapproach:

1)Thevendordoesawalkthroughexplanationtotheratingorganizationsselectedintermediateor

expertclinicalusers.

2)Thevendorperformsthetesttasks;tasktimesarerecordedasmeasuresofexpertperformance.

nBrief:Weofferspecific

recommendationsforactiontorating

organizations:

Startsmall.

Developmeasurements.

Definetheprocess.

Createa5starratingsystem.

Improvewithtime.

Encourageotherstodotheirpart.

19


22/42


3)Theratingorganizationsselectedusersperformtesttasksforintermediateusermeasurement.As

muchaspossible,thegroupofusertestparticipantsshouldrepresentamixtureofroletypes(e.g.,

physician,nurse,medicalassistant,physicaltherapist)performingtasksappropriatetotheirrole.The

ratingorganizationsselectedusersshouldbeexperiencedintwoormoreEMRs,butnottheonebeing

tested.Thenumberofuserparticipantsnecessarytoproducemeaningfulresultswillneedtobe

evaluated.In

general,

summative

testing

requires

more

participants

than

formative

testing.

Foreffectivenessevaluation,anevaluatorwillneedtodeterminethepresenceorabsenceofpatient

safetyitemsfromthechecklistdeveloped.SeeAppendixCforexampleeffectivenesstestsforpatient

safety.

Improve w i th t ime .

Usabilityratingprogramsshouldevolveinsophisticationoveramultiyearperiod.Eventuallythey

shouldincludemeasuresofcognitiveloadandeaseoflearning.Infutureyears,consideralsotesting

naveusers.Evaluatetheeffectivenessoftheprogramitselfonanannualbasis.Updatescenarios,tasks,

methods

and

measures

to

reflect

any

needed

improvements

as

well

as

evolution

in

the

EMR

marketplaceandusabilitybestpractices.

Itmaybehelpfultotheconsumertobreakthestarsystemintocategoriesasitbecomesmorecomplex,

e.g.,aproductmayscore4starsforefficiency,3starsforeffectivenessand3starsforeaseoflearning.

Duetoitssubjectivenature,werecommendthatusersatisfactionbeleftforpotentialcustomersand

thirdpartiestoevaluate.

Encourage o thers t o do the i r par t .

Encouragevendorstoutilizeiterativedesignwithformativeuserbasedresearchthroughoutthedesign

anddevelopmentprocesswithsummativeusabilityevaluationbeforelaunch.Atthesametimeeducate

clinicaldecision

makers

to

assess

EMR

usability

as

part

of

their

EMR

purchase

and

system

configuration

processes.

REFERENCES

1.AccessibilityForum.QuickReferenceGuidetoSection508ResourceDocuments.September15,2003.

AccessedJune8,2009.Availableat:www.accessibilityforum.org/paper_tool.html.

2.AmericanMedicalAssociation.AMAspecialtycodes.(n.d.).AccessedApril23,2009.Availableat:

www.ehealthlist.com/ama_specabbrev.asp.

3.Usability.gov.AutomatedUsabilityTestEnvironment(UTE)ToolFactSheet.(n.d.).AccessedJune7,

2009.Availableat:www.usability.gov/refine/UTEfactsheet.html.

4.BaileyB.GettingtheCompletePicturewithUsabilityTesting.March2006.AccessedJune7,2009.

Availableat:www.usability.gov/pubs/030106news.html.

20


23/42


5.BangorA,KortumPT,MillerJA.AnempiricalevaluationoftheSystemUsabilityScale(SUS).IntJHu

CompInteract.2008;24(6):574594.

6.BatesDW,KupermanGJ,WangS,GandhiT,KittlerA,VolkLetal.Tencommandmentsforeffective

clinicaldecisionsupport:makingthepracticeofevidencebasedmedicineareality.JAMIA.

2003;10(6):523530.

7.BrookeJ.SUS:aquickanddirtyusabilityscale.InJordanPW,ThomasP,WeerdmeesterBA,

McClellandAL(Eds.).UsabilityEvaluationinIndustry.London:TaylorandFrancis.1996.

8.CampbellEM,SittigDF,AshJS,GuapponeKP,DykstraRH.Typesofunintendedconsequencesrelated

tocomputerizedproviderorderentry.JAMIA.2006;13(5):547556.

9.EdsallRL,AdlerKG.UsersatisfactionwithEHRs:reportofasurveyof422familyphysicians.FamPrace

Manage.2008;15:2532.

10.EmbreyD,BlackettC,MarsdenP,PeacheyJ.DevelopmentofaHumanCognitiveWorkload

AssessmentTool.July2006.AccessedJune8,2009.Availableat:

www.mcga.gov.uk/c4mca/research_report_546.pdf.

11.FoneJ,LewisK.Safety&usabilitymethodsforevaluatingtheuserinterface.April28,2009.

AccessedJune19,2009.Availableat:www.mit.edu/~juhan/nhs_public/Safetyfocusedusability

evaluation_28Apr2009_KitLewis_for_HC_v2.pdf.

12.FrkjrE,HertzumM,HornbkK.Measuringusability:areeffectiveness,efficiency,andsatisfaction

reallycorrelated?ProceedingsoftheSIGCHIconferenceonHumanFactorsincomputing

systems,(pp.345352).NewYork,NY,USA.2000.

13.GaretsD,DavidM.Electronicmedicalrecordsvs.electronichealthrecords:yes,thereisadifference.

HIMSSAnalytics.January26,2006.AccessedJune17,2009.Availableat:

www.himssanalytics.org/docs/WP_EMR_EHR.pdf.

14.GrossmanT,FitzmauriceG,AttarR.Asurveyofsoftwarelearnability:metrics,methodologiesand

guidelines.Proceedingsofthe27thinternationalConferenceonHumanFactorsinComputing

Systems(Boston,MA,USA,April04 09,2009)(pp.649658).NewYork,NY:CHI'09.ACM.

15.HumanFactorsandErgonomicsSociety.2008.ANSI/HFES200:HumanFactorsEngineeringof

SoftwareUserInterfaces.SantaMonica,CA:HumanFactorsandErgonomicsSociety.

16.HumanFactorsImplicationsoftheNewGMPRuleOverallRequirementsoftheNewQualitySystem

Regulation.2009.AccessedJune6,2009.Availableat:

www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketRequirements/Human

Factors/ucm119215.htm.

21


24/42


17.InformationforManufacturersandDistributors.2009.AccessedJune7,2009.Availableat:

www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/PostmarketRequirements/Human

Factors/ucm119190.htm#1.

18.InstituteofMedicine.PatientSafety:Achievinganewstandardforcare.2004.(AspdenP,Corrigan

JM,Wolcott

J,

Erickson

SM,

Eds.)

Washington,

D.C.:

The

National

Academies

Press.

19.JhaAK,DesRochesCM,CampbellEG,DonelalnK,RaoSR,FerrisTG,etal.Useofelectronichealth

recordsinU.S.hospitals.NEnglJMed.2009;360:16281638.

20.KhajoueiR,JaspersMWCPOEsystemdesignaspectsandtheirqualitativeeffectonusability.Studies

inHealthTechnologyInformatics.2008;136:30914.

21.KrallMA,SittigDF.(2002).Clinician'sassessmentsofoutpatientelectronicmedicalrecordalertand

reminderusabilityandusefulnessrequirements.AMIA2002SymposiumProceedings,(pp.400

404).

22.KrugS.(2006).Don'tMakeMeThink!(2nded.).Berkeley,CA:NewRiders.

23.KushnirukA,BoryckiE,AndersonJG,AndersonMM.(2008).Combiningtwoformsofsimulationto

predictthepotentialimpactofinterfacedesignontechnologyinducederrorinhealthcare.

Proceedingsofthe2008Springsimulationmulticonference(pp.497504).SanDiego,CA:The

SocietyforComputerSimulation,International.

24.MaddoxME,BarnesHJ.(2004).RiskyBusiness:AnalysisToolsforMedicalDevicesandProcesses.

Workshoppresentedatthe48thMeetingoftheHumanFactorsandErgonomicsSociety.New

Orleans,LA.

25.MayhewDJ.KeystrokeLevelModelingasaCostJustificationTool.InBiasRG,MayhewDJ(Eds.),

CostJustifyingUsability,AnUpdatefortheInternetAge.SanFrancisco,CA:MorganKaufmann

Publishers.2005.

26.MicrosoftCorporation.WindowsUserExperienceInteractionGuidelines.November6,2008.

AccessedJune8,2009.Availableat:msdn.microsoft.com/enus/library/aa511258.aspx.

27.NationalInstituteofStandardsandTechnology.CommonIndustrySpecificationforUsability

Requirements(NISTIR7432).June2007.AccessedJune7,2009.Availableat:

zing.ncsl.nist.gov/iusr/documents/CISURIR7432.pdf.

28.NationalResearchCouncil.(2009).ComputationalTechnologyforEffectiveHealthCare:Immediate

stepsandstrategicdirections.(SteadWW,LinHS,Eds.)Washington,D.C.:TheNational

AcademiesPress.

29.NielsenJ.DesigningWebUsability:ThePracticeofSimplicity.Indianapolis:NewRidersPublishing.

1999.

22


25/42


30.NielsenJ..UsabilityEngineering.ChestnutHill,MA:AcademicPress.1993.

31.PoissantL,PereiraJ,TamblynR,KawasumiY.Theimpactofelectronichealthrecordsontime

efficiencyofphysiciansandnurses:asystematicreview.JAMIA.2005;12:505516.

32.PrinciplesforUsableDesign.(n.d.).AccessedJune8,2009.Availableat:

www.usabilitybok.org/design/p287.

33.QuesenberyW.(2001).WhatDoesUsabilityMean:LookingBeyondEaseofUse'.Proceedingsofthe

48thAnnualConference,SocietyforTechnicalCommunication.Availableat:

www.wqusability.com/articles/morethaneaseofuse.html.

35.RoseAF,SchnipperJL,ParkER,PoonEG,LiQ,MiddletonB.Usingqualitativestudiestoimprovethe

usabilityofanEMR.JBioInform.2005;38:5160.

36.RosenbloomST,MillerRA,JohnsonKB,ElkinPL,BrownSH.Amodelforevaluatinginterface

terminologies.JAMIA,2008;15:6576.

37.SaleemJJ,PattersonES,MilitelloL,AschSM,DoebbelingBN,RenderML.(2007).UsingHuman

FactorsMethodstoDesignaNewInterfaceforanElectronicMedicalRecord.AMIA2007

SymposiumProceedings,(pp.640644).

38.SandersMS,McCormickEJ.HumanFactorsinEngineeringandDesign.NewYork,NY:McGrawHill.

1993.

39.SchoeffelR.Theconceptofproductusability.ISOBulletin.2003;34:67.

40.SmelcerJB,MillerJacobsH,KantrovichL.Usabilityofelectronicmedicalrecords.JUsabilityStudies.

2009;4:7084.

41.SmithCS,RockwellTE,CollinsLA,KimSD,ParkBT.(2009).NHTSASchildsafetyseatusabilityrating

program.AccessedJune8,2009.Availableat:

www.lifesaversconference.org/handouts2009/Smith2.pdf.

42.StepbyStepUsabilityGuide.(n.d.).AccessedJune7,2009.Availableat:www.usability.gov/.

43.StephansRA.SystemSafetyforthe21stCentury:TheUpdatedandRevisedEditionofSystemSafety

2000.Hoboken,NJ:JohnWiley&Sons,Inc.2004.

44.System

Usability

Scale.

(2009).

Accessed

June

8,

2009.

Available

at:

http://en.wikipedia.org/wiki/System_Usability_Scale_(SUS).

45.TheJointCommission.Safelyimplementinghealthinformationandconvergingtechnologies.

December11.2008.Availableat:

www.jointcommission.org/SentinelEvents/SentinelEventAlert/sea_42.htm.

23


26/42


46.TognazziniB.(n.d.).FirstPrinciplesofInteractionDesign.AccessedJune6,2009.Availableat:

www.asktog.com/basics/firstPrinciples.html.

47.Tullis,T.S.,&Albert,W.MeasuringtheUserExperience:Collecting,Analyzing,andPresenting

UsabilityMetrics.Burlington,MA:MorganKaufmann.2008.

48.TullisTS,StetsonJN.(2004).AComparisonofQuestionnairesforAssessingWebsiteUsability.

UsabilityProfessionalAssociationConferenceProceedings.

49.U.S.DepartmentofHealthandHumanServices.(2006).ResearchBasedWebDesign&Usability

Guidelines.AccessedJune8,2009.Availableat: www.usability.gov/pdfs/guidelines.html.

50.WalkerJ.Usability.InWalkerJM,BieberEJ,RichardsF.(Eds.),ImplementinganElectronicHealth

RecordSystem(pp.4759).Springer.2005.

51.ZhengK,PadmanR,JohnsonMP,DiamondHS.Aninterfacedrivenanalysisofuserinteractionswith

anelectronichealthrecordssystem.JAMIA.2009;16:228237.

CONTRIBUTORS

Authors

JefferyL.Belden,MDChair,HIMSSEHRUsabilityTaskforce

RebeccaGrayson

JaneyBarnes,PhD

Usabi l i t y Pr inc ip les Workgr oup

PatriciaAlafaireet,MHA

JaneyBarnes,PhD

JefferyL.Belden,MDChair,HIMSSEHRUsabilityTaskforce

EdnaBoone,MASS,CPHIMSHIMSSSeniorStaffLiaison

JonDuke,MD

RebeccaGrayson

AndrewHutson

JasminPhua

24


27/42


TianaThomasLeadershipCouncil,andCochairEHRUsabilityPrinciplesWorkgroup

JuhanSonin

PennWhite,MD LeadershipCouncilandCochairEHRUsabilityPrinciplesWorkgroup

Editors

MelanieBrodnik,PhD,RHIAViceChair,HIMSSEHRUsabilityTaskforce

HelenVolger,MSHA,CPHIMS

InAcknowledgement:

WewouldliketothankthemanyHIMSSEHRUsabilityTaskForcemembersandotherswhoprovided

theirtimeandexpertiseforreviewandcommentonthefinaldrafts.Appreciationisalsoextendedto

theTIGER

Initiative

for

permitting

us

use

of

the

compendium

of

material

organized

by

their

Usability

and

ClinicalApplicationDesignWorkgroup.Thisdocumentisthesynthesisofnumerousexperienced

professionalsintheHealthcareITandHumanFactorscommunities.

HIMSS EHR Usabi l i ty Task forc e

PatriciaAlafaireet,MHA

CeciliaBackman,MBA,RHIA,CPHQ LeadershipCouncil

JaneyBarnes,

PhD

JefferyL.Belden,MD Chair

LyleBerkowitz,MD,FHIMSS

JudiBinderman,MD,MBA

EdnaBoone,MASS,CPHIMSHIMSSSeniorStaffLiaison

MelanieBrodnik,PhD,RHIA ViceChair,HIMSSEHRUsabilityTaskforce

JonDuke,

MD

RobertDuthe,MBA LeadershipCouncil

RebeccaGrayson

DoronGutkind

25


28/42


ShannonHouser,PhD,MPH,RHIA

AndrewHutson

RonRibitzky,MD LeadershipCouncil

JuhanSonin

MaryAnneSterling LeadershipCouncilandCochairEHRUsabilityUserPainPointsWorkgroup

SandraStork

CarolynSwanson LeadershipCouncilandCochairEHRUsabilityUserPainPointsWorkgroup

TianaThomas LeadershipCouncilandCochairEHRUsabilityPrinciplesWorkgroup

JuanitaThreat HIMSSCoordinator

RoyceUehara

MichaelVanOrnum,RPh,RN,BCPSViceChair,HIMSSEHRUsabilityTaskforce

AlbertVillari,MD LeadershipCouncilandChairEHRUsabilityVendorBestPracticesWorkgroup

HelenVolger,MSHA,CPHIMS

PennWhite,MD LeadershipCouncilandCochairEHRUsabilityPrinciplesWorkgroup

26


29/42


APPENDICES

A.Test task and scenar io examples

Toplevel:

Clinical

Scenarios

These

are

entire

workflows

consisting

of

anumber

of

associated

componenttesttasks.Thesescenariosaresufficientlycomplextorepresentaclinicianworkflow

worthyoftesting.

Nextlevel:TestTasksThesearecomponenttasksthatoccurfrequentlyinclinicalsettings,orare

tasksthatareatriskforusererror.Individually,theywouldbetoosimpletoconstituteatest

workflow.

Thissetofexamplesisnotmeanttobeexhaustive,buttoserveasastartingpointfortypesof

scenariosandtasksthatmightbeapartofusabilitytesting.

27


30/42


TableA1.ClinicalScenarios

Theseareentireworkflowsconsistingofanumberofassociatedcomponenttesttasks.Thesescenariosaresuffi

clinicianworkflowworthyoftesting.

D

No. ScenarioName ScenarioDescription Features/Rationale

Simplicity

Naturalness

Consistency

A B C

1

AcuteUTI

with

heydocrashNew

patient

presents

with

3days

of

dysuria,hematuria,urgencyand

frequency.Nofever,chills,orback

pain.Lateron,sheremembersthat

shehasanitchyrashbetweenthe

toesofherleftfoot.Doctororders

BactrimDS1tabbidx3days,

phenazopyridine200mgtidafter

mealsandrecommendsOTC

terbinafinecreamtoapplybidx10

days.

Ohby

the

way

complaint

notincludedintheinitial

reasonforvisit.

Demonstrateshowprogram

handlesmulticomplaint

visitsandeaseofchartinga

lastminuteaddition.

+

28


31/42


2 Chroniccomplex

diabeticwithLDL

elevation

ChronicpatientwithHTN,Obesity,

Type2DiabetesandelevatedLDL

comesinforarecheckofhisweight

anddiabetes.DoctorwantsfastingBS

(inoffice),Lipidpanel&HbA1c(sent

out),VSincludingweight,diabetic

footexam,andinterveninghistory

beforeseeingpatient.Atendofvisit,

doctorincreasesglipizidefrom5mg

bidto10mgbid.

Careforchronicdisease.How

doesprogramhandle

instructionstostaffbefore

seeingthepatient?How

easilyareoutsidelaborders

handled?Howefficientlyare

medicationorderschanged?

Doessystemofferdecision

supportfortargetLDL,

aspirintherapyindications,

remindersforperiodic

testingandimmunizations?

+

3 Medicationrefill

request

Respondtoamedicationrefill

request.Checkmedicationhistory,

patientproblemlist,drugprescribing

informationandlabtrackingstudies

recommendedforthismedication.

Doesdesignofdisplay

providenecessary

informationinaterse,

aggregatedfashion?

+ +

4

Depressioninitial

visit

Established53

yr

old

male

with

3

monthsofdepressionsymptoms.Not

suicidalorpsychotic.Orderlabtests

tolookformedicalcausesof

depression.Initiatetreatmentwith

SSRI.Printoutapatienteducation

handoutforthepatient.

OrderTSH,

CMP

or

BMP.

Efficientaccesstopatient

educationmaterials,and

wayfindingtotheproper

handout.LinkingtheEHR

diagnosistothePatient

Educationresourcewould

savetime.

+ + +

29


32/42


5 Labresultletter&

orders

Sendaletter(oremail)tothepatient

reportingonherabnormalthyroid

testresult,orderthyroidmedicine,

andschedulerepeattestingforsix

weeksfromnow.

Effectiveuseoflanguage

appropriateforthepatient

receivingthecommunication.

Pagelayoutintheletterthat

makescommunication

effective.Clinician

efficiency.

Simplifieddatadisplayfor

clinician.Nonintrusive

decisionsupportforselecting

theproperthyroidtest.How

isreceiptofthenotification

tothepatientverified?What

happensifthepatientmisses

herretest?

+

30


33/42


TableA2.TestTasks.

Thesearecomponenttasksthatoccurfrequentlyinclinicalsettings,oraretasksthatareatriskforusererror.In

simpletoconstituteatestworkflow.

No.

Task

Name

Task

Description

Features

/

Rationale

Simplicity

Naturalness

Consistency

A B C1 FindLDL FindthepatientslatestLDLresult. Dontmaketheclinician

calculatetheLDLresult.

2 CountCADrisk

factors

Howmanycoronaryarterydisease

riskfactorsdoesthepatienthave?

Doesthesystemaggregate

riskfactordataandpresentit

conciselyandappropriately

forthetaskathand?Reduce

cognitiveload.Simpledata

presentation.

+

31


34/42


3 CADriskscore Whatisthepatientsriskofhavinga

coronarydiseaserelatedeventinthe

next10years?

Doesthesystemaggregate

riskfactordataandpresentit

conciselyandappropriately

forthetaskathand?

PresentingFraminghamrisk

scorecan

guide

clinician

in

makingdecisionsaboutlipid

reductiontherapy.

+ +

4 Changedefault

pharmacy

Howdoyouchangeapatients

pharmacyofchoice?Whathappensif

thenewpharmacyisnotapprovedby

thepatientsinsuranceplan?

ForgivenessandFeedbackin

eventoferrorindataentry.

Appropriatesystemdefaults. + + +

5 Druginteraction

alert

&

response

Prescribingnewdrugbringsupdrug

interactionwarning.

Physician

reviewswarning,completes

prescriptionorder,andmakeschange

indefaultsettingforDIseveritylevel

threshold.

Avoidsalertfatigue.Patient

safetyis

at

stake.

Is

informationterseand

actionable?Isseverity

thresholdeasilyadjustable?

+ + +

32


35/42


TableA3.PotentialTasksthatNeedAdditionalWorktobeconsideredaTestTask.

Thesearetoovaguetobecomponenttasks,orhavecomponentsthathavenoclinicalconsensusastoappropria

No. TaskName TaskDescription Rationalefornoninclusion

Simplicity

Naturalness

Consistency

A B C

1 Neworders. Placenewordersonacomplex

patient

Thistaskneedsmoredetailin

ordertodefinethetaskasa

testtask.+ +

2 Manage

hypertension

Reviewyourplanformanagingthe

patientshypertension.

Thistaskistoovague.In

ordertobeatesttask,the

taskwouldneedtohave

moredetail.

3 Whatkindof

penicillinallergy?

Whatisthenatureofthepatients

penicillinallergy?

Thistaskistoosimpleas

statedandneedscontext

regardingthe

need

for

the

informationandhowthe

informationisgoingtobe

usedinaclinicaldecision.

+ + +

33


36/42


4 Orthopedicconsult

order

Orderanorthopedicsconsult,with

appropriateprevisittesting.

Thistaskneedsmoredetail

asthereisnotastandard

approachforordering

consults.Shouldtheclinician

ordertheMRIoftheknee,or

letthe

orthopedic

surgeon

decideifitisneeded?

+ +

34


37/42


B.Benchmark examp les

Forusabilityratingstooffermorethancomparativeinformation,usabilitybenchmarkmetricsneedto

beestablished.Benchmarkmetricscanbedevelopedbymeasuringclinicianusersinactualclinical

environmentsperformingeachtaskorscenario.Clinicianuserpanelswouldthencomparethebest

actualperformancesagainstusersperceivedidealperformance,inordertodevelopatargetscorethat

betterreflectsactualuserneeds,asopposedtothecurrentstateoftheartEMRperformance.Some

targetcriteriawouldbemorestraightforward,aseitherpresentorabsentfeatures.

SeeNationalInstituteofStandardsandTechnology(2007)foradetaileddescriptionforbenchmarking

usabilitycriteria.

Table1presentsexamplesoftargetcriteriaformeasuringeffectivenesstiedtopatientsafety.

Table 1. Target Cri t er ia for Evaluat ing Pat i ent Safety.

Pat ient Safe ty Check l is t Effect iveness: Pass/Fai l of Pat ient

Safe ty I t em

MedicationlistdisplayedinTallmanlettering Pass/Fail

Patient'sdrugallergiesdisplayedonmedication

orderingscreen

Pass/Fail

Table2presentsexamplesoftargetcriteriaformeasuringtheefficiencyofanEMR.Inthisexample

efficiencyisdefinedastheaveragetimefortestparticipantstocompleteeachspecifictaskorscenario.

Usertimeandsystemresponsetime(e.g.,downloadtimes)shouldbeincludedinthetasktime.

Table 2 . Target Cr i te r ia for Measur ing Ef f ic iency.

Task or Scenar io Eff ic ienc y: Maxim um

accep tab le task t ime

Scenario1.AcuteUTIwithheydocrash minutes

Scenario2.ChroniccomplexdiabeticwithLDLelevation minutes

35


38/42


C. Pat ient sa fe ty check l is t examples

Thislistprovidessamplesofthetypesofdesignfactorsthatcouldleadtousererrorswhichwouldhave

patientsafetyimplications.TheyareintendedtobestraightforwardPass/Failtests.Theresulting

effectivenessmeasurecouldsimplybehowmanyofthechosenteststheproductpassed(e.g."8outof

10").Opportunitiesforcheckingsomeofthesemayneedtobebuiltintotestingscenarios,butmostcan

bescenarioindependent.

These"tests"focusonpreventionofthefollowingusererrorsorpractices:

1. Selectionofthewrongpatientorpatientencounter.

2. Selectionofthewrongmedicationordosage.

3. Steppingawayfromaterminalwithoutloggingoutorsuspendingthesession.

4. Steppinguptoaterminalandtakingactionwithinsomeoneelse'sactivesession.

5. Overlookingorbeingunawareofcriticalpatientinformation.

Pat ient Se lect ion and Ident i f ic a t ion

Patient'sfullname,uniqueID,age(orDOB)andgenderareprominentlydisplayedonallchart

screens.

Patient'sfullname,uniqueID,DOBandgenderistheminimumsetofidentifiersdisplayedwhen

selectingapatienttoaccesstheirchart.

CPOE/CDS/ePrescribing

Patient'sdrugallergiesaredisplayedonthemedicationorderingscreen.

Patient'scurrentmedicationsaredisplayedonasinglescreen.

Viewingofthepatientscurrentmedicationlistisatmostoneclickawayfromthemedication

orderingscreen.

Actionstorenew,discontinueorcancelaredonedirectlyfromthecurre