drchrono safety-enhanced design usability report · 2019-01-01 · of evaluating the ehr’s...

drchronoSafety-EnhancedDesignUsabilityReport1

drchronoSafety-EnhancedDesignUsabilityReport

Report based on NISTIR 7742 Customized Common Industry Format Template forElectronicHealthRecordUsabilityTesting.

drchronoEHR11.0 DateofUsabilityTesting: October31,2017–Nov21,2017DateofReport: May16,2018ReportPrepareBy: drchrono,Inc. 328GibraltarDr Sunnyvale,CA94089


Table of Contents ExecutiveSummary..................................................................................................................................4Majorfindings........................................................................................................................................6Areasforimprovement......................................................................................................................6

Introduction.................................................................................................................................................7UserCenteredDesignProcess..............................................................................................................7UnderstandingUsers..........................................................................................................................7UserFeedback........................................................................................................................................8Efficiencyofworkflow.......................................................................................................................8IterativeDesign.....................................................................................................................................8UserExperience....................................................................................................................................9Multidisciplinarydevelopmentteam...........................................................................................9

Methods..........................................................................................................................................................9Participants.............................................................................................................................................9StudyDesign........................................................................................................................................12Tasks.......................................................................................................................................................13Procedures...........................................................................................................................................14TestLocation.......................................................................................................................................15TestEnvironment..............................................................................................................................15TestFormsandTools......................................................................................................................15ParticipantInstructions..................................................................................................................16UsabilityMetrics................................................................................................................................17

DataScoring..............................................................................................................................................17Results..........................................................................................................................................................19DataAnalysisandReporting........................................................................................................19DiscussionofFindings.....................................................................................................................25Effectiveness........................................................................................................................................25Efficiency...............................................................................................................................................26Satisfaction...........................................................................................................................................26MajorFindings....................................................................................................................................26

APPENDICES.............................................................................................................................................27Appendix1:drchronoEHRUsabilityStudyScreener......................................................28


Appendix2:ParticipantDemographics...................................................................................28Appendix3:Moderator’sGuide.................................................................................................30Appendix4:SystemUsabilityScaleQuestionnaire............................................................36


ExecutiveSummary

The drchrono product team conducted a usability test of the drchrono EHR (the“EHR”) from October 31, 2017 – Nov 21, 2017. The drchrono product teamconducted this test over several testing session.Theproduct teamconducted thistestinordertovalidateanddocumenttheusabilityofthecurrentuserinterfaceinthe EHR. Ten (10) healthcare providersmatching the target demographic criteriaparticipatedinthetestandusedtheEHRinsimulated,butrepresentativetasks.

Thisstudycollectedperformancedataon12taskstypicallyconductedonanEHR:

1-(a)(1)ComputerizedProviderOrderEntry(CPOE)–medications2-(a)(2)CPOE–LaboratoryOrder3-(a)(3)CPOE–DiagnosticImaging4-(a)(4)Drug-drug,Drug-allergyInteractionChecksforCPOE5-(a)(5)Demographics6-(a)(6)ProblemList7-(a)(7)MedicationList8-(a)(8)MedicationAllergyList9-(a)(9)ClinicalDecisionSupport10-(a)(14)ImplantableDeviceList11-(b)(2)ClinicalInformationReconciliationandIncorporation12-(b)(3)ElectronicPrescribing

Duringthe60-minuteusabilitytest,theadministratorgreetedeachparticipantandasked the participant to review his or her signed informed consent/release form.The administrator notified each participant that he or she couldwithdraw at anytime.ParticipantshadvaryingamountsofpriorexperiencewiththeEHR.

The administrator introduced the test, and instructed participants to complete aseriesoftasks(givenoneatatime)usingtheEHR.Theadministratortimedthetestand,alongwithoneormoredataloggers,recordeduserperformancedatainbothpaper and electronic formats. The administrator did not assist any participant inhow to complete the task, but every participant had access to a wide-range oftrainingmaterials available to all EHR users. Thismaterial included, butwas notlimited to, information available to users during registration, videos, customersupportandvaryinglevelsofimplementationtraining.

Thefollowingtypesofdatawerecollectedforeachparticipant:

• Number of tasks successfully completed within the allotted time withoutassistance

• Timetocompletethetasks


• Numberandtypesoferrors• Pathdeviations• Participant’sverbalizations• Participant’ssatisfactionratingsofthesystem

Thereportanonymizesallparticipantdatasuchthattheidentityoftheparticipantcannot be discerned from the data collected and contained in this report. Afterconcluding the test,participantswereasked tocompleteapost-testquestionnaireandwerecompensatedwithnogreaterthan$100fortheirtime.

Various recommended metrics, in accordance with the examples set forth in theNISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronicHealthRecords, were used to evaluate the usability of the EHR. A summary of theperformanceandratingdatacollectedontheEHRfollows.

Tasks TaskSuccess

PathDeviation

TaskTime Errors TaskRating1=Easy,5=

Hard Observed/

OptimalMean(SD) Deviations

(Observed/Optimal)

CPOE:MedicationOrder

Mean:100%SD:0

7.2/7 Mean:52.4SD:15.1

52.4/60 Mean:0SD:0

Mean:1.1SD:0.32

CPOE:LaboratoryOrder

Mean:100%SD:0

8.3/8 Mean:64SD:15.6

64/80 Mean:0SD:0

Mean:1.4SD:0.52

CPOE:ImagingOrder

Mean:100%SD:0

6/6 Mean:41.2SD:9.4

41.2/60 Mean:0SD:0

Mean:1.1SD:0.3

Drug-Drug,Drug-AllergyInt.

Mean:100%SD:0

13.3/13 Mean:66.7SD:18.4

66.7/80 Mean:0SD:0

Mean:1.3SD:0.48

Demographics Mean:100%SD:0

2.4/2 Mean:15.8SD:6.7

15.8/20 Mean:0SD:0

Mean:1.2SD:0.42

ProblemList Mean:100%SD:0

1/1 Mean:6.9SD:2.69

6.9/10 Mean:0SD:0

Mean:1SD:0

MedicationList Mean:100%SD:0

1/1 Mean:4.2SD:1.4

4.2/10 Mean:0SD:0

Mean:1SD:0

MedicationAllergyList

Mean:100%SD:0

1/1 Mean:3.8SD:1.13

3.8/10 Mean:0SD:0

Mean:1SD:0

ClinicalDecisionSupport

Mean:100%SD:0

9.1/8 Mean:108.5SD:35.3

108.5/120 Mean:0SD:0

Mean:2.3SD:0.82

ImplantableDeviceList

Mean:100%SD:0

5.1/5 Mean:33.6SD:22.0

33.6/40 Mean:0SD:0

Mean:1.2SD:0.42

ClinicalInformationReconciliation

Mean:100%SD:0

6.5/6 Mean:77.4SD:29.5

77.4/100 Mean:0SD:0

Mean:1.4SD:0.52

ElectronicPrescribing

Mean:100%SD:0

7.2/7 Mean:60.5SD:25.0

60.5/80 Mean:0SD:0

Mean:1.3SD:0.63


UsingtheSystemUsabilityScale,thesubjectivesatisfactionscorewascalculatedas:91.2%.

In addition to the performance data, the following qualitative observations weremade:

Majorfindings

Finding#1:Sinceourfirstusabilitytestin2014,we’vefoundthatprovidersareasfluent with the EHR as they are with their staff. Initial findings suggest thatproviderswerereliantontheirstaffmembersforadministrativetasks,butournewusability test shows that doctors have taken advantage of the system’s improvedusabilityovertheyearsandarestartingtoknowtheins-and-outsofthesystem.

Finding #2: We’ve found that users within the age range of 23-59 showconsiderableknowledgewhenusingthewebandcomputertechnology.BecauseofthephasingoutofphysicalrecordsandhavingprovidersmovingintoEHRs,they’vequicklyadaptedandtheadditionalexperiencehasclosedthegapbetweenyoungerandolderusers.

Finding#3:Someparticipantswhohadnotpreviouslyusedafeaturethatwasontheusability test,hadaharder timecompleting the task.OnesuchexamplewasauserwhodidnothadpriorexperiencesendinganElectronicPrescriptionspendingamuch longer timeon that taskcompared tootherusers.AnotherwasaddinganImplantableDevice,whichisnewtothisusabilitytest.

Areasforimprovement

Duringtheusabilitytesting,we’venoticedthatmanyusersgetconfusedwhiledoingacoupleof the tasks.Theywouldmakeanentryand immediatelyclickon theOkbutton (that was disabled) without realizing there was another step. We canimprovethisworkflowbyaddinganalertorasetofprominentinstructionstotrainthe users. A number of users commented that the position of the buttons isinconsistent, causing them to be confused on where to click. Additionally, wenoticedthat input isnotconsistentwithin thepatientdashboardareaandwewillneed to take steps to improve the overall usability and consistent of this section.Overall,therearealotofimprovementsthatneedtobemadeinthesystemandthedrchrono team is committed tomaking improvements whilemaking each screenconsistentwiththerestofthesystem.


Introduction

Theversionof theEHR tested for this studywasdrchronoEHR11.0 (the “EHR”).The EHR is designed to present medical information to healthcare providers inambulatoryphysicianpractices.Theusability testingrepresents realisticexercisesandconditionsconductedbyprovidersintheirday-to-daypractices.

ThedrchronoproductteamconductedthisstudytotestandvalidatetheusabilityoftheEHR’suserinterface.ThisstudyestablishestheusabilityoftheEHRforpurposesof evaluating the EHR’s compliance with the 2015 ONC requirements forCertification. To this end, the drchrono product team captured measures ofeffectiveness,efficiencyandusersatisfaction,suchas(e.g.,timeontask),duringtheusabilitytesting.

UserCenteredDesignProcess

drchrono’sprimaryusabilitygoal isanEHR that is simpleandeasy touse.As thefirstEHRalsoavailableontheiPad,thedrchronoproductteamfocusesondesigningtheEHRtopresentcomplextasksinawaythatthatisbothintuitivefortheusertounderstand while also allowing users to perform even complex tasks efficiently.Both drchrono’s web and iPad interfaces reflect this emphasis on “intuitive tounderstand,”“keepingsimplethingssimple”and“makingcomplextasksefficient.”

The iPad interface follows the Human Interface Guidelines created by Apple fordeveloperscreatingmobileapps.ThewebinterfacefollowsthedesignguidelinesintheISO9241-210standardinadditiontomobile-centricdesignprinciples.

drchrono’sdesignprocessisbasedonthefollowing:

UnderstandingUsers

Every user is different, and every user has a different level of comfort andexperiencewithtechnology.Thus,theEHRmustbeusableby,andrelatableto,anyindividual without regard to age or gender. In additional, we try to design thesystemtobeusedbydifferenttypesofspecialties.InordertocontinuallyimprovetheusabilityoftheEHR,thedrchronoproductteamregularlymeetswithadiverserangeofproviderstoprovideassessmentsoftheEHR.Wealsoattendtradeshowsto lookathowotherEHRshandledifferentspecialtiesanditeratively improveoursystem over time. On occasion, we would also visit our users at their office andobservetheirworkflowandseehowtheyuseoursystem.


UserFeedback

UsersmayalsoprovidedrchronofeedbackandideasonhowtoimprovetheEHRviaanonline forum.The forumallowsusers topresent their own ideas aswell as tovoteonideaspresentedbyotherusers.Theideaswiththemostvotesarereviewedandreceivehighdevelopmentpriority.Inadditiontouserfeedbackviatheform,thedrchrono product team regularlymeetswith a diverse spectrum of physicians toprovide more detailed feedback. We also have a beta testing team made up ofproviders from different specialties. We sent out beta version of our applicationoftenandtheywouldgiveusfeedbackonchangesandimprovements.Wetakethisfeedbackandtryto integrateasmuchaswecanwithoutmakingthesystemmoredifficult to use. In addition, we regularly invite providers to visit our office onoccasionandhavethemprovideanassessmentofthesystem.

Efficiencyofworkflow

The drchrono product team is constantly trying to find ways to help the userimprovehisorherworkflow.Theseeffortshavehistoricallyproducedsoftwarethatdrchronousersseeasawaytoenhanceandsupporttheirexistingworkflowsratherthanburden it.We try to design the system formaximumcustomizability so thattheycancreateaworkflow that fits theirneedswithinourEHR.Forexample, theEHRincorporatescustomizabletemplatesystemsformedicalformssothatdoctorscan tailor their forms to the particular way they see clients. Users also have theabilitytocreatemacrosandlistsofcommonlyusestermstoimprovethespeedofdataentrytoparticularspecialtiesortoprovidersthathavepracticesthatfocusonparticular classes of patients. Similarly, the EHR incorporates specialized speechrecognitionsoftwaretoallowproviderstodictatetheirclinicalnoteswhilehavingthedictationtranslateaccurately.This focusonefficiencyofworkflowisbasedonone of drchrono’s core beliefs: if an EHR can reduce the amount of time doctorsspend entering in data documenting a visit, it is allowing the doctor to providebettercaretohisorherpatients.

IterativeDesign

Thedrchronoproductteamunderstandsthatitsdevelopmentanddesignteamswillnotbuildperfectfeaturesonthefirsttrybecauseeveryfeaturecanbeoptimizedinterms of both speed and ease of use. In order to support iterative design anddevelopment of features, the product team designed architected the EHR with a“verticaldesignphilosophy”—i.e.,insuchawaythattheEHRcaneasilybeadaptedto incorporatenewfeaturesandnewfunctionality.The iPad interfaceandAPIareupdated with design and feature improvements on a quarterly basis. The webinterfaceisupdatedatleastonceaweekwithnewfeaturesandbugfixes.Becausedrchrono users constantly provide feedback, the product team can focus onaddressingactualuserconcerns.


UserExperience

“Simplicity” is one of the key tenets of drchrono’s design philosophy. The EHRshould make accomplishing tasks as easy as possible using as few actions aspossible.UsershouldbeabletogetfrompointAtopointBwithintheEHRwithasfewactionsaspossible.This isespecially importantwhenaprovider, forexample,mustswitchbetweenpatientsorswitchbetweencomplextasksforapatient(suchasrecordingfamilyhistorytodocumentingclinicalnotesorviceversa).

Multidisciplinarydevelopmentteam

TheEHRshouldnotbegoodatallowinguserstoaccomplishingor finishonlyonetask.Thus,ourproductteamhasexpertiseinavarietyofdesignfieldsrangingfromvisualdesign,user interfacedesign,graphicdesign, touserexperiencedesign.Thedrchrono product team continuously experiments on new ideas and new designtechniques and look at design trends like Material Design and Human InterfaceDesign. The EHR does (and should continue to) set new standards for designingsimplicityandease-of-use.

Methods

The sections below discuss the methodology and results of the testing that wasconductedbythedrchronoproductteam.

Participants

A total of 10 participants participated in usability test. Participantswere selectedfrom drchrono’s existing user base and included medical professionals from adiversespectrumofpractices.drchronoreachedouttoanumberofexistinguserstorequest that they participate in the study. Participantswere compensatedwith a$100gift card for their time. Inaddition,participantshadnodirect connection tothedevelopmentofororganizationproducingtheEHR.drchronodidnotsolicititstestingorsupportorganizations forparticipants in thisstudy.Asexistingusersofthe EHR, participants had access to all of the materials and orientation madeavailabletoallusersoftheEHR.

For the testpurposes, drchrono identified end-user characteristics that translatedinto a recruitment screenerused to solicit potential participants; an example of ascreenerisprovidedintheAppendix.

Recruited participants came from a mix of backgrounds and demographiccharacteristics conforming to the recruitment screener.The following isa tableofparticipants by characteristics, including demographics, professional experience,computing experience and user needs for assistive technology. Participant names


werereplacedwithParticipantIDssothatthepresenteddatacannotbetiedbacktoindividualidentities.

ID M/F Age Specialty Occupation/Role YearsinHealthCare

ProfessionalExperience

P1 F 30-39 Gastroenterology MedicalAssistant 11-20 162Months

P2 M 30-39 Endocrinology MedicalAdmin 11-20 82Months

P3 F 50-59 Cardiology MedicalAdmin 21-39 417Months

P4 F 20-29 Dermatology MedicalAdmin 11-20 182Months

P5 M 50-59 OB/GYN MedicalAdmin 6-10 97Months

P6 M 30-39 Ophthalmology Physician 11-20 237MonthsP7 M 30-39 Gastroenterology Physician 11-20 154Months

P8 F 50-59 FamilyMedicine/UrgentCare

Physician 21-39 295Months

P9 M 30-39 InternalMedicine/Pediatrics


P10 M 40-49 InterventionalPainManagement


ID Education ComputerExperience

AssistiveTechnology

TimeusingEHR

P1 AssociateDegree 240Months No 5Years,2Months

P2 Bachelor’sDegree 240+Months No 6Years,4Months

P3 Trade/technical/vocationaltraining 321Months No 5Years,6Months

P4 Master'sdegree 303Months No 5Years,6Months

P5 Master'sdegree 448Months No 5Years

P6 Doctoratedegree 297Months No 5Years,2Months

P7 Doctoratedegree 154+Months No 5YearsP8 Doctoratedegree 194Months No 3YearsP9 Doctoratedegree 240Months No 5Years,

2MonthsP10 Doctoratedegree 240+Months No 3Years


Ten(10)participantswererecruitedandallparticipatedintheusabilitytest.

Participants participated in hour-long sessions. In between sessions, theadministrator and data logger would debrief participants and reset the testingsystemstopropertestconditions.Theadministratoranddataloggermaintainedaspreadsheet that included each participant’s demographic characteristics to tracktheparticipantschedule.


StudyDesign

Overall, the objective of this test was to uncover areas where the applicationperformed well—that is, effectively, efficiently, and with satisfaction—and areaswhere the application failed tomeet the needs of the participants. The data fromthistestmayserveasabaselineforfuturetestswithanupdatedversionofthesameEHRand/orcomparisonwithotherEHRsprovidedthesametasksareused.Inshort,this testing serves asboth ameans to recordorbenchmark currentusability, butalsotoidentifyareaswhereimprovementsmustbemade.

Duringtheusabilitytest,participantsinteractedwithoneEHRandtheywereaskedtousethewebversionoftheEHRonanyoperationsystemandanywebbrowseroftheir choice with the exception of Internet Explorer. The administrator and datalogger monitored participant use of the system using Zoom (zoom.us) meeting,which allowed the administrator and data logger to monitor exactly how theparticipant was interacting with the EHR. The administrator and data loggerfurnishedeachparticipantwiththesameinstructions.Thesystemwasevaluatedforeffectiveness, efficiency and satisfaction as defined by measures collected andanalyzedforeachparticipant:

• Number of tasks successfully completed within the allotted time withoutassistance

• Timetocompletethetasks• Numberandtypesoferrors• Pathdeviations• Participant’sverbalizations(comments)• Participant’ssatisfactionratingsofthesystem

Additional information about the various measures can be found in the sectionUsabilityMetrics.


Tasks

The testing of a number of tasks were constructed that would be realistic andrepresentativeofthekindsofactivitiesausermightdowiththisEHR,including:

1. ComputerizedProviderOrderEntry(CPOE)—MedicationOrdersa. AddMedicationOrderb. ChangeMedicationOrder

2. ComputerizedProviderOrderEntry(CPOE)—LabOrdersa. AddLabOrderb. ChangeLabOrder

3. ComputerizedProviderOrderEntry(CPOE)—DiagnosticImaginga. AddaImagingOrderb. ChangeImagingOrder

4. Drug-Drug,Drug-AllergyInteractionChecka. Drug-DrugInteractionb. Drug-AllergyInteraction

5. Demographicsa. AccessPatientDemographics

6. ProblemLista. AccessPatientProblems

7. MedicationLista. AccessPatientMedications

8. MedicationAllergyLista. AccessPatientMedicationAllergies

9. ClinicalDecisionSupporta. CreateaCDSRuleb. CreateanAppointmenttodisplaythatCDSRule

10. ImplantableDeviceLista. AddanImplantableDeviceusingauniquedeviceID.

11. ClinicalInformationReconciliationa. ReconcilePatientMedicationListb. ReconcilePatientProblemListc. ReconcilePatientMedicationAllergyList

12. ElectronicPrescribinga. Createaprescriptionandpreviewitb. Reviewprescriptionandsendit.

ThetestingteamselectedthesetasksbasedontherequirementsoftheMeaningfulUse2015SEDsection.


Procedures

Uponarrival,participantsweregreeted; the testing teamverified theparticipants’identities and matched a name on the participant schedule. The testing teamassigned each participant a unique participant ID. Each participant reviewed andsigned an informed consent and release form. The test team confirmed theparticipant’ssignaturebeforeeachsession.

Toensurethatthetestransmoothlyandquickly,thetestingadministratorinformedtheparticipantofwhat thenext taskon testwasupon successful completionof atask.Thetestingteamconductingthetestwereexperiencedusabilitypractitionerswith degrees in Communication Design, International Business Communicationsand Biomedical Engineering and has over ten years of experience in design andusabilitytesting.

The testing administrator moderated the session including administeringinstructionsandtasks.Theadministratoralsomonitoredtasktimes,obtainedpost-taskratingdataandtooknotesonparticipantcomments.Asecondpersonservedasthedataloggerandtooknotesontasksuccess,pathdeviations,numberandtypeoferrorsandcomments.

Thetestinggavetheparticipantsthefollowinginstructions:

Tasksshouldbecompleted:

• Asquicklyaspossiblemakingasfewerrorsanddeviationsaspossible;• Withoutassistance1and• Withoutusingathinkaloudtechnique.

For each task, the testing informed the participant of the nature of the task. Theadministratorbegantimingtheparticipant’suseoftheEHRuponfinishingreadingthequestion.Theadministratorstoppedthetimeroncetheparticipantindicatedheorshehadsuccessfullycompletedthetask.

Following the session, the administrator gave the participant the post-testquestionnaire, compensated them for their time and thanked each individual fortheirparticipation.

Participants' demographic information, task success rate, time on task, errors,deviations, verbal responses and post-test questionnaire were recorded into aspreadsheet.

1Thetestingadministratoronlygaveimmaterialguidanceandclarificationontasks,butnotinstructionsonuse.


TestLocation

Each participant performed the test from his or her location of choice while theadministratoranddata loggerwereatdrchrono’soffices.Duringeachsession, themoderator and data loggerwere seated in an empty roomwith two laptops or alaptop accompanied with a tablet and spreadsheets for logging data and takingnotes. Each usability session was conducted using a virtual meeting tool called“Zoom.” Zoom meeting allowed the moderator and data logger to view theparticipant’sscreenandcommunicatewithparticipantsverbally.Theadministratoranddataloggerkeptnoiselevelstoaminimumtoensurethatparticipantswerenotdisturbedorinterruptedwiththeirtesting.

TestEnvironment

User of the EHR typically used it in a healthcare office or facility. Participantsconducted the test from their own computers while the administrator and datalogger monitored the participants’ actions via Zoommeeting. The EHR has web-baseduser interfaceandisthuscompatiblewithanyoperatingsystemsthathasacompatiblewebbrowser.2

ThetestrequirednospecialsetupbecausethedrchronoEHRusesaweb-baseduserinterface. They tested the software using varying computer screen sizes, rangingfrom11 inch to27 inchesataminimumof1366x768resolutionup toa2560x1440resolution.Allparticipantsperformedthetestusingatestaccountsuppliedbydrchrono. The browser platform used for this test was Chrome, Firefox or Safariover a LAN network. The testing team initialized the test account with relevantsample data required for the usability study. Because the EHR is web-based, theparticipants’internetconnectionspeedscanaffecttheEHR’sperformancefromtheend-user’sperspective.Asobservedduringtheusabilitytest, theEHR’sperformedsimilarlytoitsdaily,real-worldoperationintermsofresponsetime.

TestFormsandTools

During the usability test, the testing team utilized various documents and testingtools,thatincludedthefollowing:

1. ParticipantDemographics;2. Moderator’sGuide;3. Post-testquestionnaire;4. Atimerand5. ZoomMeeting

2TheEHRsupportsthefollowingwebbrowsers:Firefox,ChromeandSafari.


Appendices 2–4 contain respective examples of documents (1)–(3) above. AModerator’sGuidewasdevisedsoastobeabletocapturerequireddata.

The participant’s interaction with the EHR was captured and recorded digitallyusingscreencapturesoftwarerunningonthedatalogger’smachine.Thecomputers’built-in microphones recorded any verbal communications between participantsandthetestingteam.

ParticipantInstructions

Thetestingadministratorreadthefollowinginstructionsaloudtoeachparticipant:3

Thank you for participating in this study. Your input isvery important. Our session today will last about 60minutes.DuringthattimeyouwillbeusingthedrchronoEHR. I will ask you to complete a few tasks using thissystemandanswersomequestions.Youshouldcompletethe tasksasquicklyaspossiblemakingas fewerrorsaspossible. Please try to complete the tasks on your ownfollowing the instructions very closely. Please note thatwe are not testing you, we are testing the system,therefore if you have difficulty all this means is thatsomethingneeds to be improved in the system. Iwill behere incaseyouneed specifichelp,but Iamnotable toinstructyouorprovidehelpinhowtousetheapplication.

Overall,weare interested inhoweasy (orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Wewouldlikeyoutobehonestwith your opinions. All of the information that youprovidewillbekeptconfidentialandyournamewillnotbe associated with your comments at any time. Shouldyou feel it necessary, you are able to withdraw at anytimeduringthetesting.

Following the procedural instructions, participants were shown the EHR and astheir first task,were given time to explore the systemandmake comments.Oncethistaskwascomplete,theadministratorgavethefollowinginstructions:

Foreach task, Iwill read thedescription toyouandsay“Begin.” At that point, please perform the task and say“Done”onceyoubelieveyouhavesuccessfullycompleted

3TheappendixcontainsthefulltextoftheModerator’sGuideusedintheusabilitytest.


the task. I will ask you for your impression of the taskwhenyouaredone.

Participants were then given 12 tasks to complete. Tasks are listed in themoderator’sguideintheAppendix.

UsabilityMetrics

AccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronicHealthRecords,EHRsshouldsupportaprocessthatprovidesahighlevelofusabilityforallusers.Thegoalisforuserstointeractwiththesystemeffectively,efficiently, and with an acceptable level of satisfaction. To this end, metrics foreffectiveness, efficiency and user satisfaction were captured during the usabilitytesting.Thegoalsofthetestweretoassess:

1. EffectivenessofEHRbymeasuringparticipantsuccessratesanderrors;2. Efficiency of EHR bymeasuring the average task time and path deviations

and3. SatisfactionwithEHRbymeasuringeaseofuseratings.

DataScoring

The following tabledetailshow taskswere scored, errorsevaluated, and the timedataanalyzed.

Measure RationalandScoringEffectiveness:TaskSuccess

Ataskwascountedasa“Success”iftheparticipantwasabletoachievethecorrectoutcome,withoutassistance,withinthetimeallottedpertasktimelimit.The testing team calculated the total number of successes foreachtaskandthendividedbythetotalnumberoftimesthattaskwas attempted. This report provides these results as apercentage.Thereportalsorecordstasktimesandtaskcompletionwereforsuccesses.Optimal task performance time, as benchmarked by expertperformance under realistic conditions, is recorded whenconstructingtasks.

Effectiveness:TaskFailures

Iftheparticipantabandonedthetask;didnotreachthecorrectanswer or performed it incorrectly or reached the end of theallotted time before successful completion, the testing teamcounted the taskasa “Failure.” The testing teamdidnot track


tasktimesforfailures.Thetestingteamcalculatedthetotalnumberoferrors foreachtaskandthendividedthattotalbythenumberoftimesthattaskwasattempted.Thetestingteamdidnotcountalldeviationsaserrors.

Efficiency:TaskDeviations

The testing team recorded the participant’s path (i.e., steps)throughtheapplication.Theparticipants’actionswerecountedas deviations if, for example, the participant went to a wrongscreen,clickedonanincorrectmenuitem,followedanincorrectlink, or interacted incorrectly with an on-screen control. Thetesting teamcomparedtheactualusepath to theoptimalpath.The testing team then divided the number of steps in theobservedpathbythenumberofoptimalstepstoarriveataratioofpathdeviation.

Efficiency:TaskTime

The testing team timedeach task.The testing teamstarted thetask timerwhen theadministrator said “Begin,” andended thetimer when the participant reported “Done.” If a participantfailedtosay“Done,”thetimewasstoppedwhentheparticipantstoppedperformingthetask.Onlytasktimesfortasksthatweresuccessfully completedwere included in the average task timeanalysis.Thisreportincludesaveragetimepertaskandvariancemeasures(standarddeviationandstandarderror).

Satisfaction:TaskRating

Thetestingteammeasuredparticipants’subjectiveimpressionsoftheeaseofuseoftheEHRbyadministeringasimplepost-taskquestionnaire as and a post-session questionnaire. After eachtask, theparticipantwasasked to rate “Overall, this taskwas:”onascaleof1(VeryDifficult)to5(VeryEasy).This report follows the common convention that an averagescoreof3.3oraboveis“easytouse.”Tomeasureparticipants’confidenceinandlikeabilityoftheEHRoverall,thetestingteamadministeredtheSystemUsabilityScale(SUS) post-test questionnaire. Questions included, “I think Iwouldliketousethissystemfrequently,”“Ithoughtthesystemwaseasytouse,”and“Iwouldimaginethatmostpeoplewouldlearntousethissystemveryquickly.”


Results

DataAnalysisandReporting

TheresultsoftheusabilitytestwerecalculatedaccordingtothemethodsspecifiedintheUsabilityMetricssectionabove.Participantswhofailedtofollowsessionandtaskinstructionshadtheirdataexcludedfromtheanalyses.

Theusability testingresults for theEHRaredetailedbelow.TheresultsshouldbeseeninlightoftheobjectivesandgoalsoutlinedinSection-StudyDesign.Thedatashouldyieldactionableresults that, ifcorrected,yieldmaterial,positive impactonuserperformance.

1. CPOE–MedicationOrder

User Task

SuccessPathDeviation TaskTime

(seconds)Errors TaskRating

1=Easy,5=Hard

Observed Optimal Observed Optimal

1 Yes 7 7 62 60 0 1

2 Yes 8 7 94 60 0 2

3 Yes 7 7 59 60 0 1

4 Yes 7 7 68 60 0 1

5 Yes 7 7 59 60 0 1

6 Yes 7 7 44 60 0 1

7 Yes 7 7 46 60 0 1

8 Yes 7 7 45 60 0 19 Yes 8 7 47 60 0 110 Yes 7 7 55 60 0 1 Rate:100% Mean:7.2 Mean:52.4 Mean:0 Mean:1.1 SD:15.1 SD:0 SD:0.32


2. CPOE–LaboratoryOrder

User TaskSuccess

PathDeviation TaskTime(seconds)

Errors TaskRating1=Easy,5=Hard

Observed Optimal Observed Optimal 1 Yes 8 8 62 80 0 12 Yes 8 8 45 80 0 13 Yes 8 8 91 80 0 24 Yes 8 8 54 80 0 15 Yes 9 8 75 80 0 26 Yes 8 8 63 80 0 17 Yes 8 8 51 80 0 18 Yes 8 8 84 80 0 29 Yes 9 8 47 80 0 210 Yes 8 8 68 80 0 1 Rate:100% Mean:8.2 Mean:64 Mean:0 Mean:1.4 SD:15.6 SD:0 SD:0.52

3. CPOE–DiagnosticImaging

User TaskSuccess




1 Yes 6 6 48 60 0 1

2 Yes 6 6 31 60 0 1

3 Yes 6 6 36 60 0 1

4 Yes 6 6 60 60 0 2

5 Yes 6 6 40 60 0 1

6 Yes 6 6 34 60 0 1

7 Yes 6 6 47 60 0 1

8 Yes 6 6 33 60 0 19 Yes 6 6 49 60 0 110 Yes 6 6 34 60 0 1 Rate:100% Mean:6 Mean:41.2 Mean:0 Mean:1.1 SD:9.4 SD:0 SD:0.3


4. Drug-Drug,Drug-AllergyInteractionCheck

User Task



1=Easy,5=Hard


1 Yes 15 13 82 80 0 2

2 Yes 14 13 78 80 0 2

3 Yes 13 13 107 80 0 2

4 Yes 13 13 56 80 0 1

5 Yes 13 13 46 80 0 1

6 Yes 13 13 68 80 0 1

7 Yes 13 13 64 80 0 1

8 Yes 13 13 64 80 0 19 Yes 13 13 51 80 0 110 Yes 13 13 51 80 0 1 Rate:100% Mean:

13.3 Mean:66.7 80 Mean:0 Mean:1.3

SD:18.4 SD:0 SD:0.48

5. Demographics

User TaskSuccess




1 Yes 2 2 22 20 0 1

2 Yes 2 2 11 20 0 1

3 Yes 3 2 23 20 0 2

4 Yes 2 2 16 20 0 1

5 Yes 3 2 21 20 0 1

6 Yes 2 2 5 20 0 1

7 Yes 3 2 17 20 0 2



6. AccessProblemList

User Task



1=Easy,5=Hard


1 Yes 1 1 4 10 0 1

2 Yes 1 1 7 10 0 1

3 Yes 1 1 6 10 0 1

4 Yes 1 1 11 10 0 1

5 Yes 1 1 9 10 0 1

6 Yes 1 1 11 10 0 1

7 Yes 1 1 4 10 0 1

8 Yes 1 1 6 10 0 19 Yes 1 1 7 10 0 110 Yes 1 1 4 10 0 1 Rate:100% Mean:1 Mean:6.9 Mean:0 Mean:1 SD:2.69 SD:0 SD:0

7. AccessMedicationList

User TaskSuccess




1 Yes 1 1 3 10 0 1

2 Yes 1 1 4 10 0 1

3 Yes 1 1 6 10 0 1

4 Yes 1 1 4 10 0 1

5 Yes 1 1 7 10 0 1

6 Yes 1 1 5 10 0 1

7 Yes 1 1 3 10 0 1

8 Yes 1 1 4 10 0 19 Yes 1 1 3 10 0 110 Yes 1 1 3 10 0 1 Rate:100% Mean:1 1 Mean:4.2 Mean:0 Mean:1 SD:1.4 SD:0 SD:0


8. AccessAllergyList

User TaskSuccess




1 Yes 1 1 3 10 0 1

2 Yes 1 1 5 10 0 1

3 Yes 1 1 5 10 0 1

4 Yes 1 1 3 10 0 1

5 Yes 1 1 6 10 0 1

6 Yes 1 1 3 10 0 1

7 Yes 1 1 3 10 0 1

8 Yes 1 1 3 10 0 19 Yes 1 1 4 10 0 110 Yes 1 1 3 10 0 1 Rate:100% Mean:1 Mean:3.8 Mean:0 Mean:1 SD:1.13 SD:0 SD:0

9. ClinicalDecisionSupport–Create&View

User TaskSuccess




1 Yes 11 8 151 120 0 4

2 Yes 9 8 96 120 0 2

3 Yes 9 8 118 120 0 2

4 Yes 8 8 60 120 0 2

5 Yes 8 8 73 120 0 2

6 Yes 8 8 75 120 0 1

7 Yes 9 8 120 120 0 2



10. ImplantableDeviceList

User TaskSuccess



Observed Optimal Observed Optimal 1 Yes 5 5 19 40 0 12 Yes 6 5 70 40 0 23 Yes 6 5 73 40 0 24 Yes 5 5 29 40 0 15 Yes 5 5 28 40 0 16 Yes 5 5 18 40 0 17 Yes 6 5 21 40 0 18 Yes 5 5 12 40 0 19 Yes 5 5 18 40 0 110 Yes 5 5 20 40 0 1 Rate:100% Mean:5.1 Mean:33.6 Mean:0 Mean:1.2 SD:22.0 SD:0 SD:0.42

11. ClinicalInformationReconciliation&Incorporation

User TaskSuccess




1 Yes 7 6 109 100 0 2

2 Yes 7 6 89 100 0 2

3 Yes 6 6 68 100 0 1

4 Yes 6 6 60 100 0 1

5 Yes 7 6 143 100 0 2

6 Yes 6 6 45 100 0 1

7 Yes 6 6 78 100 0 1



12. ElectronicPrescribing

User Task



1=Easy,5=Hard


1 Yes 7 7 39 80 0 1

2 Yes 7 7 38 80 0 1

3 Yes 7 7 65 80 0 1

4 Yes 7 7 69 80 0 1

5 Yes 9 7 125 80 0 3

6 Yes 7 7 49 80 0 1

7 Yes 7 7 55 80 0 1


TheresultsfromtheSUS(SystemUsabilityScale)scoredthesubjectivesatisfactionwiththesystembasedonperformancewiththesetaskstobe:91.2%.4

DiscussionofFindings

Following the drchrono Moderator’s Guide, the testing team collected all thenecessaryinformationtopopulatethetablesabove.Thetestingteamfirstcollectedparticipantdemographicsandthenaskedeachparticipanttocompleteallthetasks.Each taskwasseparately timed.Each testerwas thenasked tocompleteaSystemUsabilityScaleQuestionnaire.Aftereachusabilitysession,thetestingteamupdatedthetablesabovewiththeinformationcollected.

Onceall participantshad completed testing, the testing teamaggregated thedata,tabulating the task success completion rate, pathdeviations, task time,numberoferrors and task ratings. The testing team also calculated the mean and standarddeviationforeachcategory.

Effectiveness

All participants showed high effectiveness using the EHR. The participants wereable to complete all tasks successfully, with no task failures. The testing team 4 Broadly interpreted, scores under 60 represent systems with poor usability;scoresover80wouldbeconsideredaboveaverage.


observednomorethanone(3)deviationpertasks: implyingthatusersgenerallyunderstandhowtousetheEHRwellandcompletetheirevery-daytaskseffectively.

Efficiency

Duringtheusabilitystudy,nearlyall thetaskstookplacewithintheEHR’spatientdashboard. The participants’ test results confirmed that the EHR is logicallyorganized and allows users to complete tasks efficiently. Because of the EHR’slogical organization, the participants very rarely deviated from the optimal path.Mostpathdeviationsoccurredwhentheparticipanthadnotpreviously interactedwithaparticularfeaturebeingtested,suchasclinicaldecisionsupportrules,clinicalinformation reconciliation, implantable devices and—in one case—sending anelectronic prescription. Even though some participants had not used a featurepreviously,theywerenonethelessabletocompletethetasksquickly.

Satisfaction

Overall,theparticipantsindicatedthattheyweresubjectivelysatisfiedwiththeEHRwith a total satisfaction rating of 91.2%. As discussed above, participants fromdifferentdemographics showed similar satisfaction ratingsnow that they all haveexperienceusingourEHR.AllparticipantsindicatedtheywouldcontinueusingtheEHRandfurtherindicatedthattheyfounditeasytousetheEHRtoperformday-to-day tasks. While some providers showed more dissatisfaction than staff medicaladministers, theynonetheless showahigher satisfaction rating thanourprevioususability test. There were a number of testers requesting changes to the userinterface,whichtellsusthatthereareimprovementsthatneedtobemade.

MajorFindings

Sinceourfirstusabilitytest in2014,we’vefoundthatprovidersareasfluentwiththe EHR as they are with their staff. Initial findings suggest that providers wererelianton their staffmembers for administrative tasks, butournewusability testshows thatdoctorshave taken advantageof the system’s improvedusability overtheyearsandarestartingtoknowtheins-and-outsofthesystem.We’vefoundthatuserswithintheagerangeof20-59showconsiderableknowledgewhenusingthewebandcomputertechnology.


APPENDICES

The following appendices include supplemental data for this usability test report.Followingisalistoftheappendicesprovided:

1. SampleRecruitingScreener2. Participantdemographics3. ExampleModerator’sGuide4. SystemUsabilityScaleQuestionnaire


Appendix1:drchronoEHRUsabilityStudyScreener

drchronoisconductingausabilitystudytoimprovethedrchronoEHR.ThisusabilitystudywillhelpustoimprovetheEHRexperienceandmakechartingmoreefficient.ThisstudyconsistsaseriesoftasksusingourEHRandquestionsregardingtheEHR.Ifyouhaveadditionalideasandsuggestions,wewillbehappytohaveadiscussionafterthesession.Thisusabilitystudywilllastabout60minutesandcanbeconductedeitherinpersonatthedrchronoofficesorviaZoomMeeting.Ifyouareinterestedandqualifyforthestudy,youwillreceivea $100giftcardafterthesession.Name_________________________________________Email________________________________________________1. Haveyouparticipatedinafocusgrouporusabilitytestinthepast6months?

�Yes �No2. Doyou,ordoesanyoneinyourhomeworkinmarketingresearchorusability

research?

�Yes �No3. Do you, or does anyone in your home, have a commercial or research interest in

anelectronichealthrecordsoftwareorconsultingcompany?

�Yes �No4. HowlonghaveyoubeenusingdrchronoEHR?

�Lessthan6months �Morethan6months5. Doyouuseelectronicprescribingandlaborders?

�JusteRx �JustLabs �Both�Neither

6. Whatisyourcurrentposition?

�Physician �MedicalAssistant(MAorCMA)�RegisteredNurse(RN) �AdministrativeStaff�Resident �Other(pleasespecify)________________________


Appendix2:ParticipantDemographics

1. Gender

�Male �Female2. AgeRange

�23to39 �40to59 �60to74 �75+3. YearsinHealthcare

�0to5 �6to10 �11to20 �21to39 �40+4. Specialty __________________________________________________________________________________________5. Occupation/Role

�Physician �MedicalAssistant(MAorCMA)�RegisteredNurse(RN) �AdministrativeStaff�Resident �Other(pleasespecify)____________

6. TimeusingdrchronoEHR Years__________ Months__________


Appendix3:Moderator’sGuide

Introduction

• IntroductionofModerator• IntroductionofParticipant• Overviewofdesignstudy• Discussionofexpectationsandencourageopendiscussionafterthesession

ParticipantInformation

• NDA&InformedConsentForm• ParticipantDemographics

LogIn

• Logintotestaccount• Assumeroleoftestaccountspecificallycreatedforthisstudy

Task1:CPOE–MedicationOrder

• AddaMedicationOrder• ChangetheMedicationOrder

TaskTime:

PathDeviation:TaskOutcome:

Errors:

DifficultyRating:

Task2:CPOE–LaboratoryOrder

Use:PARoxetineChange:AddSignote“PatientMedication”


• AddaLabOrder• ChangetheLabOrder

TaskTime:


Errors:DifficultyRating:

Task3:CPOE–DiagnosticImaging(ImagingOrder)

• AddaImagingOrder• ChangetheImagingOrder

TaskTime:PathDeviation:

TaskOutcome:Errors:

DifficultyRating:

Task4:Drug-Drug,Drug-AllergyInteractionCheckforCPOE

• Drug-DrugInteractionCheck• Drug-AllergyInteractionCheck

UsetheFollowing:Status:OrderEnteredDateTestPerformed:TodayLOINCCode:12000-6ChangeLOINCCodeto:LOINCCode:13000-5

UseCPTCode:2000F-BloodpressuremeasureChangeCPTCodeto:30000-Drainageofnoselesion


TaskTime:

PathDeviation:

TaskOutcome:Errors:

DifficultyRating:

Task5:Demographics

• Accesspatientdemographics

TaskTime:


Errors:

DifficultyRating:

Task6:ProblemList• Accessthepatient’sproblemlist

TaskTime:

PathDeviation:

TaskOutcome:Errors:

DifficultyRating:

Usefordrug-drug:-PARoxetine10mgoraltablet-FasprinUsefordrug-allergy:SpecificDrugAllergy-d00170aspirinViewdrug-drug&drug-allergyinteractions.


Task7:MedicationList• Accessthepatient’smedicationlist

TaskTime:

PathDeviation:

TaskOutcome:Errors:

DifficultyRating:Task8:MedicationAllergyList

• Accessthepatient’sallergyListTaskTime:PathDeviation:

TaskOutcome:


Task9:ClinicalDecisionSupport

• CreateaCDSrule• CreateapatientappointmentthatdisplaystheCDSrule

TaskTime:


Errors:

DifficultyRating:

CreateCDSRule:Description:AgeCDSRulePatientDOEShave:AgeRangeinYears20–24yearsold.


Task10:ImplantableDeviceList

• AddImplantableDevice

TaskTime:


Errors:

DifficultyRating:Task11:ClinicalInformationReconciliationandIncorporation

• ReconcilePatientMedicationList• ReconcilePatientProblemList• ReconcilePatientAllergyList


TaskOutcome:


Task12:ElectronicPrescribing

• SendanElectronicPrescription

Participantsweregivingauniqueimplantabledevicecode.

InPatientDashboard–ReconcileMedication,Problem&Allergysections

Drug:Aspirin500mgoraltabletSIG:1gramoralonceadayIndication:1NotetoPharmacy:Testprescription



TaskOutcome:


SystemUsabilityQuestionnaire• Requestparticipantstofilloutquestionnaire


Appendix4:SystemUsabilityScaleQuestionnaire

StronglyDisagree StronglyAgree

1.IthinkthatIwouldliketousethissystemfrequently

12345

2.Ifoundthesystemunnecessarilycomplex

12345

3.Ithoughtthesystemwaseasytouse

12345

4.IthinkthatIwouldneedthesupportofatechnicalpersontobeabletousethissystem

12345

5.Ifoundthevariousfunctionsinthissystemwerewellintegrated

12345

6.Ithoughttherewastoomuchinconsistencyinthissystem

12345

7.Iwouldimaginethatmostpeoplewouldlearntousethissystemveryquickly

12345

8.Ifoundthesystemverycumbersometouse

12345

9.Ifeltveryconfidentusingthesystem

12345

10.IneededtolearnalotofthingsbeforeIcouldgetgoingwiththissystem

12345

drchrono safety-enhanced design usability report · 2019-01-01 · of evaluating the ehr’s...

Documents