ehr usability test report product: chirotouchsales@chirotouch.com introduction the ehr tested for...

9265SkyParkCourtSanDiego,CA92123Corp619-488-9800Sales800-852-1771Fax619-528-0050Sales@chirotouch.com www.chirotouch.com

EHRUsabilityTestReport

Product:ChiroTouch

Version:7.0

Reportbasedon(NISTIR7742)CustomizedCommonIndustryFormatTemplateforElectronicHealthRecordUsabilityTesting

DateofUsabilityTest: October11,2017toOctober13,2017

DateofReport: October16,2017

ReportPreparedBy:

TheUsabilityPeople,LLC.

703-687-6300 Info@TheUsabilityPeople.com 4000LegatoRoad,Suite1100 Fairfax,VA22033

Note:ThefollowingstudywasdevelopedusingtheNISTIR7742templateasaguideforreportingourfindings:CustomizedCommonIndustryFormatTemplateforElectronicHealthRecordUsabilityTesting.

ExecutiveSummaryAusabilitytestofChiroTouchVersion7.0,acompleteEHRwasconductedonOctober11toOctober13th,2017InavirtualenvironmentbyIntegratedPracticeSolutions,Inc.Thepurposeofthistestwastotestandvalidatetheusabilityofthe2015EHRuserinterface,andtoprovideevidenceofthisusabilityintheChiroTouchsoftware.

Duringtheusabilitytest,8chiropracticprovidersand2healthcareprofessionals(EHRusers)servedasparticipantsandusedtheEHRinsimulated,butrepresentativetasks.

Thisstudycollectedperformancedataon18taskstypicallyconductedonanEHR.Thetasksfocusedonthetwelve(12)2015EditioncertificationcriteriaspecifiedbyONC:

• Section170.315(a)(1)Computerizedproviderorderentry–medications• Section170.315(a)(2)Computerizedproviderorderentry–laboratory• Section170.315(a)(3)Computerizedproviderorderentry–diagnosticimaging• Section170.315(a)(4)Drug-drug,drug-allergyinteractionchecks• Section170.315(a)(5)Demographics• Section170.315(a)(6)Problemlist• Section170.315(a)(7)Medicationlist• Section170.315(a)(8)Medicationallergylist• Section170.315(a)(9)Clinicaldecisionsupport• Section170.315(a)(14)Implantabledevicelist• Section170.315(b)(2)Clinicalinformationreconciliationandincorporation• Section170.315(b)(3)Electronicprescribing

Duringthe60to90-minuteone-on-oneusabilitytest,eachparticipantwasgreetedbytheadministratorandaskedtoreviewandsignaninformedconsent/releaseform;theywereinstructedthattheycouldwithdrawatanytime.

MostparticipantshadpriorexperiencewiththeEHR.Theywerealsoprovidedtrainingmaterials(usermanualdocumentation)priortothetesttointroducethemtonewconcepts.

Theadministratorintroducedthetest,andinstructedparticipantstocompleteaseriesoftasks(givenoneatatime)usingtheChiroTouchsoftware.Duringthetesting,theadministratortimedthetestand,alongwiththedatalogger(s)recordeduserperformancedataonpaperandelectronically.Theadministratordidnotgivetheparticipantassistanceinhowtocompletethetask.Eachsessionwasrecordedforsubsequentanalysis.

TestLogisticsAllparticipantsweretestedontheChirotouchsystemduringremoteconferencingsessionsusingGotoMeeting.

TheGotoMeetingusabilitytestsessionwasconductedbyatestadministratorfromthetestingorganization(TheUsabilityPeople,LLC)workingfromasmallconferenceroomatTheUsabilityPeople’sFairfax,VAlocation.Adataloggerfromthetestorganizationalsotookdetailednotesoneachsession,includingusercommentsandotherratingsfollowingeachtask.Duringasessionboththetestadministratorandthedatalogger(s)couldseeonlytheparticipant’sscreenandheartheparticipant’scomments,questions,andresponses.

ChiroTouchwasevaluatedforeffectiveness,efficiencyandsatisfaction(ISO9241-11)asdefinedbythefollowingmeasurescollectedandanalyzedforeachparticipant:

• Numberoftaskssuccessfullycompletedwithintheallottedtimewithoutassistance• Timetocompletethetasks• Numberandtypesoferrors• Pathdeviations• Participant’sverbalizations

Participant’ssatisfactionratingsofthesystem.

ThefollowingisasummaryoftheperformanceandratingdatacollectedontheEHR.

Deviations Satisfaction

Task Participants TaskSuccess

TimetoComplete(Avg)

Total AverageAverage(outof5)

Task1DemographicInformation10 100% 1:49

60.60 4.20

Task2ProblemListView/Intervention10 100% 0:42

111.10 4.20

Task3ProblemListIntervention-Remove10 100% 0:57

40.40 3.40

Task4ClinicalInfoRecon/Incorporation10 70% 1:49

191.90 3.10

Task5MedicationAllergy-Add10 90% 1:33

171.70 3.50

Task6Drug-AllergyInteractionCheck10 100% 1:02

50.50 3.50

Task7ElectronicPrescribe10 100% 1:49

20.20 3.80

Task8eRXCancel/Change10 90% 2:46

111.10 3.90

Task9eRX-Renew10 100% 0:45

40.40 4.60

Task10MedOrder;Drug-DrugIntervention10 100% 1:39

10.10 4.30

Task11Medicationlist-Stop10 100% 0:21

00.00 4.80

Task12MedAllergyList-Remove10 100% 0:27

00.00 4.70

TheresultsfromtheSystemUsabilityScalescoredtheoverallsubjectivesatisfactionwiththesystembasedonperformancewiththesetaskstobe:53.

Inadditiontotheperformancedata,thefollowingqualitativeobservationsweremade:Inadditiontotheperformancedata,thefollowingqualitativeobservationsweremade:

1. MajorFindingsTheoveralltaskperformanceforthisstudywasveryhigh,withanoveralltaskcompletionrateofninety-seven(97)percent.Althoughtheperformancedatawasveryhigh,andtheindividualtaskperformanceratingwasalsohigh,thesubjectiveusabilityratingscollectedviatheSUSscalewerelowerthanexpected.

2. AreasforImprovementThissystem,likemanyotherEHRswouldbenefitfromseveralminorenhancementstotheuserexperience.ThetaskthatcreatedthemostnumberofissuesfortheparticipantswastheClinicalInformationReconciliationtask.Thistask,andseveralothers,couldbemadetobeeasiertoperformbyadditionsomeadditionalworkflowindicationtohelp“guide”theuserstowardscompletionofthetask.Severalotherareasforimprovementarediscussedlaterinthisdocument

Task13CPOEDiagImaging10 100% 1:44

10.10 4.10

Task14CPOEDiagImaging-Modify10 100% 1:18

60.60 4.30

Task15CPOELabOrders10 100% 1:36

10.10 4.20

Task16CPOELabOrders-Modify10 100% 0:59

30.30 4.70

Task17ImplantableDevice-Add10 100% 0:59

90.90 4.20

Task18ImplantableDevice-Modify10 100% 0:48

101.00 4.20

IntroductionTheEHRtestedforthisstudywasChiroTouchVersion7.0,acompleteEHR.Thissystemisdesignedtorepresentacompleteandcomprehensivedigitalversionofapatient’spaperchartwithreal-time,patient-centeredrecordsandpresentmedicalinformationtohealthcareprovidersinanambulatorysetting.ThissystemcomplieswithallONCcertificationcriteriaandisbeingsubmittedforverificationofcompliancewith2015certificationcriteria.Theusabilitytestingwasconductedwitheveryefforttorepresentrealisticexercisesandconditions.

Thepurposeofthisstudywastotestandvalidatetheusabilityofthecurrentuserinterface,andprovideevidenceofusabilityintheChiroTouchEHRsoftware.Tothisend,measureofeffectivenessandefficiency(timetoperformtasks;totalnumberofdeviations;totalnumberoferrors;etc.)werecapturedduringtheusabilitytesting.

Method

ParticipantsAtotalof10participantsweretestedontheChiroTouchEHRSoftware.Participantsinthetestwereindividualsthatworkwithinanambulatoryhealthcareenvironment.ParticipantswerecontactedbyChiroTouchdirectlyandfromadatabaseofmedicalprofessionalsmaintainedbythetestingorganization(TheUsabilityPeople,LLC).Inaddition,participantshadnodirectconnectiontothedevelopmentofororganizationproducingtheChiroTouchEHRSoftware.ParticipantswerenotfromIntegratedPracticeSolutions,Inc.Participantsweregiventheopportunitytohavethesameorientationandleveloftrainingastheactualenduserswouldhavereceived.

Forthetestpurposes,end-usercharacteristicswereidentifiedandtranslatedintoarecruitmentscreenerusedtosolicitpotentialparticipants.AnexampleofascreenerisprovidedinAppendix1.

Recruitedparticipantshadamixofbackgroundsanddemographiccharacteristicsconformingtotherecruitmentscreener.Thefollowingisatableofparticipantsbycharacteristics,includingdemographics,professionalexperience,userrole,userneedsforassistivetechnology,andproductexperience.ParticipantnameswerereplacedwithParticipantIDssothatanindividual’sdatacannotbetiedbacktoindividualidentities.

ParticipantID

Age Education Gender ProfessionalExperience

AssistiveTechnologyNeeds

UserRole ComputerExperience(years)

ProductExperience(Years)

1 P0101 50to59

Bachelor’sdegree Female 20

None Officemanager

2 P010250to59

Somecollegecredit,nodegree Female 30

Officemanager

23 P0103

40to49

Tradetechnicalvocationaltraining Female 22

Officemanager

44 P0104 50to

59Doctorate

degree(e.g., Female 35None Chiropractor

Atotaloftwelve12participantsrecruitedforthetestshoweduptoparticipateinthetest.Two(2)participantsfailedtoshowforthestudy.

Participantswerescheduledforsixty(60)toninety(90)minutesessionswithuptothirty(30)minutesinbetweeneachsessionfordebriefbytheadministrator(s)anddatalogger(s),andtoresetsystemstopropertestconditions.Aspreadsheetwasusedtokeeptrackoftheparticipantschedule,andincludedeachparticipant’sdemographiccharacteristicsasprovidedbythetestingfirm.

StudyDesignOverall,theobjectiveofthistestwastouncoverareaswheretheapplicationperformedeffectively,efficiently,andwithsatisfaction,andareaswheretheapplicationfailedtomeettheneedsoftheparticipants.ThedatafromthistestmayserveasabaselineforfuturetestswithanupdatedversionofthesameEHRand/orcomparisonwithotherEHRsprovidedthesametasksareused.Inshort,thistestingservesasbothameanstorecordorbenchmarkcurrentusability,butalsotoidentifyareaswhereimprovementsmustbemade.

Duringtheusabilitytest,participantsinteractedwithone(1)EHR.Eachparticipantwasprovidedthesamesetofinstructions.Thesystemwasevaluatedforeffectivenessandefficiencyasdefinedbymeasurescollectedandanalyzedforeachparticipant.

• Timetocompletethetask• Numberandtypesoferrors• Pathdeviations• Participantsverbalizations• Taskscompletedintheallottedtime• Participant’ssatisfactionratingsofthesystem• Numberoftaskscompletedsuccessfullywithintheallottedtimewithoutassistance

MD,DNP,DMD,PhD)

5 P0105 20to29

Bachelor’sdegree Female 5

None OfficeManager

6 P0106 40to49

Associatedegree Female 12

ChiropracticAssistant

7 P0107

50to59

Doctoratedegree(e.g.,MD,DNP,DMD,PhD) Male 27

Owner/chiropractor

38 P0108 20to

29 NursingDegree Female 9None RN

9 P0109

60to69

Highschoolgraduate,

diplomaortheequivalent Female 7

Ins.ClerkandCA

310 P0110 40to

49MastersDegree Male 4

None HealthITInstructor

AnumberoftaskswereconstructedthatwouldberealisticandrepresentativeofthekindsofactivitiesausermightdowiththisEHR,including:

• ModifyPatientDemographicinformation• Addanitemtopatientproblemlist• Removeanitemfrompatientproblemlist• Reconcileandintegrateinformationfromanotherprovider• AddamedicalAllergy• Prescribeadrug/ViewCDS• Prescribeadrugw/oCDSIntervention• CancelaneRX• RenewaneRX• Orderamedication;viewCDS• Stopamedication• Removeanallergy• Createimagingorders• ModifyImagingorder• CreateLaborder• ModifyLaborder• AddanImplantableMedicalDevice• ModifyanImplantableMedicalDevice

Taskswereselectedbasedontheirfrequencyofuse,criticalityoffunction,andthosethatmaybemosttroublesomeforusers.

ProceduresUponcommencementoftest,participantsweregreeted,remotelyloggedin,andtheiridentitywasverified.ParticipantswerethenassignedaparticipantID.Allparticipantssignedaninformedconsentformpriortothetesting(SeeAppendix1).Thetestadministratorprovidedtheinstructionsforeachtest,notedallcommentsfromtheparticipants;whilethedataloggernotedalltimes,pathdeviations,comments,successes,anderrors.

Participantswereinstructedtoperformthetasks:

• Afterlisteningtotheinstructionsfromthetestingadministrator• Asquicklyaspossiblewithasfewdeviationsorerrorsaspossible• Withoutassistance–participantswereallowedtogiveclarificationontasks,butnotguidanceonuse• Withoutusingathinkaloudtechnique

Foreachtask,theparticipantswereshownacopyofthetaskonthescreenandwerefreetoswitchback-and-forthbetweenthetaskandtheEHR.Tasktimingbeganafterthecompletionoftheverbalinstructionsfromtheadministrator;andafteranacknowledgementfromtheparticipantthattheywerereadytobegin.Thetasktimewasstoppedoncetheparticipantindicatedtheyhadsuccessfullycompletedthetask.

Followingthetest,theadministratorgavetheparticipantthepost-testquestionnaire;andthenthankedthemfortheirtime.Participants'demographicinformation,tasksuccessrate,timeontask,errors,deviations,verbalresponses,andpost-testquestionnairewererecordedintoaspreadsheet.

TheUsabilityPeople,LLCstaffmemberswereresponsibleforloggingdatarecordedallparticipants’demographicinformation,tasksuccessrate,timeontask,errors,comments,anddeviationsintoaspreadsheet.

TestLocationThetestwasadministeredinaremotesettingwhereparticipantswereisolatedfromotherparticipantsinthestudy.Onlythetestadministratorandloggerwereonthecallwiththeparticipantswhilethestudywasbeingadministered.

TestEnvironmentTheChiroTouchEHRSoftwarewouldbetypicallybeusedinaprivatepractice,andinthisinstancethetestingwasconductedremotelyineachparticipant’sactualpractice.Althoughthetypeofcomputer,operatingsystemanddisplayresolutionoftheremoteparticipantsystemwasunknown,thesystemthatwasusedbythetestadministratorandcontrolledbytheremoteparticipantwasaDellInspironLaptoprunningtheWindows10operatingsystemataresolutionof1366x768pixels.UsersalsousedamouseandkeyboardwhileinteractingwiththeEHR.

TheChiroTouchapplicationisaprimarilyclient-serversolutionwithsomewebcomponents;andallcomputerswererunningonhighspeedinternetconnections.Allparticipantsindicatedthatsystemperformanceduringthetestwaswhattheywereusedtoseeingduringtheirtypicalworkday.Additionally,participantswereinstructednottochangeanyofthedefaultsystemsettings(suchascontroloffontsize).

TestFormsandToolsDuringtheusabilitytest,variousinstrumentsanddocumentswereused,including:

• Informedconsent• ModeratorGuide• Post-testquestionnaire• Study

ExamplesofthesedocumentscanbelocatedintheAppendixsection.Theparticipant’sinteractionwiththeChiroTouchEHRSoftwarewascapturedandrecordeddigitallywithscreencapturesoftwarerunningonthemoderator’smachine.

ParticipantInstructionsTheAdministratorreadthefollowinginstructionsaloudtoeachparticipant:

“Thankyouforparticipatingintoday’susabilitystudyofChiroTouch.Inafewminutes,youwillbeaskedtoperformaseriesoftasksandcompleteausersurvey.Pleaseattempttocompletethesetasksasquicklyaspossible.ThepurposeofthisstudyisforChiroTouchtoobtaininformationonwhereenhancementsareneededintheapplicationbasedonhowquicklyandeasilythetaskisbeingperformedintheChiroTouchEHRsystem.

Pleasenotethatwearenottestingyou;wearetestingthesystem.Thereforeifyouhaveanydifficultythismaymeanthatsomethingneedstobeimprovedinthesystem.Iwillbehereincaseyouneedspecifichelp,butIamnotabletoinstructyouorprovidehelpinhowtousetheapplication.

Overall,weareinterestedinhoweasy(orhowdifficult)thissystemistouse,whatinitwouldbeusefultoyou,andhowwecouldimproveit.Ididnothaveanyinvolvementinitscreation,sopleasebehonestwithyouropinions.

Alloftheinformationthatyouprovidewillbekeptconfidentialandyournamewillnotbeassociatedwithyourcommentsatanytime.Shouldyoufeelitnecessaryyoumaywithdrawatanytimeduringtesting.”

Participantswerethengiven18taskstocomplete.TasksarelistedintheDataAppendix3.

UsabilityMetricsAccordingtotheNISTGuidetotheProcessesApproachforImprovingtheUsabilityofElectronicHealthRecords,EHRsshouldsupportaprocessthatprovidesahighlevelofusabilityforallusers.Thegoalisforuserstointeractwiththesystemeffectively,efficiently,andwithanacceptablelevelofsatisfaction.Tothisend,metricsforeffectiveness,efficiencyandusersatisfactionwerecapturedduringtheusabilitytesting.

Thegoalsofthistestweretoassess:

1. TheefficiencyofChiroTouchbymeasuringthelengthoftimeittakesforausertocompletethetask;andthesuccessoftaskcompletion.

2. TheefficiencyofChiroTouchbymeasuringthepathdeviationstakenbytheuserduringthetask.3. TheefficiencyoftheChiroTouchbymeasuringtheaveragetasktimeandpathdeviations.4. TheeffectivenessofChiroTouchbymeasuringthenumberandtypesoferrorsexperiencedbytheuserduringthetask.5. ThesatisfactionoftheuserwithChiroTouchbyloggingtheircommentsonthetask.

DataScoringThetablebelowdetailshoweachtaskwasscored.

Measure RationaleandScoring

TaskTime Timingstartedwhentheadministratorsaid‘Begin’.Thetimeendedwhentheparticipantsaid‘Done’.Intheeventthattheparticipantfinished,anddidnotsay‘Done’,theadministratorstoppedtheclockwhenitwascleartheparticipanthadcompletedthetask.Tasktimewasonlycountediftheparticipantcompletedthetaskintheallottedtime.Averagetimepertaskwascalculatedforeachtask.

Errors Thetaskresultedinanerroriftheparticipant:failedtofinishthetaskintheallottedtime;or,iftheybecame‘stuck’andcouldnotproceedwithoutaskingforassistance.Tasktimewasnotcountedwhenthetaskresultedinanerror.

PathDeviations Pathdeviationswererecordedasactionstakenduringthetaskthatwerenotpartofthenecessaryactionsneededtocompletethetask.

Wecalculatedpathdeviationsbytakingthetotalnumberofobserveddeviationsanddividingthatnumberbythetotalnumberofstepstakenusinganoptimalpathtoprovidearatioofpathdeviation.

TaskSuccess Thetaskwasconsideredasuccessiftheparticipantcompletedthetaskintheallottedtimewithoutassistance.

TaskRating Participant’ssubjectiveimpressionoftheeaseofusewasmeasuredbyadministeringapost-sessionquestionnaireandapost-taskquestion.Theparticipantswereaskedtoratetheoveralleaseofeachtask on a scale of 1 (Very Difficult) to 5 (Very Easy) To measure participants’ confidence in andlikeabilityoftheoverallsystem,thetestingteamadministeredapost-testquestionnaire.Questionsincludedthoseaddressingtheeaseofuseofthesystemandparticipants’thoughtsonhowquicklyausercouldlearnthesystem.SeefullquestionnaireinAppendix2.

Results

DataAnalysisandReportingTheresultsoftheusabilitytestwerecalculatedaccordingtothemethodsspecifiedintheUsabilityMetricssectionabove.

AsummaryofthetestingresultsforChiroToucharedetailedbelow.Thetablebelowidentifiesthetasksperformedandtheperformancelevelandsatisfactionforeachtask.

Task MeanTaskTime

CompletionRate(%)

Mean#PathDeviations

MeanTaskSatisfaction

Task1DemographicInformation 1:49 0:54 100% 0.60 0.66 4.20 0.98

Task2ProblemListView/Intervention 0:42 0:10 100% 1.10 0.70 4.20 0.75

Task3ProblemListIntervention-Remove 0:57 0:37 100% 0.40 0.92 3.40 1.20

Task4ClinicalInfoRecon/Incorporation 1:49 1:01 70% 1.90 1.81 3.10 1.45

Task5MedicationAllergy-Add 1:33 0:40 90% 1.70 1.62 3.50 1.02

Task6Drug-AllergyInteractionCheck 1:02 0:21 100% 0.50 0.81 3.50 1.20

Task7ElectronicPrescribe 1:49 0:26 100% 0.20 0.40 3.80 1.08

Task8eRXCancel/Change 2:46 0:40 90% 1.10 1.87 3.90 0.94

Task9eRX-Renew 0:45 0:42 100% 0.40 0.92 4.60 0.92

Task10MedOrder;Drug-DrugIntervention 1:39 0:22 100% 0.10 0.30 4.30 0.64

Task11Medicationlist-Stop 0:21 0:07 100% 0.00 0.00 4.80 0.40

Task12MedAllergyList-Remove 0:27 0:12 100% 0.00 0.00 4.70 0.64

Task13CPOEDiagImaging 1:44 0:28 100% 0.10 0.30 4.10 0.83

Task14CPOEDiagImaging-Modify 1:18 0:21 100% 0.60 0.66 4.30 0.64

Task15CPOELabOrders 1:36 0:43 100% 0.10 0.30 4.20 0.75

Task16CPOELabOrders-Modify 0:59 0:15 100% 0.30 0.64 4.70 0.46

Task17ImplantableDevice-Add 0:59 0:24 100% 0.90 0.83 4.20 0.60

Task18ImplantableDevice-Modify 0:48 0:20 100% 1.00 1.18 4.20 0.75

Asthetableaboveshows,relativetooptimalperformancestandardsasdefinedbyChiroTouchandTheUsabilityPeople,

participantperformanceintheChiroTouch EHRusabilitytestwasquitesatisfactory.Theoverallaveragetaskcompletionrate

wasninety-seven(97)percent.

ReliabilityDuringtheentiredatacollectionphase,itwasobservedthatthesystemprovidedaconsistentandreliableinterfacetoeachparticipantastheycompletedtheirtasks.Aseachparticipantcompletedtheirassignedtasks,thesystemprovidedthesameinformationandrespondedtotheirinputwiththesameverbiageandusingthesamemodeofcommunication(e.g.Pop-upmessage,orembeddedassistance).

RiskAssessmentThefollowingtablepresentsaprioritizedlistoftasks(alongwithadescriptionofthetaskandanidentificationofthecriteria

evaluated)prioritizedbytheriskoferrorasobservedduringthetesting.

TaskCriteria

Evaluated DescriptionPercentComplete RiskStatus

Task1DemographicInformation a(5) ModifyPatientDemographicinformation 100% None

Task2ProblemListView/Intervention a(6) Addanitemtopatientproblemlist 100% None

Task3ProblemListIntervention-Remove a(6) Removeanitemfrompatientproblemlist 100% Low

Task4ClinicalInfoRecon/Incorporation b(2)Reconcileandintegrateinformationfromanotherprovider 70% Moderate

Task5MedicationAllergy-Add a(8) AddamedicalAllergy 90% Low

Task6Drug-AllergyInteractionCheck a(1);a(9) Prescribeadrug/ViewCDS 100% None

Task7ElectronicPrescribeb(3);a(4);

a(9) Prescribeadrugw/oCDSIntervention 100% None

Task8eRXCancel/Change b(3) CancelaneRX 90% Low

Task9eRX-Renew b(3) RenewaneRX 100% None

Task10MedOrder;Drug-DrugInterventiona(7);a4;a(9) Orderamedication;viewCDS 100% None

Task11Medicationlist-Stop a(7) Stopamedication 100% None

Task12MedAllergyList-Remove a(8) Removeanallergy 100% None

Task13CPOEDiagImaging a(3) Createimagingorders 100% None

Task14CPOEDiagImaging-Modify a(3) ModifyImagingorder 100% None

Task15CPOELabOrders a(2) CreateLaborder 100% None

Task16CPOELabOrders-Modify a(2) ModifyLaborder 100% None

Task17ImplantableDevice-Add a(14) AddanImplantableMedicalDevice 100% None

Task18ImplantableDevice-Modify a(14) ModifyanImplantableMedicalDevice 100% None

EffectivenessParticipantsinthestudyexperiencederrorsinallbuttwo(2)tasks.However,thebulkoftheerrorsoccurredinthreeareas:“ClinicalInformationReconciliation”,“AddanAllergy”,and“Cancelanelectronicprescription.”ErrorsfoundintheseareaswereofnosurprisetoChiroTouch.TheClinicalInformationReconciliationapplicationisanewercomponenttoChiroTouchusers.Thisfeature,aswellastheallergyandeRXfeaturesarenotusedtypicallybythethoseintheChiropracticprofession.

EfficiencyParticipantswhosuccessfullycompletedtasksgenerallycompletedthosetaskswithinanacceptabletime.Sometaskswerecompletedmorequicklythanthecalculatedoptimaltime,whileseveraltaskstookslightlylongerthanexpected.Thetasksthattookthelongestrequiredtheparticipantstonavigatetoaparticularunfamiliarportionofapage,interactwithaworkflow,locateandselectspecificactions.

Someparticipantsmadeerrorswhenattemptingtonavigatetowardsolvingtheirassignedtasks.TheseerrorsmaybeassociatedwiththoseparticipantsnotbeingfamiliarandnotunderstandingthepresentedinformationarchitectureoftheEHRsystem.Asnotedabove,priorexperiencewithEHRsystemswasrelatedtosuccessfultaskcompletion.Similarly,experienceandpracticewiththegivensystemmayhavepositiveeffectswithregardtouserefficiency.

SatisfactionIndividualTaskSatisfaction

Participantsverballyindicatedtheirsatisfactionwiththeeaseofuseforeachtaskusingascaleof“1”(“VeryDifficult”)to“5,”

(“VeryEasy”).Asthefigurebelowshowsindividualtasksatisfactionrangedfromalowof3.1outof5onTask4(Clinical

InformationReconciliation)toahighof4.8outof5onTasks11(Medicationlist–Stopamedication).Theoverallaveragetask

satisfactionwas4.09indicatingthatoveralltheparticipantswerequitesatisfiedwiththeirtasks.

SystemUsabilityScale

TheSystemUsabilityScale(SUS)isasimple,10-itemLikert-typeattitudescaleprovidingaglobalsubjectiveassessmentof

usabilityfromtheuser’sperspective(JohnBrookeatDigitalEquipmentCompanydevelopedtheSUSin1986).TheSUSscaleis

scoredfrom0to100;scoresunder60representsystemswithlessthanoptimalusability,scoresover80areconsideredbetter

thanaverage.

ThemeantotalSUSscorefortheChiroTouch EHRwasfifty-three(53)andrangedfromalowofthirty(30)andahighofninety

(68).Overall,participant-usersratedtheirsatisfactionwiththeChiroTouch EHRsystemtobelowerthantheexpectedrange

giventhehighindividualsatisfactionratingsandexcellenttaskperformancedata.

ThefollowinggraphshowstheSUSscorebyeachparticipant:

0.00 1.00 2.00 3.00 4.00 5.00

Task1DemographicInformationTask2ProblemList…Task3ProblemList…Task4ClinicalInfo…

Task5MedicationAllergy- AddTask6Drug-AllergyInteraction…

Task7ElectronicPrescribeTask8eRXCancel/Change

Task9eRX- RenewTask10MedOrder;Drug-Drug…Task11Medicationlist- Stop

Task12MedAllergyList-…Task13CPOEDiagImagingTask14CPOEDiagImaging-…

Task15CPOELabOrdersTask16CPOELabOrders- ModifyTask17ImplantableDevice- Add

Task18ImplantableDevice-…

4.204.20

3.403.103.503.503.803.904.604.304.804.70

4.104.304.204.70

4.204.20

AverageTaskSatisfaction

Onthepost-studyquestionnaire4out8ofexperiencedChirotouchusersexpressedtheywere“VeryorSomewhatSatisfied”withtheChiroTouchsystem.

MajorFindings

Theoveralltaskperformanceforthisstudywasveryhigh,withanoveralltaskcompletionrateofninety-seven(97)percent.Althoughtheperformancedatawasveryhigh,andtheindividualtaskperformanceratingwasalsohigh,thesubjectiveusabilityratingscollectedaspartoftheSystemUsabilityScale(SUS)weremuchlowerthanexpected.Thisseemstoimplythatalthoughthesystemiseasytouse,andthatparticipantsareabletosuccessfullycompletetheirtasks,somethingisnegativelyinfluencingtheiroverallsatisfactionwiththesystem.

AreasforImprovement

Theoveralltaskperformancerateforthistestwasveryhigh,atninety-seven(97)percent.Thisindicatesthatamajorityofthesystemishighlyusableanddoesnotrequireanysignificantimprovement.Therearealways,however,severalareasthatcouldbeenhancedtoachieveevenhighertaskcompletionrates,andperhapscouldincreaseoverallparticipantsatisfaction.

• Indicationoftaskworkflowo Inseveralareasoftheproduct,theusermustnavigateacrossseveraltabsorotheruserinterfaceelementinto

ordertocompletetheirtask.Insteadofpresentingtheseitemsacrosscommontabs,theworkflowcouldbeindicatedusinga1->2->3typearrangementthatwouldservetoguidetheusertocompletionofthetask.

o Inaddition,someembeddedassistancecouldbeprovidedtohelpusersunderstandandcompletetheirtasks.

• Buttons;PrimaryandSecondarybuttonso Acrossthesystemtherewasaninconsistentuseofuseractionbuttons,links,buttonnamesandoccasionally

buttonarrangement.Thesystemcouldbenefitfromprovidingamoreconsistentuseofbuttonsorlinksandperhapsamoresalientindicationofprimaryvssecondaryactionbuttons.

• Betterintegrationof3rdpartytoolso Thesystemtakesadvantageofseveral3rdpartytoolsinordertoachieverequiredfunctionality.Unfortunately,

eachofthesetoolspresentsadifferentuserinterface,andthiscanhavenegativelyimpactedtheperceptionoftheoveralluserexperience.

Appendix1:SampleConsentForm

ChiroTouchUsabilityStudyPracticeConsent

ChiroTouchwouldliketothankyouforconsideringparticipationinthisstudy.ThepurposeofthestudyistoevaluatetheusabilityoftheChiroTouchEHRsystem.YourparticipationinthisstudywillincludeperformingaspecifictaskwithinChiroTouch;andcompletingashortsurveyfollowingthestudy.Thestudyshouldtakeapproximately60-90minutes.TheinformationcollectedbyChiroTouchduringthestudyisforresearchpurposesonly.Yourparticipationinthisstudyisvoluntary,soyouarefreetowithdrawatanypointduringthestudy.

Pleaseanswerthefollowingscreeningquestions:

☐☐ Doyou,oranyoneinyourhome,haveacommercialorresearchinterestinanEHRsoftwareorconsultingcompany?☐☐ Doyou,ordoesanyoneinyourhome,workinmarketingresearch,usabilityresearch,webdesign[…etc.]?☐☐ Doyourequireanyassistivetechnologiestouseacomputer?☐☐ Areyouactivelypracticingasalicensedmedicalproviderwithinyourclinic?

NameofPractice

NameofParticipant

Age ☐23to39|☐40to59|☐60-to74|☐75andolder

Gender ☐Male☐Female☐Other

YearsinPractice(ifDC)

Position ☐Provider☐Other ☐FrontOffice☐BackDesk/ProviderAssistants

Experiencew/ChiroTouch(Yrs)

Location

Bysigningbelow,Iagreetoparticipateinthestudy.

SignatureDate PrintedNameDate

Appendix2:SamplePost-TestQuestionnaire

ChiroTouchUsabilityStudyPost-TestQuestionnaire

1. Howwouldyoupreferwecommunicatenewenhancementstoyourpractice?�Email �MembersAreaPage �Newsletter �Wedonotcaretobenotified

2. WhatisyourpreferredmethodofworkingwiththeChiroTouchSupportdepartment?�Phone�Email �OnlineChat(ifthiswereavailable)

3. Whatisyourpreferredmethodforcreatingaprescription?�DrugSearch �Favorites

4. HowlongdoesittaketoenteranewpatientinChiroTouch(Demographicdataonly)?�0-3minutes �3-5minutes �>5Minutes

5. IfyoucouldchangeonepartofChiroTouch,whatwouldyouchange

6. DoyoufeelthattheChiroTouchsystemmakesreportingandmaintainingcomplianceeasy?

7. Whatcompliancefeature(s)wouldyouaddtotheChiroTouchreportingsystemtomakeiteasierforyoutoreport?

8. Onascaleof1-5,howwouldrateyouroverallsatisfactionwithChiroTouch?�1(VeryDissatisfied) �2(SomewhatDissatisfied) �3 �4(SomewhatSatisfied) �5(VerySatisfied)

9. WhichofthefollowingfunctionsareavailablethroughChiroTouch?(Youmaycheckmorethan1option)�PatientEducationMaterials�ElectronicallyPrescribeControlledSubstances�PrescriptionCouponsfordistributingtoPatients�Secureemailmessaging

10. DidyoufindtheChiroTouchsystemoranypartofitunnecessarilycomplexorcumbersome?Pleaseexplain.

11. IfyoucouldaddonepieceoffunctionalitytoChiroTouch,whatwouldyouadd?

Appendix3Data

1. TasksPerformedDuringStudyPreparation Task DesignatedTaskTime OptimalPaths

Task1DemographicInformation

ModifyPatientDemographicinformation

120(seconds)7

Task2ProblemListView/Intervention

Addanitemtopatientproblemlist 30(seconds)

5Task3ProblemListIntervention-Remove

Removeanitemfrompatientproblemlist

30(seconds)

2Task4ClinicalInfoRecon/Incorporation

Reconcileandintegrateinformationfromanotherprovider

180(seconds)

11Task5MedicationAllergy–Add

AddamedicalAllergy 60(seconds)9

Task6Drug-AllergyInteractionCheck

Prescribeadrug/ViewCDS 120(seconds)

5Task7ElectronicPrescribe Prescribeadrugw/oCDSIntervention 180(seconds) 6Task8eRXCancel/Change CancelaneRX 180(seconds) 12Task9eRX-Renew RenewaneRX 30(seconds) 2Task10MedOrder;Drug-DrugIntervention

Orderamedication;viewCDS 60(seconds)6

Task11Medicationlist-Stop Stopamedication 30(seconds) 2Task12MedAllergyList–Remove

Removeanallergy 30(seconds)2

Task13CPOEDiagImaging Createimagingorders 120(seconds) 7Task14CPOEDiagImaging-Modify

ModifyImagingorder 60(seconds)

6Task15CPOELabOrders CreateLaborder 60(seconds) 7Task16CPOELabOrders–Modify

ModifyLaborder 60(seconds)

6Task17ImplantableDevice–Add

AddanImplantableMedicalDevice 60(seconds)

5Task18ImplantableDevice–Modify

ModifyanImplantableMedicalDevice 30(seconds)4

2. ParticipantsDemographicsGender

Male 2Female 8Total 10

ChiroTouchUserRoleProvider 2BackDesk/Prov.Asst. 1FrontDesk 5Other 2

YearsofExperiencew/ChiroTouch<1 2<2 3>2 5

YearsinPractice<1 02-5 25-10 2>10 6

Appendix4:ModeratorGuide

ChiroTouchUsabilityStudyModeratorGuide

Prior to testing § Confirm schedule with Participants § Ensure the client’s ChiroTouch software is running properly

Prior to each participant:

§ Reset application § Start session recordings with Morae

Prior to each task:

§ Reset application to starting point for next task

After each participant: § End session recordings with Morae

After all testing

§ Back up all video and data files

Orientation (5 minutes)

Thank you for participating in this study. Our session today will last 60 to 90 minutes. I will ask you to complete a few tasks using this system and answer some questions. We are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. You will be asked to complete these tasks on your own trying to do them as quickly as possible with the fewest possible errors or deviations. Do not do anything more than asked. If you get lost or have difficulty I cannot help you with anything to do with the system itself. Please save your detailed comments until the end of a task or the end of the session when we can discuss freely.

I did not have any involvement in its creation, so please be honest with your opinions.

The product you will be using today is describe the state of the application, i.e., production version, early prototype, etc. Some of the data may not make sense as it is placeholder data.

We are recording our session today. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time.

Do you have any questions or concerns?

TASK1:DemographicInformation (120 Seconds)

Direct the participant to the starting point for the task.

Yourpatient,KerryKennedy,hasarrivedfortheirscheduledappointment.Beforebeginningtheirtreatment,youneedtoverifyorentersomeofthedemographicinformationthatisstoredwithinthesystem.NavigatetotheDemographicinformationsectionfromtheFrontDeskapplicationandmakesurethatthefollowinginformationisstoredintheEHR:

Asyoureviewthepatienthistory,theymentionthattheyprefertospeakSpanishasherfirstlanguageandsheisnotascomfortablereadingEnglish.

• SetthepatientLanguageto‘Spanish’• SetthepatientRaceas‘African-American’• SetthepatientEthnicityas‘HispanicorLatino’

Asyoucontinuethroughtheirsocialhistory,theyinformyouthatwhilehersexassignedatbirthwasmale,shehaslegallychangedhernametoRandyandnowidentifiesasafemale.Documentthefollowinginformationinthechart.

• SexualOrientationas'Choosenottodisclose'• IndicatetheGenderIdentityas‘Male-to-Female(MTF)transsexual'

EnterandsavethisinformationintotheEHR.

Success:

¨ Easilycompleted¨ Completedwithdifficultyorhelp::Describebelow¨ NotcompletedComments:

Task Time: Seconds

Comments:

Observed Errors and Verbalizations:

Rating: Overall, this task was:

Ask participant to rate the task from : “Very Difficult” (1) to “Very Easy” (5)

Administrator / Notetaker Comments:

TASK2:ProblemListViewandIntervention. (30 Seconds)

Yourassistanthasmeasuredthepatient’sbloodpressureandrecordedthefollowingvalues:150/90.Consideringtheseresults,youhavediagnosedyourpatientwiththefollowing:

401.9-Hypertension

Success: ¨ Easilycompleted¨ Completedwithdifficultyorhelp::Describebelow¨ NotcompletedComments:

Task Time: Seconds

Comments:

TASK3:ProblemListIntervention--Remove. (30 Seconds)

Youalsonoticethattheyhavealisteddiagnosisof“(M54.5)LowBackPain“ofwhichtheyclaimthattheynolongersufferfrom.

• Removethe“(M54.5)LowerBackPain“diagnosis.”

Task Time: Seconds

Comments:

TASK4:ClinicalInformationReconciliation/Incorporation (180 Seconds)

ThepatienthascometoyourofficewithaCCDAdocumentwithmedicalinformationfromanotherprovider.Thisinformationhasalreadybeenpre-loadedintotheChiroTouchsystem.UsingtheReconciliationfeaturesforCCDAlocatedonthepatientsearchscreenonthePatientsapplicationpleasedothefollowing:

• AccesstheCCDDocumentforyourpatient• ReconciletheMedications• ReconciletheMedicationAllergies• ReconciletheProblems.

Task Time: Seconds

Comments:

TASK5:MedicationAllergyList-Addanallergy.(60 Seconds)

Asyoubeginyourconsultationthepatienttellsyouthattheyrememberthataftertakingamedicationthatcontained“sulfa”awhilebacktheyhadexperiencedsevereurticaria(hives).

• AddanallergyforSulfadixineorSulfa(SulfonamideAntibiotics)totheirrecord.o Youmayseeawarningmessage.

Task Time: Seconds

Comments:

TASK6:Drug-AllergyInteractionCheck.(120 Seconds)

Yourpatientsaidthatthemainreasontheycameintoyourofficeisthattheystartedtofeelpoorlyaftereatingasaladbowlfromalocalfast-foodMexicanrestaurantchain.Youthinkthattheymayhavecontractedane-Coliinfection.

YoudecidetoprescribeKeflex(cephalexin)capsule250MG.However,onaddingKeflexyounoticeadrug-allergyinteractionalertsoyoudecidetostopthisprescription.

EntertheinformationaboveintotheEHRandidentify/viewthedrug-allergyalert.

Task Time: Seconds

Comments:

TASK7:ElectronicPrescribing. (180 Seconds)

Youstillneedtoprescribeamedicationtowhichthepatientisnotallergic.YoudecidetoorderDoxycyclineinstead.

Pleaseprescribe:

• DoxycyclineHyclate20MGtabletstobetakenonetablettwiceadayaftermealsfor5days.

• SendtheRXtothechosenpharmacyusingthefollowingSIG:1234

Task Time: Seconds

Comments:

Task8.ElectronicPrescribingCancel/Change. (180 Seconds)

Oops,youjustrealizedthatyouprescribedandsentelectronicallytothepharmacytheantibioticaboveat20mgdose,butyoureallywantedtoprescribethe100mgDoxycyclinetablet.Youneedtomakemodificationssuchthatthepharmacydispensesthecorrectdosage,andmakesurethepatientsmedicationlistisalsocorrect.

• Navigatetoareportoftheprescriptionsthatweresentelectronicallyandcancelthe20mgtablet.• PrescribeDoxycyclineHyclate100MGtabletstobetakenonetablettwiceadayaftermealsfor5days.

• MakesuretheMedicationlistreflectsthischange.

Task Time: Seconds

Comments:

Task9.ElectronicPrescribing–renew. (30 Seconds)

YoupatienthasbeentakingLipitor(atorvastatin)forseveralyearswithoutanyproblemsandithasbeendoingagoodjobcontrollingtheircholesterollevels.Theyhaveaskedthatyourenewtheircurrentprescription.

• ElectronicallyrenewtheexistingRXforLipitor(atorvastatin)

Task Time: Seconds

Comments:

TASK10:MedicationOrder,ViewDrug-DrugInteractions,MedicationListInterventions (60 Seconds)

SeveraltasksagoyoudiagnosedyourpatientwithHypertension.Inordertotreatyourpatient’selevatedbloodpressure,youdecidetoaddAccupriltotheirhypertensiondiagnosis.

Enteraprescriptionfor:

• AccuprilOralTablet10MGtotakeonetablettwiceadayfor30days.

Task Time: Seconds

Comments:

Task11:Stopamedication. (30 Seconds)

Uponexaminationofthepatient’scurrentmedicationlist,younoticethatthedrugNaproxenislistedasanactivemedication.YoumayalsonoticethatthisdrugisassociatedwithareactiontoapatientAllergy.YouneedtostopthepatientfromtakingNaproxenandindicatethisstopintheirrecord.

• EnteramedicationstopforNaproxen.

Task Time: Seconds

Comments:

TASK12:MedicationAllergyList-Removeanallergy. (30 Seconds)

Duringtheconsultation,yourpatienthastoldyouthattheynolongerhaveanallergytoAmoxicillin.Youdecidetoremovethisallergyfromthepatientrecord.

• RemovetheAmoxicillinallergyfromthepatientrecordintheEHR.

Task Time: Seconds

Comments:

Task13:RecordanoutpatientRadiologyorder. (120 Seconds)

Yourpatienthasbeencomplainingofanklepainafterinjuringthemselveswhileplayingbasketball.Uponfurtherexamination,youdecidetoorderanx-ray,usingtheCPOEfeaturesofthesystem.EnterthefollowingintotheEHR:Diagnosis:ICD-10code:S93.01XASubluxationofRightAnkleJointTest:Radiologicexaminationankle2viewsReview:RequestingPhysicianCraigPalmer

Task Time: Seconds

Comments:

Task14:ModifyoutpatientRadiologyorder. (60 Seconds)

YourPatienthasalsocomplainedofKneepainafterinjuringthemselveswhileplayingbasketball.Youdecidethatyouwanttoaddanadditionalitemtotheradiologyorderthatyouplacedabove.Modifytheorderyourplacedinthetaskabovetoincludethefollowingtest:

• Radiologicexaminationknee3views

Task Time: Seconds

Comments:

TASK15:LaboratoryOrders. (60 Seconds)

Yourpatienthasbothhighbloodpressureandelevatedcholesterollevels.Youwanttoordersomeotherbasiclabworktomakesurethatthepatientdoesn’thaveanyotherunderlyingproblems,suchasakidneyissue.UsingtheCPOEsystempleaseenteranorderforlabworkasfollows:

• Diagnosis:EssentialPrimaryHypertension• Tests:

o Creatinine,24-Hoururineo Fastingglucoseinserumorplasma

EnterthelaborderintotheEHR.

Task Time: Seconds

Comments:

TASK16:LaboratoryOrders–Modify (60 Seconds)

Uponreviewofthelaborderthatyoujustcompletedintheprevioustask,youdecidetochangethislaborderandaddanadditionaltest.Addthefollowinglabtesttothepreviousorder.

• Urobilinogen,Semi-Qn

Makethenecessarychangessothattheadditionaltestisalsoordered.

Task Time: Seconds

Comments:

TASK17:ImplantabledevicelistTASK7:ElectronicPrescribing. (60 Seconds)

Thepatienttellsyouthatafewyearsbacktheyhada“Calciumcompoundbonevoidfiller”implantedintotheirleftarm.UponexaminationoftheimplantabledevicelistpresentedintheEHR,thereisnorecordofthismedicaldevice.

• AddamedicaldevicewiththefollowingUDIcode.

=/W4146EB0010T0475=,000025=A99971312345600=>014032=}013032&,1000000000000XYZ123(Note:thisinformationisstoredonyourclipboardandsavedasa.txtfileontheWindowsDesktop)

• Parsetheinformationassociatedwiththeimplantedmedicaldeviceandverballystatetheexpirationdate.

Task Time: Seconds

Comments:

TASK18:Implantabledevice-Modify. (30 Seconds)

Theimplanteddevicethatyouenteredintheprevioustaskisexpiredandshouldnolongerclassifiedas“active.”ModifytheinformationfortheBoneVoidFillersothattheDeviceStatusissettoINACTIVE.

Task Time: Seconds

Comments:

Appendix5.SpecificTaskData

ParticipantNumber

Task1DemographicInformation

TaskTime Outcome#Path

DeviationsTask

SatisfactionP0101 1:54 Success 1 4P0102 1:23 Success 0 3P0103 2:08 Success 0 5P0104 2:42 Success 2 2P0105 1:06 Success 0 5P0106 1:45 Success 0 5P0107 1:47 Success 0 5P0108 0:20 Success 1 4P0109 3:48 Success 1 4P0110 1:18 Success 1 5

ExpectedTimeonTask 2:00 (SD) AverageTimeonTask 1:49 0:54 AverageTaskSatisfaction 4.20 0.98 Average#PathDeviations 0.60 0.66 PercentSuccess 100%

ParticipantNumber

Task2ProblemListView/Intervention

DeviationsTask

ExpectedTimeonTask 0:30 (SD)

AverageTimeonTask 0:42 0:10 AverageTaskSatisfaction 4.20 0.75 Average#PathDeviations 1.10 0.70 PercentSuccess 100%

ParticipantNumber

Task3ProblemListIntervention-Remove

DeviationsTask

ParticipantNumber

Task4ClinicalInfoRecon/Incorporation

DeviationsTask

SatisfactionP0101 4:23 Success 3 1P0102 1:50 Success 1 4P0103 3:15 Fail 3 1P0104 2:09 Fail 6 1P0105 1:17 Success 0 4P0106 0:44 Success 0 5P0107 1:16 Success 0 4P0108 1:42 Success 2 3P0109 2:06 Fail 3 4P0110 1:31 Success 1 4

ParticipantNumber

Task5MedicationAllergy-Add

DeviationsTask

SatisfactionP0101 1:20 Success 1 4P0102 1:48 Success 2 3P0103 1:13 Success 1 5P0104 2:29 Success 3 2P0105 0:58 Success 1 5P0106 1:33 Success 1 2P0107 1:38 Success 0 4P0108 1:31 Success 1 3P0109 3:20 FAIL 6 3P0110 1:24 Success 1 4

ParticipantNumber

Task6Drug-AllergyInteractionCheck

DeviationsTask

ParticipantNumber

Task7ElectronicPrescribe

DeviationsTask

ParticipantNumber

Task8eRXCancel/Change

DeviationsTask

SatisfactionP0101 2:28 Success 0 3P0102 3:48 Fail 6 3P0103 2:13 Success 0 5P0104 3:26 Success 0 2P0105 1:43 Success 0 4P0106 3:27 Success 1 4P0107 2:25 Success 0 5P0108 2:23 Success 0 4P0109 3:15 Success 3 5P0110 3:32 Success 1 4

ParticipantNumber

Task9eRX-Renew

DeviationsTask

ParticipantNumber

Task10MedOrder;Drug-DrugIntervention

DeviationsTask

ParticipantNumber

Task11Medicationlist-Stop

DeviationsTask

ParticipantNumber

Task12MedAllergyList-Remove

DeviationsTask

ParticipantNumber

Task13CPOEDiagImaging

DeviationsTask

ParticipantNumber

Task14CPOEDiagImaging-Modify

DeviationsTask

ExpectedTimeonTask 1:00 (SD) AverageTimeonTask 1:18 0:21 AverageTaskSatisfaction 4.30 0.64 Average#PathDeviations 0.60 0.66 PercentSuccess 100%

ParticipantNumber

Task15CPOELabOrders

DeviationsTask

ParticipantNumber

Task16CPOELabOrders-Modify

DeviationsTask

ParticipantNumber

Task17ImplantableDevice-Add

DeviationsTask

ParticipantNumber

Task18ImplantableDevice-Modify

DeviationsTask

ehr usability test report product: chirotouchsales@chirotouch.com introduction the ehr tested for...

Documents

ehr usability test report of cloudmd365 version 1a usability...

ehr usability toolkit: a background report on usability...

himss ehr usability task force report on defining and...

ehr usability test report of advancedmd version 12 · ehr...

ehr usability toolkit: a background report on usability and

usability is the key to stimulating ehr innovation and

ehr usability test report for epiccare inpatient ehr...

ehr usability test report -revised€¦ · report based on...

turf a unified framework for ehr usability jiajie zhang...

ehr usability test report for practicesuite ehr...

ehr usability test report of exscribe ehr (version 6.0)

a human factors guide to enhance ehr usability of critical...

ehr usability style guide...ehr usability style guide •...

ehr usability test report of continuum 2

ehr usability test report of srs ehr v10 user centered...

ehr usability test report of e-medsys ehr version 8 ·...

sharpc’s 3rd annual pre-amia symposium, the ehr usability...

assessing ehr implementation decisions to improve...

ehr usability test report of medinformatix ehr v7 · ehr...

ehr usability test report of greenway intergy ehr 11 · a...