ehr usability test r emr 7 - drummond group · records, exchange health information between...

8000 SW 117 Ave PH-C | Miami, Florida 33183

EHR USABILITY TEST REPORT OF SMARTEMR 7.0

REPORT BASED ON ISO/IEC 25062:2006 COMMON INDUSTRY FORMAT FOR USABILITY TEST REPORTS

SMARTEMR 7.0

Date of Usability Test: June 2018

Date of Report: June 28, 2018

Report Prepared By: Vipa Health Solutions

2

EHR Usability Test Report of smartEMR 7.0 1

smartEMR 7.0 1

Executive summary 3

Introduction 7

Method 7

Participants 7

Study Design 8

Tasks 9

Procedures 11

Test Location 11

Remote Testing 11

Healthcare Organization in South Florida 12

Test Environment 12

Remote 12

Physical 12

For both Scenarios 12

Test Forms and Tools 12

Participant Instructions 13

Usability Metrics 13

Data Scoring 13

Results 14

Data Analysis and reporting 14

Discussion of Findings 19

Effectiveness 19

Efficiency 21

Satisfaction 21

Major Findings 21

Areas for Improvement 21

Appendices 22

3

EXECUTIVE SUMMARY

A usability test of the newest version of smartEMR 7.0 ambulatory EHR was conducted from June 4 to June 29 in Miami, Florida. The purpose of this study was to test and validate the usability of the current user interface and provide evidence of usability in the EHR Under Test (EHRUT).

During the usability test, fifteen healthcare providers, medical assistants, and administrators matching the target demographic criteria served as participants and used the EHRUT in simulated, but representative tasks.

• Task 1: a.1 - CPOE Medications

• Task 2: a.2 - CPOE Labs

• Task 3: a.3 - CPOE Diagnostic Imaging

• Task 4: a.4 - Drug-drug

• Task 5: a.5 - Demographics

• Task 6: a.6 - Problem List

• Task 7: a.7 - Medication List

• Task 8: a.8 - Medication Allergy List

• Task 9: a.9 - Clinical Decision Support

• Task 10: a.14 - Implantable Device List

• Task 11: b.2 - Clinical Information Reconciliation

• Task 12: b.3 - Electronic Prescribing

During the 60-minute one-on-one usability test, each participant was greeted by the administrator and asked to review and sign an informed consent/release form (included in Appendix 3). Participants were instructed that they could withdraw at any time.

Participants had prior experience with EHR systems. The administrator introduced the test and instructed participants to complete a series of tasks (given one at a time). During the examination, the administrator timed the test and, along with the data logger recorded user performance data on paper and electronically. The administrator did not give the participants assistance on how to complete the task.

The EHRUT tested for this study was smartEMR 7.0 Designed to present medical information to clinicians in an ambulatory setting. EHRUT consists of several modules to capture patient medical records, exchange health information between stakeholders and suggested quality improvement based on patient-specific data. The test was limited to the tasks explained above. The usability testing attempted to represent realistic exercises and conditions.

4

The purpose of this study was to test and validate the usability of the current user interface and provide evidence of ease of use in the EHR Under Test (EHRUT).

To this end, measures of effectiveness, efficiency and user satisfaction, such as time on task, ability to locate information, and ease of executing the task were captured during the usability testing.

Participant screens, headshots, and audio were recorded for subsequent analysis.

The following types of data were collected for each participant:

● Number of tasks completed within the allotted time without assistance

● Time to complete the tasks

● Number and types of errors

● Path deviations

● Participant's verbalizations

● Participant's satisfaction ratings of the system

All participant data were de-identified, no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire; participants were not compensated for their time, as participation was voluntary. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT

Table 2

N Task Time

(success) Path Deviation Task Time Errors Task Ratings

(1=Easy)

Module TASK LIST # Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a1 Prescribe new medication order 10 0.6 0.52 3 3 39.6 22.73 0.2 0.42 1 0

a1 Update Medication Order (adjust dosage) 10 0.8 0.42 3.1 3 18.2 9.75 0.3 0.48 1 0

a2 Access lab order module 10 0.9 0.32 1.3 3 7.8 3.68 0 0 2.1 1.45

a2 Record a new Lab Order with diagnosis 10 0.4 0.52 3.8 6 39.4 18.92 0.4 0.52 2.1 1.45

a2 Route lab order 10 0.5 0.53 2.8 2 32.8 24.39 0.2 0.42 2.1 1.45

a2 Update and route existing Lab Order 10 0.9 0.32 8.1 8 39 15.3 0 0 2.1 1.45

a3 Access diagnostic imaging order module 10 1 0 1 1 5.1 1.97 0 0 1.3 0.67

a3 Record a new Diagnostic Imaging Order and

include diagnosis 10 0.9 0.32 4.9 6 32.6 15.58 0.1 0.32 1.3 0.67

5

a3 Route current order 10 1 0 1.1 1 5.8 2.44 0 0 1.3 0.67

a4 Access d2d/d2a module 10 0.6 0.52 1.2 1 13.9 13.74 0.3 0.48 1.3 0.67

a4 Configure severity level and interaction

notification of d2d/d2a interventions (admin) 10 0.6 0.52 5.2 6 23.5 12.57 0.6 0.97 1.3 0.67

a4 Access Med module in current visit 10 1 0 1 1 4.6 1.84 0 0 1.7 0.95

a4 Prescribe, Trigger and acknowledge d2d

intervention 10 1 0 1.4 1 28.4 9.78 0 0 1.7 0.95

a4 Prescribe, Trigger and acknowledge d2a

intervention 10 0.4 0.52 1.8 1 43 22.09 0.3 0.48 1.7 0.95

a5 Access smartEMR clinical 10 0.8 0.42 1.1 1 35.3 18.35 0 0 1.6 0.84

a5 Access and Record New Patient on demographics

module 10 0.4 0.52 2.4 2 82.5 47.78 0.3 0.48 1.6 0.84

a5 Update patient demographics information 10 0.5 0.53 3.1 3 29.9 9.29 0.4 0.52 1.6 0.84

a6 Access problem list within an encounter (Hx) 10 0.7 0.48 3.5 3 25.9 14.11 0 0 2.4 1.71

a6 Access and record a new active problem 10 0.5 0.53 5.5 6 51 32.17 0.5 0.85 2.4 1.71

a6 Update an existing problem 10 0.4 0.52 3.5 5 43.2 22.87 0.6 0.52 2.4 1.71

a7 Access hx Medication List 10 0.8 0.42 1.9 1 16.8 24.11 0.2 0.42 1.5 0.71

a7 Access and Document a new medication on active

medication module (pill) 10 0.2 0.42 3.4 3 36.1 24.04 0.7 0.48 1.5 0.71

a7 Document a new liquid/gel/patch medication

(non-pill) 10 0.3 0.48 3.9 4 57.2 20.9 0.7 0.48 1.5 0.71

a7 Update an existing medication (DC) 10 0.9 0.32 2.1 2 14.9 12.97 0 0 1.5 0.71

a8 Access allergy module (hx) 10 0.7 0.48 1.5 1 14.5 18.78 0 0 2.5 1.43

a8 Access and record a new active medication allergy 10 0.8 0.42 2.1 2 25 26.91 0.2 0.42 2.5 1.43

a8 Update medication allergy -add a manifestation 10 0.2 0.42 1.8 3 35.5 33.8 0.8 0.42 2.5 1.43

a9 Access admin console 10 0.9 0.32 1 1 23 13.5 0 0 1.3 0.67

a9 Access CDS management module 10 1 0 1 1 5.8 2.53 0 0 1.3 0.67

a9 Configure a CDS intervention and/or reference

resource for each of the required elements 10 0.8 0.42 14.7 15 25.8 8.2 0.1 0.32 1.3 0.67

6

(admin)

a9 Access and trigger CDS interventions 10 0.8 0.42 1 1 29.2 16.61 0.2 0.42 1.7 1.16

a9 Check for CDS interventions generated by

reconciled C-CDA data 10 1 0 1.3 2 13.4 3.63 0.3 0.48 1.7 1.16

a9 Access reference information for a CDS

intervention (source info) 10 0.8 0.42 0.9 1 36.4 26.7 0.1 0.32 1.7 1.16

a9 Identify the guideline for a CDS intervention

(therapeutic info) 10 0.8 0.42 0.9 1 33.1 15.83 0.2 0.42 1.7 1.16

a9 Access reference information (info-button) 10 0.7 0.48 1 1 26.4 12.95 0.3 0.48 1.7 1.16

a14 Access implantable device list 10 0.5 0.53 1.1 1 18.6 18.14 0.2 0.42 2.1 1.37

a14 Record a new implantable device 10 0.4 0.52 2.9 3 54 26.1 0.6 0.52 2.1 1.37

a14 Update implantable device list 10 0.8 0.42 1.8 2 10.7 7.47 0.2 0.42 2.1 1.37

b2 Access Patient's Documents and locate C-CDA 10 0.5 0.53 3.3 3 25 17.39 0.5 0.53 1.7 0.95

b2 Reconcile patient’s current medication allergy list

with external source 10 0.4 0.52 1.5 2 11.5 3.44 0.6 0.52 1.7 0.95

b2 Reconcile patient’s current medication list with

external source 10 0.6 0.52 2.2 3 10.1 6.92 0.4 0.52 1.7 0.95

b2 Reconcile patient’s active problem list with

external source 10 0.6 0.52 2.2 3 7.2 2.15 0.4 0.52 1.7 0.95

b3 Create new prescription

10 0.6 0.52 3 3 39.6 22.73 0.2 0.42 1 0

b3 Access and Update pharmacy information

10 0.6 0.52 5 5 46.8 23.04 0.1 0.32 1.5 0.85

b3 Route a medication to a pharmacy

10 1 0 3.4 3 40.5 14.09 0.4 0.7 1.5 0.85

b3 Change prescription (dosage or duration)

10 0.8 0.42 3.1 3 18.2 9.75 0.3 0.48 1 0

b3 Cancel prescription

10 0.90 0.32 2.3 2 10.30 3.80 0.30 0.48 1 0.00

b3 Request and receive medication history

information

10 0.90 0.32 1.3 1 19.20 5.98 0.20 0.42 1.2 0.42

b3 Receive fill status notification

10 0.70 0.48 4.8 4 39.60 22.73 0.80 0.79 1.2 0.42

b3 Refill prescription

10 0.90 0.32 1.5 1 10.50 5.54 0.60 0.84 1.2 0.42

7

The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be: 86.00

In addition to the performance data, the following qualitative observations were made:

-Major findings

● Drug-to-Allergy interaction alert demised by drug-to-drug interaction alert when both are triggered simultaneously.

● Adding a link to add a new pharmacy or update the current pharmacy information for the prescriber workflow.

-Areas for improvement

● Participants identify opportunities for improvement in several areas. The need to streamline and optimized workflows and make information more accessible to the user when it is needed. Refinements to the interface and load times were also mentioned by participants.

INTRODUCTION

The EHRUT for this study was smartEMR version 7.0. This system is designed to present medical information to clinicians in an ambulatory setting. smartEMR consists of several modules to record and retrieve patient health data. The usability testing attempted to represent realistic exercises and conditions.

The purpose of this study was to test and validate the usability of the smartEMR system's user interface, provide evidence of usability and feedback on user experience. To this end, measures of effectiveness, efficiency and user satisfaction, such as time on task and success/failure ratings, were captured during the usability testing.

METHOD

PARTICIPANTS

A total of ten users were tested. Participants were doctors, medical assistants, and administrators. Participants were recruited by Vipa Health support staff members and were not compensated for their time. Participants had the opportunity to receive a brief orientation before testing.

Recruited participants had a mix of backgrounds and demographic characteristics conforming to the recruitment screener. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. All names were replaced with participant identifiers so that an individual's data cannot be tied back to individual identities.

8

Table 3

Participant

ID Participant

Gender Participant

Age Participant

Education Participant

Occupation/Role

Participant

Professional

Experience

Participant

Computer

Experience

Participant

Product

Experience

Participant

Assistive

Technology

Needs

P01 Female 50-59 Master's Degree Administrative

(front office) 120 120 96 No

P02 Female 40-49

High school

graduate, diploma or

the equivalent (for

example: GED) Administrative

(back office) 120 120 12 No

P03 Female 50-59 Trade/technical/voc

ational training Administrative

(back office) 108 48 60 No

P04 Female 30-39 Trade/technical/voc

ational training Other 120 12 12 No

P05 Female 50-59

High school

graduate, diploma or

the equivalent (for

example: GED) Administrative

Staff 120 120 24 No

P06 Male 60-69

Doctorate degree

(e.g., MD, DNP,

DMD, PhD) Physician 120 120 120 No

P07 Female 60-69

Doctorate degree

(e.g., MD, DNP,


P08 Male 70-79

Doctorate degree

(e.g., MD, DNP,


P09 Male 50-59

Doctorate degree

(e.g., MD, DNP,


P10 Male 40-49

Doctorate degree

(e.g., MD, DNP,


A total of ten users were signed up to participate. The examination took place during several weeks, testing one or two users per day. Each participant was scheduled for a fifty-minute session with fifteen minutes in between each session for debriefing by the administrator and data logger, and to reset the system to proper test conditions. A spreadsheet was used to keep track of the users scheduled and included demographic components.

STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well, In other words, it was useful, efficient, and with satisfaction. As well as areas where the application failed to meet the needs of the users. The data from this test may serve as a baseline for future testing once a new

9

version of the same EHR is available and also for comparison with other EHRs provided the same tasks are performed. In short, this testing serves as both a means to record or benchmark current usability, but also to identify areas where improvements must be made.

During the test, participants interacted with one EHR, smartEMR version 7.0. Each user worked the system in different physical locations and some choose to participate remotely connecting via a web conference system.

The system was evaluated for effectiveness, efficiency, and satisfaction as defined by measures collected and analyzed for each participant:

● Number of tasks completed within the allotted time without assistance

● Time to complete the tasks

● Number and types of errors

● Path deviations

● Participant's verbalizations (comments)

● Participant's satisfaction ratings of the system

TASKS

Several tasks and scenarios were constructed using synthetic data. A realistic representation of the types of activities a clinical user performs in an EHR system was intended.

Table 4

TEST ORDER Task ID Task Description Criteria

1 t.13.1 Access Admin Console a9

2 t.13.2 Access CDS management module a9

3 t.13.3 Configure CDS intervention rules (admin) a9

1 t.13.4 Access d2d/d2a module a4

5 t.13.5 Configure severity level and interaction notification of d2d/d2a interventions (admin) a4

6 t.5.1 Access smartEMR clinical a5

7 t.5.2 Access and Record New Patient on demographics module a5

8 t.5.3 Update patient demographics information a5

9 t.8.1 Access Allergy Module (Hx) a8

10 t.8.2 Access and record new active Medication Allergy a8

11 t.8.3 Update Medication Allergy - add a manifestation a8

12 t.10.1 Access implantable device list a14

13 t.10.2 Record a new implantable device a14

14 t.10.3 Update implantable device list a14

10

15 t.11.1 Access Patient's Documents and locate C-CDA b2

16 t.11.2 Reconcile patient’s current medication allergy list with external source b2

17 t.11.3 Reconcile patient’s current medication list with external source b2

18 t.11.4 Reconcile patient’s active problem list with external source b2

19 t.6.1 Access problem list module within an encounter (Hx) a6

20 t.6.2 Access and record a new active problem a6

21 t.6.3 Update an existing problem a6

22 t.7.1 Access Medication List (Hx) a7

23 t.7.2 Access and document a new active medication (pill) a7

25 t.7.3 Document a new liquid/gel/patch medication (non-pill) a7

24 t.7.4 Update an existing medication (DC) a7

26 t.4.1 Access Med module in current visit a4

27 t.4.2 Prescribe, Trigger and acknowledge drug-to-drug intervention a4

28 t.4.3 Prescribe, Trigger and acknowledge drug-to-allergy intervention a4

29 t.1.1 Prescribe New Medication Order a1

30 t.1.2 Update Medication Order (adjust dosage) a1

31 t.12.1 Create new prescription b3

32 t.12.2 Access and Update pharmacy information b3

33 t.12.3 Route a medication to a pharmacy b3

34 t.12.4 Change prescription (dosage or duration) b3

35 t.12.5 Cancel prescription b3

36 t.12.6 Request and receive medication history information b3

37 t.12.7 Receive fill status notification b3

38 t.12.8 Refill prescription b3

39 t.2.1 Access Lab Order Module a2

40 t.2.2 Record a new Lab Order with diagnosis a2

41 t.2.3 Route Lab Order a2

42 t.2.4 Update and route existing Lab Order a2

43 t.3.1 Access Diagnostic Imaging Order Module a3

44 t.3.2 Record a new Diagnostic Imaging Order and include diagnosis a3

45 t.3.3 Route current order a3

11

46 t.9.1 Access and trigger CDS interventions a9

47 t.9.2 Check for CDS interventions generated by reconciled C-CDA data a9

48 t.9.3 Access reference information for a CDS intervention (source info) a9

49 t.9.4 Identify the guideline for a CDS intervention (therapeutic info) a9

50 t.9.5 Access reference information (info-button) a9

Tasks were selected based on their frequency of use, criticality of function, and those that may be most troublesome for users.

PROCEDURES

Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a participant identification number. Each participant reviewed and signed informed consent and a release form. A member of the testing team witnessed the participant's signature.

To ensure that the test ran smoothly, two staff members participated in this test, the usability administrator, and the data logger. Both team members had experience conducting this type of user testing

The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. A second person served as the data logger and took notes on task success, path deviations, number and type of errors, and comments.

Participants were instructed to perform the tasks (see specific instructions below):

● As quickly as possible making as few errors and deviations as possible.

● Without assistance; administrators were allowed to give immaterial guidance and clarification on tasks, but no instructions on use.

● Without using a think-aloud technique. For each task, the user was given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had completed the task. Scoring is discussed below in Section 3.9.

Following the session, the administrator gave the participant a post-test questionnaire (see Appendix 5) and thanked each for their time.

Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet.

TEST LOCATION

REMOTE TESTING

12

Usability tests were conducted remotely over a web conference technology. Each participant joined the conference from their location. Both members of the testing team were connected to the participant’s screen and were listening to the audio of the session. Participants were asked to join from a location that was relatively quiet and free from distraction.

HEALTHCARE ORGANIZATION IN SOUTH FLORIDA

The test facility included a waiting area and a quiet conference room provided by the medical practice. The room was equipped with a table, a computer for the participant, and a recording computer for the administrator. Participants at this location were employees of the organization. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a reasonable range.

TEST ENVIRONMENT

REMOTE

This usability testing was conducted remotely, with participants interacting with the smartEMR version 7.0 via Cisco WebEx Meetings, a video conferencing software. Using remote testing allowed the participants to use the EHRUT from their usual office location. Participants were given instructions on how to access the meeting online. Once in the meeting, they were passed keyboard and mouse control of the test system by one of the testing team members.

PHYSICAL

The EHRUT would typically be used in a physician's office. In this instance, the testing was conducted in a conference room. The computer utilized for testing was a MacBook Pro running OS X version 10.11.5. The participants used a mouse and attached keyboard when interacting with the application. SmartEMR version 7.0 was displayed on a fifteen inches screen on 1920x1200 resolution in full range of colors.

FOR BOTH SCENARIOS

The application was set up by the vendor for testing purposes. The EHR system is web-based and was running on Mozilla Firefox using a production database with a staging web server. Technically, the system performance (i.e., response time) was representative of what actual users would experience in a real-life scenario. Additionally, participants were instructed not to change any of the default system settings (such as control of font size).

TEST FORMS AND TOOLS

During the testing, various documents and instruments were used, including:

● Informed Consent

● Moderator's Guide

● Post-test Questionnaire Examples of these documents can be found in Appendices 3-5 respectively. The Moderator's Guide was devised so as to be able to capture required data.

The participant's interaction with the EHRUT was captured and recorded digitally with screen capture software running on the test machine for the physical locations and the administrator computer using the

13

web-conference tools for the remote sessions. An embedded digital web camera recorded each participant's facial expressions synced with the screen capture, and verbal comments were recorded with a microphone.

PARTICIPANT INSTRUCTIONS

The administrator read the instructions aloud to each participant (moderator's guide in Appendix 4). Following the procedural instructions, participants were shown smartEMR version 7.0 and as their first task, each participant was given five minutes to explore the system and make comments. Once the time was up, the administrator proceeds with the remaining tasks. All Participants were given thirteen clinical or administrative tasks to complete. Tasks are listed in detail in Appendix 6.

USABILITY METRICS

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing.

The goals of the test were to assess:

1. The effectiveness of smartEMR by measuring participant success rates and errors 2. The efficiency of smartEMR by measuring the average task time and path deviations 3. Satisfaction with smartEMR by measuring ease of use ratings

DATA SCORING

The following table details how tasks were scored, errors evaluated, and the time data analyzed.

Table 5

Rationale and Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time

allotted on a per task basis.

The total number of successes were calculated for each task and then divided by the total number of times that task was

attempted. The results are provided as a percentage.

Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal

efficiency.

Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when

constructing tasks. Target task times used for task times in the Moderator’s Guide must be operationally defined by taking

multiple measures of optimal performance and multiplying a factor of 1.5 that allows some time buffer because the participants

are presumably not trained to expert performance. Thus, if expert, optimal performance on a task was 24 seconds then allotted

task time performance was 36 seconds. This ratio should be aggregated across tasks and reported with mean and variance

scores.

14

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the

allotted time before successful completion, the task was counted as an “Failures.” No task times were taken for errors.

The total number of errors was calculated for each task and then divided by the total number of times that task was attempted.

Not all deviations would be counted as errors. This should also be expressed as the mean number of failed tasks per participant.

On a qualitative level, an enumeration of errors and error types should be collected.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application was recorded.

Deviations occur if the participant, for example, went to a wrong screen, clicked on an incorrect menu item, followed an

incorrect link, or interacted incorrectly with an on-screen control. This path was compared to the optimal path. The number of

steps in the observed path is divided by the number of optimal steps to provide a ratio of path deviation.

It is strongly recommended that task deviations be reported. Optimal paths (i.e., procedural steps) should be recorded when

constructing tasks.

Efficiency:

Task Time

Each task was timed from when the administrator said “Begin” until the participant said, “Done.” If he or she failed to say

“Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successful ly

completed were included in the average task time analysis. Average time per task was calculated for each task. Variance

measures (standard deviation and standard error) were also calculated.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task

question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a

scale of 1 (Very Easy) to 5 (Very Difficult). These data are averaged across participants.

Common convention is that average ratings for systems judged easy to use should be 3.3 or below.

To measure participants’ confidence in and likeability of smartEMR 7.0 overall, the testing team administered the System

Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought

the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See full System

Usability Scale questionnaire.

RESULTS

DATA ANALYSIS AND REPORTING

The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses.

The usability testing results for the EHRUT are detailed below (see Table 6). The results should be seen in the light of the objectives and goals outlined in section 3.2 Study Design. The data should yield actionable results that, if corrected, yield material, positive impact on user performance.

15

Table 6 - CPOE – Medications (170.315 (a)(1)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a1 Prescribe New Medication Order 10 0.6 0.52 3 3 39.6 22.73 0.2 0.42 1 0

a1 Update Medication Order (Adjust Dosage) 10 0.8 0.42 3.1 3 18.2 9.75 0.3 0.48 1 0

Table 7 - CPOE – Laboratory (170.315 (a)(2)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a2 Access Lab Order Module 10 0.9 0.32 1.3 3 7.8 3.68 0 0 2.1 1.45

a2 Record A New Lab Order with Diagnosis 10 0.4 0.52 3.8 6 39.4 18.92 0.4 0.52 2.1 1.45

a2 Route Lab Order 10 0.5 0.53 2.8 2 32.8 24.39 0.2 0.42 2.1 1.45

a2 Update and Route Existing Lab Order 10 0.9 0.32 8.1 8 39 15.3 0 0 2.1 1.45

Table 8- CPOE – Diagnostic Imaging (170.315 (a)(3)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a3 Access Diagnostic Imaging Order Module 10 1 0 1 1 5.1 1.97 0 0 1.3 0.67

a3 Record a New Diagnostic Imaging Order and

Include Diagnosis 10 0.9 0.32 4.9 6 32.6 15.58 0.1 0.32 1.3 0.67

a3 Route Current Order 10 1 0 1.1 1 5.8 2.44 0 0 1.3 0.67

16

Table 9 - Drug-drug, Drug-allergy Interaction 170.315 (a)(4)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a4 Access D2D/D2A Module 10 0.6 0.52 1.2 1 13.9 13.74 0.3 0.48 1.3 0.67

a4 Configure Severity Level and Interaction

Notification of D2D/D2A Interventions (Admin) 10 0.6 0.52 5.2 6 23.5 12.57 0.6 0.97 1.3 0.67

a4 Access Med Module in Current Visit 10 1 0 1 1 4.6 1.84 0 0 1.7 0.95

a4 Prescribe, Trigger and Acknowledge Drug-To-

Drug Intervention 10 1 0 1.4 1 28.4 9.78 0 0 1.7 0.95

a4 Prescribe, Trigger and Acknowledge Drug-To-

Allergy Intervention 10 0.4 0.52 1.8 1 43 22.09 0.3 0.48 1.7 0.95

Table 10 - CPOE – Demographics 170.315(a)(5)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a5 Access SmartEMR Clinical 10 0.8 0.42 1.1 1 35.3 18.35 0 0 1.6 0.84

a5 Access and Record New Patient on

Demographics Module 10 0.4 0.52 2.4 2 82.5 47.78 0.3 0.48 1.6 0.84

a5 Update Patient Demographics Information 10 0.5 0.53 3.1 3 29.9 9.29 0.4 0.52 1.6 0.84

Table 11 - Problem List 170.315 (a)(6)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a6 Access Problem List Module (Hx) 10 0.7 0.48 3.5 3 25.9 14.11 0 0 2.4 1.71

a6 Access and Record a New Active Problem 10 0.5 0.53 5.5 6 51 32.17 0.5 0.85 2.4 1.71

a6 Update an Existing Problem 10 0.4 0.52 3.5 5 43.2 22.87 0.6 0.52 2.4 1.71

17

Table 11 - Medication List 170.315 (a)(7)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a7 Access Medication List (Hx) 10 0.8 0.42 1.9 1 16.8 24.11 0.2 0.42 1.5 0.71

a7 Access and Document a New Active

Medication (Pill) 10 0.2 0.42 3.4 3 36.1 24.04 0.7 0.48 1.5 0.71

a7 Document A New Liquid/Gel/Patch Medication

(Non-Pill) 10 0.3 0.48 3.9 4 57.2 20.9 0.7 0.48 1.5 0.71

a7 Update an Existing Medication (DC) 10 0.9 0.32 2.1 2 14.9 12.97 0 0 1.5 0.71

Table 12 - Medication Allergy List 170.315 (a)(8)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a8 Access Allergy Module Hx 10 0.7 0.48 1.5 1 14.5 18.78 0 0 2.5 1.43

a8 Access and Record A New Active Medication

Allergy 10 0.8 0.42 2.1 2 25 26.91 0.2 0.42 2.5 1.43

a8 Update Medication Allergy -Add a

Manifestation 10 0.2 0.42 1.8 3 35.5 33.8 0.8 0.42 2.5 1.43

Table 13 - Clinical Decision Support 170.315 (a)(9)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a9 Access Admin Console 10 0.9 0.32 1 1 23 13.5 0 0 1.3 0.67

a9 Access CDS Management Module 10 1 0 1 1 5.8 2.53 0 0 1.3 0.67

a9 Configure CDS Rules (Admin) 10 0.8 0.42 14.7 15 25.8 8.2 0.1 0.32 1.3 0.67

18

a9 Access and Trigger CDS Interventions 10 0.8 0.42 1 1 29.2 16.61 0.2 0.42 1.7 1.16

a9 Check for CDS Interventions Generated by

Reconciled C-CDA Data 10 1 0 1.3 2 13.4 3.63 0.3 0.48 1.7 1.16

a9 Access Reference Information for A CDS

Intervention (Source Info) 10 0.8 0.42 0.9 1 36.4 26.7 0.1 0.32 1.7 1.16

a9 Identify the Guideline for A CDS Intervention

(Therapeutic Info) 10 0.8 0.42 0.9 1 33.1 15.83 0.2 0.42 1.7 1.16

a9 Access Reference Information (Info-Button) 10 0.7 0.48 1 1 26.4 12.95 0.3 0.48 1.7 1.16

Table 14 - Implantable Devices 170.315 (a)(14)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

a14 Access Implantable Device List 10 0.5 0.53 1.1 1 18.6 18.14 0.2 0.42 2.1 1.37

a14 Record a New Implantable Device 10 0.4 0.52 2.9 3 54 26.1 0.6 0.52 2.1 1.37

a14 Update Implantable Device List 10 0.8 0.42 1.8 2 10.7 7.47 0.2 0.42 2.1 1.37

Table 15 - Clinical Information Reconciliation and Incorporation 170.315(b)(2)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

b2 Access Patient's Documents and Locate C-CDA 10 0.5 0.53 3.3 3 25 17.39 0.5 0.53 1.7 0.95

b2 Reconcile Patient’s Current Medication Allergy

List with External Source 10 0.4 0.52 1.5 2 11.5 3.44 0.6 0.52 1.7 0.95

b2 Reconcile Patient’s Current Medication List

with External Source 10 0.6 0.52 2.2 3 10.1 6.92 0.4 0.52 1.7 0.95

b2 Reconcile Patient’s Active Problem List with

External Source 10 0.6 0.52 2.2 3 7.2 2.15 0.4 0.52 1.7 0.95

19

Table 16 - Electronic Prescribing170.315 (b)(3)

N Task Time


(1=Easy)


(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD Mean

(%) SD

b3 Create New Prescription 10 1 0 3.4 3 40.5 14.09 0.4 0.7 1 0

b3 Access and Update Pharmacy Information 10 0.6 0.52 5 5 46.8 23.04 0.1 0.32 1.5 0.85

b3 Route A Medication to a Pharmacy 10 1 0 3.4 3 40.5 14.09 0.4 0.7 1.5 0.85

b3 Change Prescription (Dosage Or Duration) 10 1 0 3.4 3 40.05 14.09 0.30 0.48 1 0.00

b3 Cancel Prescription 10 0.90 0.32 2.3 2 10.30 3.80 0.30 0.48 1 0.00

b3 Request and Receive Medication History

Information (PBM)

10 0.90 0.32 1.3 1 19.20 5.98 0.20 0.42 1.2 0.42

b3 Receive Fill Status Notification (Refill

Request)

10 0.70 0.48 4.8 4 39.60 22.73 0.80 0.79 1.2 0.42

b3 Refill Prescription 10 0.90 0.32 1.5 1 10.50 5.54 0.60 0.84 1.2 0.42

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be: 86.00. Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.

Mean P01 P02 P03 P04 P05 P06 P07 P08 P09 P10

SUS RAW SCORE 34.4 40 29 33 32 40 37 36 33 38 26

SUS FINAL SCORE 86 100 72.5 82.5 80 100 92.5 90 82.5 95 65

DISCUSSION OF FINDINGS

EFFECTIVENESS

• a.1 - CPOE Medications, all participants were able to order a medication using the CPOE with minor deviations from the optimal path. All participants were able to update an active medication, minor deviations were also observed on this task.

20

• a.2 - CPOE Laboratory, all participants were able to access and edit and existing lab order. Though, some participants especially the users who perform administration functions, had difficulty adding a diagnosis to a new lab order and routing it correctly. In most cases, users neglect to click on the save button that will attached the diagnosis to the order.

• a.3 - CPOE Diagnostic Imaging, all participants were successful at completing all tasks associated with this module.

• a.4 - Drug-drug and drug-allergy interaction, all participants were able to complete this task without major deviations. However, most users agreed that drug-allergy interaction should take precedence over drug-drug interactions when displaying notifications. Also, the admin tasks were quickly performed by non-clinical users, while clinical users found it irrelevant.

• a.5 – Demographics, all participants were able to complete this task with minor deviations from the optional path.

• a.6 - Problem List, all participants were able to complete this task with minor deviations from the optional path.

• a.7 - Medication List, all participants were able to complete this task with some deviations. This is a tool that users access constantly and therefore have their own path for reaching and achieving the task. Most of the users did not follow the optimal path to complete the task; however, all tasks associated with this module were completed successfully.

• a.8 - Medication Allergy List, participants were able to complete most tasks with ease. Most participants had difficulty adding a manifestation to an active medication allergy. Participants found the task cumbersome and opted for deleting the existing medication allergy and creating a new entry with all accurate information. However, since this workflow was a major deviation from the original path, we did not count it as successful.

• a.9 - Clinical Decision Support, participants were successful at completing all tasks associated with this module with ease.

• a.14 - Implantable Device List, this module in their current state was new to most participants. It was a good opportunity to assess the ease of use and the how intuitive was this module to work with. The module was intuitive enough that most users found their way to achieve the task, but some minor deviations occurred. All participants were able to complete this task

• b.2 - Clinical Information Reconciliation, most participants were able to execute this task. Users found the tool convenient and easy to use. However, CDAs were not easy to access because there is only one path to reach reconciliation documents. Also, users were concerned with the quality of the information obtained via electronic exchange.

• b.3 - Electronic Prescribing, all participants were able to send an electronic prescription to a pharmacy. Picking between two or more pharmacies and sending prescriptions to different pharmacies were completed with ease by most participants (e.g., mail-order for medications for chronic conditions, and local retail pharmacy for short-term antibiotics). Users experienced a setback when updating pharmacy information because it required the user to exit their current workflow, yet most users were successful at accomplishing this task with minor deviations. All

21

participants were able to update and cancel a current prescription without difficulties. Users were able to acknowledge a refill request notification and were successful in acting on the information to fill the prescription. Users had no significant delays or interferences at requesting medication history from PBM and visualizing the information.

EFFICIENCY

● Participants completed all tasks promptly. Some participants clicked on different links or had minor deviations from the optimal path but were able to find their way back to the original task or found alternative ways to complete the task. The system allows for different workflows to achieve a task, and users look for familiar ways to complete the task at hand. In all, participants were able to complete all tasks successfully with additional clicks.

SATISFACTION

● Overall the users were satisfied and content with the system. For new users, the system was intuitive enough, and most were able to complete the tasks with ease while the experience users were able to navigate the system quickly and without significant interruptions.

MAJOR FINDINGS

● During the usability test two main issues were found. When prescribing a medication that has triggered both drug to drug and drug to allergy interactions, we discovered that the drug to drug notification warning was taking precedence while the drug to allergy notification warning was displayed to the user in a less noticeable manner. We found this to be a major red flag and took action to corrected.

● The second major finding was related to the selection of a pharmacy when prescribing a new medication electronically, if there was not pharmacy assigned to the patient, then the prescribing user had to interrupt their current workflow and add a pharmacy on a different screen. We found this inconvenient and we suggested to add a link to the pharmacy selection module directly from the e-prescriber workflow.

AREAS FOR IMPROVEMENT

● Modules associated with CDA reconciliation need to be incorporated into the workflow efficiently. System feedback upon completing reconciliation.

● The EHR has two sections for medication favorites, the system learns the most commonly prescribed medications by a user and displays a top 50 most frequently used. The second section, the user selects a medication on demand and adds it manually to this section, and there is no limit on the numbers of medications that can be attached. The opportunity to filter medication by conditions was brought to our attention by some users.

● For drug-to-allergy interaction checks, a color-contrast background was determined to be potentially useful in making the alerts standout.

● Optimize the workflow for the Clinical Decision Support module, the placement of a direct link to the Plan of care next to the information to make it more accessible is helpful.

● Improve workflow for creating lab orders with diagnosis. Make the save button more obvious.

22

APPENDICES

The following appendices include supplemental data for this usability test report:

o Appendix 1 - Recruiting Screener Form o Appendix 2 - Participant Demographics o Appendix 3 - Informed Consent Form o Appendix 4 - Moderator’s Guide o Appendix 5 - Post-test Questionnaire

Appendix 1: RECRUITING SCREENER

The purpose of a screener to ensure that the participants selected represent the target user population as closely

as possible. (Portions of this sample screener are taken from www.usability.gov/templates/index.html#Usability

and adapted for use.)

Recruiting Script for Recruiting Firm

Hello, my name is , calling from [Insert name of recruiting firm]. We are recruiting individuals to participate

in a usability study for an electronic health record. We would like to ask you a few questions to see if you qualify

and if would like to participate. This should only take a few minutes of your time. This is strictly for research

purposes. If you are interested and qualify for the study, you will be paid to participate.

Would you be able to participate on [date, time]? [If so, collect contact information]

May I get your contact information?

Name of participant:

Address:

City, State, Zip:

Daytime phone number:

Evening phone number:

Alternate [cell] phone number:

Email address:

Before your session starts, we will ask you to sign a release form allowing us to videotape your session. The

videotape will only be used internally for further study if needed. Will you consent to be videotaped?

This study will take place at [location]. I will confirm your appointment a couple of days before your session and

provide you with directions to our office. What time is the best time to reach you?

Appendix 2: PARTICIPANT DEMOGRAPHICS

Following is a high-level overview of the participants in this study.

Participant Gender

n (N=10) %

Male 4 40

Female 6 60

Other 0 0

Participant Age

n (N=10) %

20-29 0 0

30-39 1 10

40-49 2 20

50-59 4 40

60-69 2 20

70-79 1 10

80-89 0 0

90 + years 0 0

Participant Education

n (N=10) %

No high school degree 0 0

High school graduate, diploma or the equivalent 2 20

Some college credit, no degree 0 0

Trade/technical/vocational training 2 20

Associate degree 0 0

Bachelor’s degree 0 0

Master’s degree 1 10

Doctorate degree (e.g., MD, DNP, DMD, PhD) 5 50

Participant Roles

n (N=10) %

RN/BSN 0 0

Physician 5 50

ARNP 0 0

Physician Assistant 0 0

Administrative Staff 4 40

Medical Assistant 0 0

Other 1 10

Participant Specialties

n (N=10) %

General Medicine 1 10

Cardiology 9 90

Other 0 0

Participant Professional Experience

Month Years

Mean 120 10

Min Max Min Max

Range 120 120 10 10

Participant Computer Experience

Month Years

Mean 66 5.5

Min Max Min Max

Range 12 120 1 10

Participant Experience with Product

Month Years

Mean 66 5.5

Min Max Min Max

Range 12 120 1 10

Appendix 3: INFORMED CONSENT FORM

Vipa Health would like to thank you for participating in this study. The purpose of this study is to evaluate an

electronic health records system. If you decide to participate, you will be asked to perform several tasks using

the prototype and give your feedback. The study will last up to 60 minutes.

YES, I have read the above statement and agree to be a participant.

NO, I choose not to participate in this study

Moderator's Guide

❏ Task 1: a.1 - CPOE Medications

❏ Task 2: a.2 - CPOE Labs

❏ Task 3: a.3 - CPOE Diagnostic Imaging

❏ Task 4: a.4 - Drug-drug

❏ Task 5: a.5 - Demographics

❏ Task 6: a.6 - Problem List

❏ Task 7: a.7 - Medication List

❏ Task 8: a.8 - Medication Allergy List

❏ Task 9: a.9 - Clinical Decision Support

❏ Task 10: a.14 - Implantable Device List

❏ Task 11: b.2 - Clinical Information Reconciliation

❏ Task 12: b.3 - Electronic Prescribing

Appendix 4

Scenarios

Task 1: a.1 - CPOE Medications:

After examining the patient history, physician has decided to start the patient on a Toprol XL

regimen.

Create a medication order (prescription) for Toprol XL 100 MG Tablet Extended Release 24 Hour,

1 tablet every day for 90 days. Adjust prescribed medication to Toprol XL 200 MG Tablet

Extended Release 24 Hour.

Task 2: a.2 - CPOE Labs:

Patient came to the office complaining of chest pain. Patient has active diagnoses of diabetes,

hypertension and Post myocardial infarction syndrome. After evaluating the patient, physician

has decided to order routine Lipid Panel. After the order is completed, physician realizes that

Hemoglobin A1C needs to be included in the lab test order. Go back to Lab orders and modify

current order to include Hemoglobin A1C.

● Lipid Panel (80061) diagnosis: Overweight E66.3 (from task 3). Send

● Hemoglobin A1c diagnosis Diabetes type 2 (E11.9) (from task 7). Send

Task 3: a.3 - CPOE Diagnostic Imaging:

Patient came to the office complaining of chest pain. Patient has active diagnoses of diabetes,

hypertension and Post myocardial infarction syndrome. After evaluating the patient, physician

decides to order an EKG, after the order is completed, physician decides to include a medical

necessity for the procedure.

Select one of the provided clinical information for this patient into this imaging order:

● Post myocardial infarction syndrome I25.2 (from task 3)

Task 4: a.4 - Drug to drug and drug to allergy interaction check:

Patient has aspirin 81 mg tablet and Lisinopril 2.5 in his current medication list. Physician starts

patient on Coumadin and Bactrim.

Case 1: Aspirin (from task 7) and Coumadin 1 mg tablet. (drug to drug)

Case 2: Sulfa (allergy. task 5) and Bactrim 400 mg-80 mg tablet (drug to allergy)

Task 5: a.5 - Demographics

A new patient, Mr. William John, that has been referred by Dr. Smith to see an specialist. Sign on

to the System. Open the patient record for Mr. William John. Enter/verify the provided

demographic information for this patient into the System

● Patient = Mr. William John

● DOB = 03/15/1951 (67 years)

● Ethnicity = Hispanic

● Sex = Male

● Race= White

After clicking Insert button, update demographic information with a phone number.

Task 6: a.6 - Problem List:

Review the problem list for this established patient. Add two new problems and update current

problem list.

Enter both of the following: (NEW)

● Post myocardial infarction syndrome I25.2

● Overweight E66.3

Update:

● Strep throat J02.0 (set as inactive/resolved) - preloaded

Task 7: a.7 - Medication List:

Review the medication list for this established patient. update current medication list, and

discontinue medication: amoxicillin 500 mg tablet

>Add: current medication: Lidoderm 5% Topical Film (1 patch, topical, daily, for 10 days)

>Add: Crestor tablet 10 mg - (1 Every day for 30 days)

>Discontinue: current med amoxicillin 500 mg tablet preloaded

Task 8: a.8 - Medication Allergy List:

Patient has amoxicillin 500 mg tablet in his current medication list and no allergies are listed.

Patient states that he is allergic to sulfa medications.

Add Sulfa to the allergy list. Update Sulfa allergy with the ‘Manifestation’ (Hives).

Task 9: a.9 - Clinical Decision Support:

Access CDS interventions and acknowledge the rules. Read out loud the description and the

source information on High Blood Pressure rule. Click on therapeutic info link and acknowledge

the guideline. Click on the infobutton link (Hypertension) and acknowledge the problem

information.

Task 10: a.14 - Implantable Device List: (new)

Add new Implantable Device (Pacemaker) and update Implantable Device (Cochlear implant) to

inactive status.

Case1:

● Patient Carol Britton, age 67, has a biotronik pacemaker.

● UDI: (01)04035479109790(21)1234

● Brand Name: Entovis DR-T

● Maker/Device: Biotronik pacemaker:

● Implant date: March-22-2018

Case 2: Preloaded

● Patient John David, DOB 04/01/2013, has a cochlear implant.

● UDI: (01)07630016830679(21)4321

● Brand: Hires 90

● Maker/Device: Advanced Bionics CI

● Implant date: January-10-2014

Task 11: b.2 - Clinical Information reconciliation:

While updating patient’s clinical information (problem list, medication list or allergy list)

physician realizes there is a list to be reconcile from a different physician. Please proceed

to reconcile external list.

Case 1:

● Practice: demointernal/drtest

● Patient:

● Patient chart > Clinical tab > [reconciliation] button

● current info in chart:

CCDA contains:

● CCDA Meds: patient has:

○ aspirin 81 mg Tablet (1 tablet every day for 30 days) [RxCUI = 315431]

○ Lisinopril 2.5 mg tablet (1 Tablet every day for 30 days) [RxCUI = 316152]

● CCDA Problem:

○ Diabetes type 2 (E11.9) and

○ Hypertension(I10) (both active)

● CCDA Allergy:

○ Sulfa and Iodine [reconcile with task 5]

Task 12: b.3 - Electronic Prescribing:

After examining the patient history, physician has decided to start the patient on a Toprol XL

regimen.

Send prescription electronically to pharmacy. (from med module outside)

• T12.1 - Create new prescription

▪ Create a medication order (prescription) for Toprol XL 100 MG Tablet Extended Release 24 Hour, 1 tablet every day for 90 days.

▪ Update Patient’s pharmacy information. (Pharmacy phone number: 305-274-6550)

▪ Send prescription electronically to pharmacy. (from med module outside)

• T12.4 - Change prescription (dosage or duration)

▪ Adjust prescribed medication to Toprol XL 200 MG Tablet Extended Release 24

Hour.

• T.12.5 - Cancel prescription

▪ Cancel prescribed medication Toprol XL 200 MG Tablet Extended Release 24 Hour. (click on discontinue and select a reason)

▪ [Send canceled/DC prescription (CANRX) to pharmacy]

• T.12.6 - Request and receive medication history information

• T.12.7 - Receive fill status notification (main home screen)

▪ Go to Home / log into system / landing screen Check Refill request screen / select patient

• T.12.8 - Refill prescription (re-prescribe/Refill)

▪ Refill Toprol XL 200 MG Tablet Extended Release 24 Hour.

Access Medication history tab in medication module

Task 13 - Admin functions: CDS and D2D/D2A admin tools

Activate all core rules for CDS including CDS for Problem List, Medication List, Allergy List,

demographics, Labs, and Vital signs. Interventions should display: source and therapeutic

information.

As a second task go to the warning configuration utility tool and select the severity level of

drug to drug interaction to display ALL severity levels warnings and select drug to allergy

warning to display ‘Open’.

● CCDA Allergy:

○ Sulfa and Iodine [reconcile with task 5]

Preliminary Task: First Impressions

Optimal Time: 125 seconds

This is the application you will be working with. Have you heard of it?

❏ Yes

❏ No

If so, tell me what you know about it.

▪ Show test participant the smartEMR 7.0 prototype. ▪ Please don’t click on anything just yet. ▪ What do you notice? What are you able to do here? Please be specific.

Notes / Comments:

Rating:

Overall, this task was: _______

Show participant written scale: “Very Easy” (1) to “Very Difficult” (5)

Administrator / Notetaker Comments:

Task 1: a.1 - CPOE Medications:


Take the participant to the starting point for the task logged in as a physician.

Please go to Patient’s Medications inside a visit and prescribe Toprol and adjust dosage.

Start Time: __________ AM/PM Number of Steps Taken: _______ Number of Errors: ________

End Time: ___________ AM/PM Number of Deviations: ________ Total Task Time: _______ sec

Optimal Path:

Select medication (Toprol XL 100 MG Tablet Extended Release 24 Hour) from Favorites list → Select SIG

from previously prescribed list → Click ‘Ok’ button → Click on ‘Adjust’ → Select new dosage (Toprol XL

200 MG Tablet Extended Release 24 Hour) → Click ‘Ok’ button

Optimal Path:

❏ Correct

❏ Minor Deviations / Cycles:

Describe

❏ Major Deviations: Describe

Completed Task Successfully:

❏ Yes

❏ No

Success:

❏ Easily completed

❏ Completed with difficulty or

help: Describe below

❏ Not completed

Rating:



Task Description Number of Clicks Task Time

Prescribe New Medication Order

Update Medication Order (adjust dosage)

Administrator / Notetaker Comments

Task 2: a.2 - CPOE Labs


Take the participant to the starting point for the task.

Please go to Patient’s open visit and add Lipid profile to Lab order.



Optimal Path:

Click on ‘Lab Orders’ link on ‘Summary’ tab → Click on checkbox for Lipid profile → Click on ‘MN’

link next name → Click on Diagnosis (Overweight E66.3) under Patient’s Problem List → Click ‘Save’

button → Click on ‘OK’ button → Click on ‘X’ to exit MN section → Click on ‘Send’ button to Front

Desk → Click on checkbox for Hemoglobin A1C → Click on ‘MN’ link next name → Click on Diagnosis

(Diabetes type 2 (E11.9)) under Patient’s Problem List → Click ‘Save’ button → Click on ‘OK’ button →

Click on ‘X’ to exit MN section → Click on ‘Send’ button to Front Desk

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access Lab Order Module

Record a new Lab Order with diagnosis

Route Lab Order

Update and route existing Lab Order


Task 3: a.3 - CPOE Diagnostic Imaging



Please add Diagnosis for previously ordered EKG procedure.



Optimal Path:

Click on ‘Diagnostic Imaging Orders’ link on ‘Summary’ tab → Click on ‘EKG’ checkbox → Click on

‘MN’ button next to EKG → Click on checkbox next to Diagnosis (Post myocardial infarction

syndrome I25.2) → Click on ‘Save’ button→ Click on ‘OK’ button → Click on ‘X’ to exit MN section

→ Click on ‘Send’ button to Front Desk

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access Diagnostic Imaging Order Module

Record a new Diagnostic Imaging Order and include diagnosis

Route to current order


Task 4: a.4 - Drug-drug and Drug to Allergy



After examining the Patient, you have decided to put this patient on Coumadin and Bactrim. Check for any

interactions and generate a prescription for these medications.



Optimal Path:

Click ‘Prescribe’ tab → Click on Coumadin from Favorites list → Verify/Acknowledge that drug to drug

interaction displays → Click on Bactrim from Favorites list → Verify/Acknowledge that drug to allergy

interaction displays

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access Med module in current visit

Prescribe, Trigger and acknowledge d2d intervention

Prescribe, Trigger and acknowledge d2a intervention


Task 5: a.5 – Demographics



A new patient has been referred by Dr. Smith to see a specialist. Create a new patient in the system with the

provided patient demographics.



Optimal Path:

Login to smartEMR→ Click on ‘New Patient Registration’ link→ Enter Name, DOB, Race, Ethnicity, and Sex →

Click on ‘Insert’ button → Click on ‘Edit Demographics’ → Click on ‘Patient Address’ tab → Enter phone

number → Click ‘Save’

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access smartEMR clinical

Record New Patient

Update Patient demographics information


Task 6: a.6 - Problem List



Please add provided clinical information to Patient’s record.



Optimal Path:

Browse Chart → Click on open visit name under ‘Unsigned’ Encounters tab → Click on ‘Assessment’ link

on ‘Summary’ tab → Type in first Diagnosis (Overweight - E66.3) in search bar→ Click on ‘Search

ICD10’ button → Click on the Description →Click on ‘Save’ button

Type in second Diagnosis (Post myocardial infarction syndrome - I25.2) in search bar→ Click on ‘Search

ICD10’ button → Click on the Description and then click on ‘Save’

Locate ‘Strep Throat - J02.0’ diagnosis under Current Problem List → Click on the ‘Add to Encounter’

button → Click on the ‘Update’ button → Click on ‘Resolved’ status → Click on ‘Save’ button →Click on

‘Remove from encounter’

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access problem list module within an encounter

Record a new problem

Update an existing problem

Administrator / Notetaker Comment

Task 7: a.7 - Medication List



Review Patient’s medication list. Document two new current medication (Crestor tablet 10 mg)

and (Lidoderm 5% Topical Film). Then, please discontinue a medication (Amoxicillin 500 mg

tablet) from Patient’s Active Medication list.



Optimal Path:

From the Summary page, click on ‘Medications’ link → Click ‘Document’ tab → Click on Crestor tablet 10 mg

from Favorites list → Enter SIG information→ Click on ‘Ok’ button → Search for Lidoderm 5% Topical Film →

Click on Medication Name → Click on Dosage → Enter SIG information→ Click on ‘Ok’ button → Locate

‘Amoxicillin 500 mg tablet’ on Active Medications list → Click on ‘Discontinue’ link under medication name →

click on ‘OK’ button next to desired ‘Reason’

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access Current Medication List

Document a new medication (pill)

Document a new liquid/gel/patch medication (non-pill)

Update an existing medication (dc)

Administrator / Notetaker Comment

Task 8: a.8 - Medication Allergy List



Add Sulfa to the Patient’s allergy list.



Optimal Path:

Click on ‘Allergies’ → Search for ‘Sulfa’ in the ‘Medication allergy search form’ → Click on ‘Add’ button

→ Click on “X’ delete button → Search for ‘Sulfa’ in the ‘Medication allergy search form’ → Type in

‘Manifestation’ type → Click ‘Add’ button

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access Allergy Module

Record New Medication Allergy

Update Medication Allergy - Add hives as manifestation


Task 9: a.9 - Clinical Decision Support


Take the participant inside an encounter.

Instruct the participant to interact (view, acknowledge and act upon) with the CDS module.



Optimal Path:

Scroll all the way down to the Clinical Decision Support section → Click on “more” link next to Controlling

High Blood Pressure→ Click on ‘Therapeutic Information’ link → Click on the ‘Hypertension’ link

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access and trigger CDS interventions

Check for CDS interventions generated by reconciled C-CDA (no action)

Access reference information for a CDS intervention (source)

Identify the guideline for a CDS intervention (therapeutic info)

Access reference information (info-button)


Task 10: a.14 - Implantable Device List



Instruct the participant to add an Implantable Devices to the Patient’s chart.



Optimal Path:

Click on ‘Implantable Devices’ under ‘Relevant Clinical Data’ → Enter the ‘UDI’ number and click on ‘Search’

→ Click on ‘Add’ → Click on ‘OK’

Click on ‘Make Inactive’ button next to Cochlear Implant → Click on ‘OK’ button

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access implantable device list

Record a new implantable device

Update implantable device list


Task 11: b.2 - Clinical Information Reconciliation


Take the participant to the Patient’s Chart.

Instruct the participant to reconcile patient’s clinical information.



Optimal Path:

Browse Chart → ‘Clinical’ tab → Click on ‘Reconciliation’ button → Select corresponding ‘Allergies’ → Click on

‘Reconcile allergies’ → Select corresponding ‘Medications’ → Click on ‘Reconcile medications’ → Select

corresponding ‘Problem 1’ → Select corresponding ‘Problem 2’→ Click on ‘Reconcile problems’

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access Patient's Documents and locate C-CDA

Reconcile patient’s current medication allergy list with external

source

Reconcile patient’s current medication list with external source

Reconcile patient’s active problem list with external source


Task 12: b.3 - Electronic Prescribing

Optimal Time: 222.5 seconds

Take the participant to Patient’s Chart.

Instruct the participant to update pharmacy and send prescription electronically



Optimal Path:

Click on ‘Medications’ link →Click on ‘Prescribe’ tab → Select medication from Favorites list → Select SIG

from previously prescribed list → Click ‘Ok’ button

Click on ‘Pharmacy’ tab → Enter pharmacy phone number and click ‘Search’ button→ Click on ‘Assign’ button

next to pharmacy name →

Click on “Active” tab → Check “eRx” box → Select Pharmacy from dropdown→ Click on “SEND” → Verify

prescription and click on “SEND” button

Click on Rx’s “Adjust” function → Select new dosage → Click ‘Ok’ button

Click on Rx “Discontinue/Cancel” → Select ‘OK’ button next to reason

Click on “Rx History” tab and wait for data to be downloaded.

Click on ‘Home’ button → Click on ‘Refill Requests’ link → Click on Patient with corresponding

Medication from list → Click on circle next to matching patient name

Click on “OK” button in the approve section

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Create new prescription

Access and Update pharmacy information

Send eRx (NEWRX)

Change prescription (dosage or duration)

Cancel prescription

Request and receive medication history information

Receive fill status notification (REFILL REQUEST

NOTIFICATION)

Refill prescription (REFILL RESPONSE)


Task 13: - Clinical Decision Support Admin Configuration


Take the participant to Admin Console.

Instruct the participant to adjust functionality for clinical decision support.



Optimal Path:

Login Page→ Admin Console Home → Click on ‘CDS Management’ link → Activate all CDS Core rules →

Activate all rules to ‘show therapeutic info’ → Activate all rules to display ‘Source; info

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access Admin Console

Access CDS management and configure a CDS rules (admin)


Task 13.1: - Drug to Drug and Drug to Allergy warning configuration


Take the participant to Admin Console.

Instruct the participant to adjust functionality for drug to drug and drug to allergy.



Optimal Path:

Under Warnings Configuration, click ’See Utility’→ Click on DDI dropdown menu → select “ALL”

→Click ‘OK’ → Click on Drug-Allergy Interactions dropdown menu →select ‘Open’ →Click ‘OK’

Optimal Path:

❏ Correct


Describe



❏ Yes

❏ No

Success:




❏ Not completed

Rating:




Access CDS management and configure a CDS rules


Appendix 5: Post-Test Questionnaires

SYSTEM USABILITY SCALE QUESTIONNAIRE

In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems

usability” known as the System Usability Scale or SUS.16

Lewis and Sauro (2009) and others have elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).

1. I think that I would like to use this

system frequently

2. I found the system unnecessarily

complex

3. I thought the system was easy

to use

4. I think that I would need the

support of a technical person to

be able to use this system

5. I found the various functions in

this system were well integrated

6. I thought there was too much

inconsistency in this system

7. I would imagine that most people

would learn to use this system

very quickly

8. I found the system very

cumbersome to use

9. I felt very confident using the

system

10. I needed to learn a lot of

things before I could get going

with this system

Strongly Strongly

disagree agree

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

http://www.usabilitynet.org/trump/documents/Suschapt.doc

Final Feedback Survey

1. What was your overall impression of this system?

2. What aspects of the system did you like most?

3. What aspects of the system did you like least?

4. Were there any features that you were surprised to see?

5. What features did you expect to encounter but did not see? That is, is there anything that is missing in this

application?

6. Compare this system to other systems you have used. Would you recommend this system to your colleagues?