onc hit program test results summary for 2014 edition … · test results summary for 2014 edition...
Post on 14-May-2018
214 Views
Preview:
TRANSCRIPT
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
Part 1: Product and Developer Information1.1 Certified Product Information
1.2 Developer/Vendor InformationPicis Clinical Solutions
Part 2: ONC‐Authorized Certification Body Information2.1 ONC‐Authorized Certification Body Information
Signature and Date
This test results summary is approved for public release by the following ONC‐Authorized Certification Body
Representative:
Adam Hardcastle
ONC‐ACB Authorized Representative Function/Title
San Luis Obispo, CA 93401
Website: www.infogard.com
Email: ehr@infogard.com
Phone: (805) 783‐0810
EHR Certification Body Manager
ONC‐ACB Name: InfoGard Laboratories, Inc.
Address: 709 Fiero Lane Suite 25
Email: PicisMU@picis.com
Phone: (844) 240‐9554
ONC‐ACB Contact: Adam Hardcastle
ONC HIT Certification Program
Test Results Summary for 2014 Edition EHR Certification
Product Name: Picis ED PulseCheck
Product Version: 5.4
Domain: Inpatient
Developer/Vendor Contact: Merrily King Turnbull
Rosemont, IL 60018
Website: www.picis.com
Test Type: Modular EHR
Developer/Vendor Name:
Address: 10275 W. Higgins Road Ste. 250
©2016 InfoGard. May be reproduced only in its original entirety, without revision 1
6/28/16
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
2.2 Gap Certification
(a)(1) (a)(19) (d)(6) (h)(1)
(a)(6) (a)(20) (d)(8) (h)(2)
(a)(7) (b)(5)* (d)(9) (h)(3)
(a)(17) (d)(1) (f)(1)
(a)(18) (d)(5) (f)(7)**
*Gap certification allowed for Inpatient setting only
**Gap certification allowed for Ambulatory setting only
2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification
(a)(19)
(a)(20)
(a)(13)
(c)(2)
(e)(3) Amb. only
No inherited certification
(f)(5) Amb. only
(f)(6) Amb. only
(f)(7) Amb. Only
(h)(1)
(h)(2)
(h)(3)
(a)(11)
(b)(7) (e)(1)
(g)(3)
(a)(12)
(c)(1)
(b)(9)
(a)(18)
(e)(2) Amb. only
(g)(4)
(a)(9)
(b)(5) (d)(8)
(g)(1)
(a)(10)
(b)(6) Inpt. only (d)(9) Optional
(g)(2)
(b)(8)
(d)(6) (a)(8)
(b)(4) (d)(7)
(a)(5)
(b)(1) (d)(4) (a)(6)
(b)(2) (d)(5)
(a)(14)
(c)(3)
(f)(1)
(a)(2)
(a)(15)
(d)(1)
(f)(2)
The following identifies criterion or criteria certified via gap certification
§170.314
No gap certification
§170.314
(a)(3)
(a)(16) Inpt. only
(d)(2)
(f)(3)
(a)(4)
(a)(17) Inpt. only
(d)(3)
(f)(4) Inpt. only
(a)(1)
(a)(7)
(b)(3)
©2016 InfoGard. May be reproduced only in its original entirety, without revision 2
Test Results Summary for 2014 Edition EHR Certification
16-2423-R-0022-PRI Vl.O, June 20, 2016
Part 3: NVLAP-Accredited Testing Laboratory Information
Report Number: N/A
Test Date(s): N/A
3.1 NVLAP-Accredited Testing Laboratory Information ATL Name: Accreditation Number:
Address:
Website:
Email:
Phone:
ATL Contact:
lnfoGard Laboratories, Inc.
NVLAP Lab Code 100432-0
709 Fiero Lane Suite 25
San Luis Obispo, CA 93401
www.infogard.com
ehr@infogard.com
(805) 783-0810
Milton Padilla
For more information on scope of accreditation, please reference
http://ts.nist.gov/Sta ndards/scopes/1004320. htm
D
Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory
Representative:
Milton Padilla EHR Test Body Manager
ATL Authorized Representative Function/Title
s.~~~4 3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
. . . . . Functionality provided by Add1t1onal Software Applicable Cntena d ft
A ditional So ware
g
D No additional software required
©2016 lnfoGard. May be reproduced only in its original entirety, without revision 3
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
3.2.2 Test Tools
Version
No test tools required
3.2.3 Test Data
3.2.4 Standards
3.2.4.1 Multiple Standards Permitted
(a)(13)
§170.207(a)(3)
IHTSDO SNOMED CT®
International Release July 2012
and US Extension to SNOMED
CT® March 2012 Release
§170.207(j)
HL7 Version 3 Standard: Clinical
Genomics; Pedigree
The following identifies the standard(s) that has been successfully
tested where more than one standard is permitted
Criterion # Standard Successfully Tested
(a)(8)(ii)(A)(2)
§170.204(b)(1)
HL7 Version 3 Implementation
Guide: URL‐Based
Implementations of the Context‐
Aware Information Retrieval
(Infobutton) Domain
§170.204(b)(2)
HL7 Version 3 Implementation
Guide: Context‐Aware
Knowledge Retrieval
(Infobutton) Service‐Oriented
Architecture Implementation
Guide
HL7 v2 Laboratory Restults Intervace (LRI) Validation Tool
HL7 v2 Syndromic Surveillance Reporting Validation Tool
Transport Testing Tool
Direct Certificate Discovery Tool
Alteration (customization) to the test data was necessary and is
described in Appendix [insert appendix letter ]
No alteration (customization) to the test data was necessary
Test Tool
Cypress
ePrescribing Validation Tool
HL7 CDA Cancer Registry Reporting Validation Tool
HL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting
Valdiation Tool
Edge Testing Tool
©2016 InfoGard. May be reproduced only in its original entirety, without revision 4
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
None of the criteria and corresponding standards listed above are applicable
3.2.4.2 Newer Versions of Standards
No newer version of a minimum standard was tested
Common MU Data
Set (15)
§170.207(a)(3)
IHTSDO SNOMED CT®
International Release July 2012
and US Extension to SNOMED
CT® March 2012 Release
§170.207(b)(2)
The code set specified at 45 CFR
162.1002(a)(5) (HCPCS and CPT‐
4)
The following identifies the newer version of a minimum standard(s) that has
been successfully tested
Newer Version Applicable Criteria
(e)(1)(ii)(A)(2)
§170.210(g)
Network Time Protocol Version 3
(RFC 1305)
§170. 210(g)
Network Time Protocol Version 4
(RFC 5905)
(e)(3)(ii) Annex A of the FIPS Publication 140‐2
(b)(7)(i)
§170.207(i)
The code set specified at 45 CFR
162.1002(c)(2) (ICD‐10‐CM) for
the indicated conditions
§170.207(a)(3)
IHTSDO SNOMED CT®
International Release July 2012
and US Extension to SNOMED
CT® March 2012 Release
(e)(1)(i) Annex A of the FIPS Publication 140‐2
(a)(16)(ii)
§170.210(g)
Network Time Protocol Version 3
(RFC 1305)
§170. 210(g)
Network Time Protocol Version 4
(RFC 5905)
(b)(2)(i)(A)
§170.207(i)
The code set specified at 45 CFR
162.1002(c)(2) (ICD‐10‐CM) for
the indicated conditions
§170.207(a)(3)
IHTSDO SNOMED CT®
International Release July 2012
and US Extension to SNOMED
CT® March 2012 Release
(b)(8)(i)
§170.207(i)
The code set specified at 45 CFR
162.1002(c)(2) (ICD‐10‐CM) for
the indicated conditions
§170.207(a)(3)
IHTSDO SNOMED CT®
International Release July 2012
and US Extension to SNOMED
CT® March 2012 Release
(a)(15)(i)
§170.204(b)(1)
HL7 Version 3 Implementation
Guide: URL‐Based
Implementations of the Context‐
Aware Information Retrieval
(Infobutton) Domain
§170.204(b)(2)
HL7 Version 3 Implementation
Guide: Context‐Aware
Knowledge Retrieval
(Infobutton) Service‐Oriented
Architecture Implementation
Guide
©2016 InfoGard. May be reproduced only in its original entirety, without revision 5
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
3.2.5 Optional Functionality
No optional functionality tested
Common MU Data
Set (15)
Express Procedures according to the standard
specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD‐
10‐PCS)
(b)(2)(ii)(C)
Transmit health information to a Third Party using
the standards specified at §170.202(b) and (c) (SOAP
Protocols)
(f)(3)
Ambulatory setting only – Create syndrome‐based
public health surveillance information for transmission
using the standard specified at §170.205(d)(3) (urgent
care visit scenario)
Common MU Data
Set (15)
Express Procedures according to the standard
specified at §170.207(b)(3) (45 CFR162.1002(a)(4):
Code on Dental Procedures and Nomenclature)
(e)(1)
View, download and transmit data to a third party
using the standard specified at §170.202(d) (Edge
Protocol IG version 1.1)
(f)(7) Ambulatory setting only – transmission to public
health agencies – syndromic surveillance ‐ Create Data
Elements
(b)(1)(i)(B)
Receive summary care record using the standards
specified at §170.202(a) and (b) (Direct and XDM
Validation)
(b)(1)(i)(C) Receive summary care record using the standards
specified at §170.202(b) and (c) (SOAP Protocols)
(b)(2)(ii)(B)
Transmit health information to a Third Party using
the standards specified at §170.202(a) and (b) (Direct
and XDM Validation)
Criterion # Optional Functionality Successfully Tested
(a)(4)(iii) Plot and display growth charts
©2016 InfoGard. May be reproduced only in its original entirety, without revision 6
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
3.2.6 2014 Edition Certification Criteria* Successfully Tested
TP** TD*** TP** TD***
*For a list of the 2014 Edition Certification Criteria, please reference
http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)
**Indicates the version number for the Test Procedure (TP)
***Indicates the version number for the Test Data (TD)
(b)(7)
(g)(3)
(c)(1)
(g)(4)
(c)(2)
(b)(3)
(f)(6) Optional & Amb. only
(b)(4)
(b)(5)
(g)(1)
(b)(6) Inpt. only
(g)(2)
(b)(8)
(b)(9)
(f)(7) Amb. only
(h)(1)
(h)(2)
(h)(3)
(a)(16) Inpt. only (f)(3)
(a)(17) Inpt. only (f)(4) Inpt. only
(b)(1)
(f)(5) Optional & Amb. only
(b)(2)
(a)(13) (e)(3) Amb. only
(a)(14) (f)(1)
(a)(15) (f)(2)
(a)(18)
(a)(20)
(a)(19)
(a)(10) (d)(9) Optional
(a)(11) (e)(1)
(a)(12) (e)(2) Amb. only
(a)(7) (d)(6)
(a)(8) (d)(7)
(a)(9) (d)(8)
(a)(4) (d)(3)
(a)(5) (d)(4)
(a)(6) (d)(5)
(a)(1) (c)(3)
(a)(2) (d)(1)
(a)(3) (d)(2)
Criteria #Version
Criteria #Version
©2016 InfoGard. May be reproduced only in its original entirety, without revision 7
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:
CMS ID Version CMS ID Version CMS ID Version CMS ID Version
2 90 136 155
22 117 137 156
50 122 138 157
52 123 139 158
56 124 140 159
61 125 141 160
62 126 142 161
64 127 143 163
65 128 144 164
66 129 145 165
68 130 146 166
69 131 147 167
74 132 148 169
75 133 149 177
77 134 153 179
82 135 154 182
CMS ID Version CMS ID Version CMS ID Version CMS ID Version
9 71 107 172
26 72 108 178
30 73 109 185
31 91 110 188
32 100 111 190
53 102 113
55 104 114
60 105 171
*For a list of the 2014 Clinical Quality Measures, please reference
http://www.cms.gov (navigation: 2014 Clinical Quality Measures)
Ambulatory CQMs
Inpatient CQMs
Ambulatory
Inpatient
No CQMs tested
©2016 InfoGard. May be reproduced only in its original entirety, without revision 8
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
3.2.8 Automated Numerator Recording and Measure Calculation
3.2.8.1 Automated Numerator Recording
Automated Numerator Recording was not tested
3.2.8.2 Automated Measure Calculation
Automated Measure Calculation was not tested
3.2.9 Attestation
(a)(17) (b)(5)
(a)(9) (a)(17) (b)(5)
(e)(2)
(e)(3)
(a)(6) (a)(15) (b)(3)
Safety‐Enhanced Design* A
(a)(5) (a)(14) (b)(2)
Attestation Forms (as applicable) Appendix
(a)(11) (a)(18)
Automated Numerator Recording Successfully Tested
(a)(1) (a)(11) (a)(18) (b)(6)
(e)(1)
(e)(3)
(a)(9)
(a)(7) (a)(16) (b)(4)
(a)(3) (a)(12) (a)(19) (b)(8)
(a)(4) (a)(13) (a)(20) (b)(9)
(a)(7) (a)(16) (b)(4)
Automated Numerator Recording Successfully Tested
(a)(1) (b)(6)
(a)(5) (a)(14) (b)(2)
Quality Management System** B
Privacy and Security C
*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16),
(a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)
**Required for every EHR product
(e)(1)
(a)(6) (a)(15) (b)(3)
(a)(3) (a)(12) (a)(19) (b)(8)
(a)(4) (a)(13) (a)(20) (b)(9)
(e)(2)
©2016 InfoGard. May be reproduced only in its original entirety, without revision 9
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
Appendix A: Safety Enhanced Design
©2016 InfoGard. May be reproduced only in its original entirety, without revision 10
10275 West Higgins Road, Suite 250 Rosemont, IL 60018 Tel: 844 240 9554 picis.com
June 16, 2016 Milton Padilla InfoGard, a UL Company 709 Fiero Ln, Suite 25 San Luis Obispo, CA 93401 Dear Mr. Padilla, Per 170.314(g)(3) criteria, this letter serves as attestation of our User-Centered Design practices. Picis PulseCheck follows Solution Architectural Design documents (specifically sections 4.0 – 4.4) for the development of each version, including ED PulseCheck versions 5.4, 5.5, and 5.6. The Solution Architectural Design documents are part of the Picis overall Quality Management System which is designed to meet the requirements of the international standard ISO 13485 Medical Devices – Quality Management Systems – Requirements for Regulatory Purposes Standards as well as the FDA Quality System Regulation 21 CFR Part 820. Additionally, Picis maps our procedures to IEC 62366-1 Application of Usability Engineering to Medical Devices. The procedures were followed for planning, development, testing, and release of each application version and capability for which Picis sought certification, including:
170.314(a)(1) Computerized provider order entry
170.314(a)(2) Drug-drug, drug-allergy interaction checks
170.314(a)(6) Medication list
170.314(a)(7) Medication allergy list
170.314(b)(3) Electronic prescribing Please reference the attached Solution Architectural Design documents for further details and please do not hesitate to contact me with any outstanding questions. Thank you,
Merrily King Turnbull, MBA Director, Product Management Emergency Department Solutions
Page 1 of 56
EHR Usability Test Report of ED PulseCheck Version 5.4
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Optum ED PulseCheck Version 5.4
Date of Usability Test 12/03/2013 – 12/12/2013 Date of Report 01/27/2014 Report Prepared By John Romadka, Senior User Experience Researcher, Optum
(612) 632-2694 john.romadka@optum.com
Jamie Peffley, User Experience Manager, Optum
(612) 632-2687 jamie.peffley@optum.com
Scott Stewart, Senior User Experience Designer, Optum
(602) 745-7916 scott.r.stewart@optum.com
Page 2 of 56
EXECUTIVE SUMMARY ................................................................................................................................................... 4
PERFORMANCE DATA ....................................................................................................................................................... 5 Physicians .................................................................................................................................................................. 6
Major Findings ......................................................................................................................................................................... 7 Areas for Improvement ............................................................................................................................................................ 7
RNs ............................................................................................................................................................................ 8 Major Findings ......................................................................................................................................................................... 8 Areas for Improvement ............................................................................................................................................................ 9
INTRODUCTION ............................................................................................................................................................... 9
METHOD ........................................................................................................................................................................... 9
PARTICIPANTS ................................................................................................................................................................. 9 RNs .......................................................................................................................................................................... 11
STUDY DESIGN .............................................................................................................................................................. 12 TASKS ........................................................................................................................................................................... 12
Physicians ................................................................................................................................................................ 12 RNs .......................................................................................................................................................................... 13
PROCEDURES ................................................................................................................................................................ 14 TEST LOCATION ............................................................................................................................................................. 14 TEST ENVIRONMENT ...................................................................................................................................................... 14 TEST FORMS AND TOOLS ............................................................................................................................................... 15 PARTICIPANT INSTRUCTIONS .......................................................................................................................................... 15 USABILITY METRICS ....................................................................................................................................................... 16
DATA SCORING ............................................................................................................................................................. 17
RESULTS ........................................................................................................................................................................ 18
DATA ANALYSIS AND REPORTING .................................................................................................................................... 18 Physicians ................................................................................................................................................................ 19
Discussion of the Findings ..................................................................................................................................................... 20 Major Findings ....................................................................................................................................................................... 23
RNs .......................................................................................................................................................................... 25 Discussion of the Findings ..................................................................................................................................................... 25 Major Findings ....................................................................................................................................................................... 27
Combined (Physician and RN) Areas for Improvement .......................................................................................... 28
APPENDICES ................................................................................................................................................................. 29
APPENDIX 1: PARTICIPANT DEMOGRAPHICS .................................................................................................................... 30 Physicians ................................................................................................................................................................ 30 RNs .......................................................................................................................................................................... 31
APPENDIX 2: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM ................................................................. 32 APPENDIX 3: EXAMPLE MODERATOR’S GUIDE FOR PHYSICIANS AND RNS ........................................................................ 33 APPENDIX 4: SYSTEM USABILITY SCALE QUESTIONNAIRE ................................................................................................ 34 APPENDIX 5: TASKS ....................................................................................................................................................... 35
Physicians ................................................................................................................................................................ 35 Task 0 – Login ....................................................................................................................................................................... 35 Task 1 - Enter a laboratory, radiology, and imaging orders ................................................................................................... 36 Task 2 - Enter medication orders .......................................................................................................................................... 37
Page 3 of 56
Task 3 – Cancel or hold a medication order .......................................................................................................................... 38 Task 4 – Delete or cancel a medication order ....................................................................................................................... 39 Task 5 – Update a medication order...................................................................................................................................... 40 Task 6 –Update a radiology order ......................................................................................................................................... 41 Task 7 – Approve a medication order with a drug-allergy reaction ........................................................................................ 42 Task 8 – Approve a medication order with a drug-drug interaction ....................................................................................... 43 Task 17 - Configure pharmacy for e-prescription .................................................................................................................. 44 Task 18 - Change pharmacy for e-prescription ..................................................................................................................... 45
RNs .......................................................................................................................................................................... 46 Task 0 – Login ....................................................................................................................................................................... 46 Task 10 - Enter active medications ....................................................................................................................................... 47 Task 11 - Update a medication currently being taken............................................................................................................ 48 Task 12 - Inactivate an active medication .............................................................................................................................. 49 Task 13 – Find active and inactive allergies .......................................................................................................................... 50 Task 14 - Enter active allergies ............................................................................................................................................. 51 Task 15 - Update the effects of an allergy ............................................................................................................................. 52 Task 16 - Inactivate an allergy ............................................................................................................................................... 53
APPENDIX 6: DEMOGRAPHICS QUESTIONNAIRE ............................................................................................................... 54 APPENDIX 7: DATA SCORING DETAILS ............................................................................................................................. 56
Page 4 of 56
Executive Summary A summative usability test of ED PulseCheck version 5.4 was conducted 12/03/2013 – 12/12/2013 by the
Optum User Experience Center of Excellence. The test was conducted remotely using a web-based testing
tool, UserZoom, and WebEx. The purpose of this test was to provide evidence of usability in the EHR Under
Test (EHRUT) and identify areas for incremental usability improvement.
Ten physicians and 13 RNs matching the target demographic criteria participated in the usability test, using
the EHRUT in simulated but representative tasks.
During the 60-minute one-on-one usability sessions, each participant was greeted by the administrator and
asked to review and sign a non-disclosure/consent form (see Appendix 2). Participants were instructed that
they could withdraw at any time. All participants had prior experience with the EHRUT.
The administrator introduced the test and instructed participants to complete a series of tasks (presented
randomly one at a time) using the EHRUT. During the testing, the testing tool (UserZoom) recorded user
performance data electronically. A Data logger also recorded user performance. The administrator did not
give the participant assistance in how to complete the task (and if it became necessary, that task was
marked as failed).
Physicians performed the following tasks:
Area of Certification
Tasks
§170.314(a)(1)
Computerized
provider order
entry
1 Enter laboratory and radiology/imaging orders
2 Enter medication orders
3 Cancel or hold a medication order
4 Delete or cancel a medication order
5 Update a medication order
6 Update a radiology order
§170.314(a)(2)
Drug-drug, drug-
allergy
interaction
checks
7 Approve a medication order with a drug-allergy reaction
8 Approve a medication order with a drug-drug interaction
§170.314(b)(3)
Electronic
prescribing
17 Configure pharmacy for e-prescription
18 Change pharmacy for e-prescription
Page 5 of 56
RNs performed the following tasks:
Area of Certification
Tasks
§170.314(a)(6)
Medication list
10 Enter active medications
11 Update a medication currently being taken
12 Inactivate an active medication
§170.314(a)(7)
Medication
allergy list
13 Find a list of active and inactive allergies
14 Enter active allergies
15 Update the effects of an allergy
16 Inactivate an allergy
Each participant’s interactions with the EHRUT were recorded for subsequent analysis. The following types
of data were collected:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations
• Participant’s satisfaction ratings of the tasks and system
All participant data was de-identified to avoid linking the identity of the participant to the data collected.
Following the conclusion of the testing, participants were asked to complete a post-test questionnaire.
Physicians and RNs were compensated $250 and $150 respectively for their time.
Performance Data Various recommended metrics, in accordance with the examples set forth in the NISTIR Guide to the
Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the
usability of the EHRUT. Following is a summary of the performance and rating data collected on the
EHRUT.
Page 6 of 56
Physicians
Measure
N Task
Success Path
Deviation Task Time Errors
Task Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
1. Enter laboratory and
radiology/imaging orders 10
70%
(±48%) 1.7
99
(±54) 179%
40%
(±70%)
3.9
(±74%)
2. Enter medication orders 10 50%
(±53%) 2.1
180
(±50) 279%
70%
(±95%)
3.8
(±63%)
3. Cancel or hold a
medication order 10
90%
(±32%) 0.9
44
(±22) 98%
10%
(±32%)
4.6
(±84%)
4. Delete or cancel a
medication order 10
80%
(±42%) 1.5
46
(±19) 133%
20%
(±42%)
4.7
(±67%)
5. Update a medication
order 10
90%
(±32%) 1.4
98
(±40) 121%
10%
(±32%)
4.3
(±67%)
6. Update a radiology order 10 90%
(±32%) 1.8
78
(±29) 144%
10%
(±32%)
4.0
(±105%)
7. Approve a medication
order with a drug-allergy
reaction
10 80%
(±42%) 3.0
75
(±30) 123%
20%
(±42%)
4.9
(±32%)
8. Approve a medication
order with a drug-drug
interaction
10 80%
(±42%) 2.3
76
(±21) 148%
20%
(±42%)
4.5
(±53%)
17. Configure pharmacy for
e-prescription 10
50%
(±53%) 2.4
167
(±56) 207%
50%
(±53%)
3.4
(±97%)
18. Change pharmacy for
e-prescription 10
50%
(±53%) 1.9
184
(±142) 196%
80%
(±103%)
3.7
(±82%)
The System Usability Scale was used to score participants’ subjective satisfaction based on their overall
experience with the EHRUT. The overall score was 86 (±10).1
In addition to the performance data, we recorded the following qualitative observations.
1 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.
Page 7 of 56
Major Findings Major findings include:
• Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness
• Participants indicated an above-average level of product satisfaction
• The EHRUT has multiple navigation paths that participants can use to complete tasks; participants
used a variety of the methods available (in some cases, this resulted in slower task times)
• Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT
As well as issues with:
• E-prescription search
• Data loss when users entered free text
• Incomplete record entries/updates (e.g., without schedules)
• Multiple action buttons too close to each other
• Dropdown lists (number of items and inconsistencies in position)
Areas for Improvement Recommendations for improvement include the following:
• Improve flexibility in search entries and results presentation for e-Prescription searches
• Alert the user when necessary data (e.g., medication schedule) are skipped to avoid incomplete
entries/updates
• Help users find the correct action (e.g., visually distinguishing the button required to complete the
action)
• Recognize free text data entries (e.g., medical abbreviations) to avoid data loss
• Consistent dropdown list display (location and length)
Page 8 of 56
RNs
Measure N
Task Success
Path Deviation
Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
10. Enter active medications 13 60%
(±52%) 2.3
172 (±29)
235% 80%
(±104%) 4.6
(±70%)
11. Update a medication
currently being taken 13
69% (±48%)
1.8 120
(±45) 187%
54% (±88%)
3.9 (±126%)
12. Inactivate an active
medication 13
85% (±38%)
2.0 114
(±30) 165%
15% (±38%)
4.4 (±96%)
13. Find a list of active and
inactive allergies 13
46% (±52%)
1.4 59
(±25) 198%
62% (±51%)
4.3 (±125%)
14. Enter active allergies 13 92%
(±28%) 1.4
113 (±33)
145% 31%
(±85%) 4.5
(±78%)
15. Update the effects of an
allergy 13
77% (±44%)
1.4 82
(±57) 133%
31% (±48%)
4.4 (±119%)
16. Inactivate an allergy 13 69%
(±48%) 1.2
98 (±32)
128% 23%
(±44%) 4.7
(±48%)
The System Usability Scale was used to score participants’ subjective satisfaction with the system based on
their overall experience with the EHRUT. The overall score was 87(±17).2
In addition to the performance data, we recorded the following qualitative observations.
Major Findings Major findings include:
• Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness
• Participants’ indicated an above-average level of product satisfaction
• The EHRUT has multiple navigation paths that participants can use to complete tasks; participants
used a variety of the methods available (in some cases, this resulted in slower task times)
• Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT)
As well as issues with:
• Difficulty finding “Show inactive” allergies
• Multiple action buttons too close to each other
2 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149). Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average.
Page 9 of 56
• Difficulty finding how to update the effects of an allergy
• Consistent dropdown list display (location and length)
Areas for Improvement Recommendations for improvement include the following:
• Make “Show inactive” more visually identifiable
• Help users find the correct action buttons
• Don’t rely on a hyperlinked line with no label or text for navigation
• Consistent dropdown list (location and length)
Introduction The EHRUT tested for this study was ED PulseCheck version 5.4. Designed to present medical information
to healthcare providers in an emergency department, the EHRUT consists of a tracking board, patient
charts, medication ordering system, lab ordering system, and e-prescription system. The usability testing
represented realistic tasks.
The purpose of this test was to provide evidence of usability in the EHRUT and identify areas for
incremental usability improvement. To this end, measures of effectiveness, efficiency, and user satisfaction
(such as task completion, time on task, and task efficiency) were captured during the usability testing.
Method
Participants A total of 23 participants, consisting of 10 physicians and 13 RNs, were tested on the EHRUT. All
participants were given the same orientation instructions. Physicians and RNs were compensated $250 and
$150 respectively for their time.
Participants were recruited by Leede Research. Participants had no direct connection to the development of
or organization producing the EHRUT. Participants were not from the testing or supplier organization. All
participants currently use the EHRUT in a professional workplace and were recruited based on job title,
gender, and age.
The following table summarizes the characteristics of the physicians who participated in the usability test,
including demographics, professional experience, computing experience and use of assistive technology.
Participant names were replaced with Participant IDs so that data cannot be tied back to identities.
Page 10 of 56
Part ID
Part ID
Gender Age Education Occupation Professional Experience
Computer Experience
Product Experience
Assistive Technologies
1 P4 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 20-29
years
1-3 years No
2 P11 Male 50-59 Professional
degree (e.g.
MD)
Physician 20-29 years 7-9 years 4-6 years Yes, Dragon
3 P13 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 10-19
years
7-9 years No
4 P18 Male 50-59 Professional
degree (e.g.
MD)
Physician 10-19 years 20-29
years
4-6 years No
5 P25 Female 50-59 Professional
degree (e.g.
MD)
Physician 20-29 years 7-9 years 4-6 years No
6 P26 Male 60-69 Professional
degree (e.g.
MD)
Physician 30-39 years 20-29
years
1-3 years No
7 P14 Male 60-69 Professional
degree (e.g.
MD)
Physician 20-29 years 7-9 years Less than
1 year
No
8 P29 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 10-19
years
4-6 years No
9 P28 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 4-6 years 4-6 years Yes, Dragon
10 P23 Male 30-39 Professional
degree (e.g.
MD)
Physician 4-6 years 7-9 years 4-6 years No
Fourteen physicians were originally recruited, including two pilots.
• Three sessions were eliminated from results because of an inconsistency between the script and the
test environment in task 2
• One participant was found not to meet the requirements
• There were two no-shows
Two additional physicians were recruited to provide test results for 10 physicians.
Page 11 of 56
Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each
session for debrief by the administrator and data logger and to reset systems to proper test conditions. A
spreadsheet was used to track the participant schedule.
RNs
The following table summarizes the characteristics of the RNs who participated in the usability test,
including demographics, professional experience, computing experience, and use of assistive technology.
Participant names were replaced with Participant IDs so that data cannot be tied back to identities.
Part ID
Part ID
Gender Age Education Occupation Professional Experience
Computer Experience
Product Experience
Assistive Technologies
1 P2 Female 30-39 Associate’s
degree
Registered
Nurse
10-19
years
10-19 years 7-9 years No
2 P3 Female 40-49 Doctoral
degree
Nurse
Practitioner
10-19
years
10-19 years 1-3 years No
3 P5 Male 50-59 Bachelor’s
degree
Registered
Nurse
7-9 years 10-19 years 7-9 years No
4 P6 Male 40-49 Bachelor’s
degree
Registered
Nurse
10-19
years
10-19 years 1-3 years No
5 P8 Female 30-39 Bachelor’s
degree
Registered
Nurse
1-3 years 7-9 years 1-3 years No
6 P9 Female 30-39 Associate’s
degree
Registered
Nurse
4-6 years 1-3 years Less than 1
year
Yes, Dragon
7 P12 Female 40-49 Bachelor’s
degree
Registered
Nurse
20-29
years
20-29 years 7-9 years No
8 P19 Male 60-69 Associate’s
degree
Registered
Nurse
20-29
years
4-6 years 4-6 years No
9 P20 Female 40-49 Bachelor’s
degree
Registered
Nurse
10-19
years
10-19 years 1-3 years No
10 P21 Male 40-49 Bachelor’s
degree
Registered
Nurse
4-6 years 10-19 years 4-6 years Yes, Dragon
11 P22 Female 40-49 Bachelor’s
degree
Registered
Nurse
20-29
years
20-29 years 4-6 years No
12 P15 Female 18-29 Bachelor’s
degree
Registered
Nurse
1-3 years 4-6 years 4-6 years No
13 P27 Male 40-49 Bachelor’s
degree
Registered
Nurse
10-19
years
10-19 years 4-6 years No
Page 12 of 56
A total of 14 RNs were recruited for the test. One participant failed to show for the study. Test results for 13
RNs are included in this report.
Participants were scheduled for 60-minute sessions. A minimum of 60 minutes was reserved between each
session for debrief by the administrator and data logger and to reset systems to proper test conditions. A
spreadsheet was used to track the participant schedule.
Study Design
Overall, the objective of this test was to measure how well the application performed with regard to
effectiveness, efficiency, and user satisfaction; to identify areas for improvement; and to recommend
improvements. Additionally, the data from this test will serve as a baseline for future tests for comparison
with updated versions of the EHR and/or other EHRs. In short, this testing serves as a means to benchmark
current usability and to identify areas where improvements can be made.
All participants were given the same orientation instructions and accessed the EHRUT using the same
remote testing environment. Two task sets were defined—one for physicians and one for RNs. All
participants used the same test data. The EHRUT was evaluated using the following measures:
• Number of tasks successfully completed within the allotted time without assistance
• Time to complete the tasks
• Number and types of errors
• Path deviations
• Participant’s verbalizations (comments)
• Participant’s satisfaction ratings of the tasks and system
Additional information about the various measures can be found in the Usability Metrics section of this
document.
Tasks Tasks were selected based on how frequently they are used, how critical they are, and how likely they are
to be troublesome for users. Task details were defined to be realistic and representative of the kinds of
activities users might perform with the EHRUT.
Physicians
Physicians performed the following tasks:
Area of Certification
Tasks
§170.314(a)(1)
Computerized
provider order
1 Enter laboratory and radiology/imaging orders
2 Enter medication orders
Page 13 of 56
Area of Certification
Tasks
entry 3 Cancel or hold a medication order
4 Delete or cancel a medication order
5 Update a medication order
6 Update a radiology order
§170.314(a)(2)
Drug-drug, drug-
allergy
interaction
checks
7 Approve a medication order with a drug-allergy reaction
8 Approve a medication order with a drug-drug interaction
§170.314(b)(3)
Electronic
prescribing
17 Configure pharmacy for e-prescription
18 Change pharmacy for e-prescription
RNs
RNs performed the following tasks:
Area of Certification
Tasks
§170.314(a)(6)
Medication list
10 Enter active medications
11 Update a medication currently being taken
12 Inactivate an active medication
§170.314(a)(7)
Medication
allergy list
13 Find a list of active and inactive allergies
14 Enter active allergies
15 Update the effects of an allergy
16 Inactivate an allergy
Page 14 of 56
Procedures Upon arrival, participants were greeted and their identity was verified and matched with the participant
schedule. Participants were then assigned a participant ID.3 Each participant digitally signed a non-
disclosure and informed consent form. See Appendix 2: Non-Disclosure Agreement and Informed Consent
Form for an example of the form.
The test was conducted by two staff members, the administrator, and the data logger. The individuals
conducting the test were usability practitioners with over 15 years of experience and educational
backgrounds in Human Factors.
The administrator moderated the session, delivered initial instructions, and took notes on participant
comments, path deviations, and errors. The data logger also took notes on task success, path deviations,
errors, and comments. The testing tool (UserZoom) led participants through the tasks. UserZoom recorded
task times, post-task rating data, post-test satisfaction data, and demographic data.
Participants were instructed to perform the tasks (see the Participant Instructions below):
• As quickly as possible making as few errors and deviations as possible
• Without assistance
• Without commenting until they had completed each task (if possible)
UserZoom provided participants with instructions before and during each task. Tasks were presented to
each participant in a randomized order to avoid learning bias. Task timing began once the participant clicked
the “Start Task” button in UserZoom. Task timing ended once the participant clicked the “Completed Task”
button in UserZoom. Scoring is discussed in the Data Scoring section.
Following the session, UserZoom displayed the post-test survey, including a System Usability Scale (SUS)
questionnaire and demographic questions. See Appendix 4: System Usability Scale Questionnaire for
additional information on SUS questionnaires. The administrator told participants to expect to receive
compensation in 2-3 weeks.
Test Location
Tests were conducted remotely, with each individual at their own location. Only the participant,
administrator, and data logger were present.
Test Environment The EHRUT typically is used in an emergency department. Tests were conducted remotely, using the
administrator’s computer to run WebEx, Windows Remote Desktop Connection (RDC), and TechSmith
Camtasia (for recording). RDC was used to connect remotely to the EHRUT test environment via a Virtual
Private Network (VPN) using a broadband cable connection with a minimum throughput of 10Mbps
download and 3Mbps upload. Participants used their own PCs during studies. They were given control of
the administrator’s computer to access the application during the session, but used their mouse, keyboard,
and display. (Note, many participants indicated they were not using the computer setup they normally use
with the EHRUT.) Participants PC information was not collected.
3 All participant data has been de-identified and will be kept confidential.
Page 15 of 56
Additionally, participants were told to maximize the WebEx sharing window to full screen and not to change
any default system settings (such as control of font size). They also were told to close any applications
other than the WebEx session.
The administrator’s computer is an HP EliteBook 8570w laptop with a Core i7 Processor, 180GB SSD
hard drive, 16GB RAM, running the Windows 7 Basic Theme. The display resolution was set to
1440x900. This computer ran WebEx screen sharing, the VPN broadband connection, the RDC to the
EHR test environment, and Camtasia.
The EHR test environment on the RDC used Internet Explorer 8.0 (zoom level of 125%). The RDC
displayed at a resolution of 1920x1080 in full screen mode with 24-bit True Color and was set for high-
speed broadband (2-10Mbps) with persistent bitmap caching.
Test Forms and Tools During the usability test, UserZoom presented various documents, including:
1. Informed Consent
Participants were emailed the Informed Consent prior to session so they could review it. At the
beginning of session, they were asked to indicate in UserZoom if they agreed or disagreed with the
terms.
2. Test instructions
3. Post-task Questionnaire
4. Post-test Questionnaire
5. Participant Demographics Questionnaire
Examples of these documents can be found in Appendix 2: Non-Disclosure Agreement and Informed
Consent Form, Appendix 3: Example Moderator’s Guide, and Appendix 4: System Usability Scale
Questionnaire. The Moderator’s Guide was designed to enable the data logger to easily capture data.
The script for participant orientation follows:
Participant Instructions
Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at
Optum, makers of ED PulseCheck. Jamie Peffley (or Scott Stewart) is also on the on line, and they’ll
be helping take notes.
Before we begin, I’d like to take a moment to describe the study we’re conducting.
We’re collecting feedback on a software product called ED PulseCheck.
I know someone already asked you this, but I just wanted to verify that you currently use ED
PulseCheck to work with patient records.
Has not used ED PulseCheck Has used ED PulseCheck
We’re going to ask you to perform a series of tasks. You should know that the tasks were created to
be independent of each other and some may sound redundant. You will be asked to complete these
tasks on your own, trying to do them as quickly as possible with the fewest possible errors or
Page 16 of 56
deviations. Please do not do anything more than the task asks.
You are not being tested on your ability to complete the tasks. We’re evaluating how intuitive the
product is to use, so just do your best to complete them as you typically would. We didn’t have
anything to do with designing the product, and your comments are entirely anonymous, so please
feel free speak candidly. However, please save your comments until the end of a task, or the end of
the session as a whole, when we can then discuss them freely.
We’re going to be using a study tool that will lead you through the tasks. We’d ask that you please
read the task information out loud, so we can follow along.
I’m going to share my screen with you in the WebEx. You’ll be using Ed PulseCheck on my
computer, but I’ll give you control of my computer’s mouse and keyboard for the duration of the
study. Because we can’t always control the bandwidth between our two locations, you may notice a
tiny lag or latency when you’re in control of the mouse and keyboard. However, we don’t anticipate
this will affect your ability to use ED PulseCheck.
Before I do that, please quit or close any other applications that are currently running on your
computer (email and Instant Messaging), with the exception of your browser and WebEx. Also,
please set any mobile devices to silent during the study.
Participants were then presented with tasks one at a time, randomly assigned by UserZoom. See Tasks for
a complete list of tasks.
Usability Metrics According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for
users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this
end, metrics for effectiveness, efficiency, and user satisfaction were captured during the usability testing.
The goals of the test were to assess:
1. Effectiveness of the EHRUT (ED PulseCheck) by measuring participant success rates and errors
2. Efficiency of the EHRUT by measuring the average task time and path deviations
3. Satisfaction with the EHRUT by measuring ease-of-use ratings
Page 17 of 56
Data Scoring The following table details how the testing team scored the tasks, evaluated the errors, and analyzed the time
data.4
Measures Rationale and Scoring
Effectiveness:
Task Success
A task was counted as a “Success” if the participant was able to achieve the correct
outcome, without assistance, within the time allotted.
The total number of successes were calculated for each task, and then the results were
divided by the total number of times that task was attempted. The results are provided as a
percentage.
The total Task times were recorded for successes. Observed task times were divided by the
optimal performance time for each task to provide a measure of optimal efficiency. The
results are provided as a percentage mean.
Optimal task performance time was benchmarked when constructing tasks by using
Keystroke Level Modeling (KLM) to measure the clicks, keystrokes, and task time to obtain
an expected time based on the optimal path. (KLM was used rather than expert time
because the product includes many paths to accomplish the tasks, and experts frequently
didn’t follow optimal paths.) Additional time, based on test environment measurements, was
added to the KLM results to account for the following:
1. User’s need to review information while interacting with the user interface (medical
professionals double-check their work routinely to avoid error)
2. System response times
See Appendix 7: Data Scoring Detail for additional information regarding how the KLM
was calculated.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct answer, or performed it
incorrectly, the task was counted as a “Failure.”
The total number of errors was calculated for each task, and the results were divided by the
total number of times the task was attempted. This was also expressed as the mean
number of failed tasks per participant.
Efficiency:
Task Deviations
The participant’s path (i.e., steps) through the application was recorded. Deviations
occurred if the participant went to a wrong screen, clicked an incorrect menu item, followed
an incorrect link, or interacted incorrectly with an on-screen control. Each participant’s path
was compared to the optimal path. The number of steps in the observed path was divided
by the number of optimal steps to provide a ratio of path deviation.
4 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman. Also see
www.measuringusability.com.
Page 18 of 56
Deviations were defined as:
• Task Recognition: failure to recognize task or completion of a different task
• Data Entry: failure to enter complete data or entry of incorrect data
Efficiency:
Task Time
Each task was timed from when the participant clicked the “Start Task” button until the
participant clicked the “Task Complete” button. Only completion times for successfully
completed tasks were included in the average task time analysis. Average times per task
and variance measures (standard deviation and standard error) were calculated for each task.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease-of-use of the application was measured by
administering a post-task question and a post-session questionnaire.
After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1
(Very Difficult) to 5 (Very Easy). These data are averaged across participants.5 Common
convention is that average ratings for systems judged easy to use should be 3.3 or above.
To measure participants’ confidence in and satisfaction with the EHRUT overall, the testing
team administered the System Usability Scale (SUS) post-test questionnaire. Questions
included, “I think I would like to use this system frequently”, “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.”
See Appendix 4: System Usability Scale Questionnaire for a complete list of SUS
questions.6
Results
Data Analysis and Reporting The results of the usability test were calculated according to the methods specified in the preceding Usability
Metrics section. We excluded data for participants who failed to follow session and task instructions.
We conducted four sessions with physicians and discovered an issue with one task. The original script
asked the participant to administer the medication nitro “as needed,” but there was no “as needed” option in
the application test environment. We removed “as needed” from the script to avoid confusion. Sessions
impacted by this were eliminated from the results. We recruited additional physicians so that 10 physician
results could be included in the analysis.
The usability testing results for the EHRUT are detailed below. The results are presented in the context of
the objectives and goals outlined in Study Design. The report includes actionable results that, if corrected,
should yield a material, positive impact on user performance.
The following table presents data for physicians.
5 See Tedesco and Tullis (2006) for a comparison of post-task ratings for usability tests. Tedesco, D. & Tullis, T. (2006)
A comparison of methods for eliciting post-task subjective ratings in usability testing. Usability Professionals
association Conference, June 12 – 16, Broomfield, CO. 6 The SUS survey yields a single number that represents a composite measure of the overall perceived usability of the
system. SUS scores have a range of 0 to 100 and the score is a relative benchmark that is used against other
iterations of the system.
Page 19 of 56
Physicians
Measure N
Task Success
Path Deviation
Task Time Errors Task
Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
1. Enter laboratory and
radiology/imaging orders 10
70%
(±48%) 1.7
99
(±54) 179%
40%
(±70%)
3.9
(±74%)
2. Enter medication orders 10 50%
(±53%) 2.1
180
(±50) 279%
70%
(±95%)
3.8
(±63%)
3. Cancel or hold a medication
order 10
90%
(±32%) 0.9
44
(±22) 98%
10%
(±32%)
4.6
(±84%)
4. Delete or cancel a
medication order 10
80%
(±42%) 1.5
46
(±19) 133%
20%
(±42%)
4.7
(±67%)
5. Update a medication order 10 90%
(±32%) 1.4
98
(±40) 121%
10%
(±32%)
4.3
(±67%)
6. Update a radiology order 10 90%
(±32%) 1.8
78
(±29) 144%
10%
(±32%)
4.0
(±105%)
7. Approve a medication order
with a drug-allergy reaction 10
80%
(±42%) 3.0
75
(±30) 123%
20%
(±42%)
4.9
(±32%)
8. Approve a medication order
with a drug-drug interaction 10
80%
(±42%) 2.3
76
(±21) 148%
20%
(±42%)
4.5
(±53%)
17. Configure pharmacy for
e-prescription 10
50%
(±53%) 2.4
167
(±56) 207%
50%
(±53%)
3.4
(±97%)
18. Change pharmacy for
e-prescription 10
50%
(±53%) 1.9
184
(±142) 196%
80%
(±103%)
3.7
(±82%)
The SUS scored participants’ subjective satisfaction with the system based on their overall experience with
the EHRUT. The overall score was 86 (±10). Broadly interpreted, scores under 60 represent systems with
poor usability; scores over 80 would be considered above average.7
7 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).
Page 20 of 56
Discussion of the Findings
Findings for each major area’s research question follow.
§170.314(a) (1) Computerized provider order entry
Can a user order medication, laboratory, and radiology / imaging orders? (Task 1-2)
• Participants were more successful entering lab/radiation/imaging orders (70%) than entering a
medication order (50%). Entering medication orders accounted for some of the highest incidences of
data entry errors (70%), primarily because participants frequently didn’t use the clinical pathways or
chose the incorrect clinical pathway (e.g., Data set 1). These two tasks also accounted for some of
the lowest task satisfaction ratings.
Can a user change an order? (Task 3-6)
• Overall, participants had the most success completing these three tasks, the lowest number of
errors, and some of the highest task satisfaction ratings.
§170.314(a) (2) Drug-drug, drug-allergy interaction checks
Can a user order a medication with a drug-allergy interaction? (Task 7)
• While the task success was comparatively high (80%), participants exhibited more path deviations
than with any other task. Compared to the expected path, participants needed three times more
clicks to complete the task, mainly because they were unfamiliar with the schedule and route
dropdown lists and had to search more. (The applications they use in their ERs have customized
dropdown lists.) However, this task had the highest task satisfaction.
• 30% of participants entered medical notations (e.g., PO or one time stat) for the medication
schedule, were unaware that the application didn’t accept the entry, and submitted the order without
a schedule.
Can a user order a medication with a drug-drug interaction? (Task 8)
• The task success (80%) and satisfaction (4.5) was comparatively high, and errors (20%) were
relatively low.
§170.314(b) (3) Electronic prescribing
Can a user issue e-prescriptions? (Task 17-18)
• These two tasks represented the lowest task success ratios (50%) and had comparatively high error
rates. These tasks have the lowest task satisfaction scores primarily because search results were
very difficult to obtain, and the results display was inconsistent. Participants entered correct data, but
if they included apostrophes or capital letters, no results appeared. Also, sometimes results
appeared “below the fold” of the pop-up window, and participants didn’t realize that they needed to
scroll down.
Page 21 of 56
Effectiveness
Task 1
• 20% of participants entered the incorrect Gas & Carbon panel.
Task 2
• 20% of participants didn’t enter the schedule for one (or both) of the medications.
• 30% of participants entered medical notations (e.g., PO or one time stat) for the medication
schedule, were unaware the application didn’t accept the entry, and submitted the order without a
schedule.
Tasks 3-6
• These four tasks represented the highest task success ratios.
o Task 3: 90%
o Task 4: 80%
o Task 5: 90%
o Task 6: 90%
Task 17-18
• 50% of participants failed to complete these two tasks. This was because: (1) search did not accept
entries that included capitalization or apostrophes and (2) results occasionally were “below the fold”
of the pop-up window where participants didn’t notice to would have needed to scroll down.
Half of the participants who successfully completed the task made multiple attempts entering search
criteria.
Efficiency
In general, participants who were the most efficient understood that they could type the first letter or two into
dropdown lists to narrow their results. The user interface does not indicate that this is possible.
The greatest variability in the actual task time vs. expected task time was due to the following:
• Use of Search instead of Quicklists for medications
• Unfamiliarity with dropdown list items and length
• Limited use of navigational shortcuts (e.g., GO button)
Task 1
• Average task time was 79% longer than expected because participants either didn’t use the
predefined clinical pathway (Abdominal and chest pain) or selected the “Data Set 1” or “Data Set 2”
clinical pathway. Many opted to search for the orders. This was likely a result of the participants’
unfamiliarity with the test environment data and customizations.
Page 22 of 56
Task 2
• 70% of participants searched for medications, rather than using Quicklists. As a result, the average
task time was 179% longer than expected, higher than any other task, and they had more than twice
the path deviations than the optimal path. Not using Groups accounted for a 70% error rate,
primarily because the participants then had to enter the medication schedules manually. Fifty per
cent of participants either didn’t enter a medication schedule or entered a medical abbreviation (e.g.,
PO or one time stat) which the EHRUT did not accept. The EHRUT did not alert the participants,
who thought they had submitted the medication order correctly.
Task 3-6
• These four tasks represented the lowest average error rates. Participants were generally able to
successfully complete the task at, or very close to, the expected task time.
o Task 3:
Errors: 10% (Only an average of 10% of participants made errors)
Task Time: -2% (Participants, on average, performed this task about the same, but
slightly faster than, expected)
o Task 4:
Errors: 20%
Task Time: 33% longer than expected, primarily because some participants used the
“cancel” link rather than “delete.” This added an additional step (and click)
o Task 5:
Errors: 10%
Task Time: 21% longer than expected as a result of inefficiencies typing the search
entry for penicillin. Some participants either misspelled or didn’t type enough letters
using “filter/find as you type” to easily find the item in the dropdown list
o Task 6:
Errors: 10%
Task Time: 44% longer than expected. When performing a search for “wrist,” some
participants either misspelled or didn’t type enough letters using “filter/find as you
type” to easily find the item in the dropdown list
Task 7
While participants’ task times were only slightly longer (23%) than expected, there were three times
the path deviations than was optimal. This was a direct result of inefficiencies typing the search entry
for “Vicodin ES.” Most participants either misspelled or didn’t type enough letters using “filter/find as
you type” to easily find the item in the dropdown list
Page 23 of 56
Task 8
• The average task time was 48% longer than expected, because most participants used “Search”
rather than “Groups” to enter the medication. This required them to manually enter the dose, unit,
and route.
Task 17-18
• Because participants had difficulty finding the Search link, which was below the fold, the task times
for task 17 and 18 were higher than any other task. Participants took 107% (for task 17) and 96%
(for task 18) longer than the expected task times. Successful participants attempted the search
several times by editing the search parameters until they achieved a result.
• This was the first time any participants had used the e-prescription search, so they weren’t familiar
with its location. These task times should improve with greater familiarity and improved flexibility in
search entries.
Satisfaction
Task 17-18
• These two tasks represented the lowest satisfaction score, which is likely due to participants’
frustration with search results not being returned as expected.
Major Findings
1. Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness
2. Participants consistently reported above average levels of product satisfaction. Many reported that
of all the similar applications they’ve used, this was “the most intuitive and easy to use.” Other
comments were:
"Initially, I was skeptical about (ED PulseCheck) but a month after using it I found I was faster than
(hand-written notes and dictated orders).”
“I’ve grown fond of the product even though I “came to the dance” reluctantly.”
3. Participants often used mixed methods to navigate. Participants who used the GO button would
occasionally also use the top menu. Using the GO button is more efficient.
4. Multiple and often less efficient navigational paths are used to accomplish tasks, increasing users’
cognitive load.
5. Only 50% of participants successfully completed Tasks 17 and 18 (relating to e-prescription search)
because the search results were not provided or were not visible. Those participants that were
unsuccessful made multiple attempts to change the search entry before they gave up.
E-prescription search did not accept search entries that included capitalization or apostrophes. For
example, an entry of “Walgreen’s”, which is correct, did not yield results, while an entry of
“walgreens” did yield results. Results sometimes displayed “below the fold” of the pop-up window. It
wasn’t evident to participants that they needed to scroll down to view them.
Participants were very encouraged with the prospect of being able to enter e-prescriptions.
Page 24 of 56
6. Data was lost and record updates were incomplete. This occurred when participants entered
medical notations (e.g., PO or one time stat) for the medication routes or schedule, were unaware
the application didn’t accept it, and submitted the order without the entry.
7. Because action buttons are too close to each other, participants accidently clicked (or hovered over
and nearly clicked) the Print Triage button rather than Enter button. The proximity and similarity of
various buttons (e.g., Print Triage, Enter/Task, Enter/Med, Repeat) conflicts with the Enter button.
8. Dropdown lists displayed an inconsistent number of items depending on where they were on the
page. For example, if the dropdown list for Quicklist medication schedules was towards the bottom
of the page, it displayed 9-10 items, whereas, if the dropdown was near the middle, it displayed
approximately 25 items.
9. Regarding medication orders, one participant commented, "I checked 'bulk' (medication for one of
the tasks). It's the same mistake I make at work. Sometimes those differentiations don't mean
anything to a physician. I just delete the medication, go back, and search again."
10. In general, task times and path deviations that were higher than anticipated may have been because
participants normally have localized customizations (e.g., predefined pathways, drug lists, and
schedules) but those familiar customizations weren’t available in the test environment. (They are
specific to each ER.) Using an EHR with the customizations participants are familiar with would likely
improve task times and reduce path deviations to more closely match the expected task times and
optimal paths.
One participant said “90-95% of the meds I use are on my Quicklist and 70-75% of the time they're
on my summary page. It's the same with orders. It's a wonderful shortcut."
11. One participant suggested a feature request: “I want to be able to draw on a picture [to show] big
lacerations on a body.”
12. Three participants suggested that customized order set personalization (by a user) would be very
useful and more efficient.
Page 25 of 56
RNs
Measure
N Task
Success Path Deviation Task Time Errors
Task Ratings (5=Easy)
Task # Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Deviations (Observed /
Optimal)
Mean (SD)
Mean (SD)
10. Enter active medications 13 60%
(±52%) 2.3
172 (±29)
235% 80%
(±104%) 4.6
(±70%)
11. Update a medication
currently being taken 13
69% (±48%)
1.8 120
(±45) 187%
54% (±88%)
3.9 (±126%)
12. Inactivate an active
medication 13
85% (±38%)
2.0 114
(±30) 165%
15% (±38%)
4.4 (±96%)
13. Find a list of active and
inactive allergies 13
46% (±52%)
1.4 59
(±25) 198%
62% (±51%)
4.3 (±125%)
14. Enter active allergies 13 92%
(±28%) 1.4
113 (±33)
145% 31%
(±85%) 4.5
(±78%)
15. Update the effects of an
allergy 13
77% (±44%)
1.4 82
(±57) 133%
31% (±48%)
4.4 (±119%)
16. Inactivate an allergy 13 69%
(±48%) 1.2
98 (±32)
128% 23%
(±44%) 4.7
(±48%)
The SUS scored participants’ subjective satisfaction with the system based on their overall experience with
the EHRUT. The overall score was 87 (±17). Broadly interpreted, scores under 60 represent systems with
poor usability; scores over 80 would be considered above average.8
Discussion of the Findings
Findings for each major area’s research questions follow.
§170.314(a)(6) Medication list
Can a user enter active medications? (Task 10)
• This task resulted in the highest error rate (80%) and over twice the number of path deviations than
the optimal path, primarily due to incorrect medication being updated. Task satisfaction was the
second highest (4.6), likely due to the ease with which the EHRUT allows participants to efficiently
and effectively enter active medications.
8 See Tullis, T. & Albert, W. (2008). Measuring the User Experience. Burlington, MA: Morgan Kaufman (p. 149).
Page 26 of 56
Can a user update active medications? (Task 11-12)
• 69% of participants successfully completed Task 11 and did so in nearly the same as the expected
task time. However, this task had the lowest task satisfaction (3.9), primarily because participants
had difficulty finding the control needed to update the medication schedule. This is a line with no text
that is a hyperlink; its functionality is not obvious.
• 85% of participants successfully completed Task 12 and had the fewest errors (15%). However,
there were two times more path deviations (clicks) than the optimal path, primarily because
participants used search rather than the Quicklist to find the medication.
§170.314(a)(7) Medication allergy list
Can a user find a list of active and inactive allergies? (Task 13)
• 54% could not complete the task, primarily because participants couldn’t find the “Show Inactive” link
to display the list of active and inactive allergies. This looks like a column heading rather than a link
or button, and the instructional test is not close to the control.
Can a user enter active allergies? (Task 14)
• 92% of participants successfully completed this task.
Can a user update active allergies? (Task 15-16)
• 77% of participants successfully completed Task 15.
• 69% successfully completed Task 16. However, it took 28% longer than expected, primarily because
several participants used Search to find the medication rather than the Quicklist.
Effectiveness
Task 10
• Low task success (60%) was primarily due to data entry errors (80%). Participants:
o Added notes to the wrong medication
o Selected an extra medication
o Recorded a schedule in the notes field
o Did not enter a schedule or entered it incorrectly.
Task 11
• Errors were made by 54% of participants which included:
o Choosing a medication unit "mcg" rather than "mg"
o Clicked on "Make Inactive'
o Clicked Print Triage Report rather than Enter. This resulted in the entries disappearing. The
EHRUT did not alert the user.
Page 27 of 56
Task 13
• 54% of participants couldn't find the "Show Inactive" link and gave up without completing the task.
Task 14
• 92% of participants successfully completed this task.
Efficiency
• In general, the most efficient participants understood they could type the first letter or two into
dropdown lists to narrow their results. The user interface does not indicate that this functionality is
available.
• The greatest variability in the actual task time vs. expected task time was due to:
o Use of Search instead of Quicklists for medications
o Unfamiliarity with the dropdown lists items and length
o Limited use of navigational shortcuts (e.g., GO button)
• Longer than expected task times for Tasks 10, 11, 12, and 16 were primarily due to participants’ use
of Search rather than the Quicklist. Participants didn't notice that the medications they needed to
enter were in the Quicklist.
o In task 10, some participants had to navigate extremely long lists in the schedule dropdown.
If the participant didn't "type ahead" to narrow the list, they had to scan as many as 36
choices.
o In task 11, several participants had difficulty finding the hyperlink line, causing longer than
expected task times. One participant couldn’t figure out how to modify the current
medication, so they inactivated it and created a new entry.
• 46% of participants weren't familiar with or couldn't find the "Show Inactive" link in Task 13. Those
that were successful, but unfamiliar with it, eventually found it after reading the instructional note
below the table.
• The higher number of path deviations (compared to optimal) for Tasks 10, 12, and 16 was primarily
due to participants’ use of Search rather than the Quicklist. Most participants didn't notice that the
medications they needed to enter were in the Quicklist.
Satisfaction
• Task 11: The lower satisfaction rating was due to participants’ difficulty finding the control, which
was a hyperlinked line with no text.
Major Findings 1. Overall, participants were satisfied with the EHRUT’s workflow, efficiency, and effectiveness.
2. Participants expressed above average level of product satisfaction.
3. Significant efficiencies when participants used Quicklists and filter/find as you type (FAYT).
Page 28 of 56
4. The EHRUT has multiple navigation paths that participants can use to complete tasks; participants
used a variety of the methods available (in some cases, this resulted in slower task times). For
example, some participants clicked the patient radio button (which is unnecessary) before they used
the GO button.
5. Participants often used mixed methods to navigate. Participants who used the GO button would
occasionally also use the top menu. Using the GO button is more efficient.
6. Participants had difficulty finding “Show inactive” allergies.
7. Because action buttons are too close to each other, participants accidently clicked (or hovered over
and nearly clicked) the Print Triage button rather than Enter button. The proximity and similarity of
the Print Triage button conflicts with the Enter button.
8. Participants had difficulty identifying the control (hyperlinked line) for editing an allergy. Participants
frequently clicked on the drug name link, expecting to be able to edit. This displayed a drug
information help window.
9. Dropdown lists displayed an inconsistent number of items depending on where they were on the
page. For example, if the dropdown list for Quicklist medication schedules was towards the bottom
of the page, it displayed 9-10 items, whereas, if the dropdown was near the middle, it displayed
approximately 25 items.
10. Pharmacy search parameters were too restrictive (case-sensitive and not permitting punctuation).
Results sometimes required scrolling to view.
11. Users are not made aware when the system deletes unacceptable entries, resulting in data loss.
12. Users are unaware of “filter/find as you type” that would increase their efficiency.
Combined (Physician and RN) Areas for Improvement
1. Continue to emphasize to users that the GO button will improve their efficiency.9
2. Make “Show inactive” more visually identifiable.
3. Help users distinguish and click on the desired action button; consider the proximity and order of
importance.
4. Don’t rely on hyperlinked lines for navigation; use text.
5. Provide consistent behavior regarding how many items a dropdown list will display regardless of
where the list is clicked on in the page.
6. Provide more flexibility in the e-prescription search entry; allow for punctuation and capital letters.
Consistently present the results where users can see them without needing to scroll.
7. Allow entry of medical notations (e.g., PO or one time stat) for the medication routes or schedules to
avoid data loss, or alert users before loss of data.
8. Provide hints for “filter/find as you type” functionality.
9 Cognitive modeling indicates that users will save an estimated .5/sec. by using the GO button vs. the top-level menu.
When .5/sec. is extrapolated over the number of times a single user interacts with patient clinical documents using the GO button, over the course of a year, the efficiency improvement can be significant.
Page 29 of 56
Appendices The following appendices include supplemental data for this usability test report. Following is a list of the
appendices provided:
1. Participant demographics
2. Non-Disclosure Agreement (NDA) and Informed Consent Form
3. Example Moderator’s Guide
4. System Usability Scale Questionnaire
5. Tasks
6. Demographics Questionnaire
Page 30 of 56
Appendix 1: Participant Demographics Following is a high-level overview of the participants in this study.
Physicians
The following table summarizes the characteristics of the physicians that participated in the usability test,
including demographics, professional experience, computing experience and use of assistive technology.
Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to
individual identities.
Part ID
Part. ID
Gender Age Education Occupation Professional Experience
Computer Experience
Product Experience
Assistive Technologies
1 P4 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 20-29
years
1-3 years No
2 P11 Male 50-59 Professional
degree (e.g.
MD)
Physician 20-29 years 7-9 years 4-6 years Yes, Dragon
3 P13 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 10-19
years
7-9 years No
4 P18 Male 50-59 Professional
degree (e.g.
MD)
Physician 10-19 years 20-29
years
4-6 years No
5 P25 Female 50-59 Professional
degree (e.g.
MD)
Physician 20-29 years 7-9 years 4-6 years No
6 P26 Male 60-69 Professional
degree (e.g.
MD)
Physician 30-39 years 20-29
years
1-3 years No
7 P14 Male 60-69 Professional
degree (e.g.
MD)
Physician 20-29 years 7-9 years Less than 1
year
No
8 P29 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 10-19
years
4-6 years No
9 P28 Male 30-39 Professional
degree (e.g.
MD)
Physician 7-9 years 4-6 years 4-6 years Yes, Dragon
Page 31 of 56
10 P23 Male 30-39 Professional
degree (e.g.
MD)
Physician 4-6 years 7-9 years 4-6 years No
RNs
The following table summarizes the characteristics of the RNs that participated in the usability test, including
demographics, professional experience, computing experience and use of assistive technology. Participant
names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual
identities.
Part
ID
Part.
ID Gender Age Education Occupation
Professional
Experience
Computer
Experience
Product
Experience
Assistive
Technologies
1 P2 Female 30-39 Associate’s
degree
Registered
Nurse 10-19 years 10-19 years 7-9 years No
2 P3 Female 40-49 Doctoral
degree
Nurse
Practitioner 10-19 years 10-19 years 1-3 years No
3 P5 Male 50-59 Bachelor’s
degree
Registered
Nurse 7-9 years 10-19 years 7-9 years No
4 P6 Male 40-49 Bachelor’s
degree
Registered
Nurse 10-19 years 10-19 years 1-3 years No
5 P8 Female 30-39 Bachelor’s
degree
Registered
Nurse 1-3 years 7-9 years 1-3 years No
6 P9 Female 30-39 Associate’s
degree
Registered
Nurse 4-6 years 1-3 years
Less than 1
year Yes, Dragon
7 P12 Female 40-49 Bachelor’s
degree
Registered
Nurse 20-29 years 20-29 years 7-9 years No
8 P19 Male 60-69 Associate’s
degree
Registered
Nurse 20-29 years 4-6 years 4-6 years No
9 P20 Female 40-49 Bachelor’s
degree
Registered
Nurse 10-19 years 10-19 years 1-3 years No
10 P21 Male 40-49 Bachelor’s
degree
Registered
Nurse 4-6 years 10-19 years 4-6 years Yes, Dragon
11 P22 Female 40-49 Bachelor’s
degree
Registered
Nurse 20-29 years 20-29 years 4-6 years No
12 P15 Female 18-29 Bachelor’s
degree
Registered
Nurse 1-3 years 4-6 years 4-6 years No
13 P27 Male 40-49 Bachelor’s
degree
Registered
Nurse 10-19 years 10-19 years 4-6 years No
Page 32 of 56
Appendix 2: Non-Disclosure Agreement and Informed Consent Form Understanding Your Participation
I agree to participate in this study conducted by Optum.
I understand Optum will record my participation for later review. By signing/acknowledging this form, I give
Optum permission to use the data gathered and my comments for the purposes of evaluating the product in
the study and showing the results of these evaluations, combined with other participants, to the product
team. Optum will not include my name or any identifying characteristics in the results.
Any information I acquire about the product in the study is confidential and proprietary and is being
disclosed only so that I can participate in the evaluation. By signing this form, I agree not to talk about this
product to anyone. I may only tell others that I helped to evaluate the software. I understand that if I disclose
my health information in my comments, I have disclosed it voluntarily, and Optum may use it anonymously
for research and evaluation purposes only.
I understand that I can stop participating in this study at any time.
I agree to raise any concerns or areas of discomfort with the study moderator.
Name (print): __________________________________________________
Signature: __________________________________________________
Date: __________________________________________________
Page 33 of 56
Appendix 3: Example Moderator’s Guide for Physicians and RNs
Thank you for joining us today; we really appreciate your time. I’m John Romadka; I work at Optum,
makers of ED PulseCheck. Jamie Peffley (or Scott Stewart) is also on the on line, and she’ll be helping take
notes.
Before we begin, I’d like to take a moment to describe the study we’re conducting.
We’re collecting feedback on a software product called ED PulseCheck.
I know someone already asked you this but I just wanted to verify that you currently use ED PulseCheck to
work with patient records.
Has not used ED PulseCheck Has used ED PulseCheck
We’re going to ask you to perform a series of tasks. You should know that the tasks were created to be
independent of each other and some may sound redundant. You will be asked to complete these tasks on
your own, trying to do them as quickly as possible with the fewest possible errors or deviations. Please do
not do anything more than the tasks asks.
You are not being tested on your ability to complete the tasks. We’re evaluating how intuitive the product is
to use, so just do your best to complete them as you typically would. We didn’t have anything to do with
designing the product, and your comments are entirely anonymous, so please feel free speak candidly.
However, please save your comments until the end of a task, or the end of the session as a whole, when
we can then discuss them freely.
We’re going to be using a study tool that will lead you through the tasks. We’d ask that you please read this
task information out loud, so we can follow along.
I’m going to share my screen with you in the WebEx. You’ll be using Ed PulseCheck on my computer, but
I’ll give you control of my computers mouse and keyboard for the duration of the study. Because we can’t
always control the bandwidth between our two locations, you may notice a tiny lag or latency when you’re in
control of the mouse and keyboard. However, we don’t anticipate this will affect your ability to use ED
PulseCheck.
Before I do that, please quit or close any other applications that are currently running on your computer
(email and Instant Messaging), with the exception of your browser and WebEx running. Also, please set any
mobile devices to silent during the study.
Page 34 of 56
Appendix 4: System Usability Scale Questionnaire In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems
usability” known as the System Usability Scale or SUS.16 Lewis and Sauro (2009) and others have
elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper,
at http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).
16 Brooke, J.: SUS: A “quick and dirty” usability scale. In: Jordan, P. W., Thomas, B., Weerdmeester, B. A., McClelland (eds.) Usability Evaluation in Industry pp. 189--194. Taylor & Francis, London, UK (1996). SUS is copyrighted to Digital Equipment Corporation, 1986.
Lewis, J R & Sauro, J. (2009) "The Factor Structure Of The System Usability Scale." in Proceedings of the Human Computer Interaction International Conference (HCII 2009), San Diego CA, USA
Page 35 of 56
Appendix 5: Tasks
Physicians
There are multiple correct paths to successfully complete the tasks; however, the “optimal path” is indicated
in bold.
Task 0 – Login Role: Physician
Starting URL: http://localhost/ibex23.mpex
For the purpose of this study, you are Chris Huffman, a medical professional working in the emergency
department of PulseCheck Hospital. The emergency department uses ED PulseCheck to triage incoming
patients. Your ED PulseCheck information is below.
Username: chuffman
Password: 123
Log in to ED PulseCheck and view the tracking board.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
No time tracking required.
No completion required.
Page 36 of 56
Task 1 - Enter a laboratory, radiology, and imaging orders
Robert Michael O’Brien was brought to the emergency department for abdominal and chest pain. A patient
record has already been created for Robert in ED PulseCheck and has been assigned to you.
Enter the following lab, radiology, and imaging orders:
Complete blood count (CBC)
Hemoglobin A1c/hemoglobin total
Gas & carbon monoxide panel
CT abdomen & pelvis w/ contrast material
Myocardial image PET perfus single study rest/stress
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Orders page.
a. Directly from the Tracking Board page
i. Select the patient > hover over All on the menu bar > click Orders
ii. Select the patient > click Order on the menu bar
iii. Select the patient > click Go > Orders on the patient row
b. Via the My Patients page (selecting the patient first is not necessary)
i. Click All on the menu bar > My Patients > select patient if necessary > Orders
ii. Click My Patients on the menu bar > select patient if necessary > Orders
2. Enter orders.
i. Select orders from Abdominal and chest pain clinical pathway
ii. Search for individual order by name
Successful Completion: User clicks Enter, Enter/Task, Enter/Med, or Repeat after entering the orders.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 37 of 56
Task 2 - Enter medication orders
Robert Michael O’Brien was brought to the emergency department for abdominal and chest pain. A patient
record has already been created for Robert in ED PulseCheck and assigned to you.
Task: Enter the following medication orders:
Digoxin 0.25 mg tablet by mouth once daily
Morphine 2 mg IV push x1 for pain , every 4 hours as needed
Nitro-bid oral 2.5 mg oral, as needed
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab
2. Enter orders.
a. Click Group>Abdominal & chest pain; select medications > Enter
b. Search for individual medications by name > Enter
Successful Completion Criteria: User clicks Enter after the orders are entered.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 38 of 56
Task 3 – Cancel or hold a medication order
Robert Michael O’Brien was brought to the emergency department for abdominal and chest pain. A patient
record was created for Robert in ED PulseCheck and assigned to you. You ordered Aspir-81 for Robert
earlier but he is refusing the medication.
Task: Update Robert’s medication orders to reflect the fact that he is refusing the medication. Enter a note
stating, “Patient refused.”
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the Medication Services tab
2. Cancel medication
a. Locate medication > click Cancel link > check Medication refused > clicks checkbox “Patent refused”
>click Cancel button
b. Locate medication > click Hold link > check Medication refused > enter note >click Hold button
Successful Completion Criteria: User clicks Cancel or Hold button after entering a note.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 39 of 56
Task 4 – Delete or cancel a medication order Role: Physician w/ experience entering medication orders
Monica Webster was brought to the emergency department for a laceration to her left hand. It has come to
your attention that you accidently ordered Norcuron 10 mg IVP for her instead of your other patient.
Task: Update Monica’s medication orders to reflect the fact that Norcuron 10 mg IVP is not needed.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the
Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the
Medication Services tab
2. Delete medication
i. Locate medication > click Delete link
ii. Locate medication > click Cancel link > click Cancel button
Successful Completion: User clicks Delete link or Cancel button.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 40 of 56
Task 5 – Update a medication order Role: Physician w/ experience entering medication orders
Javier Knight was brought to the emergency department for a sore throat. It has come to your attention that
Penicillin V Potassium 125mg/5ml oral was inadvertently ordered for Javier instead of Penicillin G
Potassium 1 million units, IM.
Task: Update Javier’s medication orders to reflect the fact that Penicillin G Potassium 1 million units, IM
should be given.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the
Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the
Medication Services tab
2. Cancel, delete, or hold medication.
a. Cancel: Locate medication > click Cancel > check Medication refused > enter note
b. Delete: Locate medication > click Delete
c. Hold: Locate medication > click Hold > check Medication refused > enter note
3. Enter order.
a. Search for medication (Penicillin) by name.
Successful Completion: User clicks Enter, Enter/Task, Enter/Med, or Repeat after entering medication order with
correct dosage.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 41 of 56
Task 6 –Update a radiology order Role: Physician w/ experience entering lab and imaging orders
John Smith was brought to the emergency department for an injury to his right wrist. A patient record was
created for John in ED PulseCheck and assigned to you. John needs a 2 view wrist x-ray but a 2 view
chest x-ray was accidently ordered for him.
Task: Update John’s record to reflect the fact that he needs a 2 view wrist x-ray rather than chest x-rays.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
Navigate to the My Tasks page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Orders
ii. Select the patient > click Order on the menu bar
iii. Select the patient > click Go > Orders on the patient row
iv. Click the Orders icon
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > Orders
ii. Click My Patients on the menu bar > select patient if necessary > Orders
2. Cancel the chest x-ray.
i. Click Cancel
3. Click Orders
4. Order the wrist x-ray.
i. Search by SVC Name > type wrist > select wrist or wrist (2 views) from list
Successful Completion: Depends on order in which user completes tasks. User can click Enter after entering the wrist
x-ray or click Cancel after selecting the chest x-ray.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 42 of 56
Task 7 – Approve a medication order with a drug-allergy reaction Role: Physician w/ experience entering medication orders
John Smith was brought to the emergency department for an injury to his right wrist. A patient record has
already been created for John in ED PulseCheck and assigned to you. John is allergic to Tylenol but has
been treated with Vicodin in the past and has responded well with no side effects.
Task: Enter an order for one tablet of Vicodin ES for John.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the
Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the
Medication Services tab
2. From the Medication Services page, search for medication by name.
3. In Allergy Reaction alert, check Patient tolerated same medication(s) in the past.
4. Click Enter.
Successful Completion: Interaction alert has been acknowledged and user clicks Enter.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 43 of 56
Task 8 – Approve a medication order with a drug-drug interaction Role: Physician w/ experience entering medication orders
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already
been created for Zackary in ED PulseCheck and assigned to you. Zackary is currently taking HCTZ. You
believe the benefits of taking lithium carbonate outweigh the risks associated with drug interactions.
Task: Order two lithium carbonate 150mg capsules for Zackary.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Medication Services page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Med SVC
ii. Select the patient > click Med SVC on the menu bar
iii. Select the patient > click Go > Med SVC on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Medication Services
ii. Click My Patients on the menu bar > select patient if necessary > click Medication Services
c. Via the Orders page:
i. Select the patient > hover over All on the menu bar > click Orders > click the Medication Services tab
ii. Select the patient > click Orders on the menu bar > click the Medication Services tab
iii. Select the patient > click Go > Orders on the patient row > click the Medication Services tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Orders > click the
Medication Services tab
v. Click My Patients on the menu bar > select patient if necessary > click Orders > click the
Medication Services tab
2. Enter orders.
i. Use Groups to enter medications
ii. Search for individual medications by name
iii. In Severe Interaction alert for HCTZ and lithium carbonate, check Benefits outweigh risks.
iv. Click Enter
Successful Completion: User clicks Enter after the orders are entered. Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 44 of 56
Task 17 - Configure pharmacy for e-prescription Roles: Physician
Janet Kikura was brought to the emergency department for a severe migraine headache. A patient record
has already been created for Janet in ED PulseCheck and assigned to you. You have learned that you can
use the newest version of ED PulseCheck to send an electronic prescription to Janet’s pharmacy so it will
be available for her to pick up on the way home.
Task: Use ED PulseCheck to write a prescription for Ibuprofen 800mg, 1 tablet taken orally every 8 hours
as needed, no refills, 20 tablets to be dispensed. The closest pharmacy to Janet’s home is Walgreen’s,
located at 930 Elk Grove Town Ctr. Elk Grove Village, IL.
**Formulary information is not currently available. You can ignore any message regarding formulary
information.**
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Prescriptions page.
a. To access the Prescriptions page from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Rx
ii. Select the patient > click Rx on the menu bar
iii. Select the patient > click Go > Rx on the patient row
b. To access the Rx page from the My Patients page (selecting the patient first is not necessary):
i. Select patient (if necessary) > hover over All on the menu bar > click Rx
ii. Select patient (if necessary) > click Rx on the menu bar
iii. Select patient (if necessary) > click Prescriptions in the right panel
c. To access the Rx page from the Disposition page:
i. Scroll to bottom > click Rx button
2. Add medication.
a. Find medication
i. Click Search All ii. Type medication name
iii. Select medication from list
b. Enter dose, schedule, and dispense instructions.
3. Send to pharmacy.
i. Click Pharmacy
ii. From the Pharmacy popup, enter pharmacy name
iii. Select pharmacy from list
iv. Click OK
v. Click Enter Successful Completion: User clicks Enter after selecting medication and pharmacy.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 45 of 56
Task 18 - Change pharmacy for e-prescription Role: Physician
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already
been created for Zackary in ED PulseCheck and assigned to you. You have learned that you can use the
newest version of ED PulseCheck to send an electronic prescription to Zackary’s pharmacy so it will be
available for him to pick up on the way home.
Task: Zackary usually fills prescriptions at Osco Drug, located on 6200 W. Higgins Avenue Chicago IL,
which is close to his home. He is planning on staying with a family member when he is discharged and
would like to fill his prescription at another pharmacy instead. Use ED PulseCheck to write a prescription
for Amoxicillin (SUSPENSION, RECONSTITUTED, ORAL (ML) : 125 mg/5 mL ) ; 5mls every 6 hours ;
dispense 1 bottle and send the prescription to Costco Pharmacy located at 505 W. Army Trail Road
Bloomingdale IL.
**Formulary information is not currently available. You can ignore any message regarding formulary
information.**
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Prescriptions page.
a. To access the Prescriptions page from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Rx
ii. Select the patient > click Rx on the menu bar
iii. Select the patient > click Go > Rx on the patient row
b. To access the Rx page from the My Patients page (selecting the patient first is not necessary):
i. Select patient (if necessary) > hover over All on the menu bar > click Rx
ii. Select patient (if necessary) > click Rx on the menu bar
iii. Select patient (if necessary) > click Prescriptions in the right panel
c. To access the Rx page from the Disposition page:
i. Scroll to bottom > click Rx button
2. Send to pharmacy.
i. Click Pharmacy
ii. From the Pharmacy popup, enter pharmacy name
iii. Select pharmacy from list
iv. Click OK
v. Click Enter
Successful Completion: User clicks Enter after selecting medication and pharmacy.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 46 of 56
RNs
There are multiple correct paths to successfully complete the tasks; however the “optimal path” is indicated in bold.
Task 0 – Login Starting URL: http://localhost/ibex23.mpex
For the purpose of this study, you are Chris Huffman, a medical professional working in the emergency
department of PulseCheck Hospital. The emergency department uses ED PulseCheck to triage incoming
patients. Your ED PulseCheck information is below.
Username: chuffman
Password: 123
Log in to ED PulseCheck and view the tracking board.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
No time tracking required.
No completion required.
Page 47 of 56
Task 10 - Enter active medications
Susan McCoy was brought to the emergency department for stomach pain that was previously diagnosed
as gastro esophageal reflux disease (GERD). A patient record has already been created for Susan in ED
PulseCheck and assigned to you. Susan neglected to mention that she takes several over-the-counter
medications regularly for GERD.
Task: Update Susan’s record to reflect the medications she takes.
Pepcid AC 10mg chewables, 2 daily
TUMS E-X 300mg chewables, 4 daily
Zegerid OTC 20mg capsule, 1 daily
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Current Medications page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Current Meds
ii. Select the patient > click Current Meds on the menu bar
iii. Select the patient > click Go > Current Meds on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Current Meds
ii. Click My Patients on the menu bar > select patient if necessary > click Current Meds
c. Via the patient’s Charting page:
i. Click the patient’s name > click Current Medications (only valid if current medications have not been
previously entered)
d. Via the Allergies page:
i. Select the patient > hover over All on the menu bar > click Allergies > click Medications tab
ii. Select the patient > click Allergies on the menu bar > click Medications tab
iii. Select the patient > click Go > Allergies on the patient row > click Medications tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Allergies > click
Medications tab
v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click Medications
tab
vi. Click the patient’s name > click Allergies (only valid if current medications have not been previously
entered) > click Medications tab
2. Enter medications.
a. Use the QuickList to add the medications > click Enter
Successful Completion Criteria: User clicks Enter or Enter/Allergies after entering three medications.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 48 of 56
Task 11 - Update a medication currently being taken Role: Nurse
Zackary A Pearson was brought to the emergency department for fever. A patient record has already been
created for Zackary in ED PulseCheck and assigned to you. Zachary said he took digoxin but neglected to
say how often he took it.
Task: Update Zachary’s record to reflect the fact that he takes digoxin, orally, 0.25 mg once a day (in the
morning).
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Current Medications page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Current Meds
ii. Select the patient > click Current Meds on the menu bar
iii. Select the patient > click Go > Current Meds on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Current Meds
ii. Click My Patients on the menu bar > select patient if necessary > click Current Meds
c. Via the patient’s Charting page:
i. Click the patient’s name > click Current Medications (only valid if current medications have not
been previously entered)
d. Via the Allergies page:
i. Select the patient > hover over All on the menu bar > click Allergies > click Medications tab
ii. Select the patient > click Allergies on the menu bar > click Medications tab
iii. Select the patient > click Go > Allergies on the patient row > click Medications tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Allergies > click
Medications tab
v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click
Medications tab
vi. Click the patient’s name > click Allergies (only valid if current medications have not been
previously entered) > click Medications tab
2. Add dosage information.
i. Click on the hyperlinked line to the right of the medication name
ii. In the Schedule field, select once a day
iii. Click Update
Successful Completion: User clicks Update after entering dosage.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 49 of 56
Task 12 - Inactivate an active medication Role: Nurse
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already
been created for Zackary in ED PulseCheck and assigned to you. Earlier, Zackary said he took Zegerid
OTC once a day. A family member has arrived at the hospital with his medications and you learn that
Zackary is taking Tums EX, not Zegerid OTC.
Task: Update Zackary’s record to reflect the fact he is taking Tums EX 300 mg calcium (750 mg), 1 tab
oral, once a day, not Zegerid OTC 20 mg-1.1 gram 1 tab oral once a day.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Current Medications page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Current Meds
ii. Select the patient > click Current Meds on the menu bar
iii. Select the patient > click Go > Current Meds on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Current Meds
ii. Click My Patients on the menu bar > select patient if necessary > click Current Meds
c. Via the patient’s Charting page:
i. Click the patient’s name > click Current Medications (only valid if current medications have not
been previously entered)
d. Via the Allergies page:
i. Select the patient > hover over All on the menu bar > click Allergies > click Medications tab
ii. Select the patient > click Allergies on the menu bar > click Medications tab
iii. Select the patient > click Go > Allergies on the patient row > click Medications tab
iv. Click All on the menu bar > My Patients > select patient if necessary > click Allergies > click
Medications tab
v. Click My Patients on the menu bar > select patient if necessary > click Allergies > click
Medications tab
vi. Click the patient’s name > click Allergies (only valid if current medications have not been
previously entered) > click Medications tab
2. Inactivate medication.
i. Click on Make inactive to the right of Zegerid
3. Add medication.
i. Type medication name in the Quick List tab > enter dosage and schedule information >
click Enter ii. Type medication name in the Search tab > enter dosage and schedule information > click
Enter Successful Completion: User clicks Enter after adding medication.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 50 of 56
Task 13 – Find active and inactive allergies Role: Nurse
Zackary A Pearson was brought to the emergency department for a fever. A patient record has already
been created for Zackary in ED PulseCheck and assigned to you. Zackary and his family have provided
conflicting lists of allergies for Zackary. You want to read through a list of allergies that have been
mentioned so you can verify which ones affect him.
Task: Display a complete list of Zackary’s active and inactive allergies that you would use to verify each
allergy with him and his family.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Display active and inactive allergies.
i. Click Show Inactive to include inactive allergies in the list of allergies
Successful Completion: User clicks Show Inactive.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 51 of 56
Task 14 - Enter active allergies Role: Nurse
Susan McCoy was brought to the emergency department for gastro esophageal reflux disease (GERD). A
patient record has already been created for Susan in ED PulseCheck and assigned to you. Susan says
she is allergic to several medications.
Allergies: Aspirin
Bactrim
Cipro
Nausea
Hives
Difficulty breathing
Mild
Moderate
Severe
Task: Update Susan’s record to reflect her allergies.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Add allergies.
i. From the Known Allergies page, type medication names in the Quick List tab > Click
checkbox, enter reaction, severity and source > click Enter
ii. From the Known Allergies page, type medication names in the Search tab > enter reaction,
severity and source > click Enter
Successful Completion: User clicks Enter.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 52 of 56
Task 15 - Update the effects of an allergy Role: Nurse
Richard Bird was brought to the emergency department with a broken arm. A patient record has already
been created for Richard in ED PulseCheck and assigned to you. Richard said he was allergic to penicillin
but the severity and nature of his allergy was not noted. A family member mentioned that Richard had
difficulty breathing due to a swollen throat when he’d been given penicillin in the past.
Task: Update Richard’s record to reflect the fact that he is severely allergic to penicillin.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Update the allergy.
i. Click on the hyperlinked line to the right of the allergy name
ii. Enter reaction, severity and source
iii. Click Update
Successful Completion: User clicks Update after entering reaction details have been added to allergy.
Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 53 of 56
Task 16 - Inactivate an allergy Role: Nurse
John Smith was brought to the emergency department for right wrist pain. A patient record has already
been created for John in ED PulseCheck and assigned to you. Earlier, a family member said John was
allergic to Tylenol. After talking with John, you learn that he is allergic to Lortab ASA , not Tylenol.
Task: Update John’s record to reflect the fact that he is allergic (mild hives) to Lortab ASA, not Tylenol.
When you feel you're done with the task, click the "Completed" or "Didn't Complete" button.
1. Navigate to the Known Allergies page.
a. Directly from the Tracking Board page:
i. Select the patient > hover over All on the menu bar > click Allergies
ii. Select the patient > click Allergies on the menu bar
iii. Select the patient > click Go > Allergies on the patient row
b. Via the My Patients page (selecting the patient first is not necessary):
i. Click All on the menu bar > My Patients > select patient if necessary > click Allergies
ii. Click My Patients on the menu bar > select patient if necessary > click Allergies
c. Via the patient’s Charting page:
i. Click the patient’s name > click Known Allergies
2. Inactivate allergy.
i. Click on Make inactive to the right of Tylenol
3. Add allergy.
i. From the Known Allergies page, type medication names in the Quick List tab > enter
reaction, severity and source > click Enter
ii. From the Known Allergies page, type medication names in the Search tab > enter reaction,
severity and source > click Enter
Successful Completion: User clicks Enter after entering reaction details. Completed
Yes No/Assisted
Overall this task was
1 2 3 4 5
Page 54 of 56
Appendix 6: Demographics Questionnaire 1. Gender
O Female
O Male
2. Age
O 18-29 years
O 30-39 years
O 40-49 years
O 50-59 years
O 60-69 years
O 70 or more years
3. What is the highest degree or level of school you have completed?
O High School, GED, or equivalent
O Some college, no degree
O Technical certification
O Associate degree
O Bachelor degree
O Master degree
O Professional degree
O Doctorate degree
O Other: (please specify) _____________________________
4. What title best describes your current occupation?
O Emergency Medical Technician
O Health Information System Administrator
O Nurse Practitioner
O Office Administrator
O Pharmacist
O Physician
O Physician’s Assistant
O Psychologist, Counselor, or Social Worker
O Registered Nurse
O Other: (please specify) _____________________________
Page 55 of 56
5. How long have you held your current title?
O Less than 1 year
O 1-3 years
O 4-6 years
O 7-9 years
O 10-19 years
O 20-29 years
O 30-39 years
O 40-49 years
O 50 or more years
6. How long have you used a computer in a professional workplace?
O Less than 1 year
O 1-3 years
O 4-6 years
O 7-9 years
O 10-19 years
O 20-29 years
O 30-39 years
O 40-49 years
O 50 or more years
7. How long have you used ED PulseCheck?
O Less than 1 year
O 1-3 years
O 4-6 years
O 7-9 years
O 10 or more years
8. Do you use assistive technology; such as an enhanced visual display, physically adapted pointing device, or text-
to-speech engine; in the workplace?
O Yes
O No
If yes, please describe __________________________________
Page 56 of 56
Appendix 7: Data Scoring Details
As was described in the Data Scoring – Effectiveness, Task Success section, optimal task performance time was
benchmarked when constructing tasks by using Keystroke Level Modeling (KLM) to measure the clicks, keystrokes, and
task time to obtain an expected time based on the optimal path. (KLM was used rather than expert time because the
product includes many paths to accomplish tasks, and experts frequently didn’t follow optimal paths.)
To quantify the additional time added to the KLM calculations, three expert performances of three tasks with varying
complexity were reviewed (Physician tasks: 2, 4, 18; RN tasks: 10, 13, 16) based on the rules described below. An
average time for each rule was calculated and then consistently applied throughout the KLM calculations.
The rules below detail how this additional time, based on test environment measurements, was added to the KLM
results to account for the following:
1. User’s need to review information while interacting with the user interface (medical professionals double-check their
work routinely to avoid error):
• 5 seconds: Review data entry before next step
• 3 seconds: Look at information displayed (e.g., table data)
• 3 seconds: Find patient in list of Tracking Board
2. System response times:
• 7 seconds: (Tasks 17-18) After Search button is clicked until the search window appears
• 4 seconds: For next page to appear after clicking Enter button
Or
2 seconds: (If there was no Enter button) System responds and displays the next page
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
Appendix B: Quality Management System
©2016 InfoGard. May be reproduced only in its original entirety, without revision 11
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
Appendix C: Privacy and Security
©2016 InfoGard. May be reproduced only in its original entirety, without revision 12
Test Results Summary for 2014 Edition EHR Certification
16‐2423‐R‐0022‐PRI V1.0, June 20, 2016
Test Results Summary Document History Version
V1.0
END OF DOCUMENT
Description of Change
Initial release
Date
June 20, 2016
©2016 InfoGard. May be reproduced only in its original entirety, without revision 13
top related