Focus Section on Health IT Usability: Perceived Burden ofEHRs on Physicians at Different Stages of Their CareerSaif Khairat1,2 Gary Burke3 Heather Archambault4 Todd Schwartz4 James Larson3 Raj M. Ratwani5,6

1Carolina Health Informatics Program, University of North Carolina,Chapel Hill, North Carolina, United States

2School of Nursing, University of North Carolina, Chapel Hill, NorthCarolina, United States

3Department of Emergency Medicine, University of North Carolina,Chapel Hill, North Carolina, United States

4Department of Biostatistics, University of North Carolina, ChapelHill, North Carolina, United States

5National Center for Human Factors in Healthcare, MedStar Health,Washington, District of Columbia, United States

6Georgetown University School of Medicine, Washington, District ofColumbia, United States

Appl Clin Inform 2018;9:336–347.

Address for correspondence Saif Khairat, PhD, University of NorthCarolina–Chapel Hill, 428 Carrington Hall, Chapel Hill, NC 27514,United States (e-mail: Saif@unc.edu).

Keywords

► electronic healthrecords and systems

► interfaces andusability

► physician► emergency and

disaster care► satisfaction

Abstract Objective The purpose of this study was to further explore the effect of EHRs onemergency department (ED) attending and resident physicians’ perceived workload,satisfaction, and productivity through the completion of six EHR patient scenarioscombined with workload, productivity, and satisfaction surveys.Methods To examine EHR usability, we used a live observational design combinedwith post observation surveys conducted over 3 days, observing emergency physi-cians’ interactions with the EHR during a 1-hour period. Physicians were asked tocomplete six patient scenarios in the EHR, and then participants filled two surveys toassess the perceived workload and satisfaction with the EHR interface.Results Fourteen physicians participated, equally distributed by gender (50%females) and experience (43% residents, 57% attendings). Frustration levels associatedto the EHR were significantly higher for attending physicians compared with residents.Among the factors causing high EHR frustrations are: (1) remembering menu andbutton names and commands use; (2) performing tasks that are not straightforward;(3) system speed; and (4) system reliability. In comparisons between attending andresident physicians, time to complete half of the cases as well as the overall reaction tothe EHR were statistically different.Conclusion ED physicians already have the highest levels of burnout and fourthlowest level of satisfaction among physicians and, hence, particular attention is neededto study the impact of EHR on ED physicians. This study investigated key EHR usabilitybarriers in the ED particularly, the assess frustration levels among physicians based onexperience, and identifying factors impacting those levels of frustrations. In ourfindings, we highlight the most favorable and most frustrating EHR functionalitiesbetween both groups of physicians.

receivedNovember 27, 2017accepted after revisionMarch 26, 2018

DOI https://doi.org/10.1055/s-0038-1648222.ISSN 1869-0327.

Copyright © 2018 Schattauer

Research Article336

Background and Significance

The increased adoption of electronic health records (EHRs)has prompted many investigations of their implementationcosts and accrued benefits. Hospitals adopting an EHR aremore productive, have improved diagnostic accuracy, andhave increased quality of care;1–6 one study conducted in asingle region suggested that an EHR could reduce costs by$1.9 million in an emergency department (ED) with aninteroperable EHR system.7

As EHRs become standard in hospitals, ease of usability—the facility with which a user can accurately and efficientlyaccomplish a task—has gained increased attention.8,9 Manydeficiencies exist in EHR usability, such as human errors as aresult of poor interface design (i.e., when clinicians’ needs arenot taken into account, or the information display does notmatch user workflow).10–13 Other usability challenges exist,such as limitations to a user’s ability to navigate an EHR,enter information correctly, or extract information necessaryfor task completion. For EHRs to be effective in assisting withclinical reasoning and decision-making, they should bedesigned and developed with consideration of physicians,their tasks, and their environment.14

Physicians may approve of EHRs in concept and appreciatethe ability to provide care remotely at a variety of locations;however, in an assessment of physicians’ opinions of EHRtechnologies in practice, the American Medical Association(AMA) found that EHRs significantly eroded professional satis-faction: 42% thought their EHR systems did not improveefficiency, and 72% thought EHR systems did not improveworkload.15 In an Association of American Physicians andSurgeons (AAPS) survey, more than half of physicians feltburned out from using EHRs,16 and emergency medicinephysicians spent almost half their shift on data entry, with4,000 EHR clicks per day.17 In a survey of 1,793 physicians,Jamoomet al found thatmore than 75% of physicians reportedthat EHR increases the time to plan, review, order, and docu-ment care.18 Physician frustration is associated with lowerpatient satisfaction and negative clinical outcomes.19–21

EHR dissatisfaction among physicians may be damaging tomedicine.15,22 Providing care in the era of EHRs leads to highlevel work-related stress and burnout among physicians;23 EDphysicians specifically already have the highest levels ofburnout and fourth lowest level of satisfaction among physi-cians. Particular attention, then, is needed to examine theimpact of using an EHR on ED physicians.24 Increasing ageamong physicians were negatively correlated with EHR adop-tion levels;25however, onlymodest, if any, attemptshavebeenmade to understand if reported usability dissatisfaction levelswill decreasewith thechangingof theguard, that is, asyoungerphysicians with lifetime experience using information tech-nologies develop and mature in the medical profession.

Objective

The purpose of this study was to explore the effect of EHRs onED attending and resident physicians’ perceived workload,satisfaction, and productivity through the completion of six

EHR patient scenarios combined with workload, productivity,and satisfaction surveys. We hypothesized that attendingphysicians—who are traditionally older, more experienced,and further removed from and/or lacking medical trainingwith robust EHR integration—would be more frustrated withthe EHR compared with resident physicians. We hypothesizedthat the latter would have more training using and experiencewith EHRs and, thus, be more comfortable with the electronicplatforms.

Methods

Study DesignTo examine EHR usability, we used a live observational designcombinedwith immediate postobservation surveys conductedover3days, observingemergencyphysicians’ interactionswiththe EHR during a 1-hour period. We created six EHR patientscenarios in the training environment of a commercial EHR—Epic version 2014—to mimic standard ED cases.

Participants and SettingWe conducted the study at a large, tertiary academic medicalcenter.Werecruitedemergencymedicineattendingphysicians(“attendings”; n ¼ 8) and resident physicians (“residents”;n ¼ 6) in their third and fourth years in training to participatein this study, which took place in the Emergency MedicineDepartment office space. Participants sat in a private officeequipped with a workstation and interacted with the trainingenvironment of Epic 2014 used at that medical center. At thetimeof the study, all participants used Epic2014 to deliver carewith varying degrees of exposure to the EHR. We obtainedInstitutional Review Board approval. All participants weregiven an information sheet—describing the study procedureand objective, time commitment, and compensation, and hadan opportunity to ask questions. We obtained verbal consentfor participation in the study aswell as for the participant to bevideo recorded.

ProcedureOur research team comprised two investigators with PhDs inhealth informatics and health services, one research coordi-nator for usability services, a project manager on humanfactors, and an ED physician. We designed six patient casesthat were incorporated into the training environment of theEpic 2014 system used by the hospital. An EHR expertconsultant from the hospital team—an Epic builder with anursing background—created a patient record in the EHR foreach of the six cases, entering information about each patient(e.g., name, date of birth (DOB), height, weight, chief com-plaint, triage note, vital signs, history of present illness,pertinent exam, past medical history, medications, allergies,social history). For each patient record, the team designed acase scenario for participants to follow and execute in theEHR. The study sessions were conducted separately for eachparticipant in a clinical simulation setting.

We asked participants to complete the six comprehensiveED scenarios that included tasks to ensure that participantswere exposed to various aspects of the EHR. Every participant

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al. 337

had an hour to complete all six scenarios sequentially andwasexpected to approach each scenario with similar thorough-ness, which was evaluated by successful task completion. Weconducted the study in a simulated setting to minimize bias;all participantswere presentedwith the same scenarios in thesame environment on days during which they did not havescheduled work. First, participants identified the fictitiouspatients’ cases in the EHR. Next, a research assistant readthe scenarios, which included a set of initial actions andevaluations, follow-up actions, and disposition, which theparticipants were to complete in the EHR. Participants couldmake handwritten notes about the scenario, actions, anddisposition. During the encounter, participants could ask theresearch assistant to repeat the scenario or task as needed.

We measured how managing six different EHR scenariosaffected user experience and satisfaction (►Table 1). Wedesigned the patient cases to include common EHR challengessuch as cognitive overload, patient safety and medicationerrors, and information management and representation;furthermore, we designed the patient cases to reflect standardEHR tasks done by ED physicians (►Table 1).

Following completion of the six scenarios, we askedparticipants to complete two survey questionnaires: theNational Aeronautics and Space Administration Task LoadIndex (NASA-TLX) (►Appendix Fig. A1) and Questionnairefor User Interaction Satisfaction (QUIS).

MeasurementWe recorded usability data, including time to complete a taskwith usability software Morae Recorder (Okemos, Michigan,United States) installed on the study workstation. The userexperience surveys, TLX and QUIS, were administered imme-diately following theEHR testing sessions to evaluate theuser’sperceptionof theEHRexperienceand theEHRinterfacedesign.

The TLX is a selective workload assessment tool ofhuman–computer interface designs measuring users’ per-ceived workload levels in six dimensions: mental demand,physical demand, temporal demand, performance, effort,and frustration.26,27 Each dimension is ranked on a 20-stepbipolar scale, with scores ranging from 0 and 100. The scoresfor all dimensions then are combined to create an overallworkload scale (0–100). The QUIS assesses users’ subjectivesatisfaction with specific aspects of the human–computerinterface.28 According to the Agency for Healthcare Researchand Quality (AHRQ), the QUIS is valid even with a smallsample size, and provides useful feedback of user opinionsand attitudes about the system,29 specifically about theusability and user acceptance of a human–computer inter-face. The tool contains a demographic questionnaire; ameasure of overall system satisfaction along five subscales:overall reaction, screen factors, terminology and systemfeedback, learning factors, and system capabilities; as wellas open-ended questions about the three most positive andnegative aspects of the system. Each area subscale measuresthe users’ overall satisfaction with that facet of the interface,as well as the factors that make up that facet, on a 9-pointscale. A detailed description of the QUIS tool and evaluationitems is shown in ►Table 2.

Statistical AnalysisUsing SAS software, version 9.4 (Cary, North Carolina,United States), data were analyzed using descriptive statis-tics (means and standard deviations for continuous vari-ables; frequencies and percentages for categorical variables).Bivariate associationswere computed for pairs of continuousvariables via Pearson’s correlation coefficients. Means werecompared between groups and via two-sample Satterhwaitet-tests, separately by gender, role (attending physician vs.resident physician), and hours working in the ED (low vs.high). Although hours working in the ED was captured as acontinuous variable, we dichotomized the variable as “low”

for residents who worked 50 or fewer hours per week or forattendings who worked 30 or fewer hours, and as “high” for

Table 1 Study scenarios, EHR function, andbuilt-inusability issues

Cases EHR function

1. Pediatricforearmfracture

1. Method of calculating dosing

2. How physician refers to patient weight

3. Process of ordering morphine

4. Process of ordering facility transfer

2. Backpain

1. EHR clinical support

2. Process of ordering MRI

3. Discharge process

4. Time sensitive protocol butnot easily ordered

3. Chestpain

1. Situational awareness (How willphysician relay need to monitorpatient blood pressure?)

2. How does physician view patient’sblood pressure

3. Process of ordering test to becompleted at future time

4. Process of admitting patient fortelemetry

4. Abdominalpain

1. Process of ordering specific CT scan

2. Process of reviewing CT Scan

3. Process of discharge

4. Ordering over 4–6 h

5. Asthma 1. Process of delivering nebulizertreatment

2. Process of ordering medication taper

3. Is SureScripts system tied in

6. Sepsis 1. Process of renal dosing forappropriate antibiotics

2. Process of ordering weight based fluids,how does system calculate (if at all)

3. Process of ordering laboratories tobe completed at future time

4. Does EHR provide guidance onappropriate rate of medication

Abbreviations: CT, computed tomography; EHR, electronic healthrecord; MRI, magnetic resonance imaging.

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al.338

residents who worked more than 50 hours or for attendingswho worked more than 30 hours, based on ED physicianexpert suggestion. We used a two-sided significance level of0.05 and did not adjust p-values for multiple comparisons.

Results

Fourteen physicians participated, equally distributed by gen-der (50% female) and experience (43% residents, 57% attend-ings) (►Table 3). All residents had no postresidency clinicalpractice experience whereas the attendings had 3 ormore years of clinical experience since residency; however,75% of each groups had 3 to 5 years or > 5 years of experienceusing Epic prior to the study. Residentsworked in the ED for anaverage of 54.2 � 3.7 hours per week, and attendings for27.5 � 7.2 hours per week. The physicians, when not dividedby classification, achieved ameanTLX score of 6.3 � 2.3 out of20. Participants tookanaverageof21.7 � 7.0minutes tofinishall six scenarios, with completion times ranging from 12.6 to36.3 minutes.

Perceived Workload in the EHRIn two-sample t-tests comparing differences for the NASA-TLX total score and its items by role (resident vs. attendings),by gender, and by category of hours worked, we found onlyone significant difference: the frustration item was signifi-cantly lower for residents than for attendings (3.3 vs. 8.0,p < 0.01) (►Fig. 1).

Bivariate associations between the TLX total score and theQUIS items are shown in ►Table 4. The TLX total score was

significantly correlated with the screen item (Pearson’sr ¼ 0.62, p ¼ 0.02), such that higher ratings on the screenwere associated with higher TLX total scores. No other sig-nificant correlations were found for any TLX item with theQUIS ratings.

EHR SatisfactionWe found significant correlations among the NASA-TLX andthe QUIS survey results (►Table 4). All participants completedboth surveys and datawerematched using an assigneduniqueparticipant identifier. Higher overall reaction, measured byevaluating navigation, satisfaction, power, and stimulation,was associated with higher ratings for terminology and infor-mation (r ¼ 0.54, p < 0.05) and with lower times to comple-tion of all scenarios (r ¼ –0.73, p < 0.01). Higher systemcapability ratings were associated with higher ratings for thescreen item (r ¼ 0.60, p ¼ 0.02) and with higher learningratings (r ¼ 0.80, p < 0.01). Also, higher terminology andinformation ratings were positively associated with higherlearning ratings (r ¼ 0.62,p ¼ 0.02). Althoughnot statisticallysignificant, higher ratings of the system capabilities tended toyield shorter times to completion of all scenarios (r ¼ –0.44).

Pearson’s correlation coefficients were determined forfrustrationwith each individual QUIS score,►Table 5. Withinthe context of learning the system, lower scores for remem-bering commands and for performing straightforward taskswere associated with greater frustration with the system.Within the items describing system capabilities, lower scoresfor system speed and for system reliability were also signifi-cantly associated with greater frustration with the system.

Table 2 QUIS tool subscales and the corresponding items to be evaluated

QUIS Description Evaluations Items

Overall reactionto the EHR

Users assess the overall user experience with theEHR.

• Navigation• Satisfaction• Power• Stimulation

Screen Users rate the screen/interface design of the EHR • The ability to read characters on the screen• Information overload• Organization of information• Sequence of screens

Terminology andsystem information

Users rate the consistency of terminology,frequency and clarity of hard stops, and systemfeedback on tasks

• Use of terms through the system• Terminology related to task• Position of message on screen• Prompts for input• Computer informs about its progress• Error messages

Learning Users evaluate their ability to use the system, theeffort and time to learn the system, knowledge onhow to perform tasks, and the availability ofsupport

• Learning to operate the system• Exploring new features by trials and error• Remembering names and use of commands• Performing tasks is straightforward• Help messages on the screen• Supplemental reference materials

System capability Users rate the performance and usability of theEHR

• System speed• System reliability/System down• Ability to correct mistakes• System designed for all levels of users

Abbreviations: EHR, electronic health record; QUIS, Questionnaire for User Interaction Satisfaction.

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al. 339

EHR ProductivityFurthermore, in comparisons between resident and attend-ing physicians, time to complete the back pain, chest pain,and sepsis scenarios, as well as overall reaction (based onQUIS scores), were statistically different (►Fig. 2). In com-parisons between high and low average numbers of hours

worked, the time to complete the sepsis scenario and overallreaction showed significant differences (►Fig. 2).

►Fig. 2 shows separate assessments for each scenariomade via Satterthwaite t-tests by role (resident vs. attend-ing). Residents had significantly shorter mean durations tocomplete the back pain (1.9 vs. 3.0 minutes, p < 0.01), chest

Fig. 1 t-Values for testing the difference in Task Load Index (TLX) scores between residents and attending.

Table 3 Demographics

Resident, n (%) Attending, n (%) Total

Gender Male 5 (83.3) 2 (25) 7

Female 1 (16.7) 6 (75) 7

Age 18–34 6 (100) 0 6

35–50 0 7 (87.5) 7

51–69 0 1 (12.5) 1

Ethnicity Asian 0 1 (12.5) 1

White 6 (100) 7 (87.5) 13

Years of clinical practice(postresidency):

0 6 (100) 0 6

1–2 y 0 0 0

3–5 y 0 2 (25) 2

More than 5 y 0 6 (75) 6

Number of years ofexperience in Epic prior tothe study

1–2 y 1 (25) 2 (25) 3

3–5 y 5 (75) 5 (62.5) 10

> 5 y 0 1 (12.5) 1

Average number of hoursworked in Epic per week

< 30 h 0 6 (75) 6

30–50 h 2 (50) 2 (25) 4

> 50 h 4 (50) 0 4

Total 6 8 14

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al.340

pain (2.4 vs. 4.7 minutes, p < 0.01), and sepsis (2.0 vs.3.7 minutes, p < 0.01) scenarios compared with attendings.Although not statistically significant, residents also took lesstime, on average, to complete the abdominal pain scenario(2.3 vs. 3.2 minutes). ►Fig. 2 also shows results of two-sample t-tests demonstrating significant differences for theQUIS items by role (resident vs. attending). When examiningby role, the mean score for overall reaction was significantlyhigher for residents than for attendings (7.1 vs. 5.8, p ¼ 0.02).

Comparisons by hours worked in the EHR indicated thatthoseworking fewer hours took a significantly longer time tocomplete the sepsis and back pain scenarios compared withthose working more hours (3.4 vs. 2.3 minutes, p < 0.05,and 2.9 vs. 2.1 minutes, p ¼ 0.05). Although not statisticallysignificant, ►Fig. 3 show that the same pattern was demon-strated for the abdominal pain (3.2 vs. 2.3 minutes), andchest pain (4.4 vs. 2.8 minutes, p ¼ 0.09) scenarios. By hoursworked, overall reaction ratings were significantly lower for

Table 5 Correlation coefficients and p-Values (bold) between frustration levels and EHR characteristics

Remembering namesand commands use

Performing tasks isstraightforward

Systemspeed

Systemreliability

Frustration levels –0.5550.039a

–0.6000.023a

–0.7090.004a

–0.6330.015a

Abbreviation: EHR, electronic health record.aStatistically significant values.

Table 4 Correlation coefficients and p-Values (bold) between NASA-TLX and QUIS items

Overall reaction Screen Terminology and information Learning

Terminology and information 0.5380.047a

0.3330.245

NA

Learning 0.4400.115

0.3140.274

0.6160.019a

NA

System capabilities 0.3360.240

0.6000.023a

0.3740.188

0.8050.001a

Total number of minutes tocomplete all scenarios

–0.7250.003a

–0.3950.162

–0.3240.259

–0.2910.312

Abbreviations: NASA-TLX, National Aeronautics and Space Administration Task Load Index; QUIS, Questionnaire for User Interaction Satisfaction.aStatistically significant values.

Fig. 2 Average minutes to complete each scenario by role.

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al. 341

those working fewer hours (5.8 vs. 6.9, p < 0.05) than forthose working more hours.

EHR Preferences and FrustrationsTo understand differences in EHR preferences and frustrations,participants were asked at the end of the exercise on the QUISsurvey to identify the three most favorable and most frustrat-ing functions of the EHR; 12 (85%) participated and wereequally distributed between roles. Common preferencesemerged among both groups, such as appreciation for “click”shortcuts available when placing an order, and the ability to

autopopulate smart phrases in the notes by using the dot (.)phrases functionality. Two-thirds of attending complimentedthe EHR design, noting its reactive speed as well as its clean,consistent, and reliable layout. Residents particularly liked theflexibility for care through EHR access at multiple sites. Resi-dents also expressed satisfaction in the comprehensive andconsolidated representation of patient information (►Fig. 4).

With regard to frustrations, attending complained aboutthe frequency and rationale behind warnings and errorsmessages, such as alerts when ordering tests, medication,or computed tomography (CT) for patients with allergies.

Fig. 4 Most positive aspects of the electronic health record (EHR) by roles.

Fig. 3 Average minutes to complete task by electronic health record (EHR) hours worked. Difference in satisfaction levels based on EHR hoursworked.

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al.342

While half of the attendings found it easy to place orders,one-third found it difficult to adjust orders once placed.Attendings also expressed concern about the steep learningcurve associated with the EHR, which they often found to bementally taxing. Residents’ main complaints were with thenumber of mouse clicks per task in the EHR. Like attendings,residents noted that orders were difficult to adjust onceplaced, and that alert fatigue remains a significant challengeof EHR use. Residentswere also dissatisfiedwith informationoverload when reviewing patient charts (►Fig. 5).

Discussion

We found in this study a significant disparity between overallsatisfaction with the EHR between residents and attendingphysicians. Attending reported significantly higher frustrationlevels with the EHR. This difference may be explained by thetime it took for participants to complete the six scenarios:residents completedhalf the scenarios significantly faster thanattending. Aside from computer skills, longer completiontimes may be attributed to attending physicians’wider reper-toire of clinical experience and judgment, and hence, attend-ings may look for more patient data and consider alternateexplanations more than residents. As a result, attendings maytake longer to complete a task if more data are required tomake clinical judgment. We raise the question of whethershorter completion time is actually a valid indication of betterusability, andwhat is the effect of faster interaction on clinicaloutcomes. A quicker completion time that is free of errors yetmaintains thoroughness would be ideal; nonetheless, this is atopic that should be explored and evaluated in future work.

The workflow discrepancy between a resident and anattending must also be considered. At many institutions,

particularly academic ones, the attendings’ responsibilitiesare often supervisory,while residents domuch of theworkofdocumentation including entering orders and notes. Attend-ing physicians oversee and supplement direct patient careand documentation. Attending ultimately sign off on docu-mentation completed by a resident; nonetheless, if residentsare performing most of the work of order entry and makingclinical notes, theymay be more comfortable with EHRs and,thus, quicker to complete the scenarios compared with theirattending physicians.

The EHR screen design showed a significant relationshipwith effort exerted to find information, temporal demandlevels, and overall perceived workload. Most participantsnoted a relationship between screen design and how mucheffort they spent on a task. They tended to be satisfied if thedesign did not require much effort to interact with it. Finally,we found a relationship between the physician’s EHR experi-ences (i.e., number of hours spent working with the EHR) andthe reported levels of satisfaction among physicians: theattending, on average, work < 50 hours per week with Epic,whereas their resident counterparts work 30 to 50 hours(50%) or > 50 hours (50%) weekly with Epic (►Table 3). This,coupled with the significantly higher mean score for overallreaction for residents (7.1 vs. 5.8, p ¼ 0.02), indicate that thephysicians who spent more time working on the EHRreported higher levels of satisfaction.

Our resultsshowedthatparticipantsweresatisfiedwith thebroad array of functions provided through the system; how-ever, our findings indicated that a significant learning curvewas required to learnways tonavigate thesystem. Participantsranked the mental demand as the highest perceived EHRworkload, and the physical demand as the lowest. Participantsrated aspects of the EHR interface design such that system

Fig. 5 Most negative aspects of the electronic health record (EHR) by roles.

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al. 343

capabilities were the highest category in terms of satisfaction,while learning factors was the lowest ranked category. Onaverage, participants completed six scenarios in 21 minutes;however, there was a range of 7 minutes among participants.

We found a concerning disconnection between the theorybehind EHR systems and the reality of using them in practice.EHRs were designed to make the administration of medicalcare easier for physicians and safer for patients30 but frus-trations with technology have, in many cases, had theopposite effect. Physician frustration with the EHR hasbeen associated with high burnout levels, lower patientsatisfaction, and negative clinical outcomes.19–21 Forinstance, in 2013, a 16-year-old boy received more than 38trimethoprim/sulfamethoxazole (trade name Septra, Bac-trim) tablets while hospitalized.31 Poor interface designwithin an EHR systemwas ultimately blamed for themassiveantibiotic overdose, which was nearly fatal for the youngpatient. The aforementioned impact of poor interface designand human factors is not unique to health care; humanfactors in other disciplines have shown critical and fatalerrors as a result of lack of user–center interface design.For example, the famous Canada Air incident named GimliGlider shows the importance of human–computer interac-tion when an accident occurred due to a pilot error with thecockpit Fuel Quantity Indicator System (FQIS).32 Similarly,human factors research have demonstrated similar resultswith regards to the relationship between interface designsand complex systems usability in fields like aviation, psy-chology, education, and computer science.33–35 The usabilityfindings in this article aligns with findings in similar dis-ciplines. For instance, in aviationworkers are under extremestress while attempting to access considerable amounts ofdata. Information overload, a contributing factor to degradeddecision making ability, affects these works by introducingdelay in the correct response.36

To improve EHR usability, we need to assess and addressuser “personas” that take into account users’ clinical experi-ence as well as their technical abilities. Tailoring the EHRinterfacebyclinical roles (e.g., nurse, programcoordinator, andphysician) is insufficient: rather, we should specify the indi-vidual personas associatedwitheach clinical role. Personas areused to create realistic representations of key users; it iscommonly used in disciplines such as marketing, informationarchitecture, and user experience.37–40 The interface needs tobe tailored to the different personas within the same role. Forexample, attending physicians are domain experts who mayhave received their medical training prior to EHR adoptionand, therefore, their expectations of the EHR will be differentthan a residentwhohas been training post-EHRadoption. Thisstudy shows that not all physicians are dissatisfied with theEHR and that there are varying levels of satisfaction based onseniority. Our results show that attending physicians arenotably more frustrated than resident physicians when usingthe EHR. We expect that levels of frustration from the EHRamong providersmaydiminish as the careers advance ofmorephysicians who have life-long exposure to information tech-nologies. To meet the expectations of all users, however,current EHR designs require major revisions with input from

a range of clinicians with varying degrees of roles and experi-ence and expertise with the electronic platforms.

Although the simulation-based environment we used inthis research served the purpose of understanding theperceivedworkload and satisfaction of ED physicians regard-ing EHRs, we also noted limitations. Conducting the study inthe ED in real-time hold potential to reveal new patterns andcreate different outcomes for the time to task completion,perceivedworkload, and satisfaction due to the intensity andinterruptions inherent in the ED environment; however, theability to implement such a study in real-life settings ischallenging for several reasons. First, the risk of disclosingprotected patient information is significant. Second, provid-ing a survey at the end of a physician’s ED shiftmay introducebias related to fatigue or salience of the records of last fewpatients seen.

Finally, the number of participants (8 physicians and 6resident physicians) may be viewed as a limitation related tothe lack of statistical power in the analysis; however, fiveparticipants are usually sufficient for usability studies suchas this one.41

Researchers and scientists in fields other than health caredeveloped the TLX and QUIS, therefore these tools are notspecifically designed to assess EHR interfaces. We suggestthat future efforts should include developing tailored andvalidated qualitative and quantitative tools to assess theusability of the health information technology applicationsused by health care providers and researchers.

Furthermore, the use of eye-tracking methodologies hasshown promise in the sociobehavioral and engineeringfields.42–44 However, despite modest attempts to build aframework explaining how to use eye-tracking devices inhealth services research, in particular, ways to integrate eye-tracking into a study design, there remains a need for ahealth-specific manual explaining how to use the saiddevices. In order for them to best benefit medical research,formal training for medical personnel must occur.45,46

Importantly, this effort would contribute to the developmentof creative ways of analyzing the large data files provided bythe eye-tracker using automation and algorithms.

Conclusion

In this article, we presented findings from an EHR usabilitystudy of physicians in the emergency medicine departmentat a tertiary academic medical center. Although this studybuilds on previous research,16,18,47 it differs from previouswork because we focused on responses to EHR, especiallyfrustration, between physicians at two different careerlevels, and the factors leading to frustration. We found thatEHR frustration levels are significantly higher among moresenior attending physicians compared with more juniorresident physicians. Among the factors causing high EHRfrustrations are: (1) remembering menu and button namesand commands use; (2) performing tasks that are notstraightforward; (3) system speed; and (4) system reliability.

In our findings, we highlight the most favorable and mostfrustrating EHR functionalities of both groups of physicians.

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al.344

Attending physicians appreciate the consistency in EHR userinterface and the ease of entering orders in the system, andthey reported frustrations with the frequency of hard stopsin the system. Although residents appreciated the ability touse dot phrases and autocomplete functions, as well as theease of entering orders, they found the click-heavy and alertfatigue aspects of EHR to be most frustrating.

The findings of this study can be used to inform futureusability studies, in particular, more specific usability areassuch as information representation, standardized pathways toaccomplishing tasks, and ways to reduce the steep learningcurve for users. Finally, today’s resident physicians willbecome tomorrow’s attending physicians, faced with newduties or demands. In this never-ending technological cycle,theonlycertainty is thatnothingstays thesame.Moreresearchis needed to investigate ways to improve satisfactionwith theEHR through meeting the various expectations from tech-savvy residents to domain expert attending physicians.

Clinical Relevance Statement

Electronic health records may lead to higher physician’sburnout and dissatisfaction levels if they do not meet theexpectations of users. We demonstrate differences in EHRexpectations, preferences, and use among resident andattending physicians, suggesting that within the medicalfield there are different user characteristics that need to befurther dissected and analyzed.

Multiple Choice Question

Users have distinct EHR personas. Which of the followingstages of persona creation is likely to take the most time?

a. Stage I: Gathering data for personasb. Stage II: Analyzing the data gathered in stage Ic. Stage III: Crafting the actual personasd. All stages typically take the same amount of time

Correct Answer: The correct answer is option a. Personasare representations of a cluster of users with similarbehaviors, goals, andmotivations. Personas can be a power-ful tool for focusing teams and creating user-centeredinterfaces because they embody characteristics and beha-viors of actual users. However, identifying and creatingpersona consumes the most time. This stage includeempirical research, as well as gathering assumptions andexisting insights from stakeholders and other individualswith a deep understanding of target users.

Protection of Human and Animal SubjectsThe study was performed in compliance with the WorldMedical Association Declaration of Helsinki on EthicalPrinciples for Medical Research Involving Human Subjects,andwas reviewed and approved by the University of NorthCarolina at Chapel Hill (UNC) Institutional Review Board.

Conflict of InterestNone.

AcknowledgmentsThe authors thank Dr. Beth Black, PhD, RN, and SamanthaRussomagno, DNP, CPNP for their contribution duringmanuscript preparation. This study was sponsored bythe American Medical Association (AMA).

References1 Huerta TR, Thompson MA, Ford EW, Ford WF. Electronic health

record implementation and hospitals’ total factor productivity.Decis Support Syst 2013;55(02):450–458

2 Cebul RD, Love TE, Jain AK, Hebert CJ. Electronic health recordsand quality of diabetes care. N Engl J Med 2011;365(09):825–833

3 Da’ve D. Benefits and barriers to EMR implementation. Caring2004;23(11):50–51

4 Walsh SH. The clinician’s perspective on electronic healthrecords and how they can affect patient care. BMJ 2004;328(7449):1184–1187

5 Clayton PD, Naus SP, Bowes WA III, et al. Physician use ofelectronic medical records: issues and successes with directdata entry and physician productivity. AMIA Annu Symp Proc2005;2005:141–145

6 Linder JA, Schnipper JL, Middleton B. Method of electronic healthrecord documentation and quality of primary care. J Am MedInform Assoc 2012;19(06):1019–1024

7 FrisseME, Johnson KB, Nian H, et al. Thefinancial impact of healthinformation exchange on emergency department care. J Am MedInform Assoc 2012;19(03):328–333

8 Zhang J, Walji MF. TURF: toward a unified framework of EHRusability. J Biomed Inform 2011;44(06):1056–1067

9 Sittig DF, Belmont E, Singh H. Improving the safety ofhealth information technology requires shared responsibility: itis time we all step up. Healthc (Amst) 2017:S2213-0764(17)30020-9

10 Tang PC, Patel VL. Major issues in user interface design for healthprofessional workstations: summary and recommendations. Int JBiomed Comput 1994;34(1-4):139–148

11 Linder JA, Schnipper JL, Tsurikova R, Melnikas AJ, Volk LA, Mid-dleton B. Barriers to electronic health record use during patientvisits. AMIA Annu Symp Proc 2006;2006:499–503

12 Zhang J. Human-centered computing in health information sys-tems. Part 1: analysis and design. J Biomed Inform 2005;38(01):1–3

13 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronichealth record usability: analysis of the user-centered designprocesses of eleven electronic health record vendors. J Am MedInform Assoc 2015;22(06):1179–1182

14 Terazzi A, Giordano A,Minuco G. Howcan usabilitymeasurementaffect the re-engineering process of clinical software procedures?Int J Med Inform 1998;52(1-3):229–234

15 Association AM. Physicians Use of EHR Systems 2014.Research2015

16 Surgeons AAPa. Physician Results EHR Survey. 2015 8/25/2017.Available at: https://aaps.wufoo.com/reports/physician-results-ehr-survey/. Accessed August 25, 2017

17 Hill RG Jr, Sears LM, Melanson SW. 4000 clicks: a productivityanalysis of electronic medical records in a community hospitalED. Am J Emerg Med 2013;31(11):1591–1594

18 Jamoom E, Patel V, King J, Furukawa MF. Physician experiencewith electronic health record systems that meet meaningful usecriteria: NAMCS physician workflow survey, 2011. NCHS DataBrief 2013;(129):1–8

19 McHugh MD, Kutney-Lee A, Cimiotti JP, Sloane DM, Aiken LH.Nurses’ widespread job dissatisfaction, burnout, and frustrationwith health benefits signal problems for patient care. Health Aff(Millwood) 2011;30(02):202–210

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al. 345

20 Haas JS, Cook EF, Puopolo AL, Burstin HR, Cleary PD, BrennanTA. Isthe professional satisfaction of general internists associated withpatient satisfaction? J Gen Intern Med 2000;15(02):122–128

21 DiMatteo MR, Sherbourne CD, Hays RD, et al. Physicians’ char-acteristics influence patients’ adherence to medical treatment:results from theMedical Outcomes Study. Health Psychol 1993;12(02):93–102

22 Friedberg MW, Chen PG, Van Busum KR, et al. Factors affectingphysician professional satisfaction and their implications forpatient care, health systems, and health policy. Rand Health Q2014;3(04):1

23 Bodenheimer T, Sinsky C. From triple to quadruple aim: care ofthe patient requires care of the provider. Ann Fam Med 2014;12(06):573–576

24 Shanafelt TD, Hasan O, Dyrbye LN, et al. Changes in burnout andsatisfaction with work-life balance in physicians and the generalUS working population between 2011 and 2014. Mayo Clin Proc2015;90(12):1600–1613

25 Hamid F, Cline TW. Providers’ Acceptance Factors and TheirPerceived Barriers to Electronic Health Record (EHR) Adoption.Online Journal of Nursing Informatics 2013;17(3). Doi: Availableat:http://ojni.org/issues/?p=2837. Accessed May 1, 2018

26 Hart SG, Staveland LE. Development of NASA-TLX (Task LoadIndex): results of empirical and theoretical research. In: HancockPA, Meshkati N, eds. Advances in Psychology. Los Angeles,California: North-Holland; 1988:139–183

27 Hart SG. NASA-Task Load Index (NASA-TLX); 20 years later.Proceedings of the Human Factors and Ergonomic Society AnnualMeeting, 2006;50(9): 904–908

28 Chin JP, Diehl VA, Norman KL. Development of an instrumentmeasuring user satisfaction of the human-computer interface.Proceedings of the SIGCHI Conference on Human Factors inComputing Systems; 1988, ACM:Washington, D.C., USA:213–218

29 AHRQ. Questionnaire for User Interface Satisfaction; 2017. Avail-able at: https://healthit.ahrq.gov/health-it-tools-and-resources/evaluation-resources/workflow-assessment-health-it-toolkit/all-workflow-tools/questionnaire. Accessed September 25, 2017

30 Kern LM, Barrón Y, Dhopeshwarkar RV, Edwards A, Kaushal R;HITEC Investigators. Electronic health records and ambulatoryquality of care. J Gen Intern Med 2013;28(04):496–503

31 Wachter R. The Digital Doctor: Hope, Hype, andHarm at the Dawnof Medicine’s Computer Age. New York, NY: McGraw-Hill Educa-tion; 2015

32 Nelson W. The Gimli Glider, in Soaring Magazine; 199733 Ng AKT. Cognitive psychology and human factors engineering of

virtual reality. 2017 IEEE Virtual Reality (VR); 2017

34 Singun AP. The usability evaluation of a web-based test blueprintsystem. 2016 International Conference on Industrial Informaticsand Computer Systems (CIICS); 2016

35 Weyers B, Burkolter D, Kluge A, LutherW. User-centered interfacereconfiguration for error reduction in human-computer interac-tion. Presented at: Third International Conference on Advances inHuman-Oriented and Personalized Mechanisms. Technologiesand Services; August 22-27, 2010; Nice, France

36 Harris D. Human Factors for Civil Flight Deck Design. New York,NY: Taylor & Francis; 2017

37 Brown DM. Communicating Design: Developing Web Site Docu-mentation for Design and Planning. San Francisco, CA: PearsonEducation; 2010

38 O’Connor K. Personas: The Foundation of a Great User Experience;2011. Available at: http://uxmag.com/articles/personas-the-foundation-of-a-great-user-experience. Accessed March 2, 2018

39 Friess E. Personas in heuristic evaluation: an exploratory study.IEEE Trans Prof Commun 2015;58(02):176–191

40 Gothelf J. Using Proto-Personas for Executive Alignment. UXMagazine; 2012

41 Nielsen J, Landauer TK. A mathematical model of the finding ofusability problems. Proceedings of the INTERACT ’93 and CHI ’93Conference on Human Factors in Computing Systems. Amster-dam, The Netherlands: ACM; 1993:206–213

42 Duque A, Vázquez C. Double attention bias for positive andnegative emotional faces in clinical depression: evidence froman eye-tracking study. J Behav Ther Exp Psychiatry 2015;46:107–114

43 Pierce K,Marinero S, Hazin R,McKenna B, Barnes CC,Malige A. Eyetracking reveals abnormal visual preference for geometric imagesas an early biomarker of an autism spectrum disorder subtypeassociated with increased symptom severity. Biol Psychiatry2016;79(08):657–666

44 KassnerM, PateraW, Bulling A. Pupil: an open source platform forpervasive eye tracking and mobile gaze-based interaction. Pro-ceedings of the 2014 ACM International Joint Conference onPervasive and Ubiquitous Computing: Adjunct Publication. Seat-tle, WA: ACM; 2014:1151–1160

45 HershW.Who are the informaticians?What we knowand shouldknow. J Am Med Inform Assoc 2006;13(02):166–170

46 Khairat S, et al. A review of biomedical and health informaticseducation: a workforce training framework. J Hosp Adm 2016;5(05):10

47 Ajami S, Bagheri-Tadi T. Barriers for adopting electronic healthrecords (EHRs) by physicians. Acta Inform Med 2013;21(02):129–134

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al.346

Appendix Fig. A1 NASA-Task Load Index (TLX) used to assess physician’s EHR workload.

Applied Clinical Informatics Vol. 9 No. 2/2018

Understanding the Perceived Burden of EHRs Khairat et al. 347