effects of information access cost and accountability on ...xijyang/assets/yang_hf_2015.pdf ·...

Objective: We aimed to examine the effects of information access cost and accountability on medical residents’ information retrieval strategy and performance during prehandover preparation.

Background: Prior studies observing doctors’ prehan-dover practices witnessed the use of memory-intensive strat-egies when retrieving patient information. These strategies impose potential threats to patient safety as human memory is prone to errors. Of interest in this work are the under-lying determinants of information retrieval strategy and the potential impacts on medical residents’ information prepara-tion performance.

Method: A two-step research approach was adopted, consisting of semistructured interviews with 21 medical resi-dents and a simulation-based experiment with 32 medical residents.

Results: The semistructured interviews revealed that a substantial portion of medical residents (38%) relied largely on memory for preparing handover information. The sim-ulation-based experiment showed that higher information access cost reduced information access attempts and access duration on patient documents and harmed information preparation performance. Higher accountability led to mar-ginally longer access to patient documents.

Conclusion: It is important to understand the underly-ing determinants of medical residents’ information retrieval strategy and performance during prehandover preparation. We noted the criticality of easy access to patient documents in prehandover preparation. In addition, accountability margin-ally influenced medical residents’ information retrieval strategy.

Application: Findings from this research suggested that the cost of accessing information sources should be minimized in developing handover preparation tools.

Keywords: clinical handover preparation, information access cost, accountability, overconfidence, information retrieval

IntroductIonAs an essential component for providing

seamless continuity of patient care, clinical handover can be conceptualized as a three-phase process, consisting of prehandover, handover communication, and posthandover (Kerr, 2002). The majority of existing literature on clinical handovers has focused on the communication phase. Although focusing on the communication phase is extremely valuable, there is a necessity to broaden the research attention to the pre- and posthandover phases, as the extension helps to pinpoint more accurately the stages responsible for the problems observed in handover transition (Raduma-Tomàs, Flin, Yule, & Close, 2012).

It is recommended that an effective handover communication should include an up-to-date summary of patient status and insight on what to anticipate or what to do during the cross-covering period (Arora et al., 2009; Arora, Johnson, Lovinger, Humphrey, & Meltzer, 2005). Although the prehandover phase is vital to ensure the accu-rate and adequate preparation of handover infor-mation, existing studies showed that the prehan-dover preparation is often insufficient (Catchpole et al., 2007; Raduma-Tomàs, Flin, Yule, & Close, 2011). In response to this shortcoming, Raduma-Tomàs et al. (2012) proposed that an ideal process of prehandover preparation should include con-firming the status of requested diagnostic investi-gations (e.g., diagnostic tests, referral replies from other specialties), collating key patient informa-tion, and updating the written handover notes. Qualitative studies on clinical handovers reported circumstances where doctors were constrained to adopt memory-intensive strategies when pre-paring handover information, relying on their

598889 HFSXXX10.1177/0018720815598889Human FactorsMedical Residents’ Prehandover Preparation

Address correspondence to Christopher D. Wickens, Department of Psychology, Colorado State University, Fort Collins, CO 80523; e-mail: [email protected].

Effects of Information Access Cost and Accountability on Medical Residents’ Information Retrieval Strategy and Performance During Prehandover Preparation: Evidence From Interview and Simulation Study

X. Jessie Yang, Nanyang Technological University, Singapore, Christopher D. Wickens, Colorado State University, Fort Collins, CO, USA, Taezoon Park, Soongsil University, Seoul, South Korea, Liesel Fong, National University Hospital, Singapore, and Kewin T. H. Siah, National University Hospital, Singapore

HUMAN FACTORSVol. XX, No. X, Month XXXX, pp. 1 –13DOI: 10.1177/0018720815598889Copyright © 2015, Human Factors and Ergonomics Society.

at HFES-Human Factors and Ergonomics Society on September 5, 2015hfs.sagepub.comDownloaded from

http://hfs.sagepub.com/

2 Month XXXX - Human Factors

memory rather than on displayed or written docu-mentations such as patient medical records (Alem, Joseph, Kethers, Steele, & Wilkinson, 2008; Vidyarthi, Arora, Schnipper, Wall, & Wachter, 2006). Vidyarthi et al. (2006) surveyed residents at three hospitals and found many resi-dents handed over verbally “wherever and when-ever” they could find the cross-covering doctor without proper preparation. It is likely that such ad hoc processes would encourage the adoption of memory-intensive strategies. Along the same line, Alem et al. (2008) studied information envi-ronment supporting medical handovers and found that in preparation of handover outgoing doctors mainly wrote small bits of patient information on patient lists or on blank sheets of paper. The infor-mation was often used as a “memory trigger,” and much additional information would be recalled and presented during handover meetings. How-ever, as the authors stated, there was no guarantee that a doctor would remember the patient infor-mation accurately.

Such memory-intensive strategies impose potential threats to the accuracy of patient infor-mation, as relying on long-term memory with-out augmentation from external memory stores predictably leads to missing and inaccurate information (Wickens, Hollands, Parasuraman, & Banbury, 2013). However, there is a lack of empirical research that addresses the underlying reasons and quantifies the potential negative impacts. This study, therefore, aims to examine the factors influencing medical residents’ infor-mation retrieval strategy and performance.

We postulate that information access cost (IAC) is one possible factor influencing medical residents’ information retrieval strategies from either knowledge in the head (KIH; i.e., a resi-dent’s memory of patient information) or knowl-edge in the world (KIW: i.e., displayed or written patient information). Studies in human–computer interaction (HCI) have consistently shown that the effort or cost to retrieve a particular piece of infor-mation from either KIH or KIW affects one’s retrieval strategy and performance: an increasing IAC from the world encourages a more memory-intensive strategy to access KIH, and discourages access to better information in the world (Ballard, Hayhoe, & Pelz, 1995; Gray & Fu, 2001, 2004;

Gray, Sims, Fu, & Schoelles, 2006). In hospitals, IAC from the world could be manifested via vari-ous means. For example, the poor compatibility between existing health care information technol-ogy (HIT) and clinical tasks (Viitanena et al., 2011), poor usability of HIT such as deep multi-keypress menu structures (Nemeth, Nunnally, O’Connor, Klock, & Cook, 2005), and the inade-quate integration between electronic health record (EHR) and other systems (Flanagan, Patterson, Frankel, & Doebbeling, 2009; Goldschmidt, 2005) are likely to increase the temporal cost, mental workload, and the level of frustration of accessing patient information from KIW and thus bias the information retrieval strategies toward memory-intensive ones. In hospitals lacking HIT infrastructures, IAC could be represented by both the effort used to physically obtain documented information such as patient charts, and the tempo-ral cost of performing corresponding physical activities. The compounded costs may further encourage doctors to rely on internal memory over external memory stores.

The choice between relying on KIH and KIW may be considered within the context of the speed–accuracy trade-off (SATO) in applied set-tings (Drury, 1994). KIH is often a more rapid way of accessing information, but as we have seen, can be error prone. In contrast, KIW is often more accurate, but may take longer to acquire. Just as various incentives and payoffs have been employed to move people toward dif-ferent ends of the SATO (Fitts, 1966; Wickens et al., 2013), in the present study we manipulated IAC indirectly by imposing or not imposing a 5-m distance, the former presumably inducing greater cost than the latter.

The second factor of interest in the present study is medical residents’ accountability in pre-handover preparation. Accountability is defined as the “condition of being answerable to audi-ences for performing up to certain standards, thereby fulfilling responsibilities, duties, expec-tations, and other charges” (Weigold & Schlen-ker, 1991, p. 25). Studies have reported the lack of definition and measurement on accountability in handover (Jeffcott, Evans, Cameron, Chin, & Lbrahim, 2009; Petersen, Brennan, & O’Neil, 1994). Specifically, the majority of health care



Medical Residents’ PRehandoveR PRePaRation 3

organizations have not yet established explicit expectations as to who is responsible within care teams to prepare patient information for hando-ver (Jeffcott et al., 2009), let alone the account-ability on the accuracy and adequacy of infor-mation preparation.

Past research suggested that when held accountable for the outcome, people achieved superior performance in simple judgment and decision-making tasks (Pelham & Neter, 1995). Therefore we hypothesize that medical residents under high accountability conditions would show more frequent and longer access to KIW, and achieve better performance. Furthermore, we speculated that the effect of accountability could be due to a combination of two underlying mechanisms. First, accountability oftentimes implies that people who justify their perfor-mance adequately will gain rewards and people who do not will suffer from penalties (Lerner & Tetlock, 1999). Therefore, it is reasonable to expect that when accountability is high, medical residents motivated to obtain rewards and avoid penalties would make more frequent and longer access to displayed or written patient informa-tion and attain better performance. Second, accountability may boost performance by reduc-ing medical residents’ overconfidence bias. There is a growing body of knowledge that peo-ple are overconfident when judging their mem-ory accuracy (Allwood, Granhag, & Johansson, 2003; Bornstein & Zickafoose, 1999). Under such conditions, one’s confidence judgment in the accuracy of memory is overestimated, biasing the decision toward a memory-intensive strategy more than it should have and thereby impairing performance. Several studies provided consistent findings that increasing accountability reduces overconfidence (Ronis & Yates, 1987; Sniezek, Paese, & Switzer, 1990; Tetlock & Kim, 1987). Therefore, we expect that by increasing medical residents’ accountability, their overconfidence bias will decrease, and thus they would make adequate access to KIW, resulting in better prep-aration performance.

In the present study, the accuracy of KIW was assumed to be 100%. To simplify the notations, from this point onward IAC refers to the cost of accessing KIW. The present study aimed to examine the following hypotheses (Figure 1).

Hypothesis 1 (H1): As suggested by previous studies (Ronis & Yates, 1987; Sniezek et al., 1990; Tetlock & Kim, 1987), an increase of accountability will increase accuracy and reduce overconfidence.

Hypothesis 2a (H2a): An increase of IAC will reduce the number of information access attempts (Gray et al., 2006; Gray & Fu, 2001, 2004) and the information access duration.

Hypothesis 2b (H2b): An increase of IAC will harm information preparation performance.

Hypothesis 3a (H3a): An increase of account-ability will increase the number of informa-tion access attempts and information access duration.

Hypothesis 3b (H3b): An increase of account-ability will improve information preparation performance.

Two separate studies were performed: semi-structured interviews and a simulation-based experiment. The National Healthcare Group Domain Specific Review Board and Hospital Institutional Review Board approved the study and all participants gave informed consent.

SemIStructured IntervIewmethod

Participants. The semistructured interviews were conducted with 21 junior medical residents over a 3-month period at a tertiary and referral hospital in Singapore. All participants were from general medicine, with an average age of 27.2 years (SD = 1.9 years) and an average experi-ence of 2.5 years (SD = 1.2 years).

Interview questionnaire. The interview ques-tionnaire was developed based on the three-phase handover framework. The questions pertaining to prehandover preparation focused on three areas: the number of patients cared by and handed over by a primary team physician, the type of patients handed over by the physi-cian, and the sources of patient information where the physician referred to in preparation of handover.

The semistructured interviews were audio-recorded and transcribed using Transcriber (http://trans.sourceforge.net/). One human factors researcher analyzed the data following standard




procedures for inductive analysis (Johnson & Christensen, 2004).

resultsWhen a patient was hospitalized, he or she

was under the charge of a primary team usually comprising one consultant (equivalent to attend-ing physician), one or two senior residents, and one or two junior residents. Depending on the medical subspecialty, the team was responsible for 5 to 20 patients, from which normally less than 3 patients were handed over. The decision whether or not to hand over a patient was team-based, but usually made by senior members of the team during the exit rounds. Once the deci-sion was made, a junior resident was responsible for preparing and conducting the handover with another junior resident from the cross-covering team. Table 1 shows the types of patients and the corresponding frequency of mentions.

Since there was no handover preparation proto-col implemented at the research site during the study, the information preparation strategy varied appreciably across the participants. Among them, only one participant mentioned the preparation of a formal handover sheet. All participants stated that they checked the computerized medical record system for the completeness of the ordered

investigations if any to prepare for the handover. More than half of the participating residents (13/21) checked the patient charts and other infor-mation sources including patient lists and personal notes before the handover communication. The others (8/21) relied largely on memory-intensive strategies, although they also mentioned referring to displayed or written information sources. The contrast between relying on KIW and KIH can be illustrated by two participants’ comments: “I will have a computer in front of me [and] relevant investigations in front of me [and] charts in front of me and then I will handover. That means all the information I need to tell them is in front of me” versus “I have a patient list, [which] will note which patient for handover. Usually [I] will just recall the patient information cos [because] you [I] will know them pretty well.”

The semistructured interview revealed the use of different information retrieval strategies by the medical residents and the prevalence of relying on one’s own memory. We examined two possible causes for this in the following experiment.

SImulatIon-BaSed experImentmethod

Participants. Thirty-two junior medical resi-dents from two tertiary and referring hospitals in

Accountability

Accuracy

Confidence

Overconfidence

IAC

+ ̶

̶

̶̶

̶

̶

+

+

̶++

+Informa�onprepara�on

performance

Informa�onaccess a�empts

Informa�onaccess dura�on

Figure 1. Effect of information access cost (IAC) and accountability on information access attempts, information access duration, and information preparation performance.




Singapore participated in the simulation-based experiment. The results of two participants were excluded from the analysis because one of them marked confidence rating as 0/50 for all the questions and the other answered “not sure” for 90% of the recall questions. The remaining 30 participants had an average age of 28.1 years (SD = 3.1 years) and an average experience of 3.6 years (SD = 2.5 years). Of the 30 partici-pants, 26 were from general medicine, 3 from general surgery, and 1 from anesthesia. Each participant received 50 Singapore dollars (about US$39.40) for his or her participation.

Development of mock-up patient cases. Based on real patient cases, a group of subject matter experts (SMEs), consisting of one consultant, one senior resident, and one junior resident, developed a pool of mock-up patient cases from gastroenter-ology, neurology, respiratory, and cardiology. The SMEs followed five steps to develop the patient cases. First, they identified representative cases in the general medical ward. Second, for each case they selected one corresponding patient and cop-ied the patient’s hospitalization history lasting for 48 to 72 hr after admission. Third, the SMEs developed an exit round entry (i.e., an entry on the patient chart documenting a patient’s condition and status observed during the exit round and the primary team’s management plan for the patient during the on-call period; Figure 2). After that, rel-evant supporting information such as general lab results, X-rays, and patient pictures were assem-bled. Last, all identifiable information was anony-mized with fictional names, numbers, and dates.

Eight patient cases were selected from the pool. The selected medical conditions were con-gestive cardiac failure, pneumonia and do not

resuscitate (DNR), deep vein thrombosis, con-stipation colic, fast atrial fibrillation, fluid over-load secondary to cardiac failure, acute stroke, and gastrointestinal bleeding. A sample mock-up case is presented in Figure 3.

Furthermore, for each patient case, the SMEs developed 10 recognition and 10 recall ques-tions that were critical for the understanding of the patient case. An example recognition ques-tion is “Was the patient’s hemoglobin level above 7?” and an example recall question is “Besides chest pain, what is the other symptom the patient presenting with during exit round?” The 20 questions were used in the two recogni-tion/recall tests.

Experimental task. Figure 4 shows the flow-chart of the experimental task. Each participant studied four randomly assigned patient cases. After that, the accountability of a particular patient was announced. In the experiment, par-ticipants went through two tests, each of which included 5 recognition and 5 recall questions. In Test 1 designed to assess overconfidence, par-ticipants answered the questions depending only on their memory and provided confidence esti-mates. Confidence rating was a 6-point scale ranging from 50% to 100% for each recognition question and an 11-point scale ranging from 0% to 100% for each recall question, with the left-most point labeled “just guessing” and the right most point “absolutely confident.” In Test 2 designed to assess information retrieval strategy and performance, participants were allowed to access the original patient cases (KIW) from a PC, the location of which was manipulated either next to the desk or 5 m away depending on the IAC conditions.

TAbLE 1: Types of Patients Handed Over by Primary Teams and Frequency of Mentions

Type of Patients Handed Over Number of Mentions (out of 21)

Sick patients in need of review by on-call doctor 21Patients with pending diagnostic test results 19Patients with pending referral letter reply 5Patients with special status code (e.g., DNR) 2Patients with behavioral or mental problems 1Patients with consent forms to sign 1Patients with demanding family 1




The choice of IAC manipulation was deter-mined by the handover practices at the research sites. During the time of the study, neither of the research sites had implemented the EHR sys-tem. A large portion of patient information was recorded using paper-based patient charts, which could be located at various physical places such as patient bedsides and nurse stations, incurring different travel distances.

Through a pilot study with two medical resi-dents, the study time and difficulty of the test questions were determined. The study time per patient case was adjusted to 150 s to ensure the performance fell in the middle range. The diffi-culty of Test 1 and Test 2 questions were bal-anced by stratified randomization. The difficulty score of each recognition and recall question was obtained (based on memory accuracy results from Test 1), and used a priori to stratify the questions into hard, moderate, and easy ques-tions. The assignment of the 20 questions into Test 1 and Test 2 were then randomized within each stratum. Moreover, the difficulty score was constantly updated after each participant com-pleted the experiment.

Experimental design. There were two 2-level independent variables manipulated within sub-jects: accountability and IAC. After Tetlock and Kim (1987), in the low accountability condition, participants were assured that their performance would be completely confidential, whereas in

the high accountability condition they were informed that the test performance would be assessed at the end of the study. The second independent variable (IAC) was manifest as the physical distance required to access the patient information. In the low cost setting, the original patient cases were available on the same PC that was used to study the four patient cases, whereas in the high cost setting, the patient cases were available on a separate PC located 5 m away.

In Test 1, memory accuracy and confidence judgment were measured. Confidence was cal-culated by averaging the confidence judgment of question i (i = 1 to N), and overconfidence was calculated as the difference between confi-dence and accuracy (Koriat, Lichtenstein, & Fis-chhoff, 1980). A positive sign indicates overcon-fidence and a negative sign underconfidence.

ConfidenceN

Confidenceii

N

==∑11

Overconfidence Confidence Accuracy= −

In Test 2, number of information access attempts, the total information access duration to original patient cases, and information prepara-tion performance accuracy were measured.

The experiment followed a two by two within-subject design. To minimize the potential

Figure 2. A sample of exit round entry.




Figure 3. A sample mock-up patient case (male with onset of shortness of breath and left leg swelling) comprising patient photos, medical record, and lab results.




carryover effect, a 4 × 4 reduced Latin square design was employed (Table 2), with a 1-min break between experimental conditions.

Experimental apparatus and procedure. The information display program was coded in Visual Basic 2008 and ran on two PCs with two 24-inch color monitors. Participants first received general information about the study and signed the con-sent form. They completed a practice trial prior to the actual experiment. No feedback on perfor-mance was given during the experiment.

resultsTo account for potential order effects, we

conducted preliminary analyses using three-way mixed ANOVAs, with accountability and IAC as within-subject factors and order as the between-subject factor. No main effects of order or any Order × Accountability, Order × IAC, or Order × Accountability × IAC interactions

were significant (all ps > .05; please refer to the supplementary material for more details).

As the order effects were not significant, we excluded order from the final analysis. Table 3 summarizes the descriptive statistics for response variables under each experimental condition. Two-way repeated measures ANOVAs tested the main and interaction effects of accountability and IAC on memory accuracy, confidence, overconfi-dence, information access attempts, access dura-tion, and preparation performance.

The results revealed that accountability sig-nificantly increased memory accuracy from 55.2% to 61.2%, F(1, 29) = 7.695, p < .01, par-tial η2 = .210, and significantly reduced over-confidence from 13.9% to 8.5%, F(1, 29) = 4.395, p < .05, partial η2 = .132, which were con-sistent with past results (Ronis & Yates, 1987; Sniezek et al., 1990; Tetlock & Kim, 1987) and confirmed H1. Since IAC was not varied in Test

Announcement of accountability for Pa�ent X

Informa�on access a�empts (IAAs)Informa�on access dura�onInforma�on prepara�on performance

AccuracyConfidence

Test 2 for Pa�ent X:with access to original pa�ent

cases

4 ×

View 4 pa�ent cases

Test 1 for Pa�ent X: without access to original pa�ent

cases + confidence ra�ng

Figure 4. Flowchart of experimental task.

TAbLE 2: The 4 × 4 Reduced Latin Square Design in the Experiment

Accountability Information Access Cost Cases for Each Subject

Low Low Patient 1 Patient 2 Patient 3 Patient 4Low High Patient 2 Patient 1 Patient 4 Patient 3High Low Patient 3 Patient 4 Patient 1 Patient 2High High Patient 4 Patient 3 Patient 2 Patient 1 Subject 1 Subject 2 Subject 3 Subject 4




1 (represented only by the treatment that partici-pants would receive in subsequent Test 2), its effect on memory accuracy, confidence and overconfidence was not significant. The effect of accountability on confidence (as opposed to overconfidence) was not significant, F(1, 29) = .249, p = .622, partial η2 = .009. Therefore the increase in overconfidence was characterized by a loss in accuracy in low accountability condi-tion without a corresponding loss in confidence, reflecting a faulty calibration of metacognition (Wickens et al., 2013).

The results for Test 2 assessing the retrieval strategy and performance under varying IAC

conditions supported H2a. With relocation of the patient case data to the PC 5 m away, the number of information access attempts decreased from 6.2 to 1.8 times, F(1, 29) = 113.87, p < .001, par-tial η2 = .797, and the total information access duration decreased from 88.9 to 46.7 s, F(1, 26) = 26.34, p < .001, partial η2 = .503. In H2b, we hypothesized that an increase in IAC would degrade the information preparation perfor-mance. Indeed, there was a significant decrease in performance from 78.5% to 65.7%, F(1, 29) = 20.226, p < .001, partial η2 = .411. The effect on information access attempts and performance is consistent with the findings of Gray and Fu

TAbLE 3: Mean and 95% Confidence Interval of Accuracy, Confidence, Overconfidence, Information Access Attempts (IAAs), Information Access Duration, and Information Preparation Performance

Low Accountability High Accountability Average

Test 1: Accuracy (%; n = 30)Low IAC (0 m) 54.7 (50.5–58.8) 59.3 (54.5–64.1) 57.0 (53.9–60.1)High IAC (5 m) 55.7 (51.4–60.0) 63.0 (57.2–68.8) 59.3 (55.6–63.0)Average 55.2 (52.4–58.0) 61.2 (57.4–64.9)** 58.2 (55.7–60.6)

Test 1: Confidence judgment (%; n = 30)Low IAC (0 m) 69.2 (63.7–74.6) 68.2 (61.9–74.6) 68.7 (63.3–74.1)High IAC (5 m) 68.9 (63.8–74.1) 71.1 (66.3–75.9) 70.0 (65.7–74.4)Average 69.0 (64.3–73.7) 69.7 (64.7–74.6) 70.2 (65.2–75.3)

Test 1: Overconfidence (%; n = 30)Low IAC (0 m) 14.5 (8.8–20.2) 8.9 (1.7–16.1) 11.7 (6.2–17.2)High IAC (5 m) 13.3 (7.0–19.5) 8.1 (1.9–14.4) 10.7 (6.1–15.3)Average 13.9 (8.8–19.0) 8.5 (3.4–13.6)* 11.2 (6.8–15.6)

Test 2: IAAs (out of 10 questions; n = 30a)Low IAC (0 m) 6.5 (5.1–7.8) 5.9 (4.6–7.2) 6.2 (5.0–7.3)***High IAC (5 m) 1.5 (0.7–2.2) 2.1 (1.2–2.9) 1.8 (1.1–2.4)Average 4.0 (3.1–4.9) 4.0 (3.0–5.0) 4.0 (3.1–4.8)

Test 2: Information access duration (seconds; n = 27)Low IAC (0 m) 82.1 (62.3–102.0) 97.6 (71.6–123.6) 88.9 (70.5–109.3)***High IAC (5 m) 36.8 (16.6–57.0) 56.6 (31.8–81.3) 46.7 (25.9–67.5)Average 59.5 (41.8–77.2) 77.1 (54.2–100.0)† 68.3 (50.1–86.4)

Test 2: Information preparation performance (%; n = 30)Low IAC (0 m) 78.7 (71.2–86.1) 78.3 (72.4–84.3) 78.5 (72.4–84.6)***High IAC (5 m) 62.3 (55.4–69.3) 69.0 (62.3–75.8) 65.7 (59.6–71.7)Average 70.5 (64.1–76.9) 73.7 (68.7–78.6) 72.1 (66.7–77.4)

Note. IAC = information access cost.aData of 30 subjects were captured with four missing data points. The four data points were calculated by expectation-maximization imputation (Dempster, Laird, & Rubin, 1977) available in SPSS Statistics 21.†p < .10. *p < .05. **p < .01. ***p < .001 (contrasts with performance at low accountability and low IAC).




(2001, 2004; Gray et al., 2006) showing that as IAC increased the adoption of memory-intensive strategies increased and performance degraded.

Moreover, we found that the effect of account-ability on total information access duration approached statistical significance, F(1, 26) = 3.699, p = .065, partial η2 = .125, and this vari-able had no effect on information access attempts, F(1, 29) = 0.001, p = .979, partial η2 = .000. Thus, this finding does not support H3a. Finally, the increase of accountability also did not affect information preparation performance, F(1, 29) = 2.706, p = .11, partial η2 = .085, hence not sup-porting H3b. In examining this effect in more detail, we do note that the information prepara-tion performance was unchanged by the account-ability manipulation when the display was close (78.3% versus 78.7%), but benefited from increasing accountability when the display was 5 m away (62.3% versus 69.0%). When testing the effect of accountability on information prepara-tion performance at the high IAC level, the result showed a significant difference, F(1, 29) = 4.567, p < .05, partial η2 = .136, although the interaction was not statistically significant, F(1, 29) = 2.391, p = .13, partial η2 = .076.

General dIScuSSIon and concluSIon

Consistent with previous safety investiga-tions (Alem et al., 2008; Vidyarthi et al., 2006), our interview study revealed that a substantial portion of medical residents relied largely on memory for preparing their handover infor-mation. We reasoned that one of the possible causes may have been the high IAC of referring to displayed or written medical information. The other factor hypothesized to influence the retrieval strategy and performance is medical residents’ accountability.

When we varied access costs of KIW in our experiment we strongly confirmed such an inter-pretation: when people had to travel farther to access the displays, they made fewer visits and spent less time viewing them overall (H2a) and this reduced their final performance (H2b). The results suggest that patient information needs to be easily accessible in handover preparation. In addition, past studies have used various methods to manipulate IAC, including eye movement

versus mouse movement, varying levels of Fitts’s index and varying levels of lockout time (Ballard et al., 1995; Gray et al., 2006; Gray & Fu, 2001, 2004; Morgan, Patrick, Waldron, King, & Patrick, 2009; Morgan, Waldron, King, & Patrick, 2007). Results of this study showed that besides these factors, physical distance was a viable way to inducing varied IAC, a critical variable in health care, where patient records may be located at various locations (e.g., bed-side, doorway, at a ward-central desk).

When we manipulated accountability, partici-pants became more accountable and their memory accuracy for Test 1 items increased whereas confi-dence did not, creating the anticipated reduction in overconfidence in memory accuracy (H1); a reduction we infer would have been manifest in Test 2, even though we did not assess it there.

Furthermore, we found a marginal significant relationship between accountability and total information access duration (part of H3a). How-ever, this effect of accountability did not cascade through the overall information preparation per-formance (thus not supporting H3b). In inter-preting the null effect of H3b (accountability effect on performance), as noted, there remained the intriguing hint of an interaction between the two independent variables on performance. Fur-ther research is required to assess the interaction effect, particularly as this type of effect—the loss of performance due to low accountability when displays are distant—has profound safety implications for health care quality.

lImItatIonSThis research has some limitations and chal-

lenges, which should be taken into consideration when interpreting and generalizing the findings. First, the experiment used patient photos, medi-cal records, and diagnostic test results to simulate patient cases, which limited simulation fidelity. Since the interaction with a patient in real set-tings often takes place with additional modalities including auditory and tactile channels, and lasts for a longer time, the differences in interaction may affect information retrieval strategies and performance. However, we noted that pure text-based simulation has been used in past studies, generating satisfactory results (Bhabra, Mackeith, Monteiro, & Pothier, 2007; Pothier, Monteiro,




Mooktiar, & Shaw, 2005; Wallston et al., 2014). Also, the simulation was focused on the accu-racy of explicit information retrieved from either KIW or KIH. Handover preparation is not merely about information retrieval and its accuracy, but also other cognitive activities such as determin-ing what information to include, organizing and synthesizing patient information into a coherent picture. Further research is needed to examine these cognitive activities. Moreover, IAC was manipulated by imposing different physical dis-tance in the present simulation, whereas IAC can be affected by other factors and would be highly context dependent in real medical settings as we discussed in the introduction.

Second, the participants were junior medical residents with limited clinical experience. Thus the findings are limited in its external validity. Past literature showed that experience and train-ing sometimes may lead to better accuracy–con-fidence calibration (Keren, 1987; Lichtenstein & Fischhoff, 1977; although not invariably so—Kahneman & Klein, 2009). Therefore, more experienced clinicians may possess a smaller overconfidence bias, resulting in higher informa-tion preparation performance. It is also possible that experienced clinicians have higher levels of accountability toward patients and the manipula-tion of accountability might not be effective.

Third, the present study assumed that the accuracy of KIW is perfect, whereas of course displayed or written documents in real medical settings may contain errors (Arora, Kao, Lovinger, Seiden, & Meltzer, 2007) and are often not up-to-date. The issue of inaccurate KIW may be exacerbated by the use of EHR, for example, due to the propagation of error by cloning (i.e., copying and pasting from one record to another; Thielke, Hammond, & Helbig, 2007) and to increased documentation burden due to poor EHR usability (Karsh, Weinger, Abbott, & Wears, 2010), respectively. Also, inadequate information presentation may have detrimental effects on KIW accuracy. For instance, poor handwriting in medical records has long been identified as a significant problem in medicine (Sokol & Hettige, 2006). Such information illegibility at best increase IAC, at worst contributes to imperfect KIW, as it’s impossible to decipher the intended message.

Fourth, the present study primarily focused on handover senders during prehandover prepa-ration. In fact, both preparation and presentation are a two-way process. A recent study of Hilli-goss and Zheng (2013) examined how inpatient clinicians (handover receivers) used the “chart biopsy” in preparation for handovers from emer-gency department clinicians (handover senders). This two-way collaboration can then mitigate the shortcomings of prehandover preparation by the handover sender.

Finally, we acknowledge that limited avail-ability of participants required us to conduct the experiment with insufficient statistical power to accurately establish the strength of all of the influence links in the model shown in Figure 1. The model however can certainly act as a frame-work within which others can pursue answers to some of these important research questions.

These limitations notwithstanding, the current study examines two important phenomena that could compromise information preparation. In particular we point to the criticality of easy access to information source in handover preparation. This compromise has been observed in hospital settings when formal procedures are not fully established or complied with. Well-designed information technology and practice and organi-zational changes promoting accountability are needed to maximize the quality of prehandover preparation.

acknowledGmentSThis study was funded by the Ministry of Health,

Singapore, under Grant HSRG/0001/2010. We would like to thank the Departments of Internal Medicine, Surgery, and Anesthesia from Changi General Hospi-tal and National University Hospital for their support of the research. Preliminary results of the study have been presented at Human Factors and Ergonomics annual meetings.

key poIntS • Memory-intensive strategies when retrieving

patient information during prehandover prepara-tion were observed.

• The effects of information access cost and accountability on information retrieval strategy and performance were examined in a simulation-based experiment.




• Increasing information access cost led to more memory-intensive strategies and harmed informa-tion preparation performance.

• Higher accountability led to marginally longer information access to patient documents.

referenceSAlem, L., Joseph, M., Kethers, S., Steele, C., & Wilkinson, R.

(2008). Information environments for supporting consistent registrar medical handover. HIM Journal, 37, 9–25.

Allwood, C. M., Granhag, P. A., & Johansson, M. (2003). Increased realism in eyewitness judgments: The effect of dyadic collabo-ration. Applied Cognitive Psychology, 17, 545–561.

Arora, V., Johnson, J., Lovinger, D., Humphrey, H. J., & Meltzer, D. O. (2005). Communication failures in patient sign-out and suggestions for improvement: A critical incident analysis. Quality and Safety in Health Care, 14, 401–407.

Arora, V., Kao, J., Lovinger, D., Seiden, S. C., & Meltzer, D. (2007). Medication discrepancies in resident sign-outs and their potential to harm. Journal of General Internal Medicine, 22, 1751–1755.

Arora, V., Manjarrez, E., Dressler, D., Basaviah, P., Halasyamani, L., & Kripalani, S. (2009). Hospitalist handoffs: A systematic review and task force recommendations. Journal of Hospital Medicine, 4, 433–440.

Ballard, D. H., Hayhoe, M. M., & Pelz, J. B. (1995). Memory rep-resentations in natural tasks. Journal of Cognitive Neurosci-ence, 7, 66–80.

Bhabra, G., Mackeith, S., Monteiro, P., & Pothier, D. (2007). An experimental comparison of handover methods. Annals of the Royal College of Surgeons of England, 89, 298–300.

Bornstein, B. H., & Zickafoose, D. J. (1999). “I know I know it, I know I saw it”: The stability of the confidence-accuracy rela-tionship across domains. Journal of Experimental Psychology: Applied, 5, 76–88.

Catchpole, K. R., de Leval, M. R., McEwan, A., Pigott, N., Elliott, M. J., McQuillan, A., . . . Goldman, A. J. (2007). Patient hando-ver from surgery to intensive care: Using Formula 1 pit-stop and aviation models to improve safety and quality. Pediatric Anesthesia, 17, 470–478.

Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society, Series B, 39, 1–38.

Drury, C. G. (1994). The speed-accuracy trade-off in industry. Ergonomics, 37, 747–763.

Fitts, P. M. (1966). Cognitive aspects of information processing: III. Set for speed versus accuracy. Journal of Experimental Psychology, 71, 849–857.

Flanagan, M. E., Patterson, E. S., Frankel, R. M., & Doebbeling, B. N. (2009). Evaluation of a physician informatics tool to improve patient handoffs. Journal of the American Medical Informatics Association, 16, 509–515.

Goldschmidt, P. G. (2005). HIT and MIS: Implications of health information technology and medical information systems. Communication of the ACM, 48(10), 69–74.

Gray, W. D., & Fu, W.-T. (2001). Ignoring perfect knowledge in-the-world for imperfect knowledge in-the-head: Implications of ratio-nal analysis for interface design. CHI Letters, 3, 112–119.

Gray, W. D., & Fu, W.-T. (2004). Soft constraints in interactive behavior: The case of ignoring perfect knowledge in-the-world for imperfect knowledge in-the-head. Cognitive Science, 28, 359–382.

Gray, W. D., Sims, C. R., Fu, W.-T., & Schoelles, M. J. (2006). The soft constraints hypothesis: A rational analysis approach to resource allocation for interactive behavior. Psychological Review, 133, 461–482.

Hilligoss, P. B., & Zheng, K. (2013). Chart biopsy: An emerging medi-cal practice enabled by electronic health records and its impacts on emergency department-inpatient admission handoffs. Journal of American Medical Informatics Association, 20, 206–267.

Jeffcott, S. A., Evans, S. M., Cameron, P. A., Chin, G. S. M., & Lbrahim, J. E. (2009). Improving measurement in clinical handover. Quality and Safety in Health Care, 18, 272–276.

Johnson, B., & Christensen, L. (2004). Educational research: Quantitative, qualitative, and mixed approaches (2nd ed.). Boston, MA: Pearson.

Kahneman, D., & Klein, G. (2009). Conditions for intuitive exper-tise: A failure to disagree. American Psychologist, 64, 515–526.

Karsh, B. T., Weinger, M. B., Abbott, P. A., & Wears, R. L. (2010). Health information technology: Fallacies and sober realities. Journal of the American Medical Informatics Association, 17, 617–623.

Keren, G. (1987). Facing uncertainty in the game of bridge: A cali-bration study. Organizational Behavior and Human Decision Processes, 39, 98–114.

Kerr, M. P. (2002). A qualitative study of shift handover practice and function from a socio-technical perspective. Journal of Advanced Nursing, 37, 123–134.

Koriat, A., Lichtenstein, S., & Fischhoff, B. (1980). Reasons for confidence. Journal of Experimental Psychology: Learning, Memory, and Cognition, 6, 107–118.

Lerner, J. S., & Tetlock, P. E. (1999). Accounting for the effects of accountability. Psychological Bulletin, 125, 255–275.

Lichtenstein, S., & Fischhoff, B. (1977). Do those who know more also know more about how much they know? Organizational Behavior and Human Decision Processes, 20, 159–183.

Morgan, P. L., Patrick, J., Waldron, S. M., King, S. L., & Patrick, T. (2009). Improving memory after interruption: Exploiting soft constraints and manipulating information access cost. Journal of Experimental Psychology: Applied, 15, 291–306.

Morgan, P. L., Waldron, S. M., King, S. L., & Patrick, J. (2007, July). Harder to access, better performance? The effects of information access cost on strategy and performance. Paper presented at HCII 2007, Beijing, China.

Nemeth, C., Nunnally, M. E., O’Connor, M. F., Klock, P. A., & Cook, R. I. (2005). Making information technology a team player in safety: The case of infusion devices (Vol. 1). Wash-ington, DC: Agency for Healthcare Research and Quality.

Pelham, B. W., & Neter, E. (1995). The effect of motivation of judgment depends on the difficulty of the judgment. Journal of Personality and Social Psychology, 68, 581–594.

Petersen, L. A., Brennan, T. A., & O’Neil, A. C. (1994). Does housestaff discontinuity of care increase the risk of pre-ventable adverse events? Annals of Internal Medicine, 121, 866–872.

Pothier, D., Monteiro, P., Mooktiar, M., & Shaw, A. (2005). Pilot study to show the loss of important data in nursing handover. British Journal of Nursing, 14, 1090–1093.

Raduma-Tomàs, M. A., Flin, R., Yule, S., & Close, S. (2011). Doc-tors’ handovers in hospitals: A literature review. BMJ Quality & Safety, 20, 128–133.

Raduma-Tomàs, M. A., Flin, R., Yule, S., & Close, S. (2012). The importance of preparation for doctors’ handovers in an acute medical assessment unit: A hierarchical task analysis. BMJ Quality & Safety, 21, 211–217.




Ronis, D. L., & Yates, J. F. (1987). Components of probability judgment accuracy: Individual consistency and effects of sub-ject matter and assessment method. Organizational Behavior and Human Decision Processes, 40, 193–218.

Sniezek, J. A., Paese, P. W., & Switzer, F. S., III. (1990). The effect of choosing on confidence in choice. Organizational Behavior and Human Decision Processes, 46, 264–282.

Sokol, D. K., & Hettige, S. (2006). Poor handwriting remains a significant problem in medicine. Journal of the Royal Society of Medicine, 99, 645–646.

Tetlock, P. E., & Kim, J. I. (1987). Accountability and judgment processes in a personality prediction task. Journal of Personal-ity and Social Psychology, 52, 700–709.

Thielke, S., Hammond, K., & Helbig, S. (2007). Copying and past-ing of examination within the electronic medical record. Inter-national Journal of Medical Informatics, 76S, 122–128.

Vidyarthi, A. R., Arora, V. M., Schnipper, J. L., Wall, S. D., & Wachter, R. M. (2006). Managing discontinuity in academic medical centres: Strategies for a safe and effective resident sign-out. Journal of Hospital Medicine, 1, 257–266.

Viitanena, J., Hypponen, H., Laaveri, T., Vanska, J., Reponen, J., & Winblad, I. (2011). National questionnaire study on clini-cal ICT systems proofs: Physicians suffer from poor usability. International Journal of Medical Informatics, 80, 708–725.

Wallston, K. A., Slagle, J. M., Speroff, T., Nwosu, S., Crimin, K., Feurer, I. D., Boettcher, B., & Weinger, M. B. (2014). Operat-ing room clinicians’ ratings of workload: A vignette simulation study. Journal of Patient Safety, 10, 95–100.

Weigold, M. F., & Schlenker, B. R. (1991). Accountability and risk taking. Personality and Social Psychology Bulletin, 17, 25–29.

Wickens, C. D., Hollands, J. G., Parasuraman, R., & Banbury, S. (2013). Engineering psychology and human performance (4th ed.). Boston, MA: Pearson.

X. Jessie Yang is a SUTD-MIT postdoctoral fellow at Singapore University of Technology and Design and Massachusetts Institute of Technology. She earned a

PhD in mechanical and aerospace engineering (human factors) from Nanyang Technological University, Sin-gapore in 2014.

Christopher D. Wickens is an affiliated professor at the Department of Psychology, Colorado State Uni-versity. He is also professor emeritus at the Univer-sity of Illinois at Urbana-Champaign and a senior scientist at Alion Science, Boulder, Colorado. He earned his PhD in psychology from the University of Michigan in 1974.

Taezoon Park is an associate professor in the Depart-ment of Industrial Information Systems Engineering, Soongsil University, South Korea. He worked for Nan-yang Technological University, Singapore, until 2013 after earning a PhD from Purdue University in 2008.

Liesel Fong is a medical officer at the Department of Internal Medicine, National University Hospital, Singapore. She earned her MBBS degree from National University of Singapore in 2012.

Kewin T. H. Siah is an associate consultant in the Department of Internal Medicine, National Univer-sity Hospital, Singapore. He was accredited as advanced specialist in gastroenterology by the Spe-cialist Accreditation Board (Singapore) in 2014.

Date received: October 22, 2014Date accepted: July 8, 2015


effects of information access cost and accountability on ...xijyang/assets/yang_hf_2015.pdf ·...

Documents