MAN VS. MACHINE: A NEW APPROACH TO HAND

HYGIENE AUDITING Jocelyn Srigley, MD, MSc, FRCPC

Corporate Director, Infection Control, Provincial Health Services Authority Medical Microbiologist, BC Children’s & Women’s Hospital

May 5, 2015

• No conflicts of interest • Received salary support from the AMMI

Canada/Astellas Post-Residency Fellowship

• Project supported by grants from Canada Health Infoway and the Health Technology Exchange, and by Infonaut Inc. and GOJO Industries Inc.

Disclosures

• To review methods of hand hygiene compliance monitoring

• To present new evidence supporting the existence of the Hawthorne effect in hand hygiene compliance monitoring

• To review the evidence for the efficacy of hand hygiene monitoring technology (HHMT)

• To discuss the implications of these findings, including public reporting of hand hygiene rates and the use of HHMT

Objectives

Hand Hygiene Compliance Monitoring

• Improving healthcare worker (HCW) hand hygiene compliance is an important way to reduce healthcare-associated infections (HAIs)

• A multifaceted approach, including measurement of compliance, is recommended1

• Measurement options include:2

– Direct observation – Self-report – Product consumption – Hand hygiene monitoring technology

Measuring Hand Hygiene Compliance

1WHO, 2009. 2Boyce, 2011.

• “Gold standard” • Advantages

– Only method that can assess all indications for hand hygiene

– One of the few methods that can assess technique – Provides opportunity for education

• Disadvantages – Labour intensive – Can only assess small samples of hand hygiene

opportunities – Questionable inter-rater reliability – Potential for errors in measurement

Direct Observation

2Boyce, 2011. 3Haas et al., 2007.

• Gaming • Bias

– Observer bias • Unit-based observers report higher compliance

rates than non-unit-based observers4 – Selection bias – Hawthorne effect

Sources of Measurement Error

4Dhar et al., 2010.

• Hand hygiene monitoring technology – Counters – Electronic monitoring systems – e.g. real-

time locating systems (RTLS), radiofrequency identification (RFID)

– Video monitoring

A New Solution?

The Hawthorne Effect

• Tendency of people to change their behavior when they are aware of an observer3

• Original Hawthorne studies 1924-19325,6 – Concept was not mentioned until 19507

• Many authors have since questioned whether the original Hawthorne studies actually showed a Hawthorne effect8

• However, the Hawthorne effect is widely assumed to play a role in hand hygiene behavior

The Hawthorne Effect

3Haas et al., 2007. 5Mayo, 1933. 6Roethlisberger et al., 1939. 7Adair, 1984. 8Jones, 1992.

• 17 studies of the Hawthorne effect and hand hygiene • 6 in public washrooms:

– 90% vs. 16%9

– 77% vs. 39%10 – 90% vs. 70%11 – 91% vs. 55%12 – 90% vs. 44%13 – 79% vs. 73%14

• 2 in other non-health care settings – Petting zoos,15 homes16

• 9 related to hospital hand hygiene compliance monitoring17-25

Does the Hawthorne Effect Exist?

9Pedersen et al., 1986. 10Munger et al., 1989. 11Edwards et al., 2002. 12Drankiewicz et al., 2003. 13Nalbone et al., 2005. 14Monk-Turner, et al., 2005. 16Erdozain et al., 2013. 16Ram et al., 2010.

• Hand hygiene compliance increased when audits were announced to units in advance compared to when they were unannounced:

– 9.1% to 29.5%20 – 29% to 45%21

– 47.4% to 55.7%22

• Increased hand hygiene compliance on high-performing units when an overt auditor (known to the units) was compared to a covert auditor, but not on low-performing units23

• Compliance rate as measured by medical students (44.1%) was significantly

lower than those measured by infection control nurses (74.4%) and unit HH ambassadors (94.1%)24

Studies of Hospital Audits

20Tibballs, 1996. 21Eckmanns et al., 2006. 22Maury et al., 2006. 23Kohli et al., 2009. 24Pan et al., 2013.

• Pilot study of an RTLS • Hand hygiene compliance was 88.9%

during audits, compared to an overall compliance of 31.5% for the days when the audits took place

• Limited by small sample size and lack of controlling for potential confounders

Use of Electronic Monitoring to Measure the Hawthorne Effect

25Cheng et al., 2011.

• Some evidence to support the existence of a Hawthorne effect in hand hygiene compliance, but existing studies have significant limitations

• Little is known about temporal and spatial boundaries of the Hawthorne effect or modifying factors that may play a role

Summary of Literature

• To determine the magnitude of the Hawthorne effect in hand hygiene compliance monitoring using an electronic monitoring system

Study Objective

• RTLS was installed on two multi-organ transplant units from July 2012 to March 2013

• Generated continuous real-time location data via ultrasound tags worn by staff and patients

• Measured every use of alcohol-based hand rub (ABHR) and soap dispensers

Electronic Monitoring System

Patients, staff and equipment wear active tags.

Active tags send location information every few seconds over a wireless network.

Unit Floor Plan

• Retrospective cohort study • Cohort = dispensers on the two units • Exposure = presence of auditor • Outcome = hand hygiene event rate (#

dispenses per dispenser per hour)

Study Design

• Auditors wore system tags to track the exact time of auditing and their location on the units Auditors were blinded to study hypothesis and conducted

audits as per usual practice in accordance with the Just Clean Your Hands program

• Audits were performed 1-2 times monthly on each unit from November 2012 to March 2013

• Number of dispenses was determined for areas within eyesight of the auditor when he/she was in a defined location for at least 5 minutes Separate counts for dispensers in rooms and hallways

• Count was converted into an event rate per dispenser per hour

Data Collection

• Area of the unit not visible to the auditor at the same time period during the audit – Control for confounding related to time

• Same area where the auditor was located at 1, 2, and 3 weeks prior to the audit – Control for confounding related to location

• Same area where the auditor was located 5 minutes prior to auditor’s arrival – Reverse causality bias

Comparisons

0

0.5

1

1.5

2

2.5

3

3.5

4

Out* In Total**

Events per dispenser per

hour AuditorOther Area

Results: Location Comparison

*p=0.001 **p=0.008

0

0.5

1

1.5

2

2.5

3

3.5

4

Out* In Total*

Events per dispenser per

hour Auditor1 Week Prior

Results: Time Comparison

*p<0.001

0

0.5

1

1.5

2

2.5

3

3.5

4

Out* In Total**

Events per dispenser per

hour AuditorPrior to Arrival

Results: Prior to Auditors’ Arrival

*p=0.009 **p=0.003

• Hand hygiene event rate is ~3 times higher within eyesight of the auditor compared to other locations at the same time and the same location in previous weeks – The effect is seen only in hallway dispensers,

where the auditor can be seen, and not inside patient rooms

– The increase in event rate happens after the auditor’s arrival, not before

Summary of Results

• Observational study, therefore cannot attribute causality

• Measuring hand hygiene events, not HCW compliance – To get a denominator, all HCWs would have to be

wearing system tags • Some hand hygiene events may have been performed

by untagged HCWs or visitors • System itself may have caused a Hawthorne effect • Study conducted with a relatively small number of

observations on multi-organ transplant units

Limitations

Efficacy of Hand Hygiene Monitoring Technology

• Validity – How accurately do electronic/video monitoring systems

(EMS/VMS) measure hand hygiene compliance? – Limited data

• Efficacy – Do EMS/VMS lead to improvements in hand hygiene

compliance? – Potential mechanisms

• Feedback • Real-time reminders • Enhanced Hawthorne effect

Is Technology the Answer?

• To determine whether HHMT increases directly observed hand hygiene compliance among HCWs compared to usual care

• To determine whether HHMT reduces HAI incidence or improves other measures of hand hygiene compliance, including: – Hand hygiene frequency – Volume of soap and ABHR use – Compliance as defined by the individual HHMT [i.e.

system-defined compliance (SDC)]

Study Objectives

• Systematic review following PRISMA guidelines26

• Searched multiple databases from inception until Dec 31, 2013

• Eligibility criteria – Experimental and quasi-experimental studies of HHMT

conducted in acute or long-term care that measured hand hygiene and/or HAI incidence

– Excluded if HHMT was installed solely to evaluate another intervention or if study focused on hand hygiene at ward/hospital entrances or in OR

– Peer-reviewed, English language publications • All steps in selection, data extraction and risk of bias

assessment27,28,29 performed independently by 2 authors

Methods

26Moher et al., 2009. 27Higgins et al., 2011. 28Harris et al., 2004. 29Schweizer et al., 2014.

Search Results

Author, Year

Study Design

Study Setting Popula-tion

HHMT type

Events tracked

Movement tracking

Feedback Real-Time Reminders

Outcomes Compliance definition

Results

Swoboda, 200430

Pretest-posttest study

Intermediate care unit

All HCW and visitors

EMS ABHR + soap

Room exit No Voice prompt

SDC, nosocomial infection rate

Proportion of room exits with a hand hygiene event prior to or within 10s of exit

P1 (monitoring): 19.1% P2 (monitoring + reminders): 27.3% P3 (monitoring): 24.1% P2 vs. P1: +8.2%* P3 vs. P1: +5%

Ventkatesh, 200831

Pretest-posttest study

Haematology ward

All HCW and visitors

EMS ABHR Room entry/exit

No Voice and sound prompt

SDC, VRE transmission

Proportion of room entries/exits with a hand hygiene event

P1 (monitoring): 36.3% P2 (monitoring + reminders): 70.1% P2 vs. P1: +33.8%*

Reminders without Feedback

Feedback without Reminders

Author, Year

Study Design Study Setting

Popula-tion HHMT type

Events tracked

Movement tracking

Feedback Real-Time Reminders

Outcomes Compliance definition

Results

Armellino, 201232

Interrupted time series

Medical ICU All HCW VMS ABHR + soap

Room entry/exit

Aggregate, continuous

No SDC Proportion of room entries/exits with a hand hygiene event prior to or within 10s of entry/exit where time in room > 60 seconds

P1(monitoring): 6.5% P2(monitoring + feedback): 81.6% P3(monitoring + feedback): 87.9% P2 vs. P1: +75.1%* P3 vs. P1: +81.4%*

Armellino, 201333

Interrupted time series

Surgical ICU All HCW VMS ABHR + soap

Room entry/exit

Aggregate, continuous

No SDC As above P1(monitoring): 30.4% P2(monitoring + feedback): 82.3% P2 vs. P1: +51.9%*

Marra, 201034

Non-randomised, controlled trial

Step down unit (2)

All HCW and visitors

EMS ABHR Not tracked Aggregate, 2/wk

No Hand hygiene frequency, nosocomial infection rate

N/A Control (monitoring): 110718 Intervention (monitoring + feedback): 117579 Intervention vs. control: +6861

Feedback and Reminders

Author, Year

Study Design

Study Setting

Population HHMT type

Events tracked

Movement tracking

Feedback Real-Time Reminders

Outcomes Compliance definition

Results

Levchenko, 201335

Pretest-posttest study

Chronic care ward

14 nurses EMS ABHR + soap

Room entry/exit

Individual, 2/wk

Vibration SDC, hand hygiene event rate

Proportion of room entries/exits with a hand hygiene event within 60s prior to entry or 20s prior to exit (‘clean’) or within 20s of vibratory reminder (‘performed after prompt’)

P1(monitoring): 2.97 P2(monitoring + feedback): 2.84 P3(monitoring + feedback + reminders) : 6.61 P2 vs. P1: -0.13 P3 vs. P1: +3.64

Fisher, 201336

RCT 2 wards + surgical ICU

231 nurses EMS ABHR Zone entry/exit

Individual, 1/wk

Vibration SDC Proportion of zone entries/exits with a hand hygiene event within 6s of entry or 60s of exit

Intervention (monitoring + feedback + reminders) vs. Control (monitoring): +6.8%*

• No studies met primary objective (directly observed compliance)

• Study at lowest risk of bias showed no clinically significant effect of an RTLS

• VMS appear promising but studies at moderate risk of bias

• Insufficient evidence to recommend adoption of HHMT as an improvement strategy

• Future trials must include stronger designs, control groups, and system independent measures of hand hygiene

Summary of Results

Implications

• BC, Ontario, and other provinces have made hospital hand hygiene compliance rates publicly reportable

• Public reporting may increase the potential for gaming and bias37

• Indicator-based improvement vs. evidence-based improvement

Public Reporting

37Muller et al., 2011.

• BC provincial average (FY2013-14) – 72% for moment 1 and 81% for moment 438

• Ontario provincial average (FY2013-14) – 86% for moment 1 and 91% for moment 439

• A systematic review found a median HH compliance rate of 40%40

• Despite the significant increase in compliance since public reporting began in Ontario, there has been no change in HAI rates41

Reported Rates

38PICNet, 2015. 39Health Quality Ontario, 2015. 40Erasmus et al., 2010. 41Didiodato, 2013.

• Actual HCW hand hygiene compliance rates are not as high as reported – May be up to 3x lower

• Potential solutions – Stop public reporting – Change from direct observation to another

method of hand hygiene compliance measurement

Reality Check

Direct Observation Hand Hygiene Monitoring Technology

Subject to observer and selection biases Lower likelihood of bias

Hawthorne effect May be less subject to Hawthorne effect

Questionable inter-rater reliability Consistent, algorithm-based data collection

Few observations, usually during peak hours of patient care activity

Constant, real-time assessment of all hand hygiene behaviour

Measures all 4 moments of hand hygiene Often uses room entry/exit as denominator

Ability to assess technique Most systems unable to assess technique

Can provide feedback/education to HCWs May or may not provide feedback

Generally accepted by HCWs May be less acceptable to HCWs

Labour intensive Can be expensive to install and maintain

Can compare across facilities Unable to compare compliance rates between systems

Comparison of Methods

• HHMT has advantages but it is not a panacea – Institutions need to weigh costs and benefits in

their particular setting – Ideally HHMT should be installed with the goal of

conducting evaluation/research • Ongoing efforts are necessary to truly improve hand

hygiene compliance and reduce HAIs – Focus on changing individual behaviour and

organizational culture – Frontline ownership (FLO)

Conclusions

Acknowledgements

Hawthorne co-authors: Dr. Michael Gardam Dr. G. Ross Baker Dr. Colin Furness

Systematic review co-authors: Dr. Matthew Muller Dr. Michael Gardam Dr. Geoff Fernie David Lightfoot Dr. Gerald Lebovic

Research staff: Mary Jane Salpeter Nijusha Barmala Timur Sharaftinov MSc advisors: Dr. Geoff Anderson Dr. Whitney Berta Dr. Monique Herbert Dr. Laura Rosella Dr. Gerald Evans

Questions?

jocelyn.srigley@cw.bc.ca

1. World Alliance for Patient Safety. WHO guidelines on hand hygiene in health care (May 2009). Geneva, Switzerland: World Health Organization, 2009; [cited May 27, 2013]. Available from: http://whqlibdoc.who.int/publications/2009/9789241597906_eng.pdf.

2. Boyce JM. Measuring healthcare worker hand hygiene activity: current practices and emerging technologies. Infect Control Hosp Epidemiol 2011;32(10):1016-28.

3. Haas JP, Larson EL. Measurement of compliance with hand hygiene. J Hosp Infect 2007;66(1):6-14. 4. Dhar S, Tansek R, Toftey EA, et al. Research brief: observer bias in hand hygiene compliance reporting. Infect

Control Hosp Epidemiol 2010;31(8):869-70. 5. Mayo E. The human problems of an industrial civilization. New York: MacMillan, 1933. 6. Roethlisberger FJ, Dickson WJ. Management and the worker. Cambridge, Mass.: Harvard University Press, 1939. 7. Adair JG. The Hawthorne effect: a reconsideration of the methodological artifact. J Appl Psychol 1984;69(2):334-45. 8. Jones SRG. Was there a Hawthorne effect? Am J Sociol 1992;98(3):451-68. 9. Pedersen DM, Keithly S, Brady K. Effects of an observer on conformity to handwashing norm. Percept Motor

Skill 1986;62:169-70. 10. Munger K, Harris SJ. Effect of an observer on handwashing in a public restroom. Percept Motor Skill 1989;69:733-4. 11. Edwards D, Monk-Turner E, Poorman S, Rushing M, Warren S, Willie J. Predictors of hand-washing behavior. Soc

Behav Personal 2002;30(8):751-6. 12. Drankiewicz D, Dundes L. Handwashing among female college students. Am J Infect Control 2003;31:67-71. 13. Nalbone DP, Lee KP, Suroviak AR, Lannon JM. The effects of social norms on male hygiene. Individual Differences

Res 2005;3(3):171-6. 14. Monk-Turner E, Edwards D, Broadstone J, Hummel R, Lewis S, Wilson D. Another look at hand-washing behavior.

Soc Behav Personal 2005;33(7):629-34.

References

15. Erdozain G, KuKanich K, Chapman B, Powell D. Observation of public health risk behaviours, risk communication and hand hygiene at Kansas and Missouri petting zoos – 2010-2011. Zoonoses Public Hlth 2013; 60:304-10.

16. Ram PK, Halder AK, Granger SP, Jones T, Hall P, Hitchcock D, et al. Is structured observation a valid technique to measure handwashing behavior? Use of acceleration sensors embedded in soap to assess reactivity to structured observation. Am J Trop Med Hyg 2010;83(5):1070-6.

17. Pittet D, Simon A, Hugonnet S, Pessoa-Silva CL, Sauvan V, Perneger TV. Hand hygiene among physicians: performance, beliefs, and perceptions. Ann Intern Med 2004;141:1-8.

18. Chen LF, Carriker C, Staheli R, Isaacs P, Elliott B, Miller BA, et al. Observing and improving hand hygiene compliance: implementation and refinement of an electronic-assisted direct-observer hand hygiene audit program. Infect Control Hosp Epidemiol 2013;34(2):207-10.

19. Bittner MJ, Rich EC, Turner PD, Arnold WH. Limited impact of sustained simple feedback based on soap and paper towel consumption on the frequency of hand washing in an adult intensive care unit. Infect Control Hosp Epidemiol 2002;23:120-6.

20. Tibballs J. Teaching hospital medical staff to handwash. Med J Aust 1996;164:395-8. 21. Eckmanns T, Bessert J, Behnke M, Gastmeier G, Rüden H. Compliance with antiseptic hand rub use in intensive care

units: the Hawthorne effect. Infect Control Hosp Epidemiol 2006;27:931-4. 22. Maury E, Moussa N, Lakermi C, Barbut F, Offenstadt G. Compliance of health care workers to hand hygiene:

awareness of being observed is important. Intens Care Med 2006;32:2088-9. 23. Kohli E, Ptak J, Smith R, Taylor E, Talbot EA, Kirkland KB. Variability in the Hawthorne Effect with regard to hand

hygiene performance in high- and low-performing inpatient care units. Infect Control Hosp Epidemiol 2009;30:222-5. 24. Pan SC, Tien KL, Hung IC, Lin YJ, Sheng WH, Wang MJ, et al. Compliance of health care workers with hand

hygiene practices: independent advantages of overt and covert observers. PLoS ONE 2013;8(1):e53746. 25. Cheng VC, Tai JW, Ho SK, et al. Introduction of an electronic monitoring system for monitoring compliance with

Moments 1 and 4 of the WHO "My 5 Moments for Hand Hygiene" methodology. BMC Infect Dis 2011;11:151.

References

26. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009;6:e1000097.

27. Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928.

28. Harris AD, Bradham DD, Baumgarten M, Zuckerman IH, Fink JC, Perencevich EN. The use and interpretation of quasi-experimental studies in infectious diseases. Clin Infect Dis 2004;38:1586-91.

29. Schweizer ML, Reisinger HS, Ohl M, et al. Searching for an optimal hand hygiene bundle: a meta-analysis. Clin Infect Dis 2014;58:248-59.

30. Swoboda SM, Earsing K, Strauss K, Lane S, Lipsett PA. Electronic monitoring and voice prompts improve hand hygiene and decrease nosocomial infections in an intermediate care unit. Crit Care Med 2004;32:358-63.

31. Venkatesh AK, Lankford MG, Rooney DM, Blachford T, Watts CM, Noskin GA. Use of electronic alerts to enhance hand hygiene compliance and decrease transmission of vancomycin-resistant Enterococcus in a hematology unit. Am J Infect Control 2008;36:199-205.

32. Armellino D, Hussain E, Schilling ME, et al. Using high-technology to enforce low-technology safety measures: the use of third-party remote video auditing and real-time feedback in healthcare. Clin Infect Dis 2012;54:1-7.

33. Armellino D, Trivedi M, Law I, et al. Replicating changes in hand hygiene in a surgical intensive care unit with remote video auditing and feedback. Am J Infect Control 2013;41:925-7.

34. Marra AR, D'Arco C, Bravim BA, et al. Controlled trial measuring the effect of a feedback intervention on hand hygiene compliance in a step-down unit. Infect Cont Hosp Epidemiol 2008;29:730-5.

35. Levchenko AI, Boscart VM, Fernie GR. The effect of automated monitoring and real-time prompting on nurses' hand hygiene performance. Comput Inform Nurs 2013;31:498-504.

References

36. Fisher DA, Seetoh T, Oh May-Lin H, et al. Automated measures of hand hygiene compliance among healthcare workers using ultrasound: validation and a randomized controlled trial. Infect Control Hosp Epidemiol 2013;34:919-28.

37. Muller M, Detsky A. Public reporting of hospital hand hygiene compliance—helpful or harmful? J American Med Assoc 2011;304:1116-7.

38. Provincial Infection Control Network of British Columbia (PICNet). Hand cleaning compliance in BC acute care facilities: fiscal year 2013/2014. Vancouver, BC: Provincial Infection Control Network of British Columbia (PICNet), June 2014; [cited 2015-04-07]. Available from: https://www.picnet.ca/wp-content/uploads/PICNet-HCC-Annual-Report-2013_2014.pdf

39. Health Quality Ontario [homepage on the Internet]. Toronto, ON: Queen's Printer for Ontario, 2015; [cited 2015-04-07]. Public Reporting; [background screen]. Available from: http://www.hqontario.ca/public-reporting/patient-safety.

40. Erasmus V, Daha TJ, Brug H, et al. Systematic review of studies on compliance with hand hygiene guidelines in hospital care. Infect Control Hosp Epidemiol 2010;31(3):283-94.

41. DiDiodato G. Has improved hand hygiene compliance reduced the risk of hospital-acquired infections among hospitalized patients in Ontario? Analysis of publicly reported patient safety data from 2008 to 2011. Infect Control Hosp Epidemiol 2013;34:605-10.

References

#CLEANSHOTS Subtitle Goes Here

336 Clean Shots More than 11,700 gallery visits More than 5,700 votes

#CLEANSHOTS

Thank you to everyone who

took part and made this campaign a success!

#CLEANSHOTS