Public Reporting of Hospital Infection Rates:Ranking the States on Credibility and User Friendliness

CSTE 2013 Conference, June 2013Ava Amini, David Birnbaum, Bernard

Black & David Hyman

History of Mandatory Public Reporting

in the United States In 1999, the Institute of Medicine published To Err

is Human, a review of the frequency of adverse outcomes.

By the time a Healthcare Infection Control Practices Advisory Committee (HICPAC) guidance document was published (McKibben et al., 2005), four states had enacted legislation requiring public disclosure of healthcare-associated infection rates.

By 2006, eight states had mandated reporting of healthcare-associated infection rates.

The state-law mandated list kept growing. Federal initiatives effectively made the mandate

universal. 2009 - HHS Requirement of State Action Plans

2011 - CMS IPPS rules

HAI was unfamiliar territory for state health departments.

Healthcare Facility Reporting to CMS via NHSN: Current and Proposed HHS RequirementsHAI Events Facility Type Start Date

CLABSIs Acute Care Hospital ICUs Jan 2011

CAUTIs Acute Care Hospital ICUs (except NICUs) Jan 2012

SSIs Acute Care Hospitals Colon Surgeries & Abdominal Hysterectomies

Jan 2012

Dialysis Events Dialysis facilities Jan 2012

CLABSIs Long Term Care Hospitals Oct 2012

CAUTIs Long Term Care Hospitals Oct 2012

CAUTIs Inpatient Rehab Facilities Oct 2012

MRSA Bacteremia LabID Events

Acute Care Hospitals Jan 2013

C difficile LabID Events Acute Care Hospitals Jan 2013

HCW Influenza Vaccination Acute Care Hospitals Jan 2013

HCW Influenza Vaccination Ambulatory Surgery Centers (ASCs) Oct 2014

SSIs and other events ASCs and Hospital Outpatient Departments TBD

Does the information influence perceptions?

What do the higher performing places know or do that the lower performing hospitals don’t (a key feature being engagement of physicians)?

How will we set systems to detect impact, and know what motivated change?

Can we automate signal detection?

Do HAI rates improve over time and do they improve more in areas with versus without reporting?

Does the process drive itself to be continuous, or are external motivators needed?

Does the site meet the users needs?

What are the information seeking behaviors of different kinds of users?

How will people even know the information service is there?

Who are the users?

What kinds of people use this type of information source and for what purposes?

Action to Improve

Health Service Research

Hospital Epidemiology

Socio

logy/M

edical

Anthropology

Syste

ms Engin

eering

Group Dynamics and Organizational Development

Knowledge Translation

Marketing Psychology

Quality Sciences

Communication of Scientific Information

Informatics

Information Delivery

HAI Program

Risk Communica

tion

Patient Expectations and Patient-ProviderCommunication

communication with staff?

Does visiting HAI websites: influence expectations, influence level of trust, and/or promote

How does communication skills of staff reinforce or reduce patient satisfaction relative to initial expectations and perceived or actual adverse events?

Improving our understanding of what works, for whom, in what settings

Key Unanswered Questions

• What kinds of people use this type of information source and for what purposes?– What are the

information seeking behaviors of different kinds of users?

• How well does the site meet the users needs?

• Elizabeth Borycki & André Kushniruk’s team

• Bernie Black & David Hyman’s team

What Constitutes User Friendliness?

• In General– Nielsen J. Usability engineering. New York: Academic Press; 1993.– Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation

of clinical information systems. Journal of Biomedical Informatics 2004;37:56-76.– Cleveland WS. Visualizing Data. Summit, NJ: Hobart Press; 1993.– Wheildon C. Type & layout: Are you communicating or just making shapes. Mentone,

Australia: Worsley Press, 2005.– Krug S. Don’t make me think: A common sense approach to web usability, 2nd edition.

Berkeley, CA: New Riders; 2008.• HAI-Specific

– Mazor KM, Dodd KS. A qualitative study of consumers’ views on public reporting of health care-associated infections. American Journal of Medical Quality; 2009;24(5):412-418.

– Mazor KM, Dodd KS, Kunches L. Communicating hospital infection data to the public: A study of consumer responses and preferences. American Journal of Medical Quality 2009;24(2):108-115.

What Contributes to Credibility?

First-draft Scoring Tool (fall 2012)

User Friendliness Scoring 1. Free access? 2. Is consumer path to basic reports obvious? 3. Ease of finding data explanations? 4. Ease of website use for hospital comparison? 5. Introduction to the topic (what HAIs are… types…) 6. Is data meaning explained? 7. Explanation helps consumers integrate information from multiple

indicators? 8. Uses numbers as well as graphs or symbols to convey numeric or

statistical information? 9. Consumer can locate specific hospitals, sensible comparisons?10. Format is simple & brief, not too much technical language?11. Understandable & useful to different groups (average consumers,

physicians & sophisticated consumers, infection control professionals, insurers, researchers)?

Credibility Scoring 1. Make it easy to verify the accuracy of the information on your site.

You can build web site credibility by providing third-party support (citations, references, source material) for information you present, especially if you link to this evidence. Even if people don't follow these links, you've shown confidence in your material.

2. Show that there’s a real organization behind your site. Showing that your web site is for a legitimate organization will boost the site's credibility. The easiest way to do this is by listing a physical address. Other features can also help, such as posting a photo of your offices or listing a membership with the chamber of commerce.

3. Highlight the expertise in your organization and in the content and services you provide. Do you have experts on your team? Are your contributors or service providers authorities? Be sure to give their credentials. Are you affiliated with a respected organization? Make that clear. Conversely, don't link to outside sites that are not credible. Your site becomes less credible by association.

4. Show that honest & trustworthy people stand behind your site… 5. Make it easy to contact you… 6. Design your site to make it look professional… 7. Make your site easy to use… 8. Update site often (at least show that its been reviewed recently)… 9. Avoid promotional content (adds, offers)…10. Avoid errors of all types… typographical… broken links…

First-draft Scoring Tool (fall 2012)

User Friendliness ScoringFree access?0 = No1 = Yes

Is consumer path to basic reports obvious?1 = HAI report or website is not first search result. Very difficult to find actual report.2 = HAI report or website is not first search result. Requires multiple clicks (at least 3)

throughout website to locate actual report. Not necessarily clear or obvious path.3 = HAI report or website is not first search result. May take more than one (but less than

three clicks) to access report once on HAI website. Alternatively, may not necessarily be first search result, but within first five AND when click on link, user goes directly to HAI report page.

4 = First search result to state HAI website. It takes more than one additional click to access actual report, but steps to find report are clear.

5 = First search result directly to state HAI report or to HAI website, which contains clear and immediately-identifiable link

Ease of finding data explanations?1 = Little to no explanation, either within report itself or on state HAI website.2 - Explanation not within actual report but on website. Some definitions provided, but little

to nothing explaining results, significance, implications.3 - Explanation within actual report or relatively easy to find on state HAI website. More than

merely definitions; some background, explanation, significance of data.4 - Explanation of infection and infection rates and statistical significance within actual report.5 = Explanation within actual report. Provides explanation regarding calculation,

interpretation, results. and analysis. Also, explanation is located prior to each specific chart or report instead provides references to charts in headers or otherwise makes it clear for user to understand which data the explanation is referencing.

Etc.

Credibility ScoringMake it easy to verify the accuracy of the information on your site. You can build web site

credibility by providing third-party support (citations, references, source material) for information you present, especially if you link to this evidence. Even if people don't follow these links, you've shown confidence in your material. (1 = bad to 5 = excellent)

1 = No citations, references, or source material.2 = No references within actual report, but website includes one to two links.3 = Report or website include three or more references.4 = Actual report includes list of several references, but does not have links to any of them.5 = Actual report includes list of references, with links to one or more of them. Show that there’s a real organization behind your site. Showing that your web site is for a

legitimate organization will boost the site's credibility. The easiest way to do this is by listing a physical address. Other features can also help, such as posting a photo of your offices or listing a membership with the chamber of commerce. (0 = no; 1 = yes)

0 = No1 = Yes Highlight the expertise in your organization and in the content and services you provide. Do you

have experts on your team? Are your contributors or service providers authorities? Be sure to give their credentials. Are you affiliated with a respected organization? Make that clear. Conversely, don't link to outside sites that are not credible. Your site becomes less credible by association. (1 = bad to 5 = excellent)

1 = No names of individuals.2 = Difficult to find list of actual individuals; credentials may or may not be included.3 = Report or website includes list of HAI Advisory Committee members, but all credentials are not

included; somewhat difficult to locate.4 = Report or website includes list of HAI Advisory Committee members, but all credentials are not

included; relatively easy to locate.5 = Report or website includes list of HAI Advisory Committee members and members’ credentials;

easy to locate.

Etc.

First Draft Scoring Tool (cont.)

Credibility vs. Usability ScoresAugust 2012

What We’ve Learned About HAI Reporting

HAI Reporting is widespread but quality of report format varies widely.

Larger states do not necessarily do better than smaller states (e.g. NH vs. FL).

Rankings can change dramatically if states redesign their website (e.g. Washington)

Redesign of website or content is not always an improvement

Impact of Redesign

Next Steps: Refining the Scoring Tools

Separate “user friendliness into “usability” and “content”Does a site contain useful information vs. ease of finding contentDividing line often not clear

Scoring initially done by one graduate student; project now working on inter-rater reliability rcontent = 0.87 rcredibility = 0.98 rusability = low (2 students)Still a work in progress

Science of Public Reporting

Credibility score & User Friendliness score are positively correlatedPearson coefficient 0.58

Many states are not applying most of what already is known in the research literature.

Nor learning from each other about best practices?

We don’t know whether interactive format, PDF static report format, or offering both options is more effective to convey HAI rate information

Select Publications Birnbaum D, Cummings MJ, Guyton K, Schlotter J, Kushniruk A. Designing Public Web Information Systems

with Quality in Mind: Public Reporting of Hospital Performance Data. CLINICAL GOVERNANCE 2010;15(4):272-278.

Bell S, Benneyan J, Best A, Birnbaum D, Borycki EM, Gallagher TH, Goeschel C, Jarvis B, Kushniruk AW, Mazor KM, Pronovost P, Sheps S. Mandatory Public Reporting: Build It and Who Will Come? STUD HEALTH TECHNOL INFORM 2011;164:346-52 (from Information Technology & Communications in Health 2011 international conference).

Amini A, Birnbaum DW, Black B, Hyman DA. Public Reporting of Hospital Infection Rates: Ranking the States on Credibility and User Friendliness. STUD HEALTH TECHNOL INFORM 2013;183:87-92. DOI: 10.3233/978-1-61499-203-5-87.

Hyman DA, Black BS. Public Reporting of Hospital Infection Rates: Not All Change is Progress. Northwestern

University Law School Law and Economics Research Paper No. 12-21; Northwestern University Institute for Policy Research Working Paper 13-07; University of Illinois Law, Behavior & Social Sciences Paper No. LE 13-18 draft available from Social Science Research Network electronic library at http://ssrn.com/abstract= 2219510.

Birnbaum D. Unraveling a Web of Confusion. CLINICAL GOVERNANCE 2013;(in press)

Senior Partners in our Universities Council• Sigall Bell, MD

– Assistant Professor of Medicine, Harvard Medical School• Jim Benneyan, PhD

– Professor, Northeastern University College of Engineering; Director, Center for Health Organization Transformation; Executive Director, New England VA Healthcare Systems Engineering Partnership

• Allan Best, PhD– Senior Scientist, Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute; Clinical Professor, School of Population and

Public Health, University of British Columbia; Managing Director, InSource.• Bernard Black, MS, JD

– Chabraja Professor, Northwestern University Law School and Kellogg School of Management• David Birnbaum, PhD, MPH

– Adjunct Professor, University of British Columbia School of Population and Public Health, and UBC School of Nursing; Adjunct Professor, University of Victoria School of Health Information Science; Principal, Applied Epidemiology; Manager, Washington State Dept. of Health Healthcare Associated Infections Program.

• Elizabeth Borycki, RN, MN, PhD– Assistant Professor, University of Victoria School of Health Information Science

• Thomas H. Gallagher, MD– Associate Professor of Medicine, University of Washington

• Chris Goeschel, RN, MPA, MPS, ScD– Director of Patient Safety & Quality Initiatives, Manager of Operations, Johns Hopkins University Quality & Safety Research Group; Clinical Instructor,

Johns Hopkins School of Nursing; Associate faculty, Johns Hopkins Bloomberg School of Public Health• Bill Jarvis, MD

– Jason and Jarvis Associates• André Kushniruk, PhD

– Professor, University of Victoria School of Health Information Science• Kathy Mazor, EdD

– Associate Professor, University of Massachusetts Medical School• Peter Pronovost, MD, PhD, FCCM

– Professor, Johns Hopkins Schools of Medicine, Nursing, and Bloomberg School of Public Health; Director, Johns Hopkins Quality & Safety Research Group• Sam Sheps, MD, MSc, FRCPC(C)

– Professor, University of British Columbia School of Population and Public Health; Director, Western Regional Training Centre for Health Services Research