Overview of the Hospital Safety ScoreMarch 2013

Missy Danforth, Senior Director of Hospital Ratings, The Leapfrog Group

Presentation Overview• Why is Leapfrog issuing a Hospital Safety Score• Who is getting a Hospital Safety Score?• Important Dates• Scoring Overview –

• Measure Selection • Measure Weights• Changes to the Scoring Methodology Since November 2012

• Bound on negative z-scores• Dealing with Missing data

• Details of the data review process• Questions

2

What is the Hospital Safety Score? Why is Leapfrog issuing one?

• The Hospital Safety Score is an A, B, C, D, or F letter grade reflecting how safe hospitals are for patients. For the first time ever, this score empowers healthcare consumers to make informed decisions based on the safety of their hospital.

• Leapfrog is strategically expanding its focus to get consumers to use information on safety and quality that can save their lives.

3

What Hospitals Will Receive a Hospital Safety Score?• General, acute-care hospitals for which there is adequate

public data

• Excluded Hospitals include:• Specialty Hospitals (i.e. surgical centers, cancer hospitals,

women’s hospitals, etc.)• Critical Access Hospitals• Free –Standing Pediatric Facilities• Non-IPPS participating hospitals (hospitals from the state of MD)

• Hospitals Missing Too Much Data:• More than 9 process measures• More than 3 outcome measures

4

Important Dates• January 2013 – Expert panel re-convened to review feedback from November 2012 safety score

release and develop recommendations for the scoring methodology.

• March 2013 – Leapfrog’s board approved recommendations of expert panel.

• March 2013 - Letter sent to CEOs of hospitals receiving a Hospital Safety Score. Letter included:• Information about the Hospital Safety Score• A username/password to a secure website where hospitals can review the source data that Leapfrog

used to calculate their numerical safety score• Links to the Hospital Safety Score help desk and helpful documents

• March 20th to April 8th - Data Preview Period

• Mid April – data preview website refreshed with any changes received during data review period.

• Late April - Hospitals will be able to preview letter grades 48 hours prior to the pubic release (www.HospitalSafetyScore.org).

• For more information about important dates, visit: http://www.hospitalsafetyscore.org/for-hospitals/updates-and-timelines-for-hospitals

5

SCORING OVERVIEW 6

Measure Selection Criteria• Publicly-reported from national data sources, reflecting

individual hospital results• Leapfrog Hospital Survey• CMS’s Hospital Compare

• Endorsed or in use by a national measurement entity• Linked to patient safety (“freedom from harm”)

• Directly quantifying patient safety events• Assessing processes that lead to better outcomes• Identified by experts as important to patient safety

• Forty-five candidate measures identified; 26 recommended for inclusion in score (note PSI 11 not available for the scoring period) 7

Measures included in the Hospital Safety Score

Measure Name Primary Data Source Secondary Data Source

Process and Structural Measures (15)Computerized Physician Order Entry (CPOE) 2012 Leapfrog Hospital Survey AHA Annual Survey (2009 Technology

Supplement to the AHA Annual Survey) i

ICU Physician Staffing (IPS) 2012 Leapfrog Hospital Survey 2010 AHA Annual Survey i

Safe Practice 1: Leadership Structures and Systems 2012 Leapfrog Hospital Survey Safe Practice 2: Culture Measurement, Feedback and Intervention 2012 Leapfrog Hospital Survey Safe Practice 3: Teamwork Training and Skill Building 2012 Leapfrog Hospital Survey Safe Practice 4: Identification and Mitigation of Risks and Hazards 2012 Leapfrog Hospital Survey Safe Practice 9: Nursing Workforce 2012 Leapfrog Hospital Survey Safe Practice 17: Medication Reconciliation 2012 Leapfrog Hospital Survey Safe Practice 19: Hand Hygiene 2012 Leapfrog Hospital Survey Safe Practice 23: Care of the Ventilated Patient 2012 Leapfrog Hospital Survey SCIP INF 1: Antibiotic within 1 Hour CMS Hospital Compare SCIP INF 2: Antibiotic Selection CMS Hospital Compare SCIP INF 3: Antibiotic Discontinued After 24 Hours CMS Hospital Compare SCIP INF 9: Catheter Removal CMS Hospital Compare SCIP VTE 2: VTE Prophylaxis CMS Hospital Compare

Outcome Measures (11)Foreign Object Retained CMS Hospital Compare Air Embolism CMS Hospital Compare Pressure Ulcer – Stages 3 and 4 CMS Hospital Compare Falls and Trauma CMS Hospital Compare CLABSI Leapfrog Hospital Survey CMS Hospital ComparePSI 4: Death Among Surgical Inpatients CMS Hospital Compare PSI 6: Iatrogenic Pneumothorax CMS Hospital Compare PSI 11: Postoperative Respiratory Failure CMS Hospital Compare PSI 12: Postoperative PE/DVT CMS Hospital Compare PSI 14: Postoperative Wound Dehiscence CMS Hospital Compare PSI 15: Accidental Puncture or Laceration CMS Hospital Compare

[i] AHA Annual Survey © 2011 Health Forum, LLC

8

Weighting Process• Two (2) measure domains, each weighted 50%:

1. Process/structural measures2. Outcome measures

• Three (3) criteria for weighting individual measures• Strength of Evidence (rating of 1 or 2)• Opportunity (rating of 1, 2, 3), based on coefficient of variation• Impact (rating of 1, 2, or 3), based on no. of patients possibly

affected by the event and severity of harm to individual patients

• Weight score: [Evidence + (Opportunity x Impact)]

9

Z-Score Methodology• Standardizes data from individual measures with different scales

• Counts how many standard deviations a hospital’s score on the measure is away from the mean

• Mean always equals 0; worse than mean = negative z-score ; better than mean = positive z-score

• Translate raw score on measure to z-score:• Process/Structural Measures = [(Hospital Score – Mean)/Standard

Deviation]• Outcome Measures = [(Mean – Hospital Score)/Standard Deviation] 10

Overall Score• Summation of z-score for each measure × weight for each

measure

3.0 + CPOE z-score × CPOE weight + IPS z-score x IPS weight + CLABSI z-score × CLABSI weight . . . . etc.

• If measure has missing data, then weight for that measure is re-apportioned to other measures within the same domain

• 3.0 was added to each hospital’s final score to avoid possible confusion with interpreting negative patient safety scores

11

CHANGES TO THE SCORING METHODOLOGY SINCENOVEMBER 2012

12

Bounds on Z-scores• Issue: For those measures in which hospitals are clustered at the

‘best performing’ end of the scale (e.g., rare outcomes; some process measures), hospitals that do not perform well on the measure can have a substantial negative z-score (i.e. -7 to -20). Even though each individual measure carries a relatively low weight, this substantial negative z-score on one measure can have more influence on the hospital’s overall safety score than was intended.

• Change to scoring methodology: In an effort to reflect the actual variation in hospital performance, but to minimize the possibility of a single measure overwhelming a hospital’s overall score, we are implementing a lower bound on z-scores. Based on analysis of the November 2012 safety score data, we are setting the lower bound for z-scores at -5.0.

13

Small Denominators• Issue: Minimum reporting requirements vary among data sources and

measures. For measures that have a publicly reported numerator and denominator, Leapfrog will establish reporting requirements that meet Leapfrog’s standard for scientific acceptability.

• Change to scoring methodology: When a measure’s denominator is publicly available, Leapfrog applies the following minimum reporting requirement for using the measure in the safety score: the number of cases in the denominator must be >= 30.

• This minimum reporting requirement was identified from the literature, which suggests that thirty cases is generally the point when a non-normal distribution begins to approximate a normal distribution, which is important given the Safety Score’s use of z-scores for standardizing data across disparate data sets. The minimum sample size of 30 has also been used by other organizations that are engaged in evidence-based public reporting of health care performance data.

14

DETAILS OF THE DATA REVIEW PROCESS

15

Secure Website• http://

www.leapfroggroup.org/data-validation/validation-login

Details on what public reports were used to obtain source data. Links to source data with instructions.

Hospitals must confirm their Medicare Provider Number before moving on to the next page.

16

Source Data

Instructions for hospital to review their source data and a link to the help desk

For each measure, hospitals are provided with the measure name, type of measure, data source (primary or secondary), reporting period, measure score (i.e. points, rate, SIR, etc)

Hospitals are asked to confirm that each Measure Score matches their score (i.e. rate, SIR, etc) from the Data Source (i.e. Leapfrog Hospital Survey Results, CMS Hospital Compare, etc) 17

Where to Locate Source Data – Hospitals That Submitted a Leapfrog Hospital Survey by February 28, 2013

• CPOE: Visit www.leapfroggroup.org/cp, find the column labeled “prevent medication errors.” Leapfrog score used to determine points towards safety score (i.e. 4 bars equals 100 points, 3 bars equals 50 points, etc).

• ICU Physician Staffing: Visit www.leapfroggroup.org/cp, find the column labeled “Appropriate ICU Staffing.” Leapfrog score used to determine points towards safety score.

• CLABSI Standardized Infection Ratio (SIR): Visit www.leapfroggroup.org/cp, find the column labeled “prevent ICU infections.” The SIR was used to calculate in the safety score calculation, and is located in the first sentence in the first paragraph above the table.

18

Where to find your hospital's CLABSI SIR on Leapfrog’s results website

19

Where to Locate Source Data – Hospitals That Did Not Submit a Leapfrog Hospital Survey by February 28, 2013

• CPOE: Leapfrog used information you provided to the American Hospital Association through the 2009 AHA Technology Supplement.

• ICU Physician Staffing: Leapfrog used information you provided to the American Hospital Association through the 2011 AHA Annual Survey.

• CLABSI Standardized Infection Ratio (SIR): Leapfrog used information published by CMS. Visit www.data.medicare.gov. Select the “Healthcare Associated Infections” report and search for hospital by name or MPN (do not type in the hyphen sometimes included with the MPN).

20

Where to find your hospital’s CLABSI SIR on CMS’ results website

21

22

Where to Locate Source Data – CMS Data for All Hospitals

• SCIP Measures: Hospital rates are published by CMS at https://data.medicare.gov/dataset/Hospital-Process-of-Care-Measures-Surgical-Care-Im/iz5u-4mk2

• HACs: Hospital rates are published by CMS at https://data.medicare.gov/dataset/Hospital-Acquired-Condition-Measures/qd2y-qcgs

• PSIs: Hospital rates are published by CMS at https://data.medicare.gov/dataset/Agency-For-Healthcare-Research-And-Quality-Measure/vs3q-rxc5

• You can download your hospital’s results into a CSV or XLS formatted file.

23

What if the Measure Score doesn’t match the public report?

• Hospitals are asked to contact the help desk immediately once they have confirmed the measure and reporting period.

• Hospitals must provide a copy of the public report that shows a different score

• If we find a recording error, we will update the score and re-issue a numerical safety score

24

Preview Numerical Safety Score

25

More Information• Hospital Safety Score Help Desk -

scorehelp@leapfroggroup.org

• Hospital Safety Score Website – www.HospitalSafetyScore.org

• Data Review Website - http://www.leapfroggroup.org/data-validation/validation-login

•

26