For the Practice Change Fellows ProgramSeptember 24, 2009

Washington, DC

Dennis A. Ehrich, MD, FACCVice President for Medical Affairs

St. Joseph’s Hospital Health CenterSyracuse, New York

The Importance of Measurement in Health Care

Agenda for the Afternoon

1-When do we measure in health care2-Project alignment and design3-Understanding and selecting measures5-Operational definitions, change concepts, and data

collection6-Data analysis, comparative data, benchmarks and

data reporting7-Time ordered statistics and understanding variation:

the value and the pitfalls

Measurement in Health CareMeasuring for

ResearchMeasuring for

JudgmentMeasuring for Improvement

Purpose To discover new knowledge

To compare to others, to rank

To bring new knowledge into daily practice

Tests One large trial to answer a few

questions

Submission of quarterly aggregated

data for public reporting.

12 month running averages

Many small, sequential, observable tests

Bias Control for as much as possible

Severity or risk adjustment where

available

Stabilize the biases from test to test

Data Gather as much data as possible, just in

case

Measures structure, process or outcomes

Usually applied to process

Duration Requires large numbers of patients/ long periods of time

to obtain results

Ongoing data collection and periodic

public reporting

Short iterative cycles in a limited number of subjects,

followed by spread

Project Design: Selection and Alignment

• Does your project align with– The mission and values of your organization?– Your organization’s vision and strategic plan?

• Do you have support of your leadership?• Are all of the stakeholders involved in the

design process?

Selecting Your Measures

Donabedian’s Quality Triangle-It’s Relevance to Process Improvement

-Avedis Donabedian, MD, MPH (1919-2000)

Donabedian’s TriadStructure

OrganizationPeopleEquipment/Technology

ProcessThe steps taken in accomplishing the change and achieving the

outcomeResults must be client-focusedMust deliver results reliably

OutcomesClinical (mortality, complications)Client perception or satisfactionFinancial

The Three Domains of Measurement

• Structural Measures• Process measures• Outcomes Measures– Balancing measures

Donabedian

The Three Domains of Measurement

• Structural Measures– Describe the environment. How many?– Square footage of a clinical unit– Number of staff– Staff qualifications and competencies– Presence or absence of technology and its

characteristics• Process Measures

• Process step cycle time• Number of defects at key steps of the process

Donabedian

The Three Domains of Measurement

• Outcome Measures• The impact of the change initiative on mortality,

readmissions to the hospital, ED visits, etc.• The satisfaction scores of clients and staff • The cost per case, average LOS, revenue per case

• Balancing Measures – Unintended outcomes that are consequences of the

new program– Unanticipated mortality, morbidity or cost – Has the shifting of resources in an organization

compromised other client or patient populations?Donabedian

The Quality Measurement Roadmap

Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”

Defining the Project (Process)

Selecting Measures

Operational Definitions/Change Concepts

Data Collection Plan

Data Collection

Data Analysis

Defining targets and Benchmarks

Data Reporting

Establish Operational Definitions Agreed Upon By All Stakeholders• Are clear and unambiguous• Specify the measurement method, procedures and

equipment when appropriate– Clinical data (chart reviews) vs. administrative data– Client logs vs. a computer database

• Define specific criteria for the data to be collected– Define all inclusions and exclusions– For percentages or rates, or ratios, define the criteria

for inclusion in the numerator and denominator• Always ask “How might somebody be confused by this

definition?”

Lloyd, R. Quality Health Care (2004) Jones and Bartlett

Examples of Unclear Definitions

• Timely completion of the screening process• A complete medication list• The readmission rate• Medication error• Cost impact• From the acute care hospital– A patient fall– Surgical start time

Lloyd, R. Quality Health Care (2004) Jones and Bartlett

What Dimension of Performance Will You Affect?

• Appropriateness • Availability• Continuity• Effectiveness• Efficiency• Respect and caring• Financial/Viability• Safety• Time lines

Joint Commission (1996)

Which IOM Domain of Quality Will be Affected?

• Safety• Effectiveness• Patient-centeredness• Timeliness• Efficiency• Equity

IOM: Crossing the Quality Chasm (2001)

What is the “Change Concept”?• Eliminate waste• Improve work flow• Shorten a waiting list• Change the work environment• Improve the Provider/Client interface• Manage time• Focus on variation• Error proofing a process• Focusing on product or service

The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass

The Data Collection Plan and Data Collection

Do You Have Baseline Data?

• Does your project, introduce a new process to your organization?

• Does it improve an existing process?– There is probably some existing data in your

organization?– May not be useful, if its collection was flawed or it

was collected for a purpose unrelated to your work

Collecting Data

• What data will be collected and how often?– Outcome data (monthly or quarterly)?– Process data (daily or weekly)?

• On what patients and how often will data be collected?– Frequent small samples are preferable– Beware of strata

• weekends vs. weekdays, shifts, or different teams of people

• How will the data be collected?– From an existing computer database– Use of a tool such as a “traveler” check-sheet?

Comparative Data and Targets

• Sources of comparative data– Internal targets-trended data– External comparisons– National or regional population averages– Best practice (top decile performance)

• Benchmarking– Outcome benchmarking– Process benchmarking

• Don’t set arbitrary or unattainable goals

Data Reporting

• A data reporting plan– Who will receive the results?– How often will they receive the results?– How will it be formatted?

• Tabular data• Graphics• Spider diagram

– How will the data be disseminated?• Snail mail/Fax/E-mail• Internet• Intranet Online Dashboard

Data Analysis

• Is the outcome (output) of the process improving?• To improve the output, we will need to reduce the variation in

the process inputs– Evaluate the output relative to the inputs

• The inputs – People-Are particular individuals or teams performing tasks in a non-

standardized fashion?– Machines-If body weight is being measured, are all scales accurate?– Methods-Is there variation in the way a procedure is being done?– Measurements-If a cycle time is being measured, is there a standard clock

being used or are data collectors using their individual watches?– Materials-Are all patients receiving the full compliment of brochures?– Environment-Is there a resulting from different time of day, season?

Defining your project in process terms

Cycles of Learning: The IHI Model for Improvement

Establishing Measures

Defining the change concept

Testing and perfecting changes on a small scale using repetitive cycles of learning

T. Nolan et al. www.ihi.org

The Use of Iterative PDSA Cycles

Learning and ImplementingChanges

“Rapid-cycle CQI”

T. Nolan et al. www.ihi.org Multiple Simultaneous Tests of Change

Spreading the Change

1-Executive sponsorship2-Planning and set-up 3-Spread within the target population-social network theory 4-Continuous monitoring and feedback during the spread process5-Capturing and sharing organizational learning 

T. Nolan et al. www.ihi.org

Data Reporting Tools

• Would recommend tracking the data in a time-ordered fashion– Run chart– Control chart

• In preference to using– Tabular data– Bar charts– Histograms– Pie charts

Tools for Displaying Time-ordered Data

• Run charts– Plot of data over time– The centerline of the chart is the median of the

data

• Control charts– The centerline of the chart is the mean– Also has upper and lower control limits calculated

from the data– Difficult, but not impossible to construct by hand

The Pitfalls of Aggregated Data

Annual Aggregated Data Before and After Implementation of Process Change: The Change Appears to have improved outcomes

The Pitfalls of Using Aggregated Data

Time-Ordered Data: Do you still feel the same way?

Pitfalls of Aggregate Data

A trend is a series of seven consecutive data points rising or falling. Data points on the centerline are not counted for the trend, but don’t interrupt the trend.

Sample Run Chart

Time Period

Valu

e

Examples of a Trend

Understanding Variation• All data, collected over time has variation• There are two kinds of variation

• Random variation (common cause)– The changes occurring are intrinsic to the process being measured and

is caused by variation in the process inputs• Non-random variation (special cause)

– The changes are being imposed on the system by some external factor– May be unintended and un anticipated or may be by design and a

reflection of your intervention• There are sets of rules for formally diagnosing special-cause

variation on run charts and control charts

A Run Chart With Common Cause (Random) Variation

A Control Chart with Common Cause Variation

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

A Control Chart With Special Cause Variation

A Control Chart with (Deliberate) Special Cause Variation

This change is known as a “process shift”

Let’s Put Some of These Ideas to Work