For the Practice Change Fellows ProgramSeptember 25, 2008

Washington, DC

Dennis A. Ehrich, MD, FACCVice President for Medical Affairs

St. Joseph’s Hospital Health CenterSyracuse, New York

The Importance of Measurement in Health Care

Agenda for the Afternoon

1-Why we measure in health care?2-The Model for Improvement3-Selecting one’s measures4-Time ordered statistics and understanding

variation 5-Displaying and tracking results6-Deciding whether To design a new process or

improve an existing process

Why We Measure in Health CareMeasuring for

ResearchMeasuring for

JudgmentMeasuring for Improvement

Purpose To discover new knowledge

To compare to others, to rank

To bring new knowledge into daily practice

Tests One large trial Public reporting quarterly or with 12

month running averages

Many small, sequential, observable tests

Bias Control for as many as possible

Severity or risk adjustment where

available

Stabilize the biases from test to test

Data Gather as much data as possible, just in

case

Measures structure, process or outcomes

Usually applied to process

Duration Can require large numbers of patients and long periods of

time to obtain results

Ongoing data collection and periodic

public reporting

Short iterative cycles in a limited number of subjects,

followed by spread

Set aims that are measurable, time-specific, and apply to a defined population

The Model for Improvement

Establish measures to determine if a specific change leads to improvement

Select changes most likely to result in improvement

Test the changes

T. Nolan et al. www.ihi.org

The Use of Iterative PDSA Cycles

Implementing the Changes

“Rapid-cycle CQI”

T. Nolan et al. www.ihi.org Multiple Simultaneous Tests of Change

Spreading the Change

1-Executive sponsorship2-Planning and set-up 3-Spread within the target population-social network theory 4-Continuous monitoring and feedback during the spread process5-Capturing and sharing organizational learning 

T. Nolan et al. www.ihi.org

Donabedian’s Quality Triangle-It’s Relevance to Process Improvement

-Avedis Donabedian, MD, MPH (1919-2000)

Donabedian’s TriadStructure

OrganizationPeopleEquipment/Technology

ProcessThe steps taken in accomplishing the change and achieving the

outcomeResults must be client-focusedMust deliver results reliably

OutcomesClinical (mortality, complications)Client perception or satisfactionFinancial

Selecting Your Measures

The Three Domains of Measurement

• Structural Measures• Process measures• Outcomes Measures

– Balancing measures

Donabedian

The Three Domains of Measurement

• Structural Measures– Describe the environment. How many?– Square footage of a clinical unit– Number of staff– Staff qualifications and competencies– Presence or absence of technology and its

characteristics• Process Measures

• Process cycle time• The percentage of patients for whom the process achieves

its desired result

Donabedian

The Three Domains of Measurement

• Outcome Measures• The impact of the change initiative on mortality,

readmissions to the hospital, ED visits• The satisfaction scores of clients and staff • The cost per case, average LOS, revenue per case

• Balancing Measures – Unintended outcomes that are consequences of the

new program– Unanticipated mortality, morbidity or cost – Has the shifting of resources in an organization

compromised other client or patient populations?Donabedian

ACTION

Aim

Selecting A Measure

Operational Definitions

Data Collection Plan

Data Collection

Data Analysis

The Quality Measurement Roadmap

Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”

Selecting a Measure:

-When selecting a measure, have clarity as to whether the measure is one of structure, process or

outcome

-And select a balanced panel of indicators that reflect the dimensions of performance being evaluated and the change concept(s) being

employed

What Dimension of Performance Do You Want to Measure?

• Appropriateness • Availability• Continuity• Effectiveness• Efficiency• Respect and caring• Financial/Viability• Safety• Time lines

Joint Commission (1996)

What Dimension of Performance do You Want to Measure?

• Safety• Effectiveness• Patient-centeredness• Timeliness• Efficiency• Equity

IOM: Crossing the Quality Chasm (2001)

What is the “Change Concept”?• Eliminate waste• Improve work flow• Shorten a waiting list• Change the work environment• Improve the Provider/Client interface• Manage time• Focus on variation• Error proofing a process• Focusing on product or service

The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass

Relating a Change Concept to a Specific Measure

Concept Potential Indicators for this processPatient scheduling •The average number of days between the call for an

appointment and the actual appointment date•The percentage of appointments made within 3 days of the call for an appointment•The number of appointments scheduled each day

Home care visits •The number of home care visits•The average time spent during a home care visit•The percentage of time spent traveling during each home care visit•The number of visits per home care nurse

CQI Training •The number of participants attending a class•The percentage of cancellations•The percentage of no-shows•The information recall scores at 30 and 60 days

Establishing Operational Definitions That Are Agreed Upon By All Stakeholders

Operational Definitions• Is clear and unambiguous• Specifies the measurement method, procedures and

equipment when appropriate– Clinical data (chart reviews) vs. administrative data– Client logs vs. a computer database

• Define specific criteria for the data to be collected– Define all inclusions and exclusions– For percentages or rates, or ratios, define the criteria

for inclusion in the numerator and denominator• Always ask “How might somebody be confused by this

definition?”

Lloyd, R. Quality Health Care (2004) Jones and Bartlett

Examples of Unclear Definitions

• Timely completion of the screening process• A complete medication list• The readmission rate• Medication error• Cost impact• From the acute care hospital

– A patient fall– Surgical start time

Lloyd, R. Quality Health Care (2004) Jones and Bartlett

Data Analysis

• How will the measurements be expressed?– Quantities, rates, ratios, proportions, percentages

• What type of statistics will be used?– Descriptive statistics

• Measures of central tendency– Mean, median, mode

• Measures of variation or spread– Minimum, maximum, range, standard deviation

– Inferential statistics• t-tests• ANOVA• Chi Square

Data Display

• Table• Bar chart• Histogram• Line chart • Pie chart• Pareto diagram• Time-ordered data

• Run chart• Control chart

Comparative Data

• Internal targets-trended data• External comparisons-benchmarking

– Best practices– National or regional population averages

External BenchmarkingJoint Commission

CMS

Calculation of the Confidence Interval

Estimates

± t * σ/ √n

Wheret= 3 (the sigma number for 99% confidence interval)

σ =The hospital’s standard error of the mean and

n = The number of patients in the hospital’s denominator

Data Reporting

• Data reporting plan– Who will receive the results?– How often will they receive the results?– How will it be formatted?

• Dashboard• Paper reports• Spider diagram

– How will the data be disseminated?• E mail• Internet• Intranet

Displaying Time-Ordered Statistics and Understanding Variation

Tools for Displaying Time-ordered Data

• Run charts– Plot of data over time with the median of the data

set plotted as a center line

• Control charts– Plot of data over time with the mean as the center

line and with upper and lower control limits

Run Charts

• Easily constructed by hand or in available spreadsheet programs

• Provides a good idea of improvement in a change initiative

• Less sensitive to significant changes (special cause variation) than the control chart

Control Charts More sensitive to special cause variation than a run

chart Requires specialized computer software to create There are 9 types of control charts used in health

care, depending upon whether the data collected is distributed normally, is continuous (numerical) or discreet (attributes) and whether the events measured are frequent or infrequent

Have their own set of rules to identify special cause variation

Understanding Variation• All data, collected over time, varies• Random variation (common cause)

– The changes occurring are intrinsic to the process being measured

• Non-random variation (special cause)– The changes are being imposed on the system by some external

factor– May be unintended and un anticipated or may be by design

• Before process improvement can be implemented, the process must be in control (free of special cause variation)

Common Cause (Random) Variation in a Run Chart

Special Cause Variation in a Run Chart

Special Cause Variation in a Control Chart

Upper Control Limit 205 mmHg

Lower Control Limit 142 mmHg

Mean 173 mmHg

Special Cause Variation 138 mm Hg

Daily record of Blood Pressure

Special Cause Variation in a Control Chart

Deciding Whether To Design A New Process or Improve An

Existing Process

Initial Considerations

• Is the process under consideration local?– Within a department– On a clinical unit

• Is the project organization wide?– A process change in a work system that impacts

the entire organization– Requires commitment of people, funds, or new

technologies

Organization-Wide Initiatives

• Must be consistent with the organization’s Mission, Vision, and Values

• Must be aligned with the organization’s strategic plan

Strategic Goals

QUALITY

PATIENTs

PEOPLE

GROWTH

MARGIN

+ + +=

Measurement and the Strategic Plan

Analyze the inputsObtain Inputs

Determine the organizational strategies for each strategic goal

Map the data sourceLocate or design the system Write the interfacesPopulate the dashboards

Determine the departmental tactics, measures, and targets

Determine HR Requirements Formulate the IT Capital Budget

Staffing requirementsGrow or PurchaseTraining requirements

Determine the organizational measures, performance Targets

and benchmarks