![Page 1: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/1.jpg)
For the Practice Change Fellows ProgramSeptember 24, 2009
Washington, DC
Dennis A. Ehrich, MD, FACCVice President for Medical Affairs
St. Joseph’s Hospital Health CenterSyracuse, New York
The Importance of Measurement in Health Care
![Page 2: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/2.jpg)
Agenda for the Afternoon
1-When do we measure in health care2-Project alignment and design3-Understanding and selecting measures5-Operational definitions, change concepts, and data
collection6-Data analysis, comparative data, benchmarks and
data reporting7-Time ordered statistics and understanding variation:
the value and the pitfalls
![Page 3: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/3.jpg)
Measurement in Health CareMeasuring for
ResearchMeasuring for
JudgmentMeasuring for Improvement
Purpose To discover new knowledge
To compare to others, to rank
To bring new knowledge into daily practice
Tests One large trial to answer a few
questions
Submission of quarterly aggregated
data for public reporting.
12 month running averages
Many small, sequential, observable tests
Bias Control for as much as possible
Severity or risk adjustment where
available
Stabilize the biases from test to test
Data Gather as much data as possible, just in
case
Measures structure, process or outcomes
Usually applied to process
Duration Requires large numbers of patients/ long periods of time
to obtain results
Ongoing data collection and periodic
public reporting
Short iterative cycles in a limited number of subjects,
followed by spread
![Page 4: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/4.jpg)
Project Design: Selection and Alignment
• Does your project align with– The mission and values of your organization?– Your organization’s vision and strategic plan?
• Do you have support of your leadership?• Are all of the stakeholders involved in the
design process?
![Page 5: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/5.jpg)
Selecting Your Measures
![Page 6: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/6.jpg)
Donabedian’s Quality Triangle-It’s Relevance to Process Improvement
-Avedis Donabedian, MD, MPH (1919-2000)
![Page 7: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/7.jpg)
Donabedian’s TriadStructure
OrganizationPeopleEquipment/Technology
ProcessThe steps taken in accomplishing the change and achieving the
outcomeResults must be client-focusedMust deliver results reliably
OutcomesClinical (mortality, complications)Client perception or satisfactionFinancial
![Page 8: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/8.jpg)
The Three Domains of Measurement
• Structural Measures• Process measures• Outcomes Measures– Balancing measures
Donabedian
![Page 9: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/9.jpg)
The Three Domains of Measurement
• Structural Measures– Describe the environment. How many?– Square footage of a clinical unit– Number of staff– Staff qualifications and competencies– Presence or absence of technology and its
characteristics• Process Measures
• Process step cycle time• Number of defects at key steps of the process
Donabedian
![Page 10: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/10.jpg)
The Three Domains of Measurement
• Outcome Measures• The impact of the change initiative on mortality,
readmissions to the hospital, ED visits, etc.• The satisfaction scores of clients and staff • The cost per case, average LOS, revenue per case
• Balancing Measures – Unintended outcomes that are consequences of the
new program– Unanticipated mortality, morbidity or cost – Has the shifting of resources in an organization
compromised other client or patient populations?Donabedian
![Page 11: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/11.jpg)
The Quality Measurement Roadmap
Modified from Lloyd, Robert: “Quality Health Care A Guide to Using Indicators”
Defining the Project (Process)
Selecting Measures
Operational Definitions/Change Concepts
Data Collection Plan
Data Collection
Data Analysis
Defining targets and Benchmarks
Data Reporting
![Page 12: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/12.jpg)
Establish Operational Definitions Agreed Upon By All Stakeholders• Are clear and unambiguous• Specify the measurement method, procedures and
equipment when appropriate– Clinical data (chart reviews) vs. administrative data– Client logs vs. a computer database
• Define specific criteria for the data to be collected– Define all inclusions and exclusions– For percentages or rates, or ratios, define the criteria
for inclusion in the numerator and denominator• Always ask “How might somebody be confused by this
definition?”
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
![Page 13: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/13.jpg)
Examples of Unclear Definitions
• Timely completion of the screening process• A complete medication list• The readmission rate• Medication error• Cost impact• From the acute care hospital– A patient fall– Surgical start time
Lloyd, R. Quality Health Care (2004) Jones and Bartlett
![Page 14: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/14.jpg)
What Dimension of Performance Will You Affect?
• Appropriateness • Availability• Continuity• Effectiveness• Efficiency• Respect and caring• Financial/Viability• Safety• Time lines
Joint Commission (1996)
![Page 15: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/15.jpg)
Which IOM Domain of Quality Will be Affected?
• Safety• Effectiveness• Patient-centeredness• Timeliness• Efficiency• Equity
IOM: Crossing the Quality Chasm (2001)
![Page 16: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/16.jpg)
What is the “Change Concept”?• Eliminate waste• Improve work flow• Shorten a waiting list• Change the work environment• Improve the Provider/Client interface• Manage time• Focus on variation• Error proofing a process• Focusing on product or service
The Improvement Guide by Langley, Nolan, Nolan, Norman and Provost. Jossey-Bass
![Page 17: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/17.jpg)
The Data Collection Plan and Data Collection
![Page 18: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/18.jpg)
Do You Have Baseline Data?
• Does your project, introduce a new process to your organization?
• Does it improve an existing process?– There is probably some existing data in your
organization?– May not be useful, if its collection was flawed or it
was collected for a purpose unrelated to your work
![Page 19: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/19.jpg)
Collecting Data
• What data will be collected and how often?– Outcome data (monthly or quarterly)?– Process data (daily or weekly)?
• On what patients and how often will data be collected?– Frequent small samples are preferable– Beware of strata
• weekends vs. weekdays, shifts, or different teams of people
• How will the data be collected?– From an existing computer database– Use of a tool such as a “traveler” check-sheet?
![Page 20: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/20.jpg)
Comparative Data and Targets
• Sources of comparative data– Internal targets-trended data– External comparisons– National or regional population averages– Best practice (top decile performance)
• Benchmarking– Outcome benchmarking– Process benchmarking
• Don’t set arbitrary or unattainable goals
![Page 21: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/21.jpg)
Data Reporting
• A data reporting plan– Who will receive the results?– How often will they receive the results?– How will it be formatted?
• Tabular data• Graphics• Spider diagram
– How will the data be disseminated?• Snail mail/Fax/E-mail• Internet• Intranet Online Dashboard
![Page 22: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/22.jpg)
Data Analysis
• Is the outcome (output) of the process improving?• To improve the output, we will need to reduce the variation in
the process inputs– Evaluate the output relative to the inputs
• The inputs – People-Are particular individuals or teams performing tasks in a non-
standardized fashion?– Machines-If body weight is being measured, are all scales accurate?– Methods-Is there variation in the way a procedure is being done?– Measurements-If a cycle time is being measured, is there a standard clock
being used or are data collectors using their individual watches?– Materials-Are all patients receiving the full compliment of brochures?– Environment-Is there a resulting from different time of day, season?
![Page 23: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/23.jpg)
Defining your project in process terms
Cycles of Learning: The IHI Model for Improvement
Establishing Measures
Defining the change concept
Testing and perfecting changes on a small scale using repetitive cycles of learning
T. Nolan et al. www.ihi.org
![Page 24: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/24.jpg)
The Use of Iterative PDSA Cycles
Learning and ImplementingChanges
“Rapid-cycle CQI”
T. Nolan et al. www.ihi.org Multiple Simultaneous Tests of Change
![Page 25: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/25.jpg)
Spreading the Change
1-Executive sponsorship2-Planning and set-up 3-Spread within the target population-social network theory 4-Continuous monitoring and feedback during the spread process5-Capturing and sharing organizational learning
T. Nolan et al. www.ihi.org
![Page 26: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/26.jpg)
Data Reporting Tools
• Would recommend tracking the data in a time-ordered fashion– Run chart– Control chart
• In preference to using– Tabular data– Bar charts– Histograms– Pie charts
![Page 27: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/27.jpg)
Tools for Displaying Time-ordered Data
• Run charts– Plot of data over time– The centerline of the chart is the median of the
data
• Control charts– The centerline of the chart is the mean– Also has upper and lower control limits calculated
from the data– Difficult, but not impossible to construct by hand
![Page 28: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/28.jpg)
The Pitfalls of Aggregated Data
Annual Aggregated Data Before and After Implementation of Process Change: The Change Appears to have improved outcomes
![Page 29: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/29.jpg)
The Pitfalls of Using Aggregated Data
Time-Ordered Data: Do you still feel the same way?
![Page 30: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/30.jpg)
Pitfalls of Aggregate Data
A trend is a series of seven consecutive data points rising or falling. Data points on the centerline are not counted for the trend, but don’t interrupt the trend.
![Page 31: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/31.jpg)
Sample Run Chart
Time Period
Valu
e
Examples of a Trend
![Page 32: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/32.jpg)
Understanding Variation• All data, collected over time has variation• There are two kinds of variation
• Random variation (common cause)– The changes occurring are intrinsic to the process being measured and
is caused by variation in the process inputs• Non-random variation (special cause)
– The changes are being imposed on the system by some external factor– May be unintended and un anticipated or may be by design and a
reflection of your intervention• There are sets of rules for formally diagnosing special-cause
variation on run charts and control charts
![Page 33: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/33.jpg)
A Run Chart With Common Cause (Random) Variation
![Page 34: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/34.jpg)
A Control Chart with Common Cause Variation
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
![Page 35: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/35.jpg)
A Control Chart With Special Cause Variation
![Page 36: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/36.jpg)
A Control Chart with (Deliberate) Special Cause Variation
This change is known as a “process shift”
![Page 37: For the Practice Change Fellows Program September 24, 2009 Washington, DC Dennis A. Ehrich, MD, FACC Vice President for Medical Affairs St. Joseph’s Hospital](https://reader036.vdocument.in/reader036/viewer/2022062519/5697bfa01a28abf838c9575c/html5/thumbnails/37.jpg)
Let’s Put Some of These Ideas to Work