hamad medical corporation launch event navigating the quality...
TRANSCRIPT
Navigating theQuality Measurement Journey
2 October 2013
Prepared and presented byRobert Lloyd, PhDExecutive Director Performance Improvement
Hamad Medical Corporation Launch Event
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
The Improvement Guide, API, 2009
The Model for Learning and Change
When you combine
the 3 questions with the…
…the Model
for Improvement.
PDSA cycle, you get…
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd3
“You can’t fatten a cow by weighing it”- Palestinian Proverb
Improvement is NOT just about measurement!
However, without measurement you will never be able to connect the dots and
answer question #2 in the MFI.
The Role of Measurement
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
• The purpose of measurement in QI work is for learning not
judgment!
• All measurement has limitations, but the limitations do not negate
its value for learning.
• Build a balanced set of measures that reflect the VOC and VOP.
• All measurement should be linked to the team’s Aim.
• Measurement should be used to guide improvement and test
changes.
• Measurement should be integrated into the team’s daily routine.
• Data should be plotted over time on annotate graphs.
• Focus on the Vital Few!
Measurement is Central to the Team’s Ability to Improve
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd5
1. By understanding the variation that lives within your data
2. By making good management decisions about this variation (i.e.,
don’t overreact to a special cause and don’t think that random movement of your data up
and down is a signal of improvement).
How will we know that achange is an improvement?
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
The Role of Measurement: Connect the Dots!
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
What is this? (dots 1 – 25)
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Any idea? (dots 1 – 50)
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
How about now? (dots 1 – 100)
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Ahhh…now I see it! (dots 1 – 150)
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
It was so obvious wasn’t it?
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
So, where do we start to connect the dots?
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Improvement?(improving the effectiveness or
efficiency of a process)
Accountability Judgment?(making comparisons;
no change focus)
Research?(testing theory and building
new knowledge)
Why are you measuring?
©Copyright 2010 Institute for Healthcare Improvement/R. Lloyd
The Three Faces ofPerformance Measurement
Aspect Improvement Accountability Research
Aim Improvement of care
(efficiency & effectiveness)
Comparison, choice,
reassurance, motivation
for change
New knowledge
(efficacy)
Methods:
• Test ObservabilityTest observable
No test, evaluate current
performance Test blinded or controlled
• Bias Accept consistent bias Measure and adjust to
reduce bias
Design to eliminate bias
• Sample Size “Just enough” data, small
sequential samples
Obtain 100% of available,
relevant data
“Just in case” data
• Flexibility of
Hypothesis
Flexible hypotheses,
changes as learning takes
place
No hypothesis
Fixed hypothesis
(null hypothesis)
• Testing Strategy Sequential tests No tests One large test
• Determining if achange is animprovement
Analytic Statistics
(statistical process control) Run & Control
charts
No change focus
(maybe compute a
percent change or rank
order the results)
Enumerative Statistics(t-test, F-test,
chi square,
p-values)
• Confidentiality ofthe data
Data used only by those
involved with improvement
Data available for public
consumption and review
Research subjects’
identities protected
Adapted from: Lief Solberg, Gordon Mosser and Sharon McDonald,Journal
on Quality Improvement vol. 23, no. 3, (March 1997), 135-147.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
0
0.1
0.2
0.3
0.4
0.5
0.6J
an
-01
Fe
b-0
1
Ma
r-0
1
Ap
r-0
1
Ma
y-0
1
Ju
n-0
1
Ju
l-0
1
Au
g-0
1
Se
p-0
1
Oc
t-0
1
No
v-0
1
De
c-0
1
Ja
n-0
2
Fe
b-0
2
Ma
r-0
2
Ap
r-0
2
Ma
y-0
2
Ju
n-0
2
Ju
l-0
2
Au
g-0
2
Se
p-0
2
Oc
t-0
2
No
v-0
2
De
c-0
2
Pro
po
rtio
n
Control Chart (p-chart)Percent Primary Cesarean Section
Data for
Improvement
0
0.1
0.2
0.3
0.4
0.5
0.6
Pro
port
ion
Data for Judgment
These data points are all common cause (random) variation
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
So, how do you view the Three Faces of Performance Measurement?
Or,
As… As a…
Imp
rovem
en
t
Ju
dg
men
t
Researc
h
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Relating the Three Faces ofPerformance Measurement to your work
The three faces of performance measurement should not be seen as mutually exclusive silos. This is not an either/or situation.
All three areas must be understood as a system. Individuals need to build skills in all three areas.
Organizations need translators who and be able to speak the language of each approach.
The problem is that individuals identify with one of the approaches and dismiss the value of the other two.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd18
Even with this thing, I have no
idea where we’re headed!Do you have a plan to
guide your quality measurement
journey?
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd19
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
AIM (Why are you measuring?)
Concept
Measures
Operational Definitions
Data Collection Plan
Data Collection
Analysis
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using
Indicators. Jones and Bartlett, 2004.
ACTION
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Moving from a Concept to a Measure
“Hmmmm…how do I move from a concept
to an actual measure?
Every concept can have MANY
measures.
Which one is most appropriate?
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd22
Concept Potential Measures
Hand Hygiene Ounces of hand gel used each day
Ounces of gel used per staff
Percent of staff washing their hands (before & after visiting a patient)
Medication Errors Percent of errors
Number of errors
Medication error rate
VAPs Percent of patients with a VAP
Number of VAPs in a month
The number of days without a VAP
Every concept can have many measuresSource: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
• Outcome Measures: Voice of the customer or patient. How is the system performing? What is the result?
• Process Measures: Voice of the workings of the system. Are the parts/steps in the system performing as planned?
• Balancing Measures: Looking at a system from different directions/dimensions. What happened to the system as we improved the outcome and process measures (e.g. unanticipated consequences, other factors influencing outcome)?
Three Types of Measures
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Balancing Measures: Paying Attention to Unintended Consequences
• Outcome (quality, time)
• Transaction (volume, number of patients)
• Productivity (cycle time, efficiency, utilisation, flow,
capacity, demand)
• Financial (money, staff hours, materials)
• Appropriateness (validity, usefulness)
• Patient satisfaction
• Staff satisfaction
©Copyright 2010 Institute for Healthcare Improvement/R. Lloyd
Potential Set of Measures for Improvement in the ED
Topic
Outcome Measures
Process Measures
Balancing Measures
Improve waiting time and patient satisfaction in the ED
Total Length of Stay in the ED
Patient Satisfaction Scores
Time to registration
Patient / staff comments on flow
% patient receiving discharge materials
Availability of antibiotics
Volumes
% Leaving without being seen
Staff satisfaction
Financials
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
AIM (Why are you measuring?)
Concept
Measures
Operational Definitions
Data Collection Plan
Data Collection
Analysis
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using
Indicators. Jones and Bartlett, 2004.
ACTION
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd27
An Operational Definition...
… is a description, in quantifiable terms, of what to measure and the steps to follow to measure it consistently.
• It gives communicable meaning to a concept
• Is clear and unambiguous
• Specifies measurement methods and equipment
• Identifies criteria
Source: R. Lloyd. Quality Health Care: A Guide to Developing and
Using Indicators. Jones and Bartlett Publishers, 2004.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
What is a goal?The whole ball or half the ball?
?
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
How do you define the following healthcare concepts?
• World Class Performance
• Unplanned readmissions
• Teenage pregnancy
• Cancer waiting times
• Health inequalities
• Asthma admissions
• Childhood obesity
• Patient education
• Health and wellbeing
• Adding life to years and years to life
• Children's palliative care
• Safe services
• Smoking cessation
• Urgent care
• Delayed discharges
• End of life care
• Falls (with/without injuries)
• Childhood immunizations
• Complete maternity service
• Patient engagement
• Moving services closer to home
• Successful breastfeeding
• Ambulatory care
• Access to health in deprived areas
• Diagnostics in the community
• Productive community services
• Vascular inequalities
• Breakthrough priorities
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
The Good News!We already have measures and definitions
30
Outcome Measures Data Collection Guidance
VAP rate
(CCO1)
• Numerator: The total
number of ventilator
acquired pneumonia
episodes in the month
• Denominator: The total
ventilator days in the month
• The VAP rate is calculated
by dividing the total number
of VAPs occurring in the
month (the numerator) by
the total number of
ventilator days in the month
(the denominator) and then
multiplying the result by
1000 to create a VAP rate
per 1000 ventilator days
• Report monthly infection rate for the months of
October – December 2013. This serves as your
baseline. Continue to report monthly data over the life
of the Campaign into the Extranet. Provide
numerators and denominators when entering the data.
The annotation section should be used to reflect any
interventions that were made to reduce the VAP rate.
• There should be no sampling for this measure. If an
infection control practitioner reports data quarterly,
please disaggregate and report the VAP data by
month.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
You will need to develop measures specific to the
change concepts and ideas you plan to test in your
workstreams.
31
But, you will need to develop additional measures!
©Copyright 2010 IHI Improvement Advisor Professional Development Program
• Use the Driver Diagram for your workstream as a reference and guide to build measures.
• Make sure you identify a balanced set of measures (outcome, process and balancing measures).
• Use the Measurement Plan Worksheet to record your work
• Then select several of your identified measures and develop a clear operational definition for each measure.
• Use the Operational Definition Worksheet© to record your work.
• The Questions for Building Operational Definitions© will provide guidance on the specific issues which need to be addressed if you want to develop clear and concise operational definitions.
Developing a Set of Measures andOperational Definitions (prep for LS1)
©Copyright 2010 IHI Improvement Advisor Professional Development Program
Measure NameType
(Process, Outcome or Balancing)
Driver addressed bythis measure
1.
2.
3.
4.
5.
6.
7.
8.
9.
Measurement Plan Worksheet
Source: R. Lloyd 2013
©Copyright 2010 IHI Improvement Advisor Professional Development Program
Measure Name(Be sure to indicate if it
is a count, percent, rate, days between, etc.)
Operational Definition(Define the measure in very specific terms.
Provide the numerator and the denominator if a percentage or rate.
Be as clear and unambiguous as possible)
Data Collection Plan(How will the data be collected?
Who will do it? Frequency? Duration? What is to be excluded?)
Operational Definition Worksheet
Name of team:______________________________ Date:__________
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
©Copyright 2010 IHI Improvement Advisor Professional Development Program
Example of a Complete Operational Definition
Measure Name: Percent of inpatient medication orders with an error
Numerator: Number of inpatient medication orders with one or more errors
Denominator: Number of inpatient medication orders received by the pharmacy
Data Collection Plan:
• This measure applies to all inpatient units
• The data will be stratified by shift and by type of order (stat versus routine)
• The data will be tracked daily and grouped by week
• The data will be pulled from the pharmacy computer system
• Initially all medication orders will be reviewed. A stratified proportional random
sample will be considered once the variation in the process is fully understood and the volume of orders is analyzed.
©Copyright 2010 IHI Improvement Advisor Professional Development Program
Team name: _____________________________________________________________________________
Date: __________________ Contact person: ____________________________________
WHAT PROCESS DID YOU SELECT?
WHAT SPECIFIC MEASURE DID YOU SELECT FOR THIS PROCESS?
OPERATIONAL DEFINITIONDefine the specific components of this measure. Specify the numerator and denominator if it is a percent or a rate. If it is an average, identify the calculation for deriving the average. Include any special equipment needed to capture the data. If it is a score (such as a patient satisfaction score) describe how the score is derived. When a measure reflects concepts such as accuracy, complete, timely, or an error, describe the criteria to be used to determine “accuracy.”
Questions for Building Operational Definitions©Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
©Copyright 2010 IHI Improvement Advisor Professional Development Program
DATA COLLECTION PLANWho is responsible for actually collecting the data?How often will the data be collected? (e.g., hourly, daily, weekly or monthly?)What are the data sources (be specific)?What is to be included or excluded (e.g., only inpatients are to be included in this measure or only stat lab requests should be tracked).How will these data be collected?Manually ______ From a log ______ From an automated systemAre these data:
Attributes data? ______ or Variables data? ______
BASELINE MEASUREMENTWhat is the actual baseline number? ______________________________________________What time period was used to collect the baseline? ___________________________________
TARGET(S) OR GOAL(S) FOR THIS MEASUREDo you have target(s) or goal(s) for this measure?Yes ___ No ___
Specify the External target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)
Specify the Internal target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)
Questions for Building Operational Definitions© (continued)
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
©Copyright 2010 IHI Improvement Advisor Professional Development Program
DATA COLLECTION PLANWho is responsible for actually collecting the data?How often will the data be collected? (e.g., hourly, daily, weekly or monthly?)What are the data sources (be specific)?What is to be included or excluded (e.g., only inpatients are to be included in this measure or only stat lab requests should be tracked).How will these data be collected?Manually ______ From a log ______ From an automated system
BASELINE MEASUREMENTWhat is the actual baseline number? ______________________________________________What time period was used to collect the baseline? ___________________________________
TARGET(S) OR GOAL(S) FOR THIS MEASUREDo you have target(s) or goal(s) for this measure?Yes ___ No ___
Specify the External target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)
Specify the Internal target(s) or Goal(s) (specify the number, rate or volume, etc., as well as the source of the target/goal.)
Questions for Building Operational Definitions© (continued)
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett, 2004.
©Copyright 2010 IHI Improvement Advisor Professional Development Program
AIM (Why are you measuring?)
Concept
Measures
Operational Definitions
Data Collection Plan
Data Collection
Analysis
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using
Indicators. Jones and Bartlett, 2004.
ACTION
We will address data collection topics in the Foundations Program
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd40
Now that you have selected and defined your measures, it is time to head out, cast your net and actually gather some data!
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd41
Key Data Collection Strategies
Stratification• Separation & classification
of data according to predetermined categories
• Designed to discover patterns in the data
• For example, are there differences by shift, time of day, day of week, severity of patients, age, gender or type of procedure?
• Consider stratification BEFORE you collect the data
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Non-probability Sampling Methods
• Convenience sampling
• Quota sampling
• Judgment sampling
Probability Sampling Methods
• Simple random sampling
• Stratified random sampling
• Stratified proportional random sampling
• Systematic sampling
• Cluster sampling
Sampling MethodsSource: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd43
Sampling Options
Simple Random Sampling
Population Sample
Stratified proportional Random Sampling
Population Sample
Medical Surgical OB Peds
Judgment Sampling
Jan March April May JuneFeb
S S M PM M MOB OB S
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
44
Judgment Sampling
• Include a wide range of
conditions
• Selection criteria may change
as understanding increases
• Successive small samples
instead of one large sample
Especially useful for PDSA testing. Someone with
process knowledge selects items to be sampled.
Characteristics of a Judgment Sample:
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd45
We are absolutely crazy around here between 9
and 11 AM!
But, things are pretty quiet after 3 PM.
Judgment Sampling takes advantage of the knowledge of those who own the process
What do I know? I usually work
afternoon shift and that is a different
process altogether!
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd46
You have performance data.Now what the heck do you do with it?
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd47
“If I had to reduce my message for
management to just a few words, I’d say it
all had to do with reducing variation.”
W. Edwards Deming
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Variation Exists!
48
Variation exists in all that we do, even in the simplest of activities. For example,
consider writing your name. This is a simple activity that you probably do each day.
What if your annual performance review, however, was based on being able to write
the first letter of your first name three times with no variation in the form, structure or overall appearance of the letter. If you are able to perform this simple task, you will
receive a 50 percent increase in your salary. Remember that there can be no
variation in the letters. Give it a try.
Now here is the second part of the your performance evaluation. Place your pen or
pencil in your opposite hand and write the same letter three times. How many of you passed the performance evaluation test? If you are like me your results look
something like this:
Source: R. Carey & R. Lloyd. Measuring Quality Improvement in Healthcare, ASQ Press, 2001.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Variation Exists!
49
Variation exists in all that we do, even in the simplest of activities. For example,
consider writing your name. This is a simple activity that you probably do each day.
What if your annual performance review, however, was based on being able to write
the first letter of your first name three times with no variation in the form, structure or overall appearance of the letter. If you are able to perform this simple task, you will
receive a 50 percent increase in your salary. Remember that there can be no
variation in the letters. Give it a try.
Now here is the second part of the your performance evaluation. Place your pen or
pencil in your opposite hand and write the same letter three times. How many of you passed the performance evaluation test? If you are like me your results look
something like this:
Bob’s left hand Bob’s right hand
Source: R. Carey & R. Lloyd. Measuring Quality Improvement in Healthcare, ASQ Press, 2001.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
QualityBetter
Old Way(Quality Assurance)
QualityBetter Worse
New Way(Quality Improvement)
Action taken on all
occurrences
Reject defectives
Old Way versus New Way
Source: Robert Lloyd, Ph.D.
Requirement,Specification or Target
No
action
taken
here
Worse
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd51
The Problem
Performance against a target or the use of aggregated data presented in
tabular formats with summary statistics, will not help you measure
the impact of process improvement/redesign efforts.
Aggregated data can only lead to judgment, not to improvement!
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd52
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd53
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Average CABG MortalityBefore and After the Implementation of a New Protocol
54
Perc
ent
Mort
alit
y
Time 1 Time 2
3.8
5.2
5.0%
4.0%
WOW!
A “significant drop”
from 5% to 4%
Conclusion -The protocol was a success! A 20% drop in the average mortality!
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Average CABG MortalityBefore and After the Implementation of a New Protocol A Second
Look at the Data
55
Perc
ent
Mort
alit
y
24 Months
1.0
9.0
Now what do you conclude about the impact of the protocol?
5.0
UCL= 6.0
LCL = 2.0
CL = 4.0
Protocol implemented here
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd56
The average of a set of numbers can be created by many different distributions
X (CL)
Me
as
ure
Time
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd57
If you don’t understand the variation that lives in your data, you will be tempted to ...
• Deny the data (It doesn’t fit my view of reality!)
• See trends where there are no trends
• Try to explain natural variation as special events
• Blame and give credit to people for things over which they have no control
• Distort the process that produced the data
• Discredit the messenger!
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd58
“What is the variation in one system over time?” Walter A. Shewhart - early 1920’s, Bell Laboratories
time
UCL
Every process displays variation:• Controlled variation
stable, consistent pattern of variation“chance”, constant causes
• Special cause variation“assignable”
pattern changes over time
LCL
Static ViewS
tatic
Vie
w
Dynamic View
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd59
Types of Variation
Common Cause Variation• Is inherent in the design of the
process
• Is due to regular, natural or ordinary causes
• Affects all the outcomes of a process
• Results in a “stable” process that is predictable
• Also known as random or unassignable variation
Special Cause Variation
• Is due to irregular or unnatural causes that are not inherent in the design of the process
• Affect some, but not necessarily all aspects of the process
• Results in an “unstable” process that is not predictable
• Also known as non-random or assignable variation
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Good versus Bad Variation
Common Cause does not mean “Good Variation.” It only means that the process is stable and predictable.
For example, if a patient’s systolic blood pressure averaged around 165 and was usually between 160 and
170 mmHg, this might be stable and predictable but completely unacceptable.
Similarly Special Cause variation should not be viewed as “Bad Variation.” You could have a special cause that
represents a very good result (e.g., a low turnaround time), which you would want to emulate. Special Cause
merely means that the process is unstable and unpredictable.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd61
3 Questions …
1. Is the process stable?
2. Is the process predictable?
3. Is the process capable?
The chart will tell you if the process is stable and predictable.
You have to decide if the output of the process is acceptable!
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
12/9
5
2/9
6
4/9
6
6/9
6
8/9
6
10/9
6
12/9
6
2/9
7
4/9
7
6/9
7
8/9
7
10/9
7
12/9
7
2/9
8
4/9
8
6/9
8
8/9
8
10/9
8
12/9
8
2/9
9
4/9
9
6/9
9
month
Perc
ent
C-s
ections
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
UCL=27.7018
CL=18.0246
LCL=8.3473
Percent of Cesarean Sections Performed Dec 95 - Jun 99
Common Cause Variation
Normal Sinus Rhythm (a.k.a. Common Cause Variation)
Week
Nu
mb
er
of M
ed
ica
tio
ns E
rro
rs p
er
10
00
Pa
tie
nt D
ays
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
22.5
UCL=13.39461
CL=4.42048
LCL=0.00000
Medication Error Rate
Atrial Flutter Rhythm (a.k.a. Special Cause Variation)
Special Cause Variation
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Attributes of a Leader WhoUnderstands Variation
Leaders understand the different ways that variation is viewed.
They explain changes in terms of common causes and special causes.
They use graphical methods to learn from data and expect others to consider variation in their decisions and actions.
They understand the concept of stable and unstable processes and the potential losses due to tampering.
Capability of a process or system is understood before changes are attempted.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd64
Question Response Options1. The Board evaluates our data using
criteria for common and special cause
variation
Strongly Agree Agree
Not Sure
Disagree Strongly Disagree
2. Senior Management evaluates our data
using criteria for common and special cause
variation
Strongly Agree Agree
Not Sure
Disagree Strongly Disagree
3. Front-line Managers evaluate our data
using criteria for common and special cause
variation
Strongly Agree Agree
Not Sure
Disagree Strongly Disagree
4. Staff Members evaluate our data using
criteria for common and special cause
variation
Strongly Agree Agree
Not Sure
Disagree Strongly Disagree
If you responded Disagree or Strongly Disagree to any question, what criteria are used by this group to determine if data or improving or
getting worse?
Attributes of a Leader WhoUnderstands Variation
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
AIM (Why are you measuring?)
Concept
Measures
Operational Definitions
Data Collection Plan
Data Collection
Analysis
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using
Indicators. Jones and Bartlett, 2004.
ACTION
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd66
How can I depict variation?
STATIC VIEW
Descriptive StatisticsMean, Median & Mode
Minimum/Maximum/RangeStandard Deviation
Bar graphs/Pie charts
DYNAMIC VIEWRun Chart
Control Chart
(plot data over time)
Statistical Process Control (SPC)
Ra
te p
er
100
ED
Pa
tients
Unplanned Returns to Ed w/in 72 Hours
M41.78
17
A43.89
26
M39.86
13
J40.03
16
J38.01
24
A43.43
27
S39.21
19
O41.90
14
N41.78
33
D43.00
20
J39.66
17
F40.03
22
M48.21
29
A43.89
17
M39.86
36
J36.21
19
J41.78
22
A43.89
24
S31.45
22
Month
ED/100
Returns
u chartu chartu chartu chart
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
0.0
0.2
0.4
0.6
0.8
1.0
1.2
UCL = 0.88
Mean = 0.54
LC L = 0.19
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd67
How do we analyze variation for quality improvement?
Run and Shewhart Charts
are the best tools to
determine if our
improvement strategies
have had the desired effect.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
Point Number
Po
un
ds o
f R
ed
Ba
g W
aste
3.25
3.50
3.75
4.00
4.25
4.50
4.75
5.00
5.25
5.50
5.75
6.00
Median=4.610
Measu
re
Time
Four simple run rules are used to determine if non-random data patterns are present
X (CL)~
Elements of a Run Chart
The centerline (CL) on a Run Chart is the Median
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Jan01 Mar01 May01 July01 Sept01 Nov01 Jan02 Mar02 May02 July02 Sept02 Nov02
Month
Nu
mb
er
of
Co
mp
lain
ts
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
A
B
C
C
B
A
UCL=44.855
CL=29.250
LCL=13.645
Elements of a Shewhart Chart
X (Mean)
Me
as
ure
Time
An indication of a
special cause
(Upper Control Limit)
(Lower Control Limit)
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd70
February April
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
16 Patients in February and 16 Patients in April
Min
ute
s
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
22.5
25.0
27.5
30.0
A
B
C
C
B
A
UCL=15.3
CL=10.7
LCL=6.1
XmR Chart
How all this works:Wait Time to See the Doctor
Baseline Period
Intervention
Where will the
process go?
Freeze the Control Limits and Centerline, extend them
and compare the new process performance to these
reference lines to determine if a special cause has been
introduced as a result of the intervention.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd71
February April
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
16 Patients in February and 16 Patients in April
Min
ute
s
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
22.5
25.0
27.5
30.0
A
B
C
C
B
A
UCL=15.3
CL=10.7
LCL=6.1
XmR Chart
Freeze the Control Limits and compare the
new process performance to the baseline
using the UCL, LCL and CL from the
baseline period as reference lines
A Special Cause is detected
A run of 8 or more data points on one side of the centerline reflecting a sift in the process
Baseline Period
Intervention
How it works:Wait Time to See the Doctor
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd72
February April
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
16 Patients in February and 16 Patients in April
Min
ute
s
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
22.5
25.0
27.5
30.0
A
B
C
C
B
A
UCL=15.3
CL=10.7
LCL=6.1
XmR Chart
Intervention Make new control limits for
the process to show the
improvement
Baseline Period
How it works:Wait Time to See the Doctor
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Run and Control Charts don’t tell you
• The reasons(s) for a Special Cause
• Whether or not a Common Cause process should be improved (Is the performance of the process acceptable?)
• How the process should actually be improved or redesigned
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd75
1. Which process do you want to improve or redesign?
2. Does the process contain non-random patterns or special causes?
3. How do you plan on actually making improvements? What strategies do you plan to follow to make things better?
4. What effect (if any) did your plan have on the process performance?
Run & Control Charts will help you answer Questions 2 & 4.
YOU need to figure out the answers to Questions 1 & 3.
A Simple Improvement Plan
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd76
Sustaining improvements and Spreading changes to other locations
Developing a change
Implementing a change
Testing a change
Act Plan
Study Do
Theory and Prediction
Test under a variety of conditions
Make part of routine operations
The Sequence of Improvement requires Measurement
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd77
AIM (How good? By when?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis ACTION
The Quality Measurement Journey
Source: R. Lloyd. Quality Health Care: A Guide to Developing and Using Indicators. Jones and Bartlett Publishers, 2004.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
ExerciseMeasurement Self-Assessment
This self-assessment is designed to help quality facilitators gain a better understanding of where
they personally stand with respect to the milestones in the Quality Journey. What would your
reaction be if you had to explain the PDSA cycle to your colleagues? Why is it preferable to plot
data over time rather than use aggregated statistics and tests of significance? Can you construct a
run chart or help a team decide which control is most appropriate for their data?
You may not be asked to do all of the things listed below today or even next week. But, if you are
facilitating a QI team or expect to achieve the goals o our collaborative, sooner or later these
questions will be posed. How will you deal with them?
The place to start is to be honest with yourself and see how much you know about QI concepts and
methods. Once you have had this period of self-reflection, you will be ready to develop a learning
plan for yourself and those on your improvement team.
Use the following Response Scale. Select the one response which best captures your opinion.
1 I could teach this topic to others!
2 I could do this by myself right now but would not want to teach it!
3 I could do this but I would have to study first!
4 I could do this with a little help from my friends!
5 I'm not sure I could do this!
6 I'd have to call in an outside expert!
Source: R. Lloyd, Quality Health Care:
A Guide to Developing and Using
Indicators. Jones & Bartlett
Publishers, 2004: 301-304.
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
Exercise: Measurement Self-AssessmentSource: R. Lloyd, Quality Health Care: A Guide to Developing and Using Indicators.
Jones & Bartlett Publishers, 2004: 301-304.
Measurement Topic or SkillResponse Scale
1 2 3 4 5 6
1. Build clear aim statements for our work
2. Move my team from concepts to specific quantifiable measures
3. Building clear and unambiguous operational definitions for our measures
4. Develop data collection plans (including sampling strategies and stratification)
5. Explain why plotting data over time is preferable to using aggregated data and summary statistics
6. Describe the differences between common and special causes of variation
7. Construct and interpret run and control charts
8. Identify specific ideas that we believe will achieve the results we desire
9. Set up and run PDSA tests on the ideas we have for improvement
10. Apply the sequence of improvement (testing, implementing and spreading) to the specific workstreams we are working on
©Copyright 2013 Institute for Healthcare Improvement/R. Lloyd
It must be remembered that there is nothing more difficult to plan, more doubtful of success, nor more dangerous to manage than the creation of a new system.
For the initiator has the enmity of all who would profit by the preservation of the old institution and merely lukewarm defenders in those who would gain by the new one.
A closing thought…
Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513Machiavelli, The Prince, 1513