![Page 1: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/1.jpg)
Are We Making a Difference? Using Measurement to Guide Improvement
Andrew Wray
Quality Forum 2017
![Page 2: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/2.jpg)
Presenter Disclosure
Presenter: Andrew Wray
Relationships with commercial interests: – Nothing to disclose
![Page 3: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/3.jpg)
Performance Measurement in Health Care
• Accountability: for judgement, to reassure the public, to decide how much people are paid, to choose between two options.
• Research: to develop new knowledge, often generalizable
• Improvement: to guide our efforts to change, to monitor our progress, to understand the problem.
![Page 4: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/4.jpg)
![Page 5: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/5.jpg)
Reference: The Improvement Guide, 2nd ed. Langley,
Moen, Nolan, Nolan, Norman & Provost, p. 24
![Page 6: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/6.jpg)
Purpose of Measurement for Improvement
Data can tell us:
- How we are progressing over time
- Where we can get some change ideas
![Page 7: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/7.jpg)
What to measure?
• Each project will need to identify a couple of key measures related to the work
• Will be a mix of process, outcome and balancing measures
• Needs to be meaningful for the team, and reflective of the processes you are improving
We’ll also have measures as part of our PDSA cycles – more later.
![Page 8: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/8.jpg)
Guideline: between 3-8 measures per project
• at least one outcome measure
• at least one process measure
A Few key indicators to track project progress
![Page 9: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/9.jpg)
Family of Measures
![Page 10: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/10.jpg)
![Page 11: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/11.jpg)
Example: Improving Diabetes Care
• Outcome: – HbA1C meeting clinical target
• Process:
– % who have attended structured education session – % screened for retinopathy in the last year – % reporting sufficient social and spiritual supports
• Balancing:
– Staff satisfaction
![Page 12: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/12.jpg)
• For your work, what sorts of indicators are important to track?
![Page 13: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/13.jpg)
Operational Definitions
• Full description of a measure: – What is the measure
– Inclusion/exclusion criteria
– Calculation
– Sample size
– Sampling strategy
– Subgroup frequency
– Data collection strategy
– By whom
– Etc.
![Page 14: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/14.jpg)
Measurement Plan Worksheet
Measure Operational
Definition Outcome, Process or Balancing
Data Collection Strategy
Frequency of Data Collection
How will measure be displayed
Baseline result Target result
![Page 15: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/15.jpg)
Operational Definitions • Option 1
– Number of people who fell
– All patients over 5yrs
– Number of patients with a fall/total number of patients
– Complete enumeration
– Monthly – Data source: DAD
• Option 3
– Number of falls with an injury
– All patients
– Count of all falls in the facility that caused harm
– Sample of 50 charts
– Audit by quality department staff
– Quarterly
• Option 2
– Number of falls
– Patients 70+
– Count of all falls in the facility
– Complete enumeration
– Reports to PSLS
– Monthly
![Page 16: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/16.jpg)
Sampling?
• Complete enumeration is best
• Sampling saves resources – usually recommended – use judgment samples
![Page 17: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/17.jpg)
![Page 18: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/18.jpg)
DATA COLLECTION
• Start right away
• Small, frequent measures
• Integrate into workload
• Timely
![Page 19: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/19.jpg)
• For the indicators you started thinking about, use the measurement plan worksheet to start working on an operational definition.
![Page 20: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/20.jpg)
• So we’re collecting data…
what do we do with it?
![Page 21: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/21.jpg)
• So we’re collecting data…
what do we do with it?
Share it with your team!
![Page 22: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/22.jpg)
The Run Chart: Tracking progress over time
Data displayed in time order
Data is collected and displayed weekly or monthly.
![Page 23: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/23.jpg)
Pre and Post Change Bar Chart – What is the Interpretation?
![Page 24: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/24.jpg)
Scenario 1 . Data displayed in a run chart over time.
0
2
4
6
8
10
change made between week 7 and 8
Scenario 1 . Pre-post data.
Let’s Look at the Data in a Run Chart: Scenario 1
![Page 25: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/25.jpg)
0
1
2
3
4
5
6
7
8
9
10
Scenario 2 . Pre-post data.
Scenario 2 . Data displayed in a run chart over time.
Scenario 2
change made between week 7 and 8
![Page 26: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/26.jpg)
Scenario 3. Pre-post data.
Scenario 3. Data displayed in a run chart over time.
change made between week 7 and 8
Scenario 3
![Page 27: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/27.jpg)
What if we use a t-test? Average Before
Change =70.0
Average After Change =30.1
0
10
20
30
40
50
60
70
80
The t-test shows a significant difference:
t(22)=7.6, p<.001
Adapted from Perla R.J., Provost L.P., & Murray S.K.
(2011). The run chart: a simple analytical tool for
learning from variation in healthcare processes. BMJ
Quality & Safety, 20(1):46-51.
![Page 28: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/28.jpg)
0
10
20
30
40
50
60
70
80
90
Week1
Week2
Week3
Week4
Week5
Week6
Week7
Week8
Week9
Week10
Week11
Week12
Week13
Week14
Week15
Week16
Week17
Week18
Week19
Week20
Week21
Week22
Week23
Week24
pre-change
post-change
Average Before Change
=70.0
Average After
Change =30.1
0
10
20
30
40
50
60
70
80
t(22)=7.6, p<.001 Adapted from Perla R.J., Provost L.P., & Murray S.K.
(2011). The run chart: a simple analytical tool for
learning from variation in healthcare processes. BMJ
Quality & Safety, 20(1):46-51.
![Page 29: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/29.jpg)
Data in real-time
• Data in real time allows us to track progress of the work
• Tells us if we getting better or worse
• Tells us if the changes we are testing are working
– The longer the lag, the slower the learning cycle
![Page 30: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/30.jpg)
Visual analysis of run charts
No improvement. Random fluctuation.
Improvement. Trend going up.
![Page 31: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/31.jpg)
Run charts: Evidence of non-random patterns
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 1. Shift
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 2. Trend
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 3. Runs
Data line crosses once Too few runs: total 2 runs
0
5
10
15
20
25
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
Rule 4. Astronomical Point
![Page 32: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/32.jpg)
“Measurement should be used to speed things up, not to slow them down”
• - IHI Breakthrough Series Guide
![Page 33: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/33.jpg)
Reference: The Improvement Guide, 2nd ed. Langley,
Moen, Nolan, Nolan, Norman & Provost, p. 24
![Page 34: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/34.jpg)
What about PDSA cycles?
• We’ll have a family of measures to track over the duration of the project but we will also need other measurement
– We’ll want to use measurement to learn what is working when we test changes.
![Page 35: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/35.jpg)
PDSA measurement
• Change idea: phone call reminders
• P: try 5 phone calls
• D: first 5 patients on Monday
• S: Number reached, length of call, did they attend
• A: What next?
![Page 36: Building a Measurement Plan... Where Do I Start?](https://reader031.vdocument.in/reader031/viewer/2022020301/58d0f7291a28abc00b8b547b/html5/thumbnails/36.jpg)
Generate light, not heat, with data
• Reinertsen JL, Gosfield AG, Rupp W, Whittington JW. Engaging Physicians in a Shared Quality Agenda. IHI Innovation Series white paper. Cambridge, Massachusetts: Institute for Healthcare Improvement; 2007