creating, critiquing and interpreting run charts and spcc

Creating, Critiquing and Interpreting Run Charts and SPCC

Heather Kaplan MD, MSCE Assistant Professor of Pediatrics, Perinatal Institute and The James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital Medical Center Cincinnati, OH Heather Kaplan MD MSCE is an Assistant Professor of Pediatrics in the Perinatal

Institute and the James M. Anderson Center for Health Systems Excellence at Cincinnati Children's Hospital Medical Center (CCHMC). Heather is a neonatologist and health services researcher interested in enhancing care delivery and studying how systems of care can be improved using innovative approaches. She completed her neonatal-perinatal fellowship training, including earning a Master's degree of science in clinical epidemiology, at The Children's Hospital of Philadelphia/University of Pennsylvania. She joined the faculty at CCHMC in August 2007 . Heather's early research focused on understanding variation in adoption of evidence-based practices in neonatal care and quality improvement as a strategy for implementing evidence in practice. With funding from the Robert Wood Johnson Foundation, she studied the role of context in the success of quality improvement initiatives and developed a model, the Model for Understanding Success in Quality (MUSIQ). MUSIQ is a tool for developing theories about which aspects of context help or hinder a specific project, and designing and implementing tests of changes to modify those aspects of context. Her current work examines the way research and improvement networks ("learning networks") can be used to improve care delivery and outcomes. She is specifically interested in scaling improvement to reach entire populations of patients and the ways technology, quality improvement methods, and N-of-1 trial methods can be combined to create a personalized learning healthcare system for the individual. Heather also has extensive experience with front-line quality improvement in perinatal care. Dr. Kaplan serves as the Improvement Advisor for the Ohio Perinatal Quality Collaborative (OPQC) neonatal improvement work. She also serves as a faculty expert for Vermont Oxford Network quality collaboratives and has been working with teams to improve their system of improvement by using MUSIQ to identify and modify key aspects of context that are affecting the success of the quality improvement projects and to help them engage with senior leadership around their improvement work.

Munish Gupta MD, MMSc Neonatologist Beth Israel Deaconess Medical Center Boston, MA Munish Gupta MD, MMSc, is a staff neonatologist and the Director of Quality and Safety for the Department of Neonatology at Beth Israel Deaconess Medical Center in Boston MA. He is also chair of the Neonatal Quality Improvement Collaborative of Massachusetts.

Annual Quality Congress Breakout Session, Sunday, October 29, 2017 Creating, Critiquing and Interpreting Run Charts and SPCC Objective: Participate in a workshop aimed at best practices in interpreting QI data including run charts and SPCC.

Creating, Critiquing and Interpreting Run Charts and Control Charts

Heather Kaplan MD, MSCE / Munish Gupta MD, MMSc

October 29, 2017 1


Heather Kaplan MD, MSCE

Munish Gupta MD, MMSc

VON Annual Meeting & Quality Congress Workshop

October 2017

Disclosures

Heather Kaplan MD, MSCE and Munish Gupta MD, MMSc

have no relevant financial disclosures related to the

content of this workshop.

Heather Kaplan MD, MSCE Munish Gupta MD, MMSc

Learning Objectives

Participate in a workshop aimed at best practices in

interpreting QI data including run charts and SPCCA Quick Review of the Basics…

Data Over Time Understanding Variation In quality improvement, we are

looking for change in key data

But natural background variation

in all things we do – fact of life

Need tools to identify true

changes versus natural variation

And, we would like to detect true

change fast



October 29, 2017 2

Signal vs. NoiseSIGNAL

means something

contains information

difference with a distinction

special cause variation —

specific causes not part of

usual process (good or bad)

NOISE

statistically indistinguishable

from other data points

contains no new information

difference without a distinction

common cause variation —

causes inherent as part of usual

process (good or bad).• Takes 28‐35 min to get to work• Variation due to factors inherent in the process

• Lead‐footed‐ness• Departure time (6:40 vs. 6:43 AM)• Starbucks mobile order speed

noise

• Takes 56 min, 58 min to get to work• Variation due to a specific cause—road closure in

the square by my house• Eliminate special cause (construction complete),

drive time returns to stable process (28‐35 min)

signalWhy this is important

Type of variation type improvement action

Type of variation

Reduce unnatural variation

Reduce natural variation, improve basic process

Common cause

Special cause

Establish stable work process

Improve overall outcomes

Tampering

Responding to a value thinking it’s a signal when actually it’s noise

Increases process and system variability

Walter Shewhart

Signal vs. Noise

Statistical Process Control

Tools to help distinguish signal from noise to limit tampering

Plot data over time

Interpret visually and statistically … easy for those on the frontlines!

Walter Shewhart

Signal vs. Noise



October 29, 2017 3

Statistical Process Control Tools

1. Run charts – minimal standard

2. Control charts

Keys:

Plot and evaluate over time

Interpret visually and statistically

What is a Run Chart

Visual display of data over time, annotated

Center line: Median or mean value

Run Charts

Minimum standard for QI project data

Can start with first few data points!

Need at least 10 data points to use rules for detecting

special cause

Simple to create (no software needed)

Can be used with all types of data

But… not as powerful as a control chart

Value of Result

Unit of Time(e.g. days, weeks, months, quarters)

Running record of the process over time

Run chart: Center line is the median.

Control chart: Center line is often the mean.

Control limits that reflect inherent variability in data or the extent of common cause variation

Mean

Upper Control Limit (UCL)

Lower Control Limit (LCL)

Run Charts and Control Charts

Median

9181716151413121111

65

60

55

50

45

40

35

Time

_X=49.77

UCL=58.77

LCL=40.77

1

1

11

11

1

• Each time you add a data point…ask your self, “Based on preceding data, are additional data points consistent with predicted performance (likely) or not consistent with predicted performance (unlikely)?”

• A stable process performs predictably over time, with an expected amount of variation• If your next data point is consistent with predicted performance, there is common cause variation (noise)• If your next data point is not consistent with predicted performance, there is special cause (signal)

Control Charts

Continuous Data

1. Numerical value for each unit in a group

Discrete (Integer) Data

2. Classification: Presence or not of an attribute

3. Count: How many attributes occur in sample

Type of data

Sample Size

Type of Chart

Math(software)

Constructing Control Charts



October 29, 2017 4

Types of Data & Control Charts

Common cause probability model Example

Discrete

Classification: Binomial

Parameter: p

Patient develops an SSI (Y/N)

Count: Poisson

Parameter: l

Number of catheter‐associated HAIs

Continuous

Normal

Parameters: m, s

Time to deliver thrombolytics

Which Control Chart To Use

Adapted from Provost & Murray, The Health Care Data Guide, 2011, and Carey, Improving Healthcare with Control Charts, 2003.

Type of Data

Discrete / Attribute(data is counted or classified)

Continuous / Variable(data is measured on a scale)

Count(events/errors are counted; numerator can be greater

than denominator)

Classification(each item is classified;

numerator cannot be greater than denominator)

Equal or fixed area of

opportunity

Unequal or variable area of opportunity

Equal or unequal subgroup size

Subgroup size = 1(each subgroup is single

observation)

Subgroup size > 1(each subgroup has

multiple observations)

C chartCount of events

U chartEvents per unit

P chartPercent classified

X and MR chartsIndividual measures and

moving range

X-bar and S chartsAverage and standard

deviation

ACTION

NO Special Cause is occurring in

System

Special Cause is occurring in System

Take action on individual outcome (treat special) MISTAKE 1 OK

Treat outcome as part of system; work on changing the system (treat common)

OK MISTAKE 2

ACTUAL SITUATION

Provost, LP and Murray S. The Data Guide. 2008

Identifying Common vs. Special Cause Rules for Identifying Special Cause (Signals)

Perla et al, BMJ Qual Saf 2011; 20:46‐51

Run Charts

Rules for Identifying Special Cause (Signals)Control Charts

TEST 1:1 point outside outer control limit

TEST 2:2 out of 3 points more than 2 SD from center line

TEST 3:Run of 8 points in a row on one side of center line

TEST 4: Trend of 6 points in a row increasing or decreasing

Stable Process Process well defined and predictable

Range of variation (performance)

intrinsic to the process

• Common Cause: sampling error, noise; no signals

Changing results achieved by a

statistically controlled process requires

a new process…Current process

perfectly designed to get what it gets

noise



October 29, 2017 5

Unstable Process Process not defined and unpredictable

Range of variation (performance) not intrinsic–

influenced by external perturbations

• Special cause: outside chance variation –signals

Changing results achieved by a statistically

uncontrolled process begins with removing the

special causes to establish what the core

process represents

Learning from experience with unstable

processes is severely limited…all one can say is

that no one knows what will happen next!

signal

Interpreting a Run Chart or Control Chart1. Is this the right chart?

• Are the measures appropriate (y‐axis)?

• Is the time analysis appropriate (x‐axis)?

• Is the sample size for each data point adequate?

• Is the control chart right for the type of data/measure?

2. Are there enough data points for meaningful conclusions?

3. Is there evidence of special cause variation?

4. Does the chart match your knowledge of your system and

its context?

Examples Example 1: Providence St. Vincent’s

Providence St. Vincent’s, Portland OR Aim: increase morbidity‐free survival of extremely preterm infants

Measures: many, including Standardized Process Assessment (SPA)

scores on each infant, measuring performance on bundles of

potentially best practices in each of 8 infant care areas

Changes: creation of Small Baby Team (SBT) and Small Baby Unit

(SBU), then multiple PDSAs addressing each care area

Data shown: one of their SPA scores

Describe the chart!

• What kind of chart is it?

• What is the unit of measurement on the x-axis?

• What is the main measure?

• What is the measure of central tendency?

• What are the different lines?

• What’s the sample size for each period?

• Are the process changes labelled (annotated)?

• Which direction is better?

• Is this the right chart?

• Is there evidence of improvement?



October 29, 2017 6

Date / Observation Value Mean Goal

1 6 6 82* outborn 3 6 8

3 7 6 84 7 6 85 4 6 86 5 6 87 6 6 88 7 6 89 7 6 8

10 6 6 811 9 8.2 812 8 8.2 813 9 8.2 814 7 8.2 815 9 8.2 816 9 8.2 8

17*outborn 7 8.2 818*outborn 8 8.2 8

19 8 8.2 821 9 8.2 822 9 8.2 8

23*outborn 9 8.2 824 7 8.2 8

25*outborn 8 8.2 826 8 8.2 8

27*outborn 8 8.2 828 8 8.2 829 6 8.2 831 9 8.2 8

33*outborn 9 8.2 834 8 8.2 835 8 8.2 8

Thoughts?

• Clear visual evidence of improvement

• Clear shift (pts 1 to 10)

• Just enough runs (8, with expected range 7 to 17)

Run chart shows clear signal, and supports impact of improvement efforts.

Using for QI

• Start run chart with baseline data

• After at least 10 data points, can use rules for identifying signal

• If no signal noted, and if process is stable, can extend mean and add new data points

Using for QI

• Signal VERY evident –would have been seen by patient 14

• Can add annotations and goal line

Using for QI

• With evidence of signal after patient 10, and knowledge that process changed, can shift the center line to reflect new performance

Using for QI

• Tells the ‘story’!

• Significant improvement with first intervention

• Further performance compared to new median

• Process stable since first change, and at goal

• Need new interventions if further improvement desired



October 29, 2017 7

Run Charts: Good Practices Need title, labelled axes including primary measure, center line,

direction of improvement, & indication of sample size

Should generally use median for center line

Can start with just a few data points, but at least 10 points

needed to establish median and use rules for signal

If process appears stable, can extend median and compare

future performance to established median

Shift median if clear signal and knowledge of process change

Use annotations and goal line!

Example 2: Emory

Emory, Atlanta GA Aim: decrease prolonged use of antibiotics in NICU infants

Measures: treatment with greater than 2 days of initial antibiotics

among infants started on antibiotics

Changes: multiple PDSAs, including pharmacy audits, provider

feedback, and practice guidelines

Data shown: main outcome chart

Describe the chart!











Monthly period (n)Infants treated

>48hrInfants receiving initial antibiotics

Jan 2015 (n=30) 13 30Feb 2015 (n=11) 7 11Mar 2015 (n=22) 16 22Apr 2015 (n=29) 11 29May 2015 (n=36) 15 36Jun 2015 (n=22) 11 22Jul 2015 (n=34) 14 34

Aug 2015 (n=34) 13 34Sep 2015 (n=24) 6 24Oct 2015 (n=33) 13 33Nov 2015 (n=28) 14 28Dec 2015 (n=36) 15 36Jan 2016 (n=24) 10 24Feb 2016 (n=22) 5 22Mar 2016 (n=23) 7 23Apr 2016 (n=19) 7 19May 2016 (n=23) 14 23Jun 2016 (n=23) 12 23Jul 2016 (n=23) 4 23

Aug 2016 (n=22) 8 22Sep 2016 (n=22) 10 22Oct 2016 (n=22) 9 22Nov 2016 (n=24) 12 24Dec 2016 (n=19) 4 19Feb 2017 (n=21) 9 21Mar 2017 (n=34) 14 34

Apr 2017 (n=20) 5 20May 2017 (n=29) 11 29Jun 2017 (n=20) 3 20Jul 2017 (n=48) 11 48

Aug 2017 (n=27) 10 27

Thoughts?

• Appropriate control chart, with overall rate of > 2d of antibiotics of 39%, and outer control limits of approximately 10% to 70%

• While visual interpretation may not suggest clear trend in data, statistical interpretation shows evidence of improvement with special cause

• Could add annotations and goal line to make chart even more effective



October 29, 2017 8

Thoughts?

• With annotations, appears improvement was noted after third PDSA, to better than goal

• If special cause variation seen, with change in system that is expected to last, can update mean and limits

Thoughts?

• Tells more of the story, suggesting improvement from 41% to 25% after three PDSA cycles

• Since 2nd mean is only based on 3 data points, need to update mean monthly until 15-20 data points available

• Future performance should be compared to 2nd mean

Control Charts: Good Practices Need clear title, axes labels, center line, control limits, direction

of improvement, and ideally sample size

Use statistical rules to identify special cause variation

Match graph to knowledge of system

Need 15‐20 data points to establish reliable mean and limits

If evidence of special cause variation and knowledge of change

in system that is expected to last, can revise mean and limits

Example 3: Nationwide Children’s

Nationwide Children’s, Columbus OH Aim: decrease post‐operative complications of surgically placed

gastrostomy tubes in NICU patients

Measures: complication rate in surgically‐placed G‐tubes

Changes: multiple PDSAs, including G‐tube crib card, standardized

dressing changes, and standardized feeding plans

Data shown: main outcome chart

Describe the chart!













October 29, 2017 9

# of G‐tubes # of Complications Complication Rate

Mar‐15 5 0 0%

Apr‐15 5 3 60%

May‐15 4 1 25%

Jun‐15 5 2 40%

Jul‐15 5 3 60%

Aug‐15 4 1 25%

Sep‐15 4 3 75%

Oct‐15 8 2 25%

Nov‐15 8 3 38%

Dec‐15 5 0 0%

Jan‐16 4 2 50%

Feb‐16 3 1 33%

Mar‐16 6 1 17%

Apr‐16 3 0 0%

May‐16 4 3 75%

Jun‐16 5 2 40%

Jul‐16 6 2 33%

Aug‐16 3 0 0%

Sep‐16 8 1 13%

Oct‐16 7 0 0%

Nov‐16 4 3 75%

Dec‐16 1 0 0%

Jan‐17 4 1 25%

Feb‐17 3 1 33%

Apr‐17 1 0 0%

May‐17 3 0 0%

Jun‐17 8 1 13%

Jul‐17 7 1 14%

Aug‐17 4 0 0%

Sep‐17 4 0 0%

Thoughts?

• Overall trend seems to be in right direction

• Control limits are very wide, particularly low limits – sample size is small

• No special cause variation seen (although does not seem to be a stable process)

Using for QI

• Start with baseline data

• Initial chart with 12 data points reflecting first ‘process stage’

• Mean fixed at 35%?

Using for QI

• Start with baseline data

• Initial chart with 12 data points reflecting first ‘process stage’

• Mean fixed at 35%?

• “Additional points added, compared to fixed mean –special cause variation!

• But… hard to designate process as stable (and fix mean) with just 12 data points, and with such wide control limits

• Other approaches?

Bimonthly Control Chart

• Some improvement in sample size, with narrower control limits (although lower limit still at zero)

Quarterly Control Chart

• Further improvement in sample size, with trend that seems more meaningful

• Lower control limit still zero

• However, only 10 data points – should start with run chart until additional data points available



October 29, 2017 10

Control Charts: Good Practices Need to interpret control charts visually, as well as using

statistical rules

Adequate sample size important – try to avoid control limits

that are mostly at the extremes

After 12 data points, can create trial mean and limits, but

update them until 15‐20 data points are available

After 20 points, if process is stable, fix mean and limits

Other examples? Questions?

ReferencesBenneyan, J.C., R.C. Lloyd, and P.E. Plsek, Statistical process control as a tool for research and healthcare improvement. Qual Saf Health Care, 2003. 12(6): p. 458‐64.

Benneyan, J.C., The design, selection, and performance of statistical control charts for healthcare process improvement. Int J Six Sigma and Competitive Advantage, 2008. 4(3):p.209‐239.

Carey, R.G., Improving healthcare with control charts : basic and advanced SPC methods and case studies. 2003, Milwaukee, WI: ASQ Quality Press. xxiv, 194 p.

Gupta, M, Kaplan, HC, Using Statistical Process Control to Drive Improvement in Neonatal Care: A Practical Introduction to Control Charts. Clinics in Perinatology, 2017, 627‐644.

Jordan, V, J.C. Benneyan, Common Errors in Using Healthcare SPC, in Statistical Methods in Healthcare. 2012, Wiley, pp. 268‐285.

Langley, G.J., R.D. Moen, K.M. Nolan, T.W. Nolan, C.L. Normal, and L.P. Provost, The Improvement Guide. 2nd ed. 2009, San Francisco, CA: Jossey‐Bass. 490 p.

Perla, R.J., L.P. Provost, and S.K. Murray, The run chart: a simple analytical tool for learning from variation in healthcare processes. BMJ Qual Saf, 2011. 20(1): p. 46‐51.

Provost, L.P. and S.K. Murray, The Health Care Data Guide: Learning From Data for Improvement. 1st ed. 2011, San Francisco, CA: Jossey‐Bass. 445 p.

creating, critiquing and interpreting run charts and spcc

Documents