m1 presentation identifying improvement measures -...

11/26/2014

1

Identifying and Defining Improvement Measures

M1December 8, 2014

Following the CAUTI Case…

1. Baselines, Gaps, Aims, OutcomesWhere are we now, and what are we trying to accomplish?

2. Building a Theory of Improvement (Driver Diagram)

What should we measure and why?

3. Mapping the measures (Measure Tree)How will we calculate the measures?

4. Defining the MeasuresAttributes of Useful Improvement Measures

5. Collecting Data and Testing Changes

P2

11/26/2014

2

What Are We Trying to Accomplish?

Dimensions of Quality

Baselines, gaps, outcomes

Aim Statement

Three Types of Measures

Outcome Measures� Point to qualities that stakeholders value.

� Is this system meeting the needs of those who care about its operation?

� Is our improvement work making a meaningful impact?

Process Measures� Voice of the process.

� Are the parts/steps in the system performing as planned? Are processes reliable? Efficient? Patient-Centered?

� Are we on track to improve?

Balancing Measures� Are we producing perverse unintended consequences in our

efforts to improve? What other factors may be affecting results?

11/26/2014

3

Balancing Outcomes: IHI Triple AimP5

IOM Report: Dimensions of Care Quality

Safe - as safe in healthcare as in our homes

Effective - matching care to science; avoiding overuse of ineffective care and underuse of effective care

Patient-centered - honoring the individual and respecting choice

Timely - less waiting for both patients and those who give care

Efficient - reducing waste

Equitable - closing racial and ethnic disparities in access and health status

Institute Of Medicine (2001). Crossing the quality chasm : a new health system for the 21st century. Washington, D.C., National Academy Press.

11/26/2014

4

IHI Triple Aim: Examples of Measures

Systems of Care P8

D: The environment (policy,payment, accreditation, etc.)

C: Organizations that supportmicrosystems

B: Microsystems

A: Lives of Patients

11/26/2014

5

Case Background: Reducing CAUTIs

A medium sized acute care hospital has noticed that there has been an increasing occurrence of catheter associated urinary tract infections (CAUTIs) over the past year. Not only has the occurrence of CAUTIs been gradually going up but also the severity of the infections has been increasing.

Indwelling urinary catheters are commonly used medical devices within acute and non-acute settings. But their use does increase the risk of CAUTIs by:

• Enabling organisms to gain entry to the bladder via external surface or opened connections

• Reducing the body's defense of flushing out organisms during urination

• Facilitating biofilm formation

Reducing CAUTIs would contribute to:

• Improving the patient experience

• Reducing the cost of antibiotic prescribing

• Reducing inpatient length of stay

• Reducing readmissions

• Improving patient outcomes

Baseline Data – Key Outcome P10

11/26/2014

6

Aim P11

AIM: Reduce CAUTI infections in all units

below 1.6 (10th percentile) within 12

months and to zero within 24 months.

Exercise

1. Case Discussion

� Why are catheter-associated infections measured as

‘Number of CAUTIs per 1000 Foley catheter days?’

� What is the evidence that the rate of infections has

actually been increasing?

2. Own Project: Reflect and discuss in pairs

� What are you trying to accomplish (your aim?)

� What is the outcome measure that best captures the

aim of your project?

� What is the baseline level of performance on the

outcome? How much does the outcome need to

improve?

3. Share with the group

P12

11/26/2014

7

Building a Theory of Improvement

Driver diagrams

Prioritization

Linking drivers and measures

Theory Drives Improvement

“Without theory, there are no questions; without questions, there is no learning.”

P14

11/26/2014

8

A Theory of How to Improve a System

CauseEffectDrives

P15

Version:

11/26/2014Theory for CAUTI Reduction

Reduce catheter associated urinary tract infections by 50% in one year

P1 Leadership and aligned policy for catheter use

S1 Clear policies for infection control

Outcomes Primary Drivers Secondary Drivers Changes / Interventions

P2 Eliminate unnecessary catheter insertions

P3 Reliable compliance with catheter insertion protocol

P4 Reliable compliance with catheter maintenance protocol

S2 Transparent reporting of process failures

S3 Staff training, with feedback on observed protocol compliance

S4 Insert catheters only for appropriate indications

S6 Minimize use of catheters for patients at risk for infections

S8 Insertion only by trained staff

S9 Standard insertion procedure

S10 Daily assessment of need, removal at earliest opportunity

S5 Consider alternative methods

S11 Standard cleaning and maintenance procedure

Maintenance Bundle:A Tamper seal intactB Secured in placeC Hand hygieneD Meatal hygieneE Disposal & clean containerF Maintain unobstructed flow

Insertion Bundle:A Hand HygieneB Sterile gloves, materialsC Aseptic insertionD Unobstructed flow

S7 Remove when no longer required

11/26/2014

9

Version:

11/26/2014Needed Measures for CAUTI Reduction




















A fundamental assumption of

clinical QI:

Reliable execution of key clinical

driver processes improves

outcomes measured at the

population level

Measuring Improvement

Measures let us

• Monitor progress in improving

the system

• Identify effective changes

11/26/2014

10

Prioritizing Drivers

Limitations of resources, attention or will usually mean we cannot work on (or measure!) everything.

Priorities:

Where is the ‘Bang for Buck?’ Which drivers do we believe will deliver the biggest impact?

Which ones will be easiest to work on? Most difficult? Are some ‘beyond our control’?

What is our current level of performance on these drivers?

P20

CAUTI Driver Rankings

Difficulty

Impact

HIGH

LOW

HIGHLOW

P21

11/26/2014

11

CAUTI Priority Measure Concepts

S4: Insert catheters only for appropriate indications.

The most effective way to eliminate the possibility of a CAUTI is to eliminate an unneeded catheter.

S7: Remove when no longer required.

Since the risk of infection is roughly proportional to the time the catheter is in place, removing catheters as soon as possible will reduce the risk.

S9: Standard insertion procedure.

If trained staff follow strict protocols for aseptic insertion of catheters, the risk of bacterial infection will be minimized.

S11: Standard cleaning and maintenance procedure.

Similarly, careful adherence to the components of the maintenance bundle will reduce risk.

P22

Exercise

Use the Driver Diagram Rubric to guide the following:

Case Discussion

� Do you have questions or issues about the CAUTI

driver diagram?

� Discuss and resolve. If you get ‘stuck’, raise the

question to the group.

Own Project Discussion

� Review (or create) the driver diagram for your project

� Discuss in pairs (or to table)

� Be prepared to share with the group

P23

11/26/2014

12

Driver Diagram Rubric

1. Does the aim of the diagram focus on OUTCOMES?

2. Do the driver labels refer to the improvements needed to accomplish the aim?

3. Are all of the secondary drivers necessary for achieving the aim?

4. Are the secondary drivers sufficient to achieve the aim?

5. Do the drivers consider needed process, leadership, cultural, and structural changes?

6. (Optional) Does the diagram include change concepts or specific change ideas that might be tested as part of an improvement initiative?

Testing the diagram: Show the driver diagram to a knowledgeable person who is naïve to the system you want to improve. Ask them to explain what you are trying to accomplish and how. Identify areas of confusion, and consider revising your diagram.

P24

Mapping the Measures

Measure Tree Diagram

11/26/2014

13

Version:

11/26/2014Measures for CAUTI Reduction




















M1

M5

M2

M6

M3

M4

CAUTI Measures P27

11/26/2014

14

CAUTI Reduction Measures

D1 Count of patients with

catheters in situ in measurement

month

M1 Percent of patients

with appropriate

catheter placements

N1 Count of patients

meeting critiera for

catheter insertion

D2 Count of catheters inserted in

measurement month

M2 Average catheter

duration

N3, M3 Count of CAUTIs

in measurement month

N4 Count of catheter

insertions with all

insertion bundle

elements in compliance

D3, N2 Sum of

days with

catheters in situ

M4 CAUTIs per 1000

patient days

N5 Count of catheters

with all maintenance

bundle elements in

compliance

M5 Percent of catheter

insertions with all

insertion bundle


M6 Percent of catheter

insertions with all

maintenance bundle


Denominators Numerators Measures

M4 (alternate) Catheter

days between CAUTI

events

P28

Exercise

Case Discussion

� Do you have questions or issues about the CAUTI

measure tree?

� Discuss and resolve. If you get ‘stuck’, raise the

question to the group.


� Based on your own driver diagram, identify the

outcome and key process measures you will need (1

outcome, no more than 4 process)

� Create a measure tree that shows the numerators

and denominators for your measures.

� Be sure your process measures are linked to drivers.

P29

11/26/2014

15

Attributes of Useful Improvement Measures

Responsive

Valid

Comprehensible

Timely

Feasible

Relevant

Attributes of Useful Improvement Measures

Responsive The measure is sensitive to changes in the system state.

When the system improves, the measure says so.

Valid The measure aligns with the theory of changes (driver

diagram). Improvement in the measure means improvement

in the system.

Comprehensible The intended audience understands the meaning of the

measure for system improvement.

Timely The data are available soon enough to inform improvement

decisions (project planning, PDSA testing).

Feasible The data can be collected with minimum effort and cost, and

in a timely fashion.

Relevant The measure supports problem identification and testing at

the appropriate level of management.

Consistent The measure has a clear definition: it yields consistent

results when applied in different places and at different

times.

Ownership Someone is explicitly assigned to monitor the measure on a

regular basis, detect problems, and initiate change.

11/26/2014

16

Why Time Is Important for Measuring Improvement

“Improvement is temporal!” – Lloyd Provost

Displaying data over time (using run charts or control charts) allows us to make informed predictions, and thus manage effectively

Did We Improve?

Did we improve?

What will happen next?

Should we do something?

Source: R. Lloyd

Percent of ER patients with Chest Pain Seen by a

Cardiologist within 10 min

P35

11/26/2014

17

Version:

11/26/2014Validity: Aligned with Improvement Theory




















M1

M5

M2

M6

M3

M4

Validity: Alignment with Improvement Work

• Improvement in a pilot population (1 practice, 1 unit, etc.) will not be

evident in measures based on the total population (city, hospital system)

Total

Population Measurement

SamplePilot Unit

P39

11/26/2014

18

Validity: Alignment with Improvement Work

Total

Population Target

population

• To track improvement, we must measure in the same target population

where we are working to improve.

P40

Comprehensible?

Percentage of patients discharged in the measurement month that suffered a CAUTI

Number of CAUTIs per 1000 Foley catheter days during measurement month

Number of CAUTIs per 1000 inpatient days during the measurement month

Count of CAUTIs in the measurement month

Number of catheter days since the last CAUTI event

11/26/2014

19

Importance of Timely Data

Consistency: Operational Definition

A procedural description of what to measure and the steps to follow to measure it consistently…

� Gives communicable meaning to a concept

� Tells what you need to count or measure, and how to

do it

� Specifies measurement methods and equipment

� Provides guidance on sampling

� Identifies detailed criteria for inclusion and exclusion

… is the basis for reliable measurement

Source: R. Lloyd

11/26/2014

20

Operational Definition Example

Measure: Percentage of patients undergoing hip and knee replacement surgery during the measurement period who have had preoperative nasal swabs to screen for Staphylococcus aureus carriage

Goal: 95%Measurement Period Length: MonthlyNumerator Definition: Number of patients undergoing hip or knee replacement surgery who have had a nasal swab specimen processed to screen for Staphylococcus aureus carriage prior to surgeryDenominator Definition: Number of patients undergoing elective hip or knee replacement surgeryNumerator and Denominator Exclusions:• Patients who are less than 18 years of age • Patients who had a principal or admission diagnosis suggestive of preoperative

infectious diseases• Patients with physician-documented infection prior to surgical procedures• Patients undergoing non-elective hip or knee replacement surgeryDefinition of Terms:Hip or knee replacement surgery includes operations involving placement of a nonhuman-derived device into the hip or knee joint space. ICD-9 Codes include 00.70-00.73, 00.85-00.87, 81.51-81.53, 00.80-00.84, 81.54, and 81.55.Calculate as: (numerator/denominator*100)

Responsive

Percent of catheters removed during the measurement

month within 2 days of insertion

Average catheter duration by month

• Which measure better reflects the

improvement work of teams trying to reduce

unnecessary catheter placement?

• Which measure better reflects protocol

compliance?`

11/26/2014

21

January

April

June

August

September

Average Time versus Percent Conforming*

Specification = 30 min or less

Which measure is most useful to an improvement team?� % of cases with Abx within 30 min

� Average time to Abx admin

*Simulated data via @Risk

Population MeasuresThroughput =

visits…

…with reliable

care process…

Population: who’s health are we responsible for?

…have an

incremental

impact on

population.

DM pts with LDL<100

Active DM pts in practice panel

P50

11/26/2014

22

Typical Population Questions

• What is the current state of the population for whom we are responsible (even those we haven’t seen for awhile?) re: Health status? Pt. Experience? Cost of care?

• How do our population’s risk factors and outcomes compare with those of other provider organizations?

• How should we plan for the long term?

• What has the impact of our improvement work been on the population? Are there other factors effecting changes in outcomes?

Outpatient ‘Look-Back’ Measures

Percent of population with current self-management plan as of

most recent visit within the past 12 months.

Each measurement contains mostly the same patients as the previous month.These measures are slow to show improvement, but reflect the state of care for the population!

12 months

Current test

No current test

11/26/2014

23

“Current Care” Measures

Percent of patients seen last month who lacked an up-to-date

A1C and who got the test during the visit or were referred.

Each subgroup contains different patients & represents current workThese measures are great for tracking process changes!

Current test

No current test

1 mo

Exercise

Case Discussion

� The worksheet titled ‘Attributes of Useful

Improvement Measures’ shows alternative measures

that might be used to capture the key measure

concepts in the case.

� Compare the alternative measures with respect to the

attributes of useful improvement measures


� Refer to your proposed project measures

� How do your measures relate to the attributes of

useful improvement measures?

� Discuss at your table

Share your insights

P59

11/26/2014

24

P60

Data Collection

Data for PDSA testing

Concurrent data collection

Segmentation

Sampling

11/26/2014

25

Project Data Collection

Existing EMR system� PRO: data collected as component of routine care

� CON: needed process measures may not be included; data may lag by weeks or months; process failures lack context; usually requires custom reports

Paper chart review� PRO: notes may provide useful context; may be necessary

if no electronic system

� CON: labor intensive (but sampling helps); data may lag by days or weeks

Concurrent log or registry� PRO: ad hoc data can target PDSAs, project measures; no

lag; context available;

� CON: extra work for caregivers; special data process necessary

Measuring Reliability

Reliability =

Number of Actions That Achieve The Intended Result

Total Number of Actions Taken or Intended

= Percent ‘Conforming’

11/26/2014

26

Proposal that

patient requires

urinary catheter

Check pt. for past

problems,

allergies, etc.

Explain procedure

to pt. and/or

caregivers

Decontaminate

hands

Clean and prepare

the work area,

assemble materials

Prepare patient

Put on personal

protection equipment

(PPE) and sterile gloves

Is the

patient

male?

Follow male

procedures for

urinary catheter

insertion

Follow female

procedures for

urinary catheter

insertion

Record patient experience, document

technical specifications and time of

completion into the chart

Dispose of equipment and materials in

designated bag. Remove PPE and wash hands

No

Yes

A B

Alternative

methods

available?

Indications

are appro-

priate?

Yes No

No YesStop

% of cases with appropriate indication

% of females with proper insertion procedure% of males with

proper insertion procedure

% of cases with proper hand hygiene

Measuring Process: Total Joint Arthroplasty

Aim: Pre-screen all total hip or knee replacement patients for nasal Staph; those who test positive will complete a course of mupirocin.

Population: All patients undergoing TKA or HKA in our hospital (with exclusions)

Process: Screening and decolonization

Measurement interval: monthly

Process reliability measure:Percent of patients who screened positive for SA who report they had completed a course of mupirocin prior to day or surgery.

11/26/2014

27

Schedule

procedureS

ch

ed

ulin

gL

ab

Ho

sp

ita

l /S

urg

eo

n

TKA or THA?

Insert lab

request for SA

culture

Inform patient

of SA screening

Pt presents for

nasal swab

Positive for

SA?Process

specimen

Results to

surgeon &

hospital

Document in

record

Confirm Rx

complete

Surgery

1-4 weeks pre-procedure 2-3 weeks pre-procedure Day of surgery

Staph aureus (SA)Screening and Decolonization Process Example

Yes

Yes

No

No

Prescribe 5 day

mupirocin

Contact patient

% of no-shows for swab

Suggested

measures

% of positive results not acted on

KEY RELIABILITY MEASURE

% of colonized patients with completed Rx

Notify hospital

% of cases with missing lab order

Time to receive lab results

Time to notify patient

Individual Patient Data to Assess Process Improvement

0

20

40

60

80

100

120

140

160

9/1

6/1

0

10/1

/10

10/1

1/1

0

10/3

1/1

0

11/4

/10

12/2

/10

12/8

/10

12/1

9/1

0

12/2

7/1

0

1/1

7/1

1

2/2

/11

2/1

0/1

1

2/1

0/1

1

2/1

5/1

1

2/2

5/1

1

2/2

8/1

1

3/2

/11

3/5

/11

3/2

4/1

1

3/2

8/1

1

4/2

8/1

1

5/8

/11

5/1

3/1

1

5/1

7/1

1

5/2

3/1

1

5/2

3/1

1

6/1

5/1

1

6/2

5/1

1

6/2

7/1

1

7/3

/11

7/8

/11

7/1

0/1

1

7/1

8/1

1

7/2

8/1

1

7/3

0/1

1

8/4

/11

8/1

0/1

1

8/1

8/1

1

8/2

5/1

1

8/2

8/1

1

8/3

1/1

1

9/5

/11

9/8

/11

9/2

1/1

1

9/2

5/1

1

9/2

7/1

1

10/2

/11

10/5

/11

10/1

8/1

1

10/2

1/1

1

10/2

5/1

1

10/3

1/1

1

11/1

0/1

1

11/1

6/1

1

11/2

6/1

1

12/5

/11

12/1

4/1

1

1/2

/12

1/5

/12

1/2

8/1

2

1/3

0/1

2

2/8

/12

2/1

4/1

2

2/2

0/1

2

Time(min)

Individual Patients Over Time

Time to Antibiotic Medication Sept 2010 - Feb 2012

Updated: March 7, 2012

Shift = 6; Trend = 5

Mean = 46 Mean = 38

Mean = 35

Mean = 27.3

Mean = 29.9 Guidelines

#1 Guidelines

#2ED Comm

Stickies/ QI ED Comm

Excel Dose Calc

VOE Guidelines

Goal:

Source: James Moses, BMC

11/26/2014

28

Sampling

… when you can’t measure

the entire population, you

can estimate its

characteristics by sampling

• Systematic sampling

• Random sampling

• Stratified sampling

• Convenience sampling

• Judgment sampling

Sampling Methods

Source: R. Lloyd

Convenience Sample

“Gosh I’m in a hurry. Why don’t I just review these?”

11/26/2014

29

Simple

Random

Sample

Every element has an equal chance of being selected

Sampling Methods

Systematic

Sample

…then select every nth element

First element selected at random…

Possible bias if there are patterns in the sequence of elements

You might survey every 10th patient who arrives at a clinic

beginning at a randomly selected time

Sampling Methods

11/26/2014

30

Sampling Methods

Judgment Sample

Especially for PDSA testing,

someone expert with the process

selects items to measure:

• To include a range of conditions

• Selection criteria may change as

understanding increases

• Successive small samples

instead of one large sample

What Sample Size?

To be useful, samples should be large enough to reveal improvement shifts and trends.

This also depends on magnitude of the change, and the inherent variability of the measure.

30 is a good rule of thumb for current care measures

You can approach this issue empirically

Don’t sample unless you need to

Small samples ok for PDSA testing

11/26/2014

31

Tracking Change– Segment by Segment

Segment 1 - Pilot Segment 2

Jan 10 Mar 11

Segment 3

Segmentation

By

Organization

Site

Provider

Region

Diagnosis

Patient process ‘trajectory’

© R. Scoville • 76

11/26/2014

32

Exercise

How did the CAUTI team approach their data collection task?

Own Project

� What data are available to support your improvement

measurement plan? Is it possible to gather concurrent

data?

� What are some of the change ideas that you will test?

What data will be needed to assess their impact?

How will you gather those data?

m1 presentation identifying improvement measures -...

Documents