rapid cycle evaluation for improvement leaders -...

12/7/2014

1

Rapid Cycle Evaluation for Improvement Leaders

Gareth Parry, Amy Reid & Sandy Cohen, Don

Goldmann

Session L6These presenters have

nothing to disclose

December 7, 2014

Introductions, Agenda, and Objectives

Don Goldmann & Gareth Parry, MD

1:00 to 1:10pm

L6: Rapid Cycle Evaluation for Improvement Leaders

December 7, 2014

12/7/2014

2

Faculty

Sandy Cohen MSW, MPH

Research Associate

[email protected]@SandyCohen_

Gareth Parry, PhDSenior Scientist

[email protected]@GJParry03

Amy Reid, MPHSenior Research

Associate

[email protected]@_amyjreid_

3

Don Goldmann, MDChief Medical and Scientific Officer

[email protected]@DAGoldmann

Overview of the Learning Lab

1:00pm Introduction

Don Goldmann

1:10pm Background on Evaluation of QI

Gareth Parry

1:50pm Designing Programs for Formative EvaluationAmy Reid

2:30pm Practical Exercise & Break

3:00pm Planning Measurement and Dissemination

Sandy Cohen

3:20pm A Practical approach to Rapid Cycle EvaluationSandy Cohen

3:50pm Stroke 90:10 A Practical ExampleGareth Parry

4:25pm Wrap up

4

12/7/2014

3

Background to Evaluation

Gareth Parry


December 7, 2014

1:10-1:50pm

Aims of the Learning Lab

To describe what types of evaluations are most suited for improvement initiatives.

– Formative Theory-Driven Evaluation

To describe the questions an evaluator can ask to clarify how initiatives aim to bring about improvement.

To describe the data an evaluator can collect to understand the progress of an improvement initiative.

To describe the analytic approaches an evaluator can use to identify what was learnt during an improvement imitative.

To complete an Evaluation form

6

12/7/2014

4

Evaluation Workbook

A blank template outlining each assignment – to be completed by you!

Example forms for you to refer to

7

Background to Evaluation

Gareth Parry


December 7, 2014

1:10-1:50pm

12/7/2014

5

Lost in Translation?Cynefin:

Welsh: A place where a person feels they ought to live. It is where nature around you feels right and welcoming.

Meraki:

Greek: Pouring yourself wholeheartedly into anything such as cooking and doing so with soul, creativity and love.

9

• Some concepts can be expressed in a single word in one language while others require a detailed description

• Some things are just lost in translation

• What do we mean by?:Quality ImprovementEvaluation

Lost in Translation: An Illustrated Compendium of Untranslatable Words from Around the World

September, 2014, by Ella Frances Sanders

Sir Alexander Fleming

Ernst B. Chain Sir Howard Florey

The Nobel Prize for Medicine was awarded in 1945 to:

10

Penicillin

12/7/2014

6

Penicillin11

1928 1930s 1939 1940s 1945

Sir Henry Harris at the Florey Centenary lecture (1998):

– “Without Fleming, no Chain;

– without Chain, no Florey;

– without Florey, no Heatley;

– without Heatley, no penicillin.”

In other words:

– Without Fleming, no innovation;

– without Chain and Florey, no testing;

– without Heatley, no wide scale use of penicillin.

12

Penicillin

12/7/2014

7

Chat to your neighbor

Can you think of improvement-related examples that have moved through

innovation, testing, and spread & scale-up?

Why Many Improvement Initiatives Are Found to “Fail”


December 7, 2014

12/7/2014

8

Conclusions: A multipayer medical home pilot, in which participating practices adopted new structural capabilities and received NCQA certification, was associated with limited improvements

in quality and was not associated with reductions in utilization of hospital, emergency department, or ambulatory care services or total costs over 3 years. These findings suggest that medical home interventions may need further refinement.

(4) Friedberg, MW, et al. (2014). Association between participation in a multi-payer medical home intervention and changes in quality, utilization, and costs of care, Journal of the American Medical Association.

(5) Urbach, DR, et al. (2014). Introduction of Surgical Safety Checklists in Ontario, Canada, New England Journal of Medicine.

15

Mark W. Friedberg et al. February 25, 2014

Conclusions: Implementation of surgical safety checklists in Ontario, Canada, was not associated with a significant reductions in operative mortality or complications.

Donald T. Campbell

“The United States and other modern nations should be ready for an experimental approach to social reform…in which we learn whether or not these programs are effective, and in which we retain, imitate, modify or discard them on the basis of their apparent effectiveness on the multiple imperfect criteria available.”

16

(1) Donald T. Campbell (1969) Reforms as experiments. American Psychologist, 24(4): 409-29.

12/7/2014

9

Peter H. Rossi: The “Iron Law” of Evaluation

“The expected value of any net impact assessment of any social program is zero. This means that our best a priori

estimate of a net impact assessment of a program is that it will have no effect.”

(2) Peter H. Rossi (1987) The iron law of evaluation and other metallic rules. Research in Social Problems and Public Policy, 4: 3-20.

“… the data Ioannidis found were disturbing: of the thirty-four claims that had been subject to replication, forty-one per cent had either been directly contradicted or had their effect sizes significantly downgraded.”

17


Where have you seen Rossi’s Iron Law in effect?

12/7/2014

10

Innovation to Prototyping: Small Number of Settings

Applied in a narrow range of contexts

Improvement in 100% of sites

19

Initial Testing:Small Number of Settings

Applied in a wider range of contexts


20

12/7/2014

11

More Settings as Range of Contexts Begins to Expand

Applied in a wider range of contexts


21

Wide Range of Contexts

Applied in a wide range of

contexts


22

12/7/2014

12

Reduction in Effectiveness from Applying Same Fixed-Protocol Program in Different Contexts

Innovation sample

23

Innovation sample

Evaluation sample

Immediate wide-scale implementation

24Reduction in Effectiveness from Applying Same Fixed-Protocol Program in Different Contexts

12/7/2014

13

Where Can Protocol Be Amended to Work25

Identify contexts in which it can be amended to work as we move from Innovation to Prototype to Test and Spread

Innovation sample

Core Concepts & Detailed Tasks

MEWS >=5Use a reliable

method to identify deteriorating

patients in real time.

When a patient is deteriorating,

provide the most appropriate

assessment and care as soon as

possible

MEWS >=4

2 Nurses1 Physician

1 Nurse1 Physician

1 Physician

ActionTheory

Core Concepts Detailed Tasks and Local

Adaptations

12/7/2014

14


Think about the Surgical Checklist – what do you think some of the “Core Concepts” are?

Improvement Stages and Degree of Belief

Gareth Parry


December 7, 2014

12/7/2014

15

Degree of Belief

ActEvidence

29

Degree of Belief in Change Ideas

degre

e o

f belie

f

InnovationGenerate/discover new

models of care with evidence of

improvement in a small number of settings.

TestingTest whether a model

works or can be amended to work in specific contexts.

Scale up and Spread

Implementation of models shown to apply in a broad

range of contexts.

High

Moderate

Low

12/7/2014

16

Basic Evaluation Approaches


December 7, 2014

Summative Evaluation

Aim Assess program impactInterpret results

How are study results used?

Decide if program was successful and can be generalized widely

Typical components

Quantitative dataHypothesis & theory testingEstimation

Commonexamples

Cost-benefit analysisComparative effectiveness

32


12/7/2014

17

Conclusions: A multipayer medical home pilot, in which participating practices adopted new structural capabilities and received NCQA certification, was associated with limited improvements

in quality and was not associated with reductions in utilization of hospital, emergency department, or ambulatory care services or total costs over 3 years. These findings suggest that medical home interventions may need further refinement.

(4) Friedberg, MW, et al. (2014). Association between participation in a multi-payer medical home intervention and changes in quality, utilization, and costs of care, Journal of the American Medical Association.

(5) Urbach, DR, et al. (2014). Introduction of Surgical Safety Checklists in Ontario, Canada, New England Journal of Medicine.

33

Mark W. Friedberg et al. February 25, 2014

Conclusions: Implementation of surgical safety checklists in Ontario, Canada, was not associated with a significant reductions in operative mortality or complications.

Summative Evaluation Formative Evaluation

Aim Assess program impactInterpret results

Improve programs as theyevolve

How are study results used?

Decide if program was successful and can be generalized widely

Amend or replicate the program in the future

Typical components

Quantitative dataHypothesis & theory testingEstimation

Mixed & qualitative methodsHypothesis generationTheory formation

Commonexamples

Cost-benefit analysisComparative effectiveness

Implementation evaluationRealist Evaluation

34


12/7/2014

18

1) Generating the pressure (will) for ICUs to take part2) A networked community3) Re-framing BSIs as a social problem4) Approaches that shaped a culture of commitment5) Use of data as a disciplinary force6) Hard edges

(4) Milbank Quarterly, 2011

35

From an Improvement Perspective:

Initial Concepts

Concepts rather than fixed protocols are a good starting point for people to test and learn whether improvement interventions can be amended to their setting.

Social Change

Improvement requires social change and that people are more likely to act if they believe.

Context Matters

Interventions need to be amended to local settings (contexts).

Learning Learn what is takes to bring about improvement.

We need: ‘Theory-driven rapid-cycle formative evaluation’

12/7/2014

19

Five Core Components37

Core Component

1) GoalsAim Statement

2) Content TheoryDriver Diagram or Change Package

3) Execution TheoryLogic Model

4) Data Measurement & LearningMeasurement Plan

5) DisseminationDissemination Plan

The results and learning derived from the evaluation of an

improvement initiative can be clearly

communicated.

That will maximize the chances that

Five Core Components: The Model for Improvement38

Core Component






Model for Improvement Questions

What are we trying to accomplish?

What changes will we make that will result in improvement?

How will we know that a change is an improvement?

12/7/2014

20

Five Core Components: Lean A339

Core Component






Background: Why are you talking about it?

Lean A3 Questions

Current Situation: What is going on?

Goal: Where do we need to be?

Analysis: What is the root cause(s) of the

problem?

Recommendations: What is your proposed

countermeasure(s)?

Plan: What activities will be required for

implementation and who will be responsible

for what and when?

Follow-up: How will we know if the actions

have the impact needed? What remaining

issues can be anticipated?

Designing Programs for Formative Evaluation

Amy Reid

1:50-2:30pm


December 7, 2014

12/7/2014

21


Core Component






Core Component 1: Aim Statement

“Aspirational aims” are stretch goals used to inspire.

- Examples: 100,000 Homes Campaign; Equitable care for all

“Achievable goals” are measurable targets believed to be achievable during a project’s timeframe.

- Answers “What are we trying to accomplish?”

- Captured in “aim statements”

42

12/7/2014

22

Constructing an Aim Statement

“Hope is not a plan;

Some is not a number;

Soon is not a time”

43

A strong aim statement includes:

• Numeric goals for outcomes (how much)

• Timeframe (by when)

• Boundaries of the system being improved (where, who)

Example: By Oct. 1, 2015, Most Excellent Health System will reduce

hospital readmissions by 25 percent through engaging 10 cross-

continuum teams of hospital, primary care, and post-acute providers.

Aspirational Aim v. Achievable Goal

“Aspirational” aims for project motivation need to be

distinguished from realistic goals achievable in the timescale of a project

(11) Nanji et al. (2012) Overarching goals: a strategy for improving healthcare quality and safety? BMJ Qual Saf

(12) Lilford et al. Evaluating policy and service interventions: framework to guide selection and interpretation of

study end points. BMJ 2010

44

Example:

Safer Patient Initiative:(9) Benning et al. BMJ 2011: 14% reduction odds of ICU mortality (p=0.250)Study powered to detect a big difference

Keystone:(10) Lipitz-Snyderman et al. BMJ 2011: 11% reduction in odds of ICU mortality (p=0.033)Study powered to detect a smaller but still clinically meaningful difference

12/7/2014

23


What examples of aspirational aims do you know of?

Kirkpatrick Framework

Kirkpatrick Evaluation of Learning Framework has four levels:

1) Experience What was the participants’ experience? Did the participants have an excellent experience working on the improvement project?

2) Learning What did participants learn? Did they learn improvement methods and begin testing?

3) Process/Behavior Did participants modify their behavior? Did they work differently and see change in their process measures?

4) Outcomes Did the organization improve its performance (via outcome measures)?

46

12/7/2014

24

Sample Goals47

1) Experience Throughout the program, 80 percent of initiative participants rate program activities as “very good” or “excellent”

2) Learning After one year, 75 percent of participants will increase knowledge of pre-natal, intra-partum, and neonatal best practices.

3) Process & Behavior

After two years, achieve 90% compliance with best-practice bundles for: Pre-natal care; Post-partum maternal care; Post-partum neonatal care.

4) Outcomes Over two years, decrease neonatal mortality within our system by 20%.

Core Components 2 & 3

Value of a theory

Content Theory & Driver Diagrams

Execution Theory & Logic Models

12/7/2014

25

Program Theory Defined

A conceptual explanation of how we believe the program, as designed, will result in the desired improvement.

A program theory has two major components:

– Content Theory

– Execution Theory

Summary

A well described theory:

– Provides a common understanding of the intention of the model.

– Allows an evaluator to understand the core concepts that underpin the more detailed tasks carried out by those at the front lines.

– Encourages testing and local amendment, informed by underlying principles.

– Shows how all the pieces of a program are related.

– Provides a high-level summary that can be the starting point for testing in other settings.

Rossi PH, Freeman HE, Lipsey MW. Evaluation: A Systematic Approach, 7th Edition. Sage, California 2004.

Ovretveit J, Leviton L, Parry GJ. Increasing the generalisability of improvement research with an improvement replication programme. BMJ Qual Saf 2011;20:i87-i91.

12/7/2014

26

Kirkpatrick Framework

1) Experience What was the participants’ experience? Did the participants have an excellent experience working on the improvement project?

2) Learning What did participants learn? Did they learn improvement methods and begin testing?

3) Process/Behavior Did participants modify their behavior? Did they work differently and see change in their process measures?

4) Outcomes Did the organization improve its performance (via outcome measures)?

51

Inputs

Resources invested and

activities performed by

each improvementagent/entity

Level 1Participant Experience

Level 2Learning

Level 3 Process/Behavior Changes

Level 4Clinical, personal,

organizational outcomes

Content Theory:What changes will teams make that will result in improved outcomes?

Execution Theory:What will the improvement initiative do that will lead teams to adopt the process changes?

Parry et al. Recommendations for Evaluation of Health Care Improvement Initiatives, 2013,

Acad Peds.

12/7/2014

27

Cargo Cult Quality Improvement

From a reference to a 1974 commencement address by Richard Feynman at Caltech, who said: “In the South Seas there is a Cargo Cult of people. During the war they saw airplanes land with lots of good materials, and they want the same thing to happen now. So they’ve arranged to make things like runways, to put fires along the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head like headphones and bars of bamboo sticking out like antennas—he’s the controller—and they wait for airplanes to land. They’re doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn’t work. No airplanes land. So I call these things Cargo Cult Science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential, because the planes don’t land.”

Dixon-Woods et al. Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Q. 2011; 89(2):167-205

Content Theory54

The rationale for how changes in processes will improve organizational performance or patient outcomes.

Parry et al. Recommendations for Evaluation of Health Care Improvement Initiatives, 2013,

Acad Peds.

12/7/2014

28

Rapid Response Teams (RRTs)

RRTs aim to provide care to deteriorating patients, to reduce the likelihood of ICU admission and mortality.

Rapid Response Team 1 Rapid Response Team 2

Monitor all patients in the acute care setting using the Modified Early Warning Score (MEWS).

When a patient scores 5 or more on MEWS, this indicates the patient is deteriorating.

Call for a Medical Emergency Team (MET) consisting of two ICU nurses and a physician to examine the patient within 10 minutes.

Monitor all patients in the acute care setting using the Modified Early Warning Score (MEWS).

When a patient scores 4 or more on MEWS, this indicates the patient is deteriorating.

Call for a Medical Emergency Team (MET) consisting of one ICU nurse and a physician to examine the patient within 20 minutes.

Core Concepts & Detailed Tasks

MEWS >=5Use a reliable

method to identify deteriorating

patients in real time.

When a patient is deteriorating,

provide the most appropriate

assessment and care as soon as

possible

MEWS >=4

2 Nurses1 Physician

1 Nurse1 Physician

1 Physician

ActionTheory

Core Concepts Detailed Tasks and Local

Adaptations

12/7/2014

29

Content Theory ExamplesHigh-Level Content Theory Outcome

Patients that receive a chlorhexadine bath, nasal staphylococcus aureus screening, decolonization (when positive), and surgical site antiseptic wipe are less likely to experience an infection from THKA procedure.

Reduce infections from total hip and knee arthroplasty (THKA)

Flag reminders in EHR system for patients due for a mammography and colonoscopy to reach out proactively, call with reminders before appointment, and provide travel vouchers when needed

Reduce racial inequities in screening rates

Implementation of three evidence-based approaches as a bundle will reduce the incidence of ventilator acquired pneumonia.

Lower incidence of ventilator acquired pneumonia

Reliable implementation of the “Transitional Care Model” will reduce hospital re-admissions.

Reduction in hospital re-admissions.

Driver Diagram

• QI tool used to outline the system that underlies the process or outcome you want to improve.

• This ‘map’ helps improvement teams identify change concepts that are likely to impact the result of interest.

• Three components:

• Aim

• Primary Drivers

• Secondary Drivers

12/7/2014

30

PRIMARY DRIVERS SECONDARY DRIVERSAIM

Increased use of tele-health services

Reduce death,

disability, or

other

preventable

harm among

newborns and

mothers across

six-district

service area

Staff clinical knowledge and skills

Adequate supply of necessary materials

Use of data for decision making

Expand access

to pre-natal

care

Activate

community

members to

promote health

and connect

peers with

health services

Reliable

delivery of pre-

natal care

bundle

Supportive community structures

Attractiveness of Health Services

Communication & transportation

Remove cost barriers

Spread information to communities

Identify and train community leaders

Open lines of communication

Community-based education

Driver Diagram: Example

Execution Theory60

Inputs

Resources invested and

activities performed by each

improvementagent/entity

Level 1Participant Experience

Level 2Learning

Level 3 Process/Behavior Changes

Execution Theory:What will the improvement initiative do that will lead teams to adopt the process changes?

12/7/2014

31

Execution Theory Examples

High-Level Execution Theory Outcome

• bringing QI teams together in person three times over 18 months

• virtual community over 1 year• webinars to teach updated clinical content and QI methods • provide means to share what works and what doesn’t from local

tests• coaching on small tests of change and data use for

improvement• requirement of data submission• engaging leadership in supporting a culture of QI• materials accessed online• listserv to share resources• evidence simplified in to actionable tools• creating national pressure for improvement

Improved reliability of specified processes

1) Generating the pressure (will) for ICUs to take part2) A networked community3) Re-framing BSIs as a social problem4) Approaches that shaped a culture of commitment5) Use of data as a disciplinary force6) Hard edges

(4) Milbank Quarterly, 2011

62

12/7/2014

32

Logic Models

1) What are they?

2) How do I develop one for my improvement project?

3) How can I use one to evaluate my work?

What Are They?

Diagram connection between what we want to accomplish and the way we plan to do that

Teams use them to:– Lay out plan

– Expose gaps

– Get multiple parties on the same page

– Increase intentionality

– Identify measures of interest

– Outline evaluation plan

12/7/2014

33

The Basic Logic Model: “If-Then”

Mid-term Outcomes

5

Ifparticipants start testing,

thenparticipants will change

their behavior/ processes

Long-term Outcomes

6

If participants change

processes, then you will see changes in patient and organizational

outcomes

Resources/Inputs

Certain resources

are needed to operate

your program

1

Activities

If you have access to

them, thenyou can

use them to

accomplish your

planned activities

2

Outputs

If you accomplish

your planned activities,

then you will deliver the intended

amount of product/

service to the intended audience

3

Short-term Outcomes

If you accomplish

your planned activities to the extent

you intended, then

participants will see

changes in knowledge and start testing

4

Execution Theory Content Theory

Source: Adapted from WK Kellogg Foundation, Logic Model Development Guide

What are you doing?

e.g., training in QI, monthly data analysis calls

What changes in organizational and patient outcomes do you expect ultimately?

e.g., 20% decrease in under-5 mortality

What changes in learning do you expect to come from your activities and outputs?

e.g., teams complete PDSA cycles

What are the reach and products of the activities?

e.g., 3 QI teams trained in improvement methods

What changes in processes do you expect as a result of your short-term outcomes?

e.g., blood protocol implemented with 80% reliability

What are you investing to get the work done?

e.g., staff, funding, space, external partners, materials, tools

External factors

What factors outside of the project may be a barrier or facilitator to reaching your desired outcomes?

e.g., new incoming government that prioritizes x, high staff turnover in health centers

Assumptions

What are you assuming about the ability to deliver the program in the above planned way?

e.g., leadership is on board and the will is strong, surgeons will make the time to attend trainings and lead their QI team

Project Title (Month Year – Month Year)

Context

What is the 2-3 bullet elevator speech? What background info is necessary to understand this project ?

Inputs Activities Outputs Short term outcomesMedium term

outcomes Long term outcomes

41 2 3

12/7/2014

34

Medium Term

In participating hospitals, >90% adoption of:

Preoperative bathing or showering with chlorhexidine

Preoperative a)nasal screening for Staphylococcus aureus carriage followed by b) decolonization of S. aureus carriers

Preoperative skin preparation w/a long-acting antiseptic agent in combination w/alcohol

Long Term

Reduce surgical site infections in hip & knee patients in 10 U.S. states

Short Term

IHI: Establish & Support Rapid Spread Network

Node- Recruit hospitals-Strengthen/ develop relationships with & among hospitals-Increased capacity to coach hospitals-Create links to related initiatives- Track hospital progress- Link hospitals to IHI resources- Provide feedback- Identify exemplar hospitals

Hospital- Aims set; teams formed- Access and use tools- Increased knowledge of importance of bundle- Leadership buy-in- Increased surgeon, admin staff, lab, & patient buy-in- Implement and test new processes & tools- Learn from other hospitals- Provide feedback- Develop improvement tools; share w/other facilities- Customized implementation (e.g., Pre-ops joints class

External factors: - Other SSI focused projects e.g. Partnership for Patients detracted attention and also prioritized reduction of SSIs- Shifting payment mechanisms for reimbursement for SSIs

Assumptions: SSI reduction is a priority; IHI & nodes are a trusted source of content; to achieve goal must effectively recruit and then engage to support flexible implementation; developing an infrastructure for teams to learn from each other will support successful implementation

Activities

Recruitment-recruit nodes, hospitals, national partners-build/strengthen state-level relationships-states assigned to cohort 1 or cohort 2

Ongoing Development & Refinement of Content Materials -understand and summarize evidence of interventions-gather existing materials from early adopters -continuous development & refinement of materials to guide the work

Support Implementation- Build infrastructure where nodes & hospitals can share learning and adapt implementation to their setting: in-person, phone, and web-based support

Revise activities and approach based on learning from cohort 1

Outputs

Nodes in 10 states recruited & assigned to cohorts; hospitals recruited for participation; relationship & support letters from national partners

Tools and communication structure for nodes and hospitals that support implementation• IHI in-person visits • How-to guide• improvement tools• patient/family 1-pager• measurement tools• node meetings at

2011 and 2012 forum• monthly node calls• webinar call series• state-specific calls• electronic

communications (website, listserv, email)

Use of IHI’s Rapid Spread Network to Reduce Hip & Knee Surgical Site Infections in Ten States in the U.S. (Sept 2010 – Oct 2012)

Inputs

$500,000 funding

Project & communication management team: PC, PM, communications, director, field manager, clinical director

Content experts: Surgical, infection control, nursing, and improvement faculty from IHI

Network of state nodes, hospitals & national partners

Experience of sites that have done this before

Communication & technology infrastructure

Learning from campaign tools

Evidence that is ready for spread

Context

3yr Federal grant funded project designed in two waves to work with all hospitals in 10 states that do hip/knee replacements to reduce hip and knee SSIs

The project is designed to use IHI’s strong relationships w/existing nodes and hospitals and nodes’ strong relationships w/hospitals to move work forward

Aspirational

aim

Content Theory

Execution Theory

Primary Goal

Inputs ActivitiesShort-Term

Outcomes

Collaborative

OutcomesImpact

Engage each hospital’s

leadership in developing

project charter: aims,

connections to extant key

projects/strategies, cycle for

review of progress, senior

sponsor.

Establish network of QI

support across hospitals

(Extranet, common rules for

run chart displays)

Develop Content DD, change

package, measures

developed, used and

reviewed

Promote listserv dialog

Team coaching individual

coaching as needed

Carry out Learning sessions

Carry out regular monthly

Action Period calls

Teams maintain PDSA logs

and PPT storyboards

Steering Committee

reinforces system

commitment (periodic public

report to network?)

Monthly reporting &

feedback cycle

Establish, maintain Extranet

site for data sharing

Faculty and teams

customize learning

Teams are actively and

transparently engaged

in their plan

Kirkpatrick Level 3:

By 31 Dec 2013, 80%

of teams reach (1)

Nurse and Physician

communication process

reliability goals;

(2) Leadership

engagement

performance

demonstrated


By March 2013, 80%

of teams testing

changes


By April 2014,

80% of teams

demonstrate

improvement in

HCAHPS Nursing,

Physician

Communications and

Responsiveness

Patient Experience Collaborative Logic Model

Improved

HCAHPS scores

(relative to secular

trends)

Integration of

improved patient

experience into

standard practice

Teams have will and

resources

Partner:

• Funding

• Contract with IHI

• Project Steering

Committee

• Commitment and

active involvement by

hospital level leadership

• Hospitals’ QI

• Expertise

• Support for proper

team selection

• Connect to Leadership

Academy

Improvement org:

• Hospital Capacity

Assessment

• Patient Experience

Drivers and Key

Actions

• Content from prior

collaboratives

• Team structure

and processes

• Core team and faculty

expertise

• Listserv

• Evaluation support

Project infrastructure

supports teams

Teams are contributing

to an active learning

community

12/7/2014

35

Using Logic Model for Evaluation

Road map for what to evaluate

Assess fidelity to the intervention

Identify breakdowns in theory

Call out key items of interest

Link to Kirkpatrick levels

Come back to it to refine theory over time


How have you used program theories in your work?

12/7/2014

36

Exercise (25 minutes)

From the Evaluation Workbook:

On your table, work together to answer questions 1-3:

1) What are your Goals

• Draft an aim statement

2) What is your Content Theory?

• Draft a driver diagram

3) What is your Execution Theory?

• Draft a logic model

71

Break

Back at 3:00pm

12/7/2014

37

Getting Started: Measurement & Dissemination Plans

Sandy Cohen

Session 2

3:00 – 3:20pm

Web&ACTION: Rapid-Cycle Evaluation for Health Care Improvement Programs


Core Component




4) Data, Measurement & LearningMeasurement Plan


12/7/2014

38

Measurement & Evaluation Plan

How will know a change is an improvement?

• Plan suite of measures

• Design system for data collection and reporting

• Analyze data

• Synthesize knowledge

Measurement & Evaluation Plan

• Do we have measures for all goals?

• When and how will various data be collected?

• How often will data be analyzed? By what methods?

• How will we use data to inform course corrections?

• Set learning objectives

12/7/2014

39

Sample Measures77

KL Sample Goal Aligned Measures

4 Over two years, decrease neonatal mortality by 20% in three-district area.

Over two years, reduce total adverse birth events (maternal or neonatal) by 30%.

After two years, improve access to pre-natal care for mothers delivering in public hospitals by 30%.

Rate of stillbirths plus neonatal deaths related to childbirth per 1,000 deliveries

Rates of low-birth weight, birth asphyxia, and all-cause admissions to NICU per 1,000 deliveries

Percentage of births across three pilot districts for which mother received any pre-natal care

3 Use of practice bundles for pre-natal maternal care and post-partum maternal & neonatal care will reach:

• 50% after year one

• 90% after year two

Monthly, randomized chart reviews to assess:

• % deliveries for which mothers received complete pre-natal care bundle

• % neonates receiving post-partum care satisfying all bundle criteria

2 Increase knowledge of pre-natal and post-partum best practices in:

• 75% of participants after year one

• 90% of participants after year two

Participants self-assess knowledge of best practices for pre-natal/neonatal care and report changes they have tested. Self-assess at baseline and every 6 months during initiative.

1 Throughout the program, 80% of participants rate overall program as “very good” or “excellent”

Participants asked to “rate the overall quality of your experience in this initiative,” using a five-point Likert scale (poor, fair, good, very good, excellent).

Sample Measurement Strategy78

Measure Operational Definition Data Collection Plan

Urinary

catheter-

associated

urinary tract

infections rate

Numerator: # patients with a urinary catheter who are being treated for a UTI arising from their current inpatient stay

Denominator: the total number of indwelling urinary catheter days

Calculate rate by numerator/denominator,then multiplying the result by 1000 to create the CAUTI rate per 1000 catheter days

• Include all patients with a urinary catheter who have a newly diagnosed UTI where the definition for a CAUTI is met per local guidelines.

Incidence of

inpatient

hospital-onset MRSA

Numerator: # of all unique blood source laboratory-identified (LabID) events identified >3 days after admission to the facility.

Denominator: Patient days

• After entering 100 cases, conduct data entry audit on 20% of them at random. If errors exceed 50%, it is advisable to repeat data entry for the 100 cases

Pressure ulcer count

This is a cumulative count of the number of pressure ulcers acquired within the general ward setting.

• Patients should be assessed for pressure ulcers on a daily basis. Each new case of patient pressure ulcer should be recorded.

• All patients in unit will be assessed for pressure ulcers. No sampling needed.

12/7/2014

40

Dissemination Plan

• The message

• Target audience(s)

• Outputs(s) for reaching audience

• Initial work plan

• Resourcing

The Message

Woodrow Wilson, was asked by a member of his cabinet about the amount of time he spent preparing speeches.

“It depends. If I am to speak ten minutes, I need a week for

preparation; if fifteen minutes, three days; if half an hour,

two days; if an hour, I am ready now.”

12/7/2014

41

The Message

• Results

• Learning

• Improvement journey

• Methods (theory and analysis)

• Discussion/Recommendations

– Implications of your findings for others’ work

– Share new theory and suggest future projects

81

Target Audience

Internal

• Colleagues doing similar work

• Leadership

• Communications team

• Central reporting system

External

• Segments of health professionals

• Specific organizations, stakeholders, individuals

12/7/2014

42

Outputs

Considering message, target audience, and resources –what are the best methods for dissemination?

• Peer-review publication

• Grey literature

• Short informal articles (e.g. blogs)

• Professional presentations (e.g. conferences)

• Radio/Press

• Multi-media

• Project summary report

• Meeting with stakeholders (internal and/or external)

Work plan & resourcing

Work Plan

• What is the timeline for certain outputs?

• Who will lead development?

Resourcing

• Consider all who will be involved

• Budget time in advance!

84

12/7/2014

43

SQUIRE: Standards for Quality Improvement Reporting Excellence

85

(1) Davidoff F, Batalden P, Stevens D, Ogrinc G, Mooney S. (2008). Publication guidelines for quality improvement in health care: evolution of the SQUIRE project. Qual Saf Health Care, 17(Supp 1), i3-i9.

http://www.squire-statement.org/

Updated SQUIRE Guidelines will be published in summer 2015

How do you plan for communicating results and learning from of your

improvement work?

Chat with your neighbors

12/7/2014

44

Practical Approach to Rapid-Cycle Evaluation

Sandy Cohen

3:20 – 3:50pm


December 7, 2014

Overview

Evaluation across project stages: Design, Improvement, and Close-out

– Describe each stage

– Evaluation objectives

– IHI approach

– Project example

88

12/7/2014

45

Three QI project stages

ImprovementDesign

Period spanning the initial concept, iterative design, and lead-up to project launch.

Close-out

Current iteration or phase is complete

Summative analysis of final results

QI activities are underway

Consistent data reporting and real-time analysis for local tests of change

89

Clarify project’s 5 core components

Plan formative evaluation system

Compare progress to predictions and goals

Amend program theory based on results & learning

Finalize new program theory

Disseminate new theory, results and learning

Descri

pti

on

Ev

alu

ati

on

Aim

Project Design


December 7, 2014

12/7/2014

46

Evaluation as a system

Design

Clarify core componentsPlan eval.

system

Improvement activities begin

91

Inputs

Goals, Program theory, Measures, Eval Plan, Dissem.

Improvement


December 7, 2014

12/7/2014

47


Design

Clarify core components

Improvement

Study and Amend core components


Review A Review B Review CINPUTSProgram TheoryGoals, MeasuresEvaluation PlanDissem. Plan

93

Project

Milestone Reviews at IHI

Who: Project Management, Content Experts, Senior Sponsor, Improvement Advisor, Evaluation Associate, Regional Lead, Focus Area Lead

What: Suggest 90-minute call

Where: IHI office with virtual attendees

When: Quarterly and aligned with project milestones

Why: To pause and reflect on progress, data, and learning to make amendments to theory and support ongoing redesign. To zoom out.

94

12/7/2014

48

Results Report

Regular project updates to track progress, learning

Key elements:

– Recent activities: Executing as planned?

– Latest results, progress towards goals

– Qualitative data providing context for results

– Changes to program theory

– Dissemination activities

95

Objective

Obtain 90% achievement:Among 60% of children in oral comprehension

Among 80% of children in writing

Among 90% of children in vocabulary

Over 1 school yearamong schoolsparticipating in UBC

Regular school

attendance.

Goal: Reduce by 20% or

more the % of children

who are absent 2+ days

in 2 weeks

Primary Drivers

Time dedicated to

instruction of language

Goal: one hour daily

Quality of teaching

Goal: Achileve a CLASS

‘medium’ level of

quality in instruction.

Self-regulation

Goal: Increase by 20%

the children who

achieve self-regulated

behavior

Secondary Drivers

Feedback from School Leadership. (ED)

Observation of & reflection on classroom practice. (ED)

Appropriate climate conditions in school (ED)

Reduction of chaotic and non-instructional time

Organized and stimulating classroom environment

Available transportation (ED)

Reduction of non-instructional time

Frequency of application of language strategies

Children’s motivation to attend

Effective group management by teacher

Prototypes for language activities

Increase peer-to-peer learning opportunities

Increase in self-regulated activities

Un Buen Comienzo (UBC): Phase 3Key Driver Diagram, 2013

96

12/7/2014

49

Objective

Obtain 90% achievement:Among 60% of children in oral comprehension

Among 80% of children in writing

Among 90% of children in vocabulary

Over 1 school yearamong schoolsparticipating in UBC

Regular school

attendance.

Goal: Reduce by 20% or

more the % of children

who are absent 2+ days

in 2 weeks

Primary Drivers

Time dedicated to

instruction of language

Goal: one hour daily

Quality of teaching

Goal: Achileve a CLASS

‘medium’ level of

quality in instruction.

Self-regulation

Goal: Increase by 20%

the children who

achieve self-regulated

behavior

Secondary Drivers

Time dedicated to learning activities at home. (F)

Feedback from School Leadership. (ED)

Observation of & reflection on classroom practice. (ED)

Appropriate climate conditions in school (ED)

Reduction of chaotic and non-instructional time

Organized and stimulating classroom environment

Available transportation (ED)

Reduction of non-instructional time

Frequency of application of language strategies

Children’s motivation to attend

Parents’ motivation for children to attend (F)

Effective group management by teacher

Prototypes for language activities

Increase peer-to-peer learning opportunities

Increase in self-regulated activities

Un Buen Comienzo (UBC): Phase 3Key Driver Diagram, 2013

97

Formative Evaluation Challenges + Tips

Finding time to convene

– Plan milestone calls early as project sets up timeline

Build will for evaluation activities

– Clearly describe purpose; Demonstrate value with quick wins

Foster strong relationships

– Frame evaluation as service that eases work burden; Evaluators and project team co-design and co-lead strategy

Data for learning, not judgment

98

12/7/2014

50

Close-out


December 7, 2014


Design

Clarify core components

Improvement

Study and Amend core components

Close-out

Finalize new program theory


Improvement activities end

Review A Review B Review CINPUTSProgram TheoryGoals, MeasuresEvaluation PlanDissem. Plan

OUTPUTSRevised

Program TheoryDissemination

100

12/7/2014

51

Stroke 90:10Gareth Parry


December 7, 2014

Stroke 90:10

Set in the Northwest of England, aimed to improve compliance to

an average of 90% on each of nine processes by the 2010 English National Audit, from the baseline average of 72% in 2008.

The nine key processes were organized into two care bundles: one for early hours care (Bundle 1) and one for rehabilitation (Bundle 2).

We conducted a cluster randomized controlled trial, where 9 hospitals were randomly chosen to start first (intervention group), with the remaining 9 chosen to start one year later (control group).

A comparison of progress in compliance with both bundles during the first year allowed us to estimate the effectiveness of the Stroke 90:10 collaborative.

12/7/2014

52

(1) Power, M., et al. (2014).

103

(2) Power, M, et al. (2014). Did a quality improvement collaborative make stroke care better? A cluster randomized

trial, Implementation Science, 9(40). Full text available at: http://www.implementationscience.com/content/9/1/40.

Stroke 90:10 - Quantitative

Background

– Stroke can result in death and long-term disability. Fast and high-quality care can reduce the impact of stroke, but UK national audit data has demonstrated variability in compliance with recommended processes of care. Though quality improvement collaboratives (QICs) are widely used, whether a QIC could improve reliability of stroke care was unknown.

12/7/2014

53

Stroke 90:10

Methods

– Twenty-four NHS hospitals in the Northwest of England were randomly allocated to participate either in Stroke 90:10, a QIC based on the Breakthrough Series (BTS) model, or to a control group giving normal care. The QIC focused on nine processes of quality care for stroke already used in the national stroke audit. The nine processes were grouped into two distinct care bundles: one relating to early hours care and one relating to rehabilitation following stroke. Using an interrupted time series design and difference-in-difference analysis, we aimed to determine

whether hospitals participating in the QIC improved more

than the control group on bundle compliance.

Stroke 90:10

Results

– Data were available from nine interventions (3,533 patients) and nine control hospitals (3,059 patients). Hospitals in the QIC

showed a modest improvement from baseline in the odds of

average compliance equivalent to a relative improvement of

10.9% (95% CI 1.3%, 20.6%) in the Early Hours Bundle and

11.2% (95% CI 1.4%, 21.5%) in the Rehabilitation Bundle.

Secondary analysis suggested that some specific processes were more sensitive to an intervention effect.

12/7/2014

54

Stroke 90:10

Conclusions

– Some aspects of stroke care improved during the QIC, but the effects of the QIC were modest and further improvement is needed.

– The extent to which a BTS QIC can improve quality of stroke care remains uncertain. Some aspects of care may respond better to collaboratives than others.

Stroke 90:10: Small Multiples

12/7/2014

55


When communicating the results of your improvement work, how do you deal with variation across sites, units or providers?

Stroke 90:10 - Qualitative

(3) Carter, P, Ozieranski, P, McNicol, S, Power, M, Dixon-Woods, M. (2014). How collaborative are quality improvement collaboratives: a qualitative study in stroke care, Implementation Science, 9(32). Click here for full text.

110

Methods

• Interviewed 32 professionals from participating hospitals

• Focus group with the QIC faculty team

• Purposively sampled documents (e.g. reports, newsletters)

• Modified form of Framework Analysis

12/7/2014

56

Stroke 90:10

BACKGROUND:

– Quality improvement collaboratives (QICs) continue to be widely used, yet evidence for their effectiveness is equivocal. We sought to explain what happened in Stroke 90:10, a QIC designed to improve stroke care in 24 hospitals in the North West of England. Our study drew in part on the literature on collective action and inter-organizational collaboration. This literature has been relatively neglected in evaluations of QICs, even though they are founded on principles of co-operation and sharing.

Stroke 90:10

METHODS:

– We interviewed 32 professionals in hospitals that participated in Stroke 90:10, conducted a focus group with the QIC faculty team, and reviewed purposively sampled documents including reports and newsletters. Analysis was based on a modified form of Framework Analysis, combining sensitizing constructs derived from the literature and new, empirically derived thematic categories.

12/7/2014

57

Stroke 90:10

RESULTS: – Improvements in stroke care were attributed to QIC participation by

many professionals. They described how the QIC fostered a sense of community and increased attention to stroke care within their organizations. However, participants' experiences of the QIC varied. Starting positions were different; some organizations were achieving higher levels of performance than others before the QIC began, and some had more pre-existing experience of quality improvement methods. Some participants had more to learn, others more to teach. Some evidence of free-riding was found. Benchmarking improvement was variously experienced as friendly rivalry or as time-consuming and stressful. Participants' competitive desire to demonstrate success sometimes conflicted with collaborative aims; some experienced competing organizational pressures or saw the QIC as duplication of effort. Experiences of inter-organizational collaboration were influenced by variations in intra-organizational support.

Stroke 90:10

CONCLUSIONS:

– Collaboration is not the only mode of behavior likely to occur within a QIC. Our study revealed a mixed picture of collaboration, free-riding and competition. QICs should learn from work on the challenges of collective action; set realistic goals; account for context; ensure sufficient time and resources are made available; and carefully manage the collaborative to mitigate the risks of collaborative inertia and unhelpful competitive or anti-cooperative behaviors. Individual organizations should assess the costs and benefits of collaboration as a means of attaining quality improvement

12/7/2014

58

Stroke 90:10

IHI Blog: Learning from the Stroke 90:10 Collaborative

(4) Parry, G, & Goldmann, D. (2014). Learning from the Stroke 90:10 Collaborative, IHI Blog, April 2014). Post available at: http://bit.ly/1u2anUd

116

“Without this comparative evaluation, we may have been misled into thinking Stroke 90:10 had achieved a bigger impact than it had, and may not have identified valuable lessons on the experience of participants in the collaborative. In alignment with the Plan-Do-Study-Act methodology, the improvement field, including IHI, must now “Act” and consider how to apply the lessons learned in the future design of improvement initiatives.”

Gareth Parry April 22nd 2014

12/7/2014

59


What are the main lessons to the improvement field from STROKE 90:10?

Hot off the press…

(5) Khodyakov, D. et al. (2014). Project JOINTS: What factors affect bundle adoption in a voluntary quality improvement campaign? BMJ Qual Saf doi:10.1136. Full text available at: http://qualitysafety.bmj.com/content/early/2014/11/09/bmjqs-2014-003169.full

118

12/7/2014

60

Exercise (20 minutes)

From the Evaluation Workbook:

On your table, work together to answer questions 4-6:

4) Measurement & Evaluation Plan

• Draft a measurement plan

5) Dissemination Plan

• Draft a dissemination plan

6) Plan for Rapid-Cycle Evaluation

• Draft a plan to modify your theories

119

Wrap up

4:20 to 4:30pm


December 7, 2014

12/7/2014

61

Bibliography121

Campbell, DT (1969). Reforms as experiments. American Psychologist, 24(4), 409-29. Click herefor full text.

Rossi, PH (1987). The iron law of evaluation and other metallic rules. Research in Social Problems

and Public Policy, 4, 3-20. Search Google Scholar.

Parry, GJ, Carson-Stevens, A, Luff, DF, McPherson, ME, & Goldmann, DA (2013). Recommendations for Evaluation of Health Care Improvement Initiatives, Academic Pediatrics, 13(6S), S23-30.

Friedberg, MW, et al. (2014). Association between participation in a multipayer medical home intervention and changes in quality, utilization, and costs of care, Journal of the American Medical Association.

Urbach, DR, et al. (2014). Introduction of Surgical Safety Checklists in Ontario, Canada, New England Journal of Medicine.

Dixon-Woods, M., Bosk, C.L., Aveling, E.L., Goeschel, C.A., Pronovost, P.J. (2011). Explaining Michigan: developing an ex post theory of a quality improvement program. Milbank Quarterly. 89(2),167-205. Click here for abstract.

Dixon-Woods, M, et al. (2014). Did a quality improvement collaborative make stroke care better? A cluster randomized trial, Implementation Science, 9(40). Click here for full text.

Carter, P, Ozieranski, P, McNicol, S, Power, M, Dixon-Woods, M. (2014). How collaborative are quality improvement collaboratives: a qualitative study in stroke care, Implementation Science, 9(32). Click here for full text.

Benning, A., Dixon-Woods, M., Nwulu, U., Ghaleb, M., Dawson, J., Barber, N., et al. (2011). Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ. 342, d199. Click here for full text.

Bibliography122

Lipitz-Snyderman, A., Steinwachs, D., Needham, D.M., Colantuoni, E., Morlock, L.L., & Pronovost,

P.J. (2011). Impact of a statewide intensive care unit quality improvement initiative on hospital

mortality and length of stay: retrospective comparative analysis. BMJ. 342, d219. Click here for full

text.

Nanji, K.C., Ferris, T.G., Torchiana, D.F., Meyer, G.S. (2013). Overarching goals: a strategy for

improving healthcare quality and safety? BMJ Qual Saf. 22(3),187-93. Click here for abstract.

Lilford, R.J., Chilton, P.J., Hemming, K., Girling, A.J., Taylor, C.A., & Barach, P. (2010). Evaluating

policy and service interventions: framework to guide selection and interpretation of study end

points. BMJ. 341,c4413. Click here for full text.

Davidoff, F. (2014). Improvement interventions are social treatments, not pills, Annals of Internal

Medicine, 161(7), 526-527. Click here for abstract.

Landrigan CP, Parry GJ, Bones CB, Hackbarth AD, Goldmann DA & Sharek PJ. (2010). Temporal trends in rates of patient harm resulting from medical care, New England Journal of Medicine, 363, 2124-2134. Click here for abstract.

Benning A, Dixon-Woods M, Nwulu U, Ghaleb M, Dawson J, Barber N, et al. (2011). Multiple component patient safety intervention in English hospitals: controlled evaluation of second phase. BMJ. 342, d199. Click here for full text.

Benneyan JC, Lloyd RC & Plsek PE. (2003). Statistical process control as a tool for research and healthcare improvement, Quality and Safety in Healthcare, 12(6), 458-464. Click here for abstract.

rapid cycle evaluation for improvement leaders -...

Documents