Heterogeneity is not always noise

Frank Davidoff29 March 2012

Heterogeneity

Heterogeneity is not always noise

2

Difference or diversity in kind from other things

Oxford English Dictionary

The Heterogeneity Problem

Heterogeneity is not always noise

3

Heterogeneity:

You can’t live with it, and you can’t live without it

Heterogeneity is not always noise

4

Benefit from Drug X: entire treated population

Results from a standard clinical trial

RCTRx benefit:20%

Heterogeneity of treatment effect: main sources

Heterogeneity is not always noise

5

• Variation in outcome risk when the primary disease is untreated (mainly biological and behavioral variation)• Treatment-related harm• Competing risk • Direct treatment-effect modification

How summary results of trials can be misleading

Heterogeneity is not always noise

6

Control - event rate

Rx -event rate

RRR ARR NNT

Overall result 8 6 0.25 0.02 50

Average risk subjects 4 3 0.25 0.01 100

High risk subjects 20 15 0.25 0.05 20

Kent et al, Trials 2010;11:85

Heterogeneity is not always noise

7

Benefit from Drug X: high-risk patient subgroup

Results from a risk-stratified clinical trial

RCT Risk stratification

Rx benefit:50%

Heterogeneity is not always noise

8

Benefit from Drug X: specific high-risk patient

Real world results in a “usual” local care system

RCT Risk stratification

Hospital P

Rx benefit:15%

Heterogeneity is not always noise

9

Benefit from Drug X: specific high-risk patient

Real world results in a local care system that successfully supports changes

RCT Risk stratification

Hospital Q

Rx benefit:47.5%

QI Program

Change factor stratification

Heterogeneity is not always noise

10

Benefit from Drug X: specific high-risk patient

Real world results in a local care system that has trouble supporting changes

RCT Risk stratification

Hospital R

Net benefit:25%

QI program

Change factor stratification

Heterogeneity of improvement effect: main sources

Heterogeneity is not always noise

11

Improvement interventions:

• Consist of multiple components: hard to standardize; get mixed and matched

A multi-component improvement intervention: the Michigan central line infection study

Heterogeneity is not always noise

12

• Introduce checklists, prep carts, new skin antiseptic • Recruit advocates within the organization• Keep the team focused on goals• Create alliances with central administration to secure resources• Shift power relations (particularly with nurses)• Create social and reputational incentives for cooperating• Open channels of communication with units that face the same challenges• Use audit and feedback

Heterogeneity of improvement effect: main sources

Heterogeneity is not always noise

13

Improvement interventions:

• Consist of multiple components: hard to standardize; get mixed and matched

• Must first be absorbed and adapted: change in the process (easily shared, spread)

Heterogeneity of improvement effect: main sources

Heterogeneity is not always noise

14

Improvement interventions:

• Consist of multiple components: hard to standardize; get mixed and matched

• Must first be absorbed and adapted: change in the process (easily shared, spread)

• Are context-dependent: context can’t be “controlled out”

Heterogeneity of improvement effect: main sources

Heterogeneity is not always noise

15

Improvement interventions:

• Consist of multiple components: hard to standardize; get mixed and matched

• Must first be absorbed and adapted: change in the process (easily shared, spread)

• Are context-dependent: context can’t be “controlled out”

• Are unstable, by design: evolve in response to feedback (“reflexiveness”)

Change factor stratification: facilitators

An “ex post” theory of a quality improvement

program

Heterogeneity is not always noise

16

• Isomorphic pressures to join the project• Networked community with strong horizontal links• Reframing bloodstream infections as a social problem• Using interventions to shape a “culture of commitment”• Harnessing data as a disciplinary force• Using “hard edges”

Dixon-Woods et al, Milbank Quarterly, 2011;89:167-205

Change factor stratification: facilitators

The Model for Understanding Success in Quality

(MUSIQ)

Heterogeneity is not always noise

17

• Twenty-five contextual factors including:

- External environment

- Organization

- QI support and capacity

- Microsystem

- QI team

- Miscellaneous

Kaplan et al., BMJ Quality and Safety, 2012; 21:13-20

Change factor stratification: obstructers

Resisters and constipators affect infection prevention

efforts

Heterogeneity is not always noise

18

• Active resisters:

- Hospital personnel who actively and openly oppose various changes in practice

• Organizational constipators:

- Mid- to high-level executives who prevent or delay change actions through passive

resistance or control

Saint, et al, Jt Comm J Quality and Safety, 2009; 35:239-46

Change factor stratification: obstructers

Understanding rapid response systems

Heterogeneity is not always noise

19

• Boundaries between senior and junior staff:

- tend to reinforce existing power relations• Standardization of practice (including use of Early Warning Systems):

- tend to attenuate staff authority• “Downstream” response behavior, including handovers and referral across medical teams:

- are more problematical than early steps in response

Mackintosh et al, BMJ Quality and Safety, 2012:21:135-44

SUMMARY

Heterogeneity is not always noise

20

Heterogeneity is everywhere in medicine• It interferes with detection of causal relationships (noise)

AND• Is a vital source of information on individual risk (signal)

We need to use heterogeneity as a source of knowledge• More techniques like risk stratification, that can expose the effects of biological and behavioral variation on clinical outcomes• Better methods for exploring the effects of variation in social change factors on the outcomes of improvement interventions