The State of the Evidence The State of the Evidence

in Patient Safety in Patient Safety

Kaveh G. Shojania, MDCanada Research Chair in Patient Safety

and Quality Improvement

Department of Medicine

Sunnybrook Hospital

University of Toronto

• Concise, evidence-based reviews of over 75 specific patient safety practices

• Over 140,000 copies obtained since publication in 2001

Peri-operative beta-blockers to reduce cardiac complications of non-cardiac surgery

• 5 randomized trials at time of AHRQ report (2001)

– Total patients ~ 600 across all 5 trials

– Substantial benefit: 1 major event averted for every 4-8 patients treated

Received 2nd highest evidence rating in AHRQ report

• Meta-analysis of 21 trials subsequently showed questionable benefit and increases in harm (2005)

• Recent mega-trial showed increased total mortality (Devereaux et

al. Lancet 2008)

Shojania et al. Making Healthcare Safer. 2001.

Other Examples of Major Changes or Starkly Conflicting Evidence

• Universal MRSA screening

• Hip protectors to prevent fall-related injuries

• Supplemental oxygen to decrease postoperative

infections

• Acetylcysteine to prevent contrast-nephropathy

• Rapid response teams

• ‘Survival analysis’ of 100 meta-analyses reviewed in ACP

J Club

• Major qualitative or quantitative changes in evidence

occurred for 23% within 2 years and for 15% within 1 year

–7% were already out of date at time of publication

• Multifaceted interventions (13 studies) showed a borderline

significant reduction in falls but not fractures

• No other strategy (hip protectors, removal of physical

restraints, fall alarm devices, changes in the physical

environment, medication review in hospital) showed

consistent, significant effects on falls, fallers, or fractures

“Safety of Patients Isolated for Infection Control” (Stelfox et al. Jama 2003)

Isolated patients twice as likely to experience adverse events (31 vs 15 adverse events per 1000 days; P<.001).

Included significant difference in preventable events (20 vs 3 adverse events per 1000 days; P<.001)Isolated patients also more likely to• have no vital signs recorded as ordered (51% vs 31%; P<.001)

• have days with no physician progress note (26% vs 13%; P<.001)

Catheter-related bloodstream infections showed a significant decrease from 0.62 (95% CI: 0.47 to 0.81) at baseline to 0.34 (95% CI, 0.23 to 0.50) at 18 months.

• missing data for approx 40% of ICU months

• CRBSIs definition open to bias

• no corroborating blood culture results

Main Arguments Against Needing Evidence in Patient Safety

• Too challenging: interventions are too complex to

study using conventional EBM paradigm

• Some interventions are self-evident

Villages…were pair-matched and randomly allocated to receive the intervention at study onset… or 3 years later…

Loans were provided to poor women who enrolled in the intervention group. A participatory learning and action curriculum was integrated into loan meetings, which took place every 2 weeks…

Cluster RCTs in Patient Safety/QI

• Rapid response teams – Hillman et al. Lancet 2005

• Teamwork training– Nielsen et al. Obstet Gynecol. 2007

• Computerized decision support– Eccles et al. BMJ. 2002

• Feedback of acute MI performance data to hospitals– Beck et al. Jama 2005

• PDSA and Chronic Care Model

– O’Connor et al. Diabetes Care. 2005

Without these trials, we would waste huge amounts of money on

ineffective (versions of these) interventions

“Parachute use to prevent death and major trauma related

to gravitational challenge: systematic review of

randomised controlled trials.”

Lethal errors involving injections of concentrated KCl

Remove concentrated

potassium from clinical areas

“forcing function” that prevents

errors from happening

Concentrated KCl resembles other iv

solutions

A simple, obviously beneficial patient safety intervention

Sobering example of hospital where delays in receiving KCl from Pharmacy resulted in surreptitious hording of KCl on wards

Increase, rather than

decrease in hazard

State of the Evidence in Patient Safety

• Superficial knowledge about epidemiology of many

important safety problems and their causes

• Few established, highly effective interventions

– most things either don’t work or we don’t know if they

work

• Those interventions that do work are often costly,

complex, and may even create new problems

Evidence vs. Action

• Robust evidence directly

informs a minority of clinical

situations.

often treat patients on basis

of anecdotal experience

– But we don’t turn these

practices into major

recommendations

• In patient safety, robust

evidence lacking for almost

everything

individual hospitals proceed

with promising strategies

– But widespread

dissemination will require

rigorous evaluation to

confirm benefit in wide

range of settings

Auerbach et al. NEJM. 2007

Suggested Framework for EvaluationBenefit self-evident?

Y N

RCT feasible?Monitor introduction

Y N

(cluster) RCT

Controlled Before-After Study or

Interrupted Time Series

Multivariate modelling

Y N

KCl removal

RRTs, Teamwork

P-4-PWork hour

reductions

High volume providers,

Staffing ratios

Prospective evaluation

Modified from R.J. Lilford

Conclusions

• Evidence in patient safety still fairly sparse

– but about what one would expect for a field this young

• Scant attention to implementation issues, unintended

consequences, and costs

• Need to distinguish levels of evidence to proceed locally with

a given intervention and that required for dissemination

Precisely because of complexity and contextual factors, we

need large-scale evaluations to determine what intervention

components required to achieve benefit in which settings