Effectiveness of Simulation in Healthcare

Joseph Barton, MD, MHMS

September 18, 2014

Desired Benefits of Simulation Training

• Reduce patient risk

• Broadly applicable and widely

available

• Consistent training and assessment

• Reduce costs

• Provide a clinical benefit

Kirkpatrick’s 4 Levels of Learning Evaluation

– Reaction

– Learning

– Behavior

– Results

Learning Evaluation

Level Description Evaluation

1 Surveys Did the trainee like it?

2 Assessments Did the trainee learn?

3 Transfer Is a behavior changed?

4 Benefit Is there a clinical outcome?

DOES SIMULATION-BASED EDUCATION WORK?

What is the Evidence?

• How effective is simulation as a modality?

• Number of recent meta analyses

– Virtual patients

– “Technology enhanced” simulation

• Virtual patients do not require special

equipment

• Standardized patients aren’t well studied

HOW EFFECTIVE IS SIMULATION AS A TEACHING MODALITY?

Effectiveness of Simulation

• More than 600 studies comparing

36K plus subjects

• Three recent large meta analyses

Effectiveness of Simulation

• Cook DA, Erwin PJ, Triola MM. Computerized virtual

patients in health professions education: a systematic

review and meta-analysis. Acad Med. 2010;85:1589-1602.

• Cook DA, Hatala R, Brydges R, et al. Technology-enhanced

simulation for health professions education: a systematic

review and meta-analysis. JAMA. 2011;306:978-988.

• Zendejas B, Brydges R, Wang AT, Cook DA. Patient

outcomes in simulation-based medical education: a

systematic review. J Gen Intern Med. 2013.

Effectiveness of Simulation

• Virtual patients and technology enhanced

simulation show consistent, large and statistically

significant benefits in

– Knowledge

– Instructor ratings

– Computer scores

– Patient care behaviors

Direct Patient Effects?

• Airway management, endoscopy, CVC insertion

studied

– Smaller benefits, still significant (major complications,

mortality, length of stay) when compared to no

intervention

– Non simulation instruction did not reach statistical

significance

Simulation based education is more effective than no instruction.

HOW DOES SIMULATION COMPARE TO OTHER INSTRUCTION?

Simulation vs other intervention

• Over 100 studies and 7K participants

• Challenging question due to different

simulation interventions and due to

variable comparison strategy

Simulation vs Other Intervention• Cook DA, Brydges R, Hamstra SJ, et al. Comparative effectiveness

of technology-enhanced simulation versus other instructional

methods: a systematic review and meta-analysis. Simul Healthc.

2012;7:308-320.

• Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness

of instructional design features in simulation-based education:

systematic review and meta-analysis. Med Teach. 2013;35:e844-

e875.

• Lorello GR, et al. Simulation-based training in anesthesiology: a

systematic review and meta analysis. Br J Anaesth. 2014 Feb;

112(2):231-45.

• McGaghie, W. C., Issenberg, S. B., Cohen, E. R., Barsuk, J. H., &

Wayne, D. B. (2011). Does simulation-based medical education

with deliberate practice yield better results than traditional clinical

education? A meta-analytic comparative review of the evidence.

Academic Medicine, 86, 706–711.

Simulation vs Other Intervention

• Simulation is non inferior to other approaches

• Technology enhanced simulation shows small,

statistically significant benefit for knowledge

and skills outcomes

• Approached statistical significance with direct

patient benefit and provider behaviors

Simulation vs Other Intervention

• Anesthesia review showed moderate effects for

satisfaction and skills, large effect for provider

behavior, small effect for direct patient benefit

compared with non simulation instruction

• Negligible effects comparing simulation to

alternative simulation interventions

• Analysis showed inconsistencies in measurement

of non technical skills

Pooled effect sizes for studies comparing training with no training. Effect sizes represent Cohen’s d or the nearly-equivalent Hedges’ g from random-effects meta-analysis; > 0.80 is large, 0.50–0.79 is moderate. Data derived from meta-analyses of Internet-based instruction (Cook et al. 2008b) virtual patients (Cook et al. 2010a), simulation-based instruction (McGaghie et al. 2011), and simulation-based instruction (Cook et al. 2011a)

Simulation based education is probably as good as (but is not necessarily better than) other types of instruction.

There is a high degree of variability between studies, suggesting that certain simulation interventions may be more effective in certain scenarios.

HOW CAN WE IMPROVE THE EFFECTIVENESS OF SIMULATION TRAINING?

Comparative Effectiveness

• Evaluates outcomes and processes leading to

the outcome

– Considers the costs, barriers, unforeseen

consequences, and effective strategies associated

with implementing therapies in practice.

• Studies comparing different simulation based

approaches to explain what works, for what

audience and in what context.

Comparative Effectiveness

• Sample sizes must be large

• Confounding can be a problem

• Effect size?

– The difference between teaching and no

teaching should be large, differences

between teaching types may not be as

apparent

Comparative Effectiveness

• Cook DA, Hamstra SJ, Brydges R, et al.

Comparative effectiveness of instructional design

features in simulation-based education:

systematic review and meta-analysis. Med Teach.

2013;35:e844-e875.

Comparative Effectiveness

• 289 studies, 20K participants

• Feedback, repetition, range of difficulty, cognitive

interactivity, clinical variation, distributed practice,

individualized training, and longer training time

significantly improve skill outcomes

• Patient outcome analysis revealed benefits of

similar direction and magnitude that approached

statistical significance

Future research should clarify how to choose between simulation and non-simulation approaches using rigorous qualitative studies that explore the strengths and appropriateness of each approach.

IS SIMULATION TRAINING COST EFFECTIVE?

What is the Value?

• Must consider costs

– Simulator, faculty time, training

expenses, facility fees, opportunity cost

• No study has offered a complete

accounting of simulation costs

Cost Assessment

• Zendejas B, Wang AT, Brydges R, Hamstra SJ,

Cook DA. Cost: the missing outcome in

simulation-based medical education research: a

systematic review. Surgery. 2013;153:160-176.

• Norman G, Dore K, Grierson L. The minimal

relationship between simulation fidelity and

transfer of learning. Med Educ. 2012;46:636-647.

What is the Value?

• Cost reporting is infrequent and

incomplete

• Low-fidelity, low-cost training models

can yield outcomes equal to much

more expensive simulators

More expensive simulators are not necessarily better.

CONCEPTUALLY, SIMULATION TRAINING MAKES SENSE

Simulation Training = Patient Safety?

• Ziv A, Wolpe PR, Small SD, Glick S. Simulation-

based medical education: an ethical imperative.

Acad Med. 2003;78:783-788.

Simulation Education = Patient Safety?

• “Risk free” skill rehearsal

– Repetitive, deliberate and structured

practice

• “Risk free” assessment and feedback

Simulation based rehearsal is evidence supported when learning to work with “real patients.”

SIMULATION MAY NOT ALWAYS BE THE BEST LEARNING MODALITY

Is Simulation the Best Choice?

• Instructional design should be based on

learning objectives, learner needs, safety

concerns, resource utilization

• Virtual patients often used to teach clinical

reasoning

• Technology enhanced simulation is most

often for procedural training

How Do We Optimally Implement Simulation Education?

• More research to clarify how to

choose between sim and non sim

approaches

• Cost effectiveness research to

determine true and comparison value

of sim education

How Do We Optimally Implement Simulation Education?

• Comparative effectiveness research should

focus on what works in simulation – for

which audience – for what circumstances –

and at what cost.

• Task analysis should focus on critical actions

and can help determine level of fidelity

necessary for type of training

How Do We Optimally Implement Simulation Education?

• Careful selection and sequencing of

events surrounding the simulated task

• Appropriate faculty development and

support

• Institutional commitment to simulation

QUESTIONS?