simulation / high stakes exams - rocky mountain · pdf fileeducation and high stakes exams...
TRANSCRIPT
Use of Simulation in Medical Education and High Stakes Exams Neil Gibson, MD, MSc, FRCPC,FACP
Division of General Internal Medicine
University of Alberta
Irene Ma, MD, MSc, FRCPC, FACP
Division of General Internal Medicine
University of Calgary
Objectives
By the end of the session, the participants will be able to:
1) Understand the uses of simulation in teaching and assessment
2) Understand the assessment requirements for high-stakes examination
What is Simulation?
An instructional medium used for education, assessment, and research
Reproduction of certain characteristics of clinical reality
Several modalities – Ogden PE et al. Am J Med 2007
Partial-task trainers (commercial, homemade, animal models)
Full body task trainers
High fidelity mannequins
Standardized patients
Virtual reality
Computer software
Chiniara G et al. Med Teach 2012
Outline
1. Simulation is useful and is here to stay
2. For simulation to remain useful, research agenda needs to adapt
3. The use of simulation in assessment opens up a realm of assessment opportunities
Copyright © 2012 American Medical
Association. All rights reserved.
Technology-Enhanced Simulation for Health Professions Education: A Systematic Review and Meta-analysis
JAMA. 2011;306(9):978-988 609 studies (137 RCTs, 67 two-group design, 405 pretest-postest)
Copyright © 2012 American Medical
Association. All rights reserved.
Technology-Enhanced Simulation for Health Professions Education: A Systematic Review and Meta-analysis
JAMA. 2011;306(9):978-988 609 studies (137 RCTs, 67 two-group design, 405 pretest-postest)
Copyright © 2012 American Medical
Association. All rights reserved.
Technology-Enhanced Simulation for Health Professions Education: A Systematic Review and Meta-analysis
JAMA. 2011;306(9):978-988 609 studies (137 RCTs, 67 two-group design, 405 pretest-postest)
12 Ma IWY et al. Acad Med Sep 2011; 86(9):1137-47.
In the case of Central Venous Catheterization, 20 studies (9 pretest posttest; 9 two-group; 2 RCTS)
13 Ma IWY et al. Acad Med Sep 2011; 86(9):1137-47.
In the case of Central Venous Catheterization, 20 studies (9 pretest posttest; 9 two-group; 2 RCTS)
14 Ma IWY et al. Acad Med Sep 2011; 86(9):1137-47.
In the case of Central Venous Catheterization, 20 studies (9 pretest posttest; 9 two-group; 2 RCTS)
Simulation is useful and is here to stay
1. The evidence is increasingly robust for its educational utility
Simulation is useful and is here to stay
2. Simulation makes sense. Analogy from flight simulation
http://en.wikipedia.org/wiki/File:Antoinette_sim_1909.jpg
Simulation is useful and is here to stay
2. Simulation makes sense. Remains useful for flight training.
Photo Credit: NASA. http://gimp-savvy.com/cgi-bin/img.cgi?ailsorHdFtW9c861266
Simulation is useful and is here to stay
3. Simulation has already been here for some time
Asmund Laerdal with Resusci-Anne, in about 1970.
Cooper J B , Taqueti V R Qual Saf Health Care 2004;13:i11-i18
Simulation is useful and is here to stay
2. Simulation has already been here for some time
Michael Gordon with Harvey in the early 1970s
Cooper J B , Taqueti V R Qual Saf Health Care 2004;13:i11-i18
Simulation is useful and is here to stay
2. Simulation has already been here for some time, even “high-fidelity” simulators
Cooper J B , Taqueti V R Qual Saf Health Care 2004;13:i11-i18
Dr Stephen Abrahamson (seated) and Dr Judson Denson with Sim One in the late 1960s
Nat Geo Nov 1970
1. Simulation is useful and is here to stay
2. For simulation to remain useful, research agenda needs to adapt
3. The use of simulation in assessment opens up a realm of assessment opportunities
How to Kill Off a Good CRM Program
1. Not integrating CRM into LOFT, PT, and other operational training
2. Failing to recognize the unique needs of your own airlines culture
3. Allowing the CRM Zealots to run the show
4. Bypassing research and data gathering steps
5. Ignoring the checking and standards pilots
6. Having lots of diagrams, boxes, and acronyms
7. Making the program a one shot affair
8. Using pop psychology and psycho babble
9. Turning CRM into a therapy session
10. Redefining the “C” to mean charismatic
Taggart WR. THE CRM Advocate 1993; 93(1):11-2
simulation [ti] AND medical education AND year [dp] Search: PubMed, November 9, 2012
Much research has been done on medical simulation
For simulation to remain useful, research agenda needs to adapt We need to stop asking:
“Does it work?”
“If you teach them, do they learn?”
Instead, we need to ask questions that help advance educational theory, curriculum development and implementation
1. Simulation is just a tool
2. Education is complex
For simulation to remain useful, research agenda needs to adapt Instead, ask educationally useful questions:
How best to teach using simulation?
How do learners learn?
When do they learn?
Why do some not learn?
When to use to simulation?
For simulation to remain useful, research agenda needs to adapt Instead, ask educationally useful questions:
How best to teach using simulation?
How do learners learn?
When do they learn?
Why do some not learn?
When to use to simulation?
Chiniara G et al. Med Teach 2012
Organophosphate Poisoning
Septic shock
UTI, pneumonia
Eosinophilic Fasciitis
Chiniara G et al. Med Teach 2012
Organophosphate Poisoning
Septic shock
UTI, pneumonia
Eosinophilic Fasciitis
For simulation to remain useful, research agenda needs to adapt Simulation is NOT the answer for every educational activity we have
How to Kill Off a Good CRM Program
1. Not integrating CRM into LOFT, PT, and other operational training
2. Failing to recognize the unique needs of your own airlines culture
3. Allowing the CRM Zealots to run the show
4. Bypassing research and data gathering steps
5. Ignoring the checking and standards pilots
6. Having lots of diagrams, boxes, and acronyms
7. Making the program a one shot affair
8. Using pop psychology and psycho babble
9. Turning CRM into a therapy session
10. Redefining the “C” to mean charismatic
Taggart WR. THE CRM Advocate 1993; 93(1):11-2
How to Kill Off a Good CRM Program
1. Not integrating CRM into LOFT, PT, and other operational training
2. Failing to recognize the unique needs of your own airlines culture
3. Allowing the CRM Zealots to run the show
4. Bypassing research and data gathering steps
5. Ignoring the checking and standards pilots
6. Having lots of diagrams, boxes, and acronyms
7. Making the program a one shot affair
8. Using pop psychology and psycho babble
9. Turning CRM into a therapy session
10. Redefining the “C” to mean charismatic
Taggart WR. THE CRM Advocate 1993; 93(1):11-2
1. Simulation is useful and is here to stay
2. For simulation to remain useful, research agenda needs to adapt
3. The use of simulation in assessment opens up a realm of assessment opportunities
Use of simulation in assessment
1. Assesses performance at a higher level
Triangle of knowledge / clinical skills competence / performance Miller 1980
Use of simulation in assessment
1. Assesses performance at a higher level
Triangle of knowledge / clinical skills competence / performance Miller 1980
Does
Shows how
Knows how
Knows
SPs as a form of simulation has been widely implemented Objective structured clinical examination
Medical school
LMCC Part II
Royal College of Physicians and Surgeons of Canada
USMLE Step 2 Clinical Skills examination (since 2004)
National Board of Osteopathic Medical Examiners (NBOME) Comprehensive Osteopathic Medical Licensing Examination Level 2-Performance Evaluation (COMLEX-USA Level 2-PE).
Use of High-Fidelity Simulation in High Stakes Examinations Royal College of Physicians and Surgeons of Canada – Internal Medicine
Combined auscultation videos with SP, 2003 – Hatala R et al. Acad Med 2005
Harvey® cardiopulmonary simulator
Israeli Board Examination in Anesthesiology
ER Trauma management
Resuscitation – ACLS
OR - hypertension post induction
Adjustment of ventilation in response to ABG results
Framework for Simulation-based Assessment
1. Valid
2. Reliable
3. Educational Impact
4. Acceptability
5. Cost
Van der Vleuten CPM. Advances in Health Sciences Education. 1996
Validity
“Validity refers to the degree to which evidence and theory support the interpretations of test scores entailed by the proposed uses of the tests.” American Psychological Association. Standards for Educational and Psychological Testing. 1999
Traditional Trinitarian view of validity: Guion RM 1980
1. Criterion validity
2. Content validity
3. Construct validity
Replaced by umbrella term: Construct validity
Encompasses all hypothesis testing in the pursuit of validity
In the absence of a gold standard, for the “new” exam in question to be valid… Exam should be relevant and representative of the tasks in question
Those who are more competent should score higher than those who are less competent
Those who score higher on the exam should in practice make more accurate diagnoses/provide better treatment plans than those who score lower
Those who score higher on the exam should be those who did well during residency
Evidence of Construct Validity
Forrest FC et al. Br J Anaesth 2002;88:338-44
Devitt JH et al. Anesth Analg 1998;86:1160-4
Framework for Simulation-based Assessment
1. Valid
2. Reliable
internal reliability, inter-rater reliability, intra-rater reliability
3. Educational Impact
4. Acceptability
5. Cost
Van der Vleuten CPM. Advances in Health Sciences Education. 1996
Optimizing Reliability
Standardize the examination
Examine on multiple occasions (how frequent?)
Sample multiple domains (knowledge, skills, attitude)
Use multiple methods (simulation, MCQ, track clinical outcomes, peer assessment,
3600 evaluation)
Trained raters, blinded
☐ Some or all of the above?
Evidence of Inter-rater reliability
Review of 28 performances by 3-5 raters
Inter-rater reliability ICC 0.79-0.85
96% agreement in pass/fail decisions
Weller JM et al. Br J Anaesth 2003;90:43-7
Framework for Simulation-based Assessment
1. Valid
2. Reliable
internal reliability, inter-rater reliability, intra-rater reliability
3. Educational Impact
4. Acceptability
5. Cost
Van der Vleuten CPM. Advances in Health Sciences Education. 1996
1. Simulation is useful and is here to stay
2. For simulation to remain useful, research agenda needs to adapt
3. The use of simulation in assessment opens up a realm of assessment opportunities
4. But for simulation in assessment to be useful, research agenda needs to adapt
Conclusion
Simulation for education and assessment seems new and exciting, although it actually has been here for some time.
We argue that that
1. Simulation is useful (and there is robust evidence to suggest that it is so) and is here to stay.
2. For simulation to remain useful, research agenda needs to adapt in order to advance learning theories
3. The use of simulation in assessment opens up a realm of assessment opportunities
4. Research agenda needs to establish its validity and reliability