script concordance test jean paul fournier département de pédagogie médicale faculté de...
TRANSCRIPT
Script Concordance Test
Jean Paul Fournier
Département de Pédagogie MédicaleFaculté de Médecine de Nice Sophia Antipolis, France
Tempus ProgramMarch 12, 2012
Clinical reasoning
Patient’s complaint
Problem representation
Early generationof relevant hypothesis
Hypothesis check
Clinical data interpretation
Quest for newclinical / paraclinical data
Early problem identification(non analytic process)Compiled knowledge
Nendaz M. Pédagogie Médicale 2005
Pellacia Th. Annales Françaises de Médecine d’Urgence 2011
InstancesPrototypes
Semantic networks
Script activation
AnalyticprocessMicro judgements
(slots)
Tempus ProgramMarch 12, 2012
Clinical reasoning (2)
No hypothesis
Systematic approach
Diagnosis
No semantic network High complexity case
EpidemiologyPrevalence
Pathophysiology…
Forwardreasoning
Pellacia Th. Annales Françaises de Médecine d’Urgence 2011
Tempus ProgramMarch 12, 2012
Clinical reasoning (3)
Non analytic process Description
Pattern recognitionIntuitive identification of a typical signs configuration (= pattern), correspounding to diagosi(e)s
InstancesIntuitive identification of a déjà vue situation and immediate diagnosi(e)s generation
Pellacia Th. Annales Françaises de Médecine d’Urgence 2011
Tempus ProgramMarch 12, 2012
Clinical reasoning (4)
Analytic process Description
Hypothetico deductive process
The physician consciously tries to confirm / infirm his early hypotheses, with an oriented quest of clinical / paraclinical signs
Forward reasoning
The physician goes from clinical / paraclinical data to the problem solving, through application of causal or conditionnal rules
Pellacia Th. Annales Françaises de Médecine d’Urgence 2011
Tempus ProgramMarch 12, 2012
A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She suffers an ovarian carcinoma, in palliatie care. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38,2°C, BP is 135-80 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. Last creatinine level was 150 μmol/l. She has venous disease sequellae and bilateral pitting edema. You think of a possible pulmonary embolism.
Venous ultrasound
Femoral vein thrombosis
D dimers assay unesuseful in such a
context
CT pulmonary angiogram
Bilateral proximal pulmonatry embolism
D dimers assay
D dimers > 5000 g/ml
With an high pre test probability and D dimers >
4000 : pulmonary embolism probability : 88 p. cent
CT pulmonary angiogram benefit/risk ?
Heparin
Physician 1 Physician 3D dimers assay
unesuseful in such a context
Physician 2
Tempus ProgramMarch 12, 2012
What happened ?
3 different pertinent approaches but with the same result : heparin therapy ;
3 different pertinent approaches for the resolution of the same problem
Impact of new data in initial hypotheses (micro judgements).
Uncertainty : D dimer results interpretation in an old wman with ovarian cancer, CT pulmonary angiogram risk/benefit in an old woman with chronic kidney impairment ;
Script activation
Clinicalreasoningvariability
Ponderation
Uncertainty
Tempus ProgramMarch 12, 2012
Script theory
Script : Compiled knowledge network (clinical, biological, imaging,…) used for a clinical problem solving
Scripts are elaborated from early patients encounters ;
The more frequent and diverse are encounters, the richest are scripts ;
Scripts cannot be transfered ;
They vary from a physician to anaother.
Charlin B. Acad Med 2000
Tempus ProgramMarch 12, 2012
Script Concordance Test Standardized assessment tool of clinical reasoning through the presence
and functionnality of scripts in simulated clinical situations in an uncertainty context
Standardized assessment tool ; Clinical situations ; Ill-defined situation and context : no consensus, no evidence-based medicine ; Assessment of compiled clinical reasoning ; Reference to an expert panel (aggregate scoring).
SCT : Principles
Comparing the attributed impact of a new information on initial hypothesis between experts and students ;
Experts take the same test as students (same questions, same timing) and serve as the reference panel ;
All experts answers are accepted (with the restriction of discordant answer or
expert) ;
The closest to the experts’ choice is the students’ choice, the most performing are students.
Tempus ProgramMarch 12, 2012
The SCT
If you were planing to prescribe…
And then you find… The impact in your hypothesis becomes…
A D-dimer assay An ongoing ovarian carcinoma in palliative care
-2 -1 0 +1 +2
A venous ultrasound A swollen painful left calf -2 -1 0 +1 +2
A CT pulmonary angiogram Créatinine level of 150 µmol/l -2 -1 0 +1 +2
-2 : Very unuseful-1 : Unuseful
0 : Neither more or less useful +1 : Useful+2 : Very useful
A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38,2°C, BP is 135-80 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. You think of a possible pulmonary embolism.
Short clinical vignette
Pertinent hypothesis New relevant information
Anchor
Question
Likert scale : assessmentof the new information
on the initial hypotheses(Micro judgement)
Item
SCT vs rich-context MCQ
Tempus ProgramMarch 12, 2012
Is SCT an efficient assessment tool ?
A valid tool ? Face validity : depends on test construction (blueprint) ;
Construct validity :
identification of juniors vs seniors ;
prediction of scores for tests exploring the same domain ;
A reliable test ? Cronbach alpha from 0,79 to 0,82.
MCQ : 0,76 à 0,93 (2 to 8 hour testing)
Patient Management Problem : 0,69 à 0,82 (4 to 8 hour testing)
Key Feature Case : 0,66 à 0,79 (4 à 8 hour testing).
Tempus ProgramMarch 12, 2012
SCT among other assessment tests ?
MCQ ;
Clinical cases for ECN.
Tempus ProgramMarch 12, 2012
SCT vs rich-context MCQ
9523Clinical ED situations
Recommandations NBME
RecommandationsB. Charlin 2002
60 MCQ 60 SCT
Each situation appeararing in the 2 formats
Test taken by 16 EM residents
Tempus ProgramMarch 12, 2012
SCT vs rich-context MCQ
Tempus ProgramMarch 12, 2012
SCT vs true/false MCQ
Collard A. Med Educ 2009
106 medical students from 3rd to 6th year ;
Endocrinology tutorial ;
SCT vs true/false MCQ :
Significant correlation in 3rd and 4th year ;
Correlation for the 4 years with the knowledge level estimate (true vs false answers).
Tempus ProgramMarch 12, 2012
SCT vs ECN clinical case
18 medical students from 4th to 6th year ;
ECN clinical case ;
26 questions SCT.
Gibot S. Pédagogie Médicale 2008
Tempus ProgramMarch 12, 2012
SCT vs ECN clinical case
R = 0,55
Gibot S. Pédagogie Médicale 2008
Tempus ProgramMarch 12, 2012
SCT vs other assessment tools
Significant correlation, but low clinical significance : : some common points (assessment domain), but the SCT assesses some other aspects ;
Combining several assessment formats is a key-point in assessment.
Tempus ProgramMarch 12, 2012
Usual assessment tools
Knowledge
Knowledgeutilization
Competences
Action
True/false MCQ, essay-questions
Rich context MCQ, SCT, PMP, key-features
Standardized oral examination, OSCE, mini CEX simulation, SCT
Direct observation, peers, video, audits
Miller Acad. Med. 1990Compétence
Performance
Tempus ProgramMarch 12, 2012
How is it accepted ?
Bounouffe C. Am J Pharm Educ 2010
Tempus ProgramMarch 12, 2012
Technical aspects
Blueprint ;
Preparation ;
Anchor ;
Experts panel ;
Quality control.
Scoring ;
Tempus ProgramMarch 12, 2012
Blueprint4 points :
What do we do ?
Situations selection ?
Questions formats
Items and questions number
Certification ? Formative aspect ? Difficulty level ?
Prevalence ?
Severity ?
Diagnosis
Treatment
Management
Validity
Reliability
Feasibility
Tempus ProgramMarch 12, 2012
Blueprint
Cardio Pneumo Neuro Gastro Metabol. Rheumato Geriatr. Gyneco Psy Traumato
Dg X X X X X X X X X
Trt X X X X X X
Management X X X
Test developers + experts
Tempus ProgramMarch 12, 2012
Blueprint
Cardio Pneumo Neuro Gastro Metabol. Rheumato Geriatr. Gyneco Psy Trauma
Dg Chest pain
Acute shortness of
breath
Headache Abdominal pain
Mono arthritis
Altered general
condition
Ectopic pregnancy
Confusion Neck trauma
Trt CHF Pulmonary embolism
Seizure Gastric bleeding
Dehydrat. Agitation
Management ACS COPD exacerbation
Stroke
Test developers + experts
Tempus ProgramMarch 12, 2012
How many cases, how many questions ?
Case specificity ;
Many cases with one question ?
1 case with n questions ?
20 cases and 60 questions : Cronbach alpha ≥ 0,70
Formathomogeneity
Few cases with many questions ?
Tempus ProgramMarch 12, 2012
How many cases, how many questions ?
Gagnon R. Adv Health Sci Educ 2008
Tempus ProgramMarch 12, 2012
How many cases, how many questions ?
Gagnon R. Adv Health Sci Educ 2008
Tempus ProgramMarch 12, 2012
Preparation : situations, questions
Principles :
Clinical situations selection, fitted to students level ;
Relevant hypothesis selection in that context ;
New relevant informations selection, in that context
Page G. Academic Medicine 1995
Test developers+ experts
Uncertainty
Tempus ProgramMarch 12, 2012
Clinical vignette and questions preparation
Vignette : relevant informations :
Age, gender ;
Complaint, adlission, hospitalization reason ;
Context : ED, office, ward,…
Clinical data,…
1 to 2writers
Tempus ProgramMarch 12, 2012
Clinical vignette and questions preparation
Relevant new informations :
Clinical data ;
Biological test ;
Imaging ;
Video,…
1 to 2writers
Tempus ProgramMarch 12, 2012
Clinical vignette and questions preparation
Vignette, hypothesis and new informations :
Sweeping the five points of the Likert scale ;
Well-balanced variability level ;
All cases are independant ;
All questions are independant ;
One format by case (diagnosis, treatment,…).
Charlin B. Medical Education 2006
Tempus ProgramMarch 12, 2012
Question discrimination power and variability
Charlin B. Medical Education 2006
Tempus ProgramMarch 12, 2012
Lickert scale - Anchor
-1 : Unlikely-2 : Very unlikely
0 : Neither more or less likely
+1 : More likely+2 : Very likely
Recommandations :
Explicit the significance of each point of the Lickert scale, particularly the neutral point (« 0 ») ;
Keep the same presentation.
Tempus ProgramMarch 12, 2012
Lickert scale - Anchor
Lickert scales :
If you were thinking of the following diagnosis…
And then you find… This new information makes your diagnosis…
Diagnosis option New information -2 -1 0 +1 +2
-1 : Unlikely -2 : Very unlikely
0 : Neither more or less likely
+1 : More likely+2 : Very likely
Diagnosis anchor :
Tempus ProgramMarch 12, 2012
Lickert scale - Anchor
Lickert scales :
If you were considering the usefulness of the following
test…
And you find… This new information makes the following test…
Investigative option New information -2 -1 0 +1 +2
-1 : Useless-2 : Less useful
0 : Neither morer or less useful +1 : Useful+2 : Very useful
If you were considering the benefit/risk ratio of the
following test…
And you find… This new information makes the test…
Investigative option New information -2 -1 0 +1 +2
-1 Contraindicated-2 : Strongly contraindicated
0 : Neither more or less indicated
+1 : Indicated+2 : Strongly indicated
Anchor for test(utility) :
Anchor for test(benefit/risk) :
Tempus ProgramMarch 12, 2012
Lickert scale - Anchor
Lickert scales :
If you were considering the usefulness of the following
test…
And you find… This new information makes the following test…
Investigative option New information -2 -1 0 +1 +2
-1 : Useless-2 : Less useful
0 : Neither morer or less useful +1 : Useful+2 : Very useful
If you were considering the benefit/risk ratio of the
following test…
And you find… This new information makes the test…
Investigative option New information -2 -1 0 +1 +2
-1 Contraindicated-2 : Strongly contraindicated
0 : Neither more or less indicated
+1 : Indicated+2 : Strongly indicated
Anchor for test(utility) :
Anchor for test(benefit/risk) :
Inadequate prescription
Non appropriate option Adequate prescription
Tempus ProgramMarch 12, 2012
Lickert scale - Anchor
Likert scales :
If you were considering the usefulness of the following
treatment…
And you find… This new information makes the following treatment…
Treatment option New information -2 -1 0 +1 +2
-1 : Useless-2 : Less useful
0 : Neither morer or less useful +1 : Useful+2 : Very useful
If you were considering the benefit/risk ratio of the following treatment…
And you find… This new information makes the following treatment…
Treatment option New information -2 -1 0 +1 +2
-1 Contraindicated-2 : Strongly contraindicated
0 : Neither more or less indicated
+1 : Indicated+2 : Strongly indicated
Treatment anchor (utility) :
Treatment anchor (benefit/risk) :
Tempus ProgramMarch 12, 2012
Lickert scale - Anchor
Échelles :
Si vous considériez l’utilité du traitement suivant
Et qu’alors vous trouvez Cette nouvelle information rend le traitement
Treatment option New information -2 -1 0 +1 +2
-1 : Useless-2 : Less useful
0 : Neither morer or less useful +1 : Useful+2 : Very useful
If you were considering the benefit/risk ratio of the following treatment…
And you find… This new information makes the following treatment…
Treatment option New information -2 -1 0 +1 +2
-1 Contraindicated-2 : Strongly contraindicated
0 : Neither more or less indicated
+1 : Indicated+2 : Strongly indicated
Treatment anchor (utility) :
Treatment anchor (benefit/risk) :
Inadequate prescription
Non appropriate option Adequate prescription
Tempus ProgramMarch 12, 2012
Lickert scale - Anchor
Lickert scales :
Prognosis item New information The prognosis becomes
Prognosis item New information -2 -1 0 +1 +2
-1 : Worsened-2 : Very worsened
0 : Never more or less modified +1 : Improved+2 : Very improved
Prognosis anchor :
Tempus ProgramMarch 12, 2012
Experts panel
Roles :
Test preparation ;
Clinical vignette and questions validation ;
Scoring.
2 differentgroups
Tempus ProgramMarch 12, 2012
Experts panel
Quality : experts in the considered domain ;
Number : 10 to 20 minimum.
Gagnon R. Medical Education 2005
Tempus ProgramMarch 12, 2012
Mean reliability following the experts number
Gagnon R. Medical Education 2005
Tempus ProgramMarch 12, 2012
Expert panel composition
Teaching physicians vs physicians ?
Charlin B. Medical Teacher 2007
SCT in community medicine ;
Comparison of scores ;
Panel : teaching physicians vs physicians in a CME program ;
Perfect concordance of the scores (R = 0.98) ;
Scores attributed by physicians more elevated (p = NS).
Tempus ProgramMarch 12, 2012
Scoring
Scoring :
Panel choice :
Modal transformation of the panel choice :
Residents’ choice :
Residents’ scores :
Tempus ProgramMarch 12, 2012
Scoring
Excel® scoring calculator available :
www.cme.umontreal.ca/tcs
Tempus ProgramMarch 12, 2012
Summary
Test preparation Blueprint Clinical vignettes Hypotheses New informations Scoring Pass/fail decision
Experts 1 Clinical vignettes Hypotheses New informations
Experts 2 Scoring (reference panel scoring)
Validation
Quality control
A priori quality control :
Caire F. Neurochirurgie 2004Tempus ProgramMarch 12, 2012
Tempus ProgramMarch 12, 2012
Quality control
A posteriori quality control : item analysis :
Difficulty index ;
Discrimination index ;
Question impact on the test reliability (Cronbach alpha) ;
Etc…
Thank you for your attention !