![Page 1: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/1.jpg)
Blueprinting and Choosing Appropriate Tools for Assessment
of Student Performance: A Start to the Charting of the C’s
Claire Touchie, MD, FRCPC
Associate Professor, University of Ottawa
Chief Medical Education Advisor
Medical Council of Canada
Property of the Medical Council of Canada
![Page 2: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/2.jpg)
Conflict of Interest
None
2
![Page 3: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/3.jpg)
ObjectivesBy the end of the workshop, participants will be able to:• Define blueprinting in the context of specific
assessment needs• Design an appropriate blueprint for the purpose of a
specific assessment• Describe available tools for student performance
assessments• Construct an assessment tool box to achieve
assessment goals
3
![Page 4: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/4.jpg)
Agenda• Assessment and blueprinting• Exercise #1: Designing a blueprint• Group Discussion• Leg stretch• Assessment tool boxes• Exercise #2: Assembling a tool box• Group Discussion• Wrap-up and Evaluations
4
![Page 5: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/5.jpg)
5
Microsoft clip-art, 2012
![Page 6: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/6.jpg)
Professional Competence
Developmental
Impermanent
Context dependent
6
![Page 7: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/7.jpg)
Dimensions of Professional Competence
ProfessionalCompetence
Epstein and Hundert, JAMA 2002; 287:226-235 7
![Page 8: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/8.jpg)
The CanMEDS Seven C’s
8
Royal College of Physicians and Surgeons of Canada
![Page 9: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/9.jpg)
Charting the Seven C’s
Seven C’s
Epstein and Hundert, JAMA 2002; 287:226-235 9
![Page 10: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/10.jpg)
Purposes of Assessment• Certify the competence of future practitioners
• Minimal level of competence
• Discriminate among candidates for advanced training
• Rank competitively
• Provide motivation and direction for learning• Provide formative feedback
• Judge the adequacy of a training program• Evaluate teacher or courses
Epstein and Hundert, JAMA 2002; Crossley et al, Medical Education, 2002 10
![Page 11: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/11.jpg)
11
Purposes of Assessment
Learning
AssessmentFeedback
• To discover the worth• To improve the quality
![Page 12: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/12.jpg)
AssessmentDoes
Shows How
Knows How
Knows
Modification of Miller from Crossley et al. Medical Education, 2002 12
Professional Authenticity
Cognition
Behavior
![Page 13: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/13.jpg)
“If you don’t know where you’re going, you probably ain’t going to get
there”
13
Microsoft clip-art, 2012
![Page 14: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/14.jpg)
Blueprinting
Defining the “what” should be represented on your assessment• For licensure or certification
• Practice analysis
• For a course or rotation• Objectives
14
![Page 15: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/15.jpg)
Blueprint“Specifies all the elements of performance relevant to the assessment so that appropriate samples of activity and corresponding methods can be selected according to their relative importance to the overall assessment process.” Newble & Dawson, 1994
“When the items of a test are judged to adequately represent well-defined domains of content…generalizable samples…” Cronbach, 1971
15
![Page 16: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/16.jpg)
16
Domains to be tested
Items tested
![Page 17: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/17.jpg)
Blueprint
Valid assessment: interpretation of the results reflect what was meant to be tested• Appropriate blueprint
• Defines the content of the assessment• Provides validity evidence
17
![Page 18: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/18.jpg)
How do I blueprint for my assessment?
1. Define the purpose of your assessment• Summative or Formative?• To decide on competency or to rank?• To provide feedback to the student or to the
teacher?
2. Tabulate curricular content• Topics vs clinical presentation vs objectives
18
![Page 19: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/19.jpg)
Example of curricular content• Community acquired pneumonia• Hospital acquired pneumonia• Otitis media• Urinary tract infections• Meningitis• Endocarditis• Osteomyelitis/septic arthritis• Septicemia• Cellulitis/skin infections• HIV
19
![Page 20: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/20.jpg)
How do I blueprint for my assessment?3. Provide relative weighting of the content
Importance X frequency• Importance/impact
1. Non-urgent, little prevention potential
2. Serious, but not immediately life threatening
3. Life threatening emergency and/or high potential for prevention impact
• Frequency1. Rarely seen
2. Relatively common
3. Very common
20
![Page 21: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/21.jpg)
How do I blueprint for my assessment?
4. Sample opinion on weighting• Involve course chairs, teachers, evaluation
coordinators, previous learners• Consensus method• Readjust weighting if necessary
21
![Page 22: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/22.jpg)
Example of weightingContent presentation Importance Frequency I X F Weight
CAP 3 3 9 0.22
HAP 3 1 3 0.073
Otitis media 1 3 3 0.073
UTI 2 3 6 0.15
Meningitis 3 1 3 0.073
Endocartitis 3 1 3 0.073
Osteo/Septic arthritis 2 1 2 0.049
Septicemia 3 2 6 0.15
Cellulitis/Skin infn 2 2 4 0.098
HIV 2 1 2 0.049
Total 41
22
![Page 23: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/23.jpg)
How do I blueprint for my assessment?
5. Decide total number of items on your assessment• Based on your weighting, you can decide
how many items per content area
23
![Page 24: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/24.jpg)
How do I blueprint for my assessment?
6. Decide how you would like to break it down?
• Tasks• Diagnosis/Investigation/Treatment/Counseling
• Context of care• ED/Inpatient/Outpatient
• Competencies
24
![Page 25: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/25.jpg)
How do I blueprint for my assessment?Content No. of items Diagnosis Investigation Treatment
CAP 11 4 4 3
HAP 3.65 1-2 0 1-2
Otitis media 3.65 1-2 0 1-2
UTI 7.5 2 3 2
Meningitis 3.65 1 1 1-2
Endocarditis 3.65 1 1-2 1
Osteo/Septic arthritis 2.45 1 1 0
Septicemia 7.5 2-3 1-2 3
Cellulitis/Skin infection 4.9 2-3 0 2
HIV 2.45 1 1-2 0
25
![Page 26: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/26.jpg)
How do I blueprint for my assessment?
Final blueprint often looks like a matrix• Patient group X Task• Clinical presentation X Context of Care• Clinical Activities X Competencies
26
![Page 27: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/27.jpg)
Exercise #1• Using worksheet #1 start working on a
blueprint for your purpose• Get together with colleagues and choose
one blueprint to work on• Large group discussion
27
![Page 28: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/28.jpg)
28
Microsoft clip-art, 2012
![Page 29: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/29.jpg)
Assessment Tool Boxes
What will I use to assess my blueprint?
29
![Page 30: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/30.jpg)
How do I assess what I want to?
Blueprint • Establishes what you wish to test
Assessment tool box• Helps to determine how you will test
30
![Page 31: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/31.jpg)
Assessing the 7 C’s
31
![Page 32: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/32.jpg)
Assessing the Competencies
Professional Competence• More than a demonstration of isolated
competence Eraut, 1994
• “…when we see the whole, we see its parts differently than when we see them in isolation” Polanyi, 1969
32
![Page 33: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/33.jpg)
Assessing the Competencies• Context specific• What may be considered “competent” in
one context may not be in a different context.
33
![Page 34: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/34.jpg)
Assessment tools for CompetenciesRCPSC suggest:• In-training evaluations (ITERs)• MCQs• SAQs• Essays• Oral examinations• OSCEs• Simulations• Log books
34
Microsoft clip-art, 2012
![Page 35: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/35.jpg)
CanMEDS Evaluations and PGE
Med expert • ITER>Oral>MCQ>SAQ
Communicator • ITER>Oral>OSCE>sim
Other roles• ITER>Oral>OSCE
Chou et al. Medical Education, 2008 35
![Page 36: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/36.jpg)
CanMEDS Evaluation and PGEPG directors levels of satisfaction with CanMEDs evaluations• Medical Expert:
• satisfied to very satisfied
• Communicator, collaborator, scholar, and professional
• Neutral to satisfied
• Manager and health advocate• Dissatisfied to neutral
Chou et al. Medical Education, 2008 36
![Page 37: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/37.jpg)
ACGME Competencies• Patient care• Medical knowledge• Practice-based learning and improvement• Inter-personal and communication skills• Professionalism• Systems-based practice
37
![Page 38: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/38.jpg)
Assessment tools for ACGME Competencies
ACGME tool box:Record review Chart stim. Recall
Checklist Global rating
SP OSCE
Simulations 3600 global rating
Portfolios MCQs
Oral exam Log books
Patient survey
38
![Page 39: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/39.jpg)
Assessment Challenges when Charting the 7 C’s
• Socially negotiated educational competencies• Accreditation agencies mandate that competencies
be assessed• Tools don’t match single competencies
independently except for medical knowledge/expert• Other competencies/roles
• Reflect personal attributes?
• What is desirable VS what is observable?
Lurie, Medical Education, 2012 39
![Page 40: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/40.jpg)
Potential solutions?• Linking abstract competencies to
observable behaviors• Key competencies/Enabling competencies
• Mapping competencies to observable behaviors• Milestones, Entrustable Professional Activities
(EPAs)
40
![Page 41: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/41.jpg)
Word of caution
“…a tendency to describe general competencies in exhaustive detail, leading to bulky, fragmented documents that lose practical value for education as they become less and less connected with the real world.” ten Cate et al. Medical Teacher 2010
41
![Page 42: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/42.jpg)
Building your own assessment toolbox• Define the purpose of your assessment• Define the population you are assessing• From your blueprint, what is best
assessed with what tool• Is it going to be reliable? Feasible?
Acceptable?• Will the result interpretation be valid?
42
![Page 43: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/43.jpg)
Exercise #2• Create your own toolbox using
worksheet #2• Discuss your toolbox with a colleague• Group discussion
• Challenges?
43
![Page 44: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/44.jpg)
44
Microsoft clip-art, 2012
![Page 45: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/45.jpg)
In Summary• Define what you wish to assess by
designing a blueprint• Decide how you will assess• Understand the challenges and
limitations to assessing the 7 C’s• Stay tuned to further developments
45
![Page 47: Blueprinting and Choosing Appropriate Tools for Assessment of Student Performance: A Start to the Charting of the C’s](https://reader036.vdocument.in/reader036/viewer/2022062513/5579ef26d8b42abc2e8b470f/html5/thumbnails/47.jpg)
Useful references1. Epstein RM and Hundert EM. Defining and assessing
professional competence. JAMA 2002;287:226-235
2. Crossley J, Humphris G, Jolly B. Assessing health professionals. Medical Education 2002;36:800-804
3. Lurie SJ. History and practice of competency-based assessment. Medical Education 2012;46:49-57
4. Coderre S, Woloschuk W, McLaughlin K. Twelve tips to blueprinting. Medical Teacher 2009;31:322-324
47