Integrating Assessment into
CPD Programs September 26, 2017
Kate Hodgson DVM, MHSc, CCMEP
Suzan Schneeweiss MD, MEd, FRCPC
Faculty/Presenter Disclosure
• Faculty: Kate Hodgson and Suzan
Schneeweiss
• Relationships with commercial interests:
–None
Learning objectives
At the end of this session, you will be able to:
• Apply Moore’s model of outcome assessment in a
strategic curriculum
• Integrate performance and knowledge assessment
within CPD programs for Royal College or
MAINPRO+ credits
• Use a flipped classroom for deeper engagement and
relevant practice application of competency
development
Program Evaluation Versus Outcome
Measurement
• CPD Program Evaluation measures program
learning environment, speaker and content
• CPD Outcome Measurement measures
changes in learners and practice and resultant
patient and health care changes
Program Evaluation versus Outcome
Measurement
Program Evaluation
• The skill of the speaker
• Quality/relevance of
the information
presented
• The venue
• The timeframe for the
session
Outcome Measure
• Participants’
knowledge skills
attitude
• Practice Change
• Patient Care
• Health care outcomes
And now for the bigger challenge …..
Impact of CPD on Physician Performance
and Patient Outcomes
Outcome Number of Studies % positive
Knowledge 22 / 28 79
Skills 12 / 15 80
Attitude 22 / 26 85
Practice behavior 61 / 105 58
Clinical outcomes 14 / 33 42
Cervero and Gaines JCEHP 2015• Systematic reviews published since 2003• CPD has a positive impact physician performance, less
reliably on patient health outcomes
Impact of CPD on Physician Performance
and Patient Outcomes
Greater improvement in physician performance
when programs:
• Focus on outcomes considered important by
physicians
• Are more interactive
• Use multiple methods of learning
• Involve multiple exposures Cervero and Gaines JCEHP 2015
CPD Outcomes
“ … should focus on identifying,
measuring, and describing the value provided
by CME that leads to enhanced physician
performance, improved health care quality and
reduced cost.”Moore D.E. Framework for outcomes evaluations in the continuing
professional development of physicians. In Davis D et al. The
Continuing Professional Development of Physicians. Chicago. AMA
Press, 2003: Chicago. P251.
Why measure CPD outcomes?
• Leads to more effective, better-targeted
education
• Provides a road map to future education
• Demonstrates value to internal and external
clients
• Accreditation standard
Model for Assessing Outcomes in CPD
Wallace S & May SA, Vet Record 2016; Adapted from Moore 2009 and Miller 1990
Application of Moore’s Outcome
Evaluation Framework
Model combines stages of learning, instructional design
and planning, outcomes framework and assessment:
1. Start with the end in mind
2. Take stages of learning into account
3. Focus on clinical problems and
knowledge that can be used in practice
4. Provide opportunities for practice and feedback in
authentic settings.
Stages of Learning and Change
Level 3: Learning
Knowledge
(e.g. MCQ or true/false test)
Skills
(e.g. psychomotor,
decision-making,
interpersonal)
Attitude
(e.g. questionnaire)
3 domains of learning• Can be done using pre/post evaluation
Small Group Discussion
• Consider a program you are developing or
have developed.
• How might you incorporate measurement of
learning in your program?
Enhancing Learning in CPD Conference
• SickKids Annual Paediatric Update
Conference
• Integration of test-enhanced learning in
concurrent workshops
– 5 pre and post MCQ questions
– Immediate feedback on post MCQs via pop-up
– 1-month later asked new questions
• Accredited Royal College Section 3
Testing Effects
• Test-enhanced learning (TEL)
– Testing after learning
– Increases knowledge retention and transfer
• Test-potentiated learning (TPL)
– Pre-testing to enhance subsequent learning
opportunity
– Identify knowledge gaps and promote self-
regulation behaviours such as reflection, strategic
studying or learning and metacognition
Test Enhanced Learning
• Testing promotes active retrieval of information from
memory
• With repeated testing, more likely to successfully
retrieve information in future context
• Long-term retention of materials is enhanced when
learning events are separated temporally from one
another.
• Feedback enhance TEL effects
Level 4: Competence
(Shows How)
• Requires demonstration and evaluation
• Ensures that person has learned something that
could be applied accurately
• Can be done with:
– physical skills
– interpersonal skills
– reframed approaches
– assessment process
Level 4: Competence (Shows How)
Demonstration to Peers
Demonstration to Instructors
Demonstration with Simulated
Patients (Role Play)
Tested Skills
Small Group Discussion
How might you integrate competency in your
program(s)?
Clinical Skills
in Communication
▪ Work in groups of 3: • The Physician
• The Patient
• The Observer
▪ Rotate roles through each of
three cases.
▪ Physician & patient may or
may not have private
information.
Observer Checklist:
Patient Communication
As an observer, you
will have a checklist
to track physician’s
techniques in
managing challenges
presented by Patient.
Observer Checklist
Case #15: Request for Dose Escalation
Please complete the following Observer Checklist. Yes NoDid the Care Provider comply with the patient’s request for dose escalation?
□ □
Did the Care Provider non-judgmentally explore the reasons for request?
□ □
Did the Care Provider specifically track or evaluate function? □ □
Did the Care Provider explain the connection between negative mood and pain?
□ □
Did the Care Provider explore changes in pain levels? □ □
Did the Care Provider review medications trialed? □ □
Patient Communication
Abby and Dr. Doshi
Case #16:
Abby and Dr. Doshi
▪Abby is a 21 yr old female
patient who suffered an
motorcycle accident 13
months ago, resulting in a
pelvic fracture.
▪ Since this time, she has
had ongoing pain despite
normal healing.
Case #16:
Abby and Dr. Doshi
▪ She has not had a family
physician until recently;
Dr. Doshi has now taken
her on as a patient in his
family health team.
▪ She is here today to
discuss getting something
to help her pain.
Example Level 4: Competence
• SickKids Hospital competency-based
workplace learning for Paediatric
Emergency Physicians
– High-risk procedures/ resuscitation
scenarios
– Flipped classroom approach (online
videos)
– Deliberate practice and rapid cycle
debrief
– Peer coaching and feedback
– Checklist for performance assessment
Workplace Learning
• Authentic, realistic, context for learning and
assessment
• Controlled, safe learning environment with
peers
• Instruction tailored to individual or group
needs
• Repetition and deliberate practice
• Providing feedback/ drive self-awareness
Increasing Engagement with SIM
• Team competency
– Crisis resource management
– Team performance management
• Individual Competency
– Procedures
– Critical incidents
– Leadership
– Communication
– Decision analysis
In-Situ Simulation in CPD
• Team Training
• In-house training in clinical setting
• Blend of simulation in real working
environments
• Assessment of teamwork effectiveness
– TEAM (Team Emergency Assessment Measure
(TEAM)Cooper et al. Resuscitation 2010;81:446-452.
TEAM Scale
Teamwork observational scale to assess performance of emergency medical teams4 Domainso Leadership,
teamwork, task management, global score
Level 5: Performance (Does)
• The learner does what he he has learned
– Follow up of practice change
• Subjective or self-report (e.g. post activity
survey, commitment to change)
• Direct observation in practice
• Indirect objective measures – quality and
utilization measures
Methods for Capturing Self-Reported
ChangeQualitative
• Online or mailed paper surveys
• Phone interviews
• Change reported (and recorded) at subsequent
meetings or educational sessions
• Focus groups
Quantitative
• Online or mailed paper surveys
Level 5: Performance (Does)
Products sent in for feedback
(e.g. videotape)
Case Study
(e.g. chart stimulated recall)
Databases
(e.g. length of stay, return visits)
Chart Audits
Commitments to Change (CTC)
Three Stages
• Participant asked to write 1 – 5 changes they plan to
make a change as a result of activity
• Asked to indicate a level of commitment utilizing a
Likert scale ranging from 1-5.
• 30 – 45 days participant is sent a list of these changes
and asked to indicate if a change occurred, partially
occurred or did not occur & why
Commitment to Change
• Allows participants opportunity to reflect on
salient pieces of information and extract
meaning in a personalized fashion
• Rating level of commitment is a mechanism
for how strongly one feels the goals should be
actualized
• Set up sense of accountability
Commitment to Change (CTC)
• Rates of compliance to CTC between 47-87%
• Dose response relationship between time in course
and number of changes reportedLockyer et al. JCEHP 2001
• Physicians who generated their own CTC, more
likely to implement changes in observed practiceWakefield et al. JCEHP 2003
• CTC can be based on predefined set of suggested
changes and still have impact on reported behavior
change Domino FJ et al. Med Teach 2011
• What questions would you consider for a
commitment to change feedback questionnaire
45 – 60 days later?
Questions to consider using for
commitment to change
• Did you make any change or apply learning to your practice?
– Yes: considering, already doing this, doesn’t apply to me
• Describe 2 things your tried or did differently
• Did you tell colleagues about X?
• Has this course changed how you interact with the team?
• What was easiest to change? Why?
• What are the barriers to making changes you would like to
make?
• What could help you to make changes?
Safer Opioid Prescribing
Guided Self-Assessment
• Chart Review:
• Support guided self-assessment by comparing
to standard guidelines
• Adapted from UHN Chart Review Checklist
for Opioid Prescription
Safer Opioid Prescribing
Guided Self-Assessment
• Had already implemented before the
webinar/workshop
• Improved since webinar/workshop
• Improvement still needed- high priority
• Improvement still needed- low priority
• Not applicable
Safer Opioid Prescribing Guided Self-AssessmentImplemented
before the webinars/workshop
Improved Since
Webinars/Workshop
Still Needs Improve-mentHigh Priority
Still Needs Improve-ment Low Priority
1. Clear assessment of the pain condition:
2. Clear assessment of psychiatric history
3. Clear assessment of substance use disorder
4. Clear assessment of psychosocial history
5. Clear assessment of co-existing use of alcohol or illicit substances
6. Clear assessment of contraindications to opioid prescribing (including pregnancy)
Evaluating CE in the Context of Individual
Performance Improvement
• Participants audit set of charts using data collection
survey
• Intervention based on gaps identified through chart
review
• Participants asked to develop performance
improvement plan
• 3 months later participants asked to re-audit different
set of charts to see if changes implemented
Chart Audit
Question
Select Performance
Metrics
Access and Collect Data
Analyze Data
Obtain Feedback
Conclusion & Document
45
Clinical Audit Tool
• Royal College clinical audit tool and guideline
for physicians to conduct audit with feedback
to assess performance
• http://www.royalcollege.ca/rcsite/cpd/moc-
program/moc-support-tools-resources-e
Level 6: Patient Health Status
Observed
Patient health records
Administrative records
Self-reports
Physician questionnaire
Patient questionnaire
• Individual patient health status
Example of Performance Assessment and
Reported Patient Outcomes
• Safer Opioid Prescribing Webinar
• Brief Pain Inventory Patient Assessment Tool
• Application to Practice exercises
– Use of Practice Tools
– Report on Patient outcomes
Brief Pain Inventory
▪ Have patients complete a
Brief Pain Inventory.
• At initial work-up.
• At each subsequent
visit.
▪ Refer to Practice Tool
Brief Pain Inventory.
(sent by email)
Determine average Pain score/10
Sum score for 9a to 9i=Functional impairment score/90high score = low function
Application to Practice
• Have 5 chronic pain patients complete the Brief Pain
Inventory.
• For each of these patients, indicate:
– Whether this is an initial or follow-up assessment.
– The patient’s pain score out of 10.
– The patient’s functional impairment score out of 90.
– Your management decisions.
• A tracking sheet will be provided.
• Submit electronic report by DATE.
Application to Practice
Patient
Brief Pain Inventory Scores Patient Management Plans
Initial Examor
Follow-up Pain /10
Functional Impairment
/90
Non-pharmacological
treatments recommended
Medications prescribed (if any)
1o Initialo Follow-
up
o Physical Activity
o Self-management program
o Psychological therapy
o Physical therapy
Level 7: Community Health
• Ultimate goal of health care
Quantitative Sources
• Morbidity and mortality rates
– E.g. Incidence of secondary complications
• Adequate control of underlying disease
• Hospitalization and re-hospitalization rates
• Community public health data
Outcome Assessment in
Continuing Professional Development
• Outcome assessments at all levels can be
embedded throughout CPD Programs:
– Before- preparatory work
– During- active learning and assessed outcomes
– After- practice application exercises
Flipped Classroom
• Pre-classroom activities
– Support cognitive work (knowledge and
comprehension)
– Tailoring program to learner needs (survey)
• In-class activities
– Facilitate higher levels of learning (application and
analysis)
– Frees up time for interactive engagement and
introduction of innovative educational models
Flipped Classroom
• Time and resources to create pre-classroom
materials (e.g. videos)
• Educator readiness
– Training or faculty development
– Use of technology
• Learner readiness
– Active vs passive participant
– Perception of increase workload for learners
PRECEDE Model
• Predisposing, Reinforcing, Enabling Causes in
Educational Diagnosis and Evaluation
• Predisposing:
– Getting attention and identification of gaps before the
program
• Reinforcing
– Selecting methods to promote learning and retention
• Enabling
– Planning for application to practice
Multiple Interventions in
Safer Opioid Prescribing• For each webinar:
1. Pre-work – directed reading and guided practice audit based on Managing Chronic Pain Toolkit
2. During - Active learning to Share Practice Experience
3. Postwork - submit tracking sheets describing patient care- Brief Pain Inventory, Calculate MME/day, and Initiate Tapering
Multiple Interventions in
Safer Opioid Prescribing• For the workshop:
1. Pre-work - prepare a challenging case to share
with class
2. During - active learning include calculation of
MME/d and Switching, team consults and
role play
3. Post-work – Implement Structured Opioid
Therapy and (2) Survey of Improved
Implementation of Recommendations
An Active Approach to Learning
with Outcome Measures
15 distinct educational
interventions with outcome
measures over 4 months
Small Group Discussion
Using the PRECEDE model, how would you
embed higher level outcome measures before,
during and after your program?
• Predisposing
• Reinforcing
• Enabling
Take Home Messages
• Good outcomes assessment provides
opportunity for overall program improvement
• Measuring outcomes leads to enhancement in
learning, competence, performance and
potentially patient outcomes
• Use of PRECEDE model in curriculum
planning to link to outcomes
• Consider integrating one new form of
assessment in your next program
•
Selected References
• Lowe, M, Hebert,D and Rappolt, S. Occupational Therapy Now, ABCx of
CTCs: An introduction to Commitments to Change. 2009:11: 20
• Moore D et al. Achieving desired results and improved outcome.
Integrating planning and assessment through learning activities. JCEHP
2009:29:1-15.
• Lockyer, J. M. et al. (2001). Commitment to change statements: A way of
understanding how participants use information and skills taught in an
educational session. JCEHP 2001:21:82-89.
• Cervero RM and Gaines JK. Effectiveness of continuing medical
education: updated synthesis of systematic reviews. JCEHP 2015;
35(2):131–138.
• Cooper et al. Rating medical emergency teamwork performance:
development of the Team Emergency Assessment Measure (TEAM)
Resuscitation 2010;81:446-452.
•