overview of newer educational techniques and evaluation options · 2012-04-13 · overview of newer...
Post on 27-Mar-2020
4 Views
Preview:
TRANSCRIPT
Overview of Newer Educational Techniques and Evaluation Options
Judy A. Shea, PhD.Associate Dean of Medical Education Research
Professor of Medicine, Clinician-Educator
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Overview
Mission of medical education produce clinicallyMission of medical education – produce clinically competent physicians
Many players:y p y» Learners» Educators » Administrators» Evaluators
V l i il d i t Value similar endpoints» Have different foci, priorities, agendas
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Objectives T i d ti l t h i To review some newer educational techniques To present a framework for thinking about medical
education research/evaluationeducation research/evaluation To review evaluation options/issues To talk a little bit about program evaluation
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Educational TechniquesEducational Techniques
Learner EngagementLearner Engagement“How do we teach?”
C t tContent“What do we teach?”
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Learner Engagement“How do we teach?”
“How of teaching” How of teaching» Active learning» Practice» Spaced learning
Simulation – SPs Essays, writing, reflection Self-assessment Technology – fewer books Less lecture, more hands on
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Content“What do we teach?”
Data access retrie al interpretation Data access, retrieval, interpretation Social and economic sciences I t l kill d i Interpersonal skills and issues “Special topics”
EBM» EBM» Patient safety» Geriatrics women’s health» Geriatrics, women s health
Business of medicineTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
What You (Really) Need to KnowLawrence Summers – NYT – 1/20/12
More abo t ho to process and seMore about how to process and use information and less about imparting it
CollaborationCollaborationNew technologies change how knowledge is
conveyedconveyed“Active learning classrooms”English’s emergence as a global languageEnglish s emergence as a global languageMore emphasis on analysis of data
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Medical Education Research/Evaluation Framework
What don’t I mean?What don t I mean?NOT a conceptual model, e.g.,
Adult learning theorySocial learning theoryBehavioral (economic) theoryTheory of Planned BehaviorTheory of Planned BehaviorTheory of Reasoned Action
Much simpler…pwhat and who are you studying?
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Medical EducationMedical Education Research/Evaluation Framework
WHOWHAT
Focus of ResearchUnit of Analyses Process Outcome
P ( dPerson (student, resident, faculty)
x x
Program (course, residency)
x x
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Focus of Research/Evaluation
Unit of Analyses Process Outcome
P ( t d t T h t d j i A t i h hPerson (student, resident, faculty)
To what do junior faculty/fellows attribute their success and how does mentoring fit in?
Are trainees who have mentors more
successful than peers who do not?g
Program (course, residency)
How can mentoring activities provided in IM
training programs be
Do programs with structured mentoring programs produce
characterized?.
more academic clinicians than those
without?To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Reason for the Framework Clarity Focus
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Evaluation QuestionsEvaluation Questions
WHY?WHY? » What are you trying to do?
HOW?HOW? » What evidence would be convincing?
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Miller’s Pyramid
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Miller’s Pyramid
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
WHY? - Evidence - K,A,S (B) – Typical/Old
Kno ledgeKnowledge» Usually ‘written test’
AttitudeAttitude» Often survey
Skills/BehaviorsSkills/Behaviors» Self-report
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
WHY? Evidence - K,A,S (B) – CurrentKnowledgeKnowledge
» Usually ‘written test’, chart-stimulated recallAttitudeAttitude
» Often survey, observations, peers and patient ratings
S ( )Skills (a.k.a. competencies)» Simulations, technology (e.g., actiwatch)
BehaviorsBehaviors» Claims data, chart review, GPS, time-motion
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
HOW? Evaluations Tools/DevicesHOW? Evaluations Tools/Devices
Sim lation SPs OSCEs Simulation – SPs, OSCEs Essays, writing, reflection S lf t Self-assessment Technology – actiwatch, GPS Portfolios Observation – mini-CEX, DOP, 360o
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
HOW? Primary Data Qualitative Qualitative
» Focus groups» Interviews» Interviews» Observational data
Quantitative» Surveys» ‘tests’
Chart reviews
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
HOW? Secondary Data
Secondary data » Grades, ratings, career choice» Tests – MCAT, NBME, ABIM» AMA database
W b b d l ti» Web-based evaluations
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Ad d Di d fAdvantages and Disadvantages of Typical Methods
Knowledge PE skills
MCQ test ++ -
SP based test -- ++
Teachers/attendings + +
Chart reviews ? ?
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Advantages and Limitations of Four “”Newer” Methods
Obser ed enco ntersObserved encounters360° degree assessments P ti t/t i tPatient/trainee outcomesProcess of care/teaching
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Advantages of Encounter-based Methods
Pose a broad range of patient/trainee problems Pose a broad range of patient/trainee problems Support the assessment of integrated skills Support education Support education Feasible in small work-based practices and training
programs
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Limitations of Encounter-based Methods
Not many encounters will be considered Not many encounters will be considered unsatisfactory
If the trainees-doctors have some control over who examines them and indirectly over the content of the assessment
Standards across practices and programs will not be Standards across practices and programs will not be equivalent
A large scale faculty development effort is neededg y p
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Advantages of 360° AssessmentAdvantages of 360 Assessment
S pported b considerable researchSupported by considerable researchHas positive educational effects
S t lit i t t l» Supports quality improvement tools FeasibleA li bl th tiApplicable across the continuum
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Limitations of 360° AssessmentLimitations of 360 Assessment
Assessor Assessor» Need to have observed the doctor» Differ in stringency and perspective
Context» Clinical settings differ in demands, stresses, etc.
Relationships Relationships» Competition, friendship may influence assessments
Stakes» In high stakes settings, assessments might be inflated
Anonymity is importantTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Advantages of OutcomesAdvantages of Outcomes
P blicPublic» Measure of accountability
Patients trainees health care systemPatients, trainees, health care system» Offers a basis for deciding among
doctors/teachers on quality/efficiencyq y yDoctors/teachers
» Offers assessment tailored to unique practice q p» Based on real work performance
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Limitations of OutcomesLimitations of Outcomes
AttributionAttribution» Good assessment requires that the doctor/teacher
be solely responsible for outcomesCase/content mix and patient/trainee
complexity» Good assessment requires that all face the same
challengeNumbersNumbers
» Reliable assessment requires that many patients/trainees be sampled
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
p p
Advantages of Process MethodsAdvantages of Process Methods
More directl in doctor/teacher controlMore directly in doctor/teacher control » Problems of attribution are reduced
Less directly influenced by complexityLess directly influenced by complexity» HbA1c should be monitored in all diabetics» Time spent teaching» Time spent teaching
For certain processes, case mix is not a problemproblem» Most patients need immunizations
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Limitations of Process Methods
Doing the right things does not ensure a good outcome Doing the right things does not ensure a good outcome Complexity and case mix still have an effect A sizeable number of a teacher/doctor’s A sizeable number of a teacher/doctor s
patients/trainees need to be sampled
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Two more issuesTwo more issues..
DesignsDesignsAmount of data
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Designs Correlational/cohort studies Correlational/cohort studies
» observational» descriptive» descriptive
Experimental studies» intervention» post-intervention observation
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Non-experimental Si l t t t l Single group - post test onlyG1: -------------I-------------O Single group – pre-post test Single group – pre-post testG1: O-----------I-------------O
Extremely common in medical educationMultiples threats to validity
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Quasi experimental
Have a nonrandom control groupG1: UC OG1:________UC________OG2:________I__________O
Probably underutilized in medical education
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Experimental Subjects are randomly assigned to treatment groups Some level(s) of blinding
RG1:_______UC________ORG2: I ORG2:_______I__________O
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Amount of Data
Need adequate number of observations Case specificity Case specificity Sampling/bias Capture more signal than noise Capture more signal than noise
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Where is the sampling error?
MCQ test Wrong content, not enough items
SP based test SP training/specificity
Teachers/attendings Different ideas/ “hawks” and “doves”
Chart reviews A lot does not get recordedChart reviews A lot does not get recorded
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
***Need a lot of observations***
Purposes of Program EvaluationPurposes of Program Evaluation
To impro e programTo improve programTo determine next steps/make decisions
H l d id t l d l f th li i t» Help decide to replace, develop further, eliminate, accredit
To determine effectivenessTo determine effectivenessTo document successTo measure outcomesTo measure outcomes
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Many ModelsMany Models
Logic Model Goal Oriented/Objective-Based (Tyler) Kirkpatrick’s 4-level Framework Continuous Quality Improvement (CQI) and Plan-Do-
Study-Act (PDSA) - Goals-free Evaluation (Scriven)
/ Judicial/Adversary Evaluation CIPP (Stufflebeam) Situated Evaluation Connoisseurship Evaluation (Eisner) Utilization-Oriented Evaluation (Patton)
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Program Logic Model - MERCProgram Logic Model - MERC
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Tyler Model - MERCTyler Model - MERC
Objectives: Methods Content/Specifics Frequency/ PersonTiming
1. Increase their participation in medical education research
Short survey
Retrospective pre/post survey•12 closed-ended dichotomous items participation in medical d i h i i i
6-12 months after completion of MERC
MERC Evaluation Committee to launch and analyze data
activities (research presentations and publications)(outcome evaluation)
education research activities (ie, collaborating in medical education research project, publishing a peer-reviewed publication)
2. Apply medical Short
Open-ended question
6-12 months after completion MERC Evaluation pp y
education research principles from MERC to their daily work(outcome evaluation)
surveyp
of MERC Committee to launch and analyze data
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Plan-Do-Study-Act CyclePlan-Do-Study-Act Cycle
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Overview of 10 Program Evaluation Steps (Workplan)
Step 1: Identify Users Step 1: Identify Users Step 2: Identify Uses Step 3: Identify Resources Step 4: Identify Evaluation Questions/Objectives Step 4: Identify Evaluation Questions/Objectives Step 5: Choose Evaluation Design Step 6: Choose Measurement Methods and
Construct InstrumentsConstruct Instruments Step 7: Address Ethical Concerns Step 8: Collect Data Step 9: Analyze Data Step 9: Analyze Data Step 10: Report Results
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 1: Identify UsersStep 1: Identify Users
Who will use the evaluation?Who will use the evaluation?» Learners» Faculty» Faculty» Workshop developers» Administrators» Agencies» Other stakeholders
What do they want from the evaluation?
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 2: Identify UsesStep 2: Identify Uses
Generally both formative and summativeGenerally both formative and summativeIndividual and program decisionsConsider specific needs of each userConsider specific needs of each user
» Judgments about individuals» Judgments about project management and» Judgments about project management and
processesImprovement or proving?p p g
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 3: Identify ResourcesStep 3: Identify Resources
What time is needed from e er one?What time is needed from everyone?
What personnel is needed?p
What equipment?
What facilities?
What funds?What funds?
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 4: Identify Evaluation Questions/Objectives
These go back to the model chosen butThese go back to the model chosen, butGenerally
» Relate to specific measurable objectives forp j• Learner• Process• Outcomes
» Wise to include some questions that get at what was not anticipated both as strengths and weaknesses
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 4: Identify Evaluation Questions/Objectives – cont.
Evaluation questions should:» Be clear and specific» Congruent with the literature» Focus on outcomes versus process
Outcomes imply change» Outcomes imply change• Workshop will improve educator’s skill
RATHER THAN• How the workshop will be given (process)
» Align with goals and objectivesTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
What are the questions? Interface•Interaction
Process•Ease of use Presentation &
Interaction•Feedback•Clarity•Quality•Organization
Evaluation of Learning
Outcomes
Ease of use•Efficiency•Relevance•Language
Organization
Pedagogy•Instructional method
Organization
?Outcomes
•Knowledge•Attitudes•Behaviors
E l ti
•Instructional method•Structure•Active learning•Learner differences•Objectives ~ methods
Evaluation of Content
Evaluation of Cost
D l t I l t ti M i t
Objectives methods
•Authority•Accuracy•Appropriateness•BreadthD h
Development Implementation Maintenance•Needs assmt•Objectives•Materials
•Staff time•Materials•Recruitment
•Portability•Coordination•Durability
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content. •DepthMaterials•Staffing•Design
Recruitment•Facilities•Hardware
Durability•Tech support
Adapted from Elissavet & Economides (2003)
Step 5: Choose Evaluation DesignsStep 5: Choose Evaluation Designs
What ones are appropriate to the questions?What ones are appropriate to the questions?
» Posttest only X - - Oy• Satisfaction/reactions
P t t P tt t» Pretest-Posttest O - -X- - O• Changes in knowledge/attitudes
» Quasi-Experimental O - -X- - O- - - - - O • Cross-over O - - - - - O - -X - - O
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 6: Choose Measurement Methods pand Construct/Adapt InstrumentsCommon methods
» Rating forms» Self-assessments» Essays» Exams» Questionnaires» Interviews/focus groups» Direct observations» Performance audits» Existing data (AAMC ti i C l JAMA)» Existing data (AAMC questionnaires, Course evals,JAMA)
Collect appropriate demographicsTo help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 7: Address Ethical ConcernsStep 7: Address Ethical Concerns
ConfidentialityAccess to dataConsent Resource allocationSeek IRB approval
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 8: Collect DataStep 8: Collect Data
Timing and response rate
Already existing data collectionAlready existing data collection
Impact on instrument design (e.g. mail vs. web survey)web survey)
Assignment of responsibility
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 9: Analyze DataStep 9: Analyze Data
Plan at the same time as the rest of thePlan at the same time as the rest of the evaluation
Want congruence between question asked and analysis that is feasible
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Step 10: Report ResultsStep 10: Report Results
TimelTimely
Format fits needs of users
Display results in succinct and clear manner
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Other Issues
How many data collection points Sustained change Fairness in ‘taking credit’ Teams
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
Thi I Did N t T lk Ab tThings I Did Not Talk About
U i h ll i d ti Unique challenges in education:» real deadlines and timelines» small samples» small samples» many “competing” curricula» teams» apprenticeship learning
Abundant opportunity and need to conduct medical education evaluation research
To help protect your privacy, PowerPoint prevented this external picture from being automatically downloaded. To download and display this picture, click Options in the Message Bar, and then click Enable external content.
top related