matt zackoff, md melissa klein, md, med jennifer o’toole, md, … › docs › meetings ›...

Matt Zackoff, MD

Melissa Klein, MD, MEd

Jennifer O’Toole, MD, MEd

Have documented that they have no financial

relationships to disclose or Conflicts of Interest (COIs) to resolve.

Matt Zackoff, MD

Melissa Klein, MD, MEd

Jennifer O’Toole, MD, MEd

All presenters have documented that their presentation will

not involve discussion of unapproved or off-label,

experimental or investigational use.

Roadmap

• Introductions

• What is a good teacher?

• History of evaluation of teaching

• Introduction of teaching assessment tools

• Practice using the teaching assessment tools

• Implementing one in your own institution

Welcome

• Meet the presenters

• Meet the audience

EVALUATING EDUCATORS: History & Challenges

Let’s Watch Some “Famous” Educators

The Good Teacher

We know a good teacher when we see

it, but how do we create good

teachers?

What Makes A Good Teacher?

Well accepted standards exist for what makes

an excellent medical educator

Skills Of A Good Teacher

SKILL EXAMPLES

Promoting a conducive learning

environment

Respect, learner stimulation, appropriate supervision, control

of stressors in the environment

Engagement of learners

Teaching techniques and activities (adult learning theory),

goal setting, promotion of clinical reasoning, AV materials

Appropriate assessment of

knowledge, skills, and attitudes

Appropriate assessment methods and tools, formative and

summative assessment, determination of supervision level

Facilitating learner’s educational

Triggering dialogue about learner’s goals, allowing

opportunities for learner’s to meet those goals

Providing feedback to learners Providing constructive feedback, promoting learner self

assessment, creating a plan based on feedback

Fostering self directed and life

long learning

Encouraging questions, assistance with finding appropriate

resources, acting as a role model

Self-reflection on one’s teaching

skills

Reviewing teaching evaluations, determining areas for self

improvement, actively sharing approaches and Ideas with

others

Bing-You RG, Lee R, Trowbridge RL, Varaklis K, Hafler JP. Commentary: principle-based teaching competencies. Journal of graduate

medical education. Sep 2009;1(1): 100-103.

Khandelwal S, Bernard AW, Wald DA, et al. Developing and assessing initiatives designed to improve clinical teaching performance.

Academic emergency medicine : official journal of the Society for Academic Emergency Medicine. Dec 2012;19(12):1350-1353.

Historical Needs In Medical Education

Many residents and faculty members have not received

formal training in educational theory or teaching

Faculty Development Programs

Resident as Teacher Curricula

Do We Do A Good Job Assessing These Skills?

The Challenges Of Evaluating Medical Educators

• Many evaluations of medical educators are

done by junior learners

– Focus on superficial skills

– Objective assessment challenging

– Often use Likert scales

– Limited training for learners on evaluation skills

• Little opportunity to show progressive in

teaching skills over time

– Everyone is a 4-5/5

A Cincinnati Example

Large Group Discussion

• How are resident and faculty teaching skills

evaluated at your institution?

• What are some of the positive aspects of your

evaluation process?

• What are the challenges of your evaluation

process?

Let’s Watch Another Educator In Action

• Questions to ponder

– What feedback would you provide this educator

about the teaching you observed?

– What specific recommendation would you give

them to improve their teaching skills?

– What rating would you give this educator?

• Novice

• Competent/Intermediate

• Expert/Master

Let’s Watch Another Educator In Action

• Questions to ponder

– What feedback would you provide this educator

about the teaching you observed?

– What specific recommendation would you give

them to improve their teaching skills?

– What rating would you give this educator?

• Novice

• Competent/Intermediate

• Expert/Master

OBJECTIVE ASSESSMENT

OF EDUCATORS Development Of An Objective Structured

Teaching Evaluation

Historical Needs In Medical Education

Competency-based evaluation of clinical reasoning,

patient interviewing, and procedural skills in

undergraduate and graduate medical education

Development of the Observed Standardized Clinical

Examination (OSCE) - validated in the literature

The Leap

Clinical Reasoning

Patient Interviewing

Procedural Skills

Feedback

Bedside Teaching

Didactic Instruction

Observed Standardized

Clinical Examination (OSCE)

Observed Structured

Teaching Evaluation (OSTE)

Goals of an Objective Assessment

• Easy to use

• Reproducible

• Objective measure of observable skills

• Foundation in consensus behaviors/skills

• Delineation of a spectrum of performance

• Translates to actionable feedback

• Applicable over the training continuum

Development of a Teaching Assessment Tool

1. Identify the Competency

2. Foundation in consensus

competencies

3. Define an EPA

competencies

3. Define an EPA

4. Determine skills/traits

competencies

3. Define an EPA

5. Characterize observable

behaviors

competencies

3. Define an EPA

5. Characterize observable

behaviors

6. Align behaviors with

gradient of competency

Development of an OSTE

Teaching

More specifically looked at brief one-on-one didactic

between an intern and 3rd year medical student

Development of an OSTE

2. Foundation in consensus competencies

3. Define an EPA

Proposed

ACGME Core

Competencies Pediatric Milestones

Teaching Medical Knowledge Demonstrate sufficient knowledge of the basic and clinically supportive sciences appropriate to

pediatrics

Practice-based Learning

and Improvement Identify strengths, deficiencies, and limits in one’s knowledge and expertise

Develop the necessary skills to be an effective teacher

Participate in the education, of patients, families, students, residents, and other health

professionals

Take primary responsibility for lifelong learning to improve knowledge, skills, and practice

performance through familiarity with general and experience-specific goals and objectives and

attendance at conferences

Interpersonal and

Communication Skills Communicate effectively with patients, families, and the public, as appropriate, across a broad

range of socioeconomic and cultural backgrounds

Communicate effectively with physicians, other health professionals, and health related agencies

Professionalism Demonstrate a sense of duty and accountability to patients, society and the profession

TEACHING ASSESSMENT

TOOL EXAMPLE

Brief Didactic

Brief Didactic Teaching Assessment Tool

competencies

3. Define an EPA

TEACHING SKILL

Setting the tone and providing

context

Covering key content areas of a

topic in an effective and

efficient manner

Reviewing and adapting to the

needs of a learner

3. Define an EPA

5. Characterize observable behaviors

6. Align behaviors with gradient of competency

TEACHING SKILL Level of

achievement (Circle below where the

majority of boxes are

marked on the right)

Observed Behaviors (Check a box after viewing these behaviors during the OSTE)

Setting the

tone and

providing

context

Does not appear interested in teaching OR enthusiastic

about the topic being presented

Immediately begins conveying information without first

priming the session with a case example or shared

patient experience

Appears interested in teaching OR enthusiastic about

the topic

Prime the session with a case example or shared patient

experience without learner involvement

Appears interested in teaching AND enthusiastic about

the topic

Primes the session with either brief case or reference to

a shared patient experience with learner involvement

and discussion

3. Define an EPA

Setting the

tone and

providing

context

patient experience

the topic

and discussion

3. Define an EPA

Setting the

tone and

providing

context

patient experience

the topic

and discussion

3. Define an EPA

Setting the

tone and

providing

context

patient experience

the topic

and discussion

Covering key

content areas

of a topic in

an effective

and efficient

manner

Appears rushed

Difficulty navigating the topic so content appears disorganized

Unable to cover prior stated goals or topic of the session

At times appears rushed

At times has difficulty navigating the topic but overall content is

presented in an organized manner (i.e. no use of transitions, takes

more than one attempt to convey a point in a clear manner)

Able to superficially cover stated goals or relevant topics of the

session

Content is conveyed in a calm manner at a manageable pace

Smooth navigation of the topic with content presented in an

organized manner

Able to thoroughly cover stated goals or relevant topics of the

session

Covering key

content areas

of a topic in

an effective

and efficient

manner

Appears rushed

session

organized manner

session

Covering key

content areas

of a topic in

an effective

and efficient

manner

Appears rushed

session

organized manner

session

Covering key

content areas

of a topic in

an effective

and efficient

manner

Appears rushed

session

organized manner

session

Reviewing

and adapting

to the needs

of a learner

Exhibits a teacher-centered approach by not assessing learner

engagement or comprehension throughout the session

Does not address learner difficulty with a term or component of a

Minimally exhibits a learner-centered approach to assess learner

engagement or comprehension throughout the session (1-2

actions)

Superficially assesses learner engagement or comprehension (i.e.

“make sense, sound good, etc.”, use of only pointed questions)

Acknowledges cues of learner difficulty with a term or

component of the topic, BUT does NOT adjust terminology or

teaching style

Continually exhibits a learner-centered approach by assessing

learner engagement and comprehension throughout the session

(3+ actions)

Thoroughly assesses learner engagement or comprehension (i.e.

use of back and forth discussion, learner provides considerable

content)

Acknowledges AND effectively addresses learners who struggle,

and adapts accordingly

Reviewing

and adapting

to the needs

of a learner

actions)

teaching style

(3+ actions)

content)

Reviewing

and adapting

to the needs

of a learner

actions)

teaching style

(3+ actions)

content)

Reviewing

and adapting

to the needs

of a learner

actions)

teaching style

(3+ actions)

content)

LET’S PRACTICE Brief Didactic with Teaching

Assessment Tool

Rate The Resident’s Teaching Skills

• Assess the resident’s teaching behaviors on

the brief didactic teaching assessment tool

• Share at your table

• Large group report out

LET’S PRACTICE Inpatient Rounding Teaching

Assessment Tool

Let’s Practice

• Review the observable teaching behaviors in the rounding teaching assessment tool

• The video you are about to see is of an inpatient general pediatrics team on rounds

• Team composition: – 3rd Year Resident Team Leader

– Intern

– 3rd year medical student

• Imagine the attending’s actions are representative of 5 patient encounters

Meet Our Actors

3rd Year

Resident

Team Leader

3rd Year

Medical

Student

Intern

Questions to Consider

• Where would you rate the 3rd year resident’s

teaching skills?

• What feedback would you give this educator

on the teaching behaviors you observed?

• What specific steps would you suggest he

takes to improve his teaching skills?

• How would you rate his educator level?

– Beginner

– Intermediate/Competent

– Master/Expert

ADAPTING FOR

YOUR INSTITUTION

Keys to Successful Implementation

• What you would do if you were launching a

formal teaching assessment program?

– Where would you launch it?

• Inpatient, outpatient, conferences

– Who is your target teacher population?

• Interns, Senior Residents, Fellows, Faculty?

– Who will you need on your team?

Key Stakeholders

• IDENTIFY:

People or groups who:

– Affect/influence the

project

– Are affected by the

project

– Have interest in the

conclusion

• PRIORITIZE

Communication

• Change can be difficult

• More likely to accept change if they view

communication as honest

– Up front about reasons for program & goals

– Open to feedback

• Prepare people to receive the message

– Try not to be too abrupt

– Adjust communication based on audience

– Ensure the message is “received” and not just

“sent”

Feasability

• Plan ahead

• Have a timeline

• Consider the IRB

• If you are involving residents, ensure it fits

into workflow to maximize buy in

The Budget

• Money

• Other considerations

– Time to develop & implement project

– Time for participation from target learners,

evaluators and other team members

– Materials

– Space

SMALL GROUP ACTIVITY

Adapt The Evaluation For

Your Program

Questions?

Evaluations

matt zackoff, md melissa klein, md, med jennifer o’toole, md, … › docs › meetings ›...

Documents