© Medical College of Georgia 2007

Recognizing, Evaluating, and Rewarding Educators andEducational Scholarship

Health Professions Educational Research Symposium

January 14, 2007Fort Lauderdale, FL

Ruth-Marie E. Fincher, MDVice Dean for Academic Affairs

Medical College of Georgia School of Medicine

“Education Is Our Core Mission”

Research Service

Patient Care

Education

From Michael Wilkes, 2003

How It Seems

Research Service

Patient Care

Education

Modified by R Fincher, 2004

Making Us Feel Like. . .

Agenda for Presentation• Background

– Teaching & educational scholarship

• Framework– Assessing teaching & learning

• GEA scholarship project– 5 categories of educational activities– Assessment of each category

• Quantity• Quality• Engagement with educational community

– Scholarly approach– Scholarship

Changing Institutional Culture May Be More Difficult than Moving a Graveyard

The Scholarship Culture is Changing

Swanson G. In Tosteson et al. New Pathways to Medical Education.

Cambridge MA, Harvard University Press, 1994

Slide adapted from Malcolm Cox

Teaching and Its Scholarship Are Often. . .

• Overshadowed by– Research– Clinical care

• Undervalued in reward system– Promotion– Tenure– Salary

Institutions Reward What They Value

• Teaching is highlighted in most mission statements

• Teaching and related scholarship must be rewarded if they are truly important to an institution

Recent Evolution of Medical School Environment Caused a

“Crisis of Mission”

Research Teaching Clinical Care

Funding Source

(Is it?)

Cost

(Is it?)

Source

Knowledge Discovery Transmission Application

Value “Opportunity” “Obligation” Critical for survival

Concept from Cuban: How Scholars Trumped Teachers, 2001

Think outside the box!Think outside the box!

Medical Schools Only Unique Mission = Teaching

• . . . .“widespread agreement that those members of the faculty who are most committed to. . . the education of medical students must be supported and rewarded. . . . Faculty appointments, promotion and tenure policies must reflect the changing roles and responsibilities of medical school faculty.”

Whitcomb M: Acad Med 78:117-118, 2003.

Evolution of Concept of Teaching and Scholarship

• Pre-Boyer– Teaching = academic citizenship

• Everyone does it• Little to do with scholarship or promotion

• Boyer– Reframed discussion by

• Challenging concept that “everyone teaches”• Examining teaching as a form of scholarly work

Boyer: Scholarship Reconsidered, 1990.

Continued Evolution: Glassick*

• Common criteria form foundation for all forms of scholarship

Clear goalsAdequate preparationAppropriate methods

Significant resultsEffective presentation

Reflective critique

*Scholarship Assessed, 1997.

AAMC’s Group on Educational Affairs:Promotes Education Across

Continuum of Medical Education

Undergraduate

Continuing

Graduate

RIME

GEA Scholarship ProjectP

hase

119

96-2

000

Phas

e 2

2001

-200

4

Pha

se 3

2005

-200

6

Pha

se 4

2007

- ?

GOAL: Value faculty who support and advance medical education by connecting education, scholarship and academic advancement

GEA Scholarship Project

Phase 1

1996-2000Ph

ase

220

01-2

004

Phas

e 3

2005

-200

6

• Provided examples of activities and evidence

Simpson & Fincher (1999)

• Defined educational scholarship

• Outlined infrastructure needed to support educational scholarship

Fincher et al. (2000)

Pha

se 4

2007

- ?

Characteristics of Scholarship

• Demonstrates the “3 P’s”:– Product that can be reproduced and

built upon– Open to peer review – Publicly disseminated

• Scholarship moves field forward

• Assessment parallels research

Phase 2

2001-2004

GEA Scholarship ProjectP

hase

119

96-2

000

Pha

se 3

2005

-06

• Identified types of educational activities & evidence

Simpson et al. (2004)

• Characterized infrastructure – Academies

Irby et al. (2004)

Pha

se 4

2007

- ?

Education Activity Categories

TeachingCurriculum

Advising/MentoringEducational Administration/Leadership

Assessment of Learner Performance

Simpson D, et al. Acad Med 2004;79:783-790

Infrastructure to Support Education

• Faculty development centers– Career development– Teaching/educational research skills

• Societies or Academies– Recognize accomplishments based on

selection criteria

Pha

se 2

2001

-200

4

GEA Scholarship ProjectP

hase

119

96-2

000 Phase 3

2005-06

GEA Consensus Conf on Educational Scholarship

Defined areas of agreement & concerns in 3 areas

• Category inclusions• Evidence & presentation displays for each category • Areas needing further investigation

Pha

se 4

2007

- ?

Categories of Education Activity

TeachingCurriculum

Advising/MentoringEducational Administration/Leadership

Assessment of Learner Performance

Simpson D, et al. Acad Med 2004;79:783-790

Q2 + Engagement with CommunityContribution to education mission

Quantity Quality

Engagement with education community

Scholarly approachto education

activity

Educational scholarship

Dra

ws

Fro

m

Co

ntr

ibu

tes

To

Scholarly and Scholarship

• Existing literature• Best practices • Resources in field• Colleagues locally

internationally

• Dissemination– Peer reviewed forums

• Local international audiences

• Impacts the field

Scholarly work is influenced by:

Scholarship contributes to the

field:

Documentation: Teaching Category

• Brief Description– Role, activity description

• Evidence of Quantity– Who, what, when, where, how much, how many

• Evidence of Quality– Summary of student/peer evaluations

– Short excerpts from supporting letters

– Invitations to teach outside department or school

– Repeat invitations to teach same group or in course

Teaching Definition and Inclusions

• Is the design and implementation of activities to promote learning

• Includes– Course design– Development of instructional materials– Interactions with students– Formative and summative assessment

Act of teaching is not scholarship

Teaching: Quantity & Quality

Role Yr(s) Quantity Number

Learners

Quality

Family Medicine Clerkship

Seminar

Leader

1997- ~19 hr/

year

8-12/ rotation; 8 rotations/ year

’05-06: “Effective seminar leader”

Mean = 5.41 (7-point scale)*

Give comparative ratings for each year; compare with*Comparative ratings should be given over time;Compare with peer group if possible

*Give comparative ratings for each year;provide comparative ratings with peers if possible

Teaching: Quantity and Quality

Student Evaluations: Individual Faculty Teaching Ratings by Year for "Overall Effectiveness as a Teacher"

1

1.5

2

2.5

3

3.5

4

4.5

5

2003-04 2004-05 2005-06 2006-07

1=In

effe

ctiv

e to

5=

Hig

hly

E

ffec

tive

Faculty

All Faculty

Engagement with Community

• Scholarly approach: Learns from– Educational literature– Best practices

• Scholarship: Contributes to field– Product: Interactive cases– Peer review: MedEdPORTAL– Public: Web

Scholarly Approach

• Teaching - learning connection– Consult education literature– Apply intervention to enhance learning– Observe outcomes– Analyze results– Obtain peer evaluation– Use results to improve teaching

• Product reflected in student learningRichlin L: Scholarship Revisited: Perspectives on

the Scholarship of Teaching. Jossey-Bass, 2001

Evidence of Scholarship

• Product: Instructional material– Interactive, diagnostic decision making cases:

Cough, chest pain

• Describe role• Peer review and public dissemination

– Website: xx hits; xx schools adapted or adopted at least 1 case; representative feedback comments

– Accepted by national peer-reviewed venue; e.g., MedEdPORTAL

Curriculum DevelopmentDefinition and Inclusions

• Longitudinal designed educational activities– Any training level

– Various venues and delivery formats

• Includes – Goals and objectives

– Learning experiences to achieve goals & objectives

– Organization and sequencing to ensure effective learning

– Evaluation of effectiveness

Curriculum DevelopmentActivity and Role

• Title of course– Evidence-based Medicine (Yr. 1 students)

• Role: Course director– Organized all lectures and self-directed

learning exercises for 5 credit hour course• Developed objectives and content• Recruited and trained facilitators• Worked with clinicians and statisticians to

develop illustrative cases

Engagement with Community*

• Clear Goals– Create a new EBM course for all first-year

students that students perceive as clinically relevant. The predecessor course consistently received “very poor student evaluations” and EBM content was “lacking in the curriculum.”

• Adequate Preparation– PhD in Public Health– Review of “best practices: McMaster’s

curriculum, NBME test content– Review of literature; application to new course

*Applying Glassick’s Criteria

Engagement: Glassick

Appropriate Methods Multi-method approach including:

Interactive lecture series Clinical vignettes Abstract followed by article critique

Increased collaboration Small groups co-led by basic scientist/MD Assessment methods: Test questions

assessed application of knowledge

Engagement

Significant Results (Outcomes) Didactic series evaluation

Improved 1 SD from prior offering AAMC Graduation Questionnaire

EBM rating improved from inadequate exposure to appropriate/excessive

USMLE sub-test score Improved from below to above average

OSCE performance on ambulatory practice module Improved

Engagement with Community:Scholarship (Contributing to)

• Effective Presentation (Dissemination)– Results presented to curriculum committee

– Internal review; comparison over last 2 years

– Content replicated in clerkships & residencies

• Reflective Critique (Next steps for continuing improvement)

– Convert course to on-line format to minimize # faculty needed

– Submit to AAMC MedEdPORTAL (Peer review)

Advising and MentoringDefinition and Inclusions

• Advising and mentoring– Educator provides guidance or counsel to

facilitate accomplishment of a learner’s or colleague’s goals

• Advising– Limited time; advisor serves as guide to

enable advisee to achieve her or his goals

• Mentoring– Sustained relationship; mentor and protégée

obtain reciprocal benefits

Advising & MentoringName CW: Medical Student Medicine Resident

Duration/

Process

9/01 - present: 4 mtg/yr; Edit (paper, CV, application to be LCME student liaison)

Outcome MD Received; Paper published in Acad Med; student liaison to LCME

Effective-ness

“I wanted to thank you for … making me the young man I am today…not sure you realize impact you’ve had in my life…”

Engage-ment

Scholarly: Learned from literature, professional development activities, etc. Scholarship: Mentor manual adopted by others; workshop at national meeting

Educational Leadership Definition and Inclusions

• Definition– Exceptional leadership transforms educational

programs and advances field

• Inclusion criteria– Pursues excellence– Evaluates and engages in self-reflection– Builds on work of others – Disseminates results – advances the field– Garners and maximizes resources

Leadership Documentation

• Quantity – Describes nature of activity, duration, rationale

for change, goals, and leader’s role

• Quality – Describes actions, accomplishments, evaluation

and resources garnered/utilized

• Engagement with educational community– Builds on literature and best practices

– Disseminates and reflectively critiques

Leadership ProjectBasic Science/Clinical Integration

• Need– Courses lacked clinical relevance, poorly taught

• Goal– Increase integration across 4 years, clinical relevance, quality

• Methods and preparation– Revised courses – Conducted faculty development during 2004-06– Garnered resources: Support staff and faculty time

• Results (Evaluation)

• Presentation (Peer review and dissemination– 6 peer reviewed activities

• Reflective Critique– Next year plan to…

Rating 2003 2005

Overall 3.4 4.3

1 = Poor5 = Outstanding

Learner Assessment Definition and Inclusions

• Activities associated with measuring learners’ knowledge, skills, and attitudes

• Assessment-related activity(ies) include: – Development

• Identifying and creating assessment processes and tools

– Implementation • Collecting data using processes and tools

– Analysis • Comparing data to correct answer key and/or

performance standards

– Synthesis and presentation • Interpreting and reporting results

Learner AssessmentDocumentation

• Quantity– Type of assessment– Learner population

• E.g., who and how many

– Size/scope• E.g., number of items, frequency of use

– Intended uses of results– Faculty member’s role

• Quality and engagement– Evidence of adherence to Glassick’s criteria

Learner Assessment Example of Presentation

Results: Caveats & Remaining ?s

• Each institution must determine what counts for promotion– Contributions to institution’s mission (Q2)

• Q2 within and across categories• Sustained contributions?

– Contributions to the field (Engagement)• Draw from and contribute to field• Sustained contributions?

– How to value individual vs group contributions

• Core elements education infrastructure

GEA Scholarship ProjectNext Steps

Pha

se 1

1996

-200

0

Phas

e 2

2001

-200

4

Pha

se 3

2005

-06

Dissemination & Infrastructure: Individual & Institutional to Support Educators

• AAMC-GEA Annual & Regional Meetings + Publish

• Institutional Team Workshops

• Individual Faculty Workshops

Phase 4

2007- ?