faculty development issues in evaluation

Faculty Development Faculty Development Issues in EvaluationIssues in Evaluation

DefinitionDefinition Types of ProgramsTypes of Programs Types of Evaluation DoneTypes of Evaluation Done Evidence of SuccessEvidence of Success Believed UtilityBelieved Utility Reason for Continuation of programsReason for Continuation of programs• Need / Evidence of SuccessNeed / Evidence of Success

Risks to current evaluation methodsRisks to current evaluation methods Impetus for future workImpetus for future work

Faculty Development DefinitionFaculty Development Definition

activities designed to activities designed to improve an individual’s improve an individual’s knowledge and skills in knowledge and skills in areas considered essential areas considered essential to the performance of a to the performance of a faculty member……faculty member……

McLeod, Steinert, Nasmith, Conochie, CMAJ-JMAC, 1997McLeod, Steinert, Nasmith, Conochie, CMAJ-JMAC, 1997

Types of Faculty DevelopmentTypes of Faculty Development

Instructional developmentInstructional development Professional development Professional development Leadership developmentLeadership development Organizational developmentOrganizational development

Irby, 1995, from Berquist and PhillipsIrby, 1995, from Berquist and Phillips

Faculty DevelopmentFaculty Development Schemes for Program Evaluation: Schemes for Program Evaluation:

The FacultyThe FacultyTeacher Attitudes/Knowledge/BehaviorsTeacher Attitudes/Knowledge/BehaviorsPedagogyPedagogyPedagogical Content KnowledgePedagogical Content Knowledge

Content Areas, e.g., End of Life, Geriatrics, Content Areas, e.g., End of Life, Geriatrics, Organizational Leadership, Professional SkillsOrganizational Leadership, Professional Skills

Their StudentsTheir Students The SystemThe System

Faculty DevelopmentFaculty Development

Schemes for Program EvaluationSchemes for Program Evaluation:: Reaction (customer satisfaction)Reaction (customer satisfaction) Faculty Learning (change in Knowledge, Faculty Learning (change in Knowledge,

Skills, Attitudes)Skills, Attitudes) Behavior Change (result of program)Behavior Change (result of program) Results (change in the system) Results (change in the system)

Blumberg, Deveau, Medical Teacher, 1995Blumberg, Deveau, Medical Teacher, 1995

Faculty DevelopmentFaculty DevelopmentSchemes for Program Evaluation:Schemes for Program Evaluation: Academic dissemination Academic dissemination

(presentations/publications)(presentations/publications) Product development (materials/workshops)Product development (materials/workshops) Implementation ( #’s trained, maintained Implementation ( #’s trained, maintained

and disseminated)and disseminated) Institutionalization Institutionalization

Blumberg, Deveau, Medical Teacher, 1995Blumberg, Deveau, Medical Teacher, 1995

Faculty DevelopmentFaculty Development

Schemes for Program Evaluation:Schemes for Program Evaluation: Process Evaluation (participant ratings)Process Evaluation (participant ratings) Content Evaluation (Change in knowledge, Content Evaluation (Change in knowledge,

skills, attitudes)skills, attitudes) Outcome Evaluation (long term change)Outcome Evaluation (long term change) Impact Evaluation (Impact on the system) Impact Evaluation (Impact on the system)

Holly, 1992Holly, 1992


The FacultyThe Faculty Their StudentsTheir Students The SystemThe System

Faculty Development in CanadaFaculty Development in CanadaEvaluation methodsEvaluation methods

3 outcomes:3 outcomes: performanceperformance cognitive learningcognitive learning self-assessment re teaching skillsself-assessment re teaching skills

Outcome measures – positive Outcome measures – positive direction, but without statistical direction, but without statistical significance, but small N. significance, but small N.

Nasmith, Steinert, Saroyan, Daile, Franco, TchLng Med, 1997Nasmith, Steinert, Saroyan, Daile, Franco, TchLng Med, 1997

Lessons from Faculty Lessons from Faculty Development in Clinical TeachingDevelopment in Clinical Teaching

Research Evidence for BenefitsResearch Evidence for Benefits Teachers and learnersTeachers and learners

– POSITIVE RATINGS OF EXPERIENCEPOSITIVE RATINGS OF EXPERIENCE– IMPROVED EDUCATIONAL OUTCOMES FOR IMPROVED EDUCATIONAL OUTCOMES FOR

FACULTY - knowledge, skills, and attitudesFACULTY - knowledge, skills, and attitudes• SELF-RATINGSSELF-RATINGS

• RATINGS OF BEHAVIORS ON VIDEOTAPESRATINGS OF BEHAVIORS ON VIDEOTAPES

• KNOWLEDGE ACQUISITION BY TEACHERSKNOWLEDGE ACQUISITION BY TEACHERS

• IMPROVED STUDENT RATINGS OF TEACHERSIMPROVED STUDENT RATINGS OF TEACHERS

– IMPROVED OUTCOMES FOR LEARNERS -- IMPROVED OUTCOMES FOR LEARNERS -- MOSTLY IN HIGHER ED LITERATURE*MOSTLY IN HIGHER ED LITERATURE*

Institutional / Societal Benefits ?Institutional / Societal Benefits ?

Stanford Faculty Development ProgramStanford Faculty Development Program*Murray in Perry and Smart, 1997*Murray in Perry and Smart, 1997Effective Teaching in Higher EducationEffective Teaching in Higher Education

Faculty Development OutcomesFaculty Development Outcomes

Reviewed different types of programs in Reviewed different types of programs in family medicinefamily medicine• FellowshipsFellowships• WorkshopsWorkshops

ResultsResults• Changes measured, but no significant Changes measured, but no significant

performance changesperformance changes• Few publicationsFew publications• Different needs for different faculty, tenure Different needs for different faculty, tenure

vs non-tenurevs non-tenure

Reid, Stritter, Arndt, Fam Med, 1997Reid, Stritter, Arndt, Fam Med, 1997


The FacultyThe Faculty Their StudentsTheir Students The SystemThe System

Faculty Development Faculty Development /Faculty Affairs Offices: /Faculty Affairs Offices: 76 of 125 institutions -> 76 of 125 institutions -> 93 faculty affairs and 21 93 faculty affairs and 21

development officesdevelopment offices

Moahan, et al, Acad Med, May 2002Moahan, et al, Acad Med, May 2002

AAMC - 2000

Faculty Development - EvaluationFaculty Development - Evaluation

Faculty development Faculty development offices focused on offices focused on use of services and use of services and mentoringmentoring


Faculty Development Faculty Development /Faculty Affairs Offices: /Faculty Affairs Offices: 76 of 125 institutions -> 76 of 125 institutions -> 93 faculty affairs and 21 93 faculty affairs and 21

development officesdevelopment offices


AAMC - 2000

Faculty Development in CanadaFaculty Development in Canada

1986 – positive commitment – poor faculty 1986 – positive commitment – poor faculty participationparticipation

Recommendations: Recommendations: • establish committeesestablish committees• fund activitiesfund activities• Recognize participation in promotion/tenureRecognize participation in promotion/tenure

1986-96: Major changes:1986-96: Major changes:• Health care financingHealth care financing• DownsizingDownsizing• Inpatient -> outpatientInpatient -> outpatient• Multi-institutional and multidisiplinary careMulti-institutional and multidisiplinary care• Curriculum reformCurriculum reform

– Less stress on information acquisitionLess stress on information acquisition– Problem solving / interpersonal skills/attitudesProblem solving / interpersonal skills/attitudes


Stanford Faculty Development ProgramStanford Faculty Development Program


All 16 schools responded All 16 schools responded 16 with Faculty Development Program vs 12 in 198616 with Faculty Development Program vs 12 in 1986

• Sabbatical programsSabbatical programs• Grants to improve instruction/ courses, with major Grants to improve instruction/ courses, with major

emphasis on information technology/electronic mediaemphasis on information technology/electronic media• Awards for teaching excellenceAwards for teaching excellence

Increasing feedback on teaching from faculty or Increasing feedback on teaching from faculty or administrators (13/16 schools)administrators (13/16 schools)

Systematic review of teaching by administrator- 6 Systematic review of teaching by administrator- 6 schoolsschools

Senior faculty work with junior faculty – 2 schoolsSenior faculty work with junior faculty – 2 schools Workshops on management / research / scholarshipWorkshops on management / research / scholarship Fac Devel committees in 9 schoolsFac Devel committees in 9 schools ½ of schools indicate inadequate budget½ of schools indicate inadequate budget Outcomes reported at 4 schoolsOutcomes reported at 4 schools




Most effective practicesMost effective practices• Teaching instructionTeaching instruction• Use of computers/information mngtUse of computers/information mngt• Sabbatical leavesSabbatical leaves• Teaching AwardsTeaching Awards• Inexpensive faculty developmentInexpensive faculty development

Local faculty development Local faculty development stimulated by clinicians with M Ed stimulated by clinicians with M Ed Degrees Degrees



Stimuli for changeStimuli for change• Curricular change, needing more Curricular change, needing more

tutorstutors• Curricular change – Uncertainty in Curricular change – Uncertainty in

facultyfaculty• Changes in health care – Changes in health care –

examination of career and personal examination of career and personal goalsgoals




Prognosis - - Good; Prognosis - - Good; Collegiality and Institutional Collegiality and Institutional SupportSupport

Problem – lack of evaluation Problem – lack of evaluation on impacton impact

Need to identify outcomes and Need to identify outcomes and methodsmethods



Research CommentaryResearch Commentary– Hard to do controlled experimentsHard to do controlled experiments– Few measures of pre-post changeFew measures of pre-post change– Few measures of long-term changeFew measures of long-term change–Mostly participant ratingsMostly participant ratings– Few measures of actual behavioral Few measures of actual behavioral

change in facultychange in faculty– Difficult for measures of changes in Difficult for measures of changes in

student learningstudent learning– Little research on institutional changeLittle research on institutional change

Stanford Faculty Development CenterStanford Faculty Development Center

Faculty Development OutcomesFaculty Development Outcomes

Are the data hidden in the process we are Are the data hidden in the process we are observing?observing?• Curricular Change?Curricular Change?• Emphasis on Teaching?Emphasis on Teaching?• New Teaching Methods?New Teaching Methods?• New Observations of Faculty Needs?New Observations of Faculty Needs?• Requirements for Teaching Improvement Requirements for Teaching Improvement

Methods by Accrediting Agencies?Methods by Accrediting Agencies? Areas for Further WorkAreas for Further Work

• Further Enhancement of Pedagogy – Improve Further Enhancement of Pedagogy – Improve the bestthe best

• Pedagogical Content Knowledge (Shulman)Pedagogical Content Knowledge (Shulman)• Needs of the ProfessionNeeds of the Profession

Types of Faculty DevelopmentTypes of Faculty DevelopmentAreas for Further WorkAreas for Further Work

Instructional developmentInstructional development Professional development Professional development Leadership developmentLeadership development Organizational developmentOrganizational development

Irby, 1995, from Berquist and PhillipsIrby, 1995, from Berquist and Phillips

Types of Faculty DevelopmentTypes of Faculty DevelopmentYour BeliefsYour Beliefs

Instructional developmentInstructional development Should be improved? Should be improved? Can be improved? Can be improved? Has been improved?Has been improved?

Professional developmentProfessional development Should be improved? Should be improved? Can be improved? Can be improved? Has been improved?Has been improved?

Leadership developmentLeadership development Should be improved? Should be improved? Can be improved? Can be improved? Has been improved?Has been improved?

Organizational developmentOrganizational development Should be improved? Should be improved? Can be improved? Can be improved? Has been improved?Has been improved?


Current Challenges/OpportunitiesCurrent Challenges/Opportunities• INSTITUTIONAL INSTITUTIONAL

– Other AlligatorsOther Alligators– Attitude - good enough mentality?Attitude - good enough mentality?– TIME / MONEY/ ADMINISTRATIVE SUPPORT?TIME / MONEY/ ADMINISTRATIVE SUPPORT?– EVIDENCE FOR NEED: Present, ? compellingEVIDENCE FOR NEED: Present, ? compelling– WHO IS GOING TO PAY? WHO IS GOING TO PAY?

• TEACHER’S THEMSELVESTEACHER’S THEMSELVES– Philosophical/practical/self-assessment/Philosophical/practical/self-assessment/

• METHODS: Applicabililty to current METHODS: Applicabililty to current environmentenvironment


Areas for Increased Emphasis in Areas for Increased Emphasis in Faculty DevelopmentFaculty Development

INSTITUTIONAL BENEFITSINSTITUTIONAL BENEFITS– IDENTIFICATION OF NEW AREAS FOR IDENTIFICATION OF NEW AREAS FOR

IMPROVEMENT - Consolidation seminar - ->> IMPROVEMENT - Consolidation seminar - ->> • STRUCTURAL RECOMMENDATIONSSTRUCTURAL RECOMMENDATIONS

• APPLICATION OF KNOWLEDGE APPLICATION OF KNOWLEDGE RECOMMENDATIONS FOR INSTITUTIONS, E.G., RECOMMENDATIONS FOR INSTITUTIONS, E.G., MDM POLICIES, PATIENT CAREMDM POLICIES, PATIENT CARE

• EDUCATIONAL RECOMMENDATIONS, E.G. EDUCATIONAL RECOMMENDATIONS, E.G. TEACHING RESOURCES, ADMINISTRATIVE TEACHING RESOURCES, ADMINISTRATIVE RESOURCESRESOURCES

SOCIETAL BENEFITSSOCIETAL BENEFITS– IMPROVED OUTCOME FOR LEARNERS -->> IMPROVED OUTCOME FOR LEARNERS -->>

FUTURE PHYSICIANS/SCIENTISTSFUTURE PHYSICIANS/SCIENTISTS


Lessons from Faculty Development in Clinical Teaching

Data vs Belief

Importance of Faculty DevelopmentImportance of Faculty Development Needs of Educators - Needs of Educators -

Academic / Community Academic / Community Topics of interest and importance Topics of interest and importance

(Content and process) (Content and process) Benefits of Development MethodsBenefits of Development Methods Current Challenges and OpportunitiesCurrent Challenges and Opportunities

Importance of Faculty DevelopmentImportance of Faculty Development Needs of Educators - Needs of Educators -

Academic / Community Academic / Community Topics of interest and importance Topics of interest and importance

(Content and process) (Content and process) Benefits of Development MethodsBenefits of Development Methods Current Challenges and OpportunitiesCurrent Challenges and Opportunities


faculty development issues in evaluation

Documents