formal appraisal of undergraduates – worth the effort? deborah murdoch-eaton professor of medical...

Formal Appraisal of Undergraduates –worth the effort?

Deborah Murdoch-EatonProfessor of Medical Education

Leeds

http://www.ltsn-01.ac.uk

The workshop

• Who are we?

• What do you want to get out of the day?

http://www.ltsn-01.ac.uk

Proposed Outcomes

• Explored whether formal appraisal might be of benefit for students &/or faculty

• developed a model for undergraduates

• Considered whether it might be applicable in your institution ie “worth the effort”?

http://www.ltsn-01.ac.uk

What is appraisal?

• 1 to 1 interview

• Purpose?

(Sam Scallan review of educational appraisal)

http://www.ltsn-01.ac.uk

Purpose of Appraisal

• Review performance

• Provide opportunity to discuss progress

• Develop personal and professional development needs

• (contribute to University requirements for PDP)

http://www.ltsn-01.ac.uk

Is appraisal different for UG and PG and CPD?

• Purpose ?

• Type of goals set?

• Skills of reflection ?• Students?• Appraiser

• Appraiser skills

http://www.ltsn-01.ac.uk

Group work

• Benefits to student

• Benefits to faculty

• What stage of UG career might it be considered most valuable?

• Who should do the appraisals?

http://www.ltsn-01.ac.uk

• Feedback from groups

http://www.ltsn-01.ac.uk

Benefits to student?

• Individuality• Reassurance re progress and grades• Life at university• Feedback – on more than academic

criteria

• Achieve potential• Model for future ? Reflective practice?

http://www.ltsn-01.ac.uk

“A more positive feeling about how I am doing. I truly believed I was doing badly, and without this appraisal I would have maintained this disheartening and negative feeling”

http://www.ltsn-01.ac.uk

Benefits to faculty?

• Detection

• Alternative view on course

• Enhance student support system?

http://www.ltsn-01.ac.uk

Who gets appraised

• Does it depend on the medical school

• Type of curriculum?

• All students?

• Selected?• Performance• Year• Self-need

http://www.ltsn-01.ac.uk

Who are the appraisers?

• Different from PG and / or CPD model?

• Chosen• Knowledge of curriculum• Interest / understanding

• trained

http://www.ltsn-01.ac.uk

Coffee break

http://www.ltsn-01.ac.uk

methodology

• When do you do it?

• Preparation

• Documentation of outcomes?

http://www.ltsn-01.ac.uk

methodology

• How and when

• Preparatory material• Student• Appraiser

• outcomes

http://www.ltsn-01.ac.uk

Leeds Model

• When• After Easter break

• Who• All 1st and 2nd years, “selected” 3rd years• Is it “optional”?

(Murdoch Eaton & Levene Med Teacher 2004)

http://www.ltsn-01.ac.uk

Student preparation

• Written guidelines

• Preparatory questions

• Summative and formative assessment marks• Individual

• (Study habits survey – data from 2002 + 2003

available)

http://www.ltsn-01.ac.uk

Preparatory questions for students – work / course

• do you think the results you obtained were a fair reflection of the work you put in?

• Think about how you worked – was it always appropriate and efficient? Give an example of where it went well, what you are pleased about and why.

http://www.ltsn-01.ac.uk

Preparatory questions for students work / course

• Is there anything over the year you didn’t do as well in or learn as much from as you could have? How might you approach something like this differently again?

http://www.ltsn-01.ac.uk

Preparatory questions for students – career choice

• Has anything this year changed your views on what being a doctor is about?

• Either made you less or more enthused about your career choice?

http://www.ltsn-01.ac.uk

Preparatory questions for students – personal change

• How would you measure a successful next year? What would you like to do to ensure this happens (both personally and in your UG career)?

• Is there anything else you would like to discuss – personal or professional development?

http://www.ltsn-01.ac.uk

Study Habits Survey

• Who knows how hard a student works?

• What is the “norm”?

http://www.ltsn-01.ac.uk

Study Habits Survey

• One representative week in term 2• Document

• Number of hours private study(all course related activities which are unsupervised)

• Where, with whom

• Employment?• Hours/week on sport / social activities?

(Whittle S R & Murdoch Eaton D G. Study Habits audit. Medical Education 2004 (38) 566-567)

http://www.ltsn-01.ac.uk

Study Habits SurveyNumber of Hours spent on self-directed learning

0

5

10

15

20

25

30

0-5 6-10 11-15 16-20 21-25 26-30 31-35 36-40 >40

Hours per week

Nu

mb

er

of

stu

den

ts

Series1

http://www.ltsn-01.ac.uk

Study Habits SurveyHours spent sports / societies

0

5

10

15

20

25

30

0 1-2 3-4 5-6 7-8 9-10 11-12 >12

Hours per week

Nu

mb

er

of

stu

den

ts

http://www.ltsn-01.ac.uk

Use of Study Habits Survey Results

• Not the absolute values for comparison

• Relative in relation to• Academic results• Outcomes• Personal objectives

• i.e. A diagnostic tool??

http://www.ltsn-01.ac.uk

Personal Tutor involvement

• Sent copy of students results

• Form to complete• Relevant information• Not breaking confidentiality

http://www.ltsn-01.ac.uk

For the Appraisers

• Environment• Information for appraisers

• Student specific: Individual results appraisal record – (yr 2

+ 3)

PT report• General : preparatory guidelines prev results study habits survey

• Appraisal record form to complete with student

http://www.ltsn-01.ac.uk

methodology

• How and when

• Preparatory material• Student• Appraiser

• Recording the outcomes

http://www.ltsn-01.ac.uk

APPRAISAL RECORD FORM – year 1

STUDENT NAME APPRAISER DATE TOPIC OUTCOME NOTES Overview of the year’s academic progress, including assessment outcomes

Positive evaluation: What went well during the year, what achieved of last years’ objectives

Personal learning needs identified

Activities planned / to be considered over next academic year

Student signature Appraiser signature Refer to Dean

http://www.ltsn-01.ac.uk

Qns on bottom of Appraisal record form

• Option to refer to Dean / Sub-Dean of students

• Use of progress file

• Contact / met with Personal Tutor

• Consider for 3rd year appraisal

http://www.ltsn-01.ac.uk

Appraisal record form

What happens to the information?

• Student

• Personal tutor

• Student file

• Signed by both appraiser and student

http://www.ltsn-01.ac.uk

Lunch break

http://www.ltsn-01.ac.uk

Issues raised in appraisal

2 main outcome themes

• altered perception on personal progress

• Altered perception of faculty

…….feel like a person, not just a statistic – the Med school

does care!

http://www.ltsn-01.ac.uk

Video clips

http://www.ltsn-01.ac.uk

Tea break

http://www.ltsn-01.ac.uk

So is appraising undergraduates worth the

effort?Cost effectiveness ?

Students ~ 600Appraisers ~ 28

84 X 3 hour sessions = 252 hours senior academics time

Administration - complex

http://www.ltsn-01.ac.uk

Appraisers perspective

• all would do it again

• Personally rewarding

• Timing about right

• Training of future appraisers?

http://www.ltsn-01.ac.uk

Effect on Students?

Student Feedback

• Questionnaire• Quantitative and qualitative questions

Student Outcomes

• Analysis of Record of Appraisal Outcomes

http://www.ltsn-01.ac.uk

Outcomes – student perspective

Questionnaire – quantitative analysis

response rate (86%)

• 96% well informed about process

• 99% they could be honest in answers• 99% topics covered were relevant• 96% feedback was useful and accurate

http://www.ltsn-01.ac.uk

Outcomes – student perspectiveQuestionnaire – quantitative analysis

(2)

• 94% able to discuss academic problems• 98% appraisal record fair representation

of the points discussed

• 62% appraisal feedback would alter their study habits

• 33% appraisal feedback would alter attitude towards medical career

http://www.ltsn-01.ac.uk

Outcomes – student perspectiveQuestionnaire – qualitative

analysis

• Majority very satisfied with appraisal session

• Perspective of medical school changed• “feel like I’m a person, not just a statistic”• “the Med School does care”

http://www.ltsn-01.ac.uk

“2 things you will take away”

• Personal• Confidence, reassurance

• Work/social balance• Healthy balance

• Study skills advice• Time management, efficiency of working

• Academic progress• C grade is fine

http://www.ltsn-01.ac.uk

“Two things you will take away”

“A more positive feeling about how I am doing. I truly believed I was doing badly, and without this appraisal I would have maintained this disheartening and negative feeling”

http://www.ltsn-01.ac.uk

But does it change anything?

Longer term study

To identify from Appraisal Record Forms• Pattern of advice and areas discussed

• Year 1 appraisal• Year 2 appraisal

• What of year 1 goals are achieved by year 2• What are new goals for year 2, and what

goals are carried forward from year 1 to 2• Correlation between factors

http://www.ltsn-01.ac.uk

Analysis of Appraisal Outcomes

2 cohorts of students completed both 1st and 2nd year appraisals N=511

1. Anonymous student feedback questionnaires(88 % response rate)

2. Appraisal record forms (100% completion)

http://www.ltsn-01.ac.uk

Feedback questionnaire - key points

Overwhelming positive perception of the value of appraisal in both years

• 59.2% overall considered issues covered would alter study habits

• 51% of 2nd years considered they had made changes based on previous years appraisal

http://www.ltsn-01.ac.uk

Practice Points – key advice required early in undergraduates’ career

• Study skills guidance was the commonest need

• Some significant correlations were seen between types of advice required

• Evolution of personal goals - changing course requirements and maturation

• Early opportunity to address healthy work-leisure balance

http://www.ltsn-01.ac.uk

Undergraduate appraisal

• Worth the effort?

• We think so

• Data would suggest that it is, from both students and faculty perspective

• The future?• Later years????

http://www.ltsn-01.ac.uk

• Any questions / comments

Thank you for coming and participating! I hope you found it valuable.

d.g.murdoch-eaton@leeds.ac.uk