australasian higher education evaluation forum (aheef) 2008 "evidence based decision making:...
TRANSCRIPT
Australasian Higher Education Evaluation Forum (AHEEF) 2008"Evidence based Decision Making: Scholarship and Practice"
2-3 October, 2008Australasian Higher Education Evaluation Forum (AHEEF)
2008"Evidence based Decision Making: Scholarship and Practice"
2-3 October, 2008
eVALUAte: development and validation of a
teaching survey
Beatrice Tucker, Beverley Oliver and Julie-Ann Pegden
eVALUate surveys for student feedback
eVALUate unit survey (for all current students enrolled in
course work units) eVALUate teaching survey(enables students to give feedback to
individual teachers on their request)
Literature Review• Systems for Measuring Teaching Quality –
controversial, validity is questionable(Pounder 2007, Kulik 2001, Kember & Leung 2008, Remedios &
Lieverman 2008, Burdsal & Harrison 2008)
• Used for different purposes – teacher development, promotion, awards– for improving quality of courses, quality audits,
benchmarking, funding allocation – student choice of study (Tucker, Jones, Straker, & Cole, 2003, Spencer & Schmelkin, 2002,
Ballantyne, 2003; Ory, 2000, Department of Education Science and Training, 2004, Carey & Gregory, 2003)
• Students and teachers have differing opinions about what they perceive to be good teaching (McLean 2001)
eVALUate surveys at Curtin
eVALUate unit survey– for unit review, annual and
comprehensive course review, student choice of units, benchmarking, promotions
eVALUate teaching survey - for teacher reflection and development,
promotions, awards
Development of survey
• 2003 – committee – students – teaching academics– Deans of teaching and learning– academics with expertise in survey design
and evaluation systems (statisticians)
• Literature was reviewed – excellent teacher characteristics, current
instruments used• Unit and teaching items included in
one survey Oliver, Tucker, Gupta, & Yeo (2008)
Method - focus group
StudentsManagers
Teaching academics
de Bono's Six Thinking Hats
Method - item development
Characteristics of a good teacher
Focus Group Literature Other Surveys
Item
eVALUate teaching survey1. Appears knowledgeable in this subject area
The teacher seems to have a good understanding of the subject.
2. Is enthusiastic in teaching this unitThe teacher makes the subject interesting and conveys his or her enthusiasm for the subject.
3. Is well organisedThe teacher has material prepared on time, is punctual and structures activities in ways that help learning.
4. Communicates clearlyThe teacher is easy to understand in face-to-face, online, written and other formats and explains concepts clearly.
5. Is approachableThe teacher encourages students to ask questions and seek help.
6. Provides useful feedbackThe teacher provides timely and helpful feedback so you can learn.
7. Is an effective teacherOverall, this teacher helps you to learn.
eVALUate teaching survey
8. Please comment on [the teacher]'s teaching strengths.
9. Please comment on how you think [the teacher] might improve the teaching and learning in this unit.
Testing and statistical analysis
• Piloted (x4) to student groups in 2003-4 tested for face validity
• Pilot 4 - Feedback from teaching staff and students
Surveys separated• Trialled to all Faculties in 2006 • Factor analysis and Rasch analysis –
tested the categorical scale and psychometric properties of the items (instrument measures multiple teacher attributes and not just one dominant trait)
How students give feedback
Uptake by teachers
0
200
400
600
800
1000
1200
1400
1600
Sem 1 06(trial)
Sem 2 06 Sem 1 07 Sem 2 07 Sem 1 08
No
. of
su
rve
ys
/ te
ac
he
rs
Teaching surveys Teachers
Teaching surveys drive unit survey response rates
0
10
20
30
40
50
60
70
80
90
100
Sem 2 06 Sem 1 07 Sem 2 07 Sem 1 08
Re
sp
on
se
ra
te (
%)
Without teaching evaluations With teaching evaluations
** * *
*significant differencep<0.001
Conclusion• All teaching staff are required to conduct a
teaching survey annually
• Teaching surveys help drive unit response rates
• Teaching eVALUate reports are used to reward staff: for promotion and awards