Evaluating a Student Rating of Teaching Form

John Ogier

Survey & Testing Unit (STU)

University Centre for Teaching & Learning (UCTL)

john.ogier@canterbury.ac.nz

Ph 64-3-364-2850

A case of quality not being assured!

• “No single student rating item, nor set of related items will be useful for all purposes”

Cashin (1995)

•“Student ratings should be used to make only crude judgements of instructional effectiveness (exceptional, adequate, and unacceptable).”

D’Apollonia & Abrami (1997)

UOC Student Surveys• Course

– Organisation• a well organised course

– Stimulation• helped to stimulate my

interest in the course area

– Workload• The overall workload in this

course was reasonable

– Difficulty• The level of difficulty of this

course was reasonable

– Overall• Overall, this was a good

quality course

• Teaching– Organisation

• classes were well organised

– Communication• able to communicate ideas

and information clearly

– Interest• stimulated my interest in the

subject

– Attitude• attitude towards students has

been good

– Overall• Overall, the lecturer is an

effective teacher

Effective?

• How appropriate are these 5 Likert scale questions?– Do they define a “good

teacher”?– Is that measure reliable?

• 1-5 scale appropriate?– Some cultures are use to

“1” being a good score– Is 3 really “neutral”?

• Students circle 2 no’s!

• Students circle a column!!

• Is it correct to use the mean of a discontinuous scale?

• Median rather than mean?

Other survey “quality” issues

• Was the class told about it in advance?• What extra questions were asked?• When in the semester was the survey held?• What time was the survey held?• How many other surveys were given out?• Who administered the survey?• How long were they given to complete it?• Is it really anonymous?• Ad hominem comments!

Possible factors

• Course

Organisation

Stimulation

Workload

Difficulty

Overall

• Teaching

Organisation

Communication

Interest

Attitude

Overall

Correlation of Q5 (Overall) mean against the means of Q1, Q2, Q3 & Q4

weighted by responses (capped at 50)

Organisation Communication Interest Attitude

R2 = 0.7441 R2 = 0.9222 R2 = 0.8310 R2 = 0.6580

The lecturer is able to communicate ideas and information clearly

Overall, the lecturer is an effective teacher

Scatterplot & Regression Line- Overall v Communication

Where to now?• Difficulties faced by tertiary NESB students in English language lectures

• Cosmopolitan UOC academics – both NESB & ESB students must cope!

• Finegan, T.A., & Siegfried, J.J. (2000). Are student ratings of teaching

effectiveness influenced by instructors’ English language proficiency?

– they found in an introductory Economics course that “the student

ratings of ESL instructors are, on average, about 0.4 points lower,

on a scale of 1 to 5, than the student ratings of native English

speaking instructors.” The American Economist, 44, 17-29.

NESB by Subjective Classification

• Surveys (UGrad, 2 or more responses) - 7072 from ESB lecturers and 524 from NESB lecturers

Means with 95% Conf.

ESB NESB

Q2 - Comm 3.810 ± 0.002 3.216 ± 0.009

Q5 - Overall 3.922 ± 0.002 3.496 ± 0.009

Q5 Difference ~ 0.43, Q2 ~ 0.60

A similar difference, but…

• What about across

– Faculties?

– Levels?

– Departments?

Breakdown of surveys

Number of Cases within grouping variables (and no. of lecturers surveyed)

  Arts Commerce Engineering Science Law

Level Y ESL N Y ESL N Y ESL N Y ESL N Y ESL N

1(Lect)

14(8)

476(153)

12(5)

98(31)

0 38(15)

42(12)

585(167)

0 29(10)

2 (Lect)

40(16)

678(196)

24(9)

213(62)

22(6)

245(81)

89(23)

713(206)

0 88(18)

3 (Lect)

31(13)

517(151)

46(14)

267(83)

49(15)

387(96)

98(24)

820(218)

0 193(29)

4 (Lect)

0 0 0 0 57(17)

537(148)

0 0 0 0

Faculty Means

Why the faculty effect?

• Cashin (1990) mentions a hypothesis that– students’ quantitative skills are less well

developed than their verbal skills so that quantitative based courses are more difficult for students and are more difficult for lecturers to teach.

By Level

Some relevant student commentsThe following student comments from relevant courses put ‘quantitative skills’ into perspective:

~ “He did the best he could but his accent was still very strong, and made a complex subject even harder to understand.”Ratings: ‘Comms’ = 2, ‘Overall’ = 3~ “.. we should have loud, clearly spoken English from the lecturer as otherwise you have to concentrate too hard on just understanding the lecturer.”Ratings: ‘Comms’ = 1, ‘Overall’ = 1

Compare a NESB student rating an ESB lecturer ~ a strong accent, ~ “It’ll be very interesting if the lecturer can speak in Thai.” Ratings: ‘Comms’ = 2, ‘Overall’ = 2.

Implications

• For a “Transmission model” of teaching, do we need anything more than a Communication question!!! ?– Need questions that cover the important aspects of

teaching (or courses) with an appropriate weighting.• NESB lecturers may have a tougher time gaining

promotion and tenure.– Assistance with language?– Alternative communication methods?

• Assisting ESB & NESB students to understand and cope?

The NESB student’s view

• Econ104 ~ 33% NESB Students• Online survey – 353 responses ESB, 165 NESB• Overall – ESB 4.58 ± 0.07, NESB 4.28 ± 0.10

– “... And think about the language he use in the test paper some words I even do not know and if he want to use it then use it in his lecture. ECON test is not a word game its aim should be test people how well they learn for ECON!”

Ratings: ‘Comms’ = 3, ‘Overall’ = 4

Questions?

John Ogierjohn.ogier@canterbury.ac.nz

Ph 64-3-364-2850