evaluation of course evaluations

EVALUATION OF COURSE EVALUATIONS.

by

Prof. Peter Friis-Hansen, Assoc. Prof. Niels Houbak and Prof. Peder Klit

Department for Mechanical Engineering (MEK),Technical University of Denmark (DTU)

DK-2800 Kgs. Lyngby, Denmark.

Evaluation of courses ?

When is a course a good course?• When the lecturer senses the success.• When most students pass the exam• When the students are happy• When the industry are pleased with the candidates• When the students have learned enough to ….

continue

• Attendance• Percentage passing the course exam• Grade average• Student satisfaction• Lecturer satisfaction• Something else?

How do we measure it? (the engineering approach)

Final Evaluation Questionnaire

• 3 sections. A: The course, B: The lecturers, C: Written comments

• Has been used for more than 15 years, the last 3 years in electronic form.

• Must be completed one week before end of term (discussed at the last lecture)

• Is important for the lecturer and for the department management (study board)

• Information is included in overall evaluations of staffs and studies

The Questionnaire.Section A

1. Are the prescribed course prerequisites adequate?

2. How is the course material?

3. Is the form of the course adequate?

4. A standard course has an average workload of 9 hours pr week; how much time did you spend?

5. How many lectures did you attend?

6. What is your general satisfaction with the course?

7. For courses taught in English; did this influence your outcome?

The Questionnaire.Section B

1. How did the (named) lecturer present the subject?

2. Is the lecturer inspiring?

3. How did you experience the dialog/cooperation with the lecturer?

4. How is the lecturer as supervisor?

5. For group works; did you receive criticism for handed in exercises during the course?

Section C

1. I appreciate2. I criticize 3. I suggest

Answers and Weights: Section A.

1 Prerequisites Too many Suitable Too few None

4 10 0 10

2Teaching material

Very bad Bad Acceptable Good Very good

0 2,5 5 7,5 10

3 Form Less suited Reasonable Well suited

0 5 10

4Time

consumptionMuch less Less Normed More Much more

0 4 10 8 3

5 Participation 0-25 26-50 51-75 76-100

0 2,5 7,5 10

6 Satisfaction Very small Small Acceptable Much Very much

0 2 6 8 10

7 English Very negative Negative Not influenced PositiveVery

positive

0 5 10 10 10

Answers and Weights: Section B.

1 Presentation Very bad Bad Acceptable GoodVery good

0 2,5 5 7,5 10

2 Inspiration NoOnly little

To some extent

YesVery much

0 2,5 5 7,5 10

3Dialog,

CooperationVery bad Bad Satisfying Good

Very good

0 2,5 5 7,5 10

4 Supervision Very bad Bad Satisfying GoodVery good

0 2,5 5 7,5 10

5Criticism of

workNo

Only little

To some extent

YesVery much

0 2,5 5 7,5 10

Post processing.

• For each question an average value is calculated

• Questions are weighed against one another

• Average values are multiplied with weights and summed

• A course/lecturer utility number is the outcome

• This is a subjective one number [0-10] evaluation – far from any truth!

• BUT it may contain some information.

Cumulative Curves.Kummuleret fordeling for MEK kurser - skema A

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0,0 1,0 2,0 3,0 4,0 5,0 6,0 7,0 8,0 9,0 10,0

Nytte

Fra

kti

l

Fordeling for MEK kurser - skema A

0

1

2

3

4

5

6

7

8

9

Nytte

An

tal

Kummuleret fordeling for MEK undervisere - skema B

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0,0 1,0 2,0 3,0 4,0 5,0 6,0 7,0 8,0 9,0 10,0

Nytte

Fra

kti

l

Fordeling for MEK undervisere - skema B

0

2

4

6

8

10

12

14

Nytte

An

tal

Time Evolution: Fall term course curves.

Time Evolution: Fall term lecturer curves.

Evaluation action.Reactions from the study board

• Letters to excellent performing lecturers

• Letters to poor performing lecturers – filtered for good excuses. Important to notice, that the tone is: ”What do think you can do about this situation? Can we be of any help?”

• Easy to locate/spot changes in behaviour (monitoring 50+ courses is hard)

Conclusions:

1. It is always worthwhile to evaluate what you are doing.

2. Evaluations can sometimes be used for something differently.

3. The anonymous comparison allows for lecturers to see his/her own performance relative to other staff members.

4. All lecturers notice that their evaluation result has been processed. It is not only the poor performers that are noticed.

5. The weighing factors indicate what is important when improving the teaching quality.

6. The defined “Utility-system” generates a “dynamic normal”, i.e. conscientious staff will try to improve the teaching quality to obtain a good position on the cumulative curves.

7. Tests have shown that radical changes to the weights are needed to form a noticeable change in the ranking