instructor course evaluations (ices) at aub

26
1 Instructor Course Instructor Course Evaluations (ICEs) at Evaluations (ICEs) at AUB AUB Development, Research, Faculty and Student Perceptions Karma El Hassan, PhD., OIRA

Upload: alma

Post on 12-Jan-2016

37 views

Category:

Documents


0 download

DESCRIPTION

Instructor Course Evaluations (ICEs) at AUB. Development, Research, Faculty and Student Perceptions Karma El Hassan, PhD., OIRA. Outline. History of ICE Development at AUB Processing of ICEs Summary of Research on: Student Evaluations _ ICEs Student Perceptions: Survey Results - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Instructor Course Evaluations (ICEs) at AUB

1

Instructor Course Evaluations (ICEs) Instructor Course Evaluations (ICEs) at AUBat AUB

Development, Research, Faculty and Student Perceptions

Karma El Hassan, PhD., OIRA

Page 2: Instructor Course Evaluations (ICEs) at AUB

2

OutlineOutline

History of ICE Development at AUB Processing of ICEs Summary of Research on:

– Student Evaluations

_ ICEs Student Perceptions: Survey Results Faculty Perspective:

– E-mail exchanges– Survey results

Page 3: Instructor Course Evaluations (ICEs) at AUB

3

I. Development of ICEsI. Development of ICEs

The ICE forms in use were developed four years ago in collaboration between faculty and OIRA.

ICE Form includes items covering- instructor, course, learning outcomes (core), - additional items.

Based on review of the literature, OIRA proposed set of core items that were discussed, revised, and finally approved by various faculties.

As to additional items, faculties first decided on list of categories relevant to their courses, then selected relevant items from item bank provided by OIRA.

Page 4: Instructor Course Evaluations (ICEs) at AUB

4

II. Processing of ICEs - 1II. Processing of ICEs - 1

Obtain initial course lists from Banner. As Banner is not updated with more recent information or

changes to schedule, we request deans’ offices for the information.

Get initial course/instructor/section lists from dean's office of every faculty.

Verify this with respective departments, as sometimes departments do changes like pooling/canceling sections, assigning different instructors, etc.

Finalize our data base of course/instructor/section #. Use this data base to issue envelope labels (later on reports) for

every evaluation..

Page 5: Instructor Course Evaluations (ICEs) at AUB

5

Sample Envelope Label/ Ans. SheetSample Envelope Label/ Ans. Sheet

Dr. Mustapha Haidar AGRL201 Sect.:01 # of Students: 39 Dept. Code: 01 Category: Large

Lecture Class

FAFS 1 0 0 0 0 0 A 0 0FAS 2 1 1 1 1 1 B 1 1FEA 3 2 2 2 2 2 C 2 2FM 4 3 3 3 3 3 D 3 3FHS 5 4 4 4 4 4 E 4 4SB 6 5 5 5 5 5 5 5SNU 7 6 6 6 6 6 6 6Others 8 7 7 7 7 7 7 7

8 8 8 8 8 8 89 9 9 9 9 9 9

FACULTY DEPT COURSE SECTION

Page 6: Instructor Course Evaluations (ICEs) at AUB

6

Instructions to Graduate Assistants-1Instructions to Graduate Assistants-1 I. Steps to be Followed in Instructor Course Evaluation (ICE) Administration1. Graduate Assistant should arrange with course instructor with respect to the ICE administration. The

questionnaires should be given last two weeks of classes.2. The ICE should be administered in the absence of the course instructor.3. Tell the class that the instructor will not get results until after grades have been submitted.4. Students should use pencils to complete the forms.5. Graduate Assistant reads instructions to students (following second page) and guides them while filling general

information; faculty of the course and not of student, department (department code is printed on envelope label), course and section numbers. For example Arabic 201, Section 1

Under faculty, A&S #2 should be blackened Under dep’t, Arabic # 03 should be blackened Under course, 201 should be blackened Under section, # 01 should be blackened

It is important that students put their own section number correctly. In case of multi-instructor sections and large lecture courses, please check code given on envelope label for section.

6 Graduate Assistant answers any questions raised.7. No discussion of ratings among students should be allowed.8. Students should be given time needed to complete questionnaires9. After students have finished, collect surveys and thank students for their cooperation. Students should be told

that the materials will be returned directly to department secretary or chair and then to OIRA. The instructor will NOT get results or forms until after grades have been submitted. Place everything in the envelope, seal it and give it to the departmental secretary or chair to be forwarded to OIRA.

Page 7: Instructor Course Evaluations (ICEs) at AUB

7

Instructions to Graduate Assistants-2Instructions to Graduate Assistants-2

II. Administration Instructions to be Read by Graduate Assistants:1. These forms are used to provide information to the instructor and to the University on instructor, course and student development

in the course. They are intended to help the instructors improve their own teaching, and the University in decisions regarding appointment and promotion. Your input is essential for the improvement of the teaching learning process. Therefore, please read every question carefully and answer in appropriate space on the computer scannable sheet by blackening the corresponding circles.

2. First, fill in general information, i.e. faculty of the course and not student, department, course #, and section # (own section number correctly).

3. Graduate Assistant should provide this information, for example: Arab 210 section 1 Under faculty blacken 2Under dep’t. blacken 03 (dep’t. code is printed on envelope, list is also enclosed).Under course blacken 201Under section blacken 01

It is important that students put their own section number correctly. In case of multi-instructor sections and large lecture courses, please check code given on envelope label for section.

4. Answer questions starting with item # 1 on computer sheet using pencils.5. Provide your comments on attached sheet.6. No need to provide your name or ID.7. Take time you need to complete the questionnaire and when you are ready hand it in.8. No need to discuss ratings with your colleagues while filling out the questionnaire.9. The instructor will not know the results of these ratings until after the semester is over and grades have been submitted.

Page 8: Instructor Course Evaluations (ICEs) at AUB

8

Processing of ICEs - 2Processing of ICEs - 2

Provide GAs. with explicit instructions on how to administer ICE especially with respect to importance of proper coding by students

Envelopes are sent to departments 3-4 weeks before end of term.

Once received, sheets are checked for accuracy of filled-in course section information.

If discrepancies are found, they would be corrected.

Scan forms and report results.

Page 9: Instructor Course Evaluations (ICEs) at AUB

9

Processing ProblemsProcessing Problems

Course sections taught by more than one instructor, informed only after reports are released.

GAs. pool sections together while administering and codes get mixed up.

Departments change instructors and we don’t know about it.

Students attend sections other than their own and put their own section code on evaluation. This results in response rate more than 100%.

GAs do not abide by the label information printed on the ICE envelope.

Page 10: Instructor Course Evaluations (ICEs) at AUB

10

III. Summary of Research on III. Summary of Research on Student Evaluations (SE)-1Student Evaluations (SE)-1 SE are used extensively on college campuses

(Marsh, 1987; Seldin, 1993). Approximately 86% of liberal arts colleges and

100% of large research universities systematically collect SE (Seldin 1999).

Authors who researched them agree that they are the single most valid source of data on teaching effectiveness, in fact there is little evidence of the validity of any other sources of data (Mc Keachie, 1997).

Page 11: Instructor Course Evaluations (ICEs) at AUB

11

III. Summary of Research on Student Evaluations III. Summary of Research on Student Evaluations (SE)-2(SE)-2

Validity of student ratings has been sufficiently well established (Marsh, 1984; Arubayi, 1987).

Focus of research has shifted more recently to study of specific background characteristics, biasing variables, which might harm validity (Wachtel, 1998).

Characteristics associated with- Administration of student evaluations (Feldman, 1978; Chen & Hoshower, 1998; Wachtel)- The course itself (Marsh & Dunkin, 1992; Braskamp & Ory, 1994; Anderson & Siegfried, 1997)- The instructor (Anderson & Siegfried, 1997; Wachtel)- Students themselves (Tatro, 1995; Chen & Hoshower, 1998).

Page 12: Instructor Course Evaluations (ICEs) at AUB

12

III. Summary of Research on Student Evaluations III. Summary of Research on Student Evaluations (SE)-3(SE)-3

Variables little/no effect on SE:

1. Instructor (age, sex, teaching experience).

2. Student (age, sex)3. Course( time of day).

Variables affecting SE:A. Course1. University required vs.

elective course2. Higher vs. lower level

course3. Class size4. DisciplineB. Student grade

expectation

Page 13: Instructor Course Evaluations (ICEs) at AUB

13

Grade Expectancy and RatingsGrade Expectancy and Ratings The effect of a student’s expected grade on evaluation of his/her

teacher in that course. Studies generally assert that there is a positive correlation between

expected grade and student ratings. The mere existence of a correlation between a background variable

and rating scores does not necessarily constitute a bias or a threat to the validity of SE.

It does not necessarily follow that an instructor can obtain higher ratings merely by giving higher grades.

Alternative explanations include: (1) the leniency hypothesis (instructors can ‘buy’ better evaluations by giving higher grades); (2) the validity hypothesis (more effective instructors cause students to work harder, thereby earning higher grades); and (3) the student characteristic hypothesis (pre-existing student characteristics such as prior subject interest affect both teaching effectiveness and student ratings).

Page 14: Instructor Course Evaluations (ICEs) at AUB

14

III. Summary of Research: III. Summary of Research: ConclusionConclusion The literature supports that students can provide

valuable information on teaching effectiveness, given that the evaluation is properly designed.

There is a great consensus that students cannot judge all aspects of faculty performance.

Students should not be asked ‘high-inference’ questions like judging whether the materials used in the course are up to date or how well the instructor knows the subject matter of the course.

Page 15: Instructor Course Evaluations (ICEs) at AUB

15

III. Summary of Research: III. Summary of Research: Conclusion-2Conclusion-2An important quote by Mc keachie (1997)

‘Classes differ. Effective teaching is not just a matter of finding a method that works well and using it consistently. Rather, teaching is an interactive process between the students and the teacher. Good teaching involves building bridges between what is in your head and what is in students’ heads. What works for one student or for one class may not work for others’.

Page 16: Instructor Course Evaluations (ICEs) at AUB

16

Summary of ICE Validity ResearchSummary of ICE Validity Research

Reliability for all subscales was good (r=.90-.96).

Content/construct validity evidence.

Differences by:1. Gender (F>M).2. Class (HL>LL).3. Course (elective> univ.

required).4. Subject (SS/HU>

Eng./SC).

Grade expectations

1. high; 70% expecting ≥80.

2. low negative correlation with rating (-.18<r<-.22)

3. students with high expectations gave lower ratings.

Correlation between grade and rating was low (.18-.25).

Page 17: Instructor Course Evaluations (ICEs) at AUB

17

IV. Faculty Perspective: Summary of IV. Faculty Perspective: Summary of Issues in E-mail exchanges-1Issues in E-mail exchanges-1

‘Grading leniency’ and need for a correcting factor for ‘difficult courses’.

Need to incorporate other measures of teaching effectiveness and not only ICEs, like graduating students, alumni, etc.

Flaws in collection, administration, and processing/reporting (especially in sections taught by more than one instructor).

Students’ attitude, do not take evaluations seriously, popularity contest.

Need to take differences due to type of course (required/elective), subject matter, student level, faculty workload, into account.

Page 18: Instructor Course Evaluations (ICEs) at AUB

18

IV. Faculty Perspective: Summary of IV. Faculty Perspective: Summary of Issues in E-mail exchanges-2Issues in E-mail exchanges-2

Need to define exactly what is purpose of evaluations. Time of evaluation, should be made earlier in semester. Some items encourage ‘academic tourism’ and should be

given less weight. ‘High inference’ items requiring subjective judgment should be minimized.

Reporting issues: averages are skewed high, may not adequately discriminate.

Some suggested removing averages/section and for the whole ICE.

Response rates sometimes exceeded 100%.

Page 19: Instructor Course Evaluations (ICEs) at AUB

19

IV. Faculty Perspective: Survey IV. Faculty Perspective: Survey Findings -1Findings -1

Majority value input from ratings and make use of them to improve their courses.

Few agree that they should be used for making personnel decisions regarding salary and promotion.

>50% believe that faculty change their teaching to receive high ratings

Around 40% assert that what is addressed in class may be determined by content of ratings.

Nearly 50% do not believe that the ratings result in negative consequences like reducing faculty morale and job satisfaction.

50% disagree that ratings are a ‘meaningless activity’, while a third agrees with that statement.

.

Page 20: Instructor Course Evaluations (ICEs) at AUB

20

IV. Faculty Perspective: Survey IV. Faculty Perspective: Survey Findings -2Findings -2

Majority believe that demanding a lot from students will result in lower evaluations

There was a split of opinion with respect to ‘Good instructors get high course evaluations’.

As to faculty view of process students use to fill out the ratings, majority believe that students do not take evaluations seriously.

Around half perceive that students do not have enough knowledge to judge quality of instruction.

Most of faculty disagree with ‘faculty members should not be evaluated by students’.

Page 21: Instructor Course Evaluations (ICEs) at AUB

21

IV. Faculty Perspective: Survey IV. Faculty Perspective: Survey Comments -1Comments -1

Enhance communication about the system Revisit questionnaire as it was commented that i) wording of

some items is not clear, and ii) items are not tailored or contextualized to particular needs of courses and accordingly information provided is not very useful.

Expand system to include other means of evaluating faculty performance. The current system should be one of many available like peer review and others.

Revise ICE administration/collection procedures now in use. Faculty value evaluations and make use of them to make

adjustments, especially the comments.

Page 22: Instructor Course Evaluations (ICEs) at AUB

22

IV. Faculty Perspective: Survey IV. Faculty Perspective: Survey Comments - 2Comments - 2

Trend analysis recently reported over a number of semesters provided good indices of teaching effectiveness.

Faculty cited some biases associated with the system like ‘ratings affected by mid-term exam grades and difficulty of course’, ‘had to water down content of courses to make students happy and have fun’, ‘students want easy way out, etc...

Although faculty agreed with the need to be evaluated by students, yet they questioned the seriousness with which students responded.

Page 23: Instructor Course Evaluations (ICEs) at AUB

23

Student Perceptions: Survey Student Perceptions: Survey Findings -1Findings -1 70% perceive them to be a means for indicating

suggestions for improvement Around half believe that faculty value input from

them and make improvements as a result of weaknesses identified.

Around 2/3 perceive them as an effective means of evaluating faculty and do not agree that the ratings are a meaningless activity.

50% do not believe that faculty alter their teaching to get high evaluations.

Believe that the content of the rating form affect what is addressed in class

Page 24: Instructor Course Evaluations (ICEs) at AUB

24

Student Perceptions: Survey Student Perceptions: Survey Findings - 2Findings - 2 Very high percentage 1. perceives the rating process as allowing honest evaluations2. states that they are giving adequate thought and effort to it.3. assert that they are being fair and accurate in their

evaluations. However, when asked about their peer’s attitude, opinion is

equally divided between agree, disagree, and uncertain. 5-point rating scale, is well-understood by students Around half opt for a ‘3’ when they are undecided or

uninterested. However, another 40% disagree with their colleagues with regard to the use of ‘3’.

Page 25: Instructor Course Evaluations (ICEs) at AUB

25

Student Perceptions: Survey Student Perceptions: Survey Comments -1Comments -1

Recommendations were provided to 1. revise form, make it shorter, and different from one major to another2. administer ICEs earlier in semester in order to give faculty member

chance to adjust and students to see effect of evaluation.3. improve administration of Graduate Assistants.4. introduce electronic submission as it better protects identity of

students, especially while writing comments.5. improve communication regarding the evaluations like ‘students

must be informed about the importance of the surveys’ and ‘should be told how and why the results of the ICE will be used’.

6. publicize results of the evaluations. Students felt that ‘it is better to register with a certain instructor because of numerical data rather than campus rumors’, etc..

Page 26: Instructor Course Evaluations (ICEs) at AUB

26

Student Perceptions: Survey Student Perceptions: Survey Comments -2Comments -2 Students believe ratings are important but raised following concerns:1. Evaluations are not made use of and accordingly students do not take them

seriously. Comments were made like ‘we need to feel that what we say matters’, ‘we students feel that evaluations are useless as not a single time there is a consideration of our preferences’,‘ I point out positive and negative points of instructor hoping that somebody listens to what I have to say however thinking that nobody really cares’, ‘I don’t think it matters what I write about the professor’, ‘most students fill form with inappropriate choices because they don’t believe that the surveys will do any good’, and ‘we never really know how ratings are used and if instructors themselves take them into consideration’.

2. Some students fill them inaccurately. Several reasons were cited like ‘lack of time’, ‘pressure from GA’s”, ‘students rather finish in haste and get out of class sooner’, ‘some professors make light of evaluations which leads us students to think that they do in fact make nothing’, and ‘personal likes and dislikes affect ratings’.