putting the results into practice

Center for Excellence in Learning and Technology

Putting the Results from the Student Evaluation of Teaching

into PracticeJodi Sterle, Ph.D.

Harman Endowed Professor  In Teaching & Learning and

Teaching Section Leader

Animal Sciencejsterle@iastate.edu

http://www.ans.iastate.edu/

Laura Bestler, Ph.D.Program CoordinatorCELTbestler@iastate.edu 294-5357 http://www.celt.iastate.edu/

Agenda

•About Online Course Evaluations•The Process•Types of Reports•Who has access? Who views the reports?•Big Picture•How Animal Science Uses Course Evaluations•Moving Forward•Student Participation•Other Uses for Class Climate

About Online Course Evaluations

In 2002, a national study found that 98% of the nation's "most-wired" colleges, still used a paper-based for the student evaluation of teaching (Hoffman, 2003).

Paper Course Evaluations

Paper Course EvaluationsPros Cons

Done in-class - good response rates Very time consuming (preparation before and after administration)

Students feel anonymous May take 4-12 weeks for faculty to get results

Resource intensive to type up answers to open ended questions

Lots of room for error (scanning errors, student errors-using pen, etc., illegible comments, handling errors: students can tamper with data or forget to return, people often put forms in a packet for the wrong class, etc)

Students may be apathetic and just fill in anything

Costly (cost of forms, pencils, bins, envelopes, work hours, scanner)

Bad for the environment (uses lots of paper)

Students may not take time on open ended questions because of time

Online Course Evaluations

Online Course Evaluations

Pros Cons

Immediate results Lower response rate?

Far less room for error (no lost forms, scanning issues)

Has to be done on the students’time (unless technology allows for in-class)

Far less time consuming Anonymity concerns

Far less costly (no scanner, paper forms, much less work hours,etc.)

Students who have not attended class are now able to give opinions

Green- no need for paper

More student comments

Flexibility for questions/scales

Students who take the time to do them have an opinion

Students are able to fill them out 24 hours a day

Less time intensive for administrators

Iowa State UniversityISU started using Scantron's Class Climate online course evaluation system on a pilot basis in Spring 2011.

S2011 F2011 S2012 F2012

DEPTS 56 92 103 115

COURSES/SECTIONS 2071 4680 4576 5574

INSTRUCTORS 717 1540 1667 1901

EVAL INVITES 55,359 142,568 159,420 165,950

Response Rates

F2012

S2012

F2011

S2011

OVERALL AVERAGE 59.74 62.30 63.50 74.40

WEIGHTED AVERAGE 60.17 59.65 63.18 71.57

Weighted Average: is taking the difference of the number of students into accountFormula: =sumproduct(enrolled students in the course,response rate/sum(enrolled students in the course)

The Process

The Process: Departments

Each department/college has local control of:• evaluation questions• email texts• when email invitations are sent out• number/frequency of email reminders• when evaluations starts and ends• when instructors receive course reports

The Process: Email Invitation

SUBJECT: ISU Online Course Evaluation for [COURSENAME] [COURSETYPE]

Iowa State Student [PARTICIPANT_EMAIL],

This email asks you to respond to an anonymous online course evaluation survey for[COURSENAME] [COURSETYPE].Please follow the link to open the course evaluation survey, fill-out the questionnaire, and click “submit” when completed.

[DIRECT_ONLINE_LINK]

Please complete this evaluation by 12:00 p.m. (Noon) on Friday, Month XX, 2012. This survey will not be reopened.You will receive this reminder email periodically until the survey is complete. Please complete all online surveys you receive. Links to your course surveys will also appear in your Blackboard Learn page under “My Course Evaluations.”

Thank you for your participation,- [SUBUNIT] Course Evaluation Administrator

The Process: Email ReminderEMAIL HEADER: Course Evaluation Reminder for [COURSENAME] [COURSETYPE]

Iowa State Student [PARTICIPANT_EMAIL],

This email is to remind you that you have not yet completed the survey for:[COURSENAME] [COURSETYPE].

Please follow the link to open the course evaluation survey, fill-out the questionnaire, and click “submit” when completed.

[DIRECT_ONLINE_LINK]

Please complete this evaluation by 12:00 p.m. (noon) on Friday, Month XX, 2012.  This survey will not be reopened.

You will receive this reminder email periodically until the survey is  complete. Please complete all online surveys you receive. Links to your course surveys will also appear in your Blackboard Learn page under “My Course Evaluations.”Thank you for your participation,- [SUBUNIT] Course Evaluation Administrator

The Process: Students fill out the evaluation

The Process: Departments/College run and send Instructor/Course Reports

Types of Reports

Example Course Report

Instructor Profile:a combined report of all courses taught during a term.

Example Instructor Profile

Report Creator: Any Compilation (Tree structure)

Example of Department Overall report

Example Course Report: Open Ended Questions

Example of Profile Line

Report for Comparison on Questionnaire

The “Hook” Profile line into standard Report

Who has access?Who views the reports?

Who has access to Class Climate?

• Departments/Colleges:• Designated administrator(s) (Merit, P&S, or Faculty

member)• Only their respective department(s)

• System-wide:• CELT administrator for Class Climate

(2 FT P&S staff members)• IT administrator for Class Climate

(1 FT P&S staff member)

Who sees instructor reports?

• Departmental designated administrator(s) (Merit, P&S, or Faculty member)

http://bit.ly/ccadmins

• Instructor (may share with whomever they wish)

• Department Chair and/or College Administrator

• Teaching Coordinator or other departmental designee

Big Picture

Big Picture: What is your Position Responsibility Statement (PRS)?

• Research, Teaching, Extension and Outreach

• Each area needs to be documented and then evaluated

VanDerZanden, A.M.. (2012, January 30). You have course evaluations: Now what? Department of Animal Science [Presentation]. Ames, IA: Iowa State University.

Big Picture: Teaching

Evaluation of

Teaching

Peer review of teaching

Portfolio of teaching

Teaching philosophy

Student evaluation of teaching

VanDerZanden, A.M.. (2012, January 30). You have course evaluations: Now what? Department of Animal Science [Presentation]. Ames, IA: Iowa State University.

Promotion & Tenure

http://www.provost.iastate.edu/help/promotion-and-tenure

How Animal Science Uses Online Course Evaluations

• 969 undergraduates

• ~200 grad students

• >80 courses/sections per semester

• Every course and section evaluated using ClassClimate since Spring ‘11

• Senior exit surveys

• Advising surveys

Animal Science

• Evaluation results and departmental averages sent to instructors

• Teaching coordinator and Department Head review

• Teaching coordinator assists in interpretation of results and development of improvement plan

• Department Head addresses results in annual review

Using the Results

• Teaching coordinator and Department Head also pass along outstanding teaching faculty names to awards committee

• Use results (with benchmarks, over time) to show improvement

• P&T packets

• Used with Peer Evaluation of Teaching evaluations

Using the Results

• Lots of initial anxiety over move to online (and mandatory) course evals

• Love getting results early

• Open and transparent

• Departmental trends

• Focus on improvement

What I’ve Seen

Moving Forward

•Snap shot in time

•Don’t take everything at face value

•Concentrate on the 80%; not the 20%

•Discuss with a trusted mentor

• What can you do with the constructive feedback?

Moving Forward: Reflecting on Fall 2012

VanDerZanden, A.M.. (2012, January 30). You have course evaluations: Now what? Department of Animal Science [Presentation]. Ames, IA: Iowa State University.

Moving Forward: Strategies for Analyzing Feedback

• Control your defense mechanisms.

• Analyze the source of your students’ reactions in a way that sheds light on any issues and problems that have been identified.

• Work hard not to under-react or over-react to information that you receive via evaluation feedback.

• Divide the issues raised by students into actionable and non-actionable categories.Moore, S., & Kuol, N. (2005). A punitive tool or a valuable resource? Using student evaluations to enhance your teaching. In G. O’Neill, S.

Moore, & B. McMulline (Eds)., Emerging issues in the practice of university learning and teaching (pp. 141-148). Dublin: All Ireland Society for Higher Education.

• Communicate with students before and after their provision of feedback.

• Do not make the simplistic assumption that all positive responses are related to good teaching and all negative responses are related to bad teaching.

• Remember that small changes can have big effects.

• Develop a teaching enhancement strategy that takes into account the evaluation feedback (145-6).

Moore, S., & Kuol, N. (2005). A punitive tool or a valuable resource? Using student evaluations to enhance your teaching. In G. O’Neill, S. Moore, & B. McMulline (Eds)., Emerging issues in the practice of university learning and teaching (pp. 141-148). Dublin: All Ireland Society for Higher Education.

Moving Forward: Strategies for Analyzing Feedback

Comments Analysis Worksheet

Comments Analysis Worksheet

• Often multiple comments are related to the same category; for example:10 students may all make comments about the assignments being unclear. This is not really 10 different comments but rather one comment 10 times.

• The multiple mentions give it weight, but it is only one area that needs to be addressed for improvement.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada License. Winer, L., Di Genova, L., Vungoc, P.-A., & Talsma, S. (2012). Comments analysis worksheet. Excerpted from Interpreting end-of-course evaluation results. Montreal: Teaching and Learning Services, McGill University available at: http://www.mcgill.ca/tls/teaching/course-evaluations/interpretation.

Comments Analysis Worksheet

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada License. Winer, L., Di Genova, L., Vungoc, P.-A., & Talsma, S. (2012). Comments analysis worksheet. Excerpted from Interpreting end-of-course evaluation results. Montreal: Teaching and Learning Services, McGill University available at: http://www.mcgill.ca/tls/teaching/course-evaluations/interpretation.

Comments Analysis WorksheetThe worksheet has been organized alphabetically in sections according to most frequently commented categories.• Note any student comments that will help in

interpretation. • Indicate positive and negative comments.• Record the frequency of comments surrounding each

theme to help identify the areas where students felt most strongly.

• Add any personal notes that will help in the process of building on the feedback received.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada License. Winer, L., Di Genova, L., Vungoc, P.-A., & Talsma, S. (2012). Comments analysis worksheet. Excerpted from Interpreting end-of-course evaluation results. Montreal: Teaching and Learning Services, McGill University available at: http://www.mcgill.ca/tls/teaching/course-evaluations/interpretation.

How Can You Help with Student Participation?

Increasing Student Participation

• Give students examples on how the results enhance your course and teaching.

• Include the course evaluation start/end date and/or information in your course syllabus.

• Add the “My Course Evaluation” module into your BbLearn course.

• Request that your departmental administrator send you the response rate notification prior to the end of the evaluation period. You will then receive an automatic email that your course has a low response rate (set at the local level).  

Contact your departmental administrator:http://bit.ly/ccadmins

Should there be incentives for completing online course evaluations?

24% No

70% Yes

6% Don’t Care

2012 Comets survey 5000 random ISU students 763 students responded15% response rate

What would be considered good incentives?

Additional Uses for Class Climate

What else can Class Climate be used for?

• Mid-semester course/instructor evaluations

• senior exit surveys• advising surveys• program evaluations• research projects (STEM, etc.)

Contact your departmental administrator:http://bit.ly/ccadmins