putting the results into practice
TRANSCRIPT
Center for Excellence in Learning and Technology
Putting the Results from the Student Evaluation of Teaching
into PracticeJodi Sterle, Ph.D.
Harman Endowed Professor In Teaching & Learning and
Teaching Section Leader
Animal [email protected]
http://www.ans.iastate.edu/
Laura Bestler, Ph.D.Program [email protected] 294-5357 http://www.celt.iastate.edu/
Agenda
•About Online Course Evaluations•The Process•Types of Reports•Who has access? Who views the reports?•Big Picture•How Animal Science Uses Course Evaluations•Moving Forward•Student Participation•Other Uses for Class Climate
About Online Course Evaluations
In 2002, a national study found that 98% of the nation's "most-wired" colleges, still used a paper-based for the student evaluation of teaching (Hoffman, 2003).
Paper Course Evaluations
Paper Course EvaluationsPros Cons
Done in-class - good response rates Very time consuming (preparation before and after administration)
Students feel anonymous May take 4-12 weeks for faculty to get results
Resource intensive to type up answers to open ended questions
Lots of room for error (scanning errors, student errors-using pen, etc., illegible comments, handling errors: students can tamper with data or forget to return, people often put forms in a packet for the wrong class, etc)
Students may be apathetic and just fill in anything
Costly (cost of forms, pencils, bins, envelopes, work hours, scanner)
Bad for the environment (uses lots of paper)
Students may not take time on open ended questions because of time
Online Course Evaluations
Online Course Evaluations
Pros Cons
Immediate results Lower response rate?
Far less room for error (no lost forms, scanning issues)
Has to be done on the students’time (unless technology allows for in-class)
Far less time consuming Anonymity concerns
Far less costly (no scanner, paper forms, much less work hours,etc.)
Students who have not attended class are now able to give opinions
Green- no need for paper
More student comments
Flexibility for questions/scales
Students who take the time to do them have an opinion
Students are able to fill them out 24 hours a day
Less time intensive for administrators
Iowa State UniversityISU started using Scantron's Class Climate online course evaluation system on a pilot basis in Spring 2011.
S2011 F2011 S2012 F2012
DEPTS 56 92 103 115
COURSES/SECTIONS 2071 4680 4576 5574
INSTRUCTORS 717 1540 1667 1901
EVAL INVITES 55,359 142,568 159,420 165,950
Response Rates
F2012
S2012
F2011
S2011
OVERALL AVERAGE 59.74 62.30 63.50 74.40
WEIGHTED AVERAGE 60.17 59.65 63.18 71.57
Weighted Average: is taking the difference of the number of students into accountFormula: =sumproduct(enrolled students in the course,response rate/sum(enrolled students in the course)
The Process
The Process: Departments
Each department/college has local control of:• evaluation questions• email texts• when email invitations are sent out• number/frequency of email reminders• when evaluations starts and ends• when instructors receive course reports
The Process: Email Invitation
SUBJECT: ISU Online Course Evaluation for [COURSENAME] [COURSETYPE]
Iowa State Student [PARTICIPANT_EMAIL],
This email asks you to respond to an anonymous online course evaluation survey for[COURSENAME] [COURSETYPE].Please follow the link to open the course evaluation survey, fill-out the questionnaire, and click “submit” when completed.
[DIRECT_ONLINE_LINK]
Please complete this evaluation by 12:00 p.m. (Noon) on Friday, Month XX, 2012. This survey will not be reopened.You will receive this reminder email periodically until the survey is complete. Please complete all online surveys you receive. Links to your course surveys will also appear in your Blackboard Learn page under “My Course Evaluations.”
Thank you for your participation,- [SUBUNIT] Course Evaluation Administrator
The Process: Email ReminderEMAIL HEADER: Course Evaluation Reminder for [COURSENAME] [COURSETYPE]
Iowa State Student [PARTICIPANT_EMAIL],
This email is to remind you that you have not yet completed the survey for:[COURSENAME] [COURSETYPE].
Please follow the link to open the course evaluation survey, fill-out the questionnaire, and click “submit” when completed.
[DIRECT_ONLINE_LINK]
Please complete this evaluation by 12:00 p.m. (noon) on Friday, Month XX, 2012. This survey will not be reopened.
You will receive this reminder email periodically until the survey is complete. Please complete all online surveys you receive. Links to your course surveys will also appear in your Blackboard Learn page under “My Course Evaluations.”Thank you for your participation,- [SUBUNIT] Course Evaluation Administrator
The Process: Students fill out the evaluation
The Process: Departments/College run and send Instructor/Course Reports
Types of Reports
Example Course Report
Instructor Profile:a combined report of all courses taught during a term.
Example Instructor Profile
Report Creator: Any Compilation (Tree structure)
Example of Department Overall report
Example Course Report: Open Ended Questions
Example of Profile Line
Report for Comparison on Questionnaire
The “Hook” Profile line into standard Report
Who has access?Who views the reports?
Who has access to Class Climate?
• Departments/Colleges:• Designated administrator(s) (Merit, P&S, or Faculty
member)• Only their respective department(s)
• System-wide:• CELT administrator for Class Climate
(2 FT P&S staff members)• IT administrator for Class Climate
(1 FT P&S staff member)
Who sees instructor reports?
• Departmental designated administrator(s) (Merit, P&S, or Faculty member)
http://bit.ly/ccadmins
• Instructor (may share with whomever they wish)
• Department Chair and/or College Administrator
• Teaching Coordinator or other departmental designee
Big Picture
Big Picture: What is your Position Responsibility Statement (PRS)?
• Research, Teaching, Extension and Outreach
• Each area needs to be documented and then evaluated
VanDerZanden, A.M.. (2012, January 30). You have course evaluations: Now what? Department of Animal Science [Presentation]. Ames, IA: Iowa State University.
Big Picture: Teaching
Evaluation of
Teaching
Peer review of teaching
Portfolio of teaching
Teaching philosophy
Student evaluation of teaching
VanDerZanden, A.M.. (2012, January 30). You have course evaluations: Now what? Department of Animal Science [Presentation]. Ames, IA: Iowa State University.
Promotion & Tenure
http://www.provost.iastate.edu/help/promotion-and-tenure
How Animal Science Uses Online Course Evaluations
• 969 undergraduates
• ~200 grad students
• >80 courses/sections per semester
• Every course and section evaluated using ClassClimate since Spring ‘11
• Senior exit surveys
• Advising surveys
Animal Science
• Evaluation results and departmental averages sent to instructors
• Teaching coordinator and Department Head review
• Teaching coordinator assists in interpretation of results and development of improvement plan
• Department Head addresses results in annual review
Using the Results
• Teaching coordinator and Department Head also pass along outstanding teaching faculty names to awards committee
• Use results (with benchmarks, over time) to show improvement
• P&T packets
• Used with Peer Evaluation of Teaching evaluations
Using the Results
• Lots of initial anxiety over move to online (and mandatory) course evals
• Love getting results early
• Open and transparent
• Departmental trends
• Focus on improvement
What I’ve Seen
Moving Forward
•Snap shot in time
•Don’t take everything at face value
•Concentrate on the 80%; not the 20%
•Discuss with a trusted mentor
• What can you do with the constructive feedback?
Moving Forward: Reflecting on Fall 2012
VanDerZanden, A.M.. (2012, January 30). You have course evaluations: Now what? Department of Animal Science [Presentation]. Ames, IA: Iowa State University.
Moving Forward: Strategies for Analyzing Feedback
• Control your defense mechanisms.
• Analyze the source of your students’ reactions in a way that sheds light on any issues and problems that have been identified.
• Work hard not to under-react or over-react to information that you receive via evaluation feedback.
• Divide the issues raised by students into actionable and non-actionable categories.Moore, S., & Kuol, N. (2005). A punitive tool or a valuable resource? Using student evaluations to enhance your teaching. In G. O’Neill, S.
Moore, & B. McMulline (Eds)., Emerging issues in the practice of university learning and teaching (pp. 141-148). Dublin: All Ireland Society for Higher Education.
• Communicate with students before and after their provision of feedback.
• Do not make the simplistic assumption that all positive responses are related to good teaching and all negative responses are related to bad teaching.
• Remember that small changes can have big effects.
• Develop a teaching enhancement strategy that takes into account the evaluation feedback (145-6).
Moore, S., & Kuol, N. (2005). A punitive tool or a valuable resource? Using student evaluations to enhance your teaching. In G. O’Neill, S. Moore, & B. McMulline (Eds)., Emerging issues in the practice of university learning and teaching (pp. 141-148). Dublin: All Ireland Society for Higher Education.
Moving Forward: Strategies for Analyzing Feedback
Comments Analysis Worksheet
Comments Analysis Worksheet
• Often multiple comments are related to the same category; for example:10 students may all make comments about the assignments being unclear. This is not really 10 different comments but rather one comment 10 times.
• The multiple mentions give it weight, but it is only one area that needs to be addressed for improvement.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada License. Winer, L., Di Genova, L., Vungoc, P.-A., & Talsma, S. (2012). Comments analysis worksheet. Excerpted from Interpreting end-of-course evaluation results. Montreal: Teaching and Learning Services, McGill University available at: http://www.mcgill.ca/tls/teaching/course-evaluations/interpretation.
Comments Analysis Worksheet
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada License. Winer, L., Di Genova, L., Vungoc, P.-A., & Talsma, S. (2012). Comments analysis worksheet. Excerpted from Interpreting end-of-course evaluation results. Montreal: Teaching and Learning Services, McGill University available at: http://www.mcgill.ca/tls/teaching/course-evaluations/interpretation.
Comments Analysis WorksheetThe worksheet has been organized alphabetically in sections according to most frequently commented categories.• Note any student comments that will help in
interpretation. • Indicate positive and negative comments.• Record the frequency of comments surrounding each
theme to help identify the areas where students felt most strongly.
• Add any personal notes that will help in the process of building on the feedback received.
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 Canada License. Winer, L., Di Genova, L., Vungoc, P.-A., & Talsma, S. (2012). Comments analysis worksheet. Excerpted from Interpreting end-of-course evaluation results. Montreal: Teaching and Learning Services, McGill University available at: http://www.mcgill.ca/tls/teaching/course-evaluations/interpretation.
How Can You Help with Student Participation?
Increasing Student Participation
• Give students examples on how the results enhance your course and teaching.
• Include the course evaluation start/end date and/or information in your course syllabus.
• Add the “My Course Evaluation” module into your BbLearn course.
• Request that your departmental administrator send you the response rate notification prior to the end of the evaluation period. You will then receive an automatic email that your course has a low response rate (set at the local level).
Contact your departmental administrator:http://bit.ly/ccadmins
Should there be incentives for completing online course evaluations?
24% No
70% Yes
6% Don’t Care
2012 Comets survey 5000 random ISU students 763 students responded15% response rate
What would be considered good incentives?
Additional Uses for Class Climate
What else can Class Climate be used for?
• Mid-semester course/instructor evaluations
• senior exit surveys• advising surveys• program evaluations• research projects (STEM, etc.)
Contact your departmental administrator:http://bit.ly/ccadmins