Using Sakai for Teaching Evaluations

Sean DeMonner, Project Manager, University of MichiganEllen Yu Borkowski, University of MarylandBryan Bakotich, Pacific Lutheran University

Overview

• Evaluation System• Institutional stories• Q&A

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 2

Evaluation System

• Provide the ability to run evaluations (course, department, institutional, group) or surveys in a flexible way which meets the needs of most Sakai using institutions

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 3

Summary of features

• Basic template authoring system– Template – collection of items– Items – questions or response statements– Scales – Likert or M/C options

• Allow groups (instructors, departments, etc.) to create evaluations/feedback/surveys• Ad-hoc groups also supported

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 4

Summary of features

• Create evaluations which can be assigned to various groups at once and can be set up in advance

• Sends out notifications and reminders to evaluators which allow one-click access• Ability to edit email templates used for

notifications and reminders

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 5

Summary of features

• Direct links into Sakai for all evaluations are available and login is handled automatically if needed • anonymous is available

• Basic online reporting and the ability to export results as CSV, PDF, XLS

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 6

Pacific Lutheran University

• Liberal Arts University with graduate programs in Nursing, Business, and Education

• Total Enrollment: 3661• Sakai in production since 2006

July 2009 710th Sakai Conference - Boston, MA, U.S.A.

• Homegrown system using Apache, PHP, and MySQL• Pilot with 7 courses in 2004• Pilot with Rank and Tenure Committee in 2006 and

2007• Recommended to form another committee and study it

further

Early Attempts

July 2009 810th Sakai Conference - Boston, MA, U.S.A.

• Administrator Evaluations• Archaic process• Source of frustration• Good place to pilot Evaluation System

• New Student Safety Orientation• Fulfill grant requirements• Track response rates and send reminders to non respondent

students• Sakai EvalSys was recommended after success of

Administrator Evaluation

Types of Evaluations

July 2009 910th Sakai Conference - Boston, MA, U.S.A.

Administrator Evaluation Results

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 10

Paper

Online

43% Response Rate?

53% Response Rate

New Student Orientation Results

• 1 evaluation on small focus group of transfer students• 60% response rate (25 / 42)

• No time spent monitoring evaluation• Previously took a “tremendous” amount of time

July 2009 1110th Sakai Conference - Boston, MA, U.S.A.

Problems Encountered

• Lack of set up time• Email reminders treated like spam due to lack of

personalization• Unable to link numeric values with comments in

results• All evaluators email addresses show up in to

field of emails sent out by system

July 2009 1210th Sakai Conference - Boston, MA, U.S.A.

Futures

• Modify code to hide email addresses of evaluators

• “Heads up” email to increase response rates• Future plans to add video content to evaluation• Faculty Evaluations?

July 2009 1310th Sakai Conference - Boston, MA, U.S.A.

University of Michigan

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 14

Teaching Questionnaires @ U-M

• History of Teaching Evaluations at U-M• Collaboration between MAIS and CTools• Concerns over move to online system:

• Research drives tenure + teaching excellence is expected - but evaluations can impact tenure decisions = significant political implications & risks

• Multiple pilots; adjustments• Full scale rollout F08; W09 meltdown; ongoing

production• System operations and shifting expectations

(e.g. frequency, reporting)July 2009 1510th Sakai Conference - Boston, MA, U.S.A.

Teaching Questionnaires @ U-M

• U-M specific features:• External data loading • Single email• Reminder email handling• Confirmation email• Admin summary page controls

• Architectural details:• PeopleSoft order entry ->XML data exchange ->

Sakai -> PeopleSoft reportingJuly 2009 1610th Sakai Conference - Boston, MA, U.S.A.

TQs: Response Info

July 2009 1710th Sakai Conference - Boston, MA, U.S.A.

Reminders drive response but students resent spam…

TQs: Response Rates

July 2009 1810th Sakai Conference - Boston, MA, U.S.A.

TQs: Initial Outcomes

• Challenges:• Customized email job

• Load modeling and “spiky” applications

• Operational issues; system maturity

• Reporting problems; managing expectations

• Data exchange root of all evil (REST?); shifting requirements

• Fully Sakai-based system would not have this problem

July 2009 1910th Sakai Conference - Boston, MA, U.S.A.

TQs: Initial Outcomes

• Successes:• Response rates (72% students; 60+%

evals)• Favorability rates (no significant difference)• Communications team

• Faculty Steering Committee in place

• Data warehouse coming (numeric data)

July 2009 2010th Sakai Conference - Boston, MA, U.S.A.

University of Maryland

• Major public research university• Carnegie Doctoral/Research University –

Extensive

• Total Enrollment: 37,000• Sakai in production since Fall 2007 with

only EVALSYS being used

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 21

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 22

CourseEvalUM

• History of CourseEvalUM• University Senate Task Force on Course and

Teaching Evaluation report April 2005• Student Course Evaluation Implementation

Committee charged by Provost February 2006• Final recommendations to University Senate

April 2006 (approved)

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 23

CourseEvalUM

• 15 university-wide items defined• 6 used for APT purposes• 7 are public for student access• 1 open ended comment item• 1 question about fit in academic plan

• CORE• Major/Certificate/Minor/Program Requirement• Elective

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 24

CourseEvalUM

• Access to results• Two-tiered incentive system for students

• Individual student must fill in all of their evaluations the prior semester to gain access to results of public questions for students

• Course must have 70% return rate for course results to be available to students who satisfy requirement above

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 25

CourseEvalUM

• Access to results (con’t)• Instructors can view all items for their own

course(s)• Administrators can view APT items

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 26

CourseEvalUM

• Maryland-specific features• External group provider

• Data pull from SIS on courses/enrollments

• Hierarchy• University and college/school

• Multiple instructor• Reminder email handling

• Suppress initial email; use reminder email in tool

CourseEvalUM

• Architectural details:• SIS (proprietary) -> Sakai -> CourseEvalUM

Reports (proprietary)

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 28

CourseEvalUM

• Pilot results• University items only

• Summer I 2007: 40.7%• Summer II: 40.8%• Fall 2007: 63%

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 29

CourseEvalUM

• Production results• University and college level items and

multiple instructor• Spring 2008:62% (3 colleges)• Summer I 2008: 39% (3 colleges)• Summer II 2008: 45% (4 colleges)• Fall 2008: 61% (7 colleges)• Spring 2009: 66% (7 colleges)

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 30

CourseEvalUM

• Challenges• Email “spamming”

• Students complained about spamming• Starting Fall 2007, initial email notification was

sent once (one per student) and reminders were sent multiple times (one per evaluation)

CourseEvalUM

• Challenges (con’t)• Performance issues Fall 2007

• Way in which data was being parsed – patch is now in trunk

• Scope creep• Limited resources

• Next Steps• Waiting on prioritized functional list

Questions?

• University of Michigan• http://www.umich.edu/~eande/tq/

onlineevals.htm

• University of Maryland• https://www.irpa.umd.edu/Assessment/

crs_eval.shtml

• Pacific Lutheran University

July 2009 10th Sakai Conference - Boston, MA, U.S.A. 32