2015 16th annual pabug conference bursting the bubble [sheets]: moving course evaluations from paper...
TRANSCRIPT
2015 16th Annual PABUG Conference
Bursting the Bubble [Sheets]:Moving Course Evaluations from
Paper to Online
Shippensburg UniversityJohn VandeBrake
Andrea Rosenberry
2015 16th Annual PABUG Conference
General Announcements:• Please turn off all cell phones/pagers
• If you must leave the session early, please do so as discreetly as possible
• Please avoid side conversations during the session
• Questions will be answered …..
Thank you for your cooperation
2015 16th Annual PABUG Conference
To receive CPE credits for this session (if eligible), complete the CPE Attendance Form on the PABUG Annual Conference website
http://pabug.org/pennsylvania-banner-users-group-annual-conference/cpe-credits/
or via the PABUG conference app.
For additional questions please contact
Kim Fremont CPE - Coordinator http://pabug.org/contact-us/
CPE Credits - We’ve gone electronic!
2015 16th Annual PABUG Conference
About
2015 16th Annual PABUG Conference
Agenda
OverviewOpportunities
ChallengesKeys to Success
OutcomesExplorance Blue PeakFuture Enhancements
Q & A
2015 16th Annual PABUG Conference
• Legacy system using Opscan paper bubble sheets• Collective Bargaining Agreement (CBA) specifications• Collaboration between provost and faculty key• SIS integration – Explorance Blue Evaluation chosen
due to Banner integration• Data security: Explorance versus Survey Monkey or
other non-SSO options• Three year pilot launched starting with Athletics
Overview
2015 16th Annual PABUG Conference
Paper versus Online• It goes without saying…..lots of manual work!
Overview
2015 16th Annual PABUG Conference
Paper process• Pre Printing of the Opscan forms with course information• Distribution of the paper forms to department offices then
class• Collection of paper forms and security concerns• Functional issues with the Opscan machine• Issues with bubble forms filled incorrectly• Issues with pre printing on forms• Computation of results in legacy software• Printing and distribution of paper results• Storage and retrieval of paper results
Overview
2015 16th Annual PABUG Conference
• Eliminate paper processing saving time for both provost office, secretaries, and IT
• Security of documents improved• Student convenience improved• Student responses more accurate and
thoughtful• Computation, distribution and retrieval of
results easier
Opportunities
2015 16th Annual PABUG Conference
• SIS Integration: Banner to Blue• Process changes: Explorance Blue vs paper• Online response and evaluation rate variance• Data integrity and automation for provost
office• CBA requirements & faculty adoption
Challenges
2015 16th Annual PABUG Conference
SIS Integration
• Oracle Views connect to Explorance Blue
• Authentication - SSO using CAS
Challenges
2015 16th Annual PABUG Conference
Banner Data Integration
• Data is term driven• Non-distinct CRN
– Numbers reused across terms– Team Taught Courses
• Not all courses should be evaluated
Challenges
2015 16th Annual PABUG Conference
Challenges
2015 16th Annual PABUG Conference
Process Changes: Explorance Blue • Pre-survey sent to enlist “volunteers” from Blue
– Volunteer versus required to evaluate– Excel flag file data extension block used in Blue
• Volunteer list from survey combined with required• Evaluation survey scheduled for launch• One click thru for students to evaluation
– Easy to use– Mobile enabled– Only one time completion allowed
Challenges
2015 16th Annual PABUG Conference
Process limitations• Delivery mechanism changes with online
• Instrument moved to online– Challenge with collection of written responses and privacy
• Process changes– 24/7 access for two week window– Lack of “Captive audience”
• Reporting changes• Mirrored reporting as much as possible• Privacy of written responses
Challenges
2015 16th Annual PABUG Conference
Evaluation Rate Variances
• Evaluation rate slightly lower, .2 to .5 on Likert scale. – Announcement made to faculty“…a negative variance in this range may be observed
when comparing online to paper. And that such variation should be considered insignificant and may not be used as the basis for negative evaluation.”
Challenges
2015 16th Annual PABUG Conference
Response Rate Variance• Response rates have somewhat dropped
– “Captive audience” effect neutralized– Reminder emails sent to those not responding– Faculty encouraged to announce in class
• Helps increase participation
• More could be done to encourage online participation– Advertising?– Contests?– LMS Integration?
Challenges
2015 16th Annual PABUG Conference
Data Integrity and Automation• Paper process had potential inaccuracies
– Paper responses were sometimes incomplete and hard to validate
– Response rates were not always accurate– Paper slugging of forms was not always accurate
• Maintaining data integrity online in Banner• Moving offline data into Blue thru Banner
integration
Challenges
2015 16th Annual PABUG Conference
CBA Requirements – Faculty Adoption• Shippensburg works with unionized faculty
under a collective bargaining agreement (CBA)– Mirrored process as much as possible– Mirrored paper documents as much as possible– Mirrored time lines as much as possible
• Faculty adoption encouraged thru collaboration with representatives and committee as well as through the pilots
Challenges
2015 16th Annual PABUG Conference
• Communication between provost, faculty, athletics, and IT– Complete support of the Provost– Representation included leadership, athletics, faculty,
union, and IT in forming a process plan– Regular committee meetings to receive input– Implementation of changes as technology permits– Communication to the larger campus community vital
Keys to Success
2015 16th Annual PABUG Conference
• Three year pilot with three phases– Phase 1: Focus on athletic coach evaluations and early
adoption as well as training• Athletic evaluations established success• Training meetings held for pilot faculty
– Phase 2: Process focused: adjustments and improvements• Meetings held to review process and gather input• Online reporting improved to match paper reporting• Reminders tweaked
Keys to Success
2015 16th Annual PABUG Conference
• Three year pilot comprised of three phases– Phase 3: Adoption: Improve participation and tweaks
• Encourage participation• Added faculty monitoring of response rates during
evaluation cycle• Supervisory report viewing enhanced
Keys to Success
2015 16th Annual PABUG Conference
• Budget considerations help drive adoption– Good pilots and minor technical glitches made
adoption more likely– Show how budget savings could benefit faculty
• Software ease of use: Explorance Blue has served us well
• Responsive leadership will help pave the way
Keys to Success
2015 16th Annual PABUG Conference
• Slight decrease in quantitative responses• Increase in qualitative data
– Written responses more detailed and legible– Quality of responses improved. Longer and more
thoughtful• Use of own time to complete allows for more thought
• Reduces the use of paper and processing• Saves class time for faculty• Insures data integrity• Quick reporting turn around• Distance Ed and offsite course evaluations easier
Outcomes
2015 16th Annual PABUG Conference
• Evaluation Instrument: sent with specific class details to each student
Explorance Blue Sample
2015 16th Annual PABUG Conference
• Three reminder emails sent to those not completed
Explorance Blue Sample
Dear Susan Hickenbaugh,
REMINDER: We have not yet received your feedback for these course(s):
-Charles Stanley (CSC570-01 Database Mgt Systems),-Joe Leevy (CSC521-01 Operating Systems),- Joe Leevy (CSC502-01 Automation Theory)
We would really appreciate it if you can take a few minutes now to complete the student course evaluations.
We are using online survey software to make it easier for students to give us their feedback.
Click here now to provide us with your feedback.
Remember: You will need to make sure you click the "Submit" button to complete your evaluation.
If you experience any technical difficulties or have any questions please contact T Pearson in the Provost office at [email protected]
2015 16th Annual PABUG Conference
Explorance Blue Sample
2015 16th Annual PABUG Conference
Explorance Blue Sample
2015 16th Annual PABUG Conference
Subject View: Response rate review during evaluation cycle
Provides the ability for faculty members to view the cumulative response rate as the evaluations is being administered.
Explorance Blue Sample
2015 16th Annual PABUG Conference
• Mobile enabled for more input options– Potential for classroom
administration– Easy access
Explorance Blue Sample
2015 16th Annual PABUG Conference
• Further automation in the provost office– Volunteer and required selection process in SIS
• LMS Integration• Participation promotion• CBA requirements reassessment • Parts of term and interactive start/stop times• Long term storage of reporting• Survey add-on tool for Title IX, Snyder File V data
collection, and other surveys
Future Enhancements
2015 16th Annual PABUG Conference
Open to the Floor
• Questions• Comments
2015 16th Annual PABUG Conference
Please take time to complete our Conference and Session
evaluations.
YOUR INPUT MATTERS!!!