Turning Course Evaluation Into InformationKarl Molden, Senior Planning AnalystKatarina Thomson, Senior Planning Analyst

| University of Greenwich Context

37,499 Students in 2014/15 of which:• 19,671 university based• 1,603 elsewhere in UK • 16,225 wholly overseas

Of UK students• 72% Full time • 76% Undergraduate• 13% Overseas plus 7% EU students• 50% white students• Split across 3 campuses (plus partner colleges)• Main subject areas: business and administrative studies, subjects allied to

medicine (nursing, paramedics etc), education (teacher training etc), computer science & maths, biological sciences & pharmacy, engineering, architecture and building, social studies, etc.

| University of Greenwich Surveys

Numerous student surveys:• National Student Survey (NSS)• Postgraduate Taught Experience Survey (PTES) / Postgraduate research

Experience Survey (PRES• International Student Barometer• University Student Survey (USS) – including UK Engagement Survey (UKES)• In 2013 and 2014: in-house engagement survey based on National Survey

of Student Engagement (NSSE) (with licence from University of Indiana)• Course evaluations• Destinations of Leavers from Higher Education (DLHE)• etc

• Issues of survey fatigue -> to maintain the commitment of students to respond to surveys, there is a need to make sure that survey results are fully utilised

• The university is developing its systems and policy with regards to analytics and their use

| University Student Survey (USS)

• In 2014 and 2015 the University Student Survey was issued to:• On-campus, first year undergraduates• Non final-year undergraduates UK partner colleges• Wholly overseas based students

• Includes:• Key NSS questions• HEA’s UK Engagement Survey (UKES) (on campus only)• Univ of Greenwich specific questions

| Module evaluations

• From 2014/15: coordinated system of module evaluations• Every student on every module is asked to answer a standard set of questions• Uses Evasys software• Transparency: module-level reports are sent automatically on completion of

survey to:• course leaders – who are required to comment on Moodle• students who responded to the survey

• Results are also available for central analysis• Generates a vast amount of data

| Module evaluations

| Motivation

• Surveys are used across the sector and typically well understood by staff and students – potential to develop something clear and understandable for both groups which could be adapted easily by other institutions

• We know that other work carried out has found a small but significant relationship between measures of engagement and academic achievement* - this project would build on that to create “real-time” analytics from module level surveys

• Systems for collection of data already exist and are well supported

• We are "drowning in a sea of data“ – why add more if we can leverage what’s already there?

* Sheffield Hallam University, Using UKES results and institutional award marks to explore the relationship between student engagement and academic achievement

| Method

• Match University Student Survey (USS) data with academic outcomes• Statistical analysis to determine if there is a correlation between

responses to USS questions and outcomes.• Where/if a correlation is found, how strong is it?

• Process developed using R, open source statistical software.

• Code will be made available online, to allow other institutions to repeat and/or develop analysis.

• USS 2014 contained the UKES questions as a subset of the whole survey which means we can try to replicate the results from Sheffield Hallam.

This material and its contents [UKES] is developed by The Higher Education Academy (“HEA”). Some of the questions in this survey are used with permission from The College Student Report, National Survey of Student Engagement, Copyright 2001-2016 The Trustees of Indiana University. With the exception of the aforementioned, the copyright in such material and content belongs to the HEA. No reproduction, modification, or adaptation is permitted without the prior written consent of the HEA © The Higher Education Academy 2012, Amended 2015, All Rights Reserved.

| Results

• Sheffield Hallam identified 6 UKES questions which had a statistically significant correlation with academic outcomes.

• Our analysis has identified 4 of the same 6 questions as having a similar correlation.

• One further USS question also found to have statistically significant correlation.

• Three of these five questions are thematically linked – around working with/explaining material to other students.

| Results

1 2 4 5

020

4060

80

Answers for Q5a based on GPA

Q5a

GP

A

1 2 4 5

020

4060

80

Answers for Q6b based on GPA

Q6b

GP

A

| Results

| Results

• Further to the results with the UKES questions, two questions from the course evaluation surveys were found to have a correlation with academic outcome.

• As these surveys are conducted every term this could be useful in developing “real-time” interventions.

• Further work is needed to look at the potential of adding further data to the mix to develop more sophisticated statistical models.

| Conclusion

The evidence seems to support a small, positive correlation between certain survey question responses and academic outcomes.

By identifying activities which may lead to improved responses to certain questions it may therefore be possible to improve these outcomes.

Similar relationships in “in-session” course level surveys have been identified which mean we may be able to develop other processes to improve academic achievement of students.

| Questions?