presentation inted 2012 valencia

http://cede.lboro.ac.uk

Andrea Wheeler and Melanie King The Centre for Engineering & Design Education

EXPLORING THE BALANCE BETWEEN AUTOMATION AND HUMAN INTERVENTION IN IMPROVING FINAL YEAR UNIVERSITY STUDENT NON-COMPLETION

2012 Conference

INTED2012 (6th International Technology, Education and Development Conference)

The Centre for Engineering and Design Education

• Background: JISC ‘Pedestal for Progression’ project

• Issues related to progression in final year and failure to complete.

• Workshops with students, a whole lost of concerns – relationships with staff and employability.

• Methods adopted: Service Design and Data Mining.

• Data mining and how we currently collect attendance data. Enhancements to current systems / processes. How staff & students use attendance data. Evidence that links attendance with progression. Issues surrounding the case for wider adoption.

• Service Design, managing relationships, points of contact.

• Discussion of results.

Outline


Pedestal for Progression Project

This project is investigating the application of the Service Design methodology within higher education: techniques usually used within the commercial Customer Relationship Management field (CRM). The project team will be working with students, academics and staff from support services across the Institution using service design and data mining techniques in order to enhance the student experience for final year students and aid their progression to next stage - either employment or further study.

http://progression.lboro.ac.uk

March 2011 – August 2012

JISC Relationship Management Programme Phase II Strand 2 – Progression, retention & non-completion


WP1: Discovery Phase – gathering user experiences – all sorts of sources of data and discovery methods adopted


• National Student Survey data • Focus groups with finalists (Programme reps, Student Union)

• Current research on campus (History HEA mini-project, BSE)

• Interviews with staff (academics, admins, technicians, support services)

• LUFBRA.net • Staff/student committees (minutes from meetings)

• Student stories • Identification of current data collected about students (VLE activity, library

systems, attendance data, coursework hand-ins)

Expectations of: Finalists

Needs of: ACADEMICS Effective & efficient process Expert support Robust systems Easy access to data

Strategic objectives: INSTITUTION Enhanced student experience Innovation Competiveness Efficiency Sustainability

• Increase in contact time (academic and personal tutoring)

• Additional skills support (e.g. digital and information literacy)

• Better access to library books • Prioritised reading lists • Support for progression beyond graduation • Fair and balanced assessment (e.g. timing of coursework hand-ins)

• Consistent and timely feedback • Access to facilities and equipment • Ability to feedback to tutors/departments


Issues raised: Progression in Final Year


Recurrent Issues: Access to tutors

“I feel that the personal tutor should take a more active role in working with the student throughout university life.”

Sport and Exercise Science student – entering final year in October 2011

Recurrent Issues: Performance anxiety and personal development

“I think the main emphasis for final year should be attainment, with the student being able to achieve the most they can to fully reach their potential.” Chemistry student – graduated 2011


“Monitoring” : Encouraging the active tutor – monitoring engagement

• Improving the identification of absence at crucial lectures or tutorials • Improving the support for personal tutors in meeting their tutees • Help department monitors to spot students falling through the net • Identify other engagement data that could signal non-participation • Improve the notification of critical information to the right staff member at

the right time.


How Attendance data is collected

Attendance at lectures

Attendance at personal tutor meetings

ATTENDANT Creation and marking of registers for modules


Organise groups

Add comments

Schedule meetings

Upload related files

Email groups

View attendance records

View personal information

Access course marks

CO-TUTOR The staff and student relationship management system


FLEXIBILITY & CONTINUITY: Supporting a learner’s journey

Personal welfare and guidance: Personal tutors, elite athletes’ welfare officers

Academic performance:

Foundation year tutors, lecturers, key skills support

Personal development planning: personal tutors, skills development officers

Attendance monitoring: course tutors, programme monitors

Supervision of placement activities: Placement co-ordinators, industrial supervisors

Research supervision: cross-department supervisors,

Graduate School ‘training needs’ advisors

Disability, additional needs & course support: support officers, departmental admin


Main menu for access to all students, personal

messages and monitoring reports

Quick access to different cohorts of students and personalised

bookmark groups

Personal summaries of

tutee meetings, commenting and

attendance

Useful links to support personal

tutors and supervisors

Select multiple tutees and perform

group actions

Quick action links and attendance

summaries

Home Page for Tutors

Automated and custom flags can

be added

A Student’s Record

Data feeds of personal info and grades from

corporate systems

A history of comments, notices and files added to a student’s record

Comments are categorised to

aid viewing permissions e.g. only allocated

research supervisors can

see comments in the Research

section

Actions that any staff can perform

on a student’s record


“Monitoring” Approaches: Enhancements to current processes / systems

• Registers now marked as ‘critical’ or ‘optional’ • Checkbox for ‘include data in summary’ • Students can add a reason for absence • At the point of marking, an automated email can be sent to absent students • More detailed reports created on student attendance, including attendance

patterns • Students now have access to their own attendance record and are sent

emails (by welfare officer) to view it if their attendance drops below a certain level

• Automated emails generated to personal tutors on a regular basis


Data Mining: Tremendous amounts of data are being collected about student behaviour and

activities

In the 2011 Horizon report Johnson et al. predicts that in the next two to five years, “Learning analytics promises to harness the power of advances in data mining, interpretation, and modelling to improve understandings of teaching and learning, and to tailor education to individual students more effectively” [8].


Attendance Data Analysis

2004/5 2005/6 2006/7 2007/8 2008/9 2009/10 2010/11

Total marked present

25,372 37,899 62,017 79,428 116,335 151,012 271,373

Total number of records

38,715 56,395 89,930 117,062 166,027 215,031 386,247

Average attendance (%)

65.54 67.20 68.96 67.85 70.07 70.23 70.26

Diff (pp) +1.66 +1.76 -1.11 +2.22 +0.16 +0.03

Number modules 62 121 214 229 260 383 800


Attendance Data Analysis Averages for 4221 students graduating between 2004 and 2009 who have had attendance recorded on at least 10 registers per year of study.

1st 78.80% 2.i 73.26% 2.ii 61.19% 3rd 58.36% Link between final UK degree classification and average attendance


Service Design:

Data mining and data monitoring aims to identify patterns of behaviour that predict behaviour… Service Design, however, aims to manage points of contact of user and a service and improve experience /desirability.


Service Design: students as owners of services…

Service Design is concerned with providing authentic customer focused, highly desirable, and pleasurable, services. It aims to foster a sense of ownership, and to include often intangible customer feelings about a service, representing them in a visual manner. Snook, a Service Design consultancy based in Glasgow, defines a service in the term Service Design as “…a co-created event that delivers value to the parties engaged in the interaction” [1].


Service Design: students as customers, educators as panderers…

Students-as-customers, has very different connotations to students as customers: entitlement to satisfaction, a duty to complain. A University management team keen to exceed customers’ expectations.


Service Design Workshops with students and staff – employability and the placement

experience ….feeding back into Co-Tutor ‘Mentoring’ software…

PROVIDES IMPORTANT METRICS: Enhances student experience

• Provides numerous monitoring reports that make the frequency and quality of support, provided by staff to students, completely transparent to senior colleagues and departmental managers.

• Attendance information used to view trends.

• Provides audit trails and accountability for the quality of care provided to students.

• Reports include; • Staff online activity

• Total number of comments per student

• Total number of student/staff meetings both missed and attended

• Distribution of alert flags

• Frequency of comments, meetings and emails

• % attendance across programme of module, year group or level of study

• Reports specific to tutoring type.


IDENTIFICATION & MONITORING: Supporting struggling students

Detailed attendance reports highlight

struggling students

Internal messaging to notify relevant staff when comments are

added to records

Quick views of attendance summary and comment counts

Ability to send emails to personal tutors,

notifying of low attendance

Automated flagging of students with < 50%

attendance


Loughborough University, UK

IDENTIFICATION & MONITORING: Supporting struggling students



1. The balance of automation versus human intervention.

2. More automated methods of capturing attendance seamlessly linked to current system

3. Complex user permissions to view student data as well as staff activity.

4. Who is monitoring the monitors?

5. Identification of ‘touch-points’ that are critically linked to completion.

RECOMMENDATIONS: Challenges to Address

http://cede.lboro.ac.uk

Andrea Wheeler Teaching and Learning Co-ordinator (Projects), The Centre for Engineering & Design Education

2012 Conference


[email protected]

JISC Pedetal for Progression Project

presentation inted 2012 valencia

Education

attendance data

staff students

ukthe centre

data fair

sources of data

engagement data

higher education

progression project