using predictive indicators of student success at scale – implementation successes, issues and...
TRANSCRIPT
![Page 1: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/1.jpg)
Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open UniversityKevin Mayles, Head of Analytics, Open University
![Page 2: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/2.jpg)
Acknowledgements
●Kevin Mayles, Head of Analytics●[email protected]●@kevinmayles●Slideshare:
● Implementation: Avinash Boroowa, Kelly Bevis, Rebecca Ward, Zdenek Zdrahal, Martin Hlosta, Jakub Kuzilek, Michal Huptych
●Evaluation and analysis: Bart Rienties, Christothea Herodotou, Galina Naydenova
![Page 3: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/3.jpg)
OU Context
2014/15
174k students
The average age of our new undergraduate students is 29
40% new undergraduates have 1 A-Level or lower on entry
Over 21,000 OU students have disabilities
868k assessments submitted, 395k phone calls and 176k emails received
from students
![Page 4: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/4.jpg)
Current predictive indicatorsModule probabilities
4
Integrated into the Student Support Intervention Tool
Predicts the probability of a student completing and passing the module
![Page 5: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/5.jpg)
Current predictive indicatorsOU Analyse
5
Predicts the submission of next assignment weekly
Deployed through OU Analyse Dashboard
![Page 6: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/6.jpg)
Lessons
6
Start small – find and nurture your champions
Create your super-users, create your case
studies
Don’t underestimate the guidance required
Create the right story for the user
Foreground the “should we” arguments Repeat, repeat, repeat
![Page 7: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/7.jpg)
Lessons
7
Start small – find and nurture your champions
Create your super-users, create your case
studies
Don’t underestimate the guidance required
Create the right story for the user
Foreground the “should we” arguments Repeat, repeat, repeat
![Page 8: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/8.jpg)
Intervention potential
Measures●Accuracy of predictions●Volume of predictions●Timeliness of predictions
Accurate predictions at a manageable volume that are timely in relation to a student’s decision to drop out
= Ability of Associate Lecturers (or others) to act on the information
We have been working to assess the intervention potential of using frequent indicators
8
![Page 9: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/9.jpg)
Accuracy
●Using standard metrics where the condition we are predicting is the student NOT submitting (or completing/passing)
●Accuracy = % of all predictions that are correct●Precision = % of non-submission predictions that are correct (the reverse of this is the
false positive rate)●Recall = % of actual non-submissions that are identified by the predictions (the
reverse of this is the miss rate)
9
![Page 10: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/10.jpg)
Accuracy
10
![Page 11: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/11.jpg)
Intervention potential
11
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:
Predicted not to submit = Actually did not submit =New prediction this week =
![Page 12: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/12.jpg)
Intervention potential
12
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:By week 5, 2/3rds of predictions true, but …
Predicted not to submit = Actually did not submit =New prediction this week =
![Page 13: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/13.jpg)
Intervention potential
13
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:By week 5, 2/3rds of predictions true, but only identify half of non-submitters
Predicted not to submit = Actually did not submit =New prediction this week =
![Page 14: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/14.jpg)
Intervention potential
14
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:By week 5, 2/3rds of predictions true, but only identify half of non-submittersBy week 14, predictions identify 2/3rds of non-submitters
Predicted not to submit = Actually did not submit =New prediction this week =
![Page 15: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/15.jpg)
Volume
15
![Page 16: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/16.jpg)
Intervention potential
16
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:By week 5, 2/3rds of predictions true, but only identify half of non-submittersBy week 14, predictions identify 2/3rds of non-submitters
Predicted not to submit = Actually did not submit =New prediction this week =
![Page 17: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/17.jpg)
Intervention potential
17
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:By week 5, 2/3rds of predictions true, but only identify half of non-submittersBy week 14, predictions identify 2/3rds of non-submitters
Volume:Manageable – one new prediction per fortnight
Predicted not to submit = Actually did not submit =New prediction this week =
![Page 18: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/18.jpg)
TimelinessGap from FIRST TMA non-submission prediction to last engagement date
18
Mean: -14 daysSt Dev.: 80 days
-30 days to 0: 15% of all first predictions+1 days to +14: 9% of all first predictions
NB: last engagement date = Last date of either visit to VLE site, submit assessment, respond to study engagement check emailCaution – this is crude proxy measure
![Page 19: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/19.jpg)
TimelinessGap from changes in TMA non-submission prediction to last engagement
19
Mean: -42 daysSt Dev.: 65 days
-30 days to 0: 19% of all prediction changes+1 days to +14: 12% of all prediction changes
NB: last engagement date = Last date of either visit to VLE site, submit assessment, respond to study engagement check emailCaution – this is crude proxy measure
![Page 20: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/20.jpg)
Intervention potential
20
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:By week 5, 2/3rds of predictions true, but only identify half of non-submittersBy week 14, predictions identify 2/3rds of non-submitters
Volume:Manageable – one new prediction per fortnight
Predicted not to submit = Actually did not submit =New prediction this week =
![Page 21: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/21.jpg)
Intervention potential
21
Based on an analysis of predictions made on 14J presentations for 11 modules…
Accuracy:By week 5, 2/3rds of predictions true, but only identify half of non-submittersBy week 14, predictions identify 2/3rds of non-submitters
Volume:Manageable – one new prediction per fortnight
Timeliness:One in three of all new predictions fall in a six week “window of opportunity” around the last engagement date based on an analysis of predictions for non-completers
Predicted not to submit = Actually did not submit =New prediction this week =
32%68%
31%
Last engagement date
30 days before 14 days after
![Page 22: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/22.jpg)
Lessons
22
Start small – find and nurture your champions
Create your super-users, create your case
studies
Don’t underestimate the guidance required
Create the right story for the user
Foreground the “should we” arguments Repeat, repeat, repeat
![Page 23: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/23.jpg)
Champions
23
![Page 24: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/24.jpg)
Lessons
24
Start small – find and nurture your champions
Create your super-users, create your case
studies
Don’t underestimate the guidance required
Create the right story for the user
Foreground the “should we” arguments Repeat, repeat, repeat
![Page 25: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/25.jpg)
Guidance and help materials
25
![Page 26: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/26.jpg)
Lessons
26
Start small – find and nurture your champions
Create your super-users, create your case
studies
Don’t underestimate the guidance required
Create the right story for the user
Foreground the “should we” arguments Repeat, repeat, repeat
![Page 27: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/27.jpg)
Super-users
27
Briefings Catch-ups Support and feedback
![Page 28: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/28.jpg)
Outcomes of current pilots
●There is a mixed picture in the quantitative analysis on the impact in the pilot tutor groups on withdrawal rates and assignment submissions (note that tutors are self selected and the expectations to intervene are not consistent across the module piloting)
●It is a useful tool for understanding students and their participation●Predictions generally agree with tutors' experience and intuitions of which students
might potentially be at risk●A (potential) USP of OU Analyse was the information provided between the
assignment submission in relation to students' engagement with learning materials●Overall, all tutors interviewed were positive about the affordances of OUA, and are
keen to use it again (for a range of reasons) in their next module
Summary of the interim evaluation of piloting as at March 2016
28
![Page 29: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/29.jpg)
Case studies and vignettes
29
“I love it it’s brilliant. It brings together things I already do […] it’s an easy way to find information without researching around such as in the forums and look for students to see what they do when I have no contact with them […] if they do not answer emails or phones there is not much I can do.
OUA tells me whether they are engaged and gives me an early indicator rather than waiting for the day they submit”
![Page 30: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/30.jpg)
Lessons
30
Start small – find and nurture your champions
Create your super-users, create your case
studies
Don’t underestimate the guidance required
Create the right story for the user
Foreground the “should we” arguments Repeat, repeat, repeat
![Page 31: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/31.jpg)
31http://www.open.ac.uk/students/charter/essential-documents/ethical-use-student-data-learning-analytics-policy
![Page 32: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/32.jpg)
Information for students
32
![Page 33: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/33.jpg)
Lessons
33
Start small – find and nurture your champions
Create your super-users, create your case
studies
Don’t underestimate the guidance required
Create the right story for the user
Foreground the “should we” arguments Repeat, repeat, repeat
![Page 34: Using predictive indicators of student success at scale – implementation successes, issues and lessons from the Open University](https://reader036.vdocument.in/reader036/viewer/2022070523/58ed6f041a28ab6c0a8b45ff/html5/thumbnails/34.jpg)
Are there any questions?
For further details please contact:●Kevin Mayles – [email protected] ●@kevinmayles
References:Calvert, C.E., 2014. Developing a model and applications for probabilities of student success: a case study of predictive analytics. Open Learning: The Journal of Open, Distance and e-Learning. Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z. and Wolff, A., 2015. OU Analyse: Analysing At-Risk Students at The Open University. Learning Analytics Review, no. LAK15-1, March 2015, ISSN: 2057-7494