analytics4action evaluation framework: a review of evidence-based learning analytics interventions...
Post on 21-Apr-2017
184 Views
Preview:
TRANSCRIPT
A4A
Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UKAvinash Boroowa, Senior Project Manager, Open University
Acknowledgements2
Project Lead: Kevin Mayles, Head of Analytics Implementation: Dr. Bart Rienties, Avinash Boroowa, Tom Olney, Tricia WintersEvaluation and analysis: Dr. Bart Rienties, Dr. Christothea Herodotou, Galina Naydenova
@DrBartRienties@kevinmayles@nashman11178
avinash.boroowa@open.ac.uk | @nashman11178
3
Mileage at the start and end of tripGasoline/petrol
avinash.boroowa@open.ac.uk | @nashman11178
3
The Open University Mission
4
avinash.boroowa@open.ac.uk | @nashman11178
Distance LearningOpen EntryResearch
Explain Module study 30-60 credits --- 360 over 6-7 years4
OU Context
2014/15174k studentsThe average age of our new undergraduate students is 2940% new undergraduates have 1 A-Level or lower on entryOver 21,000 OU students have disabilities868k assessments submitted, 395k phone calls and 176k emails received from students
5
avinash.boroowa@open.ac.uk | @nashman11178
Open entry Distance learning5
Analytics enhancement strategy
The OU is developing its capabilities in 10 key areas that build the underpinning strengths required for the effective deployment of analytics
Adapted from Barton and Court (2012)6
avinash.boroowa@open.ac.uk | @nashman11178
The OU recognised three equally important strengths are required for the effective deployment of analyticsAvailability of DataAnalysis and Creation of insightProcesses that impact student success6
Underpinning organisational strengthsThe OU recognised three equally important strengths are required for the effective deployment of analyticsWe need to ensure we have the right architecture and processes for collecting the right data and making them accessible for analytics we need a big data mind-set
7
avinash.boroowa@open.ac.uk | @nashman11178
Kevin
Right data data gapsAccess data warehouse / integrationTechnology visualisation tool7
Underpinning organisational strengthsThe OU recognised three equally important strengths are required for the effective deployment of analyticsThe university needs world class capability in data science to continually mine the data and build rapid prototypes of simple tools, and a clear pipeline for the outputs to be mainstreamed into operations
8
avinash.boroowa@open.ac.uk | @nashman11178
Prototyping predictive modelling experimentalOperation models have to operate at scale mature modelsInterpretation cycles of activity that align with our business processes to incorporate change / enhancement8
Underpinning organisational strengthsThe OU recognises that three equally important strengths are required for the effective deployment of analyticsBenefits will be realised through existing business processes impacting on students directly and through enhancement of the student learning experience we will develop an analytics mind-set in these areas
For/in/on-action adapted from Schn (1987)9
avinash.boroowa@open.ac.uk | @nashman11178
reflectinaction (while doing something) andonaction (after you have done it)
Read more:Reflection and Reflective Practicehttp://www.learningandteaching.info/learning/reflecti.htm#ixzz49e2X361uUnder Creative Commons License:Attribution Non-Commercial No Derivatives9
Analytics enhancement strategy
Early alert indicators using predictive analyticsPolicy on the ethical use of student data for learning analyticsAnalytics for action evaluation frameworkImpact of learning design on outcomes10
avinash.boroowa@open.ac.uk | @nashman11178
10
11http://www.open.ac.uk/students/charter/essential-documents/ethical-use-student-data-learning-analytics-policy
avinash.boroowa@open.ac.uk | @nashman11178
12
avinash.boroowa@open.ac.uk | @nashman11178
Information for students13
avinash.boroowa@open.ac.uk | @nashman11178
Problem Statement
14
avinash.boroowa@open.ac.uk | @nashman11178
Explain Module studyModule TeamsTutor Student RatioCurriculum presentation cyclesInvestment & review?Speed up review cycleDist learning issue with retention.
14
15We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?
avinash.boroowa@open.ac.uk | @nashman11178
16We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?
avinash.boroowa@open.ac.uk | @nashman11178
Student Progression Reports17
avinash.boroowa@open.ac.uk | @nashman11178
Past MI module focussed
Change to student focus based on study goals
17
18Faculty Dashboard
avinash.boroowa@open.ac.uk | @nashman11178
What is the purpose of this report?
The reports are being developed to show student progression and highlight issues that can affect student retention and progression.
Students may be counted multiple times where they are active on more than one qualification.
Questions answered:
Which Qualifications - which qualifications are being studied by students within a faculty?Which Modules - which modules are being studied by students within a faculty?Multiple Module Study: All mod students - what is the study intensity of all students studying a module within a faculty?Multiple Modules: Faculty qual students - what is the study intensity of all qualifications students within a faculty?Assessment Measures: Module - which modules have the lowest TMA01 submission rates within a faculty?Assessment Measures: Qualification - which qualifications have the lowest TMA01 submission rates within a faculty?18
19Qualification Dashboard
avinash.boroowa@open.ac.uk | @nashman11178
What is the purpose of the report?
The reports are being developed to show students progression through a qualification and highlight issues that can affect student retention and progression at the qualification level.
The tabs show different sections within the qualification dashboard. Currently they are:
Qualification Study Plans how many are studying, in which modules, future planned study, last study.Concurrent Study students studying 2 or more modules within current presentation, or plans to overlap study.Study intensity - what other modules are currently being studied - i.e. combinations of 2, 3, 4 and more modulesAssessment measures - TMA01* results by modules linked to the selected qualification19
20Module Dashboard
avinash.boroowa@open.ac.uk | @nashman11178
What is the purpose of this report?
The reports are being developed to show students progression through a module and highlight issues that can affect student retention and progression at the module level.
The tabs show different sections within the qualification dashboard. Currently they are:Module Students - this shows a breakdown of what module students are studyingConcurrent Study - shows a breakdown of what other modules students are currently studyingModule Combinations - shows which other modules students are studyingWithdrawers - A view of the number of students who have either withdrawn themselves or by the university5 and 6. Assessment results by module and broken down by qualification.
20
21We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?
avinash.boroowa@open.ac.uk | @nashman11178
22We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?
avinash.boroowa@open.ac.uk | @nashman11178
Developing the Framework The Social Sciences Brief23
but what IS active presentation? We must be engaging in active presentation!
avinash.boroowa@open.ac.uk | @nashman11178
23
24
avinash.boroowa@open.ac.uk | @nashman11178
25
avinash.boroowa@open.ac.uk | @nashman11178
26
avinash.boroowa@open.ac.uk | @nashman11178
Supporting Module Teams27
Briefed over 80 staff Working with 46 modules, meeting each team at least 3 times in the year
avinash.boroowa@open.ac.uk | @nashman11178
28
avinash.boroowa@open.ac.uk | @nashman11178
The A4A Toolkit29
What is the trend in the gap between TMA submissions and registration? How does this compare with previous presentations?What is the relationship between VLE activity and TMA submission? How does this compare with previous presentations?How does the module's VLE activity / TMA submissions / retention rate compare with a comparable module on the same presentation?
How do TMA submission rates compare on the module? Are there any significant drops in submissions (pinchpoints) between TMAs?
avinash.boroowa@open.ac.uk | @nashman11178
Support for module teams30
avinash.boroowa@open.ac.uk | @nashman11178
31We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?
avinash.boroowa@open.ac.uk | @nashman11178
32We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?
avinash.boroowa@open.ac.uk | @nashman11178
32
Taking Action33
avinash.boroowa@open.ac.uk | @nashman11178
33
Taking action34
Community of inquiry framework for Online Learning (adjusted based upon Stenbom, Cleveland-Innes and Hrastinski)
avinash.boroowa@open.ac.uk | @nashman11178
SplitAdd example of L192 induction SessionE102K101
34
35
35
avinash.boroowa@open.ac.uk | @nashman11178
SplitAdd example of L192 induction SessionE102K10135
Evaluation Methodologies / Protocols36
avinash.boroowa@open.ac.uk | @nashman11178
36
Evaluation what assistance is available?37
avinash.boroowa@open.ac.uk | @nashman11178
37
38We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?
avinash.boroowa@open.ac.uk | @nashman11178
top related