analytics4action evaluation framework: a review of evidence-based learning analytics interventions...

A4A

Analytics4Action Evaluation Framework: A Review of Evidence-Based Learning Analytics Interventions at the Open University UKAvinash Boroowa, Senior Project Manager, Open University

Acknowledgements2

Project Lead: Kevin Mayles, Head of Analytics Implementation: Dr. Bart Rienties, Avinash Boroowa, Tom Olney, Tricia WintersEvaluation and analysis: Dr. Bart Rienties, Dr. Christothea Herodotou, Galina Naydenova

@DrBartRienties@kevinmayles@nashman11178

[email protected] | @nashman11178

3

Mileage at the start and end of tripGasoline/petrol


3

The Open University Mission

4


Distance LearningOpen EntryResearch

Explain Module study 30-60 credits --- 360 over 6-7 years4

OU Context

2014/15174k studentsThe average age of our new undergraduate students is 2940% new undergraduates have 1 A-Level or lower on entryOver 21,000 OU students have disabilities868k assessments submitted, 395k phone calls and 176k emails received from students

5


Open entry Distance learning5

Analytics enhancement strategy

The OU is developing its capabilities in 10 key areas that build the underpinning strengths required for the effective deployment of analytics

Adapted from Barton and Court (2012)6


The OU recognised three equally important strengths are required for the effective deployment of analyticsAvailability of DataAnalysis and Creation of insightProcesses that impact student success6

Underpinning organisational strengthsThe OU recognised three equally important strengths are required for the effective deployment of analyticsWe need to ensure we have the right architecture and processes for collecting the right data and making them accessible for analytics we need a big data mind-set

7


Kevin

Right data data gapsAccess data warehouse / integrationTechnology visualisation tool7

Underpinning organisational strengthsThe OU recognised three equally important strengths are required for the effective deployment of analyticsThe university needs world class capability in data science to continually mine the data and build rapid prototypes of simple tools, and a clear pipeline for the outputs to be mainstreamed into operations

8


Prototyping predictive modelling experimentalOperation models have to operate at scale mature modelsInterpretation cycles of activity that align with our business processes to incorporate change / enhancement8

Underpinning organisational strengthsThe OU recognises that three equally important strengths are required for the effective deployment of analyticsBenefits will be realised through existing business processes impacting on students directly and through enhancement of the student learning experience we will develop an analytics mind-set in these areas

For/in/on-action adapted from Schn (1987)9


reflectinaction (while doing something) andonaction (after you have done it)

Read more:Reflection and Reflective Practicehttp://www.learningandteaching.info/learning/reflecti.htm#ixzz49e2X361uUnder Creative Commons License:Attribution Non-Commercial No Derivatives9

Analytics enhancement strategy

Early alert indicators using predictive analyticsPolicy on the ethical use of student data for learning analyticsAnalytics for action evaluation frameworkImpact of learning design on outcomes10


10

11http://www.open.ac.uk/students/charter/essential-documents/ethical-use-student-data-learning-analytics-policy


12


Information for students13


Problem Statement

14


Explain Module studyModule TeamsTutor Student RatioCurriculum presentation cyclesInvestment & review?Speed up review cycleDist learning issue with retention.

14

15We dont need more and more data we need informationThere appears to be information overload I'd like to know how to get through this forest of dataOK, I can see there is a problem what do I do about it?




Student Progression Reports17


Past MI module focussed

Change to student focus based on study goals

17

18Faculty Dashboard


What is the purpose of this report?

The reports are being developed to show student progression and highlight issues that can affect student retention and progression.

Students may be counted multiple times where they are active on more than one qualification.

Questions answered:

Which Qualifications - which qualifications are being studied by students within a faculty?Which Modules - which modules are being studied by students within a faculty?Multiple Module Study: All mod students - what is the study intensity of all students studying a module within a faculty?Multiple Modules: Faculty qual students - what is the study intensity of all qualifications students within a faculty?Assessment Measures: Module - which modules have the lowest TMA01 submission rates within a faculty?Assessment Measures: Qualification - which qualifications have the lowest TMA01 submission rates within a faculty?18

19Qualification Dashboard


What is the purpose of the report?

The reports are being developed to show students progression through a qualification and highlight issues that can affect student retention and progression at the qualification level.

The tabs show different sections within the qualification dashboard. Currently they are:

Qualification Study Plans how many are studying, in which modules, future planned study, last study.Concurrent Study students studying 2 or more modules within current presentation, or plans to overlap study.Study intensity - what other modules are currently being studied - i.e. combinations of 2, 3, 4 and more modulesAssessment measures - TMA01* results by modules linked to the selected qualification19

20Module Dashboard


What is the purpose of this report?

The reports are being developed to show students progression through a module and highlight issues that can affect student retention and progression at the module level.

The tabs show different sections within the qualification dashboard. Currently they are:Module Students - this shows a breakdown of what module students are studyingConcurrent Study - shows a breakdown of what other modules students are currently studyingModule Combinations - shows which other modules students are studyingWithdrawers - A view of the number of students who have either withdrawn themselves or by the university5 and 6. Assessment results by module and broken down by qualification.

20





Developing the Framework The Social Sciences Brief23

but what IS active presentation? We must be engaging in active presentation!


23

24


25


26


Supporting Module Teams27

Briefed over 80 staff Working with 46 modules, meeting each team at least 3 times in the year


28


The A4A Toolkit29

What is the trend in the gap between TMA submissions and registration? How does this compare with previous presentations?What is the relationship between VLE activity and TMA submission? How does this compare with previous presentations?How does the module's VLE activity / TMA submissions / retention rate compare with a comparable module on the same presentation?

How do TMA submission rates compare on the module? Are there any significant drops in submissions (pinchpoints) between TMAs?


Support for module teams30






32

Taking Action33


33

Taking action34

Community of inquiry framework for Online Learning (adjusted based upon Stenbom, Cleveland-Innes and Hrastinski)


SplitAdd example of L192 induction SessionE102K101

34

35

35


SplitAdd example of L192 induction SessionE102K10135

Evaluation Methodologies / Protocols36


36

Evaluation what assistance is available?37


37



analytics4action evaluation framework: a review of evidence-based learning analytics interventions...

Data & Analytics