tlc2016 - learning analytics - one universities journey

20
www.derby.ac.uk Introducing Analytics: One University’s Journey Sandra Stevenson-Revill, Ruth Grindey & Chris Bell

Upload: blackboardemea

Post on 11-Apr-2017

118 views

Category:

Education


1 download

TRANSCRIPT

Page 1: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Introducing Analytics: One University’s JourneySandra Stevenson-Revill, Ruth Grindey & Chris Bell

Page 2: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

You have to do the tech bit first, sorry.

• Quite simple– We build the servers– Blackboard install the software– We install the B2 (Building Block)– Work together to get the installation pulling data– Pulling the data in takes time if you have a lot

• The complex bit– Mixing in the Student data, we have a complex course build in

Blackboard that makes re-aligning the data in to terms difficult– Terms v Academic Years

Page 3: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Testing, Test Data & Data Validation

• How do you test?

– We have true data in our test system from past years– Testing was done on real data that meant something to the

testers– Blackboard, ITS, UDOL, Learning Tech’s all in the same room

testing together

• Why test?

– Fields can be used for different data– Have we cross-referenced data correctly

Page 4: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Online Oncampus

Establishing a Pilot Group (September 2015-September 2016)

– 30 users– Across all Colleges– Identify common questions and themes from

tutors

Initial Dashboard Creation

– “Student Journey”– Personal Tutoring

• Time spent in Blackboard• Number of times accessed• Which tools and documents?

Focusing on Students at Risk

• All student support staff• All teaching areas• Continue to identify key questions

Identified four Dashboards, one for each audience using out of the box reports

• Senior Management (9 Reports), • Academic Management (13

Reports),• Customer Service Management (7

Reports)• Content Management (18 Reports)

Page 5: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Why Analytics?

• Proactive Customer Service (NOT REACTIVE)

– Pastoral Care • You’re not interacting can we help?• Struggling to study online – can we improve the materials?• Academics - what is the relationship with the student?• Is this student a one off or is it all the students on the course?

• Academic

– Programme Management• Course Design• Delivery TLA

Page 6: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

What about the Performance Dashboard, Retention Centre and Module Reports?

• Higher level access across Programmes

• 360 view of student activity across all modules

• Identify students on grade and classification boundaries and monitor ALL activity

Page 7: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

A TEL Dashboard • TEL Dashboard

– High usage (hot-spots!), can we identify and model good practice?– Low usage (target staff development resource)– Top 10 (investigate top 10 across all Colleges to identify good practice)

• College Overview of Tool Usage, are we (they) getting return on investment?

• Modules with High student numbers (High stakes/Quick wins), raise awareness of:

– Adaptive release– PeerMark– Group tool– Opportunities to develop peer support mechanisms (cohort identity)

Page 8: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

A TEL Dashboard • Single Course Query/Health Check (prior to Staff Development/one-to-one Consultations):

– Usage (student numbers, student access)– Tool usage, what is and isn’t being used?– Folder depth? High level of organisation and structure?– Compare with previous iterations? What is the history of the module?

Page 9: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Supporting Teaching

• Providing academics with more support

– How to improve student interaction/engagement– How to be more effective/efficient using technology– Evaluating the effectiveness of the technology in the delivery.– Content analysis to ensure appropriate level of pedagogy

Identifying a Student who may be at Risk

Page 10: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Identifying a Student who may be at RiskBb A4L – Learn Course At A Glance Report (Out of the Box Report)

Instructor:

Course ID:

Avg. Student Avg. Student Avg. SIS Grade Letter

6.0 35.00 58.49 Fail

6.0 32.00 58.49 Fail

6.0 35.00 58.49 Fail

6.0 35.00 58.49 Fail

6.0 10.00 58.49 Fail22 213.9 07 27.4 199 347.0Student E (100009850) 01/31/2016

Student D (100396803) 02/11/2016 17 27.4 197 347.0 178 213.9 0

162 213.9 626 27.4 125 347.0Student C (100394157) 02/15/2016 11/05/2015

123 213.9 014 27.4 1 347.0Student B (100396130) 02/10/2016

Student A (100381731) 02/22/2016 51 27.4 456 347.0 374 213.9 0

Interactions Submissions Grade Center Score SIS GradeStudent Course Access Trend Date of Last

AccessDate of Last Submission

Student Avg. Student Avg. Student Avg.Student Activity Summary (133 Students) Course Accesses Minutes

Student

Submissions Avg. vs. Department Avg.

Legend

> Avg + 10% Within Avg +/- 10%

< Avg - 10% NA

187.4

Submissions 6 4.3

Avg. ActivityInteraction Avg. vs. Department Avg.

Accesses 27 19.7

Minutes 347 344.3

Interactions 214

Tool 9.2% 61.9%

Tool 14 1.9

% of Items Accessed per

StudentAssessment 32.6% 38.8%

Content 16.8% 17.4%

% Difference

Item Count

Assessment 7 7.3Minutes Avg. vs. Department Avg.

Content 86 76.9

Accesses Avg. vs. Department Avg.

Psychology: Traditions and Skills (2015-UDOL-2015-09-28-4PS511)

Learn Course Information

This Course UDOL-Life & Natural Sciences (92 Courses)

This Course

Dept. Avg. (No SIS

Instruction Method Mapped

Courses)

College: University of Derby Online

2015-UDOL-2015-09-28-4PS511 Status: Available Students Enrolled: 133 Department: UDOL-Life & Natural Sciences

Alistair Turvill Term: 2015-2016 Academic Year

Instruction Method: No SIS Instruction Method Mapped

The report highlighted the following:

Out of 133 student enrolled, five students had submitted an assessment and ‘Failed’ (obtained a grade less that 40). This is where the investigation started to determine if there were any correlation between ‘Fail’ student interaction, and ‘Pass’ student interaction.

Page 11: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Identifying a Student who may be at RiskStudents with a Grade <40 (Pass Rate)

Some interesting data . . .

Avg. Student Avg. Student Avg. SIS Grade Letter

6.0 35.00 58.49 Fail

6.0 32.00 58.49 Fail

6.0 35.00 58.49 Fail

6.0 35.00 58.49 Fail

6.0 10.00 58.49 Fail22 213.9 07 27.4 199 347.0Student E (100009850) 01/31/2016

Student D (100396803) 02/11/2016 17 27.4 197 347.0 178 213.9 0

162 213.9 626 27.4 125 347.0Student C (100394157) 02/15/2016 11/05/2015

123 213.9 014 27.4 1 347.0Student B (100396130) 02/10/2016

Student A (100381731) 02/22/2016 51 27.4 456 347.0 374 213.9 0

Interactions Submissions Grade Center Score SIS GradeStudent Course Access Trend Date of Last

AccessDate of Last Submission

Student Avg. Student Avg. Student Avg.Student Activity Summary (133 Students) Course Accesses Minutes

Student

Grade achieved against the average grade of the group

Student interaction against the average of the group.

Q. Why has Student A ‘Failed’ with an above average interaction?

Student time in learning against the average of the group.Q. Why has Student A ‘Failed’ with an above average minutes spent in learning interactions?

Student course accesses against the average of the group.

Q. Why has Student A significantly above average for course accesses?

Page 12: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Identifying a Student who may be at RiskStudents Interaction with a Grade <40 (Pass Rate) compared to a Grade >69(Students highlighted in grey have ‘Failed’ the module, Students highlighted in taupe achieved the highest grade of 80)

Some interesting comparisons . . .Avg. Student Avg. Student Avg. SIS Grade

Letter6.0 35.00 58.49 Fail

6.0 72.00 58.49 Pass

6.0 75.00 58.49 Pass

6.0 74.00 58.49 Pass

6.0 70.00 58.49 Pass

6.0 80.00 58.49 Pass

6.0 70.00 58.49 Pass

6.0 32.00 58.49 Fail

6.0 78.00 58.49 Pass

6.0 75.00 58.49 Pass

6.0 35.00 58.49 Fail

6.0 78.00 58.49 Pass

6.0 75.00 58.49 Pass

6.0 70.00 58.49 Pass

6.0 35.00 58.49 Fail

6.0 70.00 58.49 Pass

6.0 80.00 58.49 Pass

6.0 75.00 58.49 Pass

6.0 75.00 58.49 Pass

6.0 10.00 58.49 Fail22 213.9 07 27.4 199 347.0Student E (100009850) 01/31/2016

24Student T (100374592) 01/16/2016 11/10/2015 25 27.4 355 347.0 214 213.9

Student S (100384121) 01/18/2016 10/28/2015 22 27.4 305 347.0 157 213.9 3

893 213.9 293 27.4 23 347.0Student R (100352984) 02/01/2016 11/26/2015

8Student Q (100394380) 01/25/2016 10/29/2015 33 27.4 49 347.0 175 213.9

Student D (100396803) 02/11/2016 17 27.4 197 347.0 178 213.9 0

12Student P (100310881) 01/16/2016 11/26/2015 37 27.4 945 347.0 371 213.9

7Student O (100356074) 02/05/2016 11/08/2015 28 27.4 5 347.0 494 213.9

596 213.9 1385 27.4 2263 347.0Student N (100390056) 02/19/2016 11/12/2015

162 213.9 626 27.4 125 347.0Student C (100394157) 02/15/2016 11/05/2015

0Student M (100355870) 01/15/2016 15 27.4 279 347.0 210 213.9

1Student L (100374290) 01/14/2016 09/29/2015 11 27.4 359 347.0 77 213.9

123 213.9 014 27.4 1 347.0Student B (100396130) 02/10/2016

Student K (100396735) 02/14/2016 12/01/2015 48 27.4 1946 347.0 369 213.9 17

17Student J (100383652) 02/02/2016 11/17/2015 56 27.4 905 347.0 393 213.9

492 213.9 1333 27.4 67 347.0Student I (100396781) 01/28/2016 12/10/2015

17Student H (100356192) 02/15/2016 12/06/2015 63 27.4 290 347.0 451 213.9

6

124 213.9 5

Student G (100375683) 02/04/2016 10/31/2015 16 27.4 566 347.0 95 213.9

24 27.4 230 347.0Student F (100378813) 02/01/2016 10/05/2015

Student A (100381731) 02/22/2016 51 27.4 456 347.0 374 213.9 0

Interactions Submissions Grade Center Score SIS GradeStudent Course Access Trend Date of Last

AccessDate of Last Submission

Student Avg. Student Avg. Student Avg.Student Activity Summary (133 Students) Course Accesses Minutes

Student

High engagement compared to students who ‘Failed’ with the exception of Student A

Students who ‘Failed’ had no Last Submission Date.

Q. Is this significant?

Page 13: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Identifying a Student who may be at RiskBb A4L – Student at a Glance (Out of the Box Report) for Student A

GradeCenter Not Used

Avg. Student Student Avg. Student Avg. SIS Grade Letter

27.4 0 35.00 49.69 Fail

40.8 0 72.00 55.39 Pass

X 17.1 0 0.00 Pass

X 18.3 0 0.00 Pass

X 5.0 0 0.00 Pass

X 5.0 0 0.00 Pass

Student At-a-Glance Report Back to Report List

Report Help

Student A (100381731) - Data as of 24th February 2016

General Information Student Avg. Per Course vs. Enrolled Courses Averages (463 Students)

Student ID: 100381731 Email: [email protected]

Student Enrolled Courses

Course Accesses Interactions SubmissionsStanding: No SIS Student Match Phone: 07701047962

Course Accesses vs. Avg. Student Enrolled Courses Submissions vs. Avg.

Primary Major: Bachelor of Science (Honours) in P sychology Class Level: No SIS Match

College: University of Derby Online Level: Undergraduate

> Avg + 10% Within Avg +/- 10%

< Avg - 10% NA

Interactions vs. Avg. Minutes vs. Avg.

Legend

Minutes Interactions Submissions Grade Center Score SIS GradeBlackboard Course History Course AccessesAvg. Avg.

2015-2016 Academic Year

Psychology: Traditions and Skills (2015-UDOL-2015-09-28-4PS511) - Trimester One 2015/16

133 51 456 347.0 374 213.9 6.0

Student Student Avg. StudentTerm Learn Course No Items Added

Interactions Trend Enrolled Count

417.8 10.3

2015-2016 Academic Year

Introduction to Cognitive Psychology (2015-UDOL-2016-02-01-4PS508) - Trimester Tw o 2015/16

237 32 364 319.8 207 177.4 1.8

46 786 951.3 3462015-2016 Academic Year

Research Methods and Analysis in Psychology (2015-UDOL-2015-09-28-4PS512) - Trimester One 2015/16

223

219.2 2.5

2015-2016 Academic Year

Introduction to Biological Psychology (2015-UDOL-2016-05-30-4PS507) - Trimester Three 2015/16

156 7 1 8.1 9 11.3 0.0

28 169 467.6 3232015-2016 Academic Year

Introduction to Developmental Psychology (2015-UDOL-2016-02-01-4PS509) - Trimester Tw o 2015/16

219

10.8 0.04 2 5.3 52015-2016 Academic Year

Introduction to Social Psychology (2015-UDOL-2016-05-30-4PS510) - Trimester Three 2015/16

177

Student A is on an Accelerated pathway taking two modules per trimester; failed one module and passed one module

Page 14: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Identifying a Student who may be at RiskStudent A – A Comparison of Engagement (Data as of the 24th February 2016)

GradeCenter Not Used

Avg. Student Student Avg. Student Avg. SIS Grade Letter

27.4 0 35.00 49.69 Fail

40.8 0 72.00 55.39 Pass

Minutes Interactions Submissions Grade Center Score SIS GradeBlackboard Course History Course AccessesAvg. Avg.

2015-2016 Academic Year

Psychology: Traditions and Skills (2015-UDOL-2015-09-28-4PS511) - Trimester One 2015/16

133 51 456 347.0 374 213.9 6.0

Student Student Avg. StudentTerm Learn Course No Items Added

Interactions Trend Enrolled Count

417.8 10.346 786 951.3 3462015-2016 Academic Year

Research Methods and Analysis in Psychology (2015-UDOL-2015-09-28-4PS512) - Trimester One 2015/16

223

As an Accelerated Pathway Student the data highlights that Student A has failed one module but passed the other, the data highlights a significant difference in the number of minutes engagement between the two modules, and their averages. Student A had an increase of 41% more engagement in the ‘Passed’ module in comparison to the ‘Failed’ Module.

Interestingly there is a significant difference in the average interaction for the two modules. Student A has above average engagement for the ‘Failed’ module, but below average engagement for the ‘Passed’ module

Q. What are the differences between the learning content for the two modules, can anything be deduced from this?

Page 15: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Identifying a Student who may be at RiskStudent A – A Comparison of Engagement

GradeCenter Not Used

Avg. Student Student Avg. Student Avg. SIS Grade Letter

27.4 0 35.00 49.69 Fail

40.8 0 72.00 55.39 Pass

Minutes Interactions Submissions Grade Center Score SIS GradeBlackboard Course History Course AccessesAvg. Avg.

2015-2016 Academic Year

Psychology: Traditions and Skills (2015-UDOL-2015-09-28-4PS511) - Trimester One 2015/16

133 51 456 347.0 374 213.9 6.0

Student Student Avg. StudentTerm Learn Course No Items Added

Interactions Trend Enrolled Count

417.8 10.346 786 951.3 3462015-2016 Academic Year

Research Methods and Analysis in Psychology (2015-UDOL-2015-09-28-4PS512) - Trimester One 2015/16

223

Data as of 24th February 2016

GradeCenter Not Used

Avg. Student Student Avg. Student Avg.

29.2 0 40.00 52.72

43.1 0 72.00 59.04

X 36.4 0 0.00

41.0 0 0.00484.7 5.844 289 1112.7 4252015-2016 Academic Year

Introduction to Developmental Psychology (2015-UDOL-2016-02-01-4PS509) - Trimester Tw o 2015/16

213

434.6 10.4

2015-2016 Academic Year

Introduction to Cognitive Psychology (2015-UDOL-2016-02-01-4PS508) - Trimester Tw o 2015/16

220 50 370 813.1 328 366.3 4.3

49 787 986.2 3582015-2016 Academic Year

Research Methods and Analysis in Psychology (2015-UDOL-2015-09-28-4PS512) - Trimester One 2015/16

219

Avg. Avg.

2015-2016 Academic Year

Psychology: Traditions and Skills (2015-UDOL-2015-09-28-4PS511) - Trimester One 2015/16

130 58 467 356.3 423 223.1 6.2

Student Student Avg. StudentTerm Learn Course No Items Added

Interactions Trend Enrolled Count

Minutes Interactions Submissions Grade Center Score SIS GradeBlackboard Course History Course Accesses

Data as of 4th April 2016

Trim

este

rOn

eTr

imes

ter

Two

Student A has undertaken referral work, and has now passed the ‘Failed’ module, with the increase in engagement being recorded

The data highlight Student A is well below the average interactions, and engagement minutes compared to peers studying on the same module, and have less interactions and engagement than they had for Trimester One modules studied. This highlights a risk as teaching for Trimester Two concluded on the 8th April, 2016.

Q. Is this Student ‘@ Risk’?

Page 16: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

GradeCenter Not Used

Avg. Student Student Avg. Student Avg. SIS Grade Letter

29.2 0 40.00 52.72 Pass

43.1 0 72.00 59.04 Pass434.6 10.449 787 986.2 3582015-2016 Academic Year

Research Methods and Analysis in Psychology (2015-UDOL-2015-09-28-4PS512) - Trimester One 2015/16

219

Avg. Avg.

2015-2016 Academic Year

Psychology: Traditions and Skills (2015-UDOL-2015-09-28-4PS511) - Trimester One 2015/16

130 58 467 356.3 423 223.1 6.2

Student Student Avg. StudentTerm Learn Course No Items Added

Interactions Trend Enrolled Count

Minutes Interactions Submissions Grade Center Score SIS GradeBlackboard Course History Course Accesses

Identifying Student’s Interactions/EngagementStudent A – Interactions

Clicking on the number of interaction enable you to drill down to see individuals interaction details

The interaction detail for Student A highlights that the student didn’t engage with the ‘Failed’ module content until 3rd November, five weeks after the start of formal teaching. In comparison Student A started engaging with the ‘Passed’ module on the 1st August, eight weeksprior to the start of teaching

Further analyses is required due tonot all content being held in Bb Learn

Date Item Access Date Course Item Name Item Type

01/09/2016

1/9/2016 3:17:41 PM Assessment Content Folder/Content Area

Submission Point Content Folder/Content Area

CW1 –Standard Submission Point TurnItIn

CW1 Submission Points Content Folder/Content Area

11/03/2015

11/3/2015 6:33:12 PM Assessment Content Folder/Content Area

Learning Activities Content Folder/Content Area

Submission Point Content Folder/Content Area

Unit 2 - Early Inf luences of Psychology as a Science (Traditions) / Academic Sources (Skills)

Content Folder/Content Area

Unit 3 - Early Approaches (Traditions) | What is an Essay? (Skills)

Content Folder/Content Area

Unit 4 - Behaviourism (Traditions) | Referencing (Skills)

Content Folder/Content Area

Formative Essay TurnItIn

11/3/2015 9:04:06 PM Assessment Content Folder/Content Area

Submission Point Content Folder/Content Area

Formative Essay TurnItIn

38 75

5 0

Interactions Minutes

Interaction Details Back to Report List

Report Help

Student A (100381731)

Psychology: Traditions and Skills (2015-UDOL-2015-09-28-4PS511)

Page 17: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Identifying a Student who may be at RiskStudent A is currently being reviewed, the data identifies them as being at risk of failure, however, due to analytics being in its infancy, and not all of the learning content being held within Bb Learn, the data is not yet conclusive and further information is needed to formally categorise Student A as being ‘@ Risk’ by just using the data alone.

Next steps will be . . .

1. Analyse the results for Trimester Two for Student A modules; a ‘Fail’ results will trigger

a. Is an Accelerated Pathway suitable for Student A?b. Are all Accelerated Pathway Student ‘@ Risk’ if they ‘Fail’ a module – further

investigation?

2. Analyse the differences in module content for module with higher average interactions

3. Analyse in detail Student Interactions/Engagement, now that all learning content is held within Blackboard Learn

4. Identify ‘@ Risk’ engagement thresholds

5. Create a ‘soft’ roll out plan for identifying student who may be ‘@ Risk’ includingindividual action/support plans

Page 18: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

Things to Consider• Removal of legacy tools from a dashboard?

• Tools named as per Blackboard

• Ability to create a KPI Report/Dashboard against threshold standards of minimum engagement

• Averages - they average all modules over the year including those that have not been delivered yet – hence a very low average for interactions

• Week Numbers are not aligned to UDOL teaching pattern as must align for all of Derby.

Page 19: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

The future

• Incorporate Blackboard Analytic for Learn data into the Universities data warehouses to increase analyses, reporting and business intelligence

• Incorporate Students into the audiences to provide them with their own analytical tool

• Work collaboratively with Blackboard to inform them of British reporting requirements, and enhance the reporting/dashboard functionality

• Use BI for true predictive analysis, to inform recruitment and retention

• Dashboard Consolidation (PeopleSoft/Cognos/Pyramid)?!• Student-facing View?• Plugging the Learner Analytics gap in our quest for Business Intelligence• Managing expectations!• Managing fear and suspicion!

Page 20: TLC2016 - Learning Analytics - One Universities Journey

www.derby.ac.uk

That’s All Folks !!!!