lessons learned from moodle vle/lms data in the field

Lessons from Moodle Data

“Dr. John” WhitmerDirector, Analytics and Research

MoodleMoot UK/I | 12-April 2017

1. Learning Analytics Overview & Bb Data Science

2. Research & Modeling Findings

1. Virtual South Carolina Moodle Predictive Model

2. Differences in Student Achievement by Tool Use

3. Discussion

Learning Analytics Overview

Educational Technology Assessment Hierarchy

Does it impact student learning?

(Learning Analytics)

How many people use it? (Adoption)

Does it work? (SLAs)

What is Learning Analytics?

Learning and Knowledge Analytics Conference, 2011

“ ...measurement, collection, analysis and

reporting of data about learners and their

contexts, for purposes of understanding

and optimizing learning

and the environments

in which it occurs.”

Techniques

• Simulation if X, what Y? (“With this Ultra Learning Analytics trigger rule, how many students would trip notified?”)

• Hypothesis testing: investigate if a specific relationship is true (“What’s the relationship between time spent in a course and student grade”?)

• Data mining: analyze underlying latent patterns in data (“What typical patterns in tool use characterizes BB Learn courses?”)

Key Data Sources

• Moodlerooms & X-Ray

• Learn Managed Hosting & SaaS

• Collaborate Ultra

Main Big Data Sources & Techniques

Commitment to Privacy & Openness

• Analyze data records that are not only removed of PII, but de-personalized (individual & institutional levels)

• Share results and open discussion procedures for analysis to inform broader educational community

• Respect territorial jurisdictions and safe harbor provisions

Virtual South Carolina Online

Feature Importance in Predictive Model

Predictive Accuracy for Risk Categories

Prediction vs. Final Grade

Performance of Predictions

Models Change by Week & by Course Type

So what?

• Rolling out risk model & X-Ray broadly for teachers

• Providing as useful indicator to augment their decisions (not source of absolute truth)

• Remaining challenge: help teachers interpret probabilistic results

Large Scale Research: Student LMS Use vs. Grade

Findings: Relationship LMS Time & Grade

• Question: what is the relationship between student time in LMS and their course grade?

• Investigate at student-course level (one student, one course)

• 1.2M students, 34,519 courses, 788 institutions

• Significant, but effect size < 1%

But strong effect in some courses (n=7,648, 22%)

What makes some for a stronger or weaker relationship?

Tools used? Course design?Quality of activity/effort?

Finding: Access to GradesAt every level, probability of higher grade increases with increased use. Causal? Probably not. Good indicator? Absolutely.

Finding: Course ContentsMore is not always better. Large jump none to some; then no relationship

Finding: Assessments/AssignmentsStudents above mean have lower likelihood of achieving a high grade than students below the mean

Finding: Discussion Forums with low/high avg useCompare courses with low forum use to courses with forum use >1 hour / student average

Implications

• Move beyond LMS use as proxy for effort (where more is always better), and get at finer-grained learning behaviors that are more useful (e.g. students who are struggling to understand material, students who are not prepared).

• Next Steps

– fine-grained understanding of activity over time (e.g. cramming vs. consistent hard working)

– quality of course materials and course design

Discussion & Contact Information

John Whitmer (john.whitmer@blackboard.com)johncwhitmer