lessons learned from moodle vle/lms data in the field
TRANSCRIPT
Lessons from Moodle Data
“Dr. John” WhitmerDirector, Analytics and Research
MoodleMoot UK/I | 12-April 2017
1. Learning Analytics Overview & Bb Data Science
2. Research & Modeling Findings
1. Virtual South Carolina Moodle Predictive Model
2. Differences in Student Achievement by Tool Use
3. Discussion
Learning Analytics Overview
Educational Technology Assessment Hierarchy
Does it impact student learning?
(Learning Analytics)
How many people use it? (Adoption)
Does it work? (SLAs)
What is Learning Analytics?
Learning and Knowledge Analytics Conference, 2011
“ ...measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding
and optimizing learning
and the environments
in which it occurs.”
Techniques
• Simulation if X, what Y? (“With this Ultra Learning Analytics trigger rule, how many students would trip notified?”)
• Hypothesis testing: investigate if a specific relationship is true (“What’s the relationship between time spent in a course and student grade”?)
• Data mining: analyze underlying latent patterns in data (“What typical patterns in tool use characterizes BB Learn courses?”)
Key Data Sources
• Moodlerooms & X-Ray
• Learn Managed Hosting & SaaS
• Collaborate Ultra
Main Big Data Sources & Techniques
Commitment to Privacy & Openness
• Analyze data records that are not only removed of PII, but de-personalized (individual & institutional levels)
• Share results and open discussion procedures for analysis to inform broader educational community
• Respect territorial jurisdictions and safe harbor provisions
Virtual South Carolina Online
Feature Importance in Predictive Model
Predictive Accuracy for Risk Categories
Prediction vs. Final Grade
Performance of Predictions
Models Change by Week & by Course Type
So what?
• Rolling out risk model & X-Ray broadly for teachers
• Providing as useful indicator to augment their decisions (not source of absolute truth)
• Remaining challenge: help teachers interpret probabilistic results
Large Scale Research: Student LMS Use vs. Grade
Findings: Relationship LMS Time & Grade
• Question: what is the relationship between student time in LMS and their course grade?
• Investigate at student-course level (one student, one course)
• 1.2M students, 34,519 courses, 788 institutions
• Significant, but effect size < 1%
But strong effect in some courses (n=7,648, 22%)
What makes some for a stronger or weaker relationship?
Tools used? Course design?Quality of activity/effort?
Finding: Access to GradesAt every level, probability of higher grade increases with increased use. Causal? Probably not. Good indicator? Absolutely.
Finding: Course ContentsMore is not always better. Large jump none to some; then no relationship
Finding: Assessments/AssignmentsStudents above mean have lower likelihood of achieving a high grade than students below the mean
Finding: Discussion Forums with low/high avg useCompare courses with low forum use to courses with forum use >1 hour / student average
Implications
• Move beyond LMS use as proxy for effort (where more is always better), and get at finer-grained learning behaviors that are more useful (e.g. students who are struggling to understand material, students who are not prepared).
• Next Steps
– fine-grained understanding of activity over time (e.g. cramming vs. consistent hard working)
– quality of course materials and course design
