the impact of learning analytics on elearning design patrick devey, ph.d. chief learning officer...
TRANSCRIPT
The Impact of Learning Analytics on eLearning Design
Patrick Devey, Ph.D.Chief Learning Officer
KnowledgeOne Inc.Montréal, Québec, Canada
THE MISSING STUDENTS CAPER
“I seem to be missing a significant number of students at the final exam…”
-FINA 200 Professor
0%
5%
10%
15%
20%
25%
30%
Voluntary Student Dropout
eConcordia FINA 200
“help…?”-FINA 200 Professor
WHERE DO WE BEGIN?
Student Logs (from LMS)
Over 50,000 entries!!!
Hazard Rate using Registration Data
1 2 3 4 5 6 7 8 9.000
.010
.020
.030
.040
.050
.060
FINA 200
Time (Week)
Haz
ard
Rate
2 3 4 5 6 7 8 9 10.000
.010
.020
.030
.040
.050
.060
Time (Week)
Haz
ard
Rate
Ass
ign
men
t 1
Ass
ign
men
t 2
Ass
ign
men
t 3
Hazard Rate using Registration Data
THE MISSING STUDENTS CAPER
…will return after these informative messages!
Learning Analytics
Data must be:• relevant to the learners and to the instructors,• focuses on individual learners or a course,• leads to informed decisions to improve learning
services or products at the learner/course level.
“The measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.”
-The Society for Learning Analytics Research (SoLAR)
13
Pascarella & Chapman (1983)
Longitudinal Data: Path Analysis
R2 = .120
Data Sources
LMS
Google Analytics
Surveys
Industry Statistics
& Reports
EnrolmentHR System
xAPICustom
INDUSTRY STATISTICS & REPORTS
Easy to get, but do they apply to your situation?
http://en.wikipedia.org/wiki/Usage_share_of_operating_systems
Nov-07 Mar-09 Aug-10 Dec-11 May-130.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Browser History
Internet Explorer Firefox Chrome Safari Opera
Google Chrome Safari MS Internet Explorer Firefox Other0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
Browser Comparison
W3 Concordia University
Windows Linux Mac Mobile0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
OS Comparison
w3 eConcordia
How could this data be used?
• What are the main browsers used by our learners?– Quality assurance testing– Compatibility (HTML5, plug-ins, etc…)
• What are the primary operating systems being used by your learners?
• Are your learners making use of mobile devices?• What are the trends?
– Should you be making preparations because of them?
LEARNING MANAGEMENT SYSTEM
It’s already being collected, might as well use it!
How could this data be used?
• When are learners most likely to be active?– Schedule live office hours– Schedule server maintenance
• Are learners achieving the learning objectives?– Adjustments to assessments– Adjustments to existing content– Addition of remedial content
• Is time-on-task correlated with performance?– Log-ins vs. performance– Identifying learners who have “stopped out”
GOOGLE ANALYTICS
Tuning in to the “Greatest Hits”
Demographic Data
Browser Statistics, OS, Mobile, etc…
Qui
z 1
Qui
z 2
Qui
z 3
Fina
l Exa
m
Qu
iz 1
Ass
ign
men
t 1O
rie
nta
tion
Se
ssio
n
Ass
ign
men
t 2
Qu
iz 2
Ass
ign
men
t 3
Qu
iz 3
Fin
al E
xamMidterm Break
031.aspx 042.aspx
How could this data be used?
• Is my visual design effective?– Do I have buttons that are not being used? Can the site be
optimized?• How are learners navigating through the course website?
– Are the navigation buttons being used as designed? Can they be improved or adapted?
• When are learners using particular course materials?– Is there an opportunity to send remedial content when it is
relevant?• How long are learners spending on individual sections?
– Is there an opportunity to improve the content?
DESIGN YOUR OWN DATA COLLECTION
If the data you want is not being collected, design a way to get it!
• Who are the students enrolling in online courses at Concordia University? How do they compare to the general student population at Concordia as well as to the distance education literature?
• Why are students enrolling in online courses? What expectations do they have about their online course?
• Why do students drop out of online courses? Are there demographic, attitudinal, integration, and/or behavioural variables related to persistence?
• When do students drop out of online courses?
• What interventions can the institution make to curtail attrition in their online courses?
What Patrick wanted to know
Timeframe of the Study
1 2 3 4 5 6 7 8 9 10 11 12 13
Weeks of the Semester
1. Web-Based Learning
Questionnaire
4. Registration Data3. Exit Survey
DNE DISCDropout Zone
2. Weekly Survey
44
THE MISSING STUDENTS CAPER
…and now, back to:
Finance: Course Satisfaction
2 3 4 5 6 7 8 9 1064
66
68
70
72
74
76
78
Week
Aver
age
Finance: Hazard Rate/Satisfaction
2 3 4 5 6 7 8 9 10.000
.010
.020
.030
.040
.050
.060
Time (Week)
Haza
rd R
ate
48
Ass
ign
men
t 1
Ass
ign
men
t 2
Ass
ign
men
t 3
2 3 4 5 6 7 8 9 1064
66
68
70
72
74
76
78
Week
Aver
age
The Facts
• Significant portion of the class were non-business students– Struggling with basic concepts such as time value
of money calculations• Older, worked full-time, studied part-time
– Did not have time to seek out instructor “in-between classes” or to go to campus
• Dropping out because of a fear of poor performance in the course
Interventions
• Live virtual office hours and tutorials scheduled when they were most needed– Dealt with the assignments– Sessions were recorded and posted on the course website
• Development of additional remedial content to address problem areas
• Provided links to the remedial content in the course content when they were relevant
• Altered assessments (rubric) for non-business students
CASE
CLOSED
(kind of)
WHAT DO YOU WANT TO IMPROVE?
What do you want to improve?
• Retention rate• Transfer of training• Achievement of learning objectives• Performance on assessments• Usability of the course website• Services to instructors and learners• ???
Take Action
1. Go beyond pass/fail to validate your learning solution.
2. Don’t let the system limit you. 3. Design the instruments you need to collect
the metrics you want.4. Don’t be intimidated by the numbers.5. Continuously seek ways to improve.
Thank you! Merci!
Questions?
[email protected] @patrickdevey
http://survivingelearning.blogspot.ca
http://www.knowledgeone.ca