learning analytics are about learning

Learning Analytics is about Learning

Dragan Gasevic@dgasevic

Growing demand for education!

Scalability is possible

Low effect size of class-sizeJohn Hattie

Delivery

Delivery

http://www.scientificamerican.com/article.cfm?id=massive-open-online-courses-transform-higher-education-and-science

Scientific American, March 13, 2013

MOTIVATION

Feedback loops between students and instructors

are missing!

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of educational research, 77(1), 81-112.

Learning and Collaborating

Educators

Learners

Registrations

Learning and Collaborating Networks

Videos/slides

Registrations

Mobile

Search

Educators

Learners

Networks

Learning and Collaborating Networks

Registrations

Mobile

Search

Networks

Educators

Learners

Videos/slides

DANGER

Predict-o-mania

The same predictive models for everything and everyone

Student diversity

http://www.census.gov/prod/2013pubs/acsbr11-14.pdf

Population Diversity

Female

s

Internati

onal stu

dents

Other lan

guag

e at h

ome

Living i

n non-urban

Part time s

tudent

Previously

enro

lled to

a co

urse

Early

acces

s

Did not acce

ss

Late a

ccess

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

ACCT 1 (n = 746)BIOL 1 (n = 220)BIOL 2 (n = 657)COMM 1 (n = 499)COMP 1 (n = 242)ECON 1 (n = 661)GRAP 1 (n = 192)MARK 1 (n = 723)MATH 1 (n = 194)

LMS Functionality Diversity

ACCT 1 BIOL 1 BIOL 2 COMM 1 COMP 1 ECON 1 GRAP 1 MARK 1 MATH 1

Light Box Gallery XForum X X X X X X X X XCourse X X X X X X X X XResource X X X X X X X X XTurn-it-in X X X X X XAssignment X X X X X X XBook X X XQuiz X X X XFeedback XMap XVirtual Classroom XLesson XGlossary XChat X

Predictive Power Diversity

All courses together

ACCT 1 BIOL 1 BIOL 2 COMM 1 COMP 1 ECON 1 * GRAP 1 MARK 1 MATH 10.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

Model 1MoodleModel 1 + Moodle

Model 1 – demographic and socio-economic variables* - not statistically significant

Retention is not the only challenge

It is important, of course!

But, where is learning?

How do we enhance learning if the focus is on outcomes only?

DIRECTION

Learning Analytics – What?

Measurement, collection, analysis, and reporting of data about

learners and their contexts

Learning Analytics – Why?

Understanding and optimising learning and the environments

in which learning occurs

Human agency is central to learning

Bandura, A. (1989). Human agency in social cognitive theory. American psychologist, 44(9), 1175-1184.

Modern Educational Psychology

Winne and Hadwin's model

of self-regulated learning

Knowledge society and knowledge economy

Why does it matter?!

ChallengeMetacognitive skills

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-Regulated Learning: Beliefs, Techniques, and Illusions. Annual Review of Psychology, 64, 417-444. doi:10.1146/annurev-psych-113011-143823

Why does it matter?!

ChallengeInformation seeking skills

Judd, T., & Kennedy, G. (2011). Expediency-based practice? Medical students’ reliance on Google and Wikipedia for biomedical inquiries. British Journal of Educational Technology, 42 (2), 351-360. doi:10.1111/j.1467-8535.2009.01019.x

Why does it matter?!

ChallengeSensemaking paradox

Butcher, K. R., & Sumner, R. (2011). Self-Directed Learning and the Sensemaking Paradox. Human–Computer Interaction, 26(1-2), 123-159. doi:10.1080/07370024.2011.556552

Why does it matter?!

ChallengeAsking questions and critical thinking

Graesser, A. C., & Olde, B. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524–536..

Process and context focus for learning analytics needed

to understand learning

OPPORTUNITIES

Learning Analytics

Effects of learning context

External conditions (e.g., instructional design)

Cognitive presence

the extent to which the participants in any particular configuration of a CoI are able to construct meaning via sustained communication

Garrison, D. R., Anderson, T., & Archer, W. (2001). Critical Thinking and Computer Conferencing: A Model and Tool to Assess Cognitive Presence. American Journal of Distance Education ,15(1), 7-23.

Effect size of the moderator role on critical thinking

Cohen’s d = 0.66

Effect size of an intervention on critical thinking in online discussions

d = 0.95 (non-moderators) and

d = 0.61 (moderators)

Cognitive Presence in Online Discussions – Association w/ Grades

Cognitive presence TMA1 TMA2 TMA3 TMA4 Final

Control group

Triggering event -.226 .005 -.046 -.050 -.010Exploration -.001 .141 .009 -.037 .048Integration .128 .060 .034 .043 .113Resolution .201 .027 -.023 -.054 .074Other -.028 .078 .113 .106 .154

** p < 0.01; * p < 0.05

Cognitive Presence in Online Discussions – Association w/ Grades

Cognitive presence TMA1 TMA2 TMA3 TMA4 Final

Control group

Triggering event -.226 .005 -.046 -.050 -.010Exploration -.001 .141 .009 -.037 .048Integration .128 .060 .034 .043 .113Resolution .201 .027 -.023 -.054 .074Other -.028 .078 .113 .106 .154

Intervention group

Triggering event .149 -.077 -.070 .000 .016Exploration .216 .197 .163 .223 .243

Integration .156 .396** .417** .338* .454**

Resolution -.041 .060 .154 .083 .129Other .219 .046 .050 .075 .088

** p < 0.01; * p < 0.05

Integration posts: effect on final grades

p < .001, Q1 vs. Q2; Q1 vs. Q3, Q1 vs. Q4

Q1 Q2 Q3 Q40

102030405060708090

100

Learning Analytics

Are students only driven by assessments?

Effects of external conditions

Self-reflections in video annotations

Course 1 (non-graded)

Course 2 (graded)

Course 3(graded)

Course 4 (non-graded)

Annotation total Annotation postion Q1

Annotation postion Q2

Annotation postion Q3

Annotation postion Q4

Annotation general0.00

20.00

40.00

60.00

80.00

100.00

120.00

Course 1 (non-graded)Course 2a (graded)Course 2b (graded)Course 3 (graded)Course 4 (non-graded)

Self-reflections in video annotations

Self-reflections in video annotations

Cognitive

proces

ses

Percep

tual pro

cesses

Positive

emotions

Negati

ve em

otions0

200

400

600

800

1000

1200

1400

1600

Course 1 (non-graded)Course 2a (graded)Course 2b (graded)Course 3 (graded)Course 4 (non-graded)

Tausczik, Y. R., & Pennebaker, J. W. (2010). The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24-54.

Learning Analytics

Effects of students’ own decisions

Beyond external conditions

Learner profiles – use of LMS

Effect size .75 on critical thinking &academic success

34

Learner profiles – use of LMS

Effect size .75 on critical thinking and academic success

Cluster 1 Cluster 2 Cluster 3 Cluster 40

2

4

6

8

10

12

TriggeringExplorationIntegrationResolutionOther

CHALLENGES

Learning Analytics

What to measure?

We don’t need page access counts only!

Wilson, T.D. (1999). Models in information behaviour research. Journal of Documentation, 55(3), 249 - 270, doi:10.1108/EUM0000000007145

Instrumentation

About specific contexts and constructs

Instrumentation

Capturing interventionsPrevious learning and (memory of) experience

Social networks (e.g., communication, cross-class)Interaction types (e.g., transactional distances)

Motivation in Information Interaction

Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004

Motivation in Information Interaction

Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. doi:10.1016/j.learninstruc.2012.03.004

Achievement goal orientation (2x2)

Siadaty, M. (2013). Semantic Web-Enabled Interventions to Support Workplace Learning, PhD Thesis, Simon Fraser University, Surrey, BC, Canada.

Technology and process of self-regulated learning

Scaling up qualitative analysis

Temporal processesbeyond coding and counting

Longitudinal studies

Generating reports and nice visualization is

not enough

Building data-driven culture in institutions

Manyika, J., et al., Big Data: The Next Frontier for Innovation, Competition, and Productivity, 2011, McKinsey Global Institute, http://goo.gl/Lue3qs

Privacy and ethics

Data sharing and mobility

Thank you!