learning analytics: a new frontier of possibilities or just another education fad?

EASICONNECT @ CQUni

Damien ClarkTwitter: @damoclarkyhttp://damos.world

Colin BeerTwitter: @beerchttp://beerc.wordpress.com

Rolley TicknerTwitter: @rolleyhttp://rolleys.wordpress.com

Is a Team Effort

Colin Beer Rolley

Tickner Me!

Introductions

Is a Team Effort – Team Background

Teaching

Research

Programming Data

AnalysisSystem Administration

Graphic Design

DB Design

Complementary blend of skills:

Self Appointed:Ed Tech Team (not in org. chart)

Central L&T

The teachers view of a disengaged student

Looking around your classroom - what are the signs?

http://farm1.static.flickr.com/32/64368770_7bce91daf6_b.jpg

The teachers view of a disengaged fully online student

https://www.flickr.com/photos/wfryer/6727304641/

The Institution

CQU Context (Brief Overview)

Where is CQU?

Our footprint…

Where is CQU?

Where is CQU?

My team located here ->

Where are our HE students?

20+ Locations

Source: CQU Student Records

+ VET - 2014

Distance Ed

International

Source: CQU Student Records

Distance Ed

International

Where are our HE students?

DemographicsP

redominantly

Regional/R

emote

Enrolm

ents

DemographicsP

redominantly Low

/Mid S

ES

CQU – Central Queensland University

http://www.beingabusinesscelebrity.com/when-business-gets-overcomplicated/

CQU – Complicated Queensland University

Complementary Tools

Early Alerts Student Indicators (EASI)

Demonstration

Estimate of Success (EOS)

Emphasises communication between teachers and students rather than accurate prediction

https://xkcd.com/612/

Aggressively weighted towards risk of failure More false positives than false negatives

http://www.hangthebankers.com/5-top-economic-advisers-forecast-world-war/

http://sorenandersson.com/a-sustainable-nudge/

Nudge

Nudge (verb):1. to push or poke (someone) gently, esp with the elbow, to

get attention; jog2. To push slowly or lightly3. To give (someone) a gentle reminder or encouragementhttp://www.thefreedictionary.com/nudge

Nudging in Business

Amazon Marketing


Nudging – what’s the problem with this nudge?

{$student.firstname},

You have failed to connect regarding <coursename>. Recent phone calls and emails to you have gone unanswered. YOU ARE AT RISK OF FAILING THE COURSE!

{$staff.firstname} {$staff.lastname}


{$student.firstname},

You have failed to connect regarding <coursename>. Recent phone calls and emails to you have gone unanswered. YOU ARE AT RISK OF FAILING THE COURSE!

{$staff.firstname} {$staff.lastname}

http://discoversociety.org/2015/12/01/policy-briefing-the-nudge-business/

Not encouraging

Pestering?



Dear {$student.firstname},

This is the third time I have emailed you regarding your involvement with <coursename> . It would appear that you have not as yet accessed the course Moodle website despite my messages. If you have no intention of successful completion of this course, I would suggest that you drop it immediately. Census day is on Tuesday, and if you do not drop it before then, you will be charged tutition fees for the course, and ultimately be awarded the grade of FAIL if your current lack of activity continues.

If you choose to continue in the course, would you please contact your lecturer and arrange to design a study plan that will ensure that you quickly catch up with the course. If you are unwilling to do this, I strongly urge you to drop the course.

Regards, {$staff.firstname} {$staff.lastname} Course Coordinator: {$course.code} {$staff.email}


“This is the third time I have emailed you regarding your involvement with <coursename> . It would appear that you have not as yet accessed the course Moodle website despite my messages. If you have no intention of successful completion of this course, I would suggest that you drop it immediately… “

http://www.flickr.com/photos/cuppini/2757884114/

Teacher frustration is showing.


Hi, We are sending this little nudge as our records show that you may be needing some encouragement to engage more in our course ‘<coursename>'. Research has shown that students who engage in the moodle site at least 3 times a week are less likely to fail or have issues with falling behind. Please if you are having any problems contact us. We do understand that it is week 1 and there are many things to get our heads around including the technology and the set up of the Moodle site. So please take heed of our nudge and seek help if you need it or rearrange your schedule so you can actively take part in this interesting and worthwhile course as part of your learning journey. regards <teacher names>



Hi, We are sending this little nudge as our records show that you may be needing some encouragement to engage more in our course ‘<coursename>’…

http://www.flickr.com/photos/cuppini/2757884114/

Not personalised

Examples of Use (EASI)

http://www.doityourself.com/scat/handtools/

Following slides include more effective examples of real nudges

Examples of Use (EASI) – Case Management


Case management between teachers



Assessment Non-submission


Shared notes

Examples of Use (EASI) – Shared Notes

Examples of Use (EASI) – Shared Notes


Shared notes

The teachers view of a disengaged fully online student

Remember this?



“We can see at a glance the level of student activity on Moodle - this is one of the most valuable aspects of Easi Connect. We can see exactly what weeks students have been active, and when their activity might have dropped off.”


Examples of Use (EASI) – Student Responses

Why EASI? Motivation?

CQU Ouch!

!

Source: http://highereducationstatistics.education.gov.au/

2003 - 2009

Database access

LMS Support

Unintended consequences

We started noticing trends in the data

http://farm3.static.flickr.com/2672/3684420623_796346b171.jpg

http://www.flickr.com/photos/dragnfly78/235652252/

2006 2007 2008 2009 2010 2011 2012 2013 20140

500

1000

1500

2000

2500

3000

16 15 22 1667

271

773

1460

2430

Google Scholar publications using the term "learning analytics"

http://flickr.com/photos/boskizzi/3241710/

http://flickr.com/photos/boskizzi/3241710/

Oz/NZ Horizon Report

Year Time Frame Label

2010 4 to 5 years Visual data analysis

2012 1 year or less (#2)

Learning analytics

2013 1 year or less (#1)

Learning analytics

2014 2 to 3 years Learning analytics

http://www.flickr.com/photos/mikecogh/5959192031/

I’m not familiar with (m)any universities thathave taken a systems-level view of LA.

http://bit.ly/16uz8vU

Most of what I’ve encountered to date is specificresearch projects or small deployments of LA. I haveyet to see a systemic approach to analytics use/adoption

http://www.flickr.com/photos/johnhaydon/5042881685/

Basing decisions on data and evidenceseems stunningly obvious(Siemens and Long, 2011, p.

31)

CQUni’s Moodle site doesn’t have MAV

No MAV links

MAV installed in browser

MAV Staff Self-install Page

Firefox Add-ons / User Scripts

MAV installed in browser

Moodle Activity Viewer – Under the HoodNo changes toMoodle code ✔

Examples of Use (MAV)


Responses:

“I can see which links are being clicked on and which students are clicking on those links and I can see whether or not I am wasting my time producing a video that no-one ever watches or if there is something that has got the students’ attention and there is a lot of people clicking on it, I can see what that is.”



Responses:“The additional feature - the MORE icon - which allows us to access this information from other parts of Moodle, such as individual assignments, is also proving really valuable. ”


“We can access the Handbook from EASI CONNECT - this provides us with useful contact information e.g. phone numbers, as well as a view of their study history.”


Responses:

IT Management – Making Changes

Recommendation:“Implement Vanilla systems (remove or reduce customisations unless essential or demonstrated benefit over cost)” (CQUniversity Information Technology Department Restructure, 2013)

http://www.praqma.com/services/plainvanilla

https://twitter.com/vanillagames

Moodle Activity Viewer – Under the HoodNo changes toMoodle code ✔

Remember this?

IT Management – Making Changes

http://elevator.empowernetwork.com/blog/lifes-too-short-to-be-plain-vanilla

We think…

Practice Underpinned by Research

Breaking BAD to bridge the reality/rhetoric chasm. (ASCILITE2014)

Three paths for learning analytics and beyond: moving from rhetoric to reality (ASCILITE2014)

The IRAC framework: Locating the performance zone for learning analytics. (ASCILITE2013)

David Jones, Colin Beer, Damien Clark & Rolley Tickner


Indicators of engagement (ASCILITE2010)

Academic involvement in the LMS (ASCILITE2010)

Analytics and complexity: Learning and leading for the future (ASCILITE2012)


Choose the appropriate path: Do it With the AcademicEffective Learning Analytics uses a PIRAC

If you are SET in your ways, sometimes you should break BAD

Purpose Informatio

n

Representation Affordance

s

Change

Bricolage

Ability Change

Distribution

Decision-making Philosophy

http://elevator.empowernetwork.com/blog/lifes-too-short-to-be-plain-vanilla

Have been asked:

“Can I have EASI send out automatic emails to my students?”

Our response: “No.”

I’ll explain why with an analogy

Aviation Industry – 2 major aircraft manufacturers

https://commons.wikimedia.org/wiki/File:Airbus_A380_inbound_ILA_2006.jpg

Airbus

Aviation Industry – 2 major aircraft manufacturers

https://commons.wikimedia.org/wiki/File:Boeing_787_Dreamliner_N787BX.jpg

Boeing

Automation Philosophies

http://www.aerospaceamerica.org/Documents/Aerospace America PDFs 2015/July-August 2015/AA_Jul-Aug2015_Feature3_BoeingVsAirbus.pdf

Assistant Professor Clint R. Balog

“Airbus’ philosophy is automation-centric”

“Boeing’s is human-centric”

“This can be seen in Airbus’ publicly available philosophy statement, which says that ‘all aircraft have physical limits that they must not exceed’ and that such flight-envelope limits ‘are not to be exceeded during normal operations.’”

“Automation, according to Boeing, is ‘to assist, but not replace, the flight crew member responsible for safe operation of the airplane.’”

Automation Philosophies

http://www.aerospaceamerica.org/Documents/Aerospace America PDFs 2015/July-August 2015/AA_Jul-Aug2015_Feature3_BoeingVsAirbus.pdf

Assistant Professor Clint R. Balog

“Airbus’s philosophy appears to be that automation generally knows better than the pilot and should, under most circumstances, have the final decision authority. Boeing takes the view that the pilot is best able to evaluate individual, and possibly unique, operational circumstances and should make the ultimate decision.”

https://commons.wikimedia.org/wiki/File:Boeing_787_Dreamliner_N787BX.jpg

We aspire to Boeing’s philosophies of decision-making using learning analytics

Decision-making Philosophy

Our team Boeing takes the view that the teacher pilot is best able to evaluate individual, and possibly unique, teaching operational circumstances and should make the ultimate decision.

Lessons & Outcomes

Usage of the technology

Ethical Implications

https://www.flickr.com/photos/icma/3635981474

Increased Student Activity

Usage of EASI & MAV

https://en.wikipedia.org/wiki/Choice

Staff @ CQU choose to use EASI & MAV

Usage of EASI & MAV

https://en.wikipedia.org/wiki/Choice

EASI Usage

MAV Usage

281k nudges20k individual students2k course offerings

333k page views

Since Term 1, 2014

280 staff

2k course offerings

898 courses

Study Activity Resulting from Nudges

Control: Students who didn’t receive nudges: Mean +4 clicks at same weeks

Students who received nudges:

Mean +27 clicks 3 weeks after comparedto 3 weeks prior to nudge

p < 0.0001

Two tailed independent samples t test

df = 132164

t = 11.8

http://www.stanwicklakes.org.uk/northampton/honey-bees-buzzing-after-louise-mensch-visit/

Learning Analytics - Ethical considerations

Data ownership

SurveillanceLabeling

3 Findings from unpublished paper(under peer review)

http://quotesgram.com/funny-ethics-quotes/

Our advice to others

http://www.itsnotbadatall.com/funny_pictures/html/Funny_Advice.html

The predictive potential of learning analytics is overrated. Beware of the hype!


The social/technical system that encapsulates learning analytics is not complicated, it’s complex.

http://www.dumpaday.com/funny-pictures/random-funny-pictures-36-pics-5/attachment/funny-parent-advice-twilight-2/



The socio-technical system that encapsulates learning analytics is not complicated, it’s complex.

http://quotesgram.com/funny-advice-quotes/


Less of a focus on plan-based approaches to adoption and more on real-time situation awareness and sense making


The socio-technical system that encapsulates learning analytics is not complicated, it’s complex.

http://quotesgram.com/funny-advice-quotes/


Less of a focus on plan-based approaches to adoption and more on real-time situation awareness and sense making

Balance PIRAC

Incidentally…

CQU Yay!!

Causality?

Source: http://highereducationstatistics.education.gov.au/

Thanks!

https://coub.com/view/4pyf0

learning analytics: a new frontier of possibilities or just another education fad?

Education