la the coming third wave

7/27/2019 LA the Coming Third Wave

1/4

educause.edu/eli 2011 EDUCAUSE. CC by-nc-nd.

Learning Analytics:The Coming Third Wave

Learning analytics (LA) is the third wave o large-scale developments in instruc-

tional technology that began with the advent o the learning management system.

As a compelling diagnostic tool, LA will see rapid adoption over the next several

years, as third-party applications begin to make it more afordable and practical.

The papers presented at the First International Conerence on Learning Analytics

and Knowledge (LAK11) conerence provide an overview o current research and

development, helping to show the potential o LA as a way to provide timely and

meaningul eedback or learners, instructors, and administrators.

By Malcolm Brown, Director, EDUCAUSE Learning Initiative April 20BRIEF

What Is Learning Analytics?At its core, learning analytics (LA) is the

collection and analysis o usage data associatedwith student learning. The purpose o LA is toobserve and understand learning behaviors inorder to enable appropriate interventions. Thereports that an LA application generates canbe very helpul or instructors (about studentactivities and progress), or students (eedbackon their progress), and or administrators (e.g.,aggregations o course and degree completiondata). The main elements o LA include:

Data collection: This entails the use oprograms, scripts, and other methods to gatherdata. This can be data rom a single source or

a variety o sources; it can entail large to verylarge amounts o data, and the data can bestructured (e.g., server logs) or unstructured(e.g., discussion orum postings). The specicdesign o the collection activity is inormed bythe goals o the LA project.

Analysis: Unstructured data is usually givensome kind o structure prior to analysis.The data is subjected to an appropriatecombination o qualitative and quantita-tive analysis. The results o the analysis

are reported using a combination o visu-alizations, tables, charts, and other kinds o

inormation display.Student learning: This core goal distin-guishes learning analytics rom other kindso analytics. LA seeks to tell us about studentlearning: what learners are doing, where theyare spending their time, what content theyare accessing, the nature o their discourse,how well they are progressing, and so onatthe individual or cohort level or both.

Audience: The inormation that LA returnscan be used to (1) inorm instructors, (2)inorm students, or (3) inorm administra-tors. Common to all three is that the reports

enable appropriate interventions. Typically(1) and (2) enable course-level interventions,while (3) inorms interventions at the depart-mental, divisional, and institutional levels. Thekinds o data and analysis employed dependon the intended audience.

Interventions: The reason or doing LA is toenable appropriate interventions at the indi-vidual, course, department, or institutionallevel. LA can do more than just identiystudents at risk. By analyzing the digital

Learning analytics is the use of intelligent data, learner-produced data, and analysis models to

discover information and social connections for predicting and advising peoples learning.

Wikipedia

Learning analytics is the use of data and models to predict student progress and performance,

and the ability to act on that information.

Next Generation Learning Challenges


2/4

educause.edu/eli

breadcrumbs that students generatewhen participating in all aspects o a course,it is possible to observe student progress atspecic stages and at specic activities in acourse. The potential o LA is to be able toindicate what is working and what is not ata much ner level o granularity than everbeore, even while a course is in progress.

Adoption HorizonsThe 2011 Horizon Report includes LA in the

our- to ve-year window, meaning it predictsLA will be in mainstream use within that time.This estimate, however, might too conserva-tive. A combination o actors makes a two- tothree-year horizon quite probable.

The rst actor is that LA is very compelling. Theway that LA reveals usage, trends, and patternsin learning is by itsel compelling enough to spuradoption. In addition, LA helps identiy studentswho are struggling, enabling interventions thatcan assist those students to successul coursecompletion. This is o particular importance, giventhe enormous pressures on higher education oraccountability and higher completion rates. LA isperhaps the key resource that will enable schoolsto begin to respond to those calls.

Traditional student course evaluationscapture student opinion and reections butcannot capture what the students actually did.Moreover, because those evaluations are doneonce the course is complete, they cannot enableinterventions while the course is in progress.Nor can they provide any inormation on thingssuch as how students utilized course content orthe extent o their participation in course activ-ities. LA can do that and more.

The second actor is the emergence o LAapplications. Until recently, i a school wantedto conduct LA, it had to build a system or LArom scratch. Two examples o this includePurdue Universitys Signals application and thedevelopment work done at the University oMaryland, Baltimore County, to coax LA romits Blackboard implementation. This landscapeis now changing rapidly. SunGard, Blackboard,and Desire2Learn are moving in this direction,and over the next year, more vendors will enterthis space. This development is important in thatit enables schools to begin to do LA withoutcostly and time-consuming development. In

short, we are now entering a time when institu-tions can buy instead o build and can have anLA application up and running in a ew monthsinstead o years. This actor, even more than theothers, might accelerate the adoption o LA atinstitutions worldwide.

A third actor is the increasing emphasis onthe use o metrics and rubrics or higher educa-tion in order to demonstrate student learning

and degree progress. This will encourage theinvestigation o LA as a way o meeting thosedemands. In addition, oundations are nowplacing a heavy emphasis on the use o LA inthe projects they support. A example is theNext Generation Learning Challenges program,which has issued a challenge to develop amodel that identies, improves, and scalesexisting solutions o learner analytics.1

In suggesting a more rapid adoption o LA,we are not claiming that within three years themajority o schools will be doing a sophisti-cated, highly customized orm o LA. That will

take more time. But given the environmentalactors mentioned above, it is clear that themajority o schools will be doing LA in someorm, even i it is simply installing an of-the-shel application.

LA as the Third WaveLA is the third wave o developments in instruc-

tional technology that began with the advent othe learning management system (LMS). In thelate 1990s it was clear that handcrating coursewebsites would not scale, and so the LMS was born.By 2005, the question or schools was no longerwhether they were running a LMS but instead which

one. The vision underlying the initial LMS designwas based on a traditional teaching model, beingcourse- and instructor-centric, and the transmis-sion paradigm o Web 1.0. Nevertheless, the LMShas seen large adoption rates, as its utility is clear.The adoption o the LMS 1.0 also enabled a keyaccomplishment: integration into the enterprise.Now learning, or the rst time, was connected tothe enterprise inrastructure.

The second wave is the addition o a 2.0layer to the LMS, adding social networking andcloud-based applications into the mix. At timesthis has been the proverbial square peg in around hole, given the 1.0 vision that inormedmuch o the LMS initial design. Nevertheless,because the 2.0 unctionality has been socompelling, higher education has moved, andrapidly, in this direction. Indeed, this has beenso important that aculty and others have attimes gone outside the LMS in search o 2.0unctionality. Today we are rapidly approachingthe point at which the question is not whetherthe 2.0 tools are being used but rather whichones and in what manner.

LA will provide the capability of

collecting and analyzing data from a

variety of sources to provide information

on what works and what does not


3/4

educause.edu/eli 3

LA will be the third wave. Building on the LMSand its enterprise integration, LA will provide thecapability o collecting and analyzing data rom avariety o sources to provide inormation on whatworks and what does not with respect to teachingand learning. LAs strength will come rom theenterprise integration that the LMS established,and it will become more compelling as dataretrieval and analysis become more powerul and

sophisticated. The new diversity o 2.0 tools willofer a larger set o digital breadcrumbs andhence more data. Colleges and universities mighteven be able to share anonymous, aggregateddata to produce composite views o studentlearning or an entire class o institutions.

The LMS addressed the administrative over-head o running a course, and the 2.0 wave isproducing learner-centered content and tools.LA, as the third wave, is the metacognitivecomponent, allowing individuals and institutionsto understand learning and make inormed deci-sions about resource allocations and required

interventions to promote learner success.

Privacy: The Ethics of Doing LAAt the rst International Conerence on Learning

Analytics and Knowledge (LAK11), held in Banf,Canada, February 28March 1, 2011, presentersand attendees all agreed that LA raises deep andcomplex privacy issues. LA could be construed aseavesdropping. The ethical concerns are obvious.The worry is that the LA mechanisms will prowlabout, scrutinizing what learners are doing, andso constitute an invasion o privacy.

One obvious way to address privacy concerns isto allow students to opt-out. But there are prob-

lems with this approach. The accuracy o LAsrecommendations or intervention improves as(1) the number o observable subjects increasesand (2) the amount o data available or analysisis maximized. Analysis based on only a subset othe students in a course or based on ragmentarydata will be incomplete. That could mean that theresulting recommendations or intervention mightbe less accurate. As a result, an opt-out alternativeor students could compromise the LA efort. Mucho the uture work around LA will involve ndingthe balance between learner privacy and the valueo data collection or improving learning.

One presenter at the conerence, Eric Duval,touched on this issue. He suggested the ideathat the person who generates the bread-crumb data owns that data and that the datashould be reundable or returnable to theowner. He mentioned a project, Attention Trusthttp://www.attentiontrust.org/, that or a time wasdeveloping this idea o reundability. Yet this ideawould seem to return us to the problem o accu-racy, or data reunds would create holes in the data,which could in turn lead to incomplete analyses.

Clearly a good deal o work needs to be donehere. As schools proceed, it will be necessaryto engage a variety o campus stakeholders toensure a balance between the demands o indi-vidual privacy and the goals o a LA project.2

Selected Presentations from theLAK11 Conference

The presentations at LAK11 presented acompelling picture o the potential o LA. Wesummarize just a ew o the papers to try tocapture a sense o this.

A presentation by Ravi Vatrapu (CopenhagenBusiness School) introduced the idea oteaching analytics and reported on aproject designed to provide the instructorwith real-time analytics inormation whilein the classroom. The goal is to providetimely, meaningul, and actionable orma-tive assessments to ongoing learningactivities in situ. Their system consistso three computer experts; these three

experts work by analyzing, interpreting andacting upon real-time data being generatedby students learning activities by using arange o visual analytics tools.

An interesting presentation by Chris Brooks(University o Saskatchewan) reported onusing a lecture-capture system to do LA.Utilizing a presentation-capture systemdesigned in-house, they have designed thetool to report data on the use o lecture-capture content by the students. The usageinormation sent back to the server isdetailed enough to largely reconstruct the

students session with the content. The goalo the project is to understand how studentsuse this system to augment learning andto move toward validating the construc-tivist educational theory. The inormationharvested by their system has been usedto help analyze the behavior o hundredso students over an academic term, quan-tiying both the learning approaches ostudents and their perceptions on learningwith lecture capture.

Simon Buckingham Shum (Open University)reported on research using LA to iden-

tiy exploratory dialogue (ollowing adiscourse typology ramework by Mercer andcolleagues). In a learning context, exploratorydialogue represents a high-order level odiscourse because it entails reasoning, chal-lenge, and evaluation, all three o which aremarkers or indicators o deeper learning. Theother two kinds o discourse, disputationaland cumulative, involve ewer o those threeelements. The challenge is to design an LAtool that can identiy exploratory dialogue,
http://www.attentiontrust.org/http://www.attentiontrust.org/


4/4

educause.edu/eli

The EDUCAUSE Learning Initiative (ELI) is where teaching and learning professionals come to learn, lead, collaborate,and share in the context of an international forum. Members benefit from the expansive emerging technology researchand development that takes place collaboratively across institutions. To learn more about ELI, visit educause.edu/eli.

flag it for the instructor and student,and locate dialogue exchangeswhere deeper learning appears tobe taking place. This is research inprogress; they are seeking to confirmwhether their initial set of exploratorydialogue markers really do work asidentifiers.

Mike Sharkey (University of Phoenix)

presentation conveyed the scale atwhich LA can be conducted. TheUniversity of Phoenix data repos-itory draws from over 30 sources,is modeled with 430 tables, andis 1.5TB in size (growing by 100GBper month). The goal of theirproject is to predict a studentspersistence in their program usingavailable data indicators such asschedule, grades, content usage,and demographics.

Katja Niemann from Bonn, Germany,presented a way to use LA to assistin content discovery in learning.Learners must make decisions aboutwhat to do next and what learningobjects (LOs) to use next. The chal-lenge is how then to make suitablerecommendations for LOs, especiallyfor resources the learners may notknow aboutin other words, how tomake semantic linkages apart fromusage metadata. Their work is basedon the linguistic theory of co-occur-

rence, which contends that the closer wordsappear in discourse, the more semanticallyrelevant they are. They have performed initialanalyses of 4,000 objects and have statisti-cally relevant results.

Aneesha Bakharia (University of Queensland)and Shane Dawson (University of BritishColumbia) presented on SNAPP (SocialNetworks Adapting Pedagogical Practice).This tool, by making available data visu-alisations and social network metrics inreal-time, allows emergent interactionpatterns to be analyzed and interventionsto be undertaken as required. Hence,

SNAPP essentially serves as an interac-tion diagnostic tool. The presentationdescribed the features that will be addedin version 2: The ability to view the evolu-tion of participant interaction over time andannotate key events that occur along thistimeline. This feature is useful...[for] evalu-ating the impact of intervention strategieson student engagement and connectivity.

Well over two dozen papers were presented at

LAK11. Overall, they represent the rich diversity ofresearch and project directions currently underway in the field of LA. One analogy used at theconference is that LA enables instructors, learners,and administrators to take the temperature ofthe learning that is taking place at our institutions.This ability to look inside learners activitiesprovides rich diagnostic information, much theway that an X-ray or an MRI scan provides infor-mation for medical diagnosis. This is what makesthe case for LA so compelling.

Endnotes1. See http://nextgenlearning.org/the-grants/wave-1-challenges/

learning-analytics/articles/learning-analytics.

2. FERPA does permit disclosure of education records without

interests. The full wording in the statute reads, The disclosure

or institution whom the agency or institution has determined tohave legitimate educational interests. See 34 CFR 99.31, http://edocket.access.gpo.gov/cfr_2004/julqtr/pdf/34cfr99.31.pdf .

To Explore FurtherWebsite for the First International Conference on Learning Analytics andKnowledge 2011, https://tekri.athabascau.ca/analytics/.

George Siemens, Learning Analytics: A Foundation forInformed Change in Higher Education, ELI Webinar,

January 2011, http://www.educause.edu/Resources/Learning

AnalyticsAFoundationfo/221468.

David Wiley, Openness, Learning Analytics, and Continuous Quality

Improvement, podcast, http://www.educause.edu/blog/gbayne/ELI2011PodcastOpennessLearning/225360 .

2011 Horizon Report , http://www.educause.edu/Resources/2011HorizonReport/223122 .

International Working Group on Educational Data Mining, http://www

.educationaldatamining.org/.

John Fritz, Video Demo of UMBCs Check My Activity Tool forStudents, EDUCAUSE Quarterly 33, no. 4, 2010, http://www.educause

.edu/EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/

VideoDemoofUMBCsCheckMyActivit/219113.

Kimberly E. Arnold, Signals: Applying Academic Analytics,

EDUCAUSE Quarterly 33, no. 1, 2010, http://www.educause.edu/

EDUCAUSE+Quarterly/EDUCAUSEQuarterlyMagazineVolum/

SignalsApplyingAcademicAnalyti/199385.

Syllabus to the open online course on Learning and Knowledge Analytics,

http://learninganalytics.net/syllabus.html.

Donald Norris, Linda Baer, Joan Leonard, Louis Pugliese, and Paul

Lefrere, Action Analytics: Measuring and Improving Performance That

Matters in Higher Education, EDUCAUSE Review 43, no. 1 (January/

February 2008): 4267, http://www.educause.edu/EDUCAUSE+Review/

EDUCAUSEReviewMagazineVolume43/ActionAnalyticsMeasuringandI

mp/162422.

John P. Campbell and Diana G. Oblinger, Academic Analytics,

EDUCAUSE White Paper, October 2007, http://net.educause.edu/ir/

library/pdf/PUB6101.pdf.

la the coming third wave

Documents