LearningAnalytics–
what,how,why
MarenScheffelIoanaJivet
OpenUniversiteitNederland
Outline
• Whatislearninganalytics?• Whereareweat?• Howaredashboardsused?• Howcanweevaluatelearninganalyticstools?• Howaboutaninstitutionallearninganalyticspolicy?• Wheredowegofromhere?
Insight is the buzzword in the data revolution. Peoplewho have insight into their behaviour will be able toworkmoreefficientlyfromnowon.Thisequallyappliesto organisations, higher education institutions andnations. The digital revolution has brought about atremendous increase in the volume of data and it iscomparativelyeasytoaccess.Itwouldseemtobealostopportunitytoignoresuchdata.
MarjoleinvanTrigt(2016)
https://cdn.static-econo
mist.com
/site
s/de
fault/files/im
ages/prin
t-ed
ition
/201
5022
8_LD
P001
_0.jp
g
TheUnitedNationsarecallingforactions:
“dataarethelifebloodofdecision-making”
“the new world of data,a world in which dataare bigger, faster andmore detailed than everbefore”
But what about theeducationaldomain?
Dataanalysisiscommon inbusinessandcommerce.
http://arcbridge.com
/wp-conten
t/up
loads/20
15/07/bu
siness-analytics.pn
g
Collecting,storingandanalysingstudentrecords
hasalwaysbeendone
inonewayoranother.
http://balfour.rb
e.sk.ca/sites/balfour/files/Da
vid-Oscar-Brown-Stud
ent-Re
cord-side-2.jpg
http://abo
vethelaw
.com
/wp-conten
t/up
loads/20
13/03/file-cabine
t1.jp
g
learningacquisitionofknowledgeorskills
throughstudy,experience,orbeingtaught
analyticssystematiccomputationalanalysisofdataor
statisticsor
informationresultingfromthesystematicanalysisofdataorstatistics
Learning analytics is the measurement, collection,analysis and reporting of data about learners andtheir contexts, for purposes of understanding andoptimising learningand theenvironments inwhichitoccurs.
Siemens(2011)
Outline
• Whatislearninganalytics?• Whereareweat?• Howaredashboardsused?• Howcanweevaluatelearninganalyticstools?• Howaboutaninstitutionallearninganalyticspolicy?• Wheredowegofromhere?
LevelsofLearningAnalytics
LearningAnalyticsFramework
Greller,W.andDrachsler,H.(2012).TranslatingLearningintoNumbers:AGenericFrameworkforLearningAnalytics.EducationalTechnology&
Society,15(3):42–57.
LAshouldbeforteachersandlearners
LASophisticationModel
Siemens,G.,Dawson,S.,&Lynch,G.(2014).ImprovingtheQualityandProductivityoftheHigherEducationSector–PolicyandStrategyforSystems-LevelDeploymentofLearningAnalytics.Canberra,Australia:OfficeofLearningandTeaching,AustralianGovernment.
Drachsler,H.andGreller,W.(2016).PrivacyandAnalytics–it’saDELICATEissue.AChecklisttoestablishtrustedLearningAnalytics.InProceedingsoftheSixthInternationalConferenceonLearningAnalyticsandKnowledge,LAK’16,pages89–98,NewYork,NY,
USA.ACM.DOI:10.1145/2883851.2883893.
http://www.laceproject.eu/ethics-privacy-learning-analytics/
http://www.eugdpr.org/
Outline
• Whatislearninganalytics?• Whereareweat?• Howaredashboardsused?• Howcanweevaluatelearninganalyticstools?• Howaboutaninstitutionallearninganalyticspolicy?• Wheredowegofromhere?
LearningAnalyticsDashboards
http://w
ww.learno
nsolutions.com
/wp-conten
t/up
loads/20
16/10/learning-analytics-learno
n-85
0x40
0.jpg
Structure
Source:https://www.flickr.com/photos/bruneluniversity/12926707974/
Learninganalyticsdashboardsare…
…“singledisplaysthataggregatedifferentindicatorsaboutlearner(s),learningprocess(es)and/orlearningcontext(s)intooneormultiplevisualisations”Targetusers:teachers(75%),learners(51%),administratorsandresearchersLearningscenarios:formal(91%),non-formal,informalPurpose:Supportinformeddecisionmaking
Schwendimann,B.A.,Rodriguez-Triana,M.J.,Vozniuk,A.,Prieto,L.P.,Boroujeni,M.S.,Holzer,A.,...&Dillenbourg,P.(2017).Perceivinglearningataglance:Asystematicliteraturereviewof
learningdashboardresearch.IEEETransactionsonLearningTechnologies,10(1),30-41.
Thelearninganalyticsprocess
Verbert,K.,Duval,E.,Klerkx,J.,Govaerts,S.,&Santos,J.L.(2013).Learninganalyticsdashboardapplications.AmericanBehavioralScientist,57(10),1500-1509.
Visualisation
Indicators
Educationalfoundation
• Barchart• Linegraph• Piechart• Scatterplot• Socialgraphs• Text
• Learnercharacteristics• Contextoflearning• Content• Learneractions• Learningoutcomes• Socialinteractions
• Feedbacktheory• Self-regulatedlearning• Collaborativelearning• 21stcenturyskills• Bloom’staxonomy• Socialcomparison
Learninganalyticsdashboards
Learninganalyticsdashboards
Jivet,I.,Scheffel,M.,Drachsler,H.,andSpecht,M.(2017).Awarenessisnotenough.Pitfallsoflearninganalyticsdashboardsintheeducationalpractice.InProceedingsofthe12thEuropean
ConferenceonTechnologyEnhancedLearning(EC-TEL2017),LNCS,pages82-96.Berlin.Springer.
Learninganalyticsdashboardsforteachers
Monroy,C.,Rangel,V.S.,&Whitaker,R.(2013).STEMScopes:contextualizinglearninganalyticsinaK-12sciencecurriculum.InLAK’13(pp.210-219).ACM.
Learninganalyticsdashboardsforlearners
Corrin,L.,&deBarba,P.(2014).Exploringstudents’interpretationoffeedbackdeliveredthroughlearninganalyticsdashboards.ASCILITE2014.pp.629-633.
Learninganalyticsdashboardsforlearners
Beheshitha,S.S.,Hatala,M.,Gašević,D.,&Joksimović,S.(2016).Theroleofachievementgoalorientationswhenstudyingeffectoflearninganalyticsvisualizations.InProceedingsLAK’16(pp.54-63).
ACM.
Learninganalyticsdashboardsforlearners
Aljohani,N.R.,&Davis,H.C.(2013).Learninganalyticsandformativeassessmenttoprovideimmediatedetailedfeedbackusingastudentcenteredmobiledashboard.NGMAST2013,pp.262-267.IEEE.
Learninganalyticsdashboardsforteachers
Learninganalyticsdashboardsforlearners
TheLearningTracker
Designgoals1. Traceable2. Scrutable3. ActionableIterativedesign1. Interactive2. Planningahead3. Framing
TheActivityWidget
TheLearningPathWidget
Recommendationsfordesigningdashboards
1. Define the goal of the dashboard and desired impact on the learners.
2. Research educational concepts as theoretical grounding. 3. Describe data that supports the goal. 4. Experiment with different visualisations. 5. Customise the dashboard to provide the same support to all
users. 6. Integrate the dashboard into the learning activities. 7. Focus dashboard evaluation on goal fulfillment, impact on
learners' affect and motivation, and on the usability of the tool. 8. Tool's usability should also assess how learners interpret
information. 9. Use multiple data sources to validate dashboard effects.
Jivet,I.,Scheffel,M.,Drachsler,H.&Specht,M.(2017).Awarenessisnotenough.Pitfallsoflearninganalyticsdashboardsintheeducationalpractice.EC-TEL2017.
Outline
• Whatislearninganalytics?• Whereareweat?• Howaredashboardsused?• Howcanweevaluatelearninganalyticstools?• Howaboutaninstitutionallearninganalyticspolicy?• Wheredowegofromhere?
Whereistheevidence?
http://www.laceproject.eu/evidence-hub/
Learninganalyticstools
• Demandsofthestakeholders?• Empiricalevidence?• Desiredeffect?• Indicatorsfortoolevaluation?• Comparability?• Standardisedevaluation?• Impact?• Benefit?
TheEvaluationFrameworkforLA
Finally, if there isn’t a relevant questionnaire foryoursituation,youcandeviseone.
JohnBrooke(2013)
ObjectivesforEFLAcreation
• Identifyqualityindicatorsforlearninganalytics
• Createanapplicableevaluationinstrumentbasedontheseindicators
• Validatetheevaluationinstrument
EFLA1
ObjectivesAwareness Reflection Motivation Behavioural
Change
Learning SupportPerceived Usefulness Recommendation Activity
ClassificationDetection of
Students at Risk
Learning Measures and OutputComparability Effectiveness Efficiency Helpfulness
Data AspectsTransparency Data Standards Data Ownership Privacy
Organisational AspectsAvailability Implementation Training of Educa-
tional StakeholdersOrganisational
Change
EFLA2
DATA-I know what data is being collected-I have access to my data-I understand the presented results
L E A R N E R S
-I know what data is being collected-I have access to my students’ data-I understand the presented results
AWARENESS-I am aware of my current learning status-I comprehend my current learning status-I can project my future learning status
-I am aware of my students’ current learning status-I comprehend my students’ current learning status-I can project my students’ future learning status
REFLECTION-I reflect on my learning activities-I reflect on alternative learning activities-I know when to change my behaviour
-I reflect on my teaching activities-I reflect on alternative teaching activities-I know when to change my behaviour
T E A C H E R S
IMPACT-I can detect whether I am falling behind-I study more efficiently-I study more effectively
-I can detect whether my students are falling behind-My students learn more efficiently-My students learn more effectively
EFLA3
DATA-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected-For this LA tool it is clear who has access to the data
L E A R N E R S
-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected-For this LA tool it is clear who has access to the data
AWARENESS-This LA tool makes me aware of my current learning situation-This LA tool makes me forecast a possible future learning situation given my (un)changed behaviour
-This LA tool makes me aware of my students’ current learning situation-This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour
REFLECTION-This LA tool stimulates me to reflect on my past learning behaviour-This LA tool stimulates me to adapt my learning behaviour if necessary
-This LA tool stimulates me to reflect on my past teaching behaviour-This LA tool stimulates me to adapt my teaching behaviour if necessary
T E A C H E R S
IMPACT-This LA tool increases my motivation to study-This LA tool stimulates me to study more efficiently-This LA tool stimulates me to study more effectively
-This LA tool increases my motivation to teach-This LA tool stimulates me to teach more efficiently-This LA tool stimulates me to teach more effectively
EFLA4
DATA-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected
L E A R N E R S
-For this LA tool it is clear what data is being collected-For this LA tool it is clear why the data is being collected
AWARENESS & REFLECTION-This LA tool makes me aware of my current learning situation-This LA tool makes me forecast my possible future learning situation given my (un)changed behaviour-This LA tool stimulates me to reflect on my past learning behaviour-This LA tool stimulates me to adapt my learning behaviour if necessary
-This LA tool makes me aware of my students’ current learning situation-This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour-This LA tool stimulates me to reflect on my past teaching behaviour-This LA tool stimulates me to adapt my teaching behaviour if necessary
T E A C H E R S
IMPACT-This LA tool stimulates me to study more efficiently-This LA tool stimulates me to study more effectively
-This LA tool stimulates me to teach more efficiently-This LA tool stimulates me to teach more effectively
EFLA
• isvalid&reliable,
• addressesthelackofevaluationinstruments,
• complementsevaluationonaninstitutionallevel,
• addressestheneedforevidencethathasbeencalledfor,
• iseasytousewithtemplates&interactivespreadsheet
DATAFor this LA tool it is clear what data is being collected
The Evaluation Framework for Learning AnalyticsE F L A
AWARENESS & REFLECTION
IMPACT
forL E A R N E R S
For this LA tool it is clear why the data is being collectedstrongly
disagreestrongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool makes me awareof my current learning situation
This LA tool stimulates me to reflecton my past learning behaviour
This LA tool makes me forecast my possible future learning situation given my (un)changed behaviour
strongly disagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
strongly disagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool stimulates me to adaptmy learning behaviour if necessary
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool stimulates me to study more efficientlystrongly
disagreestrongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool stimulates me to study more effectivelystrongly
disagreestrongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
DATAFor this LA tool it is clear what data is being collected
AWARENESS & REFLECTION
IMPACT
forT E A C H E R S
For this LA tool it is clear why the data is being collectedstrongly
disagreestrongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool makes me awareof my students’ current learning situation
This LA tool stimulates me to reflecton my past teaching behaviour
This LA tool makes me forecast my students’ possible future learning situation given their (un)changed behaviour
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool stimulates me to adaptmy teaching behaviour if necessary
stronglydisagree
strongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool stimulates me to teach more efficientlystrongly
disagreestrongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
This LA tool stimulates me to teach more effectivelystrongly
disagreestrongly agree
1O
2O
3O
4O
5O
6O
7O
8O
9O
10O
EFLA scoring per stakeholder groupStep 1: Calculate the average value for each item based on the answers given for that item.
Step 2: Calculate the average value for each dimension based on the average of its items.
Step 3: Calculate the dimensional scores by rounding the results of ((x-1)/9)*100 where x is the average value of a dimension.
Step 4: Calculate the overall EFLA score by taking the average of the three dimensional scores.
http://www.laceproject.eu/evaluation-framework-for-la/
Outline
• Whatislearninganalytics?• Whereareweat?• Howaredashboardsused?• Howcanweevaluatelearninganalyticstools?• HowaboutaninstitutionalLApolicy?• Wheredowegofromhere?
PolicyFramework
http://sheilaproject.eu/
PolicyFramework
ACTION CHALLENGES POLICY
culture capabilities datamanagement
ethics&privacy culture evaluation
financial&humanresources
ethics&privacy methodology
infrastructure infrastructure policymanagement
internal&externalsupport management purpose
methodology methodology stakeholderengagement
purpose
stakeholderengagement
Outline
• Whatislearninganalytics?• Whereareweat?• Howaredashboardsused?• Howcanweevaluatelearninganalyticstools?• Howaboutaninstitutionallearninganalyticspolicy?• Wheredowegofromhere?
LASophisticationModel
Siemens,G.,Dawson,S.,&Lynch,G.(2014).ImprovingtheQualityandProductivityoftheHigherEducationSector–PolicyandStrategyforSystems-LevelDeploymentofLearningAnalytics.Canberra,Australia:OfficeofLearningandTeaching,AustralianGovernment.
ProblemofLAUptake
EducationalScience
ComputerScience
LearningAnalytics
MIDDLESPACE
Solution:UseLAasarealmiddlespace
EducationalScience
ComputerScience
LearningAnalytics
MIDDLESPACE
Learninganalytics…
… is not about the creation of intelligent or stupidsystems…… but about the creation of intelligent and successfulstudents…
…andthepromotionofeducation.
Ifeducationisviewedasavehicleforadvancingtheglobaleconomy,thenitmustbetheNorthStarthatguides societies to the next big thing, illuminatingnew ideas that solve pressing challenges andcreatingopportunitiestoshapeabetterfuture.
SamanthaAdamsBeckeretal.(2017)
Nowit’syourturn…
…tocreateyourowndashboard!
Thankyouforyourattention!
[email protected] [email protected]@m_a_s_c @IoanaJJEFLA:www.laceproject.eu/evaluation-framework-for-la/LAattheOU:www.ou.nl/en/welten-open-learning-analytics