actionable learning analytics: practical lessons for teaching and learning
TRANSCRIPT
@stevelonn
ActionableLearningAnalytics:PracticalLessonsforTeaching&Learning
StevenLonn,Ph.D.
@stevelonn
Acknowledgements
• ColleaguesinDEI atMichigan• ColleaguesandstudentsinLEDLab atMichigan
• DevonCancilla andteamatUMKC• StephanieTeasley• TimMcKay• MatthewPistilli• KimArnold• JoeWaddington
2
@stevelonn
Who(ifanyone)isusingthecontentyoucreatefor
yourcourses?
3
@stevelonn
IsThis“Actionable?”
TheChronicleofHigherEducationJanuary5,2016http://chronicle.com/article/This-Chart-Shows-the-Promise/234573
4
@stevelonn
JoannaMirecki Millunchick
• ProfessorofMaterialsScienceandEngineering
• TeacheslargeintroductoryMaterialsScienceandEngineering(MSE)course
5
@stevelonn
ScreencastsforHWSolutions
6
@stevelonn
Screencastusagepositivelycorrelatedwithcourseperformance
• Chemicalengineersusedscreencaststheleast,yetreceivedthehighestgrades
• Industrial&operationsengineersusedscreencaststhemost,andreceivedaveragegrades
7
@stevelonn
What’sgoingoninthecurriculum?Materials Science&Engineering
• Physics ofMaterials
• PrinciplesofEngineeringMaterials
• ThermodynamicsofMaterials
• KineticsandTransport
• StructuresofMaterials
• MaterialsLaboratoryIandII
ChemicalEngineering
• IntrotoMaterialandEnergyBalances
• ChemicalandEngineeringThermodynamics
• FluidMechanics• MassandHeat
Transfer• Separation
Processes• Reaction
EngineeringandDesign
• ChemicalEngineeringLaboratory
AerospaceEngineering
• IntrotoAerospaceEngineering
• IntrotoAerospaceEngin.Systems
• IntrotoSolidMechanicsandAerospaceStructures
• IntrotoGasDynamics
• AircraftandSpacecraftStructures
• Aerodynamics• Propulsion
Industrial&Operations Eng.
• Economic DecisionMaking
• OperationsModeling
• Probability andStatistics
• OptimizationMethods
• MarkovProcesses• Ergonomics• LinearStatistics• DataProcessing
Screencastsshouldhaveagreaterimpactasasupplementaryresourcetoaidstudentlearningforstudentswithlessfamiliaritywiththecoursecontent.
8
@stevelonn
CaseStudyonExamPerformance
• IfIOEstudentswatchedany screencasts,theyperformedbetteronaparticular question.– r =.425p =.001
• IfIOEstudentswatchedaparticular screencast,therelationshipevenstrongerfortheperformanceonthatparticular question– r =.375p =.003
9
Green,Pinder-Grover&Millunchick,2012
@stevelonn
Summary• Analytic:LMShitsandperceptionsurveysaboutsupplementaryproblem-setscreencasts
• Finding:Sig.content-performanceimpactforlearnerswithoutpriorMSEknowledge– NoimpactforstudentswithpriorMSEknowledge
• Whatcanyoudo?:Beginbybreakingdownassessmentsbydiscipline/sub-discipline.Unpackanysig.differences– (gender,ethnicity,curriculum,etc.)
10
@stevelonn
WhatisLearningAnalytics?
11
@stevelonn
Learninganalyticsisthemeasurement,collection,analysisandreportingofdataaboutlearnersandtheircontexts,forpurposesofunderstandingandoptimisinglearningandtheenvironmentsinwhichitoccurs.
1212
@stevelonn
measurement,collection,analysisandreporting
ofdata
1313
@stevelonn
learners andtheircontexts
1414
@stevelonn
understanding andoptimising learning
1515
@stevelonn
IsUMKCpreparedtoimplement
learninganalyticstoday?
16
@stevelonn
WhenisanInstitutionReady toImplementLearningAnalytics?
17
Read�i�ness(ˈred-ē-nis)
noun1 willingness to do something.
2 the state of being fully prepared for something.
3 immediacy, quickness, or promptness.
@stevelonn 18
TheNeedforReflection
• Comprehensiveunderstanding
• Multipleperspectivesneeded
• Multiplelevelsneeded
TheLearningAnalyticsReadinessInstrument
Arnold, Lonn&Pistilli,2014
@stevelonn
LARIBetaParticipants
19
Assoc/Pub-R-L,Bac/A&S,&Bac/Diverse
Master’sLarge&ResearchUniversities(HighResearchActivity)
ResearchUniversities(VeryHighResearchActivity)
5institutions129respondents
15institutions337respondents
4institutions84respondents
@stevelonn
ReadinessFactors
20
Factor # SurveyItems
Example
Culture 22 “Myinstitutionhasaculturethatacceptstheuseofdatatomakedecisions.”
DataManagementExpertise
8 “Myinstitutionhastheabilitytostoreincreasinglylargevolumesofdata.”
DataAnalysisExpertise
6“My Institutionhasprofessionalswithmathematical/statisticalexperienceinmanipulatingandtransforming data.”
Communication&Policy Application
7 “MyInstitutionhasprofessionalswithbusinessacumeninmarketing/publicity.”
Training 3“Myinstitutionhasprofessionalswithcustomer-facingsupport experienceintrainingconstituentsontheuseofnewsystems.”
Oster,Lonn,Pistilli,Brown,2016
@stevelonn
CultureisPervasive
21
@stevelonn
GrowinganInstitutionalLearningAnalyticsCommunity
22
“StudentLearningandAnalyticsatMichigan”seminarseries
crlt.umich.edu/slam
Othercomponents:
• LearningAnalyticsfacultytaskforce(3yearcharge)• “ExploringLearningAnalytics”researchprojects• Learninganalyticsfellowsprogram• Streamliningdataaccessandestablishingnewdatasets
@stevelonn
PracticalLearningAnalyticsMOOC
23
Everyoneineducationhasquestions….PLAisaboutansweringthem.
ReturningSpring2016
@stevelonn
Summary• Analytic:Institutionalreadinessmeasuredbyinstitutionalreflection(surveyresponses)
• Finding:5factors:Culture,datamanagement,dataanalysis,communication&policyapplication,&trainingallimportantforreadiness
• Whatcanyoudo?:Helpestablishcommunicationaboutlearninganalyticswithcolleagues.– e.g.,Organizeamonthlyseriestodiscussandhavespeakers.
24
@stevelonn
Aretheredifferentkindsofanalyticsineducation?
25
@stevelonn
Howlonguntiltheygraduate?
4-yeargraduationrate:22%6-yeargraduation rate:51%
Whichstudentsareaccepted?
Roughly4,300 HSstudentsapplyUMKCaccepts65-68%ofapplicants
Ofthose,about39%enroll(F15 n=1,021)
Whichstudentsstay?
Freshman retentionrate:73%Part-timestudentretentionrate:28%
LearningAnalyticsvs.AcademicAnalytics
26
@stevelonn
LearningAnalyticsvs.AcademicAnalytics
27
AcademicLoad
EnrollmentbyAcademicLevel
EnrollmentbyDistanceEducation
Sources:irapweb.umkc.edunces.ed.gov
@stevelonn
HowtoMeasureStudentLearning
• Traditionalmeasures
• Holisticmeasures
• Psychometrics
• 21st CenturySkills
28
@stevelonn
Data+Tools=NewWaystoMeasureTeachingandLearning
29
Student InformationSystemsLearningManagementSystems
Onlinehomework, peerevaluation,discussions, practicetools,etc.
DashboardsStatisticalanalysistools
MachinelearningSocialnetworkanalysis
Naturallanguageprocessing
@stevelonn
NewWaysforInstructorstoViewStudentActivity&Performance
30
@stevelonn
UnpackingCourseResourceAnalytics
31
• NancyKerner• LecturerIV,GeneralChemistry(Emeritus)
• TaughtlargeintroductoryChemistrycourse
@stevelonn 32
• 10semestersofdata– 8,762students;703,191“hits”
• 4categoriesemerged– Courseinformation– Lecture– Assignments– ExamPreparation
UnpackingCourseResourceAnalytics
@stevelonn 33
• Raw#of“hits”:Astudents>Bstudents>Cstudents
• 25th à 50th percentileofresourceuse:
UnpackingCourseResourceAnalytics
ResourceCategory ‘A’vs.‘C’ ‘B’vs.‘C’
CourseInformation 0.856 0.907
Lecture 1.228 Not SignificantAssignments Not Significant Not Significant
ExamPreparation 1.811 1.565
Waddington &Nam,2014
@stevelonn
DoestheModelScale?HowCanitScaleforYou?
34
• Expandedto7largeSTEMcourses– 24Kstudents
• Similartrends– OnlyExamPrepresourcesrelatedtobetterperformancein5of7courses
Waddington, Nam,Lonn&Teasley,InReview
@stevelonn
Summary• Analytic:LMShitsoncourseresourcesandsemi-manualcategorizationinto4keytypes.
• Finding:Exampreparationmaterials(whenclicked)onlysignificantresourcetyperelatedtoimprovedcourseperformance.
• Whatcanyoudo?:IdentifythetypesofcourseresourcesavailableinyourLMSwebsite– whatcategoriesemerge?– WhatdoestheLMStellyouabouthow,andwho,isusingdifferentcategories?
– Whatistheeffectonlearning?
35
@stevelonn
Whatlearninganalyticsinnovationsareonthehorizon
atMichiganandbeyond?
36
@stevelonn
PresentingCourse-RelatedAnalyticstoStudents
37
@stevelonn
UsingAnalyticstoCoachStudents
38
@stevelonn
NaturalLanguageProcessing
39
• Outofthebox:Sentiment,bag-of-wordsgroupings,readinglevel• Pairs,triads,sentencestructures• Topicmodeling• Customdictionariesbydiscipline,contentarea,etc.
@stevelonn
EthicalConsiderations
40
@stevelonn
AnalyticsforLearners
41
@stevelonn 42
Knowingis notenough;wemustapply.
Wil l ingis notenough;wemustdo.
―JohannWolfgangvonGoethe
@stevelonn 43
ThankYou!
StevenLonn,Ph.D.AssistantDirectorforAssessment&EvaluationDigitalEducation&InnovationUniversityofMichigan
[email protected]@stevelonn
digitaleducation.umich.edu