christopher brooks soed 2016
Post on 26-Jan-2017
1.021 Views
Preview:
TRANSCRIPT
brooksch@umich.edu @cab938
Shape of Educational Data: Predictive Modeling as an Enabler of
Personalized Learning
Christopher BrooksResearch Assistant Professor, School of Information
Director of Learning Analytics and ResearchDigital Education and Innovation
University of Michigan
brooksch@umich.edu @cab938
Psychohistory
“…[it] combined history, psychology and mathematical statistics to create a (nearly) exact science of the behavior of very large populations of people…Asimov used the analogy of a gas: in a gas, the motion of a single molecule is very difficult to predict, but the mass action of the gas can be predicted to a high level of accuracy. Asimov applied this concept to the population of the fictional Galactic Empire, which numbered in the quadrillions.”
http://asimov.wikia.com/wiki/Psychohistory
brooksch@umich.edu @cab938
• "Averaginarianism"• Regression towards a mean that
doesn't actually naturally exist• There is a gulf between the
predictive modeling perspectives, and the explanatory modeling ones
brooksch@umich.edu @cab938
Research Perspective• Learners are individuals• There is nuance in data that is
important and being missed bystudying populations vs. individuals
• Computational modelling (esp. predictive modelling) has opportunity to help
brooksch@umich.edu @cab938
Traditional Higher Education
Low Stakes Lifelong Learning
brooksch@umich.edu @cab938
Lecture Capture• How do students integrate educational technologies into their study habits?
– (and do those technologies have any effect?)• A need for insight
– Studies largely show only student satisfaction benefits from lecture capture– Several studies show no effect to the use of lecture capture on performance
• Data mining for usage patterns– Apply unsupervised machine learning methods (k-means clustering) to viewership data by
week– Then built general model from prototypes and apply to new datasets and determine fit
(replication)
brooksch@umich.edu @cab938
Results (Chemistry 2xx 2010)
• 5 groups found, each pedagogically labelled (by investigators!)• Error and size of groups ranges considerably• The final exam period is not indicative of activity throughout semester
brooksch@umich.edu @cab938
The general model
• How well does the model generalize?
brooksch@umich.edu @cab938
(Grades Pairwise Tukey HSD, * p<0.1 ** p<0.05)
(Grades)
brooksch@umich.edu @cab938
Results• Not a predictive model,
but a more discriminate descriptive model– Showed an effect not for general use of lecture
capture, but for specific ways of using lecture capture• Replication suggests there is merit to the model, but that
it is highly contextualized (theme of course)• Data from more sources could add further detail to the
model as to causal effects
Brooks, C. A., Erickson, G., Greer, J. E., Gutwin, C. (2014) Modelling and Quantifying the Behaviours of Students in Lecture Capture Environments. In Computers & Education. Vol 75 June. pages 282-292.
brooksch@umich.edu @cab938
Bonus Calculus Slide
brooksch@umich.edu @cab938
Massive Open Online Courses• As of the end of 2014, MOOCs at Michigan have attracted
1.9 million enrollees and nearly 1 million participants• Of these participants, ~ 300K attempt some assessment
task, ~80K end up passing the course (certificate)• Can we do better in understanding student success in this
environment?• Could we predict who is at-risk for students who want to
obtain a certificate?
brooksch@umich.edu @cab938
• MOOCs lack the diversity of data we have about residential students– Previous achievement (SAT/ACT, last years course)– Socioeconomic status (distance from university, first in family,
wealth)– Gender– Ethnicity– Motivation
• Building predictive models of student achievement in learning analytics is largely done on these entry-level features
• Both frustrating and refreshing– Want accurate models, but want actionable data
brooksch@umich.edu @cab938
• Built a novel feature selection algorithm inspired by work in the text-mining community
• It looks at the pattern of engagement that a student has with course resources
• Build of historical data (last years course) to create day-by-day multilevel models (C4.5)
• Initial work is based on student certificate achievement (pass/fail)– (not the only valuable outcome variable to try and
predict!)
brooksch@umich.edu @cab938
Resource
Day of Course
1 2 3 4 5 6 7 8 9Video
Daily AccessesDay 1: YesDay 2: NoDay 3: YesDay 4: NoDay 5: NoDay 6: NoDay 7: NoDay 8: YesDay 9: No
3-Day countsDay 1-3: YesDay 4-6: NoDay 7-9: Yes
Weekly countsWeek 1: YesWeek 2: Yes
Monthly countsMonth 1: Yes
For a 104 day long course,with three resources(videos, forums, quizzes)this gives us 408 featuresfor the modelling activity.
brooksch@umich.edu @cab938
Text Mining Inspiration• Text mining often uses n-grams as features in a document
– A bigram (cat, good) is the number of pairs of these two words in a document, a trigam (cat, was not good), etc.
– We build engagement n-grams up to 5 gramDaily AccessesDay 1: YesDay 2: NoDay 3: YesDay 4: NoDay 5: NoDay 6: NoDay 7: NoDay 8: YesDay 9: No
Possible bigrams[yes, yes]: 0[no, no]: 3[yes, no]: 3[no, yes]: 2
Possible trigrams:[yes, yes, yes]: 0[yes, yes, no]: 0[yes, no, yes]: 1…
For a 104 day long course, with three resources (videos, forums, quizzes) this gives us 717 more features for the modelling activity.
brooksch@umich.edu @cab938
In a nutshell• We do not have diverse set of data, but
we do have a detailed set of data• And there is a lot of it (200 million+
clickstream events)• By pulling out patterns of resource
access, we can use supervised machine learning (C4.5) techniques to build predictive models
• But what if we did have entry data from students?– Gender & Ethnicity, certification
status, country of origin, etc.
brooksch@umich.edu @cab938
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 750
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Fliess' κ versus Time in Days
Activity Features OnlyDemographics Features OnlyActivity and Demographics Features
Day of Course Offering
Flie
ss' κ
brooksch@umich.edu @cab938
Results• It is possible to create predictive models on clickstream data for MOOCs• 3 weeks into the MOOC seems to be an interesting point for some courses• It is computationally intensive to create these models (daily!)• MOOC entry/demographics information does not seem to add value
C. Brooks, C. Thompson, S. Teasley. (2015) A Time Series Interaction Analysis Method for Building Predictive Models of Learners using Log Data. 5th International Conference on Learning Analytics and Knowledge 2015 (LAK'15)
C. Brooks, C. Thompson, S. Teasley. (2015) Who You Are or What You Do: Comparing the Predictive Power of Demographics vs. Activity Patterns in Massive Open Online Courses (MOOCs). The second annual conference on Learning At Scale 2015 (L@S2015), Works in Progress track. Vancouver BC, March 14-15, 2015. Vancouver, BC.
brooksch@umich.edu @cab938
brooksch@umich.edu @cab938
No Particular Night or Morning
“I looked at the page with my name under the title…it was some other man…the story was familiar – I knew I had written it – but that name on the paper still was not me. It was a symbol, a name.”
“I’ve always figured it that you die each day, and each day is a box…but you never go back and lift the lids...each is a different you, somebody you do not know or understand or want to understand.”
brooksch@umich.edu @cab938
Questions? Comments?
Christopher BrooksResearch Assistant Professor, School of Information
Director of Learning Analytics and Researchin Digital Education and Innovation
University of Michigan
brooksch@umich.edu@cab938
top related