learner analytics and the “big data” promise for course & program assessment
DESCRIPTION
Presentation delivered at the San Diego State University "One Day in May" conference on May 22, 201 by John Whitmer, Hillary Kaplowitz, and Thomas J. Norman Universities archive massive amounts of data about students and their activities. Students also generate significant amounts of “digital exhaust” as they use academic technologies. How can faculty and administrators use automated analysis of this data to save time and conduct targeted interventions to improve student learning? The emerging discipline of Learner Analytics conducts analysis of this data to learn about student behaviors, predict students at-risk of failure, and identify potential interventions to help those students. In this presentation, we will discuss the contours of this discipline and review the state of research conducted to date. We will then look at several examples of Learner Analytics services and hear from California State University educators who are using these tools to help their students. Finally, we will suggest some immediate ways that Analytics can be conducted at San Diego State. Presenters: John Whitmer, California State University, Chico Hillary Kaplowitz, California State University, Northridge Thomas J. Norman, CSU Dominguez HillsTRANSCRIPT
San Diego State University “Day in May” 22 May 2012
John Whitmer, CSU Chico (& Office of the Chancellor)Hillary Kaplowitz, CSU Northridge
Thomas Norman, CSU Dominguez Hills
Learner Analytics and the “Big Data” Promise for Course & Program
Assessment
Download slides at: http://bit.ly/Kb6gsV
Outline
1. Promise of Learner Analytics
2. Case Studiesa) Analytics at work in the classroom (Hillary)
b) Improving classroom discussion and mastery of program level outcomes (Thomas)
c) Evaluating course redesign (John)
3. Analytics Tools @ SDSU
4. Q & A
1. PROMISE OF LEARNER ANALYTICS
John Goodlad’s Place-Based Research
Classroom-based research: “What is schooling?”
1,000 classrooms, 27,000 individuals
14 foundations needed to support
Fundamental changes to understanding of educational practice
Steve Lohr, NY Times, August 5, 2009
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.
7http://slidesha.re/IgKSTX
Source: jisc_infonet @ Flickr.com
Source: jisc_infonet @ Flickr.com
Current GPA: 3.3First in family to attend collegeSAT Score: 877
Hasn’t taken college-level math
No declared major
8
Academic Analytics
“Academic Analytics marries large data sets with statistical techniques and predictive modeling
to improve decision making”
(Campbell and Oblinger 2007, p. 3)
9
DD Screenshot
Learner Analytics
“ ... measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs.” (Siemens, 2011)
Fundamental Questions behind Learner Analytics
1. What are students doing (or not doing)? Which students are we talking about?
2. Does it matter (re: achievement, engagement, learning)?
3. What should we do? – Changes in student behavior? – Changes in faculty/program?
SIGNALS
http://www.itap.purdue.edu/studio/signals/Purdue Signals Project
Wordcloud of student evaluations of Course Signals (Arnold, 2010)
Signals Course Outcomes
Fall 2009 Compare same course (w/Signals v. w/o Signals)
-6.41% DWF
+10.97% A/B
(Arnold, 2010)
KHAN
http://www.khanacademy.org/Khan Academy
Source: wpclipart.com
Will Learner Analytics replace educators with computer algorithms? By Peter Nowak, Macleans.ca
“Robots are grading your
papers!” -Marc Bousquet,
4/18/2012 Blog post, Chronicle of Higher
Education
Contrasting State-of-the-Art Automated Scoring of Essays: Analysis
Mark D. Shermis, University of Akron (funded by Hewlett Foundation)
Compared 8 robo-graders to human grading on standardized essay questions from 6 states
Outcome: very small difference in results (.03 - .12 score difference)
Conclusion: are scoring algorithms sophisticated …. or are standardized essays simplistic … or do we need to stop the dichotomy of computers and people?
-- Tom Vander Ark, Gettingsmart.com
Or analytics can support faculty by …
1. Providing behavioral data to investigate student performance
2. Informing faculty about students succeeding or at risk of failing a course
3. Warning students that they are likely to fail a course – before it’s too late
4. Helping faculty evaluate the effectiveness of practices and course designs
5. Customizing content and learning activities
3. CSU CASE STUDIES
How can data help teachers and students work better
together?
Hillary Kaplowitz
Instructional Designer, Faculty Technology Center
Part-Time Faculty, Cinema and Television Arts Department
California State University, Northridge
Case #1
“I'm not upset that you lied to me, I'm upset that from now on I can't believe you.”
Friedrich Nietzsche
“Hey Professor,
I just looked at my assignments and realized that my Chapter 11 summary did not get submitted, which I'm having trouble believing that I didn't submit it... especially because I see that I did it, and I always submit my assignments as soon as I finish them.”
Now the hard part….
Do I believe him?
If I only I could check…
And it was all his idea…
The student suggested that I check Moodle and if that didn’t work told me how to check the Revision History in GoogleDocs with step-by-step directions!
Case #2
“Life isn't fair. It's just fairer than death, that's all.”
William Golding
“The quiz is unfair”
Hybrid Course Weekly Structure
1. Watch lectures
2. Read textbook
3. Online chat and tutoring
4. Post questions and take practice
quiz
5. Class meets
6. Aplia quiz
But the story was not that simple…
• Reports on Moodle painted a different picture• Student was watching the lectures at 10:00 p.m.• Then immediately taking quiz
Enabled constructive feedback…
Advised the student how the structure of the course was designed to enhance learning
Student revised their study habits Improved grades and thanked the instructor!
What we can do with data now
Use Reports in Moodle to verify student claims Review participant list to see last access time Empower students to review their own reports Analyze usage and advise students how to study better Review quiz results to find common misconceptions
Could we help improve student learning outcomes if we knew the effect of…
Coffee
Sequencing
Amount
Textbook
LMS Access
LMS Activities
Mobile
Attendance
Using Learner Analytics to Improve Classroom Discussion and Mastery of Program Level
Outcomes
Thomas J. Norman, Ph. D.California State University,
Dominguez [email protected]
Solving the Student Effort Challenge
• Prior surveys revealed that a majority of Management students were reading 5 chapters or less of the assigned 15 chapters
• The course average on the cumulative final was around 70%
• Using online assessments has boosted these scores 7-8 percentage points!
• These are tools made available by McGraw Hill and Aplia that you can use too:– McGraw Hill $39- $99 with eBook– Cengage Aplia $99 with eBook
Benefits of Online Assignments
• Assignment are due Sunday at 11:45 or 11:59 p.m.
• They ensure students have read and begun working with the concepts BEFORE classroom discussion and activities
• Provides immediate feedback
• Automatically graded
Aplia Real Time Metrics Progress and Mastery
At risk
LearnSmart Tale of 3 Students
Student 1 warned to keep up, ignored warning and failed courseStudent 2 knew material, completed homework in 6 hours A studentStudent 3 struggled early, but caught up and did well A- student
2
1
3
Analysis by AACSB Categories
Performance by Learning Objective/DifficultyWhy do my students do better at
medium difficulty questions?
This is working!
Not working!
48
EVALUATING COURSE REDESIGN: INTRO TO RELIGIOUS STUDIES 180
49
LMS Learner Analytics @ Chico StateEvaluation for Program Assessment
– Academy e-Learning course redesign
– Intro to Religious Studies: increased enrollment from 80 to 327 students first semester
– Outcome: increased mastery course concepts AND increased number D/W/F students
– Why? (and for whom? And what did they do?)
– What is the relationship between LMS actions, student background characteristics and student academic achievement? (6 million dollar question)
50
Grade Distribution
51
Grades by LMS “Dwell Time”
Grades by Dwell & Tool
3. ANALYTICS TOOLS @ SDSU
Bboard Learn Tools
BLACKBOARD LEARN: Activity Stats Reports
Evaluation: Course Reports
Performance Dashboard
Evaluation: Early Warning System
Grade Center: “Smart Views”
Student View: “Report Card”
Call to Action
1. You’re *not* behind the curve, this is a rapidly emerging area that we can (should) lead ...
2. Metrics reporting is the foundation for Analytics
3. Don’t need to wait for student characteristics and detailed database information; LMS data can provide significant insights
4. If there’s any ed tech software folks in the audience, please help us with better reporting!
64
Q&A and Contact Info
Resources Googledoc: http://bit.ly/HrG6Dm
Contact Info: • John Whitmer ([email protected])• Hillary C Kaplowitz ([email protected])• Thomas Norman ([email protected])
Download presentation at: http://bit.ly/Kb6gsV
65
Works CitedAdams, B., Bienkowski, M., Feng, M., & Means, B. (2012). Enhancing Teaching Learning through Educational Data Mining and Learning Analytics: An Issue Brief. Washington, D.C.: U.S. Department of Education, Office of Educational Technology.Arnold, K. E. (2010). Signals: Applying Academic Analytics. Educause Quarterly, 33(1). Bousquet, M. (2012). Robots Are Grading your Papers. Retrieved from http://chronicle.com/blogs/brainstorm/robots-are-grading-your-papers/45833Campbell, J. P., DeBlois, P. B., & Oblinger, D. G. (2007). Academic Analytics: A New Tool for a New Era. EDUCAUSE Review, 42(4), 17. Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy. The Economist.LaValle, S., Hopkins, M., Lesser, E., Shockley, R., & Kruschwitz, N. (2010). Analytics: The new path to value. Findings from the 2010 New Intelligent Enterprise Global Executive Study and Research Project: IBM Institute for Business Value and MIT Sloan Management Review.Manyika, J., Chui, M., Brown, B., Bughin, J., Dobbs, R., Roxburgh, C., & Hung Byers, A. (2011). Big data: The next frontier for innovation, competition, and productivity.Parry, M. (Producer). (2012, 5/14/2012). Me.edu: Debating the Coming Personalization of Higher Ed. Chronicle of Higher Education. Retrieved from http://chronicle.com/blogs/wiredcampus/me-edu-debating-the-coming-personalization-of-higher-ed/36057Siemens, G. (2011, 8/5). Learning and Academic Analytics. Retrieved from http://www.learninganalytics.net/