open learning analytics strategy for student success: the north carolina state university story
TRANSCRIPT
Open Learning Analytics Strategy for Student Success: The North Carolina State University Story
Lou Harrison, Director, Educational Technology Services, North Carolina State University Josh Baron, Assistant Vice President, Information Technology for Digital Education for Marist College Kate Valenti, Senior Director of Integration Services, Unicon, Inc.
Leading provider of IT Consulting for
the education market
Optimizing learning environments to improve student
success
In Today’s Webinar
• Historical Context: OAAI Overview • Apereo Learning Analytics Initiative Update • Feature: Moving Toward Enterprise Learning Analytics at
NC State • Q&A
Josh Baron ASSISTANT VICE PRESIDENT INFORMATION TECHNOLOGY FOR DIGITAL EDUCATION MARIST COLLEGE
Open Academic Analytics Initiative (OAAI) SOME BRIEF HISTORICAL CONTEXT…
Open Academic Analytics Initiative • EDUCAUSE Next Generation Learning
Challenges (NGLC) • Funded by Bill and Melinda Gates
Foundation • $250,000 over a 15 month period • Goal: Leverage Big Data concepts to
create an open-source academic early alert system and research “scaling factors”
OAAI Early Alert System Overview
Research Design Deployed OAAI system to 2200 students across four institutions
• Two Community Colleges • Two Historically Black Colleges and Universities
Design > One instructor teaching 3 sections • One section was control, other 2 were treatment groups
Each instructor received an AAR three times during the semester: • Intervals were 25%, 50% and 75% into the semester
Predictive Model Portability Findings Conclusion •Predictive models are more “portable” then anticipated.
• It is possible to create generic models that are then “tuned” for use at specific types of institutions.
• It is possible to create a library of open predictive models that could be shared globally.
Intervention Research Findings Final Course Grades
Analysis showed a statistically significant positive impact on final course grades •No difference between treatment groups
Saw larger impact in spring then fall Similar trend amount low income students
More Research Findings…
Jayaprakash, S. M., Moody, E. W., Lauría, E. J., Regan, J. R., & Baron, J. D. (2014). Early Alert of Academically At-Risk Students: An Open Source Analytics Initiative. Journal of Learning Analytics, 1(1), 6-47.
Apereo Learning Analytics Initiative OVERVIEW AND UPDATES
Apereo Learning Analytics Initiative (LAI) Goal: Operationalize outcomes from Learning Analytics research as means to develop, maintain and sustain modular components that integrate to support an open platform for Learning Analytics
Current Apereo LAI Related Projects • Marist College – Learning Analytics Processor (LAP) • Unicon – Student Success Plan (SSP) • University of Amsterdam – Larrisa (open-source Learning Record Store) • Uniformed Services University & Unicon - OpenLRS (Learning Record Store) and
OpenDashboard Apereo Incubation Project
Apereo Endorsed Project
Collection – Standards-based data capture from any potential source using Experience API and/or IMS Caliper/Senor API
Storage – Single repository for all learning-related data using Learning Record Store (LRS) standard.
Analysis – Flexible Learning Analytics Processor (LAP) that can handle data mining, data processing (ETL), predictive model scoring and reporting.
Communication – Dashboard technology for displaying LAP output.
Action – LAP output can be fed into other systems to trigger alerts, etc.
Modular Components of an Open Learning Analytics Platform
Library of Open Models
OpenDashboard
OpenLRS & Larrisa
Learning Analytics Processor
(LAP)
Student Success
Plan
Jisc National Learning Analytics Project
• Government funded non-profit that provides technology services to all of UK higher education
• Adopted much of the Apereo LAI platform and openness strategy
• Funding two-year project to create a highly scalable cloud-based learning analytics service
• All work released under open licenses
• Initial code release in Spring 2016
• See http://analytics.jiscinvolve.org/wp
Apereo - Jisc Learning Analytics Hackathon
Two organizations leading the way worldwide
in developing open architectures for
learning analytics are coming together at
LAK16 in Edinburgh for a two-day
hackathon on April 25-26, 2016. Jisc and
Apereo will put the growing ecosystem of
learning analytics products through their
paces with experimental big data coming
from learning management systems, student
record systems and other sources.
http://lak16.solaresearch.org/
Looking to learn more?
Apereo Learning Analytics Initiative Wiki: https://confluence.sakaiproject.org/x/rIB_BQ
GitHub: https://github.com/Apereo-Learning-Analytics-Initiative
Join the mailing list! [email protected] (subscribe by sending a message to [email protected])
Moving toward Enterprise Learning Analytics at NC State
LOU HARRISONDIRECTOR OF EDUCATIONAL TECHNOLOGY SERVICES
Building a visionHow to get everyone moving in the same direction
● Organizational Strategic Planning○ Student Success - NC State○ Leverage Technology to improve student
success - DELTA
● Specific Initiatives in Support of Plan○ Conversations with peers○ Get information to the right people (dashboard?)
SOCIALIZING A PLAN AND VISION
● Scheduled series of Lunch & Learn sessions within the LA Space.
● Bring people up to speed on what questions to ask
● Start thinking about who can generate answers
WORKING WITH OUR PARTNERS
● Everyone agrees helping students succeed is an admirable goal.
● We don’t always agree on how to do so.
● I had been in touch with Josh since a 2013 Educause talk he gave about OAAI. Other quarters of the university are exploring other technologies.
A SHARED GOAL
● There are still many unanswered questions.
● Who to notify, how to notify, when, how often?
● Some of these are ours to answer, others not.
HOW TO PROCEED?
● DELTA decided to dip its toes into the game with OAAI (which had recently been renamed LAP)
● I received funding to work with Unicon and Marist
● We built a plan to build us a model, and validate it.
MOVING IT FORWARD
● While Phase 1 was getting ramped up, we started planning for Phase 2
● More automatic, more turnkey, bigger, badder...
● More Enterprise-y™
PLANNING FOR PHASE 2
● LAP was designed for SAKAI, NC State uses Moodle
● We gathered some historical data, build a crosswalk
● Customized a first pass at a model for us, and...
MODEL VALIDATION
● Gradebook● Cumulative GPA● Academic Standing ● then
○ Course logins, content access, online flag...
PREDICTIVE POWER
● Overall accuracy 75-77%● Recall rates 88-90%
○ Highest seen in populations other than Marist● False Alarms (False positives) 25-26%
○ A little high
MODEL RESULTS
From Proof to ProductionToward Learning Analytics for the Enterprise
● Small sample sizes
● Predictions at ¼, ½, ¾ points in course
● Multi-step, manual process
INITIAL STEPS
● Large sample sizes (all student enrollments)
● Frequent early runs (maybe daily)
● Automatic, no more than one click
GOAL 1: MORE ENTERPRISE-Y
● Rebuild infrastructure for scale
● Daily snapshots of fall semester data
● After fall semester ends, look for sweet spot
CURRENTLY IN PROGRESS
● Refine model even more
● Segment Model by population
● Balance between models and accuracy
FUTURE GOALS
● Refine and improve model(s) over time
● Explore ways to track efficacy over time
● Once we intervene, can never go back to virgin state
FUTURE GOALS CONTINUED
If you want more… Wednesday, Oct 28th 2:30 PM - 3:20 PM Meeting Room 235-236 Indiana Convention Center
Visit Unicon’s booth #939
Opening Up Learning Analytics: Addressing a Strategic Imperative Josh Baron Assistant Vice President, Information Technology for Digital Education Marist College Lou Harrison Donna Petherbridge Dir of Educational Technology Svcs Associate Vice Provost, DELTA NC State University NC State University Kenny Wilson Division Chair-Health Occupation Programs Jefferson College
Questions/Answers
CONTACT
Lou Harrison | [email protected] Josh Baron | [email protected]
Kate Valenti | [email protected]
Q? A.