FACT2 LEARNING ANALYTICSTask Group Report

SUNY Conference on Instructional Technologies May 2014

2013-14 Task Group Goals

• Will be to.. – develop the professional learning opportunities

for SUNY faculty and staff – “identify and share known best practices and

exemplary uses of Learning Analytics for assessment, and early intervention strategies.”

Task Group Activities Fall 2013 • Presented first year Task Group findings and best practices about

Learning Analytics at several SUNY conferences.– “Using Big Data to Enhance the Student Experience” panel presentation at

“Building a Smarter University: Big Data, Innovation and Ingenuity”, October 2013.

– “Enhancing Excellence in Assessment: Institutional Effectiveness and Learning Analytics” presented at SUNY Council on Assessment (ScoA) at The College at Brockport, Stony Brook University, and the University at Albany.

Spring 2014• Presented best practices about Learning Analytics at through SUNY

webinars.– “Learning Analytics: Best Practices for Student Assessment” – “Learning Analytics: Predicting Student Success in A Course”

Spring 2014 Pilot

• SUNY Oswego has piloted Starfish retention system over one academic year– Nearly 1,000 students targeted in programs with

known persistence issues (freshman and transfer students)

– 750 courses, 360 instructors, 100 advisors involved• Impact being assessed; compared to previous

performance of a similar cohort– scope of impact being measured

(effect vs. effort to "scale up" and track against all students)

SUNY Institutional Level Use

Course outcomes Intervention

• Student Retention

• Degree Completion

• More….

• Advising• Placement• Learning

outcomes• Degree

Completion

• Student Support• Persistence• Retention• Degree

Completion

• Learning outcomes

• Student feedback

• Instructional effectiveness

Learning Analytics - Working Definition

• Software that collects and analyzes multiple data sets related to the process of learning to PREDICT and IMPACT student success.

Learning Analytics Task Group of FACT2Learning Analytics Webinar: Tools & Best Practices for Student Assessment, 4/14/2014

Online course assignments

Social media

activities

Student data

Data potentially collected in…Blended and online learning environments, and other emergent resources connected to the teaching and learning experience.

Predictive Analytics: Building Models

Placement for success and completion..– which students should be steered toward which

courses? Which programs?

Can advising leverage student data?– If so, what are the best predictors of

performance?

Collect Data

Data Analysis

Actionable Results

Development of Large Scale Approaches

Predictive Analytics Reporting (PAR) Framework and “Data Cookbook”

Predictive Analytics: Building Models

Can we identify characteristics of a

successful outcome?

an unsuccessful

outcome?

DATA SOURCES

Grade in course

Can it be predicted by other data?• Major• High school GPA• English placement exam

score• Math placement exam

score• HS Regent scores….• SAT Verbal, SAT math• SAT Writing“every student with a HS

average of 83 or less, did not successfully complete the

course…”

Learning Outcomes

Teaching & Learning activity

Teaching & Learning activity

Student assignments

Teaching & Learning activity

Student assignments

Grading

Evaluate outcomes

met for course?

Typically, student assessment data is

collected a the end of a course and data is used to report on outcomes.

“Learning analytics is not in itself the goal but could provide a basis for decision

making for effective action.”

Learning Analytics Webinar: Tools & Best Practices for Student Assessment, 4/14/2014

Prior assessment techniques…Focused on course outcomes, but no real-time data for interventions….

Learning Analytics Webinar: Tools & Best Practices for Student Assessment, 4/14/2014

What happened?

Why did it happen?

What will happen?

How can we improve learning?

Descriptive Analytics

DiagnosticsAnalytics

PredictiveAnalytics

PrescriptiveAnalytics

2014 State of the Art

forLearning Analytics

Learning Analytics Webinar: Tools & Best Practices for Student Assessment, 4/14/2014

DIAGNOSE

FEEDBACK

NEW INSIGHTS

Learning AnalyticTools need to

• Align with learning principles & pedagogy

• Robust data analysis

• Ethical considerations• Institutional capacity

Is there a thoughtful educational plan for interventions and student feedback?

Learning Analytics Webinar: Tools & Best Practices for Student Assessment, 4/14/2014

RISKS & CONSIDERATIONS

Ethics of Data Collection. Permissions?

What is the educational plan for interventions and student feedback?

Learning Analytics Webinar: Tools & Best Practices for Student Assessment, 4/14/2014

RECOMMENDATIONSLearning Analytics Task Group

LATG Recommendations

• Develop a mechanism to encourage and support the adoption of Learning Analytics– For course placements, assessment, and degree completion – Through

• professional education programs• enabling data access

• Identify large Learning Analytics systems – for predictive analytics and intervention strategies– Expand & continue Pilots in process for system adoption, such

as StarFish. – Identify, pilot and adopt learning analytics assessment tools for

course use.

Recommendations:

Specifically:• Establish an ongoing working group to develop

Learning Analytic practices, tools and support services.

• Develop best practices for campuses interested in adoption – resource allocation, effort involved, faculty

development, organizational change management• Develop educational programs to encourage

campus adoption

Recommendation: Data Practices

• Establish common data definitions (e.g. definitions of "at risk" students) – leverage national data definition standards, such as the

“Data Cookbook” developed by the Predictive Analytics Reporting framework.

• Facilitate access to data and develop supporting policies for data access and privacy - ethical considerations.

• Develop common indicators and measurements to assess impact across multiple campuses

QUESTIONS?