analytics: from reflective to predictive
TRANSCRIPT
David Onder Director of
Assessment
Casey Iannone Consultant
Background
2
Background
3Freshmen Only
The Problem• We are asked for program-level data
• Lack of program-level retention and graduation rates
• Complicated (particularly for undergrads)
• Different programs serve different purposes
• High stakes – program prioritization (AA driven)
• Reports lumped all non-retained together (whether they graduated or stopped out)
4
Reflecting on the past
What We Wanted• Solid & simple approach (easy to explain and defend)• Fair
• Useful for all types of programs • Meaningful for decision-making (high- and low-level)• Not overly complicated display• Illuminates
• Overall performance• Historic trends• When are students lost
• Something that can be generated yearly w/o too much effort
7
5 Outcomes
• Five possible outcomes for each student that declares a given major• Retained in program
• Graduated in program
• Retained in different program
• Graduated in different program
• Not retained (stop-out/drop-out)
• Exclusive and exhaustive8
General Approach
• Based on cohorts:• A student is placed in a program cohort the 1st time
they declare a given program
• Each student in the cohort is flagged as one of the 5 possible outcomes for each ½ year interval (each regular semester)
• At each interval we report where the members of the cohort fall
• Each student will only appear in one cohort for a program (usually)
9
Technical Approach
10
Multiple IterationsStarted with term-level data
New Cohort @1 year @2 years @3 years @4 years @6 years
0.5 1 2 3 4 6
Year SemesterNew Cohort Prg Ret Prg Grad WCU Ret WCU Grad Not
Retained Total Prg Ret Prg Grad WCU Ret WCU Grad Not Retained Total Prg Ret Prg Grad WCU Ret WCU Grad Not
Retained Total Prg Ret Prg Grad WCU Ret WCU Grad Not Retained Total Prg Ret Prg Grad WCU Ret WCU Grad Not
Retained Total
1996 Fall 28 54% 18% 4% 0% 25% 100% 25% 46% 0% 0% 29% 100% 21% 61% 0% 0% 18% 100% 14% 64% 0% 0% 21% 100% 7% 79% 0% 0% 14% 100%1997 Spring 3 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 0% 33% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100%
1997 Fall 14 50% 14% 0% 0% 36% 100% 7% 43% 0% 0% 50% 100% 14% 50% 0% 0% 36% 100% 0% 50% 0% 0% 50% 100% 0% 57% 0% 0% 43% 100%1998 Spring 4 75% 0% 0% 0% 25% 100% 25% 50% 0% 0% 25% 100% 0% 75% 0% 0% 25% 100% 0% 75% 0% 0% 25% 100% 0% 75% 0% 0% 25% 100%
1998 Fall 15 80% 7% 0% 0% 13% 100% 47% 40% 0% 0% 13% 100% 13% 53% 0% 0% 33% 100% 13% 60% 0% 0% 27% 100% 7% 67% 0% 0% 27% 100%1999 Spring 5 60% 0% 20% 0% 20% 100% 40% 0% 0% 20% 40% 100% 0% 40% 0% 20% 40% 100% 0% 40% 0% 20% 40% 100% 0% 40% 0% 20% 40% 100%
1999 Fall 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100%2000 Spring 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100% 25% 25% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100% 0% 50% 0% 0% 50% 100%
2000 Fall 3 67% 0% 0% 0% 33% 100% 0% 0% 0% 0% 100% 100% 33% 0% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100%2001 Spring 5 40% 0% 0% 0% 60% 100% 20% 20% 0% 0% 60% 100% 20% 20% 0% 0% 60% 100% 0% 40% 0% 0% 60% 100% 0% 40% 0% 0% 60% 100%
2001 Fall 9 56% 22% 11% 0% 11% 100% 22% 56% 11% 0% 11% 100% 22% 56% 0% 11% 11% 100% 0% 78% 0% 11% 11% 100% 0% 78% 0% 11% 11% 100%2002 Spring 5 60% 0% 0% 0% 40% 100% 60% 0% 0% 0% 40% 100% 0% 0% 0% 0% 100% 100% 0% 0% 0% 0% 100% 100% 20% 0% 0% 0% 80% 100%
2002 Fall 10 60% 10% 10% 0% 20% 100% 20% 40% 0% 10% 30% 100% 10% 50% 10% 10% 20% 100% 10% 50% 0% 10% 30% 100% 0% 60% 0% 10% 30% 100%2003 Spring 6 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 17% 50% 0% 0% 33% 100% 0% 67% 0% 0% 33% 100% 17% 67% 0% 0% 17% 100%
2003 Fall 11 73% 9% 0% 0% 18% 100% 36% 45% 0% 0% 18% 100% 9% 73% 0% 0% 18% 100% 0% 73% 9% 0% 18% 100% 0% 73% 0% 9% 18% 100%2004 Spring 4 75% 0% 0% 0% 25% 100% 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100% 25% 25% 0% 0% 50% 100% 0% 25% 0% 0% 75% 100%
2004 Fall 12 67% 0% 17% 0% 17% 100% 25% 42% 0% 8% 25% 100% 0% 50% 0% 8% 42% 100% 0% 50% 0% 8% 42% 100% 0% 50% 0% 8% 42% 100%2005 Spring 3 67% 0% 0% 0% 33% 100% 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100% 0% 67% 0% 0% 33% 100% 0% 67% 0% 0% 33% 100%
2005 Fall 17 71% 6% 0% 0% 24% 100% 35% 41% 0% 0% 24% 100% 12% 53% 6% 0% 29% 100% 6% 59% 0% 6% 29% 100% 0% 65% 0% 6% 29% 100%2006 Spring 8 50% 13% 0% 13% 25% 100% 25% 13% 25% 13% 25% 100% 13% 25% 13% 25% 25% 100% 0% 38% 0% 38% 25% 100% 0% 38% 0% 38% 25% 100%
2006 Fall 7 57% 0% 0% 0% 43% 100% 0% 29% 29% 0% 43% 100% 0% 29% 14% 14% 43% 100% 0% 29% 0% 14% 57% 100% 0% 0% 0% 0% 0% 0%2007 Spring 9 33% 22% 22% 0% 22% 100% 0% 56% 22% 0% 22% 100% 0% 56% 0% 11% 33% 100% 0% 56% 0% 11% 33% 100% 0% 0% 0% 0% 0% 0%
2007 Fall 13 92% 0% 0% 0% 8% 100% 38% 46% 0% 0% 15% 100% 15% 69% 0% 0% 15% 100% 8% 77% 0% 0% 15% 100% 0% 0% 0% 0% 0% 0%2008 Spring 3 33% 0% 0% 0% 67% 100% 0% 33% 0% 0% 67% 100% 0% 33% 33% 0% 33% 100% 0% 33% 0% 0% 67% 100% 0% 0% 0% 0% 0% 0%
2008 Fall 18 67% 6% 6% 0% 22% 100% 22% 50% 6% 0% 22% 100% 17% 61% 6% 0% 17% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%2009 Spring 7 57% 0% 0% 0% 43% 100% 57% 14% 0% 0% 29% 100% 29% 43% 0% 0% 29% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2009 Fall 15 80% 0% 7% 0% 13% 100% 27% 53% 0% 7% 13% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%2010 Spring 3 67% 0% 0% 0% 33% 100% 67% 0% 0% 0% 33% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2010 Fall 17 76% 0% 0% 0% 24% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%2011 Spring 6 50% 0% 0% 0% 50% 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
2011 Fall 9 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0%
New Cohort @1 year @2 years
0.5 1 2
Year SemesterNew Cohort Prg Ret Prg Grad WCU Ret WCU Grad Not
Retained Total Prg Ret Prg Grad WCU Ret WCU Grad Not Retained Total
1996 Fall 28 54% 18% 4% 0% 25% 100% 25% 46% 0% 0% 29% 100% 1997 Spring 3 67% 0% 0% 0% 33% 100% 33% 33% 0% 0% 33% 100%
1997 Fall 14 50% 14% 0% 0% 36% 100% 7% 43% 0% 0% 50% 100% 1998 Spring 4 75% 0% 0% 0% 25% 100% 25% 50% 0% 0% 25% 100%
1998 Fall 15 80% 7% 0% 0% 13% 100% 47% 40% 0% 0% 13% 100% 1999 Spring 5 60% 0% 20% 0% 20% 100% 40% 0% 0% 20% 40% 100%
1999 Fall 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100% 2000 Spring 4 75% 0% 0% 0% 25% 100% 25% 25% 0% 0% 50% 100%
This is helpful for programs, in the context of program history,
but NOT administrators
11
Academic Year
Next combined data into academic years
12
New Cohort @1 year @4 years New Cohort 1 4
Year Semester Class LevelNew Cohort Prg Ret Prg Grad WCU Ret WCU
GradNot
Retained Total Prg Ret Prg Grad WCU Ret WCU Grad
Not Retained Total
2006-2007 Total 40 33% 0% 25% 0% 43% 100% 3% 15% 18% 26% 38% 100%Lower Division 36 28% 0% 25% 0% 47% 100% 3% 17% 19% 28% 36% 103%Upper Division 4 75% 0% 25% 0% 0% 100% 0% 0% 0% 0% 50% 50%
2007-2008 Total 27 56% 0% 7% 0% 37% 100% 15% 22% 26% 7% 78% 148%Lower Division 26 54% 0% 8% 0% 38% 100% 15% 12% 27% 4% 81% 138%Upper Division 1 100% 0% 0% 0% 0% 100% 0% 300% 0% 100% 0% 400%
2008-2009 Total 42 45% 0% 24% 0% 31% 100% 12% 7% 10% 2% 33% 64%Lower Division 38 42% 0% 24% 0% 34% 100% 13% 8% 11% 3% 34% 68%Upper Division 3 67% 0% 33% 0% 0% 100% 0% 0% 0% 0% 33% 33%
2009-2010 Total 38 50% 0% 11% 0% 39% 100% 11% 26% 18% 11% 45% 111%Lower Division 32 47% 0% 9% 0% 44% 100% 13% 22% 22% 13% 50% 119%Upper Division 6 67% 0% 17% 0% 17% 100% 0% 33% 0% 0% 17% 50%
2010-2011 Total 62 48% 0% 19% 0% 32% 100% 5% 8% 3% 5% 40% 61%Lower Division 55 42% 0% 22% 0% 36% 100% 5% 7% 4% 2% 40% 58%Upper Division 7 100% 0% 0% 0% 0% 100% 0% 14% 0% 29% 43% 86%
2011-2012 Total 51 51% 0% 22% 2% 25% 100% Lower Division 42 50% 0% 24% 0% 26% 100% Upper Division 9 56% 0% 11% 11% 22% 100%
2012-2013 Total 56 52% 0% 25% 0% 23% 100% Lower Division 44 52% 0% 27% 0% 20% 100% Upper Division 12 50% 0% 17% 0% 33% 100%
2013-2014 Total 11 64% 0% 27% 0% 9% 100% Lower Division 6 50% 0% 50% 0% 0% 100% Upper Division 5 80% 0% 0% 0% 20% 100%
@1 year @4 years 1 4
Year Semester Class LevelNew Cohort Program Success Non-program
Success Not Retained Program Success Non-program Success Not Retained
2006-2007 Total 40 33% 25% 43% 18% 44% 38%Lower Division 36 28% 25% 47% 19% 47% 36%Upper Division 4 75% 25% 0% 0% 0% 50%
2007-2008 Total 27 56% 7% 37% 37% 33% 78%Lower Division 26 54% 8% 38% 27% 31% 81%Upper Division 1 100% 0% 0% 300% 100% 0%
2008-2009 Total 42 45% 24% 31% 19% 12% 33%Lower Division 38 42% 24% 34% 21% 13% 34%Upper Division 3 67% 33% 0% 0% 0% 33%
2009-2010 Total 38 50% 11% 39% 37% 29% 45%Lower Division 32 47% 9% 44% 34% 34% 50%Upper Division 6 67% 17% 17% 33% 0% 17%
2010-2011 Total 62 48% 19% 32% 13% 8% 40%Lower Division 55 42% 22% 36% 13% 5% 40%Upper Division 7 100% 0% 0% 14% 29% 43%
2011-2012 Total 51 51% 24% 25% Lower Division 42 50% 24% 26% Upper Division 9 56% 22% 22%
2012-2013 Total 56 52% 25% 23% Lower Division 44 52% 27% 20% Upper Division 12 50% 17% 33%
2013-2014 Total 11 64% 27% 9% Lower Division 6 50% 50% 0% Upper Division 5 80% 0% 20%
SummaryAll students lumped into 3 groups
This is the most summarized data we can (read: are willing to) provide.The bottom line = Bold 3-group number
13
But how does that compare?
This is average program data to use as a comparison
14
New Cohort @1 year @4 years New Cohort 1 4
Year Semester Class Level New Cohort Program Success Non-program
Success Not Retained Program Success Non-program Success Not Retained
2006-2007 Total 1 64% 16% 21% 44% 24% 32% Lower Division 40 62% 19% 19% 34% 30% 36% Upper Division 37 68% 9% 23% 61% 12% 27% 2007-2008 Total 2 59% 16% 25% 38% 27% 36% Lower Division 40 55% 19% 26% 26% 33% 41% Upper Division 36 67% 11% 22% 57% 16% 27% 2008-2009 Total 0 56% 19% 25% 30% 32% 38% Lower Division 27 54% 22% 24% 23% 37% 40% Upper Division 26 60% 14% 26% 43% 23% 35% 2009-2010 Total 1 39% 38% 23% 31% 28% 41% Lower Division 42 33% 41% 26% 23% 25% 52% Upper Division 38 50% 33% 17% 45% 33% 22% 2010-2011 Total 1 67% 13% 20% 57% 12% 31% Lower Division 38 59% 17% 24% 43% 15% 42% Upper Division 32 75% 9% 16% 73% 9% 18% 2011-2012 Total 2 66% 8% 26% Lower Division 64 63% 10% 27% Upper Division 55 71% 5% 24% 2012-2013 Total 3 66% 9% 25% Lower Division 51 62% 11% 28% Upper Division 42 74% 6% 20% 2013-2014 Total 3 65% 14% 21% Lower Division 58 60% 17% 23% Upper Division 44 75% 8% 16%
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
2011-2012
2012-2013
2013-2014
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Program Success WCU Success Not Retained
Horizontal stacked bar• Normally we would never do this, but
…..
What are people asking:- How is the program performing?- How are students performing overall at institution?- How many are dropping out?
15
We graph 5 Flags too
@ 1 year @ 4 year @ 6 year
Transitions to completers and drop-outs
16
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
2011-2012
2012-2013
2013-2014
0% 20% 40% 60% 80% 100%
0% 10%20%
30%40%
50%60%
70%80%
90%100%
0% 10%20%
30%40%
50%60%
70%80%
90%100%
Compare performance over time
Main question
17
2006-2007
2007-2008
2008-2009
2009-2010
2010-2011
2011-2012
2012-2013
2013-2014
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
But what about next year?
If we study learning as a data science, we can reverse engineer the human brain and tailor learning techniques to maximize the chances of student success. This is the biggest revolution that could happen in education, turning it into a data-driven science, and not such a medieval set of rumors professors tend to carry on.
-Sebastian Thrun
Most data is only viewed in one direction.
A Small Window of Time Exist to
Reach Students.
The Rising Tide Of Data.
What is Machine Learning?
Machine learning is the science of getting computers to act without being explicitly programmed.
-Stanford University
Supervised Learning
Unsupervised Learning
Cont
inuo
usSupervised
Unsupervised
Dis
cret
eClassification
or Categorization
Regression
Clustering
Dimensionality reduction
Cont
inuo
usSupervised
Unsupervised
Dis
cret
eClassificationo KNNo Treeso Logistic
Regressiono Naïve-Bayeso SVM
Regressiono Linearo PolynomialDecision TreesRandom Forests
Association Analysiso Apriorio FP-GrowthHidden Markov Model
Clustering & Dimensionality reductiono SVDo PCAo K-means
Ready to enter the
Lab
First-year Retention Example
Student Retention
Binary response:
Retained yes/no
Brainstorm
BrainstormCampus Assessment/Effectiveness
Framework
Data Dissemination
Key Stakeholders
Education