Redefining “At-Risk” Through Predictive Analytics: A Targeted Approach to

Enhancing Student Success

Amilcah Gomes

Assistant Director, Academic Services Center

Eastern Connecticut State University

Predictive Analytics

• Using real-time data to plan for the future…

• “Predictive analytics helps your organization predict with confidence what will happen next so that you can make smarter decisions and improve business outcomes.” –IBM

Institutions That Use Predictive Analytics

• Predictive Analysis Reporting (PAR) Framework Institutions – American Public University

System – Ashford University – Broward College – Capella University – Colorado Community

College System – Lone Star College System – Penn State World Campus – Rio Salado College – Sinclair Community College – Troy University

– University of Central Florida – University of Hawaii System – University of Illinois

Springfield – University of Maryland

University College – University of Phoenix – Western Governors

University

• Purdue University • Rutgers University • University of South Florida • Eastern Connecticut State

University • Others

Eastern Connecticut State University

• Location: Willimantic, Connecticut • Institution Type: Public, Liberal Arts University • Total Enrollment (2012): 5,237 students, 4,506

undergraduate FTE • Nearly 2/3 residential, with 93% of first-time

students living on campus • 2011 FTFT cohort (N = 931):

– Female 53.4%; Students of Color 22% – Other Characteristics – First Gen Students (32.5%),

Compass Cohort (50.1%), STEP/CAP students (6.8%), Honors Scholars (2.4%), Athletes (9.7%)

• 2012 FTFT cohort (N = 979) • 2013 FTFT cohort (N = 960)

Institutional Changes

• Strategic Plan: Student Success Initiative (2008-2013) – Dual Advising Model

• Project Compass (2008-2012) – Communities of Practice

– Early Identification of At-Risk Students

– Enhancing First-Year Advising Services

– Targeting tutoring services to high-risk subjects

• Title III Student Support Services (2009-2014)

• 2013-2018 Strategic Plan & Targeted Student Success Initiatives

Development of Prediction Models

• 2008 – Multivariate model developed for predicting withdrawal prior to second year – Original assumption: students withdraw due to poor

academic performance – Problem: Original model failed to differentiate

between students who left for academic and non-academic reasons • Did not account for financial reasons, motivational variables,

engagement factors, etc. • Withdrawal risk quintiles were difficult to interpret • Professional advisors were unable to develop advising

strategies specific to a student’s needs

• 2011 – Developed additional multivariate model for academic risk; implemented two-model approach

Variable B S.E. Wald df Sig. Exp(B) Base

male -.087 .122 .508 1 .476 .917 female

black -.723 .278 6.786 1 .009 .485* white

hisp .347 .279 1.540 1 .215 1.414

oth_race -.674 .274 6.069 1 .014 .510

not_east .408 .133 9.401 1 .002 1.503 East CT River

commuter .497 .211 5.545 1 .019 1.644 Campus Pell_yr1 -.245 .161 2.316 1 .128 .783

Not Pell first_gen -.076 .124 .378 1 .539 .927

Not FGEN Athletics -.647 .218 8.816 1 .003 .524 Not Athlete HsGpa_quint1 .734 .189 15.070 1 .000 2.084

Quintile 3 HsGpa_quint2 .242 .184 1.727 1 .189 1.274

HsGpa_quint4 .277 .186 2.228 1 .136 1.319

HsGpa_quint5 .080 .223 .130 1 .719 1.084

admit_rating_le_4 -.057 .160 .125 1 .723 .945 Rate 5, 6, or 7 admit_rating_ge_8 -.573 .203 7.985 1 .005 .564

Vsat_quin1 -.069 .184 .140 1 .708 .934 VSAT Quint 3 Vsat_quin2 -.320 .180 3.157 1 .076 .726

Vsat_quin4 .191 .173 1.216 1 .270 1.210

Vsat_quin5 .172 .186 .854 1 .356 1.188

Stem -.036 .177 .042 1 .838 .964 Not STEM PreEd -.263 .165 2.536 1 .111 .769 Not PreED Undec .121 .132 .837 1 .360 1.129 Declared major ERG_none .484 .201 5.834 1 .016 1.623 ERG DEF ERG_ABC .164 .151 1.181 1 .277 1.178

ERG_GHI .269 .153 3.089 1 .079 1.309

got_schol_yr1 -.022 .165 .018 1 .893 .978 No schol got_FedLoan_yr1 -.227 .125 3.326 1 .068 .797 No Fed Loan Choice -.246 .120 4.235 1 .040 .782 Not #1/No

FAFSA Constant -1.282 .241 28.336 1 .000 .277

*Factors that are significant at the 0.10 level have been highlighted.

Withdrawal Model, 2011 Cohort (2008, 2009 Data)

Academic Risk Model, 2011 Cohort (2008, 2009 Data)

Model 2: for classifying 2011 Cohort into Academic-Risk Quintiles, Based on Data from 2008 and 2009 Cohorts.

B S.E. Wald df Sig. Exp(B)

HsGpa -1.505 .180 69.974 1 .000 .222

ERG_DEF .222 .151 2.165 1 .141 1.249

ERG_GHI .300 .157 3.657 1 .056 1.350

ERG_none .091 .204 .201 1 .654 1.096

black .365 .222 2.714 1 .100 1.441

hisp .429 .275 2.445 1 .118 1.536

oth_race -.194 .245 .629 1 .428 .824

Pell_yr1 .140 .148 .891 1 .345 1.150

first_gen .028 .123 .052 1 .820 1.028

male .344 .121 8.039 1 .005 1.410

Stem .387 .168 5.325 1 .021 1.472

PreEd -.274 .172 2.555 1 .110 .760

Athletics -.641 .216 8.770 1 .003 .527

admit_rating_le_4 .186 .144 1.663 1 .197 1.204

admit_rating_ge_8 -.414 .200 4.271 1 .039 .661

Constant 2.952 .557 28.125 1 .000 19.150

Interpreting the Models: What We Saw

Commuter

Not East

Federal Loans

FAFSA Choice

HS GPA

ERG/DRG (CT HS districts)

Athlete (negative, both)

African American identity (negative withdrawal, positive academic risk)

Admissions Rating ≥ 8 (negative, both)

Males

STEM Majors

Only Significant in Withdrawal Model

Only Significant in Academic Risk Model

Factors Significant in Both Models

Factors that had a significant impact (0.10 level):

Interpreting the Models: What We Didn’t See

• Factors that did not have a significant impact (0.10 level):

Pell eligibility (less significant in

GPA model)

1st Generation Status

Multiracial Identities (added to 2012

model)

Undeclared Majors (significant for 2nd to

3rd year retention)

Math SAT (good predictor of

GPA > 3.0)

Classifying Withdrawal & Academic Risk

Academic Risk Quintile 2011

Total 1.00 2.00 3.00 4.00 5.00

Q_2011 1.00 99 57 35 35 24 250

2.00 45 47 39 33 23 187

3.00 29 49 50 45 45 218

4.00 10 32 30 49 35 156

5.00 2 15 15 26 62 120

Total 185 200 169 188 189 931

TAC 1 = Intensive 172

TAC 2 = Tutoring 205

TAC 3 = Engaged 232

TAC 4 = Monitor 322

Cross-classification of 2011 cohort of entering first-time, full time students by the two types of risk: withdrawal and low academic performance:

Targeted Advising Cohorts (TAC)

• TAC 1 = Intensive (high risk withdrawal & high risk GPA < 2.3)

• TAC 2 = Tutoring (low risk withdrawal & high risk GPA < 2.3)

• TAC 3 = Engaged (high risk withdrawal & low risk GPA < 2.3)

• TAC 4 = Monitor (low risk withdrawal & low risk GPA < 2.3)

Targeted Advising Cohorts (TAC)

TAC 2 Tutoring

Low risk withdrawal

High risk GPA < 2.3

TAC 1 Intensive

High risk withdrawal

High risk GPA < 2.3

TAC 4 Monitor

Low risk withdrawal

Low risk GPA < 2.3

TAC 3

Engaged

High risk withdrawal

Low risk GPA < 2.3

High withdrawal

risk

Low withdrawal

risk

High academic risk

Low academic risk

Fall 2011 Cohort TAC Assignments*

18.5%

25.5%

22.8%

33.2%

TAC1=Intensive

TAC2=Tutoring

TAC3=Engaged

TAC4=Monitor

* 35 students originally in TACs 3 and 4 who did not participate in the library assessment and orientation were reassigned to TAC2.

By using the two-model approach it was determined that 18.5% of the FTFT students entering were at risk of withdrawal and low academic performance (TAC1), and another 25.5% were at risk of low academic performance (TAC2).

Observations

• Gender – Males made up less than half (46.6%) of 2011 cohort,

but represented 67% of TAC 1 & 2 students

• Race/Ethnicity – Students of color, particularly African American/Black

students, were overrepresented in TAC 2 (46.3% vs. 23.3% overall)

• Home of Record – 80.2% of TAC 1 students were “Not East”

(international, out of state, and students living west of the CT river) students (56.9% overall)

• Other Characteristics – 68.8% of Pell eligible and first generation students

were classified in TACs 2 & 4

Summary of Outcomes

• There is very little variation in credits attempted across the four TACs • Students in TAC 1 and TAC 2 on average were slightly less likely to achieve

the same level of academic momentum as their classmates in terms of credits earned

• In addition, the average GPAs for students in TAC 1 and TAC 2 are significantly lower than those in TAC 3 and TAC 4

• TAC assignment is clearly related to retention • TAC 3 and TAC 4 more likely to utilize tutoring services and devote more

hours with math and writing tutors on average

FTFT 2011 General Characteristics Outcomes (Average)

N %

Female % 1st Gen

% Students of Color # Credits Att. # Credits Earned

1st Yr. GPA Lib.Score Retention

927 53.4 32.5 22.0 29.40 26.18 2.79 1.11 75.5 TAC1 171 32.2 27.5 15.7 28.88 24.18 2.34 0.98 67.3 TAC2 235 37.0 40.0 42.4 28.83 23.13 2.38 0.99 74.9 TAC3 210 66.7 23.8 3.4 29.23 27.85 3.10 1.15 71.9 TAC4 306 69.0 35.0 22.6 30.23 28.49 3.14 1.23 83.7

Data-Informed Approach

• Reassess assumptions about student withdrawal patterns and ALANA student performance on campus

• Align institutional priorities to significantly enhance student success

• Better allocate limited staffing and financial resources to support high-impact practices

• Address persistence and performance issues among high-risk groups

Developing Targeted Interventions

• Freshman Preference Registration

• Developmental Advising Focus

• Leading Indicators Project (Library Orientation Score)

• Collaboration with Campus Departments

• Registration Holds and Financial Review

• Diversity Scholars Program

• Strategies for High-Performing Students

• Recognition of ALANA Student Success

Intervention: Major Course Selection

First-Semester GPA for FTFT 2011 & 2012 cohorts by TAC

TAC FTFT 2011 Cohort

Mean CGPA

N FTFT 2012 Cohort “Major Course” CGPA

N FTT 2012 Cohort No “Major Course”

CGPA

N

1 2.35 168 2.54 114 2.48 50

2 2.28 199 2.64 84 2.32 53

2A 2.49 32 2.90 36 3.31 3

3 3.10 205 3.28 121 3.07 55

4 3.15 304 3.25 191 3.22 77

Total 2.78 908* 2.99 546 2.83 238

* 23 students did not complete the Fall 2011 semester and were not included in the first semester GPA analysis

Intervention: Major Course Selection

TAC FTFT 2011 Cohort Returned Spring

2012

FTFT 2012 Cohort Returned Spring

2013

FTFT 2012 Cohort “Major Course” Returned Spring

2013

FTFT 2012 Cohort No “Major

Course” Returned Spring 2013

1 89.0% 90.0% 93.0% 84.9%

2 89.7% 93.0% 95.2% 90.7%

2A 82.9% 93.3% 94.4% 75.0%

3 89.4% 94.0% 95.0% 91.2%

4 94.8% 95.0% 95.3% 94.9%

Total 90.9% 93.3% 94.7% 90.7%

First-Semester Persistence for FTFT 2011 & 2012 cohorts by TAC

Intervention: Major Course Selection Year 1 Persistence to Year 2 Year 1 Academic Performance Outcomes (Average)

Cohort Overall

No Maj Crse ECO 200 BUS 201

Cume GPA

No Maj Crse

Cume GPA ECO 200

ECO 200 grade in 1st Semester

Cume GPA BUS 201

BUS 201 grade in 1st Semester

FTFT 2011 75.6% 72.5% 77.5% 82.4% 2.65 2.54 2.70 2.52 (C+/B-) 3.19 3.05 (B)

FTFT 2012 75.6% 70.0% 77.2% 80.9% 2.86 2.71 2.88 2.46 (C+/B-) 2.92 3.15 (B/B+)

Year 2 Persistence to Year 3 Year 2 Academic Performance Outcomes (Average)

Cohort Overall

No Maj Crse ECO 200 BUS 201

Cume GPA

No Maj Crse

Cume GPA ECO 200

ECO 200 grade in 1st Semester

Cume GPA BUS 201

BUS 201 grade in 1st Semester

FTFT 2011 61.1% 52.5% 65.0% 76.5% 2.72 2.48 2.91 2.52 (C+/B-) 3.16 3.05 (B)

• BUAD majors taking BUS 201 and ECO 200 during first semester via FPR

• Slight improvement in cumulative GPA for students in BUS 201 over ECO 200

• Students consistently performed better in BUS 201, regardless of TAC, etc.

• Some students may be at a disadvantage with ECO 200, particularly African American, Hispanic/Latino, FGEN, PELL, and TAC 1 & TAC 2 students

• Intervention for both at-risk and high-performing students

Intervention: Registration Holds • 36.8% of 2012 cohort had holds

just prior to Spring 2013 advising and registration, mostly financial – 44% of TAC 1 and 48% of TAC 2

students had holds – 45.7% of TAC 2A students had

holds – 76.2% of STEP/CAP students and

51.1% of nonwhite, non-STEP/CAP students had holds vs. nearly 30% of white students

– Average midterm GPA with holds: 2.53 (2.81 without holds)

• Collaboration with Enrollment Management, Fiscal Affairs, and Student Affairs – Financial Review Day – “Intrusive” Advising Strategies – Individual “Day Passes”

• Average cumulative GPA with holds: 2.74 (3.07 without holds)

TAC Returned Spring 2013

N Did Not Return

Total Not Returned

with Holds

N

1 90.0% 21 11 (52.4%) 211

2 93.0% 12 9 (75.0%) 171

2A 93.3% 3 2 (66.6%) 45

3 94.0% 14 8 (57.1%) 232

4 95.0% 16 8 (50.0%) 320

Total 93.3% 66 38 (57.6%) 979

Redefining Perceptions of “At-Risk”

Shifting from deficit-based assumptions…

Cognitive deficit assumptions mischaracterization

marginalization

…toward a success-based model!

Targeted initiatives based on student needs relevant

student support

Diversity Scholars Program

• Spring 2012 – Pilot group for FTFT SOC, TAC 1-2, < 2.0 GPA (N = 28)

• Fall 2012 – services for all FTFT ALANA students (N = 133) – Developmental advising services had positive impact on GPA for TAC 1,

limited impact for TAC 2 when given alone (“high-relational” groups)

– Peer mentoring: participants (2.79) vs. non-participants (2.47) • Hispanic/Latino student participants (3.03) vs. non-participants (2.13)

– Academic & student support interventions (e.g., taking major course in first semester, financial review)

– FTFT students of color had 3-4 times higher increase in GPA from previous year than white students, except TAC 4 African American students

– Reduction in GPA gap experienced across all ethnicities

• Fall 2013 – added targeted developmental advising outcomes and major-related opportunities/workshops (N = 138) – TAC 1-2 DSP participants (2.94) vs. ALANA non-participants (2.34)

– TAC 1-2 Hispanic/Latino students outperformed white students during first semester (2.72 vs. 2.59)

– TAC 3-4 African American students still underperform (2.89 vs. 3.10)

Recommendations for First-Year Success

• Use multiple approaches to assist students, based on individual needs and types of risk

• Consider non-academic barriers when advising students (finances, registration holds, campus integration, family issues, etc.)

• Encourage use of early alert system (APN) and sharing of information across departments

• Connect students to academic departments and major-related opportunities, particularly high performing students

Where do we go from here?

• Explore other non-cognitive measures and/or instruments and administer during the fall semester

• Revisit second-year persistence models developed during Project Compass

• Develop advising strategies for second-year students

• Communicate data findings with more faculty

• Increase access to high-impact practices, particularly for underrepresented students

ANY QUESTIONS?