Correlating Engagement and Student Success(Online version)

Student Success SpecialistWestern Oregon University

poolej@wou.edu503-838-8083

Jesse Poole

What Are We Talking About?

• Part One– Basic steps to implement an early alert system on

your campus• Part Two– Identifying any correlation between engagement

and student success at the basic level using an early alert system

What Do We Know?• Advising– Effective academic advising is seen as a key

element to student success and retention• Drake, 2013; Nutt, 2003; Swanson, 2006; Tinto, 1993

– Advisors today are addressing issues related to finance, personal relationships, decisions about coursework, choosing a major and minor, and academic progress• Drake, 2013; Miller, & Murray, 2005

What Do We Know (cont.)

• Early Alert– Electronic early alert and intervention systems

provide institutions with a methodical process used to identify and intervene with students that are exhibiting at-risk behaviors • Tampke, 2013

• Engagement– Students who engage in the at-risk process are less

likely to become at risk again, compared to those students who do not engage• Dobele et al., 2013

Western Oregon University• Western Oregon University

– Liberal Arts College– Approximately 5,200 undergraduate– Approximately 900 graduate

• Advising Structure– Central advising for core university requirements– Faculty advising for major and minor requirements– Multicultural Student Services and Programs (MSSP)– Student Enrichment Program (SEP)

• Second Year Retention– 2010 – 71.5%– 2011 – 75.1%– 2012 – 72.6%

Stakeholders and Support• Often starts from the top

– Provost– Vice President of __________

• Student affairs• Academic affairs

– Dean of _______• Students• The college of X, Y, Z

– University computing– Faculty

• Faculty senate president• Department and division chairs

Choosing a Program• Conduct preliminary research first• Create a feedback committee– Program administrator– University computing director– Director of SEP– Athletic compliance director– Faculty member– Registrar

• Preliminary needs assessment

Choosing a Program (cont.)

• Program review– Simplicity– Hobsons/Starfish– Grades First (EAB)– Student Success Plan (open source)

Implementation Team• Often four to six people• Anywhere between 60 and 90 days• Does not need to be the same people as the

feedback committee• Consider people’s unique ability/skills– Registrar’s office and FERPA– Computing services and technology

• Consider stakeholders– Advising offices– Faculty

Begin Implementation

• Review your needs assessment– What are wants and what are needs?

• Establish an implementation timeline• Begin data integration process– Export files– Single-sign-on through university Portal

• Configure your settings and permissions• Training– Train your key staff

Implementation• Training (cont.)– Train your faculty– Train your advisors– Train your students

• Program features and going live– Consider rolling out in phases• Early alert• Advising notes• Tutoring functionality• Report access

Part Two:Research Study

• Purpose– To gain access to archival data in order to evaluate

any correlation between engagement and student success using an early alert and intervention system

• The study does not– Measure the content of the intervention– Measure the approach to advising students with at-

risk tendencies• Scope– Restricted to undergraduate students only

Research Study

• Measuring Student Success– Based on the academic standing of the student for

the term in which they were referred– Academic Standing Policy for Western

Procedure• One-to-one referral system– Faculty or staff refer student using electronic system– Preset list of referral reasons w/ comment box– 24 to 48 hour response time– Outreach methods

• Email• Telephone

– Typically three attempts were made to contact student• Goal– Schedule a face-to-face appointment– Meetings were voluntary

Analysis• Analysis was quantitative

– Does not focus on personal account of the intervention or advising methods

• Two populations were evaluated– students that received an intervention– students that did not respond to an intervention request

• Three dispositions– Successful– Not successful– Neutral

What is successful?1. A student is referred AND:a. Was contacted by AALC staff regarding the RFI andb. The student is in good standing and stayed in good standing at the conclusion of the

term.

2. A student is referred AND:a. Was contacted by AALC staff regarding the RFI andb. The student was, at the time of the referral, in academic warning status but returned to

good standing at the conclusion of the term.

3. A student is referred AND:a. Was contacted by AALC staff regarding the RFI andb. The student was, at the time of the referral, in academic probation status but returned to

good standing at the conclusion of the term.

4. A student is referred AND:a. Was contacted by AALC staff regarding the RFI andb. The student was, at the time of the referral, on academic warning status AND continued

on academic warning (continued academic probation) at the conclusion of the term.

Neutral Students1. A student was in bad academic standing and maintained that same standing

at the conclusion of the term. For example, this may occur if a student was on academic warning and then withdrew from the current term. Thus, the student maintains the last term attended academic standing.

2. A student is a new freshman and did not have academic standing prior to the term. If the student then withdrew from the term, or otherwise received non-punitive grades resulting in no academic standing, the student would be considered neutral.

3. A student had no immediate prior term standing due to taking the prior term off and received non-punitive grades in the term in which they were referred.

4. A student had non-punitive grades in the prior term and received non-punitive grades in the term in which they were referred.

Analysis (cont.)

• Duplicate referrals– Because multiple referrals could be submitted

regarding the same student, duplicates within the system were identified and removed

• Summer session– Not included in the study

• Terms evaluated– winter 2013 (pilot), spring 2013, fall 2013, winter

2014, spring 2014– Ten week quarter system

Results

• Total # of referrals sent between winter 2013 and spring 2014– 732

• Average # of referrals per term– 146

• Total # of students referred (after removing duplicates)– 672

Results (cont.)

• # of students contacted– 279 (42%)

• # of students that did not respond– 393 (58%)

• Response rates are consistent with other early alert research findings (Hudson, 2006)

Results (cont.)

• Contacted students (279)– 137 of the 279 were considered successful (49%)– 131 of the 279 were not successful (47%)– 11 of the 279 were considered neutral (4%)

• Students that did not respond (393)– 204 of the 393 were not successful (52%)– 161 of the 393 were considered successful (41%)– 28 of the 393 were considered neutral (7%)

Results (cont.)

Comparing Success Rates• Percentage is based on the number of students that received

an intervention and the population that did not receive an intervention; these population numbers are not the same

• The population of students that did not respond showed an increased level of negative academic progression

• The analysis indicated a consistent increase in student success within the population of students that received an intervention

Comparing Failure Rates• There were some instances where the met-with

population had a higher failure percentage than the not-met with population

• Findings show a 15% higher rate of negative academic progress in the population that received an intervention, compared to the population that did not, 16 (34%), and 5 (19%) respectively

Total # of RFI Students (no dups) Total # of Students that were Contacted Total # of Students that did not RespondHR - HR 0 HR - HR 0 HR - HR 0HR - 00 2 HR - 00 1 HR - 00 100 - HR 0 00 - HR 0 00 - HR 000 - 00 33 00 - 00 15 00 - 00 18P1 - HR 0 P1 - HR 0 P1 - HR 0P1 - 00 6 P1 - 00 3 P1 - 00 3P1 - CP 5 P1 - CP 3 P1 - CP 2P2 - HR 0 P2 - HR 0 P2 - HR 0P2 - 00 1 P2 - 00 0 P2 - 00 1P2 - CP 1 P2 - CP 0 P2 - CP 1CP - HR 0 CP - HR 0 CP - HR 0CP - 00 0 CP - 00 0 CP - 00 0CP - CP 1 CP - CP 1 CP - CP 0SS - HR 0 SS - HR 0 SS - HR 0SS - 00 0 SS - 00 0 SS - 00 0SS - CP 0 SS - CP 0 SS - CP 0SS- RA 0 SS- RA 0 SS- RA 0RA - HR 0 RA - HR 0 RA - HR 0RA - 00 0 RA - 00 0 RA - 00 0RA - CP 0 RA - CP 0 RA - CP 0NS - HR 1 NS - HR 0 NS - HR 1NS - 00 1 NS - 00 0 NS - 00 1NP - HR 0 NP - HR 0 NP - HR 0NP - 00 0 NP - 00 0 NP - 00 0

TOTAL: 51 TOTAL: 23 TOTAL: 28

HR - P1 1 HR - P1 0 HR - P1 100 - P1 31 00 - P1 9 00 - P1 22P1 - P2 24 P1 - P2 12 P1 - P2 12P2 - SS 1 P2 - SS 0 P2 - SS 1P2 - RA 0 P2 - RA 0 P2 - RA 0CP - SS 1 CP - SS 0 CP - SS 1CP - RA 1 CP - RA 1 CP - RA 0RA - SS 3 RA - SS 2 RA - SS 1NS - P1 4 NS - P1 0 NS - P1 4NP - P1 0 NP - P1 0 NP - P1 0NP - P2 0 NP - P2 0 NP - P2 0NP - CP 0 NP - CP 0 NP - CP 0NP - SS 0 NP - SS 0 NP - SS 0NP - RA 0 NP - RA 0 NP - RA 0NS - RA 1 NS - RA NS - RA 1

TOTAL: 67 TOTAL: 24 TOTAL: 43

HR - NP 0 HR - NP 0 HR - NP 000 - NP 2 00 - NP 0 00 - NP 2P1 - P1 1 P1 - P1 1 P1 - P1 0P1 - NP 0 P1 - NP 0 P1 - NPP2 - NP 0 P2 - NP 0 P2 - NPCP - NP 0 CP - NP 0 CP - NPSS - NP 0 SS - NP 0 SS - NPRA - NP 0 RA - NP 0 RA - NPNS - NP 0 NS - NP 0 NS - NPNP - NP 0 NP - NP 0 NP - NPNS - NS 0 NS - NS 0 NS - NSP2 - P2 1 P2 - P2 1

TOTAL: 4 TOTAL: 2 TOTAL: 2

AREA TOTAL: 122 AREA TOTAL: 49 AREA TOTAL: 73

Good Standing % 42% Good Standing % 47% Good Standing % 38%Bad Standing % 55% Bad Standing % 49% Bad Standing % 59%Neutral % 3% Neutral % 4% Neutral % 3%Validate 100% Validate 100% Validate 100%

Financial Analysis• Duplicate referrals were removed• Any registration subsequent of the intervention could be seen as

gained tuition and fees• Assumes the student was in danger of attrition at the time of the

intervention