“Improving the Success of Students at Risk of Dropping Out: Changing Practices and Policies”

Dr. Andrew ParkinAssociate Executive Director

Canada Millennium Scholarship FoundationCanada

18th Annual 2009 European Access Network Annual Conference

Changing the Culture of CampusTowards an Inclusive Higher Education

June 22-24, 2009York, UK

Outline

1. Introduction to the issue of persistence

2. Who leaves PSE and why?

3. Strategies to improve persistencea) Foundations for Successb) LE,NONET

4. Reflections on experimental research

Introduction to the Issue of Persistence

Introduction to the Issue of Persistence

• Persistence: – the ability of students to continue their post-secondary studies

from one year to the next and ultimately complete their programs

• Access Success– Meaningful access requires that students brought into to PSE

must be successful in their studies

• Low levels of persistence pose a problem... – For students: left without the credential they need– For institutions: poor use of resources and poor performance– For societies:

• lower educational attainment• exacerbates social cleavages

An Increasingly Important Concern

As a recent OECD report puts it:

• “the growing portion of disadvantaged students enrolled in tertiary education makes the ongoing issue of their retention and programme completion an increasingly important concern in tertiary education.”*

* Santiago, Paulo, Karine Tremblay, Ester Basri and Elena Arnal. 2008. Tertiary Education for the Knowledge Society, Volume 2. Paris: OECD.

Persistence: How Much Do We Know?

• “We know very little about how many students drop out of programs, or why” (Rae, 2005)

• Little research on the issue of persistence in PSE in Canada until recently– Previous data on persistence rates limited to institution-specific

studies– Little program evaluation or evidence of what works– New research now becoming available:

• Canada Millennium Foundation pilot projects and program evaluations

The Need for Program Evaluation

“It is ironic that institutions that spend so much time and money insisting on evidence-based decisions, spend so little time on

research that evaluates higher education itself. Research on the postsecondary sector is inadequate and poorly publicized.

This should change.”

Bob Rae, Ontario: A Leader in Learning. Report and Recommendations. Toronto: Government of Ontario, Ministry of Training, Colleges and Universities (2005)

“Presently…there is little evidence about the effects of institutional support programmes on student outcomes.”

Paulo Santiago, Karine Tremblay, Ester Basri and Elena Arnal. 2008. Tertiary Education for the Knowledge Society, Volume 2. Paris: OECD.

The Challenges of Program Evaluation

• Student support programs are notoriously difficult to evaluate because:– Programs are designed and implemented without the

requirements for a viable evaluation in mind– Data that would facilitate evaluation are not available– It is difficult to isolate the effects of the program from

the possible effects of external factors– There is no counterfactual (no reasonable “control”

group against which to compare the outcomes of program recipients)

Who Leaves PSE and Why?

Persistence Rates in Canada

Overall Persistence Rates in Post-Secondary Education in Canada

  Graduated Still in post-secondary education

Discontinued post-secondary education

  (%) (%) (%)

College

Year 1 12.0 75.2 12.9

Year 2 36.9 45.8 17.3

Year3 57.0 25.1 17.9

Year 4 66.2 14.8 19.0

Year 5 73.1 8.8 18.0

University

Year 1 1.1 91.0 7.9

Year 2 3.6 86.7 9.6

Year3 11.2 78.8 9.9

Year 4 45.0 45.2 9.8

Year 5 69.4 20.4 10.2

Source: Finnie and Qiu, Table 6b.

Barriers to Access and Persistence

Malatest (2007), Class of 2003

33%31%

19%

8%

13%11%

22%

14%

29%27%

9%12%

10%

%

5%

10%

15%

20%

25%

30%

35%

Finances Careerindecision

Lack ofinterest

Program notwhat expected

Employment Academicchallenges

Personal/family

Never attended Discontinued

Who Leaves PSE and Why?

Self-Reported Reasons for Discontinuing Post-Secondary Studies

Class of 2003 Study YITS

Lack of interest/lack of program fit/lack of career direction

52% 32%

Financial reasons(other than desire to

work)23% 11%

Academic reasons 14% 6%

Desire to work 11% 7%

Who Leaves PSE and Why?

Factors correlated with low persistence:• Poor academic performance (both secondary and PSE levels)• Low engagement• Inadequate financial aid package or high levels of debt• Uncertainty about career goals• Lower levels of parental education (in some studies)• Aboriginal ancestry• Gender (men are more likely to drop out than women)• Age & family status (older students and students with dependent

children are more likely to drop out)

Percent of PSE Students Aged 24-26 Who Had Discontinued Their Original Stream of PSE *, By Grade Average in High School

38

34

18

11

30

22

14

7

27

0

5

10

15

20

25

30

35

40

59% and less 60%-69% 70%-79% 80%-89% 90%-100%

CollegeStream

UniversityStream

Source: Shaienks, Danielle, Tomasz Gluszynski and Justin Bayard. 2008. Postsecondary Education: Participation and Dropping Out: Differences Across University, College and Other Types of Postsecondary Institutions. Ottawa: Statistics Canada.

* Note: while some of these students will have discontinued their studies, others will have switched streams.

Percent of Post-Secondary Students who Have Discontinued Their Studies (By Age Group)

9

14 14

21

16

23

15

20

0

5

10

15

20

25

18-20 20-22 22-24 24-26

Non-AboriginalStudents

AboriginalStudents*

Source: YITS (Cohort B) – special calculation.* Excludes First Nations youth living on reserve

Who Leaves PSE and Why?

• Those who persist are more likely to attempt more than one program than are those who drop out

• A key difference between those who persist and those who leave is the ability to make adjustments along the way

• Importance of “resilience” (the capacity of overcome obstacles, adapt to change, or to survive and thrive despite adversity)– Those who persist are able to make adjustments that help them to stay

enrolled– Factors contributing to resilience in youth include supportive

relationships with adults and parental expectations

Strategies to Improve Persistence

Foundations for Success

LE,NONET

Research on Policy Responses

• Persistence should be addressed through a comprehensive approach– Need to address interconnected barriers to success – Need to work at institutional level rather than isolated policies

implemented by various departments

• There are many programs designed to improve persistence, but few of these are evaluated for impact

• Student support programs can be implemented in the context of a research endeavour that allows the impact of the program to be assessed effectively – Confirms that programs deliver the intended benefits to students– Confirms that resources are well-spent

Improving access to PSE

Career development activities

Financial incentives

Academic preparation

GRADUATION

Persistence

Academic supportMentoring & cultural

support

Overcoming Barriers to Access and Success

(Millennium Pilot Projects)

Off-campus community supportFinancial support

Community Support

Career development

Tackling the Drop-Out Rates in Community Colleges through Financial

Incentives and Case Management

Foundations for Success

Research questions:

– Do case manager-mediated support services increase the probability of completing a college program?

– Do financial incentives in combination with case manager-mediated support services increase the probability of completing a college program?

Project Participation: At-Risk Factors1) Remedial English/Communications:

• Measure: English Placement assessment results (slight variation within participating colleges)

• Intervention: students are either streamed to regular Communications or developmental course and are encouraged to undertake tutoring and related academic support

2) Mentoring:• Measure: Fast-Track survey question on whether the student self-

identified as someone who would benefit from a mentor.• Intervention: Assignment of a mentor.

3) Career indecision:• Measure: Four Fast-Track career clarity questions, with a cumulative

score greater than 10 (i.e. neutral response to strongly agree/disagree).• Intervention: two “Career Gear” workshops; Myers-Briggs; Strong-

Campbell; one group debriefing; one one-on-one with counsellor (over 2 semesters)

Innovative Features of the Foundations for Success Model

• Post-admissions testing where responses result in actual redirection to existing services

• Case management approach to advise identified at-risk students (one-on-one advisement)

• “Case managers” follow students’ progress for two-years. This involves encouragement, identification of students’ needs and challenges, and redirection to appropriate services.

Random AssignmentNew students in 2-year programs are invited to complete the Fast Track survey, language assessment and sign informed consent, making them eligible for the study

Students deemed at risk on at least one item

Students not deemed at risk(no further involvement in project)

Random Assignment occurs

Program Group 1SERVICES Academic, mentorship and career exploration support

Program Group 2SERVICES PLUSAcademic, mentorship and career exploration support and financial incentives

Program Group 3COMPARISON

Number of Participants per College

Seneca College

Mohawk College

Confederation College TOTAL

Services 404 393 242 1039

Plus 406 393 242 1041

Compensation 387 434 242 1062

PROJECT TOTAL 1197 1219 726 3142

Year 1 ResultsParticipation in FFS Activities

• Service group significantly more likely to participate in any FFS activities compared to Control group

• Service Plus group significantly more likely than Service group to participate in any FFS activities

Source: 2008 College Administration Data, n=2,008.

Participation – Sub-group Analysis

58%63%

82%73%

58%70%

59% 59%61%70%

63%

36%37%

54%

38%

55%64%

43%40%45%51%

43%

Ove

rall

ES

L

Wom

en

Men

Firs

t Gen

erat

ion

29+

Age

Inco

me

Und

er $

25,0

00

Gra

de 1

2 E

duca

tion

orLe

ss

Not

Con

fiden

t in

Suc

ceed

ing

Ski

pped

Hig

h S

choo

lC

lass

Onc

e a

Mon

th +

Hig

h S

choo

l Gra

des

of 6

5or

less

Per

cent

of r

espo

nden

ts

Semester 1 Semester 2

• Higher Semester 1 participation rates among ESL students, first generation, older, low-income and low-confidence students.

• Higher Semester 2 participation rates among ESL students, older students, women, low income and low-confidence students.

Semester 1 GPA

2.132.062.01

Control group Service Group Service Plus Group*

Per

cent

of r

espo

nden

ts

• Service Plus group GPA was higher (2.13) compared to the Control group (2.01)

Source: 2008 College Administration Data, n=2,008, missing GPAs were imputed using a regression model.

Semester 2 GPA

1.961.821.79

Control group Service Group Service Plus Group**

Per

cent

of r

espo

nden

ts

• Service Plus group had a higher GPA in Semester 2 (1.96) compared to the Control group (1.79).

Source: 2008 College Administration Data, n=2,008, missing GPAs were imputed using a regression model.

Student Retention – Beginning of Year 2

67.2%65.8%62.6%

Control group Service Group Service Plus Group*

Per

cent

of r

espo

nden

ts

• Service Plus group significantly more likely to still be in program (67.2%) compared to Control group (62.6%).

• Adjusting for the students that did not participate in any FFS activities, the adjusted effect of the Service Plus treatment is a 6.4% increase in retention one year into students’ college program.

Source: 2008 College Administration Data, n=1,711.

Sub-Group Analysis – Who Gets the Most Benefit from the Interventions?

• Retention: Service Plus ESL students, low-income students, and students with high school grades of 65 or lower show significantly higher retention rates than similar students in the Control group.

74% 73%59%

44%55%60%

ESL students Family Income Lessthan $25,000

High School Grades of65 or Lower

Per

cent

of s

tude

nts

Control group Service Plus Group

Source: 2008 College Administration Data, n=1,711, Cohort 1 only.

Improving the persistence rate of Aboriginal learners at the University of Victoria

“Success after enduring many hardships” (Salish)

LE,NONET

• Pilot project to test the effectiveness of initiatives to improve the retention of Aboriginal students at the University of Victoria

• Project recognizes that improved outcomes necessitates change not only in students but also in the university– “the onus for adjustment” should not be placed solely on Aboriginal

students

• LE,NONET project involves the creation and implementation of a series of programs and support structures:– Student Mentoring– Bursaries (regular and emergency)– Community Internship – Research Apprenticeship– Staff and Faculty Aboriginal Cultural Training (SFACT)

LE,NONET

Research question:

• will a series of interventions involving financial, academic, peer and cultural support have a demonstrable effect on Aboriginal students’ likelihood of completing their program of studies?

• is any particular type of intervention more effective and do the interventions have greater impact when delivered in combination with one another?

The Research Framework

• Research will document the process through which changes are proposed, developed, implemented, and modified at the university, faculty, department and program level over the course of the project

• Since one of the objectives of the program is to change the university culture, it is possible for the program to affect non-participants as well as participants (cannot use random assignment)

– evaluate the direct effects of participation in the various programs offered by the LE,NONET project

– evaluate the indirect effects that such programs might have on students who decline to participate

• Comparison of the persistence of Aboriginal students during the periods before and after the program was introduced

• The LE,NONET approach will allow students to determine the depth of their involvement in the programs — making differing ‘levels of exposure’ to the interventions a key variable of interest.

Participation

• Number of participants (2006 & 2007): 139– Bursaries = 87– Mentor = 17– Mentored = 35– Preparation seminar = 44– Community intership = 24– Research apprenticeship = 17

• Historical cohort: 997

• Non-participants: 728

Interim Findings

• The majority of participants (77%) reported that their participation in LE,NONET program components contributed to the development of their sense of self as an Aboriginal person.

• A strong majority (87%) agreed that their participation in the programs contributed to their sense of connection to the on-campus Aboriginal community.

• Just over half (54%) said that their involvement with LE,NONET also contributed to their sense of belonging at the university.

Interim Findings: Return to School

• Overall, about half of the research participants said that LE,NONET had contributed to their decision to return to school the following year.

• For some students, the financial support they received through the Project made it financially viable for them to afford their tuition and living expenses.

• Other respondents said that returning to school was made easier by the emotional support and sense of community provided by LE,NONET.

Registration Status (Interim): LE,NONET

48%

63%

48%

23% 27

% 31%

23%

10%

21%

0%

10%

20%

30%

40%

50%

60%

70%

Continuing Graduated Withdrawn

Historical Cohort

Participants

Non-Participants

Reflections on Experimental Research

Reflections on Experimental Research

1. These projects show that student support programs can be implemented in the context of experimental research initiatives that will lead to an increase stock of evidence about what may or may not work.

2. Experimental research is possible but not always easy to put in place. Obstacles include:

a) Moneyb) Timec) Confusion about the objectives of the researchd) Tensions between educators and researchers

• Sticking to the script• What is the measure of success?

e) Recruitment of participants

3. All of these obstacles can be overcome, but only when researchers develop effective partnerships with educators and administrators.

Canada Millennium Scholarship Foundation1000 Sherbrooke West, Suite 800Montreal, QC H3A 3R21-877-786-3999

All our publications are available online: www.millenniumscholarships.ca

Andrew Parkinaparkin@bm-ms.org