washington state’s integrated basic education and …...2.3 pace evaluation design and analysis 13...

Washington State’s Integrated Basic

Education and Skills Training (I-BEST)

Program in Three Colleges:

Implementation and Early Impact Report

Pathways for Advancing Careers

and Education

OPRE Report No. 2018-87

September 2018

Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges: Implementation and Early Impact Report

Pathways for Advancing Careers and Education (PACE) OPRE Report No. 2018-87

September 2018

Authors Karin Martinson, Sung-Woo Cho, and Karen Gardiner, Abt Associates Asaph Glosser, MEF Associates

Submitted to: Nicole Constance Federal Project Officer Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services Contract No. HHSP2332007913YC

Project Director: Karen Gardiner Abt Associates 6130 Executive Blvd. Rockville, MD 20852

This report is in the public domain. Permission to reproduce is not necessary. Suggested citation: Glosser, A., Martinson, K., Cho, S.W., and K. Gardiner. (2018). Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges: Implementation and Early Impact Report, OPRE Report No. 2018-87, Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Disclaimer The views expressed in this publication do not necessarily reflect the views or policies of the Office of Planning, Research, and Evaluation, the Administration for Children and Families, or the U.S. Department of Health and Human Services.

This report and other reports sponsored by the Office of Planning, Research, and Evaluation are available at http://www.acf.hhs.gov/programs/opre/index.html.

Sign-up for the ACF OPRE News

E-Newsletter

Like OPRE on Facebook facebook.com/OPRE.ACF

Follow OPRE on Twitter @OPRE_ACF

http://www.acf.hhs.gov/programs/opre/index.html

https://visitor.r20.constantcontact.com/d.jsp?llr=bxgbo4hab&p=oi&m=1108082727649&sit=6tfui4lgb&f=f0ac32d6-a97c-466c-bf3f-9766e5703ded



https://www.facebook.com/OPRE.ACF

https://twitter.com/OPRE_ACF

Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges Implementation and Early Impact Report

PACE

Acknowledgements This evaluation of I-BEST would not have been possible without the support and efforts of multiple individuals, organizations, and agencies. In particular, this study was dependent on the willingness of administrators at Bellingham Technical College, Everett Community College, and Whatcom Community College to participate in the study. They were instrumental in making this study a success, and we applaud their commitment to a rigorous test of the I-BEST model. Equally important were the legions of instructors, advisors, and other college staff who made implementation of I-BEST and the study possible. All of these individuals were also consistently generous with their time in helping us understand their efforts in implementing the program.

We also owe a deep debt of gratitude to the hundreds of individuals who volunteered to participate in the evaluation and shared their experiences with us in surveys and in-depth interviews.

Current and former staff at the Washington State Board for Community and Technical Colleges were valued partners, especially in supporting acquisition of critical college data that informed the evaluation.

We gratefully acknowledge financial support and technical guidance from the U.S. Department of Health and Human Services Administration for Children and Families (ACF). The Contracting Officer’s Representative Nicole Constance played a critical role in guiding the study and provided helpful comments on multiple drafts of this report. We also thank the following current and former ACF staff for their efforts on behalf of the study: Erica Zielewski, Nicole Deterding, Mark Fucello, Naomi Goldstein, Molly Irwin, and Brendan Kelly.

At Abt Associates, a large team contributed to the evaluation. David Judkins and Douglas Walton at Abt provided indispensable assistance in the preparation of technical appendices and the data analysis effort. Correne Saunders analyzed Washington State Board for Community and Technical Colleges student records. Former MEF Associates staff members Emmi Obara and Claire Ma supported site evaluation operations and implementation study data collection efforts. Former Abt staff member Rob Olsen provided insightful comments on an early draft of the report. MEF staff member Mike Fishman co-led the site evaluation operations and provided thoughtful comments on several drafts of this report. We also acknowledge assistance from Bry Pollack in editing the report, and support in production from Charmayne Walker and graphic design from Jon Saunders.


PACE

Abt Associates Contents ▌ pg. i

Contents

Overview ............................................................................................................................... O-i

Executive Summary .............................................................................................................. ES-i

Introduction ................................................................................................................. 1 1.1. Pathways for Advancing Careers and Education (PACE) Evaluation ......................... 2 1.2. Overview of the I-BEST Model ................................................................................... 3 1.3. Research Context for Key Features of I-BEST ............................................................ 6 1.4. Structure of This Report ............................................................................................. 8

PACE Evaluation Design and Data Sources .................................................................... 9 2.1. Career Pathways Theory of Change ........................................................................... 9 2.2. Research Questions for the I-BEST Evaluation ........................................................ 12 2.3. PACE Evaluation Design and Analysis ...................................................................... 13 2.4. Data Sources ............................................................................................................ 22

About I-BEST: Context and Program Design ................................................................ 25 3.1. Local Context ............................................................................................................ 25 3.2. I-BEST Program Design ............................................................................................. 29

Implementation Study Findings .................................................................................. 37 4.1. Implementation Findings ......................................................................................... 37 4.2. Education and Training Participation Patterns ........................................................ 45 4.3. Impact on Receipt of Services .................................................................................. 51 4.4. Summary of Implementation Findings .................................................................... 56

Early Impacts of I-BEST ............................................................................................... 58 5.1. Key Hypotheses and Outcomes ............................................................................... 58 5.2. Impacts on Educational Attainment ........................................................................ 62 5.3. Impacts on Early Career Progress (Secondary Hypotheses) .................................... 70 5.4. Impacts on Psycho-Social Skills, Life Stressors, and Family Structure ..................... 70

Conclusions ................................................................................................................ 73 6.1. Implications of I-BEST Programs .............................................................................. 73 6.2. What Lies ahead for the Evaluation of I-BEST ......................................................... 75

References............................................................................................................................. 77


PACE

Abt Associates Contents ▌ pg. ii

List of Exhibits

Exhibit ES-1: Participation in I-BEST, Credential Receipt, and Participation in Post-I-BEST Courses, by College and Program, within 24 Months after Random Assignment ................................................ ES-vii

Exhibit ES-2: Early Impacts on Credits Earned and Credential Receipt, within 24 Months after Random Assignment (Confirmatory and Secondary Outcomes) ............................................................. ES-ix

Exhibit ES-3: Early Impacts on Enrollment in Occupational Training and Academic Courses, within 24 Months after Random Assignment ..................................................................................................... ES-x

Exhibit ES-4: Early Impacts on the Enrollment and Completion of College-Level Algebra and English ...................................................................................................................................................... ES-xi

Exhibit 2-1: Career Pathways Theory of Change for I-BEST ........................................................................ 10

Exhibit 2-2: Number of Study Participants Randomly Assigned, by College ............................................... 16

Exhibit 2-3: Selected Characteristics of the I-BEST Study Sample .............................................................. 18

Exhibit 3-1: Location of PACE Study Colleges and Programs Included in the I-BEST Study ........................ 26

Exhibit 3-2: Characteristics of Students at Three Colleges and Statewide, 2015-16 .................................. 27

Exhibit 3-3: Comparison of Career Pathways Components Available to PACE Control Group and Treatment Group Members ........................................................................................................................ 28

Exhibit 3-4: I-BEST Programs, Courses, Credits and Certifications, and Next Steps, by College ................. 33

Exhibit 3-5: Examples of I-BEST Educational Pathways .............................................................................. 36

Exhibit 4-1: Percentage of Treatment Group Members Enrolled in I-BEST, by College and I-BEST Program....................................................................................................................................................... 46

Exhibit 4-2: Participation Rates in I-BEST Program for Treatment Group Members and Average Credits Earned and Quarters Attended by I-BEST Participants, by College and Program, within 24 Months of Random Assignment ................................................................................................................. 48

Exhibit 4-3: Credential Attainment and Post-I-BEST College Enrollment Among Treatment Group Members Who Participated in the I-BEST Program, Within 24 Months of Random Assignment .............. 50

Exhibit 4-4: Education or Training Receipt After Random Assignment ...................................................... 53

Exhibit 4-5: Receipt of Various Supports Within 24 Months of Random Assignment ................................ 55

Exhibit 5-1: Outcomes in the Impact Analysis ............................................................................................ 59

Exhibit 5-2: Early Impacts on Education Outcomes (Confirmatory Hypothesis) ........................................ 63

Exhibit 5-3: Early Impacts on Education Outcomes (Secondary Hypotheses) ............................................ 64

Exhibit 5-4: Academic Course Enrollment over Time ................................................................................. 66

Exhibit 5-5: Occupational Training Course Enrollment over Time .............................................................. 67

Exhibit 5-6: Early Impacts on Other Education Outcomes (Exploratory Hypotheses) ................................ 69


PACE

Abt Associates Contents ▌ pg. iii

Exhibit 5-7: Early Impacts on Selected Career Outcomes (Secondary Hypotheses) ................................... 70

Exhibit 5-8: Early Impacts on Other Outcomes (Exploratory Hypotheses) ................................................. 71

List of Boxes

Key Finding ................................................................................................................................... ES-i

Colleges and I-BEST Programs Included in PACE Evaluation ..................................................... ES-iii

Programs in PACE ............................................................................................................................ 2

Observations from a Precision Machining Course—Active Role for Basic Skills Instructor in Traditional Lecture Class at BTC........................................................................................ 40

Observations from a Sustainable Office Skills Course—Integrated Team Teaching at EvCC ....... 41

Observations from an Office Administration Course—Varied Instructional Techniques at WCC ................................................................................................................................... 40

How to Read Impact Tables .......................................................................................................... 52

Definitions of Types of Courses and Credentials .......................................................................... 62


PACE

Abt Associates Overview ▌ pg. O-i

Overview

This report describes the implementation and early impacts of the Washington State Integrated Basic Education and Skills Training (I-BEST) program at three colleges: Bellingham Technical College, Everett Community College, and Whatcom Community College. I-BEST is a nationally known program that aims to increase access to and completion of college-level occupational training in a variety of in-demand occupational areas. Its signature feature is team teaching by a basic skills instructor and an occupational instructor during at least 50 percent of occupational training class time. Colleges operated I-BEST programs in one or more occupational areas including automotive, electrical, office skills, nursing, precision machining, and welding. I-BEST is one of nine career pathways programs being evaluated under the Pathways for Advancing Careers and Education (PACE) study sponsored by the Administration for Children and Families.

The I-BEST program was launched in Washington in the 2006-07 academic year by the Washington State Board for Community and Technical Colleges. I-BEST aims to help students in basic skills programs (e.g., Adult Basic Education, English as a Second Language), who otherwise might have spent time in remediation, to enroll and succeed in college-level occupational training courses. Each I-BEST program is a course of study within a structured career pathway, and it offers students the opportunity to obtain credentials and college credits in in-demand occupations. Besides the team teaching, the I-BEST program evaluated in PACE also included two enhancements: financial support for tuition and associated materials; and additional advising services focused on supporting students’ academic needs, navigating college procedures, and career planning. These enhancements were funded by a grant to Abt Associates from the Open Society Foundations to support program modifications, and are not part of the standard I-BEST program at other colleges in Washington State.

Using a rigorous research design, the study found that the I-BEST programs at the three colleges increased participation in college level courses, number of credits earned and credential attainment. Future reports will examine whether the I-BEST program resulted in gains in employment and earnings.

Primary Research Questions

• Was the intervention implemented as designed?

• How did services received differ between study participants who could access I-BEST programs in the three colleges versus those who could not?

• What were the effects of access to I-BEST on short-term education outcomes including earning credits and credentials?


PACE

Abt Associates Overview ▌ pg. O-ii

Purpose

Low-income workers with only a high school education or less face poor and declining employment prospects. Postsecondary training, often at community colleges, offers one strategy for improving this population’s education and employment opportunities, especially if it is targeted to occupations where there is high and growing demand for skilled workers. How to facilitate a better match between the nation’s need for a skilled workforce and the needs of low-skilled adults for employment is a topic of great interest to policymakers, workforce development organizations, educators, and other key stakeholders.

Career pathways programs are designed to address these issues by providing well-articulated training and employment steps targeted to locally in-demand jobs, combined with a range of supports. Policymakers and practitioners have shown great interest in the career pathways approach. But, to date, limited rigorous research is available on its effects on participants’ educational and economic outcomes. To assess the effectiveness of a career pathways program such as I-BEST, the PACE evaluation uses an experimental design—that is, randomly assigning study participants to a “treatment” group who can access the program and a “control” group who cannot, then comparing their outcomes.

Key Findings & Highlights

• The three colleges implemented I-BEST as designed. The three colleges varied in how they delivered I-BEST across the different occupational programs, but implementation study findings suggest that the program was delivered largely as planned. The combination of the instructional approach (team teaching), advising, and financial supports resulted in a clear contrast between the services available to treatment group members and those available to control group members.

• The program provided access to college-level courses for individuals with low basic skills and education levels. Almost one-third of treatment group members reported having less than a high school diploma or equivalent. Overall, 73 percent of treatment group members offered the I-BEST program participated in it.

• The program had positive impacts on college course enrollment, driven primarily by enrollment in occupational training courses. I-BEST increased college enrollment by 22 percentage points, and increased enrollment in occupational training courses by 41 percentage points.

• I-BEST increased credits earned and credential attainment. Treatment group members received an average of 13 more credits compared to control group members, and there was a 32-percentage point increase in credential receipt.


PACE

Abt Associates Overview ▌ pg. O-iii

Methods

The PACE evaluation’s implementation study examined the design and operation of I-BEST programs at three colleges and students’ participation patterns; its impact study used an experimental design to measure effects on educational and employment outcomes. The PACE evaluation pools results across the three colleges and all occupational training areas.

From November 2011 to September 2014, the PACE evaluation randomly assigned 632 program applicants either to the treatment group (315 students), which could access the I-BEST program, or to the control group (317 students), which could not. Data were collected from college records for a 24-month follow-up period and a follow-up survey conducted approximately 18 months after random assignment. The evaluation also included site visits and monitoring calls to document program implementation and operations. Prior to estimating I-BEST impacts, the research team published an analysis plan specifying key hypotheses and outcome measures, and registered those outcomes.1

1 Abt Associates. 2015. Pathways for Advancing Careers and Education (PACE). Technical Supplement to the

Evaluation Design Report: Impact Analysis Plan. OPRE Report # 2015-100, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education-supplement-evaluation-design-impact-analysis-plan.

https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education-supplement-evaluation-design-impact-analysis-plan



PACE

Abt Associates Executive Summary ▌ pg. ES-i

Executive Summary

Low-skilled workers with only a high school education or less face poor and declining employment prospects. Postsecondary training, often at community colleges, offers one strategy for improving this population’s education and employment opportunities, especially if targeted to occupations where demand for skilled workers is high and growing. How to facilitate a match between the nation’s need for a skilled workforce and the needs of low-skilled adults for entry-level employment and advancement to higher-skilled jobs is a topic of great interest to policymakers, workforce development organizations, educators, and other key stakeholders.

The Integrated Basic Education and Skills Training (I-BEST) Program

This report describes the implementation and early impacts of the Washington State Integrated Basic Education and Skills Training (I-BEST) program at three colleges: Bellingham Technical College (BTC), Everett Community College (EvCC), and Whatcom Community College (WCC). Designed by the Washington State Board for Community and Technical Colleges (SBCTC), I-BEST operates at all 34 public community and technical colleges in the state. It is designed to provide occupational training and basic skills in a structured career pathway for students who have basic skills levels too low to enter college.

Without I-BEST, students whose college entrance test scores were too low to enroll directly in an occupational training program would first have to enroll in Adult Basic Education (ABE) or English as a Second Language (ESL) classes to raise their basic skills to the required levels.

The key elements of the I-BEST program as defined by SBCTC are:

• Focus on living wage jobs. I-BEST programs must incorporate coursework that will qualify workers for jobs with median wages of at least $13 per hour (or $15 per hour in King County, which includes Seattle).

• Basic skills instruction through team teaching and support classes. I-BEST occupational training courses are required to have both an occupational instructor and a basic skills instructor present for at least 50 percent of class time. As a complement to team teaching, some I-BEST programs also include support classes, taught by basic skills instructors, that clarify concepts from occupational training classes, address basic skills

Key Finding Within a 24-month follow-up period, treatment group members were more likely than a randomly assigned control group who could not access the program to • obtain workforce and academic

credits, and • receive a college credential.


PACE

Abt Associates Executive Summary ▌ pg. ES-ii

required to succeed in occupational classes, and help students prepare for assignments and tests.

• Credits. Most I-BEST programs range in length from one to two quarters, and each quarter includes credit-bearing courses. Per SBCTC guidelines, workforce credits are earned in occupational training courses that focus on specific technical skills. For the most part, I-BEST generally did not result in academic credits, which are usually transferable to four-year colleges, including most general education requirements.

• Credentials. Completion of I-BEST courses generally culminated in a workforce award from the college, although further state licensing may be required to practice in the field (e.g., for nursing occupations).2

• Subsequent training and credentials. As I-BEST programs are relatively short by definition, each program also defines a longer-term post-I-BEST educational pathway.

Within this overall approach, individual colleges have flexibility in how they design and operate their I-BEST program, including the occupations targeted.

In addition to these standard I-BEST elements, the three colleges in the PACE evaluation received additional resources for two program enhancements funded by the Open Society Foundations:

• A dedicated advisor available to I-BEST students to assist with guidance on academic issues, navigating the college’s procedures, and career planning.

• “Fill-the-gap” financial support for tuition and course-associated materials that could not be covered by other sources.

Pathways for Advancing Careers and Education (PACE) Evaluation

Abt Associates and its partners are evaluating I-BEST as part of the Pathways for Advancing Careers and Education (PACE) evaluation. Funded by the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services, PACE is an evaluation of nine programs that include key features of a career pathways framework. The framework guides the development and operation of programs aiming to improve the occupational skills of low-income, low-skilled adults, primarily older nontraditional students, by increasing their entry into, persistence in, and completion of postsecondary training. Central to accomplishing these outcomes, the framework describes strategies for overcoming barriers to education and

2 All community and technical colleges governed by SBCTC provide workforce awards (a certificate or an

Associate in Applied Sciences degree requiring more 20 or more credits that are not transferable to a four-year college); workforce completion (a certificate requiring less than 20 credits); and associate’s degrees that require two years of coursework and are transferrable to a four-year college.


PACE

Abt Associates Executive Summary ▌ pg. ES-iii

training that these students can face. Key features of programs within this framework include: a series of well-defined training steps, promising instructional approaches targeted to adult learners, services to address academic and non-academic barriers to program enrollment and completion, and connections to employment.

The I-BEST evaluation includes an implementation study that examines the design and operation of the program and enrolled students’ participation patterns. It also includes an impact study that measures differences in education and employment outcomes using an experimental design; that is, by comparing individuals randomly assigned to a treatment group that could enroll in I-BEST at the three colleges versus a control group that could not.3

The three community and technical colleges in the study offer I-BEST programs in a different set of occupational fields (see box). The colleges were selected due to the size of their I-BEST program and willingness to participate in a random assignment study. They were not selected to be representative of the I-BEST program across all community and technical colleges in the State of Washington. The PACE evaluation of I-BEST pools results across the programs in the three colleges.

Analyses used data from two baseline surveys completed at the time of study entry, an 18-month follow-up survey, 24 months of college records, and site visits. This report provides the results from the implementation study and describes the early impacts of the I-BEST program (18 to 24 months after random assignment) on education, training receipt, credits earned (the confirmatory outcome selected to assess the early effects of I-BEST), and credential receipt, as well as early employment outcomes. Though there are previous evaluations of I-BEST, PACE is the first to use a rigorous random assignment research design to determine a causal link between access to I-BEST and education outcomes.

3 Random assignment ensures that the treatment and control groups will be alike in their observed and

unobserved characteristics, and that any systematic differences in their outcomes can be attributed to the treatment group having access to program services.

Colleges and I-BEST Programs Included in PACE Evaluation Bellingham Technical College • Automotive • Electrical • Nursing Assistant • Precision Machining • Welding Everett Community College • Nursing Assistant • Sustainable Office Skills • Welding Whatcom Community College • Clerical Administration


PACE

Abt Associates Executive Summary ▌ pg. ES-iv

Findings Summary

From the Implementation Study

• I-BEST served individuals with low education and basic skills levels, including 31 percent who had less than a high school diploma or equivalent.

As discussed, I-BEST seeks to increase access to college-level classes for students with low basic skills. Reflecting this approach, the high proportion of study participants who lacked a high school diploma or equivalent reflects the low basic skills level of the population served. More than two-thirds of study participants reported never having attended college, and less than 10 percent reported having an associate’s degree or higher. I-BEST allowed students with lower skills levels to enroll in the college-level occupational training programs that included basic skills instruction.

• Though the details varied across the colleges and I-BEST programs, the three colleges implemented I-BEST team teaching by combining basic skills and occupational technical content.

A hallmark of the I-BEST model is team teaching. As noted above, basic skills instructors must be present for at least 50 percent of occupational training class time. The colleges and instructors had flexibility in how they approached team teaching, including how occupational and basic skills instructors defined their roles within the classroom and planned their time together. The implementation study documented three general approaches:

1. Basic skills instructor as an active student, sitting in class with the students and stopping the occupational instructor to ask clarifying questions or for a concept to be further explained if students were having difficulty with the material. This approach was used often in occupational courses that were primarily lecture-based and highly technical, such as Welding and Machining.

2. Basic skills instructor delivers part of the content during a designated part of the occupational classes. For example, the basic skills instructor might start each class with a discussion of study skills before turning it over to the occupational instructor to present technical information.

3. Basic skills and occupational instructors jointly deliver class content. From the students’ perspective, delivery is seamless; it may not be apparent which instructor is responsible for which type of content. Staff reported this approach was more suitable to I-BEST programs where the occupational focus was not highly technical or specialized, such as Office Skills.


PACE

Abt Associates Executive Summary ▌ pg. ES-v

• The I-BEST teaching teams reported benefiting from experience in working together, but some faced challenges in finding adequate instructional planning time.

Two colleges had teaching teams that worked together for multiple quarters. These teams tended to have clearly defined roles and often co-delivered content. I-BEST staff reported that new teams had a learning curve. For many occupational instructors, I-BEST courses were their first experience teaching low-skilled students, whereas basic skills instructors needed to learn occupational content quickly. Regardless of team tenure, college staff reported that some teaching teams had limited preparation time due in part to the adjunct status of instructors.

• Basic skills instructors provided advising and support.

Basic skills instructors at each of the three colleges provided tutoring and one-on-one academic support, especially early in the academic term when students struggled with course material. Though each of the three colleges offered tutoring services to all students, including those in I-BEST, staff reported that students were often hesitant to engage with tutors and more inclined to seek the assistance of the basic skills instructors whom they already knew.

• The I-BEST programs provided “fill-the-gap” financial assistance and dedicated student advisors (“navigators”) with funding from the Open Society Foundations.4

Many I-BEST students were new to college and had never completed the Free Application for Federal Student Aid (FAFSA) or applied for other forms of financial aid. Navigators helped new students apply for financial aid for which they were eligible. The program then provided “fill-the-gap” funding if the financial aid was not sufficient. Colleges also used the funding to cover training-related expenses such as books.

Navigators also provided proactive advising to I-BEST students regarding class progress, potential barriers to participation, and career and academic planning. Instructors and administrators at one college characterized the navigator as a “one-stop shop” who served as both a coordinator and guide for I-BEST participants. Another administrator described the navigator role as an “early alert system” to proactively identify potential issues that could inhibit student success.5

4 Programs in PACE could receive funding from OSF to enhance program services. OSF made a grant to Abt

Associates for this purpose, and funds were allocated by Abt to programs that proposed modifications. 5 Each of the colleges had advising services available to I-BEST students prior to the start of PACE. However,

these services were less comprehensive than what was available to treatment group members during the study. Feedback from staff from Washington’s State Board for Community and Technical Colleges indicates that more comprehensive navigation services are now a core component of I-BEST.


PACE

Abt Associates Executive Summary ▌ pg. ES-vi

• The I-BEST program did not offer structured employment services, but some instructors provided informal employment and job search assistance.

SBCTC designed the I-BEST program with an explicit focus on occupational training that could increase the earnings potential of participants. However, none of the three colleges had employment supports specific to I-BEST. Administrators at one college noted that the lack of employment services was a weakness of the I-BEST program, whereas another administrator indicated that the lack of such specific employment services reflected the focus on continued training and credentials.

Although none of the three colleges had dedicated I-BEST employment services, staff reported that instructors often provided informal, individualized assistance to students as requested. Some I-BEST instructors came from the relevant industry and knew employers in the local labor market. One Welding instructor, for example, stayed informed of employers who were likely to hire entry-level welders and provided applications to students and coached them on the application process.

• Recruiting students for I-BEST was challenging.

For the PACE study, the three colleges collectively planned to enroll 1,000 study participants to be equally divided between the treatment and control groups. Based on past experience, college administrators expected a high demand for occupational training among their currently enrolled basic skills students (ABE and ESL), and several I-BEST programs included in the study traditionally had waiting lists. In addition, colleges expanded their outreach to community-based organizations, employers, the state’s Temporary Assistance for Needy Families (TANF) program, and the local media. They also reached out to students whose college entrance scores were high enough to enroll in the standard college occupational classes but whose basic skills levels were in the I-BEST range. Despite efforts to increase recruitment, the three colleges struggled to recruit participants for the study and ultimately recruited two-thirds of the sample goal (632 study participants in total).

• Across the three colleges, almost three-quarters of treatment group members participated in an I-BEST program.

Seventy-three (73) percent of treatment group members participated in at least one I-BEST course. Exhibit ES-1 shows the proportion of the treatment group that participated in an I-BEST course and, for those who participated, the percentages who obtained a credential and enrolled in additional courses after I-BEST within a 24-month follow-up period.


PACE

Abt Associates Executive Summary ▌ pg. ES-vii

Exhibit ES-1: Participation in I-BEST, Credential Receipt, and Participation in Post-I-BEST Courses, by College and Program, within 24 Months after Random Assignment

College/Program

Participated in an I-BEST Program

Obtained Workforce Credential

Enrolled in Additional Courses after I-BEST

Among Those Randomly Assigned

Among Those Who Participated in a I-BEST Program

Bellingham Technical College Automotive (N=3) 100% 33% 67% Electrical (N=13) 87% 31% 77% Nursing Assistant (N=57) 71% 70% 53% Precision Machining (N=6) 60% 83% 50% Welding (N=39) 81% 38% 72% Subtotal (N=118) 76% 55% 62% Everett Community College Nursing Assistant (N=20) 61% 80% 55% Sustainable Office Skills (N=36) 86% 61% 67% Welding (N=22) 48% 95% 55% Subtotal (N=78) 65% 76% 60% Whatcom Community College Clerical Assistant (N=38) 89% 29% 76% Total (all colleges and programs) 73% 58% 63% SOURCE: SBCTC college records. NOTES: Sample size for column one is 315; sample size for columns two and three is 230 and includes all individuals who attended at least one team-taught I-BEST program.

As shown in the first column, almost 90 percent of WCC treatment group members enrolled in an I-BEST course, whereas 76 percent of BTC students and 65 percent of EvCC students did so. Participation rates were above 80 percent for the Automotive, Electrical, and Welding programs at BTC; the Sustainable Office Skills program at EvCC; and the Clerical Assistant program at WCC. The lowest rate (48 percent) was in the I-BEST Welding program at EvCC.

• Fifty-eight (58) percent of treatment group members who participated in I-BEST obtained a workforce credential, and almost two-thirds enrolled in subsequent education and training.

Each college awarded a college-issued certificate for the completion of its I-BEST programs. As shown in the second column on Exhibit ES-1, 58 percent of treatment group members who participated in I-BEST received a credential within the 24-month follow-up period, with the majority being workforce awards (20 credits or more; not shown).

Credential receipt varied by college and occupational area. More than three-quarters of EvCC students (76 percent) obtained a credential, compared with 55 percent of BTC students and 29 percent of WCC students. Credential receipt rates were above 80 percent for treatment group members who enrolled in the BTC’s Precision Machining and EvCC’s Nursing Assistant and Welding programs. EvCC had notably higher credential receipt rates than comparable programs at the other colleges for students enrolling in its Welding program (95 percent for EvCC versus 38 percent for BTC) and in its clerical program (61 percent for EvCC versus 29 percent for WCC).


PACE

Abt Associates Executive Summary ▌ pg. ES-viii

The third column of Exhibit ES-1 shows that a high proportion of treatment group members who participated in an I-BEST program continued their education beyond I-BEST, with 63 percent enrolling in additional college courses after I-BEST. This rate was similar at BTC and EvCC (about 60 percent) but higher at WCC (76 percent). On average, those who attended an I-BEST program were enrolled in college courses for four quarters—longer than the I-BEST portion of the occupational training lasted. Approximately one-quarter were still enrolled in college at the end of the 24-month follow-up period (not shown).

The variation in credential receipt, as well as the progression to additional education and training (in some cases, a greater proportion than those who received a credential through I-BEST), is likely due to I-BEST students not completing all of the I-BEST program course requirements but continuing to higher or perhaps different courses of study.

From the Impact Study

• I-BEST had a positive impact on the number of academic and workforce credits earned at colleges (confirmatory outcome).

As Exhibit ES-2 shows, I-BEST had a 13-credit impact on academic and workforce credit completion, which was the confirmatory outcome for the impact analysis. Both types of credits are college-level credits (i.e., non-remedial and applicable towards credentials), and the accumulation of these credits is a positive indicator of academic progress, either towards a credential or, for academic credits, towards transferring to four-year programs in the future. However, the majority of the credits (84 percent) earned were workforce credits (not shown).

The impact on credits earned was driven primarily by more treatment group members enrolling in college than control group members, and to a lesser extent by treatment group members who attended college earning more credits than control group attendees. These results indicate engagement in college-level courses for many of these students would not have otherwise been possible, because they did not meet the college entry basic skill requirements.


PACE

Abt Associates Executive Summary ▌ pg. ES-ix

Exhibit ES-2: Early Impacts on Credits Earned and Credential Receipt, within 24 Months after Random Assignment (Confirmatory and Secondary Outcomes)

SOURCE: SBCTC records.

• I-BEST produced large positive impacts on credential completion, particularly workforce awards.

As Exhibit ES-2 shows, I-BEST increased the completion of any credential from a SBCTC college within 24 months after random assignment by more than 32 percentage points, with more than 44 percent of the treatment group receiving some type of credential within the 24-month follow-up period compared with only 12 percent of the control group (based on SBCTC college records). The majority of the students completed a workforce award, a categorization that includes either a certificate that required more than 20 total workforce credits to complete or an Associate’s in Applied Science (AAS) degree. I-BEST also increased by four percentage points completion of short-term workforce completion certificates, which includes any certificate that requires less than 20 total workforce credits to complete.

The impact on credentials completed was driven primarily by a higher proportion of treatment group members who enrolled in college receiving a credential than did the control group members who enrolled in college. Among those who enrolled in college, 54 percent of treatment group members received a credential from a college, whereas 28 percent of control group members did so (not shown).


PACE

Abt Associates Executive Summary ▌ pg. ES-x

• I-BEST had large positive impacts on college course enrollment, driven primarily by enrollment in occupational training courses within the first six months after random assignment.

As Exhibit ES-3 shows, I-BEST increased college enrollment by 22 percentage points, with close to 90 percent of the treatment group enrolling in college compared with 68 percent of the control group. As shown in the exhibit, I-BEST increased enrollment in occupational training courses by 41 percentage points (81 percent for treatment group members compared with 40 percent for control group members over the 24 months—a 100 percent increase). The large impact on occupational training course enrollment is driven by treatment group members enrolling in I-BEST courses, whereas the control group members would likely take remediation classes first. The treatment group members could earn workforce credits for completing I-BEST courses. I-BEST also increased academic course enrollment by six percentage points.

Exhibit ES-3: Early Impacts on Enrollment in Occupational Training and Academic Courses, within 24 Months after Random Assignment


The impact in occupational training class enrollment is at its largest in the first six months after random assignment, with 73 percent of the treatment group enrolling in an occupational training course compared with only 22 percent of the control group (not shown). The impacts on cumulative occupational training course enrollment decrease over time, where eventually the treatment and control groups are roughly equal in months 19-24 after random assignment (not shown).

• There was a positive impact on the enrollment and completion of college-level algebra and English courses.

As Exhibit ES-4 shows, I-BEST increased enrollment in and completion of college-level algebra and English, both of which are credit-bearing academic courses and are often considered to be


PACE

Abt Associates Executive Summary ▌ pg. ES-xi

“gateway” courses for advancement into higher-level college courses. For algebra, I-BEST increased enrollment by 10 percentage points and completion by nine percentage points; for English, I-BEST increased enrollment by nine percentage points and completion by six percentage points. These findings are encouraging in terms of advancement in college coursework, since these algebra and English courses serve as prerequisites for many other courses required as part of two-year associate degrees and are often transferrable to four-year colleges.

Exhibit ES-4: Early Impacts on the Enrollment and Completion of College-Level Algebra and English


• I-BEST had statistically significant impacts on receipt of advising and employment services.

Reflecting the dedicated advisor that was part of I-BEST, the program had a 14 percentage point impact on receipt of career counseling (36 percent of the treatment group members versus 22 percent of control group members, statistically significant at the one percent level). There were similar impacts on receipt of help arranging supports for school, work, or family (29 percent of the treatment group versus 16 percent of the control group, statistically significant at the one percent level). Though there were no official job search assistance services, I-BEST produced an impact on receipt of job search or placement services (29 percent of the treatment group versus 19 percent of the control group, statistically significant at the one percent level), potentially reflecting the assistance received from instructors or the advisor. (See full results in Chapter 4, Exhibit 4-5).


PACE

Abt Associates Executive Summary ▌ pg. ES-xii

Implications

Because of the interest at the federal, state, and local levels in the I-BEST as a strategy to improve education and employment outcomes for low-skilled adults, replications of the model are already underway in several states and localities. These results have a number of implications for further development of related initiatives.

• Initial findings from the PACE evaluation provide strong causal evidence of the effect the I-BEST program can have on education outcomes. The PACE evaluation provides the first rigorous evidence on I-BEST from a range of occupational programs at three of the state’s 34 community and technical colleges. Though not representative of all I-BEST programs statewide, these results greatly strengthen the evidence that I-BEST can produce increases in receipt of credits and credentials. Future reports will provide information on employment effects at later follow-up points.

• Though the I-BEST programs themselves last one to two quarters, many students used them as a launching pad for additional college-level education and training. A high proportion of treatment group members who attended I-BEST programs continued their education beyond I-BEST, with 63 percent enrolling in additional college courses after I-BEST. On average, those who attended an I-BEST program were enrolled in college courses for four quarters—longer than the I-BEST portion of the occupational training lasted. Approximately one-quarter were still enrolled in college at the end of the 24-month follow-up period.

• Maintaining a connection between occupational and academic programs may be important for students interested in pursuing higher levels of education that may result in higher-paying jobs. Most of the credits earned and credentials received by I-BEST participants were for workforce rather than academic courses, meaning they are not transferable to four-year colleges. But there is some evidence of students moving to academic courses of study. Primarily through enrollment in courses after I-BEST, the program increased enrollment in and completion of college-level algebra and English, both of which are credit-bearing academic courses and are often required as part of two-year associate degrees and transferrable to four-year colleges.

• The team teaching approach benefited from planning and durability in teaching teams. Staff reported variations in how team teaching was operationalized and challenges the teams faced. To effectively implement I-BEST the instructors reported that they needed to adapt to the learning styles of students with lower basic skills as well as develop instructional approaches that effectively integrate a second instructor. Overall, the study indicates the value of planning activities to help instructors define and integrate their instruction and importance of the tenure of the instructors in working


PACE

Abt Associates Executive Summary ▌ pg. ES-xiii

together as team. In addition, several of the programs experienced challenges using staff who were part-time and adjunct, which is common in community college programs, because it limited the planning time available to the teaching teams.

• Although the I-BEST programs operated largely as designed, the implementation experience showed areas for further attention and development. First, although the program sought to target low-skilled individuals, colleges had some difficulty recruiting students. Second, connections to employment and jobs were not a formal component of the program in most of the career areas. Finally, study findings on the impacts of student advising and supports indicate that the dedicated student advisors and “fill-the-gap” financial assistance—resources not typically available to I-BEST students but available during the study through a PACE-related foundation grant—may be a contributing factor to the program effects observed.

Next Steps

The next I-BEST report will cover a 36-month follow-up period. It will focus on the effects of the program on students’ economic outcomes for the period when these are expected to occur. The report will examine employment outcomes, such as average employment and earnings over successive follow-up quarters, and job characteristics. Thus, it will begin to answer whether the services provided by I-BEST translate into economic gains in the workplace in the longer term. An analysis at 72 months after random assignment will estimate long-term effects on earnings of the program.

The many important questions that remain to be addressed in subsequent reports include:

• Will I-BEST’s impact on educational attainment remain stable, increase, or decrease?

• Will I-BEST’s impact on educational attainment translate into impacts on employment and earnings?

• Does I-BEST have other impacts on participants and their families?

• Is I-BEST cost beneficial?


PACE

Abt Associates 1. Introduction ▌pg. 1

Introduction

Workers with only a high school education or less face poor and declining employment prospects (Georgetown University Center on Education and the Workforce 2016; Pew Research Center 2014). Postsecondary training, often at community colleges, offers one strategy for improving this population’s employment opportunities, especially if it is targeted to occupations where demand for skilled workers is high and growing (Capelli 2014; Conway and Giloth 2014; Holzer 2015). How to meet the nation’s need for a skilled workforce and the needs of low-income adults for entry-level employment and advancement to higher-skilled jobs is a topic of great interest to policymakers, workforce development organizations, educators, and other key stakeholders.

Research indicates that meeting both needs is not easy. Many low-income, low-skilled adults face considerable barriers to completion of postsecondary education. Many are “nontraditional” students—that is, often they are older, are parents, lack adequate basic academic skills, and have few economic resources (National Center for Education Statistics 1996). Further, on average, nontraditional students fare worse in postsecondary settings than do traditional students (Visher et al. 2008; Cooper 2010; Goldrick-Rab and Sorensen 2010). Institutions often assign students who need to improve their basic academic skills to developmental (remedial) education, courses that do not result in college credits. Many of the students requiring remediation never progress beyond it as the amount of remedial coursework makes the transition to college-level coursework difficult (Bailey, Jeong, and Cho 2010; Rutschow and Schneider 2011). Others drop out due to financial setbacks or difficulties juggling school, work, and family responsibilities. Some have difficulties navigating the college environment, including course sequences and financial aid (Karp 2011). Many have difficulty meeting academic standards (Bridges to Opportunity Initiative 2008).

This report describes the implementation and early impacts of the Washington State Integrated Basic Education and Skills Training (I-BEST) program at three colleges: Bellingham Technical College, Everett Community College, and Whatcom Community College. I-BEST is a nationally known program that aims to increase students’ access to and completion of college-level occupational training in a variety of in-demand occupational areas.

I-BEST aims to quickly teach students basic, occupational, and college-readiness skills so they can move through college and into living wage jobs faster. The program uses a team teaching approach. Students work with two instructors in the classroom: one provides job-training and the other teaches basic skills in reading, math, or English (SBTC 2017).


PACE


Abt Associates and its partners are evaluating the I-BEST program as part of the 10-year Pathways for Advancing Careers and Education (PACE) evaluation.6 The evaluation includes both an implementation study to examine I-BEST’s design and operation and an impact study to estimate the impacts of access to I-BEST on its students’ education and training, employment, and other outcomes within an 18- to 24-month follow-up period.7 Although PACE is not the first study of I-BEST’s impacts (Zeidenberg, Cho and Jenkins 2010; Jenkins, Zeidenberg, and Kienzl 2009), it is the first to use a rigorous random assignment research design.

This chapter describes the PACE evaluation, provides an overview of the I-BEST program, summarizes findings from previous I-BEST evaluations, and provides a roadmap to the rest of the report.

1.1. Pathways for Advancing Careers and Education (PACE) Evaluation

PACE is funded by the by the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services. It is the first large-scale, multi-site experimental evaluation of programs that include key features of a “career pathways framework.” This framework articulates signature strategies for overcoming the barriers that nontraditional, occupational students often face. For example, strategies typical of programs within this framework include having a series of well-defined training steps, promising instructional approaches, supportive services, and connections to employment (Fein 2012). The career pathways framework is flexible, however, and not a specific program model. Thus, which components a local program adopts and how it implements them can vary greatly.

Reflecting this diversity, each of the nine programs in the PACE evaluation represents a different program model (see box). All share some program

6 For more information on the PACE study, go to www.acf.hhs.gov/opre/research/project/pathways-for-

advancing-careers-and-education. 7 The time frame was selected because completion of the 15-month follow-up survey occurred on average 18

months after random assignment and college records were available for the full sample for 24 months post random assignment.

Programs in PACE • Bridge to Employment in the Health Care Industry

at San Diego Workforce Partnership, San Diego, CA

• Carreras en Salud at Instituto del Progreso Latino, Chicago, IL

• Health Careers for All at Workforce Development Council of Seattle-King County, Seattle, WA

• Integrated Basic Education and Skills Training (I-BEST) program at three colleges (Bellingham Technical College, Everett Community College, and Whatcom Community College), Washington State

• Pathways to Healthcare at Pima Community College, Tucson, AZ

• Patient Care Pathways Program at Madison College, Madison, WI

• Valley Initiative for Development and Advancement (VIDA), Lower Rio Grande Valley, TX

• Workforce Training Academy Connect at Des Moines Area Community College, Des Moines, IA

• Year Up (Atlanta, Bay Area, Boston, Chicago, National Capital Region, New York City, Providence, Seattle)

http://www.acf.hhs.gov/opre/research/project/pathways-for-advancing-careers-and-education



PACE


components that are part of the career pathways framework; but each also has distinct and unique elements, reflecting the target populations, occupational trainings offered, and industries of focus. Because of this variation, PACE evaluates and reports findings for each program individually.8

The central goal of the PACE evaluation is to determine the effectiveness of each of the nine programs using a common evaluation design and conceptual framework (impact study). The most critical element of the evaluation design is random assignment of eligible applicants either to a treatment group that can access the career pathways program or to a control group that cannot. A randomized control design ensures that the study’s treatment and control groups will be equivalent in their observed and unobserved characteristics—and thus, that any systematic differences in their subsequent outcomes can be attributed to the treatment group having access to program services (i.e., the program’s impacts). Systematic differences in outcomes due to the characteristics of individual members in each group can be ruled out.

Consistent with this career pathways framework and the career pathways theory of change (described in Chapter 2) guiding the PACE evaluation, the key outcomes for which the PACE study estimates effects are in two areas—education and training and employment—although the study also estimates effects in other areas, such as individual and family well-being.

The PACE evaluation implementation and early impact program reports analyze outcomes over a follow-up period of about 18 to 24 months after random assignment. The impact analyses rely primarily on surveys and college records for study participants in the treatment group and control group. Future reports will analyze outcomes three years and six years after random assignment.9 These future reports will also include benefit-cost studies for some of the nine PACE programs.

1.2. Overview of the I-BEST Model

Developed by the Washington State Board for Community and Technical Colleges (SBCTC), the I-BEST program has been operating in all 34 public community and technical colleges in the state since the 2006-07 academic year. The I-BEST model grew out of a concern that adult basic skills students—adults who lack high school-level skills—were not advancing beyond basic skills classes to college-level occupational programs, and therefore were not earning credentials. In Washington State, adult basic skills programs that are offered at two-year colleges, including

8 All PACE-related reports can be found on www.career-pathways.org as well as

www.acf.hhs.gov/opre/research/project/pathways-for-advancing-careers-and-education. 9 These future reports will be developed as part of the Career Pathways Intermediate Outcomes and the Career

Pathways Long-term Outcomes studies, respectively.

http://www.career-pathways.org/



PACE


English as a Second Language (ESL) and Adult Basic Education (ABE), serve approximately 60,000 students per year (Wachen, Jenkins, and Van Noy 2010).

One influential study (Prince and Jenkins 2005) found that only 13 percent of students who enrolled in ESL programs and 30 percent of students who entered ABE programs continued on to earn any college credits. Very few (four to six percent of each group) earned the credit equivalent of two full-time semesters or a certificate within five years. However, ESL and ABE students who did complete a year of credits and a credential earned on average $7,000 and $8,500 per year more, respectively, than those who did not. This one-year threshold of credits and credential completion is known as the “tipping point,” a marker by which students are more likely to have successful longer-term outcomes such as employment. However, there is no rigorous evidence that indicates that the tipping point causes these outcomes to occur. Perhaps it is the more determined and academically ready students who were already likely to have successful outcomes who reach this tipping point.

SBCTC funded a pilot at 10 of Washington’s community and technical colleges in the 2004-05 academic year to increase the rate of transition of ABE and ESL students to college-level occupational training programs. It rolled out the program to all 34 colleges over the next two academic years.

Core Components of the I-BEST Standard Model

SBCTC created the I-BEST model with the following core components:

• Courses are part of a structured career pathway, defined by SBCTC as “a sequence of courses that leads directly to a postsecondary credential and to jobs that are in demand in the local labor market.” Pathways ensure students do not have to “find their way on their own” (SBCTC 2005).

• A team teaching instructional model pairs basic skills instructors and occupational instructors for at least 50 percent of occupational training class time.

• Enhanced funding from SBCTC reimburses colleges 1.75 times the regular rate for a full-time equivalent (FTE) student to help cover the costs associated with implementing I-BEST, including development of a curriculum, instructor preparation, and supportive services such as a dedicated program coordinator.

Within this I-BEST model, the colleges have flexibility in creating a new program. A college must complete a comprehensive application that details the local demand for that specific occupational field, as well as a roadmap to the full educational pathway for that program.


PACE


PACE Enhancements to the I-BEST Model

In addition to these core components, the three colleges in the PACE study received additional funding for program enhancements from the Open Society Foundations. The enhancements were available only to I-BEST students while they were members of the PACE study’s treatment group:

• Dedicated advising. Each college had a dedicated advisor (“navigator”) to provide students with guidance on academic issues, navigating the college’s procedures, and career planning.

• “Fill-the-gap” financial support for training and associated materials beyond typical sources available to any student. For students who were not able to secure funding through Pell Grants, Washington State Opportunity Grants, or other sources, the three colleges covered the tuition costs. They also provided funds for books, tools, other course materials, or transportation.

Replication of I-BEST

There have been several efforts in other states and localities to replicate the I-BEST program or some of its key components such as team teaching. For example, Maryland has implemented the I-BEST model in some of its community colleges, and Mississippi has implemented it statewide.10,11 Other efforts include the Breaking Through program, which works with 41 community colleges in 22 states.12 Like I-BEST, Breaking Through provides low-skilled students with accelerated, contextualized training in in-demand fields, plus comprehensive supports.

Yet another effort to replicate the I-BEST model, Accelerating Opportunity, operates at the state-system level rather than the college level. It aims to transform basic skills education in five states.13 The program emphasizes support through a “navigator” or “coach” more than I-BEST itself does. A recent study in four of the five states used non-experimental design to determine impacts of Accelerating Opportunity on academic and labor market outcomes (Anderson et al. 2017). It found mixed evidence on the effectiveness of this program, with only one of the four study states found to have mostly positive impacts for both types of outcomes. Rigorous research documenting replications of I-BEST has not yet been released.

Prior Research on I-BEST

Previous study of I-BEST suggests that the program can improve outcomes such as credential and credit completion. Two often cited studies from the Community College Research Center

10 See http://insidemc.montgomerycollege.edu/details.php?id=25781. 11 See http://news.hindscc.edu/index.php/hinds-cc-mi-best-program-focus-of-hearing-before-lawmakers/. 12 See http://www.jff.org/initiatives/breaking-through. 13 See http://www.jff.org/initiatives/accelerating-opportunity.

http://insidemc.montgomerycollege.edu/details.php?id=25781

http://news.hindscc.edu/index.php/hinds-cc-mi-best-program-focus-of-hearing-before-lawmakers/

http://www.jff.org/initiatives/breaking-through

http://www.jff.org/initiatives/accelerating-opportunity;%20http:/www.urban.org/research/publication/new-evidence-integrated-career-pathways


PACE


(CCRC) used system-wide administrative data from SBCTC and non-experimental quantitative methods to determine impacts, but they are not as rigorous as a well-implemented randomized study.

Zeidenberg, Cho, and Jenkins (2010) estimated the impact of I-BEST among basic skills students who had enrolled in at least one career or technical education course on their own. Using the same administrative data collected from SBCTC on individual students, their transcripts, and their credentials, the authors used two different methods—a difference-in-differences approach in addition to a separate propensity score matching approach—to determine the impact of I-BEST on outcomes. This research found that I-BEST had a positive impact on college credit accumulation and on basic skills, and mixed findings on credential completion. These results replicated largely similar findings from an earlier study (Jenkins, Zeidenberg, and Kienzl 2009). Neither study found a positive impact on wages nor hours worked, which may have been due to students entering the job market during a major recession.

Though these two studies used rigorous, non-experimental quantitative methods, the resulting impact estimates might suffer from selection bias – the more motivated students may have elected to enroll in I-BEST. This possibility highlights the potential contribution of new evidence based on random assignment, which protects against selection bias.

1.3. Research Context for Key Features of I-BEST

Though previous evidence on the impact of I-BEST on outcomes is limited, numerous studies exist on its major components.

Improving Basic Skills

There is considerable evidence that the usual approach of requiring remediation before allowing students to enroll in college-level coursework does not help them complete college coursework. Most students referred to developmental education to raise their skills to college standards never enroll in college-level courses (Bailey, Jeong, and Cho 2010). Research using the National Education Longitudinal Survey found that fewer than half of students referred to developmental coursework even complete the required sequence of developmental courses (Bailey et al. 2010). However, there is little evidence about whether the alternative that I-BEST offers is more effective. I-BEST is an acceleration strategy that compresses the time required to succeed in an occupational program by simultaneously providing basic skills training and occupational training. There is some evidence that compressing developmental education into shorter periods can improve outcomes for low-skilled students, but there is no rigorous evidence of its effectiveness (Rutschow and Schneider 2011).


PACE


Contextualization

I-BEST delivers basic skills content in a contextualized manner, using materials and examples from the student’s occupational field of interest. Advocates for contextualization argue that it can improve student motivation and success, because students may be more likely to engage with basic skills materials that use occupational subject matter that interests them. The evidence on the effectiveness of contextualization among adult learners is mixed, with very few randomized studies at the college level and with some studies finding limited effects (Perin 2013; 2011). However, non-experimental evidence does suggest the possibility of impacts on college readiness and progress toward vocational certificates (Bragg and Barnett 2009).

Enhanced Advising

The I-BEST model at the three colleges included more individualized advising and navigational support than is typically available at two-year colleges in Washington State. This advising was designed to ease students’ transition to college-level classes and to proactively address any academic or non-academic barriers that emerged while students were enrolled.

Evidence from random assignment evaluations has demonstrated that enhanced counseling and more personalized advising can increase the number of courses passed and credits completed (Bettinger and Baker 2011; Scrivener and Weiss 2009.) The City University of New York’s Accelerated Study in Associate Programs (ASAP) has had a strong and convincing impact on associate’s degree completion, and the program strongly emphasizes advising. However, it is not certain whether the advising is responsible for these results, as it is only one component of the program (Scrivener et al. 2015). Some other programs in PACE, notably Valley Initiative for Development and Advancement (VIDA), also strongly emphasize advising among other components; early results from VIDA are promising (Rolston, Copson, and Gardiner, 2017). Project QUEST, a program similar to VIDA that provides comprehensive support through community colleges to help individuals complete occupational training programs, has also seen promising results through a randomized research design (Elliott and Roder 2017).

Added Financial Support for Training and Related Costs

In their CCRC study, Jenkins, Zeidenberg, and Kienzl (2009) found that I-BEST students were more likely to receive financial aid than were students enrolled in career and technical education classes but not in I-BEST. During the PACE study, the program at the participating colleges provided additional financial supports over and above those provided by Washington State that were funded by the Open Society Foundations (OSF), to ensure that lack of financial aid would not be a barrier to enrollment or program completion.14 A large body of evidence indicates that insufficient resources are a barrier to entry and completion of education and 14 Programs in PACE could receive funding from OSF to enhance program services. OSF made a grant to Abt

Associates for this purpose, and funds were allocated by Abt to programs that proposed modifications.


PACE


training for low-income students, and that financial assistance can increase postsecondary attendance and persistence (Deming and Dynarski 2010; Dynarski and Scott-Clayton 2013).

Similarly, OSF funds covered costs associated with transportation, books, and other school supplies, as well as testing fees and certification exams. The Opening Doors Demonstration, which had positive effects on student outcomes, included financial supports as part of the program model (Scrivener and Weiss 2009), though the contribution of those financial supports to the program’s positive effects is uncertain. The Performance-Based Scholarship Demonstration (Mayer et al. 2015) included a range of academically incentivized financial aid supports and had modestly positive effects on students’ academic progress and degree completion outcomes.15

1.4. Structure of This Report

The organization of the remainder of this report follows:

• Chapter 2 presents the PACE evaluation as applied to I-BEST, beginning with its conceptual framework and research questions; details the evaluation design; describes the study sample; and summarizes the evaluation’s data sources.

• Chapter 3 describes the context and design of I-BEST in the three colleges.

• Chapter 4 describes the implementation study findings, including training programs and instructional approaches, participation in training and comparisons of participation in education and training across the treatment and control groups, academic and nonacademic advising, employment supports, and financial assistance provided by the program.

• Chapter 5 presents the impact study’s early findings, focusing on two main impacts—hours of training, and credentials received over a 24-month follow-up period—but also looking at other career and life outcomes.

• Chapter 6 summarizes the implementation and impact findings and discusses their implications for the PACE study in the longer term.

The appendices provide additional details about baseline data (Appendix A); college records data, a key source for the impact study (Appendix B); survey-based outcomes (Appendix C); and the approach to outliers (Appendix D).

15 See https://www.mdrc.org/sites/default/files/designing_scholarships_FR.pdf.

https://www.mdrc.org/sites/default/files/designing_scholarships_FR.pdf


PACE

Abt Associates 2. PACE Evaluation Design and Data Sources ▌pg. 9

PACE Evaluation Design and Data Sources

This chapter describes the larger PACE evaluation design and its application to I-BEST. It begins with a discussion of the PACE career pathways theory of change and the research questions that the theory of change implies. It then briefly describes the evaluation design and analysis procedures for the impact study, including the random assignment process and the outcome of that process. A brief description of the implementation study analysis follows.16 Finally, the chapter summarizes the main data sources for the implementation and impact studies.

2.1. Career Pathways Theory of Change

The career pathways theory of change guides both the implementation study (that is, it identifies which aspects of program services are expected to affect outcomes) and the impact study (that is, it identifies which outcomes the program is expected to affect). The theory of change also generates key hypotheses about the direction of expected effects that the impact evaluation will test for statistically significant change.17 In addition, the theory of change implicitly assumes time horizons by which the program is expected to have effects, and thus the theory determines the key outcomes at any particular time of follow-up.

Exhibit 2-1 depicts the PACE career pathways theory of change, as applied to I-BEST in the three participating colleges. It shows how a program (inputs) is hypothesized to produce effects on intermediate outcomes, which in turn will lead to effects on main outcomes. Effects on intermediate outcomes are expected to occur earlier than effects on main outcomes, but the exact timing depends on particular features of the program, such as the length of occupational training and what, if any, steps precede it. In addition, because effects on intermediate outcomes may persist over time, the study will also measure them at later points in time.

16 The research team developed a detailed evaluation design report and analysis plan for the PACE evaluation

(Abt Associates 2014, 2015). 17 The implementation study describes the set of services that students in the treatment group experienced. In

addition to descriptive statistics, it includes a small number of impact estimates that show the difference in services received between treatment and control group members. The impact study focuses solely on estimates of the effects of the program on intermediate and main outcomes.


PACE


Exhibit 2-1: Career Pathways Theory of Change for I-BEST


PACE


As shown in Exhibit 2-1, starting in the box at the left, the career pathways theory of change begins with two types of program inputs:18

• Organization. Organizational inputs include the three colleges, program standards and funding from SBCTC, and supplemental funding from the Open Society Foundations.

• Participants. This individual input includes the characteristics of the target population, such as ABE and ESL students scoring in the specified range on the Comprehensive Adult Student Assessment System (CASAS) assessment and their interest in an occupational area offered by a college’s I-BEST program.

This same box includes four types of program components that are expected to improve participant outcomes by addressing barriers that are hypothesized to impede successful entry into and completion of occupational training:

• Assessment. I-BEST programs administer the CASAS to assess English and math skill levels and to determine whether the applicant is eligible for I-BEST.19

• Supports. I-BEST supports include career and academic advising by dedicated program advisors, and “fill-the-gap” financial support for tuition (not covered by financial aid) and for training-related tools, books, and transportation.

• Instruction. I-BEST integrates basic skills and occupational training. Courses are team taught—basic skills instructors and occupational training instructors must be in the training classroom together at least 50 percent of the time. Some programs also offer instructor-led study groups, open labs, or tutoring.

• Employment. I-BEST students could access their college’s general career services and career counselors. Two of the I-BEST programs included in the study, Nursing Assistant and Sustainable Office Skills, made placements in clinical or internship settings.

The middle box shows intermediate outcomes—targeted improvements expected to lead to better main outcomes. These include improved basic academic skills; improved psycho-social skills such as grit and academic self-confidence; improved skills in occupational areas; and increased career knowledge.

18 Program inputs can include components available only to treatment group members as well as those available

to both treatment group and control group members, because the interaction of the former components with the latter can lead to impacts.

19 Entities such as colleges, employment programs, and employers use CASAS to assess basic skills, literacy, and English language skills need for both academic and workplace success.


PACE


In the far right box, the main outcomes are the primary targets that I-BEST seeks to change:

• Increased postsecondary attainment, namely accumulated hours of training and credits (as measures of progress toward a credential) and occupational training credentials.

• Successful employment, including obtaining employment in occupations that align with I-BEST training programs and pay at least $13 per hour ($15 per hour in King County), increased earnings and job benefits, and career advancement.

• Improvements in individual and family finances and well-being.

Influencing expected effects are a number of contextual factors, including the types and number of postsecondary training systems in the local area and strength of the local economy. Other community factors are the size and characteristics of the target population, and the number and nature of other service providers.

2.2. Research Questions for the I-BEST Evaluation

The implementation study documented I-BEST in the three colleges as implemented and captured participation patterns of treatment group members in training (see Chapter 4 for implementation findings). The impact study (see Chapter 5) aimed to measure the effectiveness of I-BEST in improving students’ short-term (within 18-24 months) intermediate and main outcomes.

Implementation study research questions:

• What is the intended program model? What is its institutional and community context?

• What intervention was actually implemented? Did it deviate from plans or expectations?

• What were the treatment group’s participation patterns and experiences with program services?

• What are the differences in services, including training, received by treatment and control group members?

Impact evaluation research questions:

• What were the main effects of I-BEST on:

− Educational attainment, including hours of occupational training received, credits received, credentials received, and other education outcomes?

− Entry into career-track employment, higher-wage jobs, earnings, and perceptions of career progress?


PACE


− Participant and family well-being, including income and material hardship?

• To what degree did the program affect intermediate outcomes in the theory of change, such as:

− Confidence in career knowledge and access to career supports?

− Psycho-social skills such as grit, academic self-confidence, core self-evaluation, and social belonging at school?

− Life stressors, such as financial hardship, life challenges, and perceived stress?

As discussed, the program’s theory of change not only describes hypothesized causal connections, it also identifies time horizons over which they are expected to occur. For example, with respect to I-BEST, the students’ first step along their chosen career pathway is an initial occupational certification that typically takes one to two academic quarters at the college to complete. Program completers can look for employment commensurate with the certification or continue on the next step of the pathway. Those who continue their education after I-BEST enroll in coursework for more advanced certifications (e.g., one lasting three quarters) or for an associate’s degree. Given that some advanced courses of study are longer than others, it is possible that some students will still be enrolled in training at the end of the 24-month study period. Thus, this early impact report focuses primarily on training outcomes that do not require a study participant to complete a program and earn a credential.

For this early impact report, the primary data sources for addressing the impact research questions are SBCTC records of course enrollment, completion, and credentials; two surveys administered at “baseline” (when students joined the study and were randomly assigned to the treatment or control group); and a follow-up survey of all study participants initiated at approximately 15 months after random assignment. The implementation study questions are addressed with information gathered during site visits and monitoring calls. A more complete description of data sources is in the concluding section of this chapter.

Later PACE reports will focus more on employment outcomes and on outcomes resulting from education and training activities that require longer to complete (e.g., associate’s degree programs). The continued measurement of such outcomes will be important, given that the career pathways framework implies that workers may alternate between education and employment as they move along a pathway.

2.3. PACE Evaluation Design and Analysis

As mentioned in Chapter 1, the PACE evaluation uses a random assignment research design to estimate the impact of access to a program on students’ outcomes. The great benefit of such a design is that when properly implemented, it ensures that estimated effects reliably can be


PACE


attributed to access to the program—and not to unmeasured differences in characteristics or external circumstances between the treatment group (who can access the program) and the control group (who cannot).

However, maintaining the comparability of the treatment and control groups requires comparing all of those in the treatment group with all of those in the control group, regardless of whether or not treatment or control group members actually enrolled in the program (what researchers refer to as an “intent to treat” analysis). A critical implication of this is that the evaluation estimates the impact of access to the entire program—to the entire I-BEST program, in this case—as opposed to the impact of the program’s specific components. The evaluation does so by comparing the entire control group with the entire treatment group, regardless of the treatment group’s actual take-up of one or many program components.

A second feature of the PACE impact study design is that both treatment and control group members can access education, training, and support services available in the community that are not exclusive to the program PACE is evaluating. In the case of I-BEST, the evaluation estimates the effect of the program’s components above and beyond what was otherwise available at the three colleges and elsewhere in their communities during the study period. For example, both treatment and control group members could potentially access their college’s occupational training courses that were not part of I-BEST, subject to availability and meeting entry requirements. Thus, the control group’s experiences represent what would have happened absent I-BEST.

In summary, the impact study assessed whether the existence of this multi-component career pathways program led to better outcomes for students who were offered the chance to participate, given what these students could have obtained without the program.20

2.3.1. College Selection, Study Intake, and Random Assignment

The PACE evaluation of I-BEST is designed to pool results across the three colleges in the study and their various I-BEST occupational programs. The PACE research team began recruiting colleges for the I-BEST study in 2010. The team sought to recruit large I-BEST programs at multiple colleges, with an ultimate goal of enrolling at least 2,000 study participants across the colleges. Over the course of a few years, the team contacted colleges across the state, had

20 Four technical appendices provide additional details about analysis methods. Appendix A describes data

collected at baseline, gives further detail on baseline characteristics of treatment and control group members, and explains procedures for using these data to adjust for imbalances arising by chance during random assignment. Appendix B describes college records data serving as the main source for measuring program impacts on educational progress. Appendix C provides detail on survey-based outcome measures, adjustments for item nonresponse, and analyses of survey nonresponse. Finally, Appendix D documents the research team’s approach to outliers.


PACE


discussions with program administrators to learn more about the I-BEST programs (e.g., historical and projected enrollment, presence of waitlists, interest in participating in an experimental evaluation) and conducted site visits to the most promising eight community or technical colleges. Five of these colleges ultimately declined to participate in PACE. Reasons for not moving forward included concern about their ability to over recruit for a control group and resistance to changes in their enrollment processes to accommodate the evaluation. Bellingham Technical College (BTC), Everett Community College (EvCC), and Whatcom Community College (WCC) agreed to participate. However, because two of these colleges had small I-BEST programs, the evaluation team reduced the target sample size to 1,000 study participants.

The research team worked with each college to design and implement study procedures and enhancements to the standard I-BEST model using OSF funds (see Chapter 1). The research team and program staff developed research procedures that allowed for a rigorous test of I-BEST while minimizing the effect of the study on normal college operations. Additionally, the team provided ongoing technical assistance to the colleges to support recruitment and enrollment of participants into the study. Each college designated key staff, typically the I-BEST navigator, to serve as the primary liaison throughout implementation of the study.

Interested program applicants took the CASAS math and reading assessments to determine whether they were eligible for the I-BEST program of interest.21 Eligible applicants then met with I-BEST staff to confirm their interest in the program and to address any program-specific eligibility requirements (e.g., Nursing Assistant applicants were screened for tuberculosis).

Program staff then enrolled eligible applicants into the study (intake):

1. Informed Consent. Program staff discussed the PACE evaluation and offered its informed consent form. Applicants who declined to sign the informed consent form were excluded from the study and could not enroll in the I-BEST program(s) at the colleges that were part of the PACE study. Those who consented became study participants and proceeded to the next step.

21 The colleges varied slightly in required CASAS reading and math scores. At BTC and WCC, students were

required to score between 221 (4th to 5th grade equivalent) and 256 (12th grade equivalent) on both tests to be eligible for I-BEST. The exception was the Nursing Assistant program at BTC, which required only 211 on math. EvCC required students to score at least 201 on the math and reading CASAS tests, but typically recommended that students should score above 211 before enrolling in I-BEST.


PACE


2. Baseline Data. Staff asked study participants to complete the study’s Basic Information Form (BIF) and Self-Administered Questionnaire (SAQ). The BIF collected demographic and economic information. The SAQ measured a variety of attitudes, beliefs, and psycho-social dispositions, as well as more sensitive personal characteristics such as perceived ability to stay calm in stressful situations. Again, applicants who declined were excluded from the study and the I-BEST program(s).

3. Random Assignment. Program staff used an online system to randomly assign study participants to either the treatment group or the control group. The random assignment ratio was 1:1, so that the treatment and control groups would each include approximately half of the study sample.

4. Services According to Random Assignment Status. Study participants assigned to the study’s treatment group could access the I-BEST program(s) at their college. Under the evaluation’s “intent-to-treat” design (discussed earlier), they were included in the analysis whether or not they actually enrolled in I-BEST training. Participants assigned to the control group could not access the I-BEST program(s) at their college. Both treatment and control group members could access comparable other services available in the community, including those offered by the same college in which I-BEST was operating.

Between November 2011 and September 2014, program staff randomly assigned 632 study participants: 315 to the treatment group and 317 to the control group. Exhibit 2-2 shows the distribution by college.22 Random assignment occurred separately at each college, but all three colleges followed the same general procedures.

Exhibit 2-2: Number of Study Participants Randomly Assigned, by College College Treatment Control Total

Bellingham Technical College (BTC) 156 159 315 Everett Community College (EvCC) 121 120 241 Whatcom Community College (WCC) 38 38 76 Total 315 317 632

22 Random assignment at BTC occurred between November 2011 and September 2014. Random assignment at

EvCC occurred between February 2013 and April 2014. Random assignment at WCC occurred between October 2012 and April 2014.


PACE


2.3.2. Characteristics of the Study Sample

Exhibit 2-3 shows the percentage distributions of the treatment and control group members across a set of characteristics. The p-values in the last column test the hypotheses that there are no systematic differences between the groups for these characteristics.23

As shown, random assignment produced treatment and control groups without significant differences in observed baseline characteristics with three exceptions: age, food assistance receipt, and public assistance receipt. These differences are most likely due to chance,24 given the number of characteristics tested (see Hypothesis Testing section below). In conducting impact analyses, the research team controls for any bias resulting from these and other differences by using baseline values as covariates to adjust for chance differences (described in the Impact Estimation section below).

Exhibit 2-3 also shows the composition of the study sample. Participants have low levels of education, with 31 percent reporting less than a high school diploma or equivalent. Less than 10 percent reported having attended one or more years of college.

Sample members are low income, with a mean annual income of $22,110 and 47 percent reporting income of less than $15,000. Consistent with these low levels of income, 59 percent of study participants received benefits from the Supplemental Nutrition Assistance Program (SNAP, formerly known as Food Stamps) or Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) in the 12 months prior to study intake. About 21 percent of study participants received public assistance or welfare (Temporary Assistance for Needy Families) in the prior year. Receipt of food assistance and public assistance in the previous 12 months was somewhat higher in the treatment group than in the control group.

Many study participants were older than traditional college students. More than 60 percent were older than age 25, although 22 percent of the sample was age 20 or younger. There was a statistically significant difference in ages between the treatment and control groups at the 10 percent level. The majority of study participants (58 percent) were female. Slightly more than half (55 percent) were non-Hispanic white, and about one quarter (26 percent) identified as Latino or Hispanic. Two-thirds (67 percent) were not working at the time of random assignment, with only 13 percent working 35 hours or more.

23 The p-value summarizes results from chi-squared tests of the likelihood that the difference in the observed

value or larger would occur if there were no differences between the two samples. For example, a p-value of .32 means that even if the characteristics of the members in the treatment and control groups were identical, the observed difference or larger would occur 32 percent of the time.

24 See Appendix A.2. On average, one would expect that three out of 28 variables would have statistically significant mean differences. Regression adjustments also help to control for any effects that chance differences might have on the impact estimates.


PACE


Exhibit 2-3: Selected Characteristics of the I-BEST Study Sample

Characteristic All Study

Participants Treatment

Group Control Group p-Value

Age (%) .067* 20 or younger 22.2 23.2 21.1 21 to 24 14.9 11.1 18.6 25 to 34 29.8 31.4 28.1 35 or older 33.2 34.3 32.2

Gender (%) .231 Female 57.5 55.1 59.9 Male 42.5 44.9 40.1

Race/Ethnicity (%) .346 Hispanic 26.0 28.9 23.1 Black, Non-Hispanic 7.6 6.2 9.1 White, Non-Hispanic 54.9 53.1 56.7 Other, Non-Hispanic 14.1 13.4 14.8

Family Structure (%) .591 Not Living with Spouse/Partner and Not Living with Children 47.2 48.7 45.8 Not Living with Spouse/Partner But Living with Children 16.6 14.6 18.6 Living with Spouse/Partner and Not Living with Children 17.3 18.2 16.3 Living with Spouse/Partner and Children 18.9 18.5 19.3 Living with Parents 28.6 27.2 30.1 .412

Current Education (%) .497 Less Than a High School Diploma or Equivalent 30.7 28.2 33.1 High School Diploma or Equivalent 40.0 42.0 38.0 Less Than 1 Year of College 11.1 12.1 10.2 1 or More Years of College 9.5 10.1 8.9 Associate’s Degree or Higher 8.8 7.7 9.8

Income (%) .551 Less than $15,000 47.3 46.5 48.1 $15,000-$29,999 23.9 26.0 21.9 $30,000 or More 28.8 27.6 30.0 Mean $22,110 $23,002 $21,240 .378

Public Assistance/Hardship in Past 12 Months (%) Received WIC or SNAP 58.6 55.0 62.1 .092* Received Public Assistance or Welfare 21.3 18.1 24.3 .094* Reported Financial Hardship 48.5 49.8 47.1 .499

Current Work Hours (%) .993 0 66.6 66.9 66.3 1 to 19 8.5 8.5 8.5 20 to 34 11.7 11.7 11.6 35 or more 13.2 12.8 13.6

Sample Size 631 315 316 SOURCE: PACE Basic Information Form. SNAP is Supplemental Nutrition Assistance Program. WIC is Special Supplemental Nutrition Program for Women, Infants, and Children. NOTE: * denotes statistically significant differences at the p = .10 level. Appendix A provides a fuller set of baseline characteristics random assignment generated well-balanced treatment and control groups. Some percentages do not add up to 100.0% due to rounding. Public Assistance/Hardship in Past 12 Months does not add to 100% because the categories are not mutually exclusive and exhaustive.


PACE


2.3.3. Analysis Plan for the Impact Study

The PACE evaluation of I-BEST is designed to pool results across the three colleges in the study and their various I-BEST occupational programs. Because only three of Washington State’s 34 community and technical colleges are included in the study, the results should not be considered representative of all community and technical colleges in the state (see box).

Three of 34 Community and Technical Colleges

The three colleges included in the PACE study were not randomly selected and are not representative of community and technical colleges in Washington. Nor are the college I-BEST programs representative of I-BEST programs implemented across the state. However, the three colleges do reflect the diversity of student populations and approach to program delivery. As shown below, there is variation across the colleges in terms of overall student population size, size of I-BEST programs, and the percentage of students who receive need-based financial aid.

The three colleges fall across the distribution of SBCTC colleges in terms of overall enrollment and receipt of need-based financial aid. However, none of the three colleges had an especially large I-BEST program: BTC’s was roughly average and EvCC and WCC had smaller than average I-BEST programs. Chapter 3, Section 3.1 provides more details on the local context of the three colleges.

School Overall Enrollment

Student Headcount in

I-BEST (#) Need-based Financial

Aid Receipt (%) Bellingham Technical College 5,526 125 54 Everett Community College 19,388 73 21 Whatcom Community College 11,292 22 46 SBCTC Average 10,581 126 38

https://www.sbctc.edu/about/facts-publications/field-guide-2017.aspx https://www.sbctc.edu/resources/documents/colleges-staff/research/academic-year-research/2015-16/ayr-1516-3-selected-programs.pdf Data from 2015-2016 Academic Year

Prior to estimating I-BEST impacts, the research team published an analysis plan (Abt Associates 2015) specifying key hypotheses and outcome measures and registered the plan on the What Works Clearinghouse and Open Science Framework websites.25 The team subsequently

25 The analysis plan was posted to the What Works Clearinghouse (WWC) online registry of randomized control

trials (RCT) on May 20, 2016. In September 2016, under the terms of a grant from the Institute of Education Sciences, the RCT registry information was removed from the WWC website and transferred to the Society for

https://www.sbctc.edu/about/facts-publications/field-guide-2017.aspx

https://www.sbctc.edu/resources/documents/colleges-staff/research/academic-year-research/2015-16/ayr-1516-3-selected-programs.pdf


PACE


assessed data quality, refined the plan for I-BEST, and publicly registered it on the Open Science Framework website.26 The purpose of the analysis plan and registration was to guide the work of the research team and publicly commit to particular hypotheses and an estimation approach, in alignment with ACF’s commitment to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations.27 Pre-specification and registration help to establish the scientific rigor of research by documenting that inspection of early results did not influence the selection of findings in PACE reports.

Hypothesis Testing

An essential principle in the PACE analysis plan is to organize and discipline the number of statistical tests conducted. Like most social policy evaluations, the nine PACE studies target an array of different outcomes. If the evaluation did not adjust in some way for multiple hypothesis tests, a potentially large number of the tests would reach conventional levels of statistical significance by chance, even if there were no effect on any outcome. This is known as the problem of “multiple comparisons.” To address this issue, the team established three categories of hypotheses: confirmatory, secondary, and exploratory:

• Confirmatory hypotheses involve outcomes most critical to judging whether the program seems to be on track—that is, producing the results expected at a given follow-up duration. Given the relatively small sample sizes in the PACE studies, the research team limited such tests to one per program in the early impact report (at 18-24 months after randomization for I-BEST) and two tests in each subsequent report (at three and six years after random assignment)—selecting outcome(s) under the “main” category in the program’s theory of change (see Exhibit 2-1). Each confirmatory hypothesis has an expected direction of change, an increase or decrease in the outcome. Therefore, the research team tests each confirmatory hypothesis for significance only in the specified direction, ignoring possible effects in the other, by applying a one-tailed test of statistical significance.

• Secondary hypotheses involve a set of additional indicators consistent with expected effects within the period covered by the study report. As with the confirmatory hypothesis, each secondary hypothesis has an expected direction of change, and therefore, the research team tests each secondary hypothesis for significance only in the specified direction by applying a one-tailed test of statistical significance. Secondary

Research on Educational Effectiveness (SREE). SREE plans to re-launch the registry in late 2018, at which time the analysis plan will be available in a searchable online database.

26 See https://osf.io/6n5ua/. 27 See https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy.

https://osf.io/6n5ua/

https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy


PACE


analyses for I-BEST included tests of hypotheses for additional education outcomes, as well as a number of indicators of early career progress.

• Exploratory hypotheses cover an additional set of possible effects that still fit within the framework of the theory of change, but whose direction and timing are less certain. Accordingly, the team applies two-tailed tests to these hypotheses to account for the uncertainty in the direction of impact. For this report, the research team selected several indicators of psycho-social outcomes (e.g., grit) and education success (e.g., enrolled in developmental math courses).

Chapter 5 identifies the specific hypotheses in each category tested for I-BEST.

Impact Estimation

Random assignment ensures that, on average, samples of treatment and control group members will have similar characteristics at the outset. As a result, measured differences in outcomes provide unbiased estimates of program impacts. Following common conventions, the team compensated for chance differences in measured baseline characteristics when estimating program outcomes. Doing so also helps to increase the precision of estimates.

For PACE, the research team estimated a statistical model relating each outcome to baseline variables for the control group sample, then calculated average differences between actual and predicted values in both groups and subtract the control group average from the treatment group average to generate the impact estimate. Appendix A provides a detailed description of this method.28

The research team estimated this approach both for continuous outcomes (e.g., total college credits earned) and for binary outcomes (e.g., yes/no questions). For survey-reported outcomes, weights were used to average outcomes. Additional details can be found in the technical appendices.

Formally, estimation uses the following equation:

( ) ( )( )1 1ˆ ˆ ˆ1i i i i i ii iT C

T Y Y T Y Yn n

δ = − − − −∑ ∑,

where δ is the estimated impact of being in the treatment group (whether or not the person attended the program or used any of the offered services); 𝑌𝑌 is the observed outcome of

interest (e.g., credits); Y is a prediction of Y based on baseline variables measured at random

28 As explained in section A.3 of Appendix A, the approach is a variant on the traditional approach to regression-

adjustment methods used in impact analyses. The latter typically involves linear regression of each outcome on an indicator of treatment status and a series of baseline variables. In this approach, the coefficient on the treatment indicator provides the regression-adjusted impact estimate.


PACE


assignment; 𝑇𝑇 is an indicator of treatment status (which is set equal to 1 if the individual is assigned to the treatment group and 0 if the individual is assigned to the control group); Tn

and Cn are the respective sample sizes in the treatment and control groups; and the subscript i

indexes individuals.

2.3.4. Analysis Plan for the Implementation Study

The PACE evaluation’s implementation study relies on both qualitative and quantitative analyses, as well as a broad variety of data sources. Key analyses include the following:

• Descriptive. Describing each program’s design and context and developing its theory of change relied primarily on review of program materials (e.g., prior evaluations of I-BEST, college-specific documentation on the components of the I-BEST program); in-person discussions with program staff and leadership during two rounds of site visits; and biweekly or monthly calls between study and program leadership during the study period when random assignment was ongoing.

• Quantitative. A quantitative analysis of the proportion of program participants who reached major program milestones served to systematically document their experience in the program. This relied on college records and follow-up surveys of treatment and control group members.

• Fidelity. The study relies primarily on field observations and interviews with program staff and participants to assess the delivery of the I-BEST program. To address the question of how program delivery changed over time, the research team asked program staff about internal or external obstacles and how staff altered the program in an attempt to overcome them. Quantitative analysis of how and the extent to which students moved through the program also enables the comparison of the actual delivery of the program versus its design.

• Service Differences. Because the random assignment design of the impact study implicitly ensures that any effects of the program result from the different experiences of treatment and control group members, a key task of the implementation study is to describe the difference in services the two groups received. This is particularly important for the PACE evaluation, as the control group is not barred from receiving services similar to those the study’s treatment group received.

2.4. Data Sources

The PACE evaluation’s implementation and impact studies use a variety of data sources.

• Baseline Surveys. Prior to random assignment into the evaluation, program applicants completed two baseline surveys: The Basic Information Form (BIF) collected


PACE


demographic and economic information. The Self-Administered Questionnaire (SAQ) measured a variety of attitudes, beliefs, and psycho-social dispositions, as well as more-sensitive personal characteristics.

• Follow-Up Survey. The research team sought to survey all study sample members starting at 15 months after random assignment. On average, the survey occurred 18 months after random assignment. The survey asked questions on participants’ training and service receipt, postsecondary educational attainment, employment, income, debt, and participation in income support programs. It used a mixed-mode approach, conducted initially by telephone and then in person for those participants not reached by telephone. The I-BEST study collected surveys from 238 treatment group members and 225 control group members, yielding response rates of 76 percent and 71 percent, respectively.29

• Administrative Records. The research team relied on the administrative records of each program evaluated, both to describe the experience of treatment group members in their program and to estimate program effects. The I-BEST study used records from SBCTC to measure treatment and control group members’ participation in education and training, as well as their credential receipt.

• National Student Clearinghouse (NSC). The NSC is a national repository of information that captures the vast majority of college enrollments at public and private nonprofit institutions. The I-BEST study used data from the NSC to impute outcomes for participants who attended colleges not part of the SBCTC system.

• Site Visits and Monitoring Calls. For the implementation study, the research team made two site visits to each of the three colleges. The goal of the first round of visits was to document the program’s theory of change and key components and to assess implementation of evaluation procedures. The goal of the second round of visits was to document any modifications to operations or the provision of services, as well as implementation challenges and plans for sustaining the program beyond the study period.30 During both sets of visits, the research team interviewed program managers; staff involved in evaluation activities (e.g., recruitment, intake, random assignment); instructors; advisors; case managers; and staff at any partner agencies with an important role in service delivery. In addition to these visits, the research team had regular conference calls with program staff during the random assignment period to discuss program updates, recruitment activities, intake and random assignment

29 See Appendix C for response bias analyses. 30 Visits to BTC occurred in March 2012 and November 2014. Visits to EvCC occurred in August 2013 and July

2014. Visits to WCC occurred in May 2013 and April 2014.


PACE


processes and any challenges, engagement in the program by treatment group members, and staffing changes.

• Program Documents. The research team obtained and reviewed program documents including funding applications; course catalogs; program materials such as applications, assessment tools, syllabi, and pre-training checklists; annual reports; and reports to funders.


PACE

Abt Associates 3. About I-BEST: Context and Program Design ▌pg. 25

About I-BEST: Context and Program Design

The context in which I-BEST operates and its design at the three colleges participating in PACE are important to understanding its implementation and potential impacts. This chapter begins with a description of the local context for each of three colleges during the PACE evaluation enrollment period (November 2011 to September 2014). Next, it describes the comparable services available at the colleges and in the communities and expected treatment differential. Finally, it describes the I-BEST program design, course offerings, and instructional and support services.31

3.1. Local Context

Three aspects of the local environment are important to understanding I-BEST’s implementation and impacts—the characteristics of the population in the area served by each college, the local labor market, and the services and opportunities available to control group members.

3.1.1. Population and Labor Market

As discussed in Chapter 2, of Washington State’s 34 community and technical colleges, three took part in the PACE evaluation of I-BEST—Bellingham Technical College (BTC), Everett Community College (EvCC), and Whatcom Community College (WCC). Exhibit 3-1 below shows the location of each, as well as the occupational courses of study for I-BEST programs included in the evaluation. (Exhibit 3-3 describes each occupational program in detail.)

BTC and WCC are located in Bellingham, which is 90 miles north of Seattle. Bellingham has a population of approximately 80,000 and is the largest city in Whatcom County, which extends north to the U.S.-Canadian border. The median household income in Bellingham in 2015 was $43,536, lower than the median for Washington ($61,062) and the United States ($53,889). The 2015 poverty rate for the city was 23 percent.32 Everett Community College is located about 30 miles north of Seattle, in Snohomish County. Everett has a population of approximately 106,000. As in Bellingham, the median income in Everett ($49,578) is lower than in both Washington and the United States. Everett’s 2015 poverty rate was 18 percent.33

Overall, the local economies in both Bellingham and Everett improved during the years the study operated. When BTC began random assignment in November 2011, the unemployment

31 For additional information about the program see Glosser et al. (2014). 32 U.S. Census Bureau, 2011-2015 American Community Survey 5-Year Estimates. 33 U.S. Census Bureau, 2011-2015 American Community Survey 5-Year Estimates.


PACE


rate in Bellingham was 8.2 percent. In September 2014, when random assignment was completed, the unemployment rate was 6.0 percent. In Everett, the unemployment rate declined from 6.9 percent to 5.4 percent between February 2013 and April 2014, while random assignment was ongoing.

At the outset of the study, program administrators reported high demand for most occupations in I-BEST programs included in PACE, particularly for nursing assistants, welders, and machinists. However, BTC stopped its I-BEST Electrical program partway through the study because program faculty were concerned that demand for electricians in the construction industry had decreased (see further detail below).

Exhibit 3-1: Location of PACE Study Colleges and Programs Included in the I-BEST Study

Exhibit 3-2 shows that the three colleges vary in size, student age, and student academic goals. BTC serves a much smaller student body (5,526 students) than do EvCC and WCC (19,388 and 11,292, respectively). BTC also serves a slightly older population (median age of 27 compared with 23 or 22). The exhibit also shows the SBCTC average.


PACE


Exhibit 3-2: Characteristics of Students at Three Colleges and Statewide, 2015-16

College Number of Students Enrolled

Median Age of

Student

Enrolled in Occupational Skills Training Courses (%)

Enrolled in Academic/ Transfer

Courses (%)

Enrolled in Basic Skills Courses (%)

Bellingham Technical College 5,526 27 93 2 4 Everett Community College 19,388 23 40 48 11 Whatcom Community College 11,292 22 37 56 5 All SBCTC Colleges 380,918 26 40 41 11 SOURCE: https://www.sbctc.edu/about/facts-publications/field-guide-2017.aspx.

At BTC, a much higher percentage students are enrolled in occupational training compared with the two other colleges. As shown on Exhibit 3-2, more than 90 percent of BTC students are enrolled in occupational skills training, including both short-term certificate and two-year degree programs. This is more than double the percentage of students at WCC and EvCC who are enrolled in similar programs. At EvCC and WCC, a much higher percentage of students (48 and 56 percent, respectively) pursue academic or transfer programs—that is, a course of study designed to easily transfer to a four-year institution. Finally, the share of students enrolled in basic skills classes was low at all three campuses, but less so at EvCC (11 percent) than at BTC (4 percent) or WCC (5 percent).

3.1.2. Control Group Environment

A critical contextual factor for understanding the impact of I-BEST is the degree to which educational opportunities and supports comparable to I-BEST’s were otherwise available locally. Programs have the greatest potential to produce observable impacts (treatment vs. control differences) when the services they offer are clearly distinguishable from those already available in the community. The nature of those other services also affects program completers’ ability to build on initial training successes.

PACE control group members had access to educational and occupational supports through several means. These included alternative training programs in their communities, non-I-BEST courses at the three colleges in the study, and other supports in their communities. Exhibit 3-3 summarizes the difference in services available to the study’s control group and treatment group members, and then each component is discussed below.

https://www.sbctc.edu/about/facts-publications/field-guide-2017.aspx


PACE


Exhibit 3-3: Comparison of Career Pathways Components Available to PACE Control Group and Treatment Group Members

Career Pathway

Component

Standard Community Offerings Available to

Treatment and Control Groups

I-BEST Components Available to

Treatment Group Only Assessment • Accuplacer • CASAS Instruction • ABE or ESL classes at the three

colleges or other community locations • Non-I-BEST classes at the three

colleges if students met entry requirements

• I-BEST classes at other community and technical colleges

• Access to college-level classes and concurrent support for basic skills needs

• Team-taught college-level academic and vocational classes

• Contextualized basic skills instruction • Cohort-based learning • Instructor-led study group, open lab, tutoring

Supports • Academic advising through the three colleges’ advising centers

• College tutoring services • Financial aid based on eligibility and

availability (e.g., Pell grants, Washington State Opportunity Grants)

• Dedicated I-BEST navigator at each college supported by funding from the Open Society Foundations

• Fill-the-gap financial support for tuition and course materials

• Funding for support services (e.g., uniforms, transportation, licensure testing)

Employment Services

• Standard employment services from the three colleges’ career services offices

• Job search supports from other community providers (e.g., Goodwill Industries, American Job Centers)

• Career counseling from I-BEST navigator • Clinical placements (for Nursing Assistant students) • Internships (for Sustainable Office Skills students)

SOURCE: Program documents and site visits.

Assessment. The three colleges assessed students using CASAS prior to random assignment. This assessment identified students whose basic skills might not be high enough to otherwise qualify them for college-level classes. Students seeking to enroll directly in college-level classes (not remedial and not part of I-BEST) take the Accuplacer placement test and need to score high enough to qualify for admission. Thus, the colleges use the CASAS to assess basic reading, math, and English language skills and the Accuplacer to assess college placement.

Instruction. Treatment group members could access I-BEST programs and the associated occupational and basic skills instruction, including team teaching. Control group members could not access I-BEST programs and courses at the three colleges; however, they could access other education and training opportunities available to them, including:

• Non-I-BEST courses. Students in the control group could enroll in non-I-BEST courses at the three study colleges, as well as at other colleges in the area if students were eligible. For example, both BTC and EvCC offered non-I-BEST versions of Nursing Assistant and Welding. Control group students could follow the more typical pathway—taking ABE, ESL, or developmental education classes until their skills/placement scores were sufficient to qualify them for entry into college-level coursework.

• I-BEST programs in other local colleges. All three colleges are near at least one other community or technical college that also offered one or more I-BEST programs. As a


PACE


condition of participating in the study, control group members were barred from enrolling for a designated embargo period in the I-BEST program at the college where they applied.34 While control group members could enroll in I-BEST programs at other colleges that were not a part of the PACE study, SBCTC administrative data show that no control group members enrolled in I-BEST courses at other colleges.

Supports. Treatment group members had access to a dedicated navigator who could provide a range of supports throughout students’ enrollment in the I-BEST program. These students also could access I-BEST’s fill-the-gap funding and additional funding for supportive services available through I-BEST. Both treatment and control group members could access general college advising, tutoring, and financial aid services that were available to all students at the colleges. Both treatment and control group members could potentially access financial support through Pell grants, Washington State’s Opportunity Grants, its Basic Food Employment and Training (BFET) program, veteran’s benefits, and Temporary Assistance for Needy Families (TANF), depending on eligibility.

Employment Services. Both treatment and control group members could access their college’s employment and job placement services designed to help program completers find jobs. Whether they enrolled in college classes or not, they also could access other employment assistance in the community, such as the job search and job readiness services at local American Job Centers. In addition to these generic services, treatment group members could receive job placement and advising from the I-BEST navigator. Several of the I-BEST programs in the study also offered clinical placements (e.g., Nursing Assistant) or internship components (Sustainable Office Skills) intended to provide relevant work experience as part of the I-BEST program.

In summary, there were alternative options for control group students seeking basic skills, academic, and occupational training. They could enroll in I-BEST programs at other schools. Or they could enroll in non-I-BEST offerings at their original college or at other public or private training providers in the area if they qualified. However, placement elsewhere was subject to availability of slots and meeting eligibility requirements. This, as well as the lack of I-BEST’s dedicated advising support and fill-the-gap funding, added up to a clear difference in available services between the control and treatment groups.

3.2. I-BEST Program Design

SBCTC set the general parameters of the I-BEST program, but colleges had flexibility to design and structure their I-BEST implementation, most notably how they identified students to enroll

34 Applicants were informed at intake that if randomly assigned to the control group, they would not be allowed

to enroll in I-BEST programs at that college for 36 months from the date they entered the study.


PACE


in the program and the industry focus and structure of the I-BEST programs. For each of the colleges in the PACE evaluation, this section describes their goals, recruitment process, and the structure and courses for their I-BEST programs.

3.2.1. Program Goals

As discussed, the SBCTC designed I-BEST to provide the first step on an articulated career pathway in a range of occupations, particularly to students whose skill levels were too low to enroll in occupational training otherwise. However, the specific focus and content of the three colleges’ I-BEST programs varied, partly reflecting their institutional type:

• Bellingham Technical College seeks to provide career and technical education geared toward developing specific occupational skills. However, the college designed its I-BEST programs so that these short-term credentials were a step on a path toward more advanced credentials or associate’s degrees.

• Everett Community College started its I-BEST program to serve its ESL population. Initially it focused on helping ESL students obtain short-term occupational credentials for employment in high-demand occupations, and then it expanded to serve the broader basic skills population. Although employment can be a shorter-term goal for these students, some could choose to continue at EvCC for more advanced certificates or associate’s degrees or choose to transfer to a four-year college.

• Whatcom Community College designed its I-BEST program to serve students with specific occupational interests but whose low level of basic skills either made them ineligible for admission into their college-level program of interest or suggested they would need additional basic skills instruction to succeed in the program. College leadership reported that the program was primarily a step toward more advanced college certificate and degree programs. This assumption that students would continue their education is a key distinction of WCC’s I-BEST program versus the other two colleges’.

3.2.2. Recruitment

To meet study enrollment goals, participation in the PACE evaluation required the colleges to recruit twice as many students for their I-BEST programs, with the goal of recruiting at least 1,000 study members across the three colleges. Each college was responsible for recruiting potential I-BEST participants and developed a recruitment plan that was based on historical demand for its I-BEST courses and estimates of the number of students currently in its ABE and


PACE


ESL programs.35 At the outset of the study, all three of the colleges anticipated that they would identify and recruit study participants primarily from these existing student pools. Specifically:

• Bellingham Technical College expected to recruit students primarily from its ABE population. The occupational skills programs that BTC included in its I-BEST program, including welding, electrical, and health care occupations, had traditionally been in demand by basic skills students at the college.

• Everett Community College administrators expected to recruit a large share of potential I-BEST students from its ABE and ESL programs. In addition, EvCC provides education and training services to TANF recipients through WorkFirst (the state’s TANF program). Administrators considered the short-term certifications available through I-BEST as a good fit for this population, given limitations on the length of training allowed for recipients of WorkFirst cash assistance.36 The college also expected to recruit students new to EvCC through outreach to local community-based organizations and high schools.

• Whatcom Community College anticipated recruiting most potential Clerical Assistant program participants from students currently enrolled in its basic skills classes, including ESL. Staff also expected that some students enrolled in developmental education classes would benefit from the added support of I-BEST, and planned to recruit from these classes, too.

As discussed in Chapter 4, recruitment proved challenging as the evaluation progressed. As a result, each college expanded its recruitment effort to include a larger than anticipated outreach to the broader community beyond the existing ESL and ABE classes, as well as additional efforts to identify low-skill students who qualified for occupational training but who could still potentially benefit from the I-BEST approach.

3.2.3. I-BEST Program Course Structure, Services, and Staffing

Exhibit 3-4 summarizes the I-BEST programs included in the PACE evaluation in each of the three colleges. It provides the occupational focus, the length of the program, specific courses required in each occupation area, credentials that could be attained, and how the program is designed to place participants on a pathway toward additional credentials. Each is discussed below.

Occupational focus and program length. As shown, BTC offered the I-BEST programs Automotive, Electrical, Nursing Assistant Certified, Precision Machining, and Welding; EvCC 35 Each college received funding through a subcontract with Abt Associates to support a recruiter. 36 TANF recipients have a 12-month lifetime limit on vocational education that counts as a core activity, meaning

that it counts toward a TANF program’s work participation rate.


PACE


offered Nursing Assistant Certified, Sustainable Office Skills, and Welding; and WCC offered Clerical Assistant. The I-BEST programs ranged in length from one quarter to three quarters. EvCC included a “pre-I-BEST” quarter focused on basic skills improvements as part of two of its three programs.

Team teaching. All programs included courses that were team taught (these courses are indicated in italics in Exhibit 3-4). The team teaching model paired a basic skills instructor and an occupational instructor in the delivery of occupational training.

Basic skills support classes. In addition to the basic skills instruction delivered via team teaching, I-BEST programs also often included support classes, taught by basic skills instructors, to provide students with the necessary academic foundation to succeed in occupational training. Support classes are an opportunity for basic skills instructors to clarify concepts presented in the training that might be confusing to students; to address basic skills such as reading or math that are required to succeed in training; and to help students prepare for upcoming tests, quizzes, and assignments. These basic skills classes are meant to complement the occupational courses.

Credits. As designed by SBCTC, the I-BEST program focuses on the attainment of workforce credits that lead to workforce credentials, usually within one to two quarters of programming. If they desire, students can progress from there to additional education and training, including at the college level, to receive additional workforce credits and credentials. They also can earn academic credits and credentials, consisting of credit-bearing courses and credentials that are transferrable to a four-year college. Basic skills or developmental courses, including I-BEST support classes, ABE, and ESL classes, do not count toward credential attainment.

Credentials. A range of credentials are available from the I-BEST program, depending on the college and occupational field. These include college-awarded credentials that appear on a student’s transcript; certificates of completion that indicate the student has completed a series of courses but do not appear on a transcript; and state- or industry-recognized credentials that require passing an exam administered by a public agency or licensing body. As discussed further in Chapter 5, SBCTC awards the following types of credentials: workforce completion (a certificate requiring less than 20 credits); workforce award (a certificate requiring 20 or more credits but that are not transferable to a four-year college); and associate’s degrees that require two years of coursework and are transferrable to a four-year college. Completion of the I-BEST courses generally culminated in a workforce award from the college, although further state licensing might be required to practice in the field (e.g., nursing or welding).


PACE


Exhibit 3-4: I-BEST Programs, Courses, Credits and Certifications, and Next Steps, by College I-BEST Program /

Length Courses (team taught are in italics) Credits and Certification (if applicable) Examples of Next Step(s) on Educational Pathway

Bellingham Technical College Automotive (1 quarter)

• Transportation Services and Systems • Occupational Math • Basic Academic Skills

23 Workforce Credits Vehicle Service Technician Certificate

• General Automotive Repair Certificate • Associate’s in Applied Science

Degree in Automotive Technology Electrical (1 quarter)

• Trade Safety • Direct Current (DC) Circuits (including lab) • Electrical Drawings and Blueprints • Applied Mechanics • Occupational Math • Basic Academic Skills—Math • Basic Academic Skills—Electrical

25 Workforce Credits • Electrician Construction Certificate • Associate’s in Applied Science

Degree

Nursing Assistant Certified (1 quarter)

• Nursing Assistant Essentials • Basic Academic Skills • clinical placement • one-day courses in CPR and HIV/AIDS

12.5 Workforce Credits Nursing Assistant Certification (requires passing state exam)

• Phlebotomy • Medical Coding • Dental Assisting

Precision Machining (2 quarters)

• Machine Lab Safety • Intro to Measuring and Inspection • Intro to Machining • Intro and Advanced Manual Lathe • Occupational Math • Applied English • Machine Fundamentals • Precision Grinding • Blueprint Reading • Intro to Manual Mill

45 credits Principals of Precision Machining Certificate

• Associate’s in Applied Science Degree

Welding (2 quarters)

• Welding Safety I and II • Shielded Metal Arc Welding • Thermal Cutting • Gas Metal Arc Welding • Occupational Math • Applied English • Career Opportunities for Welders and

College Success Foundations • Basic Academic Skills (one per quarter)

30 Workforce Credits Washington Association of Building Officials (WABO) Certification (requires passing certification exams)

• Basic Industrial Welding Skills Certificate

• Associate’s in Applied Science Degree in Welding Technology


PACE


I-BEST Program / Length Courses (team taught are in italics) Credits and Certification (if applicable) Examples of Next Step(s) on

Educational Pathway Everett Community College Nursing Assistant Certified (1 quarter, plus ESL/ABE pre-program quarter)

• Pre-I-BEST course • Nursing 101 • I-BEST Basic Skills

18 Workforce Credits (no credits for pre-program) Nursing Assistant Certification (requires passing state exam)

• Phlebotomy Technician Certificate • Associate’s in Applied Science

Degree in Medical Assisting

Sustainable Office Skills (2 quarters)

• Beginning Keyboarding • Computer Literacy • Sustainable Office • Job Readiness • Cooperative Work Experience

19 Workforce Credits Sustainable Office Certificate

• Legal Office Support Certificate • Medical Administrative Certificate • Associate’s in Technical Arts Degree

Welding (1 quarter, plus ESL/ABE pre-program quarter)

• Pre-I-BEST course • Sustainable Industrial Standards for Welding • Advanced Arc • Gas Metal Arc/Flux Cord Arc Welding • I-BEST Basic Skills course

16 Workforce Credits (no credits for pre-program) WABO Certification (requires passing certification exams)

• Advanced Tungsten Inert Gas Welding

• Aerospace Fabrication & Welding Certificate

• Associate’s in Technical Arts Degree, Welding

Whatcom Community College Clerical Assistant (2-3 quarters)

• Introduction to Business Computing • Office Procedures • Introduction to Accounting • Customer Service for Professionalism • mediated labs in Accounting, Business

Computing, and Office Administrationa • Basic Academic Skills (one per quarter)

28 Workforce Credits Clerical Assistant Certificate of Proficiency

• Accounting Certificate • Hospitality and Tourism Business

Management Certificate • Associate’s in Science Degree in

Business Administration

a Mediated labs are self-directed computer lab courses that allow students to choose from an array of courses in a lab setting with an instructor and teaching assistants, who are available to provide support and review work. The class is semi-structured, with quiz and test dates set by the instructor.


PACE


Subsequent training and credentials. Exhibit 3-4 also shows potential subsequent credential options for each of the I-BEST programs in the PACE evaluation. Each of the colleges defined a longer-term educational pathway following the I-BEST program that mapped to available employment options. Students did not necessarily need to complete all the I-BEST courses in a program to move to additional education and training options; in some cases, specific courses rather than the completion of the entire I-BEST program were needed to progress to additional coursework at the college.

Exhibit 3-5 below provides two examples of I-BEST educational pathways—the Welding program at EvCC and the Clerical Assistant program at WCC. As shown, they provided multiple opportunities for occupational training and credential attainment built on the skills students acquired during the I-BEST program.

• EvCC’s Welding program emphasized specialized industry certifications related to various welding techniques. As students accumulated these skills, they progressed first toward a four-quarter program certificate, and then an Associate Degree in Technical Arts.

• WCC’s Clerical Assistant program provided academic training that could be applied to an array of workforce and academic degrees, including Accounting, Business, and Entrepreneurship. Students could also earn shorter-term certificates in more specialized vocational fields such as Retail Management and Hospitality and Tourism.

“Fill-the-gap” financial assistance. The colleges expected that most students would receive the needed financial support through Pell grants, TANF funds, or other sources such as Washington State Opportunity Grants. However, in the event a treatment group member did not qualify for aid or tuition costs were not fully covered, the colleges participating in PACE could use OSF funding to fill this gap. Thus, fill-the-gap assistance ensured that all treatment group members would pay no tuition.

Dedicated advising. The colleges used foundation funds to add dedicated advisors (“navigators”) for I-BEST treatment group members. The advisor was intended to be a regular presence in I-BEST classes and available to assist students through all phases of the program, including during enrollment, while they were attending I-BEST courses, and with career planning.


PACE


Exhibit 3-5: Examples of I-BEST Educational Pathways


PACE

Abt Associates 4. Implementation Study Findings ▌pg. 37

Implementation Study Findings

Prior chapters described the structure of and services planned by the I-BEST program in the three colleges, as well as the contextual factors that could facilitate or impede program implementation and outcomes for participants. As described in this chapter, the research team examined the extent to which the I-BEST program operated as planned while it was part of the PACE study, including efforts to expand and diversify existing recruitment methods in order to scale up for the PACE evaluation.

This chapter describes the implementation findings, based on site visits conducted at two points in time. In the second section, it describes patterns in how treatment group participants experienced the I-BEST program in the colleges, based on SBCTC program records. It concludes by comparing education and training and service receipt for the treatment group versus the control group, based on responses to the 18-month study participant follow-up survey.

4.1. Implementation Findings

• Recruitment of students for I-BEST was challenging due to lower than expected demand for the programs among existing ABE and ESL students. In response, the colleges expanded outreach, but still missed their recruitment goal.

As discussed in Chapter 3, the colleges in the study expected a high proportion of I-BEST enrollments would come from current basic skills students, primarily those in ESL and ABE courses. Colleges administrators anticipated a high demand among this population for occupational training, and several of the I-BEST programs included in the study (e.g., Welding, Electrical) had waiting lists in the quarters leading up to random assignment.

Despite these expectations, all three colleges struggled, recruiting only 63 percent of the total sample goal of 1,000 (or 632 students).37 To address recruitment shortfalls, colleges expanded their outreach to the general community and students who had qualified college-level occupational training based on the entrance exam but still had low basic skills. In particular, the colleges adopted two new strategies:

• Targeting individuals not currently enrolled in the colleges. External recruitment efforts included outreach to community-based organizations and employers, and advertisements in local media. Additionally, the colleges marketed I-BEST to individuals enrolling through the TANF WorkFirst program.

37 Lower than expected enrollment in I-BEST may also have been related to a broader decline in enrollment in

community and technical colleges in Washington. Between 2011 and 2013, enrollment at two-year institutions in the state declined 5.9 percent (WSAC 2015).


PACE


• Targeting students who were eligible to enroll in non-I-BEST college classes based on their Accuplacer scores but whose CASAS scores placed them in the eligible range for I-BEST. At BTC, administrators reported that many of the students entering the standard occupational training courses had basic skills levels, especially in math, that made it hard for them to succeed without extra basic skills supports. In the final year of study enrollment (2014), BTC required all applicants to non-I-BEST versions of programs (e.g., welding) whose Accuplacer scores were on the low end of the eligibility range to also take the CASAS to determine basic skills levels and I-BEST eligibility. The theory was that students with low but acceptable Accuplacer scores had basic skills levels that made them eligible for I-BEST. These students were offered the opportunity to enroll in the study. College leadership reported that they saw this both as a way to potentially expand the recruitment pool as well as a way to augment supports for students who, though they qualified for college-level classes, could benefit from extra instructional support.

• Ultimately, recruitment struggles led to smaller class sizes and blended classes containing both I-BEST and non-I-BEST students.

Each college expected to enroll sufficient study participants to fill its I-BEST courses. However, lower than expected study enrollment often resulted in the colleges not being able to fill all available slots in the I-BEST courses, in part because the study required that half of eligible applicants be randomly assigned to the control group. Often, this meant smaller class sizes. As a result, the colleges filled the extra spaces in the I-BEST classes with students who technically were not eligible for I-BEST due to their higher skill levels. In these blended classes, the basic skills instructor focused his or her attention on the I-BEST students but also provided some support to the non-I-BEST students.38

• I-BEST served students with low education and basic skills levels, including 31 percent who had less than a high school diploma or equivalent.

The I-BEST program seeks to increase access to college-level classes for students with low basic skills. As discussed in Chapter 2, the high proportion of sample members who had less than a high school diploma or equivalent reflects the low basic skills level of the population served. More than two-thirds of the sample reported never having attended college, and less than 10 percent had an associate’s degree or higher. This is due to eligibility criteria that targeted students with lower skills levels.

38 Staff at each of the colleges ensured that the non-I-BEST students in these classes were not control group

members.


PACE


• Though there was variation across the colleges, the three colleges implemented I-BEST team teaching as planned, combining basic skills and occupational technical content.

A central component of the I-BEST model is team teaching, which pairs an occupational instructor and a basic skills instructor in the same course. SBCTC required team teaching, but allowed flexibility in how instructors approached their teaching, including how they defined their roles within the classroom, planned their time together, and established the rapport to work collaboratively.

Though their role varied across the programs, the basic skills instructors were a key element across all I-BEST programs included in the PACE study. As required, the basic skills instructors were present for 50 percent of class time for occupational courses; in some cases, they were present for almost all of the class time as well as lab time in occupational courses.

• Shared course time and team teaching were standard across all I-BEST programs, but instructors had flexibility in how they approached their teaching.

Based on discussions with instructors and classroom observations, the research team identified three approaches to team teaching:

• Basic skills instructor as an active student—The basic skills instructor sits in class with the students and stops the occupational instructor to ask clarifying questions or for a concept to be further explained if students seem confused. This approach was used often in occupational courses that were primarily lecture based and highly technical, such as Welding and Machining. The approach also was easier to implement for teams who had not worked together before or had limited planning time. The box Observations from a Precision Machining Class below provides an example of this approach to team teaching.

• Basic skills instructor delivers part of the content during a designated part of the occupational classes—For example, a basic skills instructor might start each class with a discussion of study skills (e.g., test-taking) before turning the class over to the occupational instructor to present technical information. The box Observations from an Office Administration Class below contains an example. Staff reported that this approach was easier to implement for instructor teams new to each other.

• Basic skills and occupational instructors jointly deliver class content (co-teaching)—From the students’ perspective, delivery is seamless; it may not be apparent which instructor is responsible for which type of content. Staff reported this approach was more suitable to I-BEST programs where the occupational focus was not very technical or highly specialized, such as office skills versus welding. Staff also reported this approach was used more often when the team had experience teaching together. The box


PACE


Observations from a Sustainable Office Skills Class below describes an example of this approach.

Observations from a Precision Machining Course—Active Role for Basic Skills Instructor in Traditional Lecture Class at BTC

The second course in a two-quarter introductory machining sequence was a math-intensive course that emphasized “shop essentials.” The objective of the course was to familiarize students with measurement tools that they would be using in the shop. The class was primarily lecture based and led by the occupational instructor. The instructor explained the use of different tools and parts, the application of geometry concepts to the machining tools, and demonstrated measurement techniques using the tools.

The basic skills instructor frequently summarized and reiterated key points and clarified concepts that seemed to confuse students. The basic skills instructor regularly posed questions to the occupational instructor for further clarification, such as differences in certain cutting techniques and when certain techniques should be used. This resulted in regular dialogue between instructors.

Both instructors responded to student questions, answering depending on their expertise. When answering questions, the basic skills instructor focused primarily on math concepts and terminology, whereas the occupational instructor described tools and measurement strategies with some reference to math concepts.

Observations from an Office Administration Course—Varied Instructional Techniques at WCC

Office Procedures was a required course in the Clerical Assistant sequence. It focused on business English, time and resource management, telecommunications knowledge and phone etiquette, organizational skills, basic financial tasks, administrative skills, oral presentations, business terminology, and conflict resolution in an office setting.

Roughly half the observed class was an interactive large group discussion, with the remainder evenly split among lecture, independent work, and small group work. The lecture component began with the basic skills instructor giving a brief presentation on test-taking strategies. This was followed by a seminar-style discussion led by the occupational instructor. Students had the option to work independently or with a partner on a worksheet activity about time management, followed by group work in which small groups of students were tasked with developing strategies for effective time management given scenarios that were assigned by the instructor. The two instructors were purposeful in drawing out students’ personal experiences and attempting to make them relevant to the course content.

The occupational instructor delivered most of the content, and the basic skills instructor offered additional clarification or examples. The two instructors traded off in assisting students or calling on students to speak during activities. The content was not overly technical, so the occupational and basic skills instructors both spoke about general success strategies, skills, and concepts.


PACE


Observations from a Sustainable Office Skills Course—Integrated Team Teaching at EvCC

The Sustainable Office Skills class was structured around the creation of a portfolio for each individual student. It included both independent and small group work, as well as some interactive content often based on assigned readings.

For the group work, instructors spent a substantial amount of time orienting students to the task and explaining the goal of the project. Students worked in pairs or groups of three to develop a sample question to ask an employer to improve sustainability in the workplace (e.g., “Have you considered using a recycling bin?”). After working in groups, the instructors ask students to share their ideas, and the basic skills instructor defined terms that English language learners may have found difficult to understand. Students worked independently to write individual 30-second elevator speeches about themselves in preparation for job interviews. The occupational skills and basic skills instructors also role-played as job candidate and potential employer, demonstrating interview and job readiness skills.

Instructors were highly integrated in delivering content, with limited differentiation of the basic skills and occupational instructor roles. The two instructors interacted constantly throughout the class, with both clarifying concepts and delivering instructions on activities and assignment expectations. The basic skills instructor asked clarifying questions both to students and to other instructor, provided feedback to student responses, and explained classroom tasks.

• The use of basic skills support courses varied by college and I-BEST program.

As discussed in Chapter 3, in addition to team teaching of basic skills and occupational training, I-BEST programs can also provide separate non-credit basic skills courses designed to complement the occupational courses. In these support courses, the basic skills instructors review occupational material with the students, help them with their homework, and help them prepare for tests and quizzes.

Both EvCC and WCC required I-BEST students to attend support courses concurrent with occupational courses for all their programs. College staff viewed the support courses as way to build basic skills concurrent with occupational training. In addition, for students who did not have a high school diploma or equivalent and were considering pursuing a GED, these courses could help them prepare.

At the outset of the study, BTC also included support courses for all its I-BEST programs. However, in the final year of study enrollment, the college administrators eliminated these courses, reporting that instructors found greater value in increasing team teaching time to reinforce key course content and to address basic skills gaps. For example, administrators were concerned that BTC’s Nursing Assistant students were not being adequately prepared for the state certification test. To address this concern, the basic skills instructors spent more time in the occupational training classes to help ensure comprehension of core course principles and to support increased pass rates on the state exam.


PACE


• College staff reported that instructors with experience in ABE and ESL programs were well suited to serve as basic skills instructors for the team teaching in the I-BEST program.

Occupational instructors must have content knowledge, and many come from the relevant industry, but basic skills instructors can and do come from a variety of backgrounds. College staff reported positive experiences in I-BEST with basic skills instructors who could align their instruction with the basic skills deficit of I-BEST students and that many of these instructors had prior experience teaching ABE and/or ESL classes.

In the interviews conducted for this study, I-BEST basic skills instructors emphasized how the learning styles and required supports of the I-BEST students differed from non-I-BEST students, in large part due to the former’s lower basic skills or more limited educational experiences. For example, staff consistently reported the importance of repeating concepts because I-BEST students were reticent to ask clarifying questions. The I-BEST instructors also noted that the occupational instructors with less prior experience teaching I-BEST students often overestimated what students knew and understood of core concepts, such as math skills for Welding students or basic anatomy for Nursing Assistant students.

• Experienced teaching teams had well-developed roles and teaching strategies for their courses.

As discussed, the I-BEST program emphasizes the importance of a collaborative teaching model in which instructors work together to develop and deliver curriculum, but is less prescriptive in defining their specific roles. The research team observed how instructional roles varied across teaching teams and across the programs in the three colleges. In more established teaching teams (i.e., instructors who worked together in at least one previous course), basic skills instructors took a more active role in the occupational courses. These instructors reported that their earlier work with the occupational instructor gave them a greater understanding of course content, and thus they felt more comfortable taking an active role. Occupational instructors in these teams reported that their earlier work with the basic skills instructor gave them a greater appreciation for the needs of I-BEST students and for adult education techniques.

Both EvCC and WCC had established teaching teams that worked with each other for multiple quarters. These teams noted that they had a clearer shared understanding regarding their roles within the classroom. In some teams, instructors had equal responsibility for delivering course content. Examples included instructors splitting class time, with each addressing a particular topic during a given lecture, or classes where there was less emphasis on lecture and more on jointly facilitating small group work with students. Other established teaching teams adhered to a more traditional division of labor, with the basic skills instructor taking a secondary role.

Instructors also reported that the earlier team experience was beneficial because the basic skills instructor typically then had command of the occupational material, including asking


PACE


relevant clarifying questions, providing examples that were more likely to resonate with students, or providing advice about preparation for upcoming assignments. Additionally, staff reported that the accumulated expertise of these basic skills instructors was helpful when they attended courses with lab components (e.g., Welding, Nursing Assistant), as they were better equipped to provide students with guidance on the technical aspects of the applied coursework.

• Reliance on adjunct instructors resulted in limited planning time for teaching teams.

As discussed in Chapter 1, colleges are reimbursed at 1.75 times the regular rate for a FTE student to help cover the costs of implementing I-BEST, including instructor preparation. In interviews conducted for the PACE study, college staff reported that teaching teams did not have much, if any, preparation time. This was due in part to the adjunct status of the basic skills instructors, who did not have full-time positions with the colleges and who were paid by the course. Though these instructors had experience with ABE and ESL students, initially they often had no experience with the occupational content area of the I-BEST program to which they were assigned. Many of the occupational instructors also were adjunct faculty. This staffing approach is common for community and technical colleges across the country, in large of part to reduce staffing costs.39 However, the reliance on adjuncts resulted in instability in teaching teams over time and reduced opportunities for collaborative course planning and professional development.

College administrators reported that the adjunct staffing model most affected the time instructors might meet in advance of the quarter to discuss their team teaching strategies. Instructors at all three colleges reported meeting in advance of the start of the quarter, but they typically were not paid for this planning time. Staff reported that most teaching teams developed the necessary rapport to effectively deliver the I-BEST program, but that newer teaching teams and those with limited planning time often continued to develop and finalize their approach to team teaching in the early weeks of the quarter.

• Opportunities for instructor training were limited, which most affected the newer teaching teams.

SBCTC provided professional development opportunities for I-BEST instructors across the state at various points during the study, and instructors at the three colleges reported that they attended one or more of these. These opportunities were typically conferences offered annually that featured presentations from experienced I-BEST instructors and SBCTC staff. Examples of presentations included a demonstration of a team teaching approach to content

39 A recent study found 68 percent of instructional staff at public two-year colleges were classified as part-time

(Ginder et al. 2017).


PACE


delivery, a group discussion among instructors of approaches they found to be effective, and information provided by the SBCTC of planned changes to I-BEST program offerings.

Generally, it was the basic skills instructor who attended these professional development conferences; it was rare for both members of a teaching team to have attended. This, combined with the lack of paid preparation time, meant that newly paired instructors had to learn each other’s styles and preferences as the course proceeded. Conversations with instructors and classroom observations suggested that all instructors delivered the core components of the I-BEST model as intended, but it was this learning curve, especially early in the quarter, which made integrating their roles difficult.

• As designed, the I-BEST programs used additional funding during the PACE evaluation to provide “fill-the-gap” financial assistance and student advising through paid “navigators.”

I-BEST staff reported that their students received a range of financial assistance and supports that were not available to other students in the college. This included assistance in completing the Free Application for Federal Student Aid (FAFSA). Many I-BEST students were new to college and had never filled out the FAFSA or applied for other forms of financial aid. Navigators regularly helped new students apply for financial aid for which they were eligible. The I-BEST program then provided “fill-the-gap” funding if the financial aid was not sufficient. These funds also supported books, tools, or work clothes (e.g., steel-toed boots for welding labs, uniforms for nursing assistants during clinical placements), transportation (e.g., bus passes, gas cards), and licensing/certification fees.

The implementation study documented that navigators proactively provided advising to I-BEST students that was not available elsewhere on campus. Each navigator reported regularly meeting with students to discuss class progress, identify potential barriers to participation, and help with career and academic planning. They also occasionally attended I-BEST classes to check in with students. Navigators reported that often, by virtue of relationships that they developed during recruitment for the program, students came to see them as a regular fixture in their campus life. For example, instructors and administrators at BTC characterized the navigator as a “one-stop-shop” who served as both a coordinator and a navigator of student services on campus for I-BEST treatment group members. Another administrator described the navigator role as an “early alert system” to proactively identify potential concerns that would inhibit student success.

• Basic skills instructors provided advising and support, specifically around academic issues.

In addition to the advising and support role played by the navigators, the basic skills instructors at each of the three schools also developed relationships with I-BEST students and provided ongoing support, particularly around academic issues. Though each of the three colleges


PACE


offered tutoring to all students, college administrators reported that I-BEST students often were hesitant to engage with these services and were more inclined to seek the assistance of the basic skills instructors, whom they already knew.

Based on the research team’s classroom observations and interviews with instructors, the basic skills instructors appeared to actively cultivate these relationships, seeking to build trust with students to identify and address their academic needs. Though this support most often occurred in the classroom or in a lab session, the basic skills instructors made themselves available to students outside of class if they required additional academic assistance. Some of the basic skills instructors reported providing tutoring and one-on-one academic support, especially in the beginning of the quarter, for those students who appeared to be struggling with course material. These instructors also provided more informal ongoing supports to students. For example, instructors would regularly remind students about upcoming assignments, discuss positive time management habits, or provide examples of approaches to studying for upcoming tests or quizzes.

• The I-BEST program did not offer structured services to students in finding employment, but some instructors provided informal employment and job search assistance.

SBCTC designed the I-BEST program with an explicit focus on occupational training that could increase the earnings potential of enrolled students. However, none of the three colleges had specific employment supports for I-BEST students. Administrators at one college noted that employment services were a weakness of the I-BEST program at the college, whereas another administrator indicated that the lack of employment services specific to I-BEST reflected the focus on continued training and credentials.

In part because their colleges’ formal employment services were limited, staff reported that basic skills instructors and occupational instructors provided informal, individualized assistance to students as requested. For example, some I-BEST instructors maintained ongoing relationships with employers and mapped instructional content to skills that employers reported were in demand. These were often instructors who had come out of industry and had knowledge of the local labor market. I-BEST instructors provided examples of how they connected students directly with employers. For example, one Welding instructor kept apprised of employers likely to hire students who completed the one-quarter entry-level certificate. The instructors provided applications for employers and coached students on the application process.

4.2. Education and Training Participation Patterns

One of the central objectives of I-BEST is to increase students’ participation in and receipt of workforce and other credentials at two-year colleges, as well as for them to move on to


PACE


additional education and training after I-BEST, as appropriate. For each of the three colleges, this section analyzes participation in the I-BEST programs, credential receipt, progression to additional training after I-BEST, and length of stay in I-BEST as well as other college courses, for students assigned to the study’s treatment group. The analysis, based on program records from SBCTC, reports the overall level of participation and credential receipt over a 24-month follow-up period.

• Welding and Nursing Assistant were the most common programs in which students randomly assigned to the treatment group enrolled.

As discussed in Chapter 2, PACE study participants were randomly assigned to the treatment or control group for the occupational program in which they were interested. For example, a student interested in enrolling in the Welding program at BTC would apply for that specific I-BEST program and would be able to enroll if he or she was randomly assigned to the treatment group. Exhibit 4-1 shows the distribution of treatment group members across I-BEST programs for each of the three colleges and overall. Among those seeking to enroll in I-BEST courses across the three colleges, the Nursing Assistant I-BEST programs had the greatest proportion of treatment group members 36 percent of the 315 treatment group members (25 percent at BTC and 10 percent at EvCC). Another 30 percent of those randomly assigned had applied to I-BEST Welding programs (split evenly at BTC and EvCC). One quarter of treatment group members were randomly assigned to the I-BEST programs related to office and clerical work: Sustainable Office Skills (13 percent at EvCC) and Clerical Assistant (12 percent at WCC). Only a small portion of the treatment group sought training as machinists, electricians, and automotive workers. The largest proportion of the treatment group (50 percent) was randomly assigned at BTC, followed by EvCC (38 percent), and WCC (12 percent).

Exhibit 4-1: Percentage of Treatment Group Members Enrolled in I-BEST, by College and I-BEST Program

I-BEST Program BTC EvCC WCC Total Automotive 1% 1% Clerical Assistant 12% 12% Electrical 5% 5% Nursing Assistant 25% 10% 36% Precision Machining 3% 3% Sustainable Office Skills 13% 13% Welding 15% 15% 30% Total 50% 38% 12% 100% SOURCE: SBCTC program records. NOTES: Sample size is 315 and includes all individuals assigned to the treatment group. Percentages may not sum to total due to rounding.

• Across the three colleges, close to three-quarters of treatment group members randomly assigned to I-BEST programs participated in one.

Of the 315 treatment group members randomly assigned to an I-BEST program, some did not participate in the I-BEST courses. Exhibit 4-2 shows the proportion of the treatment group that


PACE


participated in an I-BEST program; and for those who participated, the average number of credits earned and average length of stay while in I-BEST (in quarters).

As shown in the second column, 73 percent of treatment group members participated in at least one I-BEST course at one of the three colleges (defined as attending at least one team-taught I-BEST course in their chosen occupational area). Almost 90 percent of the WCC treatment group participated in an I-BEST course, whereas 76 percent of BTC students and 64 percent of EvCC students did so. Participation rates were above 80 percent for the Automotive, Electrical, and Welding programs at BTC, the Sustainable Office Skills at EvCC, and the Clerical Assistance program at WCC. The lowest rate of participation (48 percent) was in the I-BEST Welding program at EvCC.

The third column of Exhibit 4-2 shows the number of workforce credits that treatment group members earned while in I-BEST. Across the three colleges, the 230 who participated in I-BEST (as opposed to the 315 in the full sample of treatment group members) earned an average of 14 credits. However, as expected given the different credit requirements and length of each I-BEST program, the number of credits treatment group members earned varied by program. Treatment group members who participated in the two-quarter Precision Machining and Welding programs at BTC earned the largest number of credits (43 and 33, respectively), whereas those who participated in the one-quarter Nursing Assistant program earned among the lowest number (7.1 at BTC and 9.5 at EvCC).

The final column of the exhibit shows the average number of quarters that treatment group members attended in the I-BEST program. This again reflects only the 230 who attended as opposed to all 315 treatment group members randomly assigned. By design, I-BEST programs are short, and students attended 1.5 quarters on average. Again, the number of quarters attending the I-BEST program was longer in those programs with longer requirements (BTC’s Precision Machining and Welding, 2.2 and 2.7 quarters, respectively). In most instances, the average number of quarters students attended is longer than the planned length, reflecting that some students likely did not complete their I-BEST programs in the specified number of quarters and thus enroll in I-BEST courses in a subsequent quarter. This is likely a combination of students needing more time to complete the course requirements associated with a given I-BEST program and students re-enrolling in the same classes a second time after having dropped out of the course prior to completion at an earlier date.


PACE


Exhibit 4-2: Participation Rates in I-BEST Program for Treatment Group Members and Average Credits Earned and Quarters Attended by I-BEST Participants, by College and Program, within 24 Months of Random Assignment

College/Program, Number Randomly Assigned,

and Planned Length of I-BEST Program

Participated in I-BEST Programb

Average Number of Workforce Credits Earned While in I-BEST Programc

Average Number of Quarters Attended in

I-BEST Program Among Those

Randomly Assigned Among Those Participating in I-BEST Bellingham Community College Automotive (N=3) (1 quarter)

100% 18.3 1.0

Electrical (N=15) (1 quarter)

87% 13.1 1.2

Nursing Assistant (N=80) (1 quarter)

71% 7.1 1.1

Precision Machining (N=10) (2 quarters)

60% 43.2 2.2

Welding (N=48) (2 quarters)

81% 32.7 2.7

Subtotal (N=156) 76% 18.3 1.7 Everett Community College Nursing Assistanta (N=33) (1 quarter)

61% 9.5 1.0

Sustainable Office Skills (N=42) (2 quarters)

86% 11.6 1.5

Weldinga (N=46) (1 quarter)

48% 10.5 0.9

Subtotal (N=111) 65% 10.7 1.2 Whatcom Community College Clerical Assistant ) (N=38) (2-3 quarters)

89% 9.1 1.7

Total 73% 14.4 1.5 SOURCE: SBCTC records. NOTES: Sample size is 315 and includes all individuals in the treatment group for second column, and 230 (all individuals who attended at least one team-taught I-BEST course) for the remaining columns. a In the case of the Nursing Assistant and Welding programs at EvCC, the occupational training occurred within a single quarter, but students were first required to attend a pre-I-BEST quarter that focused on ABE and ESL training, immediately prior to the start of the I-BEST program. The proportion who attended this pre-quarter activity is not shown on the table. Program records show that 67 percent of treatment group members enrolled in the Nursing Assistant course at EvCC attended its pre-I-BEST course, whereas 50 percent of Welding enrollees did so. b This is defined as enrolling in at least one team-taught I-BEST course. c This includes a small number of academic credits (0.7) on average across the colleges and programs. As discussed in Chapters 3 and 5, basic skills credits and developmental education credits do not count toward credential attainment and are not included in the PACE analysis.

More than one quarter (27 percent) of treatment group members did not participate in an I-BEST program. However, based on SBCTC records, approximately half of those who did not participate in I-BEST enrolled in at least one other course at one of Washington’s community or technical colleges but did not enroll in I-BEST (not shown).40 The PACE 18-month follow-up survey also asked treatment group members who did not attend any education or training

40 SBCTC records show that no treatment group members participated in I-BEST at any college besides the three

involved in the PACE study.


PACE


(I-BEST or otherwise) for the most important reasons they did not. The most common reasons reported were not having enough time due to work or family, or the belief that they could not get enough financial assistance (not shown).

• Fifty-eighty (58) percent of treatment group members who enrolled in I-BEST obtained a credential, and a large proportion (63 percent) of those who participated in I-BEST attended additional education and training.

A key goal of I-BEST is to provide students with credentials in their area of study. As discussed in Chapter 3, each of the colleges awarded a college-issued certificate for the completion of the I-BEST program, with some programs (e.g., Nursing Assistant, Welding) also requiring additional certification from an outside organization or the state. Exhibit 4-3 reports credentials awarded by each college, based on SBCTC program data; values do not reflect any credentials earned from outside licensing bodies.

As shown in the first column on Exhibit 4-3, based on SBCTC data, 58 percent of treatment group members who participated in I-BEST received a credential by the end of the 24-month follow-up period. Most were workforce credentials (not shown).41 One limitation of these data is that they do not indicate the occupation or type of workforce credential obtained, nor whether it was associated with participation in I-BEST. However, 60 percent of the credentials earned were within the first six months of the follow-up period, (not shown). This suggests that many were earned as a result of completing an I-BEST program.

Credential receipt varied by college and occupational area. More than three-quarters of EvCC I-BEST students who enrolled in an I-BEST program (76 percent) obtained a workforce credential, compared with 55 percent of BTC students and 29 percent of WCC students. Credential receipt rates were 80 percent or above for treatment group members who participated in BTC’s Precision Machining and EvCC’s Nursing Assistant and Welding programs. EvCC had notably higher credential receipt rates than did comparable programs at the other colleges for both Welding (95 percent for EvCC versus 38 percent for BTC) and clerical programs (61 percent for EvCC versus 29 percent for WCC).

41 Of the credentials awarded, 84 percent were workforce credentials (requiring 20 or more credits), 13 percent

were workforce awards (requiring less than 20 credits), and 1 percent were academic credentials.


PACE


Exhibit 4-3: Credential Attainment and Post-I-BEST College Enrollment Among Treatment Group Members Who Participated in the I-BEST Program, Within 24 Months of Random Assignment

College Program Obtained

Workforce Credential

Enrolled in Additional

Courses After I-BEST

Average Number of Quarters

Attended, I-BEST Plus Additional

Courses

Still Enrolled at End of the

Follow-up Period

Bellingham Technical College Automotive (N=3) 33% 67% 5.0 0% Electrical (N=13) 31% 77% 4.3 23% Nursing Assistant (N=57) 70% 53% 3.7 28% Precision Machining (N=6) 83% 50% 4.0 0% Welding (N=39) 38% 72% 5.3 18% Subtotal (N=118) 55% 62% 4.4 22% Everett Community College Nursing Assistant (N=20) 80% 55% 3.4 20% Sustainable Office Skills (N=36) 61% 67% 4.0 25% Welding (N=22) 95% 55% 3.1 9% Subtotal (N=78) 76% 60% 3.6 19% Whatcom Community College Clerical Assistant (N=38) 29% 76% 4.5 32% Total 58% 63% 4.1 23% SOURCE: SBCTC program records. NOTES: Sample size is 230 and includes all individuals who attended at least one team-taught I-BEST course.

The second column of Exhibit 4-3 shows that a high proportion of treatment group members who participated in I-BEST programs continued their education beyond I-BEST, with 63 percent enrolling in additional college courses following their enrollment in I-BEST. This rate was similar at BTC and EvCC (about 60 percent) but higher at WCC (76 percent). Students in the clerical programs at both EvCC and WCC participated in additional training at a relatively high rate (more than two-thirds did so); whereas the rate was lower for both Nursing Assistant programs and BTC’s Precision Machining (about 50 percent). Again, there were differences in the two Welding programs, with 72 percent of those participating in additional training in the BTC program and 55 percent doing so at EvCC.

The variation in credential receipt rate, as well as the high rate of progression to additional education and training (in some cases greater than receiving a credential through I-BEST), results from I-BEST students not completing all of the course requirements but still continuing on to higher or perhaps different courses of study. Staff reported some staff and students focused on progressing to the next level of training. That is, students may not have completed all the requirements to obtain the college-issued credential, but still met the prerequisites for proceeding to the next level of training. Staff also reported that some students changed their


PACE


focus or interests while attending I-BEST, and thus continued without completing an I-BEST credential.

In addition, because of the labor market, some programs de-emphasized obtaining the short-term college-issued certificates of I-BEST completion and encouraged progression to higher levels of training. For example, EvCC’s Welding program focused on helping students attain an industry-recognized credential after the one-quarter program because there were entry-level production welding job opportunities (hence its credential receipt rate of 95 percent). By comparison, BTC’s Welding faculty focused less on near-term certifications, instead they believed that students would be most employable in the local labor market after a full year of training. As a result, BTC Welding students were more likely to continue in welding without necessarily receiving an occupational credential; 72 percent of those who attended the BTC I-BEST Welding program progressed to additional education and training. Similarly, WCC staff encouraged their Clerical Assistant students to pursue education and training beyond an entry-level clerical position (with only 29 percent receiving the I-BEST credential and 76 percent attending subsequent education and training).

Based on SBCTC program records, for the 230 treatment group members who attended an I-BEST program, the average length of participation for all education and training programs (including I-BEST) over the 24-month follow-up period was 4.1 quarters.42 Length of stay was more than five quarters for those who had attended BTC’s Automotive and Welding I-BEST programs, and lowest (about three quarters) for those who had attended EvCC’s I-BEST Nursing Assistant and Welding programs.

At the end of the 24-month follow-up period, about one-quarter (23 percent) of the treatment group that attended I-BEST was still participating in at least one course at an SBCTC college. This proportion was lowest at EvCC (19 percent) and highest at WCC (32 percent). The proportion still participating was relatively high at both clerical programs, as well as at BTC’s Electrical and Nursing Assistant programs. No students who participated in BTC’s Automotive or Precision Machining I-BEST programs were still participating at the end of the follow-up period.

4.3. Impact on Receipt of Services

The previous section focused on treatment group member experiences, using SBCTC data. This section focuses on the degree to which I-BEST increased receipt of education and training, advising, and employment services and is based on data from the 18-month survey. An implication of the career pathways framework is that any improvements in the main outcomes

42 This length of participation is truncated because some students were still attending programs at the end of the

follow-up period.


PACE


(discussed in Chapter 5) will result primarily from impacts on the treatment group’s experiences and services tied to education and training.

The purpose of this section is to compare levels of education and training receipt between the treatment and control groups—that is, the service differential that is expected to lead to impacts. (The box below briefly explains how to read impact tables.) Specifically, the following section discusses impacts on education or training receipt after random assignment (Exhibit 4-4) and receipt of advising and employment services (Exhibit 4-5).

How to Read Impact Tables

Exhibit 4-4 and Exhibit 4-5, as well as exhibits in Chapter 5, list the outcome measure in the analysis in the left-most stub column (Outcome), with the unit of that outcome in parentheses (e.g., “(%)”).

The next column (Treatment Group) presents the treatment group’s regression-adjusted mean outcome, followed in the next column by the control group’s actual mean outcome (Control Group). The regression adjustments correct for random variation in baseline covariates between the two groups (and thus differ slightly from the raw means). The next column (Difference) lists the impact—that is, the difference between the treatment and control group means. The Standard Error column is a measure of uncertainty in the estimated impact that reflects both chance variation due to randomization and any measurement error. The final column, p-Value, is the probability that observed or larger difference between the treatment and control group would occur by chance, even if there was no difference in the characteristics of the two groups. There are several common standards for judging statistical significance. In this report, tests are considered statistically significant and highlighted in tables if the p-value is less than or equal to .10. Tests with smaller p-values are separately flagged:

* for .10 (ten percent level) ** for .05 (five percent level) *** for .01 (one percent level)

Outcomes in italics apply to a subset of survey respondents (e.g., those who attended education or training). These estimates are not impacts, but unadjusted, non-experimental comparisons.

• I-BEST had a statistically significant impact on education and training receipt.

Exhibit 4-4 shows statistically significant impacts on study participants’ receipt of any education and training activities based on responses to the 18-month follow-up survey. I-BEST increased receipt of training in any subject by 23 percentage points (76 percent of treatment group versus 53 percent of control, statistically significant at the one percent level). Though the only I-BEST healthcare training program was Nursing Assistant (at BTC and EvCC), it was one of the largest, and overall it increased receipt of healthcare-related training by 13 percentage points (29 percent versus 16 percent, statistically significant at the one percent level).43

43 These proportions represent the percentage of treatment and control group members who reported on the

follow-up survey that they participated in an education or training program at the three colleges or elsewhere. Though similar, this value differs from the proportion who participated in a program based on administrative data. This difference is due to variation in the data source (self-reported measures are subject to recall error).


PACE


Exhibit 4-4: Education or Training Receipt After Random Assignment

Outcome Treatment Group

Control Group Difference Standard

Error p-Value

General Aspects of Education & Training Receipt Received education or training since random assignment (%)

In any subject/field 76.0 53.3 +22.7 *** 4.5 <.001 In a healthcare occupation 28.9 15.6 +13.3 *** 3.5 <.001

Since random assignment, ever attended (%) Two-year college 70.7 45.9 +24.8 *** 4.5 <.001 Four-year college 1.5 1.5 +0.0 1.3 .978 Proprietary school 1.2 0.9 +0.4 0.9 .665 Adult high school/education 0.9 0.5 +0.4 0.9 .675 Community/nonprofit organization 0.9 1.7 –0.8 1.2 .522 Other 4.5 6.9 –2.4 2.0 .224

Time spent at school and work at first place attended (%) Full-time school and full-time work 5.0 2.5 +2.5 2.1 .231 Full-time school with no or part-time work 42.2 47.4 –5.2 6.1 .392 Part-time school and full-time work 11.1 8.4 +2.7 3.6 .453 Part-time school with no or part-time work 41.7 41.7 +0.0 6.1 .999 Total 100.0 100.0

Views of classes at first place attended (%) Strongly agrees relevant to life/career a 57.3 43.8 +13.6 ** 6.1 .027 Used active learning methods most/all of the time b 39.1 32.7 +6.4 5.8 .273 Perceived strong emphasis on community at first place of instruction (%)

9.0 14.5 –5.5 4.1 .180

Basic Skills Instruction and Tests Received basic skills instruction since random assignment (%)

Academic skills 33.8 26.2 +7.6 * 4.4 .081 English as a Second Language 18.1 12.5 +5.6 * 3.2 .076

Took college placement exam (%) English 51.7 43.3 +8.4 * 4.7 .076 Math 48.5 44.4 +4.1 4.7 .387

Passed college placement exam (%) English 39.3 31.3 +7.9 * 4.7 .090 Math 33.4 30.2 +3.2 4.4 .466

Life Skills Instruction Received life skills instruction since random assignment (%)

26.8 24.3 +2.5 4.0 .538

Sample size (full survey sample) 238 225 SOURCE: Abt Associates calculations based on data from PACE short-term follow-up survey. NOTES: Where not italicized, outcomes apply to the full survey sample, and impact estimates are fully experimental and regression-adjusted. Outcomes in italics apply to subset of survey respondents (e.g., those who attend education or training)—for these estimates, between-group differences are unadjusted, non-experimental comparisons. Statistical significance levels, based on two-tailed t-tests of differences between research groups, are summarized as follows: ***statistically significant at the one percent level; ** at the five percent level; * at the ten percent level. a Percentages who either strongly agreed that classes were relevant to career interests or who strongly disagreed that classes did not relate to anything else in life. b Refers to first place of instruction if went to more than one place. Gives the average percentage who described classes as involving each of a series of active learning approaches at least often, or at least most of the time (items used different scales, as discussed in Appendix C).

In addition, treatment group members who did not enroll in the I-BEST program may have enrolled in other education and training programs in the community.


PACE


The program produced an eight percentage point impact on receipt of basic skills instruction (34 percent of treatment group members versus 26 percent of control group members, statistically significant at the 10 percent level). Given that the I-BEST program allowed members of the treatment group to enroll in credit-bearing courses without remediation, this difference could be due to respondents in the treatment group interpreting this question to apply to basic skills support classes required for many I-BEST certificates, or the basic skills instruction they received through the team teaching approach. There was also an eight percentage point impact on completion of a college placement exam in English (52 percent of treatment group members versus 43 percent of control group members, statistically significant at the 10 percent level). There was no statistically significant impact on the percentage reporting haven taken a math placement exam. I-BEST significantly increased the likelihood of passing a college placement exam in English, but had no effect in math.

Exhibit 4-4 also shows non-experimental comparisons for the subset of treatment and control group survey respondents who reported attending any education and training (see italicized rows). Treatment group members attending an education and training program were more likely than the control group to agree that the courses at the first place they attended were relevant to their life/career (57 percent versus 44 percent, statistically significant at the five percent level).

• I-BEST had statistically significant impacts on receipt of advising and employment services.

Exhibit 4-5 shows impacts on advising and employment services for all treatment and control group members, regardless of whether they received services or not. Reflecting the dedicated advisor that was part of I-BEST, the program increased receipt of career counseling by 14 percentage points (36 percent of the treatment group members versus 22 percent of control group members, statistically significant at the one percent level). There were similar impacts on receipt of help arranging for supports for school, work, or family (29 percent of the treatment group versus 16 percent of the control group, statistically significant at the one percent level). Though there were no official job search assistance services, I-BEST produced an impact on receipt of job search or placement services (29 percent of the treatment group versus 19 percent of the control group, statistically significant at the one percent level), potentially reflecting the assistance received from instructors or the advisor.

Though these impacts are relatively large, only a minority of the treatment group reported receiving advising or employment services. Roughly two-thirds of the treatment group did not report this support.


PACE


Exhibit 4-5: Receipt of Various Supports Within 24 Months of Random Assignment

Outcome Treatment

Group Control Group Difference

Standard Error p-Value

Received assistance from any organization since random assignment (%) Career counseling 36.1 22.3 +13.8 *** 4.2 .001 Help arranging supports for school/work/family 28.9 16.4 +12.5 *** 3.7 <.001 Job search or placement 29.3 19.0 +10.3 *** 3.9 .008

Received supports at first place of instruction attended (%) Career counseling

Ever 26.1 27.4 –1.3 5.5 .814 Three or more times 13.3 7.8 +5.5 3.6 .129

Academic advising Ever 62.3 57.0 +5.3 6.1 .384 Three or more times 40.2 41.3 –1.1 6.1 .857

Financial aid advising Ever 54.0 48.4 +5.6 6.2 .366 Three or more times 19.0 23.5 –4.4 5.1 .384

Tutoring Ever 41.9 37.6 +4.3 6.0 .475 Three or more times 34.3 33.5 +0.8 5.9 .894

Help arranging supports for school/work Ever 23.2 24.6 –1.4 5.3 .792 Three or more times 13.4 10.5 +2.9 3.9 .461

Job search/placement assistance Ever 25.6 22.3 +3.3 5.2 .525 Three or more times 10.7 9.2 +1.4 3.5 .687

Received financial assistance at first place of instruction (%) a Grant/scholarship 80.2 67.9 +12.3 ** 5.4 .023 Loan 19.0 10.9 +8.1 * 4.2 .055

Cited financial support as challenge in enrollment or persistence (%) b

56.6 58.9 –2.2 4.7 .636

Offered opportunities for related work experience as part of training at first place of instruction (%) Clinical internship 36.5 23.3 +13.2 ** 5.5 .017 Visits to local employer 25.1 19.1 +6.0 5.0 .228 Work-study job 37.9 25.6 +12.3 ** 5.6 .029 Apprenticeship 14.9 11.2 +3.7 4.0 .360 Any related work experience (including other) 64.7 49.0 +15.6 ** 6.0 .010

Sample size (full survey sample) 238 225 SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. Where not italicized, outcomes apply to the full survey sample, and impact estimates are fully experimental and regression-adjusted. Outcomes in italics apply to subset of survey respondents (e.g., those who attend education or training)—for these estimates, between-group differences are unadjusted, non-experimental comparisons. Statistical significance levels, based on two-tailed t-tests of differences between research groups, are summarized as follows: ***statistically significant at the one percent level; ** at the five percent level; * at the ten percent level. a Reported receiving grant or loan to help cover either tuition/school expenses or living expenses. b Cited financial support challenges as a reason for non-enrollment or leaving school or as a difficulty while attending school.

Exhibit 4-5 also shows non-experimental comparisons for a subset of treatment group members who attended any education or training program. Reflecting the fill-the-gap financial assistance advising, among students who enrolled in a program, a significantly higher proportion of treatment group members reported receiving grants or scholarships (80 percent) than did control group members (68 percent).


PACE


Among those who attended, there were also positive impacts related to work experience as part of the training. Treatment group members were more likely than control group members to report participating in any work-related experience (65 percent versus 49 percent), as well as in a clinical internship (37 percent versus 23 percent) or a work-study job (38 percent versus 26 percent).

4.4. Summary of Implementation Findings

Though there was variation across the three colleges and within programs at each college, each college delivered a version of I-BEST that incorporated the key features of the model as designed by the SBCTC. This included increased access to college-level classes for students with low basic skills or limited English proficiency, team-taught classes providing both occupational training and basic skills instruction, and proactive services related to advising and financial assistance. There was variation across the occupational programs in how they implemented the team teaching and the extent to which additional support classes providing basic skills instruction were added to the occupational training. In general, the I-BEST approach was more challenging for new staff members who had limited experience with team teaching and/or the specific course content; they experienced a learning curve and had to adapt their styles and approaches as the course progressed. The established teaching teams tended to have clearly defined roles and strong working relationships.

Administrative data show that more than 70 percent of treatment group members participated in at least one I-BEST course. Among those who participated, 58 percent of treatment group members received some type of credential, mostly commonly a workforce credential requiring 20 more credits; 63 percent of the treatment group participated in additional college classes beyond I-BEST (with substantial differences across colleges and occupational programs). The high rate of progression to additional education and training (even in the absence of credential receipt) indicated some I-BEST students did not complete all of the course requirements for a given I-BEST program, but still continued on to more advanced or perhaps different courses of study. Interviews conducted for the study indicated some students may not have completed all the requirements to obtain the college-issued credential, but may have met the course prerequisites for proceeding to the next level of training without the short-term credential. Staff also reported that some students changed their focus or interests while attending I-BEST and thus continued in a different course of study without completing I-BEST. In addition, because of college staff’s perceptions of the local labor market, some programs de-emphasized obtaining the short-term, college-issued certificates after I-BEST. In such cases, the focus was instead on progression to higher levels of training or degree programs, even if it meant not obtaining the shorter term credential.


PACE


I-BEST in the three colleges had significant effects on participants’ receipt of education and training, as provided by the 18-month follow-up survey. The I-BEST program produced a 23 percentage point difference in the proportion of treatment group members who received training in any subject compared with the control group (76 percent versus 53 percent). There was a similarly large effect on enrollment in two-year colleges overall. Treatment group members also received increased advising and supports at their colleges, including career counseling; help arranging supports for school, work, or family; and receipt of job search or placement services. Though these impacts were 13 or 14 percentage points, the overall proportion of treatment group members who reported receiving these services was still relatively low.


PACE

Abt Associates 5. Early Impacts of I-BEST ▌pg. 58

Early Impacts of I-BEST

This chapter reports estimates of I-BEST’s early impacts on educational attainment, career progress, and a set of non-economic outcomes across the three study community colleges. The main estimates are based on administrative data and cover impacts over a 24-month period after random assignment ended for the full sample of 632 study participants. Additional estimates are based on data from the 18-month follow-up survey, which had a response rate of 73 percent.

The chapter begins by describing hypothesized impacts and the various outcomes that the team analyzed. Subsequent sections in this chapter present findings on education, career progress, and non-economic outcomes, respectively. Each of these sections distinguishes between confirmatory, secondary, and exploratory analyses.

5.1. Key Hypotheses and Outcomes

Through the combination of team teaching strategies that paired basic skills instructors and occupational training instructors for at least 50 percent of occupational training class time, support classes, advising, and financial assistance, I-BEST aimed to increase enrollment, credit attainment, credential completion, and career knowledge. In the longer term, I-BEST positioned students for more advanced credentials and career-track employment. Each I-BEST program was part of a career ladder, and students could exit and return or continue their education. Subsequent reports will focus on additional education and employment outcomes.

The research team delineated outcomes as confirmatory, secondary, and exploratory. Exhibit 5-1 lists and describes each outcome.

The confirmatory outcome in the I-BEST analysis is the total number of academic and workforce credits earned. Academic credits are attached to courses that are suitable for academic transfer to another college, such as English Composition I and Introduction to Statistics. Workforce credits are attached to training courses that are categorized as occupational in nature and include I-BEST courses such as Spreadsheets I and Applied Mechanics.44 The research team determined that these two types of earned credits were the best educational indicator as to whether I-BEST was effective. The receipt of these credits reflects the ability of the treatment

44 Approximately 93 percent of all I-BEST courses are categorized as occupational training courses according to

SBCTC course records. The remaining I-BEST courses are categorized as basic skills, developmental, or academic courses. Among all courses taken by students in the sample, 87 percent of all academic courses were worth five credits, while the vast majority of workforce courses ranged from one to five credits, with five credits being the most frequent occurrence (31 percent).


PACE


group to engage in the array of academic and occupational activities that can support longer-term career success. (See Definitions of Types of Courses and Credentials box below.)

Secondary analyses included tests of hypotheses for additional education outcomes, including certificate and degree completion, and enrollment in various categories of courses at SBCTC colleges, as well as separately delineated academic and workforce credits earned. Secondary analyses also included measures of career progress.

Finally, exploratory analyses provided additional evidence on the program’s impacts on a number of educational and social outcomes. Educational outcomes include measures at six-month intervals after random assignment of whether students continued to enroll in basic skills and developmental courses, whether students enrolled specifically in developmental math and reading courses, and whether they completed college-level coursework in algebra or English (“gatekeeper” courses).45 As I-BEST is a diversion for many students who otherwise would have faced basic skills or development courses that do not result in college credit, these tests will examine whether I-BEST diminished flows into this coursework, and instead diverted students into college-level coursework. Additionally, the research team expected I-BEST to have some impact on insights, habits, and functioning in a number of domains, including perceived measures of psycho-social skills, life stressors, and family structure.

Exhibit 5-1: Outcomes in the Impact Analysis

Outcome Operationalization Source Sample Size46

Treatment Control Confirmatory Outcome Total number of academic and workforce credits earned

Within 24 months of random assignment

SBCTC records 315 316

Secondary Outcomes: Educational Achievement Received a certificate or degree from a college

Within 24 months of random assignment


Enrolled in college academic courses, occupational training courses, and any college courses

Earned within: Months 1-6; Months 7-12; Months 13-18; Months 19-24; Any month


Cumulative number of academic credits, workforce credits, total academic and workforce credits

Earned by end of: Month 6; Month 12; Month 18; Month 24


Received an occupational credential

Within 18 months of randomization

SBCTC records, PACE short-term follow-up survey

SBCTC: 315 Survey: 238

SBCTC: 316 Survey: 225

45 Developmental math and reading courses, as well as college-level algebra and English courses, were flagged

using specific Classification of Instructional Programs (CIP) codes in this analysis to distinguish the courses that are traditionally categorized as being in the remedial course sequence and “gatekeeper” course framework.

46 One student who was randomized to the control group left the sample at the time that outcomes were measured.


PACE



Treatment Control Secondary Outcomes: Self-Assessed Career Development Perceived career progress 3-item scale of self-assessed

career progress; response categories range from 1=strongly disagree to 4=strongly agree

PACE short-term follow-up survey

238 225

Confidence in career knowledge

7-item scale of self-assessed career knowledge; response categories range from 1=strongly disagree to 4=strongly agree


238 225

Access to career supports 7-item scale counting number of types of career-supportive relationships in workforce and education settings; ranging from 0 to 7 possible types; response categories are 0=no or 1=yes


238 225

Secondary Outcomes: Employment Success Working in a job paying at or above specified wage

Earning $12/hour or morea PACE short-term follow-up survey

238 225

Working in a job requiring at least mid-level skills

Employed in a job requiring calibrated set of skills based on federal standardsb


238 225

Exploratory Outcomes: Education Success Enrolled in basic skills or developmental education courses

Enrolled in: Months 1-6; Months 7-12; Months 13-18; Months 19-24; Any month


Enrolled in developmental any math course


Enrolled in any developmental reading/writing course


Enrolled in and completed college-level algebra course


Enrolled in and completed college-level English course


Exploratory Outcomes: Psycho-Social Skills, Life Stressors, and Family Structure Grit 8-item scale capturing

persistence and determination; response categories range from 1=strongly disagree to 4=strongly agree


238 225

Academic self-confidence 12-item scale; response categories range from 1=strongly disagree to 6=strongly agree


238 225

Core self-evaluation 12-item scale; response categories range from 1=strongly disagree to 4=strongly agree


238 225


PACE



Treatment Control Social belonging in school 5-item scale capturing sense of

belonging; response categories range from 1=strongly disagree to 4=strongly agree


238 225

Financial hardship 2-item scale reported as either an inability to pay rent/mortgage or not enough money to make ends meet; response categories are 0=no and 1=yes


238 225

Life challenges 7-item scale capturing life challenges that interfere with school, work, or family responsibilities; response categories range from 1=never to 5=very often


238 225

Perceived stress 4-item scale capturing perceived stress; response categories range from 1=never to 4=very often


238 225

Living with spouse Response categories are 0=no and 1=yesc


238 225

Had child since random assignment / currently pregnant

Response categories are 0=no and 1=yesd


128 144

a Threshold selected because it was close to the 60th percentile of hourly wages among employed control group members. b Skill levels based on the federal O*NET system, with thresholds targeted to PACE program target occupations. Occupational categories were coded for PACE by Census Bureau staff from standard open-ended survey items. c Living with an unmarried partner is not counted as living with a spouse. d Asked of women only.


PACE


Definitions of Types of Courses and Credentials

Courses:

• Academic: Credit-bearing courses that are transferrable to four-year colleges, and are not considered basic skills or developmental. Most general education requirements fit this category.

• Workforce: Credit-bearing occupational training courses that focus on specific technical skills.

• Basic skills: Non-credit Adult Basic Education courses, GED coursework, and English as a Second Language courses.

• Developmental education: Also known as pre-college or remedial education; non-credit courses outside of the basic skills courses listed above, that are specifically designed to prepare students for college-level algebra and English courses.

• College-level algebra and English: Academic courses that are designated as having a course number of 100 or above with specific Classification of Instructional Programs (CIP) codes of 23 (algebra) and 27 (English).

Credentials:

• Workforce award: A certificate that requires 20 or more credits to complete (e.g., Vehicle Service Technician Certificate) or an Associate’s in Applied Science (AAS) degree (e.g., AAS in Welding Technology). The AAS is not fully transferrable to a four-year bachelor’s program.

• Workforce completion: A certificate that requires less than 20 credits to complete.

• Associate’s of Arts (AA) or Associate’s of Science (AS): Degrees that usually require two years of coursework, and are usually transferrable to any public university, as well as to many private universities.

5.2. Impacts on Educational Attainment

This section presents impact estimates for key measures of educational progress for the I-BEST sample, aggregated across the three colleges. It first highlights the confirmatory test to determine whether early impacts are on the expected path. Next, it examines findings for the secondary and exploratory education outcomes.

• I-BEST increased the total number of academic and workforce credits earned at colleges (confirmatory hypothesis).

Exhibit 5-2 shows that I-BEST had a 13-credit impact on academic and workforce credits earned. Over the 24-month period, treatment group members completed 24 credits compared with 11 credits for the control group. Both types of credits are college-level credits (i.e., non-remedial and applicable towards credentials), and the accumulation of these credits is a positive indicator of academic progress, either towards a credential or, for academic credits, towards transferring to four-year programs in the future. The majority of the credits (84 percent) earned were workforce credits.47

47 The average academic and workforce credits received by treatment students is 24.3 (see Exhibit 5-2) and 20.5

of those credits are workforce credits. The same figures for control group students are 11.2 and 8.2, respectively (73 percent).


PACE


This finding demonstrates that students with measurably low skills can earn college-level credits through I-BEST. A closer look at enrollment (the top panel of Exhibit 4-4) shows that the treatment/control difference in credits earned is driven primarily by more treatment group than control group members enrolling in college.

Exhibit 5-2: Early Impacts on Education Outcomes (Confirmatory Hypothesis)

Confirmatory Outcome Treatment Group

Control Group

Impact (Difference)

Standard Error

Effect Size p-Value

Total number of academic and workforce credits earned at colleges within 24 months

24.3 11.2 +13.1*** 2.2 +0.47 <.001

Sample size 315 316 SOURCE: Abt Associates calculations based on data from SBCTC records. NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between research groups, are summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level.

• The I-BEST program increased credential completion, particularly completion of workforce awards.

Beyond earning credits, a key I-BEST objective is for students to obtain a credential. Community college completion rates, even for programs that result in a short-term certificate,48 are typically low (Shapiro et al. 2014).49 Many students have competing demands in their lives, and finishing a program with a credential is a goal that can be derailed by various circumstances.

With this in mind, it is particularly encouraging to see I-BEST increase the completion of any credential within the 24-month follow-up period by 32 percentage points, from 12 percent to 44 percent (Exhibit 5-3). The vast majority of the students completed a workforce award, which could include an AAS degree. I-BEST increased the share of study participants who received workforce awards from eight percent to 41 percent. I-BEST also increased the attainment of the workforce completion credential by four percentage points, from five percent to nine percent. The small share of study participants who completed this last certificate reflects the fact that I-BEST primarily provided programs requiring more than 20 credits to obtain a credential.

I-BEST did not have an impact on completion of associate’s degrees transferrable to a four-year college. This is not surprising, as such degrees usually take more than two years to complete—longer than the follow-up period for this report. In addition, completion of a transferrable AA or AS degree was not an intended short-term outcome of I-BEST.50

48 Defined as any non-degree credential that takes less than one year of full-time study. 49 A 2014 study from the National Student Clearinghouse finds that students who started at a two-year public

institution had two-year award completion rates of about 30 percent within eight years. Approximately 56 percent of students who started at a four-year public institution received an award within eight years.

50 This is in contrast to the AAS degrees, which are non-transferrable to four-year bachelor’s programs and are often the next step in the pathway of several of these programs.


PACE


The impact on credentials completed was driven primarily by a higher completion rate among treatment group members enrolling in college than among enrolling control group members. That is, among students who enrolled in any course, 49 percent of treatment group members received a credential from the college (44 percent divided by 90 percent) versus 18 percent of control group members (12 percent divided by 68 percent). This difference of 31 percentage points for enrolled students is comparable to the impact on credential receipt reported above for the full sample. Thus, it appears that the I-BEST program was more effective in engaging students in programming that resulted in a credential than were other education and training opportunities in the community.

The 18-month follow-up survey results on occupational credential receipt (the bottom panel of Exhibit 5-3) show that in addition to its impact on receiving a certificate from a SBCTC college (31 percentage points), I-BEST had a 15 percentage point impact on receiving an occupational credential from a licensing or certification agency. Almost a third of the treatment group (32 percent) received such a credential, compared with 17 percent of the control group. Combining credential receipt from a college and a licensing or certification agency, almost three-quarters of the treatment group received a credential within 18 months of random assignment, compared with less than a quarter of the control group.

Exhibit 5-3: Early Impacts on Education Outcomes (Secondary Hypotheses)


Control Group

Impact (Difference)

Standard Error

Effect Size p-Value

Received a certificate or degree from a college within 24 months (N=631)

Workforce completiona Workforce awardb Associate’s degree or higher Any credential

8.5 41.3 0.3

44.2

4.7 7.9 0.3

12.0

+3.7** +33.3***

0.0 +32.1***

1.9 3.1 0.4 3.3

+0.16 +0.86 –0.01 +0.78

.026

<.001 .535

<.001 Enrolled in college (%) (N=631)

Academic courses Months 1-6 Months 7-12 Months 13-18 Months 19-24 Any month

Occupational training courses Months 1-6 Months 7-12 Months 13-18 Months 19-24 Any month

Any coursesc Months 1-6 Months 7-12 Months 13-18 Months 19-24 Any month

14.7 16.0 14.2 10.0 31.8

73.3 40.0 22.5 18.6 80.5

85.3 52.5 36.6 27.7 89.6

15.2 14.9 11.4 7.0

25.6

21.5 21.2 17.1 16.5 39.6

58.2 40.2 29.1 22.8 67.7

–0.5 +1.1 +2.8 +3.0* +6.1**

+51.7*** +18.8*** +5.4** +2.2

+40.9***

+27.1*** +12.3*** +7.5** +5.0*

+21.9***

2.8 2.8 2.6 2.2 3.5

3.4 3.6 3.1 3.0 3.6

3.4 3.9 3.7 3.4 3.2

–0.01 +0.03 +0.09 +0.11 +0.14

+1.20 +0.42 +0.14 +0.06 +0.91

+0.63 +0.25 +0.16 +0.11 +0.55

.570 .347 .142 .090 .041

<.001 <.001

.041

.237 <.001

<.001 <.001

.022

.075 <.001


PACE



Control Group

Impact (Difference)

Standard Error

Effect Size p-Value

Cumulative credits (N=631) Academic credits through

Month 6 Month 12 Month 18 Month 24

Workforce credits through Month 6 Month 12 Month 18 Month 24

Academic and workforce credits through

Month 6 Month 12 Month 18 Month 24

0.9 1.8 2.9 3.8

6.9 13.3 16.8 20.5

7.8 15.1 19.7 24.3

0.8 1.7 2.5 3.0

1.4 3.7 6.0 8.2

2.1 5.4 8.5

11.2

+0.1 +0.1 +0.4 +0.7

+5.5*** +9.6***

+10.8*** +12.4***

+5.7*** +9.7***

+11.2*** +13.1***

0.2 0.4 0.5 0.7

0.5 1.0 1.5 2.0

0.5 1.1 1.7 2.2

+0.05 +0.02 +0.06 +0.09

+0.92 +0.73 +0.57 +0.49

+0.87 +0.68 +0.53 +0.47

.263 .389 .239 .129

<.001 <.001 <.001 <.001

<.001 <.001 <.001 <.001

Received an occupational credential through 18 months from

A college (%) (N=631) Other place (%) (N=463) Licensing or certification agency

(%) (N=463) Any place/authority (%) (N=463)

40.6 1.0

32.0

51.5

9.8 4.3

16.9

23.9

+30.8*** –3.3

+15.1***

+27.6***

3.1 1.4 3.8

4.2

+0.78 –0.22 +0.37

+0.60

<.001 .989

<.001

<.001 Sample size 315 316 SOURCE: Abt Associates calculations based on data from SBCTC records and from the PACE short-term follow-up survey. NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between research groups, are summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. a Certificates requiring 1-19 credits. b Certificates requiring 20 or more credits, including Associate’s in Applied Sciences (AAS) degrees. c Any academic, workforce, basic skills, or developmental courses.

The size of the impact on credential receipt is similar or larger than the impact estimates from previous studies based on quasi-experimental designs (Jenkins, Zeidenberg, and Kienzl 2009; Zeidenberg, Cho, and Jenkins 2010). That the large impact sizes hold after using a rigorous research design provides strong evidence of a causal link between access to I-BEST and credential receipt.

• Includes I-BEST substantially increased enrollment in college courses, driven primarily by enrollment in occupational training.

The research team examined I-BEST’s impact on enrollment in academic and occupational training courses within 24 months and by six-month intervals. The latter was intended to determine how the program’s impact changes over time. The results indicate that for academic course enrollment, there was a cumulative six percentage point increase in enrollment over the 24 months (see Exhibit 5-3). This suggests that while I-BEST offered occupational training, it still spurred academic course enrollment among this population of disadvantaged students. Exhibit


PACE


5-4 shows academic course enrollment for each six-month period, in which the impact increases over time.

Exhibit 5-4: Academic Course Enrollment over Time

However, the majority of students pursued occupational training. I-BEST increased enrollment in credit-bearing occupational courses by 41 percentage points (Exhibit 5-5). This makes sense, given that all I-BEST courses fall into this workforce category, and most treatment group students enrolled in I-BEST courses. About four in five treatment group members (80 percent) enrolled in occupational training courses over the 24-month period (Exhibit 5-3), compared with 40 percent in the control group. Exhibit 5-5 shows enrollment in six-month increments. The impacts are particularly strong in months one through six (52 percentage points), indicating that more students in the treatment group enrolled in occupational courses quickly, compared with their control group counterparts.


PACE


Exhibit 5-5: Occupational Training Course Enrollment over Time

Overall, when accounting for all courses in college, including basic skills and developmental courses, I-BEST increased enrollment by 22 percentage points within the 24 months following random assignment (Exhibit 5-3). Almost 90 percent of the treatment group enrolled in some coursework within this timeframe, compared with 68 percent of the control group.

• I-BEST’s impact on cumulative credits increased over time, though the difference in enrollment between treatment and control declined.

As noted earlier, I-BEST increased completion of academic and workforce credits over the 24 month follow-up. This impact was driven by enrollment in and completion of occupational courses, as there is no overall impact on credits earned in academic courses. Over time, the gap in credit completion grew between the treatment and control groups. As the third panel in Exhibit 5-3 shows, through month six, the impact on cumulative workforce credits was about six credits; the impact increased to about 10 credits by month 12, 11 credits by month 18, and 12 credits by month 24—all results are statistically significant at the one percent level.

Conversely, I-BEST’s impact on course enrollment faded over time, as seen in the second panel of Exhibit 5-3. The impact on enrollment in any course was at its highest in the first six months at 27 percentage points, and dropped to five percentage points by months 19 through 24 (statistically significant at only the 10 percent level). A possible explanation for the decline in impact size over time is that I-BEST has a strong focus on course enrollment at the front end. Another explanation is that I-BEST students may have completed their courses faster than their


PACE


control group counterparts. The control group may have taken longer to complete the necessary developmental or ABE coursework to get into college-level coursework, thereby depressing the enrollment impact over time. Declining impacts on course enrollment and increasing impacts on course completion over time, with positive impacts on credentials received, is what we might expect from a career pathways program designed to provide initial training that leads to employment.

• The I-BEST program’s impact on enrollment in basic skills and developmental courses faded over time. Though there was no impact on developmental course enrollment by subject, there were positive impacts on both enrollment in and completion of college-level algebra and English courses.

The results in the first panel of Exhibit 5-6 indicate that there was a 19 percentage point increase in the enrollment of basic skills and developmental education courses in the first six months after randomization.51 This impact, however, faded over the next 18 months. There was a sizeable drop-off in impact in months seven through 12 (six percentage points, statistically significant at only the 10 percent level), and no statistically significant impacts over the next 12 months. Over the course of the entire 24 months, there was a 21 percentage point impact on enrollment in these courses. Much of the impact in the enrollment of these types of courses, particularly early on, is attributable to basic skills support courses required for I-BEST certificates.52

To better discern any possible impacts of I-BEST specifically on developmental coursework, the research team also examined separately enrollment in developmental math courses and in developmental reading courses. The team separated out these two courses because the enrollment and completion of these two types of courses can often differ from each other. As shown in panel two, the findings suggest that most of the positive impacts in the first six months are likely due to basic skills course enrollment, as there is a lack of impact on developmental math and reading course-taking across all time periods.

However, within the 24-month follow-up period, I-BEST increased enrollment in and completion of college-level math (algebra) and English—credit-bearing academic courses with course numbers of 100 or above. For algebra, I-BEST increased enrollment by 10 percentage points and completion by nine percentage points; for English, I-BEST increased enrollment by

51 In considering basic skills courses, the research team chose to analyze course enrollment and did not measure

credit attainment. Basic skills courses do not count towards the course requirements for credential attainment. Though the colleges attached credits to basic skills courses for their own institutional purposes, we do not analyze basic skills credit attainment, but instead focus on enrollment in basic skills courses.

52 See Chapter 3 for the courses required for specific I-BEST programs across the three colleges. Almost all programs have a basic skills course requirement, which is why there is a high percentage of students enrolling in basic skills or developmental courses using course transcript information.


PACE


nine percentage points and completion by six percentage points. These findings (shown in the third panel of Exhibit 5-6) are encouraging in terms of advancement in college coursework, because these algebra and English courses serve as prerequisites for many other courses and are often easily transferrable to four-year colleges.

Exhibit 5-6: Early Impacts on Other Education Outcomes (Exploratory Hypotheses)


Control Group

Impact (Difference)

Standard Error

Effect Size p–Value

Enrollment in basic skills or developmental courses in (%)

Months 1-6 Months 7-12 Months 13-18 Months 19-24 Any month

64.7 29.2 14.6 11.6 73.2

45.9 22.8 12.7 8.2

51.9

+18.8*** +6.4* +2.0 +3.4

+21.3***

3.9 3.4 2.7 2.4 3.8

+0.38 +0.15 +0.06 +0.11 +0.45

<.001 .061 .471 .151

<.001 Enrollment in developmental courses over time (%)a

Enrolled in any developmental math course in:

Months 1-6 Months 7-12 Months 13-18 Months 19-24 Any month

Enrolled in any developmental reading/writing course in

Months 1-6 Months 7-12 Months 13-18 months 19-24 Any month

5.1 5.5 6.0 5.6

14.5

4.6 3.7 4.1 2.4

12.3

6.6 6.6 3.8 3.2

13.0

6.3 3.8 2.8 1.9

12.0

–1.5 –1.1 +2.2 +2.4 +1.5

–1.7 –0.1 +1.3 +0.5 +0.3

1.9 1.8 1.7 1.6 2.7

1.8 1.5 1.5 1.2 2.6

–0.07 –0.05 +0.10 +0.12 +0.05

–0.07 –0.01 +0.07 +0.03 +0.01

.414 .548 .200 .145 .565 .348 .948 .381 .675 .913

Enrollment and completion of college-level academic courses (%)b

Enrolled in Introductory college-level math Introductory college-level English

Completed with a grade of C or better: Introductory college-level math Introductory college-level English

22.0 25.5

19.4 20.7

11.7 16.8

10.1 14.2

+10.3*** +8.7***

+9.3*** +6.4**

2.9 3.1

2.7 2.9

+0.28 +0.22

+0.27 +0.17

<.001 .006

<.001 .029

Sample sizec 315 316 SOURCE: Abt Associates calculations based on data from SBCTC records. NOTES: Statistical significance levels, based on two-tailed t-tests tests of differences between research groups, are summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. a Developmental coursework enrollment was derived using administrative data. Courses were flagged developmental if they had a course number below 100. b Also referred to as “gatekeeper” courses, these are the first college-level courses that many community college students take as prerequisites for higher-level courses. c Sample size is 315 treatment cases and 316 control cases for all analyses shown in this table. Completion of introductory college-level math or English course is not conditional on enrollment.


PACE


5.3. Impacts on Early Career Progress (Secondary Hypotheses)

This section presents impact estimates for employment outcomes and self-assessments of career progress.

• I-BEST improved perceived career progress, but there was no evidence of impact on skilled employment.

The estimates in Exhibit 5-7 show a positive impact on perceived career progress, but no impacts on the other assessments of career progress in the first 18 months. The results also show no evidence of impacts for the other two indicators of career pathways employment: working in a job (1) earning more than $12 per hour and (2) that requires at least mid-level skills.

Exhibit 5-7: Early Impacts on Selected Career Outcomes (Secondary Hypotheses)


Control Group

Impact (Difference)

Standard Error

Effect Size p-Value

Indices of Self-Assessed Career Development (average):

Perceived career progressa 3.32 3.19 +0.13** 0.08 +0.17 .043 Confidence in career knowledgeb 3.14 3.08 +0.06 0.05 +0.10 .136 Access to career supportsc 1.72 1.77 –0.04 0.03 –0.14 .936

Indicators of Career Pathways Employment (%):

Working in a job paying $12/hour or more

23.0 23.8 –0.8 4.1 –0.02 .576

Working in a job requiring at least mid-level skillsd

5.1 9.5 –4.4 2.6 –0.16 .953

Sample sizee 238 225 SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between research groups, are summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. a Three-item scale tapping self-assessed career progress; response categories range from 1=strongly disagree to 4=strongly agree. b Seven-item scale of self-assessed career knowledge; response categories range from 1=strongly disagree to 4=strongly agree. c Seven-item scale of types of career-supportive relationships in workforce and education settings, ranging from zero to seven possible types; response categories are either 1=no or 2=yes. d Whether employed in a job requiring calibrated set of skills based on federal standards. e Sample sizes in this row apply to sample members responding to the PACE follow-up survey.

Given I-BEST’s short-term focus on building basic and occupational skills, it is not surprising that the effects for perceived career progress are more positive than actual employment outcomes at 24 months. Though the observed progress in training might foster participants’ sense of career progress, the fact that more treatment than control group members continued to be enrolled in training programs at the end of the 18-month survey suggests that it may be too early to expect unambiguously positive impacts on actual attainment of career-path jobs. Whether I-BEST helps participants obtain career path jobs in the longer term will be addressed in a later report.

5.4. Impacts on Psycho-Social Skills, Life Stressors, and Family Structure

The large positive impacts on course enrollment and credential completion, as well as the positive impact on self-assessed career progress, suggest possible positive effects on other self-assessed


PACE


outcomes associated with college success. However, individuals in a program that stresses psycho-social of skills may come to have higher expectations of their own performance than control group members, rating the same objective level of performance more negatively (Duckworth and Yeager 2015). This potential for measurement bias means the predicted direction of effect is unclear. As such, the study treats these analyses as exploratory (i.e., subject to two-sided tests).

• I-BEST slightly increased social belonging in school and decreased perceived stress.

Results in Exhibit 5-8 show a positive effect of I-BEST on social belonging in school, with treatment group students reporting a slightly higher level of belonging on a four-point scale (statistically significant at the 10 percent level).

Exhibit 5-8: Early Impacts on Other Outcomes (Exploratory Hypotheses)


Control Group

Impact (Difference)

Standard Error

Effect Size p-Value

Indices of Psycho-Social Skills (average):

Grita 3.03 2.99 +0.03 0.04 +0.07 .434 Academic self-confidenceb 4.49 4.48 +0.01 0.07 +0.02 .842 Core self-evaluationc 3.20 3.14 +0.06 0.04 +0.14 .150 Social belonging in schoold 3.25 3.15 +0.09* 0.05 +0.18 .054

Indices of Life Stressors (average): Financial hardshipe 0.63 0.64 –0.01 0.04 –0.03 .783 Life challengesf 1.52 1.55 –0.04 0.05 –0.07 .423 Perceived stressg 2.15 2.29 –0.14** 0.07 –0.19 .043

Family Structure (%): Living with spouse 35.9 33.2 +2.7 3.5 +0.07 .444 Had child since random assignment/currently pregnant

9.7 13.7 –4.0 3.6 –0.14 .266

Sample sizeh 238 225 SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. NOTES: Statistical significance levels, based on two-tailed t-tests tests of differences between research groups, are summarized as follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. a Eight-item scale capturing persistence and determination, response categories range from 1=strongly disagree to 4=strongly agree. b Twelve-item scale; response categories range from 1=strongly disagree to 6=strongly agree. c Twelve-item scale; response categories range from 1=strongly disagree to 6=strongly agree. d Five-item scale capturing sense of belonging; response categories range from 1=strongly disagree to 4=strongly agree. e Two-item scale capturing financial hardship, reported as either an inability to pay rent/mortgage or not enough money to make ends meet; response categories are 0=no and 1=yes. f Seven-item scale capturing life challenges that interfere with school, work, or family responsibilities; response categories range from 1=never to 5=very often. g Four-item scale capturing perceived stress; response categories range from 1=never to 4=very often. h Sample sizes in this row apply to sample members responding to the PACE follow-up survey.

I-BEST decreased perceived stress, with I-BEST students reporting a lower level of stress. There were no significant impacts on the remaining indices of psycho-social skills or life stressors. There were also no significant impacts on family structure.

These early findings suggest that in light of the financial and academic supports provided through I-BEST, the program may reduce stress among program participants and give them a


PACE


greater sense of social belonging. Whether these impacts persist, or even increase, as participants transition to employment is an open question to be addressed in later reports.


PACE

Abt Associates 6. Conclusions ▌pg. 73

Conclusions

Since its inception, I-BEST has garnered the attention of educators and researchers as a promising model to support educational and occupational advancement for students with low basic skills. The I-BEST program grew out of a concern that adult basic skills students were often not advancing beyond basic skills classes to college-level occupational programs, and therefore were not earning credentials. To better serve this low-skilled population, the I-BEST program modified instructional staffing, pairing basic skills instructors and occupational instructors for team teaching in occupational training courses. The I-BEST programs evaluated here also assisted students with school-related expenses and offered advising support. Without I-BEST, students whose college entrance test scores were too low for them to enroll directly in their desired occupational training program would have had to successfully complete remedial classes to increase their basic skills to the required levels.

This study found that the evaluated I-BEST programs were implemented largely as designed, with various forms of team teaching, as well as financial and advising supports delivered to a population with low education levels (about 30 percent did not have a high school diploma or GED). The program substantially increased enrollment in credit-bearing courses, driven primarily by students’ enrollment in occupational training classes. It also increased treatment group members’ receipt of career counseling and other supports. Importantly, I-BEST programs also produced increases in the number of academic and workforce credits earned and the percentage of students who had earned a workforce credentials within the 24-month follow-up period. This concluding chapter examines the implications of these early results for policy and practice and looks ahead to questions the evaluation will address in the longer term.

6.1. Implications of I-BEST Programs

Because of the interest at the federal, state, and local levels in I-BEST as a strategy to improve education and employment outcomes for low-skilled adults, replications of the model are already underway in several states and localities. The PACE results, which are the first from an experimental evaluation of the I-BEST model, have a number of implications for further development of related initiatives.

• Earlier non-experimental research showed the potential of I-BEST to increase academic success for low-skilled adults. Initial findings from the PACE evaluation provide strong causal evidence of the effect the I-BEST program can have on education outcomes.

The PACE evaluation provides the first rigorous evidence on I-BEST from a range of occupational programs at three of Washington State’s 34 community and technical colleges. Though not representative of all I-BEST programs statewide, these results greatly strengthen prior evidence


PACE


(e.g., Zeidenberg, Cho, and Jenkins 2010) that I-BEST can produce increases in receipt of credits and credentials. In particular, the results demonstrate that some students who may not initially qualify for college-level programming because of low basic skills levels can complete college-level work when additional basic skills instruction, financial assistance, and advising supports are built into the occupational training. Future reports will provide information on I-BEST’s employment effects. Longer follow-up, planned as part of the PACE study, is needed to determine I-BEST’s effects on participants’ economic outcomes.

• Though the I-BEST programs themselves last one to two quarters, many students used them as a launching pad for additional college-level education and training.

A high proportion of treatment group members who attended I-BEST programs (63 percent) also enrolled in additional college classes. On average, those who attended an I-BEST program were enrolled in college courses for four quarters—longer than the I-BEST portion of the occupational training lasted. Approximately one-quarter were still enrolled in college at the end of the 24-month follow-up period. In addition, the impact on earning credentials was driven primarily by the higher rate of completion among treatment group members who enrolled in college than among the control group. Thus, it appears that the I-BEST program was more effective in engaging students in programming that resulted in a credential than other education and training programs were.

• For students interested in pursuing higher levels of education, it may be important to maintain connections between occupational and academic programs.

Most of the credits and credentials completed by I-BEST participants were from workforce courses, meaning they are not transferable to four-year colleges. But there is evidence that some students also incorporate more academic coursework in their education. By design, workforce credits are earned in occupational courses that focus on specific technical skills and do not include academic credits that are generally transferable to four-year colleges (and that can be used to meet general education requirements for higher-level degrees). However, primarily through enrollment in courses after I-BEST, the program also increased enrollment in and completion of college-level algebra and English, both of which are credit-bearing academic courses. These findings are encouraging for potential advancement in college coursework, because these introductory math and English courses serve as prerequisites for many other college courses and are often transferrable to four-year colleges. Some I-BEST instructors were proactive in encouraging this connection to higher-level education courses, particularly if there were limited job advancement opportunities for program completers with short-term workforce credentials. However, whether and when this transition from workforce to academic credits is appropriate is an important consideration when designing career pathways beyond the initial I-BEST programming.


PACE


• Based on discussions with I-BEST staff, team teaching was most successful when co-instructors had planning time and worked together over multiple quarters.

Although team teaching was used across all the programs in the study, I-BEST staff reported variations in how it was operationalized and challenges the teams faced. The I-BEST program represents a departure for many occupational instructors, who are used to being alone in a classroom and teaching students with higher academic skills. To effectively implement I-BEST, the instructors reported needing to adapt to the learning styles of students with lower basic skills. They also needed to develop instructional approaches that effectively integrated a second instructor. The implementation study underscores the value of professional development activities to help instructors define and integrate their instruction and the benefit of instructors in working together as team for more than one quarter. Several of the programs reported that part-time and adjunct staffing was a challenge to deeper integration. While common in community colleges, the use of part-time and adjunct staffing limited the planning time available to the teaching teams.

• Although the I-BEST programs operated largely as designed, the implementation study identified areas for further attention and development.

First, although the program sought to target low-skilled individuals, colleges had some difficulty recruiting students. In order to scale up, sites were required to modify their outreach strategies. Second, formalized connections between education and employment were not strong in most of the I-BEST programs. Though some students proceeded to additional coursework, strengthening this connection would likely benefit students who are focused on earning workforce credits and credentials. I-BEST occupational instructors, some of whom had industry experience, sometimes provided informal individualized job assistance, but this kind of connection to the labor market was not consistent across the programs. Finally, foundation funding during the PACE study enabled the colleges to implement additional resources for dedicated student advisors and “fill-the-gap” financial assistance; this support not typically available at in other I-BEST programs. Study results show increased receipt of these supports. This indicates supports may contribute to the observed program effects, and may be important for other program administrators to consider.

6.2. What Lies ahead for the Evaluation of I-BEST

This report finds that I-BEST had a strong positive impact on college enrollment, credits earned, and credential receipt over a 24-month period. Though promising, longer-term follow-up on a broader range of outcomes, particularly employment and earnings, is needed to more definitively assess the I-BEST program:

• Will I-BEST’s impact on educational attainment remain stable, increase, or decrease? In particular, it will be important to assess whether students receive additional


PACE


credentials, beyond those offered through the short-term I-BEST program. The positive early impacts might indicate permanent improvements of the treatment group relative to the control group. Conversely, control group members could catch up over time, so that their education outcomes are no longer significantly different from those of the treatment group.

• Will I-BEST’s impact on educational attainment translate into impacts on employment and earnings? The 24-month analyses examined only a very few employment-related outcomes. Future analyses will assess the impact of I-BEST on a broader array of employment outcomes, including earnings, hourly wages, receipt of fringe benefits, and stability of employment.

• Does I-BEST have other impacts on participants and their families? Key outcomes to address in the future include the effect of the program on individual and household income and material well-being.

• Is I-BEST cost beneficial? Future analyses will explore the costs of the program relative to the benefits it produces for participants and society.

The next PACE report on I-BEST will cover a 36-month follow-up period and provide a systematic look at impacts on employment for a period when any such impacts can be expected to emerge. That report will examine a broad variety of employment outcomes, including average employment and earnings over successive follow-up quarters, job characteristics (e.g., occupation, hourly wage rate, receipt of fringe benefits, career progress). Thus, the report will begin to answer whether the education and training experiences of I-BEST treatment group members will translate into economic gains in the workplace in the longer term.

In addition, a later report will examine the long-term effects of I-BEST on earnings at approximately 72 months after random assignment.


PACE

Abt Associates References ▌pg. 77

References

Abt Associates. 2015. Pathways for Advancing Careers and Education (PACE). Technical Supplement to the Evaluation Design Report: Impact Analysis Plan. OPRE Report # 2015-100, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education-supplement-evaluation-design-impact-analysis-plan.

Abt Associates. 2014. Pathways for Advancing Careers and Education Evaluation Design Report. OPRE Report # 2014-76, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education-evaluation-design-report.

ACT, Inc. 1997. ACT assessment and technical manual: Student Readiness Inventory (SRI). Iowa City, IA: ACT, Inc. (2,3,4,5).

Anderson, Theresa, Kuehn, Daniel, Eyster, Lauren, Barnow, Burt, and Lerman, Robert. 2017. New Evidence on Integrated Career Pathways Final Impact Report for Accelerating Opportunity. Washington, DC: The Urban Institute. https://www.urban.org/research/publication/new-evidence-integrated-career-pathways.

Andridge, R. R., and R. J. A Little. 2010. “A review of hot deck imputation for survey non-response.” International Statistical Review, 78, 40-64.

Bailey, Thomas and Dong Wook Jeong, and Sung-Woo Cho. 2010. “Referral, Enrollment, and Completion in Developmental Education Sequences in Community Colleges.” Economics of Education Review 29(2): 255-270.

Bettinger, Eric P. and Rachel Baker. 2011. “The Effects of Student Coaching in College: An Evaluation of a Randomized Experiment in Student Mentoring.” NBER Working Paper No. 16881. Accessed September 1, 2016. http://www.nber.org/papers/w16881.

Betz, N.E. and Taylor, K.M. 2001. Manual for the career decision self-efficacy scale and CDMSE—Short form. Columbus: The Ohio State University.

Bragg, Debra D., and Elisabeth Barnett. 2009. Lessons Learned from Breaking Through. In Brief. Champaign, IL: Office of Community College Research and Leadership.



https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education-evaluation-design-report

https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education-evaluation-design-report

https://www.urban.org/research/publication/new-evidence-integrated-career-pathways

http://www.nber.org/papers/w16881


PACE


Bridges to Opportunity Initiative. 2008. “Bridges to Opportunity for Underprepared Adults: A State Policy Guide for Community College Leaders.” http://ccrc.tc.columbia.edu/media/k2/attachments/bridges-opportunity-underprepared-adults.pdf.

Bureau of Labor Statistics. 2016. “May 2015 Metropolitan and Nonmetropolitan Area Occupation and Wage Estimates: Tucson, AZ.” United States Department of Labor. http://www.bls.gov/oes/2015/may/oes_46060.htm.

Capelli, Peter. 2014. “Skill Gaps, Skill Shortages, and Skill Mismatches: Evidence for the US.” NBER Working Paper No. 20382. Cambridge, MA: National Bureau of Economic Research. http://www.nber.org/papers/w20382.

Cohen, Sheldon, Tom Kamarck and Robin Mermelstein. 1983. “A Global Measure of Perceived Stress.” Journal of Health and Social Behavior 24(4):385-396.

Conway, M. and Giloth, R.P., eds. 2014. Connecting People to Work: Workforce Intermediaries and Sector Strategies. Washington, DC: Aspen Institute. https://www.aspeninstitute.org/publications/connecting-people-work/.

Cooper, Michelle. 2010. “Student support services at community colleges: A strategy for increasing student persistence and attainment.” Paper presented at the White House Summit on Community Colleges, Washington, D.C. https://www.ed.gov/white-house-summit-community-colleges-conference-papers.

Deming, David and Susan Dynarski. 2010. “Into College, Out of Poverty? Policies to Increase the Postsecondary Attainment of the Poor.” In Targeting Investments in Children: Fighting Poverty when Resources are Limited. University of Chicago Press.

Duckworth, Angela L. and David Scott Yeager. 2015. “Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes” Educational Researcher 44(4): 237-251.

Duckworth, Angela L., C. Peterson, M.D. Matthews, and D.R. Kelly. 2007. “Grit: Perseverance and passion for long-term goals.” Journal of Personality and Social Psychology 92(6): 1087-1101.

Dynarski, Susan and Judith Scott-Clayton. 2013. “Financial Aid Policy: Lessons from Research” Future of Children 23(1):67-91. Accessed September 1, 2016. http://www.futureofchildren.org/futureofchildren/publications/docs/23_01_04.pdf.

Fein, David. 2012. Career Pathways as a Framework for Program Design and Evaluation. OPRE Report # 2012-30, Washington, DC: Office of Planning, Research and Evaluation,

http://ccrc.tc.columbia.edu/media/k2/attachments/bridges-opportunity-underprepared-adults.pdf

http://ccrc.tc.columbia.edu/media/k2/attachments/bridges-opportunity-underprepared-adults.pdf

http://www.bls.gov/oes/2015/may/oes_46060.htm

http://www.nber.org/papers/w20382.

https://www.aspeninstitute.org/publications/connecting-people-work/

https://www.ed.gov/white-house-summit-community-colleges-conference-papers

https://www.ed.gov/white-house-summit-community-colleges-conference-papers

http://www.futureofchildren.org/futureofchildren/publications/docs/23_01_04.pdf


PACE


Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/opre/resource/career-pathways-as-a-framework-for-program-design-and-evaluation-a-working.

Georgetown University Center on Education and the Workforce. 2016. America’s Divided Recovery: College Haves and Have-Nots. https://cew-7632.kxcdn.com/wp-content/uploads/Americas-Divided-Recovery-web.pdf

Ginder, S.A., Kelly-Reid, J.E., and Mann, F.B. (2017). Enrollment and Employees in Postsecondary Institutions, Fall 2015; and Financial Statistics and Academic Libraries, Fiscal Year 2015: First Look (Provisional Data) (NCES 2017-024). U.S. Department of Education. Washington, DC: National Center for Education Statistics. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017024.

Glosser, Asaph, Hamadyk, Jill, Gardiner, Karen and Mike Fishman. 2014. Pathways for Advancing Careers and Education Career Pathways Program Profile: Washington’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges. OPRE Report #2014-38, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/opre/resource/pace-career-pathways-program-profile-integrated-basic-education-and-skills-training-i-best-program.

Goldrick-Rab, Sara, and Kia Sorensen. 2010. “Unmarried parents in college.” Future of Children 20(2): 179-203.

Holzer, H. 2015. Job Market Polarization and U.S. Worker Skills: A Tale of Two Middles. Washington, DC: The Brookings Institution. https://www.brookings.edu/research/job-market-polarization-and-u-s-worker-skills-a-tale-of-two-middles/.

Jenkins, Davis and Sung-Woo Cho. 2012. “Get With the Program: Accelerating Community College Students’ Entry into and Completion of Programs of Study.” CCRC Working Paper No. 32. Accessed September 1, 2016. http://ccrc.tc.columbia.edu/media/k2/attachments/accelerating-student-entry-completion.pdf.

Jenkins, D., Zeidenberg M. and Kienzl, G. 2009. “Educational outcomes of I-BEST, Washington State Community and Technical College System’s Integrated Basic Education and Skills Training Program: Findings from a multivariate analysis.” New York: Community College Research Center. https://ccrc.tc.columbia.edu/publications/i-best-multivariate-analysis.html.

https://www.acf.hhs.gov/opre/resource/career-pathways-as-a-framework-for-program-design-and-evaluation-a-working

https://www.acf.hhs.gov/opre/resource/career-pathways-as-a-framework-for-program-design-and-evaluation-a-working

https://cew-7632.kxcdn.com/wp-content/uploads/Americas-Divided-Recovery-web.pdf

https://cew-7632.kxcdn.com/wp-content/uploads/Americas-Divided-Recovery-web.pdf

https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017024

https://www.acf.hhs.gov/opre/resource/pace-career-pathways-program-profile-integrated-basic-education-and-skills-training-i-best-program

https://www.acf.hhs.gov/opre/resource/pace-career-pathways-program-profile-integrated-basic-education-and-skills-training-i-best-program

https://www.brookings.edu/research/job-market-polarization-and-u-s-worker-skills-a-tale-of-two-middles/

https://www.brookings.edu/research/job-market-polarization-and-u-s-worker-skills-a-tale-of-two-middles/

http://ccrc.tc.columbia.edu/media/k2/attachments/accelerating-student-entry-completion.pdf

http://ccrc.tc.columbia.edu/media/k2/attachments/accelerating-student-entry-completion.pdf

https://ccrc.tc.columbia.edu/publications/i-best-multivariate-analysis.html

https://ccrc.tc.columbia.edu/publications/i-best-multivariate-analysis.html


PACE


Judge, T.A. 2009. “Core self-evaluations and work success.” Current Directions in Psychological Science 18(1): 58-62.

Judkins, David. R. and Kristin E. Porter. 2016. “Robustness of ordinary least squares in randomized clinical trials.” Statistics in Medicine 35(11): 1763-1773. doi: 10.1002/sim.6839.

Karp, M. M. 2011. “Toward a new understanding of nonacademic support: Four mechanisms encouraging positive student outcomes in the community college.” CCRC Working Paper No. 28, Assessment of Evidence Series. New York: Community College Research Center. https://ccrc.tc.columbia.edu/publications/non-academic-student-support-mechanisms.html.

Keshavarz, Mohsen. 2011. “Measuring course learning outcomes.” Journal of Learning Design 4(4):1-9. Accessed September 1, 2016. http://files.eric.ed.gov/fulltext/EJ963323.pdf.

Kessler, R. C., G. Andrews, D. Mroczek, B. Ustun, and H.U. Wittchen. 1998. “The World Health Organization composite international diagnostic interview short-form (CIDI-SF).” International Journal of Methods in Psychiatric Research 7(4): 171-185.

Koch, G.G, Tangen, C.M., Jung, J.W., and Amara, I.A. 1998. Issues for covariance analysis of dichotomous and ordered categorical data from randomized clinical trials and non-parametric strategies for addressing them. Statistics in Medicine 17: 1863-1892.

Le, H., A. Casillas, S. Robbins, and R. Langley. 2005. “Motivational and skills, social, and self-management predictors of college outcomes: Constructing the Student Readiness Inventory.” Educational and Psychological Measurement 65(3): 482-508.

Lesaffre E, Senn S. 2003. A note on non-parametric ANCOVA for covariate adjustment in randomized clinical trials. Statistics in Medicine 22: 3586-3596.

Lin, W. 2013. Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique. The Annals of Applied Statistics 7: 295-318.

Lumley T, P. Diehr, S. Emerson, and L. Chen. 2002. “The importance of the normality assumption in large public health data sets.” Annual Review of Public Health 23: 151-169.

National Center for Education Statistics. 1996. “Nontraditional Undergraduates / Definitions and Data.” Accessed September 1, 2016. https://nces.ed.gov/pubs/web/97578e.asp.

Perin, Dolores. 2013. “Facilitating Student Learning Through Contextualization.” In Teaching Developmental Reading (2nd Edition). Bedford/St. Martin’s.

Perin, Dolores. 2011. Facilitating Student Learning Through Contextualization (CCRC Working Paper no. 29). New York, NY: Columbia University, Teachers College, Community College

https://ccrc.tc.columbia.edu/publications/non-academic-student-support-mechanisms.html

https://ccrc.tc.columbia.edu/publications/non-academic-student-support-mechanisms.html

http://files.eric.ed.gov/fulltext/EJ963323.pdf

https://nces.ed.gov/pubs/web/97578e.asp


PACE


Research Center. https://ccrc.tc.columbia.edu/publications/facilitating-student-learning-contextualization.html.

Peterson, C. H., A. Casillas, and S.B. Robbins. 2006. “The Student Readiness Inventory and the Big Five: Examining social desirability and college academic performance.” Personality and Individual Difference 41(4): 663-673.

Pew Research Center. 2014. “The Rising Cost of Not Going to College.” Pew Research Center, Social & Demographic Trends (blog), February 11. http://www.pewsocialtrends.org/2014/02/11/the-rising-cost-of-not-going-to-college/.

Prince, David and Davis Jenkins. 2005. “Building Pathways to Success for Low-Skill Adult Students: Lessons for Community College Policy and Practice from a Longitudinal Student Tracking Study.” CCRC Brief 25: April. New York: Community College Research Center. https://ccrc.tc.columbia.edu/publications/low-skill-adults-policy.html.

Randall, Vernillia R. 1994. “Learning Domains or Bloom’s Taxonomy: The Three Types of Learning.” The University of Dayton School of Law. Accessed September 1, 2016. https://academic.udayton.edu/health/syllabi/health/Unit01/lesson01b.htm.

Research Triangle Institute. 2012. SUDAAN Language Manual, Volumes 1 and 2, Release 11. Research Triangle Park, NC: Research Triangle Institute.

Rolston, H., Copson, E. and K. Gardiner. 2017. Valley Initiative for Development and Advancement: Implementation and Early Impact Report, OPRE Report #2017-83, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. https://www.acf.hhs.gov/opre/resource/valley-initiative-development-advancement-implementation-early-impact-report.

Rutschow, Elizabeth Zachry and Emily Schneider. 2011. “Unlocking the Gate: What We Know About Improving Developmental Education.” MDRC. Accessed September 1, 2016. http://www.mdrc.org/publication/unlocking-gate.

SBCTC [Washington State Board for Community and Technical Colleges]. 2005. I-BEST: A Program integrating adult basic education and workforce training. Olympia, WA: Author. https://eric.ed.gov/?id=ED496215.

SBCTC [Washington State Board for Community and Technical Colleges]. 2017. “Integrated Basic Education Skills Training (I-BEST).” Website: https://www.sbctc.edu/colleges-staff/programs-services/i-best/default.aspx. Accessed September 5, 2017.

https://ccrc.tc.columbia.edu/publications/facilitating-student-learning-contextualization.html

https://ccrc.tc.columbia.edu/publications/facilitating-student-learning-contextualization.html

http://www.pewsocialtrends.org/2014/02/11/the-rising-cost-of-not-going-to-college/

https://ccrc.tc.columbia.edu/publications/low-skill-adults-policy.html

https://academic.udayton.edu/health/syllabi/health/Unit01/lesson01b.htm

https://www.acf.hhs.gov/opre/resource/valley-initiative-development-advancement-implementation-early-impact-report

https://www.acf.hhs.gov/opre/resource/valley-initiative-development-advancement-implementation-early-impact-report

http://www.mdrc.org/publication/unlocking-gate

https://eric.ed.gov/?id=ED496215

https://www.sbctc.edu/colleges-staff/programs-services/i-best/default.aspx

https://www.sbctc.edu/colleges-staff/programs-services/i-best/default.aspx


PACE


Scott-Clayton, Judith. 2011. “The Shapeless River: Does a Lack of Structure Inhibit Students’ Progress at Community Colleges?” CCRC Working Paper No. 25, Assessment of Evidence Series. Accessed September 1, 2016. http://ccrc.tc.columbia.edu/media/k2/attachments/shapeless-river.pdf.

Scrivener, Susan, Michael J. Weiss, Alyssa Ratledge, Timothy Rudd, Colleen Sommo, and Hannah Fresques. 2015. “Doubling Graduation Rates: Three-Year Effects of CUNY’s Accelerated Study in Associate Programs (ASAP) for Developmental Education Students.” New York: MDRC.

Shapiro, Doug, Afet Dundar, Xin Yuan, Autumn Harrell, and Phoebe K. Wakhungu. 2014. Completing College: A National View of Student Attainment Rates – Fall 2008 Cohort (Signature Report No. 8). Herndon, VA: National Student Clearinghouse Research Center.

Scrivener, Susan and Michael J. Weiss. 2009. “More Guidance, Better Results? Three-Year Effects of an Enhanced Student Services Program at Two community Colleges.” New York: MDRC. https://www.mdrc.org/publication/doubling-graduation-rates.

Tsiatis, A.A., Davidian, M., Zhang, M. and Lu, X. 2008. Covariate adjustment for two-sample treatment comparison in randomized clinical trials; A principled yet flexible approach. Statistics in Medicine 27: 4658-4677.

Visher, Mary G., Heather Wathington, Lashawn Richburg-Hayes, and Emily Schneider. 2008. “The Learning Communities Demonstration: Rationale, sites, and research design.” New York, NY: National Center for Postsecondary Research. https://eric.ed.gov/?id=ED501563.

Wachen, John, Davis Jenkins, and Michelle Van Noy. 2010. "How I-BEST Works: Findings from a field study of Washington State's Integrated Basic Education and Skills Training Program." New York: Community College Research Center. https://ccrc.tc.columbia.edu/publications/how-i-best-works.html.

Walton, G. M., and G.L. Cohen. 2007. “A question of belonging: Race, social fit, and achievement.” Journal of Personality and Social Psychology 9: 82–96.

Walton, G. M., and G.L. Cohen. 2011. “A brief social belonging intervention improves academic and health outcomes of minority students.” Science 331: 1447-1451.

Washington Student Achievement Council. 2015. 2015 Roadmap Report: Measuring Our Progress. Olympia, WA: Author. http://www.wsac.wa.gov/sites/default/files/2015.Roadmap.Report.pdf.

Zeidenberg, M., Cho, S.W., & Jenkins, D. 2010. Washington State’s Integrated Basic Education and Skills Training program (I-BEST): New evidence of effectiveness (CCRC Working Paper

http://ccrc.tc.columbia.edu/media/k2/attachments/shapeless-river.pdf

https://www.mdrc.org/publication/doubling-graduation-rates

https://eric.ed.gov/?id=ED501563

https://ccrc.tc.columbia.edu/publications/how-i-best-works.html

http://www.wsac.wa.gov/sites/default/files/2015.Roadmap.Report.pdf


PACE


no. 20). New York, NY: Columbia University, Teachers College, Community College Research Center. https://ccrc.tc.columbia.edu/publications/i-best-new-evidence.html.

https://ccrc.tc.columbia.edu/publications/i-best-new-evidence.html