executive summary of the pima community college pathways ... · technician, implemented weekly...
TRANSCRIPT
Pima Community
College Pathways to
Healthcare Program:
Implementation and
Early Impact Report
Executive Summary
OPRE Report No. 2017-10
April 2017
Pima Community College Pathways to Healthcare Program: Implementation and Early Impact Report
Pathways for Advancing Careers and Education (PACE)
OPRE Report No. 2017-10
April 2017
Authors
Karen Gardiner, Abt Associates
Howard Rolston, Abt Associates
David Fein, Abt Associates
Sung-Woo Cho, Abt Associates
Submitted to:
Nicole Constance
Federal Project Officer
Office of Planning, Research and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
Contract No. HHSP2332007913YC
Project Director: Karen Gardiner
Abt Associates Inc.
4550 Montgomery Ave.
Bethesda, MD 20814
This report is in the public domain. Permission to reproduce is not necessary. Suggested citation:
Gardiner, K., Rolston, H., D., Fein, D. and S. Cho (2017). Pima Community College Pathways to Healthcare
Program: Implementation and Early Impact Report, OPRE Report No. 2017-10, Washington, DC: Office of
Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health
and Human Services.
Disclaimer
The views expressed in this publication do not necessarily reflect the views or policies of the Office of
Planning, Research and Evaluation, the Administration for Children and Families, or the U.S. Department
of Health and Human Services.
This report and other reports sponsored by the Office of Planning, Research and Evaluation are available
at http://www.acf.hhs.gov/programs/opre/index.html.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. i
Executive Summary
Over the next ten years, the demand for workers in healthcare jobs is expected to grow quickly as the
population grows and ages.1 Successfully meeting the need for more healthcare workers is important to
both the national economy and providing quality healthcare to people. This also creates opportunities
for low-income adults to find entry-level employment and advance to higher-skilled jobs. Almost all jobs
in healthcare require some training after high school. Policymakers, workforce development
organizations, educators, and other key stakeholders are very interested in how to enable the match
between the nation’s need for a skilled workforce and low-income adults’ need for employment.
Pathways to Healthcare Program
This report offers early evidence on the implementation and impacts of one promising effort to meet
both needs. The Pathways to Healthcare program, operated through a partnership between Pima
Community College (PCC) in Tucson, Arizona, and Pima County One Stop (PCOS), the local workforce
agency, aimed to help low-income adults in Pima County access occupational training in the growing
healthcare field. Over its first 18 months, Pathways to Healthcare program participants were more likely
than a randomly assigned control group who could not access the program to:
attend more hours of college occupational training—the primary outcome measured in this
report;
enroll in occupational training; and
earn college credentials (degrees, certificates)
The Administration for Children and Families (ACF), within the U.S. Department of Health and Human
Services, awarded a five-year grant in 2010 to fund the program under the Health Profession
Opportunity Grants (HPOG) demonstration. The two program partners in Pathways to Healthcare
improved existing PCC training programs and PCOS services, seeking to engage low-income adults in
occupational training and aid their academic and career progress.
Pathways to Healthcare combined several key components:
Mapping 16 existing occupational training programs into five “pathways” helped students who
might otherwise find the course catalog confusing understand how completing one credential
can lead to a higher credential and higher paying job.
Intensive and proactive staff guidance and advising helped students and program staff identify
and address barriers to participating in the program, select a training program, persist in and
complete the chosen program, and subsequently enter employment.
Scholarships for tuition, books, and other supplies to reduce the cost of training.
1 http://www.bls.gov/news.release/ecopro.nr0.htm.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. ii
Two compressed and contextualized basic skills (“bridge”) programs helped students with low
levels of basic skills enroll directly in one of the 16 occupational programs.
Employment supports helped program completers locate employment through coaching for job
interviews and help with resume writing.
Pathways for Advancing Careers and Education (PACE) Evaluation
Abt Associates and its partners are evaluating Pathways to Healthcare as part of the Pathways for
Advancing Careers and Education (PACE) evaluation. Funded by ACF, PACE is an evaluation of nine
programs that include key features of a “career pathways framework.”
The career pathways framework guides the development and operation of programs aiming to improve
the occupational skills of low-income adults by increasing their entry into, persistence in, and
completion of postsecondary training. These students are primarily older and nontraditional students.
The framework describes strategies for overcoming barriers to education and training that these
students can face. Key features of programs within this framework include:
a series of well-defined training steps;
promising instructional approaches targeted to adult learners;
services to address academic and non-academic barriers to program enrollment and
completion; and
connections to employment.
The Pathways to Healthcare evaluation includes an implementation study that examines the design and
operation of the program and enrolled students’ participation patterns, and an impact study that used
an experimental design to measure differences in educational and employment outcomes between
individuals randomly assigned to a group that could receive Pathways to Healthcare (treatment group)
and a group that could not (control group).2 Using data from baseline surveys, a follow-up survey,
program records, and site visits, this report provides the results from the implementation study and it
describes the early impacts of the program (18 months after random assignment) on education, training
and employment, including hours of college occupational training received since random assignment,
the primary outcome to assess the early effects of Pathways to Healthcare.3
2 Random assignment ensures that the treatment and control groups will be alike in their observed and
unobserved characteristics, and that any systematic differences in their outcomes can be attributed to the treatment group having access to program services.
3 See the PACE analysis plan. Abt Associates, Inc. (2014). The Pathways to Healthcare analysis plan was also
registered on the What Works Clearinghouse online registry of randomized control trials (RCT) on May 20, 2016 and the Open Science Framework site. In September of 2016, under the terms of a grant from the Institute of Educational Sciences, the RCT registry information was removed from the WWC website and transferred to the Society for Research on Educational Effectiveness (SREE). SREE plans to re-launch the registry in late 2017, at which time the analysis plan will be available in a searchable online database.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. iii
Key Findings
From the Implementation Study
Pathways to Healthcare mapped 16 of PCC’s existing occupational training programs into five
distinct pathways.
By mapping individual training programs into defined career pathways, PCC leadership hoped to simplify
enrollment and placement in the right program and focus students on longer-term training goals. As
Exhibit ES-1 shows, Pathways to Healthcare designated each of the 16 training programs into one of five
pathways: Medical Office, Nursing, Medical and Physician Support, Emergency Medicine, or Other.
Within each pathway, they designated programs by “level” (1 for entry level, 2 for mid-level, and 3 for
advanced level).
The program successfully implemented two bridges to help participating students improve basic
skills prior to enrollment in occupational training. Twenty-six percent of treatment group members
participated in at least one bridge.
As Exhibit ES-1 shows, Pathways to Healthcare implemented two versions of a bridge program for those
students whose basic academic skills (i.e., math, reading, writing) were too low to enter an occupational
program directly. The 10-week College Readiness class and the open lab were designed to help students
quickly increase basic skills and score high enough on the required ACT Compass® assessment to begin
their training of choice.
Exhibit ES-1. Pathways Overview
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. iv
Pathways to Healthcare developed and implemented instructional supports to help students
persist in and complete their training programs.
Instructors and PCC staff members noticed that some students had difficulty persisting in their programs
and proposed a number of course-specific supports to help participants master content and learn
college success skills. Two training programs, Pharmacy Technology and Nursing Assistant/Patient Care
Technician, implemented weekly instructor-led study groups to help students bolster basic skills and
learn occupational content. PCC also implemented the Pre-LPN Reading Group for students trying to
achieve the Compass reading score (90 or above) necessary to qualify for the Licensed Practical Nurse
program. The Nursing Assistant Readiness Class was implemented after staff determined that many
Nursing Assistant students did not have the college success skills, such as time management, needed for
the course.
Pathways to Healthcare implemented three advising positions designed to identify and address
barriers to enrollment, persistence, and completion.
The three advisors—PCOS’s Workforce Development Specialist (WDS) and PCC’s Student Services
Advanced Specialist (SSAS) and Student Services Specialist (SSS)—were expected to collaborate to
address the range of potential academic and non-academic barriers that could affect a student’s ability
to enroll in and complete a training program.
In order to engage students quickly, the first advising meeting, with the WDS, was supposed to occur
within two weeks of random assignment and focus on identifying barriers and necessary supports to
overcome them. Within a week of this meeting, the student met with a SSAS to begin discussing pre-
training activities at the college. According to a PCC staff member, these initial meetings were intended
to help students not lose their way early in the Pathways to Healthcare program. The SSAS helped
students navigate the steps to enrollment; once a student enrolled in a training program, the SSS
provided guidance to help ensure completion.
All Pathways to Healthcare students approved for occupational training received scholarships.
Pathways to Healthcare provided scholarships to all participants with an approved Training Plan. During
the study period, the per-student scholarship (for tuition and books) was roughly $3,500. Recipients who
subsequently pursued a second course of study could receive additional funds, based on availability.4
Pathways to Healthcare staff designed and implemented services to assist in the transition to
employment.
Program staff members observed that many students completing the program had difficulty
transitioning from training to employment. Based on the recommendations of a workgroup, staff
created new PCOS workshops to help students customize their job searches to the healthcare industry
and provide guidance ranging from resume development to practice interviewing; implemented the
4 In Year 2 of the program, there was a waitlist for scholarships due to PCC awaiting approval of Year 1 carryover
funds and more Year 1 participants than expected, who were still in training in Year 2. As a result, some Year 2 treatment group members relied entirely on Pell grants and WIA funds.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. v
New Employee Transition program to encourage employers to hire program graduates by funding
training on the job to help them acclimate to the specific workplace5; and launched the Student and
Alumni Network to help students and graduates develop a professional network, build a sense of
community, and foster college success skills.
Most treatment group members participated in Pathways to Healthcare’s College Readiness
program or its occupational training. Fewer transitioned from College Readiness to occupational
training or from an initial training program to a higher-level one.
As shown in Exhibit ES-2, 62 percent of all treatment group members participated in College Readiness
or occupational training, with 35 percent enrolling directly in occupational training and 26 percent
enrolling in the College Readiness class or open lab. Half of the latter (13 percent) enrolled in training
after completing College Readiness.6 Of the 48 (35 + 13) percent who enrolled in training, 29 percent
(60 percent of trainees) received a credential and another nine percent (19 percent of trainees) were
still in their first training at the end of the follow-up period. The remainder dropped out.
There are many potential reasons why only half of College Readiness completers enrolled in
occupational training. Some who did not may have chosen not to take the Compass assessment required
for occupational training or may not have passed it. Some may have decided to enroll in other types of
training at PCC. Responses to the follow-up survey suggest that some students did not have enough time
for work and family, or would not receive enough financial aid.
The exhibit also demonstrates that close to one-quarter of the 48 percent who enrolled in occupational
training went on to attend a second training during the follow-up period. Most of these combined the
Level 1 Nursing Assistant and the Level 2 Patient Care Technician programs. One likely reason more
students had not progressed along their chosen pathway during the study period is that program staff
encouraged training completers to work for six months in the occupation before returning for an
additional credential. The Nursing Assistant/Patient Care Technician pairing was the only case in which
students could enroll directly in a second program upon completion of the first.
5 HPOG funds could be used for training but not wages.
6 Due to rounding the figures do not add up to 62.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. vi
Exhibit ES-2. Participation in and Completion of Education and Training among Treatment Group
Members within an 18-Month Follow-Up Period
SOURCE: Pima Community College records.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. vii
A sizable subgroup of treatment group members did not enroll in College Readiness or
occupational training.
Thirty-eight percent of treatment group members attended neither College Readiness nor occupational
training. There are a number of possible reasons. Responses to the follow-up survey suggest that some
may have found it more difficult than they expected to juggle school with other commitments,
particularly work. Similar to the group of participants who did not advance beyond College Readiness,
for the subset of Pathways to Healthcare treatment group members who did not enroll in training at all,
the most common reasons reported as very important to not enrolling were not enough time for work
and for family and not enough financial aid. Also, as noted below, treatment group members had to wait
an average of four months to start their programs which may have discouraged them from enrolling.
Nursing was the most common pathway and Nursing Assistant was the most commonly attended
and completed training program.
As Exhibit ES-3 shows, of those who participated in education or occupational training, more than one-
third (36 percent) were in the Nursing pathway; within this pathway, Nursing Assistant was the most
common program (84 percent of those in the pathway and 30 percent of participants overall). The
Nursing pathway completion rate was 82 percent, driven largely by the five-week Nursing Assistant
program (88 percent) and four-week Patient Care Technician add-on (92 percent). The third program in
the pathway, the 13-month Level 3 Licensed Practical Nurse program, had a lower completion rate
(29 percent), but a large proportion of participants were still enrolled at the end of the follow-up period
(54 percent, not shown).
The next most common pathway was Medical and Physician Support (18 percent of participants overall);
within this pathway, the 15-month Medical Assistant program was the most common. Its completion
rate (25 percent) was lower than for other programs in the pathway, notably Phlebotomy (80 percent),
but the duration of the program was longer and more participants were still in training at the end of the
follow-up period (not shown). This pathway had the largest proportion of participants still enrolled at
the end of the follow-up period (44 percent).
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. viii
Exhibit ES-3. Type of Program Attended, Completion Rates, and Average Length of Stay among
Treatment Group Members in Pathways to Healthcare over 18-Month Follow-Up Period
Education and Training Program Participation
Rate
Of Participants in Specified Program
Completion Rate
Average Length of Stay
(mos.)
Participating at End of
Follow-Up
College Readiness 42.4% 99.4% 2.4 1.6%
Occupational Training 78.4% 59.9% 6.4 24.8%
College Readiness and Occupational Training 20.8% 47.4% 7.0 30.8%
Attended Any Training Program or College Readiness 100.0% 53.1% 6.0 19.5%
Occupational Training Programs
Medical Office–Health Information Management Pathway
11.5% 51.2% 6.4 14.0%
Medical Office (Level 1) 5.9% 63.6%
Medical Records Technician (Level 2) 3.2% 66.7%
Health Information Technology (Level 3) 2.4% 0.0%
Nursing Pathway 35.7% 82.1% 3.2 15.7%
Nursing Assistant (Level 1) 29.9% 87.5%
Patient Care Technician (Level 2) 16.0% 91.7%
Licensed Practical Nurse (Level 3) 7.5% 28.6%
Medical and Physician Support Pathway 17.6% 40.9% 10.5 43.9%
Phlebotomy (Level 1) 4.0% 80.0%
Medical Assistant (Level 2) 11.7% 25.0%
Clinical Research Coordinator (Level 3) 1.9% 28.6%
Emergency Medicine Pathway 2.9% 45.5% 5.6 18.2%
EMT–Basic (Level 2) 2.9% 45.5%
EMT–Paramedicine (Level 3) 0.0% –
Other Pathway 12.8% 41.7% 8.9 31.3%
Community Health Advisor (Level 1) 0.5% 50.0%
Direct Care Professional (Level 1) 0.0% –
Behavioral Health Specialist (Level 1) 5.3% 55.0%
Surgical Technician (Level 2) 1.3% 40.0%
Pharmacy Technology (Level 3) 4.8% 16.7%
SOURCE: PCC records.
NOTES: Sample size is 375 and includes all students who participated in at least one Pathways to Healthcare training program.
Completion rate and length of stay are calculated for those who attended the specified program. Individual items may not sum
to totals because students can attend more than one training.
While the average length of stay in a training program was six months, an average of almost
11 months elapsed between enrollment in Pathways to Healthcare and completion of training.
Some 60 percent of those who participated in occupational training completed their program in an
average of six months, and 25 percent were still in their training program at the end of the follow-up
period (see Exhibit ES-3). However, the average time between random assignment (enrollment in
Pathways to Healthcare) and last date of participation was 11 months, with more than 45 percent of
students attending a year or longer. A key factor in this total length of stay in the Pathways to
Healthcare program is the amount of time students spent between enrollment and the start of training.
Students averaged almost four months before starting their training, with one-third waiting longer than
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. ix
four months. Several factors likely contributed to the delay in training enrollment, including getting a
Training Plan approved, completing pre-enrollment activities, and waiting for the training start date.
The treatment group was significantly more likely than the control group to participate in
education and training.
All 1,217 study participants (treatment and control group) could enroll in PCC’s training programs and
receive services from PCC and PCOS, but only treatment group members could access the additional
components that comprised the Pathways to Healthcare program. Students in the treatment group
were significantly more likely to participate in training than members of the control group. This held true
both for training in any subject (60 percent of the treatment group versus 47 percent of the control
group) and for healthcare-related training (37 percent versus 30 percent). Treatment group members
were significantly more likely to participate in basic skills instruction than were control group members
(18 percent versus 10 percent), with the College Readiness class and lab likely accounting for the
difference.
The treatment group was significantly more likely than the control group to participate in advising
and employment services; a minority of both groups reported they used the services, however.
Treatment group members were significantly more likely than control group members to receive career
counseling (28 percent versus 19 percent) or help arranging supports (14 percent versus 7 percent).
Those in the treatment group also were significantly more likely to receive job search assistance than
were control group members (23 percent versus 14 percent). Significantly fewer treatment group
students (58 percent) cited financial support as a challenge to enrollment or persistence in the program
than did control group members (69 percent).
From the Impact Study
The Pathways to Healthcare program increased the total hours of college occupational training
that students received. Since this is the primary outcome for the early analysis of this program, the
finding indicates that the program was on track toward achieving its main goals.
As Exhibit ES-4 shows, Pathways to Healthcare had a statistically significant impact on the primary
outcome of interest for this analysis period: hours of occupationally focused college training. The
program had a 63-hour impact on total hours of training (over the 18-month follow-up period,
190 hours for the treatment group compared with 127 hours for the control group), a finding
statistically significant at the 1-percent level.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. x
Exhibit ES-4. Early Impacts on Educational Attainment (18 Months after Random Assignment)
Outcome Treatment
Group Control Group Difference
Standard Error p-Value
Primary Outcome
Total hours of college occupational training (average) 190.1 127.0 +63.1 *** 15.2 <.001
Secondary Outcomes
Enrollment in college occupational training in successive months after random assignment (%)
Months 1-6 37.3 27.3 +10.0 *** 2.6 <.001
Months 7-12 45.4 28.0 +17.5 *** 2.6 <.001
Months 13-18 39.6 24.5 +15.1 *** 2.6 <.001
Any month 55.8 36.0 +19.7 *** 2.7 <.001
Total hours of college occupational training in successive months after random assignment (average)
Months 1-6 56.4 49.1 +7.2 7.2 .156
Months 7-12 72.3 40.2 +32.1 *** 6.2 <.001
Months 13-18 61.4 37.6 +23.8 *** 6.4 <.001
All months 190.1 127.0 +63.1 *** 15.2 <.001
Total hours of occupational training at (average)
A college 190.1 127.0 +63.1 *** 15.2 <.001
Another place 17.5 32.7 -15.2 8.2 .968
Any place 210.3 159.2 +51.1 *** 17.6 .002
Total credits earned from colleges by 18 months 1.5 1.7 -0.2 0.3 .799
Received a credential from (%)
A college 23.1 10.4 +12.7 *** 2.1 <.001
Another education-training institution 2.1 9.0 -6.8 1.6 1.000
A licensing/certification body 24.1 22.5 +1.6 2.7 .282
Any source 34.6 29.4 +5.2 ** 2.9 .040
Sample sizea
609 608
Covariate procedure used for these tables: Residualization.
SOURCE: Abt Associates calculations based on data from Pima college records and the PACE short-term follow-up survey.
NOTES: Statistical significance levels, based on one-tailed t-tests of differences between research groups, are summarized as
follows: ***statistically significant at the one percent level; ** at the five percent level; * at the ten percent level. a Sample sizes in this row apply to estimates based on college records for the full sample. In the last two panels (total hours and
credentials by place), estimates for activity at a college are based on college records for the full sample, while all other
estimates (including those for activity at any source) are based on the subsample who responded to the PACE follow-up survey,
including 500 treatment and 477 control group members.
This impact on participation in occupational training is the result of the Pathways to Healthcare program
enrolling more students in the treatment group (20 percentage points) in training compared to the
control group, rather than increasing the hours of participation among those who otherwise would have
enrolled. From this perspective, the average effect of an increase of 63 hours across the entire
treatment group translates into roughly a 320 hour increase for the approximately 20 percent of
treatment group members who in the absence of Pathways to Healthcare would not have enrolled in
occupational training.
There were also impacts on a number of training-related secondary outcomes, including enrollment in
and total hours of occupational training in successive months after random assignment and receipt of a
credential from a college. The second and third panels in Exhibit ES-4 show that both fractions of
students enrolled and average total hours of college occupational training received are higher for
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates Executive Summary ▌pg. xi
treatment group members than for control group members in each of the three six-month follow-up
intervals after random assignment. Impacts on enrollment are smaller in the first six-month period
(10 percentage-point difference), grow in the next six months (18 percentage-point difference), and
remain positive in the last six months (15 percentage point difference). Impacts on average total hours
follow a similar pattern. The bottom panel shows that treatment group members were also significantly
more likely than control group members to obtain a credential from a college (23 percent versus
10 percent) and from any source (35 percent versus 29 percent).
Pathways to Healthcare had a limited effect on employment-related outcomes 18 months after
random assignment.
The evaluation team also investigated the impact of program participation on two dimensions of career
progress. Self-assessed progress towards career goals included measures of perceived career progress,
confidence in career knowledge, and access to career supports. The analyses found positive impacts for
these measures, with the largest effect on perceived career progress. At this point in time, there is no
evidence of impact on employment outcomes, indicated by working in a job that pays at least $12 per
hour, requires at least mid-level skills, or is in a healthcare occupation.
It is not surprising that the effects for self-assessed career progress are more positive than for actual
employment outcomes after 18 months. The treatment group’s greater progress in training compared
with the control group’s might well increase their sense of career knowledge and prospects. However,
more treatment than control group members were still enrolled in training at the end of the 18-month
follow-up period, suggesting it may be too early to expect positive effects on career-path jobs.
Next Steps in the Pathways to Healthcare Evaluation
This report on Pathways to Healthcare focuses on the implementation of the program and its early
effects on participating students’ education and training. At 18 months after students were randomly
assigned into the program or not, the key program goal examined was increased occupational training,
with limited analysis of employment and earnings. This reflects expectations that many students
participating in the program would still be engaged in training at the end of 18 months.
The next Pathways to Healthcare report will cover a 36-month follow-up period. It will take a more
systematic look at program effects on students’ economic outcomes for a period when these are
expected to occur. The report will examine employment outcomes, such as average rate of employment
and average earnings over successive follow-up quarters, and job characteristics, such as occupation,
hourly wage, receipt of benefits, and career progress. Thus, it will begin to answer whether the
occupational training gains that Pathways to Healthcare achieved after 18 months will translate into
economic gains in the workplace in the longer term. An analysis at 72 months after random assignment
will estimate long-term effects of the program.
Pima Community College Pathways to Healthcare Program
Implementation and Early Impact Report
PACE
Abt Associates References ▌pg. xii
References
Abt Associates Inc. (2015). Pathways for Advancing Careers and Education (PACE). Technical Supplement
to the Evaluation Design Report: Impact Analysis Plan. OPRE Report # 2015-100, Washington, DC:
Office of Planning, Research and Evaluation, Administration for Children and Families, U.S.
Department of Health and Human Services.
Bureau of Labor Statistics. 2015. “Employment Projections: 2014-24 Summary.” Accessed September 1,
2016. http://www.bls.gov/news.release/ecopro.nr0.htm.