Evaluating Tulsa’s Two-Generation CareerAdvance® Program:

Early Indications

Christopher T. King Ray Marshall Center, LBJ School of Public Affairs

The University of Texas at Austin

Administration for Children and Families Welfare Research & Evaluation Conference

Grand Hyatt Hotel, Washington, D.C.

May 28, 2014

Acknowledgements: Collaborators

•  Lindsay Chase-Lansdale, Teresa Eckrich Sommer, Terri J. Sabol, Mumbe Kithakye, Rayane Alamuddin, Emily Ross, Allison Frost & Katie Dahlke, Northwestern University

•  Jeanne Brooks-Gunn, Columbia University •  Tara C. Smith, Kristin Christensen, The

University of Texas at Austin •  Hirokazu Yoshikawa, New York University •  Celia Gomez, Harvard University •  Steven Dow & Monica Barczak, Community

Action Project-Tulsa County

Acknowledgements: Funders

Funding for this project was provided by: •  The U.S. Department of Health and Human

Services, Administration for Children and Families, Grant #90PH0020 & Grant #90FX00100

•  W.K. Kellogg Foundation •  Bill and Melinda Gates Foundation •  George Kaiser Family Foundation

Ø  Two-Generation Programs Ø  CareerAdvance®: A Model 2-Gen Program Ø  Evaluating CareerAdvance®

l  The CareerAdvance® Implementation Study l  The CAP Family Life Study

Ø  Early Indications l  Program Changes, Lessons Learned l  Persistence Findings l  Qualitative Findings via Participant Voices

Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views or policies of the Office of Planning, Research and Evaluation, the Administration for Children and Families, or the U.S. Department of Health and Human Services.

Presentation Overview

Ø  Wide variation in two-generation program types Ø  Our focus is on variant emphasizing human

capital investments, programs that intentionally and simultaneously connect: l  High-quality education and job training

programs for parents and l  High-quality early education for their young

children

Two-Generation Programs Defined

Theory of Change

•  Improved cognitive and social development

•  Higher attendance •  Readiness for kindergarten

•  Motivation to pursue education and careers

•  Defined education and career goals

•  Higher rates of adult basic education •  Higher rates of education and career

training enrollment

Child

Parent

Two Generation Programs

Early Childhood Centers

•  High-Quality Classrooms

•  Family Support Services

PSE/Workforce Development

•  High-Quality Community Colleges

•  High-Quality Job Training Programs

•  Employers

Theory of Change

•  Higher rates of persistence in education and job training

•  Improved job training skills and career development

•  Higher rates of employment

•  Higher wage growth

•  Higher motivation and engagement in school

•  Academic success in elementary school

•  Social competence Child

Parent

Two Generation Programs

Early Childhood Centers

•  High-Quality Classrooms

•  Family Support Services

PSE/Workforce Development

•  High-Quality Community Colleges

•  High-Quality Job Training Programs

•  Employers

Theory of Change

•  Increased high school graduation rates

•  Increased training and postsecondary education attainment

•  High expectations and positive future orientation

•  Stable career •  Family supporting wage

•  Greater life stability

•  Better functioning family system

Child

Parent

Two Generation Programs

Early Childhood Centers

•  High-Quality Classrooms

•  Family Support Services

PSE/Workforce Development

•  High-Quality Community Colleges

•  High-Quality Job Training Programs

•  Employers

CareerAdvance®: A Model 2-Generation Education and Workforce Program

Target Population: Parents of Head Start and Early Head Start children, mainly single mothers, though some fathers as well

Early Childhood Program

Career Coaches

Peer Support

Incentives & Support Services

Sectoral Training/

Career Paths Employers

Elementary Schools

Local Colleges

Basic Education &

ESL

CareerAdvance®

Evidence-Based Components Ø  Stackable skills training in growth sectors offered

via structured career pathways (e.g., healthcare, healthcare IT, allied health)

Ø  Contextually-based adult ed, GED, & ESL instruction

Ø  Peer supports Ø  Weekly career coach meetings Ø  Mutual obligations agreements Ø  Workforce intermediary services, connecting

participants, providers, and employers Ø  Conditional cash (in-kind) incentives

Evaluating CareerAdvance® Implementation Study •  Focus on program implementation and progress

for all participant cohorts (2009-2015) •  Staff & partner interviews, focus groups, program

data analysis

CAP Family Life Study •  Mixed-methods Approach: Quantitative &

Qualitative •  Quasi-experimental •  Longitudinal outcomes for Cohorts 4-10

(2011-2015)

Implementation Study Research Questions

1.  How has CareerAdvance® changed over time and why?

2.  What progress have CareerAdvance®

participants made?

3.  What program and institutional factors contribute to or impede participant progress?

8 wks 4-8 wks 16 wks 15-17 mos. 3 years

Certified Nurse Assistant

$9-$12

CNA I

Geriatric Technician

$8.51-$14.13

Advanced Unlicensed Technician

$11-$15

Licensed Practical Nurse

$16-$20

Registered Nurse

$20-$30

Registered Nurse

CNA II

Patient Care Technician

Licensed Practical

Nurse

LPN to RN

Bridge

Program Changes: Nursing Career Ladder

Certified Medical Coder

$11-$20

Health Information Technology Associate $13-$30

Health Information Technology

Program

Medical Coding

Program

3 years 7 mos.

Program Changes: Health Information Technology (HIT) Ladder

Certified Medical Assistant

$10-$18

Medical Assistant Program

9 mos.

Program Changes: Medical Assisting

Changes in Basic Skills Preparation

Ø  Participants’ basic skills measured by TABE® and COMPASS® tests

Ø  Higher baseline basic skills in reading than math l  Mean TABE® reading scores at 10th-12th grade level

l  Mean TABE® math scores at 7th- 8th grade level

Ø  Also, wide range of basic skills within cohorts and between pathways

l  Some participants test at 1st & 2nd grade levels, others at 12th grade levels

Ø  Group and individual tutoring Ø  Advanced Nursing Skills course Ø  Individual instruction at Sylvan Learning Ø  Added new Educational Pathways Program (Jan.

2013) to build CAP parents’ basic skills for future program entry. Parents sorted into three groups: 1.  ‘Skill-ready,’ testing at 6th-8th grade level

2.  ‘College-bound,’ testing at 9th-12th grade level

3.  ‘Career-ready,’ testing at college level

How CareerAdvance® Partners Strengthened Basic Skills Preparation

Factors Impeding Program Progress

Ø  Time management (e.g., work/school/family balance) and perceived value of components

Ø  Partner meeting concerns for participants further into their pathway

•  Meetings frequency reduced

Ø  New pathway start-up problems

Factors Supporting Program Progress Ø  High parent motivation for obtaining a career, not a

job

Ø  Peer supports and cohort approach

Ø  Career coaches

Ø  Performance-based financial incentives

Ø  Customized program schedule

Ø  CAP connection: “CAP treats you like family.”

Family Life Study Research Questions

1.  Does participation in CareerAdvance ® relate to positive short- and long-term outcomes for parents and children?

2.  Are there any negative outcomes?

3.  To what extent do parent and child outcomes vary as a function of CareerAdvance® dosage and persistence?

Mixed Methods: Quantitative Ø Computer-assisted parent survey (90 mins.)

Ø Direct child assessments

Ø Teacher survey (i.e., basic/professional demographics; classroom characteristics; child’s social skills, executive functioning, temperament; quality of parent relationship)

Ø Oklahoma administrative data (e.g., UI employment and earnings, TANF, SNAP records)

Ø Head Start Agency (CAP Child-Plus) data (e.g., dosage, persistence, baseline characteristics)

Overview of Constructs: Parent Survey

•  Demographic characteristics •  Education, employment, & income •  Mobility •  Self-confidence and self-efficacy •  Academic expectations for child •  Parenting attitudes and practices •  Mental and physical health •  Moving from job to career

Parent Survey: Overview of Constructs

Psychological Measures l  Conscientiousness l  Optimism l  Self-Esteem l  Impulsivity l  Financial Worry l  Psychological Distress (K6, Kessler et al. 2002): 6%

of CAP Family Life Study sample reported “severe psychological distress” v. 3% in large NHIS sample

l  Material Hardship (6 item scale adapted from New Hope)

Overview of Measures: Child Assessments

Basic Academic Skills

Bracken School Readiness Scale

Math

Woodcock Johnson Achievement Test- Applied Problems

Executive Functioning

Pencil Tap

Language

Peabody Picture Vocabulary Test (PPVT-III)

Gift task

Internal Representations Adapted Berkeley Puppet Interview

CAP Family Life Study: Sample (Cohort 4-6; n=163)

Parent Demographics M(SD) Age 29.47(6.64) Male 3% Adult race/ethnicity (%) White 29% Black/African American 40% American Indian 7% Latino 10% Other/Two or more races/ethnicities 13% Country of origin (%) United States 94% Mexico 5% European Country 1%

CAP Family Life Study: Sample (Cohort 4-6; n=163)

Household Characteristics and Family Structure M(SD) Household Size 4.31(1.40) Number of children in home <18 2.27(1.06) Income-to-needs ratio 1.13(0.73) Relationship Status Married 34% Steady romantic relationship 31% On-again/off-again relationship 8% Single, not in a relationship 27%

0 5

10 15 20 25 30 35 40

Perc

ent

Baseline Parent Education Level Cohort 4- 6 (n=163)

Parent Interview and Child Assessment Year 1 Year 2 Year 3 Year 4 Year 5

Cohort 2011 2012 2013 2014 2015

S F S F S F S F S F 4 60 60 60 60 5 60 60 60 60 6 60 60 60 60 7 60 60 60 8 60 60 60 9 60 60

10 60 60 Total Interviews/

Assessments Per Year

60 180 300 420 420

* S=Spring; F=Fall

Data Collection: Quantitative

Preliminary Persistence Findings

Program /Study   Sample   Results  

Opening Doors Program (2003-2004)

537 low-income students across two community colleges

32% enrolled one year later

New Chance Demonstration (1989-1992)

1,401 low-income, young mothers

52% attained GED or diploma after 42 months

Learning, Earning, and Parenting Program (LEAP) (1987-1997)

267 participants enrolled in school

46% attained GED or diploma after 36 months

Teenage Parent Demonstration (TPD) (1987-1991)

1,739 participants 56% attained GED or high school diploma

Targeted Educational Programs

Program /Study   Sample   Results  

Year-Up (2007-2008)

120 participants (18-26 years)

64% participants completed year-long program

Project QUEST, Inc. (1998-1999)

1,059 participants over 3 years

72% average graduation rate across 3 years

Job Training Partnership Act Title II-A Programs (1987-1989)

6,102 adult women across 16 sites

32% of women who needed a GED or diploma received one within 30 months

Past Workforce Training Programs

Cohort  4  Weeks                                                                                                                                                                                      0   8   16   24   32   40   48   56   64   72   80   88   96   104   112   120   128   136   144   152   160   168   176   184   192   200                                                                                                                                                                                                                  

ANS

CNA  1  

CNA2        

                             CN

A3  

                                                                                                                                                   

PCT   RN  Pre  Req   RN      

                                                                                                                                               

LPN   RN  Pre  Req   RN      Cohort  5  Weeks                                                                                                                                                                                      0   8   16   24   32   40   48   56   64   72   80   88   96   104   112   120   128   136   144   152   160   168   176   184   192   200                                                                                                                                                                                                                  

ANS

CNA  1  

CNA2                      

                                                                                                                                               

PCT   RN  Pre  Req   RN      

                                                                                                                                               

LPN   RN  Pre  Req   RN  

KEY      

    Cer%fica%on  

Pink  =   CNA  Courses    

Blue=   PCT  Track  

Orange  =   LPN  Track  

Green  =   RN  Courses  

(1  semester  =   16  weeks)  

Certification

Cohort  6  Weeks                                                                                                                                                                                      0   8   16   24   32   40   48   56   64   72   80   88   96   104   112   120   128   136   144   152   160   168   176   184   192   200                                                                                                                                                                                                                  

ANS

CNA  1    

CNA2  

                                                                                                                                               

PCT   RN  Pre  Req   RN      

                                                                                                                                               

LPN   RN  Pre  Req   RN  Cohort  7  Weeks                                                                                                                                                                                      0   8   16   24   32   40   48   56   64   72   80   88   96   104   112   120   128   136   144   152   160   168   176   184   192   200                                                                                                                                                                                                                  

ANS    

CNA1  

   CNA2  

                                                                                                                                                   

PCT   RN  Pre  Req   RN      

                                                                                                                                               

LPN   RN  Pre  Req   RN      

CareerAdvance® Nursing Cohorts 4-7 (n=99)

Defining Educational Success in CareerAdvance®

Ø  Persistence l  Enrolled at end of year one l  Number of days enrolled

Ø  Transitions l  Exit->re-enter

Ø  Educational advancement l  Certificates (number, proportion, y/n) l  Classes (number, proportion)

Ø  Program participation l  Number of partner meetings

Enrollment & Certification in CareerAdvance® at One Year (Cohorts 4-7)

CareerAdvance® Participants

(n=99)

Exited (n=32)

Attained certificate

(n=15)

Did not attain certificate

(n=17)

Enrolled (n=67)

Attained certificate

(n=51)

Did not attain certificate

(n=16) 67% enrolled at end of Yr 1 66% attained certificate by end of Yr 1

Predictors of Persistence in CareerAdvance® (Cohorts 4-7; n=99)

Probability of educational success after one year

(odds ratios)

Financial Circumstances Enrollment Certification

Material Hardship 1.88* 1.69*

Income to needs 1.41 2.08* Employed adult in home 0.95 0.91

Psychological wellbeing Psychological distress 0.77 0.53* Optimism 1.20 0.66

Includes controls for baseline demographic characteristics *p<.05

Predictors of Persistence in CareerAdvance®

In-depth interviews at baseline, 2 Cohorts (n=21)

•  Among parents who expressed financial worry at baseline, many completed certification in first year

•  Among parents who did not advance academically in year one, most described psychological distress

Predictors of Persistence in CareerAdvance®

In-depth interviews (n=21)

•  Almost all parents expressed high levels of educational motivation

•  Among parents who completed certification

•  Describe plans for financial impact of program

•  Identify potential barriers to or worries about potential to succeed

Preliminary Qualitative Findings

Types of Annual Data Collection

PARENT INTERVIEWS

FOCUS GROUPS, PARENTS

FOCUS GROUPS,

STAFF

CA Parents

MC Parents

CA Exits

CA Coaches & Program Manager

Family Support Staff

Once-in-a-Lifetime Opportunity

It’s amazing. I mean, you can’t go anywhere to where any program will literally pay for anything and support you all the way. That’s what amazed me the most - the before and after care, the tuition, the scrubs, stethoscopes, books, dictionaries, shoes, watches, gas money.... You’d seem ungrateful if you dropped out and said ‘No I’m not doing that.’ You really wouldn’t have an explanation.

Increased Confidence

It’s been so long since I’ve been in school. I spent almost the last 5 years at home being a mom, and you start doubting yourself and you don’t think you can do it. Once you’re back in the routine of it, and you’re like, ‘Yeah, it’s still in there. Just clean off the cobwebs.’

Coordinated Parent-Child Schedule

I like how they’ve made the program fit around the youngest child’s schedule... how they’ve tailored it to fit around those hours, which really would tailor around all school-age children’s hours. So only during clinical times do you have to really worry about before and after care. But for the most part, all of us can still take the kids, kiss them goodbye, do our thing, and then be there to pick them up.

Extraordinary Peer Support

I know if I tried to leave this program, I would have some people on my phone. And that’s the good thing about us being, that’s the one good thing about us being a small group of people. If one of us tried to leave it, oh, we gonna be on that phone quick, ‘Wait a minute what are you doing?’

Intensive Support from Career Coaches and Center Staff

My favorite part is so much support we’re getting. We can pretty much call her [the coach] anytime and be like, you know.... We constantly have the support not only from our classmates but also from our teachers and our coach. You know, and when I was in college before, it was just me against the world basically you know. So if I dropped out, nobody cared. It was just, I was only just disappointing myself. Now if anybody is missing too much class we’d call them and are like, you know ‘Where are you at? Come to class’.

Stress and Strain

It’s just harder for me because now I’m having to work, like, with any free time that I have, on the weekends or at nights or anything else. You know, me and my husband, If it comes down to choosing between school and work, I have to pick work.

Engagement and Learning

I have found on a positive note, what school has done in our house is…Like my 9-year old has always struggled in math. And I have always struggled in math. It’s never been a strong suit. I’ve always told her that, you know, sorry I can’t really help you. And she’s relied on that, “Well mommy can’t help me. She doesn’t get numbers…”

Engagement and Learning …

Well when I got put in this math tutoring class, I felt like I could then relate to her more, and I felt like it was empowering me because it was giving me those skills that I left behind somewhere in high school and junior high. And so when I would get home, for the first couple of weeks, I’d be like, “I can help you.” She’s like, “No you can’t, you don’t know how to do this” and I was like, “No, really, I know how to do it now.”

Engagement and Learning …

So I feel like, I wasn’t getting so upset with her because now I know the material and understand it and I’m getting it so it’s helping her to feel better about herself, and I feel better about myself because for all those years, it was embarrassing to tell your 9-year old, “Sorry I can’t help you with this because I don’t know it myself…So I feel like that’s been a positive is that I can guide them better now, that I have the information, I can help them better.

Role Modeling

I’m the first person to even go to school. So it feels good to me to just know that I’m gonna make a better, like pave a better path for my son. The chances of him going to school if I complete school are so much higher. And that’s you know, not only will I create a better life for him as a child, but it’ll give him some encouragement and motivation, and I can be a better role model for him to go to school when he’s older. So it makes me feel a lot better I think.

Role Modeling …

My son knows at 4 years old that he’s not stopping his education after high school, he’s gonna keep going. And he knows that now, and he’s, you know...I think he’s gonna be much more prepared than I was when I was in high school.

Less Time with Children

So it’s really hard cause you want to spend time with your family but you don’t have the time to spend with them you want to. Or you have your, you know, your toddler coming up to you, “Mama, mama” and you’re like, oh, I have to write a paper, you know, that’s really hard.

“This program has changed my life; it’s changed my future, my family’s future definitely. I mean, this has opened up so many opportunities for me and my family.”

Hope for the Future

Ø  Long-term follow-up of parents and children in CAP Family Life Study

Ø  Deepen understanding of two-gen programs Ø  Scale-up, sustain CareerAdvance® and expand

beyond Tulsa Ø  Conduct RCTs with varying 2-gen program

models Ø  Develop strategies for informing broad policy and

practitioner communities Ø  Address tension between fostering political will

and building research evidence

Future Goals