advancing evidence-...wisconsin making the future a hope, college c consortium for bioscience a...

Grant Blume, Elizabeth Meza, Ivy Love, and Debra Bragg

ADVANCING EVIDENCE-

BASED POLICYMeta-Analysis Findings from the Trade Adjustment Assistance Community College Career Training Grant (TAACCCT) Program

This project was funded by the Lumina Foundation

WOODGROVEBANK

CONTEXT AND BACKGROUND

2019ASHE

The Great Recession of 2008-2009

Congress authorizes the Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program

TAACCCT’s goal: “Provide workers with the education and skills to succeed in high-wage, high-skill occupations” (DOL, 2016, p. 3).

WOODGROVEBANK


2019ASHE



From 2011-2018, more than 200 federal grants invest nearly $2 billion in 700+ community colleges

WOODGROVEBANK


2019ASHE



A unique aspect of TAACCCT: required third-party evaluation reports, ideally using quasi-experimental design (QED) to evaluate impact of TAACCCT funds

WOODGROVEBANK

RESEARCH QUESTIONS

2019ASHE

1. To what extent does evaluation data suggest that TAACCCT funding had an observable impact on student-level outcomes?

2. What are the benefits and drawbacks of synthesizing third-party evaluation data to measure impact across federal grants to community colleges?

WOODGROVEBANK

METHODS

2019ASHE

Meta-analysis & thematic heat mapping

Quantitative Qualitative understanding impact of of TAACCCT’s impact

TAACCCT

WOODGROVEBANK

METHODS

2019ASHE

Meta-analysis

Synthesizes research findings on a particular construct or outcome of interest

EducationProgram Completion

Credential Completion

EmploymentPost-Program Wage Gain

Post-Program Employment

WOODGROVEBANK

METHODS

2019ASHE

Meta-analysis

ln($OR') =⁄a' c'⁄b' d'

Y=1Y=0

WOODGROVEBANK

METHODS

2019ASHE

Meta-analysis

ln($OR') =⁄a' c'⁄b' d'

se{ln $𝑂𝑅4 } = ⁄1 a' + ⁄1 b' + ⁄1 c' + ⁄1 d'

WOODGROVEBANK

METHODS

2019ASHE

Meta-analysis

ln($OR') =⁄a' c'⁄b' d'

se{ln $𝑂𝑅4 } = ⁄1 a' + ⁄1 b' + ⁄1 c' + ⁄1 d'

8𝑤4 = ⁄1 { se :θ4< + �̂�<

WOODGROVEBANK

METHODS

2019ASHE

Heat mapping

Systematic review of studies meeting inclusion criteria, coding 36 programmatic components across six core elements:

Career Pathways Online & Open Educational ResourcesTransfer and Articulation PartnershipsRetention Strategies Evaluation & Sustainability

WOODGROVEBANK

DATA

2019ASHE

Phase One

Phase Two

Phase Three

Phase Four

WOODGROVEBANK

FINDINGS: EDUCATION OUTCOMES

2019ASHENOTE: Weights are from random effects analysis

.

.

Overall (I-squared = 97.9%, p = 0.000)

Kan-TRAIN

Golden Triangle

Consortium for BioScience B

New Mexico SUNAdv. Manufacturing for Global EconomyLA HealthcareHope, College E

Northeast Resiliency Consortium B


Northeast Resiliency Consortium

HOPE, College D

Northeast Resiliency Consortium A

Retraining the Gulf Workforce

ID

North Dakota AM

Minneapolis MAAC A

Iowa Advanced Manufacturing

HOPE, College A

Ohio Tech Net

Allied Health Expansion A

CT Health and Life Sciences

HOPE, College B

Amplifying Montona

Florida XCEL-IT A

Greater Memphis Alliance

Adult Competency Based Education AProgram Completion

Minneapolis MAAC B

Healthcare Careers Work

ACT for Healthcare ABridging the Gap

Online2Workforce

Maine is IT

UDC Construction and Hospitality

Building Illinois Bioeconomy

AME Manufacturing A

Mission Critical Operations

Heroes for Hire


Wisconsin Making the Future A

HOPE, College C

Consortium for BioScience A

Project IMPACT

Competency-Based Education in Community Colleges

Study

1.91 (1.49, 2.43)

1.56 (1.33, 1.81)

10.67 (7.30, 15.58)

0.52 (0.29, 0.91)

4.58 (4.05, 5.19)6.27 (2.25, 17.46)6.67 (6.06, 7.35)6.91 (0.83, 57.34)

1.18 (1.02, 1.35)

0.95 (0.81, 1.12)

1.19 (0.37, 3.82)

4.85 (4.12, 5.71)

1.59 (1.40, 1.81)

ES (95% CI)

2.30 (1.40, 3.79)

3.76 (1.47, 9.62)

2.32 (1.24, 4.35)

1.40 (0.13, 14.74)

1.65 (1.37, 1.98)

1.77 (0.99, 3.17)

1.14 (0.91, 1.42)

1.83 (0.63, 5.30)

0.90 (0.58, 1.38)

7.67 (4.75, 12.40)

1.76 (1.43, 2.15)

0.74 (0.60, 0.91)0.75 (0.41, 1.38)

1.72 (1.31, 2.26)

2.74 (2.42, 3.10)2.81 (1.71, 4.62)

2.30 (1.66, 3.18)

1.45 (0.76, 2.76)

2.22 (1.52, 3.24)

0.85 (0.40, 1.78)

1.37 (0.90, 2.07)

1.10 (0.96, 1.25)

1.82 (1.26, 2.62)

0.98 (0.69, 1.40)

2.15 (1.94, 2.39)

2.88 (0.77, 10.77)

0.55 (0.37, 0.81)

11.75 (5.71, 24.18)

0.82 (0.75, 0.89)

100.00

2.78

2.63

2.43

2.791.882.790.90

2.78

2.77

1.71

2.77

2.79

Weight

2.51

1.98

2.37

0.78

2.76

2.42

2.74

1.83

2.58

2.53

2.75

2.752.39

2.71

2.792.52

2.67

2.35

2.63

2.22

2.60

2.78

2.64

2.65

2.79

1.54

2.62

2.25

2.80

%

1.91 (1.49, 2.43)

1.56 (1.33, 1.81)

10.67 (7.30, 15.58)

0.52 (0.29, 0.91)

4.58 (4.05, 5.19)6.27 (2.25, 17.46)6.67 (6.06, 7.35)6.91 (0.83, 57.34)

1.18 (1.02, 1.35)

0.95 (0.81, 1.12)

1.19 (0.37, 3.82)

4.85 (4.12, 5.71)

1.59 (1.40, 1.81)

ES (95% CI)

2.30 (1.40, 3.79)

3.76 (1.47, 9.62)

2.32 (1.24, 4.35)

1.40 (0.13, 14.74)

1.65 (1.37, 1.98)

1.77 (0.99, 3.17)

1.14 (0.91, 1.42)

1.83 (0.63, 5.30)

0.90 (0.58, 1.38)

7.67 (4.75, 12.40)

1.76 (1.43, 2.15)

0.74 (0.60, 0.91)0.75 (0.41, 1.38)

1.72 (1.31, 2.26)

2.74 (2.42, 3.10)2.81 (1.71, 4.62)

2.30 (1.66, 3.18)

1.45 (0.76, 2.76)

2.22 (1.52, 3.24)

0.85 (0.40, 1.78)

1.37 (0.90, 2.07)

1.10 (0.96, 1.25)

1.82 (1.26, 2.62)

0.98 (0.69, 1.40)

2.15 (1.94, 2.39)

2.88 (0.77, 10.77)

0.55 (0.37, 0.81)

11.75 (5.71, 24.18)

0.82 (0.75, 0.89)

100.00

2.78

2.63

2.43

2.791.882.790.90

2.78

2.77

1.71

2.77

2.79

Weight

2.51

1.98

2.37

0.78

2.76

2.42

2.74

1.83

2.58

2.53

2.75

2.752.39

2.71

2.792.52

2.67

2.35

2.63

2.22

2.60

2.78

2.64

2.65

2.79

1.54

2.62

2.25

2.80

%

1.1 1 10

32 studies41 effects

WOODGROVEBANK

FINDINGS: EDUCATION OUTCOMES

2019ASHE


NOTE: Weights are from random effects analysis

.

.

Overall (I-squared = 97.9%, p = 0.000)

Kan-TRAIN

Golden Triangle

Consortium for BioScience B

New Mexico SUNAdv. Manufacturing for Global EconomyLA HealthcareHope, College E

Northeast Resiliency Consortium B


Northeast Resiliency Consortium

HOPE, College D

Northeast Resiliency Consortium A

Retraining the Gulf Workforce

ID

North Dakota AM

Minneapolis MAAC A

Iowa Advanced Manufacturing

HOPE, College A

Ohio Tech Net

Allied Health Expansion A

CT Health and Life Sciences

HOPE, College B

Amplifying Montona

Florida XCEL-IT A

Greater Memphis Alliance

Adult Competency Based Education AProgram Completion

Minneapolis MAAC B

Healthcare Careers Work

ACT for Healthcare ABridging the Gap

Online2Workforce

Maine is IT


Building Illinois Bioeconomy

AME Manufacturing A

Mission Critical Operations

Heroes for Hire


Wisconsin Making the Future A

HOPE, College C

Consortium for BioScience A

Project IMPACT

Competency-Based Education in Community Colleges

Study

1.91 (1.49, 2.43)

1.56 (1.33, 1.81)

10.67 (7.30, 15.58)

0.52 (0.29, 0.91)

4.58 (4.05, 5.19)6.27 (2.25, 17.46)6.67 (6.06, 7.35)6.91 (0.83, 57.34)

1.18 (1.02, 1.35)

0.95 (0.81, 1.12)

1.19 (0.37, 3.82)

4.85 (4.12, 5.71)

1.59 (1.40, 1.81)

ES (95% CI)

2.30 (1.40, 3.79)

3.76 (1.47, 9.62)

2.32 (1.24, 4.35)

1.40 (0.13, 14.74)

1.65 (1.37, 1.98)

1.77 (0.99, 3.17)

1.14 (0.91, 1.42)

1.83 (0.63, 5.30)

0.90 (0.58, 1.38)

7.67 (4.75, 12.40)

1.76 (1.43, 2.15)

0.74 (0.60, 0.91)0.75 (0.41, 1.38)

1.72 (1.31, 2.26)

2.74 (2.42, 3.10)2.81 (1.71, 4.62)

2.30 (1.66, 3.18)

1.45 (0.76, 2.76)

2.22 (1.52, 3.24)

0.85 (0.40, 1.78)

1.37 (0.90, 2.07)

1.10 (0.96, 1.25)

1.82 (1.26, 2.62)

0.98 (0.69, 1.40)

2.15 (1.94, 2.39)

2.88 (0.77, 10.77)

0.55 (0.37, 0.81)

11.75 (5.71, 24.18)

0.82 (0.75, 0.89)

100.00

2.78

2.63

2.43

2.791.882.790.90

2.78

2.77

1.71

2.77

2.79

Weight

2.51

1.98

2.37

0.78

2.76

2.42

2.74

1.83

2.58

2.53

2.75

2.752.39

2.71

2.792.52

2.67

2.35

2.63

2.22

2.60

2.78

2.64

2.65

2.79

1.54

2.62

2.25

2.80

%

1.91 (1.49, 2.43)

1.56 (1.33, 1.81)

10.67 (7.30, 15.58)

0.52 (0.29, 0.91)

4.58 (4.05, 5.19)6.27 (2.25, 17.46)6.67 (6.06, 7.35)6.91 (0.83, 57.34)

1.18 (1.02, 1.35)

0.95 (0.81, 1.12)

1.19 (0.37, 3.82)

4.85 (4.12, 5.71)

1.59 (1.40, 1.81)

ES (95% CI)

2.30 (1.40, 3.79)

3.76 (1.47, 9.62)

2.32 (1.24, 4.35)

1.40 (0.13, 14.74)

1.65 (1.37, 1.98)

1.77 (0.99, 3.17)

1.14 (0.91, 1.42)

1.83 (0.63, 5.30)

0.90 (0.58, 1.38)

7.67 (4.75, 12.40)

1.76 (1.43, 2.15)

0.74 (0.60, 0.91)0.75 (0.41, 1.38)

1.72 (1.31, 2.26)

2.74 (2.42, 3.10)2.81 (1.71, 4.62)

2.30 (1.66, 3.18)

1.45 (0.76, 2.76)

2.22 (1.52, 3.24)

0.85 (0.40, 1.78)

1.37 (0.90, 2.07)

1.10 (0.96, 1.25)

1.82 (1.26, 2.62)

0.98 (0.69, 1.40)

2.15 (1.94, 2.39)

2.88 (0.77, 10.77)

0.55 (0.37, 0.81)

11.75 (5.71, 24.18)

0.82 (0.75, 0.89)

100.00

2.78

2.63

2.43

2.791.882.790.90

2.78

2.77

1.71

2.77

2.79

Weight

2.51

1.98

2.37

0.78

2.76

2.42

2.74

1.83

2.58

2.53

2.75

2.752.39

2.71

2.792.52

2.67

2.35

2.63

2.22

2.60

2.78

2.64

2.65

2.79

1.54

2.62

2.25

2.80

%

1.1 1 10

WOODGROVEBANK

FINDINGS: EMPLOYMENT OUTCOMES

2019ASHENOTE: Weights are from random effects analysis

.

.Overall (I-squared = 91.9%, p = 0.000)

Golden Triangle

Kan-TRAIN

I Am STAR

New Mexico SunAllied Health Expansion C

New Mexico SUN

Project IMPACT

Florida XCEL-IT

ShaleNET

Allied Health Expansion B

PluggedIn and Ready to Work

Adult Competency Based Education B

Wage Gain

ACT for Healthcare B

Online2Workforce

ID

ACT for Healthcare C

AME Manufacturing B

Wisconsin Making the Future B

AME Manufacturing C

Employment

Study

1.27 (1.00, 1.61)

3.68 (2.67, 5.08)

0.89 (0.69, 1.14)

1.54 (0.47, 5.01)

1.30 (1.17, 1.46)1.64 (1.06, 2.53)

2.19 (1.96, 2.46)

0.86 (0.27, 2.80)

0.83 (0.57, 1.21)

1.81 (1.13, 2.91)

1.11 (0.51, 2.38)

2.12 (0.96, 4.70)

0.57 (0.46, 0.70)

1.40 (1.08, 1.82)

0.61 (0.38, 0.98)

ES (95% CI)

1.29 (1.07, 1.56)

0.86 (0.47, 1.58)

1.21 (0.97, 1.51)

1.64 (0.71, 3.81)

100.00

6.41

6.73

2.58

7.155.84

7.15

2.60

6.14

5.64

4.12

3.98

6.88

6.67

5.64

Weight

6.95

4.91

6.84

3.78

%

1.27 (1.00, 1.61)

3.68 (2.67, 5.08)

0.89 (0.69, 1.14)

1.54 (0.47, 5.01)

1.30 (1.17, 1.46)1.64 (1.06, 2.53)

2.19 (1.96, 2.46)

0.86 (0.27, 2.80)

0.83 (0.57, 1.21)

1.81 (1.13, 2.91)

1.11 (0.51, 2.38)

2.12 (0.96, 4.70)

0.57 (0.46, 0.70)

1.40 (1.08, 1.82)

0.61 (0.38, 0.98)

ES (95% CI)

1.29 (1.07, 1.56)

0.86 (0.47, 1.58)

1.21 (0.97, 1.51)

1.64 (0.71, 3.81)

100.00

6.41

6.73

2.58

7.155.84

7.15

2.60

6.14

5.64

4.12

3.98

6.88

6.67

5.64

Weight

6.95

4.91

6.84

3.78

%

1.1 1 10


WOODGROVEBANK

FINDINGS: HEAT MAPPING

2019ASHE


Core Elements

Program Completion (19 Studies)

Credential Completion (17 Studies)

Employment (16 Studies)

Wage Change

(5 Studies)Career PathwaysStacked & latticed credentialsCredit for Prior Learning & PLAComprehensive student successCareer guidance & servicesDevelopment education reformIntegrated Basic Skills & Career TrainingNon-credit to credit bridgesCBE, CB assessments & CB training Modularized curriculumWork-based learningPre-apprenticeships & ApprenticeshipsTransfer & ArticulationTransfer and articulation of creditApplied baccalaureate degreesRetention StrategiesAccelerated program modelSelf-paced learningRestructured course scheduling

76-100% QED studies51-75% QED studies26-50% QED studies1-25% QED studies0% QED studies

WOODGROVEBANK

FINDINGS: HEAT MAPPING

2019ASHE


Core Elements

Program Completion (19 Studies)

Credential Completion (17 Studies)

Employment (16 Studies)

Wage Change

(5 Studies)Online & OEROnline & technology-enabled learningAccelerated course delivery/HybridMOOCsPersonalized tech-enabled instructionOnline CBE & CB assessmentsEducational gamingTech-enabled tutoringRolling & open enrollmentAsynchronous & real-time collaboration"Next Gen" assessmentOERPartnershipsEmployer engagementPublic workforce partneringExplicit use of sector strategiesState Gov/Governor partneringPhilanthropic/External funding partneringLabor partneringTAACCCT grant partneringEvaluation & SustainabilitySustainabilityEvaluation & Continuous Improvement

WOODGROVEBANK

RESEARCH QUESTIONS

2019ASHE

1. To what extent does evaluation data suggest that TAACCCT funding had an observable impact on student-level outcomes?

Of the studies meeting our inclusion criteria, we find TAACCCT funding had a positive effect on student-level education and employment outcomes.

WOODGROVEBANK

RESEARCH QUESTIONS

2019ASHE

2. What are the benefits and drawbacks of synthesizing third-party evaluation data to measure impact across federal grants to community colleges?

Third-party evaluation reports represent a rich source of quantitative and qualitative data; methods and rigor varied widely and were largely a result of limited data.

Grant Blume, Elizabeth Meza, Ivy Love, and Debra Bragg

ADVANCING EVIDENCE-

BASED POLICYMeta-Analysis Findings from the Trade Adjustment Assistance Community College Career Training Grant (TAACCCT) Program

This project was funded by the Lumina Foundation

advancing evidence-...wisconsin making the future a hope, college c consortium for bioscience a...

Documents