www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 1

NEHA NANDA, PH.D, CAROLYN COREA M.S.W., ANDRES P. ROMUALDO

ISSUE BRIEF

Workforce Innovation Fund Grant Evaluation Best Practices & Lessons Learned

OVERVIEW WIF grantees set themselves up for a successful evaluation by identifying a knowledgeable and experienced evaluator that is familiar with the tasks required for evaluation planning and startup to communicating evaluation findings. In addition to evaluation knowledge and experience, a strong evaluator of WIF grant programs must also bring an awareness of navigating the relationships between the grantee, the NEC, and DOL. IMPAQ is currently serving as the third-party evaluator for six WIF grants as well as nine DOL funded Trade Adjustment Assistance Community College and Career Technical Training (TAACCCT) grants.

The WIF grant evaluation recommendations are presented below by implementation topic area. For each task, we note whether the grantee or the evaluator holds primary responsibility for ensuring completion. As shown below, the responsibility for many of these tasks is shared by the grantee and the evaluator, underscoring the importance of a strong working relationship.

WIF Evaluation Communication Flow

Workforce Innovation Fund (WIF) grantees are required by the U.S. Department of Labor (DOL) to hire third-party evaluators to evaluate their proposed programs. All grant evaluations are overseen by DOL’s National Evaluation Coordinator (NEC). Evaluations require grantees and their evaluators to build strong working relationships to facilitate communication among stakeholders, collect program data, and disseminate evaluation findings. The evaluation findings will contribute to the evidence-base supporting WIF grant funded interventions. Evidence supporting program effectiveness could garner additional programmatic funding. This issue brief presents WIF grant evaluation recommendations across the lifecycle of the grant program and evaluation. These recommendations grew out of IMPAQ’s experience evaluating a number of WIF grant programs and were presented at the 2015 WIF Conference as part of the plenary session. They will help new WIF grantees build an understanding of the essential steps involved in a successful grant evaluation.

www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 2

ISSUE BRIEF Workforce Innovation Fund Grant Evaluation Best Practices & Lessons Learned

Evaluation Planning and Startup

Grantee Evaluator Task

ü Provide your evaluator with program documentation early on to build a solid understanding of

the grant program. A program guidebook may help to ensure successful grant implementation as well as evaluation.

ü ü Work with your evaluator, regions/sites, and other program partners/stakeholders to ensure all parties have a solid understanding of the purpose, approach, and requirements of the evaluation. Discuss the roles and responsibilities of the grantee, evaluator, NEC, and DOL.

ü Ensure your evaluator has planned for meetings to discuss the logistical requirements of the grant program. Webinars offer a good opportunity to walk all involved parties through the evaluation process, respond to initial questions, and build rapport.

ü Ensure your evaluator has planned and budgeted for the IRB approval process. Work with your evaluator to ensure grant and evaluation activities are operating in compliance with IRB.

ü If the evaluation requires the collection, tracking, and submission of participant data, request that your evaluator provides data collection templates and how-to guides for tracking and submitting participant data

ü ü Work with your evaluator to prepare grantee/regions/sites to respond to participant questions/concerns about data collection.

Implementing Evaluation Activities

Grantee Evaluator Task

ü Request that your evaluator conduct refresher webinars throughout the evaluation to remind grantee staff and regions/sites of the evaluation process/requirements and respond to questions/concerns.

ü ü Work with your evaluator to gather real time feedback in order to address implementation issues early on in the grant period. Do not wait to receive feedback in the next deliverable.

ü ü Work with your evaluator to share feedback with any regions/sites implementing the grant program.

ü Leverage existing relationships to negotiate access to UI and other administrative data needed for the evaluation.

www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 3

ISSUE BRIEF Workforce Innovation Fund Grant Evaluation Best Practices & Lessons Learned

Communicating Evaluation Findings

Grantee Evaluator Task

ü Request that your evaluator customize reports and deliverables to meet the needs of your organization. Work with your evaluator to communicate those needs.

ü Request that your evaluator frame evaluation findings in terms of replicability/scalability/sustainability.

ü ü If possible, hold meetings with your evaluator and program stakeholders to discuss draft deliverables and provide feedback to be incorporated into final deliverables.

ü ü Work with your evaluator to share reports and other deliverables with program stakeholders.

Implications for the Program and Broader Workforce System

Grantee Evaluator Task

ü

Request that your evaluator discuss implications of evaluation findings in the broader context of workforce development, highlighting any findings that advance overarching workforce development goals such as employer/stakeholder engagement and developing career pathways.

ü Request that your evaluator reflect on the value of building local/regional community efforts/movements around specific workforce development issues such as self-employment/entrepreneurship, and employer engagement.

An experienced evaluator should be ready to work with the grantee to facilitate each of these evaluation tasks. For more information on IMPAQ’s experience conducting third-party evaluations, please

contact info@impaqint.com

www.impaqint.com JANUARY 2016 Evaluating & Enhancing Public Programs & Policies 4

ISSUE BRIEF Workforce Innovation Fund Grant Evaluation Best Practices & Lessons Learned

BIOS Neha Nanda (Ph.D., Economics, Florida State University) is a Research Associate at IMPAQ International, LLC, Dr. Nanda is an applied economist in the fields of labor economics, econometrics and program evaluation methods. Her research focuses on employment and soccioeconomic

issues among the low-income women, children and the youth population. Dr. Nanda has substantial experience leading and managing several projects for DOL and other clients. Currently, for DOL, she is the leading the “Feasibility Study and Evaluation of Non-Traditional Occupation (NTO) Demonstrations project” as well as the recently-funded “Labor Market and DOL-Funded Employment Assistance for Older Workers.” She was also the co-Principal Investigator on the “Evaluation of Women Veterans Economic and Employment Characteristics and Federal Program Service Inventory”. She is the Principal Investigator for the Workforce Innovation Fund (WIF)-funded Evaluation of Startup Quest and part of the research team in a few other WIF-funded evaluations.

Carolyn Corea (M.S.W., University of Maryland, Baltimore) is a Research Associate at IMPAQ International LLC with seven years of experience conducting evaluations of social policies and programs. Ms. Corea is currently serving as IMPAQ's process

study lead on a number of evaluation studies for the U.S. Department of Labor’s (DOL) Workforce Innovation Fund (WIF) and Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant programs. Her responsibilities include developing site visit interview and focus group protocols, training site visitors, conducting site visits, analyzing site visit findings, and providing continuous recommendations for program improvement.

Ms. Corea is also working to support the Feasibility and Design Study on Non-Traditional Occupation (NTO) Demonstration for DOL’s Employment and Training Administration (ETA). The aim of this study is to describe the relevant trends and issues for DOL related to non-traditional occupations, identify evidence-based research findings of promising strategies, identify relevant strategies to test, and prepare an evaluation design for a demonstration of promising strategies. In addition, Ms. Corea worked to support DOL’s Green Jobs and Healthcare Implementation study, an implementation evaluation of four grant programs awarded with American Recovery and Reinvestment Act funds by DOL.

Andrés P. Romualdo, (B.A., University of Maryland, Baltimore County, Psychology and Sociology) is a Research Analyst at IMPAQ International with five years of experience in process/implementation evaluations, qualitative research,

interviews, focus groups, trainings, facilitation, and data collection and analysis. Additionally, he collaboratively identifies problem areas and solutions with clients in process improvement projects. Mr. Romualdo is currently supporting several Department of Labor grant evaluations which includes Workforce Innovation Fund (WIF) and Trade Adjustment Assistance Community College and Career Training (TAACCCT) grants. Throughout these evaluations, Mr. Romualdo has conducted interviews with key stakeholders and program participant focus groups to extract success, challenges, and lessons learned of the program. In this experience, Mr. Romualdo has deep understanding of grant funded program implementation. In addition to his work at IMPAQ, Mr. Romualdo is currently pursuing a M.A. in applied sociology at University of Maryland, Baltimore County.