National Council for Workforce Education (NCWE)Darlene G. Miller, Ed.D., Executive Director

Heldrich Center for Workforce Development, Rutgers UniversityMichelle Van Noy, Ph.D., ResearcherJennifer Cleary, Ph.D., Researcher

Burning Glass TechnologiesKelly R. Bailey, Director of Partnerships and Alliances

Aligning Workforce Programs with the Labor Market: Strategies and Tools for TAACCCT Applications

What this presentation can do for you

Provide a framework for thinking about how to build TAACCCT-funded workforce programs that align with the labor market, as well as the needs of students and other stakeholders.

Offer tools and information to aid in building strong programs.

Provide tips on how to build an evaluation-friendly workforce program.

Overview of SGA and Labor Market Alignment

USDOL Priorities Challenge Colleges to Create New Program Models

Key Goals: Align programs with the needs of the labor

market and students Use sector-based & evidence-based approachesEvaluate program implementation and

outcomes

Data systems and administration goalsOpen sharing of curriculaCoordinated data collection across programs

Evidence-Based Design

Three levels of evidence:

Strong

Moderate

Evidence of Promise

Strong Theory

Priority program approaches:

• Sector-based• Career Pathways & Stackable

Credentials• Employers involved in ALL PARTS OF

PROGRAM• Use Real-time and other LMI (program

selection, student advising)• On-line learning• Accelerated, contextualized remedial

education• Competency-based & Work-based

learning (including entrepreneurship)• Share outcomes and LMI with students • Credit granting for non-credit work

Higher Education Labor Market Program-level Alignment Model

Program Selection

Recruitment &

Enrollment

Curriculum, Teaching,

& Assessmen

t

Support Services

Employer Needs

Student Needs

Data Collection/Validation

Incorporate

Relationship-building

Outcomes

Credential Attainment

Employment outcomes

Employer/Student Satisfaction

Incorporating data collected into the program is a multifaceted, complex process. It is more than program selection!

Employer & Student Needs

Contextual Factors

•Institutional Factors

•Program/ Institutional Structure

•Labor Market Dynamics

The SGA requires alignment activities occur at many organizational levels

System

Class

- Data systems to share student outcomes- Policies for open sharing curriculaPolicies/systems/facilities to

allow:- Credit for non-credit work- Flexible schedule /enrollment- IT/On-line instruction- Accelerated & competency-based learning - Work-based/hands-on learning

- Policies to allow for sharing of faculty for contextualized ed.

Systems and skilled staff for: -Data collection and analysis-Incorporating data into program selection and all program activities-Measure interim goals-Build relationships with stakeholders

- Effective pedagogy for contextualized and accelerated learning

Strategies for Successful Program Design

Alignment Activities: Systemic Change

Grant Leads to System Change within the Institution (not a boutique program)

Embedded into the culture of the institution Difficulty of navigating internal barriers

College level commitment Administrative Engagement Cross-functional Leadership Team

For Consortiums Consortia Leadership Team Consortium Director

Alignment Activities: Relationship Building

Relationships to Improve Outcomes andMeet Grant Priorities

DOL Priority on Long Term Unemployed Relationship with Workforce Investment Boards

Strong Employer Engagement and Commitment Career pathways and stackable credentials

Employer validation and data validation Employer validation of supply/demand Role of Industry Associations Employer commitment to work-based learning

Alignment Activities: Relationship Building

Relationships to Improve Outcomes andMeet Grant Priorities

For Consortium Grants – Peer Learning Networks Learning best practices from each other Sharing curriculum: Don’t reinvent the wheel Online and face-to-face

Platform and compatibility Data collection and sharing

Data systems in synch

Alignment Activities: Data Collection and Validation

Alignment with Needs of Labor Market and Students

Real Time Jobs Analysis* Allows understanding of job vacancy and skill needs of

local labor market Utilizing multiple “reliable” sources

LMI Data, Local WIB data, state data, industry association data

Gap analysis of competing programs What are other colleges in your area doing What about local CBOs or apprenticeship programs Understand available supply to fill job vacancies

Alignment Activities: Data Collection and Validation

Alignment with Needs of Labor Market and Students Convening stakeholder groups to collect information on job

vacancy and skill demand Multiple relationship building activities (WIBs, employers, etc) Ensures multiple sources of evidence for demand and supply

Validation by the right person (HR manager v CEO) Allows for collection of skills information for curriculum

development Locally recognized industry-based credentials

Aligning enrollment targets with job vacancies and gaps Employment data over time

Alignment Activities: Program Implementation

Curriculum Development Open sharing of curriculum

Curriculum is no long proprietary or owned by the faculty Open Sharing On-line using DOL platform

• Faculty support in budget for making the curriculum public and accessible

Acceleration strategies Evidence-based design Integration and contextualization for low-skilled adult

learners

Alignment Activities: Program Implementation

Curriculum Development Credit vs Noncredit

Noncredit must be linked to industry-recognized credentials (NIMS, NCCER, etc)

Articulation between noncredit and credit Acceleration and Credit for Prior Learning

BE CLEAR WITH THE FACULTY UPFRONT REGARDING PLA College policy Accreditation Transparency

“Real Time” Jobs Data

Evaluation Approaches and Considerations

Evaluation Consideration : Outcomes Measurement and Program Design

Randomized Controlled Trials Plan for recruitment and assignment?

Regression Discontinuity Assessment tests with cut-off scores? Common across colleges?

Comparison Group with Controls Drawn from similar programs at college either from prior or

current years

Evaluation Consideration : Outcomes Measurement & Data Collection

Develop a plan to collect key student data Including Student Name, SSN, DOB

Labor market outcomes data Partnerships with state to link with UI wage data

Other important data also need to be collected Courses, grades, services received, assessments, scores, #

attempts, activity in on-line courses, attendance

Plan to store data & ensure consistent data entry

Evaluation Considerations: Program Implementation Goals

Document implementation to inform outcomes analysis Provide detail on how program was implemented

including strengths and weaknesses

Uncover opportunities to strengthen implementation Evaluation can be an opportunity for on-going

learning, reflection, and feedback

Evaluation Considerations: Program Implementation Components

Multiple data collection efforts to provide a full understanding of implementation may include: Focus groups Surveys Interviews

Include the perspectives from a variety of stakeholders including: College staff, students, employers, other partners

Evaluation Considerations: Program Implementation Issues

How did you choose program design? Employer involvement? Evidence?

Does the delivery method fit student and employer needs? Are the assessments sufficient and well implemented?

Abilities, skills and interests What ? Who? How implemented? How used?

Is the program structure sufficient to ensure data collected on skills, etc. is incorporated in all program components? Is there a communication or quality control system in place to ensure all parts of

the program are coordinated around stakeholder priorities. Are your supportive services directly addressing employer, as well

as, student needs? Career guidance for students, support of employer skill priorities Are your partners involved in all aspects of the program

What relationship building activities have you created to recruit, retain, and engage partners?

Final Thoughts

Final Thoughts

Create a dashboard of metrics right up front along with a workplan for all of the partners Make sure all of your partners can live with the plan and

understand their role in accountability Don’t bite off more than you can chew

Be realistic in your outcomes Just because you think a reviewer wants to see this,

don’t put things into the grant that you cannot accomplish

Increased scrutiny by DOL Congress wants to know that their money was well spent Importance of a strong evaluation plan Take audits seriously

NCWE Team of TechnicalAssistance Providers

Darlene G. Miller, Ed.D., NCWE Executive Director

Nan Poppe, Ed.D., Campus President Emeritus, Portland Community College

Rebecca Nickoli, Ed.D., former Vice President of Corporate College (Workforce and Economic Development), Ivy Tech Community College

Nick Kremer, Chair, CA Vocational Research and Accountability Committee and former Dean of Career and Technical Education, Cerritos College

For More Information…

Darlene Miller, executivedirector@ncwe.org 603-714-1918

Michelle Van Noy, mvannoy@rutgers.edu 732-932-4100 x6213

Jennifer Cleary, jcleary@ejb.rutgers.edu 732-932-4100 x6215

Kelly R. Bailey, kbailey@burning-glass.com 732-800-2484