contextualized essential skills demonstration project...contextualized essential skills...

March 2019

Contextualized Essential Skills Demonstration Project

Final Report

This project is funded in part by the Government of Canada and the Government of Ontario

Executive Summary

A strong link exists between essential skills and labour market success. Research has demonstrated that workers with lower essential skills face barriers to career progression and higher earnings. A significant proportion of Ontario’s workforce has gaps in essential skills - directly impacting Ontario’s ability to develop a more productive and resilient labour pool.

The Contextualized Essential Skills Demonstration Project was a response to this challenge. The project tested an innovative approach to delivering workplace essential skills training to frontline workers in the hospitality and retail sectors in Ontario. Blueprint was engaged by the Ontario Centre for Workforce Innovation (OCWI) to lead this project.

The project also addressed the question of how to deliver a sector-specific essential skills training program given Ontario’s current skills training landscape by:

• Engaging Ontario’s Literacy and Basic Skills (LBS) providers—who are mandated to deliver essential skills training to low-skilled adults in the province—to support the delivery of training.

• Using an employer-focused approach to participant recruitment and training coordination to explore whether this type of training could be funded in the future through the Canada-Ontario Job Grant (COJG)—a funding envelope driven by an employer’s decision to invest in training.


ii

ABOUT THE PROJECT

The Contextualized Essential Skills Demonstration Project was based on the Workplace Training Program—an innovative contextualized essential skills program implemented at Douglas College in British Columbia in 2012.

The Workplace Training Program offered workplace essential skills training to employees in the hospitality and tourism and retail industries, and was evaluated with promising results. The curriculum was adapted and updated to create a new Ontario-branded program called Customer Service Results (CSR). The CSR curriculum contains a series of training modules that integrate technical sector-specific content with essential skills training. In addition to providing training for frontline employees, the program also offers leadership training for supervising staff—as research has found that a supportive supervisor can reinforce the positive effects of workplace essential skills training.

We engaged OTEC, a training, consulting, and workforce development organization, to lead recruitment and training delivery. OTEC collaborated with LBS providers to co-deliver training, and with LBS network organizations to provide employer engagement and coordination support. Douglas College supported the adaptation of the curriculum and delivered “train-the-trainer” sessions to OTEC and LBS trainers.

EVALUATION APPROACH

Blueprint used a three-part strategy to evaluate the CSR program:

• Participant results – Participants in frontline training completed surveys and skills assessments to measure gains in essential skills and other outcomes—such as attitudes towards learning, job motivation, and self-esteem.

• Employer results – Employers completed surveys on training expectations, relevance, and perceived impact. Blueprint also conducted in-depth interviews with a sub-sample of employer partners to explore their business needs and satisfaction with the program in more depth.

• Process evaluation – We carried out a process evaluation to identify success factors, challenges, and opportunities for improving the model. The process evaluation explored the potential role of CSR in the skills training ecosystem in Ontario and identified opportunities for sustaining the model as well as key learnings that could inform the implementation of similar projects. It was also intended to generate some broader learnings about the benefits of the model for industry and government. The process evaluation was based on ongoing engagements with project staff and partners, including OTEC staff, LBS trainers, LBS network organizations, and employers.

PARTICIPANT AND EMPLOYER RESULTS A total of 146 frontline workers in 14 cohorts completed the CSR frontline training.

The gains in essential skills experienced by participants were modest:

• There were small but statistically significant gains in numeracy skills overall.

• There were statistically significant gains in


iii

document use among participants who entered the training with low skill levels (but not for participants overall).

Generally, both participants and employers were very satisfied with the training:

• 92% of participants in frontline training described the program as “good”, “very good”, or “excellent”.

• Employers were satisfied with the quality of the CSR and leadership training with most employers reporting they would recommend CSR and leadership training to new staff (90%) or other businesses (90%). Many also reported that the training improved employees’ customer engagement, listening ability, sales strategies, and attitude on the job.

PROCESS EVALUATION RESULTS: SUCCESSES AND CHALLENGES

What worked well?

• CSR trainers reported that the curriculum was relevant, useful, and aligned with the needs and interests of employees in the hospitality and tourism and retail sectors.

• Many employers perceived the program’s blend of essential skills and sector-specific corporate training to be well-aligned with their business objectives. They also felt that the leadership training offered to supervisors was extremely valuable.

• Trainers reported that the co-delivery approach involving LBS providers and OTEC trainers was a very positive experience for trainers. It facilitated capacity building and added value for participants.

• Adopting a coaching-based approach—with skills assessment scores discussed with participations and used as a way to motivate participants to

grow and learn—was found to be effective in increasing participant focus and confidence.

What challenges were encountered?

• Coordinating training proved difficult. Scheduling training for cohorts with employees from multiple small and medium-sized businesses was particularly challenging, as training had to be aligned with multiple schedules and held at mutually convenient locations.

• Diverse cohorts presented an additional challenge. Trainers reported that it was difficult to ensure that training content was relevant to participants from different types of business and with different skill levels.

• Integrating the essential skills assessments into the training proved demanding given time constraints and the logistics of administering the assessment. These challenges may have affected the motivation and effort of participants.

• Contextual factors such as seasonal business cycles, policy changes related to the minimum wage and guaranteed time off for employees, and local labour market pressures also presented challenges. For example, many employers in the target sectors noted that it is difficult to invest in employee training during busy summer months.

LESSONS FOR WORKFORCE DEVELOPMENT ORGANIZATIONSThis project highlighted a number of lessons that could directly inform the future efforts of workforce development organizations interested in implementing workplace contextualized essential skills training in Ontario:

• There is a need for an employer recruitment strategy tailored to both the industry sector and more broadly the way in which small to


iv

medium-sized businesses in the sector operate. OTEC noted that face-to-face engagement is often most effective in working with employers to see the business case for this type of training. Other effective strategies included leveraging existing relationships with employers and within networks of employers.

• There is a need for training to be designed and delivered with a clear target audience in mind to ensure it is closely aligned with participant needs and skill levels.

• Sector-based essential skills projects must consider the impact that policy, seasonal, and regional factors could have on the overall success of the project, and particularly on the ability to recruit employers and employees.

BROADER LEARNINGS

This project generated some important lessons about the broader opportunities presented by workplace essential skills training models. These findings are relevant to decision-makers, employers, and service providers interested in addressing the skills gaps within Ontario’s workforce, increasing job retention, and improving workforce productivity and resilience:

• Contextualized essential skills training programs align well with the business needs and goals of particular sectors. This is helpful to employers in understanding exactly how the training will benefit their businesses. This point was reinforced by participating employers, who reported that the CSR training was useful, valuable, and relevant to their business needs.

• Training programs like CSR offer a standardized product, in contrast to the more individualized approach to essential skills training adopted by other organizations, including LBS providers. This standardization allows employers to ensure

employee training is consistent—which is key to building and maintaining service quality. The standardized curriculum also creates efficiencies through the participation of multiple employees in a single training cohort.

• Sector-focused approaches to workplace essential skills training can combine the expertise of multiple partners including essential skills trainers, industry organizations, and employers. This lays the groundwork for collaborative efforts to address industry skills needs and contribute to a more skilled and resilient labour force.

This project also generated a series of valuable learnings on factors that could support the success and enable the scaling of workplace contextualized essential skills training programs in Ontario:

• More resources for coordination - Coordinating training for small to medium-sized employers is particularly challenging, and more support for training coordination is required if training is to be accessible to these employers. OTEC and partner organizations are actively working to build new regional training consortiums in the hospitality and tourism sectors to help address this challenge.

• Strategic partnerships – Opportunities to build stronger partnerships between industry and training organizations are important for ensuring that training aligns with industry needs, meets business objectives, and adds value for both participants and employers.

• Funding envelopes - The Canada-Ontario Job Grant is a promising avenue for funding workplace contextualized essential skills training projects, but training offerings need to align with Job Grant requirements and guidelines (e.g. length of training).

Table of Contents

Executive summary .......................................................................................................................... i

Introduction ....................................................................................................................................... 1

Report purpose............................................................................................................................................ 1

Background ................................................................................................................................................... 1

Evaluation approach .................................................................................................................................. 2

In this report ................................................................................................................................................. 2

1. About the project ......................................................................................................................... 3

Rationale ........................................................................................................................................................ 3

About the Workplace Training Program .............................................................................................. 4

Project partners ........................................................................................................................................... 6

Key features .................................................................................................................................................. 7

Design ............................................................................................................................................................. 7

Incentive structure ...................................................................................................................................... 7

Outcomes evaluation design .................................................................................................................. 10

Participant outcomes ................................................................................................................................. 8

Employer outcomes ................................................................................................................................... 8

Leadership ..................................................................................................................................................... 10

Continued monitoring ............................................................................................................................... 10

Process evaluation design ........................................................................................................................ 11

2. Implementing the model ........................................................................................................... 12

3. Results for participants ............................................................................................................... 16

Who was reached? ..................................................................................................................................... 18

Training expectations ................................................................................................................................. 19

Satisfaction with training .......................................................................................................................... 21

Short term outcomes ................................................................................................................................. 22

Essential skills ............................................................................................................................................... 22

Other short-term outcomes .................................................................................................................... 26

Longer term outcomes ............................................................................................................................. 26

Leadership training participants ............................................................................................................. 28

4. Results for employers ................................................................................................................. 30

Employer characteristics ........................................................................................................................... 31

Satisfaction .................................................................................................................................................... 32

Utility ............................................................................................................................................................... 34

Perceived impacts ....................................................................................................................................... 35

Employer case studies ............................................................................................................................... 36

5. Process evaluation findings ....................................................................................................... 39

What worked well ....................................................................................................................................... 39

Challenges ..................................................................................................................................................... 40

Context ........................................................................................................................................................... 41

6. Key takeaways .............................................................................................................................. 42

Results ............................................................................................................................................................. 42

Lessons for workforce development organizations ......................................................................... 42

Broader learnings ........................................................................................................................................ 43

Appendix A: Comparing CSR and the Workplace Training Program ..................................45

Appendix B: Discussion of Learnings .......................................................................................... 46

References .......................................................................................................................................... 49

Introduction

Report purposeThis report presents findings from the Contextualized Essential Skills Demonstration Project. The project tested the feasibility and effectiveness of an innovative approach to workplace essential skills training for frontline workers in the hospitality and retail industries.

BackgroundIn December 2017, as part of their mission to drive innovation and generate new evidence about the effectiveness of employment and training services, the Ontario Centre for Workforce Innovation (OCWI) engaged Blueprint to lead a demonstration project on contextualized essential skills training. Research has identified contextualized essential skills training as an evidence-informed strategy for improving the skills and labour market outcomes of individuals with skills gaps.

Recognizing that a significant proportion of Ontario’s workforce has essential skills gaps, Blueprint opted to test an innovative workplace training model that focused on upgrading essential skills for frontline workers in the hospitality and retail sectors. The demonstration was based on the Workplace Training Program—a model that aimed to increase the skills and productivity of frontline hospitality and retail workers through sector-specific essential skills training and that showed promising results in British Columbia. The curriculum from the Workplace Training Program was adapted and updated for the Ontario context. The resulting Ontario-branded program was called Customer Service Results (CSR).


2

Evaluation approachBlueprint used a two-pronged evaluation strategy to understand the feasibility and effectiveness of the CSR program in the Ontario context:

1. Outcomes evaluation – We assessed outcomes for participants (including essential skills gains, job performance, and career satisfaction) and employers (including relevance and impact of training)

2. Process evaluation – We identified success factors, challenges, and opportunities for strengthening the model, and also explored opportunities for scaling and sustaining the model.

In this reportThis report has six sections:

1. About the project - Provides an overview of the project and the evaluation design

2. Implementing the model - Describes how the model was implemented

3. Results for participants - Presents an overview of participant results

4. Results for employers - Presents an overview of employer results

5. Process evaluation findings - Outlines the process evaluation findings including context, success factors, and challenges

6. Key takeaways - Provides key takeaways, lessons learned, and broader implications

1 | About the project

RationaleA significant proportion of Ontario’s workforce has essential skills gaps. Policymakers are increasingly concerned about the need to address these gaps to create a more productive, resilient workforce. Previous research found a strong relationship between essential skills and earnings and demonstrated that individuals with skill gaps may need help to upgrade their skills in order to achieve labour market success (Hampf & Widerhold, 2016).

While many Ontario employees with skills gaps have likely benefited from the increase in minimum wage, it is also possible that employers may now expect more of them in terms of productivity, and individuals with lower skills may struggle to keep up in this environment.

At the same time, many hospitality and retail employers are struggling to maintain profit margins. In an increasingly competitive

marketplace, businesses face pressure to compete on price and may be reluctant to invest in their employees through training. This creates a ‘vicious cycle’, as growing evidence exists that consistent underinvestment in employees leads to poor performance, high employee turnover, and lost profits—ultimately resulting in even fewer training opportunities (Ton, 2012).

The Contextualized Essential Skills Demonstration Project was implemented to test a potential solution to these challenges. The project tested the feasibility and effectiveness of a workplace training model that focuses on upgrading essential skills for entry-level hospitality and retail workers in Ontario. The project was designed to add value for both employees and employers by increasing the skills and productivity of frontline workers. The model also aligns with the Ministry of Training, Colleges and Universities’ broader skills agenda, including the focus on sectoral approaches.


4

Leadership training for the supervisors and managers of participating businesses was also included, based on previous findings that a supportive supervisor can reinforce the positive effects of workplace essential skills training. The leadership training also acted as an added incentive for employers to participate in the training.

About the Workplace Training ProgramThe Contextualized Essential Skills Demonstration Project is based on the Workplace Training Program, an innovative contextualized essential skills program implemented in British Columbia. The program was funded by the BC Government and designed to provide training opportunities to lower-skilled individuals working in the hospitality and retail sectors. The program curriculum integrated essential skills into training programs designed for specific clusters of frontline service occupations.

The Workplace Training Program produced positive results for both employees and employers. In addition to improvements in on-the-job performance, participants experienced modest gains in essential skills—including document use and numeracy. Employers perceived the training to be relevant and valuable, and most reported that it improved the skills and productivity of their staff.

Box 1 outlines additional key findings from the evaluation of the Workplace Training Program.

BOX 1 | RESULTS FROM THE WORKPLACE TRAINING PROGRAM

Results for participants

Participants were generally satisfied with the training. Almost all participants (99%) rated the program as “good”, “very good”, or “excellent”. Participants experienced average improvements of 19 points in document use skills and 17 points in numeracy skills. Participants also showed gains in their motivation and engagement at work, and their attitude towards learning. One month after training, 68% of participants reported improvements in job performance in all four performance areas.

Results for employers

Employers were satisfied with the quality of the Workplace Training Program. 88% of employers communicated that they would recommend the training to other employers and 81% said that the training had enhanced employee performance.


5

Given its success in British Columbia, Blueprint identified the Workplace Training Program as a promising model that could be replicated in Ontario. The curriculum from the Workplace Training Program was adapted and updated for the Ontario context, with the new Ontario program named Customer Service Results (CSR). See Appendix A for a summary of the key differences between the CSR program and the Workplace Training Program.

Sustainability was a key goal of the CSR model design and so LBS providers were engaged as partners to explore the broader applications of the model in the province’s essential skills training system. In addition, we used an employer-focused recruitment approach to test whether the model could be sustained in the future through the Canada-Ontario Job Grant which offers matching funds for employers who invest in training for employees (see box 2).

BOX 2 | CANADA-ONTARIO JOB GRANT

The Canada-Ontario Job Grant provides funding for employers or consortia of employers who want to purchase training for existing or new employees. The funding is provided by the federal government but administered by the Government of Ontario, with employer contributions ranging from 15%-50% of the training costs depending on business size.


6

Project partnersBlueprint engaged several partners to contribute to the development and execution of the Contextualized Essential Skills Demonstration

Project. The role of each project partner is outlined in Table 1.

TABLE 1 | PROJECT PARTNER ROLES

Project partner (Partner description) Role

OTECA workforce development organization, specializing in tourism, hospitality, and retail

• Marketed the project and recruited partners in collaboration with employer associations

• Coordinated training• Delivered training to frontline employees• Engaged LBS network organizations and providers to

identify co-delivery partners• Delivered leadership training• Collaborated with Blueprint to collect data

Literacy and Basic Skills (LBS) providersProviders with expertise in literacy and basic skills training for adults

• Co-delivered CSR training with OTEC

Literacy and Basic Skills (LBS) network organizationsSupport organizations that provide planning and support for LBS providers in a given region

• Supported employer recruitment• Supported the engagement and selection of

LBS providers

Douglas CollegePost-secondary institution that led the Workplace Training Program

• Adapted and ‘refreshed’ the Workplace Training Program curriculum

• Delivered ‘train-the-trainer’ sessions to OTEC and LBS providers

Employer AssociationsOntario Restaurant Hotel and Motel Association, Restaurants Canada, Ontario Chamber of Commerce, regional tourism organizations, and business improvement areas

• Supported marketing and recruitment


7

Key features

DESIGN

The CSR curriculum contained a series of training modules that integrated technical sector-specific content with essential skills training. The curriculum included tailored training modules for food & beverage servers and retail sales associates.

The project also included a leadership training program called Leadership Xelerator that offered two full days of training for supervisors and/

or managers in the hospitality and retail sectors. The Leadership Xelerator curriculum, designed by OTEC, focused on developing the communication, team building, and coaching skills of supervisors. Leadership training participants also received instruction on the goals and objectives of the CSR frontline training program to enable them to reinforce key messages on the job.

Table 2 outlines the training modules included in the curriculum.

TABLE 2 | PROGRAM TRAINING MODULES

FOOD & BEVERAGE RETAIL ASSOCIATE LEADERSHIP XELERATOR

Introduction to training and essential skills

Introduction to training and essential skills

Communication and EQ for leadership

Business and personal success Business and personal success Learning to lead

Effective communication Effective communication Team building and trainer excellence

Product knowledge Product knowledge Coaching for excellence

Successful sales strategies Customer service

Productivity Successful sales

Delivering “wow” guest service Productivity

The initial CSR training sessions were delivered by OTEC, with LBS providers engaged to co-deliver training mid-way through the project. The initial target locations for training were Kitchener-Waterloo, Fort Frances/Kenora, and Ottawa. The training was expanded to the Greater Toronto Area, London, Barrie, and Mono due to employer interest.

The frontline training curriculum was generally delivered over three full-day sessions. Leadership training consisted of two full-day sessions.

Most training cohorts were made up of employees from multiple participating businesses. Three training cohorts consisted of employees from a single business.

INCENTIVE STRUCTURE

Employees were paid wages by their employer for the time spent participating in the training. Employees also received $30 each time they completed a skills assessment or survey in support of the evaluation. See Appendix A for a discussion of the differences in the incentive structures used for CSR and the Workplace Training Program.


8

Outcomes evaluation designThe outcomes evaluation focuses on results the CSR program delivered for frontline training participants, employers, and leadership training participants.

FRONTLINE PARTICIPANT OUTCOMES

CSR training participants were asked to complete:

• Surveys (on the first and last day of training, and at follow-up)

To assess participant characteristics and collect information on measures of labour market

outcomes, personal development, job-related well-being, and career satisfaction

• Skills assessments (on the first and last day of training)

A customized version of an online essential skills test to measure gains in numeracy and document use (see Box 3 for additional details)

Figure 1 (below) shows the participant outcomes evaluation timeline.

FIGURE 1 | PARTICIPANT OUTCOMES EVALUATION TIMELINE

4

4


FIGURE 2 | EMPLOYER OUTCOMES EVALUATION TIMELINE

CSR Training

Period

Baselinesurvey

Exitsurvey

Follow-upsurvey1 3 5

FIRST DAYOF TRAINING

LAST DAYOF TRAINING

Exit skillsassessment

2

CSRTraining

Baselinesurvey

Follow-upsurvey1 3

Baselineinterview

LeadershipTraining

Follow-upinterview

Month following training


2Baseline skillsassessment

ONE MONTHAFTER TRAINING

EMPLOYER OUTCOMES

All employers were asked to complete surveys—during the initial engagement and one month after training completion—that captured information on their training expectations, satisfaction, and perceived changes in the performance of employees. We also conducted case studies with a

small sample of employers. These employers were asked to participate in interviews that explored their business needs and their experience with the training in more depth.

Figure 2 (below) shows the outcomes evaluation timeline for employers.


4

4



CSR Training

Period

Baselinesurvey

Exitsurvey



LAST DAYOF TRAINING

Exit skillsassessment

2

CSRTraining

Baselinesurvey

Follow-upsurvey1 3

Baselineinterview

LeadershipTraining

Follow-upinterview



2Baseline skillsassessment



9

BOX 3 | ESSENTIAL SKILLS ASSESMENT

We used a customized Essential Skills Group (ESG) Essentials Skills Test to test the essential skills of CSR participants. This version was an abbreviated version of the standardized ESG Essential Skills Test. It contained questions that were contextualized to participant occupation. The test was administered through an online portal and was self-adaptive, with participants receiving harder or easier questions based on their ongoing test performance. The test assessed skills in numeracy and document use as outlined below:

DIMENSION DOMAIN WHAT WAS TESTEDNUMBER

OF QUESTIONS

Document Use

Document Use

The ability to locate and use information embedded in lists, tables, and visual displays 18

Numeracy Whole Numbers

The ability to solve problems using whole numbers 5

Decimals The ability to solve problems using decimals 5

Fractions The ability to solve problems using fractions 5

Per cents The ability to solve problems using per cents 5

Formulas and equations

The ability to solve problems using common formulas and equations 5

Tests scores were reported as both a raw score (1 to 500) and as an Essential Skill Level (1 to 5). The levels corresponded with commonly used essential skills levels and reflected a rough grouping of essential skills as follows:

Level 1 – 2: Basic Skills

Level 3 – 4: Generally Acceptable Skills

Level 5: Highly Advanced Skills

In general, Level 3 is considered the minimum desired level for safe and productive work in a knowledge-intensive society.


10

FIGURE 3 | LEADERSHIP PARTICIPANTS OUTCOMES EVALUATION TIMELINE

LeadershipTraining Period

Baselinesurvey

Exitsurvey



LAST DAYOF TRAINING



LEADERSHIP TRAINING PARTICIPANT OUTCOMES

Leadership training participants were asked to complete surveys on the first and last day of training and one month following training completion (see Figure 3, below). The surveys assessed motivation, satisfaction with training, and changes in job performance.

Figure 3 (below) shows the outcomes evaluation timeline for leadership training participants.

FIGURE 3 | LEADERSHIP PARTICIPANTS OUTCOMES EVALUATION TIMELINE

CONTINUED MONITORING

Blueprint continuously monitored and analyzed the quality, completeness, and results of data from each cohort throughout the project. This approach allowed us to adapt data collection processes in order to ensure data quality and inform course corrections to enhance the program’s impact. For example, after experiencing low response rates in

early cohorts, we incorporated survey reminders by phone and email to increase the response rates to follow-up surveys. Similarly, ongoing analysis of survey and skills assessment results allowed us to provide real-time updates to the trainers and the training provider (OTEC) that helped them stay engaged with the project and project participants.


11

Process evaluation designTo complement the outcomes evaluation, we also conducted a process evaluation that focused on understanding the context, success factors, and challenges associated with the implementation of the CSR program. The process evaluation drew on several sources of data:

• Ongoing conversations with OTEC staff—including mid- and post-project debriefs—to understand the successes and challenges in recruiting, and the coordination and delivery of training

• Data collected by OTEC during their outreach to businesses in order to recruit training participants

• Short reflection surveys completed by OTEC trainers following the completion of each training cohort which documented perceived impacts and challenges, as well as any adaptations to training

• A short survey completed by employers who showed an interest in the CSR training but chose not to take part to allow us to better understand some of the barriers to participation

• Interviews with participating LBS trainers to assess their experiences in co-delivering the training

• Interviews with representatives from LBS network organizations to explore their perceptions of the CSR program and potential opportunities for alignment with the LBS mandate

In addition to conducting interviews with participating LBS trainers and network organization partners, we also held a learning session with representatives from LBS network organizations across Ontario on February 5, 2019. At the session, we presented key learnings from the project and engaged participants to reflect on potential next steps for implementing workplace essential skills training models in Ontario.

Appendix B provides a summary of the key themes that emerged from this discussion.

2 | Implementing the model

This section provides an overview of the implementation of the CSR program.

Figure 4 (below) gives an overview of the implementation timeline.

FIGURE 4 | IMPLEMENTATION TIMELINE

NOV DEC JAN FEB MAR APRIL MAY JUNEProjectplanning

Curriculumadaptation

Employerengagementrecruitment

JULY AUG

“Train-the-trainer”sessionsProjectlaunchOngoingdelivery

LBS co-delivery

LBSsymposium

Timeline of CSR implementation

SEPT OCT NOV DEC JAN

2017 2018 2019FEB


13

December 2017

Blueprint proposes the Workplace Training Program as a promising contextualized essential skills training model for replication in the Ontario context, and identifies OTEC as delivery lead based on their expertise in designing and delivering training to employees in the Ontario hospitality and retail industries.

Blueprint and OTEC meet with Douglas College to develop a strategy for replicating the Workplace Training Program in Ontario. It is agreed that Douglas College will update the Workplace Training Program curriculum and deliver “train-the-trainer” sessions to partners in Ontario.

January-February 2018

OTEC begins initial discussions with employers and LBS network organizations across the province to gauge interest and identify potential sites for the demonstration. Based on these discussions—as well as considerations related to regional balance—Kitchener-Waterloo, Ottawa, and Kenora-Fort Frances are chosen as target sites for the initial launch.

March 2018

Douglas College leads a four-day “train-the-trainer” session to familiarize OTEC trainers with the curriculum and training materials, and offer tips and strategies for effectively delivering the training. Blueprint also provides trainers with an orientation on the data collection tools and processes needed to support the project evaluation.

May-June 2018

OTEC intensifies recruitment efforts, incorporating outreach tactics such as e-blasts, newsletters, project announcements through social media, and calls to businesses.

An initial “pilot-within-a-pilot” is launched in Toronto. This first cohort presents an opportunity to live test the delivery of the CSR curriculum, leadership training, and evaluation processes.

July-August 2018

OTEC continues their efforts in recruitment, training coordination, and delivery.

In response to lower than anticipated enrollments, OTEC begins exploring interest in other markets including the Greater Toronto Area and Thunder Bay. After finding that small to medium-sized businesses are unable to commit multiple employees for training, OTEC expands recruitment to include larger employers.

OTEC begins the process of engaging LBS providers to co-deliver training. An expression of interest is sent to LBS providers, coordinated through LBS network organizations. A number of providers express interest in co-delivery:

• Ottawa - Centre Moi J’Apprends; People, Words, & Change; and the Rideau-Ottawa Valley Learning Network

• GTA – Peel-Halton-Dufferin Adult Learning Network; and the Peel Adult Learning Centre

• Kitchener-Waterloo – Waterloo Regional District School Board – Essential Skills Upgrading; and the Literacy Group of Waterloo

• Kenora/Fort-Frances - Valley Adult Learning Association

August 2018

OTEC and Douglas College lead a four-day “train-the-trainer” session for trainers from interested LBS providers. Similar to the initial “train-the-trainer” session, this session familiarizes LBS providers with the curriculum and strategies for training delivery. Some OTEC trainers participate in the session to refresh their training on the CSR curriculum. Representatives from LBS network organizations


14

also attend the first day of the session to learn more about the project and discuss its relevance to the LBS mandate.

In response to limited uptake in Kenora-Fort Frances, OTEC changes strategies and leverages the connections of a local Fort Frances LBS provider—the Valley Adult Learning Association (VALA). VALA assists OTEC in recruitment in the area and helps to coordinate training.

September 2018

Three more CSR cohorts are launched in Ottawa, Toronto, and Fort Frances. These cohorts are the first to be co-facilitated by OTEC and LBS trainers.

October 2018

A second cohort is launched in Ottawa. OTEC expands recruitment efforts to include London, where businesses have expressed interest. The first London cohort is launched.

Blueprint and OTEC hold a mid-project check-in to assess early results and identify opportunities for improvement and course corrections, including:

• Recruitment strategy –marketing the project directly to employees, increasing interest, and clearly communicating the value of the project

• Emphasizing essential skills – engaging Douglas College to offer follow-up tips and strategies on coaching, and motivating participants to increase their essential skills—including sharing and discussing baseline scores with the participants

November 2018 – December 2018

Three more hospitality cohorts launch in Toronto, Mono, and Kenora/Fort Frances.

The first retail cohort launches in Toronto.

January 2019

The final CSR cohort is launched and completed in Toronto. This cohort is the first and only to receive the training in four, six-hour sessions. This brings the total number of participants that were enrolled in CSR training to 159.

February 2019

Blueprint organizes a symposium with LBS training providers. The event gives participants the chance to discuss their experiences coordinating support for the CSR program, and the broader implications of the project for the skills training ecosystem in Ontario (see Appendix B for a summary).

Figure 5 (below) shows the timing of the CSR and leadership cohorts by location.


15

FIGURE 5 | TIMELINE OF CSR AND LEADERSHIP COHORTSFIGURE 5 | TIMELINE OF CSR AND LEADERSHIP COHORTS

APR MAY JUNE JULY AUG SEPT OCT NOVGreater Toronto Area

Kitchener

Ottawa

11 8

14 13

13 10

DEC JAN

Fort Frances

London

Kenora

Mono

9 8

15 14

7 9

7 6

5 5

6 6

5 6

11 9

11 9

10 8

14 13

7 7

19 19

Program

CSR LeadershipSummary

Sites 8 4

Cohorts 14 5

Entered training 159 45

Completed training 146 40

number of participants starting

number of participants completing

CSR cohort diagrams

Leadership training employees

10 10

6 6

21 20Barrie

3 | Results for participants

CSR training participantsThis section outlines the results for participants who enrolled in and completed the CSR (frontline) training. Of the 159 individuals who enrolled, 146 participants (92%) completed the full 24-hour training.

Our evaluation was guided by the project logic model and informed by the logic model and evaluation framework used in the Workplace Training Program, as well as additional input from OTEC.

Based on the logic model, we expected to see pre- to post-training changes for participants in four areas:

• Essential skills – Training in essential skills should have an immediate impact on the skill levels of participants.

• Job motivation and engagement – Training was designed to increase job motivation and

engagement. Participants in the Workplace Training Program experienced immediate and statistically significant gains in this area.

• Attitudes towards learning – The curriculum was designed to increase participant interest in and engagement with learning. Participants in the Workplace Training Program experienced statistically significant positive gains in attitude towards learning between baseline and exit.

• Industry attachment – Taking part in a sector-based training program was hypothesized to make participants feel more confident in their occupations and perceive that investments were being made in their careers. This should strengthen attachment to the sector.

With the exception of essential skills, these short-term outcomes were also measured at follow-up, but the data was not of sufficient quality to be usable in our analysis.


17

We also analyzed longer-term outcomes in four areas one month after training: locus of control, future orientation, compensation, and perceived impact on job performance.

Table 3 (below) outlines the points at which each outcome was measured and how many participants completed the survey or assessment.

TABLE 3 | OUTCOMES AND DATA COLLECTION POINTS

DATA COLLECTION POINTS

OUTCOMES BASELINE EXIT1 MONTH

FOLLOW UP #COMPLETE

Short term outcomesEssential skills

132

Job motivationAttitudes towards learningIndustry attachment

1145

Longer term outcomesLocus of controlFuture orientationCompensationPerceived job performance

62

1 Short-term outcomes were also collected at follow-up, but the data was not of sufficient quality to be usable.


18

Who was reached?The 157 participants who completed intake surveys ranged in age from 17 to 73 years, with a mean age of 33. About half (52%) were under the age of 30 and about a third (30%) were under the age of 25 (see Figure 6, below). About half (56%) were post-secondary graduates, 39% had a high school

education, and 5% had less than a high school education. Participants were more likely to be female (55%) than male (45%), and more likely to be Canadian born (69%) than born outside of the country (31%). Most participants (84%) were native English speakers (see Table 4, below).

FIGURE 6 | AGE RANGE OF PARTICIPANTS

FIGURE 6 | AGE RANGE OF PARTICIPANTS

30%

0%

10%

20%

40%

17-24

29%

25-29 30-44 45-59 60+

22%

31%

16%

2%

Age group

% o

f par

ticip

ants

TABLE 4 | DEMOGRAPHIC CHARACTERISTICS OF PARTICIPANTS

CHARACTERISTIC PERCENTAGE OF PARTICIPANTS

GenderFemale

Male

55%

45%

Highest educationDid not graduate high schoolHigh school diplomaPost-secondary graduate

5%39%56%

Country of birthCanadaOther

69%31%

Native languageEnglishOther

84%16%


19

Training expectations Most participants had multiple motivations for participating in training. The most common motivation (54%) was their employer asking them to participate in the training. But many participants also had internal motivations. Participants commonly cited a desire to do their job better (52%), improve

their communication (42%), and improve their problem solving (37%), as reasons for taking the training. Participants also took the training as a means to advance their careers (35%) and to further their education (28%) (see Figure 7, below).

FIGURE 7 | REASONS FOR TAKING TRAINING


Because my employer asked

25% 75%0% 50% 100%

54%

52%

42%

37%

35%

28%

5%

To do my job better

To improve my communication

To improve my problem solving

To advance my career

To further my education

Other reason not listed

% of participants selecting

Participants overwhelmingly agreed that they were motivated to do the best they could in the course (91%), that their supervisors supported them in taking the course (93%), and that they were looking forward to taking the training (80%). Participants anticipated that the course would be

useful and most agreed or strongly agreed that the training topics were highly relevant to their jobs (81%), and that the training would help them further their career (79%). Most participants were not nervous about taking the training, with only 16% reporting being anxious (see Figure 8, below).


20

FIGURE 8 | PRE-PROGRAM MOTIVATION

FIGURE 9 | PRE-PROGRAM MOTIVATION

How much do you agree with these statements?

This training will help meget further in my career

Disagree or strong disagree Neutral Agree or strongly agree

25% 75%0% 50% 100%

The topics that will be coveredin the training are highly relevant

to my job tasks

My supervisor supportsme in taking this course

79%

81%

93%

64%I am nervous abouttaking this course

20%

I am motivated to do thebest I can in this course 91%

7%

16%

9%

19%

I am looking forward totaking this training 80%17%

16%

3%

3%

2%



21

Satisfaction with trainingParticipants who completed an exit survey (145) were generally satisfied with training. Almost all participants (92%) rated the overall training as good (18%), very good (31%), or excellent (43%).

A large majority of participants were extremely or somewhat satisfied with all of the components of training, including the instructor (91%), the location (84%), the content (81%), and the schedule (77%) (see Table 5, below).

TABLE 5 | SATISFACTION WITH TRAINING COMPONENTS

HOW SATISFIED ARE YOU WITH EACH OF THE FOLLOWING?

EXTREMELY OR SOMEWHAT SATISFIED NEUTRAL

EXTREMELY OR SOMEWHAT DISSATISFIED

The location of the training 84% 13% 3%

The scheduling of the training

77% 13% 9%

The instructor 91% 6% 3%

The training content 81% 10% 9%

In addition to being satisfied with the training, participants reported that the training was relevant, useful, and motivating. The majority of participants agreed or strongly agreed that

the topics covered were highly relevant to their job tasks (82%) and that they would have opportunities in their job to apply what they had learned (87%).


22

Short-term outcomesWe analyzed essential skills outcomes for the 132 participants who completed the training as well as the assessments at program baseline and exit. We also analyzed attitude towards learning, job motivation and engagement, and industry attachment for the 145 participants who completed the baseline and exit surveys.

ESSENTIAL SKILLS

Overall skills gains

Essential skills were measured using an assessment completed by participants before and after receiving the CSR training. The assessment tool was a customized version of the Essential Skills Group (ESG) Essential Skills Test which included questions focused on the hospitality and retail sectors.

58% of participants showed an increase in their numeracy score from baseline to exit, with an average increase of 13.5 points—a statistically significant (p <0.001) change. Just over half (51%) of participants achieved improved scores for document use from baseline to exit. However, at

the group level, the overall change (+0.5%) was not statistically significant (see Table 6, below).

While the training accounted for a share of the difference between pre- and post-curriculum test scores, there were also additional factors that could have affected the results. It is possible that individuals may have reacted to the assessment in different ways at baseline and exit. For example, some participants may have become disengaged over the course of the training and so put less effort into completing the exit assessment. Conversely, participants who were more engaged by the curriculum may have made a more concerted effort to do well in the assessment at exit than baseline.

It is unlikely that the overall results are driven entirely by these testing effects, given the large sample size. There is also no reason that participants as a whole should be systematically more motivated at exit than at entry. However, findings for specific sub-groups of participants should be treated with more caution due to the smaller sample sizes.

2 Statistical significance to 95% (p = < 0.05) is denoted by an asterisk (*). Significance to 99% (p = < 0.01) is marked by two asterisks (**).

TABLE 6 | ESSENTIAL SKILLS GAINS

ESSENTIAL SKILLS MODULE

AVG. BASELINE SCORE AVG. EXIT SCORE AVERAGE CHANGE

% OF PARTICIPANTS WITH SCORE

IMPROVEMENT

Document Use 270.1 269.6 -0.5 51%

Numeracy 240.2 259.1 + 13.5** 58%


23

Skills gains by starting level

The evaluation of the Workplace Training Program found that learners who entered the program with lower essential skills levels experienced the largest improvements. To explore this relationship in the context of CSR, we analyzed skills gains across different skills levels at baseline.

The extent to which document use and numeracy changed during training was associated with the learner’s skill level at baseline. Overall, learners who had lower skills levels entering the training showed gains in both essential skills (document use and numeracy), whereas participants who had higher starting skills levels showed decreases in essential skills scores (see Figure 9).

FIGURE 9 | AVERAGE SKILLS CHANGE BY STARTING LEVEL

FIGURE 10 | AVERAGE SKILLS CHANGE BY STARTING LEVEL

0 25-25-50 0 25 50-25-50

Document Use Numeracy

Parti

cipan

t sta

rting

skill

leve

l

Change in skill (+/- points)

12.3

-13.3

-11.8

-75-100 -75

3.1

29.4

-73.5

-27.6

23.6

LEVEL 5

LEVEL 4

LEVEL 3

LEVEL 2

LEVEL 1

-78.3

20.4

TABLE 7 | AVERAGE SKILLS CHANGE BY STARTING LEVEL

DOCUMENT USE NUMERACY

Starting Level # of learners

Average change

% of learners with score improvement # of learners

Average change

% of learners with score improvement

1 24 20.4* 71% 54 29.4 67%

2 47 12.3 55% 42 3.1 52%

3 44 -11.8 41% 21 23.6 62%

4 11 -13.3 45% 7 -27.6 43%

5 6 -78.3 17% 4 -73.5 0%


24

More specifically, participants who started at document use skills levels of 1 or 2 showed gains of 20.4 and 12.3 points respectively, whereas learners who had starting document use skills levels of 3, 4, or 5 showed losses of 11.8, 13.3, and 78.3 points respectively. Document use gains for Level 1 learners were statistically significant (p < 0.05) (see Table 7).

Similarly, learners who had starting numeracy levels of 1, 2, or 3 showed an average gain of 29.4, 3.1, and 23.6 points respectively, whereas learners who had baseline numeracy levels of 4 or 5 showed an average loss of 27.6 and 73.5 points respectively.

A similar trend was observed in the Workplace Training Program where participants with starting essential skills levels of 1 or 2 saw overall gains in their scores at exit, and participants with essential skills scores of levels 3 or higher saw overall losses at exit. This suggests that training may be most effective for participants with lower levels of essential skills, especially Level 1 learners. However, these results should be interpreted with caution given the relatively small number of participants in each sub-group.

Skills gains by cohort timing

In response to early findings showing minimal changes in essential skills, OTEC held a mid-project check-in where an expert trainer from Douglas College provided trainers with a refresher session on the training curriculum. The refresher focused on the importance of the essential skills assessments and encouraged trainers to work with participants using a coaching-based approach to build participant motivation and emphasize to them the importance of increasing their essential skills. As part of the check-in, trainers committed to sharing and discussing essential skills assessment scores with participants going forward.

Learners who participated in the program after this debrief showed larger gains in document use (+2.1 points) and numeracy (+14.6 points) than overall averages (-0.5 points and +13.5 points respectively) suggesting that this process may have contributed to more effective facilitation from trainers and/or improving the motivation of learners (see Table 8, below).



Cohort characteristic# of

learnersAverage change

% of learners with score

improvement# of



improvement

All cohorts 132 -0.5 51% 128 13.5 58%

Post mid-project debrief 84 2.8 56% 82 17.1 65%


25

Skills gains by participant characteristics

Changes in essential skills also varied by participant characteristics. Participants who were female, youth (age 16-24), or had some post-secondary education had the largest average changes in essential skills (see Figure 10 and Table

9, below). Although participants born outside of Canada, non-native English speakers, and those without a high school degree all experienced large changes, the sample sizes for each are too small to draw meaningful conclusions. Overall, none of the changes for each subgroup of participants were statistically significant.

FIGURE 10 | AVERAGE SKILLS CHANGE BY PARTICIPANT CHARACTERISTICS

FIGURE 11: | AVERAGE SKILLS CHANGE BY PARTICIPANT CHARACTERISTICS

0 25 50-25-50 0 25 50-25-50

Document Use Numeracy

PSE graduate

Youth

Female

Born outsideof Canada

Non-nativeEnglish speaker

No highschool degree

Parti

cipan

t cha

ract

erist

ics

Change in skill (+/- points)

27

11.8

15.3

0.0

0.08

5.6

30.27

45.67

-20.71

0.0

0.1

-46.7



Characteristic# of



improvement# of



improvement

PSE graduate 74 -0.0 50% 71 27.0 68%

Youth 41 -.8 51% 40 11.8 55%

Female 69 5.6 57% 67 15.3 57%

Born outside of Canada 13 -0.0 51% 11 -30.3 65%

Non-native English speaker 23 0.1 52% 21 45.7 71%

No high school degree 7 -46.7 14% 4 -20.7 14%


26

OTHER SHORT-TERM OUTCOMES

In addition to essential skills changes, we measured changes in attitude towards learning, job motivation and engagement, and industry attachment. Overall, these outcomes were largely unaffected by the CSR training:

• Attitude towards learning – Attitude towards learning refers to how positively individuals view the act of learning and the benefits learning can deliver. We measured attitude to learning using three questions used to evaluate Skills for Life—a longstanding initiative of the UK government aimed at improving essential skills. Each question was scored on a 5-point Likert scale. Although more participants showed positive changes than negative changes, these changes were not statistically significant.

• Job motivation and engagement - Job motivation and engagement’ refers to the extent to which individuals find the work they do engaging and are motivated to do their best in their job. We measured this outcome with questions drawn from the Motivation and Engagement in Work Scale. Each question was scored on 5-point Likert scale. There was no change in this outcome.

• Industry attachment - Industry attachment refers to the extent to which individuals are satisfied with the career and the industry they are employed in. We measured this outcome with a three-question scale adapted from the Career Activation Scale. Each question was scored on a 5-point Likert scale. There was no change in this outcome.

While we did expect to see some changes in these outcomes, the lack of improvement is not completely unanticipated. It is possible that the three-day duration of the training was not sufficient to bring about a change in these outcomes.

Longer-term outcomesBased on the program’s theory of change, we expected to see changes in participant outcomes one month after training in four areas:

• Perceived impacts on job performance – participants could see changes in their job performance, in particular their ability to engage with customers, listen to customer needs, implement new sales strategies, and overall attitude towards their work

• Compensation – improved job performance could lead to increased compensation, either in the form of direct compensation, or customer tips

• Self-esteem – enhanced performance and higher compensation could boost the participant’s self-esteem

• Future orientation – experiencing successful outcomes following participation in the training could make participants more willing to consider, or invest time in, pursuing larger, more longer-term rewards

Of the participants that completed the training, 62 (43%) completed the one-month follow-up surveys. The results of these surveys on the four outcomes listed above are summarized below:

• Perceived impact on job performance – participants were asked if they had noticed any change across four dimensions of job performance:

▪ 86% reported an improvement in customer engagement (i.e. greeting customers warmly, allowing them to take the lead in customer interactions, and asking open-ended questions to identify needs)

▪ 86% thought their listening abilities had improved (i.e. using non-verbal cues, clarifying what customers say, and responding to customer questions)


27

▪ 79% said their sales strategies had become more effective (i.e. matching products to customer, describing product features and benefits, and suggesting complementing products)

▪ 87% saw an improvement in their attitude (i.e. a sense of ownership, service, initiative, professionalism, and commitment)

• Compensation - participants were asked to provide their hourly wage, hours worked, and an estimate of the value of tips received. At follow-up, participants earned +$0.42/hour in wages, but reported a -$0.60/hour drop in tips, which left their level of compensation effectively unchanged at $20.10/hour. Statistical testing found no significant change in compensation levels before and after the training.

• Self-esteem – we measured self-esteem using four questions adapted from the Rosenberg Self-Esteem Scale. We observed no changes in this scale.

• Future orientation - we measured future orientation using five questions adapted from the Zimbardo Time Preference Inventory. Again, we observed no changes in this scale.


28

Leadership training participantsThis section outlines the results for participants who enrolled in and completed the leadership training. Of the 34 employees who participated in the leadership training, 30 completed the program. Of the 26 leadership training participants

who completed the exit surveys, only 10 completed the follow-up survey.

Most leadership participants had multiple motivations for taking training (see Figure 11, below).


24%

11%

5%

FIGURE 12: | REASONS FOR TAKING TRAINING

To improve my management skills

25% 75%0% 50% 100%

74%

74%

53%

53%

41%

38%

27%

To improve my leadership skills

To improve my communication skills

To do my job better


Because my employer asked


To further my education

To improve my resume

Other reason not listed


Participants were very satisfied with the quality of leadership training at exit. Almost all participants (95%) rated the quality of leadership training as excellent (70%) or very good (25%).

In addition to being satisfied with the training, participants reported that the training was relevant and useful. The majority of participants agreed or strongly agreed that the topics covered were highly relevant to their job tasks (96%) and that what they learned in the training was important information to know for their job (96%).


29

Leadership training participants were also asked one month after completing training whether they had observed performance improvements among frontline employees who had participated in the CSR training. While only a small sample (10 participants) completed the survey, results

suggested that leadership training participants did see some positive improvements in their staff members’ performance, with 7 out of 10 observing improvements in customer engagement and sales strategies skills.

FIGURE 12 | SATISFACTION WITH LEADERSHIP TRAININGFIGURE 13: | SATISFACTION WITH LEADERSHIP TRAINING


I would recommend this trainingto my coworkers (or other employees

in similar organizations)


25% 75%0% 50% 100%

What I learned in this trainingwill help me get further in my career

What I learned in this trainingis important informtion

to know for your job

92%

96%

96%

96%What I have learned from the trainingwill enhance my job performance

4%

The topics we covered in the trainingare highly relevant to my job tasks 96%

4%

4%

4%

4%4%


4 | Results for employers

Thirty-four employers participated in the CSR and leadership training across the eight locations. 19 (56%) of these employers completed input surveys and 25 (74%) completed employer feedback surveys. Of the 25 employer feedback surveys,

15 (58%) were completed by senior managers, 7 (27%) were completed by owners/co-owners, 3 (12%) were completed by a human resources representative, and 1 (4%) was completed by the CEO.


31

Employer characteristicsTable 13 below shows the number of employers that participated in the training at each location. It includes a breakdown of employers by business area, and then by the category of hotel chain or

dining experience. On average, each employer sent six participants to CSR training, with some employers sending as few as a single employee and one employer sending 19 employees to training.

TABLE 13 | PARTICIPATING EMPLOYERS

Location Employers

GTA 6

Fort Frances

6

London 7

Kitchener- Waterloo

6

Ottawa 3

Barrie 3

Kenora 2

Mono 1

Total 34

Business area % Business type %

Hotels 57% Independent hotel chains 47%

National hotel chains 53%

Food and beverage 40% Full service restaurant 50%

fast casual restaurant 50%

Retail 3%


32

SatisfactionEmployers who completed the follow-up survey reported a high level of satisfaction with the overall training package, with all employers rating the quality of the overall training package as good (10%), very good (70%), or excellent (20%). No employers rated the program quality as fair or poor.

Employers reported similarly high levels of satisfaction with the frontline training component and the leadership training component.

Overall, employers were satisfied with the quality of frontline training (see Figure 12). All employers rated the training as good (28%), very good (52%), or excellent (20%). 80% of employers agreed or strongly that the training fulfilled their expectations. Most employers were willing to recommend the program, with 84% agreeing (48%) or strongly agreeing (36%) that they would recommend frontline training to their new hires, and 22 out of 25 employers agreeing (44%) or strongly agreeing (44%) that they would recommend the training to other businesses.

BOX 4 | EMPLOYERS WHO DECLINED TO PARTICIPATE IN CSR

To better understand potential barriers to participation, OTEC administered a short survey to employers who had shown interest in the CSR project, but chose not to participate. The survey asked these employers about perceived challenges in participating and potential changes that would have made it easier to take part. The survey was completed by 17 businesses.

The employers who completed the survey reported that they chose not to participate primarily due to scheduling issues—it was too difficult to schedule days off for their employees to attend training. Cost was also a concern, as many businesses said they could not afford to pay employees to participate in the training. The timing of the training also proved difficult for employers, as the summer season is particularly busy for the hospitality and tourism sector.

Employers identified a number of program changes that would make it easier for them to participate. They would like to see training scheduled to fit business cycles, additional monetary incentives for participation, and frontline training more closely tailored to the needs of their specific business.

The survey findings should be interpreted with caution as the sample is small. However, these findings were corroborated in informal feedback collected from OTEC through their employer engagement efforts.

Leadership training was also positively reviewed. Almost all employers rated the leadership training as good (24%), very good (38%), or excellent (33%). Only one employer rated the training as

fair, and none rated it as poor. About two-thirds of employers agreed (36%) or strongly agreed (32%) that they would recommend the leadership training to their new managers.


33

FIGURE 12 | EMPLOYER SATISFACTION WITH TRAINING

FIGURE 14: | EMPLOYER SATISFACTION WITH TRAINING


I would recommend leadershiptraining to other businesses

in the industry


25% 75%0% 50% 100%

Leadership training hasfulfilled my expectations

I would recommend mynew managers to participate

in leadership training

68%

63%

68%

84%I would recommend Customer

Service Results to other businessesin the industry

16%

Customer Service Results traininghas fulfilled my expectations 76%

27%

36%

20%

32%

I would recommend my new hiresparticipate in Customer Service Results 88%4%

5%

4%

8%



34

UtilityAt intake, businesses were asked about their expectations for the training. The most common expectations were that the training would develop leadership in the organization (100%) and improve their customer’s experience (100%).

Figure 13 shows the extent to which training met or exceeded employer expectations. Employers on average reported that the training met or exceeded their expectations on four of the seven

possible benefits. Employers were most likely to report that CSR met or exceeded expectation in developing leadership in their organization (78%), increasing the basic skills of their employees (72%), helping retain quality employees (72%), and improving their customers’ experience (68%). Employers were least satisfied with the increases in sales as a result of training. Only 35% said that the training met or exceeded their expectation of increasing their company’s sales.

FIGURE 13 | EXPECTED AND REALIZED BENEFITS OF TRAINING

100%

100%

59%

35%

55%

FIGURE 15: | EXPECTED AND REALIZED BENEFITS OF TRAINING

Develop leadership in my organization

25% 75%0% 50% 100%

Improve my customers’ experience

Improve the company’s profitability

Increase the basic skills of my employees

Help retain quality employees

Increase the company’s sales

Increase the productivity of my employees

84%

95%

74%

89%

72%

Expected benefit of training Training met or exceeded expectation

78%

68%

72%79%

% of employers selecting

In addition, we asked employers how much they would be willing to pay to send a single employee to each training. On average, employers said they would be willing to pay up to $677 to send a single employee to frontline training, and $585 to send a single participant to leadership training. But given that eliciting accurate figures for the willingness of an individual or business to pay for an item or service is difficult and often requires

a more complex methodological approach than the direct question used in the CSR surveys, these findings should not be interpreted as precise figures. However, even if the amount is somewhat uncertain, they do show that employers are willing to invest in this type of training, and that the amount is likely to be less than the cost of delivery ($1,800 for CSR training and $1,000 for Leadership training).


35

Perceived impactsOverall, employers perceived some positive impacts on both frontline and leadership performance, with 80% of employers agreeing (64%) or strongly agreeing (16%) that the CSR curriculum had enhanced the job performance of their frontline staff (see Figure 14). However, in general at the time of the survey, training had not yet impacted their bottom line. Only 12% of employers said that they had seen an improvement in their bottom line one month after training.

In addition to general impacts, we also asked employers the percentage of frontline employees who had improved in the four specific aspects of customer service: customer engagement, actively listening to customers, sales strategies, and attitude. Nearly half of employers (48%) said that more than half of the employees who attended training had improved across all four aspects of customer service. 16% said more than half of their employees who attended training improved on three dimensions, and 8% said they improved on two dimensions.

FIGURE 14 | EMPLOYER PERCEPTIONS OF EMPLOYEE PERFORMANCE AFTER TRAINING


What managers have learnedfrom the leadership training hasenhanced their job performance

25% 75%0% 50% 100%

What front-line staff have learnedfrom the Customer Service Results

training has enhanced theirjob performance

36% 64%

20% 80%

4%

12%

88%

No, I have not seen a changeYes, an increase

PARTICIPANT RESPONSE

Have you seen any change in your profit margin or bottom line?




36

Employer case studiesWe prepared in-depth case studies on four employers, drawing on interviews we conducted regarding their experiences with CSR training.

These case studies focus on the motivation of the employers for taking part in the training, and the perceived impact of the training on their employees.

BOX 5 | EMPLOYER CASE STUDY #1

Motivation

A Toronto-based hotel signed up for CSR because they believed it represented a “win-win” scenario. The hotel has struggled to implement staff training due to time constraints and small profit margins. The CSR project allowed the hotel to explore whether staff training would be a worthwhile investment. The hotel was interested in improving food and beverage specific skills, increasing motivation in frontline employees, and honing the leadership skills of supervisors.

Perceived impact

The hotel reported an improvement in the skills of frontline employees, as well as an increase in productivity and customer satisfaction. The training was also useful for on-boarding new staff. Staff who participated in the leadership training showed improved leadership skills, including a new focus on the “people side” of the business (i.e. effectively managing frontline staff as a team). Overall, employee satisfaction increased as their staff felt invested in and cared for.

The employer noted that leadership training was most effective for junior and mid-level managers, but would not be appropriate for more senior staff.

Conclusion

The hotel reported positive impacts for both frontline and supervisory staff. The combination of frontline and leadership training in a single program was seen as unique among training programs in the industry and beneficial to employers.


37


Motivation

A hotel in Kitchener viewed CSR as a chance to improve knowledge and motivation among its frontline employees. It also saw the leadership training course as an opportunity to enhance knowledge, accountability, and management skills among mid-level supervisory staff for whom there is limited in-house training.

Perceived impact

The hotel reported that the CSR curriculum had a positive impact on the knowledge and skills of new frontline employees, which led to improved customer experience, and increased profitability through increased up-selling. The training was also seen as useful for on-boarding new employees.

The leadership training helped supervisors improve and refine their leadership styles. The modules on accountability and empathy with frontline staff were seen as particularly useful.

The employer noted that both the frontline and leadership training curriculums could have been more closely aligned with current industry needs, and would have been more impactful if they contained more advanced content for experienced staff. They also felt that frontline staff should be allowed to participate in leadership training in preparation for a future supervisory role. This could help address the shortage of employees in Kitchener who perceive hospitality as an attractive career option.

Conclusion

Overall, the hotel has seen a positive impact on staff. In the long term, participation in the program could lead to the development of more in-house training at the hotel.


38


Motivation

Their training needs and the lack of local training opportunities motivated a small Fort Frances hotel to sign up for CSR. Their existing training typically involves briefing new employees on policies and instructing them to shadow more experienced colleagues. But rapid turnover means the business is always in “training mode”, and sometimes customers are lost due to untrained staff. There is no training for managers.

Perceived impact

The employer reported that the CSR training improved the motivation and assertiveness of frontline staff, which helped them gain more respect from other staff members. The employer saw the training as potentially cost efficient, as the improved skills would enable the frontline servers to bring in more business on slow nights. The leadership training improved the communication and patience of managers when dealing with difficult issues. It was seen as particularly useful for more experienced managers who would benefit from some refresher training.

Conclusion

The training proved to be a great compliment to the shadowing. The employer welcomed the opportunity to send employees to training in the area, noting that few such opportunities were available locally.


Motivation

A large, name-brand hotel in Ottawa signed up for the CSR training based on OTEC’s reputation and good past experiences with OTEC’s training programs. In particular, the business was keen to refer new managers to the leadership training program.

Perceived impact

Employees reported that the training was highly relevant and that they were able to put parts of it into action immediately. The training had a clear and immediate impact on customer service, with guests at the hotel giving excellent feedback on employees who had taken part in the training.

Conclusion

The hotel was very positive about the quality of the training and its impact on employees. However, it did caution that branded hotels typically opt for in-house training to retain brand identity and standardization. But they did point to a significant market for the training among non-branded businesses.

5 | Process evaluation findings

We conducted a process evaluation to understand the success factors, challenges, and context in which the project took place. The process evaluation addressed three main questions:

• What worked well? What components of the project were implemented successfully? What positive impacts did partners observe? What factors contributed to success?

• What were the challenges? What components of the project were more difficult to implement? What factors created barriers?

• What role did context play? How did seasonality, geography, and the policy and labour market context influence project success?

WHAT WORKED WELL

Many aspects of the CSR program worked well. Both employers and participants felt that the curriculum was aligned with their needs and helped address important skills gaps in the industry. The coaching-based approach and co-delivery of the curriculum were also seen as particularly positive aspects of the project.

• Curriculum design – The combination of essential skills training and a strong industry training component resonated with many employers. Instructors reported that the curriculum was highly relevant to the needs of the industry, and that most participants approached the course with motivation and enthusiasm—which was reflected in high levels of participant satisfaction.

• Alignment with business goals – A number of employers expressed strong interest in CSR because of its alignment with their business


40

needs. As the skills training was sector-specific, it was easy for employers to see how CSR directly connected to better on-the-job performance in their industry. Employers felt CSR was particularly useful for on-boarding new employees—especially those with high potential. Others noted that the training improved morale by making employees feel valued and recognized.

• Coaching-based approach – Instructors noted that the emphasis on coaching, supporting, and building the motivation of participants was important for ensuring success in the project. Some of the trainers began sharing initial skills assessment results with participants as a way of encouraging them to focus on improving their skills. Based on additional input from Douglas College at the mid-project debrief, all trainers committed to sharing assessment results with participants for the second half of the project. They also received additional tips and strategies for coaching and supporting participants.

• Co-delivery was a positive learning experience – While the challenges in coordinating scheduling resulted in fewer instances of co-delivery than expected, the LBS providers who co-delivered with OTEC trainers reported it to be a very positive experience. Co-delivery helped LBS providers learn more about the industry-specific aspects of the curriculum that they were less familiar with. It also enabled providers to build capacity and offer professional development opportunities to their trainers. Some LBS providers welcomed the chance to directly connect essential skills training with employment outcomes.

CHALLENGES

The project also provided lessons on some of the challenges involved in delivering workplace essential skills training in the hospitality and retail sectors.

• Training coordination – Finding suitable training locations, enrolling participants, and scheduling training dates were significant challenges throughout the project. Employers in the hospitality and retail sectors typically have little infrastructure or capacity to coordinate training, especially when working with multiple other employers. Most training cohorts contained employees from multiple businesses, which meant training could not take place on site for all employees. This created logistical challenges relating to transportation, and made scheduling particularly tricky, as coordinators had to find training dates that worked for all participants.

• Coordination with LBS – While LBS providers were generally positive about the training experience, coordinating training to allow for the participation of LBS trainers proved challenging. Many of the training sessions were scheduled on short notice, making it difficult for LBS providers to plan their activities accordingly. In addition to the tight timelines, LBS providers found it hard to accommodate the lack of scheduling flexibility in the project, given the need to accommodate employers’ schedules.

• Diverse cohorts – Differences in skill levels ( job skills, language skills etc.) within cohorts presented some challenges for instructors. Instructors reported difficulties in keeping training relevant to employees from different businesses. For example, many of the skills required to work in high-end dining are very different from those essential for performing well in a busy bar. They also noted that English as a Second Language participants and those with lower skills levels


41

tended to struggle with the curriculum, while more experienced workers and those with higher skills levels were often not sufficiently challenged and could become disengaged.

• Integrating essential skills assessments - Integrating the essential skills assessments into training sessions proved challenging. Most trainers had little experience administering essential skills assessments prior to participating in CSR. In addition, the initial “train-the-trainer” session—which provided instruction on the essential skills assessments—took place several months before the first cohorts were launched. Trainers also found it challenging to provide enough time for participants to complete the pre- and post-assessments while still delivering the full course curriculum.

CONTEXT

Contextual factors played a significant role in determining the successes and challenges experienced over the course of the project.

• Seasonality – The timing of the training was a significant barrier to participation for employers. The summer months are a very busy time in the hospitality sector, and employers were hesitant to remove staff from frontline duties for training. Poor fit with seasonal business cycles was the main reason that the project was unable to launch any cohorts from the retail sector. Employers cited January-April as the best time for training.

• Policy – Some employers were reluctant to send staff to training on their days off as they were unsure who would be working (or taking time off) on any given day due to the new ‘zero-notice time-off’ provision under Bill 148. Some employers reported wanting to keep employees available on their days off in order to cover shifts at short notice if needed. Some employers also indicated that the recent increase in the provincial minimum wage had put pressure on their margins, making them reluctant to invest in staff training.

• Regional variations – Regional factors also played a role in the project. Many employers in smaller labour markets like Kitchener-Waterloo noted that people are less likely to see hospitality as a potential long-term career path, making investing in training a less attractive option. In more remote areas (such as Fort Frances and Kenora in north-western Ontario), medium-sized businesses were particularly interested in training as they have access to fewer local training opportunities; however, these regions also presented unique challenges, including tight labour markets and a high proportion of seasonal workers.

6 | Key takeaways

Results The CSR project achieved some promising results for participants and employers:

• Participants experienced a small but statistically significant gain in numeracy skills. While there was no significant improvement in document use overall, participants entering training with lower skills levels did experience a statistically significant improvement in document use skills. Participants also reported high levels of satisfaction with training. A large majority found the training relevant and useful to their work.

• Participating employers were satisfied with the training and reported that the training was relevant to their business needs. They noted positive impacts for employees in terms of on-the-job performance and productivity.

Lessons for workforce development organizationsThe CSR project highlighted a number of lessons for workforce development organizations interested in workplace contextualized essential skills training in Ontario:

• Employer recruitment strategy – CSR has highlighted the need for an approach to employer recruitment that is tailored to both the sector and to the way in which small to medium-sized businesses operate in the sector. For example, OTEC found that in-person contact was the most effective method for recruiting employers in the hospitality sector. Face-to-face meetings enabled recruiters to explain to employers how the model could help them to meet their HR goals. Leveraging existing relationships with local employers, especially those known to be leaders in training and human resources, was also an effective strategy.


43

• Broadening relevance – The project emphasized the importance of ensuring that training content is aligned with the needs of the sector. Training that is too narrow in scope (i.e. applicable to only a few occupations within the sector) can be more difficult to coordinate, as small to medium-sized businesses may only be able to send one or two employees to training. Conversely, training that is too broad risks losing the contextual focus. One potential way forward would be to combine more generic training content with occupation-specific tasks and exercises, an option that OTEC is currently exploring.

• Targeting specific employees – This project highlighted the need for careful consideration of the type of employees who will benefit most from contextualized essential skills training. Clarity on the target population (defined by skill level, experience, and/or other criteria) would help clarify the value proposition for employers. It would also provide the option of marketing the program directly to employees who could benefit from the training.

• Importance of context – Any training project that aims to work with employees in the hospitality sector must consider the impact of context on project success, including policy (e.g. employment standards), seasonality (i.e. busy times), and regional factors (e.g. local labour markets and local economic profiles).

Broader learnings More broadly, this project highlighted some important learnings about the potential benefits of workplace essential skills training for participants, industry, and government:

1. Contextualized approach aligns well with industry needs – Many essential skills training programs are generic. CSR is more attractive to employers as a contextualized essential skills model as it aligns training with their business needs and goals. This helps employers understand exactly how the training will benefit their businesses—a point reinforced by participating employers who reported that the CSR training was useful, valuable, and relevant to their business needs. The model also aligns well with the Ministry of Training, Colleges and Universities’ focus on sectoral approaches.

2. Standardized product – Training programs like CSR offer a standardized product, in contrast to the more individualized approach to essential skills training adopted by other organizations, including LBS providers. Standardization allows employers to ensure that employee training is consistent, which is key to building and maintaining service quality. The standardized curriculum also creates efficiencies, as multiple employees can participate in training within a single cohort.

3. Partnership and relationship building – The CSR project brought together a diverse set of project partners to engage in collaborations that have broken new ground for each partner. The project’s emphasis on building stronger connections between essential skills trainers, sector organizations, and employers has transformative potential for the delivery of essential skills training to low-skilled workers in entry-level positions.

The CSR project also generated a number of valuable lessons about the factors that could enable


44

the design and delivery of workplace essential skills training programs in Ontario, and support their long-term sustainability and replicability:

• More support for training coordination – Further support for training coordination could support the implementation of programs like CSR moving forward. Coordinating training is difficult for small employers and requires a significant amount of time and resources. Recognizing this, OTEC is exploring the development of local training consortia that support multiple small to medium-sized businesses to identify common training needs and co-invest in training (see Box 9 for more information). Such collaboration may be in the best interests of employers in locations with small labour pools, as employees in the hospitality sector are likely to circulate between different employers in the area over a period of time—providing a strong incentive to create a consistently well-trained local workforce.

• Building relationships between key partners – More opportunities for collaboration and coordination between employers, industry organizations, and training providers could help foster effective workplace essential skills training opportunities. These opportunities could be supported through partnership funding or other targeted investments.

• Leveraging funding opportunities – There is an opportunity to ensure the sustainability of workplace essential skills training programs like CSR by aligning them with existing funding envelopes. OTEC is already exploring the potential for sustaining the delivery of CSR through the Canada-Ontario Job Grant. Some modifications to the CSR curriculum would be required to meet eligibility requirements (e.g. adjusting the length of training) and the needs of consortia of small employers (e.g. building training into work days, and linking training activities to employer priorities).

BOX 9 | CONSORTIUM BUILDING

OTEC is taking steps to develop industry consortia through the Tourism SkillsNet Ontario initiative. One of the aims of the initiative is to increase the ability of small to medium-sized businesses to assess their training needs, and then identify and coordinate the appropriate training. OTEC is applying learnings from the CSR project to address this systemic challenge in the hospitality and tourism sector.

Established in 2018, Tourism SkillsNet Ontario was designed to foster collaboration between multiple employers and training providers in order to improve training coordination and increase investment in training. The network has a provincial advisory group, regional coordination partners, and working groups made up of employers, training partners, referral networks, industry intermediaries, and municipal stakeholders with 12 locations across Ontario.


45

Appendix A

Comparing CSR and the Workplace Training ProgramWhile CSR is based on the Workplace Training program there are some key differences between the two:

Scale – The Workplace Training Program engaged 14 training providers to deliver training to 1208 employees across British Columbia. CSR was more modest in scope with a target of delivering training to 150 employees.

Leadership training – The Workplace Training Program did not include leadership training. Leadership training was included in CSR based on previous research which demonstrated that participants showed greater improvement as a result of the training when they felt supported by their managers. Leadership training also acted as an incentive for employers, who could send one employee to leadership training for every two employees they sent to frontline training.

Incentive structure – The Workplace Training Program offered a $500 stipend to participating employees. CSR participants received $30 for each survey they completed, and were also compensated by their employers for the hours they attended training.

Delivery partners – The Workplace Training Program was delivered by post-secondary institutions, while CSR was delivered by OTEC in collaboration with LBS providers.

Delivery structure – CSR sessions were primarily delivered as three, 8-hour sessions. The Workplace Training program offered some sessions as three, full day sessions but most were delivered as 6-hour sessions over four days. In addition, the majority of cohorts in the Workplace Training Program consisted of employees from a single business, while the majority of CSR cohorts were mixed cohorts.

Eligibility – The Workplace Training Program restricted eligibility for training to those with high school education or less. CSR was targeted to employees with low skills but did not include strict eligibility criteria.


46

Appendix B

Discussion of Learnings from the Contextualized Essential Skills Demonstration Project

IntroductionOn February 5, 2019, partners from OTEC, Blueprint, and the Ontario Centre for Workforce Innovation were joined by representatives from 12 Literacy and Basic Skills (LBS) network organizations across Ontario to discuss learnings from the Contextualized Essential Skills Demonstration Project. The session was facilitated by Angela Hoyt of Evolution Group.

The purpose of the session was to share project learnings, learn more about the experiences of project partners, and discuss implications and next steps for the LBS network and other stakeholders.

The discussion centered around three themes:

Promising features – What are the most promising aspects of the project? How can it add value for workers, employers, the LBS network, and other partners and stakeholders?

Challenges – What are the challenges involved in implementing this type of model? What are some of the barriers for LBS in delivering programs like Customer Service Results (CSR)?

Opportunities moving forward – What are some potential next steps for building on the promising features of this project?

Promising features Employer-focused approach

Programs such as CSR provide an important opportunity for the LBS network to build their expertise and develop their capacity to engage employers and deliver sector- and occupation-specific training. Session participants noted a strong alignment between CSR and LBS efforts in the design and delivery of essential skills training that is relevant to employer needs and that supports access to and success in employment. LBS network organizations that actively participated in the project noted that it helped them build stronger connections with local employers, raise their profile within the community, and increase their capacity to deliver more employment-focused training going forward.

Partnership opportunities

The partnership between LBS providers and OTEC was identified as another promising feature of the project. LBS networks communicated the importance of working with sector-focused organizations in engaging employers and communicating the value of training programs. The involvement of OTEC—a sector-focused workforce development organization with deep ties and a positive reputation in the hospitality sector—was identified as critical in motivating employers to participate in CSR. LBS networks also viewed these partnerships as crucial for accelerating their own


47

capacity-building and expanding their reach. They highlighted the need to develop partnerships with other sector organizations in order to expand the development and delivery of essential skills training to new sectors.

Evaluation

LBS networks emphasized the importance of evaluating programs like CSR in order to understand their effectiveness in achieving significant outcomes for learners and employers. They noted that rigorously assessing and sharing the results of CSR and similar projects is critical in helping to identify the most promising training models, refine program components, and demonstrate the ways in which essential skills training can add value for all participants.

ChallengesEmployer participation

LBS networks noted that getting employers to invest in training for lower-skilled workers can be difficult. While many employers are eager to invest in training for supervisors and managers, some have difficulty seeing the potential value and return on investment from training programs focused on enhancing the skills of frontline staff. These challenges are exacerbated for small to medium-sized employers who often have fewer resources and less flexibility to provide training opportunities. Session participants highlighted the need to work closely with employers to help them understand the business case for investing in training for frontline employees.

Defining target population

LBS networks highlighted difficulties in clearly defining the target population for programs like CSR. They discussed the risks associated with delivering training to learners with very low essential skills levels or other challenges—such as being a recent newcomer—that affect the suitability of training and can lead to participants becoming overwhelmed and disengaged. They also noted that learners with extensive work experience within an industry may find the training too basic. At the same time, working with employers to identify and select frontline staff who could benefit from programs like CSR poses some challenges and risks—including the possibility that workers identified as having lower skills may feel they are being placed in a vulnerable position with their employers. Session participants discussed potential strategies for reaching the appropriate CSR target population, such as directly engaging frontline workers in the target industries, and targeting the training to new employees as part of the onboarding process.

Coordination and delivery

LBS networks identified some challenges in coordinating and delivering programs like CSR that are largely driven by employer needs and timelines. Those who actively participated in the CSR project noted it was difficult for LBS providers to prepare to co-deliver training sessions on short notice and to adjust their schedules to deliver training at the times and locations required to make the training work for employers. These challenges were amplified in Northern Ontario communities, where the geographical distance between LBS networks, providers, and employers created additional coordination challenges. Session participants discussed the need for more training coordination infrastructure in order to increase predictability and lower barriers to training participation.


48

Opportunities moving forward Exploring alignment between program model and LBS requirements

While LBS networks agreed that the CSR program aligns with the LBS mandate and mission, they noted an opportunity to further clarify and address potential areas of friction between programs like CSR and LBS requirements/guidelines. Session participants discussed the implications of CSR for meeting LBS suitability and eligibility criteria and ensuring alignment with the program curriculum framework. They noted that clarity on these points is critical for understanding their ability to commit CSR delivery in the short-term, and for identifying opportunities to strengthen LBS program guidelines to better facilitate opportunities for workplace essential skills training that meets employer and learner needs.

Continue to strengthen partnership opportunities and explore business model

LBS networks expressed a strong interest in exploring further opportunities to work with OTEC to deliver the CSR program and initiated discussions about potential business models going forward—including options for delivering CSR as part of their core services or on a fee-for-service basis. They also highlighted a need to work with providers in their networks to develop a consistent approach to training delivery and harmonize messaging to employers who are interested in the training.


49

References

Hampf, F. and Wiederhold F. (2016) Skills, earnings, and employment: exploring causality in the estimation of returns to skills. Large-scale Assessments in Education 5: 12.

Ton, A. (2012). Why “good jobs” are good for retailers. Harvard Business Review https://hbr.org/2012/01/why-good-jobs-are-good-for-retailers

The opinions expressed in this report do not necessarily reflect those of the Government of Canada or the Government of Ontario.

contextualized essential skills demonstration project...contextualized essential skills...

Documents