programme evaluation and impact programme...

Programme Evaluation and Impact Assessment of SMART programme

1

Programme Evaluation and Impact Assessment of SMART programme by

the Tech Mahindra Foundation

Prepared by KPMG Advisory Services Private Limited


2


3

Table of Contents

Disclaimer and notice to the readers 03

List of tables 04

List of figures 04

Abbreviations used 04

Executive Summary 1. Context and Overview of SMART programme

2. Study Design

2.1. Objectives of the assignment

2.2. Stakeholder identification, sampling and data collection tools

2.3. Analysis framework

3. Observations and Findings

3.1. IRECS

3.2. Impact Assessment using Social Return on Investment (SROI)

3.2.1. Impact mapping

3.2.2. Evidencing outcome and giving them a value

3.2.3. Establishing impact

3.2.4. Calculating SROI

4. Conclusion and Recommendations

05 07

10

10

11

13

13

22

22

24

24

26

27


4

Disclaimer and notice to the readers The information contained herein is of a general nature and is not intended to address the circumstances

of any particular individual or entity. No one should act on such information without appropriate

professional advice after a thorough examination of the particular situation.

We have prepared this report solely for the purpose of providing select information on a confidential

basis to the management of Tech Mahindra Foundation in accordance with the letter of engagement

dated 30th April 2015 executed between Tech Mahindra Foundation and KPMG (“Engagement Letter”).

This report is confidential and for the use of management only. Any distribution beyond the

management (including any copying, circulation, reference or quoting) in correspondence or discussion

with any other party, in whole or in part, shall only be with our prior written consent, as per the terms

agreed between KPMG and Tech Mahindra Foundation.

This report sets forth our views based on the completeness and accuracy of the facts stated to KPMG

and any assumptions that were included. If any of the facts and assumptions is not complete or accurate,

it is imperative that we be informed accordingly, as the inaccuracy or incompleteness thereof could have

a material effect on our conclusions.

While performing the work, we assumed the genuineness of all signatures and the authenticity of all

original documents. We have not independently verified the correctness or authenticity of the same.

We have not performed an audit and do not express an opinion or any other form of assurance. Further,

comments in our report are not intended, nor should they be interpreted to be legal advice or opinion.

While information obtained from the public domain or external sources has not been verified for

authenticity, accuracy or completeness, we have obtained information, as far as possible, from sources

generally considered to be reliable. We assume no responsibility for such information.

Our views are not binding on any person, entity, authority or Court, and hence, no assurance is given

that a position contrary to the opinions expressed herein will not be asserted by any person, entity,

authority and/or sustained by an appellate authority or a court of law.

Performance of our work was based on information and explanations given to us by the staff of the Tech

Mahindra Foundation. Neither KPMG nor any of its partners, directors or employees undertake

responsibility in any way whatsoever to any person in respect of errors in this report, arising from

incorrect information provided by the Tech Mahindra Foundation staff.

Our report may make reference to ‘KPMG Analysis’; this indicates only that we have (where specified)

undertaken certain analytical activities on the underlying data to arrive at the information presented; we

do not accept responsibility for the veracity of the underlying data.

In accordance with its policy, KPMG advises that neither it nor any partner, director or employee

undertakes any responsibility arising in any way whatsoever, to any person other than Tech Mahindra

Foundation (TMF) in respect of the matters dealt with in this report, including any errors or omissions

therein, arising through negligence or otherwise, howsoever caused.

In connection with our report or any part thereof, KPMG does not owe duty of care (whether in contract

or in tort or under statute or otherwise) to any person or party to whom the report is circulated to and

KPMG shall not be liable to any party who uses or relies on this report. KPMG thus disclaims all

responsibility or liability for any costs, damages, losses, liabilities, expenses incurred by such third party

arising out of or in connection with the report or any part thereof.

By reading our report the reader of the report shall be deemed to have accepted the terms mentioned

hereinabove.


5

List of tables

Table 1: Expected shortfall in manpower by 2022 ................................................................. 8

Table 2: SMART models of implementation and courses offered........................................10

Table 3: Stakeholders, sampling and data collection tools ..................................................12

List of figures

Figure 1: Cities with SMART centers ................................................................................................................ 9

Figure 2: Key stakeholders ................................................................................................................................ 11

Figure 3: Number of respondents from each city ....................................................................................... 12

Figure 4: Education qualification of students ............................................................................................... 14

Figure 5: Age profile and marital status of students .................................................................................. 15

Figure 6: Number of months after which placement was received and current nature of

employment of the students ............................................................................................................................ 17

Figure 7: Mobile public address system mounted on bike ....................................................................... 20

Figure 8: Pamphlets distributed in communities ........................................................................................ 20

Figure 9: SMART+ center at Hyderabad ........................................................................................................ 21

Figure 10: TMF SMART board on the entrance and inside the center ................................................... 22

Figure 11: Charts on Microsoft Word, Excel and Power point at the SMART center ......................... 22

Figure 12: Information pamphlets used by SMART centers .................................................................... 22

Figure 13: Attribution to various stakeholders as per inputs from the students ................................ 26

Abbreviations used

CRS Customer Relationship and Sales

CSR Corporate Social Responsibility

HO Head Office

IGIT Indo-German Institute of Technology

IRECS Inclusiveness, Relevance, Expectations, Convergence, Service Delivery

IT Information Technology

ITES Information Technology Enable Services

ITI Industrial Training Institute

MIS Management Information System

MOU Memorandum of Understanding

M&E Monitoring and Evaluation

NGO Non-Governmental Organizations

NSDC National Skills Development Centre

SMART Skills-For-Market Training

SOP Standard Operating Procedures

SROI Social Return on Investment

TMF Tech Mahindra Foundation

TOT Training of Trainers

WPR Work Place Readiness


6

Executive Summary

Established in 2007, Tech Mahindra Foundation is driven by the vision of ‘‘Empowering through

Education“. The Foundation is working in ten cities across India with over 110 partners in

education and skills development sector.

The Foundation’s flagship programme for skills development, SMART (Skills-for-Market Training)

was conceptualised to enhance employability of youth from underprivileged background and

linking them with employment opportunities. The programme has scaled up from one center in

September 2012 to 65 centers by March 2015. In this span of three years, the programme has

benefited more than 15,923 youth.

The evaluation study presented in this document assesses the SMART programme on IRECS

(Inclusiveness – Relevance – Expectations – Convergence – Service Delivery) parameters and

measure SROI (Social Return on Investment). Key stakeholders consulted during the study include

over 3500 SMART students using online questionnaire, 200 students through focus group

discussion and 5 location mangers, 10 center managers, 20 trainers, 10 community mobilisers and

20 employers through in-depth interviews.

The performance of SMART programme on IRECS parameters is provided below:

Inclusiveness: The programme charges zero fee for trainings, which ensures that every

youth, from economically marginalized communities serviced by the SMART centers, can

access training. Specialized SMART+ centers are established to provide training to people

with disabilities. Furthermore, the programme has achieved gender equity in access to

trainings, as equal number of male and female students have been trained at the SMART

centers.

Relevance: There is a clear linkage between skills training provided at the SMART centre

and jobs available in the local market. The SMART centers are able to place an average of

72.19 percent of students after completing the training. It is also important to note that,

more than 66.33 percentage of students are still serving in the same job where they were

placed by SMART center. Employers also recognised that the SMART students are more

punctual, disciplined and prepared for workplace.

Expectations: The SMART programme has largely met the expectations of the students by

ensuring that students get desired job after the training. Students reported increase in self-

confidence, aspirations and respect from the community. One of the key observation made

by girl students was the change in attitude by their parents towards their career. Girls

informed that their parents are more supportive of their career now and they experience

lesser pressure for marriage.

Convergence: The SMART programme leverage the skills and experience of the

implementation partners. It has tie-ups with experienced knowledge partners and

corporates who are supporting the programme with technical support or infrastructure.

The TMF has also planned to align the SMART programme with government policies on

skilling.

Service Delivery: The SMART programme is being implemented across ten cities through

registered non-profit implementation partners. TMF supports the implementation partner

with required infrastructure and provide curriculum and programme implementation


7

guidelines. The Center Managers play key role in selecting partners and monitoring

programme implementation. The synergy between the implementation partners and TMF

is identified as key to the efficient delivery of the programme.

It was found that, over the last three years, the SMART programme has created social value worth

INR 13.29 for each rupee invested by the Tech Mahindra Foundation (SROI of 13.29). The outcomes

of the training programme experienced by the students are provided below:

Increased skills, employability and access to employment opportunities

Increased self-confidence, self-esteem and aspirations

Improved personality in terms of ability to have a conversation with people, manage facial

expressions and dressing style

Increased respect from the parents, friends, relatives, neighbours and community

Financial independence and financial support for siblings and parents

Increased support for career development for girls and reduction in pressure for marriage

after completing the education

Increased understanding of the course and improved teaching methods for trainers


8

1. Context and Overview of SMART programme

Established in 2007, Tech Mahindra Foundation is driven with the vision of ‘‘Empowering through

Education“. The Foundation is working in ten cities across India with over 110 partners in

education and skills development sector. In order to address issues concerning the teaching-

learning process, the Foundation has adopted a holistic approach entailing core objectives like

improving learning outcomes; enhancing teaching-learning environment; providing teacher

training; facilitating effective pedagogy; enhancing employability and employment opportunities

for youth; recognition for teachers and formulating coherent policies for the issues mentioned.

1.1. Context

India is facing a two-pronged challenge of unemployment and un-employability, implying the

scarcity of jobs as well as the lack of manpower with appropriate skill sets. In this light, the country

not only requires to generate sufficient employment opportunities, but also needs to focus on

harnessing and developing potential manpower.

While an estimated unemployment rate of 4.9 percent at all India level1 is disconcerting, the

unemployment figure of approximately 50 percent2 for the people with graduation and above

degree is a cause for huge concern. Additionally, as per the study conducted by Ministry of Skill

Development and Entrepreneurship, the expected shortfall in manpower by 2022 is estimated as

below3:

Key Sectors Expected shortfall by 2022 (in million)

Building, Construction & Real Estate 31.13

Organized retail 17.13

Transport & logistics 11.66

Tourism, Hospitality & Travel 6.48

Textile & clothing 6.31

Education & skill development services 4.29

Auto & auto components 3.90

Healthcare 3.80

IT & ITES 2.35

Table 1: Expected shortfall in manpower by 2022

Recognizing this as a challenge as well an opportunity, the Twelfth Five Year Plan focuses on

improving the employability of youth through skill development and fostering high value

employment in primary and secondary sectors. To boost the efforts, a Ministry of Skill

Development and Entrepreneurship (MoSDE) was established and a flagship programme named

1 Fourth Annual Employment-Unemployment Survey 2013-14 conducted by Labour Bureau; www.labourbureau.gov.in 2 Report on Youth Employment-Unemployment Scenario 2012-13 3 “Minister of Skill Development and Entrepreneurship Shri Rajiv Pratap Rudy Launches Human Resource and Skill Requirement

Reports for 24 Priority Sectors”, 9 April 2015, Press Information Bureau, Ministry of Skill Development and Entrepreneurship,

Government of India


9

‘Skill India Programme’ (National Skill Development Mission) on July 15, 2015. This programme

seeks to provide an institutional capacity to train a minimum 400 million people by 2022.

1.2. Overview of SMART programme

The SMART (Skills-for-Market

Training) programme was

conceptualised and designed to

address the need for employability

skills among underprivileged youth.

The programme, started with its first

center in September 2012 and scaled

up to 65 centers by March 2015 i.e.

within a span of just three years. In

this short span, the SMART

programme has benefited more than

15,9234 youth across ten cities -

Bangalore, Bhubaneshwar,

Chandigarh, Chennai, Delhi- NCR,

Hyderabad, Kolkata, Mumbai, Pune

and Vishakhapatnam.

The distinctiveness of SMART

programme lies in the three models,

namely; SMART (for women and men

from socio-economically deprived

communities), SMART+ (for

differently abled youth) and SMART-

T (technical skills for youth). This design is in alignment with the mission of the Mahindra Group

which says, ‘Educated, skilled and able women and men are a country's true strength. We aspire

to see children who are purposefully engaged, youth that is constructively employed, and a society

that provides equal opportunities to people with different abilities. Through its corporate social

responsibility initiatives, the Mahindra Group commits itself to this’.

The SMART programme is a 3 month course imparting a combination of different skillsets-

foundation skills, technical skills and field exposure.

Foundation courses: This course ranges up to a total of 210 hours and consists of the

subjects on functional English (Yuva English - 120 hours), soft skills (workplace readiness -

40 hours) and basic IT skills (50 hours). Foundation courses are common to all students at

SMART.

Technical course: This course ranges up to a total of 90 hours and consists of specific

subjects under Customer Relationship and Sales, Tally accounting, Hospitality and IT

enabled services. For SMART-T students the subjects include Automobile Technician, Civil

4 Data provided by Tech Mahindra Foundation

Bhubaneswar

Delhi-NCR

Chandigarh

Bengaluru

Kolkata

Chennai

Mumbai

Pune

Hyderabad Visakhapatnam

Figure 1: Cities with SMART centers


10

Works, CNC Machine Technician, AC and Refrigeration mechanic, Electrician, Fitter and

Maintenance Technician and Welding.

Field exposure: Students are given field exposure in the Technical skills for minimum of 4

hours. For example, under Customer Relationship and Sales course students, students

would observe representatives pitching the client and also interact with prospective clients.

SMART Models

Following table provides details of SMART programme implementation models.

Table 2: SMART models of implementation and courses offered

SMART Model Target Group Courses / Trades Covered

SMART Youth over 18 years of age

Urban-socio-economically

disadvantaged individuals

Able to read and comprehend

simple English

Job seeker

Foundation course: Spoken English,

Workplace readiness and Basic IT and

computer.

Specialized courses: Customer Relationships

and Sales, Hospitality, ITES/ BPO, Nurse

Aides, Office Administration, Pharmacy

Assistant, Quick Service Restaurants, Tally

SMART+ Young women and men with visual and

physical disabilities - people with

hearing/speech impairment

visual-impairment

locomotive disabilities

Specialized courses: Hospitality, ITES/ BPO,

Customer Relationships and Sales,

Multimedia, Tally

SMART-T Youth over 18 years of age

Urban-socio-economically

disadvantaged individuals

Job seeker

Specialized courses: CNC machine operator

– Milling and Turning, Automobile

Mechanics, AC and refrigeration

Technicians, Electrical Maintenance and

Technicians, Basic and Advanced welders,

Fitter Maintenance Technicians

The programme has outlined a robust set of processes for each of the programme components

including identification of implementation partner, selection and training of students, placements

of students in appropriate jobs and tracking students post placements.


11

2. Study Design

2.1. Objective of the assignment

The SMART program is operational for last three years with 65 SMART centres established as on

March 2015. Tech Mahindra Foundation hired KPMG Advisory Services Private Limited’s Social

Sector Practice team to undertake an evaluation of the programme and conduct an impact

assessment study. Broader objectives of the study were to:

assess the programme design and implementation model

assess programme management processes and organisation capacity

measure impact of the programme on students

provide inputs to further strengthen the programme and enhance the impact

The study was designed to evaluate the programme using IRECS (Inclusiveness – Relevance –

Expectations – Convergence – Service Delivery) parameters and assess the impact using SROI

(Social Return on Investment) methodology. Key highlights of the study design are provided

below:

2.2. Stakeholder identification, sampling and data collection tools

Stakeholders: Key stakeholders involved

in the programme are represented in the

figure. KPMG team engaged in

consultation with each of the

stakeholders to meet the objectives of

the assignment.

While the students provided vital inputs

to measure impact, other stakeholders

provided information on the programme

design, implementation model,

management processes and

organisation capacity.

Sampling: highlights of the sampling are

provided below:

Number of respondents to the online

questionnaire is proportionate to the

total number of students passed out

from respective city

Five cities – Delhi-NCR, Bengaluru, Hyderabad, Mumbai, and Kolkata were covered to

conduct face to face consultations

Two centers per city based on the number of students passed out were selected

A mix of SMART (5 centers), SMART+ (3 centers) and SMART-T (2 centers) were

selected

Telephonic interview with Technical Advisor for SMART-T was conducted

Students

TMF HO Team

Location Manager

Center Manager

Trainer

Community Mobiliser

Employer

Figure 2: Key stakeholders


12

Following table summarises the sample selected for data collection and the tools used.

Table 3: Stakeholders, sampling and data collection tools

Stakeholder Data collection tools used Number of respondents

Students Questionnaire (closed ended)

Focus Group Discussion checklist

Case study format

3500 (online responses collected from

all the cities)

TMF HO team In-depth interview checklist 2

Location Manager In-depth interview checklist 5

Center Manager In-depth interview checklist 10

Trainer In-depth interview checklist 20

Community Mobilisers In-depth interview checklist 10

Employers In-depth interview checklist 20

Number of respondents across the SMART centers as captured in the online responses to the

questionnaire are represented in the figure below:

Figure 3: Number of respondents from each city

2.3. Analysis framework

The performance of SMART programme was assessed using the IRECS framework and the impact

of the programme was measured using the SROI methodology.

2.3.1. IRECS framework

IRECS framework measured the performance of programme on five parameters – Inclusiveness,

Relevance, Expectation, Convergence and Service Delivery. Overview of areas assessed under

each of these five parameters is provided below:

Inclusiveness: Assesses the extent to which communities equitably access the benefits of

assets created and services delivered. It also assesses the role of different stakeholders in the

design and implementation of the project.

Relevance: Assesses if the project is geared to respond to the felt needs of the communities.

Expectations: Examines the extent to which the process of project implementation meets the

expectations of the communities.

352

108

119

475

1049

1051

258

75

199

113

Bengaluru

Bhubaneshwar

Chandigarh

Chennai

Delhi

Hyderabad

Kolkata

Mumbai

Pune

Vishakhapatnam


13

Convergence: Judges the degree of convergence with government/other partners; the degree

of stakeholder buy-in achieved; and the linkages with concurrent government programmes in

the field.

Service Delivery: Assessing the state of operations of the programme outputs in terms of

delivering the intended services to the beneficiaries.

2.3.2. Social Return on Investment (SROI)

Social Return on Investment (SROI) is a methodology to monetize the social and environmental

impact of the project and measures how much value has been created for each rupee invested/

spent on the project. This method is an analytic tool and an accepted form of stakeholder centered

evaluation. The evaluative aspect of SROI quantifies the value of the social impact of the projects.

It endeavors to measure the social, environmental and economic value created as a result of

investment.5

Through SROI, organizations are able to evidence the social impact, gain deeper insight into what

impact they are having for their stakeholders and can thus use this as an input for their

programme strategy.

5 The SROI Guide: http://www.thesroinetwork.org/publications/doc_download/241-a-guide-to-social-return-on-investment-2012 as on 5th March 2015


14

3. Observations and Findings

This section provides a snapshot of the performance and impact of the SMART programme. The

information presented in this section is based on the observations made during the field visits to

the 10 SMART centres and interaction with the stakeholders including executive team of TMF,

implementation partners, knowledge partners, trainers, employers and students. The field

observations are substantiated with the data collected from more than 3500 students through an

online questionnaire. The SMART programme was evaluated using the IRECS parameters and an

impact assessment was conducted using the SROI methodology. Details of our observations are

provided below:

3.1. IRECS6

The IRECS framework was used to assess the programme on five parameters: Inclusiveness,

Relevance, Effectiveness, Convergence and Service Delivery.

3.1.1. Inclusiveness

The SMART programme was observed to be inclusive as is evidenced below: Profile of Students: SMART imparts training to

students from the socio-economically

marginalised youth. The age group of students

is primarily 20 – 25 years whereas the education

qualification varies from 8th and 12th pass to

graduates (mostly B.Com. and B.Tech.)

depending on the nature of centre (SMART or

SMART –T). Distribution of education profile of

the students is presented in the adjacent graph.

Ratio of female to male is 1:1 which is in line

with the guidelines by SMART programme. Few

SMART centers are also exclusively for female

operated by specialised organisations.

It is important to note that, the SMART

programme caters to youth with disabilities

through specialised SMART+ centres.

Age profile and marital status of the students is

showcased in the figures below.

6 Primary data collected from interviews with TMF HO team, location managers, center managers, trainers and employers, and FGDs with the students

0 200 400 600 800

10th

12th

Below 10th

Graduation

Incomplete Graduation

ITI

Others

Male

Female

Figure 4: Education qualification of students


15

Figure 5: Age profile and marital status of students

Cost of training: SMART training centres do not charge any fees from the students to ensure

that students from very poor economic background can access the training.

Selection Process: The selection process does not ask for any pre-requisite skill apart for the

minimum level of education required for the course. Center manager assess the seriousness

of the candidate towards training and taking up job after the course completion.

Assessment of student performance: Students undergo regular performance assessments

during the course and one post-training assessment. As per current format, same questions

are used for pre-training and post-training assessment.

The training certificates are provided to students who perform up to a defined standard in

these tests. However, students who do not perform well and hence do not complete the

training, can continue to visit the centres and learn till they are able to reach the required level

of skill to complete the training. Almost 97% of the students have shown satisfaction with the

assessment process.

3.1.2. Relevance

The SMART programme addresses the specific problem of employability of the youth and

unmet demand for skilled labour in the market. The relevance of SMART programme in selected

communities was observed as is showcased below:

Skills for job: There is a clear linkage between skills training provided at the SMART centre

and available jobs in the local market. The training courses being imparted at the center

are chosen based on the baseline data and skills gap assessment done by the

implementation partner.

Communication skills and personality development: It was specifically noted during the

interaction with the students that more than employment or new skills learned, students

were motivated due to improvement in their communication skills and personality.

Students attributed their success and ability to secure job to their improved communication

skills and personality.

953

2178

364

104

19 and below

20-25

26-31

31 and above

Age profile

1440

356

3

1658

105

3

Unmarried

Married

Divorced

Marital Status

Male Female


16

Curriculum design: The courses offered at the SMART centres has a adequate mix of technical

skills, soft skills and field exposure needed for the job. Students appreciated the component of

exposure visits and practice sessions for interviews. They further stated that course on

Workplace Readiness enable them to stand-out in the early days of employment. Almost 98%

students observed that the course content is good.

Placements: The SMART programme is able to place about 72.19 percent students enrolled for

training, which is equivalent to the placement figures mandated by NSDC for an effective

training programme.

Almost 63.00 percent students received placement within one month of completion of

training. Some students also informed that they received placement even before

completing the training.

It is also important to note that 66.33 percent students who were placed through SMART

center are still continuing in the same job.

“I did not understand English earlier; but after completing the training I can communicate fluently in English”

Rahul acknowledged that before undertaking training at the

SMART centre, he had difficulty in understanding the English

language and wasn't confident enough to talk to new people.

The 'Spoken English' course, delivered as part of the training,

has helped him to hone his English language skills and his

communication skills.

The most significant change, that he observed, is the change

in his confidence level and personality. He attributes this

change to his comfort with conversing in English language.

He further quoted that his current job at Optol Infocom Pvt.

Ltd. (placement from SMART centre) requires him to

communicate with client in English, and he is able to do his

job effectively because of the skills acquired at SMART.

The transformation in his personality is acknowledged by his friends and Rahul thinks that he

has gained respect in his friend circle. His inhibitions and fear of communicating due to lack

of the knowledge of the English language are things he was able to change through this

course.

Rahul’s father is a Tea stall owner. Rahul has now started supporting his family with

increased income which has doubled from INR 4000/- per month to INR 8000/- per month.

Increase in income has enabled him to purchase assets for the family like fridge, cycle and

contribute to the household rent. Moreover, Rahul is saving money to buy the house he is

currently renting.

Rahul enjoys music and reading books. He has passed 12th standard and has a younger

brother in 9th standard.

Rahul

SMART, Kolkata


17

Figure 6: Number of months after which placement was received and current nature of employment of the students

Employer satisfaction: The relevance of the training is further substantiated by the employers.

During the interaction with employers, KPMG team had following observations:

- The students trained at SMART centres are very punctual and disciplined. The students are

better prepared for the workplace environment.

- The students have basic/ foundation skills and understand the subject matter faster;

reducing the effort and time spent on on-job training.

63%

33%

1%

4%

0-1 month

1-5 months

Above 5 months

Not received

0% 10% 20% 30% 40% 50%

Jobless

Permanent job

Others

Student

Contract/ Probationary job

“course modules are highly relevant to the employment offered” Sushma, currently employed as trainee at AP Mahesh Bank,

Hyderabad agrees that the modules she learned at SMART

center are relevant to her current job. She quips that she would

have never afforded the Tally course as the fees charged by

private coaching institutes are very high. She further stated that

Tech Mahindra Foundation has helped her realise her dream, as

she was able to learn Tally and got a job that she wanted.

Commenting on the relevance of course curriculum, she said

that, “the course on Tally was exceptionally good and provided

her with an excellent job offer at the completion of the course. The modules covered during

the training were relevant to the employment offered. The spoken English and Work place

readiness component of the course were helpful during job interviews. The course on spoken

English was very specific to the regional requirements of the location and the Telugu

speaking students in terms of comprehensiveness of the subject and teaching methods

used”.

Sushma enjoys dancing and has two elder sisters and a brother. A daughter of an electrician,

she is looking forward to a career in the Banking sector which she thinks would not have been

possible without SMART training programme.

P. Sushma,

SMART, Hyderabad


18

3.1.3. Expectations

The SMART programme has largely met the expectations of the students. The benefits from the

programme, as identified by the students, are listed below:

Desired Job: Almost everyone joining SMART center has an aspiration to get a better job

profile apart from the increased remuneration. Students specifically mentioned that the jobs

received after the training meet their expectations of better working environment, on-job

facilities provided by the employer, kind of client they interact with, working hours and

scheduled holidays. While approximately 31% students have a permanent job and 69% are on

contract / probationary job out of those who are employed; almost 89% students have shown

satisfaction with the placement they received. Around 7.5% of the passed out students have

not yet received desired job offers.

Increase in self-confidence: About 96.30% respondents have experienced increase in self-

confidence and self-esteem and 88% respondents reported increase in their decision-making

abilities due to SMART training. This increase in self-confidence has not only helped students

but their families as well.

Supporting Family Income: More than 57% students stated that they support their family with

their income. The average salary earned by SMART students is INR 7733/- per month which

enables them to support family with monthly grocery shopping, paying house rent, paying for

the education of siblings or their entertainment.

Purchase of Assets: Increase in income has enabled students to afford assets like mobile phone

and bicycle for themselves or refrigerator for the family.

Support for career development for girls: Most of the girl students reported that before

enrolling in the programme and the job placement, their families wanted to marry them; but

that is not the case now. After seeing the progress of their daughters, parents realize that their

“Gained respect in community with better job profile” Ayan was coaching students in his community after

completing his BA in 2012. But the people in his community

never respected him and threw taunts like “he is coaching

kids as he does not have any other job”. After securing job

with Aditya Birla Group, Ayan’s income has significantly

increased from INR 1,000 to INR 6,000. This has enabled him

to not only contribute to the household income but also be

financially independent.

There has been a substantial increase in the level of his

self-esteem and he has a got a sense of achievement and

pride. Ayan is confident that he will be able to grow

professionally and fulfil his dream of travelling and start-up

his own business.

Ayan Ganjan SMART, Kolkata


19

daughters can be successful if given an opportunity. There is clear change in attitude towards

marriage for the girls in families.

This change can further lead to increased respect for the girls in society and better health of

the girls. Right age for marriage will also ensure better family planning and better health for

the mother and child.

3.1.4. Convergence

SMART programme has achieved convergence, to an extent, with the agencies working in skills

development sector and with corporates with similar vision. Primarily, the SMART programme

leverage the skills and experience of the implementation partners at all training centers.

Tie-ups with knowledge partners: SMART programme, specifically SMART-T and SMART+, is

delivered with support from institutions with expertise in the respective fields. The technical

support on SMART-T is provided by Indo-German Institute of Technology (IGIAT),

Vishakhapatnam. IGIAT was formed through collaboration between the Government of Andhra

Pradesh and Government of Germany through the PPP model and has been operational for 9

years. Similarly, SMART+ programme is implemented by organisations with expertise and

proven track record in providing trainings to persons with disability.

Tie-ups with Corporates: SMART has tied up with Tally Solutions Pvt. Ltd. to design the

curriculum, undertake assessments and issue joint certificates (Tally and SMART in this case)

at the end of the training programme. Such a partnership will increase the market demand for

the trained student, which would result in better placements. SMART has also tied up with

Schneider Electric India Pvt. Ltd. and Mahindra and Mahindra Ltd. who provided machinery

and skilled staff for the training.

Tie-up with NSDC: Tech Mahindra Foundation has planned to align the SMART programme

with government policies on skilling. The course curriculum for the SMART programme is

currently being reviewed and restructured to ensure adherence with curriculum recommended

by Sector Skills Councils.

3.1.5. Service Delivery

The SMART programme has grown from one center in Sep 2012 to 65 centers in June 2015. The

programme is now being implemented across ten cities of India through registered non-profit

implementation partners. Some of the salient features of the service delivery model are discussed

below:

Selection of Partner: The process of selecting implementation partner is very efficient, as the

implementation partners are generally recommended by the location managers or the

knowledge partners who understand the local community and strengths of various non-profits

operating in the proposed area.

Community Mobilization: This is one of the key activity to encourage youth to sign up for

SMART programme. Community mobilization is undertaken by a community mobilisers and

supported by Centre Managers. It was observed that both the centre managers and community

mobilisers have good understanding of the community they operate in and are able to mobilize

sufficient number of youth for trainings. However, few female community mobilisers

expressed concern over access to interiors in slum areas where they find it uncomfortable to

reach and at times they also need to work till late hours.


20

The list of community mobilization tool used

by implementation partners are:

Referral by SMART students: encourage

passed out students and current students

to refer probable candidates from their

community

Support from other non-profits:

Information on the SMART programme is

shared with other organisations to

motivate them for referral

Awareness camps: set up employment

awareness camps in the target community

and collect data of interested candidates

Door to door campaigns: community

mobilisers visit door to door in selected

communities

Mobile public address system:

Community mobilizers make

announcements about SMART

programme to create awareness

Social and print media: Social media

platforms, pamphlets and newspaper

advertisements are also used for outreach

Mass mailers: Implementation partners

collect database from local employment

exchange office and send mail to the

unemployed candidates

Special days: Events are organised on the

International Day for DEAF and Blind

Location of SMART Centres: Most of the

SMART centres are located close to the

communities they serve. Also, centres are

easily reachable by public transport in cases

where centres located away from the

community. More than 72% students

commented that they stay within 5 km distance from the center.

Infrastructure: All SMART centres are equipped with basic infrastructure. One of the student

observed that,” Facilities provided at the centre like lighting, clean toilets are good. Also, the

number of computers provided are sufficient for the number of students in a given batch”. A

picture of SMART+ center at Hyderabad is shown below.

Figure 7: Mobile public address system mounted on bike

Figure 8: Pamphlets distributed in communities


21

Figure 9: SMART+ center at Hyderabad

Relationship between teachers and students: At all the SMART centres, students appreciated

the support provided by trainers and center manager during the training, preparation for

placements and support after placements. In words of one of the respondent, “trainers were

exceptionally helpful and provided necessary guidance which enabled respondent to

comprehend the subject properly. The trainers are very patient and conduct multiple rounds

of exercises to make sure students understand the concept well”.

Placements: The placement process according to the student respondents is very well

organized where trainers arrange for mock interview rounds to make them comfortable with

the interview process. Additionally, a lot of effort is spent in grooming the students for jobs

through courses on English and Workplace Readiness. Regular sessions are conducted with

the students to appraise them of the prevailing conditions in the job market and set their

expectations in terms of the salary and job profile.

Monitoring of the programme: The concurrent monitoring of the SMART centres is undertaken

by the Location Managers. Additionally, team from TMF HO, including the Project Head,

undertake regular monitoring visits to different centers.

Relationship between Implementation Partners and TMF: All the implementation partners

visited during the field visit were highly appreciative of the support provided by the TMF team.

One of the centre managers noted that he has worked for many training institutes in his career

and SMART is one of the well managed programmes and there is never a delay in the

payments. He further appreciated the processes and documentation instituted for the SMART

programme describing it as a great learning experience in efficient delivery of a training

programme.

Branding: SMART programme is formulated and propagated by TMF through implementing

partners who deliver the programme at the ground level. Advertisements play a significant

role in the process of community mobilization, the first step in implementation. It is therefore,

important to ensure that set of guidelines devised for moderating the content and mode of

dissemination of programme information, are adhered to by all centres. However, it is also

equally important to provide some flexibility in context specific requirements. This would

prevent the programme from issues of rigidity and exclusiveness.


22

Figure 10: TMF SMART board on the entrance and inside the center

Figure 11: Charts on Microsoft Word, Excel and Power point at the SMART center

Figure 12: Information pamphlets used by SMART centers


23

3.2. Impact assessment using Social Return on Investment (SROI)

About SROI methodology

Social Return on Investment (SROI) is “a method, an analytic tool and an accepted form of

stakeholder centered evaluation.” The SROI quantifies the value of the social impact of projects,

programmes and policies. It endeavors to measure the value created as a result of investment –

namely social, environmental and economic value which is not fully reflected in conventional cost-

benefit analysis.7 Through SROI, organizations are able to evidence the social impact, gain deeper

insight into what impact they are having for their stakeholders and can thus use this as an input

for their programme strategy.

Type of SROI

KPMG has used Evaluative type of SROI conducted retrospectively and based on actual outcomes

that has already taken place.

Principles used to assess SROI

Assessment of SROI was based on following seven principles:

Involve stakeholders

Understand what changes

Value the things that matter

Only include what is material

Do not over-claim

Be transparent

Verify the result

The following section covers impact mapping and SROI calculations based on the evidenced

outcomes during the primary data collection phase. Key recommendations based on the SROI

results are provided in the conclusion and recommendations chapter.

3.2.1. Impact mapping

The Impact Map provides link between resources contributed to the programme (inputs), the

results of the activity (outputs) and the outcomes of the programme that are a vital part of SROI

analysis. It is to be noted that we have focused on the impact on students and considered trainers

and implementation partner for specific inputs from Tech Mahindra Foundation.

The impact map for SMART programme is given below. The evidence outcomes and the SROI

calculations follow this impact map.

7 http://www.thesroinetwork.org/publications/doc_download/241-a-guide-to-social-return-on-investment-2012 as on 5th March 2015


24

Table 4: Impact map for the SMART programme

Stakeholder

Primary Objectives

Input Outputs Outcomes Impact

For each stakeholder

What is the type of

investment?

Summary of activity

Description of the change

Long term change

Youth beneficiaries / Students

Provide skills development trainings for youth empowerment

Time Students took admission at the SMART center

Improved skills level for employment

Reduced unemployment in the community

Training conducted for Basic IT skills, Spoken English, Workplace Readiness, CRS, ITES, Tally and other courses. Exposure visits conducted

Increased employability and access to desired employment opportunities for the students

Increased self-confidence, self-esteem and aspirations among students

Enhanced social and financial status of student’s family

Improved personality in terms of ability have conversation with people, facial expressions and dressing style

Students became financially independent

Improved wellbeing of family

Students provided financial support to their sibling’s education / extra-curricular activities

Improved education status of family

Increased respect from the parents, friends, community members, neighbours and relatives

SMART students being considered as a role model in the community

Increased support for career development for girls and reduced pressure for marriage

Improved physical and mental health of family

Trainers Provide training for increasing capacity of the trainers

Time Training of Trainers conducted at each location for trainers

Increased understanding of the course content and improved teaching methods

Improved teaching-learning environment in the SMART center

Implementation Partners

Develop infrastructure for implementing SMART programme

Time Develop infrastructure for SMART center

Assets created for the SMART center

Increased efficiency and outreach of the organisation


25

3.2.2. Evidencing outcome and giving them a value

Evidencing outcomes

The impact map was validated during primary data collection phase and the outcomes were

evidenced based on the indicators observed. These outcomes are quantified based on the data

collected from the online questionnaire and focus group discussions conducted at the selected

SMART centers. The quantification of outcome is based on the percentage of students who

experienced the change and total number of students passed out from the SMART training centers

i.e. 15,923 students till 31st March 2015.

Some outcomes can last throughout a person’s life and some outcomes maybe short lived i.e. the

outcomes lasts only as long as the activity lasts. For the purpose of this analysis, the visibility of

the outcomes have been considered to be a maximum of 5 years. This is based on the assumption

that, the students will need to build new skillsets according to their job profile and advancement

in technology and job market. It is further assumed that the students may learn or acquire new

skills while they are employed or under any other training programme, thereby diminishing the

relevance of the training provided through SMART.

Other impact assessment methodologies stop at identifying outcomes while SROI methodology

goes beyond to value them and calculate the social value of impact.

Giving Value to an Outcome - Identifying Financial Proxies

The distinguishing aspect of SROI methodology, compared to other impact assessment

methodologies, is the usage of financial proxies to value the outcome. This process of valuation

is often referred to as monetization because we assign a monetary value to things that do not

have market price. All the prices that we use in our day-to-day lives are approximations – ‘proxies’

– for the value that the buyer and the seller gain and lose in the transaction. Value is, in the end,

subjective. Identifying financial proxy for certain outcomes like monetary saving or increased

income is straight forward; while for outcomes like, increased confidence or becoming financially

independent involves certain assumption for assigning the monetary value. For example, if we

evidence that our intervention results in increase in confidence, the scenario can be equated with

individual attend a seminar or training session specifically focused on increasing confidence.

3.2.3. Establishing impact

After giving value to the outcome, we assessed how much of the outcome would have happened

anyway and what portion of the outcome can be isolated as being added to project activities. To

reduce the risk of over claiming the impact, following four aspects were used. These help gain the

credibility of impact calculation and identify measures to enhance the impact. All these aspects

are generally expressed as percentages.

These percentages are applied to the financial proxy of each outcome to arrive at the total impact

for the outcome.

Attribution is the process of considering impact in isolation; that is independent of other projects

operating in the same geographic area.


26

Following is the attribution to various stakeholders as per inputs from the primary beneficiary-

students. Attribution was commonly collected for all outcomes.

Outcome attribution for SMART programme = 37%

Figure 13: Attribution to various stakeholders as per inputs from the students

Displacement are the positive impacts on one stakeholder at the cost of a negative impact on other

stakeholder. KPMG team did not observe any negative impact on other stakeholder during the

primary or secondary data collection. Thus,

Displacement for the SMART programme = 0%

Deadweight is an estimation of the social benefits that would have accrued anyway, even without

the intervention.

It was noted that few students who enrolled for the course were either working part-time or full-

time. As per data collected from sample students few students were earning salary before joining

the SMART classes.

Deadweight for the outcome - increased employability and access to desired employment

opportunities = 16.49%

According to the students at sample locations, there was no other institute in their community

which provide similar combination of courses at free of cost. As the target beneficiaries are from

economically backward families they could not have afforded private coaching classes. With this

reasoning, we can assume that

Deadweight for other outcomes except the increased employability = 0%

Drop-off is the proportion of outcomes which are not sustained.

It was assumed that the impact of training will last for 5 years. As students will learn new skills as

part of their employment or through training programmes. Also due to change in technology,

there will be changes in skills requirement from the students. Thus the impact will be reduced

gradually in 5 years.

SMART37%

Family Members /Relatives /Friends

22%Your Employer

11%

Government / Local Administration

9%

Self21%


27

Drop-off for all outcomes of the SMART programme = 20% each year

Impact for each outcome

Considering the above parameters, the impact for each outcome is calculated with following

formula:

Impact = Quantity of outcome * Proxy Value * Attribution – Deadweight – Displacement – Drop-

off for each year

3.2.4. Calculating SROI

We are assuming that the outcomes of SMART will last for 5 years. The duration is calculated from

the year of training. Thus the outcome for students who passed out in 2012-13 batch will last till

2016-17. We have used inputs till 2014-15, thus the projections are done till 2018-19.

SROI is a ratio of cumulative present value for each outcome against the total investment in the

project

Total NPV of social value

SROI = --------------------------------------------------

NPV of investment

The detailed calculation for each outcome are available in the detailed report submitted to the

Tech Mahindra Foundation. Based on the values from the calculation tables;

SROI = 13.29

For each 1 rupee invested by Tech Mahindra Foundation, total 13.29 rupees of social value is

generated.


28

4. Conclusion and Recommendations

KPMG team conducted an impact assessment and analysis of SMART programme vis-à-vis its

design and implementation. The achievements of the programme, given the brief duration since

the time of its inception, is noteworthy. This can be attributed to the fervent focus ensured by the

top level management team; coordination between the former and location managers across the

SMART centers and persistent efforts of implementation partners to ensure effective attainment

of set objectives.

While the programme is overall well designed, TMF team ensures that knowledge is transmitted

from the design level to implementation level and the void between theory and practice is bridged.

This is done through a series of discussions, query resolution and training of trainers. To further

enhance the impact, TMF has successfully partnered with agencies like Tally Solutions Private

Limited, Mahindra & Mahindra Limited and Schneider Electric India Private Limited. However, at

this crucial juncture where SMART can be perceived as a role model for other CSR skills

development programmes, we suggest following recommendations for TMF and the

implementation partners.

1. Guidelines for branding, communication material and infrastructure

It is recommended that the TMF should set guidelines for branding and IEC material design.

Also wherever possible, internal infrastructure design should be standardized to ensure

consistent experience across SMART centers.

2. Reporting and sharing of good practices and learnings

Monthly review meetings are being conducted with implementation partners but the

process is not yet formalized. It is suggested that minutes of meetings conducted at each

location are recorded and shared it with the HO. Also each partner should share their

learnings with other partners and with the TMF HO to replicate in other centers.

3. Partner due-diligence reporting

The partner selection process and monitoring is being managed by location managers

efficiently. However the due-diligence report for each partner should be documented

before on-boarding the partner.

4. Guidelines for community mobilization

Community mobiliser play crucial role in the programme success. However it was noted

that in certain instances there are challenges with regard to community outreach which can

cause discomfort for the staff, primarily for female mobilisers. It then become pertinent to

define guidelines entailing necessary precautions to be undertaken for community

mobilization so as to ensure safety, efficiency and effectiveness of the process.

5. Guidelines for trainer selection process

TMF should define trainer selection process to ensure that trainers have required aptitude

and skills for training in the SMART center.


29

6. Review student evaluation process

It is recommended that the student evaluation process can be further reviewed to make it

more stringent.

7. Formation of alumni groups

TMF can facilitate initiation of Facebook or Whatsapp group for each center and outgoing

batches so that the alumni stay connected and the network can be leveraged.

8. Establishing employer helpline and candidate web-portal

As there is high demand for the skilled candidates, TMF can establish a helpline which can

connect employers with the candidates. Also a web-platform showcasing the profile of

students and connect them to the employers can be developed for enhancing employment

opportunities for the students and facilitating smoother hiring process.

9. Increase in field work and hands-on component

As per request from the students, it is recommended that the field work component and

hands-on experience for the students can be increased for further enhancement in

exposure and confidence subsequently in students.

10. Student selection criteria

The SROI calculation indicates that, if only the jobless candidates are selected for SMART

training, the SROI value would increase from 13.29 to 15.01. Thus, it is recommended to

modify the student selection criteria from selecting ‘job seekers’ to ‘currently unemployed’

candidates.

11. Smart-Talk: Special sessions on confidence building and personality development

It was noted during the conversation with students that, increase in confidence level and

enhanced personality had significant impact on them. As per SROI calculations, this

outcome contributes over 12.43% of total social value created. Thus, it is recommended to

focus on the confidence building and personality development aspect of the students which

can enhance the impact further. ‘SMART-Talk’ can be organised at the centers where

alumni, employers or even the real life heroes can share their experiences with the

students.


30

average salary per month for SMART students

96.30%

INR 7,733

72.19%

students experienced increase in confidence level

students are placed from SMART centers

students are still continuing with the same job placed by SMART center 66.33%

43.60% students received information about SMART from friends/ family/ relatives

13.29 rupees of social value is generated for each rupee invested

Achievements of the programme8

15,923 youth with socio-economically deprived background were trained till March 2015

INR 17,614

INR 3.78 Cr of assets created for SMART centres till March 2015

spent per year by students on household goods & supporting education of siblings

students received job offer within one month of completing the training 63.00%

864 youth with visual and physical disabilities were trained till March 2015

Tech Mahindra Foundation

Loveleen Kacker

Chief Executive Officer [email protected]

KPMG Advisory Services Private Limited

Dr. Amit Chatterjee, MD

Technical Director – Social Sector Practice [email protected]


31


32


33

programme evaluation and impact programme...

Documents