roi toolkit

8/4/2019 ROI Toolkit

http://slidepdf.com/reader/full/roi-toolkit 1/75

ROI of Human Capital Interventions

A Practice Toolkit



Description Icon

Relevant quote

Concept/Framework

Tools, Techniques

Template



Step 1

Presenting Problem to Real Problem



Step 1. Presenting Problem to Real Problem

The first task of the practitioner is almost always analysis

Starts with a“presenting”

problem – not a

“hunch” but a

problem … as it

presents itself




Example…

Let's take an

example – higher

staff turnover for

Graduate

Engineers (let say

compared to an

industry standard)

The first question should not be, what skills and knowledge are required. It should be which

metric(s) are affected negatively.




Example…

Unit cost perreplacement

Cost of 'covering' avacant position

(Calculated costs of other employees

'filling in' while theposition is vacant)

Cost to fill a vacantposition

On boarding &Orientation cost

Cost of productivityramp-up




1.1 Calculating benchmark employee cost

Nr Metric Element Description Example

1. Departing employee annual base

salary

N/a 100,000

2. Calculated annual benefits cost Estimated at 20% of

base salary

20,000

3. Calculated monthly salary + benefits 10,000

4. Calculated daily salary + benefits Based on 230 (8hr)

working days

522




1.2 Cost of 'covering' a vacant position (Calculated costs of other employees 'filling

in' while the position is vacant)


1. Number of days until the vacant

position is filled

N/a 90

2. Calculated daily cost of 'covering' a

vacant position

33% of departing

employee's daily

salary + benefits

522/3

3. Total cost to 'cover' vacant position 15644




1.3 Cost to fill a vacant position


1. HR/Hiring manager's annual salary N/a 200,000

2. Calculated HR/Hiring manager's hourly rate Based on 230 (8hr) working

days and 20% fringe rate

130

3. Cost of advertising (online and/or print) Enter cost 3000

4. Cost of resume screening HR/Hire manager hours *hourly rate

50*130=6500

5. Cost of interviews (telephone screening, 1st and

2nd)

HR/Hire manager hours *

hourly rate

50*130=6500

6. Cost of behavioural and skills assessments Enter cost 3000

7. Cost of background checks (criminal, credit,

reference, education)

Enter cost 4000

8. Cost of travel/moving expenses (if applicable) Enter cost 5000

9. Total cost to fill a vacant position: 28,000




1.4 On boarding & Orientation cost


1. Trainer/Manager annual salary: N/a 100,000

2. Calculated trainer/manager daily rate Based on 230 (8hr) working

days and 20% fringe rate

522

3. Total training days Enter number of days 10

4. Total on boarding and orientation cost N/a 5220




1.5 Cost of productivity ramp-up (During the first 3 months, an average new

employee performs at 50% productivity of a tenured top performing employ


1. Daily employee cost (salary + benefits) N/a 522

2. Number of working days during first 3 months: Enter number of days (avg.

58 days

50

3. Cost of productivity ramp-up N/a 26100




1.6 Unit cost for filling a vacancy

R 74964

Two important learning points:

I. Know your metrics

II. Negotiate, and even request help from your client to determine the right metrics. They are

closer to the business and at the same time you obtain by in for the intervention.




1.7 Total cost of replacing graduate engineers

R 74964 x 6

= R 449784

Let's assume that we had to replace five graduate engineers the

past six months



Analysis andDefinition of

Business Need

Identification ofPerformance

Need

Competence GapSolution

GenerationIntervention

Strategy

PresentingProblem/

Opportunity

Non TrainingSolution


Now we can move on to analysis

We know now which

metrics are affected

Consultant speciality




The basic tasks in analysis

4. Perspective and informationfor decision making

3. Dataanalysis

2. Dataconsolidation

1. Datagathering




Two types of data

Data

Quantitative

Qualitative

” More about context, how

people experience in their

own words”.

”The truth is in a number” –

however does not say much

about context




Data gathering

I. Questionnaires

II. Surveys

III. Tests

IV. Interviews

V. Focus groupsVI. Observation

VII. Performance records

VIII.Knowledge and skills testing

IX. Program follow up

X. Project assignments




Data gathering

I. Target population/participants;

II. Supervisors of target

population/participants;

III. Subordinates of target

population/participants;

IV. Peer Group;V. HRD Staff;

VI. Documentation:

• Organizational Performance

Records:

• Disciplinary records;

• Safety records;• Scorecards




Contextual factors to consider when selecting data collection methods

I. Time for participants to respond;

II. Costs;

III. Amount of disruption of normal

activities;IV. What is the level of certainty that

you will achieve with the result;

V. Practicality;

VI. Culture/Philosophy – example the

10% syndrome.




Requirements for effective data collection instruments and techniques

I. Construct validity ( methods,

techniques and test instruments

should be evaluated in terms of its

theoretical grounding);II. Predictive validity (methods,

techniques and test instruments

should be evaluated regularly in

terms of its ability to predict what it

purport to predict);

III. Face validity – for line managementvery important!




Requirements for effective data collection instruments and techniques

Reliability:

I. Produce consistent results

over time, that differentiateseffectively, and are

independent of the

assessor/data collector, or

context.




Data consolidation

Data Consolidation Techniques areWide and Varied

Descriptive statistics

Qualitative description, using organizational

and other models and concepts




Data analysis

Data Analysis Varies According toSituation

Descriptive and inferential statistics

Qualitative decisions about the situation,

based on decision making and techniques



Brief Exposition of Problem

Our analysis shows that the HR and line

managers involved in selection decisions do

interviews in a haphazard and nonscientific

fashion

Solution

Following discussions with consultants, HR

experts and line all agreed we need to

implement competency based interviewing


Lets assumethis based

on the core

example



The training department is instructed to

develop an appropriate program and put all

line and support staff through training – they

however indicate that a change in results is

required quickly and that ROI for the trainingneed to be justified soon after

implementation…


Lets assume this

based on the core

example



Presenting Problem to Real Problem

So we have addressed this even before we asked any questions about training – and

now we can move on to the next step

Starts with a“presenting”

problem – not a

“hunch” but a

problem … as it

presents itself



Step 2a

Planning the Intervention (Starting With the End

In Mind)

St 2 Pl i th I t ti (With ROI i Mi d)



Step 2a: Planning the Intervention (With ROI in Mind)

When we plan an intervention – considering that we want to measure ROI- we really reverse

engineer (or start with the end in mind) – We use the Kirkpatrick Model to achieve this

4. Impact

What is the business impact we need to achieve (refer step 1where we got a firm grip on the metric that must change)

3. Transfer

What is the work place behaviour that must change

2. Learning

What knowledge, skills and attitudes need to be acquired

1. Reaction

What is the required reaction of the learner to theintervention

Step 2a Planning the Inter ention (With ROI in Mind)




Back to our case study to show how we plan “backwards” – remember these objectives and

measures for clarification, based on our core example/case study (re graduate engineers)

Kirkpatrick Level Objectives Measures

Impact Train target population (define) in behaviour based

interviewing techniques, with the aim to:

a. Improve predictive validity of selection decisions

b. Driving down cost of filling vacancies for graduate

engineers

25% reduction in total cost of filling

vacancies for graduate engineers

Transfer Correct application of behaviour based interviewing

techniques by target populations during selection

interviewing

Expert observation with the aid of

checklists show that all involved in

selection interviews use the correct

interviewing structure (agenda) and

questioning techniques at least 85%

of the time. Observation is

conducted for two months following

the initial training and corrective

coaching is implemented where

required, following interviews.






measures a merely examples


Learning Using Blooms Taxonomy of Learning Objectives here are

a few examples of the objectives that will be outlined

here:

a. The learner should demonstrate that they can

analyse a CV, with the aim to develop behaviour

based questions for an interview;b. The learner should demonstrate the ability to

develop interview questions according to the

behavioural interview format

c. Etc. etc…

100% compliance with the Bloom

Taxonomy, alignment with SAQA Unit

Standards and also SAQA guidelines

and requirements for assessment

and moderation;

Learners score 75% on theoreticalpaper to pass and 80% on practical

simulation interview.

Remember this is asnapshot – there is a

whole learning map and

course development

process underlying

these objectives






measures a merely examples


Reaction The reaction to the learning material, environment,

perception of preparation, facilitator style and

Using a Likert type scale participants

rate the following on average no lessthan “agree”:

a. Preparation

b. Facilitator style

c. Learning environment

d. Participation

e. General expectations met





What have we achieved by planning the intervention this way?

4. Impact

What is the business impact we need to achieve (refer step1 where we got a firm grip on the metric that must change)

3. Transfer

What is the work place behaviour that must change

2. Learning

What knowledge, skills and attitudes need to be acquired

1. Reaction

What is the required reaction of the learner to theintervention

A p a t h –

t r a i l l i n k i n g t h e

b u s i n

e s s n e e d w i t h t h e

i n t e r v e n t i o n

Results

Results

Results

Results

ROI ?





We can also go some steps further and plan where we will get the information to evaluate

the intervention and how…

Kirkpatrick Level Objectives Measures Where Will We Find

Information For

Evaluation

In What Format

Impact See previous slides See previous

slides

HR Budget vis Actual

Staff Turn Over Records

Business Report

HR Report

Transfer See previous slides See previous

slides

Audit/Observation results Possibly checklists

Learning See previous slides See previous

slides

Test scores

Practical simulation

scores

Tests papers and

answer sheets

Simulation checklists

and grading papers

Reaction See previous slides See previousslides Survey results Possibly tabulatedand presented in

graphs



Step 2b

Planning the Intervention (Planning Beyond

Training – With ROI In Mind)

Step 2b: Planning the Intervention (With ROI in Mind)




Usually once we have done the planning and developed the training intervention we are

ready to implement…

a. With ROI in mind our planningneeds to go some steps further…

b. We need to determine whattechnique(s) we will use to showthat the intervention actually hadan impact





We need to determine what technique(s) we will use to show that the intervention actually

had an impact – but our dilemma is this…

External Factors

Management

Attention

Incentives

Systems/Procedures

Changes

Here is our training

program

AND YES THERE IS

IMPROVEMENTAFTER

PROGRAM

IS IT THETRAININGPROGRAM?

OR SOMETHING

ELSE?





We have to isolate the effects of our intervention!

I. Isolating the results/effect of our intervention is the most difficult and also most

important element of building the case for value (ROI);

II. Because of this, value at the higher levels is seldom determined;

III. However, the credibility not only of your study to show value, but also the training

function is based on this element.




p g ( )


I. The complexity of isolating the effect, is a function of the variables that are in play

at any moment in time in an organization;

II. The analyst should however always try to prove a causal relationship to an

acceptable degree of accuracy;

III. Acknowledging the forces that influence individual, group and business

performance is the first step in this process.




p g ( )


Remember that we are trying to isolate the effect of out intervention to an acceptable

degree of accuracy – this makes very important to involve and agree with the client:

a. What metrics should change as result of the intervention (see step 1)

b. What isolation technique will be used (because they differ in terms of accuracy)

Let’s consider the isolation techniques…




p g ( )

Let's look at the techniques!

a. Trend line analysis of performance

data;

b. Control groups;

c. Participant’s estimation of impact;

d. Supervisor’s estimation of impact;

e. Management’s estimation of impact;

f. Use of previous studies

g. Subordinate’s report of other factors;

h. Estimating the impact of other factors;

i. Use of customer input.

With these

techniques we

try to isolate the

effects of our

intervention




Let's look at the techniques!


data;

b. Control groups;








With these

techniques we

try to isolate the

effects of other

interventions

and then declare

the rest as due

to our effort




Trend Line Analysis

I. Draw a trend-line using previous

performance data as a base;

II. When intervention is conducted,

actual performance is plotted and compared to the trend line;

III. Improvement in actual performance

over what the trend line predicted

can then be reasonably attributed to

the intervention.




Advantages Disadvantages

I. Show quick results

II. Easy to calculate

III. Easy to understand

IV. Inexpensive

V. Not difficult to organize (basically just

monitor changes in metrics)

VI. A good first order analysis

I. Uses only one factor to isolate the effect

of the intervention (time- before and

after implementation)

II. Thus less accurate and client could be

sceptical to accept only an analysis of

the trend line.

Trend Line Analysis- Advantages and Disadvantages




Example- Not from our central case study…

AveragePost Program 35

AveragePre Program 55

TIME

C O M P L A I N T S

SexualHarassment

Prevention Program O N D J F M A M J J A S O N D J F M A M J J

A S O

Effect of the

intervention




Control Groups Technique

I. Similar to classical research the control group method, involves an experimental group and a

control group;

II. The so called experimental group will be the target of our intervention;

III. The control group would be a group that is paired with the experimental group, but would

not undergo the intervention;

IV. Important things to remember with this method are:

a. Paring need to be thought about carefully so as not to dilute the impact of the

intervention;

b. Although not as rigid as classical research key factors that could dilute the impact or

assessment of the impact of the intervention need to be controlled.




Advantages DisadvantagesI. More accurate than most techniques

II. If the pairing and control of variables are

agreed with the client the outcome has high

level of credibility

III. With this control groups, given the level of

accuracy it is possible to benchmark the ROI

of certain interventions given a

predetermined set of variables

I. More complicated

II. Require more planning, negotiation,

control and monitoring

III. As result of above many practitioners areunwilling to use this technique

IV. Requires absolute agreement with the

client environment that variables will be

kept constant for the duration of the ROI

evaluation.

Control Groups- Advantages and Disadvantages




Schematic Explanation of control groups design

Control

Group

Experimental

GroupM1 Intervention

M2

M2

M1

*M = measurement

1. Pairing 2.

Control

Variables




Looking at pairing and control of variables

Select your

experimental and control groupcarefully

Note:

I. Too small then meaningful pairing is not

possible – too big then you are not going

to be able to control the variables during

and after the implementation of your

intervention;II. Remember we go for reasonable

accuracy therefore you pair the majorelements and also control the majorvariables!!

III. The above makes it very important toagree upfront with your clientenvironment what how pairing will be

done and which variables will becontrolled.

PAIRING

VARIABLES




So what do you typically pair?

Negotiate creafully

S i z e

N a t u r e o f w

o r k

L e a d e r s h i p s t y p e

P e r f o r m a n c e

l e v e l

G e n e r a l d e m o g r a p h y

Pair for (Primarily)




So what do you typically try to control?

Negotiate creafully

S i z e

N a t u r e o f w

o r k

L e a d e r s h i p s t

y p e

P e r f o r m a n c e l e v e l

G e n e r a l d e m o g

r a p h y

Changes in:

Negotiate creafully

s t r u c t u r e

F u n c t i o n

R e c o g n i t i o n a n d

r e w a r d

T u r n o v e r

S y s t e m s a n d p r o c e d u r e s

Changes in:

+




Remember your decision about pairing and control of variables….

= (f) (your knowledge and

judgement) + agreement withthe client environment +

practicability




Estimation of impact by third parties ( this technique can include an estimation of the

perceived impact of the intervention, by participants, participants supervisors, participants

managers)

This technique provides participants with a questionnaire, and based on the

improvement ask them to respond to the following:

I. What percent of this improvement can be attributed to the application of

skills/techniques/knowledge gained in the training program?

II. What is the basis for this estimation?

III. What confidence do you have in this estimate, expressed as a percentage?

IV. What other factors contributed to this improvement in performance?

V. What other individuals or groups could estimate this percentage or determine

the amount?




Estimation of impact by third parties – how do you increase the confidence level for this

technique?

I. Individuals who do not respond to the questionnaire or provide non- usable data

on the questionnaire are assumed to report no improvement/impact;

II. Extreme data and unrealistic claims are omitted from the analysis;

III. Only annualized values are used, it is assumed that there are no benefits fromthe program after the first year of implementation;

IV. The confidence level, expressed as a percent, is multiplied by the improvement

value to reduce the amount of the improvement by the potential error;

V. If value is expressed it is also factored by the confidence value.




Advantages Disadvantages

I. Easy to implement;

II. Based on the assumption that participants

are capable of determining or estimating

impact;

III. Although an estimate, this value will usually

have considerable credibility with

management because participants are at the

centre of the change or improvement;

IV. High face validity.

I. Less accurate than control groups

II. Based on perceptions – could be

regarded as Return on Expectation

rather than ROI

III. Could become victim of “group think”

Estimation Of Impact By Third Parties - Advantages And Disadvantages




Example from an actual coaching intervention- Not from our central case study…

All indicators

were pre

agreed with the

client

Participantsand

management

agreed which

the respective

groups will

rate



Note – because the techniques related to isolating the impact of other factors are

not used often it is only referred to in this toolkit…


data;

b. Control groups;








With these

techniques we

try to isolate the

effects of other

interventions

and then declare

the rest as dueto our effort



Step 3

Implementation and Data Gathering (With ROI in

Mind)

Implementation and Data Gathering (With ROI in Mind)



Referring back to our original example (turnover in graduate engineers) – let us assume the

following:

Mine A

Control

Correctpairing

Correctcontrol of

variables

Agreementon metrics

Mine B

Experiential

Correctpairing

Correctcontrol of

variables

Agreementon metrics

We opted for a control group

evaluation method



Impact

Transfer

-Evaluation completed

Learning -Evaluation completed

Reaction – Evaluation

completed


following:

In the control group the

need for replacement of

graduate engineers also

down by 10%...possibly

because of anticipation

of changes in HR

approach




So because to the method we used we can assign the

change in te metric to our intervention, mainly because we

controlled major variables. We do this as follows…



Step 4

Converting the Change As Result of the

Intervention to Value and Calculating ROI

Step 4.a Converting the Change As Result of the

Intervention to Value



Referring back to our original example (turnover in graduate engineers)…this is how it is

dome…

Identify the unit of

improvement

In our case study:

Number of

replacements of

graduate engineers in

the past six months

Value per unit =

R 74964 (see slide 12

AND 13

Determine the

performance levelchange

Original replacements =

6Six months after the

intervention = 3

Improvement of 50%

Original cost of

replacementR 449784

Our latest cost of

replacement following

intervention

R 224892

Calculate theperformance

improvement

50% saving R 224892



Always try to link to a metric –

alternatively use the method of

Estimation Of Impact By Third Parties – See slide 55 and 56 (this method is also

referred to as Return on Expectation)





See whether one can link tangible values to these?

OUTPUT

Units Produced

Tons Manufactured

Items Assembled

Money Collected

Items Sold

Forms Processed

Loans Approved

Inventory Turnover

Patients Visited

Applications Processed

Tasks Completed

Output Per HourProductivity

New Accounts Generated

COSTS

Budget Variances

Unit Costs

Cost By Account

Variable Costs

Fixed Costs

Overhead Cost

Operating CostsNumber of Cost Reductions

Project Cost Savings

Accident Costs

Sales Expense

TIME

Equipment Downtime

Overtime

Time to Project Completion

Processing Time

Break in Time for New Employees

Learning Time

Meeting Schedules

Repair Time

Work Stoppages

Order Response

Late Reporting

Lost Time DaysQUALITY

Scrap

Waste

Rejects

Error Rates

Rework

Shortages

Product Defects

Deviation From Standard

Product FailuresInventory Adjustments

Time Card Corrections

Percent of Tasks Completed Properly

Number of Accidents





And these….

WORK HABITS

Absenteeism

Tardiness

Visits to the Dispensary

First Aid Treatments

Violations of Safety Rules

Number of Communication Break-downs

Excessive Breaks

Follow-Up

WORK CLIMATE/SATISFACTION

Number of Grievances

Number of Discrimination Charges

Employee Complaints

Job Satisfaction

Employee Turnover

Litigation

Organization Commitment

Employee Loyalty

Increased Confidence

CUSTOMER SERVICE

Customer Complaints

Customer Satisfaction

Customer Dissatisfaction

Customer Impressions

Customer Loyalty

Customer Retention

Customer Value

Lost Customers

EMPLOYEE DEVELOPMENT/ADVANCEMENT

Number of PromotionsNumber of Pay Increases

Number of Learning Programs Attended

Requests for Transfer

Performance Appraisal Ratings

Increases in Job Effectiveness

INITIATIVE/INNOVATION

Implementation of New Ideas

Successful Completion of Projects

Number of Suggestions Implemented

Setting Goals and Objectives

New Products and Services Developed

New Patents and Copyrights



In our case example we did determine the value of

the change and can therefore now move on to

calculate cost benefit and therefore also ROI

Step 4.b Calculating ROI

First thing to do now is to determine the cost of the intervention – again back to our main




example …

Cost element Notes ZAR value (for the sake of our example)

Development Costs If you use more than one trainer or one

trainer on more than one project – pro-rate

R50,000

Program Materials R10,000

Instructor/Facilitator Costs If you use more than one trainer or one

trainer on more than one project – pro-rate

R10,000

Facilities Costs R10,000

Travel and lodging R10,000

Salaries and time loss of participants Will need to consult HR and line and pro-rate R50,000

Administration costs R5,000

Evaluation costs R 5,000

Total cost of the intervention R150,000





First thing to do now is to determine the cost of the intervention again back to our main

example …

Benefit/Cost Ratio =Net Program Benefits (R224892- R150,000)

Program Costs (R150,000)

Ratio = 0.499928





Net Program Benefits (R224892- R150,000)

Program Costs (R150,000)

First thing to do now is to determine the cost of the intervention again back to our main

example …

Benefits ROI =X 100

1

Benefit ROI = 50% thus for

every R1 I spent on this

program you got your money

back through saving as well

as an additional 50 cents.

roi toolkit

Documents