chapter 07 slides 4e

CHAPTER 7

Werner & DeSimone

1

Evaluating HRD Programs

Effectiveness

Werner & DeSimone (2006)

2

The degree to which a training (or other HRD program) achieves its intended purpose

Measures are relative to some starting point

Measures how well the desired goal is achieved

Evaluation


3

HRD Evaluation


4

Textbook definition:“The systematic collection of descriptive and judgmental information necessary to make effective training decisions related to the selection, adoption, value, and modification of various instructional activities.”

In Other Words…


5

Are we training: the right peoplethe right “stuff”the right waywith the right materialsat the right time?

Evaluation Needs


6

Descriptive and judgmental information needed Objective and subjective data

Information gathered according to a plan and in a desired format

Gathered to provide decision making information

Purposes of Evaluation


7

Determine whether the program is meeting the intended objectives

Identify strengths and weaknessesDetermine cost-benefit ratioIdentify who benefited most or leastDetermine future participantsProvide information for improving HRD

programs

Purposes of Evaluation – 2


8

Reinforce major points to be madeGather marketing informationDetermine if training program is appropriateEstablish management database

Evaluation Bottom Line


9

Is HRD a revenue contributor or a revenue user?

Is HRD credible to line and upper-level managers?

Are benefits of HRD readily evident to all?

How Often are HRD Evaluations Conducted?


10

Not often enough!!!Frequently, only end-of-course participant

reactions are collectedTransfer to the workplace is evaluated less

frequently

Why HRD Evaluations are Rare


11

Reluctance to having HRD programs evaluated

Evaluation needs expertise and resources

Factors other than HRD cause performance improvements – e.g., Economy Equipment Policies, etc.

Need for HRD Evaluation


12

Shows the value of HRDProvides metrics for HRD efficiencyDemonstrates value-added approach

for HRDDemonstrates accountability for HRD

activitiesEveryone else has it… why not HRD?

Make or Buy Evaluation


13

“I bought it, therefore it is good.”“Since it’s good, I don’t need to post-test.”Who says it’s:

Appropriate? Effective? Timely? Transferable to the workplace?

Evolution of Evaluation Efforts


14

1. Anecdotal approach – talk to other users 2. Try before buy – borrow and use

samples 3. Analytical approach – match research

data to training needs 4. Holistic approach – look at overall HRD

process, as well as individual training

Models and Frameworks of Evaluation


15

Table 7-1 lists six frameworks for evaluationThe most popular is that of D. Kirkpatrick:

Reaction Learning Job Behavior Results

Kirkpatrick’s Four Levels


16

Reaction Focus on trainee’s reactions

Learning Did they learn what they were supposed to?

Job Behavior Was it used on job?

Results Did it improve the organization’s effectiveness?

Issues Concerning Kirkpatrick’s Framework


17

Most organizations don’t evaluate at all four levels

Focuses only on post-trainingDoesn’t treat inter-stage improvementsWHAT ARE YOUR THOUGHTS?

Other Frameworks/Models


18

CIPP: Context, Input, Process, Product (Galvin, 1983)

Brinkerhoff (1987): Goal setting Program design Program implementation Immediate outcomes Usage outcomes Impacts and worth

Other Frameworks/Models – 2


19

Kraiger, Ford, & Salas (1993): Cognitive outcomes Skill-based outcomes Affective outcomes

Holton (1996): Five Categories: Secondary Influences Motivation Elements Environmental Elements Outcomes Ability/Enabling Elements

Other Frameworks/Models – 3


20

Phillips (1996): Reaction and Planned Action Learning Applied Learning on the Job Business Results ROI

A Suggested Framework – 1


21

Reaction Did trainees like the training? Did the training seem useful?

Learning How much did they learn?

Behavior What behavior change occurred?

Suggested Framework – 2


22

Results What were the tangible outcomes? What was the return on investment (ROI)? What was the contribution to the organization?

Data Collection for HRD Evaluation


23

Possible methods:InterviewsQuestionnairesDirect observationWritten testsSimulation/Performance testsArchival performance information

Interviews


24

Advantages:FlexibleOpportunity for

clarificationDepth possiblePersonal contact

Limitations:High reactive effectsHigh costFace-to-face threat

potentialLabor intensiveTrained observers

needed

Questionnaires


25

Advantages:Low cost to

administerHonesty increasedAnonymity possibleRespondent sets the

paceVariety of options

Limitations:Possible inaccurate

dataResponse conditions

not controlledRespondents set

varying pacesUncontrolled return

rate

Direct Observation


26

Advantages:NonthreateningExcellent way to

measure behavior change

Limitations:Possibly disruptiveReactive effects are

possibleMay be unreliableNeed trained

observers

Written Tests


27

Advantages:Low purchase costReadily scoredQuickly processedEasily administeredWide sampling

possible

Limitations:May be threateningPossibly no relation to

job performanceMeasures only

cognitive learningRelies on normsConcern for racial/

ethnic bias

Simulation/Performance Tests


28

Advantages:ReliableObjectiveClose relation to job

performanceIncludes cognitive,

psychomotor and affective domains

Limitations:Time consumingSimulations often

difficult to createHigh costs to

development and use

Archival Performance Data


29

Advantages:ReliableObjectiveJob-basedEasy to reviewMinimal reactive

effects

Limitations:Criteria for keeping/

discarding recordsInformation system

discrepanciesIndirectNot always usableRecords prepared for

other purposes

Choosing Data Collection Methods


30

Reliability Consistency of results, and freedom from

collection method bias and error

Validity Does the device measure what we want to

measure?

Practicality Does it make sense in terms of the

resources used to get the data?

Type of Data Used/Needed


31

Individual performanceSystemwide performanceEconomic

Individual Performance Data


32

Individual knowledgeIndividual behaviorsExamples:

Test scores Performance quantity, quality, and timeliness Attendance records Attitudes

Systemwide Performance Data


33

ProductivityScrap/rework ratesCustomer satisfaction levelsOn-time performance levelsQuality rates and improvement rates

Economic Data


34

ProfitsProduct liability claimsAvoidance of penaltiesMarket shareCompetitive positionReturn on investment (ROI)Financial utility calculations

Use of Self-Report Data


35

Most common methodPre-training and post-training data Problems:

Mono-method bias Desire to be consistent between tests

Socially desirable responses Response Shift Bias:

Trainees adjust expectations to training

Research Design


36

Specifies in advance:the expected results of the studythe methods of data collection to be usedhow the data will be analyzed

Research Design Issues


37

Pretest and Posttest Shows trainee what training has accomplished Helps eliminate pretest knowledge bias

Control Group Compares performance of group with training against

the performance of a similar group without training

Recommended Research Design


38

Pretest and posttest with control groupWhenever possible:

Randomly assign individuals to the test group and the control group to minimize bias

Use “time-series” approach to data collection to verify performance improvement is due to training

Ethical Issues Concerning Evaluation Research


39

ConfidentialityInformed consentWithholding training from control groupsUse of deceptionPressure to produce positive results

Assessing the Impact of HRD


40

Money is the language of business.You MUST talk dollars, not HRD

jargon.No one (except maybe you) cares about

“the effectiveness of training interventions as measured by and analysis of formal pretest, posttest control group data.”

HRD Program Assessment


41

HRD programs and training are investments

Line managers often see HR and HRD as costs – i.e., revenue users, not revenue producers

You must prove your worth to the organization – Or you’ll have to find another organization…

Two Basic Methods for Assessing Financial Impact


42

Evaluation of training costsUtility analysis

Evaluation of Training Costs


43

Cost-benefit analysis Compares cost of training to benefits gained

such as attitudes, reduction in accidents, reduction in employee sick-days, etc.

Cost-effectiveness analysis Focuses on increases in quality, reduction in

scrap/rework, productivity, etc.

Return on Investment


44

Return on investment = Results/Costs

Calculating Training Return On Investment


45

Results Results

Operational How Before After Differences ExpressedResults Area Measured Training Training (+ or –) in $

Quality of panels % rejected 2% rejected 1.5% rejected

.5% $720 per day

1,440 panels 1,080 panels

360 panels

$172,800 per day per day per year

Housekeeping Visual 10 defects 2 defects 8 defects

Not measur- inspection (average) (average) able in $

using

20-item

checklist

Preventable Number of 24 per year 16 per year 8 per year

accidents accidents

Direct cost $144,000 $96,000 per

$48,000 $48,000 per

of each per year year

year

accident

Return

Investment

Total savings: $220,800.00

ROI = =

SOURCE: From D. G. Robinson & J. Robinson (1989). Training for impact. Training and Development Journal, 43(8), 41. Printed by

permission.

Operational Results Training Costs

= $220,800$32,564

= 6.8

Types of Training Costs


46

Direct costsIndirect costsDevelopment costsOverhead costsCompensation for participants

Direct Costs


47

Instructor Base pay Fringe benefits Travel and per diem

MaterialsClassroom and audiovisual equipmentTravel Food and refreshments

Indirect Costs


48

Training managementClerical/AdministrativePostal/shipping, telephone, computers,

etc.Pre- and post-learning materialsOther overhead costs

Development Costs


49

Fee to purchase programCosts to tailor program to organizationInstructor training costs

Overhead Costs


50

General organization supportTop management participationUtilities, facilitiesGeneral and administrative costs,

such as HRM

Compensation for Participants


51

Participants’ salary and benefits for time away from job

Travel, lodging, and per-diem costs

Measuring Benefits


52

Change in quality per unit measured in dollars Reduction in scrap/rework measured in dollar cost of

labor and materials Reduction in preventable accidents measured in

dollars ROI = Benefits/Training costs

Utility Analysis


53

Uses a statistical approach to support claims of training effectiveness: N = Number of trainees T = Length of time benefits are expected to last dt = True performance difference resulting from

training SDy = Dollar value of untrained job performance (in

standard deviation units) C = Cost of training

U = (N)(T)(dt)(Sdy) – C

Critical Information for Utility Analysis


54

dt = difference in units between trained/untrained, divided by standard deviation in units produced by trained

SDy = standard deviation in dollars, or overall productivity of organization

Ways to Improve HRD Assessment


55

Walk the walk, talk the talk: MONEYInvolve HRD in strategic planningInvolve management in HRD planning and

estimation efforts Gain mutual ownership

Use credible and conservative estimatesShare credit for successes and blame for

failures

HRD Evaluation Steps


56

1. Analyze needs.2. Determine explicit evaluation strategy.3. Insist on specific and measurable training

objectives.4. Obtain participant reactions.5. Develop criterion measures/instruments

to measure results.6. Plan and execute evaluation strategy.

Summary


57

Training results must be measured against costs

Training must contribute to the “bottom line”

HRD must justify itself repeatedly as a revenue enhancer

chapter 07 slides 4e

Documents