training evaluation. training evaluation training evaluation provides the data needed to demonstrate...
TRANSCRIPT
TrainingTrainingEvaluationEvaluation
Training evaluationTraining evaluation provides the data needed to demonstrate that training does provide benefits to the company.
What are the differences What are the differences among: among:
Training effectivenessTraining effectivenessTraining outcomesTraining outcomesTraining evaluationTraining evaluationEvaluation designEvaluation design
Types of EvaluationTypes of Evaluation
FormativeFormativeSummativeSummative
Why Evaluate Training Why Evaluate Training Programs?Programs?
Why Evaluate Training Why Evaluate Training Programs?Programs?
ObjectivesObjectivesSatisfactionSatisfactionBenefitsBenefitsComparisonComparison
Objectives = FoundationObjectives = Foundation
Terminal behaviorTerminal behaviorConditions under which Conditions under which
terminal behavior is expectedterminal behavior is expectedThe standard below which The standard below which
performance is unacceptable performance is unacceptable
--> criteria by which the --> criteria by which the trainee is judgedtrainee is judged
The Evaluation ProcessThe Evaluation Process
Conduct a Needs AnalysisConduct a Needs Analysis
Develop Measurable Learning Outcomes Develop Measurable Learning Outcomes and Analyze Transfer of Trainingand Analyze Transfer of Training
Develop Outcome MeasuresDevelop Outcome Measures
Choose an Evaluation StrategyChoose an Evaluation Strategy
Plan and Execute the EvaluationPlan and Execute the Evaluation
Training Outcomes: Training Outcomes: Kirkpatrick’s Four-Level Kirkpatrick’s Four-Level Framework of Evaluation Framework of Evaluation CriteriaCriteria
Business results achieved by traineesBusiness results achieved by traineesResultsResults44
Improvement of behavior on the job; akaImprovement of behavior on the job; akaskillsskills
BehaviorBehavior33
Acquisition of knowledge, skills, attitudes, Acquisition of knowledge, skills, attitudes, behavior; aka cognitivebehavior; aka cognitive
LearningLearning22
Trainee satisfaction; aka affectiveTrainee satisfaction; aka affectiveReactionsReactions11FocusFocusCriteriaCriteriaLevelLevel
How do you know if your How do you know if your outcomes are good?outcomes are good?
Good training outcomes Good training outcomes need to be:need to be:
RelevantRelevantReliableReliableDiscriminativeDiscriminativePracticalPractical
Good Outcomes: Good Outcomes: RelevanceRelevance
Criteria relevanceCriteria relevance – – the extent to which training the extent to which training programs are related to learned capabilities programs are related to learned capabilities emphasized in the training programemphasized in the training program
Criterion contaminationCriterion contamination – – extent that training extent that training outcomes measure inappropriate capabilities outcomes measure inappropriate capabilities or are affected by extraneous conditionsor are affected by extraneous conditions
Criterion deficiencyCriterion deficiency – – failure to measure training failure to measure training outcomes that were emphasized in the training outcomes that were emphasized in the training objectivesobjectives
Criterion deficiency, relevance, Criterion deficiency, relevance, and contamination:and contamination:
Outcomes Measured in Evaluation
Outcomes Identified by
Needs Assessment and Included in Training Objectives
Outcomes Related to Training
Objectives
Contamination Relevance Deficiency
Good Outcomes Good Outcomes (continued)(continued)
ReliabilityReliability – – degree to which outcomes can degree to which outcomes can be measured consistently over timebe measured consistently over time
DiscriminationDiscrimination – – degree to which trainee’s degree to which trainee’s performances on the outcome actually performances on the outcome actually reflect true differences in performancereflect true differences in performance
PracticalityPracticality – – refers to the ease with which refers to the ease with which the outcomes measures can be collectedthe outcomes measures can be collected
Training Evaluation Training Evaluation PracticesPractices
79%
38%
15% 9%
0%
20%
40%
60%
80%
Re
ac
tio
n
Co
gn
itiv
e
Be
ha
vio
r
Re
su
lts
Outcomes
Perc
enta
ge o
f Cou
rses
Usi
ng
Out
com
e
Evaluation ProceduresEvaluation Procedures
UtilityUtility
[(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)][(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)]
Ns = number of applicants selectedNs = number of applicants selectedT = tenure of selected group in years T = tenure of selected group in years r = correlation between predictor and job r = correlation between predictor and job
performance (VALIDITY)performance (VALIDITY)SDy = standard deviation of job performanceSDy = standard deviation of job performanceZs = average standard predictor score of selected Zs = average standard predictor score of selected
groupgroupN = number of applicantsN = number of applicantsC = cost per applicantC = cost per applicant
[(Nc)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)][(Nc)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)]
Nc = number of trainees who complete programNc = number of trainees who complete programT = T = duration of training benefitduration of training benefit r = correlation between r = correlation between training criteriontraining criterion and job and job
performance (VALIDITY)performance (VALIDITY)SDy = standard deviation of job performanceSDy = standard deviation of job performanceZs = average standard Zs = average standard criterioncriterion score of score of traineestraineesN =N = totaltotal number of number of trainees enrolledtrainees enrolledC = cost per C = cost per traineetrainee
Training CostsTraining Costs
DirectDirectIndirectIndirectDevelopmentDevelopmentOverheadOverheadCompensation for TraineesCompensation for Trainees
50 = Ns = number of trainees who complete program 50 = Ns = number of trainees who complete program 1 = T = duration of training benefit 1 = T = duration of training benefit ..50 50 = r = correlation between training criterion and job = r = correlation between training criterion and job
performance (VALIDITY)performance (VALIDITY)4800 = SDy = standard deviation of job performance 4800 = SDy = standard deviation of job performance
(assume 40% of base pay . . . $12,000 * .40)(assume 40% of base pay . . . $12,000 * .40) .80 = Zs = average standard criterion score of trainees.80 = Zs = average standard criterion score of trainees100 = N = total number of trainees enrolled100 = N = total number of trainees enrolled 150150 = C = cost per trainee = C = cost per trainee
For On the Job Training For On the Job Training $81,000$81,000
[(Ns)*(T)*(r)*(SDy)*(Zs)]-[(N)*(C)](50 * 1 * .50 * 4800 * .8) - (100 * 150)
Experimental DesignsExperimental Designs
Experimental DesignsExperimental Designs
ChoicesChoicesPretest/posttestPretest/posttestControl GroupsControl Groups
Experimental DesignsExperimental Designs
1: 1 group, posttest only1: 1 group, posttest only2: 1 group, pretest/posttest2: 1 group, pretest/posttest3: Pretest/posttest control group3: Pretest/posttest control group4: Solomon four-group 4: Solomon four-group 5: Time-series5: Time-series6: Nonequivalent control group6: Nonequivalent control group
Experimental DesignsExperimental Designs Validity Validity
InternalInternalExternalExternal
Experimental DesignsExperimental Designs Threats to Internal ValidityThreats to Internal ValidityHistoryHistory
MaturationMaturationTestingTestingInstrumentationInstrumentationRegression toward the Regression toward the
meanmeanDifferential selectionDifferential selectionExperimental mortalityExperimental mortality
InteractionsInteractionsDiffusion/imitation of Diffusion/imitation of
treatmentstreatmentsCompensatory Compensatory
equalization of equalization of treatmentstreatments
Rivalry/desirability of Rivalry/desirability of treatmentstreatments
DemoralizationDemoralization
Experimental DesignsExperimental Designs Threats to External ValidityThreats to External ValidityReactive effect of pretesting Reactive effect of pretesting Interaction of selection & Interaction of selection &
treatmenttreatmentReactive effects of Reactive effects of
experimental settingsexperimental settingsMultiple-treatment interferenceMultiple-treatment interference
Issues in Training Issues in Training ValidityValidity
Training validityTraining validityTransfer validityTransfer validityIntra-organizational validityIntra-organizational validityInter-organizational validityInter-organizational validity