kirkpatrick four level evaluation certification program presented jim kirkpatrick, phd hosted by...

Kirkpatrick Four Level Evaluation Certification Program

Presented Jim Kirkpatrick, PhD

Hosted by Booz Allen Hamilton Herndon, VA

November 23-24, 2009

These slides are provided as a courtesy to those who have attended one of our classes or presentations.

These are for internal use within your organization only; all rights reserved.

We thank you in advance for respecting our intellectual property and refraining from duplicating, distributing, selling, or otherwise using this presentation in whole or part for your own profit or promotion.

The following marks are the property of Kirkpatrick Partners, LLC : The Kirkpatrick Business Partnership ModelSM, KBPMSM, Return on ExpectationsSM, ROESM, and Chain of EvidenceSM

Kirkpatrick Foundational Principles

1. The end is the beginning.

2. ROESM is the ultimate indicator of value.

3. Business partnership is necessary to bring about positive ROE.

4. Value must be created before it can be demonstrated.

5. A compelling Chain of EvidenceSM

demonstrates your bottom line value.

©2009, all rights reserved.

©2009, all rights reserved.

©2009, all rights reserved.

©2009, all rights reserved.

Kirkpatrick Four Levels

Level 4: RESULTS The final outcomes that occur as a result

of the training and reinforcement

©2009, all rights reserved.

Kirkpatrick Four Levels

Level 3: BEHAVIOR The extent to which participants apply

what they learn on the job

©2009, all rights reserved.

Kirkpatrick Four LevelsLevel 2: LEARNING

The extent to which participants acquire intended knowledge, skills, and

attitudes

©2009, all rights reserved.

Kirkpatrick Four Levels

Level 1: REACTION How training participants react to the

training

©2009, all rights reserved.

Why Evaluate?

• How can the program be improved?

• How can we maximize training effectiveness by reinforcing knowledge, skills, and attitudes?

• How can we demonstrate the value of training?

©2009, all rights reserved.

“What is your job here at the hotel?”

“I am a window washer.”

©2009, all rights reserved.

“What is your job here at the resort?”

“I am part of a team that creates great experiences for our guests!”

©2009, all rights reserved.

LearningResults Behavior Reaction

The Kirkpatrick Business Partnership ModelSM

ROESM

Business need

identified

Measure L1 Reaction

Measure L2 Learning

Measure L3 Behavior

Present L1 Reaction findings

Present L2 Learning findings

Present L3 Behavior findings

Present L4 Results findings

Prepare Chain of EvidenceSM to demonstrate ROE

TARGET critical behaviors and

required drivers

Determine required KSAs,

Learning Objectives

REFINE expectations to define outcomes

An

aly

ze fi

nd

ing

s,

ad

just

, re

peat

step

s as

nece

ssary

Consider necessary learning

environment

Design and build learning program and evaluation tools

Initiate ongoing reinforcement and monitoring

Measure L4 Results

Deliver learning program

ADDRESS jury issues

Identify NECESSITIES For

Success

PLEDGE to work together

© 2009. All rights reserved.

CriticalBehaviors

Business Results

Recognition

Coaching

Refreshers

Mentoring

Accountability

Level 3 Evaluation

ExecutiveModeling

Drivers – encouragers or discouragers

©2009, all rights reserved.

DOT Example

1. Jury – safety managers, district engineers, traffic control, risk managers, maintenance supervisors.

2. Expectations – smooth traffic flow; increased safety for workers and motorists in highway work zones; better planning.

3. Success Outcomes – less than 15 minute delays; reduction in injuries and fatalities; work zone projects completed on time and within budget.

4a. Key New Behaviors – written work zone plans; plan communicated to public; proper work zone set-up; flaggers following procedures.

4b. Success Drivers – observation, feedback and coaching by supervisors; ongoing compliance tracking; ongoing execution of formal and informal recognition programs.

©2009, all rights reserved.

Methods, Tools, and Tips

Evaluation Methods – the process that is used to gather evaluation data

Evaluation Tools – the actual document that is utilized to gather the evaluation data and/or information

Evaluation Tips – what you will do to gain elicit the desired candor and response rates

©2009, all rights reserved.

Guidelines for Evaluating Reaction

• Determine first what you want to find out• Consider a blend of objective and

subjective questions• Consider your rating scale• If indicated, get delayed reactions• Develop with the learner in mind• Set the table for your learners

©2009, all rights reserved.

WHO HOW? WHEN?

WHEN?WHO?WHO?WHAT? WHY?

Level 1

• Reaction Sheets

• Interviews

• Focus Groups

©2009, all rights reserved.

Guidelines for Evaluating Learning

• Start with the issues of usefulness and credibility

• Measure before and after attitudes, knowledge, and/or skills if indicated

• Beware of an overemphasis on assessing knowledge

• Evaluate during the course as well as after

• Use a performance test for skills

©2009, all rights reserved.

WHO HOW? WHEN?

WHEN?WHO?WHO?WHAT? WHY?

Level 2

• Knowledge Checks• Knowledge Tests• Skills Observation• Refreshers

• Teach Backs• Developing Action

Plans• Case Studies

©2009, all rights reserved.

Guidelines for Evaluating Behavior

• Start with need and importance• Be sure to set the table for this• Don’t overwhelm – more is not better• Measure on a before/after basis if indicated• Consider immediate implementation• Repeat if indicated• Consider more than one source• Answer the ?s, “If so, why?” and “If not, why not?”

©2009, all rights reserved.

WHO HOW? WHEN?

WHEN?WHO?WHO?WHAT? WHY?

Level 3

• Behavior Survey• Behavior Checklist• Work Review

• Focus Groups• Monitor Action Plans• Action Learning

©2009, all rights reserved.

Guidelines for Evaluating Results

• Determine based on responses from stakeholders – ROESM

• Borrow metrics when possible• Gather data and information when indicated• Consider a tiered approach (short and long term

measures)• Consider evidence versus proof• Don’t make it harder than necessary

©2009, all rights reserved.

WHO HOW? WHEN?

WHEN?WHO?WHO?WHAT? WHY?

Level 4

• Borrowing Metrics– Business– HR

• Surveys

• Focus Groups

©2009, all rights reserved.

Evaluation MethodsEvaluation Levels

Methods

1 Reaction

2 Learning

3 Behavior

4Results

Survey ● ● ● ●

Questionnaire / Interview ● ● ● ●

Focus Group ● ● ● ●

Knowledge Test / Check ●

Case Studies ●

Work Review ● ●

Skills / Behavior Observation ● ●

Presentation / Teach Back ●

Action Planning ● ● ●

Action Learning ●

Key Business HR Metrics ●©2009, all rights reserved.

1

Post Session L1

Level 1

Level 2

Level 3

Level 4

Was session delivered effectively?

Post Session Reinfcmt, Coaching & L3

Expected results will occur.Build your chain of evidence & showcase

Post Session L2

Did targeted learning occur?

Was there successful transfer of learning to behavior?

Improve Reinforcement

and/or Coaching

NO

YES

NO

NONOYES

YES

YES

Improve Session 2

3

4

5

6

7

8

9

10

11

Delivery of Learning EventL1 & L2 Formative Evaluation

Was failure due to session?

Kirkpatrick Four Level EvaluationFeedback Loop to Eliminate ‘Snags’

©2009, all rights reserved.

Percentage of learning transfer

1975: Percentage of formal learning that is actually applied to the job:

15%

2005: Percentage of formal learning that is actually applied to the job

15%

Dana Robinson, ASTD ICE, 2008

©2009, all rights reserved.

Brinkerhoff Study

Josh Bersin and Associates, 2008

Training Application

Did not try new skills: 15%

Tried new skills and failed: 70%

Achieved sustained new behaviors: 15%

©2009, all rights reserved.

Preparation and Readiness: 20%Learning Intervention: 10%Application Environment: 70%

Causes of “Training Failure”

2006 ASTD Study

©2009, all rights reserved.

Typical Learning Investment

P re-Work 10%

L earning E vent 85%

F ollow-Up 5%

Dr. Brent Peterson, University of Phoenix, 2004

©2009, all rights reserved.

The increase in “informal learning”

Where Learning Takes Place

Prior to being "trained": 20%During "training": 10%On the job: 70%

Josh Bersin and Associates, 2008

©2009, all rights reserved.

Transferring Learning to Behavior

Balancing two major forces:

ACCOUNTABILITYSUPPORT

©2009, all rights reserved.

Data Collection PlanLevel Methods Tools Timing

1 Surveying

Focus Group #1

Reaction Sheet

F.G. Questionnaire

Immediately after course

1 week after course

2 Knowledge test

Skills Observation

Teach backs

Focus Group #1 (see above)

Objective test

Checklist

Instructions

F.G. Questionnaire

Pre and post course

During course

During course

1 week after course

3 On-the-job Observation

Surveying – 180

Focus Group #2

Action learning

Checklist

Survey (partic / mrg.)

F.G. Questionnaire

Instructions

1 week + monthly

4 months after course

4 months after course

6 months after course

4 Survey – Customer

Focus Group #2 (see above)

Review Bus. & HR Metrics

Survey

F.G. Questionnaire

Business & HR Metrics

4 & 8 months after course

4 months after course

6 &12 months after course

©2009, all rights reserved.

Chain of EvidenceSM

Gather data at all four levels and show that your training delivers true value to your organization

Level 1Reaction

Level 2Level 2LearningLearning

Level 4Results

Level 3Behavior

©2009, all rights reserved.

Four Practical Ways to Make a Good Business Case

1. Show them the data

2. Conduct an impact study and showcase it

3. Review your training curriculum and redeploy your resources

4. The Brunei Window Washer

©2009, all rights reserved.

More Information

kirkpatrickpartners.com

jim.kirkpatrick@kirkpatrickpartners.com

Feb. 2010