kirkpatrick four level evaluation certification program presented jim kirkpatrick, phd hosted by...

Kirkpatrick Four Level Evaluation Certification Program

Presented Jim Kirkpatrick, PhD

Hosted by Booz Allen Hamilton Herndon, VA

November 23-24, 2009

These slides are provided as a courtesy to those who have attended one of our classes or presentations.

These are for internal use within your organization only; all rights reserved.

We thank you in advance for respecting our intellectual property and refraining from duplicating, distributing, selling, or otherwise using this presentation in whole or part for your own profit or promotion.

The following marks are the property of Kirkpatrick Partners, LLC : The Kirkpatrick Business Partnership ModelSM, KBPMSM, Return on ExpectationsSM, ROESM, and Chain of EvidenceSM

Kirkpatrick Foundational Principles

1. The end is the beginning.

2. ROESM is the ultimate indicator of value.

3. Business partnership is necessary to bring about positive ROE.

4. Value must be created before it can be demonstrated.

5. A compelling Chain of EvidenceSM

demonstrates your bottom line value.

©2009, all rights reserved.

Kirkpatrick Four Levels

Level 4: RESULTS The final outcomes that occur as a result

of the training and reinforcement



Level 3: BEHAVIOR The extent to which participants apply

what they learn on the job


Kirkpatrick Four LevelsLevel 2: LEARNING

The extent to which participants acquire intended knowledge, skills, and

attitudes



Level 1: REACTION How training participants react to the

training


Why Evaluate?

• How can the program be improved?

• How can we maximize training effectiveness by reinforcing knowledge, skills, and attitudes?

• How can we demonstrate the value of training?


“What is your job here at the hotel?”

“I am a window washer.”


“What is your job here at the resort?”

“I am part of a team that creates great experiences for our guests!”


LearningResults Behavior Reaction

The Kirkpatrick Business Partnership ModelSM

ROESM

Business need

identified

Measure L1 Reaction

Measure L2 Learning

Measure L3 Behavior

Present L1 Reaction findings

Present L2 Learning findings

Present L3 Behavior findings

Present L4 Results findings

Prepare Chain of EvidenceSM to demonstrate ROE

TARGET critical behaviors and

required drivers

Determine required KSAs,

Learning Objectives

REFINE expectations to define outcomes

An

aly

ze fi

nd

ing

s,

ad

just

, re

peat

step

s as

nece

ssary

Consider necessary learning

environment

Design and build learning program and evaluation tools

Initiate ongoing reinforcement and monitoring

Measure L4 Results

Deliver learning program

ADDRESS jury issues

Identify NECESSITIES For

Success

PLEDGE to work together

© 2009. All rights reserved.

CriticalBehaviors

Business Results

Recognition

Coaching

Refreshers

Mentoring

Accountability

Level 3 Evaluation

ExecutiveModeling

Drivers – encouragers or discouragers


DOT Example

1. Jury – safety managers, district engineers, traffic control, risk managers, maintenance supervisors.

2. Expectations – smooth traffic flow; increased safety for workers and motorists in highway work zones; better planning.

3. Success Outcomes – less than 15 minute delays; reduction in injuries and fatalities; work zone projects completed on time and within budget.

4a. Key New Behaviors – written work zone plans; plan communicated to public; proper work zone set-up; flaggers following procedures.

4b. Success Drivers – observation, feedback and coaching by supervisors; ongoing compliance tracking; ongoing execution of formal and informal recognition programs.


Methods, Tools, and Tips

Evaluation Methods – the process that is used to gather evaluation data

Evaluation Tools – the actual document that is utilized to gather the evaluation data and/or information

Evaluation Tips – what you will do to gain elicit the desired candor and response rates


Guidelines for Evaluating Reaction

• Determine first what you want to find out• Consider a blend of objective and

subjective questions• Consider your rating scale• If indicated, get delayed reactions• Develop with the learner in mind• Set the table for your learners


WHO HOW? WHEN?

WHEN?WHO?WHO?WHAT? WHY?

Level 1

• Reaction Sheets

• Interviews

• Focus Groups


Guidelines for Evaluating Learning

• Start with the issues of usefulness and credibility

• Measure before and after attitudes, knowledge, and/or skills if indicated

• Beware of an overemphasis on assessing knowledge

• Evaluate during the course as well as after

• Use a performance test for skills


WHO HOW? WHEN?


Level 2

• Knowledge Checks• Knowledge Tests• Skills Observation• Refreshers

• Teach Backs• Developing Action

Plans• Case Studies


Guidelines for Evaluating Behavior

• Start with need and importance• Be sure to set the table for this• Don’t overwhelm – more is not better• Measure on a before/after basis if indicated• Consider immediate implementation• Repeat if indicated• Consider more than one source• Answer the ?s, “If so, why?” and “If not, why not?”


WHO HOW? WHEN?


Level 3

• Behavior Survey• Behavior Checklist• Work Review

• Focus Groups• Monitor Action Plans• Action Learning


Guidelines for Evaluating Results

• Determine based on responses from stakeholders – ROESM

• Borrow metrics when possible• Gather data and information when indicated• Consider a tiered approach (short and long term

measures)• Consider evidence versus proof• Don’t make it harder than necessary


WHO HOW? WHEN?


Level 4

• Borrowing Metrics– Business– HR

• Surveys

• Focus Groups


Evaluation MethodsEvaluation Levels

Methods

1 Reaction

2 Learning

3 Behavior

4Results

Survey ● ● ● ●

Questionnaire / Interview ● ● ● ●

Focus Group ● ● ● ●

Knowledge Test / Check ●

Case Studies ●

Work Review ● ●

Skills / Behavior Observation ● ●

Presentation / Teach Back ●

Action Planning ● ● ●

Action Learning ●

Key Business HR Metrics ●©2009, all rights reserved.

1

Post Session L1

Level 1

Level 2

Level 3

Level 4

Was session delivered effectively?

Post Session Reinfcmt, Coaching & L3

Expected results will occur.Build your chain of evidence & showcase

Post Session L2

Did targeted learning occur?

Was there successful transfer of learning to behavior?

Improve Reinforcement

and/or Coaching

NO

YES

NO

NONOYES

YES

YES

Improve Session 2

3

4

5

6

7

8

9

10

11

Delivery of Learning EventL1 & L2 Formative Evaluation

Was failure due to session?

Kirkpatrick Four Level EvaluationFeedback Loop to Eliminate ‘Snags’


Percentage of learning transfer

1975: Percentage of formal learning that is actually applied to the job:

15%

2005: Percentage of formal learning that is actually applied to the job

15%

Dana Robinson, ASTD ICE, 2008


Brinkerhoff Study

Josh Bersin and Associates, 2008

Training Application

Did not try new skills: 15%

Tried new skills and failed: 70%

Achieved sustained new behaviors: 15%


Preparation and Readiness: 20%Learning Intervention: 10%Application Environment: 70%

Causes of “Training Failure”

2006 ASTD Study


Typical Learning Investment

P re-Work 10%

L earning E vent 85%

F ollow-Up 5%

Dr. Brent Peterson, University of Phoenix, 2004


The increase in “informal learning”

Where Learning Takes Place

Prior to being "trained": 20%During "training": 10%On the job: 70%

Josh Bersin and Associates, 2008


Transferring Learning to Behavior

Balancing two major forces:

ACCOUNTABILITYSUPPORT


Data Collection PlanLevel Methods Tools Timing

1 Surveying

Focus Group #1

Reaction Sheet

F.G. Questionnaire

Immediately after course

1 week after course

2 Knowledge test

Skills Observation

Teach backs

Focus Group #1 (see above)

Objective test

Checklist

Instructions

F.G. Questionnaire

Pre and post course

During course

During course

1 week after course

3 On-the-job Observation

Surveying – 180

Focus Group #2

Action learning

Checklist

Survey (partic / mrg.)

F.G. Questionnaire

Instructions

1 week + monthly

4 months after course



4 Survey – Customer

Focus Group #2 (see above)

Review Bus. & HR Metrics

Survey

F.G. Questionnaire

Business & HR Metrics

4 & 8 months after course


6 &12 months after course


Chain of EvidenceSM

Gather data at all four levels and show that your training delivers true value to your organization

Level 1Reaction

Level 2Level 2LearningLearning

Level 4Results

Level 3Behavior


Four Practical Ways to Make a Good Business Case

1. Show them the data

2. Conduct an impact study and showcase it

3. Review your training curriculum and redeploy your resources

4. The Brunei Window Washer


More Information

kirkpatrickpartners.com

[email protected]

Feb. 2010

kirkpatrick four level evaluation certification program presented jim kirkpatrick, phd hosted by...

Documents

chain of evidence sm

expectations sm

kbpm sm

jim kirkpatrick

levels level

value of training

learning objectives

training participants