case study · models for measuring training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80...
TRANSCRIPT
![Page 1: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/1.jpg)
Case StudyHow to Create a Training Measurement Strategy
![Page 2: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/2.jpg)
Join audio:• Choose “Computer audio” to use VoIP• Choose “Phone call” and dial using the
information provided
Questions/Comments:• Submit questions and comments via the
Questions panel.• Please download the materials from the
materials pane before leaving today’ssession.
• Please continue to submit your textquestions and comments using theQuestions Pane
Note: Today’s presentation is being recorded and will be posted at www.mimeo.com/webinars.
Your Participation
![Page 3: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/3.jpg)
Jessica CoburnDirector of Training
Social Solutions
Ann McDonaldDirector of Talent Development
Mimeo
![Page 4: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/4.jpg)
![Page 5: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/5.jpg)
1. Overview of measurement strategies
2. Background on Social Solutions
3. Deep dive in Jessica’s training methods
4. Key takeaways and tips for you
![Page 6: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/6.jpg)
![Page 7: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/7.jpg)
Models for Measuring Training73.7
32.6
14.7 13.7
8.4
26.3
0
10
20
30
40
50
60
70
80
Kirkpatrick's 4Levels of Evaluation
Phillips' ROIMethodology
Six Sigma Brinkerhoff'sSuccess Case
Method
Bersin's TrainingEvaluation Method
Other
From Learning Analytics: Measurement Innovations to Support Employee Development, Mattox and Van Buren, Kogan Page Limited, 2016, p.24
![Page 8: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/8.jpg)
Kirkpatrick Model
Level 1: Reaction
Level 2: Learning
Level 3: Behavior
Level 4: Results
![Page 9: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/9.jpg)
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
How effective is yourtraining?
How effective is yourstrategy to measure
training?
![Page 10: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/10.jpg)
Training at
User training:
• Customer training
• Employee training
• In-person workshops
• E-learning
• Microlearning
![Page 11: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/11.jpg)
![Page 12: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/12.jpg)
What to Measure
• Consider your audience• Determine your objectives and
ideal results• Then figure out how to measure
that
![Page 13: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/13.jpg)
![Page 14: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/14.jpg)
Reaction
What We Do: Clicktool Survey
Why It’s Important: High Correlation between learning and CSAT scores
Tip: If you’re not collecting this type of feedback, start now!
![Page 15: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/15.jpg)
Learning
What We Do: Grading matrix for course assignments
Why This is Important: “Learning” must take place to ensure behavior change
Tip: Use pretest/post test if you don’t want to assume a zero baseline start
![Page 16: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/16.jpg)
Behavior
What We Do: Pre-test and post-test before, immediately after, and 6 months after training
Why This is Important: It allows you to see a change in behavior
Tip: Consider your training objectives when you develop a strategy for measuring behavior
![Page 17: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/17.jpg)
ResultsWhat We Do: Correlation metrics to show training impact on business
Why Is It Important: This is what it’s all about! Show your value to your organization
Tip: Much of this data exists elsewhere in your organization. You just need to find it.
![Page 18: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/18.jpg)
Kirkpatrick Model
Clicktools survey
Grading matrix
Pre and post tests
Correlation metrics
Level 1: Reaction
Level 2: Learning
Level 3: Behavior
Level 4: Results
![Page 19: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/19.jpg)
Lessons Learned
• Focus on metrics
• Get your trainers’ thoughts and opinions about your strategy and incorporate what is important to them
• Have a strategy to get the data you want
• Even if you don’t have the time or resources to analyze your data now, don’t let that stop you from getting started!
![Page 20: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/20.jpg)
Additional Resources
• Learning Analytics: Measurement innovations to support employee development
by John R Mattox II and Mark Van Buren
published by CEB
• Spotlight Report on Training Measurement 2017
Published by Mimeo
![Page 21: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/21.jpg)
Q&A
Next Webinar:
Take Your Training from In-Person to Online
Brian Washburn, CEO, Endurance Learning
October 182 pm ET
![Page 22: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/22.jpg)
BRING YOUR CONTENT TO LIFE
![Page 23: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/23.jpg)
AppendixSelected tables with more advice for working with learning
analytics
![Page 24: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/24.jpg)
Efficiency Measures Effectiveness Measures Outcome Measures
# of people trained Satisfaction training Increase in customer satisfaction
# of people trained by learning methodology
Knowledge and skills gained due to training
Increase in employee performance
Reach (% of people trained in target population)
Intent to apply learning on the job Decrease in risk
Cost of training per program Expectation that training will improve individual performance on the job
Decrease in costs
Cost of training per learner Expectation that individual performance improvement will lead to organizational performance improvement
Increase in sales
Cost of training per hour Return on expectations Increase in revenue
From Learning Analytics: Measurement Innovations to Support Employee Development, Mattox and Van Buren, Kogan Page Limited, 2016, p.175
![Page 25: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/25.jpg)
Relevant High quality Timely
Avoid analyzing data simply because of ease of collection and analysis
Provide oversight to data collection and analyses to ensure that insight is applicable to the problems it intends to address
Refresh data frequently, ideally whenever new training instances take place or when recurring processes occur
Gather data that has relevance to stakeholders beyond L&D function
Keep data collections methods consistent over business units, regions, and functions to ensure accurate comparisons at the functional level
Ensure that ad hoc data can be collected in a timely manner, so information supporting business decisions is updated
Make sure data collected can be cut for specific business units, functions, or regions so strengths/weaknesses can be pinpointed
Ensure that accurate thresholds are in place to notify management when data indicated declining performance
Keep KPIs/metrics and data collection methods consistent over time for accurate tracking of progress and improvement
From Learning Analytics: Measurement Innovations to Support Employee Development, Mattox and Van Buren, Kogan Page Limited, 2016, p.189
![Page 26: Case Study · Models for Measuring Training 73.7 32.6 14.7 13.7 8.4 26.3 0 10 20 30 40 50 60 70 80 Kirkpatrick's 4 Levels of Evaluation Phillips' ROI Methodology Six Sigma Brinkerhoff's](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8b9d35927900464a5e8f67/html5/thumbnails/26.jpg)
- Connect all data back to original desired business outcomes and organizational goals
- Outline how L&D and stakeholders share same priorities
- Do not assume stakeholders will make accurate assumptions about the data on their own
- Emphasize logical, clear connections between the data and desired business outcomes and organizational goals
- Focus on the central message and avoid including peripheral information
- Explicitly note areas where L&D is creating value for the organization
- Clearly and succinctly outline justification for key findings
- Suggest improvements and potential next steps based on data implications
- Establish productive environment for continuing conversation on L&D progress and improvement
From Learning Analytics: Measurement Innovations to Support Employee Development, Mattox and Van Buren, Kogan Page Limited, 2016, p.191
Set the Scene
Plot Development
Resolution