Beyond Measuring Usage - Tools Beyond Measuring Usage - Tools & Techniques for Demonstrating & Techniques for Demonstrating Program ValueProgram ValueSkillSoft and Knowledge AdvisorsSkillSoft and Knowledge Advisors
Learning: Re-Imagined
Workshop AgendaWorkshop AgendaWorkshop AgendaWorkshop Agenda
• Welcome and Introduction
• Metrics in Times of Economic Uncertainty Jeffrey Berk - Knowledge Advisors
• Metrics That Matter ™ Case Study –CNATom Simon and Mike Castillo - CNA
• SkillSoft Tools & Techniques Heather McCain, Laura Rexford, Lori Turley - SkillSoft
• Wrap up
Economic Uncertainty Emphasizes Human Capital Measurement
Jeffrey Berk
KnowledgeAdvisors
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 4Apr 19, 2023
Measurement in Economic UncertaintyMeasurement in Economic Uncertainty
Agenda
Measurement for Performance Management
Measurement for Performance Management
Business Case for MeasurementBusiness Case for Measurement
© 2008, KnowledgeAdvisors
Bumpy Economic Times
S&P 500 Down Double Digits $700 Billion Bailout DJIA Record Daily Losses Unemployment on the Rise Well Known Brands in
Trouble
© 2008, KnowledgeAdvisors 5Apr 19, 2023
© 2008, KnowledgeAdvisors
What are Business Leaders Saying?
© 2008, KnowledgeAdvisors 6Apr 19, 2023
Source: UK Commission for Employment and Skills, November 10, 2008
• Firms that don’t train are 2.5 times more likely to fail than those that do train.
•“Investing now in building new skills will put us in the strongest position as the economy recovers.”
•“The skills of our people are our best guarantee of future prosperity - - and the best investment a business can make in challenging times.”
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 7Apr 19, 2023
The Economy Changed in 16 Years19920.6:1
20085:1
Tangible Assets
Intangible Assets
MarketValue
MarketValue
Intangible Assets
Tangible Assets
Intangible/Tangible = HCI(Human Capital Index)
66% Productivity Gain in 16 Years
Gary Becker Wins Nobel Prize in Human Capital
S&P 500 Grows at Record Levels
Organizations Invest More in L&D, Leadership, HRIS, LMS, Analytics, Performance Management,
Engagement & Recruiting
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 8Apr 19, 2023
Measuring Talent Maximizes Economic Value
Shareholder Returns of companies investing in L&D measurement have 15.18% higher returns than the S&P 500
Source: Human Capital Measurement and Its Impact on Stock Performance, KnowledgeAdvisors, 2008
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 9Apr 19, 2023
Identify Gaps in Human Capital Processes
PE
RF
OR
MA
NC
E
RE
CR
UIT
ING L
&D
TA
LE
NT
LE
AD
ER
SH
IP
EN
GA
GE
ME
NT
PE
RF
OR
MA
NC
E
Forecast Workforce RequirementsRecruit, Select, and Hire International AssignmentMobile WorkforceEmployee Turnover
On-boardingL&DCoachingKnowledge Mgmt
Competencies Assessment
Succession PlanningLeadership Development
Compensation and BenefitsEmployee SatisfactionEmployee EngagementWork/Life BalanceWorkforce Diversity
Performance Appraisal
© 2008, KnowledgeAdvisors
Case: Talent Gap
04/19/23 © 2008, KnowledgeAdvisors 10
Background: Printing and publishing company
Business Issue: Gaps identified in skilled resources for changing business strategy
Measurement Solution: Provided general skills competency set, collected and reported data w/analysis
Results: Data used for L&D strategy and for recruiting job descriptions.
© 2008, KnowledgeAdvisors
Case: Leadership Evaluation
04/19/23 © 2008, KnowledgeAdvisors 11
Background: Major health benefits provider
Business Issue: Turnover in first 90 days of employment was 25 to 30%
Measurement Solution: Built a leadership program for the Blue Chip hires and used smarter evaluations to measure it and improve it
Results: 93% of Blue Chip new hires were retained and no longer flight risks. 73% reduction in admin burden in the measurement process itself
“Metrics that Matter has provided objective data that supports “gut feelings” about where improvements can be made and where things are going well.”
-Director of Leadership & Associate Development
© 2008, KnowledgeAdvisors
Case: Template Links L&D to Sales
04/19/23 © 2008, KnowledgeAdvisors 12
Background: Consumer Packaged Goods firm
Business Issue: Needed reasonable evidence of sales impact from training
Measurement Solution: Used actual results and reasonable assumptions via impact template
Results: Proved that $74,256/person of sales dollars were tied to a $5,000/person training investment.
Example of template used to showcase ROI from Sales trainingSource: KnowledgeAdvisors
© 2008, KnowledgeAdvisors
Case: ERP Technology Company
Apr 19, 2023 © 2008, KnowledgeAdvisors 13
Background: ERP company performing customer education
Business Issue: Show value of training to customers in terms of increased productivity and quality
Measurement Solution: Collected 12,770 end of program evaluations within 6 months asking participants to forecast impact
Analytics Results: 18.2% estimated increase in productivity from the training (adjusted down for self-reported bias)
Business Results: ProductivityPost
EventTotal % improvement in productivity including training 51.0%Training's contribution to improved productivity 57.0%Percent improvement due to training 29.1%Adjustment factor for confidence in estimations 65.0%Adjusted percent improvement due to training 18.9%
Business Results: QualityPost
EventTotal % improvement in quality, including training 54.0%Training's contribution to improved quality 57.0%Percent improvement due to training 30.8%Adjustment factor for confidence in estimations 65.0%Adjusted percent improvement due to training 20.0%
Business Results: Customer SatisfactionPost
EventTotal % improvement in customer satisfaction including training 52.0%Training's contribution to improved customer satisfaction 57.0%Percent improvement due to training 29.6%Adjustment factor for confidence in estimations 65.0%Adjusted percent improvement due to training 19.3%
Analyze & monetize this result
Analyze & monetize this result
Analyze & monetize this result
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 14Apr 19, 2023
Measurement in Economic UncertaintyMeasurement in Economic Uncertainty
Agenda
Measurement for Performance Management
Measurement for Performance Management
Business Case for MeasurementBusiness Case for Measurement
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 15Apr 19, 2023
Fact
1. Most organizations invest 1% to 4% of revenue in talent development source ASTD stated L&D is 2 to 3% of revenue
2. Most organizations waste 50% or more of their talent development expenses source Metrics that Matter 600,000 follow up data points is 55%, IDC sites
50% waste, Brinkerhoff cites 80% or more waste
3. Most organizations waste .5% to 2% of revenue on ineffective talent development calculation from item 1 x 2 above
4. Effective investments in predictive analytics increases performance by 2.4% source IDC
Talent Measurement Results: Increases Performance * Reduces Waste * Slashes Admin Burden
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 16Apr 19, 2023
Fact
According to IDC, programs that use predictive analytics solutions, such as Metrics that Matter, generate 60% better returns than those that don’t.
Example: If one invests $1,000,000 in an L&D program that generates a $1,000,000 profit, without using a predictive measurement solution, effective use of one, could have generated an additional $600,000 in profit.
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 17Apr 19, 2023
Measurement Impact on Increased Performance
1. Major services firm realized $200,000 increased annual profit by using Metrics that Matter predictive analytics capabilities.
2. Through measurement a leading federal agency significantly increased output and business impact without increasing budget over a 3 year period.
3. Major telecommunications firm using Metrics that Matter was able to increase sales by $1.25 million dollars through better supply chain management.
4. 15,000 hours in improved productivity by a floral company in identifying appropriate training investments.
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 18Apr 19, 2023
Measurement Impact on Reduced Waste1. 25 retail stores avoided $100,000 of wasted training. Retail store training
measured after first of six weeks saw 1 set of ineffective trainers and discontinued using them.
2. Instructor compared herself to other instructors and changed approach. 6 months of additional sessions performed better to avoid at least $100,000 in scrap learning.
3. $500,000 in funding gained by measuring scrap learning. Services firm measured training results and found high quality but low impact. The program was retained and additional funding rewarded to consult with managers on increasing impact.
4. 88% of learners not applying training is NOT due to bad training. Waste is due to unsupportive employee environment. A billion dollar company can save at least $1,000,000 if it can reduce waste by just 20% by performance managing the employee environment.
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 19Apr 19, 2023
Measurement Impact on Reduced Admin
1. 75% reduction in administration costs from automating measurement process vs. manual measurement.
2. 15 seconds to retrieve actionable data for decision-making vs. 15 hours prior to automating the measurement process. This automation of 1 report alone saved over $36,000 per year.
3. $400,000 savings by major accounting firm that used smarter measurement in a practical way to yield similar results compared to complex and burdensome alternative measurement techniques
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 20Apr 19, 2023
Calculate Performance Gain
$1,000,000,000
Revenue
25% Labor Percentage
$250,000,000 Labor Cost
1%Increased Productivity from Measurement%
$2,500,000 Increased Performance from Measurement
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 21Apr 19, 2023
Calculate Waste Reduction
Description
$1,000,000,000
Revenue
1% L&D as % of Revenue
$10,000,000 L&D Investment
50% Waste or Unrealized Value %
$5,000,000 Waste or Unrealized Value
5%Waste Reduction from Measurement%
$250,000Waste Reduction from Measurement
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 22Apr 19, 2023
Calculate Admin Savings
Description
$55,000 Salary of 1 FTE admin for metrics
$50,000Cost avoided from unnecessary analysis
$50,000Cost to replication fraction of MTM benchmarks
75%Admin savings from MTM measurement and benchmarks
$116,250Administration Savings from Measurement
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 23Apr 19, 2023
Total Value
Value Summary
$2,500,000 Performance Gain
$250,000 Waste or Scrap Avoided
$116,250 Administration Savings
$2,866,250
Total Quantified Value
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 24Apr 19, 2023
Measurement in Economic UncertaintyMeasurement in Economic Uncertainty
Agenda
Measurement for Performance Management
Measurement for Performance Management
Business Case for MeasurementBusiness Case for Measurement
© 2008, KnowledgeAdvisors
Talent and L&D Measurement Tips
Identify and Audit Gaps in Human Capital Processes
Build a Robust Evaluation Process
Track and Link to Actual Business Results
Reduce Errors and Scrap Leverage Technology Use Metrics for
Performance Management
© 2008, KnowledgeAdvisors 25Apr 19, 2023
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 26Apr 19, 2023
Benchmarks for Performance Management
Benchmarks: Internal or External
Provide a point of referenceMotivate by exampleAvailable thru Metrics that Matter
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 27Apr 19, 2023
Goals for Performance Management
Goals: Actual to Goal
Set in Metrics that MatterMove in right directionChallenging yet attainable
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 28Apr 19, 2023
Trends for Performance ManagementTrends: Time based analysis
Available in Metrics that MatterShows improvement over time
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 29Apr 19, 2023
Dashboards for Performance Management
Dashboards: •Concise Data•Summary Data•Visual View•Available in Metrics that Matter for course managers, instructors, impact and operations
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 30Apr 19, 2023
A template for the learning team to conduct more comprehensive business impact and ROI analysis. Templates bring stewardship and collaborative guidance to the process of linking training to results.
Track & Link to Actual Business Results
© 2008, KnowledgeAdvisors
Build a Robust Evaluation Process
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 32Apr 19, 2023
Measurement in Economic UncertaintyMeasurement in Economic Uncertainty
Agenda
Measurement for Performance Management
Measurement for Performance Management
Business Case for MeasurementBusiness Case for Measurement
© 2008, KnowledgeAdvisors© 2008, KnowledgeAdvisors 33Apr 19, 2023
Experts in metrics that
matter
Experts in metrics that
matter KnowledgeAdvisors is a human capital analytics firm. We are the world’s largest provider of learning and talent measurement solutions.
KnowledgeAdvisors
© 2008, KnowledgeAdvisors
KnowledgeAdvisors Clients
04/19/23 © 2008, KnowledgeAdvisors 34
© 2008, KnowledgeAdvisorsApr 19, 2023 Page 35
www.knowledgeadvisors.com
For further information on the information contact:
Jeffrey Berk
Chief Operating Officer
+1 312 676 4411 (phone)
+1 312 676 4401 (fax)
Measurement at CNA
Measurement at CNATom Simon & Michael Castillo Area of Presenter
Measurement
CNA At A Glance
•1 million business and professionals insured•7th largest commercial lines insurer•U.S. and international operations•9,000 employees•$7.8 billion in revenues•Assets of $52 billion•“A” rated for financial strength by A.M. Best
Measurement
Measurement
Measurement
Measurement
Measurement
Why ROI?Why ROI?Why ROI?Why ROI?
Benefits of ROI:
• Business cases benefit strongly from ROI
• Measuring financial impact helps illustrate importance of learning
• Value-based measurements support Learning Growth Model progression
Challenges of ROI:
• Limited time and resources
• Hard to evaluate ROI for intangibles (learning)
• Securing internal data difficult
SkillSoft Measurement ToolsSkillSoft Measurement ToolsSkillSoft Measurement ToolsSkillSoft Measurement Tools
• EZ ROI Calculator - “soft measure”
• Business Impact Analysis – Validated by internal survey
• Value Measurement Toolkit (VMT) – ROI, Phillips Level 5
Business Business Impact Impact
AnalysisAnalysis
EZ ROI EZ ROI CalculatorCalculator
ValueValue Measurement Measurement
ToolkitToolkit
time, resources, data involvedtime, resources, data involved
estimate vs. rigorous measureestimate vs. rigorous measure
EZ ROI CalculatorEZ ROI CalculatorEZ ROI CalculatorEZ ROI Calculator
Business Business Impact Impact
AnalysisAnalysis
EZ ROI EZ ROI CalculatorCalculator
ValueValue Measurement Measurement
ToolkitToolkit
time, resources, data involvedtime, resources, data involved
estimate vs. rigorous measureestimate vs. rigorous measure
Objectives and AttributesObjectives and AttributesObjectives and AttributesObjectives and Attributes
Objectives:• Deliver more value-based metrics • Help strengthen business cases• Prospective use (Should we train?) and retrospective use
(How did we do?)
Attributes:• Easy to use• Requires little time and very little data • Defensible • Immediate analysis• Cost displacement and performance improvement measures
EZ ROI CALCULATOREZ ROI CALCULATOREZ ROI CALCULATOREZ ROI CALCULATOR
Components:
• Introduction and assumptions
• Cost displacement calculator
• Cost displacement sample
• Productivity impact calculator
• Productivity impact sample
Microsoft Excel Worksheet
Application Ideas for EZ ROI CalculatorApplication Ideas for EZ ROI CalculatorApplication Ideas for EZ ROI CalculatorApplication Ideas for EZ ROI Calculator
• SkillSoft versus another learning option
• Monetary value of productivity gains
• Limited licenses? Select wisely
Business Impact AnalysisBusiness Impact AnalysisBusiness Impact AnalysisBusiness Impact Analysis
Business Business Impact Impact
AnalysisAnalysis
EZ ROI EZ ROI CalculatorCalculator
ValueValue Measurement Measurement
ToolkitToolkit
time, resources, data involvedtime, resources, data involved
estimate vs. rigorous measureestimate vs. rigorous measure
Objectives and AttributesObjectives and AttributesObjectives and AttributesObjectives and Attributes
Objectives:
• Measure satisfaction, determine motivation and determine job impact by the following:
• Determine individual skill requirements and needs• Evaluate program satisfaction• Gauge skill improvement (based on individual
responses/group roll-up) to determine productivity gains
Attributes:• Easy to use • Company-specific • Requires limited time and data• Longer timeline• Stronger measurement story• Potential testimonials
Business Impact AnalysisBusiness Impact AnalysisBusiness Impact AnalysisBusiness Impact Analysis
Components
• Custom survey
• Communications to launch the survey
• Summary of feedback collected
• Impact/Results Summary
Microsoft Excel Worksheet
Sample Survey QuestionsSample Survey QuestionsSample Survey QuestionsSample Survey Questions
1.Job Category Drop Down of Job Families
2.Salary Range Drop Down of Salary Ranges
3.Would you recommend the SkillSoft resources to a colleague?
Select One(Yes, Somewhat, No)
4.What was the main reason why you accessed SkillSoft training?
Drop down with list of likely options plus an “other” option that has a blank field
5.Primarily, what skill were you seeking to develop? Open short text field
6.What percent of your job requires you to use this skill?
Drop down with a range of percentages(0%, 1-10%, 11-20%, 21-30%, 31-40%, 41-50%,
51-60%, 61-70%, 71-80%, 81-90%, 91-100%)
7.As a result of the training, how much of a skill increase did you experience?
Drop down with a range of percentages(0%, 5-10%, 11-15%, 15-20%, 21-25%, 26-30%,
More than 30%)
8.If you did not have access to SkillSoft resources, what alternatives would you have considered?
Drop down of common training types including “no alternatives” and “other” options
9.In approximate terms, how much do you think the alternate option would have cost?
Open numerical field, includes the options “no cost” and “not sure”
10.When are you planning to use SkillSoft resources again?
Select One (Not sure, within 2 weeks, within 30 days, within 90
days, more than 90 days)
Output of SurveyOutput of SurveyOutput of SurveyOutput of Survey
• Survey Results document including:• Background• Survey Responses (graphical representation)• Comment Highlights• ROI Impact
Application Ideas for Business Impact Application Ideas for Business Impact AnalysisAnalysis
Application Ideas for Business Impact Application Ideas for Business Impact AnalysisAnalysis
• Translate productivity gains into monetary terms specific to your company
• Utilize survey results for ongoing marketing and increased usage
• Create targeted programs based on skill needs identified
Value Measurement ToolkitValue Measurement ToolkitValue Measurement ToolkitValue Measurement Toolkit
Business Business Impact Impact
AnalysisAnalysis
EZ ROI EZ ROI CalculatorCalculator
ValueValue Measurement Measurement
ToolkitToolkit
time, resources, data involvedtime, resources, data involved
estimate vs. rigorous measureestimate vs. rigorous measure
Objectives and AttributesObjectives and AttributesObjectives and AttributesObjectives and AttributesObjectives:
• Measure the value derived from the implementation and ongoing support of learning initiatives.
• A method offered to aid clients to establish a process for measurement.
• Deliver value in terms your stakeholders will believe in – cot savings, productivity impact, increased revenue, etc.
Attributes:• Longer term study based on a defined process• Requires internal, validated data• Defensible and credible due to stakeholder input• Return on investment
Notice that the 5 levels are not each a step in the process. The process can be used to any level.
The Value Measurement ProcessThe Value Measurement ProcessThe Value Measurement ProcessThe Value Measurement Process
• White Paper
• Readiness Worksheet
• Planner
• Calculator
• Case Studies
Value Measurement ToolsValue Measurement ToolsValue Measurement ToolsValue Measurement Tools
Microsoft Excel Worksheet
Microsoft Word Document
Application Ideas for Value Measurement Application Ideas for Value Measurement Application Ideas for Value Measurement Application Ideas for Value Measurement
• Validate the investment in technology enabled learning
• Meet stakeholder expectations/shape expectations
• Measuring key strategic programs with high visibility
Hands-On ExerciseHands-On ExerciseHands-On ExerciseHands-On Exercise
• Divide into teams
• Review your scenario
• Agree on appropriate tool
• Be prepared to present considerations for tool selection
Thank you for participating!Thank you for participating!Thank you for participating!Thank you for participating!