knowledgeadvisors and metrics that matter a leading ... articles/brandon hall... ·...

16
BERSIN & ASSOCIATES © 2011 BERSIN & ASSOCIATES Research Bulletin | 2011 THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY. Introduction : Talent Management Continues to Grow Integrated talent management continues to be one of the most important strategies in business today. Companies are integrating their recruiting, talent development, succession and leadership strategies to drive growth, productivity, and employee retention and engagement. As our Talent Management Framework 1 illustrates (see Figure 1), these strategies include many initiatives and potential areas on which to focus. Retailers, for example, which deal with high-volume recruiting and high turnover rates, focus heavily on assessment, talent acquisition and onboarding strategies to drive their business. Professional services firms focus on deep skills development, knowledge-sharing and organizational collaboration to drive strong levels of expertise and client value. All organizations understand that investment in leadership development at all levels is core to business success. 2 1 For more information, The Talent Management Framework: A Modern Approach for Developing and Mobilizing Talent, Bersin & Associates / Josh Bersin, Stacey Harris, Kim Lamoureux, Madeline Laurano and David Mallon, May 2010. Available to research members at www.bersin.com/library . 2 For more information, High-Impact Leadership Development, Bersin & Associates / Laci Loew and Stacia Sherman Garr, October 2011. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/hild. November 29, 2011 Volume 6, Issue 59 BERSIN & ASSOCIATES, LLC 180 GRAND AVENUE SUITE 320 OAKLAND, CA 94612 (510) 251-4400 [email protected] WWW.BERSIN.COM KnowledgeAdvisors and Metrics that Matter TM : A Leading Platform for Learning and Talent Measurement About the Author Josh Bersin, Principal Analyst

Upload: nguyenkhanh

Post on 03-May-2018

217 views

Category:

Documents


1 download

TRANSCRIPT

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

Introduction : Talent Management Continues to Grow

Integrated talent management continues to be one of the most

important strategies in business today. Companies are integrating their

recruiting, talent development, succession and leadership strategies to

drive growth, productivity, and employee retention and engagement.

As our Talent Management Framework1 illustrates (see Figure 1), these

strategies include many initiatives and potential areas on which to

focus. Retailers, for example, which deal with high-volume recruiting

and high turnover rates, focus heavily on assessment, talent acquisition

and onboarding strategies to drive their business. Professional services

firms focus on deep skills development, knowledge-sharing and

organizational collaboration to drive strong levels of expertise and

client value. All organizations understand that investment in leadership

development at all levels is core to business success.2

1 For more information, The Talent Management Framework: A Modern Approach for Developing and Mobilizing Talent, Bersin & Associates / Josh Bersin, Stacey Harris, Kim Lamoureux, Madeline Laurano and David Mallon, May 2010. Available to research members at www.bersin.com/library.

2 For more information, High-Impact Leadership Development, Bersin & Associates / Laci Loew and Stacia Sherman Garr, October 2011. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/hild.

November 29, 2011 Volume 6, Issue 59

BERSIN & ASSOCIATES, LLC180 GRAND AVENUE

SUITE 320OAKLAND, CA 94612

(510) [email protected]

KnowledgeAdvisors and Metrics that MatterTM: A Leading Platform for Learning and Talent Measurement

About the Author

Josh Bersin,Principal Analyst

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 2

Training and Capability Development Play a Foundational Role

As our Framework points out, training, coaching and what we call,

“capability development,” form a foundation to almost every piece of

the talent management puzzle. In fact, our research shows that, among

all the various talent management areas on which to focus, excellence

in development planning, career development and training drive the

highest value of all.3 Companies that excel in these areas actually drive

almost twice the revenue per employee over those which do not invest

in training and development planning.

3 For more information, Talent Management Factbook 2010: Best Practices and Bench-marks in U.S. Talent Management, Bersin & Associates / Karen O’Leonard and Stacey Harris, September 2010. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/tmfactbook.

Source: Bersin & Associates, 2010.

Figure 1: Bersin & Associates Talent Management Framework®

Companies that excel

in development

planning, career

development and

training drive almost

twice the revenue

per employee over

those which do not.

KEY POINT

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 3

The spending on training and development is large. Companies spend

close to $50 billion a year in all areas of training and development

(around $680 per employee on average in 2010)4, and this investment

goes into a wide array of programs – onboarding, functional and

technical training, leadership development, certificate programs,

employee skills assessment, coaching, compliance training, performance

support, and social and informal learning. When any manager tries to

improve the performance of an employee, ultimately his / her challenge

is to assess performance, coach the employee to fill gaps and train them

to perform better.

So while training itself is often run by a training department, or

learning and development team, the work that training does is diverse

and broadly used throughout the organization. Our research continues

to show high-impact learning organizations outperform their peers

through their focus on the tools, technologies, skills and organizational

structure of their training function.

Training and Capability Measurement Continue to be a Challenge

Despite the proliferation of learning management systems (LMSs) and

talent management software platforms in the market5, companies

still struggle to measure the effectiveness, efficiency and alignment

of training. Because training is a highly customized and somewhat

“indirect” driver of business impact, there are many models for training

measurement (i.e., the Bersin Impact Measurement Model6, the

4 For more information, The Corporate Learning Factbook® 2011: Benchmarks, Trends and Analysis of the U.S. Training Market, Bersin & Associates / Karen O’Leonard, January 2011. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/factbook.

5 For more information, Talent Management Systems Buyer’s Guide, Bersin & Associates / Josh Bersin and Katherine Jones, November 2011. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/tms.

6 For more information, please read the following two publications: (1) High-Impact Talent Management: Trends, Best Practices and Industry Solutions, Bersin & Associates / Josh Bersin, May 2007, available to research members at www.bersin.com/library or for purchase at www.bersin.com/hitm; and, (2) The Training Measurement Book: Best Practices, Proven Methodologies, and Practical Approaches, Josh Bersin / Pfeiffer, 2008, available for purchase at www.bersin.com.

In 2010,

companies spent

close to $50

billion in all areas

of training and

development,

with around $680

per employee on

average in 2010.

KEY POINT

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 4

Kirkpatrick Model7, the Success Case Methodology8 and many others).

Our research shows that only six percent of HR organizations rate

themselves “highly skilled” at talent measurement, and 56 percent rate

themselves “poor” at understanding their organization’s skills and what

programs best drive business value.9

Not only is training measurement imperfect and complex, there are

many tools and systems to use. In order to adequately measure and

monitor capability development programs, organizations must use

survey tools and assessments – and then match this data against many

data elements in the LMS and HR management systems. So while

nearly every learning and HR software vendor sells a measurement and

analytics module to their systems, companies typically tie together as

many as seven different systems to build an integrated approach to

training measurement.10

Enter KnowledgeAdvisors: A Company Laser Focused on Learning and Talent Measurement

Enter KnowledgeAdvisors. Founded in 1999, this company developed a

platform and related services to help organizations to solve the puzzling

problem of creating an integrated solution for learning measurement.

KnowledgeAdvisor’s product, Metrics that MatterTM, was originally

designed as a highly scalable system to distribute employee surveys

and assessments to people, and then collect, analyze and manage this

information around a given training program and across the entire

training function.

7 Donald Kirkpatrick’s four-level Measurement Model has been widely published in many articles and its terminology is well-known to most training professionals. The origi-nal model was published in Training and Development Handbook, R. L. Craig, McGraw-Hill, 1976.

8 Source: The Success Case Method: Find Out Quickly What’s Working and What’s Not, Robert O. Brinkerhoff, Ed.D. / Berrett-Koehler Publishing Inc., 2003.

9 For more information, The High-Impact HR Organization: Top 10 Best Practices on the Road to Excellence, Bersin & Associates / Stacey Harris, January 2011. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/hihr.

10 For more information, Talent Management Factbook 2010: Best Practices and Benchmarks in U.S. Talent Management, Bersin & Associates / Karen O’Leonard and Stacey Harris, September 2010.

Our research

shows that only

six percent of HR

organizations rate

themselves “highly

skilled” at talent

measurement, and

56 percent rate

themselves “poor”

at understanding

their organization’s

skills and what

programs best drive

business value.

KEY POINT

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 5

Because the company originally focused primarily on the measurement

of training, KnowledgeAdvisors built a wide array of reports,

dashboards and tools to make training measurement easy. The system

includes many predefined measurement templates that implement

the most modern training measurement models in the market. Data is

collected and aggregated precisely the way a CLO or training manager

wants to see it; reports and dashboards give users the ability to measure

program efficiency and effectiveness, manager satisfaction and value,

learning transfer, and a wide variety of other measures. Since training

measurement has a number of workflow elements (e.g., end-of-

course surveys, follow-up surveys, manager surveys, etc.), the platform

is also designed to address all of the various models for collecting

measurement data.

As KnowledgeAdvisors grew, the company started to realize that, if

customers used standard assessment questions, they could benchmark

various learning measures against other companies. So over the

years, KnowledgeAdvisors has built a large database of training-

related metrics which let companies compare their training programs’

effectiveness against similar programs in companies of their size and

industry. This type of benchmarking may not specifically tell a company

what they are doing well or poorly, but it gives them tremendous

insights into what possible elements of a training program they may

need to radically improve.

As the company’s customer base grew, KnowledgeAdvisors realized that

businesses need much more than a tool set – they need advice, standard

reports and consulting to build the right training and capability

measurement solution. So KnowledgeAdvisors built up its consulting

team and worked closely with companies, like Bersin & Associates,

to identify new models for measurement and to help their clients

implement them. Today KnowledgeAdvisors and Bersin and Associates

have worked together at many large companies to help organizations

build an end-to-end training and talent measurement strategy.

Our experience with KnowledgeAdvisors over the years has shown

that the company’s platform and services provide an invaluable service

to any midsize or large organization that wants to better understand

how to understand, manage, improve and benchmark its training

Our experience with

KnowledgeAdvisors

over the years has

shown that the

company’s platform

and services provide

an invaluable

service to any

midsize or large

organization that

wants to better

understand how

to understand,

manage, improve

and benchmark its

training investment.

A N A LY S I S

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 6

investment. Cognizant, a global IT consulting firm, standardized on

Metrics that Matter for its corporate university which trained more than

100,000 employees in 2010. Cognizant now uses KnowledgeAdvisors’

product as a standard tool to measure feedback, learning effectiveness

and blended-learning design for all courses – and has improved its

learning effectiveness measures by more than 10 percent in the last

year. (Read our case study on Eaton University’s implementation of

Metrics that Matter for more details on how KnowledgeAdvisor’s

training measurement solutions work.11)

Evolving into a SaaS12 Talent Measurement Solution

As organizations have evolved their end-to-end talent management

strategies, they are now integrating the L&D function into many more

areas of HR. High-impact learning organizations typically create a

“federated” model13 for L&D – in which the team supports not only

operational training and skills development, but a wide variety of other

areas of the business.14 This includes providing support for onboarding,

leadership development, sales enablement, customer training and

general performance consulting. So, while “training measurement”

continues to be an important and highly complex problem,

KnowledgeAdvisor’s platform can be extended to measure these other

areas, as well.

11 For more information, Eaton University: Training Analytics Excellence in a Corporate University, Bersin & Associates, April 2004. Available to research members at www.bersin.com/library.

12 “Software as a Service” (or SaaS) refers to the business of selling software over the Internet as a web service. In this business model, the software vendor charges an annual “rental” fee and can predict recurring revenues far more reliably than with the licensed software model. As SaaS becomes proven and the delivery strategies more refined, most analysts agree that more and more enterprise software will be sold in this way.

13 A “federated model” has a small core team that manages some technology and corporate programs, and empowers business and functional units to run their own training programs.

14 For more information on the federated model, The High-Impact Learning Organization: WhatWorks® in the Management, Governance and Operations of Modern Corporate Training, Bersin & Associates / Josh Bersin, May 2008. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/highimpact.

Three of

KnowledgeAdvisors’

most recent

extensions focus

on the areas

of onboarding,

leadership

development and

sales training.

K E Y P O I N T

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 7

Over the last several years, the company has extended its solution

to provide more specific application-level measurement solutions,

leveraging the same technology, workflow and benchmarking available

in its core training offering. Three of KnowledgeAdvisors’ most recent

extensions focus on the areas of onboarding (a critical part of talent

acquisition), leadership development and sales training.

• Onboarding – Since onboarding is essentially a workflow-based

training process, it has many of the same characteristics of a training

program. Employee onboarding requires a high degree of content

and workflow management and tools. (Vendors such as Taleo,

Oracle / PeopleSoft and SilkRoad offer very powerful automation

systems to manage the entire onboarding process.) But how do you

measure the effectiveness of various onboarding programs? How do

you determine the level of readiness of new employees when they

finish? Even more importantly, how do you continuously monitor

the effectiveness of the company’s onboarding after the initial

program is completed?

KnowledgeAdvisors now offers a solution in this area. Using much

of the same underlying infrastructure for training measurement,

KnowledgeAdvisors has now built a set of dashboards, reports

and models to help companies measure onboarding effectiveness

and impact.

Onboarding typically generates a very high return on investment –

but, since it is not formal training, it is often difficult to measure.

One of KnowledgeAdvisors’ clients showed 15,000 hours of

increased productivity to the business as a result of the increased

speed to competency. This is the type of analysis which can help to

both justify the program and also find areas for improvement.

• Leadership Development – Leadership development is a special

type of training process. It often includes a highly blended model,

which includes 360-degree assessments15, action learning16, formal

15 A “360-degree assessment” incorporates feedback from the worker, and his / her man-ager, peers, superiors, subordinates and customers.

16 “Action learning” is a process in which groups of learners collaborate to solve actual workplace problems. In this way, organizations benefit from gaining solutions to critical

Onboarding

typically generates

a very high return

on investment –

but, since it is not

formal training, it

is often difficult to

measure.

K E Y P O I N T

No two leadership

development

programs are

identical.

K E Y P O I N T

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 8

classroom and e-learning, developmental assignments, special

projects, simulations, books and special reading assignments, and

competency-based assessments. No two leadership development

programs are identical; companies change and improve these

programs regularly, and often use third-party consultants and

content on a regular basis. Again, KnowledgeAdvisors has extended

its platform, reports and dashboards to give companies easy-to-use

tools to measure the satisfaction, efficiency and effectiveness of

leadership development. Applying the concepts developed for the

training industry, KnowledgeAdvisors gives customers easy-to-use

reports to measure satisfaction, job impact, alignment, efficiency

and follow-up measures to quantify actual results.

Executive and leadership development is particularly hard

to measure. Many of our clients invest heavily in leadership

development but find that budgets are cut during tough times. The

KnowledgeAdvisors system would not only help to maintain that

ongoing investment, but would also give the program manager

detailed feedback on the satisfaction and effectiveness of each

element of the program across a broad range of leaders.

• Sales Training – Sales training, like leadership development, is

one of the most universal and important development investment

areas in talent management. Like any other form of training,

organizations want to measure the effectiveness, efficiency, business

alignment and impact of sales training, so that these programs can

be continuously improved. KnowledgeAdvisors has built special

reports and dashboards to make sales training easy to measure and

to help companies benchmark their sales training programs against

peers in their industry.

Sales training is also a program that we know organizations

need, but is often difficult to cost-justify directly. By using a tool

like Metrics that Matter, companies can diagnose different

modalities, audiences and geographies, as well understand which

elements work best and which elements of the program need the

most improvement.

challenges and participants benefit by learning from their experiences.

Companies change

and improve

these programs

regularly, and

often use third-

party consultants

and content on a

regular basis.

K E Y P O I N T

Sales training is also

a program that we

know organizations

need, but is often

difficult to cost-

justify directly.

K E Y P O I N T

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 9

Figure 2: Where KnowledgeAdvisors Impacts in Talent Management Areas

Source: Bersin & Associates, 2011.

The Value of a Single Measurement Platform across Many Types of Programs

All of the solution areas discussed here (including many other training-

related applications like customer service training, technical training,

etc.) are built in different ways. Some are highly instructor-led, some are

blended, some are highly informal or online, and some use many forms

of content. It is very inefficient (and usually impossible) for a company

to put in place separate measurement programs, tools and back-end

systems for each.

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 10

Our High-Impact Learning Measurement17 research (and also The

Training Measurement Book®18) describes a complete model for

measurement of any program – but as we point out in our research,

once you build a model you still need a way to manage this

measurement program. Surveys must be developed and they must be

deployed – and then lots of information must be collected, filtered,

sorted and analyzed. Ultimately, what makes measurement useful is

having the time to analyze the data (i.e., to view it by program, by

audience group, by program element, etc.).

KnowledgeAdvisors’ system can become this single measurement

platform for all of the various different types of development programs

in a company. Because the system has a standard toolset to build

surveys, develop reports and benchmark programs against many other

companies, organizations are beginning to “standardize” their talent

development measurement on the KnowledgeAdvisors platform.

Extensible beyond the LMS

How does KnowledgeAdvisors’ platform compare to a company’s

LMS? It turns out that most large companies (more than 70 percent

according to our research) have some type of learning management

system today and ideally this platform is designed to be the single

repository for training information.19 While LMS systems have built-in

testing and assessment tools, they have not been finely crafted with

the templates, reports, dashboards and SaaS-enabled benchmarks

available in Metrics that Matter™. In fact, during the last few years,

LMS vendors have shifted their technologies so heavily toward talent

17 For more information, High-Impact Learning Measurement: Best Practices, Models, and Business-Driven Solutions for the Measurement and Evaluation of Corporate Training, Bersin & Associates / Josh Bersin, November 2006. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/measurement.

18 For more information, The Training Measurement Book: Best Practices, Proven Methodologies, and Practical Approaches, Josh Bersin / Pfeiffer, 2008, available for purchase at www.bersin.com.

19 For more information, Learning Management Systems 2011: The Definitive Buyer’s Guide to the Global Market for Learning Management Solutions, Bersin & Associates / David Mallon, December 2010. Available to research members at www.bersin.com/library or for purchase at www.bersin.com/lms.

According to our

research, more than

70 percent of large

companies have

some type of LMS

today and ideally

this platform is

designed to be the

single repository

for training

information.

KEY POINT

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 11

management that most have actually reduced their investments in

training measurement tools.

This gives KnowledgeAdvisors an even more important role in the

market. As companies build more diverse types of training programs

(e.g., informal and social learning, simulations, and gaming, as well

as e-learning and mobile applications), the need for an extensible,

easy-to-use measurement platform becomes bigger. Today, many

large organizations (such as NetworkAppliance, Defense Acquisition

University and Deloitte) use KnowledgeAdvisors in complement with

their learning management systems. The LMS manages employee

enrollment, course administration, and tracking and reporting

information – KnowledgeAdvisors serves as the “training analytics”

system, giving the organization all of the information it needs to

measure program-level effectiveness, efficiency, alignment and impact.

In these companies, KnowledgeAdvisors’ platform itself is the “training

analytics” solution and is totally complementary to the tools available

in most LMS systems. Data is often summarized and extracted from

KnowledgeAdvisors, and then moved into other HR analytics tools for

broader analysis.

Creating Standards for Training Measurement: TDRP – Talent Development Reporting Principles

This past year, KnowledgeAdvisors took the lead to pull together a

variety of organizations (including Bersin & Associates) to create an

open standard for the reporting of training and talent development

metrics. This program is called TDRP.20

Think for a minute about the accounting profession. There are

“generally accepted accounting principles (GAAP)” that spell out what

an income statement, balance sheet and cashflow statement should look

like. When finance professionals go about their work, they can capture

and analyze information, and then present it in a standard format.

20 For more information on the TDRP standards and program, please see the following report which includes am introduction by Josh Bersin, TDRP – The Beginning of Standards in Training Measurement, Bersin & Associates / Josh Bersin, July 21, 2011. Available to research members at www.berisn.com/library.

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 12

TDRP is trying to do this for the training profession. Today, TDRP

founding members include Qualcomm, Deloitte, Lowes, Lockheed

Martin, Cigna, NetApp, Sodexo, BP, CNA Insurance and ADP.

The initiative has focused on the following two major areas.

• First, create a set of standard “outcome statements” – similar

to GAAP. These include a business outcome statement, training

effectiveness statement and training efficiency statement.

◦ The business outcome statement shows how various L&D

investments tie to business outcomes.

◦ The effectiveness statement summarizes results in terms of

satisfaction, effectiveness, utility and alignment (some of

these measures come from the Bersin & Associates Impact

Measurement Framework – see Figure 3).

◦ The efficiency statement standardizes measures of cost,

volumes and utilization.

• Second, create a set of definitions (still a work in progress), so

that someone new to training measurement can review this to

understand what these various terms mean.

In the Bersin & Associates Impact Measurement Framework®, we find

four key elements to training measurement:

1. Effectiveness – How well does the L&D program or investment

actually work as designed;

2. Efficiency – How well is the L&D team using its resources efficiently

to build the right types of programs in the right blend;

3. Alignment – How well does the L&D investment align with current

urgent business priorities; and,

4. Impact – What real business impact is being driven by the programs.

We identified nine measurement areas – and recently added three more to

cover the effectiveness, efficiency and alignment of informal learning.21

21 For more information, The Impact Measurement Framework: An Update for the Measurement of Informal Learning, Bersin & Associates / Josh Bersin, May 3, 2011. Available to research members at www.bersin.com/library.

In the Bersin &

Associates Impact

Measurement

Framework, we

identified nine

measurement

areas – and recently

added three

more to cover

the effectiveness,

efficiency and

alignment of

informal learning.

KEY POINT

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 13

TDRP, which is an open initiative, borrows from some of this work to

build a set of “standards” that companies can use to get started with

their reporting programs.

Note that TDRP is a set of “reporting” principles, not “measurement”

principles. The idea of TDRP is to help companies create standard

reports, so that tools like Metrics that Matter (and others) can deliver

standard reports which business people can start to understand. This

makes TDRP very valuable to CLOs and training managers because,

if it succeeds, we can all compare more standardized outputs across

different organizations and different systems.

TDRP does not replace or compete with the Bersin & Associates

Measurement Framework (or any other model for that matter), because

it does not specify all the training measures a company needs, nor

does it try to describe how these measures are captured or many of the

details of an ongoing measurement program.

Figure 3: Bersin & Associates Impact Measurement Framework®

Source: Bersin & Associates, 2011.

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 14

As one of the leaders in training measurement (and by far the most

active company in this highly fragmented area), KnowledgeAdvisors is

taking on a very important role in making TDRP come true – and we do

hope that, over time, more and more companies see TDRP as a standard

way of reporting training effectiveness, efficiency and impact.

Summary: Measurement Really Matters

Because training and HR are somewhat “soft” areas of business,

organizations continually look for better ways to measure effectiveness,

efficiency and impact. Our research shows that companies do not

“implement” measurement programs – rather they “evolve” them. Early

on, companies start measuring things and they find that some of these

measures are highly effective and others are not, and then they improve

the measurement over time.

Our industry studies on The High-Impact Learning Organization22

and The High-Impact HR Organization23 both looked at the maturity

of measurement programs in many companies. What both of these

research programs found is that organizations which buckle down

and focus on measurement year after year significantly improve their

overall impact. As the old maxim goes, “You can’t improve what you

can’t measure,” which applies heavily to all areas of human resources.

KnowledgeAdvisors understands this – and focuses on giving companies

the tools they need to start fast and grow their measurement

sophistication over time.

In summary, this bulletin is designed to help organizations to better

understand the KnowledgeAdvisors offering, and to give readers the

Bersin & Associates perspectives on the company’s product, strategy and

direction. We believe that KnowledgeAdvisors plays a vital role in the

measurement of training and other talent development programs, and

that the company will continue to grow.

22 For more information, The High-Impact Learning Organization: WhatWorks® in the Management, Governance and Operations of Modern Corporate Training, Bersin & Associates / Josh Bersin, May 2008.

23 For more information, The High-Impact HR Organization: Top 10 Best Practices on the Road to Excellence, Bersin & Associates / Stacey Harris, January 2011.

Our research shows

that companies do

not “implement”

measurement

programs – rather

they “evolve”

them.

KEY POINT

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 15

While there are many talent platforms in the market (i.e., LMS

companies, talent management software companies, HRMS companies),

none do what KnowledgeAdvisors does. The company’s product is

highly complementary to these systems, and forms the basis for a

standard talent development measurement platform across many

training and development applications.

We believe that

KnowledgeAdvisors

plays a vital role in

the measurement of

training and other

talent development

programs, and that

the company will

continue to grow.

A N A LY S I S

BERSIN & ASSOCIATES © 2011

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES, LLC6114 LA SALLE AVENUE

SUITE 417OAKLAND, CA 94611

(510) [email protected]

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

BERSIN & ASSOCIATES

Research Bulletin | 2011

BERSIN & ASSOCIATES © 2011

THIS MATERIAL IS LICENSED TO KNOWLEDGEADVISORS FOR DISTRIBUTION ONLY.

KnowledgeAdvisors and Metrics that MatterTM Josh Bersin | Page 16

The Bersin & Associates Membership ProgramThis document is part of the Bersin & Associates Research Library. Our research is provided exclusively to organizational members of the Bersin & Associates Research Program. Member organizations have access to the largest library of learning and talent management related research available. In addition, members also receive a variety of products and services to enable talent-related transformation within their organizations, including:

• Research – Access to an extensive selection of research reports, such as methodologies, process models and frameworks, and comprehensive industry studies and case studies;

• Benchmarking – These services cover a wide spectrum of HR and L&D metrics, customized by industry and company size;

• Tools – Comprehensive tools for HR and L&D professionals, including tools for benchmarking, vendor and system selection, program design, program implementation, change management and measurement;

• Analyst Support – Via telephone or email, our advisory services are supported by expert industry analysts who conduct our research;

• Strategic Advisory Services – Expert support for custom-tailored projects;

• Member Roundtables® – A place where you can connect with other peers and industry leaders to discuss and learn about the latest industry trends and best practices; and,

• IMPACT® Conference: The Business Of Talent – Attendance at special sessions of our annual, best-practices IMPACT® conference.

• Workshops – Bersin & Associates analysts and advisors conduct onsite workshops on a wide range of topics to educate, inform and inspire HR and L&D professionals and leaders.

For more information about our membership program, please visit us at www.bersin.com/membership.