scorecards, learning metrics and measurement strategies

You can listen to today’s webinar using your computer’s speakers or you may dial into the teleconference.

If you would like to join the teleconference,

please dial 1.866.469.3239 and enter access code xxx xx xxx.

You will be on music hold until the seminar begins.

Scorecards, Learning Metrics and Measurement Strategies

#CLOwebinar

Speaker: Dennis Bonilla Senior Vice President GP Learning Solutions Group

Andrew Snoey Senior Program Manager, Operations Microsoft Corporation

Moderator: Kellye Whitney Managing Editor Chief Learning Officer magazine


#CLOwebinar

•  Q&A

–  Click on the Q&A panel (?) in the bottom right corner

–  Type in your question in the space provided

–  Click on “Send.”

Tools You Can Use

#CLOwebinar

•  Polling

–  The poll will appear on the right side of your screen

–  Select the best option for each question

–  Click on “Submit”

#CLOwebinar

Tools You Can Use

Frequently Asked Questions 1. Will I receive a copy of the slides after the webinar?

YES

2. Will I receive a copy of the webinar recording? YES

Please allow up to 2 business days to receive these materials

#CLOwebinar

#CLOwebinar

Kellye Whitney Managing Editor Chief Learning Officer magazine


#CLOwebinar

Dennis Bonilla Senior Vice President GP Learning Solutions Group Andrew Snoey Senior Program Manager, Operations Microsoft Corporation


LEARNING METRICS, SCORECARDS MEASUREMENT STRATEGIES

9

POLL QUESTION

Where does your role fit in your Learning and Development Organization? •  Not at all

•  Instructional Designer

•  Instructor

•  Training Coordinator

•  Training Manager

•  In the Line of Business

10

POLL QUESTION

Does your organization formally track and report learning metrics?

•  Not at all

•  Sometimes

•  I Don’t Know

•  Yes

11

POLL QUESTION

Does your organization formally track and report learning metrics?

•  Not at all

•  Sometimes

•  I Don’t Know

•  Yes

If you answered yes to the previous poll, are the metrics:

•  Just Learning Process and Event Statistics (butts in seat, courses completed, etc….)

•  Tied to Organizational Business Outcomes

•  Both

SECTION 1 – CRITERIA FOR DEVELOPING EFFECTIVE SCORECARDS

12

SCORECARD OPERATING PRINCIPLES

•  Drive Business Leadership •  Aim for Simplicity •  Set Realistic, But Stretched, Targets •  Ensure Consistency in Taxonomy and Calculations

•  Ensure Similar Metric Types have Similar Thresholds •  Align Related Scorecards’ Metrics and Targets

13

IS LEARNING MAKING A DIFFERENCE?

•  Are business objectives being met?

•  Business leaders are demanding more linkages to business outcomes and measurements of results, yet it’s the outcome least often measured.

•  Huge confusion between internal process metrics and actual results.

•  Sponsors want to see evidence of IMPROVED BUSINESS PERFORMANCE, not internal training operations metrics.

14

SECTION 2 – DISCUSS THE EVOLUTION OF LEARNING MEASUREMENT FRAMEWORKS

15

16

POLL QUESTION

How does your organization / company measure the quality of the training programs for your key strategic initiatives?

•  Not at all

•  Data collection measuring attendee satisfaction

•  Data collection measuring attendee competency

•  Data collection measuring manager satisfaction in attendee performance

•  Data collection measuring business impact after course completion

WHAT METHODS ARE USED IN EVALUATION?

•  Evaluation is traditionally represented as the final stage in a systematic approach to training

•  More recent Instructional Systems Design (ISD) models incorporate evaluation throughout the process, not as end state

•  There are six general approaches to educational evaluation with Goal-based and Systems-based approaches as the predominant modes used in the evaluation of training programs

•  Various frameworks have been proposed under the influence of these two approaches

•  The most influential framework has come from Kirkpatrick which follows the Goal-based approach and the most commonly used today

17

KIRKPATRICK LEVELS

18

BASELINE INFO SUCH AS STUDENT PRESENCE AND PARTICIPATION

ATTENDANCE LEVEL 0

STUDENT PERCEPTIONS OF THE COURSE AND ITS VALUE

REACTIONS LEVEL I Most organizations have

Level 0 to I

STUDENT UNDERSTANDS AND CAN RECALL PRIMARY CONCEPTS FROM THE TRAINING

LEARNING LEVEL II

STUDENTS’ BEHAVIORAL CHANGE RESULTING IN PERFORMANCE AT A HIGHER LEVEL

TRANSFER LEVEL III Assessing Level II and III takes

more time and effort –this is where our certification

program is located

BUSINESS IMPACT AS A CONSEQUENCE OF TRAINING

RESULTS LEVEL IV

But Level IV is the only place where we can easily perform

an ROI calculation

SECTION 3 –ALIGNING TRAINING MEASURES WITH BUSINESS MEASURES

19

TALK ABOUT A DISCONNECT

Data from the Forum Corporation, Customer Driven Training Organization 20

0

1

2

3

4

BUSINESS LINKAGE REGISTRATION & ALIGNMENT

PROGRAM DELIVERY MEASUREMENT

WHAT EXECUTIVES WANT

TALK ABOUT A DISCONNECT

Data from the Forum Corporation, Customer Driven Training Organization 21

0

1

2

3

4

BUSINESS LINKAGE REGISTRATION & ALIGNMENT

PROGRAM DELIVERY MEASUREMENT

CURRENT EFFORT ALLOCATION WHAT EXECUTIVES WANT

MEASUREMENT STRATEGIES

•  1st Question that should always be asked: “Does this metric actually measure an outcome for which the training program was created?”

•  For some, it could be a behavior change; for others, improved perception by stakeholders, and others a quantitative measure of business impact (revenue, market share, market position, etc.).

•  Measure what matters to the customer in a way that leads to informed action.

•  Don’t pick a process before defining the purpose. Let form follow function!

22

MEASUREMENT STRATEGY CORE PRINCIPLES

•  Relevant

•  Credible and Compelling

•  Simple and Consistent

•  Efficient

23

“For every complex problem there is an answer that is clear, simple and wrong!”

– H.L. Mencken

24

POLL (TRIVIA) QUESTION

When was the first, documented, formal use of an educational program evaluation conducted in the USA?

•  1792

•  1815

•  1897

•  1916

•  1958

“NOT EVERYTHING THAT CAN BE COUNTED, COUNTS.” So What Counts?

•  Outcomes that are of interest to the business to which training contributed in a demonstrable way.

•  Sponsors may not be as wedded to proof of financial ROI as many HR professionals assume.

•  An evaluation that provides sound evidence to support informed decisions that are in the best interest of the organization.

•  They prove to an appropriate level of certainty that the program did or did not achieve the desired outcomes and they provide insights to improve subsequent courses or iterations; actionable Information (Bersin)

25

PRESENTING TRAINING RESULTS Power of Narrative

•  “Stories are easier to remember – because in many ways, stories are how we remember.” Pink (2005): A Whole New Mind

•  Stories bring statistics to life

•  Stories will be repeated and retold

Sell the Sizzle: It doesn’t matter how good your results are if no one knows about them.

Results x Marketing = Perception of Value

26

SECTION 4 – SHARING EXAMPLES OF MICROSOFT’S EFFORTS

27

PROJECT PICASSO In the Fall of 2009, the Microsoft training organization to applied our measurement principles to our course assessment survey.

Remember •  Relevant: questions refer to key course success metrics à eliminated non-action-

oriented questions •  Simple: shortened the question quantity to improve response rate •  Consistent: aligned format, wording, and calculations à minor wording revisions

Compelling: added business impact questions

•  Manager engagement on the material •  Different actions will the employee take due to the course

28

OLD SURVEY NEW SURVEY

OLT 12 + 4 6 + 4

cILT 11+7 12 + 4

vILT 11+7 12 + 4

PROFESSIONAL CERTIFICATION

29

•  Microsoft has an extensive certification program for internal and external candidates using third-party testing organizations

•  Our technical staff can be certified in 18 tracks; when looking across the technologies and levels, resulting in 64 discrete certifications

TECHNICAL EMPLOYEES

•  Sales staff was taught the “soft” (professional) skills, but how do you measure the knowledge?

•  In January 2010, Microsoft started an effort to train Sales & Margeting staff through an academy program that would have certifications associated with certain skills

SALES & MARKETING EMPLOYEES

http://www.microsoft.com/learning/en/us/certification/cert-default.aspx

SALES AND MARKETING ACADEMY Our Sales & Marketing Academy is part of our People focus

Purpose •  Accelerate sales and marketing staff readiness •  Promote management coaching and re-enforcement

Options considered •  Certification of the growth of a skill •  Certification of the growth of a skill set •  Separate certifications for each skill •  Certify high-priority skills

Built

•  Curriculum roadmaps •  Course knowledge assessments – demand pass rate of 80% •  Single certification activity across all courses in a level involving case study/simulation,

live work product review, role play, oral review panel

30

Basic Competent Master

Business Management A101 A201 A301

Customer Relationship Management B101 B201 B301

Products & Services C101 C102

-- C301

Organizational Awareness

A101 A102

B201 D201

--

… E101 B201 --

… F101 F201 F301 F302

MEASURING TRAINING IMPACT Almost two years ago, Microsoft started an effort to measure the business impact of our core sales training curriculum (CST)

Key questions asked •  Does CST increase the Account Manager’s annual quota attainment? •  Does CST increase Customer Satisfaction? •  Does CST increase the Account Manager’s pipeline velocity?

Tried to control for •  Region / country

•  Tenure / experience level •  Account Manager’s satisfaction level with CST courses •  Account Manager's self-assessed impact of CST courses

31

ALIGNING TRAINING TO A BUSINESS SCORECARD

32

Metric Color?

Root Cause?

Y/R

G

OTHER PLANS

OTHER

DEVELOP CORPORATE SCORECARD

Sell $ Share % Cust Sat #

SALES & MARKETING EXECUTES TO STRATEGIC INITIATIVES

QUARTERLY SCORECARD RESULTS

Course Metric A Metric B Metric C Metric D Metric E

Sell… X X X

Market… X X

Present… X

Create… X

Close… X X

FORMULATE RESPONSE PLAN 1.  Drive existing course(s) 2.  Bolster manager reinforcement 3.  Re-vamp existing course(s) 4.  Develop new course(s)

ASSESS TRAINING ADOPTION

TRAINING

Feedback Loops

SUMMARY

33

•  Align Your Scorecard with the Desired Business Results

•  Do Not Abandon Measurement at the Smiley Face

•  Count Training Results, Not Effort

LET’S CONTINUE TALKING

Andrew Snoey Senior Program Manager, Operations Microsoft Corporation [email protected] 425-722-8510 office

Dennis Bonilla Senior Vice President, Learning Solutions General Physics Corporation [email protected] 425-636-3410 office 702-885-5814 mobile

Join Our Next CLO Webinar

To the Agile Go the Spoils: Why Learning Must Evolve

Tuesday, June 21, 2011

CLO Webinars start at 2 p.m. Eastern / 11 a.m. Pacific

Register at www.clomedia.com/events Join the CLO Network: http://network.clomedia.com/

#CLOwebinar

scorecards, learning metrics and measurement strategies

Technology