september/october 2015 measuring elearning to create … · measuring elearning to create virtuous...

Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.comSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn

Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US

Measuring eLearning to Create Virtuous Cycles of Continuous Improvement

DevLearn – September/October 2015

Bridging Gap between Research and Practice

Slides available at: www.is.gd/willstuff

Research

Simplicity is the Ultimate Sophistication

Leonardo da Vinci

Engagement &Understanding

Remembering

Application

The Decisive Dozenfor Learning Design and Learning Measurement

http://is.gd/ddResearch

Baseline1. Content

2. Exposure

3. Guiding Attention

4. Creating Correct Conceptions

5. Repetition

6. Feedback

7. Variation

8. Retrieval Practice

9. Context Alignment

10. Spacing

11. Persuasion

12. Perseverance

http://is.gd/DecisiveDozen

Available at: is.gd/manifesto22

The 22 Principles

www.Debunker.Club

http://Work-Learning.com/catalog

Quite simply, the BEST book on smile sheet creation and utilization, Period!

Karl M. KappProfessor of Instructional Technology

Bloomsburg University

Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.

Robert O. BrinkerhoffProfessor Emeritus, Western Michigan

University & Director, Brinkerhoff Evaluation Institute

Evidence‐based practice at the master level.

Julie DirksenAuthor of Design For How People Learn

How effective are your organization’s smile sheets—those that you use in your elearning?

We DO NOT USE smile sheets in our

elearning.

1

We DO NOT USE THE DATA we

collect.

2

Our smile sheets are

VERY EFFECTIVE

4

Our smile sheets are NOT VERY EFFECTIVE

3

How do you typically measure Level 2 Learning Results—in your elearning?

With items that focus on KEY TERMINOLOGY

1

With items that focus on

KEY CONCEPTS

2

With SIMULATIONSor HANDS‐ON EXERCISES

4

With REALISTICSCENARIO‐BASED

DECISIONS

3

Which of the following practices is done routinely in your organization to measure learning results?

We measure learning results a week or more after learning events.

1

We measure post‐training on‐the‐job application of what

was learned.

2

We track on‐the‐job learning.

4

We watch learners as they use our

elearning programs.

3

The Kirkpatrick Model Rules!Level 1

ReactionLevel 2

LearningLevel 3

BehaviorLevel 4

Results

LearningIntervention

PerformanceSituation

LearningOutcomes

IndividualResults

OrganizationalResults

On-the-Job Learning

Prompting

The Learning Landscape

Learning

Remembering

On-the-Job Performance

Later on the job, learners remember

what they’ve learned.

The learners get a return on their efforts.

Performance can be prompted through job aids, signage,

intuitive cues, performance

support, management, etc.

Learners can learn on-the-job through

trial & practice, insight learning, help from others,

social media, studying on their

own, etc.

© Copyright 2009-2015 Work-Learning Research, Inc.

YouTube: http://is.gd/LearningLandscape

Most Smile Sheets Not Correlatedwith Learning!

Level 1to

Level 3

r=.16

Alliger, Tannenbaum, Bennett, Traver, & Shotland (1997). A meta-analysis of the relations among training criteria.

Personnel Psychology, 50, 341-357.

Very Weak Relationship between Levels

Correlation between levels?

Level 1to

Level 2

r=.09

Sitzmann, T., Brown, K. G., Casper, W. J., Ely, K., & Zimmerman, R. D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied

Psychology, 93, 280-295.

Correlation between levels?

No

Practical Significance

Weak Relationship is below .30 and .09 is VERY WEAK

So…SMILE SHEETS tell us VERY LITTLE about Learning

Level 1to

Level 2

r=.09

The Critical Importance Of Remembering



LearningOutcomes

IndividualResults




Remembering

Learning

© Copyright 2009-2015 Work-Learning Research, Inc., www.work-learning.com

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

After LearningDuring Learning

Learning and Forgetting Curves

Learners

Learners

If our learners start here.

But end up here.

Have we maximized the

learning benefits?

Less Remembering

More Remembering

© Copyright 2012-2015 by Work-Learning Research, Inc. (www.work-learning.com)

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15


On-the-JobLearning Curves

On-the-JobForgetting

Curves

LearningCurve


There are many possible after-training results,

depending on:

(1) Design of the Learning(2) After-Learning Follow-up


YouTube: http://is.gd/LearningForgettingCurves

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15


On-the-JobLearning Curves

On-the-JobForgetting

Curves

LearningCurve



What does an end-of-course

assessment tell us?

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15




If we measure

here?BiasedMetric!!

Understanding

Remembering

Application

DelayedTests

Three Biases in the WayWe Measure Level 2 Learning

WHEN did you Measure Learning?

90 %

WHERE did you Measure Learning?

91 %

Aggleton, J. P., & Waskett, L. (1999). The ability of odours to serve as state-dependent cues for real-world memories: Can Viking smells aid the recall of Viking experiences? British Journal of Psychology, 90, 1-7.

Balsam, P. D. (1985). The functions of context in learning and performance. In P. D. Balsam & A. Tomie (Eds.) Context and Learning. Hillsdale, NJ: Erlbaum.Bjork, R. A., & Richardson-Klavehn, A. (1989). On the puzzling relationship between environmental context and human memory. In C. Izawa (Ed.) Current Issues in Cognitive Processes: The Tulane

Floweree Symposium on Cognition (pp. 313-344). Hillsdale, NJ: Erlbaum.Bouton, M. E. (1993). Context, time, and memory retrieval in the interference paradigms of Pavlovian learning. Psychological Bulletin, 114, 80-99.

Bower, G. H., Monteiro, K. P., and Gilligan, S. G. (1978). Emotional mood as context for learning and recall. Journal of Verbal Learning and Verbal Behavior, 17, 573-585.Bransford, J. D., Franks, J. J., Morris, C. D., & Stein, B. S. (1979). Some general constraints on learning and memory research. In L. S. Cermak & F. I. M. Craik (Eds.), Levels of processing in human

memory (pp. 331-354). Hillsdale, NJ: Erlbaum.Cassaday, H. J., Bloomfield, R. E., Hayward, N. (2002). Relaxed conditions can provide memory cues in both undergraduates and primary school children. British Journal of Educational Psychology,

72(4), 531-547.Cousins, R., & Hanley, J. R. (1996). The effect of environmental context on recall and category clustering scores following relational and individual item processing: A test of the outshining

hypothesis. Memory, 4, 79-90.Dalton, P. (1993). The role of stimulus familiarity in context-dependent recognition. Memory & Cognition, 21, 223-234.

Davies, G. (1986). Context effects in episodic memory: A review. Cahiers de Psychologie Cognitive, 6, 157-174.Dijkstra, K., Kaschak, M. P., & Zwaan, R. A. (2007). Body posture facilitates retrieval of autobiographical memories. Cognition, 102, 139-149.

Dulsky, S. G. (1935). The effect of a change of background on recall and relearning. Journal of Experimental Psychology, 18, 725-740.Eich, E. (1985). Context, memory, and integrated item/context imagery. Journal of Experimental Psychology: Learning, Memory, and Cognition, 11, 764-770.

Eich, E. (1995). Mood as a mediator of place dependent memory. Journal of Experimental Psychology: General, 124(3), 293-308.Eich, J. E. (1980). The cue dependent nature of state dependent retrieval. Memory and Cognition, 8, 157-173.

Fernandez, A., & Glenberg, A. M. (1985). Changing environmental context does not reliably affect memory. Memory & Cognition, 13, 333-345.Gartman, L. M., & Johnson, N. F. (1972). Massed versus distributed repetition of homographs: A test of the differential encoding hypothesis. Journal of Verbal Learning and Verbal Behavior, 11, 801-

808.Godden, D. R., and Baddeley, A. D. (1975). Context dependency in two natural environments: on land and underwater. British Journal of Psychology, 91, 99-104.

Godden, D., & Baddeley, A. (1980). When does context influence recognition memory? British Journal of Psychology, 71, 99-104.Grant, H. M., Bredahl, L. C., Clay, J., Ferrie, J., Groves, J. E., McDorman, T. A., & Dark, V. J. (1998). Context-dependent memory for meaningful material: Information for students. Applied Cognitive

Psychology, 12, 617-623. Herz, R. S. (1997). The effects of cue distinctiveness on odor-based context-dependent memory. Memory & Cognition, 25(3), 375-380.

Jacoby, L. L. (1983). Remembering the data: Analyzing interactive processes in reading. Journal of Verbal Learning and Verbal Behavior, 22, 485-508.Johnson, A. J., Miles, C. (2008). Chewing gum and context-dependent memory: The independent roles of chewing gum and mint flavour. British Journal of Psychology, 99(2), 293-306.

Marian, V., & Kaushanskaya, M. (2007). Language context guides memory content. Psychonomic Bulletin & Review, 14(5), 925-933.Marian, V., & Neisser, E. (2000). Language-dependent recall of autobiographical memories. Journal of Experimental Psychology: General, 129, 361-368.

Mead, K. M. L., & Ball, L. J. (2007). Music tonality and context-dependent recall: The influence of key change and mood mediation. European Journal of Cognitive Psychology, 19(1), 59-79.Pan, S. (1926). The influence of context upon learning and recall. Journal of Experimental Psychology, 9, 468-491.

Parker, A., & Gellatly, A. (1997). Moveable cues: A practical method for reducing context-dependent forgetting. Applied Cognitive Psychology, 11, 163-173.Prestera, G. E., Clariana, R., & Peck, A. (2005) Memory-Context Effects of Screen Color in Multiple-Choice and Fill-in Tests. Journal of Educational Multimedia and Hypermedia, 14(4), 2005, 415-436.

Riccio, D. C., Richardson, R., & Ebner, D. L. (1984). Memory retrieval deficits based upon altered contextual cues: A paradox. Psychological Bulletin, 96, 152-165.Roediger, H. L., III, & Guynn, M. J. (1996). Retrieval processes. In E. L. Bjork & R. A. Bjork (eds.), Memory (pp. 197-236). San Diego, CA: Academic Press.

Russo, R., Ward, G., Geurts, H., & Scheres, A. (1999). When unfamiliarity matters: Changing environmental context between study and test affects recognition memory for unfamiliar stimuli. Journal of Experimental Psychology: Learning, Memory, and Cognition, 25, 488-499.

Schab, F. R. (1990). Odors and remembrances of things past. Journal of Experimental Psychology: Learning, Memory, and Cognition, 8, 648-655.Schroers, M., Prigot, J., & Fagen, J. (2007) The effect of a salient odor context on memory retrieval in young infants. Infant Behavior & Development. 30(4), 685-689.

Smith, S. M. (1979). Remembering in and out of context. Journal of Experimental Psychology: Human Learning and Memory, 5, 460-471.Smith, S. M. (1982). Enhancement of recall using multiple environmental contexts during learning. Memory & Cognition, 10, 405-412.Smith, S. M. (1984). A comparison of two techniques for reducing context-dependent forgetting. Memory & Cognition, 12, 477-482.

Smith, S. M. (1985). Background music and context-dependent memory. American Journal of Psychology, 98, 591-603.Smith, S. M. (1988). Environmental context-dependent memory. In G. M. Davies & D. M. Thomson (eds.) Memory in Context: Context in Memory (pp. 13-34), Chichester, UK: Wiley.

Smith, S. M. (1995). Mood is a component of mental context: Comment on Eich (1995). Journal of Experimental Psychology: General, 124(3), 309-310.Smith, S. M., & Rothkopf, E. Z. (1984). Contextual enrichment and distribution of practice in the classroom. Cognition and Instruction, 1, 341-358.

Smith, S. M., & Vela, E. (2001). Environmental context-dependent memory: A review and meta-analysis. Psychonomic Bulletin & Review, 8, 203-220.Smith, S. M., Glenberg, A., and Bjork, R. A. (1978). Environmental context and human memory. Memory and Cognition, 6, 342-353.

Spear, N. E. (1978). The processing of memories: Forgetting and retention. Hillsdale, NJ: Erlbaum.Thompson, L. A., Williams, K. L., L'Esperance, P. R., Cornelius, J. (2001) Context-dependent memory under stressful conditions: The case of skydiving. Human Factors, 43(4), 611-619.

Tulving, E., & Thompson, D. M. (1973). Encoding specificity and retrieval processes in episodic memory. Psychological Review, 80, 352-373.Vela, E. (1984). Memory as a function of environmental context. Paper presented at the 30th annual meeting of the Southwest Psychological Association, New Orleans, LA.

Weiss, W., & Margolis, G. (1954). The effect of context stimuli on learning and retention. Journal of Experimental Psychology, 48, 318-322.Wright, D. L., & Shea, C. H. (1991). Context dependencies in motor skills. Memory & Cognition, 19, 361-370.

Tested in the learning room or in a different room. Does this matter?

Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.

0

5

10

15

20

25

Tested ina

DifferentRoom

Tested inthe

LearningRoom

MeasurementLevels

Level AReal World

Level C Scenario-Based

Decisions

Level DMemorization

Questions

Level B Simulation

Level EAttendance

Level FAffiliation

Certification

Quasi-Certification

“Level C represents the last level of certification that can be considered to assess an ability to perform on the job. Level D represents the first quantum jump away from fidelity in assessment and should be

used with caution.”

Scenario‐Based Question Example

Alena wants to start a firm that helps farmers grow food organically. She has a degree in sustainable agriculture and has worked for a non‐profit organization for seven years doing similar work. She’s developed a marketing plan, a financial plan, and has found several farmers who would pay her if she went out on her own. What should Alena do first—before she tells her boss that she’s quitting to start her own firm?

A. Create a cash flow statement to determine whether her predicted income will support the business through the first year.

B. Form a group of advisors with experience in both small‐business management and agriculture.

C. Determine whether she has enough seed money and start‐up capital to get started.

D. Analyze her values and goals to ensure that the proposed business will support them.

Smile Sheets Can be Improved!

Likert-like Scales provide Poor DataA. Strongly AgreeB. AgreeC. Neither Agree Nor DisagreeD. DisagreeE. Strongly Disagree

54321

4.1

Compared to:

• Previous• Standard• Others

Sharon Shrock and Bill Coscarelli, authors of the classic text, now in its third edition, Criterion‐Referenced Test Development, offer the following wisdom:

On using Likert‐type Descriptive Scales (of the kind that uses response words such as “Agree,” “Strongly Agree,” etc.):

“…the resulting scale is deficient in that the [response words] are open to many interpretations.” (p. 188)

We’d like to trust our learners…

But the research shows that they don’t always know their own learning…

Learners are Overly Optimistic Zechmeister & Shaughnessy (1980).

Learners can’t always OvercomeFaulty Prior KnowledgeKendeou & van den Broek (2005).

Learners Fail to Properly Use ExamplesRenkl (1997).

Learners Fail to Give ThemselvesRetrieval PracticeKarpicke, Butler, & Roediger (2009).

Two Recent Reviews Emphasize Learners’ Lack of Knowledge of LearningBrown, Roediger & McDaniel (2014); Kirschner & van Merriënboer (2013).

Transmogrify

Strongly AgreeAgree

Neither Agree Nor DisagreeDisagree

Strongly Disagree

We Start with Fuzzy

Adjectives

54321

Magically We Turn Adjectives

Into Numbers

We Average

Responses, Losing

More Info

3.9

4.2

3.7

4.0

4.1

4.0

4.2

3.4

We Choose One

Question and Report Results

4.1

My Journey in tryingTo create a better

Smile Sheet

My “New” Smile Sheet

Specific Concepts

How much Value?

ConceptNew?

HowWell

Taught?

Simple Overall Ratings

Helping Learners Calibrate

Best Feedback Comes from Comments

http://is.gd/TrainingResearch2012

http://is.gd/ddResearch

Performance-FocusedSmile Sheet

Quite simply, the BEST book on smile sheet creation and utilization, Period!

Karl M. KappProfessor of Instructional Technology

Bloomsburg University

Thoughtful and sensible advice for feedback tools that will provide valid and actionable data.

Robert O. BrinkerhoffProfessor Emeritus, Western Michigan

University & Director, Brinkerhoff Evaluation Institute

Evidence‐based practice at the master level.

Julie DirksenAuthor of Design For How People Learn

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET

Primary Goals Secondary Goals Tertiary Goals

LEARNINGEFFECTIVE?

Will the learning be effective in supporting

on-the-job performance?

RESULTS ACTIONABLE?

Will the Smile-Sheet results communicate

with clarity and urgency to guide

action?

LearnersUnderstand?

LearnersRemember?

LearnersMotivated to Apply?

After-Training Supports in Place?

Learners Smile Sheet Decisions

Accurate? Avoiding GIGO?

Do Smile Sheet Results Distinguish between Different

Levels of Success?

Are We Measuring the Things that

Matter?

Are We Using the Smile Sheet

Opportunity to Educate Our

Stakeholders?

LearnersEngaged

Cognitive Supports Effective

Realistic Retrieval

SpacingSit-Action Triggers

Belief in Value of Concepts

Self-Efficacy in Skill Area

InoculatedJob Aids

Supervisors Follow-up

Do learners remember enough to answer the questions, are the questions focused on most

important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?

From the information, can we determine whether a course needs to be maintained, improved, or

removed? Are we avoiding numeric averages that discourage a standards-based decision on

success and failure?

While smile sheets are not capable on their own to determine effectiveness, we should at least try

to examine the four goals, (1) understanding, (2) remembering, (3)

motivation to apply, (4) after-training support.

We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,

instructional designers.

Three Key Goals:

1. Are the Questions Well-Designed?

2. Are Learners Making Good Smile Sheet Decisions?

3. Are The Data we’re Getting Clear and Actionable?

Ensuring that we are focused on the science-

of-learning factors that matter!

Ultimate Goal

MAXIMALLY EFFECTIVE

SMILE SHEET

Primary Goals Secondary Goals Tertiary Goals

LEARNINGEFFECTIVE?

Will the learning be effective in supporting

on-the-job performance?

RESULTS ACTIONABLE?

Will the Smile-Sheet results communicate

with clarity and urgency to guide

action?

LearnersUnderstand?

LearnersRemember?

LearnersMotivated to Apply?

After-Training Supports in Place?

Learners Smile Sheet Decisions

Accurate? Avoiding GIGO?

Do Smile Sheet Results Distinguish between Different

Levels of Success?

Are We Measuring the Things that

Matter?

Are We Using the Smile Sheet

Opportunity to Educate Our

Stakeholders?

LearnersEngaged

Cognitive Supports Effective

Realistic Retrieval

SpacingSit-Action Triggers

Belief in Value of Concepts

Self-Efficacy in Skill Area

InoculatedJob Aids

Supervisors Follow-up

Do learners remember enough to answer the questions, are the questions focused on most

important factors, are the answers calibrated to provide granularity, are leading questions avoided, do questions avoid areas of bias?

From the information, can we determine whether a course needs to be maintained, improved, or

removed? Are we avoiding numeric averages that discourage a standards-based decision on

success and failure?

While smile sheets are not capable on their own to determine effectiveness, we should at least try

to examine the four goals, (1) understanding, (2) remembering, (3)

motivation to apply, (4) after-training support.

We should use smile sheets to send stealth messages to our stakeholders, including senior decision makers, instructors,

instructional designers.

What Most Smile Sheets

Provide

QUATERNARYGOALS

Traditional Smile Sheets

Learners rate instructors as credible and engaging?

Learners say that classroom environment was conducive

to learning?

Learners satisfied with experience?

Learners think course was well-organized?

Ultimate Goal Primary Goals Secondary Goals Tertiary Goals

In regard to the course topics taught, HOW ABLE ARE YOU to put what you’ve learned into practice on the job?

A. I’m NOT AT ALL ABLE to put the concepts into practice.

B. I have GENERAL AWARENESS of the concepts taught, but I will need more training/practice/guidance/experience TO DO ACTUAL JOB TASKS using the concepts taught.

C. I am ABLE TO WORK ON ACTUAL JOB TASKS, but I’LL NEED MORE HANDS-ON EXPERIENCE to be fully competent in using the concepts taught.

D. I am ABLE TO PERFORM ACTUAL JOB TASKS at a FULLY-COMPETENT LEVEL in using the concepts taught.

E. I am ABLE TO PERFORM ACTUAL JOB TASKS at an EXPERT LEVEL in using the concepts taught.

A Better Smile Sheet Question

Ultimate Goal Primary Goals Secondary Goals Tertiary Goals

In regard to the concepts taught in the course, how motivated WILL YOU BE to UTILIZE these skills in your work?

A. I will NOT MAKE THIS A PRIORITY when I get back to my day-to-day job.

B. I will make this a PRIORITY—BUT A LOW PRIORITY when I get back to my day-to-day job.

C. I will make this a MODERATE PRIORITY when I get back to my day-to-day job.

D. I will make this a HIGH PRIORITY when I get back to my day-to-day job.

E. I will make this one of my HIGHEST PRIORITIES when I get back to my day-to-day job.

A Question About Motivation…

Traditional Smile Sheet

Design Learning

Design Smile Sheet

Deploy‐‐‐‐‐

Collect Data

Examine Results

Determine Standards

Performance‐Focused Smile

Sheet

Design Learning

Design Smile Sheet

Deploy‐‐‐‐‐

Collect Data

Examine Results

Determine Standards

OPEN TO

BIAS

© Copyright 2009 Work-Learning Research, Inc.

Now that you’ve taken the course, how well do you feel you understand the concepts taught in the course?

A. I have some significant CONFUSIONS AND/OR BLIND SPOTS.

B. I have a BASIC FAMILIARITY with the concepts.

C. I have a SOLID UNDERSTANDING of the concepts.

D. I have a COMPREHENSIVE UNDERSTANDING of the concepts.

E. I have an EXPERT‐LEVEL UNDERSTANDING of the concepts.

Standards

Unacceptable

Unlikely

?

Acceptable

?

AcceptableUnacceptable

AcceptableSuperior

This learning event will make a significant contribution to my work, in terms of the following: SELECT AS MANY AS YOU LIKE!

A. Improving my personal productivity.

B. Increasing my ability to innovate.

C. Enabling me to collaborate more effectively.

D. Improving my management performance.

E. Enabling me to generate more revenue.

F. Enabling me to lower costs.

The Future ofLearning Measurement

Nano-Coaching CyclePerform Task

Submit Work Product

Notify Coach

Coach Reviews Work Product

Specific Feedback

Approved?

• Photo• Checklist• Text• Video• Audio• Document• Direct

Observation

Email & Dashboard

Coaching Support

• Checklists, Guidelines

• Coach the Coach

• Photo• Checklist• Text

Comment• Audio• Document• Direct

Goal Achieved!

YES NO

©2014-2015 Cognitive Advisors LLC Used With Permission

MartyRosenheck

CEO

Any Learning

Experience

Learning Record Store(LRS)

Analytics and

Reporting

+ Context, Results, and Extensions

Activity Stream

<Actor> <Verb> <Object>

Learner Played Simulation

Learner Cleaned Crankshaft

Mike HruskaPresident/CEO

©2015 Problem SolutionsUsed With Permission

xAPI

xAPI Example

www.pipelinexapi.com



• HPML is an XML Schema‐based language.

• “Takes raw data as input and specifies the computations required to turn those data into measurements and assessments.”

• Provides a list of constructs, which can represent experiences, tasks, conditions, etc.

• Based as an XML Schema

• Study underway at Simulation Interoperability Standards Organization (SISO) for standard

References:• Stacy, W., Ayers, J., Freeman, J., & Haimson, C. (2006). Representing Human Performance

with Human Performance Measurement Language. Washington, DC: Aptima, Inc. • Walker, A., Tolland, M., & Webb, S. (2015) Using a Human Performance Markup Language

for Simulator‐Based Training. Woburn, MA: Aptima, Inc.

Human Performance Measurement Language (HPML)



Correlating learning performance with actual performance.

Andrew Downes

Comparing two versions of a program.

Subscription Learning

Learners Subscribe or are Subscribed

Many Learning Events

Spaced Over Time

Usually Short Nuggets

Usually Relies onPush Technology

Usually Utilizesthe Spacing Effect

To Learn More: SubscriptionLearning.com

Subscription Learning can be – THEORETICALLY – a series of informational nuggets

October

Nuggets

Nuggets Nuggets

Nuggets Nuggets

Nuggets Nuggets

Nuggets

Nuggets

Subscription Learning IS MORE THAN INFORMATIONAL NUGGETS

OctoberCEO Video

Challenge Q3FeedbackReflection

Challenge Q4Challenge Q5FeedbackAssignment

Assign. Review

Self‐Assessment

Challenge Q6Challenge Q7Feedback

Trainer Video

CEO VideoReflection

Challenge Q8Assignment

Assign. ReviewChallenge Q9Feedback

Reinforcer 1

Challenge Q10Challenge Q11

FeedbackReinforcer 2



Challenge Q14Feedback

Reinforcer 4Change Effort

Welcome!Challenge Q1Challenge Q2Feedback

Subscription Learning ALSO ENABLES STEALTH EVALUATION !!!

OctoberCEO Video

Challenge Q3FeedbackReflection

Challenge Q4Challenge Q5FeedbackAssignment

Assign Review

Self‐Assessment

Challenge Q6Challenge Q7Feedback

Trainer Video

CEO VideoReflection

Challenge Q8Assignment

Assign ReviewChallenge Q9Feedback

Reinforcer 1





Challenge Q14Feedback

Reinforcer 4Change Effort

Welcome!Challenge Q1Challenge Q2Feedback

GuerrillaEvaluation

• Our goal: Creating better feedback loops.

• Gather satisfactory data at a good cost point.

• Consider taking a user-testing approach.

· With Online Meeting Tools, it’s easy, it’s almost FREE, it takes hardly any time at all.

• Sample across cohort of users.

• Moral licensing.

How many of you have EVER seen a learner

using the elearning you’ve

developed?

Julie Dirksen, AuthorDesign for How People Learn



LearningOutcomes

IndividualResults


On-the-Job Learning

Prompting


Learning

Remembering


© Copyright 2009-2015 Work-Learning Research, Inc.Regular

smile sheets

Immediate tests of

remembering & decision-

making

Does Job Performance

Improve?

Does Business Performance

Improve?

IR

L1-Delayed

L2-Delayed

L3-Supports

PD

RS JS PS

L1

L2Are Prompt Supports

Used? Effective?

Are Just-in-Time Learning

Supports Used?

Effective?

Are Retrieval Supports

Used? Effective?

Smile sheets

Retrieval &

Decision Making

Evaluation of Coaching & Learner Learning

Delayed smile

sheets

Delayed tests of remembering & decision-making

Do the learners actually benefit

from the learning in their work or

career?

Does the work environment

support performance?

Are Prompting Devices Used?

Effective?

On smile sheets, ask about

motivation to apply

L1

L2

L3

L4

Problems with the Kirkpatrick/Phillips 4- or 5-Level Models of Learning Evaluation

• Pushes us to focus on weighing outcomes. Is largely silent on learning support and learning-design improvement.

• Training centric. Ignores prompting mechanisms & on-the-job learning.

• Ignores the role that management and the business side must play.

• Implies that higher levels are more important than lower levels.

• Ignores the causal chain from learning to remembering to performance to results.

• Ignores the fact that learners forget and that learning interventions can be good at creating understanding but poor at minimizing forgetting.

• Pushes us to value learner ratings as predictive of learning and on-the-job performance.

“Historically, organizations and training researchers have relied on Kirkpatrick’s [4-Level] hierarchy as a framework for evaluating training programs…

[Unfortunately,] The Kirkpatrick framework has a number of theoretical and practical shortcomings.

[It] is antithetical to nearly 40 years of research on human learning, leads to a checklist approach to evaluation (e.g., ‘we are measuring Levels 1 and 2, so we need to measure Level 3’), and, by ignoring the actual purpose for evaluation, risks providing no information of value to stakeholders… (p. 91)

http://is.gd/TrainingResearch2012

“The goal of training evaluation is not to prove the value of training; the goal of evaluation is to

improve the value of training.” (p. 94‐95)

Tim Mooney and Rob BrinkerhoffCourageous Training:

Bold Actions for Business Results.

Final Thoughts:

We have a responsibility to build learning programs that are effective.

Valid feedback enables improvement.

We must work to get good feedback.

We should aim to create virtuous cycles of continuous improvement.

Phone: 888‐579‐9814Email: info@work‐learning.comWebsite: Work‐Learning.comAudits: LearningAudit.comSmiles: SmileSheets.comBlog: WillAtWorkLearning.comTwitter: @WillWorkLearn

Will Thalheimer, PhDPresidentWork‐Learning Research, Inc.Somerville, Massachusetts, US

Measuring eLearning to Create Virtuous Cycles of Continuous Improvement

DevLearn – September/October 2015

Thank You!!

september/october 2015 measuring elearning to create … · measuring elearning to create virtuous...

Documents