capturing elusive level 3 data: the secrects of survey design · 3/22/2018 · phillips associates...

Phillips Associates 1

Capture Elusive Level 3 Data:

The Secrets of Survey Design

Presented by:Ken PhillipsPhillips AssociatesMarch 22, 2018


Agenda

1. Examine Level 3 evaluation facts2. Analyze survey creation errors in a

sample Level 3 evaluation

3. Discover 12 tips for creating valid, scientifically sound Level 3 evaluations


Kirkpatrick / Phillips Evaluation Model

Level 1: Reaction Degree to which participants find the training favorable, engaging, and relevant to their jobs

Level 2: Learning

Degree to which participants acquire the intended knowledge, skills, attitude, confidence and commitment based on their participation in the training

Level 3: Behavior Degree to which participants apply what they learned during training when they are back on the job

Level 4: ResultsDegree to which targeted outcomes improve as a result of the training, and the support and accountability package

Level 5: ROI Degree to which monetary program benefits exceed program costs


Level 3 Evaluation Facts

Source: ATD Research Study, “Evaluating Learning Getting to Measurements That Matter,” 2015

60%Organizations

evaluate some

programs

at Level 3

of live classroom

programs being

evaluated33%75%Organizations view

data collected

as having high

or very high value18% of tech based programs beingevaluated


Source: Donald & James Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 2006.

5

Data Collection Methods

Sheet1

EVALUATION LEVELS

METHODS1Reaction2Learning3Behavior4Results

Survey/Questionnairesllll

Interviewsllll

Focus Groupsllll

Knowledge Testingl

Work Reviewll

Structured Observationll

Action Planningll

HR/Business Unit Reportsll

Key Performance Indicatorsl

Sheet2

Sheet3


Possible Survey Respondents

External customers

Managers of learners

Direct reportsPeers/ ColleaguesLearners


How to Decide

has first-hand knowledge of learners’ behavior?

credible do results need to be?

Who

How


Sample

Level 3

Participant Survey


Instructions

Note: Survey respondents are the direct reports of managers/ supervisors who attended an interpersonal feedback learning program.

1. Form a group of 3, 4 or 5 persons2. Review sample Level 3 participant survey in

handout and see how many different survey creation errors you can find (Hint: 9 different errors are built into the survey)

3. Be prepared to discuss your findings with the whole group


Scientifically Sound Survey Design

Measurement

ContentFormat

Phillips Associates

Content


8. Before providing employees with feedback about their job performance, my manager considers whether or not he or she is knowledgeable about their job.

What’s Wrong With These?

25. When giving feedback to an employee my manager considers whether it should be done privately or in the presence of others.


Tip 1: Content

Source: Palmer Morrel-Samuels, “Getting the Truth into Workplace Surveys”, Harvard Business Review, 2002.

Focus on observable behavior

not thoughts or motives.


18. My manager provides employees with regular ongoing feedback about their job performance and speaks in a normal conversational tone or manner when delivering the feedback.


14. My manager gives his or her employees feedback just as soon as possible after an event has happened and avoids getting emotional or evaluative.


Tip 2: Content

Limit each item to a

single description of behavior.

Source: Palmer Morrel-Samuels, 2002


Example

My manager gives his or her employees feedback just as soon as possible after an event has happened.

My manager avoids getting emotional or evaluative when giving feedback to his or her employees.



2. My manager doesn’t get to know his or her employees as individuals before providing them with feedback about their job performance.

7. When giving employees feedback about their job performance, my manager doesn’t distinguish between patterns of behavior and random one-time events.


Tip 3: Content

Word about 1/3 of the survey items so that

the desired answer is negative.



Format



Building Trust

Credibility

Feedback Sign

Feedback Timing

Feedback Frequency

Message Characteristics


Tip 4: Format

Keep sections of the survey unlabeled.




Design sections to contain

a similar number of items

and questions to contain

a similar number of words.

Tip 5: Format



Tip 6: Format

Place questions regarding

respondent demographics

(e.g. name, title, department, etc.)

at end of survey, make completion optional

and keep questions to a minimum.


Measurement


Source: Ken Phillips, “Capturing Elusive Level 3 Data: The Secrets of Survey Design”, Unpublished Article, 2013.

Tip 7: Measurement

Collect data from multiple observers

or a single observer multiple times.


What’s Wrong With This?

StronglyAgree Agree Disagree

StronglyDisagree N/A

4 3 2 1


*Palmer Morrel-Samuels, 2002.Source: Palmer Morrel-Samuels, 2002

Tip 8: Measurement

Create a response scale

with numbers at regularly spaced intervals

and words only at each end.


ExamplesThis:

Not This:

Or This:

Not at allTrue

CompletelyTrue

1 2 3 4 6 75

Not at allTrue

CompletelyTrue

1 2 3 4 6 75

RarelyTrue

Occasionally True

SomewhatTrue

MostlyTrue

Frequently True

Not at allTrue

CompletelyTrue

RarelyTrue

Occasionally True

SomewhatTrue

MostlyTrue

Frequently True



Tip 9: Measurement

Use only one response scale

with an odd number of points

(7, 9 & 11 point scales are best)


Odd vs. Even Scale

This:

Not This:


Tip 10: Measurement

Use a response scale

that measures frequency

not agreement or effectiveness.



Examples

This:

Or this:

Never Always

1 2 3 4 6 75

Not at all True

Completely True

1 2 3 4 6 75


Source: Ken Phillips, 2013

Tip 11: Measurement

Place small numbers

at left or low end of scale

and large numbers at right

or high end of scale.


ExamplesThis:

Not at all True

Completely True

1 2 3 4 6 75

Completely True

Not at all True

7 6 5 4 2 13

Not This:



Tip 12: Measurement

Include a

“Did Not Observe” response choice

and make it different.


Example

Not at all True

Completely True

1 2 3 4 6 75

Did Not Observe


Summary: Content

Focus on observable behavior

Limit ideas to a single description of behavior

Word 1/3 of items as reverse score


Summary: Format

Keep survey sections unlabeled

Design sections to contain similar number of items & questions similar number of words

Place questions regarding respondent demographics at end of survey, make completion optional and keep questions to a minimum


Summary: Measurement

Collect data from multiple observers or multiple times

Create a response scale that: Has words only at each end Has an odd number of points Measures frequency Has small numbers at left and large numbers

at right Includes a “Did Not Observe” that is different



The difference between a good survey

and a bad one… quite simply, is careful

and informed design.


Phillips, Ken, “Eight Tips on Developing Valid Level 1Evaluation Forms”, Training Today, Fall 2007, pps. 8 & 14.

Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8.

Phillips, Ken, “Capturing Elusive Level 3 Data: The Secrets of Survey Design”, Unpublished article, 2013.

Phillips, Ken, “Level 1 Evaluations: Do They Have a Role in Organizational Learning Strategy?”, Unpublished article, 2013.

Phillips, Ken, “Business Results Made Visible: Designing Proof Positive Level 4 Evaluations”, Unpublished article, 2013.

Free Articles


Phillips [email protected](847) 231-6068

www.phillipsassociates.com

34137 N. Wooded Glen Drive

Grayslake, Illinois 60030

Ken Phillips

Slide Number 1AgendaKirkpatrick / Phillips Evaluation ModelLevel 3 Evaluation FactsData Collection MethodsPossible Survey RespondentsHow to DecideSample�Level 3�Participant SurveyInstructionsSlide Number 10Slide Number 11Slide Number 12Slide Number 13Slide Number 14Slide Number 15ExampleWhat’s Wrong With These?Tip 3: ContentSlide Number 19Slide Number 20Slide Number 21Tip 5: FormatTip 6: FormatSlide Number 24Slide Number 25What’s Wrong With This?Slide Number 27ExamplesSlide Number 29Odd vs. Even ScaleSlide Number 31ExamplesSlide Number 33ExamplesSlide Number 35ExampleSummary: ContentSummary: FormatSummary: MeasurementSlide Number 40Slide Number 41�Free ArticlesSlide Number 43

capturing elusive level 3 data: the secrects of survey design · 3/22/2018 · phillips associates...

Documents