cets 2011, ken phillips, slides for construct the ultimate multiple-choice test! stop measuring...

Phillips Associates

CONSTRUCT THE ULTIMATE MULTIPLE CHOICE TEST!

STOP MEASURING RECALL AND START MEASURING

APPLICATION Presented by: Ken Phillips Phillips Associates August 16, 2011

Phillips Associates 2

AGENDA ➤  Review Kirkpatrick/Phillips five level

evaluation model

➤  Examine Level 2 evaluation facts ➤  Analyze multiple choice test creation

errors in a sample Level 2 knowledge test

➤  Learn 10 tips for creating valid, scientifically sound multiple choice test questions


Levels of Evaluation Measurement Focus Time Frame

Level 1: Reaction Participant reaction to a learning program

Conclusion of program

Level 2: Learning Degree to which participants acquired new knowledge, skills or attitudes

Conclusion of program or within 6 to 8 weeks after

Level 3: Behavior Degree to which participants applied back-on-the-job what was learned

2 to 12 months

Level 4: Results Degree to which targeted business outcomes were achieved

9 to 18 months

Level 5: ROI Degree to which monetary program benefits exceed program costs

9 to 18 months

KIRKPATRICK/PHILLIPS EVALUATION MODEL


% LIVE CLASSROOM VS. TECH. BASED PROGRAMS EVALUATED AT LEVELS 1 – 5*

Level 1-Reaction 2-Learning 3-Behavior 4-Results 5- ROI

Live Classroom

81%

50%

25%

14%

6%

Tech. Based

52%

43%

15%

10%

6%

*ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009


➤  81% of organizations evaluate at least some learning programs at Level 2

➤  55% of organizations view data collected as having high or very high value

*ASTD Research Study, “The Value of Evaluation: Making Training Evaluations More Effective,” 2009

LEVEL 2 EVALUATION FACTS*


WHY THE DISCONNECT? ➤  Knowledge tests created are overly

simplistic

➤  Knowledge test administered too close to learning event

➤  Test items focus on recall not application

➤  The data collected is not used


WHATʼS THE SOLUTION ➤  Create valid, scientifically sound Level 2

knowledge tests that measure job application not just recall

➤  Include knowledge tests as part of learning program design and use for both evaluation and reinforcement

➤  Aggregate knowledge test score data and use to demonstrate program value as well as improve test item validity


SAMPLE LEVEL 2 MULTIPLE CHOICE

KNOWLEDGE

TEST


Whatʼs one thing that all 16 multiple

choice test questions have in

common?


They test for knowledge or recall – the

lowest level of cognitive ability


TEST CREATION TIP 1

➤  Test for application or understanding not

just knowledge


EXAMPLES ➤  What does TV stand for?

➤  What is the main function of a TV?

➤  What physical principle is used to display images on a TV?

➤  Your TV is not working properly. What is the most likely cause of the following set of symptoms:…?


You sent a detailed program design document to a group of SMEs to solicit their feedback, and have incorporated a number of their suggestions into the program design. This is an example of what element in the ADDIE model?

A. Analysis D. Implement

B. Design E. Evaluation

C. Development

EXAMPLE


10.   Effective listening is defined as?

A.  Attempting to understand the person speaking from his or her perspective and not your own.

B.  Encouraging another person to talk.

C.  Consolidating the important parts of a discussion.

D.  Creating a win-win communication situation.

WHATʼS WRONG WITH THIS?


➤  Keep response options equivalent in

length & writing style

TEST CREATION TIP 2


4.  A Collaborative Communication Style is also known as a?

A.  Win/Win communication style. B.  Low concern for self and high concern

for other communication style. C.  Engaging and outgoing communication

style. D.  Intermediate concern for self and high

concern for other communication style.



➤  Question should not reveal correct

answer

TEST CREATION TIP 3


4.  A Collaborative Communication Style is also known as a(n)?

A.  Win/Win communication style. B.  Low concern for self and high concern

for other communication style. C.  Engaging and outgoing communication

style. D.  Intermediate concern for self and high

concern for other communication style.

EXAMPLE


14.   What communication technique is used to consolidate what has been previously discussed and to move a conversation from one topic to another?

A.  Arguing

B.  Interrupting

C.  Summarizing

D.  Initiating



➤  All response choices must be plausible

TEST CREATION TIP 4


6.  A benefit of being an effective listener is?

A.  Facilitates getting to know another person better.

B.  Increases the likelihood of having a mutual exchange.

C.  Helps another person to get to know himself or herself better.

D.  All the above.



➤  Avoid “All the above/None of the above”

TEST CREATION TIP 5


12.   Which is not one of the four communication styles?

A.  Intuitive

B.  Avoiding

C.  Aggressive

D.  Collaborative



➤  Write questions same way material taught

TEST CREATION TIP 6


9.  A perception check is a? A.  Technique for moving a conversation

from the objective, where the focus is on facts, to the subjective, where the focus is on feelings.

B.  Technique for pulling together the important parts of one long conversation.

C.  Technique for helping someone to express his or her real self.

D.  Technique for conveying interest in what another person is saying.



➤  Include central idea and most of phrasing in

stem

TEST CREATION TIP 7


9. A Perception Check communication technique is used to? A.  Move a conversation from the

objective to the subjective. B.  Pull together the important parts of

one long conversation. C.  Help someone express his or her real

self. D.  Convey interest in what another

person is saying.

EXAMPLE


WHATʼS WRONG WITH THIS? 2.  ____________________ are used to report

specific, observable actions or details in an objective, non-evaluative manner.

A.  Behavior descriptions

B.  Open probes

C.  Evaluative statements

D.  Direct statements


➤  Write stem as a question – responses can

finish question, but donʼt use fill-in-the-

blank

TEST CREATION TIP 8


WHATʼS WRONG WITH THIS? 1.  Interpersonal communication is often

rendered ineffective due to the presence of communication barriers. What are the three categories of communication barriers?

A.  Environmental, Interpersonal and Intrapersonal

B.  Environmental, Interpersonal and Cultural C.  Structural, Interpersonal and Cultural

D.  Organizational, Intrapersonal and Structural


➤  Eliminate excessive verbiage or irrelevant information from the stem

TEST CREATION TIP 9


EXAMPLE 1.  What are the three categories of

communication barriers?

A.  Environmental, Interpersonal and Intrapersonal

B.  Environmental, Interpersonal and Cultural

C.  Structural, Interpersonal and Cultural

D.  Organizational, Intrapersonal and Structural


➤  Spread correct answers across A – D

TEST CREATION TIP 10


➤  Develop Level 2 Multiple Choice Tests that are:

•  Valid

•  Fair to the learner

•  Fair to the organization &

•  Test for understanding

OVERALL SUMMARY


FREE ARTICLES

Phillips, Ken, “Developing Valid Level 2 Evaluations”, Training Today, Fall 2009, pps. 6-8.

Phillips, Ken, “Eight Tips on Developing Valid Level 1Evaluation Forms”, Training Today, Fall 2007,pps. 8 & 14.


Ken Phillips

Phillips Associates

34137 N. Wooded Glen Drive

Grayslake, Illinois 60030

(847) 231-6068

www.phillipsassociates.com

[email protected]

cets 2011, ken phillips, slides for construct the ultimate multiple-choice test! stop measuring...

Documents