1 producing your assessment question mark “software for creating and delivering assessments with...
TRANSCRIPT
1
Producing Your Assessment
Question Mark “Software for creating
and delivering assessments with powerful reports”
Copyright 2000 QuestionMark. All rights reserved.
2
Objectives
204/20/23
Compute your passing score
Define your passing score
04/20/23Copyright 2000 QuestionMark. All rights reserved.
At the end of this presentation, you will understand how to:
Beta test
Alpha test
3
Best practices for developing tests
04/20/23Copyright 2000 QuestionMark. All rights reserved.
1.Analyze the job 2.Document the test blueprint 3.Survey the blueprint 4.Develop items 5.Alpha test and revise items6.Beta test the exam 7.Select items and set the cut score 8.Go live with the test
4
Costly to revise Inexperienced test writers New test content or program New test design
Why test your test?
04/20/23Copyright 2000 QuestionMark. All rights reserved.
5
What does a stepped evaluation process give you?
Verify technical accuracy Ensure implement-ability Test content effectiveness Gauge organizational or industry
acceptance
04/20/23Copyright 2000 QuestionMark. All rights reserved.
6
Stepped evaluations for alpha & beta testing
04/20/23Copyright 2000 QuestionMark. All rights reserved.
Self evaluation
Expert review
Small group review- alpha test
Large group review – beta test
7
Inside evaluation steps
04/20/23Copyright 2000 QuestionMark. All rights reserved.
SelfRough version
Expert Sm.Group Lg.GroupPolished version
High interaction
Low interaction
Content & clarity
Usability & performance
8
What do you want to know?
04/20/23Copyright 2000 QuestionMark. All rights reserved.
Reaction ? Customer or trainee satisfaction
Learning ?? Knowledge learnedSkills developedAttitudes changed
Results ?!??! Productivity, quality, profits increased
Certify !!!!!!! Mastery achieved
9
Types of measurements
04/20/23Copyright 2000 QuestionMark. All rights reserved.
Reaction ? “Smile” sheet Attitude survey
Learning ?? Self-assessmentSkill/knowledge check
Results ?!??! Training ROI Behavioral impact
Certify !!!!!!! Certification exams Legal defensibilityConfidentiality policies
Low Stakes
High Stake
s
10
Relative scores Absolute scores Compromised scores Failure rates Arbitrary, but not capricious
Scoring terminology
04/20/23Copyright 2000 QuestionMark. All rights reserved.
11
Contrasting groups method
Experts set score arbitrarily
Experts’ consensus opinion
Ways to set passing scores
04/20/23Copyright 2000 QuestionMark. All rights reserved.
12
Contrasting Group method Expert performers Poor performers Consider the overlap Pick a score in the zone of acceptance
Low stakes score setting
04/20/23Copyright 2000 QuestionMark. All rights reserved.
13
Contrasting group sample
04/20/23Copyright 2000 QuestionMark. All rights reserved.
Score Hi Group Low Group
24+ xxxxxx
24 x
23 x
22 x
21 x x
20 x
19 xxx
<+=18 xxxxx
Zone of Acceptance
14
Use your judgment to assign points to test items
Estimate a percentage to pass Variations on estimating a percentage Compromise on a passing score Some practical hints
High stakes score setting
04/20/23Copyright 2000 QuestionMark. All rights reserved.
15
Identify what it would take to “just” pass
Rate & categorize characteristics Set required number of characteristics
Compute minimum points
Assigning points
04/20/23Copyright 2000 QuestionMark. All rights reserved.
16
Agree on “borderline” characteristics
Estimate the percentage of candidates
Compute average
Estimating a percentage
04/20/23Copyright 2000 QuestionMark. All rights reserved.
17
Supplying correct answers Omitting result data Working individually
Common variations on estimating
04/20/23Copyright 2000 QuestionMark. All rights reserved.
18
Both relative and absolute Define acceptable standards Compute curves Compute cut score
Compromise on your passing score
04/20/23Copyright 2000 QuestionMark. All rights reserved.
19
Minimum failure rate Maximum failure rate Minimum passing point Maximum passing point
Acceptable standards for the compromise method
04/20/23Copyright 2000 QuestionMark. All rights reserved.
20
50
40
30
20
10
0<45 50 55 60 65 70
Compromise curve
04/20/23Copyright 2000 QuestionMark. All rights reserved.
%Correct Score
% Fail
20% acceptable failure rate
Minimum fail score
Maximum passing score
21
Multiple judges Provide correct answers to items Provide result data Collect relevant demographics Avoid abstraction Perform additional surveying
Common themes & practical hints
04/20/23Copyright 2000 QuestionMark. All rights reserved.
22
Thank you for your attention
Thank you for your time and attention. We hope you have enjoyed learning about how to write test blueprints and test items.
We at Question Mark strive to improve our products and services. To let us know how you felt about this presentation, visit our North American home page and complete the Learning Center Survey: http://www.questionmark.com/home.htm#Surveys
Question Mark provides numerous presentations to help our customers, and prospective customers, successfully deploy computerized assessments. We provide public “teleseminars” that use a combination of conference calls and web technology to discuss computerized assessment issues. You can register for a public seminar by going to: www.questionmark.com/seminars/index.asp
For other material that you might find useful, please visit our Learning Center at: www.questionmark.com/learningcenter/
Thank you once again. This is the end of the presentation.
04/20/23Copyright 2000 QuestionMark. All rights reserved.