The ABC’s of The ABC’s of Pattern ScoringPattern ScoringThe ABC’s of The ABC’s of
Pattern ScoringPattern Scoring
Dr. Cornelia OrrDr. Cornelia Orr
Vocabulary• Measurement – Psychometrics is a
specialized application• Classical test theory • Item Response Theory – IRT
(AKA logistic trait theory)• 1 – 2 – 3 parameter IRT models• Pattern Scoring
General & SpecializedMeasurement• Assigning
numbers to objects or events
• Ex. – time, height, earthquakes, hurricanes, stock market
Psychometrics• Assigning
numbers to psychological characteristics
• Ex. – personality, IQ, opinion, interests, knowledge
Different Theories of Psychometrics
Classical Test Theory• Item discrimination
values • Item difficulty values
(p-values) • Guessing (penalty)Number correct
scoring
Item Response Theorya) Item discrimination
valuesb) Item difficulty
valuesc) Guessing (pseudo-
guessing) valuesPattern scoring
Similar constructs – Different derivations
Different Methodsof Scoring
Number-Correct Scoring
• Simple Mathematics
• Raw scores (# of points)– Mean, SD, SEM, % correct
• Number right scale
• Score conversions – Scale scores, percentile
ranks, etc.
Pattern Scoring• Complex Mathematics • Maximum likelihood
estimates– Item statistics, student’s
answer pattern, SEM
• Theta scale (mean=0, standard dev=1)
• Score conversions– Scale scores, percentile
ranks, etc.
Comparison: Number Correct and Pattern Scoring
Similarities• The relationship of
derived scores is the same
For example, a scale score obtained in a test corresponds to the same percentile for both methods.
Differences• Methods of deriving
scores• The number of scale
scores possible– Number right =
limited to the number of items
– IRT = unlimited or limited by the scale (ex. 100-500)
Choosing the Scoring Method
• Which model?• Simple vs. Complex?• Best estimates? • Advantages/Disadvantages?
Ex. – Why do the same number correct get different scale scores?
Ex. – Flat screen TV – how do they do that?
Disadvantages of IRT and Pattern Scoring
• Complex Mathematics – Technical– Difficult to explain
– Difficult to understand
• It doesn’t add up!
• Perceived as Hocus Pocus
Advantages of IRT and Pattern Scoring
• Better estimates of an examinee’s ability – the score that is most likely, given the student’s
responses to the questions on the test (maximum likelihood scoring)
• More information about students and items are used
• More reliability than number right scoring• Less measurement error (SEM)
Item Characteristic Curve (ICC)
Achievement Index (Theta)
Pro
ba
bili
ty o
f C
orr
ect
Re
spo
nse
-4 -2 0 2 4
0.0
0.2
0.4
0.6
0.8
1.0
Discrimination=1
Difficulty =0.5
Pseudo-Guess. =0.13
Examples5 Items (Effects of Item
Discrimination)No Type a b c1 MC 0.0250 300.000 0.2 2 MC 0.0200 300.000 0.2 3 MC 0.0150 300.000 0.2 4 MC 0.0100 300.000 0.2 5 MC 0.0050 300.000 0.2
4 examinees’ response patterns (1=correct)
Pattern SEM SS1234511100 39 30001110 46 27800111 61 25810011 94 260
Examples5 items (Effects of item
difficulty)No Type a b c1 1 MC 0.0150 250.000 0.1
2 1 MC 0.0150 275.000 0.1
3 1 MC 0.0150 300.000 0.1
4 1 MC 0.0150 325.000 0.1
5 1 MC 0.0150 350.000 0.1
4 examinees’ response patterns (1=correct)Pattern SEM SS1234511100 43 30001110 43 30500111 43 29910011 43 310
Missing easy items can result in a lower scores.