Generalized Mixed-effects Models Generalized Mixed-effects Models for Monitoring Cut-scores for for Monitoring Cut-scores for Differences Between Raters, Differences Between Raters,
Procedures, and TimeProcedures, and Time
Yeow Meng ThumHye Sook Shin
UCLA Graduate School of Education & Information StudiesNational Center for Research on Evaluation,Standards, and Student Testing (CRESST)
CRESST Conference 2004
Los Angeles
RationaleRationale• Research shows that cut-scores vary as a
function of many factors: raters, procedures, and over time.
• How does one defend a particular cut-score? Averaging several values, use of collateral information are current options.
• High-stakes accountability hinges on the comparability of performance standards over time.
• Some method is required to monitor cut-scores for consistency across groups and over time. (Green, et al)
Purpose of StudyPurpose of Study
• An approach for estimating the impact from procedural factors and rater characteristics and time.
• Monitoring the consistency of cut-scores across several groups.
Transforming Judgments into Transforming Judgments into Scale ScoresScale Scores
-4 -2 0 2 4480 540 600 660 720
0.0
0.2
0.4
0.6
0.8
1.0
Pro
bab
ility
Logit
Scale Score
Cut-Score
0.633 logits
619 scale-score points
Figure 1: Working with the Grade 3 SAT-9 mathematics scale
Performance DistributionPerformance Distributionfor Four Urban Schoolsfor Four Urban Schools
Figure 2: Grade 3 SAT-9 mathematics scale score distribution for four schools
400 450 500 550 600 650 700 750 8000.000
0.004
0.008
0.012School A32% Proficient
619
400 450 500 550 600 650 700 750 8000.000
0.004
0.008
0.012School B70% Proficient
619
400 450 500 550 600 650 700 750 8000.000
0.004
0.008
0.012School C19% Proficient
619
400 450 500 550 600 650 700 750 8000.000
0.004
0.008
0.012
School D32% Proficient
619
Potential ImpactPotential Impactof Revising a Cut-score of Revising a Cut-score
Revised Cut-score
(as fraction of sem)
school -1 -0.5 0 0.5 1
A 41% 37% 32% 29% 26%
B 78% 75% 70% 67% 63%
C 25% 23% 19% 15% 13%
D 40% 36% 32% 28% 25%
Table 1: Potential impact on school performance when cut-score changes
Data & ModelData & Model• Simulate Data for a standard setting study
design : a ramdomized block comfounded factorial design (Kirk, 1995)
• Factors of standard setting study
a. Rater Dimensions (Teacher, Non-Teacher, etc.)
b. Procedural Factors/Treatments
1. Type of Feedback (Outcome or impact Feedback- “yes” or “No”, etc)
2. Item Sampling in Booklet (Number of items, etc)
3. Type of Task (A modified Angoff, a contrasting group approach, or
Bookmark method, etc)
Treating Binary OutcomesTreating Binary Outcomes
1 for "pass"
0 for "fail",ijty
ln1
ijtijt
ijt
p
p
(2)
Binary outcome
(1)
Logit link function
(pass if rater j thinks, the passing candidate has a good chance of getting the ith item right in session t)
s= + jt sjt jts
K S
IRT Model for Cut-score - IIRT Model for Cut-score - I
Procedural Factors Impacting A Rater’s Cut-scores
(3)
Where s is the fixed effect due to session characteristics s
is random effect, which evolves over time ROUNDjt, and a function of rater characteristics, Xpj
jt
Item Response Model (IRT)
= - ijt jt ijtK d
(4)
Estimating Factors Impacting A Rater’s Cut-scores
(5)
0 1
0 00 0 0
1 10 1 1
jt j j jt
j p pj jp
j p pj jp
ROUND
X u
X u
0 1( , )j ju u are distributed bivariate normal
with means (0, 0) and variance-covariances
00 01
10 11
T
IRT Model for Cut-score - IIIRT Model for Cut-score - II
LikelihoodLikelihood
( , )jg T
(1 )( ; ) [1 ]ijt ijty yj j ijt ijt
t i
f y p p
(7)
Prior distribution of j
Conditional posterior of the rater random effects j is
( ; ) ( ; )
( , , )j j j
j
f y g T
h y T
where ( , , ) ( ; ) ( ; )j
j j j j jh y T f y g T
Condition on , y has probability
(6)
Joint marginal likelihood
( , , )jj
h y T (8)
= + jt sg sjt jts
K S
Multiple StudiesMultiple StudiesConsistency & StabilityConsistency & Stability
Procedural Factors Impacting A Rater’s Cut-scores for separate study g (g=1.2.3….,g)
Where sg is the fixed effect due to session characteristics s
is random effect, which evolves over time SESSIONjt, and a function of rater characteristics, Xpj
jt
(9)
Group Factors Impacting A Rater’s Severity
0 1
0 00 0 01 1
1 10 1 11 1
jt j j jt
G G
j gj g gj g p pj jg g p
G G
j gj g gj p pj jg g p
ROUND
GROUP GROUP X u
GROUP GROUP g X u
(10)
SimulationSimulationSAS Proc NLMixed
150 raters who are randomly exposed 4 rounds to STD setting exercise varying on 3 session factors.
Session Factor 1: Feedback type
Session Factor 2: Item Targeting in Booklet
Session Factor 3: Type of Standard Setting Task
Rater Characteristics: Teacher, Non-Teacher
Change over Round (time)
Selected Results Selected Results
• Model (reasonably) recovers parameters within sampling uncertainty across 3 studies.
• Average cut-score (All Teachers) for each rater group at the last Round is not significantly different from 619, while the first Round results were significantly different.
• Results from the model for multiple studies are similarly encouraging.
SuggestionsSuggestions
• Large-scale testing programs should monitor their cut-score estimates for consistency and stability.
• For stable performance scale, estimates of cut-scores and factor effects should be replicable to a reasonable degree across groups and over time.
• The model in this paper can be modified based on actual data so that we verify and balance out the variation due to the relevant factors of the study.
