diagnostic classification models
TRANSCRIPT
PARCC Diagnostic Assessments forMathematics Comprehension:
A Diagnostic Classification Model Approach
Laine P. BradshawThe University of Georgia
June 24, 2015
• A brief introduction to diagnostic classification models (DCMs) » How are they different from the models that are typically used
in assessment?» Why are they particularly useful for diagnostic assessments?
• How did we use the DCM framework to develop PARCC’s diagnostic assessment system for mathematics comprehension in Grades 2-8?
• What new challenges in reporting exist when transitioning to a new psychometric framework?
Overview
Big Picture: Math Content to be Measured
Typical End of Grade Summative Test
• Traditional testing procedures measure an overallability in an area with a continuous latent variable
7th gradeMath Ability
Cluster 4 Item
Math ability is continuous
Responses to items are observed
The more math ability a person has the more likely he or she is to answer an item correctly
Traditional Measurement Models for Summative Tests
Cluster 2 Item
Cluster 1 Item
Cluster 3 Item
…
Cluster 2 Item
Traditional Testing and Classification Methods
• Information from Continuum: » Spencer has more math ability than Sue
» Hugh scored a 240 on the test
» Juan scored in the 70th percentile
• Diagnosis from Cut Score:» Sue scored below the cut score
» Sue is not proficient in 7th grade math
Not Proficient
7th Grade Math Ability
ProficientHIGHLOW
Question difficult to answer: What are Spencer’s weaknesses?
7th gradeMath Ability
Cluster 2 Item
Cluster 6 Item
Cluster 1 Item
Cluster 3 Item
…
Diagnostic Approach
Instead of measuring an overall math ability in 7th grade, we can break “math” down into a set of skills or attributes:
Cluster 1Cluster 2Cluster 3Cluster 4
Cluster 1
Diagnostic Approach
Cluster 4 Item
Cluster 2 Item
Cluster 1 Item
Cluster 3 Item
…
Cluster 2 Item
Cluster 2
Cluster 3
Cluster 4
Each cluster has two levels
A student can be in one of two groups, or levels, for each cluster• Higher group (masters; on-track)• Lower group (non-masters; needs attention)Masters are more likely than non-masters to answer items correctly.
Diagnostic Classification Models
Subtract
Add
Multiply
Divide
Masters Non-masters
• DCMs uses responses to items to place students into groupsaccording to multiple skills
–No cut score is used to put students into groups; the model is built to do that
Cluster 1
Cluster 2
Cluster 3
Cluster 4
Diagnostic Classification Models
Subtract Multiply DivideAdd
• Students receive attribute-specific feedback
• The model provides a probability each attribute is mastered• Notice there is no “score” in a traditional or grading sense
Spencer has mastered Cluster 1 and 2, but should improve his understanding of Cluster 3 and 4.
.84 .76 .35 .28
Cluster 1 Cluster 2 Cluster 3 Cluster 4
• Aligns with a standards-based view of achievement where mastery of all standards is monitored» Modeling multiple “dimensions” or traits» Instead of answering: Did this student meet the standards?» Answer: Which standards has the student met?
• DCMs provide valuable information with fewer data demands» Higher reliability per dimension than IRT/MIRT models» Potential to drastically reduce testing time » Quick and dirty categorical feedback
Benefits of Using DCMs
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
3 13 23 33 43 53 63 73 83 93
Rel
iabi
lity
Number of Items
DCM
IRT
DCM Reliability
8 ItemReliability
Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251-275.
Psychometric Approach for Mathematics Comprehension
Non-summative Assessments
Mathematics Comprehension
Example: Cluster Level Attributes in MathNumber Systems Learning Progression
Attribute 1: Divide Fractions
Attribute 2: Understand Rational Numbers
Attribute 3: Operate with Rational Numbers
Example: Cluster Level Attributes in MathNumber Systems Learning Progression• Create a separate diagnostic test for each individual
cluster
… ……
Divide Fractions
Understand
Rational Numbers
Operate with
RationalNumbers
Diagnostic Feedback to Students
Divide Fractions
Understand Rational Numbers
Operate withRational Numbers
Num
ber S
yste
m
Resu
lts
Diag
nost
ics
Example Student A
Example Student B
Example Student C
On-track Needs ImprovementDiagnostic
Key:
Mas
tery
Pro
babi
lity
00.10.20.30.40.50.60.70.80.9
1
Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways
Individual Cluster Test1 attribute ~8 items
Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways
Grade Level Test6 attributes~48 items
Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways
Learning Progression Test6 attributes~48 items
Key to Flexible Diagnostic Assessment System: One Test Per Cluster, Packaged in Different Ways
Grade Level Test6 attributes~48 items
Learning Progression Test6 attributes~48 items
Individual Cluster Test1 attribute ~8 items
• DCMs do not provide a score!» They provide classifications
• How will teachers and students interpret classifications?
• How will they interpret the probabilities of mastery?
• What is the best way to present the results so that interpretations are clear and accurate?
Challenges in Reporting
• Teachers can monitor the percentage of students who have mastered each cluster for a given grade level» Aggregate feedback
At the class level
• Can show the multidimensional mastery profiles for each student in the class
• Who has mastered each cluster?
At the class level
• Should the probability of mastery be provided? Where?» Analogous to
decisions for where to put standard errors on score reports for IRT-based assessments
At the student level
• DCMs are parametric, latent class models» Other examples of these models exist» For example, Bayes Nets
• Models in the same family produce the same student estimates—mastery classifications and probabilities of mastery» These models face similar challenges in reporting
Classification-based Reporting
