copyright © 2004, graduate management admission council ®. all rights reserved. 1 expected...

1Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Expected Classification Accuracy

Lawrence M. RudnerGraduate Management Admission

Council®

LRudner@gmac.com

2Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

I will discuss

Classification Accuracy with IRT

False Positives – False Negatives

Repeat opportunities

Lower operational cut scores

2-categories / n-categories

Easy estimate

3Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Classification table

classified mastertrue master

classified non-mastertrue master

classified mastertrue non-master

classified non-mastertrue non-master

4Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Example

5Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

False Positive

P c i( | )

is the area to the right of

6Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Illustration 1• Dataset

– 50 items: [a,b,c]=[1.43, 0.00, 0.20]– N(0,1): mean see= .393, r= .92– θc = -.25; 40th %ile

Table 2: Expected and true examinee classifications

    Expected Classification

    Master Non-Master

TrueMaster 53.9     5.0    

Non-Master 6.5     34.6    

Accuracy = 88.5%    

7Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Application 1• add 10 items with parameters

a=2.0, b=-.25 and c=0

:

Expected and true examinee classifications adding 10 optimal items    Expected Classification

    Master Non-Master

TrueMaster 55.4     3.6    

Non-Master 4.8     36.3    

Accuracy = 91.7%    

8Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Application 2

Expected and true examinee classifications using an adjusted operational cut score    Expected Classification

    Master(   c'>-.40)

Non-Master(   c'<-.40)

True

Master(  c>-.25) 56.2     2.7    

Non-Master(  c< -.25) 9.8     31.3    

Accuracy = 87.5%    

9Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Multiple categories

P ro b ( | ) ( ( ) ) ( ( ) )a b bse

ase

10Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Simplified estimate

sea

b ase se seb a a( )

11Copyright© 2004, Graduate Management Admission Council®.  All Rights Reserved.

Simplified Estimate

• 81.7% calculated correct classification

• 78.0% estimated correct classification

10-item 2001 Maryland State Performance Assessment Program Grade 8 Reading Test, Form A.