copyright © 2004 educational testing service listening. learning. leading. using differential item...

Copyright © 2004 Educational Testing Service

Listening. Learning. Leading.

Using Differential Item Functioning to Analyze a State English-language Arts Assessment

Linda CookFred ClineEducational Testing Service


Relationship to DARA

• Investigating how tests work for students with disabilities now is one way to determine how to build more appropriate tests in the future.

• Analyzing tests as a whole and item by item can be done via Factor Analysis and Differential Item Functioning


Differential Item Functioning (DIF)

• DIF refers to a difference in item performance between two comparable groups of test takers

• DIF exists if test takers who have the same underlying ability level are not equally likely to get an item correct


Why DIF Procedures are Useful

• DIF studies first carried out on a frequent basis in 1960s– Evaluate ethnic differences– Identify and remove biased items

• Examples– Deaf and HH examinees– Test item draws on knowledge of popular

music


Some Issues Related to Using DIF Procedures For Students With Disabilities

• Definition of groups

• Sample Size

• Matching criterion

• Ability differences between groups

• Very little research focusing on DIF for students with disabilities


Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities

• Lewis, Green & Miller, 1999– Read aloud and extended time– 35 different groups with disabilities– More ELA items had DIF than Math

items


Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.)

• Bielinski, Thurlow, Ysseldyke, Freidebach & Freidebach, 2001– Read aloud accommodations

– 41 multiple choice grade 3 reading items

– 32 multiple choice grade 4 math items

– DIF for 30 reading and 7 math items

– Results indicated read aloud accommodation may affect comparability of items



• Bolt,2004– Read aloud accommodation is less

appropriate for reading tests than for non-reading tests

– Accommodations are more appropriate for students with sensory and physical disabilities than for students with cognitive disabilities



• Barton& Finch, 2004– Second edition of the TerraNova

– Language and math subtests for grades 3,5,8

– DIF for items with high amount of text provided via read aloud conditions to accommodated students

– Concluded some level of boost for students with disabilities with read aloud accommodation


Current Study (ETS/DARA)

• Grade 4 and Grade 8, State English Language Arts test

• Evaluated three criteria– Total Test– Reading– Writing

• Analyzed items in total test


Description of the Tests

• Grade 4 ELA test contains reading and writing strands for a total of 75 items• Reading subtest has three strands for a total

of 42 items

• Writing subtest has two strands for a total of 33 items


Description of the Sample

• Four groups of students– Students without disabilities– Students with LD who took the test

without an accommodation– Students with LD who took the test

with an accommodation defined by 504 plan or IEP

– Students with LD who took the test with a read-aloud accommodation


Comparison Groups Used for DIF Analyses

Reference Group Focal GroupWithout disabilities LD, no

accommodationsWithout disabilities LD, IEP/504

accommodationsWithout disabilities LD, read-

aloud accommodation

LD, no accommodations LD, IEP/504 accommodations

LD, no accommodations LD, read-aloudaccommodation


Characteristics of Groups Used for DIF Studies

Group Sample Size Mean Standard

Deviation

No Disability 30,225 48 14

LD, no Accommodation

9,045 29 12

LD, 504/IEP Accommodation

4,724 27 10

LD, Read-aloud accommodation

1,367 29 11


No. of DIF Items Identified Using Total Test as Criterion

Reference Group Focal Group B DIF C DIF

No Disability LD, No Accommodation

1R

No Disability LD, 504/IEP Accommodation

1R, 1W

No Disability LD, Read-aloud Accommodation

6R 1W

LD, No Accommodation

LD, 504/IEP

Accommodation


LD, Read-aloud

Accommodation

1R, 1W


No. of DIF Items Identified Using Reading Subtest as Criterion



1R, 3W 1W


2R, 4W 1W


6R, 10W 2W


LD, 504/IEP Accommodation


LD, Read-aloud accommodation

1R, 1W


No. of DIF Items Identified Using Writing Subtest as Criterion



5R 1R


5R, 1W 2R


8R, 2W 1W

LD, No Accommodation LD, 504/IEP Accommodation

LD, No Accommodation LD, Read-aloud accommodation

2R, 2W 1R


Summary of Results

• Criterion– Total test score most reliable– Writing subtest score least reliable– Total test may be multi-dimensional– Total test identifies least amount of DIF items– Reading and writing subtests identify similar

amounts of DIF– Using Reading as the criterion identifies

mostly writing items as having DIF and using writing items as the criterion identifies mostly reading items as having DIF


Summary of Results (cont.)

• When reference group is students without disabilities, students with disabilities who took test with accommodations showed more DIF than students with disabilities who took test without accommodations

• Read-aloud accommodations result in increased DIF

• DIF is decreased for accommodated groups if reference group is students with disabilities


Conclusions

• Choice of matching criterion impacts results

• Disability, alone, results in DIF• Accommodations result in DIF • Read-aloud accommodations result in the

most DIF• Accommodations specified in 504/IEP do

not result in DIF when reference group is students with disabilities who took test without accommodations

copyright © 2004 educational testing service listening. learning. leading. using differential item...

Documents