Accommodations Research & Universally Designed Assessments: Where We’ve Been & Where We’re

Going

Sandy ThompsonChristopher JohnstoneAmanda Blount Morse

National Center on Educational Outcomes

Goals Define and describe the

purposes of test accommodations

Update and summarize what we know from research on accommodations use

Provide direction for future research on the effects of accommodations

Goals

Define “universally designed” assessments within the current context of large-scale assessments and accountability

Identify elements of universally designed assessments

Identify relevant research

Educational Accountability Driven by federal law states have

implemented statewide accountability systems.

One of the most challenging tasks has been determining appropriate ways to include students with disabilities in statewide assessments.

One of the most feasible ways to increase the participation of students with disabilities in assessments is through the use of testing accommodations.

What are Assessment Accommodations?

Definition – changes in assessment materials or procedures

Purposes To provide students with disabilities access

to assessments, so that more can participate.

Allow student’s to demonstrate their knowledge and skills to be assessed rather than the student’s disability.

To comply with federal and state regulations.

Types of Accommodations

SettingStudy carrelSmall groupIndividualized

PresentationRepeat directionsLarge print editionBraille edition

TimingExtended timeFrequent breaksUnlimited time

ResponseMark test bookletWord processorUse references

SchedulingSpecific time of daySubtests in different orderAcross multiple days

OtherTest preparationOut-of-levelMotivational cues

Accommodations:The National Picture

All states have accommodation policies.

States vary in terms of which accommodations they consider “standard” and “non-standard.”

Accommodation use is on the rise. About 50% of students with learning

disabilities received an accommodation during testing.

Summary of Research on the Effects of Test

Accommodations

The first effort to summarize all currently available research on the topic of assessment accommodations was written by Tindal & Fuchs and published in 1999.

The current analysis was conducted to summarize research on the effects of test accommodations published from 1999 through 2001.

Summary of Research on the Effects of Test

Accommodations Forty-six research studies were reviewed. All of the studies are empirical – include

an analysis of data. The studies have the following

characteristics: Purpose Types of assessment, content areas, and

accommodations Participants Research Design Findings Limitations

Purpose of the ResearchMost commonly cited purposes of research Determine the effect of the use of

accommodations on test scores of students with disabilities

Investigate the effects of accommodations on test score validity

Study institutional factors, teacher judgment, or student desirability of accommodation use

Examine patterns of errors across items or tests

Types of Assessment, Content Areas, and Accommodations

The majority of the studies used criterion referenced tests.

Mathematics was assessed in half of the studies, and reading/language arts was assessed in about one third of the studies.

Presentation accommodations were investigated most frequently. “Oral presentation" was examined in almost

half of the studies.

Participants in Reviewed Research

The number of participants ranged from 3 to nearly 21,000.

The largest number of studies included elementary school students.

Learning and cognitive disabilities were most frequently investigated among studies that documented participants types of disabilities.

Research Design

The studies were identified as representing one of four group research designs, a single subject research design, or a non-experimental or other design.

Over one third of the studies applied non-experimental or other designs to the study of accommodations effects.

Findings from Reviewed Research Three accommodations showed a

positive effect on student test scores across at least four studies: Computer administration Oral presentation Extended time

However, additional studies on each of these accommodations also found no significant effect on scores or alterations in item comparability.

Findings from Reviewed Research

All of the meta analyses of accommodated conditions found a positive effect on scores.

All of the studies examining differential item functioning (DIF) found some items that exhibited DIF under accommodated conditions.

Limitations Cited in Reviewed Research

Frequently cited limitations were: Unknown variations among

students included in the study Sample sizes too small to provide

adequate statistical support Nonstandard administration of the

accommodations across proctors and schools

Recommendations for Future Research

Recommendations cited by researchers:

Replicating the research for validation and generalization

Investigating associations to specific disabilities

More detailed non-experimental studies be conducted to provide richer data

Increase researcher control of the testing process

Study larger groups of students

Conclusions

Observations and recommendations: Clear definition of the constructs

tested Greater clarity in accommodations

needed by individual students Desirability and perceived usefulness

of accommodations by students themselves- the “End Users”

Consider Universal Design

Universal Design Research? Pieced together from:

Specific accommodations research

Policy statements Other fields (vision, ergonomics,

graphic design, Universal Design of architecture research)

Universally designed assessments:

are designed from the beginning to be accessible and valid for the widest range of students

provide optimal standard assessment conditions

Who Benefits?

Universal design does not apply exclusively to people with disabilities or limited English proficiency

It applies to all individuals, with wide ranging characteristics

Think About Universal Design in Architecture and Tool

Design

Curb cuts and ramps

Elevators that talk to you

Door handles rather than knobs

Special pen shapes that are easier to hold

Remember This? Official Ballot, Palm Beach County,

Florida

Elements of UD Assessments

Inclusive assessment population

Precisely defined constructs Accessible, non-biased items Amenable to accommodations

Elements of UD Assessments

(continued)

Simple, clear, and intuitive instructions and procedures

Maximum readability and comprehensibility

Maximum legibility

Preliminary Research in Universal Design

Sample of 230 students taken from four schools in US Southwest.

Two schools were “town” schools (pop. 20,000) and two were “rural” schools.

Students chosen from sixth grade teams that had populations of students with disabilities.

Research Design Two tests were created, one from

sample statewide test items, the other re-designed using UD principles.

Each student took both tests. Students randomly assigned to take a

particular test first to prevent practice effect.

Constructs held constant for each item.

Community Involvement

Advisory Board trained in principles of Universal Design and asked to comment / suggest improvements based on their perspectives.

Team consisted of three parents of children in special education program (one Navajo, one Latina, one Anglo) and one community member with dyslexia.

Sample Item

Ramón is building a doghouse. He wants the roof of the doghouse to be at an angle that is more than 90° but less than 110°. Which angle below could he use for the roof?

A. B.

C. D.

Revised Item

Which angle is more than 90° and less than 110°?

A. B.

C. D.

What Changed?? Design element #2: construct more precisely

defined. Design element #3: bias eliminated (dog

house, Ramón) Design element #4: “built in

accommodations” – un-timed, students circled answer on paper, did not bubble

Design element #5: simple instructions and procedures

Design element #6: more comprehensible language, underlined key words

Design element #7: larger font

Results Means of two tests were compared and

t-tests performed. A difference of 8.16 (1.67 sig.) was

found between means, a statistically significant finding.

Effect size calculated using Cohen’s d. Effect of design = .061 (or 6/10 Standard Deviation difference) – a “moderate effect”

Why? Students with largest difference between

two tests were interviewed to determine difference for them.

Students noted that: more direct language made it easier for them to “understand” items and unlimited time helped them to “think better” about items. Students also said they “remembered” content better on UD test.

“Simplex Sigillum Veri”“The simple is the seal of the true”

Tests that remain true to constructs, are easy to understand, and contain language that is accessible to all will give the truest readings of what students do and do not know. Universal Design does not mean “dumbing down” a test.

What Have We Learned? Design matters!! How a test is designed

may effect how a student scores on that test.

Items that are better designed appear to aid students that are English Language Learners and with disabilities “show what they know” better.

This leads to more valid assessment of traditionally “under-performing” students.

Development Techniques for

Universally Designed

Assessments

“Think aloud” Recently interviewed 90 students using think aloud protocol

4th and 8th grade Used multiple choice and constructed response items from state math test

Logistics

10 researchers 5 days Worked in pairs All sessions videotaped Primary accommodations

included oral administration and sign language interpretation

Student Characteristics

Grade 4Grade

8

Learning Disability 10 10

Deaf/Hard of Hearing 10 10

Mild Cognitive Impairment 5 5

English Language Learner 10 10

No Disability 10 10

Overall Observations

Students who were confident of content did not have problems with design

Students who had no idea how to solve the problem did not have problems with design

Students “in the middle” – not sure of content, some reading difficulty, design made a difference

Examples of Student Perceptions

Many students didn’t see one of the cities on a map

The name of one of the cities was “Independence” - uncommon meaning

Box between top and bottom of item – some students did not read entire item

Sign for parallel gave away the answer Some students read fraction 3 5/8 as

“35 divided by 8” Students unfamiliar with settings –

“Glee club does number,” “fitness club”

Other Observations Some students got a lot of “help” from

sign language interpreters and teachers who wanted to make sure they understood the problem – what happens on test day?

Some student forms reported need for oral administration when they could clearly read the items independently – are some accommodations inappropriate?

For English language learners - is oral administration in English an appropriate accommodation on a Math test?

Should sign language interpreters have a script to follow so they don’t “give away” some answers?

Future Plans

Interview students with vision impairments Partner with American Printing House for the

Blind Explore differential item analyses

Partner with researchers from CTB/McGraw-Hill Produce short awareness video

Partner with Institute on Community Integration at University of Minnesota

Design and pilot training for item reviewers and item developers Partner with Missouri Department of Education

More Information…

Visit: http://education.umn.edu/nceo or Search for NCEO

Web site includes: Topic introduction Frequently Asked Questions Online and Other Resources