NCME 2009 1

Alternate Assessments of Modified Achievement Standards:Research on More Accessible & Less Difficult Items

Stephen N. Elliott Vanderbilt University

Designing & Evaluating Modified Items for Students with Disabilities:Research Results

NCME 2009 2

Project, Partners & Presenters

CAAVES = Consortium for Alternate Assessment Validity and Experimental Studies

(USDE funded; October 2006 - March 2009; 6 states involved)

Partners = AZ, HI, ID, & IN + Vanderbilt Measurement Group + Discovery Ed. Assessment

Presenters Andrew Roach, Georgia State University Ryan Kettler, Vanderbilt University Michael Rodriguez, University of Minnesota

NCME 2009 3

Terminology

Modification: alternations or adjustments of test items to facilitate access.

Appropriate modifications Remove extraneous material, Maintain the same depth of knowledge (DOK), Do NOT change the grade-level construct being

measured, and Increase the validity of the inference from the test score.

NCME 2009 4

Session Goals: Federal Policy, Item Modifications, & Results

Briefly review AA-MAS policy from NCLB

Review item modification development approaches & methods

Discuss results and share observations about the findings and the implications for the design and use of alternate assessments of modified achievement standards

NCME 2009 5

Policy Context for Our Study

Our research has the potential to inform current testing practices and policies concerning students with disabilities who have experienced persistent academic difficulties and poor performance on statewide assessments.

USDE Regulations 34 CFR Part 200 (2007)

NCME 2009 6

2% Alternate Assessment

Recent (i.e., 2007) NCLB regulations have permitted states to develop an alternate assessment based on modified academic achievement standards (AA-MAS) and report proficiency for 2% of the population based on the results of this new test.*

The assessment must test grade-level content standards.

To be eligible for participation in the 2% test, students must:

Have an IEP;

Be receiving high-quality instruction at grade-level;

Have a history of poor test performance;

Have persistent academic difficulties.

NCME 2009 7

CAAVES Project Goal #2

Goal #2 of the CAAVES Project is to “investigate feasibility of item modification strategies for future alternate assessments.”

This goal was accomplished by (a) developing a common set of test items from existing reading and mathematics tests using modification principles that facilitate reading access and valid responses and (b) using a computer-based delivery system to experimentally examine student preferences, score comparability, and item statistics of the modified items for students with and without disabilities.

NCME 2009 8

To Accomplish Goal #2

We proposed and completed the following….. Modified a common set of existing reading and math items to

create items designed to be more accessible and still measure the same grade-level content as the original items.

Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties.

Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability.

NCME 2009 9

Original Motivating Questions

We wanted to answer the following questions about item modifications:

1. Will modifications in testing conditions change the skill being measured?

2. Will taking the test under modified conditions change the resulting scores?

3. Will non-disabled examinees benefit if allowed the same modifications?

NCME 2009 10

Participation Criteria for Students with Disabilities

NCME 2009 11

Multi-State Sample

NCME 2009 12CAAVES Item Mod Data 2008 12

Order of Forms and Conditions

Students were

randomly assigned to

one of 36 possible

reading and math tests

comprised of 39 items

that represented three

types of multiple choice

items: unmodified,

modified, and modified

with reading support.

NCME 2009 13

TAMI: A Tool to Guide Item Modifications

The Test Accessibility and Modification Inventory (TAMI; Beddow, Kettler, & Elliott, 2008) was developed as a decision-making tool to facilitate the analysis of new and existing tests and test items with the purpose of enhancing their accessibility.

The TAMI was influenced by 4 primary areas of study:1. Universal design principles;2. Cognitive load theory;3. Research on test and item development; and4. Guidance on web and computer accessibility.

NCME 2009 14

Anatomy of an Item

Stimulus

Visual

Stem

Answer Choiceskey (B) and

distractors (A & C)Page Layout

X

X

NCME 2009 15

Passage / Item Stimulus

1. Passage / Item Stimulus1.1 Passage and/or item stimulus vocabulary and sentence complexity are grade-appropriate.1.2 Paragraphs are appropriate in length.1.3 Each line in poems is numbered.1.4 Idioms or jargon are avoided (unless they are defined and related to the item content).1.5 Sentence structure is as simple as possible given the construct being measured.1.6 Text includes only words essential for responding to the item(s), with minimal extraneous verbiage.1.7 Essential words or vocabulary terms use bold font to facilitate identification.1.8 Readability analyses indicate appropriate grade level.1.9 Passage or stimulus does not require construct irrelevant knowledge and skills for item response.1.10 Passage and/or stimulus is viewable on the same page as visuals, item stem, and answer choices.1.11 Other: ___________________________________________

NCME 2009 16

Item Stem

2. Item Stem2.1 Item stem reflects intended content standard(s) and/or

objective(s).

2.2 Text includes all requisite information for responding.

2.3 Text includes only words essential for responding to the item, with

minimal extraneous verbiage.

2.4 Readability analyses indicate appropriate grade level.2.5 Item

vocabulary and sentence complexity are grade-appropriate.

2.6 Item stem is as direct as possible and uses the active voice.2.7 Idioms

or jargon are avoided (unless defined and related to the item content).

2.8 Essential words or vocabulary terms use bold font to facilitate

identification.

2.9 Item stem is positively worded (i.e., avoiding not questions).

2.10 When the item references a specific part of a passage or poem, the

relevant sentence or excerpt is included with the item stem.

2.11 Other: ___________________________________________

NCME 2009 17

Visuals

3. Visuals13.1 Visuals are included when necessary.

3.2 Visuals are relevant to essential item content.

3.3 Visuals are connected to the text (e.g., pictures, graphics, direction

icons).

3.4 Visuals clearly represent intended images.

3.5 Visuals use colors with sufficient contrast.

3.6 Visuals convey information using more than distinctions between

colors.

3.7 Essential words or vocabulary terms use bold font to facilitate

identification.

3.8 Visuals include labels and supportive text if necessary (e.g., titles,

row and column headers for graphs).

3.9 Circles, arrows, or highlighting are used for important elements of

visuals.

3.10 Supportive text is located close to related visuals.

3.11 When visuals are self-explanatory, words are not added.

3.12 Other: ___________________________________________

NCME 2009 18

Answer Choices

4. Answer Choices4.1 Text includes only words essential for responding

to the item, with minimal extraneous verbiage.

4.2 Answer choices are about equal in length.

4.3 All distractors are plausible (“attractors”).

4.4 All answer choices (of a multiple choice item) are necessary.

4.5 No distractors could be eliminated without reducing the validity of the

inference(s) intended by the item.

4.6 The order and construction of the answer choices are logical and

deliberate.

4.7 Other: ____________________________________________

NCME 2009 19

Page / Item Layout

5. Page / Item Layout5.1 All text is printed in standard typeface, using

a minimum of 12-point text.5.2 There is sufficient space between lines (leading).5.3 There is high contrast between text and background colors.5.4 All text can be read in a traditional, left-to-right, top-to-bottom way, with consistent left-margins.5.5 Text is formatted appropriately with regard to breaks and unbroken blocks.5.6 The entire item (stimulus, visuals, stem, and answer choices) is visible on the same page/screen.5.7 Page includes ample white space to prevent the item from appearing cluttered.5.8 Right margin is unjustified (staggered).5.9 Visuals are integrated into the item stimulus and item stem, rather than placed off to the side.5.10 Other: ___________________________________________

NCME 2009 20CAAVES Item Mod Data 2008 20

Item Summary Reports: An Example

NCME 2009 21

Group by Condition Overview

In the remainder of this session we focus on two sets of initial analyses: (1) group performance comparisons on reading and math tests and (2) item difficulty and distractor analyses.

NCME 2009 22

Thanks!

Thank you very much for your time and joining us for this session.

Please provide follow-up questions and suggestions in writing to:Steve.Elliott@vanderbilt.edu

http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml