modified achievement tests for students with disabilities: design strategies and experimental...
TRANSCRIPT
CAAVES Item Modification 2008 1
Modified Achievement Testsfor Students with Disabilities:Design Strategies and Experimental Results
Stephen N. Elliott Vanderbilt University
CCSSO’s National Conference on Student AssessmentJune 2008
CAAVES Item Modification Study 2008 2
Project, Partners & Presenters
CAAVES = Consortium for Alternate Assessment Validity and Experimental Studies
(USDE funded; October 2006 - March 2009; 6 states involved)
Partners = AZ, HI, ID*, IN, MS, & NV + Vanderbilt Measurement Group + Discovery Ed. Assessment
Presenters Beddow & Kettler, Vanderbilt University Palmer, DEA Roach, Georgia State University Rodriguez, University of Minnesota Compton (ID), Bruen (AZ), Hinton (HI), & McGrath (IN)
CAAVES Item Modification Study 2008 3
Session Goals: Design & Data Review w/ Application Focus
Review item modification development methods, the resulting items, and suggested steps to improve the effectiveness of these strategies for assessing students with persistent academic difficulties.
Review Reading (Total, Vocabulary, & Comprehension) for 3 groups of students who took a 39 item test with 3 different item conditions. Also review Math (Total, Numbers, & Data) for the same students.
Discuss results and share observations about the findings and the implications for the design and use of alternate assessments of modified achievement standards.
Hear state assessment leaders perspectives on this study and their take-away lessons.
CAAVES Item Modification Study 2008 4
Policy Context for Our Study
Our research has the potential to inform current testing practices and policies concerning students with disabilities who have experienced persistent academic difficulties and poor performance on statewide assessments.
USDE Regulations 34 CFR Part 200 (2007)
CAAVES Item Modification Study 2008 5
CAAVES Project Goal #2
Goal #2 of the CAAVES Project is to “investigate feasibility of item modification strategies for future alternate assessments.”
This goal was accomplished by (a) developing a common set of test items from existing reading and mathematics tests using modification principles that facilitate reading access and valid responses and (b) using a computer-based delivery system to experimentally examine student preferences, score comparability, and item statistics of the modified items for students with and without disabilities.
CAAVES Item Modification Study 2008 6
To Accomplish Goal #2
We proposed and completed the following….. Modified a common set of existing reading and math items to
create items designed to be more accessible and still measure the same grade-level content as the original items.
Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties.
Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability.
CAAVES Item Modification Study 2008 7
Original Motivating Questions
We wanted to answer the following questions about item modifications:
1. Will modifications in testing conditions change the skill being measured?
2. Will taking the test under modified conditions change the resulting scores?
3. Will non-disabled examinees benefit if allowed the same modifications?
CAAVES Item Modification Study 2008 8
Participation Criteria for Students with Disabilities
CAAVES Item Modification Study 2008 9
Multi-State Sample
CAAVES Item Modification Study 2008 10CAAVES Item Mod Data 2008 10
Order of Forms and Conditions
Students were
randomly assigned to
one of 36 possible
reading and math tests
comprised of 39 items
that represented three
types of multiple choice
items: unmodified,
modified, and modified
with reading support.
CAAVES Item Modification Study 2008 11
Psychometric Outcomesto be Examined
CAAVES Item Modification Study 2008 12
Questions & Evidence from Analyses
1. Will modifications in testing conditions change the skill being measured?A. Content analysis by panelB. DIF analysisC. Factor analyses (across groups for same condition and within groups for different conditions)C. Depth of Knowledge of items (Compare mean DOK for 3 conditions)
2. Will taking the test under modified conditions change the resulting scores?A. MANOVA with groups and conditions as IVs & Reading and Math Scores as DVsB. Effect size calculations (comparing modified conditions to the unmodified condition for the total sample and each of the 3 groups)C. Item difficulties (Compare mean item difficulties for 3 conditions)
3. Will non-disabled examinees benefit if allowed the same modifications?A. MANOVA
B. Chi-square analysis (use 50% on Unmodified as cut score for Proficiency level) C. Descriptive analysis of post-assessment survey
4. Do item modifications change key psychometric characteristics of items or the test?A. Reliability estimates (Cronbach alphas and test-retest)B. DIF analysis
CAAVES Item Modification Study 2008 13
Group by Condition Overview
In the remainder of this session we focus on two sets of initial analyses: (1) group performance comparisons on reading and math tests and (2) item difficulty and distractor analyses.
CAAVES Item Modification Study 2008 14CAAVES Item Mod Data 2008 14
Item Summary Reports: An Example
CAAVES Item Modification Study 2008 15
Thanks!
Thank you very much for your time and joining us for this session.
Please provide all follow-up questions and suggestions in writing to:Steve Elliott at Vanderbilt