Establishing the Validity of Test Accommodations for Students with

Disabilities: A Collaborative of State-based

Research

CTEAG Project Summary of Accomplishments and

Dissemination of Final ProductsJohn F. Olson, Ph.D.

Principal Investigator & Project Coordinator,

CTEAG Project

CCSSO NCSA and Joint ASES-TILSA MeetingDetroit, MI

June 23, 2010

CTEAG Project - OverviewGoals • Conduct research that focuses on validity of

test accommodations and interpretations of scores from accommodated state assessments

• Use a common research design to conduct the studies and to analyze data for comparisons across states and accommodations

• Build a shared body of research-based validity evidence on test accommodations across multiple states

• Share findings from project that provide states with additional evidence (e.g., to submit to the USED for peer reviews) in support of their decisions on the use of various accommodations

2

CT EAG Project Structure• Lead State – Connecticut

• Other Participating States – Kentucky, Michigan, and Nevada

• Additional State Input and Involvement – from ASES and TILSA members

• Project Management Team – group of SCASS advisors, experts, and consultants that oversaw and managed all validity studies and reporting activities

• CCSSO -- provided logistical support, staffing, and consultants to do tasks for the project

CTEAG Research Questions1. Does the accommodation affect item

and test form statistics?

2. Does the accommodation change the content structure of the test, i.e., is there an impact on the test structure under accommodated and standard administrations?

3. Does the accommodation increase performance of students? Does it increase performance more for SWD than it does for SWOD?

CT EAG – Validity of Accommodations Project

 State Participation – Details of Studies  

5

Topics/Issues CT #1 CT #2 KY MI NVAccommodation Computerized

text reader (computer voice)

Computerized text reader with human voice and other mods made since CT#1

Read aloud Enhanced directions and fewer response options

Read aloud by a trained individual (e.g teacher or aide)

Grade 7 5 4 6 7

Content Area Reading comprehension

Math Reading ELA Math

Administration:Test DateStudy DesignItems/Forms development

May 2008.Computer-based test forms with 20 MC items per form.

Spring 2009.Computer-based test forms.Used old secure math test with 26 MC items per parallel form.

Mar 2009.Developed forms from a pool of non-state items that were similar in content to KY forms with 22MC items per form.

Oct 2008 and Jan. 2009.Used released items – 24 MC and 1CR per form.

Nov-Dec 2008. Used 20 MC items per form.

Summary of Results

John Olson Establishing the Validity of Test Accommodations June 2009

6

• In most cases, accommodations did not alter item stats (difficulty, discrimination)

• Accommodations also did not change test form stats/characteristics (mean item difficulty, reliability)

• Accommodations did not have an effect on content structure of test forms

• Accommodations did not change dimensionality of tests

Summary of Results (continued)

John Olson Establishing the Validity of Test Accommodations June 2009

7

• Overall, results from ANOVA were mixed• Accommodations increased performance

of students in KY study; enhanced directions increased performance in MI

• Differential boost effect found in some cases, i.e., SWD scores increased more than SWOD in CT#2, KY, and MI studies

• Key areas of focus – • Differential boost effects • Increasing the accessibility of the

assessment

CTEAG Project - Accomplishments

• Assisted and guided states in conducting 5 different validity studies on test accommodations

• Analyzed all data from the studies and reviewed results with the PMT and states (and sometimes with states’ vendors)

• Summarized results in a series of detailed state reports

• Shared updates and information on progress of project at 8 ASES and TILSA meetings from 2007 to 2010

• Completed and disseminated 3 reports and a Project Database

• Presented information on the project at 4 national conferences and meetings (2007-2010)

• Conducted 3 evaluations of the work, surveying states, and summarizing the info in an Evaluation Report

C

8

CTEAG Final Reports/Products• Guidebook for Test Accommodations

Validity Studies --provides overall summary of results/findings and guidance to states on validity studies in a nontechnical manner [report was distributed to ASES and TILSA in Feb. 2010]

• CTEAG Technical Report – provides technical information on the project and details of each state’s study, e.g., designs, methods, analyses, statistics, results, etc.

9

CTEAG Final Reports/Products• Lessons Learned from State Validity

Studies – provides information on the background and context in which the validity studies were conducted, issues that states may need to consider, and lessons learned by the four participating states

• CTEAG Project Database -- provides additional info from the five studies, such as, samples, variables, data files, and statistical data for use in further research

10

Dissemination Activities• Presentation Session at 2010 NCSA on

June 22−State participants present results and

perspectives from conducting validity studies−Representative from USED discusses

importance of findings from federal perspective

• Dissemination Workshop at Joint ASES-TILSA meeting on June 23−Share all remaining products and summary of

accomplishments−State participants discuss lessons learned and

how data and findings can be used to benefit states

−Expert advisors and USED representative discuss implications at national level

11

Final Dissemination Activities• Distribute the reports/products widely to

states, the USED, and others

• Place PDFs of all final products on websites (CSDE , CCSSO/EAG)

• Issue formal announcement (press release?) and Chiefline newsletter article

• Coordinate with project partners (NCEO and NASDSE) to share information with others

• Submit final report on project to USED12

Questions &

CommentsFor more information, contact:

John F. OlsonPresident/Founder

Olson Educational Measurement & Assessment Services

617-965-1490 (home/office)

210-859-5593 (cell)jmclkolson@yahoo.com