center for assessment considerations for the alternate assessment based on modified achievement...

Center for Assessment

Considerations for the Alternate Assessment based on Modified Achievement Standards

Briefing on a white paper commissioned by the New York Comprehensive Center in collaboration with the New York State Education Department

January 11, 2010

The contents of this publication were developed under cooperative agreement S283B050019 with the U. S. Department of Education. However, the contents do not necessarily represent the policy of the Department of

Education, and you should not assume endorsement by the Federal Government.

Center for Assessment 2

Introductions and Project Overview NYCC received a letter from USED informing

them that funds were available for a research project

NYSED agreed that an expert panel meeting culminating in a white paper on issues surrounding the AA-MAS would be useful

NYCC applied for the funds and contracted with NCIEA to support this work


Issues Facing NY and Most States New regulation without much discussion with states

led to many questions Should states spend limited resources to develop an

AA-MAS? Will it yield useful information to guide instruction? Is that the best way to support the learning of this

population? Would this test only be helpful to schools for accountability

purposes? Are there current best practices with this population

in assessment development that should be followed?

What does the research say?


NYSED Initial Questions Regarding the AA-MAS Which students are best served by this assessment? How different are they from the rest of the special

education population? What is an “appropriately challenging” achievement

standard? Which modifications make the most sense in the

context of the AA-MAS? How do the modifications affect the validity and

reliability of the interpretation? What is the credential that is most appropriate for

students participating in the AA-MAS and what does it lead to in terms of post-secondary potential?


Overview of Project NYCC contracted with the Center for Assessment to

assemble an expert panel and develop a white paper on issues concerning the development of an AA-MAS

NYSED provided guidance on important policy questions and recommended addressing issues in Filbin (2008) report

Expert panel met twice in person and several times over WebEx

Developed a 10-chapter report exploring many issues surrounding the conceptualization and development of an AA-MAS


Project ParticipantsProject Director: Larry Hirsch, New York Comprehensive

Center

Project Manager/Editor: Marianne Perie, Center for Assessment

New York State Department of Education: David Abrams, Assistant Commissioner

for Standards, Assessment, and Reporting

Candy Shyer, Bureau Chief of Test Development, Office of State Assessment

Rebecca Cort, Deputy Commissioner, Office of Vocational and Educational Services for Students with Disabilities

Expert Panel: Jamal Abedi, University of California, Davis Chris Domaleski, Center for Assessment Steve Dunbar, University of Iowa Howard Everson, Fordham University Claudia Flowers, University of North Carolina,

Charlotte Brian Gong, Center for Assessment Meagan Karvonen, Western Carolina University Suzanne Lane, University of Pittsburgh Scott Marion, Center for Assessment Jim Pellegrino, University of Illinois, Chicago Marianne Perie, Center for Assessment David Pugalee, University of North Carolina,

Charlotte Rachel Quenemoen, National Center on

Educational Outcomes Robert Rickelman, University of North Carolina,

Charlotte Katherine Ryan, University of Illinois, Urbana

Champagne Gerald Tindal, University of Oregon Cathy Welch, University of Iowa Phoebe Winter, Pacific Metrics


Authors and Reviewers NYSED and the Center for Assessment

identified chapter authors and reviewers and sent requests prior to the first expert panel meeting

Eleven expert panelists served as authors The remaining seven experts served as

reviewers Discussions were organized by section &

each reviewer was assigned a section Three reviewers read all chapters


Organizing Structure of Report Report is organized into three sections

Identifying and understanding the population Test Design: Understanding content and achievement

standards and incorporating appropriate item modifications Technical considerations and practical applications

Or… Should I develop an AA-MAS? How should I develop it? How should I evaluate it?


Organization (continued) 10 chapters within the three sections

1. IntroductionIdentifying & Understanding the Population2. Identifying the Population3. Standards-based IEPs4. Cognition of Low AchieversTest Design5. Reading and Math Content6. Test Development7. Standard SettingTechnical Considerations8. Comparability9. Validity10. Accountability and operationalization


Section I: Identifying and Understanding the Population Intro and first chapter provide a policy context

for the AA-MAS and a historical perspective on educating students with disabilities

Includes a discussion framework for state policymakers on assessment options and on improving student access to grade-level curriculum


Section I (continued) Second chapter describes how a standards-based

IEP supports an educational program that meets the needs of AA-MAS-eligible students, through access to grade-level curriculum using effective instruction; supports to address learner characteristics; and setting and monitoring goals to support a transition into grade-

level achievement. Third chapter describes relationships among

assessment, curriculum, and instruction, and conceptualizing assessment as a process of reasoning from evidence driven by theories and data on student cognition, and includes examples related to the AA-MAS.

11


Section II: Test Design Starts with a chapter describing content

domains for ELA and Math with a focus on ways to modify content so that we can measure greater depth with easier items

Next chapter discusses item and test development and reviews approaches to modification of items and assembly of test forms with respect to psychometric consequences and standards-based interpretations of proficiency for the AA-MAS population.


Section II (continued) Final chapter in this section focuses on

developing modified achievement standards Includes a discussion on how to define

“modified” proficiency, synthesizing information from the previous sections on cognition and test design, and provides ideas for setting cut scores on assessments with fewer items and lower sample sizes


Section III: Technical Considerations First chapter in this section discusses comparability

between AA-MAS and general assessments from different perspectives including content and construct, psychometrics, scale and score, linguistic structure, text features, and depth of knowledge.

The next chapter describes why and how to develop a validity argument using Kane’s argument-based approach as a framework for considering validity issues related to AA-MAS.


Section III (continued) The final chapter in this section describes

operational and accountability issues and is specifically geared towards policymakers

It focuses on the interrelationship of the AA-MAS to the existing state assessment and accountability system considering several practical, technical, and policy issues that must be considered when implementing a new program.


Other Features Other resources developed at the request of

NYSED List of additional web-based resources on

instructing and assessing students with disabilities Tool for policymakers that includes guiding

questions for state policymakers considering the development of an AA-MAS and references back to specific sections within the report

Glossary of terms related to special education, assessment, and federal policy


Overarching Themes Clear link between assessment, instruction, and student

cognition, Several of the chapters focus on the importance of bringing lessons

learned in studying the students and designing the assessments into the classroom.

Similar to accommodations, certain modifications will only be successful to the degree they are incorporated into a student’s daily instruction.

Important to develop a validity argument for this assessment early in the process and test the various assumptions throughout. Testing the assumption that a new assessment is needed is one of the

first important recommendations. Many chapters address ongoing collection and evaluation of validity

evidence to ensure that the development is in line with the expected goals.

Consider how to incorporate the recommendations into an existing assessment and accountability system Including how to work with current state content standards and grade-

level achievement standards.


Suggested Uses State policymakers should find this report

useful in determining whether or not developing an AA-MAS makes sense for their state, and if so, how.

Even in states with no intention of designing an AA-MAS, this report could be a useful tool for designing professional development activities.

Contains suggestions for several areas: special education, curriculum & instruction, assessment, and accountability


Lingering Concerns Still concerned about identifying right population

Kids are classified differently from one school to the next

We need to understand the need for such classifications and find ways to better classify them into instructionally useful categories

If AA-MAS modifications work well for these kids, why not expand the approach for all assessments Need more focus on universal design, better

accommodations, and aligning assessments with diagnostic approaches to learning.

19


Lessons Learned from the Report This report is applicable to multiple tests and

types of students – not just the “2%” population.

We need to consider how the AA-MAS fits in with the general assessment.

It highlighted the importance of an integrated test design, meaning we must consider assessment, curriculum, and instruction together in consideration of how tests will be used and validated.

20


Bigger Lessons Learned Key decision points should be identified and

informed by research and best practices. There is a value in triangulating federally-

funded research with states’ operational work: States need time to have thoughtful discussion to

process issues. This type of work increases professional

development of state department staff. It helps inform several areas like RTTT and allows

measurement research to inform policy.

21


Lessons Learned on Next Steps We need to strengthen our understanding of

learning progressions and integrate that understanding into classroom instruction and assessment development. Where do students with disabilities differ from

general population? Need to understand more than scope and

sequence Should provide information on next steps to help

strengthen nuances of student learning

22


Next Steps for New York Will not develop an AA-MAS under current

political context Need to see what happens with ESEA

reauthorization and RTTT In the meantime…

This work is helping us think through formative and interim assessment strategies.

We are also working with our colleagues in special education to rethink IEP design.

23

center for assessment considerations for the alternate assessment based on modified achievement...

Documents