considerations for transitioning students from operational ... · considerations for transitioning...
TRANSCRIPT
National Center on Educational Outcomes (NCEO)
Considerations for Transitioning Students from Operational 2% Assessments to General
Assessments: Challenges for States
Moderator: Martha Thurlow (National Center on Educational Outcomes)
Presenters: Sheryl Lazarus (National Center on Educational Outcomes) Wendy Stoica (Ohio Department of Education) Trinell Bowman (Maryland Department of Education) Susan Weigert (U.S. Department of Education, Office of
Special Education Programs)
Discussant: Marianne Perie (CETE, University of Kansas)
National Conference on Student Assessment National Harbor, MD
June 20, 2013
Lessons Learned Chapter Authors • Parker, Gorin, and Bechard • Perie, Fincher, Payne, and Swaffield • Elliott, Kettler, Zigmond, & Kurz • Lazarus & Thurlow** • Elliott, Rodriguez, Roach, Beddow, Kettler, and Kurz • Cohen, Danielson, Stoica, Wothke, and Zhang** • Bechard • Bowman** • Dean and Roberts • Loving-Ryder and Siler • Zigmond, Kloo, Lemons & Lupp • Nagle and Cameto
National Center on Educational Outcomes (NCEO)
.
• Projects funded by the U.S. Department of Education in 2006-2007 under three funding sources (General Supervision Enhancement Grants, Enhanced Assessment Grants, and Supplemental Funding) compiled their findings in chapters that discussed the lessons learned from their studies on alternate assessments based on modified achievement standards (AA-MAS).
• This presentation highlights lessons from three chapters, plus perspectives from OSEP
Lessons Learned in Federally Funded Projects That Can Improve the Instruction and Assessment of Low Performing Students with
Disabilities (Thurlow, M., Lazarus, S., & Bechard, S. (Eds.))
Available at: www.nceo.info
National Center on Educational Outcomes (NCEO)
Sheryl Lazarus
.
The Characteristics of Low Performing Students with Disabilities
Sheryl Lazarus
National Center on Educational Outcomes
(NCEO)
Email: [email protected]
Demographic Characteristics of Low Performing Students with Disabilities
• 4 projects looked at demographic characteristics of low performing students. More likely to be: – Male – From a racial/ethnic minority – From a low socio-economic background – Have ELL status
• Both students with and without disabilities were low performing.
National Center on Educational Outcomes (NCEO)
Disability Categories • 4 projects looked at the disabilities categories of students
who might be candidates for an AA-MAS. • All found that a majority of the students had specific
learning disabilities. • Other categories:
o Intellectual disabilities o Speech and language impairments o Other health impairments o Emotional/behavioral disabilities o Autism o Other disability categories
National Center on Educational Outcomes (NCEO)
Opportunity to Learn
4 projects looked at whether students who were candidates for the AA-MAS had the opportunity to learn the content. All found that some students may not have had
the opportunity to learn.
National Center on Educational Outcomes (NCEO)
Learning Characteristics/Barriers 10 projects found one or more of these learning characteristics/barriers:
• Difficulty interacting with print • Difficulty solving problems that require multi-step solutions • Easily distracted • High vocabulary load • Lack of availability of needed accommodations • Limited meta cognition • Need clarification of instructions • Poor organizational skills • Self-monitoring skills • Slower work pace • Text structure (passage length and formatting) • Working memory capacity
National Center on Educational Outcomes (NCEO)
Wendy Stoica
National Center on Educational Outcomes (NCEO)
AA-MAS Test Development: Item Modifications
Wendy Stoica June 20, 2013
Research Questions 1. Do Target and GenEd group students
differ on the cognitive variables? 2. Are the cognitive variables accessible via
teacher judgment?
Item Modifications Fall 2008 Modifications of Reading Items
Bold/Underlined Important elements of the reading passage are bolded or underlined. This will facilitate structured recall of the content passage by AA-MAS students.
Boxed
Questions are interspersed within passages and offset with a box.
Simplified Language Language is simplified beyond that which is typical for universal design.
Thought Questions Additional questions designed to focus students’ attention on particular aspects of a passage are interspersed with the passage. These questions are not answered but serve to help students focus attention.
Modifications of Mathematics Items
Bold/Underlined (1 level) Important elements of the reading passage are bolded or underlined. This will facilitate structured recall of the content passage by AA-MAS students.
Modified Graphics Graphics may have been enlarged or simplified in order to increase readability.
Relevant Pictures Information in the stem of the item is given as a picture or table to help students organize and understand the information necessary to answer the question.
Scaffolding Complex items are decomposed into simpler parts.
Simplified Language Language is simplified beyond that which is typical for universal design.
Spring 2009 Modifications of Reading Items Bold or Underlined Important elements of the reading passage are bolded or underlined. This will facilitate
structured recall of the content passage by AA-MAS students.
Primed Items
A specially designed priming item is presented immediately before a test item. The priming item assists the students’ memory of the cognitive process so that they will more readily see the solution to the test item.
Cued Items Additional items break up complex questions into a series of simpler steps to reduce the planning load.
Graphic Organizers Graphic organizers provide a graphical structure to help students organize their thoughts.
Passage Primed Thought questions are introduced before a reading passage to help the students engage in the content of the passage.
Modifications of Mathematics Items Bold or Underlined Important elements of the problem are bolded or underlined. This will facilitate structured recall
of the content passage by AA-MAS students.
Primed Items
A specially designed priming item is presented immediately before a test item. The priming item assists the students’ memory of the cognitive process so that they will more readily see the solution to the test item.
Cued Items Additional items break up complex questions into a series of simpler steps to reduce the planning load.
Relevant Pictures Added icons help students visualize the problem at hand.
Results of interaction of item modifications with cognitive traits
for reading Working Memory Executive Function/
Planning Focused Attention Sustained Attention
Bold
Increases Influence Decreases Influence
Cue
Increases Influence
Graphic Organizers Decreases Influence
Prime
Passage Prime
Results of interaction of item modifications with cognitive traits
for mathematics
Working Memory Executive Function/ Planning
Focused Attention Sustained Attention
Bold
Increases Influence Increases Influence
Cue
Decreases Influence
Decreases Influence Increases Influence Increases Influence
Graphic Organizers
Decreases Influence
Increases Influence
Prime
Decreases Influence
More Research Questions 1. Do the revised items improve access? 2. Does performance exceed chance? 3. Do the revisions improve performance and
is the discrepancy reduced between the Target and GenEd group?
4. How do the revisions relate to the student’s cognitive traits?
Conclusions/Recommendations 1. 17% of general education students not identified as having
disabilities actually had significant deficits in sustained attention and 11% of the general education students had significant deficits in executive functioning.
2. Students often had deficits only on one, or perhaps two, cognitive traits.
3. Question the ability of classroom teachers to identify those students with disabilities who have specific cognitive processing deficits which has implications for how appropriate data might be collected to inform IEP team decision making.
4. IEP teams require additional training to help them understand that the indiscriminate use of accommodations may actually make tests less accessible for some students with disabilities.
education.ohio.gov
Social Media
@OHEducation
ohio-department-of-education
Ohio Families and Education Ohio Teachers’ Homeroom
OhioEdDept
storify.com/ohioEdDept
National Center on Educational Outcomes (NCEO)
Trinell Bowman
Maryland’s Alternate Assessment Based on Modified Achievement Standards (AA-MAS)
What did we learn?
2013 National Conference on Student Assessment
National Harbor, Maryland June 2013
Trinell Bowman Maryland State Department of Education
22
Lessons Learned from the Development of the AA-MAS • Cognitive Labs were critical to the development process
and for the first time, students were asked to describe the “why” and “how” as they answered test questions.
• Online testing offers more standardization of certain
accommodations such as the use of the audio test and calculator.
• Online testing may offer additional support to students, which they normally may not receive when taking a paper and pencil test. Examples include: eliminating a choice, highlighting critical information or marking a test question for review.
23
24
How does the student’s disability affect progress in the general curriculum? The student has difficulty with the following:
Percentage
Comprehending written materials 51.1%
Difficulty following multiple steps 51.1%
Overwhelmed with many sentences/words on a page
48.8%
Difficulty completing written tasks 44.1%
Maintaining time on tasks 39.5%
How are methods of instruction different for this student compared to special education students taking the regular assessments?
Percentage
Requires repetition of directions 62.7%
Extended time 58.1%
Scaffolding questions 55.8%
Slower pace of instruction 53.5%
Decreased level of content complexity 46.5%
Increased wait time 44.2%
More visual cues 32.5%
Greater use of assistive technology 30.2% 25
Lessons Learned from the Monitoring Process
1. Special educators must be included in professional development training to fully understand state standards.
2. All teachers must understand how to write and implement standards based IEPs.
3. All students must receive appropriate research-based
and/or evidenced-based interventions in order to address any gaps or deficit documented in an IEP.
Contact Information
Trinell Bowman Program Manager
Division of Curriculum and Assessment (410) 767-2498
27
National Center on Educational Outcomes (NCEO)
Susan Weigert
Perspectives of U.S. ED on Transition challenges and opportunities:
Students currently eligible for an AA-MAS
Transitioning to Improved Assessments
• AA-MAS Flexibility to states in 2007
• Students seen as unable to attain grade level proficiency
• Era of accountability reflected in NCLB State
percent proficient with add-on flexibilities.
• We can do better
Changes in Accountability
• Permitted States to incorporate student growth • For students performing below grade level this
changed what was required of an assessment— • Features we asked for became part of the vision
for next generation assessment. • Moved from a simply easier test • To an assessment system more sensitive to lower
levels of performance, and capable of measuring progress toward grade level proficiency.
Improved Design of Test Items • AA-MAS research identified a key problem in
accessibility of items: non-construct cognitive load.
• Evidence Centered Design has led to improvement in measurement.
• Item accessibility features that do not undermine construct validity
• Accommodations that are linked to student access profiles.
Instructionally-Embedded Assessment
Next Generation Formative assessment Offer better evaluation of student’s problem-solving approaches Feedback to guide the student correctly along the instructional path. Instructors can get a detailed analysis not just of whether the student reached the final answer correctly, but how they solved the problem.
Focus on Continual Progress
Assessments sensitive to student performance range and growth toward standards. Provide students and their teachers with a far better understanding of where they are, where they need to go next Will Provide empirical evidence of how far they can actually go with targeted instruction.
National Center on Educational Outcomes (NCEO)
Marianne Perie