Contingencies for the use of effective educational practices: Developing Utah’s Alternate Assessment

Tim Slocum

Wing Institute First Annual Summit

April 2006

Main Topics

Contingencies and evidence-based special education

Designing an alternate assessment that can bring powerful NCLB contingencies to bear on promoting evidence-based special education.

Contingencies and Evidence-Based Practices

Contingencies A contingency is a relationship between

behavior and consequences. If a certain behavior occurs, then a certain

consequence will occur. Many contingencies are “natural” - that is, not

intentionally constructed Physical – Effects of jumping out of a window on falling Social – Effects of speaking unclearly on interaction

Other contingencies are intentionally constructed to influence behavior Parenting – Family rules Education – Classroom motivation systems Legislation and policy – Laws and regulations

Contingencies and Evidence-Based Practices

When we consider factors that may promote or undermine a culture of evidence-based practices, we should consider contingencies that may make such a culture more or less relevant.

What legislative and policy contingencies provide a context for evidence-based practices?

Effects of Contingencies Contingencies can be very powerful for

changing behavior. When powerful reinforcers are given contingent

on a particular desirable behavior, changes in that behavior can be dramatic.

However, if contingencies are not carefully constructed, they can produce unintended outcomes. A teacher may offer a reward contingent on

increased oral reading rate. This may result in increased oral reading rate, but also increased errors.

Contingencies – Defining Targets There are two ways to define behaviors in a

contingency: By their form – specifying what must be done:

Examples Study 2 hours per night… Reading First – a federal program that specifies forms of

behavior (research-based reading instruction) necessary to get a reward (funding).

This requires a specific form of behavior but leaves the outcome unspecified.

Unintended outcomes can come from completing the behavior in ways that do not produce the intended outcomes. Consider all the ineffective “study behaviors” that a student

might engage in. Consider how mandated processes (IEP, functional analysis,

implementation of research-based practices) become formalized and fail to achieve intended purposes.

Contingencies – Defining Targets By their results – specifying what must be

achieved: Examples

Get a GPA above 3.75… No Child Left Behind – a federal program that specifies

outcomes (AYP) necessary to avoid punishment (public scorn, loss of control, loss of funding)

This specifies what is to be accomplished but leaves the process unspecified.

Unintended consequence can come from ways of producing the narrowly defined outcomes without producing the intended outcomes. Getting a high GPA by taking easy classes Raising test scores by excluding lower performing

students.

Contingencies

So, whether we aim at what a person must do (form) or what they must accomplish (achieve), we must be very careful about defining what constitutes fulfilling the target of the contingency.

When contingencies specify difficult behaviors and promise strong consequences, the chances of unintended consequences are most serious.

Contingencies and NCLB We can understand NCLB as a set of

contingencies that specify results but leave specific forms of behavior unspecified. NCLB has produced a lot of behavior oriented

toward achieving adequate yearly progress (AYP). Some of this behavior is desirable:

Searching for and implementing evidence-based practices,

Focus on effectively teaching core subject matter. Some of this behavior is undesirable:

Implementing narrow test-preparation programs, Excluding lower performing students, Reduced focus on important outcomes that do not affect

AYP.

Contingencies and Evidence-Based Practices

When we talk about promoting an evidence-based culture in special education, we must consider contingencies. Contingencies that demand effectiveness provide a context for an

evidence-based culture. With such contingencies, many of the most serious barriers to

implementation of evidence-based practices are reduced or eliminated.

In the absence of such contingencies, an evidence-based culture is likely to be less relevant. Effective implementation of evidence-based practices requires

substantial effort from many players. In the absence of strong contingencies, this kind of effort is unlikely.

When we think about each part of the Wing model of evidence-based education, we should consider the contingencies for key actors at each point.

Alternate Assessment

Contingencies and Alternate Assessment

NCLB is designed to include all students in a school’s AYP – and that is a very good thing.

Given the strength of NCLB’s contingencies, behavior that is not relevant to meeting its demands is likely to be reduced in value. Exclusion from contributing to AYP does not just imply

avoiding the pain and hassle of test-taking. Exclusion makes the student’s educational progress

irrelevant to the powerful contingencies of NCLB. Those who are not counted, don’t count.

NCLB includes a provision specifying that students with significant cognitive disabilities (SCD) take an alternate assessment that will contribute to AYP.

Utah’s Alternate Assessment We (Karen Hager and I) became interested in

whether an alternate assessment could be constructed that would take advantage of the strong contingencies of NCLB to: motivate effective educational practices and improve educational outcomes of students with SCD.

We worked with the State of Utah to attempt to construct an effective alternate assessment.

We realized that validity of the alternate assessment would be critical. Validity is a technical term for the degree to which an

assessment successfully accomplishes the purposes for which it is being used.

Validity is critical for producing intended outcomes and avoiding unintended outcomes from contingencies.

Importance of Validity

The contingency of NCLB is based on test scores, not important student learning per se. To the degree that alternate assessment

results are valid, the contingencies of NCLB could have very positive impacts.

To the degree that alternate assessment results are invalid, the contingencies of NCLB could be very detrimental.

Defining the Construct

We began with the question, “What are we trying to measure?”

Federal mandate – AA must assess: Domains of Language Arts and Math for students with

SCD. Our values – AA should assess:

Skills that will contribute to independence and effectiveness of students with SCD.

The construct that drove the development of Utah’s Alternate Assessment: Language Arts and Math skills that will contribute to

students’ independence and effectiveness for students with SCD.

Sources of Invalidity There are 2 main sources of invalidity:

Construct-Irrelevant Variance – the assessment may be influenced by factors that are not part of the target construct. May result in strengthening the wrong behaviors.

Construct Under-representation – the assessment may ignore some important component of the construct. May result in failing to strengthen some appropriate

behaviors.

Construct Irrelevant Variance Sources of Construct Irrelevant Variance

Direct assessment of factors other than student behavior

E.g., degree of inclusion, attendance, and other issues Assessment of skills in such a way that outcomes

are heavily influenced by how the teacher documents student behavior.

E.g., some forms of portfolio assessment Inclusion of skills that are not important.

E.g., academic skills that are not functional Problems with administration

E.g., Low fidelity of implementation and low reliability Trivial tasks

E.g., assessments on which all students will be “proficient” no matter what their skills.

Construct Irrelevant Variance

Implications of Construct Irrelevant Variance These are sources of unintended effects

of the contingency The contingency may strengthen

behaviors relevant to these other factors at the expense of actually improving the learning of important skills.

Construct Under-representation Sources of construct under-

representation Some important skills are not assessed.

E.g., actual differences in critical communication skills do not impact test results

Implications of under-representation The powerful contingencies of NCLB may

reduce focus on these important skills.

NCLB appears to be very powerful in changing behavior of teachers and administrators.

If this power is to benefit students with significant cognitive disabilities, Alternate Assessments must be highly valid. That is, they must reflect actual student

performance on important tasks and must exclude other factors as much as possible.

Construct Under-representation

Design of Utah’s Alternate Assessment

Standardization among diversity

The greatest challenge in designing an alternate assessment is to: Produce meaningful, standardized

outcomes For a highly diverse group of students.

This requires a balance of standardization and individualization.

Standardization is necessary to enable those not directly involved in the administration to interpret scores. Standardized bank of tasks

UAA includes approximately 115 tasks in language arts and math

Standardized administration procedures Each task requires direct observations of student performance Tasks are to be embedded in naturally occurring routines in

natural settings Any communication system or assistive technology the student

typically uses is acceptable during the assessment

Standardization in UAA

Standardization in UAA

Performance criteria standardized for each task: Performance criteria are designed to reflect a

“functional” level of performance, i.e., a level that makes the skill meaningful/useful

Students perform the task independently (e.g., no physical, verbal, or gesture assistance)

Measurement tasks specify assessment of generalization: Three performances in situations that vary in relevant variables (e.g., people, settings, materials)

Individualization in UAA

Individualization is necessary to ensure relevance to specific students. IEP team selects targets from list of

standardized tasks IEP team selects content on some tasks

E.g., specific sight words to be assessed

Individualization in UAA

IEP team adapts the task to some individual circumstances IEP team specifies the natural routine during which to

assess

IEP team selects appropriate generalization settings E.g., particular individuals, settings, materials

Defining Adequate Yearly Progress

AYP for general education students Standards set by grade level

AYP for students with SCD Grade level not relevant basis for defining

standards Tasks that define AYP must be identified

individually. In UAA, IEP team selects relevant tasks from task bank.

Given a relevant task, performance level necessary to declare AYP is standardized.

Example UAA Task Description

Example UAA Task Description

Example UAA Task Description

Example UAA Task Description

Example UAA Task Record

Example UAA Task Record

Example UAA Task Record

Accessing UAA

The entire manual for Utah’s Alternate Assessment and the Technical Manual can be found at: http://www.usoe.k12.ut.us/sars/Data.htm