Raggedy Ann or Student Performance:

Construct Relevance in Alternate Achievement

StandardsAssessments

How can we support students but not interfere with the construct ?

• Does the assessment truly measure that which it intended to measure?– Content Standards in• Reading• Math• Science

• How can we support students maximizing the assessment population but not interfere with the construct?

Where it gets messy…..• AA-AAAS necessarily require the teacher to – Directly observe student performance or recall

observation using ongoing (formative assessment) techniques• Performance tasks

– Level of prompt required to make a response– Select the correct response from an array of responses

• Portfolios– Compile on-going data collection sheets based either based on

non-instructional probes or on level of prompt required to make a response or a combination of both

– EVEN Construct items that measure the intended construct…

Historically, in Effective Practices in Special Education

• The purpose of assessment was– Formative (instructional improvement)

• Instruction and assessment in severe disabilities which follows – Techniques from Applied Behavior Analysis– Relies on heavily on

• Observational techniques• Task analysis and chains of behaviors• Collection of performance data during instruction• Identifying “controlling prompts” which ensure the intended

behavior will occur under the specified stimulus conditions

Ongoing Instruction or formative assessments

Prompt Definition

• Prompt – teacher behaviors that increase the probability of a correct response – used in the acquisition of behaviors/targets and faded as learning improves.– ( Wolery et al. 1992; Wrestling & Fox, 2004)

Support Definition• Support – modifications in the

presentation of an instructional target, materials, response modes

• referring to instruction and levels of participation in general curriculum, inclusive education.

( Weymer, Lattin, & Agran 1992)

Probe Definition

• Probe – an opportunity for the student to perform the skill given only the target stimulus with NO OTHER input from the teacher. – (Wolery, et al. 1992; Wrestling & Fox, 2004)

Scaffold Definition• Instructional scaffolding is the provision of sufficient support to

promote learning when concepts and skills are being first introduced to students. These supports may include the following:– Resources– A compelling task– Templates and guides

• These supports are gradually removed as students develop autonomous learning strategies, thus promoting their own cognitive, affective and psychomotor learning skills and knowledge

.• The teacher’s level and type of support change over time from

directive, to suggestion, to encouragement, to observation.

– (Bruner; Vytgosky; Cazden & others)

The Question of the Day!

How do we “control” for irrelevant variance in the design of assessment of students with significant cognitive disabilities who need supports in order to provide an independent, discriminatory response?

Prompt Confusion• The task direction is not a prompt

–”Find one more” is NOT a verbal prompt; it’s the cue to respond. Another example, “Read the word”

• Support to get student ready to respond is not a prompt– Allowing student to select one answer gets Kyle ready to respond. Another example- placing students hands above all answers so can slide hand to correct card is NOT a physical prompt.

Wakeman adapted from Browder (2008) Prompting: Impact on Inferences about Student Learning

Student Achievement vs. Something else

• Student achievement• – Select picture for

main idea• Full credit eye gaze, ‐

point• Find main idea across

stories• Solve problems- eye

gaze to correct answers for TA

• “Something Else”• Select picture with

model prompt- point where I point

• Student works with a peer to select picture

• Student did not select picture but could select not my best work

Wakeman adapted from Browder (2008)

Worksheet• What is the targeted skill? – Is it to trace numbers or add or subtract decimals?

• Could make very little inference about student learning for tracing numbers given.

• Better way to assess AND support learning using a scribe?– One idea: Steps in a TA to solve problem and have

student eye gaze at or select from written choices correct answer for each step. Teacher or student could place correct answers in the equation.

Wakeman

Teaching vs. Testing• This teacher may have previously assessed the

student without prompts on this task and is now teaching the student.

• Assessment= unprompted independent responses

• Teaching= prompts (changes in definitions of terms to support student understanding; review of individual items; provides correct answers when student is wrong)

Wakeman

Scaffolding• Interesting/confusing when considering scaffolds in

assessment- does the scaffolded support change the construct?

• Behaviorist would consider a verbal prompt hierarchy• Where do we get apples?• – Let’s read the paragraph again (rereads paragraph)• – Let me read some key sentences (rereads sentences• with correct answer)• – Listen to this sentence “The apples came from an ‐

orchard of well established trees.”Wakeman adapted from Browder (2008)

Consideration for Partial Credit?• When reducing the number of choices, it

can become an issue of independent response vs. an accurate response.– What is the priority within the state scoring

system?– Inferences about student learning in

assessment- with only two options, guessing plays a significant role. How does that play out in scoring of the assessment?

– Good teaching practice regarding distractors (relevancy – “What did they ride in the story- a bus or a goldfish?” vs. “a bus or a car?”) may be based upon student need.

Wakeman

Response Prompts

• Excellent example of Least to Most prompts• Valuable in teaching to support students • Could demonstrate learning for some students

(growth) in an assessment if they demonstrate a change over time in the level of prompt needed across different content – (not master Romeo and Juliet and then right back to

needing physical prompts for Macbeth but show change over time and content needing fewer physical prompts for each story)

Wakeman

Lack of Appropriate Response Options or Support = Bias in the Assessment• When an item does not include the

appropriate support (picture symbols) as this one does not, the assessment is then biased against the student (no way to participate in the item).

• By defining an active response the student CAN make; minimize the barrier of response mode

Wakeman adapted from Browder (2008)

Construct Relevance

Using the AA-AAAS Test Administration Manual to Communicate the Appropriate Use of Prompts

Test Administration Handbook

• Definitions of Prompts and Prompt Hierarchies

• Use of prompts in task/item/artifact development

• Recording use of prompts • Scoring rubrics and prompts• Instructional use of prompts

Different prompt hierarchies

• Verbal-Gesture-model-partial physical-full physical

• Gesture-verbal-visual-partial physical-full physical

• Auditory-visual-physical• Verbal-gesture-physical

Different definitions for prompts

• Verbal prompt – This level of prompt requires the teacher to give a specific verbal direction in addition to the task direction. (indirect/direct)

• Verbal: Auditory Statement (e.g., more than repeat prompt)

• Auditory - Any sound or verbal direction given to help the student initiate or participate in the completion of the skill in an activity.

Different Definitions of Prompts

• Visual - Any picture, gesture, or modeling given to help the student initiate or participate in the completion of the skill in an activity.

• Gesture prompt – This level of prompt requires the teacher to move his/her finger, hand, arm, or make a facial expression that communicates to the student specific information (e.g., teacher taps scanner switch button).

• Model prompt – This level of prompt requires the teacher to demonstrate the correct response for the student, and the student imitates the teacher’s model (e.g., the teacher demonstrates how to push the switch and then asks the student to repeat).

Code of EthicsIT IS A BREACH OF PROFESSIONAL ETHICS FORSCHOOL PERSONNEL TO PROVIDE VERBAL AND NON-VERBAL CLUES OF ANSWERS, TEACH ITEMS ON THE TEST, SHARE WRITING PROMPTS, COACH, HINT, OR IN ANY WAY INFLUENCE A STUDENT’S PERFORMANCE DURING THE TESTING SITUATION.

A BREACH OF ETHICS MAY RESULT IN INVALIDATION OF TEST RESULTS AND LEA OR MSDE DISCIPLINARY ACTION.

Prompts in Task/Item/Artifact Development

Various levels of direction• Data Charts. The template for the Full Physical

Prompt data chart includes an area to list the various assistive technology devices a student may use to explore the full range of assistive technologies to support demonstration of skills.

• Prompts may not be used in the baseline• Use of Prompts No prompts are allowed. Be

careful not to use physical or verbal prompts to encourage student to make a choice (for example, gesturing, looking, pointing)

Recording of Prompt Use During Assessment

• Record on data charts

• Report on student work

• State after videotaping or audio taping of student performance

Prompts: Scoring RubricPrompts That Prevent Grading for AccuracyThe teacher must grade the item as incorrect;

the prompt has made it impossible to determine student performance.

Full Physical Prompt Model of Exactly the Same TaskTelling Student the Answer

ScoringLevel of Independence• The student seldom requires cues or prompts

when demonstrating skills based on the documented AGLIs. (80-100%Independence)

• The student requires limited cues or prompts to demonstrate skills based on the documented AGLIs.(60-79%Independence)

• The student requires extensive cues or prompts to demonstrate skills based on the documented AGLIs. (30-59% Independence)

Scoring• The number of steps not requiring prompts

per number of steps or items in the assessment task determines the percentage of independence.

• 80% or greater is attained, and the prompt level and number do not exceed prompt level and number of prompts indicated in the mastery objective.

I have noticed some significant changes with our students (who represent

those with the greatest challenges of all who are tested). One struggle in

working with students with SCD is that when they develop skills, these are so

often “prompt dependent” skills. Students display these behaviors only once

others in their environment elicit them. Despite professional inservice training

to minimize over prompting, staff often seems to persist with giving many

prompts. This is one of the areas I have worked on in training but the testing

process seems to have made the most significant difference in helping staff be

more mindful of prompts they are giving as well as providing more

wait time for students to respond without prompts. This certainly moves them

toward greater independence and that is a wonderful and positive thing!

— Submitted by Roslyn Canosa, Principal; Baltimore City Public Schools

Advice for Minimizing Irrelevant Variance in AA-AAAS

• Use procedures that support independent responses.– If prompted responses are accepted, score appropriately.

• Provide structure (e.g. scripts, protocols) for the administration of “items”.

• Specify appropriate and inappropriate “supports”, “scaffolds”, “prompts” in the administration manual.

• Provide items or provide structure/ protocol for item development specifying the construct.

• Caution is warranted when using systematic least to most prompting systems as they may– Increase the number of item presentations for students who

may require higher levels of prompts– Prompts are aimed at producing a correct response