vertically articulated performance standards

Vertically Articulated Performance Standards

An Exploratory Examination of their Interpretability

Steve Ferrara, Gary Phillips, Paul Williams, and Shannon Mahoney

American Institutes for ResearchOctober 19, 2006

Vertically Articulated Standards (VAS) 2

Overview for this talk

Background and purpose of the study Vertically articulated standards

Definition, procedures The assessment program This study

Procedures Results, interpretation, implications

Conclusion


Purposes of the study

Evaluate a system of articulated standards that we created for a state content area assessment

Examine definitions of Proficient performance and achievement growth as defined by test items around the Proficient cut scores

Consider implications for the panelist’s judgmental task in selecting cut scores and articulating standards


Vertically articulated performance standards

Coherent system of performance standards When performance standards are articulated,

they form a system of standards in an orderly progression across grades in the same content area and across content areas

An orderly progression of standards is indicated by the percentages of examinees at/above each performance level in each grade


Vertically Articulated Performance Standards

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

3 4 5 6 7 8

Grade

Per

cen

tag

e o

f S

tud

ents

At

or

Ab

ove

Basic

Proficient

Accelerated

Advanced


Vertically Disarticulated Performance Standards

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

3 4 5 6 7 8

Grade

Per

cen

tag

e o

f S

tud

ents

At

or

Ab

ove

Basic

Proficient

Accelerated

Advanced


Importance of vertically articulated standards

Discrepancies can cause confusion in the general public and media and enable misinterpretations of standards and performance Misinterpreting cross-grade differences in

performance standards as achievement differences

Mistargeting resources to a grade level or content area

Blaming or rewarding teachers


The assessment program

Typical state assessment program (e.g., MC and CR items) Ambitions well beyond NCLB requirements (e.g.,

K-2 diagnostic assessments, other content areas, accountability system, growth modeling)

Reading and math, grades K-8 plus high school

Science, social studies, and writing in an elementary and middle school grade and at high school level


The assessment program (cont.) Within grade assessments aligned with grade level

content standards Within grade scaling Vertical articulation of standards across the within grade

scales Vertical scale for future score reporting (not

considered in standard setting) Linking items included from the lower adjacent grade only

(“down linking”) Previous year, math: Higher linking error when include

upward linking items Joint calibration, grades 3-8; also chain linking, other

tryouts


Setting cut scores within grades

Mathematics grades 3-8 Separate 3-5 and 6-8 panels Reading grades 4-5 and 6-8

Bookmark method “Place your bookmark on the page that

(approximately) two-thirds of those students who are just barely Proficient would be able to answer successfully”

RP criterion Just Barely Proficient students


Proficiency level descriptions (PLDs)

In this program, the PLDs define Proficient performance explicitly in terms of growth from the previous grade


Grade 3 Mathematics PLD Students performing at the Proficient level (1) show adequate

progress by using grade 3 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) use place value concepts, apply basic measurement and geometry concepts to describe attributes of shapes or objects, and interpret graphs. They usually can use informal reasoning and make appropriate decisions about what procedure to use to solve routine problems. Students typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students communicate mathematical thinking and solutions using a combination of informal and mathematical language.



progress by using grade 4 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) solve routine problems involving whole numbers, decimals and simple fractions; describe perimeter and area; compare geometric figures; write an equation to describe a situation; and describe data. They usually can use informal reasoning and make appropriate decisions about what procedure to use to solve routine problems. Students typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students communicate mathematical thinking and solutions using a combination of informal and mathematical language.



progress by using grade 5 concepts and skills to solve familiar problems. They apply mathematical concepts, terms and properties to problem situations. Most of the time, students can (2) solve routine problems involving sums and differences of fractions or decimals, measuring angles, describing the probability of events, and converting units in the same measurement system. They typically can interpret or provide a visual or symbolic representation to match a problem situation and purpose. Students use informal and some formal reasoning to evaluate and justify the reasonableness of a solution. They communicate mathematical thinking and solutions using a combination of informal and mathematical language.


PLDs for Proficient

Each definition refers to growth from the previous grade

Each describes articulation of content and skill requirements across grades 3, 4, and 5

Some features are constant across these grades


Achieving vertically articulated standards

Set cut scores for anchor grades in typical way

For interim grades, provided the target cut score (i.e., page number) that would provide an orderly progression of standards across grades

We asked panelists to try to make the bookmark judgment within three or so pages of the target


Achieving vertically articulated standards (cont.)

After all standards were set, table leaders participated in a moderation process to achieve articulated standards Guidance: Cut scores that corresponded to

orderly percentages of students reaching Proficient (and other levels) in each grade

Within content area, then across content areas They chose not to adjust one standard


This study: Analytic procedures

Identified items that define Just Barely Proficient performance

Examined the knowledge and skills they Target Require for successful responses We refer to this as Item Demands Analysis

Summarized and compared the knowledge and skill demands


Analysis (cont.)

Specifically, we focused on the items that define Just Barely Proficient performance at grades 3 and 4

We examined those items and how they define: Just Barely Proficient performance for standard

setting panelists Growth in achievement


Analysis (cont.)

We used mathematics in grades 3 and 4 from a state assessment program

We focused on the standard for Proficient performance, established using the Bookmark procedure

We focused on the concept of the Just Barely Proficient student because this is the concept that panelists use to make the cut score judgment


Analysis (cont.)

Items: RP 67 locations on the vertically linked scale

Just Barely Proficient: Items at the cut score, three items below, three items above


Example: grade 3 mathematics

Item ID RP67Stan-dard Load Item description CAK CAS

O3MO4016 -0.764 NS LowDivision - missing

numberPK: Division fact UA 11h

O3MO894 -0.758 ME ModMeasurement -

customaryCU: Measurement

unitsUA 11h

O3MO3414 -0.656 GS Mod Geometry # 3 - obtuse PK: acute angleUA 11b,

11h

O3MO4018 -0.576 NS Mod Estimation - subtractionCU: Reasonable

estimateAE 12a

O3MO567 -0.572 NS LowIdentify - mixed

numbersPK: Mixed number (2

1/4) as visual rep'nUA 11b,

11h

O3MO968 -0.520 PA LowPatterns - identifying

numbersRecog: subtraction

patternACH 13c

O3MO8225 -0.487 NS High Packages of buns

PK: Multiplication facts for 48;calculate correct cost; select correct bills and coins

AE 12a


Item demands analysis

Targeted content standards Identified in the state content standards and test

blueprints (see App. A) Our judgments about:

Reading load Content area knowledge demands Content area skill demands/question types


Definitions: Reading load

High Lots of text, complex text, interrelatedness of

elements in the text, complex format Moderate

Multiple steps, short phrases, scaffolded format Low

Small amount of text, mostly mathematics, how to proceed and respond is explicit or obvious


Definitions: Content area knowledge

Prior knowledge Supply or apply an arithmetic fact or taught

algorithm Conceptual understanding

Use or explain a mathematical concept Recognition

Recognize and complete a numerical or other pattern, recognize a visual representation of a mathematical concept or procedure


Definitions: Content area skills

Use/apply E.g., visual information given with the item

Answer and explain E.g., defend the answer given

Analyze, categorize, hypothesize E.g., observe and describe patterns in data,

procedures, or results Empirically supported (Ferrara, Duncan, et

al., 2004); adapted from science


Selected results: Grades 3 and 4

Grade 3 JBP Student Grade 4 JBP StudentJBP Student at the Grade 34

Cut Score

Standard NS=4ME=1PA=1GS=1DA=0

NS=2ME=1PA=1GS=2DA=1

NS=2ME=2PA=1GS=0DA=2

Region of RP 67 locations of items

-0.76, -0.49Difference=.27

-0.13, +0.26Difference=.39

-0.78, -0.33Difference=.45

Scale location of cut score

-Easy (CS=-0.58)-CS at 29 of 52 (56%)

-Moderate (CS=-0.01)-CS at 15 of 52 (29%)

-Easy (CS=-0.45)-CA at 5 of 52 (10%)

Item types 5 MC, 1 SA 1 of 2, 1 SA 2 of 4 4 MC, 1 SA 1 of 2, 1 SA 2 of 2, 1 SA 1 of 4

5 MC, 2 SA 1 of 2

Content area knowledge requirements

PK=3CU=2Recog=2

PK=3CU=1Recog=3

PK=5CU=Recog=1

Content area skill requirements

UA=4AE=2ACH=1

UA=5AE=2 (1 UA and AE)ACH=0

UA=5AE=2ACH=0

Reading and language load

Low=3Mod=3High=1

Low=1Mod=3High=3

Low=3Mod=3High=1


Selected results: Grades 3 and 4 cut scores

Standards targeted More number sense at grade 3, more geometry at

grade 4 Scale location of cut scores and JBP items

CS grade 3 = -0.58, CS grade 4 = -0.01 Locations grade 3 = (-0.76, -0.49), grade 4 =

(-0.13, +0.26) Item types

Similar


Selected results: Grades 3 and 4 (cont.)

Content area skills Similar

Reading load Grade 3 low-moderate Grade 4 moderate-high


Content area knowledge requirements

Grade 3 Grade 4

Prior knowledge

Use division and multiplication facts; calculate cost, select bills and coins (2)

Calculate statistics, use to support conclusion; calculate change, select bills and coins (3)

Conceptual understanding

Measurement units for liquid, reasonable estimate (2)

Parallel lines (1)

Recognition

Subtraction pattern, pizzas as mixed number, acute angle (3)

Addition pattern, symmetric rotation, calculate area by counting squares (3)


Summary of results: Grades 3 and 4

Item demands on JBP students similar for the grade 3 and 4 cut scores except in three areas Location of cut score on vertical scale Content area knowledge requirements Reading load

Item demands for grade 3grade 4 scale Similar to grade 3 except more emphasis on prior

knowledge requirements


Interpretation of results: Three areas

System of articulated standards Definitions/interpretations of what it means to be

Proficient and what is growth in achievement Implications for setting and articulating standards

We are interested in practical steps for setting performance standards that: Are vertically articulated Enable valid interpretations about what students know and

can do as they progress through school


Interpretation: System of articulated standards

Articulating standards across grades using statistical considerations can enable meaningful interpretations from a content area demands point of view

Content knowledge, reading load, and difficulty demands increased in meaningful ways across grades 3 and 4 for this mathematics assessment

That increase is small and consistent with the grade level means


Student proficiency estimates, vertically linked theta scale

Grade Mean SD

3 0.00 0.97

4 0.58 0.96

5 1.00 1.00

6 1.27 0.88

7 1.52 0.91

8 1.66 0.94


System of articulated standards (cont.)

A concern: The articulation worked out well, but it did not happen completely by design

Content standards, PLDs, and performance standards are articulated by design

Item locations are not articulated by design Item writers aim at difficulty targets but cannot/do not control

empirical item difficulty A point being made often: Scientifically principled test design and

development and item difficulty modeling E.g., Ferrara & DeMauro, 2006; Gorin, 2006; Mislevy & Haertel

(2006)


Interpretations of Proficient and growth in achievement

Articulating standards in a meaningful way enables inferences about growth in achievement; however,

Typical PLDs illuminate what that growth entails only in a very general way “Students performing at the Proficient level show

adequate progress by using grade 3 concepts and skills to solve familiar problems. …”


Interpretations of Proficient and growth in achievement (cont.)

Meaningful inferences about grade-to-grade growth in achievement require that four elements are articulated: Content standards that are the basis for test

design and development PLDs that are the performance standards The cut scores that effect those standards The items around a cut score that panelists use to

define the cut scores


Implications for setting and articulating standards

In item mapping procedures, panelists consider item demands to locate their cut scores

They may or may not consider whether the items they examine will result in articulated cut scores Even when we give them articulation information It seems likely that they do not

Consider using item demands summary tables as input to: The standard setting process Subsequent moderation processes to articulate standards


Conclusion

Exploratory examination; we need to look at additional grades and content areas

It seems unlikely that things will work out as fortunately in all grades and content areas


ReferencesFerrara, S., & DeMauro, G. E. (2006). Standardized assessment of individual achievement in K-12. In

R. L. Brennan (Ed.), Educational measurement (4th ed.). Westport, CT: American Council on Education/Praeger.

Ferrara, S., Duncan, T. G., Freed, R., Velez-Paschke, A., McGivern, J., Mushlin, S., Mattessich, A., Rogers, A., & Westphalen, K. (2004). Examining test score validity by examining item construct validity: Preliminary analysis of evidence of the alignment of targeted and observed content, skills, and cognitive processes in a middle school science assessment. Paper presented at the annual meeting of the American Educational Research Association, San Diego.

Gorin, J. (2006 in press). Test design with cognition in mind. Educational Measurement: Issues and Practice, 25 (4).

Mislevy, R. J., & Haertel, G. D. (2006 in press). Implications of evidence-centered design for educational testing. Educational Measurement: Issues and Practice, 25 (4).


Appendix A: Content Standards

Number, Number Sense and Operations Standard

Measurement Standard Geometry and Spatial Sense Standard Patterns, Functions and Algebra Standard Data Analysis and Probability Standard Mathematical Process Standard

vertically articulated performance standards

Documents

performance level

system of standards

content areas

content areablaming

grade level

accountability system

achievement growth

grade vertical articulation0