Comparing State Reading and Math Performance Standards

Using NAEP

Don McLaughlinVictor Bandeira de Mello

National Conference on Large-Scale AssessmentSan AntonioJune 2003

Another look at NAEP’s role in the context of NCLB

Confirm state gains. Help state assessment programs

to help each state’s schools and students. Answer questions for/about the state

assessment. Provide a test bed for new solutions.

Complexities in confirming state gains

Sampling Exclusions, Absences, Small samples

Measurement Accommodations, Framework, Format,

Time of year, Motivational context Differential sensitivity

Analysis Gaps

Design Grade-to-grade growth

NAEP’s strengths for helping state assessment

programs

NAEP is administered with uniform procedures to a representative sample of public school students in every state. This can illuminate cross-assessment comparisons.

NAEP can carry out validity research to address questions relevant to all assessments (e.g., validity of accommodations).

Questions NAEP can answer for state assessments

1. How do your state’s performance standards for math and reading compare with other states’?

2. How do your state’s assessment instruments compare to others in focus and difficulty?

3. Can your state assessment’s identification of schools that need improvement be enhanced?

Questions NAEP can answer for state assessments

4. Is your state’s assessment more (or less) sensitive to race or poverty differences in achievement than other states’?

5. Are differences between types of schools in your state similar to analogous differences in other states?

6. How do your identification and testing procedures for SD/ELL students differ from those in other states?

Comparison of state performance standards, using

NAEP

Achievement levels are set separately in each state, using a variety of standard-setting methodologies.

Does “meeting the standard” mean something different in each state? … How different?

How do states’ targets for adequate yearly progress differ?

Comparison of state performance standards, using

NAEP

By comparing NAEP results with state assessment results in the same schools, we can approximate where on the NAEP scale the cutpoint for the state’s standard lies.

This requires matching a database of state assessment results to the NAEP database.

National Longitudinal School-Level State Assessment Score

Database

Scores provided by State Education Agencies

Processed by the American Institutes for Research

Funded by the U.S. Department of Education

Policy and Program Studies Service

School-level scores in the public domain.

National Longitudinal School-Level State Assessment Score

Database

Scores for > 80,000 public schools

in 49 states, Puerto Rico, and DC

from 1997-1998 through 2001-2002(but not all years in all states)

merged with Common Core of Data

National Longitudinal School-Level State Assessment Score

Database

Available at

www.schooldata.org

www.edtrust.org www.greatschools.net

Contact: statedata@air.org

Equivalents of State Math Standards: Grade 8, 2000

WY Advanced

WY Proficient

WY Partial

VT Meets

VA Pass

TX Pass

SC Advanced

SC Proficient

SC Basic

RI MeetsOR Exceeds

OR Meets

OK Advanced

OK Satisfactory

OK A Little Know ledge

NY Exceeds

NY Meets

NY Needs Improve

NV PR75

NV PR25

NC Superior

NC Consist Mastery

NC Inconsist Mastery

MO Advanced

MO Proficient

MO Near Proficient

MO Progressing

MN Pass

ME Exceeds

ME Meets

ME Partial

MD Excellent

MD Satisfactory

MA Advanced

MA Proficient

MA Pass

LA Advanced LA Proficient

LA Basic

LA Approach Basic

IN Meets

IL Exceeds

IL Meets

HI Stanine 5

GA Exceeds

GA Meets

CT At goal

CA PR75

CA PR50

CA PR25

AZ Exceeds

AZ Meets

AZ Approach

200.0

220.0

240.0

260.0

280.0

300.0

320.0

340.0

360.0

0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0

Standard Deviation

Method

Match the “percent meeting the state’s standard” in each school to the distribution of NAEP plausible values in the school’s NAEP sample.

e. g.,

If 30 percent of the students in a school are reported by the state testing program to meet the state’s standard,

Find the NAEP plausible value that has 30 percent of the distribution higher than it.

(If the NAEP mean in the school is 220, that value might be 230.)

o o o o o N o o o o o N o o o o o N oN o o o o o o N o o o o No o o o o No N o o o o N o o o o o o N o o o o oo o N o o o o o N o o o N o o o o o oo o o o N o o o o N o o o o o N o o oo o o N o o o o o o N o o o N o o o o

in a school selected to participate in NAEP

o o o o o o o o o o o o o o o o o o oo o o o o o o o o o o o o o o o o o oo o o o o o o o o o o o o o o o o o oo o o o o o o o o o o o o o o o o o oo o o o o o o o o o o o o o o o o o oo o o o o o o o o o o o o o o o o o o

o o o o o N o o o o o N o o o o o N oN o o o o o o N o o o o N o o o o o No N o o o o N o o o o o o N o o o o oo o N o o o o o N o o o N o o o o o oo o o o N o o o o N o o o o o N o o oo o o N o o o o o o N o o o N o o o o

Grade 4 Students

Some are selected for NAEP

Some meet state standard

o N o o o N o o o N o o o o o N o o N o o o N o o o o o o o N o o o o o o o o N o o N o o o o o o o o o o o N o o o o o N o o o o o N o o o o o o N o o o o o o N o o o o o o N o o N o o o o o o o o N o o o o o o o o N o o N o-----------------------------------------

Achievement Scale

o o o o o N o o o o o N o o o o o N oN o o o o o o N o o o o No o o o o No N o o o o N o o o o o o N o o o o oo o N o o o o o N o o o N o o o o o oo o o o N o o o o N o o o o o N o o oo o o N o o o o o o N o o o N o o o o

in a school selected to participate in NAEP

o N o o o N o o o N o o o o o N o o N o o o N o o o o o o o N o o o o o o o o N o o N o o o o o o o o N o o o o o o o o N o o o o o N o o o o o o N o o o o o o N o o o o o o N o o N o o o o o o o o N o o o o o o o o N o o N o------------------------------------------

Achievement Scale

Achievement frequency distribution in the school

n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n n

Achievement Scale (NAEP Reading)

o N o o o N o o o N o o o o o N o o N o o o N o o o o o o o N o o o o o o o o N o o N o o o o o o o o o o o N o o o o o N o o o o o N o o o o o o N o o o o o o N o o o o o o N o o N o o o o o o o o N o o o o o o o o N o o N o

Achievement Scale(State Reading Assessment)

in a school selected to participate in NAEP

30 % of the distribution in one school

might correspond to 230 on the NAEP scale, in that school.

Mapping a Standard for Two Schools with Different Performance Levels

(when tests are correlated)

0

1

2

3

4

5

6

7

8

9

0 0.5 1 1.5 2 2.5 3

70% 10%

State report of percents meeting the standard

NAEP achievement distribution in each school

NAEP Mean: 230 200

Standard at: 220 220

A B

cutpoint

Mapping a Standard for Two Schools with Different Performance Levels

(when tests are not correlated)

0

1

2

3

4

5

6

7

8

9

0 0.5 1 1.5 2 2.5 3

70%

10%

State report of percents meeting the standard

NAEP achievement distribution in each school

NAEP Mean: 230 230

Standard at: 220 250

A B

cutpoint

cutpoint

Assumption

State reading assessment score variation

is strongly related to

NAEP score variation.

Assumption

State reading assessment score variation

is strongly related to NAEP score variation.

If so, then we should get approximately the same estimate of the position of the state standard on the NAEP scale from nearly all NAEP schools in the state.

Test of the Assumption

Grade 4 Grade 8

Random matching 21.5 18.7

State to NAEP (matched) basic standardsState to NAEP (matched) proficient standards

State to NAEP (matched) advanced standards

NAEP to NAEP (matched) 7.0 7.3

Average standard deviation of scale values for standards, across schools in each state

Test of the Assumption

Grade 4 Grade 8

Random matching 21.6 18.7

State to NAEP (matched) basic standards

14.0 12.2

State to NAEP (matched) proficient standards 11.4 9.6

State to NAEP (matched) advanced standards

10.3 9.6

NAEP to NAEP (matched) 7.0 7.3

Average standard deviation of scale values for standards, across schools in each state

Test of the Assumption

State standard NAEP Standard Grade 4 Grade 8

Partial Basic 0.68 0.71

Partial Proficient 0.63 0.63

Meets Basic 0.74 0.73

Meets Proficient 0.72 0.71

Exceeds Basic 0.63 0.62

Exceeds Proficient 0.69 0.66

Average correlation between percent achieving state standards and percent achieving NAEP standards,

across schools in each state

Equivalent "Proficient" State Standards: Grade 4 Reading 2002

FL Success

SC Proficient MA ProficientWY ProficientME Meets

TX MasteryMI Satisfactory

NY ProficientAR Proficient

WA MeetsCT Proficient OH Pass

CA PR50MT Proficient VT MeetsNC Meets

WI Proficient

GA Meets

MS Proficient

160

180

200

220

240

0 5 10 15 20 25 30 35

Standard deviation

Standard

Equivalent "Proficient" State Standards: Grade 8 Reading 2002

GA MeetsNC Meets

MN PassCT Proficient WI ProficientIL Meets

IN MeetsDE Meets VA PassCA PR50MT Proficient

MS Proficient OR MeetsTX MasteryVT MeetsPA Proficient

NY ProficientWY Proficient ME Meets

AR ProficientSC Proficient

MD SatisfactoryKS Proficient

FL Success

220

240

260

280

300

0 5 10 15 20 25 30 35

Standard Deviation

Standard

Equivalent "Above Proficient" State Standards: Grade 4 Reading 2002

GA ExceedsCT Goal

CA PR75NC Exceeds

WA ExceedsWI Advanced

NY AdvancedMT AdvancedWY Advanced

FL Most Success AR AdvancedSC AdvancedMA Advanced

ME Exceeds

190

210

230

250

270

290

0 5 10 15 20 25 30 35

Standard deviation

Standard

Equivalent "Above Proficient" State Standards: Grade 8 Reading 2002

CT Goal

GA Exceeds

NC ExceedsCA PR75OR Exceeds

PA Advanced WI AdvancedMT Advanced IL Exceeds

NY AdvancedWY Advanced

SC AdvancedKS Advanced

AR AdvancedFL Most SuccessMD Excellent

ME Exceeds

240

260

280

300

320

340

0 5 10 15 20 25 30 35

Standard Deviation

Standard

Equivalent "Below Proficient" State Standards: Grade 4 Reading 2002

MS Basic AR Basic

WA PartialNC Partial TX Pass WI BasicNY Basic ME Partial

CA PR25 MT Near ProficientSC Basic

MI ModerateMA Pass

WY PartialCT Basic

FL Limited

FL Partial

150

170

190

210

230

250

0 5 10 15 20 25 30 35Standard deviation

Standard

Equivalent "Below Proficient" State Standards: Grade 8 Reading 2002

FL Partial

KS Satisfactory

SC BasicFL LimitedWY Partial PA Basic

MT Near ProficientME Partial

MS BasicCT Basic KS BasicDE Near WI BasicCA PR25 AR BasicNY Basic

TX PassNC Partial200

220

240

260

280

300

0 5 10 15 20 25 30 35

Standard Deviation

Standard

High "Proficient" Standards Are Harder to Achieve:Grade 4 Reading

MS ProficientGA Meets

WI ProficientNC Meets

VT MeetsMT Proficient

CA PR50

OH Pass

CT ProficientWA Meets

AR ProficientNY ProficientMI Satisfactory

TX Mastery ME MeetsWY Proficient

MA Proficient

SC Proficient

FL Success

y = - 0.79x

R2 = 0.70

0

20

40

60

80

100

160 180 200 220 240

Standard Scale Value

Percent Achieving Standard

High "Proficient" Standards Are Harder to Achieve:Grade 8 Reading

MS Proficient

OR Meets

TX Mastery

VT MeetsPA Proficient

NY ProficientWY Proficient

ME Meets

AR Proficient

SC ProficientMD Satisfactory

KS Proficient

FL Success

MT Proficient

CA PR50

VA PassDE Meets

IN Meets

IL Meets

WI Proficient

CT ProficientMN Pass

NC Meets

GA Meets

y = -1.11x

R2 = 0.83

0

20

40

60

80

100

220 240 260 280 300Standard Scale Value

Percent Achieving Standard

High "Proficient" Standards Are Unrelated to High Achievement on NAEP: Grade 4 Reading

MS Proficient

GA Meets

WI ProficientNC Meets

MT Proficient

CA PR50

CT Proficient

WA Meets

AR Proficient

NY Proficient

MI Satisfactory TX Mastery

ME MeetsWY Proficient

MA Proficient

SC ProficientFL Success

y = 0.14x

R2 = 0.12

0

20

40

60

160 180 200 220 240

Standard Scale Value

Percent Achieving NAEP Standard

High "Proficient" Standards Are Unrelated to High Achievement on NAEP: Grade 8 Reading

GA Meets

NC Meets

MN PassCT Proficient

WI ProficientIL Meets

DE Meets

VA Pass

CA PR50

MT Proficient

FL Success

KS Proficient

MD Satisfactory

SC ProficientAR Proficient

ME Meets

WY ProficientNY ProficientTX Mastery

OR Meets

MS Proficient

y = - 0.01x

R2 = 0.002

0

20

40

60

220 240 260 280 300Standard Scale Value

Percent Achieving

NAEP Standard

Conclusions

NAEP can provide a link between state achievement standards, especially at the proficient and advanced levels.

There is substantial variation in standards between states.

Percentages of students meeting state standards are lower in states in which the standards are set high.

Most of the links for “below proficient” standards are weak.

Summary

NAEP can contribute most to Leaving No Child Behind by helping state assessment programs give useful feedback to parents, teachers, and school administrators.