Using MEAPTo Evaluate

Teachers

MERA Fall 2011 ConferenceDave Treder

Study Methods

• Looked at MEAP growth/change over two consecutive years (grade 4 MEAP to grade 5 MEAP):

Both Fall 07 to Fall 08 and Fall 08 to Fall 09

2007-08 School Year 2008-09 School YearSchools 71 72Teachers 188 192Students 4576 4751

With MEAP Scores, Yr1 & Yr2Schools 71 72Teachers 186 189Students 4223 4318

(92%) (91%)

Counts-- 20 of the 21 school districts In Genesee ISD

Description of Study Data

• 4th Grade Teachers • 20 of the 21 Districts In Genesee ISD• Based on the Districts’ SIS system• Business rules:

- If schedules weren’t used in the district, the homeroom teacher was coded as the math and reading teacher – in the districts that used schedules in the elementary grades, math and reading classes were used to designate the math and reading teacher

- Teachers were included if they were the teacher of record at the beginning of the school year (1st semester grades) and the end of the school year (last semester grades)

- Students included if they had the same teacher at the beginning of the school year (1st semester grades) and the end of the school year (2nd semester grades)

Number of 4th Grade Teachers, by District:F07-08, F08-09, and in the District Both Years

District Number N0708 N0809 Both Years1 36 37 122 24 22 173 15 15 144 12 12 115 11 14 96 11 12 107 9 9 88 9 9 69 9 9 810 9 9 811 7 6 512 5 6 513 5 5 314 5 5 415 4 3 316 4 4 417 4 4 418 3 3 319 2 3 220 2 2 1 TOTAL 186 189 137 74%

Clas

s Size

F07-08 Class Size Frequency

What was the size of the classrooms, for the teachers in this study?(2007-08 School Year)

Study MethodsMethod 11) Compute by-student residuals (non-linear regression, student predicted - actual scale score)2) Compute Mean Teacher residual

Method 21) Compute by-student Performance Level (PL) Change Index2) Compute Mean Teacher PL change index

• For each method, compute indices for F07-F08 and F08-F09.

• Look at the relationship between the two years.Also Tried:1) Converting Scales Scores to Z Scores2) Compute by-student ‘’change”: Year2 Z_Score – Year1 Z_Score3) Compute mean teacher change

Computing By-Student Residuals (Predicted – Actual)Math F07-F08

Scal

e Sc

ore,

Gra

de 5

Scale Score, Grade 4

How Much of the Variance can be Explained at the Teacher Level?By Teacher, Mean Residual, Fall07-08

Math: 15%

Reading: 1.5%

What is the Relationship Between Teacher Level Mean Residuals, F07-08 & F08-09?

Correlation:Math: .32 Reading: .19

Method 1 – General Results

Rank – F07 – F08

Ran

k –

F08

– F0

9

Math –Teacher Ranks (Mean Residual), F07-08 X F08-F09

How do Teachers’ Ranks Compare, F07-08 to F08-F09?

MATH Within District Relationship - Quintile Rank – SS residual

How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?

Rank – F07 – F08

Ran

k –

F08

– F0

9

Reading –Teacher Ranks (Mean Residual), F07-08 X F08-F09

How do Teachers’ Ranks Compare, F07-08 to F08-F09?

READING Within District Relationship - Quintile Rank – SS residual

How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?

State Top-to-Bottom RankingPerformance Level Change IndexFOR EACH THE SCHOOL, THE PERFORMANCE LEVEL CHANGE SCORES ARE SUMMED ACROSS STUDENTS AND AN AVERAGE IS TAKEN TO CREATE THE TWO-YEAR AVERAGE PERFORMANCE LEVEL CHANGE INDEX. THE INDEX FOR EACH SCHOOL IS THEN COMPARED TO THE REST OF THE SCHOOLS IN THE STATE TO CREATE A SCHOOL-LEVEL PERFORMANCE LEVEL CHANGE Z-SCORE. THAT Z-SCORE IS THEN MULTIPLIED BY 1/4 TO CONTRIBUTE TO THE OVERALL SCHOOL INDEX IN THE CONTENT AREA.

Method 21) Compute by-student Performance Level (PL) Change Index2) Compute Mean Teacher PL change index

-- Computed with the New Cut Score Sub-Level Cut Scores --

2011 Top-to-Bottom PL Change Business Rules

-60

-50

-40

-30

-20

-10

0

10

20

30

40

50

60

ELEM_MS_RD_IMP_8_9

-60 -50 -40 -30 -20 -10 0 10 20 30 40 50 60

N = 2500r = .032p = .11

Relationship between: PL Change/Growth F07-to-F08 & PL Change/Growth F08-to-F09

Top-to-Bottom List*- School-Level Correlation - Reading, Grds 3-8F0

7 to

F08

F08 to F09*Original Top-to-Bottom Rules: (N Imp – N Dec )/ N Matched

ELEM/MS PL Change & Achievement Status

How much of the Variance can be Explained at the Teacher Level?By Teacher, Mean Residual, Fall07-08

Math: 15%

Reading: 4%

What is the Relationship Between Teacher Level Mean Residuals, F07-08 & F08-09? Correlation

Math: .42 Reading: .15

Method 2 – General Results

PL Change Index – F07 – F08

PL

Cha

nge

Inde

x –

F08

– F0

9

R = .42

Math –By Teacher PL Change Index, F07-08 X F08-F09

Relationship Between Teacher PL Change Index, F07 to F08 X F08 to F09

Mean MEAP Scl Score Z– F07

PL

Cha

nge

Inde

x –

F07

– F0

8

R = -.46

MATH –By Teacher, Mean MEAP Scl Score Z– F07 X PL Change IndexF07-08

What’s the Relationship Between a Teacher’s Classroom Starting Point (Mean Scale Score) and Their PL Change Index?

2011 State Top-to-bottom Math – Pl Change X Achievement Status(Old Cut Scores, New Rules)

-3

-2

-1

0

1

2

3

4

5

-5 -4 -3 -2 -1 0 1 2 3

PL C

hang

e Z

scor

e

Achievement Status z score

R = - .81

MATH –Teacher Ranks (PL Change Index), F07-08 X F08-F09

Rank – F07 – F08

Ran

k –

F08

– F0

9How do Teachers’ Ranks Compare, F07-08 to F08-F09?

MATH –Within District Relationship quintile rank - PL Change Index

How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?

READING –Teacher Ranks (PL Change Index), F07-08 X F08-F09

Rank – F07 – F08

Ran

k –

F08

– F0

9How do Teachers’ Ranks Compare, F07-08 to F08-F09?

READING –Within District Relationship quintile rank - PL Change Index

How do Teachers’ Ranks Compare, Within Districts, F07-08 to F08-F09?

PL

Cha

nge

Inde

x –

F07

– F0

8

Mean Residual

Relationship Between Teacher Mean Residual & PL Change Index

READING –By Teacher, Mean Residual X PL Change Index (F07to 08)

R = .88

PL

Cha

nge

Inde

x –

F07

– F0

8

Mean Residual – F07 – F08

Relationship Between Teacher Mean Residual & PL Change Index

MATH –By Teacher, Mean Residual X PL Change Index (F07to 08)

R = .77

…Error rates for teacher-level analyses will be about 26 percent if three years of data are use for estimation. This means that in a typical performance measurement system, more than 1 in 4 teachers who are truly average in performance will be erroneously identified for special treatment, and more than 1 in 4 teachers who [exceed] average performance by will be overlooked.

-17%-16%-6%

-12%

6%

-6%

11%3% 1%

-4%

-29%

-3% -2%-13%

-4%

13%11%

-4%-13%-14%

-50%-40%-30%-20%-10%

0%10%20%30%40%50%

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

Reading

MATH

-23%-12%

-22%-25%

-12%-9%-17%

-12%

4% 8%16%

22%

1%12% 10%

-1%

-41%-33%

23%12%

-50%-40%-30%-20%-10%

0%10%20%30%40%50%

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

F06-

F07

F07-

F08

F08-

F09

F09-

F10

Math

Gr3-Gr4 Gr4-Gr5 Gr5-Gr6 Gr6-Gr7 Gr7-Gr8

MEAP PL Change Index, Math & Reading, F06 -- F10 (GISD Students)

Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School)?

Original Top-to-Bottom Rules: (N Imp – N Dec )/ N Matched

PL C

hang

e In

dex

Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School?

Math

Reading

(OLD Cut Scores, NEW PL Index Rules)MEAP PL Change Index, Math & Reading, 08-09 & 09-10 (GISD Students)

PL C

hang

e In

dex

MATH

READING

Old & New Cut Scores

School-Level Std Dev - Math: .21 (08-09) .27 (09-10) Reading: .16 (08-09) .18 (09-10)

Math

Reading

Reading

Is Your PL Change Index Helped (or Hurt), based on the Grade You Teach? (Or, On the Grade Levels in Your School?

MEAP PL Change Index, Math & Reading, 08-09 & 09-10 (GISD Students)(NEW Cut Scores, NEW PL Index Rules)

PL C

hang

e In

dex

Developing a Growth/Progress Measure, Based on a Single Summative Assessment, that:

1) is stable across years;2) doesn’t punish high performing students/

classes/schools;3) incorporates the need to expect low performing

students/classes/schools to “grow” at a greater rate than low performing students/classes/schools.

Ain’t easy

Conclusion