how much value is added? an evaluation plan for the achievement challenge pilot project

How Much Value is Added? An How Much Value is Added? An

Evaluation Plan for the Achievement Evaluation Plan for the Achievement Challenge Pilot ProjectChallenge Pilot Project

Little Rock Merit Pay Year 1 Evaluation

2

What’s wrong? Teachers affect student performance, however… Policy problem

General & specific teacher shortages Measuring teacher effectiveness Providing incentives to teachers

Need System to recruit, retain, and reward high quality

individuals in the teaching field


3

Can current system rectify the problems? Status Quo:

Single Salary System Based on tenure and degree

Arguments for single system: Fair Simple (critics call it a “breathing bonus”) Status quo

Concerns: Lacks extrinsic reward for innovation, creativity, hard work Lacks extrinsic reward for innovation Does not encourage or reward outcomes Does not recruit, retain, or reward effective teachers

If status quo isn’t working, what alternatives do we have?


4

Policy Solutions “Lump Sums” (recruit and retain)

Often in the form of lump increases Intuitively lacks motivation to work harder

Differential Pay (recruit and retain) Hard-to-staff schools Specific subjects Disadvantaged students

Merit Pay (rewards) Teacher characteristics Teacher behavior Student performance gains

Literature: Johnson, 2000; Lazear, 1996; Murnane & Cohen, 1986


5

Merit Pay Literature Supporters believe performance improves:

Innovation Work harder Salary satisfaction

Opponents believe performance decreases Counter-productive competition Degraded work environment Focus on high-performing students

Evidence: Very few evaluations Policy questions:

Effects of merit pay programs on student performance? Effects of merit pay programs on teachers?


6

Possible Policy Implications

Possible options: Improves student achievement, and teachers like

program: Improves student achievement, but teachers

dislike program Does not improve student achievement, but

teachers like program; Does not improve student achievement, and

teachers dislike program.


7

Achievement Challenge Pilot Project (ACPP)

Program Goals: Increase student performance Reward effective teachers Make positive influences to school culture Ultimately, recruits, retains, and rewards effective

teachers 5 elementary schools in Little Rock School District Financial rewards based on student performance

payouts computed as NCE gains between fall and spring tests (SAT-9; SAT-10) Meadowcliff payouts per student gain Wakefield payouts based on class average gains


8

ACPP Addresses Literature ConcernsTable 1: Payouts for Wakefield for 2006-07

Employee Type / Position0-4%

Growth5-9%

Growth10-14% Growth

15%+ Growth

Maximum Payout

Principal $2,500 $5,000 $7,500 $10,000 $10,000

Teacher (Grades 4 -5) $50 $100 $200 $400 $11,200

Teacher (Grades 1-3) $50 $100 $200 $400 $10,000

Teacher (Kindergarten) $50 $100 $200 $400 $8,000

Coach $1,250 $2,500 $3,750 $5,000 $5,000

Specialist; Spec. Ed. $1,000 $2,000 $3,000 $4,000 $4,000

Music Teacher $1,000 $2,000 $3,000 $4,000 $4,000

Physical Examiner $500 $1,000 $1,500 $2,000 $2,000

Aide $250 $500 $750 $1,000 $1,000

Secretary & Custodian $125 $250 $375 $500 $500


9

“Observable” School Characteristics

School Name

Enrollment, 2005-06

% Free/ Reduced Lunch, 2005-06

% Black, 2005-06

2-Year % Proficient,

Math, 2003, 2004

2-Year % Proficient, Literacy,

2003, 2004

Meadowcliff 270 90% 80% 45.8 49.4Wakefield 365 94% 75% 47.2 54.0Baseline 202 96% 97% 54.9 59.2Chicot 367 89% 75% 37.3 44.1Franklin 281 95% 96% 25.0 45.3Treatment 635 93% 78% 46.5 51.8Control 850 93% 82% 38.1 48.4

Table 2: Demographic Characteristics of Schools in 2005-06 ACPP Evaluation


10

Research Question 1: Student Effects Question:

What is the impact of the ACPP on the math performance of students?

Method: Student level fixed-effects regression model Data provided by the Little Rock School District

Test scores Stanford Achievement Test-9 (2003; 2004) Iowa Test of Basic Skills (2005; 2006)

Reduces “gaming effect”

Demographic data Race, Poverty (FRL), Gender, Age


11

Methods: Data – Why Math Only?

SubjectCohorts, 2005-06 ITBS, 2006

ITBS, 2005

SAT-9, 2004

SAT-9, 2003

Reading 1 Voc K: Voc

2 Total 1: Total

3 Total 2: Total

4 R Comp; Total 3: R Comp 2: Total

5 R Comp; Total 4: R Comp 2: Total

Math 1 Total K: Total

2 Total 1: Total

3 Total 2: Total

4 Total 3: Total 2: Total

5 Total 4: Total 2: Total

Table 3: Summary of Tests by Grade and Year for Fall 2006 Report

ITBS 2005, Language subtest not administered to Grade 4 & 5


12

Methods: Analytic Strategy

Regression Student-level Individual Fixed effects

Compares the difference in test scores for treatment students to the difference in test scores for control students

This model only applies to 4th and 5th grade students because they are the only students who possess pre-gains (2002-03 or 2003-04 to 2004-05) post-gains (2004-05 to 2005-06) Meadowcliff removed – no pre-gain scores


13

RQ1: Study Sample CharacteristicsTreatment

(n = 132)Comparison

(n = 334)Total

(n = 466)

Schools 1 3 4

Grade Level, 2005-06

Grade 4 55 166 221

Grade 5 77 168 245

Race

% African-American 78.8% 83.5% 83.2%

% Caucasian 4.5% 4.8% 4.7%

% Hispanic 15.2% 8.9% 10.7%

% Other 1.5% 2.4% 2.2%

Free and Reduced Lunch Status

% Free or Reduced 90.8% 92.1% 91.7%

% Full Pay 9.2% 7.9% 8.3%


14

Teacher Effects

What are the attitudes regarding merit pay of ACPP teachers compared to those of teachers in the comparison schools?


15

Teacher Survey

Advantages

Innovation Work harder Salary satisfaction

Disadvantages Counter-productive competition Degraded work environment Focus on high-performing students

Teacher effectiveness


16

Policy Implications & Conclusions ACPP improves student performance

Student performance increased 3.5 NCE points

Teachers support the ACPP Significantly more satisfied with ACPP than single salary

system Believe the program did not lead to counterproductive

competition Believe the school environment is more positive with ACPP Teachers believe ACPP has positive impacts for students

Based on student performance increases and teacher opinions, program should be expanded to other elementary schools.


17

Limitations & Policy Concerns Receptivity is a factor

Teacher support may be vital to program success Limited sample of teachers (58 treatment) Limited sample of students (132 treatment)

All from same school Only two grades used

Funding $225,000 / school


18

Survey Practice & Questions

how much value is added? an evaluation plan for the achievement challenge pilot project

Documents