reversing the widget effect teacher union reform network i november 9 -10

Reversing The Widget EffectTeacher Union Reform Network I November 9 -10

2© The New Teacher Project 2009

Agenda

About The New Teacher Project

Studying Teacher Evaluation

The Current Problem

The Current Opportunity

Preliminary Data About Peer Review Programs

How to Build a Better Evaluation System

Appendix


The New Teacher

Project (TNTP)

helps school

districts and states

fulfill the promise of

public education by

ensuring that all

students—

especially those

from high-need

communities—get

excellent teachers.

• National nonprofit, founded by teachers in 1997

• Partners with school districts, state education agencies, and charter schools

• Targets acute teacher quality challenges

• Delivers a range of customized services and solutions on a fee-for-service basis

• Approx. 200 employees, most embedded in school district offices; majority are former teachers

• Past and present clients include:

Districts: Baltimore, Chicago, Denver, Memphis, New Orleans, New York, Oakland, Philadelphia, San Antonio, Washington, DC

States: Alaska, Louisiana, Texas, Virginia


Agenda



The Current Problem




Appendix


Study Scope

In Fall 2008 through Spring 2009, TNTP partnered with 12 school districts in four states to analyze each district’s evaluation, tenure, remediation and dismissal policies and practices as well as each state’s teacher performance management policies.

Represented:

• Urban, suburban and rural districts

• Collective bargaining and non-collective bargaining states

• Peer review and traditional evaluation approaches


Data Collection

Arkansas Colorado Illinois Ohio TOTAL

Districts Studied 4 2 3 3 12

Teachers Surveyed 2,196 2,428 7,482 3,070 15,176

Principals Surveyed 117 184 794 186 1,281

Former Teachers Surveyed N/A 344 227 227 798

Interviews Conducted* 36 23 36 41 136

Teacher Evaluation Records Collected 1,588 5,855 28,147 7,040 42,630

*Interviews conducted with some combination of teachers, principals, local teacher union leaders, local principal professional organization leaders, superintendent, district HR staff, legal counsel, and school board members.


Advisory Panels

Membership• One panel per state• Approximately 80 stakeholders

total• Advisory Panel members

include:o District leaders and HR staffo Local and state union

leadershipo School board memberso State policy makerso Education advocateso Foundation representatives

• Three formal meetings, 2-3 hours long.

TNTP’s research benefited from the extensive input and feedback of Advisory Panels in each of the four states studied.

Role

• Share information and perspective on state/district context.

• Suggest issues for additional exploration and sources of information.

• Comment on study methodology and data collection methods and instruments.

• React to preliminary findings and recommendations.

• Craft brief responses to the study for publication in unedited form.


Agenda



The Current Problem




Appendix


The Widget Effect

“When it comes to measuring instructional performance,

current policies and systems overlook significant

differences between teachers. There is little or no

differentiation of excellent teaching from good, good from

fair, or fair from poor. This is the Widget Effect: a

tendency to treat all teachers as roughly

interchangeable, even when their teaching is quite

variable. Consequently, teachers are not developed as

professionals with individual strengths and capabilities,

and poor performance is rarely identified or

addressed.”

The New Teacher Project, 2009


Teacher Evaluation: Where We Codify The Widget Effect

Treating teachers as interchangeable parts

All teachers are rated “good” or “great.”

Although teachers and principals report that poor performance is common, less than 1 percent of teachers are identified as “unsatisfactory” on performance evaluations.

Excellence goes unrecognized.

When excellent ratings are the norm, truly exceptional teachers cannot be formally identified. Nor can they be compensated, promoted or retained.

Professional development is inadequate.

Almost 3 in 4 teachers did not receive any specific feedback on improving their performance in their last evaluation.

Novice teachers are neglected.

Low expectations for beginning teachers translate into benign neglect in the classroom and a toothless tenure process.

Poor performance goes unaddressed.

Half of the 12 districts studied have not dismissed a single non-probationary teacher for poor performance in the past five years. None dismisses more than a few each year.


When multiple ratings are available, teachers tend to be assigned the highest ratings and are very rarely assigned poor ratings.


In districts that use binary “Satisfactory/Unsatisfactory” rating systems, the “Unsatisfactory” rating is almost never used.

Evaluation Ratings for Tenured Teachers in Districts with Binary Rating Systems


These data often stand in sharp relief against current levels of student achievement, as this example from Denver illustrates.

Frequency of Unsatisfactory Ratings in Denver Public Schools that Did Not Meet AYP

In Denver schools that did not make adequate yearly progress (AYP) for three consecutive years (2005-06 to 2007-08), more than 98 percent of tenured teachers received the highest rating—satisfactory.


Teachers say districts aren’t doing enough to recognize and retain top performers as measured by their impact on student learning…

59%of teachers report their district is not doing enoughto identify, recognize, compensate, promote and retain the most effective teachers as measured by their impacton student learning.

“All the good quality teachers leave the district after just a few years. They need more incentive to stay.”

“Some sort of recognition or praise would be nice. Those doing a good or great job are never told so.”

“If you pay the shining stars the same as the slackers, you will dim the shining stars and reinforce the sloth of the slackers.”

“I, and others, work hard because we have a conscience, but I don't think [the district] sees us as any different than the lower performing teachers. Teachers who work hard receive very little praise or notice.”TNTP survey of 7,318 teachers across four sites conducted May 2008 to April 2009


Percent of teachers who had development areas identified on their most recent evaluation.

Tenured/non-probationary

teachersProbationary

teachers

Most likely (Denver) 32% 55%

Average 22% 37%

Least likely (Springdale) 2% 4%

Source1: TNTP survey of 15,176 teachers across 12 sites conducted May 2008 to April 2009Source2: TNTP survey of 1,863 Denver Public School teachers conducted November to December 2008

…or to ensure that evaluation practices and systems provide them with meaningful feedback.

of Denver teachers who had

a development area identified on their most recent evaluation “do

not know” which

performance standard they failed to meet.

39%


64%58% 63%

40%

72%78% 75%

46%

Akron Chicago Little Rock Springdale

Teachers Principals

And despite the fact teachers and principals agree that poor instruction is pervasive…

Source: TNTP survey of 7,318 teachers and 751 administrators across 4 sites conducted February to April 2009; teacher evaluation data provided by each district.

“Are there tenured/non-probationary teachers in your school who deliver poor instruction?”

(Percent responding “Yes”)

0% 0.4% n/a 0%

Percent of All Ratings that Indicated “Unsatisfactory” Performance


…dismissal for poor instructional performance almost never occurs.

Frequency of Teacher Dismissals for Performance (Non-Probationary Teachers)


Agenda



The Current Problem




Appendix


Research has shown that teachers have a greater impact on student learning than any other school-based factor.

Dallas students who start 3rd grade at about the same level of math achievement…

57

55

0 20 40 60 80 100

Group 2

Group 1

Average Percentile Rank

Beginning of 3rd Grade

27

77

0 20 40 60 80 100

Group 2

Group 1

Average Percentile Rank

End of 5th Grade

After 3 EFFECTIVE

Teachers

After 3 INEFFECTIVE

Teachers

…may finish 5th grade math at dramatically different levels depending on the quality of their teachers.

Original analysis by the Education Trust.

Source: Heather Jordan, Robert Mendro, and Dash Weerasinghe, The Effects of Teachers on Longitudinal Student Achievement, 1997.

50


Teacher Effectiveness (e.g., Value Add, Growth, Eval. Rating)

Boost effectiveness of all teachers through effective evaluation and targeted professional development

Improve or exit persistently less effective teachers and replace with more effective

teachers.

Retain and leverage most effective teachers

5

2

4

1 Optimize new teacher supply by hiring from preparation programs whose teachers consistently achieve higher student outcomes.

Increase the concentration of effective teachers in high-need

schools.

3

Current Performance

Potential Performance

Therefore, dramatic improvements in student achievement cannot occur without a sustained focus on maximizing teacher effectiveness.

5Goals for Optimizing Teacher Effectiveness


TNTP Human Capital Continuum

Education leaders must use clear evaluations of teacher effectiveness to inform decisions at each step of the human capital continuum.

Supportive policy context that removes barriers and publicizes results / outcomes data

OptimizingSupply

ManagingEffectiveness

Recruitment

Selection

Training /Certification

Hiring / Placement

On-Boarding

School-LevelHuman Cap.

Mngmnt.

Evaluation / Prof. Dev.

WorkingConditions

Retention / Dismissal

Compensation

Evaluation system that differentiates

teachers based on student academic growth


Most of the districts we studied considered teacher effectiveness only when making decisions about remediation and dismissal.


• Improvement: All teachers should report receiving ongoing feedback that helps them improve, and there should be evidence of that improvement in their observations and in student performance.

• Higher Standards: Teachers should see a high standard of instruction being upheld for themselves and others, and they should not report instances of poor performance going unaddressed.

• Credibility: Teachers should feel evaluations are accurate and credible.

• Differentiation: Each school should know which teachers are currently performing at great, good, fair, and poor levels.

• Recognition: Examples of outstanding instruction should be recognized and made available to other teachers to help them improve.

• Support: Professional development should be aligned to the evaluation process.

Outcomes of an Effective Teacher Evaluation System


Agenda



The Current Problem




Appendix


For many years, peer review has been an innovative approach to teacher evaluation, including in Toledo, a Widget Effect study site.

Components of peer review:

• Early career mentoring and evaluation as a component of the tenure granting process.

• Joint labor-management decision-making about renewal, support for struggling teachers and teacher dismissal, usually through the use of lead or master teachers.

• Intervention for poor performance where teachers or principals refer tenured teachers for intervention when necessary.

Beliefs guiding peer review:

• Teachers have a vested interest in the quality of the workforce and will hold their peers to even higher standards than administrators would.

• The best way to ensure quality is to provide intensive support and feedback to beginning teachers and create avenues for intervention with struggling veterans.

• Teachers and administrators will make the appropriate referrals for intervention when they observe cases of poor performance.


Compared to teachers in other districts we studied, teachers in Toledo are very satisfied with the supports they receive.

*Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.”**Source: Survey of 13,664 teachers in 10 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.”

66% 65%63% 60%

56%52% 51%

43% 42% 42%

20%

71%

64%

57%

52% 52% 51%48% 47% 46%

38%

The probationary period provides new teachers with adequte support to develop the skills and knowledge they need to be successful in the classroom.*

The district’s remediation provides poorly performing tenured teachers with an opportunity to improve their instructional performance.**

Districts studied in The Widget Effect

Toledo


But Toledo ranks in the middle among districts whose teachers believe the evaluation system accurately assesses them and helps them improve their instruction.

64%

59%53%

48% 48%42% 42% 40% 39% 38%

32%

62%58%

55%50% 50%

43% 43% 42% 39% 38% 36%

The evaluation process allows evaluators to accurately assess teachers’ instructional performance.*

The evaluation process helps teachers improve their instructional performance.**

*Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.”**Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.”

Toledo



About half of Toledo teachers say there is at least one tenured teacher in their school whose instruction is poor enough to warrant dismissal.

79%

64%

51% 50% 49%43% 43%

34% 34% 33% 30%

23%

Percent of teachers who say there is a tenured teacher in their school who should be dismissed for poor instructional

performance, but has not been

Source: Survey of 15,176 teachers in 12 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Yes.”


Toledo


According to data from the Toledo Federation of Teachers, the peer review program resulted in 26 departures over a four period.

Non-Tenured Teacher Dismissals Tenured Teacher Dismissals

School Year Formal Informal Formal Informal

2004-2005 4 2 0 2

2005-2006 4 0 0 0

2006-2007 2 0 0 2

2007-2008 4 5 0 1

TOTAL 14 7 0 5

6.5tenured teachers exited annually through formal and informal means, on average. The average number of teachers in Toledo during this

time was 2,084.


And less than half of Toledo teachers believe the district maintains high instructional standards—among the lowest percentages of the districts we studied.

72%68% 67%

58%52% 49% 49%

45% 44% 42% 41%

Percent of teachers who say their district enforces a high standard of instructional performance for all teachers

*Source: Survey of 14,611 teachers in 11 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Strongly agree” or “Agree.”**Source: Survey of 15,176 teachers in 12 districts, including 773 teachers in Toledo Public Schools. Percent that responded “Yes.”

Toledo



• Continue visits throughout a teacher’s career, whether they are referred or not. Conduct random observations to audit the quality of instruction, and refer teachers to PAR if performance is poor.

• Actively canvas teachers about the quality of instruction in their team, grade level and school. Do not wait for teachers to come forward and report problems.

• Apply multiple ratings rather than only satisfactory and unsatisfactory. Nuanced provides richer feedback, allows progress to be tracked, and recognizes excellence.

• Use PAR for more than determining which teachers are dismissed. The most comprehensive evaluation systems drive decisions about teacher development, assignment, promotion, compensation, etc.

We recommend several strategies to ensure peer review is as effective an evaluation system as possible.


Agenda



The Current Problem




Appendix


Evaluator Staffing• Staffing structures that allow evaluators

to dedicate the time necessary to observe and evaluate teachers.

Multiple Evaluation Criteria Teacher contributions to student learning.

Teacher actions, as compared to clearly articulated criteria that reflect what exemplary performance looks like at a given experience level.

District Policies• Shift away from centralized PD to tailored

school- level development.

• Requirement that teachers’ evaluations drive renewal, compensation, career path opportunities, and other rewards / consequences.

• Probationary periods that last long enough to assess effectiveness comprehensively.

• Reasonable documentation requirements.

• Efficient processes for recognizing excellence and addressing ineffectiveness.

Strong evaluation systems are based on a comprehensive set of strategies .

School and District Leader Accountability

• Accountability for school and district leaders for conducting accurate, useful performance evaluations and improving teacher performance.

• Processes and checks to ensure accuracy of evaluations (e.g., tenure review board).

Evaluator Training and Support• Training and ongoing support for

administrators and other evaluators to ensure consistency in evaluations.

• Training on how to use evaluations to inform feedback and individualized developmental interventions.


Traj

ecto

ry

Lo

wH

igh

Effectiveness

Low High

Teacher evaluations should assess both effectiveness and trajectory.

Place on a rigorous

improvement plan or replace

Recognize and appreciate, but offer targeted

support

Renew for another year,

but expect more

Reward, retain, and provide

chances to lead from the

classroom

• Effectiveness: The teacher’s positive impact on student learning during the evaluation period

• Trajectory: The rate of improvement the teacher has demonstrated during the evaluation period


Expectations should increase with experience, and tenure should be awarded only if a teacher meets those expectations consistently.

Eff

ecti

ven

ess

Year 1 Year 2 Year 3 Year 4 Year 5

Minimum expectations for

first year teachers to get

renewed

Minimum expectations

for teachers to get tenure

Duration of District’s Probationary Period (e.g., 5 years)

Expectations

The level of effectiveness at which the teacher’s performance could hold steady and still fully meet expectations in the future.


Research suggests that precise language will improve calibration of raters.

Less Precision More Precision

• Criteria that are articulated in such a vague or confusing manner that we are not confident principals are able to minimize subjectivity and make repeatable assessments.

• Criteria that are oversimplified, too high level, and not actionable, so much so that one can interpret excellence to be very different things (e.g., may allow lower expectations, rather than higher expectations).

• Criteria that are so poorly worded that it is unclear what is being evaluated.

• Criteria that evaluate something entirely different from what it can reasonably assumed it was intended to measure.

• Criteria that bundle multiple, distinct, and elements criteria together and force the principal to make one evaluation.

• Criteria that are articulated in such a way (e.g., with precise and clear language) that principals can use the criteria to make repeatable evaluations while minimizing false positives.

• Criteria that are precisely defined for the teacher and the principal.

o The criteria help principals know what to look for and minimize interpretation and subjectivity.

o The criteria help teachers know what is expected of them, and know what excellence looks like.

“One way to increase reliability is to ensure that the evaluation instrument has clearly defined, non-subjective criteria that require minimal interpretation.” (NCCTQ, 2008)


Agenda



The Current Problem




Appendix



The New Teacher Project (TNTP) helps school districts and states fulfill the

promise of public education by ensuring that all students—especially those

from high-need communities—get excellent teachers. A national nonprofit

organization founded by teachers, TNTP is driven by the knowledge that

although great teachers are the best solution to educational inequality, the

nation’s education systems do not sufficiently prioritize the goal of effective

teachers for all. In response, TNTP develops customized programs and policy

interventions that enable education leaders to find, develop and keep great

teachers and achieve reforms that promote effective teaching in every

classroom.

Since its inception in 1997, TNTP has recruited or trained approximately

33,000 teachers—mainly through its highly selective Teaching Fellows™

programs—benefiting an estimated 4.8 million students. TNTP has also

released a series of acclaimed studies of the policies and practices that affect

the quality of the nation’s teacher workforce, most recently including The

Widget Effect: Our National Failure to Acknowledge and Act on Differences in

Teacher Effectiveness (2009). Today TNTP is active in more than 25 cities,

including Baltimore, Chicago, Denver, New Orleans, New York, and Oakland,

among others. For more information, please visit www.tntp.org.

http://www.tntp.org/


For more information:www.tntp.org

http://www.tntp.org/

reversing the widget effect teacher union reform network i november 9 -10

Documents

new teacher project

teacher evaluation records

districts evaluation

new york

new orleans

local teacher union

school districts

virginia slide