dcps-tntp impact conference: overview and key … · 4. impact is a key part of dc public...

DCPS-TNTP IMPACT Conference: Overview and Key LessonsApril 2011

2© The New Teacher Project 2010

Contents

IMPACT Overview

Developing IMPACT: Highlights & Key Lessons

Implementing IMPACT: Highlights & Key Lessons


In October 2010, representatives from leading districts and states met in Washington, D.C. to learn from DCPS’ experiences with IMPACT.Conference ObjectivesThe New Teacher Project is focused on improving teacher effectiveness, and IMPACT represents one of the first of a new generation of evaluation approaches. It is critical to learn as much as we can from it. TNTP facilitated the conference to provide participants with the opportunity to:

• Learn about the IMPACT model in depth• Explore the challenges faced and lessons learned from development and implementation• Learn from peer districts and states that are innovating in different areas of teacher evaluation

and development• Engage in discussions with other district, union, and state leaders committed to achieving

large-scale student achievement gains through a fundamental shift in the way teachers are developed, evaluated, and held accountable for student learning

BaltimoreCharlotte-MecklenburgChicagoColorado

DenverIllinoisIndianaMemphis

MinneapolisNew HavenNew York CityRecovery School District (Louisiana)

Conference Participants – District, State, and Union Leaders from:

The conference was made possible with the generous support of The Joyce Foundation and The Michael & Susan Dell Foundation. Representatives from the DC Public Education Fund, the Broad Foundation, Fight for Children, and the Walton Family Foundationalso attended the conference.

Conference Funders and Other Participants

Rhode IslandTennessee


IMPACT is a key part of DC Public Schools’ effort to close its achievement gap by focusing on teaching and learning.

Teaching and Learning is at the heart of the Effective Schools Framework.

• The Framework is aligned to the district’s overarching Master Education Plan.

• The six elements of The Framework are the foundational elements of district reform efforts at the school level.

• IMPACT is a primary tool of reform, designed to put classroom instruction and teachers at the center of the conversation.

The Framework was designed to meet simple, yet powerful goals.

• Fairly and accurately assess teacher effectiveness.

• Improve practice by establishing clear expectations while providing meaningful support.

• Be as manageable and straightforward as possible – easy to understand and execute.


Grounded in a few key principles, IMPACT provides DCPS with the data needed to answer key human capital questions.

• Produce differentiation

• Be grounded in student outcomes

• Be rigorous

• Be relatively simple

• Have multiple measures

• Establish clear expectations

• Catalyze growth

• Provide the necessary data to answer key human capital questions

• Who are our top performers?

• Who needs additional support?

• Who needs to be separated from the system?

• What is the distribution of high-performing teachers across the school system?

• Is our professional development changing teacher practice?

• Which talent pipelines are producing the most effective teachers?

• In what subjects do we have an “effectiveness shortage?”

• Are we retaining our best teachers?

Guiding Principles Human Capital Questions


Contents

IMPACT Overview




IMPACT covers all school-based staff in DCPS, not only classroom teachers.

• Over 6,300 people across 125 schools• ~3,600 classroom teachers

Group 1: Teachers with Value-AddedGroup 2: Teachers without Value-AddedGroup 3: Special Education TeachersGroup 3a: Teachers of Students with AutismGroup 4: Non-Itinerant ELL TeachersGroup 5: Itinerant ELL TeachersGroup 6: Shared TeachersGroup 7: Visiting Instruction Service TeachersGroup 8: Student Support ProfessionalsGroup 9: Librarians

Group 10: CounselorsGroup 11: Social Workers and PsychologistsGroup 12: Related Service ProvidersGroup 13: Special Education CoordinatorsGroup 14: Program Coordinators and DeansGroup 15: Instructional CoachesGroup 16: Mentor TeachersGroup 17: Educational AidesGroup 18: Office StaffGroup 19: Custodial StaffGroup 20: Other

Faculty and Staff IMPACT Groups


IMPACT was developed and launched in one school year after an initial research phase to gather input from teachers and administrators.

Initial research

phase

Jun ‘07 Aug ‘07- Aug ’08 Sept ‘08 – Aug ‘09 Sept ’09 Jan’10 – Aug ’10 Sept ‘10

Initial design phase

Launch of IMPACT

v1.0

Implementation feedback and

system refinement

• Part of DCPS design team’s core responsibilities

• 100+ focus groups and feedback sessions on current experiences and opinions about IMPACT, with a heavy focus on the Teaching and Learning Framework

• Engaged 1,000+ teachers and administrators

• District-led audits for school-level quality control and compliance

• Part of DCPS design team’s core responsibilities

• 50+ focus groups and feedback sessions on various iterations of the design

• Engaged 500+ teachers and administrators

• Task forces• Rubric reviewers

• “Listening Tour”

• Engaged 1,500+ stakeholders through meetings (teachers, principals, and other school-based staff)

Launch of IMPACT

v2.0

Chancellor Rhee

appointed


Before intensive design work started, the IMPACT team considered several key design questions.

Key Questions DCPS’ Answers

Do you have the necessary student data?

DCPS made it a priority to have usable student data linked to teachers.

Do you want to use an existing system or design a new one yourself?

DCPS honed its vision of good teaching by designing its own system. However, districts with limited resources could choose to use an existing system and focus resources on the all-important implementation phase.

What is your timeline?DCPS spent one year to develop IMPACT 1.0, and much of the first year of implementation working on revisions for IMPACT 2.0.


Key Design Questions (cont’d)

Key Questions DCPS’ Answers

How should you staff your design team?

DCPS Design Team: two full-time employees, augmented by a large number of central office and school-based staff.

How should you solicit stakeholder input?

DCPS made a big effort to engage stakeholders by holding a “listening tour” and numerous focus groups. But the design team retained decision-making authority throughout the process.

How should you roll out the system?

DCPS rolled out the system all at onceand decided to make everything “count” in the first year instead of piloting or soft launching different parts of the system.


Major Design Decision Points (In Chronological Order)

• All school-based staff, excluding administrators

Other options consideredo Only teacherso All instructional staffo All school-based staff, including administrators

• Student achievement• Instructional practice• Broader contribution to the school • School-wide student achievement• Professionalism• Subject-relevant measures (e.g. completion of IEPs)

Other options consideredo Student work sampleso Lesson or unit planso Quality and use of assessmentso Peer evaluationso Self-assessmentso Student or parent surveys

What elements will each educator be assessed on?

Who will be included in your evaluation system?1

2

Design Decision Point DCPS’s Answer

Continued


Overall Design Decision Points In Order of Sequence (cont’d).

What will be the consequences for each group of scores?

4

5

How will each element be assessed?

• Student achievement: Individual Value-Added Student Achievement Data or Teacher-Assessed Student Achievement Data

• Instructional practice: Teaching and Learning Framework • Broader contribution to the school: Contribution to School

Community • School-wide student achievement: School Value-Added

Student Achievement Data • Professionalism: Core Professionalism• Subject-relevant measures: Individual Education Plan

Timeliness and Ordering Assessment Timeliness for special education teachers

3

How will these measures be combined to calculate a final score?

• Percentages: Easier to combine multiple components in an understandable way.

Other option consideredo Matrix

• Highly Effective: Eligible for ImpactPlus (Additionalcompensation program.)

• Minimally Effective: Additional support, salary step hold

• 2 Consecutive Years of Minimally Effective: Dismissal• Ineffective: Dismissal at the end of the school year

Design Decision Point DCPS’ Answer


DCPS hired 42 Master Educators whose primary responsibilities are to help evaluate and support teachers at schools across the district.

• Principals are the primary evaluators, but MEs support with 2 additional observations

• Each ME conducts about 200 30+ minute observations per year.

• Rate teachers on all 9 TEACH standards.

• Provide written explanation of ratings and guidance for growth, based on a detailed, multi-page template

Evaluate Teachers Support Teachers

•Provide specific guidance in post-observation conferences, including feedback and resources to develop each of the 9 TEACH standards.

•Hold weekly, informal “office hours”

•Conduct subject-specific “study groups”

•Provide additional support through informal mentoring

Master Educators are a highly selective corps of teachers with content area expertise, and a track record of success in low-income schools.


For most classroom teachers, IMPACT assesses the following components to determine the teacher’s final rating at the end of each school year.

Student Learning Individual Value-Added

(IVA) OR Teacher-Assessed Student Achievement Data

(TAS) for those teachers without individual value-

add scores

Teaching and Learning Framework

(TLF)

Commitment to School

Community(CSC)

School Value-Added(if data exists)

(SVA)

Core Professionalism

(CP)


IMPACT Components and Measures

IMPACT Component Measures

Individual Value-Add (IVA) Component *

(For Teachers with Value-Added Data)

• Measures a teacher's contribution to student growth on district-wide standardized tests.

• Controls for student's past academic history, poverty level and other factors beyond the teacher's control,including special education status, ELL status, and over-age status.

• Covers around 15% of teachers. (4th-8th grade reading and math).

Teacher-Assessed Student Achievement Data (TAS) Component

(For teachers without Value-Added Data)

• Measures a teacher’s impact on student achievement growth using pre-determined student performance targets on pre-determined assessments

• Teacher selects or creates assessments and sets annual student achievement goals (growth and/or mastery); Principal must approve assessments and goals

• All teachers have a TAS component, but it only counts (10%) for those without value-added scores.


IMPACT Components and Measures (Cont’d)

IMPACT Component Measures

Teaching and Learning Framework (TLF) Component

• The Teaching and Learning Framework has three domains: Plan, Teach, and Increase Effectiveness, however, teachers are assessed only on the TEACH domain

• The TEACH domain of the TLF includes 9 standards that are observable and imbedded in classroom/instructional practice *

• Five 30-minute observations per year on all 9 TEACH Domain standards

o 3 by an administrator; 2 by a master educatoro First administrator observation is announced;

all others are unannouncedo Post-conference within 15 calendar days after

each observation

Continued

*Based on stakeholder feedback during IMPACT 1.0, DCPS revised the TEACH rubric to be less rigid for IMPACT 2.0. The rubric language was adjusted to be more student-centered, and the 13 TEACH standards were streamlined down to 9—essentially collapsing 5 standards related to student behavior and classroom management into 1 standard about building a learning focused classroom culture.


IMPACT Components and Measures

IMPACT Component Description

Commitment to School Community (CSC) Component

• Measures support of school initiatives, partnership with families, and willingness to collaborate.

• Included because of DCPS’ belief that students’ success depends on the collective efforts of everyone in the school

• Assessed by Principals

School Value-Added (SVA) Component

• Measures impact of entire school on student learning

• Every employee in the school receives the same SVA score, which is weighted 5% for nearly all teachers

Core Professionalism (CP) Component

• Measures four basic tenets of professionalism: attendance, tardiness, policies and procedures, respect

• Assessed by Principals


Different combinations of components are used to evaluate teachers and staff, depending on the group. • Each component scored on a 1 (lowest) to 4 (highest) scale• All classroom teachers have at least a student learning component and an

instructional practice component• Nearly all teachers and staff have a school-based component*

* Itinerant ELL teachers, visiting instruction service teachers, related service providers, and mentor teachers are the only groups that do not include the Commitment to School Community and School Value-Added Student Achievement Data components as part of their evaluation criteria.

General Education With Value Added Data

General Education Without Value Added Data


DCPS created position-specific rubrics for groups evaluated on performance standards other than the Teaching and Learning Framework.

For example, a Special Education, Individualized Instruction rubric:

DCPS created rubrics for various groups, including:

• Special education teachers who work within the autism program

• Counselors• Instructional coaches• Mentor teachers• Custodians

• Student support professionals• Library media specialists• School-based social workers and psychologists• Program coordinators and deans• Visiting instructional service teachers• Related service providers• Office staff


How is the final rating calculated?

Step 1: Each component is scored on a 1 (lowest) to 4 (highest) scale and averagedo Example 1: TLF is assessed five times throughout the year through five observations.

These five scores are averaged to calculate the overall TLF component score. o Example 2: CSC is assessed twice a year. These two scores are averaged to calculate

the overall CSC component score.

Step 2: Each overall component score is multiplied by its assigned weight percentageStep 3: Weighted scores are added to come up with a total IMPACT scoreStep 4: Total IMPACT score is adjusted by the Core Professionalism score (if applicable) to

arrive at the final rating o If the teacher meets CP standards, no adjustments are made to the final score. If the

teacher is below standards, either 10 or 20 points are deducted.


What is the overall rating scale and what do the ratings mean?

Effective

oPerformance that is below expectations

oAdditional professional development

oSalary “hold”oIf excessed, 60 days to

find mutual consent placement

o2 consecutive years = separation

Highly Effective

oUnacceptable performance

o1 year = separation

Ineffective Minimally Effective

oSolid performanceoNormal salary

advancement

oOutstanding performance

oAdditional compensation for WTU members (As determined through the IMPACTpluscompensationsystem)


Words of Advice on Design from the IMPACT Team

People matter. DCPS focused on getting the right staff to design and implement IMPACT, beginning with the central office.

Prioritize implementation. You have to balance comprehensive, ideal design with support during implementation. Target your resources to give teachers and evaluators as much support as possible.

Continuous improvement. Embrace transparency and communicate regularly to teachers that reform is a process. You don’t know what works and what doesn’t until you try it, and you improve based on what you learn.

Work with urgency. Two years is enough time to develop an effective, accurate system We have to work quickly, and we can.

Communicate clearly and regularly. Be prepared to be unpopular for a while. Know what you want and how to explain your rationale to various stakeholders.


Other Design Lessons Synthesized by TNTP.

Design is not a one-time cost. DCPS did not disassemble the design team. They immediately started re-designing in January of Year 1 with focus groups of teachers and master educators to improve on the process and framework.

Singular focus. DCPS made choices to stop doing other “stuff” to focus on IMPACT. School staff are accustomed to hearing about new initiatives on a monthly basis. Concentrating on IMPACT, and only IMPACT, allowed staff to work with unquestioned focus and clarity.


Contents

IMPACT Overview




IMPACT Operations Team Overview

Responsibilities (As of Year 1 Implementation)• Data Collection and Management

• DCPS uses QuickBase (Intuit), a customized off-the-shelf program, to collect assessment data through an online portal: impactdcps.dc.gov

• Track and report principal completion data• Manage the IMPACT platform (online data portal)• Fulfill internal IMPACT data requests• Monitor quality of IMPACT assessments and data

• Training and Support• Train superintendents, principals, APs, teachers, and other school staff

• Other Logistics• Coordinate operational details of value-added “roster confirmation”• Manage and respond to the IMPACT Helpline and email address• Produce preliminary and final IMPACT reports for 6300+ employees• Manage operational details of IMPACTplus, DCPS’ performance based compensation

system.

Mission• To ensure that IMPACT is implemented faithfully • To ensure that every stakeholder has a strong understanding of the system

Team Structure• 6 full-time employees: 1 Manager, 4 Coordinators (1 for Special Education), 1

Program Assistant• Larger districts will require more staff


Instructional superintendents, principals, and teachers required different kinds of support as IMPACT was implemented

Group Support Need Support Provided

Instructional Superintendents

(principals’ managers)

Supporting principals on conducting observationsand giving feedback

•Twice per month Instructional Superintendent meetings

•One-on-one meetings

Principals

Conducting observations and giving feedback; meeting all IMPACT requirements

•Monthly cluster meetings

•Monthly Principals’ Academy

•One-on-one support

•IMPACT Helpline

Teachers (and other school-based staff)

Understanding IMPACT and the Teaching and Learning Framework; accessing IMPACT data

•Prof. development days

•School-based Q&A sessions, and trainings by IMPACT team

•IMPACT Helpline


DCPS invests heavily in the training and support of their Master Educators.

Training and Support Provided to Master Educators

• Six-week Master Educator Institute

• Three weeks of rubric analysis, video norming, report writing, and post observation conference practice

• Two weeks of informal observations

• One week of logistics training and teacher resource development

• Ongoing support

• Data analysis

• Master educator professional development

• Resource development

• Teacher professional development planning

• Mentoring and problem-solving in senior master educator cohorts


What DCPS looks for in their Master Educators

Core Competencies Description of Competency

Core Values/Beliefs Ethical, commitment to equity

Professional Expertise Content knowledge, pedagogy, professionaldevelopment

Entrepreneurial Spirit Self-starter, independent, flexible, success under adverse conditions

Interpersonal/ Communication Skills

Relate one-on-one with others, work in group, engaging presence

Problem Solving/CriticalThinking/Prioritizing Skills

Prioritizing in multiple scenarios, analyze data, respond quickly to changing situations

Writing SkillsWrite clearly and correctly, write quickly, write analytically, write in a manner appropriate for situation

Personal Development Self awareness, continuous improvement, open to feedback


Year One Overall Performance Distribution, PPEP (prior system) vs. IMPACT

n=3,469


Words of Advice on Implementation from the IMPACT Team

• Even though student achievement data carries a small weight for some teachers (10% for those without value-added data), it has been enough to motivate big changes in behavior and instructional practice.

• Without high stakes, people are not going to pay attention. High stakes incentivize people to focus on data and on performance. Teachers in DCPS are now proactively starting conversations about data and performance within the Teaching and Learning Framework.

• Feedback is more powerful when it is rooted in data. People assume data is only used to fire people, but IMPACT uses data to help teachers do their day-to-day jobs more effectively.


Key Implementation Lessons Synthesized by TNTP

• Implementation is where the game is won or lost. If districts are not willing to commit the resources to hire a sufficiently sized and highly talented team to build sustainable support structures, they are not going to be able to implement a new system.

• Anticipate operations challenges. Even simple tasks can become complex and time consuming. Think early about the obstacles unique to your district, and hire with those challenges in mind.

• Quality training is critical to successful implementation. DCPS trained their MEs for six weeks over the summer before they did a single observation. Districts must allocate the time to train intensely over the summer and maintain support throughout the school year.

• Districts will need systems to efficiently collect, analyze, and share evaluation data. DCPS invested upfront in an online platform and dedicated resources to provide data reports for all of Instructional Superintendents to use with their principals.

• Focusing on teacher evaluation and development offers an outstanding return on investment. IMPACT costs about $1,000 per employee but has led to fundamental changes in every classroom in the district.


To hear D.C. teachers speak out about IMPACT…

Click here for a video of D.C. teachers speaking about how IMPACT evaluations have improved their teaching practice. [Video courtesy of the Thomas B. Fordham Institute.]

Visit the Thomas B. Fordham Institute’s weekly bulletin The Education Gadfly for a series of videos on IMPACT and teacher evaluation.

http://www.youtube.com/watch?v=LoaidrMJ6wk&feature=youtu.be�

http://www.youtube.com/user/educationgadfly�


For more information about improving teacher evaluations…

Contact:Daniel Weisberg

Vice President of Policy and General CounselThe New Teacher Project

[email protected]

mailto:[email protected]�

dcps-tntp impact conference: overview and key … · 4. impact is a key part of dc public...

Documents