the use of student assessments in teacher evaluations through the next decade national conference on...

The Use of Student Assessments in Teacher Evaluations Through the Next DecadeNational Conference on Student AssessmentJune 27, 2012Minneapolis

edc.org

Agenda

• Introduction: Julie Kochanek• Study profile: Julie Riordan

– Performance-based teacher evaluation systems in five states

• Study profile: Karen Shakman– Changing culture and building capacity: An exploration of district

strategies for implementation of teacher evaluation systems

• District profile: David Heistad– The use of value-added in Minneapolis public schools

• Q & A discussion: Julie Kochanek

04/19/23 2

edc.org

An Examination of Performance-based Teacher Evaluation Systems

Karen ShakmanJulie Riordan

Maria-Teresa SanchezKyle DeMeo CookRichard Fournier

Jessica Brett

04/19/23 3

edc.org 04/19/23 4

Rationale for Study

• Regional need• Recent research• Race to the Top• State policy context

edc.org

Selection Criteria

• Required for all practicing general educators• Operational statewide in 2010–11 school year• Multiple rating categories• Multiple measures of teacher effectiveness

04/19/23 5

edc.org

Research Questions

• What are the key characteristics of state-level performance-based teacher evaluation systems in the study states?

• How do state teacher evaluation measures, the teaching standards the evaluations are designed to measure, and rating categories differ across states that have implemented statewide systems?

04/19/23 6

edc.org

Approach

• Examined 50 SEA websites• Conducted additional online search• Reviewed evaluation guides/manuals, rubrics,

training materials, regulations, program reports• Contacted state directors

04/19/23 7

edc.org

States Included in Study

• Delaware (RTT Phase 1)

• Georgia (RTT Phase 2)

• North Carolina (RTT Phase 2)

• Tennessee (RTT Phase 1)

• Texas (did not apply)

04/19/23 8

edc.org

Evaluation Measures

• Classroom observations• Principal evaluations• Analysis of classroom artifacts• Analysis of teaching portfolios• Teacher self-reports of practice• Student ratings of teacher performance• Value-added (student growth) strategies

(Goe, Bell, and Little, 2008)

04/19/23 9

edc.org

Findings: RQ1 – Key Characteristics

04/19/23 10

edc.org

Findings: RQ 2 – Teaching Standards

04/19/23 11

edc.org

Findings: RQ 2- Rating Categories

04/19/23 12

edc.org 04/19/23 13

Future Directions

• Further investigation about specific nature of the measures in place, including how student data are used in the overall assessment of teachers • Identify the types of knowledge and skills that are

commonly observed and evaluated in performance-based systems • Investigate the fidelity of implementation of these

systems

edc.org

Changing Culture and Building Capacity:

Karen ShakmanNicole BreslowJulie KochanekJulie RiordanTom Haeford

04/19/23 14

An Exploration of District Strategies for Implementation of Teacher Evaluation Systems

edc.org

Context

• National attention on teacher evaluation

• Focus on districts

• Systems change as framework

04/19/23 15

edc.org

Research Questions

1. What challenges are districts facing as they design and implement new teacher evaluation systems?

1. What strategies have districts developed to address these challenges?

04/19/23 16

edc.org

Approach

• Conducted semi-structured interviews with 16 district representatives

• Employed an emergent code strategy to transcripts and interview notes to identify key themes

04/19/23 17

edc.org

Findings: 2 Major Challenges

1. Changing district and school level culture

1. Building district and school level capacity

04/19/23 18

edc.org

Findings: Strategies for Changing Culture

1. Focus on effective teaching

2. Engage stakeholders in the design process

3. Involve the teachers union

4. Align professional development

5. Establish differentiated and direct communication

strategies

04/19/23 19

edc.org

Findings: Strategies for Building Capacity

1. Invest in training evaluators and teachers

2. Establish district-level, cross-functional teams

3. Identify new roles and fill gaps in expertise

4. Sequence implementation strategies

04/19/23 20

edc.org

Remaining Challenges

• Redefined role for principals

• Use of student achievement measures

• Lack of alignment between local and state level reform efforts

04/19/23 21

edc.org 04/19/23 22

Future Directions

• Research–Implementation studies–Measurement studies–Impact studies

•Building a community of practice

edc.org 04/19/23 23

edc.org

History of Value-Added Use in Minneapolis Public Schools

• Evaluation of the Public School Academy (1992-94) Longitudinal Achievement Effects – Rob Meyer and U of W

• Teachers who Beat the Odds in 2nd Grade Reading (1997-99)• Quality Performance Awards (1995–2005) Multiple Measures with

School Awards • Teacher Advancement Program (TAP) School, Grade level and

individual teacher bonuses (2006-2010)• Teachers who Beat the Odds in Kindergarten Literacy (2006-2010)• Grade level value-added to all K-8 Principals (2010-11) – Bush

Foundation and VARC• Classroom level value-added to all K-8 Teachers (2012-13) – VARC

and Dr. Chris Moore

edc.org

Predictive Validity of the Minneapolis Public Schools Beginning of Kindergarten Assessment (BKA)

25

Measures (MPS Beginning of Kindergarten Assessment)N= 1674

9 months later (EKA Total Literacy)

1.75 years later (Gr. 1 Oral Reading)

2.75 years later (Gr. 2 MAP Reading)

3.75 years later (Gr. 3 MCA Reading)

Alphabetic Principle - letter names - letter sounds

.74

.74

.66

.71

.71

.60

.64

.64

.54

.58

.52

.50Phonological Awareness - initial sounds - rhyming

.70

.63.65

.60

.58

.53

.63

.57

.59

.63

.57

.58Vocabulary - picture naming .61 .47 .54 .52Listening Comprehension .63 .52 .65 .67

Concepts of Print .55 .49 .55 .57

Total Literacy .81 .74 .70 .65

edc.org

Value-added Kindergarten Teacher Effects “beat the odds teachers”

• Post-test reading score=– Pretest literacy score– + Free or reduced price lunch– + Racial/ethnic code (4 dummy codes)– + Gender – + English Language Learner status– + Special Education status– + Lives with single parent– + Teacher effects

edc.org

Follow-up with teachers who beat the odds

• Surveys, observations, video tape• These teachers can be leads in staff development,

mentors, etc.• What about inviting these teachers to work in

schools with the high concentrations of poverty and students of color?

• Videos at: http://rea.mpls.k12.mn.us/uploads/teachers_who_beat_the_odds.pdf

edc.org

Current Work with the Bush Foundation and Value-added Research Center (VARC)

1) Teacher level value-added rolled out K-8 in 2012-13 (Rubric Scores and Value-added)

2) Graduates from 14 institutions tracked into MPS system for feedback to first year teachers and the Institutions of Higher Education (IHEs)

3) Studies of the human capital teacher pipeline

04/19/23Presentation Title Appears Here 28

edc.org

For more information….

• jkochanek@edc.org• jriordan@edc.org • kshakman@edc.org• dheistad@mpls.k12.mn.us

04/19/23Presentation Title Appears Here 29