education policy initiative at carolina unc teacher quality research initiative: collaborating for...

Education Policy Initiative at Carolina

UNC Teacher Quality Research Initiative: Collaborating for Program

Accountability and Improvement

September 29, 2015

Kevin C. Bastian

Director, Teacher Quality Research Initiative


UNC Chapel Hill


The University of North Carolina at Chapel Hill

2

Outline• Background on the UNC Teacher Quality Research Initiative

• In-depth discussion on key research projects

• Individual-level data sharing and TPP improvement



3

UNC System Priorities

UNC System Overall Goal: • Preparing more, higher quality teachers and school leaders for North

Carolina Public Schools

Key Strategies to Address the Goal• Recruitment• Preparation• Beginning Teacher Support• Teacher Quality Research – Focused on outcomes/impacts



4

What is the Teacher Quality Research Initiative?

• Originated in 2008

• Partnership between the University of North Carolina General Administration, the Education Policy Initiative at Carolina (EPIC), and the 15 UNC System institutions that prepare teachers and school leaders

• Main goal: To conduct high-quality research and disseminate research evidence to improve the quality of teachers and school leaders prepared in North Carolina

• Integral component of the UNC system’s mission to prepare more and better teachers and school leaders for the public schools of North Carolina



5

Motivations for the Teacher Quality Research Initiative

• North Carolina as an accountability state

• Stalled progress in student performance and concerns about educational inequities

• Widespread attention on evidence-based reform

• Compelling evidence on the importance of teachers and teaching



6

Structure of the Teacher Quality Research Initiative

• Annual scope of research work between the UNC-GA and EPIC

• UNC Council of Education Deans Meetings—opportunities for dialogue, feedback and reflection

• Production of policy briefs, reports, and journal articles

• Benefits of working within a university system



7

Teacher Quality Research Initiative Studies

• Teacher portals analyses

• Program effectiveness analyses

• Recent Graduate Survey

• Job placement rates

• North Carolina Teaching Fellows evaluation

• Returns to teacher experience

• North Carolina New Teacher Support Program evaluation

• Graduate degrees in NC public schools

• edTPA validity and reliability studies

• NCTQ predictive validity study

• Personality traits and beginning teacher outcomes

• UNC MSA and Principal Fellows evaluations



8

Teacher Portals Analyses

• Static characteristic capturing teachers’ preparation prior to entering teaching

• Track demographic and compositional data on the NC teacher workforce

• Assess the performance of teachers entering the profession through different routes

• Value-added models: Statewide and within-school comparisons

• Teacher evaluation ratings

• Defined Portals• UNC undergraduate, graduate degree, and licensure only

• NC private undergraduate, graduate degree, and licensure only

• Out of state undergraduate, graduate degree, and licensure only

• Teach For America

• Visiting International Faculty

• Alternative Entry

• Unclassifiable



9

Teacher Portals Distribution



10

Summary of Portals Value-Added Results

Teacher PortalSignificantly

More Effective than UNC UG

Significantly Less Effective than UNC UG

No Different than UNC UG

UNC Graduate Degree1

(HS Math) 0 9

NC Private Undergraduate Degree 03

(ES Science, MS Math, MS Science)

8

NC Private Graduate Degree1

(HS Science) 0 7

Out-of-State Undergraduate Degree 0

4(ES Math, ES Science, HS

Math, and HS Social Studies)

7

Out-of-State Graduate Degree1

(HS English) 0 10

UNC Licensure Only1

(HS Science) 0 8

Out-of-State Licensure Only 0 1(ES Reading) 2

Teach For America

9(ES Math, ES Science, MS

Math, MS Reading, MS Science, MS Algebra, HS

Math, HS Science, HS Social Studies)

0 2

Visiting International Faculty2

(ES Math and ES Reading)1

(HS Math) 6

Alternative Entry 03

(MS Math, HS Math, and HS Social Studies)

8

Unclassifiable 0 2(MS Science and HS Math) 8



11

Summary of Portals Evaluation Rating Results

Teacher Portal Leadership Classroom Environment

Content Knowledge

Facilitating Student Learning

Reflecting on Teaching

UNC Graduate Degree 0.048* 0.059* 0.085* 0.044* 0.058*

NC Private Undergraduate Degree 0.013 0.011 -0.003 0.009 0.009

NC Private Graduate Degree 0.057* 0.066* 0.071* 0.034 0.043

Out-of-State Undergraduate Degree -0.005 -0.008 -0.004 -0.010 -0.004

Out-of-State Graduate Degree 0.010 0.016 0.011* 0.008 -0.007

UNC Licensure Only -0.017 0.018 0.045* -0.008 -0.014

Out-of-State Licensure Only -0.058 -0.076 0.038 -0.002 0.034

Teach For America 0.105* 0.073* 0.066* 0.067* 0.066*

Visiting International Faculty -0.043* 0.037 0.037 0.014 -0.025

Alternative Entry -0.051* -0.032* -0.017* -0.041* -0.045*

Unclassifiable -0.079* -0.055* -0.035* -0.059* -0.067*



12

Beginning Teacher Retention by Portal

Alternative Entry

Visiting International Faculty

Teach For America

Out-of-State University

NC Private University

UNC System

0 10 20 30 40 50 60 70 80 90 100

55.52

28.64

10.22

57.20

75.23

75.43

72.76

61.79

31.16

74.01

86.13

85.18

3 and 5 Year Persistence Percentages in NC Public Schools

Stay for 3 Stay for 5



13

UNC Program Effectiveness Analyses

• Track employment outcomes and school characteristics

• Assess the performance of initially-prepared teachers from each UNC system institution

• Value-added models: Statewide, within-school comparisons, student subgroups, selection vs. preparation

• Teacher evaluation ratings

• Overall analyses compare graduates of UNC system institutions to a common reference group of all non-UNC system initially-prepared teachers

• Program-specific analyses compare graduates of a specific institution to teachers entering through other routes of preparation



14

Job Placement Rates by UNC System Institution

WSSU

WCU

UNCW

UNCP

UNCG

UNCC

UNCCH

UNCA

NCSU

NCCU

NCA&T

FSU

ECU

ECSU

ASU

0 10 20 30 40 50 60 70 80 90 10076.98

72.93

66.50

85.79

74.29

79.10

74.12

75.66

72.10

72.00

69.08

76.22

78.24

68.30

73.51

60.89

61.08

55.85

79.07

66.50

71.73

67.03

58.65

64.55

60.22

63.34

70.58

69.33

55.72

62.44

Teach Within One Teach Within Three



15

Where Do UNCC Initially-Prepared Teachers Work?



16

School Characteristics for UNC System Graduates

School Characteristic ASU ECU ECSU FSU NCAT NCCU NCSU UNCA UNCCH UNCC UNCG UNCP UNCW WCU WSSU

% Free or Reduced-Price

Lunch56.08 62.08 65.74 65.04 65.63 64.42 43.14 55.05 47.34 55.03 61.55 73.12 57.05 59.07 74.98

Minority Percentage 36.22 56.41 55.65 68.60 68.76 73.61 48.05 32.14 52.21 49.57 54.03 68.07 46.85 28.76 71.77

Performance Composite 70.34 64.63 61.47 63.13 60.27 60.78 70.96 70.45 71.34 68.59 64.74 61.13 68.79 69.51 55.52

NBC Teacher Percentage 15.02 12.73 13.53 6.43 10.95 10.75 15.39 18.11 15.78 13.94 12.39 8.19 13.14 17.06 9.40

Returning Teacher

Percentage82.79 80.33 80.06 78.33 78.92 77.25 81.34 83.51 81.07 80.51 81.47 79.06 81.39 83.43 77.50

School Type

Elementary 53.61 61.86 59.95 57.11 42.59 58.60 19.62 46.01 54.10 66.68 60.55 64.27 61.79 54.66 61.62

Middle School 18.82 15.61 17.48 18.55 18.30 18.67 26.25 15.83 15.87 15.61 17.08 11.57 16.17 17.08 16.67

High School 26.87 21.97 22.09 23.90 38.17 22.73 54.00 37.67 29.72 16.84 21.85 23.99 21.56 26.05 20.20

Other 0.70 0.57 0.49 0.45 0.95 0.00 0.13 0.49 0.31 0.88 0.52 0.17 0.48 2.21 1.52

Teacher-School Year Count 8984 9861 824 1569 951 1157 3935 815 2621 5797 6682 2376 4986 3443 594



17

Summary of Programs Value-Added Results

Institution Number of SignificantlyHigher Value-Added Comparisons

Number of SignificantlyLower Value-Added Comparisons

Number of Statistically Insignificant Comparisons

ASU 0 0 10

ECU 2(MS Read & MS Algebra I) 0 8

ECSU 0 0 5

FSU 0 0 8

NCA&T 0 2(5th Grade Science & MS Math) 5

NCCU 1(5th Grade Science) 0 6

NCSU 2(ES Math & HS Algebra I) 0 8

UNCA 0 1(ES Math) 7

UNCCH 1(MS Math)

1(MS Algebra I) 8

UNCC 3(MS Read, HS Biology, & HS English I/II) 0 7

UNCG 1(ES Math)

3(MS Read, HS Biology, & HS English I/II) 6

UNCP 0 2(8th Grade Science & HS English I/II) 6

UNCW 3(ES Math, 5th Grade Science, & MS Math) 0 7

WCU 1(5th Grade Science) 0 9

WSSU 0 0 5



18

UNC Programs Value-Added Results

Al l Oth e rs

UNCA* -4.70

UNCG** 2.50UNCW* 2.80

NCSU** 6.20

WSSU -2 .5 0

ECU -1 .6 0UNCP -1 .6 0

UNCC 0 .1 0

WCU 0 .6 0ASU 0 .9 0FSU 0 .9 0NCA&T 1 .1 0NCCU 1 .2 0

ECSU 1 .6 0

UNCCH 2 .6 0

-6.0

00

.00

6.0

0S

tan

da

rd D

ev

iati

on

Dif

fere

nc

es

in

Pe

rce

nt

El e me n ta ry Sc h o o l Ma thPrograms Comparison

Al l Oth e rs

UNCA* -6.30

UNCG** 2.70

NCSU** 6.10

UNCP -1 .4 0

ECU -0 .9 0WSSU -0 .5 0

UNCCH 0 .4 0WCU 0 .5 0ECSU 0 .6 0NCA&T 0 .6 0UNCC 0 .8 0ASU 1 .2 0FSU 1 .2 0NCCU 1 .3 0

UNCW 2 .3 0

-7.0

00

.00

7.0

0S

tan

da

rd D

ev

iati

on

Dif

fere

nc

es

in

Pe

rce

nt

FRL El e me n ta ry Sc h o o l Ma thPrograms Comparison

Al l Oth e rs

UNCA** -8.30

UNCCH* 3.10

NCSU** 7.10

WSSU -2 .1 0ECU -1 .7 0

UNCP -1 .5 0

WCU -0 .3 0

ASU 0 .4 0UNCC 0 .4 0

ECSU 0 .6 0NCCU 0 .8 0

FSU 1 .8 0NCAT 1 .8 0

UNCG 1 .8 0

UNCW 2 .4 0

-10

.00

-5.0

00

.00

5.0

01

0.0

0S

tan

da

rd D

ev

iati

on

Dif

fere

nc

es

in

Pe

rce

nt

REM El e me n ta ry Sc h o o l Ma thPrograms Comparison

Al l Oth e rs

UNCA** -12.30

NCSU** 10.60

WCU -4 .8 0

WSSU -3 .3 0UNCP -2 .7 0

UNCG 0 .4 0

FSU 0 .5 0NCAT 0 .5 0

NCCU 0 .5 0

ECU 1 .2 0

ECSU 1 .6 0UNCC 2 .3 0

ASU 2 .9 0UNCW 2 .9 0

UNCCH 3 .6 0

-12

.00

-6.0

00

.00

6.0

01

2.0

0S

tan

da

rd D

ev

iati

on

Dif

fere

nc

es

in

Pe

rce

nt

L P El e me n ta ry Sc h o o l Ma thPrograms Comparison



19

UNC Programs Evaluation Rating Results

Al l Oth e rs

NCAT* 0.72

UNCP* 0.85

ASU* 1.11

UNCW* 1.17

UNCC* 1.25

UNCA* 1.71NCSU* 1.72

UNCCH* 1.80

WSSU 0 .8 1

FSU 0 .8 9

ECSU 0 .9 8UNCG 0 .9 9ECU 1 .0 0

NCCU 1 .0 3

WCU 1 .0 9

0.0

01

.00

2.0

0O

dd

s R

ati

os

Sta n d a rd 1Programs Comparison

Al l Oth e rs

WSSU* 0.80

UNCC* 1.14UNCW* 1.19

UNCA* 1.26

NCSU* 1.58

UNCCH* 1.88

NCAT 0 .8 4ECSU 0 .8 5

UNCP 0 .9 1FSU 0 .9 6

UNCG 0 .9 8

ECU 1 .0 1WCU 1 .0 2ASU 1 .0 3NCCU 1 .0 5

0.0

01

.00

2.0

0O

dd

s R

ati

os


Al l Oth e rs

WSSU* 0.66

ECSU* 0.77

ASU* 1.10

UNCW* 1.19

UNCA* 1.82

NCSU* 1.94

UNCCH* 2.06

UNCP 0 .8 7FSU 0 .9 0NCAT 0 .9 1

ECU 0 .9 7UNCG 0 .9 9UNCC 1 .0 4 WCU 1 .0 6NCCU 1 .0 8

0.0

01

.00

2.0

0O

dd

s R

ati

os


Al l Oth e rs

WSSU* 0.74NCAT* 0.79

ASU* 1.09

UNCC* 1.17

UNCW* 1.26

UNCA* 1.69

NCSU* 1.75

UNCCH* 1.92

UNCP 0 .8 7

ECSU 0 .9 4FSU 0 .9 9UNCG 1 .0 0NCCU 1 .0 2ECU 1 .0 4

WCU 1 .0 9

0.0

01

.00

2.0

0O

dd

s R

ati

os


Al l Oth e rs

NCAT* 0.68

UNCP* 0.77 WSSU* 0.79

ASU* 1.10UNCC* 1.15UNCW* 1.17

UNCA* 1.64

NCSU* 1.79

UNCCH* 1.95

FSU 0 .9 0

ECSU 0 .9 5UNCG 0 .9 5

NCCU 1 .0 3ECU 1 .0 4WCU 1 .0 4

0.0

01

.00

2.0

0O

dd

s R

ati

os




20

edTPA Validity Analyses

• Partnering with East Carolina University to conduct construct and predictive validity analyses on locally-evaluated edTPA data

• Locally-evaluated edTPA can provide faculty and staff:

• Common language and expectations for candidate performance

• Direct evidence about the extent to which candidates demonstrate specific knowledge and skills

• edTPA can leverage systems of continuous improvement

• Locally-evaluated edTPA may not guide TPPs to adapt/adopt more effective practices if the data are not valid and predictive of outcomes for program graduates



21

Construct Validity of Locally-Evaluated Portfolios

edTPA Construct edTPA Rubric Factor 1 Factor 2 Factor 3

Planning

Planning for Content Understanding 0.73 0.22 -0.11

Planning to Support Varied Student Learning Needs 0.72 -0.04 0.18

Using Knowledge of Students to Inform Teaching 0.45 0.01 0.35

Identifying and Supporting Language Demands 0.77 -0.06 0.12

Planning Assessment to Monitor and Support Student Learning 0.69 0.16 -0.02

Instruction

Learning Environment 0.20 0.61 -0.03

Engaging Students in Learning -0.01 0.82 0.04

Deepening Student Learning 0.03 0.76 0.07

Subject-Specific Pedagogy 0.01 0.61 0.19

Analyzing Teacher Effectiveness 0.03 0.18 0.62

Assessment

Analysis of Student Learning 0.03 0.11 0.73

Providing Feedback to Guide Further Learning 0.04 0.03 0.71

Student Use of Feedback -0.09 0.02 0.89

Analyzing Students’ Language Use 0.14 -0.02 0.72

Using Assessment to Inform Instruction 0.13 0.03 0.73



22

Predictive Validity of Locally-Evaluated Portfolios

edTPA Measures Std. EVAAS Estimate

Planning factor 0.063

Instruction factor 0.213**

Assessment factor 0.178*

Planning for Content Understanding 0.121

Planning to Support Varied Student Learning Needs 0.050

Using Knowledge of Students to Inform Teaching 0.029

Identifying and Supporting Language Demands 0.092

Planning Assessment to Monitor and Support Student Learning 0.102

Learning Environment 0.156+

Engaging Students in Learning 0.271**

Deepening Student Learning 0.210**

Subject-Specific Pedagogy 0.229*

Analyzing Teacher Effectiveness 0.134

Analysis of Student Learning 0.124

Providing Feedback to Guide Further Learning 0.194**

Student Use of Feedback 0.195*

Analyzing Students’ Language Use 0.215*

Using Assessment to Inform Instruction 0.154+

Standardized Total Score 0.184**

Cases 209



23

Predictive Validity of NCTQ Ratings

• Established formal collaboration with NCTQ in spring 2014

• Assess the relationship between NCTQ’s TPP ratings and measures of teacher performance in North Carolina

• Study premise: An underlying justification for standards for teacher preparation programs is that meeting the standards should lead to higher quality preparation practices, better teacher performance, and better student outcomes

• If premise holds, TPP have an incentive to undertake reforms to increase their scores/ratings on standards

• If premise does NOT hold, TPP efforts could be better spent undertaking reforms that do lead to improve teacher and student outcomes



24

Research Background for NCTQ Analyses

• Full sample includes 4,513 first and second year teachers in 2011-12 and 2012-13 in NC public schools

• Ran overall program rating, standards, and indicators models

• Ran models for 1st year teachers only and for 1st and 2nd year teachers combined

• Took three approaches to address missing data—case-wise deletion, multiple imputation, and dummy variable replacement



25

Summary of NCTQ Results

NCTQ Ratings

Associations with Teacher Performance MeasuresTeacher Value-Added Teacher Evaluation Ratings

Total Tests Significant Positive Associations

Significant Negative

AssociationsTotal Tests Significant Positive

AssociationsSignificant Negative

Associations

NCTQ Overall Program Rating 42 1 0 30 8 0

Totals with NCTQ Standards 124 15 5 140 31 23

Selection Criteria 14 3 0 10 8 0

Early Reading 4 0 0 10 0 0

English Language Learners 4 0 0 10 8 0

Struggling Readers 4 0 0 10 0 1

Elementary Mathematics 4 0 0 10 0 0

Elementary Content 4 0 1 10 9 0

Middle School Content 4 0 1 10 0 8

High School Content 6 0 0 10 0 1

Classroom Management 14 0 0 10 0 10

Lesson Planning 14 0 0 10 0 3

Assessment and Data 14 4 2 10 0 0

Student Teaching 14 2 1 10 0 0

Secondary Methods 10 0 0 10 6 0

Outcomes 14 6 0 10 0 0



26

Limitations and Takeaways

• Limitations: Generalizability, missing data, and conceptual alignment between NCTQ standards and teacher outcome variables

• Takeaways

• In NC there is not a strong relationship between NCTQ’s ratings and meeting their standards and the performance of teacher preparation program graduates

• Results do not suggest that teacher preparation is unimportant or that the preparation components rated by NCTQ are unimportant

• Results suggest creating outcome-rich environments for preparation programs, setting higher standards for admission into programs, and continuing to identify ways to measure program quality



27

From Accountability to Program Improvement Research

• Current research may encourage programmatic reforms and help teacher preparation programs better consumers of evidence but it CANNOT formatively drive evidence-based improvements

• With what data can teacher preparation programs make evidence-based improvements?

• Teacher preparation programs need individual-level data on program graduates

• Helps teacher preparation programs determine whether preparation practices are aligned with the working environments of their graduates

• Helps teacher preparation programs assess how variation in graduates’ preparation experiences explain variation in their characteristics and performance



28

Data Sharing in the UNC System

• Initiative designed to stimulate a culture of evidence and program improvement by providing teacher preparation program with individual-level data on their program graduates

• In the UNC system we are providing teacher preparation programs with separate data files per school year, with each file containing data on all the individuals who were initially prepared to teach by a teacher preparation and employed as teachers in that school year



29

What Data are Being Shared?

Employment Status Teacher Characteristics Classroom Characteristics School Characteristics Teacher Outcomes

1. District and school2. Number of pay

periods 3. First pay period4. Last pay period5. Amount of time

worked (full-time equivalency status)

1. Teaching experience

2. Graduate degree status

3. National Board Certification status

4. Licensure areas5. Licensure basis6. Exams taken7. Exam scores8. Teaching a tested-

grade/subject-area

1. Number of classes taught

2. Average class size3. Grade level(s) taught4. Subject-area(s)

taught5. Race/ethnicity

proportions6. Free and reduced-

price lunch proportions

7. Gifted proportion8. Disabled proportion9. Limited English

Proficient proportion10. Average days absent11. Average prior

achievement scores 12. Average prior

achievement level

1. Urbanicity2. School size3. Percentage free and reduced-

price lunch4. Short-term suspension rate5. Violent acts rate6. Race/ethnicity percentages7. Total per-pupil expenditures8. Per-pupil expenditures in

spending categories (e.g. regular instruction)

9. AYP percentage10. State accountability status and

growth11. Performance composite12. Teacher credentials—percentage

fully-licensed, novice, holding an advanced degree or NBC

13. Pupil to teacher ratio14. Teacher stay ratio

1. Returns to the state’s public schools

2. Returns to the same school

3. Teacher value-added estimate (across 10 separate subject-areas)

4. Quintile for value-added estimate



30

Connections to Program Improvement



31

Thoughts on Program Improvement

• These data tell you where to look

• Programs need individual-level data on program graduates

• Important to conduct drill-down studies to understand WHY

• Important to have multiple outcomes to assess TPP performance

• Program improvement aided when outcomes point in the same direction

• Opportunities to leverage performance assessment data and to examine candidate success on new licensure examinations

• Program improvement requires leadership, capacity, and culture



32

Future Research Directions

• edTPA analyses

• Student teaching and clinical placements

• Personality traits/non-cognitive characteristics

• School leader programs analyses

• Assistant principal experiences

• Incorporation of additional outcome measures

• Program-specific research and data use



33

Thank You!

• Contact Information

[email protected]

http://publicpolicy.unc.edu/

http://tqdashboard.northcarolina.edu/

mailto:[email protected]





education policy initiative at carolina unc teacher quality research initiative: collaborating for...

Documents