ncsu · 2016-02-10 · epp progress against the targets are evaluated annually by the epp and...

Teacher Production Evidence

Our EPP works collaboratively with the University of North Carolina System to carefully monitor the teacher workforce needs for the state of North Carolina. This process begins with the development of Proposed

Graduation Targets. The established targets are used by the EPP in targeted recruitment efforts. EPP progress against the targets are evaluated annually by the EPP and University System and adjusted as

appropriate. Each year the University system provides each EPP with the productivity totals for the full system as a Teacher Productivity Report. In addition to teacher production our EPP collaborates with the University

System in the evaluation of each individual program. The University system, through the Carolina Institute for Public Policy (CIPP) (now called Education Policy Institute at Carolina EPIC), develops a program effectiveness

report. The UNC Program Effectiveness Report evaluates each campus program through the results of completers K-12 standardized test scores. This data is next disaggregated by campus and results in the NCSU

Program Effectiveness Report. Our EPP reviews this annual report and triangulates the data with existing sources to determine if an area for improvement exists. Finally, the UNC System, again through CIPP/EPIC,

develops a Teacher Portals Policy Brief. This policy brief makes comparisons of each of the pathways to teaching in North Carolina which includes UNC System, NC Independent Colleges, Out of State colleges, and

alternative licensure programs. The brief routinely points to UNC System programs as the most effective source of teachers for North Carolina and the UNC Program Effectiveness report demonstrates the quality of our

EPP completers.

GA Proposed Graduation Targets (Traditional, Alternative, Middle Grades)

08-‐09 09-‐10 10-‐11 11-‐12 12-‐13 13-‐14 14-‐15 15-‐16

GA Projections -‐ Total 215 245 280 320 355 359 374 388 Traditional

GA Projections -‐ Middle Grades

37 40 47 52 60 64 79 93

GA Projections -‐ Total 200 245 295 350 410 422 434 449 Alternative

GA Projections -‐ Middle Grades

14 20 30 42 54 66 78 93

Total CED Completers 415 490 575 670 765 780 808 837

CIP Title

2012-13 Traditional Undergrad Educ

Grads CIP13-Education--

Licensure All Other CIPs

with TOTAL Track Programs Flags

Education, General 17 0 17 Educational Leadership and Administration, General 0 0 0 Educational Leadership and Administration, General 0 0 0 Elementary Education and Teaching 56 0 56 Junior High/Intermediate/Middle School Education and Teaching -- Language Arts (Middle Grades) 0 0 0 Junior High/Intermediate/Middle School Education and Teaching -- Mathematics (Middle Grades) 0 0 0 Junior High/Intermediate/Middle School Education and Teaching -- Science (Middle Grades) 0 0 0 Junior High/Intermediate/Middle School Education and Teaching -- Social Studies (Middle Grades) 0 0 0 Junior High/Intermediate/Middle School Education and Teaching -- Language Arts and Social Studies (M Grades) 22 0 22 Junior High/Intermediate/Middle School Education and Teaching -- Mathematics & Science (M Grades) 0 0 0 Junior High/Intermediate/Middle School Education and Teaching -- Middle Grades (masters level) 0 0 0 Agricultural Teacher Education 19 0 19 Agricultural Teacher Education 0 0 0 Agricultural Teacher Education 0 0 0 English/Language Arts Teacher Education 7 0 7 Technology Teacher Education/Industrial Arts Teacher Education 23 0 23 Technology Teacher Education/Industrial Arts Teacher Education 0 0 0 Sales and Marketing Operations/Marketing and Distribution Teacher Education 6 0 6 Mathematics Teacher Education 44 0 44 Mathematics Teacher Education 0 0 0 Mathematics Teacher Education 0 0 0 Science Teacher Education/General Science Teacher Education 16 0 16 Science Teacher Education/General Science Teacher Education 0 0 0 Social Studies Teacher Education 1 0 1 Social Studies Teacher Education 0 0 0 Social Studies Teacher Education 0 0 0 French Language Teacher Education 3 0 3 Spanish Language Teacher Education 3 0 3 Biology/Biological Sciences, General 0 6 6 Mathematics, General 0 8 8 Statistics, General 0 1 1 TOTAL 217 15 232

CIP Title

2012-13 MAT/MEd Grads. CIP13-Education--Licensure

Track Programs Teacher Education and Professional Development, Specific Levels and Methods, Other -- English

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Mathematics 6

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Reading 1

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Science 11

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Social Sciences 21

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Elementary Education 23

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Middle Grades 11

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Middle Grades-Social Studies 2

Teacher Education and Professional Development, Specific Levels and Methods, Other -- English as a Second Language 4

Teacher Education and Professional Development, Specific Levels and Methods, Other -- Special Education: General Curriculum 4 Teacher Education and Professional Development, Specific Levels and Methods, Other -- Technology Education

1 TOTAL 92

UNC Teacher Quality Research:2013 Teacher Preparation

Program Effectiveness Report- MAY 2013 -

Gary T. Henry, Vanderbilt University

Kristina M. Patterson, UNC–Chapel Hill

Shanyce L. Campbell, UNC–Chapel Hill

Pan Yi, UNC–Chapel Hill

UNC Teacher Quality Research: 2013 Teacher Preparation Program Effectiveness Report

May 2013

Gary T. Henry, Vanderbilt University Kristina M. Patterson, UNC–Chapel Hill Shanyce L. Campbell, UNC–Chapel Hill

Pan Yi, UNC–Chapel Hill

Contents Acknowledgements ………………………….………………………… i Introduction ………………………………………………….………. 1 Data and Methods……..…….………………………………….…….. 3

Results ……………………………………………………………... 8 Conclusion ………………………………………………………….. 17 Calculating Days Equivalency ……………………………………….….. 18 Tables: Table 1: Standard Model Control Variables ……………………………….. 4

Table 2: Institution Abbreviations ……………………………………….. 5

Table 3: UNC Institutional Counts ………………………………….…… 7

Table 4: Summary of Results ………………………………….….….… 15

Table 5: Teacher Counts by UNC System School, Level and Subject …………… 16

Table 6: Key for the Interpretation of Coefficients (Days Equivalency) ……..…… 17 Figures: Figure 1: UNC Programs vs. All Other Sources of Teachers: Elementary School ……. 12 Figure 2: UNC Programs vs. All Other Sources of Teachers: Middle School ……..… 13 Figure 3: UNC Programs vs. All Other Sources of Teachers: High School …….…… 14

i

Acknowledgements

We wish to recognize Alisa Chapman and Keith Brown with the University of North Carolina General Administration for their vital contributions in providing data and working as partners throughout the research and communication processes. We also wish to thank the deans and department heads from the colleges, schools and departments of education at the 15 UNC institutions engaged in teacher education for their valuable input during the development of the models and discussions of the findings. We gratefully acknowledge the many contributions made by our current and former researchers and fellows at the Education Policy Initiative at Carolina (EPIC), including Kevin C. Bastian, C. Kevin Fortner, David C. Kershaw, Jade V. M. Jenkins, Charles L. Thompson, and Rebecca A. Zulli. Finally, we wish to acknowledge the editing and formatting work done by Elizabeth D’Amico, who is responsible for the overall look and polish of the report. All authors accept responsibility for any remaining errors in the report.

Department of Public Policy Education Policy Initiative at Carolina University of North Carolina at Chapel Hill 140 Friday Center Drive, Chapel Hill, NC 27599-9000 919-962-0668 publicpolicy.unc.edu

http://www.publicpolicy.unc.edu/

Introduction This report, produced in collaboration with the UNC General Administration, presents the third1 set of results assessing the effectiveness of undergraduate teacher preparation programs at North Carolina public universities. UNC established this research agenda in 2009 in order to quantify the effects of UNC traditional (undergraduate) preparation programs on student achievement in North Carolina and to provide results to assist UNC Schools of Education in the evaluation and improvement of their programs. The purpose of this analysis is to accurately estimate the average test score gains for public school students taught by recent graduates of each of UNC’s fifteen undergraduate teacher preparation programs, net of other variables that affect student performance. Thus, this report applies the value-added models used in prior studies, with extensive student, classroom, and school covariates, in order to isolate the effects of traditional UNC teacher preparation programs on student achievement in the public schools of North Carolina. Generating quantitative estimates of program effectiveness allows the UNC institutions to see where their program graduates perform well, provides a starting point for inquiry about program characteristics that affect student achievement scores, and identifies programs that need improvement. The information in this report covers programmatic effects across the entire range of grade levels and subjects tested by public schools in North Carolina. Estimates of program effectiveness vary widely across these subject and grade level combinations, and therefore, there is no single measure of overall effectiveness for each program. Two types of models were employed for this analysis: 1) models comparing the graduates of each of the 15 public undergraduate preparation programs to all other teachers in the state, and 2) individual campus models comparing each undergraduate program to 12 other categories of teacher preparation, including alternative entry and teachers prepared out-of-state. These two approaches serve different purposes. The first allows comparisons between each of the undergraduate teacher preparation programs and a common reference group representing the average of all other types of teacher preparation. Essentially, this first analysis shows how effective each of the undergraduate programs is by comparing it to the teaching corps that North Carolina would have if the traditional UNC programs did not exist. This analysis is the focus of this report. The second set of models provides a more detailed comparison between undergraduate prepared teachers from an individual UNC institution and teachers prepared through the other major means by which teachers are prepared to teach in North Carolina. This set of value-added models specifies each UNC institution as the reference group and makes direct comparisons between the reference institution and 12 other categories of teacher preparation, including teachers from other in-state public institutions, in-state private institutions,

1 See Henry, G.T., Thompson, C.L., Fortner, C.K., Zulli, R.A., and Kershaw, D.C (2010). The Impact of Teacher

Preparation on Student Learning in North Carolina Public Schools. Chapel Hill, NC: The Carolina Institute for Public Policy. Available online at: http://publicpolicy.unc.edu/research/Teacher_Prep_Program_Impact_Final_Report_nc.pdf and Henry, G.T., Thompson, C.L., Bastian, K.C., Fortner, C.K., Kershaw, D.C., Marcus, J.V., and Zulli R.A. (2011) UNC Teacher Preparation Program Effectiveness Report. Chapel Hill, NC: The Carolina Institute for Public Policy. Available online at: http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

http://publicpolicy.unc.edu/research/Teacher_Prep_Program_Impact_Final_Report_nc.pdf

http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

2

out-of-state universities, Teach For America, and other alternative entry teachers. The 12 other categories of teachers are defined in the 2011 UNC Teacher Preparation Program Effectiveness Technical Report.2 We provide results from this second set of models to the deans of each institution in a separate institution-specific report. These estimates of the effects of traditional UNC teacher preparation programs reflect both the knowledge and skills that teachers develop while they are prospective teachers at a particular university and the capacity or learned ability that the prospective teachers bring with them into a particular preparation program. While it may be useful to separate these effects to determine which programs add more value during the preparation process, the education that a teacher provides to a student in North Carolina public schools is a product of both preparation and academic ability. Therefore, these estimates are intended to capture the total effect of teacher preparation and selection in a single estimate of value-added model outcomes. The effectiveness of teacher preparation programs should be judged, at least in part, on the extent to which teachers prepared by these programs are able to produce gains in students’ knowledge and skills. To assess these gains we use the standards-based state assessments as the measures of student learning. In this report, we estimate the effects of teachers on student performance on the available End-of-Grade (EOG) and End-of-Course (EOC) tests administered by the North Carolina Department of Public Instruction (NC DPI). These test scores have strengths and weaknesses as a method for estimating the effectiveness of teachers. The student assessment program in North Carolina serves as a model for other states and the state’s test scores are widely used in rigorous, high quality research and evaluation studies. North Carolina’s state tests are aligned with the NC Standard Course of Study within each grade and/or subject, and tests in reading and mathematics in grades 3 to 8 are vertically scaled to allow for comparisons across years and grades. Student EOG and EOC test scores serve as the basis for the ABCs accountability program in North Carolina as well as for calculating the value-added by teachers and schools to student test score growth which will be used in the state’s new system for evaluating teachers and principals. Because EOG and EOC tests are closely aligned with the intended curriculum, the test score gains produced by program graduates are appropriate for estimating the effectiveness of teachers who graduate from UNC teacher preparation programs. The weaknesses of using student test score outcomes to measure teaching effectiveness include the limited subjects and grades in which testing is conducted—students below grade three are not tested; students in 3rd through 8th grade are tested only in reading and mathematics, annually, and in science in only the 5th and 8th grade; and the number of EOC tests in high school has decreased in recent years. This prevents any estimates of teacher effectiveness based on test scores in other subject or grade level combinations and means that program effectiveness estimates are based on a subset of program graduates and not on all teachers prepared by the preparation program (see Table 3). Other important outcomes, such as graduation, attitudes toward school and learning, or capacity to function as a citizen within a democracy, are not captured by these standardized tests. The tests measure the extent to which individual teachers succeed in teaching their students the state-adopted curriculum in specific grades and subjects. So while limited, they do provide useful outcome-based information for assessing the effectiveness of teacher preparation programs. 2 Henry, G.T., Thompson, C.L., Fortner, C.K., Bastian, K.B. and., Marcus, J.V. (2011) Technical Report: UNC

Teacher Preparation Program Effectiveness Report. Available online at: http://publicpolicy.unc.edu/research/ProgramEffect_TechRpt_July2011.pdf

http://publicpolicy.unc.edu/research/ProgramEffect_TechRpt_July2011.pdf

3

This report provides an overview of the data and methodology of the current analysis and the changes between this evaluation of teaching program effectiveness and the prior report.3 Subsequent sections include the results of the analysis—comparing each undergraduate program to the aggregate of all other sources of teacher preparation—and a conclusion section. We provide separate campus reports (not included here) to each institution with results from the individual campus models.

Data and Methods This 2013 edition of the UNC Teacher Preparation Program Effectiveness Report continues to utilize a value-added modeling approach that compares the average student learning gains for students whose teachers have different types of preparation prior to entering the classroom in North Carolina public schools. These value-added models are estimated within a multi-level model specification with extensive controls at the student, classroom, and school level. Limited controls for teachers (years of experience and out-of-field teaching) are included in the models in order to generate comparable estimates of the total effect of teacher preparation programs on student achievement. For example, although we expect teachers with higher Praxis scores and National Board Certification to generate greater learning gains for their students, on average, we exclude these variables from the analysis since teachers from certain institutions may be more likely to have some of these characteristics, and including them would downwardly bias program effect estimates. Table 1 includes a complete list of control variables included at each of the three levels in the multi-level model. The definitions for three of the variables may not be obvious: structural mobility refers to students who changed schools due to the grade configuration of a school (i.e. 6th grade students in a 6 - 8 middle school); between-year mobility refers to students who either attended or completed assessments at a different school in the prior academic year; and within-year mobility refers to students who were enrolled in the school where they took their tests for substantially less than the full school year (more than two weeks less than a full year).

3 Henry, G.T., Thompson, C.L., Bastian, K.C., Fortner, C.K., Kershaw, D.C., Marcus, J.V., and Zulli R.A. (2011) UNC Teacher Preparation Program Effectiveness Report. Chapel Hill, NC: The Carolina Institute for Public Policy. Available online at: http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

http://publicpolicy.unc.edu/research/TeacherPrepEffectRpt_Final.pdf

4

Table 1: Standard Model Control Variables Student Classroom & Teacher School

1. Prior test scores (reading & math)

2. Classmates prior test scores (peer effects)

3. Days absent 4. Structural mobility 5. Between-year mobility 6. Within-year mobility 7. Gender 8. Race/ethnicity 9. Poverty 10. Gifted 11. Disabled 12. Currently limited English

proficient 13. Previously limited

English proficient 14. Overage for grade (held

back or retained at least once)

15. Underage for grade (promoted two grades)

16. Years of experience 17. Teaching out-of-field 18. Number of students 19. Advanced curriculum 20. Remedial curriculum 21. Dispersion of prior

achievement within classroom

22. School size (ADM) 23. School size squared 24. Suspension rate 25. Violent acts rate 26. Total per pupil expenditures 27. District teacher supplements 28. Racial/ethnic composition 29. Concentration of poverty

Table 2 (below) lists the 15 campuses included in the program comparison results contained in this report and the abbreviations for each campus used throughout the report. Separate reports present the results of each of these campuses compared to 12 other categories of teacher preparation. To identify graduates of traditional UNC preparation programs for this analysis, we rely on data from the UNC General Administration, which includes information on the undergraduate major(s) and campus enrollment for each individual who attended a North Carolina four-year public institution. Individuals are classified as program graduates if they graduated from a UNC institution with an education major or graduated with another major and simultaneously received a teaching license. These individuals are considered traditionally prepared teachers in this report regardless of any additional preparation (such as a Master’s degree from another institution) obtained between graduating from the traditional teacher education program and starting work as a classroom teacher. This definition allows the UNC teacher preparation program comparisons to include all of the traditional undergraduate prepared teachers that graduated from each institution, but creates slight differences between the teacher categories specified in this report and the Education Policy Initiative at Carolina’s prior publication on portal effectiveness.4

4 Portal Report: Teacher Preparation and Student Test Scores in North Carolina. (2010). Chapel Hill, NC: The

Education Policy Initiative at Carolina (EPIC) formally known as the Carolina Institute for Public Policy. Available online at: http://publicpolicy.unc.edu/research/Teacher_Portals_Teacher_Preparation_and_Student_Test_Scores_in_North_Carolina_2.pdf

http://publicpolicy.unc.edu/research/Teacher_Portals_Teacher_Preparation_and_Student_Test_Scores_in_North_Carolina_2.pdf

http://publicpolicy.unc.edu/research/Teacher_Portals_Teacher_Preparation_and_Student_Test_Scores_in_North_Carolina_2.pdf

5

Table 2: UNC Institution Abbreviations Teacher Preparation Program Abbreviation

Appalachian State University ASU East Carolina University ECU

Elizabeth City State University ECSU Fayetteville State University FSU

North Carolina Agricultural & Technical State University NCA&T5 North Carolina Central University NCCU

North Carolina State University NCSU University of North Carolina – Asheville UNCA

University of North Carolina – Chapel Hill UNCCH University of North Carolina – Charlotte UNCC

University of North Carolina – Greensboro UNCG University of North Carolina – Pembroke UNCP

University of North Carolina – Wilmington UNCW Western Carolina University WCU

Winston-Salem State University WSSU The data used for this analysis includes student, classroom, and school characteristics from public schools in North Carolina from the 2006-07 through the 2010-11 school years. This five-year span updates the data used (2005-06 through 2009-10) in our previous analysis. In addition, we restrict the models to teachers with less than five years of teaching experience. This allows us to assess the effectiveness of relatively recent graduates from each program. The cutoff of less than five years of experience balances the need for a sample of program graduates that is large enough to calculate effects and recognizes that the effects of a teacher’s training program will diminish over time as a teacher learns from classroom experience, principal and peer feedback, and other professional development. This report includes results from 11 distinct models for student test score outcomes: elementary school mathematics, reading (grades 4 and 5), and science (grade 5 only); middle school mathematics, reading (grades 6-8), science (grade 8 only), and algebra I; and high school (grades 9 – 12) algebra I & II, English I, science (biology and physical science), and social studies (US history and civics & economics). The previous report did not include analysis of elementary school science. High school models are slightly different from the prior analysis due to the elimination of a number of EOC tests (North Carolina discontinued the geometry, chemistry, and physics tests during our study period). Finally, we eliminated the high school overall model due to the changes in tests previously included in that analysis. We present results in a graphical format that resembles a thermometer and displays institutions or teacher preparation categories vertically, corresponding to their estimates of effectiveness. These graphs provide an accessible visual presentation of the data, but introduce the need for some explanatory notes. First, only comparisons with categories that contain at least ten teachers

5 Abbreviated as NCAT in Figures 1, 2, and 3

6

from a given teacher preparation program are reported. This count is based on unique teachers across years and may include observations of student outcomes over numerous years that are linked to a given teacher. Second, results which are statistically significant at the 0.05 level are marked with asterisks and bolded to indicate they are significantly different from the reference category, teachers from all sources other than undergraduate preparation programs at UNC institutions. We should also note that these estimates are reported as a percentage of a standard deviation unit, meaning for example, that a value of 25 indicates that the students taught by a teacher from a specific program are expected to generate test score outcomes 25 percent of a standard deviation higher than similar students in similar schools and classrooms taught by teachers not prepared by undergraduate preparation programs at UNC institutions. Table 3 presents the total number of teachers from each institution who were teaching in North Carolina public schools during the five year study period, the number of teachers with less than 5 years of experience teaching in NC public schools over the time period, and the number of teachers with less than 5 years of experience who were teaching in tested grades and subjects, and therefore, could be included in the models. In cases of missing or unmatched data, teachers are excluded from models. (See Table 5 for exact counts of teachers included in each model).

7

Table 3: UNC Institutional Counts

Teacher Preparation

Program

Total Teachers from

2006-07 to 2010-111

Total Teachers with Less than 5 Years’ Experience

from 2006-07 to 2010-112

Total Teachers in Tested Grades and Subjects with Less than 5

Years’ Experience from 2006-07 to 2010-113

ASU 6850 3074 1530 ECU 6537 2828 1263

ECSU 716 238 101 FSU 1253 555 282

NCA&T 698 282 128 NCCU 862 356 167 NCSU 1852 1030 614 UNCA 381 211 135

UNCCH 1924 629 281 UNCC 3342 1783 940 UNCG 3956 2019 839 UNCP 1784 681 316

UNCW 3204 1630 808 WCU 2425 1053 514

WSSU 614 197 77 1Represents the total number of unique, traditionally prepared teachers paid as certified instructors in North

Carolina public schools at least one time between 2006-07 and 2010-11. Counts in this column are for all teachers,

regardless of experience or subject/grade taught. 2 Represents the total number of unique, traditionally prepared teachers with less than five years of experience paid

as certified instructors in North Carolina public schools at least one time between 2006-07 and 2010-11. Counts in

this column are for all teachers with less than five years of experience, regardless of subject/grade taught. 3Represents the total number of unique, traditionally prepared teachers with less than five years of experience paid

as certified instructors in North Carolina public schools at least one time between 2006-07 and 2010-11 who taught

in tested subjects and grade levels during that time period.

8

Results

Graduates of traditional teacher preparation programs at each UNC institution were compared to all other sources of teachers on 11 different standardized tests (EOG or EOC), with three outcomes in elementary school, four in middle school, and four in high school. Results are not reported in cases where a program had fewer than ten graduates teaching the grade and subject associated with the EOG or EOC test. For elementary and middle school mathematics and reading we report results in terms of equivalent days of instruction gained (or lost) by comparable students whose teacher graduated from a particular program compared to all other sources of teachers. We cannot report days equivalency in elementary school science, middle school science, middle school algebra I, nor any high school EOC tests because the tests are not vertically equated to a prior test score in the same subject.

The Effects of UNC Teacher Preparation Programs on Student Achievement

Appalachian State University (ASU): On average, graduates of Appalachian State University’s undergraduate teacher preparation program were more effective than all other sources of teachers in one comparison – elementary school mathematics. It may be useful to think of this impact in terms of days of learning. A student with a teacher prepared at ASU could gain an average of 6 days of learning per year more than a student instructed by a teacher in the reference category.6 ASU graduates performed similarly to all other sources of teachers in the remaining ten comparisons – elementary school reading, elementary school science, middle school mathematics, middle school reading, middle school science, middle school algebra I, high school algebra I and algebra II, high school science, high school English I, and high school social studies.

East Carolina University (ECU): On average, graduates of East Carolina University’s undergraduate teacher preparation program were more effective than all other sources of teachers in three comparisons – elementary school reading, middle school algebra I, and high school social studies. In elementary school reading, a student with a teacher prepared at ECU could gain an average of 5 ½ days of learning per year more than a student instructed by a teacher in the reference category. ECU graduates performed similarly to all other sources of teachers in the remaining eight comparisons – elementary school mathematics, elementary school science, middle school mathematics, middle school reading, middle school science, high school algebra I & II, high school English I, and high school science.

Elizabeth City State University (ECSU): On average, graduates of Elizabeth City State University’s undergraduate teacher preparation program performed similarly to all other sources of teachers in five comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, and middle school reading. There were insufficient (<10) ECSU graduates teaching middle school science, middle school algebra I, high school algebra I and algebra II, high school science, high school English I, and high school social studies to report results from these comparisons.

6 See Appendix for instructions on calculating days equivalency.

9

Fayetteville State University (FSU): On average, graduates of Fayetteville State University’s undergraduate teacher preparation program were more effective than all other sources of teachers in one comparison – high school algebra I & II. FSU graduates performed similarly to all other sources of teachers in seven comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, middle school reading, high school English I, and high school social studies. There were insufficient FSU graduates teaching middle school science, middle school algebra I, and high school science to report results from these comparisons.

North Carolina A&T State University (NCA&T): On average, graduates of North Carolina A&T State University’s undergraduate teacher preparation program performed similarly to all other sources of teachers in five comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, and high school algebra I & II. On average, NCA&T graduates were less effective than all other sources of teachers in one comparison – middle school reading. There were insufficient NCA&T graduates teaching middle school science, middle school algebra I, high school English I, high school science, and high school social studies to report results from these comparisons.

North Carolina Central University (NCCU): On average, graduates of North Carolina Central University’s undergraduate teacher preparation program were more effective than teachers from all other sources in one comparison – elementary school science. NCCU graduates performed similarly to all other sources of teachers in four comparisons – elementary school mathematics, elementary school reading, middle school mathematics, and middle school reading. There were insufficient NCCU graduates teaching middle school science, middle school algebra I, high school science, and high school social studies to report results from these comparisons.

North Carolina State University (NCSU): On average, graduates of North Carolina State University’s undergraduate teacher preparation program were more effective than all other sources of teachers in one comparison – high school social studies. NCSU graduates performed similarly to all other sources of teachers in the remaining ten comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, middle school reading, middle school science, middle school algebra I, high school algebra I and algebra II, high school English I, and high school science.

University of North Carolina – Asheville (UNCA): On average, graduates of University of North Carolina at Asheville’s undergraduate teacher preparation program performed similarly to all other sources of teachers in seven comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, middle school reading, high school English I, and high school science. On average, UNCA graduates were less effective than all other sources of teachers in one comparison – high school algebra I & II. There were insufficient UNCA graduates teaching middle school science, middle school algebra I, and high school social studies to report results from these comparisons.

University of North Carolina – Chapel Hill (UNCCH): On average, graduates of University of North Carolina at Chapel Hill’s undergraduate teacher preparation program were more effective than all other sources of teachers in one comparison – middle school mathematics. In terms of days of learning, a student with a teacher prepared at UNCCH could gain an average of 44 days of learning per year more than a student instructed by a teacher in the

10

reference category. UNCCH graduates performed similarly to all other sources of teachers in five comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school reading, and middle school science. There were insufficient UNCCH graduates teaching middle school algebra I, high school algebra I and algebra II, high school science, high school English I, and high school social studies to report results from these comparisons.

University of North Carolina – Charlotte (UNCC): On average, graduates of University of North Carolina at Charlotte’s undergraduate teacher preparation program were more effective than all other sources of teachers in two comparisons – high school science and high school social studies. UNCC graduates performed similarly to all other sources of teachers in eight comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, middle school reading, middle school science, high school algebra I & II, and high school English I. On average, UNCC traditionally prepared teacher were less effective than all other sources of teachers in one comparison – middle school algebra I.

University of North Carolina – Greensboro (UNCG): On average, graduates of University of North Carolina at Greensboro’s undergraduate teacher preparation program were more effective than all other sources of teachers in one comparison – elementary school mathematics. A student with a teacher prepared at UNCG could gain an average of 7 ½ days of learning per year more than a student instructed by a teacher in the reference category. UNCG graduates performed similarly to all other sources of teachers in nine comparisons – elementary school reading, elementary school science, middle school mathematics, middle school reading, middle school algebra I, high school algebra I and algebra II, high school science, high school English I, and high school social studies. There were insufficient UNCG graduates teaching middle school science to report results from this comparison.

University of North Carolina – Pembroke (UNCP): On average, graduates of University of North Carolina at Pembroke’s undergraduate teacher preparation program performed similarly to all other sources of teachers in eight comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, middle school reading, high school algebra I & II, high school science, and high school social studies. There were insufficient UNCP graduates teaching middle school science, middle school algebra I, and high school English I to report results from these comparisons.

University of North Carolina - Wilmington (UNCW): On average, graduates of University of North Carolina Wilmington’s undergraduate teacher preparation program were more effective than all other sources of teachers in two comparisons – elementary school science and middle school mathematics. A middle school mathematics student with a teacher prepared at UNCW could gain an average of 32 ½ days of learning per year as compared to a student instructed by a teacher in the reference category. UNCW graduates performed similarly to all other sources of teachers in the remaining nine comparisons – elementary school mathematics, elementary school reading, middle school reading, middle school science, middle school algebra I, high school algebra I and algebra II, high school science, high school English I, and high school social studies.

11

Western Carolina University (WCU): On average, graduates of Western Carolina University’s undergraduate teacher preparation program were more effective than all other sources of teachers in one comparison – high school English I. WCU graduates performed similarly to all other sources of teachers in the remaining ten comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, middle school reading, middle school science, middle school algebra I, high school algebra I & II, high school science, and high school social studies.

Winston-Salem State University (WSSU): On average, graduates of Winston-Salem State University’s undergraduate teacher preparation program performed similarly to all other sources of teachers in five comparisons – elementary school mathematics, elementary school reading, elementary school science, middle school mathematics, and middle school reading. There were insufficient WSSU graduates teaching middle school science, middle school algebra I, high school algebra I and algebra II, high school science, high school English I, and high school social studies. The figures below present the results of models that compare the average effectiveness of teachers from each of UNC’s fifteen traditional undergraduate teacher preparation programs to the average effectiveness of teachers from all other sources of preparation. Where estimates are statistically different from all other sources of teachers (significant at the 0.05 level), institution names appear in bold text, are noted with a *, and have points designated with diamonds. Institutions with insufficient data to report results (fewer than ten teachers in the model) are excluded from graphs. The scaling of graphs may differ to maximize the readability of each graph. The results are summarized in Table 4.

12

Figure 1: UNC Traditional Preparation Programs vs. All Other Sources of Teachers – Elementary School

13

Figure 2: UNC Traditional Preparation Programs vs. All Other Sources of Teachers – Middle School

14

Figure 3: UNC Traditional Preparation Programs vs. All Other Sources of Teachers – High School

15

Table 4: Summary of Results

The results summarized in Table 4 (above) are based on the number of statistically significant positive or negative comparisons for each institution across the 11 models. Table 5 (below) displays the number of teachers, by institution, included in each model. Higher numbers of teachers in a particular group decrease the standard errors and make statistically significant findings more likely. Schools with small sample sizes are unlikely to show statistically significant results, even when the estimated effect is large in absolute value terms.

Institutions with Two or More

Positive Results

Outperforming the Reference Group Comparisons

Underperforming the Reference Group Comparisons

ECU ES Reading; MS Algebra I; HS Social Studies ----

UNCC HS Science; HS Social Studies MS Algebra I UNCW ES Science; MS Mathematics ----

Institutions with One Positive Result



ASU ES Mathematics ---- FSU HS Algebra I & II ---- NCCU ES Science ---- NCSU HS Social Studies ---- UNCCH MS Mathematics ---- UNCG ES Mathematics ---- WCU HS English I ----

Institutions with No Statistically Significant

Results



ECSU ---- ---- UNCP ---- ---- WSSU ---- ----

Institutions with One Negative Result and

No Positive Results



NCA&T ---- MS Reading UNCA ---- HS Algebra I & II

16

Table 5: Teacher Counts by UNC System School, Level and Subject

Teacher Preparation

Program

ES Math

ES Read

ES Science

MS Math

MS Read

MS Science

MS Algebra

I

HS Algebra I & II

HS English

HS Science

HS Social

Studies

ASU 666 666 183 177 180 30 33 116 112 35 147

ECU 620 617 192 139 140 30 23 81 70 40 79

ECSU 59 59 20 16 10 2 1 1 5 2 3

FSU 102 102 32 43 48 5 4 39 18 4 18

NCA&T 58 58 18 17 11 2 6 16 8 3 3

NCCU 88 88 27 23 17 2 8 12 11 3 8

NCSU 31 31 13 86 122 32 34 146 58 43 65

UNCA 46 46 15 10 17 2 4 14 19 13 7

UNCCH 164 164 43 40 41 12 9 8 5 8 1

UNCC 497 497 138 113 91 28 21 60 34 12 50

UNCG 406 405 114 77 105 8 13 39 69 14 42

UNCP 143 143 42 24 34 7 1 29 9 13 25

UNCW 416 416 113 96 81 18 21 59 58 12 29

WCU 215 215 61 68 70 10 11 38 30 18 37

WSSU 39 39 10 10 12 1 1 3 1 0 3 *Highlighted cells have fewer than ten teachers and therefore do not have any results reported.

17

Results in elementary and middle school mathematics and reading models may be interpreted in terms of the equivalent days of instruction gained (or lost) by comparable students whose teacher graduated from a particular program compared to all other sources of teachers. Table 6 contains values for interpretation of effectiveness estimates (coefficients) depending on the model under consideration. For example, comparable students in similar classrooms and schools are expected to score as if they had attended 14 and one-third extra days of school when they are taught by a teacher whose effectiveness estimate (coefficient) is five percent of a standard deviation higher than the reference group in elementary school mathematics. These estimates vary based on the subject and grade level, and the formula for calculating values based on different results is found in the appendix of this report. Table 6: Key for the Interpretation of Coefficients (Days Equivalency)

Result Values (Coefficient/Effectiveness) ES Math ES

Reading MS Math MS Reading

15.00 43.13 days 49.16 days 97.71 days 79.35 days 10.00 28.76 days 32.78 days 65.14 days 52.90 days 5.00 14.38 days 16.39 days 32.57 days 26.45 days 2.00 5.75 days 6.56 days 13.03 days 10.58 days

Note: These result values show days equivalency in relation to the reference group; negative result values

have negative days equivalency results. See the appendix section for directions on calculating the days

equivalency.

Conclusion Of the fifteen traditional teacher education programs at UNC institutions, most are performing about as well, or better than, all other sources of teachers in terms of their estimated impact on student EOG and EOC achievement tests. Only two of the campuses had results with more negative model outcomes than positive, ten campuses demonstrated more positive model outcomes than negative results, and three campuses had no statistically significant results. Overall, on average, UNC traditionally prepared teachers are likely to outperform teachers from all other sources combined, and some programs perform substantially better than others in specific subjects or grade levels. We present the estimates comparing each campus to each of the twelve “portals” or categories of teacher preparation in separate campus-specific reports.

18

Appendix: Calculating Days Equivalency Table A.1. Necessary Information for Days Equivalency Calculations

End of Grade Test Standard Deviation Average Yearly Gains

Elementary School Mathematics 9.373 5.867 Elementary School Reading 9.614 5.280 Middle School Mathematics 9.174 2.535 Middle School Reading 8.931 3.039

Days Equivalency Equation: (((Result value/100) x Standard Deviation)/ (Avg. Yearly Gain))) x 180 Example for Elementary School Mathematics:

Step One

Result value from institutional comparison graph = 2.10 Standard Deviation (9.373) and Average Yearly Gains (5.867) from the table above

Step Two

Insert the result value into the days equivalency equation

(((2.10/100) x 9.373)/(5.867))) x 180 = 6.04 days of learning

Days Equivalency Note: Days equivalency values can be calculated for elementary and middle grades mathematics and reading tests because the tests are interval scaled and students have prior test scores for the subject. We cannot calculate days equivalency in elementary grades science, middle grades science and algebra, and in all high school tests (EOC) because these assessments are not vertically equated to a prior test score in the same subject.

publicpolicy.unc.edu

UNC Teacher Quality Research: 2013 Teacher Preparation Program Effectiveness Report

North Carolina State University

Carolina Institute for Public Policy

February 2013

Contents

Model Information .................................................................................................................. 1

Summary of Results ................................................................................................................. 2

Table 1: Teacher Preparation Category Abbreviations ………………………………………..………..3

Thermometer Graphs………………………………………………………………………………………..……………....4

Table 2: Teacher Counts by Preparation Category, Grade Level and Subject…..…......…….7

Distribution of Program Graduates by Effectiveness .............................................................. 8

Top Hiring Districts for Program Graduates …................................................................…....14

Appendix

NCSU Traditionally Prepared Teachers vs. All Other Teacher Categories with Less than Five Years Experience, Teaching in the Same School (2006-07 through 2010-11).………………..….16 Appendix Tables: Table A.1. Elementary School: NCSU Traditionally Prepared Teachers vs. All Other Teachers in the Same School with Less than Five Years Experience ……..….17 Table A.2. Middle School: NCSU Traditionally Prepared Teachers vs. All Other Teachers in the Same School with Less than Five Years Experience………….18

Table A.3. High School: NCSU Traditionally Prepared Teachers vs. All Other

Teachers in the Same School with Less than Five Years Experience …..…....19

1

2013 Teacher Preparation Program Effectiveness Report

North Carolina State University The University of North Carolina (UNC) established an initial agenda of teacher quality research in

summer 2009 to address the entry, persistence, and quality of teacher and school leader preparation in

North Carolina. Since that time, the research has become a very important component of the

University’s efforts to prepare more, higher quality teachers and school leaders for North Carolina public

schools. The results of the research are enabling campus leaders to identify areas of strength, potential

areas of weakness, and the areas where their graduates’ performance is comparable to the performance

of teachers from a variety of other sources. This important work should help educators and policy

makers understand where their policies and investments are paying off as well as where they are not

achieving the intended purposes.

For this report, the Carolina Institute for Public Policy (CIPP) compared the performance of traditionally

prepared undergraduates at North Carolina State University to the performance of all other teachers in

the state based on the education and training that qualified them to enter teaching. Detailed in the

remainder of the report are key features of the models, a summary of the findings, graphs of the results,

and a table of teacher counts for each model.

Model Information Data for this report spanned the 2006-07 through 2010-11 school years and was limited to teachers with

less than five years of experience in order to best capture the effects of teacher preparation on student

achievement. For this analysis, the performance of traditionally prepared undergraduates at North

Carolina State University is compared to the performance of all other teachers in North Carolina

according to the training that qualified them to enter the teaching profession. Models were generated

across eleven grade-subject areas—elementary mathematics, reading, and science (5th grade); middle

grades mathematics, reading, science (8th grade), and Algebra I; and high school Algebra I & II, English I,

science (Biology and Physical Science) and social studies (U.S. History and Civics & Economics) —with

control variables for prior student achievement and demographic information, classroom and teacher

indicators, including years of teaching experience and out-of-field teaching, and school level contextual

factors.

Supplementary Models that limited comparisons to the performance of teachers in the same school

were included to control for aspects of school context that could not be included in the models.

Additionally, individual teacher by year value-added estimates were computed to provide information

on program graduates by quintiles of effectiveness. Finally, teacher preparation categories needed at

least ten teachers in the grade-subject area for results to be reported.

2

Summary of Results The following key points summarize the results for traditionally prepared undergraduates from North

Carolina State University (NCSU).

NCSU traditionally prepared undergraduates outperformed teachers from the following preparation

categories:

Other UNC Program Traditionally Prepared in High School Social Studies

UNC Graduate Degree Prepared in Middle School Science and High School Social Studies

NC Private University Undergraduate Degree in High School Science and High School Social

Studies

NC Private University Graduate Degree in Elementary School Mathematics and High School

Social Studies

Out of State University Undergraduate Degree in Elementary School Mathematics, High School

Science and High School Social Studies

Out of State University Graduate Degree in High School Social Studies

Other Licensure Only in Elementary School Mathematics

Visiting International Faculty in High School Algebra I & II

Alternative Entry in High School Social Studies

Unclassifiable in Middle School Science and High School Social Studies

NCSU traditionally prepared undergraduates underperformed:

Teach For America in Middle School Mathematics, Middle School Reading, Middle School

Science, Middle School Algebra I, High School Algebra I & II, and High School English

Visiting International Faculty in Middle School Reading

Graphs

The following pages present graphical results at the elementary, middle, and high school levels. For

interpretation, four points are important:

1) North Carolina State University is set to zero for all graphs and each plot represents the relative

difference between the indicated category and NCSU.

2) All model results have been multiplied by 100 to express findings as percentages of standard

deviations.

3) Significant results are expressed with black labels, diamond symbols and asterisks; non-

significant results are grayed out with square symbols

4) Scaling for graphs differs depending on the magnitude of results.

3

Table 1: Teacher Preparation Category Abbreviations

Teacher Preparation Category Abbreviation

Other UNC Traditionally Prepared Undergraduates OtherUNC

UNC Graduate Degree Prepared UNCGD

NC Private University Undergraduate Degree Prepared NCPUGD

NC Private University Graduate Degree Prepared NCPGD

Out of State Undergraduate Degree Prepared OSUGD

Out of State Graduate Degree Prepared OSGD

UNC Licensure Only UNCLO

Other Licensure Only OtherLO

Teach For America TFA

Visiting International Faculty VIF

Alternative Entry AE

Unclassifiable Unclass

4

Elementary School: NCSU Traditionally Prepared Teachers vs. All Other Teacher Categories with Less than 5 Years Experience (2006-07 through 2010-11)

NCSU

NCPGD* -11.74 OtherLO* -11.61

OSUGD* -9.00Unclass -8.39

OSGD -8.20AE -8.02

NCPUGD -7.54OtherUNC -7.20

UNCGD -5.52 UNCLO -5.42

VIF -3.60

TFA -1.30

-12.0

0-1

0.0

0-8

.00

-6.0

0-4

.00

-2.0

00.0

02.0

04.0

06.0

08.0

010

.00

12

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

Elementary School Math

NCSU Comparison

NCSU

OtherLO -7.10

NCPGD -6.38

OSUGD -2.76Unclass -2.40

OSGD -2.33NCPUGD -2.17

OtherUNC -1.89 UNCGD -1.84

AE -1.24

UNCLO -0.66

TFA 0.20

VIF 1.52

-8.0

0-6

.00

-4.0

0-2

.00

0.0

02.0

04.0

06.0

08.0

0

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

Elementary School Reading

NCSU Comparison

NCSU

NCPGD -12.73

UNCGD -11.15

OSUGD -9.07AE -9.05

NCPUGD -8.57

OSGD -7.30

OtherUNC -5.51

Unclass -5.48

UNCLO 0.11

VIF 0.65

TFA 2.45

-16.0

0-1

2.0

0-8

.00

-4.0

00.0

04.0

08.0

012

.00

16

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

Elementary School Science

NCSU Comparison

5

Middle School: NCSU Traditionally Prepared Teachers vs. All Other Teacher Categories with Less than 5 Years Experience (2006-07 through 2010-11)

NCSU

TFA* 12.60

NCPUGD -0.90

Unclass -0.20

UNCGD 0.50UNCLO 0.60AE 0.90

OSGD 1.40

OSUGD 1.80OtherUNC 2.00

VIF 2.80

OtherLO 3.30

-10.0

0-5

.00

0.0

05.0

010

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

Middle School Math

NCSU Comparison

NCSU

TFA* 3.30

VIF* 3.70

NCPGD -1.60

UNCLO 0.00UNCGD 0.30

OSUGD 0.70OtherUNC 0.70OSGD 0.80AE 1.00NCPUGD 1.10

Unclass 1.60

OtherLO 3.30

-10.0

0-5

.00

0.0

05.0

010

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

Middle School Reading

NCSU Comparison

NCSU

Unclass * -16.90

UNCGD* -15.80

TFA* 18.50

NCPUGD -8.50

AE -4.90

OSGD -3.90

OSUGD -2.80

OtherUNC -1.50

VIF 2.60

-10.0

0-5

.00

0.0

05.0

010

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

Middle School Science

NCSU Comparison

NCSU

TFA* 35.00

OSGD -10.40

NCPUGD -7.40AE -6.90

VIF -6.60

OtherUNC -3.40

OSUGD -0.50

-10.0

0-5

.00

0.0

05.0

010

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

Middle School Algebra

NCSU Comparison

6

High School: NCSU Traditionally Prepared Teachers vs. All Other Teacher Categories with Less than 5 Years Experience (2006-07 through 2010-11)

NCSU

VIF* -11.30

TFA* 14.90

Unclass -8.90

OSGD -5.20

AE -3.40OSUGD -3.00

NCPUGD -2.50

OtherUNC -1.20 UNCLO -1.20NCPGD -1.00

UNCGD 4.50

-10.0

0-5

.00

0.0

05.0

010

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

High School Algebra I & II

NCSU Comparison

NCSU

TFA* 5.00

AE -1.40 OSUGD -1.40

OtherUNC -0.80

Unclass -0.60

UNCLO 0.50

NCPUGD 0.70UNCGD 0.80

VIF 1.00

OSGD 1.90

NCPGD 3.00

-5.0

0-2

.50

0.0

02.5

05.0

0

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

High School English

NCSU Comparison

NCSU

NCPUGD* -13.50

OSUGD* -10.00

OSGD -8.50

OtherUNC -5.90

AE -5.20

VIF -4.20

UNCLO -0.80UNCGD -0.50

Unclass 1.90

NCPGD 10.70

TFA 11.50

-10.0

0-5

.00

0.0

05.0

010

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

High School Science

NCSU Comparison

NCSU

OSUGD* -8.80

AE* -8.60NCPGD* -8.40OSGD* -8.30

Unclass* -7.20NCPUGD* -7.00

UNCGD* -6.00 OtherUNC* -5.90

UNCLO -4.70

TFA 7.40

-10.0

0-5

.00

0.0

05.0

010

.00

Sta

nd

ard

De

via

tio

n D

iffe

rences in P

erc

en

t

High School Social Studies

NCSU Comparison

7

Table 2: Teacher Counts by Preparation Category, Grade Level, and Subject

Teacher Preparation Category

ES Math

ES Read

ES Science

MS Math

MS Read

MS Science

MS Algebra

I

HS Algebra

I &II HS

English I HS

Science

HS Social

Studies

NCSU 31 31 13 86 122 32 34 146 58 43 65

Other UNC 3519 3517 1008 853 853 158 155 515 449 177 452

UNC Graduate Degree Prepared 134 134 37 24 68 10 5 53 105 52 108

NC Private Undergraduate Degree Prepared 1454 1454 377 223 219 28 20 176 119 44 145

NC Private Graduate Degree Prepared 39 39 16 8 15 2 3 27 29 16 31

Out of State Undergraduate Degree Prepared 3133 3131 797 772 826 212 136 371 287 188 323

Out of State Graduate Degree Prepared 687 687 175 157 217 52 36 105 98 101 105

UNC Licensure Only 122 122 34 36 49 6 5 12 34 17 47

Other Licensure Only 33 33 3 5 13 1 1 5 4 7 2

Teach For America 123 123 36 102 116 38 19 76 64 58 47

Visiting International Faculty 215 215 53 96 74 15 14 39 36 26 1

Alternative Entry 762 759 175 1067 1295 406 184 731 607 727 427

Unclassifiable 187 187 45 42 58 12 7 23 33 24 59 Note: Cells highlighted in grey have less than ten teachers and therefore do not have any results reported.

8

Distribution of Program Graduates by Effectiveness1

0%

5%

10%

15%

20%

25%

30%

Bottom Quintile Middle Quintile Top Quintile

Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

sNCSU - Elementary School Math

(N=38)

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - Elementary School Reading(N=42)

1 Expected value of each quintile is 20%

9

0%

5%

10%

15%

20%

25%

30%

35%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - Elementary School Science(N=19)

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - Middle School Math (N=176)

10

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - Middle School Reading (N=253)

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - Middle School Science (N=55)

11

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - Middle School Algebra I(N=57)

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - High School Algebra I & II (N=293)

12

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - High School English (N=126)

0%

5%

10%

15%

20%

25%

30%

35%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - High School Science(N=116)

13

0%

5%

10%

15%

20%

25%

30%


Pe

rce

nta

ge o

f Te

ach

er

by

Ye

ar E

stim

ate

s

NCSU - High School Social Studies (N=155)

14

Top Hiring Districts for Program Graduates North Carolina State University

Early Career Graduates (< 5 Years’ Experience)

All Program Graduates

Elementary School

Wake County Public School System 44.12% Wake County Public School System 40.74%

Harnett County Schools 8.82% Johnston County Schools 7.41%

Winston-Salem/Forsyth County Schools 5.88% Harnett County Schools 5.56%

Charlotte-Mecklenburg Schools 5.56%

Winston-Salem/Forsyth County Schools 3.70%

Durham Public Schools 3.70%

Cleveland County Schools 3.70%

% Employed by Top 3 Districts2 58.82%

% Employed by Top 7 Districts2 70.37%

Middle School

Wake County Public School System 44.77% Wake County Public School System 41.71%

Johnston County Schools 12.55% Johnston County Schools 11.52%

Durham Public Schools 3.35% Durham Public Schools 3.00%

Guilford County Schools 2.93% Guilford County Schools 2.53%

Franklin County Schools 2.51% Harnett County Schools 2.07%

Craven County Schools 2.09% Franklin County Schools 1.84%

Charlotte-Mecklenburg Schools 1.67% Craven County Schools 1.84%

Sampson County Schools 1.67% Charlotte-Mecklenburg Schools 1.61%

Sampson County Schools 1.38%

Wayne County Schools 1.38%

Lee County Schools 1.38%

Granville County Schools 1.38%

Cumberland County Schools 1.38%

% Employed by Top 8 Districts2 71.55% % Employed by Top 13 Districts1 73.04% 1Labor market was expanded beyond 10 school districts when multiple districts had the same percentage of program graduates

2A number of districts hired 1-3 program graduates

15

Top Hiring Districts for Program Graduate North Carolina State University

Early Career Graduates (< 5 Years’ Experience)

All Program Graduates

High School Wake County Public School System 39.63% Wake County Public School System 34.48%

Johnston County Schools 9.31% Johnston County Schools 7.16%

Guilford County Schools 4.52% Guilford County Schools 4.18%

Charlotte-Mecklenburg Schools 2.93% Charlotte-Mecklenburg Schools 2.69%

Winston-Salem/Forsyth County Schools 2.39% Harnett County Schools 2.24%

Harnett County Schools 2.39% Durham Public Schools 1.94%

Durham Public Schools 2.13% Franklin County Schools 1.94%

Franklin County Schools 2.13% Winston-Salem/Forsyth County Schools 1.79%

Cumberland County Schools 1.60% Cumberland County Schools 1.49%

Cleveland County Schools 1.60% Union County Public Schools 1.49%

% Employed by Top 10 Districts 68.62% % Employed by Top 10 Districts 59.40%

16

Appendix: North Carolina State University (NCSU) Traditionally Prepared Teachers vs. All Other Teacher Categories with Less than 5 Years Experience, Teaching in the Same School (2006-07 through 2010-11)

Compared to teachers in the same school, NCSU traditionally prepared undergraduates outperformed:

Other UNC Program Traditionally Prepared in High School Social Studies

UNC Graduate Degree Prepared in High School Social Studies

NC Private University Undergraduate Degree in High School Social Studies

Out of State University Undergraduate Degree in High School Social Studies

Out of State University Graduate Degree in High School Algebra I & II, High School Science, High

School Social Studies

UNC Licensure Only in High School Social Studies

Other Licensure Only in Elementary School Mathematics

Visiting International Faculty in High School Algebra I & II

Alternative Entry in High School Social Studies

Unclassifiable in High School Algebra I & II and High School Social Studies

Compared to teachers in the same school, NCSU traditionally prepared undergraduates

underperformed:

UNC Graduate Degree Prepared in High School Algebra I & II

NC Private University Graduate Degree in High School Science

Teach For America in Middle School Mathematics, Middle School Reading, Middle School

Science, Middle School Algebra I, High School Algebra I & II, High School English, and High School

Science

Visiting International Faculty in Middle School Reading

The following pages provide for results from models which limit comparisons to teachers in the same

schools. NCSU is set to zero for all tables and each value represents the relative difference between

teachers prepared by the marked category and NCSU traditionally prepared teachers, in average student

test scores (expressed in standard deviations).

17

Table A.1. Elementary School: NCSU Traditionally Prepared Teachers vs. All Other Teachers in the Same School with Less than

Five Years Experience

NCSU as Reference

Elementary School Math

Elementary School Reading

Elementary School Science

Value Standard

Error Value Standard


Error

Other UNC Program Traditionally Prepared -0.039 0.039 0.005 0.029 -0.033 0.058

UNC Graduate Degree Prepared -0.025 0.041 0.018 0.031 -0.059 0.068

NC Private University Undergraduate Degree -0.052 0.040 0.004 0.029 -0.082 0.058

NC Private University Graduate Degree -0.095 0.049 -0.031 0.035 -0.064 0.086

Out of State Undergraduate Degree -0.061 0.039 -0.004 0.029 -0.076 0.058

Out of State Graduate Degree -0.059 0.040 0.000 0.029 -0.045 0.059

UNC Licensure Only -0.035 0.042 0.019 0.031 -0.001 0.063

Other Licensure Only -0.109* 0.051 -0.066 0.037 NR NR

Teach For America 0.040 0.044 0.026 0.033 -0.013 0.070

Visiting International Faculty 0.004 0.042 0.043 0.031 0.094 0.069

Alternative Entry -0.049 0.040 0.008 0.030 -0.073 0.062

Unclassifiable -0.054 0.042 -0.002 0.031 0.004 0.067

18

Table A.2. Middle School: NCSU Traditionally Prepared Teachers vs. All Other Teachers in the Same School with Less than Five

Years Experience

NCSU as Reference

Middle School Math

Middle School Reading

Middle School

Science

Middle School Algebra I

Value Standard




Error

Other UNC Program Traditionally Prepared 0.023 0.015 0.012 0.008 0.011 0.043 -0.033 0.063

UNC Graduate Degree Prepared 0.021 0.031 0.004 0.014 -0.097 0.076 NR NR

NC Private University Undergraduate Degree -0.024 0.018 0.012 0.010 -0.090 0.058 -0.085 0.086

NC Private University Graduate Degree NR NR -0.003 0.026 NR NR NR NR

Out of State Undergraduate Degree 0.011 0.015 0.009 0.008 -0.011 0.040 0.072 0.066

Out of State Graduate Degree 0.005 0.020 0.006 0.010 -0.062 0.047 0.121 0.096

UNC Licensure Only 0.011 0.026 0.005 0.012 NR NR NR NR

Other Licensure Only NR NR 0.033 0.029 NR NR NR NR

Teach For America 0.128* 0.026 0.032* 0.013 0.330* 0.060 0.501* 0.166

Visiting International Faculty 0.022 0.021 0.055* 0.016 -0.005 0.072 -0.258 0.145

Alternative Entry 0.012 0.016 0.014 0.008 -0.023 0.040 0.010 0.070

Unclassifiable -0.017 0.028 0.007 0.013 -0.164 0.076 NR NR

19

Table A.3. High School: NCSU Traditionally Prepared Teachers vs. All Other Teachers in the Same School with Less than Five Years

Experience

NCSU as Reference

High School Algebra I & II

High School English I

High School

Science

High School Social Studies

Value Standard




Error

Other UNC Program Traditionally Prepared -0.008 0.020 -0.007 0.012 -0.014 0.040 -0.062* 0.020

UNC Graduate Degree Prepared 0.073* 0.033 0.006 0.015 0.057 0.044 -0.048* 0.023

NC Private University Undergraduate Degree -0.023 0.023 0.000 0.013 -0.081 0.046 -0.062* 0.023

NC Private University Graduate Degree 0.051 0.039 0.032 0.019 0.193* 0.069 -0.063 0.036

Out of State Undergraduate Degree -0.041 0.021 -0.007 0.012 -0.058 0.040 -0.096* 0.022

Out of State Graduate Degree -0.069* 0.035 0.013 0.016 -0.088* 0.042 -0.082* 0.027

UNC Licensure Only -0.017 0.051 0.002 0.019 -0.018 0.062 -0.090* 0.030

Other Licensure Only NR NR NR NR NR NR NR NR

Teach For America 0.197* 0.040 0.052* 0.023 0.115* 0.057 0.051 0.044

Visiting International Faculty -0.182* 0.046 0.019 0.040 -0.016 0.058 NR NR

Alternative Entry -0.032 0.021 -0.011 0.011 -0.020 0.038 -0.091* 0.021

Unclassifiable -0.142* 0.052 0.003 0.020 0.081 0.048 -0.080* 0.034

EDUCATION POLICY INITIATIVE at CAROLINA 1

Teacher Preparation and Performance in North Carolina Public SchoolsThis policy brief examines the distribution, effectiveness, and persistence of individuals entering the teaching profession with different forms of preparation. In comparison to teachers prepared by the University of North Carolina system, we find that: (1) North Carolina Independent College and University prepared teachers are less effective in middle grades mathematics and 5th and 8th grade science; (2) Out-of-state prepared teachers are less effective in elementary and high school grades and exhibit high rates of attrition; (3) Alternative entry teachers are less effective at the high school level, receive lower teacher evaluation ratings from their principal, and attrite at high rates; and (4) Teach For America Corps Members are more effective in secondary grades and STEM tested subjects and receive higher evaluation ratings from their principal but persist at very low rates. From these findings we conclude that:

1. Teacher selection into preparation categories and the preparation or training received therein has significant effects on early-career teacher effectiveness and persistence.

2. North Carolina should consider policies, such as improved hiring procedures and more intensive supports for beginning teachers, to improve the effectiveness and persistence of out-of-state prepared and alternative entry teachers.

3. School districts and teacher preparation programs should consider ways to utilize best practices from Teach For America’s recruitment, selection, and support model.

IntroductionGiven the importance of teachers to student achievement and later life outcomes, better research evidence is needed to understand the effectiveness and persistence of teachers entering the profession with different forms of preparation. At the state level, policy makers and university leadership can use this research to structure policies around teacher quality and improving teacher preparation, while at the local level, this research can assist school districts and principals with teacher hiring decisions. To address this research need, we classified North Carolina public school teachers into one of six policy-relevant categories that capture the preparation and formal education an individual held prior to first entering the teaching profession. Three of the categories

are for traditional teacher preparation: teachers prepared at the undergraduate, graduate, or licensure only level by the University of North Carolina (UNC) system, North Carolina Independent College and University (NC ICU) system or at an out-of-state university. The remaining three categories are for alternative preparation: Teach For America (TFA), Visiting International Faculty (VIF), and all other alternative entry teachers.1 In this policy brief we examine: (1) the distribution of teachers to these categories; (2) the effectiveness of teachers from these categories, as measured by their value-added to student achievement and evaluation ratings; and (3) the persistence rates of teachers from these categories for three and five years, respectively, in North Carolina public schools.

1 There is one additional category containing individuals who cannot be classified based on available data. We include these teachers in our analyses but do not report their results in this policy brief.


BackgroundIn response to the rapid expansion of alternative routes into the teaching profession, many prior studies have investigated the effects of teacher preparation or certification on student achievement. Specifically, these studies have asked whether traditionally prepared/certified teachers produce larger student test score gains than their alternatively prepared/certified peers. While an important advancement, this research has generally suffered from two shortcomings: (1) the teacher categories examined have been too broad to facilitate informed policy action, particularly in light of the diversity in teacher preparation practices within traditional and alternative routes and (2) the categories have conflated teacher preparation and certification—preparation is the formal education and training an individual brings with them into the teaching profession, while certification is a teacher’s changing status with respect to a particular course or group of students. Therefore, we created preparation categories to address these concerns. For example, instead of one traditional preparation category we have three—UNC system, NC ICU system, and out-of-state—to better inform policy action. Furthermore, our categories are fixed characteristics, identifying a teacher’s preparation when first entering the profession. In the following sections we address our research objectives and provide more detail regarding our sample and methods.

How Are Teachers Distributed Across Preparation Categories? As shown in Figure 1a, the UNC system was the largest source of teachers in North Carolina public schools during the 2011-12 academic year, comprising 36.6 percent of the teacher workforce, followed by out-of-state prepared, alternative entry, and NC ICU prepared teachers at 28.9, 14.7, and 12.5 percent of the workforce, respectively. While Teach For America garners significant research and policy focus, their Corps Members accounted for only 0.5 percent of the state’s teachers in 2011-12. To complement these static values, Figure 1b displays trends in the distribution of teachers from the 2004-05 through 2011-12 academic years. In the figure’s initial years, the share of UNC system and NC ICU prepared teachers was decreasing as the share of out-of-state prepared and alternative entry teachers increased. More recently, however, the percentage of UNC system prepared teachers is increasing, while the share of out-of-state prepared and alternative entry teachers has leveled off.

Figure 1a: Teacher Preparation Categories in the 2011-12 Academic Year

Figure 1b: Trends in Teacher Preparation Categories, 2004-05 Through 2011-12

NOTE: In the 2011-12 academic year there were 95,534 individuals paid as teachers in North Carolina public schools. This figure displays the teacher preparation categories arranged from largest (top) to smallest (bottom).

NOTE: This figure displays the percentage of teachers in each preparation category from 2004-05 through 2011-12.

480

858

5679

11955

14013

27565

34984

Teach For America

Visiting International Faculty

Unclassifiable

NC ICU System

Alternative Entry

Out-of-State

UNC System

0 5000 10000 15000 20000 25000 30000 35000 40000

0%

5%

10%

15%

20%

25%

30%

35%

40%

04-05 05-06 06-07 07-08 08-09 09-10 10-11 11-12

UNC System NC ICU System

Out-of-State Visiting International Faculty

Teach For America Alternative Entry


NOTE: This figure illustrates the value-added of NC ICU, out-of-state, Teach For America, Visiting International Faculty and alternative entry teachers in reference to UNC system prepared teachers. An *at the end of a horizontal bar indicates statistically significant differences at the 0.05 level.

VisitingInternational

Faculty

Teach For America

Alternative Entry

Out-of-State

NC ICU System

Math Reading 8th Grade Science -0.15 -0.10 -0.05 0.00 0.05 0.10 0.15 0.20 0.25

How Effective Are Teachers With Different Forms of Preparation?To assess teachers’ value-added to student achievement gains we focused on teachers with less than five years experience in the 2007-08 through 2011-12 academic years. In elementary and middle grades we estimated models in mathematics, reading, and science (5th and 8th grades), while in high school, we estimated models in mathematics (algebra I, algebra II , and geometry), science (biology and physical science), social studies (U.S. history and civics), and English I. For our high school models, data were available in geometry in the 2007-08 through 2009-10 academic years and for all other tests in the 2007-08 through 2010-11 academic years. In 2011-12, North Carolina only assessed students in algebra I, biology, and English I. In our value-added models, UNC system prepared teachers were the reference category and we used a multi-level model with controls for student, classroom, teacher, and school characteristics. As a check on our preferred multi-level model, we also implemented models to compare the effectiveness of UNC system prepared teachers with that of other teachers in the same schools. Except where noted, all results were similar.

When examining Figures 2a-c several value-added findings stand out. First, UNC system prepared teachers outperform NC ICU prepared teachers in three comparisons—5th grade science, 8th grade science and middle grades mathematics.

It may be useful to consider this impact in terms of days of student learning. For example, students in middle grades mathematics taught by a UNC system prepared teacher gain an average of nearly nineteen additional days of learning as compared to similar students taught by a NC ICU prepared teacher.2 When comparing within schools, NC ICU prepared teachers were also less effective in elementary grades mathematics but were more effective in high school science. Second, UNC system prepared teachers are more effective than out-of-state prepared teachers in elementary grades mathematics and 5th grade science and more effective than out-of-state prepared and alternative entry teachers in high school mathematics, science, and social studies.

In terms of days of student learning, elementary grades mathematics students taught by a UNC system prepared teacher gain an average of five days of additional learning as compared to similar students taught by an out-of-state prepared teacher. These results are particularly significant since out-of-state prepared teachers are highly concentrated in elementary grades and alternative entry teachers are highly concentrated in high school STEM subjects. Third, TFA Corps Members are the most effective early-career teachers in the state, outperforming UNC system prepared teachers in elementary mathematics and 5th grade science, middle grades mathematics, reading, and 8th grade science, and high school mathematics, science, and social studies. As shown in Figure 1a, however, the practical significance of

Figure 2b: Teacher Value-Added in Middle Grades

Figure 2a: Teacher Value-Added in Elementary Grades


-0.04 -0.02 0.00 0.02 0.04 0.06 0.08

Alternative Entry

VisitingInternational

Faculty

Teach For America

Out-of-State

NC ICU System

Math Reading 5th Grade Science

2 For more information on calculating days equivalency see Henry, G.T., et al. (2011). Technical Report: UNC Teacher Preparation Program Effectiveness. Chapel Hill, NC: Carolina Institute for Public Policy.


these results is limited since TFA Corps Members comprise less than one percent of the state’s teacher workforce. Finally, VIF teachers were more effective than UNC system prepared teachers in elementary grades mathematics and reading but were less effective in high school mathematics.

Since many important aspects of teaching quality may not be well-captured by value-added estimates and only a minority of teachers teach in tested grades/subjects, we also assessed the effectiveness of each preparation category using teachers’ North Carolina Professional Teaching Standards (NCPTS) evaluation ratings. There are five NCPTS – leadership, classroom environment, content knowledge, facilitating learning, and reflecting on practice – and for

each standard principals can rate teachers at one of five levels – not demonstrated, developing, proficient, accomplished, and distinguished. For these analyses we focused on teachers with less than five years experience in 2010-11 or 2011-12 and for each standard, we used models controlling for teacher and school characteristics to estimate the probability, relative to UNC system prepared teachers, that a teacher was rated as accomplished or distinguished (‘above proficient’). Overall, Table 1 shows that principals rate UNC system prepared teachers no differently than NC ICU or out-of-state prepared teachers. Across all five NCPTS, TFA Corps Members are more likely to be rated above proficient (e.g. Corps Members are 10.1 percent more likely than UNC system prepared teachers to be rated above proficient on Standard 1). Conversely, alternative entry teachers are less likely to be rated above proficient on all five NCPTS.

How Long Do Teachers From These Categories Persist in North Carolina?Because teacher attrition entails significant financial costs to school districts and may adversely impact school stability and student achievement, we identified first year teachers in North Carolina public schools in 2005-06, 2006-07, 2007-08, and 2008-09 and longitudinally tracked each cohort’s persistence over a five year period. Figure 3 displays the three and five year persistence rates for each teacher preparation category. Overall, it is clear that those teachers prepared in North Carolina – from the public and private colleges and universities – display the greatest commitment to teaching in the state. Conversely, out-of-state prepared and alternative entry teachers exit North Carolina public schools at higher rates than their in-state prepared peers. This is likely attributable to a desire to return home or to the lowered costs of entry into the profession for alternatively prepared teachers. Finally, consistent with the two year teaching commitments of TFA, we find that Corps Members are the least likely to persist for three or five years in North Carolina public schools.

Figure 2c: Teacher Value-Added in High School


Math English Science Social Studies

-0.10 -0.05 0.00 0.05 0.10 0.15

VisitingInternationalFaculty

Teach For America

Alternative Entry

Out-of-State

NC ICU System

Teacher CategoryStandard 1:Leadership

Standard 2: Classroom

Environment

Standard 3: Content

Knowledge

Standard 4: Facilitating Learning

Standard 5: Reflecting on

Practice

NC ICU System 0.013 0.010 -0.005 0.007 0.008

Out-of-State -0.005 -0.008 -0.009 -0.009 -0.009

Teach For America 0.101* 0.068* 0.058* 0.064* 0.062*

Visiting International Faculty -0.046* 0.032 0.029 0.011 -0.029

Alternative Entry -0.054* -0.037* -0.024* -0.044* -0.048*

NOTE: This table displays the probability of rating above proficient for each of the five NCPTS in reference to UNC system prepared teachers. * Indicates statistical significance at the 0.05 level.

Table 1: Rating ‘Above Proficient’ on the NCPTS


DiscussionIn this policy brief we examined the distribution, effectiveness, and persistence of those entering the teaching profession with different forms of preparation. Overall, we found that the selection of individuals into these preparation categories and the training received therein has significant effects on student achievement gains, evaluation ratings, and the persistence of teachers. These results imply that state policies on teacher quality, the selection and preparation practices of teacher education programs, and the hiring practices of school districts can have substantial effects on outcomes of interest. Below, we discuss key implications of our findings.

First, the size of each preparation category or the concentration of teachers from a preparation category in a particular grade/subject influences the practical significance of our findings. For example, alternative entry teachers’

negative value-added effects in high school mathematics and science are especially troubling since these teachers comprise 30 and 47 percent of the early-career workforce in those subjects. Second, despite the average effectiveness differences between UNC system prepared and out-of-state prepared and alternative entry teachers, further research shows a high degree of overlap in the distributions of teacher effectiveness across these groups. Essentially, there are many highly effective out-of-state prepared and alternative entry teachers. This means that North Carolina should not take action to broadly curtail alternative entry programs or licensure reciprocity agreements, but rather, focus on improving the quality and persistence of these groups through more effective screening/hiring procedures and more intensive supports for beginning teachers. Finally, we argue that the effectiveness of Teach For America is not a challenge to traditional teacher preparation, especially since Corps Members comprise such a small percentage of the workforce, but rather, an opportunity to identify highly-effective recruitment, selection, and support practices that can be scaled-up to a university, district, or statewide level. Teacher education programs and school districts in North Carolina should implement selected practices, with future research investigating the effects of these reforms.

For more research on this topicBoyd, D., Goldhaber, D., Lankford, H., & Wyckoff, J. (2007). The effect of certification and preparation on teacher quality. The Future of Children, 17(1), 45-68.

Henry, G.T., Bastian, K.C., Fortner, C.K., Kershaw, D.C., Purtell, K.M., Thompson, C.L., & Zulli, R.A. (2014). Teacher preparation policies and their effects on student achievement. In press, Education Finance and Policy.

Henry, G.T., Purtell, K.M., Bastian, K.C., Fortner, C.K., Thompson, C.L., Campbell, S.L., & Patterson, K.M. (2014). The effects of teacher entry portals on student achievement. Journal of Teacher Education, 65(1), 7-23.

National Research Council. (2010). Preparing Teachers: Building Evidence for Sound Policy. Washington, D.C.: National Academies Press.

Figure 3: Teacher Persistence in North Carolina Public Schools

NOTE: For four cohorts of first-time teachers (2005-06, 2006-07, 2007-08, and 2008-09) this figure depicts the percentage of teachers who return for a 3rd and 5th year of teaching in North Carolina public schools.


Study Authors: Kevin C. Bastian and Kristina M. Patterson

The Education Policy Initiative at Carolina is a policy research and outreach unit affiliated with the Department of Public Policy and housed in the College and Arts and Sciences at the University of North Carolina at Chapel Hill.

ncsu · 2016-02-10 · epp progress against the targets are evaluated annually by the epp and...

Documents