evaluation of the professional development lab …...evaluation of the pdl programs in community...

Evaluation of the Professional Development Lab (PDL) Programs in Community School District 20

2002 – 2003

CENTER FOR RESEARCH ON TEACHING AND LEARNING

Department of Teaching and Learning

82 Washington Square East, Suite 700

New York, NY 10003 | 212 998 5872 | 212 995 3636 fax

www.steinhardt.nyu.edu/teachlearn/crtl

Evaluation of the PDL Programs in Community School District 20

CRTL Research Report Series, EE-0304-01

i

Evaluation of the Professional Development Lab (PDL) Programs in

Community School District 20

2002 – 2003

Robert J. Tobias, Director

Rosa L. Rivera-McCutchen, Graduate Assistant

CRTL External Evaluation Report Series

EE-0304-01

March 2004

Center for Research on Teaching and Learning

Department of Teaching and Learning

Steinhardt School of Culture, Education, and Human Development

New York University

© Copyright 2004 by the Center for Research on Teaching and Learning



ii

ACKNOWLEDGEMENTS

The Center for Research on Teaching and Learning would like to thank a number of individuals

who were instrumental in the research and writing phases of this report. Ognjen Simic, Research

Scientist, conducted statistical analyses of the data. Laura Carroll and Emily Hole, graduate

student workers, assisted with data collection and entry. Alexandra Snyder, also a graduate

student worker, compiled the school profiles. Beth McDonald, Research Scientist, reviewed

early drafts of the report and provided valuable feedback.

Finally, this report would not have been possible without the cooperation of the PDL staff,

consultants and participants, including the New Teacher Facilitators and Resident Teachers.



iii

TABLE OF CONTENTS

List of Figures and Tables ........................................................................................................... v

Executive Summary..................................................................................................................... vi

I. Introduction ............................................................................................................................. 1

II. The Programs .......................................................................................................................... 2

The Professional Development Laboratory (PDL) ............................................................... 2

PDL’s Programs in CSD 20................................................................................................... 2

PDL for New Teachers .................................................................................................... 2

PDL for Middle School Social Studies Teachers ............................................................ 6

Learning Through Practice Leadership Program............................................................. 7

Context................................................................................................................................... 8

III. The Evaluation ....................................................................................................................... 9

The Evaluators ....................................................................................................................... 9

Evaluation Questions ............................................................................................................. 9

Methods of Data Collection ................................................................................................... 10

New Teacher Staff Training Observations ...................................................................... 10

NTF and RT Questionnaires and Interviews ................................................................... 10

New Teacher Surveys ...................................................................................................... 11

Individual Performance Assessment Logs (IPAL’s) ....................................................... 11

Teacher Efficacy Scale .................................................................................................... 12

New Teacher Classroom Observations ........................................................................... 12

Social Studies Institute Feedback Questionnaire............................................................. 13

Teacher Retention Data.................................................................................................... 13

Standardized Achievement Test Scores........................................................................... 13

Assumptions and Limitations ................................................................................................ 14 IV. Findings ................................................................................................................................. 16

New Teacher Participants ...................................................................................................... 16

Social Studies Teacher Participants ....................................................................................... 17

Evaluation Question 1............................................................................................................ 18

Evaluation Question 2. .......................................................................................................... 19

PDAQ .............................................................................................................................. 19

IPAL’s .............................................................................................................................. 21

Interview Data on the Cycle Component......................................................................... 23

Evaluation Question 3A......................................................................................................... 23

Evaluation Question 3B ......................................................................................................... 27





Evaluation Question 6............................................................................................................ 33



V. Conclusions and Recommendations ....................................................................................... 40

Conclusions............................................................................................................................ 40



iv

Evaluation Question 1...................................................................................................... 40

Evaluation Question 2. .................................................................................................... 40

Evaluation Question 3A................................................................................................... 41

Evaluation Question 3B ................................................................................................... 41





Evaluation Question 6...................................................................................................... 43



Recommendations.................................................................................................................. 44

PDL for New Teachers and LTPLP................................................................................. 44

PDL for Middle School Social Studies Teachers ............................................................ 46

References .................................................................................................................................... 47



v

LIST OF TABLES AND FIGURES

Figure 1. PDL for New Teachers Program Model....................................................................... 4

Figure 2. Three-year Implementation of NTF & TLI Components............................................. 5

Figure 3. Social Studies Institute Program Model ...................................................................... 7

Table 1. PDL for Middle School Social Studies Teachers Three-Year Model ........................... 8

Table 2. Distribution of New Teacher Program Interns by School ............................................. 16

Table 3. Descriptive Characteristics of New Teacher Program Interns ...................................... 17

Table 4. Subjects Taught by New Teacher Interns ..................................................................... 17

Table 5. Distribution of Social Studies Program Participants by School .................................... 18

Table 6. New Teachers’ Importance Ratings and Reports of PDL Program Support

on the Professional Development Activities Questionnaire .................................................. 21

Table 7. Mean Self Ratings on the Items of the Elements of Effective Classrooms

and Instruction ....................................................................................................................... 26

Table 8. Correlations Between New Teachers’ Self-Reported Degree of Professional

Skill and the Extent of Program Mentoring in Four Essential Areas of Effective

Classroom Teachers ............................................................................................................... 27

Table 9. Mean Self Ratings on the Teacher Efficacy Scale (New Teachers) .............................. 29

Table 10. Mean Self Ratings on the Teacher Efficacy Scale Pre-Test (S.S. Teachers)............... 30

Table 11. Mean Observation Ratings in the Domain Referenced Teacher Observation

Tool ........................................................................................................................................ 31

Table 12. Three-year Comparison of the Retention Rates of New Teachers in PDL

Cohort Schools and Comparison Schools.............................................................................. 34

Table 13. PDL for New Teachers: Means and SD for Pre and Post ELA Scale Scores

of Students of PDL-Served and Non-PDL Served Teachers ................................................ 36

Table 14. PDL for New Teachers: Summary of T-Tests for the Significance of the

Difference in ELA Gain Scores for Students of PDL Served and Non-PDL

Served Teachers ..................................................................................................................... 37

Table 15. PDL for Middle School Social Studies Teachers: Means and SD’s for

Social Studies and ELA Scale Scores of Students of PDL-Served and Non-PDL

Served Teachers ..................................................................................................................... 38

Table 16. PDL for Middle School Social Studies Teachers: Summary of Stepwise

Multiple Regression Analysis of the Effects of PDL on Grade 8 Students

Social Studies Scale Scores Controlling for 2002 ELA Scale Scores .................................. 39



vi

EXECUTIVE SUMMARY

INTRODUCTION This report presents the findings of an evaluation of the Professional Development

Laboratory (PDL) programs in New York City’s Community School District (CSD) 20 during

the 2002-2003 academic year. Two of these programs were designed to provide support for the

professional development of two groups of middle school teachers in the district: first-year

teachers who required mentoring by NYS state law and social studies teachers who wished to

enhance their instructional skills. A third program was designed to provide professional

development in the area of leadership for staff developers and exemplary teachers.

The evaluation was conducted by the Center for Research on Teaching and Learning

(CRTL) of the Department of Teaching and Learning in New York University’s Steinhardt

School of Education. The purpose of the evaluation was to document the implementation of the

PDL programs for middle school teachers in CSD 20 and to assess its impact upon four groups

of participants as follows:

• First, the impact of PDL training upon the mentoring capacities of school-based

professional-development staff in CSD 20 middle schools;

• Second, the effects of the program upon the professional competence, sense of efficacy,

and retention in the profession of new middle school teachers;

• Third, the effects of the program upon the instructional skills and sense of efficacy of

selected middle school social studies teachers; and

• Last, the indirect effects of the program upon the academic achievement of students

served by the programs’ teacher participants

THE PROGRAMS

PDL

PDL was instituted in 1989 and currently functions under the aegis of the Department of

Teaching and Learning’s Ruth Horowitz Center for Teacher Development at New York

University’s Steinhardt School of Education. Over the past fourteen years, PDL has

implemented a variety of professional development and training programs for teachers and

school administrators in 18 New York City school districts. At the heart of the PDL philosophy

is the belief that by opening classroom doors, teachers have the opportunity to reflect on their

teaching practice and analyze student work to develop instructional approaches, all while

collaborating with one another. The PDL model centers on the belief teachers learn best to

improve their practice when the learning takes place in actual classrooms during the school day.

PDL designed and implemented the three programs in CSD 20 that are described below.



vii

PDL for New Teachers Program This program was a three-year induction model for first-year teachers that was designed

to build their professional competency in planning lessons, analyzing student academic growth

and student work, delivering appropriate instructional strategies, and using cooperative learning

strategies. The program also aimed to develop the capacity of CSD 20 to provide high-quality

mentoring for new teachers. Toward that end, PDL provided various training opportunities for

New Teacher Facilitators (NTF’s) and Resident Teachers (RT’s) who, in turn, mentored new

teachers. Another component of this was the PDL Cycle Program. The goal of this component

was to provide the new teachers with opportunities to improve their pedagogical skills by

participating in focused visits to RT classrooms. Additionally, new teachers were encouraged to

enroll in one of four Teacher Leadership Institutes, courses that were designed by PDL to assist

new teachers in developing effective teaching strategies and leadership capacity.

PDL for Middle School Social Studies Program

This program was designed to help CSD 20 social studies teachers develop professional

competency in planning lessons, understanding student development, analyzing student work,

developing appropriate instructional strategies, and integrating technology in the social studies

classroom. To achieve its goals, the program provided a series of training institutes for 20

middle school social studies teachers.

Learning Through Practice Leadership Program (LTPLP)

This program was designed to develop the leadership potential of selected district

teachers and staff developers whose classrooms exemplified standards-based teaching and

learning and best practices. All participants also played a role in the PDL for New Teachers

program. LTPLP consisted of seven full-day training sessions that focused on various leadership

strategies, differentiated instruction, multiple intelligences, using protocols, observation

techniques, and using data to guide instruction.

THE EVALUATION A CRTL research team conducted the evaluation during the period September 2002

through June 2003. The evaluation used a variety of methods to collect qualitative and

quantitative data on the effects of the programs, including interviews, observations, self-report

questionnaires, affective surveys, internal program records, human resources data, and student

achievement test scores. Specific data collection methods and instruments included the

following:

• NTF and RT Questionnaires and Interviews assessed the impact of PDL training they

received on their leadership skills and on their work with new teachers;

• Participant Information Form collected basic demographic information from new

teachers;



viii

• Professional Development Activities Questionnaire (PDAQ) asked teachers to

identify the support they received through the program in a number of professional

development areas, and to rate the level of importance of each;

• Elements of Effective Classrooms and Instruction (EECI) asked new teachers to rate

their level of development as a teacher in twenty-seven areas, and the extent to which

they had received mentoring in those areas;

• Teacher Efficacy Scale (TES) measured the teachers’ sense of professional efficacy

and its relationship to teacher behavior and student achievement;

• Domain Referenced Teacher Observation (DRTO) protocol recorded ratings from

observations and post-observation conferences to measure the new and social studies

teachers’ proficiency in each of four domains of practice, including Planning and

Preparation, Classroom Environment, Professional Responsibility, and Classroom

Instruction;

• Social Studies Feedback Questionnaire assessed social studies teachers’ ratings of

program professional development in seven areas that were identified in the

program’s literature, and the extent to which the training influenced their practice;

• Teacher Retention data obtained from the NYC DOE.

Contextual Caveat The evaluation was conducted in the context of the ongoing implementation of the

program in a school system that was undergoing major restructuring. Throughout the year, CSD

20 was in transformation as it prepared for the transition from a semi-independent school district

to incorporation in a larger region. The reorganization had an impact on the district and school

climate, leaving many personnel at the district and school level unsure of their job security. This

uncertainty was exacerbated by the announcement in February 2003 that CSD 20’s

superintendent would be resigning his position in the spring. The reorganization of the DOE also

led the district to rely heavily on NTF’s to fulfill additional administrative responsibilities which

were beyond the scope of their mentoring roles. These factors no doubt affected the

implementation of the program and its effects upon the participants in ways that could not be

measured.

FINDINGS AND CONCLUSIONS

The data collected in this evaluation were analyzed to address six questions concerning

the effects of the program. The findings and conclusions pertaining to these questions are

summarized below.

Evaluation Question 1: How well did PDL prepare NTF’s and RT’s for their

mentoring and modeling roles, respectively, in the new teacher program?

The findings of this report suggested that the PDL training provided to NTF’s was

effective and appropriate. NTF’s reported that the various training components aimed at

preparing them for their mentoring roles, including the Author Study Exchanges, weekly NTF

meetings, and the NTF Institutes, were useful to them. The LTPLP sessions, though not directly

affiliated with the PDL for New Teachers program, were also noted as highly useful training



ix

opportunities. Observations of the sessions suggested that NTF’s were given opportunities to

reflect on and practice learned skills. Data collected from the PDAQ questionnaire supported

this conclusion, indicating that NTF’s responded to the most important needs of their new

teacher interns.

Other data collected pointed to school-level contextual obstacles that had to be overcome

by the NTF’s in their work with their interns. NTF’s commented that in some cases their

mentoring schedules were interrupted by emergency coverages that were assigned to their

interns. Also, several of the NTF’s assumed other functions in the schools, and they noted that

the additional responsibilities often interfered with their mentoring responsibilities. The latter

may have been a manifestation of the resource demands attendant to DOE restructuring, a

contextual effect that may not generalize to other program implementations.

Evaluation Question 2: In the perception of new teachers, to what extent did

program services address their most important staff professional development and

support needs?

Evidence from the PDAQ indicated that new teachers perceived the PDL for New

Teachers program to be responsive to their most important professional development and support

needs, including motivating students, classroom management, lesson planning, and clarifying

instructional goals. The sample IPAL’s collected also suggested that NTF’s responded to the

needs of their interns. IPAL’s consistently pointed to support provided in the areas of classroom

management and lesson planning, two areas reflected in the PDAQ as important to the new

teachers.

Additionally, many of the teachers who participated in the Cycle Program reported that

they benefited from the program’s activities, particularly the opportunity to observe a veteran

teacher using instructional strategies that incorporated group work, manipulatives, and multiple

intelligences. Participants did note that the program could have been enhanced with a second

round of visits to the RT’s classrooms.

Overall, the evidence suggested that the PDL for New Teachers program had a structured

model for implementation, but also had sufficient flexibility to meet the individual professional

development needs of new teachers.

Evaluation Question 3A: To what extent did new teachers believe that the

program facilitated their professional growth in the competencies of effective

teachers?

Multiple analyses of the data collected from the EECI showed a relationship between the

areas in which new teachers reported receiving PDL support and the areas where teachers

believed themselves to be most effective. The data also showed that more program participants

felt competent and supported in the domain of Planning and Preparation, which included areas

such as content knowledge, assessment, and goal setting. The relationship between the teachers’

beliefs about their competencies and the support they received from PDL was also strongest in

this domain. On the other hand, a notably low number of respondents indicated high skill levels



x

in the areas of gaining access to school resources for students and engaging families.

Correspondingly, a low number of respondents indicated high levels of support from PDL in

those same areas.

Evaluation Question 3B: To what extent did social studies teachers believe that the

program facilitated their professional growth in the competencies of effective teachers?

Participants in the PDL for Middle School Social Studies Teachers program believed the

program facilitated their professional growth as teachers. The Social Studies Feedback Survey

and PDL’s internal program feedback forms indicated that participants believed their ability to

integrate technology, primary source documents, literature, and cooperative learning strategies

into their classrooms was enhanced by this PDL program. Some participants did note, however,

that their ability to integrate technology in their practice was hampered due to their schools’ lack

of technological resources. This varied from school to school.

Evaluation Question 4A: Did new teachers served by the program show an

increase in their sense of efficacy as a teacher?

The evidence with respect to PDL’s impact on new teachers’ sense of efficacy was

inconclusive. There was no significant increase in the new teachers’ sense of efficacy, due in

part to the low return rate on the post-TES. Additionally, the literature regarding teacher

efficacy suggested that the construct of “general teacher efficacy” may be problematic and

difficult to measure. These findings highlight the need for more research on the measurement of

this construct, and teacher beliefs and attitudes in general.

Evaluation Question 4B: Did social studies teachers served by the program show an


The evidence with respect to PDL’s impact on the middle school social studies teachers’

sense of efficacy was similarly inconclusive. There was no significant increase in the social

studies teachers’ sense of efficacy, due in part to the low return rate on the post-TES.

Additionally, the cautionary literature cited above obtains here as well.

Evaluation Question 5A: To what extent did new teachers served by the program

demonstrate the professional competencies of effective classroom teachers?

On average, participants in the PDL for New Teachers program were either partially

proficient or were approaching proficiency in the professional competencies of effective

teachers. The data from the DRTO demonstrated a slight increase in the mean scores of the

sample of teachers over time. There was a notable increase in the mean scores of six areas. Four

of the areas were in the Planning and Preparation domain, and the remaining two were in the

Instruction and Classroom Environment domain, respectively. In general, the changes in mean

scores were in the expected direction. It is expected that with a larger sample and/or a longer

intervention, the difference in mean scores would have been greater.



xi

Evaluation Question 5B: To what extent did social studies teachers served by the

program demonstrate the professional competencies of effective classroom

teachers?

The sample of social studies teachers who participated in the observation phase of the

research generally demonstrated at least “partial proficiency” in the professional competencies of

effective classroom teachers. In three of the four cases, scores generally remained constant or

increased slightly in the second observation. Conversely, in the fourth case, the teacher’s ratings

decreased in a number of the twenty-nine areas from “proficient” to “partially proficient.”

Evaluation Question 6: Were the retention rates of new teachers served by the

program better than new teachers in comparable schools that were not?

The analysis of new teacher retention data showed a trend toward a favorable effect of

PDL upon reducing the rate of new teachers leaving the New York City school system. For both

cohorts of PDL schools, the attrition rates showed a slight increase in the first year of service, but

far less than the increase in comparison schools. Further, there was a decline in system leavers

for the schools that were served by PDL for a second year. The data on new teachers who stayed

in the same school were more equivocal. Teacher retention is a complex issue and is affected by

many factors. Moreover, the evaluation of the program took place under the influence of the

confounding effects of new state regulations and the systemic restructuring of the New York City

public schools.

Evaluation Question 7A. Did students of new teachers served by the program

show gains in ELA achievement test scores that were equal to those for all CSD

20 middle school students?

The analysis of ELA achievement test score data indicates that PDL had a favorable

affect on students of PDL-served teachers in CSD 20. The mean gains on ELA achievement of

the PDL group were higher than those for the non-PDL group in three out of the four schools and

four out of the six grade-within-school comparisons. Mean gains for the two groups were similar

in the other two analyses. The mean gains of the PDL group were also higher for the combined-

schools analyses. In general, PDL had statistically significant and educationally meaningful

effects on the achievement of students of new teachers who were served by the PDL for New

Teachers program.

Evaluation Question 7B. Were the State Intermediate Level Social Studies

Achievement Test (SILSSAT) scores of students of teachers served by the

program higher than those of students of non-PDL teachers in the same schools?

The analysis of SILSSAT scores indicates that PDL did not have a statistically significant

effect on scores beyond the influence of prior general achievement. The mean 2003 SILSSAT

scale scores tended to be higher for the non-PDL group than for the PDL group both within

schools and across the four schools combined. However, the mean 2002 ELA scale scores also

tended to be higher for the non-PDL group. Accordingly, the differences in SILSSAT means



xii

may have been attributable to differences in the general achievement of the two groups prior to

the program.

RECOMMENDATIONS Based on the findings and conclusions of the evaluation, the following recommendations

are offered to enhance the effects of future implementations of these programs.

PDL for New Teachers

♦ Make principals and other supervisory staff more aware of PDL’s mission and goals through

by strengthening orientation and conducting ongoing updates.

♦ PDL and District staff should continue to work with school leadership to devise ways to

encourage and facilitate structured visits to classrooms.

♦ Help the NTF’s identify the impact of their work on students. All NTF’s who responded to the

CRTL questionnaire indicated that they believed their interns were positively impacting their

students, yet they often could not cite “concrete evidence.” Being able to cite specific

evidence would not only help to enhance the new teacher’s craft, but would also go a long way

toward helping the role of PDL and the NTF become more valued.

♦ Consider ways to schedule and deliver training so that NTF’s are out of their school buildings

less often. All of the NTF’s who responded to the questionnaire noted that the various training

components were valuable and reinforcing. The biggest drawback they noted, however, was

that the numerous training sessions took them out of the building and often interfered with

their mentoring schedule. This was exacerbated by the additional administrative

responsibilities placed on them by the District.

♦ Expand the PDL for New Teachers program emphasis to include explicit training on gaining

access to school resources for students and working with families, two areas that the literature

suggests are important features of effective instruction. The data collected on the EECI

indicated that teachers’ self-ratings of their professional skill and the emphasis of mentoring

particularly in these two areas were relatively low.

♦ PDL should consider expanding the pool of teachers eligible for The Cycle Program so more

cohorts of teachers, including new teachers who have already met the mentoring requirement,

can take advantage of this highly effective model. PDL should also consider beginning the

Cycle Program earlier in the year and expand it to include a visit by the Resident Teachers to

their Visiting Teachers’ schools.

While the above recommendations are aimed at strengthening the quality of

implementation of future replications of the program, it should be emphasized that there is

considerable evidence of the program’s impact upon the professional growth of new teachers and

the quality of teaching and learning in their classrooms. Particularly notable in this regard is

evidence suggesting that the program’s basic model is effective and should be retained as a

foundation.



xiii

PDL for Middle School Social Studies Teachers

♦ PDL and District staff should ensure that participants in the PDL for Middle School Social

Studies program, and their students, have access to the technology resources that would allow

them to implement learned approaches.

♦ Evaluations of future replications should pay greater attention to the collection of more

complete data. This can be achieved by identifying the students served by program teachers

prospectively, i.e. at the beginning of the program, so that plans can be developed to capture

their achievement test scores later in the project year. These achievement test data should be

supplemented by other evidence of growth in student achievement, chief among which is the

structured analysis of student work.

IMPLICATIONS FOR RESEARCH AND EVALUATION

Along with higher education institutions across the country, the NYU Steinhardt School

of Education is in the process of developing evaluation and accountability systems for its teacher

education programs. Accountability through accreditation is a requirement under federal and

state legislation aimed at improving teacher education. The work conducted in this evaluation of

the Professional Development Laboratory has direct implications for the evaluation system the

Steinhardt School of Education is creating for its teacher education programs. This evaluation

gave CRTL the opportunity to pilot a variety of instruments, including the Teacher Efficacy

Scale, the Domain Referenced Teacher Observation protocol, the Elements of Effective

Classrooms and Instruction teacher questionnaire, and the Professional Development Activities

Questionnaire. Piloting these instruments gave CRTL valuable insight in furthering the creation

of an effective evaluation system for the NYU Steinhardt School of Education teacher education

programs


CRTL Research Report Series, RR-0304-01

1

I. INTRODUCTION

This report presents the findings of an evaluation of the Professional

Development Laboratory (PDL) programs in New York City’s Community School

District (CSD) 20 during the 2002-2003 academic year. These programs were designed

to provide support for the professional development of two groups of middle school

teachers in the district: first-year teachers who required mentoring and social studies

teachers who wished to enhance their instructional skills. A third program was designed

to provide professional development in the area of leadership for staff developers and

exemplary teachers.

The evaluation was conducted by the Center for Research on Teaching and

Learning of the Department of Teaching and Learning in New York University’s

Steinhardt School of Education. The purpose of the evaluation was to document the

implementation of the PDL programs for middle school teachers in CSD 20 and to assess

its impact upon four groups of participants as follows:

• First, the impact of PDL training upon the mentoring capacities of school-based

professional-development staff in CSD 20 middle schools;

• Second, the effects of the program upon the professional competence, sense of

efficacy, and retention in the profession of new middle school teachers;

• Third, the effects of the program upon the instructional skills and sense of

efficacy of selected middle school social studies teachers; and

• Last, the indirect effects of the program upon the academic achievement of

students served by the programs’ teacher participants.

In the objectives of the evaluation stated above, direct effects refer to those that

are attributable to the work of PDL with the individuals who are exhibiting growth or

improvement. Indirect effects, on the other hand, are changes in individuals that are

mediated through the work of other individuals who impact upon them.

By focusing on these participant groups, the evaluation assesses the cascading

impact of the programs beginning with the training provided by PDL staff and

consultants to school-based professional developers, to the work of the latter with new

and experienced teachers, and ending with the effects of this work upon the academic

performance of students.

This evaluation report is organized in five sections. Following the introduction,

the second section describes the programs, including its sponsoring agency, PDL,

descriptive features of its context, CSD 20, and the rationale for the program. The third

section describes the evaluation, including the evaluation agency, the methods, and

assumptions and limitations. The fourth section describes the major evaluation findings,

and the last section presents conclusions and recommendations based on the findings.



2

II. THE PROGRAMS

The Professional Development Laboratory (PDL)

PDL designed and implemented the professional development programs that were

the focus of this evaluation. PDL was instituted in 1989 and currently functions under

the aegis of the Department of Teaching and Learning’s Ruth Horowitz Center for

Teacher Development at New York University’s Steinhardt School of Education.

Additional PDL partners include the New York City Department of Education (DOE),

the UFT, and the business sector. Over the past fourteen years, PDL has worked with 15

New York City school districts. Currently, PDL runs programs in Brooklyn’s

Community School Districts (CSD) 18 and 20. Over this period, PDL has implemented a

variety of professional development and training programs for teachers and school

administrators. The breadth of this work is best described by the goals stated in PDL’s

awareness literature:

• Developing leadership capacity in teachers and administrators for the purpose of

improving student learning.

• Preparing facilitators to convene groups of educators and school teams to examine

student work and analyze student data in order to improve student performance.

• Establishing networks to share practice across school districts.

• Integrating technology into the teaching and learning process.

• Evaluating the impact of professional development on student learning.

• Collaborating with other school change initiatives.

• Providing access to expertise in the field of education research.

• Incorporating current educational research into program design.

At the heart of the PDL philosophy is the belief that by opening classroom doors,

teachers have the opportunity to reflect on their teaching practice and analyze student

work to develop instructional approaches, all while collaborating with one another. The

PDL model centers on the belief teachers learn best to improve their practice when the

learning takes place in actual classrooms during the school day.

PDL’s Programs in CSD 20

The focus of this evaluation is on three of PDL’s programs in CSD 20: PDL for

New Teachers, PDL for Middle School Social Studies Teachers and the Learning

Through Practice Leadership Program (LTPLP). Although these three programs are

discrete, all focus on the same ultimate goal—to improve teaching and learning by

helping to build the professional competency of teachers.

PDL for New Teachers

The first year of teaching is extraordinarily overwhelming for teachers and

involves a number of phases, starting with anticipation, then survival, disillusionment,



3

rejuvenation, and reflection (Moir, 1990). Research shows mentoring is an essential

component of support and retention of new teachers as they go through these phases

(Moir & Bloom, 2003; Black, 2001; Mauer & Zimmerman, 2000). Highly qualified

mentors are critical elements in any teacher induction program, and cannot be easily

replaced. However, the literature cautions that poorly trained mentors can hinder the new

teacher’s induction and increase attrition (Moir & Bloom, 2003). Quality mentorship

programs involve ongoing mentor training, with opportunities for building knowledge,

reflection, and problem-solving as mentors work with their interns (Moir & Gless, 2003;

Feiman-Nemser, 2003). States with rigorous and high-quality mentor training programs

have a marked decrease in teacher attrition (Darling-Hammond, 2003). However, in

order to be successful, mentoring and professional development programs require a

financial and logistical “buy-in” on the part of the school administration and their district.

PDL for New Teachers was a three-year model designed to build the capacity of

CSD 20 to provide the high-quality mentoring that research has found to be integral to

the professional growth and retention of new teachers. The program was designed to

build professional competency in planning lessons, analyzing student academic growth

and student work, delivering appropriate instructional strategies, and using cooperative

learning. In addition, the program aimed to integrate new teachers into the school

community, assist them in achieving certification, and increase their retention in the

profession. The program aimed to achieve its goals by linking the new teachers to New

Teacher Facilitators (NTF), who were trained to serve as mentors by both PDL staff and

PDL-trained district staff.

The program was organized around a comprehensive array of training

components that were designed to ensure that district staff, responsible for mentoring the

new teachers, engaged in collaborative professional activities that enhanced their

mentoring skills. Figure 1 displays the program model as depicted by the organization of

the program components. The NTF’s were integral to the success of the program, since

they directly provided mentoring to the new teachers and coordinated the delivery of

program resources as well. As Figure 1 shows, the program model trained NTF’s

through three components: NTF Weekly Meetings/Author Study Exchanges, the 10-day

NTF Institute, and the LTPLP.

The weekly meetings were facilitated by PDL staff and were conducted at a PDL

Lab in one of the middle schools or the district office, and were attended by the five first-

year NTF’s. From November through May, the weekly meetings were replaced with an

Author Study Exchange that was attended by all NTF’s. During the Author Study

Exchanges, also facilitated by PDL staff, NTF’s read and discussed the current research

literature on topics such as effective coaching and feedback strategies, analyzing student

work, and differentiated instruction.

The seven Author Study Exchanges were half-day training sessions at the district

office. Figure 2 displays the program’s designed three-year implementation scheme, with

two cohorts of new teachers receiving support from NTF’s. For each year of

implementation, the NTF’s provide formal mentoring support to the new teachers, and



4

informal support to returning teachers. The model also displays the additional support

provided to new teachers through the Teacher Leadership Institute (TLI) courses,

commensurate with their level of need.

FIGURE 1

PDL FOR NEW TEACHERS PROGRAM MODEL

The NTF Institutes involved ten full days of training, on NYU’s campus, focused

on building the leadership capacity of the NTF’s, as well as developing a professional

community to share experiences and knowledge with one another. Sessions focused on

effective mentoring strategies, differentiated instruction, linking the INTASC standards1

to practice, and using protocols to assess the progress of the NTF’s work with new

teachers. NTF’s were expected to integrate the strategies learned during the Institutes into

their mentoring practice.

1 INTASC stands for Interstate New Teachers Assessment and Support Consortium.

NTF Institutes

NTF Weekly Meetings/Author

Studies

LTPLP

TLI

♦ Monthly course taught by NTF

♦ Focus on INTASC standards

NTF

♦ Weekly Observations & Conferences

♦ Schedules NT observations of veteran

staff

♦ Models effective practice

♦ Provides access to resources

RT

♦ Models effective practice

♦ Conducts post-observation conferences

New

Teachers

Students



5

FIGURE 2

THREE-YEAR IMPLEMENTATION OF NTF & TLI COMPONENTS

Another training opportunity that both NTF’s and some RT’s received was the

LTPLP. While this program is not directly linked to the PDL for New Teachers program,

it provided training for those leaders involved with the program in the form of seven

training days at the district focused on various leadership strategies, differentiated

instruction, multiple intelligences, using protocols, observation techniques, and using

Department of Education data to guide instruction.

Another component of the PDL for New Teacher program was the PDL Cycle

Program. The goal of this component was to provide the new teachers with opportunities

to improve their pedagogical skills while working with Resident Teachers (RT),

experienced teachers who were recognized by the district for their professional expertise

and exemplary classroom practice. New teachers earned new teacher credit for their

participation in the Cycle Program.

Yr 1 NTF’s Yr 1 New

Teachers TLI 1

Year One (2001-2002)

Year Two (2002-2003)

Yr 1& 2

NTF’s

Yr 2 New

Teachers TLI 1

Yr 1 New

Teachers TLI 2

Year Three (2003-2004)

Yr 1& 2 NTF’s

Yr 1 New

Teachers

Yr 2 New

Teachers

TLI 2



6

New teachers were also encouraged to enroll in the PDL-designed TLI, where

they could earn, NYU graduate credit, NYC Department of Education “G” credit, or new

teacher credit. There were three basic TLI’s (TLI 1) offered for first year teachers, and

one advanced TLI (TLI II) intended for second year teachers. All courses were offered at

three of CSD 20’s middle schools, in fifteen two-hour sessions over the 2002-2003

academic year. The goals of the basic TLI programs were stated in the course outline as

to:

• Assist the new teacher in developing the requisite skills to become an effective

classroom teacher;

• Enhance the new teacher’s ability to assume a leadership role as a learner and

teacher; and

• Enhance the new teacher’s ability to use collaboration with colleagues to analyze

and solve problems.

The goals of TLI II incorporated and expanded those of the basic TLI course:

• Support and coach teachers in providing meaningful and sustained assistance to

improve teaching and learning;

• Stimulate discussion and dialogue in order to deepen the understanding of how

students think and learn;

• Actively design programs of instruction to increase student engagement in

learning;

• Utilize ongoing assessments, rubrics and portfolios to diagnose and assess student

performance; and

• Provide teachers with resources to select goals, determine improvement methods

and monitor student progress.


The research literature on effective professional development indicates that

traditional in-service models of professional development are generally unsuccessful

(Fickel, 2002; Corcoran, 1995). Professional development programs that have been

shown to have a positive impact on teachers are sustained over a substantial period of

time, involve uninterrupted day-long commitments with release time for staff, and

opportunities to use and reflect on new knowledge and skills (Fickel, 2002; Guskey,

2002; Guskey, 1998).

The PDL for Middle School Social Studies Teachers program was a research-

based program designed to help CSD 20 social studies teachers develop professional

competency in planning lessons, understanding student development and analyzing

student work, developing appropriate instructional strategies, and integrating technology

in the social studies classroom. Figure 3 displays the program model as depicted by the

organization of program components. To achieve its goals, the program provided a series

of Social Studies Institutes for middle school social studies teachers.



7

Designed to be implemented over a period of three years, the program also aimed

to develop the leadership potential of middle school social studies teachers by enhancing

their facilitation and communication skills and teaching them the principles of mentoring.

Year two of implementation was designed to train Cohort 1 participants in facilitation

and mentoring principles, so that they could serve as Resident Teachers for Cohort 2. At

the same time, Cohort 2 social studies teachers would receive training in the SSI. Finally,

the third year of implementation was designed so that Cohort 1 would continue to receive

leadership training and Cohort 2 would begin leadership training. Both Cohorts 1 and 2

would open their classrooms as Resident Teachers for Cohort 3, who would also receive

training in the Social Studies Institute (SSI). (See Table 1.)

FIGURE 3

SOCIAL STUDIES INSTITUTE PROGRAM MODEL

Learning Through Practice Leadership Program

The Learning Through Practice Leadership Program (LTPLP) was designed to

develop the leadership potential of selected district teachers and staff developers whose

classrooms exemplified standards-based teaching and learning and best practices, through

ongoing training (Refer to Figure 1). The CSD 20 leadership selected eleven participants

for this program, all of whom also played a role in the PDL for New Teachers program

(all eight NTF’s and five RT’s). The program consisted of seven full-day training

sessions that focused on various leadership strategies, differentiated instruction, multiple

intelligences, using protocols, observation techniques, and using Department of

Education (DOE) data and the GROW Report (DOE diagnostic skills analysis) to guide

instruction.

Social Studies Institutes

♦ Integrating technology

♦ Finding & using primary documents

♦ Integrating literature

♦ Using cooperative learning strategies

Cohort 1 Social Studies Teachers

Students



8

TABLE 1.

PDL FOR MIDDLE SCHOOL SOCIAL STUDIES TEACHERS THREE-YEAR

MODEL

Context

The PDL programs served teachers in nine middle schools located in CSD 20,

which encompasses the neighborhoods of Bay Ridge, Bensonhurst, Dyker Heights,

Kensington, Flatbush, and Bath Beach in Brooklyn, New York. The district’s population

is varied linguistically and ethnically with over 60 languages spoken in the homes of the

students in those neighborhoods. Although diverse, the ethnic distribution of CSD 20

students differs from that for the city overall. Based on the 2001 – 2002 Annual District

Report (NYC Department of Education, 2003), almost 37 percent of CSD 20’s students

are White, 5 percent Black, 25 percent Hispanic, and 33 percent Asian/Pacific Islander,

compared to 15 percent, 33 percent, 39 percent and 12 percent for the city overall.

Almost 13 percent of the students arrived in the United States within the last three years,

compared to 7 percent for the city, and more than 20 percent are limited in English

proficiency, compared to 13 percent for the city. More than 70 percent are eligible for

free lunch, which is about the same as the 73 percent citywide. The teaching staff in

CSD 20 tends to be slightly more experienced and educated than the faculty citywide.

Nevertheless, 14 percent of the teachers were provisionally appointed, 45 percent had less

than 5 years total teaching experience, and 34 percent had no more than two years

teaching experience in their schools. The student population is large, with almost thirty-

thousand students distributed among the district’s 30 schools, and the schools tend to be

crowded, serving 107.5 percent of capacity, compared to 91.9 percent citywide. CSD 20

has consistently outperformed the city overall on standardized tests in English Language

Arts (ELA) and mathematics. For the three years between 2000 and 2002, approximately

half of CSD 20 students in grades 3 thru 8 met grade-level performance standards on

state and city tests in ELA and mathematics, compared to less than 40 percent for the city

overall. However, there has been a steep decline in test performance—steeper than that

for the city overall—between grade 4 and grade 8. In 2002, 59 percent of CSD 20 4th

graders, but only 34 percent of 8th

graders, met state standards on the state ELA test.

CSD 20’s 4th

grade performance was 12 percentage points higher than the city’s, but CSD

20 held only a 3 percentage point advantage in grade 8. A similar pattern was displayed

in mathematics with the percentage of students meeting state standards declining from 73

percent in grade 4 to 41 percent in grade 8, and a 21 percentage point advantage over the

city in grade 4 shrinking to 11 percentage points in grade 8. The declining standardized

Cohort 1 Cohort 2 Cohort 3

Year 1

(2002-2003)

SSI N/A N/A

Year 2

(2003-2004)

Mentoring and Residency

Training (10 Days)

SSI

Cohort 1 Resident Teachers

N/A

Year 3

(2004-2005)

Mentoring & Residency

Training (5 Days)

Mentoring & Residency

Training (10 Days)

SSI

Cohort 1 & 2 Resident

Teachers



9

test performance between grades 4 and 8 signaled a need to consider ways to bolster

teaching and learning in CSD 20’s middle schools. Profiles of each of the nine middle

schools in CSD 20 are presented in Appendix A.

Another contextual feature that may have affected the implementation of PDL

programs during the 2002-03 academic year was the reorganization of the DOE.

Throughout the year, CSD 20 was undergoing transformation as it prepared for the

transition from a semi-independent school district to incorporation in a larger region. The

reorganization had an impact on the district and school climate, leaving many personnel

at the district and school level unsure of their job security. This uncertainty was

exacerbated by the announcement in February 2003 that CSD 20’s superintendent would

be resigning his position in the spring. The reorganization of the DOE also led the

district to rely heavily on NTF’s to fulfill additional administrative responsibilities which

were beyond the scope of their mentoring roles.

III. THE EVALUATION

The Evaluators The evaluation was conducted by the Center for Research on Teaching and

Learning (CRTL). CRTL was organized in February, 2002 and functions within the

organization of the Department of Teaching and Learning of New York University’s

Steinhardt School of Education. CRTL’s mission is to support the department in the

development of teacher education and professional development programs that prepare

highly qualified educators to work with diverse student populations. CRTL fulfills its

mission by conducting evaluations of the department’s pre-service and in-service

programs, establishing systems of accountability, and supporting research done by

department faculty and students. Five CRTL staff members conducted this evaluation as

follows: The director developed the evaluation design and oversaw all aspects of its

implementation; a CRTL graduate research assistant served as the principal investigator

with responsibility for designing the instruments, scheduling and conducting site visits

and interviews, administering the instruments, and collecting all evaluation data; two

CRTL graduate student workers coded, key entered, and prepared data for analysis; and a

CRTL research scientist conducted all data analyses. CRTL staff members were

supported by two adjunct faculty members from the Department of Teaching and

Learning who conducted structured observations in the classrooms of a sample of

teachers served by the program.

Evaluation Questions The evaluation was designed to track the flow of services across the program

models, as depicted in Figures 1 and 2 above, documenting program implementation and

assessing its impact for each model component.

In this way, for the PDL for New Teachers program, the evaluation traced and analyzed

the path of program effects as they rippled through the model from the training of NTF’s

and RT’s by program and district staff, through the mentoring of new teachers by NTF’s



10

and the modeling of teaching techniques by RT’s, to the impact of new teacher training

upon the teaching and learning process and, consequently, student achievement. A

similar design was used to document and assess the effects of the PDL Program for

Middle School Social Studies Teachers. The evaluation was designed to address the

following questions about the effectiveness of the PDL program models at CSD 20:

1. How well did PDL prepare NTF’s and RT’s for their mentoring and modeling

roles, respectively, in the new teacher program?

2. In the perception of new teachers, to what extent did program services address

their most important staff professional development and support needs?

3. To what extent did new teachers believe that the program facilitated their

professional growth in the competencies of effective teachers?

4. Did new teachers served by the program show an increase in their sense of

efficacy as teachers?

5. To what extent did new teachers served by the program demonstrate the

professional competencies of effective classroom teachers?

6. Were the retention rates of new teachers served by the program better than new

teachers in comparable schools that were not?

7. Did students of new teachers served by the program show gains in ELA

achievement test scores that were equal to those for all CSD 20 middle school

students?

The evaluation also addressed Evaluation Questions 3, 4, 5, and 7 for the teachers

who participated in the PDL for Middle School Social Studies Teachers program, with

performance on the grade 8 state examination in social studies added as data for

Evaluation Question 7.

Methods of Data Collection

The evaluation used a variety of methods to collect qualitative and quantitative

data on the programs’ components, including interviews, observations, self-report

questionnaires, affective surveys, human resources data, and student achievement test

scores. The data that were collected are described below.

New Teacher Staff Training Observations

CRTL staff conducted several observations of training sessions held for NTF’s

and RT’s. CRTL staff attended three of the five LTPLP training sessions and two NTF

Author Study Exchanges to collect ethnographic data. These data were used to address

Evaluation Question 1.

NTF and RT Questionnaires and Interviews

All participants in the LTPLP program—NTF’s and RT’s— were asked to fill out

a questionnaire which asked them to discuss the impact of all the PDL training they

received on their leadership skills and on their work with new teachers. Four of the



11

participants—three NTF’s and one RT—also agreed to participate in a focus group

regarding the same. However, due to scheduling conflicts, individual interviews were

conducted with each of the four volunteers at their respective schools. Questions were

taken from the questionnaire and expanded with follow-up questions. The four

volunteers were compensated for their participation in the interview. The data from these

questionnaires and interviews were used to assess how well PDL prepared the NTF’s and

RT’s to work with new teachers as mentors and models, respectively (Evaluation

Question 1).

New Teacher Surveys

In May, NTF’s were asked to distribute three surveys to their interns—the term

used to describe the new teacher facilitators’ mentoring relationship with the new

teachers. The first was a Participation Information Form that asked new teachers to

provide basic demographic information that would be used for descriptive analysis. The

second, the Professional Development Activities Questionnaire, asked teachers to identify

the support they received through the program in a number of professional development

areas, and to rate the level of importance of each. The list of support areas was

developed by CRTL evaluators to reflect the key professional issues that concern new

teachers in their first year of teaching. To develop this survey, CRTL evaluators

interviewed new teachers in selected non-program public schools in New York City.

CRTL evaluators compiled a list of areas of concern that were mentioned during the

interviews. The list was culled to a final set of issues that represented the most

frequently-mentioned areas of concern. This survey was used to assess the extent to

which new teachers perceived that the program addressed their most important

professional development needs (Evaluation Question 2). The third survey, the Elements

of Effective Classrooms and Instruction, asked teachers to rate their current level of

development as a teacher in twenty-seven areas, and the extent to which they had

received mentoring in those areas. The elements were based on the characteristics of

effective teachers as presented in the work of Charlotte Danielson (1996). The twenty-

seven areas were organized into four distinct domains—Planning and Preparation,

Classroom Environment, Professional Responsibility, and Classroom Instruction. The

data from the third survey were used to assess the extent to which the new teachers

showed developing expertise in the areas in which they received program support

(Evaluation Question 3). All of the above surveys were individually addressed in manila

envelopes, and were returned directly to CRTL via business-reply envelopes. Some of

these surveys were mailed directly to the teachers at their schools, while others were

delivered to the teachers by their NTF’s.

Individual Performance Assessment Logs (IPAL’s)

The IPAL’s are internal program documents used by the NTF’s to track the

weekly progress of their interns. The form is used to specify areas of strength and

weakness, and calls for the NTF and the intern to identify subsequent activities to support

the new teacher’s growth. The form is also used to document the new teacher’s

fulfillment of the NYS mentoring requirement.



12

Teacher Efficacy Scale

New teachers’ sense of their effectiveness as teachers (Evaluation Question 4)

was assessed with the Teacher Efficacy Scale (Gibson and Dembo, 1984). The TES was

developed to measure a teacher’s sense of efficacy and its relationship to teacher

behavior and student achievement. The original scale consisted of thirty items rated

on a six point Likert-type scale. However, a factor analysis indicated that two substantial

factors emerged from only sixteen of the items. The first factor was labeled personal

teacher efficacy because the items relate to the individual teacher’s belief that her actions

have an impact on student behavior and learning. The second factor, or teacher efficacy,

relates to the belief that any teacher’s ability to influence student behavior and/or learning

is limited by external elements such as family and home environment. In fall 2002, each

NTF was given copies of the TES to distribute to all of their interns. NTF’s were also

given postage-paid reply envelopes for teachers to return the surveys directly to CRTL.

In spring 2003, NTF’s were provided with the names of teachers who had returned the

TES and were asked to follow up with those teachers who had not yet returned them.

NTF’s were provided with additional surveys and postage-paid envelopes. Teachers who

filled out the TES prior to March were mailed a post-test TES in May. The TES was also

redistributed to those teachers who failed to complete it during the first round of

distribution. The TES was also administered pre and post to all teachers in the PDL

Middle Schools Social Studies Program.

New Teacher Classroom Observations

New teachers were recruited by their NTF’s to participate in the observation

phase of the evaluation. Two (one pre and the other a follow-up) observations were

scheduled with all 13 volunteers. The pre-observation was conducted on all 13; due to

scheduling conflicts, only ten follow-up observations were conducted. Two NYU

adjunct faculty members, who serve as student teacher supervisors, were hired as

consultants to conduct the observations, using the Domain Referenced Teacher

Observation (DRTO) protocol (adapted from Danielson, 1996). The consultants were

trained by a CRTL staff member who underwent one full day of training with the

DRTO’s creator, Charlotte Danielson. The consultants were trained over a period of

three days, during which they viewed un-narrated videos depicting classroom instruction.

After viewing each video, the consultants used the DRTO rubric to individually rate the

instruction. Areas of discrepancy were discussed and resolved before moving to the next

video. Training proceeded until the raters agreed on 90% of the ratings. During the

course of training, it became evident that the ratings of one of the consultants who was

originally recruited to conduct observations, was consistently in the outlier range.

Accordingly, this consultant was not employed in the operational study.

The protocol consisted of twenty-nine items organized into four domains:

Planning and Preparation, Classroom Environment, Professional Responsibility, and

Classroom Instruction. The consultants rated the teachers’ proficiency in each area.

Twenty-seven items on the DRTO were matched with the items on the aforementioned



13

Elements of Effective Classrooms and Instruction. At the time of the first observation,

teachers were asked to collect three samples of typical assignments from three students

who varied in their level of work, with one approaching state standards, one meeting the

standards, and the other exceeding them. Participants in this phase of the research were

compensated for each observation they completed. Two of the observed teachers also

participated in the PDL Middle School Social Studies Teachers program. Data from the

observations were used to assess the extent to which new teachers served by the program

demonstrated the professional competencies of effective classroom teachers (Evaluation

Question 5).

Social Studies Institute Feedback Questionnaire

The SSI Feedback Questionnaire consisted of ten items designed to determine if

the goals of the program, as stated by the Institute’s facilitators, had been met. First,

teachers were asked to rate the extent of professional development in seven areas that

were identified in the program’s literature. For those same areas, teachers were also

asked to rate the extent of the influence of the professional development on their practice.

The questionnaire also included three open-ended questions designed to elicit specific

examples of the skills participants learned in the program. (Evaluation Question 3B)

Teacher Retention Data

Data on teacher retention were obtained from the Division of Assessment and

Accountability (DAA) of the New York City Department of Education (NYC DOE) and

project staff. The DAA data files were extracted from the NYC DOE’s human resources

electronic record system and contained information on all teachers in the New York City

public schools for fall 2000, fall 2001, and fall 2002. For each year, the files contained

information on each teacher’s school, subjects taught, number of years of experience in

their current schools, and total years of experience in education. Using encrypted

identification numbers, the files for the three years were matched. Using SPSS

programming, flags were applied to each matched teacher record to indicate their status

in 2001 and 2002 as follows: continued employment in the same school, continued

employment in NYC DOE but in a different school, and left NYCDOE. Project staff

determined the retention status of the new teachers served by PDL in 2002-2003 through

telephone interviews with the staff of CSD 20 middle schools. These data were sorted

into the same three status categories.

Standardized Achievement Test Scores

To address Evaluation Question 7, the standardized achievement test scores of

students served by the PDL-trained teachers were compared to those of students served

by more experienced teachers in the same schools. The Division of Assessment and

Accountability prepared two electronic files of standardized test files for CSD 20

students. The first file contained the matched 2002 and 2003 state and city English

language arts (ELA) test scores for every CSD 20 student on register in grades 6 – 8 in

2003. This file was used to compare the gains in ELA scale scores obtained by students



14

of teachers in the PDL for New Teachers program with the scale-score gains of students

in the same schools who were not served by the program. The second file contained the

2003 Grade 8 State Social Studies Examination scores for all CSD 20 students matched

with each student’s 2002 ELA test scores. The social studies test scale scores for

students of teachers served by the PDL for Middle Schools Social Studies Teachers

program were compared to the scores of students of other social studies teachers in the

same schools. ELA scores were used as a covariate in this analysis to control for pre-

program differences in the general achievement levels of the two groups of students.

Assumptions and Limitations The evaluation was conducted in the context of the ongoing implementation of the

program in its natural context using a variety of measurement techniques, including

interviews, observations, record reviews, questionnaires, surveys, and student test data.

The evaluators have no reason to believe that the normal assumptions of naturalistic

research have been violated in this study. Specifically, the evaluation assumes that:

• the responses of the subjects to all instruments were candid and reflected their

actual beliefs and understandings;

• the activities observed by the evaluators were representative of normal program

implementation and were not contrived for the evaluation; and

• the data used to describe the program and assess its effects are based, in whole or

in part, on the activities of the PDL program.

In addition to the general assumptions that apply to naturalistic research, the

reader is cautioned to be mindful of certain limitations that are specific to the

circumstances of this evaluation.

• The instruments were developed or adapted specifically for the purpose of this

evaluation. As such, there was limited time to pilot these instruments and assess

their validity and reliability.

• The period of time between pre- and post-administration of the instruments was

relatively brief. Ideally, we would have administered pre-tests prior to or closer

to the start of the intervention.

• While the return rate on some of the surveys was reasonably high, we would have

preferred a greater response rate on others. We also would have liked a more

consistent response from the participants on all surveys, particularly those

teachers who took part in the observation phase of the research.

• Student work collected from teachers was not consistent with some handing in

tests and others projects. Also in some cases teachers only submitted work for

one student, not three. In short, we were unable to use the student work in any

substantive way. Better planning in the collection and assessment of student work

in ways that would permit the teachers to better understand how to use the work

for evaluation and instruction should be explored in future evaluations.



15

• The evaluation focused on the effects of one year of program implementation.

The true magnitude of the effects of this program can only be assessed in

longitudinal follow-up, which is beyond the scope of the evaluation.

• Standardized test data could not be matched to all of the teachers served by the

programs. The evaluators believe the resultant attrition of subjects was random

and did not substantially bias the internal validity of the analyses for Evaluation

Question 7.

Although the above assumptions and limitations warrant some caution, the

evaluators believe that the findings from this evaluation provide a fair and reasonable

assessment of the program and its effects.



16

IV. FINDINGS

This section describes the results of the 2002-03 PDL evaluation. The section

begins with a description of the new teachers who were targeted for mentoring and

professional support by the PDL for New Teachers program. The section continues with

the presentation of the findings organized by the study’s evaluation questions.

New Teacher Participants The PDL for New Teachers program served a total of 67 new teachers in CSD 20

during the 2002 – 2003 school year. There were large differences in the numbers of new

teachers served among the district’s middle schools. The numbers ranged from as few as

2 (3.0% of all participants) new teachers in two of the schools to as many as 14 (20.9%)

in one school. (See Table 1.) Since each school was served by one NTF, there was a

wide range in caseload among the program’s NTF’s. It should be pointed out that the

several NTF’s assumed other functions in the school such as reading specialists, assistant

principal, staff developer, etc.

Consistent with the program’s proposed target population, the new teacher interns

were young and inexperienced. Their median age was 28, and their median total

experience in education was 2 years. All were in their first year in the school in which

they were served. About one quarter had earned their master’s degrees, and 67% had

only a bachelor’s degree. (See Table 2.)

TABLE 2

DISTRIBUTION OF NEW TEACHER PROGRAM INTERNS BY SCHOOL

School N of New Teachers % of Participants

A 2 3.0%

B 10 14.9%

C 6 9.0%

D 11 16.4%

E 9 13.4%

F 14 20.9%

G 2 3.0%

H 13 19.4%

TOTAL 67 100%



17

TABLE 3

DESCRIPTIVE CHARACTERISTICS OF NEW TEACHER PROGRAM

INTERNS

Characteristic Median/%

Median Age 28 Years

Median Yrs Teaching Anywhere 2 Years

Median Yrs in Current School 1 Year

% Bachelor's Degree 67.0%

% Master's Degree 24.7%

The new teachers taught a wide range of subjects in the 2002 – 2003 school year.

The most frequently taught subjects were math and science and English/literature, 25.9 %

of respondents each. (See Table 3.)

TABLE 4

SUBJECTS TAUGHT BY NEW TEACHER INTERNS

Subjects Taught N % of Respondents

Math/Science 7 25.9%

Social Sciences 3 11.1%

Arts/Electives 5 18.5%

English/Literature 7 25.9%

Special Education/ Bilingual 4 14.8%

Other 1 3.7%

TOTAL 27 99.9%

Social Studies Teacher Participants

Twenty Social Studies teachers from eight of nine middle schools in CSD 20 were

participants in the 2002-2003 PDL for Middle School Social Studies Teachers program2.

There were large differences in the number of years of experience among program

participants, ranging from 1 to 22 years, with a mean of 6.6 years and a median of 4

years. (See Table 5.) Five of the participating schools had 3 participants, and one school

had 1.

2 One participant withdrew from the program due to an injury she sustained in the line of duty.



18

TABLE 5

DISTRIBUTION OF SOCIAL STUDIES PROGRAM PARTICIPANTS BY

SCHOOL

School

N of SS

Teachers

% of

Participants

A 1 5%

B 3 15%

C 3 15%

D 3 15%

E 3 15%

F 3 15%

G 2 10%

H 2 10%

Total 20 100%

Evaluation Question 1. How well did PDL prepare NTF’s and RT’s for

their mentoring and modeling roles, respectively, in the new teacher

program? During the 2002-2003 academic year, there were a total of eight NTF’s in eight of

nine middle schools in CSD 20, with a range of two to fourteen interns assigned to each.

Three of the NTF’s were selected and trained in the fall of 2001, and were reappointed in

2002-2003. The remaining five NTF’s were selected in September and October, with

two selected in late November and December 2002.

Topics covered during the weekly meetings included the stages of new teacher

development, understanding the role of the mentor, using the INTASC standards to

formulate instructional strategies, using the IPAL's, learning and practicing mentoring

strategies, and using protocols to facilitate meetings.

The data from interviews, internal evaluation forms and a CRTL questionnaire

suggested that NTF’s believed the training they received from the NTF Institutes, the

weekly NTF meetings, and the Author Study Exchanges enhanced their leadership

abilities, with respect to their listening, communication and facilitation skills. A number

of NTF’s commented on the usefulness of the various protocols that were used during the

training sessions to facilitate their own meetings with interns. The first year NTF’s who

participated in the weekly meetings also suggested that their understanding of new

teacher development increased as a result of the extensive work they did with the

INTASC standards.

While the data indicates that PDL training effectively prepared the NTF’s to

mentor their new teacher interns, some data suggest obstacles at the schools that may

have made their job more difficult. One concern was that while CSD 20 was committed

to the goals of PDL, several NTF’s believed that this commitment was not always shared

by the school principals. NTF’s commented that in some cases, new teachers were



19

assigned emergency coverages that often interfered with the mentoring schedule. In at

least one school, the NTF was also assigned coverages. Additionally, the NTF’s reported

that they were often pulled away from their mentoring responsibilities in order to assist

with grading and/or other administrative duties. This was especially true of the six

NTF’s who were district reading specialists. Several NTF’s indicated that their presence

in the school was not formally legitimized by their principals, through introduction or

other measures. Some NTF’s reported that veteran teachers felt threatened by the NTF,

and did not allow them and new teachers to visit their classrooms. At least one NTF

indicated that an antagonistic relationship between teachers and school administrators led

both new and veteran teachers to be wary of her position in the school.

Observations of training sessions also suggested that the NTF’s were provided

with valuable opportunities to learn and practice leadership skills. During one LTPLP

session, NTF’s and RT’s were divided into two groups with each group visiting one of

the participating PDL CSD 20 middle schools. Each group observed a teacher

conducting a lesson using a balanced literacy strategy. After the observations, the session

focused on strategies for both “warm” and “cool” feedback. As the participants practiced

the strategies, a PDL facilitator interjected with suggestions for improving the strategies,

such as using proper nouns rather than pronouns, and making eye contact.

The observations also suggested that the NTF’s were encouraged to transform

theories of mentoring and leadership into practical strategies. This process occurred

through first individual, then group reflection, culminating in the creation of a formal

plan of action. For example, during one Author Study, the teachers were asked to choose

one of the instructional strategies outlined in the book that they preferred. They were

asked to share their choices in small groups. Then, the groups were asked to choose one

of the instructional strategies discussed and create an instructional objective and

strategies they could use with one of their interns.

All NTF’s felt that the various modes of PDL training gave them the vocabulary

and strategies to help them with their interns. They indicated that while test scores might

not have indicated that their work with interns positively impacted students, their efforts

to assist their new teachers in becoming competent and reflective practitioners had a

positive impact on student performance. As one NTF stated, “I don’t have any concrete

evidence [of PDL’s impact on student achievement] but I can say that the students do

well if the teacher is prepared and confident in his/her craft.” Another NTF indicated that

at least one of her intern’s levels of expectations for her students had increased; this had a

positive impact on her students’ performance.

Evaluation Question 2. In the perception of new teachers, to what extent

did program services address their most important professional

development and support needs?

PDAQ



20

In order to determine the extent to which the new teachers perceived the program

as meeting their professional development needs, all new teachers were asked to

complete the Professional Development Activities Questionnaire (PDAQ). Respondents

were asked to answer two sets of questions on the PDAQ. First, respondents used a five-

point Likert-type scale to rate how important it was that they, as new teachers, receive

support in 26 areas of professional development. The scale ranged form “Not Important

at All” to “Very Important”. The second set of questions asked them to check off each

area in which they received support during their first year through the program. Results

of the questionnaire for the 32 teachers who responded are displayed in Table 4. Table 4

shows the percentage of respondents rating each area “Very Important” or “Important”

and the percentage who indicated they received support in those areas from the program.

(For purposes of this discussion these ratings are combined under the label important.)

Overall, all of the areas were rated important by more than half of the

respondents. This is to be expected since the list was constructed through interviews with

new teachers concerning areas of importance to them for first-year professional

development. However, there were considerable differences in the ratings among the

areas. All respondents rated motivating students as important to them and over 93%

percent gave the same rating to classroom management. These data corresponded with

data retrieved from the sample of IPAL’s that indicated NTF’s concentrated their

mentoring in these two areas. (See discussion of IPAL data below.) In addition, 90.6%

of the respondents gave ratings of important to dealing with diverse student needs. In at

least two cases, the sample IPAL’s suggest that this area was being met by the NTF’s.

Overall, 10 areas were rated important by at least 80% of respondents. Conversely, 5

areas, excluding “other”, were rated important by less than 65% of the respondents.

Among these, was classroom and/or school inter-visitation (58%), a significant feature of

the PDL for New Teachers program. When the percentage of respondents who reported

that they received program support was averaged separately for the 10 areas with the

highest percentage of important ratings and the five areas with the lowest percentage of

important ratings, the mean support percentage was meaningfully higher for the former

than the latter. The mean support percentage was 22.0% for the most important areas

versus 14.2% for the least important areas. These findings suggest that the new teacher

interns perceived that the PDL program was focused on their most important needs.

Moreover, each area showed at least some respondents reporting that they received

program services. This suggests that the program addressed a wide range of new teacher

needs and that the services were tailored to the needs of the individual teacher.



21

TABLE 6

NEW TEACHERS’ IMPORTANCE RATINGS AND REPORTS OF PDL

PROGRAM SUPPORT ON THE PROFESSIONAL DEVELOPMENT

ACTIVITIES QUESTIONNAIRE

Percent of New Teachers

(N = 32):

Areas of Importance

Rating Very

Important/

Important

Receiving

Program

Support

1. Motivating students 100.0 14.5

2. Classroom management 93.8 20.3

3. Dealing with diverse learning needs 90.6 21.7

4. Certification 87.5 18.8

5. Curriculum materials for content area/grade level 87.5 26.1

6. Clarification of departmental/school-wide instructional goals 83.9 21.7

7. Observations with feedback 83.9 27.5

8. Support in the content area 83.3 23.2

9. Orientation (including payroll, your rights as a teacher, etc) 81.3 21.7

10. Modeling of classroom practice in a classroom 80.0 24.6

11. Mentoring 75.0 29.0

12. Individual meetings to discuss classroom practice 75.0 31.9

13. Lesson planning/design 75.0 24.6

14. A collegial environment 75.0 13.0

15. Support with curriculum development 71.9 26.1

16. Navigating the Department of Education 68.8 15.9

17. Approaches to assessing student performance 68.8 21.7

18. Materials (paper, markers, rulers, etc) 68.8 15.9

19. Forums for discussing issues/ideas 68.8 13.0

20. Time management 65.6 17.4

21. Setting up your classroom 63.3 11.6

22. Building trust among colleagues 62.5 14.5

23. Inter-visitations to other classrooms/schools 58.1 23.2

24. Avoiding isolation 56.3 13.0

25. Other 55.6 5.8

26. Support groups 51.7 8.7

IPAL’s

Each week, NTF’s were expected to observe their assigned interns and meet with

them afterwards to discuss their observations. The outcomes of the meetings were

recorded on IPAL’s and transmitted to PDL staff who turned them over to DOE

personnel, who in turn awarded the new teachers with credit towards fulfilling their

mentoring requirements. Whenever possible, NTF’s were also expected to take their

interns to observe experienced teachers in their school. Effective strategies and



22

techniques used in those lessons were discussed and incorporated into the interns’

planning.

A review of a sample of IPAL’s revealed a number of common themes. NTF’s

noted that the new teachers were concerned about planning and classroom management.

With respect to planning, teacher concerns and questions referred to curriculum mapping,

pacing the lesson to fit within the class period, and incorporating instructional strategies

(group work, using manipulatives, etc.) that motivated students. Many of the concerns

surrounding classroom management centered on getting students on task and dealing with

students who were disruptive. These themes are consistent with the new teachers’

importance ratings on the PDAQ (See Table 4), where over 93% of teachers indicated

that motivating students and classroom management were important or very important.

The IPAL’s suggest a number of strategies taken by the NTF to assist the new

teachers. Most often, the IPAL suggested that the NTF’s helped the teacher plan lessons,

or they coordinated meetings between the new teacher, the content area staff developer,

and themselves. This was especially evident in the area of curriculum mapping. NTF’s

also team taught lessons that incorporated group activities to demonstrate effective group

management and facilitation. In many cases, the NTF’s scripted new teachers’ lessons to

help them with pacing. These themes were also reflected in the new teachers’ importance

ratings on the PDAQ. 75% of new teachers reported that lesson planning was important

or very important, while 80% indicated that modeling practices was important (See Table

4).

When classroom management was a concern, NTF’s generally observed how the

teacher’s movement around the classroom and instructional strategies impacted student

behavior. The IPAL’s indicate that suggestions for managing classroom behavior often

centered on modification of instructional strategies, such as incorporating more student-

centered and group activities. NTF’s also made suggestions that new teachers make a

more concerted effort to call on withdrawn or misbehaving students, to draw them into

the lesson. In only one of the IPAL’s, the NTF instructed the new teacher to use a

disciplinary approach in managing the behavior of some students. The NTF’s emphasis

on classroom management corresponds to the new teachers’ importance ratings on the

PDAQ, where 93% of the new teachers reported that classroom management was

important or very important (Table 4).

Another common area of concern for the new teachers was preparing their

students for the NYS and NYC examinations. While some teachers in the sample IPAL’s

expressed these concerns in the fall of 2002, this was a consistent area of concern for

most of the teachers in the spring, coincident with the onset of the state and city testing

programs. NTF’s provided teachers with copies of previous state and city examinations,

and worked with the new teachers to identify major concepts that appeared on the exams

frequently. Additionally, two of the NTF’s in the sample collaborated with their new

teachers to use the GROW Report to plan instructional strategies that addressed the

different needs of their students.



23

Interview Data on the Cycle Component

In the fall, the NTF’s were asked to identify new teachers who might benefit

from participating in the Cycle Program. Seven new teachers were identified and

matched with Resident Teachers (RT), who were trained and certified by PDL and CSD

20 as exemplary role models. Wherever possible, new teachers and RT’s were matched

by discipline. The new teachers met with the RT’s, the NTF’s, and the Cycle Program

facilitator during two after school sessions to discuss the program goals and objectives, as

well as the goals of the new teachers. During one of two periods in January and

February, the new teachers visited the RT’s schools over a period of three days. Prior to

the visits, the Cycle Program PDL facilitator, RT’s, and new teachers met individually to

discuss goals and focus on areas that meet the needs of the new teacher. During the first

two days of the visit, the RT modeled pedagogical skills that addressed the goals set

earlier, and the facilitator led a pre and post-conference where the new teacher had the

opportunity to discuss the lesson with the RT. On the third day the new teachers used the

skills learned in the Cycle Program to teach all of the RT’s classes. The new teachers

were expected to return to their home schools and use the newly learned skills with their

own students.

Internal program feedback suggested that most of the new teachers found the

Cycle Program to be beneficial. A number of participants commented that the

opportunity to visit a veteran teacher in his/her own environment was helpful in

demonstrating how strategies, such as group work, using manipulatives, and multiple

intelligences, might work in a classroom. One concern that was raised was that while the

strategies worked for the RT’s, classroom management issues that new teachers were

facing might interfere with the application of these strategies in the classroom. One new

teacher suggested that having the RT visit the classroom to provide feedback on learned

strategies might be useful in overcoming these challenges. One of the RT’s commented

that it would have been beneficial if they had the opportunity to meet with the new

teachers a second time. However, this same RT indicated that frequent disruption in the

VT’s and RT’s teaching schedule, while positive with respect to new teacher induction

and reflective practice, was too costly for the school to support. The RT also stated that

the new teacher’s students would be negatively impacted by the frequent absence

required by an additional Cycle Component.

Evaluation Question 3A. To what extent did new teachers believe that

the program facilitated their professional growth in the competencies of

effective teachers? The Elements of Effective Classrooms and Instruction (EECI) was used to assess

new teacher perceptions of the extent to which the program facilitated their professional

growth in the first year of teaching in their schools. The EECI was comprised of 27

elements associated with effective classroom teachers, clustered in four areas: Planning

and Preparation, the Classroom Environment, Instruction, and Professional

Responsibilities. (See Appendix B.) The new teachers were asked to rate each item with

respect to two different dimensions. First, the teachers rated their degree of development



24

in each element using a 5-point Likert-type scale, ranging from (1) Minimal to (5)

Advanced. Second, they rated the emphasis project mentoring placed on each element

using a 5-point Likert-type scale, ranging from (1) No Emphasis to (5) Very Heavy

Emphasis. Results of the questionnaire for the teachers who responded are displayed in

Table 6. The number of responses for each item varies because all respondents did not

respond to every item. Table 6 shows the items displayed according to the total

percentage of respondents who rated the items at level 4 or 5 for each of the 27 effective

elements. These items are displayed in rank order and are organized by domain. The

means and standard deviations for each item are also shown. Additionally, the table

displays the percentage of respondents who rated the emphasis of program mentoring at

levels 4 or 5, as well as the means and standard deviations for each item.

Of the four domains, respondents indicated that they felt most prepared in the area

of Planning and Preparation, with 56.2% of respondents indicating a level of professional

skill at 4 or 5. Overall, the mean score for program emphasis at level 4 or 5 in this

domain was 55.9%. The domain with the second highest ratings was Classroom

Environment, with a mean score in professional skill at 53.7%, and a mean score in

mentoring emphasis at 57.8%. The mean ratings for professional skill level in each of the

two remaining domains (Instruction and Professional Responsibilities) were each below

50%, and the mean ratings for mentoring emphasis were each below 52%.

In the domain of Planning and Preparation, 60% of respondents rated their skill

level at 4 or 5 in five of the fourteen areas. Those items related to assessment, content,

multicultural and gender issues, and creating appropriate goals. In these same areas, over

65% of the respondents rated the emphasis of the PDL program mentoring at levels 4 or

5. In comparison, only 28.1% of the respondents indicated that they were at a level 4 or 5

in “gaining access to school resources for students.” Similarly, 28.2% indicated that

program emphasis in this area was at level 4 or 5. This indicates a strong relationship

between program emphasis and the level of professional skill teachers reported.

In the Classroom Environment domain, an average of 60.4% of the respondents

rated their level of professional skill at 4 or 5 in three of the six areas, which deal

primarily with managing student behavior and engagement. In these same areas, the

mean emphasis of mentoring received from PDL was 61.4%. In the remaining three

areas, which relate primarily to organization and classroom routines, the mean rating for

level of professional skill at level 4 or 5 was 46.9%. The mean emphasis of PDL training

in these areas was 54.2%. This indicates a moderate relationship between the PDL

program emphasis and the teachers’ self-reported level of professional skill.

No greater than 59% of the respondents rated their professional skill level at 4 or

5 in any of the four items in the domain of Instruction. The same is true for respondents’

ratings of mentoring in the four areas. The four items deal primarily with instructional

strategies that foster student contributions in classroom discussions, group work, and the

learning process. The mean ratings for professional skill and mentoring emphasis in all

four areas were 48.5% and 51.2%, respectively.



25

In the last domain, Professional Responsibilities, at least 50% of the respondents

rated their professional skill level at 4 or 5 in those areas pertaining to being reflective

and working with other teachers (64.5% and 50%, respectively). For those same areas,

the mean rating of program mentoring emphasis at levels 4 or 5 was 54.8% and 46.9%,

respectively. In the area of engaging families, only 15.6% of the respondents rated their

level of professional skill at 4 or 5. 21.9% rated the program emphasis in this area at

level 4 or 5.

To further explore the relationship between program mentoring and the self-

reported professional development of program-served new teachers, evaluators applied

correlation analysis to the EECI data. Table 7 displays a summary of the correlation

analysis between the teachers’ ratings of their level of professional growth in the four

areas of professional development measured by the EECI and the extent of PDL

mentoring they reported receiving in each of those areas. The Pearson product moment

correlation coefficients for all four areas were moderate to moderately high and all were

statistically significant beyond p < .01 for two-tailed tests with 31 degrees of freedom.

Professional responsibility was the area that showed the strongest relationship between

professional growth and extent of mentoring with a coefficient of determination equal to

.483, meaning that nearly one-half of the variance in professional growth was shared by

the extent of mentoring provided by PDL. The extent of the relationship in the other

areas was more modest, with coefficients of determination indicating between one-fifth

and one-quarter shared variance.

Thus, findings from multiple analyses of EECI data suggest that in those areas

where teachers believed their skill level to be advanced, or approaching advanced, they

generally reported PDL support to be strongest. Moreover, even in those areas where

fewer teachers indicated having a high level of professional skill, a comparable

percentage of teachers indicated that they had received heavy or very heavy emphasis of

mentoring. Similar to the PDAQ findings, the results of the EECI also demonstrate the

flexibility of the program model as it adjusts to the needs of the individual new teacher.



26

TABLE 7

MEAN SELF RATINGS ON THE ITEMS OF THE ELEMENTS OF EFFECTIVE CLASSROOMS AND INSTRUCTION Level of Professional Skill Emphasis of Mentoring

Cluster Item# Effective Elements N Mean S.D. % Levels

4 & 5 N Mean S.D.

%

Levels 4

& 5

Planning and Preparation

1 Use of a Variety of Assessment Methods 32 3.84 0.81 71.9 32 3.81 1.12 68.8

2 Knowledge of Content Areas 32 3.88 0.79 68.8 31 3.19 1.05 35.5

3 Establishment of Goals in Level of Expectations, Clarity and Suitability 32 3.81 0.90 65.6 32 3.69 1.18 65.6

4 Multicultural and Gender Issues 32 3.84 0.85 62.5 32 3.31 1.33 43.8

5 Classroom Assessments Aligned w/ Instructional Goals 32 3.69 0.69 62.5 32 3.72 1.02 65.7

6 Lesson Plans 32 3.69 0.93 59.4 32 3.5 1.16 56.2

7 State Content Standards 32 3.84 0.88 59.4 32 3.53 1.08 59.4

8 Different Approaches to Learning 32 3.69 0.69 56.3 32 3.66 1.24 65.7

9 Ability to Adjust Instruction 32 3.78 0.83 53.1 32 3.66 1.12 59.4

10 Classroom Assessment Criteria and Standards 32 3.63 0.91 53.1 32 3.69 1.15 68.8

11 Pedagogical Practices 31 3.47 1.07 51.6 31 3.16 1.19 65.5

12 Establishment of Goals for Different Types of Learning 32 3.56 0.91 50.0 32 3.53 1.14 56.3

13 Knowledge of All Resources Available through School 32 3.28 1.22 43.8 32 3.16 1.22 43.8

14 Knowledge of Gaining Access to School Resources for Students 32 2.81 1.25 28.1 32 2.81 1.09 28.2

Classroom Environment

15 Being Alert to Student Behavior 32 3.88 0.87 62.5 32 3.72 1.2 68.7

16 Polite and Respectful Classroom Environment 32 3.84 1.08 59.4 32 3.37 1.39 50

17 Students Generally Engaged 32 3.75 0.95 59.4 32 3.63 1.24 65.6

18 Smooth Transitions 32 3.44 1.01 46.9 32 3.47 1.29 59.4

19 Routines for Handling Materials and Supplies 32 3.5 1.08 46.9 32 3.44 1.32 46.9

20 Standards of Conduct are Clear 32 3.55 1.16 46.9 32 3.56 1.27 56.2

Instruction

21 High Quality Questions 32 3.88 0.98 59.4 32 3.67 1.19 59.4

22 Creation of Appropriate Instructional Groups 32 3.56 0.98 53.1 32 3.44 1.11 53.1

23 Ability to Have Students Assume Responsibility for Discussions 32 3.47 0.98 46.9 32 3.41 1.21 46.9

24 Creation of a System for Maintaining Info on Student Progress 32 3.25 1.08 34.4 31 3.16 1.24 45.2

Prof Responsibilities

25 Being a Reflective Practitioner 31 3.81 0.79 64.5 31 3.42 1.23 54.8

26 Ability to Work as Part of a Team w/ other Teachers 32 3.47 1.19 50.0 32 3.03 1.38 46.9

27 Ability to Engage Families 32 2.56 1.16 15.6 32 2.72 1.25 21.9



27

TABLE 8

CORRELATIONS BETWEEN NEW TEACHERS’ SELF-REPORTED DEGREE OF

PROFESSIONAL SKILL AND THE EXTENT OF PROGRAM MENTORING IN FOUR

ESSENTIAL AREAS OF EFFECTIVE CLASSROOM TEACHERS

Evaluation Question 3B. To what extent did the social studies teachers believe

that the program facilitated their professional growth in the competencies of

effective teachers? CSD 20 leadership, in collaboration with the eight participating middle school principals,

selected twenty teachers to participate in the PDL for Middle School Social Studies Teachers

program. Participants were given release time to attend eight SSI’s, facilitated by PDL and

District staff. During the SSI’s, two facilitators modeled instructional strategies that

incorporated literature and literacy, cooperative learning, and simulation activities. They also

used protocols to look at the participants’ student work. The sessions also focused on the use of

technology as both a planning and instructional tool.

A review of a CRTL questionnaire, as well as PDL’s internal program feedback forms,

suggests that participants believed this program was useful to them in a number of areas,

particularly in integrating the use of technology in the classroom as a learning tool and as a

resource for students. Teachers indicated that the extent of professional development in all of the

program areas, including developing professional competency in planning lessons, understanding

student development and analyzing student work, developing appropriate instructional strategies,

and integrating technology in the social studies classroom, was at least moderate, with several

indicating that the emphasis was large. Teachers commented that the SSI’s gave them the skills

and the opportunities to access resources, such as primary source documents and activities on the

internet. Teachers found this skill particularly useful in integrating the use of document-based

questions in their instruction. Teachers also noted their increased confidence in using literature

Area of

Professional

Development

Correlation

Coefficient

r2

Coefficient of

Determination

N Level of

Significance

Planning and

Preparation

.439 .193 32 .012

Classroom

Environment

.471 .222 32 .006

Professional

Responsibility

.695 .483 32 .001

Classroom

Instruction

.484 .234 32 .005



28

in their classrooms, as well as sharing ideas and materials with their colleagues. Most

participants also commented that the time given to them for planning was extremely useful. All

respondents indicated that the professional development had at least a moderate influence on

their practice, with one participant indicating a large influence in all but one of the areas.

Many of the participants’ perceptions of the SSI’s are best demonstrated through one

teacher’s comment about the program: “I feel it gave me a much needed boost in the

classroom—the mundane was replaced by a new outlook on teaching.”

Evaluation Question 4A. Did new teachers served by the program show an


Items on the TES that related to the Personal Teacher Efficacy (PTE) factor dealt

primarily with the individual teacher’s impact on his/her students. They imply that if the

respondent makes a concerted effort, it will influence the academic achievement of his/her

students. The seven items relating to the General Teacher Efficacy (GTE) factor imply that,

regardless of any teacher’s effort, home environment and parental/family background dictate the

academic achievement and behavior of students.

The statements are generally worded to demonstrate the limited impact that any teacher

in general may have on students. The mean score of those items on the TES pre-test (see Table

8) that related to PTE were all above level 4 (agree slightly more than disagree).

Four of the items had a mean score above 4.6, with item five showing the highest mean

score of 4.75. Conversely, only three of the items related to GTE had a mean score over 4.0.

There were no statistically significant differences in mean pre- and post-test scores for PTE or

GTE.



29

TABLE 9

MEAN SELF RATINGS ON THE TEACHER EFFICACY SCALE (NEW TEACHERS)

Item # N Mean S.D.

Pre Test

Personal Teacher Efficacy

1 48 4.61 1.26

5 48 4.75 1.14

6 48 4.23 1.13

7 48 4.73 1.05

9 48 4.48 1.11

10 47 4.49 1.02

12 48 4.48 1.01

13 47 4.66 1.34

15 48 4.13 1.10

General Teacher Efficacy

2 46 3.15 1.59

3 48 2.92 1.51

4 48 4.00 1.60

8 47 3.32 1.53

11 47 4.72 1.21

14 48 4.13 1.39

16 46 3.96 1.60

Post Test


1 14 4.86 0.66

5 14 4.86 1.23

6 14 4.36 1.15

7 14 5.00 1.30

9 14 4.93 0.83

10 14 4.86 1.10

12 14 4.43 1.50

13 13 4.85 1.07

15 13 4.77 0.83


2 14 3.93 1.14

3 14 2.86 1.23

4 14 4.86 1.46

8 14 3.86 1.75

11 14 5.29 0.91

14 13 3.69 1.80

16 13 4.84 0.90



30

Evaluation Question 4B. Did Social Studies teachers served by the program

show an increase in their sense of efficacy as a teacher?

The TES pre-test was administered to participants of the PDL for Middle School Social

Studies program during their December 2002 meeting. Table 9 displays the mean scores and

standard deviations for each item. The items are grouped by the two factors measured on the

TES, personal teacher efficacy (PTE), and general teacher efficacy (GTE). The average of the

mean scores of those items related to PTE is 4.07 (agree slightly more than disagree).

Conversely, the average of the mean scores of the items related to GTE is 3.31 (disagree slightly

more than agree). In two of the items related to GTE, the mean scores were between 2.21 and

2.63, or moderately disagree.

Four of the nineteen program participants who completed the TES pre-test completed the

post-test. While no statistically significant findings emerged from the matched pre and post-tests

of the four participants, on average the participants did show at least a 1 point increase in their

sense of efficacy scores on four of the items. On items 4 and 11, which relate to GTE, the

participants’ efficacy scores increased by 1.25 and 1.00. Participants showed an increase of 1.25

on item 7, and 1.50 on item 15, both of which measured PTE.

TABLE 10

MEAN SELF-RATINGS ON THE TEACHER EFFICACY SCALE PRE-TEST

(S.S. TEACHERS)

Item # N Mean S.D.


1 18 3.28 1.23

5 19 4.79 0.79

6 19 3.37 1.01

7 19 4.37 1.38

9 19 3.74 1.24

10 19 3.84 1.17

12 19 3.95 0.97

13 19 5.00 0.88

15 18 4.33 1.09


2 18 4.06 1.21

3 19 2.63 1.61

4 19 4.16 1.46

8 19 3.63 1.38

11 19 2.21 0.98

14 19 3.89 1.20

16 19 2.58 0.96



31

Evaluation Question 5A. To what extent did new teachers served by the

program demonstrate the professional competencies of effective classroom

teachers?

The DRTO was used by CRTL consultants who conducted structured observations in the

classrooms of samples of teachers served by the program. The DRTO was comprised of 29

elements associated with effective classroom teachers, clustered in four areas: Planning and

Preparation, the Classroom Environment, Instruction, and Professional Responsibilities. (See

Appendix B.) The observers rated the teachers on a continuum, which included Not Proficient,

Partially Proficient, Approaching Proficient, Proficient, and Distinguished. Table 10 shows the

mean scores of the observations broken down into the four domains, with observations 1 and 2

displayed separately. The standard deviations are also shown. The analysis employed pair-wise

deletion, therefore only those cases with pre and post observations were included. This explains

why the N varies from one domain to the next.

Table 10 shows that on average, teachers were rated as “partially proficient” in the

domains of Planning and Preparation and Classroom Environment, and as “approaching

proficient” in the domains of Instruction and Professional Responsibilities during the first

observation. During the second observation, the mean scores increased slightly in all four

domains.

TABLE 11

MEAN OBSERVATION RATINGS ON THE DOMAIN REFERENCED TEACHER

OBSERVATION TOOL

Observation 1 Observation 2

DOMAIN N Mean S.D. N Mean S.D.

Planning and Preparation 11 2.89 0.64 11 3.00 1.01

Classroom Environment 11 2.90 0.72 11 3.06 0.99

Instruction 10 3.08 0.78 10 3.16 0.98

Professional Responsibilities 10 3.07 0.72 10 3.17 0.85

Table 10 shows that on average, teachers were rated as “partially proficient” in the

domains of Planning and Preparation and Classroom Environment, and as “approaching

proficient” in the domains of Instruction and Professional Responsibilities during the first

observation. During the second observation, the mean scores increased slightly in all four

domains.

There were six items on the DRTO where the mean scores increased notably between the

first and second observations. Four of the items (5, 9, 10, 11), were in the Planning and

Preparation domain. In item 5, which pertained to teacher’s knowledge of students’ varied

approaches to learning, the mean scores increased .57 points on the second observation from

2.45. Item 9, which was concerned with selecting balanced instructional goals, increased .55



32

points on the second observation from a mean score of 2.50 on the first observation. The mean

score on item 10, which related to finding resources for teaching, increased from 2.73 on the first

observation to 3.27 on the second. Item 11 dealt with teachers’ knowledge of resources available

for students. The mean observation scores on item 11 increased .30 points from 2.90. Item 21,

which showed an increase of .30 points from an original mean score of 3.00, referred to

monitoring student behavior. The item was in the Classroom Environment domain. The last

area that showed a notable increase in mean scores was item 25, “grouping of students.” This

item, under the Instruction domain, increased from 3.31 in the first observation to 3.75 in the

second. Four of these six items showed sampled teachers moving from a classification of

“partially proficient” to “approaching proficient” in the professional competencies of effective

teachers.

The mean scores increased slightly during the second observation, but not significantly.

Overall, the changes in mean scores were in the expected direction (positive), and it is likely that

with a larger sample, the difference might have been greater. Similarly, the results suggest that

conducting the second observation after a longer period of intervention might also have yielded

greater differences in the mean scores.

Evaluation Question 5B. To what extent did social studies teachers served by

the program demonstrate the professional competencies of effective classroom

teachers?

The sample of social studies teachers with both pre and post-observations was too small

(4 cases) for statistical analysis. However, a visual inspection of the data collected on the DRTO

(See Appendix B) shows that in three of four cases the scores generally remained constant or

increased. One teacher’s ratings increased from the level of “approaching proficient” to

“proficient” in seven areas, including knowledge of students’ approaches to learning and skills,

selecting instructional goals, discussion techniques, and instructional groups. Another teacher’s

ratings increased by .5 in four areas, and by 1 point in another four areas. In three of those areas

where the teacher increased most, the ratings went from “partially proficient” to “approaching

proficient.” Those areas included knowledge of content, and knowledge of students’ approaches

to learning and skills. Conversely, another teacher’s ratings decreased in thirteen of the twenty-

nine areas. In some areas, the teacher’s ratings decreased two points from “proficient” to

“partially proficient.”

The results of the DRTO for the sample of Social Studies teachers suggest that there were

a number of limitations in data collection. The sample of teachers with both pre and post-

observation data was extremely low. A larger sample, as well as a longer period of time between

the two observations may have yielded more substantive data.



33

Evaluation Question 6. Were the retention rates of new teachers served by the

program better than new teachers in comparable schools that were not?

The impact of PDL on the retention rates of new teachers was looked at in two ways.

First, the new-teacher retention rates for PDL schools were compared for the years prior to

program service and during program service. Second, the changes in retention rates for PDL

schools were compared to those for comparison schools during the years that the former were

being served by the program. The comparison schools were all middle schools in two

neighboring community school districts that served populations that were demographically

similar to CSD 20.

For purposes of this analysis, the PDL schools were divided into two cohorts. Cohort 1

included the three CSD 20 schools that began PDL in the 2001 – 2002 school year. Cohort 2

included the five CSD 20 schools that were first served in 2002 – 2003. Using the human

resources data files provided by the NYC DOE, the evaluators identified all teachers in each

cohort who were new to the profession and required mentoring as of September 2000 and

September 2001, separately. A new teacher was operationally defined as holding a Preparatory

Provisional Teaching license (PPT) and having less than one year of total teaching experience.

The evaluators identified new teachers meeting the same criteria in the comparison schools for

the same time periods. Next, the status of the identified new teachers was determined one year

later; that is September 2001 for the September 2000 new teachers and September 2002 for the

September 2001 new teachers. There were three status categories for this one year follow-up:

teachers who stayed in the same school, teachers who stayed in the NYC DOE but transferred to

a new school, and those who left the NYC DOE. The data collected by project staff for PDL-

served new teachers in 2002 – 2003 were similarly categorized. These data were not available

for the comparison schools at the time this analysis was conducted.

Table 12 summarizes the three-year analysis of new teacher retention data for the two

PDL cohorts and the comparison schools. To facilitate the interpretation of the data in Table 12,

the following explanation is offered. The first three cells in row 1 give the status in September

2001 of teachers who were new to Cohort 1 schools in September 2000. The next three cells in

the row give the status in September 2002 of teachers who were new to these schools in

September 2001. Of the new teachers in Cohort 1 schools in September 2000, 19 (86.4%) stayed

in the same school in September 2001, and 3 (13.6%) left the NYC DOE completely. None of

these new teachers transferred to other schools in the NYC DOE. Analyzed this way, the

retention data indicate some general trends in teacher retention for all schools in the analysis

including a trend for Cohort 1 that is suggestive of a positive impact of PDL.

Table 12 shows an overall increase in the percentage of new teachers leaving the DOE

system for all three groups between the periods September 2000–September 2001 and September

2001–September 2002. However, the three groups differed in the magnitude of this increase.

The percentage of teachers leaving the NYC DOE for Cohort 1 increased by 1.2 percentage

points for the September 2001 new teachers (14.8%), those who were served by PDL, compared

to the September 2000 new teachers (13.6%) who were not served. This increase was far less



34

than the 8 percentage-point increase for the Comparison Schools for the same period (from

13.8% to 21.8%). Cohort 2, whose new teachers were not served in either of these years, showed

the largest increase (16.2%) in attrition rate for this period (from 3.8% to 20.0%). For the

September 2002 new teachers, Cohort 1 schools, in their second year of PDL, showed a decline

of 5.7 percentage points in system leavers, while Cohort 2, in its first year of PDL service,

showed a small increase of 2.7 percentage points. Thus, these data show evidence of a favorable

effect of PDL upon new teacher attrition rates for Cohort 1 over two years of PDL service.

Program effects for Cohort 2 cannot be assessed until Comparison School data become available

for September 2002-September 2003.

TABLE 12

THREE-YEAR COMPARISON OF THE RETENTION RATES OF NEW TEACHERS

IN PDL COHORT SCHOOLS AND COMPARISON SCHOOLS Gray shaded areas indicate PDL program intervention.

Cohort 1 data for new teachers staying in the same school are less clear. Cohort 1

showed an 8.6 percentage-point decline in new teachers staying in the same school for the

September 2001 new teachers compared to September 2000, about the same as the 8.1

percentage-point decline for the Comparison Schools. For Cohort 1, the percentage staying in

the same school remained about the same for September 2002. Cohort 2 schools showed a

decline in the percentage staying in the same school of 14.8 percentage points for the September

2001 new teachers even though PDL didn’t begin until the next year, during which Cohort 2

showed a 1.0 percentage point decline. Thus, the retention data suggest a positive effect of PDL

on retaining new teachers in the NYC DOE system, but there is no evidence of an effect upon

retention in the same schools.

3 September 2000 – September 2002 data obtained from the DOE

4 September 2002 – September 2003 data obtained from PDL staff

Sept. 2000 - Sept. 20013 Sept. 2001 - Sept. 2002 Sept. 2002 - Sept. 2003

4

Stayed in

Same

School

% (N)

Stayed

in DOE

% (N)

Left DOE

System

% (N)

Stayed in

Same

School

% (N)

Stayed

in DOE

% (N)

Left DOE

System

% (N)

Stayed in

Same

School

% (N)

Stayed in

DOE

% (N)

Left DOE

System

% (N)

Cohort 1

Schools 86.4 (19) 0 (0) 13.6 (3) 77.8 (21) 7.4 (2) 14.8 (4) 77.3 (17) 13.6 (3) 9.1 (2)

Cohort 2

Schools 88.5 (23) 7.7 (2) 3.8 (1) 73.7 (22) 6.7 (2) 20.0 (6) 72.7 (16) 4.6 (1) 22.7 (5)

Comparison

Schools 75.4 (49) 10.8 (7) 13.8 (9) 67.3 (37) 10.9 (6) 21.8 (12) N/A N/A N/A



35

Evaluation Question 7A. Did students of new teachers served by the program

show gains in ELA achievement test scores that were equal to those for all

CSD 20 middle school students? The academic progress of New York City public school students is assessed annually

through the administration of standardized tests in ELA and mathematics in grades 3 - 8. Test

performance is reported through scale scores that range from approximately 450 to 800 and

performance levels from 1 to 4, with Level 3 signifying that students have met the learning

standards for their grade. The scale scores for each grade are vertically aligned, meaning that

performance across grades can be tracked by analyzing a student’s scale score gains from one

year to the next. Students are expected to show scale-score gains from one grade to the next,

reflecting increases in their mastery of skills and their depth of understanding of the subject

matter from one year to the next. Because of their utility for tracking student progress, scale

scores were used in the analysis of test data to address Evaluation Question 7A.

The PDL for New Teachers program is designed to upgrade the skills of novice teachers,

thereby enhancing teaching and learning in their classrooms. One way to measure the

effectiveness of teaching and learning is through the standardized achievement test scores of the

students. A common concern for the students of new teachers is that their academic progress

may be interrupted as the new teachers learn their craft. By providing support to strengthen the

skills of new teachers, PDL has the potential to support the continuity of learning for students in

their classes. One manifestation of this continuity would be continued progress in standardized

test scores. Accordingly, the evaluators tested the hypothesis that the gains in mean ELA scale

scores for students served by PDL-trained new teachers would not differ significantly from the

mean gains of students served by other teachers, i.e. more experienced teachers, in the same

schools. In order to test this hypothesis, the evaluators needed ELA test scores for two years—

the pre-project year 2002 and the project year 2003—for each student served by a PDL-trained

teacher and for students served by other teachers in the same schools. The evaluators obtained

these data for N = 12 PDL-trained teachers in four middle schools served by the program. ELA

test score data describing the gains in performance of the students of PDL teachers and those of

non-PDL teachers in the same school are presented in Table 13. Means and standard deviations

are presented by grade within school and by grade across schools combined. Analyses could not

be performed for students in Grade 8 because a scoring error by the test publisher nullified grade

7 scores for 2002. The mean gains of the PDL group were higher than those for the non-PDL

group in three out of the four schools and four out of the six grade-within-school comparisons.

Mean gains for the two groups were similar in the other two analyses. For one school,

School H, the PDL group had higher mean gains for both Grade 6 and Grade 7. For most

analyses, the PDL group had substantially higher mean scale-score gains than the non-PDL

group; for grade 7 in School A the mean gain for the PDL group was nearly 10 scale scores

higher than the non-PDL group, 20.84 versus 10.98, respectively. The mean gains of the PDL

group were also higher for the combined-schools analyses. The mean gains in Grade 6 were 6.35



36

for PDL compared to 3.03 for non-PDL, and in Grade 7 they were 20.98 for PDL 3.03 compared

to 15.80 for non-PDL.

TABLE 13

PDL FOR NEW TEACHERS

MEANS AND SD FOR PRE AND POST ELA SCALE SCORES OF STUDENTS OF PDL-

SERVED AND NON-PDL SERVED TEACHERS

ELA Scale Scores

Pre (2002) Post (2003)

School Group N Students Mean SD Mean SD

A PDL 24 697.04 34.93 717.88 38.58

Non-PDL 151 664.90 31.04 670.74 29.93

B PDL 66 667.97 15.61 669.21 18.56

Non-PDL 901 672.06 38.3 682.37 40.56

C PDL 49 667.96 32.13 682.35 28.51

Non-PDL 593 649.01 30.46 657.77 30.39

H PDL 128 635.83 26.04 649.52 25.96

Non-PDL 553 641.02 31.59 652.39 31.52

All Above Schools PDL 267 655.17 32.92 666.55 33.26

Combined Non-PDL 2619 657.54 36.69 667.39 37.53

T-tests of the statistical significance of the differences in mean scale-score gains of the

PDL and non-PDL groups are summarized in Table 14. Three grade-within-school analyses

showed statistically significant differences (p < .05) in favor of the PDL group: Grade 7 in

School B, t = 2.03, df = 457, p = .043; Grade 6 in School H, t = 1.99, df = 358, p = .047; and

Grade 7 in School H, t = 1.99, df = 317, p = .047. None of the grade-within-school tests of

significance showed significant differences in favor of the non-PDL group. Table 14 also

displays the Effect Size for the mean differences. Unlike tests of statistical significance, Effect

Sizes are not affected by the size of the samples being compared. The magnitude of Effect Size

is interpreted using a rubric for educational meaningfulness. Effect Sizes of .20 - .49 indicate

differences that have small educational importance, .50 - .69 have moderate educational

importance, and .70 and higher indicate large education importance. Using this rubric, four of

the mean differences suggested that PDL had educationally meaningful effects upon gains in

ELA test scores; two of these effects were small—Grade 7 in School B and Grade 6 in School H,

E.S. = .29 and E.S. = .22, respectively— and two were moderate—Grade 7 in School A and

Grade 7 in School H, E.S. = .35 and E.S. = .38, respectively. The mean difference in Grade 7 for

all schools combined showed an effect of PDL on ELA gain scores that was both statistically

significant (t = 2.37, df = 1,191, p = .018) and educationally meaningful to a small degree (E.S. =

.21).



37

TABLE 14

PDL FOR NEW TEACHERS

SUMMARY OF T-TESTS FOR THE SIGNIFICANCE OF THE DIFFERENCE IN ELA

GAIN SCORES FOR STUDENTS OF PDL SERVED AND NON-PDL SERVED

TEACHERS

ELA Scale Score Gains

School Group N Mean SD

Mean

Difference t df p

PDL 24 20.83 42.88 A

Non-PDL 151 5.84 23.63 14.99 2.53 173 0.012

PDL 66 1.24 16.41 B

Non-PDL 901 10.31 27.47 -9.07 -2.65 965 0.008

PDL 49 14.39 22.88 C

Non-PDL 593 8.76 22.97 5.63 1.65 640 0.100

PDL 128 13.69 20.93 H

Non-PDL 553 11.36 24.79 2.33 0.98 679 0.326

PDL 267 11.38 23.79 All Above Schools

Combined Non-PDL 2198 9.85 25.41 1.53 0.94 2463 0.350


Achievement Test (SILSSAT) scores of students of teachers served by the

program higher than those of students of non-PDL teachers in the same

schools?

All middle school students in New York State public schools are administered the

SILSSAT once, usually in eighth grade but sometimes in seventh. SILSSAT assesses each

student’s level of mastery of the state social studies learning standards at the intermediate level.

Test scores are reported in scale scores and performance levels. The analysis used scale scores to

increase its sensitivity for detecting program effects.

The analysis compared the mean 2003 SILSSAT scale scores of students served by PDL-

trained social studies teachers and those of non-PDL trained social studies teachers in the same

schools. To control for prior differences in achievement, differences in 2002 ELA scale scores

between the two groups were partially controlled through covariance. Test data were obtained

for (N = 5) PDL teachers in four CSD 20 middle schools. Table 15 displays the test score means

and standard deviations for each group by school and for the four schools combined. The mean

2003 SILSSAT scale scores tended to be higher for the non-PDL group than for the PDL group

both within schools and across the four schools combined. However, the mean 2002 ELA scale

scores also tended to be higher for the non-PDL group. Accordingly, the differences in



38

SILSSAT means may have been attributable to differences in the general achievement of the two

groups prior to the program.

TABLE 15.

PDL FOR MIDDLE SCHOOL SOCIAL STUDIES TEACHERS MEANS AND SD’S FOR

SOCIAL STUDIES AND ELA SCALE SCORES OF STUDENTS OF PDL-SERVED AND

NON-PDL SERVED TEACHERS

2003 Soc Stud 2002 ELA

School Group N Mean SD Mean SD

PDL 59 60.10 7.54 688.08 15.70 B

Non-PDL 414 68.09 15.34 707.42 41.12

PDL 18 46.22 7.18 675.89 13.56 D

Non-PDL 209 52.91 9.30 685.26 22.57

PDL 10 57.50 9.36 679.40 16.96 E

Non-PDL 321 59.20 12.91 692.63 26.20

PDL 26 53.65 5.04 680.42 11.07 H

Non-PDL 231 52.65 9.54 677.39 24.72

PDL 113 56.18 8.70 683.61 15.17 All Above Schools

Combined Non-PDL 1175 59.93 14.29 693.54 33.52

Table 16 displays a summary of the multiple regression analysis for the indirect effects of PDL-

training upon SILSSAT scores, controlling for pre-program differences in ELA test scores. The

R-square change for PDL was small for all analyses and not statistically significant for the

combined analysis and for three of the four schools. Thus, PDL did not have a statistically

significant effect upon SILSSAT scores beyond the influence of prior general achievement.



39

TABLE 16

PDL FOR MIDDLE SCHOOL SOCIAL STUDIES TEACHERS

SUMMARY OF STEPWISE MULTIPLE REGRESSION ANALYSIS FOR THE

EFFECTS OF PDL ON GRADE 8 STUDENTS SOCIAL STUDIES SCALE SCORES

CONTROLLING FOR 2002 ELA SCALE SCORES

School Step

Variables

Entered R

Adjusted R

Square

R Square

Change

F

Change df p

1 ELA 0.839 0.703 0.703 1116.35 1,471 <.0001 B

2 PDL 0.840 0.704 0.002 2.90 1,470 0.089

1 ELA 0.711 0.504 0.506 230.26 1,225 <.0001 D

2 PDL 0.720 0.514 0.013 5.99 1,224 0.015

1 ELA 0.800 0.638 0.639 583.45 1,329 <.0001 E

2 PDL 0.801 0.639 0.002 2.02 1,328 0.156

1 ELA 0.644 0.413 0.415 180.91 1,255 <.0001 H

2 PDL 0.644 0.410 0.000 0.03 1,254 0.862

1 ELA 0.811 0.658 0.658 2471.85 1,286 <.0001 All Above Schools

Combined 2 PDL 0.811 0.657 0.000 0.14 1,285 0.712



40

V. CONCLUSIONS AND RECOMMENDATIONS

This section presents the conclusions of the evaluation in the form of answers to the

questions that it was designed to address. These conclusions are based on the analyses of the

wide range of data that has been collected on the implementation of the PDL program in CSD 20

during the 2002 – 2003 school year and its effects upon program participants, including the

implementers and the recipients of program services. Following the conclusions,

recommendations are offered that are aimed at enhancing future program implementation and its

effects.

Conclusions

Evaluation Question 1: How well did PDL prepare NTF’s and RT’s for their

mentoring and modeling roles, respectively, in the new teacher program?

The findings of this report suggest that the PDL training provided to NTF’s was effective

and appropriate. NTF’s reported that the various training components aimed at preparing them

for their mentoring roles, including the Author Study Exchanges, weekly NTF meetings, and the

NTF Institutes, were useful to them. The LTPLP sessions, though not directly affiliated with the

PDL for New Teachers program, were also noted as highly useful training opportunities.

Observations of the sessions suggested that NTF’s were given opportunities to reflect on and

practice learned skills. NTF’s characterized the varied training opportunities offered to them as

reinforcing and enhancing newly learned skills. They did not perceive them to be repetitive in

any way. Data collected from the PDAQ questionnaire support this conclusion, indicating that

NTF’s responded to the most important needs of their new teacher interns. This suggests that

PDL effectively prepared NTF’s for their roles as mentors. The sample IPAL’s collected also

indicate that NTF’s assisted their interns using techniques learned during PDL training sessions,

including instructional strategies that focused on the needs of the students, mapping and

diagramming lessons, and test preparation.

Other data collected pointed to school-level contextual obstacles that had to be overcome

by the NTF’s in their work with their interns. NTF’s commented that in some cases their

mentoring schedules were interrupted by emergency coverages that were assigned to their

interns. Also, several of the NTF’s assumed other functions in the schools, and they noted that

the additional responsibilities often interfered with their mentoring responsibilities. Many of the

NTF’s also indicated that they were not formally introduced to the staff, making it difficult for

them to encourage the participation of veteran teachers in their intern’s training.

Evaluation Question 2: In the perception of new teachers, to what extent did program

services address their most important staff professional development and support

needs?



41

Evidence from the PDAQ indicates that new teachers perceived the PDL for New

Teachers program to be responsive to their most important professional development and support

needs, including motivating students, classroom management, lesson planning, and clarifying

instructional goals. The sample IPAL’s collected also suggest that NTF’s responded to the needs

of their interns. IPAL’s consistently pointed to support provided in the areas of classroom

management and lesson planning, two areas reflected in the PDAQ as important to the new

teachers.

Additionally, many of the teachers who participated in the Cycle Program reported that

they benefited from the program’s activities, particularly the opportunity to observe a veteran

teacher using instructional strategies that incorporated group work, manipulatives, and multiple

intelligences. Participants did note that the program could have been enhanced with a second

round of visits to the RT’s classrooms. Some participants indicated that RT’s should have been

given the opportunity to visit the VT’s to observe their implementation of strategies learned

during the Cycle Program. However, one participant did note that the time commitment for the

Cycle Program was extensive, and any additional components would interfere with both the RT’s

and VT’s students learning.

Overall, the evidence suggests that the PDL for New Teachers program had a structured

model for implementation, but also had sufficient flexibility to meet the individual professional

development needs of new teachers.

Evaluation Question 3A: To what extent did new teachers believe that the program

facilitated their professional growth in the competencies of effective teachers?

Multiple analyses of the data collected from the EECI suggest that there was a

relationship between PDL support and those areas where teachers believed themselves to be

most effective. The data also suggests that more program participants felt competent and

supported in the domain of Planning and Preparation, which included areas such as content

knowledge, assessment, and goal setting. The relationship between the teachers’ beliefs about

their competencies and the support they received from PDL was also strongest in this domain.

The evidence suggests that on average, fewer teachers believed themselves to be highly

competent in the domains of Classroom Environment, Instruction, and Professional

Responsibilities. Additionally, a notably low number of respondents indicated high skill levels

in the areas of gaining access to school resources for students and engaging families.

Correspondingly, a low number of respondents indicated high levels of support from PDL in

those same areas.

Evaluation Question 3B: To what extent did social studies teachers believe that the

program facilitated their professional growth in the competencies of effective teachers?

The evidence collected suggests that participants in the PDL for Middle School Social

Studies Teachers program believed the program facilitated their professional growth as teachers.

The Social Studies Feedback Survey and PDL’s internal program feedback forms indicate that



42

participants believed their ability to integrate technology, primary source documents, literature,

and cooperative learning strategies into their classrooms was enhanced by this PDL program.

Some participants did note, however, that their ability to integrate technology in their practice

was hampered due to their schools’ lack of technological resources. This varied from school to

school.

Evaluation Question 4A: Did new teachers served by the program show an increase in

their sense of efficacy as a teacher?

The evidence with respect to PDL’s impact on new teachers’ sense of efficacy was

inconclusive. There was no significant increase in the new teachers’ sense of efficacy, due in

part to the low return rate on the post-TES. Additionally, the literature regarding teacher

efficacy suggest that the construct of “general teacher efficacy” may be problematic and difficult

to measure.

Evaluation Question 4B: Did social studies teachers served by the program show an


The evidence with respect to PDL’s impact on the middle school social studies teachers’

sense of efficacy was similarly inconclusive. There was no significant increase in the social

studies teachers’ sense of efficacy, due in part to the low return rate on the post-TES.

Additionally, the cautionary literature cited above obtains here as well.

Evaluation Question 5A: To what extent did new teachers served by the program

demonstrate the professional competencies of effective classroom teachers?

The evidence collected for this evaluation suggests that, on average, participants in the

PDL for New Teachers program were either partially proficient or were approaching proficiency

in the professional competencies of effective teachers. The data from the DRTO demonstrated a

slight increase in the mean scores of the sample of teachers over time. There was a notable

increase in the mean scores of six areas. Four of the areas were in the Planning and Preparation

domain, and the remaining two were in the Instruction and Classroom Environment domain,

respectively. Four of these six items showed sample teachers moving from a classification of

“partially proficient” to “approaching proficiency” in the professional competencies of effective

teachers. In general, the changes in mean scores were in the expected direction. It is expected

that with a larger sample and/or a longer intervention, the difference in mean scores would have

been greater.

Evaluation Question 5B: To what extent did social studies teachers served by the

program demonstrate the professional competencies of effective classroom teachers?

The data collected revealed that the sample of social studies teachers who participated in

the observation phase of the research generally demonstrated at least “partial proficiency” in the

professional competencies of effective classroom teachers. In three of the four cases, scores



43

generally remained constant or increased slightly in the second observation. Conversely, in the

fourth case, the teacher’s ratings decreased in a number of the twenty-nine areas from

“proficient” to “partially proficient.”

Evaluation Question 6: Were the retention rates of new teachers served by the program

better than new teachers in comparable schools that were not?

The analysis of new teacher retention data showed a trend toward a favorable effect of

PDL upon reducing the rate of new teachers leaving the New York City school system. For both

cohorts of PDL schools, the attrition rates showed a slight increase in the first year of service, but

far less than the increase in comparison schools. Further, there was a decline in system leavers

for the schools that were served by PDL for a second year. The data on new teachers who stayed

in the same school were more equivocal. Teacher retention is a complex issue and is affected by

many factors. Moreover, the evaluation of the program took place under the influence of the

confounding effects of new state regulations and the systemic restructuring of the New York City

public schools. The complex factors that affect new teacher retention are exemplified by the

stories of 44 new CSD 20 teachers who required mentoring by PDL in 2002 – 2003. Data

collected internally by PDL staff showed that 33 teachers returned to their schools. Of the 11

who did not return, 4 continued to teach in the NYC DOE system, with 2 of the 4 teaching in

another of the PDL schools in CSD 20. Two of the teachers did not receive their certification,

and consequently had to leave the system, 1 left teaching to pursue a Master’s degree, another

left to pursue medical credentials, and another had lost her position due to cut-backs. It is

evident that many teachers left for reasons that could not be changed by professional

development. Within the limitations of these complexities, the data suggest that PDL has had a

valuable effect on the professional growth of teachers in CSD 20 and the probability that they

will remain in the New York City schools.

Evaluation Question 7A. Did students of new teachers served by the program show

gains in ELA achievement test scores that were equal to those for all CSD 20 middle

school students?

The analysis of ELA achievement test score data indicates that PDL had a favorable

affect on students of PDL-served teachers in CSD 20. The mean gains on ELA achievement of

the PDL group were higher than those for the non-PDL group in three out of the four schools and

four out of the six grade-within-school comparisons. Mean gains for the two groups were similar

in the other two analyses. The mean gains of the PDL group were also higher for the combined-

schools analyses. The mean gains in Grade 6 were 6.35 for PDL compared to 3.03 for non-PDL,

and in Grade 7 they were 20.98 for PDL 3.03 compared to 15.80 for non-PDL. In general, PDL

had statistically significant and educationally meaningful effects on the achievement of students

of new teachers who were served by the PDL for New Teachers program.



44


Achievement Test (SILSSAT) scores of students of teachers served by the program

higher than those of students of non-PDL teachers in the same schools?

The analysis of SILSSAT scores indicate that PDL did not have a statistically significant

effect on scores beyond the influence of prior general achievement.

The mean 2003 SILSSAT scale scores tended to be higher for the non-PDL group than for the

PDL group both within schools and across the four schools combined. However, the mean 2002

ELA scale scores also tended to be higher for the non-PDL group. Accordingly, the differences

in SILSSAT means may have been attributable to differences in the general achievement of the

two groups prior to the program.

Recommendations

PDL for New Teachers and LTPLP

1. PDL and District staff should consider ways to make principals and other supervisory staff

more aware of PDL’s mission and goals, as well as the program components.

Support from the school leadership is one area that NTF’s perceived was lacking.

Successful professional development programs must have strong support from both the school

and the district leadership. An orientation and ongoing updates of the program’s progress would

facilitate the NTF’s transition into his/her role, which is particularly important for those NTF’s

that are new to the school community. In addition, it makes it clear to all parties involved that

the professional development of new teachers is a priority. The PDL and District leadership

should also communicate to the school leadership that every effort should be made to ensure that

new teachers do not receive emergency coverages or additional responsibilities that interfere

with the mentoring schedule. The same efforts should be made with respect to NTF’s in regard

to the mentoring and training schedule.

2. PDL and District staff should work with school leadership to devise ways to encourage and

facilitate structured visits to classrooms.

Some NTF’s reported that veteran staff members felt threatened by the NTF’s, and were

reluctant to allow them or new teachers to visit their classrooms. School leadership should

encourage structured visits to classrooms and conferences among veteran and new staff, without

fear of reprisal. This will assist the NTF’s in providing new teachers with models of successful

teaching strategies, as well as legitimize their work.

3. NTF’s need additional training in helping new teachers assess the impact of their

instructional approaches on their students.

All NTF’s who responded to the CRTL questionnaire indicated that they believed their

interns were positively impacting their students, yet they often could not cite “concrete



45

evidence.” Being able to cite specific evidence would not only help to enhance the new

teacher’s craft, but would also go a long way toward helping the role of PDL and the NTF

become more valued.

4. PDL should consider ways to schedule and deliver training so that NTF’s are out of the

building less often—no more than once a week, if possible.

All of the NTF’s who responded to the questionnaire noted that the various training

components were valuable and reinforcing. The biggest drawback they noted, however, was that

the numerous training sessions took them out of the building and often interfered with their

mentoring schedule.

5. PDL should consider expanding its programmatic goals and focus to include more explicit

training in knowledge of gaining access to school resources for students and working with

families.

PDL for New Teachers was responsive to the most important needs articulated by new

teachers. However, the data collected on the EECI indicated that some of the teachers’ self-

ratings of their professional skill and the emphasis of mentoring in some areas were relatively

low, in particular knowledge of gaining access to school resources for students and working with

families.

6. PDL should consider expanding pool of teachers eligible for The Cycle Program so more

cohorts of teachers, including new teachers who have already met the mentoring requirement,

can take advantage of this highly effective model.

The data suggest that those new teachers who participated in the Cycle Program found it

to be very beneficial. An additional component whereby the RT visits and works with the VT at

his/her home school might be added. This will encourage the VT’s to practice the skills they

learned during the initial phase of the Cycle Program. Additionally, RT’s can help the VT

modify the approaches based on the needs of the VT’s students. The Cycle Program should also

begin prior to the spring semester to avoid disrupting the classroom schedules in a concentrated

period of time.

While the above recommendations are aimed at strengthening the quality of

implementation of future replications of the program, it should be emphasized that there is

considerable evidence of the program’s positive impact upon the professional growth of new

teachers and the quality of teaching and learning in their classrooms. Particularly notable in this

regard is evidence suggesting that the program’s basic model is effective and should be retained

as a foundation.



46


Most of the participants in the PDL for Middle School Social Studies Teachers program

noted that learning to use technology as a resource was a positive experience. However, some

noted that it was difficult to take advantage of this new skill because their schools lacked the

appropriate resources. While this is not a PDL program flaw, participants would most benefit if

the District and the schools ensured that all participants in the program, and their students, have

access to technology resources that allow them to implement the approaches they learned.

Additionally, to better assess the individual effects of the PDL for Middle School

Social Studies program on student achievement in Social Studies, it is recommended that

evaluations of future replications pay greater attention to the collection of more complete data.

This can be achieved by identifying the students served by program teachers prospectively, i.e. at

the beginning of the program, so that plans can be developed to capture their achievement test

scores later in the project year. In addition, these achievement test data should be supplemented

by other evidence of growth in student achievement, chief among which is the structured

analysis of student work.



47

REFERENCES

Archer, J. (1999). Mentoring New Teachers. Education Week, 18(27), 1.

Black, S. (2001). A Lifeboat for New Teachers. The American School Board Journal, 188(9),

46-48. [online] http://www.asbj.com/2001/09/0901research.htm June 17, 2003

Feiman-Nemser, S. 2003. What New Teachers Need to Learn. Educational Leadership. 60(8),

58-60. [online] www.ascd.org/members/ed_lead/200305/feimannemser.html May 30, 2003

Fickel, L. H. (fall 2002). Quality Professional Development: Suggestions about Process and

Content. The Educational Forum, 67, p47-54

Gibson, S. & Dembo, M.H. (1984). Teacher Efficacy: A Construct Validation. Journal of

Educational Psychology, 76(4), 569-582.

Guskey, T.R. (2002). Does It Make a Difference? Evaluating Professional Development.

Educational Leadership, 59(6), 45-51.

Guskey, T.R. (Aug 1998). Making Time to Train Your Staff. The School Administrator Web

Edition. [online] http://www.aasa.org/publications/sa/1998_08/focGuskey.htm May 30, 2003.

Halford, J.M. (1998). Easing the Way for New Teachers. Educational Leadership, 55, 33-36.

Mauer, E., & Zimmerman, E. (2000). Mentoring New Teachers. Principal, 79(3), 26-28.

Moir, E. (1990). Phases of First-Year Teaching. [online]

http://www.newteachercenter.org/article3.html June 6, 2003.

Moir, E. & Gless, J. (2003) Quality Induction: An Investment in Teachers [online]

http://newteachercenter.org/article-CCETQInd.html June 20, 2003.

McDonald, J.P. (2002). Teachers Studying Student Work. Phi Delta Kappan, 84(2), 120-127.

Moir, E. & Bloom, G. 2003. Fostering Leadership Through Mentoring. Educational Leadership.

60(8), 58-60.[online] http://www.ascd.org/members/ed_lead/200305/moir.html May 30, 2003

O’Shea, M.R. (2002). Teaching to Standards. Leadership, 31(3), 22-23.

evaluation of the professional development lab …...evaluation of the pdl programs in community...

Documents