ma ssip phase iii report - massachusetts … · web viewmassachusetts department of elementary and...

Massachusetts Department ofElementary and Secondary Education (MA ESE)

State Systemic Improvement Plan (SSIP)Phase III Report

FFY 2015Submitted to the Office of Special Education Programs March 29, 2017

Table of ContentsA. Summary of Phase III.........................................................................................................................................4

1. Massachusetts SIMR and Theory of Action...................................................................................................4

2. Summary of Improvement Strategies...........................................................................................................6

3. Evidence-Based Practices Implemented to Date..........................................................................................8

4. Evaluation Activities, Measures, and Outcomes...........................................................................................8

5. Highlights of Changes to Implementation and Improvement Strategies....................................................10

B. Progress in Implementing the SSIP..................................................................................................................11

1. Description of the State’s SSIP Implementation Progress...........................................................................11

2. Stakeholder Involvement in SSIP Implementation......................................................................................17

C. Data on Implementation and Outcomes..........................................................................................................20

1. How the State Monitored and Measured Outputs to Assess Effectiveness................................................20

2. How the State has Demonstrated Progress and Made Modifications to the SSIP......................................27

3. Stakeholder Involvement in the SSIP Evaluation........................................................................................29

D. Data Quality Issues..........................................................................................................................................30

1. Data Limitations..........................................................................................................................................30

E. Progress in Achieving Intended Improvements...............................................................................................32

1. Assessment of Progress toward Achieving Intended Improvements..........................................................32

F. Plans for Next Year...........................................................................................................................................47

1. Additional Activities to be Implemented Next Year (with timeline)............................................................47

2. Planned Evaluation Activities Including Data Collection, Measures, Expected Outcomes..........................48

3. Anticipated Barriers and Solutions.............................................................................................................50

4. State’s Need for Additional Support/TA.....................................................................................................50

APPENDIX.............................................................................................................................................................51

Tables and Figures

TABLES

Table 1. SSIP Training Sequence...................................................................................................................12

Table 2. SSIP Implementation Progress (May 2015 – January 2017)........................................................16

Table 3. MA SSIP Evaluation Plan................................................................................................................20

Table 4. Data Collection Plan.........................................................................................................................25

Table 5. SSIP Pyramid Model Training Participant Totals (May 2015 – January 2017)........................33

Table 6. Quality of SSIP Training Events: Average Ratings on Four-Point Agreement Scale..............34

Table 7. Usefulness of Pyramid Model Practices Training: Comparison of Year 1 and Year 2 Feedback............................................................................................................................................................35

Table 8. Usefulness of PTR-YC Training: Comparison of Year 1 and Year 2 Feedback.......................35

Table 9. Progress toward Implementation of Pyramid Strategies among SSIP Districts and Schools. .39

Table 10. Benchmarks of Quality – Ratings of Critical Elements..............................................................40

Table 11. TPOT Training: Participants’ Knowledge of TPOT Before and After Training.....................45

Table 12. SPP Indicator 17 Data Summary Statement 2..............................................................................45

Table 13. SPP Indicator 17 Data.....................................................................................................................46

Table 14. SSIP Implementation: February 2017 – Spring 2018..................................................................47

Table 15. SSIP Evaluation Plan: February 2017 – January 2018................................................................49

Table 16. Anticipated Barriers and Solutions.......................................................................................................50

FIGURES

Figure 1. MA SSIP Theory of Action..............................................................................................................6

Figure 2. 2016 Mid-year Leadership Team Survey: Ratings of District Team Progress toward Implementation “Over the Past 6 Months”....................................................................................................42

............................................................................................................................................................................42

............................................................................................................................................................................43

A. Summary of Phase IIIMassachusetts’ State Systemic Improvement Plan (MA SSIP) is designed to result in improved social emotional outcomes for preschool children with IEPs. Based on input from a wide array of stakeholders, the Massachusetts Department of Elementary and Secondary Education (MA ESE) has selected implementation of Preschool Positive Behavior Supports (PBS) through Pyramid strategies as its evidence-based practice (EBP) to positively impact students. Massachusetts has a long history with the Pyramid Model, having first engaged in trainings to support replication in preschool classrooms in 2009. Since that time, MA has been a member of the Pyramid Model Consortium, and has built upon this experience to design the MA SSIP. Now in Phase III, MA ESE is engaged in implementation of the Pyramid Model with 19 districts, and is evaluating all components of the work as planned during Phase II. To implement the SSIP, MA ESE has contracted with national experts from the Pyramid Consortium to collaborate with state personnel to provide training, and to assist in building capacity among coaches, practitioners, and other educators to implement the model with fidelity in Massachusetts’ schools. Further, MA ESE has hired an external evaluator to provide additional expertise with data analysis and systems so that additional refinement of the SSIP will continue to promote improved child outcomes.

1. Massachusetts SIMR and Theory of Action

Indicator 17: SSIP – State Identified Measurable Result (SIMR)

Reported Data

FFY 2013 2014 2015

Target ≥ 100% 100%

Data (Summary Statement 1) 85.44% 87.20% 79.14

FFY 2013 2014 2015

Target ≥ 90% 90%

Data (Summary Statement 2) 44.49% 47.81% 53.57%

FFY 2016 – FFY 2018 Targets

FFY 2016 2017 2018

Target ≥ (Summary Statement 1) 100% 100% 100%

FFY 2016 2017 2018

Target ≥ (Summary Statement 2) 90% 90% 90%

4 | P a g e

Description of Measure

Indicator 7: Preschool Outcomes measures the percent of preschool children aged 3 through 5 with IEPs who demonstrate improved:

a. Positive social-emotional skills (including social relationships); b. Acquisition and use of knowledge and skills (including early language/communication and early

literacy); andc. Use of appropriate behaviors to meet their needs.

MA ESE, in collaboration with stakeholders, will focus on improving social-emotional outcomes, or Indicator 7a, for the SSIP. Child level data are collected via the Child Outcomes Summary (COS) and aggregated. These aggregated data are measured by Summary Statements.

Summary Statement 1: Of those preschool children who entered or exited the preschool program below age expectations in Outcome A, the percent who substantially increased their rate of growth by the time they turned 6 years of age or exited the program.

Measurement for Summary Statement 1: Percent = # of preschool children reported in progress category (c) plus # of preschool children reported in category (d) divided by [# of preschool children reported in progress category (a) plus # of preschool children reported in progress category (b) plus # of preschool children reported in progress category (c) plus # of preschool children reported in progress category (d)] times 100.

Summary Statement 2: The percent of preschool children who were functioning within age expectations in Outcome A by the time they turned 6 years of age or exited the program.

Measurement for Summary Statement 2: Percent = # of preschool children reported in progress category (d) plus # of preschool children reported in progress category (e) divided by [the total # of preschool children reported in progress categories (a) + (b) + (c) + (d) + (e)] times 100.

Further detailed information about data collection samples, methods, and tools can be found in MA ESE’s FFY 2015 Part B State Performance Plan (SPP)/Annual Performance Report (APR).

Theory of Action

The MA SSIP Theory of Action (TOA), shown in Figure 1 on page 6 is based on a cascading structure of collaboration and support that flows from the state level (inter-agency and MA ESE Early Childhood Special Education (ECSE) initiatives), to programs, classrooms, and students. Family engagement is essential throughout, as is an ongoing process of inquiry and improvement.

5 | P a g e

Figure 1. MA SSIP Theory of Action

2. Summary of Improvement Strategies

Principal SSIP Activities

The principal activities that comprise the SSIP began in spring 2015, and are being conducted across the state (i.e., infrastructure for the SSIP), district, school, and community levels as they align with MA ESE’s Theory of Action. These activities include the following:

National Expert Support

MA ESE contracted with Pyramid Model Consortium to design and provide training for leadership teams and external coaches in Pyramid Model implementation and support, as well as training for all participants in Pyramid Practices; Prevent, Teach, Reinforce - Young Children (PTR-YC); and use of the Teaching Pyramid Observation Tool (TPOT).

Statewide Coaching

MA ESE funded eight external coaches with Pyramid Model experience to support participating districts’ implementation of the model in their schools and classrooms. External coaches have participated in all statewide training activities for districts and their personnel, and co-trained on the Practices modules. External coaches also received coaching and training from the Pyramid Model Consortium and MA ESE staff.

6 | P a g e

District Leadership Teams

District leadership teams were formed within the 19 participating districts to include representation from both administration and practitioner levels, as well as to bring district and school level perspectives to the work. Each team consists of a district-based administrator, a special education administrator, an early childhood administrator or principal, a teacher, an educator with behavior expertise, and a classroom coach for the district.

Training Practitioners

At the core of building statewide infrastructure for implementing PBS through Pyramid strategies is the provision of high-quality trainings to build capacity at the local level. The SSIP has offered multiple statewide trainings in Pyramid Model Practices, TPOT, and PTR-YC. More than 300 classroom teachers, external and classroom coaches, behavior specialists, and community members have taken part in training since SSIP implementation began in spring 2015.

Training District-Based Coaches

District-based classroom coaches receive additional training and preparation to work with their district teams and teachers to implement Pyramid strategies with fidelity. In the fall of 2016, MA ESE hosted a half day “Coaches Day” training on practice based coaching for classroom coaches; approximately 30 people attended. External coaches are also providing on-site support to classroom coaches including co-scoring TPOTs, helping them to develop implementation plans, and creating coaching plans.

Infrastructure Improvements

The MA ESE is also engaged in numerous statewide activities related to the implementation of the Pyramid Model more broadly across Massachusetts. A sample of these activities is described in Section B with respect to activities that have taken place since the Phase II report was submitted in 2016. The broad areas of focus include the following:

Intra- and Inter-Agency Initiatives, such as Pyramid Model Trainings for Families, outreach to Approved Private Special Education schools, and participation in the Pyramid Model State Leadership Team and Pyramid State Summit.

Professional Development and Guidance, including supporting early literacy skills of young children with disabilities, promoting inclusive opportunities for children with disabilities, and the early childhood personnel action plan, among others.

Data Systems, specifically new Memorandum of Understanding with MA DPH to increase data sharing activities.

Grants and Fiscal Support, including increased funding to other state agencies and local communities to support the implementation of the Pyramid Model in community-based and Head Start programs to improve inclusive services for children with IEPs.

Accountability and Monitoring, including alignment of the SSIP with the Every Student Succeeds Act (ESSA) plan, alignment with Early Literacy initiatives designed to align social/emotional and literacy outcomes, and increased collaboration with districts in Massachusetts accountability Level 4.

7 | P a g e

3. Evidence-Based Practices Implemented to Date

All aspects of MA ESE’s SSIP are based upon Evidence-based practices (EBP), namely, the Pyramid Model, to support the social and emotional development of pre-school students. The specific state-level trainings being carried out by the state and its partner, the Pyramid Model Consortium, are described below. The goal of the SSIP at the state and local levels is geared toward classroom implementation of PBS through Pyramid strategies, with a focus on fidelity to the model.

Pyramid Model Practices

This training teaches practitioners how the Pyramid Model promotes social emotional development, relationship-building, targeted supports for at-risk children, and individualized intensive interventions for children with persistent challenges. Tools specific to early childhood environments, child observations, and data collection are introduced.

Prevent, Teach, Reinforce Young Children (PTR-YC)

PTR-YC is a model of intervention for young children with the most persistent and severe challenging behaviors; the training is based on positive behavior support strategies, and designed to support fidelity of implementation.

Teaching Pyramid Model Observation Tool (TPOT)

The TPOT training is geared toward classroom coaches who will be administering the TPOT, an observation tool designed to assess the fidelity of the Pyramid Model strategies implementation in the classroom. In addition to a measure of implementation of the Pyramid Model in the classroom, it can also be used to guide teachers in supporting children’s social and emotional development.

Positive Solutions Train-the-Trainer

The Positive Solutions for Families, train-the-trainer model addresses the skills and strategies necessary for professionals to support families to implement PBS approaches in the home. This training is an evidence-based parent training series during which professionals learn how to partner with parents of preschoolers to promote positive and effective parenting strategies. In partnering with families, children’s social and emotional development is promoted and challenging behaviors and mental health needs of children in preschool addressed. Positive Solutions aligns with Massachusetts’ priority to enhance family and community engagement and supports improved strategies for working with families of young children in preschool settings.

4. Evaluation Activities, Measures, and Outcomes

MA ESE worked closely with internal partners and key stakeholders to develop an evaluation plan for the SSIP that is well-aligned with its Theory of Action. A series of eight evaluation questions across the four levels of the SSIP TOA – state, program/district, schools, and classrooms – were identified to determine progress toward both implementation and outcomes. The questions can be summarized as follows:

To what extent was the SSIP implementation carried out as planned (i.e., trainings delivered, adequate participation levels, implementation with other related initiatives, district-level planning and implementation, classroom implementation of Pyramid strategies)?

8 | P a g e

Have intended outcomes have been achieved (i.e., high-quality training to increase coach and practitioner knowledge and skills, increased district capacity to implement, increased number of classrooms implementing, improved student outcomes as aligned with the SIMR)?

Intended outcomes of the SSIP were also established, and data collection tools were selected or created to measure these outcomes. The data collection activities include the collection and review of project documentation (e.g., training participation, inter-agency meeting notes, etc.), Statewide Training Evaluation Forms, External Coach Contact Records, Benchmarks of Quality (BOQ) Assessments, Mid-year Leadership Team Survey, TPOT (locally administered and reviewed), and Indicator 7 data via the Child Outcomes Summary (COS).

With respect to early outcomes of the SSIP, Progress Findings prepared by the external evaluator are shown below. Section E of this report describes the findings in more detail.

As specified in the SSIP planning and design phases, MA ESE has successfully organized and conducted multiple statewide trainings in evidence-based practices (EBP) to foster implementation of PBS through Pyramid strategies among participating districts and their schools.

Approximately 326 individuals across 19 districts have taken part in the EBP trainings to date. Participants have included external coaches, district leadership teams, practitioners (teachers), classroom coaches, and behavior specialists.

Trainings in Pyramid Model strategies and supports have been well-received, and have largely been found by participants to be useful in developing their skills toward implementation. Over time, the effectiveness of the trainings has been improved, guided by participant feedback.

MA ESE has embedded the SSIP within its larger strategic planning efforts in Early Childhood Special Education (ECSE) to improve special education outcomes for all pre-K children statewide.

MA ESE leadership and staff overseeing the SSIP are engaged in ongoing efforts to integrate the work of PBS through Pyramid strategies throughout multiple initiatives at the state and local levels.

Among the 19 districts participating in the SSIP, 17 districts have schools and classrooms engaged in implementing the Pyramid Model. In all, 88 classrooms are in the process of adopting Pyramid.

On average, district leadership teams reported substantial progress across most component areas of the BOQ over time. The greatest areas of improvement were staff buy-in, establishment of a functional leadership team, and establishing and communicating program-wide expectations.

The majority of participating districts reported making progress toward implementation during the first year of participation. More than a third categorized their progress as “moderate” or better by March 2016.

During the first year of the SSIP, district teams expressed a need for additional facilitated planning meetings with their teams, as well as training for teachers in Pyramid Practices.

External coaches most often provide in-person support to their districts, followed by phone and email contacts. Support is typically focused on support for leadership teams, but also extends to classroom coaches, data-driven instructional decision making, and behavior systems, among other areas.

9 | P a g e

The percent of preschool children functioning within age expectations in positive social-emotional skills has increased over each of the past two years.

SIMR data have not yet evidenced improvements in the number of children exiting from preschool with greater than expected growth in their social emotional functioning.

5. Highlights of Changes to Implementation and Improvement Strategies

MA ESE made a number of changes to their SSIP implementation strategy based on stakeholder feedback. These changes were all based on a review of evaluation data, and ongoing conversations with key stakeholders, see Section C (2)(c).

Increased internal alignment at SEA to include students with disabilities in the ESSA plan, Curriculum and Instruction initiatives, District Support activities, and inter-agency agreements

Reduced the format of Pyramid Practices training from 4 days to 3 days Changed practices training to regional, in-district trainings to make them more convenient for

participants Refined the content of the Practice training to be more sophisticated to better meet attendees’ needs Added a “Coaches Day” to enhance the preparation of classroom coaches and external coaches for

their roles Added a monthly external coaches Professional Learning Community (PLC) to foster collaboration

among the group, and for the state to continue to learn about the progress of the SSIP at the district level

Incorporated data-based decision making into the leadership team meeting in September 2016 Supported increased site visits with external coaches and MA ESE staff Provided aggregate BOQ data back to coaches to debrief with their teams for planning purposes Modified external coach contact record based on coach feedback to better reflect the types of support

they are providing Repeated Mid-year Leadership Team Survey to allow for a comparison over time Added additional external coaches for a total of 8 with additional applications pending Offered PTR-YC training more often based on districts’ requests Expanded contract for Comprehensive System of Professional Development (CSPD) coaches that

differentiates between coaching and training to individualize and provide targeted support for districts Expanded the contract with MA’s Parent Training and Information center (PTI), the Federation for

Children with Special Needs (FCSN), to support Positive Solutions in districts Through participation on the MA Pyramid Model SLT, supported inclusive practices through funding

and other initiatives that will include a Pyramid Summit which focuses on district implementation and will feature presentations by participating SSIP districts, and the identification of community based child-care Pyramid Model Sites

10 | P a g e

B. Progress in Implementing the SSIP

1. Description of the State’s SSIP Implementation Progress

Implementation Highlights

The Massachusetts SSIP began in 2013 with the identification of its SIMR, followed by its planning and design phases (Phases I and II). Phase III began ahead of schedule, with 19 districts committing to participation in 2015 (with districts joining in May, September, and October). One additional district joined the SSIP cohort in 2016.

All participating districts have identified leadership teams who are charged with guiding the implementation process in their districts, and have participated in a series of trainings and planning meetings. Personnel in their districts have attended trainings sessions – more than 300 educators have been trained in Pyramid Model Practices to date. Among the 19 districts, 17 have begun to implement the model in at least one school in their district, with a total of 88 classrooms implementing the model. Districts are also receiving on-site coaching from eight external coaches hired by the state. External coaches bring expertise in the Pyramid Model and childhood trauma, and many are licensed mental health clinicians and social workers.

In addition to the principal activities being conducted with these districts, MA ESE project personnel continue to extend the SSIP focus to a broader set of initiatives at the state level, through alignment in general education initiatives including the development of the MA ESSA plan, active participation on the Massachusetts Pyramid Statewide Leadership Team (SLT), integration of the SSIP within Early Childhood Special Education (ECSE) initiatives, and alignment with MA DPH Part C SSIP and the MA ESE Positive Behavior Intervention and Support (PBIS) initiative specifically.

(a) Description of extent to which the State has carried out its planned activities with fidelity; accomplishments and milestones; and adherence to intended timeline

This section of the report is organized around two major areas of SSIP activities: 1) Implementation of the Pyramid Model (the primary SSIP activities being conducted with 19 districts, and those being directly evaluated), and 2) Infrastructure Improvements.

Implementation of the Pyramid Model

The 19 districts participating in the SSIP began the process by identifying district leadership teams to plan and oversee the work of implementing the Pyramid Model in preschool classrooms in their districts. These teams of approximately eight individuals each include district and school level personnel, representing both administrators and practitioners.

Leadership teams participated in a number of events facilitated by the state, including in a kick-off webinar, a two and a half day Leadership Academy, and two additional meetings during the school year (please see the Appendix for slides from a leadership team meeting). In addition, personnel in participating districts (i.e., teachers, classroom coaches, behavior specialists) are encouraged to attend a series of state-led training sessions that will prepare them to implement the Pyramid Model at the school level. These trainings include sessions in Pyramid Practices, PTR-YC, and using the TPOT. Each district is also assigned one of eight external

11 | P a g e

coaches hired by the state; the external coaches are expected to make contact with their district teams at least monthly to support their implementation efforts. External coaches meet with districts either in-person or virtually, and record all of their substantive interactions with their districts. External coaches also receive state level training throughout the year to help prepare them for their roles, and are expected to attend other trainings with their district teams. In addition to these activities, the state offers other Pyramid-related trainings throughout the year, such as a train-the-trainer series to help build district level capacity for sustaining the initiative.

Table 1 below shows the training and support sequence for participating districts.

Table 1. SSIP Training Sequence

ACTIVITY

TARGET AUDIENCE

Leadership Teams

External Coaches

Practitioners (Teachers)

Classroom Coaches

Behavior Specialists

Kick-off Webinar

Leadership Team Academy (2.5 days)

TPOT (Teaching Pyramid Observation Tool) Training (2 days)

Five, 1-hour external coach training calls (virtual)

Pyramid Practices Training (Part 1)

Pyramid Practices Training (Part 2)

Prevent, Teach, Reinforce – Young Children (PTR-YC) Training (1 day)

Leadership Team Mid-Year Meeting (1 day)

Leadership Team End of Year Meeting (1 day)

Provision of Ongoing District Support (bi-monthly, on-site or virtual)

The main features of the SSIP, as they align with this training and support sequence, are described below, with respect to fidelity to the original plan presented in Massachusetts’ Phase II SSIP report (FFY 2014 SPP/APR, pages 70-131).

National Expert Support

MA ESE contracted with the Pyramid Model Consortium in 2014 to support the implementation of the Pyramid Model in the state—this consortium includes developers of the model, and as such, consortia members are considered national experts in Pyramid strategies. MA ESE worked with the Consortium in designing the scope and sequence of statewide training activities. In addition, experts from the Consortium facilitated the Beginning, Middle, and End of Year Leadership Team Meetings, ensuring fidelity to model practices remained at the core of district-level planning. For example, in the fall of 2016, MA ESE hosted a one-day beginning of the year leadership team meeting for returning districts focused on using data for implementation planning including family engagement with the Pyramid Model. This meeting also incorporated a “Coaches Day” component, which consisted of a half-day coaches’ session for classroom-based coaches on the practice-based coaching model, and a half-day session for MA ESE-funded external coaches on how to support program wide implementation of the Pyramid Model.

12 | P a g e

The Consortium also provided TPOT, PTR-YC, and Practices training to groups including classroom and external coaches, practitioners (teachers), and behavior specialists. In effort to further build capacity within the state to continue and extend the Pyramid model, MA ESE-funded external coaches also received coaching support and training from a national trainer including the opportunity to co-train on the practices modules and receive feedback. External coaches participated in more than 10 hours of coaching calls to support their delivery of the Pyramid Model practices modules.

Statewide Coaching

MA ESE funded eight external coaches with Pyramid Model experience to support participating districts’ implementation of the model in their schools and classrooms. External coaches participated in all statewide training activities for districts and their personnel, and co-trained on the practices modules. External coaches also received coaching and training from Pyramid Model consortium and MA ESE staff. These experiences were designed to help prepare the external coaches for the provision of embedded support to the 19 districts working toward implementation, referenced below.

District Leadership Teams

Nineteen districts were selected for participation in the SSIP activities based on an assessment of their capacity for implementation, as well as readiness to take on a substantial initiative. More information about the selection process can be found in the Phase II report on page 105. Once on board, district leadership teams were formed within the 19 participating districts to include representation from both administration and practitioner levels, as well as to bring district and school level perspectives to the work. Each team consists of a district-based administrator, a special education administrator, an early childhood administrator or principal, a teacher, someone with behavior expertise, and a classroom coach for the district. District leadership teams participated in beginning (i.e., Leadership Academy), middle, and end of year leadership team meetings facilitated by MA ESE.

Training Practitioners

At the core of building statewide infrastructure for implementing PBS through the Pyramid Model is the provision of high-quality trainings to build capacity at the local level. The SSIP has offered multiple statewide trainings in Pyramid Model Practices, TPOT, and PTR-YC. More than 300 classroom teachers, external and classroom coaches, behavior specialists, and community members across the 19 participating districts have taken part in training since SSIP implementation began in spring 2015.

Training District-Based Classroom Coaches

District-based classroom coaches receive additional training and preparation to work with their district teams and teachers to implement Pyramid strategies with fidelity. In the fall of 2016, MA ESE hosted a half day “Coaches Day” training on practice based coaching for classroom coaches; approximately 30 people attended. External coaches are also providing on-site support to classroom coaches, including co-scoring TPOTs, helping them to develop implementation plans, and creating coaching plans.

13 | P a g e

Infrastructure Improvements

The MA ESE continues to leverage and extend a wide range of other statewide activities to promote and support the Pyramid Model in Massachusetts. These activities are described below with respect to the progress that has been made since the Phase II report was submitted in April 2016; references to relevant page numbers within the Phase II report are also provided.

Intra-and Inter-Agency Initiatives (FFY 2014 SPP/APR, page 96)

Pyramid Model Trainings for Families – Positive Solutions: the Massachusetts Parent Training and Information center has been awarded a contract to significantly expand a train the trainer model for Positive Solutions for Families, a Pyramid Model family training program.

Outreach to Approved Private Special Education schools and collaborative for students who are Deaf and hard of hearing: MA ESE has provided support to the MA Commission for the Deaf and Hard of Hearing (MCDHH) to work with national Pyramid Consortium staff and American Sign Language (ASL) experts to leverage the Pyramid model and Positive Solutions to increase language for children who use ASL.

Pyramid Model State Leadership Team and Pyramid State Summit: MA ESE has provided funding and support to the MA DPH for the following projects:

Host four Communities of Practice (CoP) across the state to promote community implementation of the Pyramid Model in an effort to support young children and their families in public preschools, Head Start programs, community child care, and Preschool Expansion Grant (PEG) programs.

Increased attendance at the fourth annual MA Pyramid Statewide Summit in spring 2016 – 125 attendees. The Summit also included a presentation by one of the 18 SSIP districts and external coaches.

Professional Development and Guidance (FFY 2014 SPP/APR, page 98)

Supporting Early Literacy Skills of Young Children with Disabilities: integrating language and literacy with social/emotional competencies in everyday activities.

Promoting Inclusive Opportunities for Children with Disabilities: collaborative effort, building on Pyramid Model Implementation to promote inclusive opportunities for children with disabilities in community based preschool programs.

Early Childhood Personnel Action Plan Collaboration with MA DPH and MA Early Education and Care (EEC) to develop a plan aligned

across Parts C and B Draft of Part B plan created in 2016 with the support of national TA center; three focus areas:

leadership, coordination, and sustainability Professional Development Statewide Survey

In January of 2017, MA ESE conducted a survey of special educators, including teachers and administrators about their professional development (PD) needs

More than 2500 responses will shape MA ESE PD offerings moving forward

14 | P a g e

Training on the new Massachusetts Standards for Preschool and Kindergarten in the Domains of Social and Emotional Learning

MA ESE offered a train the trainer model in the spring of 2016; participants were required to offer the training again for free within one year of training

SPP/APR Indicator Data Training New Indicator 7 recorded instructional videos for the web, one on the COS process and one on

data submission requirements and how to use the SmartForm Use of Early Childhood Technical Assistance Center (ECTA) modules on the COS process for

Indicator 7 training New Reusable Learning Objects (RLOs) to support data collection activities

Data Systems (FFY 2014 SPP/APR, page 100)

MA ESE and MA DPH executed a Memorandum of Understanding (MOU) to share aggregated data for students in Part C who were referred to Part B, 619 with the intention to begin analysis of services, eligibility, and longitudinal analysis. Further, state Information Technology infrastructure has renewed its efforts to design a comprehensive, interagency statewide system for assigning unique identifiers for students throughout the Commonwealth.

Grants and Fiscal Support (FFY 2014 SPP/APR, page 100)

New Memorandum of Understanding with MA DPH to increase data sharing activities.

Accountability and Monitoring (FFY 2014 SPP/APR, page 101)

Alignment of the SSIP and the Every Student Succeeds Act (ESSA) plan – MA ESE is leveraging the SSIP and ESSA plans in order to:

articulate a coherent vision for students with disabilities support continuous improvement efforts at the local level provide differential approaches of technical assistance for districts leverage stakeholder engagement support alignment of evidence-based practices (EBPs) promote capacity-building for teachers and administrators at the local level to improve

outcomes for students with disabilities deepen collaboration at the SEA to develop and employ plans for improving student outcomes

and aligning resources MA ESE increased internal collaboration to support districts that are in accountability Level 4 by

working with all internal staff and stakeholders and District Support and Accountability Centers (DSAC).

(b) Intended outputs that have been accomplished as a result of the activities

The MA ESE carried out all principal SSIP activities described above as established by the timeline set in Phase II, and with each intended target audience as planned. All activities conducted between May 2015 and January 2017 are displayed in Table 2 on the following page. As shown in the table, some training events were offered on a “rolling” basis as districts joined the project. For example, the Leadership Team Academy, which was a start-up event for each district, was conducted three times, including May 2015, September 2015, and October 2015. It is also worth noting that the MA ESE has chosen to include information and data in this report dating

15 | P a g e

http://www.doe.mass.edu/sped/training.html

http://ectacenter.org/eco/pages/training_resources.asp#COSFTopics

http://www.doe.mass.edu/sped/ta/indicator7-training-videos.html

back to spring 2015, since the implementation of the SSIP began early and this report represents the first opportunity to document outputs and outcomes of the work. With respect to the intended outputs of these activities, the last column in the table indicates the number of participants who took part in each meeting or training session, or other relevant information that demonstrates progress toward implementation. Finally, evaluation data were also collected across each of these activities as specified in the Phase II report. The full evaluation plan is presented in Section C; evaluation findings are presented in Section E.

Table 2. SSIP Implementation Progress (May 2015 – January 2017)

DATE ACTIVITIES AUDIENCE OUTPUTS/PROGRESS

Ongoing Communication with Pyramid Model Consortium Staff

N/A Ongoing communication; on-site trainings conducted as planned

Ongoing External Coach Contact Records completed as substantive contacts with districts occur

N/A 59 External Coach Records logged by external coaches from 9/28/15 to 1/24/17

Ongoing Site Visits by MA ESE Staff at participating districts

District Leadership Team, External Coaches, Teachers

Visits underway as of 2016-17 school year; conducted in conjunction with external coaches

5/20-5/22/15 Leadership Team Academy (2.5 Days) District Leadership Team and External Coaches

5 districts participated

6/29-6/30/15 Pyramid Practices Training Part 1 (Days 1 and 2)

Practitioners (teachers), Classroom Coaches, Behavior Specialists, and External Coaches

Approximately 30 participants took part in the four day series


9/22/15 Kick Off Webinar – District leadership teams from May 2015

District leadership teams and External Coaches

Approximately 5 districts participated (follow-up from May 2015 meeting)

9/30-10/2/15 Leadership Team Academy (2.5 Days) District leadership team and External Coaches


10/2/15 Positive Solutions Train-the-Trainer External Coaches and Community Trainers

17 individuals participated

10/21-10/22/15 TPOT (Teaching Pyramid Observation Tool) Training (2 Days)

Classroom Coaches and External Coaches

28 participants including community partners

10/27-10/29/15 Leadership Team Academy (2.5 Days) District leadership team and External Coaches


11/5, 11/12, 11/19, 12/3, & 12/10/15

Practice-Based Coaching: Five, 1-hour training calls – Virtual


Approximately 10 participants



Approximately 90 participants took part in the four day series


2/1-2/2/16 TPOT (Teaching Pyramid Observation Tool) Training (2 Days)


Approximately 30 classroom coaches trained

2/26/2016 Prevent/Teach/Reinforce Young Children (PTR-YC) Training (1 Day)

Behavior Specialists and External Coaches


3/1/16 Leadership Team Mid-Year Meeting (1 Day)

District Leadership Teams and External Coaches


16 | P a g e

DATE ACTIVITIES AUDIENCE OUTPUTS/PROGRESS

3/3-4/14/16 Practice-Based Coaching: Five, 1-hour training calls – Virtual





Approximately 30 participants took part in the 3 day series

4/25/16 Pyramid Practices Training Part 2 (Day 3)


Behavior Specialists, Teachers, and External Coaches


6/2/16 Leadership Team End of Year Meeting (1 Day)



9/26-9/27/16 Beginning of the Year Leadership Team Meeting & Coaches Day



11/15-12/12/1611/18-12/15/16

Pyramid Model Practices Training (Offered twice, 3 days for each course section)


Approximately 24 participants across two course sections


Behavior Specialists and Teachers


2. Stakeholder Involvement in SSIP Implementation

This section of the report addresses both:

(a) How stakeholders have been informed of the ongoing implementation of the SSIP, and (b) How stakeholders have been involved in decision-making regarding ongoing implementation

By design, the MA SSIP provides multiple opportunities for key stakeholders at the state, district, program/classroom, and community levels to learn about the SSIP such as project philosophy and goals, the provision of project training events and embedded supports, and the availability of resources to support and extend implementation. Stakeholders also have occasion to participate in decision-making about the direction of the project as it progresses. Key aspects of this design include sharing information and gathering feedback, via state-level leadership and steering committee meetings, with stakeholders who are working toward the common purpose of improving the social and emotional outcomes for preschool children; and by engaging in ongoing feedback sessions with the participants (administrators, teachers, community members) who comprise the SSIP districts. Specific mechanisms for communicating with and responding to feedback from stakeholder groups are described below.

State Pyramid Leadership Team

MA ESE staff play a significant role on the state’s Pyramid State Leadership Team (SLT), which typically meets for three hours each month to collaborate on initiatives to support the adoption of Pyramid strategies throughout the state. This team of approximately 12 individuals also includes representatives from the MA DPH, MA EEC, and the Department of Mental Health (MA DMH), Connected Beginnings Training Institute (CBTI) at Wheelock College, University of Massachusetts (UMASS) Boston, and the Head Start Technical

17 | P a g e

Assistance agency. Through MA ESE’s active role on this team, progress on the SSIP is communicated to this broad group of stakeholders whose feedback helps to refine MA ESE’s efforts, and opportunities for collaboration are identified. Involvement in the SLT has been instrumental in expanding MA ESE’s work to a larger audience, and refining plans to identify model Pyramid sites in the coming year.

Special Education Advisory Council (SAC), and Special Education Steering Committee

The SSIP Leadership Team’s participation on other statewide working groups provides opportunities for MA ESE to connect with stakeholders about the SSIP. The SAC is comprised of members appointed by the Commissioner on behalf of the MA Board of Education. Over half of the voting members are individuals with a disability or a parent of a child with a disability. MA ESE personnel provide updates about the SSIP to the SAC at least annually in order to solicit feedback and provide updates. The Special Education Steering Committee provides another venue for communicating progress on the SSIP. This group is composed of members of the SAC plus advocacy organizations, other state agencies, and MA ESE representatives from the Office of District and School Turnaround (ODST), Systems for Student Success Office (SSSO), Special Services (including Special Education), System Accountability, Data and Accountability, Center for Curriculum and Instruction, and Student Support. During a statewide meeting in January 2017, MA ESE presented information about the activities and progress of the SSIP. Through this presentation, MA ESE provided updates about the evaluation plan, implementation progress, and solicited feedback and input about future directions and additional opportunities for collaboration.

District Leadership Team Survey Feedback

The 2016 Mid-Year Leadership Team Survey was completed during February and March by 52 leadership team members across 18 districts. By responding to the survey, leadership team members indicated their progress toward implementation of the Pyramid Model, as well as ongoing support needs from both External Coaches and MA ESE. Based on this feedback, MA ESE provided external coaches with additional support to continue to prepare them to respond to districts’ areas of need identified in the survey. MA ESE also modified its training schedule in response to feedback. Selected results from the survey are presented in Section E.

Participant Feedback at Trainings

MA ESE reviews feedback forms completed by participants at each training, and at statewide meetings for leadership teams and external coaches. The feedback forms allow MA ESE to gauge the general quality of the sessions, usefulness of the information, and perhaps most importantly, areas for improvement. Some of the changes that have been made in response to this feedback include reducing the duration of the Practices training from four full days to three days; as a result, the new format has been better received by participants. In addition, many participants indicated that traveling to a statewide training was a challenge. Going forward in 2016, MA ESE modified its plan to include regional trainings to reduce travel time. Finally, the content of the Practices training was modified slightly based on participant input.

External Coach Feedback

External coaches have a key role in supporting district and school personnel in moving toward implementation of PBS through Pyramid strategies. Since these individuals are making direct contact with district teams on a regular basis, they are in the best position to determine some of the ways the initiative might be improved at the local level, and to understand challenges to implementation. MA ESE maintains ongoing communication with coaches primarily through regular calls and emails. MA ESE also established external coaching expectations in collaboration with coaches in September 2016 to help guide their work going forward.

18 | P a g e

Some of the improvements that have been put into place based on feedback from coaches include Professional Learning Communities (PLSs) for external coaches to allow for sharing tips and strategies among their group—PLCs began monthly starting in December 2016. Also, revisions have been made to the External Coach Contact Record format to better capture the information about the technical assistance they provide. Finally, MA ESE staff have begun to accompany external coaches on some of the district site visits in an effort to increase SEA/LEA collaboration; provide additional training and TA for data collection efforts, specifically Indicator 7; and support the development of external coaches.

19 | P a g e

C. Data on Implementation and Outcomes

1. How the State Monitored and Measured Outputs to Assess Effectiveness

During Phase II of the SSIP, the MA ESE worked closely with internal partners and key stakeholders to develop a comprehensive evaluation plan for the SSIP to be carried out during Phase III. Data collection instruments were also developed and/or selected at that time. This evaluation plan was presented in the Phase II report – please see pages 112-129 of MA’s FFY 2014 Part B SPP/APR – and has largely been carried out as planned over the past year and a half of SSIP implementation.

In February of 2017, MA ESE contracted with an external evaluator to review and analyze the evaluation data collected to date, and to contribute to the writing of this report with respect to evaluation findings related to both progress toward implementation, and outcomes. MA ESE will continue to work with the external evaluator going forward to refine the evaluation plan and data collection procedures as the SSIP implementation evolves.

(a) How the evaluation measures align with the Theory of Action

The SSIP Theory of Action articulates activities at the state level with respect to infrastructure for principal activities, as well as ongoing interagency collaboration to support the SSIP. The next level of activities engages districts/programs, followed by classrooms and students. Across these four levels, the plan incorporates key questions to help focus the evaluation, as well as appropriate short, intermediate, and long term outcomes to assess progress and impact of the SSIP. Data collection instruments have been selected or developed, and processes for collecting data at regular intervals have been put into place.

Table 3 below displays the SSIP evaluation plan, including key questions at each level of the Theory of Action, as well as the intended outcomes and data sources.

Table 3. MA SSIP Evaluation Plan

EVALUATION QUESTIONS INTENDED OUTCOMES DATA SOURCES

State Level Infrastructure

EQ1aIn what ways is MA ESE using the SSIP, including statewide implementation of PBS through Pyramid strategies, to build state-level capacity to support improved social emotional outcomes for young children with disabilities?

S1. Short Term and IntermediateIn order to build state capacity, MA ESE will…

a. leverage the cadre of PBS external coaches to support districts and communities;

b. collaborate with community and social services agencies to provide additional training and support to families;

c. engage in ongoing collaboration with colleagues in Part C and K-12 PBIS initiatives to build community liaison and data sharing to promote effective transitions and improve social emotional outcomes.

Extant project documents (e.g., state and district meeting notes, inter-agency meeting minutes, implementation notes and planning tools)

Statewide demographic training data (i.e., internal project records of participation numbers)

Statewide training evaluation forms

EQ1bTo what extent is implementation of PBS through Pyramid strategies in MA integrated with other early

S2. IntermediateMA ESE will…

a. support the implementation of the newly created PBS/PBIS crosswalk designed to


20 | P a g e


childhood and/or MA ESE initiatives at the community/local and state levels?

promote state level collaboration to create a seamless model of Positive Behavior Supports (PBS) for preschool through secondary schools; and

b. will engage in ongoing collaboration to continue to identify strategies and actions to promote local level integration of PBS.

Program/District Infrastructure

EQ2aIs the state-level plan resulting in the number of schools and classroom participating in PBS through Pyramid strategies growing over time?

S3. Long TermMA ESE will provide adequate training in Pyramid Model strategies so that local school district leadership teams, teachers, and classroom coaches are participating in Pyramid Model trainings provided by district classroom coaches and other staff.

D1. Short TermDistrict administrators and educators will participate in statewide trainings on PBS through Pyramid strategies including Leadership Team Academies, Practices Trainings, and Coaches’ Trainings to become familiar with the tenets of PBS and PBS through Pyramid strategies and classroom and program-wide implementation.

External Coach Contact Records Benchmarks of Quality (BOQ)

assessments Mid-year Leadership Team Survey Statewide demographic training

data Statewide training evaluation

forms Extant documents (e.g., state and

district meeting notes, inter-agency meeting minutes, implementation notes and planning tools)

Pyramid Model Implementation Database

EQ2bAre districts developing systems to support sustainable training and coaching practices at the local level?

D2. IntermediateThe participating districts have built sustainable internal capacity to train additional teachers to implement Pyramid strategies in their classrooms and sustainable improvement plans to support implementation of PBS through Pyramid and PBIS strategies.

External Coach Contact Records BOQ assessments Mid-year Leadership Team Survey

Classroom Level

EQ3aAre teachers implementing PBS through Pyramid strategies in their classrooms?

D2. IntermediateThe participating districts have built sustainable internal capacity to train additional teachers to implement Pyramid strategies in their classrooms and sustainable improvement plans to support implementation of PBS through Pyramid and PBIS strategies.

External Coach Contact Records Mid-year Leadership Team Survey TPOT training session data Benchmarks of Quality

assessments Pyramid Model Database

EQ3bDoes the fidelity of classroom implementation improve over time?

D3. Long TermTeachers will be able to implement PBS through Pyramid strategies with fidelity to improve the social/emotional development of young children with disabilities.

TPOT (the feasibility of state access to summary data to be explored going forward)

Student Level

EQ4aIs the number of children in MA, aged 3-5, with disabilities, exiting

C1. Long TermChildren with disabilities, aged 3-5, will exit preschool with social/emotional competencies that

Indicator 7 data - Child Outcomes Summary (COS): Summary Statement 2

21 | P a g e


from preschool with age-expected social emotional functioning increasing?

will allow them to access and participate in the general curriculum and in all aspects of the school.

EQ4bIs the number of children in MA, aged 3-5, with disabilities, exiting from preschool with greater than expected growth in their social emotional functioning increasing?

Indicator 7 data - Child Outcomes Summary (COS): Summary Statement 1

(b) Data sources for each key measure

Data sources are described below. All data are collected by the MA ESE for its own internal review, and will also be shared with the external evaluator for analysis and reporting.

Extant Project Documents

Project personnel maintain and collect SSIP documentation that includes ECSE Leadership Team meeting notes, inter-agency meeting minutes, implementation notes and planning tools, training participation data, and other details. This documentation is used to facilitate internal planning and tracking of progress, as well as for review and analysis by the external evaluator to help determine progress toward intended outcomes of the initiative.

District Benchmarks of Quality (BOQ)

The Pyramid Model Consortium has developed a Benchmarks of Quality document for programs and districts implementing the Pyramid Model. This self-assessment identifies 47 benchmarks of quality across nine Critical Elements on which programs can assess their implementation. Participating districts complete the BOQ when they attend initial Pyramid Model Leadership Team Academies. Thereafter they complete the tool at least twice a year: at the beginning and end of the school year. External coaches support the district leadership team in the completion of this measure and can provide additional guidance on the meaning of each element and what it means for each element to be in place, the criteria for fidelity on the tool. This support by coaches in intended to promote more accurate self-assessment by each participating district.

Teaching Pyramid Observation Tool (TPOT)

Hemmeter, Fox, and Snyder (2013) have developed and published a classroom implementation fidelity tool for the Pyramid Model called the Teaching Pyramid Observation Tool for Preschool Classrooms (TPOT). The TPOT uses a two hour classroom observation and subsequent interview with a teacher to score implementation of key Pyramid Model practices in 14 areas, identify the presence of red flags during instruction or in the classroom, and understand an educator’s response to challenging behavior. With respect to measures of implementation fidelity in classrooms throughout the SSIP districts, the TPOT is currently used by classroom coaches and teachers for purposes of teacher reflection and improvement. The goal is for each coach and teacher to complete the TPOT process at least once each year. External coaches and classroom coaches received training in TPOT administration, to ensure reliability with the process.

External Coach Contact Record

22 | P a g e

A critical component of the Pyramid Model is ongoing practice-based coaching of educators implementing the model. MA ESE has developed an extensive External Coach Contact Record for external coaches under contract with MA ESE. External coaches complete this online form each time they have a substantive contact with a district. This measure captures information including the mode of contact (e.g., site visit, phone call, email, etc.), the type of support provided, the number of individuals trained, and the status of the district’s current Pyramid Model implementation. This tool provides ongoing information about the kinds of support districts need, as well as an ongoing account of the numbers of classrooms and teachers trained, and numbers of classrooms that have adopting the Pyramid Model. The data collected are coded and analyzed for themes. These aggregate analyses will inform decisions about training topics, training frequency, and support for external coaches.

Training Evaluation Forms

MA ESE, in collaboration with the Pyramid Model Consortium, designed brief Training Evaluation Forms for use at the conclusion of all training sessions and meetings. The evaluation forms are designed to assess the quality and usefulness of the sessions, and to solicit suggestions for improvements going forward. The data are used internally for ongoing refinement of the trainings and meetings, and will also contribute to documenting progress with respect to the overall evaluation.

Mid-Year Leadership Team Survey

MA ESE designed the Mid-Year Leadership Team Survey to capture district teams’ assessment of their progress toward implementation, the contribution of external coaches, challenges, and their specific needs for assistance from MA ESE and external coaches as they move forward. The online survey is distributed annually to all leadership team members.

Pyramid Model Implementation Database

In 2017, MA ESE created an electronic form for use by external coaches to record each district’s implementation status. The form captures the total numbers of schools and classrooms with preschool programs in the district, as well as the numbers of schools and classrooms implementing the Pyramid Model. Comments are also captured throughout, to provide additional information about the particular circumstances of each district. The objective of the database is to gather a snapshot of the depth and breadth of Pyramid in each district at different points in time. These data are collected using Google docs, and will be updated by external coaches twice a year – at the beginning and end of each school year.

Indicator 7 data

SPP Indicator 7 data collected via the Child Outcomes Summary (COS) are collected annually by SSIP districts. Presently, the SSIP team is working toward collecting Indicator 7 data in a targeted way that will allow the evaluation to assess progress toward the SIMR for implementing classrooms specifically, in addition to statewide results.

(c) Description of baseline data for key measures

Overview of Participating Districts

23 | P a g e

MA ESE originally recruited 18 districts1 to participate in the first year of the implementation of the Pyramid Model as part of the SSIP beginning May 2015 with one additional district joining in September 2016. While these 19 districts represent only a fraction of the more than 400 districts with preschool programs in Massachusetts, at the time of the initial implementation, they educate more than 21% of the total number of public school students with and without IEPs enrolled in preschool through grade 12 in the Commonwealth. In addition, these 18 districts are responsible for educating 3,041 preschool children with IEPs or 28.3% of the total statewide preschool population with IEPs. The students served by these districts are primarily educated in a fully inclusive environment (42% of students) or a partially inclusive environment (26%). Approximately 22% of students in the participating districts are educated in a substantially separate setting and the remaining 10% receive their special education services in the home or at a service provider’s office. Similar to the statewide demographics, preschool students with disabilities in the 18 participating districts fall into three primary disability categories: developmental delay (46%), communication (33%) and autism (21%), with the remaining 6% representing other low incidence disability categories.

The full set of 19 participating districts are geographically dispersed throughout the state of Massachusetts but a majority of them are located in the eastern half of the state. While many of the largest districts in the state are included in this sample, there are also several smaller districts serving approximately 2,500 students in total and both rural and urban populations are represented. Lastly, the 19 districts represent communities of diverse economic means, including ones that serve predominately families who are identified as economically disadvantaged and those that serve more affluent communities.

Training Participation

Training data collected as part of this study provide information on the scope and sequence of training activities during the study period. By examining the order of training activities, participant engagement as measured by the number of districts attending, and the revisions to the training schedule based on participant and national trainer feedback, these data provide information on the ways in which MA ESE approaches statewide implementation. In addition to the trainings outlined in the SSIP Phase I implementation plan, MA ESE identified the need for additional trainings for the external coaches. MA ESE provided individual and group coaching opportunities for the external coaches to develop their ability to provide statewide trainings in the Pyramid Model practices. These trainings were co-led by MA ESE staff and a Pyramid Model Consortium trainer with expertise in the Pyramid Model. In addition, each external coach had at least one opportunity to participate in the Pyramid Model Practices Training through a co-training model with the national trainer to further develop their presentation skills. Each coach participated in a private debriefing meeting or phone call with the national trainer and MA ESE staff after they were co-trained. With respect to feedback on the quality and usefulness of the training sessions, baseline ratings across a sample of trainings are presented in Section E.

1 As indicated previously, one additional district was brought onto the project in 2016-17 based on its prior experience with the Pyramid Model and readiness to begin implementation, bringing the present total of participating districts to 19. While this district is not represented in the demographic profile presented here, the district did participate in data collection activities, and is represented in evaluation findings presented throughout this report.

24 | P a g e

District Benchmarks of Quality

The Pyramid Model Benchmarks of Quality (BOQ) is a self-assessment that identifies 47 benchmarks of quality in nine Critical Elements across which programs can assess their implementation on a scale of not in place, partially in place, or in place. Created by the Pyramid Model Consortium, this tool allows programs and districts implementing the Pyramid Model to assess their own progress over time and provides the state with data on programs’ implementation status, including current challenges. Eighteen participating districts completed the Benchmarks of Quality at one of the three initial statewide Leadership Team Meetings which occurred in May 2015, September 2015, and October 2015. Districts reassessed their implementation of the model at the Midyear Leadership Team Meeting in March 2016 and again at the End of Year Leadership Team Meeting in June of 2016. Baseline data on this measure suggest that districts are having the most success toward implementation in the following Critical Areas: Staff Buy-in, Establish Leadership Team, and Program-wide Expectations. The areas in which districts have the most work to do going forward include: Family Involvement, Strategies for teaching and acknowledging program-wide expectations, monitoring implementation and outcomes, and Staff Support Plan. Full results from the BOQ are presented in Section E.

Mid-year Leadership Team Survey

The Mid-year Leadership Team Survey was administered in March 2016, during the first year of implementation. The 2017 version of the survey is currently being completed by district teams, and will serve as a basis for comparison. Baseline data on this measure suggest that early in the project, districts were beginning to make progress bringing Pyramid strategies to their schools—at the time of the 2016 survey, 62% reported making “a little progress”, 32% reported “moderate progress”, and 6% felt their teams had made “significant progress”. District team members indicated some of their specific needs for moving forward with implementation, such as more facilitated meeting time with their teams, and additional Pyramid Practices training for teachers in their districts. Section E provides more details about these findings.

(d) Data Collection Procedures and Associated Timelines

The data collection procedures described above are carried out per the schedule shown in Table 4.

Table 4. Data Collection Plan

DATA COLLECTION TOOL/METHOD PROCESS TIMELINE

MA ESE meeting notes Collected at each meeting Ongoing

Training Evaluation Forms Collected by participants at conclusion of each training Ongoing

Meeting Evaluation Forms Completed by attendees at conclusion of each meeting Ongoing

Benchmarks of Quality (BOQ) Assessment Completed by leadership teams and external coaches at leadership meetings Bi-Annually

External Coach Contact Records Completed via electronic form by external coaches after each substantive contact with a district Ongoing

Mid-year Leadership Team Survey All leadership team members invited to complete online survey annually

Annually in March

Pyramid Model Implementation Database Completed by external coaches Bi-Annually

TPOT Collected at the local level for teachers implementing Pyramid Annually

Indicator 7 Student Outcomes Districts began collecting data for Indicator 7 for new students in classrooms implementing the Pyramid Model in the fall of 2016

Annually in spring

25 | P a g e

(e) Sampling Procedures

Sampling procedures are not being been used in the collection of evaluation data. All data used to assess progress toward implementation and to assess outcomes have been conducted with all participants as appropriate, including training evaluation forms, surveys, and documentation of technical assistance provided by external coaches. Indicator 7 data are currently collected statewide, while approaches to collecting program-specific data (i.e., school and/or classroom level) are in progress for spring 2017.

(f) Planned data comparisons

A number of data sources will be analyzed in comparison to either earlier data collected across the same measures, or in comparison to statewide data (i.e., Indicator 7). These comparisons include the following:

Results from the BOQ assessment have been compared across beginning, middle, and end of year assessments. Going forward, the BOQ will be completed by district teams bi-annually, and results will continue to be compared over time in the aggregate. At the local level, district teams, in coordination with external coaches, will compare their own results over time to assess progress toward the Critical Elements of Implementation.

The Mid-year Leadership Team Survey is being administered to all district team members annually. Results on this survey in the aggregate across all participating districts will be compared over time to assess progress toward implementation, as well as the progression of successes and challenges.

Indicator 7 data collected via the Child Outcomes Summary (COS) are used to address the two key evaluation questions that relate to the SIMR, which are aligned with Summary Statements 1 and 2. Indicator 7 data are currently being reported for students statewide (using the state’s cohort model; see FFY 2014 SPP/APR page 43), and are compared over time. Going forward, the project seeks to collect Indicator 7 data annually from all classrooms within the SSIP cohort that are implementing the Pyramid Model, which will be used to assess year-to-year progress across the 19 participating districts.

(g) How data management and data analysis procedures allow for assessment of progress toward achieving intended improvements

MA ESE has put in place several data collection procedures during the implementation phase that allowed an assessment of progress early in the project, and that will continue throughout Phase III. For example, External Coach Contact Records record the method (e.g., in person, phone, etc.) and focus of each interaction between coaches and their assigned district, allowing for real time updates of implementation activities in district. The documentation of that work has improved over time, as the contact record is refined and coaches become more accustomed to keeping records.

The online Mid-Year Leadership Team Survey captures the implementation status and challenges that may impede achieving intended outcomes. By reviewing these results as soon as they are received, the MA ESE is able to address the needs identified by district team members with respect to the content and format of future training sessions and leadership meetings, as well as in supporting external coaches so that they can respond to district needs.

Training Evaluation Forms, distributed at the conclusion of each training session and statewide meeting allow MA ESE to modify training as needed to meet the needs of participants. This feedback has been extremely

26 | P a g e

useful for informing MA ESE about the effectiveness of the sessions; refinements continue to be made to the content, logistics, and format of the sessions.

Finally, maintaining meeting notes from both inter- and intra-agency meetings and internal implementation progress logs allows the MA ESE team to track and review their own progress toward SSIP milestones and goals, and the intersection of the SSIP work with the larger work in Massachusetts.

All data sources are also being shared with the external evaluator for end of year reporting and for sharing recommendations for adjustments to the evaluation plan and procedures in subsequent years of the project.

2. How the State has Demonstrated Progress and Made Modifications to the SSIP

(a) How the State has reviewed key data that provide evidence regarding progress toward achieving intended improvements to infrastructure and the SIMR

MA ESE reviewed data on an ongoing basis to address any areas of concern, and to make improvements mid-course. For example, MA ESE reviewed External Coach Contact Records to understand the extent to which districts are implementing the Pyramid Model, the ways in which they are using data for decision making, and the challenges they are facing. This information has been used to inform the ways in which external coaches and MA ESE can best support districts going forward.

Furthermore, results from the 2016 Mid-Year Leadership Team Survey allowed MA ESE to target training for external coaches, modify the training schedule and locations, and to provide additional supports to district teams based on need, including the implementation of site visits by MA ESE which began in the 2016-17 school year. MA ESE also regularly reviews training documents to determine the ways in which the content, method of delivery, and/or timing need to be modified to best meet participants’ needs.

MA ESE has made several other changes in response to its review of data toward achieving improvements that are designed to meet the SIMR, which include the following:

A review of district and school implementation status, provided by external coaches, was used to inform activities planned for next year.

PLCs have been put in place with external coaches to allow for greater collaboration, and to inform changes MA ESE needs to make to best support coaches

Information about the frequency of TPOTs in district supports MA ESE’s modification of the support external coaches are providing (based on this information, external coaches now co-score at least one TPOT with each classroom coach to ensure reliability among the ratings).

(b) Evidence of change to baseline data for key measures

While the data for most key measures as this stage represent baseline, there is evidence of change that can be noted in three areas. With respect to implementation at the state level, an analysis of training evaluation ratings suggested that the quality and usefulness of the sessions has improved over time. At the district level, the BOQ Assessment revealed substantial progress toward fidelity by participating districts from the initial to the third BOQ completion over the first 18 months of the initiative. Growth was reported across all nine Critical Elements measured, suggesting that district teams have made notable progress. Finally, at the student level, SIMR data aligned with Summary Statements 1 and 2 revealed mixed results over baseline (2013) and 2014 data (these results are shown on page 1 of this report). For Summary Statement 1, growth was documented

27 | P a g e

from 2013 to 2014, but outcomes decreased in 2015. For Summary Statement 2, growth has been documented in both 2014 and 2015 over baseline.

(c) How data support changes that have been made to implementation and improvement strategies

The MA ESE made a number of modest changes to their SSIP implementation strategy based on stakeholder feedback. These changes were all based on a review of evaluation data, and ongoing conversations with key stakeholders, and included the following:

Increased internal alignment at SEA to include students with disabilities in the ESSA plan, Curriculum and Instruction initiatives, District Support activities, and inter-agency agreements

Reduced the format of Pyramid Practices training from 4 days to 3 days Changed practices training to regional, in-district trainings to make them more convenient for

participants Refined the content of the Practice training to be more sophisticated to better meet attendees’ needs Added a “Coaches Day” to enhance the preparation of classroom coaches and external coaches for

their roles Added a monthly external coaches Professional Learning Community (PLC) to foster collaboration

among the group, and for the state to continue to learn about the progress of the SSIP at the district level

Incorporated data-based decision making into the leadership team meeting in September 2016 Supported increased site visits with external coaches and MA ESE staff Provided aggregate BOQ data to coaches to debrief with their teams for planning purposes Modified external coach contact record based on coach feedback to better reflect the types of support

they are providing Repeated Mid-year Leadership Team Survey to allow for a comparison over time Added additional external coaches for a total of 8 with additional applications pending Offered PTR-YC training more often based on districts’ requests Expanded contract for Comprehensive System of Professional Development (CSPD) coaches that

differentiates between coaching and training to individualize and provide targeted support for districts Expanded the contract with MA’s Parent Training and Information center (PTI), the Federation for

Children with Special Needs (FCSN), to support Positive Solutions in districts Through participation on the MA Pyramid Model SLT, supported inclusive practices through funding

and other initiatives that will include a Pyramid Summit which focuses on district implementation and will feature presentations by participating SSIP districts, and the identification of community based child-care Pyramid Model Sites

(d) How data are informing next steps in the SSIP implementation

With respect to next steps, MA ESE is looking closely at the implementation status and needs of each district based on feedback and information provided by participants and external coaches, as well as through MA ESE site visits. Indicator 7 data, extant data, and qualitative data are expressly driving the plan for future implementation (see Section F).

28 | P a g e

(e) How data support planned modifications to intended outcomes (including the SIMR)

There are no planned modifications to intended outcomes at this time.

3. Stakeholder Involvement in the SSIP Evaluation

Stakeholder involvement in the SSIP evaluation has included ongoing input and feedback from participants at the state, district/program, school, and classroom levels. Much of this information has been shared in Section B (2) of this report, with respect to how stakeholders have been informed about the SSIP, and the ways in which they have provided input toward implementation and evaluation. The following list are examples of the ways in which stakeholders have been directly involved in the evaluation.

(a) How stakeholders have been informed of the ongoing evaluation of the SSIP

Information about the evaluation purpose, timeline, and methods have been shared at leadership team meetings

Data collection procedures and purpose of evaluation have been shared during site visits with districts Evaluation purpose and plan has been shared with the Special Education Steering Committee; the

same information has been shared with the Special Education Advisory Council (SAC) MA ESE staff working on the SSIP have shared through updates to other MA ESE offices such as the

Office of District and School Turnaround (ODST), Systems for Student Success Office (SSSO), Special Services (including Special Education), System Accountability, Data and Accountability, Center for Curriculum and Instruction, and Student Support

Evaluation purpose and plan has been shared at ECSE leadership team meetings Provided evaluation updates at MA Pyramid State Leadership Team meetings Ongoing communication with MA DPH and MA EEC regarding SSIP evaluation as part of alignment with

Part C SSIP

(b) How stakeholders have had a voice and been involved in decision-making regarding the ongoing evaluation of the SSIP

Feedback received from external coaches through External Coach Contact Records, Training Evaluation Forms, and informally throughout the project

Feedback from Leadership Team Meetings Feedback from Mid-year Leadership Team Surveys Special Education Steering Committee meeting feedback SAC meeting feedback Interagency Coordinating Council (ICC) feedback Ongoing work with MA EEC and MA DPH on alignment with Part C SSIP Feedback from participants at all statewide activities via Training Evaluation Forms, as well as

informally Feedback from MA ESE ECSE Leadership Team Feedback solicited from other MA ESE offices Feedback from the Federation for Children with Special Needs (FCSN)

D. Data Quality Issues

29 | P a g e

1. Data Limitations

(a) Limitations related to the quality or quantity of the data used to report progress or results

Several limitations have been confronted with respect to the quality and quantity of data that have been collected and analyzed by project leaders throughout the implementation phase. Examples of these limitations are identified below.

Indicator 7 – Prior to 2017, statewide Indicator 7 data to address the SIMR have been available for a representative sample of districts each year, based on the cohort schedule currently in place in Massachusetts. This data collection and reporting process (i.e., districts reporting on Indicator 7 once every four years depending upon their cohort assignment) has not allowed for the annual assessment of progress toward the SIMR across the 19 targeted districts specifically, in addition to statewide progress overall.

As the project moves forward, MA ESE is requesting that all participating districts provide Indicator 7 data annually for all classrooms implementing Pyramid. In doing so, there will be several considerations to be confronted by the SSIP team and stakeholders such as establishing the criteria used to define “implementing” classrooms, as well as how the state can best support the collection and reporting of high-quality data via the Child Outcomes Summary to inform Indicator 7.

TPOT Assessment – TPOT data are collected at the school level, and are intended to be both a gauge of fidelity of implementation, as well as a teaching tool for teachers and classroom coaches. As such, TPOT data are not provided to the state. That said, districts are not conducting TPOT assessments as often as hoped due to staffing and contract issues.

External Coach Contact Records – Coaches had not been using the coach contact form for each substantive contact in the beginning of the project; MA ESE also needed to refine the form to more closely align with activities coaches were engaged in with leadership teams and educators.

Mid-year Leadership Team Survey – The survey was not completed by all districts in 2016, as some individuals were not able to attend the meeting and did not complete it virtually.

Benchmarks of Quality – The BOQ was not completed by all districts at each leadership team meeting. In addition, there was potentially an overestimation of implementation status, especially at beginning of the year leadership team meeting.

(b) Implications for assessing progress or results

At this time, there are no significant concerns about the quality of quantity of data for assessing progress. However, MA ESE will be actively supporting districts to collect and report Child Outcomes via Indicator 7 on all children with IEPs. MA ESE reviews data from all sources to gain a perspective on what the entire cohort is doing, and what challenges need to be addressed. As MA ESE works with the external evaluator, further refinements to the data collection plan are anticipated.

30 | P a g e

(c) Plans for improving data quality

MA ESE identified a number of ways in which data quality could be improved, which included:

Implementing new Indicator 7 resources including FAQ, SmartForm, and coaching from MA ESE and external coaches

Ongoing revisions of the External Coach Contact Records to collect more detailed information Ongoing revisions of the Mid-year Leadership Team Survey to capture additional information; collecting

data online to improve response rate Promoting on-site support from external coaches to complete at least one TPOT per teacher annually

31 | P a g e

E. Progress in Achieving Intended Improvements

1. Assessment of Progress toward Achieving Intended Improvements

This section of the report addresses MA ESE’s progress toward achieving the intended improvements of the SSIP at the state, district, and school/classroom levels to support PBS through Pyramid strategies. This summary has been prepared by the external evaluator based on a review and analysis of multiple data sources collected by SSIP project personnel, as described in Section C above. Progress findings are organized around the key evaluation questions which serve as the foundation for the SSIP evaluation; supplemental documentation is presented in the Appendix as indicated.

(a) Infrastructure changes that support SSIP initiatives

Evaluation Question 1a: In what ways is MA ESE using the SSIP, including statewide implementation of PBS through Pyramid strategies, to build state-level capacity to support improved social emotional outcomes for young children with disabilities?

Progress Finding: As specified in the SSIP planning and design phases, MA ESE has successfully organized and conducted multiple statewide trainings in evidence-based practices (EBP) to foster implementation of PBS through Pyramid strategies among participating districts and their schools.

In coordination with its partner the Pyramid Model Consortium, MA ESE has conducted a series of trainings since SSIP implementation began in spring 2015—all training events and approximate numbers of participants are shown in Table 2 on page 16. As noted previously, MA ESE has worked with Pyramid Model experts since 2009 with a focus on enhancing the capacity of schools to adopt the model and increasing the number of high quality trainers and coaches statewide. The SSIP has allowed this work to continue with greater focus and district-level support via the establishment of dedicated leadership teams, external coaches, and train-the-trainer events. The trainings conducted as part of the SSIP have been organized around the following topics:

Pyramid Model Practices

This training teaches practitioners how the Pyramid Model promotes social emotional development, relationship-building, targeted supports for at risk children, and individualized intensive interventions for children with persistent challenges. Tools specific to early childhood environments, child observations, and data collection are introduced.

Prevent, Teach, Reinforce Young Children (PTR-YC)

PTR-YC is a model of intervention for young children with the most persistent and severe challenging behaviors. The training is based on positive behavior support strategies, and designed to support fidelity of implementation. Individuals who serve as behavior specialists or coaches in their districts are encouraged to attend.

Teaching Pyramid Model Observation Tool (TPOT)

The TPOT training is geared toward classroom coaches who will be administering the TPOT, an observation tool designed to assess the fidelity of the Pyramid model strategies implementation in the classroom. In addition to a measure of implementation of the Pyramid Model in the classroom, it can also be used to guide

32 | P a g e

teachers in supporting children’s social and emotional development. This training is geared toward classroom coaches and external coaches.

Practice-Based Coaching

Each series of five, one-hour virtual coaches’ calls with national experts is designed to provide training in practice-based coaching and to offer techniques and strategies for classroom coaches as they begin to support teacher implementing PBS through Pyramid strategies. Calls are recorded so that individuals unable to attend can access the content at their convenience. External coaches are also invited to participate.


The Positive Solutions training addresses the skills and strategies necessary for professionals to implement Positive Solutions for Families, an evidence-based parent training series, in their school districts. Professionals learn how to partner with parents of preschoolers to promote positive and effective parenting strategies. In partnering with families, children’s social and emotional development is promoted and challenging behaviors and mental health needs of children in preschool addressed. Positive Solutions aligns with Massachusetts’ priority to enhance family and community engagement and supports improved strategies for working with families of young children in preschool settings.

In addition to these topical trainings, the SSIP team also conducts beginning and end of year meetings with district leadership teams and external coaches to work together to align policies, procedures, and practices to support teachers in using Pyramid strategies.

Progress Finding: Approximately 326 individuals across 19 districts have taken part in the EBP trainings to date. Participants have included external coaches, district leadership teams, practitioners (teachers), classroom coaches, and behavior specialists.

As shown in Table 5 below, the MA ESE and its partners have trained approximately 326 educators in Pyramid strategies and related coaching and assessment methods (i.e., TPOT) since spring 2015, as supported directly through SSIP implementation. The participants represent both the district and school levels, including external coaches who support multiple districts and schools with Pyramid implementation; district leadership teams who guide implementation within their districts and schools; and school/classroom level personnel such as teachers, coaches, and behavioral specialists.

Table 5. SSIP Pyramid Model Training Participant Totals (May 2015 – January 2017)

TRAINING TYPE AUDIENCE NUMBER OF INDIVIDUALS TRAINED

Pyramid Model Practices Practitioners (teachers), Classroom Coaches, Behavior Specialists, and External Coaches Approximately 170

PTR-YC Behavior Specialists, Teachers, and External Coaches Approximately 58

TPOT Classroom Coaches and External Coaches Approximately 80

Practice-Based Coaching Classroom Coaches Approximately 25


External Coaches, District Based Personnel and Community Members Approximately 17

Approximate Total Trained326 Individuals

(Total excludes potential duplicate count of external coaches across trainings)

33 | P a g e

Progress Finding: Trainings in Pyramid Model strategies and supports have been well-received, and have largely been found by participants to be useful in developing their skills toward implementation. Over time, the effectiveness of the trainings has been improved, guided by participant feedback.

MA ESE has regularly collected evaluation feedback forms from participants at scheduled training events and meetings. The evaluation forms are designed to assess the quality and usefulness of the sessions, and to solicit suggestions for improvements going forward. A sample of findings across the total 444 evaluation forms entered and analyzed to date are shown below.

With respect to the overall quality of the SSIP trainings, respondents consistently reported that the necessary resources had been provided, and that the sessions were well-delivered in terms of time and pacing. A sample of training responses for these items is provided in Table 6 below. For each item, the average score across a four-point agreement scale is shown where 1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree. Quality ratings were similarly strong across all evaluation forms, including those for TPOT trainings in October 2015 (2016 data were unavailable)—these data are not shown as the survey items differed from those included in the table below.

Table 6. Quality of SSIP Training Events: Average Ratings on Four-Point Agreement Scale

SURVEY ITEMPTR-YC

LEADERSHIP TEAM

MEETING

LEADERSHIP TEAM

MEETING

COACHES DAY

PYRAMID MODEL

PRACTICES

PYRAMID MODEL

PRACTICESPTR-YC

4/2016n=21

6/2016n=37

9/2016n=35

9/2016n=26

11-12/2016n=16

11-12/2016n=6

1/2017n=26

Received necessary information prior to event 3.62 3.68 3.37 3.42 3.69 3.72 3.58

Schedule was well-paced 3.62 3.54 3.20 3.46 3.78 3.94 3.25

Facilities appropriate 3.57 3.49 3.20 3.30 3.49 3.89 3.36

Materials provided were useful and sufficient 3.71 3.64 3.43 3.42 3.70 3.94 3.46

Sufficient time for all activities 3.67 3.70 3.34 3.48 3.78 3.94 3.32

Participants also rated the usefulness of the training sessions. A sample of responses over time across two different types of training, Pyramid Model Practices and PTR-YC, are shown in Tables 7 and 8 on the following pages—these tables show a comparison of results across the first and second years of training. While usefulness ratings were consistently positive, growth over time is also evident, where the mean level of agreement increased from the first to second year training of each type. The averages shown in the tables are based on the same four-point agreement scale used above, where 1 = strongly disagree, 2 = disagree, 3 = agree, and 4 = strongly agree. Percentages in the second row of Table 6 refer to the percent responding “yes” to whether expectations had been met.

34 | P a g e

Table 7. Usefulness of Pyramid Model Practices Training: Comparison of Year 1 and Year 2 Feedback

SURVEY ITEM

PYRAMID MODEL

PRACTICES: PART 1 (2

DAYS)

PYRAMID MODEL

PRACTICES: PART 2 (2

DAYS)

PYRAMID MODEL

PRACTICES: 1ST REGIONAL

(3 DAYS)

PYRAMID MODEL

PRACTICES: 2ND REGIONAL

(3 DAYS)11/2015

n=781/2016n=60

11-12/2016n=16

11-12/2016n=6

The information will be useful for my work 2.94 3.40 3.68 3.94

The training met my expectations 56% 91% 97% 100%

The training helped me understand the foundation of the Pyramid Model

NA(question not asked) 3.76% 3.89

Table 8. Usefulness of PTR-YC Training: Comparison of Year 1 and Year 2 Feedback

SURVEY ITEMPTR-YC PTR-YC

4/2016n=21

1/2017n=26

Deepened my understanding of how to use Pyramid Strategies to support children 3.43 3.68

As a result of the training I have new strategies and skills to implement in my work 3.38 3.50

The training provided a framework that will support the implementation of the Pyramid Model in my

district/program3.52 3.57

Participant comments were also reviewed across the various sessions. A small sample of the common themes that emerged is presented below. As described earlier in this report, MA ESE has been using feedback in a timely way to improve the trainings for subsequent sessions. For example, the first round of Practices trainings was found to be somewhat repetitive, and four days was considered too long for the material presented. The next series of trainings was then streamlined and the duration was shortened to 3 days. In addition, suggestions about content changes, based on the demographics of the audience with hands-on make-it/take-it activities, were also made to subsequent trainings.

35 | P a g e

“What was the most important thing you learned?” Common Themes

Pyramid Model Practices Training

Year 1

Many participants indicated having learned how to structure and develop an individualized support plan. Some discussed the value of the support planning chart and the visual of the strategies. A few respondents specifically discussed the value of the break out activity which allowed them to see different points of view from different roles/perspectives (e.g., administrator vs. teacher, etc.)

Others found learning more about how to identify behaviors and change behaviors to be particularly valuable. Examples included how to breakdown behaviors, how to teach replacement skills, and prevention strategies.

Year 2

Almost a third of participants reported that the real-life suggestion and the sharing of practical ideas were the most important aspects of the three-day event. Some examples included viewing videos, viewing other educators’ lesson plans and learning how they can be applied in the classroom, and acquiring tools that can immediately be used in their professional practice.

Some participants mentioned goal setting strategies and setting out steps to reach those goals as valuable information. Others mentioned three areas of social skills modeling, including skills taught in the classroom, the role families play in creating a behavior plan, and strategies to address certain behaviors.

Prevent, Teach, Reinforce – Young Children (PTR-YC) Training

Year 1

Approximately one-third of the respondents indicated finding value in the resources they received, and believed they would be easy to use. Specifically, they appreciated the manual, the “forms”, and strategies on how to implement the interventions.

Almost as many specifically mentioned the Functional Behavioral Assessment (FBA), and found the steps easy to follow to support implementation in their schools.

Year 2

Like Year 1, many respondents indicated they most liked having received forms that would help facilitate implementation – examples included the PTR-YC goals sheet, fidelity checklist, and the assessment forms.

The Functional Behavioral Assessment was mentioned by a number of others as the most valuable aspect of the training.

36 | P a g e

Evaluation Question 1b: To what extent is the implementation of PBS through Pyramid strategies in MA integrated with other early childhood and/or MA ESE initiatives at the community/local and state levels?

Progress Finding: MA ESE has embedded the SSIP within its larger strategic planning efforts in Early Childhood (EC) and other MA ESE initiatives to improve special education outcomes for all pre-K children statewide.

MA ESE plays a key role on a leadership team of approximately seven individuals charged with carrying out the work of the SSIP within the larger context of ECSE in Massachusetts. The ECSE’s strategic plan for improving special education student outcomes – Building Blocks: Building Better Longitudinal Outcomes for Children in pre-K (please see Appendix) – has been created in partnership with MA DPH and MA EEC. This plan identifies three strategic areas of focus: 1) Improve systems to engage effectively with families, 2) Improve systems to assist transition from early intervention to pre-K and from pre-K to K, and 3) Improve instruction to increase educational outcomes (including social/emotional skills and social relationships). The SSIP is directly aligned with the third area of focus specifically.

A review of project documents suggests that the ECSE leadership team, including representatives from MA ESE, MA DPH, and MA EEC, meets regularly to engage in cross-agency collaboration to support the Strategic Plan. The team has a stated mission, vision, and target metrics for success, and maintains a detailed internal tracking system of progress toward its objectives across the three areas of focus in the plan.

Examples of the overarching statewide activities in which the team is engaged include:

an annual ECSE retreat in collaboration with MA DPH and MA EEC to review the strategic plan, related data, and current initiatives;

creation and maintenance of an ECSE website to share interagency resources and information about PD opportunities; and

implementation of a Comprehensive System of Professional Development (CSPD) model of statewide trainers for child level outcomes.

A sample of the current initiatives under the purview of the ECSE team includes the following:

Early Literacy Early Childhood IEP Family Engagement and PBS Indicator 7 – Preschool Outcomes Indicator 8 – Family Involvement Survey Indicator 12 – Timely transition to EI from Pre-K Promoting Inclusive Communities in Preschool - Preschool LRE / Indicator 6 Providing TA and PD to LEAs related to family engagement SSIP Phases I-III

37 | P a g e

The ECSE team’s commitment to success is further indicated by the completion of the Early Childhood Technical Assistance (ECTA)/DaSy Framework Self-Assessment to gauge the status of their state system, and to set priorities for improvement.

Progress Finding: MA ESE leadership and staff overseeing the SSIP are engaged in ongoing efforts to integrate the work of PBS through Pyramid strategies throughout multiple initiatives at the state and local levels.

In addition to the work of the ECSE leadership team, MA ESE staff are active members of the Pyramid Model State Leadership Team (SLT). A review of project documentation spanning May 2015 through February 2017 indicates that MA ESE personnel, via the Pyramid Model SLT, are actively engaged in a wide range of activities to support the SSIP among broader statewide initiatives and to continue to expand that statewide infrastructure to extend the implementation of Pyramid strategies.

Some of the SLT’s activities have included collaborating on the delivery of an annual statewide Summit to train and inform practitioners about Pyramid practices; forming workgroups to address focus areas such as communication and marketing, professional development, and data collection and use strategies around Pyramid; and designing a statewide website devoted to Pyramid Model information and resources. In addition, the group recently began work on an initiative to identify Model Schools implementing the Pyramid Model, and also continues to work on creating a crosswalk of PBS with PBIS to help facilitate the transition of practices that ensure consistency across these interventions for students moving from pre-K (PBS through Pyramid) to K-12 (PBIS) classrooms.

Further, MA ESE is engaged in promoting the integration of students with IEPs across MA ESE initiatives, including:

the development of curriculum modules and training videos designed to integrate literacy standards with social/emotional guidelines

the alignment of ECSE objectives with Early Literacy grants distributed through the Office of Literacy promoting the alignment of outcomes for students with disabilities within the state’s ESSA plan sponsoring the TA series “Building Inclusive Communities in Preschools” to promote inclusive

preschool options for children with IEPs

(b) Evidence that SSIP’s evidence-based practices are being carried out with fidelity and having the desired effects

Evaluation Question 2a: Is the state-level plan resulting in the number of schools and classrooms participating in PBS through Pyramid strategies sample growing over time?

Progress Finding: Among the 19 districts participating in the SSIP, 17 districts have schools and classrooms engaged in implementing the Pyramid Model. In all, 88 classrooms are adopting Pyramid.

As evidenced by External Coach Records, external coaches are engaging with SSIP districts and schools on a regular basis to support the training and implementation of the Pyramid Model, and to document their progress over time. Table 9 below indicates a recent assessment of the numbers of schools and classrooms implementing the model, as reported by external coaches. In subsequent years of SSIP reporting, the extent to which the number of schools and classrooms increases will be assessed.

38 | P a g e

As indicated earlier in this report, 19 districts are participating in the SSIP, having established leadership teams to help guide Pyramid adoption, and taken part in trainings and planning meetings. Across these 19 districts, 17 have begun to implement the model in at least one school in their district. The total number of schools reported to be implementing is 25; among these 25 schools, 88 total classrooms are involved. Among the 16 districts for which complete data were provided, 24% of their preschool classrooms are implementing.

Finally, as reported by ESE staff with input from District Leadership, the two districts of the 19 that have not begun to implement are currently managing several challenges regarding their readiness to do so, such as staff turnover and competing priorities within their districts. MA ESE intends to continue to support these districts in moving forward, and is optimistic the districts will begin implementation in the future.

Table 9. Progress toward Implementation of Pyramid Strategies among SSIP Districts and Schools

IMPLEMENTATION MEASUREPROGRESS TO DATE

(AS OF JANUARY 2017)

Number of districts participating(i.e., attendance at trainings, establishment of leadership team) 19

Number of districts implementing 17

Number of schools implementing 25

Number of classrooms implementing 88

Number of districts implementing in 100% of preschool classrooms (includes 8 districts with only 1 school) 10

Number of districts implementing in 50% of preschool classrooms 2

Overall percentage of classrooms implementing within districts (based on 16 districts for whom complete data were provided) 24%

Evaluation Question 2b: Are districts developing systems to support sustainable training and coaching practices at the local level?

Progress toward implementation of the Pyramid Model within SSIP districts and programs is primarily assessed using the BOQ tool, created by the Pyramid Model consortium. As described in Section C, district leadership teams completed the assessment at three points in time to gauge their district’s progress across nine critical elements of implementation. In all, 47 different benchmarks comprise the tool (please see the Appendix for the full BOQ with benchmarks).

Progress Finding: On average, district leadership teams reported substantial progress across most component areas of the BOQ over time. The greatest areas of improvement were staff buy-in, establishment of a functional leadership team, and establishing and communicating program-wide expectations.

Table 10 on the following page presents results for each critical element of the BOQ on the beginning and end of year assessments. The percentages represent ratings across each element on a scale of 0-2 where 0 = not in place, 1 = partially in place, and 2 = in place. The last column in the table provides the mean implementation level on the same scale. Finally, these results represent 18 districts that completed a beginning of year BOQ,

39 | P a g e

and 14 districts that completed an end of year BOQ. Therefore, these results may not represent the totality of the changes or lack thereof across the entire group of participating districts.

As shown, the aggregate results across all participating districts indicate that the greatest amount of growth from beginning to end of year BOQs were made in the following areas:

Staff Buy-in: 15% reported “in place” at beginning vs. 61% at end Establish Leadership Team: 32% (beginning) vs. 63% (end) Program-wide Expectations: 20% (beginning) vs. 60% (end) All Classrooms demonstrate adoption of “Teaching Pyramid: 13% (beginning) vs. 43% (end)

In addition, these are four of the five areas in which districts are having the most success toward implementation, meaning that they reported the highest percentages of benchmarks “in place” at the end of year BOQ, regardless of where they started. The fifth area in this category was Procedures for responding to challenging behavior, for which 58% of respondents reported “in place” on the end of year BOQ. The areas that reflected the least amount of growth from beginning to end of year were the following:

Family Involvement: 5% reported “in place” at beginning vs. 9% at end, and Strategies for teaching and acknowledging program-wide expectations: 20% (beginning) vs. 21% (end)

These two critical elements also represent the areas in which districts have the most work to do going forward, meaning that they reported the lowest percentages of benchmarks “in place” at the end of year BOQ. Two other elements in this category would include Monitoring implementation and outcomes (10% reporting “in place” at end of year) and Staff Support Plan (26% reporting “in place”).

Table 10. Benchmarks of Quality – Ratings of Critical Elements

40 | P a g e

Finally, MA ESE project staff have suggested that the BOQ completed at the Midyear Leadership Team Meeting (data are not shown in Table 7) likely reflected some self-correction by leadership teams as they became more familiar with the model. External coaches reported that leadership teams rated more items as “not in place” or “partially in place” after participating in additional Pyramid Model trainings and therefore the aggregate results on some of the critical elements reflects lower implementation than was indicated on the initial Benchmarks of Quality. For example, the “family involvement” benchmarks went from 5% of benchmarks in place at the beginning administration to 2% of benchmarks in place at the midyear administration. This process of re-calibration is not uncommon in a pre- post- assessment model; since the teams are now familiar with tool and training content/material, comparisons in subsequent years would be considered more reliable.

(c) Outcomes regarding progress toward short-term and long-term objectives that are necessary steps toward achieving the SIMR

Evaluation Question 3a: Are teachers implementing PBS through pyramid strategies in their schools?

As indicated above, 88 classrooms across 17 districts are currently implementing PBS through Pyramid strategies. With respect to progress toward short-term and long-term objectives that are designed to achieve the SIMR, data collected via the 2016 Mid-year Leadership Team Survey provides some indication of progress at the district and school, including successes and challenges.

Progress Finding: The majority of participating districts reported making progress toward implementation during the first year of participation. More than a third categorized their progress as “moderate” or better by March 2016.

The first Mid-year Leadership Team Survey was completed by 52 individuals across 18 districts in March 2016. Respondents included preschool classroom teachers, early childhood program directors and coordinators, school adjustment counselors, special education administrators and coordinators, and school administrators, among others. Figure 2 below shows the groups’ assessment of the progress their teams had made looking back over the first six or so months of implementation. Almost two-thirds (62%) indicated having made “a little progress”. Another third (32%) reported “moderate progress”, while 6% reported “significant” progress. It is

41 | P a g e

worth noting that these results represent feedback collected almost one year ago; the 2017 survey is currently being completed by district teams and will serve as a basis for comparison going forward.

Figure 2. 2016 Mid-year Leadership Team Survey:Ratings of District Team Progress toward Implementation “Over the Past 6 Months”

Leadership team members also indicated that their external coaches had contributed to some extent to their progress by that time; 58% categorized their coaches’ contribution as “a little”, while 39% indicated that their contribution had been “moderate” or “significant”. Survey ratings of coaches’ knowledge and skills were also strong, for example with respect to their knowledge of Pyramid content, understanding of how to coach adult learners, and familiarity with the public school context. Finally, open-ended survey comments suggested that during that early phase of the project (up through March 2016) district teams who had worked with their coaches were pleased with the support they had received; some indicated that they would like additional time with their coach. Others indicated that they had not yet had a chance to meet with their coach.

Progress Finding: During the first year of the SSIP, district teams expressed a need for additional facilitated planning meetings with their teams, as well as training for teachers in Pyramid Practices.

The Mid-Year Leadership Team Survey is designed to capture the ongoing challenges with implementation, and the specific needs for assistance from MA ESE and external coaches. As shown in Figure 3 below, nearly two-thirds (62%) of the 52 survey respondents indicated that additional facilitated time with their teams for planning would help them to expand implementation. In keeping with this finding, comments on evaluation forms collected at leadership trainings over the course of the project so far have also consistently pointed to teams’ desire for more time to work together on their implementation plans during meetings. Other needs identified by district team members included the availability of district-based Practices training for teachers (60%), as well as guidance about the importance of early childhood PBS (29%), and finally, how to blend funding sources to support the Pyramid Model initiative (both 29%).

42 | P a g e

Figure 3. 2016 Midyear Leadership Team Survey: Types of Assistance Needed to Expand Implementation

Survey comments from 37 of the respondents provided some additional details about the specific challenges districts were facing at that time:

24% (n=9) mentioned difficulty securing substitutes to enable educators to attend trainings. Just over 20% (n=8) mentioned educator buy-in as a challenge. One of the reasons for educator

concern was perceived to be the presence of multiple initiatives in their districts. 15% (n=6) discussed the need to continue to align the initiative with others in the district, especially

behavioral initiatives such as PBIS. Time required out of the building for trainings was mentioned by 15% (n=6) of respondents.

Extending beyond the timeframe captured by the Mid-Year Leadership Survey, External Coach Contact Records from 59 contacts made between June 2015 and January 2017 were also reviewed to learn about the types of services and supports that have been provided to districts to date. These records reflect work conducted by the eight external coaches with all 19 districts, where the number of contacts reported for each district ranged from 1 to 10; the average number of contacts with each district was three.

Progress Finding: External coaches most often provide in-person support to their districts, followed by phone and email contacts. Support is typically focused on support for leadership teams, but also extends to classroom coaches, data-driven decisions, and behavior systems, among other areas.

Records revealed that external coaches most often met with district and school staff in-person during site visits to programs or classrooms (56% of contacts). Other contacts were made via phone or email (33%), followed by training (19%). Figure 4 below shows the types of support typically provided by external coaches. As shown, 92% of contacts were focused around leadership team support; an additional 24% were related to classroom coach support.

43 | P a g e

Figure 4. The Most Frequent Types of Support Provided by External Coaches

During these contacts, external coaches provided support to leadership teams such as:

Assisting team leaders in building meeting agendas/processes (46%) Assisting with professional development and coaching (31%) Attending meetings (31%) Assisting with locating resources (17%)

Contacts with classroom coaches, while less frequent, included examples such as:

Debriefing with the classroom coach Support for using the TPOT Assisting with classroom coaches’ professional development events Assisting classroom coaches in explaining their role and providing an overview to teachers

Evaluation Question 3b: Does the fidelity of classroom implementation improve over time?

With respect to measures of fidelity of implementation in the classroom, the TPOT assessment is currently administered and reviewed at the local level for purposes of teacher reflection and improvement. As such, the SSIP includes trainings in TPOT administration for classroom coaches and external coaches. Therefore, the extent to which the TPOT training is preparing these individuals for using the TPOT effectively is an important consideration.

Data collected via the evaluation feedback forms from 32 participants who attended the 10/2015 TPOT training is shown in Table 11 below (data for 2016 were unavailable at the time of this report). As indicated, before the training, participants rated their knowledge of the TPOT as 2.4 (between limited and moderate), on a four-point scale where 1 = none, 2 = limited, 3 = moderate, and 4 = extensive. After the training, 91% of the group felt prepared to conduct a TPOT within a classroom.

44 | P a g e

Table 11. TPOT Training:Participants’ Knowledge of TPOT Before and After Training

SURVEY ITEM

TPOT

10/2015n=34

My knowledge of the TPOT before the training 2.4 (on four-point scale)

Following the training, I feel prepared to conduct a TPOT within a classroom

50% “agree”41% “strongly agree”

Going forward, the state will consider working with district teams to assess fidelity of implementation through local level reporting of the TPOT in summary or aggregate form to state leadership, with assurances of de-identified data that are not tied to individual teachers. The feasibility of this approach will be explored with the district leadership teams and in collaboration with the external evaluator.

(d) Measureable improvements in the SIMR in relation to targets

Evaluation Question 4a: Is the number of children in MA, aged 3-5, with disabilities, exiting from preschools with age-expected social emotional functioning increasing?

Progress Finding: The percent of preschool children functioning within age expectations in positive social-emotional skills has increased over each of the past two years.

The State Identified Measurable Result (SIMR) for the MA ESE SSIP is the improvement of social emotional outcomes for young children with disabilities, aged 3-5. The first way in which MA ESE is measuring achievement of the SIMR is by assessing over time the percent of preschool children functioning within age expectations in positive social-emotional skills by the time they turn 6 years of age or exit the program. These data are collected via the Child Outcomes Summary (COS) process designed by the Early Childhood Outcomes Center as a way for states to summarize data on children’s movement toward age expectations in specific outcome areas. As can be seen in Table 12, the state’s SIMR (data point 2) increased from 44.49% in FFY 2013 and 47.81% in FFY 2014 to 53.57% in FFY 2015, providing preliminary information regarding positive SSIP outcomes at a statewide level.

Table 12. SPP Indicator 17 DataSummary Statement 2

SIMR DATAFFY 2013 FFY 2014 FFY 2015

n=472 n=479 n=420

The percent of preschool children who were functioning within age expectations in Outcome A (positive social-emotional skills) by the time they turned 6 years of age or exited the program.

44.49% 47.81% 53.57%

Target 90% 90% 90%

Note: Summary Statement 2 is calculated by: # of preschool children reported in progress category (d) plus # of preschool children reported in progress category (e) divided by [the total # of preschool children reported in progress categories (a) + (b) + (c) + (d) + (e)] times 100.

45 | P a g e

MA ESE has set targets for Indicator 7 based on the state’s expectation that a majority of young children should be functioning within age expectations in each outcome by the time they exit the preschool program or turn six. While the FFY 2015 target was not reached for Summary Statement 2, MA ESE has decided to keep the FFY 2016-FFY 2018 targets set at 90%. The ambitious targets reflect MA ESE’s high expectations for all early childhood outcomes and MA ESE’s vision for the SSIP.

Evaluation Question 4b: Is the number of children in MA, aged 3-5, with disabilities, exiting from preschool with greater than expected growth in their social emotional functioning increasing?

Progress Finding: SIMR data have not yet evidenced improvements in the number of children exiting from preschool with greater than expected growth in their social emotional functioning.

The second way MA ESE is measuring achievement of the SIMR is by assessing over time the percent of preschool children who entered or exited the preschool program below age expectations in positive social emotional skills who substantially increased their rate of growth by the time they turned 6 years of age or exited the program, as measured by the Child Outcomes Summary. As can be seen in Table 13 below, the percentage declined approximately 8 percentage points from FFY 2014 to FFY 2015 (87.20% compared to 79.14%). It is not clear whether the slippage for this measure is attributable, in part, to the smaller data set (see Indicator 7 report of the FFY15 MA SPP/APR). Although the FFY 2015 target was not met, MA ESE kept the targets for FFY 2016-FFY 2018 at 100% to reflect MA ESE’s high expectations for all students, even the State’s youngest learners.

Table 13. SPP Indicator 17 DataSummary Statement 1

SIMR DATAFFY 2013 FFY 2014 FFY 2015

n=419 n=422 n=350

Of those preschool children who entered or exited the preschool program below age expectations in Outcome A (positive social emotional skills), the percent who substantially increased their rate of growth by the time they turned 6 years of age or exited the program

85.44% 87.20% 79.14%

Target 100% 100% 100%

Note: Summary Statement 1 is calculated by: # of preschool children reported in progress category (c) plus # of preschool children reported in category (d) divided by [# of preschool children reported in progress category (a) plus # of preschool children reported in progress category (b) plus # of preschool children reported in progress category (c) plus # of preschool children reported in progress category (d)] times 100.

46 | P a g e

F. Plans for Next Year

1. Additional Activities to be Implemented Next Year (with timeline)

After a thorough review of the current implementation data, MA ESE has identified a number of strengths and areas for improvement which will drive the activities for next year (see Table 14 for timelines).

Activities that will be continued without adjustment: Actively engage with state and district stakeholders Support state infrastructure, including intra- and inter-agency collaboration Sustain capacity of External Coaches Implement training and improving curriculum based on feedback Gather and monitor data Reinforce systems for data collection

Activities that address areas for continued growth: Promote additional family engagement and evaluation of Positive Solutions Strengthen professional development for classroom coaches Increase training for data-based decision making (state, district, classroom and child level) using

Indicator 7 data Bolster technical assistance for local level district data analysis Develop improved data collection methods for qualitative data by streamlining open-ended items on

all data collection tools to facilitate summary of common themes Develop statewide disaggregated data analysis for indicator 7 Improve methods for collecting qualitative data Build state infrastructure to support local level integration of early literacy initiatives

Table 14. SSIP Implementation: February 2017 – Spring 2018

Date Activities Audience

Ongoing and Continuing

Communication with Pyramid Model Consortium Staff

N/A


External Coach Contact Records completed as substantive contacts with districts occur

N/A


Site Visits by MA ESE Staff at participating districts, including intensive support for Indicator 7 data collection

District Leadership Team, External Coaches, Teachers


Monthly Professional Learning Community (PLC) meeting

External Coaches


State-level leadership team meetings MA ESE, MA EEC, and MA DPH staff

New activity Quarterly Coaches Convening, to include: in-depth data-based decision making

(using indicator 7) fidelity measures integration of literacy instruction support for family engagement

Classroom and External Coaches

2/9 & 2/10/17 or TPOT (Teacher Pyramid Observation Tool) Classroom Coaches and External

47 | P a g e

Date Activities Audience

3/9 & 3/10/17 Training Coaches

4/24/17 Prevent Teach, Reinforce – Young Children (PTR-YC)

Teachers, Classroom Coaches, Behavior Specialists, and External Coaches

3/17/17 and 5/17/17 Positive Solutions Train-the-trainer w/MA EEC Teachers, Classroom Coaches, Behavior Specialists, and External Coaches

5/25/17 End of Year Leadership Team Meeting District Leadership Teams and External Coaches

FY 18

Summer/Early FallNew Activity

New Staff “ramp up” District Leadership Teams, Classroom Coaches, and Teachers

Fall Beginning of the Year Leadership Meeting, to include: gathering baseline BoQ data implementation planning for Pyramid and

Positive Solutions implementation data-based decision making, district,

school and child level


Practices Training Teachers

Positive Solutions Train-the-trainer Teachers, Classroom Coaches, Behavior Specialists, and External Coaches

TPOT (Teacher Pyramid Observation Tool) Training


Prevent Teach, Reinforce – Young Children (PTR-YC)


Spring Practices Training Teachers

Positive Solutions Train-the-trainer Teachers, Classroom Coaches, Behavior Specialists, and External Coaches

TPOT (Teacher Pyramid Observation Tool) Training


Prevent Teach, Reinforce – Young Children (PTR-YC)


End of Year Leadership Team Meeting District Leadership Teams and External Coaches

2. Planned Evaluation Activities Including Data Collection, Measures, Expected Outcomes

The evaluation plan for the coming year will be consistent with the plan presented in Phase II, and discussed throughout this report. MA ESE will continue to collaborate with internal and external stakeholders to monitor and refine this plan including the key questions, intended outputs, and the process of data collection. In coordination with the external evaluator, MA ESE will consider fine-tuning some of the data collection tools and procedures to help ensure high response rates, and a continued focus on valid and reliable data across all measures. MA ESE will review all data as they become available to continue its cycle of inquiry for continuous

48 | P a g e

improvement, and data will also be provided to the external evaluator throughout the year for analysis and reporting.

Table 15 shows the evaluation plan for the current year including key questions and an abbreviated list of intended outcomes, as well as data sources and timelines, based on the previous evaluation pan with slight modifications.

Table 15. SSIP Evaluation Plan: February 2017 – January 2018

EVALUATION QUESTION INTENDED OUTCOMES DATA SOURCESANTICIPATED

TIMELINE

State Level InfrastructureEQ1aIn what ways is MA ESE using the SSIP, including statewide implementation of PBS through Pyramid strategies, to build state-level capacity to support improved social emotional outcomes for young children with disabilities?

Build state capacity, including: leverage cadre of PBS external

coaches collaborate with social service

agencies to providing training/support to families

collaborate in Part C and K-12 PBIS initiatives


Statewide demographic training data

Statewide Training Evaluation Forms

Ongoing

At each training

EQ1bTo what extent is implementation of PBS through Pyramid strategies in MA integrated with other early childhood and/or ESE initiatives at the community/local and state levels?

MA ESE will: Support implementation of newly

created PBS/PBIS crosswalk to promote state level collaboration, and

Engage in ongoing collaboration to continue to identify strategies to promote integration of PBS

Extant project document review

Ongoing

Program/District InfrastructureEQ2aIs the state-level plan resulting in the number of schools and classroom participating in PBS through Pyramid strategies sample growing over time?

MA ESE will provide adequate training in Pyramid Model strategies to promote district-levels trainings

District administrators and educators participate in statewide trainings

Families participate in Positive Solutions Training

External Coach Contact Records

Benchmarks of Quality (BOQ) Mid-year Leadership Team

Survey Statewide training data and

evaluation forms Pyramid Model Database

Ongoing Bi-annually Annually in

March At each

training Bi-annually

EQ2bAre districts developing systems to support sustainable training and coaching practices at the local level?

Participating district have built internal capacity to train additional teachers in Pyramid strategies



Survey

Ongoing Bi-annually Annually in

March

Classroom LevelEQ3aAre teachers implementing PBS through Pyramid strategies in their classrooms?

Participating districts have built internal capacity to training additional teachers in Pyramid strategies

Family engagement increased over time



Survey Pyramid Model Database

Ongoing Bi-annually Annually Bi-annually

49 | P a g e

EVALUATION QUESTION INTENDED OUTCOMES DATA SOURCESANTICIPATED

TIMELINE

EQ3bDoes the fidelity of classroom implementation improve over time?

Teachers will be able to implement PBS through Pyramid strategies with fidelity

Teaching Pyramid Observation Protocol (TPOT) (currently collected at local level; the feasibility of sharing aggregate data the state to be explored going forward)

Annually

Student LevelEQ4aIs the number of children in MA, aged 3-5, with disabilities, exiting from preschool with age-expected social emotional functioning increasing?

Children with disabilities, aged 3-5, will exit preschool with social/emotional competencies that will allow them to access and participate in the general curriculum and in all aspects of the school.

Indicator 7 data – Child Outcomes Summary (COS): Summary Statement 2

Annually in spring

EQ4bIs the number of children in MA, aged 3-5, with disabilities, exiting from preschool with greater than expected growth in their social emotional functioning increasing?

Indicator 7 data – Child Outcomes Summary (COS): Summary Statement 1

Annually in spring

3. Anticipated Barriers and Solutions

MA ESE has planned for the following contingencies in implementing the next steps of the MA SSIP.

Table 16. Anticipated Barriers and Solutions

Anticipated Barrier Solution

Attrition of External Coaches. State staff recruits early childhood experts throughout the region.

Attrition of trained local staff. Provide new staff “ramp up” training and support at the beginning of the school year.

State model does not adequately account for diverse local level needs.

Stakeholders and ESE staff will engage in a continuous cycle of data analysis, action planning, and evaluation to address diverse community needs and provide responsive, individualized supports for district staff and families.

4. State’s Need for Additional Support/TA

MA ESE continues to appreciate the technical assistance and support the agency has received to date on the development of the SSIP and reporting in Phases I – III. MA ESE has participated in a variety of individualized, targeted, and universal TA from OSEP directly and from OSEP-funded TA centers, including, the IDEA Data Center (IDC), the National Center for Systemic Improvement (NCSI), the Early Childhood Technical Assistance Center (ECTA), and the Center for IDEA Early Childhood Data Systems (DaSy). The expertise of these organizations, coupled with the dissemination of information via Grads 360 and OSEP TA calls, has proved invaluable for ongoing implementation and progress monitoring for Phase III. MA ESE plans to continue to

50 | P a g e

participate in cross state learning communities and request additional assistance with communication tools, such as info-graphics, as offered by IDC and NCSI.

APPENDIX

Beginning of the Year Leadership Team Meeting Presentation

MA ESE Department of Early Childhood Special Education (ECSE) Strategic Plan

Benchmarks of Quality (BOQ) Assessment

51 | P a g e

52 | P a g e

53 | P a g e

54 | P a g e

Early Childhood Program-Wide PBS Benchmarks of Quality

Program Name: Location: Date:

Team Members: *** Only one answer may be checked ***

Critical Elements Benchmarks of QualityCheck One

Not In Place

Partially in Place

In Place

Establish Leadership Team

1. Team has broad representation that includes at a minimum a teacher, administrator and a member with expertise in behavior support. Other team members might include parent, teaching assistant, related service specialists and other program personnel.

2. Team has administrative support. Administrator attends meetings and trainings, is active in problem-solving to ensure the success of the initiative, and is visibly supportive of the adoption of the model.

3. Team has regular meetings. Team meetings are scheduled at least 1x per month for a minimum of 1 hour. Team member attendance is consistent.

4. Team has established a clear mission/purpose. The team purpose or mission statement is written. Team members are able to clearly communicate the purpose of the leadership team.

5. Team develops an implementation plan that includes all critical elements. A written implementation plan guides the work of the team. The team reviews the plan and updates their progress at each meeting. Action steps are identified to ensure achievement of the goals.

6. Team reviews and revises the plan at least annually.

Totals (%): 0%

Staff Buy-In

7. Staff are aware of and supportive of the need for a program wide system for addressing children’s social emotional development and challenging behavior. A staff poll establishes buy-in before the initiative is launched.

8. Staff input and feedback is obtained throughout the process - coffee break with the director, focus group, suggestion box. Leadership team provides update on the process and data on the outcomes to program staff on a regular basis.

Totals (%): 0%

55 | P a g e


Not In Place

Partially in Place

In Place

Family Involvement

9. Family input is solicited as part of the planning process. Families are informed of the initiative and asked to provide feedback on program-wide adoption and mechanisms for promoting family involvement in the initiative.

10. There are multiple mechanisms for sharing the program wide plan with families including narrative documents, conferences, and parent meetings to ensure that all families are informed of the initiative.

11. Family involvement in the initiative is supported through a variety of mechanisms including home teaching suggestions, information on supporting social development, and the outcomes of the initiative. Information is shared through a variety of formats (e.g., meetings, home visit discussions, newsletters, open house, websites, family friendly handouts, workshops, rollout events).

12. Families are involved in planning for individual children in a meaningful and proactive way. Families are encouraged to team with program staff in the development of individualized plans of support for children including the development of strategies that may be used in the home and community.

Totals (%): 0%

Program-Wide Expectations

13. 2-5 positively stated program wide expectations are developed.

14. Expectations are written in a way that applies to both children and staff. When expectations are discussed, the application of expectations to program staff and children is acknowledged.

15. Expectations are developmentally appropriate and linked to concrete rules for behavior within activities and settings.

16. All program staff are involved in the development of the expectations.

17. Expectations are shared with families and staff assist families in the translation of the expectations to rules in the home.

18. Expectations are posted in classrooms and in common areas in ways that are meaningful to children, staff and families.

Totals (%): 0%

Strategies for Teaching and

Acknowledging the Program-

Wide Expectations

19. Instruction on expectations is embedded within large group activities, small group activities, and individual interactions with children.

20. A variety of teaching strategies are used: teaching the concept, talking about examples and non-examples, scaffolding children’s use of the expectations in the context of ongoing activities and routines. Instruction on expectations and rules occurs on a daily basis

21. Strategies for acknowledging children’s use of the expectations are developmentally appropriate and used by all program staff including administrative and support staff (e.g., clerical, bus drivers, kitchen staff).

Totals (%): 0%

56 | P a g e


Not In Place

Partially in Place

In Place

All Classrooms Demonstrate the Adoption of the

“Teaching Pyramid”

22. Teachers and program staff have strategies to promote positive relationships with children, each other, and families in place and use those strategies on a daily basis.

23. Teachers and program staff have arranged environments, materials, and curriculum in a manner that promotes social-emotional development and guides appropriate behavior.

24. Teachers and program staff are proficient at teaching social and emotional skills within daily activities in a manner that is meaningful to children and promotes skill acquisition.

25. Teachers and program staff respond to children’s problem behavior appropriately using evidence-based approaches that are positive and provide the child with guidance about the desired appropriate behavior.

26. Teachers and program staff provide targeted social emotional teaching to individual children or small groups of children who are at-risk for challenging behavior.

27. Teachers and program staff initiate the development of an individualized plan of behavior support for children with persistent challenging behavior.

Totals (%): 0%

Procedures for Responding to

Challenging Behavior

28. Strategies for responding to problem behavior in the classroom are developed. Teachers use evidence-based approaches to respond to problem behavior in a manner that is developmentally appropriate and teaches the child the expected behavior.

29. A process for responding to crisis situations related to problem behavior is developed. Teachers can identify how to request assistance when needed. A plan for addressing the child’s individual behavior support needs is initiated following requests for crisis assistance.

30. A process for problem solving with other teachers around problem behavior is developed. Teachers can identify a process that may be used to gain support in developing ideas for addressing problem behavior within the classroom (e.g., peer-support, classroom mentor meeting, brainstorming session).

31. A team-based process for addressing individual children with persistent challenging behavior is developed. Teachers can identify the steps for initiating the team-based process including fostering the participation of the family in the process.

32. An individual or individuals with behavioral expertise are identified for coaching staff and families throughout the process of developing individualized intensive interventions for children in need of behavior support plans.

33. Strategies for partnering with families when there are problem behavior concerns are identified. Teachers have strategies for initiating parent contact and partnering with the family to develop strategies to promote appropriate behavior.

Totals (%): 0%

57 | P a g e


Not In Place

Partially in Place

In Place

Staff Support Plan

34. A plan for providing ongoing support, training, and coaching in each classroom on the Pyramid Model practices is developed and implemented.35. A data-driven coaching model is used to assist classroom staff with implementing the Pyramid Model practices to fidelity.

36. Staff responsible for facilitating behavior support processes are identified and trained.

37. A needs assessment is conducted with staff to determine training needs on the adoption of the Pyramid Model.

38. Individualized professional development plans are developed with all staff.

39. Group and individualized training strategies are identified and implemented.

40. Plans for training new staff are identified and developed.

41. Incentives and strategies for acknowledging staff are identified.

Totals (%): 0%

Monitoring Implementation and Outcomes

42. Process for measuring implementation fidelity is developed.

43. Process for measuring outcomes is developed.

44. Data are collected and summarized.

45. Data are shared with program staff and families.46. Data are used for ongoing monitoring, problem solving, ensuring child response to intervention, and program improvement.47. Implementation Plan is updated/revised as needed based on the ongoing data.

Totals (%): 0% Final Total (%): 0%

Benchmark Averages for graphs Not In Place

Partially in Place

In Place

Establish Leadership Team 0.00 0 0 0Staff Buy-In 0.00 0% 0% 0%Family Involvement 0.00 Program-Wide Expectations 0.00 Strategies for Teaching and Acknowledging the Program-Wide Expectations 0.00All Classrooms Demonstrate the Adoption of the “Teaching Pyramid” 0.00Procedures for Responding to Challenging Behavior 0.00Staff Support Plan 0.00 Monitoring Implementation and Outcomes 0.00

58 | P a g e