Slide 1

Measuring Performance within School Climate Transformation Grants Rob Horner OSEP TA-Center on PBIS www.pbis.org Slide 2 Goal Define resources for measuring performance within SCTG grants that may improve efficiency, consistency and effectiveness of evaluation efforts. Provide a brief summary of schools using PBIS in the U.S. Slide 3 Why a Framework for Multi-tiered Behavior Support? The fundamental purpose of a Multi-tiered Behavior Framework is to make schools more effective and equitable learning environments. Predictable Consistent Positive Safe Slide 4 All the parts Logic Model Intervention Strategies/Programs Implementation Process Fidelity of Implementation Student/ Family Outcomes Multi-tiered Why the strategies work Build Capacity to Scale and Sustain Are we doing the strategy/program Valued outcomes/ Academic, Behavior, Social Emotional Slide 5 Four ways the OSEP TA-Center may be helpful Coordination and Communication o Webinars and information dissemination Intervention Content o Tier I o Tier II o Tier III Measuring Fidelity of Multi-tiered Behavior Support Measuring Student Outcomes Aligning multiple initiatives begins by defining common fidelity and outcome measures Slide 6 Resources Grantees are held accountable for what they submitted. But some consistency across projects may help all evaluation efforts Resources that may be helpful: o Evaluation Blueprint (www.pbis.org)www.pbis.org o Evaluation Annual Reports (www.pbis.org)www.pbis.org o State and District Capacity Assessments www.sisep.org o Multi-tiered Fidelity Assessment (Tier I, Tier II, Tier III) www.pbisapps.org o Multi-tiered student discipline data systems SWIS, CICO-SWIS, ISIS-SWIS www.swis.org, www.pbisapps.org www.swis.orgwww.pbisapps.org Slide 7 Slide 8 Performance Indicators: SEA The number of training and technical assistance events provided by the SEA School Climate Transformation Grant Program to assist LEAs in implementing a multi-tiered behavioral framework. The number and percentage of LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that report an improvement in knowledge and understanding of the implementation of a multi-tiered behavioral framework. The number and percentage of schools in LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that implement a multi-tiered behavioral framework. Slide 9 Performance Indicators: LEA Number and percentage of schools that report an annual decrease in office disciplinary referrals. Number and percentage of schools that report an annual improvement in the attendance rate. Number and percentage of schools that report an annual decrease in suspensions and expulsions, including those related to possession or use of drugs or alcohol. Number and percentage of schools annually that are implementing the multi-tiered behavioral framework with fidelity. Slide 10 Performance Measure Considerations Use a Formal Evaluation Model Measure Fidelity at All Three Tiers Consider Measuring Improved State/ District Capacity Measure student outcomes within a formal decision system Slide 11 Multi-Tiered Behavior Support Evaluation Model: Evaluation Blueprint/ www.pbis.org Slide 12 Context Who is participating How selected Initial Assessment Process for support plan development Slide 13 Input Type and amount of technical assistance Training Coaching Systems for Performance Feedback Development of evaluation capacity Development of Leadership Team Capacity Slide 14 Fidelity Are schools using Tier I practices and systems of behavior support? Are schools using Tier II practices and systems of behavior support? Are schools using Tier III practices and systems of behavior support? Slide 15 Impact State: Capacity of State o Has the state established the capacity to (a) improve district capacity, (b) implement evidence- based, multi-tiered behavior support in schools, (c) monitor the fidelity of school-level implementation and (d) benefits for children and families? District/Region: Capacity of LEA or Region o Has the district/region established the capacity to (a) implement evidence-based, multi-tiered behavior support in schools, and (b) monitor the fidelity of implementation in schools and (c) benefits for children and families? Schools: Fidelity of Imp and Impact on Students o Is multi-tiered behavior support being implemented with fidelity in target schools? o Is there an effect of multi-tiered behavior support on student behavior as measured by office discipline referral (ODR) patterns? o Is there an effect of multi-tiered behavior support on student attendance? o Is there an effect of multi-tiered behavior support on student suspension and expulsion? o Is there an effect of multi-tiered behavior support on student bullying and violence? o Is there an effect of multi-tiered behavior support on juvenile arrests? o Is there an effect of multi-tiered behavior support on student academic performance? Slide 16 Schools Are Schools Implementing Multi-tiered Systems of Behavior Support? Slide 17 Possible Measures of Multi-Tiered Behavior Support Fidelity Measures (www.pbis.org) Tiered Fidelity Inventory (TFI) Tier III Individual Student System Evaluation Tool (ISSET) Benchmark of Advanced Tiers (BAT) Measure of Advanced Tiers Tool ( MATT) Tiered Fidelity Inventory (TFI) Tier II Tiered Fidelity Inventory (TFI) Tier I School-wide Evaluation Tool (SET) Team Implementation Checklist (TIC) Benchmarks of Quality (BoQ) Regardless of the Intervention approach used, TA-Center Fidelity measures are available and appropriate Slide 18 Available from OSEP TA-Center www.pbis.org www.pbis.org No Cost Assessors Training PowerPoint available Assessors Training Video in development TFI- Technical Adequacy (McIntosh, Massar et al. 2014) Content Validity (.95;.93;.91) Usability (12 of 14 > 80%) (15 min per Tier) Inter-rater Agreement (.95;.96;.89) Test-retest reliability (.98;.99;.99) Factor Analysis (in process) Slide 19 Slide 20 TFI Data Tier I: 2014 Slide 21 Tier II Tier III Slide 22 Slide 23 Action Planning Slide 24 State/Region Capacity Does a state have the capacity to implement multi-tiered systems of behavior support? Is Technical Assistance improving the capacity of a state/region to implement multi-tiered systems of behavior support? Slide 25 Possible Measures of SEA/LEA Capacity www.sisep.org State Capacity Assessment (SCA) o Assessment of extent to which state department has capacity to implement evidence-based practices at scales of social significance? o No Cost, from OSEPs SISEP TA-Center www.sisep.org; www.scalingup.orgwww.sisep.org www.scalingup.org o Scores: Total, Sub-scale, Item o Used for initial assessment, action planning, progress monitoring District Capacity Assessment (DCA) o Assessment of extent to which a school district (or region) has capacity to implement evidence-based practices as scales of social significance? o No Cost, from OSEPs SISEP TA-Center o Scores: Total, Sub-scale, Item o Used for initial assessment, action planning, progress monitoring Implementers Training Video Slide 26 SCA/ DCA Format (Team with Coach/ Assessor) Item Data Sources Scoring Criteria Score 0 1 2 Slide 27 Minnesota State Slide 28 Minnesota Slide 29 Minnesota Slide 30 State SCA Item Analysis: Action Planning Slide 31 Slide 32 District Capacity Is Technical Assistance improving the capacity of a district to implement multi-tiered systems of support? Slide 33 District Capacity Slide 34 District Capacity Assessment Slide 35 District Capacity Assessment: Duda, et al. 2012 Slide 36 District Capacity Assessment District Capacity Assessment: Duda, et al. 2012 Slide 37 District Capacity Assessment Slide 38 Slide 39 Measures of Student Benefit Office Discipline Referrals o Tier I= SWIS o Tier II= CICO-SWIS o Tier III= ISIS-SWIS School Climate o Under development School Safety Survey o Risk Factor/ Protective Factor Slide 40 Performance Monitoring Summary Consider including measures of MTBSF fidelity at all three tiers. o Many Tier I options o Several Tier II options o One new measure that does all three (Tiered Fidelity Inventory (TFI) Consider assessing SEA/ LEA Capacity to implement evidence- based interventions o State Capacity Assessment o District Capacity Assessment Outcome measures o Office discipline referrals (SWIS, CICO-SWIS, ISIS-SWIS) later o School Climate o School Safety Survey Excellent Examples of State and District Reports at www.pbis.org