Implementing School-Wide Positive Behavior Supports with Fidelity: Exploring the

Evidence

Nicholas A. Gage, Ph.D.University of Florida

A little about me…

Goals• We will:

• Provide an overview of School-wide Positive Behavior Interventions and Supports

• Discuss the evidence-base supporting the implementation of SWPBIS• Describe a series of state-level quasi-experimental design studies evaluating

the effect of SWPBIS on disciplinary exclusions

• We will: • Be moderately entertaining• Not talk too fast• Not hop too much

What are School-Wide Positive Behavior Interventions and Supports?

Continuum of Multi-Tiered Systems of Support

Primary Prevention:School-/Classroom-Wide Systems for

All Students,Staff, & Settings

Secondary Prevention:Specialized Group

Systems for Students with At-Risk

Tertiary Prevention:Specialized

IndividualizedSystems for Students

with High-Risk

~80% of Students

~15%

~5%

IMPLEMENTATION W/ FIDELITY

CONTINUUM OF EVIDENCE-BASEDINTERVENTIONS

CONTENT EXPERTISE &

FLUENCY

TEAM-BASEDIMPLEMENTATION

CONTINUOUSPROGRESS

MONITORING

UNIVERSAL SCREENING

DATA-BASEDDECISION MAKING

& PROBLEM SOLVING

CORE FEATURESSWPBIS

Universal Prevention

1. Leadership team2. Behavior purpose statement3. Set of positive expectations & behaviors4. Procedures for teaching SW & classroom-wide

expected behavior5. Continuum of procedures for encouraging expected

behavior6. Continuum of procedures for discouraging rule

violations7. Procedures for on-going data-based monitoring &

evaluation

School-Wide

Implementation Fidelity!

Why is Fidelity of Implementation Important?

• Fidelity data is necessary to make valid conclusions about outcomes.• Implementation acts as a potential moderator of the relationship

between interventions [programs] and their intended outcomes• A practical example: an antibiotic

What Fidelity of Implementation Measures Are Available for SWPBS?

Tier 1 Fidelity Measures Multiple Tiers Fidelity Measures• School-wide Evaluation Tool

(SET)• Benchmarks of Quality (BoQ)• Team Implementation

Checklist (TIC)• PBIS Self-Assessment Survey

(SAS)

• Monitoring Advanced Tiers Tool• Benchmarks for Advanced Tiers

(BAT)• PBS Implementation Checklist for

Schools• Individual Student Systems

Evaluation Tool (ISSET)• SWPBS Tiered Fidelity Inventory

(TFI)

Evidence Supporting the Effectiveness of SWPBIS

• Horner, Sugai, & Anderson (2010)• Narrative review found positive effect on school organizational health,

student perceptions of school safety, and problem behavior • Problem: not systematic and unclear how large an effect

• Solomon et al., (2012)• Meta-analysis of 20 single-case design studies that found small effects on

ODR and problem behavior• Problem: combined school and student outcomes

Evidence Supporting SWPBS

• Gage, Whitford, & Katsiyannis (2018)• Conducted a What Works Clearinghouse Review of SWPBIS effects on

disciplinary exclusion

Correlation n k g LL UL QOverall 90 4 -0.61 -1.27 0.05 8.69*Disciplinary Exclusion Type

ODR 53 3 -0.62 -1.29 0.06 8.43*

Suspension 72 2 -0.86* -1.68 -0.44 0.34WWC Standards

Meets Without Reservations 43 2 1.49 -0.24 3.21 1.92

Does Not Meet Standards 47 2 -0.97 -1.68 -0.26 0.10

Table 2.Fixed-Effect Meta-Analysis Results for SWPBS Effect on Disciplinary Exclusion

Limitations of the Literature

• Only four experimental studies with a treatment group and control group conducted to date at the school-level

• Those studies have only include 90 schools • Limited data on outcomes directly related to problem behavior

Solution!State-level analyses comparing schools implementing SWPBIS (with

fidelity) to schools that have never received training

***

Study #1: Kentucky

• Research Question:• R1: Are there differences in the number of behavioral events, and in- and out-

of-school suspensions between schools that have received SWPBIS training (Trained) and matched comparison schools (ITT models)?

• R2: Are there differences in the number of behavioral events, and in- and out-of-school suspensions between schools that have received SWPBIS training and implemented the behavior support practices as prescribed (Fidelity Behavior) and matched comparison schools (TOT models)?

Sample and Measures• Data from all public schools in the state of Kentucky (n =

1,171 schools)• 29 received SWPBIS training between the fall of 2013 and

spring of 2016, including two high schools, five middle schools, and 22 elementary schools

• School demographics and disciplinary exclusions (i.e., behavioral incidents and suspensions)

Research Design• We took two approaches in the analysis:

• an intent-to-treat model treating all schools ever receiving SWPBIS training, and

• a treatment-on-the-treated focusing on fidelity of implementation of the SWPBIS model.

• Missing data: < 10% missingness, thus used multiple imputation• Propensity Score matched Treatment schools to Control schools• Treatment Effect: Poisson Regression• Effect Sizes: WWC procedures for dichotomous outcomes by

calculating the Cox index based on the log odds ratio

Results

• Equivalence• Treatment Effect• Effect Sizes: -0.41 for ABRI schools, - .60 ABRI schools implmeneting

with fidelity for in-school suspensions and -0.33 for out-of-school suspensions for schools implementing with fidelity.

All Comparison Schools

(n = 1,138)

PSM Comparison Schools(n = 29)

Trained(n = 29)

Fidelity(n = 13) Equivalence

Covariate M SD M SD M SD M SD g1 g2Enrollment 545.4 291.4 419.2 223.6 450.2 112.7 464.9 134.0 0.18 0.24% Male 51.4 51.7 52.0 51.8 0.01 0.00% Female 48.7 48.5 48.2 48.3 -0.01 0.00% White 80.8 75.4 71.0 74.7 -0.14 -0.11% Black 8.8 12.3 15.2 13.8 0.15 0.07% Hispanic 5.7 6.3 7.5 6.5 0.11 0.09% Asian 1.3 0.7 1.0 1.2 0.00 -0.13% Native American 0.1 0.1 0.1 0.1 0.00 0.13% Hawaiian 0.1 0.1 0.1 0.0 0.00 0.00% Multi-Racial 3.3 5.2 5.5 3.9 0.02 0.21% Free/Reduced Lunch 63.8 68.3 69.1 63.4 0.02 0.15

Attendance Rate 94.8 95.1 95.0 95.3 -0.01 -0.04# of Teachers 35.9 16.9 29.3 13.8 31.8 6.5 30.6 7.4 0.23 0.12% Male Teachers 0.22 0.19 0.17 0.17 -0.08 -0.11% Female Teachers 0.78 0.81 0.83 0.83 0.08 0.11% White Teachers 0.96 0.93 0.95 0.94 0.21 0.05% Black Teachers 0.03 0.06 0.04 0.05 -0.14 -0.12% Hispanic Teachers 0.01 0.00 0.00 0.01 0.15 0.02Average Years of Experience

11.9 2.6 10.8 2.7 10.5 2.6 10.7 2.8 -0.09 -0.02

% of Teachers with BA 23.7 25.4 23.9 20.9 -0.05 -0.15

% of Teachers with MA 46.8 47.0 51.1 53.8 0.10 0.16

% of Teachers Rank 1 28.6 26.1 24.0 24.2 -0.07 -0.06% of Specialists 0.7 0.7 0.6 0.8 -0.04 0.09% of Proficient or Above: Language Mechanics

50.1 36.8 39.9 46.1 0.08 0.23

% of Proficient or Above: Math

49.0 41.2 42.8 48.1 0.04 0.17

% of Proficient or Above: Reading

56.4 46.0 46.9 51.2 0.02 0.12

Trained Schools Fidelity Schoolsβ SE 1-(exp(β)) β SE 1-(exp(β))

Total Suspensions -0.37*** 0.03 31% -0.40*** 0.03 33%

In-School Suspensions -0.36*** 0.03 30% -0.13** 0.04 12%

Out-of-School Suspension

-0.37*** 0.04 31% -0.84*** 0.06 57%

Behavioral Events -0.05* 0.03 5% 0.02 0.03 <1%

Study #2: Georgia

• Research Question #1: Is there a statistically significant difference in the number of disciplinary incidents and in-school and out-of-school suspensions for schools implementing SWPBIS with fidelity compared to matched comparison schools?

• Research Question #2: Is there a statistically significant difference in the number of disciplinary incidents and in-school and out-of-school suspensions for schools implementing SWPBIS at different fidelity levels compared to matched control schools?

Sample and Measures

• Data from all public schools in the state of Georgia during the 2015-2016 school year were collected from the Georgia Department of Education (GaDOE) website (n = 1,755)

• A total 119 schools implemented SWPBIS with fidelity during the 2015-2016 school year.

• BoQ, demographics, behavioral incidents, school suspensions

Research Design• Three levels of fidelity of implementation: Installing, Emerging,

Operational• We focused exclusively on schools implementing SWPBIS with fidelity

(Emerging and Operational), and excluded schools (n =218) that were “Installing”.

• Used Propensity Score Matching• Conducted ToT models using Poisson Regression• Calculated effect sizes as described above

ResultsOut-of-School Suspension In-School Suspension Disciplinary Incidents

Parameter β s.e. β s.e. β s.e.Intercept 3.79*** 0.259 2.86*** 0.221 3.74*** 0.149

SWPBIS -0.80*** 0.020 -1.60*** 0.021 -0.89*** 0.011

% White 0.00 0.002 0.01*** 0.002 0.01*** 0.001

% Black 0.01*** 0.003 0.01*** 0.002 0.02*** 0.001

% Hispanic 0.00 0.003 0.01* 0.002 0.01*** 0.002

% SPED 0.05*** 0.003 0.09*** 0.002 0.05*** 0.002

% Proficient: Math -0.05*** 0.002 -0.12*** 0.002 -0.07*** 0.001

% Proficient: Language Arts

0.03*** 0.002 0.08*** 0.002 0.04*** 0.001

% Proficient: Science 0.01*** 0.002 0.04*** 0.001 0.03*** 0.000

Out-of-School Suspension In-School Suspension Disciplinary IncidentsParameter β s.e. β s.e. β s.e.

Intercept 3.51*** 0.263 2.87*** 0.222 3.53*** 0.150

SWPBIS: Emerging -0.63*** 0.028 -1.62*** 0.033 -0.75*** 0.016

SWPBIS: Operational

-0.92*** 0.025 -1.59*** 0.024 -0.98*** 0.014

% White 0.00 0.002 0.01*** 0.002 0.01*** 0.001

% Black 0.02*** 0.003 0.01*** 0.002 0.02*** 0.001

% Hispanic 0.00 0.003 0.01* 0.002 0.01*** 0.002

% SPED 0.05*** 0.003 0.09*** 0.002 0.05*** 0.002

% Proficient: Math -0.05*** 0.002 -0.12*** 0.002 -0.07*** 0.001

% Proficient: Language Arts

0.03*** 0.002 0.08*** 0.002 0.04*** 0.001

% Proficient: Science 0.01*** 0.002 0.04*** 0.001 0.03*** 0.001

SWPBIS Schools PSM Comparison Schools

Outcome M SD M SD gOSS 29.9 34.2 69.2 95.8 -0.54ISS 24.6 55.7 123.1 188.2 -0.71Disciplinary Incident

92.1 97.5 232.6 291.2 -0.64

Study #3: Florida• Primary research questions.• 1. Is there a statistically significant difference in the number of

disciplinary exclusions between schools implementing SWPBIS with fidelity and propensity score matched comparison schools?

• Exploratory research questions• 2. Are there differences in disciplinary exclusions for subgroups of

students in schools implementing SWPBIS with fidelity compared with propensity score matched comparison schools?

Sample and Outcomes

• Data from all Florida public schools for all available discipline outcomes for the 2013-2014 school year (n = 3,513)

• During that year, 1,129 regular schools were trained to implement SWPBIS. Of those, 593 implemented SWPBIS with fidelity and were retained, while the remaining 536 schools were removed from the dataset

Research Design• Variables: Demographics, and disciplinary actions: (a) corporal

punishment, (b) in-school suspension (ISS), (c) out-of-school suspension (OSS), (d) expelled, and (e) contact with law enforcement.

• Propensity score matching of schools implementing with fidelity• Zero-predicted Poisson Regression Models• Effect sizes as described above

ResultsISS OSS Corporal Punishment

Estimate OR se Estimate OR se Estimate OR seIntercept -0.85*** 0.113 0.27 0.291 2.11*** 0.330SWPBIS 0.06 1.06 0.147 -0.99*** 0.37 0.243 -0.11 0.90 0.283Middle School -0.77* 0.46 0.302 -0.30 0.74 0.610 -0.26 0.77 0.460High School -0.65 0.52 0.426 0.71 2.04 0.784 -0.95 0.39 0.712Other -14.02 0.00 566.001 1.71 5.53 1.392 -2.60*** 0.07 0.617Total # of Students 0.00* 1.00 0.000 0.00*** 1.00 0.000OSS 2011 0.00*** 1.00 0.001 -0.07*** 0.93 0.010 0.00 1.00 0.002

Referral to Law Enforcement School-Related Arrest ExpulsionEstimate OR se Estimate OR se Estimate OR se

Intercept 2.17*** 0.202 3.64*** 0.390 6.85*** 1.078SWPBIS -0.18 0.84 0.146 0.27 1.31 0.234 -0.16 0.86 0.418Middle School -1.62*** 0.20 0.238 -1.88*** 0.15 0.314 -2.97** 0.05 1.102High School -1.09* 0.34 0.434 -1.90*** 0.15 0.488 -3.49** 0.03 1.211Other -1.90** 0.15 0.687 -0.82 0.44 1.192 -3.97** 0.02 1.331Total # of Students 0.00*** 0.00 1.00 0.000 0.00 1.00 0.000 0.00 1.00OSS 2011 0.00*** 1.00 0.001 0.00 1.00 0.001 0.00* 1.00 0.001

ResultsSample Discipline Outcome OR dAll Students

OSS 0.37 -0.55***ISS 1.06 0.03Corporal Punishment 0.9 -0.06Referral to Law Enforcement 0.84 -0.10

School-Related Arrest 1.31 0.15Expulsion 0.86 -0.08

SWDOSS 0.36 -0.56***one OSS 0.43 -0.46***more than one OSS 0.40 -0.50***

BlackOSS 0.57 -0.31**one OSS 0.63 -0.26**more than one OSS 0.62 -0.27**

ResultsOSS Only One OSS More than One OSS

Estimate OR se Estimate OR se Estimate OR se

Intercept 0.32 0.29 2.05 0.28 0.37 0.29

1-2 Years -0.82 0.44 0.44 -0.75** 1.10 0.20 -0.63 0.53 0.39

3-5 Years -1.53*** 0.22 0.37 -1.00*** 1.21 0.25 -1.33*** 0.26 0.33

6-10 Years-0.36 0.69 0.35 -0.63* 1.22 0.28 -0.38 0.68 0.34

Middle School

-0.32 0.73 0.61 0.34 2.50 0.36 -0.44 0.64 0.60

High School 0.78 2.19 0.79 1.14 1.71 0.61 0.62 1.85 0.78

Other 1.93 6.90 1.34 0.80 1.77 0.92 1.62 5.04 1.32

Total # of Students

0.00* 1.00 0.00 0.00 1.00 0.00 0.00* 1.00 0.00

OSS 2011 -0.07*** 0.93 0.01 -0.04*** 1.00 0.00 -0.07*** 0.94 0.01

Academic Achievement and SWPBS in FL

• Gage, N. A., Leite, W. L., Childs, K., & Kincaid, D. (2018). Average treatment effect of school-wide positive behavior supports (SWPBIS) on school-level academic achievement in Florida. Journal of Positive Behavior Interventions.

FCAT Mathematics FCAT ReadingEmpty Model Full Model Empty Model Full Model

Estimate s.e. Estimate s.e. Estimate s.e. Estimate s.e.Fixed Effects:

Intercept 57.07*** 1.89 64.00*** 1.94 58.43*** 1.87 62.37*** 1.76Treatment 1.65*** 0.33 1.94*** 0.39Years SWPBIS -0.50*** 0.01 -0.59*** 0.03Typical School 33.30*** 0.79 30.83*** 0.67SPED School -7.15*** 1.13 -9.45*** 0.95Title I -3.32*** 1.71 -3.50*** 0.16City -0.66*** 0.19 0.01 0.00Suburban -0.06 0.17 0.00 0.01Town -0.31 0.29 -1.29*** 0.25Grade 4 -2.39*** 0.13 -0.04 0.11Grade 5 -11.4*** 0.13 -1.21*** 0.11Total # of Students 0.02*** 0.01 -3.50*** 0.15# Free/Reduced Lunch -0.39*** 0.00 -0.04*** 0.00# White Students 0.01*** 0.00 0.01*** 0.00# Black Students -0.01*** 0.00 -0.01*** 0.00#Hispanic Students 0.01*** 0.00 0.01*** 0.00

Teacher/Student Ratio -0.13*** 0.01 -0.08*** 0.00Random Effects:

Time 79.92 33.34 100.49 28.41Treatment 8.33 0.33 11.03 0.94District 83.53 28.33 92.10 14.63

Residual 290.69 153.76 241.28 113.95

Fit:ICC (Within) 0.64 0.71 0.56 0.73ICC (Time) 0.18 0.15 0.23 0.18ICC (District) 0.19 0.13 0.21 0.09AIC 498,815 461,490 488,171 444,143BIC 498,869 461,687 488,225 444,340Deviance 498,803 461,446 488,159 444,099

Table 3. Mixed-Effects Regression Model Results for the Effect of SWPBIS with Fidelity on FACT Mathematics and Reading Tests

Study 4: SWPBIS in California

Primary research questions.1. Is there a statistically significant difference in the number of disciplinary exclusions between schools implementing SWPBIS with fidelity and propensity score matched comparison schools? Exploratory research questions2. Are there differences in disciplinary exclusions for subgroups of students in schools implementing SWPBIS with fidelity compared with propensity score matched comparison schools?

Sample and Outcomes

• Data from all California public schools for all available discipline outcomes for the 2015-2016 school year (n = 7,775)

• During that year, 1,384 regular schools were trained to implement SWPBIS. Of those, 544 implemented tier 1 SWPBIS with fidelity based on the TFI and were retained.

• Overall, there were 396,194 students in SWPBIS schools implementing with fidelity.

Research Design• Variables: Demographics, and disciplinary actions: (a) corporal

punishment, (b) in-school suspension (ISS), (c) out-of-school suspension (OSS), (d) expelled, (e) contact with law enforcement and (f) removal to alternative school for behavior.

• Propensity score matching of schools implementing with fidelity• Zero-predicted Poisson Regression Models• Effect sizes as described above

Demographic Characteristics

Treatment Effect Estimation

• Zero-inflated Poisson (ZIP) regressionoAccounts for data scaled as frequency counts and excessive zeros in outcomes

• Odds ratioso Likelihood that an event (e.g., suspension) will occur

• Standardized mean differences (g)o Size of difference between treatment and control schools

Results

Sample Discipline Outcome OR gAll students

ISS 1.05 0.03One OSS 0.66 -0.23More than one OSS 0.78 -0.14All OSS 0.63 -0.25OSS Incidents 0.69 -0.21Days missed due to OSS 0.65 -0.24Expulsions 1.05 0.03Referrals to alternative schools for disciplinary reasons 1.12 0.06Referrals to law enforcement 0.94 -0.03School-related arrests 1.05 0.03

SWDReferrals to alternative schools for disciplinary reasons 0.31 -0.65

HispanicOne OSS 0.67 -0.22

BlackMore than one OSS 0.72 -0.18

ParameterOne OSS All OSS Days Missed due to OSS

Estimate OR SE Estimat

e OR SE Estimate OR SE

Intercept -2.00 1.08 -2.12 1.16 -2.32* 1.16SWPBIS -0.42* 0.66 0.20 -0.45* 0.63 0.22 -0.43* 0.65 0.22Suburb 0.26 1.29 0.21 0.36 1.43 0.23 0.33 1.39 0.23Town -1.52** 0.22 0.55 -1.36* 0.26 0.56 -1.34* 0.26 0.56Rural 0.15 1.16 0.45 0.05 1.05 0.49 0.07 1.07 0.49Title I ineligible

1.25*** 3.48 0.35 1.12** 3.07 0.38 1.23** 3.43 0.38Middle school -0.41 0.66 0.36 -0.28 0.76 0.38 -0.26* 0.77 0.38High school -0.24 0.79 0.68 -0.04 0.96 0.70 -0.02 0.98 0.70Other configuration 2.70* 14.93 1.07 2.88** 17.84 1.11 2.86 17.39 1.12Total enrollment 0.00 1.00 0.00 0.00 1.00 0.00 0.00 1.00 0.00FTE Teachers -0.01 0.99 0.04 -0.01 0.99 0.04 -0.02 0.98 0.04Meet or exceed ELA standards in 2015-16 0.03*** 1.03 0.01 0.03** 1.03 0.01 0.03* 1.03 0.01FRL 1.39 4.00 0.80 1.61 4.98 0.85 1.81 6.12 0.86SWD -2.14 0.12 2.23 -1.85 0.16 2.36 -1.58 0.21 2.37LEP 0.46 1.59 0.77 -0.02 0.98 0.83 -0.01 0.99 0.83Black students -2.67* 0.07 1.30 -3.20* 0.04 1.47 -3.21 0.04 1.47White students -0.10 0.90 0.74 -0.11 0.90 0.80 -0.08 0.92 0.80

Does it really matter though?

What does it all mean

• SWPBIS is an effective approach for addressing disciplinary exclusions• Next Steps: MORE STUDIES

Thank You!