Critical Incidents in the Implementation

of a Multi-Tiered System of Supports

Christian Sabey & Cade Charlton

Brigham Young University

Melanie Dawson

Utah State University

Dan Pyle

Weber State University

Scott Ross

Colorado Department of Education

Session Objectives

2

• Describe current status of MTSS • Discuss the various definitions of MTSS • Describe our CIT study, and its implications for

scaling up MTSS across the country

Acknowledgements• Special thanks to the generous support and

investment of time from the project directors interviewed for this study. Also, special thanks to the co-authors and collaborators on this study

• Melanie Rees Dawson, PhD - Utah State University

• Dan Pyle, PhD - Weber State University• Emily Lund, PhD - Boston College • Sara Moulton, PhD - Brigham Young University • Kent McIntosh, PhD - University of Oregon• Don Kincaid, PhD - University of South Florida

The old model of support

Universal Interventions:School-/Classroom-Wide Systems for

All Students,Staff, & Settings

Targeted Interventions:Specialized Group

Systems for Students with At-Risk Behavior

Intensive Individual Interventions:Specialized

IndividualizedSystems for Students

with High-Risk Behavior

LAYERED CONTINUUM OFINSTRUCTIONAL &

BEHAVIORALSUPPORTS

Layered Continuum of Supports

Adapted from Practitioner Voices

PreventionContinuum–LikeLayersofaSieve

GreenZone(Level0/1)istransformedintoasolidsystem

YellowZoneInterventions(Level2,3and4)arecreatedandtransformedintosustainablesystems

RedZonesareestablishedandindividualized

Level 0/1

Level 2

Level 3

Level 4

Level 5

Level 6

Labelsupports…notstudents

MTSSIntegrated Continuum

Academic Continuum

Behavior Continuum

Adapted from the OSEP TA Center for PBIS

McIntosh & Goodman (2016) p. 7* Modifications

● Specific academic assessment and interventions

● Use of published curricula selected by school or district

● Direct assessment of skills

● Periodic assessment (benchmarking)

● Focus on grade-level teaming

● Described in IDEA as special education eligibility determination approach

● Specific social behavior assessments and interventions

● Use of free materials that are adapted to fit the school’s context

● Use of indirect assessment of behavior

● Continuous assessment of social behavior

● Focus on schoolwide teaming

● IDEA -- schoolwide prevention and individual intervention approach

Academic RTI PBIS

MTSS Elevator Pitch

•Share your MTSS Elevator Pitch with your neighbor:

• Who is it for?• Why is it important?

• Remember: The typical elevator ride in NYC lasts less than 118* seconds!

•Compare your definition with you neighbor’s

Image credit: Brian Honigmanhttp://www.brianhonigman.com/competitive-b2b-content/*Hayzlett, J. (January 2012). Why you need a better elevator pitch. Harvard Business Review. Boston, MA.

Defining MTSS• UT MTSS

• a framework for integrating assessment and intervention to maximize student achievement, reduce behavior problems, and increase long-term success

• FL MTSS• the systematic use of multi-source assessment data to most

efficiently allocate resources in order to improve learning for all students, through integrated academic and behavioral supports

• CO MTSS• a prevention-based framework of team-driven data-based

problem solving for improving the outcomes of every student through family, school, and community partnering and a layered continuum of evidence-based practices applied at the classroom, school, district, region, and state level.

Research Questions

1. What specific events helped facilitate the statewide implementation of MTSS in the participating states?

2. What specific events hindered the statewide implementation of MTSS in the participating states?

3. What do project directors or staff wish had been in place to support their efforts?

Method

•Critical Incident Technique (CIT)• “A set of procedures for collecting direct

observations of human behavior in such a way as to facilitate their potential usefulness in solving practical problems and developing broad psychological principles” (Flanagan, 1954).

• “The most quantitative of the qualitative approaches” Kent McIntosh (Personal Communication, 2016)

CIT

•Structured Interview• All interviews were

recorded & transcribed• What events helped your

implementation of MTSS?• What events hindered you

implementation of MTSS?• What do you wish would have or

could happen to enhance your improvement efforts?

UTAH STATE UNIVERSITY MTSS Critical Incidents Study

1

First Interview Script

(estimated duration = 30-45 minutes)

Date Interviewer

State “Thank you for taking the time to participate in this study. Our purpose in conducting this study is to identify the critical incidents associated with the statewide implementation of a tiered system of supports for academic and social behavior. To begin, I’d like to ask you a few questions about you and your project.”

1) “What is your educational and professional background?” 2) “What is your current role in the [insert project name]?” 3) How long have you been involved with the project?”

- If less than 1 year skip questions 8, 9, and 10 - Ask “Is there anyone else we should talk to about the

project in your state?”

4) “How long has the project been underway?” - If less than 1 year skip questions 8, 9, and 10

5) “Approximately, how many districts and/or schools are actively participating with your project?”

6) “Could you briefly describe the top three goals for your project and the strategies

you are using to accomplish these goals” # Goal Strategies Measures (if applicable) 1 2 3

------------------------------- Continue to the next page -----------------------------

CIT

•Credibility Checks• Critical Incident (CI) Extraction

• Percent agreement between two independent coders extracting Cis from the interviews

• IOA = 98%• Categorization

• Percent agreement between two independent coders categorizing Cis using operational definitions

• IOA = 86%• Participant cross-check• Expert review

Overview of Participating State Projects

Cross-Disciplinary Leadership, 25

Access to Professional

Development, 23

Consistent Language and Practices, 17

Consultation with External Partners, 14

Access to Funding , 12

Supportive Policies and Projects, 12

Student Outcomes Drive Planning and

Evaluation , 9

Efficient Data Systems, 9

COUNT OF HELPING CRITICAL INCIDENTS

Competing Priorities, Philosophies, or

Practices, 23

Ineffective Professional

Development Models, 15

Hiring, Retention, and Turnover, 13

Varying Levels of Readiness, 10

Limited and/or Restricted Funding, 9

Inadequate Data Systems , 7

Inadequate Support from State Leaders, 7

COUNT OF HINDERING CRITICAL INCIDENTS

Consistent Language and Practices, 16

More & Better Trained Personnel, 10

Access to Funding , 9

Effective Practices, 9

Access to PD, 7

Effective Data Systems, 7

Cross-Disciplinary Leadership, 6

Consultation with External Partners, 5

Supportive Policies and Projects, 2

COUNT OF WISHLIST ITEMS IN HELPING & UNIQUE CATEGORIES

Implications

•State MTSS projects would benefit from…• a concerted effort to build strong supportive

coalitions across traditional barriers (e.g., GenEd, SpEd, Admin),

• consensus on specific core components of MTSS (e.g., problem solving, tiered interventions, etc.),

• efficient strategies for evaluating implementation and impact,

• readily accessible professional development, and

• better guidance on MTSS in non-ES settings (e.g., secondary, talent dev., etc.)

Implications cont.

•Supporting agencies have a critical role including…

• preparing future teachers to engage in systems of supports for all students (IHE),

• engaging in mutually beneficial and sustainable partnerships with researchers,

• requiring the use of implementation science to inform high quality PD, and

• accepting responsibility for aligning initiatives when supporting local LEAs.

Limitations

•Not all MTSS projects participated• Terminology

• “Active participation of LEAs”• “… helped your implementation of MTSS”

• Idiosyncrasies of the states• Politics• Demographics• Policies

Future Research

• Identify strategies to enable or enhance helping factors

• Identify strategies to escape or avoid hindering factors

•Compare and contrast state helping and hindering incidents with district experiences

•Explore the resources or events that help administrators integrate academic and behavioral supports