Harvesting Data for Problem Solving
REBECCA PIERMATTEIWayne HickmanChristina Jordan
PBIS Maryland Coaches MeetingApril 30, 2014 & May 5, 2014
WHAT CAN WE LEARN FROM DATA?
What we are doing well and what challenges we face
Which students, teachers, systems need more support
Whether our support is successful
In a NutshellKnow what data you need and how to access the
data.
Make sure your data is reliable
Review and share data on a regular basis.
Keep it simple and reasonable.
Use your data to guide your decision making.
Use your data to evaluate individual progress as well as program and school-wide success.
BEFORE YOU START…Resource Mapping
Resource Mapping-Inventory of Current Practices
• What are the practices in place at each tier of the triangle?
• Are they evidence-based practices?• How are you measuring effectiveness of
practices (data)? • Who are the service delivery
teams/personnel (e.g., graduation coach, PALS teacher, Math Coach)
Tier 3
Tier 2
Tier 1
Triangle Activity:
Applying the Three-Tiered
Logic to Your School
7
Practices, Initiatives, Programs for a FEW
Practices, Initiatives, Programs for SOME
Practices, Initiatives, Programs for ALL
BEFORE YOU START…Resource Mapping
Early Warning Indicators
Early Warning IndicatorsCourse
Performance inCore Subjects GPA Credits
FCAT/Concordance
ScoresAttendance
Office Discipline Referrals
AdditionalFactors
On-Track Indicators
On-Track
Meeting all graduation requirements Cs or better in all areas
2.5 or more Meeting credit graduation requirement for grad plan year
Level 3 or Above or concordant scores within the same school year
4% or less absences per quarter or semester
3 or less Level I and/or minor referrals
DisengagementNo extra curricular involvementSubstance AbuseHigh MobilityMental health issuesFree/Reduced lunchFoster/group homeTransient/HomelessParent unemployment Student employmentChanges in behavior/ appearance More recent traumatic eventMissed guidance appointmentsNo show for yearbook picture
At-Risk forOff Track
Lacking 1 graduation requirement
2.0 to 2.49 Behind 1 Credits
Level 2 on FCAT 5% or more absences per quarter or semester
4 or less Level I and/or minor referralsLevel II ODRs per semester
Off-Track
Lacking 2 graduation requirementsFailing 1-3 classes
Less than 2.0 Behind 3 credits Not passed both sections of 10th grade FCAT or retakesNo concordant scores
10% absences per quarter or semester
5 or more Level I and/or Level II ODRs per semester
Highly Off-Track
Lacking 2 or more graduation requirementsCurrently failing 3 or more classes
Less than or equal to 1.5
Behind 4 or more credits
Not passed 10th grade FCAT or retakesNo concordant scores
15% or more absences per quarter or semester
5 or more Level II ODRs for fighting/ profanity/ disruption per semester
ExtremelyOff-Track
Meeting no graduation requirements2-3 Years Behind
Less than or equal to 1.0
Not meeting cohort graduation plan
Not passed 10th grade FCAT or retakesNo concordant scores
20% or more absences per quarter or semester
Established pattern of severe behavior Level II & III ODRs
Part A: SCHOOL-WIDE DATAIdentify what academic and behavioral data you need
ACADEMIC: Homework completion, GPA, Credit Accrual, Benchmark Assessments
BEHAVIORAL: ODRs (Big 5), attendance, suspension/expulsion, minor incidents, nursing and counseling logs
Identify the sources of that data
Identify the person responsible for getting and presenting the data
Identify how often and it what manner it will be shared with the team, faculty, and administration
Part B: WHAT DO WE DO WITH THE DATA?
Celebrate success!
Identify problems with PRECISION using the 5 W’s:What, Where, When, Who, Why
Determine whether your data indicates a need for school-wide practice or small group/individual response?
Develop Solution Options
Create Problem Solving Action Plan
Evaluate Solution
HELPFUL TOOLSData Decision RulesProblem Solving Action Plan
IF: Focus On:
> 35% of students receive 1 or more referralsAverage referrals per student > 2.5
Schoolwide Systems
> 35% of referrals come from non-classroom settings> 15% of students who receive a referral are referred from non-classroom settings
Non-Classroom Systems
>50% of referrals are from classroom settings>40% of referrals come from less than 10% of the classrooms
Classroom Systems
At least 10-15 students have 5 or more referrals Targeted Group Interventions
<10 students with 10 or more office referrals<10 students continue w/referrals after targeted interventionSmall number of students destabilizing overall functioning
Individual Student Systems
EXAMPLEPRECISION PROBLEM STATEMENT:
Many students from all grade levels are engaging in disruption, inappropriate language, and harassment in the cafeteria and hallway during lunch, and the behavior is maintained by peer attention.
Identify 5 W’sIdentify which system you will target
TIER 2/3: Sorting Students into Interventions
ACTIVITY
BEFORE YOU START…Resource Mapping
Early Warning Indicators
Decision Rules for Access to Interventions
Decision Rules for Access to Advanced Tiers
Specific to each intervention
Identify objective variables/criteria as well as the minimum/maximum for eachEX: GPA between 1.0 and 1.5
You may need to scale up or down depending on your capacity
Part C: Sorting Students into Interventions
1. How will students be identified for this intervention? What are the data decision rules for access – criteria used, min and max for each variable?
2. What data do we need?
3. Where will we get the data?
4. Who will be responsible for collecting the data?
HELPFUL TOOLSEARLY WARNING INDICATORS
RESOURCE MAPPING
DECISION RULES FOR ACCESS TO ADVANCED TIERS
Part D: Progress Monitoring and Program Evaluation
1. How will student progress be measured/monitored? Who is responsible for measuring/monitoring progress?
2. What indicators/benchmarks will show that a student is responding to the intervention? Not responding?
3. What indicators/benchmarks will show that a student is ready to exit the program?
4. How will overall program success be measured/monitored?
HELPFUL TOOL
Intervention Tracking Tool
*Illinois PBIS Network
OH NO!What do you do if your data shows
your program is not working?
FIDELITY MEASURESDetermine whether you are
implementing with fidelity(Are you doing it the way you are supposed to be doing it?)
Consider BOQ, SET, PBIS-TIC
Consider intervention specific fidelity measures
IMPORTANT CONSIDERATION
DATA MUST BE RELIABLEEXAMPLE: Behavior Referrals
Define your behaviors
Agreement re: major/minor
Standardized procedures for gathering referrals
All staff trained in how to complete and submit referrals
In a NutshellKnow what data you need and how to access the
data.
Make sure the data is reliable
Review and share data on a regular basis.
Keep it simple and reasonable.
Use your data to guide your decision making.
Use your data to evaluate individual progress as well as program and school-wide success.
Acknowledgements MDS3 is funded by a grant from the USDOE.
Federal Grant CFDA# Q184Y100015
Sheppard Pratt Health System: Rebecca Piermattei, M.S. [email protected] Wayne Hickman, Ed.D. [email protected] Christina Jordan, M.Ed. [email protected]
Maryland State Department of Education
Johns Hopkins University