School-Wide Positive Behavioral Support

School-wide Evaluation Tool

January 2007

School-wide Evaluation Tool (SET) Developed by

•George Sugai

•Terri Lewis-Palmer

•Anne Todd

•Rob Horner University of Oregon (2001,

revised 2005)

SET Purposes Research validated instrument

• Results are primarily used for research and reporting as a way to validate SWPBS practices

Designed to determine• The extent to which schools are using SW-PBS• If training and technical assistance efforts

result in change when using SW-PBS• If use of SW-PBS procedures is related to

valued change in the safety, social culture, and violent behaviors in schools

SET can be used to Assess features of SWPBS that are

in place Evaluate on-going efforts Design and evaluate procedures Determine annual goals and build

action plans Compare year to year efforts in

SWPBS

Do Schools Already Use School-Wide PBS?

SET Scores Oregon and Hawaii

0

20

40

60

80

100

a b c d e f g h I j k l m n o p q r

Schools

SE

T T

ota

l S

core

Pre

Can Schools Adopt School-Wide PBS Systems?

SET Scores Oregon and Hawaii

0

20

40

60

80

100

a b c d e f g h I j k l m n o p q r

Schools

SE

T T

ota

l S

core

Pre

Post 1

Post 2

LHS SET Scores

25%

33%

16%

62.50%

37.50%

55%

67%

100% 100%

83%

67%63%

100%

83%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Defined Taught Rew ards Violations Evaluation Leadership District

LHS 1/01

LHS 12/02

SETElementary School K pre/post

features

Expec

t. de

fined

Expec

t tau

ght

Ackno

wledgm

ent

Corre

ction

s

Evalua

tion

Lead

ersh

ip

Distric

t Sup

port

mea

n0

20

40

60

80

100

% o

f fe

atur

es im

plem

ente

d

fall 98

fall 99

Linwood Middle SchoolCaddo Parish

Linwood Middle SET Scores 4/05

100

60

6763

100 100

50

77

0

10

20

30

40

50

60

70

80

90

100

(A)Expectations

Defined

(B)Expectations

Taught

(C) RewardSystem

(D) ViolationsSystem

(E) Monitoring (F)Management

(G) DistrictSupport

Mean Score

% o

f fe

atu

res

imp

lem

ente

d

Henry V. Adams ElementaryMorehouse Parish

Henry V. Adams Elementary School SET Scores 5/05

84%

50%

100%100%

88%

83%

70%

100%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

(A)Expectations

Defined

(B)Expectations

Taught

(C) RewardSystem

(D) ViolationsSystem

(E) Monitoring (F)Management

(G) DistrictSupport

Mean Score

% o

f fe

atu

res

imp

lem

ente

d

SWPBS Evaluation SET is only one means of evaluating a

school’s level of implementation on SWPBS

It should be used in conjunction with• Office discipline referral patterns

• Staff survey results

• Safety surveys

• Team Checklist information

• Etc.

Louisiana Plans for SET To identify schools that have

exemplary SWPBS practices in place •Validation of self-report surveys for

schools reporting 80% or higher on the Self-Assessment Survey

Conducted by personnel who have met reliability criteria

It is not a test or a way to pass judgment on schools

Features of SETThe SET has 28 items focused on 7

areas:A. Expectations definedB. Behavioral expectations taughtC. Acknowledgement proceduresD. Correction proceduresE. Monitoring and evaluationF. ManagementG. District level support

Data Sources for the SET Interviews Observations Product Reviews

Conducting the SET (1 of 3)

Interviews (avoid any disruption to instruction)

•Administrator - 30 minutes

•Staff – at least 10 randomly selected

•Students - at least 15 randomly selected

Observation - tour school building•School rules posted in 10 specified

locations

Conducting the SET (2 of 3)

Product Review •Discipline Handbook•Office discipline referral forms, records

and summary reports•Instructional materials for teaching and

correcting behavioral expectations•School Improvement Plan•Annual action plan for SWPBS activities•Other related written products

Conducting the SET (3 of 3)

Approximately 2 hours to complete•30 minute interview with administrator

•Interviewing staff and students who are not actively engaged in instruction

•Tour of the school building

•Review school-wide discipline records, related materials, plans, etc.

Preparing School for SET Send confirmation letter to principal

one week before scheduled meeting date•Include the SET Overview

•Include a list of products to review Begin every SET with the

Administrator interview•Requires 30 minutes of uninterrupted

time!

SET Administrator Interview Items

Areas Items Discipline System 1-5 School Expectations 6-11 Leadership Team 12-21

The responses provided to these questions will be the basis for the rest of the interviews

Attempt to begin interview 45 minutes before lunch/recess

Student Interviews Interview 15 random students

•During administrator interview find out the lunch and recess times

Avoid interrupting instruction time

•On the scoring sheet, note the student grade level and gender

Get a representative sample of the student population (different grade levels, genders, ethnicity, etc.)

Remember small, medium, large

Student Interview Questions What are the school expectations?

•Count how many the students can name that match administrator’s responses

(i.e., 5/5, 2/3,…)

Have you received a reinforcer in the last two months?•Try to equate to a holiday or special event

(e.g., since Mardi Gras, since Christmas, etc.)

Staff Interview Procedures Request the administrator

introduce you to the SWPBS team members

Interview staff members individually•Avoid group responses

Staff Interview Questions (1

of 3)

What are the school expectations? • Count how many the students can name

that match administrator’s responses (i.e., 5/5, 2/3,…)

Have you taught the school expectations/rules to students this year? yes/no

Have you given out any reinforcers in the last 2 months? yes/no

Staff Interview Questions (2

of 3)

What type of student behaviors do/would you refer to the office?• Write exactly what behaviors they name

Yes (or +) if response matches administrator’s No (or –) if response does not match admin.

What is the procedure for dealing with a stranger?• Write exactly what they describe

Yes (or +) if response matches administrator’s No (or –) if response does not match admin.

Staff Interview Questions (3

of 3)

Is there a team in your school to address school-wide behavior support systems?•Yes or no

Are you on that team?•Yes or no

If no, then they are done If yes, move to 3 team member questions

Team Member Interview Questions Does your team use discipline data to

make decisions? Has your team taught the School-wide

program to staff this year? Who is your team leader?

• Name the person

Scoring items• Yes (or +) if response matches admin• No (or -) if response does not match admin

Product Reviews It helps if the administrator stays

around in case you have questions on the products. Often the notebooks lack organization

Ensure you have all the products you need before the administrator leaves the room

Observation During the interviews, tour the

school and observe school rules and crisis plan posted in 10 locations

Do not enter classrooms while teaching is in progress

If empty rooms are locked, request the administrator to allow you to quickly observe

Scoring the SET Write administrator answers directly

on the Interview Sheet Match administrator responses with

staff and student responses Staff and student responses and

observations are written directly on the Interview and Observation Form

Product Review is scored directly on the SET Scoring Guide

What does the SET score mean? Overall SET score

•General index of SWPBS implementation

Criteria for SWPBS implementation•80% total score AND

•80% “Expectations Taught”

Exercises SET Matching Exercise

SCORING THE SET TAB PAGES 14-15

Interview and Observation Score Calculation

Example #1 Scoring the Set Tab (pg. 19 A-D)

•Calculate score of interview and observation items

•Omit items requiring a product (P) Example #2 Scoring the Set Tab (pg. 21 A-D) Example # 3 Scoring the Set Tab

(pg. 23 A-D)

Product Score Evaluation (p.27) Teaching Behavioral Expectations

• Examples 1-3 Rewarding Student Behavior

• Examples 4-5 Dealing with and reporting behavioral

violations• Examples 6-8

Current Action plan with specific goals• Examples 9-12

Case Studies Score the 28 SET evaluation questions for CASE STUDY #1

Scoring the Set Tab (pg. 44 A-D and CS #1 pgs.1-15)

Score the 28 SET evaluation questions for CASE STUDY #2

(pg. 46 A-D and CS #2 pgs. 1-17)