tools for evaluating social validity in a rapidly ... · tools for evaluating social validity ......

Tools for Evaluating Social Validityin a Rapidly Expanding PBS

ProgramJulie Graden, Jill Norman, Molly Gadd,

Michelle Marchant, & K. Richard Young

Brigham Young University

Presentation Outline

• Overview of the Peaceable Schools Project• Obstacles to collecting Social Validity in a

growing program• Review of the outcomes of our Social Validity data• Implications for the future

Peaceable Schools Program:An Overview

Our Priority. . .

To create a positive school environmentthat nurtures the social, emotional and

academic well-being of all studentsthrough the use of Positive Behavior

Support (PBS) practices.

“If schools attend to the emotional andbehavioral needs of students on a broadscale, it is likely that they will create theconditions necessary for socialcompetence and academic success ofstudents.”

(Young, Marchant, & Wilder, 2003)

School-Wide Systems

Approach

Individual

Classroom Non-Classroom

Model

Why Measure Social Validity?“It is important to assess social acceptability ofintervention procedures…in order to ensure that allrelevant parties (e.g., teachers, parents, and otherinterventionists) agree that the procedures arereasonable for the classroom, home, or wherever theintervention procedures take place…Namely, if anintervention is viewed as socially acceptable there ishigher probability that it will be implemented withtreatment integrity than if the intervention procedureswere initially viewed to be unacceptable.”

- Lane & Beebe-Frankenberger, 2004

Obstacles to MeasuringSocial Validity

• Developing instruments of social validitymeasurement

• Collecting data

Developing Instruments ofSocial Validity Measurement

• ISQ (Indicators of School Quality)• SVQ (Social Validity Questionnaire)

ISQ: Indicators of School Quality

The ISQ is a comprehensive survey system forschool administrators to evaluate and monitorschool improvement efforts. It summarizes theperceptions of parents, teachers, students, andother school staff regarding more than 30 crucialcharacteristics of the school in 7 major areas.

Development

• Center for the School of the Future (CSF)Utah State University– http://www.schoolquality.org/– Original Instrument

• Peaceable Schools Program, BYU– Supplement

Instrument:Student

Survey ResultsWestside:2002-2003

School Climate Westside:2002-2003

Social Development Westside:2002-2003

Social Validity QuestionnaireSVQ

• Development– Questions addressing school-wide PBS and

specific interventions– Administration

Social Validity Analysis

• Four areas being measured: school-wide,classroom, non-classroom, individual

• Components being measured: procedures,outcomes

• Populations being surveyed: parents,students, teachers/staff

Research QuestionsWhat are• educators’ perceptions of procedures?• educators’ perceptions of outcomes?• parents’ perceptions of outcomes?• students’ perceptions of outcomes?

--School B

--School AIndividual

0.973.40School B

0.834.07School ANon-classroom

0.854.22School B

0.934.22School AClassroom

0.854.00School B

1.004.17School ASchoolwide

SDM

Educators’ Perceptions of Peaceable Schools Project’s Procedures

0.493.75 Teachers

1.532.64 Staff

0.953.19School B

0.864.41 Teachers

0.83.95 Staff

0.834.18School AIndividual

--Non-classroom

0.394.24 Teachers

0.654.27 Staff

0.474.25School B

0.854.53 Teachers

0.784.07 Staff

0.824.3School AClassroom

0.413.93 Teachers

1.373.01 Staff

0.743.48School B

0.774.37 Teachers

0.743.96 Staff

0.754.17School ASchoolwide

SDM

Educators’ Perceptions of Peaceable Schools Project’s Outcomes

0.833.91School B

0.784.23School AIndividual

--School B

--School ANon-classroom

0.794.08School B

0.674.37School AClassroom

0.734.08School B

0.694.28School ASchoolwide

SDM

Parents’ Perceptions of Peaceable SchoolsProject’s Outcomes

0.592.62School B

0.632.59School A

SDM

Students’ Perceptions of Peaceable Schools Project’s Outcomes

General Results

• For both schools the lowest rankingquestion was “The Peaceable Schoolsproject required too much teacher time toimplement.”

• The highest ranking questions were “Myadministrator supports and values the PSP”and “It is important to praise appropriatestudent behavior”.

Methods of Collecting Data

1.       Paper/pencil - first year School A: 55% School B: 87.5%

2.       Survey Monkey - second year School A: 76% School B: 57%

Benefits of Technology

Technology:• Is fast and easy• Produces a better return rate• Makes data storage less cumbersome• Allows for easy manipulation of data• Cuts costs• Eliminates errors of human calculation

Drawbacks of Technology

Technology:• Requires specific skills and equipment• Distances us from data• May distance us from the people

Creating Social Validity Surveys

• Four areas: schoolwide, classroom, non-classroom, individual

• Components: procedures, outcomes• Populations: parents, students,

teachers/staff• Benefits of technology

Future Research Questions

• How are social validity data influencingprogram change?

• How does technology impact return rate –among different groups?

• How does the social validity of electronicsurveys compare to paper pencil surveys?

• Does the PBS model become more sociallyvalid over time?