brian r. flay, d.phil. department of public health oregon state university

Presented at Society for Research on Educational Effectiveness, Lansdowne VA, December 11 2006 OSU

School-Based Randomized Trials of Social and Character

Development ProgramsWith preliminary results of the

Positive Action program

Brian R. Flay, D.Phil.

Department of Public Health

Oregon State University


Why Social and Character Development

• Increasing behavioral problems in schools– Including violence, substance use, unsafe sex,

and other disciplinary referrals

• Seems to have worsened with increased attention to instruction for NCLB

• Link of behavior management to teacher time on task (teaching)

• Link of student behavior to student time on task (learning)


Limitations of Other Programs• Domain Specific

– Usually only one behavior or one skill

• Start Too Late – Upper elementary or middle school

• Limited Intensity and Dose– Often only once a week for 10-20 sessions

• Ecologically Limited– Usually only classroom– Also need school-wide, parent, community

• Limited Effect Sizes– Average effect sizes in the 0.2 to 0.4 range

• Effects not Sustained– Few effects beyond one year, let alone H. S.


IES Program of Research on Social and Character Development (SACD)

• To conduct rigorous research on comprehensive social and character development (SACD) programs

• To compare 7 comprehensive SACD programs in a national cooperative agreement

• UIC/OSU to conduct a clustered (school-based) randomized controlled trial of the effectiveness of the Positive Action program on:– Parent involvement with schools– Teacher classroom management– Student character and behavior– Student school attendance and achievement test scores


Funded Projects• New York University:

– Reading, Writing, Respect and Resolution (4 R’s)

• Vanderbilt University: – Love in a Big World (LBW) Program

• U of North Carolina, Chapel Hill: – The Competence Support Program (CSP)

• U of Illinois at Chicago with Oregon State University: – The Positive Action (PA) Program– Brian Flay, David DuBois and Carol Allred

• U of Maryland: – Second Step

• Children’s Institute: – Promoting Alternative Thinking Strategies (PATHE) Curriculum

• State U of NY Buffalo: – Academic and Behavioral Competencies (ABC) Program


The Positive Action Program Targets Multiple Behaviors

• By teaching that doing positive actions helps:– individuals develop a positive self identity.

– families develop a positive family identity.

– schools develop a positive school identity.

– communities develop a positive community identity.

• By teaching that:– When you do good, you feel good

– And there’s always a positive way of doing things


The Positive Action content is taught school-wide through six units:

• Unit 1. Self-Concept: What It Is, How It’s Formed, and Why It’s Important (the Thoughts-Actions-Feelings Circle).

• Unit 2. Physical and Intellectual Positive Actions for Body and Mind (includes motivation to learn)

• Unit 3. Social/Emotional Positive Actions for Managing Yourself Responsibly

• Unit 4. Social/Emotional Positive Actions for Getting Along with Others by Treating Them the Way You Like to be Treated (Character Education and Social-Emotional Learning)

• Unit 5. Social/Emotional Positive Actions for Being Honest with Yourself and Others (Mental Health)

• Unit 6. Social/Emotional Positive Actions for Improving Yourself Continually


The POSITIVE ACTION Program Components

K–12 classroom curriculumover 1,200 lessons - using Teacher’s Kits (manuals and materials for each grade), classroom teachers present 15–20-minute lessons

Principal’s Kits (Elementary and Secondary)a school-climate program to promote the practice and reinforcement of positive actions in the whole school population (students and staff)

Counselor’s Kitused with selected individual students, small groups and families

Family Kit contains prepared weekly home lessons paralleling the school program along with school parent-involvement activities

Community Kitmanuals and materials that align and encourage collaboration of all the environments (schools, families and community) involved in the program


Grade 3 Instructor’s Kit


Elementary Climate Kit• Principal’s manual• Support Staff manual• Parents’ manual• Assemblies

– Certificates of Achievement– Positive Action Balloons

• Positive Action Calendar• Words-of-the-Week Cards• Reinforcement Stickers • ICU Boxes and Notes• Positive Action Tokens

• Positive Notes • Notepads• Positive Action News • Music – 27 songs

– CDs– Song books

• Posters• Video

– Overview

• Implementation Plan• Scope and Sequence

Booklet


In a prior quasi-experimental study (Flay, Allred & Ordway, 2001), relative to elementary schools

without PA, matched schools with PA reported:• Major reductions in problem behaviors

• Up to 85% reductions in violence

• Up to 71% reductions in substance use

• Up to 90% fewer general disciplinary actions (with effects sometimes being larger in schools with higher levels of student poverty)

• Up to 80% fewer suspensions

• Up to 94% reductions in criminal bookings

• Major improvements in school performance• Up to 60% reductions of absenteeism,

• Up to 13% lower rates of chronic absenteeism.

• Up to 100% improvements on standardized achievement scores,

• Many schools report moving from one of the lowest scoring to one of the highest scoring in their district or state,

• The matched-control study was replicated by Flay & Allred (2003) with the addition of pretest matching data.


Flay & Allred (2003) also found long-term effects in middle high schools

• Relative to middle/high schools with a low proportion of PA graduates, middle/high schools with a high proportion of PA graduates reported:– Major reductions in problem behaviors– Major improvements in school attendance and retention– Major improvements in academic achievement

• Including proportions graduating from high school• And higher proportions going on to higher education

– With some of these effects being larger in high-mobility schools

– Strong dose-response relationship for all outcomes, with stronger effects occurring in middle/high schools with greater numbers of PA graduates


Chicago Trial of Positive Action• UIC/OSU school-based randomized trial• Schools randomly assigned from 7 matched pairs

– Matched on school-level demographic variables, achievement and disciplinary referrals

• Data collected from students and their teachers and parents, and school leadership (principal and PA Coordinator)– Beginning and end of grades 3 & 4 and end of grade 5

• Program Implementation and Outcomes– Teacher amount and integrity of program delivery– Parent involvement with program and schools– Teacher classroom management– Student character and behavior– Student school attendance and achievement test scores– Also school records data on achievement, attendance and

disciplinary referrals• Last wave of data collection Spring 2007


Research Design• 7 matched pairs of schools randomly assigned to 2

conditions:– Early Starters: Start the program in the 2004-05 school year OR– Late Starters: Start the program 3 years later (2007-08 school year)

• Schools eligible for inclusion were:– Community-based (No Magnet or Charter schools), Have not

already used Positive Action, Not already participating in related projects (Project Northland, All Stars),

– Enrollment > 50 and < 140 students per grade– Mobility rate no greater than 40%– More than 50% of students receive free or reduced price lunch– Less than 50% of students met achievement criteria on the ISAT

• Schools matched into pairs before randomization:– Achievement scores, School size – enrollment, Ethnic Distribution,

% Mobility, % Free/Reduced Lunch, Attendance and Truancy Rates, % Parent Involvement, % Teachers not meeting minimal requirements, CPS Region, Community crime statistics


Comparability of Matched Sets of SchoolsChicago Study (No significant differences)

0

10

20

30

40

50

60

70

80

90

100

Pair

Readin

ess

% W

hite

% B

lack

% H

isp

% A

sian

Achiev

e

% A

ttten

d

% T

ruan

cy

% P

over

ty

% M

obilit

yEnr

ol

% P

aree

nt P

atici

p

Quality

Tea

cher

s

Crimes

Pe

rce

nt

ES (Program)

LS (Control)


Research Design 2• Follow one cohort of students from the beginning of grade

3 to the end of grade 5• Signed parental consent – 98.3% returned, 79.7% Yes• Surveys of students, teachers and parents at beginning

and end of grades 3 and 4, and end of grade 5.• Teachers complete behavior rating scales on students in

their class at beginning and end of grades 3 and 4, and end of grade 5.

• Some of the measures are multi-site – collected by a national contractor (Mathematica Policy Research – MPR) at each of the 7 sites – and not available to PIs until about a year later

• Some of the measures are site-specific – specific to our evaluation of the Positive Action program – collected by UIC research staff


Measures and Analysis Plans• Cross-site data collection includes measures of

– social competence, – character, – behavior and – achievement.

• Site-specific data includes extensive process measures of – fidelity of implementation and – dosage of exposure for all components of the PA program

• As well as additional measures of hypothesized mediators of the intervention’s effects.

• We will use hierarchical statistical models (random regression models and GEE) that can accommodate longitudinal data with nested observations and missing observations.


Hawai`i Trial of Positive Action• Funded by NIH-NIDA• Schools randomly assigned from 10 matched pairs

– Matched on school-level demographic variables, achievement and disciplinary referrals

• Data collected from students and their teachers and parents, and school leadership (principal and PA Coordinator)– Beginning and end of grades 1/2 and end of grades 2/3, 3/4, 4/5

and 5/6• Program implementation and outcomes

– Teacher amount and integrity of program delivery– Parent involvement with program and schools– Teacher classroom management– Student character and behavior– Student school attendance and achievement test scores– Also school records data on achievement, attendance and

disciplinary referrals• Last wave of data collection Spring 2006


Baseline Equivalence: 2000-01

No differences are close to being statistically significant except % free/reduced lunch p = .099

0

10

20

30

40

50

60

70

80

90

100E

nrol

/10

Sta

bilit

y %

Lunc

h %

Bud

get

/day

Per

Cap

/100

0

Sp

Ed

%

LEP

%

Bel

owA

vera

ge

Aab

ove

Ave

rage

Bel

owA

vera

ge

Abo

veA

vera

ge

Abs

ente

es%

Dis

cplu

nary

Ref

eral

s

Dis

p. R

's p

er10

stu

dent

s

SAT Read: % SAT Math: %

School Demographics Achievement Behavior

Program Schools

Control Schools


Some Preliminary Results from the Hawai`i Trial

• Final wave of data not yet completely available or merged/cleaned

• Preliminary results from the end of the 3rd year of the intervention

• Simple graphs shown of examples of significant effects

• Some variability in dose, so graphs show effects by level of implementation.


Unit 4 Get Along With Others

2.35

2.40

2.45

2.50

Hi Med Low Control

Teacher rating of child behavior scale


Student Self-Report of Behavior Scale by Level of Implementation and Control

Overall Behavior (18 DO Items) by Classroom-level of Implementation and Controls

3.15

3.20

3.25

3.30

3.35

Hi Med Low Control


Student Rating of Feeling About Self When They Do Each Behavior by Level of Implementation and Control

Overall Feelings about Self when Do the 18 Behaviors by Classroom-level of Implementation

3.45

3.50

3.55

3.60

3.65

Hi Med Low Control


Fig. 13: % of 5th graders reporting substance use by condition

-2.00

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00

16.00

Ever triedtobacco

Ever triedalcohol

Ever beendrunk

Ever triedillegal drugs

C P A


Fig. 15: % of 5th graders reporting ever having had voluntary sex

-1.00

0.00

1.00

2.00

3.00

4.00

5.00

Girls Boys

C PA


Figure 17: Teacher ratings of student disruptive behaviors at Wave 4 by condition

1.1

1.15

1.2

1.25

1.3

1.35

1.4

1.45

1.5

Destroysothers'

property

Gets into lotsof f ights

Bullies otherkids

Disruptive inclass

C PA


Parents Listen/Attend/Help School

Parents Negative Attention To Child

Parents Talk with Child

PA Helps Their Child on Unit 2 - Intellectual

Parents Help Their Child on Unit 6 – Self-Development


Figure 8: Average Daily Absences by Condition

8

8.5

9

9.5

10

10.5

11

11.5

2001-02 2002-03 2003-04 2005-05

Sch

oo

l Yea

r

CONTROL PA STATE Standard


Suspensions per 100 students by year and condition

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

susp99 susp00 susp01 susp02 susp03 susp04 susp05

CONTROL Means

PA Means


SAT Reading Scores by Year and Condition

70

71

72

73

74

75

76

77

78

79

2002 2003 2004 2005

CONTROL Means

PA Means

STATE


SAT Math Scores by Year and Condition

74

75

76

77

78

79

80

81

82

83

84

2002 2003 2004 2005

CONTROL Means

PA Means

STATE


Relationships between teacher perceptions of school climate, beliefs about SACD, attitudes

toward PA and three elements of program implementation


Student responsiveness partially mediates program outcomes

The first (estimates outside the parentheses) is the model run with the clustering option specified using Wave 4 classrooms. The second (estimates inside parentheses) does not specify the clustering option.

This is the reason for the varying Ns between the two model fit indices.


Summary of Findings• Baseline equivalence was established

– Matching and random assignment was successful

• We observed some variation between schools in level of PA Implementation– Evidenced in teacher process data– Also reflected in student behavior and achievement

• Important positive program effects are emerging (and should be larger at later waves):– Teacher ratings, Student ratings, School Records

• Clear differential effects by level of implementation– Reflected in both teacher and student data– The more PA is implemented, the larger the effects for both

teachers and students, on both behavior and achievement

• Some important causal models of effects of school leadership and student responsiveness


How Does Positive Action Work?

Students:ImprovedBehavior

Students:Improved Feelings

About Self

Students:IncreasedMotivationTo Learn

Students:ImprovedAcademic

Scores

Teachers andStudents: Increased

Time On Task

Teachers: Improved

ClassroomManagement

Students:Increased

Attendance

Parents: Increased Involvement in School

ThePositiveAction

Program


Future Work/Needs• Larger scale trials

– ICCs for attitudes (.03-.1) and behavior (.01-.05) are generally smaller than for achievement (.15-.2)

– Still need Ns of 10-20 per condition rather than 5-7

• Improved measures of integrity and dosage delivered and received– Teacher, student and observer reports– Contractual reporting systems?

• Longer term follow-ups– Effects take several years to even start emerging– Prior work suggests important long-term effects are possible

• Methods of analysis to accommodate differential implementation– Propensity scoring, CACE, instrumental variable

brian r. flay, d.phil. department of public health oregon state university

Documents