mark w. fraser, pi, school of social work, unc-chapel hill steven h. day, school of social work

Social and Character Development in Elementary School:

The Effectiveness of the Making Choices Program

Mark W. Fraser, PI, School of Social Work, UNC-Chapel Hill Steven H. Day, School of Social Work Shenyang Guo, School of Social Work Alan Ellis, Sheps Center and School of Social Work Roderick A. Rose, School of Social Work Maeda J. Galinsky, School of Social WorkKim Dadisman, Co-PI, Center for Developmental Science, UNC-CH Dylan Robertson, Center for Developmental Science Tom Farmer, School of Education, Pennsylvania State University

This presentation was given at the School of Social Work, University of Maryland, Baltimore, MD, on April 9, 2009. Portions of this report were presented at the annual meeting of SACD Project grantees on June 13, 2008 in Washington, DC

Preliminary Findings

Agenda• Theoretical Bases and programs

• Design and challenges

• Analytic strategies

• Analytic methods (skim – see slides)

• Findings

Acknowledgments

• This project was support by a cooperative agreement (R305L030162) with the Institute of Education Sciences at the U.S. Department of Education (US DOE). Funding for the project was appropriated by the US DOE and the Centers for Disease Control and Prevention.

• We thank Paul Rosenbaum (U Penn), Ben Hansen (U of Michigan), and Matthias Schonlau (Rand Corp) for their consultation on methodological issues related to this presentation.

Teachers Talk about Making Choices

• Changes in Classroom Atmosphere

• Observable Differences in Student Behaviors

• Measurable Academic Achievement

Classroom Atmosphere

“I noticed that the classroom

started working more as one big group instead of

individuals.”

Gr.5 Sandy Grove Elementary,Hoke County

Observable Behaviors

“The students tend to be less critical of each other and

more understanding of

each other’s differences.”

Gr. 5 Sandy Grove Elementary,Hoke County

Academic Achievement

“ The program uses excellent books to support the goals of being a good friend and not hurting others.… I use them during Language Arts time.” Gr. 4 Tommy’s Road Elementary, Wayne County

“It provided a way for students to put their feelings into words.”

Gr. 2, Bunn Elementary, Franklin County

Observable Behaviors …

I am feeling really mad!

Academic Achievement

“My students spend more time on task. They seem less distracted by annoying behavior.”

Gr. 5 Scurlock Elementary, Hoke County

Children are actually stopping and thinking about making the right choices, and I have heard a lot of children say to themselves,

“Make the right choice.” It is great to hear.

Kdg. Bunn Elementary, Franklin County

Make the right choice!

“It made a difference with teaching children how to deal with their feelings using better methods rather

than having tantrums or hitting.”Kdg. Bunn Elementary, Franklin County

Oh, boy! I need that Making Choices program.

Classroom Atmosphere

“This program provided a

foundation on which we

could build a classroom

community.” Gr. 1 North Drive,

Wayne County

…a program designed to reduce disruptive behavior and promote academic achievement.

…lessons that teach children respect toward others and responsibility for their own actions.

…social skills to make friends and deal with interpersonal problems.

Children in my school need…

Making Choices

Does social and character education work?

Research Question:

(i.e., is Making Choices effective?)

Intervention Research Perspective:The Design and Development Approach

1. Specify the problem and develop a program theory

2. Create and revise program materials

3. Refine and confirm program components (sequential experimentation perspective)

4. Assess effectiveness in a variety of practice circumstances and settings

5. Disseminate findings and program materials

Source: Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social programs. New York, NY: Oxford University Press.

PROBLEM THEORY:Perspectives on Conduct Problems and Academic

Achievement in Elementary School

•Developmental risk perspective

•Ecological theory

•Social information processing theory

Social and Character Development in Childhood:A Risk and Resilience Orientation

Biological RisksParentingFamily-School Pre-School ClimateNeighborhood

School ReadinessProcessing SkillsParentingFamily-SchoolSchool ClimateNeighborhood

Peer RejectionAcademic FailureParentingFamily-SchoolSchool ClimateNeighborhood

Increasingly Broad Repertoire of Potentially Damaging and

Aggressive Behaviors

Eco-Developmental Risk CascadePOINT: Risk factors for poor developmental outcomes vary over time. Lacking effective

intervention, the potential for poor outcomes increases – and cascades – as function of complex bio-social processes. To promote positive outcomes, we must disrupt malleable risk mechanisms.

Cognitive Mediation Model(in Developmental Sciences)

Biological Biological PredispositionPredisposition




Sociocultural Sociocultural ContextContext

•Stress/povertyStress/poverty•RacismRacism•Street codesStreet codes•Acute/chronic stressAcute/chronic stress



Adapted from: Dodge, K. A., & Pettit, G. S. (2003, p. 351). A biopsychosocial model of the development of chronic conduct problems in adolescence. Developmental Psychology, 39(2), 349-371.

PeersPeers•Deviancy trainingDeviancy training•Contagion effectContagion effect•False consensusFalse consensus effecteffect

PeersPeers•Deviancy trainingDeviancy training•Contagion effectContagion effect•False consensusFalse consensus effecteffect

ParentingParenting•MonitoringMonitoring•BondingBonding

ParentingParenting•MonitoringMonitoring•BondingBonding

Mental Mental ProcessesProcesses

•Social knowledgeSocial knowledge•ScriptsScripts•Schema/skillsSchema/skills

Mental Mental ProcessesProcesses

•Social knowledgeSocial knowledge•ScriptsScripts•Schema/skillsSchema/skills

Conduct Conduct ProblemsProblems

•Conduct disorderConduct disorder•FightingFighting•Drug useDrug use

Conduct Conduct ProblemsProblems

•Conduct disorderConduct disorder•FightingFighting•Drug useDrug use





Social Information Processing Theory:SIP Skills and Emotional Regulation as Malleable

Mediators?State the

problem

Generate

potentialsolutions

Evaluate potentialsolutions

Select &enact the

best solution(s)

Assessoutcomes

Encode social cues

Interpretsocial cues

Arousal, Emotions,Social Knowledge

Setgoal(s)

Social Knowledge: Life experiences producing scripts, schemata, skills, and beliefs

PROGRAM THEORY(specifies how a program is to work)

InterventionProgram StructureSuch as:Targeting Unit: Classroom Entire school Other (after school, family) Curriculum Structure: Distinct activities Embedded in curriculumActivities to address SACD GoalsSuch as:Character educationViolence prevention/peace promotionSocial and emotional developmentTolerance and diversityRisk prevention and health promotionBehavior management

Social - Emotional

Competence (mediator)

Attitudes about aggressionSelf-efficacyEmpathy

School Climate (mediator)

School connectednessVictimizationFeelings of safety at schoolParent involvement

BehaviorPositive BehaviorResponsible behaviorProsocial behaviorSelf-regulationCooperationNegative behaviorAggressionMinor delinquencyDisruptive classroom behavior

Moderating FactorsChild Family CommunityGender Parenting practices Community risk factorsSocioeconomic status Home atmosphere Social capitalRace/ethnicityRisk status Program SchoolPrior test scores/grades Fidelity Activities to promote social and character development

Intensity and dosage Organizational structure

Social and Character Development: Social and Character Development: Prevention ModelPrevention Model

AcademicsAcademic competenceSchool engagementGradesStandardized test scores

Social Development Model PerspectiveInstruction in social & emotionalskills . . .

EmpathyAnger managementProblem solving & impulsecontrol

Opportunity to . . .Discuss and identify feelingsAcquire language andcommunication skillsPractice solving problemsObserve models

Reinforcement andgeneralization of learningthrough . . .

Naturally occurringopportunities in schoolHome discussion of materials

Engaged school behavior. . .

Focusing on workPaying attentionFollowing instructions

School success

Reductions in . . .Classroom disruptivebehaviorAnxietyAnger

Reductions in . . .Problem behaviorAggressionPeer rejection

Social & emotional skills . . .EmpathyAnger managementProblem solving &impulse control

Intervention Immediate outcomes Knock-on outcomes Outcomes

PROGRAMS

*Developed by the program investigators, the intervention simultaneously focuses on the characteristics of children and on the classrooms in which they learn. The intervention combines three components.

Social Skills Training for students

ClassroomBehavior

Management Training and Consultation

for teachers

Social Dynamics Training for teachers

Group randomization:

Cohort 1: Hoke and Wayne Counties (10 schools randomized to 5 intervention; 5 control)

Cohort 2: Franklin County (4 schools randomized to 2 intervention; 2 control)

The Competence Support Program*

Program Elements• Making Choices: Skills Training curriculum for students in

elementary school. In-service training introduced teachers to the risks of peer rejection and social isolation, including poor academic outcomes and conduct problems. Throughout the school year, teachers received consultation and support (2 times per month) in providing lessons designed to enhance children’s social information processing and other skills. As a part of the Standard Course of Study, the program was integrated into routine class instruction.

• Classroom Behavior Management provided teacher consultation on classroom management strategies designed to strengthen engagement in instructional activities.

• Social Dynamics Training provided teacher consultation on classroom contexts, social groupings, and interactional patterns that can be used to reinforce academic achievement and prosocial behavior.

Theory of Change: Theory of Change: Making ChoicesMaking Choices

Training the Teacher or Counselor

Application of Making Choices by Teacher or Counselor

SIP skills of the Children in the School

Impact on Social Engage-ment and Peer Rejection

Impact on Disruptive Behavior and Academics

Characteristics of the Teacher or Counselor

Characteristics of the Children and the Classroom

Ran

dom

Ass

ignm

ent

Core #1

Core #2

Core #3

Core #4

Core #5

Note. In a randomized trial, you must figure out a way to measure each of the core elements.

Treatment as Usual Control Condition

•Assess implementation of training•Assess if teacher acquires skills from training/supervision

•Test the degree to which the intervention is delivered as intended, e.g., specific activities

Make Program Manuals• From risk mechanisms, mediators, and

logic models to the design of a program

• Specifying program activities that target the malleable mediators and have cultural congruence

• Example: Making Choices

For a discussion of issues in the development and use of treatment manuals, see: Galinsky, M. J., Terzian, M. A., & Fraser, M. W. (2006). The art of group work practice with manualized curricula. Social Work with Groups, 29(1), 11-26.

Warning: It is easy to under estimate the difficulty of developing a program manual.

“That Sunk Feeling”

Source: Don Moyer, Harvard Business Review (October, 2004, p. 160)

If you start in the wrong place, it usually does not help to dig deeper!

Start with theory and research, plus practice

experience…

Develop a template for each lesson or session

How to begin in the right place…

Recognizing Your Feelings

Objectives:

The learner will recognize that certain situations bring out feelings in all of us. The learner will practice recognizing their own feelings. The learner will use personal experiences and knowledge to interpret written

and oral messages. (SCS- LA 3.01) The learner will write structured, informative presentations and narratives

when given help with organization. (SCS- LA 4.08)

Materials:

Penguin Facts page, Response Sheets, Write About It worksheets A and B

Introduction

Review the idea that we all experience a variety of emotions and responses to emotions. Even when we experience the exact same situation, we may have different responses to the situation. Our responses to our feelings can cause us to do good things, but at times they can also cause us to do things that are not helpful.

Activity I: Pete the Penguin

Using two columns, list on the board the emotions presented in Lesson 1 of the book, The Way I Feel. Column I- Emotions that Feel Good: happy, silly, excited, proud, or

thankful Column II- Emotions that Don’t Feel Good: scared, sad, disappointed, bored, angry, or jealous Introduce the students to Pete the Penguin using the penguin puppet. Pass out the Penguin Facts page and discuss the factual information about penguins. Explain to the students that Pete has experienced events that have brought out many different emotions. Sometimes his emotions feel good, but at other times they don’t feel very good at all.

Review the emotions listed in the columns on the board. Then give each student four small pieces of paper (about the size of a note card). Read aloud the following events involving Pete the Penguin. After reading each event, ask the students, “How would you feel?” Give the students enough time to record their responses on one of

Grade 2Grade 2 Lesson Lesson 22

Activity 1

Overview

Review

PropAnswers

Process Tip

Standard Course

of StudyPrep Material

s

their blank pieces of paper. They can use the emotions on the board to express how they would feel or they may provide their own responses.

After you read each situation, collect a few responses randomly and read them aloud (so as not to bring attention to specific student responses). As you read through each response, discuss whether the event brought out a good feeling or a not-so-good feeling. The texts are ambiguous so that students can develop their own interpretations—not all students will feel the same way about each situation. Discuss the idea that everyone heard the same event, yet the feelings were different in many instances.

Today Pete walked in the classroom. As he walked to his desk, Pete noticed Susan and Tony talking quietly and laughing. They both looked up at Pete and giggled. If you were Pete, how would you feel?

When Pete was on the playground, he saw a group of students playing ball. He went to join them, and they told him he could play as soon as they started the next game. If you were Pete, how would you feel?

At lunch, Pete was sitting next to Jermaine. Jermaine opened his lunch and Pete looked inside. All he saw was two cookies and a drink box. If you were Pete, how would you feel?

Pete’s teacher told him he could play a game with Juan as soon as he finished his writing assignment. If you were Pete, how would you feel?

After discussing the above events, ask the students how they recognize when they are feeling certain emotions. “What happens when you start to feel angry?” “Happy?” “Frustrated?” and so on. (Example response: When you are getting angry- you might get hot, start to shake, get tense, grit your teeth, etc.)

Leave the list of emotions on the board to use in Activity II .

Activity II: Write About It

Give the students the Write About It page. On the top of the sheet, have students write about an event in their life that caused them to experience an emotion that made them feel good. On the bottom of the sheet they can write about an experience that caused an emotion that didn’t feel good. Each

narrative should describe the emotion, what caused it, and how they responded to the emotion. Students can refer to the columns on the board to choose the emotions they want to write about. Share the following examples aloud or on a transparency: Example 1: Once I felt excited when I was going to my friends party. I knew I felt this way because I was smiling and jumping around.

Avoid labeling

Scenarios

Activity 2: Write About It!

Develop all worksheets and artwork

G r a d e 2 - L e s s o n 2

A good feeling: Once I felt ___________________ when ____________________________________ ____________________________________ ____________________________________ ____________________________________ I knew I felt this way because__________ ____________________________________ ____________________________________

Activity II: Sheet A NAME: _________________

A not so good feeling: Once I felt _____________________ when ___________________________________ ___________________________________ ___________________________________ ___________________________________ I knew I felt this way because__________ ___________________________________

___________________________________

Activity II: Sheet B NAME: ____________________

““Pete the Penguin” Poster for Pete the Penguin” Poster for Grade 2Grade 2

Sample Lesson

Activities

from

Making Choices

Gr. 3 Lesson - Intentions

SYMBOLS

FRIENDLY

Grrrrr!

MEAN CAN’T TELL

Grade 3 Lesson 10

Intentions: Mean or Friendly?

LAUGHSWITHYOU

HITSYOU

SHARESWITHYOU

MAKESA FACEAT YOU

HUGSYOU

IGNORESYOU

TALKSABOUT

YOU

BITESYOU

HELPSYOU

Grrrrr!

MEAN

FRIENDLY

FRIENDLY

FRIENDLY

CAN’T TELL

GOAL SETTING

GOAL: Something a person wants or something a person wants to see happen.

RELATIONSHIP GOAL: Goals that involve wanting to get along with another person.

Grade 4 Lesson 6

Are these Relationship Goals?

• I want to make an “A” on my math test.• I want to play more often with my friend.• I want a new video game for my birthday.• I want to eat out at a restaurant for dinner.• I want to become friends with the new student.• I want to join in the basketball game at recess. • I want to sit with Jose on the bus.• I want to be in the class play this fall.• I want to stop getting upset when friends ignore me.

(thumbs up or thumbs down)

GOAL SETTING

• Set a relationship goal for these situations:

I was playing basketball at recess with some friends. Terrell, who is not very good at basketball, asked if he could play with us.

Set a Relationship Goal

Denise just made me really upset. She tried to pick a fight with me by saying things that are not true. I am feeling angry with her right now.

Set a Relationship Goal

Yesterday, my mom gave me a really cool pen that writes in all different colors. When I brought it to school this morning, Stacey asked me if she could borrow it. Last time I let Stacey borrow something she lost it, but if I say no she might get angry with me.

EVALUATION DESIGN:Cluster Randomized Trial

with Ten Schools Randomly Assigned to Treatment (j=5) and Control (j=5) Conditions

Cohort Design: Intervention provided in grades 3, 4, and 5

Prior Studies 1.Single-group qualitative trial of MC intervention (8th grade girls)2.Two-group cluster randomized trial at classroom level in one middle school (6th grade only)3.Two-group cluster randomized trial at classroom level in one school (3rd grade)4.Two-group, MC+SF intervention randomized trial (11 sites, 3rd – 4th grade)5.Cohort sequential study by classroom in two schools (3rd grade)6.(Current) Two-group cluster randomized trail at 14 elementary schools

SAMPLE

Two Overlapping Samples

* One treatment school was reorganized into a different building and dropped the program between 4th and 5th grade; students from that school were excluded from the 3-4-5 sample.

Grade 3-4-5 Sample• 3rd, 4th, and 5th graders • 9 schools*• Only consented students on a 5th grade roster• Change=addition of entrants

n=370

n=414

Grade 5n=433

Grade 3-4 Sample• 3rd and 4th graders • 10 schools• Any consented students on a 3rd or

4th grade roster• Change=entrants-leavers

Grade 4n=557

Grade 3n=571

N Total Control Intervention p-value

Child (analysis sample)

Male child, % 618 47.7 50.0 45.3 .243

Age of child, years 618 7.92 7.94 7.89 .235

Black child, % 618 34.5 21.9 48.0 .001

White child, % 618 48.1 59.3 35.8 .001

Hispanic child, % 618 10.0 13.7 6.1 .001

Family (analysis sample)

Primary caregiver, not a HS graduate , %+ 564 13.5 15.0 11.7 .259

Two biological parents not in household, %+ 563 48.5 43.7 54.0 .014

Income-to-needs < = 1, %+ 550 26.0 24.6 27.6 .416

School (aggregate, school level)

Black, %* 10 41.3 28.4 51.4 .001

Free Lunch, %* 10 44.7 37.4 50.7 .001

Adequate Yearly Progress*# 10 81.9 84.7 79.0 .068

Pupil/teacher ratio, mean* 10 16.3 16.2 16.4 .825

Note. + MPR data from baseline child level data file. * NCES school level data (CCD 2003-2004) across all schools. # AYP Performance Composite score, Year 1 of the SACD study.

Equivalence of Intervention and Control Groups on Selected Child, Family, and School Attributes: Grade 3 Cohort 1

Difference in School-Level Academic Performance: Percentage at Grade Level

Cohort 1 AYP Performance Composite

50

55

60

65

70

75

80

85

90

02-03 03-04 04-05 05-06 06-07

Per

cen

t of s

tud

ents

ach

ievi

ng

at g

rad

e le

vel

Treatment

Control

Test results for 2005-06 and 2006-07 are based on a revised accountability model and are not comparable to those from previous years.

Equivalence of Intervention and Control Groups on Selected Site Specific Outcomes Grade 3 Sample (Pretest): Cohort 1 N Total Control Intervention p-value

SIP Skill Level Assessment - Child

Encoding 420 44.2 43.0 45.6 .149

Goal formulation 412 67.6 65.5 70.1 .074

Response decision making 410 66.0 64.1 68.3 .225

Carolina Child Checklist - Teacher

Social contact 549 3.8 3.7 3.8 .121

Cognitive concentration 549 3.2 3.2 3.2 .952

Social competence 549 3.3 3.3 3.3 .945

Social Aggression 549 4.1 4.1 4.0 .183

Interpersonal Competence Scale - Teacher

Aggression 548 2.5 2.4 2.6 .095

Academic competence 548 5.1 5.1 5.1 .948

Popularity 548 4.9 4.9 4.8 .798

Peer Interpersonal Assessment

Aggression 502 38.7 42.6 34.4 .152

Prosocial skills 502 82.8 84.8 80.6 .552

Sample sizes vary because pretest measures were collected from different respondents (teachers, students) at different times. SLA and Peer assessment pretest were collected from students at the end of 2nd grade. CCC and ICST were collected from teachers at the beginning of 3rd grade. SLA=Skill Level Assessment (SIP skill – HOME Scale adaptation by Dodge, 1980). CCC=Carolina Child Checklist (Macgowan et al. 2002 – Research on Social Work Practice). ICST-Interpersonal Competency Scale – Teacher (Xie et al., 2002, Social Development)

Teacher and Classroom Characteristics by Intervention Status

Characteristic

Intervention Teachers (n = 21)

Non Intervention Teachers (n = 23) Difference p-value

Demographics White %+ 73.7 95.5 -11.8 .051 Greater than a bachelor degree %+ 28.6 4.6 24.0 .033 Years of teaching, mean+ 11.6 14.1 -2.5 .385 Years teaching at current school, mean+ 4.1 8.0 -3.9 .040 Has regular/advanced teaching certificate, %+ 85.0 90.9 -5.9 .566

SACD classroom activities

N of SACD classroom strategies, mean+ 14.8 13.7 1.1 .267 Violence prevention hours, mean+ 7.8 2.7 5.1 .039 Social and emotional development hours, mean+ 10.9 2.9 8.0 .002

Classroom observations Number of feedback and structure exemplars in place, mean# 6.2 4.2 2.0 .019

Note. +MPR baseline data (spring). #Observations were conducted with the Classroom Observation Form (COF) by an intervention specialist blind to the treatment or control condition of each school. The COF assesses seven domains relevant to the intervention: daily routines, time and task management, consequences and follow-through, teaching alternative behaviors, communication and feedback, and group processes and peer support.

Causation in Research Design:Randomization Is Supposed to Produce the Counterfactual

Note. We let the control group serve as evidence for what would have happened counter to the fact of participation in intervention (the stat class). Randomization is supposed to create equivalence or balance between the intervention (taking the stat class) and control (not taking the stat class) groups. But it didn’t. On several observed and an unknown number of unobserved measures, the intervention and control group schools differ.

Four evaluation challenges

Selection Bias: Covariates are not balanced between treated and control groups

Missing Data: No baseline data on enterers and lost data on leavers = constant churning of sample

Rater Effects: Outcome ratings were made by the same teachers within grades, but different teachers over grades 3, 4, and 5• Piecewise analyses – change scores within grade level

Treatment Contamination/History: High intervention content in control schools*

Note. Student Citizen Act (SL 2001-363) was passed into law by the Legislature in 2001. The Act required local boards of education to develop and incorporate character education instruction into standard curricula. Local boards of education began implementation in the 2002-2003 school year.

MEASURES

Site-Specific Outcomes

• Skill Level Assessment Activity (SLA): Based on the Dodge Home Scale (1980), the SLA uses students’ responses to questions about hypothetical social situations. After viewing picture scenarios, students answer questions measuring different aspects of social information processing skill: encoding (α=.78), goal formulation (α=.76), and response decision making (α=.80).

• Carolina Child Checklist (CCC): The CCC is a 35 item teacher questionnaire that yields factor scores on children’s behavior including social contact (α=.90), cognitive concentration (α=.97), social competence (α=.90), and social aggression (α=.91).

• Interpersonal Competence Scale-Teacher (ICST): The ICST is an 18-item teacher questionnaire that yields factor scores on children’s behavior including aggression (α=.84), academic competence (α=.74), and popularity (α=.78).

• Peer interpersonal assessments: Peer interpersonal assessments were used to examine classmates’ perceptions of participants’ social and behavioral characteristics including aggression (α=.92), prosocial skills (α=.84 ), and internalizing behavior (α=.67 ).

Grade 2 Grade 3 Grade 4 Grade 5 Wave 1 Wave 2 Wave 3 Wave 4 Wave 5 Wave 6 Wave 7 Wave 8 Wave 9 Time of year March Sept March April Sept March April Sept April Cohort 1 (C1) 2003-04 2004-05 2004-05 2004-05 2005-06 2005-06 2005-06 2006-07 2006-07 Cohort 2 (C2) 2004-05 2005-06 2005-06 2005-06 2006-07 2006-07 2006-07

Instrument (in-house data)

Making Choices Student report (SLA)

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C1

Friends and Groups Student report (Social Cognitive Maps)

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C1

Peer groups Teacher report (PG)

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C1

Carolina Child Checklist Teacher report (CCC)

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C1

Interpersonal Competence Scale Teacher report (ICST)

C1 C2

C1 C2

C1 C2

C1 C2

C1 C2

C1 C1

Peer Interpersonal Assessments Student report (PNOMS)

C1 C2

C1 C2

C1 C2

C1

Ratings of school adjustment Teacher report (TASS)

C1 C2

MPR data (across-site data)

Student reported C1

C2 C1

C2 C1

C1

C2 C1

Teacher reported C1

C2 C1

C2 C1

C1

C2 C1

Parent reported C1

C2 C1

C2 C1

C1

C2 C1

Summary of Data Collection Occasions

Minutes of Skills Training Instruction in 3rd and 4th Grades by Student

Benchmark=1,140 minutes

Below benchmark: 19%Above benchmark: 81%

(overall n=571)

ANALYTIC PROCEDURES

Analytic Procedures: Flow Chart for Use of “Bias-Correcting” Statistical Methods

Multiple Imputation of Missing Data(The imputation models employed both predictor and outcome variables, but the analysis models employed imputed missing values for predictor variables only). 50 imputations for each outcome variable

Piecewise change score HLM analysis using propensity score weighting (propensity scores estimated by gbm)

Estimation of propensity scores using Generalized Boosted Modeling (gbm) -- aims to optimize balance on observed covariates between treated and control groups

Heckman sample selection Model (Predictors of the selection equation are similar to the input of gbm)

Optimal pair matching using propensity scores estimated by gbm

Optimal full matching using propensity scores estimated by gbm

Post-pair-matching with regressionadjustment

Post-full-matching with Hodges-Lehmann aligned rank test

Dose (efficacy subset) analysis) using Abadie et al. Matching estimator

Procedures for multiple imputation of missing data

Test for MCAR (Little, 1988) confirms models are not MCAR.

Assumption of MCAR not required if imputation model is informed (i.e., data may be missing at random)

A diagnostic stage identified models that resulted in 99% relative efficiency for all analysis variables.

50 simulations (copies of the raw data set) generated using MI.

DVs and predictors both used in imputation; imputed DVs discarded after imputation (MID procedure; von Hippel, 2007).

Missing Data Diagnostics: Proportion without Missing Data and Proportion of Missing Data Points

Step 2: MatchingGreedy match (nearest neighbor with or without calipers) Mahalanobis with or without propensity scores Optimal match (pair matching, matching with a variable number of controls, full matching)

Step 1: Logistic regression Dependent variable: log odds of receiving treatmentSearch an appropriate set of conditioning variables (boosted regression, etc.) Estimated propensity scores: predicted probability (p) or log[(1-p)/p].

General Procedure for Propensity Score Analysis

Step 2: Analysis using propensity scores: Multivariate analysis using propensity scores as weights

Step 2:Analysis using propensity scores Analysis of weighted mean differences using kernel or local linear regression (difference-in- differences model of Heckman et al.)

Step 3: Post-matching analysis Multivariate analysis based on matched sample

Step 3: Post-matching analysis Stratification (subclassification) based on matched sample

Estimating propensity scores Need relevant conditioning variables Obtain “best” logistic regression (i.e., best

functional forms); however, no way to know Used Multiple Additive Regression Trees (MART)

to run logistic regression. Rand Generalized Boosted Modeling (gbm): Aims for best balance on observed covariates between treated and controlled groups. Iteration stops when the sample average standardized absolute mean difference (ASAM) is minimized.

Example of gbm output: Does gbm reduce the difference between treated and control schools?

STR=treatment group; LTR=control group; ASAM= average standardized absolute mean difference between treatment and control cases; pretreatment covariates: red solid diamonds= p-values before use of gbm weights (if below line then significant); black diamond outline = p-values after weights applied

Point: After using gbm propensity score weights, all pretreatment differences are ns.

Predictors of the propensity score model __________________________________________________________________________________________________________________________________________________________________

Outcome__________________________________________________________________________________________________________________________________________________________ICSTAGG ICSTACA ICSTINT CCCSCOM CCCPROS CCCEREG CCCRAGG

Predictor AggressionAcademic

Competence InternalizingSocial

Competence ProsocialEmotion

RegulationRelational

Aggression

Age at baseline (year) Gender female (male is reference) Race (Other is reference)

African American White Hispanic Primary caregiver's education (years of schooling) Ratio of income to poverty threshold Primary caregiver full-time employment (part-time is referrence) Father's presence in family: Yes (absence is reference) ICST-aggrestion at baseline ICST-academic competence at baseline ICST-internalizing at baseline CCC-emontion regulation at baseline CCC-social competence at baseline CCC-prosocial at baseline CCC-relational aggression at baseline __________________________________________________________________________________________________________________________________________________________________

Note. Predictors vary by outcome variable. Following the convention of propensity score analysis, we did not include predictors that are highly correlated with the outcome variable.

Propensity score weighting When estimating the treatment effect, can use propensity

scores as sampling weights.(Hirano & Imbens, 2001; McCaffrey et al., 2004; Rosenbaum, 1987)

Suppose p is the propensity score of receiving treatment. Then: Average treatment effect for the treated (ATT):

control weight = p/(1-p) treatment weight = 1

Average treatment effect for the population (ATE): control weight = 1/(1-p) treatment weight = 1/p

Post-optimal-matching analysis For the matched sample created by optimal pair

matching, regress pairwise differences in Y between treated and control cases on pairwise differences in X vector between treated and control cases (Rubin, 1979). In doing so, use the intercept of the regression to estimate the treatment effect and its p-value as a significance test.

For matched sample created by optimal full matching or optimal variable matching, use the signed-rank test of Hodges and Lehmann (1962) to estimate the average treatment effects.

Dose analysis using Matching estimator

• The dose analysis evaluates the outcome difference between a dose group (i.e., low, benchmark, or high) and a comparison group using Matching estimator developed by Abadie et al. (2004).

• Under the exogeneity assumption, this method imputes the missing potential outcome by using average outcomes for individuals with “similar” values on observed covariates.

• The estimator uses the vector norm (i.e., ||x||v=(x’Vx)1/2 with positive definite matrix V) to calculate distances between one treated case and each of the matched multiple nontreated cases, and chooses the outcome of the nontreated case whose distance is the shortest among all as the predicted outcome for the treated case.

Comparing model features

___________________________________________________________________________________________________________________________________Model The Model Controls for:

____________________________________

Level at which treatment was tested

Multiple imputation of missing data

Rater's effect

Selection bias Clustering

____________________________________________________________ ________ ________ ________ ________

Piecewise change with propensity score weighting (ATE) School Yes Yes Yes YesPiecewise change with propensity score weighting (ATT) School Yes Yes Yes YesOptimal pair matching with regression adjustment School Yes Yes Yes YesOptimal full matching with Hodges-Lehmann test Student Yes Yes Yes NoEfficacy subset analysis using Matching estimator Student Yes Yes Yes No___________________________________________________________________________________________________________________________________

Note. Regression models include covariates age at baseline, female, black, white, latino, primary caregiver education, income-to-poverty ratio, primary caregiver fulltime employment, father in household, and midyear change in teacher.

FINDINGS

Findings: Treatment effects measured by changes in the 3rd Grade (g34)

Outcome VariableHypothetical

Sign

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 1:

ATE Grade 3

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 2:

ATT Grade 3

Change Score Using

Optimal Pair-Matching (gbm) and Regression Adjustment

Grade 3

Change Score Using Optimal Full-

Matching (gbm) and Hodges-

Lahmann Aligned

Rank Test Grade 3

Approximate Sample Size Used in Analysis ≈571 ≈571 ≈542 ≈571ICSTAGG - Aggression - -.10 -.04 -.10 -.01ICSTACA - Academic competence + .12+ .08 .11 -.08***ICSTINT - Internalizing - .14+ .13+ .14 .17CCCSCOM - Social competence + -.22 -.25+ -.25 -.25CCCPROS - Prosocial + -.25 + -.26 + -.25 -.19+CCCEREG - Emotion regulation + -.16 -.18 -.20 -.24CCCRAGG - Relational aggression - .17 .21* .18 .28__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.

Findings: Treatment effects measured by changes in the 4th Grade (g34)


Sign

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 1:

ATE Grade 4

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 2:

ATT Grade 4

Change Score Using

Optimal Pair-

Matching (gbm) and Regression Adjustment

Grade 4

Change Score Using

Optimal Full-


Lahmann Aligned

Rank Test Grade 4

Approximate Sample Size Used in Analysis ≈557 ≈557 ≈550 ≈557ICSTAGG - Aggression - -.13 -.14 -.14 -.17ICSTACA - Academic competence + -.13+ -.10 -.11 -.08ICSTINT - Internalizing - -.00 .04 -.02 .18CCCSCOM - Social competence + -.01 -.02 .06 .05CCCPROS - Prosocial + -.00 -.03 .05 .07+CCCEREG - Emotion regulation + -.00 -.01 .06 .03CCCRAGG - Relational aggression - -.12 -.12 -.13 -.06__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.

Findings: Treatment effects measured by changes in the 3rd Grade (g345)


Sign

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 1:

ATE Grade 3

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 2:

ATT Grade 3

Change Score Using

Optimal Pair-

Matching (gbm) and Regression Adjustment

Grade 3

Change Score Using Optimal Full-


Lahmann Aligned

Rank Test Grade 3

Approximate Sample Size Used in Analysis ≈370 ≈370 ≈314 ≈370ICSTAGG - Aggression - -.15 -.12 -.12 -.08ICSTACA - Academic competence + .15 .12 .03 .28ICSTINT - Internalizing - .09 .09 .13 .19+CCCSCOM - Social competence + -.02 -.03 -.04 -.03CCCPROS - Prosocial + -.03 -.05 -.04 -.01CCCEREG - Emotion regulation + -.01 -.01 -.04 .03CCCRAGG - Relational aggression - .09 .09 .07 .07__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.



Sign

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 1:

ATE Grade 4

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 2:

ATT Grade 4

Change Score Using


Grade 4

Change Score Using

Optimal Full-


Lahmann Aligned

Rank Test Grade 4

Approximate Sample Size Used in Analysis ≈414 ≈414 ≈380 ≈414ICSTAGG - Aggression - -.17 -.21 -.18 -.21+ICSTACA - Academic competence + -.08 -.08 -.06 -.13ICSTINT - Internalizing - -.07 -.03 -.02 .13CCCSCOM - Social competence + .15+ .18* .20 .13CCCPROS - Prosocial + .15 + .14 + .17 .11+CCCEREG - Emotion regulation + .17+ .20+ .22 .21+CCCRAGG - Relational aggression - -.15 -.18 -.18 -.15+__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.



Sign

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 1:

ATE Grade 5

Piecewise Change

with Propensity

Score (gbm)

Weighting Model 2:

ATT Grade 5

Change Score Using


Grade 5

Change Score Using

Optimal Full-


Lahmann Aligned

Rank Test Grade 5

Approximate Sample Size Used in Analysis ≈433 ≈433 ≈350 ≈433ICSTAGG - Aggression - -.08 -.08 -.12 -.01ICSTACA - Academic competence + .20* .20* .16 .16ICSTINT - Internalizing - -.17+ -.18+ -.17 -.24+CCCSCOM - Social competence + .27** .25** .29 .28*CCCPROS - Prosocial + .29*** .27*** .29 .28*CCCEREG - Emotion regulation + .24* .22* .27 .27+CCCRAGG - Relational aggression - -.20 -.22 -.24 -.24+__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.

Findings: Growth curve and dose models (g345)


Sign

Growth curve

(intervntn*months)

ATE

Growth curve

(intervntn*months)

ATT

Dose (cum. minutes)Grade 3

ATE


ATE


ATE(9 months) (9 months) (8 hours) (8 hours) (8 hours)

Approximate Sample Size Used in Analysis ≈472 ≈472 ≈370 ≈414 ≈433ICSTAGG - Aggression - -.01 -.02 -.00 -.09 .12ICSTACA - Academic competence + .03 .03+ .03+ .08 -.03ICSTINT - Internalizing - -.13*** -.14*** .02 -.06 -.08CCCSCOM - Social competence + .07** .06** .03 .10 -.05CCCPROS - Prosocial + .06*** .05* .02 .07+ -.10CCCEREG - Emotion regulation + .07*** .07*** .02 .07 -.10CCCRAGG - Relational aggression - -.05* -.05* .00 -.19** .22+__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.

Low Confidence:Do Not Cite

Findings of Efficacy Subset Analysis: Treatment effects measured by changes in the 3rd and 4th Grades (g34)

__________________________________________________________________________________________________________________________________________________________________________________


Sign

Low Exposure

(<900) versus

Comp. (0) Grade 3

Benchmark Exposure (900-1044)

versus Comp. (0) Grade 3

High Exposure (1045+) versus

Comp. (0) Grade 3

Adequate Exposure

(240+) versus Comp. (0) Grade 4

Approximate Sample Size Used in Analysis 343 372 446 545ICSTAGG - Aggression - .11 -.09 .01 -.25**ICSTACA - Academic competence + -.10 .10 .06 -.12ICSTINT - Internalizing - -.04 .10 .15 .09CCCSCOM - Social competence + -.50*** -.17+ -.23* .08CCCPROS - Prosocial + -.51*** -.20* -.24** .06CCCEREG - Emotion regulation + -.39** -.21* -.16+ .08CCCRAGG - Relational aggression - .38*** .10 .21* -.22**__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.

Findings of Efficacy Subset Analysis: Treatment effects measured by changes in the 3rd, 4th, and 5th Grades (g345)


Sign

Low Exposure

(<900) versus

Comp. (0) Grade 3

Benchmark Exposure (900-1044)


High Exposure (1045+) versus

Comp. (0) Grade 3

Adequate Exposure


Adequate Exposure


Approximate Sample Size Used in Analysis 240 260 295 354 354ICSTAGG - Aggression - .16 -.08 -.10 -.30*** -.10ICSTACA - Academic competence + .10 .14 .03 -.06 .10ICSTINT - Internalizing - -.17 .19 .14 .04 -.24*CCCSCOM - Social competence + -.28 -.08 -.02 .22* .29**CCCPROS - Prosocial + -.43* -.08 -.07 .20* .30**CCCEREG - Emotion regulation + -.27 -.09 .03 .23* .27**CCCRAGG - Relational aggression - .36+ .05 .11 -.31** -.20+__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.

Findings of Efficacy Subset Analysis: Treatment effects measured by changes in the 4th & 5th Grades using subsets of Grade 3 exposure (g345)


Sign

G3 Low Exposure

(<900) versus

Comp. (0) Grade 4

G3 Benchmark Exposure (900-1044)


G3 High Exposure (1045+) versus

Comp. (0) Grade 4

G3 Low Exposure

(<900) versus

Comp. (0) Grade 5

G3 Benchmark Exposure (900-1044)


G3 High Exposure (1045+) versus

Comp. (0) Grade 5

Approximate Sample Size Used in Analysis 221 246 285 234 252 288ICSTAGG - Aggression - -.09 -.45*** -.32** .34 -.05 -.26*ICSTACA - Academic competence + .61** -.09 -.22* .19 .17 .02ICSTINT - Internalizing - -.16 -.02 .12 -.50+ -.12 -.17CCCSCOM - Social competence + .38* .09 .19* .27 .29* .28**CCCPROS - Prosocial + .36* .12 .15 .33 .25+ .28**CCCEREG - Emotion regulation + .39* .10 .22* .09 .29* .28**CCCRAGG - Relational aggression - -.19 -.24+ -.32** .09 -.09 -.33**__________________________________________________________________________________________________________________________________________________________________________________

*** p <.001, ** p < .01, * p < .05, + p < .1, two-tailed.

SummaryFrom different methods of analysis, a pattern of small, cumulative program effects emerges across grades 3, 4, and 5. These analyses exclude one poorly performing that was dissolved in third year of the study.

Positive cumulative effects on: Social competence – including

• Prosocial behavior and • Skill in regulating emotions

Internalizing behavior Relational aggression

By HLM and efficacy subsets, promising effects observed on: Academic competence Aggression

Focuses on Program Development and Steps in Intervention Research

Focuses on (Selection) Bias-Correction Statistical Methods

For a description of Making Choices and copies of sample lessons, see http://ssw.unc.edu/jif/makingchoices/

ReferencesAbadie, A., Drukker, D., Herr, J. L., & Imbens, G. W. (2004). Implementing matching estimators for average

treatment effects in Stata. The Stata Journal 4(3), 290-311.

Fraser, M. W., Day, S. H., Galinsky, M. J., Hodges, V. G., & Smokowski, P. R. (2004). Conduct problems and peer rejection in childhood: A randomized trial of the Making Choices and Strong Families programs. Research on Social Work Practice, 14(5), 313-324.

Fraser, M. W., Galinsky, M. J., Smokowski, P. R., Day, S. H., Terzian, M. A., Rose, R. A., & Guo, S. (2005).Social information-processing skills training to promote social competence and prevent aggressive behavior in third grade. Journal of Consulting and Clinical Psychology, 73(6), 1045-1055.

Fraser, M. W., Nash, J. K., Galinsky, M. J., & Darwin, K. E. (2000). Making choices: Social problem-solving skills for children. Washington, DC: NASW Press.

Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social programs. New York, NY: Oxford University Press.

Galinsky, M. J., Terzian, M. A., & Fraser, M. W. (2006). The art of group work practice with manualized curricula. Social Work with Groups, 29(1), 11-26.

Guo, S., & Fraser, M. W. (in press). Propensity score analysis: Statistical methods and applications. Thousand Oaks, CA: Sage Press.

Hansen, B. B. (2007). Optmatch: Flexible, optimal matching for observational studies. R News, 7, 18-24.

Heckman, J. J. (1979). Sample selection bias as a specification error. Econometrica, 47, 153-161.

Heckman, J. J. (2005). The scientific model of causality. Sociological Methodology, 35, 1-97.

Hirano, K., & Imbens, G. (2001). Estimation of causal effects using propensity score weighting: An application to data on right heart catheterization. Health Services and Outcomes Research Methodology, 2, 259-278.

Hodges, J. L., & Lehmann, E. L. (1962). Rank methods for combination of independent experiments in the analysis of variances. Annals of Mathematical Statistics, 33, 482-497.

McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9, 403-425.

Rosenbaum, P. (1987). Model-based direct adjustment. Journal of the American Statistical Association, 82, 387-394.

Rosenbaum, P. (2002). Observational studies (2nd ed.). New York: Springer-Verlag. Rubin, D. B. (2008). For objective causal inference, design trumps analysis. The Annals of Applied

Statistics, 2(3), 808-840.

Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association,74(366), 318-328.

Rubin, D. B. (1979). Using multivariate matched sampling and regression adjustment to control bias in observational studies. Journal of the American Statistical Association,74(366), 318-328.

Von Hippel, P. T. (2007). Regression with missing Ys: An improved strategy for analyzing multiply imputed data. Sociological Methodology, 37(1), 83-117.

Note. CCC=Carolina Child Checklist; ICST = Interpersonal Competency Scale - Teacher

S t e p s i n I n t e r v e n t i o n R e s e a r c h

Step 1:

Specify Problem and Develop Program

Theory

Step 2:

Create and Revise Program Materials

Step 3:

Refine and Confirm Program Components

Step 4:

Assess Effectiveness in Variety of Settings and Circumstances

Step 5:

Disseminate Findings and Program Materials

Stage 1

Formulation

Stage 3

Differentiation

Stage 2

Revision

Stage 4

Translation/Adaptation

Four Stages in the Development of Program Manuals Integrated across the Five Steps in Intervention Research

Source: Fraser, M. W., Richman, J. M., Galinsky, M. J., & Day, S. H. (2009). Intervention research: Developing social programs. New York, NY: Oxford University Press.

mark w. fraser, pi, school of social work, unc-chapel hill steven h. day, school of social work

Documents

elementary school

school of education

school of social work

school of social work

social skills

bunn elementary

scurlock elementary

sandy grove elementary