OUT-OF-SCHOOL TIMEAS A DEVELOPMENTAL CONTEXT
Lessons from Successful (and Unsuccessful Afterschool Programs
Deborah Lowe Vandell
SRHD Biennial Conference
San Antonio TX
March 2010
The Policy Context• Disconnect between the length of the school day and children’s
academic, social, behavioral, and physical needs
• High rates of maternal employment– 74% of the mothers of children 6-17 yrs
• Concerns about– Low test scores– Unmet needs of English Language Learners– Negative effects of low supervision– Youth as victims & perpetrators of crime – Childhood obesity
• Some evidence of beneficial effects of afterschool programs but not always
• An element in Race to the Top and other school reform efforts
What is an after-school program?
• Narrow definition – programs that are offered by schools or other organizations on a daily basis throughout the school year
• Broad definition – includes extracurricular activities, sports, clubs, and sports offered by schools, libraries, and youth organizations
After-School Programs Sometimes Offer
Academic enrichment & remediation• Homework help• Science, computer & math clubs• Book clubs
Non-academic enrichment• Organized sports & recreational games• Music, drama• Arts and crafts• Scouts, 4-H, YMCA
Public and Private Investments in Programs
• After-school programs– Serve 7 million+ children– 21% of 6 to 9 yr olds & 14% of 10-12 yr olds– CA After-School Education and Safety Program– 21st Century Community Learning Centers– National Network of State Afterschool Networks
• Participation in at least 1 organized activity in a yr – 81% of 6- to 11-yr-olds– 83% of 12- to 17-yr-olds– 90% of non-poor vs 60% of poor children
Do afterschool programs have positive effects on child
developmental outcomes?
Effect Size
a statistic that measures the magnitude of a program’s impact on a particular outcome
One common metric – Cohen (1988)“small” - d = .2"medium” - d = .5“large” - d = .8
Effect sizes also can be benchmarked against those reported in other studies.
1. Aspirin on heart disease d = .03 2. School-based substance abuse prevention programs on
drug & alcohol use d = .093. Class size reductions on math achievement d = .23
A RECENT META-ANALYSIS
Durlak, J. A., & Weissberg, R. P. (2007)
49 reports of 73 programs
Meta-analyses of
1. All programs
2. Programs with Sequential & Active activities and Focused & Explicit content (SAFE)
• .
Durlak & Weissberg Meta-AnalysisOutcomes # of
StudiesOverall Effect Size
Met SAFE Criteria
Did not meet SAFE criteria
Self-perceptions
22 .34 .35 .14
School bonding
28 .14 .26 .03
+ social behaviors
35 .19 .30 .06
Misconduct 42 .18 .26 .07
Drug use 27 .11 .22 .03
School Achievement
20 .16 .31 .03
Grades 25 .11 .24 .05
School Attendance
20 .16 .31 .03
ADDITIONAL COMPONENTS OF PROGRAM QUALITY (NRC, Eccles & Gootman, 2002)
• Positive relationships between students and staff
• Positive relationships between students• Mix of academic and non-academic skill-
building activities• High levels of student engagement• Mastery orientation• Appropriate levels of structure• Opportunities for autonomy and choice
Study of Promising Afterschool Programs(Vandell, Reisner, Pierce et al. 2007)
• A longitudinal study of almost 3,000 low-income, ethnically diverse elementary and middle school student
• Eight states: CA, CO, CT, MI, MT, NY, OR, RI
• 14 communities: rural, small towns, mid-size cities, large cities
• All located in high-poverty communities
• First year devoted to identifying & describing promising programs
• Studied student academic & social outcomes over two
years
Programs received consistently high ratings on all of the program quality dimensions
• Positive staff-student relationships • Positive relationships between
students• Mix of skill-building activities• High levels of student engagement• Mastery orientation• Appropriate levels of structure• Opportunities for autonomy and
choice
Study ParticipantsElementary School Sample
1,796 students in Grades 3 & 4 from 19 schools
89% free- or reduced-price school lunch
88% students of color
Middle School Sample
1,118 students in Grades 6 & 7 from 16 schools
63% free- or reduced-price lunch
69% students of color
• Organized activities such as team sports, school-based activities and lessons
• Home alone and home with a sibling• “Hanging out” with peers without adult supervision
• Each reported on 4-point scales– 1 = not at all/ once or twice– 2 = about once a week– 3 = 2 – 3 times a week– 4 = 4 or more days a week
Other Afterschool Experiences
– Programs only (PO)
– Programs plus other activities (PP)
– Low supervision with activities
– Other
FOUR CLUSTERS WERE IDENTIFIED
Participation in Promising Programs and Other Activities
Over 2 YearsElementary Sample
• 54% regular program participants (about 90 day/yr)
2/3 Program Only; 1/3 Program Plus
• 15% low supervision
Middle School Sample
• 49% regular program participants (about 55
days/yr)
2/3 Program Only; 1/3 Program Plus
• 16% low supervision
Child OutcomesMeasured at Baseline, at the end of
Year 1, and at the end of Year 2
Test Scores Self-Reports Teacher Reports
Math
Reading
Work habits
Misconduct
Substance use
Work habits
Task persistence
Academic performance
Prosocial behavior
Aggression
• HLM analyses
• Level 1 - child
• Level 2 – school/program
• Difference scores
• 10 multiple imputations
Analytic Plan
Covariates
• Gender • Race-ethnicity • Grade level• Household income (per $1000)• Maternal (or guardian) education (in yrs)• Mother (or guardian) works full time• Household structure
Elementary Sample: Child Report Program Only vs. Low Supervision
Effect Sizes
-.58
.17
-.66
0.24
-0.70
-0.60
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
Work habits Misconduct
Year 1Year 2
Elementary Sample: Child Report Program Plus vs. Low Supervision
Effect Sizes
-.43
.36
-.51
.41
-0.60
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
0.50
Work habits MisconductYear 1Year 2
Elementary Sample: Teacher Report Program Only vs. Low Supervision
Effect Sizes
-.17
.19
.24 .23
.18 .17
-.34
.23.21
.23
.31
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
Academic Work habits Taskpersistence
Social skills Prosocialbehavior
Aggression
Year 1Year 2
Elementary Sample: Teacher Report Program Plus vs. Low Supervision
Effect Sizes
-.29
.21
.30.30
.35
0.25
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
Academic Work habits Taskpersistence
Social skills Prosocialbehavior
Aggression
Year 1Year 2
Elementary Sample Math Achievement Test Scores
Effect Sizes
.61
.73
.52
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Program Only Program Plus
Year 1
Year 2
Middle School Sample: Youth ReportProgram Only vs. Low Supervision
Significant Effect Sizes
-.32 -.31
-.47
-.55
0.20
-0.60
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
Work habits Misconduct Substance use
Year 1Year 2
Middle School Sample: Youth Report Program Plus vs. Low Supervision
Effect Sizes
-.37
-.31
0.33
-.64-.67-0.70
-0.60
-0.50
-0.40
-0.30
-0.20
-0.10
0.00
0.10
0.20
0.30
0.40
Work habits Misconduct Substance use
Year 1Year 2
Middle School Sample Math Achievement Test Scores
Effect Sizes
.57.55
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Program Only Program Plus
Year 1
Year 2
Conclusions: Elementary School Sample
Standardized TestsGains in math for PO after one year; gains for both PO and PP groups after 2 years
Child Reports Gains in work habits & reductions in misconduct for both PO & PP groups; larger effects after 2 years
Teacher ReportsGains in work habits, task persistence, social skills, prosocial behaviors & reductions in aggression for PO after first year; improvements for both PO and PP after 2 years
Conclusions: Middle School Sample
Standardized TestsGains in math achievement for PO & PP groups after 2 years (not 1 year)
Youth Self-Reports Both PO and PP youth reports reported gains in work habits after 2 years (not 1 year) and larger reductions in misconduct and substance use after two years
Teacher ReportsLittle evidence of differences between Program and low supervised youth
Why are after-school programs beneficial?
Study 2: Experience Sampling - Out-of-School Time
Vandell, Shernoff, et al. (2007, 2009) • 191 8th grade students, primarily low-income students of
color
• 8 middle schools in 3 communities
• Students wore watches that beeped them at random times
during non-school hours – afterschool, evenings, weekends
• 35 signals during a one week period in the fall and 35 signals during a one week period in the spring
Students Filled Out Logbooks
• After each signal, students recorded
– Who they were with
– Where they were
– What they were doing
– How they were feeling
– And their levels of effort, concentration, motivation
Very Little Missing Data!!
• On average, students responded to 33 of the 35 signals during the week.
• 12,143 after-school, evening, and weekend experiences were reported.
• 5, 136 of the experiences occurred after school.
Students’ Activities at Programs and Elsewhere
Student Activities
0 5 10 15 20 25 30 35
Homework
Enrichment
Eating
TV
Sports
Service
% of time
At program
Not at program
No program
Differences in Supervision (% of time)
No No programprogram
Not at Not at programprogram
At At ProgramProgram
Unsuper’d Unsuper’d peers peers
16.716.7 21.021.0 7.27.2******
Sup. PeersSup. Peers 26.426.4 26.026.0 91.191.1******
No adultsNo adults 38.438.4 38.438.4 7.5*7.5*****
AloneAlone 12.912.9 11.711.7 0.00.0
Sib careSib care 5.85.8 3.53.5 0.10.1
Differences in Motivation, Effort, and Feelings (4-pt ratings)
No No programprogram
Not at Not at programprogram
At At programprogram
MotivationMotivation 2.72.7 2.92.9 3.03.0******EffortEffort 1.91.9 1.91.9 2.52.5******ImportancImportancee
2.42.4 2.52.5 2.92.9******
Positive Positive EmotionsEmotions
2.22.2 2.32.3 2.52.5******
Negative Negative
EmotionsEmotions1.31.3 1.21.2 1.31.3
Choice, Concentration, and Effort
Study 3: Are Particular Program Features Associated with Child Developmental
Outcomes? (Pierce, Bolt, Vandell, 2010)
3-year longitudinal study of children who attended 30 programs of varying quality
n = 150 in 1st grade n = 120 in 2nd grade
n = 91 in 3rd grade
49% male89% white60% parents have college degrees
Observations Of Program Quality
4 program observations in 2nd grade and 3 observations in 3rd grade
staff-child relationships availability of age-appropriate activities
programming flexibility
Analyses control for family background and child prior functioning
Child Developmental Outcomes
• Teacher ratings – Mock Report Card
• Reading• Math
– Work habits (6 items, 5 pt ratings)– Social skills with peers (7 items, 5 pt ratings
• Collected from classroom teachers in G1, G2, and G3.
Analytic Strategy
HLM analyses in Grade 2 & Grade 3
Level 1 – child
Level 2 – program
Covariates:
gender, minority (yes/no), maternal education, family income, 2-parent household (yes/no), firm/responsive parenting, prior child functioning ,
Effects Associated With Program Quality Features in G2 and G3
Five Take-Home Messages
• Regular participation in high quality afterschool programs is linked to positive social and academic outcomes
• Gains are more evident after two years than after one year, suggesting that duration is important
• Youth reports of experiences reveal differences in activities, social partners, motivation, effort, and affect while attending afterschool programs
• There is evidence that particular aspects of children’s experiences are related to child developmental outcomes
• Out of school time IS an important developmental context
http://childcare.gse.uci.edu/
for more information