school of education university of california, irvine · table 27: skill development scores by grade...

i

Report to David and Lucile Packard Foundation Children, Families and Communities Program Grant# 2013-38940 Reporting Period: May 1, 2013 - October 31, 2013

December 2013

Deborah Lowe Vandell, Principal Investigator Valerie Hall, Project Scientist

Pilar O’Cadiz, Project Scientist Andrea Karsh, Research Specialist

School of Education

University of California, Irvine

Summer Learning Outcome Measures Report – December 2013 ii

TABLE OF CONTENTS CHAPTERS 1. SUMMER LEARNING TOOLBOX YEAR THREE REPORT 1

2. STUDENT SELF-REPORTS OF PROGRAM EXPERIENCES 5 A. Elementary Student Outcomes B. Middle/High School Student Outcomes

3. STUDENT SELF-REPORTS OF SKILL DEVELOPMENT, ATTITUDES AND BELIEFS, AND POSITIVE BEHAVIOR 12

A. Elementary Student Outcomes B. Middle/High School Student Outcomes

4. ASSOCIATIONS BETWEEN QUALITY OF PROGRAM EXPERIENCES AND STUDENT OUTCOMES 45 A. Elementary Student Outcomes B. Middle/High School Student Outcomes

5. 2013 IMPLEMENTATION EXPERIENCE AND PROGRAM FEEDBACK 50 6. CONCLUSIONS 52 REFERENCES 54

Summer Learning Outcome Measures Report – December 2013 iii

LIST OF TABLES

Table 1: Participating Summer Learning Programs and Number of Sites, 2011, 2012, and 2013 4 Table 2: Program Experiences, Elementary Student Self-Reports 8 Table 3: Program Experiences, Middle/High School Student Self-Reports 10 Table 4: Surveys Completed by All Summer Matters Programs, Summer 2013 12 Table 5: Psychometrics at Time 1 (Pre) and Time 2 (Post), Elementary Student Self-Reports of Skill

Development 14 Table 6: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Elementary Student Self- Reports of Skill Development, by Gender and Grade Level 14 Table 7: Skill Development, Elementary Student Self-Reports 15 Table 8: Skill Development Scores by Gender, Elementary Student Self-Reports 16 Table 9: Skill Development Scores by Grade Level, Elementary Student Self-Reports 17 Table 10: Skill Development Scores by Ethnicity, Elementary Student Self-Reports 18 Table 11: Psychometrics at Time 1 and Time 2, Elementary Student Self-Reports of Positive

Attitudes and Beliefs 19 Table 12: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Elementary Student Self-

Reports of Positive Attitudes and Beliefs, by Gender and Grade Level 19 Table 13: Positive Attitudes and Beliefs, Elementary Student Self-Reports 20 Table 14: Positive Attitudes and Beliefs Scores by Gender, Elementary Student Self-Reports 21 Table 15: Positive Attitudes and Beliefs Scores by Grade Level, Elementary Student Self-Reports 22 Table 16: Positive Attitudes and Beliefs Scores by Ethnicity, Elementary Student Self-Reports 23 Table 17: Psychometrics at Time 1 and Time 2, Elementary Student Self-Reports of Positive

Behavior 24 Table 18: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Elementary Student Self-

Reports of Positive Behavior, by Gender and Grade Level 24 Table 19: Positive Behavior, Elementary Student Self-Reports 25 Table 20: Positive Behavior Scores by Gender, Elementary Student Self-Reports 26 Table 21: Positive Behavior Scores by Grade Level, Elementary Student Self-Reports 26 Table 22: Positive Behavior Scores by Ethnicity, Elementary Student Self-Reports 27 Table 23: Psychometrics at Time 1 (Pre) and Time 2 (Post); Middle/High School Student Self-

Reports of Skill Development 29 Table 24: Internal Consistency (Cronbach alpha) at Time 1 (Pre) and Time 2 (Post); Middle/High

School Student Self-Reports of Skill Development, by Gender and Grade Level 29 Table 25: Skill Development, Middle/High School Student Self-Reports 30 Table 26: Skill Development Scores by Gender, Middle/High School Student Self-Reports 31 Table 27: Skill Development Scores by Grade Level, Middle/High School Student Self-Reports 32 Table 28: Skill Development Scores by Ethnicity, Middle/High School Student Self-Reports 33 Table 29: Psychometrics at Time 1 and Time 2, Middle/High School Student Self-Reports of Positive

Attitudes and Beliefs 34 Table 30: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Middle/High School

Student Self-Reports of Positive Attitudes and Beliefs, by Gender and Grade Level 34 Table 31: Positive Attitudes and Beliefs, Middle/High School Student Self-Reports 35 Table 32: Positive Attitudes and Beliefs Scores by Gender, Middle/High School Student Self-Reports 36 Table 33: Positive Attitudes and Beliefs Scores by Grade Level, Middle/High School Student Self-

Reports 37 Table 34: Positive Attitudes and Beliefs Scores by Ethnicity, Middle/High School Student Self-

Reports 38

Summer Learning Outcome Measures Report – December 2013 iv

Table 35: Psychometrics at Time 1 (Pre) and Time 2 (Post), Middle/High School Student Self-Reports of Positive Behavior 39

Table 36: Internal Consistency (Cronbach alpha) at Time 1 (Pre) and Time 2 (Post), Middle/High School Student Self-Reports of Positive Behavior, by Gender and Grade Level 39

Table 37: Social Competencies, Middle/High School Student Self-Reports 40 Table 38: Positive Behavior Scores by Gender, Middle/High School Student Self-Reports 41 Table 39: Positive Behavior Scores by Grade Level, Middle/High School Student Self-Reports 42 Table 40: Positive Behavior Scores by Ethnicity, Middle/High School Student Self-Reports 43 Table 41: Elementary Student Reports of Quality of Program Experiences—Positive Relationships

with Staff—Are Linked to Self-Reported Student Outcomes 46 Table 42: Elementary Student Reports of Quality of Program Experiences—Interest in Program

Activities—Are Linked to Self-Reported Student Outcomes 46 Table 43: Elementary Student Reports of Quality of Program Experiences—Positive Peer

Relationships—Are Linked to Self-Reported Student Outcomes 47 Table 44: Middle/High School Student Reports of Quality of Program Experiences—Positive

Relationships with Staff—Are Linked to Self-Reported Student Outcomes 48 Table 45: Middle/High School Student Reports of Quality of Program Experiences—Interest in

Program Activities—Are Linked to Self-Reported Student Outcomes 49 Table 46: Middle/High School Student Reports of Quality of Program Experiences—Positive Peer

Relationships—Are Linked to Self-Reported Student Outcomes 49

1

CHAPTER 1

SUMMER LEARNING TOOLBOX YEAR THREE REPORT

Project Overview This report presents findings from the third year of the Summer Learning Outcome Measures Project. The project extends the use of an online assessment tool to the summer learning context. The pilot (Year One) and initial field-testing (Year Two) of the Summer Learning Toolbox was conducted to determine the reliability, validity and feasibility of administering the summer learning outcome assessments via an online process.1 The Year One Summer Learning Outcome Measures pilot study demonstrated the reliability and validity of these measures for use in summer learning programs. The field test provided additional data on student outcomes. The Year Three Implementation of the Online Toolbox at six programs served to further document learning outcomes for youth participating in summer learning programs in the Packard Summer Matters Initiative. Key findings for the third year of implementation of the Summer Learning Outcome Measures Project are outlined here:

• The majority of both elementary and middle/high school students reported having good or excellent quality experiences.

• The majority of students reported good or excellent work habits (62-‐70% elementary; 65-‐70% of middle/high school)

• The majority of students reported good or excellent reading, math, and science efficacy scores:

- Reading efficacy (60-‐68% elementary; 59-‐62% middle/high school) - Math efficacy (74-‐78% elementary; 61-‐63% middle/high school - Science efficacy (58-‐64% elementary; 47-‐57% middle/high school

• The majority of students reported good or excellent social competencies (57-‐60% elementary; 66-‐68% middle/high school)

• Over half of elementary students (52-‐55%) and over a third of middle/high school students (34-‐37%) reported good or excellent levels of interest in science.

• Over half of elementary students (55-‐56%) and over a third of middle/high school students (38-‐43%) reported a good or excellent likelihood of pursuing a career in science.

• The majority of students (92-‐94% elementary an 95% middle/high) reported good or excellent likelihood of being successful in the future.

• Students reported good or excellent social competencies (57-‐60% of elementary; 66-‐68% of middle/high school students)

• Almost all elementary and middle/high school students reported low or medium levels of misconduct (96-‐99%)

1 Validity is the strength of the measure to support conclusions, inferences, and generalizations based on the measure. Reliability is the consistency of the measure over time, the degree to which it measures the same way under similar conditions with similar subjects.

Summer Learning Outcome Measures Report – December 2013 2

• For both elementary and middle/high school students, significant associations were found between program experiences and outcomes (p < .01). These associations were consistent with the results obtained in all previous implementations of the Online Toolbox.

Findings in this report can be used for the Packard Summer Matters Initiative’s efforts to support “the development of quality summer programs” and inform stakeholders (educators, parents, and leaders from the target communities) regarding the benefits of participation in summer learning programs. This first chapter provides an overview and background on the third year (2013) of the Summer Learning Outcome Measures Project. Chapter one also includes a description of program participation across the three years of the project. Chapter 2 presents results of the student self-reports of their experiences with program staff, activities and peers in year 3. Chapter 3 provides the findings from the student self-reports of skill development and positive behavior change in year 3. Chapter 4 shows the obtained associations between the quality of program experiences reported by students and the student outcomes in year 3. Chapter 5 summarizes feedback from participating programs for the current third year implementation experience. Chapter 6 highlights key findings of the 2013 Implementation of the Summer Learning Outcome Measures Online Toolbox. Background All of the scales included in the 2013 Summer Learning Outcome Measures Online Toolbox have been used successfully in evaluations of afterschool programs and in other research studies with elementary and middle/high school students (Vandell, et. al., 2013a, 2013b, 2012a, 2012b, 2012c, 2012d). In addition, the measures of Positive Behavior Change and Skill Development have demonstrated excellent psychometric properties that were consistent across subgroups of students in both the Summer 2011 pilot and Summer 2012 field test and in other previous studies. Pre-post student and staff surveys administered at the beginning and at the end of the summer programs measured changes in students’ skill development and positive behavior change. The post participation survey also included a measure of students’ program experiences: supportive relationships with peers, supportive relationships with program staff, and interest in program activities. Programs received technical assistance via email and phone during the survey administration. Feedback from program leads and site staff on their experience using the Online Toolbox was collected by UC Irvine researchers by phone and online. Customized reports were developed and provided to programs that participated in the 2013 Summer Learning Outcome Measures Implementation Project. These reports included summary statistics for both Time 1 and Time 2 scores. Programs were able to compare their students’ performance from the beginning of summer to the end of summer program administration at the site and program level and compare their scores to the aggregate of program scores across all Summer 2013 programs. Although an individual program’s results were not made available to


the other programs, each program was able to compare their own results to the aggregate results of all Summer 2013 programs that used the Online Toolbox. In addition, in 2013, programs had the option to include additional variables to the analysis of their program data (gender, ethnicity, ELL, FRL). Program Participation The Summer Learning Online Toolbox was made available to all 10 Summer Learning Communities that were part of the 2013 Summer Matters Initiative. Six programs opted to participate in the 2013 Summer Learning Outcome Measures Project. These programs were: Fresno, Los Angeles, Sacramento, San Bernardino, San Francisco, and Whittier. Five Summer Matters Initiative programs (Fresno, Los Angeles, Sacramento, Santa Ana THINK Together, and Whittier) participated in the Summer 2011 Pilot. Fresno, LA’s Best and Sacramento continued from Year One to Year Two. San Bernardino and Gilroy also joined in the 2012 Online Toolbox Field Test, in addition to three National Summer Learning Association (NSLA) grantees, for a total of 8 programs participating in 2012. Altogether, across the three years of the Summer Learning Outcome Measures, eight Summer Matters programs have participated, with three programs (Fresno, Los Angeles and Sacramento) participating across the three years. Four of the eight programs that have used the Online Toolbox to measure summer learning outcomes for their participating students will be transitioning to alumni status in 2014. Table 1 summarizes participation in the Summer Learning Outcome Measures Project of eight Summer Learning Communities of the Summer Matters Initiative for 2011, 2012 and 2013.


Table 1. Participating Summer Learning Programs and Number of Sites, 2011, 2012 and 2013

Summer Matters Program Participation Year

2013 Participating

Sites Surveys

Administered 2013

2011 2012 2013 Elem. Middle Fresno County Of Education*

Fresno Recreation, Enrichment, Scholastic Help (FRESH)

Yes Yes Yes 1 [K-8] Youth

Los Angeles Unified School District LA’s BEST Yes Yes Yes 12 0

Youth

Staff Sacramento City Unified School

District Youth Development Support Services

Yes Yes Yes 0 5 Youth

San Bernardino City School District Creative Before-And-Afterschool Programs

For Success (CAPS) No Yes Yes 5 1 Youth

San Francisco Department of Children, Youth and Their Families*

Hillcrest Elementary Opportunity Impact

No No Yes

1 Elem

Youth

Staff 1 [K-8]

Whittier City School District* Reach for the Stars Yes No Yes 3 2 Youth

Gilroy Unified School District* Super Power Summer Camp No Yes No

Santa Ana Unified School District THINK Together Yes No No

Totals: 5 5 6 31 sites (2013) *Programs transitioning to alumni status in 2014


CHAPTER 2

STUDENT SELF-REPORTS OF PROGRAM EXPERIENCES One important indicator of the success of any out-of-school program is the quality of students’ experiences. In the 2013 implementation of the Summer Learning Outcome Measures Project, this indicator of program success was measured by the Program Experiences Survey, which was adapted from the After-School Experiences Survey developed by Rosenthal and Vandell (1996). The After-School Experiences Survey items have been used extensively in studies of afterschool programs, including the NICHD Study of Early Child Care and Youth Development, the Study of Promising After-School Programs, and evaluations of THINK Together and the Tiger Woods Learning Center. It has excellent psychometric properties. The value of this measure is informing programs about how students perceive and assess the effectiveness of the program, their relationships with teachers and staff, and their relationships with other students. The measure is valuable because prior research has demonstrated associations between quality of students’ reported program experiences and measures of students’ skill development and positive behaviors. Students’ reports of more positive experiences in their afterschool program, for example, are linked to better skill development and behavior change as reported by teachers and program staff (Rosenthal & Vandell, 1996; Kataoka & Vandell, 2011; Vandell, O’Cadiz, Hall, & Karsh, 2012). The Student Self-Report of Program Experiences consists of 16 items that are rated using four-point rating scales: 1) Not at all true; 2) A little true; 3) Mostly true; 4) Really true. The measure includes three subscales, Positive Relationships with Staff (6 items), Interest in Program Activities (5 items), and Positive Peer Relationships (5 items). An example of an item in the Interest in Program Activities scale is “I like the activities here”. An example of an item in the Positive Relationships with Staff scale is “I trust the program staff here”. The Positive Peer Relationships subscale measures students’ relationships with the other students in the afterschool program with items such as “I get to know other kids really well here”. The Student Self-Report of Program Experiences measure was administered during the post-participation survey administration. Both elementary and middle/high school students completed the measure. The scale was administered only at the end of the summer programs, so that students could report on their experiences over the summer. From the Summer Matters programs, a total of 1,314 students completed the Student Self-Report of Program Experiences: 865 elementary students and 449 middle/high school students. Calculation of Program Experiences Scores For the Program Experiences measure, mean scores and categorical scores were calculated for each of the three subscales (Positive Relationships with Staff, Interest in Program Activities, and Positive Peer Relationships). These scores were calculated for each student who completed the Program Experiences measure. The reliability coefficients for the three subscales were good. For the Relationships with Staff subscale, the Cronbach’s Alpha was .76 for elementary students and


.82 for middle/high school students. For the Program Activities subscale, the Cronbach’s Alpha was .82 for elementary students and .87 for middle/high school students. For the Positive Peer Relationships subscale, the Cronbach’s Alpha was .88 for elementary students and .89 for middle/high school students. Calculation of each student’s Program Experiences mean scores was based on the average score of the item ratings for the items of each subscale. Each item was assigned a value of 1 to 4; this value corresponded to the rating selected by the student for that item (e.g. “Really true” = 4). For items that were negative statements, such as “I am really bored here”, the ratings were reverse-coded (e.g. “Really true” = 1). The mean scores for Positive Relationships with Staff were computed as the mean (average) of the six Positive Relationships with Staff item scores, the mean scores for Interest in Program Activities were computed as the mean of the five Interest in Program Activities item scores, and the mean scores for Positive Peer Relationships were computed as the mean of the five Positive Peer Relationships items. The mean scores ranged from 1.00 to 4.00; a score of 3.00 or higher would indicate that a student had good experiences in their summer program. After all individual student scores were calculated for the three Program Experiences subscales, the combined mean scores of the two main groups of students were calculated (elementary and middle/high school students). These overall scores were calculated from all the individual student scores from all the students who participated in the post-participation survey administration of the Summer 2013 Online Toolbox. These scores can provide a benchmark for comparison to an individual summer program or site. In addition to calculating mean or average scores, categorical designations for Positive Relationships with Staff, Interest in Program Activities, and Positive Peer Relationships were determined. The categorical scores were created to designate Low, Fair, Good, and Excellent Quality experiences. The cut-points were established based on the four-point item rating scale: 1) Not at all true; 2) A little true; 3) Mostly true; 4) Really true. A score of 3.00 to 3.59 was categorized as Good, because this score would show that students rated the majority of statements about their afterschool program as “Mostly true”. A mean score of 3.6 to 4.0 was categorized as Excellent, because this score would show that the majority of items were rated as “Really true”. Similarly, a mean score of 2.0 to 2.9, which corresponded to the rating “A little true”, was categorized as Fair, and a mean score of 1.0 to 1.9 was categorized as Low. Table 2 summarizes all the Program Experiences scores for elementary students and Table 3 summarizes all scores for middle/high school students. Each table also includes scores for groups of students: scores by gender, grade level, and ethnicity. The average Relationships with Staff score for elementary students in the Summer Matters programs was 3.2, the average score for Program Activities was 3.3, and the average score for Positive Peer Relationships was 3.2. The categorical scores show that about 70% of elementary students in the Summer Matters programs reported good (41%) or excellent (21%) relationships with staff, about 26% of students reported fair quality relationships, and 5% reported low quality relationships with staff. For the Program Activities subscale, about 68% of students reported good (21%) or excellent (47%) interest in program activities, about 28% of students reported fair


quality experiences, and 4% reported low quality experiences. For quality of Peer Relationships, about 68% of students reported good (23%) or excellent (45%) quality of experiences with peers; 26% of students reported fair experiences; and 5% reported low quality of experiences with peers at their summer program. Overall, program experiences scores were consistent across groups of students. Elementary girls reported slightly higher scores than those reported by elementary boys, but all scores were high, ranging from 3.2 to 3.4. Scores across grade levels were also consistently high (3.2 to 3.3), with 3rd-4th grade students reporting slightly higher scores than 5th-6th grade students. Scores across ethnicity groups were comparable, ranging from 3.0 to 3.4.


Table 2: Program Experiences, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)

% Excellent (3.6-4.0)

Elementary (All Programs) Positive Relationships with Staff 864 3.21 4.5% 25.6% 40.7% 29.2% Interest in Program Activities 865 3.28 4.3% 27.6% 20.9% 47.2% Positive Peer Relationships 864 3.21 5.3% 26.4% 23.4% 44.9% Scores by Gender

Boys Positive Relationships with Staff 415 3.15 5.3% 27.2% 45.1% 22.4% Interest in Program Activities 415 3.20 4.6% 32.3% 21.7% 41.4% Positive Peer Relationships 415 3.17 5.1% 28.7% 24.6% 41.7%

Girls Positive Relationships with Staff 449 3.26 3.8% 24.1% 36.7% 35.4% Interest in Program Activities 450 3.35 4.0% 23.3% 20.2% 52.4% Positive Peer Relationships 449 3.24 5.6% 24.3% 22.3% 47.9% Scores by Grade Level

3rd-4th Positive Relationships with Staff 476 3.26 4.6% 22.7% 44.1% 28.6% Interest in Program Activities 476 3.29 4.4% 26.5% 21.6% 47.5% Positive Peer Relationships 476 3.24 5.0% 26.5% 21.2% 47.3%

5th-6th Positive Relationships with Staff 388 3.15 4.4% 29.1% 36.6% 29.9% Interest in Program Activities 389 3.27 4.1% 29.0% 20.1% 46.8% Positive Peer Relationships 388 3.16 5.7% 26.3% 26.0% 42.0% Scores by Ethnicity

African; African American Positive Relationships with Staff 41 3.14 9.8% 17.1% 53.7% 19.5% Interest in Program Activities 41 3.23 4.9% 29.3% 19.5% 46.3% Positive Peer Relationships 41 3.02 14.6% 31.7% 12.2% 41.5%

Asian; Asian American Positive Relationships with Staff 60 3.09 6.7% 30.0% 40.0% 23.3% Interest in Program Activities 60 3.20 10.0% 26.7% 11.7% 51.7% Positive Peer Relationships 60 3.04 10.0% 30.0% 25.0% 35.0%

Caucasian; European American Positive Relationships with Staff 27 3.17 7.4% 25.9% 37.0% 29.6% Interest in Program Activities 27 3.38 3.7% 18.5% 29.6% 48.1% Positive Peer Relationships 27 3.23 7.4% 25.9% 11.1% 55.6%

Hispanic; Latino Positive Relationships with Staff 532 3.17 3.4% 23.3% 41.4% 32.0% Interest in Program Activities 532 3.26 4.3% 28.4% 23.1% 44.2% Positive Peer Relationships 532 3.23 3.4% 27.8% 23.9% 44.9% N= number of students; Mean = average score


For all middle/high school students in the Summer Matters programs, the average Staff & Activities score was 3.2; the average Program Activities score was 3.3; the average Positive Peer Relationships score was 3.2. Categorical scores show that about 71% of students reported good (41%) or excellent (30%) relationships with staff, about 25% of students reported fair quality relationships, and 4% reported low quality relationships with staff. For the Program Activities subscale, about 74% of students reported good (26%) or excellent (47%) quality experiences with program activities, about 24% of students reported fair quality experiences, and about 3% reported low quality experiences. For Positive Peer Relationships, about 72% of students reported good (33%) or excellent (39%) quality of experiences with peers; 24% of students reported fair experiences; and 4% reported low quality of experiences with peers at their summer program. Overall, program experiences scores were consistent across groups of students. Scores for middle/high school girls and boys were 3.2 to 3.3. Scores for 6th-7th grade students were also 3.2 to 3.3, and scores for 8th-9th grade students were 3.1 to 3.2. Students in 10th-12th grade had slightly lower scores: 2.7 to 3.1. Scores across ethnicity groups were ranged from 3.0 to 3.6. Overall, Asian American students and Hispanic/Latino students reported slightly higher quality program experiences than African American and Caucasian students.


Table 3: Program Experiences, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Middle/High (All Programs) Positive Relationships with Staff 449 3.19 4.0% 24.7% 41.2% 30.1% Interest in Program Activities 449 3.31 2.7% 23.8% 26.1% 47.4% Positive Peer Relationships 449 3.20 4.0% 23.6% 33.4% 39.0% Scores by Gender Female Positive Relationships with Staff 242 3.20 4.1% 24.0% 41.3% 30.6% Interest in Program Activities 242 3.30 4.1% 21.9% 23.1% 50.8% Positive Peer Relationships 242 3.19 3.7% 24.8% 32.6% 38.8%

Male Positive Relationships with Staff 207 3.19 3.9% 25.6% 41.1% 29.5% Interest in Program Activities 207 3.32 1.0% 26.1% 29.5% 43.5% Positive Peer Relationships 207 3.21 4.3% 22.2% 34.3% 39.1% Scores by Grade Level 6th-7th Positive Relationships with Staff 391 3.22 3.8% 23.5% 40.9% 31.7% Interest in Program Activities 391 3.32 2.8% 23.0% 25.6% 48.6% Positive Peer Relationships 391 3.21 4.1% 22.3% 34.3% 39.4%

8th-9th Positive Relationships with Staff 48 3.10 4.2% 29.2% 45.8% 20.8% Interest in Program Activities 48 3.22 27.1% 35.4% 37.5% Positive Peer Relationships 48 3.19 2.1% 31.3% 27.1% 39.6%

10th-12th Positive Relationships with Staff 10 2.73 10.0% 50.0% 30.0% 10.0% Interest in Program Activities 10 3.12 10.0% 40.0% 50.0% Positive Peer Relationships 10 2.86 10.0% 40.0% 30.0% 20.0% Scores by Ethnicity African; African American Positive Relationships with Staff 50 3.07 4.0% 34.0% 36.0% 26.0% Interest in Program Activities 50 3.30 2.0% 30.0% 22.0% 46.0% Positive Peer Relationships 50 3.19 2.0% 30.0% 32.0% 36.0%

Asian; Asian American Positive Relationships with Staff 18 3.45 16.7% 27.8% 55.6% Interest in Program Activities 18 3.59 11.1% 27.8% 61.1% Positive Peer Relationships 18 3.30 5.6% 16.7% 33.3% 44.4%

Caucasian; European American Positive Relationships with Staff 28 3.15 35.7% 32.1% 32.1% Interest in Program Activities 28 3.29 28.6% 35.7% 35.7% Positive Peer Relationships 28 2.97 17.9% 21.4% 28.6% 32.1%

Hispanic; Latino Positive Relationships with Staff 157 3.38 1.3% 16.6% 39.5% 42.7% Interest in Program Activities 157 3.49 1.9% 14.0% 22.9% 61.1% Positive Peer Relationships 157 3.26 2.5% 18.5% 36.9% 42.0% N= number of students; Mean = average score


Chapter Summary

Analyses of the student reports of Program Experiences show that a substantial majority of elementary and middle/high school students in the Summer Matters programs had high quality program experiences. Students reported positive relationships with program staff, satisfaction with program activities, and good relationships with peers. All participating programs were provided with the summary Program Experiences scores for all sites, as well as the scores for their overall program and each of the participating sites in their program. These scores can be used to compare the Program Experiences scores of individual programs and sites to the Program Experiences scores of all participating programs in the Summer 2013 implementation. Programs can use these results to help guide program improvement. The next chapter summarizes the findings pertaining to all student outcomes (skill development, positive attitudes and beliefs, and positive behavior change).


CHAPTER 3 STUDENT SELF-REPORTS OF SKILL DEVELOPMENT, ATTITUDES AND BELIEFS,

AND POSITIVE BEHAVIOR This chapter presents the primary results of the online assessment of students’ skill development, attitudes and beliefs, and positive behavior collected during Summer 2013. Reports of student outcomes were obtained from the students themselves (self-reports), and from program staff (reports on individual students). From the six Summer Matters programs that participated during Summer 2013, over 2,500 youth outcome reports were obtained (n = 2,618): 1,267 pre-participation surveys (718 from elementary students, 71 from middle/high school students, and 71 from program staff), and a total of 1,351 post-participation surveys were completed (888 elementary students, 454 middle/high students, 9 program staff). 848 of the students in the Summer Matters programs completed both pre- and post-participation surveys (50% of the 1,690 student participants). Table 4 provides the overall participation rates for all programs that participated in the Summer 2013. Table 4: Surveys Completed by All Summer Matters Programs, Summer 2013

Pre-Participation Surveys Post-Participation Surveys

# of reports,

elementary students

# of reports, middle/high

students

# of reports, program

staff

# of reports, elementary

students

# of reports, middle/high

students

# of reports, program

Staff Summer Matters 2013 (All Programs) 718 478 71 888 454 9

Totals by Program Creative Before-and-Afterschool Programs for Success (CAPS) San Bernardino City School District

235 8 -- 186 8 --

Fresno Recreation, Enrichment, and Scholastic Help (FRESH) Fresno County of Education

111 211 -- 93 160 --

LA’s BEST Los Angeles Unified School District 300 -- 13 393 0 --

San Francisco Department of Children, Youth, and Their Families 72 17 58 76 26 9

Whittier City School District -- -- -- 140 95 -- Youth Development Support Services Sacramento City Unified School District

-- 242 -- -- 165 --


Calculation of Student Outcome Scores Similar to the scores for program experiences (see Chapter 2), outcome scores are calculated and reported both as mean scores and as categorical designations. The mean score for each outcome scale represents the average of all the items of that scale. In the case of negative statements or problem behaviors, the ratings are inverted (e.g. “Not at all true” = 4), so that higher scores represent better student outcomes. In addition to mean scores, student outcome scores are reported as categorical designations that are grounded in the terminology of the specific scale (e.g. 1 = Not at all true; 2 = A little true; 3 = Mostly true; 4 = Really true”). These categorical designations indicate the proportions of students whose outcomes receive Low scores (scores between 1.0 and 1.9), Fair scores (scores between 2.0 and 2.9), Good scores (scores between 3.0 and 3.59), or Excellent scores (scores between 3.6 and 4.0). The next sections of this chapter present the findings pertaining to the student outcome scores. The findings for elementary students are presented first, followed by the findings for middle/high school students. Program Staff Reports of student outcomes were completed by only two programs at Time 1 (Pre) and only one program at Time 2 (Post), so those scores are not reported.

A. ELEMENTARY STUDENT OUTCOMES

This section presents the student outcome scores for elementary students (Grades 3 to 6). Elementary student self-reports of student outcomes are organized in three broad areas of skill development, positive attitudes and beliefs, and positive behavior. Within each section, the following scores are presented: psychometrics for all elementary students and for groups of students (gender, grade level), mean scores and categorical scores for all Summer Matters 2013 elementary students (aggregate), and mean scores and categorical scores for groups of elementary students (gender, grade level, ethnicity). Elementary students in grades 3, 4, 5, and 6 completed self-reports of outcomes in which they rated their own skills, attitudes, and behaviors. In the area of skill development, four measures were included: work habits, reading efficacy, math efficacy, and science efficacy. In the area of positive attitudes and beliefs, two measures were included: science interest and view of future. In the area of positive behavior, two measures were included: social competencies and misconduct. From all Summer Matters 2013 programs, a total of 718 elementary school student self-reports were completed at the beginning of the summer programs (Time 1, Pre) and 888 were completed at the end of the summer programs (Time 2, Post). Skill Development The Elementary Student Self-Report includes four measures of skill development: work habits, reading efficacy, math efficacy, and science efficacy. All four measures are assessed on a 4-point scale (1 = not at all true, 4 = really true). Sample items on the Work Habits scale include: “I


work well by myself” and “I finish my work on time.” Items on the Reading Efficacy scale assess a student’s sense of reading competency (e.g. “I expect to do well in reading”) and subjective task value (e.g. “I am interested in reading.”). The Math and Science Efficacy scales includes items such as “I expect to do well in math” and “I am interested in science.” For all four measures of skill development, an overall score is computed as the mean of all the items of each scale. Psychometrics for the skill development measures were calculated for all elementary student participants in the Summer 2013 implementation of the Online Toolbox, as well as for subgroups of students (by gender, grade level, and ethnicity). Table 5 summarizes the psychometrics for the whole sample of elementary students and Table 6 summarizes the reliability coefficients (Cronbach’s Alpha) by gender and grade level. These analyses show that the measures had excellent psychometric properties, consistent across subgroups of students. The figures are consistent with those obtained in the Summer 2011 Pilot Study, the Summer 2012 implementation, the California Afterschool Outcome Measures Project Field Test (2011), and also with results reported in other previous studies (Vandell et al., 2011). Table 5: Psychometrics at Time 1 (Pre) and Time 2 (Post), Elementary Student Self-Reports of Skill Development

N Mean SD Range Alpha # Items Work Habits, Pre 718 3.05 .61 1.00 - 4.00 .77 6 Work Habits, Post 888 3.15 .61 1.00 - 4.00 .81 6

Reading Efficacy, Pre 717 2.99 .82 1.00 - 4.00 .81 4 Reading Efficacy, Post 883 3.12 .78 1.00 - 4.00 .84 4

Math Efficacy, Pre 717 3.31 .80 1.00 - 4.00 .86 4 Math Efficacy, Post 883 3.37 .77 1.00 - 4.00 .88 4

Science Efficacy, Pre 716 2.96 .90 1.00 - 4.00 .88 4 Science Efficacy, Post 883 3.07 .89 1.00 - 4.00 .90 4

Table 6: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Elementary Student Self-Reports of Skill Development, by Gender and Grade Level

Gender Grade Level Boys Girls 3rd- 4th 5th- 6th

Work Habits, Pre .76 .76 .78 .75 Work Habits, Post .78 .83 .79 .83

Reading Efficacy, Pre .82 .79 .80 .82 Reading Efficacy, Post .82 .84 .83 .84

Math Efficacy, Pre .87 .84 .83 .87 Math Efficacy, Post .86 .89 .84 .90

Science Efficacy, Pre .88 .88 .88 .87 Science Efficacy, Post .89 .91 .88 .91 Table 7 summarizes the elementary student self-reported outcome scores for the four measures of skill development: work habits, reading efficacy, math efficacy, and science efficacy. For all elementary students in the Summer Matters programs, mean scores for work habits were 3.1 at


Time 1 (Pre) and 3.2 at Time 2 (Post), the mean scores for reading efficacy were 3.0 and 3.1, mean scores for math efficacy were 3.3 and 3.4, and mean scores for science efficacy were 3.0 and 3.1. Categorical scores show that about 62-70% of the elementary students in the Summer Matters programs self-reported good (42% Pre; 39% Post) or excellent work habits (26% Pre; 29% Post), and about 30-38% reported fair (33% Pre; 27% Post) or low work habits (5% Pre; 3% Post). The distributions for reading efficacy and science efficacy were comparable. About 60-68% of the reading efficacy scores were in the good (31% Pre; 35% Post) or excellent range (29% Pre; 33% Post) and about 58-64% of the science efficacy scores were in the good (26% Pre; 27% Post) or excellent range (33% Pre; 37% Post). Math efficacy scores were slightly higher, with about 74-78% in the good (24% Pre; 25% Post) or excellent range (51% Pre; 53% Post). Only 6% of elementary students reported low efficacy in math at Time 2 (Post). Table 7: Skill Development, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Summer Matters 2013 (All Programs)

Work Habits, Pre 718 3.05 4.7% 33.4% 40.5% 21.3% Work Habits, Post 888 3.15 3.3% 26.8% 42.0% 27.9%

Reading Efficacy, Pre 717 2.99 11.3% 29.0% 31.2% 28.5% Reading Efficacy, Post 883 3.12 7.2% 24.6% 34.9% 33.3%

Math Efficacy, Pre 717 3.31 7.4% 18.4% 23.7% 50.5% Math Efficacy, Post 883 3.37 6.0% 16.2% 24.5% 53.3%

Science Efficacy, Pre 716 2.96 13.7% 28.2% 25.6% 32.5% Science Efficacy, Post 883 3.07 11.6% 24.3% 27.3% 36.8% N= number of students; Mean = average score

Tables 8-10 shows elementary student self-reported outcome scores for the four measures of skill development by gender, grade level, and ethnicity. Mean scores for all measures, all groups, were within the good to excellent range (2.9 to 3.5). Within each group, students reported slightly higher math efficacy scores than for the other three measures. For all elementary boys in the Summer Matters programs, mean scores for work habits, reading efficacy, and science efficacy were 2.9 at Time 1 (Pre) and 3.1 at Time 2 (Post). The mean scores for math efficacy were 3.3 and 3.4. For all elementary girls, mean scores for work habits were 3.2 at Time 1 (Pre) and 3.2 at Time 2 (Post), scores for reading efficacy were 3.1 and 3.2, scores for math efficacy were 3.3 and 3.3, and scores for science efficacy were 3.0 and 3.1. For 3rd-4th grade students, mean scores for work habits, reading efficacy, and science efficacy were 2.9 at Time 1 (Pre) and 3.1 at Time 2 (Post). The mean scores for math efficacy were 3.4 and 3.5. For 5th-6th grade students, mean scores for work habits were 3.0 at Time 1 (Pre) and 3.1


at Time 2 (Post), scores for reading efficacy were 3.1 and 3.2, scores for math efficacy were 3.2 and 3.3, and scores for science efficacy were 2.8 and 2.9. Overall, 3rd-4th grade students reported slightly higher scores than 5th-6th grade students. Ethnicity data were provided for approximately 75% of elementary students. Scores are reported for the following groups: African or African American, Asian or Asian American, Caucasian or European American, and Hispanic or Latino. Two elementary students were categorized as Pacific Islander; the scores for these students are not reported. Across all four ethnicity groups, reported scores were high (3.0 to 3.5). For Caucasian students, reading efficacy scores were slightly higher than scores for the other skill development measures. For all other ethnicity groups, the math efficacy scores were slightly higher than for the other measures. Table 8: Skill Development Scores by Gender, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Gender

Boys Work Habits, Pre 344 2.92 6.7% 41.9% 35.2% 16.3% Work Habits, Post 430 3.05 4.2% 30.7% 44.0% 21.2%



Science Efficacy, Pre 342 2.88 14.9% 32.2% 23.1% 29.8% Science Efficacy, Post 427 3.07 10.8% 26.7% 25.8% 36.8% Girls Work Habits, Pre 374 3.17 2.9% 25.7% 45.5% 25.9% Work Habits, Post 478 3.24 2.4% 23.1% 40.2% 34.3%





Table 9: Skill Development Scores by Grade Level, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Grade Level

3rd-4th Grade Work Habits, Pre 337 3.10 4.5% 31.8% 38.9% 24.9% Work Habits, Post 495 3.22 2.2% 23.8% 41.0% 32.9%



Science Efficacy, Pre 334 3.12 10.4% 23.3% 25.4% 40.9% Science Efficacy, Post 487 3.20 10.0% 19.8% 27.8% 42.4% 5th-6th Grade Work Habits, Pre 381 3.00 5.0% 34.9% 42.0% 18.1% Work Habits, Post 393 3.05 4.6% 30.5% 43.3% 21.6%





Table 10: Skill Development Scores by Ethnicity, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Ethnicity

African; African American Work Habits, Pre 39 3.20 2.6% 30.8% 38.5% 28.2% Work Habits, Post 42 3.19 31.0% 35.7% 33.3%

Reading Efficacy, Pre 39 3.26 25.6% 38.5% 35.9% Reading Efficacy, Post 42 3.43 21.4% 21.4% 57.1%


Science Efficacy, Pre 39 3.13 2.6% 43.6% 15.4% 38.5% Science Efficacy, Post 42 3.33 4.8% 23.8% 19.0% 52.4% Asian; Asian American Work Habits, Pre 50 3.13 2.0% 28.0% 48.0% 22.0% Work Habits, Post 66 3.17 3.0% 27.3% 42.4% 27.3%



Science Efficacy, Pre 50 3.22 6.0% 24.0% 24.0% 46.0% Science Efficacy, Post 66 3.28 4.5% 21.2% 34.8% 39.4% Caucasian; European American Work Habits, Pre 25 3.39 20.0% 36.0% 44.0% Work Habits, Post 27 3.38 3.7% 11.1% 29.6% 55.6%

Reading Efficacy, Pre 25 3.61 16.0% 20.0% 64.0% Reading Efficacy, Post 27 3.60 7.4% 37.0% 55.6%

Math Efficacy, Pre 25 3.51 4.0% 16.0% 12.0% 68.0% Math Efficacy, Post 27 3.49 18.5% 25.9% 55.6%

Science Efficacy, Pre 25 3.09 8.0% 36.0% 16.0% 40.0% Science Efficacy, Post 27 3.19 7.4% 22.2% 25.9% 44.4% Hispanic; Latino Work Habits, Pre 346 3.08 4.6% 31.8% 39.9% 23.7% Work Habits, Post 544 3.19 2.8% 25.9% 41.9% 29.4%





Positive Attitudes and Beliefs The Elementary Student Self-Report includes two measures of positive attitudes and beliefs: science interest and view of future. Both measures are assessed on a 4-point scale (1 = not at all true, 4 = really true). An overall score is computed as the mean of the items. The science interest measure was developed at the Harvard University Program in Education, Afterschool & Resiliency (PEAR), and includes items such as “I get excited about learning about new discoveries or inventions.” The view of future measure includes two subscales: science career and view of future. The science career subscale includes items such as “I will get a job in a science-related area.” The view of future subscale includes items such as “I will go to college.” Psychometrics for the positive attitudes and beliefs measures and subscales were calculated for all elementary student participants in the Summer 2013 implementation of the Online Toolbox, as well as for subgroups of students (by gender and grade level). Table 11 summarizes the psychometrics for the whole sample of elementary students and Table 12 summarizes the reliability coefficients (Cronbach’s Alpha) by gender and grade level. These analyses show that the attitudes and beliefs measures had excellent psychometric properties, consistent across subgroups of students. Table 11: Psychometrics at Time 1 and Time 2, Elementary Student Self-Reports of Positive Attitudes and Beliefs

N Mean SD Range Alpha # Items Summer Matters 2013 (All Programs)

Science Interest, Pre 715 2.95 .66 1.00 - 4.00 .91 6 Science Interest, Post 881 2.99 .69 1.00 - 4.00 .93 6

Science Career, Pre 713 2.90 .81 1.00 - 4.00 .84 4 Science Career, Post 879 2.92 .84 1.00 - 4.00 .87 4

View of Future, Pre 714 3.71 .49 1.00 - 4.00 .90 4 View of Future, Post 879 3.67 .54 1.00 - 4.00 .93 4

Table 12: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Elementary Student Self-Reports of Positive Attitudes and Beliefs, by Gender and Grade Level



Science Interest, Pre .90 .92 .91 .90 Science Interest, Post .92 .93 .92 .92

Science Career, Pre .83 .85 .81 .85 Science Career, Post .86 .88 .84 .89

View of Future, Pre .90 .91 .90 .91 View of Future, Post .94 .93 .93 .95 Table 13 below summarizes the elementary student self-reported outcome scores for the three scales measuring positive attitudes and beliefs: science interest, science career, and view of future. Overall, mean scores reported by elementary students in Summer Matters programs were


in the good to excellent range (2.9 to 3.7). Scores for view of future were slightly higher than for the other two scales; 3.7 at Time 1 (Pre) and 3.7 at Time 2 (Post). Scores for science interest were 3.0 and 3.0 and the mean scores for science career were 2.9 and 2.9. About 52-55% of elementary school students reported good (34% Pre; 33% Post) or excellent (18% Pre; 22% Post) interest toward science and about 45-48% of students reported fair (40% Pre; 37% Post) or low (8% Pre; 8% Post) interest in science. About 55-56% of elementary school students reported good (32% Pre; 31% Post) or excellent (24% Pre; 25% Post) likelihood of pursuing a career in science, and about 44-45% of elementary school students reported fair (35% Pre; 33% Post) or low (10% Pre; 11% Post) likelihood of pursuing a career in science. About 92-94% of elementary school students reported good (22% Pre; 23% Post) or excellent (72% Pre; 69% Post) likelihood of being successful in the future. At Time 2 (Post), only 1% of elementary students reported low likelihood of future success. Table 13: Positive Attitudes and Beliefs, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



Science Interest, Pre 715 2.95 8.4% 39.9% 33.8% 17.9% Science Interest, Post 881 2.99 8.3% 37.1% 32.5% 22.1%

Science Career, Pre 713 2.90 9.5% 35.3% 31.6% 23.6% Science Career, Post 879 2.92 10.6% 33.0% 31.4% 25.0%

View of Future, Pre 714 3.71 1.0% 5.2% 22.0% 71.8% View of Future, Post 879 3.67 1.7% 5.9% 23.2% 69.2% N= number of students; Mean = average score

Tables 14-16 shows elementary student self-reported outcome scores for the two measures (three scales) of positive attitudes and beliefs by gender, grade level, and ethnicity. Across all groups, mean scores for science interest ranged from 2.8 to 3.2 and mean scores for science career ranged from 2.7 to 3.1. Within each group, students reported slightly higher view of future scores than for the other three measures (3.0 to 3.9). For all elementary boys in the Summer Matters programs, mean scores for view of future were 3.7 at Time 1 (Pre) and 3.6 at Time 2 (Post). The mean scores for science interest were 3.0 and 3.0 and scores for science career were 2.9 and 3.0. For all elementary girls, scores for view of future were 3.8 at Time 1 (Pre) and 3.7 at Time 2 (Post), scores for science interest were 3.0 and 3.0, scores for science career were 2.9 and 2.9. For 3rd-4th grade students, mean scores for view of future were 3.6 at Time 1 (Pre) and 3.0 at Time 2 (Post). The mean scores for science interest were 3.1 and 3.1, and scores for science career were 3.1 and 3.0. For 5th-6th grade students, mean scores for view of future were 3.8 at


Time 1 (Pre) and 3.7 at Time 2 (Post), scores for science interest were 2.8 and 2.8, and scores for science career were 2.7 and 2.8. Across all four ethnicity groups, reported scores were high (2.9 to 3.9). Consistent with the other groups of elementary students (gender, grade level), scores for view of future were slightly higher (3.6 to 3.9) than scores for science interest (3.0 to 3.2) and scores for science career (2.9 to 3.2). Table 14: Positive Attitudes and Beliefs Scores by Gender, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Gender

Boys Science Interest, Pre 341 2.95 7.3% 42.2% 32.3% 18.2% Science Interest, Post 426 3.02 6.6% 37.6% 31.7% 24.2%


View of Future, Pre 340 3.65 1.2% 7.6% 24.7% 66.5% View of Future, Post 424 3.64 1.9% 7.8% 22.9% 67.5%

Girls Science Interest, Pre 374 2.95 9.4% 37.7% 35.3% 17.6% Science Interest, Post 455 2.97 9.9% 36.7% 33.2% 20.2%




Table 15: Positive Attitudes and Beliefs Scores by Grade Level, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



3rd-4th Grade Science Interest, Pre 335 3.12 6.9% 28.1% 40.3% 24.8% Science Interest, Post 488 3.13 5.5% 30.1% 36.5% 27.9%


View of Future, Pre 335 3.64 1.2% 7.8% 22.4% 68.7% View of Future, Post 487 3.02 1.8% 6.2% 25.7% 66.3% 5th-6th Grade Science Interest, Pre 380 2.80 9.7% 50.3% 28.2% 11.8% Science Interest, Post 393 2.82 11.7% 45.8% 27.5% 15.0%




Table 16: Positive Attitudes and Beliefs Scores by Ethnicity, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Ethnicity

African; African American Science Interest, Pre 39 2.97 51.3% 38.5% 10.3% Science Interest, Post 42 3.11 4.8% 31.0% 40.5% 23.8%


View of Future, Pre 39 3.77 5.1% 23.1% 71.8% View of Future, Post 42 3.80 2.4% 9.5% 88.1% Asian; Asian American Science Interest, Pre 50 3.22 2.0% 30.0% 42.0% 26.0% Science Interest, Post 66 3.17 4.5% 28.8% 37.9% 28.8%


View of Future, Pre 50 3.76 6.0% 24.0% 70.0% View of Future, Post 66 3.59 12.1% 27.3% 60.6% Caucasian; European American Science Interest, Pre 27 3.07 48.0% 20.0% 32.0% Science Interest, Post 25 3.07 7.4% 25.9% 48.1% 18.5%


View of Future, Pre 27 3.87 16.0% 84.0% View of Future, Post 25 3.81 3.7% 7.4% 88.9% Hispanic; Latino Science Interest, Pre 344 3.02 7.8% 35.2% 35.8% 21.2% Science Interest, Post 537 3.03 6.9% 35.9% 33.9% 23.3%



Positive Behavior The Elementary Student Self-Report includes two measures of positive behavior: social competencies and misconduct. Social Competencies are assessed by items such as “I work well with other kids” and “I can tell other kids what I think, even if they disagree with me.” These skills are rated on a 4-point scale (1 = not at all true, 4 = really true) and an overall social competencies score is computed as the mean of the items. The items of the Misconduct measure are rated on a 4-point scale (1 = never, 4 = more than once a week). Sample items include “I


have done something my parents told me not to” or “I have taken something that belongs to someone else.” An overall misconduct score is computed as the mean of the seven item scores. Low scores of misconduct are indicators of positive outcomes in behavior. Psychometrics for the social competencies and misconduct measures are calculated for all elementary students in the Summer Matters 2013 programs, as well as for subgroups of students (gender and grade level). Similar to the results for the measures of skill development and the measures of attitudes and beliefs, results show that the measures of social competencies and misconduct have good psychometric properties that are consistent across subgroups of students. The psychometrics for the whole sample (all Summer Matters 2013 programs) are also consistent with results reported in other studies. Table 17 summarizes the psychometrics for the whole sample of elementary students. Table 18 summarizes the reliability coefficients (Cronbach’s Alpha) by gender and grade level. Table 17: Psychometrics at Time 1 and Time 2, Elementary Student Self-Reports of Positive Behavior


Social Competencies, Pre 716 3.02 .62 1.00 - 4.00 .73 7 Social Competencies, Post 882 3.04 .63 1.00 - 4.00 .76 7

Misconduct, Pre 717 1.62 .55 1.00 - 4.00 .81 7 Misconduct, Post 887 1.59 .54 1.00 - 4.00 .81 7

Table 18: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Elementary Student Self-Reports of Positive Behavior, by Gender and Grade Level



Social Competencies, Pre .73 .71 .71 .75 Social Competencies, Post .75 .76 .72 .80

Misconduct, Pre .82 .78 .81 .80 Misconduct, Post .78 .83 .82 .80 Table 19 shows the mean scores and categorical distributions for elementary students’ self-reports of two areas of behavior: social competencies and misconduct. Higher scores in social competencies and lower scores in misconduct are indicators of positive outcomes in behavior. For all elementary students in the Summer Matters 2013 programs, mean scores for social competencies were 3.0 at Time 1 (Pre) and 3.0 at Time 2 (Post). The mean scores for misconduct were 1.5 at Time 1 and 1.5 at Time 2. For all elementary students in the Summer Matters 2013 programs, mean scores for social competencies were 3.0 at Time 1 (Pre) and 3.0 at Time 2 (Post), the mean scores for misconduct were 1.6 at Time 1 and Time 2.


The categorical scores show that about 57-60% of the elementary students in the Summer Matters programs reported good (45% Pre; 38% Post) or excellent social competencies (16% Pre; 19% Post). The categorical scores for misconduct show that 96-97% of students reported low (79% Pre; 78% Post) or medium (18% Pre; 19% Post) levels of misconduct. Only 3-4% of students reported high levels of misconduct. Table 19: Positive Behavior, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



Social Competencies, Pre 716 3.02 5.4% 34.2% 44.6% 15.8% Social Competencies, Post 882 3.04 4.8% 38.0% 38.0% 19.3%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0) Summer Matters 2013 (All Programs)

Misconduct, Pre 717 1.62 78.7% 17.7% 3.6% Misconduct, Post 887 1.59 77.9% 18.9% 3.2% N= number of students; Mean = average score

Tables 20-22 show elementary student self-reported outcome scores for the two measures of positive behavior by gender, grade level, and ethnicity. Across all groups, mean scores for social competencies ranged from 2.9 to 3.3 and mean scores for misconduct ranged from 1.4 to 1.7. For all elementary boys in the Summer Matters programs, mean scores for social competencies were 2.9 at Time 1 (Pre) and 3.0 at Time 2 (Post), and mean scores for misconduct were 1.7 and 1.7. For all elementary girls, scores for social competencies were 3.1 at Time 1 (Pre) and Time 2 (Post), and scores for misconduct were 1.5 and 1.6. Across all grade levels, mean scores for the two measures of positive behavior were comparable. Students in 3rd-4th grade and students in 5th-6th grade reported social competencies scores in the range of 3.0 to 3.1. For misconduct, 3rd-4th grade students reported scores of 1.6 and 5th-6th grade students reported scores of 1.7. Across all four ethnicity groups, reported scores for social competencies ranged from 3.0 to 3.3. Consistent with the other groups of elementary students (gender, grade level), scores for misconduct were low, ranging from 1.4 to 1.6.


Table 20: Positive Behavior Scores by Gender, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Gender

Boys Social Competencies, Pre 342 2.92 7.3% 39.2% 41.2% 12.3% Social Competencies, Post 427 2.96 6.1% 41.2% 37.7% 15.0%

Girls Social Competencies, Pre 374 3.11 3.7% 29.7% 47.6% 19.0% Social Competencies, Post 455 3.11 3.5% 34.9% 38.2% 23.3%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0) Scores by Gender

Boys Misconduct, Pre 343 1.71 75.2% 19.2% 5.5% Misconduct, Post 429 1.65 74.4% 22.6% 3.0%

Girls Misconduct, Pre 374 1.53 81.8% 16.3% 1.9% Misconduct, Post 458 1.55 81.2% 15.5% 3.3% N= number of students; Mean = average score

Table 21: Positive Behavior Scores by Grade Level, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



3rd-4th Grade Social Competencies, Pre 335 3.02 5.1% 34.6% 44.2% 16.1% Social Competencies, Post 489 3.07 3.9% 37.2% 39.1% 19.8%

5th-6th Grade Social Competencies, Pre 381 3.03 5.8% 33.9% 44.9% 15.5% Social Competencies, Post 393 3.00 5.9% 38.9% 36.6% 18.6%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0) Scores by Grade Level

3rd-4th Grade Misconduct, Pre 336 1.57 80.7% 15.8% 3.6% Misconduct, Post 494 1.55 78.9% 17.2% 3.8%

5th-6th Grade Misconduct, Pre 381 1.65 76.9% 19.4% 3.7% Misconduct, Post 393 1.65 76.6% 21.1% 2.3% N= number of students; Mean = average score


Table 22: Positive Behavior Scores by Ethnicity, Elementary Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Ethnicity

African; African American Social Competencies, Pre 39 3.23 2.6% 33.3% 30.8% 33.3% Social Competencies, Post 42 3.12 4.8% 31.0% 40.5% 23.8%

Asian; Asian American Social Competencies, Pre 50 2.97 2.0% 46.0% 42.0% 10.0% Social Competencies, Post 66 2.99 6.1% 40.9% 34.8% 18.2% Caucasian; European American Social Competencies, Pre 25 3.34 20.0% 52.0% 28.0% Social Competencies, Post 27 3.31 25.9% 44.4% 29.6% Hispanic; Latino Social Competencies, Pre 345 3.02 6.4% 31.0% 45.5% 17.1% Social Competencies, Post 538 3.05 4.3% 38.1% 39.8% 17.8%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0) Scores by Ethnicity

African; African American Misconduct, Pre 39 1.55 87.2% 12.8% Misconduct, Post 42 1.57 81.0% 14.3% 4.8%

Asian; Asian American Misconduct, Pre 50 1.48 88.0% 12.0% Misconduct, Post 66 1.47 87.9% 10.6% 1.5%

Caucasian; European American Misconduct, Pre 25 1.39 88.0% 12.0% Misconduct, Post 27 1.52 88.9% 7.4% 3.7%

Hispanic; Latino Misconduct, Pre 346 1.56 81.5% 15.6% 2.9% Misconduct, Post 543 1.57 78.5% 18.4% 3.1% N= number of students; Mean = average score


B. MIDDLE/HIGH SCHOOL STUDENT OUTCOMES This section presents the outcome scores and analyses for middle/high school students (Grades 6 to 12). Students in Grade 6 were included in the middle/high school sample if they attended an intermediate school summer program; they were included in the elementary school sample if they attended an elementary school program. Identical to the previous section on elementary student outcomes, middle/high school students’ self-reports of outcomes are organized in three broad areas of skill development, positive attitudes and beliefs, and positive behavior. The middle/high school students in grades 6, 7, 8, and 9 completed self-reports of outcomes in which they rated their own skills, attitudes, and behaviors. In the area of skill development, four measures were included: work habits, reading efficacy, math efficacy, and science efficacy. In the area of attitudes and beliefs, two measures were included: science interest and view of future. In the area of positive behavior change, two measures were included: social competencies and misconduct. From all Summer Matters 2013 programs, a total of 478 middle/high school student self-reports were completed at the beginning of the summer programs (Time 1, Pre) and 454 were completed at the end of the summer programs (Time 2, Post). Skill Development The Middle/High School Student Self-Report included four measures of skill development: Work Habits, Reading Efficacy, Math Efficacy, and Science Efficacy. All four measures were assessed on a 4-point scale (1 = not at all true, 4 = really true). Sample items on the Work Habits scale include: “I work well by myself” and “I finish my work on time.” Items on the Reading Efficacy scale assess a student’s sense of competency in reading (e.g. “I expect to do well in reading”) and subjective task value (e.g. “I am interested in reading.”). The Math Efficacy and Science Efficacy scale includes items such as “I expect to do well in math” and “I am interested in science.” For all four measures of skill development, an overall score is computed as the mean of all the items of each scale. Psychometrics for the skill development measures were calculated for the whole sample of middle/high school students (all participating programs in Summer Matters 2013), as well as for subgroups of students (gender and grade level). Table 23 summarizes the psychometrics for the whole sample of middle/high school students, and Table 24 summarizes the reliability coefficients by gender and grade level. Results conducted with the data from the middle/high school student self-reports are similar to those obtained with the elementary student self-report data. The measures of work habits, reading efficacy, math efficacy, and science efficacy had excellent psychometric properties, consistent across subgroups of students. The psychometrics for the whole sample are also consistent with those obtained in Summer 2011 and Summer 2012, and with results reported in other studies.


Table 23: Psychometrics at Time 1 (Pre) and Time 2 (Post); Middle/High School Student Self-Reports of Skill Development


Work Habits, Pre 478 3.07 .57 1.00 - 4.00 .78 6 Work Habits, Post 454 3.07 .60 1.00 - 4.00 .83 6

Reading Efficacy, Pre 476 2.87 .83 1.00 - 4.00 .85 4 Reading Efficacy, Post 454 2.95 .83 1.00 - 4.00 .88 4

Math Efficacy, Pre 476 3.03 .90 1.00 - 4.00 .89 4 Math Efficacy, Post 454 3.07 .90 1.00 - 4.00 .90 4

Science Efficacy, Pre 476 2.86 .88 1.00 - 4.00 .89 4 Science Efficacy, Post 454 2.90 .90 1.00 - 4.00 .91 4

Table 24: Internal Consistency (Cronbach alpha) at Time 1 (Pre) and Time 2 (Post); Middle/High School Student Self-Reports of Skill Development, by Gender and Grade Level

Gender Grade Level Boys Girls 6th-7th 8th-9th


Work Habits, Pre .77 .77 .78 .82 Work Habits, Post .80 .83 .82 .86

Reading Efficacy, Pre .85 .86 .85 .89 Reading Efficacy, Post .88 .88 .88 .87

Math Efficacy, Pre .89 .90 .89 .93 Math Efficacy, Post .91 .90 .90 .89

Science Efficacy, Pre .91 .88 .89 .92 Science Efficacy, Post .93 .89 .91 .90

Table 25 summarizes the middle/high school student self-reported outcome scores for the four measures of skill development: work habits, reading efficacy, math efficacy, and science efficacy. For all participating middle/high school students in the Summer Matters 2013 programs, the aggregate mean scores for all measures ranged from 2.9 to 3.1, on a scale of 1 to 4. The mean scores for work habits were 3.1 at Time 1 (Pre) and Time 2 (Post), the mean scores for reading efficacy were 2.9 and 3.0, the mean scores for math efficacy were 3.0 and 3.1, and the mean scores for science efficacy were 2.9 and 2.9. At Time 1 (Pre) about 70% of the middle/high school students in the Summer Matters programs reported good (44%) or excellent (26%) work habits; about 26% reported fair scores and 4% reported low scores. For reading efficacy, 34% of students reported good scores and 28% reported excellent scores; 29% reported fair scores, and 9% reported low scores. About 27% of students reported good scores in math efficacy and 34% reported excellent scores; 24% reported fair scores and 15% reported low scores. For science efficacy, about 30% of students reported good scores and 17% reported excellent scores; 33% reported fair scores and 12% reported low scores.


At Time 2 (Post) about 65% of the middle/high school students in the Summer Matters programs reported good (43%) or excellent (21%) work habits; about 31% reported fair scores and 3% reported low scores. For reading efficacy, about 59% of the middle/high school students reported good (32%) or excellent (27%) scores. About 63% of the students reported good (26%) or excellent (37%) scores in math efficacy, and about 53-57% of the students reported good (28%) or excellent (29%) scores in science efficacy. Table 25: Skill Development, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



Work Habits, Pre 478 3.07 3.3% 31.8% 47.7% 17.2% Work Habits, Post 454 3.07 4.2% 31.3% 43.4% 21.1% Reading Efficacy, Pre 476 2.87 14.5% 30.7% 33.0% 21.8% Reading Efficacy, Post 454 2.95 11.2% 30.0% 32.2% 26.7% Math Efficacy, Pre 476 3.03 13.9% 22.7% 28.6% 34.9% Math Efficacy, Post 454 3.07 12.3% 24.0% 26.4% 37.2% Science Efficacy, Pre 476 2.86 15.3% 31.7% 27.5% 25.4% Science Efficacy, Post 454 2.90 12.8% 30.2% 28.0% 29.1% N= number of students; Mean = average score

Tables 26-28 shows middle/high school student self-reported outcome scores for the four measures of skill development by gender, grade level, and ethnicity. Overall, middle/high school students reported slightly lower scores than elementary students. Mean scores across all measures were in the range of 2.0 to 3.2. Scores for middle/high school boys were comparable to scores for middle/high school girls. Scores for boys were 3.0 for work habits (at Time 1 and Time 2), 2.8 to 2.9 for reading efficacy, 3.1 for math efficacy, and 2.9 to 3.0 for science efficacy. Scores for girls were 3.2 for work habits, 2.9 to 3.0 for reading efficacy, 3.0 to 3.1 for math efficacy, and 2.8 for science efficacy. For 6th-7th grade students, mean scores for all skill development measures were 2.8 to 3.1 at Time 1 (Pre) and 2.9 to 3.1 at Time 2 (Post). Scores were comparable for 8th-9th grade students: 3.0 to 3.1 at Time 1 (Pre) and 3.0 to 3.1 at Time 2 (Post). Only 10 students in 10th-12th grade completed surveys, and only at Time 2 (Post). Scores for 10th-12th grade students ranged from 2.0 to 3.0. Ethnicity data were provided for approximately 38% of middle/high school students. Scores are reported for the following groups: African or African American, Asian or Asian American, Caucasian or European American, and Hispanic or Latino. Five middle/high school students were categorized as American Indian or Native American, and two students were categorized as Pacific Islander; the scores for the students from these two groups are not reported. Across all


four ethnicity groups included in the aggregate scores, scores were comparable. Scores for African American students ranged from 2.7 to 3.0, scores for Asian American students ranged from 3.0 to 3.3, scores for Caucasian students ranged from 3.0 to 3.2, and scores for Hispanic/Latino students ranged from 2.6 to 3.1. Table 26: Skill Development Scores by Gender, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Gender

Boys Work Habits, Pre 224 2.96 4.0% 39.3% 44.6% 12.1% Work Habits, Post 208 2.95 4.3% 40.9% 39.4% 15.4%



Science Efficacy, Pre 223 2.89 16.1% 28.7% 27.8% 27.4% Science Efficacy, Post 208 3.00 12.5% 25.5% 26.9% 35.1% Girls Work Habits, Pre 254 3.16 2.8% 25.2% 50.4% 21.7% Work Habits, Post 246 3.17 4.1% 23.2% 46.7% 26.0%





Table 27: Skill Development Scores by Grade Level, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



6th-7th Grade Work Habits, Pre 440 3.06 3.2% 32.3% 47.7% 16.8% Work Habits, Post 396 3.08 3.8% 30.8% 43.7% 21.7%



Science Efficacy, Pre 432 2.84 16.0% 32.2% 26.7% 25.1% Science Efficacy, Post 395 2.89 13.6% 29.5% 27.5% 29.3% 8th-9th Grade Work Habits, Pre 38 3.12 5.3% 26.3% 47.4% 21.1% Work Habits, Post 48 2.98 6.3% 37.5% 39.6% 16.7%



Science Efficacy, Pre 37 3.11 7.9% 26.3% 36.8% 28.9% Science Efficacy, Post 48 3.05 6.3% 27.1% 35.4% 31.3% 10th-12th Grade Work Habits, Pre 0 - - - - - Work Habits, Post 10 3.02 10.0% 20.0% 50.0% 20.0%

Reading Efficacy, Pre 0 - - - - - Reading Efficacy, Post 10 2.98 10.0% 30.0% 40.0% 20.0%

Math Efficacy, Pre 0 - - - - - Math Efficacy, Post 10 2.03 50.0% 30.0% 20.0%

Science Efficacy, Pre 0 - - - - - Science Efficacy, Post 10 2.48 10.0% 70.0% 10.0% 10.0% N= number of students; Mean = average score


Table 28: Skill Development Scores by Ethnicity, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Ethnicity

African; African American Work Habits, Pre 49 3.04 4.1% 32.7% 46.9% 16.3% Work Habits, Post 50 3.00 4.0% 36.0% 42.0% 18.0%



Science Efficacy, Pre 49 2.71 22.4% 28.6% 26.5% 22.4% Science Efficacy, Post 50 2.75 20.0% 28.0% 26.0% 26.0% Asian; Asian American Work Habits, Pre 24 3.13 4.2% 25.0% 50.0% 20.8% Work Habits, Post 19 3.28 26.3% 47.4% 26.3%


Math Efficacy, Pre 24 2.96 8.3% 29.2% 41.7% 20.8% Math Efficacy, Post 19 3.20 26.3% 47.4% 26.3%

Science Efficacy, Pre 24 3.05 12.5% 20.8% 37.5% 29.2% Science Efficacy, Post 19 3.13 5.3% 26.3% 31.6% 36.8% Caucasian; European American Work Habits, Pre 34 3.14 2.9% 29.4% 38.2% 29.4% Work Habits, Post 28 3.15 3.6% 32.1% 28.6% 35.7%



Science Efficacy, Pre 34 3.13 5.9% 32.4% 23.5% 38.2% Science Efficacy, Post 28 3.04 3.6% 39.3% 25.0% 32.1% Hispanic; Latino Work Habits, Pre 95 3.04 6.3% 30.5% 45.3% 17.9% Work Habits, Post 159 3.06 4.4% 34.6% 38.4% 22.6%





Positive Attitudes and Beliefs The Middle/High School Student Self-Report includes two measures of positive attitudes and beliefs: science interest and view of future. Both measures are assessed on a 4-point scale (1 = not at all true, 4 = really true). An overall score is computed as the mean of the items. The science interest measure was developed at the Harvard University Program in Education, Afterschool & Resiliency (PEAR), and includes items such as “I get excited about learning about new discoveries or inventions.” The view of future measure includes two subscales: science career and view of future. The science career subscale includes items such as “I will get a job in a science-related area.” The view of future subscale includes items such as “I will go to college.” Psychometrics for the positive attitudes and beliefs measures were calculated for all middle/high school student participants in the Summer Matters 2013 programs, as well as for subgroups of students (by gender and grade level). Table 29 summarizes the psychometrics for the whole sample of middle/high school students and Table 30 summarizes the reliability coefficients (Cronbach’s Alpha) by gender and grade level. These analyses show that the measures had excellent psychometric properties, consistent across subgroups of students. Table 29: Psychometrics at Time 1 and Time 2, Middle/High School Student Self-Reports of Positive Attitudes and Beliefs


Science Interest, Pre 470 2.68 .65 1.00 - 4.00 .90 6 Science Interest, Post 454 2.71 .70 1.00 - 4.00 .92 6

Science Career, Pre 469 2.53 .85 1.00 - 4.00 .88 4 Science Career, Post 453 2.57 .86 1.00 - 4.00 .90 4

View of Future, Pre 469 3.76 .41 1.00 - 4.00 .91 4 View of Future, Post 453 3.78 .42 1.00 - 4.00 .93 4

Table 30: Internal Consistency (Cronbach’s Alpha) at Time 1 and Time 2, Middle/High School Student Self-Reports of Positive Attitudes and Beliefs, by Gender and Grade Level



Science Interest, Pre .91 .89 .90 .90 Science Interest, Post .93 .92 .92 .93

Science Career, Pre .88 .88 .88 .88 Science Career, Post .89 .90 .90 .90

View of Future, Pre .93 .87 .91 .87 View of Future, Post .92 .94 .94 .91 Table 31 summarizes the middle/high school student self-reported outcome scores for the three scales measuring positive attitudes and beliefs: science interest, science career, and view of future. For all middle/high school students in Summer Matters programs, mean scores for science interest were 2.7 at Time 1 (Pre) and Time 2 (Post), the mean scores for science career


were 2.5 and 2.6, and mean scores for view of future were 3.8 and 3.8. Similar to the reports from elementary students, scores for view of future were higher than for the other two measures: 3.8 at Time 1 and Time 2. The categorical scores for the positive attitudes and beliefs measures show that across all Summer Matters programs, about 34-37% of middle/high school students reported good (26% Pre; 28% Post) or excellent (7% Pre; 10% Post) interest in science, and about 63-66% of students reported fair (52% Pre; 48% Post) or low (15% Pre; 15% Post) interest in science. About 38-43% of students reported good (26% Pre; 30% Post) or excellent (11% Pre; 13% Post) likelihood of pursuing a career in science. Over 95% of students reported a good (20% Pre; 19% Post) or excellent (76% Pre; 78% Post) likelihood of being successful in the future. Only 3-4% of students reported a fair or low likelihood of future success. Table 31: Positive Attitudes and Beliefs, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



Science Interest, Pre 470 2.68 14.7% 51.7% 26.2% 7.4% Science Interest, Post 454 2.71 14.5% 48.2% 27.8% 9.5%



Tables 32-34 shows middle/high school student self-reported outcome scores for the two measures (three scales) of positive attitudes and beliefs by gender, grade level, and ethnicity. Across all groups, mean scores for science interest ranged from 2.1 to 2.9 and mean scores for science career ranged from 1.9 to 2.9. Within each group, students reported higher scores for view of future than for the other three measures (3.7 to 3.9). For all middle/high school boys in the Summer Matters programs, mean scores for view of future were 3.7 at Time 1 (Pre) and 3.8 at Time 2 (Post). The mean scores for science interest were 2.8 and 2.8 and scores for science career were 2.7 and 2.7. For all middle/high school girls, scores for view of future were 3.8 at Time 1 (Pre) and Time 2 (Post), scores for science interest were 2.6 and 2.6, scores for science career were 2.9 and 2.9. Scores were comparable across all grade levels. Mean scores for view of future were high; 3.8 at Time 1 (Pre) and 3.7 to 3.8 at Time 2 (Post). Scores for science interest were 2.7 to 2.8. For 6th-7th and 7th-8th grade students, scores for science career ranged from 2.5 to 2.7. For 10th-12th grade students, the Time 2 (Post) score for science career was 1.9.


Scores were also comparable across ethnicity groups, reported. Consistent with other groups of middle/high school students, scores for view of future were also slightly higher (3.7 to 3.9) than scores for science interest (2.4 to 2.9) and scores for science career (2.4 to 2.9). Table 32: Positive Attitudes and Beliefs Scores by Gender, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Gender

Boys Science Interest, Pre 220 2.78 13.6% 45.9% 30.5% 10.0% Science Interest, Post 208 2.84 11.1% 42.8% 34.6% 11.5%


View of Future, Pre 220 3.72 0.9% 3.6% 22.3% 73.2% View of Future, Post 208 3.76 1.0% 1.4% 22.1% 75.5% Girls Science Interest, Pre 250 2.60 15.6% 56.8% 22.4% 5.2% Science Interest, Post 246 2.60 17.5% 52.8% 22.0% 7.7%


View of Future, Pre 249 3.79 2.8% 18.1% 79.1% View of Future, Post 245 3.79 0.4% 3.3% 15.9% 80.4% N= number of students; Mean = average score


Table 33: Positive Attitudes and Beliefs Scores by Grade Level, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



6th-7th Grade Science Interest, Pre 433 2.68 15.0% 51.0% 26.6% 7.4% Science Interest, Post 396 2.71 14.9% 47.5% 28.5% 9.1%


View of Future, Pre 432 3.76 0.5% 3.0% 19.9% 76.6% View of Future, Post 395 3.78 0.5% 1.8% 19.2% 78.5% 8th-9th Grade Science Interest, Pre 37 2.70 10.8% 59.5% 21.6% 8.1% Science Interest, Post 48 2.83 6.3% 52.1% 27.1% 14.6%


View of Future, Pre 37 3.75 5.4% 21.6% 73.0% View of Future, Post 48 3.74 2.1% 6.3% 14.6% 77.1% 10th-12th Grade Science Interest, Pre 0 - - - - - Science Interest, Post 10 2.13 40.0% 60.0%

Science Career, Pre 0 - - - - - Science Career, Post 10 1.93 20.0% 80.0%

View of Future, Pre 0 - - - - - View of Future, Post 10 3.69 10.0% 20.0% 70.0% N= number of students; Mean = average score


Table 34: Positive Attitudes and Beliefs Scores by Ethnicity, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Ethnicity

African; African American Science Interest, Pre 48 2.48 25.0% 47.9% 20.8% 6.3% Science Interest, Post 50 2.43 26.0% 46.0% 22.0% 6.0%


View of Future, Pre 48 3.80 2.1% 2.1% 8.3% 87.5% View of Future, Post 50 3.76 2.0% 22.0% 76.0% Asian; Asian American Science Interest, Pre 24 2.77 8.3% 58.3% 29.2% 4.2% Science Interest, Post 19 2.85 10.5% 47.4% 36.8% 5.3%


View of Future, Pre 24 3.85 12.5% 87.5% View of Future, Post 19 3.85 10.5% 89.5% Caucasian; European American Science Interest, Pre 34 2.65 11.8% 52.9% 26.5% 8.8% Science Interest, Post 28 2.63 17.9% 53.6% 25.0% 3.6%


View of Future, Pre 34 3.73 2.9% 23.5% 73.5% View of Future, Post 28 3.84 21.4% 78.6% Hispanic; Latino Science Interest, Pre 94 2.55 18.1% 58.5% 16.0% 7.4% Science Interest, Post 159 2.87 9.4% 44.7% 30.2% 15.7%


View of Future, Pre 94 3.78 4.3% 19.1% 76.6% View of Future, Post 159 3.77 2.5% 21.5% 75.9% N= number of students; Mean = average score

Positive Behavior The Middle/High School Student Self-Report includes the same two measures of behavior change as are included in the Elementary School Student Self-Report: social competencies and misconduct. Social Competencies are assessed by items such as “I work well with other kids” and “I can tell other kids what I think, even if they disagree with me.” These skills are rated on a 4-point scale (1 = not at all true, 4 = really true) and an overall social competencies score is computed as the mean of the items. The items of the Misconduct measure are rated on a 4-point


scale (1 = never, 4 = more than once a week). Sample items include “I have done something my parents told me not to” or “I have taken something that belongs to someone else.” An overall misconduct score is computed as the mean of the seven item scores. Low scores of misconduct are viewed as a positive indicator. Psychometrics for the social competencies and misconduct measures are calculated for all middle/high school students who participated in the Summer 2013 implementation, as well as for subgroups of students (by gender and grade level). Similar to the results for the measures of skill development and attitudes and beliefs, results show that the measures of social competencies and misconduct had excellent psychometric properties, consistent across subgroups of students. The psychometrics for the whole sample were also consistent with those obtained in Summer 2011, Summer 2012, and with results reported in other studies. Table 35 summarizes the psychometrics for the whole sample of middle/high school students in the Summer Matters 2013 programs. Table 36 summarizes the reliability coefficients by gender and grade level. Table 35: Psychometrics at Time 1 (Pre) and Time 2 (Post), Middle/High School Student Self-Reports of Positive Behavior


Social Competencies, Pre 474 3.14 .59 1.00 - 4.00 .79 7 Social Competencies, Post 454 3.16 .57 1.00 - 4.00 .79 7

Misconduct, Pre 476 1.60 .48 1.00 - 4.00 .81 7 Misconduct, Post 454 1.58 .44 1.00 - 3.14 .78 7

Table 36: Internal Consistency (Cronbach alpha) at Time 1 (Pre) and Time 2 (Post), Middle/High School Student Self-Reports of Positive Behavior, by Gender and Grade Level



Social Competencies, Pre .81 .77 .77 .90 Social Competencies, Post .78 .80 .79 .84

Misconduct, Pre .83 .78 .80 .86 Misconduct, Post .77 .79 .78 .76

Table 37 shows the mean scores and categorical distributions for middle/high school students’ self-reports reports of two areas of behavior: social competencies and misconduct. Higher scores in social competencies and lower scores in misconduct are indicators of positive outcomes in behavior. For all middle/high school students in the Summer Matters 2013 programs, mean scores for social competencies were 3.1 at Time 1 (Pre) and 3.2 at Time 2 (Post). The mean scores for misconduct were 1.6 at Time 1 and Time 2.


Categorical scores show that about 66-68% of the middle/high school students reported good (44% Pre; 45% Post) or excellent social competencies (22% Pre; 23% Post). The categorical scores for misconduct showed that 98-99% of students reported low (80% Pre; 82% Post) or medium (19% Pre; 17% Post) levels of misconduct. Only 1-2% of students reported high levels of misconduct. Table 37: Social Competencies, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



Social Competencies, Pre 474 3.14 2.7% 31.4% 43.7% 22.2% Social Competencies, Post 454 3.16 2.6% 29.1% 45.2% 23.1%

N

Mean

% Low

(0.0-0.9)

% Medium (1.0-1.9)

% High

(2.0-3.0) Summer Matters 2013 (All Programs)

Misconduct, Pre 476 1.60 79.6% 18.7% 1.7% Misconduct, Post 454 1.58 82.2% 17.0% 0.9% N= number of students; Mean = average score

Tables 38-40 shows middle/high school student self-reported outcome scores for the two measures of positive behavior by gender, grade level, and ethnicity. Across all groups, mean scores for social competencies ranged from 3.1 to 3.2 and mean scores for misconduct ranged from 1.5 to 1.7. Scores for social competencies and misconduct were comparable for boys and girls. Mean scores for social competencies for both boys and girls ranged from 3.1 to 3.2 and mean scores for misconduct were all 1.6. Mean scores for the two measures of positive behavior were also comparable across grade levels. Social competencies scores for students in 6th-7th grade and students in 8th-9th grade were 3.1 at Time 1 (Pre) and 3.2 at Time 2 (Post). Social competencies scores were also 3.2 for students in 10th-12th grade. For all grade levels, misconduct scores were low, ranging from 1.6 to 1.7. Across all four ethnicity groups, reported scores for social competencies ranged from 3.0 to 3.3. Consistent with the other groups of middle/high school students (gender, grade level), scores for misconduct were low, ranging from 1.5 to 1.7.


Table 38: Positive Behavior Scores by Gender, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Gender

Boys Social Competencies, Pre 222 3.11 2.7% 33.8% 42.3% 21.2% Social Competencies, Post 208 3.17 1.9% 30.3% 46.2% 21.6%

Girls Social Competencies, Pre 252 3.17 2.8% 29.4% 44.8% 23.0% Social Competencies, Post 246 3.15 3.3% 28.0% 44.3% 24.4%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0) Scores by Gender

Boys Misconduct, Pre 223 1.64 76.7% 21.1% 2.2% Misconduct, Post 208 1.60 81.3% 17.8% 1.0%

Girls Misconduct, Pre 253 1.56 82.2% 16.6% 1.2% Misconduct, Post 246 1.56 82.9% 16.3% 0.8% N= number of students; Mean = average score


Table 39: Positive Behavior Scores by Grade Level, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)



6th-7th Grade Social Competencies, Pre 437 3.14 2.5% 31.1% 45.3% 21.1% Social Competencies, Post 396 3.15 2.5% 28.8% 46.2% 22.5%

8th=9th Grade Social Competencies, Pre 37 3.13 5.4% 35.1% 24.3% 35.1% Social Competencies, Post 48 3.18 2.1% 33.3% 39.6% 25.0%

10th-12th Grade Social Competencies, Pre 0 - - - - - Social Competencies, Post 10 3.23 10.0% 20.0% 30.0% 40.0%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0) Scores by Grade Level

6th-7th Grade Misconduct, Pre 438 1.58 81.1% 17.6% 1.4% Misconduct, Post 396 1.57 82.3% 17.2% 0.5%

8th-9th Grade Misconduct, Pre 38 1.74 63.2% 31.6% 5.3% Misconduct, Post 48 1.64 79.2% 16.7% 4.2%

10th-12th Grade Misconduct, Pre 0 - - - - Misconduct, Post 10 1.57 90.0% 10.0% N= number of students; Mean = average score


Table 40: Positive Behavior Scores by Ethnicity, Middle/High School Student Self-Reports

N

Mean

% Low

(1.0-1.9)

% Fair

(2.0-2.9)

% Good

(3.0-3.59)


Scores by Ethnicity

African; African American Social Competencies, Pre 49 3.28 2.1% 18.8% 50.0% 29.2% Social Competencies, Post 50 3.25 2.0% 28.0% 36.0% 34.0%

Asian; Asian American Social Competencies, Pre 24 3.04 33.3% 58.3% 8.3% Social Competencies, Post 19 3.20 31.6% 57.9% 10.5%

Caucasian; European American Social Competencies, Pre 34 3.12 5.9% 23.5% 35.3% 35.3% Social Competencies, Post 28 3.15 3.6% 21.4% 53.6% 21.4%

Hispanic; Latino Social Competencies, Pre 95 3.18 2.1% 32.6% 43.2% 22.1% Social Competencies, Post 159 3.13 1.9% 32.7% 45.3% 20.1%

N

Mean

% Low

(1.0-1.9)

% Medium (2.0-2.9)

% High

(3.0-4.0) Scores by Ethnicity

African; African American Misconduct, Pre 49 1.73 71.4% 24.5% 4.1% Misconduct, Post 50 1.67 76.0% 22.0% 2.0%

Asian; Asian American Misconduct, Pre 24 1.57 87.5% 12.5% Misconduct, Post 19 1.48 100.0%

Caucasian; European American Misconduct, Pre 34 1.62 76.5% 20.6% 2.9% Misconduct, Post 28 1.73 71.4% 21.4% 7.1%

Hispanic; Latino Misconduct, Pre 95 1.60 78.9% 17.9% 3.2% Misconduct, Post 159 1.48 87.4% 12.6% N= number of students; Mean = average score


Chapter Summary

The student outcome scores show that the majority of elementary and middle/high school students in the Summer Matters programs reported good or excellent scores in all of the measures of skill development, attitudes and beliefs, and positive behavior change: work habits, reading efficacy, math efficacy, science efficacy, science interest, science career, view of future, social competencies, and misconduct (lower scores of misconduct are a positive indicator). All participating programs were provided with the summary student outcome scores for all sites, as well as the scores for their overall program and each of the participating sites in their program. Programs can use these results to help guide program improvement. The next chapter summarizes the findings pertaining to associations between student self-reports of program experiences (positive relationships with staff, interest in program activities, positive peer relationships) and student outcomes (skill development and positive behavior change).


CHAPTER 4

ASSOCIATIONS BETWEEN QUALITY OF PROGRAM EXPERIENCES AND STUDENT OUTCOMES

The Program Experiences scale, administered as part of the post-participation survey, is an important part of the Online Toolbox that provides an assessment of the quality of students’ experiences at their afterschool program. In this chapter, association between these reported experiences and students’ outcome scores are reported. For these analyses, the quality of program experiences was categorized as poor/fair (ratings of 1.0 to 2.9), good (ratings of 3.0 to 3.59), and excellent (ratings of 3.6 to 4.0). Analyses were conducted separately for elementary and middle/high school students. Within each section, the first analysis of variance (ANOVA) tested associations between quality of experiences with program staff (categorized as Low/Fair, Good, or Excellent) in relation to the student skills and behaviors. A second ANOVA examined associations between experiences with program activities in relation to student skills and behaviors, and a third ANOVA examined associations between experiences with peers in relation to student skills and behaviors.

A. ELEMENTARY STUDENTS

For the participating Summer Matters programs, all of the ANOVAs show significant associations between elementary students’ program experiences and their self-reports of student outcomes (p < .01). Table 41, below, presents the mean student outcome scores at each of the three levels of quality of experiences with program staff. For each successive level of quality of Positive Relationships with Staff (Low/Fair, Good, Excellent), the mean student outcome scores for misconduct decreased; the scores for all other outcomes (work habits, reading efficacy, math efficacy, science efficacy, science interest, science career, view of future, and social competencies) increased (p < .01). Tables 42-43 summarize student outcome scores at each level of quality of Interest in Program Activities and each level of quality of Positive Peer Relationships. All tested associations were significant (p < .01). For each successive level of quality of program activities and for each successive level of quality of relationships with peers, the mean outcome scores for misconduct decreased and all other outcome scores increased.


Table 41: Elementary Student Reports of Quality of Program Experiences—Positive Relationships with Staff—Are Linked to Self-Reported Student Outcomes

Positive Relationships with Staff

Low/Fair Quality Good Quality Excellent Quality Mean Mean Mean All Programs, Summer 2013 (N = 864) Work Habits** 2.84 3.22 3.36 Reading Efficacy** 2.71 3.22 3.41 Math Efficacy** 3.10 3.47 3.49 Science Efficacy** 2.75 3.12 3.31 Science Interest** 2.68 3.06 3.22 Science Career** 2.61 3.00 3.10 View of Future** 3.49 3.69 3.81 Social Competencies** 2.76 3.11 3.23 Misconduct** 1.81 1.60 1.39 **p<.01 *p<.05

Table 42: Elementary Student Reports of Quality of Program Experiences—Interest in Program Activities—Are Linked to Self-Reported Student Outcomes

Program Activities



Table 43: Elementary Student Reports of Quality of Program Experiences—Positive Peer Relationships—Are Linked to Self-Reported Student Outcomes

Positive Peer Relationships



B. MIDDLE/HIGH SCHOOL STUDENTS

For the participating Summer Matters programs, ANOVAs show significant associations between elementary students’ program experiences and their self-reports of student outcomes (p < .01). Table 44, below, presents the mean student outcome scores at each of the three levels of quality of experiences with program staff. For each successive level of quality of relationships with staff (Low/Fair, Good, Excellent), the mean student outcome scores for misconduct decreased; the scores for all other outcomes (work habits, reading efficacy, math efficacy, science efficacy, science interest, science career, view of future, and social competencies) increased (p < .01). Tables 45-46 summarize student outcome scores at each level of quality of Interest in Program Activities and each level of quality of Positive Peer Relationships. All tested associations were significant (p < .01). For each successive level of quality of program activities and for each successive level of quality of relationships with peers, the mean outcome scores for misconduct decreased and all other outcome scores increased. Table 44: Middle/High School Student Reports of Quality of Program Experiences—Positive Relationships with Staff—Are Linked to Self-Reported Student Outcomes

Positive Relationships with Staff



Table 45: Middle/High School Student Reports of Quality of Program Experiences—Interest in Program Activities—Are Linked to Self-Reported Student Outcomes

Interest in Program Activities


Table 46: Middle/High School Student Reports of Quality of Program Experiences—Positive Peer Relationships—Are Linked to Self-Reported Student Outcomes

Positive Peer Relationships


Chapter Summary

The analyses presented in this chapter show that students’ experiences in their summer program are consistently linked to their skill development and positive behavior change. Students who reported higher quality program experiences with program staff and peers had higher self-reported outcome scores in work habits, reading efficacy, math efficacy, science efficacy, science interest, science career, view of future, and social competencies, and lower self-reported scores in misconduct.


CHAPTER 5 2013 IMPLEMENTATION EXPERIENCE AND PROGRAM FEEDBACK

In fall 2013, program directors and staff were interviewed by phone following the summer survey administration to collect feedback on the survey implementation experience. After receiving their program reports, program directors were contacted again to gather feedback on how useful they found the reports on student summer learning outcomes. Online Toolbox Implementation

In general, program staff and program administrators interviewed stated that the online survey administration process went well, and that students were able to complete surveys without any major issues. One program director noted, “The only issue was making sure students were taking it seriously and understood the questions.” At another program the staff assigned to overseeing the survey administration at several sites noted: “Once I got the instructions it was easy to read to the kids and they got comfortable doing the surveys.” Another program staff suggested that it would be desirable to reduce the length of the survey due to students’ limited “attention span.” Notably, programs had the option this year to select scales they wanted to use but none took advantage of that flexibility, in some cases stating that they did not have time to think through which ones would be most valuable to them. Specific feedback on the measures was also provided. One program director pointed out that the students preferred the Online Toolbox measures over the programs’ internal assessments [which measured content knowledge], she added “They love to give their opinions,” and appreciated that the Online Toolbox measures got at students’ sense of efficacy “[...] it’s hard to measure attitudes,” she stated. As in past years, the time constraints related to the short duration of summer programs was an issue. For some programs, such as LA’s BEST, pre and post surveys were only administered at sites that had a longer program period of 6-8 weeks. At sites with a shorter duration, only post surveys were administered. At another program, the site administrator noted that because of the short time span between pre and post, it was hard to get the kids to do the post surveys when they felt they had just answered the same questions a few weeks prior.

In addition, the short duration of the summer program made it difficult for staff to complete reports on their students. Only two programs completed staff surveys. A program director for a site that opted to administer staff surveys noted that it was more of a challenge to get staff “on board” to complete student reports as it took them 5 to 10 minutes per student and it took time out of their normal planning period. She liked the idea of doing post surveys only for staff “…since it's such a short program and there is little time.” She further expressed that she was “…excited about getting the data back. Right now it’s just hypothetical and staff don’t see the usefulness. Once we get the data we will be able to show them why they did it and that would be good.”


Finally, three programs mentioned the challenge of access to computer labs for the online administration of surveys at their summer learning program sites. Program Reports Program directors also provided feedback on the usefulness the program reports that they received. They found the information in the reports useful as a program improvement tool and as a resource in working toward building support from stakeholders and toward achieving their program sustainability goals. The director of a program that has participated in previous years stated how this year’s report was particularly helpful in her staff development and program improvement efforts: “[The report] …provides more information and is more helpful to us, more relevant to our work. The STEM related measures reflect the current focus of our program. The information leads to discussion in doing debriefs with staff in discussing what we need to do and what we do well." One Program Director stated, “I find the format very easy to read and understand. The breakdown of ethnicities is particularly helpful.” She also suggested that the report should include a page that provides a summary of highlights of the outcomes reported. The program director elaborated:

Although the individual communities can create their own highlight page with their program goals in mind, I still feel it would be helpful to have some highlights pulled out for them. Secondly, is it feasible to include open-ended responses for some questions? In particular, it would be helpful for us to know what the vast majority of students are looking for when rating activities. I am surprised they rated “fair”. Perhaps there could be optional questions and the data can be sent to the communities to view and categorize, as I understand it would be incredibly difficult for your staff to review and categorize that type of data.

Another program director that has participated in previous years viewed the Online Toolbox Outcome Measures as “…more applicable and recognized as valuable to school administrators” given their district’s current focus on “common core with habits of mind”. She continued, “[the Online Toolbox] …is easy to administer, kids don’t mind it, it gives us good data, it helps move the [Summer Matters] campaign forward, it’s a win win all around.” In summary, program staff and administration feedback on experience in using the Summer Learning Toolbox was positive, noting the ongoing issues of limitations of time and computer access particular to the summer program context. Program directors were satisfied with the reports received and indicated that they planned to use them toward there program development efforts.


CHAPTER 6 CONCLUSIONS

The Year Three (Summer 2013) implementation of the Online Toolbox—developed by UC Irvine, with previous support from Packard Foundation to assess student summer learning outcomes—is the third iteration of the Summer Learning Outcome Measures Project. The Summer 2013 administration of the Online Toolbox was conducted at 31 sites from six programs participating in the Packard Summer Matters Initiative. All of the participating programs were able to successfully complete at least one administration of student surveys; some programs were able to complete both the pre- and post student reports. In all, over 2,500 student and program staff reports (n = 2,618) were completed. This included 1,267 pre-participation surveys (718 from elementary students, 478 from middle/high school students, 71 from program staff), and 1,351 post-participation surveys (888 elementary students, 454 middle/high school students, 9 program staff). 848 of the students in the Summer Matters programs completed both pre- and post-participation surveys (50% of the 1,690 student participants). The measures of Skill Development and Positive Behavior, which were piloted at summer learning programs in Summer 2011 and Summer 2012, demonstrated excellent psychometric properties that were consistent across subgroups of students and with results reported in other studies. Additional measures in the area of Positive Attitudes and Beliefs about STEM content were included in the Summer 2013 implementation of the Online Toolbox. These measures were adapted from measures used in the Online Toolbox for afterschool programs. They also demonstrated excellent psychometric qualities, consistent with results obtained from piloting at afterschool programs across California during the 2011-2012 and 2012-2013 school years. Reports on student program experiences show that the majority of both elementary and middle/high school students in the Packard Summer Matters programs had good or excellent quality experiences. Students reported positive relationships with program staff (70% elementary and 71% middle/high), satisfaction with program activities (68% elementary and 74% middle/high), and good to excellent relationships with peers (68% elementary and 72% middle/high). In the area of skill development, categorical scores show that about 62-70% of the elementary students and 65-70% of middle/high school students in the Summer Matters programs reported good or excellent work habits. The distributions for reading, math, and science efficacy were also high. For elementary students, about 60-68% of the reading efficacy scores, about 74-78% of the math efficacy scores, and about 58-64% of the science efficacy scores were in the good or excellent range. For middle/high school students, about 59-62% of the reading efficacy scores, about 61-63% of the math efficacy scores, and about 47-57% of the science efficacy scores were in the good or excellent range. In the area of positive attitudes and beliefs, about 52-55% of elementary students and 34-37% of middle/high school students in the Summer Matters programs reported good or excellent levels


of interest in science. Distributions for likelihood of pursuing a career in science were comparable—55-56% of elementary school students and 38-43% of middle/high school students reported a good or excellent likelihood of pursuing a career in science. Scores for view of future were high. 92-94% of elementary students and over 95% of middle/high school students reported a good or excellent likelihood of being successful in the future. In the area of positive behavior, about 57-60% of elementary students and 66-68% of middle/high school students in the Summer Matters programs reported good or excellent social competencies. Only 3-4% of elementary students and 1-2% of middle/high school students reported high levels of misconduct; almost all elementary and middle/high school students (96-99%) reported low or medium levels of misconduct. For both elementary and middle/high school students, significant associations were found between program experiences and outcomes (p < .01). Higher quality of relationships with staff, greater interest in program activities, and more positive relationships with peers were associated with higher scores in work habits, reading efficacy, math efficacy, science efficacy, science interest, science career, view of future, and social competencies, and with lower scores in misconduct. These associations were consistent with the results obtained in all previous implementations of the Online Toolbox. All participating programs were provided with the summary scores for all sites, as well as the scores for their overall program and each of the participating sites in their program. These scores can be used to compare the scores of their individual programs and sites to the scores of all participating programs in the Summer 2013 implementation. Program leads can use these results to discuss program strengths with site staff, as well as to help guide program improvement. Program leads and staff provided feedback, through interviews conducted in fall 2013, on their experiences using the Summer Learning Toolbox and on the program reports they received. For the most part, the experience of participating programs was positive. Program staff and directors found the online toolbox easy to administer and reported few issues. The main issues identified were (1) limited time to implement surveys given the short duration of the summer program, and (2) limited access to computers to administer the online surveys, particularly at school sites during the summer months when computer labs are closed. Summer Learning Programs using the Online Toolbox should have the option to do post only surveys, particularly those that are short in duration and find it challenging to administer both pre and post surveys. Program directors were satisfied with the reports they received. They found the outcome measures relevant to their program goals and useful to their program management and development needs. Program leads were able to articulate ways that they could use the data as part of their program improvement activities, to inform stakeholders of their outcomes and build program sustainability.


REFERENCES

Vandell, D. L., O’Cadiz, M.P., Hall, V., Karsh A. (2013a). Field Test of Outcome Measures for Summer Learning Programs. Report to the David and Lucile Packard Foundation. Vandell, D. L., Pierce, K. M., Hall, V., O’Cadiz, M. P., Karsh, A., & Westover, T. (2013b). Youth outcome measures online toolbox: 2013. University of California, Irvine. Vandell, D. L., Hall, V., O’Cadiz, M.P., Karsh, A. (2012a). California Afterschool Outcome Measures Project: Field Test of the Online Toolbox. Report to the California Department of Education and to the David and Lucile Packard Foundation and the California Department of Education. Vandell, D. L., O’Cadiz, M.P., Hall, V., Karsh A. (2012b). Piloting Outcome Measures for Summer Learning Initiative Programs. Report to the David and Lucile Packard Foundation. Vandell, D. L., Hall, V., O’Cadiz, M.P, Karsh, A (2012c), Power of Discovery: STEM2 Year 1 (2011-12): Report of Findings. Report to the UC Davis California Afterschool Network. Vandell, D. L., O’Cadiz, M.P., Hall, V. (2012d), Tiger Woods Learning Center Evaluation Study—Phase 3: Assessing Student Outcomes In Sporadic And Regular Program Participants And Understanding Sources Of Attrition. Report to the Tiger Woods Foundation.

school of education university of california, irvine · table 27: skill development scores by grade...

Documents