a study of the woodcraft rangers’ · woodcraft rangers: assessment of quality executive summary...

Assessment of Program Quality and Youth Outcomes:

A Study of the Woodcraft Rangers’ Nvision After-School Program

Prepared by

For Woodcraft Rangers

Page i

ACKNOWLEDGEMENTS The authors would like to acknowledge and thank a number of individuals for contributing their resources, time and input to support the development of this report. To begin, we thank Woodcraft Rangers for making this assessment of quality in their after-school program sites possible. Special thanks go to Chief Executive Officer Cathie Mostovoy, as well as Program Director Pablo Garcia, and Director of Education Liz Zarate for making available all of the necessary resources for this study. They collaborated with us throughout the process and this report would not be possible without their assistance. We would also like to thank Monica Kaiser for her participation and input into the development of the Site coordinator survey. Special thanks are also extended to the Woodcraft Rangers Site Coordinators who provided the site-level information that was the foundation of this study, Woodcraft Rangers participants who provided much of the youth outcome information, and the Los Angeles Unified School District and Garvey School District who also contributed youth outcome information. The wealth of information they all provided was invaluable to the development of this report.

Page ii

TABLE OF CONTENTS Acknowledgements .......................................................................................................................... i

Executive Summary .......................................................................................................................... v

I. Introduction ................................................................................................................................ 1

A. Research Context ................................................................................................................... 2

B. Woodcraft’s Nvision Model ................................................................................................... 3

C. Study Approach ...................................................................................................................... 4

II. Methods ..................................................................................................................................... 5

A. Data Sources .......................................................................................................................... 5

B. Measuring Quality – Two Approaches ................................................................................... 9

C. Analyses of Quality with Outcomes ..................................................................................... 10

III. Quality Factors ........................................................................................................................ 11

A. Item Selection ...................................................................................................................... 11

B. Factor Analysis Results ......................................................................................................... 11

C. Reliability of Factors ............................................................................................................. 14

D. Average Quality Factor Scores ............................................................................................. 14

IV. Quality Benchmarks ................................................................................................................ 15

A. Creating the Quality Benchmark Scores .............................................................................. 15

B. Summary of Sites’ Quality Benchmark Scores ..................................................................... 17

C. Overall Quality Score ........................................................................................................... 19

V. Relationship of Quality Indicators with Youth Outcomes ....................................................... 20

A. Introduction to Quality-Outcome Results ........................................................................... 20

B. Quality-Outcome Findings for Elementary Schools ............................................................. 21

C. Quality-Outcome Findings for Middle School Sites ............................................................. 27

Page iii

VI. Conclusions ............................................................................................................................. 35

VII. Recommendations .................................................................................................................. 37

Appendix A – Youth Development Outcomes Mapped to Youth Survey Items ........................... 38

Appendix B – School Community Sociodemographic Correlation Analyses ................................ 40

Appendix C – Factor Analytic Approach Detail ............................................................................. 42

Appendix D – Quality Benchmark Categories and Scores, Mapped to Site Coordinator Survey

Items and High-Quality Response Expectations ........................................................................... 46

Appendix E – Multiple Regression Models for Quality Factors, Elementary Schools .................. 54

Appendix F – Multiple Regression Models for Quality Factors, Middle Schools .......................... 55

Appendix G – Multiple Regression Models for Quality Benchmark Categories, Elementary

Schools .......................................................................................................................................... 56

Appendix H - Multiple Regression Models for Quality Benchmark Categories, Middle Schools . 58

Page iv

TABLES Table 1. List of Youth Development Outcomes ............................................................................. 6

Table 2. Number of Matched Baseline and End-Year Surveys by Survey Type ............................. 7

Table 3. List of Youth Academic Outcomes ................................................................................... 7

Table 4. Number of Matched School Academic Records by Data and School Type ...................... 8

Table 5. School Sociodemographics by School Type ..................................................................... 9

Table 6. Quality Factors and Site Coordinator Surveys Items ..................................................... 12

Table 7. Alpha Coefficients of Quality Factors ............................................................................. 14

Table 8. Average Scores on Quality Factors by School Type ....................................................... 14

Table 9. Quality Benchmark Categories from Site Coordinator Survey ....................................... 16

Table 10. Quality Benchmark Scores for Elementary School Sites .............................................. 18

Table 11. Quality Benchmark Scores for Middle School Sites ..................................................... 18

Table 12. Overall Benchmark Quality Score by Level of Quality.................................................. 19

Table 13. Elementary School Site Findings – Significant Relationships Between Quality and Outcomes ...................................................................................................................................... 22

Table 14. Desired Student Outcomes Associated with Quality Benchmarks for Elementary School Programs ........................................................................................................................... 25

Table 15. Univariate Analyes of Variance Models – Average Youth Outcome Scores By Overall Quality Rating, Elementary Schools .............................................................................................. 26

Table 16. Middle School Site Findings – Significant Relationships Between Quality and Outcomes ...................................................................................................................................... 28

Table 17. Desired Student Outcomes Associated with Quality Benchmarks for Middle School Programs ....................................................................................................................................... 33

Table 18. Univariate Analyes of Variance Models – Average Youth Outcome Scores by Overall Quality Rating, Middle Schools ..................................................................................................... 34

Woodcraft Rangers: Assessment of Quality Executive Summary

Page v

EXECUTIVE SUMMARY

Formally established in 1922 in Los Angeles, Woodcraft Rangers (Woodcraft) is a youth development non-profit agency with a mission to guide young people as they explore pathways to purposeful lives. Woodcraft developed its after-school program, known as Nvision, to serve elementary through high school youth in impacted and high-need neighborhoods throughout Los Angeles County. The program is provided at more than 60 public school sites five days a week from 3:00 pm to 6:00 pm.

The field of after-school research has shown that regular attendance in high quality after-school programs is associated with positive developmental outcomes for school-age youth. Yet, few studies have directly tested the relationships between after-school program quality and student outcomes. In 2009, the Woodcraft administration directed its evaluation focus toward exploring quality as it relates to targeted youth outcomes. Woodcraft contracted with EVALCORP, a research and evaluation consulting firm, to conduct a secondary analysis study using quality and outcome data from the 2008/09 program year. It focuses on the Nvision elementary and middle school programs, which includes 43 elementary school sites and 14 middle school sites located in the Los Angeles Unified and Garvey school districts.

The study’s primary purposes are to:

Explore the quality of implementation of the Nvision after-school program model;

Examine whether and how quality is associated with student outcomes; and,

Inform further development of a way to monitor quality and develop improvement strategies.

Study Methods The first part of the study involved using site coordinator survey data from a 2008/09 in-depth implementation study to identify key components of program implementation. Two different methods, a factor analysis approach and a benchmark approach, were used to identify the key components of the Nvision model. The factor analysis approach used statistical methods to identify quality indicators. The benchmark approach used agency-identified expectations of quality as a benchmark to which site assessments were compared. Both approaches were used in order to take full advantage of the available data and to explore which approach would be most useful for Woodcraft to use in creating a quality monitoring tool.

The next part of the study was to explore how the factor and benchmark quality indicators were related, or not, to student outcome data from that same 2008/09 year. Data from participants’ school academic records and from program entry and 2008/09 follow-up surveys were used to measure the Nvision outcomes (see table below). The goal was to begin to understand program quality as it relates to youth outcomes, and in what combinations. Both the quality factors and benchmarks were used in correlation and regression analyses to determine if they could predict student outcomes. Two other data sources, after-school program attendance days for participants, and sociodemographic and economic indicators for the communities in which Woodcraft sites are located, were used in the regression analyses to


Page vi

account for the contribution of any known factors beyond quality on student outcomes. Regression analyses were performed separately for elementary and middle schools due to their somewhat different nature and focus of the Nvision model.

Woodcraft Youth Development and Academic Outcomes

More positive attitude about after-school experience

Improved academic performance in English/Language Arts

Higher involvement in making choices Improved academic performance in Math

Higher use of academic skills in activities Improved school attendance

Higher involvement in fitness Improved pro-social skills

Improved homework and school skills Improved self-confidence

Improved school attitude Decreased problem behaviors

Improved leadership skills (MS only)

Quality Factors The factor analysis of the site coordinator survey identified five areas that may be considered when conceptualizing and understanding site quality (see table below for factors and items).

Quality Factors with Related Site Coordinator Survey Items

Core Elements

To what extent do you have a collaborative relationship with school administrators? Generally, how often do you follow the site schedule? To what extent are students involved in decisions that impact the program design?

Cycle Plans

To what extent do you implement cycle plans? How useful are your cycle plans during the implementation of club activities at your site? To what extent are the objectives of your cycle plans met?

Value of Ad Hoc Assistance

How valuable is the contribution for each staff that help to deliver activities at your site: WR Parent Volunteers? Youth Volunteers?

WR Traveling Specialists?

Connections

On average, how many planned activities do students attend or participate in that promote direct involvement with the community?

In general, how involved are parents in your program? How would you describe the access you have to school facilities that you need to implement your

programs’ activities?

Educational Supports


Page vii

How valuable is the contribution for each staff that help to deliver activities at your site: WR Teacher Liaisons?

How valuable is the contribution for each staff that help to deliver activities at your site: WR School Personnel?

To what extent are you using academic concepts in club activities?

Internal consistency for each of the quality factors was examined using Cronbach’s alpha to determine whether the items can be used to create scales to explore program implementation. The alpha coefficients for the factors ranged from .214 – .682. For exploratory research, a minimum of .600 is required to establish the reliability of a scale. Only the first three factors met this criterion: Core Elements, Cycle Plans and Ad Hoc Assistance. The other two areas, Connections and Educational Supports, do not have strong enough reliability to be considered stable factors and so were not included in the subsequent quality-outcome analyses.

Quality Benchmarks The Woodcraft Rangers executive and management staff completed the site coordinator survey as a team to identify how they would expect a high quality site to respond. These ratings serve as the benchmark or minimum criteria for a “high quality” site response. For each item, if the site’s response met or exceeded the management team’s benchmark, a point for that item was assigned. Benchmark expectations were the same for both elementary school (ES) and middle school (MS) sites in most cases, with a few exceptions of higher expectations for middle school sites (e.g., more frequent leadership activities).

The survey items were then organized into 13 categories of implementation, reflecting the major areas of program operations and values. Once a site’s survey had its benchmark points assigned to each item, the points were totaled within each category and divided by the benchmark total for that category. For example, the benchmark score for the category Cycle Plans is five. If a site met the high quality criteria for each item except one in that category, their score would be 80% (i.e., 4 divided by 5). These scores can be considered reflections of the site’s progress toward what is considered a high quality implementation of the program.

Quality Benchmark Categories with Summary of Related Site Coordinator Survey Items

Quality Categories

Summary of Category Item Content and Expectations Benchmark Goal #

1. Cycle Plans Cycle plans are used and useful for club activity implementation 5

2. Time Distribution

Students are involved for expected number of minutes in required components of: homework, fitness, nutrition/snack, interest-based club activities, and closing activities/all-together time (ES)

5

3. Club Activities

Club activities offered include all six types of: Sports, Performing arts, Visual arts, Recreation activities, Computers or Technology / multi-media, Leadership opportunities/youth development

6

4. Student Engagement

Students are engaged in appropriate frequencies of additional youth development activities such as sport tournaments, club culmination

9


Page viii

activities, middle/high school transition activities, etc.

5. Club Contributors

Contributors to the delivery of club activities (beyond site coordinator and club leaders) include teacher liaison, activity consultants, traveling specialists (MS), and school personnel

3 (ES) 4 (MS)

6. Ad Hoc Staff Value

Club delivery contributors (same list as 5 above) have appropriately important roles in the selection and delivery of site activities

5 (ES) 6 (MS)

7. Club Selection

Reported staff that help to select the club activities include: regional manager, site coordinator, club leader, youth council, youth participants, principal/school administrator

6

8. Student Involvement

Extent to which student involvement is incorporated by program, such as needs and interests of students considered, students have choice of club activities, leadership opportunities offered, etc.

5 (ES) 6 (MS)

9. Parent Involvement

Extent to which parents are generally involved in the program and involved in specific activities, such as communicating with staff, attending special events, volunteering, etc.

14

10. School Collaboration

Extent to which the site generally collaborates with their host school and engages with school staff in specific communications & activities

8

11. Facilities Access

Extent to which site has access to school facilities as well as the specific types of school spaces available to the program

8

12. Site Coordinator Qualities

Importance of a variety of possible characteristics for a site coordinator to have, including items related to management style, organization skills and staff communication

11

13. Program Staff Qualities

Importance of a variety of possible characteristics for program staff to have, including leadership, organization and student interactions

11

In addition to the category scores, a total score was calculated that indicates how close the program is to one composite benchmark, and is considered the overall quality indicator. The distribution of overall quality scores (reported as a percent of the benchmark) was categorized into three levels: low, medium and high. Category cut-offs were determined by creating levels with a similar number of schools in each. For elementary sites, the range of average score is not that large, with an average of 60% for low quality sites and 74% for high-quality sites. The opposite is true for the middle school sites. Low-quality sites have an average score of 48% while the high-quality sites average an 80% score.

Relationship of Quality Indicators with Youth Outcomes When analyzing the quality indicators (factors and benchmarks) with youth outcomes, the desired, or favorable, result is a positive relationship between the two. This indicates that as quality increases, positive youth outcomes increase (greater positive change or higher positive level). The only exception is in the instance of problem behavior. In this case, the outcome variable is scored so that higher scores are reflective of greater problem behaviors.


Page ix

Elementary School Site Findings Overall, the ability of the quality factors and benchmarks to predict elementary youth outcomes is mixed, showing that the different components of quality are differentially associated with student outcomes. Specifically, there is no one quality indicator favorably (i.e., in the expected direction) associated with all twelve outcomes, and no one outcome associated with all the quality indicators. Rather, each indicator has its own profile of outcome relationships. Of the 16 indicators, five (5) have relationships that are only favorable with a sub-set of the youth outcomes. This includes the Ad Hoc Assistance factor and the following benchmarks: Student Engagement, Club Contributors, Access to Facilities and Site Coordinator Qualities. This indicates that when program quality is higher in each of these areas, the more positive are at least one or more of the desired youth outcome results. Other quality indicators either have a combination of negative and positive relationships with outcomes (8), only negative relationships (3), or no relationships (1).

To illustrate the findings from a different perspective, the table below presents the statistically significant associations between desired student outcomes with quality benchmarks for elementary sites. Outcome areas in which at least four of the quality benchmarks have a favorable impact are: decreased problem behaviors, higher level of use of academic skills in activities, and higher level of involvement in making choices.

Desired Student Outcomes Associated with Quality Benchmarks for Elementary School Programs

Desired Student Outcome Significantly Associated Quality Benchmarks

Increased CST Math scores Club Activities, Access to Facilities

Increased school attendance Club Contributors

Increased homework/school skills Ad Hoc Staff Value, Club Selection

Increased pro-social skills Time Distribution, Club Contributors

Higher level of use of academic skills in activities

Student Engagement, Club Contributors; Student Involvement, Program Staff Qualities

Increased positive attitude toward school

Club Contributors

More positive attitude toward after-school experience

Ad Hoc Staff Value, Site Coordinator Qualities

Higher level of involvement in making choices

Club Activities, Club Contributors, Student Involvement, School Collaboration, Program Staff Qualities

Decreased problem behaviors Club Contributors, Ad Hoc Staff Value, Club Selection, Access to Facilities, Site Coordinator Qualities

Analyses were also conducted with the overall benchmark ratings created for the sites. At elementary sites, this overall quality rating does not appear to be helpful in understanding or predicting student academic outcomes. Regression analyses show that the overall score is not related to any youth outcome indicator.


Page x

Middle School Site Findings Similar to the elementary site findings, the ability of the quality factors and benchmarks to predict middle school youth outcomes is mixed, indicating the different components of quality are differentially associated with student outcomes. Unlike elementary school sites findings, there are many more favorable relationships found between the quality indicators and outcomes. Of the 16 quality indicators (3 quality factors and 13 quality benchmarks), seven (7) have only favorable relationships with a sub-set of youth outcomes. This includes the Cycle Plans factor as well as the following benchmarks: Cycle Plans, Student Involvement, Parent Involvement, School Collaboration, Facilities Access and Site Coordinator Qualities. This indicates that when program quality is higher in each of these areas, the more positive are at least one or more of the desired youth outcome results. Other quality indicators either have a combination of negative and positive relationships with outcomes (6), only negative relationships (2), or no relationships (1). While there is no one indicator favorably associated with all thirteen middle school outcome indicators, five of the indicators are associated with at least half of the outcomes.

While the discrepancies between which outcomes the quality indicators are favorably related to and not help highlight that there is no one quality indicator that can be used to determine effectiveness related to outcomes, quality does seem to have a more important impact on middle school outcomes than seen at elementary sites. There are nine outcome areas with favorable associations to at least one quality benchmark (see table below). Unlike elementary site findings, most of the middle school outcomes are associated with several quality benchmarks, versus just one or two. In addition, unlike elementary, significant findings are found between overall site quality and a few key outcomes, including increased CST ELA scores, decreased problem behaviors and increased positive attitude toward school. These findings point to some utility in understanding overall quality of middle school sites, along with the different components of quality.

Desired Student Outcomes Associated with Quality Categories for Middle School Programs

Desired Student Outcomes Significantly Associated Quality Benchmarks

Increased CST ELA scores Club Contributors, Club Selection, Student Involvement, Parent Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities, Program Staff Qualities

Increased CST Math scores Student Involvement, School Collaboration

Increased school attendance Access to Facilities, Site Coordinator Qualities, Program Staff Qualities

Increased homework/school skills Ad Hoc Staff Value, Student Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities

Higher use of academic skills in activities

Cycle Plans, Time Distribution, Ad Hoc Staff Value, Access to Facilities, Site Coordinator Qualities

Improved school attitude Cycle Plans, Time Distribution, Ad Hoc Staff Value, Student Involvement, Access to Facilities

More Increased positive attitude about after-school experience

Ad Hoc Staff Value, School Collaboration, Access to Facilities


Page xi

Higher involvement in fitness

Cycle Plans, Student Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities

Decreased problem behaviors Cycle Plans, Ad Hoc Staff Value, Student Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities

Conclusions Of the two methods used to quantify program quality, factor analysis versus benchmark, the benchmark approach of comparing sites’ responses to a management-established quality expectation is more useful because it includes more content areas than can be included in the factor approach. However, the factor analysis is important in that it produced program areas consistent with the described program model and provides credibility to the site coordinator survey as a good measure of the Woodcraft Nvision program model.

In both approaches, some of the quality indicators are better at predicting positive outcomes than others, but none of the indicators are positively associated with all the outcomes. Using an overall score is even less predictive of outcomes, especially at the elementary level. These findings suggest that overall quality may not be a useful tool when trying to improve program operations with the intent of affecting student outcomes. Rather, more specific areas should be assessed and targeted to affect positive change in students.

Several middle school youth outcomes appear to be influenced by multiple quality indicators. Improvements in English/Language Arts standardized test scores and decreases in school problem behavior are related to at least eight quality areas. Higher levels of involvement in fitness activities, improvements in positive school attitudes and more use of academic skills in activities are related to at least five areas. The most critical quality indicators for middle school programs appear to include: Access to Facilities, School Collaboration, Student Involvement and Site Coordinator Qualities. Cycle Plans and Ad Hoc Staff Value are also important.

The findings for elementary sites are not as strong or as consistent as those found for middle school sites. This may be a result of a greater number of sites such that variability in program implementation and approach differ beyond measured items. Or, outcomes for younger students may be less influenced by quality as it has been examined in this study. At elementary sites, key quality areas that have the most relationships to favorable student outcomes include: Club Contributors (to program delivery), Access to Facilities, and Site Coordinator Qualities. Of the factors, Ad Hoc Assistance is also associated with desired outcomes. There are three elementary student outcome areas that have at least four quality indicators favorably associated with them: decreased problem behavior, higher involvement in making choices, and more use of academic skills in activities.

The importance of this study for Woodcraft Rangers, and potentially for the field at large, is the confirmation that after-school program quality needs to be understood and defined in terms of outcomes. Given the finding that different aspects of quality impact different outcomes, the next step is to explore how to balance or refine quality expectations so that the program’s priority outcomes are achievable. The study also helps identify a process for how to measure


Page xii

quality efficiently and accurately while producing program improvement information that is “actionable” for staff. This addresses Woodcraft’s stated need to begin developing a data-based quality monitoring tool. In addition, the benchmark process may be of use to other after-school providers, even though specific categories and benchmarks will differ. The model can potentially work for any provider looking for a way to assess quality that is not too resource-heavy, relies on well-informed input and engages staff in the improvement process.

Woodcraft Rangers: Assessment of Quality Introduction

Page 1

I. INTRODUCTION

Formally established in 1922 in Los Angeles, Woodcraft Rangers (Woodcraft) is a youth development non-profit agency with a mission to guide young people as they explore pathways to purposeful lives. Woodcraft developed its after-school program, known as Nvision, to serve elementary through high school youth in impacted and high-need neighborhoods throughout LA County. The program is provided at more than 60 public school sites five days a week from 3:00 pm to 6:00 pm. Nvision is designed to help students achieve:

Improved school attendance and behavior/attitudes toward school;

Enhanced academic performance and language development;

Strengthened social/leadership skills; and,

Improved physical activity.

From 2000 to 2008, Woodcraft Rangers worked with an external evaluation consultant to complete a series of outcome evaluation studies to assess its impact on youth. The studies show that youth who attend Nvision more often experience better outcomes than those who do not attend.1 During the 2008/09 fiscal year, a site implementation study was conducted by EVALCORP, a contracted research and evaluation consulting firm, to assess how Woodcraft’s Nvision program model is implemented across sites. This assessment examined the extent to which the Woodcraft model, vision and assumptions about program implementation across sites are supported by field staff and key stakeholder experiences (“Woodcraft Rangers Site Implementation Report, 2009”).

An underlying purpose of the 2008/09 study was to create a dataset that could be used to explore the quality of the implementation of the Nvision model. This was in response to the Woodcraft administration’s request, which had directed its evaluation focus toward exploring quality as it relates to targeted youth outcomes. In 2009, Woodcraft contracted with EVALCORP to conduct a secondary analysis study using select data from the quality study and youth participant outcome data from 2008/09. The study’s primary purposes are to:

Explore the quality of implementation of the Nvision after-school program model;

Examine whether and how quality is associated with student outcomes; and,

Inform further development of a way to monitor quality and develop improvement strategies.

This report focuses on understanding quality for the Nvision elementary and middle school programs. It includes 43 elementary school sites and 14 middle school sites located in the Los Angeles Unified and Garvey school districts. Woodcraft’s high school sites are not included because the high school program is still in early stages of development and includes only three sites (not a large enough number of sites for this type of study).

1 See http://www.hfrp.org/out-of-school-time/ost-database-bibliography for a summary of outcome studies.

http://www.hfrp.org/out-of-school-time/ost-database-bibliography


Page 2

A. RESEARCH CONTEXT

The field of after-school research has shown that regular attendance in high quality after-school programs is associated with positive developmental outcomes for school-age youth, such as academic performance, task persistence, study skills, and social skills.2 Many evaluations have been conducted to measure the impact of program attendance on outcomes, with the assumption that the program is of high-quality. At the same time, there are studies that indicate program quality, not just participation, is a key determinant of positive outcomes. Yet, few have directly tested the relationships between after-school program quality and student outcomes. Many after-school programs also struggle with understanding and measuring the quality of their programs.

Knowing the powerful influence that high quality programs have on determining good outcomes for youth who participate in after-school programs, there is a need to better understand the challenges of developing and implementing quality after-school programs. While several program quality assessment tools exist, they are not widely used and are not designed to assess how quality program components relate to youth outcomes.3 It is difficult to define quality across after-school programs because of the variety of academic, enrichment, recreational and other activities that programs use to affect youth development, as well as different goals and desired outcomes for participants.4,5

A recently published review identified key elements of program quality after examining available quality assessment tools.6 Key elements identified from this review include engagement, interaction, a supportive environment and a safe environment. However, this is but one approach to understanding quality. Differing reports and evaluations exist that identify other key factors in program models and areas for quality assessment. Nationally, states are moving towards the creation of after-school quality measures, improving quality through licensing and accreditation, professional development and incentives for improving quality.7

Generally, quality assessment tools for after-school programs are broad, complicated and require an objective observer or multiple observers to gather information about how the

2 Vandell, D., Reisner, E., Brown, B., Dadisman, K., Pierce, K., Lee, D. and Pechman, E. (2005). The study of

promising after school programs: Examination of intermediate outcomes in year 2. Madison, Wisconsin: Authors. Available at http://www.wcer.wisc.edu/childcare/pdf/pp/year_2_report_final.doc 3 Little, P. (2007) The quality of school-age child care in after-school settings. New York: Child Care & Early

Education Research Connections. Available at: http://www.researchconnections.org/childcare/resources 4 Smith, C., Devaney, T.F., Akiva, T., and Sugar, S.A. (2009). Quality and accountability in the out-of-school-time

sector. New Directions for Youth Development, 121, 109-127. 5 Huang, D., La Toree, D., Harven, A., Huber, L.P., Jiang, L., Leon, S., and Oh, C. (2008). Identification of key

indicators of quality in afterschool programs. Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 6 Yohalem, N., and Wilson-Ahlstrom, A. (2007). Measuring youth program quality: A guide to assessment tools.

Washington DC: Forum for Youth Investment. 7 National Institute on Out-of-School Time at the Wellesley Centers for Women at Wellesley College. (2009).

Making the case: A 2009 fact sheet on children and youth in out-of-school time. Wellesley, Massachusetts: Authors.

http://www.researchconnections.org/childcare/resources


Page 3

program is implemented. These types of assessments can be costly as they require significant resources to collect and analyze findings. Some tools allow for program staff to perform a self-assessment and complete their own quality analysis. Although this approach can empower program sites and encourage their buy-in to the importance of quality improvement, self-assessment is subject to reporting bias in that sites may be likely to report more positively.

Using assessments of quality as a means to establish quality improvement processes can aid in the management of after-school programs. Such processes can help keep program sites more consistent with the program model and key quality indicators by allowing early identification of challenges and the development of improvement plans that can be employed in order to avoid programs drifting away from the model. The challenge is to identify a way to measure quality efficiently and accurately while producing information that is helpful and actionable.

B. WOODCRAFT’S NVISION MODEL

Just as it can be difficult to define quality in a meaningful way across providers, so can it be for just one provider with multiple sites. At Woodcraft Rangers, sites are expected to follow the Nvision model, but that model is flexible so that each site can also make sure it meets its specific community’s needs and interests. For example, student engagement is a critical component of the Nvision model and there are general parameters for how that should look at the elementary, middle and high school level (e.g., types of activities, frequency). Each site then decides how best to address it given those parameters. Following is a brief description of the Nvision model to help set the context for how implementation quality was assessed.

The Nvision elementary and middle school after-school program time is divided into structured segments, including a grade- or subject-specific homework segment, fitness segment and nutrition segment with healthy snacks provided. Enrichment club activities make up the last segment of the afternoon. Although some sites may vary the order of these segments, all are provided at elementary school and middle school sites on a daily basis. In addition, sites provide expanded learning opportunities through special guest speaker assemblies, field trips and community volunteer events.

The daily club activities are the core of the Nvision program. They are student-driven, based on youth’s current interests or the latest trends, and designed as ‘disguised learning’ experiences that are aligned with California Content Standards. The goal is to simultaneously support academic progress while keeping students engaged. A sampling of clubs offered includes: drama, music, dance (from Folklorico to Hip-Hop), team sports, martial arts, photography, and robotics. The clubs run in either six- or eight-week cycles depending upon the school calendar, and can meet two, three or five days per week. Each cycle builds skills towards a specific achievement, such as a new dance or a graphic arts exhibit, and the cycle culminates with participants presenting their accomplishment to their community of students, parents and school staff.


Page 4

To successfully implement the program and help change students lives, the Nvision model requires that each site:

Be staffed by a site coordinator (3/4 to full-time) who has leadership skills and club leaders (part-time) staff who have expertise in at least one enrichment area;

Access the support available from Woodcraft’s administrative and cross-site staff, such as regional managers, activities consultants, traveling specialists;

Work closely with an identified teacher liaison (stipend for about 8 hours a week) and collaborate with other school staff, including administrators, teachers, support staff;

Implement all segments every day: homework, fitness, nutrition, enrichment clubs;

Be interest-based and comprehensive, making sure to offer clubs each year in: Language Arts, Science/Math, Technology, Visual Arts, Performing Arts and Sports;

Create a plan for each club cycle that identifies the cycle goal, the academic concepts to be addressed and the disguised and project-based learning techniques that will be used;

Use age-appropriate methods for engaging students in leadership activities;

Offer several types of opportunities for parents to be involved; and,

Link students to community events and volunteer/service opportunities.

C. STUDY APPROACH

The plan for this study was to conduct exploratory analyses using secondary implementation and outcome data from 2008/09. The purpose was to assess how the critical components of the Nvision model could be used in a data-based strategy for monitoring quality and developing site and program improvement strategies.

The first part of the study involved using the site coordinator 2008/09 survey data to identify the key components of program implementation and assess how they could be best used to measure quality. This would aid Woodcraft in the future in developing a quality assessment tool that is effective for their program model. Two different methods, a factor analysis approach and a benchmark approach, were used to identify the key components of the program model. The factor analysis approach used statistical methods to identify quality indicators. The benchmark approach used agency-identified expectations of quality as a benchmark to which site assessments were compared. Both approaches were used in order to take full advantage of the available site data and to explore which approach would be most useful for Woodcraft to use in creating a quality monitoring tool.

The next part of the study was to explore how the factor and benchmark quality measures were related, or not, to student outcome data from that same 2008/09 year. Data from student academic records and from program entry and 2008/09 follow-up surveys were used to measure outcomes such as achievement, attendance, attitude and behavior. The goal of this part of the study was to begin to understand program quality as it relates to youth outcomes, and in what combinations. Both the quality factors and benchmarks were used in correlation and regression analyses to determine if they could predict student outcomes.

Woodcraft Rangers: Assessment of Quality Methods

Page 5

II. METHODS

This study focuses on the program year of 2008/09 and relies on quantitative data collected by Woodcraft staff from staff, student participants and school districts. Quality analyses included data from all sites where a site coordinator completed the program implementation survey in 2009. Analyses of youth outcomes use youth survey and school records data from 2008/09 and years prior. For youth outcome data, change was computed using the earliest survey data available for an individual (as early as July 2006) and the end-year data for 2008/09. Change in outcomes that use school records was computed using 2007/08 and 2008/09 time points. In all cases, errors and outliers found in data were excluded from analyses.

As with any study, particularly an exploratory one, there are some limitations. For example, the staff and youth instruments were not standardized or statistically validated prior to use. Both instruments have high content (face) validity because they were created in concert with Woodcraft staff so the questions could be tailored to the program. Reliability is promoted by using multiple items per quality or outcome indicator (with a few exceptions among the outcome indicators). Another limitation is related to the short time period under study – the length between the outcome comparison time points could be as little as one year. Youth outcomes may not be immediately affected by the after-school program, even if the program is of the highest quality. While staff and youth self-report may be considered a limitation, this type of data was purposefully used for this study. Woodcraft’s goal is to create an outcome-linked quality assessment process that is based on well-informed input while not being too burdensome or resource-heavy. Staff and youth participants have the deepest experience with the program and data collection is already incorporated into expectations of their involvement.

A. DATA SOURCES

Several data sources were used for this study: Woodcraft site coordinator survey, Woodcraft youth participant baseline/end-year survey, school academic and attendance records for Woodcraft participants, after-school program attendance days for youth participants, and sociodemographic and economic indicators for the communities in which the Woodcraft sites are located. Each source is described below.

1. Site Coordinator Survey

Data collected from site coordinators during Spring 2009 were used to examine how sites implemented the Nvision program model. It is believed that site coordinators have the most comprehensive and accurate assessment of program implementation, as their responsibility is to implement the Nvision program at their respective sites. A total of 55 of the 57 elementary and middle school site coordinators completed the survey tool (42 of 43 elementary sites; 13 of 14 middle school sites).

The survey tool consisted of more than 150 quantitative and qualitative questions about how the site operates. Questions focused on how the site operates in terms of:


Page 6

Staffing (pattern, experience, desired qualities and other staff or volunteer supports);

Site structure and activities (schedule, activity types and frequency);

Daily club activities (types, selection process, cycle plan use and usefulness);

Youth engagement and leadership;

School collaboration (frequency and types of interactions, facility access);

Parent involvement (frequency and types of involvement); and,

Community involvement.

Quality indicators were calculated using the quantitative items from the site coordinator survey. Additional details about the how these were calculated are presented in the Factors and Benchmarks sections.

2. Youth Baseline/End-Year Survey Data

Woodcraft participants are administered a baseline survey when they first join the Woodcraft program. At the end of every academic year, an end-year survey is also collected. There is a version of the surveys for youth in grades two through five, and another version for students in grades six through 12. Many of the items are the same across the two age levels, but the response scales differ or the wording differs to be more age-appropriate. The baseline and end-year surveys within each grade group are nearly identical, as they were designed to measure change in attitudes, academic and social skills, sense of efficacy and risk-related activities. The end-year surveys also include questions about students’ experience and satisfaction with the Nvision program.

Youth developmental indicators, which are linked to youth survey items, were selected from the Nvision program logic model in order to explore how quality is related to student outcomes. Relevant youth outcomes are provided in Table 1. For some outcomes, the desired result is measured by assessing change over the course of students’ involvement in Nvision – from when they first started to the end of the most recent program year (2008-09 for this report). These are identifiable by the word “improved” or “decreased.” For other outcomes, the desired result is measured by assessing the level of students’ involvement at end-year, and these are identifiable by the word “higher” or “more.”

TABLE 1. LIST OF YOUTH DEVELOPMENT OUTCOMES


Improved school attitude

Higher involvement in making choices Improved self-confidence

Higher use of academic skills in activities Improved pro-social skills

Higher involvement in fitness Decreased problem behaviors

Improved homework and school skills Improved leadership skills (MS only)


Page 7

The logic model youth development outcomes are linked to questions in the Woodcraft youth survey. Appendix A maps the relevant youth outcome indicators to specific youth survey items. Several indicators include more than one survey item. In these cases, the average of all items of that indicator was calculated by using the numerical values assigned to responses from students (usually from 1 to 4, or 1 to 5 where lower values reflect the absence of a behavior or a negative response). In indicators where more than one item is used to calculate a score, if one response was missing, a score for that indicator was not calculated.

In order to assess degree of change, change scores were calculated by subtracting the indicator score from the latest available end-year survey from the student’s baseline survey (referred to as a “match”). Thus, the desired result is a positive change value for all outcomes with the exclusion of “decreased problem behaviors.” For the decreased problem behaviors outcome, a negative change value is the desired result, reflecting a decrease in the level of problem behaviors. In the cases where only the end-year level of involvement is assessed, the desired result is a positive value for all of those outcomes.

Table 2 describes the number of baseline and end-year surveys available for analyses and the number of schools represented.8

TABLE 2. NUMBER OF MATCHED BASELINE AND END-YEAR SURVEYS BY SURVEY TYPE

Number of Students with Matched Baseline and Year-End Survey

Number of Schools

Grades 2-5 Survey 2,304 46

Grades 6-12 Survey 406 18

3. Academic and School Attendance Records

In addition to measuring youth development outcomes, Woodcraft evaluations measure youth academic outcomes. These are also in the Woodcraft Nvision logic model and are linked to data sources available through requests to local districts. Relevant youth academic outcomes are provided in Table 3.

TABLE 3. LIST OF YOUTH ACADEMIC OUTCOMES

Improved school attendance

Improved academic performance in English/Language Arts

Improved academic performance in Math

Academic outcomes were measured using data provided by Garvey and Los Angeles Unified School Districts. Academic data included scores from the 2008/09 academic year and from the 2007/08 academic year on the California Standards Test (CST) in English/Language Arts (ELA)

8 Because some schools have grade levels that overlap the two survey types, they are counted twice in this table,

once in Grade 2-5 and again in Grade 6-12.


Page 8

and in Mathematics. Proficiency scores on the CST range from one to five (1 = Far Below Basic, 2 = Below Basic, 3 = Basic, 4 = Proficient, 5 = Advanced). Change scores for both ELA and Math CSTs were calculated by subtracting their score in 2008/09 from the 2007/08 score, thus, only those students with data from both years were included in analyses.9

School attendance from the academic years of 2007/08 and 2008/09 were used. Attendance was calculated by dividing the number of days attended by the number of days enrolled in school in order to create a percent of days attended. Similar to CST data, only those students with data from both years were included in analyses (Table 4).

TABLE 4. NUMBER OF MATCHED SCHOOL ACADEMIC RECORDS BY DATA AND SCHOOL TYPE

Number of Students with Matched: Number of

Schools CST ELA Scores

CST Math Scores

School Attendance

Elementary Schools 3,285 3,298 4,696 37

Middle Schools 4,905 4,889 4,717 13

4. After-School Program Attendance

For this study, it was also important to account for the contribution of any known factors beyond quality on student outcomes. Research from the field and Woodcraft’s own outcome evaluations have shown that there is a relationship between after-school program attendance and youth development outcomes. As a result, the number of days of after-school program attendance during the 2008/09 program year was collected from the Woodcraft Rangers data system for the students included in this study.

5. School Community Sociodemographic and Economic Indicators

There were concerns that sociodemographic and economic characteristics of the communities in which the schools are located and most of the participants live may be related to or impact program quality and youth outcomes. Rather than use individual student demographics, demographics for the communities in which the schools are located were used in analyses since the focus of this study is understanding the quality of each site and how that impacts student outcomes rather than trying to understand the nature of individual-level outcomes. To that end, data from the U.S. Census 2000 for each school region was collected. Though school boundaries are not determined by zip code, it was concluded that data by zip code would approximate the region around the schools. Indicators used include:

Percent of families in poverty;

Median family income;

9 CST proficiency scores rank performance according to what students are expected to know and do in each grade

according to California content standards. At each grade, students are expected to score proficient or advanced. While scores are not directly comparable year to year, computing a change score helps indicate whether or not students are moving towards higher levels of proficiency even as academic expectations increase.


Page 9

Percent of persons (16 and older) in labor force;

Percent of persons with high school degree or greater; and,

Percent of persons who speak a language other than English in their home.

Averages by school type (elementary or middle school) are presented in Table 4. Correlations between school community sociodemographic and economic indicators and youth outcomes were performed. Significant correlations were found suggesting that controlling for their contribution to student outcomes is necessary. See Appendix B for extensive detail on the statistical analyses and findings.

TABLE 5. SCHOOL SOCIODEMOGRAPHICS BY SCHOOL TYPE10

Percent of Families in

Poverty

Median Family Income in

Dollars

Percent in Labor Force (16 years old and

older)

Percent who Graduated from High School (25 years old and

older)

Percent who Use Language

Other Than English in Home

Elementary Schools (n=45)

19.1% Range= 3.4-48.8

$37,774 Range=

19,388-79,646

55.9% Range=

47.2-66.4

50.4% Range=

29.6 – 90.8

73.7% Range=

28.1-90.5

Middle Schools (n=12)

26.5% Range=

11.2-46.4

$30,943 Range=

16,757-52,608

54.9% Range=

49.5-65.0

42.7% Range=

25.1 – 77.4

77.3% Range=

45.9-90.5

B. MEASURING QUALITY – TWO APPROACHES

As mentioned, two methods or approaches were used to explore ways to measure quality. Within each approach, quality indicators were developed and explored to determine if there was any significant relationship between quality (as measured by these quality indicators) and youth outcomes.

The first approach used a factor analysis of the site coordinator survey to derive a set of program areas that could be considered as quality indicators (referred to as quality factors) based on the factors generated by the statistical method. Factors that demonstrated “good enough” reliability (alpha coefficient of .6 or greater) were used in correlation and regression analyses to determine if these quality factors could predict student outcomes. Section III – Quality Factors – presents a summary of the factor analysis. See Appendix C for extensive detail on the factor analytic, or quality factors, approach and process.

The second approach used the site coordinator survey in a slightly different way. In order to take advantage of the data available rather than use only a select number of items required by the factor analytic approach, a benchmark was developed. The Woodcraft executive and management team was asked to complete the site coordinator survey in a manner that they would expect a high-quality site to do. High-quality was then defined as full and successful

10 Data Source: U.S. Census 2000


Page 10

implementation of that definition of the Nvision program model. Their survey was considered the benchmark and was used as a comparison to the responses of the site coordinators. The more similar a site’s response was to the benchmark, the more their implementation was considered to be reflective of a “high-quality” site in that area.

The benchmark approach was used in two ways: (1) to examine site scores in several different program areas and the relationship of each to youth outcomes, and (2) to transform overall site scores into categories (low-, medium- and high-quality) and compare these groups based on observed student outcomes. The categorical method was performed in anticipation that this approach would be easiest to adopt for quality improvement purposes if the method was determined to be useful. Section IV – Quality Benchmarks – provides a summary of the benchmark approach. See Appendix D for detail on the benchmark items and scores.

C. ANALYSES OF QUALITY WITH OUTCOMES

Multiple regression analyses were used to explore the relationships between the quality factors and benchmarks and youth outcomes. In all of the regression analyses, school sociodemographic and economic indicators along with the students’ attendance in Woodcraft were included as control variables. Separate regression models were run for elementary sites and middle school sites because expected outcomes differ by these two age groups and because the benchmark approach yielded different quality expectations in some cases between the elementary and middle school program model. Section V - Relationship of Quality Indicators with Youth Outcomes –summarizes the statistically significant relationships found between the quality indicators (term used to refer to both the factors and benchmarks) and youth outcomes. See Appendices E and F for the quality factor regression results and Appendices G and H for the quality benchmark regression results.

Woodcraft Rangers: Assessment of Quality Factors Approach

Page 11

III. QUALITY FACTORS

The factor analysis of the site coordinator survey identified five areas that may be considered when conceptualizing and understanding site quality. These areas may be useful in understanding how sites are implementing the Nvision model and identifying areas in which Woodcraft may want to focus program improvement efforts for sites needing implementation support. The quality factors and statistical process used to create them are described below.

A. ITEM SELECTION

Before factor analyses could be completed, the available data had to be reviewed to determine which items were appropriate for this statistical technique. The Site coordinator survey had a total of 146 items that could be quantified and used in statistical analyses, though many of the items were variations of a general theme and some had response types that could not be used in a factor analysis (i.e., yes/no, or absent/present). The number of surveys available (55) is relatively low for the demands of a factor analysis so that items had to be removed in order to create a stable model. A total of 130 items were removed: 65 due to redundancy with a common theme and another 65 due to their response type. Once these items were removed, a factor analytic model was created.

The 16 remaining items produced a six-factor solution; however, one factor (value of activity consultants) was eliminated because the factor was a single item. Generally, factors with fewer than three items are considered weak and unstable. Thus, the final factor solution consisted of 15 items, represented in five factors. The solution met minimum criteria used to establish that the data are appropriate for use in a factor analysis (see Appendix C for a detailed description of these criteria).

Items that were maintained in the model reflected the practices, attitudes and values of various aspects of the program model. The items were related to adherence to the site schedule, cycle plan implementation, the value of the contribution of different types of stakeholders (staff and volunteers), collaboration with schools, involvement in the community, and the level of involvement of parents, students and the school staff.

B. FACTOR ANALYSIS RESULTS

Items were examined using principal components analysis with Varimax (orthogonal) rotation. The final factor analytic model produced a five factor solution, explaining 68.2% of the total variance. Details regarding how factors were selected can be found in Appendix C.

Each quality factor is comprised of three items. A theme, based on the item(s) within each factor, was identified and used to create quality factor names. Quality factors were described using knowledge of the Nvision model and information gleaned from site visits of program sites conducted in 2008 (see Woodcraft Rangers Site Visit Summary Report, 2009). The five quality factors are presented in Table 6 below and described in more detail after the table.


Page 12

TABLE 6. QUALITY FACTORS AND SITE COORDINATOR SURVEY ITEMS

Core Elements To what extent do you have a collaborative relationship with school administrators? Generally, how often do you follow the site schedule?

To what extent are students involved in decisions that impact the program design?

Cycle Plans

To what extent do you implement cycle plans? How useful are your cycle plans during the implementation of club activities at your site? To what extent are the objectives of your cycle plans met?

Value of Ad Hoc Assistance

How valuable is the contribution for each staff that help to deliver activities at your site: WR Parent Volunteers?

How valuable is the contribution for each staff that help to deliver activities at your site: Youth Volunteers?

How valuable is the contribution for each staff that help to deliver activities at your site: WR Traveling Specialists?

Connections


In general, how involved are parents in your program?

How would you describe the access you have to school facilities that you need to implement your programs’ activities?

Educational Supports



To what extent are you using academic concepts in club activities?

1. Core Elements

The first factor consists of items that reflect core components of the program’s infrastructure, and are considered core because they were both described by program staff and observed in site visits. These critical elements include collaboration with the host school, incorporation of the voice of youth participants, and ensuring that major activities are always incorporated into daily programs. School collaboration is required because the host school helps Woodcraft to deliver the program by providing staff, space and support. Youth assistance in decision-making empowers participants, promotes their ownership and involvement in the program, and may result in more positive outcomes. Lastly, adherence to the expected program structure aids in keeping the site consistent. This factor may be considered a way to measure program fidelity.


Page 13

2. Cycle Plans

The Nvision model covers a variety of activities in its program and promotes thoughtful planning and development of activities by requiring that site staff complete cycle plans. Cycle plans are descriptions of goals and objectives of activities delivered at program sites. Club leaders develop their own cycle plans with the consultation and input from site coordinators, activity consultants, and at times, youth participants. It is believed, based on site visit observations and interviews with program staff in the previous evaluation, that program staff are much more likely to use and follow “home-grown” club activities because club leaders have developed the plans. Three of the cycle plan items fall into this factor, and reflect the perceived utility, level of implementation and success of cycle plans.

3. Value of Ad Hoc Assistance

The value of contributors other than the regular site staff, which include youth and parent volunteers and traveling specialists, falls within the same factor. This factor reflects the importance of the skills that youth, parents and enrichment specialists bring to the program. For example, youth volunteers may provide tutoring to younger program participants or help plan program infrastructure. Traveling specialists are professionals who run clubs related to their field of expertise (e.g., photography, dance, etc.). They are often brought to the program after student requests for club activities in areas that regular Woodcraft program staff does not have. Ad hoc assistance adds value to the Woodcraft Rangers Nvision after-school program because it supports the delivery of the program, infuses additional knowledge and creativity to the program, and keeps youth interested in club activities.

4. Connections

The fourth factor includes the availability of community-based activities for youth participants, the level of parent involvement in the program, and access to the host school’s facilities. These items reflect opportunities to tap into various networks. Connections underlie the program to promote youth development – connecting youth to the community, to their parents and to their school. Ultimately, with a program’s greater access to such connections, implementation of the program is broader and potentially more beneficial to students.

5. Educational Supports

Woodcraft program sites disguise academic learning by creating club activities that are built upon academic concepts and objectives but are delivered in non-academic ways. Though academics are purposefully not an overt aspect of the program, they are always included in activities that are more interesting to youth. Two of the items that make up this factor, the value of teacher liaisons and school personnel, reflect the educational supports that make Woodcraft work. These school staff often provide support to program staff in the development of activities and facilitation of program staff development. The third item, the incorporation of academic concepts into the cycle plan, has a negative relationship with the other two items, suggesting that as the value of these school staff increases, the use of academic concepts in cycle plans decreases. One explanation of the negative relationship may be that programs that incorporate academic concepts into their cycle plans explicitly are more familiar with how to do this, and less likely to need the educational support from teacher liaisons and school personnel.


Page 14

C. RELIABILITY OF FACTORS

Internal consistency for each of the quality factors was examined using Cronbach’s alpha to determine whether the items can be used to create scales to explore program implementation. The alpha coefficients for the factors ranged from .214 – .682 (descriptive statistics are presented in Table 7). For exploratory research, a minimum of .600 is required to establish the reliability of a scale. Only the first three factors met this criterion: Core Elements, Cycle Plans and Ad Hoc Assistance. The other two areas, Connections and Educational Supports, do not have strong enough reliability to be considered stable factors using the current survey measurement tool. Therefore, the Connections and Educational Supports quality factors were not included in the subsequent quality-outcome analyses.

TABLE 7. ALPHA COEFFICIENTS OF QUALITY FACTORS

Quality Factors Alpha Coefficient Scale Mean (Standard Deviation)

Core Elements .682 10.94 (1.38)

Cycle Plans .606 9.84 (1.29)

Value of Ad Hoc Assistance .676 6.35 (2.30)

Connections .512 8.67 (2.00)

Educational Supports .214 3.75 (1.94)

D. AVERAGE QUALITY FACTOR SCORES

In Table 8, average scores on quality factors are presented by each school type. Average scores were calculated by summing the responses across all items that make up each factor. Higher scores reflect higher quality of implementation, as all items on the site coordinator survey were scored so the more positive the response the higher the score assigned to that response.

For both Core Elements and Cycle Plans, sites’ average score is on the high end of the possible range. This indicates that across Woodcraft, site coordinators tend to rate their sites similarly and positively in these areas. Average ratings for Ad Hoc Assistance tend to be more in the middle of the range, indicating more differences across sites in how highly they rate the value of other stakeholder contributions to program delivery. There are no significant differences in the quality factor scores between elementary and middle schools.

TABLE 8. AVERAGE SCORES ON QUALITY FACTORS BY SCHOOL TYPE

Core Elements Score

Cycle Plans Score

Ad Hoc Assistance Score

Elementary (N=42) Average

Range

10.9 (5 – 12)

10.0 (7 – 11)

6.4 (4 – 10)

Middle (N=13) Average

Range

11.1 (5 – 12)

9.3 (5 – 11)

6.0 (3 – 12)

Woodcraft Rangers: Assessment of Quality Benchmark Approach

Page 15

IV. QUALITY BENCHMARKS

The factor analysis of the site coordinator survey demonstrated that a select number of survey items can be informative and provide some guidance about what elements may be critical in the delivery of Woodcraft. However, findings were limited due to the demands of a factor analysis, where a high sample size for a large survey is required. Because the Site coordinator survey provides a rich dataset with a much larger number of items than could be included in the factor analytic approach, a second approach was designed for understanding quality. Referred to as the quality benchmark approach, it uses almost all of the quantitative items on the survey and organizes them into major areas that each has an expected potential best score.

A. CREATING THE QUALITY BENCHMARK SCORES

The Woodcraft Rangers executive and management staff completed the site coordinator survey as a team to identify how they would expect a high quality site to respond. These ratings serve as the benchmark or minimum criteria for a “high quality” site response. For example, sites could indicate the frequency with which they offer “youth leadership activities” and “club culmination activities” on a five-point scale from “not at all” to “at least once a week.” Woodcraft management indicated they would expect a high quality Nvision site to offer youth leadership activities at least once a week and club culmination activities at least once every 2-3 months. For each item, if the site’s response met or exceeded the management team’s benchmark, a point for that item was assigned. No bonus points were given for exceeding the benchmark. Benchmark expectations were the same for both elementary school (ES) and middle school (MS) sites in most cases, with a few exceptions of higher expectations for middle school sites (e.g., more frequent leadership activities).

The survey items were then organized into 13 categories of implementation, reflecting the major areas of program operations and values, (i.e., Cycle Plans, Student Engagement, Parent Involvement). Each category consists of several related survey items and is named based on the content of the items. Once a site’s survey had its benchmark points assigned to each item, the points were totaled within each category and divided by the benchmark total for that category. For example, the benchmark score for the category Cycle Plans is five. If a site met the high quality criteria for each item except one in that category, their score would be 80% (i.e., 4 divided by 5). These scores can be considered reflections of the site’s progress toward what is considered a high quality implementation of the program, with 100% for each category as the goal.

See Appendix D for the site coordinator survey items arranged by benchmark category, along with the item rating expectations and the best possible score for each category. This also includes which benchmarks differ for elementary versus middle schools. Table 9 below presents the list of quality categories with short definitions for each.


Page 16

Table 9. Quality Benchmark Categories from Site Coordinator Survey

Quality Categories

Summary of Category Item Content and Expectations Benchmark Goal #

14. Cycle Plans Cycle plans are used and useful for club activity implementation 5

15. Time Distribution

Students are involved for expected number of minutes in required components of: homework, fitness, nutrition/snack, interest-based club activities, and closing activities/all-together time (ES)

5

16. Club Activities

Club activities offered include all six types of: Sports, Performing arts, Visual arts, Recreation activities, Computers or Technology / multi-media, Leadership opportunities/youth development

6

17. Student Engagement

Students are engaged in appropriate frequencies of additional youth development activities such as sport tournaments, club culmination activities, middle/high school transition activities, etc.

9

18. Club Contributors

Contributors to the delivery of club activities (beyond site coordinator and club leaders) include teacher liaison, activity consultants, traveling specialists (MS), and school personnel

3 (ES) 4 (MS)

19. Ad Hoc Staff Value

Club delivery contributors (same list as 5 above) have appropriately important roles in the selection and delivery of site activities

5 (ES) 6 (MS)

20. Club Selection

Reported staff that help to select the club activities include: regional manager, site coordinator, club leader, youth council, youth participants, principal/school administrator

6

21. Student Involvement

Extent to which student involvement is incorporated by program, such as needs and interests of students considered, students have choice of club activities, leadership opportunities offered, etc.

5 (ES) 6 (MS)

22. Parent Involvement

Extent to which parents are generally involved in the program and involved in specific activities, such as communicating with staff, attending special events, volunteering, etc.

14

23. School Collaboration

Extent to which the site generally collaborates with their host school and engages with school staff in specific communications & activities

8

24. Facilities Access

Extent to which site has access to school facilities as well as the specific types of school spaces available to the program

8

25. Site Coordinator Qualities

Importance of a variety of possible characteristics for a site coordinator to have, including items related to management style, organization skills and staff communication

11

26. Program Staff Qualities

Importance of a variety of possible characteristics for program staff to have, including leadership, organization and student interactions

11


Page 17

B. SUMMARY OF SITES’ QUALITY BENCHMARK SCORES

Reviewing the quality benchmark score data for elementary and middle school sites reveals several trends. First, quality tends to be high across sites in 5 of the 13 benchmark areas at each program level of elementary and middle school. In Tables 10 (elementary sites) and 11 (middle school sites) below, the first five benchmarks listed are those in which a majority of sites achieve a 75% or higher quality rating (indicated both by average benchmark score and number of sites reaching 75% quality score). Three of those top quality benchmarks are the same for elementary and middle school, specifically: Site Coordinator Qualities, School Collaboration, and Program Staff Qualities. For elementary, the two other areas in which a majority of sites reach at least a 75% quality rating include Cycle Plans and Student Engagement. For middle school, the other two top areas are Club Activities and Club Contributors. These findings indicate that most sites are implementing the Nvision program in a few key areas according to the Woodcraft model’s expectations.

Another trend is that the range of quality across sites is quite large for almost all the areas. For example, the average quality score for Student Involvement is high at 74% for elementary sites, yet that ranges from as low as 20% to the highest possible score of 100% (Table 10). The range is wider across sites at the middle school level than at the elementary level. Seven of the thirteen quality categories have a range that starts as low as 0%, with 0% meaning that the site’s responses to how they implement the program in that area does not meet any of the Woodcraft model’s expectations. While the ranges are large, the average scores for both elementary and middle school tend to be above 50% of the benchmark. This combination of findings indicates that there are typically only one or a few sites with such low scores in a given category.

There are only a couple of benchmark categories in which sites’ quality scores are consistently lower than the Woodcraft model’s expectations. For Time Distribution, the average percentage of the benchmark reached is 44% for elementary sites and 38% for middle school sites. This indicates that most sites do not spend the same amount of time in each of the daily required components as expected by the Woodcraft model. The Ad Hoc Staff Value to club delivery is also an area for both program levels where quality tends to be lower than the Woodcraft-set benchmarks.

While these trends are useful to review, the larger question is how quality in each of these areas relates to youth outcomes. That data, in Section V, in conjunction with these trends can help Woodcraft to identify which program areas and sites on which to focus improvement efforts. It can also be helpful in reviewing the quality expectations to create a quality monitoring process. Given that this is an exploratory study, it is the first time the benchmarks have been used and these data can help inform if they are appropriately set or need refining.


Page 18

TABLE 10. QUALITY BENCHMARK SCORES FOR ELEMENTARY SCHOOL SITES

Quality Categories % of Benchmark Score Reached* Number Sites that Reach at least 75% of Benchmark** Average Range

Site Coordinator Qualities 88% 27% - 100% 33

Cycle Plans 82% 40% - 100% 32

School Collaboration 81% 50% - 100% 32

Student Engagement 77% 44% - 100% 29

Program Staff Qualities 77% 27% - 100% 26

Student Involvement 74% 20% - 100% 31

Club Activities 73% 33% - 100% 21

Facilities Access 72% 33% - 83% 27

Club Contributors 64% 25% - 100% 21

Club Selection 64% 17% - 100% 11

Parent Involvement 50% 7% -93% 5

Ad Hoc Staff Value 46% 0% - 75% 7

Time Distribution 44% 20% - 80% 1

* Note: This does not represent a percentage of sites. Rather it represents what percentage of the quality benchmark expectation sites were able to reach, with the goal to reach 100% of the benchmark. ** N=42 sites

TABLE 11. QUALITY BENCHMARK SCORES FOR MIDDLE SCHOOL SITES

Quality Categories % of Benchmark Score Reached* Number Sites that Reach at least 75% of Benchmark** Average Range

Site Coordinator Qualities 86% 27% - 100% 11

Club Activities 83% 50% - 100% 10

School Collaboration 83% 25% - 100% 11

Program Staff Qualities 75% 18% - 100% 10

Club Contributors 75% 0% - 100% 10

Student Engagement 74% 0% - 89% 11

Student Involvement 70% 0% - 100% 9

Cycle Plans 69% 20% - 100% 7

Parent Involvement 69% 14% - 93% 7

Club Selection 67% 0% - 100% 6

Facilities Access 61% 0% - 83% 5

Ad Hoc Staff Value 60% 0% - 100% 6

Time Distribution 38% 0% - 60% 0

* Note: This does not represent a percentage of sites. Rather it represents what percentage of the quality benchmark expectation sites were able to reach, with the goal to reach 100% of the benchmark. ** N=13 sites


Page 19

C. OVERALL QUALITY SCORE

In addition to the category scores, a total score was calculated that included almost all items. The total score for each site was divided by the benchmark total score (100 for elementary school sites, 103 for middle school sites). This percent of benchmark total indicates how close the program is to one composite benchmark. This percent of the expected total is considered the overall quality indicator. The distribution of overall quality scores (reported as a percent of the benchmark) was categorized into three levels: low, medium and high. Category cut-offs were determined by creating levels with a similar number of schools in each. The distribution of schools is presented in Table 12.

For elementary sites, the range of average quality score from low to high quality sites is not that large, with an average of 60% for low quality sites and 74% for high-quality sites. The opposite is true for the middle school sites. Low-quality sites have an average score of 48% while the high-quality sites average an 80% score. There is also an incredible range within the category of low-quality middle school sites, with some sites as low as 23% and others more similar to a medium-quality site at 67%.

TABLE 12. OVERALL BENCHMARK QUALITY SCORE BY LEVEL OF QUALITY

Low Quality Sites Medium Quality Sites High Quality Sites

Elementary (N=42)

Average 59.9% 66.5% 74.2%

Std. Dev. 2.3% 2.7% 2.9%

Range 55% - 62% 63% - 70% 71% - 80%

N 13 16 13

Middle

(N=13)

Average 48.3% 75.2% 80.1%

Std. Dev. 20.5% 2.9% 1.9%

Range 23% - 67% 72% - 79% 80% - 83%

N 4 6 3

Std.Dev=Standard Deviation; N=Number of sites

Woodcraft Rangers: Assessment of Quality Quality and Outcomes: Intro

Page 20

V. RELATIONSHIP OF QUALITY INDICATORS WITH YOUTH OUTCOMES

A. INTRODUCTION TO QUALITY-OUTCOME RESULTS

The purpose of this study is not just to explore ways to assess quality but to specifically explore quality as it relates to youth outcomes. Program quality matters because it is assumed to impact how youth experience and benefit from a program. It is not meant to be a stand-alone goal. For example, if a site has high-quality practices but these do not help youth improve their skills or behaviors, then the purpose of assessing the quality of these becomes less clear. Yet, to date, after-school research in each of these areas has been largely separate. Connecting the two allows program quality to become a more concrete and useful measurement for program monitoring and improvement.

To accomplish this part of the study, multiple regression analyses were performed using the quality factors and benchmarks (referred to as quality indicators when discussing factors and benchmarks together), along with school community sociodemographics and number of days students attended Woodcraft, as predictor variables of positive outcomes in student academic performance and youth development. Regression analyses were performed separately for elementary and middle schools due to their somewhat different nature and focus of the Woodcraft Nvision model. Summary data related to regression models for quality indicators in elementary schools and middle schools can be found in Appendices E through H.

In almost all cases, the desired, or favorable, result is a positive relationship between the quality factor or benchmark and the youth outcome. This indicates that as quality increases, positive youth outcomes increase (greater positive change or higher positive level). The only exception is in the instance of problem behavior. In this case, the outcome variable is scored so that higher scores are reflective of greater problem behaviors. The ideal direction then is negative - as quality increases, problem behaviors decrease. For this study, problem behaviors is primarily defined by school-related issues such as receiving detention or getting sent to the office.

It was not expected that the relationships between the quality indicators and outcomes would be the same for elementary and middle schools, and this was true in most cases. Thus, the findings are presented separately by each school level. Each section summarizes findings only for those relationships where a statistically significant relationship occurred between the quality indicator and outcome when controlling for sociodemographics and program attendance.

Woodcraft Rangers: Assessment of Quality Quality and Outcomes: Elementary

Page 21

B. QUALITY-OUTCOME FINDINGS FOR ELEMENTARY SCHOOLS

Overall, the ability of the quality factors and benchmarks to predict elementary youth outcomes is mixed, showing that the different components of quality are differentially associated with student outcomes. Specifically, there is no one quality indicator favorably (i.e., in the expected direction) associated with all twelve outcomes, and no one outcome associated with all the quality indicators. Rather, each indicator has its own profile of outcome relationships. Table 13 shows all 16 quality indicators (3 quality factors and 13 quality benchmarks) and their relationship to each of the youth outcomes measured for this study. Of the 16 indicators, five (5) have relationships that are only favorable with a sub-set of the youth outcomes. This includes the Ad Hoc Assistance factor and the following benchmarks: Student Engagement, Club Contributors, Access to Facilities and Site Coordinator Qualities. This indicates that when program quality is higher in each of these areas, the more positive are at least one or more of the desired youth outcome results. Other quality indicators either have a combination of negative and positive relationships with outcomes (8), only negative relationships (3), or no relationships (1).

The discrepancies between which outcomes the quality indicators are favorably related to and not help highlight that there is no one quality indicator that can be used to determine effectiveness related to outcomes. This is supported by the findings in the overall quality section. All these components are important to the Nvision model, but at times may compete with each other in terms of their impact on outcomes. For example, youth choice and leadership is a critical component to the Nvision model, yet the higher the quality of an elementary site in this area, the more likely there is an increase in youth problem behaviors. On the other hand, a decrease in problem behaviors is found when quality is high in terms of the roles of staff in the program. Thus, while both youth and staff involvement are important to the model, it seems there would need to be a balance between quality expectations if a consistent impact on decreased problem behaviors is to be seen.

A description of the relationships between each quality indicator and the outcomes follows Table 13.

Woodcraft Rangers: Assessment of Quality Quality and Outcomes: Elementary Sites

Page 22

TABLE 13. ELEMENTARY SCHOOL SITE FINDINGS – SIGNIFICANT RELATIONSHIPS BETWEEN QUALITY AND OUTCOMES

After school

attitude

Make choices

Use academic

skills

Fitness involve-

ment

Problem behavior

Pro-social skills

School attitude

Self-confi-dence

Homework / school

skills

School attend-

ance

Math perform-

ance

ELA perform-

ance

FACTORS Core Elements Cycle Plans ^ ^ ^ Ad Hoc Assistance

X X X X

BENCHMARKS Cycle Plans ^ ^ ^ Time Distribution

^ X ^

Club Activities X ^ ^ ^ X Student Engagement

X

Club Contributors

X X X X X X

Ad Hoc Value X ^ X X ^ Club Selection ^ X X Student Involvement

X X ^ ^

Parent Involvement

^ ^

School Collaboration

X ^

Access to Facilities

X X

Site Coordinator

X X

Program Staff Qualities

X X ^

X Relationship is in desired direction: higher quality is associated with a more positive level or positive change in outcome (negative change for problem behavior) ^ Relationship is not in desired direction: higher quality is associated with more negative occurrences of outcomes (positive increase in problem behaviors)


Page 23

1. Quality Factors

Ad Hoc Assistance – Ad Hoc Assistance is favorably related to 4 of the 12 outcomes. That is, as ratings increase for the value of parent volunteers, youth volunteers and traveling specialists in activity delivery, greater improvement is seen in youth’s homework/school skills and ELA CST academic scores. Higher values are also associated with youth having more positive attitudes toward the after-school experience and higher involvement in fitness activities.

Cycle Plans – The Cycle Plans factor is related to three outcome areas, though not in the desired direction. As Cycle Plans scores increase, ELA CST scores decrease, problem behaviors increase and attitudes toward the after-school experience are more negative. These findings are not consistent with the idea that higher quality implementation of cycle plans is associated with positive student outcomes. It may be possible that the current way in which cycle plans are implemented at elementary sites entails too much structure and not enough flexibility for ensuring full youth engagement.

Core Elements – Core elements is not significantly associated with any of the youth outcomes.

2. Quality Benchmarks

Cycle Plans – The Cycle Plan benchmark follows the same pattern as the Cycle Plan factor, not surprisingly since both the factor and this category share many of the same items. The more closely site ratings are to the quality benchmark (i.e., a higher rating), the more ELA CST scores decrease, problem behaviors increase, and the more negative are after-school attitudes.

Time Distribution – Distribution of minutes across program segments (e.g., homework time, club activity time) that are closer to the benchmark rating is associated with an increase in pro-social skills but also a decrease in school attendance and increase in problem behaviors. This benchmark may need more exploring given only one of the elementary sites reached even 75% of the expectation (See Section IV, Table 10).

Club Activities – The more likely a site is to meet the quality benchmark of providing the six main types of club activities, the more students improve on CST Math scores and the higher their interest in after-school activities. However, higher ratings on Club Activities is also associated with declines in homework/school skills, pro-social skills and school attitudes.

Student Engagement – The higher the benchmark rating for how frequently students are engaged in additional activities beyond the regular daily schedule, the higher is students’ involvement in using academic skills during activities.

Club Contributors – Club Contributors is the quality indicator that is favorably related to the most number of youth outcome areas. Sites that are closer to the benchmark of who contributes to the delivery of club activities have participants who: increase school attendance, increase pro-social skills, increase use of academic skills in activities, increase positive school attitudes, decrease problem behaviors and have a more positive attitude toward after-school. This could indicate that the presence of adults, beyond the site coordinator and club leaders, is one of the most important quality indicators to consider for the elementary program.


Page 24

Ad Hoc Staff Value – Ratings closer to the benchmark (i.e., higher ratings) are associated with decreased problem behaviors, increased homework/school skills, and a more positive attitude toward the after-school programs. However, it is also associated with declines in CST Math scores. Note that this benchmark is similar to the Ad Hoc Assistance factor but not the same. Both concern the value of contributions to club delivery by people other than the site coordinator and club leaders. The benchmark version includes staff only (traveling specialists, activities consultants, teacher liaison, other school personnel), while the factor includes traveling specialists and youth and parent volunteers. The factor and benchmark are similar in terms of outcomes in that both are related to improved homework/school skills and a more positive after-school attitude. In line with the Club Contributors benchmark, this also seems to suggest the importance of adults in the elementary program for decreasing problem behaviors and creating a more positive after-school experience.

Club Selection – When persons associated with the selection of club activities are closer to the benchmark, students improve homework/school skills and decrease problem behaviors, but also have lower involvement in making choices. The latter finding makes sense given that the benchmark goal is to have six different groups of people (key WR staff, youth, parents, school administrators) involved in selecting activities. The more people involved in selecting clubs, the less likely students may see themselves as being involved in the process even if they are.

Student Involvement – The more frequently students have opportunities to be involved in the program operations, the higher their involvement in making choices and in using academic skills during activities. On the other hand, they more likely they are to experience a decrease in homework/school skills and an increase problem behaviors. This benchmark is a good example of how different program components may need to be balanced. It may be that the more autonomy youth experience through after-school leadership opportunities inadvertently results in their expressing autonomy during the school day - where it may not be expected or appropriate and result in problem behaviors such as being sent to the school office.

Parent Involvement – This factor is not favorably associated with any of the outcomes, and is associated with increased problem behaviors and lower interest in after-school activities. That is, the more a site meets the benchmark for involving parents in program communications and activities, the more students increased acting out during the school day and the less engaged they felt in the after-school program.

School Collaboration – Higher quality ratings of School Collaboration are related to a higher level of students making choices in the program, but also associated with an increase in problem behavior. It is possible that more collaboration with the school is required at the elementary school level when students exhibit more problematic behaviors.

Access to Facilities – Access to school facilities is one of the indicators that has only favorable associations with outcomes rather than mixed. The better a site’s access to appropriate school spaces for after-school activities, the better behaved students are during the school day and the more likely they are to move up a proficiency level in Math. The relationship with problem behaviors may be due in part to the consistency between how space is used during the school


Page 25

day and after-school – since expectations of behavior are related to space (e.g., quiet study time in a classroom versus active play on the field or playground).

Site Coordinator Qualities – This is another one of the four indicators that has only favorable associations with outcomes. When Site Coordinator Qualities are rated similar to the benchmark, students’ problem behaviors decrease more and their after-school attitude is more positive. It seems the balance of management and inter-personal skills expected in this benchmark is effective for simultaneously reducing problem behaviors while still keeping students positive about their experience.

Program Staff Qualities – Positive effects are seen when Program Staff Qualities are similar to the benchmark in the areas of level of use of academic skills and involvement in making choices. However, declines in CST Math scores are seen among sites where Program Staff Qualities are closer to the benchmark. For this benchmark, the quality expectation is on staff having skills related to engaging students (e.g., enthusiasm, initiative and creativity, responsiveness to needs) somewhat more than it is on their formal experience or education.

3. Desired Outcomes by Quality Benchmarks

To illustrate findings from a different perspective, Table 14 presents the statistically significant associations between desired student outcomes with quality benchmarks for elementary sites. Outcome areas in which at least four of the quality benchmarks have a favorable impact are: decreased problem behaviors, higher level of use of academic skills in activities, and higher level of involvement in making choices. If an outcome is not listed in the table, it indicates that there are no favorable associations between that outcome and any of the benchmarks.

TABLE 14. DESIRED STUDENT OUTCOMES ASSOCIATED WITH QUALITY BENCHMARKS FOR ELEMENTARY SCHOOL PROGRAMS

Desired Student Outcome Significantly Associated Quality Benchmarks

Increased CST Math scores Club Activities, Access to Facilities

Increased school attendance Club Contributors

Increased homework/school skills Ad Hoc Staff Value, Club Selection

Increased pro-social skills Time Distribution, Club Contributors

Higher level of use of academic skills in activities

Student Engagement, Club Contributors; Student Involvement, Program Staff Qualities

Increased positive attitude toward school

Club Contributors

More positive attitude toward after-school experience

Ad Hoc Staff Value, Site Coordinator Qualities

Higher level of involvement in making choices

Club Activities, Club Contributors, Student Involvement, School Collaboration, Program Staff Qualities

Decreased problem behaviors Club Contributors, Ad Hoc Staff Value, Club Selection, Access to Facilities, Site Coordinator Qualities


Page 26

4. Overall Benchmark Quality Rating

Analyses were also conducted with the overall benchmark ratings created for the sites. At elementary sites, this overall quality rating does not appear to be helpful in understanding or predicting student academic outcomes. Regression analyses show that the overall score is not related to any youth outcome indicator. Another way of looking at the overall score data compared the low, medium and high quality sites. Most of the results are not significant (See Table 15). Of the few significant findings, none are in the expected direction. Compared to low and medium quality sites, participants at high quality sites tend to show less change in homework/school skills, less change in self-confidence, an increase in problem behaviors, lower levels of positive attitudes about the after-school experience and lower involvement in fitness.

These trends may be due in part to how close the average quality ratings are for the three categories (60% for low, 67% for medium, 74% for high). In general, these findings suggest that for elementary school sites, overall quality may not be a useful tool when trying to improve program operations to affect student outcomes. Rather, specific quality indicators should be targeted to affect the desired positive change in students.

TABLE 15. UNIVARIATE ANALYES OF VARIANCE MODELS – AVERAGE YOUTH OUTCOME SCORES BY OVERALL QUALITY RATING, ELEMENTARY SCHOOLS

Level of Significance

Low Quality Average Score

Medium Quality Average Score

High Quality Average Score

Change CST ELA Scores NS NS NS NS

Change CST Math Scores NS NS NS NS

Change in percent of days attended school

NS NS NS NS

Positive attitudes about after school experience (2,207)

NS NS NS NS

Positive attitudes toward school

NS NS NS NS

High involvement in making choices

NS NS NS NS

High use of academic skills in activities

NS NS NS NS

Change in homework/school skills (n=2,174)

p = .072 0.09 0.17 0.05

Level of fitness involvement (n=2,219)

p = .012 3.37 3.34 3.27

Increased self-confidence (n=2,216)

p = .008 0.11 0.26 0.06

Increased pro-social skills NS NS NS NS

Increased leadership skills NS NS NS NS

Decreased problem behaviors (n=2,211)

p = .028 -0.13 -0.24 0.07

Woodcraft Rangers: Assessment of Quality Quality and Outcomes: Middle School Sites

Page 27

C. QUALITY-OUTCOME FINDINGS FOR MIDDLE SCHOOL SITES

Similar to the elementary site findings, the ability of the quality factors and benchmarks to predict middle school youth outcomes is mixed, indicating the different components of quality are differentially associated with student outcomes. Unlike elementary school sites findings, there are many more favorable relationships found between the quality indicators and outcomes. As Table 16 shows, of the 16 quality indicators (3 quality factors and 13 quality benchmarks), seven (7) have only favorable relationships with a sub-set of youth outcomes. This includes the Cycle Plans factor as well as the following benchmarks: Cycle Plans, Student Involvement, Parent Involvement, School Collaboration, Facilities Access and Site Coordinator Qualities. This indicates that when program quality is higher in each of these areas, the more positive are at least one or more of the desired youth outcome results. Other quality indicators either have a combination of negative and positive relationships with outcomes (6), only negative relationships (2), or no relationships (1). While there is no one indicator favorably associated with all thirteen middle school outcome indicators, five of the indicators are associated with at least half of the outcomes.

The discrepancies between which outcomes the quality indicators are favorably related to and not help highlight that there is no one quality indicator that can be used to determine effectiveness related to outcomes. In the case of middle school sites, quality does seem to have a more important impact on outcomes than seen at elementary sites. Six of the thirteen outcome areas are each favorably associated with at least six of the quality indicators. In addition, significant findings are found between overall site quality and a few key outcomes.

A description of the relationships between each quality indicator and the outcomes follows Table 16.


Page 28

TABLE 16. MIDDLE SCHOOL SITE FINDINGS – SIGNIFICANT RELATIONSHIPS BETWEEN QUALITY AND OUTCOMES

After school

attitude

Make choices

Use academic skills

Fitness involve-

ment

Problem behavior

Pro-social skills

School atti-tude

Self-confi-dence

Leader-ship skills

Home-work / school skills

School attend-

ance

Math perform

-ance

ELA perform

-ance

FACTORS Core Elements X X X X X ^ X Cycle Plans X X X X X Ad Hoc Assistance

BENCHMARKS Cycle Plans X X X X Time Distribute X X ^ Club Activities ^ Student Engagement

^

Club Contributors

^

X

Ad Hoc Value X X ^ X X ^ X Club Selection ^ ^ ^ X Student Involvement

X X X

X X X

Parent Involvement

X

School Collaboration

X X X

X X X

Facilities Access X X X X X X X X

Site Coordinator

X X X

X X X

Program Staff ^ X X X Relationship is in desired direction: higher quality is associated with a more positive level or positive change in outcome (negative change for problem behavior) ^ Relationship is not in desired direction: higher quality is associated with more negative occurrences of outcomes (positive increase in problem behaviors)


Page 29

1. Quality Factors

The quality factors’ relationships with middle school youth outcomes are the direct opposite of what is found for the elementary sites. At the elementary level, Ad Hoc Assistance is favorably related to a few of the outcomes, while Cycle Plans is not and Core Elements has no relationships. For middle school, both the Core Elements and Cycle Plans factors appear to be influential with several of the outcomes, while Ad Hoc Assistance is not. The implication of these findings may be that ensuring overall program quality and structure is more important at the middle school level than at the elementary level.

Ad Hoc Assistance – Ad Hoc Assistance is not significantly associated with any of the youth outcomes. It appears as if at the middle school level, the value a site places on the role of traveling specialists and youth and parent volunteers in program delivery is not related to student outcomes.

Cycle Plans – The Cycle Plans factor is favorably associated with 5 of the 13 youth outcomes. This indicates that sites with higher quality scores for how they use cycle plans to guide activities have more students with: higher involvement in using academic skills in activities, higher involvement in fitness, decreased problem behaviors, increased school attitude, and increased performance on the Math CST. Unlike elementary sites, these findings are consistent with the idea that higher quality implementation of cycle plans are associated with positive student outcomes. It may be that the structure that cycle plans help bring to the daily program has more importance at the middle school level than at the elementary school level.

Core Elements – Core Elements is favorably associated with 6 of the 13 outcomes, including: attitude toward after-school program, use of academic skills in activities, fitness involvement, problem behaviors, school attitude and homework/school skills. Four of these outcomes are the same as those related to Cycle Plans (academic skills use, fitness, problem behaviors and school attitude). The Core Elements factor includes three site coordinator survey items and each represents a different component essential to the Nvision model (other factors tend to consist of similar items). The three items are school collaboration, youth involvement in decision-making and adherence to the site schedule. There is one outcome that is not favorably related to Core Elements - leadership skills.11

2. Quality Benchmarks

Cycle Plans – The Cycle Plan benchmark follows the same pattern as the Cycle Plan factor, not surprisingly since both the quality factor and this category share many of the same items. The benchmark is favorably associated with four of the same outcomes as the factor: use of academic skills in activities, involvement in fitness, problem behaviors and school attitude. The only outcome that is associated with the factor but not this benchmark is change in Math CST. This difference between the factor and benchmark may be explained by the benchmark including two more questions than the factor and/or to how the factor versus the benchmark

11 Leadership skills is not favorably associated with any of the quality factors or benchmarks. It is possible that it is

more a measurement issue related to the items’ response scale than to a lack of relationships.


Page 30

works. The factor yields a score based on the actual responses, while the benchmark checks the response against an established benchmark and assigns the score according to met or not.

Time Distribution – Higher scores on Time Distribution of after-school program activities is associated with decreased school attendance, similar to the finding at elementary school sites. Favorable relationships are also found at the middle school level between how well sites meet expectations for how much time is spent in each type of daily activity and a higher level of youth involvement in using academic skills in activities and more positive youth attitudes toward school. Like Cycle Plans, Time Distribution is a benchmark related to program structure.

Club Activities – Sites that are closer to meeting the benchmark of providing all six main types of Club Activities are more likely to have students whose CST Math scores declined. This is opposite of the finding for elementary sites where CST Math scores increased with higher quality in this area.

Student Engagement – The higher the benchmark rating for how frequently students are engaged in additional activities beyond the regular daily schedule, the lower is their involvement in using academic skills during activities. This is also opposite of the elementary finding where youth are more likely to be involved in using academic skills the more their site meets the quality benchmark for additional activities.

Club Contributors – Middle school sites that are closer to the benchmark related to who contributes to the delivery of club activities have students who are more likely to increase their CST ELA scores. This benchmark is also associated with decreases in leadership skills, and is not associated with any of the other outcomes (unlike elementary where it is favorably associated with several outcomes). It may be that the engagement of staff beyond the coordinator and club leaders is important for building language skills though not as important for impacting the youth development outcomes.

Ad Hoc Staff Value – Higher quality ratings of the Value of Ad Hoc Staff (traveling specialists, activities consultants, teacher liaison and other school personnel) are associated with a number of positive student outcomes. These include: increase in homework/school skills, decreased problem behaviors, higher level of involvement in using academic skills in activities, increased positive attitude toward school, and more positive attitude toward after-school. Higher scores on this quality benchmark category are also related to less involvement in fitness skills and decreased leadership skills. It is possible that the site coordinator’s value assessment of these ad hoc staff is also a proxy indicator for the quality of the staff specific to their site.

Of all the quality indicators, this is the one in which the relationship to outcomes is most similar between elementary and middle school sites. For both elementary and middle schools, the Value of Ad Hoc Staff is favorably related to the three of the same outcomes: after-school attitude, problem behavior and homework/school skills.

Note that this benchmark is similar to the Ad Hoc Assistance factor but not the same. Both concern the value of contributions to club delivery by people other than the site coordinator and club leaders. The benchmark version includes staff only, while the factor includes traveling


Page 31

specialists and youth and parent volunteers. For middle school sites, the factor and benchmark do not share any similarities in their relationship to outcomes (the factor has no relationships).

Club Selection – Higher scores on Club Selection are associated with increased CST ELA scores. That is, the more sites meet the benchmark of engaging six different groups of people (key WR staff, youth, parents and school administrators) in selecting club activities, the more likely students are to improve their English/Language Arts performance. However, higher Club Selection quality scores are associated with a number of negative outcomes, including: decreased homework/school skills, increased problem behaviors and a lower level of involvement in using academic skills in activities. This benchmark is a good example of how quality expectations may need to be balanced depending on the desired outcomes.

Student Involvement – Student Involvement is one of the four benchmarks associated with at least six outcomes. The better the sites meet the benchmark for the frequency with which they provide students with opportunities to be involved in the program operations, the more youth: increased CST ELA scores, increased CST Math scores, increased homework/school skills, decreased problem behaviors, increased positive attitude toward school and had higher levels of fitness involvement. These findings support the core belief behind the Nvision middle school program – which is that youth development and leadership represent the best pathway to changing youth’s behavioral and academic outcomes.

Parent Involvement – Higher scores on the Parent Involvement benchmark are related to improved CST ELA scores. This indicates that the more a site meets the Nvision model for how to engage parents, the more likely students improve their English/Language Arts performance. This relationship, however, may also be indicating the general level of parent involvement in students’ lives in that school community.

School Collaboration – Similar to the Student Involvement benchmark, School Collaboration is related to six desirable student outcomes: increased CST ELA scores, increased CST Math scores, increased homework/school skills, decreased problem behaviors, more positive attitude toward after-school experience and higher involvement in fitness. Five of those six outcomes are the same as those found for Student Involvement – CST ELA and Math scores, homework/school skills, problem behaviors and fitness. This finding seems to indicate that engaging the school community in appropriate and appropriately frequent ways is particularly important for impacting youth outcomes. This may also be a proxy indicator of the quality of the school staff. That is, part of the reason some Woodcraft sites have better collaboration levels may be due in part to the buy-in and commitment of the school staff to integrating school-day and after-school efforts.

Access to Facilities – Higher scores on the Access to Facilities benchmark are favorably associated with eight of the outcomes, with four of those outcomes being the same as those associated with Student Involvement and School Collaboration. Those four outcomes are: increased CST ELA scores, increased homework/school skills, decreased problem behaviors and higher level of fitness involvement. The other four outcomes with which Access to Facilities are related include: improved school attendance, higher involvement in using academic skills in


Page 32

activities, increased positive attitude toward school and more positive attitude toward after-school experience. At both the middle and elementary level, the Access to Facilities benchmark emerges as one of the most important one given the number of outcomes with which it is related.

Site Coordinator Qualities – The Site Coordinator benchmark is the fourth one that is favorably associated with at least six outcome areas. Like Student Involvement, School Collaboration and Facilities Access, the closer Site Coordinator Qualities are to the benchmark, the more likely the participants at that site have: increased CST ELA scores, increased homework/school skills, decreased problem behaviors and higher level of fitness involvement. It is also favorably associated with improved school attendance and higher use of academic skills in activities. This benchmark’s relationship to middle school outcomes is similar to its relationship to elementary school outcomes in that it is associated with a decrease in problem behaviors at both school levels.

Program Staff Qualities – Higher ratings on Program Staff Qualities are associated with increased CST ELA scores and increased school attendance, but decreased positive attitude toward school. The quality expectation for this benchmark is on staff having skills related to engaging students (e.g., enthusiasm, initiative and creativity, responsiveness to needs) somewhat more than it is on their formal experience or education. Given its relationship to CST ELA scores, it is worth exploring this benchmark in more depth to better understand how this connects, such as by conducting in-depth interviews with Woodcraft sites that reach 100% of the benchmark.

3. Desired Outcomes by Quality Benchmarks

To illustrate findings from a different perspective, Table 17 presents the statistically significant associations between desired middle school student outcomes with quality benchmark categories. There are nine outcome areas with favorable associations to at least one quality benchmark. Unlike elementary site findings, most of the middle school outcomes are associated with several quality benchmarks, versus just one or two. This may provide another indication that adherence to the whole program model at the middle school level is more critical than at the elementary level.

If an outcome is not listed in the table, it indicates that there are no favorable associations between that outcome and any of the quality benchmarks. This is the case for 4 of the 13 outcomes. There are no relationships between any of the benchmarks and the outcome areas of: higher involvement in making choices, pro-social skills, and self-confidence. As mentioned in the earlier section, the leadership outcome has only negative associations with its associated indicators.


Page 33

TABLE 17. DESIRED STUDENT OUTCOMES ASSOCIATED WITH QUALITY BENCHMARKS FOR MIDDLE SCHOOL PROGRAMS

Desired Student Outcomes Significantly Associated Quality Benchmarks Increased CST ELA scores Club Contributors, Club Selection, Student Involvement,

Parent Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities, Program Staff Qualities

Increased CST Math scores Student Involvement, School Collaboration

Increased school attendance Access to Facilities, Site Coordinator Qualities, Program Staff Qualities

Increased homework/school skills Ad Hoc Staff Value, Student Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities


Cycle Plans, Time Distribution, Ad Hoc Staff Value, Access to Facilities, Site Coordinator Qualities

Improved school attitude Cycle Plans, Time Distribution, Ad Hoc Staff Value, Student Involvement, Access to Facilities

More Increased positive attitude about after-school experience

Ad Hoc Staff Value, School Collaboration, Access to Facilities


Cycle Plans, Student Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities

Decreased problem behaviors Cycle Plans, Ad Hoc Staff Value, Student Involvement, School Collaboration, Access to Facilities, Site Coordinator Qualities

4. Overall Benchmark Quality Rating

Analyses were also conducted with the overall benchmark ratings created for the sites. Unlike elementary school where no relationships are found, regression analyses for middle school sites show that higher overall quality scores are related to three youth outcomes: increased CST ELA scores, decreased problem behaviors and increased positive attitude toward school.

Another way of looking at the overall score data compared the low, medium and high quality sites. This also shows differences in outcomes by level of overall quality (See Table 18). An increase in CST ELA scores is seen in the high quality group and less negative change in CST Math scores is seen in this group, compared to the low and medium quality sites. Students in the high quality group also appear to attend school slightly more than those in the medium and low quality groups, noted by a decrease in the percent of days attended school. Additionally, higher quality sites show increases in youth’s school attitudes, although they also show lower use of academic skills in activities. These findings point to some utility in understanding overall quality of middle school sites, along with the different components of quality.


Page 34

TABLE 18. UNIVARIATE ANALYES OF VARIANCE MODELS – AVERAGE YOUTH OUTCOME SCORES BY OVERALL QUALITY RATING, MIDDLE SCHOOLS

Level of Significance

Average Score for Low Quality

Sites

Average Score of Medium Quality

Sites

Average Score for High Quality Sites

Change CST ELA Scores (n=1,856)

p = .009 -0.02 -0.01 0.12

Change CST Math Scores (n=1,844)

p = .000 -0.20 -0.43 -0.08

Change in percent of days attended school (n=1,755)

p = .000 -13% -11% 0.2%

Attitudes about after school experience

NS NS NS NS

Positive attitudes toward school (n=354)

p = .020 3.84 3.57 4.41

High involvement in making choices

NS NS NS NS

High use of academic skills in activities (N=342)

p = .000 2.98 3.15 2.56

Change in homework/school skills

NS NS NS NS

High involvement in fitness NS NS NS NS

Increased self-confidence NS NS NS NS

Increased pro-social skills NS NS NS NS

Increased leadership skills NS NS NS NS

Decreased problem behaviors

NS NS NS NS

Woodcraft Rangers: Assessment of Quality Conclusions

Page 35

VI. CONCLUSIONS

The measurement of quality in after-school programs is becoming more important nationally. Demands for accountability and continued positive outcomes are critical to securing future funding. Measuring the quality of a program can be challenging. Knowing what to measure and how to measure it continues to be an area of development and growth in after-school program research and evaluation. There is no one agreed upon model of quality primarily due to the variety of program goals, objectives, structure, and activities within after-school models.

This study was an initial attempt at understanding quality of the Woodcraft Rangers Nvision after-school program as it relates to youth outcomes. Since this study was exploratory, two approaches, factor analysis and benchmark categorization, were used. Of the two approaches, the benchmark approach of comparing sites’ responses to a management-established quality expectation is more useful because it includes more content areas than can be included in the factor approach. However, the factor analysis is important in that it produced program areas consistent with the described program model and provides credibility to the site coordinator survey as a good measure of the Woodcraft Nvision program model.

In both approaches, some of the quality indicators are better at predicting positive outcomes than others, but none of the indicators are positively associated with all the outcomes. Using an overall score is even less predictive of outcomes, especially at the elementary level. These findings suggest that overall quality may not be a useful tool when trying to improve program operations with the intent of affecting student outcomes. Rather, more specific areas should be assessed and targeted to affect positive change in students.

Focusing on the specific quality areas shows that several middle school student outcomes appear to be influenced by multiple quality indicators. Improvements in English/Language Arts standardized test scores and decreases in problem behavior are related to at least eight quality areas for middle schools. In addition, higher levels of involvement in fitness activities, improvements in positive attitudes toward school and more use of academic skills in activities are related to at least five quality areas. The most critical quality indicators for middle school programs potentially include: Access to Facilities, School Collaboration, Student Involvement and Site Coordinator Qualities. Cycle Plans and the Value of Ad Hoc Staff are also important.

The findings for elementary school sites are not as strong or as consistent as those found for the middle school sites. This may be a result of a greater number of sites such that variability in program implementation and approach differ beyond the items measured in the survey. Or, student outcomes for younger students may be less influenced by quality as it has been defined and examined in this study. Of note, a majority of the regression models are statistically significant, but it is the control variables - school sociodemographics and number of days attended Woodcraft - that influenced the outcome so much that the quality indicator does not make enough contribution to be a significant predictor. At elementary sites, key quality areas that have the most relationships to favorable student outcomes include: Club Contributors (to program delivery), Access to Facilities, and Site Coordinator Qualities. Of the factors, Ad Hoc Assistance is also associated with desired outcomes. There are three elementary student

Woodcraft Rangers: Assessment of Quality Conclusions

Page 36

outcome areas that have at least four quality indicators favorably associated with them: decreased problem behavior, higher involvement in making choices, and more use of academic skills in activities.

The results of this study can be used to further refine the program quality expectations for elementary versus middle school sites. For example, cycle plans are a critical component of the Nvision model. Yet, it had opposite relationships with elementary and middle school outcomes. It is possible that cycle plan development is an important activity for program staff, but the quality of this activity is less influential for elementary school students. It may be that stricter adherence to cycle plans may not be the best approach for elementary school students, yet that structure is important for middle school students. Furthermore, it may be that quality measurements need to be adjusted not only in terms of the expectations by site type, but even in the importance of the categories by elementary versus middle school sites.

The importance of this study for Woodcraft Rangers, and potentially for the field at large, is the confirmation that after-school program quality needs to be understood and defined in terms of outcomes. Given the finding that different aspects of quality impact different outcomes, the next step is to explore how to balance or refine quality expectations so that the program’s priority outcomes are achievable. The study also helps identify a process for how to measure quality efficiently and accurately while producing program improvement information that is “actionable” for staff. This addresses Woodcraft’s stated need to begin developing a data-based quality monitoring tool. In addition, the benchmark process may be of use to other after-school providers, even though specific categories and benchmarks will differ. The model can potentially work for any provider looking for a way to assess quality that is not too resource-heavy, relies on well-informed input and engages staff in the improvement process.

Woodcraft Rangers: Assessment of Quality Recommendations

Page 37

VII. RECOMMENDATIONS Study findings suggest that continued quality monitoring may be important in understanding student outcomes. It is recommended that Woodcraft Rangers consider the following areas in the future management and monitoring of program sites:

Use these exploratory results to help establish an ongoing quality improvement plan that will monitor fidelity to the Woodcraft model.

o Collect annual or bi-annual assessments and compare site responses to benchmarks, track progress and change, and set goals.

Incorporate student outcomes as a measure of quality. o Student outcomes are already a part of the Nvision logic model. However,

examining how students are performing at each site may help to motivate and focus program staff to maintain fidelity to the Woodcraft model.

Consider further exploration of program activities and outcomes with students, staff, and parents by using qualitative methods such as interviews and focus groups.

o Program staff may be able to conduct small group meetings with students, staff, and parents to explore issues identified by the study to further understand how the program can impact students and what improvements may be necessary.

Identify best practices among Woodcraft sites, measure their impact, and encourage all sites to use the successful practices.

o Sites that perform closest to the benchmark can be visited to gather information about how they administer the program. Their practices can be shared with other sites that appear to have lower quality implementation.

Re-visit expected student outcomes and their relationship to key aspects of the Woodcraft Nvision model rather than link outcomes to the entire program, and test new models if possible.

o Some student outcomes did not appear to be related to quality at all (e.g., leadership skills, self-confidence), perhaps because the survey is not sensitive enough to detect change or because activities are not targeted enough. Both program activities and youth survey measurements should be reviewed to determine where changes may be helpful to make.

Woodcraft Rangers: Assessment of Quality Appendix A

Page 38

APPENDIX A – YOUTH DEVELOPMENT OUTCOMES MAPPED TO YOUTH SURVEY ITEMS ELEMENTARY SCHOOL STUDENT OUTCOMES MAPPED TO YOUTH SURVEY ITEMS

Outcome Indicator

Elementary Site Youth Survey Item(s) Survey Time Point(s) & Response Scale


I felt safe when I was at Woodcraft. I liked going to Woodcraft. The staff was helpful when I had homework questions. I felt like I belonged in Woodcraft. The staff helped me feel like I can do a good job.

End-Year 4-point scale

Higher involvement in making choices

I helped plan what we do in the clubs. I got to choose what I wanted to do.



I had to count or use math during club activities. I read or wrote during club activities.



How often did you do more active play (like running, dancing, team games or sports)?


Improved homework and school skills

How often do you feel good about your grades? How often do you do the best job you can on your homework? How often do you understand how to do your homework? How often do you turn all of your homework in on time? How often do you follow the rules at school?

Baseline & End-Year 4-point scale


How often do you like school?


Improved self-confidence

How often do you feel good about yourself?


Improved pro-social skills

How often do you feel like you have friends that you really like? How often do you try to help someone else? How often do you think about something before you do it?



How often do you make fun of other kids? How often do you get sent to the office because you are in trouble? How often do you lose recess time because you are in trouble?


Woodcraft Rangers: Assessment of Quality Appendix A

Page 39

MIDDLE SCHOOL STUDENT OUTCOMES MAPPED TO YOUTH SURVEY ITEMS

Outcome Indicator

Youth Survey Item(s) Survey Time Point(s) & Response Scale


I felt safe when I was at Woodcraft. I liked going to Woodcraft. The staff was helpful when I had homework questions. I felt like I belonged in Woodcraft. The staff helped me feel like I can do a good job.

End-Year 5-point scale (never to always)

Higher involvement in making choices

I helped plan what we do in the clubs. I got to choose what I wanted to do.



I used math during club activities. I read or wrote during club activities.



I did activities that helped me take better care of myself physically.


Improved homework and school skills

During a typical school week, I… …felt good about my grades. …did the best job I could on my homework. …understood how to do my homework assignments. …turned all of my homework in on time to my teachers. …kept working on my homework even when it was hard.

Baseline & End-Year 6-point scale (0 to 5 days)


During a typical school week, I looked forward to coming to school.


Improved self-confidence

During the last 30 days, I felt confident in myself.

Improved pro-social skills

During the last 30 days, I… …solved a problem I was having without arguing or fighting. …thought about the consequences of my actions before I did them. …had at least one good friend I trusted. …felt good about the decisions I made for myself. …tried to help someone else. …felt accepted by my groups of friends.

Baseline & End-Year 6-point scale (never to always)


During a typical school week, I… …received detention. …got sent to the office because I was in trouble. …skipped school or class.


Improved leadership skills

Were you a leader for an activity at school or in your community? Did you help organize or plan an activity or even at your school on in your community?

Baseline & End-Year 3-point scale (no, once or twice, often)

Woodcraft Rangers: Assessment of Quality Appendix B

Page 40

APPENDIX B – SCHOOL COMMUNITY SOCIODEMOGRAPHIC CORRELATION ANALYSES

Correlation analyses were conducted using student academic outcomes and school community sociodemographic information. Elementary and middle schools were examined separately. Pearson’s R, a measure of correlation or how strongly associated two variables are, is presented for each pair of relationships. A correlation may range from -1 to 1. Values that are insignificant suggest that there is no relationship. Values close to 0, even if statistically significant, suggest that there is a relationship but the strength is fairly weak.

For elementary school sites, significant relationships were found. Median family income, the percent of high school graduates in the zip code, and the percent in the labor force all have a positive association with CST scores, while the percent of families below poverty level was negatively associated with academic performance (i.e., the greater percent of families below poverty level, the lower the CST score). Of note, the percent of persons using a language other than English in their home was not generally associated with CST scores, with a single exception for CST ELA scores in Spring 2009.

Relationships between sociodemographics and school attendance were mixed. The percent of families below poverty level was negatively associated with attendance in both schools years and the percent of persons using a language other than English as their primary language at home was positively associated with a higher rate of days attended.

SCHOOL SOCIODEMOGRAPHICS AND STUDENT ACADEMIC PERFORMANCE – ELEMENTARY SCHOOLS12 2008 CST

ELA 2009 CST

ELA 2008 CST

Math 2009 CST

Math Percent of

days attended

2008

Percent of days

attended 2009

Percent of families below poverty level

-.128 -.089 -.088 -.083 -.081 -.054

Percent using a language other than English in their home

NS -.033 NS NS .088 .125

Median family income

.101 .102 .069 .100 NS NS

Percent high school graduates

.140 .062 .117 .050 .039 NS

Percent in labor force

.117 .103 .085 .104 NS NS

12 All values reported in the table are statistically significant (p<.05). Where relationships were not statistically

significant, NS (not significant) is reported.

Woodcraft Rangers: Assessment of Quality Appendix B

Page 41

For middle school sites, generally the same significant relationships were found. Median family income, the percent of high school graduates in the zip code, and the percent in the labor force all have a positive association with CST scores with three exceptions: a negative relationship between the percent of high school graduates and attendance in academic year 2008, and two insignificant relationships between median family income and percent in labor force with attendance in 2008. The percent of families below poverty level was negatively associated with academic performance and attendance in the 2009 academic year but not in 2008.

Unlike the elementary school data, the percent of persons using a language other than English in their home was positively associated with CST scores. That is, a higher percentage of persons in the area who speak a language other than English is associated with higher CST ELA and Math scores.

SCHOOL SOCIODEMOGRAPHICS AND STUDENT ACADEMIC PERFORMANCE – MIDDLE SCHOOLS13 2008 CST

ELA 2009 CST

ELA 2008 CST

Math 2009 CST

Math Percent of

days attended

2008

Percent of days

attended 2009

Percent of families below poverty level

-.156 -.147 -.118 -.147 NS -.040

Percent using a language other than English in their home

.053 .049 .075 .056 .092 .061

Median family income

.140 .132 .098 .131 NS .031

Percent high school graduates

.097 .103 .082 .142 -.035 .028

Percent in labor force

.055 .064 .068 .094 NS .035

Overall, these findings suggest that the sociodemographics of the community that the school serves should be considered when examining student academic outcomes. The strength of the correlations described above are low (all are between -.2 and .2), but these patterns indicate the need to consider the impact of these variables on student outcomes.

13 All values reported in the table are statistically significant (p<.05). Where relationships were not statistically

significant, NS (not significant) is reported.

Woodcraft Rangers: Assessment of Quality Appendix C

Page 42

APPENDIX C – FACTOR ANALYTIC APPROACH DETAIL

Before analyses could be completed, the available data had to be reviewed to determine which items were appropriate for this statistical technique. A number of items from the site coordinator survey had to be eliminated due to their dichotomous response types (i.e., yes/no, or absent/present). Dichotomous responses often result in meaningless factors because factor analysis relies on data that are interval; that is, has three or more categories or levels of responses that have a ranking from high to low. Once these items were removed, a factor analytic model was attempted. However, there were too many items in the remaining dataset for the number of cases available, so a stable model could not be created. Items were removed, beginning with items that appeared to be highly related to one another (i.e., the extent to which students are involved, the extent to which parents are involved, and the extent to which school staff are involved). Once these items were removed, a factor analytic model was created. The initial model‘s items were reviewed to determine if any correlated strongly with multiple factors. Those items that appeared to be strongly associated with multiple factors were eliminated.

Finally, 16 items remained and produced a six-factor solution; however, one factor was eliminated because the factor was a single item (i.e., the value of activity consultants). Generally, factors with fewer than three items are considered weak and unstable. It was surprising that this item did not load with any other factor, and may suggest that there was not a strong enough association with any other items. The remaining 15 items, represented by five factors, are presented as the final factor solution.

Similar to other statistical methods, data must meet certain criteria to establish that the data can be used in a factor analysis. Principal Components analysis was used to identify factors that could explain the relationships seen in Site coordinator survey responses. There are two measures that help to determine whether the data available for this procedure are of adequate size and if the observed relationships among the variables are “good enough” to support a factor analytic approach. The Kaiser-Meyer-Olkin (KMO) is a measure of sampling adequacy. For this analysis, the KMO statistic for the final factor solution was .508, meeting the minimum recommended value of .5. The Bartlett Test of Sphericity, which measures if the strength of the relationship among variables is strong enough, was significant (p = .001), indicating that the observed relationships could support a factor analysis.

Items that were maintained in the model reflected more of the attitude of various aspects of the program model. The 15 items that remained included items related to adherence to the site schedule, cycle plan implementation, the value of the contribution of various staff, collaboration with schools, involvement in the community, and the level of involvement of parents, students and the school staff.

Results

Items were examined using Principal Components analysis with Varimax (orthogonal) rotation. The final factor analytic model produced a five factor solution, explaining 68.2% of the total


Page 43

variance. Eigenvalues are calculated for each factor and indicate how much variation the total sample can be accounted for each factor. Generally, eigenvalues below 1.0 are not included in final models, thus, those factors that have a value of greater than 1.0 are included in the factor solution; however, there was one factor where the eigenvalue was greater than 1.0 but it was excluded from the final model because the factor contained only one item (value of activity consultants). The five factors are presented in the table below. Factor names are based on a shared theme among the items that make up each. These items are presented in the table on the following page.

EIGENVALUES AND VARIANCE DISTRIBUTION OF FIVE FACTOR SOLUTION

Eigenvalue Percent of Variance

Core Elements 3.974 24.8%

Cycle Plans 2.28 14.3%

Ad Hoc Assistance 1.86 11.6%

Connections 1.54 9.6%

Educational Support 1.26 7.9%

Total 68.2%

The items that make up each factor are presented in Table C-2. Item loadings are included when the primary factor loading was .4 or above. Loadings this high are considered adequate to include in a particular factor. Communalities are also included, as they indicate which single items the factor solution accounts for the variance the best, and least. In this analysis, the value of parent volunteers and school personnel, the usefulness of cycle plans, and the value of teacher liaisons have the greatest amount of variance explained by the five-factor solution presented above.

There are three items associated with each factor. A theme, based on the item(s) within each factor, was identified and used to create factor names presented (can be found in Table 5 of the report).


Page 44

FACTOR LOADINGS AND COMMUNALITIES

Core Elements

Cycle Plans Value of Ad Hoc

Assistance

Connections Educational Supports

Communal-ities

To what extent do you have a collaborative relationship with school administrators?

.858 .755

Generally, how often do you follow the site schedule? .819 .781

To what extent are students involved in decisions that impact the program design?

.731 .605

To what extent do you implement cycle plans? .858 .792

How useful are your cycle plans during the implementation of club activities at your site?

.793 .818

To what extent are the objectives of your cycle plans met? .724 .558

How valuable is the contribution for each staff that help to deliver activities at your site: WR Parent Volunteers?

.907 .906

How valuable is the contribution for each staff that help to deliver activities at your site: Youth Volunteers?

.710 .634

How valuable is the contribution for each staff that help to deliver activities at your site: WR Traveling Specialists?

.646 .800


.722 .721

In general, how involved are parents in your program? .715 .662

How would you describe the access you have to school facilities that you need to implement your programs’ activities?

.710 .735


.871 .802


.829 .825

To what extent are you using academic concepts in club activities? -.544 .793


Page 45

Internal consistency for each of the factors was examined using Cronbach’s alpha to determine whether the items can be used to create scales to explore program implementation. The alpha coefficients for the factors ranged from .214 – .682 (Descriptive statistics are presented in Table C-3). For exploratory research, a minimum of .600 is required to establish the reliability of a scale. Only the first three factors meet this criterion.

ALPHA COEFFICIENTS OF FIVE-FACTOR SOLUTION

Factors Alpha Coefficient Scale Mean (Standard Deviation)

Program Infrastructure .682 10.94(1.38)

Cycle Plans .606 9.84 (1.29)

Value of Ad Hoc Assistance .676 6.35 (2.30)

Connections .512 8.67 (2.00)

Educational Supports .214 3.75 (1.94)

Woodcraft Rangers: Assessment of Quality Appendix D

Page 46

APPENDIX D – QUALITY BENCHMARK CATEGORIES AND SCORES, MAPPED TO SITE

COORDINATOR SURVEY ITEMS AND HIGH-QUALITY RESPONSE EXPECTATIONS

Note: The same survey was administered to elementary and middle school site coordinators. The Woodcraft management team’s quality expectations sometimes differ by elementary versus middle school site level. When there is a benchmark difference between the two site types, it is noted with an “ES” or “MS” to show the school level to which it applies. Benchmark 1: Cycle Plans Implementation 5 points

To what extent do you implement cycle plans? All or most of the cycle plans are implemented as written

Some of the cycle plans are implemented as written Few of the cycle plans are implemented as written

None of the cycle plans are implemented as written Don’t know

Generally, to what extent are you knowledgeable about the objectives of the cycle plans at your site?

Very knowledgeable Somewhat knowledgeable

Not very knowledgeable Not at all knowledgeable

Overall, to what extent are the objectives of your cycle plans met? Most or all of the objectives are met

Some of the objectives are met Few of the objectives are met None of the objectives are met Don’t know

How useful are your cycle plans during the implementation of club activities at your site? Very useful, they are often referred to

Useful, they are occasionally referred to Not very useful, they are rarely referred to Not at all useful, they are never referred to Don’t know

To what extent are you using academic concepts in club activities? A lot Sometimes

A little Not at all Don’t know


Page 47

Benchmark 2: Time Distribution 5 points

What is the average number of minutes per day that students are engaged in each of the following activities at your site?

# of minutes per day

ES MS

Homework 45 45

Fitness 20 20

Nutrition/snack 15 20

Interest-based club activities 75 100

Closing activities or all-together time 30 NA

Benchmark 3: Club Activities 6 points

Which of the following topics of club activities have been offered at your site in the past year? Check all that apply.

(Note: All Woodcraft sites are expected to offer activities in each of the 6 checked areas below at some point throughout each year.)

Sports Performing arts (e.g., music, drama, dance) Visual arts (e.g., photography, graffiti arts, cartooning) Recreation activities (e.g., games) Computers OR Technology/multi-media Leadership opportunities/youth development

(Note: Woodcraft expects that sites will offer some of the following activities if there is an interest/need of that site’s school community but does not expect that all sites will offer these.)

Fitness (Note: these are fitness related clubs, such as yoga, which are different from the general fitness activity all students do listed in Benchmark 2) Nutrition education Tutoring for individual skill development Reading/literacy Mathematics/science CAHSEE preparation Education and college planning Job skills and career preparation Other


Page 48

Benchmark 4: Student Engagement 9 points

How often are students engaged in each of the following activities at your site?

At least once a week

At least once a month

Once every 2 – 3

months

Once or twice a

year Not

at all

Youth leadership activities

Club culmination activities

Field trips

Sports tournaments

Group activities such as Fun Fridays or assemblies

On-site activities open to the community such as literacy events

Career awareness events such as resume writing or professional guest speakers

MS

MS

MS

MS

ES

Middle school and high school transitional activities

Community service activities

Benchmark 5: Club Contributors 3 points (ES); 4 points (MS)

In addition to Club Leaders, who else contributes to the delivery of activities at your site? Check all that apply.

Woodcraft traveling specialists (MS) Woodcraft activity consultants Woodcraft teacher liaisons School personnel (other teachers, computer technicians, literacy coaches, etc.)

Woodcraft parent volunteers Woodcraft youth volunteers


Page 49

Benchmark 6: Staff Value 5 points (ES); 6 points (MS)

Please rate how valuable the contribution is for each that help to deliver activities at your site.

Critical for success

Very important Important

Not very important

Woodcraft traveling specialists MS

Woodcraft activity consultants

Woodcraft teacher liaisons

School personnel (other teachers, computer technicians, coaches, etc.)

Woodcraft parent volunteers

Woodcraft youth volunteers

Benchmark 7: Club Selection 6 points

Who provides input into the selection of club activities at your site? Check all that apply. Woodcraft Regional Manager Woodcraft Site Coordinator Woodcraft Club Leader Woodcraft Youth Advisory Board/Youth Council Woodcraft Youth Participants Principal/School Liaison

Woodcraft Administration Woodcraft Teacher Liaison District Liaison

Benchmark 8: Student Involvement 5 points (ES); 6 (MS)

To what extent are students involved in the program in the following ways?

A lot

Some-what

A little

Not at all

Students’ needs and interests are considered by staff in the delivery of the program

Students have a sense of belonging in the program and among peers

Students have a choice of club activities

Students have opportunities and support to take on leadership roles

Students are involved in decisions that impact the program design

MS

Students have opportunities to contribute to or interact with the community


Page 50

Benchmark 9: Parent Involvement 14 points

In general, how involved are parents at this school? A lot Somewhat A little

Not at all Don’t know

To what extent are parents involved in your program? A lot Somewhat

A little Not at all Don’t know

To what extent are parents involved in the program in the following ways?

ELEMENTARY SCHOOL BENCHMARK A lot

Some-times A little

Not at all

Receive information on program activities

Receive information on school activities

Communicate regularly with program staff

Provide input on program activities

Volunteer at program/assist with activities

Attend club culminations

Attend field trips with participants

Attend sport tournaments

Volunteer at/assist with sport tournaments

Attend workshops for parents (e.g., Literacy Nights, school planning)

Attend community activities (e.g., holiday activities)

Participate in Woodcraft Parent Council

MIDDLE SCHOOL BENCHMARK A lot

Some-times A little

Not at all

Receive information on program activities

Receive information on school activities

Communicate regularly with program staff

Provide input on program activities

Volunteer at program/assist with activities

Attend club culminations


Page 51

Attend field trips with participants

Attend sport tournaments

Volunteer at/assist with sport tournaments

Attend workshops for parents (e.g., Literacy Nights, school planning)

Attend community activities (e.g., holiday activities)

Participate in Woodcraft Parent Council

Benchmark 10: School Collaboration 8 points

To what extent do you have a collaborative relationship with school administrators? A lot

Somewhat A little

Not at all

To what extent does your site collaborate with the school in the following ways?

At least once a week

At least monthly

Once every 2

– 3 months

Once or twice a

year Not at

all

Has FORMAL meetings with the school principal/administration

Has INFORMAL meetings with the school principal/administration

Communicates with teachers other than your Woodcraft Teacher Liaison

Seeks input from the school principal on the impact of the program on students

Seeks input from school teachers on the impact of the program on students

Communicates the academic and behavioral progress of students to the school

Collaborates with general school staff (such as literacy coaches, math coaches, Title I coordinator)


Page 52

Benchmark 11: Facilities Access 8 points

How would you describe the access you have to school facilities that you need to implement your program’s activities?

Full access Adequate access

Limited access No access Don’t know

Which of the following school facilities do you have access to for your program? Check all that apply.

Classrooms Cafeteria/auditorium School yard/playground Sport fields/courts/gymnasium Fitness equipment/workout rooms Cooking facilities/classrooms Library Computer rooms Other

Benchmark 12: Site Coordinator Qualities 11 points

We know that all of the following characteristics are important for Site Coordinators to have. Some of these characteristics may be more critical in order to have an effective program. To what extent are the following qualities important in your role as a Site Coordinator?

Critical for

success Very

important Important Not very

important

Leadership skills

Management experience

Enthusiasm for working with students

Commitment to program goals

Initiative and creativity

Organization

Establish a welcoming environment

Clearly communicate expectations

Responsive to staff and student needs

Model respectful interactions with staff, students, and families

Level of education


Page 53

Benchmark 13: Program Staff Qualities 11 points To what extent are the following qualities important to have in program staff?

Critical for success

Very important Important

Not very important

Leadership skills

Experience in after school programs

Enthusiasm for working with students

Commitment to program goals

Communication skills

Organization skills

Initiative and creativity

Ability to be responsive to student needs

Ability to model respectful interactions

Level of education

Sense of fun / playfulness

Overall Benchmark

All items from the above 13 benchmark categories were used in creating the overall benchmark score, along with the following two additional items.

Site Schedule

Generally, how often do follow the site schedule? Always

Sometimes Rarely Never Don’t know Community Involvement

On average, how many planned activities do students attend or participate in that promote direct involvement with the community (e.g., writing letters to local police or fire stations or working in a community garden)?

At least once a week At least once a month Once every 2 – 3 months Once or twice a year

Not at all

Woodcraft Rangers: Assessment of Quality Appendix E

Page 54

APPENDIX E – MULTIPLE REGRESSION MODELS FOR QUALITY FACTORS, ELEMENTARY

SCHOOLS When the regression model was significant and the quality factor contributed significantly to the model, the probability value (p) is noted along with the beta weight (β) and associated level of significance. Probability values <.05 suggest that findings are not likely due to chance. Beta weights indicate how much influence that variable has on the regression model. The value of the beta weight is not important here, rather, the direction of the beta weight (either positive or negative) is critical. Similar to a correlation, the direction of the beta weight indicates whether the contribution of that variable adds, or subtracts from the predicted variable.

REGRESSION MODELS OF QUALITY FACTORS AND YOUTH OUTCOMES, ELEMENTARY SCHOOLS14 Core

Elements Cycle Plans

Ad Hoc Assistance

Change CST ELA NS p=.000

(β=-.068, p=.002) N=2,292

p=.014 (β=.024, p= .065)

N=1,540

Change CST Math NS NS NS

Change school attendance NS NS NS

Level of positive attitude toward after school NS p =.000

(β=-.040, p=.031) N=2,184

p =.000 (β=.026, p=.005)

N=1,347

Change in attitudes toward school NS NS NS

Level of involvement in making choices NS NS NS

Level of use of academic skills in activities NS NS p =.000

(β=.039, p=.002) N=1,359

Change in homework/school skills NS NS NS

Level of involvement in fitness NS NS p =.001

(β =.039, p=.001) N=1,362

Change in self-confidence NS NS NS

Change in pro-social skills NS NS NS

Change in problem behaviors NS p =.000

(β =.101, p =001) N=2,195

NS

14 Findings where the regression model or the contribution of the quality factor was not statistically significant are

marked as NS (not significant). Sample sizes are reported and vary by item due to missing data from respondents.

Woodcraft Rangers: Assessment of Quality Appendix F

Page 55

APPENDIX F – MULTIPLE REGRESSION MODELS FOR QUALITY FACTORS, MIDDLE SCHOOLS When the regression model was significant and the quality factor contributed significantly to the model, the probability value (p) is noted along with the beta weight (β) and associated level of significance. Probability values <.05 suggest that findings are not likely due to chance. Beta weights indicate how much influence that variable has on the regression model. The value of the beta weight is not important here, rather, the direction of the beta weight (either positive or negative) is critical. Similar to a correlation, the direction of the beta weight indicates whether the contribution of that variable adds, or subtracts from the predicted variable.

REGRESSION MODELS OF QUALITY FACTORS AND YOUTH OUTCOMES, MIDDLE SCHOOLS15 Core

Elements Cycle Plans

Ad Hoc Assistance

Change CST ELA NS NS NS

Change CST Math NS p=.000

(β=-.046, p =.041) N=1,844

NS

Change school attendance NS NS NS

Level of positive attitude toward after school p=.041

(β=.241, p =.008) N=256

NS NS

Change in attitudes toward school p=.006

(β=.524, p =.000) N=260

p=.020 (β=.216, p =.003)

N=355

NS

Level of involvement in making choices NS NS NS

Level of use of academic skills in activities p=.000

(β=.250, p=.024) N=257

p=.000 (β=.119, p=.034)

N=354

NS

Change in homework/school skills p=.000

(β=.236, p=.049) N=253

NS NS

Level of involvement in fitness p =.006

(β =.342, p =005) N=259

p =.064 (β =.144, p =018)

N=356

NS

Change in self-confidence NS NS NS

Change in pro-social skills NS NS NS

Change in leadership skills p =.035

(β =-.180, p =032) N=77

NS NS

Change in problem behaviors p =.000

(β =-.471, p =001) N=253

p =.000 (β =-.137, p =049)

N=359

NS

15 Findings where the regression model or the contribution of the quality factor was not statistically significant are

marked as NS (not significant). Sample sizes are reported and vary by item due to missing data from respondents.

Woodcraft Rangers: Assessment of Quality Appendix G

Page 56

APPENDIX G – MULTIPLE REGRESSION MODELS FOR QUALITY BENCHMARK CATEGORIES, ELEMENTARY SCHOOLS When the regression model was significant and the quality factor contributed significantly to the model, the probability value (p) is noted along with the beta weight (β) and associated level of significance. Probability values <.05 suggest that findings are not likely due to chance. Beta weights indicate how much influence that variable has on the regression model. The value of the beta weight is not important here, rather, the direction of the beta weight (either positive or negative) is critical. Similar to a correlation, the direction of the beta weight indicates whether the contribution of that variable adds, or subtracts from the predicted variable.

Change CST ELA

Change CST Math

Change in School

Attendance

Change in Homework/ School Skills

Change in Pro-social Skills

Change in Problem Behavior

Cycle Plans p=.000

(β=-.090, p =.000) N=2,291

NS NS NS NS p=.000

(β=.122, p =.000) N=2,211

Time Distribution NS NS p=.000

(β=-.003, p =.03) N=2,657

NS p=.000

(β=.051, p =.022) N=2,189

p=.000 (β=.076, p =.019)

N=2,211

Club Activities NS p=.000

(β=.087, p =.000) N=2,296

NS p=.003

(β=-.052, p =.001) N=2,174

p=.000 (β=-.048, p =.004)

N=2,189 NS

Student Engagement NS NS NS NS NS NS

Club Contributors NS NS p=.000

(β=.004, p =.021) N=2,657

NS p=.000

(β=.059, p =.013) N=2,189

p=.000 (β=-.073, p =.039)

N=2,211

Ad Hoc Staff Value NS p=.000

(β=-.073, p =.003) N=2,296

NS p=.028

(β=.042, p =.015) N=2,174

NS p=.000

(β=-.118, p =.000) N=2,210

Club Selection NS NS NS p=.041

(β=.021, p =.027) N=2,174

NS p=.000

(β=-.069, p =.000) N=2,211

Student Involvement NS NS NS p=.051

(β=-.043, p =.040) N=2,174

NS p=.000

(β=.102, p =.002) N=2,211

Parent Involvement NS NS NS NS NS p=.000

(β=.030, p =.000) N=2,211

School Collaboration NS NS NS NS NS p=.000

(β=.086, p =.000) N=2,211

Access to Facilities NS p=.000

(β=.089, p =.000) N=2,296

NS NS NS p=.000

(β=-.060, p =.008) N=2,211

Site Coordinator NS NS NS NS NS p=.000

(β=-.032, p =.003) N=2,211

Program Staff NS p=.000

(β=-.022, p =.014) N=2,296

NS NS NS NS

Overall Score NS NS NS NS NS NS

Woodcraft Rangers: Assessment of Quality Appendix G

Page 57

Level of Use of

Academic Skills in

Activities

Change in Attitude

Toward School

Level of Attitude

Toward After-School

Level of Involvement

in Making Choices

Level of involvement

in Fitness

Change in Self-confidence

Cycle Plans NS NS p=.000

(β=-.047, p =.016) N=2,198

NS NS NS

Time Distribution NS NS NS NS NS NS


(β=-.074, p =.008) N=2,220

NS p=.000

(β=.052, p =.010) N=2,207

NS NS

Student Engagement p=.000

(β=.064, p =.002) N=2,213

NS NS NS NS NS

Club Contributors p=.000

(β=.164, p =.000) N=2,213

p=.034 (β=.083, p =.039)

N=2,220 NS

p=.000 (β=.077, p =.006)

N=2,2.07 NS NS

Ad Hoc Staff Value NS NS p=.000

(β=.037, p =.028) N=2,198

p=.000 (β=-.100, p =.000)

N=2,207 NS NS

Club Selection NS NS Ns p=.000

(β=-.061, p =.000) N=2,207

NS NS

Student Involvement p=.000

(β=.065, p =.019) N=2,213

NS NS p=.000

(β=.123, p =.000) N=2,207

NS NS

Parent Involvement NS NS NS p=.000

(β=-.020, p =.002) N=2,207

NS NS

School Collaboration NS NS NS p=.000

(β=.042, p =.023) N=2,207

NS NS

Access to Facilities NS NS NS NS NS NS

Site Coordinator NS NS p=.000

(β=.018, p =.008) N=2,198

NS NS NS

Program Staff p=.000

(β=.038, p =.000) N=2,213

NS NS p=.000

(β=.024, p =.002) N=2,207

NS NS

Overall Score NS NS NS NS NS NS

Woodcraft Rangers: Assessment of Quality Appendix H

Page 58

APPENDIX H - MULTIPLE REGRESSION MODELS FOR QUALITY BENCHMARK CATEGORIES, MIDDLE SCHOOLS When the regression model was significant and the quality factor contributed significantly to the model, the probability value (p) is noted along with the beta weight (β) and associated level of significance. Probability values <.05 suggest that findings are not likely due to chance. Beta weights indicate how much influence that variable has on the regression model. The value of the beta weight is not important here, rather, the direction of the beta weight (either positive or negative) is critical. Similar to a correlation, the direction of the beta weight indicates whether the contribution of that variable adds, or subtracts from the predicted variable.

Change CST

ELA Change CST

Math

Change in School

Attendance

Change in Homework/ School Skills

Change in Pro-social Skills

Change in Problem Behavior

Cycle Plans NS NS NS NS NS p=.000

(β=-.197, p =.034) N=362

Time Distribution NS NS p=.001

(β=-.006, p =.052) N=1,755

NS NS NS


(β=-.170, p =.000) N=1,844

NS NS NS NS

Student Engagement NS NS NS NS NS NS

Club Contributors p=.019

(β=.042, p =.012) N=1,856

NS NS NS NS NS

Ad Hoc Staff Value NS NS NS p=.000

(β=.443, p =.001) N=359

NS p=.000

(β=-.623, p =.000) N=361

Club Selection p=.015

(β=.031, p =.009) N=1,856

NS NS p=.000

(β=-.210, p =.003) N=359

NS p=.000

(β=.278, p =.000) N=362

Student Involvement p=.005

(β=.035, p =.002) N=1,856

p=.000 (β=.033, p =.014)

N=1,844 NS

p=.000 (β=.130, p =.045)

N=359 Ns

p=.000 (β=-.193, p =.005)

N=362

Parent Involvement p=.013

(β=.018, p =.007) N=1,856

NS NS Ns NS NS

School Collaboration p=.013

(β=.029, p =.007) N=1,855

p=.000 (β=.036, p =.005)

N=1,844 NS

p=.000 (β=.187, p =.012)

N=359 NS

p=.000 (β=-.288, p =.000)

N=362

Access to Facilities p=.008

(β=.037, p =.004) N=1,855

NS p=.000

(β=.003, p =.008) N=1,755

p=.000 (β=.242, p =.015)

N=359 NS

p=.000 (β=-.375, p =.003)

N=362

Site Coordinator p=.010

(β=.023, p =.005) N=1,856

NS p=.001

(β=.002, p =.050) N=1,755

p=.000 (β=.128, p =.022)

N=359 NS

p=.000 (β=-.145, p =.014)

N=362

Program Staff p=.048

(β=.012, p =.054) N=1,856

NS p=.000

(β=.001, p =.022) N=1,755

NS NS NS

Overall Score p=.013

(β=.305, p =.007) N=1,856

NS NS NS NS p=.000

(β=-3.29, p =.020) N=350

Woodcraft Rangers: Assessment of Quality Appendix H

Page 59

Level of Use of Academic

Skills in Activities

Change in Attitude Toward School

Level of Attitude Toward

After-School

Level of Involvement

in Making Choices

Level of involvement

in Fitness

Change in Self-

confidence

Change in Leadership

Skills

Cycle Plans p=.000

(β=.162, p =.031) N=354

p=.016 (β=.300, p =.002)

N=366 NS NS

p=.037 (β=.219, p =.008)

N=356 NS NS

Time Distribution p=.000

(β=.237, p =.037) N=354

p=.025 (β=.422, p =.004)

N=366 NS NS NS NS NS

Club Activities NS NS NS NS NS NS NS

Student Engagement p=.000

(β=-.083, p =.020) N=355

NS NS NS NS NS NS

Club Contributors NS NS NS NS NS NS p=.030

(β=-.385, p =.006) N=122

Ad Hoc Staff Value p=.000

(β=.238, p =.043) N=354

p=.033 (β=.418, p =.006)

N=366

p=.002 (β=.364, p =.000)

N=354 NS

p=.002 (β=-.494, p =.000)

N=356 NS

p=.031 (β=-.253, p =.006)

N=122

Club Selection p=.000

(β=-.141, p =.020) N=354

NS Ns NS NS NS NS

Student Involvement NS p=.019

(β=.215, p =.003) N=366

NS NS p=.005

(β=.214, p =.000) N=356

NS NS

Parent Involvement NS NS NS NS NS NS NS

School Collaboration NS NS p=.019

(β=.156, p =.002) N=353

NS p=.037

(β=.187, p =.008) N=356

NS NS

Access to Facilities p=.000

(β=.189, p =.028) N=354

p=.007 (β=.377, p =.001)

N=366

p=.003 (β=.251, p =.000)

N=353 NS

p=.001 (β=.376, p =.000)

N=356 NS NS

Site Coordinator p=.000

(β=.106, p =.028) N=353

NS NS NS p=.002

(β=.199, p =.000) N=355

NS NS

Program Staff NS p=.017

(β=-.153, p =.002) N=366

NS NS NS NS NS

Overall Score NS p=.039

(β=3.83, p =.010) N=354

NS NS NS NS NS

a study of the woodcraft rangers’ · woodcraft rangers: assessment of quality executive summary...

Documents