21st century community learnign centers illinois statewide ... · understand the day-to-day...

21st Century Community Learning Centers Illinois Statewide Evaluation Program Year 2006-2007

Penny Billman, Ph.D. Northern Illinois University

J. Harvey Smith, Ph.D. Northern Illinois University

NIU Outreach and Regional Development Interactive Illinois Report Card Project

July 2007

i

This research was conducted at the request of, and funded through, the Illinois State Board of Education. The contents of the report reflect the conclusions drawn by the researchers and based on the data and best research practices. The report does do not represent the views or official position of Northern Illinois University.

The 21st Century Community Learning Center program was reauthorized under Title IV, Part B, of the No Child Left Behind Act of 2001.

21st CCLC Annual Evaluation 2006-2007

ii

Table of Contents

Executive Summary ................................................................................................i

Acknowledgements................................................................................................v

Program Evaluation Overview .............................................................................. 2 Purpose of the Evaluation ........................................................................ 2 Description of Active Grants in 2006-2007................................................ 4 Methodology.............................................................................................. 7

Part I: Formative Evaluation Findings ................................................................ 10 Evidence of Meeting Program Objectives ............................................... 10 Quality of Profile and Performance Data................................................. 21 Status of Recommendations from Previous Formative Evaluation ......... 22

Part II: Summative Evaluation Findings ............................................................. 27 Factors Impacting Program Implementation and Outcomes................... 27 Differences in Mid-Cycle and End-of-Cycle Programs............................ 30 Characteristics of Effective Programs ..................................................... 30 Concluding Summative Thoughts ........................................................... 31

Recommendations from Formative and Summative Evaluations ....................... 32 The Evolving 21st CCLC Program ........................................................... 32 Developing the Front Line ....................................................................... 33 Perceptions of Competition ..................................................................... 33

Appendix A: Research Matrix and Instrumentation for Formative and Summative Evaluations ................................................................ 34


iii

Table of Exhibits

Exhibit 1. Illinois Objectives and Research Questions for 21st CCLC Evaluations 3 Exhibit 2. Active 21st CCLC Grants in 2006-2007 .................................................. 4 Exhibit 3. 21st CCLC Sites in 2006-2007 in Illinois................................................. 6 Exhibit 4. 21st CCLC Sites in Northeastern Illinois in 2006-2007 ........................... 7 Exhibit 5 Staff Perceptions of Student Improvement in Behavior and Performance 11 Exhibit 6. Percentages of Students with Half-Grade Changes in 2006................... 13 Exhibit 7. Comparison of 2006 Reading Scores on Illinois State Assessments...... 14 Exhibit 8. Comparison of 2006 Mathematics Scores on Illinois State Assessments 14 Exhibit 9. Percentages of 2003 Grantees with Sustainability Funds....................... 19 Exhibit 10. Most Common Approaches to Sustainability ....................................... 20 Exhibit 11. Status of Recommendations from Statewide Evaluation 2005-2006 ... 22 Exhibit 12. Factors Mentioned Most Often as Facilitating Positive Outcomes....... 29 Exhibit 13. Factors Mentioned Most Often as Barriers .......................................... 29


Executive Summary Since 2003, the State of Illinois through the Illinois State Board of Education has granted over 110 awards for 21st Century Community Learning Centers (21st CCLC), as authorized under Title IV, Part B, of the No Child Left Behind Act. The goal of the program is to provide opportunities “for students and their families to continue to learn new skills and discover new abilities after the school day has ended.”1 Over the past five years, these grantees provided services at over 400 locations. In 2007, the first wave of grantees completed their five-year funding cycle, and the evidence indicates that the 21st CCLC sites are making a positive impact on students’ behavior and academic achievement.

The State of Illinois established seven objectives for the 21st CCLC program. In 2006-2007, qualitative and quantitative data were collected from all of the 110 active awards, and 41 programs received an on-site visit (56%) to determine the progress Illinois has made in achieving the seven objectives.

Student Behavior and Achievement (Objectives 1, 2, and 3) 9 In 2006, 61.5% of the 40,387 students who used the 21st CCLC site attended for 30 days

or more. 9 According to their teachers, over 60% of the students needing to improve their behavior

did improve. 9 Teachers and staff believe nearly three-fourths of the students improve academically. 9 Even though the classroom teachers and program administrators recruited low-

performing students for the 21st CCLC programs, students who attended the 21st CCLC program did not differ significantly in their academic performance on the reading and math ISAT exams than their classmates who did not attend the 21st CCLC programs.

Staff Perceptions of Student Improvement in Behavior and Performance

1 Source: United States Department of Education, downloaded May 1, 2006 from USDOE web site, www.ed.gov/programs/21stccls/index.html.

Source: Compilation of Staff and Teacher Surveys, 2006-2007

21st CCLC Annual Evaluation 2006-2007 i

Parent and Community Involvement (Objective 4) 9 Over 46% of the program administrators interviewed mentioned “lack of parent

involvement” as a critical factor hindering the project. The 21st CCLC sites have tried various approaches to engage more parents and have found varying degrees of success.

9 Student achievement was not significantly related to whether or not the parents were involved in the school. On the other hand, a preliminary analysis found that the degree to which the teacher feels the parent supports the school is related to student achievement. Higher degrees of perceived parent support for the school were linked to higher student performance in math and reading.

Serving the Highest Need Schools (Objective 5) 9 Illinois established grant criteria in which programs serving high-need schools are given

preference. 9 Nearly 85% of the 21st CCLC regular attendees in grades 3 through 8 are eligible for free

or reduced lunches.

Professional Development (Objective 6) 9 ISBE provided mandatory professional development activities on sustainability and the

measurement of the program’s progress and outcomes. 9 The majority of grantees (78%) provide additional professional development

opportunities to their personnel.

Sustainability (Objective 7) 9 Only about 6% of the grants ending in 2003 had secured sustainable funding for the

coming year at the current level. About 72% have some funding and 22% are in total crisis.

Progress on Last Year’s Recommendations 9 ISBE and the individual grantees have made good progress in addressing the

recommendations from last year. Initiatives are in place and are funded. Continued progress on addressing the recommendations is expected.

Findings from Summative Evaluation of 2003 Cohort Data on the 2003 grants were coded to identify those grants that were in the top quartile on the following indicators: student achievement and positive behavior; parent and community involvement; strength of partnerships; sustainability; and evaluation tied to continuous improvement. The grantees appearing most frequently in the top quartile shared the following characteristics: 9 They focused on providing prescriptive interventions to individual students. They zeroed

in on the student’s need in a variety of ways, such as diagnostic computer instructional programs, close communication with the classroom teacher, and pre- and post-testing of the student on instruments that identify content and skills to target.

9 They focused on identifying the students most likely to benefit from their program. They knew whom they were serving and what those students needed. They did not try to be everything for everyone.

9 They had high quality staff, often certified. Staff had rich and varied backgrounds and the ability to build rapport with the students.

21st CCLC Annual Evaluation 2006-2007 ii

Recommendations 1. The revised continuation form is a tool the grantee, ISBE, the external evaluator, and the

technical assistance provider can use to track each program over time, ensuring the evaluation and sustainability plans are implemented and that the project has not strayed from its original vision and proposed delivery of services.

2. Programs which evolved effectively had a clear focus on who they served and strategically used resources to provide the highest quality programming to that population. The strategic plan for the year, the professional development, and the allocation of resources were aligned to achieve the program’s goals. he technical assistance model adopted by ISBE can be used to provide individualized, strategic planning help to the programs, as needed.

3. The evolving program needs accurate data on which to base decisions. During the next year, programs should continuously work to ensure the PPICS data and other data used for program improvement have acceptable levels of reliability and validity. This includes furthering the discussion on how to flag the ISAT/PSAE data in ways that reflect levels of attendance and include student identifiers to track individuals across years.

4. Because relationships provide the foundation for the success of the program, grantees, especially first-time grantees, might benefit from instruction on ways to use tools to assess the health of a relationship and to begin meaningful dialogues to build partnership capacity.

5. An advisory group could be formed to find solutions to problems such as a) decreasing the number of students who complete homework but fail to turn it in, b) increasing the involvement of middle school students in other school activities, and c) improving communications with parents so they understand the “community learning center” aspect of the program.

6. The “dosage of intervention” is related to student achievement; e.g., students who attend more regularly have higher achievement gains. Sites should focus on retention rather than one-time attendance at an event. Effective ways of marketing the importance of regular attendance to students and parents need to be identified and shared.

7. ISBE and the grantees may wish to explore cost effective and efficient ways of providing professional development to front-line workers, especially through the technical assistance provider funded in FY 2008.

8. Program directors, especially in the Chicago Metro area, mentioned increasing competition among after school programs, especially those programs targeting a specific ethnic group. In addition, there is a perception that the pool of schools that will qualify for 21st CCLC funds is shrinking across the state. These issues need further discussion and documentation to determine how to best use 21st CCLC funds in Illinois.

In closing, the impact of the 21st CCLC program is difficult to measure but it is observable to those watching the programs change over time. Each program has testimonials documenting the “life changing” stories of their students. History can be used to predict what the outcome could have been for many of these students if they had not been in the program. The 21st

CCLC program in Illinois is creating a future of change, hope, and opportunity a student at a time.

21st CCLC Annual Evaluation 2006-2007 iii

(Blank)

21st CCLC Annual Evaluation 2006-2007 iv

Acknowledgements

A project of this magnitude can be completed only with the assistance and collaboration of multiple agencies and individuals. The authors wish to thank the many staff members at the Illinois State Board of Education who provided time, guidance, and the data necessary to complete the analyses, especially Gail Meisner.

A special recognition goes to the administrators of the 110 grants and their staff members who kindly and enthusiastically allowed us to interview them and delve into what worked and what did not work at their after-school sites. Their valuable insights and reflective discussions provided in-depth information not readily collected via other methods.

We are especially grateful to the grant managers who coordinated on-site visits for Northern Illinois University researchers and consultants. The interviews with staff, parents, and/or community partners, as well as the observations of after-school sessions, allowed us to better understand the day-to-day operations, successes, and challenges of 21st Century Community Learning Center. We appreciated your gracious willingness and openness towards our visit, questions, surveys, and observations.

Lastly, the authors wish to thank the staff at Northern Illinois University and the technical consultants on this project. They deserve special commendations for their work and dedication to the project, especially B.A.S.S. Counseling and Educational Services, Mary Wood, Richard Carlson, Tim Butterfield, Sherrie Taylor, Lisa Bergeron, Paulette Bowman, and Andrea Lewis.

21st CCLC Annual Evaluation 2006-2007 1

Program Evaluation Overview

Purpose of the Evaluation The 21st Century Community Learning Center program (21st CCLC), authorized under Title IV, Part B, of the No Child Left Behind Act of 2001, provides funding to states to create opportunities “for students and their families to continue to learn new skills and discover new abilities after the school day has ended.”2 The collection of data on the 21st CCLC centers is mandated in the federal legislation at both the grantee and the state level. Grantees report annual data through the Profile and Performance Information Collection System (PPICS) coordinated by Learning Point Associates.3 This information is summarized in the Annual Performance Report (APR) for each 21st CCLC award. Annual reports by award, an annual state summary, and miscellaneous reports are available electronically at the Learning Point Associates website: www2.learningpt.org/ppics. The most recent annual reports and individual APR reports posted at the time of this research are for program year 2004-2005.

The statewide evaluation (this report) summarizes the state-level formative (Part I) and summative evaluations (Part II) of the 21st CCLC program in the State of Illinois during program year 2006-2007. The focus of this research went beyond the annual performance analyses to look at what worked, where improvement is needed, and where projected outcomes were realized.

The formative evaluation analyzed the seven measurable objectives for 21st CCLC programs identified by the State of Illinois (Exhibit 1). Each objective is linked to measurable performance indicators. These objectives and performance indicators provided the primary foundation for the formative evaluation.

The research findings of the formative evaluation (Part I) are organized around three topics: • Evidence of Meeting Program Objectives • Quality of Profile and Performance Data • Status of Recommendations from Previous Formative Evaluation

The summative evaluation (Part II) looked more closely at the 34 grants that ended their funding cycle in 2007. The purpose of the summative evaluation was to identify the most critical factors that facilitated or hindered the grant over the 5-year period, explore the specific features or characteristics that were associated with exemplary outcomes, and compare the outcomes of programs in mid-cycle and at the end of the funding cycle.

Specific research questions, performance indicators, and data sources were identified for the formative and summative evaluations, as presented in Appendix A.

2 Source: United States Department of Education, downloaded May 1, 2006 from USDOE web site, www.ed.gov/programs/21stccls/index.html.

3 Learning Point Associates, http://www.learningpt.org/ppics.


http://www.learningpt.org/ppics

Exhibit 1. Illinois Objectives and Research Questions for 21st CCLC Evaluations

Objective 1 Participants will demonstrate an increased involvement in school activities and in participating in other subject areas such as technology, arts, music, theater, and sports and other recreation activities.

Objective 2 Participants in the program will demonstrate increased academic achievement.

Objective 3 Participants in the program will demonstrate social benefits and exhibit positive behavioral changes.

Objective 4 The 21st CCLC programs will work toward services that benefit the entire community by including families of participants and collaborating with other agencies and non-profit organizations.

Objective 5 These programs will serve children and community members with the greatest needs for expanded learning opportunities.

Objective 6 21st CCLC program personnel will participate in professional development and training that will enable them to implement an effective program. Professional development activities must be aligned with the No Child Left Behind Act definitions and National Staff Development Council’s professional development standards.

Objective 7 21st CCLC program projects will use the funding most efficiently by coordinating and collaborating with state and other funding sources, agencies, and other community projects to supplement the program, and not supplant the funds, and to eventually become self-sustaining.

Section Questions

Evidence of Meeting Program

Objectives

What effect does the program have on youth behaviors as measured by changes in classroom behavior, attendance rates, involvement in school activities, attitudes toward school and learning, disciplinary referrals, and dropout and graduation rates? Objectives 1 & 3

What impact does the program have on student achievement, including homework completion, classroom grades, promotions, and performance on the state assessments? Objective 2

In what ways does the program serve the parents of the program participants? Is there increased involvement by participants' parents in regular school activities? Objective 4

What is the impact of the collaborations with other agencies and non-profit organizations? Objective 4

Did the RFP award process result in programs being awarded to serve the children and community members with the greatest need? Objective 5

Did the professional development activities provided through the State of Illinois to 21st CCLC program personnel adhere to No Child Left Behind Act definitions and the National Staff Development Council’s professional development standards? Objective 6

What are the current efforts toward providing for sustainability of the current programs, especially of the programs in their final year of funding? Objective 7

Quality of Profile and

Performance Data

Did the 21st CCLC program personnel find the data collection methods and evaluation resources, in particular PPICS, useful and relevant in documenting their programs and outcomes?

Factors Impacting Program

Implementation and Outcomes

To what extent are the grant recipients implementing the activities and evaluation plan proposed in their RFPs, as revised in their annual continuation requests?

Which factors hinder and which factors facilitate reaching the objectives?

Summative Evaluation

Are there specific features or characteristics associated with exemplary outcomes?

How do the outcomes of programs completing the five-year cycle differ from the outcomes of programs in mid-grant?

In retrospect, which were the most critical factors that hindered or facilitated obtaining the seven objectives? Are there barriers, which must be addressed if the program is to have a positive impact on communities in the future? How can these barriers be addressed?


Description of Active Grants in 2006-2007 In program year 2006-2007, Illinois had 110 active 21st CCLC awards. Because some organizations received awards in more than one competition (2003, 2004, 2005, and/or 2007), there are 72 different organizations with at least one active 21st CCLC award in program year 2006-2007. Exhibit 2 lists these active grants. During 2005 to 2007, researchers from Northern Illinois University researchers collected data from all of the grantees and conducted 41 on-site visits (56% of the grantees, represented in bold in Exhibit 2).

Exhibit 2. Active 21st CCLC Grants in 2006-2007 Grantee

(Bold indicates grantee hosted a site visit) 2003 2004 2005 2007

Ada S McKinley Community Services X Albany Park Community Center, Inc. X Alton Community Unit School District 11 (a) X X Aspira Inc. of Illinois X Aurora West Unit School District 129 X Beacon Street Gallery and Performance Company

X

Berwyn South School District 100 X X Bloom Township High School District 206 X Boys and Girls Clubs of Chicago X X Build, Inc. X Bureau/Henry/Stark ROE 28 X X X Cahokia Unit School District 187 X Calhoun Community Unit School District 40 X Calumet Public School District 132 X Centers for New Horizons Inc. X X Central Baptist Family Services (Kids’ Hope) X X Central States SER, Jobs for Progress X X Chicago Arts Partnerships in Education X Chicago Area Project X Chicago Public Schools 299 X X X Chicago Youth Centers X X Children’s Home and Aid Society X Christopher Unit School District 99 X Cities (Communities) in Schools X X X X Comer Science and Education Foundation X X Community Consolidated School District 65 X Country Club Hills 160 X Crete-Monee School District Unit 201 X Dallas City Community Unit School District 336 X Decatur School District 61 X Dolton West School District 148 X X Egyptian Community Unit School District 5 X Elementary School District 159 X Elverado Community Unit School District 196 X Fairmont School District 89 X Family Focus, Inc. X X X


Grantee (Bold indicates grantee hosted a site visit)

2003 2004 2005 2007

Firman Community Services X X Franklin/Williamson ROE 21 X Howard Area Community Center X Illinois Alliance of Boys and Girls Clubs X X Iroquois/Kankakee Counties ROE X X X J S Morton High School District 201 X Joliet Township High School District 204 X Jones Memorial Community Center X Little Village Community Development Corp. X Logan Square Neighborhood Association X Metropolitan Family Services X X Mexican Fine Arts Center Museum X Momence Comm Unit School District X Moody Kids Club X National Puerto Rican Forum Inc. X X Nicasa X Noble Street Charter High School X Northeastern Illinois University X X X Northwestern University Settlement X Passages Alternative Living Programs, Inc. X Prairie-Hills Elementary School District 144 X Project Success of Vermilion County, Inc. X Proviso/Leyden Council (b) X Quincy School District 172 X Rock Island County ROE 49 X X X Rockford Public School District 205 X X X X School District U-46 X X Southwest Youth Collaborative –Service X X Springfield Public School District 186 X X Thornton Township High School District 205 X Urban Solutions Association X X Urbana School District 116 X Waukegan Community Unit School District X X Westside Health Authority X X Will County ROE X X Zion Elementary School District 6 X Total Awards 32 17 45 16

(a) Alton FY03 grant combined with FY05 grant. (b) Maywood, Melrose Park, Broadview assumed by Proviso/Leyden.


Each of the grants provides services at one or more sites. Exhibits 3 and 4 show the locations of the 388 sites in the 110 grants. The majority of locations are in the Chicago Metro area; however, sites are around the state in urban, suburban, and rural areas. In 2006, 61.5% of the 40,387 21st CCLC site attendees were regular attendees (n=24,851; attended 30 or more days).

Exhibit 3. 21st CCLC Sites in 2006-2007 in Illinois


Exhibit 4. 21st CCLC Sites in Northeastern Illinois in 2006-2007

Methodology The formative evaluation used a multi-source, multi-method approach that combined quantitative and qualitative data. Analyses included anecdotal, descriptive, and multivariate methodology. This approach enabled researchers to collect the myriad of data needed to address the breadth of the objectives and the organizational, programmatic, strategic, and implementation characteristics of the various sites.


Appendix A links the performance indicators and measurement sources to specific research questions and provides more information how the instruments used. The primary and secondary sources included the following:

Primary Sources 1. Telephone interview with each 21st CCLC award administrator and/or staff

• Interviews conducted in February through May with all 72 organizations that received at least one award

• Average length 46 minutes • Additional follow-up phones calls and surveys used to garner more in-depth

information for selected organizations 2. Telephone or in-person interview with ISBE staff knowledgeable of the program

(February through July) 3. Site visits to 41 organizations (56% of all of the 21st CCLC active awards in 2006-2007)

• Included sites from southern Illinois, from central Illinois, and from Chicago and the collar counties

• Conducted visits and surveys from February through July • Site visit interviews with the program administrator, on-site teachers and tutors,

parents, and partners, as possible • Random selection (lists provided by on-site administrator) of parents, classroom

teachers, school administrators, and community partners for follow-up phone and/or mail surveys

Secondary Sources 1. Original applications for funding and annual requests for continuation for FY 2003

grantees reviewed at the ISBE offices in Springfield, Illinois. 2. ISBE Fiscal Program Reports obtained on-line from the FRIS system of ISBE 3. Program documentation and professional development records provided by ISBE staff 4. Annual Performance Report and State of Illinois Summary Reports from the 21st CCLC

Profile and Performance Information Collection System (PPICS). Latest performance data available was for program year 2004-2005 and other reports through 2006. (http://ppics.learningpt.org/ppics/reports/2005APRPDFS/IL.PDF).

5. Interactive Illinois Report Cards available on-line from Northern Illinois University (http://iirc.niu.edu) and data files from IIRC staff

6. State assessment file of individual-level student assessment data for ISAT and PSAE with identifiers for students participating in 21st CCLC programs statewide provided from IIRC through ISBE

The research was conducted during September 2006 through July 2007 by a team of NIU researchers and consultants. All organizations receiving a 21st CCLC award were called via telephone or by e-mail by NIU researchers. During this correspondence, appointments were established for a phone interview with the organization’s 21st CCLC program director and/or staff members who could best answer the interview questions, which were provided before the phone conversation, as requested. Visitation dates were arranged for those sites selected for a site visit, and organizations were sent materials outlining the site visit protocols and requests for interviews and/or focus groups with the administrator of the grant, site personnel, teachers, tutors, school officials, community partners, and parents.

Data from program directors were collected in 2006 from all but one grantee and from all grantees in 2007 through the telephone or in-person interview.


(http://ppics.learningpt.org/ppics/reports/2005APRPDFS/IL.PDF)

(http://iirc.niu.edu)

Interview and survey data were collected from 21st CCLC staff, school staff, parents, and community partners from the grantees receiving a site visit. Because the number of surveys returned varied from 2 to 78 per award, results were averaged by award, then by organization receiving the award, and then compiled. No statistical differences were found among the perceptions of the 21st CCLC administrators and staff, the teachers at the school, the parents, and the partners; thus, the data were aggregated by grant site.

The data from the primary and secondary sources were compiled by grant and summarized by research question. Two researchers independently reviewed the data and compared interpretations and codings. Any differences in interpretation were judicated through follow-up questions to the appropriate source.

In summarizing the results, the researchers did not search for causal or definitive findings. The limitations of the data did not warrant those types of interpretations: the multiple sources of data have varying degrees of reliability and only a selection of sites provided the more in-depth, on-site evaluative data. The research, however, was useful in providing information on the implementation and outcomes of the 21st CCLC program in Illinois.


Part I: Formative Evaluation Findings

The research findings are reported in parallel to the three sections of research: • Evidence of Meeting Program Objectives • Quality of Profile and Performance Data • Status of Recommendations from Previous Formative Evaluation

As a caveat, this program evaluation included open-ended survey instruments to collect data from all awardees and case study methodology for the site visits and follow-up surveys. Program administrators were asked to identify characteristics, findings, and conclusions that were the most important; therefore, the percentages reported for these findings do not represent the total domain of all programs with that characteristic. Likewise, case studies cannot be generalized to the total domain; however, the data provides a richer context to help understand the overall findings.

Evidence of Meeting Program Objectives Seven questions and their performance indicators were used to assess how well the Illinois 21st CCLC projects are meeting the objectives established for the 21st CCLC program by the State of Illinois.

STUDENT BEHAVIOR AND ATTITUDES (Objectives 1 & 3)

What effect does the program have on youth behaviors as measured by changes in classroom behavior, attendance rates, involvement in school activities, attitudes toward school and learning, disciplinary referrals, and dropout and graduation rates?

This research question addressed both the first and third Illinois objectives, which related to student behavior and attitudes. The sources of evidence included the interviews and surveys of the program administrators, site staff, teachers at the school, and parents.

Changes in Classroom Behavior The teacher survey asked respondents to rate the degree to which students’ behavior changed, using a scale of 1 to 7 with 1 being “significant decline” and 7 being “significant improvement.” In addition, respondents could indicate that the students “did not need to improve.” The results in Exhibit 5 represent the improvement indicated for only those students who needed to improve.

Over 61% of the students improved in “behaving in class,” over one-third of the students who need to change still have not done so. About one-tenth are declining in behavior. The results are not significantly different from those in the State of Illinois 2005 Annual Performance Report of all sites,4 in which 58.9% of the teachers indicated an improvement in students’ classroom behavior and 15.12% noted a decline.

4 Learning Point Associates, Illinois 21st CCLC 2005 Annual Performance Report, Generated June 2006, www2.learningpt.org/ppics. See


Attendance Rates The site survey data from 2006-2007 and the 2005 Annual Performance Report for Illinois indicate that more than half of the teachers and staff thought 21st CCLC students improved their day-school classroom attendance (2006-2007 data – 51.7% versus 2005 data - 57.9%).

Attendance in the 21st CCLC program is related to classroom attendance for many of the projects: students must be in the classroom during the day to be eligible to attend the after-school activities. About 18% of all 21st CCLC program administrators mentioned that student attendance in the after-school program was irregular due to competing programming with SES, sports, clubs, and other activities, especially at the junior high and high school level.5

Compared to the 2006 evaluation, more grantee administrators are mentioning the problem of competition, especially for those programs serving specific ethnic populations (15% versus 18%).

Programs with consistently high enrollments over the past two years6 were examined in more detail. The program administrators of these programs listed the following as their two primary factors associated with program success: 1) ensuring programming is meaningful, active, and interesting and 2) building relationships with students and parents.7 These findings match those in last year’s evaluation.

Exhibit 5. Staff Perceptions of Student Improvement in Behavior and Performance

Source: Compilation of Staff and Teacher Surveys, 2006-2007

5 NIU, Interview of Program Directors of 21st CCLC Awards, 2006-2007. 6 Attendance rates of over 85% during the past two years. 7 NIU, Interview of Program Directors of 21st CCLC Awards, 2006-2007.


Involvement in School Activities According to the interview of program directors, less than 5% of the students at the middle school level but over 45% of the students at the high school level participate in other activities offered afterschool by the school. Further study should be conducted to determine why so few of the middle school students were involved in the other activities.

Attitudes toward School and Learning Program administrators, teacher/tutors, and parents gave numerous anecdotal examples of how participation in a 21st CCLC program completely changed a student’s attitude and life. Over 89% of the program administrators mentioned the impact the program has had on at least one individual student. A survey of 90 parents of students indicated that 71.1% of their children who needed to improve their attitudes towards school and learning had done so.8

Disciplinary Referrals Program administrators and teaching staff reported fewer disciplinary referrals and behavior problems; however, over half of the program administrators mentioned that the improvements might be attributable to school-wide implementation of behavior programs, such as PBIS, which are also consistently applied in the 21st CCLC activities.9 These findings were similar to those found in the 2006 statewide evaluation.

Dropout Rates and Graduation Rates Program directors of nearly 90% of the sites serving high school students believed the 21st CCLC program was instrumental in keeping students in school and helping them graduate. All of the program directors of sites offering credit recovery programs reported lower dropout rates.

STUDENT ACHIEVEMENT (Objective 2)

What impact does the program have on student achievement, including homework completion, classroom grades, promotions, and performance on the state assessments?

This research question was addressed through the project administrator interviews, interviews and surveys of site personnel, classroom teachers, and parents, and an analysis of the State of Illinois student performance data.

Homework Completion Almost all program administrators mentioned homework assistance as a component of their 21st CCLC project. About one-fourth identified one of the key outcomes of the program was that students go to school prepared to learn because they have their homework completed. The survey of staff, teachers, and tutors at the site visit locations confirms this finding--those surveyed indicated nearly two-thirds of the students improved in completing their homework (Exhibit 5).

8 NIU, Parent Surveys from 41 site visits and follow-up phone calls, 2006-2007. 9 NIU, Teacher/Tutor Interviews at 41 site visits, Interview of Program Directors of 21st CCLC Awards, 2006-2007.


Over one-fourth of program staff (28.3%) reported that students often complete homework in the 21st CCLC program but fail to turn it in the next day. About 52% of the program administrators wish they could improve the coordination of homework with the classroom teacher. Further discussion is needed if there are ways to improve the coordination of homework.

Classroom Grades About one-fourth of the program administrators surveyed mentioned that the improvement of classroom grades is a major outcome of the project. In the 2005 Illinois Annual Performance Report, less than half of the students in the program increased their grade in reading (44.8%) or math (39.2%) by a half-grade or more.

Given the fact that grading scales and criteria differ between and within schools, this is not an unusual finding. Grades are complex measures based on a multitude of factors.

There is a relationship, however, between changes in half-grade increments and the attendance pattern of the student. Students who attended 90 or more days were more likely to increase their grades than were those students attending for fewer days. This finding supports current research on “dosage of intervention” and reinforces the need to retain students in the program.

Exhibit 6. Percentages of Students with Half-Grade Changes in 200610

Elementary Middle School and High School Reading/LA Math Reading/LA Math

30-59 Days

60-89 Days

90 & over

30-59 Days

60-89 Days

90 & over

30-59 Days

60-89 Days

90 & over

30-59 Days

60-89 Days

90 & over

Increase 39 38 46 34 32 45 38 40 51 31 36 48 Same 44 46 39 45 44 36 39 35 29 40 38 31 Decrease 17 16 15 20 24 19 24 25 19 29 27 22

Academic Performance As shown previously in Exhibit 5, approximately three-fourths of the staff, teachers, and parents surveyed in 2006-2007 as part of the 41 site visits indicated that students in the 21st CCLC program improved in terms of academic performance. The 2005 Illinois Annual Performance Report found that 72% of the teachers surveyed felt similarly.

Students often are referred to the program because they are at-risk of failure. Anecdotal evidence of how students improve academically while in the 21st CCLC program is abundant, mostly through perceptual data. The question becomes one of magnitude--did the 21st CCLC participants increase sufficiently to equal their peers who are not attending the 21st CCLC program?

State assessment data for the 2006 ISAT exams were provided by ISBE through the Interactive Illinois Report Card project. Data for those schools with at least one 21st CCLC participant were aggregated to determine if there were any statistically significant differences (p < .01) between the performance of students in the 21st CCLC program and their classmates (those in the school at the same grade who were not in the program). Even though the classroom teachers and program administrators recruited low-performing students for the 21st CCLC programs, students who attended the after school program regularly did not differ significantly in their

10 Learning Point Associates, www2.learningpt.org/ppics, July 2007 report on 2006 data.


academic performance on the reading and math ISAT exams than their classmates who did not regularly attend the 21st CCLC programs. The ISAT data, however, is limited because of when and how the students have been flagged in the database. To confirm analysis, the ISAT data provided by the grantees was compared to the official ISAT data files. The finding was duplicated in the self-report data-- students in 21st CCLC programs and students not attending any 21st CCLC sessions were not statistically different in their achievement. Further study is needed to isolate students by the “dosage of intervention” to determine if changes are noted in the ISAT scores.

Compared to previous evaluation, great strides have been made in flagging the 21st CCLC students in the state database. Follow-up conversations with selected program administrators, however, revealed some sites are still having difficulty in sharing data between the school and the community based agency.

Exhibit 7. Comparison of 2006 Reading Scores on Illinois State Assessments11

0

50

100

150

200

250

300

21st CCLC Classmates

21st CCLC 187.3 204.0 209.3 221.6 228.2 240.3

Classmates 190.8 204.7 210.9 222.5 229.2 238.7

3rd 4th 5th 6th 7th 8th

Exhibit 8. Comparison of 2006 Mathematics Scores on Illinois State Assessments12

0

50

100

150

200

250

300

21st CCLC Classmates

21st CCLC 196.4 215.5 222.6 233.8 239.3 255.8

Classmates 200.8 214.9 223.0 234.1 241.9 254.5

3rd 4th 5th 6th 7th 8th

11 Interactive Illinois Report Card Project, Northern Illinois University, 2007. No statistical differences at p < .01. 12 Interactive Illinois Report Card Project, Northern Illinois University, 2007 No statistical differences at p < .01.


Other Findings on Student Achievement The 21st CCLC program administrators provided some insights into what the centers are doing to increase student achievement:

• When program administrators and staff were asked to identify the best practices in increasing students’ academic performance, the two most noted answers concerned improving pedagogy and coordinating the “day” school with the 21st CCLC program. This was the most often noted practice in last year’s evaluation also.

• Improved pedagogy was noted most often by community-based providers, whereas school districts were most likely to see the alignment with the “day” school as most important.

• When asked to describe the “improved pedagogy,” respondents pointed to the individualized instruction provided through tutoring, increased use of differentiated instruction, and using approaches to reduce the ratio of teacher/tutor to student.

• Improved pedagogy also included using more hands-on activities. • Coordinating the “day” school and 21st CCLC program most often involved

communication of what happened in the day, any special concerns with individual students, and information on the types of academic problems with which the students may need help.

• Students in the 21st CCLC programs often were referred to the program because they were not succeeding in the traditional classroom. In the before- and after- school programs, teachers are trying new techniques and approaches to see what will work with individual students. For example, credit recovery programs are used especially in high schools to keep students progressing towards graduation.

PARENT AND COMMUNITY INVOLVEMENT (Objective 4)

In what ways does the program serve the parents of the program participants? Is there increased involvement by participants' parents in regular school activities?

The lack of parent involvement is a common concern of school systems across Illinois. This is true even for schools receiving awards for exemplary academic improvement and achievement.13

Similar to the findings in last year’s statewide evaluation, program administrators described the enrichment and support services provided to the families of the participants. The approaches used to involve parents ranged greatly across the 21st CCLC programs—from traditional approaches described by one program administrator as “build it and they will come” to programs in which parents were integral members of the decision-making process and as necessary components of the program. Older, well-established, community-based centers and organizations were more likely to be at the inclusive end of the continuum. These organizations, also, tended to have more established programming offered through the center or other on-going partnerships established before the 21st CCLC program.

13 For example, Billman, P.S. (2004). Mission Possible. DeKalb, IL: Northern Illinois University.


Parent Perception of the Program Data collected from parents from the sites visits revealed that parents vary in their expectations of the 21st CCLC programs. A significant portion of parents look to the 21st CCLC to help their child with homework and ensure that it is completed. The majority of parents (82%) did not see the 21st CCLC as important in meeting their own personal needs.

Approaches Used to Involve Parents Over 46% of the program administrators interviewed mentioned “lack of parent involvement” as a critical factor hindering the project. The 21st CCLC sites have tried various approaches to engage more parents and have found varying degrees of success.

Programs managed by school districts tended to leverage traditional involvement practices, such as Family Nights, PTO approaches, and report card pick up. Both school districts and community-based organizations found that parents were more apt to come to events in which their children were performing.

Impact of Parent Involvement How much impact does overt parent involvement have on student achievement? All of the sites were rated as being high, medium, or low in terms of parent involvement. When a site was the sole provider of 21st CCLC programming to a school, the achievement data of the students were aggregated to produce a mean score for that site. There were no statistically significant differences among the mean achievement scores for sites that had high, medium, or low overt parent involvement.

On the other hand, data from the teacher surveys and interviews were used to categorize sites as having strong, medium, and low parent support. On the surveys and during interviews, teachers were asked to rate the degree of support they felt from the parents. When the mean ISAT scores were analyzed, statistically significant differences were found between schools with teachers who felt they had strong support from parents and schools with teachers who felt they had low levels of support from parents (F=2.1573, p < .01).

These data suggest that the critical factor may be whether or not the parent supports the school, not whether or not they are physically present at school events or involved in school activities. This preliminary finding will be further explored.

Referrals to Other ServicesMost of the program administrators reported that they already had referral connections in place through previous programming, which they tapped into for the 21st CCLC project. This was especially true for community not-for-profits and public school systems. The 21st CCLC project, however, did afford the opportunity to expand the number and types of referral resources, especially in projects with many partners. This finding replicates that reported in 2006.

Collaborations with Partners Interviews and surveys of the community partners (n=31) conducted in October 2006 through May 2007 asked these organizations to identify why they were involved. The majority of the partners saw this as a way of serving the students and community; however, 48% are concerned about their ability to continue with the program after the funding cycle ends. Because programs rely heavily on the partners to provide programming, the 21st CCLC sites may need to reassess what programming is critical if the site is to be sustained.


SERVING SCHOOLS WITH GREATEST NEED (Objective 5)

Did the RFP award process result in programs being awarded to serve the children and community members with the greatest need?

A competitive RFP process was used for the 21st CCLC program. The process and eligibility requirements are documented on the ISBE website. Proposals to serve high need areas are given preference points in the scoring rubric. Proposals undergo a peer review process in which reviewers independently rate the proposals using the criteria articulated in the RFP. Peer ratings are used to prioritize the proposals for funding.

Selection and recruitment policies for participants vary from site to site, ranging from open admissions to very strict criteria based on income, test scores, grades, and/or behavioral or are eligible for free or reduced lunches (ISBE ISAT data, 2006).

PROFESSIONAL DEVELOPMENT (Objective 6)

Did the professional development activities provided through the State of Illinois to 21st CCLC program personnel adhere to No Child Left Behind Act definitions and the National Staff Development Council’s professional development standards?

Four ISBE consultants are assigned awards to monitor and provide assistance. In addition, ISBE provides mandatory staff development workshops. In fall 2006, a mandatory conference was held in three locations across the state. In addition, grantees were required to attend sessions on sustainability and evaluation. Both the sustainability and evaluation sessions were found to be helpful by nearly 90% of the grantees. ISBE staff members keep program administrators informed of items of interest through e-mail. The ISBE actions and activities were found to comply with the NCLB and NSDC standards.

In addition to the mandated professional development, individual 21st CCLC grantees provided development for their personnel. In fact, only 18% of the organizations used only the mandated professional development. Program administrators were asked to describe the most valuable professional development afforded to their staffs. Similar to the 2006 statewide evaluation results, nearly one-fourth of the centers described extensive in-house training, 22% mentioned recognized external experts, 18% rated Regional Office of Education programs as most valuable, and 18% mentioned sending staff to national conferences. Larger community-based agencies tended to provide more professional development training than did smaller agencies. School districts often provided professional development through other funds. Professional development was one of the first items to be downsized as grantees received decreased funding in the final years of their grant.

When asked what they would like offered in the coming year, the majority of grantees (56%) mentioned the need to re-think the format of the large sessions. Even though they found the evaluation and sustainability meetings helpful, they were frustrated at times because of the wide variety of needs of the participants. For example, in the sustainability training, seasoned experts in professional fund raising were grouped with novices with many questions. In order to make the material relevant to the inexperienced, some of those with strong backgrounds


thought that the sessions tended to be too long, too frequent, and/or less “meaty” than they could have been. On the other hand, some of the novice grantees found the material to be “overwhelming”. Regardless of their point of view, the grantees realized the benefits from learning from each other and did not have a solution to the dilemma of balancing the needs of the expert and the novice.

As part of the external evaluation, several grantees were contacted to talk about the evaluation and the type of data collected. A common recommendation from these program directors was that the evaluator work with the grantees to build a benchmarking tool that could be used in helping grantees identify areas of strengths and weaknesses as compared to other grantees. These data could be used to identify professional development that was needed for the continuous improvement of their program.

A significant percentage of program directors (46%) requested more professional development be made available for the front-line workers, especially in the area of positive behavior techniques, alternative instructional approaches to engage students, the Illinois State Standards, and how to recognize and diffuse student conflicts before they surface.

Administrators were most likely to ask for programming concerning legal issues, such as background checks, liability issues, and personnel issues.

Several program administrators who have attended national conferences recommended ISBE bring in some national speakers or representatives from nationally recognized exemplary programs for the mandatory meetings.

Over three-fourths of the program administrators (75.6%) requested more time at mandatory meetings and/or additional opportunities be available for grantees to network, share ideas, and learn from each other. Some of the time should be dedicated to similar programs sharing ideas, and some of the time for different types of programs to hear from each other. Similar to the findings in the 2006 evaluation, the majority of program administrators were interested in learning how other sites are structured, how instruction is scheduled, how they overcame problems, and what is working. They are looking for innovative ideas for academic activities, use of community partners, tutoring, enrichment activities, and day-to-day operations. In addition to learning about their peers in general, subgroups included those in rural programs, early childhood programs, high school programs, and summer programs. High school programmers were most interested in how to retain and motivate students.


SUSTAINABILITY (Objective 7)

What are the current efforts toward providing for sustainability of the current programs, especially of the programs in their final year of funding?

The majority of program administrators (83%), especially those in public education, felt replacing the 21st CCLC funding at the current level is nearly impossible. Each award has a sustainability plan; however, on average, the majority of the programs do not begin thinking about sustainability until late in year three of the grant.

Nearly all of the 2003 grants did not have sustainable funding identified at the time of the interviews; however, a few community-based organizations believed they had 100% of the funding identified for the next year.

Exhibit 9. Percentages of 2003 Grantees with Sustainability Funds

Re-applying for the 21st CCLC grant was the strategy most of the 2003 grants were going to use. Compared to the previous year, program administrators were more apt to identify strategies they wanted to pursue, and some (26%) talked about avenues that they heard about during the sustainability training.


Exhibit 10. Most Common Approaches to Sustainability

Approach

# of Awardees Mentioning

Using Approach (n=110)

Re-apply for the 21st CCLC Grant 95 Other grants and foundations 64 Replace funding with specified alternative sources 57 Fundraising 56 Increase awareness of program in the community through public relations, marketing, newspaper articles, etc.

34

Use grant writer 18 Reallocate other Title or federal funds 17


Quality of Profile and Performance Data Two research questions evaluated the process and quality of data collected on the 21st CCLC program.

USEFULNESS OF DATA COLLECTION AND EVALUATION RESOURCES

Did the 21st CCLC program personnel find the data collection methods and evaluation resources, in particular PPICS, useful and relevant in documenting their programs and outcomes?

In general, program administrators (86.7%) found the PPICS system useful; however, many (42.6%) voiced concern that changes to the types of data to be collected are not always communicated at the beginning of the year.

Several grantees mentioned that they had increased the accuracy of their data collections systems used for PPICS.

OTHER DATA

Would additional data and/or data collection methods have helped document the outcomes of the programs and provided supplemental information for decision-making?

Compared to last year, significantly more program administrators (23.4% compared to 36.2%) described the use of pre- and post-testing of students as a way to individualize or differentiate instruction. Similar to last year, over 20% of the program administrators mentioned using auxiliary data systems created to track individual student performance.

Nearly one-third of the program directors wished they could easily access benchmarking data on the Illinois 21st CCLC programs.


Status of Recommendations from Previous Formative Evaluation Several recommendations were offered for consideration in the 2005-2006 Illinois Statewide Formative Evaluation. The following summarizes the status of the recommendations as of July 2007. ISBE and the individual grantees have made good progress in addressing the recommendations from last year. Initiatives are in place and funded to continue the progress. In the exhibit below, the progress made on each of the recommendations is shown. An empty thermometer indicated little or no process, whereas a completely filled (red) thermometer indicates the recommendation has been completed.

Exhibit 11. Status of Recommendations from Statewide Evaluation in 2005-2006

Indicator Recommendation From Last Year Status or Action Taken

Improving Instruction

Each program should maximize the amount of time students have to connect with a positive role model in a one-to-one or small group setting.

Approximately one-third of the programs mentioned that they are trying to keep the adult-to-student ratio low. Last year, only one-fourth noted this goal.

The 15% to 20% of students showing a decline in behavior should be tracked to determine why they were not successful and if other interventions or changes in the 21st CCLC program could have helped them improve.

Similar rates were found in declines in behavior. More research is needed to address this recommendation.

ISBE could create an instrument for 21st CCLC programs to rate themselves on quality factors, such as how well they engage individual students, how well they align instruction to the Illinois learning standards, and the effectiveness and types of pedagogical approaches used. Based on the ratings received, programs can be directed to appropriate professional development materials and opportunities to visit exemplary peers.

Selected program administrators discussed this with the external evaluator. Funding has been provided to create an on-line benchmarking instrument in FY 2008-2009.

Parent Involvement The 21st CCLC programs experiencing difficulty in recruiting and retaining students should reassess the programming offered to determine if it is meaningful and interesting to the students.

The use of an external or peer evaluator should be considered. The technical assistance budgeted for 2008 should be able to help with this recommendation.



Recruiting and Retaining Students The 21st CCLC programs experiencing difficulty in recruiting and retaining students should reassess the programming offered to determine if it is meaningful and interesting to the students.

The use of an external or peer evaluator should be considered. The technical assistance budgeted for 2008 should be able to help with this recommendation.

Changing Leadership Seamlessly Because the roles of program administrator and site coordinator are critical to the success of the program, changes in leadership in either of these two positions can affect the effectiveness of the 21st CCLC. When such changes occur, customized professional development activities may be needed. ISBE may wish to consider providing exemplary 21st CCLC programs with additional funding to help centers in transition.

Fewer programs had changes in leadership this year. A promising practices project is underway to identify exemplary programs. This information is now tracked on the continuation form. The technical assistance budgeted for FY 2008 can help with this also.

Selecting Strategic Partners Programs should strategically select and maintain partnerships that enhance their capacity to provide specific programming and services needed to best serve their students and families.

Maintaining quality partners is still challenging, especially as funding decreases. For at least 5% of the sites, program administrators eliminated some partners to allow more time to focus on academic skills development.

Growing Quality Staff Through Professional Development Program administrators should continuously assess the skills of the staff and provide necessary professional development to ensure that all tutors and front-line personnel build positive relationships with the students.

Compared to last year, more program administrators requested specific training for their front line staffs. The technical assistance funded for FY 2008 will address this issue.

Increase the amount of time at mandatory meetings dedicated to networking and sharing of promising practices.

In 2006-2007, the mandatory meetings, especially the sustainability and measurement training sessions, provided time to network and share ideas.

Investigate ways to allow grantees to electronically share information.

Two initiatives are currently in planning stages that will address this. ISBE issued a request for technical assistance and will be awarding a contract soon, which stipulates that the provider create this platform. The second funded initiative will create an on-line benchmarking tool.



Find ways to facilitate site visits among grantees with common concerns.

In the past year, the external evaluator has connected numerous programs with common concerns with each other. Some of the program administrators have followed up with site visits. The technical assistance funded for FY 2008 will address this issue.

Provide dedicated development activities on sustainability, improving evaluations, communication, and addressing common staffing issues.

In the past year, the first two topics were covered in a series of mandated meetings. The continuation form asked grantees to provide the types of professional development needed in the coming year. This information will be used by the technical assistance provider to create web-based programs.

The timing of mandated trainings should be carefully considered. Late spring meetings can conflict with end-of-school activities and be difficult to attend.

The mandated trainings were scheduled throughout the year. There were some concerns voiced by program administrators that they did not have sufficient lead-time to accommodate some of the trainings.

Program administrators should receive professional development in how to use testimonial and case study data to effectively document student growth.

This was covered in the mandatory professional development sessions this year.

Exemplary sites need resources to share what they have learned with newer sites to ensure the progress made so far will not be lost and to help newer sites build capacity quicker. Sites could be asked to formally provide examples of promising practices to be made available to all 21st CCLC projects and to include contact information to facilitate site visits and follow-up by other sites interested in learning more about the practice.

The external evaluator is piloting a promising practices form at this time to be used in the 2008 evaluation and benchmarking initiative. The technical assistance provider will help with this concern as well.

ISBE should continue to focus on linking 21st CCLC projects together, especially for professional development that might be more cost effective and as a way to build a network among the grantees. Annual requests for continuation of funding should request grantees to specify the professional development activities anticipated during the next year.

The continuation form was changed to include this item. The technical assistance funded in FY 2008 will address this as well.



Sustaining Programs The U.S. Department of Education and state boards of education should help 21st CCLC programs become sustainable through entrepreneurial, innovative approaches and/or additional funding.

ISBE funded mandatory sustainability training for all grantees. The 2008 competition provided an avenue for programs ending in 2007 to submit proposals and compete in the new competition. There are concerns if the 2008 cohort will have the same opportunity.

Most projects are well into the grant cycle before energies are directed toward sustainability. Sites falling behind should be counseled through professional development, one-on-one assistance from ISBE, and/or through mentoring by an exemplary site administrator.

The mandatory sustainability training addressed this recommendation, and the technical assistance funded for 2008 will address these issues.

Improving Data Quality Standards of quality for the reporting of PPICS data should be established. The accuracy, reliability, and validity of the data have not been established. The methodology used in collecting survey data, steps taken to prevent selection bias, and the confidence intervals for interpretation are not reported. The PPICS data should be perused for discrepancies in data, provide information on the methodologies used to collect the data, and include the reliability and validity information, including confidence intervals for survey data.

The 2006-2007 annual reports through PPICS are not available at this time. Once released, comparisons among PPICS and the various data sources used for the external evaluation will be compared.

Add unique identifiers to the Illinois student achievement records to allow changes in academic performance to be tracked by student from year to year.

This is under discussion at the state level for implementation.

A strength of the 21st CCLC program is the social and emotional growth of the students. Pre- and post-tests of established reliability and validity should be considered as a means of documenting this growth.

The external evaluator will work with an advisory group to explore the use of this instrument for a selection of sites during the external evaluation in 2008.

Grantees, schools, and ISBE need to ensure the students in 21st CCLC programs are coded as such on the student achievement records along with their level of participation. Provide any necessary professional development to accomplish this.

The 2006 student performance data provided by ISBE through the IIRC had significantly more students coded as 21st CCLC when compared to last year. The continuation form asks the grantee how they are addressing this. The SIS system at the state level does not allow for late identification.



Documenting program success involves looking at how far individual students progress. A student may not meet the state standards; however, he may significantly decrease his achievement gap. Pre- and post-data needs to be available for each student in the 21st CCLC program. The evaluation of this data should be in terms of gains made on decreasing the achievement gap. In order to use the state student performance data, unique identifiers would need to be used.

This has not been resolved.


Part II: Summative Evaluation Findings

In 2007, the first cohort of 21st CCLC grantees funded in the state competition in 2003 ended their funding cycle. All of the grantees except one hosted an on-site visit in 2006 or 2007. During the visit, researchers collected data to identify (1) the critical factors that facilitated or hindered the grant over the 5-year period, (2) the differences between programs in mid-cycle and at the end of the funding cycle, and (3) the specific features or characteristics associated with exemplary outcomes.

Factors Impacting Program Implementation and Outcomes Researchers collected data across the multiple sources in order to asses the level of implementation of the grant and to identify factors that helped or hindered the implementation of the program and realization of the outcomes. Before looking at specific factors, researchers looked at how the implementation of the grant changed over time.

To what extent are grant recipients implementing the activities and evaluation plan proposed in their RFPs, as revised in their annual continuation requests?

Over the five-year period, the majority of programs changed in various ways. Three changes were most evident: time the site was open, lack of adherence to items in the proposal, and changes in partners.

The most common adjustments were changes in the hours of operation, especially the elimination of Friday programming by many of the grantees or adjusting the ending time to accommodate parents. Some programs decreased the number of sites due to closing of schools, inactivity at the site, and/or lack of coordination between the school and the after-school program. Some sites found it necessary to go to a team teaching model to keep qualified staff.

All sites provided programming within the recommendations of the legislation; however, the focus of the instruction changed. Sites grappled with the pressure from parents, and often teachers, to focus heavily on homework completion. In fact, the majority of the sites found it necessary to work closely with parents and teachers to help them better understand the intent, guidelines, and goals under which the 21st CCLC operated.

In order to implement their activities, 21st CCLC staff found it necessary to create an ongoing dialogue and partnership with the school day staff and administrators. The day-to-day issues consumed enormous amounts of time.

Programming as articulated in the original proposal changed; however, some form of programming continued. This was not the case with the Evaluation Plan and the Sustainability Plan. A random selection of 2003 grantees (n=10) was tracked from their proposal to the last year of operation. The majority of the programs began with a measurement plan and often worked with an external evaluator. Over time, however, especially when budgets became tight, the evaluation component was dropped and staff became the de-facto evaluators. This is not to say that evaluation stopped, though it did in some cases. Rather, the evaluation component was greatly reduced and relied heavily on survey methodology.


The 2003 cohort had a wide variation in the implementation of their evaluation plans. In 2006, over one-fourth used external evaluators and over 15% used grant writer/evaluators to document the outcomes of their project. About 10% described a comprehensive, integrated evaluation process in which data is used for substantive project planning and improvement. On the other hand, over 10% of the awardees barely collect the minimum PPICS data and do not use it for program improvement

The sustainability plans, also, changed. During the first two to three years, most programs were so busy trying to implement day-to-day programming that they did not have time to address sustainability. By year four, program administrators were beginning to worry about sustainability and had to implement something quickly, often very different from what was originally proposed.

The parent and community focus, also, was an under-implemented component of the proposal. Programs offered activities and involvement avenues to the parents; however, most sites were fortunate if they could maintain a core of parents to be involved.

Maintaining the partnerships was a tremendous challenge for many of the grantees, especially as funding decreased. The strongest relationships appear to be between a grantee and a partner with whom the grantee has had a long-term, evolving relationship. At first, many grantees underestimated the time, energy, and resources needed to maintain the partnership.

Which factors hinder and which factors facilitate reaching the objectives?

Interviews with the 32 program administrators included questions on which factors hindered and which factors facilitated successful outcomes of the 21st CCLC project. Two raters independently coded the factors into the categories. The categorizations were compared; differences were investigated and judicated by contacting the appropriate 21st CCLC program for clarification. The percentages below reflect those 2003 grantees who offered the factor in an open-ended question; it does not reflect the domain of those actually using the approach.


Exhibit 12. Factors Mentioned Most Often as Facilitating Positive Outcomes

Facilitating Factor # of Respondents

Mentioning Factor as Important (N=32)

Build strong relationships with all constituents 28 The students form strong relationships with the program staff, and these relationships lead to motivation to attend the session and the ability of staff to identify the programs and services a students needs the most

16

Provide small group or one-on-one instruction, low ratios 16 Use classroom teachers to staff center 14 Create a safe haven 14 Incentives and rewards 12 Cooperative relationship and sharing of information between classroom teachers and 21st CCLC staff

9

Create individual instructional plans for students based on pre- and post-testing and tracking

8

Clearly articulated behavioral expectations, the same as those used during the day and uniformly enforced

6

Match programming to needs of students; eliminate activities and partners with little value-added

4

Exhibit 13. Factors Mentioned Most Often as Barriers

Hindrance

# of Respondents Mentioning Factor as

Important (N=32) Difficulty in getting parents involved 29 Home life is not good for student 20 Difficulty in recruiting students 18 Difficulty in working with schools 18 Inconsistent attendance of students 18 Lack of classroom teacher buy-in and/or cooperation 14 Transporting students expensive and logistical problem 12 Influence of gangs 12 Peer pressure to use drugs 15 Lack of partners for key aspects of programming 14

.


Differences in Mid-Cycle and End-of-Cycle Programs When the summative evaluation was designed, researchers hypothesized that programs would move along a continuum over the five years. The first year would be primarily one of building relationships and establishing relationships. The middle years would be one of tweaking the programming to better meet the students needs, and the final years would be characterized by a smoothly running program with administrative staff time allocated more on sustainability issues. In some ways, the hypothesized continuum was correct, and many of the more effective programs did complete this track. For other programs, however, it became extremely difficult to move beyond the labor-intensive beginning tasks of building relationships.

The success of a 21st CCLC program is related to the health of the partnerships and relationships needed to provide effective day-to-day programming. The majority of program administrators described how time consuming the partnership-building process is. Their most common advice to fledgling programs included identify point people for each of the partners, be flexible, and have contingency plans in place at all times.

Characteristics of Effective Programs Data on the 2003 grants were coded to identify those grants that were in the top quartile on the following indicators: student achievement and positive behavior; parent and community involvement; strength of partnerships; sustainability; and evaluation tied to continuous improvement. Of course, many of the variables were co-linear in nature and a complete hierarchical model was not possible with only 32 cases. However, three characteristics emerged as being related to being in the top quartiles. First, these programs focused on providing prescriptive interventions to individual students. They zeroed in on the student’s need in a variety of ways, such as diagnostic computer instructional programs, close communication with the classroom teacher, and pre- and post-testing of the student on instruments that identify areas to target.

The second characteristic involved the strategic focus of the program. All of the top-quartile programs focused on the student, but they also focused on identifying the students most likely to benefit from the program. For some programs, they identified the student on the cusp—the student needing more instruction time to learn the material. For others, the targeted students were those who had the largest achievement gaps to bridge. The common factor was these grantees knew whom they were serving and what those students needed. They did not try to be everything for everyone.

The final characteristics was the quality of the staff. Most often the staff were certified teachers, but not always. When non-certified staff were in high performing programs, the staff had rich and varied backgrounds and could relate well to the students.


Concluding Summative Thoughts All of the projects had the same aspiration--to help students grow academically and socially. Their philosophical orientations, however, differed greatly. As they struggled with the similar challenges of day-to-day operations, the community-based grantees and the school-based grantees approached the solutions differently and had different resources to help them solve problems. In the end, however, the community-based programs had to learn the way of the school district and the schools had to learn the ways of the community-based organizations. These relationships needed continual nurturing. In the interviews, the grant leaders were asked what advice they would give new programs. They often mentioned the need to build relationships, communicate, and other such partnership actions. Many wished they had had a mentor to help them through this process. As more cohorts are formed, there needs to be some way of linking the struggling sites with sites that had overcome the same challenges. There needs to be some lighthouses showing how partnerships can work.

As a researcher stepping back and looking at the data holistically, it becomes apparent that each site had a unique community with its own challenges and strengths. Some sites had high homeless populations; others had gang activity and violent acts, even death, of neighborhood youth. Some neighborhoods were in transition demographically; others had become static. Each grantee had to work within the constraints and strengths of their community.

Though many sites became proficient at student programming, only a few of the 2003 cohort had the resources to move to the next level. An effective 21st Century Community Learning Center was to focus on the word “community” as much as the word “learning.” The education research literature is filled with references to learning communities within schools--the challenge for 21st CCLC grantees is how to create a “learning community” for the entire community. How can centers get beyond mathematics and reading to something deeper and richer for the student and the community? What does that take in terms of resources and relationships? What does it look like to the students, parents, teachers, and members of the community?

Many of the grants had this “community” vision in the beginning; however, day-to-day operations tended to turn the attention elsewhere. Programming for students kept expanding, and staff members were doing more and more. Shooting buckshot at the target does not help you understand how you hit the bulls-eye. A critical lesson learned by the 2003 cohort was the need to follow their evaluation and sustainability plans in the proposal. Use them to keep on track and not lose sight of the vision. Use data to understand where you have been and how to make the changes to get where you want to be. Use your evaluation to look back and decide on which parts of the program to keep and which need to be abandoned because they are not getting you where you need to be. Use your data to provide evidence of what should be sustained to create a true community-learning center.


RECOMMENDATIONS FROM FORMATIVE AND SUMMATIVE EVALUATIONS

The 21st CCLC program serves a population that is at risk in a myriad of ways. Just as each site serves a unique population within a unique neighborhood, within each site is a group of unique students with unique needs. The power of the 21st CCLC program lies in its ability to be flexible and innovative in combining the resources of profit, not-for-profit, public, and private organizations in ways to provide the instruction, programming, and support services needed to help individual students, their families, and their community.

The 21st CCLC sites in Illinois are addressing the seven objectives and showing progress toward achieving them. The program administrators are passionate about the programs and have supportive partners.

This annual evaluation used multiple approaches to assess the individual 21st CCLC projects in order to identify what is working and to identify ways to improve the statewide implementation of the program. The summative evaluation section looked more closely at those programs completing their funding cycle to glimpse at how they evolved over the five years. Based on these findings, the following recommendations are respectfully submitted to ISBE and the grantees for their consideration.

The Evolving 21st CCLC Program The summative evaluation provided some insights into how the 21st CCLC programs evolve over time. Beginning with the original proposal, programs appear to move through stages of development—beginning with establishing partnerships, next to tweaking programming, and finally to sustainability. Some programs, however, appear to get caught at the beginning stages and have continuous challenges in establishing strong partnerships with the other stakeholders in the program. ISBE and the grantees may wish to consider the following recommendations that are designed to help programs evolve into effective, efficient programs that impact student academic achievement and build community.

1. The revised continuation form is a tool the grantee, ISBE, the external evaluator, and the technical assistance provider can use to track each program over time, ensuring the evaluation and sustainability plans are implemented and that the project has not strayed from its original vision and proposed delivery of services.

2. Programs which evolved effectively had a clear focus on who they served and strategically used resources to provide the highest quality programming to that population. The strategic plan for the year, the professional development, and the allocation of resources were aligned to achieve the program’s goals. he technical assistance model adopted by ISBE can be used to provide individualized, strategic planning help to the programs, as needed.

3. The evolving program needs accurate data on which to base decisions. During the next year, programs should continuously work to ensure the PPICS data and other data used for program improvement have acceptable levels of reliability and validity. This includes furthering the discussion on how to flag the ISAT/PSAE data in ways that reflect levels of attendance and have student identifiers to track individuals across years.


4. Because relationships provide the foundation for the success of the program, grantees, especially first-time grantees, might benefit from instruction on ways to use tools to assess the health of a relationship and to begin meaningful dialogues to build partnership capacity.

5. An advisory group could be formed to find solutions to problems such as a) decreasing the number of students who complete homework but fail to turn it in, b) increasing the involvement of middle school students in other school activities, and c) improving communications with parents so they understand the “community learning center” aspect of the program.

6. The “dosage of intervention” is related to student achievement; e.g., students who attend more regularly have higher achievement gains. Sites should focus on retention rather than one-time attendance at an event. Effective ways of marketing the importance of regular attendance to students and parents need to be identified and shared.

Developing the Front Line Effective 21st CCLC programs were adept at building relationships with the students and delivering high-quality instructional programming and enhancements. Those working directly with the students need a tremendous toolkit to help students learn. ISBE and the grantees may wish to explore cost effective and efficient ways of providing professional development to front-line workers, especially through the technical assistance provider funded in FY 2008.

Perceptions of Competition Program directors, especially in the Chicago Metro area, mentioned increasing competition among after school programs, especially those programs targeting a specific ethnic group. In addition, there is a perception that the pool of schools that will qualify for 21st CCLC funds is shrinking across the state. These issues need further discussion and documentation to determine how to best use 21st CCLC funds in Illinois.

In closing, the impact of the 21st CCLC program is difficult to measure but it is observable to those watching the programs change over time. Each program has testimonials documenting the “life changing” stories of their students. History can be used to predict what the outcome could have been for many of these students if they had not been in the program. The 21st

CCLC program in Illinois is creating a future of change, hope, and opportunity.


Appendix A

Research Matrix and Instrumentation for Formative and Summative Evaluations Research Focus Questions Performance Indicator Source of Data

FORMATIVE EVALUATION Objective 1: Participants will demonstrate an increased involvement in school activities and in participating in other subject areas such as technology, arts, music, theater, and sports and other recreation activities.

Objective 3: Participants in the program will demonstrate social benefits and exhibit positive behavioral changes.

1. What effect does the program have on youth behaviors as measured by changes in classroom behavior, attendance rates, involvement in school activities, attitudes toward school and learning, disciplinary referrals, and dropout and graduation rates?

� Involvement in school activities

� Participation in other subject areas

� Attendance rates � Graduation rates � Dropout rates � Positive classroom

behavior � Student attitudes

toward learning and school

� Disciplinary referrals

�

�

�

�

�

Program administrators interview/survey School administrators focus group and survey Classroom teacher survey Parent focus group and survey PPICS Data

Objective 2: Participants in the program will demonstrate increased academic achievement.

2. What impact does the program have on student achievement, including homework completion, classroom grades, promotions, and performance on the state assessments?

� Academic performance on state assessments by grade and by subject matter

� Homework completion rates

� Classroom grades � Promotion/retention

rates

�

�

�

�

�

� �

�

Program administrators interview/survey School administrators focus group and survey Classroom teacher survey Site teacher/tutor survey Parent focus group and survey PPICS Data Interactive Illinois Report Card State assessment data at individual level

Objective 4: The 21st CCLC programs will work toward services that benefit the entire community by including families of participants and collaborating with other agencies and non-profit organizations

3. In what ways does the program serve the parents of the program participants? Is there increased involvement by participants' parents in regular school activities?

� Evidence and quality of enrichment and support services for families of participants

� Parent involvement in regular school activities

� Parent attitude towards learning and school

� Parent satisfaction with program and services

� Parent perception of impact of program on students and community

�

�

�

�

�

�

�

Program administrators interview/survey School administrators focus group and survey Classroom teacher survey Site teacher/tutor survey Direct observation on site visit Parent focus group and survey PPICS Data

Objective 4: The 21st CCLC programs will work toward services that benefit the entire community by

4. What is the impact of the collaborations with other agencies and non-profit

� Type and extend of collaborations

� Parent satisfaction with referrals to other

�

�

Program administrators interview/survey Parent focus group and survey


Research Focus Questions Performance Indicator Source of Data including families of participants and collaborating with other agencies and non-profit organizations

organizations? agencies and non-profit agencies

� Community partners perception of impact of program and satisfaction with program

�

�

�

Community Partner survey PPICS Data

Objective 5: These programs will serve children and community members with the greatest needs for expanded learning opportunities.

5. Did the RFP award process result in programs being awarded to service the children and community members with the greatest need?

� Sites selected met the selection criteria based on free- and reduced-lunch eligibility, participants’ test scores, grades, and promotion rates.

� �

�

� �

ISBE staff interview Grantees’ applications for funding ISBE Fiscal Program Reports PPICS Data Interactive Illinois Report Card

Objective 6: 21st CCLC program personnel will participate in professional development and training that will enable them to implement an effective program. Professional development activities must be aligned with the No Child Left Behind Act definitions and National Staff Development Council’s professional development standards.

6. Did the professional development activities provided through the State of Illinois to 21st

CCLC program personnel adhere to No Child Left Behind Act definitions and the National Staff Development Council’s professional development standards?

� Participation of programs in the professional development activities

� Review of satisfaction data collected on the professional development activities

� Adherence to NCLB and NSDC definition and standards

� �

�

�

ISBE staff interview Program administrators interview/survey Program documentation and professional development records from ISBE PPICS Data

Objective 7: 21st CCLC program projects will use the funding most efficiently by coordinating and collaborating with other and state funding sources, agencies and other community projects to supplement the program and not supplant the funds, and to eventually become self-sustaining.

7. What are the current efforts toward providing for sustainability of the current programs, especially of the programs in their final year of funding?

� Implementation of sustainability plan

� Degree to which sustainability efforts are on-target to reach appropriate level of sustainability based on year in project

�

�

�

�

ISBE staff interview to confirm sustainability requirements per year of grant Program administrators interview/survey Community Partner survey Grantees’ applications for funding

Quality of Evaluative Data 8. Did the 21st CCLC program personnel find the data collection methods and evaluation resources, in particular PPICS, useful and relevant in documenting their programs and outcomes?

9. Would additional data and/or data collection methods have helped document the outcomes of the programs and

� Ease of use of PPICS and data collection

� Relevancy of PPICS data and data collected

� Usefulness of PPICS data in decision making

� Satisfaction with data collection methods

� Satisfaction with types of data collected

� Gap analysis of reporting and undocumented

� �

�

ISBE staff interview Program administrators interview/survey PPICS Data


Research Focus Questions Performance Indicator Source of Data provided supplemental information for decision-making?

outcomes

Assessment of Overall Implementation

10. To what extent are the grant recipients implementing the activities and evaluation plan proposed in their RFPs, as revised in their annual continuation requests?

� Activities match program objectives and goals

� Evidence of implementation and on-target progress of evaluation plan

� Evidence of implementation and on-target progress of sustainability plan

� �

�

�

�

�

ISBE staff interview Program administrators interview/survey Direct observation site visit Grantees’ applications for funding (RFPs and continuation requests) ISBE Fiscal Program Reports PPICS Data

Assessment of Overall Implementation

11. What factors hinder and which factors facilitate reaching the objectives?

� Identification of facilitating factors

� Identification of barriers � Identification of

strategies to overcome barriers

� �

�

�

�

�

�

�

�

ISBE staff interview Program administrators interview/survey School administrators focus group and survey Classroom teacher survey Site teacher/tutor survey Parent focus group and survey Community Partner survey Relevant research literature PPICS Data

SUMMATIVE EVALUATION Summative Evaluation 12. Are there specific

features or characteristics associated with exemplary outcomes?

� The identified characteristics are associated with a statistically significant difference in program outcomes (p < .05).

� Program characteristics will be evaluated for co-linearity before conducting the multivariate analyses.

� Outcomes will include student achievement and increases in positive behaviors, stakeholders’ levels of satisfaction, and perceptions of the positive impact of the program on the community.

� Characteristics will include budgetary

�

�

�

�

� �

�

�

�

�

Program administrator surveys School administrators focus group and survey Classroom teacher survey Site teacher/tutor survey Student focus group Parent focus group and survey Community Partner survey Grantees’ applications for funding ISBE Fiscal Program Reports Relevant literature on 21st CCLC programs and research on the program components


Research Focus Questions Performance Indicator Source of Data characteristics, school characteristics, student characteristics, type of community, school-based or community-based, and the type and duration of specific activities and interventions.

� � �

PPICS Data IIRC State assessment data at the student level

Summative Evaluation 13. How do the outcomes of programs completing the five-year cycle differ from the outcomes of programs in mid-grant?

� Statistically significant differences in program outcomes (p < .05) of programs in their last year as compared to programs in mid-grant cycle.

�

�

�

�

� �

�

�

�

�

� � �

Program administrator surveys School administrators focus group and survey Classroom teacher survey Site teacher/tutor survey Student focus group Parent focus group and survey Community Partner survey Grantees’ applications for funding ISBE Fiscal Program Reports Relevant literature on 21st CCLC programs and research on the program components PPICS Data IIRC State assessment data at the student level

Summative Evaluation 14. In retrospect, which were the most critical factors that hindered or facilitated obtaining the seven objectives? Are there barriers that must be addressed if the program is to have a positive impact on communities in the future? How can these barriers be addressed?

� Same as research focus #2.

� Same as research focus #2.


21st century community learnign centers illinois statewide ... · understand the day-to-day...

Documents