the community schools evaluation toolkit: moving the research agenda forward reuben jacobson,...

The Community Schools Evaluation Toolkit:

Moving the Research Agenda Forward

Reuben Jacobson, University of Maryland

Shital C. Shah, Coalition for Community Schools

Agenda

www.communityschools.org

2

Research summaryCommunity Schools Evaluation ToolkitDiscussionWork time: work on your evaluations and

network

Research Summary


3

Are community schools effective at improving outcomes for students, families, and the

community?• Collected 153 studies of community school

and community school-like initiatives– CS models: lead-agency, community agency/CBO,

university-assisted, and staff-initiated– Scale: boutique, local, state, national

• No studies in peer-reviewed journals

Characterizing the Literature


4

• Mix of internal/external evaluators• Great variation in study design– Process/implementation studies (e.g., number of

students served)– Outcome studies (e.g., improved achievement)

• Number of outcomes depends on the unique strategies of each community school – found at least 30 outcomes– Tied to theory of action

• Selected 22 quasi-experimental studies (comparison group, interrupted time series, controls)

Achievement


5

The most rigorous studies indicate that community schools are improving student achievement for some students.

• High-implementing CIS schools scored +6% than comparison non-CIS schools in percent proficient in grade 8 math (p<.01) and +5.1% in grade 8 reading (p<.05)

• Mixed results out of Chicago for school-level data– Student-level demonstrates that OST participation matters

• More students who participated in Children’s Aid Society (CAS) after-school programs demonstrated a steady increase from 2004 to 2007 in their math performance levels as measured by the state assessment compared to students who did not attend CAS activities (p<.05)

Attendance


6

The most rigorous studies indicate that community schools are improving attendance.

• CIS: High-implementing elementary schools had higher attendance levels than comparison group (+0.2%, p<.05). So did high-implementing high schools (+0.3%, p<.01)

• CAS: students who participated in CAS after-school programs for 3-4 years had better attendance than students with lower or no participation (p<.05)

• SF Beacons: participants who attended 30+ days of Beacon after-school had 3.9% less total days of unexcused absences than those who participated less than 30 days (p<.001)

Graduation Rate


7

While a focus of many community schools, there is little evidence of improving the graduation rate.

• CIS: high-implementing community schools had a significantly better graduation rate than comparison schools (+4.8 percent, p<.01)– Compared to other large-scale dropout programs, high-

implementing CIS schools had the highest effect size for graduation rate (ES=.31) and second-highest for dropout rate (ES=.36)

• CPS: schools that have been in CSI longer have better rates of ninth-grade students “on-track” to graduate than comparison group (p<.05)

Behavior


8

While a focus of many community schools, there is little evidence of improving student behavior.

• CPS schools had significantly less disciplinary incidents than their matched comparison group from 2002 to 2006

• CAS teachers reported that they saw greater improvement in students getting along with others for students with higher rates of participation in CAS than comparison students during the 2006-2007 school year (significance not reported)

Other Outcomes


9

Overall, few studies measure, evidence is weak, for the following outcomes: Parental engagement Relationships with adults Student engagement Impact on instruction

Future Directions for Evaluating Community Schools


10

• Utilize high-quality study designs that establish causality Why? Funding. Have to demonstrate effectiveness and grant-makers

are increasingly aware of study design (e.g., i3)• Evaluate the CS strategy, not just programs

Community schools are more than after-school programming Is the strategy or particular programming causing changes in

outcomes?• Use appropriate unit of analysis (school, student, or both)• Measure:

level of implementation: fidelity to design matters (e.g., CIS) longevity intensity of the intervention (after-school participation, receiving

health services, …• Evaluate outcomes for families and community; also student

health• Develop standard measures: e.g., how do you measure

student engagement?• Assess effectiveness of particular CS components to modify

and strengthen the design (e.g., do CS coordinators matter?)

Community Schools Evaluation Toolkit


11

Rationale for Results FrameworkJohn W. Gardner CenterCommunity School EvaluatorsCoalition for Community SchoolsJP Morgan Chase

What is in the Toolkit?


12

• 35 Page step-by-step manual • Coalition for Community Schools Logic

Model • Indicators for 5 short-term results• Indicators for 4 long-term results• Descriptions of school, city, county, and state

data that is available to sites• 45 surveys –for sites that are ready to collect

additional data• Examples from real sites

Goals of the Toolkit


13

• Introduce the Coalition for Community Schools (CCS) Results Based Logic Model

• Support sites to use existing data sources and collect additional data (if needed)

• Support sites to identify areas of success and areas in need of improvement

• Provide evaluation planning tools

4 Parts to the Toolkit:


14

1. Before you Start: Begin with the End in Mind

2. Get Ready: Prepare to Evaluate 3. Get Set: Designing the Evaluation4. Go!: The Evaluation Process


15

Discussion


16

Describe your experience evaluating CS.What challenges have you experienced?

Successes?What are your evaluation needs?

Work Time


17

Work on your evaluations and network

How to Get the Toolkit


18

• Coalition for Community Schools Web site: www.communityschools.org

• Questions? Comments?• [email protected]

Contact information


19

Reuben Jacobson [email protected]

Shital C. Shah [email protected]

the community schools evaluation toolkit: moving the research agenda forward reuben jacobson,...

Documents

studies of community

community schools effective

community agencycbo

outcome studies

rigorous studies

comparison noncis schools

network slide

number of students