21st cclc conference - wordpress.com · presentation overview 1. meet the state evaluation team 2....

43
21st CCLC Conference Evaluation Presentation Strategic Research Group September 19, 2014

Upload: doandang

Post on 20-Jan-2019

215 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

21st CCLC ConferenceEvaluation Presentation

Strategic Research Group

September 19, 2014

Page 2: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Presentation Overview

1. Meet the State Evaluation Team

2. Evaluation Results

3. From Whys to Wise

4. Local Evaluation Manual

5. Plans for the 2014-2015 Evaluation

Page 3: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

1. MEET THE STATE EVALUATION TEAM

Page 4: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

State Evaluation Team

Kathleen Carr

Tina Kassebaum

Lisa Riegel

Jillian Hmurovic

995 Goodale Blvd

Columbus, OH 43212

614.220.8860

www.strategicresearchgroup.com

Page 5: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

2. EVALUATION RESULTS

Page 6: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Key Areas Needing Improvement

1. Academic Growth

2. Student Retention

3. Student Engagement/Behavior

Page 7: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Key Area 1: Academic Growth

Overall, limited evidence of academic growth

Minimal change in math and reading grades

30%

26%

44%

30%

28%

42%

Increase

Decrease

No Change

Math

2013

2012

31%

24%

44%

32%

26%

43%

Reading

2013

2012

Page 8: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Key Area 1: Academic Growth

Daytime teachers generally report improvement in academic performance

32%

3%

65%

Improvement Decline No Change

Academic Performance

Page 9: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Types of Programming

Math and reading programming:

Nearly all sites have HW help

Many also have customized help

Individual assistance & tutoring

Computer software (Math: 77%; Reading: 73%)

YD programming

More “incidental” than “intentional”

Page 10: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Amount of Programming

Almost two-thirds of sites require daily academic participation

Math: 61%

Reading: 64%

Avg. minutes spent

Nearly half of sites avg. >30 min.

Largest % of sites avg. 21-30 min.

2% 3% 3%

17% 20% 19%

33% 28% 29%

22% 22%13%

21% 24%

20%

4% 3%16%

Math Reading YD

>60 min.

46-60 min.

31-45 min.

21-30 min.

1-20 min.

None

Page 11: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Number of Site Staff

Nearly half of sites: < 10 staff

Avg. # (per site)Teachers*: 3.23

Volunteers: 3.45

All Staff: 16.2

Student-Teacher Ratio: 8 to 1

Possible trend“Newer” sites: lower ratio

“Older” sites: higher ratio

10%

21%

16%28%

12%

13% 1 to 3 Staff

4 to 6 Staff

7 to 9 Staff

10 to 15 Staff

16 to 20 Staff

> 20 Staff

Page 12: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Type of Site Staff

Compensated vs. Volunteer Staff

Most sites pay all staff (85%) and have no unpaid volunteers (81%)

Staff Certification

60% of sites: at least half of staff are licensed teachers

66% of sites: have paraprofessional staff

Communication with daytime teachers

Tends to be informal and ad hoc (less programmatic)

Page 13: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Features of High-Improvement Sites

(↓) Lowest avg. student-to-staff ratios

both paid and volunteer staff

(↑) Highest avg. # partners providing $$

(↑) Highest avg. # partners providing evaluation services

(↑) Highest # of hrs. of operation (per week)

Suspect academic programming associated with high improvement, but quality of data is limiting

Page 14: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Key Area 2: Student Retention

Only about half of students served are regular attendees

Possible greater difficulty HS students

Most HS students are not regular attendees

Little change from 2012 to 2013

65%

44%

26%

Elementary

Middle

High

% Who are Regular

Attendees

Page 15: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Program Expansion

Minimal program growth

Little change in avg. # students per site

Slight ↑ for “newer” sites

Slight ↓ for “older” sites

No change in avg. # regular attendees per site

“Older” sites: greater % of regular attendees

Page 16: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Factors Influencing Attendance

Site administrators’ attribution of change

↑ due to increased awareness of program (90%)

↓ due to decreased funding (38%)

Student recruitment efforts (Push vs. Pull)

Push: academic weakness

Pull: student interest

Page 17: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Factors Influencing Attendance

Other commonly perceived barriers:

Program Perception (students & staff)

School Administrator Support

Competition with Other Activities

Transportation

Page 18: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Key Area 3: Student Engagement

Fair amount of off-task behavior

Quality of student-teacher interaction associated with engagement

Most observed sites reported using specific YD framework

Pervasive integration into programming not observed

Page 19: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Daytime Classroom Behavior

Daytime teachers generally report improvement in classroom behaviors

14%

20% 20% 21%

25% 25%

29% 30%33%

2% 2% 3%

14%

3% 3% 2% 3% 3%

AttendingRegularly

Getting Alongw/ Others

Behaving Volunteering Motivation toLearn

BeingAttentive

Participating Turning in HW CompletingHW

Satisfactorily

Improvement Decline

Page 20: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

3. USING WHYS TO BECOME WISE

Page 21: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Root causes of challenges we face?

From WHYs to WISE

Why is my attendance inconsistent?

Because students want to go home rather than attend

Page 22: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Root causes of challenges we face?

From WHYs to WISE

Why do students want to go home instead of coming here?

Students think the program is boring

Page 23: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Root causes of challenges we face?

From WHYs to WISE

Why do students think the program is boring?

Because we do not have enough activities outside of homework help and our activities are repeated (e.g., we have open gym every day and kids play basketball)

Page 24: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Root causes of challenges we face?

From WHYs to WISE

Why are your activities repeated so often?

We do not have good community partnerships to help us develop activities

Here is the problem you need to address for program improvement

After you evaluate the root causes that are impacting your data, you can implement targeted interventions for

program improvement.

Page 25: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Now You Try…Define the Root Cause

Individually, select one of the 3 key areas for growth: academic growth, student retention, and student engagement

Within that area, identify a challenge you are facing

Team (table) members will ask you five “WHY” questions to dive deeper into the root cause 5 minutes per person

Afterwards, as a group, discuss ideas about how to overcome these root causes

Page 26: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

3. LOCAL EVALUATION MANUAL

A tool to help you improve your programming.

Page 27: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Your evaluation should answer…

1. What are our goals?

2. What did we do to try

to meet them?

3. Did it work?

4. How do we know?

5. What should we do now?

This formative process can help you ensure you are getting the most out of your 21st Century grant.

It can also help you communicate your story to local partners.

A LOCAL EVALUATION

PROCESS

Page 28: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Four Phases of Evaluation

Preparation

PlanningImplementation

Using Results

A stakeholder team should be engaged in your evaluation and improvement process.

Potential team members: School principal

Staff members

Student representatives

Business and community partners

A parent representative

Your local evaluator

Program director

Site administrator

Page 29: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Local Evaluation Manual Overview

The manual is a valuable tool to help you facilitate the evaluation process

Provides an

overview of the

local evaluation

and sets the

foundation for

using evaluation

results

Se

ctio

n 1 Describes the

local evaluation

process,

including

assembling your

stakeholder

team and hiring

a local evaluator

Se

ctio

n 2 Helps you

understand and

develop the

reporting

requirements of

your evaluation

Se

ctio

n3

Page 30: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Four Phases of Evaluation

Preparation

PlanningImplementation

Using Results

Assemble stakeholders &Hire a local evaluator

How to find a local evaluator

What to expect from your evaluator

How to interview for a local evaluator

Page 31: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Four Phases of Evaluation

Preparation

PlanningImplementation

Using Results

Developing a Logic Model & Evaluation Plan

What are our goals?

What did we do to try to meet them?

Did it work?

How do we know?

What should we do now?

Page 32: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Four Phases of Evaluation

Preparation

PlanningImplementation

Using Results

Data Collection & Analysis

What do you collect?

When do you collect it?

Who gathers it?

How do you analyze it?

Page 33: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Four Phases of Evaluation

Preparation

PlanningImplementation

Using Results

Stakeholder Meetings

Communicating Results

To Whom?

When?

How?

Page 34: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Who is Required to Use the Local Evaluation Manual?

Required:

New Grantees

Second Year Programs

Strongly Recommended:

All other programs!

Page 35: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Accessing Local Evaluation Manual

Accessible online: www.websrg.com

Email Kathleen Carr:

[email protected]

Page 36: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

4. PLANS FOR 2014-2015 EVALUATION

Page 37: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Emphasis on Local Evaluation

Will take into account what we have learned so far to improve Local Evaluation and, thus, programs

Goals: 1. Develop best practices for Local Evaluation

2. Learn how of use Local Evaluation for continuous program improvement

3. Increase positive outcomes for students served by 21st CCLC programs

Page 38: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Evaluation Questions

1. How well did programs follow the Local Evaluation Manual requirements?

2. How rigorous and effective were the local evaluations conducted by the 37 2013-2014 grantees?

3. How were local evaluations used for program improvement?

4. Did the local evaluations, based on the guidance provided by the Local Evaluation Manual, result in more useful feedback than those local evaluation results from programs that did not use the manual?

5. How can the Local Evaluation Manual be improved?

Page 39: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

2014-2015 Evaluation Components

1. Document Review

2. Local Evaluation Survey

3. Intensive Case Studies

Page 40: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Document Review

Review of documents submitted by 2013 Grantees

Purpose:

Look for best practices

Uncover challenges and barriers

Use findings to make improvements to the Local Evaluation Manual

Page 41: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Local Evaluation Survey

Web survey of local evaluator for each program

Each program should supply SRG with contact information of person responsible for Local Evaluation Complete Evaluator Information Sheet

Questions will include:What evaluation activities were conducted?

How were data analyzed?

How were the results used for continuous improvement?

Conducted in late fall (2014)

Page 42: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Intensive Case Studies

Will select 6 sites for more intensive case study

Case studies will include:

Interviews with program administrators

Interviews with stakeholder committee members

Interviews with local evaluators

Review of local evaluation data

Conducted in late winter to early spring (2015)

Page 43: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for

Thank You

For more information, please contact:

Kathleen Carr

[email protected]

614.220.8860