![Page 1: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/1.jpg)
21st CCLC ConferenceEvaluation Presentation
Strategic Research Group
September 19, 2014
![Page 2: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/2.jpg)
Presentation Overview
1. Meet the State Evaluation Team
2. Evaluation Results
3. From Whys to Wise
4. Local Evaluation Manual
5. Plans for the 2014-2015 Evaluation
![Page 3: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/3.jpg)
1. MEET THE STATE EVALUATION TEAM
![Page 4: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/4.jpg)
State Evaluation Team
Kathleen Carr
Tina Kassebaum
Lisa Riegel
Jillian Hmurovic
995 Goodale Blvd
Columbus, OH 43212
614.220.8860
www.strategicresearchgroup.com
![Page 5: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/5.jpg)
2. EVALUATION RESULTS
![Page 6: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/6.jpg)
Key Areas Needing Improvement
1. Academic Growth
2. Student Retention
3. Student Engagement/Behavior
![Page 7: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/7.jpg)
Key Area 1: Academic Growth
Overall, limited evidence of academic growth
Minimal change in math and reading grades
30%
26%
44%
30%
28%
42%
Increase
Decrease
No Change
Math
2013
2012
31%
24%
44%
32%
26%
43%
Reading
2013
2012
![Page 8: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/8.jpg)
Key Area 1: Academic Growth
Daytime teachers generally report improvement in academic performance
32%
3%
65%
Improvement Decline No Change
Academic Performance
![Page 9: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/9.jpg)
Types of Programming
Math and reading programming:
Nearly all sites have HW help
Many also have customized help
Individual assistance & tutoring
Computer software (Math: 77%; Reading: 73%)
YD programming
More “incidental” than “intentional”
![Page 10: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/10.jpg)
Amount of Programming
Almost two-thirds of sites require daily academic participation
Math: 61%
Reading: 64%
Avg. minutes spent
Nearly half of sites avg. >30 min.
Largest % of sites avg. 21-30 min.
2% 3% 3%
17% 20% 19%
33% 28% 29%
22% 22%13%
21% 24%
20%
4% 3%16%
Math Reading YD
>60 min.
46-60 min.
31-45 min.
21-30 min.
1-20 min.
None
![Page 11: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/11.jpg)
Number of Site Staff
Nearly half of sites: < 10 staff
Avg. # (per site)Teachers*: 3.23
Volunteers: 3.45
All Staff: 16.2
Student-Teacher Ratio: 8 to 1
Possible trend“Newer” sites: lower ratio
“Older” sites: higher ratio
10%
21%
16%28%
12%
13% 1 to 3 Staff
4 to 6 Staff
7 to 9 Staff
10 to 15 Staff
16 to 20 Staff
> 20 Staff
![Page 12: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/12.jpg)
Type of Site Staff
Compensated vs. Volunteer Staff
Most sites pay all staff (85%) and have no unpaid volunteers (81%)
Staff Certification
60% of sites: at least half of staff are licensed teachers
66% of sites: have paraprofessional staff
Communication with daytime teachers
Tends to be informal and ad hoc (less programmatic)
![Page 13: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/13.jpg)
Features of High-Improvement Sites
(↓) Lowest avg. student-to-staff ratios
both paid and volunteer staff
(↑) Highest avg. # partners providing $$
(↑) Highest avg. # partners providing evaluation services
(↑) Highest # of hrs. of operation (per week)
Suspect academic programming associated with high improvement, but quality of data is limiting
![Page 14: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/14.jpg)
Key Area 2: Student Retention
Only about half of students served are regular attendees
Possible greater difficulty HS students
Most HS students are not regular attendees
Little change from 2012 to 2013
65%
44%
26%
Elementary
Middle
High
% Who are Regular
Attendees
![Page 15: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/15.jpg)
Program Expansion
Minimal program growth
Little change in avg. # students per site
Slight ↑ for “newer” sites
Slight ↓ for “older” sites
No change in avg. # regular attendees per site
“Older” sites: greater % of regular attendees
![Page 16: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/16.jpg)
Factors Influencing Attendance
Site administrators’ attribution of change
↑ due to increased awareness of program (90%)
↓ due to decreased funding (38%)
Student recruitment efforts (Push vs. Pull)
Push: academic weakness
Pull: student interest
![Page 17: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/17.jpg)
Factors Influencing Attendance
Other commonly perceived barriers:
Program Perception (students & staff)
School Administrator Support
Competition with Other Activities
Transportation
![Page 18: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/18.jpg)
Key Area 3: Student Engagement
Fair amount of off-task behavior
Quality of student-teacher interaction associated with engagement
Most observed sites reported using specific YD framework
Pervasive integration into programming not observed
![Page 19: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/19.jpg)
Daytime Classroom Behavior
Daytime teachers generally report improvement in classroom behaviors
14%
20% 20% 21%
25% 25%
29% 30%33%
2% 2% 3%
14%
3% 3% 2% 3% 3%
AttendingRegularly
Getting Alongw/ Others
Behaving Volunteering Motivation toLearn
BeingAttentive
Participating Turning in HW CompletingHW
Satisfactorily
Improvement Decline
![Page 20: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/20.jpg)
3. USING WHYS TO BECOME WISE
![Page 21: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/21.jpg)
Root causes of challenges we face?
From WHYs to WISE
Why is my attendance inconsistent?
Because students want to go home rather than attend
![Page 22: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/22.jpg)
Root causes of challenges we face?
From WHYs to WISE
Why do students want to go home instead of coming here?
Students think the program is boring
![Page 23: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/23.jpg)
Root causes of challenges we face?
From WHYs to WISE
Why do students think the program is boring?
Because we do not have enough activities outside of homework help and our activities are repeated (e.g., we have open gym every day and kids play basketball)
![Page 24: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/24.jpg)
Root causes of challenges we face?
From WHYs to WISE
Why are your activities repeated so often?
We do not have good community partnerships to help us develop activities
Here is the problem you need to address for program improvement
After you evaluate the root causes that are impacting your data, you can implement targeted interventions for
program improvement.
![Page 25: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/25.jpg)
Now You Try…Define the Root Cause
Individually, select one of the 3 key areas for growth: academic growth, student retention, and student engagement
Within that area, identify a challenge you are facing
Team (table) members will ask you five “WHY” questions to dive deeper into the root cause 5 minutes per person
Afterwards, as a group, discuss ideas about how to overcome these root causes
![Page 26: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/26.jpg)
3. LOCAL EVALUATION MANUAL
A tool to help you improve your programming.
![Page 27: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/27.jpg)
Your evaluation should answer…
1. What are our goals?
2. What did we do to try
to meet them?
3. Did it work?
4. How do we know?
5. What should we do now?
This formative process can help you ensure you are getting the most out of your 21st Century grant.
It can also help you communicate your story to local partners.
A LOCAL EVALUATION
PROCESS
![Page 28: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/28.jpg)
Four Phases of Evaluation
Preparation
PlanningImplementation
Using Results
A stakeholder team should be engaged in your evaluation and improvement process.
Potential team members: School principal
Staff members
Student representatives
Business and community partners
A parent representative
Your local evaluator
Program director
Site administrator
![Page 29: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/29.jpg)
Local Evaluation Manual Overview
The manual is a valuable tool to help you facilitate the evaluation process
Provides an
overview of the
local evaluation
and sets the
foundation for
using evaluation
results
Se
ctio
n 1 Describes the
local evaluation
process,
including
assembling your
stakeholder
team and hiring
a local evaluator
Se
ctio
n 2 Helps you
understand and
develop the
reporting
requirements of
your evaluation
Se
ctio
n3
![Page 30: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/30.jpg)
Four Phases of Evaluation
Preparation
PlanningImplementation
Using Results
Assemble stakeholders &Hire a local evaluator
How to find a local evaluator
What to expect from your evaluator
How to interview for a local evaluator
![Page 31: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/31.jpg)
Four Phases of Evaluation
Preparation
PlanningImplementation
Using Results
Developing a Logic Model & Evaluation Plan
What are our goals?
What did we do to try to meet them?
Did it work?
How do we know?
What should we do now?
![Page 32: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/32.jpg)
Four Phases of Evaluation
Preparation
PlanningImplementation
Using Results
Data Collection & Analysis
What do you collect?
When do you collect it?
Who gathers it?
How do you analyze it?
![Page 33: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/33.jpg)
Four Phases of Evaluation
Preparation
PlanningImplementation
Using Results
Stakeholder Meetings
Communicating Results
To Whom?
When?
How?
![Page 34: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/34.jpg)
Who is Required to Use the Local Evaluation Manual?
Required:
New Grantees
Second Year Programs
Strongly Recommended:
All other programs!
![Page 35: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/35.jpg)
Accessing Local Evaluation Manual
Accessible online: www.websrg.com
Email Kathleen Carr:
![Page 36: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/36.jpg)
4. PLANS FOR 2014-2015 EVALUATION
![Page 37: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/37.jpg)
Emphasis on Local Evaluation
Will take into account what we have learned so far to improve Local Evaluation and, thus, programs
Goals: 1. Develop best practices for Local Evaluation
2. Learn how of use Local Evaluation for continuous program improvement
3. Increase positive outcomes for students served by 21st CCLC programs
![Page 38: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/38.jpg)
Evaluation Questions
1. How well did programs follow the Local Evaluation Manual requirements?
2. How rigorous and effective were the local evaluations conducted by the 37 2013-2014 grantees?
3. How were local evaluations used for program improvement?
4. Did the local evaluations, based on the guidance provided by the Local Evaluation Manual, result in more useful feedback than those local evaluation results from programs that did not use the manual?
5. How can the Local Evaluation Manual be improved?
![Page 39: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/39.jpg)
2014-2015 Evaluation Components
1. Document Review
2. Local Evaluation Survey
3. Intensive Case Studies
![Page 40: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/40.jpg)
Document Review
Review of documents submitted by 2013 Grantees
Purpose:
Look for best practices
Uncover challenges and barriers
Use findings to make improvements to the Local Evaluation Manual
![Page 41: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/41.jpg)
Local Evaluation Survey
Web survey of local evaluator for each program
Each program should supply SRG with contact information of person responsible for Local Evaluation Complete Evaluator Information Sheet
Questions will include:What evaluation activities were conducted?
How were data analyzed?
How were the results used for continuous improvement?
Conducted in late fall (2014)
![Page 42: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/42.jpg)
Intensive Case Studies
Will select 6 sites for more intensive case study
Case studies will include:
Interviews with program administrators
Interviews with stakeholder committee members
Interviews with local evaluators
Review of local evaluation data
Conducted in late winter to early spring (2015)
![Page 43: 21st CCLC Conference - WordPress.com · Presentation Overview 1. Meet the State Evaluation Team 2. Evaluation Results 3. From Whys to Wise 4. Local Evaluation Manual 5. Plans for](https://reader031.vdocument.in/reader031/viewer/2022030104/5c43582d93f3c34c32307920/html5/thumbnails/43.jpg)
Thank You
For more information, please contact:
Kathleen Carr
614.220.8860