what does the research say? october 2014. agenda program description research overview key findings...
TRANSCRIPT
What Does the Research Say?
October 2014
Agenda
• Program Description • Research Overview• Key Findings • Conclusions and Continual Improvement• Lessons Learned: Mentoring for Academic Gains• Action Planning
Kelvin
Program Model
• Afterschool, Summer, High School Placement
• 650 hours/year, + 1000 hours in school• Weekly academic mentoring• Accelerated lessons, college trips,
career exposure• Positive culture and youth
development approach
RCT Study Design$4 million over 7 years
Four-year results published in 2013
By MDRC
Two-year results and Summer
Snapshot published in 2011
by P/PV
Researchers test & survey:
Baseline and three follow-ups
50% accepted through lottery
50% assigned to control group
Over 3 years (2006, 2007,
2008),951 students &
families apply and interview for 3
cohorts
Only 2% of nonprofits have this “gold-standard” randomized research
Randomized Control Trial (RCT) Study: funded by Wallace, WT Grant, Smith Richardson, Spencer, Atlantic Philanthropies, Bank of America
Randomization – in Brief
1. Hear about HA at school, neighborhood, etc.
2. Interview99% - enter lottery.
3. Lottery50% enterprogram:“treatment”
“Treatment”Access to full program – may never attend.
“Control”Never able to access program.But given list of other after-school & summer options
Annual testing ($120 payment), surveys,
Follow-up from researchers
Researchers
• Dr. Jean Grossman– Princeton University, P/PV, MDRC
• Dr. Carla Herrera– P/PV
• Dr. Leigh Linden– Columbia University, University of
Texas- Austin
Outcome Measures Collected• Hypothesis:
- With high dosage and structure, Higher Achievement would have a measurable impact on academic outcomes and high school placement, by first impacting attitudes and behaviors.
• 2 key outcomes:– Standardized Test Scores– Application, Acceptance to, and Matriculation at Competitive High Schools
Other outcomes of interest:– Behavior– Academic Attitudes– Perceptions of Peer and Adult Support– Participation in HA and Other OST Programs– Engagement in Academic Activities and High School-Related Activities
KEY FINDINGS
“Higher Achievement’s intensive year-round program had a
significant impact on youth’s standardized reading and math test
scores.”
“The longstanding Higher Achievement model is making a
difference in the academic lives of motivated, at-risk students who
could easily fall through the cracks.”
Two-year follow-up study
No summer learning loss.
Also - no impact on test scores, compared to control group,
over the course of one summer: 2010
High levels of attendance and retention, during tricky middle
school years: 97% of summer participants also in
after-school
Summer 2010 Snapshot
Four-year follow up study
Math gains persist at significantly higher levels than
the control group. Reading gains level with control group.
Significant impact on certain types of family engagement in
education
Significant impact on placement in competitive
private schools and avoiding low-performing high schools
Academic Impacts
First-Year FU
Second-Year FU
Fourth-Year FU
Math problem-solving 0.03 0.10* 0.11*
Reading comprehension 0.02 0.08† 0.04
* = Statistical significance.
Academic and Enrichment Activities
ActivityFirst-Year FU Second-Year FU Fourth-Year FU
Treatmt (%)
T-C Diff (%)
Treatmt (%)
T-C Diff (%)
Treatmt (%)
T-C Diff (%)
Community service 53 0 60 4 74 4
Presented ideas to a group out of school
63 6† 64 9* 69 11*
Visited a college 72 28** 73 28** 78 25**
Read books out of school 74 3 79 4 85 8†
Writing out of school 73 7* 74 3 82 14**
Visited a business 56 4 61 8* 70 14**
Events with OST 77 10** 80 6† 84 9*
Academic contests at OST 68 13** 68 11** 71 16**
Conclusions
• Year-round, multi-year:high dosage yields results
• No effect after 1 year, only after 2 years:– Academic results take time
• Reading gains level with control group in 4-year follow-up:– Control and treatment made gains.
Reading can be more self-directed. Math requires more instruction.
Continual Improvement• Common Core Standards Alignment:– Curricula and Technology
• Explicitly teach writing skills• Staff training to improve inference skills
for reading comprehension• “How Children Succeed” &
Social/Emotional Skills– Piloted three assessments:
PEAR HSA, SAYO, Gallup Student Poll + Grit Scale
Annual Operating Plan and Individual Work Plans, tied to Strategic Plan.
Monthly Dashboard and Site Observation Discussions:
National & Local StaffSchool Partners
Quarterly Dashboard, linked to Financial Model
Mid-Year Retreats: Org & City course
correctionsUpdate individual work
plans
Annual Retreat: Outcomes and RCT Results
Continuous ImprovementCycle
HOW MENTORING LEADS TO ACADEMIC GAINS
Maximizing Mentors
• One full-time staff member per site devoted to mentor support (80 mentors)
• Orientation (4 hours)• Ongoing mini-trainings
– lesson pacing, differentiated instruction, behavior management• Scripted lessons• Nightly session feedback• Monthly in-person observations • Quarterly report card data sharing with mentors
"Whatever affects one directly, affects all indirectly. I can never be what I ought to be until you are what you ought to be.”- Dr. Martin Luther King