presentation to the before and afterschool advisory committee
DESCRIPTION
Pilot Findings and Field Test: An Update. Presentation to the Before and Afterschool Advisory Committee California Department of Education January 11, 2011. California Afterschool Outcome Measures Project. Last report to the Advisory Committee —February 24, 2010 - PowerPoint PPT PresentationTRANSCRIPT
Presentation to the Presentation to the
Before and Afterschool Advisory Committee Before and Afterschool Advisory Committee
California Department of EducationCalifornia Department of Education
January 11, 2011January 11, 2011
Pilot Findings and Field Test:Pilot Findings and Field Test:An Update An Update
California Afterschool Outcome Measures ProjectCalifornia Afterschool Outcome Measures Project
Last report to the Advisory Committee —Last report to the Advisory Committee —February 24, 2010 February 24, 2010
Pre-Pilot activities and Key Findings:Pre-Pilot activities and Key Findings: Completed pre-pilot test of initial versions of measures Completed pre-pilot test of initial versions of measures
and online administrationand online administration Measures performed well in online format; preliminary Measures performed well in online format; preliminary
outcomes comparable to previous studiesoutcomes comparable to previous studies Minor modifications made to measures and Minor modifications made to measures and
procedures based on pre-pilot findingsprocedures based on pre-pilot findings• Changes to wording in items, scales removed and addedChanges to wording in items, scales removed and added• Survey administration instructions revised—clear step-by-Survey administration instructions revised—clear step-by-
step step • Special instructions created to support ELL students and Special instructions created to support ELL students and
slow readersslow readers
Highlights of Today’s ReportHighlights of Today’s Report
Phase 2—Report on the Pilot Phase 2—Report on the Pilot TestTest Creation of project websiteCreation of project website Results of the spring 2010 pilot workResults of the spring 2010 pilot work
Phase 3—Update on the Field Phase 3—Update on the Field TestTest Fall survey administration experienceFall survey administration experience Dissemination plansDissemination plans
Phase 2: PILOT PHASEPhase 2: PILOT PHASE
Pilot DataPilot DataNumber of Respondents & Time to complete Number of Respondents & Time to complete
Survey Survey 1,656 Students completed surveys1,656 Students completed surveys
• 1,175 Elementary; 481 Middle1,175 Elementary; 481 Middle• On average less than 10 minutes to complete survey (observed)On average less than 10 minutes to complete survey (observed)• 51% elementary students report they took less than 10 minutes51% elementary students report they took less than 10 minutes• 61% middle school students report taking less than 10 minutes61% middle school students report taking less than 10 minutes
105 Staff completed 769 student reports105 Staff completed 769 student reports• 54% report taking less than 5 minutes per student report54% report taking less than 5 minutes per student report• 46% report taking 5 to 9 minutes46% report taking 5 to 9 minutes
70 Classroom Teachers completed 535 student reports70 Classroom Teachers completed 535 student reports• 81 % report completing each student report in less than 5 minutes81 % report completing each student report in less than 5 minutes• 18% report taking 5-10 minutes per report18% report taking 5-10 minutes per report• 1% took more than 10 minutes 1% took more than 10 minutes
FIELD TESTFIELD TEST (2010-2011) (2010-2011)FIELD TESTFIELD TEST (2010-2011) (2010-2011)Overview of key elements: Overview of key elements:
Participation open to all programs in the stateParticipation open to all programs in the state
Outreach effort to all regionsOutreach effort to all regions::• Presented on Field Test at Regional Lead Meeting (October 13, 2010)Presented on Field Test at Regional Lead Meeting (October 13, 2010)• Sent Field Test materials electronically to RLs for dissemination to all granteesSent Field Test materials electronically to RLs for dissemination to all grantees
Web-based surveys for students, program staff, and Web-based surveys for students, program staff, and classroom teachers completed in fall 2010 and spring classroom teachers completed in fall 2010 and spring 2011 2011
• Programs select which surveys to administerPrograms select which surveys to administer
Confidential summary report of survey resultsConfidential summary report of survey results• students’ scores of positive behavior change and skill development provided to students’ scores of positive behavior change and skill development provided to
participating programsparticipating programs
All materials and instructions available on Project All materials and instructions available on Project websitewebsite
Technical assistance provided to programsTechnical assistance provided to programs• Support provided by email and telephone during pre and post survey administrations Support provided by email and telephone during pre and post survey administrations
& in interpreting their scores at the end of Field Test & in interpreting their scores at the end of Field Test
Measures modified based on Pilot results and feedback Measures modified based on Pilot results and feedback from fieldfrom field
FIELD TESTFIELD TEST (2010-2011) (2010-2011)Outcome MeasuresOutcome Measures
StudentStudent—Elementary & Middle School——Elementary & Middle School—(5 subscales)(5 subscales)• Work HabitsWork Habits• MisconductMisconduct• Reading Efficacy Reading Efficacy • Math EfficacyMath Efficacy• Social EfficacySocial Efficacy
Staff (5 subscales)Staff (5 subscales)• Work HabitsWork Habits• Aggressive Behavior with PeersAggressive Behavior with Peers• Prosocial Behavior with PeersProsocial Behavior with Peers• Task PersistenceTask Persistence• Social SkillsSocial Skills
TeacherTeacher (6 subscales)(6 subscales)• Same as Staff plus Mock Report CardSame as Staff plus Mock Report Card
FIELD TEST FINDINGS FIELD TEST FINDINGS (Fall 2010 – Spring 2011)(Fall 2010 – Spring 2011)
Diverse programs across the state participate in fall 2010 Diverse programs across the state participate in fall 2010 field test field test • 156156 Sites (elementary & middle school) Sites (elementary & middle school)
• 2323 Programs (10 rural, 8 urban, 5 suburban) Programs (10 rural, 8 urban, 5 suburban)
• 8 8 Regions representedRegions represented
Pre-Surveys Completed Pre-Surveys Completed (November 1 through December 17) (November 1 through December 17) • 3934 3934 elementary studentselementary students• 1847 1847 middle school studentsmiddle school students• 360 360 program staff completed program staff completed 4611 4611 student reportsstudent reports• 29 29 classroom teachersclassroom teachers completed completed 185 185 student reportsstudent reports
Initial feedback from Field Test sites positiveInitial feedback from Field Test sites positive• Instructions clearInstructions clear• Online administration easyOnline administration easy• Happy with the quality of technical support Happy with the quality of technical support
Field Test psychometrics consistent with Pilot results and Field Test psychometrics consistent with Pilot results and previous studiesprevious studies
FIELD TEST FINDINGS FIELD TEST FINDINGS Time to Complete Survey Time to Complete Survey
Elementary and Middle School Students Reported Time to Complete Survey
Pilot Field Test
Elementary Students N=1,175 N=3934
< 5 minutes 23% 30%
5-9 minutes 28% 27%
10-15 minutes 16% 10%
More than 15 minutes 2% 3%
Don’t Know 31% 30%
Middle School Students
N=481 N=1847
< 5 minutes 24% 37%
5-9 minutes 38% 33%
10-15 minutes 8% 5%
More than 15 minutes 3% 2%
Don’t Know 27% 23%
FIELD TEST FINDINGS FIELD TEST FINDINGS Time to Complete Survey Time to Complete Survey
Pilot Field Test
Teachers N = 70 N = 29
< 5 minutes 81% 50%5-9 minutes 18% 46%
10 minutes or more 1% 4%
Staff N = 105 N = 360
< 5 minutes 54% 61%
5-9 minutes 46% 31%
10 minutes or more — 4%
Classroom Teacher and Staff Reported Time to Complete Each Student Report
NEXT STEPSNEXT STEPS Can programs sign up now? ign up now?
• Yes, but they will not get a report of pre/post change scores, only Yes, but they will not get a report of pre/post change scores, only the results of the spring survey administration.the results of the spring survey administration.
Provide support to new participants Provide support to new participants enlisting in Field Test for spring 2011enlisting in Field Test for spring 2011
Contact current Field Test participants to Contact current Field Test participants to prepare for spring survey administration prepare for spring survey administration (April - May 2011)(April - May 2011)
Dissemination of findings in summer 2011 Dissemination of findings in summer 2011 in collaboration with California Afterschool in collaboration with California Afterschool Network Regional Leads and CDENetwork Regional Leads and CDE