university of california, irvine caomp email: [email protected] university of california, irvine...
TRANSCRIPT
Background and Rationale for Online Toolbox
Research indicates high quality afterschool programs can have beneficial effects on students both socially and academically
The California Outcome Measures Project (CAOMP) developed an Online Toolbox in response to the State’s legislative mandate for alternative accountability measures to standardized test scores for publically funded afterschool programs*
Toolbox includes reliable and valid measures of Positive Behavior Change and Skill Development to be used by State funded programs as part of their accountability measures
Field Test of measures administered using online toolbox at over 200 afterschool and summer learning program sites concluded in Fall 2011
Online Toolbox will be made available for general use in Fall 2012
**In partnership with UC Davis and with support David and Lucile Packard Foundation & California Department of Education
OVERVIEWAfterschool Outcome Measures Online Toolbox
Web-based surveys of student performance collected from three respondents: 1. Students (grades 3-8)2. Program staff3. Classroom teachers
PRE and POST Surveys collected at TWO time points, for example: Early Fall & Late Spring Beginning of Summer & End of Summer
Technical Assistance provided through: Online Instructions and Resources Email and Telephone Communication
Programs receive reports of their survey results at program and site Programs receive reports of their survey results at program and site level and aggregate results of California statewide data for comparisonlevel and aggregate results of California statewide data for comparison
DESCRIPTION OF MEASURESDESCRIPTION OF MEASURES
PROGRAM STAFF & CLASSROOM TEACHER REPORTS—of Elementary & Middle School Student Performance (5-6 subscales)
Skill Development Work Habits Task Persistence Academic Performance (classroom teachers only)
Positive Behavior Change Aggressive Behavior with Peers Prosocial Behavior with Peers Social Skills
Programs have option to administer one, two or all three surveys:Programs have option to administer one, two or all three surveys:(1) (1) Student (grades 3-8; Student (grades 3-8; (2) (2) Program Staff; and/or Program Staff; and/or (3) (3) Classroom TeachersClassroom Teachers
SIGNIFICANT FINDINGS
PRE-POST OUTCOMES in MATCHED SAMPLE
Staff Reports of Elementary Students Increase in Prosocial Behavior **
Staff Reports of Middle School Students Increase in Prosocial Behavior **
Teacher Reports of Elementary Students Increase in Academic Performance ** Decrease in Aggressive Behavior**
Student Reports of Quality Program Experience Positive student reports of program experience linked to positive results in ALL student outcomes
** p<.01 * p<.05** p<.01 * p<.05
From 2010-11 Field Test
Staff Reports on Student Prosocial Behavior Increase in pre-post scores—matched sample
Elementary School Students Middle School Students
SIGNIFICANT FINDINGS
Teacher Reports on Elementary Students Academic Performance
Increase in pre-post scores—matched sample
Aggressive Behavior Decrease in pre-post scores—matched sample
SIGNIFICANT FINDINGS
Program Quality Distribution
Staff & Activities Peer Affiliation
Student reports of higher quality experiences with staff & activities are linked to greater Math Efficacy
Student reports of higher quality experiences with peers are linked to less Aggressive Behavior
Quality to Outcomes
How does it work?
Updated Website to Launch
JANUARY 2012
Updated Website to Launch
JANUARY 2012
Online Toolkit Available to public
FALL 2012
EASY TO ADMINISTERStep-by-Step Instructions Available to
download from CAOMP Website
afterschooloutcomes.orgafterschooloutcomes.org
GO TO PROJECT WEBSITE: afterschooloutcomes.orgafterschooloutcomes.org
1. Click on “Field Test Surveys” tab at top.
GO TO PROJECT WEBSITE: afterschooloutcomes.orgafterschooloutcomes.org
1. Click on “Field Test Surveys” tab at top.
2. SELECT appropriate survey:
Once survey link is selected you should see this first page of the survey.
Example: Example:
Elementary Student Elementary Student SurveySurvey
Example: Example: Elementary Student Elementary Student
SurveySurvey
3. Student a. Selects language [English or Spanish] b. Enters assigned survey code:
3. Student a. Selects language [English or Spanish] b. Enters assigned survey code:
Click downFor Spanish
Click downFor Spanish
4. When student finishes survey—in about 10 minutes—this final page should appear.
4. When student finishes survey—in about 10 minutes—this final page should appear.
PProgram Staff & Teacher Surveysrogram Staff & Teacher SurveysReports on Students in Afterschool ProgramReports on Students in Afterschool Program
Program Staff and Classroom teachers are assigned Program Staff and Classroom teachers are assigned individual access codes individual access codes
Use both individual access code and the ID code for each Use both individual access code and the ID code for each student for whom they complete a reportstudent for whom they complete a report
Each Each student report student report takes takes 3-5 minutes 3-5 minutes to complete.to complete. Student reports may be completed in multiple sessions. Student reports may be completed in multiple sessions.
For exampleFor example: : A staff completing reports for 20 students can do four A staff completing reports for 20 students can do four reports a day (a 20-minute session) on five reports a day (a 20-minute session) on five
different days.different days.
Expands accountability tools beyond standardized test scores and other academic measures
Provides a battery of valid, reliable and easy to administer youth outcome measures • Currently include: Skill Development, Positive Behavior Change & Program Experience• Process of testing measures of STEM (Science, Technology, Engineering and Math)
learning in afterschool • Aim is to make available a menu of measures that match a range of student outcome
goals for diverse out-of-school time programs
Value in tracking individual student performance over time with matched pre-post surveys—providing more powerful & interpretable data• Matched = Respondent data is matched for both pre and post survey administrations
(i.e. same students, staff and/or teachers in fall and spring)
Information can be useful to to guide program improvement based on student reports of program experience AND on student performance