CaMSP Cohort 8 OrientationCaMSP Cohort 8 Orientation

Cohort 8

State and Local Evaluation

Overview, Reporting Requirements, and Attendance Database

February 23, 2011

California Department of Education

Mikala L. Rahn, Ph.D

Public Works, Inc.

Who Are We?Who Are We?

Public Works, Inc. a nonprofit in Pasadena dedicated to working with communities, government, schools and parents by providing services and resources to educate and inform children, youth and families. Our work is in three areas:

• Education Reform

• Workforce Development

• Intervention/Prevention

Our TeamOur Team

Project Director:

Mikala L. Rahn, PhD.

Team: Patty O’Driscoll, Velette Bozeman, Albert Chen, Scott Phelps, James Maloney

Multiple Evaluation Multiple Evaluation RequirementsRequirements

1) Statewide Evaluation

2) Local Evaluation

3) Federal Reporting

Levels of EvaluationLevels of Evaluation

Central versus local

• CDE/Public Works, Inc.

• CDE/Federal

• Partnership/ Local evaluators

Statewide Evaluation Statewide Evaluation Research QuestionsResearch Questions

1. How have the Partnerships ensured that all students have access to, are prepared for, and are encouraged to participate and succeed in challenging and advanced mathematics and science courses?

2. How have the Partnerships enhanced the quality of the mathematics and science teacher workforce?

3. What evidence-based outcomes from the Partnerships contribute to our understanding of how students effectively learn mathematics and science?

Key FeaturesKey Features

Features of the programs reviewed:

• Partnership driven

• Teacher quality

• Challenging courses and curricula

• Evidence-based design and outcomes

• Institutional change and sustainability

Balanced Evaluation Approach in Balanced Evaluation Approach in the Statewide Evaluationthe Statewide Evaluation

Process measures focus on how a program is implemented and the outcome measures focus on the results of the program or intervention.

Public Works, Inc. uses both quantitative and qualitative data collection methods.

Qualitative AnalysisQualitative Analysis

• Qualitative research including site visits to Cohort 8 partnerships and selected additional partnerships from previous cohorts in Spring 2012. • Site visits include interviews, focus groups and observations to gain general patterns across partnerships for a statewide picture, and best practices discovered throughout the state. •Telephone interviews in years you are not visited (Spring 2013).

Quantitative Data CollectedQuantitative Data Collected

• Teacher Database: Incorporates PAIF data on all teachers in participating districts and attendance in professional development for all participating teachers (ongoing)

• Partner Survey (Winter)• Participating Teacher Survey (Spring)• Student rosters of comparison and

treatment teachers for statewide study/CST Data (first in Summer-Fall 2012)

Teacher outcomes examined Teacher outcomes examined in state evaluationin state evaluation

TeachersNumber and characteristics of teachers who

participate in professional development; satisfaction with training

• Demographics/Years teaching• Hours of Training• Qualifications/Assignments• Treatment vs. Comparison Student

Performance on CSTs

Student outcomes examined Student outcomes examined in state evaluationin state evaluation

Students

• Improved student academic achievement on state mathematics and science assessment across the state and at the partnership level

Local EvaluationLocal Evaluation

Two Goals:

• Fulfill state evaluation requirements

• Fulfill commitments made in Local Evaluation Plan

Internal or External evaluator (not a training provider) qualified to conduct a research study

Local EvaluationLocal Evaluation

Fulfill commitments in CaMSP proposal:• Pre- and post- assessments of teacher content knowledge• Matched control/comparison group of teachers• Local assessment data/CSTs • Teacher observation, portfolio, lesson study, video• Surveys• Qualitative/case studies• Value added/comparison/treatment methodologies

Most important for local: Teacher knowledge and instructional strategies & measuring student knowledge with local assessments. Also, the completion of an evaluation report to attache to the March CDE YTD report and another for the fall Federal Report.

Measuring the UnmeasurableMeasuring the Unmeasurable

• Capturing your story….

• Narrative/journaling professional growth from the teacher’s perspective

• Demonstration lessons/video

• Teacher portfolios

• Performance tasks from students

How do we document our work in math and science?

Being Clear on TermsBeing Clear on Terms

What is your model for…• Coaching

• Lesson study

• Hands-on

• Classroom follow-up

• Mentoring

Bottom Line on Local Bottom Line on Local EvaluationEvaluation

• Documentation is crucial• High profile at the state-level: outcomes

matter!• Impact: How does professional

development translate into improved classroom practice?

• Demonstrate best practices in content delivery/pedagogy within state content

Federal Reporting Cohort 8Federal Reporting Cohort 8

Completion of APR is partnership and local evaluator responsibility

PW provides the Student GPRA data and technical assistance to partnerships

Cohort 8 First APR: PW Data Collection for APR Begins July 15, 2012

First Annual Performance Report (APR) due to CDE October 15, 2012

Block it now!

Narrative, Project Profile, & Local Evaluation Report are RequiredDue for Cohort 8 from CDE to ED by

November 30, 2012

PW State Evaluation Work Plan: PW State Evaluation Work Plan: Cohort 8Cohort 8

• Contacts/Partnership Profiles on CaMSP site (March 2011) • Conduct site visits, observations, and phone interviews

(Spring 2012)• Administer Partner Survey (Nov. 2011)• Web-based Database (ongoing)• Toolbox of instruments (ongoing)• Administer teacher survey (Feb-March 2012)• Determine list of treatment/control teachers based on

attendance through June 30, 2012 by July 15, 2012• Create matched comparison list of teachers by Aug 15, 2012• Collect rosters of students of treatment and comparison

teachers from partnership districts by September 15, 2012• Match to 2012 CSTs and provide student GPRA # to

partnerships by October 15, 2012

Next StepsNext Steps

• After Lunch: Demonstration of CaMSP Website and PW Attendance and Teacher Participation Database

• CDE provides updated PAIF (working from 08-09); CDE provides statewide CST data

• Determine who should get access to the database

• Input attendance data into Web system frequently (used by CDE for YTD’s!!)

• Closeout attendance data every June 30th by July 15th

• All professional development activities are expected to be logged as soon as possible following the activity.

Web-based Teacher Web-based Teacher Database RemindersDatabase Reminders

• Demographic teacher data is updated once per year from statewide PAIF (October).

• Input training data as you go.

• Database information will be reviewed by CDE as part of the YTD process.

To Do ListTo Do List

• Provide Updated contact information to CDE and PW (including database contacts)

• Review and approve PW partnership profile for CaMSP Website

• Work with Public Works, Inc. to begin data entry of attendance for periodic reports

Web-based CommunicationWeb-based Communication

www.publicworksinc.org

• Contacts (check)

• Profiles (check)

• Administrative (use)

• Database (use)

Contact InformationContact Information

• Mikala Rahn, mikala@publicworksinc.org

• Patty O’Driscoll, patty@publicworksinc.org

• Albert Chen, achen@publicworksinc.org

• Velette Bozeman, velette@publicworksinc.org

• Phone: (626) 564-9890