Research Indicators for Sustaining and Institutionalizing Change

CaMSP Network MeetingApril 4 & 5, 2011

Sacramento, CA

Mikala L. Rahn, PhD

Public Works, Inc.

Session OverviewSession Overview

• CaMSP Evaluation Overview--What we can learn moving forward

• Emerging themes from MSP meeting, morning presentation, and panels on Institutionalization and Sustainability

• Time for discussion and next steps after the Network meeting

Statewide Evaluation Statewide Evaluation Research QuestionsResearch Questions

1. How have the Partnerships ensured that all students have access to, are prepared for, and are encouraged to participate and succeed in challenging and advanced mathematics and science courses?

2. How have the Partnerships enhanced the quality of the mathematics and science teacher workforce?

3. What evidence-based outcomes from the Partnerships contribute to our understanding of how students effectively learn mathematics and science?

Key Features of CaMSPKey Features of CaMSP

Features of the programs reviewed based onthe legislation:

• Partnership driven• Teacher quality• Challenging courses and curricula• Evidence-based design and outcomes• Institutional change and sustainability

L. Desimone L. Desimone Conceptual FrameworkConceptual Framework

Increased teacher

knowledge and skills; change in

attitudes and beliefs

CriticalAttributes ofProfessionalDevelopment:

Content focusActive

learningCoherenceDurationCollective

participation

Change in Instruction

Improved student learning

FIGURE 1: Desimone Conceptual Framework for studying the effects of professional development on teachers and students

What’s the Match?What’s the Match?

• Content Focus

• Active Learning

• Coherence

• Duration

• Collective Participation

IHE’s bring content lens

Strengthened partnerships-stronger models

Standards/textbooks/ assessments embedded

84 hours/3 years per teacher

Collaboration embedded in classroom follow-up

CaMSP in CaliforniaCaMSP in California

• Nine cohorts funded so far; science grades 3 through 8 or mathematics grades 3 through Algebra I.

• 59 partnerships included in the 07-08 outcome study; 88 partnerships in the 08-09 and 09-10 outcome study

• PD models currently incorporate significant hours of training for three funding cycles--less attrition as implementation requirements tightened in California

• Same cohort of teachers--we have data to measure based on consistent dosage and rules for implementation.

CaMSP Outcome StudyCaMSP Outcome Study

• 07-08 Outcome Study completed (Report available)

• 08-09 and 09-10 Outcome Study will be combined and reported Spring 2011

• Large studies: – 07-08: 284,538 treatment and comparison

students/1,581 treatment teachers– 08-09 and 09-10: 165,209 treatment and

comparison students/1,594 treatment teachers

CaMSP Evaluation and CaMSP Evaluation and Outcome Study ResultsOutcome Study Results

• A positive and statistically significant—though small—effect on overall mathematics CST scores in 2008 and 2009. Also showed a significant, and more dramatic, effect on Algebra I test scores in 2008 and 2009. However, not in 2010.

• Science partnerships appeared to have a slight, though positive, impact on science learning in California in 2008, 2009, and 2010. This is mostly attributable to 8th grade performance. Limited to the 5th and 8th grades where the science CST is administered.

CaMSP Evaluation and CaMSP Evaluation and Outcome Study Results (cont.)Outcome Study Results (cont.)

• A more in-depth analysis in the 2008-09 and 2009-10 study of partnerships by dosage/funding cycles completed reveal the diversity of mathematics and science achievement among partnerships. Some partnerships showed marked improvement, while others were little different from the comparison groups.

What are we learning from CaMSP?What are we learning from CaMSP?

• There is a lot of variation in implementation despite rules--ability to be innovative continues to exist yet there are challenges to maintain cohort

• LEA’s are getting used to their role in building partnerships with individuals from IHE’s and building capacity for managing high quality PD, especially in medium size districts; however, rural districts are particularly challenged by participation rules

What are we learning from CaMSP?What are we learning from CaMSP?

• Individuals from IHE’s are tailoring their training and challenging teachers with content knowledge in ways that meet teacher needs often building on experience in Subject matter projects

• Subject matter projects, CPEC grants and other IHE-led professional development exist in parallel with CaMSP but not necessarily integrated

What are we learning from CaMSP?What are we learning from CaMSP?

• Structure and personnel in the classroom follow-up piece is essential to successful implementation and has improved; adaptation to teacher needs is important but fidelity to follow-up model supports institutionalization

• Teacher leaders are emerging in many partnerships and being incorporated in training as facilitators and in district curriculum decision-making committees

What are we learning from CaMSP?What are we learning from CaMSP?

• Lack of involvement of site administrators in planning and PD continues as an area of concern

• Visibility of and continued focus on local evaluation is beginning to lead to higher quality instrumentation and documentation but there is little sharing of what we are learning

Emerging ThemesEmerging Themes

• General panel on Institutionalization to address changes in policy and practice among the partners and what lessons the partnership has taken from the MSP project and infused into the district's culture.

• General panel on Sustainability to address funding the effort once the grant sunsets.

• ESEA Reauthorization and preparing to respond to the direction for MSP program

Discussion & Wrap-upDiscussion & Wrap-up

• What has been most successful in moving our partnership forward?

– Institutionalization of the model

– Integration in district or IHE policy and practice

– What change can we document as a result of this effort?

• How do these efforts contribute to a competitive position for California in mathematics and science education?