measuring educational interventions in context

Measuring educa-onal interven-ons in context: A system for iden-fying how and why instruc-onal materials are effec-ve

Amy Cassata, PhD and Jeanne Century, PhD Flash Presenta5on at the Third Biennial Global Implementa5on Conference Dublin, Ireland, May 29, 2015

Educa&onal interven&ons are curricula, tools, teaching approaches, programs, or prac5ces designed to improve student outcomes.

Instruc&onal materials are one type of educa5onal interven5on. They are resources designed for use by teachers in classrooms to guide their instruc5on.

To help guide selec5on decisions, instruc5onal materials are evaluated based on evidence of their effec5veness in improving student achievement.

The evidence is organized and summarized to help school and district leaders decide which instruc5onal materials to adopt in their seLng.

The expecta5on is that teachers’ use of these materials will lead to large-‐scale changes in prac5ce and in turn, increased levels of student achievement.

Evidence-‐based Instruc5onal materials

Large-‐scale changes in teacher prac5ce

Improved student achievement

But how do schools and districts get from point A, to point B, to point C?

D Evidence-‐based Instruc5onal materials

Large-‐scale changes in teacher prac5ce


Con5nued posi5ve outcomes

TIME

And, how do they ensure that any changes in teacher prac5ce, and effects on student outcomes, will be sustained over 5me?

For the past several years, our research team has asked these same ques5ons.

We developed a theory of implementa5on to understand and measure interven5ons in prac5ce, the factors that influence their use, and their rela5onships to outcomes.

Our theory begins by clearly describing an interven5on and its essen5al components.

We then consider the context in which the interven5on is used, which also has many elements.

THE CONTEXT THE CONTEXT

Characteris5cs of individual end-‐users include demographics, disposi5ons, and percep5ons.

Percep5ons

Individual beliefs

Mo5va5on

Self-‐efficacy

Innova5veness

Resourcefulness

Experience

Demographics

THE CONTEXT

Characteris5cs of the organiza5on include people, structures, and strategies.

THE CONTEXT THE CONTEXT

Ongoing improvement structures

Organiza5onal innova5veness

Time

Locus of decision making

Clarity of

organiza5onal goals

Learning opportuni5es

Leadership

Elements of the environment include policies, poli5cs, and community characteris5cs.

THE CONTEXT

Poli5cal environment

Extraneous events

Community

characteris5cs

Community beliefs

Learning

opportuni5es

THE CONTEXT

Networks are systems of interconnected people or organiza5ons.

THE CONTEXT THE CONTEXT Network structure

Member

rela5onships

Trust

Communica5on

Change and evolu5on

The interven5on is enacted in a context over 5me, which means that elements of the context change.

THE CONTEXT THE CONTEXT THE CONTEXT

And in turn, interven5on components may be adapted to fit the changing context.


All of these elements together make up the IMPLEMENTATION PROCESS.


In short, Implementa5on is complex!


Large-‐scale changes in Teacher prac5ce



TIME

Based on this theory, we created a suite of instruments to measure the implementa5on of instruc5onal materials in school systems.





TIME

Part of this work involved a research study to understand the implementa5on of the Everyday Mathema&cs curriculum in five school districts.

“poten5ally posi5ve effects” on math achievement -‐ What Works Clearinghouse (2010)





TIME

We focused in par5cular on measuring teacher varia5on in EM use, and understanding why teachers used EM in par5cular ways.

We administered two ques5onnaires to 387 K-‐5 teachers to collect data on which EM components they were using, and why.

EM essen5al components

The first ques5onnaire asked teachers to report the extent to which they used each of the EM components in the most recent unit they taught.

Items measured teachers’ use of EM structures (ac5vi5es, procedures, materials, and content), their interac5ons with students, and level of student par5cipa5on.

EM essen5al components

The second ques5onnaire asked teachers to report on selected teacher and school-‐level factors.

THE CONTEXT

Percep5ons about EM

Beliefs about student ability & behavior

Extrinsic mo5va5on

for EM

Self-‐efficacy for EM

Innova5veness

Resourcefulness

Years of experience

Ongoing improvement structures

Organiza5onal innova5veness

Time

Locus of decision making

Clarity of

organiza5onal goals

Facilita5on of Small Group Par5cipa5on

Facilita5on of Cogni5vely Demanding Work

Facilita5on of Students Taking Risks

Facilita5on of Student Interest

Use of Assessment to Inform Instruc5on Use of Differen5a5on

Through confirmatory factor analysis, we iden5fied six specific EM instruc5onal prac5ces.

We conducted mul5ple regression analysis to see which factors were most strongly related to which EM prac5ces.

THE CONTEXT

Beliefs about student ability Beliefs about student behavior Perceived effec5veness for EM Extrinsic mo5va5on for EM Self-‐efficacy for EM Innova5veness Resourcefulness Years of experience

Ongoing improvement structures Clarity of organiza5onal goals Organiza5onal innova5veness Locus of decision making Time sufficiency

? EM prac5ces

?

We found that teacher characteris5cs were more influen5al than school characteris5cs.

THE CONTEXT

Beliefs about student ability Beliefs about student behavior Perceived effec5veness for EM Extrinsic mo5va5on for EM Self-‐efficacy for EM Innova5veness Resourcefulness Years of experience

Ongoing improvement structures Clarity of organiza5onal goals Organiza5onal innova5veness Locus of decision making Time sufficiency

? EM prac5ces

?





TIME

Next, we were interested in which of these EM prac5ces were associated with changes in student math achievement.

We conducted a path analysis to predict student math achievement using data from one district (N=125 teachers).

We es5mated a path model to predict math achievement for students in grades 2-‐5.

Variables controlled for Independent variables Dependent variable

The model contained the 6 EM instruc5onal prac5ces we iden5fied in CFA.

EM Instruc-onal prac-ces

Facilita5on of small group par5cipa5on

Facilita5on of cogni5vely demanding work

Facilita5on of students taking risks

Facilita5on of student interest

Use of assessment to inform instruc5on

Use of differen5a5on

The model also contained 6 EM structures.

EM Structures

Number lessons taught in order

Number of lessons with parts in order

Number of lessons omiked

Number of lesson with parts omiked

Amount of content covered in the unit

Non-‐EM materials supplementa5on

Significant, posi5ve direct effects on student achievement included: ✔ Non-‐EM supplementa5on ✔ Prior achievement

Significant, posi5ve indirect effects on student achievement included: ✔ Facilita5on of cogni5vely demanding work ✔ Facilita5on of student interest ✔ Number of lessons taught in order ✔ Prior achievement

For more informa5on, contact Amy Cassata at [email protected] Century, J., & Cassata A. (2014). Conceptual founda5ons for measuring the implementa5on of educa5onal innova5ons. In L.M.H. SaneL & T.R. Kratochwill (Eds.), Treatment Integrity: A Founda&on for Evidence-‐Based Prac&ce in Applied Psychology (pp. 81-‐108). Washington, DC: American Psychological Associa5on.

This work was funded by the Na5onal Science Founda5on (#1109595) and the US Department of Educa5on -‐ Ins5tute of Educa5on Sciences (#R305A110621).

Acknowledgements

measuring educational interventions in context

Education

opportuni5es thecontext

andeectsonstudent outcomes

forthepast severalyears

implementa5on iscomplex