researchers without borders webinar 4 a framework and suite of instruments for examining fidelity of...
TRANSCRIPT
Researchers Without Borders Webinar 4
A Framework and Suite of Instruments for Examining Fidelity of Implementation
Jeanne CenturyCenter for Elementary Mathematics and Science Education
University of ChicagoJuly 7, 2010
This work has been supported in part by the National Science Foundation Project ESI-0628052
Why Measure Fidelity of Implementation (FOI)?
Determine if programs are effective
Determine which parts of programs are effective
Determine why and under what conditions parts of programs are effective
What is actually happening?
NSF Project Challenges
Instruments that could be used across multiple programs
A suite of instruments that could be used alone or in combination depending on resources and goals
Provide detailed and specific information about FOI
Five Dimensions Approach Dane and Schneider (1998)
39/162 studies measured integrity In the 39 studies, the authors defined “integrity” along one or more of five dimensions They are: 1) adherence; 2) exposure; 3) quality of delivery; 4) participant responsiveness and
5) program differentiation
Critical Components Approach
Hall and Hord (1987), Bond et al. 2000; Huntley, 2005; Sabelli & Dede (2001) and others
Some called them model dimensions, fidelity criteria, critical parts
Structure and Process ApproachMowbray et al. (1987), Wang et al. (1984)
Structure (composition of the intervention)Process (human interactions that take place during delivery)
FOI Approaches…
Combining Frameworks Approach
Ruiz Primo (2005) – matrix of 5 dimensions and structure/processLynch and O’Donnell (2005); Lastica and O’Donnell (2007) – matrix of 5 dimensions and structure/process
CEMSE Approach
combine critical component approach and structure process approachshow relationships to 5 dimensions approach
use combination of approaches to create categories of critical components and a framework that can apply to multiple interventions
Questions?
FOIThe extent to which an enacted program is consistent with the intended program model
Critical ComponentsThe elements of a program model that are essential to its implementation
FOI OperationalizedThe extent to which the critical components of a program are present when the program is enacted
Definitions
Written Materials
Developers
Critical Component Identificationtheory…
Users
ChallengesTheories aren’t always explicitCritical components very in “grain size”The “it” can vary, depending on how the intervention is framed
Critical Component Identificationevidence…
FOI of Instructional Materials
Categories of Critical Components
Structural Instructional
Procedural Educative Pedagogical Student Engagement
FOI Framework
FOI of Instructional Materials
Categories of Critical Components
Structural Instructional
Procedural Educative Pedagogical Student Engagement
Inclusion of all essential segments
within a lesson
Use of class structures
Content background information
Lesson Notes
Teacher facilitation of group work
Teacher facilitation of student autonomy
Students contribute to small group work
Students demonstrate autonomy
FOI Framework and Critical Components
FOI of Instructional Materials
Categories of Critical Components
Structural
Procedural Educative
Time 1. Duration of unit 2. Time spent on instruction
Order 3. Lesson order 4. Order of segments and activities within lesson
Inclusion 5. Inclusion of all essential segments within a lesson 6. Inclusion of all essential lessons
Essential Program Elements 7. Materials presence 8. Writing structures 9. Readings 10. Assessments and assessment tools 11. Content of lesson (procedures, facts, concepts, processes) 12. Use of class structures 13. Use of instructional delivery formats
Non-Essential Program Elements* 14. Projects* 15. Extensions* 16. Additional Resources* 17. Homework*
1. Content background information2. Pedagogy background information3. National standards and benchmarks information*4. Lesson notes
FOI Framework and Critical Components
FOI of Instructional Materials
Categories of Critical Components
Instructional
Pedagogical Student Engagement
Facilitating student engagement with others1. Teacher facilitation of group work2. Teacher facilitation of student discussion
Facilitating student engagement with content3. Teacher facilitation of students doing intellectually challenging
work4. Teacher emphasis on types of content
Facilitating student role as learner5. Teacher facilitation of student autonomy6. Teacher facilitation of students taking risks7. Teacher facilitation of student interest
Pedagogical strategies14. Teacher facilitation of materials, manipulatives and tools15. Teacher use of assessment to inform instruction16. Teacher use of differentiation
Students engage with others1. Students contribute to small group work2. Students engage in discussion
Students engage with content3. Students do potentially intellectually challenging work
Students develop role as learner4. Students demonstrate autonomy5. Students take risks
Students use the materials6. Students do/complete essential activities7. Students do/complete optional or non-essential activities
FOI Framework and Critical Components
Questions?
FOI of Instructional Materials
Categories of Critical Components
Structural Instructional
Procedural Educative PedagogicalStudent
Engagement
Categories of Differentiation
MathematicsAnd Science
Science Only
Mathematics Only
Program Specific
FOI Framework and Differentiation
FOI of Instructional Materials (adherence)
Categories of Critical Components
Structural (structure) Instructional (process)
Procedural (exposure;
dosage)Educative
Pedagogical (quality)
Student Engagement (participant
responsiveness)
Categories of Differentiation (differentiation)
Common Across Interventions
Unique to Interventions
CEMSE Framework Aligned with (Others’ Approaches)
Questions?
InstrumentsTeacher QuestionnaireTeacher Instructional Observation ProtocolTeacher Interview ProtocolTeacher Instructional LogTeacher Attitude QuestionnaireSchool Leader QuestionnaireSchool Leader Interview ProtocolSchool-wide Observation Protocol
Structural Procedural
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
TeacherQuestionnaire
X X X X X X X X X X X X X X X X
School LeaderQuestionnaire
X X
Teacher Interview Protocol
X X X X X X X
School Leader Interview Protocol
X X
Classroom Observation Protocol
X X X X X X X X X X X X
School Wide Observation Protocol
X X X X X X
Teacher Log X X X X X X X X X X X X
First Teacher Log X X X X X
Teacher Attitude Questionnaire
X
Sample Instrument Construct Matrix
FOI of Instructional Materials
Categories of Critical Components
Structural Instructional
Procedural Educative Pedagogical Student Engagement
Categories of Differentiation
Common
Program A
Program B
Program C
Impact of a single programCompare two or more programsCompare programs to “business as usual”Compare two versions of a program
Uses of the Framework and Measures
Developers
U.S. Satellite, Inc. – additional elements
EvaluatorsSchool–based change model – Lindblom High SchoolProgrammatic model – E2SP
Researchers
Trailblazers – look at a critical component in more depth
Programs Science and Technology for ChildrenFOSSSEPUPIESEveryday Mathematics
Science Companion
Jeanne CenturyCenter for Elementary Mathematics and Science Education
University of [email protected]
www.researcherswithoutborders.org
FOI Supplement Work Researchers Without Borders
Resources FAQs; Research; Tools and Instruments
Learning Webinars; Collaboration
Community Development www.researcherswithoutborders.org;
This work has been supported in part by the National Science Foundation Project ESI-0628052