1 a tool for designing evaluations of paradigm shifts in complex system interventions beverly...
TRANSCRIPT
1
A Tool for Designing Evaluations
of Paradigm Shifts in Complex System Interventions
Beverly ParsonsInSites
Pat JessupInSites
Marah MooreI2i Institute
Handouts and PowerPoint presentation available at:
www.insites.org/AEA2010
2
Session Objective
To build capacity of evaluators to design evaluations of interventions in complex systems
3
Four Phases of Evaluation
Collect Data
MakeMeaning from
Data
Design Evaluation
Shape Practice
4
Tool’s Role in Evaluation Design
Tool represents the intervention’s theory of change and relationship of intervention to larger system/situation
Evaluation designers also need to know:– Whether evaluation is part of intervention (internal) or is
separate from intervention (external)– If evaluation is part of intervention, does it participate through
• Evaluation processes and/or• Evaluation findings
5
Session Agenda
Introduction to tool for designing evaluations of interventions in complex systems
Work in small groups to begin tool development
Reconvene to discuss completion of tool and its use
6
The Tool
7
Theory of Change in Paradigms, Structures, and Conditions of Complex SystemsExample from Cross-Site Evaluation of Quality ImprovementCenter on Early Childhood
8
Tool Characteristics
Tool focuses on deep patterns of change in systems (i.e., changes in structures and paradigms)
Tool recognizes multiple system dynamics at play in complex systems
9
Differences from Typical Logic Model
Will almost always depict more aspects of the system than just the intervention
Depicts phases of change over time with variation in rates and patterns across components of the system
Likely to depict the intervention through a different lens
10
Background Concepts
11
What is a System?
A group of interacting, interrelated, and interdependent elements forming a complex whole
A configuration of parts connected and joined together by a web of relationships
The whole is greater than (different than) the sum of its parts
12
The Systems Iceberg
Events and Behaviors
Patterns
StructuresParadigms Conditions
What is happening now?
How do patterns play out over time and space?
What are the drivers? How are they related?
13
Deep System Change
Deep system change involves shifts in paradigms and structures of system boundaries, relationships, perspectives, and dynamics over time and space.
Change is evident in mid-level patterns through data analysis/synthesis.
Change is evident in observable events and behaviors.
14
Paradigm Shift: Risk to Protective Factors
Risk Factors• Substance Abuse• Poverty• Stress
Protective Factors• Parental resilience• Social connections• Concrete support in time
of need• Knowledge of parenting
and child development• Social and emotional
competence of child
15
Dynamics of a Social System and Its Context
Unorganized (random)
Organized(simple,
complicated)
Self-Organizing(complex, adaptive)
RelationshipsHigh
PredictabilityLow
Predictability
Pers
pect
ives
Hig
h
Ag
reem
en
tLo
w
Ag
reem
en
t
Context
16
Group Task
Succinctly describe the paradigm shift.
Identify the ultimate beneficiary of the paradigm shift.
Identify the likely key points of leverage to shift the system.
Order the likely points of leverage along left side of page beginning with the ultimate beneficiaries of the system and those close by.
17
Phases of Shift
Baseline understanding of fundamentals and system dynamics
Trying out interventions in system
Tipping point to new fundamentals and system dynamics balance
Sustainable adaptive balance of new fundamentals and system dynamics in a shifting context
18
Theory of Change in Paradigms, Structures, and Conditions of Complex SystemsExample from Cross-Site Evaluation of Quality ImprovementCenter on Early Childhood
19
Other Key Evaluation Design Questions
Is the evaluation internal or external to the intervention?
What are the boundaries of the intervention?
What are the focal points of the evaluation?
How will the evaluation attend to the larger system (beyond the intervention)?
20
Example of Evaluation Design
21
QIC-EC Cross-Site Analysis Framework
22
A Tool for Designing Evaluations
of Paradigm Shifts in Complex System Interventions
Beverly ParsonsInSites
Pat JessupInSites
Marah MooreI2i Institute
Handouts and PowerPoint presentation available at:
www.insites.org/AEA2010