unpacking the notion of program theory from a complexity lense: can indigenous and cross-cultural...
TRANSCRIPT
Unpacking the notion of program theory from a complexity lense: Can
indigenous and cross-cultural perspectives help?
Sanjeev Sridharan and Janet SmylieCentre for Research on Inner City Health
University of Toronto
Benita van Wyk, Feedback Research & Analytics
Outline
• 1. Why is program theory important and useful?
• 2. What are the limitations of the standard approach to program theory? The view from evaluation
• 3. Ideas for a potential research project
Conceptualizing and Critiquing Programs:
Program Planning and Evaluation
Evaluation Literature
Indigenous/ Cross-Cultural
Literature
Identify concepts to enhance program planning and
evaluation
Implement Concepts in
Two Programs
Evaluate Programs
Basis for dialogue
Sustainability
Connectivity
Dynamics
Heterogeneity
The important questions
What is a program?
Does the nature of the causal assumptions make a difference
in the way we plan programs and evaluations?
Dealing with complexity
An Example of a complex intervenion
Dealing with complexity
An Example: Primary Prevention Have a Heart Paisley
The nature of the causal problem
• Note that the causal assumptions are not clearly specified
• Most assumptions are implied and of the form:
• If (A and B and C and D and E and at time T) then good outcomes might result
• The problem:– Causal contingencies not clearly specified or
understood– Dynamic complexity not well understood
Another example: The Smoking Ban in Scotland
The incompleteness of program theories The logic model while complex is still incomplete
A policy such as the smoking ban might lead to proximal impacts through perhaps an uncomplicated pathway; however the logic of the causal chain for intermediate and long-term outcomes might be incomplete
Additional “programmatic” inputs might be needed.
Information on the nature of such additional interventions can perhaps come from a synthesis of literature plus additional dialogue with stakeholders with an intimate knowledge of the processes by which programmes works.
Defining a complex intervention (MRC, 2000)
“The greater the difficulty in defining precisely what exactly are the ‘active ingredients’ of an intervention and how they relate to each other, the greater the likelihood that you are dealing with a complex intervention.”
System Dynamic Approaches (Sterman, 2006)
Constantly changing; Governed by feedback; Non-linear, History-dependent; Adaptive and evolving; Characterized by trade-offs; Policy resistance: “The result is policy resistance, the
tendency for interventions to be defeated by the system’s response to the intervention itself.”
Features of complex health interventions (Pawson et al., 2004)
The intervention is a theory or theories The intervention involves the actions of people. The intervention consists of a chain of steps These chains of steps or processes are often not linear,
and involve negotiation and feedback at each stage. Interventions are embedded in social systems and how
they work is shaped by this context. Interventions are prone to modification as they are
implemented. Interventions are open systems and change through
learning as stakeholders come to understand them.
An additional purpose of evaluation for complex interventions
Explication
Why?
Multiple components– be explicit about the “active ingredients”
Timeline of impact unclear
Non-uniformity– Contextualizing
Stakeholder involvement: Co-construction of intervention
On causality and complexity
Does a cause operate the same way in simple systems as compared to complex systems?
Are there different kinds of causes? E.g. Famines
Precipitating causes Amplifying causes Causes of vulnerability
Innovations in evaluation design and approaches
So what do programmes do?
A need for a richer vocabulary on programmes as causes– can we learn from other sciences such as evolution or chemical kinetics?
Different metaphors: Programmes as catalysts, programmes as foundations, programmes as operating systems
Understanding complexity in multiple light– The system-as-cause– Understanding dynamic complexity
System-as-Cause
Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.
Meadows DH. Leverage points: places to intervene in a system. Sustainability Institute, 1999. Available at <http://www.sustainabilityinstitute.org/pubs/Leverage_Points.pdf>.
Richardson GP. Feedback thought in social science and systems theory. Philadelphia, PA: University of Pennsylvania Press, 1991.
Sterman JD. Business dynamics: systems thinking and modeling for a complex world. Boston, MA: Irwin McGraw-Hill, 2000.
“Solutions” Can Also Create New Problems
Meadows DH, Richardson J, Bruckmann G. Groping in the dark: the first decade of global modelling. New York, NY: Wiley, 1982.
Merton RK. The unanticipated consequences of purposive social action. American Sociological Review 1936;1936:894-904.
Forrester JW. Counterintuitive behavior of social systems. Technology Review 1971;73(3):53-68.
Policy resistance is the tendency for interventions to be delayed, diluted, or defeated by the response of the system to the intervention itself.
-- Meadows, Richardson, Bruckman
Unpacking the causal contingencies of change
•If (A AND B AND C AND D AND E and at time T) then good outcomes might result
•What does this really mean? Unpacking the complexity of the change process
•What is missing in our understanding of how programs work?
Seeing and understanding connection“You think you understand two because you understand one and one. You think you understand two because you understand one and one.
But you must also understand ‘and’.”
You think you understand two because you understand one and one.
But you must also understand ‘and’.”
-- Sufi Saying
1. The role of sustainability
Why sustaining programs might be key to success
Going further – sustainability of programs at the individual level – the critical importance of
Continuity of Care
2. The limited understanding of heterogeneity
Heterogeneity of needs
Heterogeneity of mechanisms of change
Heterogeneity of interventions
3. The role of connectivity between interventions
Far too much focus of most evaluations on a single intervention as the unit of analysis
Understanding connectivity between programs
Better understanding of ecology of interventions
4. Understanding the relationship between multiple levels of interventions
Many complex interventions require programming at the community, neighborhood, school and individual level
Multilevel alignment across the interventions
4. Multilevel alignment complexity
5. Understanding Dynamic Complexity
Timeline of impact
Developmental Synergies
Anticipated timeline of impact
Dynamic complexity (1)
Dynamic complexity (2)
5. Coordination complexity
The advantages of partnerships are poorly understood
More importantly, collaborative advantages of individual recipients rarely made
Coordination complexity
Cracks in a network complexity
6. Sequential Complexity
Relationship between sequences of interventions poorly understood
Most of our methodologies are extremely limited in understanding such relationships
6. Sequential intervention complexity
7. Structural complexity
Policy resistance
Translating programs to fit within the structures
The system-as-cause
Next Steps
Identify partners
Identify concepts from Evaluation and Indigenous/Cross-Cultural Field/Literature that can help enhance the concepts
Translate concepts into practical testable experiments
