cdi seminar: 3d impact analysis
TRANSCRIPT
3D impact analysisA new tool to approach impact evaluations
April 23, 2015
CDI is a joint initiative between:
1
and and
For more information: www.ids.ac/cdi or email: [email protected]
Rob D. van den Berg
Visiting Fellow, IDS
Overview
• What is impact?
• What is evidence?
• What is causality?
• What is attribution/contribution?
• Time
• Space
• Scale
2
Impact
• Impact is an ordinary word in the English language
– “the effective action of one thing or person on another; the effect of such action; influence; impression”
• Its meaning cannot be scientifically claimed
• Demand for impact evidence can refer to a wide variety of effects, influences and impressions
3
Evidence
• Evidence is an ordinary word in the English language
– “the quality or condition of being evident; clearness; evidentness”
• Its meaning cannot be scientifically claimed
• Demand for impact evidence can refer to a wide variety of qualities or conditions
4
Causality
• The word “cause” is an ordinary word in the English language
– “A person or thing that gives rise to an action, phenomenon, or condition”
• Its meaning cannot be scientifically claimed
• Demand for evidence of cause and effect can refer to a wide variety of actions, phenomena and conditions
5
Attribution / Contribution
• Both words are ordinary words in the English language, with great variety in meaning
– Attribution: in copyright law, requiring an author to be credited; in marketing, assigning a value to a marketing activity based on desired outcome; journalism, practice of attributing information to its source
– Contribution: donation, sharing, payment, publication, a song by Mica Paris
6
Impact Evaluation
• Focus of Impact Evaluations:
– Impact = evidence of causality between an intervention and the desired effect by establishing a counterfactual through controlled experimentation, which attributes part of the effect to the intervention
• This partially meets the demand for impact evidence in politics, the media and society
• So what to do with other demands?
7
Meeting impact demand
• Broaden the concepts of impact and causality
• Broaden the range of scientific methods and tools
• Develop a framework for understanding demand for impact evidence
• Incorporate issues of time, space and scale
• This is urgent, given the adoption of the Sustainable Development Goals in September 2015
8
Understanding causality
• Schaffer (2013) proposes two kinds of causality: “difference” and “production”– Difference: with/without (counterfactual) analysis – Production: A “produces” B (natural systems, physics &
technology)
• Concepts that include causality:– Catalytic roles (the change agent speeds up change but is
not involved in the change itself)– Dynamic and chaotic systems (feedback loops, iterative
processes, Fibonacci sequences)
• Terry Pratchett: “hardly anything important has a single cause”
9
Issues of time, space and scale
• Some changes can be observed immediately – others take decades– Short-term results: vaccinations, technology transfer, new
livelihood approaches etc.– Either short- or long-term: market transformations, societal
change– Long-term results: health trends, ecosystem services, ozone
layer
• Some changes are local, other regional, national or even global
• Some changes concern one actor, intervention or institution, others involve multiple actors or institutions, and multiple sectors
• Sustainable development involves longer time horizons, overlapping locations and many scales
Matrix of evaluable impact
• Impact can be evaluated at different moments in time: ex ante, in real time and ex post– These can be refined: ex ante tends to be done once, but real
time and ex post have many possibilities– Different processes tend to have different time horizons
• Geographical space runs from local to global– These can also be refined: the boundaries of societies,
economies and natural systems are different from each other and may overlap
• Scales of involvement can go from one actor to a multiplicity, from one market to a full economic system, from one governance level (or sector) to many – Actors, markets and governance may not fully overlap
12
Appraisal
Inception
Implementation
End-of-project
2 years ex-post
5-8 years ex-post
13
Local Regional National Global
One
Multi-
actor
Multi-
sector
Matrix dimensions space and time
14
Ex ante Inception Mid-term End of project
Ex post < 2 years
Ex post 5-8 years
Local
National
Regional
Global
Ecosystem(overlap with other rows)
Experimentation
Mixed methods /
theory of change
approachesMonitoring and data
analysis (including “big
data”)
Matrix dimensions space and scale
15
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Marketchange
Markettransform-ation
Climatechange
Local
National
Regional
Global
Ecosystem(overlap with other rows)
RCTs
Data
analysis
Monitoring
Mixed methods / theory of change
Double evaluand evaluations
Matrix dimensions scale and time
1616
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Marketchange
Markettransform-ation
Climatechange
Ex-ante
Inception
Real-time
End-of-project
Ex-post
RCTs and
quasi-
experimentalData
analysis
Monitoring
Mixed methods / theory of change
Counterfactual analysis
17
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Marketchange
Markettransform-ation
Climatechange
Local
National
Regional
Global
Ecosystem(overlap with other rows)
RCTs
Modelling of data
and
experimentation
(quasi- and natural)
Quasi-
experimental
and QCA
Social
Network
analysis
Delphi
Production causality
18
One inter-vention
Multiple inter-ventions
Enabling environ-ment
Marketchange
Markettransform-ation
Climatechange
Local
National
Regional
Global
Ecosystem(overlap with other rows)
Inspection, validation
before/after data
Verification of data, trend analysis
Systems evaluation
