ukes process tracing presentation
TRANSCRIPT
![Page 1: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/1.jpg)
1
Process tracing for impact evaluationPotentials and Pitfalls
Melanie PuntonUK Evaluation Society Conference (12 May 2015)
![Page 2: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/2.jpg)
2
Punton & Welle (2015) Straws-in-the-wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and-smoking-guns-what-can-process-tracing-offer-to-impact-evaluation
Stedman-Bryce (2013) Health for All: Towards Free Universal Health Care in Ghanahttp://policy-practice.oxfam.org.uk/publications/effectiveness-review-health-for-all-towards-free-universal-health-care-in-ghana-306376
![Page 3: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/3.jpg)
3
Process Tracing:
A qualitative method for assessing causal inference within a single case
Sits in the ‘evaluator’s toolbox’ alongside• Theory-based evaluation• Contribution analysis• Realist evaluation
![Page 4: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/4.jpg)
4
Why process tracing?
0 Useful where pathways of change are uncertain 0Tells you WHY change happened0 Rigorous approach through ex-post design without a
control group
0Time intensive0 Requires the outcome to be known0 Uncharted territory
![Page 5: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/5.jpg)
5
A brief history of process-tracing
0 A method stemming from the social and political sciences. Major contributors: Alexander George, Andrew Bennett, David Collier
0 Originally used to explain historical events, e.g. the cause of the Cuban Missile Crisis (Allison and Zelikow 1999)
0 A ‘contested method’0 Beach and Pederson’s book (2013): detailed theory and
step-by-step guidance0 Limited examples of its application in evaluation
NB: See Punton & Welle (2015) for references
![Page 6: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/6.jpg)
6
Counterfactual framework
Generative framework
Experimental (RCTs)Quasi-experimental
Process tracingRealist evaluation
Intervention A
Outcome B????
Opening up the black box
![Page 7: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/7.jpg)
7
Causal mechanisms
Intervention A
Outcome B????
Causal mechanism
Generative causal framework
![Page 8: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/8.jpg)
8
Causal mechanisms
Part 3Part 2Part 1
Intervention A
Entity 1 Entity 2 Entity 3
Activity 1 Activity 2 Activity 3
Outcome B
![Page 9: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/9.jpg)
9
The Universal Health Care Campaign in Ghana
0Collaborative advocacy effort by civil society to promote universal free access to health care
0Core funding from Oxfam until 20130Lobbying, mobilising, media and research0Effectiveness review in 2012-13, using elements of
process tracing
Stedman-Bryce (2013) Effectiveness Review: Health for All: Towards Free Universal Health Care in Ghanahttp://policy-practice.oxfam.org.uk/publications/effectiveness-review-health-for-all-towards-free-universal-health-care-in-ghana-306376
![Page 10: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/10.jpg)
10
A (simple) example of a process tracing mechanism
Universal Health Care
campaign
Free universal
health care
Civil society
Conduct coordinated advocacy activities for free universal health care
Part 1
Public
Part 2
Become aware of the limitations of current health care financing
Public
Part 3
Demand free universal health care from government actors
Govern-ment
actors
Part 4
Increasingly support free universal health care, based on desire for public support
Govern-ment
Part 5
Amend policies and processes to move towards free universal health care
Activity
Entity
![Page 11: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/11.jpg)
11
Differences between a mechanism and a theory of change…
0No leaps in logic between the parts0Every part should be necessary0Every part should be observable and empirically
measurable
![Page 12: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/12.jpg)
12
Collecting evidence
• Examples of coordinated advocacy linked to campaign (e.g. events, platforms, meetings)
• Interview data suggesting that the campaign increased knowledge about and interest in healthcare reform among civil society actors
• Number of advocacy events held before the campaign started, compared to number held during campaign
Hypothesis: civil society conducts coordinated advocacy for free healthcare as a result of the Campaign
![Page 13: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/13.jpg)
13
Collecting evidence
• Examples of coordinated advocacy taking place before the campaigns
• Interviews suggesting that the campaign made little difference to activities
Alternative hypothesis: The advocacy campaign made little difference to the coordination or activities of CSOs
![Page 14: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/14.jpg)
14
“We balance probabilities and choose the most likely. It is the scientific use of the imagination.” – The Hound of the Baskervilles
![Page 15: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/15.jpg)
15
Assessing evidence: four tests• Analogous to a criminal trial• The evidence for each hypothesis
is weighed….• …in order to increase the
researcher’s confidence in the hypothesis.
• Bayesian probability logic followed to assess the strength of each part of the causal chain.
• Evidence examined using concepts of necessity and sufficiency.
![Page 16: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/16.jpg)
16
Assessing evidence: four tests
Straw in the wind test
Evidence: hotel receipts, suggestive text messages- Neither necessary
nor sufficient
Hypothesis: John shot Mary
Hoop test
Evidence: John lacks an alibi- Necessary, but not
sufficient
Smoking gun test
Evidence: John was found holding a gun over Mary’s body- Sufficient, but not
necessary
Double decisive test
Evidence: A tamper-proof CCTV camera recorded the murder- Necessary and
sufficient
![Page 17: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/17.jpg)
17
Assessing evidence: four testsU
niqu
enes
s of
evi
denc
e
Low (evidence is insuffi cient for h )
High (evidence is suffi cient for h )
Hig
h(e
vide
nce
is
nece
ssar
y fo
r h
)
Low
(evi
denc
e is
not
ne
cess
ary
for
h)
Hoop tests Double-decisive
Straw in the wind tests
Smoking gun tests
![Page 18: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/18.jpg)
18
Assessing evidence: four testsStraw in the wind test
Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care
Evidence:1. Campaign members were given the opportunity to
discuss and debate a report on the complex National Health Insurance Scheme, in order to increase their understanding on the topic
2. Before the report was published, there was little coordinated advocacy for free universal healthcare
Increase the plausibility of the hypothesis but do not prove it or disprove alternative hypotheses. BUT together they provide stronger evidence than they do alone
![Page 19: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/19.jpg)
19
Assessing evidence: four tests
Hoop test
Hypothesis: The Campaign significantly increased the capacity of member CSOs to plan and work together on the issue of free health care
Evidence: Until the campaign started, there was little notable coordinated advocacy on this issue.
Not enough on its own to prove the hypothesis, BUT evidence must pass the test in order to keep the hypothesis under consideration
![Page 20: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/20.jpg)
20
Smoking gun test
Hypothesis: The National Health Insurance Authority in Ghana revised methodology for calculating membership because of pressure created by the campaign
Evidence: Ghana delegation at international meeting stated that the campaign’s report ‘prompted us to revise our figures’ (despite initially publically dismissing the report)
Evidence is sufficient to give high confidence that the hypothesis is true. BUT this type of evidence is hard to come by.
Assessing evidence: four tests
![Page 21: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/21.jpg)
21
The end goal…
0Researcher can express level of confidence in each part of the mechanism
0Mechanism is only as strong as its weakest part0 If the researcher is confident in the whole
mechanism…0…the researcher can express confidence that the
mechanism holds in that particular case: that A caused B, and why.
![Page 22: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/22.jpg)
22
Why process tracing?0 Useful where pathways of change are uncertain 0 Rigorous approach through ex-post design without a control
group0Tells you WHY change happened0Useful in a mixed methods design, or in combination with e.g.
Contribution Analysis?
0 Time intensive0 Requires the outcome to be known0Uncharted territory0 Risk of one weak link causing the whole mechanism to be
rejected
![Page 23: Ukes process tracing presentation](https://reader033.vdocument.in/reader033/viewer/2022042818/55c327f5bb61eb5b348b47da/html5/thumbnails/23.jpg)
23
Questions?
Punton & Welle (2015) Straws-in-the-wind, Hoops and Smoking Guns: What can Process Tracing Offer to Impact Evaluation? http://www.ids.ac.uk/publication/straws-in-the-wind-hoops-and-smoking-guns-what-can-process-tracing-offer-to-impact-evaluation