A new way of impact evaluation Who values Whose values Value for what Contents Why do we do this? Shortcomings of current evaluation methods How do we do this? Some first findings Shortcoming 1 Mostly not on impact Focus on ongoing projects Focus on implementation (did we do well?) Focus on direct results: Outputs: What did the project produce? Outcomes: How did the target group make use of the outputs of the project? Impact: What are the sustainable results in the communities? Shortcoming 2 Lack of attention for context (even in impact assessments) What did other projects do? External factors? backdonor INGO NGO Project a In community x Activities Results Outputs Outcomes Impact M&E Project c Project b backdonor INGO NGO 1 Project a Community x Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Private initiatives Government Local State National Companies (e.g. telecom) Other INGOs Other backdonors Project c Project b backdonor INGO NGO 1 Project a Community x Very poor poor average rich very rich Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Government Local State National Companies (e.g. telecom) Private initiatives Other INGOs Other backdonors Local influences National influences Global influences Natural Physical Economic Human Social Political Cultural Community x Very poor poor average rich very rich Government Local State National Companies (e.g. telecom) Private initiatives Project c Project b backdonor INGO NGO 1 Project a Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Other INGOs Other backdonors Local influences National influences Global influences Natural Physical Economic Human Social Political Cultural Project c Project b backdonor INGO NGO 1 Project a Community x Very poor poor average rich very rich Project c Project b NGO 2 Project a Project c Project b NGO 3 Project a Government Local State National Companies (e.g. telecom) Private initiatives Other INGOs Other backdonors Local influences National influences Global influences Natural Physical Economic Human Social Political Cultural Shortcoming 3 Often mainly based on expert opinion What the expert defines as impact Often looking for predetermined results only Little attention for unexpected effects Shortcoming 4 If evaluations look at target group perceptions, there is almost always bias yes sir, we like the project and we want some more Shortcoming 5 Focus on what and little on how, with which values or why projects were done How do we do this? Prisma with ICCO and Woord en Daad with University of Amsterdam and University of Development Studies and Expertise Dveloppement Sahel In Burkina Faso and Ghana backdonor INGO NGO Project a In community x Not this perspective backdonor INGO NGO Project a In community x backdonor INGO NGO Project Project a In community x Community x Very poor poor average rich very rich But this perspective ProjectsActors Changes in context 3 day workshops 60 people from area of 20,000 Subgroups: men, women, old, young (officials and project staff separate group) Individual life history questionnaire + data about parents + data about brothers / sisters + data about children (total about 600 persons per workshop) 4 rounds of 3 workshops each Part 1: context What were major events? What are changes over past 30 years? Are these positive or negative (natural, physical, human, economic, social/political, cultural) Who are considered the Very rich Rich Average Poor Very poor by local standards? Part 2: interventions Which interventions happened? (by government, private sector, secular ngos, christian ngos / churches, muslim ngos, individuals) How do you assess their impact? (negative impact, no impact, impact in past, positive impact) + explanations !!! Part 3: Best and Worst What were best 5 and worst 5 projects? and why? What did you tell your friends about these projects when they started? And what do you tell your friends about them now? What is the effect of these projects on the 5 wealth classes? What is the effect of these projects on the different domains? What is the link with the changes in context? (cause / mitigation) And then 60 small reports per workshop Analyse, compare and combine all subgroups First findings Incredibly detailed historical profile Big picture: Natural domain: negative Human domain (education / health): positive Socio/political: mixed Cultural: mixed Economic: we have more but we feel poorer Wealth categories: rich anthropological descriptions (meals, behaviour, jobs, funeral, family life, properties, etc.) Very different per region Numbers of Projects Best projects by actor Worst projects by actor (Ghana) But different in Burkina Effects best pr. on Wealth Classes (Sandema, Ghana) Effects best pr. on Wealth Classes (T, Burkina Faso) Reasons exclusion very poor Miss preconditions for benefiting Knowledge about the existence of an intervention Access to the place where selection takes place Having time available for themselves (or their children in case of education) to take part in the activities of an intervention Having the feeling that they are allowed to participate (e.g. feeling that their dresses are sufficiently appropriate to be part of a meeting) Being involved in the activities at which the intervention is focused (e.g. veterinary services will not benefit the very poor in some societies, since they do not have livestock) Reasons for best/worst Hard results Relevance Manner in which projects are done Respect Mutual trust (we can trust them and we feel that they trust us) Participation (decision making, monitoring) Long term commitment (they have us at heart) Honesty (something went wrong and they did not explain to us) Local presence (they are one of us) Results / expectations ratio Expectations about length of stay (they left and never came back) Expectations about results (poverty will be ended here) Negative side effects Some final conclusion Projects have many side effects Best projects have effects in more than 1 domain (often including economic) There are no black and white conclusions Always nuances and exceptions Christian among best and worst Differences between men / women, old / young Some of the biggest negative changes are hardly addressed Climate change Loss of cultural identity Negative sides of individualism Men, Women and Project officers all have different opinions Questions ?