comprehensive evaluation of the european union ......comprehensive evaluation of the european union...
TRANSCRIPT
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 1
Comprehensive evaluation of
the European Union humanitarian aid, 2012-2016
Annexes to the Draft final report
Volume 1
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 2
Table of Contents
Contents
Annex 1 Cost-effectiveness assessment framework ............................................ 5
A1.1 Introduction ...................................................................................... 5
A1.1.1 Structure .................................................................................... 5 A1.1.2 Methodological notes ................................................................... 5
A1.2 Cost-effectiveness of ECHO as a donor ................................................. 7 A1.3 Cost-effectiveness of ECHO-funded actions ..........................................12 A1.4 Additional descriptive indicators ..........................................................14
Annex 2 Full list of interviewed stakeholders .....................................................15 Annex 3 Methodological notes on the sources and use of the quantitative data .....18
A3.1 EVA data .........................................................................................18
A3.1.1 Source of EVA data .....................................................................18 A3.1.2 Content of EVA data ...................................................................19
A3.2 UN OCHA FTS funding data ................................................................20
A3.2.1 Source of FTS funding data ..........................................................20 A3.2.2 Content of FTS data ....................................................................20 A3.2.3 Comparison of ECHO’s EVA data and UN OCHA FTS data .................21
A3.3 Appeals data ....................................................................................24
A3.3.1 Introduction to the appeals process ..............................................24 A3.3.2 Data on the Appeals and Response Plans .......................................25 A3.3.3 Limitation to the FTS response plan data extracted .........................27 A3.3.4 Appeals coverage and funding gap ...............................................28
Annex 4 Profile of respondents to the three surveys ..........................................29
A4.1 Respondents to the ECHO field survey .................................................29 A4.2 Respondents to the framework partner survey......................................30 A4.3 Respondents to the local partner survey ..............................................33
Annex 5 Meta-evaluation ................................................................................35
A5.1 Introduction .....................................................................................35 A5.2 Findings of the meta-analysis by evaluation questions ...........................36
A5.2.1 Relevance .................................................................................36 A5.2.2 Coherence .................................................................................53 A5.2.3 EU added value ..........................................................................71 A5.2.4 Effectiveness .............................................................................76 The extent to which ECHO funded actions effectively promote the visibility of
ECHO activities ...........................................................................................87 A5.2.5 Efficiency ..................................................................................88 A5.2.6 Sustainability .............................................................................96
Annex 6 Field reports ...................................................................................... 1 Annex 7 Rapid evaluations .............................................................................. 1 Annex 8 Relevance ......................................................................................... 1
A8.1 Supporting evidence EQ1: Needs-based approach in ECHO ..................... 1
A8.1.1 Review of the needs assessment process in ECHO ........................... 1 A8.1.2 Level of needs, ECHO funding, global funding, humanitarian appeals . 3
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 3
A8.2 Supporting evidence for EQ2: ECHO’s choice of partners ......................... 9 A8.3 Supporting evidence on EQ3: Appropriate, comprehensive and context-adapted strategies .........................................................................................17 A8.4 Supporting evidence on EQ4: Field network .........................................19
Annex 9 Coherence .......................................................................................31
A9.1 Survey results ..................................................................................31 A9.2 Activation of the UCPM in context where ECHO was covering the same crisis from a humanitarian aid perspective ........................................................32
Annex 10 EU added value ..........................................................................35 Annex 11 Effectiveness .............................................................................41
Figure 1. Local implementing partners: In your view what has been ECHO’s role in… 42
Annex 12 Efficiency ..................................................................................44
A12.1 Statistics on use of different transfer modalities ....................................44 A12.2 Statistics on portfolio levels by TA for different parts of the world ...........46 A12.3 ECHO budget funding sources ............................................................46 A12.4 Indicators for cost-effectiveness analysis .............................................48 A12.5 Survey results ..................................................................................50 A12.6 Available benchmarks for TCTRs .........................................................53
Table 1. Average TCTRs in ECHO funded projects 2011-2014 .......................53
Annex 13 Prospective evaluation ................................................................55
A13.1 List of EU commitments for the World Humanitarian Summit ..................55
Annex 14 OPC analysis ..............................................................................58
A14.1 Introduction ..................................................................................58 A14.1.1 Purpose and scope of the open public consultation .................58 A14.1.2 Structure of the report ............................................................58 A14.2 Overview of responses to the open public consultation .................58 A14.2.1 Number and distribution of replies received ............................58 A14.2.2 Distribution by level of familiarity with DG ECHO humanitarian aid activities ...............................................................................................59 A14.3 Analysis of the survey responses ...................................................59 A14.3.1 DG ECHO performance 2012-2016 ...........................................59 A14.3.1.1 Relevance ................................................................................60 A14.3.1.2 Coherence ...............................................................................61 A14.3.1.3 EU added value ........................................................................66 A14.3.1.4 Effectiveness ...........................................................................68 A14.3.1.5 Sustainability ..........................................................................69 A14.3.1.6 Visibility ..................................................................................71 A14.3.2 Follow-up to DG ECHO's international commitments and global humanitarian challenges .............................................................................71
Annex 15 Terms of reference .....................................................................76 1. General context ......................................................................................76 2. EU Humanitarian Aid ...............................................................................78
2.1. Legal and strategic framework ................................................................78 2.2. Objectives of EU humanitarian aid operations ...........................................81 2.3. The EU intervention in the humanitarian field ...........................................81 2.4. Implementation and financial framework .................................................83 2.5. Evaluations and related exercises ...........................................................84
Overall exercises .........................................................................................84 Recently completed geographic and thematic evaluations ................................84
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 4
Recently launched evaluations (on-going) ......................................................85
2.6. Cost-effectiveness guidance for ECHO evaluations ....................................85
3. Purpose and scope of the evaluation ..........................................................85
3.1. Purpose ...............................................................................................85 3.2. General scope ......................................................................................86 3.3. Evaluation questions .............................................................................87 3.4. Supplementary tasks.............................................................................88
4. Management and supervision of the evaluation ...........................................89 5. Specific requirements ..............................................................................89
5.1. Methodology ........................................................................................89 5.2. Evaluation team ...................................................................................90
6. Content of the Offer ................................................................................91 7. Budget of the contract .............................................................................91 8. Timetable ...............................................................................................92 9. Phases of the evaluation ..........................................................................92
9.1. Inception Phase ....................................................................................93 9.2. Desk phase ..........................................................................................93 9.3. Field Phase ..........................................................................................94 9.4. Synthesis Phase ...................................................................................95 9.5. Dissemination and Follow-up Phase .........................................................96
10. Award ....................................................................................................97 Annex 1: Cost-effectiveness guidance for ECHO evaluations ..................................98 Annex 2: The final report ...................................................................................98
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 5
Annex 1 Cost-effectiveness assessment framework
A1.1 Introduction
A1.1.1 Structure
The general EQ for this framework is: “To what extent did ECHO achieve cost-
effectiveness in its response?”
In line with ADE guidance, the framework assesses the following two dimensions:
1- Cost-effectiveness of ECHO as a donor
2- Cost-effectiveness of ECHO funded actions.
This framework closely follows the ADE guidance; we have not tried to re-invent the
wheel although some improvements have been made to the ADE guidance.
Specifically, we have added decision rules for each indicator to make the process of
drawing evaluative judgement clear, transparent and as objective as possible.
The framework is focused on key aspects directly linked to cost-effectiveness – coming
from relevance and efficiency sections. Other aspects also feed into cost effectiveness:
‘Effectiveness’ itself, coordination/ coherence, configuration of field network. These are
covered under separate evaluation questions of the evaluation framework.
For easier reference, the evaluation framework from the inception report is pasted in
annex and those questions covered by CEA framework are highlighted.
We also included a table with descriptive background indicators which are not
associated to any decision rules (and are therefore not part of the CEA framework per
se but are nevertheless there to help the team have a better understanding of the
issues).
A1.1.2 Methodological notes
As our evaluation is a comprehensive, strategic one, we have more indicators related
to the first part ‘cost-effectiveness of ECHO as a donor’.
The analysis is also mostly based on level 1 tools1 rather than level 3 tools2 – as
current evaluation budget and timescales do not permit intensive data collection and
analysis foreseen for level 3 analysis. Moreover, past evaluations do not allow us to
carry out a deeper analysis at the intervention level (CEA-related issues are only
covered to a limited extent and mostly qualitatively).
More specifically, the following list indicates which aspects - related to ‘cost-
effectiveness of ECHO funded actions’ - are not covered in the CEA framework and for
which reasons:
Cost-effectiveness or cost benefit ratio (CER): calculated by using the total costs as the numerator and the quantitative benefits as the denominator.
Producing CER even by sector would be too difficult. Aside the from usual
1
Level one tools require low to medium additional effort with a focus on qualitative evidence (and quantitative evidence where possible and easy to access) through document review and interviews that can be easily added to existing evaluation interview schedules or scanning for new information within a broader review of documents. 2
Level three tools require the most effort, time and skills as the data required for the analysis is not generated routinely in ECHO management and monitoring systems. It should not be expected for all evaluations to use level three tools unless dedicated resources are provided, however these tools may be of particular importance in specific contexts or may become more relevant over time with management and monitoring system changes that more readily enable the analysis.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 6
problems of lack of indicators on which to base the analysis and attribution
issues, the main issues are that:
Not all benefits can be monetised and summed up: Projects even within the same sector produce multiple outcomes including non-quantifiable ones which
cannot be aggregated.
Only the health sector has its routinely used summary measure of health outcome, namely quality-adjusted life year (QALY) and even QALYs (which
incorporate the impact of interventions on both the quantity and quality of life)
have several limitations which matter a lot in HA context e.g. the fact that a
QALY is a QALY regardless of who accrues it (equity aspect - the distribution of
health is neglected)3.
Other sectors do not have a single, measurable outcome indicator.
Such CER ratios are not very helpful in absence of benchmarks: Comparable benchmarks often do not exist due to specificity of crisis situations and donor
interventions: there is lack of comparability of interventions in general
(differences in timing, programme design and location -e.g. more accessible,
less accessible-, scale, programmes with start-up costs vs on-going projects). .
Alpha ratio: value of transfers reaching beneficiaries. Calculating the alpha ratio for a larger sample than the Inspire study would have implied level 3 analysis
which is not feasible within the scope of the assignment – e.g. it would have
implied completing a fiche for each project based on financial and narrative
reports submitted to ECHO (like we did for project mapping but in more detail
onbudget data of partners) but also collecting additional data provided by
partners in response to questions of clarification.
Cost per beneficiary or cost per unit of result is calculated by dividing the budget of a programme or an output by the number of beneficiaries or units of
result. Calculating these ratios was not possible in the present case.
Project mapping illustrated that even for key results, costing data is not available in a format to calculate cost per unit.
Interpreting cost per beneficiary or cost per unit of result implies making some comparisons: examples of possible comparisons include cash transfers v. food
aid to deliver an amount of calories or nutritional value; funding an intervention
through one NGO v. another NGO; the cost of an intervention v. the average
cost of similar interventions in that context. Interventions even within the same
sector are not comparable when looking at the global portfolio of ECHO which
comprises interventions implemented in very different contexts (see earlier
point on lack of comparability of interventions in general).
Overall number of beneficiaries: an aggregate figure (in total or by sector) cannot be retrieved from the information systems because of double counting,
e.g.
- The same project has activities in various sectors: summing counts of beneficiaries per sector could actually imply counting all the same
beneficiaries again if they benefit from interventions in several sectors
(double counting of the beneficiaries).
- The same beneficiaries can also benefit from a follow up project in the same year
3 Due to its limitations and ethical issues, this approach was not adopted in the sector specific health
evaluation
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 7
- Also difficult to give a limit to the definition of a beneficiary. There can be indirect beneficiaries to some projects too. Ex: building of public
infrastructure.
A1.2 Cost-effectiveness of ECHO as a donor
[Relevance section]
Judgement criterion: Available evidence suggests that ECHO’s budget allocations
are based on needs (including gaps lefts by other donors) rather than political
considerations
Indicator Indicator
type
Decision Rule(s) Source of
evidence
The extent to which ECHO’s
strategic programming (GGOPHAs / HIPs) is underpinned by
a robust and impartial needs assessment
Qualitative INFORM, FCA and IAFs are robust and based on reliable
methodologies and data
There is clear link between ECHO funding allocations to specific crises and needs while
taking account of gaps left by other donors
Countries/crises with the most needs and vulnerabilities have been allocated more funding as compared to countries with fewer needs
GGOPHA review
Review of outputs
of INFORM, FCA and IAF and HIPs
Interviews with ECHO staff and
experts
The extent to
which stakeholders believe that ECHO’s budget
allocations are based on needs
Qualitative The general opinion across
different categories of stakeholders interviewed is that ECHO’s budget allocations are
based on needs while taking account of gaps left by other donors
Stakeholder
interviews
% of Framework partners who
strongly agree/ agree with the following statement: “ECHO funding allocations to
specific regions/ countries are based needs assessment rather than other factors
such as political considerations”
Quantitative At least 75% strongly agree/ agree
Q.21 of online survey of
framework partners
ECHO funding
allocations to specific crises/ sectors overtime
Quantitative Echo funding to specific sectors
and crises over time has evolved with needs of the sector/ crises
ECHO funding
allocations as per EVA / contracted amounts per HIPs
http://dgecho-partners-helpdesk.eu/financing_decisions/dgecho_strategy/start
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 8
Indicator Indicator
type
Decision Rule(s) Source of
evidence
% of ECHO funding to
different regions over time
Share of funding has evolved to reflect changing needs of
regions
HOPE / EVA database
/ contracted amounts per HIPs
% of ECHO funding allocated to forgotten crises
Quantitative ECHO's allocation to forgotten crises is reasonable in relation to needs.
HOPE / EVA database
Forgotten crisis index
ECHO funding allocated to forgotten crises as
a % of funding to forgotten crises
Quantitative ECHO is one of the main donors to forgotten crises
Forgotten crisis index
OCHA FTS
ECHO funding to specific sectors/ crises as a share of total needs
Quantitative A significant share of ECHO funding (ECHO total funding) is allocated to sectors/ crises with largest needs.
OCHA FTS appeals data
ECHO funding patterns take into account the gaps
left by other donors as measured by ECHO funding to specific sectors/
crises as a share
of total funding
Quantitative List of crises / sectors where ECHO is a major donor corresponds to the list of crises
/ sectors with largest gaps
OCHA FTS appeals data
OCHA FTS
NB: for the indicators under this Judgement criterion, the basis for our decision rules
will need to be fine-tuned after follow-up interviews with IAF experts at DG ECHO and
once we confirm with DG ECHO whether OCHA FTS / OCHA FTS appeals data can be
used at all.
Where ECHO funding needs to be put in perspective with what other donors are doing
at the global level, there is no other option but to use OCHA FTS for both ECHO
funding and other funding and stick to their definitions (of sectors for instance).
Despite the limitation related to some (ECHO) funding not being captured as
contributing to an appeal, FTS OCHA appeals data are the only source of information
available to give an overview of needs and how those are filled across sectors with the
same indicator
[Efficiency section]
Judgment criterion: ECHO balanced cost in relation to effectiveness and timeliness
in making strategic choices about its portfolio of assistance (i.e. including about its
partners, sectors, approaches, geographical locations, beneficiaries and transfer
modalities)
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 9
Indicator Indicator
type
Decision Rules Source of
evidence
Extent to which ECHO has a clear view of costs throughout the delivery
chain
Qualitative ECHO has a clear view of costs for each project
Inspire report
% of funding implemented via local implementing partners
Quantitative
% of funding implemented via local implementing partners from ECHO falls between the benchmarks of global average of 2% and target of 25%
Inspire study
% of projects with different transfer
modalities
Quantitative
Cash / in kind / vouchers
a growing share if echo funding is distributed in form of cash.
ECHO C&V database
Inspire
study
Adoption of innovations and/ or best practices driving cost effectiveness e.g. Cash vouchers, multi sectoral approaches, consortiums, multi annual funding
Qualitative ECHO pioneered or adopted quickly new approaches
Meta-evaluation
Interviews
Other evidence of ECHO
weighing up the costs and benefits of choices made
Qualitative Number of assessments
available about the choices made: e.g. funding different types of partners (NGOs versus UN), consortia versus individual
partners, different transfer modalities, and the balance between different approaches
(preparedness versus response) – especially in protracted crises
Depth and breadth of these assessments
Even in the absence of formal
assessments, the general opinion across ECHO is that there is strategic thinking about the choices made: e.g. funding different types of partners (NGOs versus UN), consortia versus individual partners,
different transfer modalities, and the balance between different approaches (preparedness versus response) – especially in protracted crises
Previous evaluation of ECHO
funded actions reflect upon choices made
Interviews
with ECHO staff (HQ and field)
Field missions
Meta-evaluation
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 10
Indicator Indicator
type
Decision Rules Source of
evidence
There are feedback and learning mechanisms (examples of efficient
delivery are identified and disseminated)
Qualitative There is evidence of some mechanisms and tools being in place (e.g. meetings,
workshops, trainings, community of practice, documents) to share results.
There are other qualitative elements confirming dissemination efforts
Literature review (monitoring
reports & evaluation)
Interviews with ECHO HQ and field officers
Online
survey with ECHO field
officers
% of Framework partners who strongly agree/ agree with the following statement: ECHO funding allocations take efficiency
and cost-effectiveness criteria into account in an appropriate way
Quantitative
At least 75% strongly agree/ agree
Q.21 of online survey of framework partners
Evidence of flexibility mechanisms being in place to find extra funding to beef up initial budget when needs emerge
Qualitative Mapping of channels available show various flexibility mechanisms exist to ensure timeliness of ECHO funding actions): funding available
through the channel, speed at
which it can be mobilized, cases where it has been used
Mapping of the decision making processes
Interview with HQ
Evidence of ECHO response being timely
Quantitative and
qualitative
Field interviews suggest ECHO response is timely
At least 75% of local implementing partners agreeing ECHO’s response is comprehensive, appropriate and timely given the identified needs and context in my country (Q24)
At least 75% of FPA partners
think ECHO’s speed of response to emerging crises facilitated effective delivery (Q31)
Interviews
Online survey local implementing partners
Online survey FPA
partners
Evidence of ECHO funding being predictable
Quantitative and qualitative
Whether the duration of ECHO funding cycles have generated any inefficiencies e.g. staff turnover etc
At least 75% of FPA partners
Desk research
Interviews
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 11
Indicator Indicator
type
Decision Rules Source of
evidence
think ECHO’s 12-18 month project funding cycles facilitated effective delivery
(Q31)
Online survey FPA partners
Evidence of funding allocations being revised when needs arise
Quantitative
Comparison of annexes to initial worldwide decisions vs revised worldwide decisions to assess revisions made if any
Desk research
Judgment criterion: ECHO took appropriate actions to ensure cost-effectiveness
throughout the project cycle
Indicator Indicator
type
Decision Rules Source of
evidence
The extent to which cost data and quantitative indicators of efficiency are available and used to drive efficiency
% sampled projects referring to quantitative indicators of efficiency, including:
Admin costs as % of
budget
"cost per beneficiary (value in $,€/beneficiary)”
"cost per output/outcome (value in $,€)"
Quantitative Majority of projects reviewed refer to them
Project mapping
Share of projects where there is any evidence of
ECHO considering cost-effectiveness, efficiency, vfm issues in project design and implementation; and nature of this evidence
Quantitative and
qualitative
Majority of projects reviewed have strong
evidence
Project mapping
Evidence of ECHO considering economy,
efficiency and cost-effectiveness in selecting partner proposals / negotiating contracts and monitoring partners
Qualitative Selection process includes criteria related the
economy, efficiency and cost-effectiveness.
ECHO officers examine economy, efficiency and cost-effectiveness issues during project
Review of the selection
process
Review of monitoring process
Interviews with
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 12
Indicator Indicator
type
Decision Rules Source of
evidence
implementation
There are other qualitative
elements confirming these are duly considered in the selection/ monitoring process
ECHO / field and partners
A1.3 Cost-effectiveness of ECHO-funded actions
Judgement Criterion: Humanitarian actions funded by ECHO were efficient / cost-
effective
Indicator Indic
ator
type
Decision Rules Source of
evidence
Alpha ratio Quantitative
Substantial share of ECHO reached the final beneficiary as per summary of Inspire study report which had the objective to answer the question, ‘how much funding
reaches the beneficiary?’
Inspire study
Previous thematic and geographical evaluations concluded that ECHO-funded actions are efficient / cost-effective
Qualitative
Majority of evaluations made this conclusion.
Meta-evaluation
ECHO staff share the view that projects are generally efficient / cost-effective– or
have an understanding of the factors influencing efficiency and take these into account
Qualitative
Majority of ECHO staff share this view
ECHO Interviews
Field
missions
ECHO sectoral staff have an understanding of main cost drivers for their sectors, cost per beneficiary for
selected KRI
Qualitative
Majority of ECHO staff have this understanding
Rapid evaluations
Judgement Criterion: partners systematically consider cost-effectiveness, efficiency,
vfm issues in project design and implementation
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 13
Indicator Indi
cato
r
type
Decision Rules Source of
evidence
Share of projects where there is any evidence of partners considering cost-effectiveness, efficiency, vfm issues in project design and implementation; and nature of this evidence
Qualitative and quantitative
Majority of projects mapped have strong evidence
Project mapping
Results of the multiple-answer question (Q38. What actions are typically taken by your organisation to ensure value for money in the delivery of ECHO funded actions?) indicate a range of actions are typically taken to ensure vfm
Quantitative
A majority of FPA respondents tick at least three actions
Online survey with FPA and local implementation partners
Judgement Criterion: Obstacles to / success factors for cost-effectiveness are clear
and appropriate adaptation strategies are put in place as a response (e.g. measures
mitigating the obstacles, adapting the intervention)
Indicator Indi
cato
r
type
Decision Rules Source of
evidence
Survey respondents have a shared understanding of main obstacles to / success factors for cost-effectiveness as per Answer to question: Based on your experience of implementing ECHO funded actions, please indicate the extent to which the following factors impede or facilitate their effective delivery.
Quantitative
Key items are selected by majority of respondents
Online survey with FPA and local implementation partners
Qualitative feedback from fieldwork shows the facts impeding / fostering cost-effectiveness are well understood and appropriate adaptation strategies are put in place as a response
Qualitative
Interviewees predominantly have this understanding
Fieldwork interviews with other relevant actors
Survey respondents and interviewees point towards the same factors identified as
Qualitative
Interviewees and survey respondents predominantly have this awareness
Meta-evaluation
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 14
Indicator Indi
cato
r
type
Decision Rules Source of
evidence
influencing cost-effectiveness in previous ECHO evaluations
% projects reviewed incorporating lessons learned
Quantitative
More than 75% do so project mapping
A1.4 Additional descriptive indicators
Indicator Indicator type Source of evidence
Average grant size Quantitative HOPE / EVA
% of projects which get funding several years in a row
% of ECHO funding to UN, NGOs and IOs Quantitative HOPE / EVA
% of funding to each partner Quantitative HOPE / EVA
Number of partners Quantitative HOPE / EVA
Value / Share of assistance reaching partners
Share of 23.02 chapter in total ECHO budget
Share of 23.02 chapter (minus cost field network, ECHO flight/ contribution to UNHAS) in total ECHO budget
Quantitative Analysis of ECHO budget
% of projects using different approaches (preparedness/prevention versus response)
Share of budget line reserved for response (23 02 01) vs preparedness/prevention (23 02 02): out of total intervention related budget (23 02 chapter)
Quantitative Analysis of ECHO budget
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 15
Annex 2 Full list of interviewed stakeholders
Table A2.1 Scoping interviews
Category Organisation
ECHO staff HQ ECHO - A3
ECHO staff HQ ECHO - B1
ECHO staff HQ ECHO - B3
ECHO staff HQ ECHO - B4
ECHO staff HQ ECHO - C1
ECHO staff HQ ECHO - C2
ECHO staff HQ ECHO - C3
ECHO staff HQ ECHO - C4
ECHO staff HQ ECHO - D1
ECHO staff HQ ECHO - D4
Other EU entities DEVCO - C,G,H-4
Other EU entities DEVCO - C1
Other EU entities NEAR - A5
Other EU entities NEAR - A3
Table A2.2 Sectoral and thematic interviews
Category Organisation
ECHO staff HQ ECHO - C1
ECHO staff HQ ECHO - B1
ECHO staff HQ ECHO – C1
ECHO staff field ECHO field
ECHO staff field ECHO field
ECHO staff field ECHO field
ECHO staff field ECHO field
ECHO staff field ECHO field
ECHO staff field ECHO field
ECHO staff field ECHO field
ECHO staff field ECHO field
ECHO framework partners FAO
ECHO framework partners Solidarites International Association
ECHO framework partners UNHCR
ECHO framework partners DRC
ECHO framework partners DRC
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 16
ECHO framework partners IOM
ECHO framework partners IOM
ECHO framework partners UNICEF
ECHO framework partners UNICEF
ECHO framework partners UNICEF
ECHO framework partners CIRC
Global forums and clusters Food Security Cluster
Global forums and clusters Global WASH Cluster
Global forums and clusters Global WASH Cluster
Global forums and clusters Global Shelter Cluster - IFRC
Global forums and clusters Global Shelter Cluster - IFRC
Global forums and clusters Global Shelter Cluster - REACH
Global forums and clusters Global protection cluster
Global forums and clusters Global protection cluster
Table A2.3 Third round of interviews
Category Organisation
ECHO HQ ECHO – D1
ECHO HQ ECHO
ECHO HQ ECHO
ECHO HQ ECHO – D
ECHO HQ ECHO – B1
ECHO HQ ECHO – C2
ECHO HQ ECHO – C3
ECHO HQ ECHO – D4
ECHO HQ ECHO
ECHO HQ ECHO – D1
ECHO HQ ECHO – D3
Other EU DEVCO
Other EU DEVCO
Other EU DEVCO
Framework partners World Vision International
Framework partners CARE
Framework partners CARE
Framework partners CARE
Framework partners WFP
Framework partners ICRC
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 17
Framework partners MSF
Framework partners UN OCHA
Framework partners UN OCHA
Other NGO Voice
EU Member States Austria
EU Member States Estonia
EU Member States Sweden, Ministry for Foreign Affairs
EU Member States Swedish International Development Cooperation
Agency
EU Member States Belgium, FPS Foreign Affairs, International Trade and
Development Cooperation
EU Member States UK, DFID & Foreign & Commonwealth Office
Academics/Think tanks ALNAP
Academics/Think tanks Falk School of Sustainability and Environment,
Chatham University
Table A2.4 Participants to the validation workshop
Category Organisation
DG ECHO ECHO – A3
DG ECHO ECHO – D1
DG ECHO ECHO – C1
Other Voice
Other Voice
Framework partners UN OCHA
Framework partners World Food Programme
Framework partners World Food Programme
Framework partners UNHCR
Framework partners ICRC
Framework partners ICRC
Framework partners Save the Children
Framework partners Action Contre la Faim
Framework partners Norwegian Refugee Council
Framework partners Danish Refugee Council
Academics/Think tanks ALNAP
EU Member States UK Department for International Development
EU Member States Belgian Ministry of Foreign Affairs
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 18
Annex 3 Methodological notes on the sources and use of the
quantitative data
Three main sources of quantitative data (EVA data on ECHO funding, FTS data on all
donors’ funding and appeals data) have been used in this report– essentially in the
relevance section. Important notes on these sources are detailed below. Three main
sources of quantitative data (EVA data on ECHO funding, FTS data on all donors’
funding and appeals data) have been used in this report– essentially in the relevance
section. Important notes on these sources are detailed below.
Before turning to that though, the box below clarifies the different angles from which
one looked at EU Humanitarian aid.
EU Humanitarian aid from different angles
The evaluation team used different sources of data on EU humanitarian aid funding to
serve different purposes and analysis. A clear reference to the exact source is
provided in all cases.
The main source used in the evaluation is the EVA database extracted by DG ECHO
from HOPE. It shows contracted amounts (i.e. amounts of contracts which have been
signed by the framework partners). The main reason to use EVA is that it allows to do
a granular analysis of the data, e.g. by sector.
In some cases, the evaluation presents committed amounts as per DG ECHO
budgetary data (i.e. amounts available to DG ECHO as per Budget Execution
summaries or Excel files directly provided by DG ECHO). The main advantage of this
source of information is that it allows for an analysis of the source of funding and for
making a distinction between different lines of the EU Budget (e.g. initial budget,
reinforcements from the EAR, Heading 4 redeployments, reinforcements from the EDF)
from other sources (e.g. external assigned revenues).
The third source of information used in the evaluation concerns the allocated
amounts as per the HIPs documents (i.e. amounts allocated to specific HIPs). This
has been used to analyse DG ECHO funding by objective (see e.g. Error! Reference
source not found.) or to have a first indication of 2017 trend as 2017 data on
contracted amounts is not yet available.
The evaluation team also used data extracted from the FTS database to put the EU
humanitarian aid funding in perspectives of the global humanitarian aid funding. The
limitation associated with the use of the FTS data are detailed below.
In general, committed amounts (if all sources are taken into account) are equal or
superior to allocated amounts, themselves equal or superior to contracted amounts.
Provided that sufficient time lag has elapsed for all amounts to be contracted and
provided that the budget is implemented in full (all resources are contracted), the
three indicators will have similar values.
A3.1 EVA data
A3.1.1 Source of EVA data
EVA dataset is managed by ECHO and was first extracted on April 19, 2017. A second
extraction was made on December 18, 2017 to make sure 2016 data was as up-to-
date as possible given the timeframe of this study (2016 resources continue to be
contracted in N+1). All graphs presenting EVA data only have been updated with the
most recent extraction. Other more sophisticated analyses which have been made on
EVA data as well as other sources of data (e.g. FTS data or IAF assessments) have not
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 19
been updated due to the time and resources it would have involved. Below each
figure, the data of extraction is clearly indicated.
A3.1.2 Content of EVA data
The EVA dataset contains ECHO’s contracted amounts per action and per result.
Because each action result is allocated to one sector, this breakdown by result allows
to do an analysis by sector without double-counting. The total contracted amount per
action simply is the sum of the contracted amount per action result. The dataset
contains other key variables as presented below:
Year. Under EVA, there are two options to present the year: The contract year refers to the year the project contract was signed and the budget was
transferred to the partner. The financial year refers to the budget year, in other
words, the year of the HIP decisions of DG ECHO. The financial year is the year
which is used as a basis for the analysis presented in this evaluation.
Beneficiary country. The beneficiary country generally refers to the country where the project is executed. In the case of multi-country projects, the single
country which appears in the dataset as the beneficiary country is the first
which appears in HOPE contract line and should correspond to the main
beneficiary country. There is no way from existing ECHO information systems to
breakdown the country allocation of these multi-country projects. This can
slightly bias country allocations (see Table A3.1for some key statistics on
multicountry projects)
Table A3.1 Statistics on multi-country projects
2012 2013 2014 2015 2016 Grand Total
Value share 20% 29% 23% 7% 3% 14%
Number of projects 67 66 84 53 36 306
Source: ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.
Sectors. There are 15 sectors, including since 2016 the multi-purpose cash transfer (for cash transfer designed to cover needs in several sectors)4.
- Child protection - Coordination - Disaster Risk Reduction / Disaster Preparedness - Education in emergencies - Food security and livelihoods - Gender - Health - Mine actions - Multi-purpose cash transfer - Nutrition - Protection - Shelter and settlements - Support to operations - WASH
4 See ECHO, 2017. Single Form Guidelines. Available at: http://dgecho-partners-
helpdesk.eu/_media/single_form_guidelines_until_21_june_2016.pdf
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 20
EVA contains information on subsectors as well. However one result can cover
several sub-sectors, so that analysis of funding per sub-sector is not possible
based on EVA without double counting5.
Number of beneficiaries: EVA contains estimated total number of direct beneficiaries targeted by the action and, where available, estimated total
number of direct beneficiaries targeted by the result, knowing that the sum of
estimated number of individuals of all results can exceed total number of direct
beneficiaries at the action level if the beneficiaries benefit from more than one
result of the project.
Partner and Partner Category. These two indicators specify the short name of ECHO’s partner and its category (International Organisation, NGO, UN agency
or other (=GIZ))
Region groupings. The countries have been grouped into regions based on HIP groupings.
A3.2 UN OCHA FTS funding data
A3.2.1 Source of FTS funding data
The Financial Tracking System is managed by UN OCHA6. The FTS website aims at
tracking humanitarian funding flows, at different levels, from the donors to the
implementing partners. It also tracks bilateral contributions and private funding. The
FTS dataset is fed by the reports on funding flows and pledges (regarding
humanitarian funding flows) and plans (regarding the appeals and humanitarian
response plans), provided voluntarily by donors and recipient organizations. FTS is
constantly updated. The fist dataset was extracted on 17th of October and the latest
dataset on 30th of October 2017.
A3.2.2 Content of FTS data
Thanks to FTS’s flow model, the data can be extracted from different perspective
(from the perspective of the organization, the emergency, the location, the project,
the plan or the sector). For the scope of this evaluation we extracted data from the
perspective of the organization (donor) or the location (recipient country) for the years
2012 to 2016.
Year. Year refers to the usage year, i.e. calendar year the resources were intended to be used, and not when resources were allocated. FTS relies on flow
model and thus usage year can be defined as from the perspective of the
source of the flow or destination of the flow. We display destination usage year
as it is the default option on FTS.
Beneficiary country. The reported “destination location” refers to the execution country of the action. FTS categories include one “Not specified” category and
one “Multiple locations (shared)” category. Taken together these two categories
represent 9% of total funding for the year 2016.
Sectors. FTS categorizes the actions into 20 different sectors.
- Camp Coordination / Management - Child Protection - Coordination and support services
5 For some sectors, e.g. protection, DG ECHO provided the team with its own weighted estimates of funding by sub-
sector, correcting for double-counting. 6 Data available on the FTS website: https://fts.unocha.org
https://fts.unocha.org/
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 21
- Early Recovery - Education - Emergency Shelter and NFI - Emergency Telecommunications - Food Security - Gender Based Violence - Health - Logistics - Mine Action - Protection - Water Sanitation Hygiene - Nutrition - Multi-sector (Multi-sector' refers to projects and activities with no one
dominant sector and often applies to UNHCR assistance for refugees)
- Multiple Sectors (shared) - All other funding - Other - Not specified
Number of beneficiaries: This is not available from FTS.
Donor: FTS contains data on all donors. To calculate ECHO’s funding share, data on all donors combined and ECHO only was extracted.
A3.2.3 Comparison of ECHO’s EVA data and UN OCHA FTS data
The purpose of this section is (i) to illustrate the differences across the two databases
and explain the origin of these and (ii) highlight what the strategy has been as part of
this study to present consistent data.
A3.2.3.1 Main differences
A naïve view would be that, since information on ECHO funding is directly transmitted
from EDRIS to FTS, differences would be minimal. In reality, when OCHA FTS receives
the data, they recode it to make it fit their classification system. The differences in
definitions highlighted in 0 and A3.2.2 result in significant discrepancies across
databases.
Differences across years
Total funding by year according to the two data sources do not fully match. Different
factors can explain this discrepancy.
Table A3.2 Comparison of ECHO’s funding to humanitarian aid from 2012 to 2016
according to FTS and EVA
2012 2013 2014 2015 2016
FTS Total ECHO funding € 1,235,706,840 € 1,314,304,793 € 1,315,362,107 € 1,623,402,332 € 1,850,089,069
EVA Total ECHO funding (financial year)
€ 1.246.934.217 € 1.247.707.070 € 1.157.282.841 € 1.553.777.523 € 1.913.940.114
EVA Total ECHO funding (contract year)
€ 1.216.455.572 € 1.259.848.373 € 1.232.052.341 € 1.327.894.096 € 2.034.823.170
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 22
Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017.
And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.
Table A3.3 FTS data converted to EUR using the following Xrates:
2012 2013 2014 2015 2016
Historical annual Xrate: USD/EUR 0.77833812 0.75294512 0.75272819 0.90129642 0.90342143
Years. Where in FTS the years correspond to the usage year, i.e. calendar year the resources were intended to be used, and not when resources were
allocated. In EVA the years correspond to the financial year (year of the HIP
decision) or the contract year (the year the project contract was signed and the
budget was transferred to the partner).
Scope of funding covered in both datasets.
- Larger scope in EVA: In EVA, the externally assigned revenues7 are included. While in FTS, external assigned revenues are separately counted.
In addition, EVA includes ECHO’s contribution to the Facility for Refugees in
Turkey while the Facility for Refugees in Turkey is excluded in FTS because
ECHO’s contribution can’t be dissociated from that of other donors.
- Larger scope in FTS: The provision of emergency support within the EU has been excluded from the EVA dataset as it was not within the scope of this
evaluation. However, this funding is not similarly subtracted in the FTS
database.
Use of different exchange rates for converting funding data. ECHO funding data is converted to USD in FTS and has been converted back to EUR here using
historical annual exchange rate.
Differences in top 10 countries
Looking at more disaggregated data, e.g. top 10 countries, differences are even larger
(see Table A3.4), especially when looking at single years (Table A3.5Table A3.5). In
addition to the above factors, other factors come into play.
Multicountry projects. In FTS, multicountry projects are shown under a specific “Not specified” category and one “Multiple locations (shared)” category.
Table A3.4 Comparison of the top 10 funded countries from 2012 to 2016
EVA Total 2012-2016 - financial year
EVA Total 2012-2016 - contract year
FTS Total 2012-2016
Syrian Arab Republic € 814.829.914 € 809.329.914 € 747.891.790
Turkey € 606.300.000 € 540.650.000
South Sudan Republic € 533.260.000 € 533.260.000 € 683.029.622
Iraq € 333.055.000 € 327.305.000 € 371.360.178
Niger € 277.744.981 € 273.544.981
Somalia € 255.450.000 € 256.150.000 € 328.937.932
Democratic Republic Of Congo
€ 241.403.672 € 241.403.672 € 393.033.887
Lebanon € 240.420.000 € 240.420.000 € 399.948.666
Jordan € 236.206.758 € 236.206.758 € 340.699.953
7 external assigned revenues are collected from and assigned by various donors to a specific programme or
action
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 23
Yemen € 227.300.000 € 230.005.202
Not Specified € 475.202.281
Ethiopia € 402.709.535
Chad € 310.918.985
Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017. And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.
Table A3.5 Comparison of the top 10 funded countries for 2016
EVA 2016 - financial year
EVA 2016 - contract year FTS 2016
Turkey € 543,650,000 € 479,500,000 € 57,070,027
Syrian Arab Republic € 162,949,965 € 160,549,965 € 203,410,073
Iraq € 157,850,000 € 153,100,000 € 141,066,051
South Sudan Republic € 122,700,000 € 142,700,000 € 142,928,477
Lebanon € 84,550,000 € 85,800,000 € 67,233,573
Jordan € 69,400,000 € 69,400,000 € 60,711,300
Yemen € 67,300,000 € 60,800,000 € 163,498,846
Nigeria € 52,800,000 € 55,779,000 € 56,644,466
Niger € 47,869,103
Democratic Republic Of Congo € 38,850,000
Yemen
€ 53,100,000
Somalia
€ 48,550,000
Greece
€ 198,882,216
Not Specified
€ 75,123,734
Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017.
And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.
Sectors. Analysis by sector is the least comparable type of analysis, as shown in Table A3.6 below.
Table A3.6 Differences in top 10 sectors
EVA 2012- 2016 - contract year FTS 2012-2016
Sector Total Amount (EUR)
Sector Total Amount (EUR)
Food security and livelihoods € 2,160,815,944
(blank) € 2,169,165,271
Health € 888,202,139 Health € 1,155,165,775
Shelter and settlements € 732,928,291
Multi-sector € 1,116,268,072
Nutrition € 728,417,699 Food Security € 1,109,137,283
WASH € 708,317,188 Coordination and
support services € 634,720,263
Protection € 649,154,471 Water Sanitation
Hygiene € 388,113,978
Multi-purpose cash transfer € 339,296,832
Emergency Shelter and NFI € 348,879,717
Disaster Risk Reduction / Disaster € 289,523,594
Protection € 210,024,099
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 24
Preparedness
Coordination € 224,617,577 Early Recovery € 155,506,179
Support to operations € 209,028,081 Education € 39,446,124
Education in emergencies € 76,101,991
Mine Action € 4,349,642
Child protection € 52,645,834 Nutrition € 2,967,927
Mine actions € 10,163,960 Total € 7,333,744,330
UNKNOWN € 1,501,304
Gender € 358,646
Total € 7,071,073,552
Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 17 October 2017. And, ICF, 2017. Analysis of ECHO EVA database, data extracted on 19 April 2017.
Additional factors explaining the differences for sectors are that (I) EVA allows to
breakdown the funding of a single project under different sectors, which is not the
case of FTS (all funding allocated to a single, main sector). And, (II) unlike EVA, FTS
data includes a large multi-sector and not specified category – to which some
protection funding certainly goes to - but this will not be included into FTS protection
data.
Strategy used
EVA contains more granular information to analyse ECHO funding. Wherever ECHO
funding only was necessary for the analysis, the preferred source of information was
EVA.
However, to put ECHO data in perspective with what other donors are doing, there is
no other option but to use FTS data. Since definitions (e.g. of years and sectors) are
different across data sources, where ECHO funding needs to be put in perspective with
what other donors are doing at the global level, there is no other option but to use
OCHA FTS for both ECHO funding and other donors’ funding and stick to FTS
definitions (e.g. of years and sectors) – in order to compare like with like.
A3.3 Appeals data
Appeals data from FTS was used to put ECHO funding into perspective with global
needs in monetary terms.
A3.3.1 Introduction to the appeals process8
Humanitarian Response Plans (HRPs), Flash Appeals and Regional Response Plans
(RRPs) reflects the needs of affected population during humanitarian crises or
emergencies. The HRPs and Flash Appeals are elaborated by the Humanitarian
Country Teams (HCT) which brings together UN agencies, NGO’s and other actors. The
Regional Response Plan (RRPs) are regional refugee response plans, coordinated by
UNHCR and its partners. For all types of appeals, the teams elaborate a plan that
includes the country or context strategy and the objectives; it details how the strategy
will be implemented and how much funding is required. The planning process usually
takes place once a year, between September and December and cover a one-year
horizon. However, plans can be issued at any time in the year. The online
planning/project system allows for the participating agencies to upload their projects
and funding requests and to update them during the course of the plan. By the end of
each calendar year, the Global Humanitarian Overview publishes known and
anticipated requirements for the year to come.
8 See also: https://fts.unocha.org/content/guide-funding-response-plans-and-appeals
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 25
A3.3.2 Data on the Appeals and Response Plans
FTS releases the data on overall finalised requirements for the year in late January.
Data based on the list of appeals made between 2012 and 2016, was retrieved from
the FTS archived website9. This list includes HRPs, Flash Appeals and RRPs. The final
list is slightly modified compared to the initially extracted list. Additional data needed
to be retrieved while overlooking some appeals as detailed below.
Additions compared to FTS list
Country breakdowns for RRPs. FTS does not break down the funding needs of the Regional Response Plans per country, but provides only the total for the
region. In order to study the funding needs per country within the region,
country breakdowns were retrieved based on own desk research (breakdowns
are available online in RRP documentation). Please note the sum of the country
breakdowns as per Regional Response Plans available online did not always
match total regional funding requirements provided by FTS. In these cases,
information from RRP prevailed since the detailed country breakdowns were
needed for the analysis.
List of the RRPs for which country breakdowns have been retrieved through
desk research:
- Burundi Regional Refugee Response Plan (DR Congo, Rwanda, Tanzania and Uganda), 2015 and 2016
- Central African Republic Regional Refugee Response Plan (Cameroon, Chad, Republic of Congo, Democratic Republic of the Congo), 2015 and 2016
- Ebola Virus Outbreak - Overview of Needs and Requirements (inter-agency plan for Guinea, Liberia, Sierra Leone, Region) - October 2014 - June 2015
- Nigeria regional refugee response plan, 2015 and 2016 - South Sudan regional refugee response plan 2014 to 2016 - Revised Syria Regional Response Plan 2012 - Syria regional refugee and resilience plan (3RP), 2013 to 2016 - Yemen Regional Refugee and Migrant Response Plan, 2015 and 2016
Addition of missing RRPs. Some HRP data were absent from the FTS dataset exports and were also added manually through desk research:
- Korea DPR Overview of Needs and Assistance 2012 - Myanmar - Kachin Response Plan, 2012 and 2013 - DPR Korea Needs and Priorities 2015 and 2016 - Zimbabwe 2013
Omissions compared to FTS list if no country breakdown available.
The Central Emergency Response Fund (CERF), is a central fund managed by the UN
for global emergency response to sudden crisis10. Although this fund processes its
appeals via the UN system, and is referred in FTS data extractions, we did not take
these into account because they have no ex-ante breakdown of needs per country.
The same goes for appeals labelled as "West Africa Regional Contribution 2015" and
"Miscellaneous 2015 (GMT and HND)" for which no country breakdown could be found
and which we are not taken into account in the study.
Figure A3.1 below gives an overview of the types of appeals that were made for each
country, per year.
9 https://ftsarchive.unocha.org/pageloader.aspx?page=searchProject-
ListSelectAppeals&CQ=cq021117194517RM9XLRdII 10
http://www.unocha.org/cerf/about-us/cerf-at-a-glance
https://ftsarchive.unocha.org/pageloader.aspx?page=searchProject-ListSelectAppeals&CQ=cq021117194517RM9XLRdIIhttps://ftsarchive.unocha.org/pageloader.aspx?page=searchProject-ListSelectAppeals&CQ=cq021117194517RM9XLRdII
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 26
Figure A3.1 List of countries and respective appeals from 2012 to 2016
Source: ICF, 2017. Analysis of UN OCHA FTS database, data extracted on 29 September
2017.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 27
A3.3.3 Limitation to the FTS response plan data extracted
Incomplete picture of needs.
Not all agencies and humanitarian actors participate in UN coordinated appeals
process. It is the case for ICRC and MSF who work independently. Hence, their funding
requirements are not included in the HRPs, Flash Appeals or RRPs and are not counted
as needs.
Lack of comparability of appeals data.
Scale of appeals requirements vary for various reasons (“valid” and “non valid ones”):
Valid reasons: different scales of need, different cost of delivering assistance per se.
Non-valid reasons (undermining comparability): data being inflated voluntarily (appeals data stemming out of politicized processes)11, differences in scope
with some appeals covering development components and others excluding it.
Scope sometimes includes development component. For example, the Syria 3RP
includes a “‘refugee component’ and a ‘resilience component’, with the latter
accounting for between a quarter and a third of the total amount requested.”12
The fact that differences in appeals requirements is not fully driven by differences in needs is well illustrated in the average cost of response per
person which varies widely based on data from UN OCHA FTS and UN-
coordinated appeals: US$801 per person for South Sudan, US$36 per person
for Nigeria13.
Possibility of double counting between RRPs and HRPs for countries covered by both types of appeals. For countries covered by both RRPs and HRPs during a
same year, it cannot be excluded that the funding requirements of the RRP are
not overlapping with the Humanitarian Response Plans: the same requirements
might be counted twice. For instance, Central African Republic was covered in
2014 by both an HRP and a RRP (Ebola Virus Outbreak, regional response
plan), however it cannot be excluded that the HRP did not integrate any funding
needs for projects related to the Ebola Virus Outbreak. The countries potentially
subject to this double counting are listed below:
- Burkina Faso, 2014 - Cameroon, 2014, 2015 and 2016 - Central African Republic, 2014 and 2015 - Democratic Republic of the Congo, 2014, 2015 and 2016 - Gambia, 2014 - Mali, 2014 - Mauritania, 2014 - Senegal, 2014 - Yemen, 2015 and 2015 - Chad, 2015 - Djibouti, 2015 and 2016 - Niger, 2015 and 2016
11
Olin, E., & von Schreeb, J. , 2014. Funding Based on Needs? A Study on the Use of Needs Assessment Data by a Major Humanitarian Health Assistance Donor in its Decisions to Allocate Funds. PLoS Currents, 6, ecurrents.dis.d05f908b179343c8b4705cf44c15dbe9. http://doi.org/10.1371/currents.dis.d05f908b179343c8b4705cf44c15dbe9 12
Development International, 2015. The 2015 UN-coordinated appeals: An ambitious plan to meet growing humanitarian needs. Available at: http://devinit.org/wp-content/uploads/2015/04/The-2015-UN-coordinated-appeals-F.pdf [Consulted on 9 November 2017] 13
Development International, 2015. Op. Cit.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 28
- Sudan, 2015 and 2016 - Somalia, 2016
A3.3.4 Appeals coverage and funding gap
FTS tracks funding flows in relation to the projects listed in the HRPs, Flash appeals
and RRPs. Each funding flow is thus labelled as either “Inside response plans/appeals”
or “Outside response plans/appeals”. Only funding flows labelled as “Inside response
plans/appeals” do count when FTS present data on needs coverage to date and
funding gap still needed for each appeal.
In contrary to this approach, we calculated, under the relevance section, funding gap
as the difference between appeals requirements and any type of funding flow
received. One reason for this is that even if recorded outside the appeals, the funding
flows targeting the same crises still serve to alleviate the same needs. In addition, DG
ECHO’s position as a contributor to appeals is generally low (funding not recorded
against the appeal or ECHO partners not participating in the appeal).
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 29
Annex 4 Profile of respondents to the three surveys
A4.1 Respondents to the ECHO field survey
Geographical distribution: responses came from 129 members based in 34
different DG ECHO field offices. Significant share of respondents were based in Kenya
(15 respondents representing 12%), Jordan (11 or 9%), Turkey (9 or 7%) and
Senegal (8 or 6%), reflecting the presence of regional office in these countries.
Table A4.1 Number of respondents per country (n=129)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 12.06.2017 to
14.07.2017
Level of experience: The respondents to the survey have different level of
experience with ECHO field network. The largest share of respondents (43%) have
been with the network for more than five years. Only 12% or 15 respondents have
been with the field network for a limited time of period (less than one year).
Table A4.2 Number of years spent as part of ECHO field network (n=129)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 12.06.2017 to
14.07.2017
Role in the field: The respondents represent a wide range of professional roles within
the network. The majority of respondents are Technical Assistants (24%), Programme
Assistants (17%), Heads of Country Officers (15%), Programme Officers (13%) or
Sectorial Experts (10%).
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 30
Table A4.3 Specific role within ECHO field network (n=127)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 12.06.2017 to
14.07.2017
Type of activities undertaken: The respondents were involved in a variety of
functions in their role, demonstrating a confidence in responding to the number of
questions in the survey. The three most visible functions of respondents were:
monitoring of projects (77% of respondents were to a very large or large extent
involved in this function), monitoring of humanitarian situation on the ground (77%)
and providing technical advice to HQ on monitoring projects (69%).
Table A4.4 Functions where respondent is most often involved (n=varies from 126
to 129)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 12.06.2017 to
14.07.2017
A4.2 Respondents to the framework partner survey
Among the 361 respondents to the survey, 89% are from framework partners and
11% are from UN agencies.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 31
Table A4.5 Share of framework partners (n=361)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 07.06.2017 to 04.08.2017
The most represented framework partners in the survey are Oxfam (6% of
respondents), CARE (6%), FAO (4%) and Save the Children (4%). In total, 88
different framework partners organisation are represented in the survey.
The 88 surveyed framework partners are based in 80 different countries. A third
(36%) of the respondents are based in Europe, mainly in Germany, France, Spain, UK,
Belgium, Netherlands, and Switzerland. The other 64% of the respondents'
organisations are based in the field; Pakistan (5% of respondents), Jordan (3%),
Turkey (3%) Nepal (3%).
The majority of the framework partners were involved in 2 to 5 ECHO funded projects
between 2012 and 2016. Roughly half of the partners had closed all their actions by
the time of the survey.
Table A4.6 In how many ECHO funded humanitarian aid actions were you involved
between 2012 and 2016? (n=361)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 07.06.2017 to
04.08.2017
Table A4.7 Have the actions you have been involved in been finalised? (n=361)
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 32
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 07.06.2017 to
04.08.2017
On average, the partners work in 3 different sectors. Food Security and Livelihoods
(51%) and WASH (47%) are the main sectors in which framework partners are active.
Disaster Risk Reduction and Preparedness is supported by 43% of the respondents'
organisations. And the Health sector is supported by 37% of the respondents'
organisations.
Table A4.8 Sector of activity of the framework partners (n=361)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 07.06.2017 to
04.08.2017. Note: This is a multiple answer question, hence the total does not equal
100%.
ECHO framework partners are active in all regions worldwide, however they are not
spread evenly. Asia and Oceania is the most covered region where almost a third of
the surveyed ECHO framework partners are active in various sectors. The Middle East
is covered by 28% of the surveyed ECHO framework partners. The Great Lakes region
and Central Africa count only 11% of framework partners each.
Table A4.9 Region of activity of framework partners (n=361)
Source: ICF, 2017. Survey: ECHO framework partners. Completed from 07.06.2017 to
04.08.2017. Note: This is a multiple answer question, hence the total does not equal 100%.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 33
A4.3 Respondents to the local partner survey
103 respondents took part to the local partner's survey, representing 74 different local
partners. CARE (8% of the respondents), Islamic Relief (8%) and the Red Cross (8%)
are the most represented local partners in this survey.
The partners are based in 49 different countries, Pakistan (12%), Ethiopia (8%) Nepal
(7%) and Sudan (7%) were the countries were most respondents were located.
Table A4.10 Share of respondents per country (n=103)
Source: ICF, 2017. Survey: ECHO local implementing partners. Completed from 07.07.2017 to 11.08.2017.
The majority of the local partners were involved in 2 to 5 ECHO funded actions
between 2012 and 2016. And for most of the partners all the actions were closed by
the time of the survey.
Table A4.11 In how many ECHO funded humanitarian aid actions were you involved
between 2012 and 2016? (n=103)
Source: ICF, 2017. Survey: ECHO local implementing partners. Completed from 07.07.2017 to 11.08.2017.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 34
Table A4.12 Have the actions you have been involved in been finalised? (n=103)
Source: ICF, 2017. Survey: ECHO local implementing partners. Completed from 07.07.2017 to 11.08.2017.
WASH (40% of active local partners) and Food security and Livelihoods (40% of active
local partners) are the main sectors addressed by local partners. 35% of the local
partners are active in the Disaster Risk Reduction and Preparedness sector.
Table A4.13 Sector of activity of the local partners (n=103)
Source: ICF, 2017. Survey: ECHO local implementing partners. Completed from 07.07.2017 to
11.08.2017.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 35
Annex 5 Meta-evaluation
A5.1 Introduction
Table A5.1 Evaluations reviewed as part of the meta-evaluation
Type of
evaluation
Publication
month-year
Name of the evaluation
Thematic Sept -17 Evaluation of the European Commission's interventions
in the Humanitarian Health sector, 2014-2016
Thematic forthcoming EU Approach to building resilience to withstand food
crises in African Drylands (Horn and Sahel), 2007-2015
– joint DEVCO and ECHO evaluation
Thematic Oct-16 Evaluation of ECHO’s actions in the Field of Protection
and Education of Children in Emergency and Crisis
Situations, 2008-2015
Thematic Jan-16 Evaluation of the use of different Transfer Modalities in
ECHO Humanitarian Aid actions (2011 – 2014)
Thematic Mar-15 Evaluation of DG ECHO’s Approach to Communication
under the Humanitarian Implementation Plans (HIP
between 2010 and 2013)
Thematic Oct-14 Mid-term evaluation of Enhanced Response Capacity
Funding
Thematic Sep-14 Joint Evaluation of Drought Risk Reduction in the Horn
of Africa and DIPECHO Central Asia and South Caucasus
(2009 ‐ 2013)
Thematic Jun-14 Evaluation of the implementation of the European
Consensus on Humanitarian Aid, 2008-2012
Thematic Aug-13 Evaluation of the European Commission’s Humanitarian
Action in the Shelter Sector
Thematic Jul-13 Food security and nutrition: Evaluation of European
Commission integrated approach of food security and
nutrition in humanitarian context (2009-2012)
Thematic Mar-13 Evaluation of DG ECHO's Fleet Management
Thematic Jan-13 Working directly with local NGOs: Evaluation of the
potential effectiveness and efficiency gains of working
directly with local NGOs in the humanitarian
interventions of the Commission
Thematic Jul-12 Evaluation and review of humanitarian access strategies
in DG ECHO funded interventions
Thematic Jun-12 Need Analysis, Review and Design of DG ECHO’s
Training in Disaster Risk Reduction and Climate Change
Adaptation
Thematic Jun-12 Evaluation of DG ECHO’s Legal Framework for Funding
of Humanitarian Actions (FPA 2008)
Thematic May-12 Review of existing practices to ensure participation of
Disaster-affected communities in Humanitarian aid
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 36
Type of
evaluation
Publication
month-year
Name of the evaluation
operations
Thematic Mar-12 Evaluation and review of DG ECHO financed
livelihood interventions in humanitarian crises
Geographic Oct-17 Evaluation of the ECHO interventions in India and
Nepal, 2013-2017
Geographic Dec-16 Evaluation of DG ECHO’s Response to the Humanitarian
Crises in Sudan and South Sudan (2011-2015)
Geographic Oct-16 Evaluation of DG ECHO's Actions on Building Resilience
in the LAC Region
Geographic Jun-16 Evaluation of the ECHO intervention in Pakistan
Geographic Jun-16 Evaluation of the ECHO response to the Syrian Crisis
(2012 – 2014)
Geographic Feb-16 Evaluation of ECHO’s interventions in the Sahel (2010-
2014)
Geographic Sep-15 Evaluation of the DG ECHO Actions in Coastal West
Africa 2008 – 2014
Geographic Dec-13 Evaluation of the European Commission's Humanitarian
and Disaster Risk Reduction Activities (DIPECHO) in
Indonesia
Geographic Nov-12 Evaluation of European Commission’s Humanitarian
Activities in Colombia
Geographic Mar-12 Evaluation of DG ECHO's interventions in the occupied
Palestinian territory and Lebanon
Thematic Jun-06 Evaluation of the European Commission’s Directorate
General for Humanitarian Aid (DG ECHO), 2000-2005
A5.2 Findings of the meta-analysis by evaluation questions
A5.2.1 Relevance
EQ1: To what extent do ECHO budget allocations consider the needs, actions
of other donors and the EU objectives on humanitarian aid?
JC1.1 ECHO’s budget allocations are based on needs, actions of other donors and ECHO objectives
I1.1.1 Evidence that ECHO’s budget allocations are based on needs
Literature and document review e.g. outputs of INFORM. FCA, IAF
Review of ECHO’s approach to needs assessment
Interviews: ECHO HQ, ECHO field, other key donors to understand their approaches to needs assessment
Surveys: ECHO field, FPA partners
Expert opinion
OPC
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 37
I1.1.2 Evidence that ECHO’s budget allocations
considered the actions of other donors
Document review
Interviews: ECHO HQ, ECHO field
Surveys: ECHO field, FPA partners
I1.1.3 Proportion of ECHO humanitarian aid funding allocated to each sector and
compared to needs
Literature review
I1.1.4 Evidence that ECHO’s budget size was commensurate to ECHO objectives and expected results
Literature review
Interviews: ECHO HQ, ECHO field
Surveys: ECHO field, FPA partners
I1.1.5 Evidence of changes in budget allocation over the evaluation period (across sectors / geographies)
Document review (e.g. review of HIPs) and HOPE data analysis
Extent to which the evaluation question has been addressed in the evaluations
The 27 evaluations reviewed were somewhat inconsistent in presenting information on the extent to which ECHO’s allocations considered the needs, actions of other donors and EU objectives on humanitarian aid. While comprehensive information was provided by some reports, others were less informative in this regard. This made comparison across sectors and geographies challenging. The review showed that not all the indicators identified in ICF’s
analytical framework were fully addressed by the 27 evaluations under observation.
Evidence that ECHO’s budget allocations are based on needs: The inclusion of information on the use of needs assessment (both at ECHO and project level) in the
evaluation reviewed was very inconsistent (i.e. not all the evaluations reviewed referred to the needs assessment aspect). Moreover, none of the evaluations looked at the DG ECHO needs assessment process for the programming of HIPs;
Evidence that ECHO’s budget allocations considered the actions of other donors – very limited information identified in the evaluations reviewed;
Proportion of ECHO humanitarian aid funding allocated to each sector and compared to needs – very limited information identified in the evaluations reviewed;
Evidence that ECHO’s budget size was commensurate to ECHO objectives and expected results: insufficient information identified to be able to provide a sound analysis;
Evidence of changes in budget allocation over the evaluation period (across sectors / geographies) – addressed by the evaluations reviewed to a moderate extent, analysis provided in section 0 below.
Extent to which ECHO’s budget allocations were based on needs
The evaluations reviewed suggested that, overall, ECHO’s budget allocations were
based on needs at the level of the actions funded by ECHO, i.e. the needs as identified
through assessments carried out at field/project level. DG ECHO was highly supportive
of assessments and consistently encouraged partners and other local actors to conduct
their own analyses. However, as further elaborated below, at the central programming
level, as part of the HIPs and other strategic decision-making by ECHO, it was found
that the budget allocations did not always fully address the needs, showing insufficient
coverage of themes and, in some cases, a disconnect with specific local needs. Not
enough information was however available to be able to draw firm conclusions on the
extent to which this “gap” might have contributed to reducing the overall relevance of
ECHO’s budget allocations to the needs of stakeholders on the ground.
-
Comprehensive evaluation of the European Union humanitarian aid, 2012-2016
October, 2017 38
At the level of ECHO funded actions, needs assessments were regularly carried out by
implementing partners in the context of the interventions developed on the ground
(field/project level). The review of evaluations, however, showed that the quality of
such assessments varied substantially, even within the same country. In Indonesia,
for example, “all partners conducted area-specific risk assessments as a basis for
designing their projects, some more comprehensively than others”.
Differences in quality were found also across themes/sectors. For example, within the
food security and nutrition sectors, some partners have developed sophisticated ways
of analysing causes and assessing nutrition problems. “Overall, partners provide good
information on food availability and food access but insufficient information on food
intake and food utilisation, despite the relevance of these elements. Partners’ analyses
of causes of undernutrition were sometimes cursory, with the implicit assumption that
food access in itself will ensure adequate nutrition”. The evaluation indicated that, in
particular in contexts where nutrition is clearly in focus, there is a need to look more
deeply into the causes of malnutrition, to shed light on types of interventions that
might be most effective.
Similarly, within the education in emergencies sector, needs assessments varied in the
level of detail and methods used (for instance, in use of participatory approaches, the
involvement of children, etc.). The specific consideration of age and gender-based
needs (and adapting activities accordingly) also varied according to partners.
In the health sector, differences in the quality of assessments conducted at
field/project level were mainly linked to the partners’ capacity to conduct proper needs
analysis. The thematic evaluation noted that while some of ECHO’s partners had
strong standards and tools to conduct needs assessments and invested in abiding to
quality standards others were altogether absent from the field and conducted such
assessments remotely or purely based on secondary data. The quality of needs
assessment conducted by partners was also influenced by factors inherent to
humanitarian contexts (shortcomings in existing data, security and access issues,
need to react fast), which made establishing proper causality and designing
interventions to address the correct factors of influence, complex.
In this context, the evaluation of the HA consensus also indicated a need for a more
consis