summative phase report - harvard university
TRANSCRIPT
i SUMMATIVEPHASEREPORT- THE DISASTERS & EMERGENCIES PREPAREDNESS PROGRAMME
EVALUATION
3 MAY 2018
SUMMATIVE PHASE REPORT
ii EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Phuong Pham, PhD, MPH*Vandana Sharma, MD, MPH*Rebecca Hémono, MPHJessica Jean-Francois, MUP, M.S. EdJennifer Scott, MD, MPH
With contributions from:Root Change, Valsa Shah, MS, and Piotr Bialowolski, PhD
*Co-leads for the Report
Suggested Citation:
Pham P, Sharma V, Hémono R, Jean-Francois J, Scott J. (2018) The Disasters and Emergencies Preparedness Programme Evaluation: Summative Phase Report, May 3 2018. Harvard Humanitarian Initiative.
© Harvard Humanitarian Initiative, 2018The views presented in this paper are those of the authors and do not necessarily represent the views of the DEPP programme, its partners or donor.
All rights, ownership and copyright of data and materials are retained by the Harvard Humanitarian Initiative.
Report design © Alice Hale / Action Against Hunger
1 SUMMATIVEPHASEREPORT-TAblEOfcOnTEnTS
ACRONYMS 2
ACKNOWLEDGEMENTS 4
EXECUTIVE SUMMARY 5 InTRODUcTIOn 6 MAiN FiNDiNGS 12 cOnclUSIOnSAnDREcOMMEnDATIOnS 18
cHAPTER1INTRODUCTION AND CONTEXT 32
cHAPTER2 EVALUATION METHODOLOGY 49
cHAPTER3RELEVANCE AND VALIDITY OF DESIGN 79
cHAPTER4RELEVANCE AND EFFECTIVENESS OF INTERVENTIONS 93
cHAPTER5EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS 128
cHAPTER6EFFICIENCY AND VALUE FOR MONEY 147
cHAPTER7SUSTAINABILITY AND LIKELIHOOD OF IMPACT 164
cHAPTER8HUMANITARIAN PREPAREDNESS AND RESPONSE CASE STUDIES 183
cHAPTER9CONCLUSIONS AND RECOMMENDATIONS 208
TABLE OF CONTENTS
2 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ACRONYMS
ADCAP AgeandDisabilitycapacityProgramme
CDAC-N communicatingwithDisasterAffectedcommunitiesnetwork
CLA countrylearningAdvisor
CSO civilSocietyOrganisation
CWC communicatingwithcommunities
DAC DevelopmentAssistancecommittee
DEPP DisastersandEmergenciesPreparednessProgramme
DFID DepartmentforInternationalDevelopment
DRC DemocraticRepublicofthecongo
DRR DisasterRiskReduction
EWEA ImprovedEarlyWarning-EarlyActions toStrengthenDisasterPreparednessEthiopia
FE financialEnablersProject
GBV Gender-basedViolence
HERR HumanitarianEmergencyResponseReview
HHI HarvardHumanitarianInitiative
IDI In-depthInterview
IDP InternallyDisplacedPerson
INGO Internationalnon-governmentalorganisation
IRB InstitutionalReviewboard
KAP Knowledge,AttitudesandPractices
L/NNGO local/nationalnon-governmentalorganisation
LPRR linkingPreparednessResilienceandResponse
M&E MonitoringandEvaluation
MEL Monitoring,Evaluationandlearning
NAHAB nationalAllianceofHumanitarianActorsbangladesh
NGO non-governmentalorganisation
NHN nationalHumanitariannetwork
NPAC nonProjectAttributablecosts
OCHA UnitednationsOfficeforthecoordinationofHumanitarianAffairs
PHEP Gambella PublicHealthEmergencyPreparednessinGambella
3 SUMMATIVEPHASEREPORT-AcROnyMS
PIP ProtectioninPractice
ROI ReturnonInvestment
SEPS StrengtheningEmergencyPreparednessSystemsinMyanmar
START STARTnetwork
STP ShiftingthePower
T1 TimePoint1(formativephase)
T2 Timepoint2(interimphase)
ToR TermsofReference
TD TalentDevelopment
TSC TransformingSurgecapacity
UEWEA UrbanEarlyWarningEarlyAction
UK UnitedKingdom
UN Unitednations
VAWG ViolenceAgainstWomenandGirls
VFM ValueforMoney
WASH Water,SanitationandHygiene
4 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ACKNOWLEDGEMENTS TheHarvardHumanitarianInitiative(HHI)evaluationteamextendsthankstoActionAgainstHunger, who commissioned this report, and Dr. Robina Shaheen and Hannah Wichterichwhomanaged theexternalevaluation.Wealso sendourdeepgratitude toniamhGibbons(managementsupport),Rootchange(networkAnalysis),ValsaShah(ValueforMoney),andPiotrbialowolski,Ph.D.(Statisticalanalyses)fortheircontributions.Wethankthemanyparticipantsinthisevaluation,includingcountry-basedDisastersandEmergenciesPreparednessProgramme(DEPP)projectteams,headquarters-basedDEPPstakeholdersandthose involved inDEPP’sleadership and governance.We are also grateful to communitymembers inMyanmar andEthiopiawhotookthetimetorespondtohouseholdandcommunitysurveys.
Particular thanks are due to our research partners who implemented data collection andcontributedtheirexpertiseandknowledgeinspecificcountriesandelementsoftheevaluation:thebusaracenterforbehavioralEconomics(Kenya),TheEthiopianPublicHealthAssociation(Ethiopia),InnovationsforPovertyAction(Myanmar,thePhilippines).WethankT-consinSouthSudanfortheirwillingnesstopartnerwithus.
WeareappreciativeofthetimeandeffortputinbytheleadershipandresearchadministrationteamsattheHarvardHumanitarianInitiative,bethIsraelDeaconessMedicalcenterandtheHarvardT.H.chanSchoolofPublicHealth,whoseon-goingsupporthasbeencrucialtothisevaluation.lastly,wewouldliketoacknowledgeourinterns,SarahThang,Julianneylan,AminaGoheer,noorZanial,MichaelDeStAubinandGinacianconefortheirvaluablesupportonthisreport.
5 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
EXECUTIVESUMMARY
6 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
INTRODUCTION Thisreportprovidesthesummativeresultsfromthethree-yearexternalimpactevaluationoftheDisastersandEmergenciesPreparednessProgramme(DEPP)conductedbyateamattheHarvardHumanitarianInitiative(HHI).TheDEPPwasa£40millionprogrammefundedbytheDepartmentforInternationalDevelopment(DfID)thataimedtostrengthenskillsandcapacityand improvethequalityandspeedofhumanitarianresponse incountriesthatareatriskofnaturaldisastersoremergencies.
The DEPP was delivered by two non-governmental Organisation (nGO) consortia, theSTARTnetwork,whichreceived£27million,andthecommunicatingwithDisasterAffectedcommunities (cDAc-n) network from 2015-2018, which received £3 million. In addition,£10millionwasreservedforan innovationwindowwhichwasseparately implementedandevaluated. TheDEPP comprised14 individual projects implemented in one ormore of 101 priority countries: South Sudan, Kenya, Ethiopia,Mozambique,Democratic Republic of thecongo(DRc),Jordan,Pakistan,bangladesh,MyanmarandthePhilippines,witheachprojectoperating in somebutnotallof thecountries.Onlyoneproject (thelearningProject)wasimplementedinall10countrieswhilefiveprojects(PublicHealthEmergenciesPreparedness(PHEP)inGambella,ImprovedEarlyWarningEarlyActions(EWEA)inEthiopia,StrengtheningEmergencyPreparednessSystemsinMyanmar,UrbanEarlyWarningEarlyAction(UEWEAinKenya),financialEnablersinthePhilippines)wereimplementedinonlyonecountryeach.The14DEPPprojects,theirconsortiamembers,locationsandbudgetsarepresentedinTable1.
The Disasters and Emergencies Preparedness Programme (DEPP)
• Investment:£40million
• Location: 10countries
• Duration: 3years
• Number of preparedness projects:14humanitariancapacitybuildingprojects
• Overall objective:Toimprovequalityandspeedofhumanitarianresponseincountries thatareatriskofnaturaldisastersorhumanitarianemergencies
Key approaches: capacitybuildingoflocalandnationalhumanitarianstaffandcommunities,earlywarning systemdevelopment, supportingcollaborativeactionand strengtheningnet-works
DEPP PROJECTPROJECT
CONSORTIA MEMBERS
COUNTRIES** DESCRIPTION BUDGET (£)
TALE
NT
DEV
ELO
PMEN
T (T
D)
SavethechildrenUK(lead),OxfamGb,ReliefInternational,cHSAlliance
bangladesh,DRc,Ethiopia,Jordanand
Kenya,lebanon
Toproducehigh-qualityprofessionalsatalllevelswhoarebetterequippedtotackletheissuessurroundingcomplexemergencies,helpingtoensurethattherightpeopleareintherightplacedoingtherightthingstoassistdisaster-affected
communities.
5,985,087
1 Indonesiawasinitiallyselectedasan11thDEPPfocalcountry,butnoselectedprojectsplannedtoimple-mentactivitiesthere.
7 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
SHIF
TIN
G TH
E PO
WER
(STP
) ActionAidInternational(lead),cAfOD(lead),christianAid,Tearfund,concernWorldwide,
Oxfam
bangladesh,Pakistan,
Ethiopia,KenyaandDemocraticRepublicofcongo(DRc)
Tosupportlocalactorstotaketheirplacealongsideinternationalactorsinordertocreateabalancedhumanitariansystemthatismoreresponsiveandaccountabletodisaster-affected
communities.
4,876,637
THE
LEAR
NIN
G
PRO
JECT
*
ActionAgainstHungerAll DEPP countries
Toevidencetheextenttowhichpreparingpeopleisaneffectiveandefficientapproachtodisastermanagementandbrokerinternalandexternalrelationshipsforlearningaboutwhatisandisnot
workingincapacityexchange.
3,343,375
BETT
ER D
IALO
GUE,
BE
TTER
IN
FORM
ATIO
N,
BETT
ER A
CTIO
N
(CD
AC-N
PRO
JECT
)
WorldVision(lead),cDAcnetwork,ThomsonReuters
foundation,bbcMediaAction,Internews
bangladesh,SouthSudan,ThePhilippines
Toensurethattwo-waycommunicationisapredicable,coordinatedandresourcedcomponentofhumanitarianresponseinordertocontributetoimprovementineffectivedeliveryofassistanceto
disasteraffectedcommunities.
3,000,000
TRAN
SFO
RMIN
G
SURG
E CA
PACI
TY
(TSC
)
ActionAidInternational(lead),Acf,christianAid,cAfOD,cARE,IMc,IslamicRelief,MuslimAid,Plan,Savethechildren,Tearfund
Pakistan,thePhilippines,RegionalHubinbangkok,Thailand
Tostrengthencivilsocietysurgecapacityatinternational,regionalandlocallevels,contributingtoadiverseanddecentralisedthirdsectorpillarbetterabletocomplementexistingUnited
nations,Redcrossandgovernmentstructuresinordertohelpcommunitiesincreaseresilience,
reduceriskandimprovecrisisresponse.
2,482,824
ALER
T
HelpAgeInternational(lead),care
international,HandicapInternational,IslamicReliefWorldwide,concernWorldwide,Oxfam,coventryUniversity
ThePhilippines,Pakistan,bangladesh,Kenya,
Mozambique,Somalia,Haiti
Todevelopasystemofapproachesandtoolsfordisasterpreparednessthatincreasestheabilityoforganisationstorespondimmediately,effectively
andappropriatelywhenadisasterstrikes.
1,987,000
FIN
ANCI
AL
ENAB
LERS
(F
E) OxfamGb(lead),Tearfund,christianAid
ThePhilippines
Totransferhumanitariancapacity,autonomyanddecision-makingtoorganisationsclosertopeopleaffectedbycrisis,asawayoffacilitatingmore
effectiveandappropriateaid.
1,637,745
AGE
AND
D
ISAB
ILIT
Y CA
PACI
TY
BUIL
DIN
G
(AD
CAP)
HelpAgeInternational(lead),cbM,
DisasterReady.org,HandicapInternational,IfRc,OxfordbrookesUniversity,RedRUK
Kenya,Pakistan
Toensureolderpeopleandpersonswithdisabilitybenefitfromimprovedaccesstoservices,asaresultofrecognitionbyhumanitarianactorsoftheirspecificneedsandincreasedcapacityamongsthumanitarianactorstodeliverinclusive,
accessibleandappropriateresponse.
1,045,159
LIN
KIN
G
PREP
ARED
NES
S RE
SPO
NSE
AN
D
RESI
LIEN
CE (L
PRR)
christianAid(lead),ActionAid,concern,HelpAgeInternational,King’scollegelondon,MuslimAid,Oxfam,Saferworld,World
Vision
bangladesh,Kenya,DRc,Pakistan,thePhilippines,colombia,Indonesia***,Myanmar
Todesignandrolloutprogrammingapproacheswhichstrengthentheresilienceofpeoplelivinginfragilestatesandbeneficiariesofhumanitarian
assistance.
1,002,964
STRE
NG
THEN
ING
EM
ERG
ENCY
PR
EPAR
EDN
ESS
SYST
EMS
IN
MYA
NM
AR (S
EPS) christianAid(lead),
DcA,RegionalIntegratedMulti-HazardEarlyWarningSystemforAfricaandAsia
(RIMES)
Myanmar
Tostrengthenthelinksamongnational,subnationalandlocal-levelpreparednessin
Myanmar,withaprimaryfocusoncapacitatinglocalcommunitiesandstructurestoaccessinformationandlinkwiththeon-going
establishmentofpreparednessandearlywarningsystemsinthecountry.
925,698
8 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
PRO
TECT
ION
IN
PRAC
TICE Oxfam(lead),
IRc,WorldVisionInternational
DRc,lebanon,Turkey,
SouthSudan,Myanmar,PakistanandthePhilippines
ToenablenationalnGOstoimplementprotectionactionsindisasterandconflictresponses,developnewtypesofpartnershipsandcollaborationsbetweenprotectionactorsandinfluencethe
internationalprotectionarchitecturesoitismoreinclusiveofnationalnGOs.
870,137
IMPR
OVE
D EA
RLY
WAR
NIN
G – E
ARLY
AC
TIO
NS T
O ST
REN
GTH
EN
DIS
ASTE
R PR
EPAR
EDN
ESS
IN ET
HIO
PIA
(EW
EA)
OxfamGbonbehalfofAfricacllimatechangeResilienceAlliance(lead),christianAid,nationalDisasterRiskManagement
commission,nationalMeteorologicalAgency
Ethiopia
Tocontributetoimprovedemergencypreparedness,timelinessandqualityofrisk
informationandresponseactionsbycommunity,government(localtofederal)andnGOsina
coordinatedmanner
784,255
URB
AN E
ARLY
W
ARN
ING
, EAR
LY
ACTI
ON
(UEW
EA)
concernWorldwide(lead),Oxfam,KenyaRedcrossSociety
Kenya
Toimproveurbanearlyactionbyimprovingthealignmentoflocalandmunicipalgovernanceinstitutionsandresponseagenciestowardsidentifyingrelevanttriggersandimplementingearlyresponsebasedonthese,supportedbyanincreaseintheallocationoffinancialsupportfromkeydonorstowardsurbanEarlyWarningEarly
Action.
675,000
PUBL
IC H
EALT
H
EMER
GEN
CIES
PR
EPA
RED
NES
S IN
GA
MBE
LLA
(P
HEP
)
christianAid(lead),AmrefHealthAfrica,EthiopiaMinistryofHealth,nationalMeteorologyAgency
EthiopiaTostrengthenearlywarning,preparednessandpromptresponseofthehealthsectortopublic
healthemergencies.548,694
Table1:ProjectsintheDisastersandEmergenciesPreparednessProgramme2
*Formerly known as the Monitoring, Evaluation and Learning Project (MEL).
**Includes non-DEPP focal countries where projects were implemented (i.e., Lebanon, Somalia, Haiti, Colombia, Turkey).
***Indonesia was initially selected as an 11th DEPP focal country but none of the selected projects planned implementation in Indonesia. However, over the course of the programme, LPRR ultimately did implement some activities in this country.
TheDEPP’slogicalframeworkoutlinedthefollowingoutputs,outcomesandimpactindicatorsforthethree-yearprogrammepresentedinTable2below.
STATEMENT
OU
TPU
TS
1.Improvedknowledgeandunderstandingofindividualsbysharingbestpracticeofhumanitarianpreparednessandresponse
2.Improvedpreparednesssystemsforearlyactionwithcommunitiesatriskofdisasters
3.Increasednumberofcoalitions,partnershipsandnetworkswhichworkingtogether,areabletoaddresshumanitarianneedsinawiderangeofemergencysituations
4.Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhumanitarianresponseandpreparednessarebettersupportedandmoresuitable
2 TheSTARTnetwork.DisasterandEmergenciesPreparednessProgrammeInfographic.August2015.Accessedon18february2016athttp://www.start-network.org/wp-content/uploads/2015/08/DEPP-In-fographic.pdf.
9 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
OU
TPU
TS 5.Strengthenedevidencebaseforwhatworkstohelpbuildhumanitariancapacityatscale
6.Human-centereddesignapproachestestedandimplementedacrossfourinnovationslabs,leveragingnon-traditionallocalactorstodeliverpreparednessapproacheswhicharecommunity-appropriate,leadingtomorepreparedandengagedcommunities
OU
TCO
MES
1.IncreasedandstrengthenedemergencypreparednesscapacityinDEPPcountries,focusingonstrengtheninglocalhumanitariancapacityandchampioninglocalisationinamannerconsistentwiththeGrandbargainandtheWorldHumanitarianSummit
IMPA
CT
1.DEPPreducessufferinganddeliversbetterhumanitarianservices,throughimprovedpreparedness,forpeoplearoundtheglobeaffectedbydisastersandemergencies
Table2:SummaryofDEPPprogramme-levellogicalframework
Thefinalversionoftheprogramme-level logical frameworkwasapprovedbyDfID in2017and fully implementedbyAugust2017, severalyearsafter thestartof theprogrammeandthecommencementoftheexternalevaluation.Whiletherewereseveraldraftversionsoftheprogramme-levellogicalframeworkinplacebeforethen,theyrequiredsignificantstrengthening3.ThereforewiththeagreementfromDfID,theevaluationwasnotdesignedinconjunctionwiththelogicalframeworkanddoesnotreportagainstit(Seebelowforfurtherdetails).Inaddition,outputarea6intheprogrammelogicalframeworkrelatestotheinnovationwindow,referredtoastheDEPPInnovationlabs.Thoughoriginallyconceivedtobeimplementedsimultaneouslywiththe14DEPPprojects,duetodelays,itsimplementationbeganrecentlyandwillcontinuefortwoyears,endingin2019.Asitwasnotimplementedduringthetimeframeoftheexternalevaluation,DEPPInnovationlabswasnotpartofthisexternalevaluation.Aseparateevaluationhasbeencommissioned.p
DEPP THEORY OF CHANGE AND SIMPLIFIED CAUSAL CHAINDuring theevaluation inceptionphase, theevaluation teamassessed theDEPPprogrammetheoryofchange(Annex3)presentedinthebusinesscase(Annex1).Theprogrammetheoryofchangewasfoundtobecomplexandnolongerrepresentedwellthesuiteofprojectsastheyhadevolvedovertime.AprocesswasundertakentocreateamoresimplifiedbutconceptuallyappropriaterepresentationoftheDEPPthatcouldserveasthebackboneoftheevaluationapproachand to reportagainstduringeachevaluationphase.figure1 illustrates theDEPPprogramme causal chain thatwas developed and thatwas used to develop a programme-level logical frameworkandsetof indicators for theevaluation4.Thecausalchain illustrates
3 forexample,DfID’sAnnualReviewoftheDEPP,March2018states:“Thereisstillmoreworktodoonthelogframe–includingfinalisingtheoutcomeindicatorsandincludinganoutputwhichtracksvalueformoney.”
4 Page17oftheDEPPbusinesscasestated,“Throughacommissionedevaluation,wewilldeveloptheTheoryofchangeinmoredetailaspartoftheevaluationinceptionstage,andwewilltestourcoreassumptionsthroughtheDEPPevaluation.”TheevaluationteamrevisedtheTheoryofchangetoenableafeasibleevaluationdesignasmandatedintheinceptionphaseandtheTermsofReference.TherevisedcausalchainwasapprovedbytheEvaluationSteeringcommitteeandthusbecamethebasisfortheeval-uationdesign,includingtheevaluationframeworkandindicators.notethatwhiletherewereseveraldraftprogrammelogicalframeworksinplacein2016,thefinalversionwasnotapproveduntil2017.
10 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
thecausallinksbetweenthe14DEPPprojectsandtheiractivities,fourexpectedevaluationoutputareas(1.capacitybuilding,2.collaboration,3.learningand4.EarlyWarningSystemDevelopment),expectedoutcomes inboth theshort term (improvedhumanitariancapacity,improvedpreparedness)andlongterm(increasedeffectivedeliveryofhumanitarianresponse).Hypothesiseddownstream impacts (mortality,morbidity,economic impactandrecovery)arealsodepictedbelow,butarenotexpectedtobeachievedwithintheprogramme’sthree-yearcycleandthuswereconsideredoutsideofthescopeoftheoverallevaluation.Thecausalchainalsospecifiesthattheactivities,outputsandoutcomesareexpectedtooccurattheindividual,organisational, community and government levels. This also captures the variety of DEPPbeneficiariesthatweretargetedateachoftheselevels(includingindividualhumanitarianstaff,humanitarian organisations, communties, governments etc). This impact evaluation focusedonlyonassessingoutputs,aswellasshort to long-termoutcomesat thesedifferent levels. notethattheoutput,outcomeandimpactareasintheprogrammelogicalframework(Table1)arequitesimilartotheevaluationteam’soutputandoutcomeareasasdepictedinfigure1.5 However,theindicatorsareunique(Seechapter2forfurtherdetail).
figure1:SimplifiedDEPPprogrammecausalchain
5 Programmeoutput1correspondstoevaluationoutput1,programmeoutput2toevaluationoutput4,programmeoutput3toevaluationoutput2,programmeoutput4doesnotdirectlycorrespondwithanyevaluationoutputarea,programmeoutput5correspondstoevaluationoutput3,programmeoutput6doesnotcorrespondtoanyevaluationoutputarea,programmeoutcome1correspondswithevaluationoutcome1,andprogrammeimpact1correspondswithevaluationimpact1.Programmeoutputarea6correspondstotheinnovationwindowwhichisnotwithinthescopeofthisevaluation.impact1.
DEPPProjects
HumanitarianCapacity
Preparedness
EffectiveDelivery ofHumanitarianResponse
MortalityMorbidityEconomic ImpactRecovery
LEVELS OF ACTION
IndividualsCommunitiesOrganisationsGovernment
INPUTS &ACTIVITIES
OUTPUTS SHORT TERMOUTCOMES
LONG TERMOUTCOMES
IMPACT
Collaboration
CapacityBuilding
Early WarningSystem
Development
Learning
11 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
PURPOSE Thethree-yearexternalimpactevaluationoftheDEPPaimedto:
1. improveprogrammeeffectivenessandenhancelearning;and
2. assesstheextenttowhichtheDEPPoverallhasprovidedanefficientandeffectiveapproachtostrengtheningresponsecapacity.
The evaluation also served to provide accountability and learning for programme, projectand external stakeholders. These stakeholders include DEPP project consortia members,DEPPbeneficiaries,programmeandproject-levelstaff,aswellasexternalstakeholders fromhumanitariannGOs,internationalorganisations,theUn,andgovernments.
Theevaluationwasimplementedinfourphases:aninceptionphaseduringwhichtheevaluationframeworkandmethodologyweredesigned;aformativephasetoevaluatetheimplementationof theDEPPby assessing the relevance of outputs and the efficiency and effectiveness ofprogrammedelivery;aninterimphasetoassessshort-termoutcomes;andasummativephasetoassessintermediateoutcomesandpreliminaryindicatorsoflikelihoodofimpact.
The research design for the external evaluation was guided by a set of questions andmethodologicalapproachoutlinedintheTermsofReference(ToR)(Annex4)andrefinedintheinceptionreport.Itusedamixedmethodsquasi-experimentaldesigntoassessimpactattheprogrammelevel.Itwasnotdesignedtoassessindividualprojectimpacts(whichwereassessedby each project’s own independent evaluation), or the £10million innovationwindow.Theexternalevaluationmethodology includedaminimumsetofevaluationactivitiescomprisinga document review and remote data collection across all countries, aswell as an intensivesetofevaluationactivities in fourcountries (Myanmar, thePhilippines,KenyaandEthiopia).Theintensivesetofevaluationactivitiesincludedquantitativeandqualitativedatacollectioncomprisingorganisational-levelassessmentsandcommunity-levelassessments;aneconomicValue for Money (VfM) assessment; analysis of humanitarian preparedness and responsenetworks;humanitarianresponseassessment/casestudies;observationandsitevisits. Inthefourintensivesetcountries,theevaluationteampartneredwithlocalresearchorganisationswhoimplementedthedatacollection.TheseincludedthebusaracenterforbehavioralEconomics(Kenya),TheEthiopianPublicHealthAssociation(Ethiopia),andInnovationsforPovertyAction(Myanmar,ThePhilippines).Datawerecollectedattwotimepointsthatwereroughly12monthsapart–timepoint1(T1)duringtheformativephaseoftheevaluation,andtimepoint2(T2)intheinterimphase,inordertoassesschangesovertime.Intotal,634documentswerereviewed,and2542quantiativesurveys,and133qualitativeinterviewswereconductedatT1and3291quantitativesurveysand149qualitativeinterviewswereconductedatT2.EvaluationfindingsrelatedtoT1werepresentedintheformativephasereportandthissummativereportpresentsT2findingsaddressingthefiveevaluationquestionsandsub-questions.Thesummativereportincludesfindingsfromallphasesoftheexternalevaluation(november2015–May2018).
Thesummativefindingscorrespondtothekeyevaluationquestionsinthefollowingfiveareas:
1) RelevanceandValidityofDesign;2) RelevanceandEffectivenessoftheInterventions;3) EffectivenessofManagementArrangements4) EfficiencyandValueforMoney(VfM);5) SustainabilityoftheInterventionandlikelihoodofImpactoftheProgramme.
Thekeyfindingsarepresentedbelow.
12 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
MAIN FINDINGS 1. RELEVANCE AND VALIDITY OF DESIGN6
The established five results areas (objectives) of theDEPPwere appropriate, and the rightpeople in the rightplaceswere targeted.The focusonbuildingnationalcapacity, improvingpreparedness and targetingvulnerable groups to ensure their inclusionduring humanitarianresponseactivitiesisconsistentwithneedsidentifiedduringliteratureandevidencereviews,aswellaswithnewerglobalcommitmentsrelatedtohumanitariansystemreformandlocalisation7 ofaid.At ahigh level, theprogramme’sobjectiveswereclear, relevant, alignedwithDfID’shumanitarian priorities and aim to fill a clear gap in humanitarian capacity. However, theprogramme’sthree-yeartimeframewasunrealistictomeettheseobjectives.
Thecountryselectionprocesswasiterative,basedonappropriatecriteriabutwasnotoptimalduetolackofstrategicdirectionandobjectivesattheportfoliolevelandultimatelysomekeycountriesatriskofhumanitariancrisesmaynothavebeenincluded.Theportfolioofinterventionsineachcountryvariedwhichcouldhaveledtolostopportunitiestomaximiseimpactandefficiencieswithineachsetting.
Thedesignprocess,atleastintheinitialphase,wasnotlogicalandcoherent.Theseissuesledtothedesignandselectionofmanyprojectsthatwereretrofittedtothebusinesscaseorselectioncriteria.localinvolvementisacriticalstepwithinthedesignprocessthatwasnotadequatelyconsidered,leadingtochallengeswithrespecttolocalownershipandstakeholderbuy-inandpotentiallyreducingwiderimpactoftheDEPP.Thedesignprocesswasparticipatorybutlocalbeneficiariesandstakeholderswerenotadequatelyinvolved.
Insufficient time and resources were allocated during the design phase to ensure projectconsortiahadthenecessarytimeandspacetogrow,andthatkeystakeholders,especiallyatthelocallevel,couldbeinvolvedinaparticipatoryway.Individualsandagenciesdidnotalwayshave the capacity to collaborate and facilitation and capacity building around collaborationcould have contributed to healthiermore effective collaborations.The design phase duringthesecondroundofSTARTnetworkprojectsaddressedmanyoftheseweaknessesandwasledby in-country teamsensuring localbuy-in, andeventually smootherandtimelierprojectimplementation.
ResourcingfortheDEPPattheportfoliolevelwasconsideredsufficient,butprojectbudgetsweretight,with insufficientnon-project attributable costs (nPAcs), inadequateallocationoffundsforcollaborativeactivitiesandvariableallocationformonitoringandevaluation(M&E).Resourceswere also needed during the design phase to ensure that smaller agencies andin-countrystaffcouldparticipateinameaningfulway.
Designattheprogrammelevelwasnotadequatelyconsideredandprojectsweredesignedinisolationwith limited localconsultationandbuy-in.Despite theflaws in thedesignprocess,
6 fullevaluationfindingsarepresentedinchapter3.7 Thereisnogloballyaccepteddefinitionoflocalisation.Onesuggesteddefinitionis:“Aidlocalisationisa
collectiveprocessinvolvingdifferentstakeholdersthataimstoreturnlocalactors,whethercivilsocietyorganisationsorlocalpublicinstitutions,tothecentreofthehumanitariansystemwithagreaterroleinhumanitarianresponse.Itcantakeanumberofforms:moreequitablepartnershipsbetweeninternationalandlocalactors,increasedand“asdirectaspossible”fundingforlocalorganisations,andamorecentralroleinaidcoordination.Underpinningthisisthequestionofpower.localisationrequiresashiftinpowerrelationsbetweenactors,bothintermsofstrategicdecision-makingandcontrolofresources.”Morethanthemoney-localisationinPractice,GroupeURD,Trocaire,2017.
13 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
theselectedprojectsdidaddresscountryneedsonce theywerecontextualisedduring theirimplementation periods. The objectives, systems and processes, the programme theory ofchange,definitionofkeyterms,andexpectationsabouthowprojectsareintendedtointerlinkandinteractwerenotdevelopedandarticulatedattheoutset.Genderinclusionwasrequiredbutspecificguidelineswerenotclearlyarticulatedduringtheprojectdesignstage.
Responsetoemergencieswasnotadequatelyconsideredduringthedesignphase–theprogrammewasdesignedtoimproveemergencypreparednessbutmechanismstoenableorfacilitatecontributiontohumanitarianresponseeffortswerenotexplicitlyintegratedintotheprogramme(leadingtoreducedpotentialimpact).Despitethis,DEPPcontributedtoatleast42humanitarianresponsesin11countries.
2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS8
Implementationdelayswereuniversalacross theDEPPprojects.Delaysweredueto limitedcontextualisationandinvolvementofin-countryteamsinthedesignphase,underestimationoftimerequiredforcollaboration,andadministrativeandcontractualbottlenecks.Thesedelaysledtoshorterperiodsforprojectactivitiesandreducedpotentialprogrammeimpact.
ThenumberofindividualsexposedtoDEPPcapacitybuildingactivities(n=33,388)exceededtheultimatetargetoftheprogrammeoftraining4,200individualsby795%.Thegendertargetwasmissedbyabout5%(i.e.,45%ofthesetraineeswerewomen).Examplesoftheseactivitiesincludes in-person trainings, developmentof trainingpackages andguidelines, developmentof earlywarning systems, collaboration activities such as learning events and conferences,developmentofcasestudiesandevidencesharinganddissemination.
DEPPcontributed to increasedcapacitybuildingefforts, collaboration,earlywarningsystemdevelopment,andlearninginbeneficiaryorganisations.However,outcome-levelchangeswithrespecttoknowledgechange,andinpreparednesslevelswerevariable.Overall,therewerenosignificantimprovementsinknowledgeoncorehumanitariancompetenciesofhumanitarianstaff,onself-reportedknowledgeondisasterpreparedness,responsetodisastersandemergenciesorageordisability-relatedissuesindisastersdespitetheincreasedexposuretocapacitybuilding.
Themosteffectiveindividual-levelcapacitybuildingapproachreportedbybeneficiariesinvolvedin-persontrainingcombinedwithastrategytoreinforceknowledge(simulations,mentoring,coaching).flexiblefundingapproaches,thoughinitiallyconsideredhighrisk,werefoundtohavehighpotentialofeffectiveness.Distanceorremotelearningwasnotreportedtobeaseffectiveasin-personapproaches.
Therehavebeensomesignificant improvements in individual,organisationalandcommunitypreparedness among DEPP beneficiaries, especially among local organisations. However,bothDEPP and non-DEPP participantswere exposed to capacity building activities relatedtoemergencypreparednessandresponse,andinsomecasessimilarimprovementshavealsooccurred in thecomparisongroups. Thechanges in thecomparisongroupmayormaynothavebeenrelatedtoexposuretoothercapacitybuildingactivities.Overall,formostindicators,the Difference-in-Difference analysis showed no overall net effect of the DEPP. However,qualitativedatasuggestssomeimprovementsinemergencypreparednessthatmaybedifficulttodetectwithquantitativeindicators.
8 fullevaluationfindingsarepresentedinchapter4.
14 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Documentreviewandin-depthinterviewsdemonstratecontributionoftheDEPPtoat least42differentemergencyresponsesin11countries.TherewerenodocumentedcontributionstoresponsesinMozambique,oneofthecountrieswiththefewestDEPPprojects(n=1).Thelargestnumberofresponsecontributions (n=16)occurred inthePhilippineswhichalsohadthehighestnumberofDEPPprojectsimplemented(n=6).notethatKenyaandPakistanalsohadsixDEPPprojectsbutinthesespecificcontexts,DEPPcontributedtosubstantiallyfewerresponsesthaninthePhilippines.ThehighperformanceinthePhilippinesintermsofabsolutenumberofhumanitarianresponseeffortswaslargelydrivenbythefinancialEnablersprojectwhich cumulatively involved contribution towards 14 different responses (out of 16DEPP-supported responses in thePhilippines).Qualitativedata collected in thePhilippines,KenyaandEthiopiasuggestmoretimely,locallydrivenresponsesinthesecontexts;qualitativedatawerenotavailabletosystematicallyassessresponsesinmanyothercountries.However,itwasevidentthat inmanycasesDEPPbeneficiarieswereunabletoresponddueto lackoffundsallocatedforemergencyresponseactivities.Incontrasttothequalitativeindicators,quantitativedataonperceptionsofimprovedspeedandeffectivenessofresponseactivitiesdidnotshowasignificantmeasurableeffectoftheDEPP.
3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS9
collaborative approaches were consistently reported through in-depth interviews andquantitative data to be the most appropriate, preferred approach to deliver humanitariancapacitybuilding, but respondents identified challenges in engagement, communication andcoordination.Qualitativedata triangulatedwithdataextracted fromproject andprogrammedocuments demonstrate that as a delivery mechanism, collaborative and consortia-basedapproacheswereparticularlyeffectivewhenthefollowingconditionsweremet:existenceofjointobjectives,valuesandcommonwaysofworking;collaborationswereorganicnotforced;sufficienttime,space,resources,capacityandwilltofosterhealthycollaborativerelationships;and streamlined decision-making and contractual processes, and fewer consortia partners.Quantitativenetworkdatademonstratedthatthetopthreeareaswhereorganisationswithinthe humanitarian networks in the four intensive set countries (Ethiopia, Kenya, Myanmarand the Philippines) collaborated were advocacy, community capacity building and projectimplementation.
Thenetworkstructuresineachofthefourintensivesetcountrieswerefoundtobesignificantlydifferentfromtheoutset.Therewassomeevidenceofstrengthenedhumanitarianresponsenetworksamongthethreeofthefourintensivesetdatacollectioncountries(thePhilippines,Kenya and Ethiopia, while the network in Myanmar remained unchanged). The Philippinesnetworkwasfoundtobehighlylocallydrivenwith70%oflinkswithlocal/nationalnGOs(l/nnGOs); network size increased betweenT1 andT2, but the network became less densewith a smaller number of connections. There was evidence of network growth in both inEthiopiaandKenya.Thenetwork inEthiopiawas internationallydrivenwith little inputfromlocalorganisations,whileinKenya,thenetworkwasbalancedbetweenlocalandinternationalactors.ThenetworkinMyanmarwasverysmall,isolatedanddominatedbyinternationalactors.
findingsoncollaborationweremixed.collaborationfrequency(aproxyindicatoroftrust)andquality of relationships increased in Kenya and the Philippines, but decreased inMyanmarandEthiopiaovertime.collaborationwithintheDEPPcohortincreasedinMyanmarandthePhilippines,butbothdidnotincreaseengagementwithnon-DEPPactors.InEthiopiaandKenyathenumberofrelationshipsbetweenDEPPandnon-DEPPorganisationsincreasedindicatingthatDEPPinstitutionsarereachingbeyondtheirDEPPpartnershipsintothebroadersystem.
9 fullevaluationfindingsarepresentedinchapter5.
15 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
basedonthenetworkanalysis, therewasnoquantitativeevidenceof increased localisationover the courseof theprogrammewithin the four intensive set countries. In thisparticularanalysis, localisationwas assessed bymeasuring the proportion of relationshipswith local/nationalorganisationswithin thehumanitariannetworks.While therewasnochange in thismetricinEthiopia,thePhilippinesorKenyaovertime,inMyanmar,thehumanitariannetworkactuallybecamemoredominatedbyinternationalnGOs(InGOs)overtime.However,qualitativedata(seechapter7)anddatafromdocumentreviewsuggestthatchangesinattitudesaroundlocalisationhaveoccurred.Takentogetherthesedatademonstratethatattitudinalchangeshaveoccurredbuthavenotyettranslatedintoquantitativebehaviouralchangerelatedtolocalisation.
Inaddition, insufficienttimepassedbetweenthedatacollectionpointstosufficientlyassessandtestthehypothesisthatstrengthenednetworksandgreatercollaborationleadtobetteremergencypreparedness.DataatT1pickedupnetworkeffectscausedbyever-shiftingstrategiescommontofirst-yearimplementation.T2likelycapturedsomedistinctiveprogrammeimpactsofDEPP,butsufficienttimetofollownetworkchangerequiresatleastanadditional12months.
According toqualitativedata, inter-project collaborationcontributed to improved sharingoflearningandevidencebutthisdidnotnecessarilytranslatetobehaviouralchange.
4. EFFICIENCY AND VALUE FOR MONEY10
focusingoneconomyandefficiency,Value forMoney (VfM) showsgoodpotential for theprogrammewithareasforimprovement.Goodindicatorsofcosteconomywereevident,thoughsomebudgetswereleanwithinsufficientresourcesforportfoliomanagementandcollaborationactivities.
Thecollaborativemodelnecessarily lends itself toadegreeof inefficiency,hightransactionscostsandslowinformationflows.Ontheflipside,thismodelhadsubstantialbenefitsandforsomeprojectsthebenefitshavelikelyoutweighedthecosts.forotherprojectsthesamebenefitsofthemodelcouldhavebeenachievedatlowercostandgreaterefficiencyifthecontracting,M&Esystemandcostingofactivitieshadbeenmoreconsciouslyconsideredfrominception.Thecollaborativemodeltiedtohierarchicallongdeliverychainshaslikelycompromisedeconomyandefficiencytoadegree.
Therewereshortfallsintermsofsystemsandresourcessetupforefficiencyingovernanceandstrategy,portfoliomanagement,decision-makingandconsortiumarrangements.Thespendingon management functions at the programme level were too economical and would havebenefittedfromsomefundsandtimebeingredeployedfromthelearningProjecttotheDEPPmanagementteamintheformofM&Eexpertise,bothattheprogrammeandproject levels.If,however,independencewasdesired,thenmoredirectlinkageandcommunicationbetweenthemanagementteamandthelearningProjectwherebydatafromM&Etransmittedregularlyanddirectlytothemanagementteamthroughbothautomaticreportingordirectdataaccessthroughadashboardfollowedbyregularmeetingscouldhavebeenconsidered.
PositivefindingsintermsofVfMreporting,adaptivemanagementandcollaborativewaysofworkingwere identified.Aftertheformativephaserecommendationto increaseuseofVfMindicators, the DEPPmanagement teammade a strong effort to increase the project staffcapacity aroundVfM concepts and reporting through the implementation of two half-daytraining events. Subsequently, some projects carried out ad hoc reports onVfM andmostprojectsincorporatedVfMasacomponentoftheirfinalevaluations.
There are no strong quantitativeVfM findings to suggest that preparedness improved the
10 fullevaluationfindingsarepresentedinchapter6.
16 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
efficiency of humanitarian response. Analysis of the empirical data showed no significantdifferencebetweenDEPPandcomparisonorganisationswithregardstoperceivedimpactofDEPPon institutionalspeedandcostofresponse,andtheextenttowhich institutionalandpolicyenvironmentaffectedthespeedandcostofresponse.
costper result indicator analyseswereundertaken toestimate theexpenditure incurred toachieveoutcomesrelatedtocapacityandpreparedness.Whilereturnoninvestmentanalysesassessespotentialorprojectedreturnoninvestment,thecostperindicatoranalysescomparesactual expenditure incurred per result achieved.This analysis demonstrated goodVfM perresultactuallyachieved.ThehighestVfMwasforMyanmar,whichdemonstratedacostperpercentage point increase in perceived organisational preparedness of only £ 1,138. It notadvisable tomakedirect comparisonsbetweencountriesbecauseof thedifferences incostbaseswithincountries,andthemanyotherfactorsthatcomeintoplay.notably,Kenyaalsohadaverylowunitcostperresultarea.fortheindividualresponseperformanceindicatorinthePhilippines,theannualunitcostfigureforeachtrainedindividualisroughly£30.Itisdifficulttomake avalue judgment on this figure because there is limited basis for comparison, butgiventhefactthatasignificantimprovementinthesatisfactionratingwasobserved,andfromexperiencewithotherprojectsthat£30isnotoutsidethereasonablerange,thisappearstobeagoodVfMfinding.
5. SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOODOF IMPACT OF THE PROGRAMME11 DEPP has contributed to strengthening national emergency preparedness systems in someoftheprogrammecountries,butnotall.forexample, theUrbanEarlyWarningEarlyAction(UEWEA)projecttodetecturbanemergenciesandenablerapidresponsehasbeenadoptedbythenairobicountygovernmentinKenya,whohavealsocommittedtoaddressingurbanfoodsecurityissues.ThePublicHealthPreparednessprojectinGambella,Ethiopia,workedwiththegovernment to improve disease and outbreak surveillance, leading to improved surveillanceandreportingatvariouslevelsofthehealthsysteminseveraldistrictsandinthestrengthenedcapacitytotestforpathogensattheregionallaboratory.InPakistan,managementofthesurgeplatformdevelopedunder theTransforming Surgecapacity project has been takenover bythenationalHumanitariannetwork(nHn),withamemorandumofunderstanding(MOU)inplacewith the localgovernmentbodytoensure implementationafter theprojectends.TheMarsabit county Government in Kenya adopted the linking Preparedness Resilience andResponse(lPRR’s)project’sconflict-sensitiveapproachduringhumanitarianresponse(includingduringthe2016-2017droughts) thusprovidingadditionalevidenceofDEPP’scontributionstonationalpreparednesssystems.Inothersettings,however,thedegreetowhichtheDEPPhas influencednational preparedness systems andgovernmentdisasterplanshasbeen lessclear.Thepositiveinfluencesthathavebeenidentifiedareimportantandrepresentsignificantinvestmentandefforts.However,itshouldbenotedthatgenerallytheseeffectshavebeenonasmallergeographicscale–oftenatthedistrictorcountylevelandinmostcases,eachpositiveexampleofchangewasdirectlyattributabletoanindividualproject.
While the DEPP business case underscored the disproportionate impact of disasters andhumanitarianemergenciesonwomeninparticularandemphasisedthattheprogrammewouldstrategicallyaddress inclusion, at least in regard togenderandviolenceagainstwomenandgirls (VAWG), sucha focuswas largelyabsent fromtheprogramme.Across theprogramme,two projects specifically focused on inclusion, Protection in Practice (PIP) which focusedon protection, and Age and Disability capacity Programme (ADcAP) which focused on
11 fullevaluationfindingsarepresentedinchapter7.
17 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
strengtheninginclusionoftheelderlyandpeoplewithdisabilitiesinemergencyresponse.bothof theseprojectsmadestrides tobuildprotectionand inclusioncapacityand insomecasesinfluencedtheworkofotherDEPPprojects.ADcAPinparticularstronglychampionedinclusionefforts includingatprogramme-leveleventsandhelpedshapeprogramme-leveldialogueonthis issue.However, the extent towhich priority groups (women, children, the elderly, andpeoplewithdisabilitiesandothervulnerableormarginalisedgroups)wereholisticallyincludedacross theprogrammeand inclusionefforts adequatelymonitoredwasminimal.The lackofmainstreamingofinclusionofgenderandprioritisedgroupsacrosstheprogrammeresultedinafragmentedprogramme-wideapproachandunevenresults.WhiletherewasnoevidenceofindividualknowledgechangeoninclusionofvulnerablegroupsoronVAWG,therewassomeevidenceoforganisationalchangewithrespecttoinclusion,inparticularinEthiopiaandamonglocal organisations in intensive set countries. InEthiopia, therewas a statistically significantincreaseininclusionoftheelderlyandpeoplewithdisabilitiesindesignandimplementationofpreparednessprogramminginDEPPorganisations,aswellasastatisticallysignificantincreasein organisationswith policies on inclusion.Therewas also a statistically significant increasein theproportionof localorganisationsacrossall intensivesetcountries thathave inclusionpoliciesinplace.
Previous external evaluation reports from the interim and summative phase documented aconsistentlackofdisaggregateddataandminimallydocumentedgenderconsiderationsacrossprojects.Recommendationstostrengthengenderreportingwithinprojectdataweretakenupaftertheformativephaseevaluationreport.Therewasanincreaseindatadisaggregationbygender over the course of the programme.However, reporting on other prioritised groupsremained low,making itdifficult toassessthetrueextenttowhichthesegroupshavebeenreachedbycapacitybuildingorotheractivities.
TherehavebeenseveralconcreteexampleswhereinDEPPhasimpactedgovernmentpoliciesor systemsor increasedpolitical commitment.forexample,UEWEAproject’s advocacyandsensitisationefforts led to anamendmentof theDisaster andEmergencyManagementAct2015(DEMAct)toincludefoodsecurityandanMOUwiththenairobicitycountyGovernmentto ensure the government’s role in addressing urban food security issues12. The linkingPreparednessResilienceandResponseprojectdemonstratedchangesattheinstitutionallevel.for example, christianAid adopted the updated resilience framework13, and aViolence toPeacestrategy14.Despitetheseexamples,therewasnostatisticallysignificantchangeacrosstheDEPPinpolicy-relatedquantitativeindicatorsinthefourintensivesetcountries.changesmaytakesignificanttimetooccurandquantitativedatamaynotfullycaptureothertypesofpolicychangeorshiftsinthelandscapethataremoredifficulttomeasurequantitatively.
changesinquantitativeindicatorsrelatedtolocalisationhavenotyetoccurredwithinthreeoutoffour intensivecountries. InEthiopia,however,thepercentageofDEPPorganisationswithpolicies inclusivetol/nnGOsincreasedby16%betweenT1andT2.Inaddition,attitudinalchangestowardslocalisationhavealsooccurred,includinginhowInGOsconsider,address,andinvolvelocalactorsandcommunities.ThisadoptionofthelocalisationapproachisconsideredtobeDEPP’smostsignificantchangebykeyinformants.Examplesofthistypeofchangeweredescribed in bangladeshwhere local actors became involved in decision-making processeswithinalargenationalpreparednessplatform,andinthePhilippineswherelocalorganisationswerethefirsttomobiliseandrespondtothecrisisinMarawi.
There is some evidence of benefits being embeddedwithin organisations and systems and
12 UrbanEarlyWarningEarlyActionfinalEvaluationReport.13 https://www.christianaid.org.uk/sites/default/files/2017-05/resilience-framework.pdf.14 https://www.christianaid.org.uk/resources/about-us/tackling-violence-building-peace-global-strate-
gy-2016.
18 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
potential for longer-term effects but this is dependent on the extent to which programcomponentsareabletocontinuebeyondtheendofDEPP.Thelikelihoodofsustainabilitywasenhancedforprojectsorprojectcomponentsthatdemonstratedthefollowingcharacteristics: builtonexistingworkandexistingpartnershipsand/orconsortia;partneredwiththenationalandlocalgovernment;hadand/ordevelopedapolicyoradvocacyelement;workedtowardssystems-levelchange;paidgreaterattentiontoexitplansearlierintheprojectcycle;developedtools, guidelines, or systems that fill an important gap; involved beneficiaries and/or localstakeholders, and exhibited good VfM; built linkages with other entities and other DEPPprojects;andimplementedincontextsthatweremorefertileforchange.
CONCLUSIONS AND RECOMMENDATIONS TheDEPPprogrammewasdesignedin2014,andwasaheadofitstimeinmanyways,especiallywithitsstrongfocusonlocalisationandmulti-stakeholderconsortia.Itincludedamixofbothflagshipprojectsusingmoretraditionalapproaches(i.e.,in-personindividualcapacitybuilding)withsomemoreinnovativeelements(i.e.,a£10millionInnovationWindow;flexiblefundingmechanismstofinancebothlocalcapacitybuildingbutalsoemergencyresponse;pooledsurgeplatforms;urbanearlywarningsystemsandinclusionoffoodsecurityasanemergency).
TheDEPPsufferedfromanumberofdesignchallengesincludingathree-yeartimeframethatwasunrealistictomeettheobjectivesanduniversal implementationdelays.ThedecisionbyDfID tono longer fund theprogramme limits thepotential long-term impactof theDEPP.Despitethisdecision,theDEPP’sfocusonemergencypreparednessandlocalisationisstillarelevantone.Italignswellwiththegrowingdialogueamongtheglobalhumanitariancommunityandwithmore recent policy commitments such as theGrand bargain and thecharter forchangewhichadvocateforincreasedvoiceforlocalactors,andforrebalancingofpowerandrelationshipswithintheexistinghumanitarianarchitecture.
Despite its initialdesignflaws,positivefindingsrelatedtothecollectiveactionoftheDEPPand to individual DEPP projects have been documented. These include the following keyachievements:
• 33,388 individuals were exposed at varying degrees to capacity building based onprojectreports
• Stakeholders across the programme noted that themost significant change due totheDEPPwasachangeinattitudestowardandincreasedsupportoflocalisation(i.e.,contributingtotheGrandbargain)
• changesincapacityoflocalorganisations(intermsofsystemsandprocesses,fundraising,conductingneedsassessments)andtheiremergencypreparedness levels,that ledtooutcomessuchasbecomingeligibleforUnpooledfunding,successinsecuringexternalemergencyresponsefunds,andbeinggivenleadershiproles
• Significant changes in organisational policy with respect to inclusion of vulnerablegroups(Ethiopia,localorganisationsacrossallintensivesetevaluationcountries),andlocalisation(Ethiopia)
• Someevidenceofstrengthenednetworksandincreasedcollaboration(Ethiopia,Kenya,thePhilippines)
19 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
• contributiontoatleast42humanitarianresponsesin11countriesandsomequalitativeevidenceinseveralsettings(ThePhilippines,Kenya)ofimprovedspeed,efficiencyandinclusivenessofemergencyresponse
• cross-country learning in termsof theamountof learningdocumentsproducedandsharingofideasacrosscountriesandprojects
• Strongcross-projectcollaborationinthePhilippinesdemonstratingbenefitsofcollectiveaction
nevertheless,agreater impact,asoriginallyenvisioned,could likelyhavebeenachievedhadtherebeenamorecohesivedesign,alongertimeframe,amorerobustprogrammeM&Esystemandamorestrategicapproachwithregardstotypeofcapacitybuildingimplementedateachlevelofaction(individual,organisational,community,systems)andthebalancebetweentheselevels.Overall,theexternalevaluationfoundthefollowingfindingsthatneedtobeexaminedfurtherinfutureevaluationsandprogrammingrelatedtoemergencypreparedness:
• noquantitativeevidencetodatethatthethree-yearprogrammehasledtomeasurableimpactwithregardstomoreefficient,andtimelyhumanitarianresponse
• noquantitativeevidencetosupport thehypothesis thatstrengthenednetworkscanlead to improved emergency preparedness and response (mainly due to the shortobservationperiodavailabletoassessthis)
• noquantitativeevidencethatDEPPimprovedindividualknowledgeoncorehumanitariancompetencies,preparedness,inclusionofvulnerablegroups,orprotectionissuesinthefourintensivesetcountries
• fewchangesinquantitativepolicy-relatedindicators,thoughthereweresomeexamplesofimprovedpoliciesdocumentedbyprojects
• Toowideofavarietyofindividualcapacitybuildinginitiativesandlackofstrategyontypeofcapacitybuildingapproachesateach level (individual,organisational, system,community)andeffectdesiredateachlevel
The three years allotted for an emergency preparedness programme to achieve fivemacroresults in10countrieswasanambitioustimelineandagenda.The levelofexposure to theDEPP programme activitieswas unevenly distributed across the 10 DEPP focus countries;eachcountry receivedonlya subsetof the14projects.forexample, threeDEPPcountriesbenefittedfromsixprojectseach,whiletwocountrieshadonlyoneprojecteach(notincludingthelearningProject;seeTable4.7forthefulllistofprojectnumberspercountry).ThisunevenlevelofexposuretoDEPPcompoundedwithvaryingcountry-specificcontextualfactorshasledtovaryingresultsinspecificcountries.HadtheDEPPbeenabletocontinueforanotherthreeyearsormoreandbeenabletointegrateitslearningsintoarevisedprogrammestrategy(i.e.,permittingtimetorestructureitsmanagementstructure,scaleupprojectsthatshowpromiseandreviseprojectsthatarelesspromising),therewouldbeamorefeasibletimeperiodtoobtainmoreconclusivefindingsaboutDEPP’sachievementstowardsitsfiveresultareas.
20 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
RECOMMENDED ACTIONS FOR PROGRAMMECOMPONENTS
Given the key evaluation findings, there is scope to replicate and further test, or scale upsomecomponentsoftheprogramme(Table3).ThiswillensurethatthepreviousinvestmentinDEPPissustained,andthatanyadditionalresourceswillbeusedeffectively,efficientlyandstrategically. based on evaluation findings that have been triangulated using multiple datasources,noneoftheDEPPprogrammecomponentshavebeenrecommendedtobeceasedandnotbepursuedinthefuture.ThisinitselfisastrongindicationthattheDEPPcoretheoryof change and initial assumptions areworthy of being re-examined for future investment.Specially,ninecomponentswererecommendedtobemodifiedandre-tested,whileeightwererecommendedtobereplicatedindifferentcontextsandpotentiallyonalargerscale,enablingmorerigorousevaluations(Table3)notethatinaprogrammeeachofthesecomponentswouldnotbeexpectedtofunctioninisolation,andtheinteractionbetweencomponentsshouldalsobeconsidered.
DEPP PROGRAMME COMPONENT
RECOMMENDED ACTION KEY FINDING / LESSONS LEARNED
Individual Capacity Building Modifyandre-test
• Prioritisemulti-prongedapproaches(in-person+coaching,mentoring,simulations)
• Qualityoftrainingsshouldbeprioritisedoverquantityofindividualstrained(webinarscanreachalotofpeoplebutmaybelesseffective)
• considerstructuralbarrierswithinorganisationsthatmayimpedeapplicationoflearning
Organisational Capacity Building Modifyandre-test
• Prioritiseapproachesthatpermitorganisationalself-assessmentsandcapacitybuildingapproachestailoredtoorganisations’needs
• focusonstrengtheningofadministrativepolicies,proceduresandsystems(i.e.,humanresources,procurement,financesystems)hasbeeneffectiveforl/nnGOs
• Approachesshouldbebetteradaptedtodifferentcontexts
• Accesstofundingiskeyfororganisationstoputlearningintopractice
• conductcomparativeassessmentsoftheeffectivenessofdifferentapproaches
Community Capacity Building Modifyandre-test
• fewprojectsaddressedthis,butthosethatdidindicatedsomepotential;furtherevidenceisneededonhowbesttobuildcommunitycapacity
Systems Capacity Building Modifyandre-test
• fewprojectsdirectlyaddressedthis,buttherehasbeensomeevidenceofqualitativeattitudinalchangethatcouldbeaprecursortoeventualsystems-levelchange;moretimeandfurtherevidenceareneededonhowbesttobuildcapacityatthesystemslevel
Early Warning Systems
Replicateandtestonlargerscaleindiffering
contexts
• UEWEA’surbansurveillancesystemshouldbetestedinotherurbansettings
• Otherearlywarningsystemprojectscouldbeadaptedforandimplementedinothercontextswithsufficientprojecttimelines
21 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
Pooled Surge Platforms
Replicateandtestonlargerscaleindiffering
contexts
• Pooledsurgeplatforms,especiallynational-levelplatformshaveshownsomeevidenceofeffectiveness.Pooledsurgeplatformsshouldensurealocallydrivendesignprocessandimplementationtoensurecontextualisationandlocalownership
• Replicateandtesttheseindifferentcontextstogeneratemorerigorousevidenceontheireffectiveness
Flexible Funding Mechanisms
Replicateandtestonlargerscaleindiffering
contexts
• flexiblefundingmechanismswerefoundtobeapromisingapproachtosupportl/nnGOsself-directedcapacitybuildingefforts.Inaddition,flexiblefundswereessentialtotranslatingemergencypreparednessintotimelyandefficientresponseactivities,especiallyforl/nnGOs
• flexiblefundingmechanisms(bothforcapacitybuildingandresponseefforts)shouldbereplicatedinothersettings,andmorerigorouslyevaluated
Multi-stakeholder Platforms Modifyandre-test
• Theseplatformsweremoreeffectiveincontextswhereexistinghumanitarianstructuresweremorematureandwithsupportinggovernments.
• furthermodificationandtestingoftheapproachisneededtounderstandhowbesttoutilisethisstrategyandinwhichspecificsettings
Consortia Modifyandre-test• consortiathatbuildonexistingrelationships,commonvaluesandwaysofworking,andthataresmallerwithinclusionofl/nnGOsshouldbefavoured
Strengthening Networks
Modifyapproachandre-test
• Useknowledgeanddataoncountrynetworkstocreatemoretargetedapproachestostrengthennetworks;allowenoughtimetotestwhetherstrongernetworkscanleadtoincreasedemergencypreparedness
Advocacy and Policy
Mainstreamacrossprogramme
• Someadvocacyeffortswereimplementedbuttherewasnosystematicadvocacycomponentattheprogrammelevel.Itwasrecognizedthatadvocacyandpolicychangeareneededtoovercomebarrierstochangeandtoincreaseprogrammesustainability
• Ensureacohesiveprogramme-widestrategyrelatedtoadvocacyandpolicy
Learning Project Modifyandre-test
• Theapproachofhavinganindividualprojectdedicatedtolearningandgeneratingevidenceisuniquebutfindingsweremixedduetothelackofprogrammemonitoringandthelackofadirectlinktothemanagementteam.Alternativemodelsforanindependentlearningprojectcouldbeenvisionedandpotentiallyassessedinafutureprogramme
Evidence Generation Modifyapproach
• Prioritisequalityoverquantity
• Includeempiricalevidencegeneration
• Adaptevidenceuseandknowledgetranslationstrategies
Protection Mainstreamacrossprogramme
• Ensureacohesiveprogramme-widestrategyrelatedtoprotection
Inclusion of Prioritised Groups
Mainstreamacrossprogramme
• Ensureacohesiveprogramme-widestrategyoninclusionofprioritisedgroups(suchaswomen,theelderlyandpeoplewithdisabilities)
Localisation
contextualiseandreplicateandtestonlargerscaleindiffering
contexts
• Definelocalisation(whatitis,whichentitiesareconsideredlocal);andtestapproachestoenableInGOstotakeamoresupportiveroletowardsl/nnGOs,andbetteroperationaliselocalisationindifferentcontexts
22 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DO NOT PURSUE FURTHER
MODIfyAnDRE-TESTcOnTExTUAlISEAnDREPlIcATEInAnOTHERSETTInG
ORMAInSTREAMAcROSSPROGRAMME
Table3:RecommendedactionsforeachDEPPprogrammecomponent
Inaddition,anumberofbestpracticesrelatedtotheDEPPprogrammecomponentshavebeencompiledtoguidefutureprogrammedesignandimplementation.ThesebestpracticescanbefoundinAnnex11.
Specificconclusionsandrecommendationsrelatedtoeachevaluationquestioncanbefoundbelow.
1. RELEVANCE AND VALIDITY OF DESIGN CONCLUSION 1: WhiletheobjectivesandtargetsoftheDEPPwereappropriate,relevantandalignedwithDfIDpriorities,theDEPPdesignprocesssufferedfromnumerousweaknessesthathindered itspotentialfor impact.AnyfutureprogrammewouldrequireasignificantredesignbasedonthelessonslearnedandtheevidencegeneratedfromtheDEPP.forfuturedesign,programmeresultsthatarerealisticwithinathree-yeartimeframemustbeset,ortheprojecttimeframeshouldbeextendedtoallowformoretimetomeetlonger-termoutcomes.Amorelogical,coherentdesignprocess,which is locally led,drawsonneedsassessments,andusesparticipatoryapproachesshouldbeprioritisedtoensurethatprojectsarecontextualisedandbasedonexistingneedsatthecountrylevel.Thedesignprocessshouldadequatelyconsiderprojectalignmentandcomplementarityaswellasrisksattheprogrammelevel.Itshouldalsoensurethedevelopmentandimplementationofprogramme-levelsystemsandprocesses,suchasarobustM&Esystem,toensurecoursecorrectionandthatprojectsfunctionasaportfoliorather than as standalone projects. This includes elaborating a programme-level theory ofchange,definitionofkeyterms,cross-projectlinkages,andstreamlinedinclusionofgenderandprioritised groups alongwith appropriate considerationof cross-cutting themes. Emergencypreparednessandresponseprojectsshouldalsoconsiderurbancontextsand issuessuchaspovertyandfoodinsecuritywhichlieatthehumanitarian-developmentnexus.budgetsshouldincludehigherallocation fornPAcs,programmemanagement functionsandM&Eacrossallprojects.
RECOMMENDATION 1.1: [DEPP PROGRAMME BOARD AND DFID] The design process should be re-envisioned to consider alternative programme models and governance structures. To increase the local relevance and effectiveness of theprogramme, the design process should be locally driven from the outset (i.e., led bylocalstakeholders),andincludesufficienttimeandfundsforprojectdesign,localisation,consortiadevelopmentandcollaboration.
Potential Model 1: The process could involve several stages: After the overallprogrammatic business case and theory of change are developed, global goalscould be established and local andnational ownership prioritised.A programme-level focal point could be appointed in each country to conduct contextualisedneedsassessmentswhichwould feed into thedevelopmentof theportfolio-levelstrategy.Theproject designprocesswouldoccur at the country level, under theoversightofthefocalpointsandtheProgrammeboard.Thisapproachwoulddrivethe localisation agenda and allow for country-based organisations to determinethemostneededinitiativeswithinthegoalsoftheoveralltheoryofchangeaswellasriskassessment.Itwouldhelpalignprojectstowardcommonprogrammegoals,
23 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
identifycomplementaryelementsofdifferentprojects,mainactorsandthosethatneedtobebolsteredaswellasmoreriskyprojectsworthyofsupport.Itwouldalsoensuresufficient investment ineachcountry inordertoaccomplishthetheoryofchange,andwouldenablemorecomprehensiveassessmentsofrisks.Also,areturnoninvestmentassessmentcouldbeconductedtoinformthefinalinvestment.focalpointscouldworkwitharegionalorcountryM&EandlearningAdvisortodevelopandimplementprogramme-andproject-levelM&Esystems.Asaprogramme-levelactor, the focalpointwouldhelpbuildcohesivenessandenhance thevisibilityofDEPPasaunifiedprogrammeineachcountry.
Potential Model 2: Alternatively, consider a multi-phase programme in which aseriesofpilotinterventionsaredevelopedandimplementedonasmallscalewithstrong but targeted M&E systems integrated to evidence changes and provideaccountabilitymechanisms.Successfulpilotinterventionscouldbescaledupduringthesubsequentphase,buildingonthelessonscapturedduringthepilot.Astrongprogramme- and project-levelM&E system during the scale up phasewould beintegraltomeasuringchangeandensureregulartrackingofoutputstomodifytheprogrammewhenneeded. Inaddition, this typeofmulti-phaseprogrammecouldincludeapre-pilotphasewhereseedfundingisprovidedtodeveloppromisingideas,toconceptualiseprojectsandtobuildconsortiaaroundtheprojects.
RECOMMENDATION 1.2: [DEPP PROGRAMME BOARD AND DFID] The programme portfolio should be balanced in terms of risk, types of activities, size of projects, geographic scope and cost. To maximise impact, the link between emergency preparedness and response should be better articulated; preparedness programmes should strongly consider embedding flexible funding mechanisms in order to facilitate contribution to humanitarian response by local actors. Theportfolioshouldincludeamixofmoreestablished ideasandprojects, those thatareconsidered riskierandwitha higher likelihood of failure, aswell as innovative elements. Prioritising humanitarianneed,examiningcountrydisasterandemergencyprofiles,aswellasaligningwithoverallprogramme strategy during country selection process would ensure a more optimal,balancedsetoffocuscountries.forafundofthissize,concentratingonasmallergroupofprojects(atleastinanyscaleupphaseofamulti-phaseprogramme),andpotentiallyfeweroverallcountriesmayhelptoensurethatinvestmentsarenotspreadtoothinandthatthereisbothsufficientinvestmentandalargeenoughnumberofprojects ineachcountry to reach a criticalmass and achieve systems-level changes.A complementaryandstrategicmixofactivitiesattheprogrammelevelandespeciallyatthecountryleveliswarranted tomaximise impacts;but this tooneeds tobe focused.forexample, theDEPP’sopenapproachledtotoomanydifferenttypesofcapacitybuilding,toogreatanemphasisonindividualcapacitybuilding,andlackofconsensusondesiredeffectsateachlevel.Instead,amoredeliberateandevidence-basedglobalcapacitybuildingstrategyisneeded,withmorebalancebetweenlevelsofaction(individual,organisational,systems),aswellasconsiderationofstructuralbarrierstochangethatmighthinderbehaviourchangeor operationalising organisation-level changes. Inmany cases advocacy, policy changeor other system-based approachesmay be critical to easing someof these obstacles;integratingaprogramme-wideadvocacyorpolicycomponentshouldbefurtherexplored.Emergencypreparednessmustnotbethoughtofinisolationandmechanismstoembedlinkagestohumanitarianresponse(includingresponsefunding)areneeded.
24 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
RECOMMENDATION 1.3: [PROGRAMME BOARD AND DFID] Portfolio-level harmonisation should be prioritised, including mainstreaming of gender and inclusion of prioritised groups. Toensurealignmentoftheprojectsandacohesiveprogramme-wideapproach,portfolioobjectives,aswellaskeyterms(e.g.,collaboration,preparednessandlocalisation),andapproaches(e.g.,typesofcapacitybuildingefforts)shouldbefullydefined,and thedefinitionsstandardisedacross theprogramme.cross-cutting themesthatareprioritisedbythedonorand/orprogrammeandprojectstakeholders, suchasgenderandaddressingVAWG,mustbegivenadequateattentionandembeddedacrossthe programme. Inclusion of prioritised groups and gender considerations are equallyimportantandshouldbemainstreamed.
RECOMMENDATION 1.4: [PROGRAMME BOARD AND DFID] Ensure objectives are realistic and feasible within the programme’s time frame. Develop programmeswithrealistic objectives that are feasiblewithin a three-year time frame.Thesemost likelywould focus on outputs and shorter-term outcomes. Alternatively, consider creatinglonger (i.e., five-yearor longer)programmetimelines if longer-termoutcomesmustbeincluded.WhilelongerprogrammetimelinesmaynotcorrespondtocurrentDfIDfundingcycles,amulti-phaseprogrammecouldbeaneffectivesolutionthatiscompatiblewithfundingtermsaswellasactivitiesaimedatlonger-termimpact.
RECOMMENDATION 1.5: [PROGRAMME BOARD AND DFID] Ensure M&E processes are integrated within the programme design from the outset and that there is an appropriate balance between M&E and learning activities. Itisimperativethataprogramme-levellogicalframeworkbeinplacepriortoimplementationofanyprogrammeor project activities.Development of this framework should occur in parallelwith theprogrammeandwithinputfromlocalstakeholders.Ideally,indicatorsshouldbestreamlinedacross theprogramme,andshouldcontainacoresetofprogramme indicatorsagainstwhichallprojects report,withpotentiallyaseriesofadditionalsite-specificorproject-specificindicatorswhererelevant.IncorporatequantitativeVfMindicatorswithinthecoresetofindicatorstoensureVfMandefficiencycanbetrackedacrosstheprogramme.TheM&Esystemshouldenablethecollectionandreportingofdatadisaggregatedbysexandotherprioritisedgroups. Inat leastoneprogrammefocuscountry,considerprioritisinga randomised controlled trial (RcT) or other rigorous evaluation design to evaluateprogrammeeffectiveness.Thiswouldrequireinvolvingtheevaluationteamintheinitialstagesoftheprogrammedesignprocesstoensurealignmentoftheevaluationquestions,randomisationoftheinterventionsites,andadequatebaselinedatacollection.Thiswouldallowfortestingoftheeffectivenessoftargetedcomponentsoftheprogrammeaswellastheprogrammeoverall,isolatingtheroleofdifferentcontextualfactors.Technologicalsolutions suchasdashboardswith real-timedatavisualisationsof key indicators couldhelpenhanceroutinemonitoring,andallowformorerapidcoursecorrection.SufficienttrainingforprogrammeandprojectfocalpointswithrespecttotheM&Esystemmustbeprovidedandappropriatemechanismstoensurehigherqualitydatamustbeinplace.learningis importantbutrequiresawell-designedM&Esysteminordertobeofmostvalue.
RECOMMENDATION 1.6: [DFID] Re-examine the development of the business case, as well as its timing and its content, and broaden definitions of disasters, emergency preparedness and response.foranyfutureprogrammeatthescaleoftheDEPP,tofacilitateprojectdesign,thebusinesscaseshouldprovidetimelyandsufficientdetailssuchasfocuscountries, the criteria for country selection, a set of definitions of key concepts (suchas collaboration, institutional arrangements, emergency preparedness and localisation),andaconcisesummaryof theDfIDpolicyon inclusionofprioritised targetgroups inprogrammeandprojectdesign.furtherconsiderationsforhowtothinkaboutemergencypreparednessinurbancontextsandconflict-affectedsettingsareneeded.considerations
25 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
forissuessuchaspovertyandfoodinsecuritywhichlieattheintersectionofdevelopmentandhumanitariansectorsshouldbeconsideredwhendesigningemergencypreparednessandresponseprojectsdependingonthecontext.
2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS
CONCLUSION 2: DEPPinterventionswereoverwhelminglyfoundtoberelevantandappropriateandweredemonstrated tofill importantgaps thathavenotbeenpreviouslywelladdressed(surveillance of emergencies in urban settings, conflict-sensitive emergency preparedness,inclusionofaginganddisability,etc).However,start-updelayslimitedprojectimplementationperiodsandreducedpotentialimpact.Optimisingthedesignprocessasdescribedaboveandstreamliningadministrativeandcontractualprocesswouldensuresmoothandtimelyprojectstart-upandminimiseimplementationdelays.Attheprogrammelevel,therewasnoevidenceof improved knowledge on core humanitarian competencies, inclusion, or emergencypreparedness due to the DEPP but significant changes in organisational capacity andpreparedness among DEPP beneficiaries occurred, in particular among local organisations.furthermore,DEPPorganisationscontributedtoimprovedresponseinseveralcontexts(ThePhilippines,KenyaandEthiopia).Agreaterimpactcouldlikelyhavebeenachievedwithalongerimplementationperiod,andwitheasieraccesstoemergencyresponsefunding.Thereisscopeforcertainprogrammecomponentstobereplicatedorscaledupormodifiedandfurthertested.
Conclusion 2a: Collaboration and Consortia
functioning of consortiawas suboptimal and contributed to implementationdelays and slow decision-making. A smaller, more strategic set of consortiamembers, with more consistency between UK and in-country members foreach projectwould streamline project delivery and should be considered infutureprogrammes. Inclusionofmorel/nnGOswithin consortiamighthelpshiftpowerbalancesandcontributetolocalisation.collaborationswithexternalstakeholdersincludinggovernmentandUn,thoughessential,weredifficultandlessonslearnedshouldinformfutureprogramming.IntwosequentialdiarrhoealoutbreaksinKenya,theconsortiamodelandcollaborationwiththegovernmentenabled coordinated effective response activities that improved with eachsubsequentemergency.
Conclusion 2b: Capacity Building
Overall individual training targets at the programme level were exceededsuggesting successful implementation of DEPP capacity building activities.However,thesedataincludedindividualswhoparticipatedincapacitybuildingeffortswithminimal exposure such aswebinars and forums.Accessibility totrainingremaineduneven.Thetargetofincluding50%womentraineesacrosstheprogrammewasnotmet,andsometrainingswereonlyavailableinEnglish.Structuralbarriers,suchasorganisationalandmanagementarrangements,wereakeybarriertoapplyingandintegratingknowledgegainedfromDEPPcapacitybuildingeffortsat theorganisational levelandshouldbeaddressed in futureprogrammes. Multi-pronged capacity building efforts that include practicalapproachestosustainandapplyknowledgewereperceivedtobemosteffective,butquantitativedatashownoevidenceofknowledgechange.Organisational-levelcapacitybuildingeffortswhereseveralstrategieswerecombinedandtailoredtothespecificgapsofeachorganisationwereperceivedtobeeffective,andevidenceofincreasedorganisationalcapacityoflocalorganisationswasdemonstratedinanumberofsettings.
26 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Conclusion 2c: Evidence Generation and Learning
Generationoflearningandevidencewashighintermsofquantity,butevidencewasmainlyanecdotalandlackinginscientificrigor.Arguablytoomuchemphasiswasplacedongenerationoflearningproductscomparedtoroutinemonitoring.Differentialreportingbyprojectsagainsttheprogramme-levellogicalframework(once itwas finalised and fully implemented inAugust 2017) occurred, andtherewasevidenceofweakqualityassurance.Inaddition,therewasalackofdisaggregateddatacollectedwithrespect to importantsubgroups.Sharingoflearningdidoccurbut there is noevidence that this translated tobehaviourchange.learningeventswereausefulapproachtosharelearning,howeveranincreasedfocusonthedisseminationofevidenceattheseeventswouldhavebeenvaluable.ThelearningPlatformwasanimportantresource(thatnowservesaspartof theDEPP’s institutionalmemory)but itwasnotusedconsistentlyacrossprojectsandfeaturedvaryingamountsofevidenceandlearningsharedbyeachproject.Suchplatformsshouldbemoreheavilypromotedtoincreaseuse by in-country partners, represent resources from all projects and ensureavailabilityofresourcesinrelevantlanguages.TheDEPPlearningProjecthadbeenoriginallyconceptualised to takeon theM&Efunctionsof theDEPP inadditiontolearning.Whileauniqueidea,thismodeldidnotworkwell–M&Efunctionsandresponsibilitywouldhavebeenbetterplacedattheprogrammemanagementlevel.Placingthesefunctionsasoneprojectalongsidetheotherswithlimitedauthorityledtodiffusionofaccountabilityandresponsibility.Asastandalonelearningproject,ithashadmixedresults,butcouldbere-envisionedforfutureprogrammes.
Conclusion 2d: Early Warning Systems
Early warning systems are important as they provide (in theory) accurate,predictive and timely data to support emergency preparedness. However,findingswithrespecttoimplementationofearlywarningsystemsaremixed.Sixdiverseearlywarningsystemsweredevelopedandfivewerefullyoperationalatthetimeofthisreport.However,becauseofimplementationdelays,thelengthoftimethesesystemswerefunctionalwasverylimited.Severalofthesystemshavebeenwellintegratedintoappropriatestructurescontributingtoincreasedlikelihood of sustainability and impact. Only two systems (UEWEA, PHEPGambella)havebeenusedinanemergencyresponse.UEWEAinparticularwasshowntobeaneffectivesystemthatwasabletodetectseveralemergenciesthatwouldhaveotherwisegoneunnoticedininformalsettlementsaroundnairobiandthiscontributedtoimprovedtimelyandeffectiveresponse.Thissystemhadbeenoperationalthelongest,andalsoestablishedaconcreteexitstrategy,withthelocalcountygovernmenttakingovermanagementofthesystem.Thisprojectbenefittedfromalonger“effective”timelinecomparedtotheotherprojectsasittookforwardworkfromapreviousproject15thatconceptualisedthesystem,developedandtestedtheindicatorsandestablishedgovernmentcollaborations.Withoutthatexistingworktobuildon,UEWEAwouldlikelyhavefacedsimilarchallengesastherestofthecohortintermsofestablishingafunctionalsystemwiththeprogrammeperiod.
15 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).UrbanEarlyWarningEarlyActionfinalEvaluationReport,p1.
27 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
Conclusion 2e: Emergency Preparedness and Response
There are some emerging examples in several countries of perceivedimprovementsinemergencypreparednessattheorganisationalandcommunitylevel.DocumentreviewhasdemonstratedthattheDEPPhascontributedtoatleast42responsesin11countriesandthereareseveralself-reportedexamplesofstrengthened,morelocalisedresponse.
RECOMMENDATION 2.1 [DEPP PROGRAMME BOARD] The Learning Project and its placement should be re-examined, with its role, responsibilities, objectives and accountability mechanisms clearly defined at the outset of any future programme. Programme-level M&E should sit within programme-level management, but the casecould bemade to test a differently structured standalone learning project in a futureprogramme.furtherconsiderationstoensureabalancebetweenquantityandqualityoflearningandevidencegenerated,includingtheburdenandtimerequirementsplacedonprojects,andthetimingofM&Eactivitiestobeginbeforeprojectsstartimplementingareneeded.
RECOMMENDATION 2.2 [DEPP PROGRAMME BOARD] Project-level M&E systems should directly feed into programme-level systems (via a common, core set of indicators), and should be in place prior to implementation with timely reporting to permit agile course correction. StrongerM&Esystems includingcollectionofempiricaldatawouldsupport projects in accessing additional funding andyield better quality projectswitha greater likelihood of impact. Accountability mechanisms, including accountability tobeneficiaries,shouldbestrengthened.
RECOMMENDATION 2.3 [DEPP PROGRAMME BOARD] To increase the effectiveness of capacity building interventions, emphasis should be placed on quality not on quantity. Attheindividuallevel,approachesthatbuildskillsandreinforcelearning,andcombineseveralstrategies,shouldbeprioritised.Attheorganisationallevel,approachesthataretailoredtoidentifiedgapsandaimtostrengthensystemsandprocessesshouldbe emphasised. flexible funding mechanisms providing small or in some cases micrograntsforcapacitybuildingwereonesuccessfulapproachtoempowerlocalorganisationsto build their capacity and should be considered in future programmes. Systems-levelcapacitystrengtheningshouldalsobefurtherpursued.furtherattentiontothebalanceofactivitiesacrossthesedifferentlevelsandthedesiredeffectateachlevelwillbeneededto maximise impacts. Structural barriers need to be addressed, and capacity buildinginterventionsshouldbebettertailoredtospecificcontexts.
RECOMMENDATION 2.4 [DEPP PROGRAMME BOARD] Development of early warning systems should be taken forward within future programmes but only when sufficient implementation periods are available. Athree-yeartimeframewasinsufficienttodesignandoperationaliseanearlywarningsystem;morerealistictimelinesareneededtoensurecompletionofthesystemandallowingtimetotestthesystemwithinemergencyresponses (or simulations). community members and government officials should beactivelyinvolvedindesigningthesurveillancesystems,beneficiarycriteria,andresponsepackagedetails.consensusonindicatorsandearlywarningsystemsshouldbereachedin collaboration with community and government stakeholders. Early and sustainedengagementshouldbeemphasisedtofostertrustandstrengthenpartnerships.
RECOMMENDATION 2.5 [DEPP PROGRAMME BOARD] Future programmes must consider emergency preparedness and response on a continuum with more attention paid to the response component. The programme should strategise on how best toenableimprovedresponsethroughemergencypreparednessactivitiesandensuresuch
28 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
mechanisms are embedded within the programme, rather than expecting responsecontributiontooccurorganicallyasanaturalby-productofpreparednessactivities.Thiscouldtakevariousforms.flexibleresponsefundscouldbebuiltintotheprogramme(andpotentially the individual projects) to provide a rapidmechanism for local programmebeneficiariestoimplementresponseactivities.Asthisapproachattheprojectlevelhasworkedwell inseveralcontexts itshouldbeviewedasapromisingstrategyworthyofreplicationinmoresettingsandtestedasaprogramme-levelcomponent.Alternatively,orinaddition,moreformallinkswithexistingemergencyresponsefunds(suchastheSTARTfund)couldbepursued.Inaddition,furtherthinkingaroundemergencypreparednessinconflict-affectedsettingsandinurbancontextsisneeded.
RECOMMENDATION 2.6 [DEPP PROGRAMME BOARD] To leverage the global momentum around localisation of humanitarian response and build on the contribution of DEPP in advancing this agenda, more nuanced considerations on localisation are needed. This includes articulating a clear definition ofwhat localisationmeans,whichentitiesareconsidered local andwhat the implicationsof thesedefinitionsare fromapractical standpoint. Strategically, laying out and potentially testing approacheswhichcouldenableInGOstotakeamoresupportiverole,andbetteroperationaliselocalisation(movefromtheoreticaltoactual).Organisationalpoliciesaroundlocalisationcouldbeonetarget,butunderstandinghowtooperationalisethepoliciesintoactualprocesseswouldbeanimportantelement.Additionally,thehumanitarianlandscapeindifferentsettingsanditsrelationshiptolocalisationneedtobeconsideredtoensurethatappropriate,contextuallyrelevantstrategiestosupportlocalisationareimplemented.finally,localisationfromthedonorperspectiveshouldbefurtherexploredtounderstandstrategieswhichmightenabledonorstomoreeasilysupportlocalcounterpartswithinthecontextofriskaversionandduediligenceprocesses.
3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS CONCLUSION 3: collaborationishighlyvaluedandDEPP’sconsortiaapproachwasuniversallydeemedtobethemostappropriate,preferreddeliverymechanismforhumanitariancapacitybuildingandemergencypreparednessandresponseactivities.Therewassomeevidencethatthe consortia model enabled coordinated, timelier response in some settings. Globally,effectivenessofconsortiacouldbe increased infutureprogrammesthroughasmaller,morestrategic set of consortia members, improved communication, opportunities for bettercoordination of activities, and provision of sufficient resources for strategic and organisedcollaborations. Inclusion of more l/nnGOs within the consortia could contribute to thelocalisationagenda.networkanalysesdemonstratedevidenceofstrengthenedhumanitarianresponse networks in the Philippines, Kenya and Ethiopia, while the network in Myanmarremained unchanged.A range of different types of emergency preparedness and responsenetworksexistinthesecountries,fromhighlyisolated(Myanmar)tohighlydistributed,locallyled(ThePhilippines).Inanyfutureprogrammeaimingtostrengthennetworks,thesedifferencesinthesize,scopeandprioritiesofnetworksacrosscountriesshouldbeappropriatelyconsideredduringthebothdesignand implementation.Humanitarian landscapesandexistingnetworksacrossfocalcountriesshouldbeconsidered,particularlyinthedesignoftheprojectsandattheprogrammeleveltoleverageexistingcollaborationsandlocalpartnerships.Targetingofnetworkmembers for capacity building and evidence sharing should be more strategic and includerelevantkeyinfluencers.
RECOMMENDATION 3.1 [DEPP PROGRAMME BOARD] In future programmes, refine and optimise the consortium model building on lessons from the DEPP, ensuring context-specific considerations. Workingeffectivelywithcollaborativestructuresinthe
29 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
futurewillrequirebetterunderstandingofwhattypesofconsortiaareappropriateandeffectivewithin different contexts. Overarching factorswhich should be incorporatedinprogrammesof thisnature include supportingcollaborative structures thatbuildonexistingsuccessfulrelationshipsandthatworkwithasmallernumberofmembers,andthatworktowardsstrategiccollaborations.Toincreasethespeedofdecision-makingandeffectiveness of consortia, streamlined contracting and reporting processes should beestablished.Ensuringl/nnGOsareincludedinconsortia,andassessmentofstrategiestoaddressriskaversionamongdonorstosupportingsuchconsortiaareneeded.
RECOMMENDATION 3.2 [DEPP PROGRAMME BOARD] Network assessments could better inform targeted strategies to enhance collaboration and to effectively strengthen emergency preparedness and response networks. country-specificnetworkmappingandassessmentsshouldbeundertaken,whenappropriate,tofacilitatetheuseofmoretargetedapproachestobuildingnetworksandpartnerships,suchasthroughtheidentificationandtargetingofkeyinfluencers.Deeperexplorationonthelevelofinfluenceandroleofdifferentorganisations(i.e.,asresourcehubs,knowledgebrokersetc.)withinthenetworkinfocalcountrieswouldbeextremelyvaluabletoinformstrategiesaroundwhichorganisationstotargetandhow.Theseassessmentscouldalsoenableoptimisationofcollaborationwithlocalandnationalactors.forexample,localisationeffortscouldberefinedinthePhilippineswherenationalnGOsarealreadyverystronglyinvolvedwithinemergencypreparednessandresponsenetworks.
4. EFFICIENCY AND VALUE FOR MONEY (VFM) CONCLUSION 4:Goodindicatorsofcosteconomywereevident,butinsomecasesprojectbudgetsweretooleanwithinsufficientresourcesforconsortiamanagementandcollaborativeactivities.Attheportfoliolevel,resourcesdedicatedtoportfoliomanagementandcollaboration,M&EandtonPAcswereinsufficientwithrespecttodesiredprogrammeobjectives.Infutureprogrammes,morestrategicallocationoffundsshouldbeundertaken.Inaddition,systemsandresources for governance, strategy, portfoliomanagement, decision-making and consortiumarrangementsshouldberestructuredtoimproveefficiency.Adaptivemanagementprocessespermittedcoursecorrectionand revisionsofprojectplans to reflectchangingcontexts, andVfMreportingimprovedoverthecourseoftheprogramme.futureprogrammesshouldbetteradoptandstandardiseVfM indicators inorder tocollectdata for internalprojectpurposes,cross-projectcomparisonsandgenerallearning.TherewerenostrongVfMfindingstosuggestthat emergency preparedness improved the efficiency of humanitarian responsewithin theprogrammeduration.
RECOMMENDATION 4.1 [DEPP PROGRAMME BOARD] Streamline contractual processes, management decisions and flow of funds. Whilerecognisingtheimportanceofsubcontractingprocessesandorganisations’internalprocedures,futureprogrammingshould takesteps tominimiseandstreamlinecontractualprocesses.forexample,pre-agreementlettersofcommitmentcouldhelpclarifysomecontractualissuespriortothestart of projects.Theprogrammemanagement could alsoprovide a template for sub-contracts and consider setting a standardtimeline for contract turnaround. If projectsare structuredwith a global grant holder and in-country host or partner, streamliningmanagementdecisionsandflowoffundssothatthesameorganisationserves inbothroleswheneverpossiblecouldbeconsidered.
30 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
RECOMMENDATION 4.2 [DEPP PROGRAMME BOARD AND PROJECTS] Adopt and standardise VFM indicators at programme and project levels and implement routine reporting. DfIDguidanceonVfMinhumanitarianprogrammesorotherrelevantVfMguidanceshouldbeusedtoenhanceVfMmonitoringandreportinginfutureprogrammes.AdoptingVfMindicatorsattheprogrammelevel,aswellastheprojectlevelwillensurethatVfMcanbecloselytrackedandanalysedatbothlevels.
RECOMMENDATION 4.3 [DEPP PROGRAMME BOARD] In a future programme, strategically allocate funds to achieve programme objectives, with greater funds designated for programme management costs, M&E and NPACs. foralargeportfolio,programmemanagementcostsshouldbeintherangeofaround10%,toenablesufficientresourcestoundertakeitskeyfunctionsofmanagement,M&Eandstrategictasks.nPAcscanbeupto20%andashighas25%,butshouldbeconsideredrealisticallytoensurethatin-countryandbackofficecostsarecovered.WithrespecttoM&E,ensuresufficientresources are available at the both the programme and project levels to efficientlyimplementbothroutinemonitoringandmorerigorousevaluations.
5. SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOODOF IMPACT OF THE PROGRAMME CONCLUSION 5: Insomesettings,DEPPhascontributedtostrengtheningnationalemergencypreparedness systems but typically on a small geographic scale, and by individual projectconsortia.TherehavebeenseveralconcreteexampleswhereDEPPhasimpactedgovernmentpoliciesorsystemsor increasedpoliticalcommitment.Stakeholdersnotedashift inthewayorganisationsconsider,addressandinvolvelocalactorsandcommunities,andattributeditasoneofDEPP’smostsignificantcontributions.However,therewasalackofadetailedstrategyfortheinclusionofgenderandprioritisedtargetgroupsattheprogrammelevelresultinginafragmentedprogrammewideapproachandunevenresults. Inaddition,cross-cuttingthemessuchasaddressingVAWGwerenotadequately integrated intotheprogramme.Ratherthanonlyincludingoneortwoprojectswithaninclusionandprotectionfocuswithinaportfolio,amoreintegratedapproachinvolvingmainstreamingacrosstheprogrammeshouldbeprioritised.Thisshouldentaildisseminationofdetailedexpectationsandguidelinesoninclusionofgenderandother prioritised groups and reporting requirements during thedesignphase to ensurethatprojectsaredevelopedaccordingly.Aswell,consistencyinimplementationofgenderandinclusionconsiderationswithregularmonitoringshouldbeestablished.Inadditiontoinclusion,sustainability planning was weak and was not clearly planned or documented at both theprogrammeandprojectlevels.ThisfailureseemstohavebeenlinkedtoexpectationsaroundthepossibilityofreceivingadditionalfundsfromDfIDforasecondphaseoftheDEPP,asarticulatedinthebusinesscase,combinedwithlackofguidancefromtheprogrammeonsustainability.Inanyfutureprogramme,moredeliberate,strategicsustainabilityplanningisneededatalllevels,beginning during the design phase, andwith transparency around the possibility of furtherfunding.Attheprogrammelevel,thereissomeevidencethatbenefitshavebecomeembedded–withexamplesofsystemandpolicychangeaswellasstrengthenednationalpreparednesssystems, albeit on a small geographic scale. However, the potential for longer-term effectsisdependentontheextenttowhichdifferentcomponentsareabletocontinuebeyondthecloseoftheprogramme.futureprogrammesshouldprioritiselongerimplementationperiodstoincreaselikelihoodofimpact,andincorporategovernmentcollaboration,systems-levelchange,policyoradvocacycomponents,involvementofbeneficiariesandgoodVfMinordertopromotesustainability.
31 SUMMATIVEPHASEREPORT-ExEcUTIVESUMMARy
RECOMMENDATION 5.1 [DEPP PROGRAMME BOARD AND PROJECTS] Develop a strategy for inclusion of gender and prioritised target groups at programme level, ensure projects adopt an approach which aligns with this strategy and monitor implementation at all levels. Inthegovernancecriteria,considerbroadeningthegenderstatementtoastatementoninclusionofprioritisedtargetgroups.Developguidelinesforprojectsonrequirementsforthisstatement.Thiscould include linkstokeydocumentsonbestpracticesandthelawandpoliciesgoverninginclusionofprioritisedtargetgroupsin development programming and project design. Ensure gender considerations areconsistentlyimplementedandreported.
RECOMMENDATION 5.2: [DEPP PROGRAMME AND PROJECTS] Within any future programme, integrate deliberate, strategic sustainability planning that takes a more holistic view on sustainability going beyond simply securing funding streams. Programme and project sustainability plans should be developed as early as possible,preferablyinthedesignphaseandupdatedregularly.However,forriskierprojectswithahighpossibilityoffailure,sustainabilityplanningdoesnotmakesenseandisnotagooduseofresourcesuntilresultscanbedemonstrated.futureprogrammesshouldincreasethe likelihoodofsustainabilityand longer-term impactsbybuildingonpreviouseffortsand existing partnerships/consortia, working with governments (when appropriate),incorporating a policy or advocacy element, developing sustainable outputs such astools, guidelines or systems, increasing cross-project linkages, fostering sustainablerelationships that can continue beyond theDEPP, improving beneficiary engagement,andstrengtheningfeedbackmechanismsforthecommunity.GoodVfMshouldalsobeprioritisedtomaximisesustainability.Programmevisibilityatthecountrylevelshouldbemaximised through increasedadvocacy initiativesand thedevelopmentof streamlinedcommunicationwithexternalstakeholders.
32 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
INTRODUCTION& CONTEXT
cHAPTER1
33 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
Eachyeartheseverityandcomplexityofnaturalandman-madedisasterscontinuestoincrease,making recovery and reconstruction efforts more challenging and costly. foreseeing thistrend, theHumanitarianEmergencyResponseReview(HERR)16urgedtheUKgovernment’sDepartment for International Development (DIfD) to invest in innovative approaches todisasterpreparednessinordertobuildlocalresilience,stating“Ifwearetomeetthechallengesahead,wehavetobe‘aheadofthecurve’ratherthanalwaysbehind;preparingfordisasters,aswellasreactingtothem.”Atthattime,lessthan5%ofallhumanitarianfundingwasinvestedindisasterpreparedness.This constituted less than1%ofOfficialDevelopmentAssistance(ODA),despitethefactthat“earlyresponseisfarmorecosteffectivethanlatehumanitarianresponse” and better preparedness is critical to more timely response17. The HERR alsohighlighted that “the levelofprofessionalism in thehumanitariansectorneeds tobe raisedthrough better investement in skills and training.” In response to theHERR,DfID createdthebusinesscase18fortheDisastersandEmergenciesPreparednessProgramme(DEPP)(SeeAnnex1)andallocated£40millionoffundingtostrengthenskillsandcapacitytoimprovethequalityandspeedofhumanitarianresponse incountriesthatareatriskofnaturaldisastersoremergencies.TheDEPPaimedtoincreasepreparednessatthelocal,regionalandnationallevelstoenablecountriestobebetterequippedtorespondtodisaster.19Itfocusedonbuildingcapacityof“nationalactorswhoareusuallythefirstonthesceneofadisaster”,anapproachnowreferredtoaslocalisation(seetheboxesbelow).
TheDEPPwasdeliveredbytwonon-governmentalOrganisation(nGO)consortia,theSTARTnetwork,(receiving£27million),andthecommunicatingwithDisasterAffectedcommunities(cDAc-n)network, (receiving £3million), to implement 14 capacity building projects. Theremaining£10millionwasreservedforaninnovationwindowwhichwasdeveloped,implemented,andmanagedseparatelywithalatertimeline.The14DEPPprojectswereimplementedinoneormoreof1020prioritycountries:SouthSudan,Kenya,Ethiopia,Mozambique,DemocraticRepublicofthecongo(DRc),Jordan,Pakistan,bangladesh,MyanmarandthePhilippines,witheachprojectoperatinginsomebutnotallcountries.Onlyoneproject(thelearningProject)wasimplementedinall10countrieswhilefiveprojects(PublicHealthEmergenciesPreparedness(PHEP)inGambella,ImprovedEarlyWarningEarlyActions(EWEA)inEthiopia,StrengtheningEmergencyPreparednessSystemsinMyanmar,UrbanEarlyWarningEarlyAction(UEWEAinKenya),financialEnablersinthePhilippines)wereimplementedinonlyonecountryeach.The14DEPPprojects,theirconsortiamembers,locationsandbudgetsarepresentedinTable1.1.
THE DISASTERS AND EMERGENCIES PREPAREDNESS PROGRAMME (DEPP)• Investment: £40 million
• Location: 10 countries
• Duration: 3 years
• Number of preparedness projects: 14 humanitarian capacity building projects
• Overall objective: To improve quality and speed of humanitarian response in countries that are at risk of natural disasters or humanitarian emergencies
Key approaches: Capacity building of local and national humanitarian staff and communities, early warning system development, supporting collaborative action and strengthening networks
16 HumanitarianEmergencyResponseReview(HERR),https://www.gov.uk/government/publications/hu-manitarian-emergency-response-review.
17 businesscaseInterventionSummary:DisastersandEmergenciesPreparednessProgramme,DfID.18 Ibid.19 ibid.20 Indonesiawasinitiallyselectedasan11thDEPPfocalcountry,butnoselectedprojectsplannedtoimple-
mentactivitiesthere.
34 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
BET
TER
D
IALO
GU
E,
BET
TER
IN
FOR
MA
TIO
N,
BET
TER
A
CTI
ON
THE
LEA
RN
ING
PR
OJE
CT
FIN
AN
CIA
LEN
AB
LER
S
SHIF
TIN
G
THE
POW
ER
TALE
NT
DEV
ELO
PMEN
T
TRA
NSF
OR
MIN
G
SUR
GE
CA
PAC
ITY
PRO
TEC
TIO
N
IN P
RAC
TIC
E
ALE
RT
AG
E &
DIS
AB
ILIT
Y
CA
PAC
ITY
PR
OG
RAM
ME
(AD
CA
P)LI
NK
ING
PR
EPA
RED
NES
S,
RES
PON
SE A
ND
R
ESIL
IEN
CE
PUB
LIC
HEA
LTH
EM
ERG
ENC
IES
PREP
AR
EDN
ESS
IN G
AM
BEL
LA
STR
ENG
THEN
ING
EM
ERG
ENCY
PR
EPA
RED
NES
S SY
STEM
S IN
M
YAN
MA
R
IMPR
OV
ED E
AR
LY
WA
RN
ING
- EA
RLY
A
CTI
ON
ETH
IOPI
A
UR
BA
N E
AR
LY
WA
RN
ING
, EA
RLY
AC
TIO
N
3-YEARPROGRAMME
£40MILLIONFUND 14
PROJECTS10COUNTRIES
45ORGANISATIONS
KENYA
PAKISTAN
BANGLADESH
PHILIPPINES
DRC
MOZAMBIQUE
ETHIOPIA
MYANMAR
SOUTH SUDAN
JORDAN
The Disasters and Emergencies Preparedness Programme uses collective networks to improve the quality and speed of humanitarian response in countries at risk of natural disaster or con� ict related humanitarian emergencies.
£500
,000
£784
,256
£954
,698
£675
,000 £3
,343
,375
£1,6
55,9
20
£3,0
00,0
00
£4,8
76,6
36
£5,9
85,0
87
£2,4
82,8
24
£804
,634
£1,9
87,0
00
£1,0
43,6
73
£978
,987
35 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
EMERGENCY PREPAREDNESS
Emergency preparedness refers to “the abil-ity of governments, professional response organisations, communities and individuals to anticipate and respond effectively to the impact of likely, imminent or current hazards, events or conditions. It means put-ting in place mechanisms which will allow national authorities and relief organisations to be aware of risks and deploy staff and resources quickly once a crisis strikes.”21
Emergency preparedness focuses narrow-ly on improving response to disasters, but it belongs to a wider set of activities to reduce the likelihood and impact of disas-ters on people’s lives called ‘Disaster Risk Reduction’ (DRR). These include activities focused on prevention, mitigation and response to humanitarian emergencies.22
The IASC Common Framework for Pre-paredness23 describes the following com-ponents of emergency preparedness:
• Hazard and risk analysis and early warning systems
• Institutional and legislative frame-works
• Resource allocation and funding• Coordination• Information management and com-
munication• Preparedness and contingency / re-
sponse planning• Training and exercises• Emergency services, standby arrange-
ments and prepositioning
The DEPP aimed to focus on those areas related to ‘people and systems’, in partic-ular hazard, risk and early warning; infor-mation management and communication; contingency/preparedness and response planning; and training and exercises.24
LOCALISATIONThere is no globally accepted definition of ‘localisation’ of humanitarian aid. The defini-tion suggested by Trocaire and Group URD is holistic and appears to align very closely to the DEPP approach to localisation: “Aid localisation is a collective process involving different stakeholders that aims to return local actors, whether civilsociety organisa-tions or local public institutions, to the centre of the humanitarian system with a greater role in humanitarian response. It can takea number of forms: more equitable partner-ships between international and local actors, increasedand “as direct as possible” funding for local organisations, and a more central role in aid coordination. Underpinning this is the question of power. Localisation requires a shift in power relations between actors, both in terms of strategic decision- making and control of resources.” 25
The OECD states that supporting local humanitarian responders contributes to the following:26
• Early response and access (including to small scale crises below the threshold for international aid)
• Improved acceptance of humanitarian aid (such as in conflict-affected areas)
• Cost effectiveness• Links with development• Increasing accountability
The growing calls to increase support for local humanitarian responders is also re-flected in a number of policy commitments including:
• Good Humanitarian Donorship, Princi-ple 827
• The Grand Bargain, Workstream 228
• The Sendai Framework for Disaster Risk Reduction29
• The 2030 Agenda for Sustainable Development30
21 https://www.humanitarianresponse.info/en/coordination/preparedness/what-preparedness.22 ibid.23 Inter-AgencyStandingcommittee(IASc)commonframeworkforPreparedness,2013.24 businesscaseInterventionSummary:DisastersandEmergenciesPreparednessProgramme,DfID.25 Morethanthemoney-localisationinPractice,GroupeURD,Trocaire,2017.26 OEcD,localisingtheResponse,2017.27 GoodHumanitarianDonorshipPrinciples,https://www.ghdinitiative.org/ghd/gns/principles-good-practice-
of-ghd/principles-good-practice-ghd.html.28 TheGrandbargain,https://interagencystandingcommittee.org/grand-bargain-hosted-iasc.29 SendaiframeworkforDisasterRiskReduction,https://www.unisdr.org/we/coordinate/sendai-framework.30 The2030AgendaforSustainableDevelopment,https://sustainabledevelopment.un.org/post2015/transformingourworld.
36 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DEPP PROJECTPROJECT
CONSORTIA MEMBERS
COUNTRIES** DESCRIPTIONBUDGET
(£)TA
LEN
T D
EVEL
OPM
ENT
(TD
)SavethechildrenUK(lead),OxfamGb,Re-liefInternational,cHS
Alliance
bangladesh,DRc,Ethiopia,Jordanand
Kenya,lebanon
Toproducehigh-qualityprofessionalsatalllevelswhoarebetterequippedtotackletheissuessur-roundingcomplexemergencies,helpingtoensurethattherightpeopleareintherightplacedoingtherightthingstoassistdisaster-affectedcommu-
nities.
5,985,087
SHIF
TIN
G TH
E PO
WER
(STP
) ActionAidInternational(lead),cAfOD(lead),christianAid,Tearfund,concernWorldwide,
Oxfam
bangladesh,Pakistan,Ethio-pia,KenyaandDemocraticRepublicofcongo(DRc)
Tosupportlocalactorstotaketheirplacealong-sideinternationalactorsinordertocreateabalancedhumanitariansystemthatismorere-sponsiveandaccountabletodisaster-affected
communities.
4,876,637
THE
LEAR
NIN
G
PRO
JECT
*
ActionAgainstHungerAllDEPPcoun-
tries
Toevidencetheextenttowhichpreparingpeopleisaneffectiveandefficientapproachtodisastermanagementandbrokerinternalandexternalrelationshipsforlearningaboutwhatisandisnot
workingincapacityexchange.
3,343,375
BETT
ER D
IALO
GUE,
BE
TTER
IN
FORM
ATIO
N,
BETT
ER A
CTIO
N
(CD
AC-N
PRO
JECT
)
WorldVision(lead),cDAcnetwork,Thom-sonReutersfounda-tion,bbcMediaAction,
Internews
bangladesh,SouthSudan,ThePhilippines
Toensurethattwo-waycommunicationisapredi-cable,coordinatedandresourcedcomponentofhumanitarianresponseinordertocontributetoimprovementineffectivedeliveryofassistanceto
disasteraffectedcommunities.
3,000,000
TRAN
SFO
RMIN
G
SURG
E CA
PACI
TY
(TSC
)
ActionAidInternational(lead),Acf,christianAid,cAfOD,cARE,IMc,IslamicRelief,MuslimAid,Plan,Savethechildren,Tearfund
Pakistan,thePhilippines,RegionalHubinbangkok,Thailand
Tostrengthencivilsocietysurgecapacityatinter-national,regionalandlocallevels,contributingtoadiverseanddecentralisedthirdsectorpillarbetterabletocomplementexistingUnitednations,Redcrossandgovernmentstructuresinordertohelpcommunitiesincreaseresilience,reduceriskand
improvecrisisresponse.
2,482,824
ALER
T
HelpAgeInternational(lead),careinterna-tional,HandicapInter-national,IslamicReliefWorldwide,concernWorldwide,Oxfam,coventryUniversity
ThePhilippines,Pakistan,ban-gladesh,Kenya,Mozambique,Somalia,Haiti
Todevelopasystemofapproachesandtoolsfordisasterpreparednessthatincreasestheabilityoforganisationstorespondimmediately,effectively
andappropriatelywhenadisasterstrikes.
1,987,000
FIN
ANCI
AL
ENAB
LERS
(F
E) OxfamGb(lead),Tear-fund,christianAid
ThePhilippines
Totransferhumanitariancapacity,autonomyanddecision-makingtoorganisationsclosertopeopleaffectedbycrisis,asawayoffacilitatingmore
effectiveandappropriateaid.
1,637,745
AGE
AND
D
ISAB
ILIT
Y CA
PACI
TY
BUIL
DIN
G
(AD
CAP)
HelpAgeInternational(lead),cbM,Disaster-Ready.org,HandicapInternational,IfRc,
OxfordbrookesUniver-sity,RedRUK
Kenya,Pakistan
Toensureolderpeopleandpersonswithdisabilitybenefitfromimprovedaccesstoservices,asare-sultofrecognitionbyhumanitarianactorsoftheirspecificneedsandincreasedcapacityamongsthumanitarianactorstodeliverinclusive,accessible
andappropriateresponse.
1,045,159
LIN
KIN
G
PREP
ARED
NES
S RE
SPO
NSE
AN
D
RESI
LIEN
CE (L
PRR) christianAid(lead),
ActionAid,concern,HelpAgeInternational,King’scollegelondon,MuslimAid,Oxfam,Saf-erworld,WorldVision
bangladesh,Kenya,DRc,Pakistan,thePhilippines,colombia,Indonesia***,Myanmar
Todesignandrolloutprogrammingapproacheswhichstrengthentheresilienceofpeoplelivinginfragilestatesandbeneficiariesofhumanitarian
assistance.
1,002,964
37 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
STRE
NG
THEN
ING
EM
ERG
ENCY
PR
EPAR
EDN
ESS
SYST
EMS
IN
MYA
NM
AR (S
EPS)
christianAid(lead),DcA,RegionalIntegrat-edMulti-HazardEarlyWarningSystemforAfricaandAsia(RIMES)
Myanmar
Tostrengthenthelinksamongnational,subnation-alandlocal-levelpreparednessinMyanmar,withaprimaryfocusoncapacitatinglocalcommunitiesandstructurestoaccessinformationandlinkwiththeon-goingestablishmentofpreparednessand
earlywarningsystemsinthecountry.
925,698
PRO
TECT
ION
IN
PRAC
TICE Oxfam(lead),IRc,
WorldVisionInterna-tional
DRc,lebanon,Turkey,SouthSudan,Myan-mar,PakistanandthePhilip-
pines
ToenablenationalnGOstoimplementprotectionactionsindisasterandconflictresponses,devel-opnewtypesofpartnershipsandcollaborationsbetweenprotectionactorsandinfluencethe
internationalprotectionarchitecturesoitismoreinclusiveofnationalnGOs.
870,137
IMPR
OVE
D EA
RLY
WAR
NIN
G – E
ARLY
AC
TIO
NS T
O ST
REN
GTH
EN
DIS
ASTE
R PR
EPAR
EDN
ESS
IN ET
HIO
PIA
(EW
EA)
OxfamGbonbehalfofAfricacllimatechangeResilienceAlliance(lead),christianAid,nationalDisasterRiskManagementcommis-sion,nationalMeteoro-
logicalAgency
Ethiopia
Tocontributetoimprovedemergencyprepared-ness,timelinessandqualityofriskinformationandresponseactionsbycommunity,government(localtofederal)andnGOsinacoordinatedmanner
784,255
URB
AN E
ARLY
W
ARN
ING
, EAR
LY
ACTI
ON
(UEW
EA)
concernWorldwide(lead),Oxfam,KenyaRedcrossSociety
Kenya
Toimproveurbanearlyactionbyimprovingthealignmentoflocalandmunicipalgovernanceinsti-tutionsandresponseagenciestowardsidentifyingrelevanttriggersandimplementingearlyresponsebasedonthese,supportedbyanincreaseintheallocationoffinancialsupportfromkeydonorstowardsurbanEarlyWarningEarlyAction.
675,000
PUBL
IC H
EALT
H
EMER
GEN
CIES
PR
EPA
RED
NES
S IN
GA
MBE
LLA
(P
HEP
)
christianAid(lead),AmrefHealthAfrica,EthiopiaMinistryofHealth,nationalMete-orologyAgency
EthiopiaTostrengthenearlywarning,preparednessandpromptresponseofthehealthsectortopublic
healthemergencies.548,694
Table1.1:ProjectsintheDisastersandEmergenciesPreparednessProgramme31
*Formerly known as the Monitoring, Evaluation and Learning Project (MEL).
**Includes non-DEPP focal countries where projects were implemented (i.e., Lebanon, Somalia, Haiti, Colombia, Turkey).
***Indonesia was initially selected as an 11th DEPP focal country but none of the selected projects planned implementation in Indonesia. However, over the course of the programme, LPRR ultimately did implement some activities in this country.
With14individualprojects,theDEPPwasbroadinscopeandcomplexindesign.Eachprojectimplementedadifferentsetofactivities,andworkedwithindifferentstructures.Theprojectsalso targeted various levels of action (or beneficiaries) including individuals, communities,organisations and governments. The programme utilised a number of strategies includingcollaboration, innovation, scale, decentralisation, complementarity, learning and addressingcross-cuttingissues.ThefiveoverallobjectivesortargetedresultareasoftheDEPPasoutlinedinthebusinesscasearepresentedintheboxbelow32.Whileeachprojectworkedtowardsoneormoreoftheseresultsareas,asawholethecollectiveactionoftheprogrammeaimedtoaddressallfive.
31 TheSTARTnetwork.DisasterandEmergenciesPreparednessProgrammeInfographic.August2015.Accessedon18february2016athttp://www.start-network.org/wp-content/uploads/2015/08/DEPP-In-fographic.pdf.
32 InallcountriesexceptJordan,thecombinedprojectscoveredallfourDEPPobjectives.TalentDevelop-mentwastheonlyprojectimplementedinJordananddidnotcoverthe3rdand4thobjectives.
38 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DEPP RESULT AREAS1. Improved knowledge and understanding of individuals by sharing best practice for
humanitarian preparedness and response
2. Improved preparedness systems for early action with communities at risk of disasters
3. Increased number of coalitions, partnerships and networks which, working together, are able to address humanitarian needs in a wide variety of emergency situations
4. Improved institutional arrangements and policy environments so that national systems for humanitarian response and preparedness are better supported and more sustain-able
5. Strengthened evidence base for what works to help build humanitarian capacity at scale
A3-year external impact evaluationof theDEPPprogrammewas commissionedbyActioncontrelafaim(Acf)(seeToRinAnnex4),andconductedbyanevaluationteam(Annex2)attheHarvardHumanitarianInitiative(HHI).Thisreportpresentsthesummativeresultsoftheimpactevaluationincudingfindingsfromallphasesoftheevaluation.
DEPP THEORY OF CHANGE AND SIMPLIFIED CAUSAL CHAIN During theevaluation inceptionphase, theevaluation teamassessed theDEPPprogrammetheoryofchange(Annex3)presentedinthebusinesscase(Annex1).Theprogrammetheoryofchangewasfoundtobecomplexandnolongerrepresentedwellthesuiteofprojectsastheyhadevolvedovertime.AprocesswasundertakentocreateamoresimplifiedbutconceptuallyappropriaterepresentationoftheDEPPthatcouldserveasthebackboneoftheevaluationapproachandtoreportagainstduringeachevaluationphase.figure1.1illustratestheDEPPprogrammecausalchainthatwasdevelopedandthatwasusedtocreateaprogramme-levellogicalframeworkandsetofindicatorsfortheevaluation33.Thecausalchainillustratesthecausallinksbetweenthe14DEPPprojectsandtheiractivities,fourexpectedoutputareas(1.capacitybuilding,2.collaboration,3.learningand4.EarlyWarningSystemDevelopment),expectedoutcomesinboththeshortterm(improvedhumanitariancapacity,improvedpreparedness)andlongterm(increasedeffectivedeliveryofhumanitarianresponse).Hypothesiseddownstreamimpacts (mortality, morbidity, economic impact and recovery) are also depicted within thecausalchain,andbasedonstakeholderinterviewswerenotexpectedtobeachievedwithintheprogramme’s3-yearcycleandthuswereconsideredoutsideofthescopeoftheevaluation.Thecausalchainalsospecifiesthattheactivities,outputsandoutcomesareexpectedtooccurat the individual, organisational, community and government levels. This also captures thevarietyofDEPPbeneficiariesthatweretargetedateachoftheselevels(includingindividualhumanitarian staff, humanitarian organisations, communties, governments etc). This impactevaluationfocusedonlyonassessingoutputs,aswellasshorttolong-termoutcomesatthesedifferentlevels.
33 Page17oftheDEPPbusinesscasestated,“Throughacommissionedevaluation,wewilldeveloptheTheoryofchangeinmoredetailaspartoftheevaluationinceptionstage,andwewilltestourcoreassumptionsthroughtheDEPPevaluation.”TheevaluationteamrevisedtheTheoryofchangetoenableafeasibleevaluationdesignasmandatedintheinceptionphaseandthetermsofreference.TherevisedcausalchainwasapprovedbytheEvaluationSteeringcommitteeandthusbecamethebasisfortheevalu-ationdesign,includingtheevaluationframeworkandindicators.notethatwhiletherewereseveraldraftprogrammelogicalframeworksinplacein2016,thefinalversionwasnotapproveduntil2017.
39 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
figure1.1:SimplifiedDEPPprogrammecausalchain
DEPP PROGRAMME-LEVEL LOGICAL FRAMEWORK AfinalversionoftheDEPPprogramme-levellogicalframeworkwasapprovedbyDfIDin2017andfullyimplementedbyAugust2017,severalyearsafterthestartoftheprogramme.ThefinalversionofthelogicalframeworkispresentedinTable1.2below.Whiletherewereseveraldraftversionsoftheprogramme-levellogicalframeworkinplacebeforethen,theyrequiredsignificantstrengthening34.Therefore,withagreementfromDfID,becausetheexternalevaluationhadalreadybegunbefore thefinalisationof the logical framework, itdoesnot reportagainst it.However, it is important to note that the output, outcome and impact areas in this logicalframeworkarequitesimilartotheevaluationteam’soutputandoutcomeareasasdepictedinfigure1.1.35Theindicators,however,areunique(SeeTable2.1intheMethodologysectionforfurtherdetail).
34 forexample,DfID’sAnnualReviewoftheDEPP,March2018states:“Thereisstillmoreworktodoonthelogframe–includingfinalisingtheoutcomeindicatorsandincludinganoutputwhichtracksvalueformoney.”
35 Programmeoutput1correspondstoevaluationoutput1,programmeoutput2toevaluationoutput4,programmeoutput3toevaluationoutput2,programmeoutput4doesnotdirectlycorrespondwithanyevaluationoutputarea,programmeoutput5correspondstoevaluationoutput3,programmeoutput6doesnotcorrespondtoanyevaluationoutputarea,programmeoutcome1correspondswithevaluationoutcome1,andprogrammeimpact1correspondswithevaluationimpact1.Programmeoutputarea6correspondstotheinnovationwindowwhichisnotwithinthescopeofthisevaluation.
DEPPProjects
HumanitarianCapacity
Preparedness
EffectiveDelivery ofHumanitarianResponse
MortalityMorbidityEconomic ImpactRecovery
LEVELS OF ACTION
IndividualsCommunitiesOrganisationsGovernment
INPUTS &ACTIVITIES
OUTPUTS SHORT TERMOUTCOMES
LONG TERMOUTCOMES
IMPACT
Collaboration
CapacityBuilding
Early WarningSystem
Development
Learning
40 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
STATEMENT INDICATOR TARGET (MARCH 2018) ASSUMPTIONSO
UTP
UTS
1.Improvedknowledgeandunderstandingofindividualsbysharingbestpracticeofhumani-tarianpreparednessandresponse
1.1#ofDEPPhuman-itariancapacitybuildingprogrammes,countriesandstrategiesthatutilisearangeofimplementa-tionmethodstoimprovethecapacityofindivid-ualsattheinternational,nationalandsub-nationallevels
12humanitariancapacitybuildingprogrammesarerevised,scaledupordevelopedin12coun-tries,usingamixof6strategiesincluding:shortcourses,longcourses,trainingoftrainers,men-toring/coaching,e-learn-ingandwebinars
capacitybuildingac-tivitieswillincludeadiversityandinclusionapproach. Individualhumanitarianworkerscantransmitlearningintoactionwith-intheirorganisationsandwideroperatingenviron-ments. chosenapproaches,methodologies,andin-novationsarethecorrectonestobuildcapacityinanincreasinglycomplexhumanitarianecosystem. Shortcourse=lessthanonemonth longcourse=morethanonemonth
1.2#ofstaffacrossmul-tiplehumanitariandisci-plineswhoaccessqualityDEPPhumanitarianca-pacitybuildingactivitiesofvaryingdurationandqualificationonarangeofsubjects
Atleast4200individuals(50%male,50%female)haveaccessedqualityDEPPhumanitarianca-pacitybuildingactivitiesacross10subjectstovariouslevelsofaccred-itation(includingshortcourses,long,courses,e-learningandToT)
2.Improvedprepared-nesssystemsforearlyactionwithcommunitiesatriskofdisasters
2.1#ofcountrieswithpreparednesssystemsforcommunities,agen-ciesandgovernmentinstitutionsdevelopedorstrengthened,includingearlywarningsystems,preparednessplans,col-laborativesurgerosters
Preparednesssystemsinuseinatleast7countriesandatleastthreegov-ernmentinstitutions,andcapacitybuiltinrespon-sibleinstitutionsatcom-munityandnationallevelandtestedviasimulationinatleastonecountryorwherearesponseisrequired
Asystemsapproachtopreparednessissuitedtoidentifiedcountriesandpopulations,aswellmul-tipleformsofhazards. Onceequippedwithacoordinatedsystemstakeholdersareabletotakeearlyaction. Resultsexpectedfromthedeploymentofthesesystemswillhaveapos-itiveimpactonincentiv-isingpreparednessandearlyaction. note:Thelanguageusedforthisindicatorhasbeenchosenonpur-posetoclarifythatthepreparednesssystemsdevelopedbytheDEPParenotnationalpre-parednesssystems.Theorganisationsthatwillusethesepreparednesssystemswillincludelocal,regionalandnationalgovernmentbodies,In-GOs,nationalnGOs,andotherinternationalactors
2.2.Improvedprepared-nesssystemsforearlyactionwithcommunitiesatriskofdisasters
6systematiccommu-nicationapproaches(includingearlywarningbulletinsandsurveillancereports)operationalinatleast3countriesatmultiplelevelsandtestedviasimulationinatleastonecountryorwherearesponseisrequired
41 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
OU
TPU
TS
3.Increasednumberofcoalitions,partnershipsandnetworkswhichworkingtogether,areabletoaddresshuman-itarianneedsinawiderangeofemergencysit-uations
3.1formalpartnershipsareestablishedandmain-tainedwithawiderangeofnationalandinterna-tionalactors(bothwithinandoutsidetheDEPPconsortium)thatincreaseactionandcommitmenttoaddresshumanitarianneedsinemergencies
Partnershipsbetweenatleast200nationalandinternationalhumani-tarianactors(includingnationalnGOs,interna-tionalnGOs,academicinstitutions,governmentagencies,privatesectorcompanies,andRedcross/crescentSocieties)demonstrateincreasedactionandcommitmenttoaddresshumanitarianneedsinemergenciescollaboratively Atleast70%oftheseac-torscommittomaintain-ingatleastoneoftheirpartnershipsbeyondtheendoftheprogramme,inordertocontinueaddressinghumanitarianneedscollaboratively.
Modelsofcollaborationbeingtestedcontributetoimprovingeffective-nessinpreparednesswork. collaborativemecha-nismscreateddonotreinforcesystematicdisempowermentandexclusionofcrisisaf-fectedpopulations,firstresponders,localandnationalhumanitarianleadership. Platforms,forumsandcommunitiesofprac-ticeactivelyreflectonstrengthsandweakness-esofdifferentapproach-essothattheyremainrelevantmechanismsforthehumanitariansys-tem.Aspectsofexistingandnewmechanismsthatdon’tworkcanbeaddressed,andthereiswillingnesstochange.
3.2collaborativemech-anismscontributetoimprovedlearning,effi-ciencyandcoordinationofDEPPstakeholdersforemergencyprepared-ness,programmingandresponse
Analysisof140collabo-rationstoriesandprojectfeedbackindicatethatcollaborationhasledtoimprovedlearning,effi-ciencyandcoordination
4.Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhu-manitarianresponseandpreparednessarebettersupportedandmoresuitable
4.1numberofstrategies(includingactionplans,guidelinesandtools)coveringvariousthe-maticareasadoptedbynationalandinternationalactorsinarangeofcoun-triestoachievemoreappropriatehumanitarianresponseforcrisisaffect-edpopulations
13project-levelstrate-gies(includingorganisa-tionalactionplans,guide-lines/standardsandtools)around7thematicareasareimplementedacross11countries
Strategiesforadvocacyactivitiesarethecorrectonessuitedtothewiderhumanitariansector. Vulnerablegroupsandcrisisaffectedpopula-tionsvoicesareconsis-tentlyintegratedintoinfluencinghumanitarianpreparedness. Initiativesandinnovationareabletoinfluencepolicyandpracticeofor-ganisationsorthewiderhumanitariansector.
4.2DEPPadvocacyandinstitutionalchangeactivitiestargetcorehumanitarianthemessuchasageanddisability,protectionandcommuni-catingwithcommunities
8corehumanitarianthemestargetedthroughadvocacyandinstitution-alchangesactivities
42 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
OU
TPU
TS
5.Strengthenedevi-dencebaseforwhatworkstohelpbuildhu-manitariancapacityatscale
5.1EvidenceacrosstheDEPPisgeneratedandcapturedsystematicallyandcontributestofutureuptakeoflessonslearnedtohelpbuildhumanitari-ancapacityatscale
Atleast200learningresourcesthatpromotetheuptakeofbestprac-ticeandlessonslearnedfromtheDEPPareproducedandshared,coveringabroadrangeofpreparednesssubjectareasincludinghuman-itariancapacitybuild-ing,resiliencebuilding,preparednesssystems,humanitariansurge,local-isation,collaboration,ageanddisabilitycapacitybuildingandprotectionmainstreaming EveryDEPPprojectdemonstratesatleastoneexampleofsignif-icantadaptiveprojectmanagementbasedonrecommendationsfromDEPPlearningresourcesonhowtoimprovehu-manitarianprogramming.
currentproblemsandgapsintheevidenceonhumanitariancapacitybuilding/enhancementforpreparednesscanbeidentifiedandsolutionscanbefacilitatedwithintheprogramme. Thereiswillingnessinidentifiedcountriesandthewiderhumanitariansectortoacceptevidencegatheredtofeedbackintoarangeofcapaci-ty-building/enhancementeffortsandactivities. Itwillbepossibletoreferencelearningandevaluationstomeasurethedegreeofuptakebyprojectsandthewiderhumanitariansector. newandexistingDfIDpreparednesscapacitybuildingprogrammesdesignand/orimplemen-tationapproachesareinformedbytheevidencefromDEPP.
5.2DEPPMElprojectpromotesgeneration,sharinganduseofevi-denceandlearning
DEPPMElprojectpro-videsprojectsupporttoallother13DEPPproj-ects,co-creates13re-searchpieceswithDEPPpartners,managestheDEPPexternalevaluationinitssummativephaseanddelivers20learningeventsacross9countries
6.Humancenteredde-signapproachestestedandimplementedacrossfourinnovationslabs,leveragingnon-tradition-allocalactorstodeliverpreparednessapproach-eswhicharecommuni-ty-appropriate,leadingtomorepreparedandengagedcommunities
6.1communitiesareactivelyidentified,en-gagedandinvolvedinallstagesoftheinnovationsprocessandoutputsareassessedusingcommu-nity-basedevaluationmethods.
Relevant/usercommu-nitieshaveevaluatedlabactivitiesforrele-vanceandusefulness.Independentcommunityperceptionsurveydatagatheredacrossfourlabstocomplementcontex-tualisedparticipatorymethodsforcoherentcomparisonsandreport-ing.
communityengagementinthedesignprocessleadstogenerationofnewideasanddevelop-mentofmoreappropri-atepreparednessmech-anisms. communitiesaremoti-vatedtoremainengagedwiththeinnovationpro-cessbecauseofthevaluetheyattributetobeinginvolvedindevelopingmoreusefultools.
43 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
OU
TPU
TS
6.cont.
6.2DEPPlabsinthefourcountriestest,pilotandsupportinnovationsthroughaninnovationprocess,learningfromwhichiscuratedandsharedwiththebroadernetwork.
Atleast48projectssupportedandtestedacrossthefourlabs.Scaledinnovationsevaluatedandsharedacrossfourcountries.lessonslearnedaboutnon-traditionalsectorengagementandcon-textualisedapproachestocommunitycentereddesigninemergencypre-parednesssharedacrossthebroadernetwork.finalevaluationincludesexternalassessmentofsustainability,qualityandeffectivenessofcommu-nityledinnovationpro-cessesbyrelevantlocalinnovationsactors.
Someinnovationshavealreadybeenpioneered,whichcanbeidentifiedandscaledupmorequicklywithlabsupport. labsupportmechanismsincludingtraining,fund-ingandmulti-sectoralpartnershipsareade-quatetoprogressinnova-tionstoapointofgreaterviability. Sharinglessonslearnedwithnetworkmemberscontributestobehaviourchangewithinthesector.
OU
TCO
MES
1.IncreasedandstrengthenedemergencypreparednesscapacityinDEPPfocuscountries,focusingonstrengthen-inglocalhumanitariancapacityandchampi-oninglocalisationinamannerconsistentwiththeGrandbargainandtheWorldHumanitarianSummit
1.Proportionofre-cipientsofqualityhumanitariancapacitybuildingactivitiesthatdemonstrateimprovedknowledge,attitudesorbehavioursincorehu-manitarianareas
Outofsampletakenfrom13humanitari-ancapacitybuildingprogrammes,70%willdemonstrateincreasedknowledge,attitudesandbehaviours
Oneassumptionisthattrainingstaffwillresultinincreasedorganisationalpreparedness-wehavetriedtoaddressthisbyincludingoneindicatoraroundstaffknowl-edge(Indicator1)and1aroundnGOreportsofcapacitystrengthen(Indi-cator4) coordinationmecha-nismsandpreparednesssystemsarewellintegrat-edintoorganisationsandstaffreceiveadequatetraining. Strategiestoaccomplishinstitutionalchangearebothdiverse,flexibleandeasytointegrateintoexistingprogramming,withappropriatesupportandtrainingprovided.
44 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
OU
TCO
MES
1.cont.
2.DEPPcoordinationmechanismsandpre-parednesssystemspro-moteefficientresponsetoemergencies,withclearpoliciesandproce-duresandinstitutionalsupport
feedbackon100%ofpilotmechanismsandsystemsindicatesthat70%ofasampleofstakeholdersreportin-creasedpreparedness,andthatthesurroundingpoliciesandproceduresareadequate. AnalysisfromaDEPPreturnoninvestmentstudydemonstratesthatDEPPsystemspromoteefficientresponsetoemergencies.
3.DEPPagencies,withaparticularemphasisonlocalandnationalnGOs,implementstrategieswhichresultinimprovedprogrammingforhumani-tarianpreparedness
AllDEPPagenciesdemonstrateimprovedhumanitarianprogram-ming(includingpolicies,plansandlearning) l/nnGOssupportedbyDEPPreportmorein-creasedcapacitytopre-pareforandrespondtodisastersthansampleofl/nnGOsnotsupportedby DEPP
IMPA
CT
1.DEPPreducessuffer-inganddeliversbetterhumanitarianservices,throughimprovedpre-paredness,forpeoplearoundtheglobeaf-fectedbydisastersandemergencies
1.Reductioninthenum-berofliveslost,affectedcommunitiesandeco-nomiclossesduetoim-provedpreparednessin10DEPPfocuscountries
In2017thenumberofreporteddeaths,totalpeopleaffected,andeco-nomiclossesduetonat-uraldisastersin10DEPPcountriesisreducedfromthepreviousyear
MoredirectfundingtolocalnGOs,aswellasmorefundingintodisas-terpreparedness,willresultinfasterandmoreefficientresponseontheground.
Itisassumedthatforindicator1thatinlinewithWHSguidelines,allagencieswillself-reportannuallyagainstcommit-ments. Reductionsinoverallnumbersofliveslostandeconomicdamagesinthe10DEPPfocuscountriesareprimarilyduetohu-manitarianandgovern-mentinitiatives,althoughannualandseasonalfluc-tuationsintheseverityandnumberofdisastersalsoplayarole.
45 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
2.Humanitarianactors,includingnGOsandgov-ernments,investmoreinemergencypreparednessprogrammingtoensurethatresponsetocrisesarebetteranticipatedandplanned
Thetotalamountofglobalofficialhuman-itarianassistancere-portedascommittedtodisasterpreventionandpreparednessin2017isUSD2500m
note:forimpactindi-cator2thedataisonagloballevel-itwasnotpossibletodisaggregatebyDEPPcountry.Also,theDAccodesforre-portingfundscommittedtodisasterpreventionandpreparednesswillchangeintheupcomingyear,sothatmayaffectdataconsistencyacrossmilestones. note:ThedataforIndi-cator1isavailabledisag-gregatedbycountry.
Table1.2:DEPPprogramme-levellogicalframework(fullyimplementedbyAugust2017)
THE DEPP EXTERNAL EVALUATION Purpose
Thethree-yearexternal,independentevaluationincludedprocessandperformanceaswellasimpactevaluations.Theoverallaimsoftheindependentevaluationare:
1) to improve programme effectiveness and enhance learning
2) to assess the extent to which the DEPP overall has provided an efficient and effective approach to strengthening response capacity.
TheresearchdesignfortheevaluationwasguidedbyasetofquestionsandmethodologicalapproachasoutlinedintheTermsofReference(ToR)(Annex4).TheevaluationusedamixedmethodsdesigninordertocaptureacomprehensivepictureoftheDEPP’seffectiveness.Theevaluationwasconductedinfourphases(inception,formative,interim,andsummativephases).Thedetailedmethodologycanbefoundinchapter2.The evaluation focuses at the programme level, and it is important to note that individualevaluationsofthe14individualprojectsarenotwithintheremitofthisevaluation.Theevaluationservestoprovideaccountabilityandlearningforprogramme,projectandexternalstakeholders.ThesestakeholdersincludeDEPPprojectconsortiamembers,DEPPbeneficiaries,programmeandproject-levelstaff,aswellasexternalstakeholdersfromhumanitariannGOs,internationalorganisations,theUn,andgovernments.
Theevaluation’sinceptionphasewascompletedinJanuary201636.TheinceptionphasereportreviewedtheDEPPtheoryofchange,andpresentedtherefinedevaluationmethodologyanditslimitationsaswellastheprocessforselectingcountriesfortargetedintensivedatacollectionduringtheformative,interim,andsummativephasesoftheevaluation.Theevaluation’sformativephase focusedon relevanceofprogrammeoutputs and theefficiency andeffectivenessofprogramme delivery. The formative phase report, completed in May 201737, assessed theprogrammedesignstage(2011toearly2015),theprogrammeimplementationstage(October
36 EvaluationoftheDisastersandEmergenciesPreparedness(DEPP)Programme:InceptionPhaseReport,feb18,2016.PhamP,SharmaV,Scott,J,Gleason,K,Gibbons,n.
37 Sharma,V,HémonoR,ScottJ,Gibbonsn,ShahV,GleasonK,yangn,HavilandM,PhamP.(2017)TheDisastersandEmergenciesPreparednessProgrammeEvaluation:formativePhaseReport,May3,2017.HarvardHumanitarianInitiative.
46 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
2014 toMarch2017),andpresentedbaselinequantitative indicators related toknowledge,attitudes,exposuretotheDEPP,andemergencypreparedness.Theinterimphasereportwascompleted inOctober2017andpresentedperceptionsonprogramme implementationandoutputs(October2014-October2017),short-termoutcomechanges,aswellasprogresssincetheformativephaseevaluationreportandadoptionofformativephaserecommendations.TheinterimreportalsoidentifiedcurrentgapsandemergingsuccessesinprogrammeimplementationandprovidedguidancetoDEPPstakeholdersabouthowtoimprovethepotentialeffectivenessoftheDEPPandevidencetoinformfuturesimilarprogrammes.
Thissummativephasereportpresentsthecumulativefindingsfromallphasesoftheexternalevaluation from november 2015 to May 2018. The aim of the summative phase was toassessintermediateoutcomesandpreliminaryindicatorsoflikelihoodofimpact.findingsareorganisedinfivemainsectionsthatcorrespondtothekeyevaluationquestions:1)RelevanceandValidityofDesign;2)RelevanceandEffectivenessoftheInterventions;3)EffectivenessofManagementArrangements(inrelationtocollaboration);4)EfficiencyandValueforMoney(VfM);5)SustainabilityoftheInterventionandlikelihoodofImpactoftheProgramme.
Evaluation Criteria and Questions
As per the evaluationToR, the evaluationwas designed based on five criteriawhichwereadapted from the Development Assistance committee’s (DAc) principles for EvaluatingDevelopment Assistance: relevance and fulfilment of objectives, efficiency, effectiveness,impactandsustainability38.Theevaluationgatheredandanalyseddatainordertoanswerthefivekeyevaluationquestionsandassociatedsub-questionspresentedbelow.MoredetailedDEPP-specificevaluationcriteriaandquestionsareincludedinAnnex5.
EVALUATION QUESTION 1: RELEVANCE AND VALIDITY OF DESIGN1. To what extent are the objectives of the programme intervention consistent with
stakeholders’ requirements and the programme design logical and coherent?
a. Has the programme targeted the right people in the right places?
b. To what extent does the programme design (theory of change) support the projects’ design (logical framework)?
c. In what ways was the programme design process participatory? Were project beneficiaries adequately engaged before, during and after?
d. To what extent was the programme design logical and coherent?
i. Were the objectives of the programme clear, realistic and likely to be achieved within the established time schedule and with the allocated resources (including human resources)?
e. Have prioritised target groups (people with disabilities, older people) and gender aspects been taken into consideration in the programme design?
38 TheDAcPrinciplesfortheEvaluationofDevelopmentAssistance,OEcD(1991),GlossaryofTermsUsedinEvaluation,in‘MethodsandProceduresinAidEvaluation’,OEcD(1986),andtheGlossaryofEvaluationandResultsbasedManagement(RbM)Terms,OEcD(2000).
47 SUMMATIVEPHASEREPORT-InTRODUcTIOnAnDcOnTExT
EVALUATION QUESTION 2: RELEVANCE AND EFFECTIVENESS OF INTERVENTIONS2. In what ways have DEPP capacity building programmes strengthened emergency
preparedness and response capacity amongst participants?
a. What delivery mechanisms are working effectively and why?
b. To what extent is DEPP contributing to greater preparedness and response among local organisations and communities?
i. Has local capacity to respond to disasters changed since the start of DEPP? If yes, how has it changed? If not, why not?
ii. Has DEPP led to improved knowledge and understanding of best practices relating to disaster and emergency preparedness and re-sponse? If yes, in what ways?
EVALUATION QUESTION 3: EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS3. To what extent was the programme’s theory proven that capacity development is
more effective when undertaken as a multi-agency collaborative approach?
a. Is the collaborative approach of multi-stakeholder platforms an effective delivery mechanism?
b. Focusing on coalitions, partnerships, and connectedness – what can be said about the effects of strengthened networks?
c. What have been the main patterns of collaboration, and the benefits and disadvantages of informal vs. formal collaboration?
d. What unique contribution did collaborative relationships and “multi-stake-holder platforms” make towards deepening cross-programme learning?
EVALUATION QUESTION 4: EFFICIENCY AND VALUE FOR MONEY (VFM)4. How economically have resources and inputs (funds, expertise and time) been con-
verted to results? To what extent does preparedness improve the efficiency of hu-manitarian response?
a. Have resources (funds, human resources, time, expertise, etc.) been allocat-ed strategically to achieve the programme objectives?
b. Have resources been used efficiently? In general, do the results achieved justify the costs? Could the same results be attained with fewer resources?
c. Have the programme funds and activities been delivered in a timely man-ner?
48 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
EVALUATION QUESTION 5: SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOOD OF IMPACT OF THE PROGRAMME
5. To what extent and in what ways have the benefits of the programme become embedded?
a. What contribution has the programme made in strengthening national preparedness systems?
b. Has the programme taken into consideration prioritised target groups (people with disabilities, older people, women, children and youth)? What contribution has the programme made in strengthening inclusion of target groups and gender aspects at the level of national and local institutions?
c. In what ways has DEPP influenced institutional and policy environ-ments?
d. What is perceived to be the be the most significant change attributed to the DEPP and why?
Evaluation Timeline
Theexternalevaluationhadfourphases(SeeTable1.3):aninceptionphaseduringwhichtheevaluation framework andmethodologywere designed; a formative phase to evaluate theimplementation of theDEPP by assessing the relevance of outputs and the efficiency andeffectivenessofprogrammedelivery; an interimphase to assess short-termoutcomes; andasummativephasetoassess intermediateoutcomesandpreliminary indicatorsof likelihoodof impact. Amodified timelinewas approved by the evaluation steering committee due todelaysincontractinglocalresearchpartners,securinglocalapprovalsandchallengeswithdatacollectionduetoareasofinsecurity.
PHASE OBJECTIVE DURATION ACTUAL END DATE*
Inception finaliseevaluationmethodology 8Months May2016*
formative Assessrelevanceofprogrammeoutputsandefficiencyandeffectivenessofdelivery 12Months May2017*
InterimAssessshort-termoutcomesdeliveredbytheprogrammeandreflectonprogrammemanage-mentprocess
6Months October2018
Summative Assessintermediateoutcomesandpreliminaryindicatorsoflikelihoodofimpact 7Months May2018
Table1.3:Externalevaluationphases,objectivesandtimeline
* Revised end date
49 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
figure2:DEPPProjectDescriptions
EVALUATIONMETHODOLOGY
cHAPTER2
50 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
EVALUATION DESIGN EVALUATION METHODOLOGY
Theevaluationteam’sapproachusedamixedmethodsdesignwithquantitativeandqualitativedatacollection,includingorganisational-levelassessmentsandcommunity-levelassessments,aneconomicVfMassessment,analysisofhumanitarianpreparednessandresponsenetworks,humanitarianresponseassessmentsandcasestudies,observationandsitevisits,anddocumentreview.findingsfromthesedatasourcesweretriangulatedtoaddressthefivemainevaluationquestionsandtheirsub-questions.AspertheevaluationToR,theevaluationframeworkwasdevelopedbasedontheDAcPrinciplesforEvaluatingDevelopmentAssistance,withtheaimto improve future aid policies and programmes and to provide a basis for accountability.39
TheframeworkwhichoutlinestheevaluationquestionsandtheircorrespondingindicatorsispresentedinTable2.1below.Datawerecollectedthroughoutthefourphasesoftheevaluationagainsttheevaluationframework,andincludedallindicatorsincludedbelow.Theseindicatorsarepresentedthroughoutthisreporttoanswerthecorrespondingevaluationquestions40.
EVALUATION DESIGN MEASURES
Main Evaluation Question
Secondary Evaluation Questions
DEPP Programme Objective / Element of Theory of Change
Indicators
1. RELEVANCE AND VALIDITY OF DESIGN
To what extent are the objectives of the programme intervention consistent with stakeholders’ requirements and the programme design logical and coherent?
•Hastheprogrammetargetedtherightpeopleintherightplaces? •Towhatextentdoestheprogrammedesign(theoryofchange)supporttheprojects’design(logicalframework)? •Inwhatwayswastheprogrammedesignprocessparticipatory?Wereprojectbeneficiariesadequatelyengagedbefore,duringandafter? •Towhatextentwastheprogrammedesignlogicalandcoherent? •Weretheobjectivesoftheprogrammeclear,realisticandlikelytobeachievedwithintheestablishedtimescheduleandwiththeallocatedresources(includinghumanresources)? •Haveprioritisedtargetgroups(peoplewithdisabilities,olderpeople)andgenderaspectsbeentakenintoconsid-erationintheprogrammedesign?
ProblemStatement(Theoryofchange):Inthecontextofrisingneed,insuffi-cientpreparednesssystemsand[few]peoplewiththerightknowledge,attitudesandskillsareavailabletoensureeffectivedeliveryofassistance,particu-larlyatthenationallevel.
•numberofcapacitydevelopmentprogrammesdevelopedandimplemented •numberofnationalstaffandcounter-partstrained •numberofplatformsestablished •numberoflearningeventsheld •numberofreports/casestudiesdevelopedandshared •Increasedadvocacytostrengthenpolicyandpractice •numberofprojectswithgovernmentinvolvement •numberofhazard,risk,andearlywarn-ingsystemsdeveloped/improved
•StakeholderperspectivesonDEPPdesignprocess,objectives,resourcingandprioritisedgroups
39 TheDAcPrinciplesfortheEvaluationofDevelopmentAssistance,OEcD(1991),GlossaryofTermsUsedinEvaluation,in‘MethodsandProceduresinAidEvaluation’,OEcD(1986),andtheGlossaryofEvaluationandResultsbasedManagement(RbM)Terms,OEcD(2000).
40 forexample,evaluationquestiontwointheframeworkbelowisassociatedwithindicatorsonknowledge(improvedknowledgeoncorehumanitariancompetencies,disasterpreparedness,humanitarianresponse,onageing,disabilityandVAWG).Thesearepresentedinchapter4,thechaptercorrespondingtothisevaluationquestion(seefigure4.6).
51 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS
In what ways have DEPP capacity building programmes strengthened response capacity amongst participants?
•Whatdeliverymechanismsareworkingeffectivelyandwhy? •TowhatextentisDEPPcontributingtogreaterpreparednessandresponseamonglocalorganisationsandcommu-nities? •HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,howhasitchanged?Ifnot,whynot? •HasDEPPledtoimprovedknowledgeandunderstandingofbestpracticesrelatingtodisasterandemergencypreparednessandresponse?Ifyes,inwhatways?
Objective 1: Toimproveknowledgeandunderstandingofpeopleinthesystemregardingbestpracticeforhumanitarianpreparednessandresponse.
Stakeholder perspectives on effective delivery mechanisms
Improved knowledge and skills of national staff and counterparts in best practices for humanitarian preparedness and response •Improvedknowledgeoncorehumanitar-iancompetencies,disasterpreparedness,humanitarianresponse•Improvedknowledgeonageing,disabili-tyandVAWG
Improved emergency preparedness among organisations and communities•Improvedperceivedlevelofprepared-nessofhumanitarianstafftorespondtoadisaster•Improvedperceivedlevelofprepared-nessoforganisationstorespondtoadisaster•Improvedperceivedlevelofprepared-nessofcommunitytorespondtodisaster•Improvedcommunityaccesstoearlywarningalerts•improvedactualleveloforganisationalpreparednesstorespondtodisasters(Organisationalpreparednessscore)•Improvedactuallevelofcommunitypreparednesstorespondtodisasters(Vulnerabilityindex/communitypreparednessscore)
# and types of disasters organisations responded to in the previous 12 months, and types of response activities
Improved delivery of humanitarian assis-tance during a humanitarian response •Improvedperceptionsofhumanitarianstaffontheirownperformanceduringapreviousresponse•Improvedperceptionsofhumanitarianstaffontheirownabilitytorespondtoadisasterinthefuture•Improvedperceptionsonorganisations’performanceduringapreviousresponse•Improvedperceptionsonorganisations’abilitytorespondtoadisasterinthefuture
•Improvedsatisfactionofcommunitymemberswithpreviousresponsetoadisaster•Increasedspeedofresponse •Increasedcollaborationduringresponse
52 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS
To what extent was the programme’s the-ory that capacity development is more effective when undertaken as a multi-agency collaborative approach proven?
•Isthe‘collaborative’approachofmulti-stakeholderplatformsaneffectivedeliverymechanism? •focusingoncoalitions,partnershipsandconnectedness–whatcanbesaidabouttheeffectsofstrengthenednetworks? •Whathavebeenthemainpatternsofcollaboration,andthebenefitsanddisadvantagesofinformalversusformalcollaboration? •Whatuniquecontributiondidcollaborativerelationshipsand‘multi-stakeholderplatforms’maketowarddeepeningcross-programmelearning?
Objective 2: Toincreasethenumberofcoalitionsandpartnershipsdeveloped.
Increased number of networks and coalitions formed •Increasednetworksize •Increasedlinkspernode •Increasednetworkdensity •numberofseconddegree,thirddegreeconnections
Increased collaboration and strength of connections•Increasednumberofcollaborationareas •Increasedfrequencyofcollaboration •Increasedstrengthofconnections•Stakeholderperceptionsoncollaboration•Typesofbenefits/disadvantagesofinformalandformalcollaboration
Emerging evidence base for what works in building humanitarian capacity •numberandtypesoflearningdocu-mentsgeneratedandshared•SharinganduseofevidenceandlessonsacrossDEPPprojectsandexternally
4. EFFICIENCY AND VALUE FOR MONEY (VFM)
How, econom-ically, have resources/ inputs (funds, expertise, time etc.) been converted to results? To what extent does preparedness improve the efficiency of humanitarian response?
•Haveresources(funds,humanresources,time,expertise,etc.)beenallocatedstrategicallytoachievetheprogrammeobjectives? •Haveresourcesbeenusedefficiently?Ingeneral,dotheresultsachievedjustifythecosts?couldthesameresultsbeattainedwithfewerresources? •Haveprogrammefundsandactivitiesbeendeliveredinatimelymanner?
All DEPP Objec-tives.
4 ‘E’ Framework •Economy •Effectiveness •Efficiency •Equity•Percentofbudgetallocationtopro-grammemanagementcosts,M&E,nPAcs•Timelinessofdeliveryoffundsandactivities•costperactivityandresultsareas•StakeholderperspectivesonVfM,deliveryoffunds
53 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
5. SUSTAINABILITY AND LIKELIHOOD OF IMPACT
To what extent and in what ways have the benefits of the pro-gramme become embedded?
•Whatcontributionhasthepro-grammemadeinstrengtheningnationalpreparednesssystems? •Hastheprogrammetakenintoconsiderationprioritisedtargetgroups(peoplewithdisabilities,olderpeople,women,childrenandyouth)?Whatcontributionhastheprogrammemadeinstrengtheninginclusionoftargetgroupsandgenderaspectsatthelevelofnationalandlocalinstitutions? •InwhatwayshasDEPPinfluencedinstitutionalandpolicyenvironments? •WhatisperceivedtobethemostsignificantchangeattributedtoDEPP,andwhy?
Objective 3: Toimproveinstitutionalandpolicyenviron-mentsforbuildinghumanitariancapacity. Objective 4: Toimproveprepared-nesssystemsforcommunitiesatriskofdisaster. Objective 5: Strengthenedevidencebaseforwhatworkstohelpbuildhumanitariancapacityatscale.
Improved national preparedness systems•Stakeholderperceptionsonstrength-enednationalpreparednesssystems
Inclusion of vulnerable groups•Percentageoforganisationswithinclusionpolicies•Perceivedinclusionduringorganisationalpreparednessactivities•Perceivedinclusionduringorganisationalresponseactivities•Perceivedinclusionincommunitypreparednessplans
Improved institutional environment/ arrangements •Perceivedindividualinfluencewithorganisation•Perceivedinfluenceoforganisationsoninstitutionalandpolicyenvironment•Perceivedinfluenceoforganisationonnationalpreparednesssystems•Inclusivenessoforganisationalpolicytowardsl/nnGOs•changeinorganisationalpolicyonworkingwithl/nnGOs
Most significant change •Stakeholderperspectivesonmostsignif-icantchangeattributedtotheDEPP
Table2.1:Evaluationframeworkcontainingkeyevaluationquestions,correspondingDEPPprogrammeobjectiveandindicators
Duetolimitedresources,theevaluationwasdesignedtoconductin-persondatacollectioninasubsetofDEPPcountries,andremotedatacollectionamongtheremainingDEPPcountries.Theevaluationmethodologythereforeincluded:
1) Anintensivesetofevaluationactivitieswithon-sitequantitativeandqualitativedatacollection in a subset of DEPP focus countries (Kenya, Ethiopia,Myanmar and thePhilippines)
2) Aminimumsetofevaluationactivitiesincludingadeskreviewandremotequantitativeand qualitiatve data collection.Qualitative datawere collected across all tenDEPPcountries, and quantitative data were collected in five non-intensive set countries(SouthSudan,bangladesh,Pakistan,Jordan,andDRc).TherewerenoDEPPactivitiesinMozambiqueatthetimeofdatacollection,thereforedatawerenotcollectedaspartoftheminimumset.
The intensiveandminimumsetofevaluationactivities,aswellas theotherdatacollectionsourcesandtheirrespectivestudylocations,aredescribedinmoredetailbelow.
Datawerecollectedattwotimepointsthatwereroughly12monthsapart–timepoint1(T1)duringtheformativephaseoftheevaluation,andtimepoint2 (T2) inthe interimphase, inordertoassesschangesovertime.AtruebaselinewasnotpossiblegiventhattheevaluationteamwasselectedandcontractedafterthestartoftheDEPPprojects.Assuch,datacollectioninT1,which serves as thefirst datapoint in the evaluation and a comparison forT2datacollection,tookplaceafterDEPPactivitieshadalreadybegunformostprojects.Thepotential
54 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
effectsofthislackofatruebaselineassessmentaredescribedingreaterdetailinthelimitationssection.Datafromrelevantcomparisongroupswerecollectedwherepossible.
COUNTRY SELECTION FOR INTENSIVE SET EVALUATION ACTIVITIES
OfthetenDEPPcountries,theexternalevaluationteaminitiallyaimedtoselectthreecountriesfortargeteddatacollectionaspartoftheintensivesetevaluationactivities.Duringthecountryselectionprocess,anumberofkeycriteriawere identifiedandapplied.These included theselectioncriteriaasdefinedinTable2.2.Toensurediversityofgeographicregions,typesofhumanitariancrises,inclusionofbothcDAcandSTARTnetworkprojectsandarangeoftypesof targetedbeneficiaries includingcommunity-leveldirectbeneficiaries, theevaluationteamoptedtoexpandthenumberofcountriesselectedfromthreetofive.Additionalcriteriathatfactoredintothedecision-makingprocessincludedensuringtheinclusionofseveralcountrieswithahighlikelihoodofanupcomingdisasteroremergencytopermitpotentialassessmentofoutcomesrelatedtorealemergencysituations.Inaddition,theteamsoughttoincludecountrieswithvaryingnumbersofDEPPprojectsanddifferentprojectcombinationstopermitstudyofwhetherthesefactorsinfluencednetworkformationandoutcomechanges.
COUNTRY SELECTION CRITERIA CONSIDERATION
Geographic Region EnsuredadequateregionalrepresentationofDEPPprojects.Type of Humanitarian Crisis
consideredtypesofhumanitariancrisesincludingarmedconflict,naturaldisastersfamine,andepidemics.
Inclusion of CDAC and START Network Projects
IncludedcDAcandSTARTnetworkprojects.
DEPP Funds Allocated to Country
DEPPfundsallocatedtoeachcountrywereestimatedbasedonavailableprojectbudgets.countrieswithlessthanandgreaterthantheaverage(10%)allocationwereincluded.
Level of Direct Benefi-ciaries (Household, Community, Organi-sation, etc.)
Assessedthetypesofdirectbeneficariesattheprojectandcountrylevels.GiventhatfewDEPPprojectsdirectlytargetcommunities,itwasimportanttoincludecountrieswithcommunity-leveldirectbeneficiaries.
Likelihood of Upcoming Disaster or Emergency
Whilethiswasdifficulttopredict,theevaluationteamconsideredcountriesthathadahighlikelihoodofadisasteroremergencyoccurringduringtheDEPPtimelineasthiswouldallowforassessmentofoutcomesduringadisasteroremergencysituation.
Contextual Factors Accountedforuniquecontextualfactorssuchassocial,economic,politicalfactorstoensurethatselectedcountriesrepresentedadiverserangeofcontexts.
Number of DEPP Projects and Project Combination in Country
AssessednumberofDEPPprojectsandthespecificcombinationofprojectsineachcountry.ThefinalcohortofselectedcountriesincludedcountrieswithdifferentnumbersofDEPPprojectsanddifferentprojectcombinations.
Table2.2:criteriaforselectingcountriesforintensive-setevaluationactivities
Using the selection criteria, fiveDEPP countrieswere selected for intensive set evaluationactivities.TheseincludedKenya(Africaregion,highnumberofDEPPprojects),Ethiopia(Africaregion,mediumnumberofDEPPprojects), thePhilippines (Asia region,mediumnumberofDEPPprojects),andMyanmar(Asiaregion,lownumberofDEPPprojects)(seeTable2.3).ThefifthcountryselectedwasSouthSudan.However,increasedviolenceinJuly2016significantlyaffectedtheDEPPprojectsoperating inthatcountryandrequiredevacuationoftheirstaffmembers;thus,datafortheintensivesetofevaluationactivitieswerenotcollectedinSouthSudan.Afterdiscussionswith thecDAc-nandProtection inPracticeprojectsoperating inSouthSudan,andclosemonitoringofthesecuritysituation,inOctober2016,theevaluationteamrevisedthemethodologyfordatacollection.Insteadofon-sitedatacollectionasinitiallyplannedandoutlinedintheinceptionreport,dataonSouthSudanwascollectedaspartofthe
55 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
minimumsetevaluationactivitiesandthroughanin-depthcasestudy.AsthecDAc-nprojecteventuallyshifteditsprogrammingtothePhilippines,thefinalfourintensivesetcountrieswerestillabletoensurerepresentationofall14DEPPprojectswithintheevaluation(seeTable2.3).
ETHIOPIA KENYA MYANMAR THE PHILIPPINES
DEP
P Pr
ojec
t*
PublicHealthEmergencyPreparednessinGambella
ImprovedEarlyWarning–EarlyActionstoStrengthenDisasterPreparednessinEthiopia
StrengtheningEmergencyPreparednessSystemsinMyanmar
UrbanEarlyWarningEarlyAction learningProject financialEnablers cDAc-n ShiftingthePower TalentDevelopment TransformingSurgecapacity ProtectioninPractice ALERT AgeandDisabilitycapacitybuilding linkingPreparednessResponseandResil-ience
Total Number of Projects (at end of DEPP) 5 7 4 7
Key
Sele
ction
Crit
eria
GeographicRegion Africa Africa Asia Asia
InclusionofcDAcandSTARTnetworkProjects
START START STARTcDAcand
START
Directcommunity-levelbeneficiaries yes yes yes no
DEPPfundsAllocatedtocountry(Estimatedbasedonavailableprojectbudgets)
>10% >10% <10% >10%
numberofDEPPProjectsattimeofcountryselection*
5 7 3 5
Table2.3:countryselectionmatrixfortheintensivesetofevaluationactivities
*There were some changes in the implementation countries over the course of the DEPP
CONTEXTS OF INTENSIVE SET EVALUATION COUNTRIES
Asintended,thefinalfourintensivesetcountriesarediverseandhaveverydifferentdisasterprofiles and contexts as detailed in the boxes below (pages 51 and 52). These contextsinfluencedtheimplementationoftheprojectsand,tosomedegree,whatcouldbeachievedineachsetting.Therefore,analysisofthecountry-levelfindingstakesintoaccountthecountry-specificcontexts.
56 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
THE PHILIPPINESThe Philippines is at great risk for natural disasters41 due to its wide-ranging terrain, vast coastline, and position on the “Ring of Fire,” an arc of active volcanoes in the Pacific Ocean basin that make this one of the most active and unpredictable geographical areas in Southeast Asia.42,43 Typhoons, earth-quakes, floods, volcanic eruptions, landslides, and fires, are examples of common natural disasters, with roughly 6-7 tropical cyclones causing significant damage annually, out of an average of 22 occurrences.44,45 Man-made threats include climate change, an overuse of coastal, marine, and forest resource, and forced displacement due to occasional conflicts in Mindanao.46,47,48
There is a strong presence of local NGOs and civil society organisations (CSOs) involved in humanitarian work in the Philippines. OCHA also maintains a sub-office in Cotabato City, in order to monitor different aspects of an emergency, such as the number of internally displaced persons (IDPs) and their evacu-ation and transition sites, as well as connect with various stakeholders, including local government, UN agencies, and NGOs.49 With regards to government disaster risk reduction (DRR), the National Disaster Risk Reduction and Management Council (NDRRMC) functions as the lead agency for DRR preparation and actions.50 Other arms of the government that are directly involved in disaster relief include the Office of Civil Defense (OCD), the Department of Social Welfare and Development (DSWD) (which leads immediate disaster relief effects), and the Armed Forces of the Philippines (AFP).51
MYANMARThe humanitarian situation in Myanmar is both complex and large, with an estimated 863,000 people requiring humanitarian assistance in 2018.52 There is a large range of threats and challenges within the country, including natural disasters, food insecurity, armed conflict, inter-communal tensions, statelessness, displacement, and trafficking.53 Natural disasters that affect Myanmar include floods, cyclones, earthquakes, droughts, fires, and landslides in the mountainous regions.54,55 Other humanitarian challenges include internal displacement, with roughly 241,000 displaced (77% women and children) and living in camp or camp-like situations in Kachin, Kayin, Shan, and Rakhine states.56 In 2012, 120,000 were displaced following inter-communal violence, and in 2016, tens of thousands were displaced due to border post attacks in northern part of the Rakhine State.57 Occasional conflicts also occur in the Kachin and Shan states, which displaced nearly 100,000 people.58 More recently, from August to November 2017, more than 600,000 refugees of the Rohingya ethnic minority left Myanmar to resettle in Bangladesh.
Currently, various INGOs have a presence in Myanmar, though the amount of activity and work conducted by these organisa-tions is lower compared to that of several other DEPP countries, as Myanmar only recently opened to humanitarian actors. The presence and level of engagement of local NGOs are lower than that of INGOs due to the current humanitarian landscape.
41 UnitednationsOcHA.AboutOcHAinthePhilippines[Internet].[placeunknown]:Unitednations;[cited2018April24].42 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook
[Internet].[Hawaii]:centerforExcellenceinDisasterManagementandHumanitarianAssistance;2018March[cited2018April24],p92.
43 ReliefInternational.Asia–Philippines[Internet].[placeunknown]:ReliefInternational;[cited2018April24].44 UnitednationsOcHA.AboutOcHAinthePhilippines45 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook.46 UnitednationsOcHA.AboutOcHAinthePhilippines.47 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook.48 ReliefInternational.Asia–Philippines.49 UnitednationsOcHA.ThePhilippines,withauniquehistoryofembracingrefugees,turnsinwardto
addressIDPs[Internet].[placeunknown]:OcHA;[cited2018April24].50 centerforExcellenceinDisasterManagementandHumanitarianAssistance.PhilippinesDisasterManagementHandbook.51 Ibid.52 MyanmarHumanitariancountryTeam(UnitednationsandPartners).2018HumanitarianneedsOverview
[Internet].[Myanmar]:ReliefWeb;2017Dec[cited2018April24],p28.53 ibid.54 UnitednationsOcHA.MyanmarcountryProfile(12-01-17)[Internet].[placeunknown]:OcHA;2017
January[cited2018April24].55 UnitednationsOcHA.AboutOcHAMyanmar[Internet].[placeunknown]:OcHA;[cited2018April24].56 Ibid.57 UnitednationsOcHA.MyanmarcountryProfile(12-01-17)[Internet].[placeunknown]:OcHA;2017
January[cited2018April24].58 Ibid.
57 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
KENYAKenya faces a range of humanitarian challenges, including drought, food insecurity and malnutrition, and outbreaks of diseases such as cholera. The country hosts nearly 500,00 refugees and asylum-seekers.59 As of January 2018, six different counties within Kenya were battling an active Cholera outbreak. Drought conditions persist over large portions of the country leading to inter-communal conflict in certain areas.60 These challenges have impacted communities dependent on agriculture and pastoralism, as the frequent recurrence of natural disasters such as drought result in a loss of crops and/or livestock. This has contributed to food insecurity, with 3.4 million people requiring food assistance, and a rise in food prices. Finally, the continued influx of refugees has also placed a strain on Kenya’s already scarce resources such as water.61
The government of Kenya has had active roles in responding to some of these humanitarian challenges. For instance, they led the drought response at the county and national levels in 2017.62 However, the country’s infrastructure is not generally able to respond to large-scale crises. In late 2017, these challenges were further exacerbated by delays, uncertainty, and civil unrest during the political election.63
INGOs and multi-lateral organisations have a strong presence in Kenya, with several closely collaborating with the government in certain humanitarian responses.
ETHIOPIAEthiopia’s key humanitarian challenges fall under the following categories: displacement of individuals, disease outbreaks, climate-re-lated issues, socio-economic difficulties, malnutrition, and a lack of safe drinking water.64 Ethiopia hosts a large population of refugees, and IDPs due to a combination of internal conflict and climate-related difficulties such as drought and flooding.65 Throughout Ethiopia, resources such as food and safe drinking water are scarce and are worsened by drought, leading to rising levels of food insecurity and malnu-trition.66 UN agencies have indicated that these challenges are contributing to higher rates of violence against women, and limited access to health facilities, particu-larly for those affected by these crises.67
Ethiopia receives support in the form of funding and in-country presence from various multi-lateral organisations and INGOs, in part to increase prevention activities, raise the level of emergency preparedness, and strengthen the system for recovery efforts.68 Many of these organisations have identified priority areas and donor funding, such as improving emergency systems and increasing the capacity of local government.69 Several organisations have also sought to collaborate with the local and national governments, such as to develop the Ethiopia Humanitarian and Disaster Resil-ience Plan (HDRP) and a comprehensive drought response plan.70
59 UnitednationsOcHA.Kenya[Internet].[placeunknown]:OcHA;[updated2018february20;cited2018April24].
60 UnIcEfKenya.KenyaHumanitarianSituationReport[Internet].[Kenya]:UnIcEf;2018January[cited2018April24],p9.
61 Savethechildren.HumanitarianResponse:TheSituationforchildreninKenya[Internet].[England]:Savethechildren;[updated2018;cited2018April24].
62 UnIcEfKenya.KenyaHumanitarianSituationReport.63 Ibid.64 USAID.Ethiopia[Internet].[placeunknown]:USAID;[updated2018April9;cited2018April24].65 Ibid.66 USAID.ActiveUSGProgramsfortheEthiopiaResponse[Internet].[placeunknown]:USAID;[cited2018
April24].67 UnitednationsPopulationfund.EthiopiaHumanitarianEmergency[Internet].[placeunknown]:UnfPA;
[cited2018April24].68 UnitednationsOfficeforthecoordinationofHumanitarianAffairs.Ethiopia:2018Humanitarianand
DisasterResiliencePlanseeksUS$1.6btoassist7.9Mpeople[Internet].[placeunknown]:OcHA;2018March[cited2018April24].
69 USAID.Ethiopia.70 Reliefweb.Ethiopia:Governmentandpartnerslaunchthehumanitariananddisasterresilienceplan
(HDRP)for2018.[placeunknown]:ReliefWeb;2018March[cited2018April24].
58 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DATA COLLECTION APPROACHES AND TOOLS
GiventhecomplexityoftheDEPPanditsrangeofactivitiestargetingdifferentlevelsofaction(includingindividual,organisational,community,andsystems)anditsobjectivestostrengthennetworksand improveefficiencyof thehumanitarianresponse,anumberofdatacollectionapproaches were conceptualised, and corresponding data collection instruments weredeveloped.Overall seven typesofassessmentswereplannedwithmanyof these includingseveraldifferent,complementarydatacollectionelements.ThesevenassessmenttypesalongwiththecorrespondingtoolsarepresentedinTable2.4.AllofthedatacollectioninstrumentsareavailableinAnnex6.
SAM
PLIN
G
GRO
UP
MET
HO
D
STU
DY
TOO
L
DES
CRIP
-TI
ON
RAN
DO
M
SELE
CTIO
N
MIN
IMU
M
SET
KEN
YA
THE
PHIL
IPPI
NES
ETH
IOPI
A
MYA
NM
AR
UK
ORGANISATIONAL ASSESSMENT T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2
countrydirector(Seniormanagementifcountrydirectornotavailable)* Q
uantitative
OrganisationalSurvey
Questionsonpreparedness,response,&capacitybuildingactivitiesimple-mented,coordinationstructures,per-ceptionsofinstitutionalenvironment,network&collaboratingpartners,&theoutputs&outcomesoftheDEPP
Organisationalhumanitarianstaff
Quantitative
Knowledge,AttitudesandPractices(KAP)Survey
Questionsonknowledge,skills,&behaviourinhumanitarianprepared-nessandresponse,corehumanitarianconcepts,prioritisedgroups,exposuretocapacity-buildingactivities,networkandcollaboratingpartners,andtheoutputs&outcomesoftheDEPP
✔
Seniormanagement(countrydirectororanotherseniormanagerifthecountrydirectorwasnotavailable)*
Qualitative
In-depthinterviews
Questionsonexperienceswithcapac-itybuilding,collaboration,institutionalenvironmentandpolicies,organisa-tionalpreparedness,improvingpre-parednesssystemsforcommunities,contextualfactors
✔
Mid-levelorSenior-levelstaff
Quantitative
Organisationalchecklist
Questionsonorganisationalemergencypreparednessplans,andorganisationalresponsecapacityandcoordination
Programmestaff/management& projectleads
Qualitative
In-depthInterviews
QuestionsonexperienceswithDEPPactivities,perceptionsofprojectandprogrammeeffectiveness,contextualfactors,numberandtypeofspecificprojectactivities
COMMUNITY LEVEL ASSESSMENT T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2
communitymembers Q
uali-
tative community
focusgroupdiscussion
Questionsoncommunityvulnerability,risk,andresiliencetodisastersandemergencies
communityleaders
Quantitative
communitysurvey
Questionsoncommunityprepared-nessplansandactions,structuralmitigationactivities,neworimprovedearlywarningsystems,politicalcom-mitmentoflocalgovernmentandcommunitystructures
communitymembers
Quantitative
Householdsurvey
Questionsonhouseholdprepar-edness,disasterriskknowledge,exposuretohazards,accesstoearlywarningalerts,householdsbehaviourrelatedtopreparedness,perceptionsofofcommunitypreparednessforemergenciesanddisasters
✔
ECONOMIC ASSESSMENT T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2
Programmestaff/management& projectleads
"Qualitative/
budget&
financialreview"
Valueformon-eyin-depthinterviews
Questionsonresourceallocation,valuechains&transationcosts,&time&resources
NETWORK ANALYSIS T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2
collaboratingpartnersidentifiedinnetworksectionoforganisationalsurveys&KAPsurveysduringorganisational Assessment
Quantitative
networksurvey
Questionsoncollaboratingor-ganisations,trustamongpartners,&frequencyofcollaborationwithinnetworks
59 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
HUMANITARIAN RESPONSE ASSESSMENT AND CASE STUDIES T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2
DEPP&externalstakeholders
Qualitative/Quantita
-tive/Documentreview/
Observation&sitevisits
In-depthin-terviews,Self-administeredquestionnaires
Questionsondisasters&emergerciesinDEPPcountries,DEPPcontributiontohumanitarianresponses
OTHER ASSESSMENTS T1 T2 T1 T2 T1 T2 T1 T2 T1 T2 T1 T2
Projectstaff,organisationalstaff&externalstake-holders O
bserva
-tion&
sitevisits
n/A
Participationincountry-levelandgloballearningevents,visitstooragnisationalpartnersandexternalstakeholdersduringdatacollection
n/A
Document
review Matrices Reviewofprogrammeandproject
leveldocumentsandresources
Table2.4:Methodology,studytools,samplepopulationanddatacollectionsiteforeachtypeofassessment
ETHICS
Ethicalconsiderationsforthisevaluationwerecarefullyassessedandthedatacollectionwasdesignedtoensureprotectionofindividualsparticipatinginthestudy.HumansubjectsresearchapprovalfortheevaluationprotocolandstudyinstrumentswasobtainedfromtheHarvardT.H.chanSchoolofPublicHealth’sInstitutionalReviewboard(IRb)aswellasfromrelevantbodiesineachoftheintensivesetcountries.Inthefourintensivesetcountries,theevaluationteampartneredwithlocalresearchorganisationswhoimplementedthedatacollection.TheseincludedthebusaracenterforbehavioralEconomics(Kenya),TheEthiopianPublicHealthAssociation(Ethiopia),andInnovationsforPovertyAction(Myanmar,ThePhilippines).Theselocalresearchpartnershiredthestudystaff.TheevaluationteamconductedsitevisitsatT1andT2toparticipateintheenumeratortraining,pilotingofinstrumentsandprotocols,andseveraldaysofdatacollectiontoensuredataquality.followingthis,theevaluationteamwasinclosecontactwiththein-countryresearchteamsinordertoremotelysupervisetheongoingdatacollection.Thisinvolvedregularmeetingsanddebriefingsessionswiththeevaluationteam,aswellasregularandfrequentchecksofincomingdatatoensurequality.QuestionnaireswerepilotedandprogrammedintoKoboToolbox,anelectronicdatacollectionsoftwarewhichpermittedtheevaluationteamtocloselymonitorincomingdatafromthefieldteams.
SAMPLE SIZE
ThesamplesizefortheevaluationisdetailedinTable2.5.Intotal,2675quantitativesurveysand qualitative interviewswere completed during T1, including 422 in theminimum set ofevaluationactivitiesand2253intheintensivesetofevaluationactivities.InT2,atotalof3440quantitativesurveysandqualitativeinterviewswerecompleted,with250intheminimumsetand3190intheintensiveset.
MINIMUM SET INTENSIVE SET NETWORK COMMUNITY VFM CASE STUDY
iDis
Organisational
surveys
KAPSurveys
Total
iDis
Organisational
checklist
Organisational
surveys
KAPsurveys
networksurveys
HouseholdSurveys
VillageSurveys
communityfGDs
VFM
iDis
caseStudyIDIs
Total
T1 51 77 294 422 47 185 185 352 453 1004 39 19 13 3 2253
T2 51 51 148 250 25 157 160 304 334 2072 65 43 8 22 3190 Table2.5:Samplesizeforevaluationactivities
60 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
METHODS FOR THE INTENSIVE SET OF EVALUATIONACTIVITIES DATA COLLECTION PROCEDURES
A local researchmanager based in each intensive set country oversaw and coordinated allof thefieldwork.Thestudy instrumentswerewritten inEnglishandtranslatedtothe locallanguageinallcountriesexceptKenya,andadaptedtothelocalcontext,pilotedandmodifiedtoensurethatthemeaningsofquestionsandresponseswerecorrect.UpdatedinstrumentswereapprovedbytheappropriateIRbs.Maleandfemaledatacollectorsaswellassupervisorswererecruited ineachcountrytaking intoaccountrelevantcharacteristicsneededforeachtypeof survey. for example, for the community andhousehold surveys, enumeratorswhocouldspeakthe local languagewererecruitedfromthestudyareas.Theenumeratorsweretrainedforaminimumoffivedays,andconductedfieldpracticeoverthreetofivedayswithorganisationsorcommunitieswhichwerenotpartofthesamplepopulation.Thispermittedenumeratorstopracticeadministeringthesurveysunderreal-worldconditions,andtoassessthetranslatedinstrumentsandaddresserrorsorproblemswiththetranslationpriortothestartofdatacollection.Atleastoneevaluationteammemberwaspresentforandparticipatedinthetraining,thepilotingandthestartofthedatacollection.Afterthis,theevaluationteammemberremained in regular contactwith the in-countrydatacollection team to remotely supervisethefieldwork.Regularmeetingswithin-countryteamswereheld,andin-comingdatawerecheckedregularlytoensurehighquality.
Verbal informedconsentwasobtained fromparticipantsand interviewswereconducted inprivatesettingstoensureconfidentiality.Inordertominimisebiasandtopreventinducementto participate, no compensationwas provided to respondentswho participated. Instead, acertificateofparticipationwasprovided.Questionnaireswereadministeredby local, trainedenumeratorsusingKoboToolboxonAndroidtabletsineitherEnglishorinthelocallanguage.In-depthinterviewslastedapproximatelyonehourandwereaudiorecordedonAndroidtablets.Quantitativeandqualitativedatawereuploadedtoasecureserver,whichisonlyaccessiblebykeyevaluationstaffthathaveethicalclearance.copiesoftheaudiorecordingonthetabletsweredeletedtoprotectprivacyandconfidentiality.Dataweremonitoredonanongoingbasistoensurehighquality.Anyerrorswerecorrectedimmediately.Enumeratorswerecontinuouslyprovidedfeedbacktoimprovetheirperformanceanddataquality.
ORGANISATIONAL-LEVEL ASSESSMENT
figure 2.1 provides an overview of the sampling approaches and survey instruments thatwereusedforeachofthesamplegroupsintheorganisational-levelassessment.first,alistoforganisationsandcommunitiesassociatedwiththeDEPPprojectswascreatedineachintensiveset country. In parallel, a list of comparison organisations were identified in each countrythroughavarietyofmeans.listsofbothnationalandinternationalorganisationsworkingonemergencypreparedness and response in each countrywere constructedwith informationfromvarious sources includingUnOcHA, humanitarianworking groups, andnationalnGOregistries.Organisationsthatweresimilarinsize,scope,andwithsimilargeographicfocusasthoseworkingontheDEPPineachcountrywereselected,andtheproportionofinternationalnGOs to nationalnGOswas kept constant across control andDEPP groups. In the eventthat there were more control organisations than needed that met the selection criteria, asubsetoforganisationswasrandomlyselected.Incountrieswherethereweremorethan50DEPPorganisations,arandomselectionoforganisationswasselectedforparticipationintheevaluation.Incountrieswheretherewerelessthan50DEPPorganisations,allorganisations
61 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
wereselectedforthesample.
foreachorganisation,thefollowingsurveyswereconducted:1)oneorganisationalsurvey,2)oneorganisationalchecklist,and3)threeknowledge,attitudes,andpractices(KAP)surveys.Inaddition,arandomselectionof15%organisationsinthesamplewereselectedtoparticipateinanin-depthinterview(IDI)inadditiontothesurveysabove.
ORGANISATIONAL SURVEY, KAP SURVEY, ORGANISATIONAL CHECKLIST, AND IN-DEPTH INTERVIEWS
In-countryresearchpartnerscarriedoutthefollowingprocedures:
1) Email invitations were sent to the country directors, or main contact of eachorganisationinthesampletoinformthemthattheirorganisationwasselectedtoparticipate in theDEPPevaluation,andorganisationswereasked toprovide theresearchpartnerswithalistofhumanitarianstaff.
2) Appointmentsweremadewithcountrydirectors(ormembersofseniormanagementwhencountrydirectorswerenotavailable)toconductanin-personorganisationalsurvey. In addition to completing the organisation survey, the individual wasasked toa)namea staffmemberwith in-depthknowledgeof theorganisationalpreparedness plans, procedures, and systems, to complete the organisationalchecklist;b)recommendanothermemberofseniormanagementtocompletethein-depthinterview(fororganisationsselected);andc)providealistofstaffwithintheorganisationifthiswasnotprovidedbytheorganisationpriortotheinterview.
3) Thestaffmemberidentifiedbythecountrydirectorormemberofseniormanagementwasapproachedtocompletetheorganisationalchecklist.
4) fortheKAPsurvey,threestaffmemberswithineachorganisationwererandomlyselected from the list of humanitarian staff, and appointmentsweremadewithselectedindividualsforanin-personsurvey.
5) Inthe15%oforganisationsthatwererandomlyselectedforthein-depthinterviews,in addition to the quantitative surveys, country directors ormembers of seniormanagementthatdidnotparticipateintheorganisationalsurveyswereselectedtoparticipateinanin-depthinterview.
figure2.1:Samplingapproachandstudyinstrumentsfororganisational-leveldatacollectionineachintensivesetcountry
*Surveys completed by different members of organisational management, depending on availability
62 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Allsurveysconductedbasedontheaboveprocedureswerecarriedout in-person,withtheexceptionofasmallnumberofsurveysconductedoverSkypeorphoneduetoextenuatingcircumstances.ThenumberoforganisationsvariedpercountryasitwasbasedonthenumberofDEPPimplementingpartnersandbeneficiariesineachcountry,andthiswasdependentontheprojectsthemselves.Inmostcases,thisdidnotexceed50DEPPorganisationsand400humanitarianstaffworkingatDEPPorganisationspercountry.Anequalnumberofcomparisonorganisations,humanitarianstaff,communitiesandhouseholdswhichweresimilarinsizeandlocationorother factors to theDEPPstudypopulationwerealso selectedusing the samesamplingapproach.
COMMUNITY-LEVEL ASSESSMENT
A list of all villages whereDEPP activitieswere implemented at the community level wascompiledinMyanmarandEthiopia,.InMyanmar,DEPPvillagesinKayinandShanstatewereselectedforthesample.Togeneratealistofcomparisonvillages,in-countryresearchteamsworkedwiththelocalgovernmentsorotherpartnerstoobtainalistofallvillagesinthearea,includingtheirpopulationsizeandotherdemographiccharacteristics.fromthislist,asetofcomparison villageswhichwere similar in size and demographics, but far enough from theDEPPvillagestominimiseriskofexposuretoDEPPintervention,wereselected.
InEthiopia, theGambella regionwas selected for the community assessmentbecause twoDEPPprojectswereimplementingcommunity-levelactivitiesthere(PHEPGambella,EWEA).DEPP interventionsbenefittedallvillageswithinadistrict (woreda),andthusa listofDEPPworedas71 andnonDEPPworedas, including the villages in eachworeda,was constructed.Anyareasorvillagesinthatlist,deemedinaccessiblebythelocalgovernmentduetoviolence,securityorfloodingwereremovedfromthesample.ArandomsampleofvillagesthatwereaccessibleintheDEPPworedaswereselectedfordatacollection.forthecomparisongroup,arandomsampleofvillagesinthenon-DEPPworedaswasselectedfordatacollection.
figure2.2:Samplingapproachandstudyinstrumentsforcommunity-leveldatacollection
Ineachsampledcommunity,thefollowingsurveyswereconducted:1)onecommunitysurvey,2)~1000householdsurveys3)onecommunityfocusgroupdiscussion.Allsurveysconductedbasedontheaboveprocedureswerecarriedoutin-person.focusgroupdiscussionquestionswereadaptedfromtheGOAlcommunityResilienceToolkit72.
71 Aworedaistheequivalentofadistrict.72 ToolkitforMeasuringcommunityDisasterResilience.GuidanceManual.Goal,2014.
63 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
COMMUNITY SURVEY, HOUSEHOLD SURVEY, COMMUNITY FOCUS GROUPDISCUSSION
In-countryresearchpartnerscarriedoutthefollowingprocedures:
1) localresearchpartnerscontactedlocalorganisationsandgovernmentsimplementingactivitiesineachvillage,whothenintroducedtheresearchpartnerstorelevantcommunityleadersandinformants.
2) Thecommunityleaderfromeachvillagewasselectedforthecommunitysurvey.
3) Arandomsampleofhouseholdswasselectedforthehouseholdsurvey.
4) community members involved in disaster preparedness were recruited for thecommunityfocusgroupdiscussion.
METHODS FOR THE MIMINUM SET OF EVALUATIONACTIVITIES IN-DEPTH INTERVIEWS
Thestudypopulationfor theminimumset in-depth interviewscomprisedUK-basedprojectleadership, project leadership in DEPP countries, DEPP programme-level leadership andtargetedstakeholders.Alistofprojectandprogrammeleadershipwasfinalisedandparticipantswererecruitedthroughperson-to-personcontactatDEPPlearningeventsandthroughemailinvitations.foreachDEPPproject,theevaluationteamaimedtosampleboththeUK-basedleadershipandthein-countryleadership.IfaprojectdidnothaveleadershipattheUKlevel,thein-countryleadershipwereinvitedtoparticipate.
fortheDEPPtargetedstakeholders,purposivesamplingwasapplied.Usingalistofcontactsfrom the inceptionand formativephases, targetedstakeholderswere invited toparticipate.Therewereseveralparticipantswhowereinvolvedattheprojectlevelandprogrammelevel;inthiscase,thoseparticipantswereinterviewedonlyonce.SomeofthekeystakeholderswhotookpartwereindividualswhohadpreviouslyworkedontheDEPPbutarenolongerformallyengagedwiththeprogramme.Additionaleffortwasmadetolocateandinvitetheseindividualsforparticipationintheinterviews.
Verbal informedconsentwasobtainedforall interviews. InterviewswereconductedbyoneofthreeevaluationteammembersinEnglish.Theinterviewswereconductedeitherinpersonor by Skype, and lasted approximately 45-75minutes. The interviewswere recorded usingcallnotesoftwareormobileaudiorecordingsoftwareforSkypeinterviewsandonanAndroidtabletusingastockaudiorecordingsoftware.Evaluationteammemberstranscribedtheaudiorecordedinterviews.TranscriptswereimportedintoDedoosequalitativeanalysissoftware.
followingthein-depthinterviewsconductedatT2,all interviewparticipantswereemailedalinktoanoptionalonlinequestionnaireonKoboToolboxtocaptureadditionalperspectivesonprojectdeliveryandimpactthatwerenotcoveredduringtheinterviews.
64 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ONLINE ORGANISATIONAL SURVEY AND KAP SURVEY
The study population for the minimum set online quantitative questionnaires comprised1) organisational leadership and 2) humanitarian programming staff from implementingorganisationsandbeneficiaryorganisationsinDRc,SouthSudan,Pakistan,bangladesh,andJordan.AlistofDEPPorganisationswasfinalisedwiththeUK-basedprojectleadsandRegionallearningAdvisors (RlAs).Participantswererecruitedby theevaluationteamthroughemailinvitation.
Abridged versions of the organisational survey and KAP survey from the intensive setof evaluation activitieswere sent via email invitation through links to online surveys usingKoboToolbox.OnlinequestionnairesforallparticipantswereadministeredinEnglishexceptintheDRcwhereitwastranslatedtofrench.QuestionnaireswereemailedtoDEPPimplementingandbeneficiaryorganisationsandwerecompletedusingKoboToolbox.Participantsprovidedconsentpriortobeginningthequestionnaire.
norewardsorcompensationwereprovidedtoparticipants.Acertificateofstudyparticipationwasprovidedtoparticipantsinprojectcountrieswhocompletedtheonlinequestionnaire.
ECONOMIC ASSESSMENT (VALUE FOR MONEY)
InterviewsfortheVfManalysiswereconductedinpersonandviaSkypebytheevaluationteam’s economistandVfMadviser.Purposivesamplingwasapplied.ThestudypopulationfortheeconomicdatacollectionincludedDEPPportfoliomanagementteammembers,DEPPboardmembersandprojectstaff.
Areviewofbudgets,financialinformationandprojectreports,wasalsoconductedtoassessVfMattheprojectandprogrammelevel.Inaddition,datafromselectquestionsoncostandefficiencyfromtheorganisational-levelassessmentoftheevaluation(organisationalandKAPsurveys)wereanalysed.costperresultindicatorswerecreatedthroughthefinancialreviewprocess.TheextenttowhichVfMrecommendationsfromformativephasewereimplementedbytheprojectswasalsoassessed.
NETWORK ANALYSIS
Thenetworksurveywascarriedoutintwophasesatbothtimepoints.Thefirstphaseincludedadministeringquestionsonnetworksandcollaboratingpartnersthatwereembeddedwithinthe quantitative organisational andKAP surveys. In the network section of these surveys,organisational leadership and humanitarian staff were asked to identify the organisationsthey collaboratedwith over the last sixmonths and to describe the nature or purpose ofthecollaboration.Theyselectedfroma listof32collaborationareas inwhichhumanitarianactors are likely to engage. Respondents were also given the option to name their ownareasofcollaboration.Respondentsthen identifiedtheirmaincontactateachcollaboratingorganisation.Theactors thatwerenamedbyfirstphasesurveyparticipantsareconsideredtobe1stdegreeactors;theyareonedegreeofseparationawayfromthesurveyinformant.Thein-countryresearchpartnersthencalledeachofthe1stdegreeactorsandthestandalonenetworksurveywasconductedwiththat individualoverthephone.These individualswerethenaskedtoidentifytheirmaincontactateachcollaboratingorganisation,andthosecontactsareconsidered2nddegreeactors, as theyare twodegreesof separation from theoriginalsurveyinformant.Thestandalonenetworksurveywasthenconductedwiththe2nddegreeactors,withtheexceptionofthePhilippinesduetolimitedtimeandresources.
HUMANITARIAN RESPONSE ASSESSMENT AND CASE STUDY
In-depthcasestudieswereaddedtotheevaluationmethodologyduringtheformativeandinterim phases to provide further perspectives on emergency preparedness and response
65 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
activities in the programme, and to assesswhether improved preparedness translated intomoreefficientandtimelyhumanitarianresponse.SimulationexerciseswereinitiallyproposedduringtheinceptionphaseasamethodtomeasureDEPPbeneficiaries’performanceduringhumanitarianresponseactivitiesinacontrolledmanner.TheintentionwastocollectsuchdataduringsimulationsthatwereplannedbyDEPPprojectsthemselves.However,overthethree-yearprogramme,projectschangedtheirplansregardingsimulations,anditwasalsonotedthatseveraloftheDEPPprojectsrespondedtoactualdisastersintheprojectcountriesprovidinganopportunity to assess contributions to real humanitarian responses.After reflectionanddiscussionwith thelearningProject, theevaluation teamdecided toconduct two in-depthcasestudies.ThefirstcasestudywasconductedinSouthSudantocaptureexperiencesandperspectivesofprojectactivitiesduringtheescalationofviolenceintheformativephaseandhowthis impactedproject implementation.Thesecondcasestudyfocusedonhumanitarianresponses inEthiopiaandKenyaduring thecourseof theDEPPtocapture thequalityandspeedofresponseactivitiesinlieuofsimulation-baseddatacollection.
ThecasestudyapproachallowedfordeeperexplorationandanalysisofactualhumanitarianresponsesthattookplaceduringtheDEPPprogrammecycle.Purposivesamplingwasapplied.Potentialparticipantswerecontactedbyemailwithabriefdescriptionofthecasestudytobeconductedaspartoftheevaluationandinvitedtoparticipate.InKenya,responsecasestudydatacollectionalsoincludedsitevisitsandobservationaldataaswellasinclusionofbeneficiarypopulationinterviews.InEthiopia,duetosecurityconcerns,asitevisitwasnotconductedinMoyaledistrictandasmallersampleofparticipantswereincludedinthedatacollection.
Inaddition,duringthedisseminationandvalidationworkshopsconductedbytheevaluationteamintheintensivesetcountries,aself-administeredquestionnairewascirculatedtodocumentdisastersandhumanitarianresponseactivities.ThequestionnaireaskedparticipantstoidentifyallofthedisastersandemergenciesthathadoccurredintheircountriessincethebeginningoftheDEPP,andtoindicatewhichcrisestheirprojecthadrespondedto.Thesedataallowedtheevaluationteamtoassessthenumberandtypeofdisastersandemergenciesineachintensivesetcountry,andprovidedadescriptionofthetypeandextentoftheresponseactivitiesthattheDEPPorganisationconducted.
OBSERVATION AND SITE VISITS
Membersoftheevaluationteamandlocalresearchpartnershadopportunitiestovisitprojectsand engage with project stakeholders in countries where the intensive set of evaluationactivitiestookplaceandincountrieswhereteammembersattendedlearningevents.Priortodatacollection,theevaluationteamconductedsitevisitsandfacilitatedinductionworkshopsin each country selected for the intensive evaluation activities to introduce the externalevaluationtoDEPPprojectleadershipandstaff,andgatherfeedbackontheproposedplanforthatcountry.Thesesessionsalsoprovidedanopportunitytoengageindialoguewithprojectleadershipandstaff,whichcontributedtotherefinementofthemethodology,studyprotocolanddatacollectioninstruments.
Evaluationteammembersalsoparticipatedinanumberoflearningandcollaborationmeetingsandconferences(Table2.6).TheseeventspermittedengagementwithDEPPprojectteamsandkeystakeholders,andalsomorestructuredassessmentandobservationonthedifferencesinperspectivesofDEPPleadershipandstaffattheUKlevelandatthecountrylevel,contextualfactorsthatimpactprojectimplementation,andthevalueofevidenceanddifferentresearchmethodologies in thehumanitarian field.During the summativephase, the evaluation teamorganisedvalidationworkshopsinthefourintensivesetcountriestodisseminatefindingsandgatherfeedbackatthecountrylevel.Theresultsofthevalidationworkshopsprovidedfurthercontextandinsightintotheevaluationfindingsandhelpedtoinformtherecommendations.
Evaluationteammemberscompilednotesduringobservationandsitevisitsandextractedkeythemes.Observationdatapresentedinthisreportarebasedonthesefieldvisitsandconferenceattendancetoenabletheevaluationteammemberstointeractwithprojects.
66 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
OBSERVATION AND SITE VISITS COUNTRY DATEMEl/learningProjectlaunch UK March2015
londoncollaborationday UK October2015
londoncollaborationday UK february2016
InceptionphaseworkshopsEthiopia,Kenya,Myanmar,
thePhilippinesMarch2016
STARTconference UK May2016
launchofT1datacollectionKenya,Myanmar,thePhilip-
pinesSeptember-november2016
londoncollaborationday UK november2016
nairobilearningconference Kenya november2016
launchofT1datacollection Ethiopia May2017
DEPPlearningconference ThePhilippines July2017
Evaluationworkshop UK July2017
launchofT2datacollectionEthiopia,Kenya,Myanmar,
thePhilippinesSeptember-november2017
formativephasedisseminationevents Ethiopia,Kenyanovember2017,January
2018
Summativephasedisseminationandvalidationworkshops
Ethiopia,Kenya,Myanmar,thePhilippines
March2018
PreparingforShockconference Switzerland March2018
Table2.6:EvaluationteammemberparticipationatDEPPevents
DOCUMENT REVIEW
The evaluation team engaged in continuous and comprehensive information collection andreview of all available documentation and materials relevant to the programme and theprojectsthroughouttheprogrammecycle.Documentswereobtaineddirectlyfromprojectandprogrammestaff,aswellasviatheDEPP’sonlineboxfolders,andtheonlineDEPPlearningPlatform73. This included project reports, documents, case studies, presentations and anyotheravailablematerials inadditiontodatacollectedthrougheachproject’smonitoringandevaluation system.Documents and learning captured through theDEPP’s learning Projectactivitieswerealsoobtained.Oncetheexistingdocumentswereobtainedandcompiled,thereviewandanalysisinvolvedassessingeachdocument,categorisingit,andextractingkeydatainamatrixwherekeythemes,suchasgenderconsiderations,couldbecompared.Table2.9presentsasummaryofthedocumentsreviewedaspartofthedocumentreview.
finally, a targeted literature review related to humanitarian capacity building activitieswasconductedandexternaldocumentswerealsoconsultedaspartofthisprocesstoinformtheoverallinterpretationofdata.AcompletelistofdocumentsreviewedisprovidedinAnnex7.
ANALYSIS
Datawere analysedover the three-yearevaluationperiod.Differencesbetween subgroupsandcountrieswereassessed,anddataweretriangulatedfrommultiplesourcestostrengthenandvalidatefindings.context-specific informationandperspectives fromthe local researchteamsandcountry-levelDEPPstakeholders(throughvalidationworkshops)werealsoappliedtocountry-leveldatatoensuresoundinterpretations.
73 ThelearningPlatformcanbefoundhere:https://disasterpreparedness.ngo/.
67 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
forthequalitativedata,in-depthinterviewsconductedinotherlanguagesweretranslatedtoEnglish.The analysisprocess included listening to audio recordingsor reviewing transcriptswhichwere thencodedandanalysedusingDedoose software forqualitativedata analysis.Mainthemes,concepts,andquoteswereextractedeitherfromtheaudiorecordingsorfromthetranscriptsandcategorisedinadetailedmatrixtofacilitatecomparisonandidentificationofpatternsoverallandwithinsubgroups.
QuantitativedatawereanalysedusingStatisticalAnalysisSystem(SAS)softwareandSTATA(version 13.1). Descriptive statistics were generated, with comparisons made betweenintensivesetcountriesaswellasacrosstheentiresampleof10countrieswhenpossible,andbetweentheDEPPandcomparisongroups.T-testsandchi-squaredtestswereconductedtoassesswhether differencesbetweenDEPPand control groupswere statistically significant.Difference-in-DifferenceEstimationwasusedtoestimatetheeffectoftheDEPPinterventionsbycomparingthechangesinoutcomesovertimebetweentheDEPPgroupandthecomparisongroup.Datawerealsodisaggregatedbygenderwhererelevant.
TheeconomicandVfManalysisfocusedontheprogrammeandcountrylevels.Theanalysisinvolvedreview,bytheevalutionteam’sVfMadvisor,ofdocumentsincludingquarterlyreports,adhocVfMreports,learningProjectdocuments,theprogramme-levelbudget,andprojectandprogrammeexpendituredata.Interviewdatawerecodedandanalysedandempiricaldataoncostsandefficiency,collectedaspartoftheintensivesetdatacollection,werealsoanalysedaspartoftheVfManalysis.formativephaseVfMrecommendationswerealsoassessedtodeterminetheextenttowhichtheywereimplemented.Theassessmentalsoincludedinternalbenchmarkingofunitcosts,costeffectivenessanalysis(includingcostperresultsassessment),analysisofadherencetoprocurementproceduresandanalysisofdecision-makingandprojectmanagementwithinDfIDandimplementingagencies,andhowVfMwasconsidered.findingsweretriangulatedwiththemultipledatasources.
networkdatawereanalysedbyRootchange,aninstitutionspecialisedinnetworkmappingandnetworkvisualisations,usingORA,acustomisednetworkanalysis toolandvisualisationplatformdevelopedbycASOSatcarnegieMellon.collaborationpatternswereidentifiedandnetworks,groupsandorganistionsfromeachcountrywerecompared.ThestatisticalanalysissoftwareRwasusedforstatisticalsignificancetestsbetweengroups.
Documentreviewincludedin-depthassessmentsofprojectreports,documents,casestudies,presentations,logicalframeworksandotheravailablematerials.
Anydifferencesinopinionswithintheevaluationteamwereresolvedthroughobtainingothersourcesofinformationforfurtherclarification.Thisincludedthecollectionofadditionaldata,furtheranalysisorsubgroupanalysis.findingssupportedbymultipledatasources(includingbothquantitativeandqualitativedata)wereweightedmoreheavily.consensuswasdeterminedthroughthereviewofadditionaldataand internaldiscussion.Therewerenounintendedorunexpectedfindingsinthisphaseoftheevaluation.Ingeneral,whiletherewassomevariationintheperspectivesofstakeholders,therewasgeneralconsensusonmostissues.Whereopinionson issuesweremixed, thesehavebeendescribed inthetext.Therewerenodifferencesofopinionwithintheevaluationteam,noconflictsofinterestandtheevaluationteamwasabletoworkfreelywithoutinterference.
DISSEMINATION AND FEEDBACK TO BENEFICIARIES
Atvarioustimes throughout theevaluation, stakeholderswereengaged toobtain feedbackonthemethodologyandprocessesortopresentfindingsfromtheevaluation.SomeoftheseeventsarepresentedinTable2.6.findingswerealsodisseminatedviasocialmediaandthroughpresentationswithexternalstakeholders.AdisseminationandcommunicationplanforthisfinalsummativereportisavailableinAnnex8.
68 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
COUNTRY OWNERSHIP, MANAGEMENT AND CAPACITY BUILDING The evaluation has been implemented in accordance with the Paris Declaration on AidEffectiveness(2015)principles74.TheintensivesetdatacollectionactivitieswereledbylocalresearchpartnersinEthiopia,Kenya,MyanmarandthePhilippines.Thelocalpartnersadaptedfieldprotocolsforthespecificcontext,ensuredsensitivitytolocalcultureandconsideredlocalethicalandlegalconcerns.capacitybuildingactivitiesforin-countrystaffoccurredatvarioustimesandincludedtrainingforeachlocalresearchmanagerinoverseeingthefieldworkandmanagingtheteam,aswellastrainingsfordatacollectorsontheinstrumentsandprotocols.capacitybuildingonevidencegenerationandhowtoappraisethequalityofevidencewasalsoconductedforalargerangeofDEPPstakeholdersataDEPPglobaleventthattookplaceinnairobi,Kenyain2016.
GENERAL CHARACTERISTICS OF THE STUDY POPULATION
Intotal,2,315interviewsorsurveyswereconductedduringT1,ofwhich347wererelatedtotheminimumsetactivitiesand1968werepartoftheintensiveset.InT2,3,440interviewsorsurveyswereconducted.Ofthosecompleted,250werefromtheminimumsetand3,190werefromtheintensiveset.
Tables2.7and2.8belowpresentthedemographiccharacteristicsofrespondentsinterviewedaspartof theevaluation’squantitativemethods:1)organisationalsurvey,2)KAPsurvey,3)householdsurvey,4)communitysurvey.Moredetailed,country-specificdemographicdataareprovidedinAnnex9.
ORGANISATIONAL SURVEY KAP SURVEY T1 T2 T1 T2
VariableDEPP
(N=172)Comparison
(N=78)DEPP
(N=143)Comparison
(N=64)DEPP
(N=479)Comparison
(N=138)DEPP
(N=333)Comparison
(N=119)
Age, mean (SD) 42.8 ± 9.7 47.4 ± 9.2 43.5 ±
10.0 46.6 ± 9.9 36.0 ± 9.1 40.0 ± 10.7 38.5 ± 9.6 40.4 ± 10.3
% Female 48 (27.9%) 19 (24.4%) 37 (25.9%) 16 (25.0%) 175 (36.5%) 40 (29.0%) 103 (30.9%) 36 (30.3%)
Education LevelSecondary School (some/ completed)
1 (0.6%) - 4 (2.8%) - 14 (2.9%) 4 (2.9%) 12 (3.3%) 4 (3.4%)
University (some/completed)
63 (37.8%) 25 (32.1%) 53 (37.1%) 18 (28.1%) 51 (53.0%) 71 (51.5%) 154 (6.6%) 66 (55.5%)
Master’s Degree 93 (54.1) 41 (52.6%) 73 (51.1%) 41 (64.1%) 184 (38.4%) 51 (37.0%) 142 (42.6%) 47 (39.5%)
Professional / Advanced Degree
8 (4.7%) 2 (2.6%) 8 (5.6%) 2 (3.1%) 12 (2.5%) 1 (0.7%) 11 (3.3%) -
Other 6 (3.5%) 10 (12.8%) 5 (3.5%) 3 (4.7%) 9 (1.9%) 5 (3.6%) 7 (2.1%) -
74 ParisDeclarationonAidEffectivenessandtheAccraAgendaforAction(2015).Availableat:http://www.oecd.org/dac/effectiveness/parisdeclarationandaccraagendaforaction.htm.
69 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
Type of OrganisationINGO 63 (36.6%) 37 (47.4%) 49 (34.2%) 28 (43.8%) 207 (43.2%) 70 (50.7%) 124 (37.2%) 59 (49.5%)
National NGO 45 (26.2%) 20 (25.6%) 37 (25.9%) 17 (26.6%) 92 (19.2%) 19 (13.8%) 60 (18.0%) 22 (18.5%)
Local NGO 52 (30.2%) 14 (18.0%) 50 (35.0%) 14 (21.9%) 144 (30.1%) 29 (24.4%) 105 (31.5%) 29 (24.4%)
Interna-tional Organi-sation (UN, World Bank etc)
- 3 (3.9%) 1 (0.7%) 1 (1.6%) 7 (1.5%) 4 (2.9%) 8 (2.4%) 3 (2.5%)
Government 4 (2.3%) 2 (2.6%) 4 (2.8%) 2 (3.1%) 15 (3.1%) 3 (2.2%) 18 (5.4%) 4 (3.4%)
Private Sector - 1 (1.3%) - 1 (1.6%) - - 1 (0.3%) -
Academic Institution 2 (1.2%) - 1 (0.7%) - 2 (0.4%) - 4 (1.2%) -
Other 6 (3.5%) 1 (1.3%) 1 (0.7%) 1 (1.6%) 12 (2.5%) 1 (0.7%) 12 (3.6%) 2 (1.7%)
Years Working in the Humanitarian Field<1 year 11 (10.7%) 3 (3.9%) 3 (3.2%) - 82 (11.7%) 8 (5.8%) 31 (9.3%) 1 (0.8%)
1-3 years 10 (9.7%) 4 (6.3%) 3 (3.2%) 4(6.3%) 90 (18.8%) 11 (8.0%) 60 (18.0%) 8 (6.7%)
3-5 years 7 (6.8%) 8 (10.3%) 16 (17.2%) 7 (10.9%) 69 (14.4%) 17 (12.3%) 53 (15.9%) 19 (16.0%)
5-10 years 30 (29.1%) 16 (20.5%) 26 (28.0%) 16 (25.0%) 123 (25.7%) 44 (31.9%) 95 (28.5%) 42 (35.3%)
10-15 years 17 (16.5%) 16 (20.5%) 20 (21.5%) 17 (26.6%) 72 (15.0%) 27 (19.6%) 63 (18.9%) 27 (22.7%)
15+ years 28 (27.2%) 31 (41.0%) 25 (26.9%) 20 (31.3%) 43 (9.0%) 31 (22.5%) 31 (9.3%) 22 (18.5%)
Table2.7:DemographicdatafororganisationalandKAPsurveysacrossallcountriesatT1andT2
HOUSEHOLD SURVEY T1 T2
DEPP Comparison DEPP Comparison
Variable Myanmar (N=589) Myanmar (N=415) Myanmar (N=570) Myanmar (N=433)
Age, mean (SD) 44.8 ± 14.4 43.3 ± 13.9 45.1 ± 14.2 44.0 ± 14.2
% Female 299 (50.8%) 211 (50.8%) 288 (50.5%) 225 (52.0%)
Attended school 452 (76.7%) 306 (73.7%) 447 (79.1%) 329 (76.5%)
Literate 424 (72.0%) 298 (71.8%) 425 (75.0%) 315 (73.4%)
Average #of family members living in dwelling
5.1 ± 2.2 5.2 ± 2.2 5.5 ± 2.3 5.4 ± 2.2
ReligionChristian 347 (58.9%) 294 (70.8%) 368 (64.6%) 311 (71.8%)
Buddhist 219 (37.2%) 117 (28.2%) 200 (35.1%) 121 (27.9%)
Other (Hindu, Islam,
Traditionalist)23 (3.9%) 4 (1.0%) 2 (0.4%) 1 (0.2%)
Community Survey T1 T2
DEPP Comparison DEPP Comparison
Variable Myanmar (N=19) Myanmar (N=20) Myanmar (N=19) Myanmar (N=19)
70 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Age 45.6 ± 11.6 38.2± 10.9 43.4 ± 11.7 38.2 ± 10.6
% Female 1 (5.3%) 1 (5.0%) 1 (5.3%) 0 (0.0%)
Table2.8:DemographicdataforhouseholdandcommunitysurveysinMyanmaratT1andT2
*Data for Ethiopia were only available at T2 and thus were not included in the Difference-in Difference analysis
ORGANISATIONAL-LEVEL ASSESSMENT
Characteristics of Organisational survey respondents at T1
TheresponseratefortheorganisationalsurveyatT1was62.6%.Overall,DEPPstudyparticipantsoftheorganisationalsurveyatT1were27.9%femaleand72.1%maleandaveraged42.8yearsof age.Most participants held a university degree (33.1%) ormaster’s degree (54.1%) andworkedatinternationalnGOs(36.6%),nationalnGOs(26.2%)andlocalnGOs(30.2%).Themajorityofparticipantswereinsenior-levelpositions(90.1%).nearlyhalfoftheparticipantshadbeenworkingattheircurrentorganisationbetween10and15ormoreyears(56.3%)andhadbeenworkinginthehumanitarianfieldfor10to15ormoreyears(72.8%).
Characteristics of Organisational survey respondents at T2
The response rate for the organisational survey at T2 was 72.1%. Overall, DEPP studyparticipantsoftheorganisationalsurveywere25.9%femaleand74.1%maleandaveraged41.9yearsofage.Mostparticipantsheldauniversitydegree(32.9%)ormaster’sdegree(51.0%)andworkedat internationalnGOs(34.3%),nationalnGOs(30.6%)andlocalnGOs(32.6%).Themajorityofparticipantswereinsenior-levelpositions(87.4%),hadbeenworkingattheircurrent organisationbetween5 and10 years (25.8%) or between10 to15ormore years(31.1%).nearlyhalfhadbeenworking in thehumanitarianfield for10to15ormoreyears(48.39%).
Characteristics of KAP survey respondents at T1
TheresponseratefortheKAPsurveyatT1was79.8%.Overall,DEPPstudyparticipantsoftheKAPsurveywere36.5%femaleand63.5%maleandaveraged36yearsofage.Mostparticipantsheldauniversitydegree(49.5%)ormaster’sdegree(38.4%)andworkedatinternationalnGOs(43.2%),nationalnGOs(19.2%)andlocalnGOs(30.0%).Themajorityofparticipantswereinmid-level (40.9%)or senior-levelpositions (44.5%).nearlyhalfof theparticipantshadbeenworkingattheircurrentorganisationbetween1and3years(25.9%)or3and5years(19.0%)andhadbeenworkinginthehumanitarianfieldbetween3and5years(14.4%)or5and10years(25.7%).
Characteristics of KAP survey respondents at T2
The response rate for the KAP survey at T2was 72.1%.Overall, DEPP study participantsof the KAP survey were 30.9% female and 69.1% male and averaged 38.5 years of age.Most participants held a university degree (42.0%) ormaster’s degree (42.6%) andworkedatinternationalnGOs(37.2%),nationalnGOs(18.0%)andlocalnGOs(31.5%).Themajorityofparticipantswere inmid-level (44.4%)orsenior-levelpositions(47.2%).nearlyhalfoftheparticipantshadbeenworkingattheircurrentorganisationbetween1and3years(24.0%)or3and5years(21.0%)andhadbeenworkinginthehumanitarianfieldbetween3and5years(15.9%)or5and10years(28.4%).
COMMUNITY-LEVEL ASSESSMENT
Characteristics of Household Survey Respondents at T1
TheresponserateforthehouseholdsurveyatT1was98.1%.Demographiccharacteristicsofthe1,004respondentswhoparticipatedinthehouseholdsurveyinMyanmararepresentedinTable2.8andAnnex9.Theaverageageofrespondentswas44.2yearsandthesamplewas
71 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
roughlyhalfmale(49.2%)andhalffemale(50.8%).Themajorityofrespondentshadattendedsomeschooling(75.5%)andwereliterate(71.9%).Theaveragehouseholdsizewas5.2individualsperdwelling,andapproximately64%ofthesamplereportedtobechristian.
Characteristics of Household Survey Respondents at T2
Theresponserateforthehouseholdsurveyin?MyanmaratT2was99.9%.Theaverageageofrespondentswas44.7yearsandthesamplewasroughlyhalfmale(48.9%)andhalffemale(51.1%).Themajorityofrespondentshadattendedsomeschooling(78.0%)andwereliterate(74.3%).Theaveragehouseholdsizewas5.4individualsperdwelling,andapproximately68%ofthesamplereportedtobechristian.
Characteristics of Community Survey Respondents at T1
TheresponserateforthecommunitysurveyatT1was100%.Thissurveywascompletedwithcommunityleaderswithin19villagesinMyanmar.Theaverageageofrespondentswas41.8years.94.9%ofsamplerespondentsweremale,whileonly5.1%ofrespondentswerefemale.
Characteristics of Community Survey Respondents at T2
TheresponserateforthecommunitysurveyatT2was100%.Thissurveywascompletedwithcommunityleaderswithin20villagesinMyanmar.Theaverageageofrespondentswas40.8years.97.4%ofsamplerespondentsweremale,whileonly2.6%ofrespondentswerefemale.
MINIMUM SET OF EVALUATION ACTIVITIES Qualitative Interviews – Minimum Set of Evaluation Activities at T1
Atotalof51 individualsparticipated in theminimumsetqualitative interviewsatT1witharesponse rateof83.6% (i.e., 61were approached and10people didnot participate in thestudy).Ofthe51studyparticipantsinterviewed,sixwereDEPPstakeholders,28country-levelprojectleadership,and17UK-basedprojectleadership.Overall,studyparticipantswere37.3%femaleand62.7%malewithanaveraged39.8yearsofage.Moststudyparticipantshavebeenworkingattheircurrentorganisationbetween1and3years(49%)and3and5years(19.6%),andhavebeenworkinginthehumanitarianfieldbetween1and3years(15.7%),3and5years(21.6%),5and10years(23.5%),10and15years(13.7%)andmorethan15years(19.6%).
Qualitative Interviews – Minimum Set of Evaluation Activities at T2
Atotalof51 individualsparticipated in theminimumsetqualitative interviewsatT2witharesponserateof88.1%(i.e.,60individualswereapproachedand9individualsrefusedordidnotrespond).Ofthe51studyparticipantsinterviewed,fivewereDEPPstakeholders,31werecountry-levelprojectleadership,and15wereUK-basedprojectleadership.Overall,57.7%ofstudyparticipantswerefemaleand42.3%weremale.Theaverageageofstudyparticipantswas40.1years.Moststudyparticipantshavebeenworkingattheircurrentorganisationbetween1and3years(48.9%)andhavebeenworkinginthehumanitarianfieldbetween5and10years(37.0%),10and15years(13.7%).
Optionalqualitativefollow-upquestionnairesweresentatT2to43projectandprogrammestakeholders following in-depth interviews to provide respondents with an additionalopportunitytoshareinformationandperspectives.Ofthissample,twoparticipantsresponded.TheresponserateforthequalitativequestionnairesatT2is4.7%.
NETWORK SURVEY
Table2.9belowprovidestheoverallnumberoforganisationswhorespondedtothenetworksurvey,thetotalnumberoforganisationstheyidentifiedascollaboratorsandthetotalnumberoflinksbetweenorganisationsidentifiedineachcountry.linkscorrespondtodifferentformsofcollaborationandaredescribedinmoredetailinchapter5.
72 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DuringT1,thetotalnumberoforganisationswhorespondedtothesurveywasnearlythesameinthePhilippinesandKenya(72and73respectively),howeverthenumberoflinksidentifiedwasmorethandoubleinthePhilippines.InT2,thenumberoforganisationswhoparticipatedinthenetworksurveyinKenyaincreasedfrom73into142,withnearlyfourtimesthenumberoftotallinks.OrganisationsinthePhilippineswhotookthesurveyincreasedbyapproximatelyonethird,howevertherewasa48%decreaseinthetotalnumberoflinks.Duringthevalidationworkshop in the Philippines, event participants attributed the decrease in number of linksto survey fatigue. largenumbers of organisational partners increased thetime required tocompletethenetworksurvey.Therefore,aftertheirexperiencesinT1,respondentslimitedthenumberoforganisationstheyreportedascollaboratingpartners,astheyknewthata largernumberofpartnersincreasedthetimeneededtocompletethesurvey.
InMyanmar,fewerorganisationscompletedthenetworksurveyinbothtimepointsbecauseof the small number of DEPP projects active there relative to the other countries. fewercollaborating organisations were identified, which in turn resulted in the identification offewerlinks.InEthiopia,thetimebetweenT1andT2waslessthantheothercountriesduetothedelays incontractingoutlinedinthe limitationssection,andthereforethenumberoforganisationsidentifiedwassimilarinbothtimepoints.However,thenumberoflinksincreasedby44%atT2.Analysesofthenetworksintheintensivesetcountriesaredescribedinfurtherdetailinchapter5.
COUNTRY AND TIME OF DATA COLLECTION
NUMBER OF ORGANSIATIONS
WHO COMPLETED NETWORK
SURVEY
TOTAL NUMBER OF ORGANISA-
TIONS IDENTIFIED
TOTAL NO. OF LINKS
The Philippines
T1 72 401 5622T2 92 254 2754
KenyaT1 73 161 2109T2 142 317 8027
MyanmarT1 33 75 207T2 30 55 226
EthiopiaT1 63 132 2438T2 59 137 4150
Table2.9:numberoforganisationsparticipatinginthenetworksurvey,identifiedorganisationsandlinksbycountry
73 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
DOCUMENT REVIEW
Intotal,634documents,includingreports,casestudies,videos,podcasts,articles,handbooksandothermaterialswerereviewed(Table2.10).Thisincluded334learningandevidenceitemsthatareavailablefordownloadontheDEPPlearningPlatform,whichrepresentsabout52.7%ofthetotalitemsreviewed.
TYPE OF RESOURCE TOTAL NUMBER REVIEWED
NUMBER (%) AVAILABLE ON LEARNING PLATFORM
Quarterly Reports 167 N/ABlog 80 77 (96.3%)Report/ Review 76 50 (65.8%)Case Study 68 43 (63.2%)Video/ Twitter Video 60 53 (88.3%)Key Findings/ Key Learnings/ Lessons Learned 38 37 (97.4%)Guidelines/ Minimum Standards 21 15 (71.4%)Article 19 9 (47.4%)Programme and Project-level Logical Frameworks 16 N/APodcast 14 14 (100.0%)Final Project Evaluation 13 0 (0.0%)Final Report Narratives 10 N/ATemplate/ How-to Guide/ Toolkit 9 8(88.9%)Concept Note 8 0 (0.0%)Summary/ Executive Summary/ Report Summary 6 6 (100.0%)Handbook/ Field Guide/ Framework 6 6 (100.0%)Terms of Reference/ Proposal 6 2 (33.3%)Presentation 4 3 (75.0%)Research/ Analysis 3 3 (100.0%)Policy Paper 2 2 (100.0%)Resource Catalogue 2 2 (100.0%)Business Case 1 0 (0.0%)Methodology 1 1 (100.0%)Slides 1 1 (100.0%)Mobile App 1 1 (100.0%)Newsletter 1 0 (0.0%)Reflection Journal 1 1 (100.0%)TOTAL 634 334 (75.7%)*
Table2.10:Summaryofdocumentsreviewed
*Does not include documents that are not applicable
74 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
HUMANITARIAN RESPONSE ASSESSMENT AND CASE STUDY
Atotalof22 in-depth interviewswereconducted for theKenyaandEthiopiacasestudies.InKenya,16 in-depth interviewswereconductedandanalysed.Of the16respondents,11(68.8%)weremaleandfive(31.2%)werefemalewithameanageof36.7years.Atotalofsevenorganisationswererepresentedintheinterviews.Respondentsreportedworkingonaverage7.6yearsinthehumanitariansectorand6.2yearsfortherespectiveorganisation.
InEthiopia,six in-depth interviewswereconducted.Ofthesixrespondents,fiveweremale(83.3%)andonewasfemale(16.7%).Themeanageofintervieweeswas42.8years.ThesixorganisationsrepresentedwerelocalbeneficiaryorganisationsaffiliatedwiththeShiftingthePowerproject.Respondentsreportedworkingonaverage7.8yearsinthehumanitariansectorand9.2yearsfortherespectiveorganisation.InSouthSudan,threeinterviewswereconducted.Giventhesmallnumberofinterviews,demographicdataarenotprovidedtoprotecttheidentityofthestudyparticipants.
LIMITATIONS Therearenotablestrengths to theevaluationactivitiesanddesign.These include thewiderangeofstudyinstrumentstocapturedataatdifferentlevelsofactionandadiversesampleofDEPPorganisations,programmeleadership,andprojectleadershipacrossDEPPcountriesandprojectswhowereinterviewedandsurveyed;however,itisimportanttoconsiderthestudy’slimitations.
Theresponseratefortheintensiveandminimumsetofevaluationactivitieswas,ingeneral,very high at both time points andwithin an expected range given the evaluation targetedsample. At T1, the response rate was 79.8% response rate for the KAP, 62.6% for theorganisationalsurvey,98.1%forthehouseholdsurvey,100%forthecommunitysurveyand83.6%forin-depthinterviews.AtT2,theresponseratewas72.1%fortheKAP,72.1%fortheorganisationalsurvey,99.9%forthehouseholdsurvey,100%forthecommunitysurveyand85%forin-depthinterviews.Responseratewaslowestfortheorganisationalsurveyatbothtimepoints.
Theorganisationalsurveywasadministeredtocountrydirectorsoforganisations,whowereoftendifficulttoinvolvebecauseofbusyschedulesandtravel.Tomitigateanypotentialbiasesa low response rate for the organisational surveymight introduce, various strategieswereemployed to increase participation. These included multiple contact attempts, flexibility intermsofinterviewdatesandtimes,andanoptionforaremoteinterviewiftheindividuals’travelschedulesdidnotpermitaface-to-faceinterview.fortheothertypesofsurveysandinterviews,non-responderswereprimarilyorganisationalleadershipandhumanitarianprogrammingstafffromimplementingandbeneficiaryorganisationsthatwereunavailabletoparticipateduetoschedulingconflicts.
HouseholdandcommunitysurveyswereconductedinMyanmarinbothtimepoints,butwereconductedinEthiopiaonlyduringT2duetocontractualdelayswiththelocalresearchpartner.Therefore, theresponseratesforthesecommunity-levelsurveysreflectonlydatacollectedinMyanmar.TheEthiopiadatawerenotusedfortheDifference-in-Differenceanalysisasthisrequirestimeseriesdata.
The majority of non-responders (n=10 at T1; n=9 at T2) for the minimum set qualitativeevaluation activities were primarily project leadership, and on subset analysis, they wererelatively evenly distributed among countries and projects indicating that there was nosystematic difference between responders and non-responders.Within the overall sample,therewerefewer interviewsconductedamongprogramme leadership,as theseparticipantsconstitutedamuchsmallergroupofstaffmembersthanthecombinedprojectleadership.The
75 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
evaluationteamalsocontactedkeyprogrammeandprojectstaffwhowerenolongerpartoftheDEPPtoensurethat theirviewswere included.Theviewsof therespondentsmaynotbe representativeofallof theprogrammestaff,fieldstaffandothers;however, therewerecommonthemesamongthepeopleinterviewed,andthesedataweretriangulatedwithotherevaluationdatatoensurevalidityandaccuracy.
It is importanttoconsiderthepotentialsourcesofbiasfortheaforementionedsurveysandinterviews. The lack of a true baseline is an important limitation for the evaluation. Datacollectionatthestartoftheprogramme,priortoanyprojectimplementationwasimpossibleduetothetimingoftheselectionandcontractingoftheevaluationteamwhichoccurredafterprojectshadbegunactivities.Sincethefirstquantitativedatacollectionpoint (T1)occurredafterDEPPimplementation,thefullimpactoftheDEPPmaynotbecaptured.AnychangesinoutcomesthatoccurredpriortoT1werenotbecapturedinthequantitativeanalysis,whichmeasureschangesbetween the twodatacollectiontimepoints.However,non-quantitativecomponentsoftheevaluationsuchasin-depthinterviewsanddocumentreviewwereassessedforevidenceofanyoutputsandchangesthatoccurredpriortoT1.
The interviewers assured respondents that privacy and confidentialitywould be respectedat all times, however social desirability bias could have affected responses. Since manyrespondentsareawareoftheexpectationsoftheDEPPandtheevaluation,theymayhavelimitedtheirresponsestowhattheythoughtDEPPmanagementortheenumeratorwouldwanttohear.OurteamutilisedenumeratorsandstaffwhoareexternaltotheDEPPandensuredthat respondents understood that their responseswould not be shared, andwould remainconfidential. Respondents were encouraged to share their honest opinions and reassuredthattheirresponseswouldhavenoimpactontheircurrentemployment.feedbackfromdatacollectionteamsinallcountriessuggeststhatrespondentswereveryforthcomingwithbothpositiveandnegativefeedbackontheDEPP,andthatsocialdesirabilitybiaswasminimised.Recallbiasmayalsohavebeenan issue,asrespondentsmaynotalwaysbeabletoprovideanaccurateaccountofpastactivitiesorexperiencesdatingfromthebeginningofDEPP. Inmanycases,therecallperiodwaslimitedtooneyearorlessinordertominimisethepotentialimpactofrecallbiasontheevaluation.Exclusionbiasmayalsohaveaffectedthestudysincein some settings such asKenya andMyanmar, security-related challenges limited access tocertainorganisations.Inthesesettings,effortsweremadetoremotelyincludeindividualswhocouldnotbephysicallyaccessedduetosecurityissues.Inaddition,topermittheDifferenceinDifferenceanalysis,manyofthesamequestionswereaskedinthesurveysduringT1andT2.Theremayhavebeenalearningeffectwhererespondentsanticipatedthequestionsaskedandpreparedtheirresponseaccordingly.ThiseffectisminimalbutitsimpactmaybeseeninthenetworkdataforthePhilippines,forexamplewhererespondents limitedthenumberofcollaboratingorganisationsdescribedduringT2sothattheirinterviewswouldbeshorter.
Inmostcases,qualitativeinterviewsfortheminimumsetofevaluationactivitieswereconductedviaSkypeandinEnglish;therefore,itispossiblethattheperspectivesofindividualswhomayhavehad limited Internetconnectivityand/orwhodidnotspeakEnglishwerenot included.However,theevaluationteamsampledindividualssystematicallyandnorespondentsdeclinedtoparticipateduetolanguageissues.Ininstanceswhereinternetconnectivitywasanissue,anoptiontoconductaphoneinterviewwasprovided.Therewereminortechnicalissueswiththeclarityandcompletenessoftheaudiofilesduringthedataextractionduetotheprogrammeplatform used to record the Skype interviews. However, the interviewers took notes andthemajority of files did not have any technical issues.Qualitativeminimum set interviewswere conducted by the same three individuals.noteswere compared and the same threeinterviewersextracteddatafromtheinterviews.ItispossiblethattherewasinterviewerbiasaseachindividualhadvaryingamountsofexposuretoandknowledgeofDEPP.However,theinterviewersfollowedadetailedsemi-structuredinterviewguideandaskedsimilarquestionswithsimilarprobestominimisepotentialinterviewerbias.Recallbiasmayalsohavebeenanissue,asrespondentsmaynotalwaysbeabletoprovideanaccurateaccountofpastactivitiesorexperiences.However,theevaluationteamprovidedthequestionareastotherespondentsaheadoftimeandencouragedthemtoshareadditional informationwiththeteamaftertheinterviewtoaddresstheissueofrecallbias.
76 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
It is also possible that individualswho did not speak Englishwere not fully represented inthe quantitative questionnaires. However, the online quantitative questionnaires from theminimumsetweretranslatedtofrenchtoensureinclusionofsampledrespondents.Intensivesetquestionnairesweretranslatedtothelocallanguagesandadministeredbylocalenumeratorstoensureinclusion.
Another limitationwhichmight have affected the evaluation is the high turnover ofDEPPprogramme and project leadership. A number of respondents interviewed during T1 wereunabletospeakdirectlyaboutthedesignprocessoftheDEPPprogrammeandofindividualprojects,astheywerehiredtoreplacestaffwhomayhavebeenengagedinthedesignprocess.HighstaffturnoverisarecurringthemethathasemergedbothwithrespecttothedesignandimplementationoftheDEPPandthathashadnotableeffects.Oneconsequencewithrespectto this evaluationhasbeen the lossof internal knowledgeabout topics suchas thedesignprocess and relevant lessons learned.To counter thispotential bias,we included staffwhowerenolongerpartoftheDEPPinoursample,andmadesubstantialefforttoensurethattheirperspectiveswereincluded.
Anotherpotentialweaknessistheinclusionofcommunity-leveldatainonlytwoofthefourintensive set countries (Myanmar andEthiopia). The inclusionof twocountrieswaspart oftheevaluationteam’smethodologyasoutlined inthe inceptionphaseandwaspartiallyduetobudgetconstraints.However,thisdesignalsotakesintoaccountthefactthatcommunity-leveleffectswerenotexpectedineverycountry,asprojectactivitiestargetingthecommunitylevelwerenotimplementeduniformlyacrosstheprogramme.Thetwocountriesselectedforcommunity-leveldatacollectionwereappropriateforassessingDEPPatthecommunitylevel.
ThedocumentreviewwasdependentonwhatwassharedbythelearningProject,andavailableonthelearningplatformandonbox,anonlineplatformusedtostoreprojectandprogrammedocuments. Documents and resources uploaded to box varied by project and programmeleadership,ascertainstakeholdersdidnotuploaddocumentssystematically.Everyeffortwasmadetolocatecertaindocumentsiftheycouldnotbelocatedinthesharedfolders.
There were unanticipated delays during the in-country data collection for the intensivesetof evaluationactivitieswhichnecessitated additionaltime tomaintain the integrity andcompletenessofthesedatacollectionmethods.Insomecases,thiswasduetodelaysinsettingupcontractualagreements,obtainingrespondentlistsandcontacts,challengeswithsecurityaswellaslimitedaccesstoareaswhereconflictorviolencewasoccurring.forexample,accesstoDEPP communities and households inMyanmarwas limited due to insecurity and datacollectionwasdelayeduntilsafeaccesscouldbeguaranteed.InSouthSudan,plannedintensiveevaluationactivitieswerenotconductedduetoanescalationofviolencethathaltedDEPPactivities in country and instead datawere collected remotely as part of theminimum setevaluationactivities.contractualdelayswereasubstantialissueinEthiopia,whereanagreementcouldnotbereachedbetweentheselectedlocalresearchpartnerandHHI.Ultimatelyanewlocalpartnerwasidentifiedandcontracted,howeverthesedelaysleddatacollectiontobeginlaterinEthiopiaforT1comparedtotheotherintensivesetcountries.ThetimeperiodbetweenEthiopiaT1andT2fortheorganisational,KAPandnetworksurveysinEthiopiaisthereforeshorter than for theothercountries. Inaddition, asmentionedabove,due to thesedelays,datacollectionatthecommunitylevelwasonlycollectedatonetimepointandthereforeaDifference-in-Differenceanalysiswasnotpossiblewiththeseparticulardata.
finally,thestudycouldalsobepotentiallyaffectedbybiasattheleveloftheevaluationteam.However it is importanttonotethattherewerenoconflictsof interest,andtheevaluationteamwasabletoworkfreelyandwithoutinterference.
77 SUMMATIVEPHASEREPORT-EVAlUATIOnMETHODOlOGy
METHODOLOGICAL CONSIDERATIONS AND APPROACH
TheevaluationmethodologywasdevelopedbasedoncriteriaadaptedfromtheDAcprinciplesfor Evaluating Development Assistance, and in alignment with the modified causal chaindevelopedbytheevaluationteam,ratherthantheprogramme’stheoryofchangepresentedin the business case which was found to be flawed, unnecessarily complex and not fullyrepresentativeofthetypesandscopesofDEPPactivities.
Arandomised-controlledtrial(RcT)wasanimportantelementoftheinitialevaluationproposalbecauseoftherigorousevidenceofimpactthatcanbeobtainedwiththismethod.OneoftheaimsduringtheinceptionphasewastothoroughlyinvestigatethefeasibilityofconductinganRcT.Afterextensiveconsultationsduringtheinceptionphaseandreviewofprojectdocumentsandplans,theevaluationteamconcludedthatanRcTwouldnotbefeasibleforvariousreasons.first, project activities across theDEPPwere extremely complex and broad in scope, andtargetedmultiplelevelsincludingcommunities,organisations,andsystems.Additionally,inmanycasestheprojectactivitieshadalreadybegunandwerebeingdeliveredtobeneficiariesbasedonnon-randomcriteria.Thus, theminimumconditions required for anRcT–a reasonablydiscreteinterventionwithalargenumberofunitsthatcouldberandomised,andtheabilitytomaintainunexposedcontrolgroupsduringthetimeframe–couldnotbemet.Theevaluationteam therefore opted to integrate quasi-experimental designs, which include a pre-postdesignandanappropriatecontrolgroupforcomparison,wherepossible.Thisapproachwassupplementedwiththeothermethods(qualitativedatacollection,documentreview,economicanalysis, network analysis, and in-depth case study of emergency responses) to provide amixedmethodsdesignwithmultiplesourcesofdatacapturedatthedifferentlevelsofaction(individuals,communities,organisationsandsystem).Theresultsweretriangulatedtotesttheprogramme’stheoryofchangeanditsunderlyingassumptionsandtoanswerkeyevaluationquestions.
The evaluation methodology – combining multiple research streams in a mixed methodsapproach–attemptedtocapturethebroadscopeoftheDEPP.Theuseofquasi-experimentalmethodsdrewuponinnovativepublichealthandmedicalresearchmethodologiestorigorouslyevaluatecapacitybuilding,andwascombinedwithqualitativeandparticipatoryapproachestocreateanintegrated,multi-dimensionalviewofDEPPprogramming.SincetheDEPPfocuscountriesdiffersignificantlyintermsofcontextualfactorssuchasgovernancestructures,anddisasterandpreparednessprofiles, itwas important toassesshowthesedifferentcontextsinfluencethetheoryofchange,programmeimplementationandoutcomechanges.contextualfactorsateachsiteweredocumentedtocontributetounderstandinghownon-DEPPfactorsinfluenceprogrammedeliveryandoutcomechanges.
ADDRESSING CROSS-CUTTING THEMES
Theevaluationmethodologywasdevelopedtotakeintoaccountthecross-cuttingissuesofgender, poverty, human rights, environment, anti-corruption, capacity building and powerrelations.HIV/AIDSwasnotaspecificcross-cuttingthemerelatedtotheprogrammeandhencenodatawascapturedonthisissue.notethattheevaluationToRonlyrequiredassessmentofgender,inclusionandviolenceagainstwomenandgirls(VAWG)acrosstheDEPP.
Gender:Documentreview,observationandsitevisits,in-depthinterviews,andorganisationalassessmentfocusedonprojectandprogramme’sactivitiesandgenderstrategyandinclusion.Inthecommunity-levelassessmenthouseholdsurvey,theevaluationteamsoughttoincludeasampleof50%maleand50%femaleparticipantstogatherinformationfrombothgenders.All organisational-level assessment study tools also aimed for equal representation of bothgenders. In-country data collection teamswere formedwith particular attention to genderbalance.
Poverty:Documentreview,observationandsitevisits,in-depthinterviews,andorganisationalassessment explored issues relating to poverty and economic inclusion. Households in thecommunity-levelassessmentwererandomlyselectedtoensurerepresentationfromdifferentsocio-economicclasses,anddatawerecollectedonsocio-economicstatus.Otherstudytoolsassessedfoodsecurity,shelter,educationandaccesstoservices.
78 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Human rights:Document reviewandobservationandsitevisitsexplored issues relating tohumanrightsandensuredtheprojectsactivitiesdidnotdiscriminateagainstrace,sex,nationality,ethnicity,languageorreligion.Theelderlyandpeoplewithdisabilitieswereprioritisedwithinall community-level data collection to ensure that their perspectiveswere captured in theevaluation. Data collection protocols were developed to ensure protection of participants,minimiseanypotentialrisksandensureconfidentiality.Onlythoseparticipantswhoprovidedvoluntary,informedconsentparticipatedinthestudy.
Environment: Environmentalfactorsrelatedtodisastersandemergencieswereconsideredaspartofthecontextualfactorsinfluencingeachsettinganditsdisasterprofile.Atthecommunitylevel, data on environmental factors and sustainable environmentalmanagementwere alsocollected.Allevaluationdatawerecollectedusingelectronicquestionnairesprogrammedontabletstominimisetheevaluation’senvironmentalfootprint.
Anti-corruption:ThelocalresearchpartnerswerecarefullyvettedinaccordancewithHarvardUniversity’santi-fraudandanti-corruptionpolicies.Theevaluationteamworkedverycloselywith in-countrypartnersandmonitoredtheirworkduringon-sitevisitsandthroughregularmeetings.Whenpossible,observationandsitevisitstoDEPPprojectsallowedverificationthatactivitieshadbeenimplemented,includingverificationofpurchasedequipmentinuse.
Capacity Building:Asamainfocusoftheprogramme,allstudytoolsaimedtocapturecapacitybuildingandeffectivenessofactivitiesamongDEPPstakeholders.
Power relations:In-depthinterviewsandobservationandsitevisitsexploredpowerrelationsbetweenInGOsandn/lnGOs,anddifferentlevelsofprojectandprogrammestaffinordertobetterunderstandrelationshipswithintheprogrammeandtheireffectonimplementation.
79 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn
figure2:DEPPProjectDescriptions
RELEVANCE AND VALIDITY OF DESIGN
cHAPTER3
80 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Evaluation Question #1: To what extent are the objectives of the programme intervention consistent with stakeholders' requirements and the programme design logical and coherent?
Sub-questions:a) Hastheprogrammetargetedtherightpeopleintherightplaces?b) Towhatextentdoestheprogrammedesign(theoryofchange)supporttheprojects’design(logical
framework)?c) In what ways was the programme design process participatory? Were project beneficiaries
adequatelyengagedbefore,duringandafter?d) Towhatextentwastheprogrammedesignlogicalandcoherent?
a. Weretheobjectivesoftheprogrammeclear,realisticandlikelytobeachievedwithintheestablishedtimescheduleandwiththeallocatedresources(includinghumanresources)?
e) Haveprioritisedtargetgroups(peoplewithdisabilities,olderpeople)andgenderaspectsbeentakenintoconsiderationintheprogramdesign?
Data sources: • Inception phase interviews (Qualitative)• Minimum set IDIs with programme and project stakeholders(Qualitative)• KAP and organisational surveys(Quantitative)• Intensive set IDIs(Qualitative)• Document review(Qualitative)Key Findings:• Thefiveresultsareas(objectives)oftheDEPPareappropriate, and the right people in the right
places were targeted• Athighlevel,programme objectives are clear, relevant and aligned with DFID priorities. However,
the three-year time frame was insufficienttoreachprogrammeobjectives• Thecountry selectionprocesswasiterative,basedonappropriatecriteriabut was not optimal
due to lack of strategic direction and objectives at portfolio level• There isgood alignment between the project’s design and the overall programme theory of
change. However, the programme-level logical framework was finalised late in programmeimplementationandreportingagainstthefinalversionbeganinAugust2017,duringthe3rdyearofDEPPimplementation
• Thedesign process was not logical or coherent,especiallyduringthefirstSTARTnetworkdesignphasewhenprojectswereretrofittedtothebusinesscase
• Designattheprogrammelevelwasnotadequatelyconsideredandprojects were designed in isolation with limited local consultation and buy in.Despite theflaws in the designprocess,the selected projects did address country needs once they were contextualised during theirimplementationperiods.However,programmesystemsandprocesses,theprogrammetheoryofchange,definitionofkeyterms,andexpectationsabouthowprojectsareintendedtointerlinkandinteractwerenotdevelopedandarticulatedattheoutset.
• Variation in number and type of projects in each country was a lost opportunitytomaximiseefficienciesandimpact
• Resourcing was sufficient at the programme level, but individual project budgets were leanwithinadequateresourcingforM&E,nPAcsandcollaboration
• All projects considered gender in their design, but the level of consideration varied. Someprojectsconsideredotherprioritisedgroupssuchastheelderlyandpeoplewithdisabilitiesintheirdesign
• Response to emergencies was not adequately considered during the design phase – theprogrammewas designed to improve emergency preparedness butmechanisms to enable orfacilitatel/nnGOcontributiontohumanitarianresponsewerenotexplicitlyintegratedintotheprogramme.Despitethis,DEPP contributed to at least 42 humanitarian responses in 11 countries.
81 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn
OVERVIEW OF THE DEPPDESIGN PROCESS ThedesignprocessfortheDEPPaswellastheindividualprojectswithintheDEPPportfolioissummarisedbelowbasedonfindingsfromin-depthinterviewsanddocumentreview.figure3.1showsthesequenceofstepsinthedesignprocess.
In response to the Humanitarian Emergency Response Review (HERR) launched in 2011,whichincludedacalltoactionforthebritishGovernmenttosubstantiallymodifyitswayofresponding tohumanitarian crises,DfID launched thebusiness casedevelopmentprocess.Evidence suggests that early in the business case development process,DfIDhad alreadyidentified theconsortiumofbritishHumanitarianAgencies (whichwould laterbecome theSTART network) and the cDAc network as potential partners. Once the business casedevelopmentwasunderway, theSTARTnetworkand thecDAcnetwork,undertook theirown,separatedesignprocessesrelatedtotheDEPP.TheSTARTnetworksecretariatsolicitedproject ideasfromits (then)19members in2013throughageneralcall thatwascirculatedinternallytoitsmembers.Intotal,overfiftypre-generatedideaswereputforwardbySTARTnetworkmembers.AllfiftywereconsideredatanIdeasWorkshopheldinSeptember2013,whichinvolvedprimarilylondon-basedheadquartersstaffwhocametogethertodiscussandconsolidatetheideas.Theideaswerereducedto18bytheendoftheday,buttheprocessinvolvedmergingsomeprojectstogether,leadingto,insomecases,theformationofartificialormandatedconsortia.
“Themashingworkshopput together four streamsof individual projects, projectsthatbuildindividual’scapacity,andmashedthemintosomethingthatperhapsifyouhadablanksheetofpapershouldn’thavelivedtogether…Wewereperhapsforcingpeopletocollaboratewhoshouldn’thavebeen...thisthenaffectedthequalityofthecollaborationbetweenthemembers.”(Programmeleadership)
Theprocessofconsolidatingandmergingprojectswasnotbasedonanyparticularstrategyordefinedcriteriaandlackedinvolvementfromin-countrystaff.Therewasapreferenceforprojects thatwere already being implemented or that agencieswere hoping to implementratherthannew,innovativeideas.Thecreationofsomeartificialconsortiahadlongreachingimpacts.OneoftheauthorsoftheDEPPdesigncasestudy75providedoneexample:“Threeagencies brought ideas for humanitarian training schemes that they were already runningwithgreatsuccess.Thesethreeagencieswerebroughttogetherandtaskedwithcreatingoneconsortiumprojectoutofthethree.Atfacevalue,thisseemedtomakesenseinthenameofcollaborationandefficiency;however,thehightransactioncostsofworkingintheconsortiummayarguablyinitiallyhaveoutweighedanyperceivedadvantageofworkingtogether.”
Thecasestudyalsodiscussessomeoftheconsequencesofthesemandatedcollaborations:
"Someconsortiathatweremandatedtoworktogetherstruggledtoreconciledifferinginterests and opinions. Partners reported that these frustrations undermined thecollaboration,andwhichrequiredsustainedefforttodisentanglelaterintheproject’slife-cycle,affectingefficiencyandeffectivenessduringdelivery.”
75 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisastersandEmergenciesPreparednessProgramme.
82 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
figure3.1:SequenceofstepsintheDEPPdesignprocess
followingtheIdeasWorkshop,teams,whichagainwereprimarilyUK-basedstaff,weregiventhreemonthsbytheSTARTsecretariattoformtheirconsortia,includinggeneratingdetailedbudgetsandproposalsbeforepresentingthemtoapanelofsixjudgesaspartofaninternalpeerreviewprocess.However,atthistime,theDfIDbusinesscasehadnotyetbeenfinalised,norhadthespecificDEPPtargetcountriesbeenselected,whichmeantthatprojectscouldnotbedevelopedwiththoseinmind.Inaddition,therewasnotenoughtimetoinvolvecountryoffices,andlondon-basedstaffdidnotfeelitwasjustifiedtousethetimeandresourcesofcountrystaffwhentherewasnoguaranteeoffunding.Therewasinsufficienttimetosetupconsortiaorfullyconsiderimplementationdetailsorappropriateresourcing.Attheendofthisthree-monthperiod,submissionswerepeer-reviewedandninewereselectedtogoforwardtobepresentedtoDfID.Thepeerreviewprocesstookalow-riskapproach.Allofthisworkwasconductedbeforethebusinesscasewasfinalisedandapproved.Thisprocessofdevelopingprojectsbeforethefinalbusinesscasehadseveraleffectswhicharedescribedlater.
According to the DEPP design case study76: “the decision-making on DfID’s investmentincludedextremelylimitedconsultationwithnGOsandotherexpertstodevelopthebusinesscase,whichmeantthatrelevantlessonsfromthehumanitariansector’sbroaderexperienceofimplementingpreparednessprojectswasnotsystematicallyincludedintheevolvingbusiness
76 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisastersandEmergenciesPreparednessProgramme.
HERR Launched
DFID launches process for business case development
START Network solicits ideas from 19 members CDAC Network
CDAC Network - submits proposal to DFID Board
2011
Sept 2013
Oct-Dec2013
Feb 2014
April 2014
July 2014
Oct 2014
Nov 2014
Jan 2015
Jan 2015
April 2015
May 2015
83 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn
casefortheDEPP.”Inaddition,evidencesuggeststhattheDfIDcountryofficesdidnotfeedintothedecision-makingprocess,andthattherewasalsoalackofbroaderconsultation.Thefinalbusinesscasedidnot containbasicdesigndetails suchas focuscountriesandprojectselectioncriteria,andtherewasonlyonemonthbeforethefirstboardmeetingtodevelopthesesubstantialareas.
InApril2014, thebusinesscasewasfinalisedandDfIDannounced£40millionof fundingfortheDEPP,ofwhich£10millionwouldbeallocatedtowardan“innovationwindow”;thus,leaving£30millionforDEPPprojects.Thiswasallocatedtothetwopre-selectednetworks,withtheSTARTnetworkreceiving£27millionandcDAcnetworkreceiving£3million.
ThefirstDEPPboardmeetingwasheldduringthatsamemonthinApril2014.Elevencountries– Kenya, Ethiopia, Democratic Republic of the congo, South Sudan,Mozambique, Jordan,Myanmar, the Philippines, bangladesh, Pakistan, and Indonesia –were selected, andDEPPprojectselectioncriteriaandmonitoringandevaluationplanswerediscussed.
Thesecondboardmeeting,inJuly2014,ledtotheapprovalofthefirstfiveDEPPprojects:TransformingSurgecapacity (TSc),Shifting thePower (STP),TalentDevelopment (TD), theAgeandDisabilitycapacitybuilding(ADcAP)projectandtheMElproject(whichwaslaterrebrandedasthelearningProject).Threemoreprojectswereapprovedduringthe3rdDEPPboardmeetinginOctober2014:linkingPreparednessResilienceandResponse(lPRR),financialEnablers(fE),andProtectioninPractice.ThefinancialEnablersprojectwasconsideredhighriskbyDfIDandwasscaledbacksubstantially.Itwasapprovedforimplementationinonlyonecountryratherthanfourasoriginallyenvisionedandwithabudgetreducedby£3million.TheAlERTprojectwasapprovedinJanuary2015duringthe4thboardMeeting.
IncomparisontotheSTARTnetwork,thecDAcnetworkinstitutedadifferentdesignprocess.OncethebusinesscasewasfinalisedinApril2014,asmallgroupofinterestedcDAcnetworkmembersworkedovereightmonthstodevelopaprojectthatspecificallyfittheDEPPbusinesscaseandselectioncriteria.Thisprojectwassubmittedinnovember2014andapprovedduringthe4thDEPPboardmeetinginJanuary2015.
Afterthe4thDEPPboardmeeting,theSTARTnetworkhadfundsremaining(fromthecutstothefinancialEnablersproject)anddecidedtolaunchasecondwaveofproposaldevelopment,buildingonlessonslearnedduringtheinitialprojectdesignphase.contrarytothefirstroundofprojectdesignduringthedevelopmentofthebusinesscase,andinanefforttoimprovetheprocess, it significantly encouraged and incentivisednational project staff from itsmemberorganisations to take the lead, and to involve local stakeholders.One-pageproposalsweresoughtonprojectsthatfocusedonearlywarningsystemsinordertofillanoticeablegapintheDEPPportfoliowithrespecttoresultareatwofocusedondevelopmentofearlywarningsystems.The shift tovery shortproposalswasdone toenablenational project staff tobesignificantlyinvolved.consortiawerepermittedtodevelopnaturallyaroundsharedinterests,andlocalownershipwasencouraged.Acommitteeoftechnicalexpertsconductedareviewofproposals, and significant involvementof in-country teamswasa selectioncriteria.fourprojectsselectedthroughthisprocessweresubmittedtotheDEPPboardandwereultimatelyapprovedinApril2015.TheexternalevaluationwasapprovedinMay2015.
Thefollowingsectionspresentevaluationfindingsrelatedtotherelevanceandvalidityofthedesignprocess,aswellasitssuccessesandlimitations.
84 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
QUESTION 1.A HAS THE PROGRAMME TARGETED THE RIGHT PEOPLE IN THE RIGHT PLACES?
The established five results areas (objectives) of the DEPP are appropriate, and the right people in the right places are being targeted. However, the country selection process, while iterative and based on appropriate criteria, was not optimal due to lack of strategic direction and objectives at the portfolio level.
Theevaluationassessedthisquestionwiththreemainlenses:first,whethertherightcountrieswere chosenandwhether the country-selectioncriteriawere clear; second,howdecisionsweremadeabout thenumberofDEPPprojectspercountry;and third,whether theDEPPdesignisconsistentwithbeneficiaryneedsandrequirementswithincountries.
DEPP Targets and Country Selection Criteria
Ingeneral,thetargetsoftheDEPPareappropriate,andtherightpeopleintherightplacesweretargeted.Thefocusonbuildingnationalcapacity,improvingemergencypreparednessandtargetingvulnerablegroupstoensuretheirinclusionduringhumanitarianresponseactivitiesisconsistentwithneeds identifiedduring theevaluationteam’s literaturereviewconductedduringtheinceptionphase.Thefocusremainsalignedwithnewerglobalcommitmentsrelatedtohumanitariansystemreformandforlocalisationofhumanitarianaid77.
Documents suggest that the country selectionprocesswas iterative andbasedon severalcriteria – humanitarian need and likelihoodof natural disasters using the InfORM Index78,followedby feasibilityandopportunity tomake thegreatest impact.Theseareappropriatecriteria.However,accordingtotheDEPPdesigncasestudy79,agenciesperceivedthatDfIDpreferredimplementationincountrieswhereitalreadydirectlyworkedin,implementationinfewercountriestomaximiseimpactandtoworkinsettingsatriskofnaturaldisastersratherthanmanmadeasitwouldbelessrisky.Inaddition,sincethetwonetworkshadalreadybeenselected, theDEPP couldonly operate in countrieswhere thosenetworks had an existingpresence.Ultimately,however,somerespondentsfeltthatkeycountriesandprioritygeographicareas with strong humanitarian need were not included and attributed this mainly to thelimitationsofoperatingonlyincountrieswheretheSTARTnetworkandthecDAcnetworkmembershadapresence.byoperatingthroughpre-selectednetworks,thereisapossibilitythatstrongerorganisationsandinterventionswerenotincluded.Minimumsetintervieweesalsodescribedalackofstrategicdirectionandobjectivesattheportfoliolevel,whichhinderedthepotentialofselectingcountriesandprojectsbasedonstrategiccriteria(e.g.,contextwithabilitytomakebiggest impact,highestdisaster risk,greatestneed,and/ormostvalue formoney).AnotherintervieweestatedthattheselectionofDEPPcountriesbytheDEPPboardhadbeenaconvolutedprocesswithtoomuchinputfromtheSTARTnetwork,andemphasisplacedonexistingmembercoveragewithinthenetwork.ThisisconsistentwiththeDEPPdesigncasestudywhichpresentedproblemswiththecountryselectionprocess,withoneagencyreferringtothefinalsetoffocuscountries:“analmostrandomlist”80.
Coverage of DEPP Interventions Across Target Countries
Of the 11 target countries that were finally selected, projects were implemented in only10 countries (Indonesia ultimately was not targeted by any DEPP projects). Despite thecommitment towards the five result areas, the portfolio of interventions in each countryalsovariedgreatly,withsomecountriesreceivingveryfewprojectsandothersmanymore.Thisseemedtobeduetominimalconsiderationat theportfolio levelof thedistributionofDEPPinterventionsacrosscountriesandtheoptimallevelofexposuretoDEPPinterventionsneededtomaximisepotentialimpact.Thiscoveragestrategy,asdescribedlaterinthereport,
77 SuchastheGrandbargain-Workstream2andTheSendaiframeworkforDisasterRiskReduction.78 http://www.inform-index.org.79 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisasters
andEmergenciesPreparednessProgramme.80 KletzingM,bevanD.PreparingforPreparedness:lessonsfromDesigningandSetting-uptheDisasters
andEmergenciesPreparednessProgramme.
85 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn
diminishedthepotentialeffectivenessoftheprogrammeinsomeoftheDEPPfocuscountrieswithfewernumbersofprojects.Onerespondentfeltthattherewasahigherleveloflearningincountrieswithmoreprojectsduetoagreatercriticalmass.forexample,Pakistanhadsevenprojects,and this led toa lotoforganicactivitiesandcostefficiencies,withpeoplesharingresourcesandstartingnewprojects.ThefindingsonprogrammeimplementationlaterinthisreportsupportthehypothesisthattherewasmorecollaborationinsettingswherethereweremoreDEPPprojects. Insomecases,thiswasfurtherenhancedbythepresenceofRegionallearningAdvisors,orlaterthecountrylearningAdvisors(clAs),thatwereembeddedinmanyoftheDEPPcountriestostimulatecollaborationandsharingoflearning.
ThefindingsfromtheprimarydataarefurthersupportedbytheDEPPdesigncasestudy81.Thereportcitesseveralchallengeswiththecountryselectionprocess,indicatingthatthiswasneitherefficientnoreffectiveintermsofaligningprojectswithneedssystematicallyidentifiedupfront.Thereportstates:
“As a result of setting geographical criteria both after the business case and aftermostof theprojectshadbeendesigned,humanitarianneedforpreparednesswasnottheprimaryincentive.Someprojectssoughtapprovaltoimplementincountriesnotonthe list,andoneoftheofficialcountriesreceivednoinvestmentatall.AndalthoughtheDEPPwasdesignedwiththeaimofachievingfiveresults,thefocusofinterventionsvariedbetweencountries.”
Beneficiary Needs in DEPP Target Countries and Relevance
At the project level, many projects did not conduct needs assessments during the designprocess,whichcallsintoquestionwhetherthemostappropriateindividualsweretargetedandwhethertheirneedsandrequirementswereadequatelyaddressedwithineachcountry.Theprocessofretro-fittingpre-designedprojectstospecificcontextsoncethegeographiclocationswereselectedbytheDEPPboardandlackofinvolvementofthein-countrystakeholders,likelyaffectedtheinitialrelevanceoftheprojects.OncetheprojectsandcountrieswereapprovedbytheDEPPboard,mostprojectsreportedspendingasubstantialamountoftimecontextualisingtheirprojectstoeachcountryorsite.Whilethisledtodelays,itwasaimedatensuringrelevanceofthedesign.However,datasuggestthatthisprocessstilloftendidnotincludeconsultationswithbeneficiariessuchascommunities,households,andl/nnGOs,whichremainsasignificantweaknessoftheapproach.
As part of the quantitativeminimumand intensive set surveys,DEPP respondents (n=245)acrossall10DEPPfocuscountrieswereaskedabouttherelevanceoftheDEPPinterventionsintheircountry(seeTable3.1).Themajorityofrespondentsreportedtheinterventionstobeeitherrelevantorextremelyrelevant(93.9%overall)82.ThehighestreportedrelevancewasinEthiopiawhere98.2%ofintervieweesreportedtheDEPPtoberelevantorextremelyrelevant.
81 Ibid.82 These findings are consistent with the findings of the project final evaluations which find each of the projects to
be relevant to existing needs.
86 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ON A SCALE OF 1-5, HOW RELEVANT ARE THE DEPP INTERVENTIONS FOR THE COUNTRY IN WHICH YOU ARE WORKING?
All DEPP countries
N=245
(%)
Local Organisations
N=152
(%)
Minimum set
N=250
(%)
Ethiopia
N=54
(%)
Kenya
N=63
(%)
Myanmar
N=31
(%)
The Philippines
N=60
(%)
Not at all relevant
0
(0.0)
0
(0.0)
0
(0.0)
0
(0.0)
0
(0.0)
0
(0.0)
0
(0.0)
Irrelevant1
(0.4)
1
(0.7)
1
(2.7)
0
(0.0)
0
(0.0)
0
(0.0)
0
(0.0)
Somewhat relevant
14
(5.7)
10
(6.6)
4
(10.8)
1
(1.85)
2
(3.2)
2
(6.5)
5
(8.3)
Relevant150
(61.2)
94
(61.8)
14
(37.8)
38
(70.4)
41
(65.1)
19
(61.3)
38
(63.3)
Extremely relevant
80
(32.7)
47
(30.9)
18
(48.7)
15
(27.8)
20
(31.8)
10
(32.3)
17
(28.3)
Table3.1:RelevanceoftheDEPPinterventions
QUESTION 1.BTO WHAT EXTENT DOES THE PROGRAMME DESIGN (THEORY OF CHANGE) SUPPORT THE PROJECTS’ DESIGN (LOGICAL FRAMEWORKS)?
There is fairly good alignment between the project’s design and the overall programme theory of change. However, the programme-level logical framework was developed late in programme implementation and reporting against the final version of the logical framework began in August 2017, during the third year of DEPP implementation. TheDEPPbusinesscase includesatheoryofchangefortheDEPPandfocusesonfivekeyresultsareasasdescribedearlier.This theoryofchangewaspresumablyutilisedduring thedesignof someof theprojects.However,evidenceon theextent towhich theprogrammetheoryofchangewasusedtoguideprojectdesignorrefinementofprojectactivitiesislimited.DatafromtheinterviewsanddocumentreviewsupportthefactthatindividualprojectsinthefirstSTARTnetworkdesignphasewerenotdevelopedspecificallytofitthebusinesscaseorprogrammetheoryofchange.However,despitethis,thereisfairlygoodalignmentbetweentheprojects’designsandtheoverallprogrammetheoryofchange.Thiscouldhavebeenstrongerhadthedesignprocessunfoldedinalogicalsequence.OneothersignificantweaknesswasthelackofanoverallDEPPprogramme-levellogicalframeworkaswellasguidanceforindividualprojectsindevelopinglogicalframeworksandM&Esystemsthataligntogetherandwiththeprogramme. A programme-level logical framework was developed very late in programmeimplementationandwasinformedbytheexistingindividualprojectlogicalframeworks–thisprocessshouldhavebeenreversedwiththeprogramme-levellogicalframeworkdevelopedatthebeginningoftheprogramme.Thefinalversionoftheprogramme-levellogicalframeworkwasapprovedandreportingagainst itbegan inAugust2017, inthethirdyearoftheDEPPimplementation.consequencesofthisaredescribedlaterinthereport.
87 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn
QUESTION 1.CIN WHAT WAYS WAS THE PROGRAMME DESIGN PROCESS PARTICIPATORY? WEREPROJECT BENEFICIARIES ADEQUATELY ENGAGED BEFORE, DURING AND AFTER? The DEPP design process was participatory, but it did not adequately involve in-country beneficiaries and stakeholders, or local needs assessments. Overall, the design process for theDEPPwas participatory in that it involved consultationwithvariousstakeholders.However,beneficiariesatthelocalinstitutionallevelandintargetedcommunities and countries were not adequately engaged in the process, especially withrespecttotheSTARTnetworkprojectsdevelopedduringthefirstphase(SeeTable3.2).TheconsultationsthatdidoccurtendedtobeprimarilyattheUKlevelonlyandwereledbylargeInGOsandlondon-basedstaff.Giventhelackofinvolvementofin-countrypartnersandsmalleragencies,thedesignprocessdidnotfollowcurrentbestpracticesrelatedtoparticipatorydesignapproaches.furthermore,theDEPPdesignprocessfocusedmainlyondevelopingindividualprojectsinisolation,withlimitedconsiderationaboutsystemsattheportfoliolevel,andabouthowprojectsmightaligntogether.
furthermore,interviewdatasuggesttherewaslimitedinvolvementoflocalstakeholdersinthedesignoftheindividualDEPPprojects,aswellasalackofneedsassessmentswithinthetargetcountries83,84.Thisledtoinadequatecontextualisationandunderstandingoftheprojectsatthecountrylevel.Projectsthenhadtousetheinitialprojectimplementationperiodtoadapttheirprojectstothelocalcontextwhichinturncontributedtodelaysinprojectimplementation85.Moretimeandresourcesbuiltintotheprojects’inceptionphasecouldhavehelpedprojectstodevelopamorerealisticprojecttimeline.
ALL DEPP COUNTRIES
N= 51
UK-BASED PROJECT
LEADERSHIP
N = 17
IN-COUNTRY PROJECT
LEADERSHIP
N = 28
PROGRAMME LEADERSHIP
N = 6
The design process was participatory ✔ ✔ ✖/✔ ✔
The design process in-volved local stakeholders ✖ ✖ ✖/✔ ✖
There was sufficient time for collaborations during design process
✖ ✖ ✖ ✖
Table3.2:PerceptionsoftheDEPPdesignprocess
✔Mostrespondentsagreewithstatement ✖Mostrespondentsdisagreewithstatement✖/✔Responsesweremixed
83 Thisfindingissupportedbydocumentreview.forexample,TScfinalEvaluationReport,p17states:“Mostcountryplatformstakeholdersandsomeregionalandinternationalstakeholdersfelttheoriginaldesigndidnotadequatelyreflecttheprioritiesatthecountrylevel(forexample,thefocusonaccesstofundingfornationalorganisationswasastrongpriorityinthePhilippinesthatwasnotreflectedintheoriginaldesign)”.TheADcAPfinalEvaluationReport,p2states:“therewascriticismthatcountryofficesandnnGOswerenotinvolvedorinvolvedquitelateintheprocess.”
84 ThePHEPGambellaprojectwasoneofthefewthatconductedabaselinethatwasusedtoguideprogramimplementation.PHEPfinalEvaluationReport,p4:“Theprojectbenefitedfromabaselineassessmentwhichhelpedtoidentifywherecapacityshouldbetargetedandeffortprioritized.”
85 forexample,seeShiftingthePowerfinalEvaluationReport,p30,orTransformingSurgecapacityfinalEvaluationReport,p31.
88 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
These issues in the design of both theDEPP and the individual projectswere due to keyflawsinthedesignprocess–anunfavourablesequenceofstepsintheprocess,theselectionofprojects in tranches, and inadequatetimeand resources toengagesmallerorganisationswith limited resources.As illustrated infigure3.1 anddescribedearlier,mostof theDEPPprojectsandproposalsweredesignedbeforethebusinesscasewasfinalised,andthecountryselectionoccurredafterthebusinesscasewasapproved.Untilthecountryselectionoccurred,it would have been difficult to engage local stakeholders in the project development. Analternativescenariowouldhavebeentoconductaglobalassessmentduringthedevelopmentof thebusinesscaseandbefore thecountry selectionwasmade, to identifycountriesandbeneficiariesingreatestneed.Onerespondentstated:
“ThefirstroundoftheDEPPprojectswereabittoolondoncentric…IthinkalsoithastodowiththefactthatthebusinesscasecameverylatefromDfIDandthatalsothecountryfocuscameverylate.Wedidn’tknowuntiltheverylastminutewhichcountrywasgoingtobethefocus.So,itwasverydifficulttoengagewithourcountryteamsononesideandalsolocalpartners,becauseyoucreateaveryhighexpectationandyou’renotevensurethatthatcountryiseveneligible.“(Projectleadership)
Several exceptions exist; interviewees confirmed that the four projects that were fundedduringthelasttranche(UrbanEarlyWarningEarlyAction(UEWEA),StrengtheningEmergencyPreparednessSystemsinMyanmar(SEPS),EthiopiaEarlyWarningEarlyAction(EWEA),andPublicHealthPreparednessinGambella(PHEP))weredesignedthroughamoreparticipatoryprocessanddidinvolveconsultationwithlocalstakeholders.Thiswaslinkedtosmootherandtimelierprojectimplementation.
“forthesecondroundofprojectsthatweretheearlywarningprojects,therewasacleardirectivethattheyhadtobedevelopedwiththecountryofficedirectingthem,and I think that is possibly one of the reasonswhy these projects havemanagedtocatchupsoquicklyeven though theystartedalmostayear later thansomeofthe other projects. It’s because they’ve been developed in collaborationwith thecountry offices so the country office already had the buy-in and the drive to getgoing.”(Projectleadership)
Asthesefourprojectsunderwentamorestreamlineddesignprocessthattookintoaccountsomeoftheshortfallsofthefirstphase,thisfindingisnotsurprising.
QUESTION 1.DTO WHAT EXTENT WAS THE PROGRAMME DESIGN LOGICAL AND COHERENT? WERE THE OBJECTIVES OF THE PROGRAMME CLEAR, REALISTIC AND LIKELY TO BE ACHIEVED WITHIN THE ESTABLISHED TIME SCHEDULE AND WITH THE ALLOCATED RESOURCES (INCLUDING HUMAN RESOURCES)? The design process was not logical or coherent, especially during the first START Network design phase when projects were retrofitted to the business case. At the macro level, programme objectives are clear, relevant and aligned with DFID priorities but the three-year time frame was insufficient to reach programme objectives. Resourcing was sufficient at the programme level, but individual project budgets were lean with inadequate resourcing for M&E, Non-project Attributable Costs (NPACs) and collaboration. Toalargeextent,theDEPPprogrammedesignprocessdidnotfollowalogicalandcoherentapproach.Asdescribedearlier,mostprojectsundertheSTARTnetworkweredesignedpriorto thefinalisationof theDfIDbusinesscase, logical frameworkand theselectionof targetcountries.Uponfinalisationof thebusiness case,nineof thesepre-designedprojectswereretrofittedtoalignwiththebusinesscase,presentedtoDfIDandapprovedintranches.ThecDAcnetworkadoptedadifferent,morelogicalapproachwithrespecttothedesignprocess.Projectsweredevelopedoveralongerperiodoftimeandweredesignedtofitthebusinesscaseandselectioncriteria.nationalprojectstaffandlocalstakeholderstookalargerroleinthedesignofthesecondroundofSTARTnetworkprojectsandconsortiaweredevelopedbased
89 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn
onsharedareasoffocus.Thus,theevidencesupportsthatthesequenceofstepsinthedesignprocess,particularlyforthoseprojectsdesignedduringthefirstSTARTnetworkphase,werenottakeninalogicalandstrategicorder.
Were objectives clear and achievable within the timeframe?
Theevaluationteamreviewedtheobjectivesof theDEPPasawholeandof the individualprojects.Atamacrolevel,theobjectivesareclear,relevant,alignedwithDfID’shumanitarianpriorities86,87 andaimedtofillacleargapinhumanitariancapacitytoprepareforandrespondtodisasters.Asdescribedintheliterature,increasedfundingisneededforpreparednessactivitiesandforstrengtheningcapacityofnationallevelactorsratherthaninternationalactors88.Thereis a shortageofpeopleandsystemswith sufficientcapacity toprepare forand respond todisasters incountriesathighriskfordisasters,particularlyatthenational level89.TheDEPPobjectivestostrengthennationalcapacitytorespondtodisastersandincreasepreparednesslevelsincountriesatriskfordisastersarerelevantforthetenDEPPfocuscountries.However,the three-year time frame to achieve the selected objectives was unrealistic. A five-yeartimeframewould have beenmore appropriate for these specific objectives 90. In fact, thebusiness case had included a conditional commitment of two years additional yearswhichwouldhaveextendedtheprogrammedurationtofiveyears.Datafromthedocumentreviewsupportstheneedforalongertimeframe91,92,93.
These findings related to the design process, objectives and timeframe are supported byinterviewdataandotherdatacollectedacrossalltenDEPPcountriesduringtheformativephase.IntervieweeswerealsoaskedaboutthevalidityoftheDEPPdesigninordertounderstandhowspecificactivitiesrelatedtoDEPPobjectives.Overall,mostagreedthattheDEPPobjectivesareclearormostlyclear(SeeTable3.3).However,someindividualsfeltthattheobjectivesandfiveresultsareasthatprojectactivitiestargetweretoobroadandwouldhavebenefittedfromincreased specificity and refinement.However, thebroadnessof the results areaswas alsoconsideredapositivepointasitpermittedthediversityofdifferentprojectsthatexistedwithintheDEPPportfolio.Theremayhavebeensomeambiguityaroundtheprecisemeaningofthedifferentresultsareas,suchasresultsareafour(Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhumanitarianresponseandpreparednessarebettersupportedandmoresuitable).Differentagenciesappear tohave interpretedkey termsandphrasesdifferently.Theprogrammewouldhavebenefittedfromclearlydefiningkeyconceptssuchascollaborationandinstitutionalarrangementsearlyon.Thisinturnwouldhaveenabledprojectstousestandardisedmeasurementsanddevelopstandardisedindicators.
86 https://www.gov.uk/government/speeches/uk-aid-in-2015-the-progress-so-far-and-the-priorities-ahead.87 https://www.gov.uk/government/publications/dfid-single-departmental-plan-2015-to-2020/single-depart-
mental-plan-2015-to-2020.88 HolmesJ.HumanitarianResponseinthe21stcentury.ThebrownJournalofWorldAffairs,vol.17,no.2,
2011,p113–121.,http://www.jstor.org/stable/24590801.89 HumanitarianResponseReview.AnindependentreportcommissionedbytheUnitednationsEmergency
ReliefcoordinatorandUnder-Secretary-GeneralforHumanitarianAffairs,OfficeforthecoordinationofHumanitarianAffairs(OcHA).Adinolfietal,August2005.
90 basedonintervieweeresponsesandevaluationteamexpertopinion.91 linkingPreparednessResilienceandResponsefinalEvaluationReport,p10.92 ShiftingthePowerfinalEvaluationreport,p30.“ProgresswasachievedbySTPinwhatwaseffectivelya
projectimplementationtimeframeof2to2.5years.However,inordertofullyachievewhattheprojectsetouttodoatimeframeofdoublethelengthwouldhavebeenmorerealistic.”
93 TransformingSurgecapacityfinalEvaluationReport,p27.“Overalltheprojecttimeframewaslargelyconsideredtooshorttoachieveeverythingitplannedtoachieve.”
90 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ALL DEPP COUNTRIES
N=51
UK-BASED PROJECT
LEADERSHIP
N=17
IN-COUNTRY PROJECT
LEADERSHIP
N=28
PROGRAMME LEADERSHIP
N=6
Objectives are clear ✔ ✔ ✔ ✔
Objectives feasible within the timeframe ✖ ✖ ✖ ✖
Collaboration adequately included in project design
✖ ✖ ✖ ✖
Resources (human and financial) were adequate ✖/✔ ✔ ✖/✔ ✖
Table3.3:DesignvalidityoftheDEPP
✔Mostrespondentsagreewithstatement ✖Mostrespondentsdisagreewithstatement✖/✔Responsesweremixed
Whenmappinghowtheobjectivesandactivitiesofeachproject feed intoeachof thefiveresultsareas,as illustratedonpage16of theDEPPlearningReport201594, it isclear thatwhilesomeprojectsonlytargetedafewofthefiveresultsareas,asagroupofprojects,allfiveresultsareaswereaddressedineverycountryexceptJordanandMozambique.Thisis,inpart,duetotheadditionofthefourearlywarningprojectswhich,asdescribedearlier,weredevelopedthroughaseconddesignprocessthatwasinitiatedwhenstakeholdersidentifiedalargegapinprojectstargetingresultareatwo(improvepreparednesssystemsforearlyactionwithcommunitiesatriskofdisasters)afterthefirsttenprojectshadbeenapproved.Whilemostrespondentsconsideredtheobjectivestobeclear,themajorityfelttheywerenotlikelytobeachievedwithintheestablishedtimeframe.
Were resources (both human and financial) adequate?
DEPP stakeholder and respondent perspectives on this issuewere assessed across all tencountries,andfiveVfMprojectcasestudieswerealsoconducted.Overall,respondentsfeltthat resourcing for theDEPPat theportfolio levelwas sufficient (seeTable3.3).However,ingeneral,most interviewees felt thatprojectbudgetswere stretched, and resourceswereinsufficient to undertake activities to their full potential. There was a need for in-countryresourcesratherthanforUKstaff.non-projectattributablecosts(rangingfrom7%to12%ofthetotalprogrammebudget)wereconsideredinsufficientbyalmostallinterviewees.Therewerealsodiscussionsabout insufficientallocationof funds forcollaborativeactivities.MonitoringandEvaluationbudgetallocationsvaried,asdidthequalityof individualM&Eplans95.TherewasaneedtoincreaseM&Eresourcesforcertainprojectsandrecommendationsweremadeintheformativephaseevaluationreport.Inmanycases,respondentsreportedthatfinancialchallengeswerealsolinkedtoimplementationdelays.Asdiscussedinmoredepthlaterinthereport,theimplementationdelayswereprimarilyduetounderestimatingthetimerequiredforcollaborationandsettinguptheconsortiaandcontracts,aswellastheadditionaltimerequiredtocontextualiseprojectswithinthecountries.
94 DEPP2015learningReport.Accessedon13January2017athttps://startnetwork.org/resource/depp-2015-learning-report.
95 forexample,thefinancialEnablersfinalEvaluationreportsan:“overallunder-developmentofprojectMEl.nooverarchingM&Eframeworkwasdeveloped.Anearlylogframewasneverembraced”,andtheTalentDevelopmentfinalEvaluationreported(p77-81)thatitsM&Eframeworksreliedtoomuchonselfassessments,lackedqualitativeinformationandthattheproject:“lackedasinglecoordinatedM&Esystem.”
91 SUMMATIVEPHASEREPORT-RElEVAncEAnDVAlIDITyOfDESIGn
Do the projects align as a portfolio?
The individualprojectsweredesignedwithoutdirect communicationbetweenprojects andhencewerenotdesigned to complement eachother. for this reason, theDEPP’sdesign islimitedintermsofitsabilitytofunctionasaportfolio.Onestakeholderstated:
“WewereunderalotofpressurefromDfIDtobasicallyjigsawtogethertheprojectstoeachother…They[DEPPprojects]weren’tdesignedtositalongsideeachotherandcomplementeachother.So,thereisabitofadesignflawinthatsenseaswewereundera lotofpressure tomake theprojects interoperable,more interlinkedwhenactuallytheyweredesignedtobeindividualprojects.Theyjusthappenedtositinthesamepot.“(Programmeleadership)
Portfolio-levelsystemsshouldhavebeendevelopedearlyon,potentiallythroughaseparateinceptionphase,toensurethattheDEPPcouldfunctionwellasaprogramme.forexample,thelackofanoverarchingDEPPlogicalframeworkattheprogrammelevelwasasubstantialweakness.Hadsuchalogicalframeworkbeendevelopedearlyontogetherwithothersystemsand processes, it might have led tomore cohesion and alignment across the projects andperhapsgreatervalidityofDEPPasawhole.
Was collaboration at the programme level addressed during the design phase?
DEPP’s emphasis on collaboration emerged relatively late during the design process andexpectations and objectives related to collaboration, especially inter-project collaboration,wereunclearformanystakeholders.Therewasanassumptionthatcollaborationwouldoccurorganically,however,therewasanunderestimationofthegeneralknowledgeandskillsamongmembersonhowtocollaborateeffectively.further,collaborationwasinterpreteddifferentlybydifferentagencies,andtherewas insufficienttimeandresourcestoeffectivelyengage incollaborativeprocessesduringthedesignphase.InthecaseoftheprojectsdevelopedduringSTART network’s first design phase, some collaborations were mandated, and this led totensionsandlowerqualitycollaborationswheresubstantialtimewasspentintryingtoreconciledifferingopinions,underminingtheefficiencyofdelivery.
QUESTION 1.EHAVE PRIORITISED TARGET GROUPS (PEOPLE WITH DISABILITIES, OLDER PEOPLE) AND GENDER ASPECTS BEEN TAKEN INTO CONSIDERATION IN THE PROGRAMME DESIGN? All projects considered gender in their design, but the level of consideration varied. Some projects considered other prioritised groups such as the elderly and people with disabilities but this was not an explicit requirement for project selection. The DEPP Programme board criteria for project selection required inclusion of a genderstatement,butdidnotprovidefurtherguidelineswithrespecttogender inclusion.Assuch,all DEPP projects considered gender in their proposals and outlined their planned genderapproachintheconceptnotessubmittedtotheSTARTandcDAcnetworks.However,sincetherequirementwasonlytoincludeagenderstatement,thelevelofdetailprovidedandtheproposed level of inclusion varied substantially across projects96. Some project proposalsdescribed disaggregating data by gender, others proposed ensuring gender equality inparticipation,andothersdescribedincorporatingastrongfocusongenderwithinthecontentoftrainingmaterialsandcurricula.Othersweremorevagueandindicatedthatgenderdimensionswouldbeconsideredwithoutanyspecificsabouthow,orstatedthattheapproachwouldbefurtherdevelopedintheprojectinceptionphase,thoughthisdidnotoccurformostprojects.
96 forexample,theTransformingSurgecapacityfinalEvaluationReport,p38states:“Theprojectdesigndidnothaveastrongfocusonwomen’srights.”TheUrbanEarlyWarningEarlyActionfinalEvaluationReport,p16reports:“thoughprojectwasdesignedtoaddressthemostvul-nerable,itdidnotexplicitlyoutlinecross-cuttingissueslikegendermainstreaming,VAWGinitsdesign.”
92 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Prioritised targetgroupswerealsonotexplicitly includedasa requirement in thedesignoftheprojectsandwerenottargetedinmostprojects’activities.However,whilenotaprojectrequirement, prioritised groups, including the elderly and people with disabilities, wereconsideredandincludedinmanyprojectsacrossDEPP.forexample,theAgeandDisabilitycapacityProgramme’sinclusioneffortsaredescribedlaterinthisreporttobeleadingthefieldandextremelyinfluentialamongstotherDEPPprojects.
Design Considerations with Respect to Emergency Preparedness and Response
Responsetoemergencieswasnotadequatelyconsideredduringthedesignphase–theprogrammewasdesignedtoimproveemergencypreparednessbutmechanismstoenableorfacilitatecontribution(especiallyofl/nnGOs)tohumanitarianresponseeffortswerenotexplicitlyintegratedintotheprogramme.forexample,accesstoresources,especiallyresponsefundswerenotsystematicallybuiltintotheprogramme.ThismeantthatinmanycaseswhileDEPPbeneficiaryorganisationsfelttheyhadthecapacitytorespond,theywereunabletosecureadditionalresponsefunds.Thelackofconsiderationtothelinkbetweenpreparednessandresponsecouldhavepotentiallyledtoreducedeffectsoftheprogramme.Despitethis,DEPPcontributedtoatleast42humanitarianresponsesin11countries(Seechapter4andAnnex10forfurtherdetailsonDEPPcontributiontohumanitarianresponseefforts).
93 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
figure2:DEPPProjectDescriptions
RELEVANCE AND EFFECTIVENESS OF INTERVENTIONS
cHAPTER4
94 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Evaluation Question #2: In what ways have DEPP capacity building programmes strenghtened preparedness and response capacity amongst participants?Sub-questions:a) Whatdeliverymechanismsareworkingeffectivelyandwhy?b) To what extent is DEPP contributing to greater preparedness and response among local
organisationsandcommunities?a. HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,howhas
itchanged?Ifnot,whynot?b. Has DEPP led to improved knowledge and understanding of best practices relating to
disasterandemergencypreparednessandresponse?Ifyes,inwhatways?Data sources: • Minimum set IDIs with programme and project stakeholders (Qualitative)• KAP and organisational surveys (Quantitative)• Organisational checklist(Quantitative)• Community and household survey(Quantitative)• Intensive set IDIs(Qualitative)• Document review (Qualitative)Key Findings:• Implementation delays were universalandduetothreemainfactors:
o limitedcontextualisationandinvolvementofin-countryteamsindesignphaseo Underestimationoftimerequiredforcollaborationo Administrativeandcontractualbottlenecks
• Delays led to increased costs and budget reallocations, and reduced period for activities and potential impact
• Overall,33,388 individuals were exposed to DEPP capacity building exceeding the ultimateprogrammetargetoftraining4,200individualsby795%.However,thegendertargetwasmissedby5%(45%oftraineesarewomen).
• Both DEPP and non-DEPP participants from comparison organisations were exposed to more didactic individual-level capacity building activitiesrelatedtopreparednessandresponsethantootheractiveskillandcompetenciesbuildingexercisessuchdrillsandsimulationsacrossallfourintensivesetcountries.ExposuretootherDEPPactivitieswashigherinDEPPthannon-DEPPbeneficiaries
• DEPP has contributed to the four output areas (corresponding to the causal chain); it hascontributedtoincreasedcapacitybuildingefforts,collaboration,earlywarningsystemdevelopmentandlearninginbeneficiaryorganisations,but success has been variable
• Themosteffectiveindividual-levelcapacitybuildingapproachreportedbybeneficiariesinvolvedin-persontrainingcombinedwithastrategytoreinforceknowledge(simulations,mentoring,coaching)
• Flexible funding approaches,thoughinitiallyconsideredhighrisk,have high potential for effectiveness• Outcome-level changes were variable.Overall,therehavebeenno significant improvements in
knowledge oncorehumanitariancompetenciesofhumanitarianstaff,onself-reportedknowledgeondisasterpreparedness,responsetodisastersandemergencesorageordisability-relatedissuesindisasters
• Therewasnochange inperceived levelof individualpreparedness,but thereweresignificantimprovements inpreparednessofDEPPorganisationsacrossall intensivesetcountries,and inparticularforl/nnGOs.AccesstoearlywarningsysteminformationwassignificantlyhigherinDEPPcommunitiesthaninnon-DEPPcommunities,theperceivedlevelofcommunitypreparednesssignificantlydecreasedinbothDEPPandnon-DEPPcommunities.
• Qualitative data suggests improvements in preparedness thatmay be difficult to detectwithquantitativedata
• DEPP contributed to at least 42 humanitarian responses in 11 countries;16oftheseresponsesoccurred in the Philippines (and 14 of these 16 responseswere driven by one project – thefinancialEnablersproject)
• QualitativedataindicatecontributionofDEPP beneficiaries towards improved responseinthePhilippines,KenyaandEthiopia,butmanyotherDEPPbeneficiarieswere unable to respond due to lack of response funds
• Individualprojectevaluationssuggestsomepositiveoutcomes,butthishasnottranslatedtoglobalprogrammeeffectslikelyduetoshortprogrammeperiodandlackofprojectalignment
95 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
QUESTION 2.A: WHAT DELIVERY MECHANISMS ARE WORKING EFFECTIVELY ANDWHY? 1. PROJECT AND PROGRAMME IMPLEMENTATION
A. What was Implemented?
A diverse range of activities of varying quality targeting all five results areas was implemented.
Toexplore interventiondelivery,keyactivitiesthatwereconductedbytheDEPPprojectshavebeengroupedtogetherbyeachofthefiveDEPPresultsareas(Table4.1).
RESULT AREA EXAMPLES OF KEY ACTIVITIES IMPLEMENTED
1 – Improve knowledge and understanding of individuals by sharing
best practice of humani-tarian preparedness and
response
• Targetedcapacitystrengtheningplans• Trainingschemesforalllevelsofhumanitarianaidworkersthrough
jobplacementsandextendedcourses• Sharingofgoodpracticesandlearningonsurgemechanismsacross
humanitarianactorsandDEPPprojects• Trainingsandworkshopstoincreasemulti-sectorknowledgeand
barrierstoageanddisabilityinclusioninpreparingandrespondingtodisasters
• betweenandwithinsectorconsultationstoimproveknowledgeandbestpractices
• Trainingstoimprovethecapacityofhealthcareworkerstodetectandrespondtoemergencies
• communicationandtrainingstargetedatnon-specialistandnon-hu-manitarianstoinformabroaderunderstandingofhumanitariansys-tems
2 – Improve prepared-ness systems for early
action with communities at risk for disasters
• Toolkitdevelopedwithguidelinesforsettingupanurbanearlywarn-ingsysteminothersettings
• Participatoryapproachesarebeingusedtodevelopnewprepared-nessandresponsesystemsattheorganisationallevelutilisingfeed-backfromusers
• Improvedearlywarningandpreparednesseffortstargetedathealthfacilitiestoensureadequateresponseduringanoutbreak
• Pursuitofcommunity-ledandownedpreparednessandresponseactivitiesthroughidentificationofcommunity-specificrisksandvul-nerabilities
• Useofbottom-upapproachtoensuresuccessfuladoptionofearlywarningsystems
96 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
3 – Increased number of coalitions, partnerships
and networks which, working together, are
able to address human-itarian needs in a wide range of emergency
situations.
• Initiationofdisasterresponseandmanagementtrainingsatthegov-ernmentlevelincollaborationwithorganisations
• Sharingofrostersandevaluationofsatisfactionforsurgepracticesatnational,regionalandinternationallevels
• EncouragementofthematicprogrammingoutsideofDEPPthroughnetworksformedatconferencesandothernon-DEPPevents
• cross-projectandcross-agencycollaborationtoimproveanden-hanceprojectprototypes,goalsandobjectivesformaximumsuccess
• Transferofprojectownershiptothelocallevelacrossdifferentsec-torsusingaflexiblefundingmechanism
• creationoflocalconsortiatoimproveownership,sustainability,andcapacityatthenationallevelthroughlocallydistributedgrants
4 – Improved institu-tional arrangements and policy environments so
that national systems for humanitarian response are better supported
and more suitable
• Supportinglocalandunderrepresentedactorstobebetterrepresent-edandheardintheirrelevantplatformsandnetworks
• Organisationalcreationandadaptationoftoolsandpoliciesforageanddisabilityinclusion
• Utilisationofaparticipatoryapproachtotransferownershipofhealthcentrepreparednessandresponseplanstogovernment
• Directadvocacyatthelocalandnationallevelforadoptionofanurbanearlywarningmechanismatthegovernmentlevel
5 – Strengthened ev-idence base for what
works to help build hu-manitarian capacity at
scale
• Assessmentofcapacityofindividualsparticipatinginhumanitarianresponserostersandtrainingsatbaselineandendline
• DEPP/non-DEPPworkshops,conferencesandlearningevents• Evidencesharinganddissemination• Individualassessmentsandtrainingscateredtowardgapsincapacity• Utilisationofcasestudiesandcommunitymember’sfeedbackand
perspectivesforinformingprogrammaticpreparednessandresponserecommendations
• creationofaDEPPlearningplatform(https://disasterpreparedness.ngo/)
Table4.1:KeyDEPPactivitiesimplementedbyDEPPresultarea
basedonthedocumentreview,itisevidentthatarangeofdiverseactivitiestargetingeachofthefiveresultsareaswasimplementedacrosstheprogramme.Examplesoftheseactivitiesincludesin-persontrainings,developmentoftrainingpackagesandguidelines,developmentofearlywarningsystems, collaborationactivities suchas learningeventsandconferences,developmentof casestudiesandevidencesharinganddissemination.feweractivitieswereimplementedaroundresultfive.Thequalityoftheseactivitieswasvariable–somewerenotablyhighinquality(forexampletheyutilisedquantitativemethodswitha largesample size),but inothercasesquality is lowerand/ormaybedifficultto independentlyverifydueto inadequatemonitoringandevaluationordocumentation97.
Ofthe336respondentsacrosstencountriesthatweresurveyedaspartofminimumsetorintensive set evaluation activities, 70% reported that their organisations had implementedcapacitybuildingactivities,37.8%reported thatearlywarning systemshadbeendevelopedand32.2% reported implementationofdrills and simulations (Seefigure4.1below).fewerrespondents reporteddrillsandsimulations inEthiopia, andparticipants in resultsvalidationworkshops attributed this to the disaster and emergency profile in Ethiopiawhere droughtis one of themost common emergencies. close to 68% reported training of national staffand44.4%reportedcommunitypreparednessactivities.OnlyaboutaquarterofrespondentsreportedthatevaluationorresearchactivitieswereundertakenaspartoftheirDEPPproject.
97 forexample,thefinancialEnablersfinalEvaluationReport,p19states:“Verylittleisknownaboutthequalityortheusefulnessofthetrainingsduetoalackofmeasurementbyconsortiaortheproject.”
97 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
figure4.1:KeyDEPPactivitiesimplementedbetween2015and2018byorganisationssurveyedduringintensiveandminimumset(all10DEPPcountries)evaluationactivities
B. Whose Capacity was Built? (Who Benefitted from DEPP Activities?)
Both DEPP and non-DEPP comparison participants in the four intensive countries were ex-posed to individual-level capacity building related to emergency preparedness and response, but both groups were less exposed to drills and simulations. Exposure to other types of DEPP activities was higher in DEPP participants.
Intermsofexposuretoindividual-levelcapacitybuildingactivities,89%ofhumanitarianstaffatDEPPorganisationssurveyedduringtheKAPsurveyhadparticipatedinacapacitybuildingactivityinthelast12monthscomparedto76%forhumanitarianstaffwithinthecomparisongrouporganisations.further,46%ofDEPPrespondentshadparticipatedinadisasterdrillorsimulationwithin the sametimeperiod compared to approximately30% in the comparisongroup(figures4.2and4.3).ThesefiguresdemonstratethatcomparisongroupparticipantsintheKAPsurveywereexposedtoafairlyhighlevelofcapacitybuildingonemergencyprepared-nessoverthesametimeframe,thoughexposurewasstillhigherby13percentagepointsintheDEPPgroup.Inaddition,exposuretosimulationswashigherby16percentagepointsintheDEPPgroup.Thissuggeststhatiftheindividual-levelcapacitybuildingeffortsinbothDEPPandcomparisongroupsweresimilarlyeffective,thentheremaybeimprovementsinsomeoutcomessuchasknowledgechangeinbothgroups.However,thiswouldnotapplytootheroutcomesre-latedtoactivitiessuchasearlywarningsystemdevelopment,ororganisationalcapacitychangewhereexposurewassignificantlyhigherinDEPPgroups98.Inthosecases,significantoutcomechangesinthecomparisongrouparenotexpected.Overall,theexposuredatadodemonstratehighpenetrationoftheDEPPwithinsampledorganisations,whichisapositivefinding.
98 forexample,figure4.9showsthatDEPPcommunitiesinMyanmarweresignificantlymoreexposedtoearlywarningsystemsthannon-DEPPcomparisoncommunities.
0
20
40
60
80
100
RESEARCH
EVALUATiONS
DRiLLS/SiMULATiONS
PREPAREDNESS ACTiViTiES
PREPAREDNESS TRAiNiNG
CAPACiTY DEVELOPMENT PROGRAMS DEVELOPED AND iMPLEMENTEDEARLY WARNiNG SYSTEMS DEVELOPED
ALL DEPP
KENYA
THE PHiLiPPiNES
ETHiOPiA
MYANMAR
DEPP ACTIVITIES IMPLEMENTEDRE
SPO
ND
ENTS
(%)
70.0
71.4
70.0
88.9
60.0
37.8
57.1
40.0
22.2
50.0
67.8 71
.4 75.0
72.2
70.0
44.4
42.9 45
.0 50.0
45.0
32.2
57.1
10.0
44.4
30.0
24.4
14.3
25.0
33.3
30.0
24.4
13.3
30.0 33
.310
.0
98 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
figure4.2:DEPPandnon-DEPPcomparisonparticipationinindividual-levelcapacitybuildingactivitiesoverpast12months
figure4.3:DEPPandnon-DEPPcomparisonparticipation in individual-level drills and/or simulationsoverpast12months
C. Timeliness
Project-level implementation was variable in terms of timeliness and delays led to increased costs, and reduced potential impact.
Universally, there were significant delays in implementation due primarily to three factors.Deficienciesinthedesignprocess, includinglimitedcontextualisationofprojectsandlowin-volvement of in-country teams, required significant time to address during project start-upand contributed to implementationdelays. for example, the lack of in-country involvementduringthedesignprocessmeantthatonceprojectswereapprovedandcontracted,timehadtobetakenfor in-countryteamstounderstandtheprojectactivities,contextualisethemforthespecificsettingandtargetaudienceandensurebuy-inofkeylocalstakeholders99,100,101 .The
99 ShiftingThePowerfinalEvaluationReport,p30.100 DEPPlearningReport,2016:https://disasterpreparedness.ngo/wp-content/uploads/2017/05 DEPP-learning-Report-fInAl-1.pdf.101 TransformingSurgecapacityfinalEvaluationReport,p31.
70%
30%54%
46%
HAVE NOT PARTiCiPATED
HAVE PARTiCiPATED
N = 329
DEPP PARTICIPATION IN DRILLS/SIMULATIONS
N = 119
NON-DEPP PARTICIPATION IN DRILLS/SIMULATIONS
11%
89%
24%
76%
HAVE NOT PARTiCiPATED
HAVE PARTiCiPATED
N = 329
DEPP PARTICIPATION IN CAPACITY BUILDING ACTIVITIES
N = 119
NON-DEPP PARTICIPATION IN CAPACITY BUILDING ACTIVITIES
99 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
timerequiredfortheseprocesseswasalsounderestimatedandfurthercontributedtodelays
inmanyprojects.Thiswascompoundedbythefactthatsomeprojectshadnot includedaninceptionphase102,whichwouldhavereservedtimespecificallyfortheseprocesses.Thosethathadincludedaninceptionphasestatedthattheamountoftimewasinsufficient.
Thesecondfactorcontributingtodelayswasunderestimationofthetimeandresourcesre-quiredforcollaborationincludingsettingupconsortia103,104,105,106. finally,lengthyadministrativeandcontractualprocessescontributedsignificantlyto implementationdelays107,108,109.Duetothese delays, the 14 projects across theDEPP portfolio had shorter project timelines thaninitiallyplannedreducing thepotential formeasurable impactwithin theprogrammeperiod.The delays also contributed to increased costs and reduced efficiencies.As a result of thedelays,projectsalsoconfrontedchallengesincompletingactivitiesontime.Sixof14projectshaverequestedandobtainedno-costextensionstolengthenprojectperiods(Seefigure4.4).Accordingto intervieweesanddocumentreview, implementationdidbecomemoreefficientovertimeasprojectsintensifiedefforts.
YEAR 2014 2015 2017 2018
MONTH
January
february
March
April
May
JuneJuly
August
September
October
november
December
January
february
March
April
May
JuneJuly
August
September
October
november
December
January
february
March
April
May
JuneJuly
August
September
October
november
December
January
february
March
April
PROJECT TIMELINESADcAPAlertcDAc-nFEEWEAlearningProjectLPRRPiPPHEPGambellaSTPSEPSTDTScUEWEA
Sub-agreementsigned/ProjectstartdateProjectimplementationstartdate(iflaterthanprojectstartdate)Projectenddateno-costextension
figure4.4:DEPPprojecttimelines
D. Perspectives on Implementation
Despite significant widespread project delays, stakeholders consistently reported effective implementation of the DEPP (particularly at the project level). In fact, 82.8% of stakeholders interviewed (N=232) across all 10 DEPP countries reported that delivery of the DEPP has either been effective or very effective. However, despite these positive perspectives,
102 forexample,figure4.9showsthatDEPPcommunitiesinMyanmarweresignificantlymoreexposedtoearlywarningsystemsthannon-DEPPcomparisoncommunities.
103 financialEnablersfinalEvaluationReport,p18.104 StrengtheningEmergencyPreparednessSystemsinMyanmar,finalEvaluationReport,p19.105 TransformingSurgecapacityfinalEvaluationreport,p31.106 UrbanEarlyWarningEarlyActionfinalEvaluationreport,p14.107 financialEnablersfinalEvaluationReport,p2:“fEPwasthelastoftheDEPPprojectstobesignedoffby
DfID,roughlyayearafterdiscussionsbeganaboutit,eatingintothetimeframeforimplementation.”108 UrbanEarlyWarningEarlyActionfinalEvaluationReport,p6:“Ittookalmost6monthstoconcludethe
project.implementationplanandcloseto9monthstofinalisesub-grantagreementsandoverarchingconsortiumagreement.This‘ate’intothetimethatwastobeusedtoimplementtheprojectactivities.”
109 cDAc-nfinalEvaluationreport,p16.
100 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
evidence demonstrates that programme implementation was fraught with significant challenges.
Manyofthechallengesrelatedtoprogrammeimplementationwerepreviouslyoutlinedintheformativephase report.110 These includea lackof systemsandprocessesat theprogrammelevel,theabsenceofafinalisedprogramme-levellogicalframework(untilAugust2017)againstwhichprogresscouldbetrackedaswellasinadequatehumanresourcesattheprogrammeleveltoeffectivelymanage theprogramme.Thenumberandtypeofprojectsbeing implementedwithin each country also varied. In countries with fewer projects implemented, such as inMozambique,Jordan,andMyanmar,therewerefeweropportunitiesforcollectiveaction,loweroverall investmentand therefore, a lower likelihoodofembedding long-termchange.Again,theseproblemscanbe tracedback to theweaknesses in thedesign including the fact thatintervention intensity at the country levelwas not adequately and strategically considered.Impact could have beenmaximised by ensuring a suite of complementary projects in eachcountry that fully addressedall levelsof theprogramme-level theoryof change.Asnonewprojectswere funded after the evaluation formative phase, andwith the exception of thecDAc-n project which shifted country locations from South Sudan to the Philippines, nochangesininterventionintensityattheprogrammeleveloccurred.Unfortunately,thedesignflawscouldnotbeundoneandcontinuedtohaverepercussionsthroughoutthecourseoftheprogramme’sthree-yearduration.
2. ACHIEVEMENT OF THE FOUR OUTPUT AREAS
DEPP contributed to increased capacity building efforts, collaboration, early warning system development and learning in beneficiary organisations, but achievements were variable.
Toassessachievementsof theDEPP,asystematicassessmentofdirectprogrammeoutputsdrawingonprimarydatacollectedviathein-depthinterviews,quantitativedataandsecondarydatafromthedocumentreview,andreferringbacktothesimplifiedcausalchaindevelopedbytheevaluationteam(seeboxbelow)wasconducted.Thefourkeyoutputareasassessedinclude:1)collaboration,2)capacitybuilding,3)generatingandsharinglearningand4)earlywarningsystemdevelopment.notethatthesecorrespondwiththeevaluationsteam’ssimplifiedcausalchain(andthustheprogramme-leveltheoryofchain)andaredistinctfromthefiveDEPPresultsareas.Alatersectionwillpresentoutcome-levelfindings(improvedemergencypreparednessand improvedresponse)oftheDEPP.notethatDEPPimpacts (reducedmortality,morbidityand economic impacts)were considered too far downstream from theDEPP activities andassessmentoftheseindictorswasnotwithinthescopeoftheevaluation.
110 forexampleseep67-68oftheformativePhaseEvaluationReport.
101 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
DEPP THEORY OF CHANGEDEPP Impact (Not being assessed by evaluation): • Reduced mortality, morbidity and economic impacts from disasters and emergencies
DEPP Outcomes:• Improved preparedness of individuals, organisations, communities• Improved response by individuals, organisations, communities
DEPP Outputs:• Improved collaboration and strengthened networks related to preparedness and response• Improved capacity to respond to disasters and emergencies (individual, organisation, com-
munity)• Improved early warning systems • Improved learning and strengthened evidence base on preparedness and response
Source: DEPP programme causal chain (see Figure 1.1)
1. Evaluation Output Area 1: Collaboration
The DEPP has contributed to increased collaborations and strengthened networks in several settings, but numerous issues impeded potential effectiveness.
TheactivitiesconductedbyDEPPprojectswereexpectedto, if thetheoryofchangeholds,leadtoenhancedcollaboration,andultimatelystrengthenednetworks,coalitionsandpartner-ships.ThecollaborativeapproachwasessentiallythecornerstoneoftheDEPP;eachofthe14projectswasdeliveredviaconsortiaandintotal45primaryorganisationsmadeuptheDEPP.collaboration within project consortia, but also between DEPP projects and with externalstakeholderswasanexpectedoutputof theDEPP, thoughasdescribedpreviouslythetimerequiredforthislevelofextensivecollaborationwasnotwellbuiltintothedesign,workplansorbudgets.Interviewdatarevealthatthecollaborativeapproach,includingcollaborationwithinprojectconsortiaandamongorganisations,wasideologicallyembracedbyallprojectsandwasperceivedtobeaneffectiveandappropriateapproachtocapacitybuildingaswellasresponse.However,asdescribedbelowtherewerevaryingdegreesofimplementationsuccess.
While the emphasiswas on localisation, and ensuring amore equal balance and role of l/nnGOpartners,thisobjectivewasnotfullymet.forexample,inadditiontothe45primaryDEPPorganisations,therewere222uniqueorganisationalpartners,butmostofthesewereInGOpartners.InternationalnGOsrepresentthelargestsharewithintheDEPPandthatwhileeffortstobemoreinclusivetowardsl/nnGOshavecertainlyincreasedinvolvement,theyhavenottranslatedtoequalopportunitiestocollaborate,engageandparticipateindecision-making.Eveninthoseprojectswithhighersharesofl/nnGOpartnerships,primarilyInGOsmakeuptheprojectconsortiaandstillplaythe“gatekeeper”rolesuggestingthatthepowerdynamicsremaininfavourofInGOs.ThereareafewnotableexceptionstothissuchasthefinancialEnablersandtheShiftingthePowerprojectswherel/nnGOswereabletotakeamoreequalrole.bothoftheseprojectsaimedtodirectlyalterpowerimbalances–financialEnablersthroughahands-offapproachwhichpermittedlocalorganisationstosteertheircapacitybuildingactivities,andShiftingthePowerbystrengtheningthecapacityoflocalorganisationsusingaprocessofself-assessments.
Collaboration within DEPP Project Consortia
Evaluationdatademonstratethestrongperceivedvalueofcollaborationandworkingthroughconsortia,butalsohighlightthechallengesofthisapproach.Theseincludethesubstantialtime,effortandresourcestosetupconsortiaandlackofclearexpectations,objectivesandprotocolsrelatedtocollaboration.forexample,theAgeandDisabilitycapacityProgrammestakeholders
102 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
expressedfrustrationaboutthelackofconsortiadecision-makingprotocolsandlackofclarityonwhohadfinalauthority111.Thesechallengeswereamplifiedincaseswherecollaborationsweremandatedorforcedratherthanorganic,andgloballyaroseduetolackofadequateplanningduringthedesignphase.Thesechallenges,inturn,areoneofthethreemainfactorsthathavecontributedtoimplementationdelays112.
Additional challenges of working in consortia included varying governance structures,administrativeprocessesandindividualandorganisationalaims113.Staffturnover,andthelackof incentivestoparticipate incollaborationsonavolunteerbasisplacedadditionalstressonprojectimplementation.Inaddition,thelargenumberofformalorganisationalpartnersthatmakeupeachconsortiabecameahindranceintermsofspeedofdecision-makingandeffectivenessofprojectimplementation114.Projectdeliverywouldhavebeenmorestreamlinedwithfewer,morestrategicconsortiamembersandalsopotentiallydifferentgovernancearrangements.Inaddition,insomeprojects,theconsortiapartnersattheUKlevelweredifferentorfunctionedindependentlyfromin-countryconsortiapartners.ThissetupwasreportedtobeproblematicespeciallywhentherewaslimitedcommunicationbetweenUKandin-countryconsortiapartnersresultinginconfusion,lackofengagementandimplementationdelays.IntheAlERTproject,theconsortiamemberswereneverformalisedwhichledtoalossofefficiencyandlimitedsupportfromotherpartnersduringtheearlyphasesoftheproject115.
collaborationsthatbuiltonestablishedpartnerships,andaroundcommongoalsandwaysofworkingwerethemostsuccessful.newercollaborationsorconsortia,includingthosethatfeltpressuredtoworktogether(suchasthosewhoweremandatedtoworktogetherduringthefirstSTARTdesignphase)tendedtostrugglethemostandwereleastefficient.
Collaboration across DEPP Projects
cross-projectcollaborationwasnotwellarticulatedasaprogrammeobjectiveandactivityduringthedesignphase116,leadingtodifferentinterpretationsofcollaboration,andunderestimationof the time and resources needed. cross-project collaboration was expected to occurorganicallybutsinceprojectsweredesignedasstandaloneprojectsratherthananintegratedportfoliothistypeofcollaborationrequiredconcertedeffort.GeographicimbalancebetweentheUK and target countries in termsof collaboration intensity across projectswas evidentsincethebeginning,thougheffortstostimulatein-countrycollaborationsthroughin-countryand international learning events, international conferences (Kenya-December 2016117, thePhilippines-July2017118)andbyembeddingcountry-levellearningadvisors(clAs119)intensifiedduringthesecondhalfofDEPPandinsomecasesweresuccessful.
cross-project and cross-country exchangevisits to learndirectly fromotherprojects and indifferentcontextswereoneoutcomeofthelearningconferencesthatwerereportedtobeparticularlyeffective.Othereffortstoengagelocalstakeholderstoparticipateinlocallearningeventsandothertypesofcross-countrycollaborationhavehadvaryinglevelsofsuccess.forexample,intervieweesstatedthatinPakistan,despitesubstantialeffortstoengagestakeholders,participationat learningeventsremained low, leadingtothedecisiontoreallocatefundsforfourremaininglearningeventstoanotheractivity.
Inordertofurtherfacilitatesharingoflearningandevidence,theDEPP’slearningProjectalso
111 ADcAPfinalEvaluationReport,p29.112 Theothertwofactorscontributingtodelaysinimplementationwerelengthyadministrativeandcontrac-
tualprocesses,aswellasretrofittingofpre-designedprojectstothebusinesscasewhichthenrequiredsubstantialtimeattheprojectstart-uptocontextualiseandsecurebuy-infromlocalstakeholders.
113 linkingPreparednessResilienceandResponsefinalEvaluationReport:“differentorganisationalcultures,timecommitment,andtheshortdurationofactual‘implementation’heldthecollaborationbackfromflourishing;assuch,theprojectmissedopportunitiesintermsofseizingnewopportunities,particularlyrelatedtojointfundingordevelopingnewprojectinitiatives.”
114 Thisfindingisalsosupportedbydocumentreview.forexampleseeADcAPfinalEvaluationReport,p5.115 AlERTfinalEvaluationReport,p5.116 Thisfindingisalsosupportedbydocumentreview.forexample,seeShiftingthePowerfinalEvaluation
Report,p24.117 102DEPPandnon-DEPPstakeholdersfrom13countriesattendedtheKenyalearningconference.118 127stakeholdersfrom9countriesattendedthePhilippineslearningconference.119 countrylearningAdvisorswereassignedtothePhilippines,Kenya,DRcandbangladesh.
103 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
setupanonlinecollaborationplatform120inMay2017.Theplatformcontainsacompilationofblogposts,reports,videosandothermaterialsgeneratedbytheprojects,aswellasaforumfordiscussions.Theplatformanditsusewillbeassessedinmoredetailunderevaluationoutputarea3:generatingandsharinglearning.
Inmanycasescross-projectcollaborationsdidnotgobeyondsharingofinformationandlearning.ThereislimitedevidenceofprojectsworkingjointlywithotherprojectswithinacountrytowardsthecommongoalsandobjectivesoftheDEPP,andtherewasminimaltonojointadvocacyorcommunicationefforts.Thisrepresentsa lostopportunityandlikelydiminishedthepotentialimpactoftheDEPPatthecountrylevel.
Collaboration with External Partners
The external evaluation assessed linkages of theDEPP to other key external partners.TheDEPPbusinesscaseoutlinesexpectationsfortheDEPPtoworkcloselywiththeHumanitarianleadershipAcademy (HlA), a global learning initiative set up to facilitate partnerships andcollaborativeopportunitiesandtoenableindividualstobetterrespondtocrisesintheirowncountries.currentdatademonstratethattheconnectionbetweenDEPPandHlAwaslimited,resultinginalostopportunity.OnlythreeintervieweesdiscussedHlA,andthesewereprimarilyindividualsattheprogrammelevel.OnerespondentdescribedajointDEPPprojectandHlAoutput,andanotherdescribedtheinvolvementofHlAinthePhilippineslearningconferenceasapartnerwhoprovidedlivestreamingoftheeventsessionsthroughtheironlineplatform.ThisconferencevideoisstillavailableontheHlAfacebookpagewhereithasreceived334views.Intotal,thelearningProjectestimatesthattherewereover10,000viewersonlineduringtheevent.Asidefromthesetwoexamples,theinterconnectednessbetweentheDEPPandHlAhasbeenminimal.SeveralotherintervieweesdescribedanintentiontodeveloplinkageswithHlAbutthatthesewerenotpursuedduetolackoftimeandresources.furthereffortstobuildaconnectionbetweenHlAandDEPPprojectswouldhavereinforcedDEPP’sactivitiesandalsoallowedforcoordinationofcapacitybuildingefforts.
TheSTARTfund,whichprovidesrapidfinancingtounderfundedsmalltomediumscalecrises,alsohashadminimallinkagestotheDEPP,despitebeinganinitiativewithintheSTARTnetwork.Atthetimeofthisreport,therewasatleastoneexamplewhereDEPPprojectpartnersinthePhilippineswereabletoaccessSTARTfundresourcestorespondtoacrisis.AccessofSTARTfundresourceswillbediscussedfurtherintheresponsesectionbelow.
A number of projects initiated collaborations with partners external to DEPP includinggovernments, ministries, and the Un. These types of collaborations and partnershipswereviewedasessentialforprojectsuccessandimportantforsustainability,butwerechallengingtodevelop,formaliseandmaintain.notabledifficultiesincludedlengthybureaucraticprocesseswithingovernments,staffturnoverwithintheUnandchangingprioritiesordirectionswithinthegovernment thatweredifficult tonavigate toensure continuityover a longerperiodoftime.Inaddition,thechallengesofworkingwithgovernmentswhentheyarepoorlyfunctioningorrestrictivewerealsodescribed.Thefewsolutionsproposedtoovercomethesechallengesincludedensuringadequatetimeandresourcestoengagewithgovernments,andbuildingintimewithinprojecttimelinestodeveloptheserelationshipsasearlyaspossible.
Strengthened Networks
Analysisoforganisationalnetworkswithinfourcountries(Myanmar,thePhilippines,KenyaandEthiopia)providessomeevidencethattheDEPPhascontributedtoincreasedcollaborationsandstrongernetworks.Pleaseseechapter5forfurtherdetailsonthefindingsfromtheanalysisofhumanitarianandresponsenetworks.
2. Evaluation Output Area 2: Capacity Building
Overall, DEPP exceeded its numeric targets for capacity building, though gender targets were not achieved. The most effective individual-level capacity building approach involved in-person training combined with a strategy to reinforce knowledge (i.e., simulations, mentoring, or coaching). At the organisational level,
120 https://disasterpreparedness.ngo/.
104 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
approaches that permit organisational self-assessments, and capacity building approaches tailored to organisations’ needs were most effective. Supporting the development of administrative policies, procedures and systems especially among L/NNGOs was also reported to lead to improvements. Eleven of the 13121 projects (not including the learning Project) implemented a total of 13differentcapacitybuildinginterventions.Theseinterventionswereimplementedin9oftheDEPPfocuscountries(Pakistan,bangladesh,Kenya,Myanmar,Ethiopia,Jordan,DRc,thePhilippines,SouthSudan),butalsoin5non-DEPPcountriesincludingIndonesia,lebanon,Turkey,Thailand,UK,andcolombia.nocapacitybuildingactivitieswereimplementedinMozambique,aDEPPfocal country. Projects conducted capacity building interventions that targeted individual,organisational,communityandsystemslevels(Seefigure4.5).122
INDIVIDUAL ORGANISATION COMMUNITY SYSTEM
ShortcourseslongcoursesTrainingoftrainersMentoringWebinarscoachingE-learningKnowledgeExchangeSimulationsOrganisationalActionPlansResourceStrengtheningTrainingandTrainingPackagesPreparednessPlansGuidelines,ToolsandResourcescommunitycommitteeesandforumscommunityAwarenessRaisingWorkshopsforumscoordinationadnPreparednessMechanismSoftware
figure4.5:DEPPcapacitybuildingstrategiesatindividual,organisational,communityandsystemlevels
Theevaluationteamcollectedindependentdatafromeachprojectregardingcapacitybuildingindicators and triangulated these against project quarterly reporting and programme logicalframeworkreporting(seeTable4.2below).Ascanbeseeninthetable,allprojectsdescribed
121 WhiletheDEPPincludedintotal14projects,thelearningProjectwasdesignedtocapturelearningonly,andthusisnotincludedinthistotal.
122 DevelopedusinglearningProject’sDEPPcapacitybuildingMap(https://kumu.io/Eamsmith/depp-capac-ity-building-activities-map#depp-capacity-building-activities-map/financial-enablers)andprojectreportsanddata.
105 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
implementing short courses (represented by green shaded cells in the first row). The nextmost common strategy was training of trainers with 10 projects employing this approach.cumulatively,acrosstheprogramme,thetriangulateddatashowthat33,388individuals(55%male,45%female)wereexposed toDEPPhumanitariancapacitybuildingactivitiesacrossanumberofkeysubjectareas.notethisisahigherfigurethanreportedbytheprogrammelogicalframeworkdata(n=24,304;seeTable4.2).Thisislikelyduetoinaccuraciesinlogicalframeworkreportingthatweredescribedintheinterimphaseevaluationreport.123ThekeysubjectareasforDEPPcapacitybuildingeffortsinclude:
1. Ageanddisabilityinclusion2. communicationwithcommunities3. Disasterandconflictresilienceandpreparedness4. Earlywarning,earlyaction/response5. Protection6. Organisationalcapacitybuilding7. Surge8. Healthsurveillanceinemergencies9. corehumanitariancompetencies10. Managementandleadership11. Research,datacollectionskills12. Wellbeinginresponse13. logistics,finance,administration
ThenumberofindividualsexposedtoDEPPcapacitybuildingactivitiesexceedstheultimatetargetoftheprogrammeoftraining4,200individualsby795%.However,thegoalthathalfofthetraineesshouldbewomenhasnotbeenmet.Only45%oftraineeswerewomen,whichrepresentsanimprovementfromtheinterimreportwhereonly41%oftraineeswerewomen.Thislikelyrepresentssomeofthecorrectiveactionstakenbyprojectstoincreaseaccessibilityforwomen,butunfortunatelybecauseofthelateimplementationofprogrammelogicalframeworkreporting,correctiveactiononlybeganinthelastquarterof2017,towardstheendofprojecttimelines.Atthatpoint,itwasdifficulttocatchupwithintheremainingtime.Itisalsoimportanttonote thatcollectionofgenderdisaggregateddatawasweakandsomeprojectswerenotabletoprovidefullygenderdisaggregateddataforall trainings. Infutureprogrammeand/orprojectiterationsinclusionofwomenshouldbestrengthened.Itisalsoimportanttonotethatthesecapacitybuildingdatawereself-reported,andmostprojectswerealsounabletoprovidedatadisaggregatedbytrainingapproach.Thetotal(n=33,388),therefore,includesalltypesofcapacitybuildingactivities,someofwhichinvolveminimalexposure(webinars,forums,shorttrainings).ItisthusasuboptimalindicatorasitmaynotaccuratelyreflectthetruenumberofindividualswhosecapacityhasbeenbuiltbytheDEPP.Thismayalsoexplainthelargeovershootofthetrainingtargetnumber.Infact,datapresentedlaterinthischapteronlackofsignificantchangeinknowledgeattheindividuallevelalsosupportsthis.
123 Sharma,V,HémonoR,Jean-francoisJ,ScottJ,Gibbonsn,PhamP.(2017)TheDisastersandEmergen-ciesPreparednessProgrammeEvaluation:InterimPhaseReport,October312017.HarvardHumanitarianInitiative,p9.
106 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
AD
CAP
ALE
RT
CDAC
-N
EWEA
FE LPRR
PHEP
G
ambe
lla
PIP
SEPS
STP
TD TSC
UEW
EA
TOTA
L
1.Shortcourses(<1month) 2,050 163 1,251 848 8,302 1,604 779 465 6,585 4,029 843 133 27,052
2.longcourses(>1month) 1,426 1,426
3.TrainingofTrainers 39 556 29 30 654
4.Mentoring 15 180 195
5.coaching -
6.Onlinelearning 2,159 3 2,162
7.Webinars -
8.KnowledgeExchange(Exchanges/placements/buddying/secondments)
-
9.Simulations 1,899 1,899
# of Capacity Building Strategies Employed
5 1 5 3 1 2 3 3 7 6 9 9 4
TOTAL # Trained based on triangulated data from document review
4,209 202 3,706 877 8,302 1,619 779 465 6,585 4,032 1,426 843 343 33,388
TOTAL # Trained as per Programme Logical framework reporting
2,050 192 3,706 1,026 5,142 1,614 782 241 5,133 4,032 1,426 624 386 24,304
Table4.2:TypesofcapacitybuildingstrategiesemployedbyDEPPprojectsandnumberofindividualstrained
* Yellow shaded cells indicate that the project implemented that particular capacity building strategy. (e.g., LPRR implemented short courses and mentoring).
Intermsofindividual-levelcapacitybuilding,universally,amulti-prongedapproachemployingseveralstrategiestogetherwasperceivedtobethemosteffectivebyDEPPstakeholders. Ingeneral,thisinvolvedanin-persontrainingcombinedwithanotherstrategysuchasmentorship,coachingorsimulations.Distanceor remote learningwasnot reportedtobeaseffectiveasin-personapproaches.Asdescribedlaterinthischapter,astherewerenomeasurablechangesinknowledge,itisnotpossibletoquantitativelyassesswhichcapacitybuildingapproachwasmosteffective.Inaddition,anumberofintervieweesdescribedinstitutionalbarrierstoapplyingknowledge or skills gained through individual capacity building and this was supported byfindingsfromthedocumentreview.124
Organisationalcapacitybuilding,especiallyofl/nnGOswasalsoseenascriticalforstrengtheninghumanitarian response and the greatest perceived organisational-level changes appear tobeoccurringwhenseveralstrategiesarecombinedandtailoredtothespecificgapsofeachorganisation.The greatest successes described by project stakeholders have often involvedprojects such as Shifting the Power and financial Enablerswhere beneficiary organisations
124 forexample,TalentDevelopmentfinalEvaluationReport,p45.
107 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
are empowered to self-assess gaps in capacity and identify their own strategies and plans.ThefinancialEnablersproject,inparticular,alsotookthestepofre-imaginingthefundingandreportingmechanismsof its localgrantees inorder tomake the fundsmoreaccessible.TheProtectioninPracticeprojectprovidedcashawardsratherthangrantstominimisereportingburdenonl/nnGOsand increaseflexibilityofhow fundswereutilised. Inboth cases, thisapproachseemstohaveworkedwell.TheShiftingthePowerprojectreportedthatstrengtheningtheadministrativepolicies,proceduresandprocesses(suchasfinancesystems,humanresourcesandprocurementprocesses)greatlybenefittedorganisationsandimprovedtheiroveralllevelofprofessionalism.125
3. Evaluation Output Area 3: Generating and Sharing Learning
Generation and sharing of learning was high across the programme but this learning was not based on empirical evidence. In addition, there is no quantitative evidence that the learning translated to behaviour change. Accessibility to learning products especially for local stakeholders was an issue. Thethirdevaluationoutputareafocusesonlearningandisdirectly linkedtoresultareafivewhich aims to strengthen theevidencebase forwhatworks tobuildhumanitarian capacityatscale.TheDEPPwasuniqueinthatoneofitsprojects,thelearningProject,wasdevotedentirelytothegoalofgeneratingandsharinglearningandevidence.Overall,findingsrelatedto the learning Project were mixed and have been documented extensively in previousreports.forexample,certainaspectssuchasthein-countrylearningadvisorswereeffectiveinanumberofcontexts,andeventssuchasthe internationalconferencesstimulatedcross-projectcollaboration.Ontheotherhand,thein-countrylearningadvisorswereineffectiveinothersettings,andtheemphasisingeneralofgeneratinglargequantitiesoflearningwithlessemphasison rigoroushighqualityevidencewasproblematic.However, thelearningProjectincorporatedpreviousrecommendationsmadebytheexternalevaluation,whichinturnledtosomeimprovementsintermsofthequantityofevidencegenerated.
Inaddition,severalpiecesofresearchwerecommissionedbythelearningProjectincludingareturnoninvestment(ROI)study,aswellasacasestudyonlocalisationduringthehumanitarianresponse to theRohingyacrisis inMyanmarandanother focusedonwomen’s leadership.126 TheROIanalysis inparticularmaynothavebeenundertakenattheappropriatetimeduringtheprogrammecycle,and likelywouldhavebeenmoreappropriatepriortothestartoftheprogramme. Given that it was implemented as the DEPPwas coming to a close, it couldpotentiallyhaveusedactualoutcomedatabutinsteadreliedonhypothetical,projectedchangesinoutcomes.ThislimitationoftheROIanalysisisimportant,especiallygiventheROIfindingshaveledtoconfusionamongstakeholderswithrespecttotheevaluationofoverallprogrammeoutcomesversusprojectedorhypotheticaloutcomes.Thisbrings intoquestiontheutilityofconductinganROIanalysispost-programmeversusbeforetheprogrammewasimplemented.WhilethisparticularROIstudywasneverintendedtoproducetheactualrateofreturn,suchanapproachwouldhavebeenavaluableexerciseattheendoftheprogramme.
125 ShiftingthePowerfinalEvaluationReport,p9.126 Thefinalproductsfromthesethreestudieswerenotavailableatthetimeofthedocumentreview.
108 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
QuarterlyReports
blog
Report/Review
caseStudy
Video
Keyfindings/Keylearnings/lessonslearnt
Guidelines/MinimumStandards
Article
ProgrammeandProjectlevellogicalframeworks
Podcast
finalProjectEvaluations
finalReportnarratives
Template/How-toGuide/Toolkit
conceptnote
Summary/ExecutiveSummary/ReportSummary
Handbook/fieldGuide/framework
TOR
Presentation/Slides
PolicyPaper
Research
Resourcecatalogue
Methodology
MobileApp
ReflectionJournal
newsletter
businesscase
Anal
ysis
TOTA
L
TOTA
L O
N T
HE
LEA
RNIN
G P
LATF
ORM
ADcAP 14 3 3 1 1 1 1 1 1 1 27(4.3%)
13(3.9%)
ALERT 12 1 1 1 1 1 17(2.7%)
1 (0.3%)
cDAc-n 12 1 2 1 1 1 12 1 1 1 1 34(5.4%)
19(5.7%)
FE 11 2 1 1 1 1 1 1 1 20(3.1%)
5(1.5%)
EWEA 10 2 1 1 1 15(2.4%)
2 (0.6%)
learningProject 10 45 19 1 37 12 10 2 3 1 2 1 1 5 2 2 153
(24.1%)118 (35.3%)
LPRR 13 1 4 12 2 1 1 1 1 5 1 1 43(6.8%)
25(7.5%)
PHEP Gambella 11 2 1 1 1 1 1 1 19
(3.0%)4
(1.2%)
PiP 13 1 1 1 1 6 1 24(3.8%)
8 (2.4%)
SEPS 11 3 1 1 1 1 18 (2.8%)
0(0.0%)
STP 14 5 14 21 6 4 2 1 1 1 1 1 2 73(11.5%)
29(8.7%)
TD 14 11 7 5 4 8 1 1 1 1 1 1 55(8.7%)
38 (11.4%)
TSc 14 3 20 8 6 4 6 10 1 1 1 1 2 1 1 2 81 (12.8%)
42 (12.6%)
UEWEA 9 4 1 1 1 1 1 18 (2.8%)
6 (1.8%)
non-DEPP 3 8 15 1 3 3 2 1 1 1 38(6.0%)
24(7.2%)
TOTAL 168 80 76 68 60 38 21 19 16 14 13 10 9 8 6 6 6 5 2 2 2 1 1 1 1 1 1 635 334
Table4.3:Typesofdocumentsproducedbyprojectsandreviewedaspartofthedocumentreview
Table 4.3 presents the type and number of documents that the evaluation team consultedaspartofthedocumentreviewprocess.Thenumberandavailabilityofdocumentscertainlyincreased sinceprevious reports, andaccessibility todocumentswasenhanced through theonlinelearningPlatform.However,themajorityofavailableoutputsdonotconstituterigorousevidence (Table4.4).Thegeneratedcontent isbasedonstoriesandperspectives related tokey DEPP concepts including collaboration, localisation, and capacity building, which werevaluableforprojectimplementationandcouldalsopotentiallyhelpprovidelessonsforfutureprojectdesign.Ofthe334documentsandreportsonthelearningPlatform,28%didincludeempiricalfindingsbuttheywereofvariablequality.Overall,thedocumentreviewrevealedthatthequalityofevidencegeneratedacrosstheDEPPbythelearningProjectandtheindividualprojectswaslow,asitwasprimarilyanecdotalandlackedscientificrigor.Partofthisstemsfromthelimitedguidanceprovidedtoprojectsduringthedesignandinceptionphasesaboutwhattypeoflearningandevidencearerequired,andpartisduetothelimitedcapacityofprojectstaffandpartnersrelatedtoM&E.TheTransformingSurgecapacityfinalevaluationalsonotedtheemphasisonquantityoverqualityandreportedthat:“fewer,bettertargetedproductsmayhavebeenmoreeffective.”127
127 TransformingSurgecapacityfinalEvaluationReport,p48.
109 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
EMPIRICAL EVIDENCE INCLUDED
(METHODOLOGY AND RESULTS)
DEPP PROJECT ACTIVITIES
PRESENTED OR DESCRIBED
LEARNINGS FROM DEPP DESCRIBED
TOTAL DOCUMENTS
ON THE LEARNING PLATFORM
yES NO yES NO yES NO
ADCAP1 2 2 1 2 1
3(33.3%) (66.7%) (66.7%) (33.3%) (66.7%) (33.3%)
ALERT1 0 1 0 1 0
1(100.0%) (0.0%) (100.0%) (0.0%) (100.0%) (0.0%)
CDAC-N3 16 17 2 16 3
19(15.8%) (84.2%) (89.5%) (10.5%) (84.2%) (15.8%)
FE1 4 4 1 4 1
5(20.0%) (80.0%) (80.0%) (20.0%) (80.0%) (20.0%)
EWEA 1 1 2 0 0 2
2(50.0%) (50.0%) (100.0%) (0.0%) (0.0%) (100.0%)
LEARNING PROJECT
20 108 85 43 81 47128
(15.6%) (84.4%) (66.4%) (33.6%) (63.3%) (36.7%)
LPRR15 10 13 12 6 19
25(60.0%) (40.0%) (52.0%) (48.0%) (24.0%) (76.0%)
PIP0 8 3 5 1 7
8(0.0%) (100.0%) (37.5%) (62.5%) (12.5%) (87.5%)
PHEP GAMBELLA
2 2 4 0 3 14
(50.0%) (50.0%) (100.0%) (0.0%) (75.0%) (25.0%)
STP9 20 26 3 24 5
29(31.0%) (69.0%) (89.7%) (10.3%) (82.8%) (17.2%)
SEPS0 0 0 0 0 0
0(0.00%) (0.00%) (0.00%) (0.00%) (0.00%) (0.00%)
TD13 25 35 3 33 5
38(34.2%) (65.8%) (92.1%) (7.9%) (86.8%) (13.2%)
TSC22 20 33 9 23 19
42(52.4%) (47.6%) (78.6%) (21.4%) (54.8%) (45.2%)
UEWEA0 6 4 2 2 4
6(0.0%) (100.0%) (66.7%) (33.3%) (33.3%) (66.7%)
NON-DEPP6 18 1 23 0 24
24(25.0%) (75.0%) (4.2%) (95.8%) (0.0%) (100.0%)
TOTAL94 240 230 104 196 138
334(28.1%) (71.9%) (68.9%) (31.1%) (58.7%) (41.3%)
Table4.4:TypesofevidencesharedonthelearningPlatform
AccessibilityofdocumentsforprojectandprogrammestakeholdersimprovedbutthebulkoflearningproductsproducedareavailableonlyinEnglish.basedonpreviousrecommendationsoftheexternalevaluation,technicalsupportwasprovidedtostrengthenindividualprojectM&Eand all projectswere required to undertake an end of project evaluation.All projects havefulfilledthisrequirement,thoughagainthequalityhasbeenvariable.
110 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Therewere some improvements in inclusion of gender disaggregated data, andVfM data,buttheseremainedbelowexpectations.InanyfutureimplementationoftheDEPPorsimilarprogrammes,guidanceandsupportforVfMassessmentstrategiesshouldbeprovidedearlyandconsistentlythroughoutthedesignandimplementationphases.
4. Evaluation Output Area 4: Early Warning System Development
Six diverse early warning systems were developed in three countries and all six are at least partially operational, though the use and testing of each system was limited.
EarlywarningsystemdevelopmentwasakeytargetareafortheDEPP.fivedifferentprojectsinanumberofdifferentcontextsfocusedonthisparticularobjective.Theseincluded:
1. ALERT2. ImprovedEarlyWarningEarlyActionEthiopia(EWEA)3. PublicHealthPreparednessinGambella(PHEPGambella)4. StrengtheningEmergencyPreparednessSystemsinMyanmar(SEPS)5. UrbanEarlyWarningEarlyAction(UEWEA)
Thesefiveprojectsaimedtodevelopsixdifferentearlywarningsystems(SeeTable4.5below)in order to enable early response towards natural disasters such as flooding and drought,urban emergencies such as food insecurity and household shocks, aswell as public healthemergencies including disease outbreaks.Oneof the projects, PHEPGambella, contributedto two systems.As described in previous evaluation reports, most of these projects facedsignificant implementation challenges and delays. In many cases, these delays were linkedtochallenges inprocurementofequipmentsuchasweatherstations128,delays insettingupagreementswiththegovernment129,outbreakofviolenceintargetcommunities130orindelaysindevelopingthesystem131.Infact,delaysweresevereenoughthattherewasanevidentriskthatnotallsystemswouldbeoperationalbeforetheendoftheprogramme.Theinterimreportrecommendedaccelerationofprojectactivitiestoavoidthispotentialoutcome.
PRO
JECT
TYPE
OF
SYST
EM
COU
NTR
Y AN
D
LOCA
TIO
N
SYST
EM
DEV
ELO
PMEN
T
GO
VERN
MEN
T U
SIN
G
SYST
EMN
GO
S, C
SOS
USI
NG
SY
STEM
SIM
ULA
TIO
NS
CON
DU
CTED
CAPA
CITY
BU
ILD
ING
AC
TIVI
TIES
TO
TRA
IN
ON
SYS
TEM
CON
TIN
GEN
C F
UN
DIN
GSY
STEM
USE
D IN
RE
SPO
NSE
ALERTPreparednesssystem(software)
KenyaPakistan,ThePhilippines,bangladesh,Mozambique
✔ ✔ ✔ ✔Trainings,Workshops
✖ ✖
EWEAMonitoringandearlywarningsystem
Ethiopia ✔ ✖* ✖** ✖
Trainings,Trainingoftrainers,Workshops
✖ ✖
128 ImprovedEarlyWarningEarlyActionEthiopiafinalEvaluationReport,p17.129 StrengtheningEmergencyPreparednessSystemsinMyanmarfinalEvaluationReport,p19.130 Ibid.131 AlERTfinalEvaluationReport,p37.
111 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
PHEP Gambella
Monitoring,earlywarningandpreparednessplansandsystems
Ethiopia ✔ ✔ ✖ ✖Trainings,Workshops, coaching
✖*** ✔
SEPS
communitypreparednessplansandearlywarning
Myanmar ✔ ✔ ✔ ✔Trainings,Workshops
✖ ✖
UEWEAMonitoringandearlywarningsystem
Kenya(nairobicity) ✔ ✔ ✔ ✔
Trainings,Workshops,Reports,Toolkits,Workshops
✔ ✔
Table4.5:EarlywarningsystemsdevelopedbytheDEPP
*Governments are expected to adopt the system when implementation is complete**NGOs/Civil Society Organisations (CSOs) are expected to adopt the system when implementation is complete*** Contingency funding in the process of being secured
Atthetimeofthissummativereport,allsixearlywarningsystemshadbeenfullydeveloped,fivewereoperationalandtwohadbeenusedinaresponse.However,becauseoftheimplementationchallenges,manyofthesystemshavebeenoperationalforaveryshortperiodoftimewhichmeansthattheyhavenothadtheopportunitytobefullytestedorusedinameaningfulway132.It alsoprevents assessmentsof the effectivenessof the systems.TheUrbanEarlyWarningEarlyActionsysteminnairobicitywasoperationalthelongest,andthereforehadthelongestperiodoftestinganduse.Themainfactorcontributingtoitsquick(relativetoothersystems)deploymentwasthatthesystemwasconceptualisedtobuildonapreviousfive-yearproject133 whereintheurbanearlywarning indicatorsweredevelopedandtested,andthesurveillancemethodologyandsystemprocesseswerearticulatedandrefined.Giventhatitwasabletobuildonthepreviousworkthathadbeenconducted, thesystemwasnotbuilt fromscratch,andthusthegoalofdevelopingaworkingsystemwithinashortprojectperiodwasachieved. Infact,thissystemwasabletodetectthreeemergenciesandtriggerresponseactions.Theseareexploredinfurtherdetailintheboxeslaterinthischapter.Thiswasnotthecasefortheotherearlywarningsystems.Itwasoptimistictosetupandtestanearlywarningsystemfromscratchwithinatwo-tothree-yearproject,andfutureprojectsofthisnatureshouldsetmorerealistictimeframes.
132 forexample,theImprovedEarlyWarningEarlyActionsysteminEthiopiawasreportedtobeonlypartiallyfunctional:TheAutomaticWeatherStationswere“notinstalledasplannedandstillnotfunctionalduetointernetunavailabilityintheareas”,ImprovedEarlyWarningEarlyActionEthiopiafinalEvaluationReport,p6.Also,theAlERTfinalEvaluation,p4reports:“noorganisationhadfullydeployedtheAlERTplatformatthetimeofthisevaluation.”
133 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).ImprovedEarlyWarningEarlyActionEthiopia,finalEvaluationReport,p1.
112 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
QUESTION 2.B, 2.C AND 2.D: TO WHAT EXTENT IS DEPP CONTRIBUTING TO GREATER PREPAREDNESS AND RESPONSE AMONG LOCAL ORGANISATIONS AND COMMUNITIES? HAS LOCAL CAPACITY TO RESPOND TO DISASTERS CHANGED SINCE THE START OF DEPP? IF YES, HOW HAS IT CHANGED? IF NOT, WHY NOT? HAS DEPP LED TO IMPROVED KNOWLEDGE AND UNDERSTANDING OF BEST PRACTICES RELATING TO DISASTER AND EMERGENCY PREPAREDNESS AND RESPONSE? IF YES,IN WHAT WAYS?
1. Knowledge Change
Overall, there have been no significant improvements in knowledge on core humanitarian competencies of humanitarian staff, on self-reported knowledge on disaster preparedness, response to disasters and emergences or age or disability-related issues in disasters.
AkeyareaofassessmenttoevaluatetheextenttowhichcapacityhasbeenbuiltasaresultoftheDEPPtraining, isknowledgechangeamonghumanitarianstaffinseveralkeythematicareasrelatedtodisasterandemergencypreparednessandresponse.figure4.6presentsdataonknowledgechangeacrosssixkeyareasusing thepooleddata fromall four intensivesetcountriesaswellastheminimumsetdata.All10DEPPcountriesarethereforerepresentedinthisanalysis.Therewasnosignificantchangeintheoverallcompositeknowledgescorerelatingto the core humanitarian competencies, or in any of the self-reported indicators related toknowledgeonspecificthemes(disasterpreparedness,humanitarianresponse,orage,disabilityor gender issues in a disaster) within DEPP or non-DEPP comparison humanitarian staff.furthermore,therewasnosignificanteffectoftheDEPPonanyoftheseknowledgeindicators,globallyorwhenrestrictedtoindividualcountriesortolocalorganisation(Seefigure4.7).Inaddition,therewasnosignificanteffectoftheDEPPwhenthesamplewasrestrictedonlytothoserespondentswhoreportedparticipatingincapacitybuildingwithinthelast12months.
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Core Humanitarian Competencies + 0 . 5% - 0 . 5% +1 . 0 %
Disaster Preparedness +1 . 8 % + 4 . 2 % -2 . 4%
Response to disasters and emergencies + 0 . 6% +3 . 4% -2 . 8 %
Age related issues in a disaster +1 . 0 % +1 . 0 % 0 . 0 %
Disability related issues in a disaster + 0 . 6% + 0 . 5% +1 .1%
Issues related to women in a disaster + 0 . 8 % - 0 . 5% +1 . 3 %
figure4.6:Individualknowledgechangeofhumanitarianstaffonkeythematicareas
* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant
ThesefindingssuggestthatwhilemanyindividualshavebeenexposedtotheDEPPcapacitybuilding, according to both logical framework data and independent data collected by theevaluationteam(figure4.6),significantknowledgechange(bothviaobjectivemeasuressuchasthecorehumanitariancompetenciesindicatorandthroughself-reportedmeasures)inseveralexpectedthematicareashasnotoccurred.
113 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Ethiopia +2 . 2 % -1 .7 % +3 .9 %
Kenya 0 . 0 % 0 . 0 % 0 . 0 %
The Philippines +1 .9 % + 0 .9 % +2 . 8 %
Myanmar -1 . 2 % +1 . 6% -2 . 8 %
Local + 0 .7 % +2 . 8 % -2 .1%
figure4.7:Individual-levelknowledgechangeofhumanitarianstaffoncorehumanitariancompetenciesbycountry
* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.
2. Emergency Preparedness
There were no significant quantitative improvements in perceived individual emergency preparedness levels in the DEPP group, but there were significant improvements in preparedness of DEPP organisations across all intensive set countries, and in particular for L/NNGOs. While access to early warning system information was significantly higher in DEPP communities than in non-DEPP comparison communities, the perceived level of community preparedness significantly decreased in both DEPP and non-DEPP comparison communities in Myanmar.
Emergencypreparednesswasassessedat the individual,organisationalandcommunity levelthrough a series of indicators using perceived (self-reported) measures or more objectiveassessments (See box below). The self-reported or perceived measures assess data fromhumanitarianstaffwhoratedtheirownortheirorganisationslevelofpreparednessaswellashouseholdswho rated their community’s level of preparedness to respond to disasters andemergencies.Atthecommunity level,preparednesswasalsoassessedthroughself-reportedexistenceofanearlywarningsystem.
Asself-reporteddatabasedonperceptionsareknowntobeproblematicandoftenbiased,moreobjectivemeasureswerealso included.Atthetimeofthedevelopmentofthemethodologyfor this evaluation, therewas an absence of standardised, objective indicators and tools toassessorganisationalandcommunityemergencypreparedness.Theevaluationteamdevelopedanorganisationalchecklist spanningsevendomains tomore rigorouslyassessorganisationalpreparedness.Theevaluation teamalsoadapted theGOAl toolkit for assessing communityresilience134 comprising a seriesofquestions across30differentdomains inorder to assesscommunitypreparedness(Seeboxbelow).Thepreparednessindicatorsthereforeincludeamixofperceivedandalsomoreobjectivemeasures.
KEY PREPAREDNESS INDICATOR DATA COLLECTED AT 2 TIMEPOINTSIndividual level:
• Perceived: KAP survey with humanitarian staff. As a responder, how would you rate your level of preparedness to respond to a disaster? Where 1=Very unprepared, 2=Unprepared, 3=Somewhat unprepared, 4=Prepared, 5=Very prepared
134 ToolkitforMeasuringcommunityDisasterResilience.GuidanceManual.Goal,2014.
114 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Organisational level:
• Perceived: KAP survey with humanitarian staff. At what level would you rate your organisa-tion’s preparedness to respond to disasters and emergencies? Where 1=Very unprepared, 2=Unprepared, 3=Somewhat unprepared, 4=Prepared, 5=Very prepared
• Actual: Organisational Checklist conducted with each organisation and assessing seven domains: 1-Emergency preparedness plans, 2-human resources and capacity, 3-logistics and supplies, 4-Legal, documentation and policy, 5-Organisation and system processes, 6-Emer-gency response team and staffing, 7-Communication and media. Aggregate score out of 100 generated for each organisation.
Community level:
• Perceived: Household survey: How prepared do you feel your community is to respond to disasters in the future? Where 1=Not prepared at all, 2=Slightly prepared, 3=Somewhat prepared, 4=Prepared, 5=Very prepared
• Perceived (early warning system): Household survey: Does your area have an early warning system?
• Actual: Community focus group using a preparedness assessment tool conducted in each community, assessing 30 domains including emergency response and recovery, contingency planning, early warning systems, capacities in preparedness and response, access to health-care in emergencies, and hazard assessment.
figure4.8presentsthefindingsrelatedtoeachofthepreparednessindicatorsusingthepooleddatafromallcountries.Attheindividuallevel,therewerenosignificantchangesinperceivedlevelsofownpreparednessintheDEPPgrouporthecontrolgroup,andtherewasnoneteffectoftheprogrammeonindividualpreparedness.Attheorganisationallevel,ontheotherhand,significantimprovementsinorganisationalpreparedness,bothwiththeself-reportedindicator,aswellaswiththemoreobjectiveassessmentusingtheorganisationalchecklistwithintheDEPPgrouponly have occurred.However, theDifference-in-Difference analysis showsnooverallsignificanteffectoftheprogrammeintermsoforganisationalpreparedness,likelyduetosimilarchangesinthecomparisongroupoverthesameperiod.figure4.9presentstheorganisationalchecklist data (the rigorous indicator of organisational preparedness) for each country andfor local organisations. Therewere significant improvements in organisational preparednessspecificallywithinKenya(anincreaseof11.1%)andalsowithinlocalorganisations(anincreaseof10.1%)acrossallintensivecountries.Again,however,theDifference-in-Differenceanalysis,which takes into account changes in theDEPP and in the control group over time, showsno statistically significant net effect of the programmeon organisational preparedness.ThemeasuredimprovementsinorganisationalpreparednessintheDEPPorganisations(especiallylocalorganisations)arealsosupportedbyin-depthinterviews,documentreviewandthecasestudy(chapter8).forexample,inEthiopia,localbeneficiaryorganisationsoftheShiftingthePowerprojectnoted the improvements in theirorganisationspreparedness levels, andcitedchangedattitudesandmindsets,inter-departmentalcoordination,improvedprocurementplansandstrengthenedorganisationalproceduresincludingcreatingemergencyresponseteams.135
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Individual Preparedness (Perceived)
+2 . 2 % - 0 . 5% +1 .7 %
135 Seecasestudy#2inchapter8.AlsotheShiftingthePowerfinalEvaluationReport,p9,alsodescribesthesechangesinorganizationalpreparedness.
115 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
Organisational Preparedness (Perceived)
+2 . 0 %* + 0 . 5% -1 . 3 %
Organisational Preparedness (Actual) +9. 3 %* +3 . 0 % + 6 . 3 %
Community Preparedness (Perceived)
-13 . 4%* -16 . 8 %* +3 . 4%
Community Preparedness (has early warning system)
+ 4 0 . 5%* + 0 .1% + 4 0 . 4%*
Community Preparedness (Actual) + 4 .9 % -7. 2 % -2 . 2 %
figure4.8:change inperceivedandactualpreparednessat individual,organisational,andcommunitylevels
* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant
Withrespecttocommunitypreparedness,therewasasignificant41%increaseinthecommunityaccess toearlywarningsystems inDEPPcommunities inMyanmarcompared tonochangein non-DEPP comparison communities in the same area.With theDifference-in-Differenceanalysis,theneteffectoftheDEPPprogrammewasa40%increaseincommunityearlywarningsystems.However,surprisingly,thissubstantialeffectdidnottranslatetoimprovedperceptionsofcommunitymembersonthepreparedness leveloftheircommunities. Infact,perceptionsonlevelofcommunitypreparednesssignificantlydecreasedby13%inDEPPcommunitiesand17% in non-DEPP communities.Attitudes related to thevalue of preparedness activities inimproving responsewerenotpositiveanddidnotchangeovertimewhichmayexplain thisfinding. furthermore, findings during the validation workshop suggest that changes in thegovernment andother contextual factorsmayhave led to increasedpessimismon thepartofcommunitymembers.finally,theobjectiveassessmentofcommunitypreparednesswhichaggregatedindicatorsacross30domainsshowednosignificantchangeinDEPP,controlorviatheDifference-in-Differenceanalysis.
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Ethiopia +13 .9 % - 6 .9 % +2 0 . 8 %
Kenya +11 .1%* +7. 6% +3 . 4%
The Philippines +5 . 0 % + 4 . 6% + 0 . 4%
Myanmar + 6 . 2 % + 4 . 6% +1 . 6%
Local +10 .1%* + 6 . 8 % + 8 . 8 %
figure4.9:changeinactualorganisationalpreparednessinEthiopia,Kenya,thePhilippines,Myanmar,andforlocalorganisationsacrossthefullsample
* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant
116 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
3. Humanitarian ResponseDEPP contributed to at least 42 different humanitarian responses in 11 countries; 16 of these responses occurred in the Philippines (and 14 of these 16 responses were driven by one project – the Financial Enablers project). Qualitative data indicate contribution of DEPP beneficiaries towards improved response in the Philippines, Kenya and Ethiopia, but many other DEPP beneficiaries were unable to respond due to lack of response funds.HumanitarianresponsewasnotconsideredbymostprojectsinthedesignphaseastheDEPPwasintendedtofunctionasadedicatedpreparednessprogramme.Assuchlinkswithresponseactivities were not explicitly developed into the design of the programme. Many projectstakeholders,infact,nowconsideranyDEPPcontributiontowardshumanitarianresponsetobeanunintendedbutpositiveconsequenceofDEPPactivities.Humanitarian response activities and outcomeswere assessed in a number ofways in thisevaluation. first, descriptive primary data were collected at the individual, organisationand community level on the types of disasters and emergencies that have occurred in theprevious12months,aswellastypesofresponseactivities implemented.Second,anumberof quantitative indicators related to performance during past humanitarian responses andabilitytorespondinthefutureweremeasuredatthetwotimepoints(Seeboxbelow).Third,the document review allowed systematic mapping of all responses that DEPP reportedlycontributedtoduringtheprogrammeperiod.fourth,additionalresponsedatawascollectedduring theevaluationvalidationworkshops inKenya,Ethiopia,MyanmarandthePhilippinesinMarch2018,tobetterunderstandwhichdisastersandemergenciesstakeholders inthosecountrieswere able to respond to and tounderstand factors related toboth response andnon-response. Inaddition, responsedatawerecollected fromeachprojectbyemail.finally,additionalqualitativeinterviewswereconductedinKenyaandtheEthiopiatocontributetoamoredetailedhumanitarianresponsefocusedcasestudy(seechapter8).figure 4.10 below presents data on the percentage of DEPP and non-DEPP comparisonorganisationsthatrespondedtoadisasteroremergencyinthepast12months.Ofnote,DEPPorganisationswereaskedtoreportonanyresponseactivityregardlessofcontributionoftheDEPP.Asdepictedinthefigure,thevastmajorityofbothDEPPandnon-DEPPorganisationsimplementedhumanitarian responseactivities in theprevious12months,andtherewasnostatisticaldifferencebetweenT1andT2ineithergroup.furthermore,localnGOswhowereDEPPbeneficiarieswerenotmore likelytorespondtoadisasterafterexposuretoDEPPorwhen compared to non-DEPP local organisations.The types of disasters did vary betweenDEPPandnon-DEPPorganisations,withflooding,droughtanddisplacementbeingthemostcommonacrosstheDEPPorganisations,anddrought,floodingandinfectiousdiseaseepidemicsbeingmostcommonamongthenon-DEPPcomparisonorganisations(figure4.11).Thetypesofresponseactivitiesaresummarisedinfigure4.12highlightingthediverserangeofactivities,though food security and water, sanitation and hygiene (WASH) activities were the mostcommoninbothDEPPandnon-DEPPorganisations.
78.8
0
20
40
60
80
100
ORGANISATION RESPONSES TO DISASTER OR EMERGENCY IN PREVIOUS 12 MONTHS
ORG
ANIS
ATIO
NS
(%)
DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON DEPP COMPARiSON
LOCAL NGOS KENYATHE PHiLiPPiNESETHiOPiAMYANMARALL
T2
T1
79.5
71.8 79
.2 84.1
69.6
47.8
85.0
68.8
95.2 10
0
100
100
100
81.082
.6
64.368
.4
92.9
78.8
66.7
76.9
67.7 73
.7 77.1
figure4.10:Percentageoforganisations that responded toadisasteroremergency in theprevious12
117 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
figure4.11:Types of disasters and emergencies that organisations responded to in theprevious12months
figure4.12:Typesofresponseactivitiesimplementedbyorganisationsintheprevious12months
Quantitative indicators to assess perceived performance during a previous response duringthe last12months, andaswell as futureability to respond toadisasteroremergencyaresummarisedinfigure4.13.Attheindividuallevel,humanitarianstaffatDEPPandnon-DEPPcomparisonorganisationswereaskedtoratetheirpastresponseperformanceandtheirabilitytorespondinthefuture(seeboxbelowfordescriptionoftheindicators).
LANDSLiDES
DISASTER AND EMERGENCY RESPONSES
COMPARiSON
DEPP
0
10
20
30
40
50
60
TYPHOONS
FLOODiNG
VOLCANO ERUPTiONS
STORM SURGE
DROUGHT
WAR AND CONFLiCT
iNFECTiOUS DiSEASE EPiDEMiC
EARTHQUAKES
MONSOON
MASS EXPOSURE TO TOXiC CHEMiCALS
DiSPLACEMENT
9.0
3.4
44.1
33.6
9.6
10.1
40.8
52.1
5.7
5.9 3.
3
1.7 8.4 3.4
16.2
29.4 30.6
22.7
34.2
26.9
0.3 0.8 9.
3
2.5
RESP
ON
DEN
TS (%
)
ORGANISATION RESPONSE ACTIVITIES
COMPARiSON
DEPP
0
10
20
30
40
50
60
70
80
12.6 16
.1
62.2
68.8
62.9
54.7
33.6
46.9
25.9 28
.1
26.6
43.8
31.5 35
.9
28.0
42.2
30.1
39.1
20.3
29.7
21.0
34.4
8.4
15.6
11.2 14
.1 16.8
14.1
20.4
31.3
ORG
ANIS
ATIO
NS
(%)
WASH
FOOD SECURiTY
NONE
EDUCATiON
SHELTER
HEALTH
MEL
NUTRiTiON
CHiLD PROTECTiON
AWARDS
FUNDRAiSiNG
LOGiSTiCS
MANAGEMENT
SURGE
TELECOMMUNiCATiONS
118 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
KEY RESPONSE INDICATOR DATA (RELATED TO RESPONSE IN THE PREVIOUS 12 MONTHS) COLLECTED AT 2 TIMEPOINTS
Individual level:
• Perceived past performance: KAP survey with humanitarian staff. How would you rate your performance during your last response? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Ex-cellent
• Perceived future ability: KAP survey with humanitarian staff. How would you rate your ability to respond to a disaster in the future? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Excellent
Organisational level:
• Perceived past performance: Organisational survey with senior leadership. How would you rate your organisation’s overall response to the disaster or emergency (in the last 12 months)? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Excellent
• Perceived future ability: Organisational survey with senior leadership. How would you rate your organisation’s ability to respond to a disaster in the future? Where 1=Poor, 2=Fair, 3=Good, 4=Very good, 5=Excellent
Community level:
• Satisfaction with previous response: Household survey: How satisfied were you with pre-vious response to disaster? Where 1=Not satisfied at all, 2=Slightly satisfied, 3=Somewhat satisfied, 4= Satisfied, 5= Very Satisfied
ThereisadownwardtrendamongstbothDEPPandnon-DEPPhumanitarianstaffreflectingdeclining perceptions of own performance and future ability to respond, though neither ofthesedifferenceswerestatistically significantexcept forpastperformance in thenon-DEPPcomparison group. Overall there was no net DEPP programme effect for either of theseindicators using the pooled data from the four intensive countries. Therewas also no netprogrammeeffectwhenrestrictingthesampletospecificintensivesetcountriesortoonlylocalorganisations.However,casestudyqualitativedatafromKenyaandEthiopiademonstratesomeindividual-levelchanges inpreparedness, includingknowledgeonprocedures,resources,andimprovedconfidencetorespond(Seechapter8).ThecDAc-nprojectfinalevaluationreportalsoreportedthatindividualsbenefittingfromtheirprojectinSouthSudanandbangladeshfeelmoreequippedandpreparedtoimplementresponseactivities,especiallythoserelatedtotwo-waycommunication136.Thisdemonstratesthattherehavebeendocumentedimprovementsinqualitativeindicatorsrelatedtoabilitytorespond,butthatthesehavenotyettranslatedintowidespreadquantitativechangeacrosstheprogramme.
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Individual Response (Perceived Past Performance)
- 0 . 6% - 4 .9 %* + 4 .1%
Individual Response (Perceived Future Ability)
-1 . 4%* - 6 . 2 % + 4 . 8 %
Organisational Response (Perceived Past Performance)
-1 . 8 %* +3 . 4% -5 .1%
136 cDAc-nfinalEvaluationReport,p47.
119 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
Organisational Response (Perceived Future Ability)
+2 . 8 %* + 4 . 5%* -2 . 4%
Community Response (Satisfaction with Past Performance)
-15 . 2 % -16 . 6% +1 .1%
figure 4.13: change in perceived response performance (both previous and future) at individual,organizationalandcommunitylevels
* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant
Attheorganisationallevel,therewerenosignificantchangesintheperceivedpastperformanceduringaresponse,orinperceivedfutureabilitytorespondinboththeDEPPandnon-DEPPcomparison groups in the four intensive set countries, and therewas no net effect of theprogramme on either of these indicators with the pooled or restricted samples. However,as described inmore detail below, qualitative data, and data extracted from the documentreviewdodemonstratesomeexamplesofimprovedorganisationalresponseactivities.Atthecommunity level, drawing on household data from Myanmar, beneficiary satisfaction withhumanitarianresponseintheprevious12monthssignificantlydecreasedby15%intheDEPPcommunitiesand17%inthecontrolcommunities.OveralltherewasnosignificantneteffectoftheprogrammeonbeneficiarysatisfactioninMyanmar.Discussionduringvalidationworkshopssuggestedthismighthavebeenduetoworseningcontextualfactorsandpoliticalclimatewithinthecountry.
Documentreviewandin-depthinterviewsdemonstratecontributionoftheDEPPtoatleast42differenthumanitarianresponsesin11countries(seeTable4.6below,andAnnex10).Therewerenodocumentedcontributions to responses inMozambique,oneof thecountrieswiththefewestDEPPprojects(oneproject).Thelargestnumberofresponsecontributions(n=16)occurredinthePhilippineswhichalsohadthehighestnumberofDEPPprojectsimplemented(n=6).notethatKenyaandPakistanalsohad6DEPPprojectsbutinthesespecificcontexts,DEPPcontributedtosubstantiallyfewerresponsesthaninthePhilippines.Thisdifferencewasalsonotedwith independentresponsedatacollecteddirectlyfromprojectsduringvalidationworkshops. In fact, according to data collected during these workshops, there was DEPPcontribution to 33 responses in the Philippines (versus 16 responses extracted from thedocument review).furtherdiscussiononwhyDEPPcontributed tomore responses than inother countries canbe foundbelow.Only three projects provided response informationbyemail (Table4.6).This is likelybecause theprojectswerewrappingup andproject staffdidnothavetimetocontributetothiscomponentoftheevaluation.completedetailsofthe42responseeffortsareinAnnex10.
COUNTRY PROJECTS
IMPLEMENTED (#)*
DOCU-MENT
REVIEW
INTENSIVE SET COUNTRY
WORKSHOPSSUBMITTED VIA EMAIL**
NUMBER
OF DEPP
RESPONSES
NUMBER
OF EVENTS
NUMBER
OF DEPP
RESPONSES
NUMBER
OF EVENTS
NUMBER
OF DEPP
RESPONSES
NUMBER
NOT RE-
SPONDED
TO
REASONS
FOR NOT
RESPOND-
ING
1The Philippines
PIP, FEP, TSC, CDAC-N, LPRR, ALERT (6)
16 33 33 1 1 0 n/A
2 MyanmarPIP, LPRR, SEPS (3)
2 2 2 4 3 1
OutoftheDEPPregionalscope
120 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
3 EthiopiaSTP, PHEP Gambella, EWEA, TD, (4)
4 12 12 3 1 2
ResponsewasnotfundedbytheDEPP
4 Kenya
STP, ADCAP, LPRR, TD, UEWEA, ALERT (6)
3 15 15 4 4 0 n/A
5 PakistanPIP, STP, TSC, ADCAP, LPRR, ALERT (6)
4 n/A n/A 6 DRC
PIP, STP, LPRR, TD (4)
2 n/A n/A 7 Bangladesh
STP, CDAC-N, LPRR, TD, ALERT(5)
6 1 1 7 7 0 n/A
8 South Sudan PIP, CDAC (2) 4 n/A n/A 1 1 0 n/A
9Jordan- Lebanon- Syria***
TD (1) 1 1 10 Mozambique ALERT (1) 0 n/A n/A 11 India**** TSC (1) 1 n/A n/A TOTAL 42 64 64 20 17 3 Table4.6.contributionofDEPPtohumanitarianresponseactivitiesinDEPPcountries
*The Learning Project was implemented in all DEPP countries and is not included, **Only four projects submitted data, ***One response reported by Talent Development in Kenya, ****TSC deployment
findings based on the document review, qualitative interviews and validation workshopssuggests thatprojects in thePhilippinesoutperformed theotherDEPP focal countrieswithrespecttocontributiontohumanitarianresponses.notethatuniversally,thesecontributionstoresponseswerenotnecessarilydesignedoutcomesfromtheoutsetbutbecameobjectivesasprojectsandplanningevolved.Somestakeholdersexpressed these responseactivities tobeunplannedsideeffects.QualitativedatasuggeststhatDEPPcontributiontosomeoftheseresponses led tomore timely, locally driven responses. In other cases, it is not possible toassesstimelinessoreffectivenessofaparticularresponseeffortwithavailabledatasources.Incontrasttothequalitativeindicators,quantitativedataonperceptionsof improvedspeedandeffectivenessof responseactivitiesdidnot showasignificantmeasurableeffectof theDEPP.furtheranalysisontheperformanceoftheDEPPtowardshumanitarianresponsesinthePhilippinesrevealsthefollowing:
• ThehighperformanceinthePhilippinesintermsofabsolutenumberofhumanitarianresponseeffortswaslargelydrivenbythefinancialEnablersprojectwhichcumulativelycontributed to14different responses (outof16DEPP responses in thePhilippinesduringtheDEPPprogramme)
• SeveralfactorscontributedtotheoverallperformanceinthePhilippinesvis-à-visothercountries.Theseinclude:
o The fact that the Philippines has a high frequency of disasters/emergencieseach year to which projects could theoretically respond. This has createdgreater incentive forgovernmental andnon-governmentpartners todeveloppreparednessprojectsandmoreopportunityfortheseactorstobuildupontheir
121 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
experiencefromeachresponse.
o networks in thePhilippinesarehighly locallydriven (seechapter5),andthegovernmentandotherkeyactorsprovidespaceforcivilsocietyaction.
o flexiblefundingmechanisms(viathefinancialEnablersproject,andtoalesserdegreeProtection inPractice)allowed localactors to identify relevantneeds,and implement accordingly. This included using funding towards responseactivities(viamechanismssuchasaquickresponsefund).ThiswasnotthecaseinnumerousothersettingswhereDEPPbeneficiarieswhowantedtorespond(andnowbelieved theyhad thecapacity to)were limited inwhat theycouldcontributeduetoabsenceofdesignatedresponsefunds.DEPPbeneficiariesinthePhilippineswerealsosuccessfulatsecuringresponsefundsexternaltotheDEPPsuchasviatheSTARTfund,orthroughotherdonors.
o The large number of DEPP projects being implemented in the Philippines(n=6)providedagreateropportunityforcollectiveaction,andalargeroverallinvestment.
o TheDEPPprojectsinthePhilippinesshowedasubstantiallyhighlevelofinter-projectcollaborationandwereable to jointlycontribute tocertain responses(i.e.,Marawiresponse).Overall, theDEPPprojects inthePhilippines, inmanyways,operatedmorelikeaprogramme(andtheinitialvisionoftheDEPP)thaninother focuscountries,albeitdespite thekeyprogramme-levelsystemsandprocessesthatwereessentialforportfoliofunctioning.
The contributions to humanitarian responses in the Philippines, Kenya and Ethiopia wereexamined in further detail and arepresented in theboxesbelow.All three settingsprovidesomeevidenceofimprovedorganisationalresponseactivities,andthePhilippinesandKenyademonstratethecontributionofthecollaborativeapproachandworkingthroughconsortiatoimprovedhumanitarianresponseactivities.
Humanitarian Response: Focus on the Marawi Crisis
What:
TheMarawicrisisbeganonMay23,2017,whenthegovernmentofthePhilippineslaunchedamilitaryoperationagainstpro-ISISmilitantsinthelanaodelSurprovinceofMarawicity.Thisfive-monthlongincursionresultedinthedisplacementofroughly98%ofthepopulation,oralmost360,000people137.
Who:
AtotalofsixDEPPprojectswereimplementedinthePhilippines:ProtectioninPractice,financialEnablers,TransformingSurgecapacity,lPRR,AlERTandthecDAc-ncommunicatingwithcommunitiesProject.
Objective:
TounderstandhowtheseprojectscontributedtoMarawiresponseefforts
Data sources:
• MinimumsetIDIswithprogrammeandprojectstakeholders(Qualitative)
• IntensivesetIDIs(Qualitative)
137 https://reliefweb.int/report/philippines/unicef-philippines-humanitarian-situation-report-marawi-re-sponse-10-november-2017;http://www.unhcr.org/ph/marawi-crisis.
122 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
• networkanalysis(Quantitative)
• Documentreview(Qualitative)
Key Findings:
Intotal,atleast16disastersandemergencies(includingflooding,typhoons,earthquakes,fires)inthePhilippineshadresponsecontributionbytheDEPP,asubstantiallyhigherfigurethanany other DEPP country. Self-reported data indicate successful inter-project collaborationandcoordination,wherecapacitybuildingoflocalorganisationsworkingtogetherinconsortiacontributedtoafaster,morelocallydrivenresponseeffort.ThemostprominentexampleofsignificantcontributionwastotheMarawicrisis.fourof thesixprojectsrespondedtotheMarawicrisesincludingthefollowingresponseactivities:
• Aprotectionnetworkof600localrespondersinMindanaoabletoimplementinclusiveprogrammingwasestablished
• localconsortiawhosecapacitywasstrengthenedbytheDEPPwerethefirsttorespondtothecrisis(anddidsoindependentlywithoutaleadInGO).TheywererecognisedforthefirsttimebyUnOcHAoncrisisresponsemapsdemonstratingincreasedpresenceandrecognitionoflocalactors
• Variousconsortiacoordinatedtheirresponsestogether,allowingforefficientuseofhumanresourcesandsharingofknowledge,contactsandresources
• Examplesdemonstratebeneficiary involvement indesignand implementationofsomeresponseactivities
• localactorsgainedconfidenceandskillsastheytookonmoreresponsibilityduringtheresponse
factorsthatenabledafastermorelocallydrivenresponseincludedthefollowing:
• Incontrasttoothercountryexamples,consortiapartnersinthePhilippineswereabletoaccessquickresponsefundsthroughanumberofmechanismswhichenabledresponseactivitiestobeimplementedrapidly
• ThePhilippines isadisaster-pronecontext,withahumanitariannetwork that ishighlydrivenby localactors.This isreflectiveofacontextwherethegovernmentandotheractorsprovidespaceforcivilsociety,which isdeeplyengrained insocietyandcultureandconsideredtobethestrongestinAsia138.TheprominenceoflocalactorswithinthePhilippinesallowscollaborativeactionatthelocalleveltooccurmoreorganicallythaninothercountrycontexts
• AkeyenablingfactorwasthemandatebythegovernmentthatonlylocalnGOscouldrespondtothiscrisis.Thisprovidedthespaceandopportunityforlocalactorstoactivelyrespond
Conclusion:
Self-reported data suggest strong inter-project collaboration, coordination between DEPPorganisationsandunifiedresponseactivitiesdeliveredbylocalconsortiaratherthanindividualorganisations.Self-reporteddatasuggestthattheresponsewasfaster,moreefficient,moreinclusiveandinvolvedbeneficiaries,andthiswasdrivenbytheimmediateavailabilityofflexiblefundsandacontextsupportiveoflocalactors.However,thereisnowaytoobjectivelyassesshowtheresponsemighthavedifferedintheabsenceoftheDEPPinthePhilippines.
138 USAID’scSOSustainabilityIndexranksfilipinocSOshighestinAsiaacross7dimensions:legalenviron-ment,organisationalcapacity,financialviability,advocacy,serviceprovision,infrastructure,andpublicimage.
123 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
Humanitarian Response: Focus on urban emergencies in Kenya
What:
• Anurbanearlywarningsurveillancesystemininformalsettlementsinnairobicountytomonitorfoodsecurity,householdshocksandpublichealthindicatorsonaroutinebasis(everytwomonths)detectedthreeemergenciesandtriggeredresponseactivities.
1. Diarrhoealoutbreak:february2017toJune20172. choleraoutbreak:April2017tofebruary20183. foodinsecurity:OctobertoMarch2018
Who:
TheDEPP’sUrbanEarlyWarningEarlyAction(UEWEA)ProjectdevelopedandimplementedtheurbanearlywarningsystemthroughitsconsortiummembersconcernWorldwide,Oxfam,andKenyaRedcrossSociety, in seven sub-counties (Kibera,Kasarani,Ruaraka,Makadara,Starehe,KamukunjiandEmbakasiEast).ThesystemaimedtoidentifyslowonsetorchronicemergenciesinurbancontextsthroughroutinemonitoringofasetofsensitiveindicatorsthatweredevelopedalongwithspecificthresholdspriortothestartoftheprojectandtosetupacoordinatedresponsemechanismwithinthenairobicitycountryGovernment.
Objective:
TounderstandhowDEPPprojectscontributedtoresponseeffortsininformalsettlements
Data sources:
• MinimumsetIDIswithprogrammeandprojectstakeholders(Qualitative)• IntensivesetIDIs(Qualitative)• Documentreview(Qualitative)• Responsecasestudydatacollection:16IDIs,sitevisitsandobservations(Qualitativeand
Observational)Key Findings:
• DEPPcontributionstoresponsesinurbansettlementsinnairobithroughearlydetectionagainstasetofsensitiveindicatorswithclearthresholds,standoutasbeingparticularlyinnovativeintheirsound,data-drivenapproachtoidentifyingurbanemergenciesandfortheirstrongcollaborationwithgovernment.This isoneofthefewsystemsspecificallytargetingurbancontextsandfillsanimportantneed,especiallyasthepopulationsinurbansettingscontinuetogrow.Thefollowingemergencyresponseactivitiesweretriggered:
o Diarrhealoutbreak:healtheducation,massscreeningofchildrenunder5years(8,338screenedinKorogocho,13,284inEmbakasi),anddistributionofwaterpurificationproducts
o cholera outbreak: health promotion (171,139 households and 2630 studentssensitised),solidandliquidwastemanagement,andwaterqualityandfoodsafetymonitoring
o foodsecurity:Massscreeningofchildrenunder5years(44,825)andcashtransfersof2000KSH(roughly20USD)providedto3,034households(2,085women,949men)whowereseverelyfoodinsecureweremadeviamobilemoneyinKorogochoandKibera
• Thefirstresponsewastheslowest–noresponsepackageexisted,andlackofimmediatefundingsource led todelays.The responseoccurred inApril2017approximately twomonthsafterinitialdetection.
124 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
• Althoughthesurveillancesystemwasnotthefirsttoidentifythecholeraoutbreak,theuseoftheconsortiummodelenabledconsortiummemberstocollaborateinthecholeraresponseoutsideoftheirDEPPactivities.
• Datacollectedsuggestthatresponsetimeimprovedsignificantlybetweenthefirstresponsetodiarrhoeatothemostrecentresponsetofoodinsecuritywhichwasundertakenwithinafewweeksofdetection.
• factors enabling success include: conducting a needs assessment and research toinformthesystem,anddevelopingthesystemalongsidegovernmentactors toensurecoordination,buy-in,alignmentwithgovernmentprioritiesandsustainability
Conclusion:
Datacollectedsuggeststhattheconsortiummanagementmodelaswellascollaborationwithgovernmentwerecriticaltosuccessfulimplementationofthissystemandpermittedcoordinatedhumanitarian responses that improved with each subsequent emergency. Availability ofimmediateresponsefundsandpreviouslyagreeduponresponseactionswerekeytoensuringmorerapidresponses.
Humanitarian Response: Focus on drought in Ethiopia
What:
Ethiopiahasbeenaffectedbylargescaledroughtssince2015.Initiallyinducedbythe2015/2016Elnino,andfurtherexacerbatedbylow2016Autumnrains,andbelowaverage2017Springrains,atleast8.5millionpeoplewereinneedofrelieffoodassistancebymid-2017inSouthernandSouth-easternEthiopia.Theeffectsof thedroughthavebeen furtherexacerbatedbydiseaseoutbreaks,largescalelossoflivelihoodassets,anddisplacement.conflictintheOromiaandSomaliregionsbeginninginSeptember2017furthercontributedtothedisplacementofhundredsofthousandsofpeopleoftenindrought-affectedareas.
Who:
AtotaloffourDEPPprojectswereimplementedinEthiopia:ShiftingthePower,PublicHealthPreparedness in Gambella, Talent Development and Improved Early Warning Early Action–Ethiopia.
Objective:
TounderstandhowDEPPprojectscontributedtoresponseeffortstodroughtinEthiopia
Data sources:
• MinimumsetIDIswithprogrammeandprojectstakeholders(Qualitative)
• IntensivesetIDIs(Qualitative)
• Documentreview(Qualitative)
• Responsecasestudydatacollection:7IDIs
Key Findings:
Self-reporteddatasuggest6outof10localpartnersoftheShiftingthePowerproject,whichhadaimedtobuildl/nnGOcapacityandpreparedness,respondedtothedroughtintheareaswheretheymanagedevelopmentprograms.Theseincluded:communityInitiativefacilitationAssistance(cIfA),communityDevelopmentInitiative(cDI),RiftValleychildrenandWomenDevelopmentOrganisation (RVcWO), Action forDevelopment, SOS Sahel, andHUnDEE.Specificresponseactivitiesledbythesel/nnGOsin2017-2018included:
125 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
• cIfA targeted35,060households in theMoyale,MiyoandGuchiworedas to reducelivestock mortality and implemented a livestock vaccination campaign which reached263,187cattle
• cDIsecureda£10,000responsegrantfromOxfamandShiftingthePowerwhichenableddistributionofseedsto200householdsincommunitiesaffectedbythedroughtinWestArsiZone,Oromia
• RcVWOobtaineda£60,000grantfromActionAidtoprovidelivestockfeedsupporttoatargeted1,030households
• Action for Development provided £60,000 in cash transfers to drought affectedcommunitiesinborena
• SOSSahelsecured$530,000directly fromUnOcHA’sEmergencyResponsefundtoprovideanimalfeedinthreeworedas(Wachille,DubuluqandDire),indirectlybenefitting28,256individuals.This isoneoftheonly instanceswherea localEthiopiannGOhasbeenabletosecurefundsthroughthismechanismwithoutanintermediarypartner
• HUnDEE obtained $410,000 from Kindernothilfe (KnH) to provide emergency foodassistanceto10,000peoplein4KebelesoftwoworedasintheborenaZone
There was no available evidence as to whether speed, costliness or effectiveness of theresponsewereaffected.factorsthatenabledthesel/nnGOstorespondinclude:
• ShiftingthePowersetupa£50,000responsefundaccessibletol/nnGOpartnersviaapeer-reviewedapplication
• Supportwithproposalwriting,andtrainingonfundraisingstrategies includingadonormapping exercise facilitated access to direct response fundingwhich in turn enabledresponseactivities
• ThemagnitudeofthedroughtcrisisinEthiopiacreatedanopportunitytopromotelearningbydoing;thus,creatingastronglinkbetweenemergencypreparednessandresponse.forexample,l/nnGOpartnersobtainedsupportandpracticeinapplyingforresponsefundsviaShiftingthePower’sresponsefundpeerreviewprocess,andonepartnerwasabletodirectlyimplementaresponseprojectasaresult.
Conclusion:
Self-reporteddatasuggestthatexperientiallearningrelatedtofundraisingandimplementationofhumanitarianresponseactivities,alongwithaccesstoflexibleresponsefundingmechanismscontributedtosixdroughtresponsesinEthiopia.
4. Linking Emergency Preparedness Capacity to Disaster Response
Despitethecontributionstowardshumanitarianresponsesthathavebeendocumentedprimarilyvia qualitative documents and through self-reported data, it is difficult to objectively (andquantitatively)assessincreasedspeed,coverage,orefficiencyofresponseinthe42responseexamplesandaswelltotracethesebacktoincreasedpreparednessevenwiththediverserangeofdatasources.Infact,acrossthefourintensivesetcountries,self-reportedquantitativedataonthespeed,efficiency,andcostlinessofresponseactivitieswerenotdifferentbetweenDEPPandnon-DEPPcomparisonorganisationsacrossthefullsample.However,thesedatapertaintoanyresponseactivitiesintheprevious12monthsandmayormaynothaveincludedthe42responseexamplesidentifiedbytheevaluationteam.further,theevaluationteamisunabletocomparetheseindicatorsamongDEPPandnon-DEPPcomparisonorganisationsthatdirectlyrespondedinthe42emergencyresponsesdescribedaboveasthereis limitedavailabledataaboutnon-DEPPcontributiontothesesameresponses.ThiswouldhaveprovidedmoreconcreteevidenceonDEPPcontributiontoimproveddeliveryofhumanitarianresponse.nevertheless,thebestevidenceacrosstheDEPPforstrengthenedresponseisinthePhilippineswherethere
126 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
issomeself-reportedlinkbetweenimprovedcapacity,improvedpreparednessandfollowedbyearlier,moretimelyresponse.Thequantitativedatadoshowstatisticallysignificantincreasedpreparedness (using the actualmeasure) amongDEPPorganisations in the full sample, andespeciallyamonglocalorganisations.However,thiswasnotsignificantinthePhilippines,likelyduetosmallsamplesize.
AnimportantfindingfromthequalitativedataaswellasthecasestudyonEthiopiaandKenyainchapter8,highlightthatinmanycasesemergencypreparednessandresponseactivitiesweredisjointed.This isnotsurprisingasmostprojectsweredesignedasstandalonepreparednessprojectswithoutclearlyarticulatingthelinktoresponseorincorporatingactivitiesthatmightbetterenablel/nnGOstorespond.Intervieweesemphasisedtheneedtostrengthenthislinkbetweenemergencypreparednessandresponseactivities,andalsodescribedtheneedtoseekseparatefundingforresponseactivitiesasasignificantbarrier.
PROJECT VERSUS PROGRAMME OUTCOMES AND ADDED VALUE OF BEING PART OFTHE DEPP
Individual project evaluations suggest some positive outcomes, but this has not translated to global programme effects likely due to short programme period and lack of project alignment.Astheexternalevaluationfocusedonglobalprogramme-leveloutputsandoutcomesaswellascountry-levelprogrammeresultsusingpooleddatafromeachcountry,individualprojecteffectscannot be assessed. Previous evaluation reports, therefore, recommended strengthening ofthe individual projectM&E systems and the implementation of end of project evaluations.Theexternalevaluationteamprovidedadetailedevaluationchecklistaswellasguidanceonindicatorstoassessinanefforttoharmonisemeasurementacrossprojects.Allprojectshavecompletedindividualevaluations,whichhavebeenreviewedaspartofthedocumentreview,thoughthequalityandrigorofeachevaluationvaries.
Anumberofindividualprojectevaluationsreportpositiveoutcomeswhichshouldbeinterpretedobjectively,inlightofthequalityofthespecificevaluationmethodologyandapproachanditslimitations.Withthisinmind,astherewaslimitedalignmentofprojectsattheportfoliolevel,eachprojectworkedtowardsitsownobjectivesandowntheoryofchange.Itisconceivablethatanindividualprojectmighthavecontributedtopositiveoutcomechangeonspecificindicators,butthatgloballyattheprogrammelevel,cumulativechangeinthisindicatorwasnotachieved.Infact,positiveoutcomesthathavebeensuggestedbyprojectevaluationshavenottranslatedintocumulativeglobalprogrammeeffectsandthisislikelyduetothelackofprojectalignmentand/ortheshortprogrammeperiods.
Added Value of Being Part of the DEPP
Giventhattheprojectsweredesignedindependentlyandwithlimitedconsiderationtoportfoliofunctioning,andevaluationdatademonstratedlimitedcohesionasaportfolio,theaddedvalueofbeingpartoftheDEPPprogrammehasbeenlimited.ThemostbeneficialaspectofworkingundertheDEPPhasbeentheabilitytosharecontactsandknowledgeandleveragepreviouslybuiltconnectionsandnetworks.However,thedatasuggestthatthelackofcohesionacrosstheprojectportfolioledtolimitedawarenessandunderstandingoftheDEPP,andlackofasenseofbelongingorconnectiontotheDEPPatthecountrylevel.Duringin-depthinterviews,mostintervieweescouldspeakabouttheirindividualprojectsbutinmostcases,wereunabletoreflectontheDEPPasawholeanddiscussitsobjectivesandfunctioningwithintheircountry.Thislowprogrammeawarenessandidentityaffectedprogramme-leveladvocacy,stakeholderbuy-inandengagementattheprogrammelevel,coordinatedprojectimplementationandlonger-termsustainabilityoftheprogramme.
DOSE –RESPONSE ASSESSMENTAsdescribedintheinceptionphasereport,theexternalevaluationaimedtoconductadose-responseassessmentwhereinprogramme-leveloutcomescouldbecomparedacrosscountriestakingintoconsiderationthenumberofprojects,sizeofinvestmentsandhumanitariancontext.Thehypothesistobetestedwasthatcountrieswithahigherinvestment,withmoreprojectsworkingcollaborativelywoulddisplaylargereffects.Infact,thefourcountriesselectedfortheintensivesetdatacollectionwereselectedtorepresentarangeofDEPP“doses”.Unfortunately,
127 SUMMATIVEPHASEREPORT-RElEVAncEAnDEffEcTIVEnESSOfInTERVEnTIOnS
as there were very few quantitative programme-level outcome changes, a dose-responseanalysisisoflimitedvalue.ItisworthwhiletohighlightthatthePhilippines,thecountrywiththehighestlevelofDEPPprojectsandinvestment,hadthelargestcontributiontohumanitarianresponse.
NUMBER OF DEPP PROJECTS (AT THE END OF DEPP)Mozambique 1Jordan 1South Sudan 2*Myanmar 3**Ethiopia 4DRC 5Bangladesh 5Pakistan 6Kenya 6The Philippines 6*
Table4.7:numberofDEPPprojectsbycountry(notincludingthelearningProjectwhichwasimplementedinall10countries)
*CDAC Network’s project moved from South Sudan to the Philippines, thus altering the project numbers in those two countries
**Myanmar originally had 2 projects, but LPRR decided to implement activities in Myanmar raising the number of projects to 3
128 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS
cHAPTER5
129 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
Evaluation Question #3: To what extent was the programme's theory that capacity development is more effective when undertaken as a multi-agency collaborative approach proven?Sub-questions:a) Isthe‘collaborative’approachofmulti-stakeholderplatformsaneffectivedeliverymechanism?b) focusingoncoalitions,partnershipsandconnectedness–whatcanbesaidabouttheeffectsof
strengthenednetworks?c) Whathavebeenthemainpatternsofcollaboration,andthebenefitsanddisadvantagesofinformal
versusformalcollaboration?d) Whatuniquecontributiondidcollaborativerelationshipsand‘multi-stakeholderplatforms’make
towarddeepeningcross-programmelearning?Data sources: • Minimum set IDIs with programme and project stakeholders(Qualitative)• KAP, organisational and network surveys(Quantitative)• Intensive set IDIs(Qualitative)• Document review (Qualitative)Key Findings:• Despite the original vision, the DEPP focused more on consortia-based collaborative
approaches rather than multi-stakeholder platforms at the national, regional and international levels.collaborativeapproacheswereconsistentlyreportedtobethemostappropriate,preferredapproach to deliver humanitarian capacity building, but respondents identified challenges inengagement,communicationandcoordination.
• As a delivery mechanism, collaborative and consortia-based approaches were particularly effective when the following conditions were met:o Existenceofjointobjectives,valuesandcommonwaysofworkingo collaborationswereorganicnotforcedo Sufficienttime,space,resources,capacityandwilltofosterhealthycollaborativerelationshipso Streamlineddecision-makingandcontractualprocessesandfewerconsortiapartnerso buildingonexistingrelationships
• Thetopthreecollaborationareas(areaswherehumanitarianorganisationsmayworktogether)inthefourintensivedatacollectioncountries(ThePhilippines,Kenya,Ethiopia,andMyanmar)wereadvocacy,communitycapacitybuildingandprojectimplementation.
• There was some evidence of strengthened humanitarian response networks among three of the four intensive set countries (The Philippines, Kenya and Ethiopia, while the network in Myanmar remained unchanged).
• The Philippines network is highly locally driven (70% links arewith l/nnGOs); network sizeincreasedbutthenetworkbecamelessdensewithfewerconnections.The Ethiopia network is internationally driven with little input from local organisations,whilein Kenya, the network is balanced between local and international actors.Therewasevidenceofnetworkgrowthinbothsettings.TheMyanmarnetworkisverysmall,isolatedanddominatedbyinternationalactors.
• Collaboration findings were mixed.collaborationfrequency(aproxyindicatoroftrust)andqualityofrelationshipsincreasedinKenyaandthePhilippinesbutdecreasedinMyanmarandEthiopiabetweenT1andT2.
• Collaboration within the DEPP cohort increased in Myanmar and the Philippines, but both did not increase engagement with non-DEPP actors. Ethiopia and Kenya had increases in relationships between DEPP and non-DEPP organisations.
• Overall,therewasnoquantitativeevidenceofincreasedlocalisation(noincreaseinproportionofrelationshipswithlocalactorsbetweenT1andT2)inanyintensivecountrywithinthenetworkdata.Infact,theMyanmarnetworkbecameevenmoredominatedbyInGOsbyT2.Howeverqualitativedatasuggestschangesinattitudesaroundlocalisationareoccurring(seecH7)
• Not enough time passed between the data collection points to sufficiently document and test the hypothesis that strengthened networks and greater collaboration lead to better preparedness. DataatT1pickedupnetworkeffectscausedbyever-shiftingstrategiescommontofirstyearimplementation.T2likelycapturedsomedistinctiveprogrammeimpactsofDEPP,butsufficienttimetofollownetworkchangerequiresatleastanadditional12months.
• betweenprojectcollaborationcontributedtoimprovedsharingoflearningandevidencebutthisdidnotnecessarilytranslatetobehaviourchange.
130 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
OVERVIEW OF THE APPROACH TO MEASURING AND ASSESSING COLLABORATION AND MANAGEMENTARRANGEMENTS collaboration is considered to be one of the defining features of theDEPP andwas usedbyallprojects inanumberofmanners.This includedcollaborationwithinprojectconsortia,collaborationamongDEPPprojectsandcollaborationwithexternalstakeholders.Programmeoutputsrelatedtocollaborationaredetailedinchapter4inthesectiononoutputarea1.Thischapterwillfurtherexaminecollaborationatthesevariouslevelsincludingthroughstrengthenednetworks. The analysis brings together several streams of data, including qualitative andquantitativedatafromtheminimumandintensivesetcountries,aswellasdocumentreviewandthemeasurementandanalysisofhumanitarianpreparednessandresponsenetworksinthefourintensivesetcountries.
NETWORK ANALYSIS: UNDERSTANDING NETWORK STRUCTURES AND TOP COLLABORATION AREAS
AcriticalfocusofthenetworkanalyseswastoidentifypatternsofcollaborationbetweenDEPPandnon-DEPPactorsandbetweenlocalandinternationalactorsinthefourintensivecountriesandtodetectanychanges inthesecollaborationpatternsduetotheDEPP. Inaddition,thenetworkanalysisaimedtoassesstheextenttowhichstrongorganisationaltiesorrelationshipsare associatedwith higher performance. Overall, 32 collaboration areas, or potential areaswhereorganisationsmightworktogetherwithinhumanitarianpreparednessandresponse,wereassessed inorder todetermine thesizeandstructureofhumanitariannetworks inEthiopia,Kenya,MyanmarandthePhilippines.
Thenetworksurveywasadministeredintwophases.ThefirstphaseinvolveddatacollectionwithanestablishedsetofactorsineachcountrywhorepresentedDEPPprogrammepartnersandbeneficiaries,aswellasotherswhowerenotpartofDEPP(whichservedasthecomparisongroup).Theactorsidentifiedbyphaseoneparticipantsascollaboratingpartnersareconsideredtobe1st degreeactorsastheyareonedegreeofseparationawayfromthesurveyrespondent(Seefigure5.1).The1stdegreeactorsidentifiedinphaseoneweretheninvitedtoparticipateinthenetworksurvey.Theactorstheyidentifiedareconsideredas2nddegreeactors,astheyaretwodegreesofseparationfromtheinitialsurveyinformant.Thesurveywasdesignedtoincludeactorsuptothreedegreesofseparationfromtheinformantforallcountries.Giventimeconstraints,only2nddegreeactorsweresurveyed in thePhilippines forbothdatacollectionperiods.
ACTiON AiD CARE KENYA1ST DEGREE ACTOR
iSLAMiC RELiEF2ND DEGREE ACTOR
EXAMPLE OF DEGREES OF SEPARATIONIn this example, Action Aid took the first phase survey and named CARE Kenya. CARE Kenya took the survey in the
second phase and named Islamic Relief. Islamic Relif is 2 degrees of separation away from Action Aid.
figure5.1:Degreesofseparationinnetworkanalysis
131 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
All network survey participants were asked to identify the organisations they collaboratedwithovertheprevioussixmonths,andtodescribethenatureorpurposeofthecollaboration.Participantsselectedfromalistof32collaborationareasandwerealsoprovidedtheoptiontonametheirownareasofcollaboration.Thisreportconcentratesprimarilyonthefullcountrynetwork, which is the combination of all collaboration areas, but also looks at individualcollaborationareanetworkstopaintalargerpictureofcountrytrends.ThenetworkdatawereanalysedwithORA, a network analysis tool developed bycASOS atcarnegieMellon.ThestatisticalsoftwarepackageRwasusedfornetworkstatisticalsignificancetests.fullnetworkanalysisdetailsandfindingsareprovidedinthereportinAnnex12.
QUESTION 3.A: IS THE ‘COLLABORATIVE’ APPROACH OF MULTI-STAKEHOLDER PLATFORMS AN EFFECTIVE DELIVERY MECHANISM? QUESTION 3.C: WHAT HAVE BEEN THE MAIN PATTERNS OF COLLABORATION, AND THE BENEFITS ANDDISADVANTAGES OF INFORMAL VS. FORMAL COLLABORATION?
Despite the original vision, the DEPP programme focused more on consortia-based collaborative approaches rather than multi-stakeholder platforms at the national, regional and international levels. Nevertheless, collaborative approaches were consistently reported to be the most appropriate, preferred approach to deliver humanitarian capacity building and response activities, while some challenges in finding efficient processes for engagement, communication and coordination remained.
Multi-stakeholder Platforms
“Multi-stakeholderplatforms”comprisealargepartoftheoriginalDEPPprogrammetheoryofchangeandintheDEPPbusinesscaseweredescribedas“platformswithabroadmembershipe.g.,InGOs,nationalnGOs,localgovernment,nationalgovernmentacademia,privatesectorcompanies,theRedcrossRedcrescentMovement,Unagencies”whichifsuccessful,should“improvepolicyatnationalandinternationallevel”.Thebusinesscasedescribedimplementinga singleplatformper countryor regionwhere theDEPPprogramme is active aswell as aninternational platform. Possible products of the platforms that were envisioned included:synthesised lessons identified, joint evaluations and common position papers for advocacy.TheplatformswerehypothesisedtobethemechanismacrosstheDEPPthroughwhichpolicychange,evidencegenerationanddisseminationwouldoccur.
Despitethisemphasisonmulti-stakeholderplatforms,onlyafewprojectssuchasTransformingSurgecapacity andcDAc-nactually implemented this approach.TheDEPP focusedmuchmore on consortia- based collaborative models rather than on larger multi-stakeholderplatforms. Those projects that did employ platforms reported challenges in setting up andsustainingtheplatformsbutdidreportsomeorganisationalchangeasaresult.Theplatformsweremoreeffectiveincontextswhereexistinghumanitarianstructuresweremorematureandwithsupportivegovernments.ThecDAc-nfinalevaluationnotesthatthethreecountry-levelplatformsitestablishedorstrengthenedwere“relevanttotheircontextandaneffectivewayof implementing joint preparedness activities” and effective in creating new links betweenlocal,nationalandinternationalactors139.However,itwasnoted,thatastrongerlinktoglobalpolicyandmoreadvocacyeffortsshouldbeintegratedintotheseapproaches,andthatfurther,systematicinclusionoflocalandnationalactorsisneededtoensurecontextualisationandlocalownership.TransformingSurgecapacityestablishedtwonationalmulti-stakeholderplatforms,anAsiaregionalplatformandaglobalplatformforcollaborativeandlocalisedsurgemechanisms.TheTransformingSurgecapacityfinalevaluationfoundthatthenationalplatformsweremosteffectiveandthatregionalandinternationalplatformsshouldtakeamoresupportiveorenablingrolethatisdefinedbythenationalplatforms140.Itwasalsoreportedthatthenationalplatformsdidnotadequatelyreflectcountry-levelprioritiesastheyweredesignedatheadquarterslevel,andthatfutureplatformsshouldinvolveanationally-leddesignprocessandimplementation.
Consortia-based Collaborative Approaches
All14DEPPprojectswereimplementedbyconsortiaoforganisationsworkingtogether,withawide rangeof typesand sizesof consortiawithin theDEPP.DatacollectedamongDEPPorganisationsandkeystakeholderssuggeststhattheconsortiaapproachwasappropriateinthe
139 cDAc-nfinalEvaluationReport,p4.140 TransformingSurgecapacityfinalEvaluationReport,p8.
132 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DEPPfocalcountrycontexts,andapreferredapproachtodeliverhumanitariancapacitybuildingactivities.Self-reporteddatafromconsortiamemberssuggestthatoverallconsortiafunctionedrelativelywell (seefigure5.1below),despite anumberof identifiedchallenges including inengagement,communicationandcoordination.Thesedatasuggestthatconsortiafunctionalityalsoimprovedbetweenthefirst(T1)andsecond(T2)datacollectionpoints.Therewerealsoseveral exampleswhere the consortia approach facilitated the delivery of capacity buildingactivitiesandalsoresponseefforts.forexample,thecasestudyinchapter8highlightshowtheconsortiumstructureoftheUrbanEarlyWarningEarlyActionprojectinKenya,includingtheestablishedproceduresandmanagementstructure,facilitatedatimelyresponsetoacholeraoutbreak.ThenextsectionwillfocusonexaminingthedegreetowhichcollaborationoccurredacrosstheDEPP,themainpatternsofcollaborationandthereportedchallengesandbenefitsofinformalandformalcollaboration.
0
10
20
30
40
50
60
70
80
T2T1T2T1T2T1T2T1T2T1T2T1
NOT WELL AT ALL
VERY WELL
WELL
SOMEWHAT WELL
NOT WELL
DEPP CONSORTIA FUNCTIONALITY
MiN SETKENYATHE PHiLiPPiNESETHiOPiAMYANMARFULL SAMPLE
figure5.2:DEPPconsortiafunctionalityattimepoint1(T1)andtimepoint2(T2)
Is Collaboration Occurring and What Types of Organisations are Involved?
Datafromthenetworkanalysis,in-depthintervieweesandintensivesetsurveyssuggestthatorganisationsinboththeDEPPandcomparisongroupscollaboratedwithoneanotherduringcapacitybuildingandhumanitarianpreparednessand responseactivities.furthermore,bothDEPPandcontrol grouporganisationsand staffstronglyvalued theapproachofworking inconsortiaandthroughnetworks.forexample,approximately94%ofrespondentsinbothDEPPandcontrolgroupintensivesetsurveysstatedthattheywouldapproachotherorganisationstocollaborateduringhumanitarianresponseactivities.SeveralchangesincollaborationpatternsbetweenT1andT2havebeenidentifiedfromthenetworkanalysisincluding:
• Increasedcollaborationfrequency(aproxyindicatoroftrustandqualityofrelationships)141 inKenyaandthePhilippines,whilebothindicatorsdecreasedinMyanmarandEthiopia
• Increasedcollaborationwithin theDEPPcohort inMyanmarand thePhilippines, butdifficultyengagingwithnon-DEPPactorsinbothsettings
• IncreasedcollaborationbetweenDEPPandnon-DEPPactorsinEthiopiaandKenya
Thesechangeswillbeexploredmorefullyinthesectiononstrengthenednetworkslaterinthischapter.
141 Whenanactorindicatesfrequentcollaboration,weassumethereishightrustandperceivedvalueincon-tinuedengagement.
133 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
What Have Been the Benefits and Disadvantages of Formal Versus Informal Collaboration?
both formal collaborations and informal collaborations occurredwithin DEPP projects, andinformalcollaborationsoccurredacrossDEPPprojectsandwithexternalstakeholders.formalcollaborationwasdefinedasanycollaborationinvolvingasignedcontractualagreementdefiningrolesandresponsibilitiesofcollaboratingentities.Informalcollaborationsweredefinedasthosecollaborations that occured more spontaneously and without formal agreements in place.Acrossallcountries,humanitarianstafffromDEPPorganisationsidentifiedtheadvantagesanddisadvantagesofformalandinformalcollaborations(seeTable5.1)
FORMAL COLLABORATION
Advantages % Disadvantages %
Sharingofresources 52.4Difficulttocollaborateeffectivelywhenorganisationshavedifferentpoliciesandprocesses
29.8
Improvednetworking 51.2 Slowandcumbersomedecision-makingprocesses 28.6
Exchangeofideas 48.4 Settingupaformalcollaborationisex-tremelytimeconsuming 20.2
Abilitytoaccessothersourcesoffund-ingwhenapplyingasaconsortium 40.5 Difficulttomanagedisagreements 20.2
Programmedeliveryanddesignmoreeffective 40.5 Unclearobjectives 13.1
INFORMAL COLLABORATION
Advantages % Disadvantages %
Exchangeofideas 48.8Difficulttocollaborateeffectivelywhenorganisationshavedifferentpoliciesandprocesses
28.8
Improvednetworking 46.4 Difficulttomanagedisagreements 18.9
Sharingofresources 41.7 Slowandcumbersomedecision-makingprocess 18.9
Informalcollaborationsleadtoformalcollaborations 25.0 Unclearobjectives 16.4
learningfromthecollaboration 21.4 Maytakealotoftimewithoutyieldinganyconcreteoutputs 9.4
Table5.1:AdvantagesanddisadvantagesofformalandinformalcollaborationRespondentscouldselectmorethanoneadvantageordisadvantageforeachtypeofcollaboration
Despite the identifiedchallenges, formal collaborationswerehighlyvalued,withadvantagessuchasnetworking,exchangeofideasandincreasedeffectivenessofprogrammedesignanddelivery. Ingeneral,many respondentshighlighted theutilityof informal collaborations, andemphasisedthatinformalcollaborationscanstimulatenewideasandprojects,whichultimatelycanturnintoformalcollaborations.PatternsofcollaborationbetweenlocalandinternationalactorsandbetweenDEPPandnon-DEPPactorswillbefurtherexploredinQuestion3.busingdatafromthenetworkanalysis.
As a delivery mechanism, collaborative and consortia-based approaches were particularlyeffectivewhenthefollowingconditionsweremet:
• Existenceofjointobjectives,valuesandcommonwaysofworking• collaborationswereorganicnotforced• Sufficient time, space, resources, capacity and will to foster healthy collaborative
134 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
relationships• Streamlined decision-making and contractual processes as well as fewer consortia
partners• buildingonexistingrelationships
QUESTION 3.B: FOCUSING ON COALITIONS, PARTNERSHIPS AND CONNECTEDNESS – WHAT CAN BE SAID ABOUT THE EFFECTS OF STRENGTHENED NETWORKS?
There was some evidence of strengthened humanitarian response networks in the Philippines, Kenya and Ethiopia, while the network in Myanmar remained unchanged. Not enough time passed between the data collection points in these four countries to sufficiently document and test the hypothesis that strengthened networks and greater collaboration lead to better preparedness.
ThissectionaimstoexaminewhethernetworksinfourDEPPoperationalcountrieshavebeenstrengthened,andanyeffectsofstrengthenednetworksinthesesettingsusingnetworkdatacollectedattwotimepointsinKenya,Ethiopia,MyanmarandthePhilippines.
Collaboration Areas
looking across all four countries, advocacy, community capacity building, and projectimplementationwereconsistentlyamongthemostactivecollaborationareasduringbothdatacollectionperiods(seeTables5.2and5.3).142
THE PHILIPPINES KENYA MYANMAR ETHIOPIA
Collaboration
Areas# of
LinksCollaboration
Areas# of
LinksCollaboration
Areas# of
LinksCollaboration
Areas# of
Links
1Advocacy 418 Advocacy 179
Communi-ty Capacity Building
29 Project Imple-mentation 207
2
Communi-ty Capacity Building
341Communi-ty Capacity Building
151 funding 20Communi-ty Capacity Building
181
3Project Imple-mentation 283 Project Imple-
mentation 123 Advocacy 14 ProjectDesign 163
4communityconnections 270 facilitation 118 Project Imple-
mentation 14 funding 153
5
communityPlanning 255 community
Planning 101 InformationSharing 11
communi-ty-basedRiskAnalysis
147
6 facilitation 249 ProjectDesign 96 WaSH 10 WaSH 139
7
communi-ty-basedRiskAnalysis
234 communityconnections 95 Education 10 ProposalWrit-
ing 131
8Dataresources 223
conflictMiti-gationExper-tise
94 communityPlanning 9
climatechangeandAdaptation
110
9Education 220
climatechangeandAdaptation
90EarlyWarningSystemsEx-pertise
7EarlyWarningSystemsEx-pertise
106
10
climatechangeandAdaptation
208communi-ty-basedRiskAnalysis
90climatechangeandAdaptation
7 facilitation 91
142 notethatforEthiopia,advocacywasranked18thinthefirstdatacollectionand13thintheseconddatacollection.
135 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
Table5.2:Top10collaborationareasbycountryatfirsttimepoint(T1)
THE PHILIPPINES KENYA MYANMAR ETHIOPIA
Collaboration
Areas# of
LinksCollaboration
Areas# of
LinksCollaboration
Areas# of
LinksCollaboration
Areas# of
Links
1 Advocacy 271 Community Ca-pacity Building 612 Project Imple-
mentation 26 Project Imple-mentation 298
2Project Imple-mentation 202 Advocacy 512 Community Ca-
pacity Building 25 ProjectDesign 255
3
Community Capacity Build-ing
194EarlyWarningSystemsExper-tise
409 Advocacy 17 ProposalWrit-ing 237
4communityPlanning 121 community
connections 396 communityPlanning 15 funding 229
5
communityconnections 108
communi-ty-basedRiskAnalysis
387 funding 14Communi-ty Capacity Building
213
6 funding 105 facilitation 372 ProjectDesign 12 WaSH 205
7facilitation 101 community
Planning 349EarlyWarningSystemsExper-tise
11EarlyWarningSystemsEx-pertise
198
8
DataResourcesincludingdatasets,collectionand analysis
98 ProjectImple-mentation 345 climatechange
andAdaptation 9 facilitation 167
9
VulnerableGroups 97 conflictMitiga-
tionExpertise 311communi-ty-basedRiskAnalysis
9 VulnerableGroups 155
ProjectDesign 96 climatechangeandAdaptation 284 ProposalWrit-
ing 8communi-ty-basedRiskAnalysis
151
Table5.3:Top10collaborationareasbycountryatsecondtimepoint(T2)
In the Philippines, the top three collaboration areas (advocacy, project implementation andcommunity capacity building) remained consistent at both data collection points, thoughtheorderdiffered.Anotable change in this countrywas the jump innumberofties in thefundingnetwork,anditsjumpfromrank11torank6betweenthetwotimeperiods.Thetopcollaborationareasinthiscountrywerealsothemostdistributednetworks143.
InKenya,twoofthetopthreecollaborationareasremainedthesame–communitycapacitybuildingandadvocacy.EarlywarningsystemsexpertisewasthethirdmostpopularcollaborationareaatT2(upfromrank11inT1).InMyanmar,thefundingnetworkdroppeddownthelist,andthecommunityplanningnetworkmovedupthelist.Otherwise,thetopfourcollaborationareasinthisnetworkremainedthesamebutwithadifferentorder.InEthiopia,projectimplementationandprojectdesignwereinthetopthreeatbothT1andT2,butproposalwritingjumpedfromrank7atT1torank3atT2.Topcollaborationareaswerealsothemostdistributednetworks.
143 A distributed networkisonewithmanylinksbetweenmultipleactorsinthenetwork,allowingformoreequalflowofinformationtoallactorswithinthenetwork.
136 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Part 1: Have the Networks in the DEPP Intensive set Countries Been Strengthened?
The DEPP theory of change is built around the premise that strengthened networks andincreasedcollaborationarecriticaltoimprovingpreparednessandresponseandshiftingtoamorelocalisedapproach.Thenetworkanalysisenabledassessmentofchangesinthehumanitarianpreparednessandresponsenetworksovertimeandprovidesevidenceofstrengthandqualityofcollaborationsbetweenvariousinstitutions.Specifically,thissectionwillexaminechangesinthenumberoflinksbetweenorganisations,thefrequencyofcollaborationaswellasthequalityofcollaborativerelationships.
A. Organisational Links
The charts below summarise the relationship data thatwas collected at both timeperiods.Theaveragenumberofcollaborationareasperuniquelinkcaptureshowmanycollaborationareasonaveragebetweentwoorganisations.Theaveragenumberoftotallinkspersurveyedorganisationrepresentsthecontributionofeachorganisationtothefullnetwork.finally,theaveragenumberofuniquelinkspersurveyedorganisationrepresentstheaveragenumberoforganisationstowhomthatorganisationistied.Overall,theDEPPcohortacrossallcountriesrepresentedonlyasmallproportionoftheentirehumanitariannetworksmappedaspartofthisanalysis(between9and24%).
Country Time Point
ORGANISATIONS LINKS
Tota
l #
Surv
eyed
Tota
l #
Iden
tified
Tota
l # o
f DEP
P O
rgan
isatio
ns
Iden
tified
Tota
l # o
f Lin
ks
# of
Uni
que
Link
s
Aver
age
# of
Co
llabo
ratio
n A
reas
per
Lin
k
Aver
age
# of
To
tal L
inks
per
O
rgan
isatio
n
Aver
age
# of
U
niqu
e Lin
ks p
er
Org
anisa
tion
The Philippines
T1 72 401 36(9%) 5622 679 8.3 78.3 9.4
T2 92 254 33(13%) 2754 489 5.6 29.9 5.3
KenyaT1 73 161 30
(19%) 2109 315 6.7 28.9 4.3
T2 142 317 28(9%) 8027 807 9.9 56.5 5.7
MyanmarT1 33 75 17
(23%) 207 103 2 6.3 3.1
T2 30 55 13(24%) 226 76 2.3 7.5 2.5
EthiopiaT1 63 132 22
(17%) 2438 329 7.4 38.7 5.2
T2 59 137 26 (19%) 4150 434 9.6 70.3 7.4
Table5.4:Organisationsandlinksbycountry
InthePhilippines,therewasanincreaseinboththenumberoforganisationssurveyedandthenumberofuniqueorganisationsonthemap,thusoverallalargernetworksize.However,therewasasubstantialreductioninthenumberofrelationshipsbetweenorganisations(theaveragenumber of links per organisation decreased from9.4 to 5.3),which suggests a decrease innetworkdensity.However,thisfindingislikelyduetobiasasaresultofthelearningeffectandsurveyfatigue.DuringthePhilippinesresultsvalidationworkshop,respondentsindicatedthatbasedontheirexperiencewiththefirstroundofnetworkdatacollection,theywereawarethatinterviewwouldtakelongeriftheyidentifiedmorecollaboratingorganisations,thustheyonlylistedmaincollaborators.
137 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
InKenya,thenumberoflinkswithinthenetworkalmostquadrupledsignifyingalargegrowthduring the time period. In addition to twice as many surveyed organisations at T2, eachorganisation also reported relationships that spanned across more collaboration areas andmoreorganisations.ValidationworkshopparticipantsinKenyasuggestedtheyhadintensifiedcollaboration efforts, which might explain the observed network growth. The Myanmarcountrynetworkremainedthesameintermsofoverallsize,thoughitisimportanttohighlightit is substantially smaller than the three other country networks. In addition, in Myanmar,organisations reported relationships with fewer organisations at T2, though relationshipsspannedmorecollaborationareassignifyingincreasednetworkdensity.
ThedatafromEthiopiashowedaslightdecreaseinthenumberoforganisationscompletingthesurveybutasmallincreaseinnumberoforganisationsinthemap.Inaddition,therewerealsoalmosttwiceasmanylinksorconnectionsbetweenorganisationsatT2–someoftheseweredueto relationshipswithmoreorganisations,butsomeweredueto increasedcollaborationacrossmorecollaborationareaswiththesameorganisations.
B. Quality of Collaboration
Twoindicatorswereusedtoassessthequalityofrelationshipswithinthenetwork.Thefirstwasfrequencyofcollaborationwitheachactoridentified(Seetable5.5)andthesecondwaslikelihoodofrecommendingthatorganisationtoothersonascaleofoneto10(alsoknownasnetpromoterscale) (See table5.6).frequentcollaboration (more thansixtimesover theprevioussixmonths)suggestsahighleveloftrustandperceivedvalueincontinuedengagement.Whenanactorindicatesahighlikelihood(scoreof9to10)ofrecommendingtheotheractor,itsuggestshightrustandreverence.Sincethismeasurecapturesbothperceptionsandactions,thismetricbetterassessestruefeelingsoneactorhasofanother.Thenetpromoterscale isadjustedforbiasbycategorisingonlythehighestscoresaslikelytotakeaction;theothersarelikelytobepassiveordetractors.144
Collaboration Frequency over Last 6
Months
THE PHILIPPINES* KENYA ETHIOPIA MYANMAR*
T1 (N=5622)
T2 (N=2754)
T1 (N=2109)
T2 (N=8027)
T1 (N=2438)
T2 (N=4150)
T1 (N=207)
T2 (N=226)
Collaborate Rarely (1-2 times)
558 (10%)
198 (7%)
95 (5%)
100 (1%)
91 (4%)
297 (7%)
18 (9%)
26 (12%)
Collaborate Often (3-4 times)
1371(24%)
449(16%)
596(28%)
889 (11%)
484(20%)
815(20%)
43 (21%)
52 (23%)
Collaborate Frequently (>6 times)
3693(66%)
2107(77%)
1418(67%)
7038(88%)
1863(76%)
3038(73%)
146(70%)
148(65%)
Table5.5:frequencyofcollaborationwithinthepast6monthsbycountry*2X2 Chi-square Test indicates significant difference between T1 and T2 at the p<0.05 level
Inallcountriesacrossbothtimeperiods, theactors interactedwithoneanother frequently.InMyanmarandEthiopiafrequencyofcollaborationdecreasedbetweenthetwotimepoints.Kenyasawthelargestincreaseinfrequencyofinteractionwithanincreaseof21%oftiesthatreportedfrequentcollaboration.fortraceablerelationships,therewasastatisticallysignificantchangeinthefrequencyofinteractionbetweenorganisationsinthePhilippines(p=0.009)andinMyanmar (p=0.028). However, the difference in the change for DEPP versus non-DEPPcomparisongroupswasnot significant in either country.This suggests thatbothDEPPandnon-DEPPorganisations’levelsofcollaborationchangedinsimilarwaysovertime.
144 Thescaleisbasedonbain’snetPromoterscale.furtherdetailsareavailablehere:http://www.netpromot-ersystem.com/about/measuring-your-net-promoter-score.aspx
138 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Likely to Recommend
Partner (scale of 1-10)
THE PHILIPPINES* KENYA* + ETHIOPIA* + MYANMAR
T1 (N=5622)
T2 (N=2754)
T1 (N=2109)
T2 (N=8027)
T1 (N=2438)
T2 (N=4150)
T1 (N=207)
T2 (N=226)
Not Likely (Score 1-6)
812 (14%)
258 (9%)
310(15%)
735 (9%)
266 (11%)
503(12%)
91 (44%)
125(55%)
Somewhat Likely (Score 7-8)
2189(39%)
1028(37%)
814(39%)
2233(28%)
902(37%)
1555(37%)
74 (36%)
63 (28%)
Very Likely (Score 9-10)
2621 (47%)
1468(53%)
984(47%)
5059(63%)
1270(52%)
2092(50%)
41 (20%)
38 (17%)
Table5.6:likelihoodtorecommendorganisationstoothers(netpromoterscale)bycountry*2X2 Chi-square Test indicates significant difference between T1 and T2 at the p<0.05 level + Statistically significant difference (p<0.05) between DEPP and non-DEPP organisations in this country
Myanmarwastheonlycountrytoshifttowardsbeinglesslikelytorecommendothersbetweentimeperiodoneandtwo.Overhalfofitsrelationshipswerereportedasnotlikelytorecommendtheotheractorbythesecondtimepoint.Inallothercountries,overhalfoftheirrelationshipswerereportedasverylikelytorecommendtheotheractor.Aswithfrequencyofinteraction,Kenyasawthelargestincreaseinlikelihoodtorecommendotherswithanincreaseof16%oflinksreportedasverylikelytorecommend.ThisagainsignifiesstrengthenedrelationshipsandislikelyduetotheincreasedcollaborationandrelationshipbuildingconductedbyKenyaDEPPprojects.
forallcountriesexceptMyanmar,theincreaseinlikelihoodtorecommendotherswasstatisticallysignificant.145InbothKenyaandEthiopia,thechangeinlikelihoodtorecommendotherswithintheDEPPcohortwassignificantlyhigherthanthechangeinthenon-DEPPcohort,providingevidencethatDEPPhascontributedtoincreasedqualityofrelationshipsinthesesettings.
Taken together, thesedata show thatnetworks inEthiopia,Kenyaand thePhilippineshavebecomestrengthenedandingeneralqualityofrelationshipshasimprovedacrossthenetworks.ThereisevidenceinKenyaandEthiopiathatDEPPhascontributedtostrengthenedrelationshipquality.
Part 2: Is there Evidence of Improved Localisation?Overall, there was no quantitative evidence of increased localisation (no increase in proportion of relationships with local actors between T1 and T2) in any intensive country within the network data. In fact, the Myanmar network became even more dominated by INGOs by T2. However qualitative data suggests changes in attitudes around localisation are occurring (see CH 7).
A. Patterns of Collaboration Between National and Local NGOs
The DEPP has emphasised localisation and promoted a more locally-driven approach toemergencypreparednessandresponse.Theevaluationassessedpatternsofcollaborationwithnational and localorganisations, andwhether thosepatterns changedovertime.figure5.3presentsdatafromtheintensivesetdatacollectiononself-reportedchangeinthenumberofpartnershipswith localornationalorganisations.As illustrated, approximately70%ofDEPPorganisationsreportedanincreaseinpartnershipswithl/nnGOsatT1comparedwith65%atT2,representingasmall,non-significantdecrease.Onlyabout50%ofnon-DEPPcomparisonorganisations reported an increase in partnershipswith l/nnGOs atT1 but this increasedto59%atT2.Thesedata suggest no significant self-reported increase in partnershipswithl/nnGOs in either group. However, as discussed earlier, perceptions may be biased andpotentially inaccurate.Aspartof thenetworkanalysis,partnershipsandcollaborationsweredirectlymeasuredtomoreobjectivelyassessthenumberofrelationshipswithl/nnGOsandwhetherthesehavesignificantlychangedovertimeineachofthefourintensivesetcountries.
145 ThePhilippines:p<0.001;Kenya:p=0.047;Ethiopia:p=0.004.
7
139 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
figure5.3:ReportedchangeinnumberofpartnershipswithlocalandnationalnGOs
figures5.4and5.5comparetheoverallnetworkstructuresinthePhilippines,Kenya,EthiopiaandMyanmaratT1andatT2.Inthesefigures,eachorganisationisrepresentedbyanode(diamondsrepresentDEPPorganisations,andcirclesrepresentnon-DEPPorganisations),whileactivecollaborations146betweenorganisationsarerepresentedbythelinesconnectingthenodes(orlinks).Arrowsrepresentthedirectionofthecollaboration.Thesizeofthenodeisakeyelementofthesefiguresasitrepresentsthetotaldegreecentrality,orthenumberoflinksonenodehastoothernodes,whichisameasureofinfluence.Themorelinksorconnectionsanorganisationhas,thelargerthesizeofthenodeinthenetwork.Given the DEPP theory of change, a distributed network where there is greater cohesion and links between multiple actors in the network, more equal flow of information to all actors in the network, and minimal bottlenecks, promotes sustainability and is more desirable than a centralised and isolated network.Whenweexaminedallthenetworksinallcollaborationareas,inallintensivesetcountriesexceptMyanmar,thetopthreecollaborationareaslistedinTable5.2and5.3werethemostdistributednetworks.Table5.7complementsthecountrynetworkvisualisationsbypresentingthepercentageofrelationshipsortiesthattarget147l/nnGOsfromvarioussourceorganisations(InGO,l/nnGOorall).Tiestargetinglocalactorsrepresentlocalisationofthehumanitarianresponsewithinthecountry.Iflocalisationincreasedbetweenthetwotimepoints,weexpecttheshareoflinkstol/nnGOsfromanysourcetoincrease.
146 Activecollaborationsarethosethathaveinvolvedinteractionwithinthelast6months.147 Thesourcegroupisthegroupfromwhichthetieiscoming,andthetargetgroupisthegrouptowhichthe
tieisgoing.
0
10
20
30
40
50
60
70
80
MORE
THE SAME
FEWERPARTNERSHIPS WITH L/NNGOS
DEPP
T2T1T2T1
COMPARiSON
7.1
23.8
69.1
8.8
26.4
64.8
5.0
45.050.0
11.1
30.2
58.7
140 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
THE PHILIPPINES T1 THE PHILIPPINES T2
KENYA T1 KENYA T2
figure 5.4: network visualisations for the Philippines andKenyaattimepoint1andtimepoint2
141 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
MYANMAR T1 MYANMAR T2
ETHIOPIA T1 ETHIOPIA T2
figure5.5:networkvisualisationsforEthiopiaandMyanmarattimepoint1andtimepoint2
142 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Data suggest that networks in the Philippines were highly locally driven, with more than 70% of links occurring with L/NNGOs (Figure 5.4 and Table 5.7). Thisisconsistentwithothersourcesthatsuggestthatcivilsocietyisdeeplyentrenchedinthiscontextandconsideredtobeoneof thestrongest inAsia148.However, therewasnochange intheproportionof linksto l/nnGOs betweenT1 andT2, suggesting no change in the level of localisation. In the Kenyan humanitarian preparedness and response network, local actors were active, but the relationships evenly targeted international and local actors(figure5.4andTable5.7).TherewasnochangeinthisdistributionbetweenT1andT2suggestingnoincreaseinprominenceoflocalactorswithinthenetwork.In Ethiopia as well, there was no change over time(figure5.5andTable5.7).However,unlikethePhilippinesandKenya,the Ethiopian network has high dominance of the INGOs with little input from local organisations. finally, thenetwork in Myanmar was substantially different from the other country networks. It was significantly smaller, and highly isolated (figure 5.5 andTable 5.7).This networkmoved fromone thathadanevensplitofrelationshipsbetweeninternationalandlocalactors,toonedominatedbyInGOs.ThismaybeduetothepoliticalsituationinMyanmar.DuringthevalidationworkshopinMyanmar, informedstakeholdersreportedthat thegovernment increasedrestrictionsandthishadaffectednGOregistration.Inaddition,localorganisationsmightbecoordinatingandcollaboratingmoreduringtimesofdisasterandresponsebutnotinbetween.
TARGET ORGANISATION: L/NNGO
Source Organisation
THE PHILIPPINES KENYA* ETHIOPIA* MYANMAR*
T1 T2 T1 T2 T1 T2 T1 T2INGO 1056
(67%)580(65%)
422(50%)
1658(47%)
411(27%)
837(29%)
41 (39%)
72 (47%)
L/NNGO 2893(70%)
1259(68%)
659(52%)
2144(48%)
202(21%)
213(17%)
39 (39%)
12 (17%)
All Organisations
3949(70%)
1839(67%)
1081(51%)
3802(47%)
613(25%)
1050(25%)
80 (39%)
84 (37%)
Table5.7:numberandpercentagesoflinkstargetingl/nnGOsfromvarioustypesofsourceorganisationsNote: The source group is the group from which the tie is coming, and the target group is the group to which the tie is going.*2X 2 Chi-square Test indicates significant at the p<0.01 level
B. Patterns of Collaboration Between DEPP and non-DEPP Cohorts
InordertounderstandcollaborationwithinoracrossDEPPandnon-DEPPgroups,theanalysisalsoexaminedthetiesorlinksthattargetoraredirectedtowardsDEPPgroupsfromvarioussources (see Table 5.8). In the Philippines, collaborationwithin the DEPP cohort increased(DEPPtoDEPPconnectionsincreasedfrom17%atT1to29%atT2).However,overallwhilebothDEPPandnon-DEPPgroupsmostlytargetednon-DEPPorganisationsatbothtimes,itwasanongoingchallengetogetthenon-DEPPcohortinvolvedandreachingouttoDEPPactors.non-DEPPactorswerenomorelikelytoreachouttoDEPPorganisationsinT2thantheywereinT1(non-DEPPtoDEPPtiesatT2was20%).
148 USAID’s CSO Sustainability Index ranks CSOs in the Philippines as the highest across 7 dimensions: legal environment, organisational capacity, financial viability, advocacy, service provision, infrastructure, and public image.
143 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
TARGET ORGANISATION: DEPP
Source Organisation
THE PHILIPPINES* KENYA ETHIOPIA* MYANMAR*
T1 T2 T1 T2 T1 T2 T1 T2
DEPP 334(17%)
379(29%)
460(59%)
1773(51%)
548(48%)
709 (41%)
42(40%)
78(75%)
Non-DEPP 562(15%)
287(20%)
470(35%)
1620(36%)
368(28%)
856 (35%)
19(19%)
8 (7%)
All Organisations
5622(16%)
666 (27%)
930(44%)
3802(47%)
916(38%)
1565(38%)
61 (30%)
86 (38%)
Table5.8:numberandpercentagesoflinkstargetingDEPPorganisationsfromvarioustypesofsourceorganisationsNote: The source group is the group from which the tie is coming, and the target group is the group to which the tie is going.*2X 2 Chi-square Test indicates significant at the p<0.01 level
Kenya’sfullcountrynetworkwasmadeupofmanyrelationshipsfromnon-DEPPactors,thoughthisdeclinedattheseconddatacollectionpoint.AtT2,theabsolutenumberofrelationshipsfromDEPPactorswasapproximately3.9timeshigherthanatT1,whereasabsolutenumberofrelationshipsfromnon-DEPPactorsatT2wasabout3.5timesthatofT1.Theselargeincreasesagain highlight the substantial overall network growth that occurred in Kenya. In terms ofcollaboration in Kenya, a slight decline in the percentage of relationships between DEPPorganisationsoccurredfromT1toT2 (from59%to51%).Also, therewasnochange inthepercentageofnon-DEPPtoDEPPlinks.However,thesedatashowhighlevelsofcollaborationbetweennon-DEPPandDEPPactors,with about35%ofnon-DEPP relationships targetingDEPPactors.KenyaisonesettingwheretheDEPPcohortseemstohavebeenabletoformrelationshipswithactorsoutsideofitsgroup,despiteasmallcohortsizerelativetotheotheractorsinthenetwork.
InMyanmar,ontheotherhand,theDEPPcohortwasnotverysuccessful inreachingthoseactors outside of the DEPP programme. In fact, non-DEPP actors actually decreased theircollaborationwithDEPPactorsfrom19%to9%.collaborationbetweenDEPPorganisationsincreasedby35%fromT1toT2with40%ofDEPPlinkstargetingotherDEPPorganisationsatT1to75%atT2.“Exclusivity,”oronlyengagingwithone’sowngroup,seemstobealargefactorinthiscountry,thoughithastheleastdisparitybetweengroupsizesofanycountrywithaboutone-quarterofallactorsinthenetworkintheDEPPcohort.
In Ethiopia, a fairly even split of network relationships came fromDEPP versus non-DEPPactors.Thenon-DEPPgroupbecameslightlylessexclusiveafterthesecondtimepoint,asitformedmorerelationshipswiththeDEPPgroup.comparedtoothercountries,thiscountry’sDEPPcohortwasfairlysuccessful(alongwithKenya)informingrelationshipswithnon-DEPPactors,as35%ofnon-DEPPtiestargetedtheDEPPgroup.
C. Influence of DEPP Actors Within the Networks
Table5.9 presents the top50organisations in each country assessed through total degreecentrality,ametricofthetotalnumberoftiesofeachorganisation.notethatthismetricdoesnottakeintoaccountthedirectionorstrengthofindividualties,ratheronlythetotalnumber.Thetableshowstheproportionofthetop50organisationsthatbelongtotheDEPPcohort,aswell as the share that are national/local compared to international. for example, in thePhilippinesatT1,14ofthetop50organisationsareDEPP(28%),whiletheremaining36(72%)werenon-DEPPorganisationsand35ofthetop50 (70%)werenational/localorganisationscomparedtotheremaining15(30%)whichwereinternational.
144 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
T1 T2
Country National / Local International All
OrganisationsNational /
Local International All Organisations
THE PHILIPPINES
DEPP 9(18%) 5(10%) 14(28%) 14(28%) 6(12%) 20(40%)
non-DEPP 26(52%) 10(20%) 36(72%) 22(44%) 8(16%) 30(60%)All Organisations 35(70%) 15(30%) 50 36(72%) 14(28%) 50
KENYA DEPP 8(16%) 11(22%) 14(28%) 6(12%) 13(26%) 19(38%)non-DEPP 12(24%) 19(38%) 36(72%) 19(38%) 12(24%) 31(62%)All Organisa-tions 20(40%) 30(60%) 50 25(50%) 25(50%) 50
ETHIOPIA DEPP 6(12%) 13(26%) 19(38%) 9(18%) 10(20%) 19(38%)non-DEPP 13(26%) 18(36%) 31(62%) 10(20%) 21(42%) 31(62%)All Organisations 19(38%) 31(62%) 50 19(38%) 31(62%) 50
MYANMAR DEPP 8(16%) 8(16%) 16(32%) 7(14%) 6(12%) 13(26%)non-DEPP 16(32%) 18(36%) 34(68%) 13(26%) 24(48%) 37(74%)All Organisations 24(48%) 26(52%) 50 20(40%) 30(60%) 50%
Table5.9:Top50influentialorganisationsbyDEPPversusnon-DEPPandinternationalversusnational/localNote: Influence is measured by total degree centrality (the total number of ties of the organisation). Statistical tests to determine statistically significant change between DEPP versus non-DEPP and International versus national/local were conducted but no significant differences were found at the p<.0.05 level.
comparingT1toT2inthePhilippinesshowsthatthetop50organisationsremainedlargelylocal(around70%ofactorsatbothtimepoints),buttherewasa12-percentagepointincreaseintheproportionoftopactorsthatwerefromtheDEPPcohort(from28%to40%).Inaddition,a 10- percentage point increase in the proportion of the top 50 organisations that werespecificallyDEPPnationalorlocalorganisationisevident(from18%to28%).Thischangewasstatisticallysignificant(p=0.046).InKenya,theproportionofnationalorlocalorganisationsinthetop50increasedfrom40%to50%byT2,representinganevensplitbetweeninternationalandnational/local.TherewasnochangeinthepercentageofDEPPorganisationsinthetop50organisationsovertime,oroverallinthetopactorsbetweenT1andT2(p=0.066).WhiletheoverallpercentageofDEPPorganisationswithinthetop50inEthiopiadidnotchange,thedistributionofnationalandinternationalDEPPorganisationsshifted.AtT112%ofthetop50wereDEPPnationalandthisincreasedto18%byT2.Therewassignificant(p=0.016forT1,p=0.005forT2)prominenceofinternationalnon-DEPPactorsinthetop50organisationsforbothtimeperiods.InMyanmar,therewasadecreasefrom32%to26%inthepercentageofDEPPactorsinthetop50.Theprominenceofnon-DEPPinternationalorganisationsisevident,asalmosthalfofthetop50actorsfallintothiscategory.
145 SUMMATIVEPHASEREPORT-EffEcTIVEnESSOfMAnAGEMEnTARRAnGEMEnTS
D. Do Strengthened Networks and Greater Collaboration Lead to Improved Preparedness?
Toexaminethequestionofwhetherstrengthenednetworksandgreatercollaborationleadtoimprovedpreparedness,theanalysesassessedchangeinnetworkingtogetherwithorganisationalpreparednessovertime.firsta‘networkingscore’forDEPPandnon-DEPPcohortswascreated,combiningin-degree,out-degree,betweennessandeigenvectorcentralitymeasures.TheT1andT2averagenetworkingscoreswereassessedusingapairedt-testtoanalysewhethertherehadbeenanysignificantchangeinacrosstimeperiods.ThechangeinnetworkingscoreswerealsocomparedbetweenDEPPandnon-DEPPgroupsusingt-testsforindependentsamples.next,correlationsbetween thenetworking score and the aggregatedorganisational preparednessscore149acrosssevendomains(seechapter4)wereassessed.Thiswascomparedwithadisasterpreparednessscoreforbothgroups.TheonlycountrywithanysignificantchangeinnetworkingscorewasEthiopia,wheretherewasasignificantdecrease(p=0.015).nocountrywasfoundtohaveasignificantcorrelationbetweenchangeinnetworkingscoresandchangeindisasterpreparednessscores.
However,notenoughtimepassedbetweenT1andT2tosufficientlydocumentandtestthehypothesis that strengthened networks and greater collaboration leads to better disasterpreparednessandultimatelyresponse.In2016,DEPPfieldoperationswerestilladaptingtolocalfeedbackandinterventionswerestillevolving.DatacollectioninT1inevitablypickedupnetworkeffectscausedbyshiftingstrategiesthatarecommontofirstyearprojectimplementation.T2more likelycapturedsomeof thedistinctiveprogramme impactsofDEPP interventions,butsufficienttimetofollownetworkchangerequiresatleastanadditional12months,ifnotmore.
QUESTION 3.D: WHAT UNIQUE CONTRIBUTION DID COLLABORATIVE RELATIONSHIPS AND ‘MULTI-STAKEHOLDER PLATFORMS’ MAKE TOWARD DEEPENING CROSS-PROGRAMME LEARNING? Cross-project collaboration contributed to improved sharing of learning and evidence at the programme level but this did not necessarily translate to behaviour change.As described in chapter 4, there was emphasis on enhancing cross-programme learningthrough a number of approaches led primarily by the learning Project. This included thedeploymentoftwoRegionallearningAdvisors,oneinEthiopiatocovertheAfricaregion,andtheotherinPakistantocovertheAsiaregion.Asthesewerereportedtobehelpfulintermsof stimulatingcollaborationandcross-programme learning, theevaluation’s formativephasereportrecommendedtoadditionallyembedlearningAdvisorsatthecountryleveltofurtheramplifycollaborationandlearning.Asreportedintheinterimreport,countrylearningAdvisorswere assigned to the Philippines, Kenya, DRc and bangladesh, in addition to Ethiopia andPakistan.countrylearningAdvisorsappearedtohavecontributedtoincreasedcollaboration.One intervieweenoted: “Initially,whenwe [didnothavea]DEPP learningadvisorhere,wewereworkingdifferently…. therewas…astandardapproach.butafterherarrival,westartedcollaboratingwitheachotherandtooktheopportunitiesformbenefitingfromeachother…wesharedlearningswitheachotherandincreasedoutreach.”
149 Organisationalpreparednessscore:Organisationalchecklistconductedwitheachorganisationandassess-ingsevendomains:1-Emergencypreparednessplans,2-humanresourcesandcapacity,3-logisticsandsupplies,4-legal,documentationandpolicy,5-Organisationandsystemprocesses,6-Emergencyresponseteamandstaffing,7-communicationandmedia.Aggregatescoreoutof100generatedforeachorganisa-tion.
146 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
figure5.6:UsefulnessoflearningProjectatT2
QuantitativedatafromKAPsurveys(figure5.6)showthatatT2,59%ofintervieweesreportedthatthelearningProjectwaseitherveryusefulorextremelyusefulinrelationtotheinterviewees’work.Approximately11%reported that thelearningProjectwaseithernotuseful at all,orslightlyuseful.Intermsofevidencesharing,89%ofrespondentsreportedsharingofevidencewithintheirproject,and72%reportedsharingofevidenceacrossprojectswhichsuggestshighlevelsofsharingbothwithinconsortiaandattheprogrammelevel.Thisprovidesevidenceofcoursecorrection,ascross-projectsharingwasreportedlylowerearlierintheimplementationoftheDEPP.
Qualitative interview data also supports the fact that collaboration and evidence sharingbetweenandacrosstheDEPPhasincreased.ThiswascatalysedtosomedegreebythelearningconferencesinKenya(December2016)andthePhilippines(July2017).botheventsbroughttogetherlargenumbersoflocalpartnersandstakeholdersatthecountrylevelovermultipledaystoshareprojectlearningsandchallenges,andstimulatecollaborationandsharing.TheintentionwastoshifttheactionfromtheUKtothecountry levelbymaximisingengagementof localstakeholders.At theconference inKenya,102DEPPandnon-DEPPstakeholders attended,representing13 countries,while at the conference in thePhilippines, 127people attendedfrom9countries,whichisinlinewiththeobjectiveofincludingadiverserangeofpartners.Thelearningconferenceswereperceivedbyintervieweestobeeffectiveinsharinglearning,creatingbridgesacrossprojectsandimprovingprogrammecohesion.forexample,onerespondentstated:
“[The]Internationallearningconference[is]wherewesharedourachievementsandbarriers…attheendoftheconference,therewere6-7projectswhoagreedtoworkwithADcAP…afterthat,someofthemevenfollowedupwithmehere.So,workingintheDEPPasawholehasadifferentkindofimpact,becauseithelpsusshareourlearningwithothercolleaguesaswellandhelpsusincreaseouroutreach.”
Another interestingoutcomeoftheseconferenceshasbeencross-projectandcross-countryexchangevisits to learn directly fromother projects and in different contexts. for example,MyanmarprojectstaffvisitedthelinkingPreparednessResilienceandResponseprojectinKenyafollowingthenairobiconference,andthePhilippinesconferenceagendaintentionallyincludedprojectsitevisits.Inaddition,UrbanEarlyWarningEarlyActionprojectstakeholdersdescribedplanstovisitaprojectinnepal,andthebeginningofacollaborationwiththeAlERTproject.nepalisoneofseveralnon-DEPPcountriesinwhichDEPPprojectsareimplementingactivities.Stafffromthebangladeshcommunicatingwithcommunities(cWc)workinggroupcompletedan exchangevisit to thePhilippinescWcgroup and this has led to on-going collaborationbetween the twoworkinggroups.Therehavealsobeen reportsof theTalentDevelopmentprojectcollaboratingwiththeProtectioninPracticeproject,andofmanyprojectsworkingwithorincludingAgeandDisabilitycapacityProgramme’sMinimumStandardsforAgeandDisabilityInclusion.
While therewasevidencethatcollaborativerelationshipscontributedtosharingof learning,therewaslessevidenceonhowlearningwasutilisedandwhetherthistranslatedintoactualbehaviourchange.
7%
42%
N = 154
USEFULNESS OF LEARNING PROJECT4%
30%
17%
NOT USEFUL AT ALL
SLiGHTLY USEFUL
SOMEWHAT USEFUL
VERY USEFUL
EXTREMELY USEFUL
147 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
EFFICIENCY AND VALUE FOR MONEY
cHAPTER6
148 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Evaluation Question #4: How economically have resources/ inputs (funds, expertise, time etc.) been converted to results? To what extent does preparedness improve the efficiency of humanitarian response?
Sub-questions:
a) Haveresources(funds,humanresources,time,expertise,etc.)beenallocatedstrategicallytoachievetheprogrammeobjectives?
b) Haveresourcesbeenusedefficiently?Ingeneral,dotheresultsachievedjustifythecosts?couldthesameresultsbeattainedwithfewerresources?
c) Haveprogrammefundsandactivitiesbeendeliveredinatimelymanner?
Data sources:
• Organisational surveys (Quantitative)
• Intensive set IDIs(Qualitative)
• Document review including financial information(Qualitative)
• IDIs on VFM(Qualitative)
Key Findings:
• focusingoneconomyandefficiency,VFM shows good potential for the programme with areas for improvement
• The collaborative model necessarily lends itself to a degree of inefficiency, high transactions costs and slow information flows. This model had substantial benefitshowever,and for some projects the benefits have likely outweighed the costs.forotherprojects thesamebenefitsof themodelcouldhavebeenachievedat lowercostandgreaterefficiencyifthecontracting,M&Esystemandcostingofactivitieshadbeenmoreconsciouslyplannedfrominception.Thecollaborativemodelthatwastiedtohierarchicallongdeliverychainsandmultipletiersofcontractsandfundingflowlikelycompromisedeconomyandefficiencytoadegree.
• Goodindicatorsofcosteconomywereevident,thoughsome budgets were lean with insufficient resources for portfolio management and collaboration activities.
• There were shortfalls in terms of systems and resources set up for efficiency in governance and strategy, portfolio management, decision-making and consortium arrangements.The spendingonManagement functionsat theprogramme levelweretooeconomicalandwould have benefitted from some funds and time being redeployed from the Learning Project to the DEPP management team in the form of M&E expertise, bothattheprogrammeandprojectlevels.If,however,independencewasdesired,thenmoredirectlinkageandcommunicationbetweenthemanagementteamandthelearningProjectteamwherebydatafromM&Eflowsregularlyanddirectlytothemanagementteamthroughbothautomaticreportingordirectdataaccessthroughadashboardfollowedbyregularlybilateralmeetingscouldhavebeenconsidered.
• Positive findings in terms of VFM reporting, adaptive management and collaborative waysofworkingwereidentified.
• Thereare no strong quantitative VFM findings to suggest that preparedness improved the efficiency of humanitarian response. TheempiricaldatafoundnosignificantdifferencebetweenDEPPandcomparisonorganisationswithregardstoperceivedimpactofDEPPoninstitutionalspeedandcostofresponse,andtheextenttowhichinstitutionalandpolicyenvironmentaffectsthespeedandcostofresponse.
• Cost per result indicator analyses demonstrated good VFM per result actually achieved.
149 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
OVERVIEW OF THE APPROACH TO ASSESSING EFFICIENCY AND VALUE FOR MONEY ThischapterreportsfindingsfromtheeconomicassessmentoftheDEPPprogrammewhichwasledbytheteam’sEconomicAdvisor(SeeAnnex2forbiography).Theeconomicanalysisusedthe4Eframework(economy,efficiency,effectiveness,andequity)toevaluatetheDEPP’sValue forMoney (VfM). conclusions on effectiveness (cost of outcomes and impacts) andequity (distributionof impacts) are presented inchapter 4 (Relevance andEffectiveness ofInterventions)andchapter7(sectiononinclusionofvulnerablegroups).Tolimitduplication,thischapterfocusesonefficiencyandeconomy,butallofthefindingsaredrawntogetherwiththeVfMworktobuildacompleteVfMpicture,withparticularfocusattheprogrammeandthecountrylevel.Themethodologyisdetailedinchapter2,butbrieflyitcomprisedquantitativeandqualitativemethods,inparticular,budgetandexpenditureanalysis,internalbenchmarkingofunitcosts,costeffectivenessanalysis(includingassessmentofcostperresultindicators),keyinformantinterviewswithintheDEPPmanagementteam,DEPPboardandwithintheselectedcasestudies,documentreview,analysisofadherencetoprocurementprocedures,andanalysisof how decision-makers and project managementwithin DfID and implementing agenciesconsiderVfM.DatacollectioninstrumentsarefoundinAnnex7.
TheformativephaseoftheeconomicevaluationcoveredportfoliolevelVfM–economyandefficiency.Thecountry-levelVfManalysisiscoveredinthesummativereport,largelybecauseitreliesontheoutcomesanalysisconductedbythewiderevaluationteaminthesummativephase.
QUESTION 4.A: HAVE RESOURCES (FUNDS, HUMAN RESOURCES, TIME, EXPERTISE, ETC.) BEEN ALLOCATED STRATEGICALLY TO ACHIEVE THE PROGRAMME OBJECTIVES?
Programme Budget
Table6.1belowshowsthetotalprogrammebudgetfortheDEPP.Theconclusionshavenotchangedsincetheformativephasereport–managementcosts (lessthan5%ofDEPPtotalbudget)aretooleanforaprogrammeofthissizegiventhehighresponsibilitiesofmonitoring,reporting,coursecorrection,strategicdirectionandgeneralsteeringof£25.5millionofprojectfunds(excludingthelearningProjectbudget).ThelearningProjecthadahealthybudget.Someof thiscouldperhapshavebeenredeployed intothemanagement functions.Seebelowforfurther discussion on this issue.Alternatively, further investment could have beenmade indeveloping and implementing an electronicM&E informationmanagement system to trackprogress inreal-timeandenablemoreagilecoursecorrection.Thissystemcouldhavebeenindependently set-up by the learning Projectwith themanagement teamhaving direct ondemandviewingprivileges.
TYPE OF COST AMOUNT (£) % OF TOTAL BUDGET
STARTnetworkManagementcosts (includes12.6%nPAcrateandProgrammeboardconsultancycosts)
936,222 3.1
cDAc-nManagementcosts (includes7%nPAcrate)
245,186 0.8
150 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
STARTnetworkProjectfunds (excludingthelearningProject)
22,720,403 75.7
learningProject (includes£1millionfortheindependentevaluation) 3,343,375 11.1
cDAc-nProject 2,754,814 9.2
Total 30,000,000 100
Table6.1:TotalDEPPprogrammebudgetbasedonactualspendinguntilDec2017andforecastbudgetuptoJune2018
Value Chains and Transactions Costs within Projects TheDEPPbusinesscaseemphasizesatvariouspointsthattheDEPPwouldbeimplementedusingcollaborativemodelthroughmulti-stakeholderplatforms.ThebusinesscasestatesthattheDEPPwill“supporthumanitariancapacitythroughthecollective,collaborativeactionofInternationalnon-GovernmentalOrganisationsandtheirpartners....Supportwillnotbelimitedtoskillsbuilding,butwillpromotepathwaysbetweenlocalandinternationalactorsthroughthecreationofjointplatformsandnetworks,andbuildcommunitypreparednesssystems.PartnersmayincludeotherInGOs,localnGOs,localgovernment,nationalgovernment,academia,privatesectorcompanies,theRedcross/RedcrescentMovement,Unagenciesandothercivilsocietyorganisations.”Inaddition,thebusinesscasehighlightsmulti-stakeholderplatformsasameanstobuildcapacity:“Analysissuggestsitisimportanttosupportnetworksandmulti-stakeholderplatformstodevelopcapacity,workingwithexistingoneswhereeverfeasible.”Itdescribesmulti-stakholderplatformsas:“a‘multiplicityoforganisationsatdifferentscalesofgovernanceworkingtowardsmorecoordinatedandintegratedactions.”150
ThebusinesscasearticulatedthatDEPPinterventionswouldbedeliveredthroughmulti-stakeholderplatformsthatwouldbeimplemented:“fromlocaltointernationallevelsthatenablecollectiveactionandcaptureandsharelessonsandgoodpractice.Theseplatformswillhaveabroadmembership-e.g.,InGOs,nationalnGOs,localgovernment,nationalgovernmentacademia,privatesectorcompanies,theRedcrossRedcrescentMovement,Unagencies.Successfulplatformsshouldimprovepolicyatnationalandinternationallevel.Weenvisageasingleplatformpercountryorregionwheretheprogrammeisactiveandaninternationalplatform.Possibleproductsoftheplatformsare:synthesisedlessonsidentified,jointevaluationsandcommonpositionpapersforadvocacy.”
Thereareclearlyrisksandcostsoftakingthisapproach.However,intheVfMsectionofthebusinesscase,therewasnodescriptionoftherisksofthisapproachintermsoflayersofmanagementoverheadsandhightransactionscosts.Itwouldbedifficulttoimplementsuchanapproachwithoutincurringsomedegreeofextratransactionscosts,compliancecostsandriskmanagementateverylevel.Overheadsarenecessaryforagentstomanagerisk,andpreventfraudandcorruption.Moreover,sincethebusinesscasewaswrittencomplianceassuranceswerestrengthenedbyDfID;inDecember2016,DfIDissuedalettertoallsuppliers,statingthatsuppliershouldmeetevengreaterrequirements.
Given that the collaborative approachwas agreed upfront, itwould have been difficult forprojectstoavoidalotofthesecosts,whichariseinnecessarymanagementoverheadsateverylevelofthedeliverysupplychain.Insomecountries,regulatorybodiesrequireagenciestobeabletomaintaincertainduediligenceprocessesthatarecoveredbyoverheadcosts.WhetherthischoiceofmodelreflectsgoodVfMdependsonwhetherthebenefitsofthemulti-sectorcollaborative approach outweigh these extra transactions costs. This differs depending onprojects,butasdescribed inchapters4,and5,themajorityofDEPPstakeholdersdescribestrong value of collaboration andworking through consortia and consistently describe thecollaborativemodeltothebethemostappropriateapproachtodeliverhumanitariancapacitybuildingandresponseactivities.Therewasatleastoneproject(theTalentDevelopmentproject)wherethebenefitsofthecollaborativemodelwerenotfelttooutweighthecostsbyanumber
150 Djalante,R.Adaptivegovernanceandresilience:theroleofmulti-stakeholderplatformsindisasterriskreduction.nat.HazardsEarthSyst.Sci.2012;12:2923-2942.
151 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
of stakeholders151.This particularprojectwas amore traditional individual capacitybuildingprogram that directly trained humanitarian staff in a number ofDEPP countries. It did notincludeotherDEPPcomponents(suchasearlywarningsystemdevelopment,andpolicyefforts)thatperhapsmaybebettersuitedforthecollaborativeapproachthantraditionaltraining.
DuringVfMinterviews,severalprojectsalsoexpressedthatthecollaborativeapproachworkedwell for their projects. Exampleswere provided of specialisation in someprojectswith thisstructureleadingtosavings.forexample,intheTransformingSurgecapacityproject,cAfODdonatedtimeandintegratedprojecttasksintotheirownwork.Thistypeofstructureisbeneficialasitcanalsocanleadtoincreasedsustainabilityandownershipandallowcontinuationoftheworkbeyondtheendoftheprojects.
Therearetwokeyissuestoconsiderwithrespecttothecollaborativemodel:1)thecollaborativemodelwillnecessarilyresultinhighertransactionscostscomparedtoanalternativemodelwithlesslayers,and2)theremaybeacasetosuggestthatthecollaborativeapproachcouldhavebeenachievedatgreaterefficiencyandlowercost.Thelatterisconsideredindetailbelow.notably,therearesomeexamplesofprojectsthatappearednottohaveoverlyhighoverheadsateverylevel.forexample,expendituredatafromtheProtectioninPracticeprojectdemonstratethatoverheadswerenotbeingappliedateverytier.Theoriginalprojectbudgetwas£804,634andatJune2017thiswasthebudgetandthedeliverysupplychainthatDfIDreviewedaspartoftheirinternalreviewofDEPP.ThenPAcsetforthiswas£89,594whichisroughly11.13%.Theprojectgainedafurther£66,998foratotalbudgetof£870,137.WhilethisdidhaveannPAcchargedtoit,agranttocEDIER(alocalnGO)bytheProtectioninPracticeprojectwasthenexecutedandnooverheadsweretaken.
Table6.2showsthetiersoffunding,onecouldmisleadinglyconcludethatthedifferencebetweenTier2and3(whichis£224,068attherevisedJune2018budget)couldbesolelyduetooverheads.Thisisinfactnottrueastheactualoverheadfigurebetweenthesetierswasactually£96,846.IntermsoftheoverheadstakenatTier3,unfortunately,thefinancialmanagementsystemwasnotsetuptocheckwhetheroverheadsweretakenornot,andtowhatextentifso.
furtherinvestigationoftheProtectioninPracticeexample,showsthatthecostsrelatedtoprojectstaffwerelistedasacombinationofdirectandindirectcosts,butinrealityallthestaffwereeitherprotectiontechnicalspecialistsorfinancestaff.Theprotectiontechnicalspecialistswerehybridprotectionadvisor-managers(theywererequiredtoprojectmanagetheirpiecesofworkbutalsotoadviseatworkshopsandontoolkitdevelopment).Inaddition,thereweresmallerpercentagesoftwofinancestafftimeincludedinthebudget.ThisincludedonefinancestaffpersonatOxfamandoneatInternationalRescuecommittee(IRc).Thesepositionswereessentialtocoverthemixtureofregularinternalaccountsmanagementalongsidetheflexiblefundinggrantsthatwentthroughbothagencies.Accordingtothedeliverysupplychainthisincludedover30differentl/nnGOorganisationsreceivinggrants.Thechaindoesnotgiveanyindicationofhowmuchworkbehindthescenesthiswas,orwhethertheSouthSudanconflictcreatedmoreworkforthefinancialmanagementoftheproject.
151 TheTalentDevelopmentProjectfinalEvaluationReport,p12,notes:“AttheendofTDP,thereweremixedfeelingsabouttheconsortium’sadded-value.HalfofrespondentsquestionedtheneedforhavingaconsortiumbecausetheythoughtthecostofrunningtheconsortiumoutweigheditsbenefitsandwouldnotwanttocommittoaTDP2,shouldtheybegiventheopportunity.”
152 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Tier2 Tier3 Tier4 Tier5
STARTnetworkSub-granttoOxfam
OxfamSub-grantto6l/nnGOsand1InGO(IRc)
Agencies
IRccontractorAgreementwithWorldVisionforWorkshops;Sub-grantto32l/
NNGOs
WorldVisioncontractorAgreement
(ThisAmountisaStraightPassThroughoffunds)
June2017budget £804,634 £571,129 £72,364 £10,439
RevisedJune2018budget £870,137 £646,069 £61,924 £10,439
Table6.2:TiersoffundingforProtectioninPractice
It’sworthnotingthatintheJune2017budget,£250,684wasspentoncapacitybuildingandpilots(PIPbudgetline12)andbasedonanalysisofactualsexpenditure,noneofitwasspentintheUKonoverheads.Rather,itwasspentincountryonacombinationofelementsincludingsalariesoftheprotectiontechnicalspecialists,costsrelatedtothel/nnGOparticipants,costsrelatedtotrainingoftrainerworkshops,andpilotcosts.Thisagainemphasisesthatprojectcontentandactivitiesarethemajorcostateachtier,andnotoverheads.
fortheProtectioninPracticeexample,thereareanumberofcoststhatsitacrossTier2andTier3.ThisoccurredbecauseOxfamdirectlyprovidedfundstoIRcandtosixl/nnGOsatthesametier.Thismeansthatmoneyhitfrontlinel/nnGOsatTier3,notattheendofalongchain.ThisisagoodexampleintheDEPPofatleastsomel/nnGOsnotsittingdowntheendofanunnecessarilylongdeliverysupplychain.
A Reconsideration of the Collaborative Multi-stakeholder Approach and VFM?
Itisclearthatthecollaborativeapproachwasmoreappropriateforsomeprojectsandinthosecases,thebenefitslikelyoutweighedthecosts.forexample,insomelocationsandprojectsitwasnecessarytoworkwiththespecialisedorganisations,especiallywhenkeyrelationshipswerenotwiththeleadagency.Asanexample,inGambella,Ethiopia,christianAidcouldnotdirectlycontractwiththeMinistryofHealthanditwasessentialtoworkwithlocalandnationalactorswiththerightlanguageskillsandcontextualisedknowledge,andexistingpartnershiproutestomanagethecollaborativeaction.Allworkwithnationalorlocalgovernments,andpotentiallynon-stateactorsinconflictsettingsneededin-countrystakeholderstodothis.
butinotherprojectsthismayhavebeenlessofanissue.Theprojectswerelikelyunnecessarilyburdenedwithahierarchyofflowoffunds,hightransactionscostswithcontracting,andcompliancecostsateverylevel.Anotherproblemrelatedtothishasbeenaroundinformationflow–longsupplychainsprolongedthetimetakenforreporting,theapprovalsprocessateverylevelwaslikelyunnecessaryandinformationflowwasslowasaconsequence.
Ifthelongsupplychainissuehadbeenconsciouslythoughtthroughattheinceptionphase,forexamplebymappingoutthechainusingtheDfIDapproachforstakeholdersandadaptingitbasedonneeds,itislikelythatprocessescouldhavebeenstreamlinedresultinginlowercostsoverall,greaterefficiencyandbetterVfM.Ifthereportingapprovalsdidnotneedtobetiedtothesupplychain,amoreproactiveMElsystemcouldhavebeensetupwhichbypassedcumbersomeapprovalsateverylevel.Separatingoutthelayersofthesupplychainwiththereportingrequirementsandinformationflowscouldhavefacilitatedthis.Also,developingbettercostmodelsonspecificactivitieswouldhaveimprovedefficiencyandloweredcosts.
Inconclusion,thecollaborativemodelnecessarilylendsitselftoadegreeofinefficiency,high
153 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
transactionscostsandslowinformationflows.Thismodelhadsubstantialbenefitshowever,andforsomeprojectsthebenefitshavelikelyoutweighedthecosts.forotherprojectsthesamebenefitsofthemodelcouldhavebeenachievedatlowercostandgreaterefficiencyifthecontracting,M&Esystemandcostingofactivitieshadbeenmoreconsciouslythoughtthroughfrominception.Thecollaborativemodelthatwastiedtohierarchicalandlongdeliverychainshaslikelycompromisedeconomyandefficiencytoadegree.
Allocation of Time and Resources for DEPP Management, Governance, Learning and M&E Functions
In terms of resource allocation to the management functions of DEPP, as explored in theformativephase,thiswasnotconsideredtobeoptimalasresourcesweretooleantoundertakefull portfoliomanagement functionsadequately, asonewouldexpect for aportfolioof thissize.nosignificantchangeshavebeenmadetoresourcesallocatedtotheDEPPmanagementteamsincetheformativephasereport.AsseeninTable6.1themanagementcostscompriseover4%of thebudget. In-depth interviews indicated thatmanagement resourceswerenotincreasedbecauseSTARTnetworkmembersfeltthatfundingacentralfunctionwashighcostandwouldtakecostsawayfromfrontlineactivities.budgettofundafull-timeadvocacypostwithinthemanagementteamwasprovidedfromthelearningteambudgetafteritwasrealised(and ‘demanded’byDEPPprojects) that thiswasneeded.Theprogrammemanagementalsoused consultancy forVfM advice –which is costed into the 4% figure of STARTnetworkadministrationcosts.TheDEPPteam,therefore,diddrawuponexternalexpertiseasandwhenrequired.
However,whilemanyprogrammesareaimingtoreducehighadministrativecost,thiswastoolean for such a complexprogramme tobemaximally effective.basedonfindings from IDIsandthesheervolumeofreportingfromafundofthissize,extraresourcesinthemanagementteamwouldhavebeenwellspent.SuchresourcescouldhavebeenspentontechnicalM&Eexpertisethatwouldhavebetterenabledthemanagementteamtoscrutiniseprojectprogressviaquarterlyreportsandothermeans,soastobetterreportresultstotheprogrammeboard.Thiswouldhavebetterenabledtheprogrammeboardtounderstandhowresults(outputsandoutcomes)wereunfolding,andtomakestrategiccoursecorrectiondecisions.
Programme-level Resource Allocation
TheVfM review explored resource allocation at the programme/DEPP level, through IDIsanddocumentreview.Inparticular,questionswereaskedaboutwhethertheportfoliomixofprojectswasconsideredoptimal.
EvidencesuggeststhatDEPPdidnotfunctionasaportfolio,andeachprojectfunctionedasastandaloneproject.ThiswasduetothedesignhistoryoftheDEPP–thesetof individualprojectswasnotdesignedasabalancedportfoliointermsofrisk,typesofactivities,size,andcost.Ideally,inaportfolio,arangeofexperimentsisneededinordertoassesswhatworksandthisshould leadtoadditional funds formoresuccessfulprojectsas implementationunfolds.Thispracticallywouldtranslateintoaseriesofpilots,followedbyscaleupandcoursecorrectionwhereneeded.Insteadofthismodel,projectsweredesignedinisolation,andretrofittedtothebusinesscaseortotheselectioncriteriaandfocalcountriesusingatop-downapproach.In-depthinterviewshaveindicatedthattheProgrammeboardwasnotabletoadequatelysteerandcoursecorrectthefundandthattheydidnothaveagoodhandleonresultsofprojects.
Theothermainvehicle for steering theDEPPProgrammewas the learningProject.At thestartofDEPPtherewasastrategicdecisiontoallocatesignificantresources(£3million)tothisproject.Thereweresomesuccesses in termsof thegenerationof learning fromtheproject(presented in chapter 4); the move of the DEPP learning Project team to have countrylearningAdvisors,andregionalandnationaleventswasgreatlyappreciatedbycountrystaff,andgavemoreopportunitiesandspacestodiscusscollaborationandshare learning.ProjectstaffindicatedthattheyhadusefulsupportfromthelearningProjecttogathercasestudiesfortheproject.forexample,theTransformingSurgecapacityteamindicatedthattheydidnothavethecapacitytogathercontentforthefutureforSurgeeventandthelearningProjectteamwasabletosupportthem.TheDEPPonlinelearningplatformwasconsideredusefulasamoreflexiblespacetoshowcaselearningfromtheproject.Thisalsosupportsthesustainability
154 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ofprojects.
Some interviewees indicated thatwhile thereweresuccesseswith thelearningProject, thedemandofprojectresourcessuchastimeandeffortsweretoohighandhenceplacedtoomuchburdenonprojects. for example, interviewees stated that therewas toomuch informationrequestedfromprojects,andsomeduplicationwithinformationthatwasrequestedfromregularreportsandfinalevaluations.RespondentsalsostatedthatwhileUKlearningeventshadbeenuseful,thefinalgloballearningeventinGenevainMarch2018wasdemandingandconfusing.
Itisquestionableastowhethertheset-upofthelearningProjectandstrategicresourceallocationwastherightone. In-depth interviews indicatedthat theDEPPwouldhavebenefittedfrommuchmoreresultsmonitoring,reporting,analysinganddata-drivendecision-making.Arguablyaprogrammeneedsgoodmonitoring,reportingandevaluationbeforeitcanengageinlearningactivities.Strong,harmonisedM&Ehasbeendistinctlymissingattheprojectandprogrammelevel.HavingsomemoneyallocatedtoM&Ewouldhavereapedbenefitsintermsofgreaterabilityofprojectstomeasureresults,andfeedtheseuptotheProgrammeboard,soitcouldbemorestrategicinitsactionsandmoreefficientlycoursecorrecttheDEPP.StrongerM&EwouldalsohavebeencomplementarytolearningProjectactivitiesandwouldhaveservedtoenhancethefluidityoflearning.
In-depthinterviewsalsoindicatedthatthelearningProjectactivitiescouldperhapshavebeenbetterplacedattheprogrammemanagementteamlevel,ratherthanasacompetingprojectwithitsownagendaandobjectives.learningobjectivesshouldnotbeseparatetoprogrammemanagement objectives, as they were in the existing DEPP structure. In-depth interviewsindicated that the learning Project team could have communicated their findings better tothemanagementteamandProgrammeboard. Ingeneral, therealsocouldhavebeenbettercommunicationbetweenthemanagementteamandtheProgrammeboardintermsofobjectivesandexpectationsrelatedtothelearningProjectactivities.Programmeboardmembersarenotlikelytoreadreportsandinsteadneededmoretargetedandconcisecommunicationsmaterials.Alternativeinnovativemodelofinformationandlessonslearnedcommunicationchannelshouldhavebeenexploredandimplemented.
The Use of VFM Indicators and Methods
TheuseofVfMindicatorshavemainlybeenwithinthequarterlyreports.Aftertheformativephaserecommendationto increasetheuseofVfMindicators, theDEPPmanagementteammadeastrongefforttoincreasetheprojectstaffcapacityaroundVfMconceptsandreportingthroughtheimplementationoftwohalf-daytrainingevents.Thistraininghadseveraloutcomes.first,someprojectscarriedoutadhocreportsonVfM.Second,mostprojectsincorporatedVfMasacomponentoftheirfinalevaluations.Duringthetraining,aVfMcompletiontemplateforprojectswasintroducedtorecordVfMqualitativeand/orquantitativefindingsatcompletion.Atthetimeofwritingthisreport,thecompletedtemplateswerenotyetavailableandthushavenotbeenreviewedbytheEvaluationteam.
ReviewofavailabledocumentsprovidedseveralgoodexamplesofadhocVfMreports.TheTalentDevelopmentprojectgeneratedsomeusefulVfMfindingsintheirfinalevaluationreportwhichfocusedonthecosteffectivenessofdifferenttrainingmodels.Inaddition,theTransformingSurgecapacityprojectcreated tworeports,oneon theVfMrelated tocollaborationwhichhighlighteditshighcostandoneonthefinancialsustainabilityofsurgecapacity.However,itisnotclearhowthesereportswillbeusedgoingforward.
Table6.3summarisestheVfMindicatorsreportedintheMay2017projectquarterlyreportswhichwerethelatestavailablereportatthetimethetablewasdeveloped.AtleasttwoofthereportshadnoVfMsections.However,theevaluationteamacknowledgesthatsomeprojectteamshaveintegratedVfMthinkingthroughoutdifferentsectionsoftheirreports.Whilethereare some useful indicators, to have hadmoremeaningful reporting ofVfM, itwould havebeennecessarytousemorespecificandindepthVfMindicatorsforeachproject.Acrosstheprojects,acoresetofcross-cuttingindicatorswhichcanbeaggregatedandcomparedwitheachother,asdescribedinformativephaserecommendationswouldhavebeenusefultoadequatelycaptureVfM.Theremainingindicatorsshouldhavebeenprojectspecificandtailoredtoeachproject’sneeds.Indicatorsshouldhavebeenbothquantitativeandqualitative.
155 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
HIGH-LEVEL SUMMARY OF VFM REPORTING IN MAY 2017 QUARTERLY REPORTS
Economy
Cost savings, benefits in kind, Oxfam procurement procedures, use of local mate-rials, pro bono work from London School of Economics, pro bono support for web-site development, discounted flights
Leveraged other donor funding in Lebanon, DRC and South Sudan
OFDA-funded consultation, leveraging and sharing of resources, working with oth-er projects and sharing resources
Efficiency
Efficiencies with integrated programming, where partners are already working, joint training, collaboration with partners to provide efficient support for confer-ences, Learning Advisors hosted by DEPP partners in country, proactive use of processes and tools, using data, prepositioning of goods
Equity Targeting the right people, partners and beneficiaries have been fully involved
Effectiveness Community-level institutions, good coordination and communication with Govern-ment. Ensuring capacity and sustainability of in country staff so programme can be more locally oriented in the future.
Table6.3:SummaryofVfMreportingfromMay2017QuarterlyReports
TheDEPP could have benefitted from the addition of indicators as it matured in order toprovideamorecomprehensivepictureofVfM.Ingeneral,themostimportantdriversofVfMareeffectivenessandequity152 andsuchdatashouldbecollectedasprojectsandprogrammesmature. Some suggested indicators are shownbelow inTable 6.3.These are not necessaryquantitativewhichmakesdirectcomparisonslessstraightforward,butneverthelesstheyofferusefuldatatoinformdecisionsaboutstrategyandfunding.Suchdataareusefulforassessingtheprogrammeasawhole,for learningandalsotomakechangesdynamically.forexample,those projects that perform particularly high on these indicators are those that likely showhigherreturnsandthuscouldbepromotedmorethanthosethatdonot.
Suggested VFM indicator Comments
Economy/Efficiency
Evidence of economies of scope and scale within operations and budget
Economy: Economies of scope through shared overheads across projects – measured by cost savings, or just narrative
Number of operational adaptations resulting in better cost effectiveness or value generation
Efficiency: Semi-qualitative, to demonstrate that operations were changed on the grounds of VFM; actual number of changes and supporting narra-tive
Effectiveness and equity (more important for strategic decision making)
Rating of the quality of partnerships Efficiency/Effectiveness: Partnership quality a key attribute of efficiency –a rating system
152 HighlyeconomicalandefficientprogrammeswillnotoffermuchVfMiftheyaretargetingthewrongthings/people(equity),ordoingthewrongthings(effectiveness).
156 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Does the programme continue to meet needs identified upfront?
Useful to revisit what the project is actually aim-ing to achieve at key break points. If the answer to the question is no, this is a breach of effec-tiveness and resources are not being allocated to where they are needed. Narrative.
Is there an exit strategy or other measures to en-sure sustainability?
Effectiveness: Without sustainability of the research uptake the interventions are compro-mised; i.e., what measures are in place to ensure that the interventions will continue post project completion? Explanatory narrative.
In terms of the methodology, is there a trade-off between cost and working with hard-to-reach populations, and if so, is this explicitly accounted for in the budget?
Equity: A key equity point – it is often more ex-pensive to reach vulnerable groups, so this needs conscious budgeting to maintain VFM. Costs and narrative.
Table6.4:ExtrasuggestedVfMindicatorsforQuarterlyReports
Insummary,intermsofstrategicresourceallocation,theevidenceshowsthattheinitialprocessofselectingprojectswasnotstrategic,andthismayhaveimpactedontheeffectivenessofthewholeprogramme.WhilethelearningProject interventionswereuseful, itmighthavebeenmorestrategictoredeploysomeofitsresourcesintothemanagementteamforexpertisefortechnicalprogrammaticandproject-levelmonitoringandreportingofoutputsandoutcomes.ThiswasnotpartofthelearningProjectteam’sremit.ThiswouldhavepermittedtheProgrammeboard to havemore regular and direct results to ensure on-time and on-demand decisionmaking,andhenceenablecoursecorrectionasneeded.
Itisnotedthatasthelogicalframeworkdatastartedtobecomeavailableandregularlyreportedonaquarterlybasisbythelearningteamthemanagementteamwasthenabletousethisforreporting toDfID, board and take course correctionmeasures such as increasing focus ongender considerations. However, thiswas insufficient in terms ofmonitoring reporting andcoursecorrection.
QUESTION 4.B: HAVE RESOURCES BEEN USED EFFICIENTLY? IN GENERAL, DO THE RESULTS ACHIEVED JUSTIFY THE COSTS? COULD THE SAME RESULTS BE ATTAINEDWITH FEWER RESOURCES?
Thediscussionaboveonvaluechainsandtransactionalcostsdemonstratesomeinefficiencieswithin the programme and help to partially respond to this question. further assessmentof empirical data related tohowDEPPpreparedness activities affected the cost of disasterresponse,andthecostperactivitiesandcostperresultsachievedwilladdressthequestiononwhetherresultsachievedjustifythecostoftheprogramme.
Empirical Findings on Disaster CostTo assess cost effectiveness of theDEPP projects, several lines of analysis come together.first,surveyquestionswereincludedintheT2organisationaldatacollection(seeboxbelow).ThesequantitativequestionnaireswerecompletedwithseniormanagementatbothDEPPandcomparisonorganisationsinthefourintensivesetcountries(Ethiopia,Kenya,MyanmarandthePhilippines),andthequestionscapturetheirperceptionsaboutwhetherDEPPpreparednessactivitieshaveimprovedthespeedandcostofresponse.Asthisdatawasonlycollectedatthesecondtimepoint,itisnotpossibletoestimatechangesovertime.Instead,theanalysisfocuseson assessing differences between theDEPP and comparison organisations in each countryandoverallwiththepooleddataset.StatisticaltestsareusedtoanalysewhetheranyobserveddifferencesbetweentheDEPPandcomparisongrouparestatisticallysignificant(p<0.05).
157 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
KEY COST EFFECTIVENESS DATA COLLECTED AT TIMEPOINT 2Organisational level: Organisational survey with senior management.
• Perceptions on preparedness and speed of response: On a scale of 1-5, to what extent have the preparedness activities helped improve the speed of response?
• Perceptions on preparedness and cost of response: On a scale of 1-5, to what extent have the preparedness activities helped reduce the cost of response?
• Perceptions on timeliness of previous response: How would you rate the timeliness of your organisation’s response to the disaster? Where 1=Very untimely, 2=Untimely, 3=Untimely, 4=Somewhat timely, 5=Timely
• Perceptions on cost of previous response: How would you rate the costliness of your or-ganisation's response to the disaster? Where 1=Very expensive, 2=Expensive, 3=Somewhat expensive, 4=Inexpensive, 5=Very inexpensive
• Perceptions on institutional and policy environment and speed of response: On a scale from 1-5, how much does the institutional and policy environment impact the speed of your response?
• Perceptions on institutional and policy environment and cost of response: On a scale from 1-5, how much does the institutional and policy environment impact the cost of your response?
Overall,noneoftheindicatorsassessedshowanystatisticallysignificantdifference153betweentheDEPPandcomparisonorganisations (i.e.,anydifferencesobservedarebychanceratherthananyrealdifference). Table6.5belowpresentsperceptionsonpreparednessandspeedandcostofresponsebycountryandforthepooledsample.MostrespondentsinbothDEPPandcomparisonorganisationsreported3to4outofthe5-pointscale154whenaskedwhetherpreparednessactivitieshelpedtoimprovethespeedofresponse.Asimilartrendisseenwhenaskedwhetherpreparednessactivitieshelpedtoimprovethecostofresponse.Overall,therewasnosignificantdifferenceinanyofthecountriesbetweentheDEPPandcomparisongrouporinthepooleddata.ThefindingssuggestthatDEPPpreparednessactivitieshavenotimprovedtheefficiencyofresponse.However,itmustbenotedthatthecomparisongrouphadbeenexposedtosomecapacitybuildingactivities(thoughtoalesserdegreethanDEPPorganisations)andthismightexplainthelackoffindings.
On a scale of 1-5, to what extent have preparedness activities helped improved the speed of response?
1 = no improvement
5 = very strong improvement
SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE
DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison
10
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)1
(5.3%)0
(0.0%)1
(1.2%)0
(0.0%)
20
(0.0%)0
(0.0%)1
(3.7%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)2
(16.7%)1
(1.2%)2
(3.7%)
33
(16.7%)4
(40.0%)15
(55.6%)8
(57.1%)8
(40.0%)8
(44.4%)7
(36.8%)3
(25.0%)33
(39.3%)23
(42.6%)
411
(61.1%)3
(30.0%)8
(29.6%)3
(21.4%)11
(55.0%)9
(50.0%)11
(57.9%)6
(50.0%)41
(48.8%)21
(38.9%)
153 Inprinciple,astatisticallysignificantdifferenceisaresultthatisnotattributedtochance.Moretechnical-ly,itmeansthatifthenullhypothesisistrue(whichmeanstherereallyisnodifferencebetweenthetwogroups),thenthere’salowprobabilityofgettingaresultthatlargeorlarger.
154 Ahigherscoreindicatesahigherlevelofperceivedpreparedness.
158 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
54
(22.2%)3
(30.0%)3
(11.1%)3
(21.4%)1
(5.0%)1
(5.6%)0
(0.0%)1
(8.3%)8
(9.5%)8
(14.8%)
On a scale of 1-5, to what extent have preparedness activities helped reduce the cost of response?
SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE
DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison
10
(0.0%)0
(0.0%)0
(0.0%)2
(14.3%)1
(5.0%)0
(0.0%)0
(0.0%)1
(8.3%)1
(1.2%)3
(5.7%)
20
(0.0%)0
(0.0%)4
(15.4%)0
(0.0%)0
(0.0%)2
(11.8%)3
(15.8%)3
(25.0%)7
(8.4%)5
(9.4%)
34
(22.2%)4
(40.0%)9
(34.6%)7
(50.0%)8
(40.0%)6
(35.3%)8
(42.1%)3
(25.0%)29
(34.9%)20
(37.7%)
412
(66.7%)4
(40.0%)10
(38.5%)3
(21.4%)9
(45.0%)8
(47.1%)7
(36.8%)5
(41.7%)38
(45.8%)20
(37.7%)
52
(11.1%)2
(20.0%)3
(11.5%)2
(14.3%)2
(10.0%)1
(5.9%)1
(5.3%)0
(0.0%)8
(9.6%)5
(9.4%)
Table6.5:contributionofpreparednessactivitiestospeedandcostofresponse
Table6.6presentstheperceivedcostlinessandtimelinessoforganisationalresponseactivitiescarriedoutintheprevious12months.Intheory,ifDEPPpreparednessactivitiesdidaffectthespeedandcostlinessof responseactivitieswewouldexpectDEPPorganisation’s responsesto be less costly andmore timely compared to the non-DEPP group. However, therewasno significant difference between DEPP and comparison organisations for either of theseindicatorsinanyofthecountriesorinthepooledsample.Thesefindingsareinlinewithotherquantitativeindicatorspresentedinchapter4whichshownosignificantdifferenceinperceivedperformanceduringapastresponseforDEPP,comparisongroupandoverall.
contrary to these quantitative findings, qualitative data and case study data presented inchapter4suggestsimprovementsinspeedandefficiencyofresponseincertaincontexts(seechapter4).Whilequalitativechangehasstartedtoappear,quantitativechangerequireslongerandmoresustainedeffortstoachieve.Itmaybetooearlytoassesswhethertheresultsachievedjustifythecost.
How would you rate the costliness of your organisation’s response to the disaster?
1= not costly at all
5=very costly
SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE
DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison
11
(6.7%)0
(0.0%)2
(7.7%)2
(14.3%)1
(5.3%)0
(0.0%)0
(0.0%)1
(10.0%)4
(5.6%)3
(5.9%)
21
(6.7%)2
(22.2%)9
(34.6%)8
(57.1%)4
(21.1%)3
(16.7%)2
(18.2%)0
(0.0%)16
(22.5%)13
(25.5%)
32
(13.3%)1
(11.1%)7
(26.9%)2
(14.3%)1
(5.3%)6
(33.3%)5
(45.5%)2
(20.0%)15
(21.1%)11
(21.6%)
48
(53.3%)6
(66.7%)7
(26.9%)2
(14.3%)13
(68.4%)9
(50.0%)4
(36.4%)7
(70.0%)32
(45.1%)24
(47.1%)
53
(20.0%)0
(0.0%)1
(3.9%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)4
(5.6%)0
(0.0%)
159 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
How would you rate the timeliness of your organisation’s response to the disaster?
1= not timely at all
5= very timely
ScoreTHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE
DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison
10
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)
20
(0.0%)0
(0.0%)2
(7.7%)1
(7.1%)1
(5.3%)0
(0.00%)1
(9.1%)1
(10.0%)4
(5.6%)2
(3.9%)
30
(0.0%)0
(0.0%)6
(23.1%)1
(7.1%)4
(21.1%)6
(33.3%)4
(36.4%)1
(10.0%)14
(19.7%)8
(15.7%)
411
(73.3%)6
(66.7%)16
(61.5%)8
(57.1%)11
(57.9%)9
(50.0%)6
(54.6%)8
(80.0%)44
(62.0%)31
(60.8%)
57
(29.2%)3
(33.3%)2
(7.7%)4
(28.6%)3
(15.8%)3
(16.7%)0
(0.0%)0
(0.0%)9
(12.7%)10
(19.6%)
Table6.6:Perceivedcostlinessandtimelinessoforganisationalresponseactivities
Table6.7presentsfindingsrelatedtotheeffectoftheinstitutionalandpolicyenvironmentonthe speedof response.Again,most respondents confirmed that the institutional andpolicyenvironment influences the speed and costliness of their ownorganisation’s response.Thishighlightsthefactthatimprovedpreparednessalonemaynotbesufficienttoimproveefficiencyofresponse.Institutionalandpolicyissueswouldalsoneedtobeaddressedinordertoimproveresponse performance. This is important to consider during future project and programmedesignprocesses.
On a scale of 1-5, how much does the institutional and policy environment impact the speed of your response?
1 = no impact
5= very strong impact
SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE
DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison
11
(5.3%)0
(0.0%)1
(3.6%)1
(7.1%)0
(0.0%)0
(0.0%)0
(0.0%)1
(6.7%)2
(2.27%)2
(3.5%)
20
(0.0%)0
(0.0%)0
(0.0%)0
(0.0%)1
(5.0%)2
(11.1%)1
(4.8%)2
(13.3%)2
(2.27%)4
(7.0%)
33
(15.8%)2
(20.0%)9
(32.1%)3
(21.4%)7
(35.0%)4
(22.2%)9
(42.9%)6
(40.0)%28
(31.8%)15
(26.3%)
413
(68.4%)7
(70.0%)11
(39.3%)7
(50.0%)9
(45.0%)7
(38.9%)8
(38.0%)5
(33.3%)41
(46.6%)26
(45.6%)
52
(10.5%)1
(10.0%)7
(25.0%)3
(21.4%)3
(15.0%)5
(27.8%)3
(14.3%)1
(6.7%)15
(17.1%)10
(17.5%)
On a scale of 1-5, how much does the institutional and policy environment impact the cost of your response?
1 = no impact
5= very strong impact
SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLE
DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison
11
(5.3%)0
(0.0%)1
(3.6%)1
(7.1%)0
(0.0%)0
(0.0%)0
(0.0%)1
(6.7%)2
(2.4%)2
(3.5%)
160 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
20
(0.0%)1
(10.0%)0
(0.0%)0
(0.0%)0
(0.0%)2
(11.1%)4
(22.2%)3
(20.0%)4
(4.7%)6
(10.5%)
37
(36.8%)3
(30.0%)12
(42.9%)5
(35.7%)11
(55.0%)4
(22.2%)7
(38.9%)3
(20.0%)37
(43.5%)15
(26.3%)
410
(52.6%)4
(40.0%)10
(35.7%)5
(35.7%)7
(35.0%)8
(44.4%)7
(38.9%)8
(53.3%)34
(40.0%)25
(43.9%)
51
(5.3%)2
(20.0%)5
(17.9%)3
(21.4%)2
(10.0%)4
(22.2%)0
(0.0%)0
(0.0%)8
(9.4%)9
(15.8%)
Table6.7:Perceptionsabout the influenceof the institutional andpolicyenvironmenton speedandcostlinessofresponse
ThelackofsignificantdifferencesbetweenDEPPandcomparisongroupswithintheempiricalfindingswouldsuggestthattheresourcesinvestedinDEPPhavenotledtoaprogramme-wideimprovementinefficiencyofthehumanitarianresponse.However,itmustbenotedthatitmaywellbetoosoontomeasureimpact,andthesefindingsarethereforenotindicativeofpotentialimpact.Moreover, thesample size is relatively small and thisparticularanalysis isbasedonperceivedtimelinessandefficiencyofresponseratherthanactual,andperceptionsdatacanbebiased.ThefactthatsomequalitativedatasuggestsimprovedspeedofresponseprovidepreliminaryindicatorsthatDEPPhasgoodpotentialtoleadtoimprovementinefficiencyofthehumanitarianresponse.Ingeneral,someofthein-depthinterviewsalsoreiteratedthatitistoosoontomonitorresultsandimpact(e.g.,cDAc-n).
Cost Per Result Indicators
Asecondlineofenquirytoassesscosteffectivenessoftheprogrammeistoassessthecostper activity and cost per result achieved. Projectswere asked to provide expenditure databrokendownbyspecificactivities,toenableestimationoftheexpenditurethatwasincurredtoachieveemergencypreparednessandcapacity-relatedoutcomes.Projectswereaskedtobreakdowntheircostsaccordingtotheactivitieslistedbelow:
• Individualcapacitybuilding• Organisationalcapacitybuilding• Systems-levelcapacitybuilding• Individualpreparedness• Organisationalpreparedness• communitypreparedness
This enables an analysis of the actual cost to achieve changes in preparedness levels andcapacityindifferentcontexts.Theunitcostscanthenbecomparedovertime(iftime-seriesdataareavailable,whichisnotthecaseforthisevaluation),oracrosscountries.Thegoalofthecountrycomparisonistoassesswhichcountryoutcomescostmore,andtounderstandwhy.
ThemethodologyinvolveslinkingthefinancialdatawiththeoutcomesfromthewiderfindingsfromtheKAPandorganisationalresponsesurveys.Asdiscussedinchapter4ofthereport,there were few significant differences in the degree of change over time between DEPPandcomparisongroupsformostofthequantitativeindicators.becauseofthenatureofthehumanitariansystem,andthefactthattherewassomeexposuretocapacitybuildingwithinthecomparisongroup(thoughatalowerlevelthantheDEPPgroup),theremighthavebeensomeimprovementsinthecomparisongroupovertime.forthisreason,thereissomemeritinassessingthechangeinoutcomesfromT1andT2intheDEPPorganisationsonlyineachoftheintensivesetcountries,andthenestimatingthecostperunitchangefortheseoutcomes.Toconduct this typeofanalysis,a fullfinancialdataset isneededpercountrywhichhadastatisticallysignificantfinding.
Table6.8summarisesthefinancialdatawhichwereceivedfromprojects.Theevaluationteamacknowledgethatthiswasnotastraightforwardtaskforprojects,whichexplainswhysomeprojectswerenotabletoprovidethedataintime,giventheburdenofothertasks.
161 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
PROJECT
FINANCIAL DATA RECEIVED FROM PROJECTS
Costs fully disaggregated by each activity for all countries combined
Costs fully disaggregated by each activity and by country
PiP Yes Yes
ALERT Yes Yes
SEPS no no
LPRR yes no
Surge yes no
UEWEA yes no
STP Yes Yes
ADcAP Yes Yes
FE no no
TD no no
EWEA no no
PHEPGambella yes no
cDAc-n Yes Yes
Table6.8:Summaryoffinancialdataprovidedbyeachproject
Withoutafullfinancialdatasetthatmatcheduptoanyoftherelevantquestionsandcountries,missingdatawereestimatedfromprojectquarterlyfinancialreportsastheyprovidedfiguresforthecostsperthefiveDEPPresultsareasforeachproject.155
Table 6.9 below summarises the results with statistically significant findings from the KAPsurveys,andthecorrespondingcostperunitchange.Asthetableillustrates,thehighestVfMisforMyanmar,whichyieldedacostperpercentagepointincreaseinperceivedorganisationalpreparednessofonly£1,138comparedtothePhilippineswherethecostperpercentagepointincrease inthesame indicatorwas£71,287.QualitativedatasuggestthatSEPSwasabletoimplementactivitiesatalowercostduetothesmallernumberofconsortiapartnersandthefactthattheeconomiccostoflivingandoperatinginMyanmarislowerthanothercountries.Generally, it isnotadvisable tomakedirect comparisonsbetweencountriesbecauseof thedifferencesincostbaseswithincountries,andthemanyothercontextualfactorsthatcomeintoplay.KenyaalsohadaverylowunitcostcomparedtothePhilippines.Thismethodologywouldbemoremeaningfulifmoredatapointsovertimewereavailableand,inmorecountries,sothatcomparisonsovertimeandacrosscountriestounderstandreasonsforthefindingscouldbemade.
155 ThefiveDEPPresultsareasare:1)Improvedknowledgeandunderstandingofindividualsbysharingbestpracticeforhumanitarianpreparednessandresponse;2)Improvedpreparednesssystemsforearlyactionwithcommunitiesatriskofdisasters;3)Increasednumberofcoalitions,partnershipsandnetworkswhich,workingtogether,areabletoaddresshumanitarianneedsinawidevarietyofemergencysituations;4)Improvedinstitutionalarrangementsandpolicyenvironmentssothatnationalsystemsforhumanitarianresponseandpreparednessarebettersupportedandmoresustainable;and5)Strengthenedevidencebaseforwhatworkstohelpbuildhumanitariancapacityatscale.
162 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
INDICATOR WITH STATISTICALLY SIGNIFICANT
CHANGE BETWEEN T1 & T2 IN DEPP GROUP
COUNTRY
WHICH COST EFFECTIVENESS
INDICATOR IS RELEVANT?
DEPP
TOTAL COST INCURRED TO MAKE
REPORTED CHANGE
COST PER
1. Perceived Organisational Preparedness: Atwhatlevelwouldyourateyourorganisation’spreparednessresponsetodisastersand
emergencies?
MyanmarOrganisationalpreparedness
+8.9% £10,129 £1,138
2. Perceived Organisational Preparedness: Atwhatlevelwouldyourateyourorganisation’spreparednessresponsetodisastersand
emergencies?
ThePhilippinesOrganisationalpreparedness
+5.5% £392,078 £71,287
3. Perceived Individual Response – Performance
in Previous Response: Howwouldyourateyourperformanceduringyourlast
response?
ThePhilippinesIndividual
capacitybuilding+6.8%
£253,784
£37,321
4. Perceived Organisational Response – Performance
in Previous Response: Howwouldyourateyouroverallresponsetothedisaster?
KenyaOrganisationalcapacitybuilding
+15.8% £220,057 £13,928
5. Actual Organisational Preparedness: compositeindicatorassessingsevendomains:1-Emergency
preparednessplans,2-Humanresourcesandcapacity,3-logisticsandsupplies,4-legal,documentationandpolicy,5-Organisationandsystemprocesses,
6-Emergency responseteamandstaffing,7-communication
andmedia
KenyaOrganisationalcapacity
buildingandorganisationalpreparedness
+11% £258,670 £23,515
Table6.9:Summaryofcostperresultindicators
fortheindividualperformanceindicatorinthePhilippines(itispossibletomatchupthedataonthenumberofpeopletrainedinthePhilippinesbyDEPP,dividedbythetotalcost.Thenumberofpeopletrainedis8,404anddividingthisfigureby£253,784givesanannualunitcostfigureof£30.20.Itisdifficulttomakeavaluejudgmentonthisfigurebecausethereisnobasisforcomparison;however,giventhefactthatweseeasignificantimprovementinthesatisfactionrating,andthatfromourexperienceofotherprojects,£30.20isnotoutsidethereasonablerangeandappearstobeagoodVfMfinding.
163 SUMMATIVEPHASEREPORT-EffIcIEncyAnDVAlUEfORMOnEy
QUESTION 4.C: HAVE PROGRAMME FUNDS AND ACTIVITIES BEEN DELIVERED IN A TIMELY MANNER?
Use of Resources during Set-up and Lags Asexplainedatlengthintheformativephasereport,resourcesandtimenecessaryforsetupwereunderestimated,andtheprocessofsettingupandcontractingwasinefficient.ThiscontinuedtohaverepercussionsthroughoutthecourseoftheDEPP,astherewaslesstimeforimplementationgiventhattimewasusedforsetup.Seechapter4forfurtherdiscussiononimplementationdelaysandno-costextensions.Projectsattheoutsetdidnothavecleardecision-makingprocesses,whichcausedfurtherdelays.Therewaslackofstrongownershipofdecisionswhichcausedadditionaldelays.Greaterclaritywasneededonstructuresandaccountabilitywithintheprojects.Also,projectswereunclearwhatthefunctionoftheProgrammeboardwas,andsomefeltitcameacrossasan“Examboard,”ratherthanafunctiontohelpimprovetheprogramme.
Underspending due to Delays Thedelaysdescribedabovecontributedtoongoingunderspendingacrosstheprogramme.Table6.10showsthevariancebetweenbudgetandactualexpenditurefromaselectionofprojectsusingdatafromtheMarch2017quarterlyreports.Ascanbeseen,thereishighvarianceinthelevelofunderspendingrangingfrom73%tolessthan10%.
However,datasuggestthatafter18monthsthespendratepickedup,andsincemid-2017thevariancebetweenbudgetandexpenditurehasbeenmuchmorereasonable.Thevariancesatthetimecouldpartlybeexplainedbytimelagsinbigexpenditureitems(suchassoftwaresystemsorprocurementofpublicequipment)whichoccurredearlyon,butactualexpenditureislagged.Thishassincebeenharmonised.
PROJECTUNDERSPEND IN MARCH 2017 QUARTERLY REPORT
COMMENTS
UEWEA 73%
PHEPGambella 35%Askedforano-costextensiontoallowfor underspend.DelaysinsigningMOUswith
government,andrecruitmentwasachallenge.
SEPS 23%
EWEA 64%
STP 41% Grantsdelayed
FE 41%
TD lessthan10%
ADcAP lessthan10%
learningProject lessthan10%
Table 6.10: Variance in spending against budgets for selection of projects reported in March 2017QuarterlyReport
164 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
figure2:DEPPProjectDescriptions
SUSTAINABILITY & LIKELIHOOD OF IMPACT
cHAPTER7
165 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
Evaluation Question #5: To what extent and in what ways have the benefits of the programme become embedded?Sub-questions:a) Whatcontributionhastheprogrammemadeinstrengtheningnationalpreparednesssystems?b) Hastheprogrammetaken intoconsiderationprioritisedtargetgroups (peoplewithdisabilities,
older people, women, children and youth)? What contribution has the programme made instrengthening inclusionof targetgroupsandgenderaspectsat the levelofnational and localinstitutions?
c) InwhatwayshasDEPPinfluencedinstitutionalandpolicyenvironments?d) Whatisperceived(qualitativeanalysis)tobethemostsignificantchangeattributedtoDEPP,and
why?Data sources: • Minimum set IDIs with programme and project stakeholders (Qualitative)• KAP and organisational surveys (Quantitative)• Intensive set IDIs(Qualitative)• Document review(Qualitative)Key Findings:• Insomesettings,DEPP has contributed to strengthening national preparedness systemsbut
typicallyonasmallgeographicscale,andbyindividualprojectconsortia• Thelack of mainstreaming of inclusion of gender and prioritised groups across the programme
resulted in a fragmented programme wide approach and uneven results.Therewasnoevidenceof individual knowledge changeon inclusion; there was evidence of organisational change on inclusion, in particular in Ethiopia and among local organisations across the intensive set countries
• Monitoring of gender data improvedovertheprogramme;reporting on other prioritised groups remained low.
• TherehavebeenseveralconcreteexampleswhereDEPP has impacted government policies or systems or increased political commitment:o UEWEAproject’sadvocacyandsensitisationeffortsledtoanamendmentoftheDisaster
andEmergencyManagementAct(DEMAct),2015toincludefoodsecurity(Kenya)o lPRRproject demonstrated changes at the institutional level (christianAid adopted the
updatedresilienceframework)o financial Enablers’ cSO partnerswere able to secure seats in the national government
DisasterRiskReductionManagementcouncil(Philippines)• Therewasno statistically significant change in policy-related quantitative indicatorsinthefour
intensivesetcountries• changesinquantitativeindicatorsrelatedtolocalisationhavenotyetoccurredwithinthreeout
offourintensivesetcountries.In Ethiopia, however, the percentage of DEPP organisations with policies inclusive to L/NNGOs increasedby16%betweenT1andT2
• Attitudinal changes towards localisation have also occurred, including in how INGOs consider, address, and involve local actors and communities.ThisadoptionofthelocalisationapproachisconsideredtobeDEPP’smostsignificantchangebykeyinformants
• There is some evidence of benefits being embedded within organisations and systems. Thepotentialforlonger-termeffectsisdependentontheextenttowhichprogrammecomponentscancontinuebeyondtheendofDEPP
• Thelikelihoodofsustainabilitywasenhancedforprojectsorprojectcomponentsthatdemonstratedthefollowingcharacteristics:o builtonexistingworkandexistingpartnerships/consortiao Partneredwiththenationalandlocalgovernmento Had/developedapolicyoradvocacyelemento Workedtowardssystemslevelchangeo Paidgreaterattentiontoexitplansearlierintheprojectcycleo Developedtools,guidelines,orsystemsthatfillanimportantgapo Involvedbeneficiariesand/orlocalstakeholderso ExhibitedgoodVfMo builtlinkageswithotherentitiesandotherDEPPprojectso Implementedincontextsthatweremorefertileforchange
166 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
QUESTION 5.A: WHAT CONTRIBUTION HAS THE PROGRAMME MADE IN STRENGTHENING NATIONAL PREPAREDNESS SYSTEMS?
In a few settings, DEPP contributed to strengthening national preparedness systems though typically on a small geographic scale, and by individual project consortia rather than by DEPP as a whole.
Documentreview,logicalframeworkdataandstakeholderinterviewsindicatesomeevidenceofinfluenceoftheDEPPonnationalpreparednesssystems.forexample,theUrbanEarlyWarningEarlyActionsystemtodetecturbanemergenciesandenablerapidresponsehasbeenadoptedbythenairobicountygovernmentinKenya,who’vealsocommittedtoaddressingurbanfoodsecurityissues.ThePublicHealthPreparednessprojectinGambella,Ethiopia,workedwiththegovernment to improve disease and outbreak surveillance, leading to improved surveillanceandreportingatvariouslevelsofthehealthsysteminseveraldistrictsandinthestrengthenedcapacitytotestforpathogensattheregionallaboratory.InPakistan,managementofthesurgeplatformdevelopedundertheTransformingSurgecapacityprojecthasbeentakenoverbythenationalHumanitariannetwork(nHn),withanMOUinplacewiththelocalgovernmentbodytoensureimplementationaftertheprojectends.TheMarsabitcountyGovernmentinKenyaadopted linking Preparedness Resilience and Response’s conflict-sensitive approach duringhumanitarianresponse(includingduringthe2016-2017droughts)thusprovidingevidenceofDEPP’scontributionstonationalpreparednesssystems.Inothersettings,however,thedegreetowhich theDEPPhas influencednational preparedness systems and government disasterplanshasbeenlessclear.
These positive influences are important and represent significant investment and efforts.However,itshouldbenotedthatgenerallytheseeffectshavebeenonasmallergeographicscale–oftenatthedistrictorcountylevelandinmostcases,eachpositiveexampleofchangewasdirectlyattributabletoanindividualproject.bydesign,theDEPPprogrammedidnotimplementall14projectsineachofthe10focalcountriesandanindividualprojectmaynotnecessarilyhavebeenimplementedacrossanentirecountry.Inaddition,notallprojectstargetedsystems-levelchange.Infutureprogramming,moresystematicconsiderationtocumulativeresultswithinageographicscalearenecessarytomaximiseimpacts.
Theshort3-yearprogrammeperiodandthewaythatDEPPprojectswereunevenlyprogrammedinthe10countries,impededtheabilitytoachievesignificantsystem-widechanges.Influencinggovernment systems is a complex task that requires sustained and evolving contextualisedefforts at multiple levels, alongwith adequate time and flexibility. Interviewees noted thatwhilepartnershipswithgovernmentactorshavebeeninstrumentalininfluencingnationalandsystems-levelchange,buildingthesetypesofrelationshipscanbeaslowandlengthyprocessduetobureaucraticandadministrativechallengesincludingchanginggovernmentsandshiftingpriorities. Incertain fragileandconflict-affectedcontextssuchasSouthSudan,partnershipswiththegovernmentmaybeevenmorechallengingorpotentiallyposeriskstotheproject.DEPPstakeholdersinSouthSudandescribedthesechallenges(seechapter8).furthermore,duetodelaysincontracting,theexternalevaluationcouldonlybegindatacollectionaftersomeprojectshadalreadybeganimplementation.furthermore,itwasrequiredtocompleteitsscopeofworkataboutthesametimeDEPPprojectsclosed.ThisledtoashortertimeperiodbetweendatacollectionpointsandfurtherreducedtheabilitytocaptureanypotentialimpactDEPPmayhavehadatthesystemlevel.
furtherassessmentofDEPPeffectsoninstitutionalarrangementsandthepolicyenvironmentaredetailedlaterinthischapter.
QUESTION 5.B: HAS THE PROGRAMME TAKEN INTO CONSIDERATION PRIORITISED TARGET GROUPS (PEOPLE WITH DISABILITIES, OLDER PEOPLE, WOMEN, CHILDREN AND YOUTH)? WHAT CONTRIBUTION HAS THE PROGRAMME MADE IN STRENGTHENING INCLUSION OF TARGET GROUPS AND GENDER ASPECTS AT THE LEVEL OF NATIONAL AND LOCAL INSTITUTIONS? The lack of mainstreaming of inclusion of gender and prioritised groups across the programme resulted in a fragmented programme wide inclusion approach and uneven results. While there was no evidence of individual knowledge change, there was evidence of organisational change with respect to inclusion, in particular in Ethiopia and among local organisations across the intensive set countries.
167 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
While the DEPP business case underscored the disproportionate impact of disasters andhumanitarianemergenciesonwomeninparticular,andemphasisedthattheprogrammewouldstrategicallyaddressinclusion–atleastinregardtogenderandViolenceAgainstWomenandGirls(VAWG)–suchafocuswaslargelyabsentfromtheprogramme.Thebusinesscasestatedthatthemanagementteamwouldberesponsiblefor“ensuringpriorityareasincludinggender,violence against women and girls…are addressed throughout the programme managementcycleprocess”,andthat“projectswillbeencouragedtobuildthecapacityoftheirpartnersonViolenceAgainstWomenandGirlsandthiswillbeakeycriteriaforthedesignedcomponenttodeliveragainst.”Whilethismayhavebeentheintention,onlyoneofthe14projects(ProtectioninPractice)focusedonprotectionandtosomedegreeVAWG.cDAc-n’sprojectinSouthSudanandlinkingPreparednessResilienceandResponse’sworkinMyanmaralsohadsmallVAWG-focusedelements.Itisnoteworthythatthebusinesscasedidnotarticulatewellinclusionofother prioritised target groups including peoplewith disabilities, older people, children andyouthorhighlightthisapriorityareafortheDEPP.
Thisabsenceofdefinitionsanddetailsinthebusinesscaseandsubsequentlackofemphasisoninclusionofprioritisedgroupsduringtheconceptualisationoftheprogrammehadseveraldownstreameffects.first,whiletheprojectselectioncriteriadidrequiregenderconsiderationstobeoutlinedinprojectproposals,projectswerenotmandatedtoprovidedetailedinclusionplans.Withoutdefinitionsandclearguidance,differingapproaches to inclusionwere incorporatedinprojectdesignsdrivenmainlybyprojectororganisational leadership.Thisyieldedvaryingdegreestowhichinclusionwasintegratedintoprojectactivitiesandultimatelyafragmentedprogrammewideapproach.ViolenceagainstWomenandGirlswasnottakenonasakeycross-cuttingthemeandassuchtheprogrammedidnotdeliverwellagainstany intendedVAWGcriteria.
Acrosstheprogramme,inadditiontoProtectioninPractice’sfocusonprotection,theAgeandDisabilitycapacityProgrammefocusedoninclusionoftheelderlyandpeoplewithdisabilities.bothof theseprojectsmadestrides tobuildprotectionand inclusioncapacityand in somecasesinfluencedtheworkofotherprojects.ADcAPinparticularstronglychampionedinclusioneffortsincludingatprogramme-leveleventsandhelpedshapeprogramme-leveldialogueonthisissue.However,theextenttowhichprioritygroups(women,children,theelderly,andpeoplewith disabilities and other vulnerable/marginalised groups)were holistically included acrosstheprogrammeand inclusionefforts adequatelymonitoredwereminimal.Arguably, amoreintegratedapproachwhereinclusionandprotectionweremainstreamedacrosstheprogrammeratherthanthespecificfocusofoneortwoprojectsmighthaveyieldedbetterresults.
Intermsofmonitoringandreporting,previousDEPPevaluationreportshighlightedaconsistentlackofdisaggregateddataandminimallydocumentedgenderconsiderationsacrossprojects.Recommendationstostrengthengenderreportingwithinprojectdataweretakenupaftertheformativephaseevaluationreport.Therewassomeevidenceofcorrectiveactionsleadingtoan increase indatadisaggregationbygenderover the courseof theprogramme.However,reportingonotherprioritisedgroupsremainedlow,makingitdifficulttoassessthetrueextenttowhichthesegroupshavebeenreachedbycapacitybuildingorotheractivities.
168 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
KEY INCLUSION INDICATOR DATA COLLECTED AT TWO TIMEPOINTSIndividual level:
• Perceived knowledge on inclusion: KAP survey with humanitarian staff. How knowledgeable do you feel you are regarding [Age related issues in a disaster; Disability related issues in a disaster; Issues related to women in a disaster]? *
• Perceived knowledge on VAWG: KAP survey with humanitarian staff. How knowledgeable do you feel you are regarding Gender-based Violence? *
Organisational level:
• Perceived inclusion during preparedness: Organisational survey with senior leadership. Were members of any of the following groups involved in the design and implementation of the preparedness activities/programming? [Women, the elderly, people with disabilities]
• Perceived inclusion during response: Organisational survey with senior leadership. How well did this response take into account the needs of vulnerable groups including the elderly, wom-en and people with disabilities? [Note this indicator was only measured at T2.]*
• Inclusion policy: KAP survey with humanitarian staff and organisational survey with senior leadership. Does your organisation/country office have a policy about inclusion of vulnerable groups such as women, children, people with disabilities, the elderly?
Community level:
• Inclusion in community preparedness plans: Household survey. On a scale of 1-5, tell me whether you agree or disagree, with the following statements: a) Community preparedness plans and actions have taken into account the needs of women, b) Community preparedness plans and actions have taken into account the needs of the elderly, c) Community preparedness plans and actions have taken into account the needs of the people with disabilities.*
*Indicatorsweremeasuredbasedona5-PointlikertScalewhere1=Poor,2=fair,3=Good,4=Verygood,5=Excellent.
Analysis of intensive set data collected in the four intensive set countries (Kenya, Ethiopia,Myanmar and the Philippines) provides additional evidence on how programme activitiesactuallyincorporatedandinfluencedinclusionattheindividual,organisationalandcommunitylevel.Theboxabovedescribesthespecificinclusion-relatedindicatorsthatwereassessedatT1andT2amongbothDEPPandcomparisongroupsandanalysedforchangeovertime.
Perceivedknowledgechangeswithrespecttoinclusion(alsopresentedinchapter4)andVAWGispresented infigure7.1.Ascanbeseen, therewasnoperceivedchange in self-reportedknowledgelevelsacrossanyofthefourindicatorsamongrandomlyselectedhumanitarianstaffatDEPPorganisations,comparisonorganisationsoroverallasaresultoftheDEPP.Inaddition,therewerenosignificantchangesinanyoftheintensivesetcountriesorwhenthesamplewasrestrictedonlytolocalorganisations.ThisfindingsuggeststhatDEPPdidnotimpactindividual-levelknowledgeoninclusion.
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Age Related Issues in a Disaster +1 . 0 % +1 . 0 % 0 . 0 %
Disability Related Issues in a Disaster + 0 . 6% + 0 . 5% +1 .1%
Issues Related to Women in a Disaster + 0 . 8 % - 0 . 5% +1 . 3 %
Gender Based Violence +1 . 2 % -1 . 4% +2 . 6%
figure7.1:PerceivedindividualknowledgerelatedtoinclusionandGender-basedviolence* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant
169 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
At the organisational level, using the pooled data from all four countries, there were nosignificantchangesintheinclusionofprioritisedgroupsinpreparednessprogrammingintheDEPP group or in the comparison group (See figure 7.2).However,DEPPorganisations inEthiopiashowedastatisticallysignificantincreaseininclusionoftheelderlyinpreparednessprogramming(+26.8%,p<0.05),andininclusionofpeoplewithdisabilities(+30.9%,p<0.05)betweenT1andT2(figure7.3).TherewasnosimilarincreaseinthecomparisonorganisationsinEthiopia.TherewasinfactastatisticallysignificantnetDEPPeffectforinclusionofpeoplewithdisabilities(+52.2%,p<0.05).Thisispositive,butsomewhatsurprisingfindingasEthiopiawasnotoneofthecountrieswheretheAgeandDisabilitycapacityProgrammewasdirectlyactive.InKenyawhereADcAPinclusionadvisorswereactive,therewasnosignificantchangeindisabilityinclusionduringpreparednessactivitiesinDEPPorganisationsoroverallasaresultoftheDEPP.ImprovementsinEthiopiacouldhavebeenduetoprogrammeleadership’sincreasingemphasis on the inclusionof prioritised groups as best practice, disseminationofADcAP’sMinimumStandardsforAgeandDisabilityInclusioninHumanitarianActionthroughtheirowneffortsand/orthroughglobalDEPPevents,ortheinfluenceoftheShiftingthePowerprojectwhichincludedsomeinclusionefforts,oracombinationofthese.
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Women involved in design and implementation of preparedness programming
+ 0 .7 % + 6 .7 % - 6 . 0 %
Elderly involved in design and implementation of preparedness planning
0 . 0 % + 0 . 3 % - 0 . 3 %
People with disabilities involved in design and implementation of preparedness planning
- 0 . 2 % - 0 .9 % + 0 . 8 %
figure7.2:Perceivedinclusionofprioritisedgroupsduringdesignandimplementationofpreparednessprogramming* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.
ADEPP
DIFFERENCE (T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Ethiopia +28 . 8 %* + 6 . 5% +2 0 . 3 %
Kenya - 8 .1% +14 . 3 % -2 2 . 4%
The Philippines -1 . 4% -2 . 3 % + 0 .9 %
Myanmar -11 . 2 % -23 .9 % +1 2 . 6%
Local +1 .7 % 0 . 0 % +1 .7 %
170 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
BDEPP
DIFFERENCE (T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Ethiopia +3 0 .9 %* -21 . 2 % +52 . 2 %*
Kenya -<11 .7 % + 4 . 8 % -16 . 4%
The Philippines -5 . 6% +18 . 8 % -24 . 4%
Myanmar -7. 5% -7.7 % + 0 . 2 %
Local - 0 . 8 % -5 .9 % +5 .1%
figure7.3. Perceived inclusionof the elderly (PanelA) andpeoplewithdisabilities (Panelb) duringdesignandimplementationofpreparednessprogrammingbycountry.* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.
Inclusionofvulnerablegroupsduring responseactivitieswasassessedonlyatT2,and thusaDifference-in-Differenceanalysis isnotpossible.T2data is insteadanalysedforstatisticalsignificanceusingchisquaretests.Table7.1presentsthefindings.Overall,therearenostatisticallysignificantdifferencesbetweenDEPPandcomparisonorganisationsinanycountryorinthefullsample.However,Ethiopiadatashowatrendofincreasedinclusionduringresponseactivities(70%inEthiopiaDEPPorganisationsreportedverygoodorexcellentcomparedto61.1%inthecomparisonorganisations).This trend is consistentwith thesignificant improvements ininclusionduringpreparednessinEthiopiadescribedabove.
How well did this response take into account the needs of vulnerable groups including the elderly, women and people with disabilities?
SCORETHE PHILIPPINES KENYA ETHIOPIA MYANMAR FULL SAMPLEDEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison DEPP Comparison
Poor 0(0.0%)
0(0.0%)
0(0.0%)
0(0.0%)
0(0.0%)
0(0.0%)
0(0.0%)
0(0.0%)
0(0.0%)
0(0.0%)
Fair 1 (5.9%)
1 (11.1%)
2 (7.7%)
0(0.0%)
3(15.0%)
2 (11.1%)
3(27.3%)
4(36.4%)
9(8.8%)
7(13.2%)
Good 7(41.2%)
2 (22.2%)
10(38.5%)
5(33.3%)
3(15.0%)
5(27.8%)
5(45.5%)
5(45.5%)
37(36.3%)
17(32.1%)
Very Good
7(41.2%)
4(44.4%)
12 (46.2%)
7(46.7%)
13(65.0%)
8 (44.4%)
3(27.3%)
2 (18.2)%
49(48.0%)
21 (40.0%)
Excellent2
(11.8%)2
(22.2%)2
(7.7%)3
(20.0%)1
(5.0%)3
(16.7%)0
(0.0%)0
(0.0%)7
(6.86%)8
(15.1%)
Table7.1:PerceivedinclusionofprioritisedgroupsduringhumanitarianresponseatT2* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.
Ethiopiaalsoshowspositivefindingswithrespecttochangesinorganisationalpolicyontheinclusionofprioritisedgroups.Asshown infigure7.4, therewasanon-significant increaseof3.5%ofDEPPorganisationsinEthiopiawithinclusionpoliciesrelatedtoprioritisedgroupsbetweenT1andT2.However,overallcomparedtothecomparisongroup,therewasasignificantnetDEPPeffect of+9.9% (p<0.05).This alignswellwith theprevious findings forEthiopiaandwhentakentogetherdemonstrateconsistently,acrossaseriesofindicators,evidenceofimprovedinclusionattheorganisationallevelinEthiopiathatcanbeattributedtotheDEPP.Withrespecttoorganisationalpolicy,strongpositivefindingsarealsoevidentwhenthesampleisrestrictedtolocalorganisations(Seefigure7.4).AmongalllocalDEPPorganisations,therewasastatisticallysignificantincreaseof+6.5%(p<0.05)inorganisationswithpoliciesonthe
171 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
inclusionofprioritisedgroups,andthis translated intoastatisticallysignificantoverallDEPPeffectof+14.0%(p<0.05).ThisisanotablefindingdemonstratingtheDEPPeffortsculminatedinstrengthenedinclusionpoliciesinlocalorganisationsacrosstheprogramme.
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Full Sample +3 .1% -2 .7 % +5 . 8 %
Ethiopia +3 . 5% - 6 . 4%* +9.9 %*
Kenya - 0 .7 % + 4 . 2 % -3 . 4%
The Philippines +1 .9 % + 0 . 8 % +1 .1%
Myanmar -1 2 . 2 % -7.9 % +2 0 .1%
Local + 6 . 5%* -7. 5% +14 . 0 %*
figure7.4:Organisationswithpoliciesontheinclusionofprioritisedgroups* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.
Inclusionwithincommunitieswasalsoanalysed.Atthecommunitylevel,householddatafromMyanmarwasusedtoconstructindicatorsontheperceivedinclusionofwomen,theelderlyandpeoplewithdisabilitieswithincommunitypreparednessplans.Asillustratedinfigure7.5,perceptionsonthelevelofinclusivenessofcommunitypreparednessplansactuallysignificantlydecreasedbetweenT2andT1inbothDEPPandnon-DEPPcomparisoncommunities.TherewasnostatisticallysignificantoveralleffectoftheDEPP.Thesedeclinesinperceivedinclusionmayhavebeenlinkedtochangesincontextualfactorssuchaspoliticalclimate,insecurityorotherfactors,orthetendencyofpeopletojudgethemselvesortheircommunitiesmorerealisticallyonceexposedtocapacitybuildingorotheractivities.SinceperceptionsoninclusiondecreasedstronglyinbothDEPPandnon-DEPPcomparisoncommunitiesatroughlythesamelevel,thereisnoevidencethatDEPPimprovedinclusionwithincommunitypreparednessplans.
172 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Myanmar: Inclusion of women in community preparedness plans
- 6 . 2 %* -5 . 0 %* -1 . 2 %
Myanmar: Inclusion of the elderly in community preparedness plans
-5 . 4%* - 4 . 4%* -1 .1%
Myanmar: Inclusion of people with disabilities in community preparedness plans
- 6 . 0 %* - 4 . 8 %* -1 . 2 %
figure7.5:Perceivedinclusionofprioritisedgroupsincommunitypreparednessplans* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.
Overall,thelackofintegrationofactivitiestopromoteinclusionacrosstheprogrammeresultedinafragmentedprogrammewideapproachandunevenresults.Whiletherewasnoevidenceofindividualknowledgechange,therewasevidenceoforganisationalchange,inparticularinEthiopia and among local organisations across the intensive countries.Whilemonitoring ofgenderdataimprovedoverthecourseoftheprogramme,reportingonotherprioritisedgroupsremainedlow.
QUESTION 5.C: IN WHAT WAYS HAS DEPP INFLUENCED INSTITUTIONAL AND POLICY ENVIRONMENTS?
There have been several concrete examples where DEPP has impacted government policies or systems or increased political commitment but there was no statistically significant change across the DEPP in policy-related quantitative indicators.
Institutionalandpolicyenvironmentchangeswerecapturedattheprogrammelevelthroughlogicalframeworkdataandthroughindicatordataprovidedtotheevaluationteambyprojects.Thiswastriangulatedwithdocumentsreviewedaspartofthedocumentreview.Accordingtologicalframeworkdata,anumberofprojectshavecreatedpolicybriefs,recommendationsandreports to influencepolicyon a rangeof topics suchas collaborativeplatforms, localisationofaidandageanddisability inclusion.Engagementwiththegovernment, theUnandotherinfluential actors alongwith advocacy eventswere also undertaken as strategies to createchangeininstitutionalandpolicyenvironments.
Thereareseveralexampleswheretheseeffortshaveimpactedgovernmentpoliciesorsystemsorincreasedpoliticalcommitment.forexample,theUrbanEarlyWarningEarlyActionproject’sadvocacy and sensitisation efforts led to an amendment of the Disaster and EmergencyManagementAct (DEMAct), 2015 to include food security and anMOUwith thenairobicitycountyGovernmenttoensurethegovernment’srole inaddressingurbanfoodsecurityissue.156 ThelinkingPreparednessResilienceandResponseprojectdemonstratedchangesattheinstitutionallevel.forexample,christianAidadoptedtheupdatedresilienceframework,157
andaViolencetoPeacestrategy.158financialEnablers’reportedexampleswherecSOpartnerswereable tosecureseats in thenationalgovernmentDisasterRiskReductionManagementcouncilsandtechnicalworkinggroupsprovidingopportunitiesforengagementandcollaboration.ThecDAcnetworkProjectadvocatedfor inclusionofcommunicatingwithcommunities inthebangladeshGovernment’snationalStandingOrdersonDisasters(SOD),whichiscurrentlyunderrevision(seeSection9).ThefinalversionoftheSODhasyettobereleasedbutisexpected
156 UrbanEarlyWarningEarlyActionfinalEvaluationReport.157 https://www.christianaid.org.uk/sites/default/files/2017-05/resilience-framework.pdf.158 https://www.christianaid.org.uk/resources/about-us/tackling-violence-building-peace-global-strate-
gy-2016.
173 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
toincludethedesiredchanges.
InordertofurtherexamineinfluenceoftheDEPPoninstitutionalandpolicyenvironments,anumberofindicatorscollectedwithinthefourintensivesetcountrieswereanalysed(seeboxbelow).
KEY POLICY INDICATOR DATA COLLECTED AT TWO TIMEPOINTSIndividual level:
• Perceived individual influence within organisation: How would you rate YOUR ability to influence the institutional and policy environment within your organisation? Where 1=No Voice or Influence, 2=Minimal Influence, 3=Some Influence, 4=Significant Influence, 5=Strong Influence, seen as equal partner
Organisational level:
• Perceived influence of organisation on institutional and policy environment: Organisational survey with senior leadership. How would you rate your organisation’s ability to influence the institutional and policy environment? Where 1=No Voice or Influence, 2=Minimal Influence, 3=Some Influence, 4=Significant Influence, 5=Strong Influence, seen as equal partner
• Perceived influence of organisation on national preparedness systems: Organisational survey with senior leadership. How well is your organisation or country office able to influence gov-ernment policies related to national preparedness systems [in country of survey]? Where 1=No Voice or Influence, 2=Minimal Influence, 3=Some Influence, 4=Significant Influence, 5=Strong Influence, seen as equal partner
• Inclusiveness of organisational policy towards L/NNGOs: Organisational survey with senior leadership. On a scale of 1 to 5, how inclusive is your organisation’s policy / approach to work-ing with local / national NGOs? Where 1=Very Exclusive, 2=Exclusive, 3=Somewhat Inclusive 4=Inclusive, 5=Very inclusive
• Change in organisational policy on working with L/NNGOs: Has there been any change in your organisation’s policy and/or approach to working with local or national NGOs in the last 1 year? Where 1=More inclusive, 2= No change, 3=Less inclusive
The indicators focus on individual influence within one’s own organisation, as well as theorganisation’s influence on the policy environment in general or specifically related tonationalDisasterManagementPlans.Toassesschangesrelatedtolocalisationpolicy,dataoninclusivenessoforganisationalpoliciestowardsl/nnGOsaswellaschangesinthesepolicieswerecollectedandanalysed.
DEPP DIFFERENCE
(T2 - T1)
CONTROL DIFFERENCE
(T2 - T1)NET DEPP EFFECT
Individual Influence Within Organisation +1 . 6% -2 . 2 % +3 . 8 %
Organisational Influence on Institutional & Policy Environment
- 4 .7 % +1 .1% -5 . 8 %
Organisational Influence on Policies related to National Preparedness Systems
+2 . 4% +3 .1% - 0 . 6%
174 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Organisational Practice with Respect to Government Disaster Management Plans
+1 . 4% +1 . 8 % - 0 . 4%
Inclusiveness of Organisational Policy to L/NNGOs
+1 . 3 % +1 . 4% - 0 .1%
Change in Organisational Policy on working with L/NNGOs
- 0 . 6% + 0 . 2 % - 0 .9 %
figure7.6:changeinIndividualandorganisationalpolicy-relatedindicators
* Denotes statistical significance. Absence of * indicates that none of the measured changes were statistically significant.
Table7.5providesthefindingsacrossallcountries.Ascanbeseen,despitesomeofthepositivepolicy changeexamples thathaveemerged from thedocument reviewprocesses, therearenoobserved significant changes in anyof the six policy-related indicators overtime.Theseindicatorsmeasurebothperceptions,aswellasthelevelofinfluencewithinacomplexpolicyenvironment.The lattermaytakesignificanttimetochangeandmaynotfullycaptureothertypesofpolicychangeorshiftsinthelandscapethataremoredifficulttomeasurequantitatively.
Changes in quantitative indicators related to localisation have not yet occurred except in Ethiopia where the percentage of DEPP organisations with policies inclusive to L/NNGOs increased by 16% between T1 and T2.
Ontheissueoflocalisation,anareathathasbeenastrongfocusoftheDEPPsinceitsconception,qualitativedatacollectedaspartoftheevaluationandduringevidencevalidationworkshopsineachofthefour intensivesetcountriessuggestslowbutsteadychanges inattitudesandmindset favouring amore localised approach.The speed andextentof this typeof changeseemed to depend to some degree on the specific contextual setting, political climate andotherexternalfactors.InKenya,forexample,severalrespondentsdescribedananti-localisationeffectoccurringasInGOssetuplocalbranchofficesandindoingsopushedoutl/nnGOs.Therepercussionsofthishavealreadybeenfeltasonel/nnGOdescribedtheirannualbudgetdropping by 90% due to the influx of InGOs into local areas. Despite this,more positiveinstancesofchangesinpowerdynamics,greateraccessofl/nnGOstoleadershippositionsandfundinghavebeendocumentedinseveralcontexts.Theseareexploredinmoredetailinthenextsection.
The question of whether this perceived shift in mindset and attitudes contributed to anymeaningfulchangeintermsoforganisationalpolicyornumberofpartnershipswithl/nnGOswasalsoassessed.Withrespectto inclusivenessoforganisationalpoliciestol/nnGOsandchange in policies related toworkingwith l/nnGOs (figure7.5), therewas no statisticallysignificantchangeovertimeduetotheDEPPinthepooleddataorinanyofthecountry-leveldataexcept forEthiopia.Respondents fromDEPPorganisations inEthiopia reporteda16%increaseinthelevelofinclusivenessoftheirorganisation’spolicestowardl/nnGOs,andthiswassignificantatthe0.05level.However,duetosmallincreasesininclusivenessofEthiopiancomparisonorganisations, thenetDEPPeffectwasnotstatisticallysignificant inEthiopia. Inothersettings,thereisnoevidencethatlocalisation-relatedpolicieshavechangedasofyet.
Inaddition,furtherdatarelatedto localisationarepresented inchapter5.networkanalysisshowsnochangeovertimeintheproportionoflocal/nationalnGOswithinthehumanitariannetworksintheintensivesetcountriesandlimitedchangeintermsofthenumberofrelationshipswithl/nnGOs.Thosefindingsareconsistentwiththefindingabovethatorganisationalpolicyrelatedtol/nnGOshasnotchanged.
Takentogetherthesefindingssuggestthatwhilechange inquantitative indicatorsrelatedtolocalisationhavenotyetoccurred(exceptinEthiopia),therehasbeensomeprogresstowardsqualitativeattitudinalchangethatisnecessarytoeventuallyleadtomoresystem-levelchanges.
175 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
Itisnotclearhowclosewearetoatippingpointwherechangesoccurandcanbestatisticallydetected,butmomentumisgrowing,especiallyincertaincontexts,anditiscriticalthatthesegainsnotbelost.PerceivedqualitativechangearoundlocalisationwillbefurtherdiscussedinthenextsectionandsustainabilityandleveragingDEPP’sinvestmentsintolonger-termchangewillbeexploredlaterinthischapter.
QUESTION 5.D: WHAT IS PERCEIVED TO BE THE MOST SIGNIFICANT CHANGE ATTRIBUTED TO DEPP, AND WHY? Qualitative data captured attitudinal change towards the localisation approach including in how INGOs consider, address and involve local actors and communities, which is considered to be DEPP’s most significant change.
consistently, project and programme level stakeholders identified the adoption of the localisation approachas themostsignificantchangeattributable to theDEPP.Stakeholdersnoted observing a shift in the way InGOs consider, address and involve local actors andcommunities,andattributedittotheDEPP.forexample,oneintervieweestated:
“[DEPP]haspushedthelocalisationagendaatsuperspeedandit’sreallyhappening.Probablyalloftheelementswerethere,butasasystemtheyreallyneededjustatinypushtomakeithappen.TheDEPPallowedthat.”
Theoften-citedtendencytoconsiderlocalorganisationsassub-contractorsorimplementers,rather than as decisionmakers or equal partnerswas also perceived as changing in certaincountries.Anin-countryrespondentinPakistandescribedthischange:
“The most significant change among international organisations, I believe, is therealisation that local organisations are not only the implementing agent but theyarepartnersandtheyneedtobetreatedaspartners.ThisissomethingthatIhaveobservedinthebehaviour,thechangethatIhaveobservedinthementororganisation…revisiting theirpartnershipmodalitiesandprocess throughwhich theycankindofreducethesubvertingapproach.Thatkindofapproachisnowdiscouragedandnewmodels are coming in by these international organisations.before this, itwas justkindofadonorandreceiverkindofrelationship,likeagrantpartnership.butnowit’sbeyondthat.Internationalorganisationsarecommittingtoinvestingmorewhilethelocalactorsarehavingalong-termengagementwiththem.”
Otherexamplesofthistypeofchangeweredescribedinbangladeshwherelocalactorsbecameinvolved indecision-makingprocesseswithin a largenationalpreparednessplatform, and inthePhilippineswherelocalorganisationswerethefirsttomobiliseandrespondtothecrisisinMarawi.Thesetypesofshiftscontributedtoincreasedconfidenceamonglocalorganisationsandnotonlybecameanopportunityforexperientiallearningbutalsoprovidedmomentumtothecause.Additionally,theopeningoftheUnPooledfundtol/nnGOs(forexample,ShiftingthePowerlocalpartnersinDRcandPakistanbecameeligibleforthesefunds)andthegreateraccessofl/nnGOstotheInter-AgencyStandingcommitteeclustersystemsignallednotableprogresstowardsthelocalisationagendawithinahumanitarianarchitecturetypicallyknownforitsslowpaceofchange.
Itwas noted that further progress in this domain towards a concrete “re-balancing” of thehumanitariansystemwouldrequireenhancedunderstandingofthebarrierstothelocalisationapproachindifferentcontexts.forexample,inEthiopia,wheretherearegovernmentrestrictionson civil society organisations, advocacy effortsmay need to take a different shape than inthePhilippineswhere civil society is engrained in the cultural fabric andprovided spacebythegovernment.nevertheless, acrossprogrammecountries,DEPPstakeholdersagreed thatideologically, a localisation approach is critical to improving preparedness and humanitarianresponseandthatincrementalchangesarebeginningtooccurontheground.Theseviewsandtheobservedchangesthathavebeendocumentedabovealignwellwiththeglobaldiscoursearound localisationofhumanitarian aidwhichhas gainedground, especially since the2016WorldHumanitarian Summit. To ensure further progress beyond the end of theDEPPwillrequiresustainedeffortstobuilduponthechangesthathavebeguntoemerge.
176 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
CONCLUSION: TO WHAT EXTENT AND IN WHAT WAYS HAVE THE BENEFITS OF THE PROGRAMME BECOME EMBEDDED?
There is some evidence of embedded benefits and the potential for some longer-term effects but this is dependent on the extent to which different programme components are able to obtain support to continue beyond the end of DEPP.
DEPP’sthree-yearprogrammecycletoestablishandimplement14projectsin10countrieswithdifferingdisasterprofilesthroughaconsortia-focusedapproachendsinJune2018;allprojectsexceptthelearningProjectcompletedtheiractivitiesbytheendofMarch2018.Toassesstheextenttowhichbenefitsoftheprogrammehavebecomeembeddedandhavethepotentialtoachievelonger-termeffects,documents,projectendevaluationsandqualitativedatawereanalysedacrossseveralareas:sustainability/exitplanningoverthecourseoftheDEPP,takeupof sustainability-focused recommendations, the sustainability of programme/project outputsandoutcomes,andlikelihoodoffutureimpact.Itisimportanttonotethatmostprojectshadapproximately twoyearsor less to implement their activitiesdue to approval of projects inwavesaswellasthedelaysinstart-up.Thisprovidedlimitedtimetosupportsustainability.
Sustainability / Exit Planning
Theinterimreporthighlightedanumberofweaknessesrelatedtosustainabilityplanningattheprojectandprogrammelevels.Individualprojectswererequiredtoprovidesustainabilityplansintheirproposals;however,thesegenerallywerenotverydetailedwithrespecttotheprojects’sustainabilitymodels,didnot includesufficient informationregardingcostsofsustainingtheproject, potential revenue sources, integration intocommunitiesor local systemsor specificactionstobetakenforhandover,andinmostcasestheywerenotfurtherdevelopedthroughoutthecourseoftheproject.Anoverallprogramme-levelsustainabilityplanwasnotadequatelyoutlinedduringthedesignoftheprogramme,whichwasaconsiderableweakness.basedonstakeholder interviews, therewas little direction provided from the programme level to theprojectsontheissueofsustainabilitywhichhasbeenproblematicforprojectsandcontributedtouncertainty.
The lack of adequate sustainability planningmay have also been influenced by the explicitdescription of a contingent allocation of £60million for a future phase of the programmeby DfID in the business case: “An extension to the programme, possibly for 2 years, willbe considered subject to the findings of amid-term review supported by the independentevaluation(in2015).ThepreviousSecretaryofState(AndrewMitchell)hasgivenin-principleagreementforupto£100millionover5years.”159Despitethisstatement,furtherclarificationonwhensuchadecisionwouldbemadeortheprocesswasnotmadetransparenttoDEPPstakeholdersoverthecourseoftheprogramme,butneverthelesssethighexpectations.Thisreportedly influencedhow intensivelyDEPP stakeholders pursuedother funding sources aswellaslikelydelayedthedevelopmentofanexitplan.forexample,onerespondentstated:
“Ithink[sustainability]hasbeenachallengebecauseIthinkifpeopleknewthat[morefundingfromDfID]wasn’tanoptiontheycouldhavethoughtfromtheverybeginninghowtostrategicallyplanforthat.”
In October 2017, DfID announced that it would not be funding a second phase of theprogramme.This announcement cameonlyfivemonthsbefore theprogrammeclose ratherthanat themid-pointof theprogrammeas laidout in thebusinesscase.At this late stage,littletimewasleftforboththeoverallprogrammemanagementteamandindividualprojectstoseekalternativesourcesoffunding.However,fromariskmanagementperspective,DEPPprogrammemanagement and leadership shouldhaveattempted to raise alternative fundingordiversified its fundingsourcesandstrengthenedotherprocessesofsustainabilitysuchasembeddingDEPPwithinlocalsystemsearlieronintheprogrammecycle.
The interimphase evaluation report provided strong recommendations formore deliberate,strategicsustainabilityplanningandtheneedforguidanceonsustainabilityplanningandopendiscussion initiatedwith projects about the availability of continued funding to extend the
159 businesscaseInterventionSummary:DisastersandEmergenciesPreparednessProgramme,DfID.
177 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
DEPP.currentdatashowsthattheserecommendationsweretakenonboardandanumberofprojectshavemadeprogresswithrespecttosustainabilityplanning.Allprojectendevaluationsandfinalprojectnarrativereportsalsoincludedasustainabilitysection.Inaddition,theSTARTnetworkrecruitedaconsultanttoevaluateoptionsforcarryingtheDEPPforward includingfundingoptions.
Current status of exit planning
Atthetimeofthisreport,anumberofprojectsdemonstratedsubstantialprogressintermsofexitplanning.forexample,theUrbanEarlyWarningEarlyActionproject’ssurveillancesystemhasbeenadoptedbythenairobicountygovernment,andmanagementofthePakistanSurgerosterhasbeentakenoverbythenationalHumanitariannetwork(nHn),withanMOUinplacewiththelocalgovernmentbodytoensureimplementationaftertheprojectends.Inaddition,local consortia that received support from the financial Enablers project havemade planstocontinuetocollaborate.Ontheotherhand,otherprojects facedisruptiondueto lackofsustainedfundingandlossordispersionofprojectstaff.Thismaylimitpotentialforsustaininganypositiveeffectsandbuildingoncurrentaccomplishments.There isalso thepotential tolosesomeofthegainsthatweremadebytheDEPP.Staffmemberswhocontinuewithinthesameorganisations,togetherwiththelearningproductsgeneratedacrosstheDEPPandbythelearningProject,willconstitutethelargestpartofDEPP’s institutionalmemory.WhilesomeprojectsandprojectcomponentsshowindicationsofbecomingembeddedandmaycontinueonaftertheDEPPinsomeform,itisclear,thattheprogrammeasawholewillnotcontinueinitscurrentconfigurationandwouldneedtoberemodelledforanyfuturephases.
Sustainability of programme/project outputs and outcomes
Atthetimeoftheprogrammeclose,therehavebeenpositivesignsofemergingsystem-levelchanges, including shifts in attitudes towards localisation and changes in organisational andgovernmental policies as reported earlier in this chapter, aswell as strengthenednetworks.Mostintervieweesexpressedthatmoretimewasneededtoembedthesechanges.Datafromproject end evaluations, narrative reports and in-depth interviewswere analysed to furtherassesstheextenttowhichDEPPoutputsandoutcomeshavebecomeembedded.
Sustainable Outputs – Achievements and Contributions to the Four Output Areas
Attheoutputlevel,anumberofachievementsoftheprojectscontributetothesustainabilityanddemonstratethatbenefitshavebecomeembedded.
Collaboration
collaboration and partnerships are important contributors to achieving sustainability.RelationshipsbuiltthroughcollaborationandconsortiawereperceivedtobeoneofthemostenduringelementsoftheDEPP,howeverthesewerecontingentonthestructureandset-upofconsortiaandalsowereinfluencedbyfactorssuchasstrongleadership,politicalwill,competingallegiances, and availability of funding. for example, the financial Enabler’s final evaluationsuggeststhatseveraloftheconsortiafundedbytheprojectinthePhilippineswilllikelybreakdownorstruggletocontinuetoworktogetherduetoreasonsincludingincompatibleplansandlackofstrongcommitmentamongstleadershiptoprioritisefuturecollectiveaction160.Oneoftheconsortiareportedlystoodoutashavingamuchhigherlikelihoodofsustainabilityduetoasharedcommitmenttolocation,priorexperienceofjointaction,complementarycompetencies,demonstratedlearningandcommittedleadership161.Stakeholdersalsoemphasisedthatbuildingsustainablecollaborationandpartnershipswaschallengingtoaccomplishwithintheprovidedtimeframe.forexample,oneintervieweestated:
“Idon’tthinkwhatwearetryingtoachieve,itcouldbeachievedinthistimeframe.I thinkbecause it’snotonlywithdeliveringprojectactivitiesanddone, itsactuallydoingmorethandeliveringactivitiesbecauseit’spairedwithchangingorganisationalculture,changingorganisationalbehaviour,bringingchangeinthemindsofdisasterpractitioners,soIthinkitrequiresmoretimethanit’sdedicatedfor.”
160 financialEnablersfinalEvaluationReport.161 Ibid.
178 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Capacity Building
buildingcapacityhasaninherentpotentialforsustainabilityatalllevelsofaction(i.e.,individual,organisational, community capacity). However, these capacities need to be maintainedor reinforced or risk being lost. likelihood of sustainability is higher among capacities builtbeyondknowledgechangesuchasthosethat involveskillbuildingandbehaviourchangeattheindividuallevel.Ifretained,improvedskillsandbehaviourscancontinuetoeffectchangeandcanalsobetransferredtoothersbeyondthelifeoftheproject.Attheorganisationallevel,improvedcapacityintheformofchangedpoliciesorgovernanceorputtinginplacesystems(humanresources,finance,procurement)tooperationalisecapacitybuildingwerealsonotedtocontributetoincreasedsustainability162.Itwasalsonotedthatsustainabilityattheorganisationallevelwouldbedependenton internal approvalof thenewstrategiesandpoliciesand theirimplementation,butthatthiswouldbeinfluencedbythenature,sizeofoperationsandscaleof humanitarian activities of the organisations.163Therewere also examples of Shifting thePowerbeneficiaryorganisationsinPakistanwithimprovedgovernance,whoarenoweligibleforadditionalfundingbecauseoftheseimprovedinternalsystems–demonstratinghowimprovedgovernancecanhelptoensurefuturesustainability.
Generating and Sharing Learning
Thegenerationoflearning,anotherkeyDEPPfocusarea,alsocontributedtowardsthegoalofsustainability.ThedevelopmentandcreationofDEPPprojectinfrastructuresuchasprotocols,tools,proceduresandguidelinesaresustainableinthattheywillcontinuetoexistaftertheendoftheprojectperiod.forexample,theAgeandDisabilitycapacityProgramme’stheGoodpracticeguide;embeddinginclusionofolderpeopleandpeoplewithdisabilitiesinhumanitarianaction,TransformingSurgecapacity’sThefutureofHumanitarianSurge,andProtectioninPractice’sGPcProtectionMainstreamingToolkit,areexamplesofsomeofthetools, learningproductsandguidesthatwillhaveacontinuedeffectaftertheDEPPends.notably,anumberoftheseareavailableinmultiplelanguagesandthroughdifferentchannelsincreasingtheirpotentialforcontinueduse.forexample,theProtectioninPracticeGPcProtectionMainstreamingToolkitisavailableinUrdu,bengali,burmese,french,Arabicandavailablethroughawebsiteandasamobileapp through theGoogleplayand iTunesAppStore.Toensure that these learningproducts and tools becomemore embedded into the humanitarian sectorwill also requiresomedegreeofcontinuedadvocacy,awarenessbuildingandpromotionaleffortswhichmaybedependentonavailabilityofadditionalfundsandorganisationalcommitment.AstheleadofthelearningProject,AcfhasmadethecommitmenttokeepthedigitallearningplatformforanadditionaltwoyearsaftertheclosingofDEPPandwillcontinueoutreachanddisseminationefforts.
Early Warning System Development
functioningearlywarningsystemsalsoincreasethepotentialofprogrammesustainabilityastheyarelikelytocontinueaftertheprogrammeperiod.However,sustainingthesesystemsisdependentonsecuringorganisationalandgovernmentbuy-in,ensuringtheadoptionanduseofthesystems,andhavingfundingearmarkedformaintenance.Severaloftheearlywarningprojects,includingtheUrbanEarlyWarningEarlyAction,ImprovedEarlyWarning–EarlyActionstoStrengthenDisasterPreparednessinEthiopiaandPublicHealthPreparednessinGambella,haveahigherlikelihoodofsystemschangeandhavebecomeembeddedwithinlocalandnationalstructureswhichwillensurelonger-termsustainability.Thesethreeprojectshaveincommontheapproachofworkingwithinestablishedsystemsandstructuresandinvolvingkeyexternalstakeholders such as the local governments and local health systems.These three projectshave confirmed sustainability plans with Kenyan and Ethiopian governments, respectively,whohavebegunincorporatingprojectactivitiesintotheirworkplansandbudgets.WithsomeinfluencefromtheUrbanEarlyWarningEarlyActionproject,theKenyangovernmentpassedlegislation requiringcounties toestablishemergencyanddisastercommitteesat thecountyandsub-county level.Thishashelpedtoensuresustainability forprojectactivitiesafter theDEPPfundingcyclefinishes.InEthiopia,theMinistryofHealthandRegionalHealthbureaus
162 ShiftingthePowerfinalEvaluationReport.163 Ibid.
179 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
providedexitstrategiesandguidelinestoensuresustainabilityofactivities.Thesethreeprojectsestablishedconcrete, feasiblestrategiestoensure long-termsustainabilityandcanthereforeprovide important lessonsfortheotherprojects intheportfolio.However, it is importanttonotethatunlessfundingissecuredtoensurecontinuedfunctioningofthesystems,thereisapotentialriskthattheimpactoftheinvestmentinthesesystemswillbediminishedorlost.
Table7.2providesexamplesofachievementsandcontributionstothefouroutputareasfromthefinalprojectnarrativereports,finalevaluationsandinterviewsthatarelikelytobesustainedbeyondDEPPwithsufficientmaintenanceandfollow-up.
PROJECT EXAMPLES OF ACHIEVEMENT/CONTRIBUTION OUTPUT AREA
ADCAP
• Developed and finalised version of the humanitarian inclusion standards for older people and people with disabilities, which are available to agencies throughout the world in various accessible formats and languages
• Developed complementary training resources and guidance on good practice relating to organisational change
Capacity building; Learning
ALERT• Created ALERT early warning system platform
• Contributed to creation or revision of organisational preparedness plans
Early Warning System Development; Capacity building
CDAC-N
• Created message libraries for use during disasters and emergencies
• Developed toolkits and resources for two-way communication with disaster-affected communities
• Established multi-stakeholder platforms
• Strengthened capacities to support two-way communication
Capacity building; Collaboration; Learning
EWEA
• Established early warning system in Ethiopia
• Strengthened capacity of district (woreda) experts and the community
• Developed/strengthened community contingency plans
Capacity Building; Early Warning System Development
FE
• Strengthened local coordination mechanisms and networks
• Empowered and strengthened confidence and experience of local actors to respond to disaster
• Strengthened technical competencies of partners
• CSO partners were able to secure seats in the national government Disaster Risk Reduction Management councils
Capacity Building; Collaboration
LPRR
• Changed attitudes on community involvement in resilience
• Kenya Marsabit County Disaster Risk Management draft Bill has recognised the role of local organisations in coordination of emergency/disaster-related events
• Tools and strategies were integrated by INGOs and LNGOs
• Platform of NGOs and peace dialogue at village level was established
• Integration of LPRR’s conflict-sensitive approach into Marsabit County Government response
Capacity Building; Collaboration
180 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
PHEP Gambella
• Provided equipment to health facilities
• Trained existing health staff to implement government strategy
• Improved surveillance and detection of disease and outbreaks
• Developed toolkits
Early Warning System Development; Capacity building
PIP
• Local partners attending Oxfam’s Global Protection Peer Group meeting shared learning with international staff and other local partners
• Developed GBP Protection Mainstreaming Toolkit
• Developed infrastructure to support protection initiatives
• Contributed to changes in organisational policy
Capacity building; Learning
SEPS• Established early warning systems at community level
• Supported the development of community preparedness plans
Early Warning System Development; Capacity building
STP
• Contributed to changes in organisational policies, systems and capacities
• Supported development of Emergency Preparedness Plans
• Set up NGO platforms/networks in five countries
Capacity building; Collaboration
TD
• Online toolkits and resources in local languages were developed
• Strengthened partner relationships
• Trained trainers
Capacity Building; Collaboration; Learning
TSC
• Organisational systems developed
• Open-access training, tools, and resources were created
• Funding secured for future use of the Philippines and Regional rosters
• Pakistan roster has been taken over by the National Humanitarian Network (NHN), with an MOU with government body to continue collaboration
Capacity building; Learning
UEWEA
• Developed and tested an urban surveillance system that has been adopted by the Nairobi county government
• Contributed to an amendment of the Disaster and Emergency Management Act (DEM Act), 2015 to include food security and a MOU with the Nairobi City County Government to ensure the government’s role in addressing urban food security issues
Early warning system development
Table7.2:contributionsofprojectstowardssustainableoutputs
Sustainable Outcomes
At the outcome level, sustainable changeswere more difficult to detect due to the shortprogramme time frame.As described in chapters 4 and 7, there have been no significantimprovements in knowledge on core humanitarian competencies of humanitarian staff, onself-reportedknowledgeondisasterpreparedness,responsetodisastersandemergenciesorongender,ageordisability-related issues indisasters.conversely, therehavebeennotableimprovements in organisational and community preparedness among DEPP beneficiaries,
181 SUMMATIVEPHASEREPORT-SUSTAInAbIlITyAnDlIKElIHOODOfIMPAcT
especiallyamonglocalorganisations.QualitativedatahavealsodemonstratedthecontributionofDEPPbeneficiariestowardsimprovedresponseinseveraloftheintensivesetcountries(thePhilippines,KenyaandEthiopia).
The extent towhich these outcomes are sustainable remains uncertain at the time of theprogramme’sclose.Sustainingpreparednesslevelsandresponsecapacitiesaredependentonmaintainingskills,knowledge,andcompetenciesattheindividual,organisationalandcommunitylevels.Thesecapacitiesarealsoaffectedbyshiftsinhumanresourcesandstaffturnover,whichwere reported as consistent and recurring issues throughout the programme. furthermore,increasedcapacity also leads to thepotential for individuals tobe recruitedelsewhere, andlocalorganisationsthatparticipatedinDEPPcapacitybuildingactivitiesrisklosingkeystafftoInGOs.Aslongastheseresourcesremaininthedisasterpreparednesssystem,however,theywillbesustainabledespitethetransferandexchangeofresources.
Additionally, preparedness and response are influenced by many factors external to DEPPincludingorganisationalpolicy,leadership,andculture,andchallengesengaginggovernment.Inmostcases,thecontributionsthatDEPPhasmadeinimprovingoutcomesaroundpreparednessandresponsewilllargelybedependentonsecuringadditionalfundstocontinueprojectactivities,maintainsystems,infrastructures,andcapacitiesbuilt,andensurequickresponsestodisasters.
The Role of Community and Beneficiary Stakeholders in Design and the Impact on Sustainability
Asoutlinedinchapter3,beneficiariesatalllevelsaswellaskeylocalstakeholderswerenotadequately involved in thedesign stage formostprojects.While thishadsignificanteffectsontherelevanceandpotentialeffectivenessoftheselectedinterventions,andledtodelaysin implementation as projects had to be contextualised, it also is a threat to the long-termsustainability of the projects and in turn the programme. Overall, beneficiary engagementand feedbackmechanisms to thecommunity shouldbeprioritisedasa strategy to increasesustainability.Infuturesuchprogrammes,beneficiariesshouldbeinvolvedinthedesignphaseinordertoachievehigherlikelihoodofsustainability.Acrosstheboard,interviewsandsitevisitsdemonstrated that using a bottom-up approach of supporting organisations and individualstoleadtheirowncapacitybuildingprocessesinfluencesthelikelihoodofbuildingsustainablecapacities.
Programme Visibility and Sustainability
Programme visibility, often built through strategic communication about a programme,contributes to community, stakeholder and government level support and is an importantconsideration with respect to sustainability. Site visit observations by the evaluation teamandintervieweereportsindicatedthatbothinternal and externalprogrammevisibilityatthecountry levelwere low. Internalvisibility,orawarenessof theDEPPby in-country staffandconsortiamembers involved inDEPPprojects increased through theDEPP learning events,butremainedlimited.Externalvisibility,orawarenessoftheDEPPbynon-DEPPstakeholderswasevenloweratthecountrylevel.WhilebothtypesofstakeholderswereawareofspecificDEPPprojects,mostwereunawareofthelargerDEPPportfoliointheirowncountrysuggestingstrategiccommunicationabouttheportfoliomayhavebeensuboptimal.
AsmanylocalDEPPprojectstaffdidnotidentifythemselvesasbeingpartoftheDEPP,orworkjointlywiththeotherin-countryprojectstowardsthecommongoalsandobjectivesoftheDEPP,itisunsurprisingthattherewereminimaljointadvocacyorcommunicationefforts.Whilemostprojectstaffdidfeelastrongsenseofconnectiontotheirownprojectsandownorganisations,thelackofconnectiontotheDEPPbrandandthelimitedfocusonoverallprogrammeobjectivesaffected longer-termprogramme sustainability through reduced community and stakeholderdialogueandprogramme-levelbuy-in.Ultimatelytherewaslessintegrationoftheprogrammeinto relevant structures inDEPP countries.At the regional or global level, one stakeholderdescribedlimitedawarenessandunderstandingoftheDEPPatimportantexternalevents,suchastheWorldHumanitarianSummit.Anumberofrespondentsalsosuggestedthat increasedadvocacyinitiativesandpromotionoftheDEPPwereneededtoraisethelocalprofileoftheprogramme.
182 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Innovation, Sustainability and Contingency Planning
TheDEPPportfoliowasenvisionedtoincludeamixofflagshipprojectsthatemployedmoretraditionalstrategiesandapproachesaswellasothermoreinnovativeandsometimesriskierprojectswithahigherpotentialtofail.Elementsoftheportfoliothatwereparticularlyinnovativeinclude:
• flexiblefundingmechanisms(i.e.,fE,PIP,cDAc-n,lPRR)• Urbanearlywarningsystemstoenabledetectionandresponseofemergenciesin
urbansettingsincludingrelatedtofoodsecurity(i.e.,UEWEA)• Regionalpooledsurgeplatforms(i.e.,TSc)
Incaseswhereinnovationmeanshigherriskoffailure,sustainabilityplanningdoesnotmakesenseandisnotagooduseofresourcesuntilresultscanbedemonstrated.Theseshouldbetreatedaspilotprojectsandthereforeshouldnotberequiredtoestablishsustainabilityplanningandconsiderationsduringthedesignphase.Insteadtheseprojectsshouldberequiredtohavestrongmonitoringandevaluationplansinordertoinformcoursecorrectionandpotentialscale-up.Inaddition,allprojects,regardlessoflevelofrisk,shouldhaveappropriatecontingencyplansinplacetonavigateexpectedorunexpectedcontextualfactorssuchasdisruptionofprojectactivitiesduetodisastersandemergencies.Therewasatleastoneexamplewheresuchplanswerenotinplace(cDAc-nprojectinSouthSudan),resultingincriticaleffectsontheprojectwhenadisasteroccurred.Therewereseveralotherexamples,albeitonasmallerscale,whereprojectactivitieswereaffectedbyemergencies(i.e.,reportingandlearningActivitiesrelatedtocDAc-ninbangladeshwerelimitedduetofocusonRohingyacrisis164,andbothdroughtandinsecurityhinderedEWEAprojectimplementationinEthiopia165).
Likelihood of Impact and Summary
Overall,moreattentiontosustainabilitywasneededearlierinthecourseoftheprogramme,withamoreholisticviewonsustainabilitygoingbeyondsimplysecuringfundingstreams.Alongerperiodforprogrammeimplementationmayhaveledtogreaterchangesinoutcomesandincreasedthelikelihoodofprogrammesustainability.Thedatasuggestthatlikelihoodofsustainabilitywasenhancedforprojectsorprojectcomponentsthatdemonstratedthefollowingcharacteristics:
• builtonexistingwork(i.e.,UEWEA,TD),andexistingpartnershipsandconsortia(i.e.,STPbangladeshandnationalAllianceofHumanitarianActorsbangladesh(nAHAb);cDAc-ninthePhilippineswherethecWcgroupsareembeddedwithinpre-existingstructures)
• Workedwithgovernmentand/orhadapolicyoradvocacyelement(i.e.,lPRRinKenya,PHEPGambellainEthiopia)
• Workedtowardssystems-levelchange(i.e.,UEWEA)• Paidgreaterattentiontoexitplanningearlierinprogrammecycle(i.e.,UEWEA)• Developedtools,guidelines,orsystemsthatfillanimportantgap(i.e.,ADcAP’s
minimumstandards,PIP’sprotectionmainstreamingtoolkit)• Involvedbeneficiariesand/orlocalstakeholders(i.e.,UEWEA)• ExhibitedgoodVfM(i.e.,cDAc-nfoundationaltrainingcalled“unsustainably
expensive”suggestinglowVfMandchallengesbringingthiscomponenttoscale)• builtlinkageswithotherentities(i.e.,STARTfund)andotherDEPPprojects(i.e.,the
Philippines)• Implementedincontextsthatweremorefertileforchange(i.e.,thePhilippines)
164 cDAc-nfinalEvaluationReport.165 EarlyWarningEarlyActionfinalEvaluationReport.
183 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
figure2:DEPPProjectDescriptions
HUMANITARIAN PREPAREDNESS AND RESPONSE CASE STUDIES
cHAPTER8
184 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Evaluation Questions: What are the impacts of violence on project implementation and lessons learned on implementing emergency preparedness and response programming in a conflict-affected setting? How has disaster preparedness translated into actual response? Sub-questions:a) InwhatwayshaveDEPPcapacitybuildingprogrammesstrengthenedpreparednessandresponse
capacityamongstparticipants?b) To what extent is DEPP contributing to greater preparedness and response among local
organisationsandcommunities?c) HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,how?Ifnot,whynot?d) HasDEPPledtoimprovedknowledgeandunderstandingofbestpracticesrelatingtodisasterand
emergencypreparednessandresponse?Ifyes,inwhatways?e) Whatwas the role of government and/or pre-existing systems?How did these facilitate the
responses?Orhowdidalackoftheseinhibitresponse?Method and data sources: Twocasestudieswereconducted.caseStudy#1examinedtheimpactofescalationofviolenceinSouthSudanonDEPPprojectimplementationandcaseStudy#2assessedthe relationship between emergency preparedness and humanitarian response in Ethiopia andKenya.Thefollowingdataformedthefindingsofthetwocasestudies:• Documentreview(Qualitative)• Semi-Structuredinterviews(Qualitative)• SiteVisits(Kenyaonly,sitevisitsinSouthSudanandEthiopiawerenotpossibleduetosecurityissues)• Datacollectedduringtheminimumsetdatacollection(Qualitative)Key Findings:Case Study #1onSouthSudandemonstratedthatoperatinginaconflict-affectedcontextrequiresprocedures that allow for flexible and adaptive management, rather than traditional and rigidcontractualrelationships.Thefindingsemphasisetheimportanceofhavingclearagreementsaboutdutyof carewhencreatingmulti-lateral agreements amongmultiple institutions in a consortiumand the responsibilities of care for the host agency in relation to consortiumpartners. findingsalso highlight the need to have a contingency plan including budget contingencies in case ofdisruptiontotheproject.finally, thecasestudycitedchallengesofoperating inconflictsettingswhereengagementwithgovernmentstakeholdersisnotalwayspossibleandthatcapacitybuildingapproachesandmaterialsmayneedmoreadaptationsforconflictsettings.Case study #2,onEthiopiaandKenya,demonstratedthattheconsortiummodelandcollaborationwithgovernmentwerecriticaltosuccessfulimplementationofearlywarningsystemsandpermittedcoordinated humanitarian responses that improvedwith each subsequent emergency in Kenya.Availabilityof immediate response funds andpreviously agreedupon response actionswere keytoensuringmorerapidresponses.InEthiopia,self-reporteddatasuggestthatexperientiallearningrelatedtofundraising,resourcemobilisationandimplementationofhumanitarianresponseactivities,alongwithaccess toflexibleresponse fundingmechanismscontributedto responsesbysevenl/nnGObeneficiariesoftheDEPPtoaddressthedrought.Keythemesthatemergedincluded:• communityandgovernmentengagementwereessentialtosuccessfulprojectdesignandimplementation• communityandorganisationalcapacitybuildingstrengthenedpreparednessandresponsethrough
improved awareness of humanitarian standards, coordination, and procedures, and increasedvisibilityoftheorganisationsandthecredibilityoftheiremergencypreparednessandresponseactivitiestodonorsandotherstakeholders
• collaborationimprovedcommunicationandefficiencyofresourcesused• Surveillanceandearlywarningsystemswerefundamentaltobothprojects(KenyaandEthiopia),
andweredescribedasthepillarsofemergencypreparednessandresponse.beyondcollectingdata, respondentsnoted that these also servedasmechanismsof collaborationandprovidedopportunitiesforbehaviouralchangeforcommunitiesandorganisations.Definitionsofdisastersdifferedbetweenurbanand rural contexts.Povertyand food insecurityare rarelyconsidereddisastersandgetvariedlevelsofattentionfromdonorsandstakeholders.
• EngagementwithDEPPprovidedopportunitiesfororganisationstosharelearningandbestpractices• Individualconfidencetorespondtoadisasterincreased;however,therewerefewexamplesof
behaviourchange• Organisationalpoliciesandprocedureswerestrengthened,andcollaborationwiththegovernment
inKenyaallowedforsystems-levelchanges,includingalignmentandintegrationofprojectactivitieswithinbroaderplatforms,suchasconstitutionalprioritiesandnationalpolicies
• Theconnectionbetweenemergencypreparednessandresponseactivitiesweredisjointedandneedtobecloselylinked
185 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
INTRODUCTION AND RATIONALE FOR THE CASE STUDIES: SOUTH SUDAN AND KENYA AND ETHIOPIA In-depth case studieswere added to the evaluationmethodologyduring the formative andinterim phases to provide further perspectives on emergency preparedness and responseactivities in the programme, and to assesswhether improved preparedness translated intomoreefficientandtimelyhumanitarianresponse.SimulationexerciseswereinitiallyproposedduringtheinceptionphaseasamethodtomeasureDEPPbeneficiaries’performanceduringhumanitarian response activities in a controlled manner. The intentionwas to collect suchdataduringsimulationsthatwereplannedbyDEPPprojectsthemselves.However,overthecourseofthethree-yearprogramme,projectschangedtheirplansregardingsimulations,anditwasalsonotedthatseveraloftheDEPPprojectsrespondedtoactualdisastersintheprojectcountries.Thisprovidedanopportunitytodirectlyassesscontributionstoactualhumanitarianresponses. After reflection and discussion with the learning Project, the evaluation teamdecided to conduct two in-depth case studies.The first case studywas focused on SouthSudantocaptureexperiencesandperspectivesofprojectstakeholdersrelatedtotheJuly2016escalationofviolenceandhowthisimpactedprojectimplementation.Thesecondcasestudyinvolved assessing humanitarian responses in Ethiopia and Kenya during the course of theDEPPtocapturetherelationshipbetweenpreparedness,inparticularearlywarningsystems,andresponseefforts.
SouthSudanwasselectedasoneofthefivecountrieswhereintensivedatacollectionwouldtakeplaceduringtheevaluation.Datacollection,however,waspostponedduetoincreasedviolenceinJuly2016,whichsignificantlyaffected theDEPPprojectsoperating in thatcountry.AfterdiscussionswiththetwoprojectsthatwereoperatinginSouthSudan(cDAc-nandProtectioninPractice),andmonitoringofthesecuritysituationinSouthSudan,adecisionwasmadeinOctober2016todiscontinuethein-countrydatacollectionplannedforSouthSudan.Instead,theevaluationteamplannedtodevelopacasestudy,toexplorehowthesetwoprojectswereaffectedbytheviolenceandthelessonsthatcanbeextractedfromthisexperience.Thiscasestudywillbepresentedfirstasitwillprovideinsighthowacrisiscanaffectprojectoperations.
Thesecondcasestudyfocusedonpreparednessandresponseeffortsintwoselectedcountries:Kenya(UrbanEarlyWarningEarlyActionproject)andEthiopia(ShiftingthePowerproject).Aspartoftheexternalevaluation’sdocumentreviewprocess,a listwasgeneratedofcountriesinwhichatleastoneresponsetoadisasteroremergencywasnotedinDEPPprojectreportsupuntilendof2017.forthecasestudy,thecountriesinwhichintensivedatacollectionwasconductedaspartoftheexternalevaluationwereconsidered(Kenya,Ethiopia,thePhilippines,andMyanmar).ThePhilippineswasexcludedasacountryoffocusgivenacasestudyhadbeenrecentlyconductedbythelearningProject.Myanmarwasexcludedgivenongoinginsecurity.Responseefforts inKenyaandEthiopiawerethereforeselectedtobethefocusofthecasestudy.
Thetwocasestudiesaimedtoexplorethefollowingevaluationquestions:
• What are the impacts of violence on project implementation and lessons learnedon implementing emergency preparedness and response programming in a conflict-affectedsetting?(SouthSudan)
• InwhatwayswereDEPP’sdesignandcorecomponents(capacitybuilding,collaboration,earlywarningsystems,andlearning)effectiveorineffectiveinstrengtheningemergencyresponseinDEPPtargetcountries?(EthiopiaandKenya)
• InwhatwayshaveDEPPcapacitybuildingprogrammesstrengthenedpreparednessandresponsecapacityamongstparticipants?(EthiopiaandKenya)
186 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
• TowhatextentisDEPPcontributingtogreaterpreparednessandresponseamonglocalorganisationsandcommunities?(EthiopiaandKenya)
• HaslocalcapacitytorespondtodisasterschangedsincethestartofDEPP?Ifyes,howhasitchanged?Ifnot,whynot?(EthiopiaandKenya)
• HasDEPPledtoimprovedknowledgeandunderstandingofbestpracticesrelatingtodisasterandemergencypreparednessandresponse?Ifyes,inwhatways?(EthiopiaandKenya)
• Whatwastheroleofgovernmentand/orpre-existingsystems?Howdidthesefacilitatetheresponses?Orhowdidalackoftheseinhibitresponse?(EthiopiaandKenya)
187 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
CASE STUDY #1: EMERGENCY PREPAREDNESS AND RESPONSE IN SOUTH SUDAN BACKGROUND
following South Sudan’s independence in 2011, political struggles led to renewed conflictin2013166.Within thecontextofongoingconflict,violenceescalated inJuly2016betweengovernment and opposition forces in Juba, affecting both civilians and aid workers167,168.Deepeninganalreadycomplexhumanitariancrisis,theviolenceresultedindisplacementandfamine,withanestimatedsixmillionSouthSudanese inneedofhumanitarianassistancebyAugust2016169.Thehumanitariansituationcontinuedtodeteriorate,andatthetimeofthiscasestudyin2017,therewasongoingviolenceinotherstatesofSouthSudantargetingbothciviliansandaidworkers170.TheviolenceandinsecurityaffectedtwoDEPPprojectsinSouthSudan:1) thecDAcnetworkcommunicatingwithcommunities (cWc)project and2) theSTARTnetworkProtectioninPracticeproject171,172.
The cWc project was delivered as part of a consortium amongWorld Vision, Internews,bbcMediaAction and national and local organisations, and aimed to improve emergencypreparednessby increasing informationexchangeandcommunityengagement173.ThecWcproject supportednovel, experimental ideas, andmodels to increase information sharing toimproveresponseandenablethosemostaffectedtoinformtheresponse.Theprojecthadfivedifferentcomponents:1)aflexiblefundingmechanismwherebylocalorganisationsappliedforfundsthroughthebroaderprojects,2)acWcinEmergencyWorkingGroupofagenciesco-chairedbytheUnitednationsOfficeforthecoordinationofHumanitarianAffairs(OcHA),3)cWctrainings,4)exchangevisitsandnetworking,and5)asimulation174.PreliminaryproposalsformajorworkstreamswerepresentedattheMarch2016meetingoftheSouthSudancWcworkinggroup.AsofJuly2016,theworkinggrouphadbeenestablished,afoundationtraininghadbeenconducted,andaspartoftheflexiblefundingmechanism,fivepotentialsub-projectswereidentifiedrightbeforetheviolenceescalatedinJuba.
TheProtectioninPracticeprojectwasdeliveredaspartofaconsortiumamongOxfam,WorldVision,andInternationalRescuecommittee,andaimedto increaseprotectionexpertiseandcapacity innational-levelorganisations175.Theprojecthadthree interlinkedaims:1) tobuildgreater capacity for protection mainstreaming among national humanitarian staff, nGOs,
166 GuardianT.AttackonaidworkersinSouthSudan:‘Therewasincrediblenaivety’2016[cited201714April].Availablefrom:https://www.theguardian.com/global-development-professionals-network/2016/oct/17/attack-on-aid-workers-in-south-sudan-there-was-incredible-naivety.
167 centerforciviliansinconflict.Underfire:TheJuly2016ViolenceinJubaandUnResponse.UnitedStates:centerforciviliansinconflict,2016.
168 GuardianT.‘Untolddevastation’inSouthSudantriggersgravehumanitariancrisis:TheGuardian;2016[cited201714April].Availablefrom:https://www.theguardian.com/global-development/2016/aug/09/untold-devastation-in-south-sudan-triggers-grave-humanitarian-crisis-un.
169 Unnewscentre.Afterattacks,UnseniorofficialinSouthSudanurgesprotectionofaidworkersandcivilians2016[updated8April2017;cited201715April].Availablefrom:http://www.un.org/apps/news/story.asp?newsID=56528-.WPIcD7uGPfy.
170 cDAcnetwork.communicatingwithcommunitiesinSouthSudan2016[updated11March2016;cited201714April].Availablefrom:http://www.cdacnetwork.org/i/20160311131929-0c4ru.
171 STARTnetwork.ProtectioninPractice[cited201714April2017].Availablefrom:https://startnetwork.org/start-engage/protection-in-practice.
172 RogueAmoeba.Piezo.2016.173 STARTnetwork.ProtectioninPractice[cited201714April2017].Availablefrom:https://startnetwork.
org/start-engage/protection-in-practice.174 Ibid.175 RogueAmoeba.Piezo.2016.
188 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
andcommunity-basedorganisations(cbOs);2)tohelpfivenationalnGOsorcbOsbecomeprotectionspecialistsbyassistingthemtoimplementprotectionactivitiesandcoordinatewithprotectionactors;and3)tocreateaneffectiveevidencebaseofwhatworksinpracticeandsharethiswithotherprojectsandinitiatives.AsofJuly2016,theProtectioninPracticeprojecthad implementedthefirstoutputonprotectionmainstreaming.Workshopsandtrainingsonmainstreamprotectionhadbeenconductedand localorganisationshadcreatedtheiractionplanstoidentifyandaddressprotectiongaps.Inaddition,localorganisationshadbeenidentifiedandselectedtoreceivefunding.
Theviolence inJubadirectly impacted thestaffandpartnersof thecWcproject176,177.Theviolence also impacted project implementation and activities for both the cWc projectandProtection inPracticeproject.Thiscasestudy,conductedaspartof theDEPPexternalevaluation,aimstoexplorehowviolenceimpactedprojectimplementationandlessonslearnedon implementingemergencypreparedness and responseprogramming in a conflict-affectedsetting.
FINDINGS FROM THE CASE STUDY
Intotal,fourindividualswereinvitedtoparticipateinaninterviewandthreeindividualsagreedtoparticipate.Demographicdatawerenotcollectedduringthecasestudyinterviewstorespecttheprivacyofindividualsinterviewed.
Keythemesemergedfromtheinterviewsrelatedtothefollowingbroadcategories:1)impactsofviolenceandinsecurityonprojectimplementation;2)impactsonprojectmanagementanddecision-makingprocesses;3)impactsonprojectmulti-stakeholderplatformsandcollaborativepartnershipswithnationalandlocalorganisations;4)emergencypreparednessandresponseinaconflict-affectedsetting;and5)lessonslearned.Withinthosecategories,multiplesub-themesemergedasdescribedbelow.Directquotesfromtheinterviewsarenot includedinordertoprotecttheprivacyofrespondents.
1.1. Impact of Violence and Insecurity on Project Implementation: Project Activities, Management, and Staffing
beyondthedirectimpactonprojectstaffbasedinJuba,theviolenceimpactedprojectactivities,projectmanagement, and the budget in differentways between the two projects in SouthSudan.
AsaresultoftheviolenceinJuba,thedecisionwasmadetotemporarilysuspendthecWcproject. In the immediate period following the violence, international staff members wereevacuated fromSouthSudan.Decisions related to staffprotectionweredeterminedby thein-countryhostorganisationasperdutyofcareprotocols.WhilethehostorganisationofthecWcprojectalsorelocateditsinternationalstaff,itwasabletocontinueitsdailyoperationsthroughouttheperiodofviolencethroughitsnetworkoflocalpartnersandstaff.Theprojectmanagerposition,previouslyaninternationalstaffposition,shiftedtoanationalstaffposition.Projectimplementationwasdelayedbyseveralmonthsbetweentheperiodofevacuationofinternationalstaffandthere-entryofstaffwithrespondentsnotinglengthynegotiationsamongconsortium partners about re-entry terms and conditions.With that said, respondents alsonotedthatimplementationdelayscouldariseinanyprojectoperatinginachallengingcontext.
SeveralmonthslaterdiscussionsresumedamongconsortiumpartnerstoconsiderthefutureofthecWcproject,re-entry,andcontractingoftheflexiblefundingmechanismprojects.Afterconductingariskassessment, the in-countryhostorganisationdeterminedthatthreeofthefivepreviouslyidentifiedsub-projectscouldproceedasplanned.Itwasultimatelydecidedtoimplementthethreeprojectsbutatareducedscaleandscope.Thecontractingprocessfor
176 centerforciviliansinconflict.Underfire:TheJuly2016ViolenceinJubaandUnResponse.UnitedStates:centerforciviliansinconflict,2016.
177 GuardianT.‘Untolddevastation’inSouthSudantriggersgravehumanitariancrisis:TheGuardian;2016[cited201714April].Availablefrom:https://www.theguardian.com/global-development/2016/aug/09/untold-devastation-in-south-sudan-triggers-grave-humanitarian-crisis-un.
189 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
thethreeprojectswasstartedinnovember2016,wascompletedinDecember2016,andtheprojectsbeganimplementationinJanuary2017.Anationalstaffprojectmanagerwashired.Infebruary2017,budgetrevisionsandreductionswerefinalised.
followingtheescalationofviolenceinJuba,theProtectioninPracticeprojectalsotemporarilyevacuateditsinternationalstaff,whileitslocalpartnerscontinuedwithprojectimplementation.OneofthefacilitatingfactorswasthatthelocalandnationalorganisationssupportedbytheprojectwerelocatedinareasofSouthSudanthatwererelativelyunaffectedbythatparticularviolenceinJuba;hence,theprojectexperiencedfewerdelaysinimplementation.Theprojecthadalsocompleteditsfirstoutputonprotectionmainstreamingbythetimetheviolenceoccurred,whichmeantfewerdelays inoverallproject implementationandprojectactivities.However,thesecondoutputoftheproject(focusedonsupportinglocalandnationalorganisations)wasaffectedbytheviolencebecausetheinternationalstaffmembersresponsibleforadministrationofthisoutputwereevacuated.
ArespondentfromtheProtectioninPracticeprojectalsocitedthattheadaptiveprojectdesigninherentlyallowedformoreflexibilityinimplementationduringtheescalationofviolence.Theprojectwas designed from the outset around the national conflict, such that themore theviolence increases, themoreflexibilityprojectmanagementhad in termsof implementation.Asnotedbyprojectmanagementmostotherprojectsinthiscontextwouldhavebeenhaltedcompletely;however,therewasflexibilitybuiltintotheprojectdesigntorespondtochangesinviolence,fromthedonortotheprojectmanagertotheincountrystaff.Theprojectwasdesignedsothatintheeventofanescalationofviolencetherewouldbelessrelianceoninternationalstaff,inthatanythingthatwouldnormallybedelegatedtoaninternationalorganisationwouldbehandledbythelocaland/ornationalorganisation.
1.2. Impact of Violence on Project Implementation: Flexible and Emergency Funding Mechanisms
bothprojectsweredesignedwith aflexible fundingmechanism;however, theviolence andsubsequentdecisionsonprojectimplementationhaddifferentimpactsontheflexiblefundingavailabletolocalandnationalorganisations.
followingtheviolenceinJuba,thecontractingprocessfortheflexiblefundingmechanismwithinthecWcprojectwasputonhold.Aspartofthere-entrynegotiations,itwasdeterminedtoproceedwiththreesub-projectsandarevisedprojectactivitymixasdescribedabove.However,giventhecWcprojectenddateofSeptember2017,theprojectactivitiesandtheoverallbudgetneededtoberealignedtoaccountforthedelays.Whiletheconsortiumpartnersconsideredapplyingforanocostextension,itwasdeterminedthatthiswouldnotbesufficienttocoverthecostsofsalarysupportforprojectmanagementthatwouldbeneedediftheextensionwasgranted.Therefore,theoverallbudgetwasreducedandfunds(approximatelyone-thirdofthebudget)werereallocatedtoacWcworkinggroupinthePhilippines.Additionally,therewassupposedtobeasecondroundofflexiblefundingtosupportthegrowthofthosepilotprojectsthatwere successful.Due to the delays and thewindowwithinwhich to apply for projectfunding,asecondroundofflexiblefundingwasnotpossible.
for the Protection in Practice project, within the budget available to local and nationalorganisations,therewasasumofmoney(approximately£10,000)setasideasemergencyfunds.Theseemergencyfundsdidnothavetobeallocatedatthebeginningoftheprojectandcouldbedrawnuponiftherewasanemergencyorescalationinviolence.Theprojectwasdesignedsuch that this emergency funding could be accessed and utilisedwith the least amount ofadministrationaspossible,suchasbysendingamessagetotheprojectmanagerthatthemoneywasneeded.ThefundingwasdisburseddirectlyfromtheInGOtothelocalorganisationsasacashaward, thus, increasing theflexibilityof the funding rather thanadministering it asagrantwith reporting.beyond the fundingprovided to the local andnationalpartners, therewas a contingency plan built into the funding so that in the event internationalnGO staffwereevacuated,the internationalnGOcouldsupportthe localpartnersremotely,providingmentoringonproject implementationandsupportwith logisticsandadvocacy, soasnot todelaythelocalimplementationoftheproject.
190 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Asdescribedbyonerespondent,fundingtranslatesintodifferentvalueindifferentcontexts—suchthattheimpactoffundinginSouthSudan,giventhehighcostsofoperatinginthecountry,ispotentiallyreducedcomparedtootherhumanitariansettings.ThisrespondentdescribedtheexpensivecontextofoperatinginSouthSudanandthatthisshouldbeaccountedforinfutureprojects.
2. Decision-making Processes and Project Management
Respondentsfrombothprojectsdescribedthecomplexitiesofthedecision-makingprocessesinthemonthsfollowingtheviolence.Whiletheimmediateconsiderationwastowardstaffsafetyandsecurity,severalotherkeyfactorsemergedduringtheinterviews.RespondentsfrombothprojectsreportedthenecessitytoadapttochangingcircumstancesinSouthSudanandofferedreflectionsonprojectmanagementinaconflict-affectedsetting.
forthecWcproject,thedecisionoptionsweretoresumeprojectactivitiesasplanned,resumeactivitiesbutwithreducedscaleandscope,ormovethefundingfromSouthSudantoanothercountry.Itwasdecided,asdescribedabove,toimplementthreeoftheplannedfivesub-projects.Asplanswereoutlined to implement thoseprojects, differingopinionsemergedamong theconsortiumpartnersabout theplannedapproach.At thattime,given travel restrictions, theinternationalstaffmembersforthein-countryorganisationwerestillbasedoutsideofSouthSudan.Theprimerecipientorganisationofthegrantadvocatedforanofficewithpermanentofficestafftoallowfordirectsupervision.Thein-countryhostorganisationnotedthatithadstronglocalteamsontheground.Itwashereinthedecision-makingprocessthat importantquestionsemerged,exposingalackofclarityanduncertaintyabouttherolesofeachconsortiumpartner.Who makes the decisions? Is it the in-country host organisation or the organisation managing the grant, or is it the donor? Where does duty of care fall?Itwasnotedthatthedialoguearoundthedecisionwasparticularlychallengingduetothemultiplelevelsofpeopleinvolved,thatevenminordecisionsbecameoverlycomplicated.
Respondents from the cWc project described the complexities of bringing numerousorganisations,eachwiththeirownindividualmandatesandorganisationalprocedures,toreachadecision.Despiteattempts tooutlineorganisationaldecision-makingmatricesevenbeforetheviolence,therewasstillalackofclarityabouttherolesofthedonor,projectmanagerorglobalgrantholder,and thehostagency,whichalsoactedasadonor for its localpartners.ThiswasfurthercomplicatedbythefactthattheonebranchofthehostagencyalsohostedthecDAcnetwork.Eventually,adecisionwasmadethroughconsensusbetweentheglobalgrant holder and the in-country host organisation.Amanagement advisory group providedfurthersupportduringthenegotiations.Thein-countryhostorganisationnotedstrongworkingrelationshipswith its localpartnersandstaff,but itwasnotclearwhat levelofengagementnationalorganisationshadinthenegotiationsaroundre-entry.AnotherchallengenotedbyarespondentfromthecWcprojectwashowthetraumaexperiencedbyorganisationsandtheirstaffasaresultoftheviolenceaffecteddecision-making.Aspeoplewerestilladaptingtothetraumaticeventsthathadtakenplace,therewastheneedtocontinuetomakedecisionsfortheprojectandorganisation,whichwasnotedtobechallenging.
TheProtectioninPracticeproject,ontheotherhand,haddesignedtheprojectinSouthSudantobeadministrativelyflexibleandwithfewbureaucraticprocedures.Asarespondentfromtheprojectnoted,withoutthatflexibility,theprojectwouldhavefailedinSouthSudan,astheywouldnothavebeenabletoimplementoutput2.Thedecision-makingprocessesfortheprojectweredescribedasadaptive,direct,andstreamlined. Itwas thedirect lineofcommunication fromthefieldwithdecisionmakersthatwasdescribedasthemostimportanttosupportongoingproject implementation in the context of insecurity. facilitating factors cited included opencommunication,trust,anddelegatedresponsibilitytotheprojectmanagerswhoarethepeopleinclosestcommunicationtothefieldteams.Projectmanagersreportedanelementofcontrolandthattheyfeltedtrustedtomakedecisions.Asanexample,theprojectwasdesignedsuchthatthepersonmanagingthefundingisalsoabletomakeprojectdecisions.
fortheProtectioninPracticeproject,oneofthekeyfactorsnotedthatallowedforadaptivemanagement was trust. This was described on one level as trust between DEPP and theconsortium organisations and then on the next level as trust between the consortium
191 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
organisationsandlocalandnationalpartnerorganisations.Thetrustbetweenprojectmanagersandthetechnicalteaminthecountrywasattributedtothebeliefthatthelocalandnationalorganisationsunderstandtheprotectionneedsandhowtobestservethoseneeds.
3. Impact on the Multi-stakeholder Platforms and Collaborative Partnerships
When asked about the impact of the violence on the multi-stakeholder platforms and oncollaborativepartnershipswith local andnational stakeholders, bothprojects acknowledgedtheimpactoftheprojectsuspensiononthelocalandnationalorganisationalpartners.
InregardtothecWcproject,asnotedabove,the in-countryhostorganisationwasabletocontinue project implementation through its network of local partners. Respondents alsopositivelynotedthatthecWcworkinggroupdidmeetduringtheperiodofprojectsuspension.However, there was concern that the group lost momentum during the period of projectsuspensionandthatthemeetingsmaynothavebeenaseffective.Asonerespondentnoted,withinthecontextofviolenceandwiththeresultingfamineandhumanitariancrisis,thecWcworkinggroupmaynothavebeenprioritisedgiven thecompetingpriorities forOcHA, theworkinggroupco-chair.Anotherrespondentacknowledgedthatthegroupwasatanascentstagewhentheviolenceoccurred.
Oneof theProtection inPractice respondentsdescribed the impact that theevacuationofinternationalorganisationshadonthelocalandnationalorganisationsthatremainedinSouthSudanwithactiveprogramming.RecognitionoftheworkbeingdoneonthegroundbylocalandnationalorganisationschangedthewaytheInGOsviewedtheirpartnershipswithintheproject.Thesepartnershipsshiftedtobeviewedascomplementarypartnerships,ratherthana partnership built on unidirectional capacity building. In doing so, power dynamics shiftedbetween the InGO and the local and national partner organisations and the relationshipsevolvedtobemorereciprocal innature.Whiletheviolencebroughtto lightthe importanceof reciprocal relationships, this approach to building partnerships existedwithin the projectevenpriortotheviolence.Asanexample,fromthebeginningoftheprojectdesign,thelocalandnationalorganisationsweregivenanopportunitytodecidewhichinternationalprotectionpartnerorganisationcouldprovidethetrainingandcapacitybuildingtheywereseeking,ratherthantheInGOdecidingwhatandwhomtoteach.
When considering these data together with the data from the evaluation’s minimum setinterviews,itwasnotedthatthedepartureofInGOsfollowingtheviolenceleftavoidinthehumanitarian community and led to a complete lack of responsibility. The respondent feltthat this void encouraged further violence by armed groups and encouraged unnecessarycompetitionbetweennGOsremaininginSouthSudan.
4. Emergency Preparedness and Response in a Conflict-affected Setting
Respondents from both projects had opinions and broader reflections about focusing onemergencypreparednessinthemidstofaconflict-affectedsetting.
ThecWcprojectwaspositionedinSouthSudanasanemergencypreparednessprojectandnot an emergency response project.Meanwhile, the in-country host organisation’s broaderportfolio,beyond thecWcproject,waspartially focusedonemergency responseactivities.Onerespondentreflectedonwhetheritisappropriatetotrytofocusonpreparednessinsuchacontextwherethereisongoinginsecurityandhowtomosteffectivelyconsideremergencypreparednessinsuchsettings.Thisrespondentnotedthatitisdifficulttogetpeopletothinkaboutpreparingwhentheyneedtobethinkingaboutfunctioningandhowtorespond.
WithinthecWcproject,thehostorganisationoperatedasbothadevelopmentprojectandahumanitarianproject.becauseofitsinternalflexibility,theorganisationisabletopivotactivitiesandfundinginresponsetoviolence.butbecauseofthemandateoftheDEPPonpreparedness,thiskindofpivotingwasnotpossible.Themandateofemergencypreparednessappliestotheentireconsortiumandmaintainingtheattentiononpreparednessamongconsortiumpartners,someofwhomalsohaveresponseactivities,wasnotedtobeparticularlychallenging inthemidstofanactiveconflictandfamine.
A respondent from the Protection in Practice project described the importance of linking
192 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
preparedness and response—that these are not necessarily two separate concepts. Thisrespondentnotedthatoneofthebestwaystostrengthenpreparednesswastoalsorespond,sothatthelessonslearnedwhilerespondinginturnstrengthenpreparednessactivities.Theprocessofrespondinginandofitselfallowsforpreparedness,andallowsforlearningtobeappliedifthesamesituationhappensinadifferentlocationatadifferenttime.combiningpreparednessandresponseactivitiesalsoallowsforopportunitiesforcombinedtraining,mentoring,andlearning,inadditiontoaperceivedbettervaluefortheinvestmentandultimatelyabetterresponse.
Itwasnoted in interviews thatmuchof theevidenceonpreparedness todatecomes fromnaturaldisastersettings,inwhichthereisacertainpredictabilityofeventsthatcanbepreparedfor inanorganisedmanner.However,preparednessforconflictwasdescribedasadifferentprocessthathasnotbeenexploredinasmuchdepthasemergencypreparednessandresponsetonaturaldisasters.Mostoftheevidenceonpreparednessfocusesondisaster-affectedareasanddevelopmentsettingsthathaveatendencytobecomehumanitariancontextsatcertaintimesduringtheyear(duetoadroughtortyphoon,asexamples).Respondentsidentifiedagapinthelearningandevidenceonpreparednessinconflict-affectedsettings.
5. Lessons Learned about Implementing a DEPP Project in a Conflict-affected Setting
Respondentswereaskedtoreflectonlessons learnedaspartofprojectmanagement inthecontextofviolence.Prominentpointsraisedrelatedtodutyofcareandtothemanagementarrangementsthatwereinplace.
5.1 Protocols, Duty of Care and Operating in a Conflict-affected Setting
RespondentswereaskedaboutprojectplanningandimplementationinSouthSudanandwhatlearningcouldbeappliedtootherDEPPprojectsoperatinginsettingsatriskofviolenceandinsecurity.Inregardtosecurityprotocols,itwasnotedthattheviolenceinJubawasunexpectedandthatthesecurityplanninghadprioritisedotherareas inSouthSudanthatwerethoughttobehigher risk. Inaddition to securityprotocols, respondentsemphasised the importanceofhavingclearagreementsaboutdutyofcarewhencreatingmulti-lateralagreementsamongmultiple institutions in a consortium and the responsibilities of care for the host agency inrelationtoconsortiumpartners.
5.2 Clarifying Consortium Agreements and Streamlining Contractual Arrangements
Inadditiontohavingclearagreementsofdutyofcare,respondentshighlightedtheimportanceofstreamliningthecontractualarrangementsamongorganisations.Keytooperatinginaconflict-affectedcontextwastohaveproceduresthatallowedforflexibleandadaptivemanagement,rather than traditionalandrigidcontractual relationships.Asanexample,cWcrespondentssuggested that futureprojectsensure that theglobalgrantholdingorganisationand the in-countryorganisationarethesamewhenpossible,suchthat ifWorldVisionUKistheglobalgrant holder that the in-countryorganisationwouldbeWorldVision in SouthSudan, as anexample.Thiswassuggestedtostreamlineadministrativeprocedures,flowoffunds,anddutyofcare.
5.3. Lessons Emerging from Minimum Set Interviews on the DEPP Approach in South Sudan
Theinterviewsconductedforthecasestudywereanalysedinconjunctionwithdatapreviouslycollected during theMinimum Set evaluation activities.These interviews offered additionalperspectives on project implementation in a conflict-affected setting and reinforced theimportanceofconsideringcontext inprojectdesignandimplementation.Thedatawerealsoanalysedwiththeexternalevaluationtheoryofchangeinmind,consideringthecorecomponentsofDEPP(capacitybuilding,collaboration,earlywarningsystems,andlearning).
OneoftheobservationsonDEPP’sconsortiumapproachandemphasisoncollaborationwithstakeholdersisthatengagementofgovernmentstakeholdersinprojectsisnotalwayspossibleinconflict-affectedsettings,suchas inSouthSudan.Giventhegovernment’s involvement intheconflict,theDEPPprojectsinSouthSudanwereunabletocollaboratewiththegovernmentasapartner.
Inregardtocapacitybuildingonemergencypreparednessandresponse,mostoftheexisting
193 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
trainingtoolsandapproachesaredesignedforotherdisasterandemergencycontexts,andnotforconflict-affectedcontexts.ArespondentfromthecWcprojectutilisedcasestudiesduringtrainingstoincreasecapacity;however,thosecasestudiesweretoodifferentfromtheSouthSudancontextandweren’taseffectiveasintended.
Respondentsreflectedonchallengespredictingviolenceinconflict-affectedsettings,citingalackofstandardisedriskassessmentsor indicators,suchasearlywarningsystems,to informprogramming.Asanexample,arespondentreflectedhowothersitesoutsideofJubahadbeenidentifiedasathigherriskofviolenceandthusprogrammingwascentredonthisassumption.
Inregardtolearning,onerespondentnotedthatexpectationsneedtoberealistictothecontext.ThisrespondentobservedinSouthSudanthatorganisationalcultureandpoliciesandownershipofcommunicationmessagesdifferedamongorganisations,suchthatorganisations,includingInGOs,nationalandlocalnGOsinSouthSudan,werenotedtobelesswillingtoshareinformationcomparedtootherDEPPprojectcountries.
KEY FINDINGS
• Thekeytooperatinginaconflict-affectedcontextwastohaveproceduresthatallowedfor flexible and adaptive management, rather than traditional and rigid contractualrelationships.Aspects of one of the projectswere able to continue because of thisflexiblemechanism.
• In addition to security protocols, respondents emphasised the importanceof havingclear agreements about duty of carewhen creatingmulti-lateral agreements amongmultipleinstitutionsinaconsortiumandtheresponsibilitiesofcareforthehostagencyinrelationtoconsortiumpartners.
• Engagementofgovernmentstakeholdersinprojectsisnotalwayspossibleinconflict-affected settings, such as in South Sudan. Given the government’s involvement inthe conflict, theDEPPprojects in SouthSudanwereunable to collaboratewith thegovernmentasapartner.
194 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
CASE STUDY #2: STRENGTHENING PREPAREDNESS AND RESPONSE IN KENYA AND ETHIOPIA Thesecondcasestudyfocusedonpreparednessandresponseeffortsintwoselectedcountries:Kenya(UrbanEarlyWarningEarlyActionproject)andEthiopia(ShiftingthePowerproject).
URBAN EARLY WARNING EARLY ACTION PROJECT
TheUrbanEarlyWarningEarlyAction(UEWEA)project178innairobi,Kenya,wasimplementedbyaconsortiumcomprisedoffournGOs(concernWorldwide,KenyaRedcross,Oxfam,andWorldVision)andnairobicountyGovernment.Theprojectaimedtodevelopearlywarningsystems and early action responses to food insecurity and health-related crises in selectedurbanslumsofnairobi(Kibera,Mukuru,Korogocho).TheprojectbuiltonconcernWorldwide’s5-year IndicatorDevelopment for theSurveillanceofUrbanEmergencies (IDSUE)project179 toestablishthresholdsforhumanitarianresponserelatedtofoodinsecurityandrelatedhealthcrises.Theprojectaimedtoachievefourkeyoutcomes:180:
• facilitatetheset-upofandcapacity-buildacoordinatedurbanearlyactionmechanismwithinthenairobicountyGovernmentwithagreedactions
• Strengthen the capacity of six nairobi sub-counties and one informal settlementcommunitytomitigateandrespondquicklytotheimpactsofslow-onsetemergencies
• Ensure routine surveillance in urban informal settlements in Kenya by the countyDisasterManagementcommittee(cDMc)bytheendof2017
• AdvocateforUEWEAasabestpracticeandmostcost-effectivemodelforrespondingtourbanemergenciesintheworld
SHIFTING THE POWER PROJECT
The Shifting the Power project was implemented in five countries (bangladesh, Pakistan,DemocraticRepublicofthecongo(DRc),EthiopiaandKenya)byaconsortiumcomprisedoffivenGOs:ActionAidandcAfOD(co-leads),Oxfam,concern,Tearfund,andchristianAid.
Theseorganisationsworkedtogethersupporting55oftheirlocalandnationalnGOpartners(10inEthiopia)whosharethevisionandambitionofplayingaleadingroleindecision-makingandrespondingtocrisesintheircountriesandregions.Theaimoftheprojectwastosupportlocal actors to take their place alongside international ones in order to create a balancedhumanitariansystem.Theprojectwastoproducefiveoutputstoachieveoneoveralloutcome:
Outcome1:nationalorganisationsinfivecountrieshaveincreasedtheircapacitytodetermineanddeliveremergencypreparednessandresponse
• Output1:nationalpartnerorganisations infivecountrieshavetheknowledge,skills,processes,andpoliciestoprepareforandrespondeffectivelytoemergencies
• Output2:nationalorganisationsarebetterrepresentedandhaveastrongervoicein
178 STARTnetwork.TheUrbanEarlyWarningEarlyActionProject:foodSecurityandnutrition,2017.179 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).UrbanEarlyWarningEarlyAction
finalEvaluationReport,p1.180 STARTnetwork.TheUrbanEarlyWarningEarlyActionProject:foodSecurityandnutrition,2017.
195 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
relevanthumanitarianplatformsandnetworksintheircountries• Output 3: Shifting thePower consortiummember InGOs recognise and respond to
local/nationalorganisations’capacity,leadershipandvoice• Output4:ShiftingthePowercollaborateswiththeotherDEPPprojectstomaximise
collaborativeadvantage• Output 5:The project provides evidence of good practice in strengthening national
organisations’ humanitarian preparedness and response work and their role andinfluenceinhumanitarianaction
FINDINGS
Atotalof22in-depthinterviews(n=16inKenyaandn=6inEthiopia)wereconductedforthecasestudy(Table8.1)witharesponserateof100%(16/16inKenyaand6/6inEthiopia).Duetosecurityreasons,theteamcouldnotreachallofpotentialrespondentsinEthiopiaandhenceasmallernumberofinterviewswerepursuedandconductedinthatsetting.
Kenya
InKenya,ofthe16respondents,11(68.8%)weremaleand5(31.2%)werefemalewithameanageof36.7years[25-56years](seeTable8.1).Eachoftheorganisationsandkeygovernmentpartners(ministriesandcountygovernment)oftheprojectwasrepresentedintheinterviews.Respondentsreportedworkingonaverage7.6yearsinthehumanitariansectorand6.2yearsfortherespectiveorganisation.
Ethiopia
InEthiopia,ofthe6respondents,5(83.3%)weremaleand1(16.7%)wasfemalewithameanageof42.8years [36-62years].Thesixorganisations representedbeneficiaryorganisationsinEthiopia affiliatedwith the Shifting thePowerproject. Respondents reportedworkingonaverage7.8yearsinthehumanitariansectorand9.2yearsfortherespectiveorganisation.
KENYA ETHIOPIAVariable N (%) N (%)Sex of RespondentMale 11 (68.8%) 5 (83.3%)Female 5 (31.2%) 1 (16.7%)Total 16 6
Mean Age of Respondent (Years) 36.7 years [25 – 56 years] 42.8 years [36-62 years]
Highest Educational Degree ObtainedBachelor Degree 7 (43.8%) 2 (33.3%)Master’s Degree 6 (37.5%) 4 (66.7%)High School Diploma 3 (18.7%)
OrganisationsInternational NGOs 7 0National and/or Local NGOs 6 6Government (i.e., Ministry, County Offices)
3 0
Table8.1:DemographiccharacteristicsofrespondentsinKenyaandEthiopia
196 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Respondentswere asked to summarise the types of emergencies that occurred during theprojectperiodandcorrespondingpreparednessandresponseactivities.Datafromsitevisitsandprojectreportswerecompiledtodescribetheresponseactivitiesandarepresentedbelow(seefigures8.1and8.2).
Humanitarian Response: Focus on urban emergencies in Kenya
What:
Anurbanearlywarningsurveillancesystemsetupininformalsettlementsinnairobicountytomonitorfoodsecurity,householdshocksandpublichealthindicatorsonaroutinebasis(everytwomonths)detectedthreeemergenciesandtriggeredresponseactivities.
1. Diarrhoealoutbreak:february2017toJune2017
2. choleraoutbreak:April2017tofebruary2018
3. foodinsecurity:OctobertoMarch2018
Who:
TheDEPP’sUrbanEarlyWarningEarlyAction(UEWEA)projectdevelopedandimplementedthe urban earlywarning system through its consortiummembers concernWorldwide,Oxfam,WorldVisionandKenyaRedcrossSociety,insevensub-counties(Kibera,Kasarani,Ruaraka,Makadara,Starehe,KamukunjiandEmbakasiEast).Thesystemaimedtoidentifyslowonsetorchronicemergenciesinurbancontextsthroughroutinemonitoringofasetofsensitiveindicatorsthatweredevelopedalongwithspecificthresholdspriortothestartof theprojectand tosetupacoordinatedresponsemechanismwithin thenairobicitycountryGovernment.
Response Activities:
TheDEPP’sUrbanEarlyWarningEarlyAction(UEWEA)projectdevelopedandimplementedthe urban earlywarning system through its consortiummembers concernWorldwide,Oxfam,WorldVisionandKenyaRedcrossSociety,insevensub-counties(Kibera,Kasarani,Ruaraka,Makadara,Starehe,KamukunjiandEmbakasiEast).Thesystemaimedtoidentifyslowonsetorchronicemergenciesinurbancontextsthroughroutinemonitoringofasetofsensitiveindicatorsthatweredevelopedalongwithspecificthresholdspriortothestartof theprojectand tosetupacoordinatedresponsemechanismwithin thenairobicitycountryGovernment.
• Diarrhealoutbreak:healtheducation,massscreeningofchildrenunder5years (8,338screened in Korogocho, 13,284 in Embakasi), and distribution of water purificationproducts
• choleraoutbreak:healthpromotion(171,139households,2630studentssensitised),solidandliquidwastemanagement,andwaterqualityandfoodsafetymonitoring
• foodsecurity:Massscreeningofchildrenunder5years(44,825)andcashtransfersof2000KSH(roughly20USD)providedto3,034households(2,085women,949men)whowereseverelyfoodinsecureweremadeviamobilemoneyinKorogochoandKibera
figure8.1:HumanitarianResponse:focusonUrbanEmergenciesinKenya
197 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
Humanitarian Response: Focus on drought in Ethiopia
What:
Ethiopiahasbeenexperiencing largescaledroughtssince2015. Initially inducedbythe2015/2016Elnino,andfurtherexacerbatedbylow2016autumnrains,andbelowaverage2017Spring rains, at least8.5millionpeoplewere in needof relief food assistancebymid-2017inSouthernandSouth-easternEthiopia.Theeffectsofthedroughtwerefurtherexacerbatedbydiseaseoutbreaks,largescalelossoflivelihoodassets,anddisplacement.conflict intheOromiaandSomali regionsbeginning inSeptember2017resulted inthedisplacementofhundredsofthousandsofpeopleoftenindroughtaffectedareas.
Who:
• fourDEPPprojectswereimplementedinEthiopia:ShiftingthePower(STP),PublicHealthPreparednessinGambella,TalentDevelopmentandImprovedEarlyWarningEarlyAction–Ethiopia.
• Self-reporteddatasuggest7outof10localpartnersoftheShiftingthePowerproject,whichhadaimedtobuildlocalandnationalnGOcapacityandpreparedness,respondedtothedroughtintheareaswheretheymanagedevelopmentprogrammes.Theseincluded:community InitiativefacilitationAssistance (cIfA),communityDevelopment Initiative(cDI),RiftValleychildrenandWomenDevelopmentOrganisation(RVcWO),ActionforDevelopment,SOSSahel,HUnDEEandAVHO.
Response Activities:
Specificresponseactivitiesledbythesel/nnGOsin2017-2018included:
• cIfA vaccinated 263,187 cattle from 35,060 target households inMoyale,Miyo andGuchiworedas.Improvedresponsetimeduetostrengthenedorganisationalprocurementpolicieswasreported
• cDIsecureda£10,000responsegrantfromOxfamandSTPwhichenableddistributionofseedsto200householdsincommunitiesaffectedbythedroughtinWestArsiZone,Oromia
• RcVWOobtaineda£60,000grantfromActionAidtoprovidelivestockfeedsupportto1,030households
• Action for Development provided £60,000 in cash transfers to drought-affectedcommunitiesinborena
• SOSSahelsecured$530,000directlyfromUnOcHA’sEmergencyResponsefundtoprovideanimalfeedinthreeworedas(Wachille,DubuluqandDire),indirectlybenefitting28,256individuals.ThisisoneofthefewinstancesalocalEthiopiannGOhasbeenabletosecurefundsthroughthismechanismwithoutanintermediarypartner
• HUnDEE obtained $410,000 from Kindernothilfe (KnH) to provide emergency foodassistanceto10,000peoplein4KebelesoftwoworedasintheborenaZone
figure8.2:HumanitarianResponse:focusonDroughtinEthiopia
belowarekeythemesidentifiedduringthedataanalysis,withexcerptsfromtheinterviewsorsummariesfrominterviewnotes.
I. Key project design and implementation factors that strengthened emergency preparedness and response included community and local organisational engagement, government engagement, and flexible funding mechanisms.
Ia. Community Engagement and Emergency Preparedness and Response
198 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Among key project design and implementation factors, respondents highlighted the importance of community engagement, especially in the design of the surveillance system, the selection of beneficiaries, and the response activities.
Inbothcountries, keycommunitymembersand stakeholderswere identifiedaspartof theprojectdesignandengagedinvariousrolesthroughouttheproject.communityengagementnotonlymeantprovidinginput,butalsobeingengagedinselectingbeneficiariesanddesigningtheresponsepackageofactivities.AsdescribedbyarespondentinEthiopia:“youhavetobetransparentandinvolveallthepeople,notonlymajorityordominantpeopleinthecommunity,aswellaswomen.Duringbeneficiaryselection,everyoneshouldbe involvedandhelpwiththeverification.”Asaresultofengagingthecommunity intheselectionofbeneficiariesandthe response activities: “…therewere no complaints frombeneficiaries about the selection.Thereusually[are]complaintsaboutselectionerrorbutthistimetherewasnot…communitymemberswereabletogiveopinionswhichmadeitsuccessful.”
Respondents inboth countries commented that communityengagementhelped tomanagecommunityexpectations,especiallyinregardtosurveillanceorneedsassessments.Oneofthechallengingaspects,asdescribedbytherespondent,was“pushback”fromthecommunityduringdatacollectionbecausecommunitymembersareaskedtoprovidedata,butmaynotreceiveanydirectaid.Toaddressthisproblem,theprojectinEthiopiacreatedseveralcommunity-basedcommittees,includingabeneficiaryselectioncommitteeandanemergencyitemsmanagementcommittee.Thisallowedtheprocessestobemoretransparenttothecommunityandprovidedamechanismbywhichcommunitymemberscouldbringforthconcernsorcomplaints.Similarly,theprojectinKenyaidentifiedcommunityfocalpeopleandcreatedopportunitiesfordialogueamongstakeholdersfromcommunityrepresentativestofoodandwatervendorstogarbagecollectors.
Ib. Government Engagement and Emergency Preparedness and Response
Respondents in both countries noted that partnership with government was crucial to project design and implementation from facilitating administrative procedures to fostering trust.
Governmentengagementwasnotedoveralltofacilitateprojectimplementation.Ononelevel,inclusionof government in theproject design and implementation facilitated administrativeprocedures.As noted by a respondent in Ethiopia: “This time itwas different because thegovernmenthashelpeduswithfacilitatingthesigningoftheagreement.Eventhoughalreadytheeffectofthedroughthasstartedtobeobserved,afterthefund[was]released…wehavebeenabletointerveneafteramonthortwo.Thisisdifferentfromotherinterventionsbecausethepreviousbottlenecksandbureaucraticstepshavebeenalleviatedandwewereallowedtoproceedwithsomeoftheimportantproceduresafterwehavestartedtheproject.”
Onanotherlevel,engagementwithgovernmentfosteredtrustbetweenprojectorganisationsandthegovernment.forexample, inKenya,theengagementwiththegovernment,earlyoninprojectdesignandimplementation,allowedfortrustinthesurveillancesystemandinthesurveillancedatacollectedbyprojectorganisations.Thistrustwasespeciallyimportantwhensurveillancedataindicatedanoutbreakofcholeraintheurbanslums,andhelpedtofacilitateamoretimelyresponse.
Ic. Funding Mechanisms and Emergency Response
Respondents from both projects reported that timely access to funding and flexible funding are essential to the speed and timeliness of humanitarian response.
InKenya,theUEWEAprojecthadaflexiblefundingmechanism,knownasa“crisismodifier”,which is funding set asidewithin the project to allow organisations to act and respond tosurveillanceindicatorsthatdetectanemergency.Aspartoftheproject,asystemwasdesignedsuchthataresponsewouldberequiredinrelationtofivesurveillanceindicatorsandthatthiswouldinturntriggertheflexiblefundingmechanism.
A respondent inKenyaexplained that thecrisismodifierwasessential to their response tothediarrhoeaoutbreakinfebruary2018.However,itwasnotedthateventhoughthecrisismodifierexisted,therewasn’tinitiallyasysteminplacebywhichtotriggerthecrisismodifier.Thissystemwassubsequentlyrefinedduringthecourseoftheproject.
199 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
Whilethecrisismodifierwasalignedwithsurveillanceindicators,therewasconcernexpressedthatitmayhavenotbeenlinkedtothemosturgentneed,suchascholera.“Ithinkthey’dalreadyplannedorhadintheirmindwhatthey’duseit[crisismodifier]for,theyweremonitoringfoodpricesthosehavegoneup,andthey’vebeenupinnairobiforacoupleofyearsso,butyouknowtheyhadtriggersthatwerethereandthattriggeredafoodsecuritycashtransferresponse.buttomeitwasfrustratingthatindicatorsmayhavepointedtowardsacholeraoutbreakandthat’sclearlyformeoneofthebiggest,oneofthebiggestthreats, immediatethreats isacholeraoutbreak.”ThedelaysinrecognisingandplanningfortheemergingcholeraoutbreakledtoaslightlydelayedresponsecompoundedbytheneedtoseekfundsfromoutsidetheDEPPtorespondtothecholeraoutbreak.
WhiletheprojectinKenyahadaflexiblefundingmechanismbuiltintotheprojectbudgetfromtheoutset, noneof the respondents inEthiopia reportedhaving access to aDEPPflexiblefundingmechanismtosupportemergency responseactivities.Document reviews,however,foundthatShiftingthePowerdidprovideasmallfundforresponseactivities.Organisationsthataimedtoimplementresponseactivitiesreportedsubmittingproposalstoexternaldonorstoseekrapidfundingwhenanemergencywasdetected.notingthatthiswasachallengeformanylocalorganisationsduetolimitedcapacity,theShiftingthePowerprojectshouldcontinuetostrengthenorganisationalcapacitytoprepareandsubmitthesefundingproposals.
Despite the increased capacity around fundraising for a response, interviewees in Ethiopianotedthatresponseeffortsweresignificantlydelayedduetothetimerequiredtoseekresponsefundingfromoutsidesources.forexample,oneintervieweeinEthiopiaexplainedthatasearchfor funds led to anearly six-monthdelay in implementationof response activities.AnotherintervieweeinEthiopiadescribedhowtheprocessofseekingfundingforresponseactivitiesalsoaffectedtheirabilitytorespondtoshiftingneeds:“Developaproposalandwithin72hoursthereshouldbearesponse,butit’sactuallynotlikethat.Insteadyouactuallyhavetowaitonemonthortwomonths,prepareaproposal,getcomments,thengetpaidfor1-3months.Thecallsforproposalsaren’tincontextofthecurrentposition...thus,someimmediateneedsdon’tgetanimmediateresponse.”
OneofthepositiveoutputsoftheShiftingthePowertrainingsinEthiopiawasthatorganisationshad proposals ready that can be adjusted immediately to the context. However, despitestrengthenedcapacitytoseekfundingforresponseactivities,manynationalnGOsinEthiopiadonotqualifytoreceivefundingfromcertaindonorsbecausetheydonothavetherequiredforeigncurrencybankaccounts.Anotherrespondentnotedthatlocalorganisationsoftenaren’teligibleforamultitudeofreasonstoreceivefundingdirectlyandthatthefundingstreamsarebasedoutsideof thecountry, suchas in theUnitedStates,UnitedKingdom,andEuropeancountries, leading to additional delays in response activities. for this reason, more flexiblefundingoptionsthatcouldpotentiallyovercomethesebarrierswouldbeaneffectivesolutiontoincreasingtimelinessofresponse.
II. How key components of DEPP – capacity building, collaboration, early warning systems, and learning – influenced emergency preparedness and response activities in Kenya and Ethiopia.
IIa. Capacity Building and Emergency Preparedness and Response
Capacity building through trainings at community and organisational levels strengthened emergency preparedness and response through improved coordination, surveillance, proposal development, and project implementation.
ThemajorityofrespondentsprovidedexamplesoforganisationalcapacitybuildingaspartoftheDEPP.However,therewerealsoseveralexamplesofcapacitybuildingatthecommunitylevel.ArespondentinKenyareportedthatafteracommunity-levelcapacitybuildingtraining,communitymemberswereabletorespondeffectivelywithoutfurtherassistance.TheprojectinKenyaalsotargetedcommunityhealthvolunteersandcommunityhealthworkerstosharemessagingandbuildcommunitycapacityinregardtoemergencypreparednessandresponse.
As described by another respondent in Kenya, “We have been doing capacity building of[nGOs],capacitybuildingofthecountygovernmentinthesurveillance,andcapacitybuildingofcoordination.Thebiggerpartofthisprojectistobuildthecapacity.Andalso,capacitybuildinginpreparedness,capacitybuildinginresponse,capacitybuildingintheoverallthinking.”During
200 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
thecholeraoutbreakinthenairobislums,Oxfamprovidedtechnicalcapacitybuildingoveraperiodofseveralmonthstoorganisationsinvolvedintheresponseandtocommunitynetworksaffiliated with the Kenya Red cross, including health workers and volunteers. However, arespondentdescribed the challenges inbuilding capacity andmeasuring change in capacityinsuchashortperiod:“fromacholeraresponseperspectiveIthinkitisverydifficulttobuildcapacity inthreemonthsperiodotherthanworkingwitheachother, trusting, learningfromthat.Howyouthenmeasurethatpastthreemonthsisadifferentquestion.”
The most common capacity building activities mentioned among respondents in Ethiopiaincluded training provided by the Shifting the Power project on conducting rapid needsassessments,preparingproposals,andimplementingcashtransferprogrammes.Thecapacitybuildingextendedbeyondtheorganisationitself,asonerespondentinEthiopianoted“…theShiftingthePowerprojecthashelpedusa lot.Weareworkingwith thezonal task force inrelationtocapacitybuilding.Wehavenotkeptsilentafterourcapacityisbuilt;wehavegonedowntotheworedalevelandwehavegiventrainingsinordertobuildthecapacityofthestaff.”
IIb. Collaboration and Emergency Preparedness and Response
Collaboration strengthened emergency preparedness and response in the project target areas through improved communication, coordination, and resource efficiency.
collaborationsupportedimprovementsincommunication,coordination,andsharingofideas.As an example from theUEWEAproject inKenya, the formalDEPP consortium facilitatedmeetings, bringing together the organisational representatives and government officers, todecideandagreeoninputsandtheinterventionstrategy.“basically,thereisalotofsharing[of] technical ideasonwhatwillworkbest, onwhatwill notworkbasedonbothpartners’experiencesontheground…Soyouarenotworkinginisolation.”
TherolesofcollaboratingorganisationsandinstitutionsinKenyaaspartoftheUEWEAprojectwereclearlyoutlinedanddescribedbyrespondents.Organisationalmandatesandrolesweredescribedtobewellalignedwiththetechnicalexpertiseofeachorganisation.OnethemethatemergedduringtheanalysisoftheKenyainterviewsistheimportanceofincludingadvocacyrolesandresponsibilitieswithinthemandateoftheconsortium(i.e.,anorganisationalpartnerwhoisfocusedonadvocacy—whetherthisbeaformalroleoramediapartner).
local nGOs interviewed in Ethiopia developed a similar model of collaboration betweenthe organisations, government, and other nGOs; however, thiswasmore often through ahumanitariancluster-basedsystemratherthanaformalconsortium.collaborationallowedfordelegationofresponsibilitiesanddutiesamongorganisationsandorganisationalstaff.Italsoallowedforimprovedresourceefficiencyandavoidedduplicationofpreparednessandresponseactivities.AsdescribedbyarespondentinEthiopia:“Whenwesayitiscluster-baseditmeanstheapproachisnotsingle-based.InborenawewereworkingwithUnOcHA,cARE,Acf…wehavebeenworkingtogetherinordertoidentifytheareaswecollaborateandwewereworkingtogetherinordertoavoidduplicationofeffortsbecausethereisconstraintofresources…wehavetouseourresourcesinanefficientwayandwehavebeendoingthat.So,bycluster-basedwemeaneverystakeholderwasinvolvedandcollaboratedinthatoperationalarea.”
OneofthemostnotablesuccessesdescribedbyUEWEAprojectrespondentsinKenyawasthat the consortium structure facilitated the response to the cholera outbreak.While thefundingforthecholeraresponsewasseparatefromtheDEPPfunding,thestructureoftheconsortiumthatwasalreadyinplacefacilitatedatimelyresponseastheywereableto“injectintotheconsortiumtodoacholeraresponsethroughtheKenyaRedcross—throughthesameconsortiumsystem.”beyondfinancing,theconsortiumstructure,includingitsproceduresandmanagementstructure,facilitatedaneffectiveresponsetothecholeraoutbreak.
Whenaskedaboutchallengesrelatedtotheconsortiumapproach,respondentsinKenyanotedthattherewereinevitablechallengestobringmultiplepartnerstogether,includinggovernmentandcountyoffices.TheyalsodescribedthelengthoftimethatittooktogetagreementsandMOUs signed among partner organisations and government officials, especially consideringtheDEPPtimelineexpectations.Despitethesechallenges,therewasoverallconsensusthata consortium approach strengthened preparedness and response activities. As noted by arespondentinKenya:“It’s[theconsortium]alwaysgoingtocreatechallenges,anditwouldn’tbeinterestingifitdidn’t.butit’sclearlygoingtohavemoreimpactifyoulocateitwithincountystructuresbothintermsofrollout,butalsointermsofinfluencinginthefuture.”
201 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
IIc. Early Warning Systems and Emergency Preparedness and Response
Surveillance and early warning systems were fundamental to effective preparedness and response in the project target areas by providing data, systems, and procedures.
AsarespondentinEthiopiaconveyed:“Thefundamentalpillarofanyemergency—itcouldbeanemergencyoritcouldbedevelopment—youhavetohaveanearlywarninginformation[system].you have to have adequate information.There needs to be an information sharing systematacommunitylevel.”ArespondentinKenyaechoedtheimportanceofdata:“Preparednessrequiresdata.Itrequiresdataaboutasetofindicators…noonewouldhaveknownthatfoodinsecurityisarealdisasterinourinformalsettlementsinnairobi.It[data]alsotellsyouaboutwhosuffersor…whereisthepotentialofthisdisasterhittingmost.”
Surveillance Systems and Procedures
The surveillance system of theUEWEA project in Kenya comprised data collection on keyindicators,dataentry,andusedadashboardtohighlightalertsandtrends.Respondentsreportedthattheprocessofdeterminingindicatorswascollaborativeamongconsortiumpartnersandtooktime, nearly fouryears (the surveillance systemhadbeen indevelopmentprior to theDEPP.)Therespondents inKenyaalsodescribedsomeof the intricaciesneededtoconductsurveillanceonfoodinsecurityinanurbancontext.Asnotedbyrespondents,foodinsecuritytendstohaveasloweronset,asopposedtoasudden,naturaldisaster;thus,requiringadifferentsurveillancesystemandsetofindicators.
SimilarlyinEthiopia,therewassurveillanceintheformofneedsassessments.OnerespondentreportedthataftercollaboratingwiththeShiftingthePowerproject,needsassessmentswereincorporated intotheorganisations’responseactivities.Theseassessmentsaimedto includerelevant stakeholders, community, and government offices, such as the disaster risk andpastoral office.Theneeds assessments subsequently identified food insecurity,malnutritionamongwomenandchildren,contaminationofwatersources,andcompromisedhealthstatus.needsassessmentsandsurveillancesystemswerealsodescribedasprovidingmechanismsforcollaborationwithgovernmentpartnersandstrengtheningthesepartnerships.
Early warning systems: people, coordination, community engagement, and trust
Dataarenot theonly importantcomponentsofanearlywarningsystem.Respondentsalsoemphasised that these early warning systems require people, coordination, communityengagement, and trust. In the case of the UEWEA project, community ambassadors werecriticaltothesurveillancesystem.Asanexample,arespondentinKenyareported:“basedonthosevariablestheyareabletotellandtheambassadorsraisetheredflag.So,theredflagcanberaisedtothecountyofficialsthatwethinkthereisaproblem.So,itdoesn’tneedtowaitforthenationalgovernmentministertoannouncethatwehaveaproblem.”
Early warning systems require coordination. Having an established surveillance system isimportant,butitdoesnotguaranteeatimelyresponseasnotedbythisrespondentinKenya.“Theresponseisdependentonhowstrongisthecoordination…becausethetimeshouldbeshort…Aswemoveon,weneedtoimprovebecausethesuccesswillbehowtimelyitis.Andhowtimelyisdependentonhowthepreparednessisdoneverywell.becausethemoreyouprepare the lesstimeyouneed.So, themechanism [earlywarningsystem,surveillance]andotherthingshelpedalotbutstillnowthereisaroomto[improve].”
202 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
furthermore,communityinvolvementinthedesignandimplementationoftheearlywarningsystemiscrucial.ArespondentinEthiopiaprovidedanexampleofcommunityengagementinthesurveillanceactivities:“Everymonthmembersofthecommunitysitanddiscusslivelihoodissues of the community (availability ofwater, livestockmarket info, conflicts, anything thatrelates to livelihoods.” Several respondents acknowledged that engagement of communitymembersandstakeholdershelpedtofostertrustintheearlywarningsystemandsurveillancedata.
Emergingfromtheinterviewdata,itwasevidentthatrespondentsbelievedthatearlywarningandsurveillancesystemsallownotonly for improvedpreparedness,suchas inpredictionofeventsandprevention,butthattheyalsoprovideanopportunityandmechanismforbehaviourchange.Itisthisbehaviourchangethatwasnotedtoleadtomoreeffectiveresponseefforts,suchasnotedbythisrespondentinKenya:“Havingbeen[involved]inthesystems,mytakehasbeennowthatifwemovetoriskmanagementwhichyougettheearlywarningsignsandactonthem,earlyactionyouactonthemitslesscostly,yousavealotofresourcesandavoiddeaths.”
IId. Learning and Emergency Preparedness and Response
Respondents identified various platforms, including committees within projects and learning events within the DEPP that facilitated sharing of best practices on emergency preparedness and response.
Respondents in both countries shared examples of learning and sharing of best practices.However,itwasnotablethatrespondentsinEthiopiaattributedimprovedaccessandvisibilitytoabroaderplatformasaresultofDEPP.RespondentsinEthiopiareportedaccessingplatformswithintheShiftingthePowerprojectthatfacilitatedsharingandlearningamongorganisations.AsdescribedbythisrespondentinEthiopia:“Thereisahumanitarianforumwhichisestablishedby the Shifting the Power project.We aremember of the steering committee there, so allmemberswereaskedtosharetheirexperiences.”
Respondents inEthiopia alsodescribed the impactofDEPP learningevents asmechanismsfor learning: “These arevery important, preparedness and response is not enoughby itself,collaborationwithotherorganisations is important.Ourparticipation in learningeventsandengagingindifferentplatformshashelpedusalot.ThankstotheDEPPprogrammewehavebeenabletoparticipateindifferentplatformswhichweappreciate.becausewecanlearnfromeachotherintheseplatforms.Ithasgivenusanopportunitytoreflectourideasandalsoheardifferentstudiesfromotherparticipants.”
Whilenotdirectlyassessedduringtheinterviews,respondentsdidnotmentionanylearningandsharingbetweenDEPPprojectsinthesamecountry.
IIe. Enhanced Visibility of Organisations as a Result of DEPP
Working as an organisation within the DEPP programme also helped to enhance visibility and credibility of local organisations to donors, government, and other stakeholders.
Thetrainingandcapacitybuildingallowedforl/nnGOstobecomemorevisibletodonorsandotherstakeholders.Asanexample,theShiftingthePowercapacitybuildingactivitiesinEthiopiahelped localorganisationsevaluatetheircommunicationanddisseminationplatformsandtodevelopwebsitestocommunicatetheirprojectactivities.
One of the respondents in Ethiopia commented how the DEPP components, especiallycapacitybuildingandcollaboration,helpedtheorganisationgainvisibilityandrecognitionasanorganisation that could respond todisasters.Prior to the capacitybuildingactivities andcollaborationwiththeShiftingthePowerproject,theorganisationwouldbeinvitedtoattenddistrictmeetings;however,now,after thecapacitybuildingactivitiesandcollaborationwithlocalgovernment,theyarecallednotonlytodistrictmeetingsbutalsotoUnOcHAmeetings.
III. Levels of action – Examples of individual, organisational, and systems-level changes in emergency preparedness and response
IIIa. Individual-level Changes Related to Emergency Preparedness and Response
Emergency preparedness and response capacity building activities resulted in increased confidence among individuals to respond to a disaster.
203 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
WhilemostofthecapacitybuildingactivitiesaspartoftheUEWEAprojectandShiftingthePowerprojectfocusedonimprovingpreparednessandresponseattheorganisationallevel,manyrespondentsnotedimpactattheindividuallevel.forthoserespondentsreportingindividual-levelchanges,thecapacitybuildingactivitiesledtoimprovedskills,knowledgeofproceduresandresources,andconfidence.
Asanexample,arespondentinEthiopiarecountedthatwithadequateresourcesandtraining,confidence in emergency response increased such that “you now knowwhere to start andwheretoend”.ArespondentfrominKenyawhoreceivedtraininginsearchandrescueandfirstaid,reportedimprovedcapabilitytorespondandto“respondwithoutfear”.
IIIb. Organisational-level Changes Related to Emergency Preparedness and Response
Emergency preparedness and response capacity building activities resulted in strengthened procedures and systems at the organisational level
Allrespondentsidentifiedorganisational-levelchangesasaresultofcapacitybuildingactivities,withthemostdetailedexamplescomingfromtherespondentsinEthiopiawhoreceivedcapacitybuilding training from the Shifting the Power project.Themost common changes reportedincludedshiftsinmindsets, inter-departmentalcoordination,andstrengthenedorganisationalprocedures—in other words, “institutionalising preparedness”. Some of the organisationalchangesincludedcreatinganemergencyresponseteamwithintheorganisationwithdefineddelegationofresponsibilitiesandimprovedprocurementpolicies.Developmentoforganisationalemergencypreparednessplanswasreportedtobecriticaltoimprovedorganisationalcapacityto respond. The capacity building activities also helped beneficiary organisations to linkpreparednessandresponseactivitiestohumanitarianstandardsandguidelines.
AsdescribedbythisrespondentinEthiopia:“Afterwehaveimplementedtheprojectwehavealsostartedtothinkasateam.Thebigthingisthereisashiftinourmindsets...AfterShiftingthePowerwestartedtolook[at]thingsindetail.Weidentifiedourgapsandwewereabletoidentifytheareaswemustengage.Eventhoughthereisnostand-alonedepartment,wehaveorganisedandformedanemergencyresponseteam.”
AsanotherrespondentinEthiopiareflected:“Previouslytherewasnoemergencypreparedness,everythingwasdoneoncetheemergencyoccurredbutnowwehaveemergencypreparednessplansandjobdescriptionsofeachmemberofresponseteamsready-made.Hazardsareidentified.Whodoeswhatisidentified…Inthiscasewheneveremergencyoccursweareready.Itisnotdifficulttotransferfrompreparednesstoresponse.Wealreadyhavepreparedproposalsontheshelf.ThankstoShiftingthePowerwearepartoftheteamnowandwegettheinformationfromthenational,regionalandlocal levelsandourvisibilityhasincreasedandinformationisreadilyavailable.”
IIIc. Systems-level Changes Related to Emergency Preparedness and Response
Collaboration with the government resulted in numerous systems-level changes to improve emergency preparedness and response.
While the respondents in Ethiopia commonly reported organisational-level changes, therespondents inKenya reportednumerous systems-level changes as a result of theUEWEAproject.Asa respondent inKenyareported: “…[when] theconsortiumwastryingtoget intogovernment,totransferthistechnologytothegovernment,initially,theywenttotheassembly.AndtheymanagedtoconvincetheassemblytopassamotiontoestablishEarlyWarningEarlyActioncentresineveryward.
Arespondent inKenyaalsohighlightedhowtheprojectwasalignedwithbroaderplatforms,includingtheconstitutioninKenya:“Theotheractivities—thechanceofbeingintegratedinthecountyishigh.Why?‘causeitis[oneofthe]fundamentalactivitiesthatneedstobeaddressedbytheKenyanconstitution.KenyanconstitutionsaysanyKenyanswholivewhetherininformalsettlementsor ina townhasa right togetbasic services,whichwasnot thecasebefore…beforesixyears,theoldKenyanconstitution,informalsettlementsareinformalsoyouarenotsupposedtoprovidethoseservices.Sonow,theconstitutionisnowgivingthatasplatform…Soitisapath…itisnotanisolatedactivity.”
204 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
IIId. Emergent Themes on Project Sustainability
Government engagement was key to sustainability of emergency preparedness and response projects.
TheinterviewswereconductedastheprojectswerereachingtheendoftheDEPPtimelineinKenyaandEthiopia. Assuch,manyrespondentsreflectedonthesustainabilityofprojectactivities.
TherespondentsfromtheUEWEAprojectinKenyahighlightedtheimportanceofgovernmentengagement to project sustainability. As a respondent in Kenya reported: “This project isfinishinginMarch[2018]now.Thecountygovernmentisgoingtotakeitforward.Thecounty[representative]said,“Weareprivilegedtohavethisprogramme.Itshouldbeustogoandaskyou,canyoujoinus?butyoudidallthejobsthatwearesupposedtodo.Anditisourdutytotake it forward.formethat isbigsuccess.”Therespondentsfromthe localorganisationsin Ethiopia expressed concern about local staff being able to continue and provided fewerexamplesofprojectsustainabilityduringtheinterviews.
IV. Emergent themes on disaster preparedness and response
IVa. Innovation is Needed in the Definitions of Disaster, Preparedness, and Response
Definitions of disasters and approaches to emergency preparedness and response differ between urban and rural contexts.
Many respondents reported the need for innovation in how the humanitarian communitydefines disasters.As an example, poverty and food insecurity, especially in urban contexts,are rarelyconsideredasdisasters;but ratherasdevelopment issues,andassuchgetvariedlevelsofattention fromdonorsandotherstakeholdersduringemergencypreparednessandresponseprogramming.Onerespondentnotedthatpovertyandfoodinsecuritygetpackagedintodroughtandfamineresponseinaruralcontext,butnotinurbancontexts.AsarespondentinKenyanoted:“It’salsoverydifficulttoraisemoneyquicklyforshorttermresponseandurbanareasinparticular…they[thedonors]lovefundingpeopleinruralareas.”Thisalsohighlightsthatdisasterandinthiscase,drought,affectsruralandurbanpopulationsdifferentlywhereruralpopulationsdirectlyloseaccesstofoodwhereasurbanpopulationsmaystillhaveaccessbutathighercostswhichtheycannotafford.
AdifferentrespondentinKenyadescribedthechallengesofaddressingfoodsecurityaspartofpreparednessandresponseprogramming: “The foodsecurityaspectof it isabitnew…Indevelopingtheresponsepackage,wearestartingfromscratchbecauseeverysystemthatisthereinresponsetofoodsecurityisrural-based,soyoucannotdothesamethings-livestockthingsandthingstheydothereforlivelihood-thatdoesnotworkhere.”
OneofthenotedsuccessesoftheUEWEAprojectinKenyaandasystems-levelchangeisthattheprojectwasabletobringattentiontofoodinsecurityaspartofdisastermanagement:“…Rightnowtheyjustputitintothelaw.Disaster,countydisastermanagement,andemergencymanagement[act].They’vejustinsertedfoodsecurityaspartofthedisastersothatitattractstheattentionof,ofbudgeting,becausenowtheyhavetoimplementthatlaw.And[through]thatlaw,afund,thedisasteremergencyfundiscreatedthatwillfundpartoftheresponsesthatwillemerge.”
Respondents inEthiopiaalsodescribedchallenges inseekingfundingtoaddressdisasters inaruralcontext.comprehensiveresponseprogrammesinaruralcontextneedtoincludelife-savingmeasuresnotonly forhumanbeings,butalso for livestock.Arespondent inEthiopiareported that due to budget restrictions and donor preferences, it ismore difficult to seekfundingforlivestockprogrammesasopposedtothoseprogrammeswithhumanbeneficiaries.
IVb. Standards on Emergency Preparedness and Response
Knowledge of and adherence to humanitarian standards strengthened emergency preparedness and response.
Respondentsinbothcountriesdescribedtheimportanceofhumanitarianstandards,particularlythosefocusedonemergencypreparednessandresponse,toallowforacommonunderstanding,
205 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
framework, and measure. The Shifting the Power project was successfully noted to havestrengthenedcapacityamonglocalorganisationsondevelopmentofemergencypreparednessplansaccordingtodefinedstandards.AsarespondentinEthiopiadescribed:“Whenweworkonsavingthelivestockweareworkingusingthelivestockguidelinestandard.nowwehaveasateambecomeawareofthespherestandardandcorehumanitarianstandardandsoasateamhavingacommonunderstandingisbigachievement.”
IVc. Linking Emergency Preparedness and Response Programming
Emergency preparedness and response were described as disjointed activities that need to be linked for effective response.
In general, respondents highlighted the importance of linking preparedness, in this case,surveillance,withresponseactivities.Manyacknowledgedthatthetwoweredisjointedintheproject,withone respondent inKenyapointingout that theirworkandpartneragreementswerefocusedonlyonsurveillance,andnotonresponse:“Aquicksurveillance,aquickresponse.butmoreimportantly,inmyview,you’dsavelives.Thereisonesayingwehavewiththisproject,“Actearly,savealife”.Thatisourmotto.Actearly,savealife.So,whatwewantissurveillancetofeedintoresponseandpreferably,immediatelysothatnolifeislost.yeah,butthetwoasofnowaredisjointed.”
Respondentswereaskedtoprovidetheirperspectivesonwhetherpreparednessleadstomoreeffectiveresponse. Ingeneral, respondents identifiedthatthecohesionbetweenemergencypreparedness and emergency response needed to be strengthened.A respondent inKenyaacknowledgedthatitshouldbutthat itwasn’tyetdoingso,andtherearestepsbeingtakenin thatdirection.As this respondentnoted: “It should. Itshouldbut it isnothappening.forreasons,wearelearning.”Oneofthereasonsmentionedbyrespondentsisthatfundingsourcesareoftenseparateforemergencypreparednessandemergencyresponse,whichimpedesthetransitionfromemergencypreparednesstoresponse.
IVd. The Transition from Preparedness to Response
Key components to transition from preparedness to response included community engagement, response training, and systems.
ArespondentinEthiopiahighlightedtheimportanceofcommunityengagementinfacilitatingthetransitionfrompreparednesstoresponse.Asanexample,theircommunityRiskReductionApproachprovidescommunitymemberswithpre-disasterinformation,trainingonpreparedness,andtrainingonwhattodoifadisasteroccurs.
AnotherrespondentinEthiopiaalsonotedtheimportanceofnotonlypreparednessactivities,but also response training and reported that the Shifting the Power project has filled thegap between preparedness and response. before the project, the organisation only knewpreparednessand thatnow,after thecapacitybuilding training fromtheproject, theyknowwhattheyneedtodotorespond.
Respondents also identified the importance of having a system in place to transition frompreparednesstoresponsetofacilitatereactivityanddecision-making.However,preparednessplansareonecomponentofasystemthatisneededtofacilitateanemergencyresponseasarespondentinKenyanoted:“Thattransitionfrompreparednesstoactualresponseneedsaveryrobustsystem.Andyouhavetobereadyalsotochangesomekeythingsveryfastiftheydon’tworkintherealcasescenario.”
IVe. The Transition from Development Activities to Emergency Response
Capacity building activities on emergency preparedness and response strengthened organisations’ capacity to transition from development activities to emergency response activities.
InEthiopia,severaloftheDEPPprojectbeneficiaryorganisationswerefocusedondevelopmentandnotondirecthumanitarianaidprovision.AsaresultoftheDEPP,theorganisationsgainedcapacityonhowtorespondtoahumanitariancrisis.Eventhoughtheynotedthatdevelopmentactivities,suchasaddressingfoodinsecurity,werenotthatdifferentfromhumanitarianactivities,theyneeded tochange their systems.Asa respondent inEthiopianoted, they revised their
206 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
organisationalstrategy.Theychangedprocurementandpolicies,andalertedstakeholdersandgovernmentthattheyarenowimplementinghumanitarianactivities.ThetrainingfromShiftingthePowerhelpedtosupportthisprocessofexpandingwhatwaspreviouslyconsideredtobedevelopment activities and shifting the organisational strategy to now focus on emergencypreparednessandresponseactivities.
IVf. Measuring Success of Emergency Response
Respondents lacked clear measures by which to evaluate the success of emergency preparedness and response activities.
Respondents were asked to identify examples of successful emergency preparedness andresponseactivitiesandalsoshareexamplesoflesssuccessfulprogramming.Themostcommoncriteriaidentifiedwereatimelyresponseandtheabilitytominimisesuffering.Onerespondentin Ethiopia described: “One of the objectives of the responsewas to minimise death andhumansuffering…becausewerespondedontime,wewereabletominimisethesuffering.”Therespondentcontinued:“[We]dotheseactivitiestosavelivesandimprovewellbeingofhumanbeings.Whenweseetheeffectofdrought,itistheworst.Itcauseshugeloss.Visionistoseeaworldwherebyallhumanbeingsliveindignityandrespecteachotherinprosperity.”
Whilerespondentsinbothcountrieshighlightedtheimportanceofminimisinghumansuffering,theylackedaframeworkbywhichtoevaluatethesuccessofprogramming.fewrespondentswereabletoprovidepost-disastermonitoringframeworksordata.Onereasonprovidedwasthattherewerebudgetrestrictionsonemergencyresponseactivitiesthatdidnotincludepost-disastermonitoring.
KEY FINDINGS
SummaryThis qualitative case study provides an in-depth analysis of two DEPP-funded projectsoperatingintwodifferentemergencycontexts—theUrbanEarlyWarningEarlyActionintheslumsofurbanKenyaandtheShiftingthePowerproject inruralEthiopia. Inbothcontexts,thepopulationswereaffectedbypoverty,foodinsecurity,andpoorhealthoutcomesincludingmalnutrition and diarrheal disease.While therewere similar components to the emergencypreparednessandresponseactivitiesbetweenthetwoprojects,respondentsfromeachprojectoffereduniqueperspectivesonstrengtheningemergencypreparednessandresponse.Thecasestudy integratesperspectivesfrominternationalandnationalnGOsandcountygovernmentrepresentationpartofaformalDEPPconsortiuminKenyaandfromlocalnGOsbenefittingfromaDEPPfundedprojectinEthiopia.
Major Result AreasResultswereanalysedandgroupedintofourbroadcategories:
1. Identification of key project design and implementation factors that strengthenedemergencypreparednessandresponse
2. characterisation of theways inwhich key components of DEPP (capacity building,collaboration, early warning systems, and learning) strengthened emergencypreparednessandresponse
3. Description of the levels of action (individual, organisational, and systems level) ofemergencypreparednessandresponseactivities
4. Recognition of gaps between emergency preparedness and response activities andopportunitiestostrengthenthelinkageandtransition
Key Findings• communityandgovernmentengagementwasnotedtobeessentialtoprojectdesignand
implementation,especiallyinthedesignofthesurveillancesystem,responseactivities,andbeneficiaries.Importantly,engagingcommunitymembersandgovernmentofficialsfostered trust and strengthenedpartnerships that facilitatedproject implementation.
207 SUMMATIVEPHASEREPORT-HUMAnITARIAnPREPAREDnESSAnDRESPOnSEcASESTUDIES
Governmentengagement,especiallyfortheprojectinKenya,wasnotedtobekeyforsustainabilityandintegrationintofutureprogrammingbeyondtheDEPPtimeline.
• capacitybuilding,intheformofcommunityandorganisationaltrainings,strengthenedemergencypreparednessandresponsethroughimprovedawareness,coordination,andprocedures.Thesetrainingsalsostrengthenedawarenessofhumanitarianstandards.forlocalorganisationsinEthiopia,thesecapacitybuildingactivitieshelpedtostrengthenthevisibilityoftheorganisationsandalsothecredibilityoftheiremergencypreparednessandresponseactivitiestodonorsandotherstakeholders.
• collaboration, both formally through project consortia and informally throughhumanitariannetworks,improvedcommunicationandefficiencyofresourcestoavoidduplicationofactivitiesinthetargetedareas.TheconsortiumapproachfortheUEWEAprojectinKenyaprovidedaneffectivemechanismforrespondingtoemergentdisastersthataroseduringtheprojectduration,suchasacholeraoutbreak.
• Surveillanceandearlywarningsystemswerefundamentaltobothprojects,describedas the pillars of emergency preparedness and response. beyond collecting data,respondentsnotedthatthesealsoservedasmechanismsofcollaborationandprovidedopportunitiesforbehaviouralchangeforcommunitiesandorganisations.
• EngagementwithDEPPprovidedopportunitiesfororganisationstosharelearningandbestpracticesthroughwidernetworksandplatforms.DEPPlearningeventswerenotedtobeparticularlyhelpfulplatformsfor learningforthosewhoattended.Engagementwith DEPP improved visibility of organisations among key stakeholders, includinggovernmentanddonors.
• Attheindividuallevel,emergencypreparednessandresponsecapacitybuildingactivitiesresultedinincreasedconfidencetorespondtoadisaster;however,therewerefewerexamples of individual-level changes. At the organisational level, capacity building,especiallyamongorganisationsinEthiopia,strengthenedproceduresandpoliciessuchasthedevelopmentofemergencypreparednessteamsandplans.collaborationwiththegovernmentfortheprojectinKenyaallowedfornumeroussystems-levelchanges,includingalignmentandintegrationofprojectemergencypreparednessandresponseactivitieswithbroaderplatforms,suchasconstitutionalprioritiesandlocalandnationalpolicies.
• Definitionsofdisastersdifferedbetweenurbanandruralcontexts.Povertyandfoodinsecurity,especiallyinurbancontexts,arerarelyconsideredasdisastersandgetvariedlevelsofattentionfromdonorsandotherstakeholders.
• Emergency preparedness and response activitieswere described by respondents asdisjointed. They described a need for a system to link preparedness and response.facilitators of this transition from preparedness to response include communityengagement, response training, and organisational procedures and standards. Theneed to seek separate funding for response activitieswas described as a barrier totransitioningfromemergencypreparednesstoresponse.
• When asked to evaluate the success of emergency preparedness and responseprogramming,respondentslackedclearmeasuresorframeworksbywhichtoevaluateeffectiveresponseprogramming.
Overall Conclusion of the Two Case Studies
TheSouthSudancasestudydemonstratedthatispossibleforprojectstooperateinhigh-risk,conflict-affectedcontexts;however,securityprotocols,flexibleandadaptivemanagementandbudgets,contingencyplans,aswellasengagementandintegrationoflocalpartnersandstaffarevital. While the case study inSouthSudan found that engagementof thegovernmentstakeholderswas not always possible, the case study in Ethiopia and Kenya demonstratedthat theconsortiummanagementmodelandcollaborationwithgovernmentwerecritical tosuccessfulimplementationofearlywarningsystemsandpermittedcoordinatedhumanitarianresponses that improvedwith each subsequent emergency. both case studies stressed theimportanceofhavingflexiblebudgetsandrapidfundingmechanismseithertomakechangesinoriginalprojectplannedactivitiesortosupportresponseactivitieswhenadisasteroccurs.
208 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
CONCLUSIONS &RECOMMENDATIONS
cHAPTER9
209 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
OVERALL CONCLUSION TheDEPPwasa3-year,£40millionhumanitariancapacitydevelopmentprogrammefundedbytheDepartmentforInternationalDevelopment(DfID).Itincluded14distinctprojectsaimingtobuildhumanitariancapacityoflocalactorsusingacollaborativeapproachin10countriesatriskforbothnaturalandmanmadedisasters.Theprogrammewasdesignedin2014,andwasaheadofitstimeinmanyways,especiallywithitsstrongfocusonlocalisationandmulti-stakeholderconsortia. It includedamixofbothflagshipprojectsusingmore traditionalapproaches (i.e.,in-personindividualcapacitybuilding)withsomemoreinnovativeelements(i.e.,a£10millionInnovationWindow;flexiblefundingmechanismstofinanceboth localcapacitybuildingbutalsoemergencyresponse;pooledsurgeplatforms;urbanearlywarningsystemsandinclusionoffoodsecurityasanemergency).
Asitsufferedfromanumberofdesignchallengesincludingathree-yeartimeframethatwasunrealistictomeettheobjectives,anduniversalimplementationdelays,thedecisionbyDfIDtono longerfundtheprogrammelimitspotential long-termimpactoftheDEPP.Regardlessof the fundingdecision,DEPP’s focusonemergencypreparednessand localisation is still arelevantone.Italignswellwiththegrowingdialogueamongtheglobalhumanitariancommunityandwithmore recent policy commitments such as theGrand bargain and thecharter forchangewhichadvocateforincreasedvoiceforlocalactors,andforrebalancingofpowerandrelationshipswithintheexistinghumanitarianarchitecture.
Despite its initialdesignflaws,positivefindingsrelatedtothecollectiveactionoftheDEPPand to individual DEPP projects have been documented. These include the following keyachievements:
• based on project reports, 33,388 individuals were exposed at varying degrees tocapacitybuilding
• Stakeholders across the programme noted that themost significant change due totheDEPPwasachangeinattitudestowardandincreasedsupportoflocalisation(i.e.,contributingtotheGrandbargain)
• changesincapacityoflocalorganisations(intermsofsystemsandprocesses,fundraising,conductingneeds assessments) and their preparedness levels, that led to outcomessuchasbecomingeligibleforUnpooledfunding,successinsecuringexternalresponsefunds,andbeinggivenleadershiproles
• Significant changes in organisational policy with respect to inclusion of vulnerablegroups(Ethiopia,localorganisationsacrossallintensivesetevaluationcountries),andlocalisation(Ethiopia)
• Someevidenceofstrengthenednetworksandincreasedcollaboration(Ethiopia,Kenya,thePhilippines)
• contributiontoatleast42humanitarianresponsesin11countriesandsomequalitativeevidenceinseveralsettings(ThePhilippines,Kenya)ofimprovedspeed,efficiencyandinclusivenessofresponse
• cross-countrylearningintermsoftheamountoflearningdocumentsproduced,andsharingofideasacrosscountriesandprojects
• Strongcross-projectcollaborationinthePhilippinesdemonstratingbenefitsofcollectiveaction
nevertheless,agreater impact,asoriginallyenvisioned,could likelyhavebeenachievedhadtherebeenamorecohesivedesign,alongertimeframe,amorerobustprogrammeM&Esystemandamorestrategicapproachwithregardstothetypeofcapacitybuildingimplementedateachlevelofaction(individual,organisational,community,systems)andthebalancebetweentheselevels.Overall,theexternalevaluationfoundthefollowingfindingsthatneedtobeexaminedfurtherinfutureevaluationsandprogrammingrelatedtoemergencypreparedness:
210 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
• noquantitativeevidence todate that the three-yearprogramme led tomeasurableimpactwithregardstomoreefficient,andtimelyhumanitarianresponse
• noquantitativeevidencetosupportthehypothesisthatstrengthenednetworkscanlead to improved emergency preparedness and response (mainly due to the shortobservationperiodavailabletoassessthis)
• noquantitativeevidencethatDEPPimprovedindividualknowledgeoncorehumanitariancompetencies,preparedness,inclusionofvulnerablegroups,orprotectionissuesinthefourintensivesetcountries
• fewchangesinquantitativepolicy-relatedindicators,thoughthereweresomeexamplesofimprovedpoliciesdocumentedbyprojects
• Toowideofavarietyofindividualcapacitybuildinginitiativesandlackofstrategyontypeofcapacitybuildingapproachesateach level (individual,organisational,system,community)andeffectdesiredateachlevel
Asnotedabove,thethreeyearsallottedforanemergencypreparednessprogrammetoachievefivemacroresultsin10countrieswasanambitioustimelineandagenda.ThelevelofexposuretotheDEPPprogrammeactivitieswasunevenlydistributedacrossthe10DEPPfocuscountries;eachcountry receivedonlyasubsetof the14projects.forexample, threeDEPPcountriesbenefittedfromsixprojectseach,whiletwocountrieshadonlyoneprojecteach(notincludingthelearningProject;seeTable4.7forthefulllistofprojectnumberspercountry).ThisunevenlevelofexposuretoDEPPcompoundedwithvaryingcountry-specificcontextualfactors ledtovaryingresultsinspecificcountries.HadtheDEPPbeenabletocontinueforanotherthreeyearsormoreandbeenabletointegrateitslearningsintoarevisedprogrammestrategy(i.e.,permittingtimetorestructureitsmanagementstructure,scaleupprojectsthatshowpromiseand reviseprojects that are less promising), therewouldbe amore feasibletimeperiod toobtainmoreconclusivefindingsaboutDEPP’sachievementstowardsitsfiveresultareas.
FINDINGS AND THE PROGRAMME THEORY OF CHANGE
TheevaluationwasnotbuiltaroundtheoriginalprogrammetheoryofchangeaspresentedinthebusinesscaseduetothefactthatitdidnotalignwellwiththetypesandscopeofDEPPactivitiesthatwereapprovedforimplementation.However,itmaystillbeworthwhiletoexaminetheDEPPfindingsagainstthattheoryofchangeandtoassesswhethertheassumptionsmadeduring thebusinesscasedevelopmenthavebeenproventrue.Table9.1presents the threekeyDEPPactivitiesintheoriginaltheoryofchange(capacitybuilding,improvingpreparednesssystemsforcommunities,establishingmulti-stakeholderplatforms),andtheexpectedoutputs,shorter-termoutcomesandlonger-termoutcomesagainsttheresultsthatwereachievedbytheDEPP.Table9.2presentstheassumptionsthatwerearticulatedintheoriginaltheoryofchangeandtheavailableevidencecorrespondingtoeachassumption.
Ascanbeseen,theevaluationfindingsdemonstratethattheoriginalprogramme-leveltheoryofchange(aspresentedinthebusinesscase)couldnotbeprovenordisproven.Mostoftheassumptionswerenotmetandanumberofcomponentsoftheoriginaltheoryofchangewerenotimplementedasoriginallydescribed.forexample,“multi-stakeholderplatforms”comprisea largepartof the theoryofchangeandweredescribed in thebusinesscaseas “platformswithabroadmembershipe.g.,InGOs,nationalnGOs,localgovernment,nationalgovernment,academia,privatesectorcompanies,theRedcrossRedcrescentMovement,andUnagencies.Successful platforms should improvepolicy atnational and international level.Weenvisageasingleplatformpercountryor regionwhere theprogramme isactiveandan internationalplatform. Possible products of the platforms are: synthesised lessons identified, jointevaluationsandcommonpositionpapersforadvocacy.”Theseplatformswerehypothesisedtobethemechanismthroughwhichpolicychangeandevidencewouldoccur.TheDEPPfocusedmuchmoreonconsortiaratherthanonlargermulti-stakeholderplatformsasdescribedinthebusinesscase(thoughsomeprojectsasTransformingSurgecapacity,cDAc’scommunicatingwithcommunitiesprojectdidincludethis).Similarly,advocacywasnotaslargeofaprogrammeemphasisasoriginallyintended.Ultimately,becauseofthedifferencesinwhatwasimplementedandthefactthatmostoftheassumptionsdidnothold,itisimpossibletoproveordisprovethetheoryofchange.Itshouldberevisited,revisedandre-testedinanyfutureprogrammedesignprocess.
211 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
DETAILS RESULTS LESSONSAC
TIV
ITIE
S
1. Capacity Building
capacitydevelopmentinterventionsreachnationalactors(nationalstaffofInGOsandnationalpartners,governmentstaff)andimprovetheorganisationalandsocialenvironmentnecessaryforeffectiveresponse
33,388individualswereexposedtocapacitybuilding,butequalgendertargetnotmet
•levelofexposurewasvariable(manyshorttrainingsorwebinars) •focusonquantityoverquality •Didnotmeetequalgendertarget •Accessibilityissuesrelatedtolackofavailabilityoftrainingsandmaterialsinlocallanguages •comparisongroupalsoexposedtocapacitybuildingindicatingthattherearemanyindividual-levelcapacitybuildinginitiativesbeingimplemented
Organisationalandsocialenvironmentnotconsistentlyorsystematicallytargetedthroughactivities
•Structuralbarriersrelatedtoorganisationalandsocialenvironmentnotadequatelytargeted;difficulttoachieveorganically
Raisingawarenessofgoodpracticeinpreparednessandresponse
learningevents,conferences,disseminationoflearningproductshelpedraiseawareness
•Amoresystematic,strategicawarenessraisingapproachmighthavebroadenedreachandbeenmoreimpactful
Enhancingskills(personalandtechnical)
Manycapacitybuildingprojectsfocusedonskillbuilding(learningbydoing,simulationsetc.)
•Structuralbarriersatorganisationswhichimpedestafffromapplyingnewskillsnotadequatelyaddressed
Improvingteamwork
Activitiesemphasisedstrengtheningcollaborativewaysofworking
•Perhapstoomuchemphasisoncollaborationwithoutenoughconsiderationformorenuancedapproachesandcontextualdifferencessuchaskeyinfluencersinthecountry-specifichumanitariannetworks
•Therewaslimitedcapacitybuildingonhowtoeffectivelycollaborate
fosteringcoalitionsandnetworks
Activitiesfocusedonstrengtheningcoalitionsandnetworks
•Greateremphasisonconsortiastrengtheningthanthebroaderhumanitariannetworks
Advocacystrengthenspolicy
Advocacyeffortswerelimitedandtherewasnoprogrammewidestrategy
•Mainstreamingofadvocacyandpolicyeffortsacrossprogrammemighthaveenhancedresults
212 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ACTI
VIT
IES
2. Projects to improve preparedness systems for response for communities at risk of disaster
Projectstoimprovepreparednesssystemsimplemented
4projectsfocusedondevelopingorimprovingpreparednesssystemsforcommunities
•Diversegroupofprojects(includesurbansettings,publichealthsurveillance,earlywarninginconflict-affectedareas)
3. Multi-stakeholder platforms established to enable collective action for capacity development
Enabledcollectiveactionforcapacitydevelopment
collaborativestructuresweresupportedacrosstheprogramme,includingmulti-stakeholderplatformsinseveralsettings
•Multi-stakeholderplatformswerenotemphasisedasmuchasconsortia-basedapproachesandmoregeneralcollaboration
•Multi-stakeholderplatformswerenotimplementedineverycountryorregionasarticulatedinthebusinesscase
capturedlessonsandevidence
Over600learningproductswereproducedacrossDEPP,butqualitywasvariable
•focuswasonquantityoverqualityandtherewasminimalempiricalevidence
Advocatedforchangebasedonemergingevidence
Someadvocacyeffortswereimplementedbutnosystematicadvocacycomponentattheprogrammelevel,andnotnecessarilybasedonevidence
•Advocacystrategyshouldbeincorporatedatprogrammelevel
OU
TPU
TS /
SH
ORT
-TER
M O
UTC
OM
ES
1. Improved knowledge and understanding of people in the system regarding best practices for humanitarian preparedness and response
Improvedknowledgeandunderstanding
noquantitativeevidenceofimprovedknowledgeoncorehumanitariancompetencies,preparedness,inclusionofprioritisedgroups,VAWG
Qualitativedatasuggestssomeimprovementinknowledge
•Toomanydifferenttypesofcapacitybuildingstrategiesused.focusshouldinsteadbeonknowledge,skillbuildingandretentionandshouldconsidermulti-prongedapproaches
2. Increased number of coalitions and partnerships developed
Increasednumberofcoalitionsandpartnerships
IncreaseinnumbersofcoalitionsandpartnershipsacrossDEPP
EvidenceofstrengthenednetworksinEthiopia,Kenya,thePhilippines
•Insufficienttimetotestthehypothesisthatstrengthenednetworksleadtoimprovedpreparedness
213 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
OU
TPU
TS /
SH
ORT
-TER
M
OU
TCO
MES 3. Emerging evidence
base for what works in building humanitarian capacity
Evidencebaseestablished
Evidencebasethathasbeencreatedismainlylearning-canteredbutnotforprojectmonitoringandimproving
notenoughempiricalevidencegeneratedthroughouttheprogrammecyclebyindividualprojects
•limiteduseofevidencebaseandlesssoatcountrylevel
•AccessibilityofevidencegeneratedwasalsoproblematicasmostproductswereonlyavailableinEnglish
4. Improved institutional and policy environment for building humanitarian capacity
Improvedinstitutionalandpolicyenvironment
Somequalitativeevidenceofstrengthenedpolicybutonlyinsomesettings
•Insomecases,multi-stakeholderplatformscontributedtothese,inothercasesitwasgeneratedbytheprojects
5. Improved preparedness systems for communities at risk of disaster
Improvedpreparednesssystemsforcommunitiesatriskofdisaster
5of6preparednesssystemsarefullyoperational
•Earlywarningsystemsrequirealongertimeframetodevelopandimplement •Thetwosystems(one-thirdofallthesystems)thathavebeentestedand/orusedinresponsehavedemonstratedimprovedpreparednesssystems
LON
GER
- TER
M
OU
TCO
MES
Improvement in effective delivery of humanitarian assistance to disaster affected communities by national individuals and organisations
Improvedeffectivedeliveryofhumanitarianassistance
Somequalitativeevidenceofimproveddeliveryofhumanitarianresponseinafewsettings
•Timeframetooshorttoachievethis •limitedaccesstoresponsefundshindered/preventedpotentialresponses
DiD NOT AcHIEVE
PARTIAllyAcHIEVED
fUllyAcHIEVED
cAPAcITybUIlDInGAnDDOWnSTREAMOUTPUTS/OUTcOMES
PREPAREDNESS SySTEMSAnDDOWnSTREAMOUTPUTS/OUTcOMES
MUlTI-STAKEHOLDER
PLATFORMS AND DOWnSTREAMOUTPUTS/OUTcOMES
Table9.1:Assessmentoftheoriginalprogrammetheoryofchangeagainstfindingsfromtheevaluation
214 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
DETAILS RESULTSA
SSU
MPT
ION
S
Capacity sufficiently improved to meet humanitarian standards relevant to target group
•Datasuggestthatthecapacitybuildinginterventionsdidnotleadtosignificantquantitativeimprovementsinindividualcapacity
Trained staff and local leaders with enhanced skills participate and remain in sector
•Staffandlocalleaderswereexposedtotrainingandthereisnoevidencetosuggestthattrainedstaffandlocalleadersleftthehumanitariansector
Platforms are the appropriate mechanism to strengthen vertical and horizontal practice, learning and accountability networks
•Somedatafoundthatcollaborativestructures(i.e.,consortia)cancontributetoimprovedpreparedness,responseandlearning
•notenoughevidencearoundplatformsandaccountabilitynetworks
Platforms and emphasis on M&E allows project learning to be taken up across both organisations and the system
•ProgrammeM&Ewasweak
•Someevidenceoftakeupofprojectlearningwithinorganisations
•limitedevidenceoftakeupacrossthesystem
•UnabletotesttheassumptionthatplatformsplusstrongM&EcontributetotakeupofprojectlearningduetotheweakM&E
Platforms improve collective advocacy or higher-level change (e.g., organisation and policy)
•challengesinsettingupandsustainingmulti-stakeholderplatforms
•Multi-stakeholderplatformswerenotaswidelyimplementedasoriginallyintended
•Platformscontributedtoorganisationalchange,butlimitedevidencethattheycontributedtoimprovedadvocacy
Sound analysis of hazard and differential capacity / vulnerability risks form basis of projects
•needsassessmentsandinvolvementoflocalstakeholdersandbeneficiariesdidnotoccurduringdesignphase
Projects inclusive of all relevant population groups under good local leadership
•Projectsfacedchallengeswithinclusionofbothwomenandofprioritisedgroups(bothintermsofdesign,implementationandinclusivenessofactivitiesandcapacitybuildingefforts)
DiD NOT AcHIEVE
PARTIAllyAcHIEVED
fUllyAcHIEVED
Table9.2:Assessmentoftheassumptionsoftheoriginalprogrammetheoryofchange
RECOMMENDED ACTIONS FOR PROGRAMME COMPONENTS
Given the key evaluation findings, there is scope to replicate and further test, or scale upsomecomponentsoftheprogramme.ThiswillensurethatthepreviousinvestmentinDEPPissustained,andthatanyadditionalresourceswillbeusedeffectively,efficientlyandstrategically.Ascanbeseen,noneoftheDEPPprogrammecomponentshavebeenrecommendedtobeceasedandnotbepursuedinthefuture.ThisinitselfisastrongindicationthattheDEPPcoretheoryofchangeandinitialassumptionsareworthyofbeingre-examinedforfutureinvestment.Specially,ninecomponentswererecommendedtobemodifiedandre-tested,whileeightwererecommendedtobereplicatedindifferentcontextsandpotentiallyonalargerscale,enablingmore rigorousevaluations.note that inaprogrammeeachof thesecomponentswouldnotbeexpectedtofunctioninisolation,andtheinteractionbetweencomponentsshouldalsobeconsidered.
215 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
DEPP PROGRAMME COMPONENT RECOMMENDED ACTION KEY FINDING / LESSONS LEARNED
Individual Capacity Building Modifyandre-test
• Prioritisemulti-prongedapproaches(in-person+coaching,mentoring,simulations)
• Qualityoftrainingsshouldbeprioritisedoverquantityofindividualstrained(webinarscanreachalotofpeoplebutmaybelesseffective)
• considerstructuralbarrierswithinorganisationsthatmayimpedeapplicationoflearning
Organisational Capacity Building Modifyandre-test
• Prioritiseapproachesthatpermitorganisationalself-assessmentsandcapacitybuildingapproachestailoredtoorganisations’needs
• focusonstrengtheningofadministrativepolicies,proceduresandsystems(i.e.,humanresources,procurement,financesystems)hasbeeneffectiveforl/nnGOs
• Approachesshouldbebetteradaptedtodifferentcontexts
• Accesstofundingiskeyfororganisationstoputlearningintopractice
• conductcomparativeassessmentsoftheeffectivenessofdifferentapproaches
Community Capacity Building Modifyandre-test
• fewprojectsaddressedthis,butthosethatdidindicatedsomepotential;furtherevidenceisneededonhowbesttobuildcommunitycapacity
Systems Capacity Building Modifyandre-test
• fewprojectsdirectlyaddressedthis,buttherehasbeensomeevidenceofqualitativeattitudinalchangethatcouldbeaprecursortoeventualsystems-levelchange;moretimeandfurtherevidenceareneededonhowbesttobuildcapacityatthesystemslevel
Early Warning Systems Replicateandtestonlargerscaleindifferingcontexts
• UEWEA’surbansurveillancesystemshouldbetestedinotherurbansettings
• Otherearlywarningsystemprojectscouldbeadaptedforandimplementedinothercontextswithsufficientprojecttimelines
Pooled Surge Platforms Replicateandtestonlargerscaleindifferingcontexts
• Pooledsurgeplatforms,especiallynational-levelplatformshaveshownsomeevidenceofeffectiveness.Pooledsurgeplatformsshouldensurealocallydrivendesignprocessandimplementationtoensurecontextualisationandlocalownership
• Replicateandtesttheseindifferentcontextstogeneratemorerigorousevidenceontheireffectiveness
216 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
Flexible Funding Mechanisms
Replicateandtestonlargerscaleindifferingcontexts
• flexiblefundingmechanismswerefoundtobeapromisingapproachtosupportl/nnGOsself-directedcapacitybuildingefforts.Inaddition,flexiblefundswereessentialtotranslatingemergencypreparednessintotimelyandefficientresponseactivities,especiallyforl/nnGOs
• flexiblefundingmechanisms(bothforcapacitybuildingandresponseefforts)shouldbereplicatedinothersettings,andmorerigorouslyevaluated
Multi-stakeholder Platforms Modifyandre-test
• Theseplatformsweremoreeffectiveincontextswhereexistinghumanitarianstructuresweremorematureandwithsupportinggovernments.
• furthermodificationandtestingoftheapproachisneededtounderstandhowbesttoutilisethisstrategyandinwhichspecificsettings
Consortia Modifyandre-test
• consortiathatbuildonexistingrelationships,commonvaluesandwaysofworking,andthataresmallerwithinclusionofl/nnGOsshouldbefavoured
Strengthening Networks Modifyapproachandre-test
• Useknowledgeanddataoncountrynetworkstocreatemoretargetedapproachestostrengthennetworks;allowenoughtimetotestwhetherstrongernetworkscanleadtoincreasedemergencypreparedness
Advocacy and Policy Mainstreamacrossprogramme
• Someadvocacyeffortswereimplementedbuttherewasnosystematicadvocacycomponentattheprogrammelevel.Itwasrecognizedthatadvocacyandpolicychangeareneededtoovercomebarrierstochangeandtoincreaseprogrammesustainability
• Ensureacohesiveprogramme-widestrategyrelatedtoadvocacyandpolicy
Learning Project Modifyandre-test
• Theapproachofhavinganindividualprojectdedicatedtolearningandgeneratingevidenceisuniquebutfindingsweremixedduetothelackofprogrammemonitoringandthelackofadirectlinktothemanagementteam.Alternativemodelsforanindependentlearningprojectcouldbeenvisionedandpotentiallyassessedinafutureprogramme
Evidence Generation Modifyapproach
• Prioritisequalityoverquantity
• Includeempiricalevidencegeneration
• Adaptevidenceuseandknowledgetranslationstrategies
Protection Mainstreamacrossprogramme
• Ensureacohesiveprogramme-widestrategyrelatedtoprotection
Inclusion of Prioritised Groups
Mainstreamacrossprogramme
• Ensureacohesiveprogramme-widestrategyoninclusionofprioritisedgroups(suchaswomen,theelderlyandpeoplewithdisabilities)
217 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
Localisation contextualiseandreplicateandtestonlargerscalein
differingcontexts
• Definelocalisation(whatitis,whichentitiesareconsideredlocal);andtestapproachestoenableInGOstotakeamoresupportiveroletowardsl/nnGOs,andbetteroperationaliselocalisationindifferentcontexts
DO NOT PURSUE
FURTHER
MODIfyAnDRE-TEST
cOnTExTUAlISEAnDREPlIcATEInAnOTHER
SETTiNG OR MAiNSTREAM AcROSSPROGRAMME
Table9.3:RecommendedactionsforeachDEPPprogrammecomponent
Inaddition,anumberofbestpracticesrelatedtotheDEPPprogrammecomponentshavebeencompiledtoguidefutureprogrammedesignandimplementation.ThesebestpracticescanbefoundinAnnex11.
SPECIFIC CONCLUSIONS AND RECOMMENDATIONS(SUMMATIVE PHASE) This section presents specific conclusions and recommendations related to each evaluationquestion.
1. RELEVANCE AND VALIDITY OF DESIGN CONCLUSION 1: WhiletheobjectivesandtargetsoftheDEPPwereappropriate,relevantandalignedwithDfIDpriorities,theDEPPdesignprocesssufferedfromnumerousweaknessesthathindered itspotentialfor impact.AnyfutureprogrammewouldrequireasignificantredesignbasedonthelessonslearnedandtheevidencegeneratedfromtheDEPP.forfuturedesign,programmeresultsthatarerealisticwithinathree-yeartimeframemustbeset,ortheprojecttimeframeshouldbeextendedtoallowformoretimetomeetlonger-termoutcomes.Amorelogical,coherentdesignprocess,which is locally led,drawsonneedsassessments,andusesparticipatoryapproachesshouldbeprioritisedtoensurethatprojectsarecontextualisedandbasedonexistingneedsatthecountrylevel.Thedesignprocessshouldadequatelyconsiderprojectalignmentandcomplementarityaswellasrisksattheprogrammelevel.Itshouldalsoensurethedevelopmentandimplementationofprogramme-levelsystemsandprocesses,suchasarobustM&Esystem,toensurecoursecorrectionandthatprojectsfunctionasaportfoliorather than as standalone projects. This includes elaborating a programme-level theory ofchange,definitionofkeyterms,cross-projectlinkages,andstreamlinedinclusionofgenderandprioritised groups alongwith appropriate considerationof cross-cutting themes. Emergencypreparednessandresponseprojectsshouldalsoconsiderurbancontextsand issuessuchaspovertyandfoodinsecuritywhichlieatthehumanitarian-developmentnexus.budgetsshouldincludehigherallocation fornPAcs,programmemanagement functionsandM&Eacrossallprojects.
RECOMMENDATION 1.1: [DEPP PROGRAMME BOARD AND DFID] The design process should be re-envisioned to consider alternative programme models and governance structures. To increase the local relevance and effectiveness of theprogramme, the design process should be locally driven from the outset (i.e., led by
218 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
localstakeholders),andincludesufficienttimeandfundsforprojectdesign,localisation,consortiadevelopmentandcollaboration.
Potential Model 1: The process could involve several stages: After the overallprogrammatic business case and theory of change are developed, global goalscould be established and local andnational ownership prioritised.A programme-level focal point could be appointed in each country to conduct contextualisedneedsassessmentswhichwould feed into thedevelopmentof theportfolio-levelstrategy.Theproject designprocesswouldoccur at the country level, under theoversightofthefocalpointsandtheProgrammeboard.Thisapproachwoulddrivethe localisation agenda and allow for country-based organisations to determinethemostneededinitiativeswithinthegoalsoftheoveralltheoryofchangeaswellasriskassessment.Itwouldhelpalignprojectstowardcommonprogrammegoals,identifycomplementaryelementsofdifferentprojects,mainactorsandthosethatneedtobebolsteredaswellasmoreriskyprojectsworthyofsupport.Itwouldalsoensuresufficient investment ineachcountry inordertoaccomplishthetheoryofchange,andwouldenablemorecomprehensiveassessmentsofrisks.Also,areturnoninvestmentassessmentcouldbeconductedtoinformthefinalinvestment.focalpointscouldworkwitharegionalorcountryM&EandlearningAdvisortodevelopandimplementprogramme-andproject-levelM&Esystems.Asaprogramme-levelactor, the focalpointwouldhelpbuildcohesivenessandenhance thevisibilityofDEPPasaunifiedprogrammeineachcountry.
Potential Model 2: Alternatively, consider a multi-phase programme in which aseriesofpilotinterventionsaredevelopedandimplementedonasmallscalewithstrong but targeted M&E systems integrated to evidence changes and provideaccountabilitymechanisms.Successfulpilotinterventionscouldbescaledupduringthesubsequentphase,buildingonthelessonscapturedduringthepilot.Astrongprogramme- and project-levelM&E system during the scale up phasewould beintegraltomeasuringchangeandensureregulartrackingofoutputstomodifytheprogrammewhenneeded. Inaddition, this typeofmulti-phaseprogrammecouldincludeapre-pilotphasewhereseedfundingisprovidedtodeveloppromisingideas,toconceptualiseprojectsandtobuildconsortiaaroundtheprojects.
RECOMMENDATION 1.2: [DEPP PROGRAMME BOARD AND DFID] The programme portfolio should be balanced in terms of risk, types of activities, size of projects, geographic scope and cost. To maximise impact, the link between emergency preparedness and response should be better articulated; preparedness programmes should strongly consider embedding flexible funding mechanisms in order to facilitate contribution to humanitarian response by local actors.
Theportfolioshouldincludeamixofmoreestablishedideasandprojects,thosethatareconsideredriskierandwithahigherlikelihoodoffailure,aswellasinnovativeelements.Prioritisinghumanitarianneed,examiningcountrydisasterandemergencyprofiles,aswellasaligningwithoverallprogrammestrategyduringcountryselectionprocesswouldensureamoreoptimal,balancedsetoffocuscountries.forafundofthissize,concentratingonasmallergroupofprojects(atleastinanyscaleupphaseofamulti-phaseprogramme),andpotentiallyfeweroverallcountriesmayhelptoensurethatinvestmentsarenotspreadtoothinandthatthereisbothsufficientinvestmentandalargeenoughnumberofprojectsineachcountrytoreachacriticalmassandachievesystems-levelchanges.Acomplementaryandstrategicmixofactivitiesattheprogrammelevelandespeciallyatthecountryleveliswarrantedtomaximise impacts;but this tooneeds tobefocused.forexample, theDEPP’sopenapproachledtotoomanydifferenttypesofcapacitybuilding,toogreatanemphasisonindividualcapacitybuilding,andlackofconsensusondesiredeffectsateach
219 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
level.Instead,amoredeliberateandevidence-basedglobalcapacitybuildingstrategyisneeded,withmorebalancebetweenlevelsofaction(individual,organisational,systems),aswellasconsiderationofstructuralbarrierstochangethatmighthinderbehaviourchangeor operationalising organisation-level changes. Inmany cases advocacy, policy changeorother system-based approachesmaybe critical to easing someof theseobstacles;integratingaprogramme-wideadvocacyorpolicycomponentshouldbefurtherexplored.Emergencypreparednessmustnotbethoughtofinisolationandmechanismstoembedlinkagestohumanitarianresponse(includingresponsefunding)areneeded.
RECOMMENDATION 1.3: [PROGRAMME BOARD AND DFID] Portfolio-level harmonisation should be prioritised, including mainstreaming of gender and inclusion of prioritised groups.Toensurealignmentoftheprojectsandacohesiveprogramme-wideapproach,portfolioobjectives,aswellaskeyterms(e.g.,collaboration,preparednessandlocalisation),andapproaches(e.g.,typesofcapacitybuildingefforts)shouldbefullydefined,andthedefinitionsstandardisedacross theprogramme.cross-cuttingthemesthatareprioritisedbythedonorand/orprogrammeandprojectstakeholders, suchasgenderandaddressingVAWG,mustbegivenadequateattentionandembeddedacrossthe programme. Inclusion of prioritised groups and gender considerations are equallyimportantandshouldbemainstreamed.
RECOMMENDATION 1.4: [PROGRAMME BOARD AND DFID] Ensure objectives are realistic and feasible within the programme’s time frame.Developprogrammeswithrealistic objectives that are feasiblewithin a three-year time frame.Thesemost likelywould focus on outputs and shorter-term outcomes. Alternatively, consider creatinglonger (i.e.,five-yearor longer)programmetimelines if longer-termoutcomesmustbeincluded.WhilelongerprogrammetimelinesmaynotcorrespondtocurrentDfIDfundingcycles,amulti-phaseprogrammecouldbeaneffectivesolutionthatiscompatiblewithfundingtermsaswellasactivitiesaimedatlonger-termimpact.
RECOMMENDATION 1.5: [PROGRAMME BOARD AND DFID] Ensure M&E processes are integrated within the programme design from the outset and that there is an appropriate balance between M&E and learning activities. Itisimperativethataprogramme-levellogicalframeworkbeinplacepriortoimplementationofanyprogrammeorproject activities.Developmentof this framework shouldoccur inparallelwith theprogrammeandwithinputfromlocalstakeholders.Ideally,indicatorsshouldbestreamlinedacrosstheprogramme,andshouldcontainacoresetofprogramme indicatorsagainstwhichallprojectsreport,withpotentiallyaseriesofadditionalsite-specificorproject-specific indicatorswhere relevant. Incorporate quantitativeVfM indicatorswithin thecoresetofindicatorstoensureVfMandefficiencycanbetrackedacrosstheprogramme.TheM&Esystemshouldenablethecollectionandreportingofdatadisaggregatedbysexandotherprioritisedgroups.Inatleastoneprogrammefocuscountry,considerprioritisinga randomised controlled trial (RcT) or other rigorous evaluation design to evaluateprogrammeeffectiveness.Thiswouldrequireinvolvingtheevaluationteamintheinitialstagesoftheprogrammedesignprocesstoensurealignmentoftheevaluationquestions,randomisationoftheinterventionsites,andadequatebaselinedatacollection.Thiswouldallowfortestingoftheeffectivenessoftargetedcomponentsoftheprogrammeaswellastheprogrammeoverall,isolatingtheroleofdifferentcontextualfactors.Technologicalsolutionssuchasdashboardswith real-timedatavisualisationsofkey indicatorscouldhelpenhanceroutinemonitoring,andallowformorerapidcoursecorrection.SufficienttrainingforprogrammeandprojectfocalpointswithrespecttotheM&Esystemmustbeprovidedandappropriatemechanismstoensurehigherqualitydatamustbeinplace.learningisimportantbutrequiresawell-designedM&Esysteminordertobeofmostvalue.
220 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
RECOMMENDATION 1.6: [DFID] Re-examine the development of the business case, as well as its timing and its content, and broaden definitions of disasters, emergency preparedness and response. for any future programme at the scale of theDEPP, tofacilitateprojectdesign, thebusiness case shouldprovidetimelyand sufficientdetailssuch as focus countries, the criteria for country selection, a set of definitions of keyconcepts(suchascollaboration,institutionalarrangements,emergencypreparednessandlocalisation),andaconcisesummaryoftheDfIDpolicyoninclusionofprioritisedtargetgroupsinprogrammeandprojectdesign.furtherconsiderationsforhowtothinkaboutemergency preparedness in urban contexts and conflict-affected settings are needed.considerationsforissuessuchaspovertyandfoodinsecuritywhichlieattheintersectionof development and humanitarian sectors should be considered when designingemergencypreparednessandresponseprojectsdependingonthecontext.
2. RELEVANCE AND EFFECTIVENESS OF THE INTERVENTIONS CONCLUSION 2:DEPPinterventionswereoverwhelminglyfoundtoberelevantandappropriateandweredemonstrated tofill importantgaps thathavenotbeenpreviouslywelladdressed(surveillance of emergencies in urban settings, conflict-sensitive emergency preparedness,inclusionofaginganddisability,etc).However,start-updelayslimitedprojectimplementationperiodsandreducedpotentialimpact.Optimisingthedesignprocessasdescribedaboveandstreamliningadministrativeandcontractualprocesswouldensuresmoothandtimelyprojectstart-upandminimiseimplementationdelays.Attheprogrammelevel,therewasnoevidenceofimprovedknowledgeoncorehumanitariancompetencies,inclusion,oremergencypreparednessduetotheDEPPbutsignificantchanges inorganisationalcapacityandpreparednessamongDEPP beneficiaries occurred, in particular among local organisations. furthermore, DEPPorganisationscontributedtoimprovedresponseinseveralcontexts(ThePhilippines,KenyaandEthiopia).Agreaterimpactcouldlikelyhavebeenachievedwithalongerimplementationperiod,andwitheasieraccesstoemergencyresponsefunding.Thereisscopeforcertainprogrammecomponentstobereplicatedorscaledupormodifiedandfurthertested.
Conclusion 2a: Collaboration and Consortiafunctioning of consortiawas suboptimal and contributed to implementationdelays and slow decision-making. A smaller, more strategic set of consortiamembers, with more consistency between UK and in-country members foreach projectwould streamline project delivery and should be considered infutureprogrammes. Inclusionofmorel/nnGOswithin consortiamighthelpshiftpowerbalancesandcontributetolocalisation.collaborationswithexternalstakeholdersincludinggovernmentandUn,thoughessential,weredifficultandlessonslearnedshouldinformfutureprogramming.IntwosequentialdiarrhoealoutbreaksinKenya,theconsortiamodelandcollaborationwiththegovernmentenabled coordinated effective response activities that improved with eachsubsequentemergency.
Conclusion 2b: Capacity BuildingOverall individual training targets at the programme level were exceededsuggesting successful implementation of DEPP capacity building activities.However,thesedataincludedindividualswhoparticipatedincapacitybuildingeffortswithminimal exposure such aswebinars and forums.Accessibility totrainingremaineduneven.Thetargetofincluding50%womentraineesacrosstheprogrammewasnotmet,andsometrainingswereonlyavailableinEnglish.Structuralbarriers,suchasorganisationalandmanagementarrangements,were
221 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
akeybarriertoapplyingandintegratingknowledgegainedfromDEPPcapacitybuildingeffortsat theorganisational levelandshouldbeaddressed in futureprogrammes. Multi-pronged capacity building efforts that include practicalapproachestosustainandapplyknowledgewereperceivedtobemosteffective,butquantitativedatashownoevidenceofknowledgechange.Organisational-level capacity building efforts where several strategies were combined andtailoredtothespecificgapsofeachorganisationwereperceivedtobeeffective,and evidence of increased organisational capacity of local organisationswasdemonstratedinanumberofsettings.
Conclusion 2c: Evidence Generation and LearningGenerationoflearningandevidencewashighintermsofquantity,butevidencewasmainlyanecdotalandlackinginscientificrigor.Arguablytoomuchemphasiswasplacedongenerationoflearningproductscomparedtoroutinemonitoring.Differentialreportingbyprojectsagainsttheprogramme-levellogicalframework(once itwas finalised and fully implemented inAugust 2017) occurred, andtherewasevidenceofweakqualityassurance.Inaddition,therewasalackofdisaggregateddatacollectedwithrespect to importantsubgroups.Sharingoflearningdidoccurbut there is noevidence that this translated tobehaviourchange.learningeventswereausefulapproachtosharelearning,howeveranincreasedfocusonthedisseminationofevidenceattheseeventswouldhavebeenvaluable.ThelearningPlatformwasanimportantresource(thatnowservesaspartof theDEPP’s institutionalmemory)but itwasnotusedconsistentlyacrossprojectsandfeaturedvaryingamountsofevidenceandlearningsharedbyeachproject.Suchplatformsshouldbemoreheavilypromotedtoincreaseuse by in-country partners, represent resources from all projects and ensureavailabilityofresourcesinrelevantlanguages.TheDEPPlearningProjecthadbeenoriginallyconceptualised to takeon theM&Efunctionsof theDEPP inadditiontolearning.Whileauniqueidea,thismodeldidnotworkwell–M&Efunctionsandresponsibilitywouldhavebeenbetterplacedattheprogrammemanagementlevel.Placingthesefunctionsasoneprojectalongsidetheotherswithlimitedauthorityledtodiffusionofaccountabilityandresponsibility.Asastandalonelearningproject,ithashadmixedresults,butcouldbere-envisionedforfutureprogrammes.
Conclusion 2d: Early Warning SystemsEarly warning systems are important as they provide (in theory) accurate,predictive and timely data to support emergency preparedness. However,findingswithrespecttoimplementationofearlywarningsystemsaremixed.Sixdiverseearlywarningsystemsweredevelopedandfivewerefullyoperationalatthetimeofthisreport.However,becauseofimplementationdelays,thelengthoftimethesesystemswerefunctionalwasverylimited.Severalofthesystemshavebeenwellintegratedintoappropriatestructurescontributingtoincreasedlikelihood of sustainability and impact. Only two systems (UEWEA, PHEPGambella)havebeenusedinanemergencyresponse.UEWEAinparticularwasshowntobeaneffectivesystemthatwasabletodetectseveralemergenciesthatwouldhaveotherwisegoneunnoticedininformalsettlementsaroundnairobiandthiscontributedtoimprovedtimelyandeffectiveresponse.Thissystemhadbeenoperationalthelongest,andalsoestablishedaconcreteexitstrategy,withthelocalcountygovernmenttakingovermanagementofthesystem.Thisprojectbenefittedfromalonger“effective”timelinecomparedtotheotherprojectsas
222 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
ittookforwardworkfromapreviousproject181thatconceptualisedthesystem,developedandtestedtheindicatorsandestablishedgovernmentcollaborations.Withoutthatexistingworktobuildon,UEWEAwouldlikelyhavefacedsimilarchallengesastherestofthecohortintermsofestablishingafunctionalsystemwiththeprogrammeperiod.
Conclusion 2e: Emergency Preparedness and ResponseThere are some emerging examples in several countries of perceivedimprovementsinemergencypreparednessattheorganisationalandcommunitylevel.DocumentreviewhasdemonstratedthattheDEPPhascontributedtoatleast42responsesin11countriesandthereareseveralself-reportedexamplesofstrengthened,morelocalisedresponse.
RECOMMENDATION 2.1 [DEPP PROGRAMME BOARD] The Learning Project and its placement should be re-examined, with its role, responsibilities, objectives and accountability mechanisms clearly defined at the outset of any future programme. Programme-level M&E should sit within programme-level management, but the casecould bemade to test a differently structured standalone learning project in a futureprogramme.furtherconsiderationstoensureabalancebetweenquantityandqualityoflearningandevidencegenerated,includingtheburdenandtimerequirementsplacedonprojects,andthetimingofM&Eactivitiestobeginbeforeprojectsstartimplementingareneeded.
RECOMMENDATION 2.2 [DEPP PROGRAMME BOARD] Project-level M&E systems should directly feed into programme-level systems (via a common, core set of indicators), and should be in place prior to implementation with timely reporting to permit agile course correction.StrongerM&Esystems includingcollectionofempiricaldatawouldsupport projects in accessing additional funding andyield better quality projectswitha greater likelihood of impact. Accountability mechanisms, including accountability tobeneficiaries,shouldbestrengthened.
RECOMMENDATION 2.3 [DEPP PROGRAMME BOARD] To increase the effectiveness of capacity building interventions, emphasis should be placed on quality not on quantity.Attheindividuallevel,approachesthatbuildskillsandreinforcelearning,andcombineseveralstrategies,shouldbeprioritised.Attheorganisationallevel,approachesthataretailoredtoidentifiedgapsandaimtostrengthensystemsandprocessesshouldbe emphasised. flexible funding mechanisms providing small or in some cases micrograntsforcapacitybuildingwereonesuccessfulapproachtoempowerlocalorganisationstobuild their capacity and shouldbe considered in futureprogrammes. Systems-levelcapacitystrengtheningshouldalsobefurtherpursued.furtherattentiontothebalanceofactivitiesacrossthesedifferentlevelsandthedesiredeffectateachlevelwillbeneededto maximise impacts. Structural barriers need to be addressed, and capacity buildinginterventionsshouldbebettertailoredtospecificcontexts.
RECOMMENDATION 2.4 [DEPP PROGRAMME BOARD] Development of early warning systems should be taken forward within future programmes but only when sufficient implementation periods are available.Athree-yeartimeframewasinsufficienttodesignandoperationaliseanearlywarningsystem;morerealistictimelinesareneededtoensurecompletionofthesystemandallowingtimetotestthesystemwithinemergencyresponses (or simulations). community members and government officials should be
181 IndicatorDevelopmentforSurveillanceofUrbanEmergencies(IDSUE).UrbanEarlyWarningEarlyActionfinalEvaluationReport,p1.
223 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
activelyinvolvedindesigningthesurveillancesystems,beneficiarycriteria,andresponsepackagedetails.consensusonindicatorsandearlywarningsystemsshouldbereachedin collaboration with community and government stakeholders. Early and sustainedengagementshouldbeemphasisedtofostertrustandstrengthenpartnerships.
RECOMMENDATION 2.5 [DEPP PROGRAMME BOARD] Future programmes must consider emergency preparedness and response on a continuum with more attention paid to the response component. The programme should strategise on how best toenableimprovedresponsethroughemergencypreparednessactivitiesandensuresuchmechanisms are embedded within the programme, rather than expecting responsecontributiontooccurorganicallyasanaturalby-productofpreparednessactivities.Thiscouldtakevariousforms.flexibleresponsefundscouldbebuiltintotheprogramme(andpotentially the individual projects) to provide a rapidmechanism for local programmebeneficiariestoimplementresponseactivities.Asthisapproachattheprojectlevelhasworkedwell inseveralcontexts itshouldbeviewedasapromisingstrategyworthyofreplicationinmoresettingsandtestedasaprogramme-levelcomponent.Alternatively,orinaddition,moreformallinkswithexistingemergencyresponsefunds(suchastheSTARTfund)couldbepursued.Inaddition,furtherthinkingaroundemergencypreparednessinconflict-affectedsettingsandinurbancontextsisneeded.
RECOMMENDATION 2.6 [DEPP PROGRAMME BOARD] To leverage the global momentum around localisation of humanitarian response and build on the contribution of DEPP in advancing this agenda, more nuanced considerations on localisation are needed. This includes articulating a clear definitionofwhat localisationmeans,whichentitiesareconsidered localandwhat the implicationsof thesedefinitionsare fromapractical standpoint. Strategically, laying out and potentially testing approacheswhichcouldenableInGOstotakeamoresupportiverole,andbetteroperationaliselocalisation(movefromtheoreticaltoactual).Organisationalpoliciesaroundlocalisationcouldbeonetarget,butunderstandinghowtooperationalisethepoliciesintoactualprocesseswouldbean importantelement.Additionally, thehumanitarian landscape indifferentsettingsand its relationship to localisation need to be considered to ensure that appropriate,contextuallyrelevantstrategiestosupportlocalisationareimplemented.finally,localisationfromthedonorperspectiveshouldbefurtherexploredtounderstandstrategieswhichmightenabledonorstomoreeasilysupportlocalcounterpartswithinthecontextofriskaversionandduediligenceprocesses.
3. EFFECTIVENESS OF MANAGEMENT ARRANGEMENTS CONCLUSION 3: collaborationishighlyvaluedandDEPP’sconsortiaapproachwasuniversallydeemedtobethemostappropriate,preferreddeliverymechanismforhumanitariancapacitybuildingandemergencypreparednessandresponseactivities.Therewassomeevidencethattheconsortiamodelenabledcoordinated,timelierresponseinsomesettings.Globally,effectivenessof consortia couldbe increased in futureprogrammes througha smaller,more strategic setof consortia members, improved communication, opportunities for better coordination ofactivities, and provision of sufficient resources for strategic and organised collaborations.Inclusionofmorel/nnGOswithintheconsortiacouldcontributetothelocalisationagenda.network analysesdemonstratedevidenceof strengthenedhumanitarian responsenetworksin thePhilippines,KenyaandEthiopia,while thenetwork inMyanmarremainedunchanged.Arangeofdifferenttypesofemergencypreparednessandresponsenetworksexist inthesecountries, from highly isolated (Myanmar) to highly distributed, locally led (The Philippines).Inanyfutureprogrammeaimingtostrengthennetworks,thesedifferencesinthesize,scopeand priorities of networks across countries should be appropriately considered during the
224 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
bothdesignandimplementation.Humanitarianlandscapesandexistingnetworksacrossfocalcountriesshouldbeconsidered,particularlyinthedesignoftheprojectsandattheprogrammeleveltoleverageexistingcollaborationsandlocalpartnerships.Targetingofnetworkmembersforcapacitybuildingandevidencesharingshouldbemorestrategicandincluderelevantkeyinfluencers.
RECOMMENDATION 3.1 [DEPP PROGRAMME BOARD] In future programmes, refine and optimise the consortium model building on lessons from the DEPP, ensuring context-specific considerations. Workingeffectivelywithcollaborativestructuresinthefuturewillrequirebetterunderstandingofwhattypesofconsortiaareappropriateandeffectivewithin different contexts.Overarching factorswhich should be incorporatedinprogrammesof thisnature includesupportingcollaborativestructures thatbuildonexistingsuccessfulrelationshipsandthatworkwithasmallernumberofmembers,andthatworktowardsstrategiccollaborations.Toincreasethespeedofdecision-makingandeffectiveness of consortia, streamlined contracting and reporting processes should beestablished.Ensuringl/nnGOsareincludedinconsortia,andassessmentofstrategiestoaddressriskaversionamongdonorstosupportingsuchconsortiaareneeded.
RECOMMENDATION 3.2 [DEPP PROGRAMME BOARD] Network assessments could better inform targeted strategies to enhance collaboration and to effectively strengthen emergency preparedness and response networks. country-specificnetworkmappingandassessmentsshouldbeundertaken,whenappropriate,tofacilitatetheuseofmoretargetedapproachestobuildingnetworksandpartnerships,suchasthroughtheidentificationandtargetingofkeyinfluencers.Deeperexplorationonthelevelofinfluenceandroleofdifferentorganisations(i.e.,asresourcehubs,knowledgebrokersetc.)withinthenetworkinfocalcountrieswouldbeextremelyvaluabletoinformstrategiesaroundwhichorganisationstotargetandhow.Theseassessmentscouldalsoenableoptimisationofcollaborationwithlocalandnationalactors.forexample,localisationeffortscouldberefinedinthePhilippineswherenationalnGOsarealreadyverystronglyinvolvedwithinemergencypreparednessandresponsenetworks.
4. EFFICIENCY AND VALUE FOR MONEY (VFM) CONCLUSION 4: Goodindicatorsofcosteconomywereevident,butinsomecasesprojectbudgetsweretooleanwithinsufficientresourcesforconsortiamanagementandcollaborativeactivities.Attheportfoliolevel,resourcesdedicatedtoportfoliomanagementandcollaboration,M&EandtonPAcswereinsufficientwithrespecttodesiredprogrammeobjectives.Infutureprogrammes,morestrategicallocationoffundsshouldbeundertaken.Inaddition,systemsandresources for governance, strategy, portfoliomanagement, decision-making and consortiumarrangementsshouldberestructuredtoimproveefficiency.Adaptivemanagementprocessespermittedcoursecorrectionand revisionsofprojectplans to reflectchangingcontexts, andVfMreportingimprovedoverthecourseoftheprogramme.futureprogrammesshouldbetteradoptandstandardiseVfM indicators inorder tocollectdata for internalprojectpurposes,cross-projectcomparisonsandgenerallearning.TherewerenostrongVfMfindingstosuggestthat emergency preparedness improved the efficiency of humanitarian responsewithin theprogrammeduration.
RECOMMENDATION 4.1 [DEPP PROGRAMME BOARD] Streamline contractual processes, management decisions and flow of funds. Whilerecognisingtheimportanceofsubcontractingprocessesandorganisations’internalprocedures,futureprogrammingshouldtakestepstominimiseandstreamlinecontractualprocesses.forexample,pre-agreementlettersofcommitmentcouldhelpclarifysomecontractualissuespriortothestartofprojects.Theprogrammemanagement couldalsoprovidea template for sub-
225 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS
contracts and consider settinga standardtimeline for contract turnaround. If projectsare structuredwith a global grant holder and in-country host or partner, streamliningmanagementdecisionsandflowoffundssothatthesameorganisationservesinbothroleswheneverpossiblecouldbeconsidered.
RECOMMENDATION 4.2 [DEPP PROGRAMME BOARD AND PROJECTS] Adopt and standardise VFM indicators at programme and project levels and implement routine reporting.DfIDguidanceonVfMinhumanitarianprogrammesorotherrelevantVfMguidanceshouldbeusedtoenhanceVfMmonitoringandreportinginfutureprogrammes.AdoptingVfMindicatorsattheprogrammelevel,aswellastheprojectlevelwillensurethatVfMcanbecloselytrackedandanalysedatbothlevels.
RECOMMENDATION 4.3 [DEPP PROGRAMME BOARD] In a future programme, stra-tegically allocate funds to achieve programme objectives, with greater funds desig-nated for programme management costs, M&E and NPACs. foralargeportfolio,pro-grammemanagementcostsshouldbeintherangeofaround10%,toenablesufficientresourcestoundertakeitskeyfunctionsofmanagement,M&Eandstrategictasks.nPAcscanbeupto20%andashighas25%,butshouldbeconsideredrealisticallytoensurethatin-countryandbackofficecostsarecovered.WithrespecttoM&E,ensuresufficientresourcesareavailableattheboththeprogrammeandprojectlevelstoefficientlyimple-mentbothroutinemonitoringandmorerigorousevaluations.
5. SUSTAINABILITY OF THE INTERVENTION AND LIKELIHOODOF IMPACT OF THE PROGRAMME CONCLUSION 5: Insomesettings,DEPPhascontributedtostrengtheningnationalemergen-cypreparednesssystemsbut typicallyonasmallgeographicscale,andby individualprojectconsortia.TherehavebeenseveralconcreteexampleswhereDEPPhasimpactedgovernmentpoliciesorsystemsor increasedpoliticalcommitment.Stakeholdersnotedashift inthewayorganisationsconsider,addressandinvolvelocalactorsandcommunities,andattributeditasoneofDEPP’smostsignificantcontributions.However,therewasalackofadetailedstrategyfortheinclusionofgenderandprioritisedtargetgroupsattheprogrammelevelresultinginafragmentedprogrammewideapproachandunevenresults. Inaddition,cross-cuttingthemessuchasaddressingVAWGwerenotadequately integrated intotheprogramme.Ratherthanonlyincludingoneortwoprojectswithaninclusionandprotectionfocuswithinaportfolio,amoreintegratedapproachinvolvingmainstreamingacrosstheprogrammeshouldbeprioritised.Thisshouldentaildisseminationofdetailedexpectationsandguidelinesoninclusionofgenderandother prioritised groups and reporting requirements during thedesignphase to ensurethatprojectsaredevelopedaccordingly.Aswell,consistencyinimplementationofgenderandinclusion considerationswith regularmonitoring shouldbeestablished. In addition to inclu-sion,sustainabilityplanningwasweakandwasnotclearlyplannedordocumentedatboththeprogrammeandprojectlevels.Thisfailureseemstohavebeenlinkedtoexpectationsaroundthepossibilityof receiving additional funds fromDfID for a secondphaseof theDEPP, asarticulatedinthebusinesscase,combinedwithlackofguidancefromtheprogrammeonsus-tainability.Inanyfutureprogramme,moredeliberate,strategicsustainabilityplanningisneededatall levels,beginningduringthedesignphase,andwithtransparencyaroundthepossibilityoffurtherfunding.Attheprogrammelevel,thereissomeevidencethatbenefitshavebecomeembedded–withexamplesofsystemandpolicychangeaswellasstrengthenednationalpre-parednesssystems,albeitonasmallgeographicscale.However,thepotentialforlonger-termeffectsisdependentontheextenttowhichdifferentcomponentsareabletocontinuebeyondthecloseoftheprogramme.futureprogrammesshouldprioritiselongerimplementationperi-
226 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT
odstoincreaselikelihoodofimpact,andincorporategovernmentcollaboration,systems-levelchange,policyoradvocacycomponents,involvementofbeneficiariesandgoodVfMinordertopromotesustainability.
RECOMMENDATION 5.1 [DEPP PROGRAMME BOARD AND PROJECTS] Develop a strategy for inclusion of gender and prioritised target groups at programme level, ensure projects adopt an approach which aligns with this strategy and monitor im-plementation at all levels. In thegovernancecriteria, considerbroadening thegenderstatementtoastatementoninclusionofprioritisedtargetgroups.Developguidelinesforprojectsonrequirementsforthisstatement.Thiscouldincludelinkstokeydocumentsonbestpracticesandthelawandpoliciesgoverninginclusionofprioritisedtargetgroupsindevelopmentprogrammingandprojectdesign.Ensuregenderconsiderationsareconsis-tentlyimplementedandreported.
RECOMMENDATION 5.2: [DEPP PROGRAMME AND PROJECTS] Within any future programme, integrate deliberate, strategic sustainability planning that takes a more holistic view on sustainability going beyond simply securing funding streams. Pro-grammeandprojectsustainabilityplansshouldbedevelopedasearlyaspossible,prefer-ablyinthedesignphaseandupdatedregularly.However,forriskierprojectswithahighpossibilityoffailure,sustainabilityplanningdoesnotmakesenseandisnotagooduseofresourcesuntilresultscanbedemonstrated.futureprogrammesshouldincreasethelikelihoodofsustainabilityandlonger-termimpactsbybuildingonpreviouseffortsandexistingpartnerships/consortia,workingwithgovernments(whenappropriate),incorpo-ratingapolicyoradvocacyelement,developingsustainableoutputssuchastools,guide-linesorsystems,increasingcross-projectlinkages,fosteringsustainablerelationshipsthatcancontinuebeyond theDEPP, improvingbeneficiaryengagement,andstrengtheningfeedbackmechanismsforthecommunity.GoodVfMshouldalsobeprioritisedtomaxi-misesustainability.Programmevisibilityatthecountrylevelshouldbemaximisedthroughincreasedadvocacyinitiativesandthedevelopmentofstreamlinedcommunicationwithexternalstakeholders.
227 SUMMATIVEPHASEREPORT-cOnclUSIOnSAnDREcOMMEnDATIOnS FORMATIVE PHASE REPORT 227
228 EVALUATiON OF THE DEPP - SUMMATiVE PHASE REPORT