routes to resilience...routes to resilience: lessons from monitoring braced 2 reflections in brief...

ROUTES TO RESILIENCELESSONS FROM MONITORING BRACEDPaula Silva Villanueva and Catherine Gould

Reflection paper

CONTACT THE AUTHORS

Paula Silva Villanueva is the director of ResilienceMonitor

and leads the BRACED Knowledge Manager Monitoring and

Results Reporting team. Over the past 10 years she has developed

a number of monitoring and evaluation frameworks and systems

in the field of climate change adaptation, disaster risk reduction

and sustainable development.

@ResilienceMonit

Catherine Gould is a senior programme manager working in the

BRACED Knowledge Manager Monitoring and Results Reporting

team. She has over 10 years’ experience designing programme

monitoring and evaluation frameworks and developing practical

tools, templates, processes and guidance for development,

emergency response and resilience building programmes.

ACKNOWLEDGEMENTS

This reflection paper was written by Paula Silva Villanueva and Catherine Gould,

based on our experiences of monitoring change in the BRACED programme.

We wish to acknowledge critical contributions from Florence Pichon, as well as

Emily Wilkinson, Blane Harvey, Katie Peters, Fran Walker and Dave Wilson from

the BRACED Knowledge Manager. The authors are also grateful to the M&E

and project leads of the BRACED Implementing Partners and to Annie Bonnin

Roncerel and Jim Djontu of the BRACED Fund Manager, for openly sharing their

experiences and reflections. The paper has benefited from critical review from

Robbie Gregorowski and Katie Peters of the Knowledge Manager and Derek

Poate (external). The donor DFID have also provided their feedback and discussed

how to ensure the lessons are taken up and applied, both within BRACED and

other similar programmes. Finally, we thank Charlotte Rye and Clare Shaw of

the Knowledge Manager for their support in the publication process.

Contents

Reflectionsinbrief 2

1. Introduction 7

1.1 Whatisthispaperabout?� 7

1.2 Whoisthispaperfor?� 8

1.3 Howhavewereflected?� 9

1.4 Howisthispaperstructured?� 10

2. MonitoringandevaluationinBRACED 11

2.1 TheBRACEDM&Eframework� 11

2.2 Theconcepts� 12

2.3 ReportingprogressinBRACED� 15

3. Whathavewelearntsofar? 17

3.1 Translatingconceptsintopractice� 18

3.2 RollingoutM&Eframeworks� 21

3.3 Reportingonresilience� 23

3.4 Aggregatingandsynthesisingdataatscale� 25

4. HowcanBRACEDbuildonthislearning? 28

4.1 Questionforfurtherreflection� 31

Annex1:MonitoringandevaluationinBRACED 33

Annex2:BRACEDtheoryofchange 34

Annex3:Project-to-programmesynthesismethodology 35

Annex4:FurtherKnowledgeManagerandImplementingPartnerreflections

onexperiencesofreportingagainsttheBRACEDM&Eframework 36

Annex5:BRACEDM&E‘Infrastructure’ 44

Annex6:RollingouttheBRACEDM&Eframework 45

2ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� 2ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED�

Reflections�in�brief

The�basis�for�reflection

Thispapersharesinsights,reflectionsandlessonslearntfromdesigning,

implementingandreportingagainsttheBuildingResilienceandAdaptation

toClimateExtremesandDisasters(BRACED)programme’sMonitoringand

Evaluation(M&E)framework.TheBRACEDprogrammeaimstobuildthe

resilienceofupto5millionpeoplevulnerabletoclimateextremesanddisasters

andsupportsinternational,regionalandlocalorganisations,workingin15

consortiaacross13countriesinEastAfrica,theSahelandSouth-EastAsia.

Tounderstandwhatworksandwhatdoesnotinbuildingclimateanddisaster

resilience,theBRACEDKnowledgeManagerisdevelopingandtestingavariety

ofresiliencemeasurementandmonitoringapproachesandframeworks.The

BRACEDM&Eframeworkisdesignedtoenabledatacollectionandevidence

generationtotrack,measureandunderstandtheprocessesofchangethatlead

toclimateanddisasterresilience.

Eachyear,theBRACEDprojectImplementingPartnersandtheKnowledge

Manager’sMonitoringandResultsReportingteamaddressthecriticalquestion:

‘How are BRACED projects contributing to building resilience?’Theanswerhas

beencapturedinourcompanionsynthesisreport–‘Routes to resilience: insights

from BRACED year 1’.�Thisreflectionpaperwaswrittenfollowingthecompletion

ofthefirstannualsynthesisreportandprovidestheMonitoringandResults

Reportingteam’sreflectionson‘What lessons have we learnt from the

monitoring and results reporting efforts to date in BRACED?’

DuringthefirstyearofBRACED,wehaveaddressedthefollowingM&Echallenges:

• movingfromconceptstopractice

• rollingoutaprogramme-levelM&Eframeworkto15projectsworking

across13countries

• triallingqualitativereportingapproachesatproject-andprogramme-level

• aggregatingandsynthesisinghighlycontextuallyspecificdata.

Theseexperienceshavegeneratednewinsightsintohowtoapproachthe

monitoringandresultsreportingofaresilience-buildingprogrammeatthescale

ofBRACED.

Developingprogramme-levelM&Eframeworksforresilience-building

programmesisarelativelynewareaofwork,withlimitedexperiencetodraw

on.ReflectionabouttheBRACEDM&Eframeworkisthereforeacriticallearning

stepforBRACEDitselftoimproveM&Epracticeandevidencegenerationwithin

theprogramme.Italsoprovidesanexcitingopportunitytocontributetobuilding

theknowledgebaseonresiliencemonitoringandmeasurementforthewider

community.Wehopethatthereflectionssharedinthispaperwillcontribute

toongoingandfutureresilience-buildingprogrammes.

http://www.braced.org/resources/i/?id=ed1d6676-a044-4f26-a99d-562785272bdb


3ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED�

Lessons�learnt�so�far

Inyear1,BRACEDprojectImplementingPartnershaveembracedanewway

ofmonitoringandreportingchange.Wehavelearntagreatdealasaresult

oftakingaprogramme-levelviewofhowresilienceisbeingbuiltinBRACED.

ThekeylessonsemergingfromourBRACEDexperiencetodateinclude:

1. Translating�concepts�into�practice:�Measuring�progress�on�resilience�

cannot�be�done�with�one�‘simple’�indicator.�It�requires�qualitative�and�

explanatory�frameworks�that�contextualise�results�against�shocks�and�

stresses,�as�well�as�the�wider�context�projects�operate�within.�Thereisa

riskoflosingandobscuringcriticallearningaboutresiliencebuildingifwe

measureresilienceusingjustoneindicator.�Understandingthedeterminants

ofclimateanddisasterresilienceiscomplexandtherearen’tanyready‘yes’

or‘no’answers.

Moredetailedlessonsontranslatingconceptsintopracticecanbefound

insection3.1ofthispaper.

2. Rolling�out�M&E�frameworks:�There�are�different�options�for�rolling�

out�programme-level�M&E�frameworks�and�systems,�but�each�comes�

with�its�own�trade-off.�Optionsandtrade-offsincludedecisionsabout

thetypeandlevelofsupporttoprovidetoprojectpartners.Therolling

outofprogramme-levelM&Eframeworksandsystemsmustfindabalance

betweenlight-touchandresource-intensiveoptions.Theyalsoneedto

allowforcontinualadjustmentsbasedontheemergingbodyofknowledge

andexperienceregardingthemonitoringandmeasuringofresilience.In

BRACED,theKnowledgeManagerwassetupaftertheprojectlogframes,

theoriesofchangeandM&Eplansweredefined.EstablishingtheBRACED

programme-levelM&Eframeworkwouldhavebeeneasierifithadbeen

developedatthesametimeasthe15BRACEDprojects’M&E.

MoredetailedlessonsonrollingoutM&Eframeworkscanbefound

insection3.2.

3. Reporting�on�resilience:�Qualitative�and�explanatory�frameworks�offer�

an�opportunity�to�complement�resilience�indicators.�However,�if�we�are�

totruly�engage�with�these�frameworks,�we�need�to�shift�mindsets�from�

accountability�to�learning-oriented�M&E.�Engagingwithqualitativeand

explanatoryframeworksrequiresM&Epracticestogobeyond‘business

asusual’andaccountability-drivenexercises.M&Eexpertsandproject

managersalsoneedtoengageinmorerefinedandcomplexdatacollection

andanalysisprocessesthaninatraditionalprogramme.

Moredetailedlessonsonreportingonresiliencecanbefoundinsection3.3.


4. Aggregating�and�synthesising�data�at�scale:�Synthesising�and�

aggregating�data�while�retaining�context�specificity�requires�time,�

resources�and�thorough�synthesis�methodologies.Qualitativeand

explanatoryframeworkscallforexhaustivesynthesisprocessesthatare

abletodealwithcomplexdataanalysis,varyinglevelsofdataquality

andself-reportingbias.Thislengthenstheleadtimebetweenproject-level

annualreportingandprogramme-levellearning,whichmaylimitthefindings’

potentialimpactonprogrammeandprojectdecision-making.

Moredetailedlessonsonaggregatingandsynthesisingdataatscalecan

befoundinsection3.4.

How�can�BRACED�build�on�this�learning?

BRACEDisnearlytwoyearsintoitsthree-yearimplementationtimeframe

andproject-andprogramme-levelM&Eisalreadysetupandestablished.

Therearethereforesomelimitationstowhatcanbeadaptedandachieved

intheremainderoftheprogramme.Inthiscontext,oursuggestionsbelow

areforboththeBRACEDprogrammeandothersimilarinitiatives.

“Genuinelyunderstandingresilienceinpracticemeansmovingawayfromalogframe-drivenand‘accountability’-focusedM&Eculture”

Monitoringresilience-buildingeffortsandreportingontheirprogressis

challenging.M&EforresilienceprogrammingisstillnascentandBRACEDis

learning-by-doing.Akeymessageemergingfromthispaper,togetherwithits

companionprogramme-levelsynthesisreport,isthatgenuinelyunderstanding

resilienceinpracticemeansmovingawayfromalogframe-drivenand

‘accountability’-focusedM&Eculture.Movingforward:

• Project�Implementing�Partners�should�enhance�their�ongoing�monitoring�

and�results�reporting�efforts�by�taking�a�more�reflective�and�critical�

approach.Thiscouldchallengeprojectassumptionsandwillbuildabetter

understandingofhowtobuildclimateanddisasterresilienceinfragileand

vulnerablecontexts.

• The�programme-level�Monitoring�and�Results�Reporting�team�should�

consider�how�to�encourage�this�critical�Implementing�Partner�reflection�

and�dialogue.Therearelimitstowhatreportingtemplatesalonecanachieve

inthisregard.WethereforeplantoprovidefurthertrainingtoImplementing

Partners,alongwithlight-touchhelpdesksupport.


• Programmes�like�BRACED�need�to�find�and�resource�efficient�ways�of�

achieving�a�sufficient�level�of�reflection�and�learning�for�the�benefit�of�

both�project-�and�programme-level�evidence�generation.Ideally,the

programme-levelM&Eframeworkshouldbedesignedinconjunctionwith

theproject-levelframeworks.

“Outcome-levelindicatorsneedtobecomplementedbysystematicmonitoringandevaluationofresilienceinthecontext

ofactualshocks”

Tobetterunderstandthestabilityofoutcome-levelchangesovertimeandhow

communitieslearnand‘bouncebackbetter’fromdisasterevents,outcome-level

indicatorsneedtobecomplementedbysystematicmonitoringandevaluation

ofresilienceinthecontextofactualshocks.Movingforward:

• Implementing�Partners�are�in�a�unique�position�to�contribute�to�

knowledge�about�how�to�quantify�the�number�of�people�whose�resilience�

has�been�built�(KPI�4)�at�the�project level.TheMonitoringandResults

Reportingteam,togetherwithwidermembersoftheKnowledgeManager

andtheBRACEDFundManager,shouldfurtherexploreoutcome-level

resilienceindicatorsindifferentcontexts:theadvantagesanddisadvantages,

aswellasopportunitiesandtrade-offs.

• When�designing�and�funding�similar�programmes�in�the�future,�the�

Department�for�International�Development�(DFID)�should�adopt�a�

pragmatic�and�realistic�view�on�the�feasible�level�of�outcome-level�

data�and�evidence�generation�in�a�three-year�programme�like�BRACED.

Resilience-buildingeffortsarenotonlycomplex,butalsoinvolveprocesses

ofchangethattaketimetomaterialise.Prioritisingannualdatacollection

effortsagainstquantitativeindicatorsmaycomeatthecostoflosing

criticalevidenceaboutwhatworksandwhatdoesnotinbuilding

resiliencetoclimateextremesanddisasters.

• Programmes�like�BRACED�should�consider�having�a�diverse�set�of�

methodologies�and�analysis�in�place�for�interrogating�quantitative�

outcome-level�resilience�indicators.Theyshouldbepragmaticabout

whatsortofoutcome-leveldataandinformationcanbeexpectedin

athree-yearperiod.


Whilemuchattentionhasbeengiventoproject-levelapproachesto

monitoringandmeasuringresilience,programme-leveleffortsfaceaunique

setofchallenges.Todate,thereisbothlimitedliteratureandexamplesfrom

otherprogrammesaddressingthesechallenges.InBRACED,wehavebeen

learning-by-doingonanongoingbasis.Movingforward:

• The�Monitoring�and�Results�Reporting�team,�together�with�Implementing�

Partners,�should�consider�ways�to�further�capture�their�monitoring�and�

results�reporting�experiences�within�BRACED.�Thiswouldbenefitboth

BRACEDandotherexistingandfutureresilience-buildingprogrammes.

• Programmes�like�BRACED�should�also�share�experiences�and�contribute�

to�building�knowledge�in�this�relatively�new�area�of�work.

Question�for�further�reflection

Thispaperaimstocontributetoongoinglearningaboutresilienceprogramming

andtoinitiatediscussion.WewishtoengageBRACEDImplementingPartners,

thebroaderBRACEDKnowledgeManager,theBRACEDFundManager,DFID

andwideraudiencesinthefollowingcriticalquestion,whichemergesasaresult

ofourcollectivereflectionsandlearningsofar.Together,weshouldcontinue

toansweritthroughouttheBRACEDprogramme.

How�complex�does�M&E�for�resilience�need�to�be?TheBRACEDM&E

frameworkbringstogetherthreedifferentlensesintotheanalysisofproject-

andprogramme-leveldata.Experiencetodate,bothfromImplementingPartners

andtheBRACEDKnowledgeManager’sMonitoringandResultsReportingteam,

indicatesthefollowing:

• Whiletheframeworkprovidesamultidimensionalviewintothevarious

levelsofcomplexity,itrequiressignificanttime.

• Italsonecessitatesdifferent,non-traditionalwaysofengagingwithdata.

• Itdoesnot–andcannot–providesimple‘yes’or‘no’answersabout

whetherresiliencehasbeenbuiltand,ifso,how.

Asoutlinedinourcompanionreport,‘Routes to resilience: insights from

BRACED year 1’,theessenceof‘resilience’isthatchangeandprogressare

notlinear–soresultsreportingshouldnotbelineareither.TheBRACEDM&E

frameworkiscomplexenoughtobeabletounderstandBRACEDresilience-

buildingefforts,butcoulditbemadetobemoreuser-friendly,whilestill

retainingthecomplexityandnuancesofresilience-buildingprojects?



7ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� IntroductIon

1.1�What�is�this�paper�about?TheDFID-fundedBRACEDprogrammecomprises15field-basedresilience-

buildingprojectsacross13countriesintheSahel,EastAfricaandSouth-East

Asia.1Theseprojectsareimplementedby15ImplementingPartners,whose

performanceanddeliveryisoverseenbytheBRACEDFundManager.The

programmealsohasadedicatedKnowledgeManager2taskedwithgenerating

andconsolidatinglearningabouttheactionsthatworkbesttostrengthen

communityresiliencetoclimateextremesanddisastersinavarietyofcontexts.

M&EactivitiesinBRACEDareundertakenatboththeprojectlevelandthe

programmelevel.(SeeAnnex1forfurtherdetails.)

AkeyareaoftheKnowledgeManager’sworkisgeneratingknowledgeabout

monitoringandevaluationpracticeinacomplexresilience-buildingprogramme.

Tothisend,itisdevelopingandtestingavarietyofresiliencemeasurement

approachesandframeworksthroughasetofmonitoringandresultsreporting,

1 www.BrAcEd.org

2 BrAcEdKnowledgeManager(2016)‘LearningaboutresiliencethroughtheBrAcEdprogramme:AnintroductiontotheroleoftheBrAcEdKnowledgeManager’.BrAcEdKnowledgeManagerinformationleaflet.London:odI.

Image:neilPalmer,cIAt

1.INTRODUCTION

http://www.BRACED.org

http://www.braced.org/resources/i/?id=c30ed2e2-0f5e-4f41-9959-72cd6077e230&com.dotmarketing.htmlpage.language=1




evaluationandresearchactivities.Asacorepartofthiswork,theKnowledge

ManagerMonitoringandResultsReportingteamhasdevelopedtheBRACED

programmetheoryofchange(seeAnnex2)andrelatedM&Eframework.The

BRACEDM&Eframeworkwasdevelopedtoestablishprogramme-levelevidence

andlearningabouthowBRACEDprojectsarebuildingresilience.

ThecomponentsoftheM&Eframeworkwerenewanduntestedwaysof

monitoring,measuringandunderstandingresilience-buildingefforts.Duringthe

firstyearofBRACED,theframeworkhasbeenadoptedbyprojectImplementing

PartnersandappliedtotheirM&E.Attheendofyear1,ImplementingPartners

providedsystematicqualitativeandexplanatoryreportingagainsttheBRACED

M&Eframeworkforthefirsttime.TheMonitoringandResultsReportingteam

havesinceundertakenaprogramme-levelsynthesisofallproject-levelyear1

annualreports.

ThispaperpresentstheMonitoringandResultsReportingteam’sreflections

sofar,fromapplyingtheBRACEDM&Eframework,overseeingyear1project-

levelreportingandcompletingtheprogramme-levelsynthesisofthisdata.The

paperaddressesthequestion:‘Whatlessonshavewelearntfromthemonitoring

andresultsreportingeffortstodateinBRACED?’Thefindingsoftheprogramme-

levelsynthesisitselfhavebeencapturedinaseparatereport,‘Routes to resilience:

insights from BRACED year 1’.3Thecompanionsynthesisreportanswersthe

questionof‘HowareBRACEDprojectscontributingtobuildingresilience?’

andexploresBRACEDprogrammeprogresstodateagainsttheBRACED

M&Eframework.

1.2�Who�is�this�paper�for?Designingandimplementingprogramme-levelM&Eframeworksforresilience-

buildingprogrammesisarelativelynewarea,wherelimitedexperienceexists.

ReflectionabouttheBRACEDM&Eframeworkisthereforeacriticallearning

stepfortheBRACEDprogramme.Itprovidesanexcitingopportunity,notonly

toimproveM&EpracticewithinBRACED,butalsotocontributetobuilding

theknowledgebaseonresiliencemonitoringandmeasurementforthewider

community.Thispaperisaimedat:

• Stakeholders�internal�to�BRACED:�project�Implementing�Partners,�

the�Knowledge�Manager,�Fund�Manager�and�the�donor�DFID.Forthis

audience,weprovideasetoflessonsonhowchangecanbemonitored,

measuredandunderstoodintheBRACEDprogramme.Thesereflections

shouldbeusedtoenhancetheBRACEDM&Esystemforyears2and3of

theprogramme.Theywillinformongoingmonitoringandresultsreporting

atboththeprojectandprogrammelevels,aswellaswiderBRACED

KnowledgeManagerworkontrackingandmeasuringresilienceoutcomes.

3 SilvaVillanueva,P.,Gould,c.andPichon,F.(2016)‘Routes to resilience: insights from BRACED year 1’.BrAcEdKnowledgeManager.Synthesisreport.Brighton:Itad.






• Outside�BRACED:�M&E�practitioners,�donors�and�programme�

staff�with�an�interest�in�designing,�implementing,�monitoring�and�

measuring�resilience-building�efforts.�Forthisaudience,weoutline

thepracticalexperiencesoftheBRACEDprogrammesofarintermsof

conceptualising,rollingoutandreportingonhowresilienceisbeingbuilt.

Wealsoprovidelessonsandreflectionsthatarerelevantandapplicable

toothersimilarprogrammes.

Thispapershouldbereadalongsideourcompanionsynthesisreport,‘Routes

to resilience: insights from BRACED year 1’.

1.3�How�have�we�reflected?InordertogatherlessonsaboutBRACEDexperiencestodateinrelationto

theM&Eframeworkanditsassociatedreportingtemplates,theMonitoring

andResultsReportingteamundertookthefollowing:

• AsystematicreviewandsynthesisofeachprojectImplementingPartner’s

annualreportagainsttheM&Eframeworkfortheprogramme-levelsynthesis

report(seeAnnex3).Here,we:

• identifiedthedatagapsandinconsistenciesintheuseofdefinitions

andindicators

• lookedatthequalityofthedataprovided,identifyingcommon

practicesandchallengesacrossproject-levelreports.

• AconsultationwiththeImplementingPartnersthemselves,gathering

informalfeedbackontheirexperienceofmonitoringandreportingagainst

theBRACEDM&Eframework(seeAnnex4).WealsoaskedtheFund

Managerfortheirexperiencesofgatheringandaggregatingquantitativedata

alongsidetheKnowledgeManager’squalitativework.

• Areflectiononwhatworkedwellandnotsowellinthedesign,rolling

outandimplementingtheM&Eframework.Wealsoheldaone-dayinternal

reflectionworkshopontheM&EframeworkwithbroaderKnowledgeManager

teammembers.Thiswasbasedontheinitialfindingsinrelationtothe

questionaddressedintheprogramme-levelsynthesis:‘How are BRACED

projects contributing to building resilience to climate extremes and disasters?’.

Thispaperisnotanin-depthtechnicalassessmentoftheBRACEDM&E

framework.Rather,itisareflectionpiecetoshareemergingreflectionsand

lessonstodate.TheKnowledgeManager’smonitoringandresultsreporting

effortssitwithinalargerBRACEDM&Einfrastructure(seeAnnex5).Detailed

analysisoftheframework,alongwithmonitoringandmeasuringresilience

inBRACED,ispartofawidercollectiveeffort.Thelessonsinthispaperare

limitedtotheMonitoringandResultsReportingteam’sareaofwork.




1.4�How�is�this�paper�structured?Insection2,wedescribetheBRACEDM&Eframeworkandprovidean

overviewoftheBRACEDmonitoringandresultsreportingapproachand

system.Insection3,wepresentourreflectionsandlessonsidentifiedfrom

theBRACEDmonitoringandresultsreportingeffortsundertakenduring

year1.Finally,insection4,weprovideconclusionsandrecommendations

forBRACEDstakeholdersandsuggestareasforfurtherconsideration.

11ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� M&EInBrAcEd

2.1�The�BRACED�M&E�frameworkMonitoringandevaluationactivitiesareundertakenatboththeprojectand

programmelevel.FurtherinformationonhowM&Eissetupandmanagedacross

theBRACEDprogrammeisinAnnex1.

TheBRACEDlogframe4andtheoryofchangearethetwocornerstone

documentsoftheBRACEDM&Eframework.Twomandatoryprogramme-wide

keyperformanceindicators(KPIs)oftheInternationalClimateFundwerein

placefromthebeginningoftheprogrammeaspartofproject-levelM&E:

• ‘The number of people supported by BRACED to cope with the effects of

climate change’ (KPI1–anoutputlevelindicatorofBRACEDlogframe).

• ‘The number of people whose resilience has improved as a result of BRACED

support’(KPI4–anoutcome-levelindicatoroftheBRACEDlogframe).

Theprogrammetheoryofchange(seeAnnex2)andBRACEDM&Eframework

weresubsequentlydevelopedbytheKnowledgeManagertofurtherunpackthe

assumptionsandprocessesbehindtheprogrammelogframe.Theyareintended

4 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/328271/revised-BrAcEd-Interim-Logframe-23June14.xlsx

Image:neilPalmer,cIAt

2.MONITORING�AND�EVALUATION�IN�BRACED

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/328271/Revised-BRACED-Interim-Logframe-23June14.xlsx



toenablecomplementaryqualitativedatagenerationinordertoexplainand

contextualisethenumbersandtounderstandhowresilienceisbeingimproved.

ThetheoryofchangeandM&EframeworkdothisbysituatingBRACEDefforts

inthebiggerpictureofchangeandencouragingcriticalthinkingandreflection

aboutchangepathways.Theyalsosurfaceandarticulateassumptionsregarding

howchangehappens.

“Thechallengewastodevelopacoherentprogramme-levelframeworkacross

interventions,flexibleenoughtoberelevantacrossdifferentsocio-political,geographical

andclimaticcontexts,whileretainingrobustnessandcoherence”

TheoriginalpurposeoftheBRACEDM&Eframeworkwastoprovidea

programme-levelvisionofchangeandensureacommonlanguageandminimum

alignmentofmonitoring,resultsreportingandevaluationeffortsacrossBRACED,

whileacknowledgingandaccommodatingproject-specificM&Eapproachesand

plansindifferentcontexts.EachprojecthaditsowntheoryofchangeandM&E

frameworkandapproach.Thechallengewastodevelopacoherentprogramme-

levelframeworkacrossinterventions.Theframeworkneededtobeflexible

enoughtoberelevantacrossanumberofdifferentsocio-political,geographical

andclimaticcontexts,whileretainingrobustnessandcoherence.

TheframeworkwasrolledoutacrossBRACEDprojectsduringthefirst

monthsoftheprogramme,throughtheprovisionofguidance,5feedbackand

one-to-onesupport.(Formoreinformationonhowwerolledouttheframework,

seeAnnex6.)Inadditiontothetwoprogramme-widequantitativemeasuresof

resilience,BRACEDImplementingPartnersmonitorandreportprojectresults

againstthethreemaincomponentsoftheBRACEDtheoryofchangeonan

annualbasis:‘AreasofChange’,the‘3As’and‘EvaluativeMonitoring’.

2.2�The�concepts

Tracking�resilience�pathways�through�‘Areas�of�Change’

ThroughreviewingtheassumptionsandcausalchainsunderpinningBRACED

projecttheoriesofchange,weidentifiedfourareaswherechangehadto

happenforBRACEDtoachieveitsoutcomes.Thesewereandcontinuetobe:

(a)Knowledgeandattitudes;(b)Capacitiesandskills;(c)Qualityofpartnerships;

and(d)Decision-makingprocesses.The‘AreasofChange’representwhatisoften

5 WrittenguidancewasprovidedbytheKnowledgeManagertoallprojectImplementingPartnersintheBrAcEdM&EGuidancenotes.

http://www.braced.org/resources/i/?id=761757df-7b3f-4cc0-9598-a684c40df788


referredtoasthe‘missingmiddle’inprojectlogframes.Theyenableustobetter

understandthesetofprocesseslinkingprojectoutputstoresilienceoutcomes

and,ultimately,impactsonhumanwell-being.Theyalsoprovidetheframework

forassessmentoftheBRACEDprogramme’strajectorytowardsimpact.

ThefourAreasofChangearedefinedas:

• Changes�in�knowledge�and�attitudesinrelationtoresilience-building,

inordertofurtherstrengthenpoliciesandpractices.

• Changes�in�the�capacities�and�skillsofnationalandlocalgovernment,

civilsocietyandtheprivatesectortomanagetherisksofclimateextremes

anddisasters.

• Changes�in�the�quality�of�partnershipstodeliverinterventions.

• Changes�in�decision-making�processesthroughinclusiveparticipation,

asonekeyaspectofaresilientsystem.

How do we track and report change?TheAreasofChangepayparticular

attentiontothestakeholdersandactorsinvolvedinBRACEDbyaskingtwo

simplequestionsacrossthem:Who is changing? And how? Theyuseanadapted

outcomemappingapproachtomeasurechange.Thisutilisesgraduated

progressmarkersfrom‘expecttosee’to‘liketosee’to‘lovetosee’changes.

MonitoringandresultsreportingagainstthefourAreasofChangebuilds

knowledgeandunderstandingaboutthe‘pathwaystoresilience’inBRACED.

MoredetailsontheBRACEDAreasofChangeareavailableintheBRACEDM&E

GuidanceNotes6(Note3).

Understanding�resilience�outcomes�through�the�‘3As’�and�transformation

InBRACED,resilienceisunderstoodasthecapacityofasystemtochangeand

adaptinthecontextofmultipleandinteractingshocksandstresses.Ananalysis

ofnearly50existingresilienceframeworksidentifiedthreeinterlinked‘capacities’

astheoutcomesoftheAreasofChangeprocesses.Theseare:thecapacityto

Anticipate,AbsorbandAdapttoshocksandstresses(the3As).The3Asaimto

measureandunderstandchangesinresilienceoutcomesatdifferentlevelsand

withregardtodifferentkindsofshocksandstresses.Insteadofspecifyingaset

ofindicatorstomeasure‘resilience’,the3AsframeworkenablesImplementing

Partnerstodevelopcontext-specificindicatorswiththeirrespectivestakeholders.

ItisanorganisingtooltoanalysetheoutcomesthatBRACEDprojectsmaybe

achieving.Monitoringandresultsreportingagainstthe3Asbuildsunderstanding

6 SilvaVillanueva,P.,Gould,c.,Gregorowski,r.,Bahadur,A.(2015)‘BrAcEdprogrammemonitoringandevaluationguidancenotes’.BrAcEdKnowledgeManager.BrightonItad.






of‘resilienceoutcomes’inBRACED.MoredetailsontheBRACED3As

areavailableintheBRACEDM&EGuidanceNotes(Note4)andthepaper

‘The3As:TrackingresilienceacrossBRACED’.7

Transformation,ontheotherhand,isnotregardedasatypeofcapacitythat

contributestoresilienceinthesamewayasthe3As.Rather,itrepresentsan

outcomerelatedtotheholisticandfundamentalwaysinwhichpeople’scapacity

toanticipate,absorbandadapttoshockscanbebuilt,reshapedandenhanced.

TheBRACEDtheoryofchangehypothesisesthatBRACEDislikelytobemore

transformationalinitsfocuscountriesifitachieveschangesthatarecatalytic,

atscaleandsustainable.

How do we track and report change?BRACEDprojectsreportagainsttwo

InternationalClimateFundkeyperformanceindicators(KPIs)attheoutcomelevel:

• KPI4is‘the number of people whose resilience has been improved’.In

BRACED,thisnumberisderivedfromcollatingproject-levelreportingat

theoutcomelevel,whereImplementingPartnershaveidentifiedtheproject-

specificoutcomeindicatorsthatwilldemonstratechangesinresilience.

Theseindicatorshavebeentaggedinrelationtotheresiliencecapacities–

anticipatory,absorptiveandadaptive.Insomecases,Implementing

Partnershaveidentifiedadditionalindicatorsfortransformativechange.

ImplementingPartnersdecidehowtoweighttheindicatorsaccordingto

theirprojecttheoriesofchangeandhowtheyexpecttoseeprogressin

buildingresilience.

• Thesecondoutcome-levelindicatorisKPI15:‘the extent to which interventions

are likely to have a transformational impact’.InBRACED,transformation

isaself-assessedqualitativeindicator.Trackingtransformationisdifficult

–suchchangescanbedeliberatelyengineered,butareoftenbeyond

thescopeofasingleintervention.TheKnowledgeManagerdevelopeda

scorecardtosupportImplementingPartnersinmonitoringthelikelihoodof

transformationalimpactattheprojectlevelinacomparableway.Through

thisscorecard,theprogrammeM&Esystemdoesnotdefinetransformational

outcomesexante,butinsteadtracksthelikelihoodoftransformationagainst

threepillarsidentifiedintheliterature:policy,empowermentandinnovation.

Itincludescertaincharacteristicsoftransformation,including‘catalytic’,‘at

scale’and‘sustainableoutcomes’.ImplementingPartnersareencouragedto

reportchangestheyinterprettoberepresentingthesepillarsorcharacteristics

oftransformation.

7 Bahadur,A.V.,Peters,K.,Wilkinson,E.,Pichon,F.,Gray,K.andtanner,t.(2015)‘the3As:trackingresilienceacrossBrAcEd’.BrAcEdKnowledgeManagerWorkingPaper.London:odI.


http://www.braced.org/news/i/?id=cd95acf8-68dd-4f48-9b41-24543f69f9f1

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/253682/ICF-KPI-summary.pdf

https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/328254/BRACED-KPI4-methodology-June2014.pdf

http://www.braced.org/resources/i/?id=cd95acf8-68dd-4f48-9b41-24543f69f9f1


Contextualising�results�through�‘Evaluative�Monitoring’

BRACEDoperatesinsomeofthemostfragileandchallengingcountriesin

theworld.8Whiletheprogrammeisnotexplicitlyconflictorsecurityfocused,

manyoftheprojectsarebeingimplementedinacontextaffectedbyconflictor

instability.However,thereisanassumptionwithintheprogrammetheory

ofchangethatBRACEDpathwaystoresilienceandresilienceoutcomesare

achievedwithinanenablingenvironmentatthelocal,sub-nationaland

nationallevels.ThroughEvaluativeMonitoring,ImplementingPartnersexplore

thenatureoftheprevailingcontext–specifically,thegovernancestructure,

decision-makingprocesses,incentivesandrelationshipsbetweendifferentgroups

andindividuals–andtheextenttowhichthiscontextsupportsorconstrains

change.Thisbuildsknowledgeandunderstandingabout‘resilienceincontext’

inBRACED.MoredetailsonBRACEDEvaluativeMonitoringareavailablein

theBRACEDM&EGuidanceNotes(Note5).

How do we track and report change?BRACEDreportingincludesEvaluative

Monitoringasacriticalpartofthereflectionprocess.EvaluativeMonitoring

bringsanevaluationlenstothereportingexercises.Itdoesthisbysituatingthe

datacollectedwithinanunderstandingoftheprevailingcontext.Theaimisto

shedsomelightonprojects’risksandassumptionsandbeexplicitaboutthe

factthatchangeoccursasaresultofmanyactorsandfactors.Monitoringand

reportingquestionsinclude:What are the key contextual factors (at local, sub-

national and national) that may enable or constrain change in the project? How

are these contextual factors enabling or constraining change from the project?

Have they contributed to any unexpected outputs or outcomes?

2.3�Reporting�progress�in�BRACEDCombined,theAreasofChange,3AsandEvaluativeMonitoringenablethe

BRACEDprogrammetotrack,measureandunderstandtheprocessesofchange

thatleadtoclimateanddisasterresiliencetospecificshocksandstresses,in

specificcontexts.Underpinningthisapproachistheneedtocriticallyreflecton

projectandprogrammetheoriesofchangeand,inturn,questionthem.Thisis

intendedtofosterinternallearningandbuildarobustevidencebaseregarding

howandwhyinterventionsaresuccessfullycontributingtoimprovingclimate

resilience(ornot).

TheKnowledgeManagerMonitoringandResultsReportingteamdeveloped

asetofreportingtemplatesduringyear1toenableBRACEDImplementing

Partnerstoreflectonandreportannuallyagainst:

• theresiliencecapacitiesbeingbuilt

• thechangeprocessesunderway

8 Wilkinson,E.andPeters,K.(Eds.)(2015)‘climateextremesandresiliencepovertyreduction:developmentdesignedwithuncertaintyinmind’.BrAcEdKnowledgeManagerresearchPaper.London:odI.


http://www.braced.org/resources/i/?id=8a8b3b72-da43-4e44-bb55-dc18d9ce5a13

http://www.braced.org/resources/i/?id=8a8b3b72-da43-4e44-bb55-dc18d9ce5a13


• ifandhowthecontextisaffectingthesechanges

• whattheimplicationsare,ifany,fortheprojecttheoryofchangedesign.

Attheirownrequest,ImplementingPartnerssubmitasingleannualprogress

reporttoboththeFundManager(againsttheprojectlogframe,foronward

reportingtoDFID)andtheKnowledgeManager(againsttheprojecttheory

ofchange,forevidencegeneration).

Figure1summarisesthestructureforreportingagainsttheM&E

framework.ImplementingPartnersalsoreporttheiruseofclimateand

weatherinformationandthelikelihoodoftransformativechange,aspart

ofthelogframereporting.ForthespecificquestionsaskedofImplementing

Partners,seethereportingtemplates.9

Figure�1:�BRACED�Knowledge�Manager�M&E�Framework

9 theM&EframeworkreportingtemplatesformthesecondoftwopartsoftheBrAcEdprojectannualreports.Inthefirstpart,ImplementingPartnersreportprogressagainsttheirlogframeindicatorstotheBrAcEdFundManager.Inthesecondpart,theyexplore,explainandcontextualisetheseresults.

How we monitor and assess change

Key questions

What is the long term change and development impact the project is seeking to support?

Theo

ry o

f cha

nge

What are the underlying assumptions about how this change would happen and did they hold?

How will change happen? Who are the key actors?

What are the contextual drivers that may enable or constrain change?

3As (Anticipatory, Absorptive & Adaptive)

Areas of Change

Evaluative Monitoring

Reflection and learning – testing theory of change

https://drive.google.com/open?id=0B2Eq4uZTAdPVeGZqazlEYVM1WWM

17ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� WhAthAVEWELEArntSoFAr?

ThissectionpresentstheMonitoringandResultsReportingteam’sreflectionsand

lessonsidentifiedtodate,fromdevelopingandimplementingtheBRACEDM&E

frameworkandconductingthefirstprogramme-levelsynthesisofBRACEDproject

resultsreporting.Inparticular,wereflectonthefourmainchallengesfacedwhen

developingaprogramme-levelM&EframeworkforaprogrammelikeBRACED,

whichcomprises15uniqueprojectsworkingacross13differentcountries:

1. Moving�from�concepts�to�practice:translatingnovelconceptsintopractical

monitoringframeworksthatareapplicableandrelevantacrossprojects.

2. Rolling�out�programme�M&E�frameworks�and�systems�to�the�project�

level:designingacoherentprogramme-levelframeworkthatisflexible

enoughtoberelevantacrossanumberofdifferentsocio-political,

geographicalandclimaticcontexts,whileatthesametimeretaining

itsrobustnessandcoherence.

3. Trialling�qualitative�reporting�approaches�at�project-�and�programme-

level:developingasetofreportingtemplatestogatherdataagainsteach

oftheM&Eframeworkcomponentstoenablestandardisedandcomparable

reportingbyallImplementingPartners.

Image:uSAId/nepal

3.WHAT�HAVE�WE�LEARNT�SO�FAR?


4. Data�aggregation�and�synthesis�at�the�programme�level:aggregating,

synthesisingandanalysingself-reportedqualitativeandexplanatorydata

from15projectsinawaythatgeneratesevidenceandlessonsabouthow

resilienceisbeingimprovedattheprogrammelevel.

3.1�Translating�concepts�into�practice1.�Encapsulating�resilience�concepts�and�thinking�into�quantitative�measures�

limits�what�outcome-level�indicators�may�tell�us.Tobetterunderstandthe

outcome-levelfiguresreportedagainstKPI4,BRACEDappliesthe3Asframework

toanalysethenatureofresilienceoutcomes.Thishasenabledtheprogramme

tounderstandhowBRACEDprojectshavebeenaddressinganticipatoryand

absorptivecapacitiestodate,butitremainsunclearhowtheywillcontribute

tolongertermadaptivecapacities.OthercomponentsoftheBRACEDM&E

frameworkhavealsorevealedhowtheinterplayofsocio-culturalandgovernance

dynamicsshapesresiliencepathways.Suchcriticaldetailswouldhavebeen

missedthroughtheuseoftheKPI4indicatoralone.

Tracking�progress�against�KPI�4�–�Emerging�insights�from�BRACED�

Each�BRACED�project�adopts�a�different�approach�to�measuring�resilience.�

This�is�context�specificandbasedontheirconceptualisationofresilience,

althoughalluseacompositeindexwithconstituentindicators‘fed’through

theuseoflargesamplehouseholdsurveys.ImplementingPartnersuseKPI4

andtheguidanceprovided,10whichencouragesthemtocontextualisetheir

approachtomeasurement.Thismeansresultsarenoteasytocompare–

alimitationofboththemethodandtheapproachtomeasurement.The

varietyinmethodologieshasbeenaparticularchallengefortheBRACEDFund

Manager,whohaveneededtoengagewithandunderstandeachproject’s

methodologyindetailinordertounderstandandaggregatethedata.

There�is�significant�risk�of�obscuring�potentially�rich�detail�in�the�data�by�

reporting�only�a�number.ThereisalsoscopeforreportingerrorswhenKPI4

numbersaregeneratedacrosshouseholds,communities,regionsandprojects,

whicharethenaggregatedtotheprogrammelevel.Suchsimpleaggregation

acrossprojectsmaynotstanduptoscrutiny.Thisisduetomultiple

interpretationsoftheKPI4guidance(WilsonandYaron,2016).11Quantitative

approachestoKPI4measurementshouldthereforebecomplementedby

qualitativedatagatheringfortriangulationandasanexplanatorytoolfor

morein-depthandnuancedunderstanding.

10 KPI4guidance:https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/328254/BrAcEd-KPI4-methodology-June2014.pdf

11 Wilson,d.andYaron,G.(2016)‘Layingthefoundationsformeasuringresilience’.BrAcEdKnowledgeManagerWorkingPaper.Brighton:Itad.



http://www.braced.org/resources/i/?id=f57ceb4f-062a-494d-b3a9-7ebe09bfcd2a


The3Asframeworkhasenabledalevelofconsistencyandcomparability,

sheddingsomelightonthenatureofresilienceoutcomes.However,experience

todatealsotellsusthatthe3Ashavebeenfoundtobeausefulanalyticaltool

ratherthanoneformeasuringchangesinresilience.

Similarinsightshaveemergedfromearlyexperiencesintrackingtransformation:

Tracking�transformation�(KPI�15)�–�Emerging�insights�from�BRACED

Over-reporting�on�transformation�is�common.�Intheiryear1reporting,

ImplementingPartnersprovidedevidenceofsmallpositivechangesas

potentiallytransformational.Incrementalchangesmaybebestunderstood

withincontext,butitisimportantnottodilutetheconceptoftransformation

byreportingall changeastransformative.Adheringtothepillarsof

transformation(strategicplanningandpolicy;leadership,empowermentand

decision-makingprocesses;andinnovativeapproaches)canclarifythetypesof

transformationchangethattheBRACEDprogrammeaimstoachieve.

Capturing�transformation�that�is�‘catalytic’,�‘at�scale’,�‘and�sustainable’,�

was�challenging�in�year�1�project-level�M&E�reporting.�Thescorecardfor

transformationattemptedtocapturethedifferent,oftencountry-specific,

dimensionsoftransformationalchange,whileremainingsufficientlysimple

tobeunambiguous.However,theseconceptsdidnotappeartobeuseful

inprobingformoredescriptiveinformationinprojectreporting.Often,

theevidenceprovidedunderonecharacteristiccouldeasilyfitunder

another.Thesedefinitionsmaynotbehelpfulforexplainingthenatureof

transformationalchangeattheprojectlevel.Ultimately,transformational

changesrequireacritical masstoovercomepolitical,marketandothersocio-

economicandpoliticalfactorsthatcannotbeaddressedbyasingleproject.

A�qualitative�outcome�indicator�for�tracking�transformation�enables�the�

monitoring�(not�measuring)�of�the�likelihood�of�transformation,�relative�to�

expected�change.�Reportingtransformationalchangecannotbeaggregatedat

theoverallprogrammelevelinthesamewayasKPI4.Attheprogrammelevel,

resultscanbesynthesised.Theycanalsoenabletheidentificationofpatterns

andtrendsasameanstoassessoverallprogress,andtoteaseoutlessons,

ratherthanformaviewontheexpectedtransformationalimpactofBRACED.

Thereisagrowinginterestinunderstandingandcreatingtransformational

changesthroughprogrammesthatbuildresiliencetoclimateextremesand

disasters.TheBRACEDprogrammeprovidesacollectiveopportunitytolearn

abouttransformationalchangeinordertoimproveresilienceanddevelopment

outcomesbothwithinBRACEDprojectsandexternally.However,thedriveto

quantifyandqualifytransformationalimpactplacesunrealisticexpectations

attheprojectlevel.


BeyondthechallengesandlimitationsofKPI4andKPI15,thisyear’s

programme-levelsynthesisofBRACEDprojectreportsalsohighlightsthe

unaddressedchallengeofsystematicmonitoringofprogressagainstsuch

indicatorsinthecontextofshocksandstresses.Todate,itremainsunclear

inBRACEDhowwecanbestcontextualiseresultsagainsttheclimaticcontext

withinwhichprojectsoperate.Moreworkisrequiredatboththeprojectand

programmelevelstofullyunderstandthepracticeofmonitoringandreporting

progressinBRACEDresilience-buildingeffortsinthefaceofshocksandstresses.

“MoreworkisrequiredatboththeprojectandprogrammelevelstofullyunderstandthepracticeofmonitoringandreportingprogressinBRACEDresilience-buildingeffortsinthe

faceofshocksandstresses”

2.�Assessing�resilience�and�transformation�involves�tracking�progress�against�

multifaceted�processes.�Concepts�and�definitions�are�critical.Conceptualising

resilienceintermsofcapacitiesputshumanagencyatthecentreofresilience

building.Effortstoquantifyresilienceneedtotakecarewhendescribingchanges

inresiliencecapacities,asthesewillultimatelybecontingentonpeople’s

attitudesandchoices.TheAreasofChangeframeworkattemptstounpack

thelesstangiblebutvitalprocessesthroughwhichBRACEDprojectsinfluence

changesinattitudesandpractice.

Findingsfromtheyear1programme-levelsynthesishighlightthatImplementing

PartnerstendedtoreportagainsttheAreasofChange‘Knowledgeandattitudes’

and‘Capacitiesandskills’withadegreeofoverlap.Thissuggestsalackof

understandingofthedifferencebetweenthetwoandaneedformoreclarity.

Capacity,inparticular,isadifficultconcepttoframe,monitorandevaluate.

Whenreportingagainst‘Partnerships’,someImplementingPartnersreferto

formalpartnershipswithamemorandumofunderstanding,whileothersrefer

moretorelationshipsofcollaborationandcoordination.InBRACED,project-

levelpartnershipsdifferenormouslyinbothscaleandscope,soitisimportant

toclarifythetypeofpartnershipbeingreferredto.Meanwhile,information

providedon‘Decision-making’variedsignificantlyfromoneImplementing

Partnertoanotherafteryear1,withmostfocusingontherepresentation

ofwomenindecision-makingstructuresratherthanthebroaderissuesof

inclusionandparticipation.

Differinginterpretationsanddefinitionsacrossprojectshavenotjustbeen

anissueforqualitativedatareportinginBRACED.Theyhavealsopresenteda

challengefortheFundManagerwhenaggregatingprojects’quantitativedatato

theprogramme-levelBRACEDlogframe.Whilemuchattentionhasbeengiven

todefinitionsandM&Emethodologiesfortrackingchangesinresiliencelevels,

ourexperiencetodaterevealsthatasimilaremphasisshouldbeplacedonclearly

definingconceptssuchascapacity,knowledgeandinclusivedecision-making.


3.�Understanding�the�processes,�outcomes�and�context�of�resilience�is�

complex�and�there�are�no�straightforward�‘yes’�or�‘no’�answers.�Oneofthe�

BRACEDprogramme’sambitionsistoconstructanevidencebaseofwhatworks

andwhatdoesnotinbuildingclimateanddisasterresilience.Thisrequires

intensiveM&Eeffortscomparedtoothermoretraditionalprogrammes,bothin

termsofquantifyingchangesinresilienceandunderstandinghowthesechanges

haveoccurred.ImplementingPartners,whorelyheavilyonlocalpartnersinthe

field,arefacingcapacityconstraintsinapplyingtheBRACEDM&Eframework

andinmeetinglogframereportingrequirements.Thisisduetothenoveltyofthe

conceptsandframeworksbeingtested,thelevelofrigourandreflectionrequired

toengagewiththese,andtheM&Eframeworkbeingintroducedafterthedesign

ofproject-levelM&E.

Theintegratednatureofresilience-buildingprojectsrequiresintegratedanalytical

frameworks.Inordertobetterunderstandthecausalpathwayslinkingoutputsto

outcomes,aswellasthefactorscontributingtoresiliencebuilding,theBRACED

M&Eframeworkinvestigateschangeprocessesfromdifferentangles.Though

theywereoriginallyintendedtobeusedasacoherentwhole,thecomponent

conceptswithinthisframeworkweredevelopedseparately.Eachaspectwas

designedtounderstandsomethingslightlydifferentabouthowBRACEDprojects

andtheprogrammeasawholearebuildingresilience.Whileeach‘lens’provides

avaluableinsightintohowresilienceisbeingbuilt,theydonotprovidesimple

answerstotheresilience-buildingquestionevenwhentakentogether.

“Understandingtheprocesses,outcomesandcontextofresilienceiscomplexandthereare

nostraightforward‘yes’or‘no’answers”

Ourexperiencetodateindicatesthat‘zoomingin’tospecificAreasof

Changeordifferentresiliencecapacitiescanmeanlosingthe‘biggerpicture’.

Forexample,improvementsinthecapacityofspecificstakeholderstomanage

aspecificshockneedtobeassessedalongsidefeaturessuchastheroleofa

project’spartnersinthatparticularprocessandthesocio-culturalnormsinthat

specificcontext.Whenundertakingtheprogramme-levelsynthesis,wefound

thatunderstandingthecontributingfactorstoresilience-buildingrequiresdata

collectionandananalysisofwhyImplementingPartnersareengagingincertain

activitiesandhowthesearecontributingtobuildingresilience.Thisisdifficult

whendatafromdifferentframeworksisreportedinisolation.Whiledetailed

analysisiscertainlyrequired,theintegrationandtriangulationoffindings,

alongwithreflectiononhowtheyrelatetoeachother,isequallyimportant.

3.2�Rolling�out�M&E�frameworksAnnex6providesdetailsregardinghowtheMonitoringandResultsReporting

teamrolledouttheM&Eframework.Thereareanumberoflessonstoconsider

fortheremainderofBRACED,aswellforothersimilarprogrammes:


1.�M&E�designs�at�the�project�and�programme�level�should�be�in�place�before�

implementation�begins.Acommonunderstandingandvisionoftheprogramme

itselfanditsM&EiscriticalforthesubsequentsuccessofanyM&Esystem.The

startingpointfortheKnowledgeManagerMonitoringandResultsReportingteam

whendesigningandrollingoutanappropriateM&Eframeworkwastodevelop

aprogramme-widetheoryofchangetoarticulateBRACEDobjectives,assumptions

andpathwaysofchange.Thisstepwasparticularlyimportant,aseachBRACED

projecthasdifferentdefinitionsandresiliencemeasurementapproaches.

“Learningaboutwhatworksandwhatdoesn’tinbuildingresiliencerequiresownershipandcommitmenttointerrogatingtheprogramme-

leveltheoryofchange”

Learningaboutwhatworksandwhatdoesn’tinbuildingresiliencerequires

ownershipandcommitmenttointerrogatingtheprogramme-leveltheoryof

change.Inthisregard,engagingimplementingpartners,researchers,thedonor

andthefundmanagerindevelopingacommonvisionisrequiredrightfromthe

beginning.However,theBRACEDKnowledgeManagerwassetuponceBRACED

projectshadalreadybeendesignedandapproved,meaningtheprogramme-level

theoryofchangeandM&Eframeworkhadtoberetrofittedtotheexistingproject-

levelM&E.Toaddressthischallenge,theMonitoringandResultsReportingteam

followedaconsultativeprocess,combiningabottom-upandtop-downapproach,

todeveloptheBRACEDprogrammetheoryofchange.Thisincludedareviewof

all15project-leveltheoriesofchangeandlogframes,alongwithaconsultation

withtheprojectImplementingPartners.Attheprogrammelevel,frameworkswere

developedtoenable,asmuchaspossible,thestandardisationofconcepts,analysis

andreportingagainsttheprogramme-leveltheoryofchange.

Combiningabottom-upandtop-downapproachmeantthattheprogramme

couldretainproject-levelcontextspecificity.However,mostImplementing

Partnershadnotplannedorresourcedfordataanalysisandreflectionfor

programme-levelM&Ebeyondthemandatorylogframereportingagainstthe

relevantInternationalClimateFundKPIs.Havingtheprogramme-levelKnowledge

ManagerM&Eteaminplacefromthestartwouldhavehelpedavoidthissituation

andmayhavealsoensuredthatinformationreportingwouldnotbeseenasatop-

downrequirement.Instead,thiswouldpresentanopportunityforcriticalanalysis

andorganisationallearning,informingdecision-makingandimpactassessment

atboththeprojectandprogrammelevels.

2.�There�are�different�options�for�rolling�out�programme-level�M&E�

frameworks�and�systems,�but�each�comes�with�its�own�trade-off.Rolling

outprogramme-levelM&Eframeworksandsystemstoensureaggregationand

comparabilityrequiresagreementonkeyconceptsanddefinitionsaswellas

standardreportingtemplates.Otherwise,project-leveldifferencesinconceptual

framingsandterminologycanmakeitdifficulttomonitorandassessprogress


attheprogrammelevel,asfoundbyboththeKnowledgeManagerMonitoring

andResultsReportingteamandtheFundManager.TheBRACEDM&EGuidance

Notessetouttheprogramme-leveldefinitionstoImplementingPartners.

Follow-uptrainingandsupportthenenhancedtheirunderstandingofthese.

However,thissupportwasprovidedinthecontextofproject-levelM&Ealready

beingdesigned.(TheBRACEDKnowledgeManagerwassetupaftertheproject

logframes,theoriesofchangeandM&Eplansweredefined.)

“Rollingoutprogramme-levelM&Eframeworksandsystemstoensureaggregationand

comparabilityrequiresagreementonkeyconceptsanddefinitions”

Optionsandtrade-offsforrollingoutprogrammeM&Eframeworksinclude

decisionsaboutthetypeandlevelofsupportprovidedtoprojectpartners.

WhencomparabilityandaggregationareakeypurposeoftheM&Esystem,

moreresource-intensiveoptionsforproject-levelsupportmaybebetter–onesuch

examplewouldbeM&EtrainingforprojectImplementingPartners,complemented

byongoingone-to-oneinteractionswiththem.Thisisunderlinedfurtherin

thecontextofaresilience-buildingprogramme,asknowledge,capacityand

experiencesarestillemerging.Wehavetakenamoreresource-intensiveapproach

atkeymoments:inthedesignoftheM&Eframeworkanditsapplicationby

ImplementingPartnersto(a)project-levelM&Eand(b)year1reporting.However,

wehavealsousedlessresource-intensiveoptions,suchaswrittenM&Eguidance

andone-offengagementswithgroupsofImplementingPartners.Rollingout

programme-levelM&Eframeworksandsystemsmustfindtherightbalance

betweenbothapproachesandallowforcontinualadjustmenttothegrowingbody

ofknowledgeandexperience.EstablishingtheBRACEDprogramme-levelM&E

frameworkwouldhavebeeneasierifithadbeendevelopedatthesametime

asthe15BRACEDprojects’M&E.

3.3�Reporting�on�resilienceThiswasthefirstyearwhereImplementingPartnerscollecteddataand

reportedaboutchangesinresilienceandtheKnowledgeManagerMonitoring

andResultsReportingteamsynthesisedandanalysedthedata.FromaKnowledge

Managerperspective,thefirstyear’sreportingprocesswentwelloverall.The

M&EframeworkitselfhasbeenfoundtoberelevanttotheworkoftheBRACED

programmeandtheBRACEDprojectsthemselves.ImplementingPartnerreports

wereparticularlyimpressive,giventhat:

• Project-levelM&Esystemswerenotsetuptoreportconsistentlyagainst

theBRACEDM&Eframework.

• Thereportingwasearly,comparedtowhenresultsintermsofresilience

buildingcouldrealisticallybeexpected.




• ImplementingPartnerswerecompletingthetemplatesforthefirsttime.

(SeeAnnex4forourfurtherreflectionsonyear1reporting.)

FromanImplementingPartnerperspective,theexperienceofcompleting

thetemplateswashelpfulforinternalreflectionforsomeandaddedvalueto

theirownprojectM&E.Therewere,however,anumberofchallengescited

incompletingthetemplatesagainsttheM&Eframework.Thiswasoftendue

toImplementingPartnersnothavingrespondedtoouroriginalfeedback

toalignproject-levelM&Etotheprogrammelevel.Ithasbeenanongoing

tensionandchallengeforImplementingPartnerstomonitorandreport

againsttheprogrammelevelM&Eframeworkwhenthiswasintroducedafter

projectimplementationbegan.(SeeAnnex4forfurtherImplementingPartner

reflectionsonthefirstyear’sreportsandreportingprocess.)

Thereareanumberoflessonstoconsiderforthereportingandanalysisfor

years2and3:

1.�The�level�of�data,�analysis�and�reflection�required�from�Implementing�

Partners�in�BRACED�is�high.Asalreadynoted,thenatureofthequantitative

andqualitativedatainBRACEDischallengingandgoesbeyondtheM&E

requirementsofmosttraditionalprogrammes.Thisongoingmonitoringwork

andannualreportingisinthecontextofImplementingPartnersalsoundertaking

projectmid-termreviews(mid-year2),wheretheyareencouragedtofurther

understandhowchangeishappeninginmoredetail,buildingontheirfirst

annualreportanalysis.Thisreportingworkloadissignificantandthelevelof

informationandanalysisthathasbeenprovidedbyImplementingPartnersin

theiryear1reportontheM&Eframeworkisadmirable.

Intermsofrecommendationsforthefuture,severalImplementingPartners

highlightedtheutilityofthetrainingworkshopheldonthereportingtemplates

andrequestedthatsimilarsupportbeprovidedagain.Itwasrecognisedthat,

thoughthetemplatesthemselvesarecomplex,thesecondannualreportwill

beeasiertocompletenowthatImplementingPartnersaremorefamiliarwith

these.Nextyear’sreportingwillseektobuildoncollectivelearningfromtheyear

1reportingprocess.Itwillaimtoachieveanoptimumlevelofdatainannual

reportsintermsofquantity,relevanceandqualitytosupportbothprogramme-

levelsynthesisandproject-leveladaptiveprogrammingintheremainder

ofBRACED.

2.�It�is�too�early�in�the�programme�to�provide�evidence�about�substantive�

change.Whengivingfeedbackonthefirstdraftsofyear1reports,werepeatedly

requestedthatImplementingPartnersprovidemoreinformationonthemain

changestheirprojecthascontributedtoandtheproject’scontributiontosuch

results,withconcreteexamplesandevidenceofhowtheyknew,wherepossible.

MostImplementingPartnerswereabletoprovidefurtherinformation,butthis

wasoftenlimitedbytheavailabilityofevidence.Reasonsforthisincluded:

theprojectM&Esystemnotbeingsetuptocollectthedata;thetimeneeded

forthesetypesofchangetohappenbeingmorethanoneyear;anddelaysin

projectstart-upandimplementation.TheFundManagerreflectedthat,although

resilience-buildingresultsmaynothavebeendemonstrablebytheendofyear1,


theImplementingPartners’workinsettinguptheprojectinfrastructuresnow

meansthatresiliencebenefitswillbegintobeseenfrommid-year2onwards.

3.�Truly�engaging�with�qualitative�and�explanatory�frameworks�requires�

shifting�mindsets�from�accountability�to�learning-oriented�M&E.�TheBRACED

M&Eframeworkaimstochallengeproject-leveltheoryofchangeassumptions

andfosterdiscussionandunderstandingaroundhowprojectsmovefromoutputs

tooutcomesandwhathappensinbetween.However,itmaybetooearlyinthe

programmetoexploretheprocessesbehindtheprogressmadetowardsoutcomes

andhowcontextualfactorsareenabling/constrainingchange.Implementing

Partnershavespentalargeproportionofyear1settingupstructuresand

partnerships,andtestingactivities.ItmayalsobethecasethatImplementing

Partnersarenotaccustomed(orincentivised)tomonitor,reflectandreporton

suchthingsfromalearningperspectiveratherthananaccountabilityone.The

FundManagerhasexperiencedasimilarchallenge.ImplementingPartnershave

spentconsiderabletimeandeffortinaddingupcomplexfigurestocalculate

thenumbersofpeoplesupportedandhowmanyhavehadtheirresiliencebuilt.

However,reflecting,analysingandreportingaboutwhatthesefiguresmeanhas

receivedlimitedattentioninyear1.TheKnowledgeManagerandFundManager

agreeontheneedtomovebeyondthenumbersandwillworktogetherinthe

remainderofBRACEDtoencouragefurtherreflectionbyImplementingPartners

onthecombinationoftheirquantitativeandqualitativedata.

3.4�Aggregating�and�synthesising�data�at�scaleTheprocessofbringingtogetherandsynthesisingevidencefromImplementing

Partners’year1projectannualreportsattheprogrammelevelhastaughtusthat:

1.�It�is�important�to�achieve�an�optimal�level�of�comparability,�while�also�

retaining�project-specific�visions�and�understanding.�Akeyobjectiveofthe

BRACEDM&Eframeworkwastoensureenoughcomparabilitybetweenproject-

levelM&Etomonitor,measureandunderstandtheresilience-buildingefforts

ofBRACEDprojectsattheprogrammelevel.However,aspreviouslymentioned,

ImplementingPartnershavestruggledtoadheretotheoverarchingprogramme-

widedefinitionsoftheM&Eframework,particularlywhenoutliningthedifferent

levelsofchange(‘expect’,‘like’and‘lovetosee’)thatwerebothanticipated

andrealisedacrossthefourAreasofChange.ItispositivethatImplementing

Partnershaveengagedwithandtakenownershipoftheseandinterpretedthem

fortheirprojectcontext.Broaddefinitionsareintendedtofacilitateproject-to-

programmedatasynthesis.However,wheretheoverarchingdefinitionshave

notbeenfollowed,comparableanalysishasbeenmademoredifficult.�We

soughttoovercomethischallengebysynthesisingproject-leveldataagainstthe

programme-leveldefinitionsassetoutintheoriginalBRACEDM&EGuidance

NotesandusingtheMonitoringandResultsReportingteam’sexpertjudgement

wheredifferencesarose.




2.�Synthesising�and�aggregating�data�while�retaining�context�specificity�not�

only�takes�time.�It�also�requires�carefully�developed�synthesis�methodologies.�

Ashighlightedinthecompanionprogramme-levelsynthesisreport,�BRACED

projectscoverawiderangeofissuesandoperateinverydifferentcontexts.

Asanexample,theseinclude:

• securing,servicingandpromotingtrans-borderlivestockmobilityacross

theSahel

• sharingskillsandtechnologytoimprovetheuptakeofclimateinformation

inEthiopia

• supportingsmallholderfarmersinNepaltotakeadvantageofeconomic

opportunitiesandinvestmentsinclimate-smarttechnologies.

Thisprojectandcontextspecificityhasproventobeachallengeforthe

programme-levelsynthesisandaggregationofdiversedata.Wesoughtto

overcomethisbyundertakingathematicsynthesisanalysis,enablingthe

identificationofcommonpatternsandthemesacrossthesetofprojects.

Thisapproachhasrequiredconsiderabletimeandresources.

3.�Dealing�with�self-reporting�bias�requires�triangulation�with�other�sources�

of�information.Intheiryear1reports,ImplementingPartnersvariedbetween

over-reportingchangesseensofaranddownplayingorunder-reportingchanges

thattheMonitoringandResultsReportingteamalreadyknewaboutfromother

sources.Thissuggeststhatthereportingprocessdidnotencourageadequate

reflectionorconsistencyacrossprojects.Toensurethattheprogramme-level

synthesiswasrepresentativeofwhatwasactuallyhappeningontheground,

wecomplementedtheproject-leveldatawithourexistingknowledgeofeach

project.Wealsocross-checkedandgap-filledbyconsultingKnowledgeManager

staffwhohadworkedwithImplementingPartnersinthefieldandreferringto

otherprojectdocumentation.TheFundManageralsohadtotriangulatethe

quantitativeinformationprovidedbyImplementingPartnersinordertocomeup

withsensible,realisticnumbers,withsomeImplementingPartnersover-reporting

onprogress.Infutureyears,wecoulddrawmoreontheFundManager’s

knowledgeandperspectivebasedontheirongoingmonitoringandinteraction

withImplementingPartners.

4.�The�lead�time�between�project-level�annual�reporting�and�programme-level�

annual�synthesis�is�long�and�limits�programme-level�real-time�learning�and�

flexibility.�Duetothescaleandlevelofanalysisrequiredforaprogrammelike

BRACED,thereportingtaskforbothImplementingPartnersandtheKnowledge

ManagerMonitoringandResultsReportingteamissignificant.InJune2016,

ImplementingPartnerssubmittedtheirreportsonprogressmadeuptotheend

ofMarch2016.Thiswasthenfollowedbyarelativelylongprogramme-level


analysisandsynthesisexercise,lastingfivemonths(JulytoNovember2016),12

todrawcross-project,programme-levelfindings,lessonsandrecommendations.

Thelengthofthisprocesswasduetothepreviouslymentionedchallenges

thatthenuancesoftheprocessesandoutcomesofBRACEDpresented,namely

thedifficultiesinprovidingclear‘yes’or‘no’answerstohowresiliencehas

beenbuilt,alongwiththeneedtorigorouslyaggregateandsynthesisedata

whileretainingcontextspecificity.Thesubsequentprogramme-levelsynthesis

findingsandrecommendationsidentifiedinNovember2016shouldthereforebe

consideredasareflectionofthesituationuptoMarch2016.Inthecontextof

evolvingchange,thisleadtimemaylimitthefindings’relevanceandpotential

impactonprogrammeandprojectdecision-making.

12 thetimegapbetweenreceivingImplementingPartnerreportsandstartingtheprogramme-levelsynthesisispartlyexplainedbythetimerequiredfromtheKnowledgeManagerMonitoringandresultsreportingteamtocontributetothedFIdBrAcEdAnnualreview(July-August2016).

28ROUTES�TO�RESILIENCE:�LESSONS�FROM�MONITORING�BRACED� BuILdInGonthISLEArnInG

M&EforresilienceprogrammingisinitsearlydaysandBRACEDislearning-by-

doingintermsofwhatkindsofprogresscanbemeasuredandreported,andhow

besttodothis.TheBRACEDprogrammeasawholeistestingasetofframeworks

andapproachesthatneedtobefine-tunedasevidenceemergesandexperience

isbuilt.Inyear1,BRACEDImplementingPartnershaveembracedanewway

ofmonitoringandreportingchange.Wehavelearntagreatdealasaresultof

takingaprogramme-levelviewofhowresilienceisbeingbuiltinBRACED.

“TheBRACEDprogrammeasawholeistestingasetofframeworksandapproachesthatneed

tobefine-tunedasevidenceemergesandexperienceisbuilt”

Inthispaper,�wehavesharedourexperiencesinBRACEDsofarandhopethatthis

willgoontocontributetoongoingandfutureeffortsindesigning,implementing

andreportingagainstM&Eframeworksofresilience-buildingprogrammes.BRACED

monitoringandresultsreportingeffortswillcontinueduringyears2and3andwe

willcarryonreflectingonexperiencesandlessonslearnt.

Image:GeorginaSmith/Worldfish

4.HOW�CAN�BRACED�BUILD�ON�THIS�LEARNING?


ThelessonsandexperiencetodatewillbeusedtoimprovetheBRACED

M&Eframeworkandreportingforyears2and3.TheMonitoringandResults

Reportingteamwillexploreoptionstoimproveproject-toprogramme-level

reporting,including:

• simplifyingreportingtemplates,wherepossible

• reducingthereportingleadtimebetweenproject-andprogramme-

levelreporting

• fosteringpeer-to-peerlearning(inconjunctionwiththebroaderKnowledge

Manager’songoingwork)

• creatingsystematicopportunitiesfordatatriangulationacrossprojects.

BRACEDisnearlytwoyearsintoitsthree-yearimplementationtimeframe

andproject-andprogramme-levelM&Eisalreadysetupandestablished.

Therearethereforesomelimitationstowhatcanbeadaptedandachieved

intheremainderoftheprogramme.Inthiscontext,oursuggestionsbelow

areforboththeBRACEDprogrammeandothersimilarprogrammes.

“M&Eexpertsneedtoengageinmorerefinedandcomplexdatacollectionandanalysis

processesthanistypicalthroughexplanatoryandqualitativeindicatorframeworks”

EstablishingM&Eframeworksandreportingsystemsforresilience-building

programmesandprojectsremainsanareawheremoreknowledge,experience

andlearningarerequired.LessonstodatehighlightthatM&Eforresilience

buildingrequiresapproachesandpracticesthatgobeyond‘businessas

usual’.ThisinvolvesM&Eexpertsengaginginmorerefinedandcomplexdata

collectionandanalysisprocessesthanistypical.Italsorequiresthesetting

upofexplanatoryandqualitativeframeworkswithstronglinkagestoresearch

efforts.Wehavefound,throughundertakingtheprogramme-levelsynthesis

ofBRACEDproject-leveldataforthefirsttime,thatgenuinelytryingtomeasure

resiliencemeansweneedtomoveawayfroman‘accountability’M&Eculture.

Goingforward:

• Implementing�Partners�should�enhance�their�ongoing�monitoring�

and�results�reporting�efforts�by�taking�a�more�reflective�and�critical�

approach.Thiscouldchallengeprojectassumptionsandwillbuildabetter

understandingofhowtobuildclimateanddisasterresilienceinfragileand

vulnerablecontexts.

• The�Monitoring�and�Results�Reporting�team�should�consider�how�to�

encourage�this�critical�reflection�and�dialogue,�as�well�as�innovative�

ways�of�capturing�rich�and�reflective�data�from�BRACED�projects.�There

arelimitstowhatreportingtemplatesalonecanachieveinthisregard.We


thereforeplantoprovidefurthertrainingtoImplementingPartners,along

withlight-touchhelpdesksupport.

• Programmes�like�BRACED�need�to�find�and�resource�efficient�ways�of�

achieving�a�sufficient�level�of�reflection�and�learning�for�the�benefit�

of�both�project-�and�programme-level�evidence�generation.Ideallythe

programme-levelM&Eframeworkshouldbedesignedinconjunctionwith

theproject-levelframeworks.

“QuantitativeindicatorslikeKPI4enabledataaggregationandsynthesisatthe

programmelevel,butmisscriticalinformationforongoingevidencegenerationandlearning

aboutwhatworksandwhatdoesnotinbuildingresilienceinBRACED”

Indicatorsenabledataaggregationandsynthesisattheprogrammelevel,but

misscriticalinformationforongoingevidencegenerationandlearningabout

whatworksandwhatdoesnotinbuildingresilienceinBRACED.�Thedesire

tounderstandifandhowresilienceisbeingbuiltisatthecoreoftheBRACED

M&Eframework.ThoughKPI4mightbenecessaryforDFIDtotrackoutcome-

levelchangesacrossawiderportfolioofprogrammes,itlimitsgeneralisable

lessonsandlosesimportantdetailaboutcontext,particularlywhenindicators

requirethequantificationofacomplexconceptlikeresilience.KPI4dataneeds

tobecomplementedbythesystematicM&Eofresilienceinthecontextofactual

shocks.Thiswillenableustobetterunderstandthestabilityofoutcome-level

changesovertimeandhowcommunitieslearnand‘bouncebackbetter’from

disasterevents.Goingforward:

• Implementing�Partners�are�in�a�unique�position�to�contribute�to�

knowledge�generation�about�how�to�quantify�the�number�of�people�

whose�resilience�has�been�built�(KPI�4)�at�the�project level.The

MonitoringandResultsReportingteam,togetherwithwidermembersofthe

KnowledgeManagerteamandtheFundManager,shouldfurtherexplorethe

advantagesanddisadvantagesofKPI4attheprogrammelevel,aswellasthe

opportunitiesandtrade-offsofresilienceindicatorsindifferentcontexts.

• When�designing�and�funding�similar�programmes�in�the�future,�DFID�

should�adopt�a�pragmatic�and�realistic�view�on�the�feasible�level�of�

outcome-level�data�and�evidence�generation�in�a�three-year�programme�

like�BRACED.�Resilience-buildingeffortsarenotonlycomplex,butalso

involveprocessesofchangethattaketimetomaterialise.Prioritisingannual

datacollectioneffortsagainstquantitativeindicatorsmaycomeatthecost

oflosingcriticalevidenceaboutwhatworksandwhatdoesnotinbuilding

climateanddisasterresilience.


• Programmes�like�BRACED�should�consider�having�a�diverse�set�of�

methodologies�and�analysis�in�place�for�interrogating�quantitative�

outcome-level�resilience�indicators.�Theyshouldbepragmaticabout

whatsortofoutcome-leveldataandinformationtheycanexpectina

three-yearperiod.

“Whilemuchattentionhasbeengiventoproject-levelapproachestomonitoringand

measuringresilience,programme-leveleffortsfaceauniquesetofchallenges”

Whilemuchattentionhasbeengiventoproject-levelapproachesto

monitoringandmeasuringresilience,programme-leveleffortsfaceaunique

setofchallenges.Theseinclude:theharmonisationofM&Eframeworksacross

awideportfolioofprojects,real-timelearningfromprojecttoprogrammeand

back,andtheaggregationofhighlycontextualquantitativeandqualitativedata

throughflexibleyetstandardframeworks.Todate,therearelimitedexamples

(andsubsequentliterature)fromotherprogrammesaddressingsuchchallenges.

BRACEDhasbeenlearning-by-doingonanongoingbasis.Goingforward:

• The�Monitoring�and�Results�Reporting�team,�together�with�Implementing�

Partners,�should�consider�ways�to�further�capture�their�monitoring�and�

results�reporting�experiences�within�BRACED.Thiswouldbenefitboth

BRACEDandotherexistingandfutureresilience-buildingprogrammes.

• More�transparent�and�reflective�discussions�are�required�to�address�

the�challenges�and�lessons�learnt�in�establishing�M&E�frameworks�for�

resilience-building�programmes.ProgrammeslikeBRACEDshouldalso

shareandcontributetobuildingknowledgeandexperienceinthisrelatively

newareaofwork.

4.1�Question�for�further�reflectionWiththeaimofcontributingtoongoinglearningaboutresilienceprogramming,

wewishtoengageBRACEDImplementingPartners,thebroaderBRACED

KnowledgeManager,theBRACEDFundManager,DFIDandwideraudiencesin

thefollowingcriticalquestionthatemergesasaresultofreflectionsandlearning

againsttheM&Eframeworksofar.Emerginginsightsfromthispapershedsome

lightforinitiatingdiscussion.However,theBRACEDprogrammeshouldcontinue

toanswerthisquestionthroughoutitslifetime:

How�complex�does�M&E�for�resilience�need�to�be?TheBRACEDM&E

frameworkbringstogetherthreedifferentlensesintotheanalysisofproject-

andprogramme-leveldata.Experiencetodate,bothfromImplementingPartners

andtheBRACEDKnowledgeManager’sMonitoringandResultsReportingteam,

indicatesthat:


• Whiletheframeworkprovidesamultidimensionalviewintothevarious

levelsofcomplexity,itrequiressignificanttime.

• Italsonecessitatesdifferent,non-traditionalwaysofengagingwithdata.

• Itdoesnotandcannotprovidesimple‘yes’or‘no’answersaboutwhether

resiliencehasbeenbuiltand,ifso,how.

“Theessenceof‘resilience’isthatchangeandprogressarenotlinear–soresultsreporting

shouldnotbelineareither”

Asoutlinedinourcompanionreport,‘Routes to resilience: insights from BRACED

year 1’,theessenceof‘resilience’isthatchangeandprogressarenotlinear–so

resultsreportingshouldnotbelineareither.TheBRACEDM&Eframeworkis

complexenoughtobeabletounderstandBRACEDresilience-buildingefforts,

butcoulditbemadetobemoreuser-friendly,whilestillretainingthecomplexity

andnuancesofresilience-buildingprojects?



33ROUTES TO RESILIENCE: LESSONS FROM MONITORING BRACED ANNEX

Annex�1:�Monitoring�and�evaluation�in�BRACEDM&Eactivitiesareundertakenatboththeprojectandprogrammelevel

withinBRACED.

Eachofthe15BRACEDprojectshasitsowntheoryofchange,logframe,M&E

planandM&Esystem.EveryImplementingPartnerreportsprogressandlearning

againstitslogframeandtheoryofchangeonanannualbasis.Eachprojectisalso

carryingoutamid-termreviewandafinalevaluation.

Progressagainsttheproject-levellogframesisreportedtotheBRACEDFund

Manager,whomanagestheprojectgrantsonbehalfofDFID.TheFundManager

thenaggregatesandreportsBRACEDprojectresultsagainsttheBRACED

programme-levellogframe.TheFundManageralsoundertakesongoing

projectperformancemonitoring.

Attheprogrammelevel,thereisanoverarchingtheoryofchange(seeAnnex2)

andasetofMonitoringandResultsReportingandEvaluationactivities,ledby

theKnowledgeManager.TheKnowledgeManagerhasprovidedsupportand

guidancetoprojectImplementingPartnerstoensurethealignmentofproject-

andprogramme-levelM&EthroughtheM&Eframework.Progressandlearning

againstproject-leveltheoriesofchangeisreportedbyImplementingPartners

totheBRACEDKnowledgeManagerusingtheM&Eframeworkinorderto

understandhow resilience is being builtat the programme level.Thispaperis

basedonthoseinputs.

WithintheBRACEDM&Esystem,theFundManagerisresponsibleforoverseeing

thedeliveryoftheBRACEDprojects.Theydothisby(a)collectingfinancial

accountabilityinformation,(b)annuallyreportingattheoutput/activitylevel,

and(c)collatingrelevantdataagainsttheBRACEDmandatorykeyperformance

indicatorsoftheInternationalClimateFund,includingKPIs1,4&15.

TheKnowledgeManagerM&Eteamfocusesonbuildingandsharingevidence

andknowledgethroughatheoryofchangeapproach.Alongsidesubstantive

monitoringandresultsreportingactivities,theBRACEDKnowledgeManager

isalsoundertakingasetofevaluationactivitiestounderstandtheextentto

whichBRACEDprojectinterventionswork(seeAnnex5).




Annex�2:�BRACED�theory�of�change

SeeNote2oftheBRACEDM&EGuidanceNotesforafullnarrativeoftheTheoryofChange.

BRACED invests in projects directly targeting:

Working with a whole variety of stakeholders:

Assumptions:effectiveness of the BRACED fund

To support changes in 7 thematic areas, which will strengthen 4 areas of change:

Assumptions:BRACED outputs

Which will directly deliver a set of 4 OUTPUTS at different scales leading to the BRACED OUTCOME:

From which BRACED will derive lessons to deliver a set of ‘amplified’ results by influencing policy making and development planning from the international to the local level:

And, in the long term will bring about:

Assumptions:BRACED amplified effect

Impact:Improved well-being of poor people, despite exposure to climate extremes and disasters

Households and community level

Components A&B

Regional/ international organisations

National government

Sub-local government

Research institutions

NGOs CSOs

Communities

Thematic areasClimate & weather information

Technology & innovation

Gender & social equality

Markets & local economic empowerment

Delivery of basic services

Governance & natural resource management

Resilience concepts

Areas of changeKnowledge & attitudes

Capacity & skills

Partnerships

Decision-making

National and local government capacity

Component D

Knowledge, learning and evidence

Component C

Output 4:Improved policies in

targeted areas

Output 2:Increased capacity of local

government, CSOs and private sector to respond to climate-related

shocks and stresses

Output 1:Poor people receive support to reduce their

vulnerability to climate-related shocks and stresses

Assumptions:BRACED outcomes

Outcome:Poor people in developing countries have improved their levels of resilience to climate-related shocks and stresses.

Measuring the three dimensions of resilience:Anticipatory, Absorptive and Adaptive capacity.

Output 3: Better understanding of w

hat works in

building resilience to climate extrem

es and disasters

BRACED

amplifie

d

results



Annex�3:�Project-to-programme�synthesis�methodology

Project-level analysis and

synthesis

Step 1: Systematic review

and screening against project-level

grid

Step 2: Characterising project-level

(organising data – key word search)

Step 3: Project-level

synthesis against analytical

framework

Step 5: Thematic analysis

Step 4: Comparative analysis and identification

of themes

Project-to-programme

Programme-level

synthesis

Framework synthesis

Thematic synthesis

Consultation with ongoing research

streams to deepen analysis

of findings


Annex�4:�Further�Knowledge�Manager�and�Implementing�Partner�reflections�on�experiences�of�reporting�against�the�BRACED�M&E�framework

Implementing�Partners’�informal�feedback�on�the�reporting�process

Basedontheyear1reports,thereseemstobebuy-in�across�all�

Implementing�Partners�to�the�key�concepts�of�the�M&E�framework

andreportingagainstthese.Onthewhole,ImplementingPartnerstold

theMonitoringandResultsReportingteamthattheyfoundtheframework�

a�relevant�way�of�telling�their�project’s�story.NotonlydidImplementing

Partnersreportinformationagainsteachpartoftheframework,many

alsoreferredtotheconceptsthroughoutthereport,suchastalkingabout

resilienceoutcomesintermsofcapacitiestoanticipate,adapttoandabsorb

shocksandstresses.ThisisasignificantachievementgiventhattheM&E

frameworkwasdevelopedafterproject-levelM&Esystemshadlargelybeen

designed,thereforemeaningithadtobe‘retrofitted’.Overall,thereisa

good�understanding�of�the�various�parts�of�the�M&E�framework.

ImplementingPartnerswereinformallyasked:Did you find the BRACED M&E

framework a relevant way to frame/structure the ‘story’ of your project? Are there

any gaps?

Overall,Implementing�Partners�seem�to�find�the�M&E�framework�a�relevant�

way�to�tell�the�story�of�the�project,withonesayingit‘has guided us to tell

our stories in a realistic way’.AnotherImplementingPartnertoldusthat‘it was

helpful in drawing lessons across the consortium’.However,membersofthesame

team‘felt it did not add much to [their] existing framework for evaluating progress

and impact, and so was more useful to the KM than to IPs themselves’.

IntermsofthespecificaspectsoftheM&Eframework,

OneImplementingPartner‘felt the request for an analysis of “changes in

capacity” in the Areas of Change [template] overlapped significantly with the

3As [template]. As a result, we struggled to parse out which unique insights the

KM was looking for in one section versus another’.

Anotherfoundit‘difficult to fit [sub-indicators] under the 3As, though

evidently they all fold up under KPI 4’asthey‘were not at the time developed

around the 3A capacities’.

Anothersuggestedthat‘there could be more consideration of “depleted

capacities”. That is, while the work of BRACED projects [is] building the

adaptive, anticipatory and absorptive capacity of the communities, there are

very often ongoing shocks and stresses that require participants to draw on

those sources of capacity in the midst of building them up. The [template]

talks about the potential trade-offs between the 3As, but I believe a common


part of many projects’ stories will be this cyclical pattern of growth in capacity

development, then a shock or stressor that calls them to draw upon them, and

then potentially another cycle of growth, etc. It is a non-linear projection that

may not clearly be evident in the annual milestone trackers’.

ImplementingPartnerswereinformallyasked:How appropriate were the Part

2 templates for capturing your project’s story? What would you change about

the M&E framework or the report template to tell your project’s story better

(remembering that, next year you will have more results to share)?

SomeImplementingPartnersreportedthattheexperience�of�completing�the�

templates�was�helpful�for�internal�reflection�and�added�value�to�their�own�

project�M&E:

‘We appreciated all the KM’s efforts to use the template to help us tell the

real story of what is happening on the ground in a factually supported way.’

Thereportingexercisewas ‘definitely a great opportunity for us to look into

our own achievements more critically and systematically – asking ourselves

various questions about the project performances/challenges, which we would

have otherwise not even thought of. In the beginning, although it appeared

highly time consuming and exhausting, with the completion of the report,

we felt very satisfied with greater insights and deeper understandings about

the project status and future needs.’

‘We felt the reporting process was useful in reducing the gap between our

project’s theory of change and BRACED programme’s theory or change,

and plan to adapt our own [theory of change] and logframe, based on key

takeaways gleaned from this exercise during the mid-term review process.’

However,oneImplementingPartnerreflectedthat:‘Some of the changes that

we were asked to report on cannot be objectively evaluated within the project

lifetime, and so felt like less of a useful exercise. For example, select aspects

of our theory of change cannot be objectively measured within the lifetime

of the project, because we are looking at long-term changes in resilience

(+5 years).’ Alongwiththis,anotherImplementingPartnerfelttherewasnot

spacetosharetheiralreadydocumentedcasestudiesandsuccessstories.

Therewerea�number�of�challenges�cited�in�completing�the�templates�against�

the�M&E�framework.ThesewereoftenduetoImplementingPartnersnot

respondingtooriginalKnowledgeManagerfeedbacktoalignproject-levelM&E

totheprogrammelevel:

Inrelationtotheidentificationofkeystakeholders/partners, Implementing

Partners asserted the following:

‘Though we, to our best of our knowledge, identified the stakeholders relevant

to Areas of Change, it was a bit confusing how to categorise them. It was also

difficult to ascertain if we have missed any important stakeholders which we

could have elaborated.’


‘Ours was one of three to four consortia who did not include markers

(‘expect’, ‘like’, ‘love to see’) in our baseline, as we were told these were not

mandatory. As a result, our project’s ‘story’ was less suited to the template

than those who set benchmarks at the start of the project.’

‘We continued to struggle to define and apply the concepts ‘scale of impact’

and ‘catalytic effect’ embedded in [the template] on transformative impact

to our analysis.’

Intermsofrecommendationsforthefuture,several�Implementing�Partners�

highlighted�the�utility�of�the�training�workshop�held�on�the�reporting�

templatesandrequestedthat,shouldonebeheldforthenextreport,they

wouldlikethistobeorganisedfurtherinadvance,toenableparticipationand

attendancebyfield-basedteammembers:

‘We’d like to request that, to the maximum extent possible, future sessions

like the one held in May either be held in Africa, perhaps as part of the annual

learning event, and/or communicated as far in advance as possible, so that

we can better plan for attendance, adequate coverage, as well as conduct

the logistics necessary for travel (visas, etc.) We recognise the limitations

surrounding this round, but just wanted to emphasise that we might have

been able to better leverage the learning for our staff and/or contribute

more efficiently had we been able to send more and/or more appropriate

staff for the workshop.’

‘Thanks again for all of your terrific work in pulling together the workshop

session. It really was quite helpful and insightful. If there are considerations

of holding a similar event next year, I would also suggest ample notification

so country staff who know the programmes best would be able to attend

and get visas… [The in-country] team were very understanding and terrific

at interpreting our insights second-hand, but I’m sure [they] could contribute

to the conversation in a richer way than I was able by attending in-person

[themselves].’

AnotherImplementingPartnerreflectedthat: ‘it can be challenging to think in

terms of strategic, observable change, rather than, say, outputs; it is difficult to

bring rigour to a largely qualitative, narrative-based section.” Inordertoimprove

thequalityoftheirreportnextyear,theyaskedfortheMonitoringandResults

Reportingteamtoshare“‘best in class’ examples for some of the answers and how

the IP substantiated it, something we could emulate for the next round, including

whether the IP used their normal M&E, conducted special surveys and/or collect

testimonials? We feel a concrete example would be extremely helpful for several

“templates”’.

Itwasrecognisedthatalthoughthetemplateitselfiscomplex,thesecondannual

reportwillbeeasiertocompletenowthatImplementingPartnersaremore

familiarwiththetemplate.


Appropriateness�of�M&E�reporting

Duetothenumberoftemplates,withImplementingPartnerscompletingthese

inafairlycomprehensivemanner,itwasahugetasktogothroughall14project

annualreports,whichoftentotalled60–80pageseach.13Anumberoffactors

increasedthesizeofthis:

• Sometimestherelevant�information�was�not�always�provided�in�the�

template�intended�for�that�data.�Forexample,changesreportedinthe

‘likelihoodoftransformation’templatewereoftenrelevantforinclusion

undertheAreasofChangebutweremissedornotreferenced.Therewas

thereforeaneedtoconsidereachreportasawhole,ratherthanlooking

tospecifictemplatesforspecificinformation,whensummarisingand

synthesisingattheprojectlevel.However,thisdidenabletheteamto

makesenseofdataprovidedbytriangulatingwithinformationelsewhere

inthereport.

• Sometimesnot�enough�detail�was�provided�by�the�Implementing�

Partner�for�the�reported�change�to�be�fully�understood�and�included

inthesynthesis.Forexample,theymayhavereportedthataparticular

stakeholderhadappliedtheirknowledgebutnotwhatknowledgeand

how.Goingforward,exampleswouldbeusefultoensurethatthechanges

reportedarenottoogeneric/vague,especiallygiventheanecdotalnature

ofevidenceinyear1.SomeImplementingPartnersweregoodatproviding

these.Also,some�were�more�self-critical�than�others,�and�downplayed�

or�under-reported�thingsthattheMonitoringandResultsReporting

teamknewtheywereactuallydoingverywell,suggestingthatthe

reportingprocessdidnotencourageadequatereflection.Toensure

thispaperisreflectiveofwhatisactuallyhappeningontheground,the

MonitoringandResultsReportingteamcomplementedthedatawith

theirexistingknowledgeoftheprojectandalsocross-checked/gap-filled

withthelogframeactivityreportingandotherprojectdocumentation

(e.g.theoriesofchange,websitesandpublications).Therewasn’tcapacity

torefertoallprojectbaselinesindetail,thoughthiswouldhavebeenuseful

forunderstandingwhatchangeshadbeenachievedduringthelifetime

oftheproject.

• Sometimes,not�all�of�the�reported�information�was�relevant�or�needed

fortheprogramme-levelsynthesis.Forexample,intheAreasofChange

template,ImplementingPartnersincludedinformationaboutactivitiesthey

haddonewithassociatednumbers(e.g.thenumberoftrainingscompleted).

However,itisacknowledgedthatthechangebeingdescribedwassoclosely

associatedwiththeactivitybeingdonetoachieveit,insomecases,itwas

difficulttodisassociateanddistinguishbetweenthetwo.IntheUse of

climate and weather informationtemplate,ImplementingPartnersoften

includedinformationonwhattheyexpectedtohappeninthecomingyear.

TheMonitoringandResultsReportingteamconsideredonlythatinformation

13 thisincludesPart1ofthereport,whichwasmostlyaimedattheFundManager.


relevantfortheprogramme-levelsynthesis.However,itmaybeusefulto

referbacktothiskindofdatawhenproducingnextyear’sversionofthe

synthesis,asthiswillenableustoseethelevelofactivity/changeover

thecourseofayear.

Thesefactorsmadeitmoredifficulttounderstandprojectstories/pathways

andthendrawcross-project,programme-levellessons.Itwillcontinuetobe

achallengetoachievethebalancebetweenImplementingPartnersproviding

enoughoftherightinformationtounderstandtheprojectanditschanges,and

toomuchdetail.TheMonitoringandResultsReportingteamwillworkalongside

ImplementingPartnerstofurtherimprovethisfornextyear’sreporting.

Thereportingtemplatesandprocessalsoseemedtoaffectthedataprovided

toacertainextent:

• TheAreasofChange,3AsandEvaluativeMonitoringtemplateseach

includedasectionforreportingprogressandanotherforproviding

reflectionandanalysis(e.g.‘Towhatextentarethesechangeprocesses

contributingtotheprojectoutcome?’)These‘bigger’,more�reflective�

questions�received�limited�answers�from�Implementing�Partners.Most

projectshaveaknowledge,learningandresearchcomponentthatcould

supportthesereflections;however,thesedidnotcomeout.Themost

effectivetemplateswerethosethatwalkedImplementingPartnersthrough

aprocessofanalysisandreflection,suchas‘Useofclimateandweather

information’.ApplyingthisapproachtotheEvaluativeMonitoringtemplate,

inparticular,wouldbehelpfulfor(a)reducingrepetitionofcontextual

factorsatdifferentlevelsandtimes,and(b)improvingspecificityaboutif

andhowthesehaveenabledorconstrainedchange.Itmayhavealsoaided

furtherreflectionbyImplementingPartnerswhenconsideringthecontinued

validityoftheirprojecttheoriesofchange,thoughprematuretimingin

relationtomid-termreviewslimitedinputshere.

• Duringthetemplatedesign,mucheffortwentintounpackingthe

BRACEDtheoryofchangeandunderstandingthechangesbeingbrought

aboutbyprojectsfromdifferentperspectives.Attimes,‘zooming�in’�to�

capacity,�decision-making�and�the�3As,�means�the�story�of�the�project�can�

be�lost(e.g.whyImplementingPartnersareengagingincertainactivitiesand

howthesearecontributingtobuildingresilience).PerhapstheKnowledge

Managerwasaskingtoomuchofprojectsinthe‘likelihoodoftransformation’

templateasclaimsheretendedtobeoverlyambitious.Inaddition,the3As

templateisnotcurrentlydesignedtothinkaboutsequencing:ifaparticular

activityhasmorefocusinyear1comparedwithyears1or3,oriftheproject

isfocusingonbuildingonecapacitymorethantheothers.

• The�factthe�report�to�the�Knowledge�Manager�was�combined�with�

reporting�to�the�Fund�Manager�(for�accountability)�may�have�affected�

what�information�Implementing�Partners�provided�for�the�M&E�

framework�sections(forlearning)andhowtheyframedandpresented

thisinformation.Reportstendedtofocusondeliveryratherthanlearning

andchangepathways,andthecontextwasconsideredintermsofrisksto


activitiesratherthanasanenablerorconstrainerofchange.Thereporting

wascombinedattherequestofImplementingPartnersatthestartofthe

programme.Whileresultsmayhavebeenaffected,thecombinednature

ofthereportmayhavealsosupportedImplementingPartnersbuyinginto

reportingagainsttheM&Eframework.

Overall,fromaKnowledgeManagerperspective,thefirstyear’sreportingprocess

wentwell.ImplementingPartnerreportswereparticularlyimpressive,giventhat

(a)project-levelM&Esystemswerenotsetuptoreportconsistentlyagainstthe

BRACEDM&Eframework,(b)thereportingwasearly,comparedtowhenresults

couldrealisticallybeexpected,and(c)ImplementingPartnerswerecompleting

thetemplatesforthefirsttime.

ItwasusefulfortheMonitoringandResultsReportingteamtobeableto

commentonthefirstdraftofeachprojectreport,highlightgapsandreceive

updatedreportsintrackedchangeswithanexplanation/responsefromthe

ImplementingPartneragainsteachgeneralpointoffeedback.Itseemedto

beconfusingforsomeImplementingPartnersthatthisKnowledgeManager

feedbackwasprovidedthroughtheFundManager.Thiswasdonebecause

thoseweretheofficialreportinglines.However,someImplementingPartners

atthatpointthoughtthatthefeedbackwascomingfromtheFundManager.

Adifferentprocesscouldbeconsiderednexttime.

Asetofreportswithboththesameconceptualisationandoptimumlevel

ofdetailandrelevanceofinformationwillgreatlyfacilitatetheprocessfor

movingfromproject-levelreportstotheprogramme-levelsynthesisreport.

TheMonitoringandResultsReportingteamwillworkwithImplementing

Partnerstoconsiderhowtoimproveboththetemplatesandreporting

process,tomaketheexercisebothasusefulandlightaspossibleforboth

theImplementingPartnersandtheKnowledgeManager.TheMonitoring

andResultsReportingsynthesisprocesscouldpossiblybefurtherrefined

bycombiningthetaskofsummarisingprojectreportsandsynthesisingthem.

TheMonitoringandResultsReportingteamwillconsiderhowtoachieve

aprogramme-levelviewthatisbothcomprehensiveandyetspecificby

referringbacktoindividualprojects.

Feedback�given�to�Implementing�Partners�on�their�initial�reporting

Therewassome�confusion�over�the�term�‘baseline’�in�the�Areas�of�Change�

template.ThisreferredtothechangestheImplementingPartnerwould

‘expect’,‘like’and‘love’toseeduringthelifetimeoftheproject,ratherthan

thesituationatthetimeofthebaselinestudy.Consequently,forasmallnumber

ofImplementingPartners,thedata‘baseline’informationreceivedwasa

mixtureofwhatthesituationactuallywasatthestartoftheprojectandwhat

theImplementingPartnercould‘expect’,’like’or‘love’toseeatitsinception.

Whiletheformerwasinterestingandusefulinformation,particularlyinterms

ofunderstandingwhathadbeenachievedwithintheprojectlifetime,itwasthe

latterthatwasbeingsought.


Insomeinstances,ImplementingPartnersreportedchangesatthe‘liketosee’

or‘lovetosee’levelswithoutdetailingtheearlierchangesintheprocess(i.e.the

‘expecttosee’aspects).Itisnotyetclearifthisisanissuethatchallengesthe

AreasofChangeasmulti-stepprocesses.ImplementingPartnersalsotendedto

reportchangesbeyond‘expecttosee’,eventhoughtheiranalysisstatedthat

mostchangesseensofarwereatthe‘expecttosee’level.Uponrequestfor

clarification,theyexplainedthatchangesseenbeyondthe‘expecttosee’

levelweremoretentative.

TherewasalsosomeoverlapinprojectreportingonAreaofChangechallenges

andEvaluativeMonitoringconstrainingfactors.Reportingprocessesofchange

inisolationfromanunderstandingofthecontextwithinwhichprojectsoperate

isnotuseful.Furtherthinkingisrequiredonhowbesttointegratethesetwo

reportingtemplates.Underpinningthegapsandchallengesisthedifficultyin

translatingkeyconceptsinprogramme-leveltheoryofchangeintomonitoring

templatesattheprojectlevel.

ItisalsothecasethatanumberofImplementingPartnersmissedthe

opportunitytoprovidetheoverallproblembeingaddressedforeachAreaof

Change.Itisthoughtthiswasduetoalackofclarityonwhetherthisinformation

wasrequired;ImplementingPartnerswereabletoprovideitonrequest.

Underthe3Astemplate,alack�of�quantitative�‘KPI�4’�data�on�how�many�

people’s�resilience�had�been�built�in�the�first�year�of�the�projectheightened

theneedforanexplanationofhow/whytheprojectindicatorswouldcontribute

tobeneficiaries’capacitytoanticipate,absorbandadapttoshocksandstresses.

SomeImplementingPartnersleftthisblankinthefirstversionoftheirreports.

However,mostwereabletofillthisonrequest.Ofallthetemplates,the3Aswas

theonemostlikelytobeadaptedorcompletedincorrectly.Itseemsthereisa

needtofurtherexplainsub-indicatorsandtheexactinformationbeingsought.

Theyear1reportsrevealedthatImplementing�Partners�are�either�not�

taking�a�continual�‘Evaluative�Monitoring’�approach�to�understanding�and�

responding�to�the�changing�context�of�their�projects,�or�not�reporting�on�it.

AllImplementingPartnerswereabletodetailthecontextualfactorsaffectingthe

projectatlocal,sub-nationalandnationallevels,bothatthestartoftheproject

andtheendofyear1.Most,however,struggled�to�provide�in-depth�analysis�

and�specific�examples�showing�how�these�factors�were�constraining�and/or�

enabling�change�in�the�project(particularlythelatter).Theywerealsounclear

onwhetherthiswasexpectedornot.Inafewcases,shocksandstresseswere

missingfromtheEvaluativeMonitoringreportingonhowtheprojectcontext

hadchangedduringtheyear.

Anotherissuewheredatawaslimitedwasinrelationtohowtheuse

ofclimateinformationintheBRACEDprojectimprovestheresilienceof

beneficiaries.OneImplementingPartnerreflectedthat,‘there is not a lot to

mention about how these processes were linking or contributing to beneficiary

resilience’.Sometimes,itwasdifficult�to�distinguish�between�changes�or�

results�that�had�actually�happened�(encompassing�whether�these�were�

during�the�lifetime�of�the�project�or�before�it�began)�and�those�that�were�


anticipated�on�the�basis�of�year�1�progress.ThiswasclarifiedbyImplementing

Partnersintheirsecondversionofthereport.

Finally,thelasttemplateofthereport–whichwasfocusedonthecontinued

validityoftheprojecttheoryofchangebasedonlearningfromreportingagainst

theM&Eframework–wasonlylightlyfilledinbymostImplementingPartners.

Themainreasongivenforthiswasthatitwastoo�early�in�the�project�cycle�to�

review�the�theory�of�changeandthatchangestoitwouldbeconsideredafter

theprojectmid-termreviewhadbeencompleted(mid-waythroughyear2).


Annex�5:�BRACED�M&E�‘Infrastructure’

BRACED Fund

Manager

FM Results Team

Quarterly & Monthly

Reporting

Direct engagement

with IPs

Annual Reporting Synthesis

Evaluation

Quarterly Performance

Reporting

Monitoring visit reports

BRACED Knowledge Manager

Project to programme evidence &

learning

Monitoring & Results Reporting

(MRR)

Consistent project results

reporting (Outcome level)

Evaluative Monitoring

(context analysis)

Areas of Change (Outcome Mapping)

3As – Resilience outcomes

Contribution Analysis

(Country Case Studies)

Realist Evaluation

Case based analysis

Quasi-Experimental

Impact Evaluation

Contribution Analysis

EA1: BRACED Programme

ToC

EA2: BRACED interventions

EA5: PHASE

EA3: BRACED Projects

EA4: Adaptive Social Protection

(System level)

Activity Method

How is BRACED performing?

How are BRACED projects building resilience?

How effectively are activities being delivered?

What results has BRACED delivered?

Does the BRACED model work? For whom?

What does this mean for future resilience programming?

What does this mean for resilience strengthening more broadly?

What have we learned about monitoring and measurement of resilience programming?

* EA: Evaluation Activity* ToC: Theory of Change


Annex�6:�Rolling�out�the�BRACED�M&E�frameworkTheBRACEDM&Eframeworkwasdevelopedandrolledoutasfollows:

• Theprogramme-levelMonitoringandResultsReportingteamcombined

abottom-upandtop-downapproachtodeveloptheBRACEDprogramme

theoryofchange.Thisincludedareviewofall15project-leveltheories

ofchangeandlogframesandconsultationwiththeprojectImplementing

Partners.Itsetoutkeyelementsoftheprogramme-levelM&Eframeworkto

whichallprojectswouldcontributebothresultsanddata.Attheprogramme

level,frameworksweredevelopedtoenablestandardisation(totheextent

possible)ofconcepts,analysisandreportingagainsttheprogramme-level

theoryofchange.

• Theprogramme-levelMonitoringandResultsReportingteamthenprovided

arangeofsupporttoprojectImplementingPartners:

• DetailedwrittenguidancewassharedwithallImplementingPartnerson

theM&Eframeworkandhowtooperationaliseit(intheBRACEDM&E

GuidanceNotes).

• One-to-oneconversationswereheldwitheachImplementingPartner

toansweranyquestionsinrelationtotheframeworkandhowtoapply

ittoproject-levelM&E.

• Writtenfeedbackwasprovidedonupdatedversionsofproject-level

M&Edocumentstocheckalignmenttotheprogrammelevel.

• Groupsupportwasprovided,bothface-to-faceandremotelyonagroup

basis,totrainImplementingPartnersinthereportingtemplatesand

showthemhowtoreportagainsttheM&Eframework.

DuringthefirstyearofBRACEDimplementation,theKnowledgeManager

MonitoringandResultsReportingteamhasworkedalongsideImplementing

PartnersandtheFundManagerto(a)ensurethealignmentofproject-levelM&E

withtheprogramme-levelM&Eframework,and(b)enhancetheoverallquality

andcomparabilityofBRACEDM&E.

Afterface-to-faceconsultationwithallImplementingPartnersduringtheBRACED

inceptionworkshopinSenegal,February2015,focusingonthedraftframework,

theKnowledgeManagerdevelopedwrittenguidanceforallImplementingPartners

tooperationalisetheframework.FollowingthesharingoftheBRACEDM&E

GuidanceNotes(March2015),theKnowledgeManagerprovidedaprogrammeof

1-2-1supporttoImplementingPartnerstohelpthemapplytheframeworkintheir

finalisationofproject-leveltheoriesofchange,logframesandM&Eplans.14

14 thisincludeda1-2-1SkypecallwitheachImplementingPartnertodiscusstheimplicationsofdatacollectionagainsttheM&Eframeworkatproject-level(April–May2015)andawrittenreviewofupdatedproject-levelM&Edocuments(June–July2015),beforetheyweresignedoffbytheFundManager.






TheMonitoringandResultsReportingteamthendevelopedthesetof

reportingtemplatestogatherdataagainsteachoftheM&Eframework

components.This,inturn,enabledallImplementingPartnerstoundertake

standardisedandcomparablereportingofyear1projectresults.TheKnowledge

Manageralsodevelopedtemplatesfortwoaspectsofprogrammelogframe

reporting.Thesewerecentredonthe‘useofclimateandweatherinformation’

andthe‘likelihoodoftransformation’.Draftversionsofthesetemplateswere

sharedwithImplementingPartnersfortheirfeedbackattheBRACEDAnnual

LearningEventinSenegal,February2016andatawebinarinMarch2016.

TheKnowledgeManagerthenworkedcollaborativelywiththeFundManager

tobothfinalisetheoverallannualreporttemplate(March2016)andtrain

ImplementingPartnersinitscompletion(May2016).15

ImplementingPartnerssubmittedtheiryear1reportsattheendofMay2016.

RevisedversionswerethenproducedduringJune2016.Theserespondedto

KnowledgeManagerfeedbackinordertoenhancethecompletenessandutility

ofthedataprovided.Allofthe14ImplementingPartnersthatsubmittedreports

completedalltemplates.

15 Atwo-dayface-to-faceparticipatorytrainingeventwasheldwithrepresentativesofallBrAcEdprojects,organisedjointlybytheFundManagerandKnowledgeManager.ImplementingPartnershadtheopportunitytoshareexamplesoftheirdraftreportswiththegroupforfeedbackandcontinuedraftingthem,whileaskingquestionsandreceivingreal-timefeedbackfromtheKnowledgeManagerandFundManager.

BRACED aims to build the resilience of up to 5 million vulnerable people against

climate extremes and disasters. It does so through a three year, UK Government

funded programme, which supports over 120 organisations, working in 15

consortiums, across 13 countries in East Africa, the Sahel and Southeast Asia.

Uniquely, BRACED also has a Knowledge Manager consortium.

The Knowledge Manager consortium is led by the Overseas Development Institute

and includes the Red Cross Red Crescent Climate Centre, the Asian Disaster

Preparedness Centre, ENDA Energie, ITAD and Thomson Reuters Foundation.

The views presented in this paper are those of the author(s) and do not necessarily represent

the views of BRACED, its partners or donor.

Readers are encouraged to reproduce material from BRACED Knowledge Manager Reports for

their own publications, as long as they are not being sold commercially. As copyright holder, the

BRACED programme requests due acknowledgement and a copy of the publication. For online

use, we ask readers to link to the original resource on the BRACED website.

The BRACED Knowledge Manager generates evidence and learning on

resilience and adaptation in partnership with the BRACED projects and

the wider resilience community. It gathers robust evidence of what works

to strengthen resilience to climate extremes and disasters, and initiates

and supports processes to ensure that evidence is put into use in policy

and programmes. The Knowledge Manager also fosters partnerships to

amplify the impact of new evidence and learning, in order to significantly

improve levels of resilience in poor and vulnerable countries and

communities around the world.

Published December 2016

Website: www.braced.org Twitter: @bebraced Facebook: www.facebook.com/bracedforclimatechange

Cover image: Mikkel Ostergaard

Designed and typeset by Soapbox, www.soapbox.co.uk

routes to resilience...routes to resilience: lessons from monitoring braced 2 reflections in brief...

Documents