2011-12 dec accountability framework assessment report

The 2011/12 DECAccountabilityFramework Assessmentfindings from the externalevaluation and peer reviewprocess

July 2012

Alice Obrecht, Christina Laybourn,Michael Hammer & Stéphanie Ray

Telephone: +44(0)20 7713 6790Email: [email protected]: www.oneworldtrust.orgCharity Commission No 1134438

About the One WorldTrustThe One World Trust is an independent charitableresearch organisation that conducts research,develops recommendations and advocates forreform to make policy and decision-makingprocesses in global governance more accountableto the people they affect now and in the future,and to ensure that international laws arestrengthened and applied equally to all.About the EvaluatorsThe 2011/12 DECAF assessment was conductedby Christina Laybourn and Alice Obrecht, bothSenior Researchers at the One World Trust, withsupport from the Executive Director MichaelHammer and Research Assistant Stéphanie Ray.All members of the consultancy team, underAlice’s lead, contributed to the final report, forwhich The One World Trust takes responsibility asthe consultant.The One World Trust would like tothank the staff from all Member Agencies whocontributed to the validation process for theirparticipation, in particular the Leads tasked withorganising their Agency’s submission. The DECSecretariat staff, through their openness toengagement and challenge, contributed invaluablyto their own accountability assessment as well asto the Member Agencies’ peer review workshops.The consultants are particularly grateful to CaitTurvey Roe for her active involvement andguidance throughout this process.

© One WorldTrust 2012. This work is licensed under a Creative CommonsAttribution-NonCommercial-NoDerivs 2.0 UK: England & Wales License.

One WorldTrustSuite 301, 3rd Floor109-111 Farringdon RoadLondon EC1R 3BW

ISSN Series 2043-7943

DEC AccountabilityFramework: findings fromthe external evaluationand peer review

Alice Obrecht, Christina Laybourn,Michael Hammer & Stéphanie Ray

One WorldTrust 2012

Way of Working 1.4: Incidences of actual/suspectedfraud or loss are investigated and actioned.

Way of Working 3.3: Disaster affected populationsparticipate in programme assessment, design,implementation and evaluation.

Way of Working 3.4: In consultation with disaster affectedpopulations appropriately designed and describedcomplaints handling procedures are implemented.

Contents

Introduction.

Findings & Conclusion.

Way of Working 1.3: Programme design andprocurement processes maximise value formoney—balancing quality, cost and timeliness ateach phase of the response.

4

7

10

12

15

18

20

Way of Working 2.6: Programmes contribute to disasterrisk reduction and build the resilience of communities,partners and governments.

Introduction Way ofWorking 1.3 Way ofWorking1.4Way ofWorking2.6

Findings &ConclusionWay ofWorking3.3Way ofWorking3.4

Appendix 21

DEC Accountability Framework One World Trust

IntroductionEach year, members of the British public placetheir trust in the Disasters EmergencyCommittee to transform their donations into life-saving assistance for people affected by disaster.The Member Agencies of the DEC honour thistrust by committing themselves to processes thatsupport and challenge them to meet the higheststandards in humanitarian accountability andperformance. One of these important processesis the DEC Members’ annual self-assessmentsagainst their shared Accountability Framework.This Report summarises the findings from the2011/12 DEC Accountability Framework(DECAF) assessment process.The purpose of theDECAF assessment is to review the policies,practices and assurance systems through whichMembers meet best practice in their humanitarianwork using funds raised through the DEC. Inorder to lend focus to the assessment, Membersare asked to provide supporting evidence fortheir self-assessments from two countries inwhich they are currently using DEC funds toassist disaster-affected people. For the 2011/12assessment, these countries were selected fromthe Pakistan Floods Appeal, for which a total of£71 million was raised from the British public, andthe East Africa Crisis Appeal, which reached atotal donation level of £79 million and has beenused to assist more than 1.3 million people.Through the DECAF assessment process,Member Agencies hold one another to accountfor mutually-recognised standards, providing aunique forum for peer challenge and horizontalaccountability. They are also held to account byexternal evaluators, who review the evidence-based self-assessments submitted by MemberAgencies, thereby supporting the externallegitimacy of the Accountability Framework.The Accountability Framework Assessmenttargets the underlying systems and policies thatare needed to support Members in their delivery

of effective and responsible aid. It does not assessthe quality of the DEC Members’ humanitarianresponse.As such, this Report is a complement tothe system of oversight of Members’ plans andprogramme reports as well as the Real-TimeEvaluations (RTEs) and other external evaluationsthat accompany each DEC appeal.Raising the bar: the DEC AccountabilityFramework (DECAF) IIThe 2011/12 DECAF process has been marked bya period of transition for the DisastersEmergency Committee.The DEC welcomed PlanUK as a new Member Agency this year, bringingits membership to 14 Agencies. DEC Membersalso revised significantly their best practicestandards, or, Ways of Working in humanitarianservice (Ways of Working). The new framework,DECAF II, was adopted in April 2011 andimplemented for the first time in this year’sDECAF assessment process. The framework is adynamic model based on four accountabilitypriorities that are considered most likely to driveimprovement:1) We use our resources efficiently and effectively.2) We achieve intended programme objectives inaccordance with agreed humanitarian standards,principles and behaviours.3) We are accountable to disaster affectedpopulations.4) We learn from our experience—taking learningfrom one emergency to the next.These are enumerated in 21 specific Ways ofWorking which reflect best practice standards(see Appendix 1).The revised Ways of Working inDECAF-II offer a substantial step change from theprevious framework and, in several key areas,

Introduction

Way ofWorking 1.3

4

1

1. Evaluations funded by the DEC are independent and aremade publically available through their Appeals page:http://www.dec.org.uk/appeals2. DEC Accountability Framework,http://www.dec.org.uk/node/1951/

1

2

Way ofWorking1.4Way ofWorking2.6


Findings &Conclusion

DEC Accountability Framework One World Trust5

Introduction

Way ofWorking 1.3

raise the bar for standards of accountability andperformance across the humanitarian sector. Forthe 2011/1 assessment, the Pakistan and EastAfrica Appeals presented unique opportunitiesand challenges for meeting the strongaccountability and performance standards set byDECAF-II. Members’ commitments to maintainingbeneficiary participation in all cycles ofprogramming were strong in countries covered bythe East Africa Appeal, while assurance processesfor value for money and detecting and respondingto suspected fraud were well-implemented inPakistan.The Assessment ProcessOn an annual basis, each Member Agency assessesitself against each Way of Working. Thisassessment targets three broad areas ofperformance:1) Policy or procedure2) Application of policy in a particularhumanitarian context (practice)3) The assurance systems that Members use toensure the quality and consistency of their workacross responses.As external evaluators, the One World Trust teamasked Agencies to submit evidence for policy,practice and assurance in support of their self-assessments for five selected Ways of Working:1.3 Programme design and procurementprocesses maximise value for money - balancingquality, cost and timeliness at each phase of theresponse1.4 Incidences of actual/suspected fraud or lossare investigated and actioned2.6 Programmes contribute to disaster riskreduction and build the resilience of communities,partners and governments3.3 Disaster affected populations participate inprogramme assessment, design, implementationand evaluation3.4 In consultation with disaster affected

populations appropriately designed and describedcomplaints handling procedures are implementedFocusing on these five standards offered a viewinto how Agencies are engaging with timely issuesin their donor environment and making progresson long-standing commitments, such asaccountability to beneficiaries, in new andchallenging contexts.After reviewing the submitted evidence, the OneWorld Trust facilitated a series of Peer Reviewworkshops, in which Members summarised theirself-assessments and received challenges andquestions from their peers, the One World Trustteam and representatives from the DECSecretariat. At the end of each workshop, eachMember’s self-assessment was discussed andconfirmed by Peers and the One World Trustteam. In some cases, a Member Agency wasrequested to submit further supporting evidencefor an area of performance (policy, practiceand/or assurance).Findings & ConclusionBroadly speaking, three performance perspectiveson each Way of Working emerged throughout theassessment process: a conventional perspectivethat represents where most Agencies are at withrespect to a Way of Working and commonthemes or strategies; a perspective of innovationthat reflects the novel interpretations of a bestpractice and the new frontiers being explored bysome Members; and a perspective ofimprovement that embodies the challengesAgencies faced in meeting a Way of Working andthe strategies some Members are using to catchup with their peers or move to the front.Theseperspectives are represented in a diagram at thebeginning of each Way of Working summary andare discussed in greater detail in the text of eachsection.The One World Trust team found that, overall,performance across the five Ways of Working





Introduction

Way ofWorking 1.3

examined for the assessment met many bestpractice standards, with strong policies andfrequent examples of humanitarian best practicebeing met in Pakistan and East Africa. However,performance on assurance was mixed across thefive Ways of Working: there were some examplesof strong quality assurance mechanisms but theprocess of strengthening or developing ways toassure quality and monitor the consistency oftheir performance continues. Innovation and theactive development of new systems to enhanceaccountability and performance are high across all14 Members.





Way ofWorking 1.3

7

Introduction Way ofWorking1.4Way ofWorking2.6



Conventional perspective

• Has position paper outlining organisational definition of

value for money

• Has strong assurance mechanisms for assessing costs,

evaluating quality, and monitoring timeliness, however

there is no overarching mechanism that unifies all three

into a single decision-making or review process

• However: confident that procurement and programme

design on the ground are being guided by a holistic

balancing of costs, benefits and value added

Perspective of innovation

• Embracing the opportunity to engage in and lead the

discussion on how value for money should be

conceptualised for humanitarian aid

• Specific attention paid to value for money considerations

in programme design, not just procurement

• Actively pursuing a holistic integration of VfM across all

aspects of organisation

“Evidence should demonstrate that we have a clear organisational position on

what ‘value for money’ means and how this should be assessed. As a minimum, quality,

cost and timeliness should be analysed together—rather than in complete isolation to

each other—during the planning, implementation and evaluation phases.” DEC

Accountability Framework Assessment Guidelines

Way of Working 1.3: Programme design and procurement

processes maximise value for money—balancing quality, cost

and timeliness at each phase of the response

Continuous

Impro

vem

ent

Perspective of improvement

• Organisational documents feature value for money as a

term, but do not define it in detail.

• Working groups established to develop an organisation-

wide approach

• Value for money will be an active area for improvement

in 2012/13, with plans for developing tools, indicators,

and/or rolling out new performance systems


Background to this Way of WorkingValue for money has become a dominant conceptin the UK donor and aid environment in recentyears. The case for value for money as a bestpractice standard is relatively straight-forward:Given the limited amount of resources availableto them, aid organisations ought to accomplishthe most they can with these resources, bymaximising the value achieved by every unit ofcost. In practice, however, value for moneystandards raise important and complex questions,such as how to assess the value and financial costof intangible or longer-term outcomes, or how tocompare value for money across organisationsthat use different approaches to assessing qualityand to balancing quality against cost andtimeliness. The evidence submitted by the DECMembers reflected a shared commitment toachieving the very best with their limitedresources, showcasing a diverse set of approachesto how organisations are conceptualising andmeasuring value for money in their work.The Conventional Perspective:The Peer Review workshops addressed thequestion of whether a Member Agency needed tohave a defined approach to value for moneyoutlined within the organisation in order to beconsidered to meet minimum standards of bestpractice. Members felt that they all should have aclear position paper or approach to value formoney and many submitted papers that outlinedtheir approach to value for money in programmedesign.There is evidence that Agencies are usingconsiderations of cost, timeliness and quality toguide programming.At present, however, evidenceof value for money in practice focuses on theprocurement of goods and services: evidenceprimarily consisted of assessments of bids ordocumentation of vender selection,demonstrating how these procedures areregularly guided by a unified decision-makingprocess that looks at all three dimensions of cost,

timeliness and quality of product.The Perspective of Innovation:Measuring and demonstrating value for moneypresents a number of challenges, which Membersare overcoming through innovation. Value formoney is sometimes viewed as a top-downaccountability priority in which aid organisationsmust direct field staff to make decisions that canbe justified as cost-effective and efficient todonors.This challenges Members to find ways tobring intended beneficiary groups into theconversation on value for money. ActionAid ismeeting this challenge by re-framing value formoney as a participatory concept, askingcommunity members to assess the value of anintervention or project. Christian Aid has alsoused a bottom-up approach to defining the valueof its outcomes, for example, by askingbeneficiaries to assess the performance andquality of contractors.Some Members reported that their organisationswere looking at mainstreaming value for moneyconsiderations across multiple departments.Oxfam, for example, is implementing a Value forMoney Organisational Programme, with projectsacross multiple organisational functions includingMonitoring, Evaluation, Accountability andLearning (MEAL), fraud response, organisationalanalysis, and data sharing. CARE UK hasdeveloped a Value for Money Framework thatidentifies guiding principles for all parts of theorganisation, examining value for money throughdifferent ‘lenses’ based on organisational level.CARE’s Framework goes beyond program qualityand management to include broad guidelines forpartnership, advocacy and policy, and safety andsecurity.The Perspective of Improvement:For some Members, improvement will involvebringing together the different perspectives andbodies of data, currently housed in different ‘silos’within their organisation, that are necessary for

8

Way ofWorking 1.3

Introduction Way ofWorking1.4Way ofWorking2.6




forming sound value for money assessments.Other Members will focus on composing aposition paper or otherwise ensuring that there isa coherent understanding of what value formoney means throughout the organisation inorder to back up the current systems which arebelieved to deliver it.

Further QuestionsHow important is the ‘value for money’ terminology?:Some Agencies explained that their organisationhad made a conscious decision to refrain frommainstreaming the language of value for money, orusing this phrase with field staff, in order to avoidconfusion. ActionAid, for example, is using theterm ‘cost-effectiveness,’ while Merlin has definedvalue for money as “Delivering the rightinterventions, to the right people, at the righttime and at a cost appropriate to the context”and will use only this broader phrasing with theirfield staff.Is the lack of a shared definition of value for moneyan opportunity or a constraint?: Some Membersindicated that the absence of a fixed definition forvalue for money presents a useful opportunity toshape a crucial debate over what constitutes‘value’ in a humanitarian response. Agencies cancontribute to this debate by articulating the valueadded by non-governmental organisations or theimportance of intangible goods such asparticipation.The One World Trust team noted that thisopportunity, while valuable, also presents achallenge: if Agencies engage with value for moneyby developing their own organisational definitions,it may become more difficult to translate thedifferent approaches and benchmark progress onvalue for money across organisations.Does value for money call for a paradigm shift, orjust repackaging?: In the workshops, manyAgencies expressed the view that, rather thansetting out a new standard of practice, value formoney as an accountability standard merely

requires Agencies to re-package their existingprocesses of cost-analysis in a new way. Thisimplies that progress on this Way of Working istherefore largely a matter of translating on-goingpractices into a new language.At present, it remains to be seen whether re-packaging is all that is necessary for all Membersto meet value for money best practice, orwhether some Members may find it useful topursue a deeper set of changes. Given thediversity of organisational systems and strategiesacross the Members, there is unlikely to be a ‘onesize fits all’ answer to this question.Regardless of how well the Members areinternally placed to respond to value for moneyrequirements, it remains important to articulatehow strong internal systems inform choices andstrategies at a higher level within eachorganisation. The principles of value for moneyare intended to direct Agencies to consider howmuch it costs them to achieve certain impacts.Such considerations ideally support organisationsto make more informed choices on those impactsthey choose to pursue and the best method ofachieving them. Insofar as current practice acrossthe Agencies emphasises the achievement of“economy” in procurement processes overassessments of the cost-effectiveness of entireprogrammes and projects, Members could domore to articulate their ongoing commitment tovalue for money and how this commitment guidestheir programme and project design.

9

Way ofWorking 1.3

Introduction

3

3. Dfid’s Approach toValue for Money, p.4:http://www.dfid.gov.uk/Documents/publications1/DFID-approach-value-money.pdf





Introduction Way ofWorking 1.3

10

Way ofWorking1.4


Way ofWorking3.4Findings &Conclusion


• Strong policies and mechanisms on the ground for

reporting and investigating suspected fraud

• Strong and reliable assurance mechanisms for identifying

and responding quickly to suspected fraud or loss

pertaining to funds provided by UK office or by DEC


• Public reporting of fraud, the amount lost, and actions

taken by the organisation

• Extensive preventative work to proactively engage with

the potential for fraud, e.g. staff trainings, extensive risk

assessments of potential partners

“Evidence should demonstrate that we have a definition of fraud and loss and a

procedure which ensures action is taken promptly by responsible persons.” DEC


Way of Working 1.4: Incidences of actual/suspected fraud

or loss are investigated and actioned

Continuous

Impro

vem

ent


• Re-drafting organisational policy or re-training field staff

on new practices following the Anti-Bribery Act

• Working on relationships with international and other

national offices to improve lines of reporting on

suspected fraud and loss to UK office


Background to this Way of WorkingProtecting the resources entrusted to them is ahigh priority for all Members of the DEC.Managing the risk of fraud is particularlychallenging in the fluid context of humanitarianemergency response, and therefore requires well-established and routinized systems to assure thatfunds are not misallocated. All 14 Membersprovided extensive evidence of such systems inplace at their organisation and, where relevant, intheir broader international families.The Conventional PerspectiveGiven the potential impact that a single case offraud can have on an organisation’s programmes,public reputation and its ability to secure funding,it was unsurprising that all Agencies performedstrongly on this Way of Working. All Memberswere able to provide evidence of explicit Anti-Fraud and Anti-Corruption policies, as well as ofprocesses for providing systematic oversight ofthe implementation and enforcement of thesepolicies. These included internal audits, site visitsby individuals responsible for monitoring fraud,and risk registers.Assessing fraud risk and responding quickly andappropriately to fraud is a particularly importantaspect of Agencies’ relationships withimplementing partners.Agencies that work heavilyor entirely through partnership discussed therange of strategies they used to ensure controlsover the appropriate use of funds. For severalMembers, partner organisations are required tocommission an external audit, in most cases,employing a local auditor. Partner assessmentmechanisms also draw together analysis of thepartner’s financial stability and solvency,governance mechanisms, financial managementand project cycle management, as well as anoverall risk assessment.The Perspective of Innovation:Anti-fraud best practice is fairly standardised and

the DEC Members all performed highly on thisWay of Working.As a result, there were few areasof innovation or examples of novel techniquespertaining to anti-fraud policies and systems. Oneexample of innovative activity is Age UK’sdecision to revise its anti-fraud policy in light ofthe UK Anti-Bribery act, and its plan to roll out anew programme of staff training in 2012 based onthe revised policy. Another potential innovationdiscussed across the workshops was increasingpublic transparency around levels of fraud andloss.The Perspective of Improvement:As this Way of Working was already an area ofstrong performance for all Members, there werefew examples of Agencies striving to improvetheir policies or practices.The primary challengefor some Agencies lies in shaping the relationshipstheir UK office has with partners and othermembers of their International families to ensurereliable and complete reporting of suspectedfraud from the field. Several Members discussedhow they might seek to strengthen these lines ofreporting in order to receive further informationin a timelier manner.Further QuestionsShould the standard for assurance be higher for fraudand loss than for other aspects of performance?:Given the particular sensitivity of fraudinvestigations and the importance of theirsuccessful resolution, it may be the case thatassurance standards for detecting and respondingto fraud should be stronger than assurancestandards for other areas of humanitarian bestpractice. While indirect reporting relationshipsmay be sufficient to assure consistent goodpractice on other Ways of Working, such asbeneficiary participation, the nature of fraudmight indicate that reporting and assurance linesbetween DEC Members and other parties(members of its international family, partnerorganisations) should be more direct.

11

Introduction Way ofWorking 1.3Way ofWorking1.4





• Disaster risk reduction or resilience building is

mainstreamed throughout organisational operations, as

reflected in a strategy or policy.

• Considerations of disaster risk reduction are captured in

planning for the intermediate or later phases of a

response

• There are members of staff tasked specifically with

disaster risk reduction or resilience building.

• Current resilience or risk reduction practice sits more at

the planning and implementation stages of the

programme cycle


• Disaster risk reduction and resilience building is

measured on a regular basis in order to monitor

whether resilience is actually built, with

recommendations issued on the basis of assessment

• Disaster risk reduction work incorporates the

appropriate role of government, different members of

the community, and, where relevant, partners

• Considerations of disaster risk reduction are reflected

in the early stage of the response

“Evidence should demonstrate how we integrated DRR and resilience into our

work: ensuring that we ‘build back better’, build DRR capacity and target advocacy work

on this topic.” DEC Accountability Framework Assessment Guidelines

Way of Working 2.6: Programmes contribute to disaster

risk reduction and build the resilience of communities,

partners and governments

Continuous

Impro

vem

ent


• Disaster risk reduction is defined and done in some

measure, but not currently mainstreamed

• Moving to develop indicators and ways of measuring

disaster risk reduction

• Working primarily with communities

12

Introduction Way ofWorking 1.3 Way ofWorking1.4 Way ofWorking2.6




Age UK, Save the Children,WorldVision andPlan UK have adopted specialised approaches toDRR due to the unique beneficiary groups withwhom they work—children in the case of Savethe Children,WorldVision and Plan UK, andolder people in the case of Age UK.Age UK’s DRR work flows out of their centralcommitment to involving older people inprocesses of development and relief.Their DRR-related projects are therefore focused on theengagement of older people’s groups in disasterrisk planning for their communities.Plan has included children in its advocacy workon disaster risk reduction so as to ensure theirperspective is represented in the global agendaon DRR.WorldVision and Save the Childrencarry out DRR trainings targeted towardschildren, and encourage children’s input on thedevelopment of community disaster risk plans.

Background to this Way of WorkingDisaster risk reduction and resilience building forfuture disasters have become increasinglymainstreamed concepts throughout humanitariangood practice. While disaster risk reduction(DRR) is now a common principle of humanitarianwork, this Way of Working reflects the moredemanding nature of DECAF-II, which requiresMember Agencies to engage not only withcommunities in DRR work, but also with partnersand governments. It was agreed that the spirit ofthe Way of Working was best interpreted asexpecting Members to work with or in support ofpartners and governments in order to build theresilience of communities. The text boxes in thissub-section discuss in greater detail examples ofDRR work with specialised communities (Textbox 1), governments (Text box 2) and partners(Text box 3).The Conventional Perspective:Performance on this Way of Working wasconsistently strong, but with room for furthergrowth.All Members make a commitment to DRRor resilience building in a policy-level document.Many have identified DRR or resilience as a keystrategic objective or programmatic theme.The Perspective of Innovation:Agencies using innovative approaches to this Wayof Working submitted evidence demonstrating theongoing evaluation and measurement of progressagainst DRR indicators. CAFOD has conductedbaseline assessments on DRR and plans to roll outassurance mechanisms in 2012 that will contributeto an improved implementation of best standardspractice. Islamic Relief submitted an Evaluationtemplate measuring disaster risk reduction that itwill use to assess its Ethiopia response this year.Oxfam has a DRR Benchmark that is measuredregularly in its RTEs.Tearfund, which performed strongly against thisWay of Working, includes DRR as one of theircross-cutting Quality Standards for emergency

Text box 1:Working with specialised communities:Age UK, Plan, Save the Children, and WorldVision

13


response used in the monitoring and evaluation oftheir partners. They are currently using two setsof indicators (for fast onset and slow onsetdisasters) as the basis for assessing partnercapacity and identifying areas for capacity building.The Perspective of Improvement:Demonstrating success in disaster risk reductionand resilience building requires a reliable and well-structured assurance system that is capable ofreviewing organisational progress against long-term goals. All 14 Members have well-structuredassurance systems for monitoring long-termprogress, however, in some cases these systemsdid not provide Members with the confidence thatresilience building activities are beingsystematically implemented and monitored.Assurance mechanisms specific to resiliencebuilding and disaster risk reduction varied instrength across the 14 Members, and a range ofpotential improvements were set out. These

Way ofWorking1.4 Way ofWorking2.6




Concern WW UK is shifting its approach toDRR towards the adoption of a longer-termperspective, which has led them to engage morewith government authorities. For instance, inEthiopia, Concern WW is increasing its work onnutrition and strengthening their workingrelationship with the Ministry of Health. Merlinhas worked for years with governmentministries to strengthen healthcare systems.Merlin views resilience as integral to its work asa provider of healthcare and, in turn, treats itsrelationships to local government as integral toresilience. In Pakistan, Merlin is currentlyworking with the Ministry of Health toundertake extensive nutrition work, while alsotrying to identify and tackle the root causes offood insecurity.

The British Red Cross works with other RedCross/Crescent National Societies (within theRed Cross Movement) as its Partners ondisaster risk reduction. For example, in theirwork with the Kenya Red Cross Society, theBritish Red Cross has directed a significantamount of funding towards longer-termprogramming designed to reduce vulnerability.

Text Box 3:Working with partners:The British RedCross

Text Box 2:Working with government: Merlin &Concern

14


included building on existing monitoring andevaluation systems, which could soon provide aconsistent organisational check on resiliencebuilding projects, and hiring members of stafftasked specifically with implementing disaster riskreduction programming.Further QuestionsWhere does DRR belong?: Throughout theworkshops, it was questioned whether anAgency’s approach to disaster risk reduction wasviewed within its organisation as part of its laterphase/development work, or as an integrated partof its humanitarian response work. While mostAgencies’ DRR and resilience staff worked underthe umbrella of humanitarian response, for some,these functions fell under the remit of theirdevelopment teams. For many Members, linkingtheir development and humanitarian effortsthrough a common theme of resilience buildingremains an important area of growth.Are we clear on our concepts?: Many Agencies raisedthe point that the terminology of resilience and

disaster risk reduction is often exchanged withthe term “disaster preparedness”, despite thesethree phrases referring to distinct approaches andtypes of activity.This indicates the importance ofmaintaining a shared understanding of resiliencebuilding, to ensure discussion of similarapproaches and activities, which could otherwisehinder meaningful comparison for this Way ofWorking.What did we learn from the Horn of Africa crisis?:Several Members submitted reports andassessments on the Horn of Africa crisis, such as ADangerous Delay (Save the Children, Oxfam, jointbriefing paper), which emphasised the importanceof early action to prevent major food crises . Theparticular complications of aiming to buildresilience when working in refugee camps alsosuggested that the very conditions of humanitarianwork in certain contexts may be in tension withlong-term resilience building.





Introduction Way ofWorking 1.3 Way ofWorking1.4Way ofWorking2.6 Way ofWorking3.3



• Strong policy-level commitment to beneficiary

participation, typically included under an Accountability

Framework or tool, sometimes in Monitoring, Evaluation,

Accountability & Learning (MEAL) guidelines

• Difficulties in executing participation in practice, with

evidence of application tipped strongly towards

implementation and needs assessments than towards

deeper participation in design and evaluation

• Weak assurance mechanisms: discovery of performance

level on participation comes about largely through

DECAF process than through ongoing organisational

monitoring


• Use of creative and novel approaches and tools to

engage beneficiaries in deep forms of participation,

including in the design stage of programming

• Systematic monitoring and evaluation processes assess

the organisation’s performance on beneficiary

participation

“Evidence should set out our understanding of what participation means (e.g.

minimum level of consultation, engagement of women and men, adults and children,

older people etc.) and our approach to ensuring this is maintained through all phases of

the response..” DEC Accountability Framework Assessment Guidelines

Way of Working 3.3: Disaster affected populations

participate in programme assessment, design,

implementation and evaluation

Continuous

Impro

vem

ent


• Evidence of application difficult to find for the DECAF

submission, indicating a significant need to improve

documentation of participation occurring

• Depth of participation is often low and rarely goes

beyond needs assessments interviews


Background to this Way of WorkingAccountability to beneficiaries is traditionally seenas a challenge for humanitarian organisationsbecause they work primarily with survivors ofdisaster. This means their relationships withintended beneficiaries are sometimes transientand are often shaped by the serious trauma thathas been experienced by the survivors. DECAF-II’s Way of Working 3.3 expresses a significantcommitment to beneficiary accountability, movingbeyond soliciting and responding to beneficiaryfeedback to a vision of participation that involvesdisaster affected populations in all stages ofprogramming, from design to evaluation. Thismirrors the 2010 HAP standards for beneficiaryaccountability and is evidence of how DECAF-II’smore demanding Ways of Working place it at thefront of best practice standards in thehumanitarian sector.The Conventional Perspective:Evidence at the policy level for this Way ofWorking was strong, reflecting the status ofbeneficiary accountability—and participation as acore component of this—as an entrenchedstandard in humanitarian work. Many Agenciesoutline their approach to accountability tobeneficiaries, including participation, in anAccountability Framework. Agencies that do nothave a distinct Accountability Frameworkprovided documents from their MonitoringEvaluation Accountability and Learning (MEAL)systems that included sections on involvingbeneficiaries throughout the programming cycle.Participation is but one component ofaccountability to beneficiaries, and, in the contextof emergency humanitarian response, it isarguably one of the more challenging criteria tomeet. While there were many good examples ofinvolving beneficiaries in the implementation ofprojects, the more difficult forms of

participation—in design and evaluation—wereharder to evidence.Evaluation, both in terms of using participation inassessments of projects, and in terms of assessingthe use of participatory approaches, is a significantarea of growth for Members, with many seekingto improve how they evaluate progress againsttheir Accountability Frameworks.The Perspective of Innovation:Of all the components of beneficiaryaccountability, participation in particularnecessarily involves building a relationshipbetween an Agency and the communities whom itaims to serve. As a result, this standard is moreamenable than other Ways of Working toindividualised approaches across the MemberAgencies that reflect each Member’sorganisational identity. It is therefore a highlyfertile area for innovation.World Vision provided examples of application forall four components of this Way of Working. InPakistan, beneficiaries participated in programmedesign and in the identification of selectioncriteria for a livelihoods project, and childbeneficiaries worked with World Vision staff todevelop Children Hygiene Action Plans. WorldVision’s LEAP system requires participationthroughout all phases of programming and theirrapid assessments support this commitment byassessing participation from the first week ofresponse onwards.CAFOD is rolling out an innovative tool toimprove its accountability to beneficiaries, calledthe Vulnerability and Inequality Analysis (VIA)toolkit.The VIA toolkit was developed as part ofCAFOD’s work in HIV/AIDS and uses poweranalysis to understand a situation from theperspective of the beneficiary and monitor theproject through the lens of empowerment,measuring how the project changes the original

16


4

4. Humanitarian Accountability Partnership,www.hapinternational.org

Way ofWorking1.4Way ofWorking2.6 Way ofWorking3.3




power dynamics over time.VIA is driven primarilythrough participation, involving the beneficiary inorder to understand the initial context and toassess progress throughout the life of a project.CARE’s Humanitarian Accountability Framework(HAF) includes benchmarks on beneficiaryparticipation, with a set of correspondingindicators. CARE has been using RapidAccountability Reviews as an assurancemechanism to assess implementing partners’performance on involving beneficiaries inprogramming. Age UK is in a similar position,having recently rolled out its AccountabilityFramework and Evaluation Pack, which togethercomprehensively address beneficiary participationthroughout the programme cycle. Age UK hasundertaken baseline assessments on beneficiaryparticipation and plans to measure and monitorprogress against their framework on a quarterlybasis.ActionAid remains a leader in this field withrespect to fully integrating participation into itsprogramming with its Action LearningParticipation System (ALPS) and its ParticipatoryReview and Reflection Process (PRRP). ActionAidinvolves beneficiaries in programme design,budgeting, and evaluation, and feeds beneficiary-written reports into its shared organisationallearning system, which senior management isrequired to monitor regularly for their ownreporting. It was observed during the peerworkshop that, while ActionAid’s innovation isnow several years old, other Agencies werelargely unaware of it and interested to learn moreon how ActionAid’s processes work, in particulararound learning from beneficiary participation. Itmight be useful therefore for ActionAid, as well asother Members, to share information on theirapproaches at a future DEC cross-learning event.The Perspective of Improvement:As mentioned above, several Agencies found in

their self-assessment that they have strongpolicies on involving beneficiaries in programming,yet implementation of these policies varies acrossdifferent responses. According to some Members,one of the challenges in demonstratingperformance in this Way of Working is theabsence of documentation which evidencesparticipation, as opposed to the absence ofparticipation itself.Further QuestionsIs it reasonable to expect beneficiary participation inall phases of programming?: Despite this Way ofWorking being revised and agreed upon by allMember Agencies in the development of DECAF-II, there was still some debate about theexpectation of integrating beneficiaryparticipation throughout all phases ofprogramming, especially in design. Given the shorttime-frame that Members have to write a projectproposal in response to a rapid onset emergency,some felt that involving beneficiaries in the designphase may be unrealistic.

Way ofWorking1.4Way ofWorking2.6 Way ofWorking3.3







• At the policy level, commitment to implementing

complaints & response mechanisms is strong, and

policies indicate that mechanisms should be designed in

consultation with disaster affected populations

• Strength of complaints mechanism is influenced by

context of application: in some areas, complaints

mechanisms are designed in consultation with

communities and are well implemented; in other areas it

is taking more time to develop an adequate complaints

process


• Beneficiary groups are regularly consulted on the design

of complaints mechanisms

• Monitoring of community response mechanism

implementation is carried out regularly, feeding back to

the Agency a consistent and high-quality performance

“Evidence should demonstrate what our complaints handling procedures are, how

these are contextualised for individual responses, in consultation with communities, and how

we ensure all complaints are responded to. We should be clear about how the leaders in our

organisation set the tone so that we are open, and respond, to complaints.” DEC


Way of Working 3.4: In consultation with disaster affected

populations appropriately designed and described complaints

handling procedures are implemented

Continuous

Impro

vem

ent


• Seeking to strengthen assurance mechanisms to ensure a

more consistent implementation of appropriate

complaints mechanisms across countries of operation

• Introducing policy-level references to the consultation of

beneficiaries in the design of complaints mechanisms


Background to this Way of WorkingParticipation and Complaints & Response areboth components of an Agency’s approach to itsaccountability to beneficiaries, and thereforeevidence at the policy and assurance level for thisWay of Working shared many overlaps with Wayof Working 3.3. Providing those affected by aproject with the opportunity to complain is nowa common practice across humanitarian work.Inspired by a similar requirement in HAP, therevised DECAF-II requires that Agencies not onlyimplement reliable and well-functioningcomplaints handling mechanisms, but also designthese mechanisms in consultation with beneficiarygroups, so as to ensure they are context-sensitiveand thus useable.The Conventional Perspective:All Members have organisation-wide complaintspolicies and evidence of CRM established in atleast one of the countries examined for evidenceof practice. Many also make policy-levelcommitments to consulting with beneficiaries inthe design of their complaints and responsemechanisms. Assuring that these mechanisms areimplemented systematically across all countries ofoperation was a common challenge identified asan area for future improvement.The Perspective of Innovation:Agencies that are innovators in this Way ofWorking were those that not only had policydocuments specifying that beneficiaries consultwith staff on the design of CRMs, but alsoprovided strong evidence of this consultationoccurring in the field. For example, Save theChildren has a 10-step Guideline on how to setup a CRM that includes a step on involvingcommunities in the design of a mechanism. InKenya, this approach was implemented separatelywith child and adult beneficiary groups, resultingin the development of two separate complaintsmechanisms. Children preferred to use acomplaints desk set up in one of Save theChildren’s Child-Friendly Spaces in the Dadaab

camps, whereas the adult beneficiaries preferredto establish a representative group that wouldbring complaints to Save the Children on theirbehalves.The Perspective of Improvement:On the basis of this year’s assessment, severalMembers indicated they would seek tostrengthen beneficiary consultation in theirapproach to establishing CRMs. Many Membersindicated they are planning to revise theircomplaints policies and guidelines in 2012/13 toinclude beneficiary consultation in design.Further QuestionsIs consulting with beneficiaries necessary to ensurecomplaints mechanisms are appropriately designed?:There was some discussion as to whethercontext-sensitive and appropriate complaintsmechanisms could be designed withoutconsultation, in cases where field staff havedeveloped strong knowledge of the surroundingcommunity and can identify the optimal means forbeneficiaries to file a complaint. However, therequirement to go beyond appropriate design tospecifically include beneficiary consultation wasupheld. First, the revision of this Way of Workingwas inspired by a similar revision in HAP, andincluded the language of consultation specificallyto match the step change reflected in HAP’srevised standards. A second point cited by someMembers in support of this requirement was that,from their experience, requiring input frombeneficiaries as a regular procedure led toobservable improvements in the usability of theircomplaints mechanisms.While in some cases fieldstaff are able to design and implement a context-appropriate complaints mechanism withoutconsulting beneficiaries, the requirement ofconsultation better ensures the achievement of acontext-appropriate complaints and responsemechanism.

19





Findings & ConclusionThe One World Trust team evaluated the 14 DECMember Agencies’ performance against theirAccountability Framework-II by reviewing theevidence submitted in support of Members’ self-assessments on five Ways of Working. Afterdiscussion of the Ways of Working and submittedevidence in the Peer Review workshops, furtherevidence was required to confirm one third of theself-assessments. In a few cases, the original self-assessments of a Member Agency were revised inlight of the evidence in order to provide a moreaccurate reflection of their performance. At theend of the assessment process, the One WorldTrust team concluded that the final assessmentsvalidated for the Members on the five Ways ofWorking are a true reflection of theirperformance.Overall, policy-level performance against thedemanding DECAF-II standards was very strong,reflecting the ability of the DEC Members toeffectively convey evolving humanitarian bestpractice standards at policy level.The assessmentprocess also found evidence of innovativeapplication of policy, despite the challengespresented by working in the context ofhumanitarian crisis. Performance on assurancewas mixed across the five Ways of Working. Asdiscussed, assurance processes for preventing andresponding quickly to suspected fraud were verystrong across all Members. Across the other fourWays of Working, assurance mechanisms forsome Members are very strong, while otherMembers are still in the process of strengtheningor developing ways to assure quality and monitorthe consistency of their accountabilityperformance.The challenging nature of the revised DECAF-IIstandards played a significant role in the contrastbetween the mixed assurance performance of theMembers and their strengths in evidencing policy

and practice. Policy and on-the-ground practiceare able to respond more quickly to significantstep-changes in accountability and performanceexpectations than assurance systems, which placegreater demands on organisational resources andtake longer to design and implement.As discussedin each of the above sections, all Members arecurrently taking on innovations or improvementsin their assurance systems for one or more Waysof Working. We therefore expect that assuranceperformance against the DECAF-II standards willsee marked improvements over the coming yearsas the DEC Members endeavour to furtherstrengthen their standing as UK leaders inhumanitarian accountability and performance.

20


Way ofWorking3.3Way ofWorking3.4 Findings &Conclusion


Findings &ConclusionWay ofWorking3.3Way ofWorking3.4

Appendix 1: DEC Accountability Priorities and Ways of Working

1. We use our resources efficiently and effectively

2. We achieve intended programme objectives in accordance with agreed humanitarian standards, principles and behaviours

3. We are accountable to disaster affected populations

4. We learn from our experience – taking learning from one emergency to the next

1.1 Documented processes are in place at the appropriate level governing the use and management of funds

2.1 The agency has defined and documented processes for programme management which are underpinned by a clear statement of standards

3.1 A defined and documented Accountability Framework is in place governing accountability to disaster affected populations

4.1 A documented process is in place at the appropriate level to ensure that learning is systematically captured, shared and utilised

1.2 Approach to the management and care of staff reflects People in Aid code of good practice

2.2 Programmes respond to clearly defined needs and are adjusted as needs change

3.2 Information on agency background, programme timelines, beneficiary entitlements and selection criteria is communicated to disaster affected populations

4.2 Key learning is effectively communicated to staff, partners and other stakeholders

1.3 Programme design and procurement processes maximise value for money - balancing quality, cost and timeliness at each phase of the response

2.3 Staff and partners understand and integrate agreed standards in to their programmes

3.3 Disaster affected populations participate in programme assessment, design, implementation and evaluation

4.3 Key learning (including from evaluations) is incorporated into processes and programmes in a systematic and timely manner

1.4 Incidences of actual/suspected fraud or loss are investigated and actioned

2.4 Agencies participate in established coordination mechanisms and support their partners to do the same

3.4 In consultation with disaster affected populations appropriately designed and described complaints handling procedures are implemented

4.4 Agencies contribute to peer and sector learning in humanitarian response

1.5 Partners are consulted and their capacity is assessed, utilised and developed where appropriate

2.5 Unintended programme impacts and outcomes are identified and acted upon in a timely manner

3.5 Agencies shall work with partners to strengthen their capacity to be accountable to disaster affected populations

1.6 Local structures (including governments, civil society organisations and markets) are consulted and strengthened

2.6 Programmes contribute to disaster risk reduction and build the resilience of communities, partners and governments

2011-12 dec accountability framework assessment report

Documents

way ofworking1

way ofworking2

introduction way ofworking

accountability assessment

programme assessment

conductedby christina

peer reviewprocessjuly

decaf assessment