final evaluation of the afcop-mfdr project ... · web vieworganization of the national high-level...

THE AFRICAN DEVELOPMENT BANK GROUP

FINAL EVALUATION OF THE AfCoP-MfDR PROJECT- “Building capacity on Managing for Development Results in the Regional Member Countries (RMCs) and the Regional Economic Communities (RECs)”

Draft Final Report

11-16-2017

THE AFRICAN DEVELOPMENT BANK GROUP

FINAL EVALUATION OF THE AfCoP-MfDR PROJECT-

“Building capacity on Managing for Development Results in the Regional Member Countries (RMCs) and the Regional Economic

Communities (RECs)”

Draft Final Report

EVALUATION TEAM:Ms. Aimtonga Amani MakawiaDr. Maximilien Tereraho

November 2017

EXECUTIVE SUMMARY

1. Evaluation Purpose and Background

This is the Final Evaluation of African Community of Practice (AfCoP) project evaluation, a multinational capacity building pilot project titled “Building Capacity on Managing for Development Results in the Regional Member Countries (RMCs) and the Regional Economic Communities (RECs)” (AfCoP-MfDR Project hereafter).

The African Community of Practice on Managing for Development Results (AfCoP) was established in 2007 and temporarily hosted by the World Bank (2007-2012). In 2012, the AfDB funded AfCoP-MfDR Project to support the AfCoP in promoting the mainstreaming of Managing for Development Results (MfDR) practices in 17 regional member countries and two regional economic communities (COMESA and WAEMU). The Project aimed also at strengthening the capacities to facilitate policy and regional programs implementation through the MfDR approach.

Originally approved for three (3) years, the initial project end date of June 30, 2016 was extended to May 31, 2017. The total cost for the three-year period was UA9.98 million, mainly from ADF grant for regional operations (90.2%).

A final evaluation needed to be conducted as part of the Bank’s commitment to ensuring that lessons learnt during the implementation process are duly internalized and used for improving project design and implementation of another phase of the Project, if deemed necessary.

2. Evaluation Framework and Methodology

The purpose of this evaluation is to assess the performance of the AfCoP-MfDR Project in the delivery of its specific expected outputs, outcomes and possible impacts during its implementation period. Specifically, the evaluation establishes the performance of the

~ ~

project in terms of: relevance; efficiency and effectiveness; outcomes and possible impact; and sustainability.

The evaluation used both primary and secondary data collected through document review and key informant interviews in a sample of seven (7) countries four of which are in COMESA, namely Kenya, Madagascar, Zambia and Zimbabwe three namely Burkina Faso, Ivory Coast and Senegal in WAEMU. Similar surveys and interviews were conducted at the project secretariats at African Capacity Building Foundation and AfDB a as well as the WAEMU and COMESA.

3. Evaluation Findings and Conclusions

Relevance: Notwithstanding the shortcomings identified in the project design, the relevance of the AfCoP-MfDR Project is overall satisfactory (3/4). There is an emerging consensus on the continued need for an MfDR capacity development program of AfCoP-MfDR Project type; a coordinated partnership to instill the culture of results and to strengthen the use of MfDR for effective implementation of policies, programs and projects in the current RMCs and RECs. Scaling-up the project to cover other African sub-regions would also be welcomed subject to improved project governance structure.

Efficiency: The AfCoP-MfDR Project efficiency is rated as moderately satisfactory with an average score of 2.6/4: (3/4) for Timeliness, (2/4) for Linking resources to results, and (3/4) for Partnership and Bank management. The AfCoP-MfDR Project’s institutional arrangements lacked mutual reinforcing steering coordination mechanisms and incentives for leveraging expected synergies between AfDB Headquarters and country offices, national and regional activities, coaches and CoPs and governments.

Effectiveness and possible impact: The AfCoP overall effectiveness is satisfactory (3/4). Levels and quality of achievements at both output and outcome levels were relatively higher on Project Component 1- Knowledge sharing than

~ ~

Component 2- Linking MfDR knowledge to regional processes and Component 3- Synergies between regional and national processes.

Progress towards achievement of the objective of strengthened MfDR capacity in COMESA and WAEMU and their member states is significant, particularly at both the individual and institution level albeit to a lesser extent in the latter case. Achieved immediate and intermediate outcomes do point to possible realization of the long-term project objective: improved development and implementation of policies, programs and projects in Africa through strengthened MfDR capacity.

However, the countries and REC’s secretariats remain stalled at the stage of planning for results in the absence of sufficient ownership and a deliberate human capacity development strategy to institutionalize MfDR, including through a framework of both academic and continuous training and a system of both positive and negative sanctions for results achieved or not achieved.

Sustainability: The AfCoP-MfDR Project sustainability is rated as moderately satisfactory with an average score of 2.75/4: (3/4) for Technical and financial sustainability, (2/4) for Institutional sustainability and strengthening of capacities, (2/4) for Ownership and sustainability of partnerships, and (4/4) for Environmental and social sustainability.

Project sustainability and extension beyond the current RMCs and RECs will depend on the capacity of the AfCoP to raise enough resources, to act upon a new designed governance structure, as it includes a legal status of the AfCoP, within the framework of a next phase of the AfCoP-MfDR Project.

4. Evaluation Lessons Learned and Recommendations

There is need for the Bank and its partners to sustain and expand the AfCoP-MfDR project gains and the MfDR momentum at RMC and REC levels taking into account the following key lessons learned from this evaluation of the pilot phase: Moving from the stage of diagnostic and planning for results to

MfDR implementation remains a challenge for both the RMCs and RECs secretariats;

~ ~

Better inter-organizational collaboration within the Bank and in countries is needed for effective MfDR capacity development project;

M&E-driven institutional arrangements are an indicator of strong highest leadership engagement, hence a key enabling factor for MfDR institutionalization;

Promoting regional integration through the Bank’s multinational capacity development projects is in need of evidence basis;

A critical mass of national/regional coaches and MfDR-skilled human resources is a determinant for MfDR capacity development;

National political or natural disaster-related crisis seems to be at the same time a barrier and enabling factor for adoption of MfDR, particularly RRIs

Based on the above what-works findings, the following are recommended changes, improvement or programmatic issues of a possible next phase of the AfCoP-MfDR Project or similar approaches to MfDR development in Africa: Ensure high project quality at entry, particularly for regional

focus; Establish clear results-based roles for the Bank’s field office and

local partners, and ensure synergy with other relevant Bank’s projects;

Consider single and independent institution approach for managing the project along with a formal legal status of the AfCoP;

Consider enhancing the project demand-driven delivery model by a differentiated approach at different levels;

Consider fostering the formulation and implementation of national and/or regional comprehensive strategy for creation of national/regional pools of MfDR leaders through a combination of academic and vocational training, results-based rewards system and institutionalization of the national coaches’ program.

~ ~

LIST OF ACRONYMS

ACBF African Capacity Building Foundation

ADF African Development Fund

AfCoP African Community of Practice

AfCoP-MfDR African Community of Practice on Managing for Development Results

AfDB The African Development Bank

AfrEA African Evaluation Association

AfriK4R The Africa for Results Initiative

AHHD Human Capital, Youth and Skills Development Department

CD Capacity Development

CLEAR Centre for Learning on Evaluation and Results

COMESA Common Market for Eastern and Southern Africa

CoP Community of Practice

CSOs Civil Society Organizations

EU European Union

ExCo Executive Committee

FGD Focused Group Discussions

G4R Gender for Results

IDEV Independent Development Evaluation

JICA Japan International Cooperation Agency

KEPSA Kenya Private Sector Alliance

M&E Monitoring and Evaluation

MED Monitoring and Evaluation Department

~ ~

MfDR Managing for Development Results

MfDR CD Managing for Development Results for Capacity Development

MNDP Ministry of National Development Planning

MTR Mid-Term Evaluation Report

NGO Non-Governmental Organization

NSA Non-State Actors

PAR Project Appraisal Report

PCR Project Completion Report

PSOs Private Sector Organizations

RBM Results Based Management

RECs Regional Economic Communities

RMCs Regional Member Countries

RRIs Rapid Results Initiatives

SDGs Sustainable Development Goals

SLDP Strategic Leadership Program

SNDR Delivery, Performance Management and Results Department

TFP Technical and Financial Partners

ToR Terms of Reference

UNDP United Nations Development Programme

UNFPA United Nations Population Fund

UNICEF United Nations Children’s Fund

USAID United State Agency for International Development

WAEMU West African Economic and Monetary Union

WB World Bank

~ ~

Y4R Youth for Results

ZPSA Zimbabwe Pharmaceutical Students Association

TABLE OF CONTENTS

EXECUTIVE SUMMARY....................................................................................................2

LIST OF ACRONYMS.............................................................................................................5

LIST OF TABLES AND BOXES......................................................................................9

SECTION I: INTRODUCTION AND BACKGROUND.......................................10

1.1. PURPOSE AND STRUCTURE OF THE REPORT.......................................10

1.2. CONTEXT OF THE AfCoP-MfDR PROJECT FINAL EVALUATION......10

SECTION II:EVALUATION FRAMEWORK AND METHODOLOGY..........13

2.1. EVALUATION OBJECTIVES AND SCOPE....................................................13

2.2. EVALUATION QUESTIONS AND CRITERIA...............................................14

2.3. THEORY OF CHANGE AND OUTCOMES USED IN THIS EVALUATION......................................................................................................................14

2.4. DATA COLLECTION AND ANALYSIS METHODS.....................................16

2.4.1. Data Collection...............................................................................................16

2.4.2. Data Analysis...................................................................................................16

2.5. EVALUATION LIMITATIONS AND MITIGATION STRATEGIES.........17

3.1. RELEVANCE OF THE PROJECT......................................................................19

3.1.1. Relevance of AfCoP-MfDR Project Objectives....................................19

3.1.2. Relevance of AfCoP-MfDR Project Design............................................19

3.1.3. Overall Rating for Relevance....................................................................21

3.2. AfCoP-MfDR PROJECT EFFICIENCY.............................................................21

3.2.1. Timeliness of Project Operations.............................................................22

3.2.2. Linking Resources to Results....................................................................22

3.2.3. Partnership and AfDB Management.......................................................25

3.2.4. Overall Rating for Efficiency.....................................................................27

3.3. AfCoP-MfDR PROJECT EFFECTIVENESS AT OUTPUT LEVEL...........28

3.3.1 General Emerging Trends..............................................................................28

~ ~

3.3.2 Quality of Outputs.............................................................................................28

3.4. AfCoP-MfDR PROJECT OUTCOMES AND POSSIBLE IMPACT...........36

3.4.1 Outcomes at Individual Level........................................................................36

3.4.2 Outcomes at Institution Level.......................................................................37

3.4.3 Outcomes at Country and Regional Level................................................38

3.4.4 Unintended Outcomes.....................................................................................41

3.4.5 Overall Rating for Achievement of Outcomes.........................................41

3.5. PROJECT SUSTAINABILITY..............................................................................42

3.5.1 Technical and Financial Sustainability......................................................43

3.5.2 Institutional Sustainability and Strengthening of Capacities...........43

3.5.3 Ownership and Sustainability of Partnerships.......................................44

3.5.4 Environmental and Social Sustainability..................................................44

3.5.5 Overall Rating for Sustainability..................................................................44

SECTION IV: LESSONS LEARNED AND RECOMMENDATIONS............45

4.1. LESSONS LEARNED.........................................................................................45

4.1.1 Project Design and Implementation...........................................................45

4.1.2 Country and REC implementation of the Road Map............................46

4.2. OVERALL CONCLUSION AND RECOMMENDATIONS..................47

ANNEX I- EVALUATION MATRIX............................................................................50

ANNEX II- LIST OF CONSULTED STAKEHOLDERS...................................54

ANNEX III- LIST OF REVIEWED DOCUMENTS AND SUGGESTED READINGS 60

ANNEX IV- EVALUATION TERMS OF REFERENCE.....................................62

~ ~

LIST OF TABLES AND BOXES

Table 1: AfCoP-MfDR Project Financing Plan..............................12

Table 2: Summary of Generic Rating Scale..................................14

Box 1: AfCoP-MfDR Project Objectives in COMESA and WAEMU......20

Table 4: Status of Physical Implementation by Project

Components.................................................................................23

Table 5: Status of AfriK4r Country Roadmap Activities Achievements...............................................................................24

Table 6: Budget Allocation and Uses at National and Regional Level..............................................................................................25

Box 2: Missed Role of AfDB Country Office.........................................26

Table 7: AfCoP-MfDR Project Achievements of Outputs............30

Box 3: AfCoP-MfDR Theory of Change at Work in Kenya...................37

Table 8: Status of MfDR in Zimbabwe, Zambia and Ivory Coast.......................................................................................................39

Box 4: State of MfDR in COMESA Secretariat....................................40

Box 5: RRIs As an Effective Post-Crisis Transition Tool......................40

Table 9: Unanticipated or Additional Outputs/Outcomes..........42

~ ~

SECTION I: INTRODUCTION AND BACKGROUND

1.1. PURPOSE AND STRUCTURE OF THE REPORT

1. This is the Final Evaluation of African Community of Practice (AfCoP) project evaluation, a multinational capacity building pilot project titled “Building Capacity on Managing for Development Results in the Regional Member Countries (RMCs) and the Regional Economic Communities (RECs)” (AfCoP-MfDR Project hereafter). It was supported by African Development Bank (AfDB) and implemented by AfCoP Secretariat at AfDB (managed component 2, 3 and 4) and the African Capacity Building Foundation (ACBF)- that managed component 1. The evaluation took place from 01 June – 15 Sept 2017.

2. The evaluation report describes the outcomes and possible impact of the AfCoP-MfDR Project in the delivery of its specific objectives, outputs, and outcomes during its implementation period as outlined in the Project Appraisal Report (PAR). The evaluation establishes the performance of the project in terms of its relevance, efficiency, effectiveness and sustainability of its results, including an exit strategy. Additionally, it provides lessons learnt and recommendations to improve a results-culture in Africa through a possible next phase of the AfCoP-MfDR Project.

3. Section 1 presents the background and context of the AfCoP-MfDR Project final evaluation, while section 2 defines the evaluation purpose, specific objectives and scope, in terms of timeframe, covered project activities, thematic and geographic areas. Then, it provides details of the evaluation criteria and the evaluation matrix (presented in Annex I). Finally, it outlines the evaluation approach and methodology, including highlights of the AfCoP theory of change and the outcomes used in evaluation, as well as the evaluation data collection and analysis methods. Section III presents the evaluation findings and conclusions, while section IV summaries the lessons learned and recommendations. The terms of reference, list of people interviewed and reviewed documents are provided at the end of the report as Annexes II-V. The report is written as per AfDB’s standard evaluation report format.

~ 1 ~

1.2. CONTEXT OF THE AfCoP-MfDR PROJECT FINAL EVALUATION

4. The African Community of Practice on Managing for Development Results (AfCoP) was established in 2006 as one of the CoPs that are supported and hosted by the regional Banks, with the aim of contributing to the promotion of results-culture and practice in Africa. Between 2007 and 2012 the AfCoP Secretariat was temporarily hosted at the World Bank as the AfDB (the Bank hereafter) was going through an institutional restructuring. At the time of tabling this pilot phase of the AfCoP-MfDR Project under evaluation in 2011, there had been a mutual agreement between the AfCoP Core Management Team, the World Bank and the AfDB for the AfCoP Secretariat to be transferred from the Results Department of the World Bank to the ORQR Department of the Bank (current SNDR).

5. The AfCoP was established as a response to developing countries’ call for a stronger ownership and leadership in shaping the results agenda and in reinforcing capacity to strengthen country institutions. AfCoP also contributed to shaping the African perspective on development effectiveness at the regional meeting held in Tunis in November 2010, which provided valuable input into the 2011 Busan Partnership for Effective Development Co-operation.

6. The Tunis consensus re-focused the attention from aid effectiveness to the broader development effectiveness agenda and recognized that development in Africa requires a robust and private sector-led growth, effective and accountable states, able to finance their development needs from their internal resources. To be truly successful, this process must also be accompanied by more efficient regional development and economic integration across the continent, thus creating economies of scale that will boost trade and investment and equip African countries to compete in the global economy.

7. In 2012, the AfDB funded AfCoP-MfDR Project to support the AfCoP-MfDR in:i) Promoting the mainstreaming of MfDR practices in 17 RMCs

and two RECs (COMESA and WAEMU). Due to an increased interest on the project, an 18th country (Rwanda) was added later; and

~ 2 ~

ii) Strengthening the capacities of the two RECs and their respective member countries to facilitate policy and regional programs implementation through the MfDR approach, hence promoting regional integration. Regional and national communities of practice (CoPs) were established in the two RECs and the 18 RMCs as a conduit to manage the activities of the AfCoP-MfDR Project.

8. To this extent, AfCoP developed the Africa for Results initiative (AfriK4R), as a regional approach to development, to help materialize that vision by mobilizing African stakeholders around the results and regional development agendas. The AfriK4R initiative aimed to: i) Strengthen public sector management systems to build capable

states; ii) Strengthen results-oriented organizations for better service

delivery;iii)Enhance regional convergence through MfDR to improve

regional integration; and iv) Promote South-South knowledge sharing.

9. The AfCoP-MfDR Project was originally approved for three (3) years with the understanding that this was a pilot which could be scaled up. As exhibited in Table 1, the total cost for the three-year period was UA9.98 million, mainly from ADF grant for regional operations (90.2%). The remaining 9.8% was came from the Bank (mainly in kind) and RMCs and RECs (training workshops and facilities to the project focal points), respectively at 4.2% and 5.6%.

Table 1: AfCoP-MfDR Project Financing Plan

Source Amount (UA)

Instrument Percentage

ADF 9.00 million Grant for Regional operations

9o.2

ADB 0.42 million Administrative budget 4.2RMCs &RECs

0.56 million 5.6

TOTAL COST 9.98 million

Source: Project Appraisal Report, page ii & 8.

10. The initial ending date of June 30, 2016 was extended to May 31, 2017 with the understanding that RMCs and RECs would have completed the activities of the agreed roadmap and plans of actions. In line with AfDB procedures, an independent evaluation

~ 3 ~

was commissioned at the end of the AfCoP-MfDR Project in order to ensure that lessons learned during the implementation process are duly internalized in future programming and they are used to improve decision-making processes, including project design, implementation and monitoring of another phase of the project, if deemed necessary.

~ 4 ~

SECTION II:EVALUATION FRAMEWORK AND METHODOLOGY

2.1. EVALUATION OBJECTIVES AND SCOPE

11. The purpose of this evaluation is to assess the outcomes and possible impact of the AfCoP-MfDR Project in the delivery of its specific objectives, outputs, and outcomes during its implementation period as outlined in the PAR. The evaluation establishes the performance of the project on the following main thematic areas, while identifying lessons learnt and best practices to be promoted in order to improve a results-culture in Africa: i) Relevance of the project;ii) Project efficiency and effectiveness;iii) Project outcomes and possible impact;iv) Project sustainability.

12. The evaluation focused solely on the activities conducted and the results achieved under the project by the implementing agencies (AfDB and ACBF). The ACBF was responsible for implementation of Component- 1 of the AfCoP-MfDR Project (Sharing knowledge on MfDR) while the AfCoP Secretariat/AfDB was responsible for the project implementation coordination and delivery on the remaining three Components: i) Component- 2: Linking MfDR knowledge to regional processes; ii) Component- 3: Synergies between regional and national

processes;iii)Component- 4: Project management.

13. The evaluation covered the two AfCoP-MfDR Project RECs and 18 of their member states, from the project effectiveness date of May 2012 to the extended project end date of May 31, 2017. However, since the project disbursement end date was set for November 30, 2017 and some engaged activities were still underway at the time of field work (July-August 2017), such activities were taken into account when assessing the progress towards achievement of the project expected results. A sample of seven (7) country cases were selected, namely:i) COMESA: Kenya, Madagascar, Zambia, Zimbabwe;ii) WAEMU: Burkina Faso, Ivory Coast, Senegal.

14. These countries were selected on the following ground:

~ 5 ~

i) They were able to complete or were well advanced in implementing the road map and in this regard, could be considered as models (Madagascar);

ii) Four (4) of them, respectively Zambia, Burkina Faso, Zimbabwe and Ivory Coast hosted AfCoP partners, namely the COMESA Secretariat, the WAEMU Commission and the AfCoP Secretariats at ACBF and AfDB;

iii)They had internalized the MfDR processes (Senegal).2.2. EVALUATION QUESTIONS AND CRITERIA

15. The assessment of progress towards achieving project expected

results used the AfDB’s rating guidance for performance of public sector as of August 2017 (Table 2). As indicated above, the AfCoP Final Evaluation criteria included: relevance, efficiency and effectiveness, outcome and possible impact, and sustainability aspects. The Evaluation Team assumed that partnership and the role of the Bank would be addressed under the efficiency criteria and gender considered under all of the evaluation criteria. The draft Evaluation Matrix in Annex I provides detail of the evaluation questions per evaluation criterion, key indicators and source of evidence.

Table 2: Summary of Generic Rating Scale

1- Very Unsatisfactory

Good performance against few or no aspects

2- Unsatisfactory Good performance only on some aspects

3- Satisfactory Good performance against the majority of aspects

4- Very Satisfactory

Good performance against all or nearly all aspects considered

Source: Adapted from BAD/ORQR.2 (2012).

2.3. THEORY OF CHANGE AND OUTCOMES USED IN THIS EVALUATION

16. As mentioned earlier and outlined in the PAR, the AfCoP-MfDR Project aimed to support the AfCoP to mainstream results-based practices of MfDR into the policies and strategies of RMCs and RECs. The capacity of RMCs and RECs in MfDR was to be

~ 6 ~

developed through sharing experiences, networking and building strong learning relationships between MfDR practitioners in Africa, and around the World. “The main expected results are related to: (i) increased effectiveness in implementing the regional policies in WAEMU and COMESA member states; and (ii) strengthened MfDR capacity of WAEMU and COMESA and their member states” (PAR, 2011: 11).

17. The AfCoP-MfDR Project theory of change was explicitly stated at the outset of the PAR, under the ‘’Strategic Thrust and Rationale’’ chapter (p. 2-3) and further articulated under Annex 1. The stated theory of change is a transformative theory of capacity development program, that is based on the empirically tested principle that for MfDR capacity development (CD) to be effective and sustainable, there must be a shift in individual, organizational and society culture from compliance, especially with technical and financial partners (TFP) accountability and results reporting requirements, to using evidence in decision making and for continuous learning, innovation and improvement.

“The process needs leaders, champions and change agents at all levels of the government to take this change-management initiative forward. The African Community of Practice for Development Results is considered a key driver of change… Its membership consists of public administration staff, parliamentarians, civil society, the private sector, academia, the media and local donors. They are seen as change agents that contribute to empowering the societal actors through MfDR learning and knowledge in order to effect transformational and sustainable change in their institutions, which in turn support achievement of development goals.” (PAR, 2011: Annex 1, p.1).

18. The AfCoP members were expected to help in building skills in priority settings, diagnosing institutional constraints and problem-solving, results focus, and strengthening the sense of accountability. In the same way, it was expected from the project initiative that sustained change behavior for regional integration will originate from relational and reputational incentives, through peer pressures and advocacy from various stakeholders including parliamentarians and non-state actors (NSAs).

19. Unfortunately, the PAR and its Results framework provided very limited operational guidance on how to go about translating the cultural shift towards results-based public administration into a

~ 7 ~

reality. In addition to unclear rationale of the planning horizon for the realization of the above quoted project outcomes, the Results framework did not trace the pathway towards strengthened MfDR capacity of WAEMU and COMESA and their member states by 2016. No immediate and intermediate outcomes were identified.

20. In fact, the theory of change above described assumes that the transformation was to happen in an interactive manner, whereby the outcomes on the individual, organizational and community/society levels recursively interact over time. In the short run, as immediate personal outcomes, the AfCoP members were expected to demonstrate increased technical, organizational and behavioral skills in the areas of AfCoP Pillars, including self-awareness, self-confidence, motivation and networking for promoting integrated result-based development management. In the medium term, as intermediate and mainly team/institutional outcomes, the AfCoP-MfDR Project beneficiaries were expected to apply and share learning and practice MfDR within the organizations where they work, affiliated or they contact. In the long run, as ultimate and mainly community/society outcomes, it was anticipated that the skilled MfDR leaders’ influence would extend to various social spheres, either directly through their routines or through their organizational work and contacts.

21. The Evaluation Team reviewed the AfCoP-MfDR Project’s Results framework contained in the 2011 PAR in light of such an operationalized transformative project theory of change and where applicable, streamlined the wording and or the understanding of short, medium and long term outcome indicators as well as the evaluation questions. Thus, project outcomes were assessed against the project’s objectives as they are stated in the PAR with as clear as possible distinction between immediate and intermediate outcomes in an effort to report on project progress towards achievement of expected outcomes and impact. However, it should be noted that this evaluation does not assess the long term outcomes and impact of the AfCoP-MfDR beyond increased capacity of the covered RMCs and RECs in the six (6) MfDR pillars.

2.4. DATA COLLECTION AND ANALYSIS METHODS

2.4.1.Data Collection

22. In search of answers to the evaluation questions, the evaluation used different methods and multiple lines of evidence. The

~ 8 ~

Evaluation Team used both primary and secondary data sources in an interactive manner: i) Focus group discussions (FGD) with national and regional CoP

members.ii) Secondary data collection was done through desk review of

project documents, log frame and progress reports, as well as content analysis of key reports and reference documents. Data collection took place by request and receipt of documents from AfCoP Secretariats at AfDB and ACBF, RECs, beneficiary countries and other partners involved in MfDR CD, as well as computer searched references;

iii)Primary data collection was done through field work in the seven sampled participating countries, the project secretariats at ACBF and AfDB as well as the two RECs.

23. Annex III provides a list of key stakeholders consulted during the field visits. Besides AfDB and ACBF and the RECs, in each of the country visited key government ministries and other relevant government entities, national and regional NSAs of the private sector, the civil society, in country development partners as well as implementing partners were interviewed.

24. The interviews and discussions were semi-structured and conducted in an interactive and complementary manner, taking into account that not all interviewees were in a position to provide valuable information on all of the issues, and where possible, within the set interview duration. Succeeding interviews were modulated and customized so as to fill in possible information gap. One member of the Evaluation Team covered COMESA region while the other covered WAEMU region. The Team then joined for data collection with the AfDB AfCoP Secretariat in Abidjan.

25. As the document review, the stakeholder consultations aimed at collecting information on: Current state of MfDR in the country and REC; perceived areas of improvement; possible actions in the short, medium and long-term for improvement of MfDR capacity strengthening. Key informant interviews included exit interview conducted at both the country and regional level with identified CoP former members and/or coaches to identify areas of possible improvement of the CoP model and the MfDR capacity.

2.4.2.Data Analysis

~ 9 ~

26. Primary data collected by interviews, group discussions and observations were triangulated against secondary data; and a content analysis and quality review of AfCoP-MfDR Project publications and materials. Interview notes and their interpretation were cross-validated by the two senior consultants.

27. This final evaluation report is a synthesis of findings and conclusions resulting from two intertwined levels of analysis: Quality at entry analysis and Portfolio strategic review. Illustrative examples drawn from cases of the sample visited countries are used to substantiate findings.

28. Quality at entry assessment aimed at identifying factors of success and failure in project implementation that are linked to the design of operations, for instance:a) Relevance and realism of the objectives of the operations;b) Participatory character of the project design; c) Quality of risk analysis and mitigation measures; d) Adequacy of the proposed institutional and organizational

arrangements, including relevance and coherence of partnership business model;

e) Adequacy of cross-cutting issues mainstreaming (gender, youth, etc.);

f) Quality of the result measurement framework and the related indicators;

g) Proper understanding of the national and regional context; and,h) Adequacy of measures to ensure sustainability of operations and

results.

29. The Portfolio strategic review aimed at: (a) Assessing performance trends emerging across the four AfCoP-MfDR Project Components; (b) Identifying factors underlying the trends observed above; and (c) Drawing lessons to be learned from the experience for future planning and M&E. From a forward-looking perspective, the Portfolio strategic review led to the answer as to whether in a possible third phase of the project, the AfCoP-MfDR Project Components and governance structure should be continued as they are, extended or narrowed or reoriented.

2.5. EVALUATION LIMITATIONS AND MITIGATION STRATEGIES

~ 10 ~

30. The following limitations should be noted for due diligence in interpreting and/or using the findings of this evaluation. In addition to the specific mitigation actions mentioned beneath, high and long experience of the two evaluators in qualitative research and evaluative interviews were used for validation of received information.

a) Consulted stakeholders were not necessary selected on random basis at individual and/or institutional level. Where possible and as necessary, non-project beneficiaries were included in the sample for opinion diversification purposes;

b) Limited access to project records at country or REC level. Succeeding interviews and readings were adjusted so as to fill in possible information gap;

c) In some cases, the consultants could not meet national development partners to gauge on their insights and experience on the project and find out about their future plans and synergy and coordination. Succeeding interviews and readings were adjusted so as to fill in possible information gap;

d) Distant coordination of field data analysis and writing up the evaluation report took more time than was anticipated as challenges related to technology and communication, among other constraints, crept in. A slight deadline extension was proactively secured from the client.

~ 11 ~

SECTION III: EVALUATION FINDINGS AND CONCLUSIONS

3.1. RELEVANCE OF THE PROJECT

3.1.1.Relevance of AfCoP-MfDR Project Objectives

31. Evaluation of relevance of the project objectives assesses the extent to which the project purpose as specified in the Results Matrix was aligned with the country/REC’s development strategies and the beneficiary needs from design to completion, including any adjustments that were made to the project. Specifically, the evaluation assessed whether the objectives falling under the four components of the AfCoP-MfDR Project were consistent with the national/regional development strategies, policies and plans, as well as those of the Bank.

32. As above described, the AfCoP-MfDR Project aimed to support the AfCoP-MfDR in promoting the mainstreaming of MfDR practices in 18 RMCs and the two RECs policies and strategies, hence promoting regional integration (see Box 1). The project's objectives as stated in the PAR are relevant given their alignment with most Results-Based Regional Integration Strategy Papers, country visions and national development plans. Indeed, the AfCoP-MfDR Project aimed to contribute to promoting capacity development on results management, hence effective policy/program/project implementation and development effectiveness.

33. In addition to the more overall Bank's strategy to strengthen the capacity of African countries, the AfCoP-MfDR Project objectives are also in alignment with the Bank’s priorities, including regional integration and monitoring of development results (one of the current High 5s), as well as with the Bank’s commitment to support the global development effectiveness agenda. The project is also consistent with the Africa Platform on Development Effectiveness established by the NEPAD Agency. To cite just a few, the following documents emphasize the central role of MfDR in Bank interventions:i) The Medium-Term Strategy 2008-2012 and the 10-year Medium-

Term Strategy 2013-2022;ii) The Knowledge Management and Development Strategy 2008–

2012 and the Knowledge Management Strategy for 2015-2020.

~ 12 ~

3.1.2.Relevance of AfCoP-MfDR Project Design

34. Evaluation of the relevance of project design assesses the quality of the initial design in terms of : (i) the extent to which the project’s objectives are clearly stated and focused on outcomes as opposed to outputs; (ii) the realism of intended outcomes in the current circumstances; (iii) the extent to which the project design adopted the appropriate solutions to the identified problems; (iv) the extent to which modifications to project design, if any, were adapted to the circumstances prevailing at the time of implementation.

Box 1: AfCoP-MfDR Project objectives in COMESA and WAEMU

AfCoP-MfDR Project objectives are aligned with the COMESA’s mandate and overall mission, that is to help accelerate the implementation of regional policies and strengthen the convergence of the policies and performance. For instance, in Madagascar, the AfCoP-MfDR Project responded to the needs for enhancing the implementation of the national development plan within the framework of reconstructing the public administration and the country from the political crisis of 2009-2013. In Zimbabwe development, AfCoP-MfDR Project objectives have complemented the national development, and the long-term reform agenda: start of focus on performance based management and smooth continuation of the program based budgeting reforms.

Consulted stakeholders at WAEMU Commission indicated that AfCoP-MfDR Project activities came at a timely moment to support the Commission's efforts to promote the culture of results within the Commission. For instance, in Senegal, the AfCoP-MfDR Project was and is still considered as relevant since the project responded to the imperative need to revitalize MfDR in the country. In Burkina Faso, the implementation of the Burkina Faso Road Map was disrupted and delayed by the socio-political crisis of 2014-16, but the country's new leadership was able to catch up in 2016-2017; with Burkina Faso becoming the first country in WAEMU to implement program results-based budgeting. In Ivory Coast, MfDR capacity reinforcement will be even more needed in the future if the country is to pursue the devolution of community development.

35. To the extent that all activities under the four (4) project components were aimed at promoting MfDR culture and practice both in RMCs and RECs, such activities were relevant. The design of the project is also relevant in that it is selective, demand-driven and tailored to clients' needs, taking into account the difference in levels of MfDR in RMCs.

~ 13 ~

36. However, and as the Medium-Term Review had already noted, the project Results framework lacked clear causal linkages between the projects outputs and its outcomes and impact. No one expected output was defined in the Results-Matrix for the fourth project component (“Project management”) and there is confusion between output and outcome when trying to define outcome performance indicators. At the same time, the Results framework falls short of the steps through which an increased MfDR capacity of WAEMU and COMESA and their member states is expected to be achieved in 2016. Instead, the making of the project business case becomes quite ambiguous and ambitious as some stated expected results make reference to those that are theoretically pursued under regional integration treaties, not a management capacity development project (PAR, 2011: 11-12). Neither at this level did the Results framework show how the effective implementation of regional policies and that of the regional policy convergence could be achieved, respectively in 2018 and 2020 or later.

37. By design, it couldn’t be assumed that demand-driven project delivery is likely to secure political commitment and functionality of the project architecture and coordination from the Bank headquarters. There was a need for clear articulation between national and regional CoPs, between the CoPs and the machines of governments and RECs, and between governments and RECs. If they existed, the regional and national steering committees were not functional and expected coordination of the project activities at the regional level by the regional focal points in COMESA and WAEMU (PAR, 2011: 18) was not effective. At both REC and RMCs, consulted stakeholders explained the disrupted and delayed implementation of the Road Map by a (wrong) assumption that the Bank was to provide the funds required for implementation of the developed action plans, including projects in the sector applying RRIs. Such misunderstanding and unmet but wrong expectations attracted long discussions between the parties and ultimately halted the project implementation at the COMESA Secretariat.

38. Implications of these design shortcomings are discussed in more detail below under the efficiency and effectiveness evaluation criteria. Nevertheless, it should be noted that the Medium-Term Review rightly underlined that this was likely to create both challenges in implementation and in measuring the effectiveness of the project. Therefore, the Medium-Term Review report recommended consideration of modifications to existing indicators, outputs, and outcomes and/or reassignment of resources and

~ 14 ~

solutions, but this evaluation could not establish that the recommendation was implemented.

3.1.3.Overall Rating for Relevance

39. In conclusion, notwithstanding the project design shortcomings mentioned above, the relevance of the AfCoP-MfDR Project is overall satisfactory (3/4). At the time of this evaluation field work, there was an emerging consensus among the stakeholders consulted, on the continued need for an AfCoP-MfDR project; a coordinated partnership to instill the culture of results, to strengthen the use of MfDR for effective implementation of policies, programs and projects in the current project beneficiaries and scale-up the project to cover other African sub-regions.

3.2. AfCoP-MfDR PROJECT EFFICIENCY

40. Evaluation of the project efficiency assesses the methods used in the course of implementing the project and discusses the adequacy and cost-effectiveness in the use of project resources in relation to planned activities, outputs and outcomes. The speed and costs involved in conversion of the project inputs into outputs are examined as well as the consequences of the centralized management of the project from AfDB Headquarters without a robust governance structure in its original design.

41. The evaluation found significant efficiency issues of AfCoP-MfDR Project implementation that could be associated with the shortcomings identified earlier under assessment of the project design relevance. The PAR did not discuss the advantages and disadvantages of implementing certain activities of RMCs and RECs directly by the Bank and others by consultant coaches. Although funding modalities were determined also by the Bank’s fiduciary requirements, some stakeholders believe that the Bank should have financed the RMCs and RECs directly or at least fully involve AfDB country offices, in order to avoid inefficiencies and thus delays. Majority of the visited AfDB country offices (2 in WAEMU and 4 in COMESA) did not feel really involved in the project implementation as their role was limited to disbursement of workshop funds.

3.2.1.Timeliness of Project Operations

~ 15 ~

42. As noted earlier and highlighted in the Project Completion Report (PCR), the AfCoP-MfDR Project was planned to be implemented in a period of three (3) years. The initial ending date of June 30, 2016 was extended to May 31, 2017 with the understanding that RMCs and RECs would have completed the activities in the agreed roadmap and plans of action. Indeed, the largest part of the planned activities was implemented in the last 12 months, but others were still on the core agenda at the time of this evaluation (June – September 2017). However, the AfCoP-MfDR Project’s delay of one year is shared with a good number of projects at the Bank.

3.2.2.Linking Resources to Results

43. Table 3 provides a summary of the project fund utilization while Table 4 does the same on the status of physical implementation per AfCoP-MfDR Project Component. The findings shows that the overall average expenditure of 87% in July 2017 does well compare to the overall implementation of the project activities estimated at 98% at the same date. The PCR indicates that the disbursement rate stands at 100% while the utilization of the funds under components 2-4, that was at an overall average of 83.9% in July 2017, is estimated at 93% at time of expenditure closure in November 2017. At this level, the use of the project financial resources seems to have been aligned with the project results (outputs) satisfactorily.

44. However, the link between resources and results was not evident along the whole project period. While the estimated physical implementation rate has been always higher than the fund utilization one from 2015-2017, sometimes, for instance by April 2016, this may mean that ACBF and the Bank have been somewhat subsidizing the implementation of the project activities for a while, with an implementation rate of 84% and a fund utilization at 51.4%. In kind Bank’s contribution included that of the Bank’s country offices in managing this project and the support provided by the Bank’s non- project staff.

45. In the same vein, some governments, for example Senegal and Burkina Faso, provided resources in cash and kind in terms of venues and transport. The PCR notifies that the project was able to enroll an additional pilot country (Rwanda) during its 4th year as a result of saving from the internalization of MfDR processes by some countries such as Senegal which used own funds to conduct

~ 16 ~

some activities of the roadmap. However, it sounds from a progress report of May 2017 by AfCoP Secretariat that the project could also rely on funds not yet disbursed in some countries. Indeed, the report shows that disbursement rate differed from COMESA to WAEMU and varied among countries inside each REC: i) From 100% in Madagascar and Zimbabwe (where the demand

remained high) to 33% in Zambia within COMESA for a regional average of 74%;

ii) From 89% in Ivory Coast to 9% and 22% respectively in Mauritania and Niger within WAEMU for a regional average of 47%.

Table 3: Summary of Financial Performance of the Project

COMPONENT Dec. 2015

April 2016

Nov 2016

July 2017

Knowledge Sharing 56.8% 56.8% 92.0% 97.0%

Linking MfDR to regional processes

80.8% 80.9% 81.3% 90.3%

Synergies between national & regional processes

29,0% 37.7% 43.8% 81.6%

Project management

27.0% 30.2% 51.9% 80.0%

Overall average 48.4% 51.4% 67.2% 87.2%

Source: Aide-Memoire, PCR Mission (July 17-26, 2017).

Table 4: Status of Physical Implementation by Project Components

COMPONENT Ap. 2015 Supervision

Dec 2015 Supervision

April 2016 Supervision

Nov. 2016 Supervision

July 2017 Supervision

1. Knowledge sharing

42% 69% 78% 93% 126%

2. Linking MfDR to regional

100% 100% 100% 106% 113%

~ 17 ~

3. Synergies between regional and national processes

38% 66% 68% 71% 78%

4. Project management

60% 93% 107% 120% 127%

Overall (weighted average)

57% 79% 84% 88% 98%


46. Likewise, insists the PCR, the project financed the Mid-Term Review along with regional report on the state of MfDR beyond the planned list of deliverables. As discussed beneath under the assessment of achieved outputs, a good number of non-planned activities were implemented while other planned activities were not. Table 5 shows that while all of the sample countries, except newly admitted Rwanda, have launched the national CoP by the end of July 2017, few had held the Implementation workshop (33%), Performance validation (39%) or the Mid-term review (41%) components of the national road map.

47. Furthermore, as exhibited in Table 6, data from the last audit report that covered the period from January to December 2016 show that de facto, the budget allocations between regional and national activities were reversed in the course of the project implementation in favor of regional activities. For all of these reasons, this evaluation could not come to the conclusion that AfCoP financial resources were adequately linked to results in the course of the project implementation.

Table 5: Status of Afrik4R Country Roadmap Activities AchievementsROADMAP STAGE OR ACTIVITY

NUMBER OF COUNTRIES having undertaken the activity

PERCENTAGE REALIZED (compared to total of 18 countries)

COUNTRIES that have undertaken the activity1

Readiness Assessment ready for

18 100% All

~ 18 ~

validation

CoP Launching 17 100% All except Rwanda2

Validation of the readiness assessment report

17 94% All except Rwanda

High level workshop

14 82% All except NE, RW, SE, TG

Learning Days of MfDR Week

13 82% All except BF, KY, MW, RW

Implementation workshop

6 33% BI, MA, SE, ZB, CI, DR

Mid-term review

7 41% BI, MA, SE, TG, ZB, CI, BE,

Performance validation

7 39% BI, MW, ZB, CI, MA, UG

Rapid Results Initiatives (RRI)

12 70% All except DR, ML, RW, TZ, UG, KY

1BE Benin; BF Burkina Faso; Bl Burundi; Cl Cote d'Ivoire ; DR Democratic Republic of Congo ; KY Kenya; MA Madagascar: ML Mali ; MT Mauritania: MW Malawi ; NI Niger; RW Rwanda; SE Senegal; TZ Tanzania; TG Togo ; UG Uganda ; ZA Zambia; ZB Zimbabwe. 2Rwanda joined the project in June 2016.


Table 6: Budget Allocation and Uses at National and Regional Level

ACTIVITIES

AMOUNT ALLOCATED (% of total)

CUMULATIVE DISBURSEMENT RATE as of 31/12/2015

CUMULATIVE DISBURSEMENT RATE as of 31/12/2016

AMOUNT SPENT BY AREAS OF ACTIVITIES (% of total) as of 31/12/2017

AfCoP Secretariat activities

2,515,000 (22.70%)

679,616 (27%) 1, 302, 367 (51.8%)

(22.2%)

Project activities at regional

3, 075,000 (27.76%)

2, 490, 854 (81%)

2, 519, 897 (81.95%)

(43%)

~ 19 ~

level

Project activities at national level

5, 488, 000 (49.54%)

1, 590, 773 (30%)

2, 034,381 (37%)

(34.7%)

TOTAL 11, 078,000 4, 761, 243 (43%)

5, 856, 644 (52.9%)

Source: AfCoP audit report, Dec 2016.

3.2.3.Partnership and AfDB Management

48. Governance structure cost-effectiveness: To the knowledge of the Evaluation Team, there is no cost-effectiveness analysis of the different modes of financing and implementation of multinational/regional projects and programs at the Bank, in order to modulate interventions from one country to another, from one activity to another within the same country, taking into account the state of the capacity being developed, and the MfDR institutionalization in this case. The qualitative analysis performed in the PAR on alternative solutions considered only project formulation (not implementation) issues pertaining to the definition of the target beneficiaries (only RMCs, only RECs or both) and the MfDR thematic pillars combination.

49. The PAR (2011:15) argued that “to ensure a quick project take-off and a smooth transition, the Bank (was) satisfied with the recruitment procedures carried out for the AfCoP facilitator and the part-time consultant (that time) working for the World Bank.” Thus, they were selected from a single source for “a continuation of the earlier assignments” while “the Bank’s Field Offices (were to) assist the focal point at the RECs and country levels to carry out the minor procurement activities related to training, and workshops.”

50. While it should be noted that the project funding modalities were determined by the Bank’s fiduciary requirements, the retained project governance architecture and processes resulted in a lack of coordination hierarchy mechanisms and incentives necessary to link the project activities at country with the REC levels, hence to increased likelihood of achievement of the results expected from the project Components 2 and 3:

~ 20 ~

i) While the two (2) consultant coaches per country were selected on merit from a list provided by the government, they did not have sufficient lobbying capacity to secure the country leadership commitment for effective implementation of the national and regional road maps. Tying their consultancy contracts to such an implementation of the roadmap on which they couldn’t have any control was a risk that was characterized by multiple contract extensions and eventual consultant withdrawal from the project.

ii) Because of the informal nature of the national CoPs, that they did not have legal existence and financial management capacities, their activities could be in competition or in conflict with the sector ministries. Organization of the national high-level workshop was instead considered by the PAR (2011: 17) as a mitigation strategy for the risk of weak partnerships of the national CoPs and the governments.

iii)AfDB country offices would have better chances of influencing project delivery at country level than a secretariat based at its headquarters (see Box 2). Thus, decentralization and proper engagement of the field offices could have increased chances of project success, ownership and accountability as well managing relationships between the Banks and RECs.

Box 2: Missed role of AfDB Country Office

Despite the reported paramount contribution of AfDB’s country offices to the success of AfCoP-MfDR Project, this evaluation established that the project could have benefited from clearer roles and responsibilities between the officers in charge of the project at the AfCoP Secretariat and the country offices. Some of the field offices visited during this evaluation claimed that they were not given any significant role besides that of “project cashier” for payments to local service providers on events and/or activities coordinated from headquarters.

In the case of Zambia for instance, the country office was even not aware that COMESA Secretariat was a partner and beneficiary of the project and had a role in coordination of the AfCoP-MfDR Project’s regional activities. Thus, they did not engage with the COMESA Secretariat on the MfDR agenda.

51. Internal and external partnership effectiveness: The efficiency and effectiveness of the AfCoP-MfDR Project

~ 21 ~

implementation framework may also have suffered from the absence of mechanisms for leveraging potential synergies with other AfDB divisions working on MfDR capacity and regional integration (M&E, Statistics, institutional governance, etc.). This conclusion emerged from the Evaluation Team’s discussions with consulted stakeholders and no trace of effective substantial collaboration with these divisions was identified in the reviewed working documents beyond the PAR.

52. Few in-country development partners collaborated with the government and the Bank in this project, despite the fact that several partners are working in different countries on strengthening RBM, including M&E at national and sub national levels. Despite the use of the AfCoP infrastructure (e.g., coach network in Senegal; annual M&E/MfDR week in Kenya) and results (RRI in Madagascar and Zimbabwe), it is not a common knowledge that AfDB is main supporter of the AfCoP-MfDR Project interventions in some countries.

53. Nonetheless, at the national level, development partners are willing to work with AfDB to advance the MfDR capacity development. For instance, in Madagascar and Kenya, the World Bank and/or JICA expressed interest in working with the AfDB to promote the results culture in the country. It will be important for future programming to consider such partnerships at the country and REC levels.

54. The AfDB PCR Mission noted that procurement issues that were raised at the time of the Medium-Term Review were addressed with overall satisfaction. Specifically, it is indicated that “procurement methods provided for in the financing agreement have been respected. A couple of changes were undertaken, including the recruitment of a facilitator on the basis of a short listing approach instead of a direct agreement to replace the resigning consultant who held the position; and the use of direct agreement instead of consultation with suppliers for the organization of some workshops”.

55. In the same vein, it was reported that although procurement of staff for the AfCoP Secretariat has been a challenge, they were able to fill all of the remaining vacant positions by May 2016. Striving to achieve expected results was also evident in the case of mainstreaming gender at both the design and the implementation

~ 22 ~

phase. The PCR Mission observed- and this was confirmed by key informants interviewed by the Evaluation Team- that during the implementation of the project, gender and youth issues were mainstreamed in the activities of the project as planned. It should be also noted that the AfCoP-MfDR Project’s low performance on horizontal management is shared with a good number of projects at the Bank.

3.2.4.Overall Rating for Efficiency

56. The AfCoP-MfDR Project efficiency is rated as moderately satisfactory with an average score of 2.6/4: (3/4) for timeliness, (2/4) for linking resources to results and (3/4) for partnership and Bank management. The absence of an overall integrated demand-driven implementation strategy focused on subsidiarity and complementarity among actors and levels of implementation has been a hindering key factor of the limited achievement of efficiency. The AfCoP-MfDR Project’s institutional arrangements lacked mutual reinforcing steering coordination mechanisms and incentives for leveraging expected synergies between AfDB Headquarters and country offices, national and regional activities, coaches and CoPs and governments. However, on a selective basis, AfCoP Secretariat demonstrated a sense of accountability being informed by lessons learned and recommendations from previous project reviews.

3.3. AfCoP-MfDR PROJECT EFFECTIVENESS AT OUTPUT LEVEL

57. This section presents evaluation findings on achievement of the project objectives as stated in the PAR. The assessment of outputs was based on the output execution ratio (realized and on track/planned targets) and the quality of outputs. The section also provides an assessment of the extent to which project beneficiaries were involved in the decision-making and implementation of the project activities.

3.3.1General Emerging Trends

58. The collected data show that the project was quite effective at the output level. Table 7 is derived from the AfCoP-MfDR activities,

~ 23 ~

outputs and outcomes reported throughout the PCR and its Aide-Memoires (June-July 2017) and various other progress reports. It provides a comparison summary of planned and reported activities & outputs per AfCoP-MfDR Project Component.

59. As illustrated in Table 7, most of the outputs were over achieved. For example, all outputs on knowledge sharing on MfDR (Component 1) were achieved by 100% and above: Online discussion 150%; MfDR knowledge products 114%; MfDR forums 100%; and regional platforms 100%. The same trend was found on Component 2- Linking MfDR knowledge to regional processes, and Component 3- Synergies between regional and national processes. However, while countries achieved over 82% of outputs from implementation of activities related to sensitization the implementation workshops (33%), Mid-term review (41%), and Performance validation (39%) were less achieved or challenging for countries to implement, as these required inputs from the high level workshop (see Table 5).

60. The evaluation findings did not show varied levels of achievement of project outputs between the two regions. However, it was noted that none of the WAEMU countries was able to implement in full the country and REC road map, while in the COMESA region, Madagascar and Zimbabwe were able to implement in full the planned road map plus additional RRI circles.

3.3.2Quality of Outputs

61. The overall quality of the AfCoP-MfDR Project’s outputs was good. The sharing of knowledge on MfDR addressed Africa topical issues and included physical and virtual discussions. The forums increased discussions on MfDR at the regional and country levels. Quality assurance was done through participant feedback for each training, forum and meeting, and collected feedbacks were used to inform future events. Key informants reported that number of universities in Africa have agreed to consider the use of the AfCoP-MfDR Project’s cases studies as training materials. For instance, the Master’s degree new Curriculum for Development Studies of the University of Namibia may include number of those case studies.

62. Expected participation, gender and youth needs considerations were achieved in most project components. The AfCoP members

~ 24 ~

and management committee were involved in selection of topics and themes for online discussions and/or annual forums. Specific youth for results and gender for results thematic groups were formed and physical forum organized. The Knowledge sharing component, for example, ensured gender and youth based topics were discussed at both French and English AfCoP Platform. Specific knowledge products such as case studies, guidelines and policy briefs were developed and widely disseminated.

63. Nevertheless, the evaluation found that the level and quality of achievements were relatively low on Linking MfDR knowledge to regional processes and on Synergies between regional and national processes. It was noted that two regional platforms were established but were inactive. The RECs were unable to mobilize and support the regional CoPs in sharing experiences on MfDR while the linkages between national CoPs and regional CoPs and processes were also inadequate. The regional platforms were developed and managed by ACBF, but ACBF was not involved in the AfCoP-MfDR Project at national and REC levels, nor did it have influence and direct links with country and the RECs road maps.

64. Thus, critical quality challenges noted on various activities within the programmatic components of the AfCoP-MfDR Project are mainly related to the project design shortcomings discussed under previous evaluation criteria. Much of the project achievements could be directly linked with the project management’s high level of commitment at ACBF, AfCoP Secretariat and AfDB’s field offices. Both ACBF and the AfDB assigned their staff to work on the project to fill the staffing gaps for a substantial amount of time. Otherwise, the evaluation noted that country and REC roadmaps were not results-based.1 Ad-hoc activity-based concept notes and budget requests were submitted to Abidjan and some core activities such as the thematic groups and the thematic clusters were not included in the Results Framework. These groups were formed in 2013 and many of their meetings and/or training workshops could be conducted towards the end of the project, as did the development and dissemination of many other knowledge products that leapt from 42% in April 2015 to 93% in Nov. 2016 and to 126% in July 2017.

1 For example, no fixed mile stones to be achieved at certain periods.

~ 25 ~

65. It was also noted that the training events conducted at country levels attracted a good number of people, especially the MfDR and M&E week, but the project’s monitoring framework (PAR, 2011:16) was by design not results-based, hence it was unable to capture adequate performance data on trainings. Capturing how participants to learning events have applied their learning is essential for sound measurement of such a capacity development project effectiveness and sustainability.

~ 26 ~

Table 7: AfCoP-MfDR Project Achievements of Outputs

PROJECT COMPONENT

S

EXPECTED OUTPUTS

(as per PAR)

ACTUAL OUTPUTSPlanned (V) / Not Planned ()

EXECUTION RATE

OUTPUTS QUALITY (V) / OBSERVED GAPS (-)

1. Sharing knowledge on MfDR

30 online discussions on MfDR including, at least 10 gender responsive

57 online discussions were facilitated of which 14 focused on gender issues.

17 online discussions; 30 Policy Briefs based on discussion

on the website developed and disseminated.

190% Selected by AfCoP members, topics focused on various critical issues on Africa development challenges. Policy briefs developed & published from discussion.

Classical approach and strategy to quickly gather AfCoP members around emerging topics.

Multiple channels of dissemination: ACBF Virtual Library, AfCoP events, AfCoP e-letters to 5000+ members and Social Media Channels.

- Partnership with specialized institutions/expert on various topics could have widened the scope of the discussion and pool of change agents.

- Website could be more flexible to allow members to use various technologies.

150 knowledge products including 50 gender-responsive

171 knowledge products, including briefs, case studies guidelines and tools were produced and disseminated in English and French, of which 52 focused on gender issues.

114% Covered topics along the 6 MfDR pillars;

Written by MfDR practitioners from Africa and abroad but focused on African states experiences and lessons;

~ 27 ~

PROJECT COMPONENT

S

EXPECTED OUTPUTS

(as per PAR)


EXECUTION RATE


Twenty-one (21) extra knowledge products were developed and disseminated;

Seven (7) Compendiums of best practices on MfDR;

Lessons learned from the RRIs produced through a comparative study.

Disseminated mainly on AfCoP website and organized in themes for easy access;

Weekly letters to over 5000 AfCoP members with links to these resources helped the dissemination;

African Universities showing interest in using them for teaching/curriculum development.

- Revamped website provides easy access to these resources, but is yet known to some members;

- Loss of information during the migration of the website;

- Products developed and disseminated mostly towards the end of the project: from 42% in April 2015 to 93% in Nov. 2016 and to 126% in July 2017.

3 annual forums organized

Three (3) African forums on MfDR organized on topical subjects.

100% Combination of physical and virtual discussions was fruitful and cost effective in exchanging and eventually up taking knowledge.

2. Linking MfDR

Two (2) regional CoPs

Two (2) regional CoPs established, one in COMESA and one in WAEMU.

100% Remaining budget used to further strengthen the capacity

~ 28 ~

PROJECT COMPONENT

S

EXPECTED OUTPUTS

(as per PAR)


EXECUTION RATE


knowledge to regional processes

created and supported to share good practices on MfDR and policy convergence

Two (2) regional platform established on AfCoP website;

Two (2) learning forums for RECs to share experience on project focused policies.

of the COMESA and WAEMU secretariats as well developing knowledge products on REC policies.

- Knowledge sharing platforms inactive on knowledge sharing by RCoP members;

- RECs did not support the COPs;

- Weak and/or less efforts to collaborate with African union/NEPAD’s Action platform on Development effectiveness.

Readiness assessment methodologies on MfDR and performance standards developed

Assessments conducted at country and regional levels for COMESA and WAEMU.

Kenya second assessment in 2017.

100% Assessment using a standardized methodology;

Readiness assessment followed by a validation meeting at country level and peer review process at regional level.

- Capacity building activities conducted only in WAEMU for 2 thematic clusters;

- Training workshop on trade facilitation couldn’t be hold due to priority conflict;

- COMESA submitted action plan that could not be implemented.

51 trainers of trainers trained on the

74 trainers of trainers (ToT) trained at regional level including staff at both COMESA and WAEMU

145% ToTs included coaches that lead national MfDR agenda;

~ 29 ~

PROJECT COMPONENT

S

EXPECTED OUTPUTS

(as per PAR)


EXECUTION RATE


methodologies carried out

Secretariats. 23 ToT trained.

Six (6) regional thematic clusters for advocacy actions established

Three (3) regional thematic clusters created on (i) trade facilitation, (ii) improving the business climate and (iii) management of public finances in WAEMU;

Three (3) additional regional thematic clusters in COMESA and WAEMU on (iv) Gender for Results, (v) Youth for Results (vi) and Natural Resources Management for Results.

Learning workshops organized to strengthen these thematic groups;

Consultancy services for the coaching of country teams supported.

100% Quality assurance was done through participant feedback for each training, forum and meeting.

Collected feedback was used to inform future events.

- Thematic groups more active on continental level than national levels;

- Not mainstreamed in country action plan/road map, no national chapter but regional;

- No clear selection processes of members to the group;

- Nominations to attend events were often done by Abidjan without consultation with national CoP (members said they used to meet at airport or in Abidjan);

- Same people regularly attend regional events.

3. Synergies between regional and national

Nine (9) national CoPs created and supported to instill results

Existing national CoPs supported. 8 new CoPs established where

needed for a total of 17 national CoPs launched.

188% National COP inclusive of MfDR practitioners from various sectors: CSOs, private sector, academia etc.;

Very active in Kenya and Zambia (leading

~ 30 ~

PROJECT COMPONENT

S

EXPECTED OUTPUTS

(as per PAR)


EXECUTION RATE


processes culture in the regional member countries

implementation of the RRI projects in Zambia).

17 readiness assessments carried out and improvement action plans developed

17 existing member countries conducted MfDR readiness.

MfDR readiness assessment conducted by newest member country, Rwanda.

105% A standardized readiness assessment tool developed and used in conducting country assessments and comparisons;

Weak project governance structure

15,000 people trained or coached on MfDR with at least one third of female (33%) and one-fourth of youth (25%)

15 countries conducted MfDR capacity building and learning events where 9,520 people participated.

Training conducted during MfDR/M&E week in Kenya;

Over 23 government officials retrained and new coaches trained in 2017 in Kenya

Coaches trained on the assessment methodology and rapid results approach

63.4% Trainings much appreciated by members states and individuals.

- Weak M&E system and effort, countries hardly able to indicate total number of people trained.

Seminars for high-level policy-makers to secure buy-in of MfDR organized (680 policy-makers

13 countries held high-level workshops (968 policy-makers trained).

Seven (7) countries launched or conducted at least one cycle of RRIs using the Rapid Results Approach.

54% for high-level workshop;

142%

Inclusive of senior policy makers from various sectors of the government;

Workshop held on time where project focal person was from the Presidency office;

Zimbabwe and Madagascar are

~ 31 ~

PROJECT COMPONENT

S

EXPECTED OUTPUTS

(as per PAR)


EXECUTION RATE


trained) for policy-makers training

case in reference on RRIs.- Project stalled where workshop

was not held on time;- Coaches being independent

consultants could hardly influence the government.

4. Project management

No output set in PAR Results framework

Recruitment of staff in the Secretariat at the Bank and ACBF completed in August 2016;

Ongoing commitments to lead to an estimated total fund utilization rate of 93% compared to actual 87.2% (July 2017);

Both ACBF and AfDB staff work on the project to fill staffing gaps.

N/A Gender and youth issues mainstreamed as planned.

- Project management and staff status and turnover affected timely project delivery;

- Weak steering coordination mechanisms and incentives.

Source: Adapted from PCR and its Aide-Memoire, PCR Mission (July 17-26, 2017) and various progress reports.

~ 32 ~

3.4. AfCoP-MfDR PROJECT OUTCOMES AND POSSIBLE IMPACT

66. Evaluation of effectiveness at outcome level assesses the change brought about throughout the results chain as anticipated from the project theory of change earlier operationalized. Actual, expected and unintended AfCoP-MfDR Project outcomes which came out during the project cycle are tracked at the individual, institutional, and country level (see illustrative Box 3). Particular attention is paid to progress made in gender and youth mainstreaming.

67. It should be noted that at the level of effects, the AfCoP-MfDR Project has little control over the institutionalization of MfDR and its effects both at the country and REC level, although its contribution may be inferred from its theory of change. The project effectiveness at outcome level is basically assessed against immediate and intermediate expected outcomes; the evaluation does not assess the long-term outcome of improved effectiveness in the implementation of the regional policies and impact of regional policy convergence in COMESA and WAEMU. From the PAR’s planning horizon, of which realism sounds doubtful, the latter was expected for 2020.

3.4.1Outcomes at Individual Level

68. As noted above, from the AfCoP Theory of change, along the results chain, outcomes at community/country level should be expected in a relatively long period while outcomes at individual and institutional levels can occur in the short and medium term, respectively. Significant MfDR knowledge acquisition, change of attitude and use of its principles and tools are key indicators of the project immediate/individual outcomes.

69. Key informants of this evaluation demonstrated mastery and ownership of the concepts and utility of MfDR for effective national and regional development management in Africa. They exhibited their capacity to report on results achieved and difficulties encountered in applying MfDR in the areas of AfCoP Pillars within the specific context of their work and/or private life. That is a good indication of use of the knowledge body developed and disseminated by the AfCoP-MfDR Project.

70. The following examples of AfCoP members testimonies are illustrative of increased technical, organizational and behavioral skills in MfDR, including self-awareness, self-confidence, motivation and networking for promoting integrated result-based development management:

i) I wish we had the RRI and MfDR way before, we could have applied the approach to solve many of our development challenges at this unit (Key informant in COMESA);

ii) The project helped me to build up a network of professionals in country and internationally, more than what I managed to develop 10 years ago (Youth for Results key informant in COMESA);

iii) It taught us how the machine of the government works; we appropriated the strategic documents; we learned how to ask the government about the results they promised themselves (Youth for Results key informant in UEMOA; free translation);

iv) All MPs know that things have changed even if they did not follow any training workshop of the project…I have a different look at everything, including managing my salary… I became more critical now and results oriented: identify what has not been done and look for a convincing explanation (Member of Parliament key informant in UEMOA; free translation).

71. Undoubtedly, this evaluation found that raising awareness and increasing the visibility of MfDR is a relatively significant success that underpins outcomes achieved at all levels. However, it should be noted that the key informants above quoted were AfCoP-MfDR Project beneficiaries and they were not necessary randomly selected. The AfCoP-MfDR Project is perceived by some non project beneficiaries as an elitist project that remains relatively unknown outside the CoPs, and it may be difficult to sort out MfDR champions from rent seekers among the participants in AfCoP/Bank’s capacity development activities.

Box 3: AfCoP-MfDR Theory of change at work in Kenya

At individual level , re-tooling and MfDR training to government officials and members of the Kenya Evaluation Society as well as national CoPs were acknowledged. Some respondents had these to say “Resources from the AfCoP platforms enriched my work as M&E”. “The project helped me to build up a network of professionals in country and internationally, more than what I managed to develop in the past 10 years.”

At institutional level , senior government managers, who attended the M&E/MfDR week and the high-level meetings are now demanding results and they also advocate for results instead of outputs. The Kenya School of Government, a training arm of the government, after learning about AfCoP and RRI, have injected these into their curriculum for strategic leadership program (SLDP). The students are showing appreciation and application of the approach. For example, one recent graduate from SLDP wrote a thesis on how to implement RBM in the Kenya Government while two others did the same on application of RRI to the Kenya National Social Security Funds. Several county governments are talking of results and have expressed request for capacity building and needs for M&E systems.

At the country level , the AfCoP-MfDR Project helped to re-energize

the MfDR capacity building and acquisition of results culture and processes at national and county levels. Consulted stakeholders were of the views that it helped in stopping the culture of public officials believing that the country has arrived due to the perceived good performance. They considered it also provided unique opportunity for comparison of the country performances with other countries in the region. The AfCoP-MfDR Project activities have increased visibility and publicity of M&E Department as well as the importance of results culture in Kenya. The MfDR language has gotten into public policy documents, papers, and speeches, as was evidenced during the 2017 Kenya Presidential campaigns, where the focus was on development results at outcome and impact levels and not outputs.

3.4.2Outcomes at Institutional Level

72. At the institutional level, and as intermediate outcomes, the AfCoP-MfDR Project beneficiaries were expected to apply and share learning and practice MfDR within the organizations where they work or they are affiliated or with which they are in business. The institutional MfDR capacity was then to be strengthened by corporate adoption and application of MfDR principles and tools across the organization.

73. It was reported that public officers trained in this project that were team leaders in the first rounds of RRIs became coaches for other ministries. In another instance, acquisitions of results culture and shift from immediate to intermediate outcomes could be validated by the level of zeal and determination shown by staff members as they started implementation of the first circle of RRIs (Zambia and Zimbabwe for examples).

74. MfDR, particularly RRIs, has been applied up to four (4) RRI cycles during the AfCoP-MfDR Project period, in different sectors and different countries. From the implementation of the RRI circles under the AfCoP-MfDR Project and as further steps towards increased MfDR capacity in the sample countries, MfDR high-level declarations, ministerial decrees, national performance management and/or M&E policies and other MfDR tools as well as related policies were developed or are in process of development or implementation. Departments dedicated to monitoring and evaluation have been established in countries. Pilot and good practice structures were established within government clusters: social, economic and governance, albeit MfDR is still relatively less applied on governance than technical sectors.

75. Overall, there is an emerging program results-based management culture and practice although hindered by weak technical capacity (state of MfDR discussed beneath). That includes the promotion of horizontal (whole-of-) government as it was reported that RRI

processes brought together various sectors leaders and officials whom never worked together previously. Also, more and more NSAS (PSOs & CSOs, TFPs) are embracing MfDR and being involved in AfCoP-MfDR Project activities and MfDR development:i) Universities using AfCoP case studies as training materials and

producing other MfDR knowledge products;ii) Media (radio, televisions, newspapers, social media) involved in

sensitization of communities after exposure to AfCoP activities;iii)Commitment by young people, especially from the private sector, to

become part of the CoP, especially to become coaches;iv) World Bank, UNICEF, JICA, etc. using AfCoP-MfDR Project

infrastructure and results or soliciting its collaboration for design and/or implementation of their own MfDR initiatives in Africa.

3.4.3Outcomes at Country and Regional Level

76. Table 8 provides an indicative picture of the evolution of MfDR in the two RECs covered by the AfCoP-MfDR Project, based on three sample countries who had conducted a credible follow-up readiness assessment by August 2017. It shows that overall state of MfDR slightly improved in COMESA and WAEMU member states. In the sample countries, the most implemented MfDR pillars remain Leadership for Results and Planning for Results while the pillars relating to Monitoring & evaluation and Accountability for results maintain the lowest scores. It is worth noting some improvement on the latter in Ivory Coast in disfavor of institutional capacity, the other generally weak pillar.

77. “What gets measured gets done”, said one of the MfDR pioneers, Peter Druker. Unfortunately, without improvement in national M&E, no significant progress in MfDR institutionalization and program implementation could be claimed beyond an early stage of results culture development. There is a tied egg-chicken type of relationship between M&E and accountability for results as no manager would like to be held accountable of non measurable results. In turn, as underpinned by the AfCoP-MfDR Project Theory of change in its quality as a capacity development project, there would be no claim of institutional MfDR capacity without MfDR-competent managers and staff working under results-based human resources management conditions.

78. Nonetheless, this evaluation found that there was an increasing region-wide political visibility and publicity of the importance of MfDR at the country level, particularly: program-based budgeting, M&E and dedicated M&E Departments, Governments using own resources to finance RRI projects and/or other MfDR tools. They constitute early indications of possible upcoming strong national results culture. Country improvement in ranking on easy of doing business was cited as example of AfCoP-MfDR Project impact in terms of more effective development and implementation of policy and program in sample

countries: For example the Registration of new business now takes much less time and it proved to be beneficial to the private sector, business men, women and youth (e.g., Burkina Faso, Ivory Coast, and in Zimbabwe).

Table 8: Status of MfDR in Zimbabwe, Zambia and Ivory Coast

Zimbabwe Zambia Ivory Coast

MfDR Pillar(s) 2013

2017 ESTIMATES

2013

2017 ESTIMATES

2013 2017 ESTIMATES

Leadership for results

2.4

3.0 2.5 3.0 3.0 3.5

Planning for results

2.4

4.0 2.9 3.2 3.0 3.8

Budgeting for results

2.3

3.5 2.5 3.0 3.5 4.0

Institutional capacity

3.5

4.0 3.1 3.5 2.2 3.0

Monitoring and evaluation

1.8

3.0 2.1 2.5 2.1 2.9

Accountability and partnership for Results

1.6

2.5 3.0 3.2 2.6 3.2

Average 2.3

3.3 2.7 3.0 2.7 3.4

Source: Afr4R, Feb 2015 and Afr4R assessment (2017 for Ivory Coast) and AfCoP evaluation field data.

79. At the REC level, performance on MfDR for regional integration was low and remained low. Nevertheless, MfDR culture seems to be slowly set up at WAEMU and COMESA, at least in discourse and institutional working documents, although the result culture is far from being spread out across the Commission and Secretariat (see Box 4). For instance, the WAEMU Commission is:i) Striving to overcome difficulties in implementing union directive on

program-based budgeting since 2009, both at country and Commission level (a priority for the new Chairperson);

ii) Working on operationalization and strengthening of national M&E systems;

iii) Producing a report on state of union trade policy since the Commission results-based reorganization of 2015;

iv) Allocating own resources to the implementation of the AfCoP Project Action Plan.

80. However, even at WAEMU Commission, the result culture is far from being spread out across the Commission, where the regional CoP exclude key actors from its membership, such as the senior officers in charge of M&E, capacity building and NSAs of the union. While the Workshop on Trade facilitation scheduled on the AfCoP regional action plan couldn’t be implemented in the absence of the responsible CoP member and presence of priority conflict, parallel initiatives for MfDR capacity strengthening both at the Commission and the country level were underway in partnership with other donors, including a program development of national M&E systems and an evaluation of RBM at the Commission. Consequently, possible impact of the AfCoP-MfDR Project activities on regional integration is likely to be blocked by such a bottleneck into the project outcomes chain.

Box 4: State of MfDR in COMESA Secretariat

According to the 2014 readiness assessment of COMESA secretariat, the MfDR overall performance was 2.8 out of 5 points. The institution's strengths were on Planning for Results (3.3), Performance Information Systems and Technology (3.0), and on Institution Management and Infrastructure (3.0). Weak MfDR pillars were the Program-based Budgeting and Financial Management (2.2), Monitoring & Evaluation (2.5), and Leadership for Results (2.6).

This evaluation also established that some modest improvement were made on information management system, which is in place but member states are yet to use it, because it is not yet endorsed by the Council. M&E policy was developed but not yet adopted and operationalized. The new institutional structure approved in 2014 has M&E as department but no staff has been recruited because of financial implications. That also affected the overall AfCoP-MfDR Project implementation since the M&E unit was the project focal at COMESA. M&E staffs are recruited on specific donor financed projects with two (2) staffs at the time of the visit for EU & USAID projects.

Although COMESA has developed result oriented 2016 -2020 medium term strategic plan where annual work plans and budge are dawn from, the planning for results remained weak as the Council’s decision comes first without necessary being informed by expected and/or achieved results.

81. In sum, indications of possible upcoming positive outcomes towards the development of a strong results culture in Africa were reported, but it is too early for confirmed long-term outcomes and impact, as many of the outputs were produced in the last year. Acceptable MfDR-consistent long, medium and short-term development plans, strategies/policies are often rewritten without really having been implemented and evaluated for learning and improvement purposes. Although COMESA and WAEMU and their member states have

developed result-oriented medium term strategic plans, where, annual work plans and budget are drawn from, evidence-based development, institutional management nd decision-making remain weak.

82. The shift from the achieved sensitization to institutionalization of MfDR principles and tools in Africa requires considerable resources and innovative means to expose stakeholders constantly to appropriate incentives. The AfCoP-MfDR Project alone could not and cannot be expected to make it happen.

Box 5: RRI as an effective post-crisis transition tool

The RRI has been used to transform service delivery in several countries in Africa. Past experiences as well as findings of this evaluation show that it works well in getting critical services back to people after the effects of a crisis. It also has a catalytic effect, crystallizing the crucial role of incentives in MfDR development. Below are key examples of various crisis contexts of Burundi, Kenya, Madagascar, Rwanda and Zimbabwe.

Burundi is considered as one champion of RRI implementation of the last decade in Africa. The first RRI pilots in Burundi were launched in October 2006 by the first post-civil war government. In July 2009, a joint decree of the two Vice-presidents institutionalized and decentralized the RRA as a tool for implementing the government program. In December 2012, one member of this Evaluation Team developed a national performance-based management policy and its guide and tools including M&E. The recent AfCoP RRI comparative study reports that under the AfCoP-MfDR PROJECT, it is in the backdrop of a political and security crisis that the first cycle of RRI was launched on 24 May 2016 in Bujumbura. “On average, Burundi has recorded a rate of achievement of results of 92.03% for the eleven (11) entities involved” (p.12).

Kenya introduced RRI for the first time and did very well during the Kibaki’s Presidency administration (2002-2013) that was set for reviving and turning around country after years of stagnation and economic mismanagement. In 2007, the head of Public service instructed all ministries and local authorities to apply RRI (AfCoP RRI comparative study, 2017: 33). Currently, the Kenya devolution program called for the county governments to provide public services to their communities, with the weak capacities, there is a danger of interruption of, and compromised quality of social services. Hence, interest of the counties and partners on MfDR capacity building in Kenya county governments.

Madagascar : The AfCoP-MfDR Project supported RRIs were not new. Madagascar successfully used RRA to turn around a declining economy and an agricultural sector that had been devastated by two powerful cyclones in 2004 (Friedman 2012). When success occurred in one district, other districts launched similar initiatives successfully. During the AfCoP-MfDR Project, the RRI were implemented within the framework of reconstructing the public administration and the country from the political crisis of 2009-2013, by all 32 ministries instead of three (3) initially planned.

Rwanda is a reference of successful RBM implementation in Africa with its institutionalized performance contracts, Imihigo. In the aftermath of the genocide, Rwanda used RRI to hasten behaviour change within the public service and economic recovery for political and social stability. It learnt about the RRA from Kenya and Madagascar where it had been applied. It was first applied in public health services and local government.

Zimbabwe has been isolated by main donors and investors for over 15 years, and this negatively affected its economy. The AfCoP-MfDR Project provided opportunity to do more with limited resource and show results. It was therefore diligently implemented and due to effectiveness of the RRI interventions, the government of Zimbabwe adopted the approach and implemented over 50 circles of RRIs in various sectors such as agriculture, transport, trade and industry.

3.4.4Unintended Outcomes

83. Table 9 presents the main unanticipated or additional outputs/outcomes and qualitatively estimated magnitude of their impact on the project. They were mainly identified at the time of the PCR Mission and were confirmed by the evaluation field work. It is worth noting that they are all related with RRIs and that a good number of them could have been expected from a sound project theory of change and risk analysis.

84. Successful implementation of RRIs was likely to be effective: (a) in response to explicit demand and personal implication of the Chief of state, along with a reward and sanction system; and, (b) where RRIs have already been used in the aftermath of natural or sociopolitical crisis (see Box 5). The Evaluation Team heard about some well-known drawbacks of RBM implementation in general and particularly RRI in the context of post-crisis fragility, whereby they could be disconnected from the strategic plan they were geared to implement. They could be selectively applied on easy-to-achieve and to measure objectives and activities for the needed political visibility.

3.4.5Overall Rating for Achievement of Outcomes

85. The AfCoP-MfDR Project overall effectiveness is satisfactory (3/4). Progress towards achievement of the objective of strengthened MfDR capacity in COMESA and WAEMU and their member states is significant, particularly at the individual level and institution level albeit at a lesser extent.

86. The project will have contributed to improving the results-culture in most of the countries covered by the project. However, countries and REC’s secretariats remain stalled at the stage of planning for results in the absence of sufficient ownership and a deliberate strategy to institutionalize MfDR, including through a framework of both academic and continuous training and a system of both positive and negative sanctions for results achieved.

87. Limitations in the number and high turnover of staff as well as a weak business governance structure have had negative repercussions for project implementation and eventually for its effectiveness. Particularly, the regional integration aspect of the project was less effective, due to project design and implementation institutional arrangements.

Table 9: Unanticipated or Additional Outputs/Outcomes

DESCRIPTION POSITIVE POTENTIAL

OR NEGATIVE

IMPACT ON THE PROJECT

High demand and adoption or re-adoption of RRIs as an implementation tool

Positive High

Training of inter-ministerial coaches to undertake RRIs for continuity and sustainability

Positive High

Utilization of MfDR case studies by researchers and universities as training materials

Positive High

Other development partners (World Bank, UNICEF, EU, JICA) supporting and/or using the RRI due to its successful implementation and emerging impact

Positive Medium

Need for multiple extension of coaches’ contracts

Negative High

Misunderstanding of the RRI agenda caused competition between the ministries against the RRI program

Negative Low

High pressure to deliver/show results in 100 days caused cheating and/or shoddy work in some sector/department

Negative High

Source: Adapted from Aide-Memoire, PCR Mission (July 17-26, 2017) & Field work.

3.5. PROJECT SUSTAINABILITY

88. At the Bank, evaluation of project sustainability assesses the extent to which the project has addressed risks during implementation and put in place mechanisms to ensure the continued flow of benefits after completion. It should also evaluate risks to the sustainability of development outcomes and/or the project’s benefits, including the resilience to exogenous factors. The overall rating of the sustainability outcome is the mean of the rating of the following four criteria: i) technical sustainability; ii) financial sustainability; iii) institutional sustainability and strengthening of capacities; iv) ownership and sustainability of partnerships and v) environmental and social sustainability.

3.5.1Technical and Financial Sustainability

89. Some countries have used own resources for implementation of RRIs and/or they are engaged in mainstreaming MfDR processes into their national budgets (e.g., Ivory Coast, Senegal, Kenya, Malawi, Zimbabwe, Niger). However, some others and the REC secretariats themselves are still recruiting M&E staffs on specific donor financed projects. In many places, M&E is still a donor business whereby it is used for justification of (continued) aid. It was reported earlier in this evaluation that some project beneficiaries have been waiting for the Bank’s financial support for implementation of the action plan that resulted from their MfDR readiness assessment.

90. With respect to technical sustainability, maintaining the AfCoP website will serve as a means of sustaining the achieved results. Should the AfCoP Secretariat be transferred to an independent organization- whatever would be the name of the new entity- the new AfCoP implementation agency could be in a better position to mobilize more resources for the long run, through a partnership-based program delivery model.

3.5.2Institutional Sustainability and Strengthening of Capacities

91. A good number of institutional mechanisms to ensure the sustainability of the AfCoP-MfDR Project results after completion are in place:

i) Availability of designated institutions to coordinate implementation of M&E at country level up to standalone ministries;

ii) The movement of devolution (“déconcentration”) program calls for a lot of support and capacity building of the sub national governments in overall management and specifically on MfDR in order for them to be able to deliver public services directly to the citizenry, and fulfill their new mandates;

iii)Members of the AfCoP have agreed during 2017 Annual forum to formalize the community into an African-wide institution advocating for development results culture. The formalization process is underway and options for hosting the successor to the current AfCoP Secretariat are under consideration.

92. However, although governments are willing to continue the activities conducted under the AfCoP-MfDR Project, especially the RRIs, this evaluation found no national or regional comprehensive strategy for creation of national/regional pools of MfDR leaders, as suggested by the project theory of change, through for instance, a combination of academic and vocational training. Such an integrated strategy entails as well, a mechanism to follow-up on post-participation career path of

trained staff in order to ensure they evolve in enabling positions. Consulted stakeholders were of the view that institutionalization of MfDR implies that it has to be perceived as a set of management principles and techniques that should be learned early at school, not just through adult on-job training itinerant workshops sponsored by donors.

3.5.3Ownership and Sustainability of Partnerships

93. The AfCoP-MfDR Project demand-driven delivery model was appropriate for ensuring ownership of the project objectives, activities, outputs and outcomes. Indeed, as underscored by the PCR and this evaluation, most governments have recognized the need to prioritize support to improvement of a results culture in their respective countries. The national development plans are increasingly results oriented. Particularly, governments have demonstrated ownership of the AfCoP-MfDR Project's outcomes by agreeing to undertake a series of RRIs which have been adopted as a means of developing the results culture in Africa.

94. Where RRI provided visibility for the government and senior officials, acceptance and appreciation of the MfDR approach and tools is higher than the rest, including a high demand for multi-stakeholder partnerships for further MfDR capacity development. Therefore, it could be expected that the needs to deliver critical public services to the population with limited resources, supportive leadership along with the needs to show off state capacity as well as supportive partnerships are likely to underpin the sustainability of the AfCoP-MfDR Project results.

95. However, as also noted by the PCR and this evaluation, although governments, including those with a 100% rate of fund utilization, are willing to continue the activities conducted under the project, especially the RRIs, they still expect support from the Bank in carrying these activities. No mechanisms were found in place to avoid that the RRIs and the entire MfDR remain a donor business primarily used for aid and/or rent seeking.

3.5.4Environmental and Social Sustainability

96. As a capacity building project, the AfCoP-MfDR Project was considered a category 3 project at appraisal, with no adverse environmental impacts expected, given that no materials or equipment was purchased. With respect to social sustainability, this evaluation noted that gender and youth issues were mainstreamed in the

activities of the project at both the design and implementation, and the associated expected outputs were achieved under the entire AfCoP-MfDR Project Components.

3.5.5Overall Rating for Sustainability

97. From the above discussion, the AfCoP-MfDR Project sustainability is rated as moderately satisfactory with an average score of 2.75/4: (3/4) for Technical and Financial Sustainability, (2/4) for Institutional Sustainability and Strengthening of Capacities, (2/4) for Ownership and Sustainability of Partnerships, and (4/4) for Environmental and Social Sustainability.

98. As underlined by the PCR and this evaluation, demand from RMCs and RECs to receive support in mainstreaming MfDR into their development processes emphasized the need to sustain AfCoP-MfDR Project activities. Project sustainability and extension beyond the current RMCs and RECs will depend on the capacity of the AfCoP community to raise enough resources to act upon a new designed governance structure, as it includes a legal status of the AfCoP, within the framework of a next phase of the AfCoP-MfDR Project. The process is underway and not clear what would be the outcome.

SECTION IV: LESSONS LEARNED AND RECOMMENDATIONS

4.1. LESSONS LEARNED

99. This section describes findings from the evaluation on what worked and why (and what didn’t work and why) and the implications for project design, planning, implementation and M&E. From this evaluation, the lessons from the performance of the AfCoP-MfDR Project are categorized into two (2) areas: 1) Project design and implementation, and (2) Country and RECs implementation of the road map.

4.1.1Project Design and Implementation

100. Project governance structure design should factor efficiency and effectiveness of procurement process. At the Bank, the AfCoP-MfDR Project management involved two separate departments: AHHD- Human Capital, Youth and Skills Development Department- for supervision and SNDR- Delivery, Performance Management and Results (former ORQR)- for implementation. Additionally, the AfCoP-MfDR Project was implemented by both ACBF and a secretariat at the

Bank headquarters. Since the Bank’s procurement procedures guided all the project procurement, this arrangement complicated the procurement processes and caused delays. These procurement processes also caused unnecessary delays in recruitment as ACBF had to first apply its own procurement procedures before it submits outputs to the Bank for clearance and/or no objection.

101. Securing commitment of the implementing agents is critical for enhanced project effectiveness. Both ACBF and the Bank subsidized the project through allocating their staff members to work on the project. This helped the project to achieve much of the intended results. Moreover, ACBF showed commitment by pre-financing the project when disbursements were delayed. Having ACBF as implementing partner was also advantageous due to its capacity and experience in knowledge management and Africa wide coverage. The effective coordination by the Bank and ACBF made it possible for the project kick-off to be effective within 3 months.

102. Cost-effectiveness of staffing Bank’s projects with consultants should be established, not assumed. The recruitment of project staff as long-term in-house consultants did not favor AfCoP-MfDR Project’s performance. It increased the rate of staff turn-over which adversely impacted on the project efficiency. Moreover, coach consultancy contracts with the Bank aligned with the country road map led to multiple contract extensions and eventual self-withdrawal of some coaches.

103. A robust M&E system is indispensable to ensure effective project coordination. The AfCoP-MfDR Project’s M&E system was not robust enough to capture all project results. Should it have worked as planned, the virtual approach to the project supervision and coordination (Monitoring schedule of PAR 2011:16) could not follow-up on the change being brought about throughout the results chain as anticipated from the project theory of change. A more robust M&E system should have been in place to ensure effective results-based coordination and a possible stronger effectiveness evaluation, as well as tracking and documentation of the project’s work and experiences for learning and innovation. A strategy to track the effectiveness and uptake of the knowledge sharing component could have served as a starting point for better demonstration of progress towards achievement of the AfCoP-MfDR Project’s expected results.

104. Management response to formative evaluation is a critical tool of managing for results. A management response to the AfCoP-MfDR Project Mid-term Review findings and recommendations may have contributed to improvement of the project performance. Specifically, the recommended modifications to the Results framework indicators,

outputs, and outcomes and/or reassignment of resources could have helped in bettering the project overall implementation efficiency and effectiveness as well as the quality of results achieved.

4.1.2Country and REC implementation of the Road Map

105. Moving from the stage of diagnostic and planning for results to MfDR implementation remains a challenge for both the RMCs and RECs secretariats. Moving from MfDR readiness diagnostic to implementation of the derived action plan was a challenge for both the RMCs and the RECs secretariats. The following factors inhibited AfCoP-MfDR Project outcomes:a) Lack of flexibility of the action plan;b) High turnover of focal points and/or coaches;c) Lack of high-level leadership commitment and rewards system

(positive and negative sanctions) confirmed beyond government branding show cases;

d) Lack of technical skills that the project’s training workshops couldn’t secure.

106. Better inter-organizational collaboration within the Bank and in countries is needed for effective MfDR capacity development project. Effective collaboration within the Bank departments implementing complementary projects and between the Bank and other development partners in countries is crucial for effective MfDR capacity development project. The evaluation findings reveal that mainstreaming the results culture into development management in RMCs and RECs as well as designing and implementing a project to promote it is a complex undertaking. It cannot be left to one promoter or implementation agency. Moreover, as above discussed under assessment of ownership and sustainability of partnerships, continued parallel MfDR capacity development initiatives are likely to undermine the MfDR capacity development through reinforcement of the negative image of MfDR as a donor business primarily used for aid justification and/or rent seeking.

107. M&E-driven institutional arrangements are an indicator of strong highest leadership engagement, hence a key enabling factor for MfDR institutionalization. Having a designated ministry/department for national planning and M&E is an advantage in the effort to advocate for results culture and institutionalize MfDR. Moreover, where the project is hosted it is also critically important for it to achieve results. The project implementation was smooth and level of achievements higher where it is housed in the Office of Cabinet or the Presidency.

108. Promoting regional integration through Bank’s multinational capacity development projects is in need of evidence basis.

According to its Results framework, the AfCoP-MfDR Project was intended to influence regional integration, up to regional policy convergence, through implementation of MfDR action plan at RMCs and RECs levels. As it is the case in many other Bank’s multinational capacity development projects, the linkages considered to that end did not work due to difficulties in ensuring that activities at the national level are mainstreamed into regional activities and the same way activities at the regional are mainstreamed into the national ones. An optimal combination of different coordination mechanism types has to be found out in search of more effective project governance structures.

109. A critical mass of national/regional coaches and MfDR-skilled human resources is determinant for MfDR capacity development. Having only two national coaches to lead the road map process as independent consultants didn’t work as expected and in some cases negatively affected the project delivery rates. At the opposite, the evaluation found that the number of conducted RRIs cycles was higher where public officers trained in the AfCoP-MfDR Project that were team leaders in the first rounds of RRIs became coaches for other ministries. A program to train national coaches from various ministries, and hence to create a critical mass of MfDR skilled people, should have been in place as way to ensuring sustainability and institutionalization of MfDR.

National political or natural disaster-related110. National political or natural disaster-related crisis seems to be

at the same time a barrier and enabling factor for adoption of MfDR, particularly RRIs. While the implementation of the MfDR could be held on during the crisis, it is also likely to be revamped and accelerated in the post-crisis period. Under the AfCoP-MfDR Project, RRIs were intensively used with relatively more success in the aftermath of a national crisis in current RRIs champion countries after a temporarily interruption of their application (See examples in Box 5).

4.2. OVERALL CONCLUSION AND RECOMMENDATIONS

111. The AfCoP-MfDR Project as a RMC and REC demand-driven venture has proven to be relevant in advancing MfDR and overall result culture in the region. It was implemented as a pilot project in 18 RMCs and two RECs, where the evaluation found early indications of positive outcomes, both at the individual and institution level. Achieved immediate and intermediate outcomes do point to possible realization of the long-term project objective: improved development and implementation of policies, programs and projects in Africa through strengthened MfDR capacity. Therefore, there is need for the Bank and its partners to sustain it and more importantly to make it work even better by taking into account lessons and knowledge gained from the pilot phase.

112. Best practices from the implementation of the AfCoP-MfDR Project include among others, those found in Madagascar where the national coaches trained ministerial coaches to lead RRI projects in their respective sector. This proved to be useful in meeting the country needs of coaches and increasing MfDR skills across the Government. Another best practice was found from Zimbabwe experience, where RRI leaders that mastered the MfDR approach during RRI of their ministries were used as coaches to lead implementation of RRIs in other sectors using government own resources. This approach shows the importance of high level country commitment to the agenda and also the need to institutionalize the coaching approach.

113. Based on the above lessons learned of this evaluation, the following are recommended changes, improvement or programmatic issues of a possible next phase of the AfCoP-MfDR Project or similar approaches to MfDR capacity development in Africa:

A. Ensure high project quality at entry, particularly for regional focus. The current AfCoP-MfDR Project needed a clear theory of change and achievable objectives for the stated project duration. The Bank should consider developing an outcome-based results framework for the successor of the current project, relying at least in part on the immediate and intermediate outcomes used in this evaluation. Particularly, the Bank could consider conducting full technical ex-ante (and ex-post) cost-effectiveness analysis for the design and implementation of its multinational projects/programs.

B. Establish clear results-based roles for Bank’s field office and local partners, and ensure synergy with other relevant Bank’s projects. The Bank could consider further clarifying and strengthening the mandate of the Bank’s field offices, and ensure they develop and nourish local partnership for advancing MfDR agenda at national level, more so to leverage on resources and ensure effective coordination in the country development efforts. It is also important for the AfDB to ensure synergy with similar projects within the Bank, especially the evaluation and statistics departments given the recognized driving role of M&E in MfDR capacity development. Complementarity-based synergies are also likely to be achieved with the governance, knowledge, natural resources, gender, youth and regional integration departments.

C. Sustain the AfCoP-MfDR Project gains and the MfDR momentum at RMC and REC levels. The Bank should consider continued support to the AfCoP-MfDR Project and scale up the coverage to more sub regions and to the remaining government

entities at both the national and sub national level2within the current RCMs and RECs. In doing so, the Bank should take into account the lessons learned and knowledge gathered during the implementation of the pilot phase (2012-2017). Specifically, the following priority results areas for a next phase of the AfCoP-MfDR Project emerged from the evaluation field work:

(i) Capacity strengthening of people and institutions;(ii) Internalization of rapid results initiatives, i.e., strengthening RRI

as a means for MfDR development;(iii) Knowledge and experience sharing; (iv) Policy dialogue and advocacy, including for stronger leadership

and accountability for results.

D. Consider single and independent institution approach for managing the project along with a formal legal status of the AfCoP. In order to avoid the complexity experienced on procurement, M&E, supervision and overall management of the project, the Bank could consider expanded partnerships with a regional institution that has a regional coverage and experience of working with the RMCs and the RECs.

E. Consider enhancing the project demand-driven delivery model by a differentiated approach at different levels: product/service, country/REC, MfDR Pillar, implementation agency, etc. in order to factor the differential level and needs of MfDR development in countries. Such a differentiated approach should also be applied for suggested extended coverage of the project to all government entities at both the national and sub national level, as well as to key PSOs and CSOs. Tailored RRIs should be considered for overall capacity rebuilding in post-conflict and fragile states.

F. Consider fostering the formulation and implementation of national and/or regional comprehensive strategy for creation of national/regional pools of MfDR leaders through a combination of academic and vocational training, results-based rewards system and institutionalization of the national coaches’ program. Such an integrated strategy entails a mechanism to follow-up on post-participation career path of trained staff in order to ensure they evolve in enabling positions. It implies also an extension of implementation partnerships to higher education, research and innovation institutions, with emphasis on public and private management development institutions, business and public management schools.

2 It should be noted that such an extension have started under the current phase, including for instance (but not limited to) the admission of Rwanda in 2016.

To establish a critical mass of MfDR coaches Africa-wide, the Bank should consider institutionalization of the national coaches’ program by building capacities of these local institutions to conduct such training sustainably. The training program should include the Master Trainer/Trainer of Trainers approach.

ANNEX I- EVALUATION MATRIX

CRITERIA EVALUATION QUESTION (EQ)

JUDGMENT CRITERIA DATA COLLECTION

Relevance EQ1. To what extent is AfCoP-MfDR Project relevant in improving the results-culture in Regional Member Countries (RMCs) and Regional Economic Communities (RECs) using MfDR principles and tools?

EQ-1.1: The extent to which the AfCoP-MfDR Project pertained to (is aligned with) the African, RECs and national priorities and the requirements of the target group(s)?

EQ 1.2. To what extent are the objectives of the project still valid?

Key informant interviews – AfCoP managers at Secretariat, ACBF, the RECs and RMCs

Interview with private sector at national and REC levels (chambers of commerce, COMESA & WAEMU business council, KEPSA, ZPSA)

Desk reviews: Project document, RECs strategic

plans, RMCs national strategic plans, regional & global policies and priorities (SDGs, high fives, Agenda 2063),

Periodic project progress and activity reports

EQ2.To what extent were partners involved in the development and implementation of the AfCoP-MfDR Project and the Africa for result initiative?

EQ 2: Quantity & quality of participation by stakeholders in project design and activities

EQ3.Were the objective, activities and outputs of the project consistent with the intended key results?

EQ3. Quality of the project Result Framework and log frame

Desk review /assessment of the project Result Framework and log frame

EQ4.Is the project relevance in increasing management efficiency in the in RMCs and RECs

EQ4: Evidence of increased capacity for MFDR as well as policy and planning, delivery, etc.

Desk reviews of annual and end of project reports

Interviews with key informants (RECs, COPs & project secretariat at AfDB)

Focused Group Discussions (FGD)

with the national/regional COPs and coaches

Effectiveness & efficiency

EQ5: Did the project achieve its stated objectives, planned outputs, outcomes, impacts and how satisfactory was the achievement?

EQ6: what are the key factors contributed to/hindered effectiveness of the AfCoP-MfDR Project?

EQ6.1 To what extent has the Bank successfully implemented a performance management strategy that focuses on performance and achievement of results

EQ5: Contributions (actual and expected) by the AfCoP-MfDR Project interventions to specific measurable benefits; extent to AfCoP interventions contribute to expected outcomes; extent to which unintended consequences (positive or negative) different from the operation were recorded after completion of the project.

EQ6: Extent to which the policy and political environment of RMCs/RECs affected or is likely to affect the outcomes of AfCoP-MfDR Project interventions; extent to which other external factors contributed to or can be likely to contribute to the achievement and non-achievement of project results.

EQ6.1. Contribution by Bank’s supervision to achieving the expected results. Were the results of the project strategy being monitored and managed?

Desk review to assess project result achievements against the planned targets in the project appraisal reports and the reported progress – annually and the project completion report, country portfolio reviews, and the project’s aide memoires

Key informant interviews of officials (RECs, ACBF & AfCoP secretariat as well as the project coordinator)

FGD with national/regional COPs & coaches

EQ7.How adequately and efficiency was the use of project resources (human and financial) in relation to planned activities,

EQ7. The extent to which delivery was undertaken by the most cost-efficient means:

Were activities cost-efficient (outputs/disbursement ratio)?

Desk review to assess project result achievements against the planned targets in the project appraisal reports and the reported progress – annually and the project completion report, country portfolio reviews,

outputs and outcomes? Were outputs achieved on time? Was the project implemented in the

most efficient way compared to alternatives?

How effective and efficient was the implementation methods used in the course of implementing the project?

What were the methodologies and were the methodologies cost effective?

Did the approach achieve the intended results?

and the project’s aide memoires Key informant interviews of experts

and officials at RECs, RMS, ACBF and AfDB, Country couches/project focal persons

Project outcomes and possible impact

EQ8. What changes brought in the lives of people in RMCs and RECs as a result of the AfCoP – MfDR project?

EQ9.What are the positive and negative changes produced by the AfCoP-MfDR Project (directly or indirectly, intended or unintended outcomes)?

EQ8. Performance indicators of outcomes as per the project Result Framework:

Changes in the way of conducting public affairs;

Empowerment of key actors in the decision-making process in the targeted beneficiaries

Improved leadership and accountability in the use of MfDR principles and tools

Status of the 5 MfDR pillars (compared to the readiness assessment conducted at the beginning of the project)

EQ9.Expected and unexpected results of the project:

What difference has the project activities made to the beneficiaries (RECs, and RMS)?

Desk review of progress reports, the mid tern evaluation report, back to office reports and end of project report to assess achievement of outcome and impact results

Interview with project staff at country, REC officials and secretariat levels

How many people have been affected?

How did impact differ across key target groups, including vulnerability categories such as gender, youth, age etc?

What were the transformational results achieved by the AfCoP-MfDR Project?

Project Sustainability

EQ10: To what extent have the AfCoP-MfDR Project achieved results continued or are likely to continue beyond the Bank’s interventions?

EQ10. Effective identification of factors facilitating/constraining the performance of the AfCoP-MfDR Project; contributions to securing RMCs’ financial and human resources for MfDR interventions; contributions to establishing or reinforcing autonomous agencies in charge of programming and managing MfDR and the AfCoP agenda:

The degree to which outputs and outcomes led to development of institutional framework (policy, laws, organizations &procedures)

Implementation of measures to ensure financial sustainability in government & REC budgets or cost recovery mechanisms

Implementation of measures to ensure human resource skills and retention

Observable changes in attitudes, beliefs and behaviors towards MfDR/acquired result culture

Key informant interviews – COPs, project staff, REC officials, other experts, and project coordinator

Review and analysis of literature, internal project documents (including Mid Term Reviews) and regional reviews and analyses that address progress and problems, risks, constraints for the project and mitigations

Potential for expansion or replication and sustainability of governance structure

Effective permanent/long-term engagement and ownership of the stakeholder network

Lessons learned, conclusions and recommendations

EQ11. What are the lessons, and recommendations from the AfCoP-MfDR Project?

What are the best practices from implementation of the AfCoP-MfDR Project and for whom?

What are the lessons learned and recommendation to improve design of future AfCoP-MfDR Project?

What are the lessons learned and recommendations in implementation of other projects?

What are the lessons learned and recommendation for scale up of the AfCoP-MfDR Project?

EQ12.What is the recommended changes, improvement or future programmatic issues of similar approaches?

EQ11.Lessons learned and recommendations

Factors underpinning effectiveness and efficiency of project design, implementation and best practices

Comparative advantages and constraints of the AfCoP model for building capacity in different context of key AfCoP beneficiaries and partners

EQ12. Programmatic recommended approaches for improvement of similar program in the future

Key informant interviews: project staff, REC officials, implementing entities and project coordinator as well as other MfDR team at AfDB

FGD with national/regional COPs Desk reviews of monitoring reports,

disbursement schedule, minutes of meetings

ANNEX II-LIST OF CONSULTED STAKEHOLDERS

BURKINA FASO

Name/Surname Title/Organization

Oulimata Faye Professionnel Economiste, Centre de Programmation Stratégique de Recherche et de Veille, Commission de l’UÉMOA

Alain S. Pykbougoum Cadre Supérieur, Chargé des Questions Douanières, Département du Marché Régional, du Commerce, de la Concurrence et de la Coopération, Commission de l’UÉMOA

Tiemoko N’Guessan Lucie Rose Epse Kabran

Chargée des Questions Commerciales, Commission de l’UÉMOA

Dr. Birane Thiam Secrétaire Administratif Permanent du Conseil du Travail et du Dialogue Social, Commission de l’UÉMOA

Halidou Sawani Chargé des Finances Publiques, Commission de l’UÉMOA

Comlan François-Xavier Degbevi

Chef de la Division Programmation Budgétaire, Centre de Programmation Stratégique, de Recherche et de la Veille, Commission de l’UÉMOA

Marou Sawadogo Chargé de l’Entreprise, Département du Développement de l’Entreprise, de l’Énergie, des Télécommunications et du Tourisme, Commission de l’UÉMOA

Clément Sékongo Ouollo Coordinateur de la Cellule d’Évaluation, Commission de l’UÉMOA

Mamadou L. Sylla Coordinateur, Programme Intégré de Renforcement des Capacités- Présidence, Commission de l’UÉMOA

Dr. Seglaro Abel Somé Secrétaire Général, Ministère de l’Économie, des Finances et du Développement

Lin Hien Chef du Service de la Mobilisation des Financements, Trésor Public, Ministère de l’Économie, des Finances et du Développement

Facinet Sylla Économiste, Burkina Faso & Niger, Groupe de la

Banque Africaine de Développement

Alfred Régis Ouedraogo Spécialiste en Développement Social, Bureau National du Burkina Faso, Groupe de la Banque Africaine de Développement

Nab Léonard Dabire Manager Associé, Projet expertise jeune, World Partner Sarl,

Représentant, Réseau Burkinabé de Suivi-Évaluation

CÔTE D’IVOIRE


Nahoua Yeo Directeur du Cabinet, Ministère du Plan et du Développement

Niamien Kadjo Administrateur des Services Financiers, Direction de la Reforme Budgétaire et Modernisation de la Gestion Publique

N’Dri Kouakou Directeur du partenariat, de l’accès aux financements et aux marchés, Ministère du Commerce, de l’Artisanat et de la Promotion de PME

Traore Idi Idrissa Député de l’Assemblée Nationale, Questeur

Sangare Yacouba Député de l’Assemblée Nationale, Membre de la Commission de Recherche, de la Science, de la Technologie et de l’Environnement

N’Dia Youssouf Conseiller Technique, Ministère du Plan et du Développement

Abbas Sanoussi Démographe en Chef, Directeur de l’Observatoire de l’Emploi et des Métiers, Ministère de l’Emploi et de la Protection Sociale

(Nom à retrouver) Représentant Jeunes pour les Résultats (Youth for Results)

(Nom à retrouver) Représentant Jeunes pour les Résultats (Youth for Results)

Victoria Chisala Chef de Division de la Performance et de la Responsabilité Institutionnelle, Département

Prestation de Service, Gestion de la Performance & Résultat, Banque Africaine de Développement

Pierre Justin Kouka Manager, AfCoP-MFDR Project Secretariat, African Development Bank

Patience U. Ekoh Senior Education Analyst, AHHD.1- Human Capital, Youth and Skills Development Department,

Cyril Blet Results Officer, SNDR- African Development Bank

Julie Ladel AfCoP Facilitator at African Development Bank AfCoP-MFDR Project Secretariat

KENYA


Dr. Samson Macguka Director – M&E, Ministry of Devolution and Planning

Dr. S. Obongo Director, Public Service Commission

Joshua Mwingira Senior Assistant Director, M&E, Ministry of Devolution and Planning

Kenneth Kadenge,

Allan Watoro,

Keneth Njoro

Dorothy Mwetele,

ICT experts at Business Development Manager, Kingsway Business Systems Ltd

Mr David Kiboi M&E Specialist, Ministry of Devolution and Planning

Magreth Githinji Economist, Ministry of Devolution and Planning

Rodgers Achein Senior Economist – Ministry of Devolution and Planning

Vivian Simwa Senior Economist, and Chief Communication Officer, Ministry of Devolution and Planning

Nell Mwema M&E Specialist, Ministry of Devolution and Planning

Mary Njoroge Programme Manager, Integrated Support Project to Devolution, United Nations Development Programme

Marry Chizororo M&E Specialist, UNICEF

Odero Water Country economist, African Development Bank Kenya Country Office

Zipporah W. Gathiti M& E Specialist, United Nations Population Fund

Esther Mwikali Youth for Results Kenya Representative, Community & Operations, METTA

Philip Brynnum Jespersen

Senior Social Development Specialist, World Bank Group

Jeniffer Mutua President, Kenya Evaluation Society

MADAGASCAR


Rakotonirina Miarisoa Patricia

Directrice des Études de la programmation et du partenariat, Ministère de la population de la protection sociale et de la promotion de la femme

Nicole Rachelle Ravelonjanahary

Directrice de la coordination interne, Secrétariat Général, Ministère des Finances et du Budget

Kaningbi Nyaki Zangbula Country Program Officer, African Development Bank

François Rakotosaona Assistant Administratif, Banque Africaine de Développement, Bureau national de Madagascar

Hasina Rakotozaflarisoa Directeur de Planification, Suivi-

Évaluation, Présidence de la République, Programme de Réforme pour l’Efficacité de l’Administration

Roger Pierre Ralala Secrétaire Général de la Présidence de la République, République du Madagascar

Raharinomena Fanja Directeur, Planification, Suivi et Évaluation, Ministère auprès de la Présidence en charge de l’Agriculture et de l’Élevage

SÉNÉGAL


Oumar Diakhate Economiste

Chef du Bureau orientation et Synthèse, Ministère de l’Économie, des Finances et du Plan, Cellule de l’Évaluation et de la Performance

Babacar Diakhate Conseiller en organisation, Secrétariat Général – Bureau Organisation et Méthodes, Présidence de la République

Sada Ly Cissé Consultant international spécialiste en finances publiques- appui budgétaire Stratégies de croissance et de réduction de la pauvreté- Microfinance, Coach national

Ibrahima Ndiaye Directeur Général, Secrétariat Générale – Bureau Organisation et méthodes, Présidence de la République

Waly Faye Consultant, Coach national

Matar Faye Président Directeur Général de M&F, Président de Jeunes pour les Résultats (Youth for Results)

Dieynaba Ndiaye Présidente de Femmes pour les Résultats (Gender for Results), Ex-Trésorière de SENEVAL (Association Sénégalaise d’Évaluation)

Omar Diakhaté Président de SenCoP (national CoP)

Abdou Karim Lo Consultant, Coach Régional

Khadidiatou Gassama

Économiste-Pays, Bureau National du Sénégal, Banque Africaine de Développement

Adalbert Économiste-Pays en chef pour la Gambie et le Cap Vert,

Nshimyumuremyi Département Régional Afrique de l'Ouest, Représentant intérimaire du Bureau National du Sénégal, Banque Africaine de Développement

ZAMBIA


Dr. Auxilia Bupe Ponga

Director, M&E, Ministry of National Development Planning

Dr. Richard Banda Permanent Secretary, Ministry of National Development Planning

Wedex Ilunga Senior Procurement Specialist, The World Bank Group

Anthony Ademola Taylor

Economic Affairs Officer, Southern Africa Office, United Nations Economic Commission for Africa

Philip Boahen Principal Country Program Officer, African Development Bank Group

Simon Oswan M&E Expert, COMESA-USAID Programmes, COMESA Secretariat

Simal O. Amor Chief, Strategic Planning & Research, COMESA Secretariat

Ambassador Dr. Kipyego Cheluget

Assistant Secretary General (Programmes), COMESA Secretariat

Dr. Francis Mangeni Director, Trade Customs and Monetary Affairs, COMESA Secretariat

Ms. Sandra Uwera Chief Executive Officer, COMESA Business Council

Peter Engbo Rasmussen

Principal Country Economist, African Development Bank

Damoni Kitabire Resident Representative, African Development Bank

John T Njovu ZAMEA- Zambian Evaluation Association

Prudence Kaoma Assistant Director, Research and Evaluation, M&E Department, Ministry of National Development Planning

ZIMBABWE


Eric D. Zinyengere National Coach Managing Consultant, Team Consulting

Eyerusalem Fasika Principal Country Program Officer, African Development Bank

Edwin F. Vela-Moyo

Director, Budgets Department, Ministry of Finance and Economic Development

Sakae Kashihara Project Formulation Advisor, Japan International Cooperation Agency Zimbabwe

Dr. Roger Atindehou

Director, Operations Department, African Capacity Building Foundation

Germana Topolovec

Attached Governance and Social Sectors, Delegation of the European Union to Zimbabwe

Johannes(Han) Herderschee

Senior Economist, The World Bank

Shumon Yoshiara Resident Representative, Japan International Cooperation Agency Zimbabwe

Mr. Crispen Mawadza

Financial Sector Specialist, The World Bank

Tachiwana Nkomo Project Officer, Japan International Cooperation Agency Zimbabwe Office

Zach Landau Economic Officer, United States Embassy Harare

Mr. Solomon Mhlanga

Senior Principal Director, Public Sector Modernization and Performance Management, Office of the President and Cabinet

Tachiwana Nkomo Project Officer, Japan International Cooperation Agency Zimbabwe Office

Amarakoon Bandara

Economic Advisor, UNDP Zimbabwe

Mr. Georges van Montfort

Country Director, UNDP Zimbabwe

Ethel Bangwayo National Economist, UNDP Zimbabwe

Faith Musuka Program Officer, Training, Japan International Cooperation Agency Zimbabwe Office

Phides Mazhawidza

Trustee, Zimbabwe Land & Agrarian Network

Katharina Guhr Advisor, Governance Programme, German Development Cooperation

Mr. Frejus Thoto, Knowledge Management Expert, African Capacity Building Foundation

Dr. Thomas Munthali

Director, Knowledge and Learning Department, African Capacity Building Foundation

Mr .Bakary Korne Director, Partnerships and Resource Mobilization, African Capacity Building Foundation

Willie Mushayu Register of Companies and Intellectual Properties, Ministry of Justice, Legal and Parliamentary Affairs

ANNEX III- LIST OF REVIEWED DOCUMENTS AND SUGGESTED READINGS

1) Abbas, S. & Ouollo, S.C. 2016. Feuille de route de la GRD-CI : Rapport d'activités des coaches nationaux. AfCoP-GRD.

2) ACBF. 2016. AfCoP supervision Aide memoirs (2013-2017).3) ACBF. 2016. Survey of the Capacity Needs of Africa’s Regional Economic

Communities; Executive Summary. Second Edition.4) ACBF. 2016. The ACBF Strategic Plan for 2017-2021. 5) AfCoP-MfDR. 2017. Aide-Memoire de la mission. Mission d’achèvement du

Projet Multinational - Project Managing for Development Results (MfDR) – AfCoP- Volet UÉMOA et Burkina Faso.

6) AfCoP-MfDR. 2017. Road Map-Progress by Country - WAEMU & COMESA.

7) AfCoP-MfDR/AHHD. 2017. Aide Memoire of the Support to Building Capacity on Managing for Development Results (MfDR) Project Completion Report (PCR) Mission (July 17-26, 2017).

8) AfCoP-MfDR. 2017. First Meeting of the Executive Council, October 3-5, 2017 – Afdb Headquarters, Abidjan, Côte d’Ivoire.

9) AfCoP/Afrik4R 2016. Guideline for Initiating and Managing Evaluations. Guideline N° 7.

10) AfCoP. 2013 – 2016 Annual Progress Reports.11) AfCoP. 2011. Building Capacity on Managing for Development Results in

the Regional Member Countries (RMCs) and the Regional Economic Communities (RECs). AfCoP.

12) AfCoP. 2012. A Stock Taking of AfCoP Secretariat Activities. 13) AfriK4R & COMESA. 2014. Management for development results

readiness assessment of COMESA Secretariat. AfCoP-MfDR. 14) AfriK4R & UEMOA. 2015. Priorité aux résultats dans l'UÉMOA. AfCoP

Secretariat.15) AfriK4R 2014. Putting results first in Africa. AfCoP-MfDR.16) AfriK4R 2015. The State of Managing for Development Results in the

COMESA Region and Associated Country. AfCoP-MfDR.17) AfriK4R. 2012. Briefs on the AfCoP-MfDR and the Africa Knowledge for

Results Initiative. AfCoP Secretariat.18) AfriK4R. 2013. Guide de mise en œuvre de la feuille de route nationale.

AfCoP-GRD.19) Afrik4R. 2013. Rapport sur la formation des coaches nationaux et des

journées d'orientation et de planification. AfCoP-GRD. 20) AfriK4R. 2014-2015. Évaluation du pays en gestion axée sur les résultats

de développement (les 3 pays UÉMOA de l’échantillon). AfCoP-GRD.21) AfriK4R. 2014. Members Survey Results. AfCoP Secretariat.22) AfriK4R. 2015. The Status of Managing for Development Results in the

COMESA Region. AfCoP Secretariat.23) Andrews, M. & Nick, M. 2015. A Study of Peer Learning in the Public

Sector. Effective Institutions Platform. 24) Asian Pacific CoP. 2011. Framework for Results-Based Public Sector

Management and Country Cases.

25) BAD/ORQR.2. 2012. Note d’orientation du personnel sur l’établissement de rapports d’achèvement de projet et leur notation.

26) CEval/UEMOA. 2016. Mission circulaire de mise en place des points focaux nationaux de la Cellule d’évaluation dans les États membres de l’UÉMOA. Rapport de synthèse et de capitalisation. Décembre.

27) Cooper, H. 2015. Mid-term evaluation: Building Capacity on Managing for Development.

28) Eyben, R. 2013. Uncovering the Politics of “Evidence” and “Results”. A Framing Paper for Development Practitioners. Learning Agenda on Local Capacity Development.

29) Halidou, S. & Issa, S. 2017. Séminaire de formation des députes membres des commissions de finances et budget des assemblées nationales des états membres de l'UÉMOA sur les finances publiques. UÉMOA.

30) Hasina, R. & al. 2016. Culture de résultats, garant de développement. AfriK4R.

31) Ian, C.D. & al. 2017. Stratégie et plan d'action de renforcement des capacités évaluatives nationales au Burkina Faso. Unicef.

32) IDEV/AfDB. 2016. Évaluation du Programme de renforcement des capacités statistiques dans les pays membre régionaux et dans les organisations sous-régionales. Sénégal. Groupe de la Banque Africaine de Développement.

33) Lare-Lantone, K. 2017. Study on Rapid Results Initiatives in Burundi, Madagascar, Senegal and Kenya. AfriK4R.

34) Tarsilla, M. 2011. AfCoP Retrospective Evaluation. AfCoP Secretariat.

35) Ministère des Finances et du Budget/Madagascar. Initiative à Résultats Rapides, 2015-2016.

36) The World Bank Group 2017. Kenya Accountable Devolution Program (KADP).

37) UEMOA. 2015. Rapport de Surveillance commerciale. 38) UEMOA. 2016. Rapport annuel sur le fonctionnement et l'évolution de

l'Union. 39) United Nations. 2011. Results-Based Management Handbook,

Harmonizing RBM Concepts and Approaches for Improved Development Results at Country Level.

40) Valentine, S. 2015. Étude de la demande de suivi et évaluation et de service de gestion de la performance au Togo. CESAG-CLEAR.

1)

ANNEX IV- EVALUATION TERMS OF REFERENCE

BACKGROUND AND CONTEXT

The African Community of Practice on Managing for Development Results (AfCoP-MfDR) was established in 2007 with the aim of contributing to the promotion of a results-culture and practice in Africa. It was established as a response to developing countries’ call for stronger ownership and leadership in shaping the results agenda and in reinforcing capacity to strengthen country institutions. AfCoP also contributed to shaping the African perspective on development effectiveness at the regional meeting held in Tunis on November 2010, which provided valuable input into the 2011 Busan Partnership for Effective Development Co-operation. The Tunis consensus re-focused the attention from aid effectiveness to the broader development effectiveness agenda and recognized that development in Africa requires a robust and private sector-led growth, effective and accountable states, able to finance their development needs from their internal resources. To be truly successful, this process must also be accompanied by more efficient regional development and economic integration across the continent, thus creating economies of scale that will boost trade and investment and equip African countries to compete in the global economy.

A. In 2012, the African Development Bank (AfDB) funded a project to support the AfCoP-MfDR in (i) promoting the mainstreaming of Managing for Development Results (MfDR) practices in 18 regional member countries (RMCs) and 2 regional economic communities (RECs) policies and strategies; and (ii) strengthening the capacities of the 2 RECs (COMESA and WAEMU) and their respective member countries to facilitate policy and regional programs implementation through the MfDR approach, hence promoting regional integration. Regional and national communities of practice (CoPs) were established in the 2 RECs and the 18 RMCs as a conduit to manage the activities of the AfCoP-MfDR PROJECT.

B. To this extent, AfCoP developed the Africa for Results (AfriK4R) initiative, as a regional approach to development, to help materialize that vision by mobilizing African stakeholders around the results and regional development agendas. The AfriK4R initiative aims to: (i) strengthen public sector management systems to build capable states; (ii) strengthen results-oriented organizations for better service delivery; (iii) enhance regional convergence through MfDR to improve regional integration; and (iv) promote South-South knowledge sharing.

C. The AfCoP-MfDR Project was originally approved for 3 years with the understanding that this was a pilot which could be scaled up. The initial ending date of June 30, 2016 was extended to May 31, 2017 with the

understanding that RMCs and RECs would have completed the activities of the agreed roadmap and plans of actions.

OBJECTIVES OF THE ASSIGNMENT

The project will come to an end in May 2017 and a final evaluation needs to be conducted as part of the Bank’s commitment to ensuring that lessons learnt during the implementation process are duly internalized. Furthermore, evaluation results are used to improve decision-making processes and to evaluate project performance while improving project design and implementation of another phase of the project, if deemed necessary.

The purpose of the planned evaluation is therefore to assess the achievements and shortcomings of the project in the delivery of its specific objectives, outputs, and outcomes during its implementation period as outlined in the project document. The evaluation will establish the performance and success of the project in terms of its relevance, efficiency, overall and cost-effectiveness, and sustainability of its results, while identifying lessons learnt and best practices to be promoted in order to improve a results-culture in Africa.

SPECIFIC OBJECTIVES

The Evaluator will assess the outcomes and possible impact of the project on the following main thematic areas:

i. Relevance of the project Relevance of the project in improving the results-culture in

Regional Member Countries (RMCs) and Regional Economic Communities (RECs) using MfDR principles and tools

Relevance of the project to the needs of the intended beneficiaries Consistence of the project objectives with its activities and

outputs Relevance of the project in increasing the efficiency in the

management of public affairs in RMCs and RECs

ii. Project efficiency and effectiveness Assessment of adequacy and efficiency in the use of project

resources (human and financial) in relation to planned activities, outputs and outcomes

Assessment of implementation methods used in the course of implementing the project

Assessment of whether the project achieved the project stated objectives as outlined in the Project Appraisal Report

Assessment of the extent to which project beneficiaries were involved in the decision-making and implementation of the project

iii. Project outcomes and possible impactThe main task is to assess the changes brought in the lives of people in RMCs and RECs as a result of the project. More specifically, the following will be assessed: Changes in the way of conducting public affairs

Empowerment of key actors in the decision-making process in the targeted beneficiaries

Improved leadership and accountability in the use of MfDR principles and tools

Status of the 5 MfDR pillars compared to the readiness assessment conducted at the beginning of the project

Outline intended and unintended outomes Recommend changes, improvement or future programmatic

issues of similar approaches

iv. Project sustainabilityThe main task is to assess the sustainability of the project to ensure continuation, maintenance and replication of the project outcomes, including: Project management arrangements Effectiveness of capacity building interventions Technical replication and financial viability Internalization of best practices by RMCs and RECs Scaling up of the project beyond current RMCs and RECs

v. Lessons learned, conclusions and recommendations Capture best practices and lessons learned from the project Draw conclusions and make recommendations

MAIN ACTIVITIES

The evaluation will focus solely on the activities conducted and the results achieved by the implementing agencies (AfDB and ACBF) under the AfCoP-MfDR Project entitled “Building capacity on Managing for Development Results in the Regional Member Countries (RMCs) and the Regional Economic Communities (RECs).” As indicated earlier, the AfCoP-MfDR Project began in 2012 with an original ending date of June 30, 2016. In November 2015, a no-cost extension was granted for 11 months to May 31, 2017 given the slow start of the project and to allow for proper closure of its activities.

DELIVRABLES

INCEPTION REPORT: An inception report should be presented and approved during an initial briefing session. The report should contain an evaluation matrix that displays a set of evaluation criteria with related questions to be answered and the methodology to be used for this purpose, including supporting data to be collected. Underlying assumptions should be clearly stated and a proposed schedule should be presented with assigned responsibilities and deliverables. The Inception report will provide as well an annotated outline of the Evaluation Report.

DRAFT EVALUATION REPORT: a draft report will be shared with the AfCoP-MfDR Project Secretariat and the Executive Committee (ExCo) of the AfCoP for comments and inputs. FINAL EVALUATION REPORT: the Final Evaluation Report taking into account comments and inputs from the AfCoP-MfDR Project Secretariat and the ExCo.

DURATION OF THE ASSIGNMENT: 75 days

final evaluation of the afcop-mfdr project ... · web vieworganization of the national high-level...

Documents