Evaluation of the Integrated Safeguards System Inception Report

Version April 2019

Task Manager: Monica Lomena-Gelis, m.lomena-gelis@afdb.org

Contents 1. Introduction ................................................................................................................................ 1

2. Background ................................................................................................................................. 2

2.1. ISS objectives and alignment with best practices of comparator organizations ...................... 2

2.2. ISS resources, processes, tools and incentives ...................................................................... 3

3. Evaluation goal, objectives, design and scope ............................................................................ 6

3.1. Evaluation goal and scope .................................................................................................. 6

3.2. Evaluation specific objectives, criteria and questions to be addressed ................................... 7

4. Data collection and analysis, sampling and evaluation limitations ................................................ 9

4.1. Data collection and analysis, including sampling ..................................................................... 9

4.2. Limitations of the evaluation .................................................................................................. 11

5. Deliverables, Level of Effort and Timeline ................................................................................ 12

6. Evaluation team and quality assurance of the evaluation process .............................................. 12

7. Communication and dissemination of findings and recommendations ....................................... 13

References ............................................................................................................................................ 16

Annexes (separate document): Annex 1. AfDB Integrated (Environmental and Social) Safeguards System Annex 2. ISS in the context of the Bank’s Project Cycle Annex 3. Reconstructed Theory of change of the Integrated Safeguards System Annex 4. Desk validation tool used to assess the projects in relation to ESS requirements. Annex 5. OS2 assessment proposal (compensation/resettlement of affected population) Annex 6. Financial intermediaries’ operations assessment proposal Annex 7. – Evaluation matrix

Acronyms list

AfDB African Development Bank Group

CRMU (or BCRM) Compliance Review and Mediation Unit at the AfDB

CSO Civil Society Organization

CSS Climate Safeguards System

DBDM Development and Business Delivery Model

ES Environmental and Social

ESAP Environmental and Social Assessment Procedures

ESIA Environmental and Social Impact Assessment

ESMP Environmental and Social Management Plan

ESMS Environmental and Social Management System

ESS Environmental and Social Safeguards

FI Financial Intermediary

GHG Greenhouse gas

IDEV (or BDEV) Independent Development Evaluation

IRM Independent Review Mechanism

ISS Integrated Safeguards System

ITST Integrated Safeguards Tracking System

MDB Multilateral Development Bank

OS Operational Safeguards

PAP Persons affected by projects

PBO Programme based Operation (Budget support)

PECG Climate Change and Green Growth Department at the AfDB

RAP Resettlement Action Plan

RMC Regional Member Country

SESA Strategic Environmental and Social Assessment

SNSC Safeguards and Compliance Department at the AfDB

ToC Theory of Change

VP Vice-Presidency

1

1. Introduction The Independent Development Evaluation (IDEV) function of the African Development Bank Group (“the AfDB or the Bank”) reports to the Board of Directors of the Bank, rather than to the senior management, through the Committee on Operations and Development Effectiveness (CODE). Its evaluators are independent and work collaboratively with external consultants, but retain final control over the delivery of evaluation products. During the ADF12 Commitments and the General Capital Increase processes, and at the request of shareholders, the Bank committed itself in 2010 to revising and upgrading its Environmental and Social Safeguards (ESS) system and procedures by preparing and adopting an Integrated Safeguards System (ISS) and by promoting the mainstreaming of climate-change considerations1. On December 17, 2013, the Board of Directors of the AfDB unanimously adopted the ISS and mandated IDEV to conduct an Evaluation of the ISS’s effectiveness in achieving the AfDB’s safeguards objectives four years after its adoption.

In February 2017, CODE approved an updated IDEV 2016-2018 work program which included (i) quality at entry; (ii) quality of supervision and (iii) integrated safeguards. IDEV proposed to complete these evaluations using a harmonized approach that leverage conceptual synergies through an Evaluation of Quality Assurance across the Project Cycle. Hence, compliance with the Bank’s ESS during project design, supervision and completion (Evaluation Question 4 below) has been partially reviewed by this first evaluation, which was presented in 2018. The Evaluation of the ISS will assess the relevance and robustness of its design, the efficiency of the systems, process, resourcing and incentives, and emerging effectiveness in achieving the Bank’s safeguard’s objectives. The evaluation audience is fully analyzed in the stakeholder mapping in Section 7 of this inception report. The Evaluation of the ISS will consider the following overarching questions: Table 1. Evaluation criteria and overarching questions.

Evaluation criterion

Overarching question

I. Relevance

1. Is the Bank’s ISS aligned with comparator organizations and identified best practices and covering relevant emerging safeguards issues?

2. Is the Bank´s ISS effectively supporting the Bank´s strategic priorities and relevant to the current portfolio and the institutional changes?

II. Efficiency

3. What are the Bank´s internal factors that facilitate or constrain implementation of the ISS as designed in relation to resources (human and financial), capacity (skills), systems, process and tools, and institutional incentives?

III. Emerging

evidence on Effectiveness

4. To what extent are the Bank´s quality assurance processes contributing to ensure compliance with ISS objectives (mitigation of environmental and social risks of Bank Group-funded projects)?

5. What is the level of ISS compliance from the emerging practice of the financial intermediaries lending (category 4 projects according to ISS)?

6. What is the level of advance in relation to the ISS commitments to

mainstream climate change?

This evaluation will coordinate with the ongoing self-assessment (Mid-term Review of the ISS) that the AfDB’s Safeguards and Compliance (SNSC) team will launch in 2019 and the revision of the Environmental and Social Assessment Procedures (ESAP). It will also consider the 3rd Review of the Independent Review Mechanism which will start in 2019.

1 In September 2010, the Bank’s Operations Committee approved the concept note, “Towards and Integrated

Safeguards System”.

2

2. Background

2.1. ISS objectives and alignment with best practices of comparator organizations The 2013 ISS was designed to be better aligned with the Bank’s new policies and strategies, including the Bank’s new Ten-Year Strategy (2013-2022), including the High 5’s2. Among others, it was intended to update the existing policies to adopt good international practice, including on climate change; to better support the evolving range of lending products and innovative financing modalities;3 and to improve internal processes and resource allocation4. The ISS is designed to promote the sustainability of project outcomes by protecting the environment and people from the potentially adverse impacts of AfDB financed projects. Specifically, the safeguards aim to:

Avoid adverse impacts of projects on the environment and affected people, while maximising potential development benefits to the extent possible;

Minimise, mitigate, and/ or compensate for adverse impacts on the environment and affected people when avoidance is not possible; and

Help borrowers/clients to strengthen their safeguard systems and develop the capacity to manage environmental and social (ES) risks.

Additional information about the ISS objectives, the process to develop the ISS, its alignment with key Bank’s strategic documents, the key issues that it tried to address and the instruments proposed to implement the ISS can be found in Annex 15. According to preliminary interviews with Bank staff and development partners and review of existing literature, the development and implementation of the ISS has resulted in a major step forward for the AfDB in relation to the management of ES risk of the portfolio. The consolidation of the safeguards has resulted in a clearer understanding of requirements, eliminated contradictions between policies, increased the number of ES issues being addressed and resulted in a broader application of the policies across the Bank’s portfolio. The Integrated Safeguards System (ISS) proposes to incorporate climate change into development efforts requiring assessments of vulnerability to climate change as part of the environmental and social assessment process for its public and private sector operations6. The Climate Safeguards System (CSS) has been piloted since 2014, and includes climate risk screening and adaptation review for project categorization, so that potential measures may be identified to protect the investment against climate risks and to promote opportunities for adaptation7. Information about the application of the CSS is stored in a different internal Bank online database from that of the other ISS applications, the Climate change tracking system. The CSS also produces country adaptation factsheets. In addition, among the requirements of Operational Safeguards (OS) 4 (pollution prevention and control, hazardous materials and resource efficiency), the ISS proposed to develop a pilot tool to track greenhouse gas (GHG)

2 The AfDB High 5s are: Light up and power Africa; Feed Africa; Industrialize Africa; Integrate Africa; and Improve the quality of life for the people of Africa. 3 The ISS documents mentioned the increased focus on Program Based Operations (PBOs) and support to financial intermediaries (FIs) which include banks, insurance, reinsurance and leasing companies, microfinance institutions, private equity funds and investment funds. 4 ISS Policy statement (2013), page 2. 5 The ISS instruments comprise an Integrated Safeguards Policy Statement, five Operational Safeguards (OS), and two practical guidance notes (Environmental and Social Assessment Procedures (ESAP) and Integrated Environmental and Social Impact Assessment (IESIA) guidance notes). 6 AfDB, 2013. Policy Statement and Operational Safeguards (page 19). 7 The Adaptation Review and Evaluation Procedures guide the identification of potentially viable climate components and activities that are integrated into project design as appropriate.

3

emissions in accordance with the provisions of the United Nations Framework Convention on Climate Change8. Since the ISS was published in 2013, several other Multilateral Development Banks (MDBs) have conducted safeguards reviews and given considerable thought to the types of issues that need to be covered in a safeguards process for a MDB9. Also, additional issues that were not included in prior safeguards policies or operational standards have started to work their way up the agenda of borrowers, clients, CSOs, communities etc. These emerging safeguards issues are sometimes related to the new types of lending operations that the Bank currently has in place (PBOs, financial intermediary operations, among others), and how the ISS system, originally crafted for major infrastructure investment, is being adapted to cover the current portfolio of the Bank Group. In other occasions, emerging issues are related to those which were not covered in detail during the Environmental and Social Impact Assessments (ESIA) conducted by the borrowers: health and safety, biodiversity offsets, influx, gender based violence, climate change mitigation and adaptation. The evaluation will explore this as part of the relevance evaluation questions.

2.2. ISS resources, processes, tools and incentives Once a policy and set of standards is in place, one of the most important factors in making its implementation successful is providing sufficient resources (human and financial), capacity (skills), systems, process and tools, and institutional incentives. These are key elements to ensure the effective operationalization of the ISS. According to the ISS and the AfDB’s Group Operations Manual, the Environmental and Social Assessment of Bank Group-funded interventions is primarily the borrower’s responsibility. The Bank is responsible for assisting the borrower at different stages in the project cycle to ensure that the Bank’s ESS requirements are respected. Within the Bank, Task Managers are responsible for ensuring compliance with the Environmental and Social Assessment Procedures (ESAP), with the support of ESS experts. Therefore, in addition to strategic and broader safeguards issues, such as helping the borrower to strengthen its capacity to address ES risks, the operationalization of the ISS at the Bank is focused on the project cycle. When the ISS was introduced, the staff responsible for ensuring its implementation were divided between three different divisions within the Bank10. The current Bank´s ESS team has been pulled together in 2016 into a single Safeguards and Compliance Department (SNSC)11 under the direction of a Chief Safeguards Policy Officer to deliver on a new mandate (the ISS). Reviewed documents and interviews suggest that the ESS support function at the Bank has been significantly under-resourced and that staffing levels are lower than comparator organizations. While in 2013, there were 20 ESS staff, in 2017 this number was reduced to 15 (five social and eight environmental safeguards specialists, a Lead environmental safeguards officer and the Officer-in-Charge (OIC) Director / Chief Safeguard Policy Officer)12. Interviews

8 AfDB, 2013. Policy Statement and Operational Safeguards (page 46). 9 The most significant recent updates of safeguards or standards for the management of environmental and social risk in development project finance have been issued by the World Bank, which completed its safeguards review in 2016, and is currently planning the roll out of its new Environmental and Social Framework (ESF), and the Development Bank of Southern Africa (DBSA) which issued its 11 Environmental and Safeguards Standards for implementation in March 2018. Other institutions which have recognized approaches to the management of environmental and social risk and are currently considering upgrading their safeguards or standards include the International Finance Corporation (IFC), which is generally considered to have the most advanced private sector focused set of standards for environmental and social performance, the Inter American Development Bank (IDB and IDB Invest) has recently released an evaluation about its Safeguards system. In addition, the European Bank for Reconstruction and Development (EBRD) updated its safeguards policy in 2014. 10 Those divisions were ORQR3, ONEC3, and OSAN3 11 AfDB organogram 2017 12 By December 2018, each region had at least one environmental and one social safeguards specialist, with the Southern and West regions having more staff. The ideal configuration according to SNSC would be five safeguards

4

with comparators suggest that they operate with a higher number of specialists in this domain, with support of more external consultants and differentiated teams to address the ES risks of public and private sector operations.13 In addition, data about ES categorization of projects show that more ES risky projects (category 1 and 2) are entering the portfolio. Recent estimates of SNSC consider that some 72% of the ongoing portfolio is of high risk14. There were various open positions to address the deficit of 715 man/weeks to cover all projects, including origination, quality control, supervision and capacity building, according to SNSC in 2017.15 The staff responsible for implementation of the Climate Safeguards System (CSS), which is part of the ISS, are housed with the rest of the climate team, in Climate Change and Green Growth (PECG) within a different Vice-Presidency, the Energy VP. Until 2015, a part of the safeguards team was supporting the sectors to accompany the borrower to develop good quality project ESIAs, while another was in charge of their revision to ensure compliance through the Readiness Review process.16 Several interviewees consider that the centralization of the ESS function in the same department may have diminished the quality assurance and enforcement of ES dimensions in project designs. They also point out that there is not sufficient time available to do internal peer review of decisions about the different milestones of the quality assurance process to ensure that ISS requirements are fully respected. The current evaluation will estimate the time allocation of ES safeguards experts among different tasks. Preliminarily findings show a skewed trend towards appraisal support, with limited time to provide support to borrowers to ensure the avoidance or mitigation of key environmental and social impacts during implementation or enhancing their capacities to fully understand the AfDB’s ISS. Interviewees also mentioned that, ensuring the good mix of skills required for ISS implementation has been challenging in relation to the constrained market for hiring staff with the breadth of environmental, social, consultation, cultural heritage, occupational-health-safety and other skills that are required. Efforts to ensure that ESS staff are at the levels and with the skills required is an important factor to facilitate the effective implementation of ISS across the Bank Group´s portfolio and considering emerging safeguards issues. Similarly, operations’ staff need a good understanding of ESS issues affecting the portfolio. The evaluation will assess the quality of the training provided to Task Managers and ESS staff. The approval of the ISS prompted the development of detailed guidance material to implement the specific Operational Safeguards (OS) across the project cycle, and entailed a series of reviews about specific safeguards issues17. Apart from the ESAP and guidance material

staff per regional office (two social and two environmental specialists and a coordinator). 13 The Bank does not record the number of projects supported by each ESS expert, although some interviewees reported having handled up to 23 projects in 2017, including due diligence of 15 category 1 projects before approval and 8 field supervisions. Only the World Bank (WB) and the International Finance Corporation (IFC) have estimated the average ratio: 10 for WB public sector operations, 12 for IFC private sector operations (5 appraisals and 7 supervisions per year). The current evaluation will try to confirm these figures. 14 No data about the ES profile of the ongoing portfolio could be found. These figures were estimated for the SNSC retreat in 2017. These numbers do not consider 339 operations for which the environmental categorization was not included in SAP. It is likely that lots of them are category 4 projects, since the SNSC data only reported one ongoing category 4 project out of the 116 analyzed. 15AfDB, 2017. Présentation retraite SNSC, Supervision de la Conformité Environnementale et Sociale des Projets. 16 For E&S review purposes, Readiness Review (RR) at the design stage involves a desk review of documentation produced for the Project Concept Note (PCN) and for the draft Project Appraisal Reports (PAR). Should the PCN meet the requirements of the RR, it is sent onward for management clearance prior to the drafting of the PAR. As per the PCN, the E&S aspects of the RR review of the draft PAR are undertaken by safeguards specialists, and given a rating of 1 to 4. PARs with an average score of 3 or below are deemed to be of “insufficient design quality” for Board submission. See Annex 2. 17 The AfDB Safeguards and Sustainability Series (available online): (i) 2013 AfDB’s Integrated Safeguards System (Volume 1, Issue 1); (ii) 2015 AfDB Assessment of the use of Country Systems for environmental and social safeguards and their implication for AfDB-financed operations in Africa (Volume 1, Issue 2); (iii) 2015 AfDB Involuntary Resettlement Policy: Review of Implementation (Volume 1, Issue 3), (iv) 2015 Environmental and

5

updated in 2015, the Bank’s Operations Manual includes specific sections to address ES risks throughout the project cycle. The Bank´s project cycle comprises key stages before Board approval (project identification, preparation and appraisal) and during implementation/completion. Across this cycle, different ES outputs are developed by the borrower and validated by the Bank, depending on the type and E&S risk of the intervention18. The operationalization of the ISS at the project level entails various due diligence, review and supervision activities that the Bank undertakes to ensure that the borrower comply with the ISS requirements. This information is summarized in Annex 2. Several steps have been taken to develop internal Bank systems and processes that will support ISS implementation since the ISS became effective in July 2014. As part of the ESAP, these include the development of an internal Integrated Safeguards Tracking System (ISTS) that tracks and stores key environmental and social information generated over the lifetime of the project which has been cleared as ISS compliant. This should facilitate periodic compliance checks. As of mid-2018, the ITST was not yet fully operational. During data collection for the evaluation scoping, IDEV encountered serious problems to find key ES outputs for the projects in the sample. Internal stakeholders recognized that the work to populate the ISTS has been less consistent than they expected due to staff shortage. SNSC used to conduct self-evaluation ISS compliance checks (status of the E&S categorization of projects approved yearly) and annual compliance review audits of selected projects. The Bank’s Independent Review Mechanism (IRM),19 administered by the Compliance Review and Mediation Unit (CRMU), also conducts spot checks reviews of projects to ensure compliance with the ISS. One of the keys to effective implementation of any set of safeguards or standards, is the ability to learn and retain learning within the organization. Initial document review and interviews suggest that this is a challenge within the context of Bank Group borrowers and clients, and also within the Bank itself20. In addition, effective implementation of new systems and processes within any organization requires an element of cultural change, which is normally best supported by a system of incentives (to follow) and disincentives (for ignoring or by-passing) the new system or process. Effective internal cultural change takes time, and requires the allocation of resources for training and capacity building both of staff and clients. A Bank’s staff survey and interviews will inform the level of knowledge of operations and management staff about the ISS. Past and ongoing challenges faced by the Bank in safeguarding the natural and human environment from the negative impacts of Bank Group -financed projects have been raised in the past and are recurrently mentioned by key stakeholders. During the scoping stage of the evaluation, a thorough review of documents and interviews with Bank safeguards and CRMU staff, helped IDEV to identify a list of issues for further assessment of the ISS. The evaluation

Social Assessment Procedures (ESAP) (Volume 1, Issue 4); (v) 2015 AfDB ISS: Guidance Materials (Volume 2, Issue 1); (vi) 2016 Development and Indigenous Peoples in Africa (Volume 2, Issue 2). 18 The main Bank-validated documents (ES outputs) are: ES sections included in the Project Appraisal Report (PAR), the Bank´s summary of the Environmental and Social Impact Assessment (ESIA), or the Environmental and Social Management Plan (ESMP), or the Strategic Environmental and Social Assessment (SESA), or the Environmental and Social Management Framework (ESMF) or the Environmental and Social Management System (ESMS); the summary of the Resettlement Action Plan (RAP), supervision reports (Implementation Progress and Results Report – IPRR- for public sector projects or Project Status Report for private sector projects, aide memoire or BTORs, according to availability, completion reports (Project Completion Report for public Projects and Expanded Supervision Report for private projects). 19 The IRM was established in 2004 to provide people with an avenue to request the Bank to comply with its own policies and procedures if they consider that they are, or are likely to be, adversely affected by a project financed by the Bank Group as a result of violation of the Bank Group’s policies and procedures, among them the ISS. 20 Examples given during preliminary interviews were limited participation of Safeguards team in project completion missions; and lack of mechanism to support the borrowers and clients to learn from mistakes, so that the same mistakes, driven by lack of capacity or other factors, are repeated project after project.

6

strategically selected certain issues to be considered as the most pressing ones to inform the Board of Directors about significant issues which remain outstanding and where additional support may be necessary to ensure the ISS achieves its overarching goal. They are detailed in the following section.

3. Evaluation goal, objectives, design and scope

3.1. Evaluation goal and scope The Evaluation of the ISS aims to assess the relevance and robustness of its design, the efficiency of the systems, process, resourcing and incentives in place, and emerging effectiveness in achieving the Bank’s safeguard’s objectives. This will provide information for the Board and management as they identify changes to further strengthen the ISS. Given the relatively recent date of approval of the ISS, which became effective in July 2014, the evaluation itself is expected to use a formative approach. The evaluation will be theory-based (see the reconstructed Theory of Change for ISS in Annex 3) and will be mainly focused on design and early implementation, on processes and efficiency, as well as how the new policy addresses the shortcomings found in the old policies. Mixed-methods will be used, combining descriptive statistics when possible with qualitative approaches that will enable the team to develop a set of concrete and practical recommendations for improvement of the ISS. In addition to the overall assessment of the system, processes and resources to implement the ISS, the evaluation will cover desk reviews of 183 public and private sector projects:

- The sample of projects analyzed in the case study about resettlement/compensation (36) and category 4 projects (58 financial intermediary operations) were approved when the ISS was already effective.

- Only 40 of the 89 projects which will be desk reviewed to inform the evaluation question 4 were approved before the ISS effective date in order to capture the supervision of environmental and social compliance.21 Some additional projects will be screened to see how the climate mainstreaming requirements of ISS were considered (evaluation question 6).

The evaluation will assess the role of ISS in supporting the delivery of the Bank´s strategic objectives, if the ISS is fit for purpose to cover emerging ES safeguards issues and if the ISS is relevant to the Bank’s current portfolio. It will also use other comparators organizations and best practices to benchmark the ISS and related processes. It will assess if the Bank has in place the right system and process to ensure the attainment of ISS objectives. In addition, the evaluation will assess in more depth the compliance with the Operational Safeguards 2 (Involuntary Resettlement: land acquisition, population displacement and compensation) including some deep dive analysis of a sub-set of key OS 2 themes (stakeholder engagement, vulnerable groups (including grievance redress mechanisms) and supervision of resettlement implementation. Finally, the evaluation will assess the compliance with ISS of operations involving lending to financial intermediaries (FI), classed as category 4 according to Operational Safeguards 122. According to preliminary interviews, this last sub-

21 E.Q.4. To what extent are the Bank´s quality assurance processes contributing to ensure compliance with ISS objectives (mitigation of environmental and social risks of Bank Group-funded projects)? 22 Financial intermediaries on-lend or invest in subprojects that may produce adverse environmental and social impacts. Financial intermediaries include banks, insurance, reinsurance providers, private equity funds and investment funds that use the Bank’s funds to lend or provide equity finance to their clients. Financial intermediaries also include private or public sector companies that receive corporate loans or loans for investment plans from the Bank that are used to finance a set of subprojects. FI operations are further classified as FI-A, FI-B and FI-C to reflect the potential ES impacts and risks of the financial intermediary’s existing or proposed portfolio of subprojects, based on the nature, type, scale and sector exposure. More details are included in ESAP, 2015, page 25.

7

objective is very important to assess the value proposition of the AfDB to its FI clients in relation to the relevance and tailoring of its ES requirements to the nature of the product or facility and client.

3.2. Evaluation specific objectives, criteria and questions to be addressed

The evaluation proposes to assess the ISS through three evaluation criteria, six overarching questions and specific evaluation questions. The Evaluation Matrix in Annex 6 provides further detail. Table 2. Evaluation criteria, overarching questions and evaluation questions.

Evaluation criteria and overarching questions

Specific evaluation questions

I. Relevance

1. Is the Bank’s ISS aligned with comparator organizations and identified best practices and covering all relevant emerging safeguards issues?

1.1 To what extent is the ISS aligned with other MDBs and best practice standards? 1.2. How effectively does the ISS address emerging issues (workers and community health and safety23, biodiversity offsets, influx,

gender based violence, climate change mitigation and adaptation)?

2. Is the Bank´s ISS effectively supporting the Bank´s strategic priorities and relevant to the current portfolio and the institutional changes?

2.1. To what extent is the Bank’s ISS relevant to the Bank’s strategic goals in relation to inclusive and green growth and implementation of the High 5s and the new changes from the DBDM? 2.2. To what extent is the Bank Group ISS relevant to the types of lending operations that the Bank currently has in place, including project finance, policy-based lending operations (PBOs) and Financial intermediaries (FI)?

II. Efficiency

3. What are the Bank´s internal factors that facilitate or constrain the implementation of the ISS as designed in relation to resources (human and financial), capacity (skills), systems, process and tools, and institutional incentives?

3.1. How adequate has been the allocation of resources and staffing (and their skills) for environmental and social safeguards work in

comparison to comparators24?

3.2. How adequate are the available guidelines, instruments and incentives to reflect the requirements of the ISS? 3.3. To what extent is there sufficient training and technical support to Bank´s operations staff to ensure efficient ISS implementation? 3.4. Are appropriate systems and tools available to ensure the quality control of ES outputs and to conduct periodic quality review? 3.5. Is there a common understanding of the objectives, principles, and requirements of the safeguards by all Bank staff? To what extent do they perceive the process to be clear, relevant, credible and useful? 3.6. To what extent the Bank has developed systems to disseminate lessons learned emerging from the application of the ISS?

23 Health and Safety (H&S) issues are included in ISS through OS5. Interviews during the evaluation inception phase indicated that this is a frequent ESS challenge during implementation. To the extent possible, the evaluation will explore how this is incorporated in the AfDB ISS and will compare it with the policy and practice in other MDBs. 24 The evaluation will compare the organizational structure of the AfDB Safeguards Department, and the level of seniority of its senior management, in comparison to other organizations, and come up with conclusions about its consequences to enforce the ISS. It will also cover how the staff is organized to deliver efficiently and how the work of different departments and complexes of the Bank is articulated to ensure ISS compliance.

8

Evaluation criteria and overarching questions

Specific evaluation questions

III. Emerging evidence on effectiveness

4. To what extent are the Bank´s quality assurance processes contributing to ensure compliance with ISS objectives (mitigation of environmental and social risks of Bank Group-funded projects)?

4.1. How adequate is the Bank’s ES categorization process in managing ES risks? 4.2. What is the overall quality of the ES outputs validated by the Bank prior to Board approval, including strategic ES assessments25 and ES Management Systems of FI? 4.3. What are the types of ES loan conditions and covenants included in the loan agreements and how are they enforced? 4.4. To what extent is the Bank in compliance with the ISS disclosure and meaningful consultation requirements before project approval and during implementation, especially in relation to resettlement and compensation? 4.5. How are project-level grievances requirements been met, especially those involving resettlement and compensation of persons affected by projects (PAP)? 4.6. To what extent is the Bank in compliance with the ISS OS2 requirements (involuntary resettlement, land acquisition, population displacement and compensation)? 4.7. To what extent the Bank’s supervision/completion documentation is able to capture the level of implementation of ES mitigation measures agreed at approval?

5. What is the level of ISS compliance from the emerging practice of the Financial intermediaries lending (category 4)?

5.1. How effectively is ISS being implemented for FIs (category 4 according to ISS) before Board approval and during implementation and completion? 5.2. Are additional resources required, with FI specific and management system expertise, to support the borrowers in achieving ISS compliance for FI projects?

6. What is the level of advance in relation to the ISS commitments to mainstream

climate change?

6.1. How well are the climate change requirements included in the

ISS integrated in the Environmental and Social Assessment

Procedures (ESAP)?

6.2. What has been the level of implementation of the tools to

operationalize the ISS commitment to mainstreaming climate

change?

25 The ESAP 2015 recognizes different types of safeguards tools at appraisal depending on the characteristics and ES identified risks of the project: ESIA, ESIA and ESMP, only ESMP (actionable document with the key ES mitigation measures), Strategic ESA (SESA) and Environmental and Social Management Framework (ESMF), Resettlement Action Plan (RAP).

9

4. Data collection and analysis, sampling and evaluation limitations

4.1. Data collection and analysis, including sampling Document and literature Review During the scoping stage of the evaluation, IDEV has undertaken a desk review of key documents about the ESS of the Bank, including the documents developed during the consultation process to develop ISS, current Bank´s procedures and guidance materials and evaluations conducted by comparator organizations.

Consultations and interviews with Bank´s stakeholders Extensive stakeholder interviews will be held both internally (AfDB) and externally with other MDBs to better understand their perspectives regarding the ISS design (appropriateness and ability to support the Bank in achieving its objectives), implementation to date and associated processes (efficiency & effectiveness). Suggested interviewees from the AfDB include Board Executive Directors, senior management, sector directors, task managers, legal experts and ESS specialists. Stakeholder interviews will be carried out using a semi-structured approach, with interviews guided by a framework of questions aligned to the evaluation matrix, and based on the potential insights or expertise of the individual stakeholder or group. All data from stakeholder interviews will be recorded and coded to identify recurrent themes across stakeholder groups. Specific interviews and focus groups will be held to assess the level of compliance of operations through FIs with the ISS (category 4 projects). A focus group based on the preliminary findings of the 36 desk reviews of projects which triggered the resettlement/compensation operational safeguards will be prepared to get further insights from operations and ESS staff. Comparator analysis A desk top document/literature review supported by focused interviews will allow to gain a more in depth understanding of ESS systems and processes of other comparator organizations. The focus of the work will be to:

Identify and assess current thinking and practice in relation to good international practice, and compare it with current AfDB policy and practice, to identify gaps;

Identify major differences in approach that might impact the ability of the AfDB to effectively co-finance projects with those organizations; and

Consider how other organizations are addressing emerging safeguards issues.

Ideally, for the purposes of this evaluation, comparators will include organizations with similar geographical coverage, similar sectors, size and scope of operations, and have conducted some level of review and upgrade to their safeguards system since the AfDB conducted its last review prior to the launch of the ISS in 2013, or are in the process of considering such a review. The potential ones are: (i) World Bank (WB) and the International Finance Corporation (IFC); (iii) Inter-American Development Bank (IDB) and IDB Invest; (iv) Development Bank of Southern Africa (DBSA); (v) European Bank for Reconstruction and Development (EBRD); (vi) European Investment Bank (EIB).

Stakeholder Surveys

A survey to Bank´s staff and Civil Society Organizations (CSOs) will be administered by email. The survey will include targeted questions to Bank´s operations staff in charge of appraisal, supervision and completion of public and private projects (task managers, investment officers, division managers of key sectors). It should capture, among others, Bank´s operations staff‘s

10

perceptions about how the current institutional arrangement for the management of ES safeguards has assisted them in identifying, managing and mitigating ES risks.

Projects Desk Review and field visits to assess compliance with ISS

The evaluation will review a total of 183 projects to respond various evaluation questions. Some of them will also entail interviews with borrowers and clients, and field visits. Table 3. Project desk reviews to inform the emerging effectiveness evaluation questions.

Evaluation component and

related EQ

Focus of the assessment and tool

used

Number of projects screened and sampling

method

Sample of projects with field visits

Analysis of ISS compliance (EQ. 4.1, 4.2, 4.3, 4.7)

Overall compliance with the Operational

Safeguards26. ESS Desk-Based Evaluation tool (ISS desk validation), Annex 4

89 public and private sector projects (40 before ISS + 49 after ISS) Sample aligned to IDEV’s Evaluation of Quality Assurance across the project cycle

4 in 3 countries (Cameroon, Kenya, Senegal)27

Case study on resettlement/ compensation (EQ. 4.4, 4.5, 4.6)

Rapid Assessment and thematic deep-dives (See Annex 5)

36 projects All public and private sector projects approved from July 2014 to December 2017 which triggered OS228

0, focus group with task managers and ESS specialists

Case study on category 4 projects (EQ 5.1 and 5.2)

Compliance with ISS requirements

58 projects All Financial Intermediary operations approved from July 2014 to December 2018, categorized as 429

10 Financial intermediary operations, 10 sub-projects (end borrowers) in Nigeria and Kenya

Assessment of climate mainstreaming (EQ 6)

Climate mainstreaming at project and CSP level

To be determined 0

26 The desk validation tool includes the assessment of the 5 Operational Safeguards approved as part of the ISS, the ESAP and the associated guidance materials, both at entry and during implementation/completion of the projects (when applicable). The tool underwent a pilot phase to ensure the relevance of the questions in relation to the Bank-validated documents used. Due to the limitations to find key ES outputs developed during the first two stages before Board approval (ES categorization memos, etc), the component about ES project compliance will be focused at the Project Appraisal Stage and the summaries of the ESIAs and RAPs done by the ESS staff. Qualitative data analysis techniques were used to analyze the observations and justifications provided by external sector experts in determining whether or not projects meet the criteria of the ISS desk validation tool. 27 Projects with interesting ESS features were purposefully chosen from the ones visited for BDEV’s Quality Assurance Evaluation in 2018. In addition to document review, interviews with task managers, executing agencies, National Environmental Management Authorities (NEMA), development partners and some clients were held. The objective of these case studies was to conduct an in-depth review of how the management of ES risks contribute to project implementation as well as how the Bank’s quality frameworks interact with contextual factors, including country and borrowers’ capacity. 28 The list was built selecting projects with a summary Resettlement Action Plan disclosed on the AfDB website

after July 2014 until December 2017. A total of 50 projects were identified, 14 of them were deleted because they are not active (not signed/cancelled), according to exchanges with task managers, legal department and ES officers. 29 The list was built selecting projects approved after July 2014 until December 2018 using the Board’s approval

information in the Bank’s intranet. Since that lsit does not include the ES categorization, the first selection was

done filtering by financing instrument (lines of credit, equity and guarantees). The evaluation team verified in the

Project Appraisal Reports that the selected projects were effectively categorized as 4. This yielded 91 category 4

projects approved during that period. 33 of the 91 projects were removed because they are not active (not

signed/cancelled), based on exchanges with the Task Managers/Investment Officers.

11

4.2. Limitations of the evaluation The main challenge of the evaluation is the comprehensive nature of the Integrated Safeguards System, which covers a very diverse range of environmental and social issues and is applied in very distinct contexts (54 countries) and to the differing financial instruments (loans, FIs, policy lending, etc). The inception phase has entailed a substantive document review and consultations to select key themes which can be representative of the issues and challenges faced in implementing the ISS. The assessment is more focused on the Bank’s perspective than the Borrowers, due to the

difficulty to embrace a high and diverse number of clients, sectors and regions. However,

during the field visits and interactions with borrowers and clients, their perspectives about the

ISS are integrated in the analysis. In addition, civil society were consulted through an e-survey.

One of the main limitations for the exercise is related to the inability to collect key ES project documents, especially about the identification and preparation stages30. To address this, IDEV will conduct a thorough document collection strategy, involving: (i) prioritized collection of required documents; and (ii) use of multiple databases to obtain the required documentation. As a last resort, IDEV will contact the safeguards department staff and task managers. According to interviews with Bank´s task managers and safeguards specialists, most of the exchanges about the ESS issues happen by email. They are not stored or summarized in any data base. Similarly, the upstream work to accompany borrowers when they prepare environmental and social assessments studies, as well as the required public consultations, was not possible to assess due to the unavailability of systematic documentation. Regular supervision reports tend to contain very limited ES information. The integrated data base to track the information about the ES due diligence conducted by the Bank across the project cycle (ISTS) is still not fully operational and little information could be extracted from it during the first stage of the evaluation. According to interviews, an additional limitation for private sector operations is that ES categorization has not always been validated by SNSC. Therefore, it is possible that some projects were mis-categorized as 3, lower ES risk. After the Bank decentralisation, SNSC has mainly worked on demand from sector departments to categorize and validate ES studies, and all stakeholders interviewed so far recognized the bulk of work has been concentrated on public sector operations31. An obvious limitation of the exercise when assessing the real compliance of projects with the ESS requirements is that it relies on the information included in Bank generated documents. This second stage of the evaluation will reinforce the analysis of borrowers’ documents about resettlement/compensation components of projects as well as financial intermediaries’ ESMS and reports.

30 The first stage of the evaluation found serious difficulties to find key ES documentation of the identification and preparation stages of the project cycle. For instance, the project brief was only found for 6% of the 85 public investment projects of the Quality at entry sample and the memo of validation of the ES categorization was found for very few projects. Moreover, 83% of the Bank summaries of ESA studies were not sign-off dated versions. This jeopardizes the capacity for evaluators to work with finalized and validated versions. 31 However, the % of compliance with categorization requirements has improved from 50% in 2012 to 92% in 2016, according to AfDB Status on ES categorization for projects approved by the Board (SNSC, 2017).

12

5. Deliverables, Level of Effort and Timeline

IDEV will work collaboratively with external consultants to complete major milestones of the

process of evaluation. The main deliverables are: (1) finalization of the inception report

(including all data collection tools), (2) Technical report responding to relevance questions, (3)

report on the results of the stakeholders’ e-survey, (4) Technical report responding to the

efficiency questions, (5) Finalization of the analysis of ISS compliance (including an in-depth

analysis of OS2 compliance), (6) Technical report about the ISS compliance of a sub-sample

of FI interventions and (7) Summary evaluation report.

The evaluation began in December 2018 and will be completed by August 2019. IDEV will have a collaborative approach and work with the consultants, ensuring quality and usefulness of the evaluation for the Bank. Within IDEV, functional responsibility for this evaluation is with a task manager who works under the guidance of IDEV management.

6. Evaluation team and quality assurance of the evaluation process The evaluation team is composed of a Principal Evaluation Officer and a junior consultant, and a communication and dissemination consultant from IDEV, supported by a senior ESS expert, an expert on compensation/resettlement issues (OS2) and an expert on the management of ES risks of financial intermediary operations (ISS category 4 operations). The evaluation counts with an internal and external peer reviewer.

The evaluation team will: (i) include the main stakeholders in decision making about evaluation design and implementation, (ii) inform on the evaluation activities and progress and (iii) communicate interim findings. This will be done through preliminary interviews, telephone and email exchanges with key Bank’s staff to scope the evaluation. The final inception report will be shared with the Reference Group and discussed in a meeting, as well as draft versions of the technical and summary evaluation reports. An audit trail will be developed to ensure transparency about the inclusion or rejection of the comments received.

The 20 members of the Evaluation Reference Group (ERG) come from the following Bank’s departments: (1) SNSC0 (Safeguards and Compliance Department), (2) RDGW4/SNSC (Safeguards and Compliance Department – West African region), (3) BCRM0 (Compliance Review and Mediation Unit (BCRM), (4) Climate Change & Green Growth PECG2, (5) Civil Society Division (AHGC2), (6) AHGC (Gender, women and civil society), (7) SNOQ1 (Operations Committee Secretariat and Quality Assurance), (8) PIFD – private sector origination team, (9) PIFD private sector supervision team, (10) PINS2 (Non Sovereign Operations - NSO & private sector support), (11) Water Security and Sanitation Division AHWS0, (12) PESD (power systems and development), (13) ECGF (governance and public financial management, (14) PICU (infrastructure and urban development), and representatives of the six AfDB’s Group regions (North, South, Western, Central, East and Nigeria Office).

13

7. Communication and dissemination of findings and recommendations The following table summarizes the main stakeholders around the evaluation, their interests, participation and proposed engagement method. Table 4. Evaluation stakeholders’ mapping.

Stakeholder and category Interests in relation to the ISS and its evaluation

Participation or role Engagement/ communication method

African Development Bank

The Board of Directors and Senior VP Offices - primary, internal

Better understand the ISS functioning and decision-making on ESS standards

Provide their view and use the ISS evaluation findings and recommendations to make decisions about changes needed for strengthening of the safeguards system.

Interviews with selected EDs, Advisors

Specific briefs about key ISS issues to be produced after the evaluation report is finished

Sectors and Regions Complexes32 - primary, internal

Understand problems with compliance and identify strategies to Improve enforcement of ESS

Provide information on implementation of ES safeguards in their Operations, share challenges faced and best practices

Use the ISS evaluation findings and recommendations to improve their internal organization to ensure ESS compliance

Participation in the Evaluation Reference Group (ERG)

Share the assessment of projects in their sectors / regions in relation to resettlement or FI operations

Active participation in organizing a focus group with task managers dealing with projects entailing resettlement/compensation

Financial sector department (PIFD) -primary, internal

Case study about the FI operations (ISS category 4 projects)

Help in documentation collection for category 4 projects assessed

Help in reaching out borrowers to be interviewed in the field visits

Participation in the ERG and continuous exchanges during the evaluation of category 4 projects (FI operations)

Safeguards & Compliance Department (SNSC)-- primary, internal

Improve their performance in dealing with Operations to ensure ESS compliance

Showcase the advances on ISS implementation and areas of improvement

Environmental and Social Safeguards Specialists to be interviewed for the evaluation

Help in documentation collection and in reaching out to borrowers to be interviewed

Participation in the ERG and exchanges during the evaluation

Jointly prepare a workshop after the evaluation about the findings of IDEV’s evaluation, their MTR and the ESAP revision

32 See the full list in Section 6. SNSC, BCRM, AHGC, PIFD and PEGC are also part of the ERG but they are treated differently in this stakeholders mapping due to their special involvement in the evaluation.

14

Stakeholder and category Interests in relation to the ISS and its evaluation

Participation or role Engagement/ communication method

African Development Bank (continuation)

Climate Change & Green Growth Officer, PECG2 - primary, internal

Showcase the advances on CSS implementation and areas of improvement

Help in documentation collection for CSS screening and GHG accounting cases

Participation in the Reference Group of the Evaluation and close exchanges about the Climate Safeguards System as part of the ISS

Compliance Review and Mediation Unit (BCRM) - primary, internal

To learn more about the ISS implementation and procedures to prevent complaints

Coordination with their ongoing IRM Review

Provide key information about key ISS issues which have triggered complaints

Provide key information learnt from the spot-checks they regularly conduct

Participation in the ERG and exchanges during the evaluation

Active participation in dissemination of lessons learned from the evaluation and their spot checks

Gender and Women Empowerment (AHGC1) – primary, internal

Improve the coordination between the social assessments of projects as part of ISS and gender assessments

Be interviewed and share key documentation

Participation in the ERG

Explore potential joint dissemination session of findings with IDEV’s evaluation synthesis on gender equality

Civil Society and Community Engagement (AHGC2) – primary, internal

Include the views of CSOs about the ISS

Provide advice about how to engage civil society groups in the evaluation, and the list of validated continental CSOs for the e-survey

General dissemination of the evaluation report

Ensuring report is disseminated to CSOs

Operations staff in general, both at headquarters and regional and country offices - internal, primary

Apply lessons learned from the evaluation to prevent and/or remedy non-compliance in on-going and future projects.

Provide information about their projects, if necessary

Participation in interviews, focus groups

General dissemination of the evaluation report

The Independent Development Evaluation

Strengthening development impact and ensuring sustainability of Bank Group’s portfolio

Mobilize the necessary expertise and resources.

Conduct the evaluation, ensuring quality is ensured.

15

Stakeholder and category

Interests in relation to the ISS and its evaluation

Participation or role Engagement/ communication method

Comparator organizations

International Finance Institutions (IFIs) with co-financed portfolio - primary, external

Learn about emerging safeguards issues to inform their own safeguards policies and operational standards

Learn about challenges of co-financing due to differing requirements, and good practices to overcome them

Provide information on ES safeguards including revised standards and policies

Exchange about their own experience with certain FI operations under evaluation

General dissemination of the evaluation report

Dissemination of the case study about the ES management of FI operations

Other development partners with ESS policies -primary, external

Bank’s clients and borrowers and other RMC’s authorities

Financial Intermediaries (FIs) – primary, external

Category 4 case study

Get information about the Bank’s ISS requirements

Provide key information about the ESMS and performance assessment

Dissemination of the case study about the ES management of FI operations

Other borrowers, including RMC - primary, external

Better understanding of Bank’s ISS requirements and their implementation

Interviewed during the country missions General dissemination of the evaluation report

National Environmental and Land Management Authorities33 – primary,

external

Better understanding Bank’s ISS requirements and their implementation

Provide information on sectoral guidelines, standards, permits and other administrative clearance guidelines that a project must comply with.

General dissemination of the evaluation report

Civil Society Organizations - secondary, external

Better understanding of the Bank’s ISS and mediation function

Potential answer to the e-survey

Provide information on project performance, concerns over specific issues

General Dissemination of the evaluation findings

The private sector - secondary, external

Better understanding of the Bank’s ISS requirements

Provide information on the challenges and opportunities for the Bank’s safeguards requirements.

Dissemination of the evaluation results to the FIs and IFIs they are associated with

The academia -secondary, external

Learn about the emerging safeguards knowledge and its impact on development

Share their expertise and provide information that could improve validity of the results, if necessary

Dissemination of the evaluation findings

Evaluation partners (e.g. ECG, OECD, EvalNet, EPRADI, AfrEA, etc) –(seconday, external)

Learn of emerging safeguards issues that could impact the activities of their members

Provide information on the safeguards issues emerging from the operations they fund

Generic methodology information of the evaluation

Dissemination of the findings of the evaluation

33 Ministries of Environment, environmental units at sector ministries, land management authorities dealing with compensation and resettlement

16

References 2001 AfDB Environmental and Social Assessment Procedures.

2003 AfDB Bank Group Financial sector policy

2005 AfDB Evaluation Guidelines for Private Sector Line of Credit Operations

2008 IEG Financing Micro, Small, and Medium Enterprises: An Independent Evaluation of IFC’s Experience with Financial Intermediaries in Frontier Countries. https://openknowledge.worldbank.org/bitstream/handle/10986/6485/448160PUB0Box327413B01official0use0only1.pdf?sequence=1&isAllowed=y

2009 Asian Development Bank, Safeguards Policy Statement. https://www.adb.org/sites/default/files/institutional-document/32056/safeguard-policy-statement-june2009.pdf

2012 AfDB, Integrated Safeguards System Working Progress: Strategic Choices made in the design of the ISS. Draft report on option for the ISS.

2013 AfDB Integrated Safeguards System, Policy Statement and Operational Safeguards, Safeguards and Sustainability Series, Volume 1, Issue 1

2013 European Investment Bank, Environmental and Social Handbook

2014 Asian Development Bank, Safeguards Operational Review. ADB processes, portfolio, country systems and financial intermediaries. Corporate evaluation study. https://www.adb.org/sites/default/files/evaluation-document/89401/files/ces-safeguards.pdf

2015 AfDB Involuntary Resettlement Policy: Review of Implementation. (Volume 1, Issue 3)

2015 AfDB ISS: Guidance Materials (Volume 2, Issue 1)

2015 AfDB Environmental and Social Assessment Procedures

2016 IADB Evaluation of Financial Intermediaries

2016 AfDB Development and Indigenous Peoples in Africa (Volume 2, Issue 2)

2017 AfDB South Africa CSPE

2017 IFC Interpretation Note on Financial Intermediaries. https://www.ifc.org/wps/wcm/connect/38d1a68049ddf966af3cbfda80c2ddf3/IN+on+FIs_Revised+April+11+2017.pdf?MOD=AJPERES

AfDB, 2017. Présentation retraite SNSC, Supervision de la Conformité Environnementale et Sociale des Projets.

2017 European Investment Bank, Evaluation of EIB Intermediated Lending through the Investment Facility in ACP. Operational Evaluation.

2017 Global Environment Facility (GEF) Review of the GEF Policy on Agency Minimum

Standards on Environmental and Social Safeguards. GEF/ME/C.52/inf.08

2018 IDEV LOC Synthesis Evaluation. Available at http://idev.afdb.org/en/document/do-lines-

credit-attain-their-development-objectives-evaluation-synthesis-2010-2017

(not dated) AfDB: E&S Management in the African Financial Sector: Training and consultancy on environmental and social management in financial intermediaries and microfinance institutions in Africa. An AfDB-FAPA project, Project Completion Report