evaluation of the bank assistance in the energy...

AFRICAN DEVELOPMENT BANK GROUP

INDEPENDENT DEVELOPMENT EVALUATION

(IDEV)

EVALUATION OF THE BANK ASSISTANCE IN THE ENERGY SECTOR

APPROACH PAPER

Version 6.0

January 2015

Evaluation of Bank Assistance in the Energy Sector -Approach Paper Ver6.0 (as of 15 January 2015)

Table of Contents

1. Evaluation Background and Context ........................................................................................ 1

Background ............................................................................................................... 1

The Bank’s Policy Framework in the Energy Sector ...................................................... 2

The Bank’s Engagement in the Energy Sector .............................................................. 3

Comprehensive Evaluation of the Bank’s Development Results .................................... 4 2. Purpose, Objectives and Scope ................................................................................................ 4

Purpose and Objectives ............................................................................................. 4

Scope ........................................................................................................................ 5 3. Key Evaluation Criteria/Issues and Questions ......................................................................... 5

Relevance ................................................................................................................. 5

Effectiveness ............................................................................................................. 6

Efficiency .................................................................................................................. 6

Sustainability ............................................................................................................. 6 4. Approach and Methodology .................................................................................................... 6

Analytical Framework ................................................................................................ 6

Overall Description of the Evaluation Approach and Methodology ............................... 7

Portfolio Review ........................................................................................................ 8

Literature and Policy Review ...................................................................................... 9

Country and Regional Case Studies ........................................................................... 10

Special Thematic Studies/Cluster Evaluations ............................................................ 11

Limitations .............................................................................................................. 12 5. Engagement and Quality Assurance Process ......................................................................... 12 6. Deliverables and Dissemination............................................................................................. 13 7. Timeframe and Management ................................................................................................ 13

Timeframe .............................................................................................................. 13

Management .......................................................................................................... 13 Annex-1: Evaluation Design Matrix ............................................................................................. A-1 Annex-2: Energy Related Policies and Initiatives alongside the Bank’s Energy Intervention ..... A-4 Annex-3: AfDB’s Results Measurement Framework for the Energy Sector ................................ A-6 Annex-4: Detailed Theory of Change ........................................................................................... A-7


1

1. Evaluation Background and Context Background

1.1 Inadequate access to modern energy services is an obstacle to economic growth and poverty reduction in Africa. Only about one-fifth of the Sub-Saharan population has access to electricity, compared with about one-half in South Asia1. Although East Asia, Latin America, and the Middle East have all added at least 20 percentage points to their electrification rates since 1990, Africa has still the lowest electrification rate of all regions (26 percent of households); as many as 547 million people do not have access to electricity2. The high level of poverty in Africa is partly due to a lack of access to electricity. 1.2 Modern energy services are, however, not affordable for the poor segments of the population. The costs of energy services are exceptionally high in Africa, due mainly to the heavy reliance on oil-based electricity generation. The cost of electricity generation is 0.18 USD/Kwh3, which is two to three times higher than the global average. Moreover, most African households live on very modest budgets and spend more than half of their recourses on food4, which in turn impedes continued access to modern energy services. The two main determining factors of affordability of energy services are the cost of services and household income. 1.3 Unreliability of energy services is another critical challenge for Africa. The entire installed generation capacity of the 48 Sub-Saharan countries, being stagnant since the 1980s, is 68 gigawatts, which is about one-tenth of that in other low-income regions. More than 30 African countries experience recurrent outages and load shedding, with opportunity costs amounting to as much as 2 percent of GDP5. Weak governance and regulatory frameworks at the national and regional levels have also hampered performance of service delivery. Poor management deteriorated facilities due to inadequate maintenance, poor performance of utilities and low-quality service. 1.4 Over the decades, however, some new developments have significantly influenced the development of the energy sector in Africa. These include increased emphasis on regional integration, with a particular focus on regional power trade through power pools6, new regional initiatives including the NEPAD Short-Term Action Plan for Infrastructure (NEPAD-STAP) and the

1 World Bank (2010), Africa’s Infrastructure: A Time for Transformation. Africa Infrastructure Country Diagnostic. V. Foster

and C. Briceño-Garmendia, eds. Washington, DC: World Bank and AFD. 2 African Development Bank (2012), Energy Sector Policy of the African Development Bank Group, AfDB. 3 World Bank (2009), World Bank Report, G8 Energy ministers meeting, Rome, 24-25 May 2009. 4 World Bank (2008), Access, Affordability, and Alternatives: Modern Infrastructure Services in Africa, Africa Infrastructure

Country Diagnostic. Sudeshna Banerjee and Quentin Wodon, eds. Washington, DC: World Bank and AFD. 5 World Bank (2009), World Bank Report, G8 Energy ministers meeting, Rome, 24-25 May 2009. 6 Namely the Southern African Power Pool (SAPP), West African Power Pool (WAPP), Central African Power Pool (CAPP), East

African Power Pool (EAPP) and Maghreb Electricity Committee (COMELEC) were established to pool energy resources and

promote the development of regional electricity markets in order to provide stable and reliable electricity at an affordable cost.


2

Program for Infrastructure Development in Africa (PIDA) 7, emergence of new donors in the infrastructure sector, and growing role of the private sector in the financing or the management of energy infrastructures. The Bank’s Policy Framework in the Energy Sector

1.5 The Bank has been involved in the energy sector since its creation. More recently, the Long-Term Strategy (LTS) recognizes infrastructure, including energy, as one of its core priorities. Over time, other policies, strategies or specific initiatives related to the energy sector have completed the relevant policy framework for the Bank’s assistance. Some of the key policy documents and initiatives are described below. 1.6 The 1994 Energy Sector Policy allowed the Bank to play a critical role, especially by supporting reforms at the national level. Nevertheless, many challenges outlined in the 1994 Policy remain, including security of supply and energy access; inadequate investments in energy; slow progress in renewable energy and energy efficiency; ineffective regional integration; inadequate energy access for rural development and agriculture; lack of capacity for implementing reforms; and inadequate capacities to manage environmental impacts. Moreover, the 1994 Policy did not adequately address challenges and opportunities presented by NEPAD and regional integration, climate change, and the Bank’s focus on infrastructure development.

1.7 The new Energy Sector Policy, launched in 2012, replaces the 1994 Energy Policy and constitutes the Bank’s response to the challenge of low access to modern energy services in Africa. It also better reconciles access and sustainability objectives, namely by addressing the emerging energy-related challenges posed by climate change. Consequently, the policy framework advances a dual objective: (i) to support Regional Member Countries (RMCs) in their efforts to provide all of their populations and productive sectors with access to modern, affordable and reliable energy services; and (ii) to help RMCs develop their energy sector in a socially, economically and environmentally sustainable manner. 1.8 To meet these objectives, nine key principles have been set in the draft Policy to guide the Bank’s interventions in the energy sector: (i) ensuring energy security and increasing access for all, (ii) moving towards a cleaner energy path, (iii) enhancing governance at the national level, (iv) innovating to increase financial flows in the energy sector, (v) integrating aid effectiveness principles, (vi) promoting social and environmental responsibility, (vii) integrating a response to climate change, (viii) fostering knowledge transfer and (ix) mainstreaming the gender dimension. 1.9 The Bank will focus on sub-sectors that are the most likely to address current and future energy demands, while contributing to the development of a sustainable energy sector. These include: (i) renewable energy, (ii) fossil fuels, namely coal, oil and gas, (iii) power transmission and distribution, (iv) regional cooperation, and (v) supply-side and demand-side energy

7 PIDA aims to support the development of priority regional and continental infrastructure projects up to the year 2040. The PIDA

Study and its recommendations are tabled for adoption by the AU Heads of State Summit in January 2012, as the priority program

to guide Africa’s infrastructure development.


3

efficiency. 1.10 Concurrently with the launch of the new Energy Policy, the Bank also developed Climate Change Action Plan (CCAP) for 2011-2015 in September 2012 in order to support its RMCs adapt to climate change and mitigate its effects while supporting the Bank’s focus on infrastructure development and regional operations. CCAP is organized around three pillars: (i) Low carbon development, (ii) Climate resilient development and (iii) Funding platform -- to help African countries strengthen their capacity to respond to climate change and to mobilize resources from existing and proposed sources of climate finance, the private sector and market mechanisms. The CCAP also includes advisory services, support to policy reform, knowledge generation and competency building that cut across all programs. The CCAP is consistent with existing Bank strategies and it takes into account the issues included in the Bank’s Long Term Strategy under development, especially on green growth. 1.11 The Bank plans to invest up to 6 billion UA between 2011 and 2015 to reach the CCAP targets.8 The resources are expected to be drawn from ADF-12 and the sixth General Capital Increase, bilateral trust funds and existing or new Climate Finance instruments, including the Climate Investment Funds (CIF), the Global Environmental Facility (GEF), the Congo Basin Forest Fund (CBFF) and the ClimDev-Africa Special Fund. The Bank has the opportunity to further expand its investments through a more streamlined and efficient use of its existing facilities. 1.12 The Bank has participated in different regional initiatives with development partners, including: (i) The NEPAD Infrastructure Project Preparation Facility (IPPF), (ii) The Infrastructure Consortium for Africa (ICA), and (iii) PIDA. The details of some of these initiatives, together with the Bank’s related policy and plan, are summarized in Annex-2. The Bank’s Engagement in the Energy Sector 1.13 Bank’s Investment Operations: The Bank’s financing of energy infrastructure programs has significantly increased over the last decade: the total commitments between 2000 and 2013 amounts to 7.6 billion UA. Around one-sixth of the total commitments, i.e. 16.1 percent9, were allocated to the energy sector, especially for financing power-supply schemes. 1.14 A total of 133 energy sector projects were approved during the period from 2000 to 2013.10 The total funds allocated to the energy portfolio increased by more than fifteen times between 2000 to 2013, from UA 40 million to approximately UA 650 million. This increase reflects efforts to align the Bank's activities with its medium-term strategies, which emphasize infrastructure development in Africa as a key priority.

8 Current funding from the United Nations Framework Convention on Climate Change (UNFCCC), as well as from other sources,

is generally reported to be very inadequate for Africa’s actual needs. Of all global climate funds disbursed in the past 4 years,

Africa’s share was 12%, or about $700 million for all project funding. Also, this funding relies primarily on voluntary contributions;

there is an enormous gap between the funds required and those made available for Africa. The requirements to address adaptation

are estimated at $10-$20 billion, yet only between $50-$100 million flows each year to Africa (Africa Progress Panel, 2010). 9 Estimated by IDEV. 10 The projects include those categorized as “power” in the SAP database and those approved between 2000 and 2013.

Supplementary loans were accounted in the original projects.


4

Figure-1: Bank’s Energy Sector Projects by Year (Left) and by Sub-Sectors (Right, net

loan base)

1.15 Bank’s Analytical Work: The Bank has delivered a number of analytical and advisory products over the decade. Although these remain limited, they have increased in the last years of the evaluation period.

Comprehensive Evaluation of the Bank’s Development Results 1.16 At the request of the Board’s Committee for Operations and Development Effectiveness (CODE) of the Bank, IDEV has launched a comprehensive evaluation of the Bank’s development results (CEDR). The comprehensive evaluation will be completed by 2016. The CEDR is meant to be a comprehensive, integrated and evidence-based assessment of the Bank’s development results across sectors, countries and regions; public and private operations; lending and non-lending activities. The main questions the CEDR seeks to examine and better understand is whether the Bank is achieving desired results; why these results are achieved or why not; and whether the Bank is optimally using and leveraging its resources to generate inclusive and sustainable growth that makes a measurable difference in the daily lives of ordinary Africans. 1.17 The study period of the CEDR covers ten years from 2004 to 2013. It starts in 2004, the year after the 2003 Strategic Plan of the Bank came into effect and just ahead of the time when results-based Country Strategy Papers (CSP) were introduced in the Bank in 2005. 1.18 The evaluation of the Bank’s assistance in the energy sector, this evaluation, is an essential part of the main building blocks within the CEDR. This evaluation is expected to contribute to provide evidence and quantitative analysis to support the synthesis of findings and lessons learned under the CEDR.

2. Purpose, Objectives and Scope Purpose and Objectives

2.1 The goal of the evaluation is to inform the Bank’s future strategic and operational approach to energy sector assistance, by identifying emerging trends in the sector, assessing how


5

the Bank has responded to these trends, taking stock of the results of the Bank’s assistance and drawing lessons for future work. The evaluation combines two objectives of (1) accountability, through determining the extent the Bank has contributed to the development of the energy sector in RMCs and (2) learning, by identifying the lessons on how the Bank can contribute most effectively to improving the energy sectors of its RMCs. Scope

2.2 Definition of “Energy Sector”: The energy sector is categorized, in this evaluation, in accordance with three sub-sectors based on physical and economic characteristics as follows:

Power-supply schemes (including power plant construction, transmission line provision, distribution network development, rural electrification, etc.);

Renewable energy projects or assistance (including climate change-related projects or assistance); and

Oil and gas projects or assistance. 2.3 Coverage: All public and private sector energy infrastructure operations, analytical work (studies, ESW, etc.) and activities related to institutional strengthening and capacity-building will be included. Currently, the universe includes at least 133 Bank-financed energy projects/studies approved during the 14-year period from 2000 to 2013. 3. Key Evaluation Criteria/Issues and Questions 3.1 The questions below are tentatively selected with regard to the issues of relevance, effectiveness, efficiency and sustainability of the Bank’s operations in the energy sector, with reference to the evaluation goal and objective. The questions will be further narrowed down and specified following the portfolio review and the literature/policy review, and they will provide a framework for the country and regional case studies as well as special thematic studies (cluster evaluations). This will allow the evaluation to conduct a more in-depth assessment of a range of limited issues that are essential to identify lessons for the Bank’s operational effectiveness in the energy sector. Relevance

Key Questions: How relevant are the Bank's policies and activities in the energy sector to the needs of recipient counties?

o To what extent do the Bank’s activities in the energy sector were aligned with the priorities of RMCs while providing an added-value by introducing reforms?

o To what extent the Bank realistically assessed RMCs’ drivers for change, re-form ownership and institutional capacity for answering to the needs of the population?

o To what extent the Bank’s interventions adapted over time, taking into account RMCs’ implementation performances and emerging challenges?


6

Effectiveness

Key Questions: To what extent has the Bank contributed to the development of the energy sector in RMCs?

o To what extent the Bank’s expected development short term (intermediate) outcomes were achieved?

o To what extent the non-lending activities (Economic and Sector Work as well as policy dialogue) and Bank's new institutional mechanisms contributed to achieving the outcomes of Bank’s energy sector projects?

o To what extent Bank’s monitoring been supportive to achieving the expected short-term (intermediate) outcomes?

Efficiency

Key Questions: To what extent has the Banks assistance been delivered efficiently? o To what extent the Bank’ identification, design and approval mechanisms and

human resources contributed to ensure a swift and efficient implementation of the energy projects?

o To what extent Bank’s energy portfolio incurred delays and cost overruns in delivering expected outputs?

o To what extent Bank’s supervision been supportive to achieving the expected outputs?

Sustainability

Key Questions: To what extent has the Bank’s assistance in the energy sector contributed to sustainable results?

o To what extent the Bank contributed to have RMCs securing financial resources to ensuring recurrent costs of the infrastructure constructed or rehabilitated?

o To what extent the Bank contributed to have RMCs operationalizing an institutional framework ensuring value-for-money of energy infrastructure maintenance funding?

3.2 The evaluation design matrix is attached as Annex-1, with sub-questions, judgment criteria, source of information and methodology to be applied, for each criterion. 4. Approach and Methodology Analytical Framework

4.1 The generic intervention logic (or theory of change) for the energy sector in Africa, illustrated below (Figure-2), is the basis for the analytical framework of this evaluation. The figure explains the multidimensional causal links that go beyond provision of physical infrastructure to encompass the broader outcomes occasioned by the Bank’s assistance. 4.2 Based on the literature and policy review, the generic theory of change shall be refined and a detailed theory of change (including assumptions and risks) should be developed tailored


7

to the sector context in Africa, by referring to the Bank’s results measurement framework for the energy sector (shown as Annex-3) and to the result-based logframes set out in the Country Strategy Papers (CSPs). The first version is presented as Annex-4. The generic theory of change is nevertheless provided to help ensure consistency in the levels and the language applied. 4.3 Information from the portfolio and country case studies will allow a mapping of the Bank’s strategic approaches and results in the energy sector with a detailed theory of change. This will help identify the intervention approaches adopted by the Bank and pathways to specific outcomes intended at country and regional level.

Figure-2: Theory of Change for Energy Intervention in Africa (First Version11)

Overall Description of the Evaluation Approach and Methodology

4.4 The evaluation will follow a phased approach. The first phase of the evaluation consists

11 Theory of change will be completed with other information including the context factors, assumptions,etc.

Increased Reliability of Energy Services

Improved Energy Network Performance

Improved Transmission/Grid System

Optimal Usage of Power Pool

Enhanced Operation of Power Pool

Reduced Cost of Input Goods and Services

Consumer Price Reduction

Competitive Environment for Private Sector

Favorable Environment for Pubic Service Delivery

Long-Term Outcomes

Increased Energy Access

Increased Affordability of Energy Services

Increased Energy Efficiency

Greater Use of Renewable Energy

Improved Grid Operation

Efficient Operation of Power Plant

Improved Fuel Purchasing Strategy

Effective Asset Management of Physical Infrastructure

Longer Life of Physical Infrastructure

Short-Term Outcomes

Physical Infrastructure

• Power plant

• Transmission lines

• Substations

• Oil and gas plant

• Oil and gas pipelines

• Renewable energy plant

• Smart grid

Policies, Management and Capacity

•Regulatory framework on energy industry

•Power pool system

•Operation and maintenance skills of energy infrastructure

•Climate change-related funds

Output

Civil Works

Capacity Building

Financial Resource

ESW

Policy Dialogue

Input

EQ-4: To what extent the Bank’s assistance in energy has contributed to sustainable results

EQ-1: How relevant are the Bank's policies and activities in the energy sector to the needs of recipient counties?

EQ-3: To what extent has the Bank contributed to the development of the energy sector in RMCs?

EQ-2: whether the Bank’s assistance is efficiently delivered or not

Energy Assistance InterventionLogic

Evaluation Questions


8

of desk reviews, including a literature/policy review and a portfolio review. These reviews will bring some evidence for answering the evaluation questions and will support the narrowing down of evaluation questions for the second phase. The second phase of the evaluation will involve the conduct of country and regional case studies as well as special thematic studies, which provide an in-depth assessment of the Bank’s assistance at both the country and regional levels, following the refinement of the evaluation questions. A synthesis report will then be prepared with key findings and results of analysis from the first and second phase, and a final summary report presenting the overall findings, lessons learned and recommendations for the Bank will be compiled by October 2015. 4.5 Bank staff interviews at headquarters will assist to collect primary and meta information for portfolio review and policy/literature review. Focus group discussions, key informant interviews and data collection, including survey questionnaires, at the Bank field offices and project stakeholders will be conducted to collect primary and secondary information for the field case studies.

Figure-3: IDEV Energy Sector Evaluation – Overall Schematic Design

Portfolio Review

4.6 The portfolio review will examine the nature of the Bank's support to the energy sector since 2000, including its composition by sub-sectors, objectives and expected results and development effectiveness. In this regard, the review will cover to the following;


9

To overview the trends of the Bank’s energy lending and approvals, and to assess the results achieved by projects and the external and internal factors of success. The review will include an analysis of the portfolio according to region, country, sub-sector, type of loan, project cost, loan amount, etc. It will include an assessment of the share of the energy sub-sector in the Bank’s portfolio, the use of instruments (investment programs, sector budget support, TA and capacity-building) and the share of co-financing in Bank’s programs. In addition, the review will identify the results achieved by completed projects and the lessons learned for increased effectiveness, efficiency and sustainability. The FIRR and EIRR will be re-calculated if sufficient data is available. Measures of efficiency and comparison with other Multilateral Development Banks (MDBs) will also be included. Information from PPERs/PCRs/PCR-ENs/XSRs/XSR-ENs will also be used.

To compare the results achieved with the general theory of change in the energy sector, and to refine the evaluation questions. The specific type of the Bank’s energy intervention will be identified. The Bank’s “input”, “output” and “outcome” (consisting of short-term and long-term ones, as shown in Figure-2 and Annex-4) in the energy projects/programs will be identified, so that the Bank’s actual results can be mapped to the general theory of change. A comparison of the two diagrams, the general theory of change and the Bank’s actual results, will elucidate the intervention approaches of the Bank over the evaluation period.

To provide and synthesize the necessary evidence for answering at this stage the evaluation questions set for relevance, efficiency, effectiveness and sustainability. The results of analysis obtained through the above tasks will be incorporated as an evidence to answer each evaluation questions.

4.7 The information for the portfolio review will be captured from an assessment of the SAP database, PARs, PCRs, PPERs, interviews with Task Managers and secondary data on socio-economic and physical indicators of the respective countries in which the projects were undertaken. Literature and Policy Review

4.8 Emerging Trends and Lessons in the Energy Sector: The main objective of the literature review is to guide the refinement of the theory of change in the energy sector, identify the main developments that have influenced the energy sector in Africa, and refine the evaluation questions. It will include:

A review of the relevant literature in order to identify the developments that have influenced the energy sector in Africa (climate change, PPP, regional integration, power pool initiative, private sector, new donors, etc.) from 2000 and examine how these concepts have influenced the development community;

A review of the evaluations/research produced by the MDBs and other institutions with regard to the successes and failures of energy assistance in Africa; and

A proposal for focusing the second phase of the evaluation on some key issues.


10

4.9 Evolution of the Bank’s Policy Framework: A policy review will compare the Bank’s energy policy with the energy policies of other development agencies especially the World Bank and the European Community as well as selected bilateral agencies (e.g. JICA) that play a key role in the energy infrastructure development in Africa. The review will also assess the extent to which the evolution of the policy framework of the Bank, including the 1994 energy policy, the Infrastructure Department Three-Year Business Plan, the Regional Integration Strategy Papers (RISPs) and other collective initiatives in which the Bank is participating (e.g. PIDA), has allowed the Bank to respond to emerging needs in the energy sector. The review will also assess the value added of the Bank’s approach to the energy sector in comparison to those of other development partners. 4.10 The literature and policy reviews will be conducted through document reviews. The methodologies used for the policy review will also include semi-structured interviews and focus groups. Country and Regional Case Studies

4.11 These country and regional case studies will assess the Bank’s overall assistance to RMCs in the energy sector by looking at the achievement of outcomes, coherence with the goals and objectives of the Bank’s related policies, and the comprehensiveness of the Bank’s approach in addressing energy sector issues in RMCs. The studies include four to five in-depth field case studies, one of which focus on power pool initiative-related projects implemented in multiple countries, while the three to four case studies examine the results achieved at the country level. The criteria for the selection of the case studies will be determined by the intermediate results of the portfolio and literature and policy reviews12. 4.12 At country level, the evaluation will firstly investigate how the Bank’s activities are implemented in line with the countries’ priorities and the Bank’s policy framework. In addition, the evaluation will assess to what extent the Bank adopted an integrated approach in the energy sector at country level, by referring to the Bank’s overall policy framework (since 1994), by assessing the use of different instruments and the synergies between lending and non-lending activities. A review of the CSPs of the selected countries will assess to what extent this integrated approach is discussed at a strategic level. Finally the appropriateness of the project design at approval will be assessed, particularly in the extent to which the Bank has integrated emerging trends13 in the design of its projects. A review of project results will also be undertaken for completed or nearly completed projects, particularly focusing on short-term or intermediate outcomes. 4.13 With regard to region-wide viewpoints, the case study will assess to what extent the Bank

12 Countries will be selected, taking into account the characteristics of projects/programs (in which the portfolio review is to identify

the type of their intervention), type of case study approach, availability of PCRs, relative weight of the different sub-sectors in the

energy portfolio and type of countries (middle income, lower income or fragile states). Country selection will also consider the

relationship between the project status and the evaluation criteria (i.e., evaluation questions relating to relevance will be answered

by both on-going and completed projects, while those of effectiveness and sustainability can only be referred to the completed

ones.). 13 These trends will be elaborated through literature and policy reviews, as mentioned.


11

adopted a holistic approach for its intervention at regional level, mainly through reference to the Bank’s regional integration strategy. To overview the region-wide intermediate outcomes (increased energy access, for instance) achieved by the Bank’s assistance, case studies may focus on the interconnector projects. 4.14 The case studies will include i) semi-structured interviews with stakeholders including executing agencies, Bank’s field office, power pool coordination committee 14 , Independent Power Producers (IPPs), energy regulators, and related private sector institutions, ii) site visits to investigate both completed and on-going projects, iii) in-depth interview with direct beneficiaries such as cargo owners, freight forwarders, truckers and farmers, and iv) use of PPER information. 4.15 The following table demonstrates the combination of planned case study approaches mentioned above and the set of possible countries/projects, with analytical viewpoints and planned survey instruments. The study will be supported by national consultants for data collection and logistics.

Table-1: Possible Approaches to Country and Regional Case Studies Approach

Types Focus on Analysis/ Methodology Survey Instruments/ Tools

Single-Country Review of country specific strategy (CSP) Appropriateness of the design at approval

for projects Review of activities, process, objectives

and results of a single-country operation Intervention performance as an integrated

and holistic approach Sustainability

Stakeholder interviews Project site visits PPER type surveys for power-supply

scheme projects and/or renewable energy projects

In-depth interview with direct beneficiaries (20-30 samples including private firms, direct beneficiaries, etc.)

Multiple-Country

RISP review Review of region-wide “outcomes” (e.g.,

increased energy access, lower cost of energy supply, etc.)

Appropriateness of the design at approval for on-going power pool-related projects

Sustainability

Stakeholder interviews Project site visits In-depth interview with beneficiaries

(20-30 samples including key industries, private firms, direct beneficiaries, etc.)

Special Thematic Studies/Cluster Evaluations

4.16 The above case studies will be further supported by special thematic studies. A cluster evaluation will be designed to provide insight into the specific theme, including regional power interconnection and rural electrification, of Bank-financed energy projects and the development benefits they have generated. 4.17 Focusing on regional power interconnection as specific theme is driven by the fact that it contributes to regional integration (which is one of the Bank’s strategic pillars) and helps reduce the cost of electricity generation, as part of “on-grid” electricity supply systems, that could

14 This may include SAPP (Southern Africa Power Pool), EAPP (East Africa Power Pool), or WAPP (West Africa Power Pool).


12

significantly improve the overall energy supply efficiency in Africa.15 Rural electrification is also selected, given that it is another critical issue for African population living in rural and sub-urban areas under “off-grid” electricity supply systems. The focus is on rural electrification and not the broader field of rural energy16. Limitations

4.18 The evaluation of the Bank’s effectiveness is limited by the number of completed projects and availability of PCRs. There are 33 projects that were completed during the period from 2000 to 2013 and only 13 PCRs are available. This limitation will be partially overcome with in-depth case studies and additional information provided by special thematic studies/cluster evaluations. 4.19 Another limitation is that data on energy interventions by the Bank are not clearly available and interventions made by non-energy departments will not necessarily be recorded as the category of “power” in the SAP database. While the SAP project recording system will be used to build a picture of energy investments, this will not capture all energy commitments (such as the energy-related institutional strengthening projects under some SAP categories) and will need to be supplemented by building data from project appraisal reports, which will be labor and time intensive. 4.20 The evaluation will not assess the individual initiatives such as the IPPF and PIDA, as these initiatives have independent mechanisms for performance review. 5. Engagement and Quality Assurance Process 5.1 The evaluation study team will maintain contact with Bank stakeholders (mainly operational complex) throughout the whole evaluation process. IDEV will circulate the approach paper, desk reviews, regional and country case studies, special thematic studies (cluster evaluation reports) and the final summary report to the Bank’s stakeholders for comment and feedback. In doing so, IDEV will put together short briefs and/or power point presentations to communicate the key messages arising from the evaluation to facilitate interaction with stakeholders. 5.2 In addition, IDEV will request the departments involved in the energy sector to nominate a reference person to facilitate daily contacts and support the process of gathering data and comments.

5.3 Some peer-reviewer(s) will be selected in order to ensure the quality and credibility of the evaluation. Those will include internal IDEV peer-reviewers as well as the external peer-

15 In this case, interconnecting electricity grids can lower electricity costs by reducing needs for electric generating capacity and

allowing cheaper fuel to be substituted for more expensive fuel Moreover, regional power pools that allow countries to share the

most cost-effective energy resources can reduce electricity costs by $2 billion a year, according to the estimation by the World

Bank. 16 For more details, refer to “Assessing the effectiveness of AfDB-funded Rural Electrification Projects - Cluster Evaluation

Approach Paper, Draft of October 2014” and “Assessing the effectiveness of AfDB-funded Power Interconnection Projects -

Cluster Evaluation Approach Paper, Draft of 25 September 2014”.


13

reviewer. 6. Deliverables and Dissemination 6.1 The outputs for the evaluation will be:

- Two desk review reports: the policy/literature review and the portfolio review, - Regional and country case studies report(s), - Special thematic study report(s), and - A final summary report to CODE

6.2 The final summary report will be sent to CODE for consideration. The other documents (see above) will contribute as key inputs to the final summary report and will serve as background papers to provide both quantitative and qualitative evidence. A two-page brief will be prepared and published on highlights of key findings, lessons learned and recommendations. 7. Timeframe and Management Timeframe

7.1 The work plan is expected to involve the following steps and timeline. Detailed work plan is attached as Annex-4.

Table-2: Work Plan (Tentative) Description of Tasks / Key Deliverables

Responsibility Inputs/ Consultation

First Draft Final Draft

Approach Paper Task Manager IDEV, ONEC, ONRI (ICA)

n.a. Mid Jan.2015

Portfolio Review Task Manager / Senior Research Assistant

IDEV, ONEC, ONRI (ICA), OPSM3

Mid Jul.2015 Mid Sep.2015

Policy / Literature Review Task Manager / Consultants


End Jun.2015 End Oct.2015

Regional and Country Case Studies

Task Manager / Senior Research Assistant / Consultants


End Dec.2015 End Feb.2016

Special Thematic Studies –Cluster Evaluation

Senior Evaluation Officer / Consultants


End Dec.2015 End Mar.2016

Final Summary Report to CODE

Task Manager n.a. End May 2016 End Jun 2017

Management

7.2 The Task Manager for this evaluation is Hajime Onishi, Principal Evaluation Officer, in IDEV1. He will be supported by Joseph Mouanda, Senior Evaluation Officer, who will lead the special thematic studies and Samson Houethossou, Senior Research Assistant, who will assist to conduct the portfolio review. Consultants will be hired to support the literature and policy review, regional/country case studies and special thematic studies.


14

7.3 Under the overall guidance of Rafika Amira, Division Manager IDEV1 and Rakesh Nangia, Evaluator General of IDEV, the Task Manager will provide inputs and lead the work of the consultants, and will produce the final summary report to CODE. The Task Manager will be responsible for organizing communication processes with stakeholder within and outside the Bank, with the support of the Knowledge Management Division (IDEV3).


Annex-1

Annex-1: Evaluation Design Matrix

1. Relevance: How relevant are the Bank's policies and activities in the energy sector to the needs of recipient counties?

Sub-Questions Judgment Criteria or Performance Indicators (Tentative) Sources of Information Methodology

To what extent do the Bank’s activities in the energy sector were aligned with the priorities of RMCs while providing an added-value by introducing reforms?

The energy sector strategy set in Bank’s CSPs (if any) is aligned with RMCs sector policy/strategy and sub-sectors’ Action Plans.

The Bank’s CSPs introduced or facilitated reforms improving the RMCs overall performance of the energy sector policy framework, notably by introducing more consistency with its energy sector strategy.

The Bank contributed to elaborate a sector or sub-sector policy/action plan when missing or incomplete (coverage, priorities, principles, institutions).

SAP PARs CSPs Departmental Records Bank Staff

Literature Review Policy Review Country and Regional Case Studies Staff Interview

To what extent the Bank realistically assessed RMCs’ drivers for change, re-form ownership and institutional capacity for answering to the needs of the population?

The Bank assessed realistically the drivers and obstacles for change and integrated them in its sector strategic response (in CSPs).

The Bank secured RMCs’ commitments to energy sector reform, in line with the Bank’s sectoral theory of change and strategy (for example as regards institutional restructuring, commercialization, cost recovery from infrastructure users and environmental sustainability).

PARs ESWs CSPs Departmental Records Bank Staff

Literature Review Policy Review Country and Regional Case Studies Special Thematic Studies(Cluster Evaluations)

To what extent the Bank’s interventions adapted over time, taking into account RMCs’ implementation performances and emerging challenges?

The design and approval stages allowed the Bank to assess realistically political economy constraints, institutional weaknesses, market failures and any other issues including underlying risks and assumptions.

PARs CSPs Country Portfolio Performance Review Departmental Records Bank Staff

Policy Review Portfolio Review Country Case Studies Staff Interview

2. Effectiveness: To what extent has the Bank contributed to the development of the energy sector in RMCs? Sub-Questions Judgment Criteria or Performance Indicators (Tentative) Sources of Information Methodology

To what extent the Bank’s expected development short term (intermediate) outcomes were achieved?

The Banks’ interventions contributed to specific measurable benefits as per each project design (results-based logical framework).

Unintended consequences (positive or negative) different from the above were recorded after Bank’s projects completion.

PCRs PPERs Country Portfolio Performance Review Bank Staff

Portfolio Review Country and Regional Case Studies Special Thematic Studies(Cluster Evaluations)

To what extent the non-lending activities (Economic and Sector

The major changes in RMCs energy policy framework and programming can be allocated to Bank’s non-lending activities.

Departmental Records PCRs

Literature Review Policy Review


Annex-2

Work as well as policy dialogue) and Bank's new institutional mechanisms contributed to achieving the outcomes of Bank’s energy sector projects?

The Bank’s new mechanisms were set in place timely and with adequate resources to fill-up their missions.

The Bank’s new mechanisms demonstrated their added-value and contributed to the outcomes achieved by the Bank's energy interventions.

PPERs Country Portfolio Performance Review Bank Staff

Portfolio Review Country and Regional Case Studies Special Thematic Studies(Cluster Evaluations) Staff Interview

To what extent Bank’s monitoring been supportive to achieving the expected short-term (intermediate) outcomes?

The Bank ensured that timely monitoring data was available from a reliable and updated set of indicators at project and sector level.

The Bank’s country teams used monitoring data for project supervision.

The Bank’s project monitoring made a consistent use of updated economic and financial analyses (IRRs, NPVs, B/C).

Supervision Reports PCRs PPERs Departmental Records Bank Staff


3. Efficiency: To what extent has the Banks assistance been delivered efficiently?

Sub-Questions Judgment Criteria or Performance Indicators (Tentative) Sources of Information Methodology

To what extent the Bank’ identification, design and approval mechanisms and human resources contributed to ensure a swift and efficient implementation of the energy projects?

The Bank’s energy projects appraisal included a comprehensive range of assessments (engineering design, sector political economy, institutional governance and performances, PFM, corruption…).

The Bank’s made a consistent use of economic and financial analysis (IRRs) at appraisal stages, including systematic testing of alternative designs.

The Bank implemented internally a specific and reliable quality control mechanism prior to approval for avoiding overambitious, overoptimistic designing or budget underestimation by task teams.

The assumptions and risks identified by each project are closely monitored afterwards.

SAP PARs Supervision Reports PCRs PPERs QaE Evaluation Reports Departmental Records Bank Staff


To what extent Bank’s energy portfolio incurred delays and cost overruns in delivering expected outputs?

The Bank’s energy portfolio faced delays and cost overruns similar to those incurred by other infrastructure sectors such as transport or water and sanitation.

Bank’s design options, unreliable cost estimates or IRRs are minor among the key issues faced during energy projects’ delayed implementation.

Procurement of Bank financed projects were conducted in a timely manner.

SAP Supervision Reports PCRs PPERs Departmental Records Bank Staff


To what extent Bank’s supervision been supportive to achieving the expected outputs?

The Bank’s staff was in a position to diligent sufficient supervision missions, with the required mix of expertise.

SAP Supervision Reports PCRs

Portfolio Review Country and Regional Case Studies


Annex-3

The Bank’s supervision reports provided with a balance and realistic view of the implementation prospects (ownership, reform undertaking, timeliness, cost, and setting of a reliable monitoring system).

PPERs Departmental Records Bank Staff

Special Thematic Studies(Cluster Evaluations) Staff Interview

4. Sustainability: To what extent has the Bank’s assistance in the energy sector contributed to sustainable results? Sub-Questions Judgment Criteria or Performance Indicators (Tentative) Sources of Information Methodology

To what extent the Bank contributed to have RMCs securing financial resources to ensuring recurrent costs of the infrastructure constructed or rehabilitated?

The Bank contributed to a realistic assessment of the range of resources available with RMCs/concession holders for energy infrastructure maintenance, modernization and extension.

The Bank’s interventions in the energy sector contributed to collect sufficient financial resources for energy infrastructure maintenance and operations.

The Bank’s interventions ensured that long-term financial RMCs’ commitments to infrastructure maintenance were enshrined by PFM or ring-fenced.

SAP PARs Supervision Reports PCRs PPERs Departmental Records Bank Staff


To what extent the Bank contributed to have RMCs operationalizing an institutional framework ensuring value-for-money of energy infrastructure maintenance funding?

The Bank contributed to establishing or reinforcing autonomous agencies in charge of programming and managing infrastructure maintenance.

The Bank contributed to the promotion of private sector for RMCs’ maintenance works and eventually excluding force account.

The Bank contributed to setting by RMCs anti-corruption and transparency policies or measures in procurement and supervision of maintenance works of energy infrastructure.

ESWs Supervision Reports PCRs PPERs Departmental Records Bank Staff



Annex-4

Annex-2: Energy Related Policies and Initiatives alongside the Bank’s Energy Intervention

Policy/Initiative Date Main Objectives Focus Remarks

AfDB Energy Policy

1994 (To set out principles and policy framework for all energy operations)

NEPAD- IPPF 2004 To assist African countries, RECs and related institutions to prepare high quality and viable regional infrastructure projects and programs, develop consensus and partnerships for their implementation with the long-term goal of reducing Africa's economic marginalisation by ensuring sustainable regional economic development and integration through cooperation among African countries, donors and the private sector.

It supports regional infrastructure development projects in the following sectors: energy, transport, ICT, and water resources management. The activities eligible for financing under the Fund are: (i) prefeasibility studies; (ii) feasibility studies; (iii) project structuring; (iv) capacity building for infrastructure development; and (v) facilitation and creation of an enabling environment for regional infrastructure development.

The Bank hosts and manages the NEPAD-IPPF. IPPF’s strategic business plan was finalized in 2011.

Infrastructure Consortium for Africa (ICA)

2006 To encourage, support and promote increased investment in infrastructure in Africa, from both public and private sources.

Energy

Transport

ICT

Water

The ICA was launched at the G8 Gleneagles summit in 2005. Housed in the Bank, the mission is to improve the lives and economic well-being of Africa's people by catalysing donor and private sector financing for infrastructure projects and programs in the region.

It contributes to the capturing of trends in financial commitments by sources of funds and beneficiary country.

AfDB Infrastructure Department Three-Year Business Plan 2008-2010

2008 To scale up sustainable and inclusive infrastructure development in the energy, transport and ICT sectors at the national and regional level by playing the triple role of financier, adviser and strategic partner.

Energy

Transport

ICT

Programme for Infrastructure Development in Africa (PIDA)

2010 (2011)

To facilitate increased regional integration in Africa through improved regional and continental infrastructure for energy, transport, ICT and water management.

PIDA will finance studies that can then be used to help identify and support large-scale investments in regional infrastructure.

Energy

Transport

ICT

Trans-boundary water

PIDA was launched in 2010 as a merger of the NEPAD Medium to Long Term Strategic Framework (MLTSF) and the African Union Commission (AUC) Master Plans and Continental Policy Initiatives.

The PIDA Study was successfully concluded in 2011 with the key outputs being the Africa Sector Outlooks 2040, the Infrastructure Development Program and the Implementation Strategy and Process.

Climate Change Action Plan (CCAP) for 2011-2015

2012 To support RMCs adapt to climate change and mitigate its effects while supporting the Bank’s focus on infrastructure development and regional operations.

CCAP is organized around three pillars: (i) Low carbon development, (ii) Climate resilient development and (iii) Funding platform -- to help African countries strengthen their capacity to respond to climate change and to mobilize resources from existing and proposed sources of climate finance, the private sector and market mechanisms.

AfDB new Energy Policy

2012 To support RMCs in their efforts to provide all of their populations and productive sectors with

(i) renewable energy, (ii) fossil fuels, namely coal, oil and gas, (iii) power transmission and

The Policy replaces the 1994 Energy Policy and constitutes the Bank’s response to the challenge of


Annex-5

Policy/Initiative Date Main Objectives Focus Remarks

access to modern, affordable and reliable energy services; and

To help RMCs develop their energy sector in a socially, economically and environmentally sustainable manner.

distribution, (iv) regional cooperation, and (v) supply-side and demand-side energy efficiency.

low access to modern energy services in Africa. It also better reconciles access and sustainability objectives, namely by addressing the emerging energy-related challenges posed by climate change.

Annex-6

Annex-3: AfDB’s Results Measurement Framework for the Energy Sector (Source: AfDB Energy Sector Policy (2012) Annex-I) Outcome Outcome indicators Output Output indicators Key activities and initiatives contributing to

achieving outcome Assumptions

/Risks

1. Increased access, affordability and reliability of energy services

1.1. % of population with access to electricity (increase from baseline) 1.2. Primary energy consumption per capita (increase from baseline) 1.3. Number of days of outages per year (decrease from baseline)

1. Investment flows at the national and regional levels to bridge infrastructure gap is increased (UA million increase from baseline)

1.1. AfDB’s lending volume (UA million increase from baseline) 1.2. Total volume of investments from the private sector and other donors (UA million increase from baseline) 1.3. MW of total generation capacity installed (increase from baseline) 1.4. Km of transmission, distribution lines built or rehabilitated (increase from baseline) 1.5. Number of newly connected households and business (grid and off-grid connections)

Support for national and regional energy production, transmission and distribution Support for rural electrification projects/programs Promotion of enabling environment to enhance private-sector participation Enhancing partnerships for project co-financing Support for capacity development

Political stability and strong commitment from RMCs Synergy with other donors Continued economic growth on the continent

2. Increased energy efficiency

2.1. Energy savings (increase from baseline)

2. AfDB investment in energy efficiency is effective and has increased

2.1. Efficiency gains (increase from baseline) Support for infrastructure rehabilitation and demand-side efficiency initiatives

3. Greater use of renewable energy and clean technologies

3.1. % of renewable energy in total generation capacity (increase from baseline) 3.2. CO2 emissions per Kwh (decrease from baseline)

3a. Financial flows in renewable and clean energy is increased (increase from baseline) 3b. Clean technology transfer, research - development are enhanced and effective

3a.1. AfDB lending volume for clean energy (UA increase from baseline) 3a.2. % of climate financing in Bank-supported projects (increase from baseline) 3a.3. % of private-sector investments for clean energy (increase from baseline) 3b.1. Tons of CO2 avoided through use of clean technologies (in particular, through retrofitting or development of greenfield coal projects) 3b.2. Number of new clean technologies schemes installed (increase from baseline) 3b.3. Number of operations and volume of financing in support of research-development and innovation.

Investments in renewable grid based and off-grid power plants Promotion of cleaner technologies in power generation Promotion of risks mitigating instruments Establishment of carbon market support mechanisms suitable for Africa Development and use of subsectors (coal, hydropower and biofuel) and options assessment guidelines Support for clean technology transfer, R&D and capacity building

4. Enhanced governance and increased viability of the sector

4.1. Financial and technical performance of energy utilities/ enterprises 4.2. Financial flow in the sector (Increase from baseline)

4. AfDB’s support for sector governance and regulatory reforms and capacity development is effective and enhanced

4.1. Number of countries where policy dialogue led to reform of energy sector aimed at increasing efficiency, governance and low carbon growth

Advisory services at three levels: policy dialogue, assistance for institutional reforms and transaction assistance

Strong commitment from RMCs Coordination with other actors

Annex-7

Annex-4: Detailed Theory of Change (Tentative, as of Jan.2015)

Context Input Output

Physical input

(incl. civil works,

materials,

equipment, etc.)

Donor coordination

Capacity building

on energy sector

resources

Goals

(Impact)

Poverty reduction

Assistance

(through TA,

ESW, policy

dialogue) for

competitiveness

improvement of

energy sector

Physical

infrastructure

• Power plant

• Transmission

lines

• Substations

• Oil and gas

plant

• Oil and gas

pipelines

• Renewable

energy plant

• Smart grid

Intermediate Outcomes

Increased power supply by

renewable energy

Increased affordability of

energy services

The lack of

energy

infrastructure is

one of the

serious

bottlenecks in

Africa.

Increased

production and

related costs

makes African

countries among

the least

competitive in the

world.

Long-Term Outcomes

Consumer price

reduction

Competitive

environment for

private sector

Favorable

environment for pubic

service delivery

Improved energy

network system

performance

Accelerated rural/

urban electrification

Greater use of eco-

friendly energy source

Regional power

pool system

Establishment of

energy-related

funds (climate

change, etc.)

Regulatory

framework

(pricing policies,

tariff setting,

emissions

trading, etc.) on

energy industry

Improved

operation and

maintenance

skills of energy

system and

infrastructure

Reduced energy

supply/distribution cost

Increased energy

production and access

Improved grid/transmission

operation

Efficient operation of

energy supply/

distribution system

Accelerated power

interchange among regions

Effective asset

management of physical

infrastructure

Regional

integration

Intra- and inter-

regional trade

facilitation

Economic

development

Increased reliability

of energy services

Longer life of physical

infrastructure

Reduced cost of

input goods and

services

Positive

environmental

impact (e.g. CO2

reduction)

Electricity tariff

reduction

Inclusive growth

Green growth

Improved fuel

purchasing strategy

Improved emissions

trading system

Reduced load

shedding

Enhanced usage of

power pool

Annex-8

evaluation of the bank assistance in the energy...

Documents