Evaluation Approach Impact Evaluation of Rural Electrification in Bhutan December 2009 Team Leader: Ganesh Rauniyar, Senior Evaluation Specialist (email: grauniyar@adb.org) Contact: evaluation@adb.org A. Background

1. The Independent Evaluation Department (IED) plans to conduct an impact evaluation of rural electrification (RE) in Bhutan in 2010 and the Study is expected to feed into Bhutan's country assistance program evaluation. The proposed study is expected to provide empirical evidence of development effectiveness of the Asian Development Bank (ADB) assistance to the country. RE is considered to have significant impact on socioeconomic welfare of rural population. ADB has been supporting Bhutan's RE initiative since 1996 with the approval of Rural Electrification Project (Loan 1375-BHU[SF]) in 1995. Three subsequent loans have been approved—Sustainable Rural Electrification Project (Loan 1712-BHU[SF]), Rural Electrification and Network Expansion Project (Loan 2009-BHU[SF]), and Green Power Development Project (Loans 2463-BHU and Loan 2464-BHU[SF]). ADB has already signaled its support for yet another project, which is expected to contribute to the Government's goal set out in the Tenth Plan to provide universal RE by 2013.1

RE accounts for slightly more than one-fourth of the ADB assistance provided to Bhutan in the last 15 years. ADB's supports for RE have had strong justification for economic growth, poverty reduction, and better quality of life for rural people. An overview of the supported projects is summarized in Appendix 1.

2. ADB has completed project completion reports (PCRs) for Loans 1375-BHU(SF), 1712-BHU(SF), and 2009-BHU(SF);2

1 The Vision 2020 target of Electricity for All was brought forward and is to be achieved by the end of the Tenth Plan

(2013).

and rated the first two projects as successful, and the last one as highly successful. IED conducted project performance and audit report (PPAR) of the first project and validation reports of the PCRs for the second and third projects. The ratings assigned by IED were consistent with PCR ratings. The PPAR of Loan 1375-BHU[SF] concluded that the evidence was sketchy on reducing income poverty as a result of the Project. The validation report of the PCR on Loan 1712-BHU[SF] restated PCR statement that insufficient project-specific data was available to assess the impact of the Project on beneficiaries. A sample survey conducted by the PCR mission suggested that project's impact varied widely, depending on location and beneficiaries. It was found that households used electricity primarily for lighting and half of them also used for cooking (primarily rice). Use of other appliances such as water boilers, refrigerators, televisions, and flying pans or curry cookers were less common. The usage of appliances was not as widespread as considered at appraisal. Similarly, the validation report of the Loan 2009-BHU[SF] noted that the PCR did not assess the achievement of two impact performance indicators (i) increased income in rural

2 ADB. 2002. Completion Report: Rural Electrification Project. Manila; ADB. 2007. Completion Report: Sustainable Rural Electrification Project. Manila; ADB. 2008. Completion Report: Rural Electrification and Network Expansion Project. Manila.

Asian Development Bank. 6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines Tel +63 2 632 4444; Fax +63 2 636 2163; evaluation@adb.org; www.adb.org/evaluation

2

2

Appendix 2

2 A

ppendix 2

areas, and (ii) improved education and health facilities through electrification. The lack of objective data did not permit the PCR mission to assess social and environmental impacts. 3. ADB Operations in Bhutan's Rural Electrification. In 1995, only 20% of rural households were electrified.3 ADB assistance to Bhutan in RE commenced with the approval of $9.5 million Rural Electrification Project4

on 19 September 1995, which became effective on 8 February 1996. At the end of the Project, ADB financed $6.64 million of the $8.21 million actual project cost from Asian Development Fund (ADF) resources. The loan closed on 6 April 2000 with approximately 18 months delay, which was largely associated with less than adequate consideration of Bhutan's mountainous terrain in some aspects of technical design, delay in procurement, malfunctioning of some transformers, incompatibility of some supplies ordered by the consultants, unavailability of sufficient community labor, and transportation caused by heavy rains. The Project began operation in December 1999; and it provided electricity connection to 3,120 households, and distributed 567 electrification kits at a minimal charge. According to the PPAR, at the end of the Project, RE in Bhutan increased to 24%.

4. ADB approved $9.8 million loan to finance the Sustainable Rural Electrification Project5 on 25 November 1999, and the loan became effective on 14 January 2000. Actual project cost was $11.6 million against the appraisal estimate of $12.5 million; and ADB financed $9 million of the project costs from ADF resources. The loan closed on 12 January 2006 with two extensions and nearly 22 months delay, largely associated with the supply of procured goods. According to the PCR,6

the project provided access to electricity to 8,090 new rural consumers, 32% more than envisaged at appraisal. In addition, it also provided off-grid electrification in the form of 100 solar panels to monasteries and other community institutions located remote from the existing grid.

5. The $12.8 million Rural Electrification and Network Expansion Project7 was approved by ADB on 30 September 2003, and loan became effective on 31 March 2004. The actual project cost amounted to $13.2 million, of which ADB financed $9.7 million. The loan closed on 19 December 2009, earlier than the expected 31 March 2007 stated at appraisal. The PCR8

stated that the project provided access to electricity to 9,206 new consumers (15% more than appraisal estimates).

6. On 29 October 2008, ADB approved the Green Power Development Project9

3 ADB. 2004. Project Performance Audit Report: Rural Electrification Project in Bhutan. Manila.

with a focus on regional clean power trade and renewable energy access for the poor. Total project cost is estimated to be $234.45 million, of which ADB is to finance $51.00 million from ordinary capital resources, $29.00 million on hard-term ADF loan, and $25.28 million as ADF grant. The first two funding are to go towards financing Dagachhu hydropower, and the last part is to support RE. The contribution of other development partners to the Project includes Asian Clean

4 ADB. 1995. Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical Assistance Grant to the Kingdom of Bhutan for the Rural Electrification Project. Manila.

5 ADB. 1999. Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical Assistance Grant to the Kingdom of Bhutan for the Sustainable Rural Electrification Project. Manila.

6 ADB. 2007. Completion Report: Sustainable Rural Electrification Project in Bhutan. Manila. 7 ADB. 2003. Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical

Assistance Grants to the Kingdom of Bhutan for the Rural Electrification and Network Expansion Project. Manila. 8 ADB. 2008. Completion Report: Rural Electrification and Network Expansion Project in Bhutan. Manila. 9 ADB. 1999. Report and Recommendation of the President to the Board of Directors: Proposed Loans, Asian

Development Fund Grant, Technical Assistance Grant, and Administration of Grant to the Kingdom of Bhutan for the Green Power Development Project. Manila.

3

Energy Fund ($1.00 million for RE), Austrian Export Credit Agency ($55.46 million for the Dagachhu hydropower), Government of Bhutan ($45.01 million for the Dagachhu hydropower, and $6.70 million for RE), and Tata Power Company ($21.00 million for Dagachhu hydropower). RE is expected to electrify 8,767 households and facilities; and provide 119 solar photovoltaic systems to schools, health clinics, monasteries and other community facilities.

7. Other Development Partners in Rural Electrification in Bhutan. Four other key players in RE are Austria, India, Japan, and the Netherlands. Under the support of the Netherlands Government, the Sustainable Rural Electrification Project (Phase II) was implemented from 2003 to 2007 in six districts, and project savings were used to electrify households in the seventh district. A sum of Nu301 million (approximately US$ 6.4 million) was provided for the project, and it covered 3,150 households.10

Data provided by the Bhutan Power Corporation (BPC) suggests that a total of 5,271 households have been electrified with funding from the Netherlands, and another 2,053 households with Austrian funds. In addition, under the energy and poverty initiative, the United Nations Development Programme has financed community-based rural energy project and a completion report of the project will be made available to IED in a month time. There is no verifiable data to suggest how many households were electrified with India's assistance, but it would be safe to assume that the first batch of electrification (20% of the households) was supported by India.

8. Future Plans for Expanding Coverage. IED estimates put RE coverage in 2009 below 55% to 60%. However, Bhutan is endeavoring to meet national goal of universal electrification by 2013 and efforts are underway towards this end. Japan International Cooperation Agency is supporting electrification of 15,272 households, and ADB will support electrification of additional 8,759 households under Loan 2464-BHU[SF]. Additional support has been sought from ADB and a project preparatory technical assistance team is already in the field.

9. Lessons. The PPAR of Loan 1375-BHU[SF] concluded that availability of RE may not be a sufficient condition for the poor to move from poor to non-poor status. This requires the poor's access to supplementary ingredients such as land, microfinance, and skill development. RE projects aimed at poverty reduction need to be accompanied by other efforts that increased poor's access to markets, skills, and seed capital. B. Rationale for the Study

10. Strategy 2020 has identified energy as a key component of the infrastructure core area of operations which drives ADB's three complementary strategic agendas of inclusive growth, environmentally sustainable growth, and regional integration. The Strategy states that ADB will also continue to invest in rural infrastructure, covering irrigation and water management, rural roads, and RE—services that particularly benefit women. In addition, the 2009 Energy Policy of the Bank recognizes that the Millennium Development Goal (MDG) targets cannot be met without modern energy services and recognizes that access to energy is essential to reducing poverty. The Policy states that ADB will support the developing member countries’ sustainable RE efforts designed to provide electricity to all, especially to the rural population. More specifically, ADB will focus especially on remote communities that are less likely to be connected to the electricity grid in the near future. ADB will develop small-scale demonstration projects that can be replicated in other locations, such as remote mountain villages or island

10 Bhutan Environs and Engineering Services (BEES). 2009. Project Evaluation of Rural Electrification Project

(Sustainable Rural Electrification – II). Final Report. Thimphu.

4

4

Appendix 2

4 A

ppendix 2

communities. Such projects will be packaged into larger bankable-size projects and, if feasible, will be added to main energy sector projects as a special energy access component. 11. The recent Independent Evaluation Group (IEG) study11

noted that quantification of benefits from RE is most commonly restricted to lighting benefits, with small number of analyses, including television viewing. Other benefits are sometimes mentioned but not quantified. For example, RE does not, in general, drive industrial development; but it can spur growth of home businesses. Such businesses mostly employ family labor and increase their hours once electricity becomes available. Electrification, thus, provides a small, but not negligible, boost to the incomes of some households. In addition, RE benefits quality of health services, and lowers costs by extending opening hours and significantly strengthening the cold chain for vaccines—though it does not increase the extent to which such services are offered. Electrification was found to reduce worker absenteeism in both health clinics and schools by improving living conditions and morale. However, the report also recognizes that the evidence base supporting above claims remains thin and, hence, further analysis is required to draw definitive conclusions.

12. While ADB has funded 28 RE projects in 11 countries from 1995 to 2009, the development effectiveness of ADB assistance in RE has not been quantified and, hence, is largely unknown. The proposed study will quantify intended and unintended impacts due to RE, and will provide a better understanding of development impacts so that future ADB assistance to the developing member countries can be more effectively directed in achieving intended economic growth, poverty reduction, and better quality of life for the rural people. The Study will also contribute knowledge to the global body of literature on impact of RE. C. Objectives and Scope of the Study

13. The Study has five specific objectives. First, the Study will evaluate the performance of Loans 1712-BHU[SF] and 2009-BHU[SF] using relevance, effectiveness, efficiency, likely impact, and sustainability based on document reviews, analysis of available data, and interviews with key stakeholders. Second, it will undertake a review of ADB portfolio of RE projects and assess the share of RE in total energy portfolio over the past 20 years, and then analyze project design and monitoring frameworks to determine quality of indicators identified at entry. Third, conduct a literature review complementing earlier work undertaken by IEG, and determine a set of pragmatic indicators for evaluating welfare impacts of RE. A provisional conceptual framework of the study appears in Appendix 2, which envisages impact in six key areas: (i) household income, (ii) education of children, (iii) health status of household members, (iv) fertility behavior of the adult household members, (v) gender roles in household decision making and control over resources, and (vi) environment. While determining impact on household welfare, the Study will also identify causal factors associated with respective impact variables. Fourth, the Study will provide a comparable baseline data for future impact evaluation of RE component under Loan 2464-BHU[SF] and panel data for future impact evaluation of preceding two loans. Fifth, relevant lessons and emerging recommendations will be derived based on empirical findings for improving design and implementation performance of future RE projects.

11 IEG. 2008. The Welfare Impact of Rural Electrification: A Reassessment of the Costs and Benefits. IEG: World

Bank. Washington, D.C.

5

14. The IED Scoping Mission in October 2009 noted that it would not be possible to find counterfactual villages for the first RE projects because majority of those electrified villages were in the peri-urban and sub-urban areas. On the other hand, villages identified for electrification under Loan 2464-BHU[SF] would be suitable as counterfactual villages for the second and third projects (Loans 1712-BHU[SF] and 2009-BHU[SF]). Hence, the proposed study will evaluate impact of the second and third projects. Furthermore, due to identical nature of intervention between ADB and other development partners, no specific differentiation will be made with respect to source of funding for RE.12

Impact will be evaluated both at the household and village levels.

D. Conceptual Framework for the Evaluation Study

15. Impact at the Household Level. The IEG study reported that the impacts of RE on indoor air quality,13

health, knowledge, and fertility reduction are quantifiable and significant. Similarly, RE has some long-term impact on home businesses. RE have potential to provide better health through reduced indoor air pollution, and reduced time burden on women of fuelwood collection. However, the report also recognizes that electricity is largely not use for cooking in rural areas. Furthermore, access to information through mass media, such as televisions and radios, is expected to raise awareness of health issues, resulting in changed health behavior and, eventually, improving health outcomes and lower fertility. The literature argues that electrification also can affect time use such as watching television, greater participation in community activities and socializing, reducing time spent on household work or shifting it to the evening, increased time spent reading—or for children doing homework, and extending hours of home businesses. The plausible channels through which electrification may affect education are (i) by improving the quality of schools, either through the provision of electricity dependent equipment or increasing teacher quality and quantity; and (ii) time allocation at home, with increased study time. The IEG report cites findings from the Philippines and takes note that children in electrified households have higher education levels and children achieve better grades in electrified households compared to the households without electricity. While evidence supporting the positive impact of RE on productive activities is limited, the number of home-based enterprises has steadily increased contributing to self employment of household members and higher household incomes.

16. The conceptual framework for this study is guided by review of relevant literature, and is based on program theory that links input, output, outcome, and impact. An operational logic model (Appendix 2) is adapted from recent IED report on impact of rural water supply and sanitation in Punjab, Pakistan. The model is based on discussions with key informants during the Scoping Mission in October 2009, qualitative findings reported in PCRs. The impacts have been grouped into four categories—economic, environmental, institutional, and social. The economic benefit has been assumed on two fronts—increase in household income as a result of income generating opportunities (specifically home-based enterprises) and savings in energy costs. In addition, spin-off impacts in terms of additional investments in infrastructure and services are also expected at the village level. The environmental impact is perceived in two ways—improved indoor air quality and preservation of forest cover (reflected by reduced use of firewood at the household level). The expected impact on institutions include better customer satisfaction, and improved financial position of implementing agency based on 12 According to the Department of Energy, the nature and scope of rural electrification was uniform irrespective of

funding source. 13 It is argued that the use of traditional solid fuels such as fuelwood, dung and crop residues exposes people—

especially women and young children—to indoor air pollution, with consequent health risks: principally, acute lower respiratory infections, but also low birth weight, infant mortality, and pulmonary tuberculosis.

6

6

Appendix 2

6 A

ppendix 2

improved tariff structure. The social impacts are foreseen in four ways—health, education, security, and gender empowerment. Tentative contributing outcomes are outlined in the logic model, and these will be refined by the Independent Evaluation Mission, in consultation with the executing and implementing agencies. An Evaluation Matrix for the study is presented in Appendix 3. E. Methodology

17. The proposed evaluation will adopt a mixed method approach. Bamberger, et al.14 and White (2008)15

argue that mixed-method evaluation combines the detailed insights and holistic understanding obtained from qualitative research with the ability to generalize to a wider population offered by quantitative data collection. Thus, it allows for a more comprehensive analysis. Mixed-method designs can be employed to strengthen validity, fine-tune sampling and instrumentation, extend the coverage of findings, conduct multi-level analysis, and generate new and diverse insights.

18. Impact Evaluation Design. Impact evaluation is conducted to answer a key question—what would have happened to that household/village receiving the electricity in the absence of electricity? Since, it is impossible to observe same household/village both with and without the electricity, it is important to develop a counterfactual similar to electrified household/village. The main challenge is to identify valid counterfactuals. There are three types of evaluation design: (i) experimental or randomized, (ii) quasi-experimental, and (iii) non-experimental. The experimental or randomized design involved randomly assigning the households into treatment (receiving electricity) and control (not receiving electricity) groups. This design is generally considered the most robust of the evaluation methodologies. The process of randomization ensures that before the intervention takes place the treatment and control groups are statistically equivalent, on average, with respect to all characteristics, observed, and unobserved. Randomized experiments solve the problem of selection bias by generating an experimental control group of people who would have participated in a program but who were randomly denied access to the program or treatment. Any differences in the average outcomes of the two groups after the intervention can be attributed to the intervention. Quasi-experimental design involves developing a comparison groups by using matching and reflexive techniques. Under such design, intervention sites are compared with non-intervention sites by using statistical methods to account for differences between the two groups. Under non-experimental design, intervention sites are compared with non-intervention sites by using statistical methods to account for differences between the two groups. 19. It is extremely difficult to randomize infrastructure projects such as electrification and road construction, and hence one is left with quasi-experimental methods to properly evaluate the impacts of RE. There are two methods commonly used to create counterfactual using quasi-experimental design: (i) matching on observables, known as Propensity Score Matching (PSM); and (ii) Instrumental Variable (IV) method. A simple comparison of households/villages with and without electricity will not tease out the true causal effect of electrification on outcomes for the simple reason that these households or villages are not similar at the outset. They may vary in initial characteristics and also their ability to access to electricity. The estimated benefits may be

14 Bamberger, M., J. Rugh, and L. Mabry. 2006. Mixed-Method Evaluation. Real World Evaluation: Working Under

Budget, Time, Data and Political Constraints: Chapter 13. Sage Publications. California. 15 White, H. 2008. Of Probit and Participation: The Use of Mixed Methods in Quantitative Impact Evaluation. Network

of Networks on Impact Evaluation Working Paper No. 6. World Bank. Washington, D.C.

7

due to the differences in households or villages characteristics rather than in having electricity. This issue can be addressed by constructing a counterfactual by using matching techniques. The PSM method calculates for both treated (with electricity) and untreated (without electricity) samples, the probability of treatment or electrification as a function of household or village characteristics from a logit or probit model. This probability of adopting electricity, calculated for households both with and without electricity, is called propensity score. The outcomes of households with electricity are then compared with those of unelectrified villages or households. The PSM method, however, only controls for selection bias due to observed characteristics but not for unobserved characteristics or endogeneity. Since the program placement is typically not a random process, it may be possible that selected villages for electrification are very different from unselected villages. To circumvent the endogeneity bias, IV method would be more relevant. The idea is to first identify suitable instruments that can influence households/village access to electricity, but not the error term in the regression equation; that is to say, instruments should not directly affect the outcome, but only through the access to electricity. 20. The Study will involve four steps: (i) preparation, (ii) data collection, (iii) data analysis, and (iv) report writing. The preparation step will entail: (i) a comprehensive literature review of evaluation methodologies and impact of RE projects implemented by selected development partners; (ii) development of detail research methodology, and determination of counterfactuals, sample size, and analytical methods; (iii) identification of indicators, and preparation of qualitative and quantitative data collection instruments; and (iv) pre-testing of data collection instruments. The data collection step will involve: (i) collection of secondary data at the village level; (ii) conduct of focus group discussions and key informant interviews; (iii) household-level data based on face-to-face interviews with responsible household male and female members; and (iv) data entry, verification, triangulation, and preparation for data analysis. The data analysis step will require subjecting data to appropriate qualitative and quantitative analysis. The fourth step will involve preparation of draft report based on data analysis and information gathered, peer reviews, interdepartmental reviews, and independent external reviews. Relevance of the ADB assistance in RE will be evaluated on the basis of (i) relevance at the time of project design, in terms of ADB and government policy and strategy; (ii) relevance of project design; (iii) relevance during implementation, and (iv) relevance at project completion and evaluation. Effectiveness will be assessed in terms of process, as well as implementation arrangements. Cost-effectiveness measure will provide an indication of efficiency. Sustainability will be analyzed based on institutional, financial as well as other enabling environment factors. 21. A review of methodological approaches in impact evaluation literature suggests that the focus group discussions at the community level; key informant interviews with local leaders and knowledgeable persons in the community, including school teachers and health practitioners, would be appropriate qualitative tools for data collection. In addition, recent studies indicate that a quasi-experimental research design will be appropriate for the quantitative analysis (Bamberger and White 2007;16 Pattanayak, et al. 2007; World Bank 2006). Since both 1712-BHU[SF] and 2009-BHU[SF] do not have household level baseline data, the Study will be limited to with and without (treatment and comparison group) design (Bamberger, et al. 2006; World Bank 2006; Vaessen and Todd 200717

16 Bamberger, M. and H. White. 2007. Using Strong Evaluation Designs in Developing Countries: Experience and

Challenges. Journal of MultiDisciplinary Evaluation. Vol 4(8):58–73.

). Selection biases in such design are inherent but these can be addressed using appropriate econometric techniques as discussed in paras. 18 and 19.

17 Vaessen, J. and D. Todd. 2007. Methodological challenges in impact evaluation: The Case of the Global Environment Facility. Discussion paper 2007-01. Institute of Development Policy and Management. University of Antwerp.

8

8

Appendix 2

8 A

ppendix 2

22. Structured questionnaires will be developed to gather village and household data.18 Since rural respondents do not keep records, responses to household surveys will rely on the recall method. The study team will make use of customer database maintained by BPC to assess trends in electricity consumption by households and local businesses. At the village level, a set of guiding questions by focus area will be used in all focus group discussions. Similarly, key informant interviews will be issue-specific and will be guided by open-ended guiding questions. The study will make use of questionnaires used in other studies, including World Bank's Energy Sector Management Assistance Programme (ESMAP) Bhandari19

, and project related documents.

1. Data Requirements and Indicators

23. The Study will require data at the household and village levels. At the household level, data will be collected to reflect socioeconomic status of household members and would include household composition by age, gender, education, occupation, health status, energy sources and consumption patterns, gender roles and time spent in accessing energy for household activities, consumption/expenditure patterns, income and household asset structure, household perception about quality of services offered by BPC, and willingness to pay for electricity. At the village level, data collection will include an inventory of businesses by typology, electrification of village facilities such as health centers, schools and monasteries, access to internet facilities, trends in fuelwood use and other relevant qualitative information relevant to outcomes and impact of RE. The Study will also make use of village and project level data maintained by village heads and BPC on electricity consumers served by the two projects and project relevant reports as deemed appropriate. IED is making efforts to retrieve data collected prior to the implementation and after the completion of Loan 2009-BHU[SF] . 24. The Study will generate four key outputs: (i) an impact evaluation report, (ii) baseline data for future impact evaluation of Loan 2464-NHU[SF], and a panel study on impact of RE; (iii) a collection of impact stories; and (iv) a set of indicators for monitoring and evaluation of RE projects. Key findings from the study will feed into Bhutan's country assistance program evaluation. The Report will be made available to both internal and external audience through multiple online channels, including ADB, Zunia, Eldis and Global Focus.

2. Survey Design and Sampling Procedure

25. According to the data provided by BPC, the two projects, Under the Loans 1712-BHU[SF} and 2009-BHU[SF] BPC has electrified approximately 17,000 rural households in 15 of the 20 districts (Dzonkhags). The coverage was all 15 districts under the first loan and only 9 under the second one. Actual sample size for the Study will be determined once geographical distribution data, including villages covered, is available. BPC is already working on it and has assured that the information will be ready in mid-December 2009. In order to permit adequate degrees of freedom in the analysis, it is expected that 1,100 (including 10% allowance for no responses or incomplete responses) of the electrified households will reflect proportional representation of the two projects and agroecology to capture spatial variations in

18 Structured questionnaires are expected to standardize the information gathering and minimize non-sampling

errors. 19 Bhandari, O. 2006. Socio-economic Impact of Rural Electrification in Bhutan. A thesis submitted in partial fulfillment

of the requirements for the degree of Master of Engineering, Asian Institute of Technology, Bankgok.

9

Bhutan. Hence, stratified random sampling method will be used. Businesses reliant on electricity will be surveyed in all sample villages (actual number to be determined). 26. BPC has already identified villages to be electrified under Loan 2464-BHU[SF] and, hence, the counterfactual villages and households will be drawn from the list of villages identified (but not electrified) for the said loan. It is also envisaged that approximately 1,100 households will be surveyed from these villages to represent comparison groups. The Independent Evaluation Mission will prepare the list of counterfactual villages in December 2009. Since electrification was done at the village level and according to BPC, all households in the treatment villages would have access to electricity; matching will be done at the village level rather than the household level based on predetermined set of similarities of village attributes, except electrification. For qualitative information and for the purpose of data triangulation, the Study will also conduct one focus group discussion in each survey village. In addition, interviews will be conducted with heads of schools, health centers and monasteries in all survey villages.

3. Estimation Methods

27. Econometric estimation procedures will be determined by the availability of previous data sets. In absence of such data set, impacts can be assessed only using single difference method instead of double difference. Similarity between treatment and comparison villages will be achieved through a propensity score matching that will be done using pre-implementation20

project household-level data. Jalan and Ravallion have argued for the superiority of a matched sample, compared to an unmatched sample in the estimation of the impact of interventions. It should be noted, however, that since matching is only based on observed community and household characteristics, the impacts of unobserved characteristics are not accounted for. The actual estimation procedure will be refined after the specific impact variables, and their determinants are identified in the preparation stage. Data permitting, the Study will also explore alternative estimation methods.

attachments: Appendix 1: Portfolio of ADB Rural Electrification Projects Appendix 2: Logic Model to Evaluate the Impact of Rural Electrification in

Bhutan Appendix 3: Impact Evaluation Matrix

20 The superiority of pre-implementation village-level data than post-implementation data as matching variables has

been argued in Pattanayak, et al. (2007). The obvious reason is that post-intervention data may be affected already by the Project. The fact finding mission seems to indicate that pre-implementation data may be poor or non-existing. If reliable pre-implementation data is not available, then there is no other recourse but to use post implementation data to implement a matching.

10 A

ppendix 1

PORTFOLIO OF ADB RURAL ELECTRIFICATION PROJECTS

Table A1.1: ADB Assistance in Energy Sector, 1969–2009 (in $ million)

Subsector Loans TAs Grants Total Loans TAs Grants Total Loans TAs Grants Total Loans TAs Grants TotalConventional Energy 344.3 0.6 0.0 345.0 1,527.7 5.2 0.0 1,533.0 2,381.2 17.1 0.0 2,398.3 277.9 11.1 0.0 288.9Transmission and Distribution 607.3 0.6 0.0 607.9 2,081.1 3.8 0.0 2,084.9 3,876.6 23.5 0.0 3,900.1 4,551.2 29.9 248.9 4,830.0Energy Sector Development 79.5 83.4 0.0 162.9 696.2 11.6 0.0 707.9 1,991.7 38.3 0.0 2,030.0 2,419.6 38.5 33.5 2,491.6Energy Utility Services 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 70.0 0.0 0.0 70.0 0.0 0.5 4.0 4.5Energy Efficiency and Conservation 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 850.0 6.2 3.0 859.2 351.1 371.2 8.5 730.8Large Hydropower 483.9 2.2 0.0 486.1 284.7 1.8 0.0 286.5 1,608.3 10.2 0.0 1,618.5 607.9 11.9 0.0 619.9Pipelines 12.2 0.2 0.0 12.4 141.3 1.6 0.0 142.9 526.0 2.7 0.0 528.7 230.0 7.6 5.0 242.6Renewable Energy 2.7 0.1 0.0 2.8 32.3 1.6 0.0 33.9 100.0 3.1 0.0 103.1 565.0 23.4 48.8 637.1

Total 1,529.9 87.1 0.0 1,617.0 4,763.4 25.7 0.0 4,789.1 11,403.8 101.1 3.0 11,507.9 9,002.7 494.0 348.6 9,845.4

1969–1979 1980–1989 1990–1999 2000–2009

Appendix 1 11

Table A1.2: ADB Loans on Rural Electrification, 1989–2009

(in $ million)

No. of Funding Amount DateProjects Source ($M) Approved Purpose

Afghanistan 1 26.52165 Power Transmission and Distribution ADF 26.5 14-Apr-05 transmission/distribution

Bangladesh 5 534.91356 Rural Electrification ADF 50.0 30-May-95 distribution/capacity building1884 West Zone Power System Development ADF 60.2 17-Dec-01 transmission/distribution1885 West Zone Power System Development OCR 138.7 17-Dec-01 transmission/distribution2038 Power Sector Development Program (Program Loan) OCR 100.0 10-Dec-03 generation/transmission/distribution/reforms2039 Power Sector Development Program (Project Loan) OCR 186.0 10-Dec-03

Bhutan 5 106.91375 Rural Electrification ADF 7.5 19-Sep-95 distribution/capacity building1712 Sustainable Rural Electrification ADF 10.0 25-Nov-99 distribution/capacity building2009 Rural Electrification and Network Expansion ADF 9.4 30-Sep-03 generation/distribution2463 Green Power Development OCR 51.0 29-Oct-08 generation2464 Green Power Development ADF 29.0 29-Oct-08 generation

Cambodia 2 64.32052 Greater Mekong Subregion Transmission ADF 44.3 15-Dec-03 transmission/distribution/capacity building2261 Second Power Transmission and Distribution ADF 20.0 04-Oct-06 transmission/distribution/capacity building

Indonesia 1 161.01982 Renewable Energy Development Sector OCR 161.0 19-Dec-02 generation/distribution/financing

Lao People's Democratic Republic 3 64.0

1308Nam Ngum-Luang Prabang Power Transmission (Supplementary) ADF 4.0 30-Aug-94 transmission/distribution

1558 Power Transmission and Distribution ADF 30.0 30-Sep-97 transmission/distribution2005 Northern Area Rural Power Distribution ADF 30.0 18-Sep-03 distribution/capacity building

Nepal 2 101.01011 Seventh Power ADF 51.0 11-Jan-90 transmission/distribution1732 Rural Electrification, Distribution and Transmission ADF 50.0 21-Dec-99 generation

People's Republic of China 2 200.01644 Yunnan Dachaoshan Power Transmission OCR 100.0 27-Nov-98 transmission/distribution1901 Shen-Da Power Transmission and Grid Rehabilitation OCR 100.0 20-Dec-01 distribution/capacity building

Sri Lanka 4 314.31021 Power System Expansion (Sector Loan) ADF 74.3 31-May-90 transmission/distribution1414 Second Power System Expansion (Sector) ADF 80.0 14-Dec-95 transmission/distribution2518 Clean Energy and Access Improvement OCR 135.0 14-Apr-09 generation/transmission/distribution/DSM2519 Clean Energy and Access Improvement ADF 25.0 14-Apr-09 generation/transmission/distribution/DSM

Thailand 1 100.01429 Rural Electrification OCR 100.0 23-Jan-96 distribution

Vietnam 2 251.01585 Central and Southern Viet Nam Power Distribution ADF 100.0 27-Nov-97 transmission/distribution/capacity building

2517Renewable Energy Development Network Expansion and Rehabilitation for Remote Communes Sector ADF 151.0 30-Mar-09 generation/financing

TOTAL 28 1,923.9

Loan No. Project Name

12 Appendix 1 12

Appendix 2

12 A

ppendix 2

Table A1.3: ADB Technical Assistance on Rural Electrification, 1989–2009 (in $ million)

No. of Funding Date

TA No. TA Title Projects Type Source Amount Approved PurposeAfghanistan 3 2,300,0004318 National Power Transmission Grid PP TASF 750,000 26-Feb-04 FS preparation4461 Poverty Reduction and Rural Renewable Energy Development AD PRF 750,000 03-Dec-04 generation/ FS preparation4662 Small to Medium-Sized Hydropower Development PP TASF 800,000 03-Oct-05 generationBangladesh 3 598,0002338 Solicitation for Private Sector Implementation of the Meghnaghat Powe AD TASF 211,000 30-May-95 distribution/capacity building

2338Solicitation for Private Sector Implementation of the Meghnaghat Power (Supplementary) AD TASF 222,000 12-Mar-97 distribution/capacity building

2338Solicitation for Private Sector Implementation of the Meghnaghat Power (Supplementary) AD TASF 165,000 03-Aug-98 distribution/capacity building

Bhutan 6 4,245,0002043 Power System Development PP JSF 245,000 29-Dec-93 distribution2400 Institutional and Financial Development of Department of Power AD TASF 400,000 19-Sep-95 capacity building2912 Second Rural Electrification PP JSF 600,000 19-Nov-97 distribution/FS preparation3825 Rural Electrification and Network Expansion PP JSF 700,000 21-Dec-01 capacity building/FS preparation4766 Accelerated Rural Electrification AD TASF 700,000 28-Feb-06 capacity building/FS preparation4916 Bhutan Power Development PP JSF 1,600,000 29-Jan-07 reforms/FS preparationCambodia 2 880,0003256 Update of Power Rehabilitation II Project Preparation Study PP TASF 150,000 17-Sep-99 FS preparation4078 Power Distribution and Greater Mekong Subregion Transmission PP TASF 730,000 10-Jan-03 FS preparationCook Island 2 340,0001102 Power System Reinforcement in Rarotonga PP SSTA 90,000 10-Jan-89 FS preparation2264 Outer Islands Power Development Study PP JSF 250,000 27-Dec-94 generation/reforms/FSFiji 1 400,0003961 Rural Electrification PP JSF 400,000 30-Oct-02 FS preparationIndia 2 1,800,0004242 Institutional Development for Rural Electrification AD UK 400,000 10-Dec-03 reforms/FS preparation7099 Integrated Renewable Energy Development PP TASF 1,400,000 21-Jul-08 distributionIndonesia 1 800,0004054 Power Welfare Scheme AD Denmark 800,000 19-Dec-02 financingLao People's Democratic Republic 5 2,043,0001080 Xieng Khouang and Sayaburi Power Transmission Study PP TASF 85,000 03-Jan-89 FS preparation1082 Institutional Improvement to EdL Luang Prabang AD TASF 198,000 03-Jan-89 capacity building2479 Power Transmission and Distribution PP JSF 250,000 18-Dec-95 transmission/distribution3087 Northern Area Rural Power Distribution PP JSF 510,000 14-Oct-98 capacity building/FS preparation7227 Small and Mini Hydroelectric Development PP Finland 1,000,000 14-Jan-09 generationMaldives 2 200,0001338 Second Power System Development PP TASF 100,000 13-Jul-90 FS preparation1944 Third Power System Development PP TASF 100,000 02-Sep-93 FS preparationMongolia 1 400,0003965 Renewable Energy Development in Small Towns and Rural Areas AD Denmark 400,000 04-Nov-02 generation/capacity building/FS preparationNepal 2 1,050,0002911 Rural Electrification and Distribution Improvement PP JSF 450,000 14-Nov-97 transmission/distribution/FS preparation4493 Rural Electrification and Renewable Energy PP JSF 600,000 17-Dec-04 FS preparationPhilippines 4 2,350,0003422 Rural Electrification Institutional Strengthening AD TASF 750,000 23-Mar-00 capacity building/reforms3516 Rural Electrification PP 0 600,000 10-Oct-00 FS preparation

4174Rehabilitation of Renewable Energy Projects for Rural Electrification and Livelihood Development AD Denmark 450,000 16-Sep-03 reforms/FS preparation

7012 Rural Electric Cooperatives Development PP JSF 550,000 11-Dec-07 FS preparationPeople's Republic of China 5 3,185,0002100 Rural Energy Development Study AD JSF 500,000 16-Jun-94 capacity building/FS preparation3105 Institutional Reform of Yunnan Electric Power Group Corporation AD JSF 785,000 27-Nov-98 distribution/capacity building4309 Renewable Energy for Poverty Reduction AD TASF/Denm 600,000 19-Dec-03 reforms/FS preparation4649 Alternative Energy Supply for Rural Poor in Remote Areas AD Other 500,000 21-Sep-05 capacity building/reforms/FS preparation4935 Gansu Rural Clean Energy Development AD DEN-E2 800,000 01-Jun-07 capacity building/FS preparationRepublic of Marshall Island 2 400,0002041 Outer Islands Power Development Study PP JSF 200,000 29-Dec-93 FS preparation2415 Ebeye Power Expansion Study PP TASF 200,000 03-Oct-95 generation/distribution/reformsSri Lanka 2 1,045,0001307 Rural Electrification Development AD JSF 445,000 31-May-90 reforms/FS preparation4262 Rural Electrification and Network Expansion PP TASF 600,000 16-Dec-03 FS preparationTajikistan 1 800,000

4423Development of Community Based Micro-Hydropower Supply in Remote Rural Areas AD PRF 800,000 05-Nov-04 reforms/FS preparation

Thailand 1 600,0002886 Rural Electrification and System Improvement PP JSF 600,000 03-Oct-97 FS preparationTonga 1 300,000

2694Institutional Development of the Tonga Electric Power Board and for Rural Electrification

AD Other 300,000 03-Dec-96 distribution/capacity building

Uzbekistan 2 650,0004173 Off-Grid Renewable Energy Development AD Denmark 350,000 15-Sep-03 reforms/FS preparation4709 Rural Renewable Energy Development PP Finland 300,000 02-Dec-05 capacity building/FS preparationVIE 1 508,0002470 Central and Southern Viet Nam Power Distribution PP JSF 508,000 12-Dec-95 distribution/capacity building

Total 49 24,894,000

Appendix 1 13

Table A1.4: ADB Grant on Rural Electrification, 1989–2009 (in $ million)

Grant No. of Source of Amount DateNo. Projects Funding (in US$M) ApprovedAfghanistan 1 23.54 Power Transmission and Distribution ADF 23.5 14-Apr-05 transmission/distributionBhutan 3 27.3119 Green Power Development CEFPF 1.0 29-Oct-08 distribution119 Green Power Development ADF 25.3 29-Oct-08 distribution9093 Rural Electricians Training Program JFPR 1.0 25-May-06Mongolia 1 2.49139 Demonstration Project for Improved

Electricity Services to the Low-Income JFPR 2.4 09-Sep-09 transmission/distribution/capacity

building/FS preparationPhilippines 1 1.59042 Renewable Energy and Livelihood

Development Project for the Poor in JFPR 1.5 19-Jan-04 generation/distribution/financing/reforms

Sri Lanka 2 3.7149 Clean Energy and Access Improvement Special Funds 2.2 06-Oct-09 distribution

9045 Power Fund for the Poor JFPR 1.5 07-Apr-04 capacity building/financingTajikistan 1 2.09089 Community-Based Rural Power Supply JFPR 2.0 15-Mar-06 generation/reforms/DSM

Total 9 60.4

PurposeProject Name

14 Appendix 1 14 A

ppendix 2

Table A1.5: Financing Sources of ADB Loans on Rural Electrification, 1989–2009 (in $ million)

ADF OCR2165 AFG Power Transmission and Distribution 26.51356 BAN Rural Electrification 50.01884 BAN West Zone Power System Development 60.21885 BAN West Zone Power System Development 138.72038 BAN Power Sector Development Program (Program Loan) 100.02039 BAN Power Sector Development Program (Project Loan) 186.01375 BHU Rural Electrification 7.51712 BHU Sustainable Rural Electrification 10.02009 BHU Rural Electrification and Network Expansion 9.42463 BHU Green Power Development 51.02464 BHU Green Power Development 29.02052 CAM Greater Mekong Subregion Transmission 44.32261 CAM Second Power Transmission and Distribution 20.01982 INO Renewable Energy Development Sector 161.01308 LAO Nam Ngum-Luang Prabang Power Transmission

(Supplementary)4.0

1558 LAO Power Transmission and Distribution 30.02005 LAO Northern Area Rural Power Distribution 30.01011 NEP Seventh Power 51.01732 NEP Rural Electrification, Distribution and Transmission 50.01644 PRC Yunnan Dachaoshan Power Transmission 100.01901 PRC Shen-Da Power Transmission and Grid Rehabilitation 100.01021 SRI Power System Expansion (Sector Loan) 74.31414 Second Power System Expansion (Sector) 80.02518 Clean Energy and Access Improvement 135.02519 Clean Energy and Access Improvement 25.01429 THA Rural Electrification 100.01585 VIE Central and Southern Viet Nam Power Distribution 100.02517 VIE Renewable Energy Development Network Expansion

and Rehabilitation for Remote Communes Sector151.0

Total 852.2 1,071.7% of Total 44% 56%

Funding Source and Amount($ million)Loan

No. Country Project Name

Appendix 1 15

Table A1.6 Financing Sources of ADB Technical Assistance on Rural Electrification, 1989–2009

(in $ million)

TA No. Country TA Name TASF JSF SSTA OthersA. PPTA - 28 projects4318 AFG National Power Transmission Grid 0.7504662 AFG Small to Medium-Sized Hydropower Development 0.8002043 BHU Power System Development 0.2452912 BHU Second Rural Electrification 0.6003825 BHU Rural Electrification and Network Expansion 0.7004916 BHU Bhutan Power Development 1.6003256 CAM Update of Power Rehabilitation II Project Preparation Study 0.1504078 CAM Power Distribution and Greater Mekong Subregion Transmission 0.7301102 COO Power System Reinforcement in Rarotonga 0.0902264 COO Outer Islands Power Development Study 0.2503961 FIJ Rural Electrification 0.4007099 IND Integrated Renewable Energy Development 1.4001080 LAO Xieng Khouang and Sayaburi Power Transmission Study 0.0852479 LAO Power Transmission and Distribution 0.2503087 LAO Northern Area Rural Power Distribution 0.5107227 LAO Small and Mini Hydroelectric Development 1.0001338 MLD Second Power System Development 0.1001944 MLD Third Power System Development 0.1002911 NEP Rural Electrification and Distribution Improvement 0.4504493 NEP Rural Electrification and Renewable Energy 0.6003516 PHI Rural Electrification 0.6007012 PHI Rural Electric Cooperatives Development 0.5502041 RMI Outer Islands Power Development Study 0.2002415 RMI Ebeye Power Expansion Study 0.2004262 SRI Rural Electrification and Network Expansion 0.6002886 THA Rural Electrification and System Improvement 0.6004709 UZB Rural Renewable Energy Development 0.3002470 VIE Central and Southern Viet Nam Power Distribution 0.508B. ADTA - 21 projects4461 AFG Poverty Reduction and Rural Renewable Energy Development 0.750

2338 BANSolicitation for Private Sector Implementation of the Meghnaghat Power 0.211

2338 BAN Solicitation for Private Sector Implementation of the Meghnaghat 0.2222338 BAN Solicitation for Private Sector Implementation of the Meghnaghat 0.1652400 BHU Institutional and Financial Development of Department of Power 0.4004766 BHU Accelerated Rural Electrification 0.7004242 IND Institutional Development for Rural Electrification 0.4004054 INO Power Welfare Scheme 0.8001082 LAO Institutional Improvement to EdL Luang Prabang 0.1983965 MON Renewable Energy Development in Small Towns and Rural Areas 0.4003422 PHI Rural Electrification Institutional Strengthening 0.750

4174 PHIRehabilitation of Renewable Energy Projects for Rural Electrification and Livelihood Development 0.450

2100 PRC Rural Energy Development Study 0.5003105 PRC Institutional Reform of Yunnan Electric Power Group Corporation 0.7854309 PRC Renewable Energy for Poverty Reduction 0.6004649 PRC Alternative Energy Supply for Rural Poor in Remote Areas 0.5004935 PRC Gansu Rural Clean Energy Development 0.8001307 SRI Rural Electrification Development 0.445

4423 TAJDevelopment of Community Based Micro-Hydropower Supply in Remote Rural Areas 0.800

2694 TONInstitutional Development of the Tonga Electric Power Board and for Rural Electrification

0.300

4173 UZB Off-Grid Renewable Energy Development 0.350Total 8.161 9.193 0.090 7.450

No. of projects 18 17 1 13

Funding Source and Amount ($M)

16 A

ppendix 1

Table A1.7: Impacts (Goals) of ADB Project Loans on Rural Electrification, 1989–2009 Primary Economic growth Poverty reduction

Loan 1011 (NEP) Loan 1021 (SRI) Loan 1308 (LAO) Loan 1356 (BAN) Loan 1375 (BHU) Loan 1414 (SRI) Loan 1429 (THA) Loan 1558 (LAO) Loan 1356 (BAN) Loan 1375 (BHU) Loan 1429 (VIE)

Loan 1585 (VIE) Loan 1644 (PRC) Loan 1732 (NEP) Loan 1884 (BAN) Loan 1901 (PRC) Loan 1982 (INO) Loan 2005 (LAO) Loan 2009 (BHU) Loan 1558 (LAO) Loan 1585 (VIE)

Loan 2038 (BAN) Loan 2052 (CAM) Loan 2165 (AFG) Loan 2261 (CAM) Loan 2464 (BHU) Loan 2517 (VIE) Loan 2518 (SRI) Loan 1712 (BHU) Loan 1732 (NEP)

Secondary Improved quality of life Improved governance Reduced environmental impacts Increased private sector participation Promote regional cooperation

Loan 1901 (PRC) Loan 1884 (BAN) Loan 1901 (PRC) Loan 2052 (CAM) Loan 2052 (CAM)

Loan 2517 (VIE) Loan 2038 (BAN) Loan 1982 (INO) Loan 2462 (BHU) Loan 2464 (BHU)

Appendix 1

17

Table A1.8: Outcome (Purpose) of ADB Project Loans on Rural Electrification, 1989–2009

1. Increased household income

Loan 1375 (BHU) Loan 1429 (THA) Loan 1712 (BHU)

Loan 1732 (NEP) Loan 1992 (INO) Loan 2005 (LAO)

Loan 2009 (BHU) Loan 2464 (BHU) Loan 2517 (VIE)

2. Better living conditions (including extended study time among children, better sanitation and hygiene, better lighting, access to information, etc)

Loan 1414 (SRI) Loan 1712 (BHU) Loan 2005 (LAO)

Loan 2009 (BHU) Loan 2518 (SRI)

3. Capacity building (including upgrading of operations and business procedures)

Loan 2261 (CAM)

4. Electricity service access and connections Loan 1011 (NEP) Loan 1021 (SRI) Loan 1558 (LAO) Loan 1585 (VIE)

Loan 1732 (NEP) Loan 1901 (PRC) Loan 2165 (AFG) Loan 2005 (LAO)

Loan 2009 (BHU) Loan 2038 (BAN) Loan 2052 (CAM)

5. Power sector development (including physical infrastructure development, additional power capacities, system loss reduction, institutional development of electric utilities, energy security, energy trading, use of indigenous energy resources, etc)

Loan 1011 (NEP) Loan 1021 (SRI) Loan 1308 (LAO) Loan 1356 (BAN)

Loan 1375 (BHU) Loan 1644 (PRC) Loan 1884(BAN) Loan 1982 (INO)

Loan 2165 (AFG) Loan 2464 (BHU) Loan 2518 (SRI)

6. Environmental protection (i.e., reduced greenhouse gas emissions)

Loan 1982 (INO) Loan 2464 (BHU)

18 A

ppendix 1

Table A1.9: Outputs (Components) of ADB Project Loans on Rural Electrification, 1989–2009 1. Construction, upgrade or rehabilitation of generation system, transmission and distribution network

Loan 1011 NEP) Loan 1021 (SRI) Loan 1308 (LAO) Loan 1356 (BAN) Loan 1375 (BHU) Loan 1414 (SRI) Loan 1429 (THA)

Loan 1558 (LAO) Loan 1585 (VIE) Loan 1644 (PRC) Loan 1732 (NEP) Loan 1884 (BAN) Loan 1901 (PRC) Loan 2005 (LAO)

Loan 2009 (BHU) Loan 2038 (BAN) Loan 2052 (CAM) Loan 2165 (AFG) Loan 2261 (CAM) Loan 2517 (VIE) Loan 2518 (SRI)

2. Electricity connections Loan 1011 NEP) Loan 1021 (SRI) Loan 1308 (LAO) Loan 1356 (BAN) Loan 1375 (BHU) Loan 1414 (SRI) Loan 1429 (THA)

Loan 1558 (LAO) Loan 1712 (BHU) Loan 1732 (NEP) Loan 1982 (INO) Loan 2005 (LAO) Loan 2009 (BHU)

Loan 2038 (BAN) Loan 2052 (CAM) Loan 2165 (AFG) Loan 2464 (BHU) Loan 2517 (VIE) Loan 2518 (SRI)

3. Fossil-fuel substitution and fuel importation savings

Loan 1982 (INO) Loan 2464 (BHU)

4. System loss reduction Loan 1011 (NEP) Loan 1732 (NEP) Loan 2038 (BAN)

5. Energy efficiency and conservation Loan 1901 (PRC)

6. Private investments in the power sector Loan 1884 (BAN)

7. Capacity building and support Loan 1901 (PRC) Loan 1732 (NEP)

Appendix 1 19

19 A

ppendix 1

Table A1.10: Summary of Indicator Analysis Used for Goals in Rural Electrification-Related Projects by Criterion

(%)

Indicators SMART Criteria

Project Document Total No. of % of Specific Measurable Achievable Relevant Time-bound

SMART SMART No. % No. % No. % No. % No. % Loans 45 10 22% 12 27% 23 51% 28 62% 38 84% 20 44% Technical Assistance 44 9 20% 10 23% 36 82% 43 98% 43 98% 14 32% Grants 30 5 17% 22 73% 30 100% 28 93% 29 97% 12 40%

Total 119 24 20% 44 37% 89 75% 99 83% 110 92% 46 39%

Table A1.11: Summary of Indicator Analysis Used for Outcomes in Rural Electrification-Related Projects by Criterion

(%)

Indicators SMART Criteria

Project Document Total No. of % of Specific Measurable Achievable Relevant

Time-bound

SMART SMART No. % No. % No. % No. % No. % Loans 63 22 35% 34 54% 38 60% 44 70% 52 83% 35 56% Technical Assistance 47 4 9% 7 15% 25 53% 39 83% 41 87% 9 19% Grants 10 8 80% 9 90% 9 90% 9 90% 9 90% 9 90%

Total 120 34 28% 50 42% 72 60% 92 77% 102 85% 53 44%

Table A1.12: Summary of Indicator Analysis Used for Outputs in Rural Electrification-Related Projects by Criterion

Indicators SMART Criteria

Project Document Total No. of % of Specific Measurable Achievable Relevant

Time-bound

SMART SMART No. % No. % No. % No. % No. % Loans 99 18 18% 34 34% 53 54% 63 64% 77 78% 52 53% Technical Assistance 114 2 2% 12 11% 63 55% 95 83% 99 87% 69 61% Grants 91 4 4% 34 37% 48 53% 53 58% 55 60% 19 21%

Total 304 24 8% 80 26% 164 54% 211 69% 231 76% 140 46%

20 A

ppendix 1

Table A1.13: Summary of Findings on the Implementation of ADB Project Loans on Rural Electrification (1989–2009)

1011 NEP Seventh Power relevant effective efficient less likely 4 years successful1021 SRI Power System Expansion (Sector Loan) highly relevant highly effective satisfactory likely 2 years and 6 months successful1308 LAO Nam Ngum-Luang Prabang Power Transmission (Supplementary) highly relevant effective efficient likely 13 months partially successful1356 BAN Rural Electrification relevant highly effective efficient sustainable 1 year highly successful1375 BHU Rural Electrification highly relevant effective satisfactory sustainable 1 year and 3 months successful1414 SRI Second Power System Expansion (Sector) highly relevant effective efficient likely 8 months successful1429 THA Rural Electrification relevant effective satisfactory 3 months successful1558 LAO Power Transmission and Distribution relevant effective efficient likely 2 years successful1585 VIE Central and Southern Viet Nam Power Distribution relevant effective efficient sustainable 2 years and 5 months successful1644 PRC Yunnan Dachaoshan Power Transmission highly relevant effective efficient likely 6 months successful1712 BHU Sustainable Rural Electrification highly relevant effective efficient likely 22 months successful1901 PRC Shen-Da Power Transmission and Grid Rehabilitation highly relevant effective efficient sustainable 11 months successful2009 BHU Rural Electrification and Network Expansion highly relevant highly effective highly efficient likely 6 months highly successful

Loan No. EffectivenessRelevanceProject NameCountry Success RatingImplementation DelaysSustainabilityEfficiencyRatings

Appendix 2

21

Output

ADB loans, TA, Grants,

National Expertise,

Government funds.

Electrification/Availability of Clean Energy

Input Outcomes

ADB = Asian Development Bank, IA = implementing agency, IED = Independent Evaluation Department, IGA = income generating activity, O&M = operation and maintenance, TA = technical assistance. Source: IED/ADB. 2009. Impact Evaluation of ADB's Rural Electrification in Bhutan. Manila.

Impact

Economic Development,

Poverty reduction,

Better quality of life.

• Improved opportunity for IGA (e.g., microenterprises)

• Reduced use of fuelwood, kerosene, and candles

• Improved ceiling using electrical appliances.

• Reduced consumption of fuelwood

• Better ventilation

• Improved response time for O&M.

• Computerized billing system • Improved tariff structure.

• Reduced smoke inside the dwellings

• Improved ventilation • Better food storage facilities • Improved teaching aids • Longer study time • Access to internet • Ease in vigilance • Reduced time spent in

collecting fuelwood • More leisure time • Improved convenience.

A. Economic • Higher household income • Lower energy expenditure

C. Institutional • Better consumer satisfaction • Better financial position of IA

B. Environmental • Improved indoor air quality • Reduced deforestation

D. Social Health

• Reduced incidence of respiratory ailments

• Reduced incidence of other health problems

Education • Improved performance of

children at school • Better quality graduates

Security • Lower incidence of crime • Reduced damages by wild

animals Gender Empowerment

• Balanced gender roles • Balanced control over

resources

LOGIC MODELS TO EVALUATE THE OVERALL IMPACT OF RURAL ELECTRIFICATION IN BHUTAN

Figure A2.1: Impact of Rural Electrification in Bhutan on Human Welfare

22 A

ppendix 2

Figure A2.2: Use of Electricity in Rural Households

Light can also be used for reading leisure, entertainment magazines and newspapers (Valencia, 1990) Light for security (fence/lamp posts) provides protection Light for making handicrafts increases productivity especially among women (Barnes, 1988) Space lighting adds to productivity since some activities could be done in the evening thus, extending working hours (Barnes, 1988) Video/sounds provides access to information that leads to non-formal education Video/sounds also builds up knowledge about health and hygiene especially among women Use of electric stoves for cooking substitutes the use of fuel wood thus, improve indoor air quality Use of electric stoves also speeds up food preparation thus, extending working hours (Barnes, 1988) Food preservation reduces time in food preparation thus, extending working hours (Barnes, 1988) Water supply enhances health and hygiene, decreases time for water collection especially among children and women resulting to better education (i.e., lower absenteeism, longer study time) and productivity.

Light for reading

Radio/TV Light bulbs

Electricity

Cook stove Refrigerator Water Pumps Fans

Video/ Sound

Light for making handicrafts

Light for security

Space Lighting

Heat for cooking

Ventilation Food storage

Water Supply

education Productivity/ Increased savings

comfort/ convenience

leisure/ entertainment/ communication

health/ hygiene protection

Appendix 2

23 Figure A2.3: Use of Electricity in Rural Communities

Electricity

comfort/ convenience

health/ hygiene

School

Street

Health

Water

education productivity protection

Space lighting

Electronic educational gadgets

Cold storage

Electronic medical equipment

Water supply for irrigation

Communal water supply

Heat for disinfection Ventilation

Street lighting led to fewer attacks at night (Valencia, 1990) Productivity of teachers are enhanced through the used of electronic teaching media while teachers are encouraged to reduce absenteeism (Chaudhury, 2003) and hold night classes with better space lighting (Barnes, 2007) Productivity of medical staff is improved with the use of electronic medical equipment while working hours are extended with better space lighting (Ramasedi, 1992)

24 A

ppendix 2

Figure A2.4: Use of Electricity in Rural Enterprises

Protection includes pest management and animal attacks (Loan 1375, PPAR) Other studies noted indirect effects like expansion of product range in existing shops to include selling of light bulbs, electrical wirings, kerosene for cooking (as substitute for fuel wood), ice cubes (Meier, 2001) There has been significant growth in small-scale activities such as steel making (Barnes 2007), horticulture, dairy farming and pig raising with the use of electricity (Ramasedi, 1992) Setting up of new shops and expansion of production/product range (especially in larger firms) engendered a structural shift in employment to the tertiary sector (Barnes and Binswanger, 1986) Mechanical processing like in coffee processing increased production by 35-40% (Mariam, 1992) Diesel motors have been replaced by electric ones, which are cheaper and more efficient (Walubengo and Onyango, 1992).

Light bulbs

Electricity

Cook stove Cold Storage Water Pumps

Light for security

Space Lighting Heat for

cooking

Food storage

Water Supply

Productivity Fuel substitution Protection

Electrical Equipment

Mechanical processing

Appendix 3

25

IMPACT EVALUATION MATRIX Evaluation Question Information Required Information Source Data Source and Method A. Economic Impact

1.a. Did rural electrification (RE) increase household income?

1.b. If yes, what was the contribution of RE towards household income increase?

Time saved in collecting fuelwood, age, educational attainment, household composition, source and proximity to energy, energy use in generating household income, type of economic activities, income sources

Household respondents in project and comparison (control) villages, project staff, household respondents in project and comparison villages, village leaders

Key informant interviews, focus group discussions and household survey based on recall method

2.a. Did RE reduce energy costs for the households?

2.b. If yes, what was the contribution of RE towards reduced energy costs?

Expenditure on energy for cooking, lighting, micro-enterprises and other economic activities

Household respondents, village leaders, project staff, businessmen, BPC

BPC records on customers, key informant interviews, focus group discussions, household survey

3 a. Did RE lead to new investments?

3 b. If yes, what kind of investments?

Inventory of new investment dependent on electrification, employment created by new investments

Village leaders, key informants, DOE, BPC

Village survey, focus group discussion, business survey

B. Environmental Impact

1.a. Did RE reduce deforestation?

1.b. If yes, to what extent?

2 a. Did RE improve indoor air quality?

2 b. If yes, to what extent?

Fuelwood consumption, forest cover, residents' perception about air quality (e.g. eye irritation, respiratory difficulties)

Departments of Forests and Environment, household survey respondents, participants in focus group discussions, key informants

Key informant interviews, focus group discussions and household surveys

C. Institutional Impact

1.a. Did RE improve consumer satisfaction?

1.b. If yes, in what ways?

2 a. Did RE improve financial status of BPC?

2 b. If yes, in what ways?

Perception about quality of service delivered by BPC, cost of expanding electrification, tariff structure, energy policy, technical competency of BPC staff, response time for repair and maintenance

Household survey respondents, business survey participants, BPC, DOE

Household and business surveys, focus group discussions, key informant interviews

26 A

ppendix 3

Evaluation Question Information Required Information Source Data Source and Method D. Social Impact

1.a. Did RE improve health status of household members?

1.b. If yes, what factors contributed to achieving this result?

Incidence and intensity of smoke related respiratory problems, provision for heating arrangements

Department of Health staff, Health facility staff, key informants, household survey respondents, focus group discussion

Department of Health records, household survey, focus group discussions

2 a. Did RE improve human capital of household members?

2 b. If yes, in what ways?

Status of school facilities (equipments), instructional methods, graduation rates by gender

Department of Education staff, school teachers, key informants, household survey respondents, focus group discussion

Department of Education records, household survey, focus group discussions

3 a. Did RE improve security for the households from animals/ poachers?

3 b. If yes, in what way?

Incidence and trends in crimes and damages by wild animals

Village leaders, focus group discussions,

Interview with village leaders, focus group discussions, household surveys

4 a. Did RE empower men and women in the households?

4 b. if yes, in what ways?

Changes in gender roles and decision making after electrification

Community leaders, health practitioners, project staff, household

Interview with NGOs representatives, household survey, focus group discussion