Document of The World Bank

Report No: ICR00001701

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD 4780 UA)

ON A LOAN

IN THE AMOUNT OF US$86.587 MILLION

TO

UKRAINE

FOR AN

EQUAL ACCESS TO QUALITY EDUCATION IN UKRAINE PROJECT

IN SUPPORT OF THE FIRST PHASE OF

THE EDUCATION SECTOR REFORM PROGRAM

June 22, 2011

Human Development Sector Unit Ukraine, Belarus and Moldova Country Unit Europe and Central Asia Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS (Exchange Rate Effective May 18, 2011)

Currency Unit = Hryvnia (UAH)

1.0 UAH = US$ 0.1267 US$ 1.00 = 7.8892 UAH

FISCAL YEAR

January 1 – December 31

ABBREVIATIONS AND ACRONYMS

APL Adaptable Program Loan Bank World Bank CAS Country Assistance Strategy CPA Country Procurement Assessment EMIS Education Management Information System FM Financial Management GDP Gross National Product GOU Government of Ukraine IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results Report IERR Internal Economic Rate of Return ISR Implementation Status and Results Report MES Ministry of Education and Science MOF Ministry of Finance MTR Mid-Term Review PAD Project Appraisal Document PDO Project Development Objectives PPD Policy Planning Division RITT Regional institutes for teacher training TIMSS Trends in international Mathematics and Science Study TTL Task Team Leader

Vice President: Philippe Le Houerou Country Director: Martin Raiser

Acting Sector Director: Mamta Murthi Acting Sector Manager: Alberto Rodriguez

Project Team Leader: Scherezad J. Monami Latif ICR Primary Author: Juan Prawda

UKRAINE EQUAL ACCESS TO QUALITY EDUCATION IN UKRAINE PROJECT

CONTENTS

D A T A S H E E T ....................................................................................................... ivA. Basic Information ..................................................................................................... ivB. Key Dates ................................................................................................................. ivC. Ratings Summary .................................................................................................... ivD. Sector and Theme Codes ........................................................................................... vE. Bank Staff .................................................................................................................. vF. Results Framework Analysis ..................................................................................... vG. Ratings of Project Performance in ISRs ................................................................... viH. Restructuring (if any) .............................................................................................. viiI. Disbursement Profile ................................................................................................ vii1.   Project Context, Development Objectives and Design ........................................ 12.   Key Factors Affecting Implementation and Outcomes ........................................ 73.   Assessment of Outcomes .................................................................................... 154.   Assessment of Risk to Development Outcome .................................................. 195.   Assessment of Bank and Borrower Performance ............................................... 196.   Lessons Learned ................................................................................................. 25Annex 1. Project Costs and Financing ......................................................................... 27Annex 2. Outputs by Component ................................................................................. 29Annex 5. Beneficiary Survey Results ........................................................................... 44Annex 6. Stakeholder Workshop Report and Results ................................................... 45Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR .................... 46Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ...................... 68Annex 9. List of Supporting Documents ..................................................................... 69

MAP 33505

i

A. Basic Information

Country: Ukraine Project Name: Equal Access to Quality Education in Ukraine Project

Project ID: P077738 L/C/TF Number(s): IBRD-47800 ICR Date: 06/23/2011 ICR Type: Core ICR

Lending Instrument: APL Borrower: GOVERNMENT OF UKRAINE

Original Total Commitment:

USD 86.6M Disbursed Amount: USD 30.0M

Revised Amount: USD 31.7M Environmental Category: B Implementing Agencies: Ministry of Education and Science Cofinanciers and Other External Partners: B. Key Dates

Process Date Process Original Date Revised / Actual Date(s) Concept Review: 09/12/2003 Effectiveness: 02/07/2006 02/07/2006 Appraisal: 07/12/2004 Restructuring(s): 04/10/2008 Approval: 05/24/2005 Mid-term Review: 02/28/2008 03/30/2008 Closing: 12/31/2009 12/31/2010 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Unsatisfactory Risk to Development Outcome: High Bank Performance: Moderately Satisfactory Borrower Performance: Unsatisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings

Quality at Entry: Unsatisfactory Government: Moderately Satisfactory

Quality of Supervision: Moderately Satisfactory Implementing Agency/Agencies: Unsatisfactory

Overall Bank Performance: Moderately Satisfactory

Overall Borrower Performance: Unsatisfactory

ii

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators QAG Assessments

(if any) Rating

Potential Problem Project at any time (Yes/No):

No Quality at Entry (QEA):

None

Problem Project at any time (Yes/No):

Yes Quality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Moderately Unsatisfactory

D. Sector and Theme Codes

Original Actual Sector Code (as % of total Bank financing) Central government administration 35 35 Secondary education 50 50 Sub-national government administration 15 15

Theme Code (as % of total Bank financing) Administrative and civil service reform 17 17 Education for all 33 33 Education for the knowledge economy 33 33 Rural services and infrastructure 17 17 E. Bank Staff

Positions At ICR At Approval Vice President: Philippe H. Le Houerou Shigeo Katsu Country Director: Martin Raiser Paul G. Bermingham Sector Manager: Alberto Rodriguez Maureen Anne McLaughlin

Project Team Leader: Scherezad Joya Monami Latif Ana Maria Parchuc de Jeria Figueroa

ICR Team Leader: Scherezad Joya Monami Latif ICR Primary Author: Juan Prawda F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The project development objective is to provide equal access to quality education and to improve the efficiency of the education system to prepare Ukraine's graduates for the knowledge society.

iii

Revised Project Development Objectives (as approved by original approving authority) The PDO was not revised. (a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Percentage of students of the 9th grades in the project hub schools achieving satisfactory results from 6 till 12 marks in the tests of Math and Ukrainian language

Value quantitative or Qualitative)

Mathematics # 77,8% Ukrainian language - 87,2%

Mathematics # 85% Ukrainian language # 92%

Mathematics-68.6 Ukraniain language-76.0

Date achieved 05/24/2005 12/31/2009 12/31/2010 Comments (incl. % achievement)

The project did not achieve indicator and during the implementation the average indicator of 9th forms# pupils# learning achievements decreased due to poor implementation progress.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised

Target Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Percentage of master trainers trained out of the total agreed target Value (quantitative or Qualitative)

0 100% 100%

Date achieved 05/24/2005 12/31/2009 12/31/2010 Comments (incl. % achievement)

G. Ratings of Project Performance in ISRs

No. Date ISR Archived DO IP Actual

Disbursements (USD millions)

1 06/30/2005 Satisfactory Satisfactory 0.00 2 06/01/2006 Satisfactory Moderately Satisfactory 0.43 3 03/23/2007 Moderately Satisfactory Moderately Satisfactory 1.11

4 06/17/2008 Moderately Unsatisfactory Moderately

Unsatisfactory 3.11

5 04/28/2009 Moderately Moderately 9.06

iv

Unsatisfactory Unsatisfactory

6 11/29/2009 Moderately Unsatisfactory Moderately

Unsatisfactory 12.86

7 02/09/2010 Moderately Unsatisfactory Moderately

Unsatisfactory 15.71

8 03/29/2010 Moderately Unsatisfactory Moderately

Unsatisfactory 15.71

9 10/12/2010 Moderately Unsatisfactory Moderately

Unsatisfactory 17.06

10 01/02/2011 Unsatisfactory Unsatisfactory 23.63 H. Restructuring (if any)

Restructuring Date(s)

Board Approved

PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes Made DO IP

04/10/2008 MU MU 3.11

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1. The Equal Access to Quality Education Project was approved on April 27, 2005, the Loan Agreement was signed on May 9, 2005, and became effective on February 7, 2006. The Project was planned as the first phase of a three-phase Adaptable Program Loan (APL) to support the Government education sector reform program.

1.1 Context at Appraisal

2. Economic landscape since independence. In Ukraine between 1990 and 2000, per capita gross domestic product (GDP) in terms of purchasing power parity decreased from US$6,725 to US$5,300; significantly lower than most of Ukraine’s neighbors such as Russia (US$8,900), Poland (US$11,000) or Hungary (US$13,900). Poor economic performance left a legacy of rising poverty and falling standards of living for most citizens. The social sectors (health, education and the social protection system) suffered the most from the years of economic collapse. These systems were largely unchanged since independence. Declining funding, salaries, and deteriorating infrastructure brought about severe worsening of the quality of services and increasing poverty and social exclusion. Social problems worsened with rising crime, alcoholism and homelessness, and falling life expectancy.

3. By 2000, the Government of Ukraine (GoU) had started to undertake significant economic reforms resulting in improved economic performance. Positive economic growth was achieved in 2000 for the first time since independence showing a GDP growth of 5.8 percent in that year and 9.1 percent in 2001. At the time of project identification in late 2002, Ukraine, the second largest country in Europe, had the natural and human resources, developed industrial based and trained labor needed to play a significant role in the world economy. However, the decade since independence in August 1991 saw the most dramatic economic collapse in the Eastern European region.

4. Main issues in the education sector. Notwithstanding the economic upturn, the economic collapse of the 1990s had substantial long-term adverse effects on Ukraine’s education system, which, together with the implications of the economic reforms at the time of preparation of the Bank-financed operation, had created new challenges to reform the education sector.

5. Ukraine’s education system had been designed to meet the needs of a centrally planned economy. Funding for education was high, resulting in high literacy levels, a majority of graduates with solid basic knowledge, a large core of skilled workers available for the industrial sector, and cultural and scientific achievements. As systemic problems remained characterized by declining quality of education and low efficiency, unaddressed during the 1990s, the strong economic performance of the early 2000s did not translate into improved performance in the sector.

6. The following five education sector issues underpinned the preparation of the Equal Access to Quality Education project partially financed by the Bank: (a) Low quality of education. The programs, curricula, teacher training and teaching practices, and

teaching aids at the time of identification were not fully appropriate to effectively meet the needs of building a knowledge economy. Every teacher, for example, was expected to upgrade his/her professional qualification in intervals of five years through rather long training courses whose content was rarely based on the current needs of teachers’ professional development. The Central In-Service Teacher Training Institute, a government institution, monopolized the teacher training provision thus preventing competition from other private potential training providers. Pre-service

2

teacher training, taking place in various post-secondary institutions, was considered to be non responsive to both the needs of the ongoing education reform and the changes in society and in the world.

The secondary education curriculum at the time of identification was overloaded. It was not attuned to the education standards and patterns adopted by other European countries in the framework of the Council of Europe, which were embraced by the GoU. The introduction of a radically new curriculum for the twelve grades of compulsory education envisaged in the reform also implied the need to modify the textbooks, teaching and learning materials, and the learning evaluation schemes. One of the most urgent and worrying problems in secondary schools was the lack of teaching and learning resources, especially acute in rural schools (about two-thirds of the total number of schools) hindering improvements in the quality of the education provided and in learning outcomes. There was an urgent need to upgrade school libraries, multimedia classes, science laboratories for chemistry and biology, foreign language teaching equipment, and social science teaching aids.

(b) Increasing enrollment and access problems and inequity in the provision of education.

Enrollment data showed an increase in primary education from 85.8 percent in 1995 to 90 percent in 2001, while at the lower secondary level there was a drop in enrollment from 96.8 percent in 1995 to 92.9 percent in 2001. Pre-school enrollment had declined from 51.4 percent in 1995 to 43.9 percent in 2001. Children living in rural, remote and poorer regions had fewer options to good learning experiences than their peers in urban centers. Unit costs in some Kyiv city schools were double those in other schools. Gross enrollment ratios for 7-15 year-old students were down to 90 percent in Ukraine in 1999, while Slovakia, Romania, Hungary and Poland increased the compulsory schooling rates to more than 98 percent.

(c) Low efficiency resulting from a system that had not adjusted to a falling student-age population.

Demographic and economic transformations affecting the enrollment pattern manifested generous average student to teacher ratios of 11.7 to 1 (18:1 in the urban areas and 7:1 in the rural ones) and a vast school network. These ratios tended to overcrowd the urban schools while rural schools tended to be significantly under-utilized. Small rural schools lack appropriate teaching spaces for laboratory or other special subjects. Due to low student-teacher ratios and low student per classroom ratios, there was significant underutilization of existing accommodation capacity, particularly in small rural towns thus contributing to uneconomic use of available and limited financial resources.

(d) Lack of a national system of educational evaluation and information on student achievement and

monitoring systems. The legacy of a centrally planned education management had kept the focus of control on inputs, and to certain extent, on processes concerning teaching, regulatory issues, and resources. Monitoring and control of education had not been concerned with learning and other key outcomes. The ongoing education reform demanded a monitoring shift towards outcomes, local management, efficiency and effectiveness, which would then become the new key descriptors of quality of education.

(e) Aging, inadequately equipped educational facilities. Maintenance had been neglected for many

years and there were scarce investments in school construction and repairs. Utilities’ costs were high due to antiquated systems and there was poor sanitation in many schools.

3

7. Government’s strategy. After independence, a GoU program to reform education was enacted by Parliament in 1993, with several subsequent efforts made to implement it. In 2001, the GoU approved a national strategy for the education sector entitled “National Doctrine for Development of Education in Ukraine in the XXIst Century” which redefined long-term priorities for education reform and improved service delivery from preschool through tertiary education. This same year, the GoU approved a new education law increasing compulsory education to 12 grades. Since then, the country’s vision and comprehensive strategies for the sector have been outlined in several other documents such as the (2003) “Strategy for Economic and Social Development of Ukraine: 2004-2015”, and the (2003) Cabinet of Ministers Program of Activities, which discusses goals for long-term reforms for all sectors.

8. At the time of project appraisal in 2004, the GoU’s goals and plans for closer ties to Europe were included in the Government’s Action Program entitled “Meeting the People” laying out an ambitious vision for the country, including inter-alia, stronger social service provision. The Government’s Action Program called for: (i) expanding access in a more cost-efficient way through further supporting and expanding an ongoing rural optimization scheme; and (ii) improving the quality of education for all Ukrainians by continuing addressing the above-mentioned issues of curriculum reform, teacher training, provision of teaching and learning resources and setting up a standardized learning assessment system. Ukraine had also committed to achieving the Millennium Development Goals (MDGs), which were adapted to suit the country’s developmental objectives, with the ultimate goal defined as raising the quality of education as a whole.

9. With about 22,000 schools (70 percent located in rural areas), half a million teachers and more than 6 million students, general secondary education was the largest sub-sector within the education sector and thus was the focus of the then ongoing Government education reform.

10. Rationale for Bank’s assistance. The GoU requested the Bank’s technical and financial assistance to help turn around the country’s quality, access and management problems in the education sector. The Bank’s project preparation team and management saw this as a compelling reason to participate since it represented a first major opportunity to support the country’s reform initiatives in the education sector.

11. The request followed joint work among the Ministry of Education and Science, United Nations Development Program (UNDP), International Renaissance Foundation (Open Society Institute) and the Bank’s work and advice on reform strategies for the sector included in the Bank’s “Ukraine- Education Reform Policy Note” disseminated in draft form in 2002 and completed in 2003.

12. The Bank’s significant international experience and its financial and technical resource mobilization capacity were envisioned to provide Ukraine with the required funds and expertise to address the issues in the education sector over the next decade. The Bank’s experience in the education sector in countries of the former Commonwealth of Independent States (former Soviet Union countries) was to provide lessons and significant resources to draw on, and brings to bear on, Ukraine’s proposed reform efforts in the education sector. The Bank’s funding, technical support and institutional capacity building was thought to help generate structural reforms that would enable lasting changes in Ukraine’s education system.

13. The proposed education sector operation was also seen as a continuation of the Bank support to the GoU through the Policy Framework and Synergies with the Programmatic Adjustment Loans, which began financing in 2001 a set of actions to improve education policy, governance and efficiency.

4

1.2 Original Project Development Objectives (PDO) and Key Indicators

14. The higher-level objectives to which the Project aimed at contributing and was an important element of the Ukraine economic and social strategy, referred to “comprehensive and harmonized human development, with emphasis on developing a healthy and highly educated citizenry, within a framework of assured rights and freedoms.” In this context the GoU planed to implement strategies geared towards increasing the quality of services and reducing service delivery costs with an emphasis on improving the existing service delivery networks and developing standards to assess performance in the education sector.

15. The project’s PDOs stated exactly the same in the PAD and the Financing Agreement were to support the GoU in its effort to provide equal access to quality education and improve the efficiency of the education system to prepare the Ukraine’s graduates for the knowledge society.

16. Progress towards achieving the PDOs during the first phase of the APL were to be measured against the following four outcome indicators and final targets agreed with the GoU: (i) Forty percent of teacher trainers conduct training with competence-based approach using modern

practices in teaching and learning; (ii) New external assessment instruments (tests and exams) accepted by education stakeholders; (iii) Increased Directorate (autonomous unit to be created within the MES to manage the project)

monitoring of programs implementation and strengthened procurement and financial management systems; and

(iv) Student/teacher ratios of rural schools in pilot districts (rayons) increased to 12-15 students per teacher.

17. It bears mentioning here that these indicators were revised in 2008 during the mid term review as will be discussed in section in section 3.2 and Annex 2 of this Implementation Completion and Results Report (ICR). Moreover, the Annex 3 in the PAD includes 19 output indicators (associated to the project components) agreed with the GoU whose attainment are fully described. In addition, this PAD Annex also includes the agreed triggers to move from one phase of the APL to the next which is discussed next.

18. A three-phase Adaptable Program Loan (APL) totaling a little over eleven years was envisioned as the financial instrument to support the attainment of the above-mentioned PDOs. The first phase of the APL was to be implemented from September 2005 to December 2009, the second from January 2010 to June 2013 and the third, and last, from July 2013 to December 2016. Triggers were agreed between the GoU and the Bank setting the key conditions, policies, processes and institutions needed to move from one phase to the next. The nine triggers to move from Phase I to II were: (i) New policy and system of certification of teachers’ competence formulated and approved; (ii) New regulatory framework for directors’ certification of competence formulated and approved; (iii) Framework document for new national curriculum completed, disseminated and agreed with all

relevant stakeholders; (iv) National external standardized exams (Matura) taken by 70 percent of secondary school leavers; (v) Sustainable, transparent and efficient policy on textbooks approval, production, distribution and

evaluation agreed with the Bank and approved by the Ministry of Education and Science (MES); (vi) General Secondary Education statistics is collected electronically from all Oblasts of Ukraine

and processed; (vii) Rayons prepared Rayon Optimization Plans; (viii) Sixty percent completion of investments in 10 piloted Rayons; and (ix) MES ready to use appropriate country systems and procedures.

5

19. Phase I was envisioned as a pilot to set in motion the processes to: (a) improve needs-based teacher training; (b) reform the curriculum; (c) scale up resource availability in non-salary expenditures; (d) improve the quality of secondary education; and (e) measure learning outcomes. Phases II and III were conceived to fine-tune and scale up the above-mentioned processes. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification

20. The PDOs were not modified during the implementation cycle. However, the results framework was simplified at the Mid-term Review (MTR) of April 2008, at the request of Bank management as further illustrated in the table in section 2.3 of the ICR.

1.4 Main Beneficiaries

21. The Project aimed at benefiting, directly and indirectly, the public general education system (primary, lower and upper secondary) in Ukraine, in particular secondary education comprising approximately 22,000 schools, about 551,500 teachers and roughly six million students. About 70 percent of the total number of secondary schools was located in rural areas. 1.5 Original Components

22. The Project had the following three components:

Component 1: Professional Development of Educators. This component costing US$12.29 million (12.8 percent of the total project cost) aimed at raising the awareness, professional knowledge and competences of teachers and school directors to implement the reforms underway in teaching and learning in general education. This component had four sub-components: (i) training of trainers to support the modernization of the system of continuing teacher profession development, (ii) in-service training for teachers to improve teachers’ awareness, knowledge and skills regarding the national education reform in general and their subject-specific professional competences in particular; (iii) leadership training for school directors, including improvement of professional knowledge and leadership skills and establishment of a professional development of school principals; and (iv) enrichment of oblasts in-training centers, comprising the improvement of the quality of training facilities, resources, and institutional culture.

Component 2: Improvement of the Learning Process. This component costing US$19.72 million (20.5 percent of the total project cost) aimed at improving the conditions of teaching and learning in secondary schools, including curriculum issues, evaluation of education outcomes, teaching and learning resources, and school improvement initiatives for classroom level education reform. This component had four sub-components: (i) the modernization of curriculum designed to strengthen the policy, strategy development and implementation capacities at the Central and Local levels and schools to support the then on-going curriculum development and revision efforts; (ii) strengthening teaching and learning tools, targeting rural schools through activities that included analytical assessment of current textbook policies, upgrading school libraries and the computerization of targeted rural schools; (iii) providing quality Monitoring and Evaluation through the setting up of a national center of Education Evaluation and Quality Assurance which carried out the external university entrance examinations and provided the technical support for the participation of Ukraine in the Trends in International Mathematics and Science Study (TIMSS); and (iv) fostering school improvements and innovations especially for disadvantaged schools.

6

Component 3: Efficiency and Management of Resources. This component costing US$63.84 million (66.3 percent of the total project cost) aimed at improving the efficiency of the general education system, strengthening the capacity for policy development of the MES and other institutions responsible for general secondary education, and supporting the MES’s plans to optimize the schools’ network and provide assistance for the rehabilitation of schools. This component had four components: (i) facilitation of policy, planning and management through establishment of the Policy Planning Division (PPD) in the MES’s General Secondary Education Department and the carrying out of a mass media campaign on the project; (ii) development of a Education Management Information System (EMIS); (iii) piloting of school optimization in six selected Oblasts; (iv) the rehabilitation of schools and furnishing of about 600 selected general education schools.

1.6 Revised Components

23. The original components were not modified during the implementation period. However, following a recommendation from Bank management, some project activities were dropped at the MTR (as further explained in section 2.2 of this ICR) because: (i) they were not being implemented; and (ii) to simplify a rather complex project design and in order to focus on only key activities which had more of a direct impact on the PDOs. 1.7 Other significant changes

24. Revised results framework. As explained in Section 2.3 of this ICR, without modifying the PDOs, the original results framework comprising 4 outcome indicators and 19 output indicators (most of them redundant and never updated during the first half of the project implementation period) was significantly streamlined during the MTRof April 2008 to include 3 outcome and 6 output indicators and to ensure connectivity of the outcome indicators with the stated PDOs. This review was undertaken at the recommendation of Bank management, agreed by the GoU and endorsed by Bank management in the Implementation Status and Results Report. The revised results framework was approved by the MES on March 26, 2009.

25. Extension to the closing date. On December 15, 2009, at the request of the GoU, the Bank approved the first and only extension to the closing date of the Loan for 12 months from December 31, 2009 to December 31, 2010, to allow for an orderly closing of the project and envisaging the partial achievement of the PDOs. The extension was granted to all categories of expenditure, except civil works, for which no disbursements had been made during the entire life of the project up to December 2009. Two new requests made by the recently elected Government (January 2010) to extend the closing date for another 12 months until December 31, 2011 were not granted by the Bank management based on observations made on the pace of project implementation and the firm understanding that an extension had a high likelihood of achieving nothing more than what the project had already achieved.

26. Partial cancellations of the Loan. There were two partial loan cancellations requested by the GoU. The first one on May 5, 2010 amounting to US$41.335 million (47.7 percent of total loan proceeds) corresponding to uncommitted financial resources allocated to the civil works category under sub-component 3.4. The second one on December 6, 2010, amounting to US$13.52 million (15.6 percent of total loan proceeds), corresponding to all categories of expenditure for uncommitted consulting services and training costs, to reduce payment of commitment charges. In total, the Bank, at the request of GoU, cancelled 63.3 percent of the original loan proceeds.

27. Alongside the second request for cancellation, the GoU also called for a reallocation of funds from the unallocated category to goods to cover unanticipated increases in the costs of laboratory

7

equipment and the purchase of 58 additional buses (not planned during project appraisal) which was an eligible expenditure under the loan agreement.

28. On December 23, 2010, seven days before the closing of the loan, the Bank processed the reallocation and cancellation.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

29. Weighing the relevant achievements and shortfalls of the preparation stage further described below, the ICR rates the project preparation, design and quality at entry of this project as unsatisfactory.

30. Strategic relevance and approach was appropriate. This project was attuned to the existing CAS (2004-2007), which focused on support and further strengthening of reform strategies and reform implementation in the education sector. The initial concept underpinning the proposed project design emerged from discussions held by the Bank with the GoU during 2002 under the framework of the then ongoing Programmatic Adjustment Loan II and the Education Reform Policy Note prepared by the Bank during 2002. These discussions that preceded the preparation activities and set the ground for the identification mission were held with the MES, the Ministry of Finance (MOF) and the Ministry of Economy and European Integration, as well as with other key stakeholder (Soros International Renaissance Foundation, the British Council and UNDP).

31. The selection of a three-phase APL (as opposed to a one-time sector investment loan) was an appropriate lending instrument at the time of preparation because: (i) it aligned project goals with the complicated and long-term implementation process involving the secondary education reform aimed by the GoU (just the completion of the curriculum reform was estimated to take about seven years); and (ii) it allowed for continuous and regular reviews of the piloting of models and activities before being scaled-up and to undertake corrective actions if they became necessary. However, the triggers agreed with GoU, specifically those for the first phase, seemed ambitious in a context where there was insufficient policy dialogue in particular with the institutional framework for Ukraine to be able to meet these APL triggers.

32. Had the project design been less complex and the GoU able to timely implement its envisaged tasks, the total duration of the implementation cycle of the three-phase APL totaling a little over 10 years would also be considered by the ICR as appropriate given the amount of time required to design and implement education reforms. Unfortunately, this timing assumption proved to be incorrect during the implementation cycle mainly for two reasons further explained in section 2.2 of this ICR. First, there was an unforeseen lack of capacity of the GoU as a whole (MES, Directorate and Treasury) to implement a very complex project design in a timely and effective fashion, a situation that was not properly assessed by the Bank preparation team. Second, there was a lack of ownership of the project design as this was prepared and implemented by consultants located in the Directorate (a legally independent unit under the MES umbrella), many of them earning more than the Minister of Education.

33. At the request of the Project Concept Note Review meeting held in September 2003, a preliminary analysis of the political environment surrounding the Project was carried out aiming to assess: (i) project support from different groups; (ii) the real and perceived needs, concerns and mechanisms to increase public trust in the education reform and identify possible incentives to engage stakeholders; and (iii) the risks associated with project preparation concerning GoU’s elections in November 2004. Notwithstanding the usefulness of the findings of this analysis, what this assessment failed to properly identify were: (a) the risks inherent in the struggles and agendas among the different political parties sitting in Parliament and within the party in government and their likely adverse impact in the

8

implementation of the project; and (b) the existing institutional bottlenecks that were to seriously hamper the project during the implementation cycle. As further shown in section 2.2 of this ICR, these two factors partially explained the long delays where the Project was at a physical and financial standstill during various implementation periods.

34. The project design was built on the key lessons learned from the analytical work undertaken by the Bank in 2002, other Bank-supported sector operations in the European and Central Asian countries and international best practices, in particular, with regard to teacher training and student assessments. A thorough economic analysis of the proposed Project was carried out during preparation. A thorough social assessment guided the design of each project sub-component by capturing the opinions and recommendations of the key project stakeholders. This social assessment consisted of focus groups with parents and students and in-depth interviews with teachers, school principals, representatives of the regional (Oblast) and district (Rayon) education administration and in-service teacher training institutions.

35. Project design was too complex. The project design including three components and 14 sub-components and a myriad of different project activities were complex and difficult to be implemented in a timely and effective fashion in a context where MES and the Directorate: (i) lacked previous experience working with the Bank, in particular the fiduciary aspects of financial management and procurement; and (ii) were not properly positioned (i.e. adequately staffed with relevant staff ) at Board approval to carry out the envisaged implementation arrangements required by such design.

36. Faulty results framework. As further detailed in section 2.3 of this ICR, the results framework is partially disconnected from the PDOs, has appropriation issues, included too many indicators, most of which were never updated and some baselines data was missing at the onset of project implementation

37. Lack of institutional capacity assessment. Despite the detailed explanation provided in Annex 6 of the PAD concerning the proposed implementation arrangements and that a final Operational Manual was made a condition for negotiations, the Bank team failed to assess the institutional capacity of the MES and of the Directorate to effectively implement the 14 sub-components included in the project design in a timely fashion; and of the MoF’s Treasury to promptly authorize eligible payments against the project.

38. The implementation readiness conditions were not fully present at the time of Board approval. For example, the Directorate was not staffed at the time of Loan effectiveness (it will take about 18 more months to have it partially staffed with the posting of the coordinator for Component 3 taking even longer). The oblasts (regions) and rayons where the school optimization network was to be undertaken had not been established (they will be selected in August 2007, 26 months after Board approval). The exact location of the schools to be rehabilitated under sub-component 3.4 (absorbing 48 percent of the total Loan) had not been identified prior to Loan effectiveness nor the architectural designs even started. Neither Bank management nor the peer reviewers raised a flag on this last serious issue at the PAD review meeting.1 The ICR team considers this to be a major project preparation weakness by the Bank team and management.

39. Risk assessments. Several risks were identified by the Bank team and listed in the PAD. One of the reasons the Bank team correctly advised on the use of an APL financial instrument was to spread

1 There was no Quality Enhancement Review (QER) of the project design as the Bank instituted the QER scheme much after this project was taken to the Board. There was, however, a QER towards the end of the implementation cycle.

9

some of these risks, in particular the ones associated with policy and institutional reforms, over almost 10 years in a three-phase APL. Some risks that were rated substantial at the preparation stage, like the inability to reach consensus concerning the assessment of learning outcomes or opposition to the school optimization program, turned out to be negligible and the corresponding sub-components timely and effectively implemented. However, the Bank team failed to assess: (i) the risk posed by the fragile political landscape existing in Ukraine at the time of preparation and its unfavorable impact it had in project implementation caused by the frequent changes at the MES and the Directorate; (ii) the lack of capacity to implement, which became a critical obstacle to the success of the project; and (iii) the adverse consequences in having this project prepared and implemented by consultants and not by staff of the mainstream institution – the MES. The Bank team seems to have failed to also recognize that the overall portfolio in Ukraine at the time of project preparation was difficult to implement with extremely low disbursements and accordingly, proposed appropriate mitigating measures for this project.

2.2 Implementation

40. The Project was not successful in delivering most of the outputs under the three components and achieving its expected outcomes (see details in Section 3.2, Annex 2 and data sheet). Project implementation was rated moderately unsatisfactory since 2008 and the ratings were never upgraded for the rest of the implementation period. What follows is amore lengthy description of the issues outside and inside the GoU’s and the implementing agency’s control that either contributed or adversely affected project implementation.

41. Issues outside GoU’s control that had an adverse effect on project implementation. The political volatility that dominated Ukraine’s nascent Democracy generated severe governance weaknesses during project implementation, that were at the root of many reform challenges in the country, including the education reforms of school optimization addressed by this project. Ownership of the project’s governance agenda was limited due to strong vested interests within the government and with the different political parties sitting in Parliament. Lack of agreement between the different political groups in power and contradictory agendas diverted the attention of the leadership of the MES away from timely and effective project implementation. This unpredictable political landscape outside the control of the project resulted in frequent changes of authorities within the MES and the Directorate in charge of managing the project, adversely affecting the project’s implementation progress. In addition, the GoU’s own bureaucratic processes to authorize eligible payments, further described in section 2.4, also contributed to severe implementation delays. This is a country rather than a sector issue, which affects implementation of all projects in Ukraine.

42. Issues within GoU and the implementation agency’s control that affected project implementation. The Directorate was established in April 2005 as an autonomous legal entity independent from the MES, and entrusted with the overall management and coordination of project activities. Establishing the Directorate may have seemed a pragmatic solution to implementation problems at the time of Project design but being legally and physically separated from its parent ministry (the MES) with project activities managed and implemented by consultants, undermined project ownership by the MES, implementation and sustainability.

43. The entire implementation cycle was characterized by: (i) frequent changes of authorities and professional staff at the Directorate; (ii) inability of the Directorate to implement most project activities in a timely and effective fashion due to a lack of capacity and lack of experience in implementing Bank funded projects; (iii) the MES, largely disengaged from project preparation and implementation (done by consultants), not ensuring the required ownership of the project; and (iv) financial bottlenecks at the MOF’s Treasury including at least six separate and sequential reviewing and authorization controls, some of these redundant and unnecessary, significantly delaying payments to contractors of eligible goods

10

delivered and services rendered, thus bringing to a partial, and sometimes total, halt the corresponding project activities. As a consequence, the physical and financial implementation progress of this operation was seriously affected during most of its implementation period. Some project activities were cancelled due to lack of any evidence of implementation.

44. A reflection of this poor implementation performance is provided by the disbursement trend. For a little over a year after Loan effectiveness, there were no disbursements of the Loan beyond the initial amount allocated to the Special Account. The total documented Loan disbursements amounted to US$3.1 million (3.6 percent of the total loan proceeds) in the first 28 months after Loan effectiveness, a very poor financial performance when compared to the expected disbursement of US$30.2 million set in the PAD. During the next 18 months, disbursements increased to US$12.86 million (14.8 percent of the total Loan). By the closing date, 59 months after Loan effectiveness, disbursements amounted to US$23.63 million (27 percent of the original loan proceeds or 74.5 percent of the amount after two cancellations). Total disbursements at the end of the grace period (April 30, 2011) are US$30 million (34.7 percent of the original loan; 94.7 percent of the amount after cancellations), reflecting a poor financial performance. It also bears mentioning, a large part as an illustration of the difficulties in processing payments under this project, that the MES and the MOF requested and received an exceptional extension to the grace period by two months in order to complete the processing of four payments including the final payment to auditor.

45. There were delays in: (a) loan effectiveness (almost seven months from the planned date set in the PAD); (b) opening the Special Account (about 12 months after Loan effectiveness); (c) partial staffing of the Directorate with qualified staff (about 18 months after its establishment; the posting of the coordinator for Component 3 took even longer); (d) creating the PPD within MES; and (e) launching the school network optimization activity, for which selection of pilot districts was completed only in 2007 and approved on August 2007 (26 months after Board approval). The Steering Committee to include representatives of different Ministries to oversee project implementation and achievement of the stated PDOs as conceived in the original project design was never established. Instead, a less inclusive and more endogenous Project Committee within the MES was established.

46. There was progress in the implementation of some project activities, as further detailed in Annex 2 to this ICR, such as: (i) the training of master trainers was completed and targets for the training of teachers and school principals was only partially achieved, as only 10 of the 28 RITTCs ended up participating in this activity (the rest decided to withdraw due to disagreements on the issue of payment with the Directorate); (ii) Ukraine participated in the TIMSS including the carrying out of standardized learning tests on a sample of 4th and 8th graders in 149 schools; (iii) the external national assessment for university entrance was scaled up and administered on an annual basis; (iv) about 530 computer classes were acquired for schools in rural areas as well as 207 laboratories of physics, chemistry, biology and math and a set of sports equipment and books; and (v) 158 buses were acquired to transport children in the rural areas in the framework of the optimization of school networks.

47. Other project activities were delivered in a much-delayed time frame with only a fraction of their agreed original targets. Some of the outputs produced in the end were also irrelevant, such as the technical concept of the comprehensive education management information system, the curriculum framework, the sociological study aimed at fostering a public information campaign and the 8 studies commissioned by the PPD and completed at the very end of the project cycle. The PPD, eventually established in the Directorate, never functioned as an incubator of education policy debate and recommendations.

48. As discussed earlier during the MTR in April 2008, 26 months after loan effectiveness, agreement was reached with the GoU to cancel the following project activities because they displayed no implementation progress: (i) rehabilitation of the Oblast’s in-service teacher training facilities including

11

the provision of equipment, supplies and other learning goods; (ii) all the remaining foreign study tours; (iii) the analytical assessment of current textbook policies; (iv) the pilot to approve, develop, produce and distribute textbooks and new teaching aids; (v) the upgrading of school libraries; (vi) the development of on-line courses materials and foreign language teaching materials as well as virtual laboratories for science and technology; (vii) the creation of an on-line data-base on available teaching and learning materials; (viii) the establishment of a web portal; and (ix) the entire sub-component 2.4 of school improvement initiatives.

49. As also discussed in Section 1.7, at the end of 2009, agreement was reached with the GoU to cancel US$41.335 million corresponding to uncommitted Loan proceeds allocated to the civil works portion of the sub-component 3.4 (rehabilitation of 600 schools). This decision was taken on the basis of no performance of this sub-component. At the same time, it was agreed that the 22 school rehabilitation architectural designs, which were likely to be completed successfully and receive the clearance of “state expertise” during 2010, would be continued and completed. State expertise clearance is a quality standard for civil works set by the GOU and the design of any structure receiving public funds for rehabilitation and/or construction have to pass State Expertise in the Ukraine in order to be rehabilitated or constructed. The 22 school designs were not completed and therefore did not even go up to State Expertise. In fact, only three school rehabilitation designs have been completed and passed State Expertise at the closing of the loan. Finally, less than one month before the closing of the Loan, another uncommitted US$13.52 million in the category of consulting services and training costs were cancelled to partially reduce the payment of commitment charges. An indication of poor implementation performance is provided by the cancellation of 63.3 percent of the original contracted loan.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

50. M&E design. The original results framework included in Annex 3 of the PAD had some design issues, which regrettably were not flagged by Bank management or by the peer reviewers during the PAD review meeting. The first issue is one of disconnect. With the exception of one outcome indicator – student teacher ratio that might provide a measure for efficiency – none of the other three outcome indicators were linked to the stated PDOs, especially with respect to equal access and quality of secondary education. Training teachers (one of the outcome indicators) by itself is not necessarily a good indicator for quality, although good teachers are necessary to ensure improved quality. The establishment of external assessment instruments (another outcome indicator) might be a means to measure quality of learning, but it is not an indicator of quality. The same comment goes with the fourth outcome indicator – increased MES monitoring of program implementation. There was no indicator to measure equity of access to secondary education.

51. Given this disconnect, it was difficult for the MES/Directorate as well as for the Bank team to gauge progress towards achieving the PDO during the first half of the implementation cycle, at which point this disconnect issue was addressed by restructuring the results framework without modifying the PDOs, as explained below.

52. The second issue concerns the aim of preparing graduates for the knowledge society as stated in the PDOs. Knowledge society was probably one of the priorities of the Bank at the time of Project design, but it has serious observable and measuring issues that Bank management did not flag at the time of the preparation cycle. No attempt was made to gauge whether graduates were being prepared for the knowledge society or not.

53. The third issue concerns the large number of intermediate output indicators – 19 – many of which were never updated, for example: (i) upgrading facilities in all RITTs; (ii) textbooks and teaching aids produced and piloted in 50 selected schools; (iii) assessment of existing textbook policies; (iv) 80 percent

12

of school improvement support approved; (v) training on planning, management and use of data for policy monitoring; (vi) MES ready to manage APL with use of appropriate country systems; (vii) information system for general secondary education statistics processing in place; and (viii) capacity of Oblasts’ and Rayons’ education authorities to manage the development and implementation of the Rayons’ optimization plans.

54. The fourth issue is one of attribution, given that some of the project interventions impacting the PDOs, like the supply of learning equipment and teaching materials, the optimization of schools, just to mention a few examples, were limited to a few rural schools, thus their likely impact in the overall education system (as stated in the PDOs) was to be very small or even negligible. The ICR team is of the opinion that the PDOs needed to be defined in more constrained terms referring only to the universe of beneficiary schools and comparing the envisaged changes with a control group of comparable non-benefited schools.

55. The fifth and last issue is the lack of a baseline of one outcome and several intermediate output indicators to compare changes during the implementation cycle.

Table 1. Revised M&E Framework

Outcome and output indicators at appraisal Revised outcome and output indicators at the MTR

Outcome indicators 40 percent of teacher trainers conduct training with competency-based approach using modern practices in teaching and learning

Modified as: Access of poorest students at the project’s hub schools; percentage of students from the poorest families attending the project’s hub schools

New external assessment instruments (tests and exams) accepted by education stakeholders

Modified as: Percentage of students of the 9th grades in the project hub schools achieving satisfactory results from 6 till 12 marks in Math and Ukranian language

Increased MES monitoring of program implementation and strengthened procurement and financial management systems

Deleted

Student/teacher ratios of rural schools in pilot rayons increased to 12-15 students per teacher

Remained the same

Output indicators for component 1 120 master trainers, 60 percent of Oblast and 30 percent of rayon level trainers of trainers prepared

Remained the same

Teacher professional development courses on modern methods of teaching piloted

Modified to: Percentage of teachers trained with improved teaching skills out of the total target

10 percent of school directors’ training on educational leadership piloted

Modified to: Percentage of school principals receiving management training out of the total target

Management training conducted in all RITTCs Deleted Upgraded facilities in all RITTCs Deleted

Output indicators for component 2 National curriculum board policies/framework document completed. Subject syllabi revised in light of overall curriculum policy

Deleted

New generation teaching aids produced and piloted in 50 selected schools

Modified to: percentage of hub schools receiving learning materials and equipment provided by the project

Assessment of existing textbooks policies completed Deleted Examination and school-based assessment pilots completed. Plans for implementation of external exams finalized

Remained the same

13

Outcome and output indicators at appraisal Revised outcome and output indicators at the MTR Ukraine participating in PISA and TIMSS Remained the same 80 percent of school improvement support approved Deleted

Output indicators for component 3 MES and Oblast education department training on planning, management and use of data for policy monitoring completed

Deleted

MES ready to manage APL with use of appropriate country systems

Deleted

Project efforts supported through information and communication campaign in a timely and effective manner

Deleted

Information system for general secondary education statistics processing is in place

Deleted

Capacity of Oblasts and rayon education authorities to manage the development and implementation of rayon optimization plans improved

Deleted

General secondary education network in 10 pilot Rayons optimized to deliver quality education services

Deleted (it is already subsumed in one of the outcome indicators)

80 percent of target schools rehabilitated Deleted Studies on issues of public finance and accountability conducted

Deleted

Note: Achievement of the outcome and output indicators is reflected in the Data Sheet of this ICR.

56. M&E implementation. During the MTR mission carried out in April 2008, Bank management requested the Bank team to retrofit the results framework to the stated PDOs and significantly reduce the number of indicators. Accordingly, as shown in the Data Sheet of this ICR and in the table 1 above, the outcome indicators were reduced to three and the output indicators to six. These changes were agreed by the GoU and endorsed by Bank Management, and have been reflected and consistently updated in the Implementation Status and Supervision Reports (ISRs) from the archived sequence Number 4 forward. The PDOs remained the same.

57. The revised outcome indicators were limited only to the beneficiary schools in the project (hub-schools) and aimed at measuring: (i) learning outcomes in 9th grade (which is an accepted proxy of quality); (ii) efficiency, but only in the rural schools included in the optimization scheme by gauging changes in the student teacher ratios (which is an accepted proxy for efficiency); and (iii) access of the poorest students at the project’s hub schools which is a good measure of equity access. Baselines for these outcome indicators were set at the mid-term review mission based on the available information as of the time of project approval in May 2005.

58. Unfortunately at the MTR mission, neither the Bank team nor the MES/Directorate made an effort to include a control group of schools not benefiting from the Project that could be used to gauge the likely impact of the Project on the newly agreed outcome indicators.

59. The six intermediate output indicators related to Component 1 (percentage of master trainers, teachers and school principals trained out of the total target set at appraisal) and to component 2 (percentage of hub schools receiving learning material and equipment provided by the project; Ukraine participation in international learning assessment schemes – TIMSS and PISA; and national external assessment implemented. No output indicator was selected for component 3, partly because: (i) the optimization of rural schools (sub-component 3.3) would impact access of poor students and this was already measured by another outcome indicator; and (ii) the rehabilitation of schools (sub-component 3.4

14

that was later dropped from the project altogether) does not really measure any of the required sub-dimensions contributing to the achievement of the PDOs. Baselines for these output indicators were set at the MTR mission based on the available information as of May 2005.

60. M&E utilization. With the exception of the information derived of the outcome indicator concerning learning outcomes and the output indicator concerning Ukraine’s participation in PISA and TIMSS, the rest of the information provided by the M&E framework was of limited use in terms of being able to inform MES’s authorities on decision-making and resource allocation. The main reason of this limited utilization of information was the significant implementation lateness of the corresponding project activity generating the required information.

2.4 Safeguard and Fiduciary Compliance

61. Financial management rated moderately unsatisfactory. The Bank team carried out a financial management (FM) assessment of MES during the preparation stage. The existing financial arrangements for handling the PHRD grant that supported project preparation activities under the Grant Coordination Unit were to continue providing such services during the implementation period, but now under the Directorate that was to be established in April 2005. These financial arrangements were found by the FM assessment to be satisfactory as reported in the PAD. Unfortunately, reality proved otherwise, as the Directorate was not really well positioned to handle the FM aspects of the project in a timely and effective manner. The reasons for the Moderately Unsatisfactory rating are provided below.

62. While the FM supervision carried out during the life of the Project found that the some elements of the Project’s accounting, reporting and and internal control arrangements were in place, a number of other FM issues adversely affected project implementation. FM arrangements were rated MU by almost all the supervision missions because of: (i) the significant delays in paying Directorate staff, as well as contractors of goods delivered and services rendered by consultants and other suppliers (requiring about 6 reviewing steps to be processed before the authorization for payment was issued, some of them redundant and unnecessary); payment processes were even further delayed by frequent changes of the Ministry officials which affected availability of project signatories. Deficiencies in the payment process led to the need to extend the application deadline by an additional two months; (ii) the long delays in submitting statement of expenditure reports at times lagging almost 22 months; and (iii) the lack of approval of a 2010 budget for the project (a temporary, but insufficient budget was given to the Directorate by the MoF later in the year); and (iv) Delays in appointment of auditors and carrying out financial audits, as well as qualifications in the audit reports and internal control issues noted by auditors. Specifically, the 2008 Audit report was not submitted on time and the one for 2009 was qualified with respect to physical observations concerning the project’s assets.

63. In addition to the above, project funds amounting to US$94,641 included in a Project transit account held in the commercial bank Ukrprombank that became bankrupt could not be accessed while transferring this transit account to a new bank – Ukreximbank. Accordingly, the Bank requested the GoU to refund this amount before the closing of the loan. This refund took place but after the project closing date and following extensive formal communication between the Bank and GoU.

64. Procurement was rated moderately satisfactory. The Bank team carried out a preliminary procurement capacity assessment during the preparation process, which was later modified by the findings of a 2005 Country Procurement Assessment (CPA). With the findings of the CPA the Directorate’s procurement capacity was reassessed and recommendations made to strengthen its procurement capacity.

15

65. The ICR team is of the view that the procurement plan agreed at appraisal was ambitious and did not take into account the lack of implementation capacity at the Directorate; particularly, for carrying out a large volume of relatively small and geographically dispersed civil works, compounded by overly complex procurement procedures. According to some Bank team supervision reports, the procurement plan rather than project activities and their objectives drove project implementation.

66. Despite the delays in procurement processes and the obvious lack of organizational and management capacity at the Directorate, most of the Bank supervision missions rated the project’s procurement as Moderately Satisfactory. By and large, all the post-review procurements carried out by the Bank team during the entire implementation cycle, including the last one undertaken in October 2010, revealed no major deviations from the Guidelines, albeit the procurement capacity of the Directorate, especially for tendering international competitive bidding, was found to be weak. There were also issues of staffing with qualified procurement (as well as financial) specialists, some of whom, once familiar with Bank procedures, tended to migrate from one international financed project to another or to the private sector for better wages. This labor migration phenomenon resulted at times in a Directorate that lacked sufficient qualified staff to ensure a prompt procurement response.

67. There were some procurement issues raised by the Bank team. One related to the bidding process for school laboratories and equipment that had to be canceled due to mismanagement of the evaluation process. This bidding process was restarted, with the result that these learning goods reached the beneficiary project schools at the end of December 2010.

68. Another procurement issue not flagged during the preparation stage concerned the intention of the RITTs to contract government employees as consultants for the training to be provided under component 1. Since there was no waiver in the Loan Agreement to include this selection process, there were significant delays in working out a waiver that would allow the RITTs as unique and specialized organizations who could provide training as designed at appraisal. This delay meant that in the end, only 10 out of 28 RITTs ended up participating in the training activities, which in turn prevented achievement of the targets set at appraisal of 4,000 teachers and 2,200 school headmasters trained.

69. Compliance with environmental safeguards not rated. The project was given a B environmental rating at appraisal and required and Environmental Management Plan. This plan was approved by Bank management on June 2004 and also immediately submitted for consultation with key project stakeholders and local non-governmental organizations (NGOs). However, the environmental safeguards were never triggered as the Directorate was unable to implement sub-component 3.4 concerning the rehabilitation of 600 schools and the civil works category was closed in December 2009.

2.5 Post-completion Operation/Next Phase

70. Most of the triggers to proceed to phase II of the APL were not achieved by the GoU (as detailed in section 3.2 of this ICR), and given the unsatisfactory physical and financial performance, it was agreed by the Bank not to pursue phase II of the APL. Given the lack of interest on the part of the government and the very poor performance of this project, the Bank is not embarking on any follow up operations in the sector in the Ukraine. Moreover, very few if any donors participate in the Ukraine education sector. Those who do, have their carefully carved out niche such as UNICEF for inclusive education and the USAID for student assessments.

3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation

16

71. At the preparation stage. The Bank’s support for education in Ukraine was in line with the Country Assistance Strategy (CAS) 2004-2007 (Report 26448-UA discussed by the Board on October 27, 2003) focusing on the further strengthening of the reform strategies and implementation in the education sector. The CAS’s aim in the continued support to Ukraine was facilitating reforms of institutions and policies. The issue of access to quality education is explicitly acknowledged in the CAS document and Objective 3 of the CAS relates to harmonized human development, improving access to modern technologies for all students and providing for rationalization of unused facilities. All of these aspects of education were supported by the project

72. At the time of the closing of the Loan. The project’s PDOs are also consistent with the strategic goal 7 (Improved Governance and Accountability) included in the second pillar of the Country Partnership Strategy 2008-2011. This second pillar refers to the Bank’s support to Public Finance and Public Sector Reform and Improved Service Delivery through the improvement of governance in the education sector, in particular with respect to the quality of education and the enhancement of the efficiency and management of the education system in Ukraine. Therefore it is clear that the current relevance of the PDOs is remains high but those for the project’s design and implementation arrangement are negligible.

3.2 Achievement of Project Development Objectives

73. Assessment of PDO achievements. Given the design issues of the results framework previously described in section 2.3 of the ICR, the achievement towards the stated PDOs - “to ensure equal access to quality education and improved efficiency of the education system to prepare Ukraine’s graduates for the knowledge society” – using the original outcome indicators (included in table 1 above) could not be assessed by the MES during the first half of the implementation cycle.

74. It was after the results framework was revised during the April 2008 MTR mission with the consent of the GoU and Bank Management and the outcome and output indicators limited to the scope of the project (see table 1 above), that the achievement towards the original PDO could be measured and reported in the ISRs numbers four to 10. The ICR authors rates the achievement of the PDOs through the revised outcome indicators as moderately unsatisfactory for the following three reasons (and further described in the Data Sheet of this ICR).

75. The learning achievements in math and Ukranian language in 2010 of students of the 9th grades in the project hub schools (outcome indicator 1) worsened as compared to the baseline and were way off the target set at appraisal. Two project interventions were included to affect learning outcomes – training of teachers and head masters and provision of learning goods. With respect to training, no on-site assessment was ever conducted about the effectiveness of the teacher and head school masters training conducted by the project, thus there is no evidence available to the ICR about the presumed impact of training in improving learning outcomes. The learning goods acquired by the project arrived at the project schools just at the end of December 2010, thus, these also did not contribute to improving learning outcomes during the project period. Based on the evidence, it can be assumed that the project did not contribute to improving learning outcomes in the project’s hub-schools, and in fact, learning outcomes in language and math have been brought down in the last five years.

76. The positive trend shown by the increased student teacher ratio in rural schools in pilot network optimization rayons (outcome indicator 2) could be a likely consequence of the optimization program undertaken in 32 rural project hub-schools including about 90 satellite schools. However, not having a comparable control group of rural schools, there is no information on where the changes in student teacher ratios in non-project rural schools during the same time period.

17

77. Even though the target of the percentage of students from the poorest families attending the project’s hub schools (outcome indicator 3) was surpassed, it is unlikely that this is a consequence of the project. If one looks at the yearly progression of this indicator – 14.2 percent in 2006, 13.6 percent in 2007, 15.5 percent in 2008 and 15.7 percent in 2009, improvement were achieved despite that the sub-component 3.3 (school network optimization) was not yet implemented (the 100 buses started operating in early 2010). The ICR authors were informed that periodic changes of the definition of the poverty level by Parliament caused this indicator to change. In addition, not having a comparable control group of rural schools, there is no information as to what were the changes in equity access from the poorest families attending non-project rural schools.

78. Output (or intermediate outcome) indicators. According to the information shown in the ICR Data Sheet the project achieved or surpassed 4 output (intermediate outcome) indicators and did not achieve two of them.

79. The following two triggers out of nine to proceed to phase II of the APL were achieved: (i) National external standardized exams (Matura) taken by 70 percent of secondary school leavers;

and (ii) Rayons prepared Rayon Optimization Plans.

80. One trigger out of nine, listed below, was only partially achieved:

81. The framework document for the new national curriculum was completed and delivered to the MES for its review and approval. However, this document was never disseminated, nor was it agreed with all relevant stakeholders, as included in the trigger. Furthermore, the MES has shown no ownership of this new curriculum framework (prepared by external consultants in the Directorate), making its likelihood for sustainability is minimal at best.

82. The following six triggers out of nine to proceed to phase II of the APL were not achieved: (i) New policy and system of certification of teachers’ competence formulated and approved; (ii) New regulatory framework for directors’ certification of competence formulated and approved; (iii) Sustainable, transparent and efficient policy on textbooks approval, production, distribution and

evaluation agreed with the Bank and approved by the MES; (iv) General Secondary Education statistics is collected electronically from all Oblasts of Ukraine and

processed; (v) Sixty percent completion of investments in 10 piloted Rayons; and (vi) MES ready to use appropriate country systems and procedures.

3.3 Efficiency

83. According to the economic and financial analysis carried out in the PAD, it was envisaged that the project would yield an Internal Economic Rate of Return (IERR) of 15.2 percent. The estimated IERR was most sensitive to the number of beneficiaries, especially the percentage of school-age population in the coming years and any increase in the parents’ out-of-pocket expenses to send their children to school. Both assumptions changed significantly and negatively during project implementation although for the latter the data used is based on anecdotal evidence rather than published data. Similarly, while the simulation was indicative of the fact that the project’s investments were robust, the constant bottlenecks and follow-up reduction in project financing suggest that the project failed to reach an

18

estimated IERR (albeit it was overly optimistic - with expectations for the benefit stream to equal almost half of the annual general secondary education budget for the period of project implementation).

84. Only close to its end was the Project able to support a number of important activities with potential for an economic return. The Project activities created conditions for optimization in pilot districts and showed the path for modernization of the education system but the benefits of the project will largely depend upon completion of the optimization exercise and education reforms by the new Government.

85. As many of the results of the Project including learning outcomes and efficiency gains, will take some time to materialize, carrying out a similar to appraisal phase cost-benefit analysis as of the end of the Project was not thought to be useful. Against this background, neither a rate of return nor a present value analysis is presented here. Overall, as further detailed in the Annex 3 of the ICR many project activities were limited to a few rural districts and its impact in the overall education system is marginal contrary to the declared PDOs; a number of important activities were either significantly delayed or fully cancelled (see section 2.2 for cancelled activities) and there are sustainability concerns on some of the project achievements. Accordingly, the efficiency of this Project is rated as Negligible.

3.4 Justification of Overall Outcome Rating

86. Given the information provided in the ICR Data Sheet and in section 3.2, the overall outcome of this Project is rated as Unsatisfactory. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development

87. The project benefited 32 rural hub-schools included in the school optimization framework under sub-component 3.3, including the provision of learning materials and equipment which is important from the standpoint of poverty gap between rural and urban households, as well as higher projected poverty risk in the country for rural areas (see poverty assessment updates for Ukraine and Country Economic Memorandum 2010). A total of 527 rural schools all over the country were equipped with teaching computer classes leading to better learning opportunities for children from rural areas. Also during the last week of the project’s implementation, the number of rural schools received laboratories of physics, biology, chemistry and mathematics, as well as books and sports equipment. However, due to the significant delay in having implemented these activities, and the lack of a control group to compare with, there is no hard evidence available to gauge the impact this project had, if any, on reducing poverty. (b) Institutional Change/Strengthening

88. One major institutional accomplishment of the project was to position the MES to undertake standardized learning assessment measurements in different levels of the education system and to participate in international learning assessment schemes, like the TIMSS.

89. The project intended to introduce a new curriculum framework in secondary education and to strengthen the analytical capacity of the MES to decide on education policy through the establishment of the PPD. As mentioned elsewhere in the ICR, the MES has not shown indications of owning the curriculum framework document prepared by consultants outside the MES and delivered in December 2010, and it is likely that no follow up will take place during 2011. Likewise, the PPD that was established as a project unit within the MES never achieved its intended incubator role to foster education policy debate and is likely to be discontinued in 2011.

19

(c) Other Unintended Outcomes and Impacts (positive or negative)

90. None. There were no other development partners working in the areas of education this project worked in. It bears mentioning that the donor participation in the education sector in the Ukraine is and continues to be limited.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

91. Not applicable.

4. Assessment of Risk to Development Outcome

92. The ICR team rates the risk to development outcomes as Significant for the following reasons. Out of the 14 key sub-components included in the original project design, only the following three will likely be financially, institutionally and technically sustained after the closing of the Loan: (i) annual carrying out of the external assessment; (ii) the optimization scheme in rural areas, which has been operating in the rural areas of Ukraine since 2003, before this project was designed, will continue to be carried out by the rayon education authorities; and (iii) the training activities under sub-component 1.2 and 1.3 depending upon the availability of financial resources in the 2011 budgets of some the 10 RITTs that participated during the implementation cycle and also contingent on the printing and distribution of the training materials that were prepared for this effect.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Unsatisfactory

93. The preparation cycle took about 20 months from pre-identification (December 2002) to appraisal (July 2004) and included 6 missions. Negotiations took place in April 2005, once the new elected Government that took over in January 2005 had the opportunity to internalize the proposed project design prepared under the previous administration. The Board approved the Project on May 24, 2005; the Loan was signed by the GoU on September 5, 2005, and declared effective on February 7, 2006. The nine months elapsing from Board approval to effectiveness were common with other Bank-financed projects in Ukraine.

94. Only one task team leader (TTL) was responsible for the entire project preparation cycle and also for the early supervision stages (until mid-2007) thus providing a welcome continuity to these two processes up to the point that this TTL was replaced because she left the Bank. As detailed below, the preparation process is a mixed story with strengths and weaknesses and ultimately faulty project design, resulting regrettably in unsatisfactory implementation progress as explained elsewhere in this ICR.

95. As already described in Section 2.1, the ICR team rates the Bank performance at the preparation stage as unsatisfactory. What follows is a more detailed explanation of the reasons for this rating, highlighting the strengths as well as the serious shortfalls, in particular those related to a complex project design, the lack of an institutional capacity assessment, a weak results framework and a lack of implementation readiness conditions, adversely affecting the physical and financial implementation progress of this project (the assessment of the project’s results framework included during the preparation cycle appears in section 2.3 while the one for the fiduciary and safeguards appears in section 2.4 of this

20

ICR).

96. The ICR team considers the: (a) strategic relevance and approach for this project as appropriate; (b) lending instrument consisting of a three-phase APL as adequate, had the first phase be less complex and had been implemented more effectively by the Directorate; (c) client orientation as effective; and (d) the economic, financial and social assessments analysis as suitable.

97. The project design comprising three components, 14 sub-components and a myriad of different activities was extremely complex and difficult to be timely and effectively implemented in the context of limited institutional capacity within the context of a newly formed Education Development Programs Directorate (the Directorate) under the auspices of the MES, which was not familiar nor appropriately prepared to manage Bank-financed operations. The Directorate was established as a legal not for profit public entity subordinate to the MES and with functions similar to that of a project management group. The ICR authors note that this complex design was not objected to or flagged by Bank management or by the peer reviewers at the Project Concept Note and the PAD review meetings.

98. The fact that this project was prepared by consultants under a PHRD Grant, and later, implemented by consultants in the Directorate, significantly diminished the MES’s engagement and ownership of the project, with detrimental consequences for the sustainability of project activities at the closing of the Loan.

99. As fully explained in sections 2.3, 3.2 and 5.1 of the ICR, the project’s Results Framework: (i) had some disconnect between the phrasing of the PDOs with the outcome indicators; (ii) had attribution issues between the scope of the project and the stated PDOs; (iii) had too many indicators, most of which were never monitored and/or updated during project implementation; and (iv) lacked baseline data for some of the indicators.

100. The ICR authors find that Bank team did not carry out a suitable institutional capacity assessment of the proposed implementation arrangements consisting of an autonomous Directorate managing the project independent from the MES through consultants, some of them earning more than the Minister of Education. This lack of operational coordination between the MES and the Directorate also hindered a more timely and effective implementation progress. Accordingly, the project lacked the implementation readiness conditions at the time the Loan was declared effective.

101. The Bank team carried out procurement and financial capacity assessments identifying some, but regrettably not all key issues that needed to be addressed prior to the start of the implementation cycle. Some of these issues that were not flagged at the preparation stage and were only discovered after Loan effectiveness related particularly to the flow of funds within the project. These were never effectively addressed during the implementation period. For example, six steps, some of them redundant and unnecessary, were required to complete the payment of an eligible expenditure of goods acquired and services rendered under the Project. The payment process was adversely affected by frequent changes of the Ministry/Directorate officials, leading to extended periods when project signatories would not be available to sign disbursement/payment documents. Delays in payment, oscillating between 15 to 22 months, plagued the entire project implementation period.

102. There was compliance with the Bank’s environmental safeguard policies at the time of project preparation with respect to the infrastructure activities included in the project design. To comply with the B environmental rating, the Bank team timely ensured the preparation of an Environmental Management Plan, approved by Bank management on June 2004 and submitted for consultation with key project stakeholders and local non-gove