samoa: education sector project ii - asian development bank...samoa: education sector project ii ......

Completion Report

Project Number: 34285-013 Loan Number: 2220 Grant Numbers: 0031 and 0032 June 2016

Samoa: Education Sector Project II This document is being disclosed to the public in accordance with ADB's Public Communications Policy 2011.

CURRENCY EQUIVALENTS

Currency Unit – tala (ST) At Appraisal At Project Completion (25 August 2005) (31 December 2014)

ST1.00 = $0.3705 $0.4185 $1.00 = ST2.6991 ST2.3895

ABBREVIATIONS

ACEO – assistant chief executive officer ADB – Asian Development Bank CEO – chief executive officer CFA – Consolidated Funding Arrangement CMAD – Curriculum, Materials, and Assessment Division DFAT – Australian Department of Foreign Affairs and Trade DMF – design and monitoring framework ESAC – Education Sector Advisory Committee ESC – Education Sector Project II Steering Committee ESCD – Education Sector Coordination Division ESPII – Education Sector Project II ITWG – Informal Technical Working Group M&E – monitoring and evaluation MEF – monitoring and evaluation framework MESC – Ministry of Education, Sport, and Culture MFAT – New Zealand Ministry of Foreign Affairs and Trade MOF – Ministry of Finance NTDF – National Teacher Development Framework NUS – National University of Samoa PINZ – Polytechnics International New Zealand PPRD – Policy Planning and Research Division SDR – special drawing rights SOD – Schools Operations Division SPECA – Samoa Primary Education Completion Assessment SPELL – Samoa Primary Education Literacy Levels TA – technical assistance TOR – terms of reference

NOTES

(i) The fiscal year (FY) of the government ends on 30 June. FY before a calendar year denotes the year in which the fiscal year ends, e.g., FY2015 ends on 30 June 2015.

(ii) In this report, "$" refers to US dollars unless otherwise stated.

Vice-President S. Groff, Operations 2 Director General X. Yao, Pacific Department (PARD) Director E. Veve, Urban, Social Development and Public Management Division,

PARD Team leader C. Thonden, Education Specialist, PARD Team member A. Dizon, Associate Project Analyst, PARD

In preparing any country program or strategy, financing any project, or by making any designation of or reference to a particular territory or geographic area in this document, the Asian Development Bank does not intend to make any judgments as to the legal or other status of any territory or area.

CONTENTS

Page BASIC DATA i

I. PROJECT DESCRIPTION 1

II. EVALUATION OF DESIGN AND IMPLEMENTATION 1

A. Relevance of Design and Formulation 1

B. Project Outputs 2

C. Project Costs 7

D. Disbursements 7

E. Project Schedule 7

F. Implementation Arrangements 8

G. Conditions and Covenants 9

H. Related Technical Assistance 9

I. Consultant Recruitment and Procurement 9

J. Performance of Consultants, Contractors, and Suppliers 10

K. Performance of the Borrower and the Executing Agency 10

L. Performance of the Asian Development Bank 10

III. EVALUATION OF PERFORMANCE 11

A. Relevance 11

B. Effectiveness in Achieving Outcome 11

C. Efficiency in Achieving Outcome and Outputs 12

D. Preliminary Assessment of Sustainability 13

E. Impact 14

IV. OVERALL ASSESSMENT AND RECOMMENDATIONS 14

A. Overall Assessment 14

B. Lessons 15

C. Recommendations 15

APPENDIXES 1. Design and Monitoring Framework for ESPII (Draft as of 20 March 2008) 16

2. Consulting Services Supplied through PINZ (January 2009–April 2013) 25

3. Status of Compliance with Loan Covenants 26 4. DFAT Evaluation Criteria: Monitoring and Evaluation, Gender, and Risk 31

5. List of Outputs 34

6. Detailed Recommendations 36

BASIC DATA A. Loan Identification 1. Country 2. Loan Number 3. Grant Numbers

4. Project Title 5. Borrower 6. Executing Agency 7. Amount of Loan

8. Amount of Grants 9. Project Completion Report Number

Samoa 2220(SF) 0031, 0032 Education Sector Project II Independent State of Samoa Ministry of Finance SDR5,655,000, or $8,060,000.00 equivalent $17,200,000.00 1572

B. Loan Data 1. Appraisal – Date Started – Date Completed 2. Loan Negotiations – Date Started – Date Completed 3. Date of Board Approval 4. Date of Loan Agreement 5. Date of Loan Effectiveness – In Loan Agreement – Actual – Number of Extensions 6. Closing Date – In Loan Agreement – Actual – Number of Extensions 7. Terms of Loan – Interest Rate – Maturity (number of years)

– Grace Period (number of years)

10 August 2005 25 August 2005 17 November 2005 19 November 2005 16 December 2005 5 June 2006 3 September 2006 14 August 2006 0 30 June 2013 22 September 2015 1 1.0% per annum during the grace period, 1.5% per annum thereafter 32 8

8. Disbursements

a. Dates Initial Disbursement

17 October 2007 Final Disbursement

31 July 2015

Time Interval 93.54 months

Effective Date

14 August 2006 Original Closing Date

30 June 2013 Time Interval

82.59 months

ii

b. Amount

Loan 2220 (in SDR)

Category Original Allocation Last Revised Allocation

Amount Canceled Amount Disbursed

01 Civil Works 2,848,000.00 3,606,000.00 258,309.68 3,347,690.32 02A Equipment: Vehicles 63,000.00 38,000.00 939.08 37,060.92 02B Equipment: Other 562,000.00 346,000.00 119,196.24 226,803.76 03 Learning Materials 330,000.00 295,000.00 84,779.64 210,220.36 04 Consulting Services 526,000.00 830,000.00 125.62 829,874.38 05 Training 316,000.00 288,000.00 78.39 287,921.61 06 O&M: Implementation

Management 49,000.00 49,000.00 33,985.63 15,014.37

07 Interest Charge 203,000.00 203,000.00 35,048.92 167,951.08 08 Unallocated 758,000.00 0.00 0.00 0.00

Total (in SDR) 5,655,000.00 5,655,000.00 532,463.20 5,122,536.80

Total (in $) 8,060,000.00 8,060,000.00 751,555.83 7,852,220.80

Grant 0031 (in $)

Category Original

Allocation Last Revised

Allocation Amount

Canceled Amount

Disbursed

3101 Consulting Services 1,490,000.00 1,400,000.00 (27,023.91) 1,427,023.91 3201 Civil Works 5,290,000.00 5,290,000.00 801,299.41 4,488,700.59 3601 Vehicles 120,000.00 61,000.00 3,413.76 57,586.24 3602 Equipment 1,030,000.00 1,030,000.00 637,177.52 392,822.48 3701 Learning Materials 570,000.00 570,000.00 211,002.48 358,997.52 3801 Training - 149,000.00 (336,189.63) 485,189.63 3901 Implementation

Management Costs 40,000.00 40,000.00 35,690.87 4,309.13

3902 Incremental O&M Costs

60,000.00 60,000.00 38,860.78 21,139.22

Total 8,600,000.00 8,600,000.00 1,364,231.28 7,235,768.72

Grant 0032 (in $)

Category Original

Allocation Last Revised

Allocation Amount

Canceled Amount

Disbursed

3101 Consulting Services 1,490,000.00 1,400,000.00 (27,023.91) 1,427,023.91 3201 Civil Works 5,290,000.00 5,290,000.00 801,299.41 4,488,700.59 3601 Vehicles 120,000.00 61,000.00 3,413.76 57,586.24 3602 Equipment 1,030,000.00 1,030,000.00 637,177.52 392,822.48 3701 Learning Materials 570,000.00 570,000.00 211,002.48 358,997.52 3801 Training - 149,000.00 (336,189.63) 485,189.63 3901 Implementation

Management Costs 40,000.00 40,000.00 35,690.87 4,309.13

3902 Incremental O&M Costs

60,000.00 60,000.00 38,860.78 21,139.22

Total 8,600,000.00 8,600,000.00 1,364,231.28 7,235,768.72

iii

10. Local Costs (Financed) - Amount ($) N/A - Percent of Local Costs N/A - Percent of Total Cost N/A C. Project Data

1. Project Cost ($ million)

Cost Appraisal Estimate Actual

Foreign Exchange Cost 19.920 22.324 Local Currency Cost 10.080 3.988

Total 30.000 26.312

2. Financing Plan ($ million)

Cost Appraisal Estimate Actual

Implementation Costs Borrower Financed 4.740 3.988 ADB Financed 8.060 7.852 DFAT Financed 8.600 7.236 MFAT Financed 8.600 7.236

Total 30.000 26.312

ADB = Asian Development Bank, DFAT = Department of Foreign Affairs and Trade, MFAT = Ministry of Foreign Affairs and Trade, O&M = operation and maintenance, SDR = special drawing rights.

3. Cost Breakdown by Project Component ($ million)

Component Appraisal Estimate Actual

Civil Works 18.27 16.63 Equipment: Vehicle 0.41 0.20 Equipment: Other 3.54 1.33 Learning Materials 1.98 1.23 Consulting Services 5.17 4.87 Training 0.00 1.66 O&M Implementation 0.14 0.04 O&M Recurrent Costs 0.20 0.05 Interest During Construction 0.29 0.30

Total 30.00 26.31

4. Project Schedule

Item Appraisal Estimate Actual

A. Component 1 Primary curriculum Q1 2006–Q1 2008 Q1 2009–Q1 2013 National assessment Q1 2006–Q4 2012 Q1 2009–Q1 2013 Community partnership program Q2 2006–Q2 2007 Q1 2009–Q1 2013

B. Component 2 Strategic framework for teacher development Q4 2006–Q42012 Q1 2009–Q1 2013

C. Component 3 Subproject appraisal report in 13 schools Q1 2006–Q3 2007 Q2 2008–Q2 2014 Detail design Q1 2006–Q2 2008 Q2 2008–Q2 2014 Construction supervision

Q3 2006–Q4 2012 Q2 2008–Q2 2014

iv

Item Appraisal Estimate Actual

D. Component 4 Education research Q2 2006–Q4 2014 Q1 2009–Q1 2013

E. Component 5 Strategic coordination and management Q1 2006–Q2 2007 Q4 2006–Q4 2007 Integrated financial management system Q1 2006–Q3 2006 Q4 2006–Q4 2007

5. Project Performance Report Ratings Implementation Period

Ratings

Development Objectives Implementation Progress

From 14 August 2006 to 31 December 2006 Satisfactory Satisfactory From 1 January 2007 to 31 December 2007 Satisfactory Satisfactory From 1 January 2008 to 31 December 2008 Satisfactory Satisfactory From 1 January 2009 to 31 December 2009 Satisfactory Satisfactory From 1 January 2010 to 31 December 2010 Satisfactory Satisfactory From 1 January 2011 to 31 December 2011 On Track On Track From 1 January 2012 to 31 December 2012 Potential Problem Potential Problem From 1 January 2013 to 31 December 2013 On Track On Track From 1 January 2014 to 31 December 2014 Potential Problem Potential Problem From 1 January 2015 to 31 December 2015 On Track On Track

D. Data on Asian Development Bank Missions

Name of Mission

Date

No. of Persons

No. of Person-

Days

Specialization of Members

a

Inception Mission 9–23 Jun 2006 2 20 a, c Joint Review Mission 1 4–6 Dec 2006 1 3 a Joint Review Mission 2 28 May–1 Jun 2007 4 3 a Special Consultation Mission 27–29 Oct 2007 1 3 a Joint Review Mission 3 11–15 Feb 2008 2 8 a, c Joint Review Mission 4 11–15 Aug 2008 2 8 a Joint Review Mission 5 23–27 Feb 2009 2 8 a, c Midterm Project Review 7–22 Oct 2009 2 24 a, c Loan Review 1 23 Feb–3 Mar 2010 1 8 a Loan Review 2 26 Jul–30 Jul 2010 2 8 a, c Joint Review Mission 6 29 Nov–3 Dec 2010 2 8 a, c Loan Review 3 6–10 Jun 2011 2 10 a, c Joint Review Mission 7 19–23 Sep 2011 2 10 a, c Joint Review Mission 8 11–15 Mar 2013 3 14 a, b, e Loan Review 4 17–21 Feb 2014 2 10 a, d Joint Review Mission 9 26–30 May 2014 1 5 a Project Completion Mission 25–29 May 2015 1 5 a a

a = education specialist, b = project administration unit head, c = operations officer, d = associate project analyst, e = consultant.

I. PROJECT DESCRIPTION

1. The Education Sector Project II (ESPII)1 objectives were to support the establishment of a more equitable and effective education system that would improve the learning outcomes for young people. The project outcome was enhanced quality of education that took into consideration the gaps in education for children from disadvantaged areas. The project specifically targeted infrastructure in the secondary schools that had not been supported under the preceding Education Sector Project, and promoted system-wide support for enhancing equity, quality, relevance, and efficiency of service delivery for mainly primary education. The project had five components: (i) introducing curriculum reform and assessment systems; (ii) developing effective teachers; (iii) improving access to quality education; (iv) strengthening capacity to undertake research, evaluation, policy analysis, and planning; and (v) strengthening capacity to implement and manage development projects.

2. When ESPII was designed, the policy and strategic taskforce that developed the Education Strategic Policies and Plan 2006–2015 identified seven priorities that needed to be addressed: (i) insufficient supply and inefficient deployment of well-trained teachers; (ii) poorly designed primary curricula and ineffective instructional practices in both subsectors; (iii) insufficient supply of quality instructional materials; (iv) shortages of teachers in disadvantaged areas and key subjects, such as mathematics and science; (v) poorly maintained school infrastructure, especially in disadvantaged areas; and (vi) limited capacity of the Ministry of Education, Sport, and Culture (MESC) to manage the system.

3. In 2005, during the design, donors and governments signed the Paris Declaration on Aid Effectiveness, committing to a set of principles emphasizing ownership, harmonization, alignment, and mutual accountability for results. ESPII was designed as a partnership between the Government of Samoa, the Asian Development Bank (ADB), and the governments of Australia and New Zealand, bringing the ongoing contributions of each development partner in curriculum, infrastructure, and institutional strengthening under one umbrella. For Australia and New Zealand, the project was also intended to support Samoa in planning and financing to improve education services as a precursor to future sector budget support.

II. EVALUATION OF DESIGN AND IMPLEMENTATION

A. Relevance of Design and Formulation

4. The design of ESPII succeeded in harmonizing and aligning the financial contributions and engagement of three development partners with the government-led sector development program. The Partner Harmonization Framework and the Consolidated Funding Arrangement (CFA) signed between the Government of Samoa, ADB, and the governments of Australia and New Zealand were new to Samoa. The total cost of ESPII was estimated at $30 million equivalent (table 1). 5. The design process was lengthy and meeting the needs of all partners proved complicated, involving two separate design missions, each designing different and sometimes incompatible components. Significant additional work was necessary to ensure that the content of components reflected MESC priorities. The resulting management structure was a hybrid using some government structures and systems alongside ADB procurement guidelines and

1 ADB. 2005. Report and Recommendation of the President to the Board of Directors: Proposed Loan and Technical

Assistance Grant to the Independent State of Samoa for Education Sector Project II. Manila.

2

financial reporting arrangements. A secretariat was established to coordinate implementation, administer project finances, and undertake monitoring and evaluation (M&E). The design focused on the ongoing contributions of each development partner: ADB in infrastructure and research studies; Australian Department of Foreign Affairs and Trade (DFAT) and New Zealand Ministry of Foreign Affairs and Trade (MFAT) in curriculum development and teacher training. The government assured alignment of ESPII with MESC’s Strategic Policies and Plan 2006–2015, as ESPII supported many of the planned development and reform initiatives.

Table 1: Financing of ESPII

Partner Foreign Exchange

$ million Local Currency

$ million Total Contributions

(up to) $ million Percentage

Inclusive of taxes

Samoa $0.38 $4.36 $4.74 15.80%

ADB $6.28 $1.78 $8.06 26.80%

New Zealand $6.63 $1.97 $8.60 28.70%

Australia $6.63 $1.97 $8.60 28.37% TOTAL $19.92 $10.08 $30.00 100.00%

6. Overall, ESPII was relevant at the time of design in that it was consistent with government strategic plans for the sector2; ADB’s country partnership strategy; the development policies of the Australian and New Zealand governments; and international principles of aid effectiveness. Each component was relevant. The secretariat intended to provide a coordination point for the five ESPII components to build an integrated and synergistic program that added up to more than the sum of its parts over time. ADB approved a project preparatory technical assistance (TA) of $350,000 in December 2003.3 Outputs focused on an updated education sector policy and strategy and a feasibility study of investments suitable for ADB financing. The TA was satisfactory but could not anticipate all aspects of a harmonized approach under a new sector-wide approach. 7. Over the course of the 9-year project, government and donor priorities in education changed globally, placing greater emphasis on student learning achievement. While the design was relevant at appraisal, and represented the then desires of the government, in retrospect the design appears dated in its focus on infrastructure and central-level reforms. B. Project Outputs

8. The following describes progress against project outputs (components). Actual achievements against the design and monitoring framework (DMF) are shown in Appendix 1.

1. Component 1: Introducing Curriculum Reform and Assessment Systems 9. New bilingual primary curriculum. Curriculum statements for Years 1 to 8 were produced in seven subject areas: Samoan language, English language, mathematics, science, social studies, the arts, and physical education and health. This was a massive undertaking that required more planning, consultation, and sequencing than the 6 months that had been allowed for in the design. Such an expectation was unrealistic and the new curriculum could not be 2 MESC’s Strategic Polices and Plan 2006–2015 was developed on 30 June 2006 separately from and approved

subsequent to the development of the ESPII components from ESPI. The government assured alignment as ESPII covered a significant proportion of the plan’s development and reform initiatives. Both focused on reducing poverty by improving the quality of and access to primary and secondary education.

3 ADB. 2003. Technical Assistance to Samoa for Preparing the Education Sector Project II. Manila. (TA 4256,

approved in December 2003 for $350,000).

3

expected to be fully embedded as this takes many years. At conclusion, curriculum statements were distributed in schools but there were delays in translating them from English into Samoan and printing and distributing teacher guides. Principals and teachers reported the new curriculum as challenging for some teachers to understand, requiring an excessive amount of time for lesson plan preparation. The project supported development of a bilingual policy, but it did not support teachers with practical transition from Samoan to English as the language of instruction, nor did ESPII provide support for improving English language competence. Overall, the output was a huge undertaking and was largely achieved. 10. Adequate supply of learning materials and teacher manuals. Teacher manuals were produced for all subjects to support the implementation of the curriculum and classroom assessment, as was a handbook for the new bilingual policy. Difficulties were experienced with translation and long delays (up to 4 years) were encountered with printing and distribution. Not all translations were available to all primary teachers even by 2015. A state of the art multimedia unit was developed which, though not yet fully functional, has produced audiovisual, digital, and print materials for secondary science subjects. During school visits, it was not easy to determine which resources were provided under ESPII compared with those from other sources.4 11. Community partnerships program for demand generation and improved learning outcomes. A community partnerships program of nonformal adult education courses was piloted through the Fagaloa Community Learning Center. Construction of a new Fagaloa center was removed from the project owing to unresolvable lease issues with the landowner. Some training activities continued but an internal evaluation in 2010 found that the center was unsustainable so they were discontinued at that time. MESC continued to look at ways to revive it and ESPII continued to engage a Fagaloa Community Learning Center coordinator although no resolution was reached by project end on the center’s sustainability. 12. National Assessment Policy Framework. Following a review of assessment and examination policies and systems, a new Samoan National School Assessment Policy Framework was developed and launched in 2010, covering all years to Year 13. Training in policies, practices, and tools was provided for core trainers and MESC staff. It was accompanied by an operations manual and a 10-day classroom assessment training program for core trainers and curriculum officers. In addition, a new Year 8 Samoan Primary Education Completion Assessment (SPECA) was developed and introduced to replace the national examination. A student education number has been introduced as a unique identifier. However, it has not been technically possible to link the student number number to the Samoa Primary Education Literacy Levels (SPELL) or Assessment Management Information System platforms. 2. Component 2: Developing Effective Teachers 13. Formulation of a National Teacher Development Framework (NTDF). The NTDF was officially launched in 2011 following a two-and-a-half-year process. It provides a policy overview of three key areas of teacher quality; ten core policy statements and processes; and the institutional arrangements for implementation. It addresses professional standards for graduate, registered, and master teachers and a system for teacher registration, providing a comprehensive pathway to support teacher effectiveness. A Leadership and Management

4 ADB. 2007. Proposed Asian Development Fund Grant to Samoa for the SchoolNet and Community Access Project.

Manila (grant 0097-SAM, approved in December 2007 for $5,900,000). Some schools also referred to substantial contributions of learning materials, IT equipment, and other in-kind resources through local and international donations.

4

Capabilities Framework for Principals was developed and a Quality Assured Performance Appraisal System for teachers. MESC created two new divisions for teacher development and for monitoring, evaluation, and review, which allowed for focused implementation of the NTDF. The passing of the Teacher Bill in 2015 has created an enabling legal framework. 14. Increased number of agricultural science, food technology, visual arts, and design technology secondary teachers. Various efforts were made to increase the number of specialist teachers. Ten teachers graduated from a fellowship program in four areas of severe secondary teacher shortage in 2013. A fast-track program was also started to train 100 primary teachers in a 3-month course. However, there were issues of quality in conceptualization and delivery as well as system issues of salary, certification, and lack of alignment with the NTDF, so the fast-track program was reconstituted as alternative teacher training and later discontinued. Thirty-eight primary teachers graduated, some of whom went on to train for their diplomas at the National University of Samoa (NUS). 15. Pre- and in-service training for new initiatives. A program of in-service school-based training was rolled out for 1,150 teachers. Twelve core trainers were recruited and trained initially in the cascade model. When it did not work well, the model was changed to school-based training. Each year for 2 years, every primary school teacher was proposed to have 10 days of professional development and subject specialist support to implement the new curriculum and associated activities.5 Training in leadership was provided for principals and most of those interviewed for this project completion report were able to explain how they were using it in their schools. A mentor teacher program was piloted with limited success. Those trained were not motivated to mentor others, owing to the additional workload, in the absence of incentives. Pre-service training started later than planned, owing to weak lines of communication between MESC and the Faculty of Education of NUS. 16. Continuous pre- and in-service teacher development and school-based support. For in-service training, a series of workshops was held to develop the six key strategies for quality pedagogy and to train the core trainers. These were followed by rollout of the first round of training in schools using the cascade model. This was not very successful and there was some negative feedback about the quality of the core trainers.

3. Component 3: Improving Access to Quality Education 17. Improved secondary school facilities. Nine secondary schools were completed as planned and one (Sagaga) had to be dropped due to unresolvable land issues (described under “Project Schedule”). The bulk of funding for infrastructure went to the new MESC headquarters (see para 21). By 2011, four schools had been completed, but the joint review mission noted that the remaining balance under civil works was only $1.36 million, leaving a funding gap of $3.99 million to complete construction of four remaining schools (Lefaga, Aana, Sagaga, and Avele). School upgrades and construction of teacher housing were therefore reprioritized, and DFAT, MFAT, and the Japan International Cooperation Agency received additional financing of $2.88 million in a parallel account to support the increased infrastructure costs.6

5 Quality of data across most outputs is such that it can be difficult to distinguish between what was supposed to be

delivered and what was actually delivered. 6 During the reprioritization of upgrades, due to a shortage of funds, it was determined that the Fagaloa Community

Learning Center would not require an upgrade under ESPII and that Faleali College would be completed using tsunami funds rather than ESPII funds for its upgrade. Vaipouli College was cancelled, and Sagaga school was eventually dropped due to land issues.

5

18. Improved maintenance. School committees are responsible for meeting maintenance needs.7 In some schools, the signs of low-quality construction and/or use of inappropriate materials were evident. A school maintenance and operations handbook was completed in 2013 but not printed until 2015 and has yet to be distributed. School committees reported receiving training in maintenance using the minimum service standards. There were no formal maintenance plans in place in any of the schools visited, although some principals had taken the initiative to overcome challenges, such as lack of water and power, which were directly affecting the safety of the learning environment and student well-being. 19. Teachers’ houses at four primary schools in rural areas. Four houses for teachers were completed. One, observed during the team visit, was being partly used as additional classroom space. In another, four teachers were sleeping in one room. One was being partially used as additional classroom space. This suggests that consultation on needs was weak and design may have been inappropriate. No data are collected on whether the availability of housing has increased recruitment and retention in remote areas. 20. MESC headquarters constructed. The cost estimates and financing plan estimated that 50% of project costs would be allocated to infrastructure, significantly below the final share of 72% of ESPII’s $30 million budget.8 The bulk of these funds went to construct MESC headquarters. It became apparent early on that headquarter construction costs were much higher than the original $7 million estimate. Unfavorable currency fluctuation of the NZ dollar against the US dollar meant a significant loss of resources for this component. Eventually, the MESC headquarters cost approximately $9 million. This affected the remaining budget to upgrade the secondary facilities and school houses (three colleges, nine secondary schools, and four teacher houses).

4. Component 4: Strengthening Capacity to Undertake Research, Evaluation, Policy Analysis, and Planning

21. Improved national capacity for research and evaluation. Training needs were analyzed during inception and used to inform a plan for professional development that was linked to MESC’s research agenda. Workshops, study visits, and on-the-job training were used to expose staff to the latest techniques in education research and analysis. This included analysis of Samoan data at the University of Auckland through workplace attachment. Most of the staff benefiting from this training have subsequently left MESC. 22. Research and evaluation program with results used for policy development and planning. An ESPII evaluation framework was developed to assess the range of program interventions and a step-by-step approach was developed to align with implementation of the new curriculum. Three international consultants provided research, information management, and national data development support to the Policy Planning and Research Division (PPRD). Various research and analytical studies were completed by 2013, including a review of the sector approach and impact evaluation of the components. These are now hard or impossible to access because they were housed by individuals who have left MESC. Some work was done to establish a research database linked to other MESC databases, but this proved technically challenging and the issues were not resolved by TA end. The intention for research and evaluation results to be used in policy development was not realized, and was probably

7 School committees have varying capacity and/or motivation to raise the amounts required for maintenance. Some

funds are also available under the school fee grants scheme and under annual school grants. 8 See ESPII Project Administration Memorandum, Section 6, Part E.

6

unrealistic, though there is evidence that the skills PPRD staff acquired in planning research activities have been used more generally in the preparation of corporate and annual plans. 23. Improved capacity to evaluate the impact of aid-funded interventions. Individual short-term impact evaluations were conducted for each component of ESPII. These identified issues such as weaknesses in the planning of TA support, the need to develop bilingual policy before writing the curriculum, better coordination of capacity building, and ensuring that preconditions are met before embarking on high-cost initiatives such as the multimedia unit.

5. Component 5: Strengthening Capacity to Implement and Manage Development Projects

24. Improved strategic management and coordination of implementation. The experience of implementing a sector approach has strengthened capacity within MESC. The divisions, especially the Curriculum, Materials, and Assessment Division (CMAD) and School Operations Division were clearly in the driving seat and, over time, worked together effectively. The ESPII Steering Committee (ESC) morphed into the Education Sector Advisory Committee (ESAC); the development of the Education Sector Plan is evidence of strategic capacity. The ESPII secretariat evolved into the new education sector coordination division under MESC’s revised organization structure, providing a strategic coordination mechanism for sectorwide initiatives while also accommodating the role of secretariat for ESPII. However, there was no intentional capacity development plan for this output and it was commonly perceived to relate to the secretariat rather than to MESC. 25. Integrated financial management system. Alignment was intended to utilize MESC systems for financial management. In fact, it was done through the secretariat rather than the MESC Corporate Services Division and some aspects of the financial management system, such as financial statements and withdrawal applications, did not meet the requirements of ADB. In order to meet the expected standards, the Ministry of Finance (MOF) took over responsibility for financial management of the ESPII. 26. Effective performance monitoring, evaluation, and reporting system. Project performance monitoring, evaluation, and reporting were weak. There were two evaluation systems in place: the DMF, which was the responsibility of development partners, and the monitoring and evaluation framework (MEF), intended to guide implementation at a more detailed level. Although these were intended to be complementary, that intention was lost, especially following the midterm review. In 2011, the joint review mission recommended TA to support the development of a more sophisticated framework for future budget support. The subsequent MEF was later replaced with a new framework developed by the coordinator and moved beyond DMF indicators. 27. One of the challenges of monitoring, and reporting, was an institutional culture that did not support information sharing. The first system established in 2011 gathered information from mid-level personnel in the divisions through continuous requests and visits. The second system required information to be collected at the level of assistant chief executive officers (ACEOs) and shared in coordination meetings. That system was unsuccessful because the ACEOs were already overloaded and did not find the information sufficiently useful to make it a priority. 28. Development partners were to jointly assess progress during the midterm review and joint review missions, which regularly raised the issue of monitoring and evaluation. However, MESC perceived the reporting to be donor driven. The system in which quarterly project

7

performance reports were provided to ESC, development partners, and the government proved inefficient in making and communicating timely decisions on resolutions or recommendations. As the focus of the ESC became increasingly sector oriented, the gap in follow-up increased. 29. Improved tendering and contract management. Visits to schools revealed poor-quality construction and the use of low-quality or inappropriate materials. In some, it was clear that improvements in the overall school environment would have been beneficial in addition to improvements in construction. Aleipata school experienced problems with the construction company, which subsequently went into liquidation. But in most cases, problems arose because of flaws in the design and quality assurance process. MESC had regular meetings with the supervisory firm for all school buildings, making joint site visits, and strengthening reporting and partnerships with the respective school committees. However, the defects clause, which was supported by the external quality assurance contract, appeared not to have been used very robustly to address outstanding construction issues. C. Project Costs

30. The total project cost was $26.31 million, against $30.00 million at appraisal. Costs comprised (i) $7.85 million from the loan; (ii) $7.24 million from DFAT; (iii) $7.24 million from MFAT; and (iv) $3.99 million from the borrower. Overall, ADB paid 29.84%, the government paid 15.16%, and DFAT and MFAT provided 27.50% each of the total project cost. The revised loan balance was SDR5,122,536.80 ($7,852,221.00) after cancellation of the unutilized loan balance of SDR532,463.20 ($751,556.00) at loan closing on 22 September 2015. The undisbursed balance of $1,364,231.28 each for the DFAT and MFAT grants was cancelled at grant closing. 31. The reduced costs were principally due to the dropping of four schools from the original 13 to be built. The reasons for dropping these schools were: (i) Fagaloa Learning Center was deemed to have sufficient buildings and facilities for the population it served; hence construction of new facilities was not a priority; (ii) Vaipouli College was closed because two other secondary schools serve the district where Vaipouli was located; (iii) Falealili College was destroyed during the 2009 tsunami, and funded under the tsunami recovery program; and (iv) Sagaga school was dropped after prolonged delays due to unresolved land issues that ran from 2012 until 2014. D. Disbursements

32. Loan disbursements totaled $7.85 million (29.84% of the revised total project amount of $26.31 million) while grant disbursements were $14.47 million from DFAT and MFAT cofinancing (55.00% of the revised total project amount). All disbursements comprised 74.41% of the approved project amount of $30.00 million. As part of the CFA agreed between the government, ADB, DFAT, and MFAT, a special purposes account was established in country. DFAT and MFAT agreed to deposit an initial advance into the account as a float to allow Samoa to incur project-eligible expenditures. MFAT, DFAT and the government were to agree the amount of the advance following the production of the project’s annual plan. Apart from the $751,556.00 from the ADB loan; $1,364,231.28 from the DFAT grant; and $1,364,231.28 from the MFAT grant, which were cancelled, the loan and grant funds were fully disbursed. E. Project Schedule

33. The project implementation was planned to cover 6 years, from 2006 to 2012. Under the CFA, Australia and New Zealand advanced an agreed amount of funds into a special purpose account. Although the funds were non-acquittable under ADB rules, a quarterly statement of

8

expenditures was required. In addition, there was provision for parallel or other contributions that required acquittals. The first steering committee meeting, in May 2007, noted that the project was very ambitious. It took 6 months to reach agreement on the project’s implementation and administration manual as it had to be harmonized among three development partners with different processes, time frames, and procedures, as well as in consultation with the government. This led to procurement delays. By 2008, there were delays in achievement of outputs of between 6 weeks and 18 months, there was no agreed MEF, and there were still delays finalizing the manual. The consulting firm providing TA for components 1, 2, and 4 was not contracted until January 2009, almost 2.5 years into the project. 34. By 2010, the midterm review noted that most of the lessons learned involved underestimating what could be achieved within the project’s time frame and that start-up required planning that normally takes place during the design phase. The review concluded that there was a sound springboard in place but that a review was needed of what could reasonably be achieved before project completion and planning for a sustainable transition thereafter. From that point, the project gained momentum and the granting of a 1.5-year extension from June 2013 to December 2014 allowed most outputs to be completed satisfactorily. 35. There were some delays due to changes in personnel including three MESC chief executive officers (CEOs), three NUS vice chancellors, two ESPII coordinators, two Polytechnics International New Zealand (PINZ) team leaders, and five ADB project officers. Two natural disasters, the tsunami in 2009 and Cyclone Evan in 2012, disrupted activities. In the infrastructure component, the proposed upgrade of Sagaga school was delayed by 2 years because the site had to be changed, first due to natural-disaster risk and again when two villages could not agree on an alternative location. It was ultimately cancelled when due diligence revealed appeals on the land. Project completion was extended once, from 30 June 2013 to 31 December 2014. The project was financially closed on 22 September 2015. F. Implementation Arrangements

36. The daily management oversight of ESPII was the responsibility of MESC’s core executive, while the division heads (ACEOs) implemented project components. Because of the heavy burden of procurement associated with the large infrastructure component, ESPI project management unit staff were retained as an asset management unit in the new ESPII secretariat. This arrangement worked well although it was challenging initially. Former ESPI staff had experience with ADB procedures but MESC itself had none, so learning ADB procedures was a major task. ESPII staff were funded by government counterpart funds and were therefore supposed be under the Public Service Commission, however MESC was involved in the recruitment process. A ban on MESC staff taking on consultancy work, although a sound decision, led to a dearth of national consultant positions to meet capacity needs. 37. There was weak coordination of implementation across the components and divisions of MESC, highlighting the need for a project coordinator in the secretariat to improve the rate of progress and information sharing across the whole project. The first coordinator was later promoted to MESC CEO, which resulted in some positive ownership effects. But the second coordinator was transferred from the role of SchoolNet project coordinator and, when her previous position was not filled, she ended up doing both jobs. For this reason, SchoolNet become known as Component 6 of ESPII although it was actually a separate project with its own implementation arrangements and reporting requirements. As a result, the coordinator was overloaded, which helps explain why the secretariat never had the capacity to manage

9

monitoring and evaluation. The existence of a coordinator also helped perpetuate the view that ESPII was a separate project rather than the core work of MESC. 38. ESC originally provided governance of ESPII. Following the midterm review, which emphasized the importance of planning beyond the lifetime of ESPII, most attention was diverted to the development of the education sector plan. During this time, the ESC morphed into the ESAC and the Informal Technical Working Group (ITWG) became the education sector working group. This led to major changes in working methods and membership of the various committees and groups and their orientation to the future sector plan but left a gap in the governance of ESPII. To fill the gap, development partners increasingly used the ITWG as a management meeting in which they sought to gather information that enabled them to meet their respective accountability needs, such as implementation tracking and financial reports. The ITWG was also used to clear documents before they moved to the ESAC although it had no official status within ESPII or government systems. G. Conditions and Covenants

39. Compliance with loan conditions and covenants, including conditions under the CFA, was generally satisfactory and the covenants were relevant to ESPII’s purpose. Covenants were also addressed under the SchoolNet project. At completion, 19 covenants have been complied with and three have been partially complied due to either delays or the covenant being covered under the Schoolnet grant. Detailed status of compliance with loan covenants is in Appendix 3. H. Related Technical Assistance

40. In addition to the project preparatory TA, ADB provided further TA,9 piggybacked onto the project, for the development of national teacher development frameworks, which made key recommendations on increasing demands for professional development, increasing the number of personnel in MESC, establishing a professional development division within MESC, and expanding support of the Curriculum, Materials, and Assessment Division; Faculty of Education; and NUS. Another small-scale TA project10 was provided for the development of Samoa’s Education Policy and Strategic Plan 2006–2014, which supported an (i) agreed process for policy reform; (ii) an education roadmap; and (iii) an agreed financial planning, management, and monitoring system for the reform of each subsector, with detailed budgeting, funding options, and prioritized activities. Both TAs were complementary and critical to the project. I. Consultant Recruitment and Procurement

41. For components 1, 2, and 4, TA was provided through PINZ. Work commenced in January 2009 and lasted until 2013. A significant issue in the PINZ contract was that the terms of reference (TOR) had been written in 2006 and were not revised even though the contract was not issued until 2009. In the meantime, key policy documents in curriculum, assessment, bilingual education, and special needs education had been developed and new initiatives such as SchoolNet had started. This led to some consultants being recruited with TOR for key tasks that had already been completed and others for tasks that were no longer needed. PINZ-implemented TA inputs are detailed in Appendix 2.

9 2005. ADB. Technical Assistance to Samoa for the National Teacher Development Frameworks. Manila. (TA 4738-

SAM, approved in December 20015 for $350,000). 10

2004. ADB. Technical Assistance to Samoa for Preparing the Education Policy and Strategic Plan 2006–2014. Manila. (TA 4433-SAM, approved in November 2004 for $150,000).

10

J. Performance of Consultants, Contractors, and Suppliers

42. Most consultants performed well. The first PINZ team leader had to be replaced after 1 year owing to MESC’s dissatisfaction with the curriculum being developed in English without adequate stakeholder consultation. Although the project attempted to maximize usage of national consultants, the only option after a failed round of recruitment was to partner with NUS. Most NUS staff only worked on ESPII part time, which contributed to delays. It was assumed that national consultants would be more effective and have context-specific expertise. However, not all national consultants had the necessary knowledge or skills. 43. Overall, consultants were heavily relied upon in the early years of implementation in an attempt to make up lost time. The result was that activities were inappropriately phased and sequenced. Inputs to assessment, for example, were concurrent with curriculum development whereas it would have been logical to proceed sequentially. Research was compromised by frontloading and longitudinal studies were not sustained for a long enough period. These factors gave rise to questions about value for money. Monitoring was weak and, although inflexibility was recognised as a major problem, nothing was done to remedy it. K. Performance of the Borrower and the Executing Agency

44. The performances of the borrower and executing agency (MOF) and the implementing agency (MESC) were satisfactory in meeting the responsibilities assigned in the implementation plan. The midterm review stated that MESC had developed significant strengths in leadership, planning, and the development of structures and administrative systems to support ESPII. The heavy staff workload put MESC’s absorptive capacity under greater strain than the design appraisal had assessed. MOF eventually took over financial management, which was weak. Momentum picked up mid-project, especially after the new ACEOs were appointed in 2013 and a restructuring exercise in MESC separated the Curriculum, Teacher Development, and Monitoring Division and the Evaluation and Review Divisions. 45. In evaluating outcomes, neither MESC nor the secretariat had the capacity to develop an appropriate M&E system. The midterm review recommended MESC develop a consolidated sector-wide monitoring and reporting system, aligned with the corporate plan and meeting ESPII needs. Subsequent joint review missions always requested monitoring information against outcomes. But different M&E frameworks were used at different times throughout the project and the tension between project M&E and MESC’s own requirements were unresolved. In part, this was because TA for M&E concluded in 2013 and the framework developed aligned fully neither with the DMF nor with the MESC framework. The absence of quality M&E has meant that a great deal of good work that has been done on the project has not been fully reported. At completion it has been difficult to prepare the DMF because it appears not to have been used and the data collected in other formats could not be read across different indicators. L. Performance of the Asian Development Bank

46. After a very complicated design process, the development partners worked well together and participated in the annual joint review missions, in formal ESC and ESAC meetings, and in informal meetings. MESC appreciated this cooperation, especially where participation helped clarify development partners’ expectations. Two areas would have benefited from more development partner input. M&E was acknowledged to be weak throughout and, although a consultant was recruited in 2011 following a joint review mission recommendation, the result was not satisfactory. By 2013, the funding allocation for TA was fully utilized. In the remaining

11

two years to 2015, it would have been useful to provide additional TA funds to maintain sight of the DMF so that it could be properly reported on at completion as well as to ensure that ESPII M&E was adapted to align with the emerging framework for the Education Sector Plan. Development partners might also have contributed more to clarify the role and purpose of the new Education Sector Coordination Division (ESCD). 47. ADB coordinated with the government and development partners in project administration. There were five ADB project officers over the course of the 9-year project. The high staff turnover hampered smooth implementation, oversight and monitoring. Nevertheless, ADB participated in all joint review missions and conducted eight review or special missions, totaling 17 missions. ADB was generally responsive to the needs of the country, providing significant support to MESC and MOF on ADB’s procurement and financial procedures and processes. In 2007, project implementation training was offered to two ESPII secretariat staff at the ADB Pacific subregional office in Suva, Fiji. Loan disbursement training was provided in 2014 at ADB’s Samoa office. More, and earlier, training would have helped the rapid learning curve during mobilization of a complex project. ADB’s performance was considered satisfactory. 48. DFAT’s and MFAT’s main nonfinancial contributions consisted of personnel in Apia, who attended all ESPII-related steering committees and working groups, and met regularly with MESC, MOF, and other sector partners. Canberra- and Wellington-based education specialists participated in joint review missions, the midterm review, and periodic in-country missions, and provided on-demand desk-based support to program teams. During the period when MESC contracted a consulting firm (2009–2013), the level of support from development partners was adequate. After 2013, when staff budgets were reduced, the pool of technical expertise was less.

III. EVALUATION OF PERFORMANCE

A. Relevance

49. The project is rated relevant. The design modality reflected commitment to the principles of aid effectiveness, harmonization, alignment with government systems, and mutual accountability for results. While highly ambitious, the individual components combining infrastructure and reform have remained relevant. Slow curriculum development means there has been little expenditure on materials for students. There has been training on assessment practices in the classroom, but the majority of assessment inputs focused on developing the National Assessment Framework and introducing SPECA. One of the successes of ESPII has been the evolution of the project-oriented ESC to ESAC.11 This transition from donor-funded projects to budget support is significant, aligning with MOF’s preferred sector-wide approaches. B. Effectiveness in Achieving Outcome

50. The project is rated effective. Nine indicators of achievement were established at the outcome level and progress against them is set out in Appendix 1. In the area of curriculum reform, the outcome of complete curriculum statements for Samoan, English, and mathematics was achieved. The target of 90% of primary and secondary school teachers effectively using the new instructional methods and new national assessment system has not been met. Although the systems have been rolled out, their effectiveness could only be measured at impact level if there was an improvement in student achievement. Curriculum reform, and implementation to

11

This was prompted by the recommendation of the 2010 midterm review to think about a sustainable transition from ESPII.

12

this level of expectation, takes at least a decade even in developed countries. The expectation that adequate numbers of trained secondary teachers for mathematics, science, agricultural science, food and textiles, visual arts and design, and technology could be available as a result of a relatively small intervention for that purpose was unrealistic. The NTDF was successful (see para 13). The Primary Completion Rate (Year 8), and the national transition rate into rural secondary schools both increased, almost reaching targets. One indicator (proportion of students from the two lowest wealth quintiles transitioning to secondary) is unmeasurable. 51. Critically, there has been no improvement in learning achievement. The percentage of students “at risk” in Year 4 and 6 Samoa Primary Education Literacy Levels (SPELL) tests is difficult to interpret reliably. There have been some improvements but, across all measures, the percentage in the “at risk” category is unacceptably high. At SPELL 2 (Year 6), 42% of students in English and 61% in math are at risk compared with 14% in Samoan.12 Beyond SPELL, the indicator of learning achievement was a 20% increase in scores in the Year 8 National Examination. This examination has been discontinued, and the new SPECA is not comparable. 52. ESPII might have been more effective had it tried to do less. The volume of reform was overwhelming and the available capacity insufficient. Activities could have been implemented in a logical and progressive sequence rather than simultaneously. Although effectiveness increased as the necessary connections between components and divisions were made, the pressure to reach targets has reduced quality. This can be seen most clearly in the curriculum. There was a critical point when dissatisfaction about development of the curriculum in English was very high and it might have been more effective to start again in Samoan. Now it is much harder to effectively implement curriculum reform, potentially undermining impact. 53. The infrastructure component has been mixed. While some school buildings are attractive, well utilized, and conducive to learning, others have fundamental problems hindering student learning and may widen inequalities between schools. A combination of poor design (focusing on the physical building rather than the whole school environment); weak quality control by the secretariat and low motivation to address issues that arise; inadequate processes for community engagement in the design and ongoing maintenance; and MESC’s lack of an institutional home for operations and maintenance, have contributed to the mixed results. 54. The least effective components were 4 and 5. Since 2013, when the consultant input concluded, little research, analysis, and evaluation has been conducted, and what reports were drafted remain inaccessible since the staff departed MESC. With Component 5 relating to the secretariat, there was a missed opportunity to plan for capacity development across MESC. Both components had the potential to be more effective had they been planned and implemented differently. In effect, they were dwarfed by the ambition of the three large components and made only a marginal contribution to achievement of outcomes. C. Efficiency in Achieving Outcome and Outputs

55. The project is rated less than efficient. Efficiency was challenged by the overambitious reforms for the project time frame. Indeed, all development partners were challenged by the project’s expected pace. Harmonization was intended to reduce transaction costs for the government and development partners, but these costs actually increased in the initial stages. Reconciling individual accountability systems and different funding modalities of

12

It is worth noting that SPELL does not cover all primary schools in Samoa, hence the results may not truly reflect learning achievements at the primary level.

13

the three development partners was protracted, causing delays to implementation that led to a race to catch up. This rush included frontloading consulting services and other project resources to try to achieve the already ambitious milestones. This process proved to be inefficient because the consultants were utilized too early when MESC could least well absorb them and person-months were exhausted by the time momentum was gained. 56. The design of the project clustered activities from previous projects but in such a way that they could not be managed as a single activity. The two separate consultancies, for the MESC headquarters construction (Component 3) and for education services (Components 1, 2, 4), were difficult to coordinate efficiently. In effect, each component became a project in itself, making it more difficult to forge necessary interconnections. It might have been more effective and efficient had the design been outcome oriented (effective teachers who bring about the learning outcomes of the curriculum) rather than output driven by components and divisions.

57. Although ESPII was intended to channel all development partner resources through a single arrangement, the partners went on to implement new projects in parallel. ADB introduced SchoolNet, which was nominally Component 6 of ESPII but actually had separate reporting requirements. When the SchoolNet coordinator became secretariat coordinator concurrently, effectiveness and efficiency were reduced. DFAT and MFAT, at the government’s request, introduced the primary school fee grants outside the scope of ESPII, as is the New Zealand 50th anniversary initiative of secondary school fee grants. D. Preliminary Assessment of Sustainability

58. The project outputs and outcomes are rated likely to be sustainable. There are signs of sustainability, as evidenced by ownership of reforms to improve student learning. There is reasonable expectation that MESC ownership will continue and perhaps increase after the end of the project. There is now more capacity both within and between the divisions and the move to contracts for senior staff is an indication that performance is being taken seriously. Increase in capacity has also contributed to Australia and New Zealand’s confidence in moving toward budget support of the education sector so that the government can allocate resources according to the priorities set out in the Education Sector Plan 2013–2018. 59. The passing of the Teacher Bill enshrines the teacher workforce reforms in law and paves the way for sustainable long-term change. It is too early to say whether the structural and technical reforms in MESC are sustainable although the progressive merging of activity areas supported by ESPII into the core functions of MESC is promising. There is still a structural separation and some duplication across curriculum, teacher development, and assessment, creating a barrier to orienting the overarching focus to student learning. The research and monitoring functions of PPRD have lost traction since TA ceased in 2013, although the skills developed are being used in wider MESC planning. Changes in PPRD management may have contributed to ending the research and monitoring functions. 60. Although signs are promising, reforms at the classroom level are fragile. The changes in pedagogy and classroom practices that have been introduced are a paradigm shift for Samoan educators. Many cannot yet make that shift either because they do not fully grasp the concepts and the methods of the new curriculum or because they are not motivated to take on the substantial additional work the new way of teaching requires. There is a risk that teachers may not improve or sustain their classroom practice and school principals may not continue to lead effective change. After such intense activity, there may be a loss of momentum especially as

14

government recurrent budgets for non-salary costs are inadequate. If this happens, students will not receive the quality education they deserve and their learning achievements will not improve. 61. The least encouraging aspect of sustainability concerns operations and maintenance. Although it is being done well in some cases, by motivated principals and willing school committees, the poor state of the school environment in other cases suggests that the current institutional arrangements are not working. This is borne out by the observation that combined expenditures on school improvements, asset management, and public library services accounted for an average of only 5.8% of total MESC spending in FY2008–2014. The government appears to have sufficient fiscal capacity to fully support the education sector. For most of the fiscal years for which complete fiscal data are available (FY2008–2014), especially more recent years, actual expenditures on education exceeded costs projected in the initial project document. MESC resources (including loans but excluding grants) more than adequately funded expenditures on education, although the buffer appears to have decreased steadily.13 E. Impact

62. The project impact is rated ‘moderate’. All involved acknowledge that there are issues with the validity and reliability of statistics; there is some consistency, however, in trend data.14 Net primary enrollment is recorded at 100% and primary completion at 85%. Net secondary enrollment is around 68%. There is modest gender disparity in favor of boys at the primary and girls at the secondary level, with a general trend for girls to outperform boys. The literacy rate in adults is claimed to be 90% in Samoan and 75% in English although this appears overstated when viewed in relation to the results of SPELL and the Pacific Islands Literacy and Numeracy Assessment. Other indicators of impact were increased enrollment at the tertiary level (achieved) and improved employment rates of high school graduates, for which there are no data.

IV. OVERALL ASSESSMENT AND RECOMMENDATIONS

A. Overall Assessment

64. Overall, ESPII is rated successful. This balances the achievement in partnership relations gained through a modality oriented to aid effectiveness (the process) with the as yet unchanged low level of student performance (the impact). ESPII was ambitious, being associated with a major reform phase in Samoa’s education system. It had a ‘slow start’ although the evidence suggests that the first several years were devoted to the critical planning exercises for curriculum, assessment, and teacher development activities that had been wholly underestimated in the design. In hindsight, it may have been too much to expect transformative changes in student performance as a project outcome in such a short period. However, the project has made a very important contribution to reforms initiated by the government and to the strong drive for positive change in the education sector. ESPII provided the enabling environment for the whole of the sector advisory committee and a comprehensive Education Sector Plan to emerge. ESPII also supported capacity development, paving the way for DFAT and MFAT to provide sector budget support. Considering the implementation challenges, the monumental reform effort in a microstate context is noteworthy.

13

Values used in this re-evaluation were indexed to FY2005 prices for consistency with the initial project document. 14

Data at impact level are from the 2013 Statistical Digest.

15

B. Lessons

65. Flexibility. Greater flexibility in design and adherence to targets might have enabled more discussion and better communication and decision making within an ambitious time frame, especially during the first year. The DMF could have been revised early on, reducing the pressure on MESC and alleviating the sense that they were not achieving results. 66. Capacity. Although capacity has been developed (though not always retained) at an individual level, the benefits are less evident within MESC or more broadly. Capacity development could have been greater had MESC capacity been thoroughly assessed during design or had design fully considered the absorptive capacity of a small ministry. High staff turnover at the management level also contributed to challenges in building capacity. 67. Curriculum development. The new curriculum provides a starting point for confident, capable teachers but is reportedly challenging for weak teachers. Developing the curriculum in the language the teachers use to plan and deliver their lessons (Samoan) would have lessened the risk of creating new vocabulary and provided a better base for weak teachers. 68. Evidence base. The loss of much of the evaluative work and studies done under ESPII when key staff departed leaving no document archive and the inability to provide quality data for the project completion report highlight the importance of institutionalizing a strong system for monitoring, evaluation, and learning. 69. Future Education Sector Program. The Education Sector Plan is ambitious. Human capacity can easily be overloaded by attempting too many simultaneous reforms. Having fewer, but well-sequenced and phased priorities, is likely to have more impact. C. Recommendations15

70. Making sure that ESCD can perform its role effectively. As a new division, ESCD will need support to ensure it is not overwhelmed and it would benefit from in-depth work where ESPII lessons have been learned, i.e., role identification, M&E, and risk management. 71. Active resource-intensive consolidation of key areas such as multi-tiered teacher training and professional development, additional resources in support of the new curriculum, and community engagement and school leadership. 72. Shifting the focus of energy and effort to the student level, including school-level early diagnostics of student learning and an emphasis on ongoing assessment.

73. Keeping curriculum review and development as an ongoing process through collection of feedback from teachers and students about the impact of the curriculum changes.

74. Improving operations and maintenance through a review of construction materials, maintenance needs, and future infrastructure needs to minimize maintenance expenditures. School maintenance plans could be part of the School Service Standards and maintenance financed under the school free fee grant scheme.

15

Detailed recommendations appear as Appendix 6.

16

Ap

pe

nd

ix 1

DESIGN AND MONITORING FRAMEWORK FOR ESPII (DRAFT AS OF 20 MARCH 2008)

Design Summary Performance Targets Actual Achievement Data Sources/

Reporting Mechanism Assumptions and Risks

Impact A more equitable and effective education system that enhances learning outcomes of young people

By 2015: 100% net primary enrollment 100% of children, boys and girls alike, will be able to complete a full course of primary education (MDG) Eliminate gender disparity at all levels of education (MDG) 85% total secondary net enrollment 100% functional literacy rates Increased enrollment at the tertiary level Improved employment rates of high school graduates

100% net primary enrollment (2014) 85% primary completion Primary 0.93 girls: 1.07 boys Secondary 1.07 girls: 0.93 boys 69% (2014) 90% literacy rate in Samoan, 75% literacy rate in English (15–40 yrs.)

a

Increased from 913 students in 2007 to 1221 in 2014

b

No data

United Nations (human development index) report

Govt. of Samoa/SPC MDGs statistics and reports

MESC statistics

FBEAP reports and meetings

NUS and University of the South Pacific data and reports

Labor market survey

Assumption

Government develops sector plan and financing framework for the sector that assure adequate resources for recurrent costs, including nonsalary inputs for basic and secondary education, to ensure long-term sustainability

Risks

External economic shocks reduce revenues available for education

Political and budget priorities do not include education

Outcome Enhanced quality of education

By 2012: 90% of primary and secondary school teachers effectively use the new instructional methods and new national assessment system Adequate numbers are available of trained secondary teachers for mathematics, science, agricultural science, food and textiles, visual arts and design and technology

In use: 100% achieved Effectively used: no data (partially achieved) No. of secondary teachers: 368 (2014)

c

Not achieved – some

Implementation reviews, progress performance reports, completion report, results from research and evaluation studies under Component 4

Classroom observations of teacher and student interaction and school processes, teacher effectiveness study under Component 4

Reports of principals and

Assumptions

Government commitment to address inequities in access, retention, and achievement is sustained

Terms and conditions of teacher service will be sufficiently attractive to recruit and retain qualified and motivated staff

Risks

Political and public support will not be sufficient to support implementation of

Ap

pe

nd

ix 1

17



% of students “at risk” in Yr. 4 & 6 SPELL tests in English, Samoan, and math reduced to under 15% Average scores in core subjects in Year 8 national examination increased by 20% Primary completion rate (Year 8) increased from 93% (2006) to 98% National transition rate into secondary schools increases from 89% (2006) to 98% Transition rates into rural secondary schools increases from 72% (2006) to 90% Proportion of students from the two lowest wealth quintiles who transition to secondary schools increases significantly Complete curriculum statements for Samoan, English, and math

deterioration SPELL 1 SPELL 2 2007 2013 2007 2013 E 13% 26% 56% 42% S 19% 16% 12% 14% M 27% 29% 59% 61% Achieved in English. Not achieved in Samoan and math

2006 2012 English 36 49 36% increase Samoan 50 46 0.8% decrease Math 44 49 11% increase PCR 97% – almost achieved 89% – almost achieved 2013–2014: 94% 82%(2006)–90% (2014) Not measurable Achieved

school review officers

Test results

Assessment of learning outcomes using new assessment system

MESC statistics

Results of study into causes of low achievement under Component 4

Education For All Mid-Decade Assessment Report, Samoa 2007 (provides 2006 baseline data)

required reforms

Project activities cannot address issues of rural school enrollment

Lack of employment opportunities available for high school graduates

18

Ap

pe

nd

ix 1



Outputs 1. Introducing curriculum reform and assessment systems 1.1 New bilingual primary curriculum for Samoan, English, social studies, math, science, social science, arts, PE/Health, visual arts, performing arts, and music 1.2 Adequate supply of learning materials and teacher manuals 1.3 Pilot community partnerships program 1.4 National assessment policy framework and systems

All students have access to a complete set of learning materials as specified in curriculum CMAD equipped with and able to use appropriate equipment for the production of print and nonprint materials Materials available in schools beginning in 2011 with teacher training in 2010 Production of teacher manuals following completion of curriculum, per schedule above Curriculum statements produced by December 2010 Fagaloa CLC pilot implemented and evaluated by December 2012 Home school literacy program evaluated by December 2012 Policy framework by 2009 CMAD staff trained and using systems effectively New database for MESC

Invalid indicator

d

Partially achieved No materials available in 2011. Partially achieved by 2015 Partially achieved Achieved in 2012 Achieved (but discontinued) No data Achieved in extension Achieved

Joint review mission reports

MESC statistics

Project progress reports

Procurement and distribution schedule

Principal and school review officers school reports

Community consultation reports

CLC participation rates and trends

Research and evaluation studies under Component 4


report generated on annual examinations and test

Risks

Procurement of textbooks and learning materials is delayed

Procurement of teacher manuals is delayed

Risks

Location and design assumptions are flawed

Community disengaged re process and outcomes

Community learning center no longer a priority

Assumption

Sufficient and appropriate expertise can be assigned

Risk

Lack of resources and

capacity limits effective

and sustainable

implementation

Ap

pe

nd

ix 1

19



combining examination and testing applications Valid and reliable assessments of student learning implemented starting in 2010

Not achieved Partially achieved

2. Developing Effective Teachers 2.1 Formulation of a national teacher development framework 2.2 Increased number of agricultural science, food and textiles, visual arts and design, and technology secondary teachers 2.3 Completion of effective pre- and in-service training to support new initiatives

Framework adopted by December 2007 Development of an implementation plan for the framework started by March 2008 2-year NUS program operational by January 2008 Full NUS graduate fellowship program completed by December 2010 and all graduates placed in schools in January 2011 Trained cadre of in-service teacher trainers in new primary curriculum by June 2009 Primary pre-service teachers trained in new curriculum by June 2010

Achievement at December 2014 Achieved Achieved Achieved Achieved Achieved Partially achieved

TA and project progress reports and joint reviews

MESC statistics

Evaluation of fellowship program by March 210 and initial impact assessment by June 2011

School review officers’ reports

Training and in-service evaluations

Research and evaluation studies

Student learning outcome data available from 2010

Risks

MESC having little control over scheduling of activities placing additional demand on its staff in implementing Component 1 and other subcomponents.

Impact of the fellowship program dissipated by inclusion of a large number of subjects for a limited number of fellowships

Assumption

Component 1 proceeds as planned.

Risks

Adoption of new instructional practices is slower than anticipated

Adoption of new assessment techniques is slower than anticipated

20

Ap

pe

nd

ix 1


Report Mechanism Assumptions and Risks

2.4 Effective continuous pre- and in-service teacher development and school-based support

Trained cadre of in-service teacher trainers in new science multimedia materials by June 2009 Science teachers trained in new multimedia materials by December 2009 All in-service teachers are trained on new classroom assessment practices by June 2010 National assessment training plan implemented by December 2010 Training for and by secondary and primary principals association in operation All teachers participate routinely in pre- and in-service training programs All pre- and in-service training courses staffed with trained instructors and delivered with 90% satisfactory rating SOD staff trained in techniques and systems to assess school performance and improvement by June 2009 SOD staff routinely use systems to assess school performance and recommend school improvement strategies from 2009

Achieved Achieved Achieved Achieved Partially achieved Partially achieved Partially achieved Partially achieved Achieved


Reports from association seminars

School improvement plans

Teacher performance plans and appraisal

School review officer reports

Training and in-service evaluations


SOD performance appraisal records

School improvement plans

School review officer reports

Assumptions

Impact assessment is undertaken to assess training effectiveness

SOD staff will have sufficient capacity and time to complete training/mentoring.

Ap

pe

nd

ix 1

21



3. Improving access to quality education 3.1 Improved secondary school facilities 3.2 Improved maintenance 3.3 Procurement of goods and services: Tendering and contract management procedures and systems established and operational 3.4 Teachers’ houses at 4

Improved learning environments in 3 secondary colleges Improved learning environments in 9 secondary schools 13 secondary schools renovated, furnished, and equipped Maintenance policies developed and operational plans in place by 2010 Funding for operational maintenance provided in each year of the project System reviewed, developed, and in use Efficient and effective procurement systems developed and in use by 2009 Tendering procedures meet Govt. of Samoa and ADB guidelines and deliver goods and services effectively and on time Each school is provided with furniture, equipment and learning material packages, on time and within budget following completion of buildings

Achieved (with some low quality) Achieved (with some low quality) Achieved (with some low quality) Plans in place by conclusion but not distributed to schools Partially achieved

e

Partially achieved Achieved Partially achieved Partially achieved Partially achieved

f


Construction reports and project database

Construction reports and contractor assessments/building inspections

Project database

Procurement reports

Project reports, school assessments

Evaluations

Construction reports

Building Contracts

Evaluation report under Component 4 about pilot program for teacher housing

Over-commitment of construction industries for other infrastructure may impact the quality of and slow down the school construction program

Curriculum sets out requirement for learning materials

Standard school design criteria set furniture and equipment needs

Highly qualified TA explicitly focused on capacity building will be available

Assumptions

Teacher housing in remote locations is a cost-effective means of attracting staff

Sites are selected by MESC and agreed with school communities

22

Ap

pe

nd

ix 1



primary schools in rural areas 3.5 MESC headquarters constructed

Most students have access to improved learning materials and equipment More qualified and experienced urban-based primary school teachers are attracted to rural areas 4 houses are designed and constructed by 2012. Better management environment and greater productivity and safety for MESC staff Staff absenteeism reduced Increased use of information technology Reduced operational costs

No data Achieved Partially achieved No data Achieved (but no definition of ‘increased’ No data


MESC statistics

Risk

Present construction boom and increase of costs of commodities worldwide will cause budget to be exceeded

4. Strengthening capacity to undertake research, evaluation, policy analysis, and planning 4.1 Improved national capacity for research and evaluation

Capacity building needs of MESC and other institutions staff assessed, associated training and technical assistance program delivered, and results determined (from 2008/09 through 2010/2011)

Achieved

Needs analysis completed

Capacity building assessments undertaken and results obtained

Assumption

Highly qualified technical assistance focused on capacity building will be available

Ap

pe

nd

ix 1

23



4.2 Research and evaluation program with results used for policy development and planning 4.3 Improved capacity to evaluate the impact of sectoral initiatives and policy analysis and planning

Design and implementation of 5 research studies based on MESC priorities implemented from 2008/09 through 2010/2011 with results integrated into policies and planning Evaluations of pilots supported by the project (by 2012) and evaluation of the impact of ESPI on agricultural science and design and technology PPRD staff effectively lead policy development, planning, implementation, and adjustment of MESC policies and SPP with very limited external assistance

Partially achieved (no data on research findings available) No data Partially achieved

Research and evaluation program design prepared

Research and evaluation studies and results published and disseminated

Evaluation reports prepared

Project progress reports and findings of joint review missions

Review of capacity of building and research/evaluation program in Year 2

Risks

Number of qualified MESC and other staff are too limited to ensure sustained capacity

Cooperative arrangements with other institutions are feasible and complementary resources are available

5. Strengthening capacity to implement and manage development projects 5.1 Improved strategic management and coordination of implementation 5.2 Integrated financial management system 5.3 Effective performance

Progress toward the project’s development objectives is on schedule and within budget Program management structure operational and effective, including secretariat and committees Financial management system established, special purpose account operational by Year 1 and integrated with government finance and reporting system by Year 2 Project progress reports

Partially achieved Achieved Achieved Partially achieved

Joint review missions, Govt. of Samoa assessments of MESC capability, sectoral plans and reviews, Govt. of Samoa budget

Sector plan midterm review (2010)

Financial reports, audit reports

Annual plans and performance progress reports

Assumptions

Progress toward sector approach is compatible with donor strategies

Donors are able to agree on financing and management arrangements

Highly qualified TA focused on capacity building will be available

Risks

Multiple donors increase administration and transaction costs for MESC/MOF

Current limitations of

24

Ap

pe

nd

ix 1



monitoring, evaluation, and reporting system

produced by the ESPII secretariat on time and providing relevant information to project management committee and development partners

Financial reports, contractor assessment reports, annual plans, analysis of lead and processing times

Govt. of Samoa financial system (GOFAR)

ADB = Asian Development Bank; CLC = Community Learning Centre; CMAD = Curriculum, Materials, and Assessment Division; ESPI = Education Sector Project I; ESPII = Education Sector Project II; FBEAP = Forum Basic Education Action Plan; MDG = Millennium Development Goal; MESC = Ministry of Education, Sport, and Culture; MOF = Ministry of Finance; NUS = National University of Samoa; PPRD = Policy Planning and Research Division; SOD = Schools Operations Division; SPC = Secretariat of the Pacific Community; SPELL = Samoa Primary Education Literacy Levels; SPP = Strategic Policies and Plans; TA = technical assistance.

a

Literacy rates are self-reported and unreliable. b

Includes the Institute of Higher Education, Institute of Technology, and The University of the South Pacific. c No definition of adequate. Baseline data are too weak to determine trends.

d Curriculum documents do not set out a ‘set of learning materials.’

e Achieved through the school fee grant project rather than ESPII.

f There are no data to support this. Rating reflects distribution of materials and equipment.

Appendix 2 25

CONSULTING SERVICES SUPPLIED THROUGH PINZ

(January 2009–April 2013)

Designation/Position

International (I) /

National (N)

Person-

Months

Component 1: Curriculum Reform and Assessment

Primary Curriculum and Materials Development Specialist I 21.00

Language and Bilingual Education Specialist I 7.00

Multimedia Specialist I 12.00

Assessment and Examination Framework Specialist/Team

Leader

I 18.32

Information System Specialist I 10.27

School Assessment Specialist I 4.87

Curriculum and Materials Writers – Samoan Language, English

Language, Mathematics, Science, Social Science, PE and Health,

Visual Art, Performing Arts, Music

N 83.58

Subject Specialist/Reviewer – Samoan Language, English

Language, Mathematics, Science, Social Science, PE and Health,

Visual Art, Performing Arts, Music

N 11.12

Multimedia Science Counterpart N 6.00

Technology Expert N 6.00

National Assessment Policy Framework Counterpart N 4.05

School Assessment Counterpart N 6.55

Community Education Development Coordinator (Phase 1) N 4.93

Program Developer: Home-school Literacy Partnership (Phase 1) N 5.77

Program Developer: Community Learning Center (Phase 1) N 6.00

International Curriculum Reviewers – English, Expressive Arts,

Health, Mathematics, Science, Social Science

I 1.18

Multi-Media Unit Support I 0.30

Subtotal 208.94

Component 2: Developing Effective Teachers

Teacher Development Policy Analyst I 10.00

School Improvement Advisor I 10.00

Primary Teacher Educator N 16.00

In-service and Pre-service Teacher Development: Expert in

Agricultural Science

N 6.00

In-service and Pre-service Teacher Development: Expert in N 6.00

Food and Textile


Visual Arts

N 6.00


Design and Technology

N 6.00

Subtotal 60.00

Component 4: Capacity to Undertake Research, Evaluation, Policy Analysis, and Planning

Research and Evaluation Specialist I 8.00

Information Management and Survey Planning Specialist I 4.00

Database Development and Data Analysis Expert N 8.48

Subtotal 20.48

GRAND TOTAL 289.42

26 Appendix 3

STATUS OF COMPLIANCE WITH LOAN CONVENANTS

Covenant

Reference in Loan

Agreement Status of Compliance

In the carrying out of the Project and operation of the Project facilities, the Borrower shall perform, or cause to be performed, all obligations set forth in Schedule 6 to this Loan Agreement.

LA Article IV, Section 4.01.

Partially complied with. Status of compliance with Schedule 6 detailed also below.

The Borrower shall (i) maintain, or cause to be maintained, separate accounts for the Project

LA Section 4.02(a)

Complied with.

The Borrower shall (ii) have such accounts and related financial statements audited annually, in accordance with appropriate auditing standards consistently applied, by independent auditors whose qualifications, experience and terms of reference are acceptable to ADB;

LA Section 4.02(a)

Complied with.

The Borrower shall (iii) furnish to ADB, as soon as available but in any event not later than 6 months after the end of each related fiscal year, certified copies of such audited accounts and financial statements and the report of the auditors relating thereto (including the auditors' opinion on the use of the Loan proceeds and compliance with the financial covenants of this Loan Agreement as well as on the use of the procedures for imprest account and statement of expenditures, all in the English language;

LA Section 4.02(a)

Complied with. Auditor’s Report for the period 1 July 2009 to 30 June 2010 was sent to ADB on 25 February 2011. Auditor’s Report for the period 1 July 2010 to 30 June 2011 was received by ADB on 3 May 2012. Auditor’s Report for the period 1 July 2011 to 30 June 2012 was received by ADB on 30 January 2013. Auditor’s Report for the ESPII for the period 1 July 2012 to 30 June 2013 was submitted to MOF on 28 November 2013 and received by ADB on 24 April 2014. Auditor’s Report for the ESPII for the period 1 July 2013 to 30 June 2014 was received by ADB on 15 June 2015. Auditor’s Report for the ESPII for the period 1 July 2014 to 30 June 2015 was received by ADB on 26 April 2016.

The Borrower shall (iv) Furnish to ADB any such other information concerning such accounts and financial statements and the audit thereof as ADB shall from time to time reasonably request

LA Section 4.02(a)

Complied with

The Borrower shall enable ADB’s representatives to inspect the Project, the goods financed out of the proceeds of the Loan, and any relevant records and documents.

LA Article IV, Section 4.03.

Complied with.

Project Executing Agency MOF will be the Project Executing Agency.

LA, Schedule 6, para 1

Complied with

Implementing Agency MESC will be the Implementing Agency. MESC will have the overall responsibility for the day-to-day management of project implementation, monitoring and evaluation and reporting of progress. The Project will be managed, through the existing MESC organizational structure, by the MESC Core Executive, comprised of the CEO,


Complied with. The ESPII Secretariat plays a coordinating role.

Appendix 3 27

Covenant

Reference in Loan


ACEOs and supported by OCEO. The responsibility for each component will rest with the appropriate ACEO and implemented by the staff of the relevant division. The Curriculum, Material and Assessment Division (CMAD) will have responsibility for the implementation of Component 1 while the School Operation Division (SOD) will have the responsibility for the implementation of Component 2 with a sub-committee consisting of ACEOs for SOD and CMAD, an OCEO representative, and Principal Education Officers. PMT (to be eventually part of MESC’s organizational structure) will be responsible for the procurement of all equipment and of the implementation of Component 3, together with an infrastructure subcommittee of MESC composed of an OCEO representative, the Head PMT, all ACEOs, representatives from Ministry of Works, and MOF. The Planning, Policy and Research Division (PPRD) will implement Component 4. CEO/MESC Core Executive will be responsible for implementing Component 5. Each division will prepare the reviews and program of activities and liaise on project implementation matters with ADB and the Cofinanciers.

Complied with.

Education Steering Committee (ESC) has changed to the Education Advisory Committee (EAC) EAC is composed of a Chair from the Public Service Commission and representatives from MESC, MOF, Ministry of Foreign Affairs, National University of Samoa, MWTI, MCIL, MWCSD, SQA, PSC, the Chairman for the National Council of Churches, the Coordinating Development Partners- AUSAID, NGO, Non Government Education Sector representatives (2) and the ESPII Secretariat. The CEO is, ex officio, a member of the EAC, while the OCEO provides support and secretariat services. EAC will (i) approve Project progress reports and endorse the proposed operational plans, and support the development of Project budgets; (ii) provide overall policy and strategic guidance in implementing the Project and decide on corrective action, if needed; (iii) ensure that Project implementation is in accordance with the Government educational sector policies and plans, and (iv) resolve issues and conflicts.


Complied with. The Education Advisory Committee convenes on a quarterly basis. With the design of the Budget Support Program for the Ministry, the EAC’s Terms of Reference was reviewed to reflect a more strategic sectoral role and the name changed to the Education Sector Advisory Committee The ESPII Secretariat provides the support and secretariat services. However, with the change of the EAC to the ESAC, the Sector Coordination Division is now the Secretariat for this committee. The ESPII/SchoolNET reports are included as part of the CEO Report to this committee Complied with. ESPII Quarterly Program Progress Reports are submitted to ESAC meeting for endorsement before to distribution to MOF and development partners. The ESPII Milestones Reports to ESAC are provided on a quarterly basis.

Procurement methods and contracts packaging The Borrower shall (i) commission an external review of the efficiency of the procurement methods and contracts packaging for equipments and materials procured under ESP I, (ii) discuss the findings of the review with ADB and the


Complied with. (i) External review of efficiency of procurement

carried out (ii) Report on findings of review was sent to DPs in

March 2007 (iii) Review of financial data conducted on a

28 Appendix 3

Covenant

Reference in Loan


Cofinanciers during the Project’s inception, and (iii) agree with ADB and the Cofinanciers on the remedial measures to be taken based on such review prior to engaging in procurement under this Project.

quarterly basis. (iv) A independent review by the design team for

Education Sector Budget Support Program completed in 2012

Education Policy and Strategic Plan Within 12 months after the Effective Date, the Borrower shall (i) develop an evidence based Education, Sports and Culture Policy and Strategic Plan (2006-2014) for the education sector, including impact studies on poverty alleviation and targeted support to the economically disadvantaged, (ii) provide a copy of such strategic plan to ADB, the Cofinanciers, and other relevant development partners for review and comments prior to tabling such policy and plan for cabinet approval, (iii) consider comments from ADB and the Cofinanciers prior to tabling such policy and plan for cabinet approval, and (iv) table it for cabinet approval.


Complied with. MESC Strategic Policy and Plan (July 2006–June 2015) was approved by the Cabinet and signed by the Minister Copies distributed to Development Partners and the Executing Agency (MOF). The Samoa Education Sector Plan (ESP) launched on the 20th January 2014. The SESP supersedes the MESC Strategic Policy and Plan.

Teacher Supply and Demand Within 3 years after the Effective Date, the Borrower shall (i) develop a policy and strategy on teacher supply and demand and conditions of their service based on, inter alia, (a) the findings and recommendations of the supply and demand study supported under ESP I, (b) the analyses and recommendations that will emerge from the TA, (c) the findings of the pilot projects as described in Component 3(ii) in Schedule 1 to this Agreement, (ii) provide such policy and strategy to ADB and the Cofinanciers for review and comments, (iii) consider comments from ADB and the Cofinanciers prior to tabling such policy and strategy for approval and (iv) table it for cabinet approval, and (v) implement it in accordance with its terms.


Complied with. The Professional Teacher Standards and Minimum Service Standards as part of the National Teacher Development Framework have been developed under Component 2 of ESPII. Established new division, Monitoring, Evaluation, and Review Division, in 2013 will take forward the recommendations of the NTDF for teacher registration and appraisal system. Established the Teacher Development Services Division will be responsible for professional development and training of teachers.

Learning Outcomes Within 3 years after the Effective Date, the Borrower shall ensure that a system to assess student learning outcomes and the impact of the Project investments is in place. This system shall develop a sample based standardized system for the assessment of student learning that will be used as a baseline for reporting of progress towards key performance indicators.


Partially complied with. Launched: the Samoa National Assessment Policy Framework provides the roadmap of activities to be implemented. The implementation of the new replacement National Year 8 examination is to complement the shift of student teaching and learning to outcomes based and the implementation of new assessment tools.

Maintenance Plan The Borrower shall cause the submission of a satisfactory maintenance plan by the respective communities as a condition precedent to their schools being upgraded under the Project.


Complied with. The School Maintenance Plan is an annex to the School Annual Plan prepared by the School Principal. A draft School Facilities Handbook for schools was upgraded under the program, fully developed in May 2006 and reviewed before December 2014 when all (except one school) schools were fully completed. Prior to a school upgraded under the Project,

Appendix 3 29

Covenant

Reference in Loan


meetings were held with the School Principal and School Committee on the Concept Design and maintenance strategies The Samoa School Fee Grant Scheme is the mechanism now used to monitor the implementation of the schools’ maintenance plans.

Counterpart Funding * The Borrower shall take, and shall cause MOF and MESC to take, all necessary measures, including the provision of additional funds, to ensure that sufficient counterpart funds are made available and provided for the Project in a timely manner. * The Executing Agency (MOF) used the term Contribution Fund to denote GoS share of ESPII costs. See also footnote 1 on page 1.


Complied with. GoS approved counterpart funds for ESPII and SchoolNet Project respectively as follows; FY2012/2013: ST$627,531.81 for ESPII and ST$50,000.00 for the SchoolNet Project. FY2013/2014: – ST$627,532.00 for ESPII and ST$50,000 for SchoolNET FY2014/2015: – $416,569.00 for ESPII and ST$50,000 for SchoolNET.

Rationalization of School By 31 December 2013, the Government shall develop and implement a plan for the rationalization of schools to ensure resources are used cost effectively.


Complied with. Preliminary discussion within the Ministry has commenced on the design of a resourcing plan, consolidation of all funding by government and the rationalization of Primary Schools. A proposal to the Ministry of Finance of the consolidation of all funding. Developed in May 2013 was a draft School Resourcing Plan as part of the requirements for the SSFGS Secondary Design MESC proposal for the consolidation of all funding by government for schools submitted to Ministry of Finance in January 2014 for review.

Employment The Borrower shall cause the contractors to maximize the employment of local persons, in particular those who suffer hardship who meet the job and efficiency requirements for the construction, rehabilitation or maintenance of the Project facilities and shall cause the contractors to give due consideration to the contribution of local women for the construction, rehabilitation and maintenance of the Project facilities.

LA, Schedule6, para 11

Complied with. All tender documents for civil works under the program incorporate this provision.

Resettlement The Borrower shall ensure that the Project will not entail any involuntary resettlement.

LA, Schedule6, para 12

Complied with. No resettlement of persons has occurred for the Program.

ICT in Secondary Education Within 3 years after the Effective Date, the Borrower shall (a) develop comprehensive action plans satisfactory to ADB to mainstream information and communication technology education in secondary schools and (b) take appropriate measures based on the lessons learnt from the ADB-supported School Net pilot project component financed by ADB under the TA 4305-SAM to expand outreach to rural areas.


Partially complied with. This component had been addressed under the SchoolNet Project approved in December 2007. The SchoolNet has since been implemented very closely with the ESPII, and at some point, considered as ESPII Component 6, especially for reporting purposes.

Plan for complete set of textbooks Within 3 years after the Effective Date, the Borrower shall (i) develop and implement a plan


Complied with. Note: Definition of textbooks is curriculum

30 Appendix 3

Covenant

Reference in Loan


for every child to have a complete set of textbooks that will address issues of finance, procurement, storage, distribution, and school based policies aimed at extending the life of textbooks and (ii) allocate in a timely manner adequate funding in the education budget for financing the development and implementation of this plan.

statements and teachers manuals. Complied with. In 2013 new primary curriculum statements in 7 subjects and teachers manuals provided to all students and teachers. The management of the textbooks is being strengthened under the School Operations Manual, through supporting visits and SSFGS reports.

Anticorruption Measures During Project implementation, the Borrower shall follow, through MOF, ADB’s Anticorruption Policy, it being understood that ADB reserves the right to investigate, directly or through its agents, any alleged corrupt, fraudulent, collusive or coercive practices relating to the Project. The Borrower shall ensure, through MOF, that (a) periodic inspections of the contractors’ activities related to fund withdrawals and settlements are carried out, (b) relevant provisions of ADB’s Anticorruption Policy are included in all bidding documents for the Project, and (c) all contracts financed by ADB in connection with the Project include provisions specifying the right of ADB to audit and examine the records and accounts of MOF and all contractors, suppliers, consultants and other service providers as they relate to the Project.


Complied with. a) Anticorruption Policy inserted in all bidding

documents b) Right of ADB to audit and examine records and

accounts of MOF, Program contractors, suppliers and service providers included in all contract documents

Review The Borrower shall undertake with ADB and the Cofinanciers two reviews in the first year of Project implementation, and a review each subsequent year. Such reviews shall focus on budgetary allocations, maintenance, staffing and other incremental recurrent costs, implementation arrangement, and achievement under the Project. In addition, the Borrower shall undertake with ADB and the Cofinanciers a midterm review at the end of the third year of Project Implementation that will cover: (i) the scope, design and project implementation arrangements; (ii) evaluate pilot programs; (iii) assess performance against targets and benchmarks; (iv) review lessons and experiences with implementation of each component; (v) review compliance with the loan agreement and bilateral funding agreement(s); and (vi) recommend any changes in project implementation, if required.


Complied with.. Complete list of missions: Inception Mission (9–23 Jun 2006) Joint Review Mission 1 (4–6 Dec 2006) Joint Review Mission 2 (28 May–1 Jun 2007) Special Consultation Mission (27–29 Oct 2007) Joint Review Mission 3 (11–15 Feb 2008) Joint Review Mission 4 (11–15 Aug 2008) Joint Review Mission 5 (23–27 Feb 2009) Midterm Project Review (7–22 Oct 2009) Loan Review 1 (23 Feb–3 Mar 2010) Loan Review 2 (26 Jul–30 Jul 2010) Joint Review Mission 6 ( 29 Nov–3 Dec 2010) Loan Review 3 (6–10 Jun 2011) Joint Review Mission 7 (19–23 Sep 2011) Joint Review Mission 8 (11–15 Mar 2013) Loan Review 4 (17–21 Feb 2014) Joint Review Mission 9 (26–30 May 2014) Project Completion Mission (25–29 May 2015

Appendix 4 31

DFAT EVALUATION CRITERIA: MONITORING AND EVALUATION, GENDER, AND RISK

1. In addition to the Development Assistance Committee’s criteria of relevance, effectiveness, efficiency, sustainability and impact, the Australian Department of Foreign Affairs and Trade (DFAT) requires specific consideration of monitoring and evaluation (M&E), gender equality, and risk. A. Monitoring and Evaluation: Was an M&E system used to generate credible

information to measure progress toward meeting outcomes and for program improvement, learning, and accountability?

2. Neither the Ministry of Education, Sport, and Culture (MESC) nor the secretariat had the capacity to develop an appropriate system to measure progress toward or evaluate outcomes. Several systems operated in parallel for part or the whole duration: the Asian Development Bank (ADB)–format design and monitoring framework (DMF), which was the joint responsibility of all development partners; the first monitoring and evaluation framework (MEF), which was supposed to guide implementation at a more detailed level than the DMF; and, following the midterm review in 2010, a more sophisticated sector-wide framework for future budget support. The intention that these would be complementary was lost, especially as development partner attention switched to what would follow the Education Sector Project II (ESPII) rather than what was currently happening. 3. The design of ESPII was one in which all partners were supposed to assess progress through joint review missions. These missions were generally helpful although changes in membership among the development partners occasionally led to recommendations that could appear inconsistent. Each joint review mission requested monitoring information against outcomes. But the secretariat found the different frameworks confusing and there was continuous tension between the requirements for project M&E and MESC’s own requirements. This tension was never resolved, mainly because there was limited capacity to understand M&E as a concept and, because implementation was fast-paced, it was never a priority. The absence of quality M&E has meant that a great deal of the work that has been done has not been fully reported on. At completion it was difficult to prepare the ADB-required and Australian Department of Foreign Affairs and Trade (DFA)–agreed DMF because although it was a development partner responsibility, it had fallen into disuse and the data collected in other formats could not be read across different indicators. 4. One of the challenges of monitoring, and therefore of reporting, was an institutional culture in MESC that did not support sharing of information. The Education Sector Advisory Committee (ESAC) recommended that sector agencies consider signing memorandums of understanding (MOUs) to allow the sharing of information for M&E purposes, although this recommendation was never implemented. The MEF established in 2011 was at least able to gather information from mid-level personnel in the divisions through time-consuming requests and follow-up. The later system required information to be collected at the senior-management level and shared in coordination meetings. This was unsuccessful because senior managers were already overloaded with other activities and did not find the information sufficiently useful to make it a priority. The result was that the quantity and quality of data fell as the project gathered momentum. 5. Component 4 of the project was directly related to evaluation and a number of analytical studies were done. Unfortunately, owing to the conclusion of technical assistance (TA) in 2013

32 Appendix 4

and the absence of a document management system, these studies were either lost or could only be accessed through individuals who had some of them in their own records. 6. This experience provides lessons for the future. The loss of the evaluative work done under ESPII and the inability to provide quality data for the project completion report highlights the importance of institutionalizing a strong system for monitoring, evaluation, and learning. This will become critically important in the future when the Government of Samoa will need to demonstrate results in order to maximize the opportunity for increased donor funding. B. Gender Equality: Has the investment made a difference to gender equality and

empowering women and girls? 7. Gender equality was not mainstreamed across the design. In the design’s underpinning analysis it was reflected as an element of equity and in the log frame it was reflected as the millennium development goal (MDG) indicator at the outcome level (gender equality in participation for all levels of education). At the policy level, the Government of Samoa commits to providing high-quality education to all Samoan students irrespective of race, ethnicity, disability or socioeconomic background. Statistical collection of education data routinely disaggregates by sex. 8. During implementation the curriculum design and review process sought to ensure that gender stereotyping was avoided in curriculum support materials. However, as there was no required monitoring of this, and as rollout of the curriculum is ongoing, there is no evaluation of the impact on boys and girls. 9. The design noted that the main gender issues relate to the underperformance of boys relative to girls. Not only do girls outperform boys, but they attend school more regularly and pay more attention in class. There was frequent mention of ‘naughty’ boys and the difficulty of ensuring attendance and managing their behavior in class. However, due to the project’s complexity, addressing gender as an issue specifically related to the performance of boys has not been a priority. There has been no further assessment of related gender issues such as the feminization of the teacher workforce. C. Risk Management and Safeguards: How was Risk Managed?

10. In design, the highest-level risks—that cofinancing arrangements and use of ADB systems would be ineffective to deliver a quality project and that MESC would not be able to manage it—were partly mitigated. The increase in construction costs and unsatisfactory maintenance of infrastructure and equipment were not properly mitigated and the risk that communities would be unwilling and unable to mobilize resources to undertake the maintenance of schools was realized in some schools. Lower-level risks that component activities would not be sequenced properly and that procurement of textbooks and learning materials would be delayed were realized. However, as the project was satisfactory overall, these risks were not deal breakers. 11. The midterm review in 2010 highlighted several risks, of which three are notable: (i) linkages between project activities and linkages between the project and MESC activities were not being clearly identified and would lessen the project’s impact. Action taken from 2011 mitigated against this with reasonable effect; (ii) analysis of data and information critical to oversight and decision making was not robust enough to identify trends and emerging risks to planned activities and achievement of objectives and goals. This was never adequately

Appendix 4 33

addressed (see M&E); (iii) quality control on non-construction consultancy contracts was lacking, which could lead to the project failing to meet its objectives. This was partially addressed. 12. With the exception of 1 year, joint review missions did not look specifically at risk. The secretariat, which should have been responsible for maintaining information to manage risk, such as a risk register or management plan, was never able to rise to that challenge, due primarily to overload but also to capacity limitations.

34 Appendix 5

LIST OF OUTPUTS

1. This list was compiled using information from the secretariat version of the project completion report. A. Component 1: Introducing Curriculum Reform and Assessment Systems

Sub-Output Summary of Major Achievements

1.1. A new bilingual primary curriculum for Samoan language, English language, mathematics, science, social studies, the arts, and physical education and health

Curriculum statements for seven subject areas developed, piloted, used to train primary school teachers, and finalized

Teachers’ manuals for 9 subjects prepared and completed. Separate teachers’ manuals for the arts – visual arts, performing arts, and music.

Translations (from English to Samoan) of some parts of the new primary education curriculum materials

1.2. An adequate supply of learning materials and teacher manuals

Instructional materials for secondary science subjects, consisting of audiovisual materials, digital materials, and print materials

Handbook on Use of Multimedia Materials

Draft of the Bilingual Policy Handbook prepared and presented to the core executive

Strategy for the development of supplementary graded readers in Samoan and English

Teacher resources to support implementation of the curriculum, and manuals related to classroom assessment

Development and refurbishment of the multimedia unit

1.3. Community partnership program for demand generation and improved learning outcomes

A Fagaloa school committee and principal of the Community Learning Centre (FCLC) were appointed

Vision and mission of FCLC and partnership strategy formulated

Further education courses provided and first batch graduated

Certificate in Adult Teaching (CAT) 101 and 102 for potential tutors successfully completed and conducted by the National University of Samoa (NUS)

The Samoa Institute of Directors conducted good governance training for FCLC committee members

Training courses on safety and hygiene at FCLC and NUS Institute of Technology were completed

1.4 A national assessment policy framework and associated technological systems

Samoa National School Assessment Policy Framework (SNSAPF) finalised

Training in policies, practices, and tools to support assessment practices in the classroom provided

Training for core trainers and Ministry of Education, Sport, and Culture (MESC) staff in classroom-based assessment practices for ongoing support to teachers

Student education number (SEN) established

Item bank system established

Business requirements for the Assessment Management Information System (AMIS) developed

Training for the Curriculum, Materials, and Assessment Division (CMAD), Assessment and Examination Unit (AEU), and Policy Planning and Research Division (PPRD) in information systems provided

B. Component 2: Developing Effective Teachers


2.1 The formulation of a National Teacher Development Framework

National Teacher Development Framework officially launched

Professional standards for graduate, registered, mentor, and master teachers developed

Mentor teacher program piloted

Quality-assured performance appraisal system developed

2.2 Increasing the number of primary

38 Fellowship participants graduated in four areas of severe secondary teacher shortage

Appendix 5 35


and secondary teachers in specialized subjects

38 primary teachers graduated in the first cohort

2.3 Completion of pre-service and in-service training to support new initiatives

Core trainers recruited and trained to support teachers in implementing the new curriculum and associated activities

1,150 teachers undertook training for the new curriculum

2.4 Effective in-service teacher development and school-based support

Training provided for Schools Operations Division (SOD) staff, School Review Officers and principals to be effective academic leaders in schools to improve teacher quality in the classroom

Resources for teachers, principals, SROs, core trainers and MESC provided to facilitate ongoing teacher support

C. Component 3: Improving Access to Quality Education


3.1 Improved secondary school facilities

Nine secondary schools completed

3.2 Improved maintenance

School maintenance and operations handbook completed and printed but not distributed. Consultations held with school principals and school committees.

3.3 Procurement of goods and services:

Project implementation and administration manual produced

3.4 Teachers’ houses at four primary schools in rural areas

Four teacher houses completed

3.5 MESC headquarters constructed

New MESC headquarters constructed

D. Component 4: Strengthening Capacity to undertake Research, Evaluation, Policy

Analysis and Planning


4.1 Improved national capacity for research and evaluation

Training needs analysis completed and capacity building plan prepared, linked to MESC research program

Workshops and on-the-job training provided for PPRD staff to expose them to the latest techniques in education research and analytical methodologies

Eight PPRD staff went on workplace attachment at the Faculty of Education at the University of Auckland

PPRD staff presented twice at the Australia & New Zealand Comparative & International Education Society conference

PPRD staff completed ten research and evaluation reports

4.2 Research and evaluation program with results used for policy development and planning

Research program developed and linked to MESC policy priorities

Research papers prepared

Action-research study undertaken for the Fagaloa Community Learning Centre pilot program

4.3 Improved capacity to evaluate the impact of sectoral initiatives

Development of an Education Sector Project II (ESPII) evaluation framework to assess program interventions

Preparation of five individual impact evaluation reports to examine the short-term impacts of ESPII interventions

Evaluation report prepared on the effectiveness of ESPII as a Sector Wide Approach.

36 Appendix 6

DETAILED RECOMMENDATIONS

1. Making sure that ESCD can perform its role effectively. The Education Sector Coordination Division (ESCD) is critical to the effectiveness of the sector and to the long-term partnership with the development partners. As a new division, it will need support to ensure that capacity is not overwhelmed. This might include in-depth work in key areas where lessons have been learned from the Education Sector Project II (ESPII):

Role identification—to ensure clarity from the outset that its strategic coordination role across the sector does not become confused with the more familiar role of program management /coordination/implementation unit within a particular ministry.

Monitoring and evaluation (M&E)—to ensure that it can report on outcomes and impact the sector plan. This is important to maximize the budget support that is performance-linked and ensure that Samoa can provide the kind of performance data that will be required for development partner financial support into the future.

Risk—to maintain an up-to-date assessment of risks and draw attention to areas that compromise achievement across the sector.

2. Active, resource-intensive consolidation of key areas

• Multi-tiered teacher training and professional development, including school-level activities; targeted in-service training; teacher qualification upgrading; Monitoring, Evaluation, and Review Division performance reviews; and high-quality pre-service training that complements the implementation of the new curriculum.

• Additional resources in support of the new curriculum, including operationalization of the Bilingual Education Policy, teacher guides and curriculum support materials in Samoan and English, additional resources related to an outcomes-based approach to education, and learning materials targeted to student learning needs.

• Community engagement and school leadership, to more fully put in place a school-community partnership in support of student learning. The Minimum Service Standards, if well understood by involved stakeholders, could be a constructive tool in a child-centered approach to education.

3. Shifting the focus of energy and effort to the student level

• Early diagnostics of student learning at the school level. A lesson from the program is that first assessing student literacy and numeracy performance at Year 4 is too late: preschool/early grade assessment—directly linked to learning materials and teacher practice—could provide the evidence to better tailor early learning, which could in turn be analyzed through Samoa Primary Education Literacy Levels (SPELL) and Samoa Primary Education Completion Assessment (SPECA) results.

• An emphasis on ongoing assessment. Samoa’s education system has embraced the formal assessment of student performance at key stages (e.g., SPELL and SPECA). A stronger move toward ongoing classroom-based assessment—linked to teacher professional development—could enable real-time adjustments to classroom practice, as encouraged by an outcomes-based curriculum. Key questions would include: what literacy and numeracy competencies are students struggling to attain? What can be done to meet their learning needs?

4. Keeping curriculum review and development as an ongoing process

Appendix 6 37

• Samoa should now be collecting feedback from teachers and students about the impact of the curriculum changes, and what barriers exist to implementation, so that there is a wealth of information with which to plan.

5. Improving operations and maintenance

• A review of the maintenance needs of schools would be beneficial in order to assess the limits of what it is reasonable to expect school committees to take on.

• Future infrastructure design and building needs to assume that there will be

minimal expenditure on maintenance; materials and standards need to take

account of this and be of more durable quality.

samoa: education sector project ii - asian development bank...samoa: education sector project ii ......

Documents