how to make m&e more results oriented · raisa venäläinen senior evaluation officer hdned...

How to Make M&E More Results Oriented

A Review of the Monitoring Arrangements of Thirty Education Sector Projects and

Some Practical Suggestions for Improvement

OCTOBER 2007

Raisa Venäläinen Senior Evaluation Officer

HDNED World Bank

47185

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Table of Contents

Acronyms ....................................................................................................................................... ii

Executive Summary ......................................................................................................................iv

Introduction ....................................................................................................................................1

1. Monitoring Systems....................................................................................................................2

2. Monitoring Arrangements.........................................................................................................8

3. Strengthening M&E Capacity.................................................................................................13

4. Implementation Support..........................................................................................................14

Conclusions and Recommendations ...........................................................................................15

Annex 1: Primary and Secondary Education Operations included in the Review.................17

Annex 2: Framework used for Assessing M&E Systems and Arrangements .........................18

Annex 3: Indicator Inventory for Reviewed Projects ...............................................................20

Annex 4: Planning for Intermediate Monitoring Systems........................................................31

Annex 5: Establishing Indicators and Baseline Data for Equity Indicators ...........................32

Annex 6: Questions for Preparing a Learning Achievement Survey.......................................33

i

Acronyms

CAS Country Assistance Strategy GER gross enrollment rate EFA Education for All project EMIS education information management system ISR Implementation Status Report NER net enrollment rate AM Aide Memoire M&E monitoring and evaluation MDG Millennium Development Goal OECD Organisation for Economic Cooperation and Development PAD Project Appraisal Document PISA International Student Assessment Program, OECD PDO Project Development Objective TIMSS Trends in International Mathematics and Science TTL Task Team Leader

ii

Box 1. Key definitions and characteristics of effective monitoring

Dissemination and utilization

Describes why and how monitoring information is collected, who will use it, and for what purpose.

Intermediate indicators Measure progress in achieving intermediate results against measurable baseline and target values.

Intermediate monitoring systems

Assess whether a project is proceeding towards intended outcomes. Intermediate monitoring systems include intermediate results and/or outcomes and indicators.

Intermediate results

The achievements and changes required during project implementation to achieve PDOs. For example, “percentage of teachers using interactive instructional methods” (outcome) or “percentage of teachers receiving professional development” (output).

Managing for results Uses monitoring information to improve decision making and steer country-led development processes toward clearly defined goals.

Monitoring arrangements Arrangements for collecting and reporting M&E data, including the sources of information and by when and to whom it will be submitted.

Project Development Objective (PDO)

Describes the changed behavior or changed conditions of primary beneficiaries, for example “Better student learning outcomes.” PDOs are measurable and clearly demonstrate the objectives for which a project is accountable.

PDO indicators (outcome indicators)

Measure the achievement of Project Development Objectives against measurable baseline and target values.

Results Sustainable improvements in country outcomes. Results answer the question: If the intervention is successful, what will be the difference for the primary target group?

Results Framework Project design is guided by a results framework, which includes the Project Development Objective (PDO), intermediate outcomes, and related indicators, as well as intended uses of M&E information.

Outcome A short- or medium-term change in the conditions or behavior of a target group resulting from an output or a set of outputs. An outcome usually contributes directly to the achievement of a higher-level outcome, such as Project Development Objectives.

Outputs Works, goods, systems, services, organizations, or activities produced by a project or program that are directly attributable to project inputs.

iii

Executive Summary A review of the Monitoring and Evaluation (M&E) findings of 30 ongoing primary- and secondary-level education operations was undertaken to obtain an overall understanding of the M&E systems of World Bank-supported projects. The review also sought to identify areas where task teams, together with their clients, could make M&E more results oriented. This report presents the findings of the review, together with some recommendations for implementing this crucial task. Certain tools for M&E planning have also been prepared and included in this report as annexes. The findings of the review suggest that while efforts have been made to establish results-oriented monitoring systems and indicators for Project Development Objectives, less work has been done to: (i) plan for results-based intermediate monitoring systems and (ii) use the information generated by such systems. Without achieving these two goals, a results-oriented M&E system does not work. The review also found that implementation support should make greater use of Results Frameworks and ensure that monitoring information is made available to project stakeholders. The projects examined by the review used more sticks than carrots in trying to make M&E operational. Some projects tried to make disbursements conditional on the production of statistical information. Only a few projects aimed to increase the awareness of and demand for monitoring information by educating and involving stakeholders and community members in the design and implementation of M&E. Yet without their participation, it is difficult for stakeholders and communities to understand the meaning and purpose of M&E. If the purpose of a project is to improve the performance of these stakeholders, it is essential that they understand what this desired performance looks like. When selecting indicators, especially at the intermediate level, “usability of M&E information” should be one of the primary selection criterion. This criterion indicates whether or not there will be immediate demand for and use of monitoring information, either during project implementation or at project close. Assessing the usability of indicators helps determine whether they can be “upstreamed.” For instance, the indicator “percentage of school improvement plans meeting approval criteria” would be more useful for project or program management than “number of school improvement plans approved.” The introduction of new indicators, such as learning outcomes or disaggregated equity indicators, requires careful planning. It is necessary to ensure that the necessary capacity and data collection arrangements are in place to establish both baseline and target values for such indicators. Project teams should accordingly define by whom, how, and when the information generated by new indicators will be used. If projects aim to make changes in classroom practices, these changes need to be measured in the classroom. Classroom observations should therefore be used more to monitor and assess the quality of education. The review found that while the majority of education projects provided capacity development for teachers, only two planned to measure changes in pedagogical or instructional practices in the classroom. Although direct classroom observation may be more expensive and demand more resources than other monitoring arrangements, without understanding what happens in the classroom, it is difficult to develop appropriate, relevant policies in order to improve the quality of education.

iv

Review of Monitoring and Evaluation Arrangements of Thirty Education Sector Projects

and Some Practical Suggestions for Implementation Introduction This report presents the findings of a review of Monitoring and Evaluation (M&E) systems of 30 ongoing primary- and secondary-level education operations. The review was conducted as a personal orientation to World Bank-supported education sector projects and the findings were used to develop recommendations and tools for M&E. The overall purpose of the report is to point out areas where task teams, together with their clients, could make M&E more results oriented. A checklist and some tools were accordingly developed to assist them in this work and are included in this report as annexes. The use of M&E information is emphasized because data should not be collected if it will not be used. Although the review focused solely on “traditional” education projects, the principles of usable and efficient M&E also apply to Sector Wide Programs. It should be noted that the review offers an overall analysis of M&E, but does not assess the appropriateness of indicators chosen for specific projects, which would require much greater familiarization with the context and content of the individual projects. Scope of Review Thirty primary- and secondary-level projects were reviewed (see annex 1). The selection criteria were that each project had a formal Results Framework and targeted primary and/or secondary education. The projects represented 60 percent of all projects approved and managed by the Education Sector Board of the World Bank in FY05 and FY06.1 Project Appraisal Documents (PADs), Implementation Status Reports (ISR), and Aide Memoires (AM) available on the Bank intranet were the primary sources of data. The framework of the review and the questions it sought to answer are attached as annex 2. M&E designs were analyzed using Project Appraisal Documents (PADs). The review looked at the following questions:

(i) What types of indicators were selected?

(ii) What was the quality of these indicators?

(iii) How did the projects plan to use M&E information?

(iv) Did the projects plan for evaluations?

(v) What strategies have been chosen to build project M&E capacity, if any?

The Implementation Status Reports (ISRs) and Aide Memoires were reviewed to answer the following questions:

(i) Were monitoring systems (i.e., indicators and baselines) in place during the first year of implementation?

(ii) How did the first implementation support mission address M&E?

1 The reviewed projects included 17 out of the 26 projects approved by the Education Sector Board in FY05 and 13 out of the 26 projects approved in FY06.

1

For one-third of the 30 projects, the inventory of PDO indicators showed that some kind of “evolution” of indicators occurred in project documentation. First, there were inconsistencies in both the indicators and their total number between various sections of the PAD (e.g., in the Results Framework, “monitoring arrangements” section, and main text). In these cases, the indicators that appeared in the monitoring arrangements section of the PAD were selected for review because this section presents baseline and target values. Second, “planned” and “actual” monitoring systems differed, given that not all indicators named in the “monitoring arrangements” section of the PAD were included in the “status of agreed outcomes” section of the Implementation Status Report (for more details, see section 1 below). The following section presents the monitoring systems of the projects analyzed, including their objectives and chosen indicators. Monitoring arrangements for these indicators are discussed in the second section, and M&E capacity development, in the third. The fourth section focuses on how implementation support addressed M&E in the selected projects and is followed by the conclusions of the review. 1. Monitoring Systems In this report, monitoring systems mean the objectives and indicators selected for a project at the PDO and intermediate level. Monitoring arrangements, in turn, consist of all actions associated with collecting data on the indicators, making this information available to users, and providing them support in using this information.

Project Development Objectives Roughly two-thirds of all projects (21) aimed to improve the quality of education; a similar percentage (19 projects in all) aimed to reduce inequities in education. Access to education was an explicit objective of nearly half of the projects (14). Nearly one-third (9) sought to improve the institutional capacity of the education system. Whereas 5 projects explicitly sought to improve student learning outcomes, a majority (16) used a learning outcome as an outcome indicator. The primary objective of one project was to increase community participation.

Figure 1. Project development objectives, FY05 and FY06

0 20 40 60 80 100

Community participation

Learning outcomes

Internal eff iciency

Institutional capacity

Access

Equity

Quality

%

Source: World Bank project documents.

2

The projects most commonly sought to reduce regional educational inequities by addressing disadvantaged, poor, or underserved communities. Although certain projects sought to meet Millennium Development and “Education for All” goals, only three identified girls as a specific target group and two, other disadvantaged groups (ethnic minorities and children with disabilities). Generally, Project Development Objectives were formulated as higher-level objectives that would have been more appropriate for demonstrating longer-term developments in the education sector than the performance of an individual operation. Many projects also had multiple objectives, such as “increase access and quality of education” and simultaneously addressed different levels of education, which would have required complex monitoring arrangements. In addition, a number of PDOs did not specify whether a project targeted primary or secondary education, or both. Although this information can be found in outcome indicators, a technically sound objective should define the key recipient of project benefits and what they will do differently as a result of the project.

Selection of indicators After project objectives are agreed, project teams select indicators. In the education sector, most indicators are drawn from the Millennium Development Goals and “Education for All” goals. These indicators are also usually used in Country Assistance Strategies. The most common indicators are therefore net enrollment rates, completion rates, and learning outcomes (see table 1 below). While conducting the review, it was learned that project teams actually select monitoring indicators twice. The first selection takes place during project preparation and the second, when indicators are chosen for the ISR. Team Task Leaders (TTLs) are required to identify at least one indicator for the PDO and for intermediate outcomes in the “status of agreed outcomes” section of the first Intermediate Status Report, in accordance with the Results Framework documented in the PAD. During project implementation, TTLs are required to provide quantifiable follow-up information on the indicators identified in the ISR, either using measured values or qualitative indicators. Alternatively, a brief explanation may be provided as to why indicators are unavailable on progress towards achieving project development objectives and intermediate outcomes, together with the date of the information. What indicators were used most frequently in Project Assessment Documents (PADs)? The summary “indicator inventory” below (see annex 3 for the full inventory) shows the indicators most frequently used in the PADs and ISRs of the projects analyzed in the review.

3

Table 1. Outcome indicators in PAD and ISRs of 30 education sector projects, FY05 and FY 06

Frequency of use in PADs Frequency of

measurement in ISRs

Indicator Number of projects %

Number of projects %

Indicators for quality of education

Learning outcomes 16

53%

9 30% Decreased pupil/teacher ratio (pupil : qualified teacher ratio in one project) 6

20%

3 10%

Teacher capacity 5

17%

4 13%

Increased instructional time/ time on task for students 1

3%

1 3%

Indicators for equity

Equity-specific and disaggregated indicators (see annex 3)

15

50%

8 27%

Indicators for access

12

40%

11 Primary and secondary school completion rates 37%

Net enrollment rate 9

30%

9 30%

Gross enrollment rate 4

13%

3 10%

Other access indicators 6

20%

6 20%

Indicators for institutional capacity and governance

15

50%

10 Several indicators (see annex 3) 33%

Indicators for internal efficiency

8

27%

4 Repetition rate 13%

Dropout rate 7

23%

3 10%

Survival rate/ retention rate 4

13%

2 7%

4

13%

1 Transition rate 3%

1

3%

1 Promotion rate 3%

1 3% 0 Failure rate 0%

Community participation

Several indicators (see annex 3) 3 10% 1 3%

Source: World Bank project documents. The most frequently used indicators in PADs were improved student learning outcomes and completion rates. Student learning achievement was a PDO indicator for a total of 16 projects (53 percent); another 12 (40 percent) planned to track project achievements using completion rates. In addition, projects used different kinds of output and outcome indicators to measure instructional capacity and equity in education. Among less frequently used indicators were those measuring access (e.g., net enrollment rate, gross enrollment rate) and internal efficiency.

4

Quality indicators other than learning outcomes were rare. A few projects planned to track input indicators such as “percentage of trained teachers” (four projects) or “teacher per pupil ratio” (six projects). While the majority of projects had a specific component for quality improvement and/or for the development of teacher capacity, only two planned to track changes in pedagogical practices in the classroom. The review revealed that more clarity is needed in indicator definition. For instance, one project used the indicator “increased instructional time,” which refers to the planned increase in time allocated to primary education from 690 to 900 hours per year by 2010, as well as the indicator “time on task.” These indicators, however, represent two different concepts. The latter, for example, istypically used to monitor teacher effectiveness, which requires classroom observation. The majority of projects planned to produce disaggregated data by gender, but failed to establish sound monitoring indicators for other equity targets. While two-thirds of the projects (19) explicitly aimed at promoting equity between different geographical areas or beneficiary groups (according to their PDOs) and half had equity-specific or disaggregated PDO indicators, only one project managed to establish disaggregated baselines and, consequently, disaggregated target values for the operation. Net enrollment rate was commonly used to track access to education. Projects also identified other access-related indicators, such as absolute enrollment numbers for project components, participation rates, or transition from one grade level to another. Using absolute enrollment numbers is appropriate if these numbers are directly linked to project components (such as building new schools) and if they can be monitored in a regional and/or national context. Institutional capacity is generally monitored using output indicators. “Timely publication of educational statistics” or “establishment of school grant programs” were commonly used by the projects that were reviewed. In addition, budget-related input indicators were used to monitor sector management and governance, such as “increased non-salary expenditure within the education sector” or “increased share of education in the national budget.” Nearly all the projects included these output indicators in their respective ISRs. Community participation was also monitored through output or input measures, such as the percentage of schools with parent-teacher associations (PTAs) or school management committees (SMCs), or the number of parents participating in school management. More outcome-oriented indicators and intermediate systems are needed, however, to effectively track changes and identify areas where policies could benefit, for instance, from the subjective perceptions of parents and school officials. Only a few projects used qualitative, or subjective, indicators. Although gathering qualitative information may demand more resources and be more costly, these indicators are as important as quantitative ones when tracking sustainable education reform. Indicators such as “level of teacher satisfaction” or “new assessment systems accepted by stakeholders” needed better formulation. Monitoring arrangements should also have been better defined, but credit should be given to those projects that attempted to address both qualitative and subjective aspects of their overall objectives. With respect to the Millennium Development and “Education for All” goals, relevant indicators were often chosen, but many of them were more useful for assessing long-term trends than the results of a single project. In such cases, the projects would have benefited from using a combination of PDO indicators. For instance, the indicator “completion rate” could have been

5

used in conjunction with such indicators as dropout rate and repetition rate. Increases in dropouts or repeats would then flag that the quality intervention was not appropriately targeted. Finally, using the existence of baseline measurements to assess the quality of project indicators did not enable the review to judge the degree to which a project was outcome oriented. For instance, 85 percent of indicators in the ISRs of the various projects had quantifiable baselines, but a closer look showed that the majority of these indicators were outputs for which it is easy to establish “zero” baselines. Such indicators, moreover, generally do no more than track the performance of a project and assess whether its activities have been completed and deliverables delivered. It is critical, therefore, to separate zero-baseline data (i.e., baselines for which the value is zero) from the absence of data. It is equally important to define whether the baselines established by a project are related to the sector as a whole or to project deliverables only. For instance, it has been unclear whether data on similar activities implemented by other donors could have been used as a baseline for the Direct Support to Schools (DSS) component of the Education Sector Support Program in Malawi.

Indicators used in Intermediate Status Reports (ISRs) As noted above, the ease with which a zero baseline could be established for many indicators made it difficult to evaluate their quality, despite the fact that baseline data is the conventional criterion for doing so. One criterion for indicators used in ISRs could thus be usability of the indicator. Because the purpose of monitoring is to improve the performance of a project, information on core indicators should be used not only to track progress, but to guide implementation. For instance, how can information on indicators such as “number of school grants delivered” or “percentage of teachers participating in professional development” be used to guide project implementation? On the other hand, upgrading indicators to track outcomes (e.g., “teachers observed using new interactive methods”) would require specific arrangements to collect both baseline and follow-up data. Indicators such as school dropout rate, grade repetition rate, or grade failure rate could be useful for tracking achievement of better completion rates in the ISRs. Surprisingly, although a number of projects (6–8) included these measures among core PDO indicators, they were not frequently selected for monitoring in ISRs (see table 1). Thus while eight projects identified the repetition rate as a PDO indicator, only four included it in their respective ISRs. Similarly, only three out of six projects included the dropout rate as an indicator in the ISR. Although the quality of data on these indicators may be questionable, as indicated in some PADs, it is necessary to keep these indicators on the monitoring agenda. On average, projects identified six PDO indicators in the PAD and the ISR, with a range of 1 to 18 indicators per project. Although it is not possible to define the correct number of indicators, some recent studies suggest that it is better to limit the number of outcome indicators because too many appear to cause less data to actually be collected.2 A smaller number of indicators also helps focus dialogue with governments on critical program or sector development components.

2 World Bank, 2006, “From Schooling Access to Learning Outcomes: An Unfinished Agenda; An Evaluation of World Bank Support to Primary Education,” World Bank, Washington, DC; and World Bank, 2006, “Measuring Results: A Review of Monitoring and Evaluation in HNP Operations in South

6

Finally, it is unclear what criteria task team leaders used when choosing indicators for the “agreed outcomes section” of ISRs. Furthermore, none of the ISRs or AMs indicated whether or not indicator selection was made in collaboration with the client. Although ISRs are internal reports meant to flag issues to Bank management, they also include valuable information for the client. When selecting indicators for the ISR, then, team leaders should take the following into account:

(i) the indicators selected for systematic review should be selected in consultation with the client;

(ii) it should be possible to establish a baseline and obtain quantifiable follow-up data;

(iii) indicator data should be used to guide project implementation;

(iv) project progress should be measured against clear annual targets; and

(v) there should be a clear dissemination and utilization plan for the indicators. Intermediate monitoring systems While it is not always possible to measure progress towards a project development objective using PDO indicators, intermediate monitoring systems are designed to help education ministries and Bank staff assess whether a project is proceeding towards intended outcomes. Analyzing why intermediate outcomes are or are not being achieved, even when project outputs are being delivered, provides an opportunity to make changes in project implementation strategies. For instance, if “improved teaching practices” are not observed even if teacher capacity development has been delivered, revision of the professional development strategies or other strategies affecting teacher performance in the classroom may be required. The review findings suggest that more attention should be given to the design of sound intermediate monitoring systems and arrangements. Efforts were made to establish sets of intermediate indicators (on average, each project had 14 such indicators), but the intermediate results themselves resembled a checklist of actions or outputs rather than targets to be achieved. In addition, some intermediate monitoring systems were incomplete and a considerable number of intermediate indicators did not have appropriate baselines or target values. Some projects also did not specify intermediate results in their Results Framework, referring instead to project components. A sound intermediate monitoring system should be built on concrete, measurable results because a project will achieve its objectives through results, not through indicators or outputs. These results should describe the behavior or change intended for target beneficiaries. Intermediate indicators in turn should produce important data that allows the project team to assess whether these changes are being made. It is also important that plans and feedback loops then use this information. Neglecting appropriate intermediate indicators during project preparation and during early stages of implementation will affect the performance of a project; it can also complicate and increase the work of task teams. For instance, it is important to have sound intermediate monitoring systems that track progress towards better learning outcomes because monitoring data on these outcomes will not be available in the short term. In this case, early grade reading assessments could provide useful information about learning achievements and preconditions for success in national achievement

Asia and Some Practical Suggestions for Implementation,” South Asia Development Sector, World Bank, Washington, DC.

7

tests. RTI International and the World Bank, for example, have developed a reasonably priced Early Grade Reading Assessment (EGRA) instrument, which can be used to assess student reading skills. This kind of instrument can be used to provide immediate feedback both to project management and education officials. Using classroom observations as an intermediate measure is highly recommended. Because the classroom is the core of education quality, direct observation of classroom practices could be helpful in defining causal chains of both capacity development for teachers and learning outcomes for students. Depending on the scope and objectives of a project, improved teaching practices could be used either as an intermediate indicator or as a PDO indicator. A smaller number of intermediate indicators would help projects stay focused on effective performance monitoring. Among the projects reviewed, the number of intermediate indicators per project ranged from 3 to 26 per project in the PAD (14 on average) and 1 to 20 indicators (6 on average) in the ISR. Finally, teams could be more specific in formulating indicators. For instance, there is a big difference between “availability of resources for school-level pedagogical support” and “availability of school-level pedagogical support.” Although both indicators are relevant, their perspectives and purposes are different. If the intention is to monitor resources and inputs, the former would be appropriate. If the purpose is to monitor implementation, the latter could be considered. Finally, if a project seeks to measure outcomes, an indicator should reflect the behavior either of the persons providing the pedagogical support or of the beneficiaries, teachers, and students themselves.

2. Monitoring Arrangements

The Results Framework of the PAD (specifically, the “arrangements for results monitoring” section) presents detailed information about project indicators, their baselines, and target values, as well as monitoring arrangements for each indicator at the PDO and component level. This information is instrumental in allowing Bank management to understand whether annual targets have been achieved. The following discussion presents certain important aspects of M&E planning. Readiness assessment While the projects reviewed for this report usually relied on domestic capacity and procedures to collect data, only a few had analyzed the roles and responsibilities associated with data collection; the project’s organizational capacity to collect, process, report, and use information; and the issues of incentives or lack thereof. This kind of readiness analysis would provide a solid basis for M&E because “[m]ost of the existing M&E systems start by jumping straight into building a results-based M&E system without knowing where a given country stands in relation to a number of critical factors, including organizational roles, responsibilities, and capabilities.”3 Issues identified in a readiness assessment would drive a strategy toward creating an M&E system that is sustainable and usable for a given country. Compared to a needs assessment, a readiness

3 Judy Kusek, 2004, “Ten Steps to a Results-based Monitoring and Evaluation System,” World Bank, Washington, DC.

8

assessment assumes that a government needs these systems and evaluates whether it is actually ready and able to move forward in building and sustaining them. For instance, the monitoring arrangements of the projects under review were usually based on an assumption that domestic monitoring systems (e.g., a school census) captured the data required for monitoring project performance. However, none of the PADs indicated that existing data collection instruments had been analyzed to assess whether they and their associated procedures were able to capture accurate data in a timely manner. In addition to shedding light on local data collection capabilities, a readiness analysis should also map out the domestic monitoring cycle to evaluate when data would become available. Certain indicators (e.g., completion, NER, etc.) in the education sector are collected only once a year and therefore the systematic reporting of ISRs (assuming more than one implementation support mission per year) cannot reflect trends or changes in the short term. Baseline studies More than two-thirds of the projects analyzed by the review had a plan to implement a baseline study, either for a specific innovation or for learning outcomes. The information in the ISRs showed delayed implementation of these studies was not unusual. Realistic time frames are therefore needed to allow for the creation of necessary baselines. If it seems impossible to create such a baseline within a reasonable period of time, alternative arrangements for creating baselines or alternative indicators might be considered. Investing significant time and resources in baseline studies may not be relevant and cause delays that have a negative effect on other project activities. A critical question to ask is whether the baseline will be used only by the project, or whether the education ministry has made a commitment to use and continue measuring the indicator in the future. The issue of baselines should thus be looked at carefully, especially when planning for learning outcome assessments. Among the projects reviewed, baselines for learning outcomes were available only for 3 out of 16 projects; the remaining projects planned to establish baselines during the first year of implementation. For projects where baselines were available, the accuracy of the baseline was questionable. For instance, it was unclear what the baselines such as 19.6 (Ghana) and 79.6 (Vietnam) actually were measuring, nor was it clear how comparable data would be obtained. When contemplating how to measure learning outcomes, a project needs to critically analyze whether it is realistic to implement both a baseline study and follow-up studies over the life of a project (4–5 years) and whether the selected monitoring indicators are appropriate for measuring the performance of the project. While the overall purpose of any education sector intervention should be improving the quality of education, establishing a good-quality baseline or building the capacity to implement and use a baseline survey may be sufficient for certain projects. Setting targets It is important to include information in project documentation on how target indicator values were set, that is, whether or not they were based on special calculations, drawn from education sector projections, or created using other criteria (e.g., calculations based on time series data of, for example, student enrollment). Although most projects included this information in the PAD, there were other indicators for which such a reference would have been needed to justify updating the baseline.

9

Creating disaggregated data Monitoring arrangements should describe how disaggregated data for equity indicators will be obtained. It should be noted here that monitoring indicators that are disaggregated by administrative unit will almost always require that a project to make complementary efforts to build the capacity of decentralized administrative units to conduct monitoring and analysis. When establishing objectives for a project, especially for equity objectives, the project design team thus needs to answer the following questions:

(i) What level of M&E data disaggregation is required?

(ii) Is a baseline is available? If not, how it will be generated and by whom?

(iii) Does the government have an interest in and commitment to establishing and using disaggregated data?

(iv) Will data collection become a regular activity that the government will undertake and finance?

New indicators While projects surveyed for this report chose to use existing indicators that are tracked by relevant domestic monitoring systems, for a number of projects new indicators were introduced to measure learning outcomes, capacity development, or sector management. For most such indicators, either the projects planned to establish baselines during the first year of implementation or a “zero” baseline was used. New indicators, however, require proper monitoring arrangements. It requires substantial effort to create baselines for new indicators; sustainable changes, moreover, can usually only be measured over a longer period of time than the life of a project. Establishing an accurate baseline for qualitative indicators may, for example, require several measurements to establish a trend over time. Task teams are advised to use the following questions to guide them in deciding whether new indicators are needed to measure the performance of a project:

(i) What will the indicator be used for during project implementation? after project completion?

(ii) Is there a need and commitment on the part of the government to continue collecting data on the indicator or is it seen solely as a project-related activity?

When developing indicators for innovations, such as school-based management, project teams should analyze existing data collection instruments and suggest changes, if needed, in line with the new policies. For example, in the course of this review, analysis of certain school census questionnaires found that information about parent participation is collected using questions such as “how active is the SMC/ PTA?” In other words, the questionnaires did not provide any criteria for how parent activity should be measured. There are good examples of data collection instruments being updated to meet new policy and information requirements. For instance, new questions were added to the standard school census in Malawi to collect data on the “Direct Support to Schools” initiative. Defining targets for new indicators for which no baselines are available is a tricky task. This is especially the case where learning outcomes are concerned. For instance, the target for some projects was “improved learning outcomes compared to the baseline, to be established during the early years of project implementation.” For other projects, quantifiable targets were defined as a

10

two- to five-percent improvement over the baseline. (One to two percent was considered adequate yearly progress, given that schools were provided necessary resources and support.4) Measuring learning outcomes Although learning outcomes should be a core indicator for any education intervention, project teams should consider whether learning outcomes are appropriate for a specific project. To do so, project designers need to have a good understanding of the purposes and types of learning outcome tests. For instance, PISA5 focuses on the evaluation of life skills; its results cannot be used for the same purpose as results from curriculum-referenced tests, such as TIMMS.6 Arrangements for measuring learning outcomes need to defined in sufficient detail in the PAD. While two-thirds of the projects surveyed planned to track learning outcomes using national and/or international educational tests, none of the PADs described in sufficient detail the arrangements for the implementation, analysis, and dissemination of such a test or test results. For certain projects the national examination council was identified as an implementing body, for others, an external consultant. Only two projects mentioned the use of end-of-term examinations or school exit examinations as a source of information about learning outcomes. Task teams also need to have knowledge of mechanisms that can be used to ensure that comparable data is obtained by baseline and end-of-project surveys. In addition, they need to know how to set quantifiable targets and be aware of the capacities needed and available to implement, analyze, and disseminate planned test results. It is also important to recognize that school-level results are not always usable because they are volatile from year to year, as student cohorts and school staff change. Finally, the benefits of learning assessments depend on whether the information generated by the assessments is actually used. In countries where there is no tradition of standardized assessments, a great deal of background work will be needed so that all stakeholders come to an understanding of the importance of these assessments. A simple tool has been developed for this report to assist TTLs to decide whether learning outcomes are an appropriate outcome indicator for a project, as well as to help them design implementation arrangements (see annex 4). Impact evaluations Six out of 30 projects mentioned an impact evaluation in the PAD, but only one had a design for a rigorous impact evaluation with treatment and control groups. Some projects did not mention an impact evaluation, but planned either to create or collect data on relevant control variables (e.g., age, sex, and grade) or evidence on relevant outcomes that could be measured over time (e.g., learning achievements). Nearly all PADs mentioned that the projects would carry out process evaluations, evaluation studies, end-of-project evaluations, or mid-term reviews. Because the Bank is now emphasizing impact evaluations of its interventions, project designs should provide evaluators with the tools to develop an initial impact evaluation strategy. This task involves (i) defining project innovations in sufficient detail; (ii) defining arrangements for the creation of control groups; (iii) establishing a baseline or determining baseline collection;

4 Robert L. Linn, E.L. Baker, and Damian W. Betebenner, 2002, “Accountability Systems: Implications of Requirements of the ‘No Child Left Behind Act 2001,’ ” Education Researcher (Aug–Sep). . 5 The international student assessment program of the Organisation for Economic Cooperation and Development (OECD). 6 Trends in International Mathematics and Science.

11

(iv) planning for the collection of follow-up data; and (v) ensuring there is a sufficient budget to implement an impact evaluation.

Use of M&E information The Results Framework has a specific section in which project teams can define the specific uses of M&E information. Yet the projects reviewed for this report usually repeated the general purposes of the M&E in this section. Sixty percent of reviewed projects indicated that M&E information would be used to track project performance and provide feedback on implementation, leading to possible re-evaluation of the project strategy. Indeed, this is the overall purpose of M&E. Only a few projects mentioned that PDO-level monitoring and evaluation would be used to monitor progress towards higher-level objectives (e.g., those mentioned in Country Assistance Strategies), or to analyze project impacts or create awareness and accountability. Uses of M&E information that were mentioned less frequently included providing feedback to sector management and/or monitoring sector performance. None of the projects mentioned that M&E information would be used to rate the project in the ISR.

Table 2. Use of outcome-level M&E information in the Results Framework

Use of M&E information Number of projects

Percentage of projects

Assess project performance, feedback on implementation, for re-evaluation of program strategy

18 60%

Improve sector management and/or monitor sector performance 7 23% Assess project impact 4 13% Monitor progress towards MDGs, EFA or CAS goals 4 13% Create awareness and accountability 4 13% Provide feedback relevant to policy development 3 10% Create a baseline for project indicators or SWAPs 2 6%

Source: World Bank project documents. Lack of thorough analysis of the potential users of monitoring information made the dissemination strategies of the projects weak and too general. Only a few projects identified to whom and how this information would be communicated. Whereas some attention was given to making ISRs more results based, the Aide Memoires paid scant attention to results. For instance, analysis of ISRs and AMs showed that while progress on core indicators was reported to Bank management in the ISR, this information was not usually communicated to the client. In fact, only 3 out of 30 projects incorporated quantifiable information about the status of agreed outcome indicators in the Aide Memoire. In general, AMs summarized the accomplishment of activities and actions that needed to be taken, with less emphasis on analysis of indicators and monitoring arrangements—both of which should be emphasized during early stages of project implementation. Based on the information available in AMs and ISRs, it was clear that certain projects discussed the use of M&E information with the relevant country education ministry and focused on disseminating monitoring information.7 The PASE-project in Nicaragua, for example, included 7 It should be noted that a lot of good work is done during implementation support missions, when important discussions are held. However, all of this work is not necessarily recorded and reported in ISRs

12

disbursement conditions for strengthening the statistical unit of the education ministry. Financing was conditioned on the preparation and dissemination of agreed educational indicators. In Sri Lanka, the World Bank mission identified the following key policy areas and levels of use for M&E information:

(i) increasing public awareness, which involves the provision of information to the public (e.g., via media, public debate, etc.);

(ii) improving policy and decision making;

(iii) improving teaching and learning in the classroom (on the basis of assessment information, teachers should be assisted in identifying necessary adjustments to their procedures); and

(iv) feedback on curricula, teacher training, and teacher support services.

Sharing ISR monitoring information with the client would help identify weaknesses in M&E systems and improve their performance. For instance, if the ISR rates the quality of M&E data as “fair,” it would be helpful, or even necessary, for the client to be informed of these ratings, the reasons behind them, and what the client can do to improve the quality of the data. Clients should also be able to learn about ratings and “flags” in order to promote accountability. One size does not fit all. Dissemination is not an “end-of-project” activity and traditional strategies for “spreading the word” are not sufficient to promote the use of M&E information. Rather, an effective dissemination strategy is based on three core elements:

(i) analysis of the information needs of the user;

(ii) dissemination via the user’s preferred media; and

(iii) tailoring content to the user’s information needs. It is also important that end-users have confidence in the data and are supported to use the information. 3. Strengthening M&E Capacity Nearly all the projects reviewed had an explicit component for improving the M&E capacity of the national education ministry or included this goal in a wider institutional capacity development initiative. Providing support to help client countries generate M&E information is justified because these countries typically have a great need to improve data collection and analysis. Also, the integration of an education management information system (EMIS) with other information systems and planning mechanisms in a given country is increasingly critical when moving toward a sector-wide approach (SWAP) to development. SWAP interventions generally focus on the supply of M&E data (e.g., the development of an EMIS) and are measured by output indicators, such as “producing educational statistics.” Production of statistical information is essential, but it is equally important to develop the capacity to use this data in planning and decision making. While country-specific M&E systems are increasingly required to produce more complex information about educational processes,

and AMs. Therefore the analysis recorded in these documents may not provide a reliable picture of the work of task teams.

13

outputs, and resource utilization, there is also a need to develop management capacity and systems for using the information generated by these systems. Demand for M&E can be achieved only if users have the skills, incentives, and authority to use the information created by the process.

4. Implementation Support

The ISRs available on the World Bank intranet for the 30 education projects reviewed were analyzed to see how the first implementation support mission addressed M&E and whether they brought any issues to the attention of Bank management. Since the projects have been only been operational for an average of 12 months, only a few provided quantitative follow-up information on core indicators in the ISR. At this stage, then, it may be too early to assess whether monitoring information has been used for decision making or creating good practices. As noted earlier, all project indicators are not systematically monitored in the ISR (see table 1). Certain projects selected one or two indicators, as noted in the instructions; task team leaders for other projects included all indicators presented in the PAD. As previously mentioned, none of the ISRs or Aide Memoires indicated whether the selection of indicators for the ISR was made in consultation with the client, neither did they make clear on what basis the indicators had been chosen. The ISRs reveal that implementation support should make better use of monitoring systems and instruments. For instance, there is little evidence in the ISRs and AMs that the Results Framework was used as a monitoring tool. Although annual-level targets were set for each indicator, they were not really used to assess project performance. In addition, the progress of individual projects was not systematically communicated in quantifiable terms to the client in the Aide Memoires, as it is in ISRs, given that the AMs focus more on the accomplishment of activities. Only a few projects reported that indicators chosen for a project were discussed with the client or revised during the first implementation support mission. This activity should be included in the agenda for first missions to guarantee that all partners understand and agree upon project M&E indicators. One-third of the projects mentioned M&E among the critical issues to be brought to management attention. In most cases, the issues were related to the lack of baseline data or delays in implementation of baseline studies. Management itself drew attention to poorly formulated indicators and encouraged task teams to assess monitoring systems with their respective clients. The ISR needs clear criteria to assess the quality of outcome information. For most projects, the starting point for ratings of information quality was “fair” or “good,” but these ratings changed as early as the first year of implementation. Surprisingly, ratings also changed for projects where follow-up information was unavailable. ISR guidelines require that the reasons for downgrading information quality to “poor” be explained, but it remains equally important to keep track of improvements and their causes. Monitoring would benefit the client to a greater degree, moreover, if there were clear definitions of “poor” or “fair” data quality. Multiple sections of the ISRs accounted for the weaknesses in M&E. For instance, downgraded or “poor” ratings were explained by the lack of a baseline in several sections of various ISRs,

14

whether the ratings concerned the quality of outcome information, M&E performance, or progress toward achievement of the PDO. More clarity is thus needed on how ISRs should address M&E. Whereas some projects used government inputs (i.e., insufficient M&E capacity in the ministry) to explain inferior information ratings, others assessed M&E performance only against project outputs (e.g., delays in the production of statistical yearbooks). When there is uncertainty about the appropriateness of indicators or monitoring systems in general, projects could be more proactive in modifying monitoring systems more quickly. Minor changes in indicators do not need approval by management, but they should be reported in ISRs and AMs. When considering modifications, the key question to ask is: Will the modified indicators lead to different project result and/or outcome than what was promised in the original PAD, or are the modifications simply refinements of the original indicators? The former constitutes a restructuring of a project, which requires World Bank Board approval, the latter does not. Refinements of original indicators are proposed, approved, and documented via routine Implementation Status Reports. Conclusions and Recommendations A project is not necessarily results based just because outcome and global indicators are included in the Results Framework. For instance, global indicators (e.g., net enrollment and completion rates) are important indicators for tracking educational sector developments, but additional indicators and sound intermediate monitoring systems are needed to show the causal chains between these indicators and project inputs, activities, and outputs. The project preparation process needs to focus on both developing appropriate monitoring systems—especially for learning outcomes—and planning to use monitoring information. Besides focusing on measurable indicators, task teams should ensure that intermediate monitoring systems create a roadmap for achieving PDOs. The following suggestions are intended to help project teams develop results-based, utilization-focused monitoring systems:

• Include M&E planning in all stages of project preparation. After the PDO and component objectives have been agreed, define parameters for M&E indicators. Choose performance indicators where there is a clear link between project outputs (i.e., what is produced) and indicators (i.e., what is measured).

• Build intermediate monitoring systems based on outcomes and targets, not indicators. A results-based intermediate monitoring system defines targets to be achieved (e.g., behavior changes of beneficiaries or changes in classroom conditions). Formulating these results and indicators in an outcome-oriented manner will lead to a more functional M&E and more sustainable improvements in performance.

• Use outcome indicators to track project performance. If a project decides to use output indicators, remember that there is always a purpose for an output—the outcome. Check whether you can upgrade output indicators to become outcomes.

• When selecting indicators for the PAD and ISR, assess the appropriateness of indicators by applying the “usability of an indicator” criterion. Determine whether there is demand or direct use for the indicator, either during project implementation or at the end of the project.

• Promote pro-equity approaches to M&E. The M&E process should be able to demonstrate equity baselines and targets by gender, language, ethnicity, location, and

15

income, if needed. Monitoring activities should be prepared and reported in a manner that permits progress to be analyzed by the specific target groups identified by the project.

• When introducing new indicators, make sure that there is a need for the indicators and a commitment and resources on the part of the education ministry to continue tracking them. Bear in mind that creating a reliable baseline for a new indicator may require several measurements and a trend over time.

• Planning for effective monitoring arrangements requires a broader approach than simply defining responsibilities for data collection. Monitoring arrangements should be based on a thorough readiness assessment, together with a capacity assessment. These arrangements should cover a broad scope of activities, starting from identification and dissemination of the indicators, utilization of monitoring information, and user support.

• Analyze the capacity and readiness of the client to collect, process, report, and use monitoring information, as well as the incentives or lack of thereof for doing so. Check existing data collection instruments. Suggest changes, if needed, to better match the data collected to new policies.

• Reserve sufficient resources for M&E. Some project manuals suggest that five to ten percent of project costs should be set aside for monitoring and evaluation, depending on the scope and scale of a project.

• Measure changes in the classroom. There are a number of classroom observation measures that could be adapted and applied to track changes made in classrooms. Use both quantitative and qualitative measures adapted to the specific country context.

• Plan carefully to disseminate and utilize monitoring information. A dissemination strategy should define: (i) the intended users of the information; (ii) how they will use the information; (iii) how, when, and in which format information will be communicated to them; and (iv) how they will be supported to use this information. Dissemination is not an end-of-project activity, but an integral part of project implementation (and implementation support).

• Dissemination should be supported by developing a client’s capacity to supply and use M&E information. Good experiences about practices that have promoted the use of such information in the education sector should be widely shared.

• Make use of the Results Framework during implementation support.

• Communicate monitoring information systematically to clients. Include a section called “Status of Agreed Outcome Indicators” in the Aide Memoire. In addition, make available to clients the information in the ISR that would help them improve their performance.

• Include M&E in the agenda of every mission. Coach clients in results-based reporting and management.

• Finally, the ultimate goal of results-based M&E is not to prove, but to improve. This report argues that an M&E culture already exists in the education sector: teachers assess student performance. Supervision and inspection systems assess school performance and teacher performance. The challenge is to make better use of these monitoring systems to improve performance and promote accountability.

16

Annex 1: Primary and Secondary Education Operations included in the Review

FY 05 Project name Lending instrument

IBRD/ IDA total commitment (US$ million)

1 Cameroon Education Development Capacity Building SIL 18.2

2 Congo, Rep Support to Basic Education SIL 20.00 3 Malawi Education Sector Support Project 1 SIL 32.2 4 Cambodia Education Sector Support SIL 28.00 5 Tonga Education Support Project SIL 1.00 6 Vietnam Targeted Budget Support for EFA SIL 50.00 7 Bosnia-Herzegovina Education Restructuring SIL 10.00 8 Kyrgyz Republic Rural Education SIL 15.00 9 Serbia and Montenegro Education (Montenegro) SIL 5.00

10 Turkey Secondary Education SIL 104.0 11 Ukraine Quality Education Equal Assess APL 86.6 12 Costa Rica Equity and efficiency of Education SIL 30.0 13 Mexico Basic Education Development APL 300.00 14 Nicaragua Education SIL 15.00

15 Morocco Basic Education Reform Support Program SIL 80.00

16 Yemen Basic Education Development Program SIM 65.00 17 Nepal Education for All project SIL 50.00

FY 06 18 Burkina Faso BF- Post Primary Education SIL 22.9 19 Gambia, Reb. GM- Education 3 Phase 2 APL 8.0

20 Mongolia MN- Rural Education and Development -READ SIL 4.0

21 Philippines PH- National Program Support to Basic Education SIM 200.0

22 Albania Education Excellence & Equity SIL 15.0 23 Croatia Education Sector Development Program SIL 85.0 24 Moldova Quality Education in Rural Areas of MD SIL 10.0 25 Argentina Rural Education Improvement project SIL 150.0

26 El Salvador Excellence and Innovation in Secondary Education SIL 85.0

27 Mexico School-Based Management programme APL 240.0 28 Djibouti Second School Access and Improvement APL 10.0 29 Pakistan Balochistan Education Support project SIL 22.0 30 Sri Lanka Education Sector Development Project SIL 60.0

17

Annex 2: Framework used for Assessing M&E Systems and Arrangements Project Development Objective (PDO)

a) Is the PDO clear, measurable, and focused? b) Does the PDO define

- What group is targeted directly by the project as the key recipient of project benefits? - What the target group will do differently after project completion that will make it better off?

N.B.: The link between PDOs and higher-level objectives are not the subject of this review. PAD: Selection of PDO indicators/ outcome indicators

a) Are the indicators related to inputs/ outputs/ outcomes/ processes? b) Quality of indicators:

- Are the indicators measurable? - Are they defined in operational detail? - Are baselines set? If not, what arrangements have been made to establish baselines? - Are target values set? - Are data requirements specified? - Are monitoring arrangements designed in sufficient detail? How is data collected and analyzed? - What are the sources of baseline information and target values?

ISR: Selection of PDO indicators/ outcome indicators

a) What indicators are selected for the ISR? b) Are baseline data for these indicators available? c) Is quantifiable follow-up information reported in the ISR?

Intermediate-level monitoring systems

Intermediate results: a) What type of intermediate results are monitored? activities? outputs? outcomes? b) What is the quality of intermediate results? Their measurability? Are they outputs or outcomes? Intermediate indicators: c) What is the number of intermediate indicators? d) What type of intermediate indicators are used? inputs? outputs? outcomes? processes? e) What is the quality of intermediate indicators? Their measurability? Baseline/ target/ monitoring

arrangements? f) Are data requirements specified?

Monitoring arrangements (PDO and intermediate) a) Are monitoring arrangements described in sufficient detail? Do they include:

- Responsibilities for data collection, data processing, analysis, reporting and dissemination? - Frequency of data collection (monthly, quarterly, annual, bi-annual, other)? - Budgets?

b) Does indicator data require specific studies or data collection arrangements? Are plans for these arrangements defined?

c) Who will provide data? d) What capacities are needed to collect data? e) What is the quality of data assessed? f) Have secondary sources of data been examined?

Impact evaluation a) Does the project design include rigorous evaluation of project impact? b) Are project innovations described in sufficient detail to enable the design of impact evaluations? c) Have funds been identified for the evaluation?

Readiness and capacity assessment a) Was there any analysis of existing M&E systems? b) Was there any analysis of client capacity to carry out M&E ? c) What strategies have been chosen to build the client’s M&E capacity? d) What indicators are used to track M&E capacity development? e) Are M&E-related risks analyzed?

18

Framework used for Assessing M&E Systems and Arrangements - continued Dissemination and utilization

a) What are the planned uses of M&E data ? b) Are specific M&E users/ user groups specified in the Results Framework? c) Does the project have a dissemination plan and strategy? d) Is dissemination budgeted for?

Implementation of M&E a) Did the first ISR contain “satisfactory baseline” for project outcome monitoring? b) Is follow-up data presented in the ISR? c) Are there any modifications related to M&E in the ISR? If yes, what kind? d) What critical M&E issues have been addressed by the ISR? e) How does the project rate the quality of M&E data? Reasons for upgrading/ downgrading? f) Are there implementation performance ratings for M&E? Reasons for upgrading/ downgrading? g) Was there any discussion of methodological issues or data quality in the ISR, PAD, or AM? h) Is M&E information included/ presented in the AM? i) What critical issues have been reported to the client in the AM?

19

Annex 3: Indicator Inventory for Reviewed Projects

Indicators for Quality of Education

Number of indicators

Country/ project Indicator in the PAD

1 Cambodia Student achievement scores in Math and Khmer (grades 3, 6, 9) 2 Croatia Performance of students in PISA, benchmarked with Croatian performance stable or just

improving relative to EU average 3 El Salvador Proportion of test takers with intermediate or advanced results in Spanish and math at the Grade 9

and PAES exams 4 Gambia Improvement in student learning, especially girls

5 Kyrgyz Republic Improved average scores in Monitoring Learning Achievement surveys for students in grades 4 and 8

6 Mexico Average test scores in math and Spanish for students in participating PEC schools by type of school (primary or secondary)

7 Moldova Learning achievement in rural schools

8 Mongolia Student literacy skills

9 Nepal Learning achievement outcomes (grade 5) 10 Nicaragua Improve attention to student achievement on national standardized tests in Spanish and math for

3rd and 6th grades 11 Pakistan Annual improvement in learning achievement 12 Philippines GD 6 Achievement (2005) (mean and SD) in math, Filippino, English, science, social studies

13 Sri Lanka Cognitive achievement scores in first language and math (proportion of students achieving higher than 50 percent)

14 Tonga Student test performance (disaggregated by gender, government/ non-government, geographic location etc.)

15 Turkey Improved performance of secondary-school graduates of schools receiving new curriculum and related support via the project, as measured by scores on national and international assessments and graduate tracer studies

16 Vietnam Achievement in G5 math and language Other Ukraine New external assessment instruments accepted by education stakeholders

Total projects: 16

Note: This inventory contains indicators selected in the PADs of the 30 projects analyzed for this report.

20

Indicator Inventory for Reviewed Projects - continued

Indicators for Quality of Education: Pupil/ Teacher Ratio



1 Cambodia Pupil : teacher ratio in primary education

2 Cambodia Pupil : teacher ratio in lower secondary

3 Congo Rep Pupil : teacher ratio by department

4 Malawi Pupil : qualified teacher ratio for Standard 1-8

5 Malawi Pupil : qualified teacher ratio for standard 1-4

6 Malawi Pupil : qualified teacher ratio for secondary schools

7 Malawi Pupil : qualified teacher ratio CDSS schools

8 Ukraine Student : teacher ratios of rural schools in pilot rayons increased to 12-15 students per teacher

9 Vietnam Pupil : teacher ratio

10 Yemen Student : teacher ratio in publicly financed basic education schools

Total number of projects: 6

Other Quality Indicators

1 Albania Increase teachers’ satisfaction with their working conditions

2 Djibouti Percentage of teachers receiving continuing training

3 Gambia Increased instructional time (eventually time on task) for students

4 Serbia and Montenegro

In schools participating in the reform, at least 70% of teachers observed to be using the skills and knowledge gained from the training provided by the project

5 Ukraine 40% of teacher trainers conduct training with competence-based approach, using modern practices in teaching and learning

6 Yemen % of qualified teachers


21


Indicators for Equity



1 Argentina Reduce the difference in access of 6-year-old children in 1st grade as compared to 5-year-old children in pre-school in rural areas

2 Argentina Increase access of students completing grades seven to eight in rural areas

3 Argentina Reduce overage enrollment at EGB1 and EGB2 levels in rural areas

4 Argentina Improve promotion rate in EGB 1 (Grades 1 and 3) in rural areas

5 Burkina Faso Rise in the average gross enrollment rate in lower secondary in the 18 less-covered provinces

6 Burkina Faso Rise in the GER of girls in lower secondary

7 Burkina Faso Schools fees supported by families in public lower secondary schools in the 18 lowest-covered provinces reduced to 10,000 FCFA per year per student

8 Cameroon Female to male primary enrollment ratio in the CIP

9 Costa Rica Increased number of low-income students beneficiaries (quintile 1 and 2) in targeted macro-regions

10 El Salvador Proportion of the 17-25 age cohort in rural areas and of the poorest 40 percent of the population that completes secondary school

11 Mexico Reduction of the failure rates of indigenous students in primary education

12 Mexico Reduction in the repetition rates of indigenous students in primary education

13 Mexico Reduction in the dropout rates of indigenous students in primary education

14 Mexico Increase in the completion rates of indigenous students in primary education

15 Moldova Difference in achievement between secondary school students in rural and urban schools

Inter-quartile ratio for elementary schools 16 Philippines

17 Philippines Inter-quartile ratio for high schools

18 Turkey For secondary schools receiving new curriculum and related support via the proposed project, reduced variance in student achievements between schools in different socioeconomic regions, between general and vocational schools, and overall rise in achievement

19 Yemen Improve ratio of girls to student population (%)

20 Yemen Gross intake rate – female/male (%)

21 Yemen Female completion rate in grades 6 (%)

22 Yemen Female completion rate in grade 9 (%)

23 Yemen Female promotion rate from grade 4 to 5 (%)

24 Cambodia* Net primary enrollment rates: nationwide/ urban/ rural/ remote

25 Cambodia* Net lower secondary enrollment rates: nationwide/ urban/ rural/ remote

22


Indicators for Equity - continued



26 Morocco* Net enrollment rate in preschool (by gender/ rural/ urban/ total)

27 Morocco* Net primary-school enrollment rate (by gender/ rural/ urban/ total)

28 Morocco* Net middle school enrollment rate (by gender/ rural/ urban/ total)

29 Nepal* Net enrollment rates by district

30 Albania* Enrollment rate in secondary education disaggregated by region and income group

31 Tonga* Improved NER (disaggregated by gender, government/ non-government, geographic location etc.)


Note: * Disaggregated indicators.

23


Completion Rate


Country/project Indicator in the PAD

1 Burkina Faso

Increase in completion rate in lower secondary

2 Cambodia

Primary education completion rates

3 Congo Rep

Primary completion rate

4 Congo Rep.

Lower secondary completion rate

5 Croatia

Completion rate: primary

6 Croatia

Completion rate: secondary

7 Djibouti Percentage of specific children completing primary schooling, of which girls

8 Gambia Rep.

Completion rate of basic education, by gender and region

9 El Salvador

Proportion of 17-25 age cohort that has completed secondary

10 Mexico

Completion rates: non-indigenous primary schools

11 Mexico

Completion rates: primary schools

12 Mexico

Completion rates: Telesecundaria education

13 Morocco End-of-cycle completion rate in primary schools (by gender, urban/rural/total)

14 Morocco End-of-cycle completion rate in middle schools (by gender, urban/rural/total.

15 Pakistan Average student completion rate no less than 70%

16 Philippines

Completion rate: elementary

17 Philippines High school completion rate (Yr 1)

18 Vietnam Primary completion rate (survival to Grade 5)


24


Indicators for Access: Net Enrollment Rate



10 Albania Enrollment rate of secondary education disaggregated by region and income group

56 Cambodia

Net primary enrollment rates: nationwide/ urban/ rural/ remote

64 Cambodia

Net lower secondary enrollment rates: nationwide/ urban/ rural/ remote

3 Cambodia Upper secondary net enrollment ratio

11 Croatia

Net enrollment ratio: preschool

12 Croatia

Net enrollment ratio: primary

13 Croatia

Net enrollment ratio: secondary

14 El Salvador

Net enrollment rate in grades 7 to 9 and 10 to 12

9 Nepal

Net enrollment rates by district*

6 Morocco Net enrollment rate in preschool (by gender, urban/rural/total)

7 Morocco

Net primary enrollment rate (by gender, urban/rural/total)

8 Morocco Net middle school enrollment rate (by gender, urban/rural/total)

15 Sri Lanka

Net enrollment rate, ages 6-14

4

Tonga

Improved net enrollment rates (disaggregated by gender, government/ non-government, geographic location, etc.)

5 Vietnam Net primary enrollment


25


Indicators for Access: Gross enrollment rate (GER)



1 Burkina Faso Increase in gross intake rate in first grade of lower secondary

2 Djibouti Total gross enrollment rate in basic education, of which girls

3 Vietnam Gross intake in grade 1 (disaggregated by gender, ethnicity, location, grade, poverty level and disability, as appropriate)

4 Yemen Gross enrollment rate in basic education (%)


Other access indicators

1 Boznia and Herzegovina

Increase in share of secondary enrollment in general and broad-based four-year vocational programs in each entity

2 Costa Rica Increased access to non-traditional secondary education modalities (Telesecundaria)

3 Gambia Rep. About 2,200 additional students access basic education in under-served areas

4 Nicaragua Increase in effective enrollment demand in autonomous schools

5 Pakistan An additional 35,000+ students in primary schools established through community and public-private partnership

6 Philippines Participation rate of ages 6-11 years


Indicators for institutional capacity, sector management, governance

1 Albania Increase teachers’ satisfaction with their working conditions

2 Burkina Faso Increase share of MESSRS in education budget

3 Argentina Bilateral agreements between national and provincial governments in satisfactory execution; annual plans regarding educational statistics and impact evaluation studies in satisfactory execution

4 Argentina Annual plans regarding the compilation, analysis, dissemination, and use of educational statistics in satisfactory execution

5 Bosnia and Herzegovina

Number of education jurisdictions funding schools through a transparent formula mechanism

6 Cameroon Degree of randomness in teacher allocation

7 Congo Rep. Administration staff to teaching staff ratio in schools

8 Congo Rep Capacity of Regional Departments of Education to develop and monitor annual department education plans

8 Djibouti Recurrent spending other than teacher salaries as percentage of total recurrent spending on basic education

9 Gambia Rep. Implementation of personnel performance monitoring system and incentive framework in all central and six regional directorates by year-end 2007

10 Gambia Rep. Functional organizational structure in place

26


Indicators for institutional capacity, sector management, governance – continued



11 Tonga A fully developed school grants program established, including tested policies and guidelines

12 Tonga 100% of all schools substantively engaging their communities in school planning, budgeting, and results assessment

13 Tonga TSGP evaluation shows positive impact and cost effectiveness of financing reforms

14 Ukraine Increased MES monitoring of program implementation and strengthened procurement and financial management systems

15 Nicaragua Formal acceptance of 3-year rolling CWP evidenced by signed MOU by all donors

16 Nicaragua Increase % of donor financing that shares harmonized financial management arrangements

17 Nicaragua Increased # of donors attending joint supervision missions

18 Nicaragua Increase # of autonomous schools as a % of all public schools

19 Nicaragua Increase % of MECD employees hired by public recruitment process

20 Serbia and Montenegro

Elimination of education arrears and increase in non-salary current expenditures from state budget

21 Tonga Fully developed school grants program established, including tested policies and guidelines

22 Tonga 100% of all schools substantively engaging their communities in school planning, budgeting, and results assessment

23 Tonga TSGP evaluation shows positive impact and cost effectiveness of financing reforms

24 Ukraine Increased MES monitoring of program implementation and strengthened procurement and financial management systems

25 Vietnam Yearly per student spending on non-salary items

26 Yemen Recurrent spending on items other than teacher remuneration as % of total recurrent spending on basic education


27


Indicators for internal efficiency: repetition



1 Cambodia

Repetition rate (grades 1,2,3)

2 Cameroon

Primary repetition rates

3 Congo Rep.

Primary education repetition rate

4 Djibouti

Primary cycle repetition rates

5 Mexico

Repetition rates: non-indigenous students in primary school

6 Mexico

Repetition rates: primary schools

7 Mexico

Repetition rates: Telesecundaria education

8 Morocco

Repetition rate in primary school (by gender, urban/rural/total)

9 Morocco

Repetition rate in middle school (by gender, urban/rural/total)

10 Tonga

Repetition (disaggregated by gender, government/ non-government, geographic location, etc.)

11 Vietnam

Repetition of grade 1


28


Indicators for internal efficiency: Dropout rate



1 Albania

Basic education dropout rate

2

Costa Rica

Reduced primary education efficiency gaps in four targeted macro regions (measured by over-age students and dropout)

3 Djibouti

Primary cycle dropout rates

4 Mexico

Dropout rates: non-indigenous students in primary school

5 Mexico

Dropout rates: primary school

6 Mexico

Dropout rates: Telesecundaria education

7 Philippines

Dropout rate: elementary/ high school

8 Tonga

Dropout rate (disaggregated by gender, government/ non-government, geographic location, etc.)

9 Vietnam

Dropout rate


Indicators for internal efficiency: Survival/ retention rate

1 Nepal Survival rates to grade five

2 Morocco Retention rate in primary school; survival rate until end of cycle (by gender, urban/rural/total)

3 Morocco Retention rate at middle school; survival rate until end of cycle (by gender, urban/rural/total)

4 Philippines Cohort survival in elementary school

5 Philippines High school cohort survival (EFA)

6 Sri Lanka Survival rate through grade 10


29

Indicator Inventory for Reviewed Projects – continued

Indicators for internal efficiency: Transition/ progression/promotion rate



1 Albania

Progression rate from basic education to secondary education, by region

2 Cambodia

Transition rate from primary to lower-level secondary

3

Cambodia

Transition rate to upper secondary

4

Cambodia

Promotion rate (grades 1, 2, and 3)

5

El Salvador

Promotion of technical education graduates who continue to tertiary

6

Tonga

Transition rates (disaggregated by gender, government/ non-government, geographic location, etc.)


Indicators for internal efficiency: Failure rate

1 Mexico Failure rates for non-indigenous students in primary school

2 Mexico Failure rates: primary school

3 Mexico Failure rates: Telesecundaria education


Indicators for internal efficiency: Community participation

1 Djibouti Percentage of schools with PTAs

2 Mexico** Number of parents reached

3 Mexico Number of children benefited

4 Mexico Number of schools participating in PEC as percentage of total number of basic education schools

5 Mexico Total number of basic education schools participating in PEC

Commitment to the goals and activities of the PETE, as expressed by local financial contributions to schools

6 Mexico

Total number of projects: 3 Note: ** Basic education development.

30

Annex 4: Planning for Intermediate Monitoring Systems

Operational feasibility of intermediate results YES NO

- Are intermediate results an outcome? (Ask the question: Do the results describe changes in conditions or behavior of the target beneficiaries?)

- If it is an output (i.e., something the project will produce), is it possible to upgrade the result to an outcome? (Ask the question: Why is this output produced?)

- Are intermediate results directly linked to both the Project Development Objective and Component Objectives?

Operational feasibility of intermediate indicators

- Is the indicator measuring the achievement of an intermediate result?

- Is a baseline available? If not, how will it be generated?

- Is the baseline “zero” or “none”? If Yes*

- Are there management systems in place to generate the required data?

- Are any specialist skills required to generate the data?

- Will the collection of data require additional resources?

- Are measurable annual targets set?

- Are end-of-operation targets achievable, given the length of project operation?

- Is there a clear need for information on the indicator during project implementation?

Note: *Check the indicator. If it is a clear output, is it possible or necessary to upgrade it to become an outcome?

31

Annex 5: Establishing Indicators and Baseline Data for Equity Indicators

When establishing equity objectives and indicators, the project design team needs to answer the following questions:

- What level of disaggregation is required?

- Does data exist or can it be generated to satisfy the scope and disaggregation required?

- Is a baseline available? If not, how will it be generated?

- Can the data be generated at the required intervals?

- Are there management systems in place to generate and analyze the required data?

- Are any specialist skills required to generate the data?

- What are the costs of collecting and analyzing data?

- Does the government have an interest in and commitment to establishing and using this data?

- Is data collection proposed as a one-time project activity or a regular activity that the government will undertake and finance? Is there commitment to and evidence of financing?

- What definitions are used? For instance, are ethnic and language minorities specified? What definitions are used for disabilities? Are the same definitions used across various ministries?

Source: Donald Hamilton, 2006, unpublished paper, World Bank, Washington, DC. Note: How does the project establish disaggregated targets?

32

Annex 6: Questions for Preparing a Learning Achievement Survey The purpose of these checklists and questions is to assist Task Teams in deciding whether learning outcomes are an appropriate outcome indicator for a project, as well as to help them design initial implementation arrangements for a learning outcome survey. This checklist is not meant to be a comprehensive guideline for an implementation plan, but an instrument used to evaluate whether essential elements and aspects of M&E implementation have been taken into account during project preparation.

YES NO(i) Does the project focus on student learning outcomes?

(ii) Is it possible to present an argument for a link between the design/ implementation of the project and changes in student learning?

(iii) Is the time frame of the project sufficiently long to allow changes istudent learning to take place?

n

(iv) Does the ministry/ government want to measure changes in slearning using an achievement test?

tudent

(v) Are there other learning assessments planned/ ongoing?

Are learning outcomes an appropriate indicator? (See checklist above)

NO YES

A. There is a baseline available B. A baseline is not available 1. When was the baseline collected? 1. When will the baseline be available? 2. Baseline survey arrangements: Who designed the

test? Are the results generally disseminated and accepted by stakeholders?

2. Baseline survey arrangements: Who will design and implement the tests?

3. What kind of sample was used? Is it applicable to the project?

3. What sample will be used?

4. What is the quality of the baseline? For example, if mastery level was used, how was it defined?

4. How will the quality of the baseline be ensured?

5. What resources and capacities to plan, implement, analyze, and use the baseline test are available? What capacities and resources are needed?

6. How will targets be set? 7. How and by whom will the information on learning outcomes be used during project implementation? after project

completion?

Is a baseline for the target groups/levels available? What other quality indicators

could be used?

Does the project focus on learning outcomes?

YES (Checklist A) NO (Checklist B)

33