manuel fernando castro the role of m&e systems in performance budgeting: good-practice and...

Manuel Fernando CastroManuel Fernando CastroThe Role of M&E Systems in The Role of M&E Systems in

Performance Budgeting: Good-practice Performance Budgeting: Good-practice and lessons learned from experienceand lessons learned from experience

V Konferencja EwaluacyjnaV Konferencja EwaluacyjnaWarszawa, 23.10.2009Warszawa, 23.10.2009

The Role of M&E Systems in Performance Budgeting: Good-practice and lessons learned from experience

Warsaw, October 2009

V Annual Evaluation Conference

Overview

► What do we mean by M&E and PB

► Strategic context for M&E and Performance Budgeting

► Role of country-level M&E systems in Performance Budgeting

► Examples of good practice countries (Mexico, Chile, UK)

► Lessons from international experience

5

Regular tracking of indicators to establish whether planned targets have been achieved

Monitoring “M”

Evaluation“E”

Analytical efforts to answer specific questions about performance of a program/activities

Impact evaluation

“IE”

Analytical efforts to relate cause and effect. Establishing what would have happened in the absence of the intervention

What is M&E?

Each tool has strengths, costs, limitations

6

What is Performance Budgeting?

• “Form of budgeting that relates funds allocated to measurable results” (OECD, 2005a)

• 3 different modalities

Presentational Performance-informed Direct/Formula funding

Linkages with allocation No links Indirect/Loose link Direct/Tight link

Main objectiveAccountability Planning and/or

accountabilityResource allocation and accountability

Evidence of utilization

Sweden, Denmark, Colombia

Most OECD (e.g., UK, Australia , Mexico) + Chile.

Mostly in sectors (e.g., Denmark's school funding, Chile's per capita funding for clinics)

Overview




► Examples of good practice countries (Mexico, Chile, UK)


There is no simple/direct relation between spending and results

• Similar increases in public spending can be associated with totally different outcomes…

Spending data for 1990s from World Development Indicators database. Child mortality data from Unicef 2002. Other data from World Bank . School completion from Bruns, Mingat and Rakatomalala 2003.

More spending does not necessarily mean better outcomes

There is no simple/direct relation either from growth rates to results

• Tanzania and Bangladesh have similar GDP per capita, but life expectancies are 55 years and 65 years.

• Philippines 2006 per capita income was 30% higher than in Moldova, but Moldova’s maternal mortality ratio was 16 as opposed to 162 per 100,000 live births in the Philippines.

• Despite remarkable economic growth, India’s maternal mortality rate has worsened and is six times worse than China's and 14 times worse than Chile's.

Higher economic growth, can mask the differences in important outcomes of development.

Budget officials and policy makers need to respond challenging questions

M&E can be associated with better project outcomes (assist with improved management)

• 80 - 90% WB projects rated high or substantial in M&E were rated satisfactory or better in their outcomes

• Those with negligible ratings had satisfactory only in 32% of cases

Overview




► Examples of two good practice countries (UK, Mexico)


13

Key factors

• Evaluation measures results, Performance Budgeting tides results to funding.

•To be relevant and sustainable PB demands continuous performance information:

• But …multiple requirements to fulfill

• Clear program logics / results chains•Timely, and quality data•Good indicators and targets •Robust methods and tools•Quality standards and procedures •Mechanisms to ensure utilization and influence

The right kind of information is frequently missing

• Conflicting demands of PI, wrong incentives, and low M&E capacities often lead to

- Lack of clarity on the role of PI for budgeting

- Duplication and overspending

- Not standardized methods and tools (limitations and strengths)

- Low quality and scarce use

M&E systems help tackle performance information problems

• Ensuring supply of performance data (records, stats, balanced M&E data, etc.)

• Stimulating demand, (use in policy, transparency, accountability, etc)

• Guaranteeing quality and credibility of data (robust methodologies, standards)

• But also increasing access to PI (integrated IT systems, reports, etc).

Country-level M&E systems are powerful budgetary tools

• Facilitate budget shift from traditional management (input/ activity-based) to performance management (output/outcome-based)

• Emphasize effective resource allocation for planning and budgeting

• Help countries assess if using scarce resources cost-effectively

• Respond to elected officials and the public’s demands for accountability in the budget

• Enhance the need for managers in the public sector

• Stress knowledge and learning for improving budget decision-making

• NO Performance Budgeting possible without systematic and good quality M&E information.

• NO effective M&E without links with the budget process

Development of PB and M&E systems need to go hand in hand

Bottom line of the M&E and PB relationship

Overview




► Good practice country cases


United Kingdom: The Public Service Agreements System

Main architecture features:

- High level Strategy (PMSU) and Delivery Units (PMDU)

- Ministerial public agreements on services to be delivered

- Targets, indicators and managers for each delivery agreement

- Ministries’ PSA Delivery Board (chaired by managers)

- Regular performance monitoring (PMDU, Cabinet)

- Dif evaluation types (expending reviews, program, impact, audits)

- Public dissemination of strategy and agreements •H.M. Treasury key player. Prime Minister’s (PMSU) and (PMDU) provide advise and regular updates directly to the Prime Minister.

United Kingdom: The Public Service Agreements System

Strengths

• “Explains what departments plan to deliver in return for…significant extra investment”

• “[Sets out] demanding national targets” (Nt. Standards)

• Multiple agencies but H.M. Treasury guidelines allow coordination

• Highly credible evaluations (externally contracted)

• “[Reflects] the Government’s key priorities and [focus] on the outcomes that matter most to the public”

• Establishes responsible “managers” of each PS

• Leading role of Treasury and Prime Minister’s office

Mexico: The Performance Evaluation System – SED


- Independent Evaluation Unit: CONEVAL council composed of academics - Single Federal Evaluation Guidelines - “Logic Frameworks” and standards for all F. Programs - 258 Performance indicators (output and outcome based) - 6 evaluation methods: Impact, consistency, indicators, process, design, perf. - Annual Evaluation Plan (More than 100 annually).

• The MoF is the system’s “champion” with active support and participation of the President’s Office.

Mexico: The Performance Evaluation System – SED

Strengths

• CONEVAL Model brings credibility

- Contracts out evaluation (market of evaluation) - increases quality, regularity, and timeliness of evaluation work- Promotes capacity development

• Unique poverty measurement schema as part of the system• Robust and credible evaluations, strengthening monitoring• All evaluations reported to Congress (with significant budget initiative)

• Powerful role of MoF and Presidency increases utilization• Active role of Ministries and agencies in defining evaluation agenda

Chile: The Management Control System


- Ex ante cost-benefit analysis for investment programs

- 1,550 performance indicators for all government programs (2006)

- “Program evaluations”: desk reviews of log frame (14 each year)

- Rigorous impact evaluations (4 each year)

- Comprehensive Spending Reviews: selective programs (2-5 each year)

- Agency management improvement programs linked to staff payments

• The MoF (Budget Directorate) designed and uses the system. Congress has no spending initiative.

Chile: The Management Control System

Strengths

• Performance information used to identify need for evaluations

• Evaluations conducted externally, in fully transparent process, and are highly credible

• All M&E findings reported publicly and sent to Congress

• M&E system closely linked to the budget information needs of MoF

• Performance information used to set targets for ministries – these are largely met

• MoF closely monitors extent of utilization of evaluation findings

Overview




► Good practice country cases


Main risks and difficulties

1. Deficient budget planning - No technical definition of program objectives, goals and indicators

2. Weak system design, unclear institutional arrangements

- How much “M” & “E” is enough? , who plan, conduct ,and use of evaluations?

- Self-evaluations or independent evaluations?

- Who funds the system? (a) earmarked budget funds?; (b) costs “absorbed” by ministries? How it links with the budget?

3. On supply side: caveat multiple, competing, uncoordinated systems

4. On demand side: Budget “rigidity”, incentives are key to achieve utilization

5. Low staff capacities (needs PS managers)

No strong preconditions but “Success” factors include

1. High Gov. commitment with quality of public spending

2. Powerful “champion”, ideally centrally driven by capable MoF Ministry

3.Careful attention to both demand and supply sides

4.Sustained, continuous effort

5. Piecemeal to ensure assimilation

6.“Quick gains” to avoid the costs of a too timid implementation

7. Avoiding complex systems (keep it simple)

Useful resource material

Keith Mackay, How to Build M&E Systems for Better Government, World Bank, 2007. http://www.worldbank.org/ieg/ecd/better_government.html

Ernesto May et al. (eds.), Towards the Institutionalization of Monitoring and Evaluation Systems in Latin America and the Caribbean, World Bank, 2006. http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,contentMDK:20893139~pagePK:146736~piPK:146830~theSitePK:258554,00.html

Manuel F. Castro, Insider Insights: Building a Results-Based Management and Evaluation System in Colombia. World Bank, 2008. http://www.worldbank.org/ieg/ecd/docs/ecd_wp_18.pdf

Gonzalo Hernandez Licona, The Role of Evaluation in Mexico: Achievements, Challenges and Opportunities. Quality of Public Expenditure. Note No. 2. World bank, 2008. http://siteresources.worldbank.org/MEXICOEXTN/Resources/MXNewsletter-QualityofPubExpenditure-No2.pdf

Useful resource material

Teresa Curristine, Performance Information in the Budget Process: Results of the OECD 2005 Questionnaire, in OECD Journal on Budgeting, 2005 5(2),pp.87-131. http://www.oecd.org/document/38/0,3343,en_2649_34119_39921702_1_1_1_1,00.html

OECD, Performance Budgeting in OECD Countries, OECD, 2007. http://www.oecd.org/document/38/0,3343,en_2649_37405_39921702_1_1_1_37405,00.html#B2

Independent Evaluation Group, M&E Tools, Methods and Approaches, World Bank, 2004. http://www.worldbank.org/ieg/ecd/me_tools_and_approaches.html

Independent Evaluation Group, Influential Evaluations: Evaluations that Improved Performance and Impacts of Development Programs, World Bank, 2004.http://www.worldbank.org/ieg/ecd/influential_evaluations.html

World Bank website on Building Government M&E Systems: www.worldbank.org/ieg/ecd/

manuel fernando castro the role of m&e systems in performance budgeting: good-practice and...

Documents

international experience

form of budgeting

uk lessons

mexico chile

experience warsaw

pb strategic context

public spending

higher economic growth