11

Presentation to a LAC PREM BBL11 January 2007

Keith MackayIndependent Evaluation Group

(kmckay@worldbank.org)

22

Structure of PresentationStructure of Presentation

1.1. What does ‘success’ look like?What does ‘success’ look like?

2.2. Chile Chile

3.3. ColombiaColombia

4.4. AustraliaAustralia

5.5. Definition of ‘good practice’ M&E systemDefinition of ‘good practice’ M&E system

6.6. LessonsLessons

7.7. Useful resource materialsUseful resource materials

33

1. What Does “Success” Look Like?1. What Does “Success” Look Like?

-- Why Countries Want an M&E -- Why Countries Want an M&E System System

To support budget decision-making To support budget decision-making = performance-based budgeting= performance-based budgeting

To support national and sectoral planningTo support national and sectoral planning

To design policies and programsTo design policies and programs

To assist sector ministries / agencies in their To assist sector ministries / agencies in their managementmanagement

To strengthen accountability relationshipsTo strengthen accountability relationships

44

2. Chile’s M&E System 2. Chile’s M&E System -- Architecture -- Architecture

Designed, managed and used by HaciendaDesigned, managed and used by Hacienda

Developed incrementally, over past decadeDeveloped incrementally, over past decade

Performance indicators (Performance indicators (∑∑1,600) for all 1,600) for all government programs (1994)government programs (1994)

Government program evaluations (Government program evaluations (∑ ∑ 185) – 185) – these are desk reviews (1996)these are desk reviews (1996)

Rigorous impact evaluations (Rigorous impact evaluations (∑∑18) (2001)18) (2001)

Comprehensive Spending Reviews (Comprehensive Spending Reviews (∑∑8) – 8) – desk reviews of all programs in a functional desk reviews of all programs in a functional area (2002)area (2002)

55

Chile’s M&E System Chile’s M&E System -- Strengths (i) -- Strengths (i)

‘‘Graduated’ approach to M&EGraduated’ approach to M&E

Evaluations conducted externally, in fully Evaluations conducted externally, in fully transparent process, and are highly credibletransparent process, and are highly credible

All M&E findings reported publicly and sent All M&E findings reported publicly and sent to Congressto Congress

M&E system closely linked to the budget M&E system closely linked to the budget information needs of Haciendainformation needs of Hacienda

Performance information used to set targets Performance information used to set targets for ministries – these are largely metfor ministries – these are largely met

Hacienda closely monitors extent of Hacienda closely monitors extent of utilization of evaluation findingsutilization of evaluation findings

66

Chile’s M&E System Chile’s M&E System -- Strengths (ii) -- Strengths (ii)

High utilization of M&E findings by High utilization of M&E findings by Hacienda in the budget process and to Hacienda in the budget process and to impose management improvements on impose management improvements on ministries / agencies (see Table) ministries / agencies (see Table)

Utilization of government evaluations -- 2000 to 2005Utilization of government evaluations -- 2000 to 2005

Minor Minor adjustment adjustment of program, of program, e.g. e.g. improved improved info systeminfo system

Major change Major change in mgment in mgment processes e.g. processes e.g. new targeting new targeting criteria, new criteria, new MISMIS

Substantial Substantial redesign of redesign of program or program or of organiz-of organiz-ational ational structurestructure

Institutional Institutional relocation relocation of programof program

Program Program terminationtermination

TOTALTOTAL

24%24% 38%38% 24%24% 5%5% 9%9% 100%100%

77

Chile’s M&E System Chile’s M&E System -- Weaknesses -- Weaknesses

Unevenness in quality of evaluations – due Unevenness in quality of evaluations – due to cost and time constraintsto cost and time constraints

Chile probably not spending enough on Chile probably not spending enough on evaluationsevaluations

Low utilization – low ‘ownership’ – of Low utilization – low ‘ownership’ – of Hacienda’s evaluations by sector ministriesHacienda’s evaluations by sector ministries

88

3. Colombia’s M&E System 3. Colombia’s M&E System (SINERGIA)(SINERGIA) -- Architecture -- Architecture

SINERGIA is managed by the Department of SINERGIA is managed by the Department of National Planning (DNP), with strong support National Planning (DNP), with strong support from the Presidenciafrom the Presidencia

On-line sub-system – SIGOB – for monitoring On-line sub-system – SIGOB – for monitoring and reporting government progress vis-à-vis and reporting government progress vis-à-vis Presidential GoalsPresidential Goals

Ambitious agenda of impact and other Ambitious agenda of impact and other evaluations (evaluations (∑∑15 underway) 15 underway)

DNP provides technical assistance to a few DNP provides technical assistance to a few ministries/agencies to develop M&E, and to ministries/agencies to develop M&E, and to municipalities to pilot SIGOB and municipalities to pilot SIGOB and performance-based budgetingperformance-based budgeting

99

Colombia’s M&E System Colombia’s M&E System -- Strengths-- Strengths

Very high utilization of SIGOB by President Very high utilization of SIGOB by President for oversight of ministers and ministries – for oversight of ministers and ministries – via performance targets – and for via performance targets – and for accountability, i.e. ‘social control’accountability, i.e. ‘social control’

Rigorous impact evaluations conducted Rigorous impact evaluations conducted externally, and have high credibilityexternally, and have high credibility

Collaborative approach between DNP and Collaborative approach between DNP and sector ministries/ agencies, and with sector ministries/ agencies, and with municipalitiesmunicipalities

1010

Colombia’s M&E System Colombia’s M&E System -- Weaknesses-- Weaknesses

Insufficient reliance on M&E information to Insufficient reliance on M&E information to support national planning and budget support national planning and budget decision-making – this may now be about to decision-making – this may now be about to changechange

Weak coordination of M&E roles / functions Weak coordination of M&E roles / functions within DNP and with central and sector within DNP and with central and sector ministriesministries

SIGOB data quality perceived as lowSIGOB data quality perceived as low

Too high reliance on donor funding for Too high reliance on donor funding for SINERGIA –SINERGIA – low level of government funding low level of government funding supportsupport

1111

4. Australia’s M&E System 4. Australia’s M&E System (1987-1997) -- Architecture(1987-1997) -- Architecture

System was managed by Finance System was managed by Finance Department – emphasis on evaluationDepartment – emphasis on evaluation

Evaluations mandatory – every 3-5 years for Evaluations mandatory – every 3-5 years for every programevery program

Sector departments required to prepare Sector departments required to prepare rolling, 3-year plans for major evaluationsrolling, 3-year plans for major evaluations

Broad range of evaluation typesBroad range of evaluation types

By mid-1990s, By mid-1990s, ∑∑160 underway at any point 160 underway at any point in timein time

Much less emphasis on performance infoMuch less emphasis on performance info

1212

Australia’s M&E System (1987-1997) Australia’s M&E System (1987-1997) -- Strengths-- Strengths

Evaluation findings heavily used in budget Evaluation findings heavily used in budget analysis, policy advice, and by the Cabinet analysis, policy advice, and by the Cabinet in its budget decision-makingin its budget decision-making

High utilization of evaluation findings by High utilization of evaluation findings by sector departments and agenciessector departments and agencies

Evaluation was a collaborative endeavor Evaluation was a collaborative endeavor between Finance Department, other central between Finance Department, other central departments, and sector departmentsdepartments, and sector departments

1313

Australia’s M&E System (1987-1997) Australia’s M&E System (1987-1997) -- Weaknesses-- Weaknesses

Uneven quality of evaluationsUneven quality of evaluations

Insufficient attention to regular performance Insufficient attention to regular performance informationinformation

An administrative burden on departments An administrative burden on departments was claimedwas claimed

1414

5. How to Define a Government M&E 5. How to Define a Government M&E System as Being “Good Practice”System as Being “Good Practice”

Can be dangerous concept – each country’s Can be dangerous concept – each country’s starting point and desired end-point are starting point and desired end-point are unique unique

What a “successful” M&E system is NOT:What a “successful” M&E system is NOT: complex set of laws, decrees, regulationscomplex set of laws, decrees, regulations # performance indicators collected# performance indicators collected # rigorous impact evaluations conducted# rigorous impact evaluations conducted # rapid evaluations conducted# rapid evaluations conducted = these simply reflect the architecture of = these simply reflect the architecture of

the system or are measures of M&E effortthe system or are measures of M&E effort

1515

How to Define a Government M&E How to Define a Government M&E System as Being “Good Practice”System as Being “Good Practice”

Chile, Colombia and Australia have good-Chile, Colombia and Australia have good-practice M&E systems becausepractice M&E systems because

1.1. the quality of their M&E work is generally the quality of their M&E work is generally reliable, andreliable, and

2.2. in particular, the monitoring information in particular, the monitoring information and evaluation findings which they and evaluation findings which they produce are used intensivelyproduce are used intensively

High utilization reflects strong demand for High utilization reflects strong demand for M&E, and can be viewed as a predictor of M&E, and can be viewed as a predictor of an M&E system’s sustainabilityan M&E system’s sustainability

1616

6. Lessons (i)6. Lessons (i)

Lessons from the 3 countries are consistent Lessons from the 3 countries are consistent with other countries’ experiencewith other countries’ experience

1.1. Key role of powerful champion of M&EKey role of powerful champion of M&E

2.2. Opportunistic development of M&E systems, Opportunistic development of M&E systems, via continuous review and modification; non-via continuous review and modification; non-linear development of the systemslinear development of the systems

3.3. Centrally-driven, by capable ministryCentrally-driven, by capable ministry

4.4. Incentives are key – cultural change to Incentives are key – cultural change to strengthen demand, achieve high utilizationstrengthen demand, achieve high utilization

5.5. Avoid competing systems – Planning, Finance, Avoid competing systems – Planning, Finance, Presidencia Presidencia

1717

Lessons (ii)Lessons (ii)

6.6. Build reliable ministry data systemsBuild reliable ministry data systems

7.7. Role of structural arrangements to ensure Role of structural arrangements to ensure M&E objectivity and qualityM&E objectivity and quality

8.8. Long-haul effort, requiring patienceLong-haul effort, requiring patience

9.9. Limitations of relying on laws, decrees, Limitations of relying on laws, decrees, regulationsregulations

10.10. An M&E system can be built and operated An M&E system can be built and operated are relatively low costare relatively low cost

Chile’s M&E system costs $0.75m p.a.Chile’s M&E system costs $0.75m p.a. Colombia’s system costs about $2m p.a.Colombia’s system costs about $2m p.a.

1818

7. Useful Resource Materials (i)7. Useful Resource Materials (i) Ernesto May et al. (eds.), Towards the Institutionalization of

Monitoring and Evaluation Systems in Latin America and the Caribbean, World Bank/IADB, 2006. http://web.worldbank.org/WBSITE/EXTERNAL/COUNTRIES/LACEXT/0,,contentMDK:20893139~pagePK:146736~piPK:146830~theSitePK:258554,00.html (Disponible en Español)

Fernando Rojas et al, Chile: Análisis del Programa de Evaluación del Gasto Publico, World Bank, 2005. http://iris37.worldbank.org/domdoc/PRD/Other/PRDDContainer.nsf/WB_ViewAttachments?ReadForm&ID=85256D2400766CC785257155005CB26B&

World Bank, A Diagnosis of Colombia’s National M&E System, SINERGIA, World Bank, 2007 (forthcoming).

Keith Mackay, Two Generations of Performance Evaluation and Management System in Australia, World Bank, 2004. http://lnweb18.worldbank.org/oed/oeddoclib.nsf/DocUNIDViewForJavaSearch/14163969A1A709BD85256E5100013AA8/$file/ecd_wp_11.pdf

1919

Useful Resource Materials (ii)Useful Resource Materials (ii)

World Bank website on Building Government M&E Systems: www.worldbank.org/ieg/ecd/

Chile: www.dipres.cl/fr_control.html

Colombia: www.dnp.gov.co/paginas_detalle.aspx?idp=266

Keith Mackay, Institutionalization of Monitoring and Evaluation Systems to Improve Public Sector Management, World Bank, 2006. www.worldbank.org/ieg/ecd/institutionalizing_me.html (Disponible en Español)

Teresa Curristine, Performance Information in the Budget Process: Results of the OECD 2005 Questionnaire, OECD Journal on Budgeting 5(2), 2005, pp 87-131. http://www.oecd.org/dataoecd/61/24/37033986.pdf