what can we learn from successful country systems? philipp krause prmpr 1
Post on 21-Dec-2015
213 views
TRANSCRIPT
1
WHAT CAN WE LEARN FROM SUCCESSFUL COUNTRY SYSTEMS?
PHILIPP KRAUSEPRMPR
2
Overview
Why bother? A selection of country cases How do differences matter? What do “successful” systems have in
common?
3
Why bother?
Two schools of thought The “best practice” school The “It’s all about the context” school
The truth lies somewhere in between – there are important characteristics shared by successful cases, but differences between countries matter tremendously
Most important reason: Countries do M&E for different reasons and different users – what you need depends on what you want to do with it
Our definition of success: High degree of utilization M&E information meets quality standards and is reliable System is sustainable over time
4
Overview
Why bother? A selection of country cases How do differences matter? What do “successful” systems have in
common?
5
Australia: Main Features
Mandate to evaluate each program every 3-5 years
Portfolio Evaluation Plans to be prepared annually for the following three years
Department of Finance: Steering & quality control Sector Departments: planning, implementation Evaluation results primarily used for budgetary
decisions: allocations of funds for new policies and reallocation of savings (i.e. the discretionary part of annual budget process)
System lasted from 1987-1996 – sustainability?
6
Australia: How does it work?
Sector Departm
ents(and outrider
agencies)
Treasury
Department of Prime
Minister and Cabinet
Department of
Finance
Parliament
Citizens
Inputs to PEPs
Inputs to PEPs
Inputs to Portfolio Evaluation Plans (PEPs)
Formal notification of
PEPsInvolvement in evaluations
Publication of department’s
evaluation reports
Reporting of keyEvaluation findings in department’s budget paper (prospective) and
in their annual reports (retrospective)
Source: Mackay 2011
7
United Kingdom: Main Features Comprehensive system of performance monitoring
and targets: Spending Reviews (multiannual budgets) and Public Service Agreements (“resources for delivery”) in return for (relative) managerial flexibility
Oversight and leadership from central executive (Prime Minister, PM’s Delivery Unit, Treasury)
No systematic evaluations in the executive government
Problem: “Gaming in Targetworld” Value-for-Money Audits: 60 per year from National
Audit Office
8
Delivery Unit*
UK: How does it work?
HM Treasury
Parliament
audits
Prime Minister’s Office is accountable to
reports to
National Audit Office
Ministers
Spending Unit
monitors priority areas
reporting/monitoring
is accountable to
Negotiations over PSAs
* Moved to HM Treasury in 2007, abolished 2010
9
Mexico: Main Features
M&E of social policies delegated to a specialized technical agency – CONEVAL
CONEVAL in charge of evaluation portfolio, development of methods, dissemination
Key oversight decisions by inter-ministerial committees – system involves many stakeholders (Congress, Finance, Presidency, Public Admin Ministry, Sector Ministries, CONEVAL)
Implementation of evaluation and day-to-day operation of monitoring done by ministries
10
Mexico: How does it work?
Inter-ministerial Committee
SectorMinistr
y
CONEVAL
SectorMinistr
y
SectorMinistr
y
CONEVAL
•Define the evaluation plan•Decide what and how to evaluate
•Coordinate and steer evaluation system•Maintain quality control•Define monitoring parameters
•Select and hire evaluators•Supervise evaluation implementation•Monitor indicators and targetsPrograms are evaluated
by ministries •Collection of evaluation results•Dissemination of evaluation results
•Utilization of evaluation results
1.
2.
3.
4.
5.Congress, Presidency, Sector Ministries, Finance
11
Chile: Main Features
System is highly centralized – closely linked to and based on a highly centralized, top-down budget process
Budget office manages most details, and is main user of information – utilization fostered by close link to budget process
Management control by hierarchical oversight Little buy-in from other (potential) stakeholders Some impact on allocations and program management
Chile: Utilization of Government Evaluations—2000–05Effect onprogram
Minor adjustmentof program, forexample, improvedprocesses orinformation systems
Major adjustmentof managementprocesses, for example,changed targetingcriteria, or newinformation systems
Substantial redesignof program ororganizational structure
Institutional relocationof program
Programtermination
Programsaffected
24% 38% 25% 5% 8%Percentage of all evaluated programs. Source: Guzman 2007
12
Chile: How does it work?12
Budget Office
Budget Office
Budget Office
Programs
Budget Office
•Define the evaluation plan•Decide what and how to evaluate
•Coordinate and steer evaluation system•Maintain quality control•Define monitoring parameters
Programs are evaluated directly by Budget Office
•Select and hire evaluators•Supervise evaluation implementation•Monitor indicators and targets
•Collection of evaluation results•Dissemination of evaluation results
•Utilization of evaluation results
1.
2.
3.
4.
5.
Budget Office
Congress
13
Centralization vs. Delegation
Inter-ministerial Committee
Mexico
SectorMinistr
y
CONEVAL
SectorMinistr
y
SectorMinistr
y
Chile
Budget Office
Budget Office
Budget Office
Programs
Budget Office
CONEVAL
Congress, Presidency, Sector Ministries, Finance
•Define the evaluation plan•Decide what and how to evaluate
•Coordinate and steer evaluation system•Maintain quality control•Define monitoring parameters
Programs are evaluated directly by Budget Office
•Select and hire evaluators•Supervise evaluation implementation•Monitor indicators and targets
Programs are evaluated by ministries •Collection of evaluation results
•Dissemination of evaluation results
•Utilization of evaluation results
Budget Office
Congress
1.
2.
3.
4.
5.
14
Overview
Why bother? A selection of country cases How do differences matter? What do “successful” systems have in
common?
Institutional Differences Matter: Index of
Legislative Budgetary Powers
Chile
Mexico
Source: Wehner 2007
16
Difference in Purpose
Budgetary: To inform budgetary decisions, best allocation of resources between sectors and programs, also to enforce operational savings in annual budget
Accountability: External accountability towards legislature, stakeholders and public. To make systematic information on performance available and strengthen the public evidence base of policy decisions
Control: To develop better central government information on implementation and service delivery of public programs as tools to hold managers to account
17
Matrix of Country Comparison
Country Purpose Leadership Operation Users
Chile Budget/Control Direct Centralized Single
Mexico Budget/Accountability Delegated Decentralized Multiple
Australia Budget Direct Decentralized Single
Canada Budget Direct Decentralized Single
UK/NAO* Accountability Direct Centralized Multiple
* Supreme Auditor
18
Features that Matter
Centralization requires the right institutional structure – a centralized M&E system design in a fragmented public sector will fail
In a system with multiple stakeholders delegation to an impartial agency might be a viable option – but beware of over-engineering and objective overload
Who is to gain and who has to worry about buying into a M&E system: The senior civil service, the legislature, ministries, service delivery units, the finance ministry, the head of government (PM or President)?
Staying in control of overall steering and quality of outputs does not equal having to internalize all aspects of M&E implementation – strategic delegation might be smart for buy-in and workload
Is it possible to imagine a long-term sustainable, well utilized M&E system that does not have a stable link to budgetary decisions?
19
Overview
Why bother? A selection of country cases How do differences matter? What do “successful” systems have
in common?
20
Lessons Start M&E Systems Successfully* (1)
Somewhere in government is substantive demand for M&E information. This is necessary to start and sustain an M&E system
Actors need to have incentives to engage with M&E. They are key for M&E to be conducted and for the information produced to be utilized
Simple is better – successful M&E systems tend to deliver just what users want, not more. They also serve only those objectives that result in utilization
Success is more likely with a powerful champion(s) to lead the push for institutionalization of M&E – rather than a legislative or technical exercise
*very liberally adapted from Mackay 2010
21
Lessons Start M&E Systems Successfully (2)
It is important to have the stewardship of a central, capable ministry that can design, develop, and manage the system
Some reforms may start with a bang, but it requires patience, determination, and a long-term effort to build an effective M&E system
For donors: It helps the process to start with a diagnosis of what M&E functions already exist in the country (and why other M&E functions do not exist – they usually don’t for a reason)
22
THANKS!
For further information, and to access the sources cited here, please visit:
http://go.worldbank.org/7MZRWD6K50