world bank document - documents & reports - all...

Document of

The World Bank

Report No: ICR00003027

IMPLEMENTATION COMPLETION AND RESULTS REPORT

(IBRD-76220)

ON A

LOAN

IN THE AMOUNT OF US$4.4 MILLION

TO THE

UNITED MEXICAN STATES

FOR A

RESULTS-BASED MANAGEMENT AND BUDGETING PROJECT

June 30, 2014

Poverty Reduction and Economic Management

Colombia and Mexico Country Management Unit

Latin America and the Caribbean Region

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

CURRENCY EQUIVALENTS (Exchange Rate Effective April 10, 2014)

Currency Unit=Mexican Pesos

US$ 1.00 = MXP $13.05

MXP $1 = US$ 0.076

FISCAL YEAR January 1 – December 31

ABBREVIATIONS AND ACRONYMS

CONEVAL National Council for the Evaluation of Social Development Policy (Consejo

Nacional de Evaluación de la Política de Desarrollo Social)

GoM Government of Mexico

IADB Inter-American Development Bank

IEG Independent Evaluation Group

IFAI Federal Institute for Access to Public Information and Data Protection (Instituto

Federal de Acceso a la Información y Protección de Datos)

IT Information Technology

M&E Monitoring and Evaluation

NAFIN Development Banking Institution (Financial Agent: Nacional Financiera, S.N.C.)

NDP/PND National Development Plan (Plan Nacional de Desarollo)

OECD

PBR

Organization for Economic Co-operation and Development

Results Based-Budgeting (Presupuesto Basado en Resultados)

PFM

PGCM

Public Financial Management

Close and Modern Government Program (Programa de Gobierno Cercano y

Moderno)

PI

PMG

Performance Information

Management Improvement Program (Programa de Mejora de la Gestión)

PSBR Public Sector Borrowing Requirements

RBB Results-based budgeting

SE Undersecretary of Expenditures (Subsecretaría de Egresos)

SED Performance Evaluation System (Sistema de Evaluacion del Desempeno)

SFP Ministry of Public Administration (Secretaría de la Función Pública)

SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito

Público)

SIDAFF

SIGAIF

Integrated Federal Financial Management System (Sistema Integral de

Administración Financiera Federal)

Harmonized Governmental Financial Information System (Sistema

Gubernamental Armonizado de Información Financiera)

SISED Performance Evaluation Program Information System (Sistema de Información

para el Sistema de Evaluación del Desempeño)

Vice President: Jorge Familiar

Country Director: Gloria M. Grandolini

Sector Manager: Arturo Herrera Gutierrez

Project Team Leader: Pedro Arizti

ICR Team Leader: Pedro Arizti

MEXICO

Results-based Management and Budgeting Project

CONTENTS

Data Sheet

A. Basic Information

B. Key Dates

C. Ratings Summary

D. Sector and Theme Codes

E. Bank Staff

F. Results Framework Analysis

G. Ratings of Project Performance in ISRs

H. Restructuring

I. Disbursement Graph

1. Project Context, Development Objectives and Design ............................................... 1

2. Key Factors Affecting Implementation and Outcomes .............................................. 6

3. Assessment of Outcomes .......................................................................................... 11

4. Assessment of Risk to Development Outcome ......................................................... 15

5. Assessment of Bank and Borrower Performance ..................................................... 15

6. Lessons Learned ....................................................................................................... 18

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners .......... 19

Annex 1. Project Costs and Financing .......................................................................... 22

Annex 2. Outputs by Component ................................................................................. 23

Annex 3. Economic and Financial Analysis ................................................................. 26

Annex 4. Bank Lending and Implementation Support/Supervision Processes ............ 28

Annex 5. Beneficiary Survey Results ........................................................................... 30

Annex 6. Stakeholder Workshop Report and Results ................................................... 31

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 32

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ....................... 33

Annex 9. List of Supporting Documents ...................................................................... 34

MAP

A. Basic Information

Country: Mexico Project Name:

MX Results-based

Management and

Budgeting

Project ID: P106528 L/C/TF Number(s): IBRD-76220

ICR Date: 06/30/2014 ICR Type: Core ICR

Lending Instrument: TAL Borrower: GOVERNMENT OF

MEXICO

Original Total

Commitment: US$ 17.2M Disbursed Amount: US$ 0.37M

Revised Amount: US$ 4.4M

Environmental Category: C

Implementing Agencies:

Secretaría de Hacienda y Crédito Publico (SHCP)

Cofinanciers and Other External Partners:

B. Key Dates

Process Date Process Original Date Revised / Actual

Date(s)

Concept Review: 09/20/2007 Effectiveness: 03/19/2010 03/19/2010

Appraisal: 10/02/2008 Restructuring(s): 11/15/2012

Approval: 12/11/2008 Mid-term Review: 02/06/2012

Closing: 12/31/2013 12/31/2013

C. Ratings Summary

C.1 Performance Rating by ICR

Outcomes: Moderately Unsatisfactory

Risk to Development Outcome: Moderate

Bank Performance: Moderately Unsatisfactory

Borrower Performance: Moderately Unsatisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)

Bank Ratings Borrower Ratings

Quality at Entry: Moderately

Unsatisfactory Government:

Moderately

Unsatisfactory

Quality of Supervision: Moderately

Unsatisfactory

Implementing

Agency/Agencies:

Moderately

Unsatisfactory

Overall Bank

Performance:

Moderately

Unsatisfactory Overall Borrower

Performance:

Moderately

Unsatisfactory

C.3 Quality at Entry and Implementation Performance Indicators

Implementation

Performance Indicators

QAG Assessments

(if any) Rating

Potential Problem Project

at any time (Yes/No): No

Quality at Entry

(QEA): None

Problem Project at any

time (Yes/No): Yes

Quality of

Supervision (QSA): None

DO rating before

Closing/Inactive status:

Moderately

Unsatisfactory

D. Sector and Theme Codes

Original Actual

Sector Code (as % of total Bank financing)

Central government administration 100 100

Theme Code (as % of total Bank financing)

Managing for development results 50 50

Public expenditure, financial management and

procurement 50 50

E. Bank Staff

Positions At ICR At Approval

Vice President: Jorge Familiar Pamela Cox

Country Director: Gloria M. Grandolini Axel van Trotsenburg

Sector Manager: Arturo Herrera Gutierrez Nicholas Paul Manning

Project Team Leader: Pedro Arizti Roberto Adrian Senderowitsch

ICR Team Leader: Pedro Arizti

ICR Primary Author: Azul Del Villar

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) The development objective of the project is to establish that federal departments and agencies in

Mexico provide decision makers and the public with rigorous, timely, user-friendly information

on the efficiency and effectiveness of government organizations and program expenditures.

Revised Project Development Objectives (as approved by original approving authority) The objective of the project is to strengthen the Ministry of Finance’s (Secretaría de Hacienda y

Crédito Público, SHCP) capacity to (i) allow the use of standardized performance information of

priority public programs during budget preparation; (ii) generate harmonized subnational fiscal

information; and (iii) disseminate budget performance information to the Borrower's citizens on a

regular basis.

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target

Values (from

approval

documents)

Formally

Revised

Target

Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 : Performance information using a performance framework is generated and used

during budget request discussions between line secretariats and SHCP.

Value

(quantitative or

qualitative)

Performance is not widely

used by line secretariats

in their budget requests.

Budget preparation

discussions are mainly

focused on inputs and are

not based on standardized

performance information

that allow to measure

relevance, efficiency of

programs.

Performance

information

(PI) is used by

SHCP during

budget

preparation in

conversation

with line

ministries.

SHCP

uses

standardized

performance

information

that allow to

measure

relevance,

efficiency of

effectiveness

of programs.

Performance

information (PI)

presented in a

timely fashion to

Congress in a

format that allows

interpretation and

meets their needs.

Content includes

outcome and output

measures. Line

ministries send

budget requests to

SHCP with PI.

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Achieved

Indicator 2 : An active network and a portal to inform civil society at large on a regular basis

about budget performance increasing transparency and accountability to citizens.

Value

quantitative or

Qualitative)

Budget performance

information is readily

available to civil society

via the new portal but has

not been expanded to all

levels of Government.

There is a

portal

containing

financial and

non-financial

performance

information

about the

budget and its

process

available for

citizens, and

the

Portal containing PI

on budget

performance is

available to citizens

+ Citizen’s Budget

+ Citizen’s Public

Accounts (Cuenta

Pública)

information

has been

expanded to

all the levels

of

Government.

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Achieved, the portal site address is: www.transparenciapresupuestaria.gob.mx

Indicator 3 : Percent of public programs with improved strategic design and reporting

Value

quantitative or

Qualitative)

In 2008 the percentage of

public programs using

performance information

was 28%.

For the 2013

budget

preparation the

percentage of

programs

using

performance

information is

set to be over

75%.

The majority of the

programs have a

more robust design

across key

dimensions

(objectives,

indicators,

baselines, links to

strategies) enabling

better reporting and

evaluation. (832

programs out of

984)

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Achieved at 84.5%

Indicator 4 : Percent of municipalities that have a simplified financial management system

installed and working.

Value

quantitative or

Qualitative)

0

Around 80%

of

approximately

1,200 small

municipalities

(of total

approximately

2,500

municipalities)

The Accounting

Unit changed

priorities to focus

on developing the

Government

Accounting System

(SCG) first.

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

http://www.transparenciapresupuestaria.gob.mx/

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target

Values (from

approval

documents)

Formally

Revised

Target Values

Actual Value

Achieved at

Completion or

Target Years

Indicator 1 : Number of municipalities that have been trained in the use of the new municipal

simplified FMIS.

Value

(quantitative

or Qualitative)

Around 80%

of

approximately

1,200 small

municipalities

Pilot of

Harmonized

Governmental

Financial

Information System

(SIGAIF) installed

in some

municipalities, not

operational though.

The Accounting

Unit changed

priorities to focus

on developing the

Government

Accounting System

(SCG) first.

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

Indicator 2 : Functional design developed for the amplified FMIS that will cover the

decentralized institutions in the Government.

Value

(quantitative

or Qualitative)

Non existent

SIDAFF has a

module

designed for

decentralized

institutions.

The Accounting

Unit changed

priorities to focus

on developing the

Government

Accounting System

first.

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

Indicator 3 : Design of a conceptual model for integration of financial information in

municipalities, states and federation.

Value

(quantitative

or Qualitative)

Non existent

SIDAFF has a

module

designed for

integrating all

three levels of

Government.

The Accounting

Unit changed

priorities to focus

on developing the

Government

Accounting System

first.

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

Indicator 4 : Design and dissemination of a number of strategic evaluations (e.g. security

policy, water policy, education).

Value

(quantitative

or Qualitative)

Non existent

Strategic

social and

non-social

evaluations

carried out and

disseminated

through the

portal.

Three strategic

evaluations were

published in the

Budget

Transparency Portal

in April 2014.

Date achieved 10/03/2012 12/31/2013 04/30/2014

Comments

(incl. %

achievement)

Partially achieved.

Indicator 5 : Design of terms of reference for non-social program evaluations.

Value

(quantitative

or Qualitative)

Non existent

Terms of

Reference

(ToR) for non-

social program

evaluations

available.

SHCP has advised

on adapting ToR to

conduct program

evaluations other

than social, and on

revising the ToR of

the Consistency and

Results evaluations

as well as the Cost

Effectiveness for

non social programs

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

Indicator 6 : Development of a methodology and guidelines for better social program

focalization.

Value

(quantitative

or Qualitative)

Non existent

Methodology

and guidelines

for better

social program

focalization

available.

Not achieved

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

Indicator 7 : Public programs portfolio review to identify program delays and development of

methodology for ex ante screening.

Value

(quantitative

or Qualitative)

Non existent

Public

programs

portfolio

review carried

out and

methodology

for ex ante

screening

available.

Not achieved

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

Indicator 8 : Design and dissemination of an evaluation for the SED

Value

(quantitative

or Qualitative)

Non existent

SED

evaluation

carried out and

disseminated

through the

portal.

Important changes

will take place

fostered by the

PND, regarding the

SED’s scope (i.e.

PGCM)

Date achieved 10/03/2012 12/31/2013 12/31/2013

Comments

(incl. %

achievement)

Not achieved

G. Ratings of Project Performance in ISRs

No. Date ISR

Archived DO IP

Actual

Disbursements

(US$ millions)

1 01/27/2009 Satisfactory Satisfactory 0.00

2 09/18/2009 Moderately Satisfactory Moderately

Unsatisfactory 0.00


Unsatisfactory 0.00


Unsatisfactory 0.00


Unsatisfactory 0.00


Unsatisfactory 0.00


Unsatisfactory 0.00


Unsatisfactory 0.02


Unsatisfactory 0.02

10 12/07/2013 Moderately Unsatisfactory 0.21

Unsatisfactory

H. Restructuring (if any)

Restructuring

Date(s)

Board

Approved

PDO Change

ISR Ratings at

Restructuring

Amount

Disbursed at

Restructuring

in

US$ millions

Reason for Restructuring &

Key Changes Made DO IP

11/15/2012 Y MS MU 0.00

The restructuring aligned the

PDO to a reduced Project scope

and partial loan cancellation.

If PDO and/or Key Outcome Targets were formally revised (approved by the original approving

body) enter ratings below:

Outcome Ratings

Against Original PDO/Targets Moderately Unsatisfactory

Against Formally Revised PDO/Targets Moderately Unsatisfactory

Overall (weighted) rating Moderately Unsatisfactory

I. Disbursement Profile

1

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

1. During the 2004-2007 period, the Mexican economy experienced a balanced and

broad-based economic expansion with an average annual gross domestic product (GDP)

growth rate of 3.8 percent. A vigorous global economic expansion contributed to the

enhanced growth performance, as the value of exports increased at a double-digit annual

average rate. Price stability also contributed to a healthy growth of domestic demand,

with improved purchasing power of wages and salaries, and an expansion of domestic

credit and consumer lending.

2. In this context, the fiscal reform approved in September 2007 marked a crucial

step in strengthening the country’s fiscal accounts. This fiscal reform enabled the

Government of Mexico (GoM) to raise tax collection rates, strengthen fiscal federalism,

and decrease tax evasion. Building on the improved macroeconomic and fiscal

framework, the GoM focused on enhancing the quality of public expenditures throughout

the budget cycle, from planning to execution and evaluation. As a result, the Government

established a results-based budgeting initiative, the Performance Evaluation System

(Sistema de Evaluación del Desempeño, SED), for which it sought Bank support.

3. Furthermore, the 2008-2013 Country Partnership Strategy (CPS) for Mexico

called for strengthening institutions to improve their functioning and enhancing citizen

perceptions of the public sector through a set of initiatives that included a move towards

results-based budgeting. The higher-level objective of the Project—improving the

quality of public expenditures in Mexico—was therefore fully consistent with the FY08-

13 CPS. The Project was also aligned with two of the primary objectives of the 2007-

2012 National Development Plan (NDP), which aimed to “improve regulation,

management, the processes and results of Federal Public Administration to satisfy citizen

needs in relation to the provision of public goods and services;” and “promote and

guarantee transparency, accountability, access to information, and the protection of

personal information in all spheres of government.”1

1.2 Original Project Development Objectives (PDO) and Key Indicators

4. The original objective of the Project was to strengthen the Borrower’s capacity

(through its federal departments and agencies) to provide decision makers and the public

with rigorous, timely, user-friendly information on the efficiency and effectiveness of

government organizations and their program expenditures.2

1 Mexico Country Partnership Strategy FY 2008-2013, March 4, 2008, Report No. 42846-MX.

2There is a slight difference between the Original PDO in the PAD and the Loan Agreement (LA). The one used in the

ICR is that of the LA.

2

5. The original PDO indicators were: (i) Performance information presented in

budget requests from line ministries to SHCP and in budget submitted to Congress; (ii)

Satisfaction of Congress and civil society organizations with the performance information

made available via government websites (as measured through user surveys); (iii) Percent

of programs with improved strategic design and reporting3

; (iv) Progression of

organizations through the steps (levels) established for each Management Improvement

Program (Programa de Mejora de la Gestión, PMG), marking increased levels of

administrative capacity and efficiency.

1.3 Revised PDO (as approved by original approving authority) and Key Indicators,

and reasons/justification

6. The Project had to be restructured (level 1) in November 2012 as a result of

changes in national priorities and most importantly, due to the fact that some activities

had been financed and implemented using local funds, in light of the loan’s start up

delays and lengthy process towards effectiveness. Therefore, the Government and the

Bank agreed on reducing the Project scope, partially cancelling the loan and redesigning

activities to better align them with Government priorities.

7. In order to align them with a reduced Project scope and partial loan cancellation,

the PDO and its key indicators and the results framework were revised. The revised PDO

is to strengthen SHCP’s capacity to: (i) allow the use of standardized performance

information of priority public programs during budget preparation; (ii) generate

harmonized sub-national fiscal information; and (iii) disseminate budget performance

information to the Borrower’s citizens on a regular basis. The table below illustrates the

change in key indicators.

Table 1. Changes in PDO indicators

Original PDO-Level Indicator

(PAD 2008)

Revised PDO-Level Indicator (restructuring

2012)

1. Performance information presented in budget

requests from line ministries to SHCP and in

budget submitted to Congress.

(Revised) Performance information using a

performance framework is generated and used

during budget request discussions between line

secretariats and SHCP.

2. Satisfaction of Congress and civil society

organizations with the performance

information made available via government

websites (as measured through user surveys).

(Revised) An active network and a portal to

inform society at large on a regular basis about

budget performance, increasing transparency

and accountability to citizens.

3. Percent of programs with improved strategic

design and reporting.

(Revised) Percent of public programs for which

there is performance information using a

standardized performance framework.

3There is evidence from previous analytic work supported by the Bank (Technical assistance to the SHCP on October

2007, corroborated by subsequent CONEVAL analysis) that the quality of strategic design of public programs is weak

along several different dimensions (objectives, indicators, link to strategies). Using equivalent dimensions, the UED

unit was supposed to be able to measure overall quality over time, using that previous work as a baseline.

3

4. Progression of organizations through the

steps (levels) established for each PMG

system, marking increased levels of

administrative capacity and efficiency.

(Dropped original and replaced with new)

Percent of municipalities that have a simplified

financial management system installed and

working.

Source: Proposed Project Restructuring of Results-based Management and Budgeting Loan 68464-MX, 2012.

1.4 Main Beneficiaries

8. The primary target groups of the Project were the users of performance

information, including line ministries, Congress and the general public. These main

beneficiaries did not change when the Project was restructured.

1.5 Original Components (as approved)4

Component 1.Management Improvement Program.

9. Provision of technical assistance and training for the design and implementation

of selected management improvement and austerity systems, including, inter alia:

(a) assistance in the design of the institutional module of the Borrower’s Programa

Especial de la Mejora de la Gestión en la Administración Pública Federal 2008-2012;

(b) assistance for technical groups established and to be established with respect to each

of said management improvement and austerity systems; and

(c) capacity building for actors responsible for the implementation of said management

improvement and austerity systems.

Component 2. Results-Based Management and Budgeting Integrated Information Systems

10. Provision of technical assistance and training to support the development of an

integrated information system, including, inter alia, the development of the necessary

institutional arrangements, procedures, and an associated electronic database to:

(a) receive budget and other expenditure information from different sources;

(b) organize and analyze said information; and

(c) produce abridged, user-friendly reports with key messages to inform various

stakeholders and improve transparency of budget execution.

Component 3.Financial Management Strengthening for Results-Based Budgeting

11. Provision of technical assistance and training to strengthen the quality and

integrity of financial data, including, inter alia:

(a) modernization of the Borrower’s public accounts (Cuenta Pública) and the underlying

government accounting systems to strengthen the usefulness, integrity and transparency

4 The components are presented as per the Loan Agreement. For a more detailed description of the original

components see the Project Appraisal Document Report No: 46271-MX.

4

of public accounting records and related accounting management systems for

management decision making and public accountability; and

(b) strengthening of the quality and timeliness of the Borrower’s budget execution

reporting.

Component 4. Results-Based Management and Evaluation System Consolidation

12. Provision of technical assistance and training to:

(a) address government’s capacity limitations to carry out, contract and use evaluations of

government programs; and

(b) consolidate a system to evaluate the effectiveness of federal policies and programs

through, inter alia:

(i) (A) creation and consolidation of a Sistema de Evaluación del Desempeño in the

Borrower’s government; and

(B) strengthening of the alignment of the Borrower’s National System for

Democratic Planning components, including, Vision 2030, National Development

Plan, sector strategies and budgetary programs and the strengthening of a selected

set of strategic indicators based on sharper program design, and ownership

building of these indicators among the Borrower’s line secretariats;

(ii) strengthening the generation of performance data for the analysis of the

Programas Federales; and

(iii) implementation of results-based management approaches at the state level.

Component 5. Monitoring and Evaluation

13. Provision of technical assistance and training to SFP and SHCP to develop the

necessary organizational and evaluation capacity to monitor the progress and to evaluate

the implementation of the Borrower’s Sistema de Evaluación del Desempeño and related

institutional reforms.

1.6 Revised Components

14. The November 2012 restructuring (level 1) introduced the following changes to

the components:

(i) Components 1 (Management Improvement Program) and 2 (Results-Based

Management and Budgeting Integrated Information Systems) were cancelled as a

result of changes in Government priorities and use of own resources;

(ii) Component 3 (Financial Management Strengthening for Results-Based

Budgeting) continued as originally designed with some changes in activities that

focused on modernizing Government accounting and public accounts, as well as

harmonizing and integrating budget execution reporting;

(iii) Component 4 (Results-Based Management and Evaluation System Consolidation)

was revised to update Project activities with respect to consolidating the

5

Government evaluation system and to include a fourth sub-component according

to the following description5:

“the provision of support to the Borrower’s fiscal transparency initiatives

including: (a) the participatory design and implementation of an integrated fiscal

transparency web portal, and (b) the design of guidelines and regulations to foster

fiscal transparency in the Borrower’s federal, state and municipal levels of

government”;

(iv) Component 5 (Monitoring and Evaluation) would continue with some changes in

activities including cancellation of activities to be originally implemented by SFP.

The revised description6 is as follows:

“Provision of technical assistance and training to SHCP to develop the necessary

organizational and evaluation capacity to monitor the progress and to evaluate the

implementation of the Borrower's Sistema de Evaluación del Desempeño and

related institutional reforms.”

1.7 Other significant changes

15. The November 2012 restructuring (level 1) introduced these additional changes:

(i) the restructured Project was to be implemented solely by the SHCP through its

Subscretaría de Egresos (SE) eliminating the participation of the Public

Administration Ministry or Secretaría de la Función Pública, SFP;

(ii) the Project financing reflected the cancellation of US$12.756 million of the

loan amount (from US$17.2 million to US$4.4 million) and revisions to the

costs by component, financing plan and expenditure categories;

(iii) the procurement arrangements were changed to apply the Consultant and

Procurement Guidelines of May 2004 as revised October 1, 2006 and May 1,

2010 to the restructured Project; this was to enable application of the Bank

Guidelines that are harmonized with the then currently authorized policies and

procedures for externally funded projects in the country; and

(iv) the procurement plan and implementation schedule was revised as a result of

the reduction in Project scope and partial loan cancellation.

16. With respect to Project financing changes, the total Project cost was reduced from

US$31.9 million to US$6.7 million (excluding front-end fee)7. In terms of the Project’s

financing plan, the reduction in total Project cost reduced both the counterpart fund

contribution as well as the required IBRD loan amount (from US$17.2 million to US$4.4

million including the front-end fee). See Table 2 for details.

5 Amendment to the Loan Agreement dated November 29, 2012. 6 Ibid. 7 See Annex 1 for more details.

6

Table 2: Summary of Project Total Costs and Estimated Financing Plan (US$ million)

As approved (PAD) After restructuring

Components/Activities Local IBRD Total Local IBRD Total

1. Management

improvement program

(PMG)

0 7.600 7.600 0 0 0

2. Development of an

Integrated Information

System for Results-Based

Budgeting and

Management (SISED)

2.100 1.900 4.000 0 0 0

3. Strengthening Financial

Information for the

implementation of Results

Based Budgeting

2.075 1.925 4.000 1.850 2.925 4.775

4. Consolidation of the

Performance Evaluation

System and Results Based

Budgeting

6.050 4.000 10.050 0.440 1.131 1.571

5. Institutional

Strengthening and

Program Monitoring and

Evaluation

4.500 1.775 6.275 0 0.3875 .3875

TOTAL 14.725 17.200 31.925 2.29 4.4435 6.7335

Source: Proposed Project Restructuring of Results-based Management and Budgeting Loan 68464-MX, 2012.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

17. The quality at entry is viewed to have been moderately unsatisfactory. During

preparation, the Project benefitted from synergies that emerged from activities organized

by the Bank, such as the Performance Budgeting workshop that took place in Mexico in

June 2008, which presented international experiences on the topic and became an

important input for the initial design of the reforms. In addition, the two original

counterparts, SFP and SHCP, also benefited from the economies of scale derived from

implementing the results-oriented agenda towards a comprehensive reform–including

both management and budget. However, the design of the Project was complex; the

potential lack of coordination between the two main counterparts (SFP and SHCP),

identified as substantial risk at appraisal, materialized during the Project’s initial stages

and implementation. Moreover, it should be noted that moving from monitoring budget

execution to results monitoring, and from compliance to results in public sector

management via a more comprehensive program, such as the PMG, implied a big reform.

Most countries in Latin America and in Asia—as highlighted by one of the peer

reviewers—have struggled with this type of changes. As such, even though counterparts

coordinated their efforts and defined their roles for the results-based management and

budgeting reform during the Project’s preparation, the significant delay in loan

effectiveness and subsequent delay in implementation suggests that the coordination

mechanisms needed for the Project were underestimated.

7

18. The PAD indicated that the Project met the criteria for readiness for

implementation.8

However, there were delays between the signing of the Loan

Agreement and the declaration of effectiveness. The Project was approved by the Board

on December 11, 2008. The Project’s legal agreement was signed on April 21, 2009, and

became effective on March 19, 2010 due to an unanticipated delay in internal GoM

administrative processes – specifically, the execution of the administrative agreements

(Contrato de Mandato)9 between the Borrower’s Representative (Public Credit Unit in

SHCP), the Financial Agent (Nacional Financiera, NAFIN), and the implementing

agencies - SFP and SHCP.

19. Overall, the long time elapsed since the loan’s approval and effectiveness (16

months), along with a challenging procurement environment involving multiple parties

(i.e., in Mexico it typically involves participation of three to four parties - the

implementing entity/ies, the financial agent and the Bank) and weak procurement

capacities in the implementing agencies, made it impossible for the Government to start

utilizing the loan proceeds until one and a half years after Board approval.

2.2 Implementation

20. The implementation of the Project is viewed to have been moderately

unsatisfactory primarily due to the lack of physical progress under the Project.

21. Start-up delays, a complex project design that included a governance framework

involving two implementing agencies, limited implementation capacity, as well as

changes in Government counterparts10

, severely hampered Project execution. Moreover,

within SHCP the reform and Project activities were not fully internalized or appropriated

in particular by the operational teams. SHCP areas (Budget Direction, Evaluation

Direction, and International Affairs) were not fully aligned conceptually regarding

Performance Based Budgeting and its priorities and targets. These resulted in the

Government requesting a partial cancellation of the Project (US$12.7 million equivalent

to 74% of Loan amount) as early as August 2011. As of Project restructuring in

November 2012, only the front-end fee for the IBRD loan of US$43,108 had been

disbursed under the loan. Moreover, lengthy procurement processes, in some cases

caused by changes in Project Government leaders and operational teams, not only

affected Project implementation but also posed a risk to its continuity, triggering internal

8 Readiness for implementation means that the Project ideally should become effective within four months from Board

approval and is able to evidence physical implementation including initial disbursement for project activities (not just front-end fee) within the first six months following effectiveness.

9There were two specific effectiveness conditions for this operation:(a) the duly executed Contrato de Mandato by the

respective parties; and (b) related legal opinions from the respective parties of the Contrato de Mandato.

10 In the SFP, there were 5 subsecretarios between 2008 and 2012.

8

Bank discussions on a possible loan closing during the process of restructuring in

2011/2012.

22. Despite this lack of progress, the Government remained committed to the overall

objective of the Project that supports a critical public sector reform, namely the

introduction of results-based management and budgeting in the context of a continued

effort to modernize and harmonize public financial management (PFM) rules and

procedures in all levels of government in Mexico. This is evidenced by some activities

being advanced and financed with local resources and which was the basis for the

continued positive assessment of the achievement of the PDO11

.

23. Using its own resources, significant progress was made by the Government in

pursuing the indicated PFM reform agenda particularly in the areas of results-based

budgeting, performance evaluation, accounting harmonization, fiscal transparency and

public procurement management. The most important achievements in these areas are

presented below: 24. Results-Based budgeting and performance evaluation: The PBR-SED

12 reform

led by the SE has successfully advanced the design and introduction of specific

guidelines and instruments to gradually adopt a results and performance orientation in the

Federal Government’s budgeting process. By developing a conceptual model, the new

system articulated two instruments aiming at incorporating the results and performance

dimensions into the resource allocation procedures, i.e., the adoption of performance

indicators for public sector programs and the implementation of systematic evaluation

activities over public sector policies and programs.

25. Accounting harmonization: As part of the overall strategy to modernize public

sector financial management practices in the country, the Government started an

ambitious program towards the harmonization of public accounts and accounting

practices across Federal and sub-national governments. Introducing results-informed

management capabilities require adequate accounting, reporting and consolidation of

information mechanisms as a sine qua non condition for success. Thus, based on

timelines clearly established in legislation passed at the request of the Administration,

efforts were deployed to modernize and harmonize accounting systems.

26. Fiscal transparency: The SE also achieved significant progress in the fiscal

transparency agenda. After a broad consultation and participatory process involving

specialized Non Government Organizations (NGOs) and experts, and in collaboration

with the Access to Information regulatory entity – IFAI, the SE launched a fiscal

transparency Portal13

that provides access to detailed information on the allocation and use

of federal government’s resources. The portal is now fully operational and has also

included a citizens’ friendly application–Presupuesto Ciudadano–that not only facilitates

11 ISR DO rating was Moderately Satisfactory from late 2009 – early 2013. 12

PBR-SED stands for Presupuesto Basado en Resultados y Sistema de Evaluación del Desempeño. 13

For more details visit http://www.transparenciapresupuestaria.gob.mx


9

access to budget information, but also describes the budgetary process in a very

accessible way for citizens.

27. The SFP had also been advancing the public sector management modernization

agenda complementing efforts in areas related to the improvement of public sector

management practices, streamlining internal regulatory frameworks, and modernizing the

Federal Government’s public procurement system and procedures.

28. Progress achieved by both the SE and SFP using own resources was in line with

the PDO and had received Bank’s feedback and guidance. In the case of the PBR-SED

effort, the Bank sustained technical dialogue during Project implementation support

missions for those activities that should be carried out through the Bank loan. On the

other hand, the PFM harmonization work was supported through both Bank

implementation support activities and a parallel advisory services activity.

29. With these advances using local funds, a Government request for a partial

cancellation was received on August 8, 2011. This was followed by a request in late

2011 to reduce the Project scope to reflect the advances and redesign of activities to

better align them with current priorities. In this regard, the processing of the cancellation

by the Bank was delayed pending validation of the final set of activities that would need

to be dropped and those that needed to be incorporated under the restructured Project to

ensure viability after restructuring. Moreover, the November 2012 restructuring was

finalized after a careful re-assessment by Government counterparts and the Bank that

took into account the potential impact of the political transition following the Presidential

elections in July 2012. Specifically, the SFP indicated its preference to cancel its

activities in the Project.

30. Moreover, soon after the Project restructuring, the new Administration took place

in 2013. Among the first impact on the Project was the change in the executing unit from

the Budget Office of the SE to the Performance Evaluation Unit (Unidad de Evaluación

del Desempeño, UED)14

. This facilitated a better alignment of presidential priorities with

sectorial plans and programs’ objectives/ targets. However, the political transition also

resulted into a set-back in Project execution since the activities needed to be revalidated

with the new counterparts. Actual Project activities implemented were minimal with

total disbursements of less than US$400K. Furthermore, priority shifts in some critical

results areas of the Project led to some results indicators not being achieved at all. This

was particularly true for the roll-out of simplified financial management systems to small

municipalities that could not be achieved because of a priority shift towards the

development of the Government Accounting System first.

14 This evaluation unit reported to the Budget Office Director (UPCP). In October 2012, the internal Regulations of the

SHCP were amended through a Decree; these reforms included the creation of the Performance Evaluation Unit with

attributions to coordinate performance evaluations established in article 111 of the Budget and Fiscal Responsibility

Federal Law (LFPRH).

10

31. Another factor affecting implementation was the existence of a similar operation

supported by IADB. The IADB supported Performance Based Budgeting through a loan,

but there was a lack of effective coordination between the two institutions. In this

context, both Banks were almost competing and not aligned in technical aspects.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

32. M&E design. The results framework presented in the PAD provided for

performance indicators for the PDO and intermediate results for each component.

However, these were largely qualitative in nature and with only the desired target value at

end of Project identified. No target values were identified for the time period between

beginning and ending which made monitoring a challenge and perhaps more susceptible

to optimistic assessments as evidenced by the Moderately Satisfactory ratings for PDO

achievement throughout the life of the Project except for the last ISR. Progress towards

achieving the PDO could have been more meaningfully and objectively assessed if target

values were identified for the implementation period and not just at exit. Moreover, with

mid-course target values, mid-course corrections could have been introduced earlier.

33. M&E implementation. During the second and third years of implementation, the

Bank scheduled periodic high level meetings with and between Deputy Ministers to

discuss the progress of the Project, but these meetings did not occur as planned and when

they did, they could not provide guidance to the Project at the political or technical level

needed. The PDO and intermediate indicators were adjusted during Project restructuring

(level one) in 2012 to establish a better link with Project activities. While the target

values were also identified only for Project exit, the indicators have improved to allow

for a more objective if not quantitative assessment during implementation post-

restructuring. The Bank regularly monitored progress under the Project through periodic

implementation support missions and were also included in the implementing agencies

progress reports.

34. M&E utilization. The PDO and intermediate results indicators, particularly with

respect to the target measures or percentages, were closely monitored by the Bank team

and reported in ISRs. The counterparts provided updates to the Bank on the results

indicators usually at the conclusion of every implementation support missions. Post-

restructuring, the Government also used the transparency portal to monitor most of the

results indicators. However, more in situ monitoring effort is done in each ministry

through their planning units instead of the UED at SHCP.

2.4 Safeguard and Fiduciary Compliance

35. No safeguard policies were triggered by the Project at design, an appropriate

decision, as no such issues arose during implementation.

36. The key issues and risks concerning procurement for Project implementation were

identified since Project appraisal; these included the lack of experience of the two

implementing agencies in applying Bank procurement guidelines. The corrective

11

measures were agreed with the executing agencies and were mainly related to keeping

enough trained personnel on Bank guidelines in the implementing agencies’ procurement

units. However, limited procurement capacity continued to be an issue during

implementation before and after restructuring which posed a substantial risk to the

Project’s success.

37. Risk for Project financial management was assessed as low and was considered to

be satisfactory throughout Project implementation. Financial monitoring reports were

submitted on time and were acceptable to the Bank. All minor inconsistencies found

during supervision were properly addressed by the client and did not impact the controls

of the Project. Annual audit reports for the Project were submitted accordingly and

reviewed by the Bank, with auditors issuing their unqualified opinions.

2.5 Post-completion Operation/Next Phase

38. Continuing commitment to the development objective beyond the scope of the

loan has been demonstrated through the decisions of actors including the UED to engage

in related activities with their own funding. There are also positive signals that future

Bank engagement with SHCP would include a focus on strengthening the UED as well as

the M&E capacities of line ministries and sub-national governments. To this end, the

lessons from this Project are expected to inform future engagements and could therefore

help to consolidate and sustain the Close and Modern Government Program (Programa

de Gobierno Cercano y Moderno, PGCM),15

as well as the reform process of SHCP in

the area of performance information, integrated financial management, use of this

information for improving commitments that result from evaluations and for management

and budget decision making.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation

39. To achieve faster growth, reduce extreme poverty and promote shared prosperity,

Mexico needed to improve public service delivery. Mexico’s public service delivery is

still hindered by expenditure inefficiencies at all levels of government: federal, state, and

municipal. In this context, sound and informed budget decision making were critical to

ensure that services were delivered efficiently, as well as in a transparent and accountable

manner.

40. The GoM recognized this challenge in the 2013-18 NDP, which lays out a cross-

cutting program for a “Close and Modern Government”. The PGCM aims at establishing

a results-based, efficient government, as well as developing monitoring and evaluation

15 See section 3.1 for more details on this cross cutting program included in the National Development Plan 2013-2018.

12

mechanisms for improving performance and service delivery. Additionally, the program

seeks to simplify procedures and regulations, allow for greater transparency and

accountability, ensure an optimal use of public resources, and seize the potential of

information and communication technologies. By focusing on enhancing the information

on performance available for government entities and citizens, the Project’s original

design and implementation are fully aligned with the GoM’s current strategy and

continue to be highly relevant for Mexico.

41. As mentioned in the Project Appraisal Document (PAD), the Project was aligned

with the CPS FY08-1316

that called for strengthening institutions to improve their

functioning and to enhance citizen perception of the public sector through a set of

initiatives including results based-budgeting. The Project was also consistent with the

World Bank thematic engagement set forth in the new CPS FY14-19, specifically with

the third theme, “Strengthening Public Finances and Government Efficiency”. 17

The

Bank strategy underscores the need for Mexico to improve public sector performance,

including expenditure quality and equity, and aims at supporting better service delivery.

Through this Project, the Bank has contributed to extending the monitoring and

evaluation techniques of key public policies, strengthening the supervision of indicator

achievements, and generalizing the follow up of intermediate results, establishing the

foundations for a more efficient and transparent public administration.

42. As mentioned earlier, design of the Project was complex but it was key to enable

the changes that needed to happen in order to pursue a results-oriented reform in budget

and public sector management and effectiveness. With respect to implementation, albeit

unsatisfactory in the end, it was relevant as it allowed for a continuing engagement and

dialogue between the Government and the Bank on the critical public sector reform areas

beyond the Project.

3.2 Achievement of Project Development Objectives

43. The overall rating for PDO achievement is Moderately Unsatisfactory.

44. Prior to the November 2012 restructuring, the achievement of outcomes is

assessed in this ICR to be moderately unsatisfactory. While credit is due to the SHCP

and SFP for their achievements using own resources, these did not enable the Project to

sufficiently progress towards achieving its desired outcomes. Nevertheless, as discussed

in Section 2, their achievements in advancing results based budgeting and performance

evaluation, accounting harmonization and fiscal transparency, albeit without any loan

disbursement except for the front-end-fee, were aligned with the PDO.

45. Post-restructuring, the achievement of outcomes with disbursement of

approximately US$400K are assessed to be likewise moderately unsatisfactory for failure

16 Report No. 42846-MX

17 Report No. 83496-MX

13

to achieve several results indicators including the PDO indicator and related intermediate

outcome indicators with respect to the roll-out of a simplified financial management

system to small municipalities and the evaluation of public programs. Nevertheless, the

following achievements after the restructuring are worth noting:

(i) Performance information (PI) is now presented in a timely manner to

Congress in a format that allows interpretation and meets their needs. The

content includes outcome and output measures.

(ii) The majority of the public programs (84.5%) have a robust design across key

dimensions (objectives, indicators, baselines, links to strategies) enabling

better reporting and evaluation.

(iii) The Portal containing the PI on budget performance is available to citizens. In

2013, the UED developed more citizen friendly applications such as Cuenta

Pública Ciudadana 2012 and 2013, as well as their English versions. These

initiatives were mostly financed by component 4 of the loan. An improved

version of the budget transparency portal is to be launched in June 2014,

including new platform, features, etc. See figure below and Annex 2 for more

details on outputs that contributed to achieve PDOs.

46. Overall, the Project supported the introduction of better management practices,

processes and tools in the Mexican public administration, including better and timely

performance information, enhanced reporting and evaluation of public programs, and

transparent and available budget information. The adoption of the Budget Transparency

Portal, which was largely funded by loan funds, resulted into tangible outcomes in terms

of generating information and enabled the country to maintain its Open Budget Index

standing.18

At the same time, the Project financed the training of officials through a

18 In 2012, Mexico has been one of the countries that provide “significant budget information” per the Open Budget

Index.

Outputs. 1. Budget Transparency Portal

2. Established methodology and manuals that set the minimum technical requirements and guidelines for budget transparency and provide orientation for performance indicators

3. Developed Training Programs based on the challenges identified and on demands (i.e. logical framework matrix)

Component 4. Consolidation of the performance evaluation system’s (Sistema de Evaluación del Desempeño – SED) institutional architecture and strengthening the system for evaluating policy and program effectiveness (Presupuesto Basado en Resultados – PBR)

Project Development Objective 2. An active network and a portal to inform society at large on a regular basis about budget performance, increasing transparency and accountability to citizens

14

series of activities including online courses, workshops and the development of a series

of manuals and methodologies to collect, monitor and evaluate budget information. This

now appear to have good results as evidenced by the work being continued and

specifically evidenced with SHCP’s presentation of the “Guidelines to register, revise,

update, schedule and monitor the Budget Programs Results Indicators Matrix (MIR)” in

the first semester of 2014.

3.3 Efficiency

47. Project efficiency is low. The reduction in scope and the limited results achieved

despite the restructuring and extension suggest that overall efficiency is much lower than

expectations at design and at restructuring. The long time elapsed between preparation

and the Project closing with minimal physical accomplishments and limited results vis-à-

vis the overall desired outcomes suggest there were opportunities missed which were

partly alluded to in the Borrower’s comments on the relevance of Project activities

(particularly those of SFP) having been reduced given the delays. In monetary terms,

from the beginning, a cost-benefit analysis was not considered to assess the impact of

these institutional reforms, as problems in the quantification of intangibles and issues of

attribution make it difficult to express their impacts. Moreover, given loan amount

disbursed (US$373,684) vis-à-vis the time-overrun in implementation and the resources

mobilized in the Bank for preparation and provision of implementation support

(US$967,977) which are more than double the loan disbursement, it could be considered

that the Project failed to be efficient. Nevertheless, the operation managed to achieve

significant results in terms of increased transparency, good governance and

accountability, as measured by greater availability and access to budget data and

performance information (see Annex for more details). Further potential efficiency gains

and cost savings are expected from these types of reforms when fully implemented.

3.4 Justification of Overall Outcome Rating

Rating: Moderately Unsatisfactory

48. As discussed above, while the relevance of the objective remains high with the

public financial management and results-based agenda at the federal and subnational

levels remaining a priority, the achievement of the PDO could only be rated moderately

unsatisfactory given the inefficient Project implementation and the non-achievement of

significant outcomes.

3.5 Overarching Themes, Other Outcomes and Impacts

(a) Poverty Impacts, Gender Aspects, and Social Development

49. No direct poverty, gender or measurable impact on the population is attributable

to the Project. However, the Project’s impact on public sector management areas has

potential indirect impact on reducing poverty through the effect of better resource

15

allocation and improved service delivery conditions, as well as by channeling generated

savings more efficiently to national priorities.

(b) Institutional Change/Strengthening

50. The M&E capacity of the UED was significantly strengthened in the latter part of

the Project. From Project start to November 2012, the Project was executed by the

Budget Office of the SE. Beginning December 2012 (when the current administration

took office) the UED19

took over Project execution. This facilitated a better alignment of

presidential priorities with sectorial plans and programs’ objectives and targets. Technical

assistance during implementation support missions by the Bank contributed to this, as

well as feedback to the reform decisions provided by TTLs and experts members of the

World Bank team.

(c) Other Unintended Outcomes and Impacts (positive or negative)

51. No significant unintended outcomes or impacts were identified.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

52. No beneficiary survey or stakeholder workshops were conducted specifically for

reviewing Project performance.

4. Assessment of Risk to Development Outcome

Rating: Moderate

53. The general direction of the reforms supported by the Project is aligned with those

of the Government’s PGCM. As such, risks to the outcomes that have been achieved to

date are considered to be moderate. While in the medium to long term, further advances

in effectiveness building off Project activities will likely require additional investments in

technology and process redesign, all of these are considered in the PGCM.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance

(a) Bank Performance in Ensuring Quality at Entry


19 This evaluation unit reported to the Budget Office Director (or UPCP). In October 2012 through a Decree the

internal Regulations of the SHCP were amended; these reforms included the creation of the Performance Evaluation

Unit (UED) with attributions to coordinate performance evaluations established in article 111 of the Budget and Fiscal Responsibility Federal Law (LFPRH).

16

54. The Project was intended to achieve challenging reforms in public management

and results-based budgeting, building from efforts supported by the Bank during the

beginning of the 2006-2012 administration and Project preparation. The Project was

prepared taking into account lessons learned from international experiences and taking

advantage of the synergies between SFP and SHCP who were then working together

towards a results oriented agenda. The Project was consistent with the federal

government’s objectives and the Bank strategy in Mexico. However, several

shortcomings in Project design posed significant challenges to implementation. In

particular, the failure to identify the challenges involved in coordinating amongst and

with several federal government entities, and the overestimation of the counterpart’s

capacity to carry out complex procurement processes (both in SFP and SHCP) caused

significant risks to the successful completion of the Project. As noted by the Borrower in

their comments, Project implementation could have benefited from having a focal point

for coordination as well from providing a venue to collaborate among all the stakeholders

during Project preparation. The delay in declaring the loan effective (16 months from

Board approval) suggests a significant under-estimation of the Project’s readiness to be

implemented.

(b) Quality of Supervision


55. The initial implementation support phase of the Project could be characterized as

overly optimistic and highly focused on technical aspects, with inadequate attention paid

to operational constraints. On the positive side, in a context where the Government was

moving fast with their reform agenda and the demands were changing, the Bank was able

to sustain technical dialogue specifically on the PBR-SED and PFM harmonization

efforts which were financed by local funds. On the negative side, the Bank failed to take

advantage of the synergies in place between SFP and SHCP in the early stages of

implementation. On top of that, the SFP faced high-level public officials’ turnover and

the World Bank team failed to recognize these implementation risk flags or adjust to

these changes in the context accurately. The Bank’s implementation support team also

suffered from lack of leadership at the beginning of implementation when the TTLs were

changed20

. Hence, realistic assessments of Project implementation constraints early on

could have facilitated a Project turnaround by addressing the changing dynamics in the

reform. There were also indications that communications between the Bank team,

fiduciary teams, financial agent and implementing agencies could have been better to

improve Project implementation support and administrative processes.

56. By 2011, it became clear that the Project scope had to be reduced and Project

activities redesigned to better align them with the incoming Government’s priorities as

20 The Project had a total of three TTLs throughout the five years of implementation,

17

was detailed in the Government requests. However, the processing of the partial

cancellation by the Bank was postponed to allow for a thorough assessment of viability

and commitment from the newly elected administration (at the end of 2012). During this

time, the Bank flagged potential issues and took action to solve them effectively

throughout the last year of implementation. The new government had just taken office

and continued to align its priorities; and as noted above, even though the Bank team

engaged in intensive implementation support, procurement processes lingered for

extensive periods given procurement capacity constraints within the implementing

agency and as such only a few contracts were executed with less than 10% of the total

restructured loan funds disbursed. Nevertheless, post-restructuring, a closer collaboration

between the Bank team and the counterparts were established that not only help move the

Project forward as well as contributed to the institutional strengthening of the UED.

(c) Justification of Rating for Overall Bank Performance


57. Based on the Bank performance in ensuring quality at entry, and the quality of

implementation support, the overall Bank performance is rated as Moderately

Unsatisfactory. A number of challenges that led to delays in Project start-up and

execution were not fully identified at design, and communication between the Bank and

the client could have been better during the initial part of implementation. However, this

ICR recognizes the Bank implementation support team’s significant efforts during the

latter part of implementation to better address the procurement issues and to try to

achieve the new Government’s goals.

5.2 Borrower Performance

(a) Government Performance


58. Despite the fact that Project implementation suffered delays, the areas supported

under this Project were part of the Government of Mexico’s priority agenda with respect

to improving results based management and budgeting as evidenced by the reform

advances that took place using their own resources. However, the political commitment

to the Project was weakened by the weak (if not lack of) coordination between SFP and

SHCP and compounded by the turnover of government officials (5 subsecretarios held

office in the SFP as the main counterparts for Components 1 and 2). Moreover, the

Project was not fully internalized or appropriated in particular by the SHCP’s operational

teams. SHCP areas (Budget Direction, Evaluation Direction, and International Affairs)

were not fully aligned conceptually regarding Performance Based Budgeting and its

priorities and targets. SFP and SHCP’s high-level commitment could have translated into

proactive measures by ensuring that the Project operational units were progressing with

Project implementation satisfactorily.

18

(b) Implementing Agency or Agencies Performance


59. The Project Implementation Units both at the SFP and SHCP were committed to

the Project but had a low capacity in fiduciary aspects, especially procurement as

identified in the risks at appraisal. During the last year of implementation, SHCP’s

implementing unit (within the UED) was very organized and engaged but lacked

familiarity with the Bank’s procurement guidelines that did not allow them to disburse

most of the resources. Nevertheless, despite their capacity constraint, they were able to

focus on limited activities that had significant contribution to the desired Project

outcomes. With their limited implementation capacity, they were able to advance on the

Budget Transparency Portal that enabled the Project to deliver on its objective of

disseminating budget performance information to the Borrower’s citizens on a regular

basis.

(c) Justification of Rating for Overall Borrower Performance


60. Considering the above arguments, the overall Borrower performance is deemed

moderately unsatisfactory.

6. Lessons Learned

61. The instrument was not the best alternative for what was needed for these reforms.

The mitigation measures in place did not lower the risks identified at appraisal. This

Project benefitted the most from the technical assistance and feedback received both

during preparation and the last years of implementation. In retrospect, it would have

been better to have supported the Government’s agenda with a Non-Lending Technical

Assistance (NLTA) program or a Reimbursable Advisory Services (RAS) to support

more effectively the type of reforms the Government was pursuing. The chosen

instrument allowed the Bank to promote the technical dialogue with the authorities and

amongst stakeholders, and also ensured agreements and commitments at the highest

levels. However, these were not fully internalized or appropriated in particular by the

SHCP’s operational teams as discussed above.

62. Assessment of Risks - Coordination mechanisms were not enough. Although there

were mitigation measures in place for this risk, the fact that there were two implementing

agencies and a financial agent was more complex than initially assessed. Moreover,

within each of the implementing agencies, coordination was not enforced thus affecting

project implementation and disbursement performance. The designation of a focal point

for coordination could have helped as noted by the Borrower.

63. Demand not well read in the case of SHCP. While Components 1 and 2 were very

much in line with the Government’s (SFP) agenda, both conceptual and technical; the

19

Components related to results-based budgeting (SE of SHCP) which included the Budget

Office (UPCP) in the first stages and the UED in the last years were not aligned. This is

largely reflected in the lack of disbursements throughout the Project.

64. Additional implementation support should be considered for projects that involve

more than one implementing unit that are not familiar with large and complex

procurement processes, such as some investments within this Project. For this Project, the

need for additional support should have been anticipated at the beginning of Project

implementation. While capacities at the Ministries were strengthened, this was not

enough. Moreover, improved harmonization of procurement procedures between the

Bank and GoM is a necessity as noted by the Borrower’s comments.

65. Ensure alignment between Government processing and Bank required action. As

reflected in the Borrower’s comments below, timely action from the Bank with respect to

issuing Bank feedback especially “no objection” is also a critical element for timely

implementation.

66. Meaningful and objective results indicators are critical for monitoring complex

reform initiatives. While reform initiatives do not always allow for quantifiable

measures of success, this Project could have benefited from a results framework that

provided for mid-course target values that could have informed counterparts and the

Bank that mid-course corrections were warranted earlier.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners

(a) Borrower/implementing agencies

67. The Borrower provided very informative and pertinent comments and these have

been taken into consideration in the assessment of overall performance of the Project, the

Borrower and the Bank as well as in the lessons learned section. These comments are as

follows:

a) The initial objective of the GoM’s “Results based Management and

Budgeting” Project aimed at having the departments and federal organizations

of the country provide decision makers and citizens in general with rigorous,

timely and easy to access information regarding the efficiency and

effectiveness of public expenditure programs.

b) The GoM sought Bank support for the financing of the Project through an

IBRD loan, with the objective of reaching the goals proposed under the five

components that integrated the Project. Given the legal attributions conferred

by the Federal Public Administration Organic Law, the Government mandated

the two main ministries involved in the PBR-SED initiative (SHCP and SFP)

with the implementation of the Project.

20

c) Having two implementing agencies was an obstacle to the successful

execution of the Project, basically due to the complexity of coordinating the

work of the two ministries and the lack of clarity in the Project objectives, as

well as its scope. This was accentuated by the absence of a coordination focal

point and the limited cooperation amongst both ministries, the Financial

Agent and the Bank. Also, the pace of the execution of the Project did not

satisfy Government expectations in terms of results.

d) In addition, the delay between the signing of the Legal Agreement (April 21,

2009) and effectiveness (March 19, 2010) had also a negative influence in the

implementation of the Project. During the first stages of Project execution, it

was not possible to hire any consultancy and advisory services due to delays

in the execution of Bank administrative processes—such as “no objections”—

through the Financial Agent. As a result, the first consultancy services were

only provided in 2011.

e) Under these circumstances, one of the executing agencies, SFP, decided not to

continue with the implementation of the Project, cancelling the total amount

of resources allocated to its components. As the entity responsible for the

PMG, SFP was counting on the Project to help advance that agenda, including

the design of documents, tools and methodologies. The initial delays reduced

the relevance of the components included in the Project from the perspective

of SFP.

f) Consequently, the Project was restructured in order to have only one

executing agency, the SHCP. The SHCP used the loan resources to: hire, inter

alia, the consultancy services to analyze and evaluate the strategic indicators

of the budget programs; conduct training sessions with regards to PBR-SED;

and implement the mechanisms to report the federal resources transferred to

state entities and municipalities. These activities had a positive impact in the

PBR-SED implementation.

g) Under the restructured Project, the Financial Agent acted in a timely manner,

which was key for the review of the documents needed to register processes

and for obtaining “no objections” from the Bank with respect to the hiring of

consultancy services, as well as disbursements. This also helped produce

Project-required financial and progress reports on time.

h) In the view of SHCP, Bank procurement and hiring processes, especially

those requiring “no objections”, could be improved. The long time elapsed

between Bank approval/rejection caused delays in the processing of contracts,

which greatly affected the proper implementation of Project activities and as a

result had a negative impact on its overall implementation.

i) In terms of areas of opportunity, greater harmonization between GoM and

Bank procurement procedures could be sought, even though there are already

21

documents streamlining Federal Government and World Bank processes. It

would be also advisable to establish work programs on time and in accordance

to reality, including clear hiring procedures that expedite processes so the

products delivered can be used to influence the final results. Finally, it would

be convenient to improve project preparation by working jointly and with

better coordination and communication among involved stakeholders.

68. The Borrower also provided additional editorial comments in other sections of the

ICR. These have been addressed in finalizing the document.

(b) Cofinanciers

Not applicable

(c) Other partners and stakeholders

69. IADB supported Performance Based Budgeting through a loan, but did not work

in coordination with the World Bank. Based on the interviews of stakeholders of the

reforms, the Banks were seen as almost competing and not aligned technically.

22

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in US$ Million equivalent)

Components Appraisal Estimate

(US$ millions)

Actual/Latest

Estimate

(US$ millions)

Percentage of

Appraisal

Component 1 7.6 0 0.00

Component 2 4 0 0.00

Component 3 4 0 0.00

Component 4 10.050

0.875*

8.70

Component 5 6.275 0 0.00

Total Baseline Cost 31.495 0.875 2.80

Physical Contingencies

0.00

0.00

0.00

Price Contingencies

0.00

0.00

0.00

Total Project Costs 0.00 0.00

Front-end fee PPF 0.00 0.00 0.00

Front-end fee IBRD 0.43 0.43 0.00

Total Financing Required 31.925 0.918** 2.90

Source: Informe de avance segundo semestre de 2013, UED - SHCP

*Sum of Loan Disbursement of US$330,576.52 and Counterpart Fund Contributions of US$543,391.

**May not equal due to rounding off.

(b) Financing

Source of Funds Type of

Cofinancing

Appraisal

Estimate (US$ millions)

Actual/Latest

Estimate (US$ millions)

Percentage of

Appraisal

Borrower 14.725 0.543 3.70

International Bank for Reconstruction

and Development 17.200 0.331 1.90

23

Annex 2. Outputs by Component

As discussed in the main section of the ICR, the executed Project activities relate to

Component 4. Below presents the outputs under said Component.

Component 4 - Results-Based Management and Evaluation System Consolidation

1. The key outputs delivered by Component 4 of the Project are described below:

a) Budget Transparency Portal (BTP): Since 2011, SHCP has had in place a Budget

Transparency Portal (www.transparenciapresupuestaria.gob.mx) on which

performance information of the budget programs can be found, as well as public

financial information. The main objectives and contents of this portal are describe in

the table below:

Table 2.1. Objectives and contents of the BTP

Objectives Content

More transparency in Public

Resources Management Public Finance Statistics

Transferred resources to the local

governments

Budget Information

Quarterly Reports in Open Data formats

Cuenta de la Hacienda Pública Federal

Budget transparency towards citizens

(friendly language)

Transparency by Thematic areas,

focused on citizens interests

Citizen Budget

Cuenta Pública Ciudadana or Public

Accounts for Citizens

Frequently Asked Questions

Glossary

Investment Projects

Performance Information (indicators and

goals)

External/Independent Evaluations

Aid and support for Natural Disasters

Reconstruction

Source: Final Report and Results Framework Matrix, UED, SHCP. April 21, 2014.

The main strength of the portal compared to other countries is the fact that all the

performance information is public, including external evaluations (more than a

thousand) and MIRs (results indicators matrix) with indicators (757). There is also

access to public investment information including those funded under special funds

such as FONDEN (or Natural Disaster Fund). The UED has also reached out to

NGOs and civil society to ensure that the information is useful for their purposes and


24

understandable for everyone. For instance, the information included in the citizen

budget for 2014 explains in a friendly manner the following topics:

Growth

Total Budget

Sources of revenue and debt

Deficit

Functional Classification of Expenditures (why the Government spends?)

Administrative Classification of Expenditures (who spends?)

Economic Classification of Expenditures (on what the Government spends?)

Investment Projects

Measures taken to reduce expenditures

SED information such as percentage of expenditures with performance indicators

Links to more detailed information

b) Methodology and Manuals were developed, including:

- “Methodology to develop and implement budget transparency portals at the

federal and local levels (technical and functional)”

- “Guide to establish and monitor goals for federal entities and agencies of the

Government”

- “Training Manual to monitor Budget Program’s objectives based on strategic

indicators”

- “Improved guide to design strategic and management indicators”

c) Various Training Programs have been and will continue to be conducted:

- Online Training

- Training for Trainers

- Training based on identified challenges on MIRs (theory, practical, continued

technical assistance, ad hoc workshops)

- Training to improve the Single Format System or Sistema de Formato Único

(SFU), i.e. information collection and registration as well as quality of

information.

2. A full account of the outputs completed by the Project under component 4 is

presented in the table below. These outputs were mostly financed by counterpart funds

and contributed towards achieving the intermediate result indicator “Consolidation of the

performance evaluation system’s (SED) institutional architecture and strengthening the

system for evaluating policy and program effectiveness” by supporting the targets

specified below as well.

Table 2.2. Completed Project outputs by source of financing

Output Original

Budget

Executed

Budget Target supported

Outputs financed with loan resources 346,000 325,884

Consultancy to prepare and facilitate the 31,000 24,259 Evaluation unit operational in SFP,

25

Project’s selection and hiring processes (1) including an institutional

performance evaluation system

Methodological guide for the technical and

functional implementation of the transparency

web portal at the federal and subnational

levels

40,000 40,000

Evaluation unit operational in SFP,

including an institutional


Development of minimum criteria and

transparency guidelines for the federal and

subnational governments

32,000 32,000




Collection of feedback on PBR program and

the SED 52,000 52,000

The SED unit manages the

evaluation of non-social programs

successfully

Training of trainers course 33,000 33,000

Set of strategic indicators has

improved after support and training

provided to Government officials in

charge of programs

Consultancy for the analysis of budget

strategic indicators for the 2013 budget cycle 18,000 18,000




Consultancy for the maintenance of the

Budget transparency portal 18,000 18,000




Consultancy for the implementation of the

SFU 18,000 18,000




charge of programs

Consultancy for the conceptualization and

analysis of the information produced by the

SFU

25,000 25,000




charge of programs

Consultancy to prepare and facilitate the

Project’s selection and hiring processes (2) 52,000 37,500




Consultancy for the establishment of a

monitoring and evaluation system for the

Project’s selection and hiring processes

27,000 28,125




Outputs financed with local resources 574,093 574,093

TOR for the carrying out of the activities to be

conducted under components iii, iv and v of

the Project

102,241 102,241 -

External audit for loan 7622-MX 2,852 2,852 -

Results-based budgeting online course

(auditing and results-oriented indicators

modules).

469,000 469,000




charge of programs

26

Annex 3. Economic and Financial Analysis

1. The overarching goals of the reforms supported by the Project aimed at increasing

transparency and strengthening good governance and accountability in the Sistema de

Evaluación al Desempeño (SED), which are difficult to measure. Therefore, the

economic and financial analysis for this operation needs to go beyond quantitative

estimations and rely on qualitative data. The assumption is that the support to the GoM’s

institution building efforts provided by the Project, notably in terms of results-based

management and budgeting, has contributed to a more transparent and accountable

governance framework. However, the outputs of the reforms carried out by the GoM,

including the Programa de Mejora de la Gestión (PMG) and Presupuesto Basado en

Resultados (PBR), as well as the SED and the sectorial strategies and specific programs,

are not fully attributable to the Project or a single reform initiative, and their impact may

not be apparent in the short-term.

2. Overall, the Project supported the introduction of better management practices,

processes and tools in the Mexican public administration, including better and timely

performance information, enhanced reporting and evaluation of public programs, and

transparent and available budget information. The most important instrument in this

regard is the Budget Transparency Portal, which was mostly financed with the loan

resources. The adoption of this portal resulted into tangible outcomes in terms of

generating information, since according to the Open Budget Index (OBI), Mexico has

been one of the countries that provides “significant budget information” in their portal

since 2012. Today most budget information is presented in Open Data format, which

allows the private sector, NGOs and civil society in general to access what is relevant for

them. In fact, online visits increased to 28.5% in 2013 compared to 2012, and as of the

first quarter of 2014, there was an increase of almost 50% compared to the same period

of 2013 and more than doubled compared to the first quarter of 2012.

3. At the same time, the Project financed the training of officials through a series of

activities, including online courses, workshops and the development of a series of

manuals and methodologies. These initiatives were aimed at increasing the capacity of

public officials to collect, monitor and evaluate budget information, and now appear to

have achieved good results. At the end of 2013, out of the 981 budget programs included

in the programmable budget (the total is 1,005), 854 programs have performance

information, which is equivalent to 87% of the programmable budget. The work has

continued in the first semester of 2014, when the SHCP presented the “Guidelines to

register, revise, update, schedule and monitor the Budget Programs Results Indicators

Matrix (MIR)”.

4. The upward trend in the use of information by the public, along with the training

received by officials, also suggests a positive impact of the Project in terms of creating an

institutional context more conducive to efficiency. Although it cannot be quantified, it

seems likely that the combined effect of greater accountability and increased capacity

27

will produce financial savings eventually, ultimately boosting the quality of public

expenditures in Mexico.

28

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/

Specialists

Lending

Pedro Arizti Senior Public Sector Specialist LCSPS TTL

Rosa Maria Hernandez-

Fernandez Senior Executive Assistant LCC1C Assistant

Mariano Lafuente Public Sector Mgmt. Spec. LCSPS Management and HR

Improvement

Karla Soledad Lopez Flores Operations Assistant CTGBD Assistant

Felix Prieto Arbelaez Senior Procurement Specialist LCSPT

Jeffrey James Rinne Consultant PRMPS HR Management

Fernando Rojas Consultant LCSPS Public Sector

Maria Guadalupe Toscano

Nicolas Public Sector Mgmt. Spec. LCSPS

Manuel Antonio Vargas

Madrigal Lead Financial Management Spec MNAFM

Supervision/ICR

Juan Carlos Alvarez Senior Counsel LEGES

Pedro Arizti Senior Public Sector Specialist LCSPS TTL

Dmitri Gourfinkel Financial Management Specialist LCSFM Financial

Management

Alberto Leyton Senior Public Sector Specialist ECSP4 TTL

Xiomara A. Morel Senior Financial Management

Specialist LCSFM

Victor Manuel Ordonez Conde Senior Finance Officer CTRLN

Jeffrey James Rinne Consultant PRMPS HR Management

Tomas Socias Senior Procurement Specialist LCSPT

Maria Guadalupe Toscano

Nicolas Public Sector Mgmt. Spec. LCSPS

Manuel Antonio Vargas

Madrigal Lead Financial Management Spec MNAFM

May Olalia Senior Operations Officer LSCPS

Azul del Villar Baston Consultant LCSPS ICR

Eguiar Lizundia Gonzalez Consultant LCSPS ICR

(b) Staff Time and Cost

Stage of Project Cycle

Staff Time and Cost (Bank Budget Only)

No. of staff weeks US$ Thousands (including

travel and consultant costs)

Lending

FY07 2.81 18.963

FY08 52.03 301.141

FY09 27.55 131.268

Total: 82.39 451.373

29

Supervision/ICR

FY09 11.19 41.890

FY10 25.14 89.238

FY11 19.33 82.145

FY12 21.75 108.016

FY13 25.39 99.577

FY14 17.20 95.645

Total: 120.00 516.604

30

Annex 5. Beneficiary Survey Results

Not applicable

31

Annex 6. Stakeholder Workshop Report and Results

Not applicable

32

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR

33

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders

Not applicable

34

Annex 9. List of Supporting Documents

Project Appraisal Document on a Results-based Management and Budgeting Loan

(Report No: 46271-MX), 2008

Mexico Country Partnership Strategy FY14-19

Mexico Country Partnership Strategy 2008-2012

Proposed Project Restructuring of Results-based Management and Budgeting Loan

(Report Number 68464-MX), 2012.

Proyecto de Apoyo a la Gestión y Presupuesto para Resultados Préstamo 7622-

MXBanco Internacional de Reconstrucción y Fomento. Informe de Progreso en la

Ejecucion del Proyecto, Segundo Semestre 2013. SHCP, abril 21, 2014.