why m&e is important in the government sector?

Jesse T. DavidMonitoring and Evaluation Staff

National Economic and Development Authority

Why M&E is Important in the Government Sector?

3 April 2019Climate Change Commission

Outline of the Presentation

• Monitoring and Evaluation Defined

• Role of M&E in the Development Process and in

Public Sector Management

• GPH Results Framework

• Reform Initiatives to Engender Results-orientation in

the Philippine Government

• National Evaluation Policy Framework: NEDA-DBM

Joint Memorandum Circular No. 2015-01

Planning

(PDP 2017-2022, PDP-RM, Sector

Plans, RDPs, Agency Plans,

LDPs, FS)Programming

(PIP, CIIP, IFPs. RIDPs, PIPs of

Agencies, LIDPs, DBCC and ICC)

Budgeting

(GAA,NEP, MTEF, ODA, PPP,

PREXC, Cash Budgetting)

Implementation

(Programs and Projects)

Monitoring (ODA Review,

BARS, BEDS, COA Audits, Agency

M&E)

Evaluation

(RM Monitoring, SER, Ex-Post,

Evaluation Studies)

M&E and the

Public Sector

Management

Process

Project Identification

Project Preparation

Project Feasibility

Study

Project Appraisal and

Financing

Detailed Engineering

Project Implementation

Project Monitoring

and Evaluation

M&E and the

Project

Development

Cycle

Monitoring - a continuing function that uses

systematic collection of data on specified

indicators to provide management and the

main stakeholders of an ongoing

development intervention with indications of

the extent of progress and achievement of

objectives and progress in the use of

allocated funds. (OECD-DAC)

M&E Defined

Purpose of Monitoring

For learning – to improve program or project

performance

• Capture the extent to which inputs are made

available, how critical activities are being carried

out, and whether the expected outputs are being

delivered; so appropriate actions/decisions can be

made;

For accountability – to provide information to

stakeholders

How do we improve ENR Management?

• Agro-forestry

• Reforestation

• Rural Infrastructure

• Land use

• Community based livelihood

• Flood controls

• Climate change adaptation and mitigation

• Waste management

• Coastal clean up

• Partnerships with LGUs

Typical Policy on ENRM Sector

Which initiative works? Which

work best? Which are the most

cost-effective? The most that

provided greater impact?

M&E Defined

For learning – to improve future

projects and programs through

feedback of lessons learned

For accountability – disclosing of

information to stakeholders

Evaluation

An assessment, as systematic

and objective as possible, of an

ongoing or completed project,

program or policy its design,

implementation and results.

(Organization for Economic Cooperation and

Development- Development Assistance

Committee)

Principal Uses of Evaluation

• Billions of pesos are spent on programs and projects;

• Given are limited resources, we need to identify which

programs work and which do not;

• Of the projects that work, which are the most cost effective?

• Knowing which programs have the greatest impact and are

the most cost effective can influence decision-making so

resources can be used more efficiently to achieve

developmental goals

Why evaluate?

• Change policy priorities;

• New programs;

• New implementation models;

• Halt programs that do not work;

• Reallocate resources to programs that

deliver higher results/impact;

• Reallocate resources toward more cost

effective programs;

Why evaluate?

What are the risks when we don’t evaluate?

• An ineffective policy/program/project will continue

to waste public money

• We will miss opportunities to improve a

policy/program/project and generating

better benefits

• We will miss opportunities for saving money which

we could have been spent on more worthwhile

projects

To know the 3 Rs

• Are we doing the right things?

• Are we doing them right?

• Are we delivering the expected results?

Why M&E?

M&E in the Project Framework

Inputs Activities Outputs Outcomes Goals

Monitoring

EvaluationBut planning for evaluation is best done at

program design, before implementation

Internal to the IA but external to the beneficiaries

External to the IA, mostly to the beneficiaries

The shift from Input-Output M&E to Results Monitoring and Evaluation

Results Monitoring and Evaluation (RME)

• Performance information on results is used for improved decision making, strategic planning, risk management, progress monitoring and outcome evaluation

• Results M&E (RME) is a basic Instrument of Managing for Development Results

• Results reflect outputs (goods and services), use of outputs, outcomes (direct benefits) and impacts of a development intervention

What are Results?

Impact - Indirect benefits in the longer term for the

target beneficiaries could be society-

wide, sectoral and geographic/area

levels. In most cases, impacts cannot be

fully and directly attributed to specific

development interventions because of

the complexity of influencing factors.

17

Outcome - Direct benefits as a result of target

groups (either intermediaries or final

target groups) using the outputs. Usually

these benefits happen/occur in the

efforts of the development interventions

of the program/project/ Department.

Included in the outcome definition is the utilization or

adoption of the target groups of the goods or services

resulting from programs/ projects and activities and

inputs/finances.

18

Outputs -

Activities -

Goods or services which are

produced by programs/projects

and activities and

inputs/finances.

Programs/projects/ activities

required to produce the goods

and services.

19

The Results Chain /Framework

IMPLEMENTATION RESULTS

Inputs Activities Outputs Outcomes Impact

Financial, human, and material resources

Tasks and actions undertaken to transform inputs to outputs

Products and services produced

Intermediate effects on clients (short- or medium-term)

Long-term improvement in society

How should this be implemented? What should be produced?

What outcomes do we expect

from this investment?

Why should we do this?

Reform Initiatives to Engender Results-orientation in the Philippine Government

AO 25: Results-

based Performance

Management

System and

Government

Executive

Information SystemNBC 532

Results-based

Budgeting:

Improving the

Link Between

Planning

(PDP-RM

Objectives)

with Major

Final Outputs

2-Tier

Budgeting

Approach

Philippine

Development

Plan (PDP) PDP

Results Matrices

(RM)

Outcome-

based PIB:

Linking

Organizationa

l Outcomes

with sectoral

outcomes and

societal goals of the PDP-RM

Program and

Project

Evaluations

Performance-

informed

Budgeting Cash Budget

System

SER

Preparation

Country Context

Features of Results-

oriented Public Sector

Management (PSM)

Presence of core result

attributes

Focus on common

results

Interdependency

among the components

Effective vertical and

horizontal linkages

Source: Based on the results-based Public Sector Management (PSM) framework developed by the Asia-Pacific Community of Practice on

Managing for Development Results (APCoP-MfDR), Adopted in PDP Results Matrix 2011-2016

Programs and Projects

Organizational

Outcomes/Major Final

Outputs

Sectoral Goals

Societal Goal:

Foundation for

Inclusive Growth, A

High Trust Society, and

Globally-Competitive

Knowledge Economy

M&E Reports

GPH Results Framework

Documents

• Philippine

Development Plan

and-Results Matrices

(RMs)

• Regional

Development Plans

and Regional RMs

• Program Expenditure

Classification

(PREXC) c/o DBM

• Project’s Logical

Framework c/o

Implementing

Agencies

• Socio-Economic

Report

• Regional

Development Report

c/o NEDA

• SONA Technical

Report c/o OP-PMS

• Budget/ Financial

Accountability

Reports c/o DBM

• Budget Execution

Document c/o DBM

• ODA Portfolio Review

• RPMES Reports

• Project-level Reports

(c/o NEDA)

• Project Evaluations

Reports (c/o Agencies/

DPs)

M&E (at the Plan, Sector, Program/Project Level)

24

1-Apr-19 25

The PDP and the Long-

Term Vision

• Results Matrices (RMs) provide an indicator

framework to the statement of goals of the Plan

PDP Results Matrices

M&E at the Plan/Sector Level

1-Apr-19 28

To lay down the foundation for inclusive growth, a high-trust society, and a globally competitive

knowledge economy

Enhancing the social fabric Reducing inequality Increasing growth potential

Ecological integrity ensured and socioeconomic condition of resource-based

communities improved

Biodiversity and functioning

of ecosystem services

sustained

Environmental quality

improved

Adaptive capacities and

resilience of ecosystems and

communities increased

PHILIPPINE DEVELOPMENT PLAN RESULTS MATRICESCHAPTER 20: Ecological Integrity

1-Apr-19 29

Monitoring of the Plan will be through:

• Monitoring of indicators indicated in the PDP-Results

Matrices

• It will be done annually through the annual

preparation of socio-economic report per chapter

• Efficiency and effectiveness of all strategies,

programs and projects supportive of the Plan will be

monitored and will be reported to various NEDA

Board Committees, Cabinet Clusters and Inter -

Agency Committees

PREXCI/OPIF Structure

MFO

Organisational

outcomes

Sectoral

Goals

Societal Goal

PAPS PAPS

MFO MFO

Organisational

outcomes

Sectoral

Goals

PAPSPAPS

MFO

Organisational

outcomes

31

Philippine

Development

Plan-Results

Matrices

Organizational

Performance

Indicator

Framework

Logical

FrameworkPrograms and Projects

Organizational

Outcomes/Major Final

Outputs

Sectoral Goals

Societal Goal:

Inclusive Growth and

Poverty Reduction

32

M&E Tools

• National Economic and Development Authority (lead)

• Department of Budget and Management (lead)

• Implementing Agencies

• National Government Agencies• Government Owned and

Controlled Corporations• Government Financial

Institutions

Key Players


M&E Reports/Documents

SONA Technical Report

(c/o OP-PMS)

Socio-Economic Report

(c/o NEDA)

Budget Accountability Reports

(c/o DBM)

Quarterly Loans Performance Report

Semestral Grants Report

Quarterly Cost Overrun Report

Semestral Alert Mechanism

Annual ODA Portfolio Review

PAPs

MFOs

Sector Outcomes

Inclusive Growth

and Poverty

Reduction

Au

dit

R

ep

ort

s


Survey on M&E Systems

• Objective

– capture existing M&E systems of IAs

– how M&E is mainstreamed in the agency

function

– institutional set-up towards effective

management of development results

Monitoring and Evaluation Staff

M&E at the Project Level: IA

• Results

– Response rate: 15 out of 28

– All 15 respondents have existing M&E systems

– Frequency and report type

Project Type IAs

ODA DENR, SC

LFP TESDA, NHA, NCDA, CHED

ODA and LFP DA, DAR, DSWD, DOH, MWSS, DOTC, DTI, DOST, ARG

Frequency Types

Weekly, Monthly ,

Quarterly,

Semestral, Annual

and as needed

Performance Reports, M&E Reports, PCRs,

BARs, Supervision Missions, Impact Evaluations

, ODA Portfolio Review Forms, BARs, DPs

Reports



• Users of M&E report

– Within IAs, OAs (NEDA, DBM, OP-PMS, COA, DOF),

Congress, DPs

• MIS

MIS Support IAs Software

Integrated MIS ARG VB.net, ASP

Individual IS,

databases and

spreadsheets

DA, DENR, DAR, DOTC, DSWD,

DTI, SC, TESDA, DOH, MWSS,

DOST, NHA, CHED, NCDA

MS Office, MySQL,

ASP, In-house

development

Ongoing MIS Development IAs

Department-wide MIS for M&E DTI, DSWD, DOST, NCDA

Use of geo-tagging for some projects DA, ARG



• M&E set-up

• M&E BudgetProvision of M&E Budget IAs

Yes. Included in the budget of the units

in-charge of M&E function of the IA

(budget for impact/ex-post not

included)

DOH, DSWD, MWSS, TESDA,

DOST, DTI, SC, DENR, DA, CHED,

NHA, NCDA, DAR

None. DOTC, ARG, MWSS

M&E Units IAs

Planning Units, Financial Management and

Regional Operations Group, Program

Management do overall M&E function

overseeing M&E activities of the agency and

also of PMOs

DOH, DSWD, DOTC,

MWSS, TESDA, DOST,

DTI, SC, ARG, CHED,

NCDA, NHA,

Same as previous but with an Special Office to

manage operations and M&E of ODA-assisted

projects

DENR, DAR, DA



• Annual M&E Plan

• Capacity Building

Conduct of Capacity Building IAs

Yes but not on a regular basis. DOH, DSWD, TESDA, DOST, DENR, DA,

ARG, CHED, NCDA, NHA, DAR

No. Relies on external entities. DTI, DA, ARG, MWSS

With M&E Plan IAs

Yes. Incorporated in the annual plan

of the units in-charge of M&E function

DOH, DSWD, TESDA, DOST,

DENR, DA, DAR, ARG, NHA,

CHED

None. DTI, DOTC, SC, MWSS, NCDA



Challenges

• At the Plan/Sector M&E Level

– Results Matrices – Organizational Performance

Indicator Framework linkage

– Emerging Management Information System

– Greater statistical support

– Vague indicators

– Poor or non-existent baselines

– Mismatch between indicator, baseline and target

– No disaggregated indicators and baselines

– Dearth of quality data

39

Challenges

• At the Program/Project M&E Level

– Need to establish M&E units for some agencies

– M&E for locally funded projects

– Coordination of oversight agencies around M&E

– Use of M&E data (e.g., poor quality of data and

analysis

– Limited engagement of stakeholders in M&E

processes

– Systems and mechanisms to support M&E

40

NEDA-DBM Joint Memorandum

Circular No. 2015-01

The National Evaluation Policy

Framework of the Philippines

Evaluation Policy

Evaluation is crucial

function that informs

decision-making. A set of policy statements on it will

promote and strengthen

the practice and use of

evaluations.

Why do we

need it? A policy framework to

guide evaluation

activities

What is it?


Circular on Evaluation

Signed on 15 July 2015

Policy Statement

Framework for the purposive conduct of

evaluations in the public sector in support of

good governance, transparency,

accountability and evidence-based decision-making.



Objectives

• support for evidence-based decisions

• promotion of program improvement

• promotion of accountability

Coverage

• All programs and projects of the Government



• evaluation criteria

• ensuring evaluation competencies

• observing standards of ethics

• evaluation plans in accordance with best

practices

• undertaking evaluations with due regard to

impartiality

• reporting, dissemination, and use of

evaluations

Guiding Principles/ Evaluation Standards



Implementing Agencies:• formulate and maintain a rolling 6-year agenda

• create neutral evaluation units initially at the central level

• evaluation plans during budget submission in accordance with best

practices

• ensure management response to evaluations and the use ofevaluations

Evaluation Task Force:• Members: NEDA, DBM, OP-PMS

• overall direction and coordination of the evaluation agenda of the

public sector

Evaluation Task Force Secretariat:• Secretariat support among others

Interim TWG:• NEDA and DBM

Major Players and Responsibilities

Annexes to the JMC

Annex Contents/Highlights

Annex A:Evaluation Criteria

Programs and projects will be evaluated in terms of the following:

(a) Relevance; (b) Effectiveness; (c) Efficiency; (d)Sustainability.

Annex B:Evaluation Competencies

Those engaged in designing, conducting and managing

evaluation should demonstrate competencies on: (a) technical

foundations; (b) leading, managing and delivering evaluations; (c)

communicating and sharing evaluation findings; and (d) integrity.

Annex C: Ethics Evaluators shall adhere to a set of ethics standards.

Annex D:Best Practices in Evaluation

Best practices in evaluation in terms of (a) evaluation scale, and

(b) evaluation design and execution are presented for guidance

of evaluators

Annex E:Impartiality

Impartiality is ensured to maximize objectivity and minimize

potential for bias.

Annex F: Reporting Guidelines on reporting, dissemination, and use of evaluations are

specified.

Major Challenges

• Varying levels of evaluation capacities

(competencies and skills) across

agencies

• Limited budget to conduct evaluations

• Set-up of evaluation unit remains a

challenge for some agencies (national

and local)



Initial Evaluation Activities

(1) Evaluation studies commissioned by NEDA:

1 Completed (NEDA-Regional Office) impact assessment–

completed agriculture project

4 Ongoing (NEDA-Central Office) impact evaluations–

completed projects in the rural roads, technical education

skills development, rail, and national shelter program

11 Ongoing (NEDA Regional Offices) –various types of

evaluation/sectors

• Training Course on Impact Evaluation for NEDA under the

M&E fund (commissioned in 2018)

(2) NEDA-UNDP Partnership “Project on Using Strategic

M&E to Accelerate Implementation of the PDP 2017-2022”

evaluations.gov.ph

Neda-Undp Strategic M&E Project

A NEDA Project under the M&E Fund

Supported by UNDP Philippines

Evaluation Studies

Policy &Guidelines

Capacity Assessmen

t

Community

of Practice

Evaluations Portal

Project Outcome:

Capacity of NEDA & select government agenciesto conduct PDP-linked evaluations is improved

Project Outputs:

• Commission eight (8) PDP-level

thematic evaluations in 2018

• Anti-Red Tape Act (ARTA)

• National Spatial Strategy (Ro-

Ro)

• National Nutrition Program

• Payapa at Masaganang

Pamayanan (Peace & Dev’t)

• Climate Change Action/Food

Security

• Early Childhood Care and

Development

• Economic Zones

• MSME

Evaluation Studies

• Support NEDA in quality

assurance, knowledge mgt.

& communication of its

evaluations

• Develop a roster of

consultants, firms &

academic institutions for

evaluations

• Develop guidelines and tools to

implement the National

Evaluation Policy Framework

• Develop National Evaluation

Agenda for PDP 2017-2022

National Evaluation Policy Guidelines

• Assess evaluation capacity in key

government agencies

• Conduct learning sessions on

NEPF guidelines & tools

• Develop core course modules &

Certification Programme for

NEDA and other stakeholders

Capacity Assessment& Learning

Evaluation Policy &Capacity Development

• Develop an online portal with a back-

end system that is structured around

the evaluation process

• Build visibility using social media

• Revitalize and expand the

M&E Network as a

community of practice

• Through the M&E Network,

conduct engagement

activities w/ stakeholders

• M&E Summit to be held in

Nov 12-13

Community of Practice

Evaluations Portal

Stakeholder Outreach &Online Visibility

evaluations.gov.ph

Government Evaluations Portal

Developed on behalf of NEDA by the

United Nations Development Programme

EvaluationsExplore independent

studies on the efficiency &

effectiveness of public

policies & programs

GuidelinesKnow more about the

National Evaluation

Policy Framework and its

implementing guidelines

LearningStudy online on various

areas of evaluation at

your own pace and

access knowledge

resources

CommunityLink up with the M&E

Network and keep up

with opportunities for

professional growth

Evaluation ManagersLog in here to manage your studies at various

stages of the evaluation process

A Decade of the Anti-Red Tape Act:

Have Frontline ServicesImproved Because of ARTA?

Read the Study Here

Jesse T. DavidNEDA-Monitoring and Evaluation Staff

[email protected]

Jesse T. DavidMonitoring and Evaluation Staff

National Economic and Development Authority

Why M&E is Important in the Government Sector?

3 April 2019Climate Change Commission