11 results-based monitoring and evaluation in bank projects hd learning week. november 6, 2006 jody...
TRANSCRIPT
11
Results-based Monitoring and Evaluation in Bank projects
HD Learning Week. November 6, 2006Jody Zall Kusek & Mohamed Khatouri
World Bank
22
Session I- Overview of Results-Based Management Concepts
Traditional Management
ApproachFocused mainly on inputs and activities.
Results-Based Management Approach
Focus on the results obtained rather than just on the inputs used or the activities conducted.
Managing for results. Use the information to improve decision making and steer development interventions towards clearly defined goals.
Results agenda. Focus on development outcomes to be achieved by Bank operations, which requires high quality project design and supervision.
1. Better indicators + Monitoring = Better results
2. High quality Project design (specific focus on outcomes) + High quality implementation = Better results
Which statement is accurate ?
33
• Socio- economic context:
Results statement should reflect local needs and priorities
Critical factors for defining results to be achieved
• Resources: Level of resources will impact on what can realistically be achieved
• Local Capacity: Existing skills, leadership, and management capacity will impact on what can be implemented to achieve expected results
• Timetable: Results framework must identify the results (changes) to be achieved in the life of the program
Session I- Overview of Results-Based Management Concepts
44
Principles of Management for Development Results*Principles of Management for Development Results*
Tools for performance measurement to increase the effectiveness of development interventions
* Roundtable on Managing for Results in 2004
1. Focusing the dialogue on results at
all phases of the development process
2. Aligning programming, M&E
with results
5. Using results information for
learning and decision making
4. Managing for, not by, results
3. Keeping measurement &
reporting simple & cost-effective
Results-based M&E
Session I- Overview of Results-Based Management Concepts
55
Session I- Overview of Results-Based Management Concepts
Ensuring accountability
Improving internal management
Marketing successes
• Results are more important than processes: provide evidence that the program is producing longer term benefits.
• Focusing interventions/reallocating inputs to achieve results. Programs focused are better able to demonstrate results over time.
• Capturing lessons-learned: Showing general program Showing general program progress progress & sharing practices and innovations.
• Preoccupation with results is a global issue : demonstrate results against money spent
Why is Results-Based M&E Important?
• Secure political & public support: “If you can demonstrate results, you can win public support “
• Better reporting: Programs have effective M&E system develop results-oriented reporting
66
Session I- Overview of Results-Based Management Concepts
Regular collection and reporting of information to track whether actual results are being achieved as planned
Clarifies program objectives
Link inputs and activities to results to be achieved
Translates results to be achieved into performance indicators
Periodically collect data on the indicators and compare actual results with target
Reports progress and alerts management to problems in implementation
Indicator
Year
30%
40%
50%
1 2 3 4 5
20 %
Monitor
ing
77
Analyses why intended results were or were not achieved
Explores unintended results
Assesses causal contributions of activities to results and validate hypothesis Examines implementation processes
Analytical efforts to answer specific questions about performance of a program activities.
Oriented to answering WHY? And HOW?
Provides lessons learned and recommendations for improvement
With
project
With out
project
Indicator
Year
30%
40%
50%
1 2 3 4 5
15 %
Evaluation
Session I- Overview of Results-Based Management Concepts
88
Outcomes Outputs
Long-term, widespread improvement in society
Effects or behavior changes resulting from program outputs
Products and services to be used to simulate the achievement of results
Utilization of resources to generate products and services
Resources committed to program activities
ImplementationResults
Long-term Goal
(Impact) Activities Inputs
PLANING FOR RESULTS Attribution Gap
Traditional M&E
Results-based M&E
Session I- Overview of Results-Based Management Concepts
Results Building Blocs
99
Social Protection and labor
Increased coverage of crop-insurance programs
Increased % of
vulnerable population
using saftey net programs
Increased population
is food secure
Health
Doctors hired
Health workers trained
Increased use of health clinics
Improved maternal mortality
Education
Teachers trained
Text Books provided
Increased student
completion rates
Increase literacy rates
Examples of Results Chain
Long-Term Goal
(Impact)
Outcomes
Outputs
Session I- Overview of Results-Based Management Concepts
1010
Session I- Overview of Results-Based Management Concepts
1. Information is available for parents about the importance of breast feeding
2. Children in Local Community healthier
3. Fewer children are having diarrhea diseases
4. Mothers Parents breast feeding rather than using formula
5. New funds available to implement a health project to reduce babies mortality rates
6. Implement information campaigns on the importance of breast feeding
Example of Results Chain (Health Sector)
Identify the sequence of inputs, activities, outputs, outcomes, and impacts
1111
Responsive to Responsive to information management and policy making of government institutions at the central and local level
M&E M&E capacity in placecapacity in place for collecting, for collecting, analyzing, and reporting performance analyzing, and reporting performance informationinformation
Built around Built around reporting requirementsreporting requirements
Active Active involvement of customersinvolvement of customers and and partners (planning, conducting, reviewing, partners (planning, conducting, reviewing, & interpreting performance information)& interpreting performance information)
Fully Fully integrated into integrated into the government’s existing management systemsmanagement systems at the central and local level
Critical Elements of an Effective M&E SystemCritical Elements of an Effective M&E System
Session I- Overview of Results-Based Management Concepts
1212
Country’s National Poverty Reduction Strategy (PRSP) Provides a central framework for donor coordination
PRSP Strategic
GoalsOutcomes
Country Assistance Country Assistance Strategy (CAS) to Strategy (CAS) to support the national support the national development development strategystrategy
CAS
Operationalize Operationalize the CASthe CAS
AAA
Policy Dialogue
Operations Projects/Programs
Align Align Projects Projects and and Programs Programs Results Results with with CAS & CAS & PRSP PRSP ResultsResults
Design Logic of Bank OperationsDesign Logic of Bank Operations
Session I- Overview of Results-Based Management Concepts
1313
Exercise 1. Develop Results Building Blocs
Assignment
• Break out into groups of 4-5 persons
• Each group should develop a results building bloc (inputs, outputs, outcomes, & impacts) for at least 2 programs. Spend no more than 15 minutes per program
• Select one case to report on in the plenary session.
Session I- Overview of Results-Based Management Concepts
1414
Session II- Development of Results Framework
• Identifies the Program Development Objective (PDO) and Intermediate Outcomes which are critical to achieve it
Program’s strategy for achieving specific objective
• Illustrates the development hypothesis and the cause & effect relationships linking all levels of a program’s objectives
(IF a and b happen THEN c should happen)
• Building consensus and ownership
An effective tool for:
• Strategic clarity and planning & managing for results
• Effective communication
Results Framework
1515
Session II- Development of Results Framework
Reduction in the child mortality
1. 1. Determine the key Determine the key
development hypothesis development hypothesis
& its contribution to & its contribution to
higher order impactshigher order impacts
THEN incidence of preventable disease will be
reduced
Causal ChainCausal Chain
Hypothesis. Results are caused by some interventions
IF… THEN IF we build health centers THEN local population will use them to for health services
IF we train teachers on new pedagogic skills THEN teachers will use these skills to improve the quality of education
IF expand child immunizations
Design with clear logicDesign with clear logic
1616
2. 2. Develop a hierarchy of objectives showing the causal linkagesDevelop a hierarchy of objectives showing the causal linkages
Reduction in Child Mortality
Expanded health care system
Improved nutrition
Provide Funds
Incidence of preventable
disease reduced
Increased immunizations
Increased family health practices
Expand facilities
Public awarenes
s
3. 3. Determine what is possible in the operation (program or project)Determine what is possible in the operation (program or project)
Vaccines availabilit
y
DESIGN. Project success depends on the development of the right activities & assumptions
IMPLEMENTATION. Each part of the results chain has a role to play in achieving the Project Development Objective
Design with clear logic (Cont.)Design with clear logic (Cont.)
Session II- Development of Results Framework
Enhance regulations
1717
Session II- Development of Results Framework
Program Development Objective (PDO)
The PDO describes the effect that the program’s outputs will have on the beneficiaries in terms of changed behavior or improved performance. It defines the program’s success.
The PDO outcomes are one step beyond the products of the activities.
At the close of the program, what problem has been solved for the key recipient of the program outputs ?
PDO example: “To achieve improved, expanded, sustainable utilization of basic social and economic services and to support a governance system where local government and communities can gradually become mutually accountable.
Is the PDO precise ?
Focus on the outcome that the project can directly influence, given its duration, resources, and approach
Focus on the expected outcome for the primary beneficiaries
Keep the PDO statement clear, precise, and stated as realistic results— not overly broad or too ambitious
1818
Example 1: To provide grant financing to educate “at risk” groups in the risks of HIV/AIDS.
Example 2: To increase the awareness of the dangers of HIV/AIDS of “at risk” groups.
Which outcome example describe what is going to be transformed?
PDO Outcomes. Capture single outcome by outcome statement
Example: PDO objectiveTo expand access and improve quality of primary education throughout the country.
Outcomes to be achieved ?
……………… ………………
Program Development Objective (Cont.)
Session II- Development of Results Framework
Focus on what is going to be transformed (not what is going to be done)
1919
Avoid selection of too many indicators Avoid selection of too many indicators Less is Less is Better!Better!
Work with stakeholdersWork with stakeholders and technical experts to and technical experts to determine which indicators are most importantdetermine which indicators are most important
7
Information about indicators should be Information about indicators should be easy to easy to gathergather and useful for management decisions and useful for management decisions
Performance indicators should be Performance indicators should be identified at all identified at all levelslevels of results chain. of results chain.
IndicatorIndicator: A variable that provides accurate and : A variable that provides accurate and reliable evidence about the achievement of a reliable evidence about the achievement of a specific result specific result
o ““What gets measured gets doneWhat gets measured gets done””o ““If you don’t measure results, you If you don’t measure results, you
can’t tell success from failure”can’t tell success from failure”
Performance Performance IndicatorsIndicators
Session II- Development of Results Framework
20208
SpecificSpecific: : measure as closely as possible what is intended to measure
MMeasurableeasurable:: clear and unambiguous about what is measured
Outcome Indicator
Parents insure that 1.Increased utilization of clinics
Children get treatment for 2. Increased use of malaria
Malaria drugs
Which indicator is more specific ?
2. % of health centers with availability of drugs
1. % of health centers without stock out of drugs x, y & z for more than a week at a time Which indicator is
measurable ?
Performance indicators should bePerformance indicators should be
SMARTSMART::SSpecific; pecific; MMeasurable; easurable; AAttributable; ttributable; RRealistic; ealistic; TTargetedargeted
Criteria for Selecting Good IndicatorsCriteria for Selecting Good Indicators
Session II- Development of Results Framework
2121
2. HIV prevalence among the total
population
1. HIV prevalence among 15-24 year old pregnant
women
1. Percent increase in employment
8
RealisticRealistic: : data obtained at reasonable cost with enough frequency Which
indicator is more realistic?
TargetedTargeted: : it should be specific about the targeted population/area
2. Percent increase in employment of graduate of technical training center X in the first year after completion of training.
Which indicator is targeted ?
1. Life expectancy
Attributable: attributable to the project’s or program’s efforts
2. % of children fully immunized at 1 yearWhich indicator is attributable ?
Criteria for Selecting Good Indicators (Cont.)Criteria for Selecting Good Indicators (Cont.)
Session II- Development of Results Framework
2222
PDO Outcomes Outcomes Indicators
Intermediate Outcomes by Component
Intermediate Outcome Indicators
• Increased coverage of clinics providing immuniatio
PDO: Increased use of health facilities
• Improved quality of doctors/nurses
• % of under 5 yr covered by DPT immunization
Component I- Extension of Quality Health Services • % of facilities without 7-day
stock outs of essential drugs (List of essential drugs defined)
• Sustained availability of essential drugs
Component II. Development of Human resources
• Reduced shortage of human resources
• % of facilities with minimum staffing norms (List of minimum staffing defined)
• % of health deliveries carried out in public health facilities (%)
Session II- Development of Results Framework
EXAMPLE
2323
DefinitionDefinition.. Clear and precise definition of Clear and precise definition of what will be measured and unity of what will be measured and unity of measure.measure.
Critical AssumptionsCritical Assumptions.. External factors that could External factors that could significantly affect the achievement of the program significantly affect the achievement of the program results targetsresults targets
7
Baseline & TargetsBaseline & Targets Baseline. Value of the indicator at the beginning of the program. Used as a point of comparison when measuring progress toward a specific result
Target values. The intended value of the indicator at the end of a specified point in time, against which actual results will be measured
Data Acquisition MethodData Acquisition Method.. Data source, Data source, frequency/schedule, and responsibility for its collectionfrequency/schedule, and responsibility for its collection
Data Analysis & Reporting MethodData Analysis & Reporting Method.. Frequency/schedule, Analysis method, and responsibility of Frequency/schedule, Analysis method, and responsibility of reporting.reporting.
Each Indicator must have a M&E Each Indicator must have a M&E planplan
Session II- Development of Results Framework
2424
Indicators
(Definition % unit)
Base Line & Target Values (/Year)
BL 1 2 3 4
Data Collection & Reporting
Frequency/Schedule
Instrument (Method)
Responsible Party
• % of health deliveries carried out in public health facilities (%):
25 28 30 35 40 Annually (June)
Routine administrative records
Ministry of Health
10
Example of a M&E plan for one indicator
Session II- Development of Results Framework
2525
Impacts
InterventionsInterventions
High level outcomes
Processes, Policies, & OutputsExample:
Provide essential obstetrical health services
Example: Reduced maternal mortality
Intermediate outcomes which the program is expect to influence directly through its defined interventions during the program period.
Intermediate Outcomes
Ways to Improve the development of Results Ways to Improve the development of Results FrameworksFrameworks
Example: Birth attended by skilled staff
Session II- Development of Results Framework
2626 Health services provided ( Heath workers trained, Health facilities built & rehabilitated, Pharmaceutical drugs purchased)
Health services provided ( Heath workers trained, Health facilities built & rehabilitated, Pharmaceutical drugs purchased)
Reduced Maternal Mortality
Example: Health Sector
Hygiene and
sanitation practices
Increased use of maternal health
services
Improved access to family planning
Increased adoption of prevention methods
Availability of ITNs
Provision of
contracep-tives and
counseling to women and youth
Increased coverage
of Antenatal, new born, emergency obstetric,
& post-natal care
Improved Communi-
cation networks
& ambulance
s
Information
on family planning available
Session II- Development of Results Framework
2727
Key Informant Interviews
Community Interviews Review of
official records
One-Time Survey
PanelSurveys
Impact Evaluation
(Experimental Design)
Census
Statistical Emphasis
Rigor/Difficulty (Cost/Time/skills)
Focus Group
Interviews
Formal M
ethods
Rapid Appraisal
MethodsDirect
Observation
Validity Reliability Credibility
Choice
depends
1. Issue to be examined 2. Quality of the information needed 3. Time frame in which information is
needed4. Cost
Data Collection Methods
Session II- Development of Results Framework
2828
Key Steps for planning an
Evaluation 1. Clarify the evaluation purpose and
audience2. Identify the evaluation questions
3. Select appropriate methods based on the questions to
answer
a. What % of women using health facilities?
b. Why didn’t more parents send their kids to school?
c. Did the CDD project contribute to the increase in income of local communities?
Example. Which data collection method is more appropriate in providing an answer to each of the following questions?
4. Prepare data collection and analysis
plan
Session II- Development of Results Framework
29297
Design/PAD
• Develop the results framework with SMART indicators
• Develop the M&E plan with baselines & targets
PCN
• Develop causal chain
• Define appropriate PDO and outcomes to be achieved
Completion (ICR)
• Review and validate the causal chain
• Report on the Project outcomes
Implementation (ISR)
- Report on outputs & outcomes
- Adjust project design (as necessary to achieve outcomes)
Focus on outcomes all phase of the project Focus on outcomes all phase of the project cyclecycle
Session II- Development of Results Framework
3030
Assignment Break out into groups of 4-5 person
Review the quality of the PDO statement and its corresponding outcome indicators for 2 projects
Report on one case in the plenary session
Exercise 2.Exercise 2. Assess the quality of the Project Assess the quality of the Project Development Objectives (PDO) & outcome Development Objectives (PDO) & outcome indicators of selected projects indicators of selected projects
Session II- Development of Results Framework