planning for measurement, monitoring and evaluation the reform
DESCRIPTION
Planning for Measurement, Monitoring and Evaluation the Reform. Assoc. Prof. Dr. Wongsa Laohasiriwong Faculty of Public Health, Khon Kaen University. What is it?. Monitoring and Evaluation (M&E) Plan. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/1.jpg)
Assoc. Prof. Dr. Wongsa Laohasiriwong
Faculty of Public Health, Khon Kaen University
Planning forPlanning for Measurement, Measurement, Monitoring and Evaluation the Monitoring and Evaluation the
Reform Reform
1
![Page 2: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/2.jpg)
2
What is it?What is it?
![Page 3: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/3.jpg)
Monitoring and Evaluation Monitoring and Evaluation (M&E) Plan(M&E) Plan
is a guide a guide as to what what you should evaluateyou should evaluate, what information what information you needneed, and whowho you are evaluating for.
3
Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159
![Page 4: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/4.jpg)
4
Characteristic of M&E Characteristic of M&E PlanPlan
UtilityUtility
- serve practical information needs of intended users.
Feasibility Feasibility - be realistic, prudent, diplomatic and frugal.
Propriety Propriety
- conducted legally, ethically, and with regard to those involved in and affected by the evaluation .
Accuracy Accuracy - reveal and convey technically accurate information.
Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159
![Page 5: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/5.jpg)
5
Developing & Developing & Implementing Implementing
an M&E Plan: Logic an M&E Plan: Logic ModelModel
Inputs
•Human resources
•Understanding of the program •Authority and mandate •Stakeholders
Processes•Advocate•Assess strategic information needs•Assess information systems capabilities •Achieve consensus and commitment•Develop mechanism for M&E plan review•Prepare document for final approval
Impacts
Improved health status
Long-term
•Evidence-based decisions for improving programs
OutcomesOutput
•M&E Plan Document
Short-term
•M&E System for obtaining Strategic Information decision making
![Page 6: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/6.jpg)
Performance Performance MeasurementMeasurement
6
is the ongoing monitoring ongoing monitoring and reporting of program and reporting of program accomplishments, accomplishments, particularly progress towards towards pre-established goals.pre-established goals.
Source: Stephanie Shipman and Joseph Wholey. 1998. Performance Measurement and Evaluation: Definitions and Relationships
![Page 7: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/7.jpg)
How to Develop a How to Develop a Monitoring and Evaluation Monitoring and Evaluation
Plan?Plan?
Step 1. - Step 1. - Identify your evaluation audienceevaluation audience.
Step 2. - Step 2. - Define the evaluation questions.evaluation questions.
Step 3. - Step 3. - Identify the monitoring questions.monitoring questions.
Step 4. - Step 4. - Identify the indicators and data indicators and data sourcessources..
Step 5. - Step 5. - Identify who is responsiblewho is responsible for data collection and timelines.
Step 6. - Step 6. - Identify who will evaluate the who will evaluate the data, howdata, how it will be reported, and when.
Step 7. - Step 7. - ReviewReview the M&E plan.7
Source: Developing a Monitoring & Evaluation Plan. From evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159
![Page 8: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/8.jpg)
Characteristics of Characteristics of Good Good
Monitoring SystemsMonitoring SystemsFollows the reform Follows the reform process.
Cost-effectiveCost-effective for the operating unit.
Describes the stages or stages or events used for rating events used for rating progressprogress .
Describes the methodology or process methodology or process used for rating the current stage of the reform.
8
Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process
![Page 9: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/9.jpg)
Characteristics of Characteristics of Good Good Monitoring Systems (cont.)Monitoring Systems (cont.)
Defines key terms Defines key terms e.g. “operational,” “fully functioning,” or “fully implemented”.
Specifies which activities activities are most critical to the are most critical to the strategic objective. strategic objective.
9
Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process
Provides a rationale Provides a rationale for how future performance targets are set.
Includes reform stakeholders as partners stakeholders as partners in reviewing the process of adoption and implementation and in setting future performance targets.
![Page 10: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/10.jpg)
Approaches to Approaches to MonitoringMonitoring
10
Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process
![Page 11: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/11.jpg)
1) Simple Quantitative 1) Simple Quantitative Approaches (SQA)Approaches (SQA)
11
Using a purely quantitative approach is a common way common way of showing the progressof showing the progress.
Typically the agency promotes a list of reforms during a three-year periodthree-year period, and each year each year it records records the number of reformsthe number of reforms that have been successfully successfully completedcompleted.Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process
![Page 12: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/12.jpg)
2) Descriptive Approaches 2) Descriptive Approaches (DA)(DA)
12
The progress in adopting and implementing reforms is reforms is frequently not readily frequently not readily quantifiablequantifiable. .
It needs clearly define in advance the key events the key events in the reform process reform process and the planned planned results results and then monitor their monitor their achievementachievement.
This permits a relatively objective comparison of planned with actual results throughout actual results throughout the performance period.the performance period.
Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process
![Page 13: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/13.jpg)
3) Composite Approaches 3) Composite Approaches (CA) (CA)
(Qualitative and Quantitative Methods) (Qualitative and Quantitative Methods)
13
Based on constructing more complex complex indicesindices.
These indices are frequently (but not necessarily) based on phases or stages phases or stages identified as milestone eventsidentified as milestone events and are expressed numericallyexpressed numerically.
Each phase or stage in the reform process is assigned a number or valuenumber or value..
Missions should provide a detailed Missions should provide a detailed explanation explanation to define each stagedefine each stage in a given index and to describe the related scoring scale. If not thoughtfully considered and well documented, can be imprecise and be imprecise and subjective.subjective.Source: Patricia Vondal. USAID. 2000. Monitoring the policy reform process
![Page 14: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/14.jpg)
MonitoringMonitoring
14
is collecting, recording, and collecting, recording, and reportingreporting information concerning performance that tracking of services and programmes performances.
Routine WorkingRoutine Working
Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control
![Page 15: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/15.jpg)
15
Why do we monitor?Why do we monitor?Simply because we know that things we know that things
don’t always go according to plandon’t always go according to plan (no matter how much we prepare).
To better assess how well a policy or how well a policy or programme is achievingprogramme is achieving its intended target.
To detect and reactdetect and react appropriately to deviations and changes to planschanges to plans
Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control
![Page 16: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/16.jpg)
16
What do we What do we monitormonitor??
Men (human resources)
MachinesMaterialsMoneyTechnology
SpaceTimeTasksQuality/Technical
Performance
What we looking for?
Progress, Schedule, Costs, Performances: Work begin,
the completionSource: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control
![Page 17: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/17.jpg)
17
When do we When do we monitor?monitor?
End of the Plan Continuously Regularly Logically While there is still time to react As soon as possible At task completion At pre-planned decision points (milestones)
Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control
![Page 18: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/18.jpg)
18
How do we How do we monitor?monitor?
Through information collection.Through meetings with stakeholders.For schedule – Update Gantt ChartsUsing Earned Value AnalysisCalculate Critical RatiosMilestonesReportsTests, Inspections, Auditing
Source: Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control
![Page 19: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/19.jpg)
EvaluationEvaluation is the systematic process of systematic process of the merit, value and worththe merit, value and worth of someone or something.
![Page 20: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/20.jpg)
Evaluation TermsEvaluation Terms EfficacyEfficacy : Can it work? (normally for
medicine) EffectivenessEffectiveness : Does it work in reality?
EfficiencyEfficiency : Is it worth doing it, compared to other things we could do with the same money? (Cost-effectiveness)
EvaluatiEvaluationon
![Page 21: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/21.jpg)
EffectivenessEffectivenessEffectiveness refers to the connection the connection between inputs and outputsbetween inputs and outputs, the achievement of objectivesachievement of objectives, means how well the job gets done i.e. the quality of the output.
![Page 22: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/22.jpg)
EfficiencyEfficiency Efficiency refers essentially to the
extent to which outputs are attained outputs are attained while minimizing production costs, while minimizing production costs, achievement of the ends with the least amount of resources.least amount of resources.
There are 2 types of EfficiencyThere are 2 types of Efficiency1. Technical efficiency 2. Allocative efficiency
![Page 23: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/23.jpg)
Cost-Benefit and Cost-Effectiveness Analyses
23
Types of Types of EvaluationEvaluation
Outcomes
Impacts
Inputs
Activities
Outputs
Implementation
Results
Impact Evaluation
Outcome Evaluation
Process Evaluation
![Page 24: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/24.jpg)
(1) Define the study question and perspectiveDefine the study question and perspectiveDescribe alternative health programs to be considered determine study perspective
(decision making context of the study)
(4) Derive decision rulesDerive decision rulesCompare cost–benefit ratios , identify other decision criteria address implementation
issues
(2) Identify, measure, and value costs and benefitsIdentify, measure, and value costs and benefitsIdentify relevant costs and benefits for each program measure costs and benefits for each
program in appropriate physical units value costs and benefits for each program, using market valuation of factor inputs, stated preference valuations of health gains, or other
valuation methods
(3) Analyse costs and benefitsAnalyse costs and benefitsAdjust costs and benefits for differential timing (discounting) where appropriate
perform incremental analysis of costs and benefits of alternatives perform sensitivity analysis to allow for uncertainty in estimates of costs and consequences
Four Stages of EvaluationFour Stages of Evaluation
![Page 25: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/25.jpg)
Once an organization has analyzed its mission, identified all its stakeholders, and defined its goals, it needs a way to measure progress toward those goals.
25
• Key Performance Key Performance Indicators Indicators are those are those measurements.measurements.
Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm
![Page 26: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/26.jpg)
Key Performance IndicatorsKey Performance Indicators, also known as KPIKPI or Key Key Success Indicators (KSI), Success Indicators (KSI), help an organization define and measure progress toward organizational goals.
26
Source: F. John Reh, Key Performance Indicators (KPI) How an organization defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm
![Page 27: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/27.jpg)
Key Performance Indicators (KPI)
“Measurable indicators that will be used to report progress that is chosen to reflect the critical success strategic”
27
Source: John McGerty, Key Performance Indicator
![Page 28: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/28.jpg)
Specific Measurable Acceptable Realistic Time frame
Source: Clara Freeman. http://www.aboutfreelancewriting.com/2012/03/what-working-smarter-means-for-the-freelance-entrepreneur/
![Page 29: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/29.jpg)
Specific Specific There has to be one
widely accepted definition of the KPI to make sure the different make sure the different users interpret it the users interpret it the same way same way and, as a result, come to the same result, come to the same and right conclusions and right conclusions which they can act on.
Source: John McGerty, Key Performance Indicator
![Page 30: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/30.jpg)
Measurable Measurable The KPI has to be measurable to define a standard, define a standard, budget or norm, to budget or norm, to make it possible to make it possible to measure the actual measure the actual valuevalue and to make the actual value comparable to the budgeted value.
Source: John McGerty, Key Performance Indicator
![Page 31: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/31.jpg)
AchievableAchievableIt is really important for the important for the acceptance of KPI’s and acceptance of KPI’s and Performance Management in Performance Management in general within the general within the organization that it possible to organization that it possible to achieved.achieved. Nothing is more discouraging than striving for a goal that you will never obtain.
Source: John McGerty, Key Performance Indicator
![Page 32: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/32.jpg)
RealisticRealistic The KPI must give
more insight in the performance of the organization in obtaining its goal and strategy. Therefore the action action need to happen.need to happen.
Source: John McGerty, Key Performance Indicator
![Page 33: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/33.jpg)
Time FrameTime FrameIt is important to important to express the value express the value of the KPI in time. of the KPI in time. The realization and standardization of the KPI therefore has to be time frame.
Source: John McGerty, Key Performance Indicator
![Page 34: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/34.jpg)
A business A business – the percentage of income that comes from return customers.
34
• A Health Promotion Hospital A Health Promotion Hospital – The percentage of coverage Polio vaccination in children age under 5 years old.
• A school A school – the graduation rates of students.• A Customer Service Department A Customer Service Department – the number of monthly customer complaint.
![Page 35: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/35.jpg)
Benefits of Evaluation Benefits of Evaluation FindingFinding
Important input for government Important input for government decision making and prioritizationdecision making and prioritization, particularly in the budget process.
Assists managers by revealing the revealing the performance of ongoing activities performance of ongoing activities at the project, program or sector levels.
Used to assess the performance of assess the performance of organizations and institutional organizations and institutional reform processes.reform processes.
Contribute to accountability Contribute to accountability mechanismsmechanisms, whereby managers and governments can be held accountable for the performance of their activities.35
Source: Keith Mackay. 1998. Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C.
![Page 36: Planning for Measurement, Monitoring and Evaluation the Reform](https://reader036.vdocument.in/reader036/viewer/2022062315/56815862550346895dc5bede/html5/thumbnails/36.jpg)
References: John McGerty, Key Performance Indicator F. John Reh, Key Performance Indicators (KPI) How an organization
defines and measures progress toward its goals, management.about.com/cs/generalmanagement/a/keyperfindic.htm
Clara Freeman. http://www.aboutfreelancewriting.com/2012/03/what-working-smarter-means-for-the-freelance-entrepreneur
Patricia Vondal. USAID. 2000. Monitoring the policy reform process. Developing a Monitoring & Evaluation Plan. From
evaluationtoolbox.net.au/index.php? option=com_content&view=article&id=20&Itemid=159.
Stephanie Shipman and Joseph Wholey. 1998. Performance Measurement and Evaluation: Definitions and Relationships.
Keith Mackay. 1998. Public Sector Performance-The Critical Role of Evaluation. The World Bank Washington, D.C.
Chapter 10 Project Monitoring and Control. http://www.scribd.com/doc/37667340/Ch10-Project-Monitoring-Control
36