workshop ii project management...1 unitar-hoap afghanishtan fellowship program 2007 ws ii 1 workshop...
TRANSCRIPT
1
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 1
Workshop IIProject Management
UNITAR-HIROSHIMA FELLOWSHIP FOR AFGHANISTAN 2007Introduction to Project Management 15–17 August 2007, Dehradun, India
Presented by: Jobaid Kabir, Ph.D.
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 2
Part 7. Monitoring and Evaluation
2
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 3
Monitoring and Evaluation
Monitoring and Evaluation basicsPurpose of Monitoring and EvaluationStakeholder Participation in Monitoring and Evaluation Planning and Managing an Evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 4
Steps of Planning and Managing an Evaluation1. Planning Evaluations 2. Defining Evaluation Questions
and Measurement Standards 3. Data Collection Process 4. Managing the Evaluation Process 5. Communicating and Using
Evaluation Results 6. Evaluation Standards
3
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 5
In planning evaluation activities decide on…
WHY: purpose, users of evaluation findings and howWHAT: objectives and the questions to addressHOW: data sources & collection methods WHO: will do evaluations, required expertise, need for external consultants, stakeholder involvementWHEN: timing of evaluations to help decision makingRESOURCES: budget needs
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 6
Evaluation Analytical Process
Evaluation Objectives
Evaluation Objectives
Overall Conclusions and lessons learned
Overall Conclusions and lessons learned
Overall Recommendations
Overall Recommendations
QuestionsQuestions Detailed Conclusions
Detailed Conclusions
Detailed Recommendations
Detailed Recommendations
Data Collection Methods
Data Collection Methods FindingsFindings Specific DecisionsSpecific Decisions
AnalysisSynthesisInterpretationAssessment
AnalysisSynthesisInterpretationAssessment
AnalysisSynthesisInterpretationAssessment
AnalysisSynthesisInterpretationAssessment
Evaluation Purpose
Evaluation Purpose
Source: Danida, 1999.
4
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 7
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 8
Performance IndicatorsMeasures:
InputsProcessesOutputsOutcomes
Enables:Tracking progressDemonstrate resultsTake corrective action
Stakeholder participation improves their understanding and allows them to use indicators for decision-making
5
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 9
Use of Performance Indicators
Setting targets and assessing progress toward achieving themIdentifying problems through early warning system to take corrective actionTells whether an in-depth evaluation is needed
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 10
Advantages of Performance Indicators
Effective means to measure progress towards objectivesFacilitates benchmarking:
With different organizational unitsWith districtsOver time
6
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 11
Disadvantages of Performance Indicators
Poorly defined indicators are not good measures of successTendency to define too many indicators, or those without accessible data sources, making system costly, impractical and likely to be underutilizedOften a trade-off between picking the optimal or desired indicators and having to accept the indicators which can be measured using existing data
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 12
Cost for Performance Indicators
Ranges from low to highDepends on:
Number of indicators collectedFrequencyQuality of information sought
7
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 13
Skills Needs for Performance Indicators
Well trained people for defining practical indicatorsMIS skills are required for implementing performance monitoring systems
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 14
Time Needed for Performance Indicators
Depends on:Extent of participatory processProgram complexity
May take 6–12 months
8
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 15
Tools of Monitoring and Evaluation
Performance indicatorsLogical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 16
The Logical Framework (LogFrame) Approach
Helps clarify objectives of a project, program, or policyA vehicle for engaging partners in clarifying objectives and designing activitiesServes as a useful tool to review progress and take corrective action
9
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 17
LogFrame Approach (Cont.)
Helps identify expected causal links between inputs, processes, outputs, outcomes and impactLeads to identification of performance indicators at each stage in this chain, as well as risks which might impede the attainment of the objectives
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 18
Contents of a logical framework
External factors realized to obtain planned outputs on schedule?Actions outside the control of the project necessary for project inception
What are sources of information?
Indicators should be included in all activities.
Activities: Undertaken to accomplish outputs.
Factors outside the control of the project which, if not present, may limit progress from outputs to achievements of project purpose?
What sources of information?
Quantity of outputs and by when will they be produced? (Quantity, quality, time)
Outputs: Produced by the project in order to achieve project purpose
Necessary external factors if achievements of project purpose is to help reaching project goal?
What sources of information exist or can be provided cost-effectively? Does provision for collection need to be made under inputs-outputs?
Quantitative measures of impacts and benefits. (Estimated time)
Purpose:
Necessary external factors for sustaining objectives?
What sources of information exist, or can be provided cost-effectively?
Quantitative judgment of objectives achieved? (Estimated time)
Goal:
AssumptionsVerificationIndicatorsSummary
10
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 19
LogFrame UseImprove quality of project and program designs by requiring the specification of clear objectives, the use of performance indicators and assessment of risksSummarize design of complex activitiesAssist the preparation of detailed operational plansProvide objective basis for activity review, monitoring, and evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 20
Advantages of LogFrameEnsures that decision-makers ask fundamental questions and analyze assumptions and risksEngages stakeholders in the planning and monitoring processWhen used dynamically, it is an effective management tool to guide implementation, monitoring and evaluation
11
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 21
Disadvantages of LogFrame
If managed rigidly, stifles creativity and innovationIf not updated during implementation, it can be a static tool that does not reflect changing conditionsTraining and follow-up are often required
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 22
Cost for LogFrameLow to mediumDepends on extent and depth of participatory process
12
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 23
Skills Needs for LogFrameFacilitation skillAdvanced facilitation skills for participatory planning and management
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 24
Time needed for LogFrame• Depends on scope and depth of
participatory process• Several days to several months
13
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 25
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 26
Theory-Based EvaluationAllows in-depth understanding of the workings of a program or activity—the “program theory” or “program logic.” Need not assume simple linear cause-and-effect relationships
14
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 27
Theory-Based Evaluation (Cont.)
By mapping causal factors and how they interact, it helps decide
Which steps should be monitoredHow well they are in fact borne out
Allows identifying critical success factors
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 28
Theory-Based Evaluation (Cont.)
When data shows that these factors have not been achieved, the conclusion is that the program is less likely to be successful in achieving its objectives
15
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 29
Example of Theory-Based Evaluation
Success of a program to improve literacy levels by increasing the number of teachers might depend on:
Availability of classrooms and textbooksLikely reactions of parents, school administration and childrenSkills and morale of teachers Districts where extra teachers are to be locatedReliability of funding
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 30
Theory-Based Evaluation Use
Mapping design of complex activitiesImproving planning and management
16
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 31
Advantages of Theory-Based Evaluation
Provides early feedback about what is or is not working and whyAllows early correction of problems as they emergeAssists identification of unintended side-effects of the programHelps prioritize issues to investigate in greater depth by using focused data collection or sophisticated M&E techniquesProvides basis to assess the likely impacts of programs
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 32
Disadvantages of Theory-Based Evaluation
Can become overly complex if the scale of activities is large or if an exhaustive list of factors and assumptions is assembledStakeholders might disagree about which determining factors they judge important, which can be time-consuming to address
17
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 33
Cost for Theory-Based Evaluation
Medium to highDepends on
Depth of analysisDepth of data collection
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 34
Skills Needs for Theory-Based Evaluation
Facilitation skillAnalytical skill
18
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 35
Time needed for Theory-Based Evaluation
Depends onDepth of the analysisDuration of the program or activityDepth of M&E work undertaken
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 36
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
19
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 37
Formal SurveysCan be used to collect standardized information from a carefully selected sample populationSurveys often collect comparable information for a relatively large population in target groups
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 38
Formal Surveys UseProvides baseline data against which performance of the strategy, program or project can be comparedCompares different groups at a given point in timeCompares changes over time in the same group
20
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 39
Formal Surveys Use (Cont.)
Compares actual conditions with the targets established in a program or project designDescribes conditions in a particular community or groupProvides a key input to a formal evaluation of the impact of a program or project
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 40
Advantages of Formal Surveys
Findings from the sample of people interviewed can be applied to the wider target group or populationQuantitative estimates can be made for the size and distribution of impacts
21
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 41
Disadvantages of Formal Surveys
Results are often not available for a long periodData processing and analysis can be a major bottleneck for the large surveysExpensive and time-consumingSome kinds of information are difficult to obtain through formal interviews
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 42
Cost for Formal SurveysRanges from roughly $30–60 per household in the United States Costs will be significantly higher if there is no master sampling frame for population to be surveyed
22
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 43
Skills Needs for Formal Surveys
Sound technical and analytical skills for sample and questionnaire design, data analysis and processing
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 44
Time needed for Formal Surveys
Depends on sample sizeMay require 6 months to 2 years
23
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 45
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 46
Rapid Appraisal MethodsA quick, low-cost way to gather the views and feedback of beneficiaries and other stakeholders, in order to respond to decision-makers’ needs for information
24
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 47
Rapid Appraisal Methods Use
Provide rapid information for management decision-making for projects or programsProvides qualitative understanding of
complex socioeconomic changeshighly interactive social situations, or people’s values and motivations
Provide context and interpretation for quantitative data collected by more formal methods
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 48
Advantages of Rapid Appraisal Methods
Low costCan be conducted quicklyProvides flexibility to explore new ideas
25
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 49
Disadvantages of Rapid Appraisal Methods
Findings usually relate to specific communities or localities—thus difficult to generalizeLess valid, reliable and credible than formal surveys
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 50
Cost for Rapid Appraisal Methods
Low to medium, depending on the scale of methods adopted
26
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 51
Skills Needs for Rapid Appraisal Methods
Non-directive interviewingGroup facilitationField observationStatistical skills
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 52
Time needed for Rapid Appraisal Methods
Four to six weeksDepends on size and location of the population interviewed and the number of sites observed
27
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 53
Rapid Appraisal MethodsKey informant interviewFocus group discussionCommunity group interviewDirect observationMini-survey
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 54
Rapid Appraisal MethodsKey informant interview
Series of open-ended questions posed to individuals selected for their knowledge and experience in a topic of interestInterviews are qualitative, in-depth and semi-structured
28
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 55
Rapid Appraisal MethodsFocus group discussion
Facilitated discussion among 8–12 carefully selected participants with similar backgroundsParticipants might be beneficiaries or program staffThe facilitator uses a discussion guideNote-takers record comments and observations
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 56
Rapid Appraisal MethodsCommunity group interview
Series of questions and facilitated discussion in a meeting open to all community membersThe interviewer follows a carefully prepared questionnaire
29
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 57
Rapid Appraisal MethodsDirect observation
Use of a detailed observation form to record what is seen and heard at a program siteInformation may be about ongoing activities, processes, discussions, social interactions and observable results
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 58
Rapid Appraisal MethodsMini-survey
A structured questionnaire with a limited number of close-ended questions that is administered to 50–75 peopleSelection of respondents may be random or ‘purposive’ (interviewing stakeholders at locations such as a clinic for a health care survey)
30
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 59
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 60
Participatory MethodsProvide active involvement in decision-making for those with a stake in a project, program, or strategy and generate a sense of ownership in the M&E results and recommendations
31
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 61
Participatory MethodsUse
Learn about local conditions and local people’s perspectives and priorities to design more responsive and sustainable interventionsIdentify problems and trouble-shooting problems during implementationEvaluate a project, program, or policyProvide knowledge and skills to empower public
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 62
Advantages of Participatory Methods
Examines relevant issues by involving key players in the design processEstablishes partnerships and local ownership of projectsEnhances local learning, management capacity and skillsProvides timely and reliable information for decision-making
32
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 63
Disadvantages of Participatory Methods
Sometimes regarded as less objectiveTime-consuming if key stakeholders are involved in a meaningful wayPotential for domination and misuse by some stakeholders to further their own interests
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 64
Cost for Participatory Methods
Low to mediumCosts vary greatly, depending on scope and depth of application and on how local resource contributions are valued
33
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 65
Skills Needs for Participatory Methods
Facilitation skill
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 66
Time needed for Participatory Methods
Varies greatly, depending on scope and depth of application
34
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 67
Commonly Used Participatory Tools
Stakeholder analysisParticipatory rural appraisal Beneficiary assessmentParticipatory monitoring and evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 68
Commonly Used Participatory ToolsStakeholder analysis
The starting point of most participatory work and social assessmentsUsed to develop an understanding of the power relationships, influence and interests of various people involved in an activity and to determine who should participate and when.
35
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 69
Commonly Used Participatory ToolsParticipatory rural appraisal
A planning approach focused on sharing learning between local people, both urban and rural, and outsidersEnables development managers and local people to assess and plan appropriate interventions collaboratively often using visual techniques so that non-literate people can participate
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 70
Commonly Used Participatory ToolsBeneficiary assessment
Involves systematic consultation with project beneficiaries and other stakeholders to identify and design development initiatives, signal constraints to participation, and provide feedback to improve services and activities.
36
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 71
Commonly Used Participatory ToolsParticipatory M&E
Involves stakeholders at different levels working together to identify problems, collect and analyze information and generate recommendations
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 72
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
37
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 73
Public Expenditure Tracking Surveys (PETS)
Tracks flow of public funds and determine the extent to which resources actually reach the target groupsExamines the manner, quantity, and timing of releases of resources to different levels of government, particularly to the units responsible for the delivery of social services
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 74
PETS (Cont.)
Often implemented as part of larger service delivery and facility surveys which focus on the quality of service, characteristics of the facilities, their management, incentive structures, etc.
38
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 75
PETS UseDiagnosing problems in service delivery quantitativelyProviding evidence on delays, “leakage” and corruption
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 76
Advantages of PETSSupports the pursuit of accountability when little financial information is availableImproves management by pinpointing bureaucratic bottlenecks in the flow of funds for service delivery
39
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 77
Disadvantages of PETSGovernment agencies may be reluctant to open their accounting booksCost could be substantial
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 78
Cost for PETSCan be high until national capacities to conduct them have been established
40
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 79
Skills Needs for PETSSound technical and analytical skills for sample and questionnaire design, data analysis and processing, and good understanding of sector to be assessed
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 80
Time needed for PETSThree to six months
41
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 81
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 82
Cost-Benefit & Cost-EffectivenessAnalysis
Cost-benefit and cost-effectiveness analysis are tools for assessing whether or not the costs of an activity can be justified by the outcomes and impactsCost-benefit analysis measures both inputs and outputs in monetary termsCost-effectiveness analysis estimates inputs in monetary terms and outcomes in non-monetary quantitative terms (such as improvements in student reading scores)
42
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 83
Cost-Benefit & Cost-EffectivenessAnalysis Use
Informing decision makers about the most efficient allocation of resourcesIdentifying projects that offer the highest rate of return on investment
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 84
Advantages of Cost-Benefit & Cost-EffectivenessAnalysis
Good quality approach for estimating program and project efficiencyMakes explicit economic assumptions that might otherwise remain implicit or overlooked at the design stageUseful for convincing policy-makers and funding agencies that the benefits justify the activity
43
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 85
Disadvantages of Cost-Benefit & Cost-EffectivenessAnalysis
Fairly technical, requiring adequate financial and human resourcesRequired data may not be available and projected results may be highly dependent on assumptions madeResults must be interpreted with care where benefits are difficult to quantify
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 86
Cost for Cost-Benefit & Cost-Effectiveness Analysis
Varies greatly, depending on scope of analysis and availability of data
44
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 87
Skill Needs for Cost-Benefit & Cost-EffectivenessAnalysis
Require technical skill in economic analysis and availability of relevant economic and cost data
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 88
Time needed for Cost-Benefit & Cost-EffectivenessAnalysis
Varies greatly depending on scope of analysis and availability of data
45
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 89
Tools of Monitoring and Evaluation
Performance indicatorsThe logical framework approachTheory-based evaluationFormal surveysRapid appraisal methodsParticipatory methodsPublic expenditure tracking surveysCost-benefit and cost-effectiveness analysisImpact evaluation
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 90
Impact EvaluationSystematic identification of the effects – positive or negative, intended or not – on individual households, institutions and the environment caused by program or project
46
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 91
Impact Evaluation (Cont.)Helps understand the extent to which activities reach the poor and the magnitude of their effects on people’s welfare
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 92
Impact Evaluation (Cont.)For large scale sample surveys project populations and control groups are compared before and after at several points during program intervention
47
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 93
Impact Evaluation (Cont.)For small-scale rapid assessment and participatory appraisals estimates of impact are obtained from combining group interviews, key informants, case studies and available secondary data
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 94
Impact Evaluation UseMeasures outcomes and impacts of an activity and distinguishes these from the influence of other external factorsHelps clarify whether costs for an activity are justified
48
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 95
Impact Evaluation Use Cont.)
Tells decision makers whether to expand, modify or eliminate projects, programs or policiesDraws lessons for improving design and management of future activities
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 96
Impact Evaluation Use Cont.)
Compares effectiveness of alternative interventionsStrengthens accountability for results
49
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 97
Advantages of Impact Evaluation (Cont.)
Provides estimates of the magnitude of outcomes and impacts for different demographic groups, regions or over timeSystematic analysis and rigor can give managers and policy-makers added confidence in decision-making
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 98
Advantages of Impact Evaluation
Provides answers to central development questions:
To what extent are we making a difference?What are the results on the ground? How can we do better?
50
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 99
Disadvantages of Impact Evaluation
Some approaches are very expensive and time-consumingReduced utility when decision-makers need information quicklyDifficult to identifying an appropriate alternative
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 100
Cost for Impact EvaluationNumber of World Bank impact evaluations have ranged from $200,000-$900,000 depending on program size, complexity and data collectionRapid impact evaluations can be conducted for as little as $10,000-$20,000
51
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 101
Training Needs for Impact Evaluation
Strong technical skills in:Social science research designManagementAnalysis and reportingQuantitative and qualitative research skills
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 102
Time needed for Impact Evaluation
Can take up to 2 years or moreRapid assessment evaluations can often be conducted in less than 6 months.
52
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 103
Examples Impact Evaluation1. Randomized pre-test post-test
evaluation2. Quasi-experimental design with
before and after comparisons of project and control populations
3. Ex-post comparison of project and non-equivalent control group
4. Rapid assessment ex-post impact evaluations
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 104
1. Randomized pre-test post-test evaluation• Design:
Subjects (families, schools, communities etc) are randomly assigned to project and control groupsQuestionnaires or other data collection instruments are applied to groups before and after the project interventionAdditional observations may be made during project implementation
53
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 105
1. Randomized pre-test post-test evaluation
Example:Water supply and sanitation services where the demand exceeds supplyBeneficiaries are selected by lottery
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 106
2. Quasi-experimental design with before and after comparisons of project and control populations• Design:
Where randomization is not possible, a control group is selected which matches the characteristics of the project group as closely as possibleWhere projects are implemented in phases, participants selected for subsequent phases can be used as the control for the first phase project group
54
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 107
2. Quasi-experimental design with before and after comparisons of project and control populations
Example:These models have been applied in World Bank low-cost housing programs in El Salvador, Zambia, Senegal and the Philippines
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 108
3. Ex-post comparison of project and non-equivalent control group
• Design:Data collected on project beneficiaries and a non-equivalent control group is selected as Model Data collected only after project implementationMultivariate analysis is often used to statistically control for differences in the attributes of the two groups
55
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 109
3. Ex-post comparison of project and non-equivalent control group
• Example:Assessing the impacts of micro-credit programs in BangladeshVillages where micro-credit programs were operating were compared with similar villages without these credit programs
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 110
4. Rapid assessment ex-post impact evaluations• Design:
Some evaluations only study groups affected by the project while others include matched control groupsParticipatory methods can be used to allow groups to identify changes resulting from the project, who has benefited and who has not, and what were the project’s strengths and weaknesses
56
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 111
4. Rapid assessment ex-post impact evaluations (Cont.)• Design:
Triangulation is used to compare the group information with the opinions of key informants and information available from secondary sourcesCase studies on individuals or groups may be produced to provide more in-depth understanding of the processes of change
UNITAR-HOAP Afghanishtan Fellowship Program 2007 WS II 112
4. Rapid assessment ex-post impact evaluations• Example:
Assessing community managed water supply projects in Indonesia