lesson 8: effectiveness macerata, 11 december alessandro valenza, director, t33 srl
TRANSCRIPT
Lesson 8: Effectiveness
Macerata, 11 December
Alessandro Valenza , Director, t33 srl
Agenda
What does Effectiveness mean?
Assessing Effectiveness
Multicriteria analysis
Effectiveness (definition 1)
The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgement about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional developmental impact.
(OCSE DAC)
Effectiveness (definition 2 )
The term effectiveness has many possible meanings. The most common definition identifies effectiveness with “achievement of objectives”. This leaves open the definition to the different meanings of “objectives”. Objectives can be expressed quantitatively in terms of expected output or results. The effectiveness is evaluated simply by comparing what has been obtained with what had been planned: outputs and results indicators are all is needed.
(European Commission – DG REGIO EVALSED GUIDE)
EFFECTIVENESS: WHAT TO EVALUATE ? EUROPEAN COMMISSIONQuality: effectiveness is evaluated
by comparing results with quality standards.
Ability of a given action to produce a desired change: comparing what is observed after the action has taken place with what would have happened without the action. One needs data that allow recovery of the counterfactual situation.
OCSE - DAC
Effectiveness assesses whether the results outlined in the logframe are delivered and if they are likely to produce the expected objective. Evaluating effectiveness should include assessment of how women and men benefit from the results brought by the project
Evaluation Questions
• To what extent were the originally defined objectives of the development intervention realistic?
• To what extent have the (direct) objectives of the development intervention been achieved in accordance with the (adjusted, if applicable) target populations?
• What are the (concrete) contributions of interventions for achieving the objectives of the development intervention?
• What factors were crucial for the achievement or failure to achieve the project objectives so far (indication of strengths and weaknesses, e.g. the monitoring and evaluation system)?
• What is the quality of development-policy, technical planning and coordination ?
Information and Data on OUTCOME
Indicators
Quantitative
- Financial
- Physical
- Procedural
Qualitative
- Opinions on the level of achievement
- Perceptions on satisfaction
Methods
• Literature review
• Interview
• Dialogic interview
• Community interview
• Project visit
• Focus group
• Case study
• Survey
Different way for collecting information
How to analyse the information?
• Quality: parameters and standard
• Economic / Financial aspects: Cost and Revenues
• Satisfaction: expectations against the perception
• Opinions: majority/ minority
• Physical: achieved against targets (logframe )
• Time/ environmental protection: savings with regard to the situation before the intervention (business as usual)
• Counterfactual: Situation with/ without intervention
Objective or Objectives?- Normally a project has more than one objective even if not too many (no,more than 3 - OCSE DAC).- For example a new road construction can make the connection:
1. Safer
2. Cheaper
3. Faster
Multicriteria
Tool used to compare several interventions in relation to several criteria.
Multicriteria analysis is used also in the ex ante evaluation for comparing proposals. It can also be used in the ex post evaluation of an intervention, to compare the relative success of the different components of the intervention. Finally, it can be used to compare separate but similar interventions, for classification purposes.
Multicriteria analysis may involve weighting, reflecting the relative importance attributed to each of the criteria. It may result in the formulation of a single judgment or synthetic classification, or in different classifications reflecting the stakeholders' diverse points of view. In the latter case, it is called multicriteria-multijudge analysis.
(from EVALSED)
Process
Step 1 define criteria
Step 2 scoring or
ranking
Step 3 weighting
Step 4
aggregating
STEP 1: setting criterion
Criterion 1:Financial Performance
Criterion 2:Procedural
Criterion 3: Physical realisation
Project 1
Project 2
Project 3
Project 4
Project n…
STEP 2: Score or rank for judgment
- It is needed to find a way to appraise the project according to different aspects since we used different measuring units for different aspects of different process
- We can opt for:- A) Scoring: by assigning a numeric value to different “interval” of
performance . For example 3 for “above average” –1 for “on line with average” – 3 for “below average”
- B) Ranking: we simply order the different projects according to their performance from the first to the last
STEP 2: Scoring
Criterion 1: Criterion 2: Criterion 3:
Project 1 1 1 1
Project 2 0 2 1
Project 3 3 1 1
Project 4 3 3 0
Project n…
STEP 3: Establishment of weight
If some Criteria is more important than others it shall be given more importance. To do it we simple apply a multiplication factor > 1 (e.g. 1,5).
Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold (For example “Physical” if some project has 0 performance, it is excluded by the analysis).
STEP 3: Apply the weight
Criterion 1:( * 1,5)
Criterion 2: Criterion 3:
Project 1 1,5 1 1
Project 2 0 2 1
Project 3 4,5 1 1
Project 4 4,5 3 out
Project n…
STEP 4: Aggregate the scoreCriterion 1:( * 1,5)
Criterion 2: Criterion 3:
Total (with weight)
Total (without weight)
Project 1 1,5 1 1 3.5 3
Project 2 0 2 1 3 3
Project 3 4,5 1 1 6,5 5
Project 4 4,5 3 out out 6
Project n…
WORK OUT: SME INCUBATOR
Criterion 1:Economic (average increase of turnover)
Criterion 2:New Jobs
Criterion 3: Satisfaction for quality service
Total
Project 1
Project 2
Project 3
Project 4
WORK OUT DATAS
Project 1 2 3 4
Economic 80% 70% 30% 70%
Physical 50 10 30 5
Quality High Low High Medium
Low Medium High
Quality levels
Apply weight
A
Economic * 1,5
Physical Veto = N. of job < 10
B
Physical * 2
Quality Veto for “Low”
