impact assessment training 10/11 th september 2013 oslo
DESCRIPTION
Impact Assessment Training 10/11 th September 2013 Oslo. Workshop Outcomes. By the end of these two days you will have : Developed a common understanding of what is meant by impact assessment; and how it differs from and complements the processes of monitoring and evaluation. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/1.jpg)
Impact Assessment Training10/11th September 2013
Oslo
![Page 2: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/2.jpg)
Workshop Outcomes
By the end of these two days you will have: • Developed a common understanding of what is meant by
impact assessment; and how it differs from and complements the processes of monitoring and evaluation.
• Identified key challenges that you face in conducting impact assessments
• Worked with pre-selected case studies to design an impact assessment process
• Considered how the results of impact assessments might be used in learning, reporting and being more accountable to both donors and stakeholders
• Identified ways in which you can take forward learning from this workshop.
![Page 3: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/3.jpg)
The vicious circle
![Page 4: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/4.jpg)
Impact
So what do we mean when we talk about Impact?
![Page 5: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/5.jpg)
Think about an event or a person who made a significant impact on your life (a birth or death of a family member or friend, a relationship, a marriage, an accident)
– What does impact mean in relation to this?
– What concepts or ideas does it include?
Impact
![Page 6: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/6.jpg)
How do you “assess” this impact?
• ... You try to understand the nature of the change that has taken place in you and to determine its significance in your life.
![Page 7: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/7.jpg)
Impact Assessment
“The systematic analysis of significant and/or lasting change – positive or negative, intended or not – in the lives of target groups, brought about by a given action or a series of actions”
![Page 8: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/8.jpg)
Its not that easy... consider Danny..
![Page 9: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/9.jpg)
Why assess impact?
• To understand the implications of our work• To become more accountable to those we work
with (stakeholders)• To support institutional learning and decision
making and improve future work• To contribute to policy development and effective
advocacy• To help demonstrate organisational performance
![Page 10: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/10.jpg)
Key areas of enquiry for impact
Five essential questions:• What has changed?• For whom? • How significant was it? • Will it last? • In what ways did we contribute to
these changes?
![Page 11: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/11.jpg)
The challenge for this workshop
To be able to develop Impact Assessment processes which are:
• Simple and user-friendly• Build on existing structures and systems• Are useful for accountability both
upwards and downwards• Are useful for your organisational
learning
![Page 12: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/12.jpg)
Session 2Relationship between M&E and
Impact Assessment
![Page 13: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/13.jpg)
M&E or Impact Assessment?• How is the assessment of impact different
from the processes of monitoring and evaluation?
• What to do?• When to do it?
![Page 14: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/14.jpg)
ExampleIn a project to build social housing for a local
community, for example: • Monitoring would relate to the purchase of
materials, and building the houses according to plans that have been drawn up.
• Evaluation would assess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective etc
![Page 15: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/15.jpg)
But...It’s possible to have well-built cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine “settlements”). Thousands of “successful projects and programmes” which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts. Impact Assessment addresses the “So what?” question: as a result of our efforts, what’s changed for whom; and how significant is this for them?
![Page 16: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/16.jpg)
Impact – so what?!Thousands of “successful” projects and programmes make no lasting difference to people’s lives.
Impact Assessment addresses the “So What?” Question
As a result of our efforts, what has changed for whom; and how significant is this for them?
![Page 17: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/17.jpg)
Differences in Brief
![Page 18: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/18.jpg)
Log Frame Approache.g. Skills training programme - a results chain...
Does it always work like this???
Inputs Activities Outputs Outcomes Impact
Funds, expertise
etc
Training
Improved skills
Improved livelihood
s
Less poverty
![Page 19: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/19.jpg)
Small groups’ taskSmall UK based NGO, working through partners
in Ethiopia in EducationGoal: All children have access to free basic
education1. What high level outcomes would ensure that
this goal is achieved?
![Page 20: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/20.jpg)
Small groups’ task• Select one outcome• Discuss what activities/outputs would lead
to the achievement of this outcome• Develop this into a mini project• Develop indicators at each level
• Use different colour post its!
![Page 21: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/21.jpg)
Small groups’ taskLevel Detail Indicators
Impact All children have access to free basic education
Outcome 1
Outputs•....•....•.....
![Page 22: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/22.jpg)
Analysing the resultsTo what extent are you able to assess the
key questions for impact assessment?– What has changed?– For whom?– How significant/lasting are these changes for
different target groups?– What -if anything - did our programme
(project) contribute?
![Page 23: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/23.jpg)
Measuring impact based on results chain
Tends to be limited to assessing what you expect/hope will change...
Good for demonstrating results to donors..Positive impact Negative
impact
Expected xxxx ?
Unexpected ? ?
![Page 24: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/24.jpg)
So...
• So how else to assess impact?• What are the challenges?• What different approaches could we use?
Answers this afternoon!
![Page 25: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/25.jpg)
Session 3Focus on change and how to
assess it
![Page 26: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/26.jpg)
Thinking about Change
• How does change happen?• How do/can development organisations
conceptualise this? • What is their realistic sphere of influence on
changes that do take place?
![Page 27: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/27.jpg)
![Page 28: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/28.jpg)
Change is..
Complex in that many different changes can take place simultaneously in people’s lives
• Continuous in that nothing in society or the environment is ever static
• Variable in pace, scale and/or over the course of any intervention
• Not necessarily lasting or sustainable.
![Page 29: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/29.jpg)
What do we need to know to assess impact?
•Who or what was involved in the change? (e.g. individual actors or state institutions)
•What strategies were used to bring about the change? (e.g. reform, mass mobilization)
•What were the contexts that affected how the change happened? (e.g. urbanization, power relationships)
•What was the process or pathway of change? (e.g. demonstration effects, cumulative progress)
•How were our efforts connected to this?
![Page 30: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/30.jpg)
Organisational response to this?
• Many organisations are developing Theories of Change to better support their understanding of how change happens and their role in the process
![Page 31: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/31.jpg)
Four Key ElementsUnderstand how change happens and your role in
this
Develop Organisation/Prog
ramme change pathway
Assess the impact of your
efforts
Critically reflect on your pathway
and how you thought change
happened
![Page 32: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/32.jpg)
How do they complement other planning and M&E processes?
• Strategic plans?• Log frames?• M&E systems?• Learning loops?
![Page 33: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/33.jpg)
3. ToC doesn’t replace results frameworks - they make them stronger
How change happens
Organisation or Programme
change pathway
Impact assessment
Reflection and adaptation of
ToC
ToC = critical analysis process
Logic Model = Accountability
Framework which is informed by ToC
![Page 34: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/34.jpg)
Summary of some of the differences
34
Logic Models Theories of Change
•Describe component parts of a projects • Make linear links between activities and results•Are used as a management tool
•Situate programme efforts in bigger picture of change •Surface and articulate assumptions•Explain causal pathways of ChangeAre used as a critical analysis process
![Page 35: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/35.jpg)
Examples of Impact Assessment Frameworks:
![Page 36: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/36.jpg)
Capacity Building to partners
Funds for partners
Funds for Capacity Building
Increased partner capacity
Activities proposed by partners
Direct
sphere
of influen
ce
Contributes to:
Long
Term
Impa
ct
Power holders at local and regional
level ensure that all women access rights,
opportunities and services
Marginalised women are
empowered and access rights,
opportunities and services
including:
• Equal access to property and
resources
• Reduction in harmful traditional
practises
• Increased representation in
local, regional and national leadership
positions
•Increased economic empowerment
• Reduction in violence and
exploitation against women?
MRDF
Contributes to:
Theory of Change – Women's Empowerment (MRDF as a stone making ripples in the water)
Contributes To change in:
Marginalised women are supported and
empowered by their families, society and
culture
Marginalised women are organised, active
and influential and play an active role in society
Marginalised women have the confidence,
knowledge and skills to access rights,
opportunities and servicesAccess to
support
Awareness of rights
Access to information
Access to resources
Access to credit
Access to training
![Page 37: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/37.jpg)
Session 4
![Page 38: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/38.jpg)
Approaches and strategies
• Three approaches:– Post programme: Testing logic of log frame ( impact “evaluation”)– Participatory ToC approach which is used to design monitor and assess
efforts– Research: looking back sometime later and assessing changes and their
relation to programme efforts
• Four strategies:– Build into existing M&E– Tracer and tracker studies– Ensure key moments of critical reflection– Commission a retrospective study
![Page 39: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/39.jpg)
Plans for tomorrow
Case Study Task
![Page 40: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/40.jpg)
Day 2
![Page 41: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/41.jpg)
Session 1Case Study Task 1
![Page 42: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/42.jpg)
Reflection on yesterday
![Page 43: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/43.jpg)
Main steps in designing an Impact Assessment
1. Define purpose, approach and scope of the assessment
2. Develop/confirm theory of change and/or dimensions of change
3. Develop a list of areas of enquiry which will help you to explore impact
4. Select tools and methods for gathering and analysing relevant information
5. Decide on reporting strategies6. Make concrete plans and timelines (who, when,
how....)
![Page 44: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/44.jpg)
The challengesTo develop a process which is:• Simple and user-friendly• Build on existing structures and systems• Are useful for accountability both upwards
and downwards• Are useful for your organisational learningAnd the challenges that we identified yesteday
![Page 45: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/45.jpg)
ACCOUNTABILITY TO THE POOR
![Page 46: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/46.jpg)
Case Study Task 1 In small groups:• Share the details of the programme you are
working on (you need to be clear about context, overall goal and top level outcomes)
![Page 47: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/47.jpg)
Case Study Task 1• What will be the main purpose of doing this
impact assessment? Organisational learning? To meet donor demands? Accountability to stakeholders? For advocacy
• Which approach (or combination of approaches?) to Impact Assessment would be most appropriate in this Why?
• What will be the scope and scale of this assessment?
![Page 48: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/48.jpg)
Session 2Case Study Task 2
![Page 49: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/49.jpg)
Case Study Task 2:• What is your realistic “scope of influence”? Which what
areas of impact will you realistically be able to “assess”, and which areas of change will you be able to “illustrate contributions to change”?
• Based on this, what “Dimensions of Change” will you be looking to assess?
• Develop a menu of areas of enquiry which will enable you to set baselines and track progress in relation to impact
•
![Page 50: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/50.jpg)
Being clear about your scope of influence
Sphere of indirect influence (there are other factors/actors
which influence changes you want to see)
Assessment process lessrigorous –
illustrations of impact
Sphere of direct influence
(you are working directly with target groups on
specific changes) Need to assess impact
rigorously
![Page 51: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/51.jpg)
DimensionTask 1
Areas of EnquiryTask 2
Tools/ Sources of Information
Task 3
Direct scope of influence
e.g. Capacity of partners
•Ability to plan and deliver•Levels of technical capacity•Approaches to networking with others• Level of sustainability•Shifts in confidence
Indirect scope of influence
![Page 52: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/52.jpg)
Areas of Enquiry and indicatorsAreas of enquiry indicators
Written neutrally to encourage expected/ unexpected
Designed to test logic of project and expected changes
Can track levels of xx, shifts in xxx, trends, perceptions, Requires open-ended questions
![Page 53: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/53.jpg)
Session 3Case Study Task 3
![Page 54: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/54.jpg)
Case Study Task 3• Propose a range of appropriate methods that
you could use to gather relevant data for both monitoring and assessing impact (including building on or adapting existing tools and mechanisms).
![Page 55: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/55.jpg)
Making the process manageable and robust
• Baselines • Sample size• Triangulation• Selecting the right tools and ways
collecting useful information• Ensuring that you have asked the right
questions in the right way
![Page 56: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/56.jpg)
Baselines• The problems?
• The solutions?– Dedicated desk research and ask others– Plan from the start and use rolling baselines– Reconstructing them
![Page 57: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/57.jpg)
Sample sizeA good sample is one that is sufficiently large
and is unbiased.How to select?
– Mathematical?– Random?– Pragmatic?
![Page 58: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/58.jpg)
Triangulation• You need three perspectives to ensure validity
of information gathered. Could include:– Three ways of checking same info– Asking the same question of three target groups
![Page 59: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/59.jpg)
A word about tools and collecting information
Less is more.. • Be aware of time resources and capacity• Stick to the “old faithfuls” as the basis• Mix of methods is good• Be led by areas of enquiry
• And perhaps the hardest thing of all.. Try to enable people to tell the real story...
![Page 60: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/60.jpg)
Checklist – use for areas of enquiry and selection of tools
In order to assess impact effectively, your areas of enquiry combined with data gathering methods needs to capture this information
1. Have there been any changes?2. How many people were affected (which target groups)?3. How were they affected (and were they affected differently)?4. Were these changes intended?5. How do they compare to baselines (have you got evidence)?6. What can be attributed to your organisational efforts?7. How confident are you in reporting these findings?
![Page 61: Impact Assessment Training 10/11 th September 2013 Oslo](https://reader031.vdocument.in/reader031/viewer/2022012922/56816506550346895dd7763d/html5/thumbnails/61.jpg)
Session 4Wrap up and the way forward