tti pec nairobi workshop - unpacking impact and influence
Post on 04-Dec-2014
81 Views
Preview:
DESCRIPTION
TRANSCRIPT
Unpacking Impact and Influence:Approaches to Monitoring & Evaluating
April 28, 2014
TTI PEC Africa workshop - Nairobi
Courtney Tolmie (Results for Development Institute)
A few notes to start
This session focuses on Organization M&E. That said, most of the lessons can be scaled down to the project level.
You will guide the session – and giving real time feedback:
2 | R4D.org
Way too fast. Please slow down.
Just right.
You can speed up.
How do we think about Impact and Influence?
1. What IMPACT does your organization seek to have?
2. What and who do you work to INFLUENCE as an organization?
3. What is the link between IMPACT and INFLUENCE in the work that you do?
3 | R4D.org
Monitoring and Evaluation – the link to Impact and Influence
In other words, a way of following whether you are doing what you planned to do and whether you are achieving the IMPACT you wanted to, including Policy INFLUENCE.
4 | R4D.org
Monitoring: a continuing function that aims primarily to provide the management and main stakeholders of an ongoing intervention with early indications of progress, or lack thereof, in the achievement of results.
Evaluation: systematic and objective assessment of an on-going or completed project, program, or policy, and its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact, and sustainability.
Definitions from the World Bank - http://siteresources.worldbank.org/INTBELARUS/Resources/M&E.pdf
What Makes an organization-level M&E System Effective?
Defining Impact and how we get there (TOC)
Defining how to measure impact & outputs
(indicators)
Carrying out M&E (Methods)
Using results to improve
work (Feedback
Loop)
5 | R4D.org
M&E Systems – Defining a Theory of Change
Defining Impact and how we get there (TOC)
Defining how to measure impact & outputs
(indicators)
Carrying out M&E (Methods)
Using results to improve
work (Feedback
Loop)
6 | R4D.org
Defining a Theory of Change Clarifying your
organization objectives and desired impacts
Mapping out what is required to achieve these impacts
Linking to the activities and actions your organization will take to achieve the impact
Also highlighting assumptions (including context issues)
Similar structures - logframes
M&E Systems – Defining Indicators
Defining Impact and how we get there (TOC)
Defining how to measure impact & outputs
(indicators)
Carrying out M&E (Methods)
Using results to improve
work (Feedback
Loop)
7 | R4D.org
Defining Indicators
Turning what you hope to achieve into what you can actually measure
Making indicators SMART
Value (and tradeoff) in collecting indicators on impact and outputs
M&E Systems – Designing Methods
Defining Impact and how we get there (TOC)
Defining how to measure impact & outputs
(indicators)
Carrying out M&E (Methods)
Using results to improve
work (Feedback
Loop)
8 | R4D.org
Defining and designing Methods
Identifying how to actually collect data on the indicators that you have defined
Timing – monitoring, evaluating or both
Things to consider – capacity, resources, what you are trying to achieve
M&E Systems – Ensuring Feedback Loops
Defining Impact and how we get there (TOC)
Defining how to measure impact & outputs
(indicators)
Carrying out M&E (Methods)
Using results to improve
work (Feedback
Loop)
9 | R4D.org
Ensuring Feedback Loops
The most forgotten step of M&E systems
Results for M&E can help improve programs and ultimately impact
Best to have formal structure to build results back into future work
Moving from culture of M&E to MEL
Mapping areas for development
Based on your existing M&E system, assess how you are doing in each element of effective M&E
Scale from “we don’t do this at all” to “we do this very well on all areas of work”
10 minutes to map your organization – then we will come back to review
10 | R4D.org
Defining Theories of Change
11 | R4D.org
Defining Theories of Change
Examples from other organizations Tips from the group about how to do it well Ideas for moving forward
12 | R4D.org
13 | R4D.org
Build evidence on better ways to
budget for higher education
Share recommendations with Minster of Education and
Finance
New budget passed with
recommendation
It would be easier if organization TOCs looked like this:
But this is not realistic
WHY NOT?
• TTs are trying to achieve bigger changes than just this• A single TT is going to likely need to take a lot of different
actions to get to that end impact• A single TT is not going to individually achieve this impact
14 | R4D.org
15
16 | R4D.org
Note about Theories of Change
A good TOC for influencing policy is incredibly complex.
Your organization may not try to affect all parts of the TOC – and thus may not choose to do M&E for all factors that go into policy influence.
But good to be aware.
Defining Theories of Change
Examples from other organizations Tips from the group about how to do it well Ideas for moving forward
17 | R4D.org
Defining Theories of Change – TOC Pitfalls to avoid from Stanford Social Science Review
1. Confusing accountability with hope2. Creating a mirror instead of a target3. Failing to take external context into account4. Not confirming the plausibility of your theory5. Creating a theory that is not measurable6. Assuming you have figured it all out
18 | R4D.orghttp://www.ssireview.org/blog/entry/six_theory_of_change_pitfalls_to_avoid
What lessons can you share from your organizations –
things that have worked and what has not worked?
Defining Indicators
19 | R4D.org
Defining Indicators
What types of indicators work well, especially for measuring policy influence and impact?
Tips from the group about how to do it well Ideas for moving forward
20 | R4D.org
Examples of good indicators – for the full organization While the focus of this program is on PEC, it is worth noting
that there are multiple components of a think tank’s performance worth monitoring:
21 | R4D.orgFrom Raymond Struyk’s forthcoming book – Improving Think Tank Management
Public Policy
Perspective
Funder Perspectiv
e
Internal Business
Perspective
Innovation & Learning Perspectiv
e
Financial Perspectiv
e
Examples of good indicators – for the full organization While the focus of this program is on PEC, it is worth noting
that there are multiple components of a think tank’s performance worth monitoring:
22 | R4D.orgFrom Raymond Struyk’s forthcoming book – Improving Think Tank Management
Public Policy
Perspective
Funder Perspectiv
e
Internal Business
Perspective
Innovation & Learning Perspectiv
e
Financial Perspectiv
e
Examples of good indicators for PEC impact and influence Some helpful indicators from the center for International Governance Innovations:
23 | R4D.orgFrom www.cigionline.org/blogs/tank-treads/communications-and-impact-metrics-think-tanks
Indicators of Exposure:• Media mentions• # and type of publications• Scholarly citations• Government citations• Think tank ratings
Indicators of Resources:• Quality, diversity, stability of
funding• Characteristics of researchers• Quality, # of networks,
partnershipsIndicators of Demand:• Events• Digital traffic and engagement• Access to officials• Publications sold, downloaded
Indicators of Policy Impact & Quality of Work:
• Policy recommendations considered or adopted
• Testimonials
• Quality of think tank work
Other indicators we have seen used
24 | R4D.org
If the objective is …Specific policy recommendation implemented
Change in the debate
… Possible indicators are:
Invitation to share findings with specific audiences (government, others)
Attendance of decision makers at dissemination events
(if available) Information on issue being debated in policy discussions
Request for research by policymakers on the new topics
Greater debate on topic in media (if available) Information on issue being
debated in policy discussions
Longer term – access to policymakers
Individual meetings with policymakers
A note about impact indicators …
In general, these are not going to be attributable.
Attribution is difficult Almost without exception, it takes a village.
This does not mean impact indicators are not helpful – especially if seeking to improve results.
25 | R4D.org
Defining Indicators
What types of indicators work well, especially for measuring policy influence and impact?
Tips from the group about how to do it well Ideas for moving forward
26 | R4D.org
Defining Indicators
What types of indicators work well, especially for measuring policy influence and impact?
Tips from the group about how to do it well Ideas for moving forward
27 | R4D.org
Designing methods for M&E
28 | R4D.org
Designing methods for M&E
What common options are others using? Tips from the group about how to do it well Ideas for moving forward
29 | R4D.org
Options for M&E approaches
30 | R4D.orgTable from ODI Background Note – A guide to monitoring and evaluating policy influence (Harry Jones)
Does not include common impact evaluation methods – RCTs, process tracing, etc.
Designing methods for M&E
What common options are others using? Tips from the group about how to do it well Ideas for moving forward
31 | R4D.org
Designing methods for M&E
What common options are others using? Tips from the group about how to do it well Ideas for moving forward
32 | R4D.org
Ensuring Feedback Loops
33 | R4D.org
Ensuring feedback loops
Do results feed back into the think tank automatically?
Tips from the group about how to do it well Ideas for moving forward
34 | R4D.org
Connecting M&E Results to Think Tank Changes
Two potential reasons to undertake M&E:
To achieve the second, M&E can’t stop at data analysis – it needs to feed back into the think tank.
35 | R4D.org
ACCOUNTABILITY to funders and others
BETTER RESULTS for think tank impact and influence
Questions to ask to design a strong Feedback Loop (Knowledge Management)
36 | R4D.org
The Question
WHO should hear the M&E results?
WHEN should they get the information?
The Answer
(1) Who is the best position to exploit the information – capacity and interest
(2) More generally, anyone who SHOULD know
(1) At a time when they can act – capacity and opportunity
(2) Generally more than once
WHAT should they receive?
(1) Format – useful but not necessarily fancy (2) Ideally comparisons to baseline data
In general, the Feedback Loop is more likely to be effective if designed early – proactive rather than reactive.
Built from Raymond Struyk’s forthcoming book – Improving Think Tank Management
Ensuring feedback loops
Do results feed back into the think tank automatically? Tips from the group about how to do it well Ideas for moving forward
37 | R4D.org
Ensuring feedback loops
Do results feed back into the think tank automatically? Tips from the group about how to do it well Ideas for moving forward
38 | R4D.org
A final word
39 | R4D.org
Conclusion
M&E is not easy – and requires capacity. This is especially true for organization-level M&E and for policy influence M&E. Match M&E plan to capacity
Moving from M&E to MEL (L = “learning”)
Linking the two objectives of M&E – accountability and better results Some methods and indicators can address both – where
possible, don’t double your work Example – how can you use the IDRC monitoring indicators for
PEC to help you achieve better results?
40 | R4D.org
Other Great Resources (in addition to those referenced on this powerpoint)
ODI work on RAPID
On Think Tanks page on Monitoring and Evaluating Influence
(http://onthinktanks.org/topic-pages/topic-page-monitoring-and- evaluating-influence/)
Research 2 Action PEC page
41 | R4D.org
top related