monitoring and evaluation of influence enrique mendizabal june 2011
TRANSCRIPT
Monitoring and Evaluation of
Influence
Click icon to add partner logo Click icon to add partner logo Click icon to add partner logo Click icon to add partner logo
Enrique Mendizabal
June 2011
Outline
• The skeptical view
• Why monitor and evaluate?
• What can we monitor and evaluate?
• Types of influencing
• Methods and tools for each
• A suggestion
• Work on your own
(on think tanks) some experts say:
From Braml (2004)
• Kent Weaver: it is hard to determine policy influence of one think tank in relation to others because they are all different, work on different aspects of a policy, influence in different ways and there are many other players involved.
• Nelson Poslky: there is no sense in looking for direct
influence of think tanks’ activities because one can only ask these questions when one ignores the complexity of political processes. Some causal relationships may only be found in a few cases, but systematic explanations of this sort remain an illusion.
5
more
Weidenbaum (2009)
• Andrew Rich: dollar for dollar, think tanks attract much more attention than any other organisation.
• John Hamre, of the Center for Strategic and International Studies:
evaluating the influence of think tanks on a particular policy would be like determining who is the father of Anna Nicole Smith’s baby. As soon as the change happens every think tank who had something to say about it is quick to claim it for itself.
• David Frum, ofthe American Enterprise Institute: since one cannot
measure influence, [no method] would do it.
6
even more
Abelson (2006)
• Abelson: by focusing on the influence of think tanks on policy change one would overlook a great deal of other types of influence that is the source of think tanks’ value.
• Leslie Gelb, form the New York Times: influence is highly
episodic, arbitrary and therefore difficult to predict. • Abelson and Ricci: as the policy community becomes more
open and complex it also becomes more difficult to say anything meaningful about the contribution of think tanks to it.
7
and more
Puryear (1994) • The most interesting and valuable contributions of
think tanks [in Chile] may not be intellectual but psychological: the hundreds of events and seminars that Chilean think tanks organised throughout the 1980s helped to restore the mutual trust and understanding missing from Chilean politics and that had led to the rupture of democratic order.
8
And:
Hoppe (2010)
The search for ‘impact’ is driven by three assumptions:• Unidirectional transfer of advice;• Policymakers and experts performing different
and exclusive tasks;• All possible impact is desirable per se
9
In practice, however:• Robust research into uptake and impact will be so
costly and time consuming, that nobody would be able to afford it.
• Lessons drawn will be marginal and irrelevant for other situations.
• Indicators focusing on that can be measured would only lead to conformance and perverse conduct.
Why should you do M&E?
• To learn• To manage better• To get more funds• To keep funders/clients happy
• (“what is the guarantee that doing this actually helps us?”)
Researchers vs research
Do we want:
1) Our research/IE to be more influential?
or
2) Policy to be based on evidence?
Visibility or substance?Visibility Substance
Short term ‘relevant’ research Long term research
Focus on solutions for ‘agreed’ problems
Engage with the definition of the problem
Media exposure Lobby, network, horse trading
Briefing papers, Opinion pieces Estimates, costed proposals, policy options
Website, Blogs, Facebook, etc. Academic publications, long reports
Online communities with millions of hits
Communities with the right people
Delegations at high level global conferences
Private meetings at Party conferences and private meetings while planning for the high level conferences
Event focused influence Problem focused influence
Global Go-To-Survey Prospect magazine Think Tank of the year
1. Strategy and
M&E & Learning
1. Strategy and direction –are you doing the right things?
2. Management –are you doing what you planned to do?
3. Outputs – are outputs up to scratch?
4. Uptake – are people aware of your work?
5. Outcomes and impacts –are you having any impact?
16
How?
Hovland (2009):
1. Strategy and direction: Logframes; Social Network Analysis; Impact Pathways; Modular Matrices
2. Management: ‘Fit for Purpose’ Reviews; ‘Lighter Touch’ Quality Audits; Horizontal Evaluation; Appreciative Inquiry
3. Outputs: Evaluating academic articles and research reports; Evaluating policy and briefing papers; Evaluating websites; Evaluating networks; After Action Reviews
4. Uptake: Impact Logs; New Areas for Citation Analysis; User Surveys
5. Outcomes and impacts: Outcome Mapping; RAPID Outcome Assessment; Most Significant Change; Innovation Histories; Episode Studies
M&E of policy influence – what’s the problem?
POLICY
Influencing
Activities
And who does the influencing?
• The researchers• Campaigners/third parties?• Policymakers themselves?
Losing control
Inputs Activities Outputs
Other Actors
Project Team
Outcomes
ImpactOutcomes ImpactOutcomes Impact
Policy changes
24
So we need to look for proxys
“Government runs in part, on the basis of memos. If a SD or DD official, or an analyst at the CIEA or the NSC, has your study in front of him and open at the time he is writing his own memo to the secretary or the director or perhaps the president himself –if, in short, he is using your ideas and analysis at the time he writes his own memo- then you have influence. If your study is not open in front of him, or worse, you do not even know who the responsible official is, you do not have influence. It is as simple as that.”
Howard Wiarda quoted by Abelson D. E., 2006
What to measure and how
3 types of influencing approaches (there are more):
• Evidence and advice• Public debate/education and
advocacy• Lobbying approaches
Evidence and advice
How? What to measure Tools
Consultancy
Research and analysis, ‘good
practice’
Evidence-based argument
Providing advisory support
Developing and piloting new policy
approaches
OutputsEvaluating research reports, policy briefs
and websites
Uptake and use
Logs;
new areas for citation analysis;
user surveys
Influence
RAPID outcome assessment;
Episode studies;
Most Significant Change
Public debate/education and advocacy
How? What to measure Tools
Public communications and
campaigns
University teaching
‘Public education’
Debate
Messaging
Advocacy
Target audience attitudes, behaviour,
etc
Surveys,
focus groups,
direct responses, career paths
Media attentionMedia tracking logs,
media assessment
Media framing and influence
Framing analysis; coverage
Lobbying approaches
How? What to measure Tools
Face-to-face meetings and discussions
Relationships and trust
Direct incentives and diplomacy
Actors;
relationships;
policy process and institutions
Recording meetings;
tracking people;
interviewing key informants;
probing influence
Recommendations
• Develop a Theory of Change based on sound theory
• Integrate PME for policy influence
• Use a mix of tools to gather data
• Ask: What does it mean?
Exercise
• Describe the change you would like to see • Describe the role you will play in it -your contribution
to this change • How will you know that you have fulfilled this role to
the best of your abilities?• How will you know whether you are doing the right
thing? • How will you incorporate lessons (that you and others
have learned) into your planning and implementation?