Why evaluate your advocacy? And how?

Identifying Indicators, Implementing Measures, Assessing Impact

Why evaluate your advocacy—from the start

• Assess implementation of your strategies• Are you doing what you set out to do?

• Test your theory of change• Does changing X really change Y, like you thought it would?

• If not, what might be other routes to your objectives?

• Know when you’re on the right track• So that you can adjust—in real time—if not

• Communicate with funders• And those you wish would invest

PHASES OF ADVOCACY EVALUATION

1• FRAME the evaluation

• Why are you evaluating?

2• ENGAGE important partners

• Who else needs to think alongside you?

3• DEFINE what you are evaluating

• What is the intervention? How is it understood to work? How do you define ‘success’?

4• DESCRIBE what happened

• Activities, outputs, outcomes, context

5• Articulate CAUSATION for the observed outcomes, including accounting for

unknowns in the causal chain

6• ASSESS the overall success

• What lessons do you want to carry forward?

AUDIENCES

OU

TCO

MES AC

TIO

NW

ILL

AW

AR

ENES

S Voter Outreach

Public EducationPolicymaker Education

Influencer Education

Political Will Campaigns

Litigation

Media Advocacy

Regulatory Feedback

Public Forums

Champion Development

Model Legislation

Policy Analysis/Research

Demonstration Programs

PUBLIC INFLUENCERS

Using an Advocacy Framework to Guide your Evaluation

Community Mobilization

Coalition BuildingCommunity Organizing

Public Will CampaignsCommunications and Messaging

Leadership DevelopmentAdvocacy Capacity Building

Public Awareness Campaigns

Public Polling

Lobbying

DECISION MAKERS

Modified from Coffman & Beer, 2015: http://www.evaluationinnovation.org/sites/default/files/Adocacy%20Strategy%20Framework.pdf

AUDIENCES

DECISION MAKERS

OU

TCO

MES

AC

TIO

NW

ILL

AW

AR

ENES

S Voter Outreach

Public Education Policymaker EducationInfluencer Education

Political Will CampaignsMedia Advocacy

Regulatory Feedback

Public Forums

Champion Development

Model Legislation

Policy Analysis/Research

Demonstration Programs

PUBLIC INFLUENCERS

Community Mobilization

Coalition Building

Community Organizing

Public Will Campaigns

Communications and Messaging

Leadership Development

Advocacy Capacity Building

Public Awareness Campaigns

Public Polling

Awareness

SalienceIssue Reframing

Public Will Political WillMedia Coverage

Attitudes & Beliefs

New Advocates New Champions

Policy Change

New Donors

Collaboration & Alignment

More or Diversified Funding

Constituency Growth

Organizational Visibility and RecognitionAdvocacy Capacity

What outcomes could tell whether you’re hitting the mark?

LitigationLobbying

AUDIENCES

DECISION MAKERS

OU

TCO

MES

AC

TIO

NW

ILL

AW

AR

ENES

SRegulatory Feedback

Public polling

Policymaker ratings

PUBLIC INFLUENCERS

Event observation

Intense period debriefsPower mapping

Public pollingFocus groups, surveys

Polling, surveys, focus groups

Bellwether interviewsMedia tracking

Constituent action tracking

Policy tracking

Partnership Analysis

Network mapping

Donor tracking

Blog analysis, social media traffic

What tools measure these outcomes?

Champion tracking

DECISIONMAKERS

Framing penetration analysis

The Best Evaluation is the One you Use• Mobilization, Constituency

Growth, Public Will

• Political Will, Champion Development

• Attitudes, Beliefs, Awareness, Salience, Alignment

• Advocacy Capacity

• Policy Change

• Salience, Visibility, Media Coverage

• Media Coverage, Framing

• Organization Records (attendance, action response)

• Policymaker Ratings

• Surveys (public, partners)

• Validated assessments + observation/reflection on practices

• Intense Period Debriefs and Bellwether Interviews

• Tracking/Content analysis

• Media analysis

IF it really measures what you want to know

• If you want to assess implementation, measure inputs and outputs.

• If you want people to do something different, measure action.

• If you think that doing differently will require changes in thinking, measure awareness. Just don’t expect strategies aimed at awareness to automatically induce action.

Measure effects on the targets toward which you actually aimed; otherwise, you may undercount your impact, because you’re just

not looking at the right people.

The U of Good Advocacy Evaluation

• While we tend to spend the most time discussing instruments and data collection, the best advocacy evaluation is U-shaped, investing the most energy in the first and final phases:

1. Considering your questions and how to approach answering them

2. Making sense of your findings and figuring out how to incorporate what you learn into your advocacy, moving forward

THOUGHT EXERCISE

What questions might you develop to evaluate advocacy within these theories of change? What indicators could you develop?

Where could you get the data needed to answer those questions?

• How much positive coverage?

• How much public support? Among whom?

• How much political will? For what? Among whom?

• Do grantees and allies have shared definition?

• Have you increased organizational capacity?

• Is a diverse base represented?

• How many, who, and toward what end, have you mobilized a constituency?

• Have necessary volunteers and staff been recruited?

• What relationships have been built with targeted journalists?

• How aware are bicycle riders of helmet safety?

• What is rate of traumatic brain injury from bicycle riding (baseline and at post-intervention points)?

AND WHAT WOULD YOU DO WITH WHAT

YOU LEARNED?

A Learning Culture for Advocacy Evaluation

• What practices do you embrace to check progress toward your goals?

• What do you ask—and seek to answer—for your own purposes, without inducement from external stakeholders?

• Who is involved in evaluation for learning? Role models among senior leaders? Substantive engagement by frontline staff?

• Are you prepared to grapple with ‘failure’? Do people feel safe learning ‘out loud’?

Learning Organizations…

• Invest resources in evaluation capacity (human and technological; internal and external)

• Embed learning into organizational routines (e.g. staff meetings)

• Signal priority for learning AND real-time response

• Model transparency

• Interrogate and display data through multiple lenses, platforms, and formats

Derrick-Mills, T., Winkler, M.K., Healy, O.& Greenberg. E.(2015). A Resource Guide for Head Start Programs: Moving beyond a Culture of Compliance to a Culture of Continuous Improvement. Washington, DC: US Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation. http://www.acf.hhs.gov/opre/resource/a-resource-guide-for-head-start-programs-moving-beyond-a-culture-of-compliance-to-a-culture-of-continuous-improvement.

Shifting to Learning Culture

Winkler & Fyffe (2016). Strategies for Cultivating an Organizational Learning Culture. Urban Institute. Available at: https://www.urban.org/sites/default/files/publication/86191/strategies_for_cultivating_an_organizational_learning_culture_3.pdf.

EXAMPLES OF ADVOCACY

EVALUATION TOOLS

There is no ‘perfect’ tool—you want a good fit with what you’re actually trying to measure, alignment with your other operations, and unobtrusive enough that you’ll really use it

Policymaker Ratings

Uses of Policymaker Ratings

• Identifying champions• Who should be with you? How can they show it?

• Isolating effective neutralization and persuasion approaches• What is moving people?

• Testing sincerity and transparency of partner relationships• Who’s sharing what they know?

• Identifying issues ready to ‘tip’• Where are the stars aligning?

Intense Period Debrief

• Often, outside evaluators engage advocates in evaluative inquiry shortly after a policy window, special event, or other intense period of action occurs.

• With training and practice, advocates can facilitate intense period debriefs for themselves, although it can be difficult to get the ‘bandwidth’ needed until longer after the period has passed.

Framing an Intense Period Debrief

• What events triggered this intense period?

• How was the organization’s response determined? How was that decision communicated?

• Which elements of the response worked well? Which could have been improved?

• What was the outcome of the intense period? – Was the immediate aftermath positive or negative?

– What are the anticipated longer-term effects?

• What challenges did we face? Which were unanticipated? What do we wish we had known?

• What insights will we take away for next time?

Uses of Intense Period Debriefs• Reevaluating historic tactics/events

• Avoiding ‘autopilot’, born of scarcity of time

• Capturing reflections during legislative cycles or other predictably unmanageable epochs

• Systematizing informal reflections and making scheduled meetings more substantive• Increasing the likelihood that you learn what you should

and use what you learn

• Reducing the personalization of shared reflection; emphasis is on the intensity of the period, not any individual’s actions/inactions

Debrief of RDP Lobby Day

Bellwether Interviews

• Evaluators or other outsiders conduct structured interviews with influential people whose positions require that they track a broad range of policy issues.

• Ideally, the same is diverse, representing both the public and private sectors. At least part of the bellwether should lack direct connection to or expertise in the policy issue of interest.

• While formal evaluation experience is not necessarily imperative, bellwether interviews cannot be conducted effectively by an organization itself. Further, bellwethers’ consent to be interviewed should not highlight the policy issue of interest.

Questions for a Bellwether Interview1. What three issues do you think are at the top of the policy

agenda?

2. Considering the state's current educational, social, and political context, do you think the state should adopt [the policy] now or in the near future?

3. Looking ahead, how likely do you think it is that [the policy] will be established in the next 5 years?

4. Currently, what individuals, constituencies, or groups do you see as the main advocates for [the policy]? Who do you see as the main opponents?

5. If [the policy] is established, what issues do you think the state needs to be most concerned about related to its implementation?

Use of Bellwether Interviews• Testing new markets/geographies/issues

• What needs to change, for you to move into this space?• Assessing messages

• What works? How much have your message efforts ‘broken through’?

• Gauging relative salience among slate of issues• How much are we ‘on the radar’? How is the issue positioned on

the policy agenda?

• Assessing political will• What do people say about your issue when you’re not the one

asking?

• Forecasting the likelihood of future policy – What do people in the business of knowing think is coming next?

MBP Bellwether Interviews

Framework for Media Analysis

1. Is this earned, social, or opinion media? (Which do you value more?)

2. To what extent is our framing used in this coverage? (can rate 1-5)

3. How are our opponents’ arguments framed? Who represents them, and how well have we prepared for that opposition?

4. Whose voices are included? Whose perspectives are ignored?

5. To what extent are ‘unlikely’ stakeholders included?

6. Whose images are included? What does the unspoken coverage ‘say’?

7. Who initiated the coverage? (What did we have to do to get it?)

8. Has this coverage been ‘echoed’ by other outlets?

Tools for Social Media Analysis

• Hootsuite

• TweetReach

• Buzzsumo

• Twazzup

• Boardreader

• Mention

• Followerwonk

Uses of Media Analysis

• Assessing framing

• How much are your messages permeating?

• Identifying promising outlets/conduits

• Who is carrying your messages?

• Gauging prominence/salience

• How much are you on the agenda?

• Identifying allies and opponents

• Who is with you? And against you?

Tactical Dashboards• Align definitions of success across the organization

• Encourage dialogue about progress toward goals

• Facilitate timely identification of successes and challenges

• Ground decisions in concrete data, presented in user-friendly visual format

• Illuminate relationships between different activities

• Create a snapshot of current status as well as trends over time

• Highlight out-of-the-ordinary results

• Not just cool pictures, the indicators you display should:– Represent business model drivers

– Reflect progress toward intended outcomes

– Guide priorities and decisions (“what gets measured gets done”)

– Be limited to a number that can realistically be monitored and periodically reassessed

Use of Tactical Dashboards

• Engaging Board members

• “This is what you should pay attention to!”

• Facilitating supervision

• “This is what I’m looking for.”

• Focusing attention on priority areas

• “We are focused on moving this.”

• Cultivate learning orientation within organizational operations

Other Tools and How You Can Use Them

• Focus groups with advocates/donors

• What do they care about? Why do they support you? What roles do they want to play?

• Advocacy capacity assessments with coalitions, partners, members

• Where are you strong? Where do you need to grow?

• A/B testing of messages

• Which one moves people more? Where are different constituencies differently motivated?

• Social network analysis

• Who is connected, and to whom, and how?

• Cost-effectiveness

• How much did you spend to get outcome X? How do you assess whether it was ‘worth it’?

Advocacy Capacity Assessments

Free Advocacy Capacity Assessment from Bolder Advocacy

http://bolderadvocacy.org/tools-for-effective-advocacy/advocacy-capacity-tool

Getting Started

• Let your evaluation be guided by your questions: what do you need to understand in order to make the change you want to see?

• Figure out what you most want to evaluate; trying to do everything at once will overwhelm you.

• Budget for evaluation, but don’t assume that you need to hire an external evaluator.

• Reach out for help—utilize developed instruments, find thought partners, build on others’ learning.

Resources• Alliance for Justice Advocacy Capacity Tool (http://bolderadvocacy.org/tools-for-effective-

advocacy/advocacy-capacity-tool)

• Center for Evaluation Innovation (http://www.evaluationinnovation.org/)

• Aspen Institute Advocacy Progress Planner (http://planning.continuousprogress.org/)

• Innovation Network Learning Center (http://www.innonet.org/resources/)

• Stanford Social Innovation Review (“Elusive Craft of Evaluating Advocacy” http://www.ssireview.org/articles/entry/the_elusive_craft_of_evaluating_advocacy/)

• Continuous Progress: Better Advocacy Through Evaluation (http://dp.continuousprogress.org/)

• Users’ Guide to Advocacy Evaluation (http://www.hfrp.org/evaluation/publications-resources/a-user-s-guide-to-advocacy-evaluation-planning)

• Monitoring and Evaluation of Policy Influence and Advocacy (https://www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8928.pdf)

• Readiness for Organizational Learning and Evaluation self-assessment (https://www.fsg.org/tools-and-resources/readiness-organizational-learning-and-evaluation-instrument-role#download-area)

• Organizational Learning Self-Assessment (https://theonn.ca/wp-content/uploads/2018/05/Self-Assessment-Tool-DRAFT.pdf)

• Unique Methods in Advocacy Evaluation (http://www.pointk.org/resources/files/Unique_Methods_Brief.pdf)

CENTER FOR COMMUNITY ENGAGEMENT AND

COLLABORATION

Melinda Lewis, LMSW, Associate Director, CCEC

University of Kansas School of Social Welfare

mlewis@ku.edu

(785) 864-1047