COMMUNITY HEALTH AND WELLNESS CONFERENCEAPRIL , 2013

Documenting and Communicating Progress on

Policy Change

Agenda

How does policy change progress, so I know what to look for?

What are some process and outcome measures that are appropriate for policy change?

What impact measures are appropriate for policy change?

What should I report and when?

Use data at every stage of policy work

Chuck: “Some date we use to get started, some data we use to know if we are on the right track, and some data lets us know how we did.”

Report out

Theory of change

Progress: process

Progress: outcome

Impact

Policy change theories

This document identifies six theories of policy change. Each is summarized and a theory of change is provided for each theory with strategies and outcomes.

Stachowiak, S. (2009). Pathways for change; Six theories about hoe policy change happens. Seattle, WA: Organizational Research Services. Funding from The California Endowment.

Global theory: Large Leaps Significant changes in policy and institutions can occur when the

right conditions are in place. Conditions are: issue is defined differently or new dimensions of the issue get attention; new actors get involved; the issue becomes more salient and receives heightened attention. Promising strategies include issue framing, mobilizing supporters, and media advocacy. Focus on changing policy, not administrative or rule changes. Issue definition and agenda setting are key to mobilizing new people.

  Global theory: Coalition

Policy changes happens through coordinated activity among a range of individuals with the same core policy beliefs. Coalitions are held together by agreement over core beliefs about policies; core beliefs are unlikely to change. Strategies include influencing decision-makers, changing incumbents, affecting public opinion, changing perceptions about policies.

  Global theory: Policy Windows

Policy can be changed during a window of opportunity when advocates successfully connect two or more components of the policy process: the way a problem is defined, the policy solution to the problem, or the political climate surrounding the issue. Three streams in the policy process are problems (including definition and salience), policies, and politics. Two streams must converge to create a policy window. Strategies include impacting problem definition, developing policy options, and influencing the political climate (through coalition building, demonstrations, or media).

 

Based on work with tobacco and underage drinking prevention coalitions

Consistent with research (including that done by Organizational Research Services, Michael Quinn Patton, etc.).

Strategy theory: Messaging and Frameworks Individuals’ policy preferences or willingness to

accept them will vary depending on how options are framed or presented. Strategies include issue framing and message development, typically embedded in a broader campaign.

  Strategy theory: Power Politics

Policy change is made by working directly with those with power to make decisions or influence decision making; power is concentrated in the hands of a few. Strategies include relationship building, focusing on decision-makers, and establishing credibility.

  Strategy theory: Grassroots

Policy change is made through collective action by members of the community who work on changing problems affecting their lives. Power is changeable and dynamic and not held by a few elites. Strategies including training, capacity building, community mobilizing, etc.

 

Coalition strategies were consistent with CDC

Coalitions utilized strategies consistent with Centers for Disease Control and Prevention guidance (2009). The most common strategies were: educating community and decision makers through

presentations and one-on-one meetings; followed by monitoring, assessment, or local

surveillance to collect data about the issue; and then advocating directly to decision-makers

through presentations or one-on-one meetings.

Comparing prevalence of strategies: successful and unsuccessful efforts

SWAT involvement

Partner involvement

Education/presentations

Surveillance

Presented to decision-maker

Personal relationships

Unsuccessful policy efforts Not yet Successful policy efforts

<---less prevalent ----- more prevalent---->

Source: Coalition interviews

The prevalence of actions taken by Coalition members to move a policy forward.

Share contacts

Distribute information

Attend advocacy meetings

Participate in events

Make presentations

Complete tasks

Conduct local research

Promote the partnership

<---less prevalent -------------------more prevalent ---->

Source: Coalition interviews

Start with a theory of change

What steps do you expect to see? What strategies are you using to move your

policy work forward?

Documenting progress

There are many types of evaluation, but thinking about evaluation in terms of process, outcome, and impact categories can be helpful.

Process evaluation is “a study of what goes on while a program (or policy) is in progress. Process evaluation relates to the phase of the program studied – in this case, program implementation “(Weiss, 1998, p 335).

Outcome evaluation is “a study of whether or not the program produced the intended program effects. Outcome evaluation relates to the phase o f the program studies - in this case, the end result of the program” (Weiss, 1998. p. 335).

Impact is “the net effects of a program (i.e., the gain in outcomes for program participants minus the gain for an equivalent group of nonparticipants). Impact may also refer to program effects for the larger community” (Weiss, 1998, p. 331).

Weiss, C. (1998) Evaluation: methods for studying programs and policies (2nd ed.). New Jersey: Prentice Hall.

Exercise: process measures for policy change

• Specific: the objective must communicate exactly what is expected, who’s involved, and where is it going to happen.

• Measurable: the objective should include concrete criteria or data for measuring progress toward the attainment of the goal.

• Attainable: the goals are neither out of reach nor below standard performance as these may be considered meaningless.

• Relevant: the objective should be worthwhile, happening at the right time, and match other efforts and needs.

• Timely: ground objectives within a time frame, giving them a target date.

How did Coalitions move policy forward?

Interim steps are an indicator of future success: 35 percent of the coalitions that achieved a policy reported at least one interim step; in contrast, when not successful, coalitions did not achieve an interim step.

What types of interim steps occurred?

A champion stepped forward. Sometimes a school board member, county commissioner, city council person, or administrator became the champion for the policy and offered to guide it to adoption.  Policy was developed. Prior to adoption, representatives from the coalition and the adopting entity often prepared drafts or worked on the actual policy components.  Supports were developed. The coalition may have assisted with implementation supports prior to the actual policy being adopted / going into effect. This included helping to obtain signs, coordinating meetings between cessation service providers and the entity, providing other resources, or working on ancillary policies such as citations.

Interim steps (continued)

Policy was placed on the policy agenda. This occurred when the issue or policy is publicly placed on the policy agenda of your target. This could be when the school board or county commission allows a presentation or when a landlord schedules a presentation at a tenant meeting.  Verbal commitment from decision-maker was received. Prior to the actual adoption of a policy, the coalition may receive a verbal commitment to adopt a policy. For example, a landlord may state he/she wishes to move forward and then schedules the next meeting to review draft policies with representatives from the coalition.  A non-coalition task force or committee was formed to adopt/implement policy. Often a task force or committee (particularly in schools) will be created to work on the policy and guide its adoption and implementation. Other examples would be a legislative committee working on a proclamation, resolution, or ordinance.

Other interim steps from the literature

Policy development, policy proposal, demonstration of support, adoption, funding, and implementation

Shift in social norms: changes in awareness, attitudes, values, or behaviors

Increased alliances: increased number of partners, increased strength of collaborations

Strengthened base of support: increased public involvement, increased advocacy activities, # of partners, increased media coverage, increased awareness among groups, increased visibility of issue

Exercise: Short and intermediate outcome measures

• Direction of change: identify if the area of change is going to be maintained, increase, decrease, improve, rise, or lower.

• Area of change: identify what will be measured.

• Target population: clarify the target population that will experience the change.

• Degree of change: identify the percentage or actual number.

• Time frame: provide the deadline by which the change will occur.

• As measured by: include the tool or source of data.

• Baseline: document the baseline.

Strategy: social host ordinance As noted, there may be additional interim and long-term outcomes associated with this strategy,

such as ordinance adoption, reduction in citations, reduction in social access, and reduction in consumption.

Outcome Measure, instrument, or question

Data collection methodology Analysis

Process: Youth to present information on the need for policy change and its positive impact to 5 municipalities between 7/1/2012 and 6/30/2013.

Agendas Source: MunicipalityHow: Collect or download agendaWhen: At time of presentationWho: Coalition staff

Compare the number of presentations planned to the number of presentations completed.

Outcome: By 6/30/2013, at least two interim steps toward a social host ordinance will have been achieved.

Policy planner checklist (see Resources section).

Source: Coalition How: Document the date that each interim step is achieved When: QuarterlyWho: Coalition staff

Compare the number of steps planned to the number of steps completed.

17

Long-term impact

Number of policies obtained

Number of residents impacted

Behaviors of interest

Others from prior studies

Students report social

access to alcohol

Policy progress: progress measures

Policy progress:

interim steps

Policy achieved: XX,XXX

students

Students report less

social access to alcohol

Reporting plan content

Who is your target audience (e.g. funder, partner, policymaker)?

When do they need information (e.g. when planning an evaluation, when conducting an evaluation, or when you have results)?

What message do you need to communicate (e.g. progress, results, impact)?

What format will you use to communicate your message (e.g. in person, formal report, email)?

How often will you communicate with this target audience (e.g. monthly, quarterly, annually)?

Policy and advocacy resources

California Endowment. (n.d). Advocacy progress planner [computer software]. Available at www.planning.continuousprogress.org

  Sponsored by the California endowment, the component definitions

provided as part of the planner are based on two reports prepared by Guthrie, K., Louie, J., David, T., and Foster, C. of Blueprint Research & Design, Inc. in 2005 and 2006 although this document is not attributed to any author.

  Interim steps towards policy include policy development, placement

on the policy agenda, policy adoption, policy implementation, policy monitoring and evaluation, policy maintenance, and policy blocking. Document lists audiences, capacity building and preparation/planning inputs, contextual factors, policy and communication activities, and capacity and policy interim outcomes and benchmarks. Users of the system create their own models from these choices.

Policy and advocacy resources

Reisman, J., Gienap, A., and Stachowiak, S. (2007). A guide to measuring advocacy and policy. Seattle, WA: Organizational Research Services. Funded by the Annie E. Casey Foundation.

Reisman, J., Gienap, A., and Stachowiak, S. (2004). A practical guide to documenting influence and leverage in Making Connections communities. Seattle, WA: Organizational Research Services. Funded by the Annie E. Casey Foundation.

Wrap up

How does policy change progress, so I know what to look for? Theory of change

What are some process and outcome measures that are appropriate for policy change? Process measures Outcome measure

What impact measures are appropriate for policy change? Impact measure

What should I report and when? Outline of a reporting plan