Vermont Partnerships for Success

Year One Process Evaluation

Amy Livingston

Process evaluation focuses on assessing and documenting grantees’ :◦Partnerships and capacity building◦Assessment and planning◦Implementation of strategies (including

successes and challenges encountered, and program-level outcomes)

Process Evaluation Questions

Outcome evaluation focuses on:◦Assessing change over time in population-based outcome measures and targeted intervening variables

◦Comparing change in outcomes in grantee communities (both individually and collectively) to the rest of the state

◦Primary sources will be YRBS and YAS

Outcome Evaluation Questions

• Formative Evaluation• Provides template for implementation activity• Helps identify implementation problems/issues that

may be correctable• Documentation/replicability • Provides record of what was done

• Performance monitoring• Provides metrics for assessing progress in achieving

implementation objectives and goals• Interpretation of program outcomes• Helps explain why or why not outcomes were

achieved

Purposes of Process Evaluation

Most of our process data for PFS comes from data submitted quarterly to CGRS

Also site visits, networking calls, meetings with District Office staff

May do some qualitative evaluation (focused interviews and/or focus groups) to help assess changes in regional capacity

How are we doing so far?

All grantees have reported an overall increase in collaboration with community partners since the beginning of the grant

Collaboration and Community Partnerships

Period

10/13 thru 3/14 4/14 thru 9/14

Sector:

Number of Grantees with at

Least One Partner

Number of Grantees with At

Least One Partner

Collaborating at High or Very High

Level

Number of Grantees with at

Least One Partner

Number of Grantees with At

Least One Partner

Collaborating at High or Very High

LevelBusiness community 5 4 6 6Civic/volunteer group 3 3 5 2Clergy/religious organization 3 1 5 3Healthcare professionals 6 3 6 4Judiciary 2 1 3 0Law enforcement agency 6 5 6 5Media 6 4 6 3Parent 5 3 5 3School 6 5 5 4State, local and/or tribal governmental agency

6 4 6 5

Youth 6 3 6 3Youth serving organization 6 4 6 4Other individual or organization involved in preventing or reducing substance abuse 4 3 6 3

Colleges 5 5 5 4Young adults 5 2 5 2

Number of Grantees with Partners in Each Sector and with Partners Collaborating at a High Level

From the beginning of the grant through the report submitted in July 2014, TA received was assessed by intervention and by source (PC, other ADAP staff, etc.)

Starting with the report submitted in October, TA is now being assessed overall by topic area (SPF step) and source

Will be used by ADAP to monitor and improve quality of TA

TA Received

CGRS includes a measure of progress on each of the key activities from the work plan for each evidence-based intervention

Grantees are asked to rate progress on each key activity as not started, partially completed, in progress, or completed

In addition, there are other process measures in CGRS that collect both quantitative and qualitative information for each intervention

Progress on Implementation

Intervention:

Number of grantees

implementing

intervention

Started on at least half of the steps

Started on all of the

steps

Completed all of the

steps

Started but not able to complete 1

or more steps

Restrictions on alcohol outlet density 2 2 0 0 0

Restrictions on alcohol in public places 5 5 0 0 4

Enhancements to social host liability 1 1 0 0 0

Enhancements to open container regulations 1 1 0 0 0

Restrictions on advertising and promotion 1 1 0 0 0

Saturation/Party Patrols 6 5 4 0 3

Community mobilization 6 6 5 0 3

Media advocacy 6 6 4 0 3

e-CHECKUP for College 2 0 0 0 1

Alcohol Edu 1 1 0 0 0

Guiding Good Choices 1 1 1 0 0

Nurturing Parenting Program

1 1 1 1 0

Parenting Wisely 1 1 1 0 0

e-CHECKUP for the Community

1 1 0 0 1

Progress on Activity Implementation, by Intervention, April 2014 through September 2014

Type of policy education Number of grantees implementing

Number of policies developed

Number of policies approved

Enhancements to open container regulations

1 1 1

Enhancements to restrictions on advertising and promotion

1 0 0

Enhancements to social host liability

1 0 0

Restricting alcohol outlet density

2 1 1

Restrictions on alcohol in public places and/or at community events

5 0 0

Education on Policy Approaches

Successes:◦ Two successful town-level policy changes so far◦ Getting health language added to town or regional

plans◦ Systematic review and assessment of current

policies and ordinances◦ Development of summary document that can be

shared with towns throughout the region Challenges:

◦ Lack of stakeholder motivation◦ Lack of existing model policies◦ Takes time!

Education on Policy Approaches

All 6 grantees are working with law enforcement to enhance saturation and/or party patrols (work plans and reporting on these two interventions were combined in July)

Successes: Getting MOUs in place Establishing relationships with law enforcementChallenges: Coordination with different law enforcement agencies Low capacity to conduct additional patrols/utilize overtime

Enhanced Law Enforcement

Law enforcement activity Total number

Proactive patrols conducted 49

Minor in possession citations issued 69

Parties responded to by patrols 21

Type of Intervention

Number of Grantees Implementing

Total number of individuals reached

E-Checkup for College 2 2

Alcohol Edu 1 342

E-Checkup for the Community

2 74

Electronic Screening and Brief Intervention/Education

Successes: High completion rate of Alcohol Edu at JSC Decrease in alcohol policy violations at JSC; from 48 in fall of 2012

to 14 in fall of 2013Challenges: Getting colleges to commit to using e-Checkup

Type of program Number of grantees implementing

Total number of parents reached

Guiding Good Choices 1 46

Nurturing Parenting Program (Nurturing Fathers)

1 16

Parenting Wisely 1 11

Parenting Programs

Successes: Centralized coordination of parenting classes around the county

worked well Positive feedback from participantsChallenges: Getting parents to sign up and follow through with sessions

Activity Number of grantees

Relevant data

Retailer recognition for passing compliance checks

6 • A total of 123 businesses were recognized

Support activities for responsible beverage service training

6 • Grantees assisted with a total of 19 Merchant Education Seminars

Sticker Shock 5 • A total of 42 businesses participated

• A total of 63 youth volunteers participated

Other supporting activities to prevent underage drinking

A total of 3 new safe drop-off locations were established

Number of PFS regions using different types of outreach to educate the community about proper storage and safe disposal of unused prescription drugs.

Prescription Drug Prevention Activities

Type of outreach Number of grantees using

Brochures 4

Community events 6

Print media 4

Press release 3

PSA 2

Radio 3

Social media 6

Television 2

Website 5

Other (e.g. flyers, posters, newsletters, Rx bag inserts) 4

Successes:◦ Creative ways to get the word out e.g. through senior

centers, Meals on Wheels◦ Hungry Heart screenings and opiate summits have

provided good forum for communicating concrete things community members can do

◦ Great collaboration with law enforcement and health care providers on this issue

◦ Collaboration with health care providers and pharmacists has led to new understanding of the issue for all involved and allows for greater saturation of messages

Challenges:◦ Waiting for statewide materials◦ Engaging pharmacists, especially at chain pharmacies

Prescription Drug Prevention Activities

Successes:◦ Youth engagement efforts very strong in some

regions◦ Increased collaboration with schools◦ Community forums on opiate use and Hungry

Heart screenings provided great opportunities Challenges:

◦ Partners and collaborating agencies don’t always identify with county-wide approach

◦ How to engage with culturally diverse sub-groups

Community Mobilization

Grantees contacted the media a total of 185 times The total number of instances of earned media reported across all PFS

regions was 488 (*true number is probably closer to 76)

Number of instances of earned media by type

*In reviewing the description of types of media included in the “other” category, it was clear that some of the instances in this total would not be considered earned media but rather grantee-generated media such as ads, flyers, press releases, etc.

Media Advocacy

Type of media Number of instances

Letter to the editor 3

Newspaper story 43

Editorial/op-ed 6

TV story 16

Radio interview or story 6

Blog story/discussion 2

Other 402*

Number of times each type of message was addressed by earned media

*The description of other types of messages included educational opportunities to prevent substance abuse, and youth engagement (specifically an Above the Influence retreat)

^These items contribute to the ADAP media advocacy performance measure on the Dashboard

Media Advocacy contd.

Type of message Number of instances

Community efforts to reduce UAD 38

Enforcement efforts to reduce UAD 11

Community efforts to reduce Rx misuse^ 68

Community specific data or info on UAD 15

Community specific data or info on Rx misuse^ 29

General info or data related to UAD (not community specific) 21

General info or data related to Rx misuse (not community specific)^ 53

Other* 329

Successes:◦ Positive relationships developed with local media◦ Good coverage of Take Back Days and summits on

opiate use◦ Invitations to be interviewed for TV or papers◦ Invitations for regular media spot on TV or

newspaper column Challenges:

◦ Developing process for coordinated media outreach and tracking at regional level

◦ Ability/capacity to monitor local media

Media Advocacy

Next Steps… PIRE is preparing a year one evaluation brief

for ADAP which is primarily a summary of CGRS data collected so far.

We are also working with ADAP to develop performance measures related to the PFS, some of which may be displayed on the Dashboard

Modify CGRS when CLI goes live