program evaluation needs assessment survey of local coalitions
TRANSCRIPT
Program EvaluationNeeds Assessment Survey of Local
Coalitions
How many evaluators does it take to change a light bulb? One to do a needs assessment One to do a feasibility study One to do a qualitative study to find out what bulb to change One to empower the bulb to change One to tender a contract for further study One to write performance indicators for success One to do a cost benefit analysis of the best bulb to buy One to do a meta-evaluation showing that all previous
studies have left everyone in the dark
SO.......how many evaluators does it take?
None, actually. Evaluators don't change bulbs, that's an implementation problem!
-Program Evaluation gets a lot of flack!
But it can save your assets!
Objectives
Understand the purpose of program evaluation
Review local coalition survey results
Most commonly asked PE question
How do I know I’m making a difference? Answer : Institute best practices… With fidelity, innovation, and critical
interpretation. Unfortunately, outcome evaluation is
expensive and rarely feasible with community intervention.
Most commonly asked PE question
How do I know I’m making a difference? Answer : Institute best practices… With fidelity, innovation, and critical
interpretation. Unfortunately, outcome evaluation is
expensive and rarely feasible with community intervention.
Process evaluation helps implement best practices with fidelity. Is my programming “evidence-based”? Nicotine patches are great. Stick one over each eye and
you can't find your cigarettes.
Is the “best practice” appropriate for my population?
What does the tobacco science say about adapting best practices to my population?
Am I appropriately following the science?
Process vs. Outcome Evaluation Process evaluation: Determines whether best-practices
were adopted properly.
Outcome evaluation: Determines whether program was successful.
Process Evaluation Assesses the delivery or implementation of a program. What happened and why? How is this different from
what was planned? To be evaluated:
• Activities• Materials• Delivery
• Numbers of staff • Audience• Logistics
Process Evaluation TATU Example: How many youth participated in TATU
trainings? What were the youths’ perceptions of
each training? What were the implementation problems?
Outcome Evaluation Measures what progress the program has
made towards its goals.
To be evaluated:
• Behaviors• Attitudes
• Knowledge• Skills
Outcome Evaluation Example goals to evaluate for TATU: Short term – youth who participate in TATU
gain outreach skills. Intermediate – TATU youth reach fellow
students. Long term – Youth who TATU participants
reach actually use skills to abstain when tobacco is available.
UW Tobacco Surveillance & Evaluation Program
“It is now proved beyond doubt that smoking is one of the leading causes of
statistics.”
http://www.medsch.wisc.edu/mep/
UW Tobacco Surveillance & Evaluation Program Indirectly provides outcome evaluation of
Wisconsin’s coordinated tobacco control activities.
Fields surveys. Analyzes data to monitor state trends. Identifies progress and emerging challenges. Future – more geographically specific
information; better disparities data.http://www.medsch.wisc.edu/mep/
Conclusion : Uses of program evaluation Integral part of every successful organization. Clarifies and re-focuses program goals and
strategies. Verifies that current strategies are still
appropriate. Refreshes staff expertise in best practices and
evidence-based programming. Documentation for supporters, funders, and
stakeholders.
Survey Results
Results reported here are for fully-funded coalitions ($20,000 or more).
95% of fully-funded coalitions responded (40 of 42 coalitions that were contacted).
76% of coalitions responded in total (55 of 72). 94% of coalitions funded at $10,000 or more
responded (44 of 47).
Survey Results: Is P.E. built into your regular program functioning?
85%
13%
0%
20%
40%
60%
80%
100%
Some currentevaluation
No currentevaluation
% o
f C
oa
litio
ns
Survey Results : The majority of coalitions (85%) engaged in some program evaluation.
85%
13%
0%
20%
40%
60%
80%
100%
Some currentevaluation
No currentevaluation
% o
f C
oa
litio
ns
Population
9%14%
19%22%
37%
0%
10%
20%
30%
40%
N W S NE SE
% o
f W
I po
pu
lati
on
Regional Distribution of Coalitions
Distribution of fully-funded coalitions by region
17%21% 21% 19%
24%
0%
10%
20%
30%
40%
N W S NE SE
% o
f C
oal
itio
ns
Population
9%14%
19%22%
37%
0%
10%
20%
30%
40%
N W S NE SE
% o
f W
I po
pu
lati
on
Coalitions distributed geographically, rather than proportionate with population.
Distribution of fully-funded coalitions by region
17%21% 21% 19%
24%
0%
10%
20%
30%
40%
N W S NE SE
% o
f C
oal
itio
ns
Number of full-time equivalent employees (FTEs) in Wisconsin:
33 FTEs statewide 0.82 per coalition
(33 hours per week) About 40¼ total
staff.
All fully-funded coalitions
33
0.820
5
10
15
20
25
30
35
FTEs in Wisconsin Average FTEs percoalition
Total number of FTEs
5.1 4.7
8.5
6.9 7.5
0
2
4
6
8
10
N W S NE SE
FTEs by Region:
Number of residents per FTE
97,678
162,405
125,844
176,440
274,948
0
50,000
100,000
150,000
200,000
250,000
300,000
N W S NE SE
Indication of urban-efficiency gains. Building partnerships is vital.
Total number of FTEs
5.1 4.7
8.5
6.9 7.5
0
2
4
6
8
10
N W S NE SE
Number of residents per FTE
97,678
162,405
125,844
176,440
274,948
0
50,000
100,000
150,000
200,000
250,000
300,000
N W S NE SE
Most coalitions are funded entirely through a TPCP contract. This raises the potential for instability given a potential change
in state government. What steps are coalitions taking to diversify?
Grants Received by Funded Coalitions(Select all that apply)
87%71%
17% 10% 7% 2% 7%
0%
20%
40%
60%
80%
100%
Only TPCPand TTAfunding
Only TPCPcontractfunding
TTA CommunityFoundation
Federal AmericanLegacy
Othergrants
% o
f C
oal
itio
ns
Priority Populations (Choose three)
Targeting youth has likely paid off.
After rising through 1999, youth cigarette use now drops every year.3%
0%
0%
7%
8%
10%
12%
24%
28%
40%
46%
53%
90%
0% 20% 40% 60% 80% 100%
Other PopulationsElderly
Vets18-24
AODA/mental hlthEmployers
Ethnic/racial minorityPregnant Women
low SESPolicymakers
AdultsGeneral Public
Youth
% of Coalitions
Priority Populations (Choose three)
More resources needed for 18-24 year olds.
3%
0%
0%
7%
8%
10%
12%
24%
28%
40%
46%
53%
90%
0% 20% 40% 60% 80% 100%
Other PopulationsElderly
Vets18-24
AODA/mental hlthEmployers
Ethnic/racial minorityPregnant Women
low SESPolicymakers
AdultsGeneral Public
Youth
% of Coalitions
Priority Populations (Choose three)
Less than 50% target policymakers.
“Other” populations reported include health care providers.
3%
0%
0%
7%
8%
10%
12%
24%
28%
40%
46%
53%
90%
0% 20% 40% 60% 80% 100%
Other PopulationsElderly
Vets18-24
AODA/mental hlthEmployers
Ethnic/racial minorityPregnant Women
low SESPolicymakers
AdultsGeneral Public
Youth
% of Coalitions
Coalition Objectives(Check all that apply)
“Other” reported goals:
Reducing tobacco use among pregnant women
Increasing the tobacco tax.
20%
33%
38%
58%
68%
73%
88%
90%
0% 20% 40% 60% 80% 100%
Other Goals
Smoke-free campuses
Smoke-free homes
Disparities
Treatment
Youth advocacy
Smoke-free worksites
Youth prevention
% of Coalitions
Major Coalition Activities (Check all that
apply)
Coalitions selected an average of 10 activities.
All selected more than one activity.8%
23%
40%
43%
43%
45%
45%
60%
63%
78%
78%
80%
80%
83%
83%
85%
93%
0% 20% 40% 60% 80% 100%
Other activities
Pursue funding
First Breath
Train health professionals
TATU
Increase community members' skills
Employers, treatment
FACT
Employers, smoke-free
Community partnerships
Recruit supporters
Educate members on advocacy
Educational materials
Media
Youth activities
Wisconsin Wins
Educate policymakers
% of Coalitions
Major Coalition Activities (Check all that
apply)
Less than 100% recruit supporters and seek community partnerships.
Less than ¼ pursue alternative funds.
8%
23%
40%
43%
43%
45%
45%
60%
63%
78%
78%
80%
80%
83%
83%
85%
93%
0% 20% 40% 60% 80% 100%
Other activities
Pursue funding
First Breath
Train health professionals
TATU
Increase community members' skills
Employers, treatment
FACT
Employers, smoke-free
Community partnerships
Recruit supporters
Educate members on advocacy
Educational materials
Media
Youth activities
Wisconsin Wins
Educate policymakers
% of Coalitions
Major Coalition Activities (Check all that
apply)
“Other” activities: working with state
partners. helping worksites
create individual policies.
asthma workgroups. the N-O-T program. Oral health
screenings in disparate populations.
8%
23%
40%
43%
43%
45%
45%
60%
63%
78%
78%
80%
80%
83%
83%
85%
93%
0% 20% 40% 60% 80% 100%
Other activities
Pursue funding
First Breath
Train health professionals
TATU
Increase community members' skills
Employers, treatment
FACT
Employers, smoke-free
Community partnerships
Recruit supporters
Educate members on advocacy
Educational materials
Media
Youth activities
Wisconsin Wins
Educate policymakers
% of Coalitions
Back to Program Evaluation…
Current Program
Evaluation Activities(Check all that
apply)
More than 50% document their basic program activities.
Herein lies a wealth of experience to share.
15%
15%
15%
18%
20%
30%
30%
30%
30%
38%
38%
40%
58%
58%
60%
65%
68%
0% 20% 40% 60% 80%
Other evaluation
Worksite survey
Survey of health professionals
Mail survey
Phone banking results
Observation data
Youth survey
Survey of general public
Track results of paid media
Solicit event feedback
Member survey
Pre and post tests
Quitline Stats
Track # meeting attendees
Track earned media
Lists of supporters
Track # event attendees
% of Coalitions
15%
15%
15%
18%
20%
30%
30%
30%
30%
38%
38%
40%
58%
58%
60%
65%
68%
0% 20% 40% 60% 80%
Other evaluation
Worksite survey
Survey of health professionals
Mail survey
Phone banking results
Observation data
Youth survey
Survey of general public
Track results of paid media
Solicit event feedback
Member survey
Pre and post tests
Quitline Stats
Track # meeting attendees
Track earned media
Lists of supporters
Track # event attendees
% of Coalitions
Current Program
Evaluation Activities(Check all that
apply)“Other”: focus groups police reports CDC’s SPF
indicators WI Wins data First Breath
statistics parent surveys oral health
surveys data from local
agencies
In your own words… Coalitions’ answers to open-ended
survey questions:
1. What questions would you want a program evaluation to answer?
2. Please describe any sources of information/data that might be available for use in program evaluation.
1. What questions would you want a program evaluation to answer? Open-ended responses were grouped
into:
process questions outcome questions technical needs
Process Questions: Is the program operating as intended?
Are training materials and meeting times best? Is the program structure optimal?
Are we reaching the intended population? Are our activities “best practice”?
Are we using best practices for helping pregnant women quit? Generally, what is the ranking of most- to least-
valuable practices? What is the need/demand for our program? How important are non-cigarette tobacco products?
Process Questions: Is the program operating as intended?
Are training materials and meeting times best? Is the program structure optimal?
Are we reaching the intended population? Are our activities “best practice”?
Are we using best practices for helping pregnant women quit? Generally, what is the ranking of most- to least-
valuable practices? What is the need/demand for our program? How important are non-cigarette tobacco products?
Outcome Questions: What difference has the program made?
Are we changing attitudes about second hand smoke?
Have our education materials impacted attitudes about smoke-free workplaces?
What behavior changes resulted? Have pledge cards increased smoke-free homes?
How many/who benefits? Progress on goals of multi-year action plan? Do results justify the costs?
Technical needs: Where can we find detailed local statistics?
Where to find numbers of youth smokers, by school and by grade?
How can City of Milwaukee data be separated from Milwaukee county data?
How do we write survey questions that identify attitude and behavior changes? Where can we find template surveys that can be changed to
fit the needs of our community? How can we use program evaluation data to promote tobacco
control? What are indicators of whether progress is being made?
2. Please describe any sources of information/data that might be available for use in program evaluation. Open-ended responses were grouped
into:
Qualitative Quantitative
Qualitative Testimony from youth, school leaders, adults who work
with youth, partners, law enforcement. Documentation of earned media. Minutes of Board of Health meetings. Supporter data base. Mid-year and year-end reports on consolidated contract
tobacco objectives. Community needs assessment (every 5 years). Community Health Assessment data.
Quantitative Statewide surveys/data: YRBS, BRFS, UW Burden of Tobacco, UW survey of smoke-free municipal
and government-owned buildings, Worksite survey, WIC data on tobacco use prevalence
Local surveys: Dane County Youth Assessment, PRIDE survey in county schools, South
West Youth Survey (SWYS), evaluation of local smoke-free ordinance, county smoke free air/tobacco use survey (400 residents surveyed), community health assessment data, survey data collected by community health improvement coalitions, oral health screening results
Program surveys/data: Friends Helping Friends (youth peer group) survey data, WI Wins
compliance checks data, First Breath data, Quit-line and Fax to Quit statistics, pre- and post-test data for participants, member survey
Next Steps…
Potentially, there may be regional trainings in program evaluation methods in each of the DHFS service areas.
Show of hands: No PE training or experience Some PE training Some PE experience Some PE success
Thanks! Local coalitions team:
Sue Marten, Chair Dianna Forrester Kathy Cahill Themis Flores Ramos Connie Olson Heidi Foster Deb Bruning Paula Silha Deb Gatzke Karen Hagemann Jody Moesch-Ebeling
T&TA Lynn Habrik Kathy Cahill Deb Bruning Mary Hilliker Sue Marten
Randy Glysch Cindy Musial Sue Marten
