frequently asked evaluation questions tobacco control evaluation center jeanette treiber, robin...
TRANSCRIPT
Frequently Asked Evaluation Questions
Tobacco Control Evaluation Center
Jeanette Treiber, Robin Kipke and Travis Satterlund
April 28, 2011
2
Where Do We Start?
Each evaluation activity is derived from the plan:
Objective plan type outcome/process data
How will data be used to move the objective forward?
Who should the data inform or convince?
Map out what pieces of information are needed, and how best to collect it (method, from whom, where, when)
3
What Is End-use Strategizing?
Identify your evaluation goals
Know the kind of information needed and specific purpose it should serve
Understand the best source for your data
Figure out when the data should be collected
Recognize what type of method is most useful
Decide an appropriate sample size
4
How Can We Use Our Data?
Your data can inform project staff by: Measuring how your message is being
perceived by various stakeholders (subgroups)
Indicating whether further community education efforts are needed
Assessing which policy options to pursue Identifying where your campaign might
encounter opposition Signaling that perhaps it is not yet the right
time to pursue policy
5
Are There Other Uses For Our Data?
Through a variety of reporting strategies, programs can use data to:
Indicate public/stakeholder readiness for a policy to decisionmakers
Generate “buzz” by sharing results in meetings, presentations, fact sheets, press releases
Build a groundswell of support and motivate more people to join the effort
Counteract critics
6
Should We Create New Instruments?
Adapt existing instruments (based on end use strategizing):
Not always necessary to reinvent the wheel
Adapt other data collection instruments for your project’s purposes
Start with high quality instrument Add questions you need data on Delete unnecessary questions Make sure it will collect what you need
7
Where Can We Find Instruments?
TCEC website has model instruments on limited number of objectives
Contact TCEC to access repository of instruments created/used by other projects
Ask other projects working on similar objectives to share their instruments
8
What Other Sites Have Instruments?
Download instruments from statewide sites
CA Health Interview Survey – http://www.chis.ucla.edu/questionnaires.html
CA Healthy Kids Survey – http://www.wested.org/cs/chks/view/chks_s/17?x-layout=surveys
CDC – http://cdc.gov/tobacco/data_statistics/surveys/
YTPS on STORE Website – http://www.tcsstore.org/stages/1_document/t_yps.pdf
9
How Can We Reach a Large Sample?
Utilize high traffic venues/locations Back-to-school packets Church service programs Grocery store flyers Classroom teachers Entertainment tickets (e.g. movie
theaters) Organizational newsletters Fairs and local events
10
What Are Other Distribution Avenues?
Online surveys (SurveyMonkey) Invite participation via email, social
networking media (Facebook, Twitter) Your organization’s website (and
those of coalition members, agencies that serve target population)
11
What Are the Pros of Online Surveys?
Cost effective Faster delivery Quick response time Better at addressing sensitive issues Ease of layout design and analysis
12
How Can We Increase Response Rates for Online Surveys?
Some tips for maximizing the response rates:
Request participation in advance
Include a cover letter
Allow sufficient time frame to complete the survey
Provide survey instructions
Design a clear survey
Send reminders to those who still have not completed the survey
Offer incentives for completion
13
Do We Need to Pilot Test the Instrument? Evaluate instrument’s effectiveness Get a sense of how long it takes to complete Create checklist of concerns to address with
the pre-test Do respondents/data collectors understand the
instrument? Is the wording clear to all segments of sample
population? Do any items produce irritation or embarrassment? Have important issues been overlooked?
14
What Should Our Key Informant Interview Protocol Look Like? Identify your goals beforehand
Know the kind of information needed and specific purpose it should serve
Create a structure to your interview
Probe, probe, probe (write probes into your interview protocol)
15
What Kind Of Outcome Data Do We Need For A MUH Objective?Depending on the objective:
Observed presence and placement of signage
Observation data of smoking and tobacco litter
Confirmed smoke-free language in leases
Tabulated count of signed leases with smoke-free language
16
How Can We Tap Into Volunteer Labor?
Develop relationships with local universities, high schools, programs or clubs
Recruit students looking for research experience to do data collection
Offer training, internships, course credit, other incentives
Entice coalition members to join in the activities
17
How Do We Make Use Of Our Volunteers?
To field test instruments/materials for appropriateness to data sources
To translate instruments/materials into other languages
To collect data To provide access to other distribution
points
18
How Do We Write Up the Results From Our Data Collection?
Summarize as you go Write “mini-analyses” See if the data can help you meet your
objective Incorporate your summaries and
analyses into your reports
19
How Can We Utilize TCEC’s Resources?
Online coalition satisfaction survey administration and analysis service
Additional model data collection instruments on a variety of topics will be developed in coming months
Model instruments translated into other languages (Spanish, Vietnamese, Hmong and Russian)
We’re working with other statewide projects to develop tools that will help you use and share your data more effectively for your policy work
20
When Should We Contact TCEC?
For individualized assistance with: evaluation planning development of data collection
instruments advice on analyzing & interpreting data guidelines for reporting & sharing info
21
Resources You Can Tap
TCEC archive of data collection instruments Evaluation associates at TCEC who can advise
you on developing instruments Recorded webinars on survey development,
sampling decisions, conducting POS Tips & Tools on creating interview questions,
sampling and end-use strategizing Electronic quarterly newsletter Your colleagues working on similar objectives