David Geddes, Ph.D.Senior Vice President & PartnerFleishman-Hillard
Institute for Public Relations Measurement Commission
21 tips for conducting 21 tips for conducting evaluation research evaluation research
on a boot laceon a boot lace
2
Today’s objectives
Advice on doing it yourself
Advice on being a good client
3
Where to get help
University departments
Social work, Sociology, Psychology, Political Science, Business, and others
Contacts via board of directors
American Marketing Association
Market Research Association
Market research firms
LinkedIn (www.linkedin.com)
National Opinion Research Center (University of Chicago) http://www.whatisasurvey.info/
4
http://www.italladdsup.gov/community_partners/dc_teachmehow_02.asp
Introduction to evaluation research
Process evaluation examples
Telephone survey with annotations
Intercept and self-administered survey
5
Build a community
Vision
Lutheran Social Services Research Community
Share best practices
Methods
Facebook or other collaboration tool
6
Set your objectives
Program objectives
Specific information needs
1
7
Do your background research2
8
Think about different types of evaluation and measurement
1. Formative evaluation
Helps planning
2. Process evaluation
How well is the program operating?
3. Impact evaluation
Building awareness and understanding?
4. Outcome evaluation
Concrete program or “business” metrics
3
9
Don’t assume you know audience attitudes and opinions
You need formative research
4
10
Set up your process metrics5
11
Measure impacts
Awareness
Knowledge
Attitudes and perceptions
Intended actions
6
12
Identify survey targets
Known respondents
Population =Sample frame Members Donors Clients Employees Civic organization
leaders
Defined population
Sample frame derived from population General public Congregation members Potential donors Potential adoptive
parents
7
13
Pick the right survey method to reach targets
Telephone
Online
Mall or store intercept
8
14
Telephone surveys
Advantages
Speed, in some cases
Control
Probe and clarify
Cost if you do it yourselfDisadvantages
Cost if you go outside
Hard to reach some socio-economic groups
15
Telephone surveys
Case: General public, 10 minute survey of 500 adults in Saint Louis metro area. $8,000 = $16 cost per interview Review cost parameters
Case: Survey of 200 adult children or guardians of seniors who receive services from LSS in St. Louis metro area. LSS provides names and phone list. $4,000= $20 cost per interview Review cost parameters
16
Telephone surveys
Case: Survey of 200 married adults in the St. Louis metro area aged 18 to 45 who are at least somewhat open to the idea of adoption. 10 minute survey. 30% incidence based on other surveys.
$7,000 = $35 cost per interview
17
Doing your own telephone surveys
Sample list
Staff
Interns, volunteers, students, etc.
Call management system
Data recording system
Discipline
Persistence
18
Going outside for telephone surveys
Smaller local firms
Be flexible on timing
19
Mail surveys
Advantages Lower cost Easy if you have a mailing list Easier to do yourself
Disadvantages Slow Lower control Lower response rates Need a mailing list
20
Doing your own mail surveys
Clear questionnaire layout
Sample list
Staff
Interns, volunteers, students, etc.
Lots of manual paper handling
Discipline
Patience
21
Online surveys
Advantages
Speed, in some cases
Control
Cost if you do it yourself (SurveyMonkey)Disadvantages
Purchase a sample list
Unrepresentative of some socioeconomic groups
22
Online surveys
Case: General public, 10 minute survey of 500 adults in Saint Louis metro area using an online survey tool. $3,300 = $6.60 cost per interview Review cost parameters
23
Doing your own Online surveys
SurveyMonkey programmer
Sample list
Staff
Interns, volunteers, students, etc.
24
Mall intercept surveys
Advantages Control Probe and clarify Cost if you do it yourself
Disadvantages Slow Need mall or store cooperation Unrepresentative of some socioeconomic
groups
25
Doing your own mall intercept surveys
Cooperation of mall or Wal-Mart
Staff
Interns, volunteers, students, etc.
Discipline
Patience .. Figure on one interview/hour
26
Select an appropriate sample size
What are possible sources of error?
How far off could we be?
What sample size do we need?
8a
27
Give advance notice
If you know the target respondents
Newsletter
Postcard
Benefit: improves response rate
9
28
Remember you are asking for their time
This applies to…
Pre-survey letters or cards
Introduction to the survey
And the questions themselves
Don’t ask meaningless questions
10
29
Write a good introduction to the questionnaire
Introduce yourself
“Research” not “survey”
Survey topic: social services
Not a sales call … not a fundraising call
You want their opinion / input
11
30
Move from general to specific12
31
Lead with an interesting question
Engage the respondent
Ask interesting questions and people will talk
13
32
Make judicious use of open-ended questions
Example:
“Based on what you have read or heard, what are your personal perceptions of Lutheran Social Services? [PROBE] Anything else?”
2 to 3 per questionnaire maximum
Finish with a “final thoughts” question
14
33
Use good response scales
Building a scale
Clearly distinguish good and bad
1 Agree
2 Disagree
15
34
Use good response scales
Building a scale
Add a polite positive and negative
1 Strongly Agree
2 Agree
3 Disagree
4 Strongly disagree
35
Use good response scales
Building a scale
Add a top positive response
1 Very strongly agree
2 Strongly Agree
3 Agree
4 Disagree
5 Strongly disagree
36
Use good response scales
Building a scale Add a don’t know/no opinion
1 Very strongly agree
2 Strongly Agree
3 Agree
4 Disagree
5 Strongly disagree
6 Don’t know / no opinion
37
Use good response scales
Building a scale Add a mid-point if you really need it ... and
balance the scale 1 Very strongly agree 2 Strongly Agree 3 Agree 4 Neither agree nor disagree 5 Disagree 6 Strongly disagree 7 Very strongly disagree 8 Don’t know / no opinion
38
Make sure you cover all possible responses
Multiple choice questions
16
39
“No opinion” does not equal “Don’t know”17
40
Always include a “refused” option18
41
Be careful with numeric responses
“About how much do you spend on gas each month” $__________
Which of the following best describes the amount you spend on gas each month: Under $25
$26 to $50
$51 to $100
Over $100
19
42
Be careful about “do you floss your teeth” items
Socially desirable responses
Responses that may imply guilt
20
43
Test your questionnaire21
44
Wrap-up
• Begin with your objectives and information needs
• Define the people you want to survey• Select an appropriate survey method• Write good questions• Test• Launch the survey• Analyze the results
45
Exercise