1 survey research nancy whelchel assistant director for survey research university planning and...
TRANSCRIPT
![Page 1: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/1.jpg)
2
Overview
We’ll cover What is a survey? Sampling Questionnaire development and design Analysis, interpretation, and reporting Using the results
![Page 2: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/2.jpg)
3
What is a Survey?
A scientific social research method that involves: Selecting a random sample of people to answer some
questions Designing & administering a standardized questionnaire
to get information about the research questions Analyzing the results to provide descriptions about the
people in the sample and find relationships between different responses
Generalizing the results to the population from which the sample was drawn
![Page 3: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/3.jpg)
4
Basic survey formats
Telephone interview Face-to-face interview Computer assisted interview Self-administered “paper & pencil”
survey (scannable or not) Self-administered Web survey Self-administered Email survey
![Page 4: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/4.jpg)
5
Why do a survey?
Really bad answer: Why not? Anyone can do a survey.
Slightly less bad answer:My boss told me to.
Better answer:I want to systematically evaluate the extent to which my program is meeting its goals and objectives in order to improve our programs through pro-active, well-informed decision making.*
*Adapted from CUPR FAQ “What is Assessment?” (http://www.ncsu.edu/provost/academic_programs/ FAQ/UAPRFAQwhatisassment.htm)
![Page 5: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/5.jpg)
6
But why do a survey?
To collect information not available from other sources
To have comparable information To be able to generalize results to the
population in which you’re interested
![Page 6: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/6.jpg)
7
Before you plan to do a your own survey…
Think about alternative sources
Information from non-survey sources Information from existing survey data
![Page 7: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/7.jpg)
8
Still think you need to do a survey?
Who is going to Identify the goals and objectives Develop the research design Design the questionnaire Identify the population and select the sample Pre-test the survey Administer the survey Analyze and interpret the results Write up the results Present the findings
![Page 8: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/8.jpg)
9
Still think you need to do a survey?
How much TIME do you have ?
Who is going to PAY for it all ?
Who is going to USE the results ?
![Page 9: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/9.jpg)
10
So you still want to do a survey …
Helpful hints! Familiarize yourself with survey research ‘best
practices’ and code of ethics (see www.aapor.org)
(Get approval from your Institutional Review Board)
Form an advisory group w/ relevant constituents
Organize a survey team
![Page 10: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/10.jpg)
11
So you still want to do a survey …
More Helpful Hints!
Develop a detailed overall research design
What? Why? When? How? Who?
![Page 11: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/11.jpg)
12
What is the purpose of the survey?
Explicitly state your objectives
Example:
Enhance employee well-being
![Page 12: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/12.jpg)
13
What is the purpose of the survey?
Identify factors related to objectives
Examples of factors related to “well-being”: Pay and benefits Physical work environment Work relationships Etc.
![Page 13: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/13.jpg)
14
What is the purpose of the survey?
Make an exhaustive list of everything you want to be able to measure related to your objective.
Examples of dimensions of “work relationships”: Communication of necessary information Feedback on performance Opportunities to provide input in decision-making Etc.
![Page 14: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/14.jpg)
15
Who do you need to survey?
Identify your population(s)
Population = The theoretically specified aggregation of all units about which information is collected.
Examples for “work relationship” populations?
Supervisors Employees Unit members Other?
![Page 15: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/15.jpg)
16
Who do you need to survey?
Survey population can influence Data collection methods you use Questions you ask Analyses you do Information you gather Assumptions you can make Recommendations you can make Action you can take Etc.
![Page 16: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/16.jpg)
17
Who do you need to survey?
All possible populations?(which technically would be a “census”)
If youMust keep everyone happyHave unlimited timeHave unlimited resources
![Page 17: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/17.jpg)
18
Who do you need to survey?
Everyone in a specific population?(which would still be a census)
If youHave a relatively small populationHave reason to expect a relatively low response
rateWant to have enough respondents to do sub-
group analyses for small groups
![Page 18: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/18.jpg)
19
Who do you need to survey?
A sample of the population?
If youUnderstand sampling proceduresCan convince others that it’s okay to sampleWant/need to limit time in the fieldWant to enhance ability for quality control over
entire processWant to keep costs down
![Page 19: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/19.jpg)
20
Select the sample
Create a “sampling frame” A list of elements from which the sample will
be selected (i.e. members of the population)
Examples?Company personnel data fileUnit directoryEtc.
![Page 20: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/20.jpg)
21
Select the sample
Probability sampling methods Simple random sample
List elements in the sampling frame and use random number chart or computer program to select elements
Systematic sampling List elements in the sampling frame and take every Nth
element, based on how many needed in sample Warning: Be attentive to any inherent patterns in how the
sampling frame is ordered. Stratified sampling
Draw appropriate proportion of elements from homogenous groups interested in
![Page 21: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/21.jpg)
22
Select the sample
Sample size? Depends on: Cost Anticipated response rate Number of responses needed per
subgroup interested in
Level of precision want (“margin of error”)
See Appendix A for more information on sampling.
![Page 22: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/22.jpg)
23
Select the sample
Sample Warnings!
BAD SAMPLE = BAD DATA
Survey results are only representative of the sampling frame from which the sample was selected.
Sample survey results are generalizable only if respondents are randomly selected.
![Page 23: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/23.jpg)
24
Select the sample
A Warning About Sampling Error!
It’s NOT the ONLY source of survey error!!!
![Page 24: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/24.jpg)
25
Select the sample
A Warning About Response Rates!
A high response rate does NOT mean good data
if those who respondedare ‘biased’ in some way
![Page 25: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/25.jpg)
26
Select the sample
A Warning about Web Surveys!
If you can not control who
accesses and responds to a web survey
the results are NOT
generalizable to the population.
![Page 26: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/26.jpg)
27
Design the questionnaire
REMEMBER YOUR OBJECTIVES!
What do you want to learn? How will you use the information?
![Page 27: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/27.jpg)
28
Design the questionnaire
Don’t reinvent the wheel!!
Develop (or borrow*) questions/items to measure your specific outcomes
*In general it is perfectly acceptable to borrow questions from other surveys, but always get permission and/or check for copyright restrictions. See Appendix B for examples of places to find questions.
![Page 28: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/28.jpg)
29
Design the questionnaire
To have confidence in your survey results your questions must be
ReliableQuestions consistently convey the same meaning
to all people in the population being surveyed.
ValidQuestions measure what they are intended to
measure
![Page 29: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/29.jpg)
30
Design the questionnaire
Warning!Bad Question = Bad Data
The way a question is wordedand the response options offered
determine the nature of the data received.
![Page 30: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/30.jpg)
31
Design the questionnaire
Open-end questions Pros
Stimulate free thought Solicit suggestions Clarify positions Richer information
Cons Respondent burden Incomplete, irrelevant, uninterpretable responses Coding, analyzing, reporting
![Page 31: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/31.jpg)
32
Design the questionnaire
Suggestion foropen-end questions!
Limit the number of open-end questions BUT always include
at least onefor additional comments.
![Page 32: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/32.jpg)
33
Design the questionnaire
“Forced-choice” (closed-ended) questions
Pros Less demanding on respondent Determine levels of intensity, frequency of
participation, etc. Use to form scales Easy to analyze and make comparisons
Cons Limits richness of data
![Page 33: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/33.jpg)
34
Good Questions…
1) Are clear and use simple language2) Are concise3) Are specific4) Are possible to answer5) Don’t overly tax the respondent’s memory6) Are not overly sensitive7) Are relevant to the respondent8) Do not use double negatives 9) Avoid biased terms10) Have only 1 part (not “double-barrel”)
![Page 34: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/34.jpg)
35
Good Questions…
Most importantly…
Can be connected to your objectivesand Provide usable information
![Page 35: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/35.jpg)
36
Good Questions…
Good Questions:Use simple language...
My supervisor gives me clear feedback on my job performance.
Is better than :
My supervisor provides lucid feedback when transmitting her thoughts regarding the feats I have accomplished in the preceding year.
![Page 36: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/36.jpg)
37
Good Questions…
Good Questions:Are concise...
How interested would you be in a flexible work schedule in which you could work at home one day a week?
Is better than:
How interested would you be in a flexible work schedule that would still require you to work 40 hours per week, but on one weekday per week you could not physically come in to the office but work from home?
![Page 37: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/37.jpg)
38
Good Questions…
Good Questions:Are specific...
In what year did you first start working in a permanent, full-time position at NC State University?
Is better than:
When did you start working here?
![Page 38: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/38.jpg)
39
Good Questions…
Good Questions:Are possible for the respondent to
answer...
NO:
The overhead money I bring in on my grants has a direct impact on the pay raises provided by the state legislature.
![Page 39: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/39.jpg)
40
Good Questions…
Good Questions:Do not overly tax the respondent’s
memory...
NO:
How many of your colleagues said hello to you within two weeks of you starting to work at NC State?
![Page 40: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/40.jpg)
41
Good Questions…
Good Questions:Are not overly sensitive…
NO:
How often do you take time out to pray
during the course of a normal work day?
![Page 41: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/41.jpg)
42
Good Questions…
Good Questions:Are relevant...
NO:
My supervisor should get a better haircut.
![Page 42: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/42.jpg)
43
Good Questions…
Good Questions:Do not use double negatives...
The lack of flexibility in my work hours makes it difficult for me to spend the amount of time I would like with my family.
Is better than:
The inflexibility in my work hours does not make it difficult for me to spend the amount of time I would like with my family.
![Page 43: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/43.jpg)
44
Good Questions…
Good Questions:Do not use biased terms or suggestions...
To what extent do you agree or disagree that all employees need to be in the office during regular business hours?
Is better than:
Don’t you agree with the Chancellor when he says that all employees need to be in the office during regular business hours?
![Page 44: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/44.jpg)
45
Good Questions…
Good Questions:Do not ask two questions in one (“double-
barrel”)...
My supervisor is knowledgeable about performance review policies.
My supervisor takes the time to make sure I understand performance review policies.
Is better than:
My supervisor is knowledgeable about performance review policies and takes the time to make sure I understand them as well.
![Page 45: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/45.jpg)
46
Response Options
Make sure your response options: Reflect the concepts you are trying to measure Fit with the question wording Get at level of precision/intensity needed Allow for comparisons between items Use “undecided/don’t know’ sparingly Are used consistently Use balanced scales Are mutually exclusive Are exhaustive
![Page 46: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/46.jpg)
47
Response Options
Most importantly
Relate back to your objectives
and Correspond to how want to use the results
![Page 47: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/47.jpg)
48
Response Options
Must use
Balanced scales
Example: NOT balanced...Q. How would your rate your supervisor’s leadership
skills? Outstanding
ExcellentVery goodGood
![Page 48: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/48.jpg)
49
Response Options
Balanced scales continued
Example: are balanced...Q. How would your rate your supervisor’s
leadership skills?
Excellent
Good
Fair
Poor
![Page 49: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/49.jpg)
50
Response Options
Must be: Mutually exclusive
can select only one appropriate answer
Example: NOT mutually exclusive...Q. On most days, what time do you leave your house
for work?6:00 AM or earlier 7:30-8:00 AM6:00-6:30 AM 8:00-8:30 AM6:30-7:00 AM 8:30-9:00 AM7:00-7:30 AM 9:00 AM or later
![Page 50: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/50.jpg)
51
Response Options
Mutually exclusive continued
Example: IS mutually exclusive...
Q. On most days, what time do you leave your house for work?
Before 6:00 AM 7:30-7:59 AM
6:00-6:29 AM 8:00-8:29 AM
6:30-6:59 AM 8:30-8:59 AM
7:00-7:29 AM 9:00 AM or later
![Page 51: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/51.jpg)
52
Response Options
Must be: Exhaustive
all possible answers are listed (including e.g., “other, “ “don’t know,” etc.)
Example: NOT exhaustiveQ. On most days, what time do you leave your house
for work?6:00-6:29 AM 7:30-7:59 AM6:30-6:59 AM 8:00-8:29 AM7:00-7:29 AM 8:30-8:59 AM
![Page 52: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/52.jpg)
53
Response Options
Exhaustive continued
Example: IS exhaustive
Q. On most days, what time do you leave your house for work?
Before 6:00 AM 7:30-7:59 AM
6:00-6:29 AM 8:00-8:29 AM
6:30-6:59 AM 8:30-8:59 AM
7:00-7:29 AM 9:00 AM or later
![Page 53: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/53.jpg)
54
Response Options
Some common scales: Excellent, Good, Fair, Poor Strongly agree, Agree, Disagree, Strongly disagree Very useful, Somewhat useful, Not very useful, Not
at all useful Definitely, Probably, Probably not, Definitely Not All of the time, Most of the time, Some of the time,
Seldom, Never Strongly agree, Agree, Neither agree nor disagree,
Disagree, Strongly disagree
![Page 54: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/54.jpg)
55
Questionnaire format
As short as possible Spread out & uncluttered Attractive & professional looking Broken into logical sections Have a good flow Uncomplicated & easy to follow Include simple instructions for answering questions Should have clear skip patterns for contingency
questions
![Page 55: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/55.jpg)
56
Questionnaire format
Question order Start with interesting, easy, non-threatening
questions
Remember –‘Question order effects’Earlier questions provide information and context that
influences responses to later questions
![Page 56: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/56.jpg)
57
Questionnaire
Suggestion!
Before you start collecting datause the questionnaire to
Prepare an analysis plan Outline the final report
![Page 57: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/57.jpg)
58
Pre-Test
A pre-test is
VERY VERY important!Test for:
Question clarity Questionnaire format Variance in responses Etc.
![Page 58: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/58.jpg)
59
Data collection
Stages (in a perfect world…)1. Preliminary announcement2. Cover letter, questionnaire w/ return
envelope and/or survey URL3. Reminder (with copy of questionnaire and
return envelope and/or survey URL)4. 2nd reminder (with contact info to request
copy of questionnaire and/or URL)5. Thank you card/page
![Page 59: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/59.jpg)
60
Cover or “welcome” letter
Include with any self-administered survey (On letterhead) “Signed” by most important person possible Briefly explain
purpose of survey how results will be used why respondent was selected why participation is important (how to access questionnaire)
![Page 60: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/60.jpg)
61
Cover or “welcome” letter
Continued
Emphasize confidentiality (if applicable) (Describe/explain incentives) (Follow IRB guidelines) Provide contact information THANK respondent for participating
![Page 61: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/61.jpg)
62
Cover or “welcome” letter
Warnings aboutEmail Announcements
Be sure to follow any mass email policies Think about spam filters:
Subject line text? ‘From’ field?
Try to have identifiable name/title in ‘from’ field
Keep track of undelivered/unopened emails
![Page 62: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/62.jpg)
63
A Word about Web Surveys….
LOTS of software to choose fromConsider:
Extent of involvement/control over administration Flexibility of survey design Population/sampling options Data collection capabilities non-response follow-ups Data management and analysis options Reporting capabilities Cost
![Page 63: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/63.jpg)
64
A Word about Web Surveys….
Warning!
Even if using web survey software you still need the skills to design and
administer a good, useful survey and to analyze the results!!!
![Page 64: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/64.jpg)
65
Clean, analyze, & interpret the data
Get input from your advisory committee
Remember your objectives Remember your report outline Remember your audience
Keep it simple… You can always do more analyses…
![Page 65: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/65.jpg)
66
Interpreting the results
Everything is relative!Suggestions for what to look for:
Do responses/ratings meet pre-defined goals? (e.g., % giving a positive rating)
How do responses/ratings for individual items compare to each other?
What factors/characteristics are related to ratings? How do responses/ratings from one group of respondents
compare to another? (How have ratings changed over time?)
![Page 66: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/66.jpg)
67
Prepare written report(s)
Get input from your advisory committee
Think about the intended audience What do they need to know? How they are going to use the report? How they like to get information…
![Page 67: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/67.jpg)
68
Prepare written report(s)
Generally include What you did (methods) Why you did it What you learned How it can be used
![Page 68: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/68.jpg)
69
Using the results
Take action: Some suggestions…
Spread the word!! Institute policies/changes to address
problem areas identified by ratings that were lower than predefined set goals.
Gear policies towards specific groups as needed.
Do follow-up research
![Page 69: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/69.jpg)
70
Want to learn more about surveys?
American Association of Public Opinion Research, “Best Practices for Survey and Public Opinion Research.” http://www.aapor.org/ethics/best.html
Babbie, Earl (2002), The Basics of Social Research 2nd ed. Wadsworth Thomson Learning: CA.
Dillman, D.A. (2000), Mail and Internet Surveys 2nd ed. Wiley: NY.Fink, Arlene and Jacqueline Kosecoff (1998), How to Conduct Surveys. Sage:
CA.Fowler, Floyd J. (1993), Survey Research Methods. Sage: CA.Fowler, Floyd J. (1995), Improving Survey Questions. Sage: CA.National Council on Public Polls, “20 Questions a Journalist Should Ask About
Poll Results.” http://www.ncpp.org/qajsa.htmSudman, Seymour and Bradburn, Norman (1982), A Practical Guide to
Questionnaire Design. Jossey-Bass: San Francisco.Tourangeau, R., and Smith, T.W. (1996), “Asking Senstive Questions: The
Impact of Data Collection Mode, Question Format, and Question Context, “ Public Opinion Quarterly, 60:275-304.
![Page 70: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/70.jpg)
71
Appendix A
Response Rate
# surveys completed and returned
net sample size*
*net sample size = sample size - undeliverable surveys
![Page 71: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/71.jpg)
72
Appendix A
Margin of ErrorBased on
# people in the sample (or population) # people who respond the response distribution
e.g., 75% said yes, 25% said no how sure you want to be of you data (“confidence
interval”) e.g., a 95% confidence interval means that you can be
sure that 95 out of 100 times the responses to a given question would be within some calculable percentage points of the actual number you got.
![Page 72: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/72.jpg)
73
Appendix A
Margin of Error Examples (at 95% confidence interval)
Population Size # Respondents Margin of Error
100 75 + 2.8
40 30 + 4.5
10 7 + 11.1
E.G. If 50% of the 7 respondents (from your population of 10) said “yes,” you can be sure that 95 out of 100 times between 38.9% and 61.1% of those in your population would say “yes.”
![Page 73: 1 SURVEY RESEARCH Nancy Whelchel Assistant Director for Survey Research University Planning and Analysis NCSU nancy_whelchel@ncsu.edu 515-4184](https://reader036.vdocument.in/reader036/viewer/2022070307/551aa904550346856e8b49ea/html5/thumbnails/73.jpg)
74
Appendix B
Survey Question ‘Banks’ Eagleton Poll Archives
http://www.scc.rutgers.edu/eagleton/ The Roper Center (fee based)
http://www.ropercenter.uconn.edu/ The Odum Institute
http://www.irss.unc.edu/data_archive The Gallup Poll
http://poll.gallup.com/ ICPSR
http://www.icpsr.umich.edu/