Developing a Questionnaire

Chapter 4

Types of Questions

• Open-ended – high validity, low manipulative quality

• Closed-ended – low validity, high manipulative quality

Open-ended

• An open-ended question is one in which you do not provide any standard answers to choose from.

1.How old are you? ______ years.

2.What do you like best about your job?

Closed-ended

• A closed-ended question is one in which you provide the response categories, and the respondent just chooses one:

What do you like best about your job?(a) The people(b) The diversity of skills you need to do it(c) The pay and/or benefits(d) Other: ______________________________

Dichotomous Questions

• Dichotomous Question: a question that has two possible responses

– Could be

• Yes/No

• True/False

• Agree/Disagree

Questions based on Level of Measurement

• Use a nominal question to measure a variable – Assign a number next to each response that

has no meaning; simply a placeholder.

• Use an ordinal question to measure a variable– Rank order preferences– More than 5 – 10 items is difficult– Does not measure intensity

Interval Level

• Attempt to measure on an interval level– Likert response scale: ask an opinion question

on a 1-to-5, 1-to-7, etc. bipolar scale • Bipolar: has a neutral point and scale ends are at

opposite positions of the opinion

– Semantic differential: an object is assessed by the respondent on a set of bipolar adjective pairs

– Guttman scale: respondent checks each item with which they agree; constructed as cumulative, so if you agree to one, you probably agree to all of the ones above it in the list

Filter/Contingency Questions

• To determine if a respondent is ‘qualified’ to answer questions, might need a filter or contingency question (also known as knowledge)

– Limit # of jumps

– If only two levels, use graphic to jump

– If you can't fit the response to a filter on a single page, it's probably best to be send them to a page, rather than a question #

How many steps in the response scale?

• Statistical reliability of the data increases sharply with the number of scale steps up to about 7 steps

– After 7, it increases slowly, leveling off around 11

– After 20, it decreases sharply

Should there be a middle category?

• Does it make sense to offer it?

• Should not be used as the “don’t know or no opinion” option.

 

– The middle option is usually placed between the positive and negative responses.

– Sometimes it’s last in an interview.

Direct Magnitude Scaling

• Method of obtaining ratio-scaled data– Idea is to give respondents an anchor point, and

then ask them to answer questions relative to that

• Example:– Suppose you are interested in the severity of

crimes.• Begin by assigning a number to one crime

and then have respondents assign numbers to the others based upon a ratio.

Filtering "Don't Know"

• Standard format – No "don't know" option is presented to the respondent,

but is recorded if the respondent volunteers it.

• Quasi filter – A "don't know" option is included among the possible

responses.

• Full filter– First the respondent is asked if they have an opinion.

If yes, the question is asked.

Question Placement• It's a good idea to put difficult, embarrassing or

threatening questions towards the end

– More likely to answer.

– If they get mad and quit, at least you've gotten most of your questions asked!

• Put related questions together to avoid giving the impression of lack of meticulousness

• Watch out for questions that influence the answers to other questions.

Wording of Questions

• Direction of Statements– Response bias– Socially desirable

• Always and never– Avoid this– Better to phrase as ‘most’, ‘infrequently’

• Language– Reflect educational level and reading ability– Need for various languages

Frequency and Quantity

• Consider both frequency and quantity

– Consider number of times

– Consider duration of times

Mutually Exclusive and Exhaustive

• Mutually exclusive: not possible to select more than one category/value

• Exhaustive: providing all possible categories/values

Forced Choice

• Choose between 2 choices

– Might not be relevant

– Other choices exist (or at least possible)

– Lesser of two evils

Recalling Behavior

• Can be difficult to remember

• Ask questions that can be answered

• Choose time frames that are reasonable

• Pilot test for time frame issues

Response Bias

• Exaggerating the truth

• Socially desirable answers

• Consider using ‘trap’ questions– Possibly fictional choice

Sensitive Items

• More comfortable answering in categories

– Minimize missing data

– Might loose statistical power

Evaluating Questions

• Pre-testing

• Cognitive interviewing

• Behavior coding

• Peer review

• Peer review has shown to be the best method but it’s the least used. 

Validity and Reliability Questions

• Evaluative strategies:

– Analysis of data to evaluate the strength of predictable relationships among answers and with other characteristics of respondents.

– Comparisons of data from alternatively worded questions asked of comparable samples.

– Comparison of answers against records.

– Measuring the consistency of answers of the same respondents at two points in time.

Coding the Questionnaire

• Create a codebook: reference guide for the data set

• Code: assigning a value to a response category– Often numeric code– Pre-coding makes it easier– Content analysis on open-ended items– Yes/No often coded as present or not (0 or 1)

Missing Responses

• Why blank?– Missed them

– Refusal to answer

– Didn’t feel it applied

– Didn’t know the answer

• To code or not– Analyze the difference

– If know why, might consider

Piloting the Questionnaire

• Test it on yourself– Possibly other experts

• Test on people similar to sample– Don’t reuse (some exceptions)

• Discuss the survey with individuals– During completion or After

Finding Respondents

• Best Methods of Selection

• Even with a good survey, poorly chosen sample leads to poor results