COLLECTING DATA ON A SAMPLE OF RESPONDENTS

Designing survey instruments

Remember:

The sample survey is a research technique used to estimate population parameters

A number of people are asked a series of questions from a standardized instrument concerning a variety of topics, and the answers are analyzed Interviews v. questionnaires

Options for administration

Personal interviewsPhone interviewsMail questionnairesComputer-mediated questionnaires

Survey design is strongly influenced by administration option chosen

The importance of quality instruments

A well-designed instrument is crucial to the success of any sample survey or census

With few exceptions, poorly written instruments represent a much greater threat to the validity of your conclusions than do sampling problems or other common threats to survey research validity

How do we go about developing the instrument?

The most important thing is knowing what we want to know What are the most significant questions

we have? What information is crucial to answering

these questions? Guided by literature review

What information can best be obtained in other ways?

How do we go about developing the instrument?

Prioritize the importance of the informationCrucial v. important v. useful but not

necessary Getting really good information on a limited set of constructs may be more valuable than a lot of scattered facts and incomplete ideas on many constructs

Don’t go fishing!

Too often researchers include many more questions than necessary Too many ‘interesting but unnecessary’

questions Tradeoff between additional questions and

completion rateTry to avoid a phone survey that lasts

more than 15 minutes Less for questionnaires

Typical instrument format

Greeting and introductionEarly general questions

Develop rapport Get the respondent thinking about the

topics of study Generate ‘top-of-mind’ data

Early to mid surveyFocus on topics of interest

Use closed-ended but still somewhat general questions

Move from category to brand questions, medium to station to program questions, etc.

Late survey questions More open-ended questions Questions dependent upon deeper review

of content Respondent has been thinking about

topics for a while Edgiest questions

Greatest level of rapport necessary

Final questionsMost embarrassing/difficult

questionsDemographics (often lose

respondents when you ask about income, etc.)

Closing Thank your respondents for their time Reassure them that their answers were

helpful and useful regardless of the quality of responses/level of knowledge of the respondent

Any necessary debriefing, especially if you were relatively guarded at the beginning or engaged in some sort of misleading or deceptive instructions, etc.

Impact of data collection method

Construct the instrument according to the method of data collection Self-administered questionnaires require

more instructions on the instrument itself than do interviews

When the instrument is administered by an interviewer, written instructions to the interviewer (usually placed in the margin) can be useful in improving response and/or validity of the information gathered

Skip Patterns

When you want to avoid asking inappropriate questions, you first ask a ‘screening question’ to identify appropriate respondents Those for whom the question is

inappropriate are not asked the question and ‘skip’ forward to the next appropriate question on the instrument

Example

If you want to ask questions about the kinds of beer someone drinks, you would first identify respondents who do not drink beer and have them skip to the next question appropriate to them

Example:1. Do you ever drink beer?

Yes (continue) No (skip to Q. 3)

2. Which brands do you drink? ______________

3. Do you ever drink coffee?

Writing the questions

Levels of measurement The higher the level of measurement, the greater the

ability to use powerful statistics to analyze your dataScale typesResponse options

Yes/no Agree/disagree

(Strongly, somewhat) Numeric responses v. categories

Question formats

Try to develop repetitive response options Increases speed Makes respondent feel more comfortable Reduces errors Boxes, check-offs, white space

Question wording

The wording of questions on any research instrument is crucial. There is usually no more important decision than how a question is asked. Results from surveys have been shown to

be strongly influenced by even subtle differences in question wording.

Instrument format

Matters of crowding, use of white space, inclusion of instructions, typeface, and so on are important.

Test the instrument

Pre-test the instrument Have trained interviewers try out your draft on

each other, Determine whether questions are misleading,

ambiguous, biased Determine how long the instrument takes to

completeUse this information honestly when recruiting

respondents Ask how the respondent interprets the question Allow for open-ended responses where closed-

ended questions have been written

Developing the instrument

Revise and reorganize the instrument based on the feedback from the testing

Try it out on respondents from the population being studied

Make final adjustments Always keep track of the changes you made

Developing the instrument

Obtain IRB approval (if this is an academic study)

Enter the field and gather data Monitor the success of the instrument and make

changes if absolutely necessary Changes must be run by the IRB

A number of concerns influence the structure of the instrument:

First and foremost the essential information needed to answer the research questions/hypotheses

Data collection method Interview, self-administered questionnaire, online v.

paper, etc. Amount and type of instruction, skip patterns,

incentivesThe normal progression of the interview/

questionnaire Developing rapport/trust

Dynamics of respondent memory Question ordering

A number of concerns influence the structure of the instrument:

Respondent facility with language and/or advanced concepts Kids Recent immigrants

Sensitivity of question topics Forms and methods of providing confidentiality

Efficiency in asking questions