27

Survey Research

I. Survey Research a. The survey is the most widely used data gathering technique in sociology, and it is used in

many other fields, as well. In fact, surveys are almost too popular. People sometimes say, "Do a survey to get information about the social world, when they should be asking: "What is the most appropriate research design?" Despite the popularity of surveys, it is easy to conduct a survey that yields misleading or worthless results. Good surveys require thought and effort. Surveys, like other scientific and technical tools, can be well made or poorly and can be used in appropriate or inappropriate ways. All surveys are based on the professional social research survey. In this section, you will learn the main ingredients of good survey research, as well as the limitations of the survey method.

b. Research Questions Appropriate for a Survey i. Behavior

1. Examples a. How frequently do you brush your teeth? b. Did you vote in the last election?

ii. Attitudes/Beliefs/Opinions 1. Examples

a. What kind of job do you think the mayor is doing? b. What is the biggest problem facing the nation today?

iii. Characteristics 1. Examples

a. Are you married? b. What is your age?

iv. Expectations 1. Examples

a. Do you plan to buy a new car in the next 12 months? b. Do you thin k the population in this town will shrink?

v. Self-Classification 1. Examples

a. Do you consider yourself liberal? b. Into which social class would you place your family?

vi. Knowledge 1. Examples

a. Who was elected mayor in the last election? c. What is a Survey?

i. A survey is a method of collecting information directly from people about their ideas, feelings, health, plans, beliefs, social background, educational level, and financial history. A survey can be a self-administered questionnaire that someone fills out alone or with assistance. Or a survey can be an interview that is done in person or on the telephone. Some surveys are on paper or disk and the respon-dent can complete them privately at home or in a central location, say, at a health center. The respondent can either return the completed survey on the disk or mail the responses electronically. Surveys can also be interactive, requiring a telephone or cable connection. Interactive surveys guide the respondent through the survey and provide audio and visual cues to help. Interviews may be conducted with or without the aid of a computer. In other words, survey researchers sample many respondents who answer the same questions. They measure many variables, test multiple hypotheses, and infer temporal order from questions about past behavior, experiences, or characteristics.

d. Constructing the Questionnaire i. Principles of Good Question Writing

1. Avoid Jargon, Slang, and Abbreviations

28

a. Jargon and technical terms come in many forms. Plumbers talk about snakes and psychologists about the Oedipus complex. Also avoid abbreviations. NATO usually means North Atlantic Treaty Organization, but for a respondent, it might mean something else (National Auto Tourist Organization).

2. Avoid Ambiguity, Confusion, and Vagueness a. Be clear when requesting information from respondents. For

example, “What is your income?” could mean weekly, monthly, or annual; family or personal; before taxes or after taxes; for this year or last year; from salary or from all sources. The confusion causes inconsistencies in how different respondents assign meaning to and answer the questions.

3. Avoid Emotional Language and Prestige Bias a. Use neutral language when constructing questions. Avoid words

with emotional “baggage” because respondents may react to the emotionally laden words rather than to the issue. Also avoid prestige bias- associating a statement with a prestigious person or group. For example, saying,” Most doctors say that cigarette smoke causes lung cancer. Do you agree?”

4. Avoid Double-Barreled Questions a. Make each question about one and only one topic. A double-

barreled question consists of two or more questions joined together. For example, “Does this company have pension and health care benefits?” a respondent is forced to answer with one response to a total of two different questions (pension benefits AND health care benefits).

5. Avoid Leading Questions a. A leading question is one that leads the respondent to choose

one response over another by its wording. 6. Avoid Asking Questions That are Beyond the Respondent’s Capability

a. Asking something few respondents know frustrates respondents and produces poor-quality responses. For example, “How did you feel about your parents divorce when you were 3 years old?”

7. Avoid False Premises a. Do not begin a question with a premise with which respondents

have not already agreed to. For example, “What time do you have to wake-up to take your children to school? Before asking this questions we must be absolutely sure that they have children before assuming that they have to wake-up at a specified time to get them to school.

8. Avoid Asking About Future Intentions a. Responses are poor predictors of behavior.

9. Avoid Double Negatives a. Double negatives in ordinary language are grammatically

incorrect and confusing. For example, “ I ain’t got no job” logically means that the respondent does have a job, but the second negative is used in this way for emphasis.

10. Avoid Overlapping or Unbalanced Response Categories a. Mutually Exclusive.

i. Mutually exclusive means that response categories do not overlap.

b. Exhaustive i. Exhaustive means that every respondent has a choice.

c. Balanced i. Keep response categories balanced. A case of

unbalanced choices is the question, “What kind of job is the mayor doing: outstanding, excellent, good, or

29

satisfactory?” Here, the only response possible is a positive one.

e. Aiding Respondent Recall and Other Issues Related to Participants’ Response Rates i. Researchers should be made aware of the realities of memory and the many factors

affecting it. In addition, researchers should also take into consideration that a non-response might be the result of some other factor besides poor memory or forgetfulness. In most cases, when a question goes unanswered, the researcher does not know if it is the result of poor memory recall, poor questionnaire design, confusion, or simply a decision by the respondent to skip the question due to the sensitive nature of the question.

1. Factors Affecting Recall a. The Topic

i. Threatening 1. In some cases, respondents will choose to

ignore questions that require very personal information.

ii. Socially Desirable 1. May not only effect response rate but also the

type of response. b. Events Occurring Simultaneously

i. Memory recall may be poor when asking about events that occurred during a time when multiple events were taking place simultaneously.

c. The Significance of an Event for a Particular Respondent i. The more significant an event for a respondent, the

more likely they will be able to recall the details of an event. The reverse is true as well.

d. Situational Conditions i. For some responses, an event may have situational

conditions specific to them that may result in better or poorer memory recall.

e. Question Wording i. Refer to the principles of good question writing on

page 2. f. Interview Style

i. Anonymous 1. Anonymity provides interviewees with a level

of protection that seems to provide them with enough security to answer questions more honestly when compared to other interview styles.

ii. Confidential g. Respondent’s Need to Have Internal Consistency h. Telescoping

i. Many respondents will telescope or compress time when asked about the frequency and over-report recent events.

1. Techniques to reduce telescoping a. Situational Framing

i. Ask a respondent to recall a specific situation and ask them about it.

b. Decomposition i. Ask several specifics and

then add them up- such as how much one drank in a

30

week then total for drinking in a year.

f. Types of Questions and Response Categories i. Threatening Versus Non-Threatening Questions

1. Threatening Questions a. Are part of a broader issue. Respondents may try to present a

positive image of themselves to interviewers or researchers instead of giving true answers. Remember, the more threatening the question, the more likely a respondent will either elect to ignore the question or answer it untruthfully.

ii. Knowledge Questions 1. Respondents sometimes want to find out whether respondents know

about an issue or topic, but knowledge questions can be threatening because respondents do not want to appear ignorant.

iii. Skip or Contingency Questions 1. Researchers avoid asking questions that are irrelevant for a respondent. A

contingency question is a two (or more) part- question. The answer to the first part of the question determines which question the respondent proceeds to.

iv. Open-Ended Questions 1. An open-ended question (unstructured, free response) asks a question to

which respondents can give any response. a. Advantages

i. Permit an unlimited number of answers. ii. Respondents can answer in detail. iii. Unanticipated findings can be discovered. iv. Permit creativity, self-expression. v. Reveal a respondent’s logic, thinking process, and

frame of reference. b. Disadvantages

i. Respondents give different degree of detail. ii. Responses may be irrelevant or buried in useless detail. iii. Comparisons and statistical analysis become very

difficult. iv. Coding responses is difficult. v. Highly articulate respondents have an advantage. vi. Respondents may lose direction when answering. vii. Responses are written verbatim, which is difficult for

interviewers. viii. Greater amount of respondent’s time, thought, and

effort. ix. Respondents can be intimidated by the questions. x. Answers require a great deal of space.

v. Closed-Ended Questions 1. A closed-ended (structured or fixed) both asks a question and gives the

respondent a fixed set of responses to choose from. a. Advantages

i. Easier and quicker for respondents to answer. ii. Easy to compare different respondents answers. iii. Answers are easy to code and statistically analyze. iv. Response categories can clarify the question. v. Respondents are more likely to respond to sensitive

questions. vi. Fewer irrelevant or confusing answers to questions. vii. Less articulate respondents are not at a disadvantage. viii. Replication is easier.

b. Disadvantages

31

i. Suggest ideas that the respondent would not normally have.

ii. Respondents with no prior knowledge or attitude may respond anyway.

iii. Respondents may become frustrated because their desired response is not found.

iv. It is confusing if too many responses are available. v. Misinterpretation of a question may go unnoticed. vi. Distinctions between respondent answers may become

blurred. vii. Clerical mistakes on the part of the respondent may

occur. viii. Force people to make choices they would not normally

have to make. vi. Non-Attitudes and Middle Positions

1. Survey researchers debate whether to include choices for neutral, middle, and non-attitudes (e.g., not sure, don’t know, or no opinion). Two types of errors can be made: accepting a middle choice or “no attitude” response when respondents hold a non-neutral opinion, or forcing respondents to choose a position on an issue when they have no opinion about it (e.g., false positives and false negatives). The issue of non-attitudes can be approached by distinguishing among three kinds of attitude questions:

a. The standard-format question does not offer a "don't know" choice; a respondent must volunteer it.

b. A quasi-filter question offers respondents a "don't know" alternative.

c. A full-filter question is a special type of contingency question. It first asks if respondents have an opinion, then asks for the opinion of those who state that they do have an opinion.

vii. Agree/Disagree, Rankings or Ratings? 1. Researchers often debate the type of response categories that are

appropriate or which will yield the most accurate results regarding respondents’ attitudes. There exists some agreement that it is better to ask respondents to choose among alternatives by ranking instead of rating items along an imaginary continuum. Respondents can rate several items equally high, but will place them in a hierarchy if asked to rank them.

viii. Wording Issues 1. Use simple vocabulary. 2. Avoid words that “trigger” responses.

g. Questionnaire Design Issues i. Length of the Survey

1. How long should a questionnaire be or an interview last? Researchers are often torn between the amount of data they would like to collect and the realistic volume a respondent will accurately and honestly complete. There exist a number of issues affecting the length of a survey, but keep in mind there is no absolute proper length.

a. The longer the survey, the more likely respondents will find it too burdensome and either elect to stop or report inaccurate data as a result of frustration or impatience.

b. Most importantly, the length depends on the survey format and on the respondent's characteristics. For example, a 10-minute telephone interview is rarely a problem and can usually be extended to 20 minutes. A few researchers stretched this to beyond 30 minutes. Mail questionnaires are more variable. A short (3- or 4-page) questionnaire is appropriate for the general population. Some researchers have had success with

32

questionnaires as long as 10 pages (about 100 items) with the general public, but responses drop significantly for longer questionnaires. Face-to-face interviews lasting an hour are not uncommon. In special situations, face-to-face interviews as long as three to five hours have been conducted.

ii. Question Order or Sequence 1. A survey researcher faces two question sequence issues.

a. Organization i. The first is how to organize items in the overall ques-

tionnaire. In general, you should sequence questions to minimize the discomfort and confusion of respondents. Make question topics flow smoothly and logically, and organize them to assist respondents' memory or comfort levels. Do not end with highly threatening questions; and always end with a "thank you.”

b. Context Effects i. The second involves context effects of answering

specific questions before others. Researchers are concerned that the order in which questions are presented may influence respondents' answers. These order effects are strongest for respondents who lack strong opinions or who are less well educated. They use previous questions as a context to help them answer later questions. You can do two things about specific question order effects:

1. Use a funnel sequence of questions that is; ask more general questions before specific ones (e.g., ask about health in general before asking about specific diseases).

2. Or, divide the number of respondents in half and give one half questions in one order and the other half the alternative order, then examine the results to see whether question order mattered. If question order effects are found, which order tells you what the respondents really think? The answer is that you cannot know for sure.

iii. Non-Response, Refusals, and Response Rates 1. Researchers are always burdened with the idea that respondents will

choose to not respond and ignore any requests to collect data. Failure to get a response from a sampled respondent can take several forms (Refer to the literature on ways to overcome, at least in part, some of the obstacles facing researchers regarding non-responses and refusals):

a. The respondent could not be contacted; he or she was contacted but was unable to complete the survey (e.g., spoke another language, had no time, was ill, etc.).

b. He or she refused to complete a questionnaire or refused to be interviewed.

c. Or he or she refused to answer some questions. iv. Format and Layout

1. There are two format or layout issues: a. Physical Layout

i. The overall physical layout of the questionnaire. Surveys should be clear, neat, organized and easy to follow. There should also be explanations and instructions located above every section of questions. The more organized and simpler the survey, the more likely

33

respondents will complete a survey accurately and honestly.

b. The Format of Questions and Responses i. Survey researchers decide on a format for questions

and responses. Should respondents circle responses, check boxes, fill in dots, or put an x in a blank? The principle is to make responses unambiguous.

h. Types of Surveys i. Mail and Self-Administered

1. Researchers can give questionnaires directly to respondents or mail them to respondents who read instructions and questions, then record their answers.

a. Advantages i. This type of survey is by far the cheapest, and a single

researcher can conduct it. ii. A researcher can send questionnaires to a wide ge-

ographical area. iii. The respondent can complete the questionnaire when it

is convenient and can check personal records if necessary.

iv. Mail questionnaires offer anonymity and avoid interviewer bias.

v. They are very effective, and response rates may be high for a target population that is well educated or has a strong interest in the topic or the survey organization.

b. Disadvantages i. Since people do not always complete and return

questionnaires, the biggest problem with mail questionnaires is a low response rate. Most questionnaires are returned within two weeks, but others trickle in up to two months later. Researchers can raise response rates by sending non-respondents reminder letters, but this adds to the time and cost of data collection.

ii. A researcher cannot control the conditions under which a mail questionnaire is completed. A questionnaire completed during a drinking party by a dozen laughing people may be returned along with one filled out by an earnest respondent.

iii. Also, no one is present to clarify questions or to probe for more information when respondents give incomplete answers.

iv. Someone other than the sampled respondent (e.g., spouse, new resident, etc.) may open the mail and complete the questionnaire without the researcher's knowledge. Different respondents can complete the questionnaire weeks apart or answer questions in a different order than that intended by researchers.

v. Incomplete questionnaires can also be a serious problem.

vi. Researchers cannot visually observe the respondent's reactions to questions, physical characteristics, or the setting. For example, an impoverished 70-year-old White woman living alone on a farm could falsely state that she is a prosperous 40-year old Asian male doctor living in a town with three children. Such extreme lies are rare, but serious errors can go undetected.

34

vii. The mail questionnaire format limits the kinds of questions that a researcher can use. Questions requiring visual aids (e.g., look at this picture and tell me what you see), open-ended questions, many contingency questions, and complex questions do poorly in mail questionnaires. Likewise, mail questionnaires are ill suited for the illiterate or near illiterate in English. Questionnaires mailed to illiterate respondents are not likely to be returned; if they are completed and returned, the questions were probably misunderstood, so the answers are meaningless.

ii. Telephone 1. The telephone interview is a popular survey method because about 95

percent of the population can be reached by telephone. An interviewer calls a respondent (usually at home), asks questions, and records their answers.

a. Advantages i. Researchers sample respondents from lists, telephone

directories, or use RDD, and can quickly reach many people across long distances. A staff of interviewers can interview 1,500 respondents across a nation within a few days and, with several callbacks.

ii. Response rates can reach 90 percent. iii. Although this method is more expensive than a mail

questionnaire, special reduced long distance phone rates help. In general, the telephone interview is a flexible method with most of the strengths of face-to-face interviews but for about half the cost.

iv. Interviewers control the sequence of questions and can use some probes.

v. A specific respondent is chosen and is likely to answer all the questions alone.

vi. The researcher knows when the questions were answered and can use contingency questions effectively, especially with computer-assisted telephone interviewing (CATI) (to be discussed).

b. Disadvantages i. Relatively high cost. ii. Limited interview length. iii. Respondents without telephones are impossible to

reach, or the call may come at an inconvenient time. iv. The use of an interviewer reduces anonymity and

introduces potential interviewer bias. v. Open-ended questions are difficult to use, and

questions requiring visual aids are impossible. vi. Interviewers can only note serious disruptions (e.g.,

background noise) and respondent tone of voice (e.g., anger or flippancy) or hesitancy but lack the insight gained by watching them visually while responding.

iii. Face-To-Face Interview 1. A face-to-face interview is a research technique that requires an

interviewer and respondent to be physically present together. During which, the interviewer asks the interviewee a series of questions that are answered only during the interview session.

a. Advantages i. Face-to-face interviews have the highest response rates. ii. Permit the longest questionnaires.

35

iii. They have the advantages of the telephone interview, and interviewers also can observe the surroundings and can use nonverbal communication and visual aids.

iv. Well-trained interviewers can ask all types of questions, can ask complex questions, and can use extensive probes.

b. Disadvantages i. High cost is the biggest disadvantage of face-to-face

interviews. The training, travel, supervision, and personnel costs for interviews can be high.

ii. Interviewer bias is also greatest in face-to-face interviews. The appearance, tone of voice, question wording, and so forth of the interviewer may affect the respondent.

iii. In addition, interviewer supervision is less than for telephone interviews, which supervisors monitor by listening in.

i. Costs i. Costs Can Vary Greatly and is Dependent on a Number of Factors

1. Factors Effecting Cost a. Size of Sample b. Amount of Time Spent on Qualitative Responses (e.g., number

of open ended questions) c. Type of Survey (e.g., $1 for a mail survey, $5 for a phone

interview, and $15 for a face-to-face interview) d. Number of Pages or Questions e. Size of Professional Staff f. Training g. Response Rates

j. The Ethical Survey i. Like all social research, people can conduct surveys in ethical or unethical ways.

1. A major ethical issue in survey research is the invasion of privacy. Survey researchers can intrude into a respondent's privacy by asking about intimate actions and personal beliefs. People have a right to privacy. Re-spondents decide when and to whom to reveal personal information. They are likely to provide such information when it is asked for in a comfortable context with mutual trust, when they believe serious answers are needed for legitimate research purposes, and when they believe answers will remain confidential. Researchers should treat all respondents with dignity and reduce anxiety or discomfort. They are also responsible for protecting the confidentiality of data.

2. A second issue involves voluntary participation by respondents. Respondents agree to answer questions and can refuse to participate at any time. They give "informed consent" to participate in research. Researchers depend on respondents' voluntary cooperation, so researchers need to ask well-developed questions in a sensitive way, treat respondents with respect, and be very sensitive to confidentiality.

3. A third ethical issue is the exploitation of surveys and pseudo-surveys. Because of its popularity, some people use surveys to mislead others. A pseudo-survey is when someone uses the survey format in an attempt to persuade someone to do something and has little or no real interest in learning information from a respondent.

4. Another ethical issue is when people misuse survey results or use poorly designed or purposely rigged surveys. People may demand answers from surveys that surveys cannot provide or may not understand a survey's limitations. Those who design and prepare surveys may lack sufficient training to conduct a legitimate survey. Policy decisions made based on

36

careless or poorly designed surveys may result in waste and human hardship. Such misuse makes it important that legitimate researchers conduct methodologically rigorous survey research.

5. Currently, there are no quality-control standards to regulate the opinion polls or surveys reported to the U.S. media. Researchers have made unsuccessful attempts since World War II to require adequate samples, interviewer training and supervision, satisfactory questionnaire design, pub-lic availability of results, and controls on the integrity of survey organizations.

II. Computer Assisted Telephone Interviewing a. Advances in computer technology and lower computer prices have enabled professional

survey research organizations to install computer-assisted telephone interviewing (CATI) systems. With CATI, the interviewer sits in front of a computer and makes calls. Wearing a headset and microphone, the interviewer reads the questions from a computer screen for the specific respondent who is called and then enters the answer via the keyboard. Once he or she enters an answer, the computer shows the next question on the screen.