COM317 800

Taejin Jung, Ph.D.

Week 11: Survey and Poll Methods

Survey Research

Data collection through communication with a representative sample

Used to:- Describe (Description)

- Learn why? (Explanation)

Advantages- Quick

- Inexpensive

- Efficient

- Generally accurate

Classifying Survey Methods Method of Communication

- Personal Interview- Self-administered survey- Telephone- Mail- Internet

Temporal Dimension (Time)- Longitudinal studies (over time)- Cross-sectional studies (a point in time)

Types of Question - Structured versus unstructured- Disguised versus undisguised

Personal Interview Characteristics

- Direct communication- Questions asked face-to-face

Types of Personal Interview- Door-to-doora. Response high if appointments usedb. May exclude large groups of peoplec. Requires “call back”

- Mall interceptsa. High refusal rateb. Can demo products

Advantages- Opportunity for feedback- Ability to probe for details- Interview can be long- High probability of complete interview- Can use props, visual aids- Higher overall response rate

Disadvantages- Interviewer influence- No Anonymity of respondent- Costs

Self-Administered Survey Respondent takes the responsibility for reading and

answering the questions.

Types- Mail- Package inserts- “Pick-up” surveys- Internet

Key Issues- Mail : non-response- Others : self-selection

Telephone Interviews Advantages

- Speed

- Low cost relative to personal

- More likely to divulge personal info than in personal interview

- Easier to obtain access and cooperation

Disadvantages

- Absence of face-to-face

- Refusal rates may be high

- Interview must be short

- Lack of visuals

Mail Surveys

Advantages

- Geographic reach and flexibility

- Relatively low cost

- Convenient for the respondent

- Anonymity can be achieved

- More likely to obtain sensitive information

- Survey can be longer than phone survey

Disadvantages

- Subject to different interpretations by respondents

- Questions should be structured

- Low response rate

- Data collection is slow

- Item non-response is high

Internet Survey Advantages- Least expensive- Faster turnarounds possible- Quick feedback from

customers and/or employees- Surveys can be interactive- Elimination of researchers- More Internet survey software

packages available- Internet surveys better than

website feedback forms- Good response rate (e.g.,

telephone survey response rate has dropped to 30% or less)

Disadvantages- Not sure who is really

responding- Sampling issues

a. May not be representative of the populationb. No guarantee that demographics are accuratec. Bad e-mail addressesd. Multiple e-mail addresses from the same persone. Inaccessibility of the poor and elderlyg. Self-selection biasf. Not considered a mainstream research methodology

Comparison of Survey

Mail Telephone Person-to-person

Internet

Cost per response

Speed of return

No. of interviews completed

Design constraints

Risk of interviewer bias

Survey control

Anonymity of response

Low

Low

Low

Medium

Low

Medium

High

Medium

High

High

High

Medium

High

Medium

High

Medium

High

Low

Medium

High

Low

Low

High

Medium

High

Medium

High

Low

Temporal Dimension

Longitudinal Studies

- Collection of data at different points in time

- Types

a. Trend

b. Cohort

c. Panel

Cross-sectional studies

- Collects data at a single point in time

- Often examines different “sections” of the population

Types of Questions

Structured vs Unstructured- Structured questions have a limited and known set of choices

- Unstructured questions are open-ended

Disguised vs undisguised- Disguised questions uses techniques to obtain information without

directly asking the question under study.

- Disguised questions used for sensitive studies.

Criteria for a Good Questionnaire

Check Points

- Does the questionnaire provide the necessary decision making information?

- Are the measures reliable and valid?

- Does the questionnaire consider the respondent?

Avoiding Common Problems

- Avoid complexity

- Avoid leading and loaded questions

- Be as specific as possible

- Avoid double-barreled questions

- Avoid making assumptions

- Avoid burdensome questions

Steps in Questionnaire Development

Step 1: Determine Survey Objectives, Resources, and Constraints

- Survey objectives: The decision-making information sought through the questionnaire.

Step 2: Determine Data Collection Method(s)- Personal Interview

- Self-administered survey

- Telephone

- Mail

- Internet

Step 3: Determine Question Response Format

- Open-ended questions: Questions that ask respondents to reply in their own words.]

- Closed-ended questions: Questions that ask respondents to choose from a list of answers.

a. Dichotomous questions: Questions that ask respondents to choose between two answers.

b. Multiple-choice questions: Questions that ask respondents to choose among a list of more than two answers.

c. Scale-response questions: Multiple-choice questions with the choices designed to capture the intensity of the respondents’ answers.

Step 4: Decide Question Wording

Four general guidelines are useful to bear in mind during the wording and sequencing of each question.

- The wording must be clear.

- Select words so as to avoid biasing the respondent.

- Consider the ability of the respondent to answer the question.

- Consider the willingness of the respondent to answer the question

Shoulds & Should-Nots of Question Wording

Shoulds- Question should be focused on a single issue or topic

- Question should be brief

- Question should be interpreted the same by all respondents

- Question should use respondent’s core vocabulary

- Question should be as grammatically simple as possible

Should-Nots- Should not assume criteria that are not obvious

- Should not be beyond the respondent’s ability or experience

- Should not use a specific example to represent a general case

- Should not ask the respondent to recall specifics when only generalities will be remembered

- Should not require respondent to guess a generalization

Step 5: Establish Questionnaire Flow and Layouts

Questionnaire Design

- Layouta. Use white space

b. Attractive and easy to follow

- Order of questiona. Sensitive questions at the end

b. Ask general questions before specific questions

c. Be aware of order bias

Questionnaire Order- Screens

- Warm-Ups

- Transitions

- Complicated and Difficult to Answer Questions

- Classification and Demographic Questions

Step 6: Evaluate the Questionnaire and Layout

Is the question necessary? Is the questionnaire too long? Will the questions provide the desired information to

accomplish the research objectives? Appearances of mail and self-Administered

questionnaires Avoid a cluttered look Allow plenty of space for open-ended responses Consider color-coding the questionnaires Instructions printed within the questionnaire should be

distinguishable from questions

Steps in Questionnaire Development

Step7: Obtain Approval from All Relevant Parties

Step 8: Pretest and Revise- Pretest: Trial run of a

questionnaire.

Step 9: Prepare Final Copy

Pre-testing- Trying it out on a small

sample, then making changes

- critical to having an easily understood questionnaire

►Translation- Best to use back translation

- English-foreign language-English

Step 10: Implementation

Supervisor’s instructions: Written directions to the field service on how to conduct the survey.

Interviewer’s instructions: Written directions to the interviewer on how to conduct the interview.

Call record sheets: Interviewers’ logs listing the number and results of a contact.

Field management companies: Firms that provide support services such as questionnaire formatting, screener writing, and data collection to full-service research companies.

Factors to Determine Survey Method

Sampling Method Budget Type of questioning

a. Need for stimuli

b. Structure

Length of questionnaire Incidence rate Time

Errors in Survey Research Random sampling errors

- Chance variations in the elements selected for the sample which make the results non-representative

Systematic errors-(Respondent or administrative error) - Result from some problem with the

design or execution of the research.

- “Mistakes”

Types of Respondent error- Non-response error

- Response bias

• Non-response errors

- Differences between the answers obtained from those who responded as compared to the answers which would have been obtained if everyone in the sample responded.- Includes refusals and not-at-homes- Often those that respond have- stronger feelings than those who do

not.

Reducing Non-Response Error- Call-backs and follow-ups- Pre-set appointments or call ahead - Incentives- Cover letters- Prestigious sponsorship

Response Bias

When respondents answer untruthfully, deliberately or not

Types of Response Bias- Acquiescence bias - a tendency to agree with all or most questions.

- Extremity bias - a tendency to answer on the extremes

- Interviewer bias - the presence of the interviewer influences responses

- Auspices bias - the sponsoring agency influences the responses

- Social desirability bias - a tendency to give a “socially correct” answer

Administrative Errors Errors that result from improper administration and

execution of the research

Types of administrative errors- Data processing errors (Coding and entry)

- Sample selection errora. Error in sample design which results in non-representative sample

- Interviewer error

- Interviewer cheating