Survey, Measurement, and Questionnaire Issues

Review, Recap, and Study Guide

Statistical Inference and Survey Research

• Surveys and Survey Modes (Chapter 9: Be familiar with the advantages of various modes)

• Measurement (Chapter 10, Levels of measurement)• Questions and Questionnaire Design (Chapter 11)• Sampling Introduction (Chapter 12)• Hypothesis Testing, T-tests, Correlation (Chapters 16,

17, 18)

Determine How the Results will be Implemented

• Will there be performance based, merit raises, bonuses for better performers?

• Will there be specific training, or plans developed for improvement on low performing branches.

• Will branches be closed, consolidated, managers moved to new locations?

2-3-4: Importance of understanding all data analysis.

• To ensure the implementation of the results, determine what the final report should contain and how it should look.

• Specify the analyses necessary to fill in the blanks in the research report.”

• Determine the kind of data that must be assembled to carry out these analyses.

• Managers must be in agreement about what data will be collected and how it needs to be presented so that it can affect decisions.

Would you rather identify the top performing branches or the bottom performers for bonuses/additional training?• What should be included in final report?—

Should top or bottom performers be identified? What are the implications?

• Should percentages be used or means?

Statistical Inference

• Generalizing to the characteristics of a population from parameters taken from samples.

• “The sample mean is an approximation of the average from a census of respondents.”

• “The true population mean will lie within +/- two units of standard error of the sample mean.

• Using sampling and statistics to create estimates of “true” population parameters.

Parameter Estimation

• Calculating a confidence interval and stating whether a stated proportion falls within or outside that proportion.

• Comparing the difference between two means to the difference that would be expected by sample-to-sample differences.

• Rejecting, or Failing to Reject

Standard Error

• The standard deviation in the theoretical sampling distribution, the sampling distribution of the mean.

• Sample-to-sample variation for samples of a specific size, given assumptions about population variability.

• Significant differences typically require two units of this statistics.

• Sample standard deviation divided by the square root of the sample size.

Standardization: The Reason

• Agreement across respondents in the nature of their information

• Agreement over time between studies• Suitability to numeric coding of responses an

computer analysis• Agreement across studies and researchers

concerning analysis—generalization and replication.

“So, you’d like to do a survey.”

• Quality of the data• Complexity and versatility• Quantity of data• Sensitivity of data• Response rate• Speed• Cost per completed survey

Quality of Data Over-riding concern of all survey research is the

control over error. Numerous opportunity for errors and biases to

contaminate the data collection effort. Sampling, sample control, and nonresponse error. Administrative and systematic error Respondent error and falsification.

Complexity and Versatility of the Survey

Ability to incorporate visual props or cues, such as packaging or advertising.

Explain unclear directions. Ability to probe, ask follow-up questions

Quantity of Data What depth of data is required to address the

research question? Do we need multiple-item questions to obtain a

valid response? How long can you keep the respondent interested

in the subject matter before termination? Anticipated length of survey directly affects

response rate. Keep the survey as short as possible.

Response Rate, Speed, and Cost

Cost of a various survey methods can only be determined via the cost per completed survey, represented by the response rate.

The majority of survey costs are fixed, rather than variable. Effective interviewer utilization is then the key factor keeping cost per completed survey low.

Speed refers to the time needed to complete the data collection process. High interviewer utilization results in high speed research, quick turn-around for clients.

Telephone Interviewing High speed—low cost per completed survey. Near all households are accessible. Convenient for respondent, good response

rates. Most popular / most frequently used survey

research technique. Intrusive.

Telephone Surveys

• Random-digit dialing eliminates the need for a fixed sampling frame.

• Caller ID, high cell phone adoption, caller refusals increasing the challenge to this mode.

• Difficult to achieve a high incidence rate on anything beyond widely adopted convenience products.

Mail Surveys

Best in maintaining respondent anonymity and best for sensitive subject matter.

Best in providing low cost geographic coverage.

Absolute worst in speed, almost completely abandoned by academic marketing research

Lowest in quality control—who’s completing the survey?

Nothing Stops the Mail• Attractive front-end costs per “piece”; still well

below $1 in many instances.• Response rates are helped with a cover letter that

confirms the importance of the survey—cannot overcome initial reluctance or refusal.

• Small cash incentives not tremendously effective, but significant $$$ necessary to achieve meaningful response rates.

• What other pieces of mail will accompany your survey?

Personal Interviewing Tremendous in flexibility, good for situations

where some probing is necessary, and questions are more complex in nature.

Easy to incorporate visual cues. Sample bias due to interviewer selection Malls are where the people are -- they're not

at accessible during the day.

Face-to-Face

• Necessary to achieve contact with executives, business-to-business customers and difficult to reach subjects

• Frequently used with systematic sampling in retail situations—appropriateness must be considered for generalizability.

• Having an interviewer present is necessary to complete lengthy, “high quantity” data collection efforts.

Computer Administered

• Use of videos, pictures, graphics and error-free data entry

• No anonymity• Simultaneous, or “real-time” capture of data• High speed data collection

Software Driven

• “Paperless” data collection• Direct entry of data into spreadsheet• Interviewer follows script, certain answers

direct subject to appropriate areas.• Set-up costs are significant, and forms a

“barrier” to non-sophisticated users.• Sophisticated non-respondent follow up

essential defeats any effort to assure anonymity.

Respondent Issues

• Incidence rate, low incident rate means difficulty to conduct random sampling and complete the survey

• Willingness to participate• Ability to participate• Diversity of respondents, high degree of diversity in

age, background, education, occupation, means greater difficulty to develop an easy to administer survey.

Taking the “Plunge”

• Are the research objectives only attainable through survey research? How many objectives are sought from a single survey?

• Is the sampling frame current? What inferences will be drawn from the sampling frame to the target population?

• What is the anticipated response rate? What will be the cost per completed survey?

…maybe a survey is not for you.

• Secondary data analysis and use of syndicated surveys

• Experiments• Observation• Depth interviewing and a qualitative research

design

Measurement and Coding

• Determine the type of questions that should be asked.

• Determine the type of analysis that can be conducted.

• Determine the validity and reliability of a measure.

Levels of Measurement

• Nominal coded data: Ability to distinctly categorize, or conversely, allows the determination of equality.

• Ordinal coded data: Allows determination of magnitude, rank, greater or less than.

• Interval coded data: Captures the distance apart two or more respondents are with respect to an attribute.

• Ratio data possess a natural or absolute zero, indicating a true absence of a characteristic. Permits statements concerning the equality of ratios.

Each level bears all the properties of the one listed above it.

Properties of Good Measures

• Sensitivity: Will it reflect differences across individuals that are meaningful to marketers?

• Reliability: Are the questions free from error, will they be interpreted the same way by different respondents?

• Validity: Are the questions measuring what they are intended to measure?