John Barrett Rudly Raphael
President, QuestionPro AudienceCEO, Priority Metrics Group
www.pmgco.com
QuestionPro Audience
20 Million Respondents
B2B and B2CSolutions
Mobile Research Panels
10 SpecialtyPanels
• Global Scalability
• Quality Recruitment
• Ongoing quality control
• Panel Respondents
• Business Panels
• Customer Panels
• Real time results
• Mobile community
• Robust platform
• Veterinarians• Building
Contractors• Pet Owners• Registered Voters• And more!
#DataQualitytips
Agenda1. Data quality2. Data quality problems3. Conversion4. Interviewer training 5. Reducing non-response bias6. Questionnaire design7. Question Construction
Improving Data Quality
Importance of Data Quality
Quality Problems – Total Survey Error
Groves (1989) identifies three categories of error:
1. Coverage – some members of the population under study do not have a known nonzero chance of being included in the sample.
2. Measurement effect – the instrument or items on the instrument are constructed in such a way to produce unreliable or invalid data.
3. Non-Response effect – nonrespondents in the survey sample differ from respondents in ways that are germane to the objectives of the survey.
Total Survey Error … for the rest of us
Measurement Effect Non-Response Effect
Data Quality Problems
1.Sampling2.Non-Response3.Questionnaire Design4.Execution/Methodology5.Quality Control6.Analysis/Reporting7.Costs
Poll: Which of the following has a direct effect on data quality, as it relates to your research?Please select the top 3. a. Poor sampleb. Poor questionnaire designc. Poor execution/methodologyd. Poor quality control procedurese. Poor survey incentivesf. Poor open-end responses g. Limited analysis/reporting toolsh. Limited research budget http://buff.ly/2n7Lk30
Non-Response
Non-Response Documentation
• Document non-response using AAPOR categories:❏Interview❏Eligible, non-interview❏Unknown eligibility, non-interview❏Not eligible
• Identify correlates of non-response
Non-Response vs. Maximizing Response Rates
Survey of healthcare leaders: web-based, self-report, four follow-ups• Overall 95% response rate, examined
results in each of five waves
Response Wave and Evaluative Attitudes Assessed Using a Multi-Item Scale: Mean and 95 Percent Confidence Intervals for the Alignment and Commitment Scale (ACS).
Conclusion: Although high response rates are desirable because of their effect on precision and power, absolute thresholds representing “adequate” survey response rates may not be accurate.
Source: “Response Rates, Nonresponse Bias, and Data Quality: Results from a National Survey of Senior Healthcare Leaders, “Meterko et al, Public Opinion Quarterly, January 27, 2015
Conversion: Non-Respondent Sees the Light!
✓ Different versions of survey introduction
✓ Study contact rules✓ Incentives✓ Interviewer style and training
Research literature reports telephone conversion rates of 5% to 40%.
Interviewer Training – The Often Overlooked Nugget
✓ Interviewer hiring practice
✓ Thorough interview training❏Survey background❏Questionnaire familiarity❏5-second rule: Introduction, survey purpose❏Timing of contact❏Personality of interviewer
✓ Multiple contacts❏May increase response rates dramatically The right person makes a
difference.
Reducing Non-Response Bias
• Good survey design increases response rates• Good survey communication increases response rates
Incentives can increase response rates
CleverForeshadow
✓ Do not cross “coercive threshold”✓ Donation to charity (business)✓ Prepaid (vs. promised) incentive
(consumer)✓ Differences decrease with follow-up✓ May lead to lengthier open-end
answers
Source: “National Survey of Physicians to Determine the Effect of Unconditional Incentives on Response Rates of Physician Postal Surveys,”Abdulaziz et al, BMJ Open, Volume 5, Issue 2
Reducing Non-Response Bias
Interviewers can gather information about non-respondents
• Observable characteristics
• Evaluations of engagement, honesty, ability
• Visual and verbal clues
• Observable characteristics
• Evaluations of engagement, honesty, ability
• Verbal clues
Questionnaire Design
Questionnaire Design Starts with Clear Objectives
Start End
Good Design Practices
✓ Promise (or at least offer) anonymity or confidentiality
✓ Some form of relationship (known brand or existing customer) increases response rate
✓ Simplify the perceived task• Preserve white space • Design a logical flow• Group common items together – similarity and proximity• Don’t ask unnecessary questions• Minimize task difficulty
✓ Avoid use of jargon and notation
✓ Include relevant questions using question and page logic based on previous answers
Good Design Practices
Improvements
Explanation of task
Question wording
Confidentiality
Positive Elements
Clean, simpleLogical flow
Comment space
Visual Display
✓ No more than four visual elements
✓ Identify new information with distinct colors and/or sizes
✓ Bold type used in favor of italics, underline, or upper case
✓ Use highlights and graphics sparingly
✓ Differentiate important elements with large, bright or distinctive colors
Visual Display
Positive Elements
Logical flow
Improvements
Jargon throughout• Contract• Self-discover• Influencing style
37 visual elements
Everything is brightEverything is newEverything is important
Question Construction
✓ Use conversational norms where possible
✓ Use common words with single primary definition
• Few letters and syllables
• Easy to pronounce
• Avoid abbreviations
✓ Avoid asking certain types of questions:
• Opinions held at prior times
• Explain prior behavior or thoughts
✓ Balance use of closed- and open-end questions appropriately
• Use open-end for numeric answers and categorical questions with unknown breadth of possible answers
Question Construction – Grice’s Maxims
The maxim of quantity, where one tries to be as informative as one possibly can, and gives as much information as is needed, and no more.
The maxim of quality, where one tries to be truthful, and does not give information that is false or that is not supported by evidence.
The maxim of relation, where one tries to be relevant, and says things that are pertinent to the discussion.
The maxim of manner, when one tries to be as clear, as brief, and as orderly as one can in what one says, and where one avoids obscurity and ambiguity.
Question Construction – Response Process
(Tourangeau Model)
Response based on recall or educated guess from cues or inferences, often requires choice of answers to report and how to report
(agree or strongly agree)
Comprehension how respondents understand the
questions and infer the question’s point
Judgment assessment of completeness or sufficiency of
information/opinion, appropriateness and manner of using inferences, and how to
transform retrieved information into appropriate answer
Retrievalrecalling information from long-term memory (behavioral questions) or a
preformed opinion (attitude questions), but retrieval is rarely complete
Question Construction – Bad Examples
1. What is your frequency of utilization of retail travel agents?
2. Are you against the restrictive House Bill 935?
3. Have you visited ___ in the past and did you enjoy your visit?
4. Please rank order the following with 1 being the most important and 7 being the least important
5. When you purchased ____ what other options did you consider?
6. Did you use the features on our website?
7. In order to speed up your shopping experience, have you used the self check-out lanes?
8. How likely are you willing to pay $50 for a ____?
Rating Scales
✓ Label options with words, not numbers
✓ Ensure range of options covers all points on continuum
✓ For bipolar constructs use 7-point scale
✓ For single constructs use 5-point scale
✓ Don’t offer “Don’t Know” response
✓ Rotate order of response on categorical questions and wherever else it is practical
How satisfied were you with the speed of check-out?
Communication Plan
✓ Invitations and advance/introductory letters
✓ Self-administered questionnaires
✓ Reminder letters/emails
Self-Administered
Interviewer-Administered
Computer-Administered
Importance of Pre-Test
Traditional pre-test – match survey method to complexity
Cognitive pre-test – think-aloud
Conclusions – Part 1
To address non-response bias:
1. Document and categorize non-responses
2. Look for correlations
3. Don’t be fooled by high response rate
4. Convert!
5. Hire the best interviewers and train them well
6. Use well-designed questionnaires
7. Use incentives thoughtfully
8. Gather information from interviewers about non-respondents (back to #2)
Conclusions – Part 2
To build the best questionnaires:
1. Start with the end in mind
2. Follow GDPs
3. Establish relationship
4. Use visual elements in limits
5. Craft questions that are clear and direct
6. Test against Grice’s Maxims
7. Think like a respondent (Tourangeau Model)
8. Benefit from research learnings about rating scales
9. Develop and implement a communication plan
10. Pretest, and if necessary, pre-test again
Poll Results
Poll: Which of the following has a direct effect on data quality, as it relates to your research?