effect of questionnaire structure on nonresponse ... - rti.org · study design (cont’d) • 6...

21
RTI International is a trade name of Research Triangle Institute www.rti.org Effect of Questionnaire Structure on Nonresponse and Measurement Error: Sequential vs. Grouped Placement of Filter Questions Lisa R. Carley-Baxter, RTI International Andy Peytchev, RTI International Michele Lynberg Black, CDC

Upload: others

Post on 20-Jul-2020

4 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

RTI International is a trade name of Research Triangle Institutewww.rti.org

Effect of Questionnaire Structure on Nonresponse and Measurement Error: Sequential vs. Grouped Placement of Filter Questions

Lisa R. Carley-Baxter, RTI InternationalAndy Peytchev, RTI International

Michele Lynberg Black, CDC

Page 2: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

2

www.rti.org

Overview

• Introduction• Sequential vs Grouped Structure• Hypotheses• Study Design• Study Specific Filter and Follow-up Questions• Results• Discussion

Page 3: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

3

www.rti.org

Introduction

• Structure of a questionnaire can impact survey estimates through multiple types of measurement error

• Researchers aim to structure questionnaires in way that respondents’ memories are organized

• Survey instruments often include follow-up questions that depend on response to a previous question (filter question)

Page 4: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

4

www.rti.org

Sequential versus Grouped Structure

• Sequential: – Filter Question 1 -> Follow-up Questions– Filter Question 2 -> Follow-up Questions

• Grouped: – Filter Question 1...Filter Question X– Follow-up Questions to Filter 1 (if endorsed)– Follow-up Questions to Filter 2 (if endorsed)

• Sequential structure is assumed to help recall, reduce measurement error, and possibly decrease item nonresponse

Page 5: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

5

www.rti.org

Literature

• Suggests that respondents may underreport in order to avoid additional questions– Kessler et al (1998), Duan et al (2007)– Respondents reported lower mental distress or mental

health service use when a sequential design was used– Both studies were very long face-to-face interviews– Both studies targeted minority groups that differ from the

general population on measures related to cognition and willingness to conduct surveys

• Research Question: Does a sequential design increase reporting in a 30 minute telephone survey of the general population?

Page 6: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

6

www.rti.org

Hypotheses

• Interview length will be longer in the sequential structure than in the grouped structure.

• Higher number of breakoffs, number of behaviors reported, and item nonresponse in the sequential structure compared to the grouped structure.

• Interviewers will prefer the grouped structure over the sequential structure.

Page 7: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

7

www.rti.org

Study Design

• National Intimate Partner and Sexual Violence Surveillance System (NISVSS)– National RDD survey of US adults 18+– Collects sensitive information on psychological aggression,

coercive control and entrapment, physical violence, stalking, and sexual violence (ever, past 3 yrs, past 12 months)

– Stratified dual-frame RDD sample design that included both landline and cell phone numbers

– Oversampled males during respondent selection– Respondent selection:

• 2 adults: Modified Rizzo method• 3+ adults: Most recent birthday method

Page 8: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

8

www.rti.org

Study Design (cont’d)

• 6 week pre-test conducted Nov – Dec, 2009– Included a 2 week nonresponse phase

• Lead letters sent to all landline cases for which an address was obtained

• Cases randomly assigned to either the sequential or grouped structure

• 784 total completed interviews– 495 sequential structure– 289 grouped structure

• AAPOR Response Rate #1 = 12.9%

Page 9: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

9

www.rti.org

NISVSS Filter and Follow-up Questions

• Filter: How many of your romantic or sexual partners have ever slapped you?– Follow-up: Please tell me the initials of the (first, etc) person

who slapped you.– Follow-up: How many times did (initials) ever slap you?

• Filter: How many of your romantic or sexual partners have ever pushed or shoved you?– Follow-up: Please tell me the initials of the (first, etc) person

who pushed or shoved you.– Follow-up How many times did (initials ever) push or shove

you?

Page 10: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

10

www.rti.org

Mean Length of Interview

Sequential 32.3Grouped 32.6

Sequential 30.3Grouped 31.0

Sequential 31.4Grouped 31.8

Females

Males

Total

Page 11: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

11

www.rti.org

Mean Breakoff Rates

Sequential 19.9Grouped 23.9

Sequential 16.7Grouped 19.8

Sequential 18.5Grouped 22.0

Females

Males

Total

Page 12: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

12

www.rti.org

Location of Breakoffs

Sequential Grouped Sequential Grouped Sequential GroupedPrior to Victimization Sections 24.2 40.4 15.9 44.1 20.9 42During Victimization Sections 63.6 51.1 70.5 55.9 66.4 53.1

After Victimization Sections 12.1 8.5 13.6 0 12.7 4.9**p<.01

Females Males** Total**

Page 13: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

13

www.rti.org

Mean Number of Behaviors Reported

Sequential 6.3Grouped 5.9

Sequential 3.5Grouped 4.6

Sequential 5.0Grouped 5.1

Males

Total

Females

Page 14: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

14

www.rti.org

Item Nonresponse

Sequential 0.5Grouped 0.2

Sequential 0.6Grouped 0.4

Sequential 0.5Grouped 0.4

Females

Males

Total

Page 15: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

15

www.rti.org

Regression Analysis

• Ran 6 Poisson regression equations by sex – Lifetime victimization– 5 of which used individual violence types as dependent

variable • Psychological aggression• Coercive control and entrapment • Physical violence • Stalking • Sexual violence

– 1 of which used any reported behavior experienced as the dependent variable

Page 16: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

16

www.rti.org

Regression Results

• Females:– Sequential structure was a significant predictor of increased

reporting:• Coercive control and entrapment• Physical violence• Sexual violence

• Males:– Sequential structure was a significant predictor of decreased

reporting:• Coercive control and entrapment• Sexual violence• Stalking

• Both: Age, age squared, and lower levels of education had stronger and more consistent effects on reports of experiencing the 5 individual violence types and any behavior

Page 17: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

17

www.rti.org

Interviewer Feedback

• Held meetings with interviewers throughout the data collection period

• Conducted an interviewer debriefing towards the end of the data collection period

• Interviewers consistently reported that they and respondents disliked the grouped structure– Main reason: Confusing to have to go back and ask follow-

up questions about specific perpetrators after the initial reports

• Reported sequential structure was easier to administer and clearer for respondents

Page 18: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

18

www.rti.org

Summary of Results

• Very few significant difference found between structures in the bivariate analysis– Only significant difference was where cases broke off: higher

breakoffs in victimization section for sequential structure

• Regression results paint a different picture– Sequential structure significant predictor of 3 violence types

each for both sexes (though different violent types, and opposite impact on reporting)

Page 19: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

19

www.rti.org

Discussion

• Most interesting finding:– Higher reports for females in sequential structure– Higher reports for males in grouped structure

• Interaction of gender and structure is of both theoretical and practical importance– Suggests that optimal questionnaire structure depends on

the sample members and topics of the questions

Page 20: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

20

www.rti.org

Discussion (cont’d)

• Sequential structure:– Higher breakoff rates within a section of detailed behavior

questions– But also overall reports for some of those questions may be

higher for the key population group of interest

• Important for studies where researchers are concerned about underreporting of experiences– Example: intimate partner and sexual violence

• Use of different structures may interfere with comparison across studies

Page 21: Effect of Questionnaire Structure on Nonresponse ... - rti.org · Study Design (cont’d) • 6 week pre-test conducted Nov – Dec, 2009 – Included a 2 week nonresponse phase •

5/18/2010

21

www.rti.org

Contact Information

Lisa [email protected]

www.rti.org/aapor