does balancing survey response reduce nonresponse bias?...does balancing survey response reduce...

14
Does balancing survey response reduce nonresponse bias? Barry Schouten and Fannie Cobben (Statistics Netherlands) Peter Lundquist (Statistics Sweden) James Wagner (University of Michigan) 68 th AAPOR, May 16 – 19, 2013, Session Implementing a Responsive Design – Moving from the Theoretical to the Practical

Upload: others

Post on 24-Jan-2020

12 views

Category:

Documents


0 download

TRANSCRIPT

Does balancing survey response reducenonresponse bias?

Barry Schouten and Fannie Cobben (Statistics Netherlands)Peter Lundquist (Statistics Sweden)James Wagner (University of Michigan)

68th AAPOR, May 16 – 19, 2013, Session Implementing a ResponsiveDesign – Moving from the Theoretical to the Practical

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 1

Summary

• Background and motivation• Empirical study• Theoretical considerations• Results• Conclusions and discussion

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 2

Adaptive/responsive survey designsAdaptive survey designs (ASD) and responsive survey designs(RSD) arise from the following viewpoints:• Persons/households may prefer different forms of

communication and interview, i.e. react differently todifferent design features

• Different design features are associated with different costs

In other words, there is a quality-cost differential.

The emergence of web, the technological advance of casemanagement systems and the gradual decrease of responserates are the main drivers for such designs.

MotivationEmpirical studyTheoryResultsDiscussion

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 3

Main components of ASD/RSDASD/RSD may be viewed as extensions to sampling designswhere instead of a single (uniform) strategy multiple candidatestrategies can be drawn.

Ingredients to ASD/RSD:• Registry data/frame data: static designs• Paradata/process data: dynamic designs• Proxy measures of survey error• Cost functions• Design features

ASD/RSD have focussed mostly on nonresponse error. Ifsurvey mode is one of the design features, then measurementdifferences need to be considered as well.

MotivationEmpirical studyTheoryResultsDiscussion

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 4

Proxy measure for nonresponse

MotivationEmpirical studyTheoryResultsDiscussion

Särndal’s BI

Särndal’s H3

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 5

Main criticisms1. Most studies/frameworks ignore variances2. Proxy measures are based on frame/registry data, and

paradata that may relate only weakly to survey variables3. Any balancing by design through ASD or RSD can be

achieved afterwards by adjustment on the same variables

This paper focusses on the last two criticisms:Does balancing survey response on available auxiliaryvariables from frame data, registry data, paradata reducenonresponse bias on other variables (most notably the surveyvariables)?

MotivationEmpirical studyTheoryResultsDiscussion

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 6

Side remarksSurvey variables and proxy measures• Survey target variables can be included in proxy measures

through imputation/projection• But leads to a contradiction: a stable model during data

collection is required to be able to compare measures. Theonly reason for continuing data collection is instability of suchmodels.

Intuition behind ASD/RSD:• A design that leads to survey response with larger detectable

nonresponse error reflects a weaker/less perfect datacollection process.

• Stronger deviation from MCAR(X) is stronger deviation fromMAR(Y,X)

MotivationEmpirical studyTheoryResultsDiscussion

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 7

Empirical study

We collected 13 datasets from three countries. Eachdataset consists of different surveys/waves/designs/levelsof effort. To each dataset a selection of auxiliary variableswas linked.

We sorted the auxiliary variables in a random order andcomputed R-indicators and coefficients of variation, addingvariables one at a time.Similarly conditional partial R-indicators and adjustednonresponse biases are computed to adjust for collinearity

Rank test constructed to investigate whether preferencesfor surveys/waves/designs/LOE are consistent

MotivationEmpirical studyTheoryResultsDiscussion

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 8

Empirical study

MotivationEmpirical studyTheoryResultsDiscussion

Type of comparisonDifferent

surveys/designsDifferentwaves

Different LOE Processingsteps

Health-NETH xCrime-NETH xHealth Crime-NETH xLabour Force-NETH xCons Sent-NETH xASD pilot-NETH x

STS business-NETH xLISS-panel-NETH xLiving cond-SWED xParty pref-SWED xSCA-USA xNSFG-USA xHRS-USA x

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 9

Some theoretical considerationsSuppose survey aims at population mean or total of some Y

It can be shown that remaining bias of HT estimator (weightedresponse mean) lies in an interval centered around

and with width

Standard estimators (GREG, IPW estimator and double robust)shift interval to zero and reduce width. But for every surveyvariable the bias interval width remains proportional to

MotivationEmpirical studyTheoryResultsDiscussion

)()()()( 22

2

2

2

2

XX cvcv

SS

),(),()()(

XX corYcorSYS

),(1),(1)()( 22

XX corYcorSYS

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 10

Some theoretical considerations• If auxiliary variables would be a random draw from the

universe of all possible variables, then• Under the same condition, a larger implies a larger

remaining nonresponse bias for all standard estimators forany other arbitrary variable.

• If auxiliary variables would be a random draw from thesubset of the universe of all possible variables that relate tothe survey topics, then the same results for any othervariable from this subset

• In surveys with many variables or in panels this may besufficient motivation.

• In surveys with one or a few main variables, the mainvariables may just be the exceptions.

MotivationEmpirical studyTheoryResultsDiscussion

)()( cvcv X )( Xcv

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 11

Results per dataset

MotivationEmpirical studyTheoryResultsDiscussion

Dataset p-value based on independence p-value based on simulationR CV Pc B R CV Pc B

HS 0.03 0.00 0.32 0.18 0.65 0.42 0.48 0.17CVS 0.50 0.03 0.97 0.12 0.93 0.41 0.96 0.12HS – CVS 0.14 0.00 0.96 0.00 0.98 0.70 1.00 0.00LFS 0.00 0.01 0.82 0.82 0.22 0.62 0.88 0.78SCS 0.00 0.00 0.00 0.00 0.17 0.02 0.07 0.01SCSASD 0.06 0.06 0.06 0.06 0.27 0.27 0.07 0.02LISS 0.00 0.00 0.03 0.00 0.00 0.07 0.27 0.00STS-IND 0.01 0.01 0.72 0.72 0.09 0.12 0.74 0.65STS-RET 0.01 0.03 0.50 0.88 0.09 0.41 0.53 0.82LCS 0.00 0.00 0.00 0.07 0.18 0.00 0.01 0.08PPS 0.00 0.00 0.00 0.10 0.06 0.00 0.00 0.13SCA 0.08 0.01 0.81 0.35 1.00 1.00 0.92 0.31NSFG 0.01 0.01 0.01 0.77 0.24 0.29 0.04 0.68HRS 0.04 0.04 0.36 0.36 0.50 0.56 0.46 0.27

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 12

Results for combined datasets

MotivationEmpirical studyTheoryResultsDiscussion

Number of inversions p-valueExpected Pc B Pc B

StatNetherlands

189.5 142 97 <0.001 <0.001

StatSweden/ISR

118.5 66 97 <0.001 0.02

All 308 208 194 <0.001 <0.001

Implementing a Responsive Design – Moving from the Theoretical to the Practical, AAPOR 2013 13

Discussion• If R-indicator and coefficient of variation prefer a design for

an arbitrary set of auxiliary variables, then in expectationthey prefer that design also for any other arbitrary variable

• Furthermore, the expected bias on any other arbitraryvariable after adjustment will be smallest for that design.

• However, many auxiliary variables are needed to have ahigh statistical power of identifying the strongest design

• The combined datasets in our study point at empiricalconsistency of design preferences and suggest thatbalancing by design on average reduces nonresponse biasregardless of adjustment afterwards

• We would like to call on others to perform similar analyses.

MotivationEmpirical studyTheoryResultsDiscussion